24-Sep-86 00:35:55-PDT,17368;000000000000 Mail-From: NEUMANN created at 24-Sep-86 00:33:45 Date: Wed 24 Sep 86 00:33:45-PDT From: RISKS FORUM (Peter G. Neumann -- Coordinator) Subject: RISKS-3.64 DIGEST Sender: NEUMANN@CSL.SRI.COM To: RISKS-LIST@CSL.SRI.COM RISKS-LIST: RISKS-FORUM Digest Wednesday, 24 September 1986 Volume 3 : Issue 64 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Sane sanity checks / risking public discussion (Jim Purtilo) More (Maybe Too Much) On More Faults (Ken Dymond) Re: Protection of personal information (Correction from David Chase) Towards an effective definition of "autonomous" weapons (Herb Lin, Clifford Johnson [twice each]) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM) (Back issues Vol i Issue j available in CSL.SRI.COM:RISKS-i.j. Summary Contents in MAXj for each i; Vol 1: RISKS-1.46; Vol 2: RISKS-2.57.) ---------------------------------------------------------------------- Date: Tue, 23 Sep 86 12:54:10 EDT From: Jim Purtilo To: RISKS@csl.sri.com Subject: Sane sanity checks / risking public discussion [Regarding ``sanity checks''] Let us remember that there are sane ``sanity checks'' as well as the other kind. About 8 years ago while a grad student at an Ohio university that probably ought to remain unnamed, I learned of the following follies: The campus had long been doing class registration and scheduling via computer, but the registrar insisted on a ``sanity check'' in the form of hard copy. Once each term, a dozen guys in overalls would spend the day hauling a room full of paper boxes over from the CS center, representing a paper copy of each document that had anything to do with the registration process. [I first took exception to this because their whole argument in favor of "computerizing" was based on reduced costs, but I guess that should be hashed out in NET.TREE-EATERS.] No one in that registrar's office was at all interested in wading through all that paper. Not even a little bit. One fine day, the Burroughs people came through with a little upgrade to the processor used by campus administration. And some "unused status bits" happened to float the other way. This was right before the preregistration documents were run, and dutifully about 12,000 students preregistration requests were scheduled and mailed back to them. All of them were signed up "PASS/FAIL". This was meticulously recorded on all those trees stored in the back room, but no one wanted to look. I suppose a moral would be ``if you include sanity checks, make sure a sane person would be interested in looking at them.'' [Regarding break-ins at Stanford] A lot of the discussion seems to revolve about ``hey, Brian, you got what you asked for'' (no matter how kindly it is phrased). Without making further editorial either way, I'd like to make sure that Brian is commended for sharing the experience. Sure would be a shame if ``coming clean'' about a bad situation will be viewed as itself constituting a risk... [I am delighted to see this comment. Thanks, Brian! PGN] ------------------------------ Date: 23 Sep 86 09:18:00 EDT From: "DYMOND, KEN" Subject: More (Maybe Too Much) On More Faults To: "risks" The intuitive sense made by Dave Benson's argument in RISKS 3.50, that >We need to understand that the more faults found at any stage to >engineering software the less confidence one has in the final product. >The more faults found, the higher the likelihood that faults remain. seems to invite a search for confirming data because there are also counter- intuitive possibilities. For example there is the notion that the earlier in the life cycle errors are detected, the cheaper to remedy them. There is a premium on finding faults early. And the further notion that with tools for writing requirements in some kind of formal language that can be checked for syntactic and semantic completeness and consistency, it's possible to detect at least some errors at requirements stage that may not have been caught till later. So SE projects using these and similar methods for other stages in the life cycle would tend to show more errors earlier. Would the products from these projects be therefore less reliable than others made with, say, more traditional, less careful, design and programming practice ? Dave makes the further argument in RISKS 3.57: >Certain models of software failure place increased "reliability" on >software which has been exercised for long periods without fault. [...] The models of software reliability exist to order our thinking about reliability and to help predict behavior of software systems based on observation of failure up to the current time. The models that show failures clustered early in time and then tapering off later do indeed model an intuition but maybe not the one that more faults mean yet more faults. Hence the need for data. I suspect that the reality as shown by data, if it exists, would be more complex than intuition allows. More errors discovered so far may just mean better software engineering methods. As far as other engineering fields, the failure vs time curve in manufactured products is often taken to be tub-shaped, not exponentially decaying. So more failures are expected at the beginning and near the end of the useful life of a "hard" engineered product. Of course, "an unending sequence of irremediable faults" should be the kiss of death for any product, whether from hard engineering or soft. But the trick is in knowing that the sequence is unending. The B-17, I seem to remember reading, had a rather rocky development road in the 1930s, yet was not abandoned. Was it just that the aeronautical engineers at Boeing then had in mind some limit on the number of faults and that this limit was not exceeded? It might be easy to say in hindsight. On the other hand, sometimes foresight, in terms of spotting a poor design at the outset, makes a difference, as in the only Chernobyl-type power reactor outside the Soviet block. It was bought by Finland (perhaps this is what "Finlandization" means ?). However the Finns also bought a containment building from Westinghouse. Ken Dymond ------------------------------ From: David Chase To: Andy_Mondore%RPI-MTS.Mailnet@MIT-MULTICS.ARPA Cc: risks@csl.sri.com Date: Tue, 23 Sep 86 08:56:18 EDT Subject: Re: Protection of personal information [The two participants requested this clarification be included for the record... PGN] You misinterpreted my message in a small way; I was writing about a university attended by a friend, NOT Rice university. To my knowledge, Rice has been very good about protecting its students' privacy. My student number is NOT my social security number, though the university has that number for good reasons. I do not want anyone to think that I was talking about Rice. David ------------------------------ Date: Tue, 23 Sep 1986 18:00 EDT From: LIN@XX.LCS.MIT.EDU To: risks@CSL.SRI.COM Subject: Towards an effective definition of "autonomous" weapons [THE FOLLOWING DISCOURSE INVOLVING CLIFF AND HERB IS LIKELY TO CONTINUE FOR A WHILE ON ARMS-D. PLEASE RESPOND TO HERB LIN, NOT TO RISKS ON THIS ONE. HERB HAS VOLUNTEERED TO SUBMODERATE, AND THEN SUBMIT THE RESULTS TO RISKS. PGN] From: Clifford Johnson An "autonomous weapon" [should be] defined to be any weapons system which is de facto preprogrammed to take decisions which, under the law of nations, require the exercise of political or military discretion. It's not a bad first attempt, and I think it is necessary to get a handle on this. With the realization that you have done us a service in proposing your definition, let me comment on it. I don't understand what it means for a weapon to "take a decision". Clearly you don't intend to include a depth charge set to explode at a certain depth, and yet a depth charge could "decide" to explode at 100 feet given certain input. What I think you object to is the "preprogrammed" nature of a weapon, in which a chip is giving arming, targeting and firing orders rather than a human being. What should be the role of the human being in war? I would think the most basic function is to decide what targets should be attacked. Thus, one modification to your definition is An "autonomous weapon" [should be] defined to be any weapons system which is preprogrammed to SELECT targets. This would include things like roving robot anti-tank jeeps, and exclude the operation of LOW for the strategic forces. But this definition would also exclude "fire-and-forget" weapons, and I'm not sure I want to do that. I want human DESIGNATION of a target but I don't want the human being to remain exposed to enemy fire after he has done so. Thus, a second modification is An "autonomous weapon" [should be] defined to be any weapons system which is preprogrammed to SELECT targets in the absence of direct and immediate human intervention. But then I note what a recent contributor said -- MINES are autonomous weapons, and I don't want to get rid of mines either, since I regard mines as a defensive weapon par excellence. Do I add mobility to the definition? I don't know. ------------------------------ Date: Monday, 22 September 1986 21:43-EDT From: Clifford Johnson To: Arms-Discussion [FORWARDED BY HERB LIN] Subject: Towards an effective defintion of "autonomous" weapons There's great difficulty in defining "autonomous weapons" so as to separate some element that seems intuitively "horrible" about robot-decided death. But a workable definition is necessary if, as CPSR tentatively proposes, such weapons are to be declared illegal under international law, as have chemical and nuclear weapons. (Yes, the U.N. has declared even the possession of nukes illegal, but it's not a binding provision.) The problem is, of course, that many presently "acceptable" weapons already indiscrminately-discriminate targets, e.g. target-seeking munitions and even passive mines. Weapons kill, and civilians get killed too, that's war. Is there an element exclusive to computerized weapons that is meaningful? I don't have an answer, but feel the answer must be yes. I proffer two difficult lines of reasoning, derived from the philosophy of automatic decisionmaking rather than extant weapon systems. First, weapon control systems that may automatically target-select among options based upon a utility function (point score) that weighs killing people against destroying hardware would seem especially unconscionable. Second, but this presumes a meaningful definition of "escalation," any weapons system that has the capability to automatically escalate a conflict - and is conditionally programmed to do so - would also seem unconscionable. Into the first bracket would conceivably fall battle management software and war games, into the second would fall war tools that in operation (de facto) would take decisions which according to military regulations would otherwise have required the exercise of discretion by a military commander or politician. The latter category would embrace booby-trap devices activated in peacetime, such as mines and LOWCs; and here there is the precedent of law which prohibits booby traps which threaten innocents in peacetime. Perhaps the following "definition" could stand alone as *the* definition of autonomous weapons to be banned: An "autonomous weapon" is defined to be any weapons system which is de facto preprogrammed to take decisions which, under the law of nations, require the exercise of political or military discretion. This might seem to beg the question, but it could be effective - military manuals and international custom is often explicit on each commanders' degree of authority/responsibility, and resolving whether a particular weapon was autonomous would then be a CASE-BY-CASE DETERMINATION. Note that this could, and would, vary with the sphere of application of the weapons system. This is reasonable, just as there are circumstances in which blockades or mining is "legal" and "illegal." Of course, a case-in-point would be needed to launch the definition. Obviously, I would propose that LOWCs were illegal. How about battle management software which decides to engage seemingly threatening entities regardless of flag, in air or by sea? Any other suggestions? Does anyone have any better ideas for a definition? ------------------------------ Date: Tue, 23 Sep 1986 18:09 EDT From: LIN@XX.LCS.MIT.EDU To: arms-d@XX.LCS.MIT.EDU, risks@CSL.SRI.COM Subject: Towards an effective definition of "autonomous" weapons In thinking about this question, I believe that ARMS-D and RISKS could perform a real service to the defense community. There is obviously a concern among some ARMS-D and RISKS readers that autonomous weapons are dangerous generically, and maybe they should be subject to some legal restrictions. Others are perhaps less opposed to the idea. It is my own feeling that autonomous weapons could pose the same danger to humanity that chemical or biological warfare pose, though they may be militarily effective under certain circumstances. I propose that the readership take up the questions posed by Cliff's recent contribution: What is a good definition of an autonomous weapon? What restrictions should be placed on autonomous weapons, and why? How might such limits be verified? Under what circumstances would autonomous weapons be militarily useful? Should we be pursuing such weapons at all? How close to production and deployment of such weapons are we? Maybe a paper could be generated for publication? ------------------------------ Date: Tue, 23 Sep 86 18:16:46 PDT From: Clifford Johnson To: ARMS-D@XX.LCS.MIT.EDU [FORWARDED BY HERB LIN] Subject: Towards a definition of "autonomous" weapons > I don't understand what it means for a weapon to "take a decision". > Clearly you don't intend to include a depth charge set to explode at a > certain depth, and yet a depth charge could "decide" to explode at 100 > feet given certain input. My concept of "decision" embraces *all* manner of conditional executions, delimited expressly by the customary law that recognizes certain "special" decisions require human participation. I know of no precedent suggesting that "aiming" at a properly comprehended target requires such discretion, which is the logical mechanism that eliminates depth charges from my definition. > What should be the role of the human being in > war? I would think the most basic function is to decide what targets > should be attacked. The question I'm concerned with is "What should/could be the role of law in precluding the unconscionable automation of war?" With the basic thrust on target selection I agree, but to include it I would amend my definition by referencing "fatal consequences" (generalization of "targets"): AN "AUTONOMOUS WEAPON" IS DEFINED TO BE ANY WEAPONS SYSTEM WHICH IS DE FACTO PREPROGRAMMED TO TAKE DECISIONS WHICH, DUE TO THEIR POTENTIALLY FATAL CONSEQUENCES, UNDER THE LAW OF NATIONS REQUIRE THE EXERCISE OF HUMAN DISCRETION AT THE TIME THEY ARE TAKEN. N.B. The "de facto" is important to exclude the excuse that a human might "override" the weapons' decision, when, in practice, he/she wouldn't be competent to do so. > But then I note what a recent contributor said -- MINES are autonomous > weapons, and I don't want to get rid of mines either, since I regard > mines as a defensive weapon par excellence. Do I add mobility to the > definition? I don't know. It seems such confusions are inevitable when seeking a primarily technological definition of "autonomous" weapons. My "trick" is to reverse the analysis, placing the burden upon the human judgment that some kinds of decisions are unconsionably automated. The nature of and context of the particular decision becomes the focus. This isn't a cop-out, but would build upon established military/international custom. As I have intimated, it would seek precedents where particular decisions would "initiate" or "escalate" a conflict, or be beyond the ordinary authority of the officer operating the system. [RESPONSES TO LIN@XX.LCS.MIT.EDU please.] ------------------------------ End of RISKS-FORUM Digest ************************ -------