19-Jan-86 18:46:23-PST,11810;000000000000 Mail-From: NEUMANN created at 19-Jan-86 18:44:48 Date: Sun 19 Jan 86 18:44:48-PST From: RISKS FORUM (Peter G. Neumann, Coordinator) Subject: RISKS-1.41 Sender: NEUMANN@SRI-CSL.ARPA To: RISKS-LIST@SRI-CSL.ARPA RISKS-LIST: RISKS-FORUM Digest, Sunday, 19 Jan 1986 Volume 1 : Issue 41 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS sponsored by the ACM, Peter G. Neumann, moderator Contents: On a Clear Day You Can See Forever ... or Nothing At All (Peter G. Neumann) Unreleased SDIO Computing Panel Report: Specialists Fault `Star Wars' Work Man in the loop and magnetic bottles (Jon Jacky) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@SRI-CSL.ARPA, Requests to RISKS-Request@SRI-CSL.ARPA.) (Back issues Vol 1 Issue n stored in SRI-CSL:RISKS-1.n.) ---------------------------------------------------------------------- Date: Sun 19 Jan 86 16:29:27-PST From: Peter G. Neumann Subject: On a Clear Day You Can See Forever ... or Nothing At All To: RISKS@SRI-CSL.ARPA This issue of RISKS once again concerns the topic of SDI, both in its own right and as being illustrative of a collection of risks from which we can learn for other applications. The problems that arise in considering SDI are some of the most fundamental problems affecting our lives at the moment. The news item that follows this message was probably seen by most of you who read newspapers. It is included here for the record, and for any of you who missed it. Occasionally I get a complaint that the RISKS Forum seems biased. Some of those comments have come from proponents of SDI. I continually seek to include both sides of any critical issue. If only one side is willing to speak out, RISKS may appear biased to those who do not want to get drawn into the discussion. That is unfortunate, particularly if one side can eschew debate and then complain that the debate is rigged. Oh, well, we are damned if we do, and damned if we don't. Our ACM sponsorship stated at the outset of this Forum that we should make every attempt to be nonpartisan. I have on various occasions offered RISKS as an outlet to the SDI panel, but have been rebuffed with "RISKS is biased". Too bad. It looks as if some serious though went into the report that is described below, and it should certainly have the benefit of exposure to the social process. But if the proponents of SDI are hiding under a protective shield other than the one they are trying to create, then that is an anti-social process instead. As I have noted earlier, it is certainly easier to criticize others than to offer alternatives. Generally, the proof of the pudding is in the eating. Since the pudding has not jelled yet (indeed, it has not been cooked, which is fortunate in that suitable recipes do not yet exist), discussion is based on such things as dietary customs, a liking for untasted goodies, bad experiences with the effects of overly rich food, or allergy to pudding. De gustibus non disputandem est. It is significant that software problems are explicitly identified in the cited report as being critical to SDI. (That conclusion was certainly self-evident a priori, but its being emphasized will certainly point to many problems that must be addressed -- not just for SDI). The risks therein are enormous, and I hope that RISKS will objectively discuss them (along with other risks) and what can be done to avoid them. SOFT-ENG should also have much valuable material on the techniques and role of software engineering. However, as a meta-issue that is certainly relevant here, it is a tragedy of our times that software engineering techniques are still not widely used by the commercial developers of critical software. Some of those developers are literally many years behind the state of the art. The fault is probably distributed in many places, such as researchers who do not make their research and development tools really amenable to realistic use; software developers who take too many short-cuts and are too concerned with profits; government administrators who do not insist on higher-quality software with carefully preestablished requirements, and who do not follow through with strict control measures and penalties; politicians who create and perpetuate the bureaucracy; and the general public -- which puts up with it all. The waste is enormous -- not only in dollars, but in human resources and misplaced efforts. (I am referring here generically to a wide class of developments, not to SDI alone.) Peter ------------------------------ Date: Sat, 18 Jan 86 12:07:27 est From: Walter Hamscher To: risks@MIT-HTVAX.ARPA Subject: Unreleased SDIO Computing Panel Report >From the Boston Globe, page 1, Saturday, 18 Jan 1986 SPECIALISTS FAULT `STAR WARS' WORK By Fred Kaplan Globe Staff WASHINGTON - In a report commissioned by the Pentagon's Strategic Defense Initiative Office, a group of top computer software experts concludes that the SDI office is going about the task of building a `star wars' missile-defense system the wrong way. The report does say developing a proper software program for SDI is feasible. However, it says the SDI office and its defense contractors are assuming they can develop the `star wars' weapons and sensors first and write its computer software afterward - when, in fact, an effective defense will be impossible unless this order is reversed. The authors of the report, all avowed supporters of the SDI program, met for 17 days last summer and held further discussions before writing the report. The report was submitted to the SDI office last month, and has not been released publicly. Software must be programmed to enable automatic communication between the satellites that detect Soviet missiles and the SDI weapons that will shoot the missiles down; between these weapons and other sensors that can distinguish missiles from decoys and assess whther the target was hit or missed; and between this entire network and political authorities on the ground. Hundreds of satellites, battle stations, sensors, giant space mirrors and other devices would be involved. Computations must be made, and orders must be given, in a matter of microseconds, with continuous updates and revisions. The report says all the various designs for strategic defense systems proposed thus far demand "excessively sophisticated software" that "cannot be adequately tested." A design "that cannot be tested ... is of no value," the report says. And "excessively complex software cannot be produced at any cost." John Pike of the Federation of American Scientists, a critic of SDI who did not serve on the panel that wrote the report, puts the problem this way: "It's like buying a home computer first and then discovering that the software you need won't run on it. Or it's like buying a Betamax and then discovering that your favorite movies are only on VHS. "This report," Pike continues, "says a lot of the money in the [SDI] budget now is wasted because you'll end up buying the wrong machines." The report emphasizes that computer software programming is still a young field with many unknown elements. The report states, "The panel expects no technological breakthrough that would make it possible to write the celebrated `10 million lines of error-free code,'" which SDI officials have acknowledged are necesary to make the system, as currently envisioned, work. Moreover, "there are no laws or formulae that can accurately predict the successs or failure of a large software development." Nor is it possible today, the report says, to measure whether a software program can be applied to an SDI battle-management system. The report says these problems are not impossible to solve. However, it says it will take at least two decades - and then only if the organization of the program is radically changed. Assuming these fundamental uncertainties can be resolved, the report cites other computer and software difficulties. Among them: * Flights of the space shuttle have frequently been delayed because of computer problems found at the last moment. Yet whereas the shuttle's computers are designed to reain in operation for 1,000 hours without breaking down, the computers on board the satellites used in an SDI system would have to be built to break down only once every 100,000 hours. David Parnas, a software specialist at the University of Victoria in British Columbia, also says the experience of several shuttle flights has allowed NASA to work out programming "bugs" over time. "This kind of thing couldn't possibly work with SDI," he says. "You can't call the Russians in the middle of a war and say, `Wait a minute, we have to recalculate some things.'" Parnas was appointed a member of the panel that wrote the report. However, he resigned a few weeks after its formation, saying its work was pointless because SDI's software requirements were impossible to fulfill. Stephen Berlin of MIT's Laboratory for Computer Sciences [sic], notes another difference between SDI and the shuttle: "The space shuttle is not being shot at. An SDI system almost certainly would be." * The system would be highly vulnerable not only to direct attack but to nuclear weapons exploded in space as far as 1,000 kilometers away. "The high-energy neutron flux from a nuclear explosion is expected to `erase' volatile semiconductor memory," the report says. "Effective shielding is difficult." The report recommends new ways of dealing with strategic defense that organize the various components of an SDI system in a "loose hierarchy," with tasks "delegated to and localized within parts of a system." Such a system would involve less complex and more testable software, and could be adapted more easily to change. The authors of the report - all software specialists at top universities - acknowledge that it is not clear how to do all this, and that the SDI office should "use independent contractors" who could "tap the talent of leading researchers in the scientific community," to study the problem further. ------------------------------ Date: Sat, 18 Jan 86 16:37:24 PST From: jon@uw-june.arpa (Jon Jacky) To: RISKS@sri-csl.arpa, djw%a@lanl.arpa Subject: Man in the loop and magnetic bottles > (Dave Wade writes...) > I ... maintained the control system for a series of Magnetic Fusion Energy > experiments. The programs ran in "real time" and attempted to contain and > shape a plasma. ... The point of this was that any failure of the control > system was capable of killing personnel. ... The software was much smarter > than the operator and certainly swifter. There was no way that I could > hit the "kill" switch before the plasma got lose. With all respect, I am having trouble accepting that there is not some misunderstanding or overdramatization in this account. Are we to believe that people stood around where they could have gotten killed if the control system failed? If this is true, the risks in question were certainly unnecessary. Why on earth were the operators not protected behind some kind of blast shield? What assurances did they require to convince themselves that the software was sufficiently bug-free that they could trust their lives with it? ------------------------------ End of RISKS-FORUM Digest ************************ -------