26-Jun-86 00:51:56-PDT,15200;000000000000 Mail-From: NEUMANN created at 26-Jun-86 00:50:15 Date: Thu 26 Jun 86 00:50:15-PDT From: RISKS FORUM (Peter G. Neumann, Coordinator) Subject: RISKS-3.13 Sender: NEUMANN@SRI-CSL.ARPA To: RISKS-LIST@SRI-CSL.ARPA RISKS-LIST: RISKS-FORUM Digest, Thursday, 26 June 1986 Volume 3 : Issue 13 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: The Risky Gap Between Two Design Cultures (Jack Goldberg) Risks of nuclear power (Dan Franklin) Research programs that pay for themselves (Rich Cowan) Having an influence from "within the system" (Rich Cowan) RISKS in running RISKS -- continued (PGN and an unhappy Mailer) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@SRI-CSL.ARPA, Requests to RISKS-Request@SRI-CSL.ARPA.) (Back issues Vol i Issue j available in SRI-CSL:RISKS-i.j. Summary Contents in MAXj for each i; Vol 1: RISKS-1.46; Vol 2: RISKS-2.57.) ---------------------------------------------------------------------- Date: Wed 25 Jun 86 12:01:12-PDT From: Jack Goldberg Subject: The Risky Gap Between Two Design Cultures To: RISKS@SRI-CSL.ARPA Over the centuries of experience in dealing with hazards, mechanical and civil engineers developed a culture of safe design, with principles and practices appropriate to the various kinds of products. This culture was expressed in the design of mechanisms that implemented various safety functions, such as barriers to undesired motion, redundancy in the event of local failures, self-adjustment to losses of tolerance, and so on. For each kind of product, particular mechanisms were developed to accomplish these functions, e.g., pawls, detents, rails, ratchets, fuses. The advent of computers and inexpensive sensors and motors made possible tremendous economies in manufacture by eliminating all those particular mechanisms and their often costly assembly (consider the dramatic comparison in complexity of mechanism between the original teletype machine and a modern typewriter/printer). Mechanical design of the new systems has been dramatically simplified, and the complex functions, including safety functions, have been relegated to a control program. In a sense, the design is created on a blank slate. Who creates that design? Generally someone who is a professional programmer, often a novice, who has inherited the culture of that profession. There are many aspects to that culture, but it rarely includes the lore and practices of safe design (and the exposure to the machinery of legal liability) that is the inheritance of mechanical and civil engineers. It is often based on a partitioning of responsibility between the hypothetical (and often anonymous) "customer" and the programmer-supplier, a partitioning that hides the ultimate users from the designer. Also, too often, the programmer's education in matters of the physical world has been compromised by the demands of training for his profession. Often, the practitioners in the new culture see themselves as generalists, able to solve any new problem, and they move frequently from one application area to another. Consequently, they seldom have the time to study and understand the things that users or designers in a particular field know or assume to be obvious, and so they must imagine and re-invent them. Tragically, those imperfectly mastered things sometime seriously affect safety. In short, the culture of safety that traditional engineers have expressed in particular mechanisms has been tossed out along with those mechanisms, and is being re-discovered, painfully, by a new generation of designers that has no connection with the traditional culture. In this light, risks arising in contemporary computer-based system design may be seen as a consequence of a gap between two design cultures. The gap is both generational and professional; there are many safety engineers in industry, but they and programmers speak different languages. In a different context, awareness of the loss of knowledge by experts in various practices, due to their lack of replacement in the work force, has stimulated some computerists to try to capture that knowledge. How well they are doing that is another matter, but it may be that some conscious gap-bridging between the cultures would save the world some amount of misfortune and misery. Jack Goldberg, SRI International ------------------------------ Date: Tue, 24 Jun 86 14:03:42 EDT From: Dan Franklin To: risks@sri-csl.ARPA Subject: Risks of nuclear power TMPLee@DOCKMASTER.ARPA discusses nuclear energy vs. solar energy and "taking all the risks into account". The risk he is primarily concerned with is the risk of falling off a solar energy device while cleaning off the snow. If we are going to take all the risks into account, let's face it: the risks to those involved in the actual energy production are simply insignificant in the debate on nuclear vs. other forms of energy. That debate focuses almost entirely on the risks to innocent bystanders. These are the risks that always matter most, precisely because people do not willfully undertake them, but rather end up subjected to them, and people are not willing to be *subjected* to nearly as much risk as they are willing to *decide* to take, or let others decide to take. The fundamental political problem of nuclear power is that it has a small probability of being disastrously more injurious to bystanders than any other form of power generation except dams. (Solar power satellites which deliver their power by microwave are a future contender.) TMPLee's mention of low-level radiation emitted by coal-fired plants is, of course, directly relevant to this issue. But in the wake of Chernobyl, as in the wake of TMI (and in the wake of Pilgrim's safety problems...), the small probability of disaster clearly needs to be discussed. Dan Franklin ------------------------------ Date: Thu 26 Jun 86 00:08:21-EDT From: Richard A. Cowan Subject: Research programs that pay for themselves To: risks@SRI-CSL.ARPA Let me add a few comments to Bob Estell's point #6: > "Going to the moon in the '60's cost the USA nothing!... > The DIFFERENCE between tax dollars paid by those wearing pacemakers, and > the "aid to their families" that would have been paid had those heart > patients died or been disabled, is more than $25 billion." There are two problems with these types of conclusions. First of all, there are plenty of big non-space or non-military government programs that we could spend our money on that are equally likely to have spinoffs; there must be a reason why SDI should be built rather than these projects. But the classification barriers of SDI will inevitably reduce spinoffs. Not only that, but some things in SDI will certainly be useless commercially. Pacemakers don't need to survive nuclear explosions. Secondly, any government program has an opportunity cost which is not factored into your calculation: when we devote scientific resources to the private sector, we lose out on the benefits we would have gained if those resources weren't used up by the government. An example is mentioned in a May issue of the weekly trade paper "Electronics News": a Japanese witness at some hearings on US competitiveness points out that the United States spends hundreds of millions on high-strength, lightweight carbon materials for aircraft wings, while the Japanese developed the same materials very cheaply for golf clubs and tennis racquets. Are there things which we could use more than we could use SDI? Are there other government expenditures (perhaps national health insurance) that would REALLY cost nothing? Well, I recently heard that aside from public police forces and the military, about $300 billion per year is spent on security (including locks, alarms, etc.) To get an idea where this comes from, consider that MIT's police force costs a couple million, and all universities put together must spend about $1 billion. Now if businesses instead spent $100 billion of this money on raising the minimum wage $2, spent $50 billion on reducing unemployment by reducing the work week, and $20 billion went to the government to improve housing programs and public facilities to keep young people occupied, then perhaps the need for so much security would be reduced, because the root causes of crime would be diminished. It would therefore "cost nothing" for the private sector to divert $170 billion of its security bill and improve the social stability and welfare of the country. The problem with such a plan is that the benefits come only in the long term; only the greater short term costs are seen on corporate balance sheets. -rich ------------------------------ Date: Thu 26 Jun 86 00:11:07-EDT From: Richard A. Cowan Subject: Having an influence from "within the system" To: risks@SRI-CSL.ARPA And now a few comments on Bob Estell's point 10 on working for SDI: > "But if we do take the opportunity, then we can use the > managers' short term interests to an advantage; i.e., we can honestly say > that "Star Wars" [R2D2 et al] is not possible today; and then diligently > work to produce what is reasonable." You have here touched upon what I believe is -- more often than not -- a delusion: that it is more effective to work within the system to change it than to protest it from without. In this case, working within the system means working on Star Wars to demonstrate part of it to be feasible or infeasible. There are several problems with this. First, within a large institution you may be isolated from resources, or a diversity of viewpoints needed to make an impartial decision. This is less true with Star Wars than with other programs because there's lots of mainstream publicity. It is also less true in a university than in a defense contractor. Second, and more importantly, what an engineer says is likely to get manipulated for political reasons -- like the ignored warnings before the space shuttle disaster. If of 10,000 engineers working on SDI, 5000 include negative critical material in their research reports, and the other 5000 are completely uncritical of SDI, what do you think Congress will hear? Well, I can guarantee that they will hear mostly glowing reports about research progress from upper-level managers and lobbying organizations of the companies doing SDI research. If your strategy to change things is to become one of those upper-level managers, you may have to compromise your values to achieve promotion, and temper your criticisms to avoid losing "credibility" once you get there. Yet Congress is hearing the other side on SDI. How? Because engineers are not relying on the companies they work for to communicate their insight. They are going outside the normal channels of communication -- like the 1600 scientists working at government labs who recently petitioned Congress to curtail SDI spending. And ultimately, communicating one's concerns directly to people in the community is necessary. What is unfortunate, and I believe dangerous in a democracy, is that people working for the government are afraid of speaking out on public policy issues for fear of reprisal. The recent statements by Undersecretary of Defense for Research and Engineering Donald Hicks may have heightened this fear. Fortunately, the Pentagon has recently dissociated itself from Hicks' statements. (Science, May 23) -rich ------------------------------ From: MAILER-DAEMON@nprdc.arpa (Mail Delivery Subsystem) Subject: Returned mail: Service unavailable [Actual Subject: RISKS in running RISKS -- continued] [One of the greatest annoyances in running a large mailing enterprise such as RISKS is fielding the incessant net-barfs, including having my mailbox cluttered with multiple copies of the Forum on net addresses that don't work now and then. (Some mailers that keep retrying periodically, and send back advisories each time.) Here is a fine example -- which of course more generally represents another type of risk in distributed systems. PGN] [FOOTNOTE: The more general problem of copious rejected mail would be an order of magnitude worse if we went to individual messages rather than the current digest format. (I now have requests from BITNET and USENET to do send out undigestified messages, and would love to let them do the undigestifying. Perhaps more regional reforwarding centers would minimize rejects and reduce mailing list maintenance substantially. But, I nevertheless get mystery rejection notices for people not even on my list, because of redistribution problems elsewhere.)] ----- Transcript of session follows ----- >>> DATA <<< 554 ... Mail loop detected >>> QUIT <<< 554 sendall: too many hops (30 max) [... but quite a brew-haha!] 554 ... Service unavailable: Bad file number ----- Unsent message follows ----- Received: from pacific.ARPA by nprdc.arpa (4.12/ 1.1) id AA00971; Tue, 24 Jun 86 03:04:28 pdt Received: from hull.aegean.arpa (hull.ARPA) by pacific.ARPA (4.12/4.7) id AA11313; Tue, 24 Jun 86 03:04:01 pdt Received: from nprdc.arpa (aegean) by hull.aegean.arpa (2.2/SMI-2.0) id AA14974; Tue, 24 Jun 86 02:50:15 pdt Received: from pacific.ARPA by nprdc.arpa (4.12/ 1.1) id AA00967; Tue, 24 Jun 86 03:04:07 pdt Received: from hull.aegean.arpa (hull.ARPA) by pacific.ARPA (4.12/4.7) id AA11309; Tue, 24 Jun 86 03:03:40 pdt [... many hops omitted ... I think you get the idea. Notice the clock drift while you're at it.] Received: from hull.aegean.arpa (hull.ARPA) by pacific.ARPA (4.12/4.7) id AA11281; Tue, 24 Jun 86 03:01:05 pdt Return-Path: Received: from nprdc.arpa (aegean) by hull.aegean.arpa (2.2/SMI-2.0) id AA14942; Tue, 24 Jun 86 02:47:19 pdt Received: from pacific.ARPA by nprdc.arpa (4.12/ 1.1) id AA00935; Tue, 24 Jun 86 03:01:10 pdt Received: from nprdc.arpa (aegean.ARPA) by pacific.ARPA (4.12/4.7) id AA11271; Tue, 24 Jun 86 03:00:39 pdt Received: from SRI-CSL.ARPA (sri-csl.arpa.ARPA) by nprdc.arpa (4.12/ 1.1) id AA00926; Tue, 24 Jun 86 03:00:30 pdt Date: Tue 24 Jun 86 01:41:53-PDT From: RISKS FORUM (Peter G. Neumann, Coordinator) Subject: RISKS-3.12 [...] [But, gee Mr. Wizard, it worked just fine on the previous issues!] ------------------------------ End of RISKS-FORUM Digest ************************ -------