1-Feb-86 01:33:46-PST,15776;000000000000 Mail-From: NEUMANN created at 1-Feb-86 01:32:16 Date: Sat 1 Feb 86 01:32:16-PST From: RISKS FORUM (Peter G. Neumann, Coordinator) Subject: RISKS-2.1 Sender: NEUMANN@SRI-CSL.ARPA To: RISKS-LIST@SRI-CSL.ARPA RISKS-LIST: RISKS-FORUM Digest, Saturday, 1 Feb 1986 Volume 2 : Issue 1 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: First Six Months of the Forum in Retrospect; *** Updated Disaster List *** (Peter G. Neumann) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@SRI-CSL.ARPA, Requests to RISKS-Request@SRI-CSL.ARPA.) (Back issues Vol 1 Issue j stored in SRI-CSL:RISKS-1.j, j = 1 to 45.) ---------------------------------------------------------------------- Date: Sat 1 Feb 86 00:18:03-PST From: Peter G. Neumann Subject: First Six Months of the Forum in Retrospect; Updated Disaster List To: RISKS@SRI-CSL.ARPA With RISKS-1.45, my MM saturated completely on its ability to cope with all of the Volume 1 issues of the RISKS Forum (at 289 pages) in one .TXT file, so it must be time to start Volume 2! Thus, let me take this opportunity to review the six months since RISKS Volume 1 Issue 1, on 1 August 1985. The SDI pallaver seems to have overwhelmed some of you, but it has opened up some serious problems -- as does the unreleased Eastport Report, which is perhaps most interesting for what it does not cover! (You may have concluded that SDI is a religious issue that cannot be decided rationally?) The Space Shuttle tragedy also opens up all sorts of questions on how much trust we can place in technology and what the risks are -- although those questions have been around all along. They are simply elevated to a higher level of public awareness now (temporarily?). For example, the issue of the safety of self-destruct mechanisms has been lurking, and is clearly a concern in general -- but is only one concern. There are lots of others. We must remember that risks come many sources -- not just from maliciousness and accidents, but also from unforeseen combinations of problems. So far we have addressed many important issues, although some of them quite superficially. We have overlapped on occasions with ARMS-D, Soft-Eng, and Security distributions. However, it seems that RISKS does provide a focus that cuts across these other mailings, and that it serves a useful purpose -- particularly when it indeed focuses on the risks to the public. A few platitudes are in order: No system is ever going to be 100% guaranteed all of the time, especially if it runs standalone. Even with humans in the loop, humans can make mistakes -- especially in real-time. Risks may come from unexpected directions. A system that has run perfectly may still suddenly malfunction. Hardware may fail. Bugs may remain undetected for years, and suddenly become activated. Besides, software may age poorly, especially if changes somewhere else interfere with old working software. The operating environment may contain risks that undermine sound design, impementation, and operations. The notion that all critical concerns -- security, reliability, etc. -- can be confined to a small portion of a computer system or distributed system (e.g., a kernel) is a fantasy, particularly with conventionally designed systems. The notion that distributed control solves problems that cannot easily be solved with central control is also often a myth -- problems of updating, synchronization, concurrency, backup, verifiability, etc., may be equally severe in many operating environments. Lowest-bidder efforts are intrinsically risky. Commercial software is commonly far behind the state of the art. That may be an advantage in some cases (!), but is often detrimental. But there are significant benefits to be gained from using certain software engineering techniques. There are inherent risks in using computer systems in critical environments. These must be continually reexamined. Critical and open discussion of critical systems and critical environments is essential. We as technologists must better understand the risks and their implications. And we must apply that increased understanding to new developments. And now, a word from our sponsor: This forum was established at the request of the Association for Computing Machinery, and chartered as an activity of the ACM Committee on Computers and Public Policy (of which I am the chairman). However, the opinions reflected herein do not constitute an endorsement by the ACM. On the other hand, if I screw up and do something the ACM does not like, I probably lose my (volunteer) job. As a summary of some of the problems that have occurred, and as a possible inspiration for undiscussed areas of concern or areas of hope, I include in this issue an update of the disaster list that those of you who have been with us since the beginning saw in Vol 1 No 1. Peter ******************************************** SOME COMPUTER-RELATED DISASTERS AND OTHER EGREGIOUS HORRORS Compiled by Peter G. Neumann (31 January 1986) The following list is drawn largely from back issues of ACM SIGSOFT Software Engineering Notes [SEN], references to which are cited as (SEN vol no), where vol 11 = 1986. Some incidents are well documented, others need further study. Please send corrections/additions+refs to PGNeumann, SRI International, BN168, Menlo Park CA 94025, phone 415-859-2375, Neumann@SRI-CSL.ARPA. Legend: ! = Loss of Life; * = Potentially Life-Critical; $ = Loss of Money/Equipment; S = Security/Privacy/Integrity Flaw -------------------------- SYSTEM + ENVIRONMENT ------------------------------ !S Arthritis-therapy microwaves set pacemaker to 214, killed patient (SEN 5 1) !S Retail-store anti-theft device reset pacemaker, man died. (SEN 10 2, 11 1) *S Auto speed changed by interference from CB transmitter (SEN 11 1) *S Failed heart-shocking devices due to faulty battery packs (SEN 10 3) *$ Three Mile Island PA, now recognized as very close to meltdown (SEN 4 2) *$ Crystal River FL reactor (Feb 1980) (Science 207 3/28/80 1445-48, SEN 10 3) !$ Deaths of 3 lobstermen in storm not predicted by National Weather Service -- 3 mos unrepaired weather buoy; $1.25M award (SEN 10 5) [NY Times 13 Aug 85] ** SAC/NORAD: 50 false alerts in 1979 (SEN 5 3), incl. a simulated attack whose outputs accidentally triggered a live scramble [9 Nov 1979] (SEN 5 3); ** BMEWS at Thule detected rising moon as incoming missiles [5 Oct 1960] (SEN 8 3). See E.C. Berkeley, The Computer Revolution, pp. 175-177, 1962. ** Returning space junk detected as missiles. Daniel Ford, The Button, p. 85 ** WWMCCS false alarms triggered scrams [3-6 Jun 1980] (SEN 5 3, Ford pp 78-84) ** DSP East satellite sensors overloaded by Siberian gas-field fire (Ford p 62) ** 747SP (China Air.) autopilot tried to hold at 41,000 ft after engine failed, other engines died in stall, plane lost 32,000 feet [19 Feb 85] (SEN 10 2) ** 767 (UA 310 to Denver) four minutes without engines [August 1983] (SEN 8 5) * F18 missile thrust while clamped, plane lost 20,000 feet (SEN 8 5) * Mercury astronauts forced into manual reentry (SEN 8 3) * Cosmic rays halve shuttle Challenger comm for 14 hours [8 Oct 84] (SEN 10 1) * Frigate George Philip fired missile in opposite direction (SEN 8 5) $ Hurricane Gloria in NY closes Midwest Stock Exchange (SEN 11 1) $S Debit card copying easy despite encryption (DC Metro, SF BART, etc.) $S Microwave phone calls easily interceptable; portable phones spoofable $S Sputnik frequencies triggered garage-door openers ------------------------------- SOFTWARE ------------------------------------ !$ 1983 Colorado River flood, faulty data/model? Too much water held back prior to spring thaws; 6 deaths, $ millions damage [NY Times 4 Jul 1983] *$ Mariner 1: Atlas booster launch failure DO 100 I=1.10 (not 1,10) (SEN 8 5) *$ Mariner 18: aborted due to missing NOT in program (SEN 5 2) *$ F18: plane crashed due to missing exception condition, pilot OK (SEN 6 2) *$ F14 off aircraft carrier into North Sea; due to software? (SEN 8 3) *$ F14 lost to uncontrollable spin, traced to tactical software (SEN 9 5) *$ El Dorado brake computer bug caused recall of all El Dorados (SEN 4 4) $$ Viking had a misaligned antenna due to a faulty code patch (SEN 9 5) $$ First Space Shuttle backup launch-computer synch problem (SEN 6 5 [Garman]) * Second Space Shuttle operational simulation: tight loop upon cancellation of an attempted abort; required manual override (SEN 7 1) * Second Shuttle simulation: bug found in jettisoning an SRB (SEN 8 3) *$ Delays of two Discovery shuttle launches due to backup computer outage [most recently 25 Aug 85] (SEN 10 5) [NY Times 26 August 1985] * Shuttle STS-6 bugs in live Dual Mission software prevented aborts (SEN 11 1) * Gemini V 100mi landing err, prog ignored orbital motion around sun (SEN 9 1) * F16 simulation: plane flipped over whenever it crossed equator (SEN 5 2) * F16 simulation: upside-down F16 deadlock over left vs. right roll (SEN 9 5) * Nuclear reactor design: bug in Shock II model/program (SEN 4 2) * Reactor overheating, low-oil indicator; two-fault coincidence (SEN 8 5) * SF BART train doors sometimes open on long legs between stations (SEN 8 5) $ IRS reprogramming delays; interest paid on over 1,150,000 refunds (SEN 10 3) $ $32 BILLION overdraft at Bank of New York (prog counter overflow) (SEN 11 1) *S Numerous system intrusions and penetrations; implanted Trojan horses; 414s; intrusions to TRW Credit Information Service, British Telecom's Prestel, Santa Clara prison data system (inmate altered release date) (SEN 10 1). Computerized time-bomb inserted by programmer (for extortion?) (10 3) PC Graphics program Trojan horse (ArfArf) wiped out users' files (SEN 10 5) *$ Union Carbide leak (135 injuries) exacerbated by program not handling aldicarb oxime plus operator error [NY Times 14 and 24 Aug 85] (SEN 10 5) * Multipatient monitoring system recalled; mixed up patients (SEN 11 1) * Pacemaker locked up when being adjusted by doctor (SEN 11 1) * Diagnostic lab instrument misprogrammed (SEN 11 1) S Chernenko at MOSKVAX: network mail hoax [1 April 1984] (SEN 9 4) S VMS tape backup SW trashed disc directories dumped in image mode (SEN 8 5) *$ C&P computer crashes 44,000 DC phones (SEN 1 1) $ 1979 AT&T program bug downed phone service to Greece for months (SEN 10 3) $ Demo NatComm thank-you mailing mistitled supporters [NY Times, 16 Dec 1984] $ Slow responses in Bankwire interface SW resulted in double posting of tens of $millions, with interest losses (SEN 10 5) $ Program bug permitted auto-teller overdrafts in Washington State (SEN 10 3) - Quebec election prediction gave loser big win [1981] (SEN 10 2, p. 25-26) - Other election problems including mid-stream corrections (HW/SW) (SEN 10 3) - SW vendor rigs elections? (David Burnham, NY Times front page, 29 July 1985) - Alaskan DMV program bug jails driver [Computerworld 15 Apr 85] (SEN 10 3) - Vancouver Stock Index lost 574 points over 22 months -- roundoff (SEN 9 1) - Gobbling of legitimate automatic teller cards (SEN 9 2, another SEN 10 5) -------------------------- HARDWARE/SOFTWARE --------------------------------- ! Michigan man killed by robotic die-casting machinery (SEN 10 2, 11 1) ! Japanese mechanic killed by malfunctioning Kawasaki robot (SEN 10 1, 10 3) [Electronic Engineering Times, 21 December 1981] ! Chinese computer builder electrocuted by his smart computer. (WWN headline: "Jealous Computer Zaps its Creator" after he built newer one!!) (SEN 10 1) * FAA Air Traffic Control: many computer system outages (e.g., SEN 5 3) * ARPANET ground to a complete halt [27 Oct 1980] (SEN 6 1 [Rosen]) *$ Ford Mark VII wiring fires: flaw in computerized air suspension (SEN 10 3) $S Harrah's $1.7 Million payoff scam -- Trojan horse chip (SEN 8 5) $ Great Northeast power blackout due to threshold set-too-low being exceeded $ Power blackout of 10 Western states, propagated error [2 Oct 1984] (SEN 9 5) $ NY Stock Exch. halted for 41 minutes; drum channel errors killed primary and backup computer systems [24 Feb 72] - SF Muni Metro: Ghost Train reappeared, forcing manual operation (SEN 8 3) *$ Computer-controlled turntable for huge set ground "Grind" to halt (SEN 10 2) *$ 8080 control system dropped bits and boulders from 80 ft conveyor (SEN 10 2) S 1984 Rose Bowl hoax, scoreboard takeover ("Cal Tech vs. MIT") (SEN 9 2) -------- COMPUTER AS CATALYST, HUMAN FRAILTIES, OR UNKNOWN CAUSES ------------- !!$ Korean Airlines 007 shot down [1 Sept 1983], killing 269; autopilot left on HDG 246 rather than INERTIAL NAV? (NYReview 25 Apr 85, SEN 9 1, SEN 10 3) !!$ Air New Zealand crashed into mountain [28 Nov 1979]; computer course data error had been detected and fixed, but pilots not informed (SEN 6 3 & 6 5) ! Woman killed daughter, tried to kill son and self; "computer error" blamed for false report of their all having an incurable disease (SEN 10 3) * Unarmed Soviet missile crashed in Finland. Wrong flight path? (SEN 10 2) *$ South Pacific Airlines, 200 aboard, 500 mi off course near USSR [6 Oct 1984] *S San Francisco Public Defender's database accessible to police (SEN 10 2) * Various cases of false arrest due to computer database use (SEN 10 3, 11 1) * Avionics failed, design used digitized copier-distorted curves (SEN 10 5) $ .5M transaction became $500M, due to "000" convention; $200M lost (SEN 10 3) $ Possible fraud on reinsurance -- message time stamp faked??? (SEN 10 5) $ N-step reinsurance cycle; SW checked only N=1 and 2 (SEN 10 5) * FAA Air Traffic Control: many near-misses not reported (SEN 10 3) !$$ Shuttle Challenger explosion, 7 killed. Cause not yet known. [29 Jan 86] ---------------- ILLUSTRATIVE OF POTENTIAL FUTURE PROBLEMS ------------------- *S Many known/past security flaws in computer operating systems and application programs. Discovery of new flaws running way ahead of their elimination. * Expert systems in critical environments: unpredictability if (unknowingly) outside of range of competence, e.g., incompleteness of rule base. StarWars $S Embezzlements, e.g., Muhammed Ali swindle [$23.2 Million], Security Pacific [$10.2 Million], City National Beverly Hills CA [$1.1 Million, 23 Mar 1979] [These were only marginally computer-related, but suggestive. Others are known, but not publically acknowledged.] ---------------------- REFUTATION OF EARLIER REPORT -------------------------- * "Exocet missile not on expected-missile list, detected as friend" (SEN 8 3) [see Sheffield sinking, reported in New Scientist 97, p. 353, 2/10/83]; Officially denied by British Minister of Defence Peter Blaker [New Scientist, vol 97, page 502, 24 Feb 83]. Rather, sinking abetted by defensive equipment being turned off to reduce communication interference? [See also anecdotes from ACM Symposium on Operating Systems Principles, SOSP 7 (SEN 5 1) and follow-on (SEN 7 1).] ------------------------------ End of RISKS-FORUM Digest ************************ -------