2-Nov-86 21:16:40-PST,23067;000000000000 Mail-From: NEUMANN created at 2-Nov-86 21:14:21 Date: Sun 2 Nov 86 21:14:21-PST From: RISKS FORUM (Peter G. Neumann -- Coordinator) Subject: RISKS DIGEST 4.1 Sender: NEUMANN@CSL.SRI.COM To: RISKS-LIST@CSL.SRI.COM RISKS-LIST: RISKS-FORUM Digest, Sunday, 2 November 1986 Volume 4 : Issue 1 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Latest version of the computer-related trouble list (Peter G. Neumann) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM) (Back issues Vol i Issue j available in CSL.SRI.COM:RISKS-i.j. MAXj: Summary Contents Vol 1: RISKS-1.46; Vol 2: RISKS-2.57; Vol 3: RISKS-3.92.) ---------------------------------------------------------------------- ILLUSTRATIVE RISKS TO THE PUBLIC IN THE USE OF COMPUTER SYSTEMS AND RELATED TECHNOLOGY Compiled by Peter G. Neumann (1 November 1986), Chmn ACM Committee on Computers and Public Policy A compendium of all of the following cases is in preparation, anthologizing back issues of ACM SIGSOFT Software Engineering Notes [SEN], references to which are cited below as (SEN vol no); e.g., (SEN 11 5) is October 1986, one vol per year, quarterly (plus an occasional special issue). Some incidents are well documented, others need further study. Please send corrections, additions, and refs to PGNeumann, SRI International, BN168, Menlo Park CA 94025, phone 415-859-2375, ARPANET Neumann@CSL.SRI.COM or RISKS@CSL.SRI.COM. Legend: ! = Loss of Life; * = Potentially Life-Critical; $ = Loss of Resources S = Security/Privacy/Integrity Problem; ["\" = multiply listed item] H = Human directly implicated (e.g., user/administrator/operator/penetrator) [NOTE: Design, implementation flaws are human problems, but not marked "H".] SPACE: !!$$ Shuttle Challenger explosion, 7 killed. [Booster sensors (removed) might have permitted early detection of booster leak?] [28 Jan 86] (SEN 11 2) $ First Space Shuttle backup launch-computer synch problem (SEN 6 5: Jack Garman, "The bug heard 'round the world", October 1981, pp. 3-10.) * Second Shuttle simulation: bug found in jettisoning an SRB (SEN 8 3) * Second Space Shuttle operational simulation: tight loop upon cancellation of an attempted abort; required manual override (SEN 7 1) $ Titan 34D, Nike Orion, Delta-178 failures follow Challenger (SEN 11 3) * Columbia return delayed; computer malfunctioned despite redundancy (SEN 9 1) * Columbia near-disaster, liquid oxygen drained mistakenly just before launch, computer output misread (SEN 11 5) *$ Delays of two Discovery shuttle launches due to backup computer outage [second one on 25 Aug 85] [NY Times 26 August 1985] (SEN 10 5) * Discovery laser aims upside down: +10,023 miles instead of feet (SEN 10 3) * Shuttle Discovery landing gear -- correlated faults (SEN 10 3) * Shuttle STS-6 bugs in live Dual Mission software precluded aborts (SEN 11 1) * Mercury astronauts forced into manual reentry? (SEN 8 3) *$ Mariner 1: Atlas booster launch failure DO 3 I=1.3 (not 1,3)? (SEN 8 5,11 5) *$ Mariner 18: aborted due to missing NOT in program (SEN 5 2) * Gemini V 100mi landing err, prog ignored orbital motion around sun (SEN 9 1) $ Atlas-Agena software missing hyphen; $18.5M rocket destroyed (SEN 10 5) $ Aries with $1.5M payload lost: wrong resistor in guidance system; (SEN 11 5) * TDRS relay satellite locked on wrong target (SEN 10 3) * Cosmic rays halve shuttle Challenger comm for 14 hours [8 Oct 84] (SEN 10 1) $ Viking had a misaligned antenna due to a faulty code patch (SEN 9 5) * Ozone hole over South Pole observed, but denied by SW for 8 years (SEN 11 5) MISSILE, AIR, AND NAVAL DEFENSE: !!$ Sheffield sunk during Falklands war, 20 killed. Call to London jammed antimissile defenses. Exocet on same frequency. [AP 16 May 86](SEN 11 3) ** Returning space junk detected as missiles. Daniel Ford, The Button, p. 85 ** WWMCCS false alarms triggered scrams [3-6 Jun 1980] (SEN 5 3, Ford pp 78-84) ** DSP East satellite sensors overloaded by Siberian gas-field fire (Ford p 62) (Ford summarized in SEN 10 3) ** BMEWS at Thule detected rising moon as incoming missiles [5 Oct 1960] (SEN 8 3). See E.C. Berkeley, The Computer Revolution, pp. 175-177, 1962. ** SAC/NORAD: 50 false alerts in 1979 (SEN 5 3), incl. a simulated attack whose outputs accidentally triggered a live scramble [9 Nov 1979] (SEN 5 3); *$ Libyan bomb raid accidental damage by "smart bomb" (SEN 11 3) * Frigate George Philip fired missile in opposite direction (SEN 8 5) * Unarmed Soviet missile crashed in Finland. Wrong flight path? (SEN 10 2) * Tomahawk cruise missile failure: program erased [8 Dec 86] (SEN 11 2) * 2nd Tomahawk failure (SEN 11 5). Bit dropped by HW triggered ABORT sequence. * Sgt York (DIVAD) radar/anti-aircraft gun -- software problems (SEN 11 5) $ Software flaw in sub-launched ballistic missile system (SEN 10 5) $ AEGIS failures on 6 of 17 targets attributed to software (SEN 11 5) - WWMCCS computers' comm reboot failed by blocked multiple logins (SEN 11 5) $ Armored Combat Earthmover 18,000 testing missed serious problems (SEN 11 5) $ Stinger missile too heavy to carry, noxious to user (SEN 11 5) $ "Spy ship" Liberty: 3 independent warnings to withdraw all lost (SEN 11 5) ** Strategic Defense Initiative -- debate over feasibility (SEN 10 5) MILITARY AVIATION: !!$ Handley Page Victor tailplane broke, crew lost. 3 INDEPENDENT test methods. 3 independent flaws, masking flutter problem (SEN 11 2,p.12;correct'n 11 3) *$ F-18 crash due to missing exception cond. Pilot OK (SEN 6 2, more SEN 11 2) * F-18 missile thrust while clamped, plane lost 20,000 feet (SEN 8 5) * F-16 simulation: plane flipped over whenever it crossed equator (SEN 5 2) * F-16 simulation: upside-down, deadlock over left vs. right roll (SEN 9 5) $H F-16 landing gear raised while plane on runway; bomb problems (SEN 11 5) *$ F-14 off aircraft carrier into North Sea; due to software? (SEN 8 3) *$ F-14 lost to uncontrollable spin, traced to tactical software (SEN 9 5) $S Pres.Reagan's command plane jams thousands of garage-door openers (SEN 11 2) COMMERCIAL AVIATION: !!$H Korean Airlines 007 shot down killing 269 [1 Sept 1983]; autopilot left on HDG 246 rather than INERTIAL NAV? (NYReview 25 Apr 85, SEN 9 1, SEN 10 3) !!$H Air New Zealand crashed into Mt Erebus, killing 257 [28 Nov 1979]; computer course data error detected but pilots not informed (SEN 6 3 & 6 5) !!H Aeromexico flight to LAX crashes with private plane, 82 killed (SEN 11 5) !!$ DC-10 indicators failed: their power came from missing engine (SEN 11 5) !!$ Electra failures due to simulation omission (SEN 11 5) !$ Computer readout for navigation wrong, pilot killed (SEN 11 2) *H South Pacific Airlines, 200 aboard, 500 mi off course near USSR [6 Oct 1984] *H 747SP (China Air) autopilot tried to maintain 41,000 ft after engine failed, other engines died in stall, plane lost 32,000 feet [19 Feb 85] (SEN 10 2) * Avionics failed, design used digitized copier-distorted curves (SEN 10 5) ** 767 (UA 310 to Denver) four minutes without engines [August 1983] (SEN 8 5) * 767 failure LA to NY forced to alternate SF instead of back to LA (SEN 9 2) * Air Traffic Control data cable loss caused close calls (SEN 10 5) * FAA Air Traffic Control: many computer system outages (e.g., SEN 5 3, 11 5), near-misses not reported (SEN 10 3) RAIL TRAVEL: !!$H Canadian trains collide despite "safe" computer; 26 killed (SEN 11 2) * SF BART train doors opened between stations during SF-Oakland leg (SEN 8 5) - SF BART automatic control disastrous days of computer outages (SEN 6 1) - SF Muni Metro: Ghost Train reappeared, forcing manual operation (SEN 8 3) AUTOMOBILES: !$ Mercedes 500SE with graceful-stop no-skid brake computer left 368-foot skid marks; passenger killed (SEN 10 3) *S Sudden auto acceleration due to interference from CB transmitter (SEN 11 1); *$ Microprocessors in 1.4M Fords, 100K Audis, 350K Nissans, 400K Alliances/ Encores, 140K Cressidas under investigation (SEN 10 3) *$ El Dorado brake computer bug caused recall of that model [1979] (SEN 4 4) *$ Ford Mark VII wiring fires: flaw in computerized air suspension (SEN 10 3) MOTOR VEHICLE DATABASE PROBLEMS: !!H Bus crash kills 21, injures 19; computer database showed driver's license had been revoked, but not checked? Also, unreported citation (SEN 11 3) *SH British auto citations removed from database for illicit fee (SEN 11 1) $ California DMV computer bug hid $400 million fees for six months (SEN 11 2) $ Toronto motor vehicle computer reported $36 million extra revenue (SEN 11 3) - Alaskan DMV program bug jails driver [Computerworld 15 Apr 85] (SEN 10 3) ELECTRICAL POWER (NUCLEAR AND OTHER): !!$H Chernobyl nuclear plant fire/explosion/radiation [26 April 86] (SEN 11 3) Misplanned experiment on emergency-shutdown recovery procedures backfired. Fatal (at least 31), serious cases continue to mount. Wide-spread effects. *$ 14 failures in Davis-Besse nuclear plant emergency shutdown (SEN 11 3) *$ Three Mile Island PA, now recognized as very close to meltdown (SEN 4 2), with 4 equipment failures plus misjudgement. SW flaw noted (SEN 11 3) !!,$ Various previous nuclear accidents -- American (3 deaths SL-1 Idaho Falls) Soviet (27-30 deaths on Icebreaker Lenin, three other accidents) (SEN 11 3) * Subsequent to Chernobyl, US Nuclear Regulatory Commission relaxed fire isolation guidelines, enabling a fire to wipe out two systems (SEN 11 3) *$ Crystal River FL reactor (Feb 1980) (Science 207 3/28/80 1445-48, SEN 10 3) *$ Great Northeast power blackout due to threshold set-too-low being exceeded *$ Power blackout of 10 Western states, propagated error [2 Oct 1984](SEN 9 5) * Ottawa power utility loses working three units to faulty monitor (SEN 11 5) * Reactor overheating, low-oil indicator; two-fault coincidence (SEN 8 5) * Bug discovered in Shock II model/program for designing nuclear reactors to withstand earthquakes shuts down five nuclear power plants (SEN 4 2) MEDICAL HEALTH AND SAFETY RISKS: !,* Misprogrammed cancer radiation machines; 1 killed, 2 injured (SEN 11 3) ! Woman killed daughter, tried to kill son and self; "computer error" blamed for false report of their all having an incurable disease (SEN 10 3) ! Arthritis-therapy microwaves set pacemaker to 214, killed patient (SEN 5 1) ! Retail-store anti-theft device reset pacemaker, man died (SEN 10 2, 11 1) * Pacemaker locked up when being adjusted by doctor (SEN 11 1) * Failed heart-shocking devices due to faulty battery packs (SEN 10 3) * Multipatient monitoring system recalled; mixed up patients (SEN 11 1) * Diagnostic lab instrument misprogrammed (SEN 11 1) * AI medical system in Nevada gave wrong diagnosis, overdose (SEN 11 2) * Video display terminal health safety a continuing concern (SEN 11 3, 11 5) * Dangers of computerized robot used in surgery (SEN 10 5) ROBOTS AND ARTIFICIAL INTELLIGENCE: ! Japanese mechanic killed by malfunctioning Kawasaki robot (SEN 10 1, 10 3) [Electronic Engineering Times, 21 December 1981] ! At least 4 more, possibly 19 more robot-related deaths in Japan (SEN 11 1) ! Michigan man killed by robotic die-casting machinery (SEN 10 2, 11 1) ! Chinese computer builder electrocuted by his smart computer. (WWN headline: "Jealous Computer Zaps its Creator" after he built newer one...) (SEN 10 1) * Two cases of robot near-disasters narrowly averted by operators (SEN 11 3) - Servant robot runs amok, winds up in court (SEN 11 5) OTHER CONTROL-SYSTEM PROBLEMS: !!$,H? 1983 Colorado River flood, faulty data/model? Too much water held back prior to spring thaws; 6 deaths, $ millions damage [NY Times 4 Jul 1983] *$ Union Carbide leak (135 injuries) exacerbated by program not handling aldicarb oxime plus operator error [NY Times 14 and 24 Aug 85] (SEN 10 5) *$ Computer-controlled turntable for huge set ground "Grind" to halt (SEN 10 2) *$ 8080 control system dropped bits and boulders from 80 ft conveyor (SEN 10 2) - Titanic photo expedition control program erratic (SEN 11 5) OTHER COMPUTER-AIDED DESIGN PROBLEMS: * Hartford Civic Center Roof collapse due to use of wrong model (SEN 11 5) * Salt Lake City shopping mall roof collapses on first snowfall (SEN 11 5) * John Hancock Building in Boston glass panels kept falling out (???) FINANCIAL LOSSES: $ $32 BILLION overdraft at Bank of New York (prog counter overflow) (SEN 11 1) $H $2 Billion goof due to test tape being rerun live (SEN 11 2) $H .5M transaction became $500M, due to "000" convention; $200M lost (SEN 10 3) $ Slow responses in Bankwire interface SW resulted in double posting of tens of $millions, with interest losses (SEN 10 5) $ California state computer wrote $4M checks accidentally (SEN 11 5) $H ATM accepts lollipop cardboard as $1M (New Zealand) deposit (SEN 11 5) $H ATM money dispensers blocked and emptied later by youths (SEN 11 5) $H Barclays Bank hacked for 440,000 pounds? (SEN 11 5) $H ATMs gave $140,000 on VISA card over a weekend -- software glitch (SEN 11 2) $ Program bug permitted auto-teller overdrafts in Washington State (SEN 10 3) $ IRS reprogramming delays; interest paid on over 1,150,000 refunds (SEN 10 3) $H San Jose library lost two weeks of records. Books, fines lost. (SEN 11 3) $H Video quiz game scam -- teams of "experts" with right answers (SEN 11 5) STOCK-MARKET PHENOMENA: $ Computer-induced big stock-market swings (SEN 11 2, 11 5) $ Vancouver Stock Index lost 574 points over 22 months -- roundoff (SEN 9 1) $ NY Stock Exch. halted for 41 minutes; drum channel errors killed primary and backup computer systems [24 Feb 72] $ London Stock Exchange computer system crashes [23 May 86] $ Hurricane Gloria in NY closes Midwest Stock Exchange (SEN 11 1) TELEPHONE PROBLEMS: $ Pac Bell loses $51 million on lost phone-call charges (SEN 11 3) $ 400 pay phones in Hackensack lost charges for half of the calls (SEN 11 3) $ GTE Sprint incomplete SW changes lost $10-$20M in Feb/Mar/Apr 86 (SEN 11 3) $ GTE Sprint billing errors from botched daylight savings cutover (SEN 11 5) *$ Michigan Bell ESS office, 2 long outages. SW updates in progress. (SEN 11 3) *$ 707 area code (above San Fran.) shut down completely for 5 hours (SEN 11 5) $* Atlanta telephone system down for 2 hours (SEN 11 5) *$ C&P computer crashes 44,000 DC phones (SEN 11 1) $ C&P computer "tape flaws" delay 100,000 bills by two months (SEN 11 5) $ 1979 AT&T program bug downed phone service to Greece for months (SEN 10 3) $ Ghost phone calls to 911 from cordless phone interference (SEN 11 2) $H Swedish phone bill of $2600 -- program error plus human error (SEN 11 5) ELECTION PROBLEMS: SH Election frauds, lawsuits (SEN 11 3, 11 5), mid-stream patches in HW/SW (SEN 10 3, 10 4), David Burnham, NY Times, 7/29, 7/30, 8/4, 8/21, 12/18 1985. - Clerical error blamed for election computer program mishap (SEN 11 5) - Quebec election prediction bug: wrong pick [1981] (SEN 10 2 pp 25-26, 11 2) INSURANCE FRAUDS: $SH Possible fraud on reinsurance -- message time stamp faked??? (SEN 10 5) $H N-step reinsurance cycle; software checked for N=1 and 2 only (SEN 10 5) COMPUTER SECURITY/PRIVACY/INTEGRITY VIOLATIONS: PENETRATIONS, BLACKMAIL, TROJAN HORSES AND VIRUSES, TIME-BOMBS, PRANKS, SPOOFS, SCAMS, AND OTHER PROBLEMS ..... General comments: *SH Many known security flaws in computer operating systems and application programs. Discovery of new flaws running way ahead of their elimination. Flaws include problems with passwords, superuser facilities, networking, reprogrammable workstations, inadequate or spoofable audit trails, ease of perpetrating viruses and Trojan horses, improper handling of line breaks, etc. Examples of UNIX flaws as illustrative. Lots of internal fraud, but external penetrations frequent. ..... Penetrations by nonauthorized personnel: SH "Captain Midnight" preempted Home Box Office program (SEN 11 3, 11 5) SH Chernenko at MOSKVAX: network mail hoax [1 April 1984] (SEN 9 4) SH 1984 Rose Bowl hoax, scoreboard takeover ("Cal Tech vs. MIT") (SEN 9 2) $SH TRW Credit information bureau breakins -- one involved gaining information on Richard Sandza (Newsweek reporter who wrote ``anti-hacker'' articles) and running up $1100 in charges. (SEN 10 1) SH British Telecom's Prestel Information Service -- demonstration for a reporter read Prince Philip's demo mailbox and altered a financial market database [London Daily Mail 2 Nov 84] (SEN 10 1) Break-in being prosecuted (1st such prosecution in Britain) (SEN 11 3) SH Milwaukee 414s broke into many computers (some with guessable passwords) *SH Santa Clara prison data system (inmate altered release date) (SEN 10 1). $SH Reps Zschau, McCain computers penetrated, mailings affected (SEN 11 2) SH Grade-changing prank at Stanford (around 1960) (SEN 8 5) $SH Southwestern Bell computer penetrated: free long-distance calls (SEN 11 3) $SH Bloodstock Research thoroughbred horse-genealogy computer system penetrated $SH Debit card copying easy despite encryption (DC Metro, SF BART, etc.) $SH Microwave phone calls interceptable; cordless, cellular phones spoofable $SH Callback security schemes rather easy to break (SEN 11 5 from RISKS-3.29) SH Systematic breakins of Stanford UNIX systems via network software (SEN 11 5) Brian Reid, "Lessons from the UNIX Breakins at Stanford", pp 29-35, Oct 1986 ..... Trojan Horses $SH PC Graphics program Trojan horse (ArfArf) wiped out users' files (SEN 10 5) SH Another Trojan horse trashes DOS -- NOTROJ (SEN 11 5) $SH Harrah's $1.7 Million payoff scam -- Trojan horse chip (SEN 8 5) SH C compiler Trojan horse for UNIX trapdoor (Ken Thompson, "Reflections on Trusting Trust", 1983 Turing Award Lecture, CACM 27 8, August 1984) ..... Internal perpetrations: $SH Nevada slot-machine ripe for $10 to 15 million phony payoffs? (SEN 11 2) *SH San Fran. Public Defender's database readable by police; as many as 100 cases could have been compromised [Feb 1985] (SEN 10 2) \SH Election frauds by vendor? by operations staff? (SEN 11 3),... [see above] \*SH British auto citations removed from database for illicit fee (SEN 11 1) *SH Software time-bomb inserted by unhappy programmer (for extortion?) (10 3) *SH Los Angeles Water&Power computer system software time-bomb (SEN 10 3) SH DC analyst in dispute with boss changed password on city computer (SEN 11 2) S Sabotage of Encyclopedia Brittania database (SEN 11 5) $SH "Goodbye, folks" software prank costs perpetrator 1000 pounds (SEN 11 3) *S Air Force sells off uncleared tapes with sensitive data (SEN 11 5) $SH Embezzlements, e.g., Muhammed Ali swindle [$23.2 Million], Security Pacific [$10.2 Million], City National Beverly Hills CA [$1.1 Million, 23 Mar 1979] Marginally computer-related, but suggestive of things to come? UNINTENTIONAL DENIALS OF SERVICE: * ARPANET ground to a complete halt; accidentally-propagated status-message virus [27 Oct 1980] (SEN 6 1: Reference -- Eric Rosen, "Vulnerabilities of network control protocols", SEN, January 1981, pp. 6-8) - Gobblings of legitimate automatic teller cards (SEN 9 2, 10 2, 10 3, 10 5) - Royal Wedding side-effect shuts down computer machine room? (SEN 11 5) * Central computer for Austin auto traffic lights & 2 lights out (SEN 11 5) $ Computer crash stops gasoline pumps (SEN 11 5) $ Many cases of point-of-sale systems crashing, business lost (SEN 11 5) - Program bug in Computerized Coke machines caused many phone calls (SEN 10 2) - Network node hit by lightning; down for weeks (SEN 11 5) $H IRS has no contingency plans for computer disasters (GAO report) (SEN 11 2) - VMS tape backup SW trashed disc directories dumped in image mode (SEN 8 5) ---> Denials of Service/Interference of Communications ('\' = noted above): \!! Sheffield (20 deaths), pacemakers (2 deaths), \*$ Challenger communications, CB auto interference, Ghost phone calls, \$ telephone outages, hurricane closes Midwest Stock Exchange, \$ bug in "Grind" stage set software halts production, \- Pres.Reagan's command plane; Sputnik effects on garage doors AGGRAVATION TO INDIVIDUALS OR TO THE POPULACE AT LARGE: $H Whistleblowing aerospace SW Quality Assurer fired, life threatened(SEN 11 3) * Carrier control unit blamed for nuclear false alarm (SEN 11 5) $S Sputnik frequencies triggered garage-door openers $ Customer declared dead by bank computer; effects propagated (SEN 11 3) $ Demo NatComm thank-you mailing mistitled supporters [NY Times, 16 Dec 1984] - Earthquakes: 3 of 5 reported never happened; microwave static (SEN 11 5) H Query of vacationing programmer starts beer panic (SEN 11 5) H Indian program to reroute bus lines trounced (SEN 11 5) - British school examination program gave erroneous grades (SEN 11 5) \*H Various cases of false arrest due to computer database use (SEN 10 3,11 1) \- Various cases of gobbled bank cards LEGAL IMPLICATIONS: !$ Deaths of 3 lobstermen in storm not predicted by National Weather Service -- 3 mos unrepaired weather buoy; $1.25M award (SEN 10 5) [NY Times 13 Aug 85] Overturned by federal appeals court. [AP, 15 May 86] (SEN 11 3) ** Launch on warning legality subject of law suit (SEN 10 2, 11 5) $ Sex-therapy software risks (SEN 11 2) $ Computerized sex ring broken; records seized (SEN 11 5) $ Israeli supreme court appeal blamed on computer malfunction (SEN 11 5) *$ Expert systems for criminal investigations (SEN 11 5) $H Lawsuit against Symphony for leaving out proposal section (SEN 11 5) S Concern over privacy of Swedish Databank (SEN 11 5) MISCELLANEOUS COMPUTER HARDWARE/SOFTWARE PROBLEMS: - Clock setting algorithm gets wrong time; other clock problems (SEN 11 2) $ Tape unit caught on fire from repeated reading of tape section (SEN 5 1) - Some destructive computer puns H Incidents on people's willingness to trust computers (SEN 11 5) - See also anecdotes from ACM Symposium on Operating Systems Principles, SOSP 7 (SEN 5 1) and follow-on (SEN 7 1). ------------------------------ End of RISKS-FORUM Digest ************************ -------