RISKS-LIST: RISKS-FORUM Digest Friday 16 June 1989 Volume 8 : Issue 80 FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Disarmament by defect (Gerard Stafleu) Even human-in-the-loop isn't foolproof. A test case. (Pete Holzmann) Single point of failure? probably not. (Ephraim Vishniac) Re: single point of failure -- Tokyo Stock Exchange (Patrick Wolfe) Qantas Airliner Mishap (John Murray) Theorem Proving by Computers (Tom Thomson) Re: Computer electrocutes chess player ... (Dave Horsfall, Joel Kirsh) Clerical error spares famed sex-fiend (Mike Albaugh) Sabre computer problems revisited (Emily H. Lonsford) Protection from Misdirected Radio Control Commands (Robert Horvitz) [*** This issue attempts to reduce the backlog. I will have limited ***] [*** net access for a week, but keep the good contributions coming. PGN ***] The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, and nonrepetitious. Diversity is welcome. * RISKS MOVES SOON TO csl.sri.com. FTPable ARCHIVES WILL REMAIN ON KL.sri.com. CONTRIBUTIONS to RISKS@CSL.SRI.COM, with relevant, substantive "Subject:" line (otherwise they may be ignored). REQUESTS to RISKS-Request@CSL.SRI.COM. FOR VOL i ISSUE j / ftp KL.sri.com / login anonymous (ANY NONNULL PASSWORD) / get stripe:risks-i.j ... (OR TRY cd stripe: / get risks-i.j ... Volume summaries in (i.j)=(1.46),(2.57),(3.92),(4.97),(5.85),(6.95),(7.99). ---------------------------------------------------------------------- Date: Thu, 15 Jun 89 11:00:29 edt From: Gerard Stafleu Subject: Disarmament by defect We have seen quite a few articles on things going wrong with the computerization of the military. The latest example is the posting by Karl Lehenbauer about "NORAD Computers: Years Late, Unusably Slow, $207 Million Over Budget". While most articles concerned the Western military, there is no doubt that our friends on the other side suffer from the same problems. After all, they are doing their level best to get their hands on as much Western computer technology as possible. (I have heard rumors that getting our technology to them is one of the most subtle and insidious plots developed by the CIA so far.) As a result, it is reasonable to suspect that the advance of computer technology into the field of the military, has made it well neigh impossible to fight any war worth its SALT. We find corroborating evidence for this position in the on-going disarmament proposals. These have been started by Gorbachov, who knows an impossible situation when he sees one, and have now been taken over by Western leaders like Bush, who perhaps reads comp.risks. So where the sheer incompetence of politicians and generals used to start wars, the sheer incompetence of us computer people has now put an end to it. No mean feat. For centuries humanity has been looking for the Weapon That Would End War Forever. We have found it. War has ended, not with the bang of a bomb, but with the gentle whisper of crashing software. Gerard Stafleu, (519) 661-2151 Ext. 6043 BITNET address: gerard@uwovax ------------------------------ Date: Wed, 14 Jun 89 14:10:29 PST From: Pete Holzmann Subject: Even human-in-the-loop isn't foolproof. A test case. I was recently witness to an event that may be of interest to those pondering safe user interfaces, man-in-the-loop questions, and the like. Not being an expert in any of these areas myself, the only comment I'll make is that it seems important to realize that there are cultural aspects of human-technology interfaces. Never assume that a sane, well-trained person will do 'the right thing'... The following is a true story. No names are given, so as to protect the participants from any further embarassment! The scenario: experienced computer user/programmer needs to get some software mailed out during the weekend. He's relatively new to the office, so he has asked where the spare floppies are kept. He is told "there's a box with a bunch of floppies over on Joe's desk". There's a small error in these instructions: the correct box of spare floppies is on Jane's desk, not Joe's. What happens: He goes into the office alone on Saturday morning. Nobody is there to watch him (not that anybody normally would -- he's an expert, remember!) He finds no box of floppies on Joe's desk. But - aha! - there's a nice big box UNDER the desk. It is sealed. (It was delivered the day before.) He opens the box, and finds a bunch of brand-new commercial software packages. Shrink-wrapped, the whole bit. Without skipping a beat, he rips open a couple, and finds sealed white envelopes inside. The envelopes have your usual dire license agreement warning, beginning with a large STOP sign... ("STOP. Read before opening! etc...") Without skipping a beat, he rips open the envelopes, reformats the enclosed floppies, puts on new labels, and uses them to mail the software he needed. Thus ruining a few thousand dollars worth of new commercial software! Now, before you read the answer, think about this puzzle: how could a sane person, an *expert* no less, completely ignore the warnings and do such a crazy thing? What are the RISKS implications of this? Here's the answer: He was able to do it, without even wondering whether it was the right thing to do, because *in his experience*, what he saw and his resulting actions were completely normal. In a previous job, his company received large quantities of commercial software for evaluation and review. So much software that they treated it like junk mail. The floppies were treated as reusable media. With that in mind, his actions become completely reasonable! He was trained to ignore dire warnings, expensive-looking software packaging, and the like. The only thing of value in a box of commercial software, in his experience, was the floppy disks themselves. And they were only useful once reformatted and with fresh labels on them. Hmmmmm... Pete Holzmann, Strategic Locations Planning {hpda,pyramid}!octopus!slp!pete ------------------------------ Date: Thu, 15 Jun 89 09:47:53 EDT From: ephraim@Think.COM Subject: single point of failure? probably not. In RISKS 8.79, Jerry Carlin (jmc@PacBell.COM) cites a story from the SF Chronicle (presumably San Francisco, and not some 'zine): The reporter quotes a story in "Manhattan, Inc" where it was disclosed that the main and backup computer for the Tokyo Stock Exchange sit right next to each other and in an area totally destroyed by the 1923 earthquake. This computer is the SOLE repository of Japan's offical records of stock ownership. Therefore if the computer is destroyed, all records of share ownership could disappear with obvious consequences. It seems very unlikely that the computer is the SOLE repository. More likely, the two computers together with the on-site and off-site backups of the data they contain are the widely distributed and highly redundant repository of the stock ownership data. That's not such an exciting story, of course. Supposing that Tokyo Exchange follows conventional backup procedures (and they could easily do much better), destruction of both computers would mean the loss of the current day's transactions; destruction of the entire site might mean the loss of as much as one week's transactions. That's expensive, but it's not catastrophic. Ephraim Vishniac, Thinking Machines Corporation, 245 First Street, Cambridge, MA 02142-1214 ------------------------------ Date: Thu, 15 Jun 89 07:27:26 cdt From: pwolfe@kailand.kai.com (Patrick Wolfe) Subject: Re: single point of failure -- Tokyo Stock Exchange > This computer is the SOLE repository of Japan's offical records of stock > ownership. Therefore if the computer is destroyed, all records of share > ownership could disappear with obvious consequences. This is why people in my position spend so much time with and are so concerned about backups, so that the computer is not the "SOLE respository" of any valuable information. Well managed computer centers keep a set of complete backups "offsite". The ones with larger budgets use an storage location complete with protection against fire and other environmental hazards. The only story I have heard about a computer center that didn't keep any backups is about US Cable in Lake County, IL. Every six months or so, they would unscramble all six pay channels for everyone for about a week, reportedly because of a "computer problem" where they lost information about who was paying for which channels. If they had reliable backups, these records could have been restored in a matter of hours, instead of a week. Patrick Wolfe (pat@kai.com, kailand!pat) System Manager, Kuck & Associates, Inc. ------------------------------ Date: Thu, 15 Jun 89 17:30 PDT From: johnm@uts.amdahl.com (John Murray) Subject: Qantas Airliner Mishap I heard an NPR report recently about a Qantas plane going out of control temporarily. It seems the autopilot suffered some sort of glitch. The (human) pilot recovered from the dive, but several people bumped their heads, etc. Since then, I've heard no follow-up, and seen nothing in comp.risks. Was I hallucinating about the original report, or do I just have my head in a bag this month?? - John Murray, Amdahl Corp. ------------------------------ Date: Thu, 8 Jun 89 09:43:04 bst From: Tom Thomson Subject: Theorem Proving by Computers Henry Spencer comments on the acceptance by mathematicians of proof by computer. I think it's important to recognise that the computer introduces no new risk here; we all believe group classification theorem, don't we, and surely no-one has ever found time to check the proofs (or even understand the underlying arguments) of all the lemmas and prior theorems involved therein. Mathematics has a long history of "proofs" that aren't (eg the omission of axioms about betweenness in geometry for a couple of thousand years); and quite a few "theorems" have been disproved. Checking a proof is no easier than checking a program. Checking that several proofs combine correctly to deliver a new proof is no easier than checking that several programs combine correctly. Do we have a new risk here - the risk that, because a computer is involved, we will assume a new risk exists even when it doesn't (or is not new)? Tom Thomson ------------------------------ Date: Thu, 8 Jun 89 11:21:29 est From: Dave Horsfall Subject: Re: Computer electrocutes chess player who beat it! (RISKS 8.75) [ Discusses receiving a strong shock from a 12-volt wiper ] More likely he received an inductive shock from the electric motor. There is no way that a mere 12 volts will cause that sensation, but a kick of a few hundred (thousand?) volts will do it, as the field collapses. ------------------------------ Date: Thu, 1 Jun 89 21:51:00 EDT From: Joel Kirsh Subject: Computer electrocutes chess player [Excerpted, from "Bioengineering: Biomedical, Medical and Clinical Engineering", by A.T. Bahill (Prentice-Hall)] The impedance of the human body can be modeled as a core of low resistance (around 500 Ohms) ... and the skin with a higher resistance (1 to 100 kiloOhms). ... the amount of electrical current necessary to induce venticular fibrillation [a "cardiac arrest"] in the human heart ... a minimum of 80 microAmps, 100 uA, and 180 uA [in three separate studies]. These values lead to estimates of the required voltage being anywhere from 240 mV (80uA times 3 kOhms) to 16 V (180 uA times 201 kOhms). Of course, this assumes that the current path crosses the chest. Also, the heart is especially susceptible to particular frequencies; good old 60 Hz is "the optimum frequency for producing ventricular fibrillation." (Bahill) Joel Kirsh, Faculty of Medicine, University of Toronto ------------------------------ Date: Wed May 31 10:54:17 1989 From: albaugh@dms.UUCP (Mike Albaugh) Subject: Clerical error spares famed sex-fiend Quoting from Colin Wilson's "The Misfits": The revolutionary Marat decided that de Sade was a typical aristocratic libertine of the old regime and ought to die; by accident, however, he denounced the Marquis de la Salle, who was executed. Marat discovered his mistake and was about to rectify it when he was murdered in his bath by Charlotte Corday. Unaware of how close he had been to the guillotine, de Sade delivered an address describing Marat as a great man. The parallels to modern wrongful arrest struck me, as well as the question of how bad the reign of terror might have been with the "help" of modern data processing. It appears the over-reliance on the accuracy of "official" orders has been around for a while. Perhaps Madame DeFarge should have used an error-correcting code in her knitting? [My remembrance of early dp is that redundancy in the form of hash totals and transaction serial numbers was used quite early, and seems to have been forgotten, rather than enhanced, as we have "advanced"] Mike ------------------------------ Date: Tuesday, 30 May 1989 10:01:46 EST From: m19940@mwvm.mitre.org (Emily H. Lonsford) Subject: Sabre computer problems revisited According to the May 22, 1989 issue of Computerworld, Sabre is run on 8 interconnected 3090-200E computers under a Sabre-modified version of ACP (Airline Control Program OS by IBM). A custom version of ACP has been used there for about 20 years. Neither ACP nor TPF 3.1 (due to be installed 3rd qtr 89) provides the required protection, according to the article. It seems the errant 'core-walker' program modified another task that was formatting disk drives - and the labels on 1080 disk drives were destroyed. "The Sabre system is down an average of six minutes a week for maintenance, Juracek noted, and is usually upgraded 'on the fly' so that service to other parts of the world is not disrupted. Because ACP cannot run without a disk subsystem, Sabre software engineers took the unusual step of rebooting the crashed system using IBM's VM operating system. Then, they had to relabel each disk drive and reset the pointers that indicate where passenger data is loca- ted....While most Sabre data was not lost, the 'pointers' to all flight reser- vation data were - and it took 100 programmers and systems engineers more than 10 hours to relabel each disk volume. The system was restarted under ACP about 7 am CDT, and the reformatting was done by 11 am. Then, due to pent-up network demand, American's systems engineers had to gradually restart Sabre, slowly ad- mitting more traffic from 27 front-end communications processors here." ACP and TPF are IBM real-time operating systems that are designed to support heavy transaction volumes. The article goes on to state that virtual storage will not be available under TPF until 1993. Apparently other protection fea- tures are not there either, such as private address spaces and multiple storage protection keys, which are implemented under MVS. Emily H. Lonsford, MITRE - Houston W123 (713) 333-0922 ------------------------------ Date: Fri, 2 Jun 89 00:45:43 pdt From: rh%well%apple@sun.UUCP (Robert Horvitz) Subject: Protection from Misdirected Radio Control Commands In RISKS 8.75, MIchael Berkley quoted a newspaper article about an accident in northern Ontario in which a radio-control signal intended for one mining machine triggered an unintended response in a second machine, which pushed a miner to his death. Berkley asked: "What kind of safeguards are possible in this situation and are the safeguards reliable?" I am not familiar with Canadian regulations for radio control, but they are probably similar to US regulations. As it happens, the FCC has just adopted new rules governing radio signals from unlicensed devices, including radio control systems (Gen. Docket 87-389: First Report & Order adopted 30 March 1989). The Commission is explicitly trying to encourage the proliferation of low-power unlicensed radio devices of all types, in the spirit of "deregulation" promoted by outgoing FCC Chairman Dennis Patrick. The primary feature of the new "Part 15" rules is to loosen restrictions on the use of radio links in appliances and systems sold publicly. The new rules begin to take effect on June 23rd. They are sure to lead quickly to a rash of new products such as wireless modems, wireless VCR/camera units, new remote monitoring and control systems for the home, etc. One aspect of the new rules relevant to the mining story is that the FCC set no maximum power limit for radio emissions in mines, caves or tunnels. A traditional feature of all "Part 15" devices is that they enjoy no right of protection from interference - either from similar devices or from licensed transmitters. Licensing confers the right of non-interference. Radio control systems are generally unlicensed. Since most of the services that the Commission regulates are for communication, they are used to thinking of interference in terms of, e.g., degradation of TV picture quality. They are not used to thinking of it in terms of misdirected control. In fact, because Part 15 devices have no recognized right of non-interference, the Commission's attitude is - and has always been - "buyer beware/you're on your own." Thus, the only safeguards we can expect in the US, to avoid accidents like the one that killed the Canadian miner, are those voluntarily adopted by manufacturers. Fortunately, there is a relatively simple fix to the problem: have each radio command begin with an identifier specifying which device is being addressed, and have the identifier be unique enough that there is little chance of two devices with the same identifier being co-located. Better, have the owner or operator be able to set the identifiers in the field, to ensure each is unique within the transmitted signal's radius. Over a dozen petitions have already been filed objecting to the FCC's new rules. I will probably be filing comments soon on behalf of the Association of North American Radio Clubs. I may raise this issue of radio control safety in my filing. But I'm sure the Commission will say that this is a matter for the marketplace can decide, and no "interference" from them is needed. ------------------------------ End of RISKS-FORUM Digest 8.80 ************************