RISKS-LIST: RISKS-FORUM Digest Monday 13 November 1989 Volume 9 : Issue 42 FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Equinox TV programme on A320 (Bev Littlewood, Chris Dalton) European Safety is not always BETTER (Bruce C. Brown) Artificial lightning (PGN) Another intrusive database with associated privacy problems (Bill Gorman) Re: "Computer Error" in Durham N.C. election results (Gregory G. Woodbury) Re: Computer errors and computer risks (Willis H. Ware, D. King) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, and nonrepetitious. Diversity is welcome. CONTRIBUTIONS to RISKS@CSL.SRI.COM, with relevant, substantive "Subject:" line (otherwise they may be ignored). REQUESTS to RISKS-Request@CSL.SRI.COM. TO FTP VOL i ISSUE j: ftp CRVAX.sri.comlogin anonymousAnyNonNullPW cd sys$user2:[risks]get risks-i.j . Vol summaries now in risks-i.0 (j=0) ---------------------------------------------------------------------- Date: 13 Nov 1989 17:42:48-GMT From: B.Littlewood Subject: Equinox TV programme on A320 Brian Randell, in RISKS-9.39, gave a brief description of this programme, which he thought was quite well done. I have a few more reservations than Brian, but this may be because most of the film of me ended up on the cutting-room floor! There were some interesting moments, though, not least in some of the assertions still being made by Airbus and its constituent companies. Throughout there was a curious belief in the power of the `systems approach'. Perhaps the strongest statements came from Gilles Pichon (Chief Engineer, A320, Aerospatiale), who said: "Safety is ensured by the system approach: the computers, the sensors, the whole environment, the power sources and the computer software. All this has to be analysed very closely for safety. And the method of analysis, which is called safety analysis, has been around for over 20 years. It was used for Concorde and it's still valid today." Pretty unexceptionable, you may think, but this statement is quickly followed in the film by a more detailed examination of the fault-tolerance of the fly-by-wire system: Jacques Troyes (Flight Control Manager, Aerospatiale): "We work with three pieces of information completely independent of each other. And we can only guarantee to protect the aircraft with certainty when we are sure of having at least two pieces of information." I assume this is a reference to the A320 software fault tolerance, but if so it is a bit confusing. The voice-over questions whether developing all versions from the same specification might introduce common faults. Then cut to John Knight (University of Virginia): "It appears to be possible to build diverse programs where the programs will allow you to recover from certain kinds of difficulties that the software may get into. The real issue is we have no way of predicting ahead of time how successful that kind of technology is going to be." Voice-over then says Airbus have stated the system is designed to fail only once 10**-9 hours [lovely slip that -- they mean 10**9!]. Then cut to me Bev Littlewood (City University): That means this system should fail every billion flight hours. A billion flight hours is about 100,000 years. There is no-one in this business believes you can design systems to that reliability." Pichon: "Having done all our safety analysis we are confident that we achieved the 10**-9. And for those who have some doubts we can also say that even with no fly-by-wire the aircraft can fly safely because we have the mechanical back-up at the end." Voice-over explains that mechanical back-up is really only meant to keep the aircraft flying until computer system is got working again. Littlewood: "Airbus could have had a fully functioning mechanical back-up on that aircraft, so that in the event of total loss of the computer system it was still flyable. Now what they've got is a vestigial mechanical back-up. Really all that you can control if you totally lose the computer system is the rudder and tail trim." Gordon Corps (Engineering Test Pilot, Airbus Industrie): "I had one Northwest Airlines pilot land the airplane totally satisfactorily using just the back-up system." Later in the film, there is an interview with Michel Asseline, who was pilot in charge of the A320 which crashed on the Mulhouse flight. Asseline: "When I pull the stick to up position, the flight controls, the elevator controls, go to down position . . . why? That would be the good question." Whereupon a man from the DGAC (the French certification agency) is asked whether he had seen any evidence to support this claim. He said he had not. Then cut to Bernard Ziegler (Vice-President, Engineering, Airbus Industrie): "By no means, never, the computer want to land the aircraft, never. I would even say, believe it or not, that we have put in our computer law to resist to land. The pilot land the aircraft, and nobody else." The voice-over then comments that, 15 months later, the official French report into the crash has not been published, but it will almost certainly clear the computer. Later there are reports of other problems pilots have met, including the following exchange Gino Scattolini (A320 pilot): "As we were coming in to land with the engines at idling speed, the two engines accelerated up to climbing speed, and as the automatic systems were not working we might have left the plane's flight path if the crew had not intervened. But safety was never at risk." Corps: "There have been fine tuning changes done in some aspects of the software and I guess they will go on for some time, as we say just to cure some of the teething problems that we have seen. But they haven't affected anything of significance associated with flight safety at all." The film ends by looking to the future, and in particular the possibility of unstable commercial aircraft. Ziegler: "It's clear that we say active control, which is a natural derivative of the fly-by-wire, we will be able to reduce the weight of the structure, to reduce the surface of the control. That is also the next step and we are working also in this direction." Littlewood: "Now making an airliner unstable would bring enormous economic benefits because it would cut down drag and the aircraft would be much more fuel efficient. But an unstable aircraft has to be controlled by computer all the time; there is no possibility of a mechanical control by the pilot. So that next step is one I think we ought to be worrying about." Brian Perry (UK Civil Aviation Authority): "There's nothing we know which would say we shouldn't consider such an approach. We believe that if you take the system approach which looks at the hazards following system failure or system non-availability, and can satisfy yourself that the safety criteria are met, then the aircraft is potentially certificatable." Certainly I agree with Brian that the film is worth seeing (I think it is to be shown in the US -- probably on Public Broadcasting). It would have been good to have more debate and less lovely pictures of the A320 doing fancy things. But a couple of things did come out. First, it seems that senior engineers (Pichon, above) are still trying to convince us that they have achieved the mythical 10**-9. Are they fools or knaves? Second, there seems to be confusion about exactly what can be expected of the back-up system. Do Airbus want us to believe that airline pilots will be able to land on this, or that they will never need to do so? (or both?) Third, there have been software problems. (I'm intrigued by the notion of `fine tuning': is this similar to `it's not a bug it's a feature'?) Fourth, this was the first formal statement I had heard that Airbus were working on active control. It seems to me that the certification agencies have to take a more active role here than is represented by Perry's statement. BEV LITTLEWOOD, Centre for Software Reliability, City University, London EC1V 0HB ------------------------------ Date: Mon, 13 Nov 89 14:53:55 gmt From: Chris Dalton Subject: Mistake in Equinox "Fly-by-wire" programme The Equinox programme mentioned by Brian Randell and Lindsay Marshall in Risks 9.39 and 9.40 has a glaring mistake in the script... I hope. The announcer quite clearly explains at one point that the system was designed to fail every "ten to the minus nine hours". Moments later, an engineer says they achieved the "10^-9 error rate". (I'd recorded the programme, so I was able to check what was said.) A case of losing something in the translation? Chris Dalton Hewlett_Packard Labs, Bristol BS12 6UF, UK +44 272 799910 crd@hplb.hpl.hp.com crd@hplb.lb.hp.co.uk ..!mcvax!ukc!hplb!crd ------------------------------ Date: Sun, 12 Nov 89 23:17:10 CST From: bcbrown%fnal.dnet@fngate (Bruce C. Brown) Subject: European Safety is not always BETTER Recent discussions in RISKS have suggested that safety standards in Europe are superior to those we enjoy here, and indeed, some recent statistics suggest that that may be true in some important senses. However, we should beware in adopting the stance that they have the answers and we have nothing to give. In particular, I was in Hamburg, Germany for a six month assignment in 1987 and was AMAZED to discover that the American safety requirement that all door open OUT and that buildings have doors which are locked such that noone can be locked in were unknown there. If I forgot my keys and worked late, I could be locked in at three separate levels: my own office, the office corridor, and the building external doors. MOST rooms had telephones, but... Like everything else, we need to be careful about adopting anything wholesale, without review. Bruce C. Brown, Magnet Test Facilty, Fermi National Accel Lab, Batavia, IL 60510 ------------------------------ Date: Mon, 13 Nov 1989 16:19:48 PST From: "Peter G. Neumann" Subject: Artificial lightning Lightning may be natural, or may actually be stimulated artificially by man-made conditions in situations in which lightning might otherwise not occur. The latter occurred in the second and third of the following cases: ... three spectacular lightning accidents involving aircraft or spacecraft: (i) In 1963, a Boeing 707 flying at 5000 feet near Elkton, Maryland, was struck and destroyed by lightning, killing all occupants (3). Lightning apparently burned through one of the metal wings, or in some other manner entered the fuel tank inside that wing, and caused the fuel vapor there to explode. (ii) In 1969, Apollo 12 artificially initiated (or "triggered") two lightning flashes, one to ground and one intracloud (IC) discharge, when it was launched through a weak cold front that was not producing natural lighting (4). Although this rocket-initiated lightning caused major system upsets and minor permanent damage, the vehicle and its crew survived and were able to complete their mission successfully. (iii) In 1987, an unmanned Atlas-Centaur vehicle (AC/67) was launched into weather conditions that were similar to those present at the launch of Apollo 12 and triggered a lightning discharge to ground (5). This discharge upset the computer memory in the vehicle guidance system and produced an unplanned yaw rotation, and the associated stresses caused the vehicle to break apart. This paragraph is excerpted from an article in the 27 October 1989 issue of _Science_, Natural and Artificially Initiated Lightning, by Martin A. Uman and E. Philip Krider, pp. 457-464. References 3-5 are given in the article. The Atlas-Centaur case was previously reported in RISKS-4.70 (1 April 1987, no joke) and RISKS-4.96 (6 June 1987). The Apollo 12 case has not -- to the best of my knowledge been noted here previously. More generally, the detailed discussion of artificially triggered lightning in the Science article should be particularly interesting to RISKS readers. ------------------------------ Date: Mon, 13 Nov 89 14:00:50 EST From: "W. K. (Bill) Gorman" <34AEJ7D@CMUVM.BITNET> Subject: Another intrusive database with associated privacy problems MEXICO AND USA SIGN TREATY TO ATTACK TAX EVASIONS The mexican Secretary of Treasury, Pedro Aspe Armella, and Nicholas Brady, northamerican Secretary of Treasury signed a treaty to detect and combat tax evasions on both countries. By means of this treaty both nations will have access to information concerning the income of mexicans living in the States and of northamericans living in Mexico. This deal also establishes the possibility to exchange information on people who evades taxes. This data will be exchanged only if the laws and rights of the citizens are respected in each country. With this information, the fiscal authorities expect to detect possible tax evasions by incomes obtained in another country and that are frequently not reported to the Government. [... another privacy "loophole" via a shared data base. The comment about protecting privacy by "observing all laws of both countries" is absurd. Once the data is in the hands of any third party, individual, corporate or national, controls imposed from without are nothing more than "gentlemen's agreements" observed out of courtesy and/or convenience. Bill] [Bill did not indicate where this appeared. I edited it lightly to fix a few typos (e.g., Treasure), but left "northamerican Secretary". PGN] ------------------------------ Date: Tue, 14 Nov 89 02:51:39 GMT From: ggw@wolves.uucp (Gregory G. Woodbury) Subject: Re: "Computer Error" in Durham N.C. election results Summary: Election worker opinion > [With a "Duke" as Governor of both Massachusetts and California, > I wonder if any Duke Univ. folks were governing this election? PGN] Well, I work at Duke University, and I was also working as an assistant to the precint registrar for my home precinct -- that might count a "governing" this election ;-) It was an enlightening experience doing an election. The machines used around here are purely mechanical. The only electricity used is for a fluroescent light over the front panel. After the election, a hand crank is used to force the counter wheels against an NCR paper (or rather vice versa) and the numbers are transcribed by hand to the official ballot reports. In my precinct, we had two calculators (electronic) to assist in the tally, and I still caught an error by doing a simple parity check on the numbers as they where called out. As for the "Computer Error" down at the Board of Elections... What goes on there is simply a convenience for the press and candidates. The BoE staff has a few PC's and spreadsheets set up to do simple calculations and the person who got to put it together this year simply messed up one of the cross-tabulations. There is not, as far as I know, a specific program that the BoE uses, just a PC set up in the County Commission meeting room used for simple arithmetic. Prior to last year they used simple hand calculators and never had a problem. The "Official Election" comes about one week after the voting when the registrars from each pricinct sit down "en banc" and canvass the actual numbers from the machines in their precincts and double check each other via whatever method is most convenient. Some of the registrars actually do the arithmetic in their heads and have the result written on their scratch pads before the various calculator people can announce what they get. All in all, its still dependent on mechanisims and mental skill. Gregory G. Woodbury, Sysop/owner Wolves Den UNIX BBS, Durham NC ------------------------------ Date: Mon, 13 Nov 89 16:05:42 PST From: "Willis H. Ware" Subject: Re: Computer errors and computer risks (Saltzer, RISKS-9.41) >> ............................. In a traditional library, it was >> possible to invade your privacy by making a list of all the books you >> have every checked out. All an investigator had to do was open every >> book in the library and look to see if you had signed the card >> inside. The information was publicly available, but actually it was >> benignly protected by an enormous collection cost, so noone every >> worried about it. In privacy discussions, one frequently hears the point about convenience of collection, magnitude of what can be obtained for little effort, etc., but the concept of "benign protection by the status quo" is a very adroit way of capturing the point and of relating it lay folks. His point also brings to mind one made very forcefully by Richard Hamming (currently on the faculty of the USN Postgraduate School at Monetery, CA) many years ago. In paraphrase, he said: "when something changes by an order of magnitude, there are fundamental new effects." Certainly from the benign library of the past to the computerized one of now, the effort to assemble one's reading list has changed by been many orders of magnitude. Hamming's Law is really what's behind so many of the computer-induced effects, and it's also the underlying issue in having such effects understood among the laity. It's certainly a big part of the problem with getting legislators to pay attention; they think everything is fine just because it has been fine in the past. Willis Ware [Hamming is also well known for not standing on Isaac Newton's feet. PGN] ------------------------------ Date: Mon, 13 Nov 89 09:54:08 PST From: king@kestrel.edu Subject: Re: Computer errors and computer risks (Saltzer, RISKS-9.41) >> In RISKS DIGEST 9.40, Randy Davis says, >> > . . . I suggest the simple test above: Ask, can the identical >> > problem can arise in the absence of computers? >> I claim that it is not that simple... I think i must respectfully disagree. Consider the two examples given ... Yes, i will concede that the cost of collecting library patron information precludes its use to send "appropriate" junkmail. The cost of collecting DMV information precluded its use for junkmail as well. But these are trivial invasions of privacy, and not the ones i'm most worried about. Consider the possibility of a new McCarthy Era. During the old McCarthy Era, readers of certain books in the library WERE found and used for purposes which i would assume many would just as soon forget. The fact that this information was available in dilute form protected nobody. Recall that both the imaginary but believable society of 1984, and the real tyrrany of Nazi Germany, were quite plausible/possible with only "human computers". Consider the case of the skiptracer. The cost of a DMV trip would be a negligible portion of his cost of doing business; no doubt he would have several cases he could service with a single trip. So, in part, i support the original thesis that for serious breaches of privacy [as opposed to trivial annoyances] lack of a computer is no protection against data collection. In part, i offer a possibility for a NEW protection. It is practical for the head librarian of even the largest city to personally walk the half-dozen disk packs containing circulation information to the library's degausser, together with the appropriate tapes, and defend the privacy of the more-than-two-month-old circulation information reasonably absolutely. It is possible for the populace to order the DMV to implement access poicies. In short, the compactness of the information implies that the privacy afforded patrons of a particular service will not be the accidental result of the way things happen to be, but the result of an explicit decision. -dk ------------------------------ End of RISKS-FORUM Digest 9.42 ************************