Subject: RISKS DIGEST 12.40 REPLY-TO: risks@csl.sri.com RISKS-LIST: RISKS-FORUM Digest Wednesdy 25 Septembr 1991 Volume 12 : Issue 40 FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Bell V-22 Osprey - correct sensor outvoted (John Wodehouse) Challenger O-ring Problem heads topics at conference on ethics (George Leach) People and Public Screens (Antony Upward, PGN) Credit bureaus, heisenbugs, and clerical errors (Peter G. Capek) Electronic locks at Harvard (David A. Holland) Bad error handling in Lamborghini Diablo engine management (Richard Boylan) Denver Hacker Hacks NASA (Andy Hawks) Re: MSAFP, utilities, and all that (Eric Eldred) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, and nonrepetitious. Diversity is welcome. CONTRIBUTIONS to RISKS@CSL.SRI.COM, with relevant, substantive "Subject:" line. Others ignored! REQUESTS to RISKS-Request@CSL.SRI.COM. For vol i issue j, type "FTP CRVAX.SRI.COMlogin anonymousAnyNonNullPW CD RISKS:GET RISKS-i.j" (where i=1 to 12, j always TWO digits). Vol i summaries in j=00; "dir risks-*.*" gives directory; "bye" logs out. The COLON in "CD RISKS:" is essential. "CRVAX.SRI.COM" = "128.18.10.1". =CarriageReturn; FTPs may differ; UNIX prompts for username, password. ALL CONTRIBUTIONS CONSIDERED AS PERSONAL COMMENTS; USUAL DISCLAIMERS APPLY. Relevant contributions may appear in the RISKS section of regular issues of ACM SIGSOFT's SOFTWARE ENGINEERING NOTES, unless you state otherwise. ---------------------------------------------------------------------- Date: 25 Sep 91 09:23:00 EST From: "John Wodehouse" Subject: Bell V-22 Osprey - correct sensor outvoted Further information about the V-22 crash from Flight International 18-24 September 1991. "A Bell-Boeing V-22 Osprey tiltrotor is flying again for the first time since the crash of aircraft number five on its first flight in June. Aircraft number three has made at least three flights, after extensive checks by the US Navy (USN). The USN has also released a brief report on the accident, which reveals that similar faults have been found in two other aircraft. It says that TWO roll-rate sensors (my capitals), know as vyros, which provide signals to the flight control computer, were reverse-wired. In the triple-redundant system the two faulty units "outvoted" the correct sensor, leading to divergent roll cycles and a crash shortly after take-off. The report says the cockpit interface unit is connected by a 120-wire plug connector in which the vyro unit uses numbers 59 and 60 - which were reversed. Examination of aircraft one and three revealed that one vyro in each was also reversed. The number three aircraft flew for 18min on 10 September in a flight cut short by extremely poor visibility. It flew again the next day, and was to complete a third flight on 13 September." What worries me is that aircraft one and three were obviously flying with one vyro reversed-wired for quite sometime. The triple-redundant system would have outvoted this vyro, but why was no indication given that there was a problem at all. What confidence does that provide for other systems, which depend on voting, if the failure is not reported. Lord John --- the programming peer [We have reported on similar cases in RISKS before. For example, see J.E. Brunelle and D.E. Eckhardt, Jr., Fault-Tolerant Software: An Experiment with the SIFT Operating System, Proc. Fifth AIAA Computers in Aerospace Conference, 355-360, 1985, where two programs written by different people to the spec of a correct program had a common flaw, and outvoted the correct program. PGN] ------------------------------ Date: Tue, 24 Sep 91 10:53 EDT From: reggie@pdn.paradyne.com (George Leach) Subject: Challenger O-ring Problem heads topics at conference on ethics >From the Tuesday, September 24, 1991 issue of the St Petersburg Times: "Event explores ethics in business" An engineer who worked on the Challenger says not "doing the right thing" can have dire consequences, but so can acting ethically. By John Craddock, Times Staff Writer TAMPA - When the engineer studying the O-rings on the space shuttle Challenger suspected they might cause a catastrophe, he told his bosses. They listened. Then they made what former Morton Thiokol engineer Roger Boisjoly called "a management decision." That decision launched a tragedy. The O-rings failed, and the Challenger exploded Jan. 28, 1986. In later statements before a presidential commission and in documents he produced, Boisjoly showed he had tried to do the right thing. But for him, doing the right thing ethically meant the undoing of his professional life. "I stepped into quicksand....It was the total destruction of myu career," he said. Discussions - and confessions - about ethical behavior and what it means to professionals, - are the theme of a two-day conference at the University of Tampa. The conference ends today. Titled "Doing the Right Thing: Revolutions in Professional Ethics," the conference Monday attracted a blue-chip panel of ethical experts, as well as politicians, lawyers, and journalists. Among those speaking Monday morning was Gov. Lawton Chiles. He told the group of about 150 that he doesn't blame the lack of ethical fiber in recent years on "the mindless materialism of the 1980's" creating a "moral vacuum across the land." He said unethical behavior has always been with us. "I'm not sure we can blame it all on the 1980's," said Chiles, who has been involved in politics since the 1950's. He noted one difference: The lack of surprise when people hear that a judge is taking bribes or other news of the public trust being betrayed. "Our citizens are no longer shocked," he said. That's why political and business leaders must step out and "be willing and able to do the right thing." He then launched into his own campaign to build trust with the people of Florida and cut state spending. [Note: the St Pete Times reported last week that the state's projected revenue will fall short by some 623 million dollars - prompting cuts, including in education - gwl] Boisjoly, who appeared in an afternoon session, said the anguish he felt from his experience at Morton Thiokol was two-fold. He wondered whether his own protests were strong enough and whether he could have prevented the Challenger tragedy. He also said his company came to view him as a traitor. The public tends to view whistle-blowers as "good guys," he said. But the perception in government and corporate circles is that "we're the bad guys. We're the messengers with bad news." Other speakers included Manuel Velasquez, director for the Center for Applied Ethics at Santa Clara University in California. He said business ethics are somehow presumed to be separate from the everyday ethical decisions people make. He said people tend to think of business as a poker game with its own rules. But business ethic "are not specialized," he said, and shouldn't be considered outside the normal bounds of fair play. ------------------------------ Date: 25 Sep 91 07:34 GMT From: UPWARD.A@applelink.apple.com (KPMG - Antony Upward,IVC) Subject: People and Public Screens I was recently returning to Paris from Birmingham (UK). Birmingham international airport has just opened a new terminal, including of course, the latest in computerised information systems to keep travellers informed. It appeared that they no longer have a direct link between the screens being updated with new information (e.g., Flight BA5310 Boarding Gate E, or flight BM540 delayed 30mins), and a public announcement to the same effect. The public announcements seemed to be about 5 minutes after the screens were updated. My flights gate details were displayed - Gate E. I, and about 100 other passengers, went to gate E, and waited. There were no airline staff present. After about 5 minutes of 100+ people waiting at Gate E the public address system announced, quite calmly (not indicating that the screens were displaying wrong information), that my flight was boarding at Gate D. *NO ONE MOVED*. No one believed the public announcement, even though there were no airline staff at Gate E. It was only when one of the airline staff at Gate D wondered why none of the passengers had turned up that they came in person to investigate. Of course we were all waiting at Gate E. Only then, when the announcement was made in person, were the information on the screens disbelieved! It seems, at least on this experience, that a majority of people now `trust' the information on screens, even when it is directly contradicted by a human announcement, and by circumstantial evidence that the screens are not correct. Antony Upward, Apple Computer Europe ------------------------------ Date: Wed, 25 Sep 91 15:14:08 PDT From: RISKS Forum Subject: Re: People and Public Screens On my previous trip East I discovered an annoying bug in United's display program. My flight was not listed on the multiscreen DEPARTURES display. After checking back several times, I discovered the problem: whichever flight should appear on the LAST LINE on the FIRST SCREEN of a multiscreen DEPARTURES display was getting truncated. An example of off-by-one programming, probably. I wonder if anyone fixed it yet? [I thought I had reported this one previously, but I cannot find it in the archives, and it seems too cute and relevant not to include. PGN] ------------------------------ Date: Tue, 24 Sep 91 00:32:15 EDT From: "Peter G. Capek" Subject: Credit bureaus, heisenbugs, and clerical errors I reported here recently about the effect which might occur to an individual's credit rating as a result of many inquiries by, say, car dealers, where that person was shopping around for a car. The dealers, in order to assess the likelihood that a person might buy a car would request a credit report on the individual, but the effect of repeated such inquiries was to give the impression that the person was overextending himself. (RISKS-12.20) The Wall Street Journal today (23Sep91) reports on credit bureaus and their difficulties. Specifically relating to the earlier comment is a description given by a headhunter who would obtain, from a candidate's credit bureau report, the names of other firms who had recently requested that report. He could then call the candidate and say, quite accurately, "You're applying to X and Y and Z; why don't you also consider W?" (I believe that the law regulating this, the 1971 Fair Credit Reporting Act, requires inclusion in the report of the names of all those to whom a copy was sent within the last 2 years; was this requirement intended to let the individual know who had seen the data, or to let the requesters coordinate amongst themselves what credit had been granted, etc?) The most interesting item in the article, however, is the intriguing lead, in which much of the citizenry of Norwich, Vermont, is abruptly flagged as bad credit risks by TRW. The problem was ultimately tracked down to an alarmingly simple error: A person working part time (for a similar, but not apparently related company) at obtaining public records and feeding them back to the credit bureaus had been asked to obtain the list of Norwich's delinquent taxpayers. She mistakenly got the list of tax receipts and carefully reported that some 1400 residents -- in a town of 3100 -- were delinquent. It took nearly three weeks to clear up; half the delay was simply in getting TRW to return repeated phone calls. [It seems as though a reasonableness check on the (size of the) delinquent list might have averted the problem.] But the article goes on to shed some light on what may be the motivations of the credit agencies: their customers, banks and stores, are anxious to obtain reports with the largest amount of negative data, thinking that it has the effect of maximizing their probability of detecting a bad risk. Since the bureaus are paid by the organizations to whom they provide the reports, and not by those whom the reports describe, one is led to speculate on their motivation. Risks? I think the effect of the requirement to record and report the names of those receiving the report may be seen here to have boomeranged. Perhaps that problem wouldn't arise if those names were recorded, but only reported to the individual. The lesson here may simply be that we need to be as conscientious in assessing the risks of our solutions, as we are in evaluating the problems they address. Peter G. Capek ------------------------------ Date: Tue, 24 Sep 91 11:50:11 EDT From: dholland@husc.harvard.edu Subject: Electronic locks at Harvard >From the Harvard Independent, Sept. 19, 1991, pg. 4: The Key to Security Computerized ID cards are the wave of the future, but for residents of the three Union dormitories - Greenough, Hurlbut, and Pennypacker - time seems to be moving faster than in other parts of the University. The Harvard University Police Department (HUPD) has replaced the standard entryway keys for each of these dorms with computerized, credit-card-like key cards. According to HUPD chief Paul Johnson, the cards prevent unauthorized persons from gaining access to the dorms, enable the police department to track the use of each key card by computer, and prevent people from jimmying locks. "It's state of the art," said Johnson. Union dorm residents feel more secure with the improved locks. Said Pennypacker resident Missy Francis '95, "Ninety percent of the upperclassmen have skeleton keys to the Yard, so this way no one can get into our dorms." If all goes as planned, other dorms will be wired by the end of the year. ---- Now, some of the risks here are obvious: tracking the usage of each key, for example. I am sure RISKS readers are familiar with the implications of that. Worse, the article implies that the police are actively aware of the possibility and may be pursuing it directly. While I have nothing against the Harvard police, I nevertheless don't see this form of surveillance as a good thing. Of course, the fundamental problem is that skeleton keys to all the dorms in Harvard Yard are readily available to anybody who wants one and has some vague idea where to go. This is not a new risk, of course, but I have severe doubts that throwing technology at the problem will make it go away. There must be card-keys somewhere that will open all the locks in question; the maintenance staff needs them. It is only a matter of time before they start circulating just as freely as any other key. I haven't seen any of these card-keys yet myself, but it strikes me as highly unlikely that they are not forgeable, and even more unlikely that (as the article claims) the locks can't be jimmied. And none of this even begins to take into account the risks of failure - power failure, for example, or electronic interference, or any of the other things that electronic devices are subject to in the real world. - David A. Holland dholland@husc.harvard.edu ------------------------------ Date: Wed, 25 Sep 91 17:14 EDT From: Richard_Boylan@vos.stratus.com Subject: bad error handling in Lamborghini Diablo engine management This is excerpted (without permission) from an article in the September 1991 issue of CAR, a British magazine. In the cover story, the writer is driving one of the first Lamborghini Diablo automobiles from the factory back to England: "Then, on the outskirts of Annecy, calamity. The power drops off suddenly, there's a soft, metallic buzz, a muffled bang, and a much louder, rattling clatter. The 'right side engine' warning light comes on. Uh-oh, time to coast over to the hard shoulder. "Tentatively, we raise the engine cover, lean over the wide wings, and peer in. The right-hand exhaust pipe is glowing like the fires of Hades. The aluminium heat shield surrounding it in the bay has melted (aluminium melts at 1000degC), and molten blobs trace a glinting trail of our move across the carriageway. . . . "Swiss Air takes us back to the Diablo a few days later. Factory troubleshooters have diagnosed and fixed the problem. There are two engine-management systems, which each look after a bank of six cylinders. If there's trouble on one side, you're still left with a straight six to get you home. Because a wire had fallen off one of the Lamdba probes for the cat[alytic converter], the right-hand side of our engine was closed down by the chip--hence the power loss. But it seems the fuel wasn't cut off at the same time, and as it reached the exhaust it ignited inside the pipe." The moral of this is that no matter how critical a piece of code is, the correctness of its error-processing paths is even more critical. It's ironic that in an attempt to provide fault-tolerance, the designers of the Diablo engine-management system actually increased risk. If the engine had simply shut down entirely when the first fault occurred, it would have undoubtedly shut down the fuel-delivery system as well. But by attempting to keep the engine running in a degraded mode, they allowed a potentially explosive situation to develop. ------------------------------ Date: Wed, 25 Sep 91 15:33:05 MDT From: ahawks@isis.cs.du.edu (Andy Hawks) Subject: Denver Hacker Hacks NASA The Denver Post, Denver & The West section p. 1 9/25/91 NASA vs. hobbyist Computer whiz accused of illegal access, mischief By. Peter G. Chronis Denver Post staff writer An Aurora computer hobbyist who allegedly used a personal computer and his home phone to penetrate NASA computers hacked off Uncle Sam enough to be indicted on seven federal counts yesterday. Richard G. Wittman, 24, the alleged "hacker," was accused of two felonies, including gaining unauthorized access to NASA computers to alter, damage, or destroy information, and five misdemeanor counts of interfering with the government's operation of the computers. Wittman allegedly got into the NASA system on March 7, June 11, June 19, June 28, July 25, July 30, and Aug. 2, 1990. Bob Pence, FBI chief in Denver, said Wittman used a personal computer in his home and gained access to the NASA systems over telephone lines. The investigation, which took more than a year, concluded that Wittman accessed the NASA computer system and agency computers at the Marshall Space flight Center in Huntsville, Ala., and the Goddard Space Flight Center in Greenbelt, Md. The NASA computers are linked to a system called Telenet, which allows qualified people to access government data bases. A user name and password are required to reach the NASA computers. Federal sources declined to reveal more information because the complex case involves "sensitive material." Wittman, a high-school graduate, apparently hadn't worked in the computer industry and held a series of odd jobs. The felony counts against him each carry a possible five-year prison term and $250,000 fine. [I suppose the Denver authorities locked up his PC to prevent him from using it. They must have used a Denver Boot Load. PGN] [For our out-of-country users, a Denver Boot is a fiendish device that police attach to a wheel to prevent you from driving your car until you have paid up all outstanding fines. Of course, more fines accumulate unless you pay immediately.] ------------------------------ Date: Wed, 25 Sep 91 14:36:43 EDT From: eldred@apollo.com (Eric Eldred) Subject: Re: MSAFP, utilities, and all that Do we really need any more discussion of medical statistics and cost/benefit analysis of tests? Yes, because after all the verbiage here I'm afraid more people are more confused than enlightened. Mark Fulk has pointed out the importance in decision analysis of assessing relevant utilities, especially those of and by the humans affected by the risk. He refers to Kahneman and Tversky (apparently as those who note the subjectivity and often seeming irrationality of individuals' risk assessments and utility analysis). It seems pretty clear now that one cannot discuss a test such as the MSAFP in isolation from utility analysis. Not all physicians, and certainly patients, are yet aware that this is true, however, so it could stand some repeating. The implication of what Mr Fulk notes is also that perhaps a test should not even be done without some counseling and interpretation to those affected, and an entire therapeutic context. For example, if an amniocentesis result predicts a certain disease state of the fetus, would an abortion be done anyway? Too often physicians do tests defensively, because they would be accused of malpractice if they didn't give the "standard" treatment to all. But that is not treating patients as individuals. For example, in a separate discussion with Jeremy Grodberg, I pointed out that utility analysis of a particular vaccine choice should involve more than just the risk of a disease or reaction to the vaccinated individual. As a good example, the US CDC (Center for Disease Control) decided after much debate (part of which was actually filmed and shown on a PBS program) that live polio vaccine should be used instead of killed virus vaccine. The latter is possibly much safer for individuals, and prevents the occasional transmitting of the virus to unprotected others in close contact (some have died, their families sued the govt, and they lost). But the live virus has a possible extra effect in increasing the resistance of the population taken as a whole (and hence the CDC chose it). Thus the risk to the individual is one thing; the risk to the entire population is another. Both factors must be taken into account when issuing a vaccine. It is quite possible, paradoxically, that the risk to an individual could be increased by a choice of one vaccine over another. (Here I'm not going to get into discussion of the risks of the Salk vaccine, which was hastily withdrawn at an earlier time when the manufacturing of it went awry and created false perceptions of its risks.) My argument with the CDC is that they have not yet apparently made it clear that those performing the vaccination should communicate to patients (or parents) that the killed virus vaccine could be safer and would be available if the patient decided for it rather than the live virus vaccine. In other countries, the decisions have been made differently. I believe this is an important point. Those exposed to risks should be able to choose responses most intelligently with full information and should not always have decisions made for them by supposedly more knowledgeable and intelligent engineers, MDs or politicians. Often, with secrecy, the necessary uncertainty of real life, or the fog of war as factors, those decisions prove quite poor ones and are hard to reverse. Generally, even rational people are willing to accept certain risks voluntarily they object to when imposed by a seeming outside force. Many teenage smokers don't put much on the chance of getting lung cancer; 40 years later, they are willing to pay a lot more money than you would predict, just in order to live a little longer, once they do have cancer. We should discuss policy openly. In the interpretation of such tests, it should also be emphasized that--also perhaps paradoxically--the prior probability of events makes a big difference in what to make of the test result. If you redraw Jon Krueger's chart of the four signal/noise possible outcomes --but place numbers in the boxes instead of the yes/no text, and then repeat, varying the incidence of the condition (and thus the numbers in the boxes), you will confirm the basis of the argument against the MMPI. A test that has a high predictive value in a population with a high prevalence of a condition may not be any good at all (less than, say, 50% predictive value of a positive result) should the prevalence be greatly decreased--even if the "accuracy" of the test stays the same. (Thus, I believe pre-employment urine drug tests for programmers are counterproductive.) Each test should be examined experimentally with two critical measures reported: the "specificity" and the "sensitivity", or essentially what lead to what we could call "false positives" and "false negatives". Without those measures reported, and without a prior estimate of the prevalence of a condition in the population tested, it is not really possible to say what to make of a specific test result. Hence, counseling and the wise therapeutic context, by which results can be verified and acted on correctly. The other interesting implication of the discussion here has been the reference to the utility put on threshold values, or on the importance of false positives or false negatives. We should realize that a medical test for a condition that could be fatal but might be prevented, and for which a false positive test result would not lead to needless suffering, anxiety, and so on, could be one with a larger number of false positives (because it is intended as a screen to be sure not to miss anybody with the condition), while one for which there might not be treatment, and for which a positive result might lead to severe consequences (say, MS, or an HIV test before AZT), might be one that one would have to be sure would not have a lot of false positives. Consequently, the threshold values of such tests should be selected so as to magnify the desired results and minimize the undesired consequences. It is quite likely that interpretation of some tests should be withheld until confirmatory results of other tests with different utility values. However, obviously the chances of false results increases with the number of tests, so testing should be done with their limits in mind. It seems to be irrational to mandate reliance solely on such tests as HIV antibodies in arbitrary populations with unknown or low disease incidence, given what we now know about testing. For those who want to look up all this, I'm sorry I don't have the exact references in hand. One book that did initiate a lot of talk on the subject is quite lucid: "Beyond Normality" by Galen and Gambino (I think it was published by Little, Brown, in about 1976). Later work by the Tufts clinical decision analysis group was published in the New England Journal of Medicine in the late '70s and early '80s, introducing the concept of the variability of patient assessment of outcome utility. I think the issues are still important today, since even the experts can make decisions poorly from time to time, and the ones who do make them correctly can't always explain the proper techniques to the rest of us, and so we end up re-arguing the same points. Eric Eldred eldred@apollo.HP.COM ------------------------------ End of RISKS-FORUM Digest 12.40 ************************