RISKS-LIST: RISKS-FORUM Digest Wednesday 9 March 1988 Volume 6 : Issue 40 FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: First Boston faces substantial loss (Dave Curry) Reliance on computers (Bahn) Number plates sans sense (Niels Kristian Jensen via Espen Andersen)[old tale] Re: New Macintosh virus... (David HM Spector) [Psychological Aspects of] Safe Systems (Hugh Davies) Re: Bank ATMs and checking your statements (Paul Fuqua) Re: waning arithmetic skills; erroneous large phone bills (Toby Gottfried) Trusting your calculator (Dan Franklin) Calculator Self Test (was: Disappearing skills) (Mark W. Eichin) Re: Disappearing skills (Bruce Hamilton) Computer Ethics in the curriculum (Rodney Hoffman) Database Correlation (Darin McGrew) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM. For Vol i issue j, FTP SRI.COM, CD STRIPE:, GET RISKS-i.j. Volume summaries in (i, max j) = (1,46),(2,57),(3,92),(4,97),(5,85). ---------------------------------------------------------------------- Date: Wed, 09 Mar 88 14:56:33 EST From: davy@intrepid.ecn.purdue.edu (Dave Curry) Subject: First Boston faces substantial loss From Information Week, March 7, 1988, p. 8: First Boston has confirmed that the company faces a substantial loss due to inaccuracies in its system that inventories mortgage-backed securities. The company won't comment on the cause of the inventory error until an internal audit is completed, but it's expected that losses will range between $10 million and $50 million. ------------------------------ Date: Tue, 8 Mar 88 14:39 MST From: Bahn@HIS-PHOENIX-MULTICS.ARPA Subject: Reliance on computers Here is another entry on the danger of relying on the machine answer. I have a friend who works at Inland Steel. They control many of their furnaces with special digital controllers made in the 60's. They then have these controllers hooked up to pdp-11 computers in a network to manage production and quality control. The 11's are responsible for lot tracking and special orders while the controllers are charged with carrying out the production. During a 2 week down spell one of the annealing furnaces was shut-down for repairs and was allowed to cool. (Usually furnaces are left idling even when not in production) When it was ready to return to production a true cold start was down to the furnace and it's controller. At power up, a power supply burned out in the digital thermometer so that it registered 0F. The controller knowing that it was in a cold start ordered 100% power to the gas tubes. The furnace should reach operating temperature within 1 hr, but with the dead thermometer was registering no increase so the controller just waited for 4 hrs. In the time period the furnace burned itself out doing several hundred thousand dollars damage and will be out of service 6 - 12 months. The amazing thing was that 4 workers were there supervising the cold start and all of them watched this knowing the computer was in control and must know what it was doing even though their experience told them otherwise. I see a few faults here in design. 1) having a single fault failure path in the controller. 2) the controller not having logic to recognize temp rise must occur rapidly in response to firing the burners. 3) blind reliance on technology by skilled workers. 4) no safety mechanisms external to the conttroller to recognize a runaway condition. 5) No computer polling of sensors to test them response (self-test) ------------------------------ Date: Tue, 08 Mar 88 09:08:47 SET From: EDB85007%NOBIVM.BITNET@CUNYVM.CUNY.EDU Subject: Number plates sans sense Having read the discussion on number plate "mismatches", I would like to submit the following. Espen Andersen Compuserve: 71361,3425 EDB85007 at NOBIVM From NUTWORKS (The Electronic Humor Magazine) Issue No 13, Submitted by Niels Kristian Jensen (C83821@NEUVM1.BITNET). The plate. ========= In the U.S. it is possible to buy a 7-character "personalized" license plate describing the car or the owner. To apply, one must list 3 choices in order of preference. If one's first choice has already been assigned to another driver, one gets one's second choice, etc. These facts and a computer system made the following possible: Mr. X wanted a new plate. He got the form and filled in his first two preferences. If these two possibilities had already been assigned to someone else, Mr. X didn't want a personalized plate at all, so for his third choice he wrote "NOPLATE". He got "NOPLATE" as his new personalized plate. At first he got mad, but he mounted it on his car anyway. Within a week he had received offers for over 100 dollars for this ingenius plate, and he began to grow partial to it. In fact, he liked it. His feelings changed by the end of the month, however, because by then Mr. X had recieved nearly a hundred parking tickets. Why? Well, any time a police officer spots a car missing a licence plate, he or she will write "NOPLATE" on the ticket/citation. The computer matched every single one of these with... Mr. X. The computer program couldn't be changed, but the ways of the parking guards could. So don't *EVER* order the custom plate: "NONE". (It would make you Mr. X the second) [This is an old warhorse. I couldn't find it in my archives, but I am sure it has been either in an early SEN or in RISKS. On the other hand, recycling is big in California as well as vanity plates. PGN] ------------------------------ Date: Wed, 9 Mar 88 18:09:54 EST From: David HM Spector Subject: Re: New Macintosh virus... (It may not have filtered all the way through the news system yet, but here's what I wrote to comp.sys.mac and to Chris...) ... There seem now to be (at least) 5 Macintosh viruses on the loose, two are of the "Brandow/MacMag" variety and the rest are based on the sources I described in Risks Digest some time ago. Its gonna be a rough year.. _David From: spector@vx2.GBA.NYU.EDU (David HM Spector) Date: 8-Mar-88 10:34 EST Subject: Re: I've got a virus and I don't like it Organization: New York University It seems you have been bitten by a virus whose sources were uploaded to Compu- serve sevral months ago... The author, a fellow in West Germany, thought it would be educational to distribute these example viruses in source form to encourage people to write defenses against them. His stated intent in writing a virus in the first place was to keep people from running possibly virus ridden program on their production Macintoshes which had been previously hit by viruses.... its signature, in the orignal sources, was a resource type of nVir... its a simple yet potent virus and very easily modified to do bad things. ... unfortunately the only way around most of these viruses is to replace your system folder. (Make sure you do this from a WRITE-LOCKED copy of the Apple System installer... or else you'll end up back where you started, with an infected system.... there is another problems, that being that the virus that was on CompuServe knows how to infect APPLICATIONS, as well as the system itself. Pretty depressing.... David HM Spector, NYU Graduate School of Business AppleLink: D1161 CompuServe: 71260,1410 (212) 285-6080 ------------------------------ Date: 4 Mar 88 04:00:09 PST (Friday) Subject: Re: [Psychological Aspects of] Safe Systems From: "hugh_davies.WGC1RX"@Xerox.COM In RISKS 6.35, Nancy Leveson says; "It describes several serious incidents that occurred because pilots put too much confidence in automatic control systems, even to the extent of rejecting their own external evidence that the system was wrong" I don't know about commercial pilot training, but I do know about PPL (Private Pilots License) training, and I have read about military training, and when learning to fly you are taught to ignore the 'evidence of your own eyes' and to rely on instruments in all but the most ludicrous of circumstances. Obviously this has the risk that the instruments may be wrong, but there are many instances of pilots ignoring the instruments and relying on their senses and subsequently doing stupid (or lethal) things. There is at least one documented case of aircrew flying at night over water, seeing the lights of fishing boats and believing them to be stars, inverting their aircraft, in direct contradiction to what their instrumentation was telling them. There is also a fairly common phenomenon called 'the leans' when the pilot develops a pronounced impression that the aircraft is not level, and rolls it in the opposite direction to correct, even when the artificial horizon and the subsequent handling of the aircraft tell him that the original roll was non-existent. In most cases, the (computer) user is not told to believe absolutely the evidence of a machine over the evidence of his senses. But in the case of aircraft he is explicitly trained to do so. This behooves us (as programmers, etc.) to make sure that the machine is telling the truth! Hugh ------------------------------ Date: Mon, 7 Mar 88 18:56:20 CST From: Paul Fuqua Subject: Re: Bank ATMs and checking your statements [RISKS DIGEST 6.36] I had something similar happen about two years ago. I made a $60 dollar withdrawal, but the machine only gave me $40; my receipt indicated $60. When I went to the office next day to report it, the teller called over a manager type. Turns out they had been puzzling all morning over the discrepancy in their records, which they reconcile daily, and were only too happy to give me my $20. It is important for *both* ends of a transaction to keep records. Paul Fuqua, Texas Instruments Computer Science Center, Dallas, Texas CSNet: pf@csc.ti.com or pf@ti-csl UUCP: {smu, texsun, im4u, rice}!ti-csl!pf ------------------------------ Date: Tue, 8 Mar 88 13:08:30 pst From: decwrl!hplabs!felix!toby@ucbvax.Berkeley.EDU (Toby Gottfried) Subject: Re: waning arithmetic skills; erroneous large phone bills (RISKS-6.35) Organization: FileNet Corp., Costa Mesa, CA Regarding waning arithmetic skills: Like pilots of high-tech planes who need to be able to tell when to observe warning devices and when to override them, people using calculators need to know when to trust the answers. Even perfectly working calculators are subject to input errors, and the consequences of a mistake can range from inconsequential to disastrous. To take a small example, once I was buying several different items at a drug store. The cashier keyed them in, and announced the total as $2.45. This was surprising to me, since the price of one of the items was $3.00 (apparently entered as $0.30). Errors like this must happen frequently, but this was so glaring it should have been noticed. A case appropriate to this time of year: It can be VERY important that adjusted gross income multiplied by tax rate (among other calculations) come out correctly. Regarding the risks of erroneous large phone bills: They occur infrequently, and they are normally fairly easy to correct, but (with even lower probability), one COULD result in phone service being cut off, just when there is a need to make an emergency call (fire, medical, police). Then the cost goes way up, as the reversibility goes way down. Toby Gottfried Costa Mesa, CA FileNet Corp {hplaps,trwrb}!felix!toby ------------------------------ Date: Wed, 9 Mar 88 13:21:35 EST From: Dan Franklin Subject: Trusting your calculator A cousin of mine is a police officer. He is not at all computer-knowledgable, but he understands very well that computers can make mistakes. Why? Because of an exercise his class did when he was training. Each member of the class was given a calculator and told to do some specific arithmetic with it. As it turned out, half the calculators were (intentionally) defective, giving wrong answers--but the class was not told this ahead of time; they had to find it out the hard way. I think this is a great idea. It ought to be mandatory for people in every profession in which non-computer-knowledgable people interact with computers (especially government bureaucrats). If my cousin is any indication, the lesson will last a lifetime. Dan Franklin ------------------------------ Date: Tue, 8 Mar 88 23:29:09 EST From: Mark W. Eichin Subject: Calculator Self Test (was: Disappearing skills) >Maybe calculators should have some sort of "self-test" program built in >which would be automatically invoked when the unit is powered up? >Al Stangenberger Dept. of Forestry & Resource Mgt. HP-10 through 16 calculators have full self test of display (light up all segments) and keyboard (you have to push all of the keys in an obvious order). Interesting that I have never had either type of failure. Mark Eichin [I interpreted what was requested was a full self-test that includes computational checks of correctness! PGN] True, I haven't seen that... but the original article mentioned an error due to a calculator with digits failing. I only wanted to point out that it there ARE groups (HP) that don't ignore the problem entirely. The HP-41c automatically sanity checks the memory... I have not heard of any that actually check the numbers they generate, though documentation of what the self tests do is often poor. _Mark_ ------------------------------ Date: 8 Mar 88 18:39:09 PST (Tuesday) From: "Bruce_Hamilton.OsbuSouth"@Xerox.COM Subject: Re: Disappearing skills Can anyone point me to some good references on closed-cycle or partly-open-cycle ecologies & economies? I'm particularly interested in such questions as the minimal population and skill sets needed to construct something close to current US middle-class lifestyles. Work on self-replicating factories might be of peripheral interest. Earth and space are both interesting, and probably have very different answers. "Living off the land" as a 19th-century trapper or 20th-century bag lady is not nearly as interesting as the more general problem of bootstrapping an economy. Now perhaps I'm getting into third-world development issues. Whoops, I'd better tie this into RISKS. As the parts lose their grasp of the whole (shared knowledge and values) and people become more specialized, I suspect that more and more types of catastrophic breakdowns of civilization are possible. Mark Vonder Haar's comment about lost knowledge of farming is right on. Let us further consider that the agricultural 5% of the US don't "feed themselves" -- they take out big loans, buy big machines, fuel, fertilizer, COMPUTERS, and pesticides, and run specialized agri-factories. --Bruce ------------------------------ Date: 9 Mar 88 10:51:59 PST (Wednesday) From: Rodney Hoffman Subject: Computer Ethics in the curriculum 'The Chronicle of Higher Education' for Feb. 24, 1988, page A15, carries a lengthy article by Thomas J. DeLoughry with the headline FAILURE OF COLLEGES TO TEACH COMPUTER ETHICS IS CALLED OVERSIGHT WITH POTENTIALLY CATASTROPHIC CONSEQUENCES Scanty attention to the topic seen as part of larger problem: Students know little about their profession The article includes quotes from RISKS moderator Peter Neumann and from other RISKS contributors and names known to RISKS readers. It includes discussions of programmer certification, computer science department accreditation, lack of research and administrative support for work in computer ethics, multi-disciplinary approaches, differences from other engineering disciplines, and details of the issue at MIT and Stanford. ------------------------------ Date: Mon, 7 Mar 88 13:41:11 PST From: ibmuupa!mcgrew@ucbvax.Berkeley.EDU (Darin McGrew) Subject: Database Correlation (Re: RISKS-6.36) Organization: IBM ACIS, PALO ALTO In RISKS 6.36, Matt Fichtenbaum (mlf@genrad.com) mentioned a cross correlation between the databases of New York State's drivers' licenses and recipients of aid to the blind. The RISK is what is done with such a cross correlation, not that it was done at all. California allows drivers with clean driving records to renew their licenses by mail. I know one elderly individual (and have read of two others) who continued to renew their licenses after attaining legal blindness, for identification purposes. The RISK is that people (eg, "The Bureaucracy") might automatically prosecute such people without finding out what's really going on. Darin (I speak for myself, not for my employer.) ------------------------------ End of RISKS-FORUM Digest ************************