2-Oct-86 21:30:48-PDT,11423;000000000000 Mail-From: NEUMANN created at 2-Oct-86 21:29:00 Date: Thu 2 Oct 86 21:29:00-PDT From: RISKS FORUM (Peter G. Neumann -- Coordinator) Subject: RISKS-3.73 DIGEST Sender: NEUMANN@CSL.SRI.COM To: RISKS-LIST@CSL.SRI.COM RISKS-LIST: RISKS-FORUM Digest, Thursday, 2 October 1986 Volume 3 : Issue 73 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Lessons from Viking Lander software (Bob Estell) Software wears out? (Rob Austein) Wrongful eviction through computer error (Bill Janssen) Deliberate override (Herb Lin, Ray Chen) Re: Piper Arrow Gear Override (Douglas Adams) Undesirable breakins and causes (Ian Davis) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@CSL.SRI.COM, Requests to RISKS-Request@CSL.SRI.COM) (Back issues Vol i Issue j available in CSL.SRI.COM:RISKS-i.j. Summary Contents in MAXj for each i; Vol 1: RISKS-1.46; Vol 2: RISKS-2.57.) ---------------------------------------------------------------------- Date: 2 Oct 86 12:18:00 PST From: "ESTELL ROBERT G" Subject: Lessons from Viking Lander software To: "risks" Angry replies? Never! To the contrary, Thank you, Nancy. In the last few weeks in the "pages" of this journal, we have established: 1. Software cannot be perfect, because it is designed by fallible humans. 2. Hardware cannot be absolutely predictable, adding to the software woes. 3. Very concise programs, though not "perfect" nevertheless have sometimes run well enough to be considered "successful" - even on a "first chance." [Other than during simulated tests.] It's time to move forward. I concur heartily with Nancy's point that what applies to small programs may not scale up to large programs. It's worth noting that in her last message, Nancy alluded to the Viking Lander as a "small" program, with "only" 18,000 words in assembly! How many of you agree that we might use orders of magnitude to distinguish sizes? e.g., IF xx,000 = small, then x,000 = tiny, and x00 = toy; and xxx,000 = large, and x,000,000 = very large; etc. Maybe it then follows that xx,000,000 is "possible?" [SDI size?] Maybe not. Depends a LOT on HOW it's designed, coded, and tested. I'd like those more knowledgable than I to address some of the following; maybe the software engineering journal is a better forum; maybe conclusions could be reprinted in RISKS. a. What are the measures of "acceptability?" [Analogy: In baseball, the 1.000 batting average [perfection] is never possible - except in trivial circumstances, then only with luck. A century of experience says that .3xx is outstanding. Likewise, the 1.000 fielding average is "impossible." But anything less than .9xx is terrible. What are similar failure rates for software? Where should we set our expectations? Where is the "point of diminishing returns?" I'm not at all clear on just HOW to set these expectations. [On a saturated UNIVAC 1110 a few years ago, we were suffering up to 3 crashes a day; we "engineered" that down to less than 3 per month. Did that improvement make us just average, or much better?] b. *IF* it's true that "small" programs can run "acceptably" albeit NOT perfectly, and that "very large" programs probably cannot (?), then why don't we get serious about building very large programs as orderly structures of small modules? (Where "small" may mean xx,000 lines.) At the moment, it seems to me that we're caught on the horns of a dilemma of our own making; the "idealists" among us are saying that very large systems cannot be perfect, hence should not be pursued; the "realists" among us are saying that the present status of large, real-time systems is a disaster; the "analysts" among us are saying that there seems to be no good formula for success, yet; and the "pragmatists" among us are saying that we can make SOME worthwhile improvements in the status quo, and thus we should. ALL FOUR VIEWPOINTS ARE "CORRECT." Isn't it time now for all of us to start "groping" forward together? Bob ------------------------------ Date: Thu, 2 Oct 1986 00:47 EDT From: Rob Austein To: risks@CSL.SRI.COM Subject: Software wears out? The other sense in which software "wears out" is that people lose their ability to maintain it. I recently had to work on a mailer daemon that is about 15 years old. Fine code, possibly one of the best mailers ever written (certainly for its day), coded according to good programming practices -- for the early 1970s. I almost went nuts trying to modify it, people just don't think that way anymore (you know, labels every ten instructions, GOTOs everywhere...). I was the person modifying the code because everybody else had better sense than to even try. At this point I can say with a fair amount of certainty that -nobody- really understands that program anymore, although the people who installed the various features (if still alive) usually remember having done so within a month or so of being asked. And this is a fairly small program compared to stuff done out in the Real World. --Rob ------------------------------ Date: Thu, 2 Oct 86 19:10:10 CDT From: Bill Janssen Subject: Wrongful eviction through computer error To: risks@sri-csl.arpa An interesting thing happened to me last month. I got home on the 5th of September to find an eviction notice on my living room floor. Something about not paying my rent. Well, I gathered up the checks and went over to the office. Turns out the problem was that I had already paid for October, as well as September, and the apartment management folks had just switched to a new computer system! There must have been a line in it something like if (last_month_paid_for != this_month AND day == trigger_day_for_eviction) issue_eviction_notice(); According to some of the office staff, 11 other people had already been in with similar complaints. Bill Janssen, MCC Software Technology, 9430 Research Blvd, Austin, Texas 78759 UUCP: {ihnp4,seismo,harvard,gatech,pyramid}!ut-sally!im4u!milano!janssen ------------------------------ Date: Thu, 2 Oct 1986 08:26 EDT From: LIN@XX.LCS.MIT.EDU To: George Adams Cc: Risks@CSL.SRI.COM Subject: Deliberate override From: George Adams Even if an incompetent driver forgot to enable the system on hard pavement, performance would be no worse than now common. Without the switch a competent driver might hit that cow instead of stopping in time. But you are assuming that the average level of competence remains the same in the presence of the automatic system. For tasks on which you can enforce some high level of performance (such as flying), this may be valid. But you can't do it for drivers. It is entirely possible that drivers will come to RELY on the automated system, perhaps even without knowing it; their abilities will decline, but they will still feel capable of overriding the system. That's a recipe for disaster. I'm not arguing that overrides should be forbidden. I'm saying that there's a trade-off that has to be addressed before they are installed. ------------------------------ Date: Wed, 1 Oct 86 02:37:57 EDT From: Ray Chen To: RISKS@CSL.SRI.COM Subject: Deliberate Overrides Organization: The Clouds Project, School of ICS, Georgia Tech Manual overrides are a nice idea, but chances are they'll be needed most during a sudden emergency when there isn't time to think about, much less trigger any kind of safety override. How would you like to have to trigger a safety override while powering into a corner trying to avoid an accident? Ugh. Personally, I don't see any way around it. Total control after all is just that -- the ability to specify exactly what the machine is going to do, even if it's beyond the normal performance envelope. Saftey restrictions on the other hand are designed to keep you from exceeding the performance envelope. There's an inherent contradiction in the two objectives which can't be neatly resolved unless the safety system and the user/operator are always in perfect agreement on the limits of the performance envelope in every possible situation. I think we have a trade-off here. Ray Chen uucp: ...!{akgua,decvax,hplabs,ihnp4,linus,seismo}!gatech!chen CSNet: chen@GATech ARPA: chen%GATech.CSNET@CSNet-Relay.ARPA ------------------------------ Date: Thu, 2 Oct 86 08:59:20 PDT From: crash!pnet01!adamsd@nosc.ARPA (Adams Douglas) To: risks@sri-csl Subject: Re: Piper Arrow Gear Override Piper could also have installed the override system because some old lawsuit or other related to a gear-up landing would have caused their insurance rates to go through the roof if they didn't implement some kind of 'fix' that they could point to. Incidentally, I don't advocate it, the problem of leaving the override on could be solved by having a flashing panel light next to the other two gear-status lights on the glareshield such as OVERRIDE ENGAGED. But that would simply add to the pilot's cockpit stimuli--which is never a good idea during takeoff. ------------------------------ Date: Thu, 2 Oct 86 17:32:33 edt From: Ian Davis To: RISKS@CSL.SRI.COM Subject: Undesirable breakins and causes Organization: U. of Waterloo, Ontario Has anyone suggested that hardware can also seriously undermine security? [YES] I tend to work from home and thus communicate via a modem, and have always logged off by merely switching from data to voice on my modem, which both drops the line, and hangs up the phone for me... unfortunately (and initially unbeknown to me) at least one of the modems that answers incoming calls from me (at a deliberately unspecified site) was hit by a current surge during a recent lightning storm, and now no longer drops the communication line to the CPU when I drop my line to it. This has disasterous consequences since the next caller to use this recieving modem finds themselves logged into my account with totally unrestricted access to the system. Fortunately most users are honest and promptly sign off, but the risks are very real. The moral for those of you who are concerned, is that one should always log off from an operating system before dropping communication lines, and that one should log back on as soon as possible if the line is dropped accidentally. Ian Davis [We have been around that one several times now, although lightning hitting the modem is a new wrinkle! PGN] ------------------------------ End of RISKS-FORUM Digest ************************ -------