RISKS-LIST: RISKS-FORUM Digest Thursday 15 March 1990 Volume 9 : Issue 75 FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: PRODIGY updating programs (Simson L. Garfinkel) Who shall guard the guards? (Robert A. Levene) Journalistic hacking (Rodney Hoffman) Caller-id by name (Gary T. Marx) Re: PSU Hackers thwarted (David C Lawrence) Re: Tracking criminals and the DRUG police-action (Brinton Cooper) RISKS of "Evolutionary Software" (Rajnish and Gene Spafford via Will Martin) Human-centered automation (Donald A Norman) Re: Airbus Crash: Reports from the Indian Press (Henry Spencer) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, and nonrepetitious. Diversity is welcome. CONTRIBUTIONS to RISKS@CSL.SRI.COM, with relevant, substantive "Subject:" line (otherwise they may be ignored). REQUESTS to RISKS-Request@CSL.SRI.COM. TO FTP VOL i ISSUE j: ftp CRVAX.sri.comlogin anonymousAnyNonNullPW cd sys$user2:[risks]get risks-i.j . Vol summaries now in risks-i.0 (j=0) ---------------------------------------------------------------------- Date: 12 Mar 90 20:44:07 EST (Mon) From: simsong@prose.CAMBRIDGE.MA.US (Simson L. Garfinkel) Subject: PRODIGY updating programs I must take issue with Eric Roskos saying that PRODIGY can only update information in the STAGE.DAT file. In doing my article on PRODIGY for The Christian Science Monitor, I was told by Prodigy's manager of software services that one of the really nifty tricks of PRODIGY is that nearly the entire system running on the PC --- including the .EXE files --- can be updated remotely. This eliminates the need to send out floppy disks with updates. (They didn't have it working well at the beginning and actually had to send out one update --- an extremely expensive proposition.) The reason for wanting to do this is based on two things: prodigy's pricing structure and its target market. Prodigy charges one $9.95 a month. At that price, it just isn't possibly to economically send out floppy disks. Especially if they want to have 1-5 million subscribers within the next 2-10 years. The other thing is their target market: they want people who don't know anything about programs or files. Automatic updates eliminate the necessity of having to have users put disks into their computers and try to figure out what is going on. The Prodigy censors is a very real problem. They have recently shut down PRODIGY groups that have ventured into "unacceptable" topics like abortion and homosexuality. ------------------------------ Date: Wed, 14 Mar 90 13:00:49 EST From: levene@aplcomm.jhuapl.edu (Robert A. Levene) Subject: Who shall guard the guards? (was: Drive-by-wire cars) In RISKS 9.74 Craig Leres writes: > ... Hopefully, auto manufacturers will be as conservative with drive by wire > systems as they have been with the computer controlled engines they are > currently building. For example, the engine in my '89 GM car has a computer > that controls functions such as fuel delivery and ignition. But nearly all > the computer controlled systems have backups that implement the "limp home > mode." ... Don't let an acute failure mode lull you into a false sense of security. My 8500-mile '89 GM car is in a dealer's repair shop due to computer failure. If the emissions computer *fails*, the car will "limp home." But if an erratic computer misinterprets the car's state, it will send faulty control signals and cause unpredictable performance. "Hey, HAL - Can you say 'Garbage In, Garbage Out?'" For two weeks, the computer failed intermittently, occasionally stalling the car without warning - not a pleasant experience, especially when it stalled while going 55mph on a 10-lane highway, and at a busy intersection while making a turn. After several such failures and three tows, the car's computer finally failed for the mechanics, giving them the required justification to replace the computer. The computer already has the capability to detect faulty sensors throughout the engine. A second, independent computer is needed to monitor the performance of the engine computer (i.e., "guard the guards") in order to detect and record intermittent failures. At minimum, the car should also have a manual override switch to enter "limp home" mode instead of the $45 "tow home" mode. Rob Levene ------------------------------ Date: 11 Mar 90 18:24:20 PST (Sunday) From: Rodney Hoffman Subject: Journalistic hacking Summarized from a story by Sheryl Stolberg in the 'Los Angeles Times' 10 March 1990: Fox Television employees in New York and Los Angeles discovered in February that someone had been trying to gain access to their computers, using the same password in both cases. Free-lance journalist Stuart Goldman was arraigned Friday and charged with violating federal and California anti-hacking laws. His personal computer and floppy disks were confiscated. According to the federal prosecutor's affidavit, Goldman made several attempts -- at least one of them successful -- to gain entry to "sensitive data files regarding ... news stories worked upon by the company's journalists." Network officials would not disclose what information was sought, but Goldman had worked briefly for the Fox-produced news tabloid 'A Current Affair', and he had recently been trying to sell an inside story on such shows to the 'Los Angeles Times'. [See 'Hacking for a competitive edge' in RISKS 8.71 for an earlier case of journalists apparently trying to steal stories via computer hacking.] ------------------------------ Date: Tue, 13 Mar 90 14:48:59 EST From: gtmarx@ATHENA.MIT.EDU Subject: Caller-id by name Newsday, February 27, 1990 Don't Give Up Your Privacy to Find Out Who's Calling Gary T. Marx The telephone is something that is usually answered, but rarely questioned. But this is changing with the proposal of the regional phone companies to introduce unblockable Caller ID. Almost everyone answers "yes" to the question "Would you like to know who is calling you before you pick up the phone?" But most people would answer "no" to the corollary question, "Would you mind if every time you made a call your phone number was automatically revealed to the person called?" This offering, for which the phone company would charge a separate fee, changes the nature of phone service by removing the control that callers have over their phone numbers. In other words, the service has consequences for the person calling -- unlike other recent developments in phone service, such as speed dialing or automatic redialing. By technological fiat, the phone company takes personal information away from the caller and sells it to the person called. This is similar to the data scavenging companies that sell credit and related information about individuals without their consent. Contrary to what most people believe, the phone companies are saying they, not you, control your phone number. Unblockable Caller ID is unlikely to become the standard in the United States. California already has a law requiring that if the service is offered, it should come with a free blocking option -- callers not wanting to reveal their number can enter three digits and the called party will see a "P" for private call. Similar federal legislation has been introduced in Congress by Senator Herbert H. Kohl (D-Wisc). That is the compromise position. Don't ban the service. Don't offer it in an unrestricted way as most phone companies propose. Give callers limited control over what is revealed - -their number, or the fact that they don't want to reveal it. While such a position is better than an unrestricted offering, it is far from ideal. Callers not wanting to reveal their number to the person called run the risk of not getting through. People receiving calls, seeing a "P" may decide that if you won't identify yourself, they won't talk to you. That is a reasonable response on the part of the called person seeking to avoid unwanted calls. The caller appears suspect - -even though in most cases what callers wish to keep to themselves is their phone number and perhaps location, not their name. The problem is that now, as proposed, the only form of identification the technology delivers is the phone number. If it were changed so that callers had the option of delivering their names, most privacy problems would be resolved. In general it would also be more useful to the people called to see a name rather than a phone number. They needn't run the risk of refusing a call from an unrecognized phone number that might in fact be a family member calling from a service station to report the car has broken down, or a surprise visit from a out-of-town friend. This is also normal phone etiquette, which begins with callers identifying themselves by name, not by telephone number. One would hope that the phone companies, as publicly regulated monopolies would feel an obligation to develop technical innovations that, beyond enhancing profits, would further important social values such as privacy and equity. At a minimum, their actions should not diminish these as unblockable Caller ID will. Investors, systems designers, and marketers need to consider how a development might be misused, or have undesirable social consequences. Services like Caller ID should be developed in consultation with citizen advisory groups. Consumers should not be put in the defensive position of having to respond to whatever radical changes the local phone service proposes. We live in a democracy, not a technocracy. As networks become more important and as invasive technologies more powerful, public utility commissions which traditionally have focused on the economic aspects must look to broader social aspects. Too often, communications technology is seen only as something that erodes, rather than enhances, privacy. But that need not be the case. Giving callers the option of providing their name would serve the interests of both the caller and the person called. ------------------------------ Date: Tue, 13 Mar 90 10:12:38 EST From: tale@turing.cs.rpi.edu (David C Lawrence) Subject: Re: PSU Hackers thwarted (Angela Marie Thomas, RISKS-9.74) Just a couple of comments on this story. These aren't criticisms but just perspective observations; as professionals in all areas of life discover there are frequently some differences of opinion about how matters of their field should be presented to the general populace. Recognising that these differences spring up even between members of the field, these are my opinions. According to records, Clark is accused of running up more than $1000 in his use of the computer account. Geyer is accused of running up more than $800 of computer time. As a user of several systems that use pseudo-monetary accounting schemes, I question whether any resources were really wasted at all, "computer time"-wise. I do not know much about the systems in question, but if they parallel those that I have had these experiences with then there were cycles to spare. To me, just handing out numbers with dollar signs attached seems to be attempting to (either knowlingly or not; I do not know the author's experience, either) garner a response of, "So much money! The waste! Clearly a terrible degree of theft!" Among the systems accessed was Internet, a series of computers hooked to computer systems in industry, education and the military, according to records. Everytime I see comment about the Internet like this it just makes me ponder what the general public thinks. Is it, "OH! Some terribly important network! Our national security might have been breached!"? I do not mock the importance of the Internet. The reality of the Internet, though, is that access to it is something that many people can get quite easily through their school or work and the fact that the Internet was "Among the systems accessed" isn't a very shocking thing. Then again, perhaps it is shocking based on the system he was coming from. My comment, however, is based on what general reaction to the above statement could be like -- the public won't know about the specifics of that system either. Matt Crawford, technical contact in the University of Chicago computer department discovered someone had been using a computer account from Penn State to access the University of Chicago computer system. Not to detract from Matt's work, but this paragraph essentially says nothing. One of the reasons that these networks exist is so that people can do work at a machine when they are half-way around the world from it. There is nothing especially surprising about someone from one university accessing a computer at another university. This is lacking in information and only makes me wonder what we are supposed to infer from it. Dave ------------------------------ Date: Tue, 13 Mar 90 9:02:04 EST From: Brinton Cooper Subject: Re: Tracking criminals and the DRUG police-action (RISKS-9.74) In Risks 9.74, J. Eric Townsend writes, in part: > "forecast crimes" -- could they have predicted the hit and run driver > who totaled my car and didn't even stop to check if my passenger and > I were injured? Maybe they should try predicting crimes by politicians > and federate [sic] employes [sic] first, just to get the bugs out of > the system.... I presume he meant "federal employees". In any case, his statement is an unwarranted defamation of the character of literally millions of public employees who work honestly, cheerfully, and carefully to give the taxpayer's an honest day's work for a fraction of an honest day's pay. As an academic, Mr (Dr?) Townsend should be a little more objective in his characterization of folks who work in other domains. _Brint ------------------------------ Date: Fri, 9 Mar 90 8:41:35 CST From: Will Martin Subject: RISKS of "Evolutionary Software" (Rajnish, Spafford) The following is extracted from the latest issue of the Computers and Society Digest. Please note the comments about this software "growing" to become so complex that it can no longer be understood by its creators. As soon as I read that, I thought of "Risks"! Regards, Will Martin ***Begin Extract*** The Computers and Society Digest, Volume 4, #9 Thursday, March 8th 1990 From: "rajnish" Date: Wed, 28 Feb 90 10:49:32 PST Subject: Evolutionary Software? I was wondering what people thought of the Darwinian software being created by the computer scientists at the University of California in Los Angeles? Their approach is pretty radical and they're creating more powerful and reliable software through self-evolution than their programmers can design by hand. I guess these small modular programs they have going interact and merge with each other to create new generations that can anticipate potential pitfalls that human programmers can't. Just like we humans think about so many things at the same time in our head, the computer runs thousands of programs simultaneously and a master program picks out the ones that suits its' needs most efficiently, integrating it to produce following generations that are even more powerful. Survival of the fittest? In Alameda and Orange Counties in California, an example of their Darwinian programming is helping the county to control their mosquito population. Each of the individual program modules are able to successfully mimic the behavior of the mosquitoes to determine growth rate, etc., to find out precisely where and how much insecticide is necessary to kill itself and its' children. Instead of the previous mass insecticide bombings at 20000 sites picked out by human programmers, this software is doing a near perfect job with only 3000 sites it picked out on its' own. Pretty impressive, you think? The computer scientist who developed this approach to software design, Danny Hillis (Founder of Thinking Machines in Cambridge, Mass.) thinks because of its constant evolution that his software is eventually going to make itself so complex that even their designers won't be able to comprehend all of its' functions. Kinda like becoming God? This guy even has software working like a biological parasite to wipe out incompetent programs and therefore forcing the master program to search for programs that are even better! This parasite even looks around for viruses to kill. Instead of taking the usual route, trying to simulate human qualities like vision and speech, Hallis' artificial intelligence is just trying to mimic unexpected behavior that all organisms exhibit and using a parallel supercomputer to accelerate the natural evolutionary process as defined by Darwin. Interesting, you think? Rajnish - - - - - - - - - - - - - - - - - - Date: 3 Mar 90 23:30:30 GMT From: spaf@cs.purdue.edu (Gene Spafford) Subject: Re: Evolutionary Software? Danny Hillis presented his work at the 2nd Conference on Artificial Life, held in Santa Fe, the week of Feb. 4. Lots of other interesting ideas were presented, too. The proceedings of the 1st conference have been published by Addison-Wesley. The second set of proceedings will be published next year, also by Addison-Wesley. You can get more information about the conference by contacting Chris Langton @ the Santa Fe Institute for Non-linear Studies, (505) 667-1444. (I was there, talking about computer viruses as a form of artificial life.) Gene Spafford ***End extract*** ------------------------------ Date: Thu, 15 Mar 90 08:10:33 PST From: danorman@UCSD.EDU (Donald A Norman-UCSD Cog Sci Dept) Subject: Human-centered automation RISKS 9.74 had a statement about the NASA human-centered aviation safety project. I don't know if I am in that project, but 1. Ed Hutchins and I have a grant with NASA-Ames on aviation safety. 2. I have strongly argued for user-centered system design (UCSD) in general. 3. We are working on the checklist problem and on the automation problem. So a quick summary of our work might be appropriate: it certainly fits the domain covered by RISKS. Automation. I have been concerned with the fact that too many automatic devices are built not only to take over the jobs performed by humans, but with no understanding of the issues that will arise when they fail. A simple example is the Air China incident in which the number 4 engine lost power, and the autopilot compensated without notifying anyone. If the 1st officer had been flying instead of the autopilot, he might have said "hmm, I seem to be compensating more and more. Wonder what's happening?" But the autopilot was silent, and when the problem finally exceeded the control authority of the autopilot, the result was an uncontrolled aircraft that almost was a total disaster. I have analyzed this an other aviation incidents in a tech report that is available upon request. Abstract at the end of this note. Checklists. In our opinion, a checklist is an admission that humans fail. After all, if we didn't we wouldn't need to check. Therefore, appropriate design of equipment and procedures can probably eliminate or at least reduce the need for checklists. We don't need a checklist to ensure that we open the door before passing through it -- unless it is a glass door. We used to try to start our autos without first inserting the key in the ignition switch, but now that the starter key and the ignition key are the same, we no longer make that error. Wiener and Asani point out that pilots sometimes take off without lowering their flaps (so there is a warning buzzer and it is a checklist item), but they never land without lowering flaps, so this condition need not be checked for. In similar fashion, the takeoff checklist has all sorts of items on it, but NOT to advance throttles. Yes it seems obvious, but that is just the point. Except for a study by Wiener and Asani that is just now being completed (for NASA-Ames) there have been NO systematic analyses of the scientific/cognitive bases of checklists. They are now constructed by a combination of the intuitions of chief pilots, experience, and the concerns of the legal staff. Lists for the same aircraft in different airlines vary dramatically. This is a real safety hazard: read the NTSB reports on the Delta Dallas crash and the Northwestern Detroit crash. Hutchins and I are doing a cognitive analysis of checklists. We will have a paper "any month now." Automated checklists can help, but 1. They are not the final answer. 2. Locating them on a front-panel CRT is probably the wrong way to go. 3. They have to be designed with an understanding of human cognition. 4. Checklists, procedures, and do-lists should probably be combined. === Abstract of tech report The "Problem" of Automation: Inappropriate Feedback and Interaction, Not "Over-Automation" As automation increasingly takes its place in industry, especially high-risk industry, it is often blamed for causing harm and increasing the chance of human error when failures occur. I propose that the problem is not the presence of automation, but rather its inappropriate design. The problem is that the operations under normal operating conditions are performed appropriately, but there is inadequate feedback and interaction with the humans who must control the overall conduct of the task. When the situations exceed the capabilities of the automatic equipment, then the inadequate feedback leads to difficulties for the human controllers. The problem, I suggest, is that the automation is at an intermediate level of intelligence, powerful enough to take over control that used to be done by people, but not powerful enough to handle all abnormalities. Moreover, its level of intelligence is insufficient to provide the continual, appropriate feedback that occurs naturally among human operators. This is the source of the current difficulties. To solve this problem, the automation should either be made less intelligent or more so, but the current level is quite inappropriate. The overall message is that it is possible to reduce error through appropriate design considerations. Appropriate design should assume the existence of error, it should continually provide feedback, it should continually interact with operators in an effective manner, and it should allow for the worst of situations. What is needed is a soft, compliant technology, not a rigid, formal one. Norman, D. A. (1990). The "problem" of automation: Inappropriate feedback and interaction, not "over-automation". Philosophical Transactions of the Royal Society of London, B (Paper prepared for the Discussion Meeting, "Human Factors in High-Risk Situations," The Royal Society (Great Britain), June 28 & 29, 1989.) Don Norman, Department of Cognitive Science D-015, University of California, San Diego, La Jolla, California 92093 USA ------------------------------ Date: Thu, 15 Mar 90 14:52:31 EST From: henry@zoo.toronto.edu Subject: Re: Airbus Crash: Reports from the Indian Press > A technical committee armed with comprehensive terms of reference > began a probe into the whole Airbus affair last week. ... Interestingly enough, it looks like somebody in authority at least suspected that the results would be embarrassing to the airline (i.e. mismaintenance or pilot error rather than technical problems). Normally, in such an accident investigation, the airworthiness authorities of the aircraft's country of origin -- i.e., the people who first certified the thing as flyable -- are involved, and the manufacturer is at least kept informed. Aviation Week reports that India refused European airworthiness authorities' request to participate, and also refused information requests from them and from Airbus Industrie. Henry Spencer at U of Toronto Zoology uunet!attcan!utzoo!henry henry@zoo.toronto.edu ------------------------------ End of RISKS-FORUM Digest 9.75 ************************