14-Apr-86 23:07:17-PST,21304;000000000000 Mail-From: NEUMANN created at 14-Apr-86 23:05:42 Date: Mon 14 Apr 86 23:05:42-PST From: RISKS FORUM (Peter G. Neumann, Coordinator) Subject: RISKS-2.42 Sender: NEUMANN@SRI-CSL.ARPA To: RISKS-LIST@SRI-CSL.ARPA RISKS-LIST: RISKS-FORUM Digest, Monday, 14 Apr 1986 Volume 2 : Issue 42 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS ACM Committee on Computers and Public Policy, Peter G. Neumann, moderator Contents: Robot safety (Ron Cain via Bill Park) Use of computer files as evidence (Rob Horn) Review of *Softwar* (Gary Chapman) Computerized Voting -- No Standards and a Lot of Questions (Summary of Eva Waskell's talk by Ron Newman) The RISKS Forum is moderated. Contributions should be relevant, sound, in good taste, objective, coherent, concise, nonrepetitious. Diversity is welcome. (Contributions to RISKS@SRI-CSL.ARPA, Requests to RISKS-Request@SRI-CSL.ARPA.) (Back issues Vol i Issue j stored in SRI-CSL:RISKS-i.j. Vol 1: MAXj=45) ---------------------------------------------------------------------- Date: Mon 14 Apr 86 13:22:55-PST From: Bill Park Subject: [Ron Cain : robot safety] To: Neumann@SRI-AI.ARPA Mail-From: CAIN created at 14-Apr-86 09:19:46 Date: Mon 14 Apr 86 09:19:46-PST From: Ron Cain Subject: robot safety To: IA.STAFF: ; For those who hadn't heard, I thought I'd mention two close calls we had out in the welding lab a week or so ago. It is worth keeping them in mind the next time you stand near a robot. In the first incident, a 68000 board in our system failed and caused the processor to jump to (of all places) a robot move routine. We were all standing around the emergency stop button looking at a terminal, and Jeff and Talia got to the button within a few milliseconds of hearing the crunching noise which marked the premature demise of a small jack belonging to the lab. With our sensor mounted on the end-effector as it was, it could have been alot worse if we had been further from a kill button. The second incident was even more sobering. Some drive motor cards in the Cincinati-Milacon box failed and joints 5 and 6 began jerking around randomly. Again, the kill button was nearby, and a potentially disastrous situation (at least for the sensor) was avoided. It could have been any other joint -- including the base or the shoulder. And someone could have been standing next to it. We do all the time. The point is just this: it can and does happen. Watch yerselves around robots. ... ron ------------------------------ Date: Mon, 14 Apr 86 13:28:33 est From: decvax!wanginst!infinet!rhorn@ucbvax.berkeley.edu (Rob Horn) To: wanginst!sri-csl.arpa!RISKS Subject: Re: Use of computer files as evidence (RISKS-2.39) The use of computerized data as evidence has been treated carefully in the environmental field (a litigious arena which includes acid rain, toxic wastes, etc.). The basic rule is: Computer-based data is NOT evidence unless ALL parties involved agree to treat it as evidence. Yet, almost all of the data acquisition and processing is performed by computers. The route around this that is used by the legal process is a dual PAPER or (only recently) MICROFILM evidence trail. Using this trail the following must be shown: 1). All instrumentation calibrations are traceable to NBS standards, with logs that are properly documented and signed by humans, in non-erasable ink on paper. (Also on numbered sheets in bound notebooks only, with countersigned dates and occasion Q/A checks). 2). The computer processing includes the processing of routine calibration so that the computer is part of the calibration loop. 3). All reports are provided in both computerized and hardcopy form. The hardcopy version is certified and signed by a Q/C person. 4). All equipment logs and records are duly signed and archived. In fact, the computer records are generally trusted and used, but all significant evidence is verified against the paper trail. This does not prevent tampering, but it does introduce several levels of human verification and record keeping on top of the computer. The legal system is comfortable with its ability to deal with human error and dishonesty, so they switch to the human trail when in doubt. These rules posed quite a problem in automating some of the data acquisition processes, because the people involved would NOT SIGN reports that they could not verify. (They had significant personal liability). Most of the reports had to be generated on the spot (so that the signer could verify that the equipment was behaving correctly), and include a hardcopy printout that showed all of the equations and intermediate computations used (so that the signer could double check whenever the numbers looked unusual or the value looked like it might have legal significance). Then from these individual data items computerized reports could be generated, but again the signers of those reports insisted on hardcopy for intermediate terms and double checked all the suspicious or signficant numbers. Did mistakes get through? Probably. But the error levels were low and bad reports had a decent chance of being corrected. Disputed reports could be re-created by hand from "raw" data if necessary. The "raw" data being computerized instrumentation reports that were paper logged and signed. Was the computerization complete? Definitely not. The people involved refused to sign reports from a program where they were unable to perform independent validation on a spot check basis, nor where they could not find a totally hardcopy re-creation path. My experience in this is now four years old, but this area changes slowly and the rules are probably still the same. The people involved are very unwilling to abandon their independent audit path. They were only willing to trust computers for the general case, not the oddball or legally significant items. For things like averages, etc. they were willing to trust computers after verifying 5% (selected at random) by hand. Rob Horn UUCP: ...{decvax, seismo!harvard}!wanginst!infinet!rhorn Snail: Infinet, 40 High St., North Andover, MA ------------------------------ Date: Fri, 11 Apr 86 09:19:00 pst From: Gary Chapman Subject: Review of *Softwar* To: glacier!RISKS@SRI-CSL.ARPA I thought participants of Risks might be interested in a recently released book called *Softwar*, by Thierry Breton and Denis Beneich, two French computer professionals. The book is a computer science thriller, so for all of you out there who have longed for computer scientist heroes and heroines who resemble Indiana Jones or Mata Hari, this book is for you. (*Softwar is published by Holt, Rinehart Winston, and is available only in hardcover right now, at $15.95.) The two principal characters in the book are computer scientists, one male and one female, one American and one Russian, who happen to have been lovers, too, of course. The American is Assistant Professor of Computer Science at MIT Brendan Barnes, who is an expert on software reliability and debugging. The Russian, who was a grad student at MIT, is Yulya Voronkov, a beautiful Soviet computer scientist who is one of the department heads at the main Soviet computing center in Krasnoyarsk in Siberia. Barnes writes a piece for *Computers and Society* that talks about the potential of using software as a weapon in the ideological war with the Soviets. This piece naturally attracts the attention of the CIA, and Barnes is gently (and without much resistance) coaxed into becoming a member of a team of military officers, CIA agents and technical experts who plan to use software bugs to plague the Soviet effort to computerize their economy. They call these "softbombs," in a "softwar" with the Soviets. As one character puts it in one of the many extemporaneous speeches about the role of computers in national security: ...any sector of society can be destabilized, even completely paralyzed--industry and defense, civil and military communications, logistics and transport, public administration, the entire economy--simply by a couple of keystrokes on a computer terminal, anywhere in the world. We do definitely see this as the electronic battleground of the future, and we definitely see ourselves of being in the process of seizing the high ground for ourselves before the other side can get there. Barnes and his colleagues start by sabotaging a piece of software bought by the Soviets from the French. It runs on a newly purchased "Craig 1" that the Soviets bought from the United States. The software is programmed to spit out garbage when the U.S. Naval Weather Station in the Virgin Islands reports a barometric pressure of 1230 millibars. Then it is programmed to restore all the data in perfect shape when the Weather Station reports that same figure again. Of course, the Naval Weather Station is instructed not to report that figure unless specifically told to do so, so the "softbomb" is detonated at the choosing of the CIA. They pick a detonation time about an hour before the "Craig 1" is to be demonstrated to a visiting delegation of the Soviet Academy of Sciences. But, aha! There is a clever programmer at the console of the "Craig 1" who is bound and determined to find out why the machine went crazy at such an embarrassing time. He eventually discovers the programming trick, and is on to how this is the product of deliberate tampering by someone outside the Soviet Union. The KGB zeroes in on Professor Barnes, and he nearly catches a hand grenade in a Paris bar. From there on out, it's a battle of wits between the American computer scientist and his Soviet counterparts, and of course gradually that becomes the gorgeous and brilliant Yulya, his former grad student and former lover. The book is a fun read most of the time, especially for those intrigued by MIT trivia, Soviet trivia and computer trivia. There are a few too many spots where some character gives a speech about the importance of computers to some such thing or other (Barnes gives a long speech to his wife about why he's mixed up with the CIA and catching hand grenades in Paris and having an affair with a beautiful Carribbean journalist, and it turns out that he's a radical democrat who wants computers used to increase the democratic process in the West). But on the whole, it's a fairly conventional thriller spiced up for computer professionals with lots of jargon and speculation, and of course, dashing, sexy and adventurous computer scientists. -- Gary Chapman ------------------------------ Subject: "Computerized Voting -- No Standards and a Lot of Questions" To: risks@sri-csl Date: Mon, 14 Apr 86 21:50:29 -0500 From: Ron Newman The following is a slightly edited version of an article I wrote for the April, 1985 issue of the Computer Professionals for Social Responsibility Boston Chapter newsletter. ~~~~~~~~~~~~~~~ Our guest at CPSR/Boston's March 19 meeting was Eva Waskell, an independent science writer, former computer programmer, and current stringer for The Economist. She spoke with considerable alarm about the rapid and unregulated spread of computerized vote-counting systems in American elections. Waskell became interested in computerized vote-counting when Severo Ornstein of CPSR National suggested that she look into several lawsuits pending against Computer Election Systems (CES) of California. CES is the leading vendor of such software; it estimates that approximately 25% of the U.S. popular vote is cast on its equipment. Losing candidates in three states have sued the company, claiming that its system produced inaccurate or fraudulent results. While investigating, Waskell was appalled to find out that only one person outside of CES, a consultant for one of the plaintiffs, had ever examined the code. Waskell's investigation resulted in several New York Times articles last summer. To use a computerized ballot system, a voter inserts a punch card into a book containing the names of each candidate for office. The voter casts a vote by pushing a stylus through a hole in the book next to the name of the candidate. thus punching out the appropriate hole in the punch card. When the polls close, punch cards from all the precincts are trucked to a central location and tabulated on a mainframe, using software provided by CES or a competitor. The first such system was developed by IBM in 1964, for use in Los Angeles elections. In 1969, there were accusations of fraud in LA's elections. Fearing unfavorable publicity, IBM got out of the election business. Four of IBM's employees left IBM to form CES. Waskell pointed out four problems with this type of system: 1) A single central computer, in a single location, is counting all the votes. This takes control away from precinct poll workers, who formerly counted the votes and could recognize deviations from traditional voting patterns in their precincts. It also makes rigging the election much easier: instead of having to buy off many individual precinct workers, who are known to the community, one need bribe only a single computer operator, who is known by almost none of the voters. 2) Election officials must now be much more than clerical workers -- they must have technical skills. Frequently, new people are hired from the outside to learn and operate the computer equipment. Officials often do not know what the new people are doing. In one state, workers rubber-stamped computer printouts without examining them. A Minnesota election official commented: "It's kind of like black magic -- we really don't know what's going on." 3) There are no standards for election software, so anyone can write a vote-counting program. Vendors often talk state legislators into writing enabling legislation which is vague and favors their company. When a state Board of Elections certifies a computer system, the board often fails to consult any computer experts, and when it does consult experts, it may ignore their advice. The state of Pennsylvania certified a computerized election system despite strong objections from two CMU professors. (One of the CMU professors, Michael Shamos, wrote a report called "The Votomatic Election System: An Evaluation" in November 1980.) 4) Vendors consider their software to be proprietary. As a result, in the last 20 years, almost nobody has examined any of the software. Compare this to accounting software, which is subjected to audit by third parties. It is hard to have confidence that software is performing accurately when you cannot look at the code. Waskell said that states and municipalities have ignored four clear warnings against adopting these systems. In 1970, a Los Angeles blue-ribbon committee recommended that all vote-counting software be independently audited. Similar recommendations have been issued by the National Bureau of Standards (1975), CMU computer science professor Michael Shamos (1980), and the independent auditing firm of Coopers & Lybrand (1982). Nevertheless, none of the programs has been audited. According to Waskell, vote-counting programs are typically 4,000-5,000 lines of COBOL "spaghetti code." Earlier this year, an Indiana consulting firm analyzed CES's program on behalf of one of the losing candidates who is suing CES. They found numerous problems, including the following: The translation between the Hollerith punch card code and characters was nonstandard. The 1971 NCR system which the software ran on did not use standard EBCDIC. The contents of memory were continually being redefined. Numerous variables and fields were overlaid in memory. The same memory locations were re-used for the vote counts of different races. There was a total lack of structure. The program contained no PERFORM UNTIL (DO-loop) statements but had numerous undocumented GOTOs. COBOL's ALTER verb was used, producing self-modifying code. A call was made to an undocumented, unknown subroutine. The program interacted heavily with the operator, who can operate the console switches to examine and modify any part of the memory or program after each set of data is tallied. The program made it easy for the operator to turn off error logging and audit trails, without leaving any trace. There was heavy use of control cards in the data deck to redefine data fields, raising the possibility that a "knowledgable" voter could punch a control card and drop it into the ballot envelope to change the program's processing of election results. CES sends undocumented "updates" to election personnel before each election. The program used a time card to set the time and required that the computer's clock be disabled. This makes it impossible to determine how long the program runs or to accurately determine when logs or printouts are produced. The program did not correctly count "crossover votes," in which, for example, a voter punches a vote for a straight Democratic ticket and punches votes for several individual Republicans. Before an election in West Virginia, newspaper publicity specifically said that such votes were allowed, yet the program failed to count them. The program failed to keep a count of invalid ballots. A report to the Illinois Board of Elections in September, 1985, revealed that of the voting systems that the state tested before elections, 28% contained errors. Although those errors were corrected, such a high error rate suggests that many errors are never detected or corrected. Waskell said that other states' election officials are unaware of the Illinois findings. It disturbed her that Illinois failed to keep a record of the errors it found, but simply sent them back to the vendors for correction. Suits against CES have been filed in Indiana, West Virginia, and Florida, but judges have dismissed several of the cases for lack of evidence, saying that computer experts' testimony is mere "speculation" and "suspicion." It is hard to successfully prosecute such a case when the computer system itself is designed to ensure that no evidence exists. In the Indiana case, the plaintiff charged that a CES representative was in the counting room on election night, turned off the program's logging, and added two extra control cards after the last votes were counted. In the West Virginia case, a CES representative allegedly connected a modem to the computer and was down on his hands and knees around the compter on election night, claiming that "a screw was loose." In addition, the West Virginia candidate alleged that the county clerk's husband manipulated the computer's switches during the count. Evidence in this case is difficult to obtain because the county clerk destroyed all the ballots 61 days after the election and returned the program deck to the vendor. According to Waskell, a company called Cronus recently purchased both CES and two of its competitors, Thornber and Governmental Data. Together, these three companies market 60-80% of the voting systems in use. Cronus is financially tied to the Tyler Corp., whose chief executive officer is Fred Meyer, the Republican Chairman of Dallas County, Texas. Meyer announced his candidacy for Mayor of Dallas one month after the city bought a CES voting system. Ms. Waskell closed her presentation with a series of recommendations. The Federal Government, using election powers outlined in the constitution, should mandate that all vendors conform to NBS standards. State election laws should be changed to show a greater understanding of the technologies. Local election officials must ensure that audit trails are always turned on and that they are continuous and unbroken. Also, local officials should count a random 5% sample of the vote by a different method, thoroughly test computer systems before adopting them, and be accountable and responsible for their use. People interested in more information about this subject may want to read New York Times articles by David Burnham, published on 7/29/85, 7/30/85, 8/4/85, 8/21/85, 9/24/85, and 12/18/85, and a letter to the editor, 8/6/85. Ms. Waskell was the source for much of the information in these articles. If you write to me (newman@mit-athena), I can tell you how to reach Ms. Waskell--I'm uncertain whether she wants her address & phone number posted on the net. ------------------------------ End of RISKS-FORUM Digest ************************ -------