Date: Thu 12 Sep 85 13:26:05-PDT From: Peter G. Neumann Subject: RISKS-1.10 To: RISKS: ; RISKS-FORUM Digest Thursday, 12 Sep 1985 Volume 1 : Issue 10 FORUM ON RISKS TO THE PUBLIC IN COMPUTER SYSTEMS Peter G. Neumann, moderator Contents: Weizenbaum, etc.; even if SDI worked.... (John Shore) SDI (John McCarthy) More on SDI reliability (Martin Moore) (Contributions to RISKS@SRI-CSL.ARPA, Requests to RISKS-Request@SRI-CSL.ARPA) (FTP Vol 1 : Issue n from SRI-CSL:RISKS-1.n) ---------------------------------------------------------------------- To: risks@sri-csl.ARPA Subject: Weizenbaum, etc.; even if SDI worked.... Date: 12 Sep 85 11:11:13 EDT (Thu) From: John Shore It's tempting to respond to Weizenbaum by arguing against the general proposition, "Don't do it if there's a way around it". After all, should we refuse to develop bullet proof vests and to equip police officers with them just because a criminal might approach from behind and stab them in the ass? Assuming that a proposed defensive system will work, the relevant question is what is the cost of developing it compared to the cost of getting around it? In the case of SDI, one should distinguish between defense against a few missiles vs. defense against a massive attack. Either defense would be enormously expensive to develop. If the goal of the attacker is to detonate a few bombs (or threaten to do so), then it is obviously easier and cheaper to get around SDI than through SDI. Here, Weizenbaum is probably right. If the goal is massive or total destruction (including destruction of our missile forces), then getting around SDI (assuming SDI works) does not appear to be either easy or inexpensive. Here, Weizenbaum is probably wrong. In this case, however, the premise is most likely also wrong. Moreover, suppose that the premise is right -- i.e. SDI works perfectly. As Parnas has pointed out, there's no way for anyone to establish this fact, which shows the absurdity of arguments like "give us SDI and we will dismantle our missiles". ------------------------------ Date: 12 Sep 85 0057 PDT From: John McCarthy Subject: SDI To: risks@SRI-CSL.ARPA Some remarks of mine about SDI on Stanford BBOARD have been referred to. For the benefit of non-readers of that BBOARD, they mainly concerned whether I, like Chris Stuart, should use the IJCAI platform to say something about it. I said nothing in my lecture, but in my press conference, added to my remarks on AI, the remark that there was no principle of computer science that says that programs of any particular task cannot be written and debugged. Not much interest was shown by the assembled press; there was exactly one question on that point. At the suggestion of Robert Jastrow, who is one of the main scientific defenders of SDI, I made the same point in letters to three Congressmen, said to be influential in the matter of SDI appropriations. Now I shall say my opinion about SDI. 1. If it can be done, it should. If it affords complete protect, that's great, and if it affords partial protection, that's good. The balance of terror is a bad thing. Here are answers to some counter arguments to its desirability. (a) Joe Weizenbaum says that it attempts a technological solution to a problem that should be solved morally. Alas, moral progress has been so slow that almost the only moral problems to be even partially solved are those that can at least partially been turned into technological problems. For example, the technology of contraception has greatly reduced human unhappiness. (b) It is argued that the Soviets would have to attack at the first sign of deployment. Every past imminent advance by either side has in principle given the other side some temptation to strike before it can be deployed. So far as we know, neither side has even come close to giving in to such temptation. One reason is that the effect of any advance is always subject to a probabilistic estimate, so temporizing has always looked better than attacking. Even if SDI works very well, it may be that no-one will be able to be sure that it is that good. However, most likely the main reason has been is that neither side ascribes the very worst intentions to the other with certainty. Each side has always said, "Perhaps they don't actually mean to attack us. Why have a nuclear war for sure instead of only a certain probability?" Anyway the Soviets have experienced a period in which we had complete nuclear superiority and didn't attack them. 2. My opinion is that if the physics of the problem permits a good anti-missile defense the programs can be written and verified. However, it will be quite difficult and will require dedicated work. It won't be done by people who are against the whole project. Computer checked proofs of program correctness will probably play some role. So will anticipating what kind of bugs would be most serious and putting the biggest effort into avoiding them. Having many people go over and discuss all the critical parts of the program will also be important. ------------------------------ Date: Tue, 10 Sep 85 13:56:45 CDT From: mooremj@EGLIN-VAX Subject: More on SDI reliability To: risks@sri-csl.arpa Cc: soft-eng@mit-xx, lin@mit-mc, mooremj@eglin-vax > From: Herb Lin > My primary complaint about your otherwise interesting table is that it > assumes independent failure modes. I think it is much more likely > that the effects of coupled failures are larger. In particular, given > the failure of one platform, it is more likely that more than one will > fail. Good point. My original post did concern only statistically independent failures. If I can be forgiven one more table, I'll address coupled failures. Independent failures are caused by events isolated to a single platform, e.g., electrical component failures. The occurrence of such a failure in platform J does not affect the probability of a similar failure in platform K, i.e., P(K|J) = P(K|~J) = P(K). Coupled failures are failures such that the probability of failure is low in any platform, but is greatly increased in all platforms when it occurs in any one of them. For example, consider that a hostile power might develop a new method for its missiles to escape detection. The probability that it will fool any one platform may be low; but if it fools one platform it is likely to fool more than one, perhaps all. For arbitrary platforms J and K, P(K|J) >> P(K|~J). The original false positive table is not affected by this, since it showed the probability that at least one platform would fail. Coupled failures do not change that probability, only the probability that if one fails, others will (although it is true that while this country might be able to explain away a single false positive, explaining a whole bunch of them could be a lot tougher!) The false negative case is where the kicker really comes in. The original false negative table applies to independent failures. The following table is structured similarly, but instead of using the probability of failure (Pn), it uses the degree of coupling, Pn(K|J). This table shows, for a 100-platform system, the probability of various numbers of successful responses, given that at least one system has experienced a coupled failure. Pn(K|J): .5 .6 .7 .8 .9 .95 .99 +------------------------------------------------------- N: 0 | 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 5 | 1.0000 1.0000 1.0000 1.0000 0.9746 0.5550 0.0033 10 | 1.0000 1.0000 1.0000 0.9973 0.5355 0.0265 0.0000 15 | 1.0000 1.0000 0.9998 0.9123 0.0677 0.0001 0.0000 20 | 1.0000 1.0000 0.9896 0.5200 0.0017 0.0000 0.0000 25 | 1.0000 0.9993 0.8740 0.1204 0.0000 0.0000 0.0000 30 | 1.0000 0.9822 0.5116 0.0097 0.0000 0.0000 0.0000 35 | 0.9988 0.8525 0.1465 0.0003 0.0000 0.0000 0.0000 40 | 0.9781 0.5054 0.0176 0.0000 0.0000 0.0000 0.0000 45 | 0.8426 0.1574 0.0008 0.0000 0.0000 0.0000 0.0000 50 | 0.5000 0.0219 0.0000 0.0000 0.0000 0.0000 0.0000 55 | 0.1574 0.0013 0.0000 0.0000 0.0000 0.0000 0.0000 60 | 0.0219 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 65 | 0.0012 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 70 | 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 For example, if the degree of coupling is 0.7 -- that is, if something that causes failure in one platform has a 70% chance of causing failure in any other platform -- then the probability is 51.16% that at least 30 of 100 platforms will respond correctly, 14.65% that at least 35 will, and so on, GIVEN THAT THIS TYPE OF FAILURE OCCURS IN THE "FIRST" PLATFORM. Don't forget that the probability that the first platform will fail is UNRELATED to the probabilities in this table! As far as the relative probabilities of independent and coupled failures, I haven't a clue. The independent failures are the easiest to get a handle on through reliability theory; the coupled failures may be the result of unknown shortcomings in design, or due to unknown hostile actions. (There is an old saying that there are always more unknown errors than known errors, because known errors are limited, but unknown errors are unbounded by definition!) Martin Moore mooremj@eglin-vax.arpa ------------------------------ End of RISKS-FORUM Digest ************************ -------