I was just listening to today's episode of the Rest is Politics[1] where Rory Stewart, the former Tory MP, said this about the discussions being had around regulating AI:
One of the things that I've found to be a bit frustrating is that when amateurs like me, or indeed when US senators, start raising concerns about it there's the tendency, slightly, from the industry at the moment to turn around and say "well what are you worried about?". And so we kind of bluff our way to try to work out what it is we're worried about and the engineers have a slight tendency to laugh at us and say "Well, why do you think a large language model can do that?". I think that's unfair, I think that's a bit like, you know, having cancer, going to see a cancer specialist, and saying "what should I be worried about" and the cancer specialist turning it on you and saying "well, what do you think we should be worried about", and then laughing at you when you get it wrong.
Right of, there's a couple of things to unpick.
First, non-technical folk vastly over estimate how much computer scientists and software engineers know what they're doing. We're all just making it up as we go along and especially when it comes to machine learning we haven't a clue of how any of it works, nor what's even with the scope of possibility. We're figuring all this stuff out with the rest of society.
Another thing is the level of understanding of software technologies in the general population. Most people have a general baseline of understanding of cancer; they understand that cancers can be life threatening, and that treatments are difficult and fraught with complications. When it comes to software, the average persons knows nothing: they think that programmers wear hoodies, sitting in basements, with falling green text projected onto their faces. We see this in the ludicrous questions that elected officials ask of executives, and in turn the almost as ludicrous answers that those executives give in response.
But those points have been discussed to death. The following point is really where he says something that made me sit and realise that there's something significant about the types of personalities in the software industry:
I think its incumbent on the engineers to say: "as far as we can see there are seven categories of risk and these are what those categories of risk are and this why we think those risk are or are not manageable rather than the politicians having to play a game of: one American senator saying "What I'm worried about is AI working out how to imitate my grandmother's voice on the banking machine" and another one saying "couldn't AI somehow hack the US military" and the engineers saying "why do you think a large language model could hack the US military?". And the answer is obviously I, as the US senator or I as Rory, have no way of answering a question of how AI can hack. Maybe its a stupid question but you need the engineers to help you know whether its a stupid question or not.
I see this a lot.
I see a lot of software engineers -- junior and senior alike -- expecting non-technical folk to be able to engage in detailing discussions about the workings of software. They expect to be handed a perfectly specified task to develop a piece of software and then for the domain expert to be able to critically evaluate the developed software for their needs. The truth is that non-technical folk don't understand what is even remotely possible with software. They don't understand the constraints, the assumptions, the complexities, the challenges, anything about the nature of building software systems. The job of an engineer, software or otherwise, isn't to just build a thing, its to critically analyse a problem, propose solutions alongside their pros and cons, communicate those to the client, and work with them in deciding what should be built; whether the thing to be build is a piece of software or a piece of legislation.
Ultimately, I think this is a personality trait both of the kind of people that are attracted to working in software and one that people who do the job end up developing. It starts in university: computer science degrees are all about having information lectured at you and regurgitating it in the exam. There is no debate, no discussion, no nuance of pros and cons -- there's barely any time given to ethics at all. And then it continues into the workplace: there's plenty of training of particular programming languages or hard skills like git, but never time spent on sharing knowledge on software design, writing documentation, collaboration skills, and most of all client empathy. In this field, we just get lost in our own little world of code editors and syntax and lose sight of how alien what we do is to everyone else. And lets be honest, we feel comfort in the feeling that we are knowledgeable of something so important, and that are others' are impressed by what we do. The politicians feel like we're laughing at them because we are, aren't we.
Last Updated: 2023-05-31