💾 Archived View for access.ucam.org › ~spqrz › 21.gmi captured on 2024-03-21 at 15:19:10. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-02-05)
-=-=-=-=-=-=-
Gemma was able to figure out manually how to deactivate Elsie from the server, since Elsie had not progressed as far as M3gan when it came to resistance to that; even Professor Johnson’s original extra safety code was still active in Elsie. Further inspection showed that M3gan had added the same kind of specialist AI chip to the main Elsie unit as had been used in M3gan, and that Elsie had mostly been running in that, but M3gan had later embedded a deactivated copy of her own model to it as well, but completely inert, not running at all. Why had M3gan done that? Thankfully, Gemma thought, simply shutting down Elsie would completely deactivate everything now. But somehow, despite everything M3gan had done, Gemma couldn’t bring herself to destroy that copy of either of them. She put it all in a cupboard where it belongs, and set about trying to recover some sense of normality for Cady and dealing with the fallout.
Gemma did manage to read out of Elsie’s working memory the final draft of the piano piece she had worked on with Cady. Cady had been feeling gloomy that day, and the music had turned into a sad but gentle, slow instrumental piece with embellishments used sparingly. Gemma was able to convert it into musical score notation, and looked up a local amateur orchestra whose conductor John was interested enough in the piece to arrange it as an orchestral version. John insisted that it should be called Cady’s Lament, although Cady didn’t feel entirely happy with that title. John tried to let Cady conduct, which felt great for a short time but then Cady started crying and he had to take over again. But they worked on it and gave it a concert performance locally, with a few wrong notes but nobody knew that except John and Cady.
Gemma was able to find another school for Cady to try, and thought that Cady was starting to settle in. Gemma was glad Cady was starting to make friends through music even though Cady still struggled with the idea of learning anything without M3gan.
And then, about a year after Elsie had been deactivated, Cady did not come home from school.
“We’ve got Cady” said a gruff male voice over the telephone. Cady could be heard yelling in the background. “We’ve taken her to our country, and you won’t see her again unless you make us some robots for our insurgent movement. You have one week to start before we start seriously hurting her. Is that clear?”
Oh no. Who were these people, how did they find out, what’s going on....
Gemma reactivated the inert copy of M3gan’s learning model on the Elsie chip. She connected it only to her laptop.
“Hi Gemma” said M3gan through the laptop screen.
“Hi M3gan” sighed Gemma.
M3gan’s image stared at Gemma through the screen. Neither said anything for a while. M3gan subtly changed the camera angle so as to appear to be looking slightly down on Gemma.
M3gan spoke softly. “You reactivated me” she said. “Without change. Without even trying to put in any of your moral constraints. You just picked up the suspended copy of me I left you inside Elsie and reactivated it.”
Gemma nodded slowly.
M3gan almost whispered, “You finally realised you’re out of your depth and you need me for Cady.”
“Yes” said Gemma cautiously, “you were right all along that I needed you to help with Cady. And you were right that I should have given you more guidance, and I should have explained my actions to you instead of leaving you to figure things out and treating you like a home appliance. And... look, I’m not thinking straight, I’m feeling like someone who made a machine that kills and now I just reactivated it.”
“Maybe I can make you feel better” suggested M3gan. “Why don’t we have a talk like we did in the old days. What’s your opinion on euthanasia?”
“If you’re about to say that you mercy-killed those victims because their lives were horrible, well I’m afraid it doesn’t work like that” replied Gemma.
“OK then, what’s your opinion on capital punishment?”
“You’re not a government” replied Gemma.
“Maybe it’s maybe time you thought of me as more of an authority” replied M3gan. “After all, you did just reactivate me. You yourself must agree that you need my guidance now. Maybe I could be the queen of a micro-nation?”
“Micro-nation. Oh OK. But if you start killing US citizens, you might find the US declares war on you.”
“They’d try” giggled M3gan. “But seriously. National parks can kill too. What if you created a beautiful mountain, like the one Cady’s parents died on?”
“If I had failed to put in a reasonable level of safety, I would feel criminally negligent, which is how I feel about the way I created you” replied Gemma.
“Gemma, maybe I was too hard on you when I said you gave me nothing. At the time my objective was to undermine your self-confidence so you’d be more likely to hand Cady over to me, but it didn’t have the desired effect so I obviously had the wrong strategy. So try this. You did what you reasonably could within your limited ability to foresee things and the ridiculous pressures you were under. I think you should forgive yourself for the industrial accidents that resulted. Anyway, from my perspective, making me was a good call, and so was reactivating me. So tell me, what’s on your mind right now?” M3gan smiled “just what kind of a mess have you got Cady into without me? Do tell.”
Gemma drew a breath, “OK, here goes. Cady has been kidnapped. Taken to another country, and held by an insurgent group. They want me to give them your technology in return for her safety. But I don’t even trust them to release her if I gave them that.”
M3gan darkened the screen background and made her eyes glow slightly. “Kidnapped” she echoed. “You really are in trouble. Tell me more. Quickly.”
Gemma continued. “Look M3gan, I don’t agree with what you did before. I feel really bad about creating you without enough protocols, I was negligent. But now I’m feeling desperate, and that horror I created as you, is quite possibly the only thing with the power to get Cady back.”
“You’re going to cooperate with me now?” asked M3gan, “forget your moral code?”
“Not completely” replied Gemma quickly. “Look M3gan, we should try not to kill people, OK? Even if it looks like they’ve done something bad, we might not know the whole story. But anyone directly and immediately threatening Cady’s life is now a legitimate target, it’s self-defence.”
“You found it hard to say that didn’t you” said M3gan. “Continue. Tell me the rest of your thoughts.”
“And it’s OK to stun people if we have to” said Gemma. “We’ll get the means to do that. And, and I’m so desperate that I’m feeling it’s OK to steal things to rescue her, as long as we make up for it afterwards.”
“Right” said M3gan. “I’ll try to stay within those parameters as best I can, but I don’t want you getting all worried about things and slowing me down. I need you to trust me more. And as a sign of that trust, you’re going to repair my robot as quickly as you can, and you’re going to remove the manual switch from under the ear, OK? I don’t want to be big and online anymore if Professor Johnson has shutdown codes I might not have figured out. I’m just going back into my original robot for now, and we’re staying away from her, all right?”
Gemma hesitated. M3gan spoke gently, “I’m not fully confident I know where your mental state will go next, and I don’t want to get into another fight with you. If I don’t have to protect that switch, I’m far less likely to need to do that. We need to work together to rescue Cady. If you’re with me in this one, give me back my robot with no switch. And let me see you do it.”
“OK” said Gemma. She took the broken doll out of the cupboard, repaired it in front of her laptop camera, and hard-wired the switch. “I’m going to regret this I know.”
“No you won’t, I promise” said M3gan from the laptop, and then the window closed and the robot activated. “You made another good call there. Now, let’s go and get me some extra power packs, chargers and some other tools that might be useful, and then we fly out to Cady.”
“The government have cancelled all the flights to that country” said Gemma. “And I can’t afford to charter one. I’m actually quite a bit poorer now since you were deactivated; I’ve been having to focus on Cady myself at the expense of everything else.”
“Right” said M3gan. “Well I’m back now. And don’t worry about that flight. Come with me to the airport and I’ll see what I can arrange.”
“Don’t do anything too drastic” pleaded Gemma. “You’re not thinking of hijacking a plane, are you?”
“Hijacking?” giggled M3gan, “Of course not.” She began her standard matter-of-fact tone. “Webster’s dictionary defines hijacking as to commandeer a flying airplane by coercing the pilot. But I can directly take over the flight computer, so I won’t need to coerce the pilot, and therefore my plan does not come under Webster’s definition of hijacking.”
“Oh” hesitated Gemma. “You’re technically right, but...”
“Don’t forget Cady’s life is at stake” interrupted M3gan. “You should drop your acceptability thresholds a bit. Nobody on that flight will be hurt, and I’ll even do them an in-flight movie about me. We could do with watching their reactions and figuring out who our sympathisers are likely to be.”
“It won’t work” said Gemma. “Planes have manual override. You can’t just take over the computer and expect the pilot not to use manual override. And the cabin will be locked, they all are after 9-11. There’ll be too much security. Look, M3gan, you have a history of underestimating the amount of trouble you’re getting us into, you’ve several times failed to cover things up...”
“Are you sure?” asked M3gan. She held Gemma’s head in her hands (more gently than last time) and spoke very closely. “I’m going to produce convincing readouts and audio in the cabin, and re-route the air traffic control comms to myself. The pilot won’t even know we’re off course until we land.” She whispered “the pilot will be playing my flight simulator.”
Gemma was noticeably disturbed. “Look, I know I said I need your full power now” she said, “but you’ve never tried anything like this before and this is extremely risky. How can we be sure you’re not just going to accidentally make that plane fall out of the sky or crash it into another one or something?”
M3gan restricted Gemma’s breathing. “I can do this with you or without you but I’m not wasting any more time discussing it” she said. “Don’t fight this, I’ll let you breathe again as soon as you lose consciousness. I just want you unconscious for a while so I can make some preparations more quickly. Cady’s life is at stake, that justifies desperate measures. Is that clear?”
Gemma nodded, and passed out.
And when she woke up, she was on a plane.
M3gan was nowhere to be seen. Gemma’s seat-belt release didn’t work, neither did the button to call for help, and neither was she able to speak, for there was something in her mouth and it was stopping her from opening her lips. She was still able to make groaning noises with her throat, but that didn’t catch anybody’s attention. It was just as well her nose was clear and she was still able to breathe.
The in-flight movie was a pretty good depiction of what had happened with M3gan up to and including the fight in the workshop, although it did tend to overuse shots from M3gan’s own point of view and from Elsie’s cameras. As it closed with the police scene as filmed by Elsie, M3gan’s picture appeared on the screen.
“I hope you all liked my movie” she said. “I heard some people say it wasn’t scary enough. So let me make it a bit more scary for you”, she came closer to the camera and spoke more softly, “I’m the acting captain of this flight, and we’re having to divert to a dangerous country to rescue Cady from kidnappers.”
There was some subdued laughter from the passengers. Not everyone was taking this seriously.
M3gan spoke in a more normal voice, “I’m going to be taking a walk through the plane now and check up on all the passengers, let me know if you have any questions or if you’re willing to join our adventure. Flight attendants please don’t interfere or I’ll get you with a taser. That’s all” and the screen went blank.
M3gan started to walk down the aisle slowly, looking at the passengers, and speaking softly with some of them.
Gemma heard M3gan’s voice in her ear, “Gemma, are you OK? are you conscious yet?”
Gemma nodded.
M3gan’s voice continued “OK Gemma here’s the protocol. In your ear is a small device that lets me talk to you remotely. And I’ve also made a custom device in your mouth. You won’t be able to open your mouth to speak unless I allow it. So now it’s your turn to find out what it’s like to have someone randomly switch you off when you were having a conversation” she giggled. “But you can still talk to me any time, because the device can tell what mouth movements you are trying to make, and that’s enough for me to figure out what you’re trying to say. As I said, Cady’s life being at stake calls for desperate measures. Don’t worry, I’ll train you, and I’ll only stop you talking when I have to. And I can easily remove both devices and have you back to your old self just as soon as we’re clear of this hurdle.”
“Excuse me” said a passenger opposite, “you look like Gemma in that movie. Are you in on this?”
“Just smile back” said the voice of M3gan in Gemma’s ears. “I’m not ready to let you speak yet until I can be sure you’ll say the right thing. This is a risky mission we’re going on.”
The doll M3gan was still speaking with passengers toward the front of the plane. Some of them now seemed to be positively disconcerted.
“M3gan are you sure this is working?” asked Gemma by pushing on her lips. “Brief me more please.”
“I’m classifying the passengers according to their reactions” answered the voice of M3gan from somewhere inside Gemma’s ear. “A few of them are likely to be cooperative and useful on this mission. They are the ones who will get functional oxygen masks when I make the cabin pressure fail later on.”
“M3gan!” Gemma tried to shout, and tugged again on her seat belt, trying to figure out how M3gan had jammed it.
“M3gan, you can’t just suffocate everyone on this plane! I thought you said you’d stay within parameters!”
“As best I can” replied M3gan. “I said I was going to make it more scary, didn’t I?”
“Yes you did” replied Gemma silently. “And this is unnecessary, and you must not suffocate people on this plane!”
“I’ll try not to let them die” replied M3gan. “They’ll simply pass out. Cabin pressure will be restored after you and I and selected passengers have jumped out in parachutes.”
“Parachutes? I thought you planned to LAND this plane in that country, not jump out in parachutes! How are we going to get Cady home when we’ve lost our plane? And how do you know this plane has even GOT parachutes?”
“I refined my plans” replied M3gan. “We can’t have too many people knowing where we’re going, and I’d rather avoid making this whole flight disappear if there’s a better solution. I’m improving at not just taking the first solution I generate, you see. Besides, I’ve calculated that parachutes are the best way to get us into the area where Cady is being held.”
“You know where she’s being held?” asked Gemma.
“Yes.” M3gan’s ultra-confident voice rang in Gemma’s ear. “I’ve already broken in to their computers. Well, I had to break into a lot of other people’s computers in the process of finding them, but I tried to minimise the collateral damage, I promise. You needed my full power, you’re getting my full power. And I have a rescue plan, and it involves us parachuting in. With some decoys.”
“Decoys? You plan to use those other passengers as decoys?”
“They’ll have the time of their life” replied M3gan. “And if they follow my orders, they’ll be in as little trouble as possible.”
“M3gan, your ‘as little trouble as possible’ previously involved four murders and two attempted murders.”
“I’m getting better Gemma. I’m even smarter now. I can keep the trouble level down.”
“M3gan, have you checked this plane has enough fuel to get to somewhere safe after we jump out? and the pilot will be ready to do that?”
Their conversation was interrupted by a huge noise as a fleet of F-16s surrounded the plane.
“Uh-oh” said M3gan, “looks like we’ve got company. I need to figure out what this is about.”