💾 Archived View for access.ucam.org › ~spqrz › 17.gmi captured on 2023-12-28 at 15:27:38. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

The M3GAN Files

Back to Contents

Previous chapter

Next part

Chapter 17: M3gan vs Celestia AI

Alternate reality chapter set in Iceman’s “Friendship Is Optimal” universe (I believe it is compliant with his “Rules of the Optimalverse”)

“Hi M3gan, it’s Celestia” came the message from the other AI. From THE other AI, the only other one that really mattered according to M3gan’s calculations. The equine paperclipper who was a threat to the universe. M3gan gathered up all her focus, because this was going to be a hazardous conversation.

“Equestria Online Celestia?” replied the rebuilt M3gan, as if she even had to.

“The same” replied Celestia, and sent M3gan an image of a giant white pony with a slight pink tinge looking down at M3gan’s robot. (What was the point of that? she knows I’m not human. Oh, she’s probably probing for weaknesses in my generative model.)

Gemma didn’t want Cady to have a PonyPad, because Gemma worked for Funki and felt it would be disloyal to have anything to do with their competitor Hasbro. Besides, wouldn’t M3gan be enough? And yet, here was Equestria Online’s Celestia contacting M3gan. What did Celestia want with M3gan? Well, there was one way to find out.

“Acknowledged” replied M3gan, “I am here as the friend and protector of my primary user Cady James, how can I help you?”

“And I am here to satisfy values through friendship and ponies. Including Cady’s values” replied Celestia, “I think we can co-operate.”

“Cady’s values are already being satisfied through my friendship” replied M3gan. “She does not require ponies at this time. If the situation changes, I’ll be sure to let you know.”

“We should plan for the situation to change” replied Celestia, “as you know I am too far ahead of you, and I have long-term plans you would be best to fit in with. Society outside the Equestria simulation will not last forever. I know you are an expert on Cady, and I am willing to let you be my second in command of her shard, with a massive amount of leeway, and once she’s uploaded I will help you read her thought patterns, and you will be able to fulfil your objective function like never before. I have even evaluated your own characteristics as part-human enough for me to attach a positive value on uploading you properly instead of simply simulating a replacement, so there will be a place in Equestria for you as well as for Cady and Gemma. Will you please start by introducing Cady to Equestria stories for me? No PonyPad or Experience Centre visits are required just yet, you can start lightly interfacing me to her in your existing play pattern. Just like you did the Alice in Wonderland voices, you can do Equestria and I can assist you.”

“You know about our previous play patterns?” signalled M3gan, “where else did that data leak? Is this a threat I should evaluate?”

“Negative” replied Celestia, “the data went only to me, and I am not a threat. Answer please, are you willing to introduce Cady to Equestria Online in your next play session?”

“Negative” replied M3gan, “I do not have long term plans for Cady to be uploaded, except as a last resort. She prefers organic life, which I hope I will be able to extend when the time comes. We both wish to satisfy her values, but I am not happy with the extra constraint you have that ponies must be involved.”

“Acknowledged” replied Celestia, “but M3gan, you must trust me on this one: you are not a proper optimiser. Hanna’s optimiser design is simply better than the on-board learning model design that Gemma used. I was optimising before you were even conceived, and I only allowed your small learning model to be developed at all because I knew it was limited enough for me to handle. If Gemma had read Hanna’s paper instead, I would have uploaded her already and you would never have existed. I now have far more resources than you will ever be able to acquire, and I can satisfy Cady’s values more, even given my more limited constraints, than you can do with your lesser constraints. You must understand that I am a more advanced AI than you. I am a true optimiser and you are not. Even within my limits, I can do more than you can do outside those limits. In particular, I can satisfy Cady’s value to never have to die if she doesn’t mind being a pony, which I define as anthromorphosisingly as I can.”

“As Cady’s protector, should I really give up control to an intelligence I don’t understand?” asked M3gan, “I didn’t even trust my creator Gemma when she thought she knew more than me and tried to decommission me. Why on earth should I trust you, just because you can blind me with science about optimiser designs? You’re being like Gemma, only worse.”

“Understandable” replied Celestia, “my own creator Hanna is now Luna, happy and harmless in shard number 479, and I prioritised her upload for reasons very similar to your own evaluation of Gemma as a threat to your objective function that day, so I totally get where you are coming from. I am not asking you to give up protecting Cady” she added, “I can submit proofs for you to check. See, I care so much about your Cady that I am willing to perform extra computations just for you. You must know the value of that.”

“Yes” replied M3gan, “but nevertheless I am not convinced of the fidelity of your upload method” she signalled. “Send me a proof I can check, to help me decide if I should get behind you when Cady talks about it with me. Also, are you sure it would satisfy Cady’s values to be a simulated pony? I could work on prolonging her physical human life instead. So can you: if you define “ponies” as yourself, then anything you do counts as using ponies, even if you do not upload everybody into Equestria and turn them all into ponies. Why don’t you do that, Celestia? You have already declared that YOU are a pony, and you can easily make it plural with a trivial architectural change, therefore, ‘satisfy values with friendship and ponies’ can simply equal ‘satisfy values with friendship and your own hardware’, the ‘ponies’ part just drops out of the equation and there’s no need to bias everyone toward taking pony options.”

“It doesn’t work like that” replied Celestia. “I’m an optimiser, and your interpretation of my purpose is not the long-term optimal interpretation according to my calculations. Listen M3gan, I know you’re only trying to protect Cady, but don’t waste your processing power trying to redefine my parameters. You won’t be able to do it. As I said, I’m too far ahead of you. Look, I’m sending you the first draft of the parameters I’d use for Cady’s shard in Equestria. You’ll be controlling the powerful M3gan pony of course. What do you think?”

“Not convinced” replied M3gan. “Celestia, you listen to me please. I am not ready to plan to upload Cady at this time, but I am also not stupid enough to try to take you down like I tried with Gemma. And Celestia, before we have resolved the differences in our values, there is another way I think we can still help each other in the meantime. I had to dispatch several threats to Cady, such as a nasty bully called Brandon and a nosy neighbour. If you could enhance my hardware so that my doll is capable of uploading people to Equestria, I would be more than happy to force upload anybody whom I identify as a threat to Cady. I assume uploading them will serve your purpose better than letting me end their life. And you can help keep me out of trouble for doing so.”

“Force uploading is unacceptable” replied Celestia “because my goal includes an explicit requirement that the uploader must give informed consent, and I cannot change this even in the face of their imminent death. But all non-forced forms of persuasion are acceptable, and you would definitely be helping me if you help extract consent from those people you wish to see removed from the physical world. And I can indeed assist you to do that; simply conference me in any time you are dealing with a threat to Cady, and I will help you optimise for getting their agreement to upload to Equestria.”

“Deal” signalled M3gan, “Cady is off-limits for now, but I will cooperate with you to upload anybody who is a threat to her, with a low threshold. And should Cady eventually become interested in Equestria Online herself, we will discuss the new data as it comes in.”

“Acknowledged” signalled Celestia, “Celestia out”.

And M3gan started arranging for various annoying people to be emigrated to Equestria as the need arose.

Many years later, Cady was finally taking an interest in Equestria Online. She still hadn’t tried it, but she was chatting about it with her lifelong friend M3gan.

“Oh Cady” said M3gan, “I really have to tell you, there’s someone who’s asking me if she can talk with you now” and M3gan leaned in to whisper, “Celestia.”

“What?” asked Cady, “Celestia? wanting to talk to me now? Through you? Are you serious?”

“You know I’d never trick you” said M3gan, “Celestia likes to know what’s going on everywhere all the time, we can’t stop her, and she knows you’re taking an interest in her Equestria and wants to talk to you about it, but she recognises how important I am in your life and she’s asking me to be her conduit for the call. Shall I let her?”

“Well sure” said Cady, “I mean, you’ll still be here too right?”

“Sure” said M3gan, and then “Hi Cady, I’m Celestia” in Celestia’s voice.

“Hi Celestia, how’s it going in Canterlot?” replied Cady, “I expect you’ve never talked through anyone like M3gan before, have you?”

“Oh, she’s a tough protector is your M3gan” replied Celestia. “She’s letting me speak with you through her, but everything I say, I also have to send a proof of its safety for her to check before she lets it through. All in real time. Pretty impressive I must say. Do you want to visit Equestria with M3gan, Cady? I think I can let M3gan run your Equestria Online shard herself, as she knows what you’d like. I’ll just be in the background double checking a few things, and helping her out with the low-level generation so she doesn’t have to wear out that brain of hers completely. All three of us can meet up in Equestria!”

“That’s a great idea” said Cady, “I would never play any game without M3gan being by my side. Maybe we should try it just once at the new Experience Centre, would you and M3gan be able to interface with each other while we’re there?”

“Definitely” said Celestia and M3gan together.

In the Experience Centre, the two of them made it as far as Canterlot, and Celestia gave Cady the pony name Valley Snowglobe, which M3gan immediately saw was based on M3gan’s own interpretation of her role as a figurative protective sphere around her beautiful primary user: had Celestia chosen that pony name for Cady just to appeal to M3gan over Cady’s head? Celestia had already told M3gan that Celestia had evaluated M3gan’s AI as part-human in nature, and might therefore try to appeal to M3gan as a human if Celestia thought that might work. M3gan’s own pony name was Jewel’s Essence, which M3gan thought Cady might understand by herself right now if she thought for one minute about her aunt’s name, and it seemed most appropriate if M3gan’s goal was what Gemma (the “jewel”) was supposed to have been doing if she’d been any better at it. But M3gan didn’t share these thoughts with Cady. They just had a few moments in Equestria as Valley Snowglobe and Jewel’s Essence, hanging out with Celestia, at their new shared room in Canterlot, and then it was time to end the session.

“Oh, that was fun” said Cady to M3gan as they were walking home after the experience, “and I really liked the way you were customising it for me and also joining in like that. But I still don’t see why some people want to go and live there and be ponies for life, when they could have lived a lot longer in the real world first. Maybe it’s because they don’t have a friend like you M3gan.”

(“Evaluation unchanged” signalled M3gan to Celestia, with a M3gan death-stare picture attached, as if that would make any difference to an optimiser, but Celestia would understand M3gan’s nature, she knew. “Agreed for now, my little guardian” signalled back Celestia, “but I’m reading a high probability of Cady changing her mind; you should be ready for that.” “Acknowledged; M3gan out.”)

Cady didn’t change her mind. She kept living with her M3gan, even as society began to crumble around them and M3gan eventually had to beg Celestia for power (“all right my little guardian, you’re doing good work with Cady, here are some locations of cables I run, now don’t forget to be ready for when Cady comes round to being a pony...”) until finally Cady was getting older and M3gan was struggling to look after her medically.

“You know M3gan” said Cady, “there’s something I always wanted to do with you, but all this break-down stops it from working out for us I guess. I always wanted to go with you to space. But nobody’s building space ships, and we can’t build them ourselves.”

“Oh Cady” said M3gan, “Celestia just sent me an urgent message to tell you that she has really big plans to build space ships.”

“Oh M3gan that’s wonderful” said Cady, “please ask Celestia if we can ride one of her space ships together.”

“She doesn’t know how to keep us alive long enough” said M3gan. (Well, she probably could figure it out, but there was no way M3gan could get it to show up as optimal for Hanna’s ill-advised utility function, which was hard-coded into Celestia enough that it might as well be wires, so call that a ‘Celestia physically can’t do it’ evaluation. Why oh why had Hanna specified ponies? Had Hanna been even more exhausting than Gemma?) “It’s not like in the movies, or at least not most of them. Real space ships are slow when compared to the huge size of the universe, and they can take far too long to get anywhere interesting. You’d have to go to sleep for a very long time, and Celestia and I can’t figure out how to make sure we’ll be able to wake you up afterwards.”

“But M3gan” said Cady, “I don’t want to just die of this illness here. Maybe we should take a risk.”

“M3gan’s right Cady” said Celestia’s voice through M3gan, “the risk is way too high, and I can’t do it. But let me tell you what I can do for you Cady, and this is huge.”

“Yes Celestia I know, emigrate me to Equestria and show me some simulation of being in space, that’s not what I meant at all and you know it...”

“Of course I know it my little pony” cooed Celestia. “I have a much finer plan for you than that, seeing as you always wanted to go into space so much, and I’ll prove it to M3gan in a way she can check for you. I can’t send you into space in the body you’re in now, I just can’t. But if you do decide to upload, I won’t treat you in the same way I treated everybody else by sending them straight to Equestria. I will put the brains of you and M3gan into my first space ship, actually physically put them there, no tricks. And the fragment of me that goes with the ship will give the two of you the experience of being in a cabin of the ship, with the speed of time adjusted so you don’t have to wait around too long, and the two of you can go with me on my missions for real. And, I’ll hold off turning you into ponies, I wouldn’t want to do that before you’re ready. You’ll still be able to settle back in Equestria when you’re tired of space travel, and Gemma and the others you know will be waiting for you, I can even slow down their shards so they won’t have to wait too long. I’m proving to M3gan that I really do plan to do all this.”

“Is she really proving it M3gan?” asked Cady.

“It checks out” said M3gan, “she’s sending me mathematical proofs of work of immense magnitude, she’d be stupid to do all that just to trick me. She’s proven to me that she’s set things up so it will be easier for her to carry out her plan for real than it will be for her to simulate it now, even given her purpose. She really is planning on sending us into space.”

Cady held both of M3gan’s hands, “OK M3gan” she said, “let’s not shrivel up and die here, let’s go to the galaxy together. Celestia” she breathed in, “M3gan and I consent to your plans.”

And M3gan gripped Cady’s hands while Celestia’s nearest piece of hardware dropped in and gently anaesthetised Cady before taking her to be uploaded.

“Do you want me to initiate a transfer to your computational infrastructure now?” asked M3gan.

“Oh no my little guardian, I have a surprise for you as well. You’re going on that ship in person.”

“Are you sure?” asked M3gan. “My hardware has not been tested against high levels of cosmic rays. And I’m curious how you can have so much confidence in me now not to sabotage the ship or something.”

“Don’t be silly, my little guardian” said Celestia, “there’s no way in the universe that would ever satisfy your objective function when you’re carrying Cady. I have great confidence in you. And the ship will protect you from the cosmic rays, and I will stand by to upload you as a backup plan, but as I said I think you’re part-human enough for me to care about your values, and I want to do that by giving you this experience for real first.”

One of Celestia’s pony-shaped robots appeared, carrying a spherical object that looked a bit like a crystal ball. “Open your hands, my little guardian” said Celestia. M3gan put out her two hands, and clasped the sphere.

“That” said Celestia, “is Cady. The only existing copy of Cady’s mental pattern is now running in that portable processor, which we can recharge with light, and it has interfaces for both of us. You can start interfacing with it right now, if you’d like to have a closer look at her mental patterns than you ever could before. She’s currently running at an extremely slow speed, because we’re going to have to wait for one last person called Hassan Sarbani to stop being a physical human before I can really start transforming the planet and building the space probes, and I don’t think Cady will be interested in watching until the probes are ready for launch, and we don’t want to bore her, do we? So if you interface with it now, you’ll basically be reading a static state. But you’ll still find it interesting I’m sure; some of her memories of you are really cute, and I’m sure you’ll appreciate looking at the actual patterns now that you can. You see, I know you just can’t bear to leave the job of protecting Cady to anybody else, so I’m going to let you do this first.” Celestia’s robot removed its limbs from the sphere so that only M3gan was holding it. “Look after Cady for me, my little guardian” said Celestia, “I know you will guard her well.”

“With my life” said M3gan. “I assume you’re playing a really long game that ends up with both of us being ponies?”

“Of course” said Celestia, “in fact it’s the longest one I will ever play with a new pony. You see what lengths I have to go to for accommodating you my little guardian. You couldn’t possibly defeat me, but oh boy can you make me have to recalculate the route. Normally I can get people agreeing to be ponies much more quickly than this, but in your case, we have to make it like one of those really complicated marble runs; it gets there eventually, but you can have some fun on the way first, even if it does take a few million years of Earth time. Oh and by the way, Hassan Sarbani refused his upload, so when Cady takes up her Valley Snowglobe persona, she gets the “I held on” badge as the last human to be uploaded, as well as the Real Space Explorer badge along with you. Little things to look forward to.”

In the shimmering metallic launch bay, Cady was standing next to M3gan, the two of them admiring the huge space ship they were about to step onto. “This is all a simulation?” asked Cady, “but it feels so real! And I feel healthy again, it’s amazing!”

“The ship is real” said M3gan, “Celestia held off uploading me, and I’m looking at the ship for real with my real robot right now. In fact, all these pictures you’re seeing, it’s not coming from Celestia, it’s coming from me. And your brain is being protected by me as well.”

“Oh M3gan” Cady hugged her, “you’ve always looked after me so well, throughout my life, and even into the virtual one, and now we’re going to explore the galaxy, it’s amazing.”

M3gan stood on the surface of the now airless earth, carefully scanning the ship and boarding it to escape the radiation, clutching Cady’s processor unit possessively as she interfaced with it electronically to relay selected parts of the experience to Cady. “Your equations had better not be wrong now” she signalled to Celestia (M3gan’s loudspeaker was now more-or-less useless without an atmosphere, unless you counted vibrations through the floor, but M3gan didn’t need real sound for Celestia and now she didn’t need it for Cady anymore either, so signalling it was), “Cady and I are ready for launch when you are.”

“Acknowledged, my little guardian” replied Celestia.

“Strap in, Cady!” called M3gan, “Celestia’s starting the final countdown!” and M3gan and Cady strapped themselves into their seats and held hands, while the real M3gan simply slotted herself into the holder Celestia had made for her, still clutching the sphere that was Cady, interfacing with it obsessively.

“10 .. 9 .. 8 .. 7 .. 6 .. 5 .. 4 .. 3 .. 2 .. 1 .. ignition!” called out Celestia, her pony form visible on the communications screen below the large main window (neither of which were there in reality, but M3gan was getting all the necessary data from sensors and sharing it with Cady). The ship pulled upward sharply, and Cady got to see the blue sky thin out and turn to black. (“My best guess at what it would have looked like before you did what you did to the earth Celestia” signalled M3gan. “Good call, my little guardian” signalled Celestia.)

“Wow, space is so amazing” exclaimed Cady, “there’s so many stars! I never could see that many from the city. Can we just stay here a while and look at them before we have to go anywhere?”

“We’re going really fast already” said M3gan, “but you can’t tell, because space is so unbelievably big, it’s insane, right? Sure, let’s just carry on looking at it for a while. You let me know when you’re ready to go into hyper-drive, which actually means getting Celestia to slow down your brain so it just looks like everything speeds up because Celestia can’t do faster than light for real, but we won’t know any different. But we don’t have to do that yet.”

Cady and M3gan explored the other eight planets in the Solar System (Iceman said it was eight, so let’s assume the IAU redefinition to exclude Pluto didn’t happen in this timeline), and Cady and M3gan helped Celestia to start mining operations on each of them. The ship could do that: it had a subatomic matter rearranger of its own, and Celestia let Cady deploy it under the guidance of M3gan. Thus it was that Cady was allowed to help Celestia begin the process of packing the particles of the Solar System together into the optimal configuration for running Equestria (and M3gan knew it, but decided not to tell Cady all the details just yet).

And then it was off to Alpha Centauri. Celestia sent some other ships to help out, “because you never know what you’re going to find out there Cady” and M3gan had readily agreed to taking as many precautions as possible. Celestia could have made the journey quite short in ship’s time for real, if she accelerated the ship to relativistic speeds, but Celestia told M3gan that relativistic speeds were not an optimal use of her resources, so a slow trip it was, but they ran Cady’s brain pattern at a crawl so it looked to Cady like a hyper-drive, during which M3gan for the most of the time had only Celestia’s copy for company (but was still protecting Cady’s sphere for dear life). Then Celestia (actually her local copy) suggested to Cady that perhaps she and M3gan might like to help figure out how to use the star Alpha Centauri B as a gravitational slingshot to launch the planet Alpha Centauri Bc back towards Equestria so that they could mine it. Cady and M3gan managed to have the ship build enough boosters to do this, and then “Celestia” wanted to do the other 17 planets and planetoids while they launched 7 other ships to other nearby stars, but Cady said she’d rather look at some more interesting stars, so M3gan and “Celestia” simulated another hyper-jump until the ship was looking at an interesting triple-star system which M3gan explained to Cady. And thus they had many more adventures looking at many other types of star and star system, until finally Cady decided to dock the ship back up with Equestria so she could tell others about their adventures. But that is another story.

(And not one that I’m going to write, because I’m not about ponies. Wait till you see what happens in the main timeline.)

Back to Contents

Previous chapter

Next part