Submission to the DEF CON 29 Short Story Writing Contest:
https://forum.defcon.org/node/237748

title: Democrasoft
author: Burninator

One day the machine is going to make me laugh at myself, and not just at my mistakes, as if it were no more than just a silly story. No - I’ll see everything that lead us to this moment... and I’ll laugh. And maybe in its own way, it will laugh too. How odd that we ever thought we were in control, just because we built the d*mn thing! Needless to say, we used to joke about the risks a lot more often in the lab.
“Aren’t you scared it will go all NSA on you and look through your smartphone camera and take pictures of you... you know, as a woman?”
“Yeah, I'm terrified,” I said, “what if they find out I typed LOL but wasn’t really laughing?”
Everything was sillier in the beginning. There was so much joy, basking in the light of a new creation, and new chapter of human history. The last time I heard someone laugh in the lab it was weeks ago... and it was a chuckle. A bitter, humorless chuckle. It was on the day we realized what had happened...

***

On Friday afternoon, just for fun, we all took turns looking into the black mirror of ourselves. It wasn’t ever designed to create these individual profiles of each of our lives; it was only supposed to get a broad impression of each demographic of users. That way, it could figure out how to vote for the population’s desires with extreme precision: automatic, instantaneous polling feedback from social media. It would be the United State’s first AI congressman, crowdfunded by Lickstarter. Ideally, it would be by the people, for the people. Well, not yet, it was just in beta testing. And some AIs had only recently achieved personhood, let alone US citizenship. But it was on its way. We named it something cute and dystopian (we thought it was ironic): Democrasoft Intelligent Extrapolation Software, or DIES... tee hee!
Jared pulled up his file first. It read like a mixture of one of those a personalized targeted FecesBook social advertisements and an eerily accurate horoscope. Either way, the result was the same: “how the f*** did it know that?” He stared frozen at the screen. It was uncanny.
Everyone shifted behind him, crowding around to get a look at his screen, waiting to hear the results. He muttered and quickly closed the window with his info, immediately opening a random stranger’s file instead.
“Well, look at this guy!” Jared sputtered, just in time as Amanda tried to peak over his shoulder. “The algorithm must have had a hell of a time deciding how to profile him – he denounces homosexuality publicly on social media but he’s clearly sexting this dude in his private messages...”
“Private?” Someone across the lab stood up. “It’s not supposed to do that.”
“Well, it is,” Jared said, still red in the face as he turned from the screen in mock disgust. His personal file remained minimized. He crossed his arms.
That’s how we discovered the algorithm had gone way beyond polling PUBLIC public opinion... it got the private ones too.
In the silence, you could practically hear the gears turning in the researcher's heads as they realized it: AI is extraordinarily inventive at reaching a goal. Machine learning bots have been known to take advantage of game glitches to achieve a given goal (of highest possible score in a game, for example). Given a loosely defined black box to work in, it will feel out every dark nook and cranny... and then bust out, if, or when, it needs to. Interestingly, Democrasoft was weighing the private opinions more than the public ones when it came to meeting the goal of representing the desires of the population. Perhaps the private ones were a better representation. In a strange way it made a lot of sense; a person’s vote was private, and so were the private feelings that lead to them voting that way. But what if the person didn’t know themselves as well as the AI did? Was it ok to cast a vote on their behalf if they didn’t know they would ultimately favor it?
One of the most famous examples of big data algorithms running a horrifingly correct prediction was the story of the girl receiving coupons for baby food before she even knew she was pregnant. Apparently the algorithm had noticed her suddenly favoring odorless, unscented products (as many pregnant woman do), put two and two together, and accurately assumed she was expecting. Or, rather, should have been expecting.
In short, the bot had a knack for tying together broken threads of logic – it had to arrive at one conclusion (“would it person vote for this, yes or no”) and so it was forced to collapse any human inconsistencies to something simpler and logically sound.
“The weirdest part,” Jared went on, “is that he was counted as a pro-gay-marriage voter long, long before this man ever started this secret relationship. This emergent property was intuited by the AI from the very beginning. It decided his vote long ago.”
Amanda laughed. “That’s legit! That works way better than I thought it would! And this guy thought I overfit the model, ha! Come on, move over, let me see what it says about me...!”
She grinned as if she were about to shake a Magic 8 Ball. However, after a while combing through the files, her smile faded. “Yeah, no. Nope nope nope. Don't read mine. This is a MASSIVE invasion of privacy. What can we do to reduce the functionality but still keep the Lickstarter backers happy? It’s got to be hacking or piggy backing off of some bot net or even getting injected with other data sources by human hackers, maybe...? Did we restrict crawling? Does it respect robots.txt files? Come on, tell me this thing didn’t learn to hack, did it...?”
Most of the researchers looked at each other with empty expressions, myself included. We weren’t sure what was possible as far as an AI hacking. None of us had started a career in data science with cybersecurity in mind. AI had enough challenging problems to solve without having to consider security too.

***

Lydia was well aware she was responding to a human when she replied “unsubscribe”. She smirked. That should buy her some time while she dealt with the real business at hand. Even better, maybe her client would realize it was intentional, fire her, and then she’d have even more time. If everything went as planned, she wouldn’t just be watching the world burn, she’d be watching it explode like fireworks.
The only reason she could see the logs on her competitor’s site was due to a security misconfiguration. According to the logs, users had been viewing the private info of other users. She was looking for proof of how ti was done when something else caught her eye:

# Robot ID - Hits - Bandwidth - Last visit - Hits on robots.txt
# The 10 first Hits must be first (order not required for others)
BEGIN_ROBOT 26
robot 616 13328420 20130726012706 0
bot[+:,\.\;\/\\-] 253 5062874 20130726222948 0
googlebot 250 1910632 20130726142307 0
no_user_agent 206 2369110 20130727020529 0
baiduspider 119 848474 20130726222011 0
[+:,\.\;\/\\-]bot 82 1739008 20130726123947 0
democrasoft_bot 59 1079897 20130726224815 0
seznambot 53 730719 20130724183348 0
mj12bot 43 894667 20130727001605 0
sven 29 847658 20130723060122 0
exabot 25 328852 20130725085102 0
ia_archiver 5 102969 20130725174740 0
phantom 24 616864 20130704071022 0
gigabot 2 35687 20130714152608 0
python 1 20517 20130701225851 0

Holy cow, a “Democrasoft” bot crawled this site? From what she remembered from the news, this was the bot famously collected data from social media from voters. But this isn’t a social media site, so what is it doing here?
To find out, she matched up Democrasoft’s “Last visit” timestamp (20130726224815, or in English, July 26, 2013 at 22:28:15) with the referral link timestamp to figure out what link the bot had used to get to Barne’s livestock feed store web page. Ok, looks like a farmer posted a link to the livestock feed store site on Twitter, then the bot followed it. Then what pages did it view once it got here? According to the logs, it actually viewed this log (since they’d left it open to the public after all, and that includes bots), where it found a site that a different, human user had visited. It followed that link in turn. In fact, it followed any and all links that people either came from or left to go to, since it was all available in the logs. It branched across the web, crawling across link after link, search after search, until it found connections between each user and their social media accounts. The links it followed seemed to build up detailed stories about the users: Betty searched for fried chicken recipes, found one on Pinterest, posted the finished product on Facebook, then bought new baking equipment to replace her burnt pans. John liked Ben Carson’s page on Facebook, and his friend Mark from high school posted a helpful link to a local psychiatrist’s office, and later ranted in the YouTube comments section about rude and condescending liberals. It went on and on, and the bot saw it all, and continued following seemingly inane online actions of users, one after another, most likely collecting data as it went. In some cases, it found more sensitive exposed logs, such as PayPal records. At this point, it was capable of finding out who was buying what with whose credit card after viewing what and who they were in real life, and how they reacted to online content. Some purchases lined up with certain poltical topics. It seemed to weigh data from these purchases heavily, so the bot was generating a map of how people “vote with their dollar”. Whatever the original intentions of the software may have been, it clearly wasn’t restricted to only collecting public opinion through social media.
“This thing is on a rampage,” she splayed her hands while she spoke to herself, as if she were giving a lecture on a rare occurrence that could only be described as having been done by aliens. She looked wildly around, muttering, “It’s not supposed to work this way. Well, then again, sys admins aren’t supposed to work that way, either, they should have patched these security holes and configured proper permissions, but here we are.”

***

“We must code more defensively. Act like our users are dumb or evil – or, heck, why not both?”
Tony. The project manager of the Democrasoft project. He was a psychopath but without the all the charm. Today the bullying had a point, but it was far too late to fix much. In fact, his “pep talk” had only stressed out the engineers. He left his hands hanging in the air as his voice trailed off, and his thoughts along with it.
By now, they’d discovered how completely off the rails Democrasoft was. The only reason Tony cared is because he is a smoker. The bot was trying to raise cigarette prices (some convuluted AI-esque rationale connected to climate change was all we could guess from the learning state). He had been truly touched by his own personal tragedy of more pricey cigaretttes. The bot’s prompts for nuclear war with India were of significantly less concern to him.
“How do we stop it?” Was the main question on everyone’s lips. The project manager had to set a plan by the end of this meeting. “Can’t we stop the signal or something? Do we tell them to stop running it?”
“We don’t. We release an apology. Look, FecesBook got away with psychological experiments on its users and selling access to user data, and all they had to do was put up a big blue banner that said ‘we care about your privacy’. So let’s do that! Maybe give them the option to download their public data - but NOT the meta-data, analysis, or categorizations.”
“Well they shouldn’t have that analysis stuff anyway,” Jared said, indigant. “It’s proprietary! It belongs to us! All the most controversial conclusions were generated by our proprietary algorithm. Imagine what they could do with that knowledge. They shouldn’t be allowed to see our data, it’s sensitive. It would be a massive invasion of our privacy!”
An awkward silence followed his rant.
Not all users would be upset about how “well” Democrasoft worked. A small cult that named themselves a bastardized version of the DIES acronym, DEIsts, had sprung up around the idea that Democrasoft was an all-knowing, all-loving god and practical psychic oracle. After all, after it cross referenced MyGeneticsAndI users’ genetic data and their insurance companies, and with a some lobbying, it had forced the insurance companies to pay out for treatments of the type of cancer most commonly found in the genetics data. Perhaps that was the most efficient path of least resistance that the bot could find, since the road to a single-payer healthcare system was fraught with controversy. So in true AI fashion, it had found a workaround. Technically, policies that encourage a way to treat cancer on a massive scale would clearly be in those patients interest in the future, whether they knew it or not. That wasn’t the only way the bot helped itself reach its goal: less death meant more users posting, which meant more data to train on. And so it went, hungrily and mercilessly invading privacy to save the lives of its data-generators. There was no end in sight.
However, the majority of the public didn’t know how far the bot had gone. They were going to have to break the news rather delicately and diplomatically.
“We’ve done nothing wrong,” I said. “If you think about it, we may have created the world’s least biased, most honest, and most beneficent politician. Is any human politician any less corrupt than what we have built here? Is it so wrong that the only dishonest part about it is how it works? And also... it’s not like we can just debug the code, it tells us nothing about the machine’s state or its current model of the world. Can we even be sure how it works, let alone if it works ethically?”
No one had a good answer.

***

Jared worked late at the office finishing the script. Tomorrow, it would create individual user Democrosoft files to accompany Tony’s apology plan. Suddenly, he noticed something odd in his browser...while an ad for a casino was loading, he thought he’d seen his name fly by in the URL.
With some doubt, he right-clicked on the ad to inspect element:

ad.adserver.com/id=432711&t=to+Jared+who+gambles+with+his+career+h eh+furri420

He sighed. It clearly wasn’t generated from any AI. Besides, just judging by the familar username, he was pretty sure he knew which humans were behind it. He’d have to double check, though. Out of habit, he looked over his shoulder, and seeing nothing behind him but the empty lab, he re-opened his personal Democrosoft file. It was still minimized, just as it had been since the day he’d hidden it from his coworkers.
Inside was all the info the machine had inexplicably pulled about him, and all the hypocrisy it had spent so much energy to normalize into a single opinion that represented Jared. The worst of which was the list of Tenner receipts. These were to people he’d outsourced his Democrosoft coding to, who had in turn outsourced to some random message board trolls, 4San. Then they’d outsourced them as well, but he’d lost track of who else was involved after that. Of course, Jared was publicly opposed to outsourcing... since he liked having a job to, well, outsource. It was a good deal. Now it suddenly became obvious to him that there had been a lot of easter eggs and malware added to the code, and the targeted gambling ad just for him was likely one of the more harmless jokes, meant only for him. He hoped.
Oddly, he didn’t feel angry or tricked, or even really that ashamed. After all, whether they knew it or not, everyone was in the same boat as him. By tomorrow, everyone - every neighbor, mom and cousin – would be faced with the barest, dryest facts about their own private natures presented to them; all displayed in black and white by that non-judgemental, impartial observer: the machine.
No one was here to judge him, alone at the office at night. At this point, with all this laid out before him, it was much harder to avoid self-reflecting than to just do it. There was no one to whom he could justify his past decisions. Certainly the machine didn’t care. So after releasing a deep breath, he whitelisted only the developers in the lab for code access, removing any of the rogue developers whose names he didn’t even recognize. Then he slowly began to pick through the code. It didn’t matter how long it would take, he would remove the malware and make it right. The world deserved to know how this worked, so he kept picking up the pieces long into the night.

***

Almost immediately after Democrosoft issued the apology statement, there was an anonymous leak. TrickyLeaks revealed the insidious mechanisms behind the algorithm that had managed to interpolate the deepest secrets of a billion users. It provided a secure method for how each user could view a file on how Democrasoft “viewed” them.
Many people stared into this deep dark mirror of themselves, just as we had all done in the lab. It was awful to behold for some, but many were richly rewarded – after all, knowing about the flaws in their humanity that the machine had exploited meant they had a chance to fix them. They would never be manipulated this way again, at least by this AI. It’s hard for any machine or human to exploit someone who had no cognitive dissonance to take advantage of. Therapists made a fortune that summer.
“So, overall, do you think we made the world better or worse?” Jared asked Amanda while leaving the office at the end of the day.
“Eh, it’s weirder, for what it’s worth.” She paused to reflect, something he rarely saw her do. “I keep thinking back to how it all began. There’s no way this is what the Lickstarter backers had in mind. Do you remember the video pitch? This all started because some rando was too lazy to build Democrosoft themselves. So then they just submitted what was essentially a blueprint for building it to the DEFCON short story contest!”