https://www.reddit.com/r/TheoryOfReddit/comments/2jkz7c/mod_tool_sockpuppet_detector/
created by jmdugan on 18/10/2014 at 04:22 UTC
50 upvotes, 10 top-level comments (showing 10)
I'm moderating a recently exploding sub, with 1000+ new subscribers per day in the last few days.
for some time now I've wanted a tool:
I want to be able to put in 2 different users into a web form, and have it pull all the posts and history from public sources on both of those users, and give me a rank-ordered set of data or evidence that either supports or refutes the idea the two accounts are sockpuppet connected.
primarily: same phrases, same subs frequented, replies to themselves, similar arguments supported, timing such that both are on at the same time or on a very different times of the day.
I want a "% chance" rating with evidence, so we can ban people with some reasonable evidence, and not have to go hunting for it ourselves when people act like rotten tards
does anyone know if this exists, or anyone who might be interested in building it?
Comment by [deleted] at 18/10/2014 at 04:34 UTC
16 upvotes, 1 direct replies
sounds like you need a super advanced bot, /r/requestabot might be able to help with that. I dont think such thing exists afaik.
Comment by [deleted] at 18/10/2014 at 06:20 UTC*
25 upvotes, 2 direct replies
^(I can only assume the sub you're speaking of is /r/ebola. Just wanted to say it.)
This is so creepy. I was thinking of this exact thing a few hours ago. I do a lot of database work and make a lot of reports that do comparisons like this, though not usually on a 1:1 basis. More like a grid of results. Lead-generating software, that kinda thing.
I have a plethora of ideas by which you could compare user's data, but I've also got a fundamental problem with it used as a tool as you've described.
If you want to ban a user, ban that user. No mod needs an excuse. That's how the system works.
But you're looking for an "evidence-bot" to justify your actions that you *already wanted to take*, and that's not how 'evidence' works. You say it here:
I want to be able to put in 2 different users into a web form..
So you already suspect these two users, and now you want evidence to back it up. They're apparently not breaking other rules, else you'd ban them for that. The problem with calling this 'evidence' is that you could make an app say anything you want. The only reason to do this is to 'avoid argument', but the argument just becomes the percentage itself. Where did it come from? Why this ratio, and not that?
I mean if it is so *blatantly* apparent as to make you think you need to automate it, surely you could do it yourself at least *once*. Open a spreadsheet, download the two suspect user's data from the API and compare it. If it's a big problem, surely it wouldn't take long to gather evidence of such a thing. Any reasonably accurate percentage is going to be based on *a lot* of data any way. If it's not, it wouldn't *be* accurate.
That's all besides the point though: the fact that you're going to manually enter two users to compare shows a glaring bias, or at the very least a huge risk of it. You say it here:
.. so we can ban people with some reasonable evidence..
You don't *need it*. Just ban them. You're looking to build a robotic 'sockpuppet' to act as your scapegoat.
That's ironic, and kinda fucked up.
Comment by shaggorama at 18/10/2014 at 12:36 UTC*
11 upvotes, 1 direct replies
I don't have the time to build this for you, but I have thought about making something similar myself and can give you a few metrics that would be useful. This way, if you get in contact with someone actually motivated to make this (really wouldn't even be that hard): you can make a more concrete request.
Collapse all of a particular users comments into a single "super document." Convert this document into vector representation by counting the occurrence of each word in the document, removing words that appear in a list of "stop words"[1] (such as 'the', 'an', 'or', etc). Scale word occurence relative normal word usage on reddit by collecting a random corpus of comments from r/all/new (a "background" corpus to help you understand what normal word usage on reddit looks like) and using the TF-IDF[2] transformation for your "document vectors." Then calculate the cosine[3] between the two vectors as your distance score. Values close to 0 indicate more similar users. Calibrate this test by performing it against randomly paired users selected from r/all/new to identify the typical distribution random cosine similarities on reddit (i.e. to determine a meaningful "these users are way too similar" cutoff).
1: http://www.ranks.nl/stopwords
2: http://en.wikipedia.org/wiki/Tf%E2%80%93idf
3: http://en.wikipedia.org/wiki/Cosine_similarity
For each comment you collect from a given user, identify which sub it came from. Do this for both users. Determine which user has the smaller number of unique subreddits visited. call this U1. Calculate a modified jaccard similarity[4] for the two users subreddits as (number of unique subreddits the tow users have in common)/(number of unique subreddits commented in by U1)
4: http://en.wikipedia.org/wiki/Jaccard_index
For each comment from each user, extract the "parent_id" attribute which identifies the comment they were responding to. Also extract the id of each comment/submission (which will need to have the appropriate "kind" prefix appended to it) created by each user. Calculate the intersection of user1's parent_ids with user2's comment/submission ids. Do this for both users separately, and report both the raw counts and as a percentage of that user's comments.
For a given user, extract the "created_date" timestamps of all their comments/submissions. extract the hour component from the timestamp and calculate an activity profile for the user. It will look something like this[5] (this plot is broken down by day of the week, but I don't think you need to get this granular). Do the same thing for both users and overly their plots. If you just want a numeric score, scale their profiles so each data point is a "percent of overall activity" instead of a raw count of comments/submissions posted that hour, and then calculate the mean squared error between the two users activity profiles. A lower error means they are active at very similar times. I don't think this is necessarily a good approach and you're probably better off doing this comparison via visual inspection.
5: http://unsupervisedlearning.files.wordpress.com/2012/09/vectorized-daily-acitivty-graphic-rdc.png
This is a really tough one. Like, a *really* tough one. I think there are a few simpler approaches that can give you the gist of this. a) construct a report on the top N most frequently used words by each user, ignoring stop words. b) Use text summarization to extract the N sentences most representative of all of each users comments. There are many free tools available for automating text summarization, but if you or your bot creator want to do it from scratch, here's a tutorial[6] for an easy approach, and here's an article[7] going into more detail. These approaches won't give you a score, but they will help you understand what these users tend to talk about.
6: http://engineering.flipboard.com/2014/10/summarization/
7: http://www.cs.cmu.edu/afs/cs/project/jair/pub/volume22/erkan04a-html/erkan04a.html
You didn't ask for this one, but I think it's important. Use the same approach as I suggested for comparing subreddit occurrence and extend that to submission ids for the comments you collect (and also each user's submissions). Additionally, given that there does exist overlap in the two users posting to the same submission, calculate the smallest time delta between the two users activity on submissions in which they both appear for all submission in which they appear together. Flag all of these submissions for more detailed investigation and calculate the mean shortest delta. You should also do something similar for the "replies to themselves" analysis: calculate the mean time it took for one user to respond to the other, given that they respond to each other.
Again, this is tough. The problem is that to really calibrate this score, you need known cases of sockpuppetry. But we can use outlier analysis as a proxy. For each of the above analyses that spits out a score, concatenate all the scores into a vector. Grab random users from r/all/new and calculate a score vector for each random pair of users so you have a distribution of these score vectors. Calculate the mean and estimate the covariance matrix for this distribution. Call these your "baseline" statistics. Now, when you have a pair of users you are suspicious of, calculate their "score" vector as above and calculate the mahalnobis distance[8] of the score vector relative to your baseline distribution to give you a score of how much of an outlier this pair is relative to what you observe at random. Pro-tip: augment your baseline by continuously scraping random pairs of users and building up your dataset. Scraping users will probably be a slow process, but the more data the better. So when you're not using your tool to investigate suspicious activity, set it to scrape random users so you can build up your baseline data. For any random user you pull down, you can permute their data against all of the other random users you've scraped data for (NB:*random* users. Don't add your "suspicious" users to this data set).
8: http://en.wikipedia.org/wiki/Mahalanobis_distance
Happy Hunting!
-- Your friendly neighborhood data scientist
Comment by c74 at 18/10/2014 at 06:55 UTC
30 upvotes, 0 direct replies
Trying to sort out sock puppets and people that upvote themselves/alternate accounts is not something that is easily identifiable as a moderator on reddit. The admins won't give that kind of access to mods... but in my experience they will give yes/no/questionable in most cases if you message them.
If you recognize user x and user y for arguing about whatever issue... you might be taking reddit too seriously.
Political agendas, marketers, creative writers, trolls... it's all here.
Comment by [deleted] at 18/10/2014 at 06:42 UTC
3 upvotes, 1 direct replies
[deleted]
Comment by MartialWay at 18/10/2014 at 05:16 UTC
5 upvotes, 1 direct replies
You could make many a bot that would give you a "% chance" rating. None we would be as accurate as good moderator familiar with the sub, personalities, politics, and writing styles.
Comment by karmabreak at 18/10/2014 at 16:16 UTC
2 upvotes, 0 direct replies
Reddective (I don't have the URL and am on mobile) will show all that data you need and you just have to compare and judge yourself.
Comment by ky1e at 19/10/2014 at 00:13 UTC
2 upvotes, 0 direct replies
Here's some data I can think of that will help you make a case for spotting alts:
Comment by vvyn at 18/10/2014 at 09:05 UTC
1 upvotes, 0 direct replies
A user analysis between posting history, frequent words and time active is doable. There are already tools to extract that for a single user.
But it really won't be as conclusive as having concrete proof like vote manipulation that the admins can look into. The % you're looking for is just supporting a hunch but still doesn't make it a reliable piece of evidence.
Comment by [deleted] at 18/10/2014 at 14:43 UTC
1 upvotes, 0 direct replies
The only real way to know is to learn their written tics. Unless they're using usernames like werebear1 and werebear23, I certainly don't join and participate on the same subreddits when I make new accounts. And when I'm trolling, I always make sure to change something about my argument and language.
