created by lift_ticket83 on 01/05/2023 at 17:12 UTC*
0 upvotes, 110 top-level comments (showing 25)
Howdy Mods,
In the interest of keeping you informed of the ongoing API updates, we’re sharing an update on Pushshift.
TL;DR: Pushshift is in violation of our Data API Terms and has been unresponsive despite multiple outreach attempts on multiple platforms, and has not addressed their violations. Because of this, we are turning off Pushshift’s access to Reddit’s Data API, starting today. If this impacts your community, our team is available to help[1].
1: https://reddithelp.com/hc/en-us/requests/new?ticket_form_id=14868593862164
On April 18 we announced[2] that we updated our API Terms. These updates help clarify how developers can safely and securely use Reddit’s tools and services, including our APIs and our new and improved Developer Platform.
2: https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/
As we begin to enforce our terms, we have engaged in conversations with third parties accessing our Data API and violating our terms. While most have been responsive, Pushshift continues to be in violation of our terms and has not responded to our multiple outreach attempts.
Because of this, we have decided to revoke Pushshift’s Data API access beginning today. We do not anticipate an immediate change in functionality, but you should expect to see some changes/degradation over time. We are planning for as many possible outcomes as we can, however, there will be things we don’t know or don’t have control over, so we’ll be standing by if something does break unintentionally.
We understand this will cause disruption to some mods, which we hoped to avoid. While we cannot provide the exact functionality that Pushshift offers because it would be out of compliance with our terms, privacy policy, and legal requirements, our team has been working diligently to understand your usage of Pushshift functionality to provide you with alternatives within our native tools in order to supplement your moderator workflow. Some improvements we are considering include:
We are already reaching out to those *we know* develop tools or bots that are dependent on Pushshift. If you need to reach out to us, our team is available to help[3].
3: https://reddithelp.com/hc/en-us/requests/new?ticket_form_id=14868593862164
Our team remains committed to supporting our communities and our moderators, and we appreciate everything you do for your communities.
Comment by RunDNA at 01/05/2023 at 17:32 UTC
187 upvotes, 9 direct replies
Does this destroy tools like removeddit? Because I use that website constantly.
Comment by MyPrivateGH at 01/05/2023 at 20:00 UTC*
184 upvotes, 3 direct replies
It's cute that you can't show us "user-deleted content" for "lawyercat reasons," but when I have someone with 1,600+ karma post in my subreddit, and every single other post or comment that person has made has been deleted, I use those tools to see if they have posted in multiple R4Rs across the country on the same day. This is the most reliable way for me verify if I'm dealing with a karma farmer.
I also use these tools to verify the authenticity of an account. I have caught minors who tried to post in my NSFW sub because I recovered posts that had their *real* age. Spammers and scammers rarely stand up to a cursory search using a Pushshift app. In many cases, these accounts are ridiculously fluid about things like age and gender. But Reddit offers no protections to moderators to prevent these accounts from changing any detail they'd like, as long as they delete their postings.
You are removing a vital tool with absolutely no replacement ready, and that is absolutely unfair to those of us who are volunteering to moderate the content on your platform. Moderation tools, at this point, should be moving forward, but Reddit is about to throw the moderators **YEARS** backwards, while the scammers, spammers, and bots continue to find new and exciting ways to spam our subreddits—which the moderators take the heat for if we fail to adequately protect the sub.
I already spend several hours a day moderating my subreddits (and I think I do a damned good job). If Reddit continues to impede my abilities as a moderator, I see no reason to keep the subreddits running. I'll take them all private, and let them die.
Comment by shiruken at 01/05/2023 at 17:38 UTC
94 upvotes, 3 direct replies
Mod of r/pushshift here, I've crossposted this to the subreddit for further discussion: https://www.reddit.com/r/pushshift/comments/134uvzz/reddit%5C_data%5C_api%5C_update%5C_changes%5C_to%5C_pushshift/[1][2]
2: https://www.reddit.com/r/pushshift/comments/134uvzz/reddit_data_api_update_changes_to_pushshift/
Comment by [deleted] at 01/05/2023 at 17:32 UTC
408 upvotes, 5 direct replies
[deleted]
Comment by dequeued at 01/05/2023 at 17:59 UTC
230 upvotes, 4 direct replies
The maintainer of Pushshift has been dealing with some stuff recently and their availability has been affected.
None of these "improvements" address the reasons why we are all using Pushshift for both subreddit bots and general moderation bots such as BotDefense, repost detection bots, and others.
Reddit is doomed to be overrun by even more reposted content, both submissions and comments, from malicious bots because they do not care about the Reddit terms of use and they will definitely not have a problem sourcing old content without Pushshift.
I think we will need to reconsider whether it is worth continuing BotDefense now that it has been crippled by this awful decision.
for lawyercat reasons
Smile at our jokes while we screw over our volunteer moderator labor force?
I am not laughing.
Comment by teanailpolish at 01/05/2023 at 17:18 UTC
154 upvotes, 1 direct replies
There are so many uses for pushshift and ban flow/removal reasons are at the bottom of that list
Comment by Stuck_In_the_Matrix at 02/05/2023 at 05:39 UTC
76 upvotes, 4 direct replies
Hey u/lift_ticket83 -- I apologize for the communications gap and not being responsive when trying to contact us. There was some internal issues and confusion on who was supposed to handle comms while I deal with family issues. I'm happy to jump on a call with you to discuss where we are deficient and how we can meet your API terms.
As you know, Pushshift is used extensively in the academic community and I have always made a good faith effort to honor user requests when a user makes a request. In fact, we now do this daily.
Could you give me some contact information so we can set up a meeting with our team and your team to discuss the best path forward?
Thanks again and I apologize for the the issue with comms.
Comment by 13steinj at 01/05/2023 at 17:18 UTC
119 upvotes, 2 direct replies
In summary, it seems you all don't care about the fact that mods and others use Pushshift for *good* to combat spam and malicious bots, and don't want to provide similar functionality for cost reasons.
I can't wait until this backfires. Enjoy your future IPO though.
Comment by Moggehh at 01/05/2023 at 17:15 UTC
194 upvotes, 4 direct replies
This just made modding 100x harder. Thanks.
Comment by Kvothealar at 01/05/2023 at 17:24 UTC
96 upvotes, 2 direct replies
Can you first natively implement the tools we depend on for moderating before nuking something almost all serious mods rely on?
This is going to seriously affect our workflow.
Comment by shiruken at 01/05/2023 at 17:44 UTC*
46 upvotes, 1 direct replies
**Providing permalinks to user- and admin-deleted content in User Mod Log for any given user in your community.** Please note that we cannot show you the user-deleted content for lawyercat reasons.
As has been previously discussed, this is still insufficient. Continued access to the user-deleted content is incredibly important because otherwise users can just delete offending content to try to hide their tracks. Yes, we can add notes (natively or via toolbox) but that requires additional effort for moderators.
Comment by Redditenmo at 01/05/2023 at 17:17 UTC
123 upvotes, 3 direct replies
Thank you for killing off a useful tool many of us use daily.
Comment by Majromax at 01/05/2023 at 18:12 UTC
42 upvotes, 2 direct replies
our team has been working diligently to understand your usage of Pushshift functionality to provide you with alternatives within our native tools in order to supplement your moderator workflow. Some improvements we are considering include:
One critical pusshift.io feature that I will miss is a comprehensive, historical list of submissions to the subreddits I moderate, and it sounds like there is no replacement in the works.
Currently, Reddit only provides an extremely limited submission history for a subreddit, of a hundred pages or so. I've previously used year-long periods to train Automoderator filters, using my own (compliant) API access to process comments into a training/test corpus when given a list of submissions.
I don't care much about the permanence of the historical record since a vanishingly small fraction of submissions are edited away or deleted; I do care about finding out what threads were posted six months ago.
Comment by justcool393 at 01/05/2023 at 17:15 UTC
73 upvotes, 3 direct replies
to be honest, it seems like the changes to the API terms were seemingly primarily targeted at Pushshift from the beginning
Comment by JonAce at 01/05/2023 at 17:31 UTC
75 upvotes, 2 direct replies
Comment bots celebrating right now.
Comment by hansjens47 at 01/05/2023 at 17:47 UTC
32 upvotes, 0 direct replies
I'm surprised this post seems to tie development of mod tools directly to third party tooling.
This adds pressure on developers, like of /r/toolbox etc., to go dark so tools get developed natively in a timely manner.
--------------------------------------------------------------------------------
Clearly admin aren't prioritizing mod tools sufficiently if going dark impacts mod tooling in any way whatsoever.
Needing third party tools to do basic community management is a pretty critical vulnerability for running the site no?
Comment by Watchful1 at 01/05/2023 at 17:59 UTC
71 upvotes, 1 direct replies
Why did you do this now instead of notifying everyone you would and letting people try to migrate tools? I have a meeting with you guys literally this week to talk about all the ways I use pushshift.
Not to mention I specifically asked if you had talked with the pushshift owner and I was told you had, so I anticipated more time.
Comment by abrownn at 01/05/2023 at 17:49 UTC*
92 upvotes, 3 direct replies
Put up a historical/research API endpoint with hyperspecific parameters like pushshift and gate access to institutions/big mods/phone interviews/NDAs/etc (hell, I'll even PAY for access!), otherwise you've just kneecapped mods only way to combat platform manipulation of every kind.
Edit: Research institutions, governments, digital forensics outfits all relied on Pushshift for historical data and the ability to craft hyperspecific requests -- have you spoken to any of them? How can they study or help your platform at all anymore?? How can we? Since day one, people have had to kludge extensions and sites built around your own because of its complete lack of functionality. Killing Pushshift and not having a replacement is insanely on-brand for ya'll.
Comment by Itsthejoker at 01/05/2023 at 19:39 UTC
59 upvotes, 1 direct replies
Why are you turning this off with ZERO warning to us? Why do you not have replacement tooling for us to migrate to? This is critical functionality that you are pulling out from under us, functionality that we rely on to do our *volunteer jobs*, and the best we get is "howdy mods" and "lawyercat" with a side of patronizing tone? This is an incredible slap in the face, especially to those of us who have been working with you all in good faith on detailing what Pushshift means to us. I'm honestly gobsmacked.
Comment by ItalianDragon at 01/05/2023 at 20:08 UTC*
59 upvotes, 1 direct replies
So basically you'll be destroying our moderating capabilities just because you can't manage to get in touch with one guy.
Considering how we routinely weed out spam and whatnot and deal with all sorts of rule-breaking behavior, axing the access to Pushshift ensures that all of those will be absolutely rampant.
You also clearly didn't "hoped to avoid" disruptions with a last minute notice like that. If anything a behavior like this on your behalf is the polar opposite of that.
Given the IPO rumors, it seems that the enshittification[1] of Reddit is well under way...
1: https://mjtsai.com/blog/2023/01/27/the-enshittification-of-all-things/
Comment by 0spore13 at 01/05/2023 at 17:24 UTC
26 upvotes, 0 direct replies
Knew this was coming and it's still extremely painful, this is going to make moderating so much harder.
Comment by TheYellowRose at 01/05/2023 at 18:21 UTC
28 upvotes, 0 direct replies
I just want to say 'fuck this shit' with my whole chest.
Comment by terminus-trantor at 01/05/2023 at 18:45 UTC
24 upvotes, 1 direct replies
I moderate a question and answers type subreddit and we use several tools that allow us to deep dive into past asked questions to find similar already asked questions, to create FAQ pages, statistics, in general go through old posts.
Reddit API as far as I can tell no longer supports search by before and after dates. Pushshift does and was indispensable tool for it. Reddit API also has some limits when going into past but as I haven't used it much I don't remember which ones exactly.
Do you plan to introduce (back) the ability to search by dates, and allow us easy search of old posts and comments by different criteria (subreddit , author, date, etc.)
If you believe such use cases are covered by your API let me know how
Comment by andrewthetechie at 01/05/2023 at 19:19 UTC
27 upvotes, 1 direct replies
Cool, thanks for continuing to make it harder to be a moderator on reddit.
Comment by ArchipelagoMind at 01/05/2023 at 20:12 UTC
25 upvotes, 1 direct replies
Hello.
Is it possible to get an update on Reddit's plans for how academics and non-profit researchers will be impacted by both this and Reddit's API changes.
Pushshift has been ***invaluable*** to the research community. One of the reasons I have been impressed by Reddit is their previous interest in allowing academics and researchers to use Reddit data (directly via the API and pushshift) to conduct research. However, this looks like researchers will be essentially shut out completely overnight, which is really disappointing.
Can someone please give an update as to what we can expect for researchers?