https://www.reddit.com/r/pushshift/comments/135cyzk/update_on_pushshift/
created by Stuck_In_the_Matrix on 02/05/2023 at 06:12 UTC
219 upvotes, 27 top-level comments (showing 25)
Unfortunately the admins have disabled our ingest due in part to my failure to maintain comms with the admins and to answer their questions related to the new terms.
First, I want to apologize to the community for my absence lately. Let me give you a thorough update and address many of the concerns from the Pushshift user community and the Reddit admins. Pushshift joined with the NCRI organization many months ago. NCRI, or the National Contagion Research Institute, does amazing work in identifying disinformation that are spead within social media platforms. NCRI is a non-profit organization that raises money through donations to help raise funds for Pushshift so that we can expand our services for the academic community as well as several government agencies like the FDA that use Reddit data and other data sources to further understand many topics mainly related to health, etc.
NCRI has raised substantial funds to allow Pushshift to expand and grow. Demand for Pushshift API services has increased substantially since I began the project in 2015. Since that time, we've helped thousands of academic universities both big and small to understand and use big data for a lot of different research proposals.
In 2013, I moved back from Denver to the Baltimore area to help my father with everyday tasks since he has suffered from a brain tumor that has grown very slowly, but unfortunately has caused some dementia over time. Around two years ago, he fell and broke his neck and that necessitated the need for me to step up and help him as much as possible. I love my father and he has been a huge influence in my passion for data science and helping society through providing tools for the academic community. Recently, my grandmother on my mother's side experienced issues that left her with dementia and I've been helping my mother deal with health insurance issues, etc. If any of you have ever dealt with medical insurance and long-term nursing care for an elderly person, you probably have experienced some of the frustrations I have experienced.
Just before the 2023 New Year, Pushshift finally made a move to a proper COLO after receiving substantial financing. The move was extremely difficult for me due to having to allocate my time across family while trying to maintain a service used by more than half a million people. I never charged for the service and my income existed solely from donations and occasional contract work very early in Pushshift's history.
Right now, I am disappointed with myself because I have left the community in the dark recently and haven't done my part in keeping up with comms. I will say that this has been the most challenging project I've ever worked on. I literally get hundreds of emails per day, lots of DMs across Twitter, Reddit and other social media platforms and even on Slack where I am a part of many different academic and non-profit communities. I hate to make excuses for my failure to maintain communication and openness with the Pushshift community, however I hope you can understand some of the unique challenges that came along when I was running Pushshift alone and trying to maintain services that were used by so many people. At first it was exciting and challenging but as Pushshift grew, it become extremely difficult just keeping up with emails let alone time for development and also time to help my father.
I want to make things right with the Pushshift community and do my best to turn things around so that you can depend on Pushshift when you need social media data for research, modding or anything else that you do with Pushshift. I want to make a promise to the community that I will personally spend a few hours each week on this subreddit and update everyone on where we are and what we're currently working on. I also want to make a promise to the Reddit admins like /u/lift_ticket83 that our team will reach out immediately to the Reddit admins and make sure we can come to an agreement on making sure we follow the new terms of service in good faith. Basically, I'm asking the community for forgiveness and another chance to show you all that I am still very invested in this project and I will do anything it takes to make sure all current technical / bug issues are addressed quickly in the next few weeks.
I will be speaking with the NCRI team to address this failure in comms so that it doesn't happen again. There were other people assigned with the task of reaching out and monitoring this subreddit and for whatever reasons that didn't happen as it should have.
Comment by No_Confidence5452 at 02/05/2023 at 06:29 UTC
35 upvotes, 1 direct replies
You are doing amazing work, don't be hard on yourself. We need you and pusshitft!
Comment by x647 at 02/05/2023 at 06:47 UTC
29 upvotes, 1 direct replies
Apologize for nothing. Life called and you answered and did what you needed to do.
Do what you need to do, you'll have lots of support, thanks and respect coming to you.
Comment by Amndeep7 at 02/05/2023 at 08:44 UTC
12 upvotes, 1 direct replies
Caring for family with dementia or other debilitating diseases is difficult in all sorts of ways that folks who haven't done so will never understand. Trying to get competent, responsible, considerate nurses and caregivers is pure luck. Dealing with the various institutions/agencies and insurance is maddeningly frustrating. I'm sure your family appreciate the time and effort that you put in immensely. However, do take care of yourself as well!
W/r to reddit and pushshift, good luck.
Comment by -Archivist at 02/05/2023 at 13:20 UTC
9 upvotes, 2 direct replies
What interesting timing.... I think we're just heading into a future in which services like PS simply aren't allowed to exist so It'll be interesting to watch how this plays out.
I've been suggesting for the last 2 years there needs to be tooling to rebuild static, consumable reddit archives from the raw PS data. However with the terms of ingest and the ability for users/subs to opt out without the transparency of who/which had done so PS is no longer a complete archive...
/u/Stuck_In_the_Matrix sorry this is the mess you're dealing with, if I can help with anything at all you know where to find me.
Comment by Watchful1 at 02/05/2023 at 15:19 UTC
8 upvotes, 0 direct replies
Sorry to hear about your family, I know how hard that is.
Have you considered getting some people to help with maintenance? There are a bunch of members of this subreddit who have both the knowledge and the time to at least help run something like pushshift.
Also could you open source your ingest code?
Comment by ExcitingishUsername at 02/05/2023 at 06:51 UTC
8 upvotes, 1 direct replies
Appreciate all the hard work, and hope there will be a way to continue it.
Will Pushshift be able to continue to archive content from NSFW communities, or will Reddit be forcing you to eliminate that from your service too? A lot of subs use access to that data for spam control, statistics, research, or even simply to exclude NSFW posters from spaces used by minors, and Reddit has thus far been pretty silent on whether they'll allow such legitimate uses after the API changes.
Assuming Reddit doesn't shut you down, will any progress be made on fixing the major search bugs and breakage that make the service largely useless for searching by author or query text? The majority of our tools using PS have not worked for many months, due to most searches returning either vast numbers of results not matching the query entered, or nothing at all.
Comment by Btan21 at 02/05/2023 at 09:12 UTC
7 upvotes, 1 direct replies
I hope things get better for you and your family JSON. Caring for the elderly and infirm is difficult and I have experienced it too with my grandparents, so I understand your difficulties.
Thanks a lot for your work!
Comment by shiruken at 02/05/2023 at 15:03 UTC
6 upvotes, 1 direct replies
I guess we'll find out whether the API blacklisting was due to the lack of response or if that was just an excuse and they were going to block Data API access regardless.
Comment by f_k_a_g_n at 02/05/2023 at 18:29 UTC
6 upvotes, 1 direct replies
Pushshift has been an invaluable tool for many for years, and all for free. You can tell from all the replies, even if they come off frustrated, how important it has been.
That said, I can empathize what you're dealing with. You should keep in mind that family is more important than anything, and you don't owe any of us anything. If you decided today to just delete the service and all the data, so you can focus on your life and family, there is nothing wrong with that.
Also, make sure you take care of **yourself** too.
Comment by iKR8 at 02/05/2023 at 09:19 UTC
5 upvotes, 1 direct replies
Best wishes to your family and more strength to you.
Comment by dniepr at 02/05/2023 at 10:03 UTC
5 upvotes, 1 direct replies
Welcome back!! No apologies needed, I can't imagine being in your place; and also what you have done with pushshift is very very very cool , I just wanted to say that.
Comment by Bot-yMcBotface at 02/05/2023 at 18:41 UTC
4 upvotes, 0 direct replies
Hi more power to you!
You have done nothing wrong, putting family first was the noble thing to do.
Secondly, reddit would have acted the same. Reddit and pushshift were never equal. They _granted_ you privileged access as long as they saw an advantage. I always wondered why they shared their data-treasure. This data has become very valuable.
There might be some bargaining in telling them, that the torrents still stay up with everything up until now and if everything fails you can open source your code.
Reddit. Will. Be. Scraped. The question is only, if the scraped data stays open.
​
Thanks for everything!
Comment by ProlesAgnstPaperHnds at 02/05/2023 at 08:41 UTC
5 upvotes, 1 direct replies
No apologies required JSON. I am very thankful for this massive contribution to science and research you have already made to date. Anything that follows is a victory lap. Hope your family is doing alright under those difficult circumstances, take care
Comment by criticool-realism at 02/05/2023 at 13:39 UTC
2 upvotes, 0 direct replies
You and Pushshift have been an asset to the academic community. Really hoping Reddit can appreciate this as you work to reestablish comms.
Comment by Twinkies100 at 02/05/2023 at 15:12 UTC
2 upvotes, 0 direct replies
Family always comes first, glad you're back. Will Pushshift continue to work via donations/crowd funding apart from NCRI to cover the API costs after new policy comes into effect?
Comment by Slopz_ at 03/05/2023 at 04:18 UTC
2 upvotes, 0 direct replies
There is absolutely nothing wrong with you valuing your family more than your work. Hopefully things get better for you, your family, and pushshift.
Good luck!
Comment by Daddy_William148 at 03/05/2023 at 17:31 UTC
2 upvotes, 0 direct replies
Thanks for your hard work. It is disappointing. I am sorry this has happened. I am glad you were able to help with your father
Comment by grejty at 02/05/2023 at 08:43 UTC*
3 upvotes, 2 direct replies
I hope you guys can resolve this. I believe it is somehow important to them as it is to us. I appreciate you and your efforts Jason.
I rely on Pushshift for my academic Bachelor project and shutting it down right now, 3 weeks before the deadline, is kinda ruining the whole work.
Comment by Starkrossedlovers at 04/05/2023 at 11:52 UTC
1 upvotes, 0 direct replies
I’m confused are you the only one in pushshift looking at Reddit?
Comment by borg_6s at 04/05/2023 at 14:06 UTC
1 upvotes, 1 direct replies
Does Twitter ingest still work or did Elon shut that off too?
Comment by yes_u_suckk at 05/05/2023 at 03:39 UTC
1 upvotes, 0 direct replies
You did the right thing.
As much as I like Pushshift, I would let it burn if I was in your position so I could take care of my family first.
Comment by ShadowOfHarbringer at 05/05/2023 at 05:15 UTC
1 upvotes, 0 direct replies
Would it be possible for you to OpenSource Pushshift if you determine that you cannot support it any more?
Maybe some alternatives will spring up this way and your life's work will not be wasted.
Have you thought about it?
Comment by notamoonshot at 05/05/2023 at 19:20 UTC
1 upvotes, 0 direct replies
Agree with the comments, thank you for bringing this tool to the community, we truly appreciate your work
Comment by MobTwo at 06/05/2023 at 00:16 UTC
1 upvotes, 0 direct replies
Your father is more important. If I were you, I would have done the same thing.
Comment by bildramer at 07/05/2023 at 07:51 UTC
1 upvotes, 0 direct replies
This wimpy narcissistic blog post of a status update makes the result of any "negotations" with the admins very predictable. NCRI is a joke.