12 upvotes, 3 direct replies (showing 3)
View submission: New Management for Pushshift
Hey everyone -- Jason here. I want to clear the air and help explain some of the changes that have been happening lately. When I started Pushshift in 2015/2016, it was a very small service used by a handful of programmers and also by researchers who wanted massive amounts of Reddit data for research purposes. Since that time, it has grown into a service that gets over 1.13 billion hits per month by over one million unique visitors.
As time went on, I was simply overwhelmed with support requests, adding additional features and just keeping things running smoothly. Literally it was all I worked on for 14+ hours a day and over weekends. I did this while also becoming a primary caregiver for an immediately family member dealing with a major health issue.
I started working with the NCRI non-profit group three years ago and they provided a lot of support behind the scenes. I felt it was a good marriage to keep the community thriving and expanding, so we made more formal agreements to work together and partner with one another.
The Pushshift-Support user is operated by a trusted member of the NCRI group and will help provide support and further communication efforts for the expanding community. It also gives me an opportunity to focus on improving Pushshift and advancing the original cause that I always stood 100% behind -- to give the research community better access to social media data to help keep social media communities engagement more transparent for researchers to better understand since disinformation is a constantly growing problem for society.
I am happy to answer questions but this is really me Jason. I'm happy to take a call with one of the moderators to prove my identity or to confirm via Twitter, etc. -- I have not been hacked.
Pushshift will continue to provide free access to researchers. Money provided via Patreon will continue to be used to further the development of Pushshift. However, if donations are made via Paypal to NCRI, NCRI is a registered 501-c3 non-profit which can be used for taxation purposes if donations are made via the NCRI paypal account. Money made through that account will be used to improve and support Pushshift services.
Again, I apologize for the lateness in responding but the past couple months have been overwhelming on a personal level as we have moved to a COLO, hired additional engineers and have worked to continue to improve the health and robustness of Pushshift services while I have had to deal with personal caregiver issues. I want to thank the community and I'll check back again shortly to answer any questions.
Comment by Watchful1 at 23/02/2023 at 21:06 UTC
4 upvotes, 1 direct replies
Thanks for posting Jason, and thanks for all your work over the years.
Do you know if the NCRI team is planning to make any substantial changes to how pushshift runs? From how removals are processed, to whether they will implement API tokens and charge for higher levels of access. There's also the long list of bugs in the top comment here[1] that need addressing.
Comment by safrax at 23/02/2023 at 21:12 UTC
3 upvotes, 1 direct replies
I'm happy to take a call with one of the moderators to prove my identity or to confirm via Twitter, etc. -- I have not been hacked.
I live in the DC area and would be happy to buy you a beer if you ever had the time.
Comment by shiruken at 23/02/2023 at 21:33 UTC
1 upvotes, 1 direct replies
I am happy to answer questions but this is really me Jason. I'm happy to take a call with one of the moderators to prove my identity or to confirm via Twitter, etc. -- I have not been hacked.
To be fair, I was far more suspicious when m.vea turned into Reddit's biggest ~~NFT~~ Collectible Avatar aficionado.
past couple months have been overwhelming on a personal level as we have moved to a COLO, hired additional engineers and have worked to continue to improve the health and robustness of Pushshift services
How big is the team now? Perhaps it would be beneficial to host an AMA here with the team to answer the community's questions.