5 upvotes, 1 direct replies (showing 1)
View submission: New Management for Pushshift
Thanks for posting Jason, and thanks for all your work over the years.
Do you know if the NCRI team is planning to make any substantial changes to how pushshift runs? From how removals are processed, to whether they will implement API tokens and charge for higher levels of access. There's also the long list of bugs in the top comment here[1] that need addressing.
Comment by Stuck_In_the_Matrix at 23/02/2023 at 21:13 UTC
4 upvotes, 1 direct replies
1. Thanks for the reminder on the list of bugs in that submission. I'm going to take time out tomorrow and this weekend to address as much of the low hanging fruit as possible and involve some of our other engineers on the larger issues (but from looking at some of them, I should be able to make a decent dent in the bugs listed).
Your question about API tokens and pricing tiers deserves a more formal reply involving more of our leadership team but I can say this -- Pushshift will continue to provide the research community with free access to our most popular API endpoints like Reddit while eventually charging for-profit and other organizations that require enhanced access and/or higher rate limits to Pushshift API endpoints.
At some point we will have a key management system / API tokens. Removals are, at present, processed manually but we are training additional people to make that process smoother and faster. Long-term goal will be to automate the process completely.
Let me know if that answers your questions -- I didn't want to get into specifics without conferring with the rest of the team but we should have more details for you and others soon.