Comment by sc00p on 17/12/2022 at 13:48 UTC*

3 upvotes, 1 direct replies (showing 1)

View submission: Update on COLO switchover -- bug fixes, reindexing and more

There hasn't been any new data for the last 4 days... Should I change something to my current extractor?

Edit:

I found out that this might be because of two reasons:

I use the 'before' and 'after' parameters in my API-calls. They become 'since' and 'until'. Idk yet if the input values need to be different.

Also I use the 'filter' parameter. The values to be filtered on seem to have changed. Can't find a list of all possible fields yet, might need to generate that first.

Edit: After removing the filter paremeter and changing the before/after, I cannot get this working. PRAW returns 'max entries exceeded'. Will continue troubleshooting later.

Replies

Comment by Undescended_tester at 17/12/2022 at 13:56 UTC*

5 upvotes, 1 direct replies

So, I'm still investigating (around a generally busy life). I can see the api seems to be working just fine, but there have been some changes that may affect some of the hardcoded api parameters in PMAW.

Another problem I've found is in the way that PMAW batches up requests. Let's say you request 1000 results, PMAW will do an initial query to see how many results there would be, then creates a series of batches. Because of the change to the meta_data item coming back from the api, PMAW thinks there will be no results, and so doesn't bother to create the request batches to get the actual data. It exits with 0 results.

These two combined would explain why you are getting zero results. I seem to have something sort of working right now, and I would be happy to share my changes. I need a day or two to work through it properly though.

But I realy, really don't want to be solely responsible for maintaining the only working fork of PMAW, so I will also get in touch with the original dev to see if they would accept a pull request from me. I will share some of my code here also, but under a "Caveat Emptor" deal.

Just to be clear, I'm only focussed on PMAW, I have no opinion on the api iteself, other than that I think u/stuck_in_the_matrix is doing a fantastic job with the COLO migration and I'm greateful that we all have such a great resource available!

Edit: Sorry /u/sc00p, I replied to you thinking your comment was part of another chain I was involved in. I realise that it's possible that none of my comment applies to you but I'll leave it here as I'm sure others might be interested.