AmputatorBot v3.0: Better bot

https://www.reddit.com/r/AmputatorBot/comments/i1wruh/amputatorbot_v30_better_bot/

created by Killed_Mufasa on 01/08/2020 at 18:14 UTC*

24 upvotes, 3 top-level comments (showing 3)

*Hi everyone! I'm glad to announce the release of AmputatorBot v3.0: Better bot.**

AmputatorBot is a little over a year old now, and although I've released multiple versions since then, the core codebase has always stayed essentially the same. But overtime, the codebase became obsolete and more spaghetti-ish. Additionally, I wanted to improve the quality of both the front-end and back-end. **I ultimately decided to rewrite AmputatorBot from the ground up**. That was challenging, but allowed me to finally fix all the things that had been bugging me for a long time. And with fairly good results:

AmputatorBot finds about 70 AMP pages more now every day
**AmputatorBot's success-rate is now about 96%, up from 89%**

Without further ado, here's the way too long changelog (sorry about that!):

New functionality

**New canonical-finder methods, bringing the total amount of methods to 9(!)** * Google manual redirect: Sometimes, Google shows an Redirect Notice for AMP pages, recognized by `url?q=`. This new method is able to find the canonicals by following the redirect - AmputatorBot.com example[1] * Google JavaScript redirect: Other times, Google automatically redirects users through JavaScript, recognized by `url?`. AmputatorBot's scraper doesn't run JavaScript, this new method has a dirty but working workaround to fix this - AmputatorBot.com example[2] * Bing original URL: Likewise, AmputatorBot can now scrape for the originalUrl element on Bing AMP pages - AmputatorBot.com example[3] * Schema mainentity: A lot of (news) websites use the Schema framework[4]. It has a tag called mainentity, often containing canonicals - AmputatorBot.com example[5] * Twitter redirect page title: You might have seen them before, they look something like this: `https://t.co/L2xLf3my3Y?amp=1` , by checking for the page title, we can find the URL and continue as usual - AmputatorBot.com example[6] * Guess-and-check: Users occasionally suggest[7] just trimming of the amp parts of URLs. With very variable results. Sometimes, it does the trick, but way too often it just breaks the page. To counter this, I've added some stuff to check the similarity between articles. If the articles are similar, we can say with some certainty that the guessed canonical is correct. If the canonical contains a `rel=amphtml` tag that points to the original URL, we know it for certain. That's the idea, but the results are simply so over the place that I'm not comfortable with enabling it elsewhere other than in mentions and online. Also, it's an extremely heavy CPU task because you can make a lot of guesses :p I'll continue experimenting, but I don't expect this method to be implemented fully any time soon.
AmputatorBot can now find AMP canonicals too, this is a word I've come up with for situations when an AMP page is no longer cached but still using the AMP framework. These will now get posted if the real canonical can't be found. The comment will include a notice that it is still AMP, but no longer cached.
Log and (empty) data files are now automatically created when running the script for the first time
**DMs are now much more specific**, there are new templates for the following situations: * Success * Error: disallowed subreddit * Error: disallowed mod (used to be merged with error: disallowed subreddit) * Error: no canonicals * Error: problematic domain (domains with known errors) * Error: reply failed * Error: user opted out * Error: unknown
**Bans are now getting automatically documented** and added to the list
**When canonicals are from not 1 but 2 or more domains, all 'alternative canonicals' gets posted**. It looks something like this: `You might want to visit the canonical page instead: www.domainA.com/example - domainB version: www.domainB/example` - AmputatorBot.com example[8]
**AmputatorBot now calculates which canonical is 'best'**. Before AmputatorBot would just return the first successful canonical but sometimes the canonical is wrong, e.g. because of a cookie-wall. Now, all canonical finding-methods are tried and the best option (there are often 3 or more) is chosen and used.
**On** **AmputatorBot.com**[9] **you can now insert more than one URLs in the input-box.** In fact, you can paste entire comments in the input-box now, AmputatorBot will automatically filter out the URLs! So when you e.g. copy a Reddit comment, you no longer have to trim out everything but the AMP URL (no example link here, because spaces are a bit tough for AmputatorBot.com)

Improvements, bugfixes & other new functionality-ish

Changed comment template from `It looks like you shared an AMP page. These often load faster but Google's AMP is a threat to your privacy and the Open Web. This page is even fully hosted by Google(!) ..` to `It looks like you shared an AMP. Fully cached AMP pages (like the one you shared) are especially problematic. These should load faster but Google's AMP is controversial because of concerns over privacy and the Open Web ..` Because I feel this is a fairer way to put things, I don't want AmputatorBot to preach and provide a service, I want AmputatorBot to provide a service and explain why it's doing so. You feel me?
Changed comment template from `You might want to visit the normal page instead:` to `You might want to visit the canonical page instead` because as one user put it: "Who the hell are you to decide what is normal?!"
Changed comment template from `Mention me to summon me!` to `Summon me with u/AmputatorBot`
Removed the article by Chris Graham from the comment template because it is getting a bit outdated / not nuanced enough
Removed the Amp-letter from the FAQ because it is too outdated / not nuanced enough
The new link is to the FAQ, which I've altered a bit again: https://www.reddit.com/r/AmputatorBot/comments/ehrq3z/why%5C_did%5C_i%5C_build%5C_amputatorbot/[10][11]. I would rather link to another article, but I couldn't find an article that is objective, nuanced and up-to-date enough.
AmputatorBot's code is now object orientated, which is huge for code-quality, readability and future-proofing
Combined all check_criteria methods into one configurable method
Added more amp keywords
Changed the way AmputatorBot finds URLs (from regex to Extractor module, which is more precise and automatically updates when stuff changes, because why re-invent the wheel right?)
All URLs (both AMP and canonicals) are now getting checked for validity
Canonical-finding method canurl works again (fixed typo)
When a canonical starts with `/` the protocol and domain gets added
When a canonical starts with `//` the protocol gets added
Messages are now categorized based on their API values rather than the subject
Duplicate URLs will now be filtered out
Made it possible to loop through URLs, or only do one
The entire amputating-process is now saved in temporary objects, making debugging a hell of a lot easier
Massively improved how and which markdown and other artifacts are removed from URLs
Added Bing to the method that checks if an AMP link is cached
Improved logging solutions and made exceptions more specific
The title and status code are now getting checked and logged for issues (such as 403's)
Database logging is now done properly through SQLAlchemy and models, instead of injecting in statements
Added tests to make it possible to test the canonical-finding process using older database-entries
Minor changes to DM templates
When checking if an item is in a list (such as if a subreddit is in a certain list), both get casefolded first to prevent issues with faulty capitalization
Expand NP-functionality to Reddit canonicals
When a comment fails, AmputatorBot tries to see if it is banned, and if it is, the subreddit gets added to the disallowed_subreddit list
Updated README
Updated configuration file, made more things configurable such as the debug level, version number and more and added static links
Website specific: * Updated site to new look (more on that later) * Added optional setting to enable and disable guess-and-check (default: enabled) * Fixed some layout-issues
Updated the subreddit to the new look
.. and I probably forgot some other things

Known issues

~~AMP-canonicals can sometimes result in false positives and other issues:~~ ~~AmputatorBot.com example~~[12] fixed in 3.0.2
The automatically changed disallowed_subreddits list does not update the Reddit version yet
Reddit auto escapes links in the displayed URLs, which breaks stuff

New look, who's dis?

Last but not least (yes this boring post is almost over now), I gave AmputatorBot a new look! The old design was.. not so good. So I updated it a bit:

1: https://www.amputatorbot.com/?q=https://www.google.com/url?q=https://www.aljazeera.com/amp/news/2018/07/ways-israeli-law-discriminates-palestinians-180719120357886.html

2: https://www.amputatorbot.com/?q=https://www.google.com/url?sa=t&source=web&rct=j&url=https://www.bbc.com/portuguese/amp/internacional-53113655&ved=2ahUKEwjd7OOStPbqAhX1HrkGHYUOD78QFjAAegQIAxAB&usg=AOvVaw1DZdguMOXC2J_W9aZgQaYq&cf=1&cshid=1596161289068

3: https://www.amputatorbot.com/?q=https://www.bing-amp.com/c/s/news.yahoo.com/amphtml/blasio-black-lives-matter-protests-115502505.html

4: https://schema.org/mainEntity

5: https://www.amputatorbot.com/?q=https://www.nytimes.com/2020/07/29/business/economy/us-gdp-report.amp.html

6: https://www.amputatorbot.com/?q=https://t.co/L2xLf3my3Y?amp=1

7: https://github.com/KilledMufasa/AmputatorBot/issues/7#issuecomment-657688697

8: https://www.amputatorbot.com/?q=https://www.businessinsider.com/chernobyl-reactors-14-years-disaster-2016-4?amp

9: https://AmputatorBot.com

10: https://www.reddit.com/r/AmputatorBot/comments/ehrq3z/why%5C_did%5C_i%5C_build%5C_amputatorbot/

11: https://www.reddit.com/r/AmputatorBot/comments/ehrq3z/why_did_i_build_amputatorbot/

12: https://www.amputatorbot.com/?q=https://github.com/orgs/ampproject/people