💾 Archived View for gmn.clttr.info › sources › geminispace.info.git › commits captured on 2023-12-28 at 15:37:08. Gemini links have been rewritten to link to archived content

View Raw

More Information

⬅️ Previous capture (2023-09-08)

➡️ Next capture (2024-02-05)

🚧 View Differences

-=-=-=-=-=-=-

Commits

67000d477416746245879dc165a5554a442e5e07

Author: René Wagner <rwa@clttr.info>

Date: Mon Nov 13 08:09:25 2023 +0100

Patch

Message: new exclude

8038f9de64eede72626d03fb6c1a964dff09f97a

Author: René Wagner <rwa@clttr.info>

Date: Mon Oct 16 19:25:52 2023 +0200

Patch

Message: switch back to release version of jetforce

b99b541e900cb19ee1e2d295702a162df33115eb

Author: René Wagner <rwa@clttr.info>

Date: Mon Sep 18 10:49:52 2023 +0200

Patch

Message: switch links to geminiprotocol.net

f5ae878d3dbecb37a6f917e7ff736fa315aaa2d5

Author: René Wagner <rwa@clttr.info>

Date: Sat Sep 16 13:59:40 2023 +0200

Patch

Message: allow robots.txt served as any text/* mimetype

e103214e870beefeb2e219c0104a685e82945207

Author: René Wagner <rwa@clttr.info>

Date: Tue Aug 22 10:14:39 2023 +0200

Patch

Message: new exclude and typo-fix on robots.txt usage doc

8feea97f7c065bb9a6b77828cc1cd8e7bc4cf84d

Author: René Wagner <rwa@clttr.info>

Date: Sat Aug 12 20:57:57 2023 +0200

Patch

Message: docs: robots.txt clarification

b2dfd8f3b26f53382350fe87a088e855479c9445

Author: René Wagner <rwa@clttr.info>

Date: Sat Aug 05 21:12:19 2023 +0200

Patch

Message: new excludes and README update

958574c5b641de21f5098982d905d5a34d5b4402

Author: René Wagner <rwa@clttr.info>

Date: Sun Jul 30 17:07:58 2023 +0200

Patch

Message: pubnix-aware fetching of robots.txt

0ba2ce59e34652eae39c4c4e815d78919e176291

Author: René Wagner <rwa@clttr.info>

Date: Sun Jul 30 16:33:42 2023 +0200

Patch

Message: make 'newest hosts' URI absolute including scheme

cfb30cb4beeba71b2da0033439a4e88ace130561

Author: René Wagner <rwa@clttr.info>

Date: Sun Jul 30 12:07:00 2023 +0200

Patch

Message: rework fetchable_url generation

54803f61105bc2da0251fcb4006b25bc9b75c9dd

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 29 20:55:55 2023 +0200

Patch

Message: fix typo in robots.txt

09f4969216e259eb57323cdb1f4d5031529acbec

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 29 10:12:25 2023 +0200

Patch

Message: update news

5fbaa742a0054bd3551ee86e3666c29011ce56a8

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 08 20:42:50 2023 +0200

Patch

Message: new excludes, fixes for infra scripts

f9842ba22a94788fba7d28193c0b1ce09d517983

Author: René Wagner <rwa@clttr.info>

Date: Thu Jun 08 16:50:34 2023 +0200

Patch

Message: fix backlinks search

511030699cd5280a0024885860e365137eb69200

Author: René Wagner <rwa@clttr.info>

Date: Wed Jun 07 20:02:01 2023 +0200

Patch

Message: further simplify URI handling and always use lowercase host part

35ebd8b71ef112145b3ebc6604f8c2e4dd0365af

Author: René Wagner <rwa@clttr.info>

Date: Sun Jun 04 19:41:54 2023 +0200

Patch

Message: fix some issues when deleting outdated pages

faf87301d736167b8b8e6f14cf4fcf2e0308b5cc

Author: René Wagner <rwa@clttr.info>

Date: Sun Jun 04 19:41:35 2023 +0200

Patch

Message: revert changes made for search.clttr.info

beb4b4e038d43d2c859bc7f6d23bc468469c0122

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 03 14:12:17 2023 +0200

Patch

Message: fix search

e93be0dbd849ccbed4a5c4b811adf9ca27a2d3b3

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 03 14:07:13 2023 +0200

Patch

Message: list filtered URIs in sorted manner

517ce30c5535fa30bdf58b7a24eb4e22c978a8ef

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 03 14:06:52 2023 +0200

Patch

Message: remove unused dependency 'twisted'

f54c132fef2bcc80f037227634b1894f5f12e8b1

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 03 11:45:03 2023 +0000

Patch

Message: update pyproject.toml

f0d22402d2a8262d8e2418bba5aa5aa4c157304b

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 03 11:31:40 2023 +0000

Patch

Message: fix deprecation warnings

af66d04b4ce4a2de97f4c32c610c031046853aa4

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 03 08:14:47 2023 +0000

Patch

Message: fix handling of requests without path segment

when a request without a path (to "/" of a host) is sent, we need to

ensure that the trailing "/" is appended.

Otherwise we might end up in redirects and not fetch a capsule at all.

fd4b8fb39f6ad739744d53bad64167ff9ba473e0

Author: René Wagner <rwa@clttr.info>

Date: Fri Jun 02 11:51:34 2023 +0000

Patch

Message: adjust params to new vps

495732be289f568d102d1c8aa5b78bdb46db6172

Author: René Wagner <rwa@clttr.info>

Date: Tue Apr 25 17:53:08 2023 +0200

Patch

Message: update about due to new donation

085471dc9ab17d935871867cdbc717c6b2d1b8fb

Author: René Wagner <rwa@clttr.info>

Date: Tue Apr 18 20:07:24 2023 +0200

Patch

Message: fix page deletion

c7d06895353747e61ab72c85f2f3446abbed4919

Author: René Wagner <rwa@clttr.info>

Date: Fri Apr 14 12:57:30 2023 +0200

Patch

Message: donation info

83a758a4db4561bf6191fd60810a736f4e85849f

Author: René Wagner <rwa@clttr.info>

Date: Sun Apr 09 10:56:45 2023 +0100

Patch

Message: pull whoosh directly from repo, preparations to move to search.clttr.info

b5fe859d448849993a0ba2b72efff45dd308cd6f

Author: Rene Wagner <rwa@clttr.info>

Date: Thu Apr 06 18:35:28 2023 +0000

Patch

Message: param tweaks for whoosh and sqlite

8ddb511c7be95e3b1bbb0c1b6944b38caecce09b

Author: Rene Wagner <rwa@clttr.info>

Date: Thu Mar 30 18:47:53 2023 +0000

Patch

Message: shell script cosmetics, switch to jetforce master branch

1e7b83f93241a1ffe2fd54236812187d7d5b235c

Author: Rene Wagner <rwa@clttr.info>

Date: Sun Mar 12 11:33:20 2023 +0000

Patch

Message: fix page diff evaluation on recrawl

16fe7694d947472b0a0b60b2bdb57cbed61287ca

Author: Rene Wagner <rwa@clttr.info>

Date: Thu Mar 09 11:32:05 2023 +0000

Patch

Message: fix storing of page content

a8879dd5d3b05f96e5fdae17a3551d5bccd5aa76

Author: Rene Wagner <rwa@clttr.info>

Date: Wed Mar 08 20:31:57 2023 +0000

Patch

Message: split content into separate table

739328b9e5aae6855f1cc92def9872dd5ba50741

Author: Rene Wagner <rwa@clttr.info>

Date: Wed Mar 08 19:02:57 2023 +0000

Patch

Message: update README

5a28dae35d06dbf00398e05584e6cf612951cfdb

Author: René Wagner <rwa@clttr.info>

Date: Sun Mar 05 16:40:33 2023 +0100

Patch

Message: include twtxt.txt feeds in known-feeds page

cfaf86dc24b3390a2a58d4c2c1cb9519abca67ef

Author: René Wagner <rwa@clttr.info>

Date: Sat Feb 11 10:37:28 2023 +0100

Patch

Message: show list of excluded uris

16eb249aa8f40774ef3613644ac8d7193d08f925

Author: René Wagner <rwa@clttr.info>

Date: Fri Feb 10 08:44:46 2023 +0100

Patch

Message: excludes, cosmetics, infra fixes

f4cbbc207f78f4be31259e6a0d5cc9859533c46d

Author: René Wagner <rwa@clttr.info>

Date: Sun Feb 05 16:02:30 2023 +0100

Patch

Message: fix crawl.lock handling

acb4b26df51e2075eecaff69827ba2442cb74390

Author: René Wagner <rwa@clttr.info>

Date: Sun Jan 29 09:48:57 2023 +0100

Patch

Message: update project url for gusmobile and logging settings

b9099d424f6c79a8a5ca550a9e6cbc7d8ee50343

Author: René Wagner <rwa@clttr.info>

Date: Fri Jan 27 15:20:58 2023 +0100

Patch

Message: new excludes & news

f3d2f34fab58a3b1261e3154bba14f07247c54f0

Author: René Wagner <rwa@clttr.info>

Date: Thu Jan 26 21:51:53 2023 +0100

Patch

Message: add new exclude & test

c562254a65a505dc9f5e5c54cb8ee6be810c644a

Author: René Wagner <rwa@clttr.info>

Date: Fri Jan 06 11:55:41 2023 +0100

Patch

Message: fix crawl, revamp crawl & indexing procedure

6ea586ae2380fcdedf5c83d5ee99438370390cad

Author: René Wagner <rwa@clttr.info>

Date: Fri Jan 06 10:51:04 2023 +0100

Patch

Message: update logging settings

2174883dd629450213786b2997ebb7b450fe385e

Author: René Wagner <rwa@clttr.info>

Date: Fri Jan 06 10:09:36 2023 +0100

Patch

Message: update db schema for improved query performance

cd8b669e4d7453d1bc4d49eecb378a386d19db1b

Author: René Wagner <rwa@clttr.info>

Date: Thu Jan 05 20:57:15 2023 +0100

Patch

Message: optimize db index for faster serve startup

c6d2cfc39343f9e07173c98726103585205a2898

Author: René Wagner <rwa@clttr.info>

Date: Sun Jan 01 19:35:44 2023 +0100

Patch

Message: news 2023-01-01

a7fb7dc171dc11fc080952ae9d8a68e43bc2a05f

Author: René Wagner <rwa@clttr.info>

Date: Fri Dec 09 21:42:13 2022 +0100

Patch

Message: remove invalid uris from seed-requests.txt

2fadb34b7487549e9b085dc0943cd923efe1c07b

Author: René Wagner <rwa@clttr.info>

Date: Fri Dec 09 21:19:43 2022 +0100

Patch

Message: only use whoosh multisegments when indexing from scratch

0b4062ed4d72021f1bf2cbbbcb89c86fe7634f01

Author: René Wagner <rwa@clttr.info>

Date: Fri Dec 09 15:56:37 2022 +0100

Patch

Message: update package versions

8d7cf178d861c9b03c0f6bd5ba8bd1883261001b

Author: René Wagner <rwa@clttr.info>

Date: Fri Dec 09 15:52:03 2022 +0100

Patch

Message: use incremental index update as default

dfe2e581d219492491e0cbcefe2de9494e307770

Author: René Wagner <rwa@clttr.info>

Date: Sun Nov 20 12:10:30 2022 +0100

Patch

Message: new exclude, make crawl usw 3 threads

332a9da344f23619cd40ac02e897311f31689f51

Author: René Wagner <rwa@clttr.info>

Date: Sun Nov 20 12:09:42 2022 +0100

Patch

Message: news 2022-11-19

3a1614d2fcf367e8fd8b2c5c0a5da82bff3771e5

Author: René Wagner <rwa@clttr.info>

Date: Wed Oct 26 19:37:46 2022 +0200

Patch

Message: fix index rebuild on first day of month

89cc95e87152cadf957b929939330dffdb6ec1d1

Author: René Wagner <rwa@clttr.info>

Date: Wed Oct 26 19:37:16 2022 +0200

Patch

Message: increase memory for indexer

a691481220a89c92282b9422b7386de16b5ed247

Author: René Wagner <rwa@clttr.info>

Date: Wed Oct 26 19:36:56 2022 +0200

Patch

Message: exclude webgate.geminet.org

d0aa349efae67c340fec377580cbb50c550335a8

Author: René Wagner <rwa@clttr.info>

Date: Thu Aug 25 09:30:02 2022 +0200

Patch

Message: fix link in latest news

83fe8ef7ac639c7c48d02dd4ff8caa30379f6bfa

Author: René Wagner <rwa@clttr.info>

Date: Tue Aug 23 16:47:26 2022 +0200

Patch

Message: properly implement deletion of capsules with outdated crawls

We now look at the whole domain and check if any page of this

domain has been successfully crawled in the last 30 days.

If not, all pages of the domain are removed from raw data and

search index.

closes #35

84c79072c9ed0f803755cfa337936bbd4e35695a

Author: René Wagner <rwa@clttr.info>

Date: Mon Aug 22 17:25:00 2022 +0200

Patch

Message: add donation link

670ac5a1f38565d9b0f36fc5758edde01ca3e4fd

Author: René Wagner <rwa@clttr.info>

Date: Thu Aug 18 10:57:23 2022 +0200

Patch

Message: news 2022-08-18

9de31b2aa50f5c35b13f33eb5506d560f180844b

Author: René Wagner <rwa@clttr.info>

Date: Tue Aug 16 19:38:41 2022 +0200

Patch

Message: fix normalizing of URIs with default port

It seems that urlsplit.port can not be string-compared for some reason.

So we simply always replace the port portion - i assume this is fine

for almost everything out there.

fixes #55

e24313a14c4ad496f418f47c1cd6162373234ef3

Author: René Wagner <rwa@clttr.info>

Date: Sun Aug 14 17:35:35 2022 +0200

Patch

Message: fix test and add additional test for special robots.txt

c2391751e9d842431c403c8b6221827dde4e6a99

Author: René Wagner <rwa@clttr.info>

Date: Sat Aug 06 15:26:27 2022 +0200

Patch

Message: fix -d param for crawl

a1160091007c3ca2fb7cbd56cc645ca982ede014

Author: René Wagner <rwa@clttr.info>

Date: Sat Aug 06 15:10:29 2022 +0200

Patch

Message: update deps

bd5944bdea9ef9f992150670cc9b41768065fd72

Author: René W <rwa@clttr.info>

Date: Sat Aug 06 15:04:39 2022 +0200

Patch

Message: Merge pull request 'Upgrade feedparser to 6.0.10' (#51) from duncan-bayne/geminispace.info:master into master

Reviewed-on: https://codeberg.org/rwa/geminispace.info/pulls/51

881b4afe5b8cbb9ed907f993f26b01ea85d5a572

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 23 08:40:16 2022 +0200

Patch

Message: newest hosts template

b98d105ad776f082da13820994ba4d0e8bd0c99f

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 23 08:37:54 2022 +0200

Patch

Message: show 50 newest hosts instead of 30

a173c6b9b7aebc782bf2d8c9de478fa23bda4225

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 23 08:36:47 2022 +0200

Patch

Message: exclude auragem.space/twitch/

980c6b4aa6c39cc366fe44febc21d64fdbb425d7

Author: René Wagner <rwa@clttr.info>

Date: Fri Jun 10 19:54:05 2022 +0200

Patch

Message: disable search suggestions due to bug

e2ec756bbdcae99d21e7f7157f9e6fb306981dc3

Author: René Wagner <rwa@clttr.info>

Date: Thu May 26 19:42:23 2022 +0200

Patch

Message: move data deletion to indexing

ca95ee48b9c2bfb317202c2500e00b178c9b6666

Author: René Wagner <rwa@clttr.info>

Date: Mon May 16 19:01:52 2022 +0200

Patch

Message: news 2021-05-16

d1dd67d9364b7a2a01cec75ae0edefca90362280

Author: René Wagner <rwa@clttr.info>

Date: Fri May 13 17:39:16 2022 +0200

Patch

Message: make crawl multi-threaded

169d39be7eb10da89a5d8d9bebcc2ab2bf432c0f

Author: René Wagner <rwa@clttr.info>

Date: Mon May 09 21:45:39 2022 +0200

Patch

Message: disable mmap-ing of whoosh index

8d3cdb64054333f937395b0056617754477dbf8f

Author: René Wagner <rwa@clttr.info>

Date: Sun May 08 19:29:48 2022 +0200

Patch

Message: switch SQLite to WAL mode

d71097373e04c47f1364e02153f263476a946104

Author: René Wagner <rwa@clttr.info>

Date: Sun May 08 11:03:00 2022 +0200

Patch

Message: some adjustments

c6e591f09fbf85254ef30e4a018fead35c379e0a

Author: René Wagner <rwa@clttr.info>

Date: Sun May 08 10:20:45 2022 +0200

Patch

Message: updated dependencies, excludes

78926405846bc500693671c29a22e0be3636efbb

Author: René Wagner <rwa@clttr.info>

Date: Fri Mar 25 21:08:51 2022 +0100

Patch

Message: news 2022-03-25

ce98af1db50829edb1531c26cfdea8d48166e605

Author: René Wagner <rwa@clttr.info>

Date: Wed Mar 23 08:38:02 2022 +0100

Patch

Message: tweak whoosh writer settings for speedup

2b6d3c7f671e8151f2c6072c0b5fd2d48fc110ca

Author: René Wagner <rwa@clttr.info>

Date: Sun Mar 20 09:18:44 2022 +0100

Patch

Message: workaround breaking change in markupsafe 2.1.x

relates to #44

602b408933a33087690a4e7c088905d5987b6664

Author: René Wagner <rwa@clttr.info>

Date: Sat Mar 19 18:55:14 2022 +0100

Patch

Message: news 2022-03-19

572a30280e1fb58002557c057f04ace03b64294e

Author: René Wagner <rwa@clttr.info>

Date: Sat Mar 19 18:54:52 2022 +0100

Patch

Message: update deps

f079ae00249ca5cac19852d9d5c681d236a5621c

Author: René Wagner <rwa@clttr.info>

Date: Thu Mar 03 19:34:29 2022 +0100

Patch

Message: fix some typos

163d97f92c49c9a576b0f0bb4035b0cb8763bc16

Author: René Wagner <rwa@clttr.info>

Date: Fri Feb 18 18:37:43 2022 +0100

Patch

Message: add info about redirect indexing

e1d7853bbc6abda3d8c3ec8f59768eb31458c136

Author: René Wagner <rwa@clttr.info>

Date: Sun Feb 06 19:53:28 2022 +0100

Patch

Message: add index for speedup

f0c091b153051a042f4763fd0eb5da97bc41bd27

Author: René Wagner <rwa@clttr.info>

Date: Sun Feb 06 19:13:30 2022 +0100

Patch

Message: some sql adjustments

0b4062fe6319107c42778b324aced2bee6b94444

Author: René Wagner <rwa@clttr.info>

Date: Sat Feb 05 16:17:35 2022 +0100

Patch

Message: precompute feeds and pages

related to #42

823e8d6a2c764cd58aa77b69b3bb2bc9da5720ca

Author: René Wagner <rwa@clttr.info>

Date: Sat Feb 05 10:37:07 2022 +0100

Patch

Message: precompute hosts statistics

relates to #42

015279c14108635a1f7639798d19589537ed5a42

Author: René Wagner <rwa@clttr.info>

Date: Sat Feb 05 10:17:46 2022 +0100

Patch

Message: update deps

55467547c97e8237e18bf8fc46a0e26d60bb953c

Author: René Wagner <rwa@clttr.info>

Date: Sat Feb 05 10:17:36 2022 +0100

Patch

Message: exclude git.skyjake.fi

ae1e797b3b7dcbda724d792d6bcd8b9bb76c864c

Author: René Wagner <rwa@clttr.info>

Date: Fri Feb 04 19:25:12 2022 +0100

Patch

Message: exclude tlgs.one

d03aa6a5969f1cade1e91aa3c9a0384bc0664981

Author: René Wagner <rwa@clttr.info>

Date: Fri Feb 04 09:41:08 2022 +0100

Patch

Message: generic exception handling for page crawling

closes #40, #41

cd9e7c853e636cef6532026cf6a3bd8b84c75cd7

Author: René Wagner <rwa@clttr.info>

Date: Fri Jan 28 13:12:44 2022 +0100

Patch

Message: new exclude: taz.de

70cc8883f8d25f51f99e012dab778cc6d40f2212

Author: René Wagner <rwa@clttr.info>

Date: Tue Jan 25 21:11:58 2022 +0100

Patch

Message: news 2022-01-25

97c0f68c1aeb90d74e93c865730163ad737bfb5a

Author: René Wagner <rwa@clttr.info>

Date: Tue Jan 04 19:57:04 2022 +0100

Patch

Message: news 2021-12-29

3cdc5efc5a5e8015b2f3066a06df3c590884db89

Author: René Wagner <rwa@clttr.info>

Date: Wed Dec 29 10:59:46 2021 +0100

Patch

Message: don't delete excluded pages from the pages table

Or we loose external backlinks to this pages as well

which might be usefull.

a61c96151612747bbcbe6e72380ad792621a5ce0

Author: René Wagner <rwa@clttr.info>

Date: Wed Dec 29 10:57:01 2021 +0100

Patch

Message: update poetry version

acbb7f5e997e6277bebbe4a7c9a157974cb34dd8

Author: René Wagner <rwa@clttr.info>

Date: Wed Nov 24 20:45:10 2021 +0100

Patch

Message: show 30 latest hosts

9ffbed7494556f9d9332261f1f73a17247b77767

Author: René Wagner <rwa@clttr.info>

Date: Sat Nov 20 17:13:32 2021 +0100

Patch

Message: exclude antenna filters

f1accd3f44a9fb0317c6e934ba35cc44c6a0d347

Author: René Wagner <rwa@clttr.info>

Date: Fri Nov 19 16:06:37 2021 +0100

Patch

Message: don't crash on URIs with non-number port

closes #37

570363cf9a3339532bbead6413b6bcb53c4ec377

Author: René Wagner <rwa@clttr.info>

Date: Tue Nov 16 16:21:25 2021 +0100

Patch

Message: update excludes

48af98a6356382e5c6db8ed8ecfdab4101058284

Author: René Wagner <rwa@clttr.info>

Date: Thu Nov 11 18:28:13 2021 +0100

Patch

Message: update contact

ca8bc01a75a08f6b5bff84b13c96280febf67ec8

Author: René Wagner <rwa@clttr.info>

Date: Tue Nov 09 18:41:24 2021 +0100

Patch

Message: dependency update

0163ff34f77330ac440bcec2bc130b72fc8506ba

Author: René Wagner <rwa@clttr.info>

Date: Sun Nov 07 17:27:36 2021 +0100

Patch

Message: cleanup excludes

7d9a46087c1de6d8c59bfb5cd80915e2b5cfb740

Author: René Wagner <rwa@clttr.info>

Date: Mon Oct 25 20:45:57 2021 +0200

Patch

Message: save first_seen_at if a page is created through a link

b30a30afe9cbd757be544d1730b6e47b287aa8a7

Author: René Wagner <rwa@clttr.info>

Date: Thu Oct 14 20:22:12 2021 +0200

Patch

Message: add link to source in geminispace

fa2db540f6a05e5be6423a0e6c4d24950f8fbdf8

Author: René Wagner <rwa@clttr.info>

Date: Thu Oct 14 18:54:12 2021 +0200

Patch

Message: more meta data for index cleanup

b484a4dadc7b58eac2f13188a748f64a83d58a26

Author: René Wagner <rwa@clttr.info>

Date: Mon Oct 11 20:03:08 2021 +0200

Patch

Message: avoid crash when normalized_url is not set

fixes #34

f928815d49189a5fcfa7753ed578ac36532ffc82

Author: René Wagner <rwa@clttr.info>

Date: Mon Oct 11 19:45:42 2021 +0200

Patch

Message: use cronjob for automated start

6eedbd4190b8f1feafb34fcca4007655237fa9df

Author: René Wagner <rwa@clttr.info>

Date: Thu Sep 16 19:53:51 2021 +0200

Patch

Message: some cleanup

- remove some unused code

- remove outdated excludes

- news update 2021-09-15

25cc314490b255581b4a792be6d59a2c0bd3bdd7

Author: René Wagner <rwa@clttr.info>

Date: Mon Sep 06 08:19:03 2021 +0200

Patch

Message: fix broken link to source code

e8d416471813553b1e75ebaefe9bf8ea0dfa9b33

Author: René Wagner <rwa@clttr.info>

Date: Sat Sep 04 09:03:14 2021 +0200

Patch

Message: do not add every single domain to the statistics file

7fa7a7d0fa5923ca93b7b22640228febf4faa6cf

Author: René Wagner <rwa@clttr.info>

Date: Wed Aug 18 17:23:23 2021 +0200

Patch

Message: news 2021-08-18

0f03e4fb66a152294bc9bb6894d4abedbfbf9b93

Author: René Wagner <rwa@clttr.info>

Date: Tue Aug 17 21:00:10 2021 +0200

Patch

Message: some minor changes

- update docs about indexing

- show historical stats in reverse order (newest first)

- some exclude cleanup

faad84dfd50d07846b5113b3c5c9d1813e750eed

Author: René Wagner <rwa@clttr.info>

Date: Tue Aug 10 18:43:19 2021 +0200

Patch

Message: ensure that scheme is given when searching for backlinks

e0fae1c4d66a301181bd9676d3d48c4374f7b4dc

Author: René Wagner <rwa@clttr.info>

Date: Tue Aug 10 18:37:46 2021 +0200

Patch

Message: update 2021-08-07

518e95dc99b4db03673a451d866c9f338ea7ef03

Author: René Wagner <rwa@clttr.info>

Date: Fri Aug 06 16:50:59 2021 +0200

Patch

Message: ensure that seed-requests use absolute URIs

e40712ca9d3297fc755984ff8091c4628b337beb

Author: René Wagner <rwa@clttr.info>

Date: Fri Aug 06 16:41:53 2021 +0200

Patch

Message: more excludes

ffbf174790574623380b494ca921f385fb464340

Author: René Wagner <rwa@clttr.info>

Date: Fri Jul 23 13:11:09 2021 +0200

Patch

Message: implemented deletion of outdated data

- pages that never had any successfull crawl

- pages with the last successfull crawl more than 30 days ago

closes #24

cbe22de43a084b7b2c288d5867cf24bbb0ce8414

Author: René Wagner <rwa@clttr.info>

Date: Tue Jul 20 19:14:39 2021 +0200

Patch

Message: small fixes and doc adjustments

b8eb04a22445fa63d716c1aaf03e0ae4bbf8b636

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 17 19:40:20 2021 +0200

Patch

Message: remove obsolete code

- threads in serve/views.py, serve/models.py and gus/lib/db_model.py

- run_index_statistics in gus/lib/index_statistics.py

f6bd88672ea163d77bc3460ce460afb29b9c5468

Author: Hannu Hartikainen <hannu@hrtk.in>

Date: Sat Jul 17 12:06:19 2021 +0300

Patch

Message: support prioritized robots.txt user-agents

Reimplement the can_fetch() function of RobotFileParser such that it

prioritizes multiple user-agents. Add unit test for said functionality

and set the user-agents this crawler uses to ["gus", "indexer", "*"] (as

they were in the past, though with bugs).

This was heavily inspired by the earlier discussion at

https://lists.sr.ht/~natpen/gus/%3C20210212070534.14511-1-rwagner%40rw-net.de%3E

1ce3f6f92b203861022554d4eac806f32ecf16c7

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 17 12:35:01 2021 +0200

Patch

Message: more excludes and less logging

e47d78ce3055828ef68ff56c4dd81daf8d06699f

Author: René Wagner <rwa@clttr.info>

Date: Wed Jul 14 21:01:05 2021 +0200

Patch

Message: treat schemeless links as non-gemini links

a scheme is mandatory per spec

https://lists.orbitalfox.eu/archives/gemini/2020/003646.html

closes #12

6b5a9f7b4c61d777d5e941ee6b93cf98c27f7410

Author: René Wagner <rwa@clttr.info>

Date: Wed Jul 14 20:56:50 2021 +0200

Patch

Message: remove pikkulog separation

73883455c273b5ccfbd16196ee8889593199ee74

Author: René Wagner <rwa@clttr.info>

Date: Wed Jul 14 08:36:25 2021 +0200

Patch

Message: minor code cleanup in db_model

6e524d1be9a8b87de6156c1f47a8bdd453f1d1f0

Author: René Wagner <rwa@clttr.info>

Date: Wed Jul 14 08:32:13 2021 +0200

Patch

Message: update to some templates

most notably:

remove the footer on pages where it's not useful

39c6540bc6a851f97493bbd8790a22451aa9af57

Author: René Wagner <rwa@clttr.info>

Date: Tue Jul 13 17:20:53 2021 +0200

Patch

Message: remove Search model

We don't store search queries, although not personalized

this is no information we want to have.

e1b3ac8ab4740ef176a3995551bcacae6244588d

Author: René Wagner <rwa@clttr.info>

Date: Tue Jul 13 13:21:28 2021 +0200

Patch

Message: enable 'newest-hosts' and 'newest-pages' sites again

closes #26

7ce66303c32c12552db2dedc7e0ab536f85e0915

Author: René Wagner <rwa@clttr.info>

Date: Tue Jul 13 09:21:06 2021 +0200

Patch

Message: remove raw data from excluded capsules

first part of #24

87d92bbfb37a77d905812d036b79c6f717c5137a

Author: René Wagner <rwa@clttr.info>

Date: Mon Jul 12 21:37:55 2021 +0200

Patch

Message: index text files up to 5 MB

fix flagging pages as indexed

80e589b1d481bb85344c9341bb61d690e58200de

Author: René Wagner <rwa@clttr.info>

Date: Mon Jul 12 19:27:57 2021 +0200

Patch

Message: commit search index only when indexing is complete

unnecessary commits during indexing are time-consuming

remove dead "feedparser" code from crawl

cddbb82dfd62c8e7e2601fd53b46359cdca1bb06

Author: René Wagner <rwa@clttr.info>

Date: Mon Jul 12 16:57:33 2021 +0200

Patch

Message: store document id in whoosh index

a9e9cf27d5460a956444d03cef784da70fb856bd

Author: René Wagner <rwa@clttr.info>

Date: Mon Jul 12 14:58:33 2021 +0200

Patch

Message: some tweaks to indexing

- simplify backlinks counter query

- only count successfull crawled domains as known domains

- increase default root recrawl time

87ef15df2ebeb3bd17e3a69fb296f0ed657c4814

Author: René Wagner <rwa@clttr.info>

Date: Sun Jul 11 19:03:15 2021 +0200

Patch

Message: restructure crawl data

The "crawl" table is now obsolete and removed, all required

information is stored in the `page` table which simplifies

queries and will make data cleanup easier.

All relevant queries have been adjusted to honor this change.

b5bf01a4454e1e904d09499038ef80ea8b5255e9

Author: René Wagner <rwa@clttr.info>

Date: Sun Jul 11 09:05:01 2021 +0200

Patch

Message: remove Crawl table, all info is stored in page table now

9efd819e3e5317965a012192bc1ec2d4fd789515

Author: René Wagner <rwa@clttr.info>

Date: Sat Jul 10 09:08:50 2021 +0200

Patch

Message: don't persist robots.txt over multiple crawls

Instead fetch them again on every crawl run and only

cache for the the crawl session

d4093761e14f5f105f1b51f53cdeac814640b19b

Author: René Wagner <rwa@clttr.info>

Date: Fri Jul 09 22:05:55 2021 +0200

Patch

Message: improve indexing speed via optimized backlinks query

the query to calculate backlinks caused massive delays during indexing.

An unused join to the `crawl` table caused this behavior.

After removing the join, speed is very fast again.

123895e2f0dfff0c35a023e037b98e3c22f12d35

Author: René Wagner <rwa@clttr.info>

Date: Fri Jul 09 17:38:45 2021 +0200

Patch

Message: again a new exclude

86365f71ae36ba3b822eb3f7246ffefb67e54550

Author: René Wagner <rwa@clttr.info>

Date: Fri Jul 09 17:37:39 2021 +0200

Patch

Message: move gusmobile to new home

gusmobile was hosted on natpen's git which is not available

anymore.

The source is now mirrored on src.clttr.info and codeberg.org

e14157666310f39c21852e367cdee4fd0e91a3a9

Author: René Wagner <rwa@clttr.info>

Date: Sun Jul 04 21:49:27 2021 +0200

Patch

Message: update 2021-07-04 & more excludes

a85534a5bf24330f6a22f0dd9a388e4eab9fa152

Author: René Wagner <rwa@clttr.info>

Date: Mon Jun 28 09:31:39 2021 +0200

Patch

Message: additional filter

6a18d99fc17f0540dc79ab8cb99f45cb8251eb4a

Author: René Wagner <rwa@clttr.info>

Date: Sat Jun 26 13:16:35 2021 +0200

Patch

Message: update 2021-06-26

c5bfdafcf56a0ae2ff88208e2aa9c4b4a62ff8f2

Author: René Wagner <rwa@clttr.info>

Date: Wed Jun 16 21:18:53 2021 +0200

Patch

Message: exclude godocs.io

05c5bd7b5d7c749c4fa495f21f9069f61c4f772a

Author: René Wagner <rwa@clttr.info>

Date: Mon Jun 14 09:13:51 2021 +0200

Patch

Message: error handling on page crawl save

acd728e7c407f72a0436f3a35a2658bfea3ce3f2

Author: René Wagner <rwa@clttr.info>

Date: Fri Jun 04 11:40:44 2021 +0200

Patch

Message: update 2021-06-04

d3b1dd8e776f97f4ab33b9500255a0fe6158bfd7

Author: René Wagner <rwa@clttr.info>

Date: Sat May 29 10:56:34 2021 +0200

Patch

Message: more exception handling on link update

3f7c0f84f9d039d4559225c7e2e97585c7fd8bc0

Author: René Wagner <rwa@clttr.info>

Date: Thu May 27 15:24:13 2021 +0200

Patch

Message: fix wrong embedding of excludes

8b004af54d160a78fdb6d261d4e2478483d2c082

Author: René Wagner <rwa@clttr.info>

Date: Wed May 26 13:06:36 2021 +0200

Patch

Message: unify capitalisation of charset in statistics

5c9e5267cf88cff4a3a0db55183fcba822b55c18

Author: René Wagner <rwa@clttr.info>

Date: Tue May 25 22:05:40 2021 +0200

Patch

Message: move exclude definition to own file

closes #18

14c39977247cd992c0a49efc1bd8416a9979a942

Author: René Wagner <rwa@clttr.info>

Date: Tue May 25 21:13:28 2021 +0200

Patch

Message: news 2021-05-25

e0fba80405ff7ea29d1680098031eaee3e165628

Author: René Wagner <rwa@clttr.info>

Date: Fri May 21 21:58:18 2021 +0200

Patch

Message: some exception handling and updated service files

52d2b4c86d365689e6927fb5b8fdfc68985fe70a

Author: René Wagner <rwagner@rw-net.de>

Date: Sun May 16 09:59:42 2021 +0200

Patch

Message: fix last wrong exception in crawl

9b6ef8a0e29b8a6697ba3e12e1f325a838cc2a65

Author: René Wagner <rwagner@rw-net.de>

Date: Fri May 14 20:59:54 2021 +0200

Patch

Message: fix wrong exception handling in crawl

06c0258323f11948f3d2e568d385f066c10fe38e

Author: René Wagner <rwagner@rw-net.de>

Date: Wed May 12 17:46:33 2021 +0200

Patch

Message: update 2021-05-12

9b21f64790e01e55a7e4a135137231abeb443fd9

Author: René Wagner <rwagner@rw-net.de>

Date: Mon May 10 17:41:06 2021 +0200

Patch

Message: rewrite statistics gathering to pure sql

the peewee functions lead to a stupid error

because to much variables are generated

fixes #21

1266d9a93ba139d53c46afb3bcbad708f0ea0ac9

Author: René Wagner <rwagner@rw-net.de>

Date: Sat May 08 21:51:48 2021 +0200

Patch

Message: exception handling on page save

20b092423339fa9fb3291b4705dc6a375a88ba2e

Author: René Wagner <rwagner@rw-net.de>

Date: Wed Apr 14 21:33:27 2021 +0200

Patch

Message: news 2021-04-14

f6c35262887211fc36bed40d4f3301661e625957

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Apr 05 08:07:46 2021 +0200

Patch

Message: delete tmp files of whoosh

61d713038c4fe1c793ac2b8618f694c2f1a2a723

Author: René Wagner <rwagner@rw-net.de>

Date: Thu Mar 25 21:33:31 2021 +0100

Patch

Message: use .fromisoformat for getting timestamp from db

tentative fix for #17

e6faa0e129dfb49380ce4d977d82356d41425f54

Author: René Wagner <rwagner@rw-net.de>

Date: Thu Mar 25 21:10:54 2021 +0100

Patch

Message: various corrections

f5ce631246ac96c7a063d55adb1ddf024b5970b3

Author: René Wagner <rwagner@rw-net.de>

Date: Sat Mar 20 20:58:58 2021 +0100

Patch

Message: hack: index update in separate dir

0b0b33610a729350a379c0ec09ee29fe04e589b4

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Mar 08 19:21:29 2021 +0100

Patch

Message: skip a capsule after 5 consecutive failed requests

This state is reset after the current crawl

closes #16

1dac97f01e9c7260cf2ef84ab4080a4835c86c63

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Mar 08 18:59:55 2021 +0100

Patch

Message: workaround for "index update blocks searches"

2ebc1a844a7b569c8df6a402c3ea9c4b44be701b

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Mar 08 18:59:09 2021 +0100

Patch

Message: news update 2021-03-08

6e54e520144aea538eccf830b2985a41878f8899

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Mar 08 18:51:28 2021 +0100

Patch

Message: Merge branch 'master' of git://natpen.net/gus

c791a758f2e2804279ef964486b03854bf07b4a7

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Mar 05 19:02:58 2021 +0100

Patch

Message: update poetry deps

e691231ec872d4dd241a69cc418eba42c32e967f

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Feb 26 18:52:51 2021 +0100

Patch

Message: gsi specific updates 2021-02-26

8520ec533ce63a745c5dbb1bafc5c23722244f94

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Feb 22 19:06:02 2021 +0100

Patch

Message: robots.txt sections "*" and "indexer" are honored

We no longer use the "gus" section for ease of implementation.

It's probably barely used anyway.

134b7f6c482cdfda95eae80c5d83ae1712fbacea

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Feb 12 08:05:34 2021 +0100

Patch

Message: correctly handle robots.txt

Honor the robots.txt entrys of "indexer" and "gus" as well

as the default * section.

The robot_file_map.p must be deleted on a live instance

after this change has been applied to refetch all robots

files, as previously only empty files have been stored.

64748f085254199e38203f84844201ac88aa27e7

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Feb 12 08:53:20 2021 +0100

Patch

Message: add verbose search to robots.txt

This was missing in the first place.

95e29af321164f011da5dc5a77e3a01e662799ac

Author: René Wagner <rwagner@rw-net.de>

Date: Wed Feb 10 19:05:47 2021 +0100

Patch

Message: Merge branch 'master' of git://natpen.net/gus

af967cc728bc97af217c8813ecb1156b80d30109

Author: René Wagner <rwagner@rw-net.de>

Date: Mon Feb 08 17:43:19 2021 +0100

Patch

Message: add some forbidden URIs & set max_crawl_depth

39edf728476cc455ee5d076b9969a6de35e4924c

Author: René Wagner <rwagner@rw-net.de>

Date: Sun Feb 07 19:11:45 2021 +0100

Patch

Message: remove seed-requests from repo

105f1ca2c60397cc0c17992a0343162f82789aef

Author: René Wagner <rwagner@rw-net.de>

Date: Sun Feb 07 17:48:36 2021 +0100

Patch

Message: Merge branch 'master' of git://natpen.net/gus

e897bc488b64e250c9f666a89accd19c965bac03

Author: René Wagner <rwagner@rw-net.de>

Date: Tue Feb 02 18:38:00 2021 +0100

Patch

Message: update python deps

e1f673fc7da4680567a4540cef42b495a9eea26f

Author: René Wagner <rwagner@rw-net.de>

Date: Tue Feb 02 17:39:42 2021 +0100

Patch

Message: updates geminispace.info 2021-02-02

b119e8b6d8186b069033f6aa1efaf05fdd22034b

Author: René Wagner <rwagner@rw-net.de>

Date: Sun Jan 31 21:08:02 2021 +0100

Patch

Message: introduce systemd-unit for indexer

The indexer is launched by systemd when the crawler finishes.

4edb3fc7d421a10a027aef7edd13c02fa1f30329

Author: René Wagner <rwagner@rw-net.de>

Date: Sun Jan 31 15:04:10 2021 +0100

Patch

Message: gsi specific updates

bb377d6f0aa3985a9f85d4fcabf9862dbd7e1d44

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jan 30 07:15:18 2021 -0800

Patch

Message: Make README heading lines more consistent

3908a24b94a8319a6df07598647c6007fd4c2826

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jan 30 07:05:46 2021 -0800

Patch

Message: Fix trailing whitespace and reformat long string

2c9c66392b50bb8e455dd1293a432f8bd2633380

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Jan 29 14:43:11 2021 +0100

Patch

Message: gsi specific updates 2021-01-29

6396d9f1869471d00977b2146b854b1158d8e50f

Author: René Wagner <rwagner@rw-net.de>

Date: Thu Jan 28 20:59:02 2021 +0100

Patch

Message: add systemd-units for automatic crawling

The template runs the crawler once a week on saturday afternoon.

If other launch times are wanted, gus-crawl.timer needs to be

modified.

3df47bfd42a8ea5142333e0cf5fa652752a04363

Author: René Wagner <rwagner@rw-net.de>

Date: Wed Jan 27 13:35:54 2021 +0100

Patch

Message: add "/robots.txt" route to views.py

It's a hard coded approach to serve a robots.txt to other crawlers.

No crawler may access /add-seed & /threads and all relevant virtual agents

may not access /search and /backlinks

a38b003bd5b90529d98350f162e7403f25353b7e

Author: René Wagner <rwagner@rw-net.de>

Date: Wed Jan 27 10:23:05 2021 +0100

Patch

Message: modify views to match geminispace.info

5915c5f6d297a9d91d1bfd6a86e9768a50fa5b1d

Author: Gogs <gogs@fake.local>

Date: Thu Jan 21 21:08:39 2021 +0100

Patch

Message: add seeds & update ignored urls

759c0493b08f7675a2943be9d93cb8b066bff71c

Author: ugla <ugla@u8.is>

Date: Sat Dec 26 18:30:35 2020 +0100

Patch

Message: Defer search requests to threads

409d9c14076a70495484ee9df304afa31c8132f8

Author: Remco <me@rwv.io>

Date: Tue Dec 22 12:46:04 2020 +0100

Patch

Message: Health test script and systemd service

Just for reference, it's already running elsewhere.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

c837404b0474e5b3ae35cef75859d33bec3ff3fe

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Dec 22 10:00:58 2020 -0500

Patch

Message: [serve] Fix copy-paste error in status endpoint function name

ba0a09765c5f0ae4cfb96d4c591ef0f14c620edb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Dec 21 12:04:41 2020 -0500

Patch

Message: [serve] Add status endpoint

974f2141967783f3cfcc42c3981c2f56430b650e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Dec 08 10:10:58 2020 -0500

Patch

Message: [serve] Improve formatting of statistics page

The right-alignment of numbers stopped working since the number of

things in Geminispace got too big, so adding two extra spaces to each

alignment block.

a5712854fde660e454456b734be56e41fd7f6c65

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Dec 06 11:29:38 2020 -0500

Patch

Message: [build_index] Import should_skip

Otherwise it breaks :)

59a14853ba375e11cc7004b8a53f62c2e985c5d2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Dec 06 11:28:56 2020 -0500

Patch

Message: Refactor change frequency constants

Put the increments in the constants file, and standardize naming.

0558ba7bc6139caa1fa2b94f9a93dc5c00dff347

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Dec 05 09:04:23 2020 -0500

Patch

Message: [crawl] Abort robots.txt parsing attempt if not text/plain

Python's built-in robots.txt parsing functionality breaks if the

content type of the robots.txt is not correctly set to text/plain. If

this is the case, simply abort the parsing attempt and allow all.

69da68b6bac395aef9c34de387b23ab987a1f651

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Nov 26 14:56:11 2020 -0500

Patch

Message: [serve] Update contributions list on about page

d401eeb5bcac81e80fb12d7261716a11fa5884cb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Nov 26 14:47:53 2020 -0500

Patch

Message: Bind to both IPv4 and IPv6

59620c13d5f8d3a6d9ef5f9068ceefd428753119

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 22 21:50:41 2020 -0500

Patch

Message: [crawl] Ignore another radio stream

097cb2a43b6e252ec4cd2a71e2a9302493641d12

Author: Remco <me@rwv.io>

Date: Fri Nov 20 23:37:03 2020 +0100

Patch

Message: Speed up get_newest_hosts

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

28a8f65d5bf8155789bbceef0b2843bfd36d2b16

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Nov 17 09:09:05 2020 -0500

Patch

Message: Add some more tests of GeminiResource

153745eb823c0962489289d85171bbdc24310855

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Nov 17 08:32:02 2020 -0500

Patch

Message: Add regex-based url exclusion support & refactor tests

This commit adds support for excluding URLs by regex, which is more

powerful than the prefix- and suffix-based exclusions GUS has so far

supported. There have been a number of cases, primarily involving

wiki-type sites, where it would be useful to match a URL by a pattern

that occurs in the middle of the URL, which is now possible. An

example of this is twinwiki's "_history" and "_revert" pages.

This commit refactors the existing test file to a more native pytest

style, from the previous unittest style. Additionally, it adds a new

set of tests for the URL exclusion functionality, covering both the

new regex-based exclusion functionality described above, as well as

the older style of prefix/suffix-based exclusion.

7b5205c27c989e2f52eec57712de4a93de326db2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 08:50:53 2020 -0500

Patch

Message: Add TODO to README

6871082e39a76cc1ac8a641f50572b318cf0440e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 08:44:31 2020 -0500

Patch

Message: Take exclusions into account when generating statistics

This will ensure accuracy of the statistics - it's relatively common

that index-excluded content ends up in the database, so this will make

sure the db-based calculations are generally more harmonious with the

index-based calculations/searches.

Note that it's not perfect, since I didn't address the calculations by

content_type/charset/etc. Those are a bit trickier to fix, so I will

have to think a bit more about the best way to deal with that. I

suspect it might warrant of bit of rearchitecting how exclusions work

generally. One idea I currently have for that is to keep the exclusion

list in the database, instead of in code like it currently is - that

would allow for inner joining against an exclusion table in db

queries, which would be really convenient.

Also, this commit removes the superfluous query for getting

domain_count - it's more performant just to count the list of domains

that were already constructed from the previous query.

99eadfccc51fc7bd71b66c0966f2e2e1cb6a7694

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 08:01:10 2020 -0500

Patch

Message: [serve] Fix formatting of dates on statistics page

Similar to the footer, these dates just need to be passed to the

datetime formatter GUS has defined for Jinja templates.

05c25d7c441f6a2d2f590d3242910884620dd720

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 07:50:07 2020 -0500

Patch

Message: Add two new TODOs to README

7e5882ef498044f0949403427e3c3de41ebaba65

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 07:49:37 2020 -0500

Patch

Message: [build_index] Only index text pages <= 1KB in size

1a1650dd7e619ef2102e5f445e1d079b06b689de

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 07:49:19 2020 -0500

Patch

Message: More exclusions

4f69e396ef8299946043471e7a3ad7fea8e742b3

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 16 07:47:53 2020 -0500

Patch

Message: [serve] Fix index closing when program is killed

This got broken during the recent commit to put search functionality

in the search.py Index class

d54e65a4cdea4f14390c395472da1ac498be28ea

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 15 10:56:01 2020 -0500

Patch

Message: [crawl] Increase increment to temp error change frequency

The crawls seem to be spending too much time on these, and there seems

to be a steady stream of new ones that all look like a common word

followed by a common TLD. Each one of these causes a long-running,

ultimately failed DNS lookup, so it ends up taking a long time. This

change should help naturally filter them out of crawls more quickly.

429b9d4de04bf8b95967cd7ad15ac46f2d751cbc

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 15 09:19:14 2020 -0500

Patch

Message: [serve] Update indexing documentation

540e3751d94e548872537902632af06ec10f1f22

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 15 08:41:44 2020 -0500

Patch

Message: [serve] Update about page

77f991d801bdc6d1059f61892e73205618c0dcf3

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 15 08:30:29 2020 -0500

Patch

Message: Bump rolling writer's batch size back up to 5000

0657f5703ef85ab8672b4a800d12ffb8df713d61

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 15 08:30:01 2020 -0500

Patch

Message: More exclusions

fe33be1c978c66af467b16787b18e63672120acb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Nov 14 11:06:51 2020 -0500

Patch

Message: Add systemd config

This is how GUS is already being run, so now checking the config into

the repository to start version controlling it.

7609836e3cd7b13f041996c2711853916aba3abe

Author: Remco <me@rwv.io>

Date: Fri Nov 13 14:24:36 2020 +0100

Patch

Message: Move all whoosh related stuff into separate module

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

af61f0143bc32d5dcbb8af0f854e5fe8140c6e85

Author: Remco <me@rwv.io>

Date: Thu Nov 12 21:03:07 2020 +0100

Patch

Message: A friend for the other duck

The second duck should acknowledge the first duck, don't you think?

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

6be6e25e7e964bee848b8551f50857be94ae2350

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Nov 11 07:27:50 2020 -0500

Patch

Message: Bump dependencies

a4e1b85f98945e9b2ab13d92102e464426465694

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Nov 11 07:18:28 2020 -0500

Patch

Message: [build_index] Fix logging statement

47fe755f7452739e724301189d0244f4c8755049

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Nov 11 07:17:25 2020 -0500

Patch

Message: [serve] Add statistics_overall_historical template

24167257f42d88aefd90de3f45962c1d6cc99e65

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Nov 06 08:56:01 2020 -0500

Patch

Message: Add .git-blame-ignore-revs file

As of Git 2.23, this can be used to exclude commits in Git blame

calculations. This is really helpful in excluding bulk

change/reformatting commits that don't do anything to affect code

functionality, but touch lots of files, and can make the commit

history more difficult to follow.

Per the official documentation

[here](https://www.git-scm.com/docs/git-blame), you can take advantage

of this manually like so:

git blame --ignore-revs-file .git-blame-ignore-revs foo.py

Additionally, you can set this up as a persistent repo-level

configuration setting like so, if so desired:

git config blame.ignoreRevsFile .git-blame-ignore-revs

b087fd6439ead0f9ff7b50e29751878ce948e96a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Nov 06 08:44:51 2020 -0500

Patch

Message: [crawl] Make logging message slightly clearer

0b11c4abfb4782c120cf41c2129b9819e731c37d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Nov 06 08:44:20 2020 -0500

Patch

Message: Check for null input in new strip_control_chars function

f0c4a784b4bc35049ab923fd68fc99c215e8e99b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Nov 06 08:43:16 2020 -0500

Patch

Message: Update default logging config to log to both console and file

43397bdda337c7a0f31019358bd56eb0ae87a993

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Nov 06 08:42:57 2020 -0500

Patch

Message: Reformat code with Black

5eebbbfc00555da619054e8129ad70bf3de99fd5

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Nov 06 07:22:02 2020 -0500

Patch

Message: [crawl] Strip control chars from URLs in crawl logging

cdba245e15f25857003c2aef03946f30b0b02a1f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Nov 03 08:38:23 2020 -0500

Patch

Message: Add exclusion improvement TODO to README

b012dd5cc733f553a34c5eb5d0e6b2bf92e489c7

Author: Remco van 't Veer <remco@remworks.net>

Date: Sun Nov 01 15:39:26 2020 +0100

Patch

Message: Ignore link like lines in preformatted text blocks

Blocks of text between ``` lines should not be interpreted as markup.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

75378967f722fbf1138d029612f68c684e8536d2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 02 08:39:15 2020 -0500

Patch

Message: Add contributors section to about page

c2dd594c459e62612c9cd57824ee031a8a21f3a1

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Nov 02 08:38:46 2020 -0500

Patch

Message: Fix the index build

1e63d8b307a42230db0a7e3fe2b2db9abcf2b608

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Nov 01 11:05:07 2020 -0500

Patch

Message: Clean up todo list in README

aa3fdeaefb1f80aa0838c2ea62b8e73f6e832d40

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Oct 31 10:06:14 2020 -0400

Patch

Message: [build_index] Flush index segments to disk periodically

2774ab2b0ec0d9c6b0f444919cd053507d6f4a86

Author: Remco van 't Veer <remco@remworks.net>

Date: Sat Oct 31 16:53:41 2020 +0100

Patch

Message: Logging

Replace all print statements in the crawler and indexer with log

statements. Use logging categories to distinguish between debug

information (level "debug"), progress (level "info"), and things that

might need attention at some point (level "warn").

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

f3f9d41aa7652d1239792652eaf972985e36e84d

Author: Remco van 't Veer <remco@remworks.net>

Date: Sat Oct 31 16:53:40 2020 +0100

Patch

Message: Drop unused imports

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

b85ce1bf4ad231b5b632da2d721bde7ab5048da5

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Oct 31 07:23:29 2020 -0400

Patch

Message: Update gusmobile clone location in pyproject.toml

fe1c2054cbd2e8985f8e94b4335a09338735e37b

Author: Remco van 't Veer <remco@remworks.net>

Date: Tue Oct 27 20:26:59 2020 +0100

Patch

Message: Include notes on updating the index

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

be45fc45968486e8669bbe0eed0163e5c853a640

Author: Remco van 't Veer <remco@remworks.net>

Date: Tue Oct 27 17:02:13 2020 +0100

Patch

Message: Describe procedure to get gus up and running

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

c858691814aadb66868f19244794bb1b47d5f0a1

Author: Remco van 't Veer <remco@remworks.net>

Date: Tue Oct 27 17:02:12 2020 +0100

Patch

Message: Fix missing database column indexed_at on Page

It's used but never defined.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

f7de0f8473861f62e2fccd13a1a29a2d73ebb098

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Oct 28 06:55:18 2020 -0400

Patch

Message: [crawl] Add a few new exclusions

72c6ccbf81941ce81541bfbbf5eb8f03ebf77b61

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Oct 28 06:50:06 2020 -0400

Patch

Message: [build_index] Perform prefix-based URL exclusion during index build

Previously this exclusion only happened while performing the crawl,

but for a number of reasons, pages have ended up in the database that

should be excluded from the index. Some due to user error, some due to

the exclusion list growing over time.

The fact that they're still in the database means they are probably

impacting db-based calculations, so longer-term there probably should

be some sort of pruning process or something to keep the db entries

pared down to only what we care about.

Even after adding such pruning functionality though, I think this

changeset would still be valuable to ensure the index only gets valid

entries.

c0210d90cf3d599060d58e6972fb6272ea322237

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Sep 16 08:56:40 2020 -0400

Patch

Message: [serve] Add "jump to page" functionality to search

5d7627a3f26d428a28a997061f977657b8abc9c9

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Sep 16 08:43:12 2020 -0400

Patch

Message: [serve] Upgrade to Jetforce v0.6.0

3756e5becfdbdc05f954b70bbc2b88ad19dd7bf2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Sep 16 07:02:39 2020 -0400

Patch

Message: [serve] Add more quotes

6ca5c355d7f317beead5b753a9b254037686158b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Sep 06 06:21:49 2020 -0400

Patch

Message: [serve] Update documentation and links a bit

6ddea2105baef4bf391c30b3cdd33e729d2fe99e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Sep 04 08:21:41 2020 -0400

Patch

Message: [serve] Add dynamic quotes to footer

c67268608f93486bba5ff7ce135549b73e98f5a7

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Sep 04 07:50:54 2020 -0400

Patch

Message: [serve] Add newest pages endpoint, revamp documentation and index

6df4e561eb1280bac4ddec89891a9851245604f4

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Sep 03 08:00:33 2020 -0400

Patch

Message: [serve] Add newest hosts route

22145a7abd5859e92c5b9967e8d71d8495209e46

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Aug 25 04:37:52 2020 -0400

Patch

Message: [serve] Remove extra quotation mark in add seeds template

86bf28edff91ef55461ae8f27492dfadd8df8b22

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Aug 11 08:30:50 2020 -0400

Patch

Message: [crawl] Print change_frequency

8a0c456fb9c75aa7c0930da9ecd67a53bc623749

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Aug 11 08:18:04 2020 -0400

Patch

Message: Fix bug in GeminiResource url construction

It wasn't adding the colon to the scheme of URLs that started with

"//".

c5b0648dcce6706d8131ef068c1587edaa54e941

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Aug 09 09:18:19 2020 -0400

Patch

Message: [threads] Only work with textual pages

d993f6cbd0a5c627490fd80a9000d793a08c0ca2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Aug 05 14:33:07 2020 -0400

Patch

Message: [serve] Add favicon.txt route

6adb5336b5e75aac90e8d31e8ff025741406f7ef

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Aug 05 09:03:56 2020 -0400

Patch

Message: [serve] Add IP addresses to about page

b6ddd91524cf60db0e7cd93516b78c8f0f38ae4a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Aug 05 09:03:27 2020 -0400

Patch

Message: [threads] Add different sort orders for threads

3d014404a2f9f01a348a516ad1c9ed03fc603d4f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Aug 03 12:55:10 2020 -0400

Patch

Message: [serve] Improve feed matching

24681773991d4aeb820ef2eb23f0f9766c6bf06d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Aug 02 09:51:17 2020 -0400

Patch

Message: Update naming

fae9d9d5fe1bfb84af91c6a93743597ddd04042c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Aug 02 09:46:53 2020 -0400

Patch

Message: [crawl] Improve handling of change_frequency

This change centralizes the logic into lib/gemini.py for a start.

Additionally it fixes a bug in that the crawl was incrementing the

change_frequency when the page *was* changed. And lastly, this now

adds some pikkulog detection, so those pages get crawled frequently as

well now (which will help them stay current in thread construction).

0b45da52c1fc5ee927f2ea9e90570a8bd38fe3fc

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Aug 02 05:45:28 2020 -0400

Patch

Message: [serve] Add Known Feeds page

34be029c6522ba5692722618af7efebe1420b355

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Aug 02 05:42:59 2020 -0400

Patch

Message: [threads] Add collapsible log variations

Currently this does some work for both duplicated content (the last

two entries) as well as redirects (the first three entries). Fine for

now, but the redirect magic could and should be made more robust by

actually resolving the redirect chain in the index when attempting to

build threads.

a2607cd721bf2828c29b3b0dc65573f8bc1fb753

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 28 08:56:01 2020 -0400

Patch

Message: [threads] Fix thread ordering

6ea24fffbb8968a5cb606fd9fd09dc918310dc10

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 28 07:04:45 2020 -0400

Patch

Message: [crawl] Index more errors

93722e67599abd7a2372f2324cd6e8925f159112

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 28 07:04:06 2020 -0400

Patch

Message: [crawl] Add change_frequency backoff

632a4cb16c0d114b83b105cf00d6743649f2d40d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 28 07:03:39 2020 -0400

Patch

Message: Bump dependencies

f75751e5b939a8b828206cfe9e3eaf0c3f73edcb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 28 07:02:50 2020 -0400

Patch

Message: Add friendly authors and titles for threads

8c1399ade960df51af3c53f6bcd8ee7fad0df56e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 27 14:50:15 2020 -0400

Patch

Message: Threads v1

ded0c0ca62fe84b119a4325a53331408328e389d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jul 24 06:43:53 2020 -0400

Patch

Message: [serve] Save searches to db

39010248c195bce521a2aaf59cc3a72a7523125a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jul 23 14:40:17 2020 -0400

Patch

Message: [build_index] [serve] Distinguish cross-capsule backlinks

c341bb82ae237de0fa3aff8b5bec1b3d5efa791c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jul 23 09:44:49 2020 -0400

Patch

Message: [crawl] Add is_cross_host_like field to db

6c187c2af2168e097c45589482e36e723b541e03

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jul 23 08:35:09 2020 -0400

Patch

Message: Gitignore all the indexes

3212bff302ea03289145d03ba4d6668e03767f26

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jul 23 08:29:27 2020 -0400

Patch

Message: Bump dependencies

b64dfafc563cef3d670e4bbbbcbdd8775d5855ce

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jul 23 06:54:55 2020 -0400

Patch

Message: Create scripts directory

96b7b661fb2c5f5eed02e4475467ef61a1b4c251

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jul 22 13:29:00 2020 -0400

Patch

Message: Add normalized url to db

4cc4e66b6c01fe54d9e5992f7f6a2147d53eba2b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 21 15:43:56 2020 -0400

Patch

Message: [serve] Add cert change to news page

962eb179053b0cce08fad5192199ef4a38485d25

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jul 21 14:49:36 2020 -0400

Patch

Message: [build_index] Account for per-page expiration

5bfa28c579ad10ddc5588109229f6093e1e4f7ef

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 20 08:19:03 2020 -0400

Patch

Message: [build_index] Build index with backlink_count instead of backlinks

This works because all the actual fetching of backlinks is now handled

by database queries, so we can slim down the whoosh index a bit with

this change.

190b9875c17508609978bafe4a88911a93bfc042

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 20 07:56:52 2020 -0400

Patch

Message: [crawl] Start indexing errors

c172c20e952abe9f34c2c78446b7bcdb81dbbd71

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 19 09:23:46 2020 -0400

Patch

Message: [crawl] Update db model, and delete links before recreating

d012217757ba8dfb3f2e6dff0dbdc79aa867ca4e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 19 08:18:31 2020 -0400

Patch

Message: [crawl] Ensure manual exclusions stay out of the database

b19d7e8a4c125c7380935bec3384eb8fc53a8ec1

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 19 07:35:19 2020 -0400

Patch

Message: [serve] minor formatting updates

df3718e3a85f47e641649f37a523ba15db81e524

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 19 07:32:05 2020 -0400

Patch

Message: [crawl] Support per-page expiration

This will allow crawls to intelligently decide which URLs to recrawl,

if any. Some pages, like site indexes, or gemlog pages, default to

expiring much more quickly than others. This way recrawls should pick

up links to e.g., new posts, fairly quickly. Conversely, existing

posts, and binary files, are considered to be more static, and will

expire much less frequently, and thus be recrawled less frequently.

a9806b3f479e017f0555372d95d591ad33f42cf2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jul 15 09:09:39 2020 -0400

Patch

Message: [crawl] Rebuild link table completely and idempotently

a8a1abd487ad5dd0587d680a136dda0a8b1fd7a7

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jul 15 08:20:03 2020 -0400

Patch

Message: [serve] Get backlinks from db instead of index

0a0d4669dea4557b24f35fdba4177203e782018a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 13 19:55:07 2020 -0400

Patch

Message: [crawl] Set cap on maxiumum redirect chain length

d10caa7aa57fdbe73ba797b5ddad8621f7a16da1

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 13 19:18:16 2020 -0400

Patch

Message: [crawl] Abort when detecting self-redirects

ec3dea118bd5ab4820b0e1a6577960f60c27625b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 13 19:17:36 2020 -0400

Patch

Message: [crawl] Ignore 80h gopher proxy

97b5864e2dc36e120af39e1b38bca3c3b4dca05c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 12 09:27:35 2020 -0400

Patch

Message: [serve] Improve pager linking back to previous page

Specifically, if we're linking back to page #1, remove the page number

component of the URL path. This way, if you page forward and back,

then reload, you'll be prompted to enter a query. This improves the

user experience slightly in Elpher.

44d21d1de5808ada3c62dcc4ef03bc52b2a30f37

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jul 11 08:33:56 2020 -0400

Patch

Message: [serve] Update backlinks links and presentation throughout GUS

46dff2549af9ca8a6408275ced77be5fddcb5397

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jul 11 06:56:49 2020 -0400

Patch

Message: [serve] Improve safety of backlinks code path

Before, it would throw an unhandled exception if the user entered an

invalid URL as their backlinks query.

c74caeb975ca120edf75f8d0a6bb865a1c144952

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jul 08 06:18:15 2020 -0400

Patch

Message: [crawl] Add feature to seed incremental crawl with atom feeds

e4b2ef0192c7d75583f6a417c260585566b5125f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 06 06:22:01 2020 -0400

Patch

Message: Make incremental build_index work

Some of the idempotency was lost during the shuffle to split the crawl

into two phases.

a3fef86b23fab26e13a97c281ef55f22cd444a19

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jul 06 06:20:01 2020 -0400

Patch

Message: DRY up the sqlite model and init_db code

613583f8ee6134346ffd8469e6761d98537b4693

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 05 08:52:26 2020 -0400

Patch

Message: [serve] Improve handling of backlink searches

They were sensitive to trailing slashes before this. An alternative

approach to consider for the future would be to add "normalized_url"

to the index. This would increase the index size, but it normalizes

away trailing slashes, so would eliminate the need for two searches

here and improve performance of backlinks queries. If they turn out to

get a lot of use, this alternative approach will probably be the

better way to go.

9a9d48e1dfc458be1c2730dba1ec6f15ceeb8817

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 05 08:02:54 2020 -0400

Patch

Message: [serve] Add historical statistics page

261d06460c3c667696aa13935b39dc3400777baf

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jul 05 07:01:16 2020 -0400

Patch

Message: [crawl] [serve] Run statistics and domains from sqlite db

a975ff45981f8d5cda29daa9f278bc4bcfc9207d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jul 04 06:43:27 2020 -0400

Patch

Message: Improve discovery of backlinks

Specifically, make sure the query picks up backlinks pointing to both

the slashed and slashless version of the URL in question.

5ad2a1e64d4b4dd19d4dd878446996d53826affb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jul 03 11:45:23 2020 -0400

Patch

Message: [serve] Fix minor bug in counting of backlinks

Empty backlink strings were getting counted as "1" instead of "0".

d7518388dd0fa358d7ae454d3feeb1fb7e6c2aa7

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jul 03 10:39:56 2020 -0400

Patch

Message: [crawl] [serve] Switch crawl to 2-phase with sqlite

2c7edac9e1fef61fb30f584cbe9700a51b5dec5f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 30 08:57:53 2020 -0400

Patch

Message: [crawl] Ignore localhost

3fdafceec0c309b1b7033e7ef7734318e935c5b4

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 30 08:54:40 2020 -0400

Patch

Message: [serve] Add backlinks news and documentation

c8fb8ed9a15c99378685e39e8ac8492ddf504355

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 30 08:28:39 2020 -0400

Patch

Message: [serve] Improve verbose mode

3d1eb23a10487cbaf838f37c32ac9f83a6113ae2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 30 08:24:56 2020 -0400

Patch

Message: [serve] Update header levels

b6ffd1fc684e26be9037d2959682590bb959ee20

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 30 07:07:36 2020 -0400

Patch

Message: [crawl] [serve] Add backlinks

b8fb69f84254a1cfd695683dd50d159e59d9ab14

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jun 22 16:57:03 2020 -0400

Patch

Message: [crawl] Ignore more bad content

d8fc5a1bec0f0e9d8eac3fad141aa96dddf3385f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jun 18 07:16:13 2020 -0400

Patch

Message: Update README

a3dd4578b051bf55728b9c250844287992b43792

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jun 18 06:58:28 2020 -0400

Patch

Message: [serve] Rearchitect serve to use templates and MVC pattern

e0ea8f1de508923e07d916898ada64527bae3e0d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 17 09:09:56 2020 -0400

Patch

Message: Add GUS licence

2b10d9000fbaa42c1e625016489a6db63f185200

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 17 07:36:13 2020 -0400

Patch

Message: [serve] Make seed request handling async again for now

2b1ff38cf9aa53855ba69a9013889eaa7191407e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 17 07:33:48 2020 -0400

Patch

Message: [crawl] Ignore some more alexschroeder pages

832865d47aebd6eb4b955657f75a496963bab269

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jun 12 09:38:24 2020 -0400

Patch

Message: [serve] Sort domains on the known-hosts page

0f7e151b3aebc673f3e327e06a6006b9e471f382

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jun 12 06:40:33 2020 -0400

Patch

Message: [serve] Add size to result rendering

cf846b8bb6f69b1bcf7c6c65e09250ca4675d51d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jun 11 06:38:56 2020 -0400

Patch

Message: [crawl] Start indexing response sizes

b833d19086bda01fa669e8a771c3d5ebc35daefe

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 10 08:09:33 2020 -0400

Patch

Message: [serve] Use preformatted blocks on the statistics page

8e51836116d0ec77e5935a35297893ae0c883f2f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 09 07:01:45 2020 -0400

Patch

Message: Bump dependencies

68f4fec7fc23d7f2d7ea13c4f1d08d41b6d25d11

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue Jun 09 06:55:11 2020 -0400

Patch

Message: [crawl] Start indexing lang parameter

e0f53546a9629c968c696cd3a15e3c88eca67a38

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jun 08 07:29:11 2020 -0400

Patch

Message: [serve] Update some copy on about page

4b42c6cbd81b0c1d2d0e8299352bf56861521be5

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Jun 08 07:28:39 2020 -0400

Patch

Message: Revert "[crawl] Index raw content for regex searches"

This reverts commit c127a0a2e9a03b60d8ea82447c27af6b12cc128b.

04785c1de606b8e05dcfc8b5c5854b0cea3afb3b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jun 07 08:32:16 2020 -0400

Patch

Message: [crawl] Ignore some more things

b2a0e6afea9f710e6d1c0a61177a337384124223

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Jun 07 07:05:02 2020 -0400

Patch

Message: [crawl] Add marmaladefoo's calculator to manual exclusions

ac4aa302ec3d2cc6af7f52682aa955fd0e2b7fb9

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jun 05 07:35:12 2020 -0400

Patch

Message: Add easy CLI way of removing domains from index

c10da9f7bfe5ef7395f1b679e91bf329073439ff

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jun 05 06:46:55 2020 -0400

Patch

Message: [crawl] Remove manual exclusions for alexschroeder.ch

They updated their robots.txt, so now the Disallow lines are parsing

correctly.

44f6e6250611aba9dd3557eba7326b67d4c4249e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Jun 05 06:41:28 2020 -0400

Patch

Message: [crawl] Add custom crawl delays

And add the first one for alexschroeder's site, which still has a

robots.txt that doesn't parse properly.

0bbf43c49ba5cccbc26d346283d66a2651261a6d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jun 04 11:27:51 2020 -0400

Patch

Message: [crawl] Improve indexing performance

I was getting index out of bound issues for optimize calls before this

change - and when looking in the index/ dirs, there were over 30

thousand files. I think this caused issues with whoosh, so now I am

waiting to commit all the writes to the end of the crawl. It's more

unfortunate if the crawl dies, but c'est la vie. On the plus side, now

the optimize call is no longer really even necessary since the final

product is only a few index segments.

ffd444663634ff94298ae07c6bbd943c8fdec880

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 19:37:37 2020 -0400

Patch

Message: Update some seeds

8a1cafaffb5a36ce9e965486a09a24927d323e07

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 16:28:13 2020 -0400

Patch

Message: [crawl] Start indexing the charset

245f74634afb8c71cfc0a73c176f99980f1d7bff

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 12:50:59 2020 -0400

Patch

Message: [crawl] Only attempt to extract contained resources from text/gemini

ec356baa82d67ea36aa3a32ce99c341e89fc9960

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 12:50:39 2020 -0400

Patch

Message: [crawl] Ignore some troublesome content from alexschroeder.ch

f8aa717e97594405e80157674ac355e15da4b8af

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 12:50:10 2020 -0400

Patch

Message: [crawl] Fix default crawl delay when not specified explicitly

bfcfec84e04a54c1c72df6858512ad6a238ec2d0

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 10:58:45 2020 -0400

Patch

Message: [crawl] Persist index & crawl statistics on non-destructive crawls

Also, add a flag to track which serialized statistics lines originated

from incremental crawls.

682feb199142c56576c21692108828efbbc43879

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 10:53:30 2020 -0400

Patch

Message: Bump dependency versions

c127a0a2e9a03b60d8ea82447c27af6b12cc128b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 10:49:16 2020 -0400

Patch

Message: [crawl] Index raw content for regex searches

dc4a32429ccd4d547319e6aacb54e500dd04ff37

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Jun 03 10:47:33 2020 -0400

Patch

Message: [serve] Use "OR" as the default connector for queries

dd46f0e29bf49df3637eccb28477d409c6e698d7

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 29 14:40:56 2020 -0400

Patch

Message: [serve] Make sure two closely-timed seed requests don't break

This will prevent seed requests' incremental crawls from stomping on

each other, but due to the way in which incremental crawls

resolve (i.e., by restarting the entire GUS serve process via

systemctl), it also means any seed requests that came in after the

first will not be handled until either A) another seed request comes

in that ends up dealing with it, or B) a manual crawl is kicked off.

The situation is no worse than before however, so this is still an

improvement in the short-term.

dbec660e8274c79bd73420d745db8356019c566e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 28 09:02:21 2020 -0400

Patch

Message: [crawl] Improve hierarchical handling of robots.txt entries

Give more priority to more specific entries - i.e., an entry for

user-agent "gus" should override an entry for user-agent "*".

80db86f5c9626a6e21fe25c2739886fa45f36ebc

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 26 09:48:26 2020 -0400

Patch

Message: [serve] Update copy on known hosts page

752b704599366b052bda583a9c163d935a7e3b11

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 26 06:57:46 2020 -0400

Patch

Message: [crawl] Ignore some Geddit URL prefixes

567916e94091b1053d8c1fbe69732b1d4b9be99f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 25 21:44:46 2020 -0400

Patch

Message: [crawl] [serve] Add fetchable URL to the index

20b2ccf59fbe3a55055d26535e7394acd0dc400d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 25 13:19:28 2020 -0400

Patch

Message: Bump version of Jetforce dependency

35176988fc825f13131be2c3f573698675c09f89

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 25 06:31:14 2020 -0400

Patch

Message: [crawl] Improve handling of quoting and unquoting URLs

Before everything got unquoted at the very beginning of GeminiResource

instantiation. This was slightly errant. It was fine for the

normalized_url and the indexable_url, but resulted in fetchable_url

not being sent quoted, which it should be.

370e53eabbc7649f4ac6e424f46efebebed4b8f8

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 24 23:05:00 2020 -0400

Patch

Message: Rename fully_qualified_url to fetchable_url

6adbcc2b60e7d7f84540b8b7e5064c54d3042ed1

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 24 23:00:23 2020 -0400

Patch

Message: Rename fully_qualified_massaged_url to indexable_url

6218770a47f53c2a6a23f0ed28c452a2fa8ff67b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 24 22:54:38 2020 -0400

Patch

Message: [crawl] Fix bug in fully_qualified_massaged_url

e61d608c8e2b9171a47ff954ecf119913e351352

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 24 10:08:37 2020 -0400

Patch

Message: [crawl] Stop storing responses in GeminiResource objects

I think this was causing memory overflows, since we were storing

potentially a lot of response content in memory without being able to

clean it up during long chains of recursive calls to crawl() of

contained resources.

0a9ac040af14daff771aa43dfbdf61b214ddb9a4

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 24 10:10:04 2020 -0400

Patch

Message: Bump version of gusmobile dependency

009873a26d4cda643e16db159e12a96c0633282a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 24 07:28:15 2020 -0400

Patch

Message: [crawl] Handle url fragments

Up to this point, fragments weren't being handled at all, so links to

two different fragments on the same page would both get indexed as

distinct results. With this change, we now strip fragments so the only

thing that ends up in the index is the fragmentless-URL one time.

59db14509562934de4e70f38d9038646ed0dc5e5

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 23 09:11:59 2020 -0400

Patch

Message: [crawl] Fix handling of robots.txt

72feabcfe86495aef4d5e0ddb074635e129e0b81

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 23 07:19:09 2020 -0400

Patch

Message: [crawl] Exclude "rss.xml" paths

a4864548ca222495d94aa8a80c7d7778e74f5fed

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 22 09:18:24 2020 -0400

Patch

Message: [crawl] Optimize the index after crawls

96731d16d3db6b277b5dacfcee6d4f4b91449a9c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 22 08:42:03 2020 -0400

Patch

Message: [serve] Update highlight scoring and rendering

a7ea73424895a1e73d0bcc7ea2dc1e6d28257ec6

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 22 07:31:18 2020 -0400

Patch

Message: [crawl] pickle and unpickle the robot_file_map

This way we don't have to re-request all the robots.txt files during

incremental crawls

cde47da62cb626be45196e66f8f5b8125a4b4827

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 22 07:20:20 2020 -0400

Patch

Message: Improve handling of unquoting URLs

Just do it once at the beginning of GeminiResource creation.

e3f879df8442e16f541954359009f369b26717c0

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 16:07:42 2020 -0400

Patch

Message: [serve] Update documentation on filters

8219abb97c98318daa21a85ed1961e8c7505e97e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 15:35:51 2020 -0400

Patch

Message: Update locked version of Gusmobile

d96abf705580f8fece03e9dc9440c94e02bf8e66

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 10:59:33 2020 -0400

Patch

Message: [crawl] Add domain field to index

941b086b7d7e8b8e1e16bd92fd80c5cc17d1650b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 09:25:18 2020 -0400

Patch

Message: Remove outdated TODO

7d609838ded87a4819ee08a2b45b0be8b2099dcd

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 09:18:25 2020 -0400

Patch

Message: [serve] Update formatting of statistics page

d07bb33e677afbe7aedff1774f32354b64de7edb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 08:39:54 2020 -0400

Patch

Message: [serve] Fix bug with first/next/previous page link formatting

Previously, it wasn't url encoding the query, so if a query had more

than one term, like e.g. "gemini hosting", the link line would get

formatted like "=> /search/2?gemini hosting Next page" so clients

would show the text "hosting Next page".

25713d69d87be8632b85c3d978452596f77f3eb7

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 07:57:13 2020 -0400

Patch

Message: [serve] Only highlight nice content types in search results

e4c042c330023335eeded1d140db47f0f9727cb8

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 07:33:36 2020 -0400

Patch

Message: [crawl] Make path exclusions more robust

1fedfc3bc57d434f3cddefcbad0a464285c35250

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 06:53:46 2020 -0400

Patch

Message: [serve] Remove broken URL count from stats page

ab9d86ca3d48a88d15801a689c2213146271ce69

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 06:45:49 2020 -0400

Patch

Message: Add houston to seeds, but ignore its search results

fbc302284a81e74f81369ccc9be6e098e4b728e8

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 21 06:45:28 2020 -0400

Patch

Message: [crawl] [serve] Add search highlights

dd1c2ffdef2f69a091d719f52bf2b27ee469ee27

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed May 20 09:33:58 2020 -0400

Patch

Message: [crawl] Index massaged URLs

Up to this point, we were indexing the URL from the gemini response

object. Instead, let's index something that's been a bit more

normalized and cleansed. We want to keep the capitalization, but strip

unnecessary ports and trailing slashes.

78ca450d9f5c7b3557834cea1a9a467d7508a401

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed May 20 09:32:32 2020 -0400

Patch

Message: [crawl] Handle trailing slash redirects better

This was recently refactored out, and it resulted in duplicate entries

in the index, like e.g. gus.guru and gus.guru/. This change should

prevent that from happening any more.

161252e750073377625a22d1e4d8cd0748116e4f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed May 20 08:15:27 2020 -0400

Patch

Message: [serve] Update the loading of statistics

Do it more dynamically, so after users submit seed requests, they will

show up immediately on the /known-hosts page.

8994b21fea2d7adf1ababecfae27971ff8390fb1

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 19 17:08:53 2020 -0400

Patch

Message: [crawl] Fix lots of bugs

97b15eaa87fcd5ba7604fd540e64767fdfe6b04c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 19 06:47:51 2020 -0400

Patch

Message: [crawl] Crawl the seed requests after the main crawl

22d4dcaa8c27660b6845521525b1882af3cf1a20

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 19 06:36:44 2020 -0400

Patch

Message: [crawl] Fix bug in relative URL parsing

f10f1fc9a0b9a8171fa69a0d206d1b2acd1f3518

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 18 15:52:48 2020 -0400

Patch

Message: [crawl] Fix bug with computing full_qualified_urls

484ef909792e8227a33fd54a648cbe36147b6627

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 18 09:12:31 2020 -0400

Patch

Message: [crawl] Use standardized print_index_statistics

c1c29b4a7474161c63b7fb7172a6e4c35d0a817b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 18 09:01:31 2020 -0400

Patch

Message: [no-op] Clean up comments in whoosh_extensions

9ffc427a6c46bafe30d57669b581118139b9e448

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 18 08:57:27 2020 -0400

Patch

Message: [serve] Crawl and index seed requests immediately

8bcf71965e48a9954bf3efdba115b084a1062f5d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 17 10:30:58 2020 -0400

Patch

Message: Update README TODOs

53ce6aa50539d49837f9a9ef3f3ed9f9e50dfd2c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 17 10:20:11 2020 -0400

Patch

Message: [crawl] Implement GeminiResource

This commit should actually be somewhat close to a no-op, but brings

substantial refactoring of the code to consolidate both functionality

related to gemini URLs as well as the source of truth for crawler

information about them (including relevant metadata) to a new class

called `GeminiResource`.

4b123933cff8b0fe8203838c87451654439d2924

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 17 07:45:25 2020 -0400

Patch

Message: [crawl] Exclude GUS search result pages from crawl

7d7422b97523544ee404c5167b8c75fb691ff5a3

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 17 06:21:18 2020 -0400

Patch

Message: [crawl] Add seeds

b0e990ca1365d081fe2a67ceb1e39038257a6fd4

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 14:51:29 2020 -0400

Patch

Message: [crawl] Add jan.bio to seeds

4c6e886c7ff89cbe3d1aa20a83f18cabc4cff11f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 11:23:25 2020 -0400

Patch

Message: Add index.bak to gitignore

7ce414234e945fb99728b6714db48f02fac0e778

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 10:57:49 2020 -0400

Patch

Message: [crawl] Create non-destructive crawl option

c2dd86ae92ca635f111d4b4c9a55464cff893a10

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 09:23:21 2020 -0400

Patch

Message: [serve] Improve documentation on content type queries

e49e877eb7e21791431ffec6bd85a339ae033ea3

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 09:05:55 2020 -0400

Patch

Message: [serve] Add verbose mode

b2026faac6ad331e3f0c70ab3512b25b32bb6798

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 08:22:57 2020 -0400

Patch

Message: [serve] Update how num_results is displayed

4136079b4e38f5d072e1aa61baa8c28e5df6409b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 08:12:22 2020 -0400

Patch

Message: [serve] Improve search result data type

20a5cb896d0eee3d271565ce420f74a598e1f94b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 08:00:35 2020 -0400

Patch

Message: [crawl] [serve] Add more statistics

d16c11de01f336034c8b4e70333c104b5c0f77a0

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 16 06:57:40 2020 -0400

Patch

Message: [crawl] Update seeds

ec6401a5230a803a36395a5ee9c9889603b69a89

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 15 08:03:04 2020 -0400

Patch

Message: [crawl] Update seeds

5ff76ac64ef29928c15dce5d4f8d2a1ff1b53b18

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 15 08:01:16 2020 -0400

Patch

Message: Update and reorder TODOs

32d12c4c5e7396fb3ca01751acf9b37a8e5b1cd6

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri May 15 06:27:44 2020 -0400

Patch

Message: [crawl] [no-op] Add a line after backup operation

Just to visually set it off from the first crawl operation.

2c002c6d76ccbef1860048fe963728a1f4418025

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 14 15:40:54 2020 -0400

Patch

Message: Update statistics TODOs

c7905a645abcf2b74f46ef3656eda3ca4300f5a8

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 14 09:17:59 2020 -0400

Patch

Message: [crawl] Add new seed

a884649816ee02eeb69bb31d6ab21eeaebafe483

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 14 08:49:56 2020 -0400

Patch

Message: [serve] Update statistics copy slightly

087e227c67413185b7db0282e21333b6fbb2548d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 14 07:56:28 2020 -0400

Patch

Message: [serve] Implement paging

a38326c3c76bd33bb7ccbfaba2ebaa7381fd4943

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu May 14 06:59:52 2020 -0400

Patch

Message: Update README ideas for more index/usage statistics

eea51a8b2172cc3c37086e50af071ea5106aa579

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed May 13 10:20:16 2020 -0400

Patch

Message: [crawl] Add new spanish site to crawl seeds

51729e59c9466cc940a141fddff740df45116028

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed May 13 09:51:06 2020 -0400

Patch

Message: [crawl] Refactor manual exclusions and add fgaz' calculator

The calculator seems to generate links dynamically, so attempting to

crawl it will yield unending pages with links to more deeply-nested

mathematical operations.

1a9a58abf2bb4e3fb3fc35b5ba6507289411f446

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 12 08:52:19 2020 -0400

Patch

Message: Add TODO for generating and sharing GUS usage statistics

b39154af7ead9bbc40a875ab94a9cdab850ba183

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 12 08:46:14 2020 -0400

Patch

Message: [serve] Add news feature

c9d588e7e168dbc785d9ce5fa52c34b20a8da817

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 12 08:18:40 2020 -0400

Patch

Message: [serve] Add page to show all known hosts

8ce3f4d55921e668a413db6242496e8f3e30dab3

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 12 07:56:30 2020 -0400

Patch

Message: [statistics] Add ability to compute and print stats easily

ffcba3395182761ec36834e566a2fac4c20df0c5

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 12 07:23:09 2020 -0400

Patch

Message: [statistics] Refactor statistics objects to pass around dicts

4c2100ff326053ce883d499f83c603a7136be247

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 12 07:07:12 2020 -0400

Patch

Message: [serve] Add page headers

1ab8d5cc9c4b9ae9d5f03d77938cce8929802488

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 11 14:51:35 2020 -0400

Patch

Message: [serve] Update copy for current index statistics

f071d9d565ab40619850746c794df92609e306d0

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 11 14:45:53 2020 -0400

Patch

Message: [serve] Stop hard-wrapping content

The Gemini spec was recently updated such that content creators are

now requested to NOT hard-wrap their content, so this commit updates

GUS to comply!

7093ad2aa8b298e8648f2c40381d1e288ef50605

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 11 13:56:48 2020 -0400

Patch

Message: [serve] Report out current index statistics

73a9aca69749a2ce6849e4aa328810dbf4cb397d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon May 11 13:16:04 2020 -0400

Patch

Message: Refactor some common/library code into separate files

ad9f0212b72fe70a47ab7f4148310cbdfd61f55a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 10 12:12:16 2020 -0400

Patch

Message: [serve] Remove TODO to add documentation for content_type

87162dcb3cd000823c899a726dcefc944f42659c

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 10 11:50:46 2020 -0400

Patch

Message: [crawl] Alphabetize and add a few more seeds

ff6a81da4088958affce70a91e60f5e248242156

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 10 10:39:05 2020 -0400

Patch

Message: [crawl] Backup old index before running crawl

cc7082f08d546dc4542263ce0068dec7b041f5a6

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun May 10 10:38:47 2020 -0400

Patch

Message: [crawl] Add indexed_at field

586127b04bf8d0b70d8714f21d6be8a571ec01f4

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 17:34:52 2020 -0400

Patch

Message: [crawl] Compute and generate index statistics after each crawl

c7ab03d8b5cd5bd98d0844c6a1c3a103c0a33809

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 17:23:54 2020 -0400

Patch

Message: [serve] Update content_type search documentation

4d825da633611e2245231cea464f49cdeb71c561

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 16:05:05 2020 -0400

Patch

Message: Add TODO to track Geminispace statistics

b70fef847f69bc7a303ca2af4909b152020bb0c2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 14:07:02 2020 -0400

Patch

Message: [serve] Add documentation for content_types

ff9804e3cf9ecb695deb0b7624731053ea8fc89d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 13:35:28 2020 -0400

Patch

Message: [serve] Add note that paging isn't implemented yet

0161ae462a27645f9720aa03c9321b2da8398d73

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 13:35:08 2020 -0400

Patch

Message: [serve] Put index generation date in footer

61f7f7e33fe57460ff8d55cd6199b16d3e95b655

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 12:38:49 2020 -0400

Patch

Message: Add a couple TODOs

6cd6e08b380aabbda95e1a7d509c9dfb3bbbfc9e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 11:54:45 2020 -0400

Patch

Message: [crawl] Add two new seeds

4e368d44e561b6e45cdece12cca9ad9fe8e17240

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 11:06:38 2020 -0400

Patch

Message: [crawl] Stop printing the sleep duration

10f44e0956d8087a20663bd02de658be7c4fe1e0

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 11:00:01 2020 -0400

Patch

Message: [crawl] Improve error recovery

The crawler is starting to hit some indexing errors, presumably with

new content popping up in Geminispace that either is malformed on its

own, or perhaps is fine, but is exposing errors in GUS' crawling code.

As an immediate-term fix, this change commits documents to the index

more frequently, and recovers gracefully from indexing errors with

individual documents. This will slow down the indexing process, but A)

I think that's worth it for the resiliecy gain, and B) in practice, it

might not actually slow things down much at all, since the extra

writing time will likely get swallowed up by the kindness-sleep in

between most requests to the same domain (it will still cause extra

waits in between requests to two different domains, and extra time

incurred opening the index for each document regardless of domain).

99c223c0adf5fb12fc683a0571e2cc5f412f949e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat May 09 10:58:18 2020 -0400

Patch

Message: [crawl] Adjust link line regex to only match at beginning of line

The crawler was starting to run into errors on source code, which some

people are now hosting in Geminispace, and which sometimes has syntax

that includes `=>` of it. I suppose this could have happened in

non-code contexts as well, but this is the first time it seems to have

loudly broken the crawl.

This fixes it.

Also, it occurs to me that I think there is a "raw-text block" type of

construct in the Gemini spec now, so I should probably add a TODO to

refactor the extract_gemini_links function to exclude any links found

within such a block.

77a824e53cb77beb01341f6570c502123132daeb

Author: Natalie Pendragon <natpen@natpen.net>

Date: Tue May 05 08:27:48 2020 -0400

Patch

Message: [crawl] Respect robots.txt crawl_delays and add a kind default

382b6facc678e2df1e4fe8f356a4af691fdeb7d2

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Apr 17 09:24:57 2020 -0400

Patch

Message: Add some TODOs

892b841f86d93848d5d9cfa7c497706fca6d1c84

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Apr 16 18:40:13 2020 -0400

Patch

Message: [serve] Fix bug in displaying "input" results

b6367fb928021ad1f62b88e863c461f7a729ae22

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Apr 16 18:39:31 2020 -0400

Patch

Message: Update dependencies

62002f4b7181d0b5b4da761b03b4dea513213206

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Apr 16 18:19:07 2020 -0400

Patch

Message: [crawl] fix crawl bug with robots.txt

3679863e4070931a32c9be7519707f2dbb4c25d5

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Apr 16 18:18:25 2020 -0400

Patch

Message: [serve] Update formatting

220709fdd46669fb36fdc321dc82f95205fe0b2d

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Mar 14 22:50:06 2020 -0400

Patch

Message: Improve it all

e537dcdb4f9686d26396870a0b263d72c4a04519

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Mar 05 08:55:16 2020 -0500

Patch

Message: [serve] Add seed request tracking

5225d303302a2d57058c4d10e79aa0a998ee518a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Mar 05 07:50:57 2020 -0500

Patch

Message: [serve] Update aesthetics

075832925cf8cf5743603aef3aed10ba5cce0d0f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Mar 04 08:08:48 2020 -0500

Patch

Message: Add search suggestions

02d9691d378fbfde7c672419046fc7cbb40c3a1a

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Mar 04 08:08:03 2020 -0500

Patch

Message: Update indexing and query parsing

7098386ec92e0edeb954782b402b9a8341678fae

Author: Natalie Pendragon <natpen@natpen.net>

Date: Wed Mar 04 08:06:25 2020 -0500

Patch

Message: Add TODO to track freshness of content

cbcbcc59a100083675a04ff2114446fcfdc7fe8e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Mon Mar 02 06:43:56 2020 -0500

Patch

Message: [crawl] Respect "indexer" robots.txt entries

9dc0120b92c51f8945fa642d1405f9a0bc6f4c39

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Mar 01 12:12:28 2020 -0500

Patch

Message: Add more feature ideas to the README

5e86473dc862db309c737f7b53f220fa500f9871

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Mar 01 12:12:16 2020 -0500

Patch

Message: Index and serve mime types

14aa70b3c65c25224721478364331ad468e10f39

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Feb 29 08:33:12 2020 -0500

Patch

Message: Improve README readability

88fa8d44136444f0f24e556da07275b5d50bfd8e

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Feb 29 08:31:15 2020 -0500

Patch

Message: Add README todo to add paging

871d4a0e9cd3aca3c119a1475ba59653f514745f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Feb 29 08:27:39 2020 -0500

Patch

Message: [serve] Remove numbers from search result rows

These were causing visual confusion with the way that some gemini

clients (like bombadillo and av-98) assign numbers to links and print

them in-line, which ends up being right next to these result numbers.

I don't think the result numbers provided much extra value, even when

not causing visual confusion, so this commit simply removes them.

09abe013da9735c2007595fdc835cf928d6798b7

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Feb 29 08:13:22 2020 -0500

Patch

Message: Update README.md

fd3a662f11bf8d1c1a52332951984ada1487c507

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Feb 27 09:06:38 2020 -0500

Patch

Message: Update README

f132b6513725aa76ca3b3884923bb5530ee5c92f

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Feb 27 08:45:55 2020 -0500

Patch

Message: Make GUS easier to run for others

This commit does the following:

1) adds a README with setup instructions

2) updates the dependency specification for gusmobile

to no longer point at a relative directory, which

is very likely unique to how I manage code directories

personally, and instead use a Git reference to the

forked version of gusmobile with the same changes.

For local hacking on the fork of Gusmobile, one should

clone that repository, update the pyproject.toml to

point to it on the local filesystem, and regenerate

their virtualenv.

I also considered simply copying out the relevant

code from the upstream gusmobile, but I have a goal

of maturing the hacks to it into more legit/robust

improvements that can eventually be contributed

back upstream :)

a4c8ddc92b939eb8d644ecc41d61f35e58e213c9

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Feb 23 09:30:52 2020 -0500

Patch

Message: Add some new seed sites

80ad98435233fab6ae7642bb603353d01b577f2b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Fri Feb 21 08:44:01 2020 -0500

Patch

Message: Respect robots.txt

b8f73c5617bd502d6d952b2714eee404d8132eca

Author: Natalie Pendragon <natpen@natpen.net>

Date: Thu Jan 30 08:47:38 2020 -0500

Patch

Message: Initial commit

df9486f3ef44214ad933629f87529d83ff424f65

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Feb 07 08:23:34 2021 -0800

Patch

Message: Add a few more url parsing test cases

788291199d1fc5f29aa12bad051153c4d6b71e66

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sun Feb 07 08:20:26 2021 -0800

Patch

Message: Update to Python 3.9 compatibility

39ca213bb5267140f38a544b2db3cca3f1d05ff2

Author: René Wagner <rwagner@rw-net.de>

Date: Thu Feb 04 21:06:57 2021 +0100

Patch

Message: update python deps

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

44812ee8b57a1adee542c8877cdc4bafce3ec546

Author: René Wagner <rwagner@rw-net.de>

Date: Thu Feb 04 21:05:38 2021 +0100

Patch

Message: introduce systemd-unit for indexer

The indexer is launched by systemd when the crawler finishes.

When launched through the unit, the output to stdout is

redirected to systemd-journald. There's no need for additional

file output, thus it has been removed.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

d9fc0b3d0b0185a1c7f875f12b6fe6b9da51f4fd

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jan 30 07:15:18 2021 -0800

Patch

Message: Make README heading lines more consistent

d4e7acc7aebb1de3cba24a56b0e8a037110da462

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Jan 29 10:08:22 2021 +0100

Patch

Message: add systemd-units for automatic crawling

The template runs the crawler once a week on saturday afternoon.

If other launch times are wanted, gus-crawl.timer needs to be

modified.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

29d601377092b43821d1d019990b193110dab12b

Author: Natalie Pendragon <natpen@natpen.net>

Date: Sat Jan 30 07:05:46 2021 -0800

Patch

Message: Fix trailing whitespace and reformat long string

7b37090a8e17f93ab5ab043a2aa0fe717870cc6e

Author: René Wagner <rwagner@rw-net.de>

Date: Thu Jan 28 11:33:45 2021 +0100

Patch

Message: add "/robots.txt" route to views.py

It's a hard coded approach to serve a robots.txt to other crawlers.

No crawler may access /add-seed & /threads and all relevant virtual agents

may not access /search and /backlinks

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

569baa0637e3d21d7423f28ef38f325ad2b888bc

Author: René Wagner <rwagner@rw-net.de>

Date: Wed Feb 10 11:06:47 2021 +0100

Patch

Message: limit max_crawl_depth to 100 for normal crawl

There are capsules out there that kill the crawler due

to a recursion exceeding the limits of python.

Python limit seems to be around 1000, so the value

can be increased if needed, but i don't think we

miss anything with the current value.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

42af8b76b338baa2fc1f66e1ec5f2d3f91771849

Author: René Wagner <rwagner@rw-net.de>

Date: Wed Feb 10 07:07:12 2021 +0100

Patch

Message: increase frequency to avoid rescanning within a single crawl

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

4817ab31490a9eb9f9587445aa3eff9292e4f21d

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Feb 12 08:53:20 2021 +0100

Patch

Message: add verbose search to robots.txt

This was missing in the first place.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

108bfe850a8439871666b06b7acdbe984aa74e9f

Author: René Wagner <rwagner@rw-net.de>

Date: Fri Feb 12 08:05:34 2021 +0100

Patch

Message: correctly handle robots.txt

Honor the robots.txt entrys of "indexer" and "gus" as well

as the default * section.

The robot_file_map.p must be deleted on a live instance

after this change has been applied to refetch all robots

files, as previously only empty files have been stored.

Signed-off-by: Natalie Pendragon <natpen@natpen.net>

4a885e8db8273f3524ff8a983779aaadb7325204

Author: Duncan Bayne <duncan@bayne.id.au>

Date: Sat Aug 06 20:37:58 2022 +1000

Patch

Message: Upgrade feedparser to 6.0.10

Fixes #49.