It took a couple years, but I have now "engineered" my gemlog.
One thing I used to enjoy about writing in my gemlog is that there was no build step. I just wrote gemtext (oh and also manually updated the feed.. and the index..) and then uploaded it to a server. Beautiful. Simple. Elegant.
I honestly really do love how gemtext bridges the divide between markdown and html. It is neither. And it eliminates the need for both. It feels great to write text in a very minimal, nearly non-existant markup and then just have that be what the client and the server both natively support.
But then the other day rkta asked me on #gemini@tilde.chat if I would consider including the body of each post in my atom feed, and the answer was of course yes.
I'm a big fan of rss/atom/twtxt/gemsub. Feeds and aggregation in whatever form. I consume a lot of it. And I provide a feed for nearly everything I publish online. And whenever I consume a feed, I always appreciate getting the full content, and am always slightly disappointed at getting anything less. So of course I immediately agreed to jazz up my feed so it can achieve its fullest potential.
I must admit that it didn't even cross my mind to do it by hand. Instead I created a little database and a build system. And then vort3, also on #gemini@tilde.chat, asked for details. Hence this post.
I already have been putting a litte metadata block at the end of each post to try to provide a little context in case somebody downloads or saves a post, and later wants to know where and when it came from. I don't know how many gemini readers out there do actually download content for offline reading. But I read a post one time about somebody who does, and they wished more people would include this kind of info in their posts. So now I do.
It looks like this:
title: Reply to Degrowther: On choosing a text editor author: dozens <dozens@tilde.team> url: gemini://tilde.town/~dozens/gemlog/12.gmi created: 2022-07-05T00:00:00-06:00 updated: 2022-07-05T00:00:00-06:00 tags: diary reply text-editors
So I spent a couple minutes making sure each page had complete and consistent metadata. And then I was able to extract that metadata, and the body of the post, and insert it into a recutils database. Because recutils are my favorite.
The schema for a post ended up looking like this:
%rec: gemlog %doc: a gemlog entry %key: id %auto: id created updated %sort: id %type: created,updated date %type: flags enum draft published unlisted %type: id int %unique: title %mandatory: title created body %allowed: id title created updated tags flags body
A little bit of cleanup later, and now I have a database of entries, with actual metadata, from which I can build not only a fully realized feed, and an index of posts, but also the posts themselves! All in one fell swoop.
One thing I'm trying out here with the feed is giving each entry an id equal to the sha256sum of the body of the post. My hypothesis is that when I update a post, the hash will change, giving the feed item a new id, which will cause your rss client to refetch it. Although maybe it already does that just based on the 'updated' attribute? I don't know, I'm trying it. We'll see how it goes.
Feed script:
#!/bin/zsh
FEEDOUT=dist/gemlog/atom.xml
RECFILE=db/db.rec
## An atom feed entry template for recfmt
ENTRYTMPL(){
Timestamp=$1
Sha=$2
cat<<EOF
<entry>
<id>$Sha</id>
<title>{{title}}</title>
<updated>$Timestamp</updated>
<link href='gemini://tilde.town/~dozens/gemlog/{{id}}.gmi' />
<content type="text/gemini">
{{body}}
</content>
</entry>
EOF
}
## Create Feed
Makefeed()(
exec > "$FEEDOUT"
cat<<EOF
<?xml version='1.0' encoding='UTF-8'?>
<feed xmlns="http://www.w3.org/2005/Atom">
<id>gemini://tilde.town/~dozens/gemlog/</id>
<title>dozens gemlog</title>
<updated>$(gdate -Iseconds)</updated>
<author>
<name>dozens</name>
<email>dozens@tilde.team</email>
</author>
<link href="gemini://tilde.town/~dozens/gemlog/atom.xml" rel="self"/>
EOF
ids=$(recsel db/db.rec -e 'flags="published"' -P id -C)
for id in $ids
do
dateorig=$(recsel $RECFILE -e id=$id -P updated)
datenew=$(gdate -Iseconds -d"$dateorig")
sha=$(recsel $RECFILE -e id=$id -P body | sha256sum | cut -d ' ' -f 1)
recsel $RECFILE -e id=$id | recfmt "$(ENTRYTMPL $datenew $sha)"
done
echo "</feed>"
)
Makefeed
Note the ENTRYTMPL function that takes a few parameters and returns a string. This is a pattern that I have been using a lot for recfmt templates because recfmt supports basicaly no scripting or logic. So I outsource any necessary formatting / scripting to the caller and make a sort of meta-template. This has helped with things like the extremely common use-case of formatting a date.
Also while I'm at it, I might as well make those "tags" at the end of each post useful. I'll (create and) link to a page for each tag, containing all the posts with that tag. And also an index of all tags. (See the bottom of this page.)
The page metadata is "live" now and is generated at build time. (See the bottom of this page.)
I also wrote a little wrapper for recins, so I have a little "CLI" now for writing a new post.
#!/bin/zsh read -p "Title?> " Title tmp=$(mktemp) $EDITOR "$tmp" Body=$(< "$tmp") read -p "[p]ublish or [d]raft?> " Flags read -p "tags?> " Tags case "$Flags" in "d") Flags="draft";; "p") Flags="published";; *) echo "Unknown type!"; exit 1;; esac recins --verbose db/db.rec -t gemlog -f title -v "$Title" -f flags -v "$Flags" -f tags -v "$Tags" -f body -v "$Body" created="recsel db/db.rec -e 'title=\"$Title\"' -P id" echo "Created $(eval $created)"
One other thing I decided to tackle here is the problem of incremental builds. For whatever reason, I thought it would be a hard problem to solve. But in actuality, it was quite easy to compare the 'updated' field of a record to the modified time of the generated file. Now the file will only be rebuilt if the db knows it has been updated recently:
for id in $(recsel db/db.rec -e 'flags="published"' -P id)
do
file_updated=$(stat -f "%m" dist/gemlog/$id.gmi)
rec_updated=$(recsel db/db.rec -P updated -e id=$id)
rec_epoch=$(gdate -d"$rec_updated" +'%s')
if [ $file_updated -lt $rec_epoch ]
then
echo "Update is fresher than $id"
else
echo "Skipping $id"
fi
done
So, that's fun! A little bash, a little awk, a little recutils, and now I have a whole system for generating my gemlog posts, indices, tags, feeds, etc. Will I miss the simplicity and austerity of simply writing gemtext by hand for forgoing a build process? Eh, probably not. I mean, I still get to write gemtext. I merely juice it up a little bit now before uploading it. And honestly I love messing around with rec and awk and sed and stuff. So this was fun to hammer out quickly, and I'll probably enjoy continuing to tinker on these tools.
:wq
Thoughts? Comments? Let me know at dozens@tilde.team
Tags:
title: my gemlog is now a recfile author: dozens <dozens@tilde.team> url: gemini://tilde.town/~dozens/gemlog/23.gmi created: 2024-08-09T11:08:32-06:00 updated: 2024-08-09T20:07:55-06:00 tags: gemini recutils