Again on feeds in Gemini format

Emilis <emilis (a) emilis.net>

Hi,

I am developing a generator and a parser for the 
frequently-discussed-but-never-agreed-on feeds in Gemini format.

I would like to share my code, see your similar code and later discuss 
(over IRC?) what format could work best for our use cases.

My code:

Gemlog generator: gemini://tilde.team/~emilis/Makefile
Aggregator: https://tildegit.org/emilis/gmi-feed-aggregator


The motivation (taken from this post: 
gemini://tilde.team/~emilis/2020/11/19-on-feeds-in-gemini-format.gmi ):


I looked through the discussions on Gemini list, read the posts by ~ew0k 
and Drew DeVault.

I wholeheartedly disagree with the opinion that Atom/RSS (or JSON feeds) 
should be enough for everybody.

The point is - some of us are not thinking about running feed 
generators, parsers and aggregators on developer laptops, workstations, 
modern servers we own, etc..

We are thinking about running these programs on computers where we have 
limited permissions, OpenWRT routers, experimental SBCs, old netbooks 
and rooted phones that cannot be updated to any recent distros, etc..

In these situations even Python (widespread as it is) may not be 
available, may be too resource-hungry or may not have the option to be 
updated or extended with libraries.

What we need is the ability to process feeds with a bare minimum of 
tools (e.g. a POSIX shell, BusyBox, etc.). Parsing XML and JSON is not 
feasible in these situations.

Therefore we want a plain Gemini feed format. Seeing how easy it is to 
generate and parse Gemini files with just plain shell script, makes us 
want it badly. We also have hopes it would have more uses than just 
gemlogging.

## What should we do about it

I think we should start by just building the tools for ourselves and 
sharing them (probably on the Gemini list). After we have a few 
implementations, we can discuss on the formal spec between the developers.

The main criteria should probably be the amount of effort and knowledge 
needed to implement a parser.

What I found in the discussions is that this may be the lowest common 
denominator at the moment:

 ```
=> URL ISO-TIMESTAMP TITLE-MAYBE-WITH-AUTHOR
 ```

We can start from this and agree that our parsers will rely on just 
these lines and ignore the rest for the moment. It could be done by this 
command:

 ```
grep -E '^=>\s*gemini://[^ ]+ 
[0-9]{4}-[0-9]{2}-[0-9]{2}(T[0-9]{2}:[0-9]{2}:[0-9]{2}(Z|\+[0-9]{1,2}:[0-9]{2}))?\s+.*



 ```


Looking forward to seeing your implementations. Please share them!


--
Emilis Dambauskas
gemini://tilde.team/~emilis/

Link to individual message.

acdw <acdw (a) acdw.net>

On 11/18/20 7:12 PM, Emilis wrote:

> I wholeheartedly disagree with the opinion that Atom/RSS (or JSON feeds) 
> should be enough for everybody.
> 
> The point is - some of us are not thinking about running feed 
> generators, parsers and aggregators on developer laptops, workstations, 
> modern servers we own, etc..
> 
> We are thinking about running these programs on computers where we have 
> limited permissions, OpenWRT routers, experimental SBCs, old netbooks 
> and rooted phones that cannot be updated to any recent distros, etc..
> 
> In these situations even Python (widespread as it is) may not be 
> available, may be too resource-hungry or may not have the option to be 
> updated or extended with libraries.
> 
> What we need is the ability to process feeds with a bare minimum of 
> tools (e.g. a POSIX shell, BusyBox, etc.). Parsing XML and JSON is not 
> feasible in these situations.

Just as a proof of concept, at least generating Atom feeds using plain 
shell script (plus a little extra) is possible.  See my new project, 
gemshimfeed:

https://git.sr.ht/~acdw/gemshimfeed

It only depends on GNU find and stat, both of which can be replaced 
easily at run time.  Or hell, it's a shell script, you can edit it 
yourself :)

As far as parsing an Atom feed goes, that's a little tricker -- but I 
think gemshimfeed shows it's not hard to generate the feed.  I 
implemented this without even reading the spec, I think; I just read the 
Wikipedia article on the protocol.

-- 
~acdw (Case)
www.acdw.net | breadpunk.club/~breadw

Link to individual message.

Robert khuxkm Miles <khuxkm (a) tilde.team>

November 18, 2020 9:45 PM, "acdw (Case)" <acdw at acdw.net> wrote:

-quote of Emilis' email snipped-
> Just as a proof of concept, at least generating Atom feeds using plain 
shell script (plus a little
> extra) is possible. See my new project, gemshimfeed:
> 
> https://git.sr.ht/~acdw/gemshimfeed
> 
> It only depends on GNU find and stat, both of which can be replaced 
easily at run time. Or hell,
> it's a shell script, you can edit it yourself :)
> 
> As far as parsing an Atom feed goes, that's a little tricker -- but I 
think gemshimfeed shows it's
> not hard to generate the feed. I implemented this without even reading 
the spec, I think; I just
> read the Wikipedia article on the protocol.

Generating a feed was never the hard part. Make a shell script that's a) 
simple and b) parses an Atom feed correctly and then we'll talk about how 
easy parsing an Atom feed with the bare minimum of tools is.

Meanwhile, we don't need to talk about how easy parsing the plain gemtext 
feed is, because it's *that* easy. Of course it's easy to parse gemtext; 
the entire point of gemtext is to be easy to parse.

Just my two cents,
Robert "khuxkm" Miles

Link to individual message.

colecmac@protonmail.com <colecmac (a) protonmail.com>

I still am not a fan of this format[1] and I think your rationale
here doesn't really make sense.

> running feed generators, parsers and aggregators

I will limit this to just parsers, as acdw has pointed out how easy
generation is. And the main difficulty of aggregating is just parsing.

> We are thinking about running these programs on computers where we have
> limited permissions, OpenWRT routers, experimental SBCs, old netbooks
> and rooted phones that cannot be updated to any recent distros, etc..

I don't see how any of these can't run C code that parses feeds. We must
remember, there is a lower bound to anything Gemini can use: it has to
be able to handle TLS. See 2.13 of the FAQ[2].

If your machine can do TLS, I don't see why it can't parse some XML.

> What we need is the ability to process feeds with a bare minimum of
> tools (e.g. a POSIX shell, BusyBox, etc.). Parsing XML and JSON is not
> feasible in these situations.

Not sure exactly what you're picturing, but can you not run compiled code
here? Do you mean the only code you can run is POSIX shell, maybe awk?
What real world environment is this?


1: gemini://makeworld.gq/gemlog/2020-11-16-re_aggregator_idea.gmi
2: gemini://gemini.circumlunar.space/docs/faq.gmi


makeworld

Link to individual message.

Omar Polo <op (a) omarpolo.com>


Emilis <emilis at emilis.net> writes:

> Hi,
>
> I am developing a generator and a parser for the
> frequently-discussed-but-never-agreed-on feeds in Gemini format.
>
> I would like to share my code, see your similar code and later discuss
> (over IRC?) what format could work best for our use cases.
>
> My code:
>
> Gemlog generator: gemini://tilde.team/~emilis/Makefile
> Aggregator: https://tildegit.org/emilis/gmi-feed-aggregator
>
>
> The motivation (taken from this post:
> gemini://tilde.team/~emilis/2020/11/19-on-feeds-in-gemini-format.gmi
> ):
>
>
> I looked through the discussions on Gemini list, read the posts by
> ~ew0k and Drew DeVault.
>
> I wholeheartedly disagree with the opinion that Atom/RSS (or JSON
> feeds) should be enough for everybody.
>
> The point is - some of us are not thinking about running feed
> generators, parsers and aggregators on developer laptops,
> workstations, modern servers we own, etc..
>
> We are thinking about running these programs on computers where we
> have limited permissions, OpenWRT routers, experimental SBCs, old
> netbooks and rooted phones that cannot be updated to any recent
> distros, etc..
>
> In these situations even Python (widespread as it is) may not be
> available, may be too resource-hungry or may not have the option to be
> updated or extended with libraries.
>
> What we need is the ability to process feeds with a bare minimum of
> tools (e.g. a POSIX shell, BusyBox, etc.). Parsing XML and JSON is not
> feasible in these situations.

With all the due respect, this is a bullshit.  Leaving aside that gemini
requires TLS by the spec, even if you don?t have root privileges on a
machine, there is a C compiler on it (how you?re gonna build a shell
without one?), and with that you can build a parser.  (I once wrote a
parser with a friend: a couple of hours of fun pair programming and we
had a JSON parser that was able to parse json objects over a socket,
without blocking etc, and was like ~200-300 lines of c code? ? ok, it
wasn?t 100% complete and spec compliant, but you get the idea).

anyway?

> Therefore we want a plain Gemini feed format. Seeing how easy it is to
> generate and parse Gemini files with just plain shell script, makes us
> want it badly. We also have hopes it would have more uses than just
> gemlogging.

I really like the idea!  A plain (but structured) text/gemini file would
be better IMHO.  It fits the goal of the protocol well: it?s simple,
effective, usable ?as it is? without special client support, easy to
generate (and consume) by humans etc?

> ## What should we do about it
>
> I think we should start by just building the tools for ourselves and
> sharing them (probably on the Gemini list). After we have a few
> implementations, we can discuss on the formal spec between the developers.

I?ve just adapted my static site generator to generate a gemini feed.
The commit for it was just ?3 files changed, 26 insertions, 5
deletions?, as git says.  It?s hosted here

=> gemini://gemini.omarpolo.com/rss.gmi

> The main criteria should probably be the amount of effort and
> knowledge needed to implement a parser.
>
> What I found in the discussions is that this may be the lowest common
> denominator at the moment:
>
> ```
> => URL ISO-TIMESTAMP TITLE-MAYBE-WITH-AUTHOR
> ```
>
> We can start from this and agree that our parsers will rely on just
> these lines and ignore the rest for the moment. It could be done by
> this command:

ignoring anything besides the links can be a good idea.  For instance,
in my feed I?ve included a title and a quote before the list of feeds:
this can improve the human consumption, but can also let advance clients
to improve the UI: for instance, they could (optionally) extract the
title of the page (the first ?#? title) and use it as a prefix for the
titles, or improve the search, or ?

> ```
> grep -E '^=>\s*gemini://[^ ]+
> [0-9]{4}-[0-9]{2}-[0-9]{2}(T[0-9]{2}:[0-9]{2}:[0-9]{2}(Z|\+[0-9]{1,2}:[0-
9]{2}))?\s+.*



> ```
>
>
> Looking forward to seeing your implementations. Please share them!

Link to individual message.

Petite Abeille <petite.abeille (a) gmail.com>



> On Nov 19, 2020, at 02:12, Emilis <emilis at emilis.net> wrote:
> 
> What we need is the ability to process feeds with a bare minimum of 
tools (e.g. a POSIX shell, BusyBox, etc.). Parsing XML and JSON is not 
feasible in these situations.

xmllint & xml2 & jq & etc would do just fine in such context.

[1] http://xmlsoft.org/xmllint.html
[2] https://command-not-found.com/2xml
[3] https://stedolan.github.io/jq/

Link to individual message.

Luke Emmet <luke (a) marmaladefoo.com>


On 19-Nov-2020 01:12, Emilis wrote:
> I wholeheartedly disagree with the opinion that Atom/RSS (or JSON 
> feeds) should be enough for everybody.
>
> The point is - some of us are not thinking about running feed 
> generators, parsers and aggregators on developer laptops, 
> workstations, modern servers we own, etc..
>
> We are thinking about running these programs on computers where we 
> have limited permissions, OpenWRT routers, experimental SBCs, old 
> netbooks and rooted phones that cannot be updated to any recent 
> distros, etc..
>
> In these situations even Python (widespread as it is) may not be 
> available, may be too resource-hungry or may not have the option to be 
> updated or extended with libraries.
>
> What we need is the ability to process feeds with a bare minimum of 
> tools (e.g. a POSIX shell, BusyBox, etc.). Parsing XML and JSON is not 
> feasible in these situations.
>
> Therefore we want a plain Gemini feed format. Seeing how easy it is to 
> generate and parse Gemini files with just plain shell script, makes us 
> want it badly. We also have hopes it would have more uses than just 
> gemlogging.

I see no harm in using Atom as a machine readable format, but from my 
point of view the main downside is expecting end users to author it, or 
have the skills to run some command line application to generate it. 
Probably fine for early adopters, but not so good for "normal" users.

Anyway a while back, I wrote a CGI script that emits Atom by parsing 
plain gmi feed listing. Details are here:

gemini://gemini.marmaladefoo.com/blog/2-Jul-2020_GMI_based_feed_generator.gmi

The parser is in Rebol, but parsing GMI is easy enough in most languages.

If we use raw GMI for feeds, one difficulty perhaps will be how the 
feeds can be discovered, particularly if they have to match some 
specification. But from an end user point of view, GMI is much nicer 
than XML for all the reasons we understand, so I think it is better way 
for authors to express their feed and can be used to generate the XML 
for automated feed clients that need it.

  - Luke

Link to individual message.

Solderpunk <solderpunk (a) posteo.net>

Hi Geminauts,

On Thu Nov 19, 2020 at 1:19 PM CET, Luke Emmet wrote:

> I see no harm in using Atom as a machine readable format, but from my
> point of view the main downside is expecting end users to author it, or
> have the skills to run some command line application to generate it.
> Probably fine for early adopters, but not so good for "normal" users.

I quite agree with this perspective.  The biggest drawback to using Atom
as the standard way to subscribe to serial content on Gemini is not the
difficulty of parsing, but that it requires the authors of such content
to generate the feed file.  I thought this could be addressed with good
tooling, such as my gemfeed program, but real world experience in
helping people use it, and using it myself, has proven this is not
actually straightforward.  The sticking point is the need for
timestamps.  File system timestamps are too easily accidentally changed
to be really useful for this.  Of course the problem is trivially solved
by some kind of static site generator which keeps its own database of
creation/update times and generates the feed, but IMHO it absolutely
should be necessary to use such a system just to make your content
subscribable.  If somebody wants to use such a tool, more power to them,
but ideally one should be able to manage a capsule entirely by hand with
nothing more than a standard text editor if that's what one wants to do,
and still participate in Geminispace as a first class citizen.  Atom
feeds don't fit well with this ideal.

Systems like Spacewalk or Moku Pona (recently updated to support
Gemini!) *do* fit with this ideal.  The content author does not need to
do anything special at all, and users can subscribe to anything at all.
These are extremely desirable properties.  The downside is that these
change-detection systems are fragile (and will erroneously declare a
site to have been updated if all the author did is change a typo or add
some ASCII art above their post list), and provide no meta-data - you
can't see post titles in Spacewalk the way you can in CAPCOM.

I have been thinking for some time now about how to get the best of both
these worlds, and have had Fediverse/email discussions with a few
people about this subject.  Recently I settled on a design and had been
meaning to make some kind of post about it.  Reading this thread, I
realise now I've independently reinvented exactly the same scheme that
Luke Emmet has already implemented!

I would be strongly in favour of some kind of "sidekick standard" (like
robots.txt) defining a process by which *any* text/gemini document can
be interpreted as a feed and subscribed to.  The details would look
roughly like this:


  document is equivalent to the <feed><title> element of an Atom feed.
  All other heading lines are ignored.

  component of the label is a recognised datestamp are treated as
  equivalent to an Atom <entry> element, where:
    - The link's URL provides <entry><link>
    - The datestamp provides <entry><updated>
    - The content of the label after the datestamp provides
      <entry><title>


Ignoring link lines without a datestamp allows for the presence of
"Home" or "Next page in LEO" (I only learned about LEO yesterday - love
it!) links in a subscribable page without having them turn up as entries
in the feed.

All this requires to become a fairly complete spec is a specification of
acceptable datestamp formats.  It seems like Luke Emmet's implementation
of this is flexible in this regard.  I would argue strongly in favour of
supporting only a single format, namely the ISO 8601 format (in which
today is 2020-11-19).  The advantages of this format are:


  maintained by somebody else

  abbreviations like "Jun" or "Mon")


If all feeds use this format, then an aggregator which subscribes to
multiple feeds (and I realise that Luke's tool is *not* one of these,
so the fact that it is flexible on dates should not be perceived as a
failing) can fetch all its feeds and sort the entries it finds from most
recent to oldest without even having to actually parse the datestamps at
all - they can just be treated as dumb strings.  This substantially
simplifies aggregator design compared to a standard which allows
multiple date formats, or which mandates any other date format.

I am aware this approach does not provide as much information as an Atom
feed does (e.g. it cannot convey per-entry authorship), but it should be
adequate for a majority of real-world cases.  Certainly it would be
adequate for something like the CAPCOM and Spacewalk aggregators
currently in popular use.

I don't want to be perceived as forcing this design upon the community,
but I have given it quite some thought and I really think it's hard to
significantly improve upon.  It would allow for aggregators with all the
best parts of CAPCOM (high resistance to false alarm updates, providing
feed and entry titles in aggregated output) with implementation burdens
much closer to Spacewalk, for both aggregator authors and content
authors.  That's a very Gemini kind of solution. :)  A quick browse of
CAPCOM and Spacewalk suggest the majority of gemlogs are already 100%
compatible with this method of subscription (including all those at
gemlog.blue).  Many which aren't could be very quickly and easily made
so (e.g. by changing 2020/11/19 to 2020-11-19), without any nasty side
effects like changing URLs.

So, if somebody can see serious shortcomings with this approach or needs
clarification of some small details, I'm very happy to discuss it, but
otherwise it would be nice to avoid this becoming yet another endless
bikeshedding expedition, and I'd be happy to bless this approach or one
very similar as "official" quite quickly.  CAPCOM is rapidly approaching
100 feeds, and I would love to see the community move toward norms of
people either running their own private aggregators to read only what
they like, and/or people setting up curated public aggregators specific
to particular special interests, rather than encouraging everybody to
read everything that anybody ever says.  Making subscription
substantially easier could help push us in that direction.

Cheers,
Solderpunk

Link to individual message.

Callum Brown <callum (a) calcuode.com>

On Thu Nov 19, 2020 15:44:06 GMT Solderpunk wrote:
> * The text of the first heading line (e.g. # My gemlog) to appear in the
>   document is equivalent to the <feed><title> element of an Atom feed.
>   All other heading lines are ignored.
> * All other lines which are not link lines are ignored.
> * All link lines with labels where the first whitespace-separated
>   component of the label is a recognised datestamp are treated as
>   equivalent to an Atom <entry> element, where:
>     - The link's URL provides <entry><link>
>     - The datestamp provides <entry><updated>
>     - The content of the label after the datestamp provides
>       <entry><title>
> * All other link lines are ignored

I like this approach. The only thing I would point out is that the
example heading "# My gemlog" should probably be "# <author>'s gemlog"
instead so aggregators get some kind of author information.

> I would argue strongly in favour of supporting only a single format,
> namely the ISO 8601 format (in which today is 2020-11-19).

The exact time of posts is rarely relevant, so I think this makes
sense. If authors want to include a timestamp or written date
(e.g. Thursday 19 November 2020) I suppose they could put that in the
title section but it may turn out a bit messy.

Callum

Link to individual message.

Jason McBrayer <jmcbray (a) carcosa.net>

"Solderpunk" <solderpunk at posteo.net> writes:

> All this requires to become a fairly complete spec is a specification
> of acceptable datestamp formats. It seems like Luke Emmet's
> implementation of this is flexible in this regard. I would argue
> strongly in favour of supporting only a single format, namely the ISO
> 8601 format (in which today is 2020-11-19). The advantages of this
> format are:

Should the datestamp be *just* a date? Or should it include a timestamp?
This would be useful mainly for posts in the same day that have some
natural ordering (ie replies to one another). If it has a timestamp,
should it be 2020-11-19T17:01:26+00:00, 2020-11-19T17:01:26Z, or
20201119T170126Z, or are any of these good?

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

Link to individual message.

Drew DeVault <sir (a) cmpwn.com>

On Thu Nov 19, 2020 at 12:59 PM EST, Jason McBrayer wrote:
> Should the datestamp be *just* a date? Or should it include a timestamp?
> This would be useful mainly for posts in the same day that have some
> natural ordering (ie replies to one another). If it has a timestamp,
> should it be 2020-11-19T17:01:26+00:00, 2020-11-19T17:01:26Z, or
> 20201119T170126Z, or are any of these good?

Timestamps are not human readable and your use-case is not a feed.

NACK.

Yes, it will break use-cases. Thus is the wisdom of Gemini.

Link to individual message.

colecmac@protonmail.com <colecmac (a) protonmail.com>

Overall this seems like a great solution, thanks a lot for chiming in.

> So, if somebody can see serious shortcomings with this approach or needs
> clarification of some small details, I'm very happy to discuss it, but
> otherwise it would be nice to avoid this becoming yet another endless
> bikeshedding expedition, and I'd be happy to bless this approach or one
> very similar as "official" quite quickly.

Please. I would be happy to see this discussion come to a rest. I'd appreciate
if you could publish a spec for this on the Web and Gemini, and maybe something
in the guide and/or FAQ to help onboard new authors, and explain whether they
should do Atom too, etc. I understand publishing the spec would probably create
more discussion, but at least it would be in a more helpful direction.

I've been told ew0k is about to send an email covering timestamps, which would
be the biggest issue I see. The largest timezone difference in the world is 26
hours, and even for smaller differences, taking dates as UTC can mess with
continuity, and have replies show up before the original posts, or have day
specific posts (like for a holiday) show up as on the wrong day.

> I would argue strongly in favour of supporting only a single format, namely
> the ISO 8601 format (in which today is 2020-11-19).

To further make my point and be annoying, I'd like to point out that today is
2020-11-20 for all the Australians out there. ;)


The other small issue I see is people who naturally separate the date and post
with some other character. Like this:

2020-02-01 - Some other post
2020-01-01 - My first post


This would mean their title starts with a dash. Not great, but I'd say that's a
acceptable issue to have. It would be nice if the spec excluded certain characters
from the beginning maybe, like all whitespace, dashes, colons, etc.


Cheers,
makeworld

Link to individual message.

acdw <acdw (a) acdw.net>

On 2020-11-19 (Thursday) at 18:09, colecmac at protonmail.com wrote:

> I've been told ew0k is about to send an email covering timestamps, which would
> be the biggest issue I see. The largest timezone difference in the world is 26
> hours, and even for smaller differences, taking dates as UTC can mess with
> continuity, and have replies show up before the original posts, or have day
> specific posts (like for a holiday) show up as on the wrong day.

I honestly don't see the problem with this.  As has been pointed out, 
CAPCOM and Spacewalk, and as has been said, this format, all have 
"problems" with ordering replies before the original post.  I've never not 
been able to figure out which actually comes first.

Now that I'm thinking about it, the same problem comes up with Twitter and 
other social medias.  People are smart.  One thing I like about Gemini is 
it's a protocol that assumes people are smart.  We can figure things out.

Let's not over-engineer something because we think people *might* be 
confused for 1/2 of a second.

-- 
~ acdw
acdw.net | breadpunk.club/~breadw

Link to individual message.

Robert khuxkm Miles <khuxkm (a) tilde.team>

November 19, 2020 12:59 PM, "Jason McBrayer" <jmcbray at carcosa.net> wrote:

> Should the datestamp be *just* a date? Or should it include a timestamp?
> This would be useful mainly for posts in the same day that have some
> natural ordering (ie replies to one another). If it has a timestamp,
> should it be 2020-11-19T17:01:26+00:00, 2020-11-19T17:01:26Z, or
> 20201119T170126Z, or are any of these good?

Seconding this. Post order *within* a day is important when it comes to 
post replies. While this may not be a feed usecase, it IS a feed 
aggregator usecase, which should be considered. Even in the case of, say, 
just a client feed reader, it'd be easier to handle posts and their 
replies if there was some sort of intra-day sort.

I would say that timestamps should be restricted to a certain timezone 
(i.e; UTC), just to maintain that idea of "you can just treat this date as 
a dumb string".

Just my two cents,
Robert "khuxkm" Miles

Link to individual message.

Luke Emmet <luke (a) marmaladefoo.com>

On 19-Nov-2020 18:09, colecmac at protonmail.com wrote:
> The other small issue I see is people who naturally separate the date 
> and post
> with some other character. Like this:
>
> 2020-02-01 - Some other post
> 2020-01-01 - My first post
>
>
> This would mean their title starts with a dash. Not great, but I'd say that's a
> acceptable issue to have. It would be nice if the spec excluded certain characters
> from the beginning maybe, like all whitespace, dashes, colons, etc.

I support this optional hyphen (not sure the others are really needed 
though tbh). My own implementation does this. It feels to me a very 
natural way to write the gemlog entry title.

I'll have to get my mind around the year/month/date format - it feels 
less natural since no one uses that in normal speech, but I'll 
definitely get there if this is conventionalised. I see it is probably 
the best way to prevent ambiguity about whether 2/1/2020 is 2nd Jan or 
1st of Feb.

On the timestamp issue, I think we can live without them, but if some 
users really want to put them in for disambiguation, we shouldn't shout 
it down. But please the *recommendation* should be just to use the date 
in almost all instances.

  - Luke

Link to individual message.

colecmac@protonmail.com <colecmac (a) protonmail.com>

On Thursday, November 19, 2020 1:16 PM, acdw <acdw at acdw.net> wrote:

> On 2020-11-19 (Thursday) at 18:09, colecmac at protonmail.com wrote:
>
> > I've been told ew0k is about to send an email covering timestamps, which would
> > be the biggest issue I see. The largest timezone difference in the world is 26
> > hours, and even for smaller differences, taking dates as UTC can mess with
> > continuity, and have replies show up before the original posts, or have day
> > specific posts (like for a holiday) show up as on the wrong day.
>
> I honestly don't see the problem with this. As has been pointed out, 
CAPCOM and Spacewalk, and as has been said, this format, all have 
"problems" with ordering replies before the original post. I've never not 
been able to figure out which actually comes first.
>
> Now that I'm thinking about it, the same problem comes up with Twitter 
and other social medias. People are smart. One thing I like about Gemini 
is it's a protocol that assumes people are smart. We can figure things out.
>
> Let's not over-engineer something because we think people might be 
confused for 1/2 of a second.

I agree, I'm not saying to dump the whole idea. I just think that if there's
a solution to be found to this, let's find it. If not, well that's too bad,
and we can use this but also encourage people to support Atom where possible.

Maybe a solution to have an *optional* line that reads: "Timezone: EST" or
something similar? Maybe "Timezone: America/Toronto" instead?

That way authors can specify it, but existing pages will work fine and just
be interpreted as UTC.


makeworld

Link to individual message.

Solderpunk <solderpunk (a) posteo.net>

On Thu Nov 19, 2020 at 6:59 PM CET, Jason McBrayer wrote:

> Should the datestamp be *just* a date? Or should it include a timestamp?
> This would be useful mainly for posts in the same day that have some
> natural ordering (ie replies to one another). If it has a timestamp,
> should it be 2020-11-19T17:01:26+00:00, 2020-11-19T17:01:26Z, or
> 20201119T170126Z, or are any of these good?

I did wonder about this.  ISO 8601 does make the time format clear.
However, a big part of the appeal for me of this approach is that it
allows a single text/gemini index to function for both machine
subscription and ordinary reading by human eyeballs.  I agree with Drew
that full blown timestamps like 2020-11-19T17:01:26+00:00 are not very
human readable.  Requiring this would make gemlog indices look ugly and
cluttered for human users, which would be a shame - especially as the
time information does not strike me as terribly important.  The
majority of phlogs and gemlogs do not receive more than one update a
day.  In the majority of exceptions to this, the two posts on the same
day do not make important references to each other and so reading them
out of order is not a big deal.  It's true this may cause occasional
problems with response threads.  But scope for serious confusion doesn't
appear until we have multiple posts in the same thread by the same
author on the same day, and I don't think it's the worst thing in the
world if some features of Geminispace act to discourage that kind of
high-paced discussion.  Most of us are here to discover a slower way of
computing, and there's always mailing lists or IRC for lengthy
discussions.

So, I think we'll probably be just fine with date stamps only.

Cheers,
Solderpunk

Link to individual message.

Rohan Kumar <rkumar (a) rkumar-dekstop>

On Thu, Nov 19, 2020 at 04:44:06PM +0100, Solderpunk wrote:
>I quite agree with this perspective.  The biggest drawback to using Atom
>as the standard way to subscribe to serial content on Gemini is not the
>difficulty of parsing, but that it requires the authors of such content
>to generate the feed file.  I thought this could be addressed with good
>tooling, such as my gemfeed program, but real world experience in
>helping people use it, and using it myself, has proven this is not
>actually straightforward.

It is true that authors generally have to run a program to generate an 
RSS/ATOM feed, but how is this different from self-hosting Spacewalk? 
Either way, if you want to own your feed, you'll have to run a command. 
The difference seems to be that the former (plain RSS/ATOM) takes the 
static-site-gen route while the latter (Spacewalk & co.) takes the route 
of dynamically-generated content. The merits and use-cases for static 
and dynamic sites is a slightly different discussion that's been going 
on for decades.

>If somebody wants to use such a tool, more power to them, but ideally 
>one should be able to manage a capsule entirely by hand with nothing 
>more than a standard text editor if that's what one wants to do, and 
>still participate in Geminispace as a first class citizen.  Atom feeds 
>don't fit well with this ideal.

I agree, but I don't think this means that we should move away from Atom 
feeds. I think that exposing an Atom feed (or newsletter) to let users 
get new posts sent to them and saved offline is a bit different than 
having a Gemini service run by someone else aggregate links. The two can 
co-exist without competing.

That being said, I find the new proposals interesting. I'm more than 
willing to revise my statements if they prove to be better. But having a 
single program (my RSS/Atom reader) from which to get updates from all 
the *logs I follow is something I hope doesn't go away. It's the main 
reason why I use Sloum's bombadillo as my main client, since it mostly 
eliminates the need to juggle multiple clients for multiple protocols.

TLDR: the new alternatives to RSS/Atom are nice, and I hope they succeed 
without making me use more than one client for browsing and one client 
for fetching updates across the text-based Internet.

Link to individual message.

colecmac@protonmail.com <colecmac (a) protonmail.com>

> It is true that authors generally have to run a program to generate
> an RSS/ATOM feed, but how is this different from self-hosting Spacewalk?

There are more content authors then there are people who want to self-host
aggregators. The proposal helps content authors who want their posts to
be interpreted properly by public aggregators without having to generate
an Atom feed themselves, which can be difficult.


makeworld

Link to individual message.

Björn Wärmedal <bjorn.warmedal (a) gmail.com>

So I wrote a gemlog about a gemini feed a few days ago, way before I
even knew it was an ongoing discussion:
gemini://tilde.team/~ew0k/is-this-aggregator-idea-good.gmi

Obviously I'd given the matter a thorough 10 or so minutes before
sharing my thoughts :) I've thought a lot more on it since, and
received a great amount of feedback.

The format I proposed was this:

=> FULLURLTOPOST TIMESTAMP, AUTHOR NAME, TITLE
For example:
=> gemini://billsdomain.com/gemlog/posts/is-this-aggregator-idea-good.gmi
2020-11-14T22:03:12+00:00, Bill P. Jones, Is This Aggregator Idea
Good?

Obviously my intended use case is *very* specific: I intended for a
feed aggregator to in turn produce a feed of the same format that it
was parsing to begin with, in order to allow recursive aggregation. In
that case the author's name becomes important, but if my "feed" is a
bunch of links from my own index page then putting my own name on
every link looks weird and feels redundant.

And timestamp... well... I hate it when posts end up in the wrong
order, but I have issues with the timestamp format I proposed myself
too:
- it's hardly human readable. Too long.
- The inclusion of a timezone makes lexicographic sort fallible. I
don't see how we can make everyone set a timestamp in UTC, though.
People will (and should, since it feels more natural to them) use
their own local time.

I don't want to be bikeshedding this issue, and I clearly see how my
own proposed format has flaws that are insurmountable if we want to
gain widespread adoption.

However... I don't see how an overly simplified format makes any sense
at all. I'd rather parse a bunch of atom feeds than aggregate a bunch
of link lists that will end up looking like this:

[1] 2020-11-21 - Re: My thoughts
[2] 2020-11-20 - My thoughts
[3] 2020-11-20 RE: My thoughts
[4] 2020-11-19 ~ew0k, A post!
[5] 2020-11-19 Diary Entry, ~someone

Cheers,
~ew0k

Link to individual message.

Drew DeVault <sir (a) cmpwn.com>

On Thu Nov 19, 2020 at 1:44 PM EST, Bj?rn W?rmedal wrote:
> However... I don't see how an overly simplified format makes any sense
> at all. I'd rather parse a bunch of atom feeds than aggregate a bunch
> of link lists that will end up looking like this:
>
> [1] 2020-11-21 - Re: My thoughts
> [2] 2020-11-20 - My thoughts
> [3] 2020-11-20 RE: My thoughts
> [4] 2020-11-19 ~ew0k, A post!
> [5] 2020-11-19 Diary Entry, ~someone

This has been rubbing me the wrong way for a while now, and I mentioned
it here:

gemini://drewdevault.com/2020/11/15/RE-Is-this-aggregator-idea-good.gmi

Gemini is simply not well-suited to discussions like this. There's a
reason this ended up on the mailing list instead of continuing to
pollute our gemlogs.

We don't need Gemini to solve every problem. In fact, we shouldn't want
it to. This is one of the main appeals of Gemini.

So yes, lack of timestamps makes this use-case break. So what? Every
use-case isn't entiteld to working well on Gemini.

Link to individual message.

John Cowan <cowan (a) ccil.org>

On Thu, Nov 19, 2020 at 1:01 PM Drew DeVault <sir at cmpwn.com> wrote:

Timestamps are not human readable


Lots and lots of people around the world are extremely used to dates
written 2020-11-19, notably in China where it is the national standard.
It's very easy for U.S. folks to interpret as a variant of their
11/19/2020, and probably not all that much work for Europeans (yes, the UK
is in Europe, just like Switzerland) with their 19/11/2020 either, as it is
consistently big-endian whereas the native format is consistently
little-endian.  A number of countries are switching over, at least for
official purposes.

I think it is very important that timestamps are sent in Universal Time
only.  Resolving local times, especially after the fact, is a Very Big Deal
involving a complex C library and timezone data files that change at least
once a year and typically more often.

But I do favor standard punctuation, so I would support the following fixed
format:  2020-11-19T18:12:00Z where Z means Universal Time.  20
characters.  Period.  If you don't have time information, use 00:00:00Z.

Feed aggregators should probably add the feed name in square brackets after
the date and before the title, so that the user can peruse the "Famous
Foovians" feed while ignoring links to the "Foovian Politicians" gemlog,
say.  This would be obtained from the first-level heading in the feed
file.  The only reason for standardizing the square brackets is so as not
to have two entries on second-level aggregations.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
Your worships will perhaps be thinking that it is an easy thing
to blow up a dog? [Or] to write a book?
    --Don Quixote, Introduction

Link to individual message.

Emilis Dambauskas <emilis (a) runbox.com>

2020-11-19 20:43 GMT+02:00 colecmac at protonmail.com:
>> It is true that authors generally have to run a program to generate
>> an RSS/ATOM feed, but how is this different from self-hosting Spacewalk?
> 
> There are more content authors then there are people who want to self-host
> aggregators. The proposal helps content authors who want their posts to
> be interpreted properly by public aggregators without having to generate
> an Atom feed themselves, which can be difficult.

I agree that it could be the job of aggregators to convert between the 
feed formats and re-share the feeds.

The Gemini feed format then could be more oriented to be human readable and writable.

The ISO8601 timestamps could be both short (e.g. 2020-11-19) and long, UTC 
based (e.g. 2020-11-19T18:51:31+00:00).

It would be up to the author to order the post links in order.

The format would work more like an index and would prevent the common 
problems with using file creation/modification times.

The aggregators could then take these index files/feeds and fill in the 
missing time parts if needed and according to their own implementations 
(e.g. keep a database of when links appeared / add "T00:00:00Z" to the 
date before sorting and so on).

--
Emilis Dambauskas

Link to individual message.

Solderpunk <solderpunk (a) posteo.net>

On Thu Nov 19, 2020 at 7:45 PM CET, Drew DeVault wrote:

> This has been rubbing me the wrong way for a while now, and I mentioned
> it here:
>
> gemini://drewdevault.com/2020/11/15/RE-Is-this-aggregator-idea-good.gmi
>
> Gemini is simply not well-suited to discussions like this.

I'm not sure I agree that gemlog posts written in response to other
gemlog posts is not a good format, but the idea that something like feed
aggregation is the best way to follow this kind of discussion, and that
it should reliably provide a time-ordered list of all contributions to
the discussion, is clearly problematic.  It doesn't scale.  It's worked
quite well so far in both Gopherspace and Geminispace, but only because
both are so small that services like CAPCOM which follow everything are
still manageable.  If growth continues at current rates this will soon
start breaking down.  CAPCOM will become an indigestible fire hose.
People will start following only their own small set of feeds, and once
that happens the probability that you see anything other than a small
random slice of the conversation becomes very small and the whole thing
breaks even if posts appear in precisely the right order.

I still think it's a good idea when writing a non-trivial response to
something in Geminispace to reach out to the original author via email,
XMPP, Fediverse, whatever, to let them know.  Authors should feel free
to add a list of links at the end of their posts to responses which they
think are worthwhile interested readers reading.  People can pick up and
follow threads of conversation this way if they want.

I don't want to tell anybody how to live their life, but Gemini indeed
can't work equally well for all use-cases, and I personally (and I get
the impression most people in the Gemini community would agree) would be
happier if we designed things to work best for the case of somebody who
has subscribed to a dozen or so gemlogs on things they're particularly
interested in or by people they think of as friends, rather than for
somebody who feels a desperate need to read every last thing anybody
on the internet says in response to something.

Cheers,
Solderpunk

Link to individual message.

Emilis <emilis (a) emilis.net>

On 11/19/20 8:43 PM, colecmac at protonmail.com wrote:
>> It is true that authors generally have to run a program to generate
>> an RSS/ATOM feed, but how is this different from self-hosting Spacewalk?
> There are more content authors then there are people who want to self-host
> aggregators. The proposal helps content authors who want their posts to
> be interpreted properly by public aggregators without having to generate
> an Atom feed themselves, which can be difficult.
I agree that it could be the job of aggregators to convert between the 
feed formats and re-share the feeds.

The Gemini feed format then could be more oriented to be human readable 
and writable.

The ISO8601 timestamps could be both short (e.g. 2020-11-19) and long, 
UTC based (e.g. 2020-11-19T18:51:31+00:00).

It would be up to the author to order the post links in order.

The format would work more like an index and would prevent the common 
problems with using file creation/modification times.

The aggregators could then take these index files/feeds and fill in the 
missing time parts if needed and according to their own implementations 
(e.g. keep a database of when links appeared / add "T00:00:00Z" to the 
date before sorting and so on).

--
Emilis Dambauskas

Link to individual message.

marc <marcx2 (a) welz.org.za>

Hi

Would it be useful to include something which tells
the software polling the feed when the next post could
be expected ? It doesn't imply an obligation, it is
just a hint...

I concede this adds more complexity in the markup, but
has the benefit that those polling the site don't need to hammer
it as much - that might be important as tls connections
are moderately expensive, and many sites run on very
small, shared or personal systems.

Maybe just the first/last link (pointing nowhere, or
to itself or some other distinguished name) with a timestamp
in the future could tell us when to poll again,
and if there is no future timestamp, the polling interval
could be some sort of backoff (eg the difference
between the newest post and the current time) so that
abandoned sites don't get polled needlessly, while
those wanting to be polled often just update their
future timestamps regularly ?

regards

marc

Link to individual message.

Sudipto Mallick <smallick.dev (a) gmail.com>

What do you think about this alternate format?

# <title of the feed>
## (by)? <author>
bla bla...
### YYYY-MM-DD
=> <link> <title> (HH:MM)?
=> <link> <title> (HH:MM)?
.
.
.
### YYYY-MM-DD
=> <link> <title> (HH:MM)?
.
.
.

--- gemini feed ---
EOF
I think this is more human readable. And that --- gemini feed --- or
anything like that which will tell the crawlers etc. that this is a
feed.

Link to individual message.

colecmac@protonmail.com <colecmac (a) protonmail.com>

On Thursday, November 19, 2020 2:30 PM, Sudipto Mallick <smallick.dev at 
gmail.com> wrote:

> What do you think about this alternate format?
<snipped>
> I think this is more human readable. And that --- gemini feed --- or
> anything like that which will tell the crawlers etc. that this is a
> feed.

The main advantage of Solderpunk's proposal was that it works with
existing pages and feeds. There's no point in creating this new format
if it needs to be written in a strict way that isn't already in use.

makeworld

Link to individual message.

Dan <dan (a) palm93.com>

On Thursday, 19 November 2020 19:30:38 GMT Sudipto Mallick wrote:
> What do you think about this alternate format?

> I think this is more human readable. And that --- gemini feed --- or

It's human readable yeh, but is this not _just as hard_ as atom to parse?

Having the information line-by-line ( link by link in this case) keeps parsing 
insanely simple. 

- Dan.

Link to individual message.

Rohan Kumar <seirdy (a) seirdy.one>

On Thu, Nov 19, 2020 at 08:08:35PM +0100, Solderpunk wrote:
>It doesn't scale [...] CAPCOM will become an indigestible fire hose.  
>People will start following only their own small set of feeds, and once 
>that happens the probability that you see anything other than a small 
>random slice of the conversation becomes very small and the whole thing 
>breaks even if posts appear in precisely the right order.
Agreed; I think that we should split feeds into two use-cases: personal 
use for receiving new content from selected gemlogs (RSS/ATOM or 
similar), and discovery of new content. For the latter, I think 
blogrolls/gemrings are the way to go. Thoughts?

(Also my neomutt config on this machine was out of date, and I've been 
sending the wrong FROM: header this whole time. Ugh.)

/Seirdy

Link to individual message.

Solderpunk <solderpunk (a) posteo.net>

On Thu Nov 19, 2020 at 8:09 PM CET, marc wrote:

> Would it be useful to include something which tells
> the software polling the feed when the next post could
> be expected ? It doesn't imply an obligation, it is
> just a hint...

...

> Maybe just the first/last link (pointing nowhere, or
> to itself or some other distinguished name) with a timestamp
> in the future could tell us when to poll again,

This would ruin the human-readability of the page, though.  I would much
rather we just included a recommendation that pages not be polled more
than two or three times a day.

I do think it would be neat if feed readers tried to learn this
themselves, by taking note of how often polls result in something new...

Cheers,
Solderpunk

Link to individual message.

Nathan Galt <mailinglists (a) ngalt.com>



> On Nov 19, 2020, at 7:44 AM, Solderpunk <solderpunk at posteo.net> wrote:
> 
> I would argue strongly in favour of
> supporting only a single format, namely the ISO 8601 format (in which
> today is 2020-11-19).

You mean like 2020-11-19T23:48:17Z? Maybe including milliseconds, too? Or 
maybe something like 2020-11-19T15:49:57-0800?

I remember seeing a list of "ISO 8601" formats that this one newsreader 
had to deal with. The list of formats was about a dozen elements long.

Link to individual message.

Rohan Kumar <seirdy (a) seirdy.one>

On Thu, Nov 19, 2020 at 03:51:43PM -0800, Nathan Galt wrote:
>> On Nov 19, 2020, at 7:44 AM, Solderpunk <solderpunk at posteo.net> 
>> wrote:
>>
>> I would argue strongly in favour of
>> supporting only a single format, namely the ISO 8601 format (in which
>> today is 2020-11-19).
>
>You mean like 2020-11-19T23:48:17Z? Maybe including milliseconds, too?  
>Or maybe something like 2020-11-19T15:49:57-0800?
>
>I remember seeing a list of "ISO 8601" formats that this one newsreader 
>had to deal with. The list of formats was about a dozen elements long.

It might be a better idea to use RFC 3339, a profile of the ISO 8601 
date format.  See https://tools.ietf.org/html/rfc3339.
It was also mentioned in the Wikipedia article for ISO 8601: 
https://en.wikipedia.org/wiki/ISO_8601. Its main difference, according 
to section 5.6 of the spec:

>NOTE: ISO 8601 defines date and time separated by "T".
>Applications using this syntax may choose, for the sake of
>readability, to specify a full-date and full-time separated by
>(say) a space character.

It also has a provision for unknown local offsets:

>If the time in UTC is known, but the offset to local time is unknown,
>this can be represented with an offset of "-00:00".  This differs
>semantically from an offset of "Z" or "+00:00", which imply that UTC
>is the preferred reference point for the specified time.  RFC2822
>[IMAIL-UPDATE] describes a similar convention for email.

If you aren't familiar with either format but you have GNU date 
installed, you can give them both a spin. For ISO 8601:

	date -I{date,hours,minutes,seconds,ns}

For RFC 3339:

	date --rfc-3339={date,seconds,ns}

Link to individual message.

Emilis <emilis (a) emilis.net>

On 11/19/20 5:44 PM, Solderpunk wrote:
> * The text of the first heading line (e.g. # My gemlog) to appear in the
>    document is equivalent to the <feed><title> element of an Atom feed.
>    All other heading lines are ignored.
> * All other lines which are not link lines are ignored.
> * All link lines with labels where the first whitespace-separated
>    component of the label is a recognised datestamp are treated as
>    equivalent to an Atom <entry> element, where:
>      - The link's URL provides <entry><link>
>      - The datestamp provides <entry><updated>
>      - The content of the label after the datestamp provides
>        <entry><title>
> * All other link lines are ignored

So I went and updated my aggregator code to match this format. Then went 
to SpaceWalk to look for pages to parse.

The output:
gemini://tilde.team/~emilis/aggregated-test.gmi

The code:
https://tildegit.org/emilis/gmi-feed-aggregator

Some notes:

1. I added support for relative URLs to my aggregator. At the moment it 
became the most complex part of the code.

2. Due to relative URL support it is possible to just parse Home/index 
pages of some of the gemlogs -- no separate feed file is needed.

3. Some gemlogs are missing the h1 or it is too generic ("Blog", 
"Updates"). I skipped those.

4. Some gemlogs have characters added to date. E.g. "2020-11-19: And 
then the title". I had to skip these too.

5. I removed the timestamps from link text and prefixed the feed-titles 
to the entry-titles to separate between sources. Seems OK, but one may 
need to hover over the link to be sure of where the link is coming from. 
I think I could add hostnames to link text. Not sure if link text will 
become too long.


--
Emilis Dambauskas

Link to individual message.

cbabcock@asciiking.com <cbabcock (a) asciiking.com>

November 19, 2020 12:08 PM, "Solderpunk" <solderpunk at posteo.net> wrote:

> On Thu Nov 19, 2020 at 7:45 PM CET, Drew DeVault wrote:
> 
>> This has been rubbing me the wrong way for a while now, and I mentioned
>> it here:
>> 
>> gemini://drewdevault.com/2020/11/15/RE-Is-this-aggregator-idea-good.gmi
>> 
>> Gemini is simply not well-suited to discussions like this.
> 
> I'm not sure I agree that gemlog posts written in response to other
> gemlog posts is not a good format, but the idea that something like feed
> aggregation is the best way to follow this kind of discussion, and that
> it should reliably provide a time-ordered list of all contributions to
> the discussion, is clearly problematic. It doesn't scale. It's worked
> quite well so far in both Gopherspace and Geminispace, but only because
> both are so small that services like CAPCOM which follow everything are
> still manageable. If growth continues at current rates this will soon
> start breaking down. CAPCOM will become an indigestible fire hose.
> People will start following only their own small set of feeds, and once
> that happens the probability that you see anything other than a small
> random slice of the conversation becomes very small and the whole thing
> breaks even if posts appear in precisely the right order.

You can't always give everyone what they want, but this might be an 
exception. Use date-only on the original post. Use T and time - minute 
granularity, author is responsible for converting to UTC, no TZ indicator 
needed or allowed (in the spirit of 'we use this spec, but not all of it') 
- when referencing a reply. Original posts will always sort ahead of 
replies, while replies generally sort in order and can be easily removed from a feed

Chris

Link to individual message.

Jason McBrayer <jmcbray (a) carcosa.net>

"Solderpunk" <solderpunk at posteo.net> writes:

> So, I think we'll probably be just fine with date stamps only.

Fair. A feed reader application that wanted to reconstruct reply-to
information could use GUS to do it.

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

Link to individual message.

Jason McBrayer <jmcbray (a) carcosa.net>

"Solderpunk" <solderpunk at posteo.net> writes:

> This would ruin the human-readability of the page, though.  I would much
> rather we just included a recommendation that pages not be polled more
> than two or three times a day.
>
> I do think it would be neat if feed readers tried to learn this
> themselves, by taking note of how often polls result in something new...

You could probably just use exponential back-off for this, without
the feed reader really having to 'learn' anything.

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

Link to individual message.

---

Previous Thread: [ANN] Gemini historical snapshot

Next Thread: [ANN] seirdy.one live on Geminispace (+questions)