Attál gemininauts, Gemini appeals to me as someone who likes plain text and keeping things simple. It also evokes a huge nostalgia for the days of gopher, bbs and slow packet radio links. Suppose there were a few shielded and solar powered Raspberry Pico based Gemini servers in orbit with radio transmitter and receiver. The long distance bandwidth would be a scarce resource and timeouts can happen frequently. There's HTTP has compression, BBS has zmodem, there's the Squid proxy. Caching and compression have been discussed before. I want to focus on compression as it surprises me as the remarks seem to be to dismissive even though compression was a huge thing in the early days of popular data networks (fidonet, bbs, simtel). On March 10, 2021 Bradley D. Thornton remarks "neither of those two things (Accessibility or gzip compression) have anything to do with, nor have any place in discussions, for spec changes" and argues:
On 17/06/2021 14:24, Francis Siefken wrote: > How would people solve such use cases elegantly and within the design philosophy? > In my personal opinion, the best solution would be to simply have it as a separate MIME-type, something along the lines of "text/gemini+gzip". I don't recall how this works with MIME-types, but it should be something like this. I do support your idea, though. I think compression would be a nice addition, but it doesn't belong in the protocol itself, but the file format. Cheers, ~almaember -- Unless you're replying to me on the Gemini mailing list, reply to almaember@almaember.com instead. Website: https://almaember.com/ Gemini capsule: gemini://almaember.com/ IRC: almaember on Libera.chat and tilde.chat
Hello, Francis Siefken <fsiefken@gmail.com> writes: > Attál gemininauts, > >snip< > One simplistic idea is to check if the gmi is binary, and > assume it has been compressed with zstd and then automatically > decompress it in the client. It'd be just one extra step in > the gemini client. To assess thethe speed and data savings I > compressed a short story by Gene Wolfe with zstandard and > lpaq9, a page or so, representative of an average gemtext. The > compression and decompression speeds are below 30ms here. I > didn't precompress with a dictionary (for added significant > size reduction) as that would mean detecting the language > (from for example word frequency). Ideally the gmi would be > compressed one time on the server to avoid compressing (and > straining the solar powered pico on space) for each request. > > 11390 xicygnus.gmi > 4098 xicygnus.gmi.0.lpaq9 > 4855 xicygnus.gmi.13.zst > 5146 xicygnus.gmi.3.zst > > How would people solve such use cases elegantly and within the > design philosophy? Well, imho you do not neccessarily need any changes to the protocol or the gemtext specs. Noone stops you from:
On Thu, Jun 17, 2021 at 04:34:54PM +0200, ew.gemini wrote: > Well, imho you do not neccessarily need any changes to the > protocol or the gemtext specs. > > Noone stops you from: > > * writing an article (arbitrary size, consider it big for the > argument) > => filename.gmi > * compressing this file with xz, say > => filename.gmi.xz > * adding an abstract which describes the article > => filename-abstract.gmi > * adding a link to the abstract > => filename.gmi.xz read more [compressed] > * the abstract is added to your feed and/or index.gmi > > The the user will decide whether to follow the link. > The client software might be able to uncompress the downloaded > file and display it similar to LaGrange displaying image as if > they were "inline". > > > Please bear in mind that the gemini protocol does not even > announce the length of the content to follow. > > > I would go the same route for extented text where I want to > control the presentation. Use \LaTeX, produce a .pdf file, offer > a download link in an plain/text abstract file, which is > indexed. No need for "complex machinery". > > Cheers, > ~ew > This is basically how I'd go, although I'd lean towards gzip over xz for being much kinder to low-memory systems. Anything that can't be done on a 68030 or an 80386SX doesn't belong in any of the core Gemini standards, or even de-facto standards like how to serve compressed gemtext. See this StackOverflow posting for more details on the justification. https://stackoverflow.com/questions/6493270/why-is-tar-gz-still-much-more-c ommon-than-tar-xz -- tidux@sdf.org SDF Public Access UNIX System - http://sdf.org
Hi ew, thanks for coming up with an interesting alternative:
On 18 Jun 2021, at 6:27 am, Francis Siefken <fsiefken@gmail.com <mailto:fsiefken@gmail.com>> wrote: > Alternatively one could reconfigure it client side to automatically decompress and display client side, with the back button or key working - but this you cannot enforce as the one serving the gemini capsule. Right. There is nothing strange or wrong about returning compressed data with a suitable MIME type, and it is logical for a client to pass the binary blob on to the default extractor app, as happened in your test. To me, this line of the spec rules out decompression as a feature of the Gemini client itself: > Response bodies are just raw content, text or binary, ala gopher. There is no support for compression, chunking or any other kind of content or transfer encoding. As with any home-grown extension you can certainly implement it for your own purposes. I just don't see a way forward for widespread compression support when it's been explicitly scoped out. Since you mentioned 1200 bps and radios in your original post, I will remind the list that Gemini is unsuitable as a protocol for any amateur radio operations due to the encryption. I used gopher for that purpose instead. Cheers, Tom
Get ready for a wall of text.
On Thu, Jun 17, 2021 at 02:24:45PM +0200, Francis Siefken wrote:
> How would people solve such use cases elegantly and within the design philosophy?
Compression could be valuable to users with poor download speeds,
especially those using an overlay network like Tor with a poor uplink.
I do like the idea of compression being optional; if a client supports
compression it can get a Gemini file compressed, but otherwise a plain
version. However, this does create a huge problem: clients would have to
declare that they support compression, which opens up a can of worms
(complexity, fingerprinting, etc.) that we should stay away from.
Almaember's approach is much better:
On Thu, Jun 17, 2021 at 03:47:48PM +0200, Almaember wrote:
> In my personal opinion, the best solution would be to simply have it as
a separate MIME-type, something along the lines of "text/gemini+gzip". I
don't recall how this works with MIME-types, but it should be something like this.
>
> I do support your idea, though. I think compression would be a nice
addition, but it doesn't belong in the protocol itself, but the file format.
I think the best solution is:
- Have clients optionally support pagination, like what most line-mode
clients (gmnlm, cgmnlm, diohsc) do. Show the first N lines/bytes instead
of downloading the whole thing; let the user scroll to trigger
downloading the rest. This probably should not be the default setting,
but that's up to client developers to decide.
- For big files, have authors link a compressed version: "Dear reader,
this gemini file is large. Here's a link to a compressed version:"
- Users can then select the link before they've finished
downloading/paging through the file.
This therefore doesn't need to be part of the Gemini "standard" but can
simply be a recommendation for authors and devs. If it gains a lot of
traction, perhaps it could be formalized using the word "MAY" in the spec
("clients MAY also support the mimetype...").
Regarding a compression algorithm to pick: it should be one that's fairly
common with a lot of libraries/implementations for a variety of platforms
and programming languages, keeping with the rationale for choosing TLS
1.2+ over other options for transport-layer encryption. Although I'd love
to pick Lizard for its speed, it's not universal enough to qualify.
=> https://github.com/inikep/lizard Lizard (formerly LZ5)
Our best options are therefore gzip/DEFLATE and perhaps zstd. Gzip can
actually get pretty small when compressing statically/ahead-of-time, where
compression speed is less of a constraint. Tools like Zopfli and
especially Efficient-Compression-Tool can get a dump of all posts on
seirdy.one (~100kb) 41.1 kb, compared to 39.9kb with zstd -f19. The
difference only becomes significant with Gemini files above 200kb.
=> https://github.com/fhanau/Efficient-Compression-Tool
I therefore vote for the following *non*-standard:
- Encourage authors of 50kb+ gmi files to link a text/gemini+gzip
somewhere near the top.
- Encourage client developers to consider supporting pagination (can be
optional and off by default), so only the first N bytes/lines are
downloaded before the user performs an action.
- Encourage client developers to support the text/gemini+gzip MIME type
Only after this gains traction and is well-received by users with
constrained bandwidth should consider adding a "MAY" statement to the
Gemini spec describing the text/gemini+gzip MIME type. Compression support
should never be required.
-- /Seirdy (seirdy.one)
On Thu, Jun 17, 2021 at 08:20:08PM -0700, Rohan Kumar wrote: > Only after this gains traction and is well-received by users with > constrained bandwidth should consider adding a "MAY" statement to the Gemini > spec describing the text/gemini+gzip MIME type. Compression support should > never be required. I thought the spec was finalized and not designed to be extended. I can see clients treating gzip files specially though, no different than lagrange supporting mp3s.
Lagrange already supports opening Zips and browsing gemtext from it. So... there you go. There's your compression. If you want different compression method, just tell gemini browsers to support it. This isn't against the spec, and it's actually very simple to use. In fact, gempubs are basically this same thing - they are zips with gemtext files inside of them. When you click on a gempub link, lagrange will literally have a button saying "Download and Open" and it'll download and open it right away for browsing. Other browsers can make this just as easy to do.
Was wondering when that would get brought up. Hadn't spoken because I wasn't sure if it hadn't already been looked into. Gwmpub/gpub is readable by Lagrange, and hopefully other browsers will follow suit (especially Ariana or another mobile browser.) Is there a reason this wouldn't work though?
I think people would probably want either a different compression, or to just compress one file instead of a whole bunch of files. So we'd probably need to come up with a different format for compression of just single files. And this is actually *easier* to support than supporting zips, etc, because those are more like compressed directories that you can navigate within. Anyways, for people wondering about the gempub project, it's the equivalent of both epubs, but with gemtest, and also has the goal of being used for gemini capsule archives. The spec for gempubs is here, if anyone wants to contribute. You can perhaps suggest this to other browsers to support: https://codeberg.org/oppenlab/gempub I do want to note that the gemini protocol doesn't care about what's send over it. This is why mimetypes were added to the protocol. You can send any binary data over gemini. This is why compression doesn't need to be in the main protocol - because you can send over the compressed stuff just as any other binary file can. It's the job of the gemini browser to handle the compressed data by reading the mimetype and dealing with that mimetype in some way. And if it's too complicated, then tell your browser maintainer and they can fix it if they want. If they don't want to, then I would consider looking into other gemini browsers that do support what you want. Finally, the other consideration is whether supporting this within clients adds too much complexity. Personally, I do not think so, as there are tons of very simple libraries you can use for supporting compression. Golang even has these in its standard library. Some gemini browser maintainers are probably going to argue that it's not the purpose of a browser to support opening zips, as a similar argument has been used for Gemini Subscription Feeds, Atom Feeds, audio/video files, and other such things. I would have to completely disagree. The point of a browser is to browse documents - that's the scope of gemini browsers. I do recognize that there are, however, two different approaches to this. Have the "gemini browser" be a full document browser for the gemini protocol, or have it just deal with the gemini protocol and pass the documents off to another program. The problem, of course, is many applications don't support gemini links, which is definitely why I prefer the 1st approach, which seems more like the approach Lagrange takes (Skyjake can correct me if I'm wrong). The point is, the point of a browser is to browse geminispace. Which means you need to either support things yourself, or have the ability to pass them off to another application that *can* support them properly.
On 2021-06-18 15:22+02:00, Christian Seibold wrote: > I think people would probably want either a different compression, or to just compress one file instead of a whole bunch of files. > So we'd probably need to come up with a different format for compression of just single files. gzip by itself only ever compresses single files, leading to the common .tar.gz combination, where the tape archive (tar) contains any directory structure. The MIME type text/gemini+gzip has been suggested for this earlier. [1] I think that would be a good solution although it is questionable how well browsers check the MIME type and if they would mistake "text/gemini+gzip" for plain "text/gemini", leading to some confusion. The +gzip suffix is registered by IANA [2]. [1] gemini://gemi.dev/gemini-mailing-list/messages/006710.gmi [2] https://www.iana.org/assignments/media-type-structured-suffix/media-typ e-structured-suffix.xml
Hi Christian, yes, there's my compression - but it's not transparent: it takes 4 user actions. So unless a client automatically opens and displays a gzipped gemtext there is no way to have a near instantly readable and extracted compressed index.gmi. Even if it did I am not sure if the back button works at once, as now I also have to hit the back 4 times after downloading and displaying a gzipped gemtext. It's not that I want a different compression codec as the usual suspects (zstd, gz, zip) all have similar ratios (40-50%), although i'd like the most efficient text compression code with a high speed decompression. For example ppmd with for example dictionary wrt preprocessing to get to ~25%. Cheers, Francis gemini://fsiefken.srht.site
The last person who emailed on the mailing list was Johann Galle, not me. Please read my own response again. If you're using zips, Lagrange doesn't take 4 user actions, it takes 1. If other browsers don't have this functionality, then tell them. Browsers should implement this, not the protocol. It's that's simple.
On Fri, Jun 18, 2021 at 08:22:32AM -0500, Christian Seibold wrote: Hi! > Finally, the other consideration is whether supporting this within > clients adds too much complexity. Personally, I do not think so, as > there are tons of very simple libraries you can use for supporting > compression. Golang even has these in its standard library. This is how bloated software starts. > Some gemini browser maintainers are probably going to argue that > it's not the purpose of a browser to support opening zips, as a > similar argument has been used for Gemini Subscription Feeds, Atom > Feeds, audio/video files, and other such things. I would have to > completely disagree. The point of a browser is to browse documents - > that's the scope of gemini browsers. I do recognize that there are, > however, two different approaches to this. Have the "gemini browser" > be a full document browser for the gemini protocol, or have it just > deal with the gemini protocol and pass the documents off to another > program. The problem, of course, is many applications don't support > gemini links, which is definitely why I prefer the 1st approach, So are you arguing that gemini client should include a postscript or PDF interpreter? Supports HTML? JavaScript? There are libraries for those in the most common languages. > which seems more like the approach Lagrange takes (Skyjake can > correct me if I'm wrong). Please this is not the place to discuss a single implementation for gemini standard. The fact is that or you are proposing a standard extension, and just let me write that i am tired that about 60% of the traffic in this mailing list is about proposing extension to the protocol that still needs works to be finalized, instead of proposing fix (there is the gitlab repo for that). If you do not like gemini propose a new protocol and i likely be the first to works on that if i think is good. Or if this is not extension (and i disagree it is not, to be clear) it is a discussion about a single implementation which produce noise in this ml. Moreover i think that promoting a "fat" client promotes client mono culture as well. Complicated clients need more resources (people or money) and lead to monopolistic behaviour (even in good faith). To be clear. No if lagrange implements X this does not means every other client should follows the same path. In my opinion if an implementation follows the path you are proposing (bloating the software) this, in my opinion, is a reason to not implements those (anti)features. To reuse (badly) a metaphor that is used often these days, to me you are proposing tho transform an old boring tree that helps people meditating with a shiny Christmas tree with colored light and decoration which only purpose is to remember us to spend our money in the commercial center nearby until it is disposed of, after the season! :) What, in my opinion, needs gemini now is not extension but fixing the standard, new contents a lot of propaganda, i think help of everyone is appreciated. Bye! C. PS: as usual, sorry for my bad English!
> This is how bloated software starts. Who says what is bloated? How much features do we have to add for something to be bloated? Was adding TLS bloat? TLS added a bunch of complexity to the protocol. What about gemtext? Is gemtext bloat? This isn't black and white. Bloat is a ratio of various things, including complexity added, size added, and performance effects vs. how much the feature adds to the program and its usefulness - it's "power to weight ratio" > > > Some gemini browser maintainers are probably going to argue that > > it's not the purpose of a browser to support opening zips, as a > > similar argument has been used for Gemini Subscription Feeds, Atom > > Feeds, audio/video files, and other such things. I would have to > > completely disagree. The point of a browser is to browse documents - > > that's the scope of gemini browsers. I do recognize that there are, > > however, two different approaches to this. Have the "gemini browser" > > be a full document browser for the gemini protocol, or have it just > > deal with the gemini protocol and pass the documents off to another > > program. The problem, of course, is many applications don't support > > gemini links, which is definitely why I prefer the 1st approach, > > So are you arguing that gemini client should include a postscript or > PDF interpreter? Supports HTML? JavaScript? There are libraries for > those in the most common languages. Nope, I never suggested such a thing. HTML isn't used in geminispace. Neither is javascript. Additionally, again, you have to look at the pros and cons of things. Javascript adds significant performance and security problems. Compression **doesn't**. Stop trying to compare compression to Javascript and HTML, especially when compression is very basic and doesn't add barely any LOC or performance decreases. Let's look at an lz4 library: https://github.com/lz4/lz4 It's 49000 MB/s on a Core i7-9700K CPU @ 4.9GHz. And it's 5699 LOC. Btw, this is the approach that Solderpunk used to determine what should go into the protocol. It's right on the front page of the gemini capsule: "Strives for maximum power to weight ratio" Imo, compression fits perfectly with this. Some people will disagree, and that's fine. That's not gonna change my opinion on this matter though. All I ask is that you stop using personal attacks, because it doesn't help your argument. > > > which seems more like the approach Lagrange takes (Skyjake can > > correct me if I'm wrong). > > Please this is not the place to discuss a single implementation for > gemini standard. Yes it is. The OP was asking for a specific feature, and I suggested a client that had that feature that they can use. I NEVER suggested every single client should implement compression, and I actually said the opposite: "Which means you need to either support things yourself, or have the ability to pass them off to another application that *can* support them properly." Passing things off to other applications is just as valid of a way to deal with this, and it's what AV-98 and various other clients do. > > The fact is that or you are proposing a standard extension, and just > let me write that i am tired that about 60% of the traffic in this > mailing list is about proposing extension to the protocol that still > needs works to be finalized, instead of proposing fix (there is the > gitlab repo for that). If you do not like gemini propose a new > protocol and i likely be the first to works on that if i think is > good. Or if this is not extension (and i disagree it is not, to be > clear) it is a discussion about a single implementation which produce > noise in this ml. I in no way proposed an extension to the protocol. In fact, I proposed the exact opposite, that this should stay *out* of the protocol, and be in client applications for people who want to support this: "I do want to note that the gemini protocol doesn't care about what's send over it. This is why mimetypes were added to the protocol. You can send any binary data over gemini. This is why compression doesn't need to be in the main protocol - because you can send over the compressed stuff just as any other binary file can." > > Moreover i think that promoting a "fat" client promotes client > mono culture as well. Complicated clients need more resources (people > or money) and lead to monopolistic behaviour (even in good faith). People are smart enough to decide for themselves what client they want to use. I'm allowed to promote a client. I think it's very presumptuous to think I'm somehow "propagandizing" people into supporting Lagrange. It's ridiculous. People are smart enough to decide for themselves what client they want to use. > > To be clear. No if lagrange implements X this does not means every > other client should follows the same path. In my opinion if an > implementation follows the path you are proposing (bloating the > software) this, in my opinion, is a reason to not implements those > (anti)features. I didn't suggest any such thing, as I've already said above.
On Sat, Jun 19, 2021 at 02:15:45AM -0500, Christian Seibold <krixano@mailbox.org> wrote a message of 3 lines which said: > If you're using zips, Lagrange doesn't take 4 user actions, it takes > 1. If other browsers don't have this functionality, then tell > them. Browsers should implement this, not the protocol. It's that's > simple. If I understand correctly your proposal, you suggest to change, neither the protocol nor the gemtext format, but to push browser authors to recognize text/gemini+gzip and act accordingly. In that case, I see two issues:
> To reuse (badly) a metaphor that is used often these days, to me you > are proposing tho transform an old boring tree that helps people > meditating with a shiny Christmas tree with colored light and > decoration which only purpose is to remember us to spend our money in > the commercial center nearby until it is disposed of, after the > season! :) Btw, soft Christmas lights can help many people, including some autistic people, as it can be used as a stimulant to calm them down. What you consider useless or "commercial" some people might consider beautiful and a way to connect and relax and spend time with their families. Finally, light, especially candles, has been used very often across history for meditation, for centuries. It's used in Judaism, Christianity, Yoga, Islam, various eastern religions, etc. Again, this is all about perspective.
Hi Christian, on Sa 19 jun. 2021 09:15 Christian Seibold < krixano@mailbox.org> you wrote: > The last person who emailed on the mailing list was Johann Galle, not me. I know, in my previous message replied specifically to your message vr 18 june 14:24. Is there something wrong with that? > Please read my own response again. [..] > If you're using zips, Lagrange doesn't take 4 user actions, it takes 1. I understood what you meant yesterday. But in my test with Langrange 1.5.2 and gmnisrv it took 4 steps. With the reference gpub (excellent choice) it takes 3 steps and it opens an addititional tab. With a gmi zip it says: "xi.zip is a compressed archive. You can save it as a file to your Downloads folder: press Ctrl+S or select "Save to Downloads" from the menu." And then a whole lot more steps. I wish it would take zero steps and a client would for example automatically gunzip a gmi.gz and display it inline immediately by default. > Browsers should implement this, not the protocol. It's that's simple Will it be? For example Asuka, Amfora, Ariane and Lagrange are nice browsers but if I read the discussion so far I am sure it will be implemented in the way I envision. Images and sounds are handled differently and take at least one or a few user actions. What I hope and wish for is that one day my Gemini capsule can serve index.gmi.zstd or index.gmi.ppmd transparently. I trust they can be automatically decompressed inline in the same tab by visiting browsers, either through the mime-type route, TLS compression or through a Gemini Content-Encoding and zstd or ppm server side compression. You mentioned the dogma browsers should implement this and not the protocol. I am curious as to what your reason for this is other then that gemini protocol is supposedly set in stone? Protocol support has been discussed on list before. On March 10, 2021 Bradley D. Thornton had a similar remark "neither of those two things (Accessibility or gzip compression) have anything to do with, nor have any place in discussions, for spec changes" and argued:
> I know, in my previous message replied specifically to your message vr 18 june 14:24. Is there something wrong with that? It looked like you were replying to Johann Galle, not me, as I didn't mention the specific compression you were talking about - I deliberately kept my message more general. I did, however, read about what you wanted. > The Gempub doesn't load immediately and I suspect there will always be the extra download and open step. Of course I can send patches, fork browsers or roll my own gemini client and encourage people to additionally compress their individual gemtexts with PPMd and their capsule with Gempub. Yes, **currently** it takes multiple steps for lagrange to open a zip, but my point is it CAN be instant without having to implement it in the protocol. It absolutely *can* be implemented in client so that it's instant. Anyways, you can make suggestions to skyjake on Station or on the lagrange github (https://github.com/skyjake/lagrange) so that it can be made more instant. There will always be 1 extra step (the "download and open" button) due to the fact that zips are more like a compressed directory - that is unless we figure out how to fix the url handling of things within zips. However, compressed single files can absolutely be instantly decompressed by a client that implements this. Also, if TLS 1.3 has compression (thanks for telling me that, I didn't know that), then it makes supporting specific compressions in clients unnecessary, because we are going to be switching to TLS 1.3 in the future, from what I've gathered.
On Sat, Jun 19, 2021 at 03:42:26AM -0500, Christian Seibold wrote: > > This is how bloated software starts. > > Who says what is bloated? How much features do we have to add for something to be bloated? Was adding TLS bloat? TLS added a bunch of complexity to the protocol. What about gemtext? Is gemtext bloat? There are reasons why TLS is in gemini and compression not, please reread the FAQ. > This isn't black and white. I agree it is not. > Bloat is a ratio of various things, > including complexity added, size added, and performance effects > vs. how much the feature adds to the program and its usefulness - > it's "power to weight ratio" I agree. > > > > > > Some gemini browser maintainers are probably going to argue that > > > it's not the purpose of a browser to support opening zips, as a > > > similar argument has been used for Gemini Subscription Feeds, Atom > > > Feeds, audio/video files, and other such things. I would have to > > > completely disagree. The point of a browser is to browse documents - > > > that's the scope of gemini browsers. I do recognize that there are, > > > however, two different approaches to this. Have the "gemini browser" > > > be a full document browser for the gemini protocol, or have it just > > > deal with the gemini protocol and pass the documents off to another > > > program. The problem, of course, is many applications don't support > > > gemini links, which is definitely why I prefer the 1st approach, > > > > So are you arguing that gemini client should include a postscript or > > PDF interpreter? Supports HTML? JavaScript? There are libraries for > > those in the most common languages. > > Nope, I never suggested such a thing. HTML isn't used in geminispace. It is, because as you correctly said, the protocol with mime types allow for any type of document. >Neither is javascript. Additionally, again, you have to look at the pros and cons of things. Javascript adds significant performance and security problems. Compression **doesn't**. So are you OK, with PDF and postscript that you left out? > Stop trying to compare compression to Javascript and HTML, especially when compression is very basic and doesn't add barely any LOC or performance decreases. You you know that compression "barely" add a few effort for every possible programming scenario? > Let's look at an lz4 library: > https://github.com/lz4/lz4 > > It's 49000 MB/s on a Core i7-9700K CPU @ 4.9GHz. > And it's 5699 LOC. > > Btw, this is the approach that Solderpunk used to determine what should go into the protocol. It's right on the front page of the gemini capsule: > "Strives for maximum power to weight ratio" Yes, please reread the FAQ. Compression is not in the standard for reasons. > Imo, compression fits perfectly with this. Some people will disagree, and that's fine. That's not gonna change my opinion on this matter though. All I ask is that you stop using personal attacks, because it doesn't help your argument. You tell me that i am presumptuous and accuse me for personal attacks? > > > > > > which seems more like the approach Lagrange takes (Skyjake can > > > correct me if I'm wrong). > > > > Please this is not the place to discuss a single implementation for > > gemini standard. > > Yes it is. The OP was asking for a specific feature, and I suggested a client that had that feature that they can use. I NEVER suggested every single client should implement compression, and I actually said the opposite: > "Which means you need to either support things yourself, or have the ability to pass them off to another application that *can* support them properly." I quote another your message: "Please read my own response again. If you're using zips, Lagrange doesn't take 4 user actions, it takes 1. If other browsers don't have this functionality, then tell them. Browsers should implement this, not the protocol. It's that's simple." And a few lines above: "...why I prefer the 1st approach" [a full gemini document browser]. > Passing things off to other applications is just as valid of a way to deal with this, and it's what AV-98 and various other clients do. But you prefer a full gemini browser, right? And clients *should* implement a form of decompression one way or another, right? > > > > The fact is that or you are proposing a standard extension, and just > > let me write that i am tired that about 60% of the traffic in this > > mailing list is about proposing extension to the protocol that still > > needs works to be finalized, instead of proposing fix (there is the > > gitlab repo for that). If you do not like gemini propose a new > > protocol and i likely be the first to works on that if i think is > > good. Or if this is not extension (and i disagree it is not, to be > > clear) it is a discussion about a single implementation which produce > > noise in this ml. > > I in no way proposed an extension to the protocol. In fact, I proposed the exact opposite, that this should stay *out* of the protocol, and be in client applications for people who want to support this: > "I do want to note that the gemini protocol doesn't care about what's send over it. This is why mimetypes were added to the protocol. You can send any binary data over gemini. This is why compression doesn't need to be in the main protocol - because you can send over the compressed stuff just as any other binary file can." > > > > > Moreover i think that promoting a "fat" client promotes client > > mono culture as well. Complicated clients need more resources (people > > or money) and lead to monopolistic behaviour (even in good faith). > > People are smart enough to decide for themselves what client they want to use. I'm allowed to promote a client. I think it's very presumptuous to think I'm somehow "propagandizing" people into supporting Lagrange. It's ridiculous. People are smart enough to decide for themselves what client they want to use. i like lagrange a lot, this is neither about lagrange nor about compression, this is about pushing for complex client that could break the geminispace. > > > > To be clear. No if lagrange implements X this does not means every > > other client should follows the same path. In my opinion if an > > implementation follows the path you are proposing (bloating the > > software) this, in my opinion, is a reason to not implements those > > (anti)features. > > I didn't suggest any such thing, as I've already said above. And i can requote your other message as above. Please reread the Bortzmeyer reply, he is wiser than me and so is able to explain where the problem is better than me. Also i encourage *everyone* that cares for gemini (including you) to helps to finalize the standard, not advocating for re-implementing web browser with an unfit protocol. Bye! C.
> If I understand correctly your proposal, you suggest to change, > neither the protocol nor the gemtext format, but to push browser > authors to recognize text/gemini+gzip and act accordingly. Ok, so, as you have probably read before :D , there's a divide on whether compression should even be put into the protocol or not. There's even a divide on whether it should be put into any clients - of course nobody can control every client and force them to do what they want it to do. So, realistically, I don't think we can get everyone to agree on a decision about compression being put into the protocol, especially when the protocol is almost finalized. However, users are allowed to suggest things for browsers, and browser developers are allowed to reject suggestions. So, as we cannot get it in the protocol, then some client developers might decide to implement it in their clients, and it's totally possible to do this. This is really the next best step that we could take, assuming we won't reach a consensus on putting it into the protocol. However, if what was said before was correct about TLS 1.3 having compression, all of this might be completely unnecessary when we switch to TLS 1.3. > * how the end user of a browser without compression support will know > in advance if the file is compressed or not? From the file extension? > It is not always present. This is a good point about client design. I think clients should tell the user the mimetype of files that are unrecognized by the client. I would also personally like that they allow the user to open the file in another program, kinda like AV-98's hooks, basically. > * today, every browser can read every Gemini file. Your proposal > seems to imply that some users will now see only a part of the > Geminispace, the uncompressed one. This is the same problem we have > when trying to use the Web with lynx or dillo: we can see only a part > of it. I mean, right. But what else can we do if we cannot reach a consensus on putting compression in the protocol? Solderpunk could just make the decision anyways, but that's dependent on him. The next best thing is for clients that want to provide this functionality to do so, and for clients that don't - they might or might not allow you to open it in a different program. We also kinda already have this problem with various filetypes. There's just too many filetypes to support them all in clients, so sending them off to be handled by another program can help resolve this.
I'm a lurker, but this discussion requires some commentary from the sidelines: On 6/19/21 11:24 AM, Christian Seibold wrote: > Btw, soft Christmas lights can help many people, including some autistic people, as it can be used as a stimulant to calm them down. As a personed diagnosed on the autistic spectrum that cannot *stand* "christmas lights", especially the "animated" or blinking kind, for a multitude of reasons, I request you refrain from dragging my "tribe" into this discussion to support your arguments. It is neither valid as an argumentative strategy, nor is it particularly welcome. > What you consider useless or "commercial" some people might consider beautiful and a way to connect and relax and spend time with their families. Noone thinks of themselves as the villain. > Finally, light, especially candles, has been used very often across history for meditation, for centuries. It's used in Judaism, Christianity, Yoga, Islam, various eastern religions, etc. Go worship in the church of feature-creep then, but leave the atheists alone with your "traditions". > Again, this is all about perspective. My perspective is: - Proposing an extension to the protocol is obviously against the core idea of gemini. - Proposing clients support an additional mime-type will lead to fracturing gem-space and is to be avoided. I for one will actively avoid any client that does this, it is *exactly* the kind of bloated "useful" features absent in gemini I find attractive. Robert
You are making a bunch of assumptions about what I do or don't want based on no evidence and reading into some of the responses that I have given. I never said clients *must* implement compression. I said it's up to the client developers if they want to or not, and that users are allowed to make suggestions. If a user doesn't have the features they want, they can easily move to a client that does have the features they want. I respect solderpunk's decision to not include it into the protocol, but adding compression to clients in no way "breaks" geminispace, and it doesn't disobey the spec either. So your arguments there are completely invalid. Let me repeat this... adding compression to clients in the way that has been described in this thread does NOT break geminispace. Gemini deliberately allows you to send whatever you want over. It's a file-transfer protocol. Using the mimetype is *exactly* what I proposed. Period. End of discussion.
> Also i encourage *everyone* that cares for gemini (including you) to > helps to finalize the standard, not advocating for re-implementing web > browser with an unfit protocol. Compression is NOT a web thing, and it predates the web, so knock that crap off. Not everything the web has remotely touched is bad. TLS is used by the web. It's not suddenly "re-implementing" the web by using TLS. Adding compression is also not "re-implementing" the web.
This is the final message that I will be sending on this thread. The Gemini protocol restricts what is added to the core protocol. Compression is not used there for a reason as outlined by the FAQ, which I have read numerous times. What the spec does NOT do is outlaw which filetypes can be sent over the protocol. Zip files can be downloaded over gemini. Same with any other filetype. The spec says how to interpret both the protocol itself, and gemtext. It has nothing to say about handling filetypes, aside from gemtext. Why? Because gemini is a file transfer protocol at its core. You transfer files over it. The spec also purposefully allows for streaming of files, and this has been talked about many times in both a mailing list thread and on Solderpunk's gemlog. What the spec does NOT do is try to censor what files can be downloaded over the protocol. Which means compressed files are allowed to be downloaded over the protocol, just like any other file. This is NOT against the protocol, as shown in the spec: > Response handling by clients should be informed by the provided MIME type information. Gemini defines one MIME type of its own (text/gemini) whose handling is discussed below in section 5. In all other cases, clients should do "something sensible" based on the MIME type. Minimalistic clients might adopt a strategy of printing all other text/* responses to the screen without formatting and saving all non-text responses to the disk. Clients for unix systems may consult /etc/mailcap to find installed programs for handling non-text types. I will now give various quotes by Solderpunk that show that Solderpunk was a fan of a variety of different types of clients that do different things - from his article on "A vision for Gemini applications" (gemini://gemini.circumlunar.space/users/solderpunk/gemlog/a-vision-for-gem ini-applications.gmi): > What this provides is a nice little, dare I say it, "containerised" Gemini application experience. You are easily and reliably identified to one service and one service only, and no external content can control what you do with that identity, and your identity can't accidentally leak out to anywhere external. > > Meanwhile, your "everyday" Gemini client, which will let you go anywhere you like and follow whichever links you like, does not know the paths to any of your client certificates. Maybe it doesn't even support client certificates at all! If you roam the dangerous wild internet frontiers with this client, and you accidentally follow a malicious link to one of your apps running on localhost, as long as that app requires an approved client certificate to do anything of consequence, no damage can be done. Later on: > Some people may still be thinking that this looks like an ugly complication, even if you can wrap these containers up in nice convenient shell scripts (and I'm sure some kind of GUI management solution could be whipped up for people who want one): "What, now I need *two* kinds of client to use Gemini? Gimme a break!". But I think it's a small surface complication which yields large simplifications deeper down. The containerised identities approach using ultra-slim clients creates two clearly separate ecological niches for clients: reading static textual content like gemlogs, technical documentation, fiction, news reports, weather forecasts, etc. on the one hand and making use of individual, certificate-secured dynamic applications on the other. This allows for client authors to target one niche only and therefore write simpler clients. > In short, different tools for different jobs, but with a common underlying protocol and markup language. > Or they can partake of both Gemini experiences. I really hope that this approach can ease some of the tensions that are building between people with different visions for what Geminispace should be, and make the lives of developers easier at the same time. In summary, and this is the most important one: > Most Gemini clients will fall into one of two distinct categories > * "Reading-centric" clients will probably have very little or no support for client certificates at all. Instead they will focus on beautiful and customisable typography, bookmarking, feed subscription, TOC generation, etc. There will be many clients of this kind, taking different approaches. Many will be graphical. People will love some and hate others. Plenty of room room for all. > * "App-centric" clients will instead have good support for client certificates and will be designed to very securely limit the use of those clients to a single app in order to totally avoid the risk of CSRF. There will be much less variation amongst these clients because they are all just doing the same basic thing. Most of them will be terminal-based, because that's just the better interface for interacting with text-based applications, but they needn't all be. > Hybrid clients attempting to address both the document reading and secure app niches will surely exist, but to avoid the risk of CSRF they will require either slightly clunky interfaces or good awareness and understanding from users. These may be the preferred clients of "power users".
Christian Seibold <krixano@mailbox.org> writes:
> Let me repeat this... adding compression to clients in the way that
> has been described in this thread does NOT break geminispace.
I'm not too too sure about this. As I see it, using *any* file type
other than text/gemini as default choice for the content will break the
geminispace.
Gemini browser have to implement text/gemini[0], not text/gemini+gzip,
text/html or text/x-markdown, and if text/gemini+gz (or whatever) starts
to gain a bit of popularity it *will* break the geminispace IMHO.
> Gemini deliberately allows you to send whatever you want over. It's a
> file-transfer protocol. Using the mimetype is *exactly* what I
> proposed. Period. End of discussion.
and I totally agree with you here. You can serve any type of content
over Gemini, so what's all this fuss about? You have a strange use case
where you need compression? Fine, you can do it. Good for you. But
please don't waste our time by telling how much cool would be to change
every browser just to read the usual 1-2K[1] pages.
Cheers,
Omar Polo
P.S.: I don't want to sound too harsh, in fact I have a local patch for
gmid to allow the `entrypoint' config option inside `location'
blocks, so that for instance one can do something like
server "xyz.com" {
...
location "*.gmi.gz" {
entrypoint "/my/gzip/script";
}
}
to automatically generate gzipped files (using an hypotetical CGI
script in /my/gzip/script) on the fly[2].
[0]: OK, not quite really, gemget/curl-like applications only implement
the protocol, but I'm talking about more "user-facing" browsers.
[1]: Stephane Bortzmeyer once shared with us the mean size of
text/gemini pages as computed by lupa IIRC.
[2]: actually, this will allow to generate *any* file on the fly, that's
why I added it.
> I'm not too too sure about this. As I see it, using *any* file type > other than text/gemini as default choice for the content will break the > geminispace. > > Gemini browser have to implement text/gemini[0], not text/gemini+gzip, > text/html or text/x-markdown, and if text/gemini+gz (or whatever) starts > to gain a bit of popularity it *will* break the geminispace IMHO. > > and I totally agree with you here. You can serve any type of content > over Gemini, so what's all this fuss about? You have a strange use case > where you need compression? Fine, you can do it. Good for you. But > please don't waste our time by telling how much cool would be to change > every browser just to read the usual 1-2K[1] pages. > Read the whole thread. I never once suggested compressed gemtext should be default. I suggested it can be supported in some clients when clicking on a link. Other people are having a problem with this, not me. I fully support sending compressed files over gemini, because it's allowed by the spec.
> As a personed diagnosed on the autistic spectrum that cannot *stand* "christmas lights", especially the "animated" or blinking kind, for a multitude of reasons, I request you refrain from dragging my "tribe" into this discussion to support your arguments. It is neither valid as an argumentative strategy, nor is it particularly welcome. It is well-known within the autistic community that not all autistic people are the same. some are like you and have problems with light sensitivity. Some have other sensory sensitivities. I request you refrain from speaking for all of the Autistic community, and assuming who is or isn't autistic within the mailing list.
> - Proposing an extension to the protocol is obviously against the core idea of gemini. And btw, nobody here was proposing an extension to the protocol.
A bit of a recap and my thoughts:
> Some people should reflect on what they said in this discussion and how they said it: what parts of it were helpful and what parts weren't. Maybe they can avoid a similar mess in the future. Moving on... It doesn't matter, because people are going to cherry pick what someone says anyways and try to put their own agenda forward as if it was the one intended by Solderpunk when it wasn't. Anyways, everything you've said I completely agree with, and is precisely what I have been saying this entire time.
Normally, I prefer being staying out of stupid arguments, but this email just has so much bullshit that I cannot not respond. >> Neither is javascript. Additionally, again, you have to look at the pros and cons of things. Javascript adds significant performance and security problems. Compression **doesn't**. > > So are you OK, with PDF and postscript that you left out? If I say that I love a person, does that imply I don't love anybody else? People don't always list everything. >> Let's look at an lz4 library: >> https://github.com/lz4/lz4 >> >> It's 49000 MB/s on a Core i7-9700K CPU @ 4.9GHz. >> And it's 5699 LOC. >> >> Btw, this is the approach that Solderpunk used to determine what should go into the protocol. It's right on the front page of the gemini capsule: >> "Strives for maximum power to weight ratio" > > Yes, please reread the FAQ. Compression is not in the standard for reasons. And nobody is saying it should be, we are simply debating a separate file format for compressed gemtext. Nobody said anything about the protocol. >> >>> >>>> which seems more like the approach Lagrange takes (Skyjake can >>>> correct me if I'm wrong). >>> >>> Please this is not the place to discuss a single implementation for >>> gemini standard. >> >> Passing things off to other applications is just as valid of a way to deal with this, and it's what AV-98 and various other clients do. > > But you prefer a full gemini browser, right? And clients *should* > implement a form of decompression one way or another, right? What exactly is wrong with advocating for something? >> I in no way proposed an extension to the protocol. In fact, I proposed the exact opposite, that this should stay *out* of the protocol, and be in client applications for people who want to support this: >> "I do want to note that the gemini protocol doesn't care about what's send over it. This is why mimetypes were added to the protocol. You can send any binary data over gemini. This is why compression doesn't need to be in the main protocol - because you can send over the compressed stuff just as any other binary file can." >> People are smart enough to decide for themselves what client they want to use. I'm allowed to promote a client. I think it's very presumptuous to think I'm somehow "propagandizing" people into supporting Lagrange. It's ridiculous. People are smart enough to decide for themselves what client they want to use. > i like lagrange a lot, this is neither about lagrange nor about > compression, this is about pushing for complex client that could break > the geminispace. It's not gonna break the geminispace, as long as clients have the capability to simply dedicate that functionality to an external software, which is absolutely possible with this suggestion. -- Unless you're replying to me on the Gemini mailing list, reply to almaember@almaember.com instead. Website: https://almaember.com/ Gemini capsule: gemini://almaember.com/ IRC: almaember on Libera.chat and tilde.chat
On 19/06/2021 10:59, Stephane Bortzmeyer wrote: > If I understand correctly your proposal, you suggest to change, > neither the protocol nor the gemtext format, but to push browser > authors to recognize text/gemini+gzip and act accordingly. > > In that case, I see two issues: > > * how the end user of a browser without compression support will know > in advance if the file is compressed or not? From the file extension? > It is not always present. > > => ilovecompression.gmiz I cannot read it > => plain.gmi I can read it The client could display a simple indicator on the UI. I'm pretty sure some browsers already display the MIME-type. This isn't our decision, but the client authors'. > > * today, every browser can read every Gemini file. Your proposal > seems to imply that some users will now see only a part of the > Geminispace, the uncompressed one. This is the same problem we have > when trying to use the Web with lynx or dillo: we can see only a part > of it. Progress comes at a price. And, this is an unfinished protocol, at least officially. -- Unless you're replying to me on the Gemini mailing list, reply to almaember@almaember.com instead. Website: https://almaember.com/ Gemini capsule: gemini://almaember.com/ IRC: almaember on Libera.chat and tilde.chat
On Sat, Jun 19, 2021 at 12:51:28PM +0200, Omar Polo <op@omarpolo.com> wrote a message of 50 lines which said: > [1]: Stephane Bortzmeyer once shared with us the mean size of > text/gemini pages as computed by lupa IIRC. gemini://gemini.bortzmeyer.org/software/lupa/stats.gmi
Hi Christian, on Sa 19 jun 2021 11:48 you wrote: > you can make suggestions to skyjake on Station or on the lagrange github [..] > compressed single files can absolutely be instantly decompressed by a client that implements this. > > Also, if TLS 1.3 has compression (thanks for telling me that, I didn't know that), > then it makes supporting specific compressions in clients unnecessary, because we > are going to be switching to TLS 1.3 in the future, from what I've gathered. I am sorry I misinformed you, TLS 1.2 has compression (for example deflate), but in TLS 1.3 it has been removed. But it introduces certificate compression to save handshake time. Although it's understandable because of the various exploits it's unfortunate as in my opinion there are use cases, like SMTP or Gemini where it makes sense and where it's not a risk. I'll make suggestions for browser handling of compressed individual Gemtexts. Just like the TLS 1.3 design committee thinks about saving time with the handshake, and compressing a certificate the size of Gemtext. I dream about saving time with optional zstandard or ppmd compression in the Gemini protocol. Making it a user or browser choice is second best. Perhaps Gemini can stay on TLS 1.2 forever? Kind regards, Francis Siefken (NL) gemini://fsiefken.srht.site
On Sat, Jun 19, 2021 at 01:13:20PM +0200, PJ vM <pjvm742@disroot.org> wrote a message of 52 lines which said: > > * how the end user of a browser without compression support will > > know in advance if the file is compressed or not? From the file > > extension? It is not always present. > > I expect authors would indicate this in the link text. So: > > => article.gmi my article > => article.gmi.gz my article (compressed version) As an user, my personal opinion is that it is not great UX.
On Sat, Jun 19, 2021 at 02:08:18PM +0200, Almaember <almaember@disroot.org> wrote a message of 34 lines which said: > I'm pretty sure some browsers already display the MIME-type. They cannot, you don't know it until you've actually send the request (and Gemini has no HEAD method so you have to retrieve everything).
Hi Folks, Just a couple of thoughts about the compression... Even though it looks pretty harmless I think is going to add unnecessary complexity. As Author I have to decide if to use or not use compression, HTML/HTTP doesn't use compression so far I know. As Client I have to ask to the server if it serves compressed GMI file. The easy way I think is using a sibling protocol, hence I use geminiz://my.url: if the server server supports it I receive the compressed page otherwise it will use the default Gemini protocol. On the other hand the Server might try to serve before compressed page and see if the client accepts those and then deciding which content serving. Anyway everything looks quite uncertain; compressing pages is an unwanted behavior that as author I would not to deal with. I would like to transfer this task to my server or to a daemon that belongs to my server, hence it would a bargain on the shoulder of the implementers. I consider that many authors and implementers would be surely unhappy about this feature. Compressing content implies a choice which eventually is not trivial; for instance, knowing myself, I will surely ended up using a compressed method to squeeze up any bit of performance more. The fact that such common feature was not included in the specification is exactly to avoid complexity and ambiguous behavior; However it would be smart if an offline client would be able to download an entire capsule and store it conveniently as compressed GMI. In such scenario compressing GMI pages would be totally useful, under control without hurting or annoying anyone. My best, Gnuserland On June 19, 2021 7:00:04 AM EDT, gemini-request@lists.orbitalfox.eu wrote: >Send Gemini mailing list submissions to > gemini@lists.orbitalfox.eu > >To subscribe or unsubscribe via the World Wide Web, visit > https://lists.orbitalfox.eu/listinfo/gemini >or, via email, send a message with subject or body 'help' to > gemini-request@lists.orbitalfox.eu > >You can reach the person managing the list at > gemini-owner@lists.orbitalfox.eu > >When replying, please edit your Subject line so it is more specific >than "Re: Contents of Gemini digest..." > > >Today's Topics: > > 1. Re: a space case for transparent gemtext compression > (Christian Seibold) > 2. Re: a space case for transparent gemtext compression > (Christian Seibold) > 3. Re: a space case for transparent gemtext compression (Omar Polo) >
I kind of skimmed over this surprisingly heated debate on compression, so I will try to keep my thoughts short:
Hi Gnuserland, you wrote today: > As Author I have to decide if to use or not use compression, HTML/HTTP > doesn't use compression so far I know. > > The HTTP protocol supports compression; for example "Content-Encoding: zstd" => https://datatracker.ietf.org/doc/html/rfc8478#section-3 If both the client and server support gzip or zstd compression a compressed stream is established. Yes this causes a developer overhead, that's why I was happy to read compression is in TLS 1.2 and dismayed that it was removed in TLS 1.3. While it's not removed in HTTP it warns against it's use in certain situations, which do or do not apply in Gemini context: => https://datatracker.ietf.org/doc/html/rfc2616 HTTP/1.1 section on compression => https://datatracker.ietf.org/doc/html/rfc7540#section-10.6 HTTP/2 compression section => https://datatracker.ietf.org/doc/html/draft-ietf-quic-http-34#page-47 HTTP/3 compression section It doesn't have to be difficult to mandate or implement an easy solution. Suppose I am in a life or death situation and I am on a 300 bps link I want the 2 kilobyte of life-saving information to be there as soon as possible. Or you have the daily top 12 HN threads mirrored as a Gemtext file (~100 Kilobyte each) and want to read those on a 300 bps link. Compression speeds up the process. With zstd compression and dictionary compression https://facebook.github.io/zstd/#small-data it can be there quicker and on low end hardware. I think it would be elegant not to rely on having two seperate files for this on the server, one index.gmi and one index.gmi.zstd - the Gemini specification could allow index.gmi to be a binary and let the browser check this after retrieval. If it's a zstd binary automatically extract and display it. Ofcourse keeping this poor man's content sniffing method as simple and secure as possible. => https://en.wikipedia.org/wiki/Content_sniffing Wikipedia on Content sniffing As mentioned by others in this thread a index.gmi.zstd could also be extracted through the existing application/zstd mime-type automatically and display the resulting gmi inline - but as the protocol doesn't enforce this each client can implement it differently. To me it seems like a minimal change to the Gemini protocol to allow gmi text to be served in zstd format and to state that the client must decompress and display it inline instead of merely offering to download it. Saluton, Francis Siefken (NL)
Ben <benulo@systemli.org> wrote: > I think rather than bothering to compress individual files, we should be thinking about compressed archives/mirrors. Oppen (author of Ariane) built this: https://codeberg.org/oppenlab/gempub: A new eBook format based on Gemini Protocol's Gemtext. Gempub can also serve as a Gemini capsule archive format. There is already support for GPUB documents in Lagrange. I don't think that support will (or should) become ubiquitous, but that was never the intention. The MIME type will be registered at some point, too. Raph
On Sat, Jun 19, 2021 at 02:53:30PM +0200, Stephane Bortzmeyer wrote: > On Sat, Jun 19, 2021 at 01:13:20PM +0200, > PJ vM <pjvm742@disroot.org> wrote >> I expect authors would indicate this in the link text. So: >> >> => article.gmi my article >> => article.gmi.gz my article (compressed version) > > As an user, my personal opinion is that it is not great UX. I'd argue that this isn't really a technical issue as much as it is a phrasing issue. Perhaps a better example would be: > # Title > > Author Note: this article is quite long; here's a compressed version: > > => compressed.gmi.gz Compressed article > > (article continues below) Gemini clients supporting pagination can support a pref to download only the first $LINES lines of a gemtext file before the user scrolls down. This way, users can choose to open a link to e.g. a compressed copy before the page finishes loading. I don't think that improving the UX should come at the cost of adding any new features to the spec. This proposed solution may not be "great UX", but it doesn't require adding any new features to clients besides the optional ability to pipe through gunzip, and it's easy enough to navigate for any user capable of navigating Gemini in the language of the capsule in question. Settling on a "good-enough" UX in pursuit of simplicity is one of the main appeals of Gemini; fine UX tuning is better suited for the Web or desktop applications. This is doubly true since plaintext pages over 30kb (the threshold at which I might consider compression worthwhile) are quite uncommon. -- /Seirdy
I am the least expert here... But as author/user if the encryption is transparent for me, in the terms that I always upload a GMI file anyway, I don't see any real objection against but just more work for the implementers. However the GMI is the lighter file we can think of, I really have hard time thinking any human made GMI file bigger than few kilobytes... Thinking that a compressed capsule will save my life while modern HTTP, despite CDN, minification, etc... Will be always way more, and incomparable, bigger that any average GMI page, just makes believing that we are over-thinking about that topic! Thanks... 😉 On June 19, 2021 12:22:49 PM EDT, Francis Siefken <fsiefken@gmail.com> wrote: >Hi Gnuserland, you wrote today: > > >> As Author I have to decide if to use or not use compression, >HTML/HTTP >> doesn't use compression so far I know. >> >> The HTTP protocol supports compression; for example >"Content-Encoding: >zstd" >=> https://datatracker.ietf.org/doc/html/rfc8478#section-3 >If both the client and server support gzip or zstd compression a >compressed >stream is established. Yes this causes a developer overhead, that's why >I >was happy to read compression is in TLS 1.2 and dismayed that it was >removed in TLS 1.3. While it's not removed in HTTP it warns against >it's >use in certain situations, which do or do not apply in Gemini context: > >=> https://datatracker.ietf.org/doc/html/rfc2616 HTTP/1.1 section on >compression >=> https://datatracker.ietf.org/doc/html/rfc7540#section-10.6 HTTP/2 >compression section >=> >https://datatracker.ietf.org/doc/html/draft-ietf-quic-http-34#page-47 >HTTP/3 compression section > >It doesn't have to be difficult to mandate or implement an easy >solution. > >Suppose I am in a life or death situation and I am on a 300 bps link I >want >the 2 kilobyte of life-saving information to be there as soon as >possible. >Or you have the daily top 12 HN threads mirrored as a Gemtext file >(~100 >Kilobyte each) and want to read those on a 300 bps link. >Compression speeds up the process. With zstd compression and dictionary >compression https://facebook.github.io/zstd/#small-data it can be there >quicker and on low end hardware. >I think it would be elegant not to rely on having two seperate files >for >this on the server, one index.gmi and one index.gmi.zstd - the Gemini >specification could allow index.gmi to be a binary and let the browser >check this after retrieval. If it's a zstd binary automatically extract >and >display it. >Ofcourse keeping this poor man's content sniffing method as simple and >secure as possible. >=> https://en.wikipedia.org/wiki/Content_sniffing Wikipedia on Content >sniffing > >As mentioned by others in this thread a index.gmi.zstd could also be >extracted through the existing application/zstd mime-type automatically >and >display the resulting gmi inline - but as the protocol doesn't enforce >this >each client can implement it differently. >To me it seems like a minimal change to the Gemini protocol to allow >gmi >text to be served in zstd format and to state that the client must >decompress and display it inline instead of merely offering to download >it. > >Saluton, >Francis Siefken (NL) -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
I think a lot of aspects were discussed already but let me add my two cents, coming from a slightly different perspective (webapp developer): It was proposed to have two links, one to the gmi, one to the compressed version. This is basically what the accept header in html is for, just deferred to the user. And you look for some kind of signalling to the server that you want a specific representation of a resource. You propose file-extensions which by itself is another convention you introduce here (or additional protocol). URLs don't know about file extensions. Webapps don't know about file extensions. Abstractly speaken: if we don't have a good way to signal to the server what kind of representation of a resource I want, we should drop this. We should not make up conventions artificially. Maybe they will emerge naturally anyway. One thing I miss in the discussion: I see comments about writing files and serving files. I got by now that this is pretty common in Gemini space but there are servers, serving generated content (blogs or games or whatever) and so no assumption about how big files are should be made...
2021 06 19 15:23 mbays <mbays@sdf.org> wrote: > > If I understand correctly, TLS 1.3 actually *removed* compression. > Yes, I saw it, I erred - such a disappointment. So in the absence of a protocol specification that leaves mime-type handling by the client using two seperate gmi files, but as Peter Much remarked somewhere else in this thread that doesn't cover compression of dynamic cgi data streams (which is less of a use case for gemini perhaps - unless). > P.S. here's how one could handle gzipped gemtext in diohsc, causing it > to be presented as if it were uncompressed text/gemini: > set geminator text/gemini+gzip gzip -dc - > I wasn't aware of diohsc, a really nice line mode browser which reminds me of the tridactyl add-on. It has www and gopher proxy support, display of images throug ascii. It's amazing! Transparent inline decompression with gmi.gz works (so hopefully other clients can follow). I am still struggling with getting 7z and zstd decompression to work - i'll figure it out. It would be nice if I can get diohsc to run on termux in the oculus quest and android phone so I can browse my gmi files with espeak (or a better android tts alternative). Thanks so much for writing it. Kind regards from the netherlands, Francis Siefken
On 6/19/21 9:12 PM, raph@raphm.com wrote: > Gempub can also serve as a Gemini capsule archive format. I'd be interested in any explorations into its utilization to that end. I did not realize gempub was even compressed. -- gemini://kwiecien.us/
On Sun, Jun 20, 2021 at 02:52:04PM +0430, Ben <benulo@systemli.org> wrote a message of 8 lines which said: > I'd be interested in any explorations into its utilization to that > end. I did not realize gempub was even compressed. It inherits that from using Zip, like EPUB. I use it for instance to store a copy of the Gemini capsules I'm interested in.
Hi Christian, Thanks for your swansong concerning this thread. Apologies that my response deviates from the topic re compression but your points and citations (btw, do you have the links) do raise some interesting areas. Given my experience of HTTP content, I have always meant to valorise my workflow across sites. I would like it, so that dependent on the site (or subsite), when I am clicking on the content is subsequently loaded in a different browser or with a different toolsets. I have seem approaches for dealing with this at an eLisp end but I havent got around to implementing it myself. In some respects, my memory association with a specific site and a specific browser allows me to fudge it. While I do relish bindings and hotkeys, I have appreciated Emacs Hyperbole's ability to do something appropriate based around contents of the content in my text buffer (though similarly I havent yet tweaked it for all my needs and workflows) - based on simple clicking. In general, I do feel that a 'workbench' approach has a utility, whereby different tools for different functions are available. One should have the ability to orientate towards a tool with minimal cost. A utility should be dependable and expectiations of deviation reasonable. Going back to HTTP, Javascripts like uMatrix (and things like NoScript) have been informative re the contemptuous/irresponsible behaviour of content providers. The combinations for minimal client supplication in order to receive content are fascinating, as over time they break down (not only from shifts in infrastructure but additionally deterring JS blocker users). Im happy to have these impediments in the way, if only to be reminded of the harm. My less tech savvy partner tollerates this to an extent but edge cases really throw her, so all switches go on (!). I recall a recent thread concerning clients and the use of favicons, with a lot of heat regarding whether an attempt at adding visual flair was detrimental. Similarly to the issue of compression, there seems to be an awkward dance which may ossilate between different producers and users of tools and content. Is it possible to avoid an approach which is heavily dependent on (ambiguous?) norms within a site (ie, non protocol hardened expectations), which can be picked up/interpreted by a client? Instead, the client could call upon a repo(s) (pre downloaded, like a CA bundle) - which gives an indication of a URI's policies. If such repo(s) existed, it might allow switching between different clients to satisfy different use cases. For example:
---
Previous Thread: Re: Gemini Digest, Vol 23, Issue 25
Next Thread: [Tech][Idea] Local (Magic) Proxy HTTP to Gemini