💾 Archived View for gemi.dev › gemini-mailing-list › 000284.gmi captured on 2024-03-21 at 17:17:57. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-12-28)
-=-=-=-=-=-=-
Hi all, I'm juhani, new here, and implementing an Android gemini client. While testing with live sites for the first time I found that some sites worked, and on others the response handling hung. Comparison of tcp streams revealed that on the working sites the last packet from the server had FIN flag, and on the failing sites it didn't. I changed the client so it doesn't try to read the whole response before parsing the header. And added timeout to make sure the connection is closed. These changes make sense to me, but what should the client do if it cant be sure it has the complete payload? With line based content it's possible to show what you've got. The clients I've tried, so far, seem to do that, but is it correct? And what about binary content? ~juhani gemini://envs.net/~juhani https://git.envs.net/juhani/minigem
> On Jul 9, 2020, at 14:27, juhani <juhani at envs.net> wrote: > > the working sites the last packet from the server had FIN flag, and on the failing sites it didn't. In principle, the server MUST close the connection. Gemini doesn't have the concept of persistent connection, or content length, or chunking, or what not. Nothing but close. If the server doesn't explicitly close the connection, then the client is toast, as there is no other mechanism to indicate end-of-transmission. > but what should the client do if it cant be sure it has the complete payload? Pray. Punt. Apply any random heuristics you see fit. But really, no close, no cigar. > With line based content it's possible to show what you've got. The clients I've tried, so far, seem to do that, but is it correct? It's a heuristic. Your milage may vary. The short of it is that there is no mechanism to indicate anything aside from the server cleanly closing the connection. > And what about binary content? Use a different protocol.
> On Jul 9, 2020, at 14:27, juhani <juhani at envs.net> wrote:
>
> Comparison of tcp streams revealed
> that on the working sites the last packet from the server had FIN flag, and on
> the failing sites it didn't.
While at it, perhaps the spec should make closing the connection more prominent.
At the moment, it's shown in 1.1 Gemini transactions:
S: Sends response header (one CRLF terminated line), closes connection
under non-success conditions (see 3.1 and 3.2)
S: Sends response body (text or binary data) (see 3.3)
S: Closes connection
And mentioned in 3.1 Response headers:
If <STATUS> does not belong to the "SUCCESS" range of codes, then the
server MUST close the connection after sending the header and MUST NOT
send a response body.
This could be perhaps reemphasized with:
If <STATUS> belongs to the "SUCCESS" range of code, then the server MUST
close the connection after sending the header and response body.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20200709/241e
8ae7/attachment.htm>
On Thursday 9 July 2020 14:52, Petite Abeille <petite.abeille at gmail.com> wrote: > > And what about binary content? > > Use a different protocol. Gemini can't send image?
> On Jul 9, 2020, at 15:14, defdefred <defdefred at protonmail.com> wrote: > > On Thursday 9 July 2020 14:52, Petite Abeille <petite.abeille at gmail.com> wrote: >>> And what about binary content? >> >> Use a different protocol. > > Gemini can't send image? Sure it can. But perhaps it shouldn't. Depends on how much reliability one is after. See "File size" in "Best practices for Gemini implementations": https://portal.mozz.us/gemini/gemini.circumlunar.space/docs/best-practices.gmi Gemini servers do not inform clients of the size of files they are serving, which can make it difficult to detect if a connection is closed prematurely due to a server fault. This risk of this happening increases with file size. Gemini also has no support for compression of large files, or support for checksums to enable detection of file corruption, the risk of which also increases with file size. For all of these reasons, Gemini is not well suited to the transfer of "very large" files. Exactly what counts as "very large" depends to some extent on the speed and reliability of the internet connections involved, and the patience of the users. As a rule of thumb, files larger than 100MiB might be thought of as best served some other way. Of course, because Gemini supports linking to other online content via any protocol with a URL scheme, it's still possible to link from a Gemini document to a large file served via HTTPS, BitTorrent, IPFS or whatever else tickles your fancy. Replace "very large file" with binary and you get the gist of it. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20200709/421a 498b/attachment.htm>
On 9.7.2020 15.52, Petite Abeille wrote: >> On Jul 9, 2020, at 14:27, juhani <juhani at envs.net> wrote: >> but what should the client do if it cant be sure it has the complete payload? > Pray. Punt. Apply any random heuristics you see fit. But really, no close, no cigar. Roger that :)
I raised an issue with jetforce for this and poked around in twisted but we might be stuck and could use help getting this fixed. I think jetforce is the most popular server with this issue. https://github.com/michael-lazar/jetforce/issues/32
Gemini absolutely can send binary files -- AFAICT that's one of the main reasons it has the MIME type included in the header. Also, I don't think it's helpful to replace "very large" with "binary" -- there are plenty of binary files that are smaller than 100 MiB, including images of almost any size, lots of binary executables, songs, and even some shorter musical albums. While it's not a great idea to serve giant files over gemini, it's certainly possible (see konpeito.media, sloum's Drift Theory on circumlunar.space), and clients should *absolutely* be able to handle them. A client author could write some checks to see if a download is taking too long or hanging, or even filter based on received MIME type, but I think the gemini experience would be lessened if a user couldn't download, e.g., images from gemini. ~ acdw On Thu, Jul 9, 2020, at 8:21 AM, Petite Abeille wrote: > > >> On Jul 9, 2020, at 15:14, defdefred <defdefred at protonmail.com> wrote: >> >> On Thursday 9 July 2020 14:52, Petite Abeille <petite.abeille at gmail.com> wrote: >>>> And what about binary content? >>> >>> Use a different protocol. >> >> Gemini can't send image? > > Sure it can. But perhaps it shouldn't. Depends on how much reliability one is after. > > See "File size" in "Best practices for Gemini implementations": > https://portal.mozz.us/gemini/gemini.circumlunar.space/docs/best-practices.gmi > > > **Gemini servers do not inform clients of the size of files they are serving, which can make it difficult to detect if a connection is closed prematurely due to a server fault*. This risk of this happening increases with file size.* > ** > *Gemini also has no support for compression of large files, or support for checksums to enable *detection of file corruption*, the risk of which also increases with file size.* > ** > *For all of these reasons, *Gemini is not well suited to the transfer* of "very large" files. Exactly what counts as "very large" depends to some extent on the speed and reliability of the internet connections involved, and the patience of the users. As a rule of thumb, files larger than 100MiB might be thought of as best served some other way.* > ** > *Of course, because Gemini supports linking to other online content via any protocol with a URL scheme, *it's still possible to link from a Gemini document to a large file served via HTTPS*, BitTorrent, IPFS or whatever else tickles your fancy.* > > Replace "very large file" with binary and you get the gist of it. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20200709/2627 6f73/attachment.htm>
> On Jul 9, 2020, at 16:17, Case Duckworth <acdw at acdw.net> wrote: > > and clients should *absolutely* be able to handle them As long as the server bother to actually *close* the darn connection :D But yes, we are in agreement: Gemini can do it. If it should -or shouldn't- be used for such purpose is another question altogether.
> I changed the client so it doesn't try to read the whole response before parsing > the header. And added timeout to make sure the connection is closed. This sounds like the way to go! You can't reply on the server being compliant. I wish we'd get a stream status code so you could know when to disable that, though... Also for the record, my Go clients never experience this "not closing the connection" issue. Amfora attempts to load all the data before displaying it (this will change), and I've never had any problems with that on any servers. It always finds the end of the data fine, presumably indicated by a closed connection. Maybe the Go TLS implementation is just more robust? makeworld
> On Jul 9, 2020, at 16:38, colecmac at protonmail.com wrote: > > You can't rely on the server being compliant. Right... (7) It is always something* That said, the server has one and only one hard requirement: close the darn connection. Perhaps this should be a minimum requirement?
On Thu, Jul 9, 2020 at 8:28 AM juhani <juhani at envs.net> wrote: > Hi all, > > I'm juhani, new here, and implementing an Android gemini client. Hi! > While testing with live sites for the first time I found that some sites worked, > and on others the response handling hung. Comparison of tcp streams revealed > that on the working sites the last packet from the server had FIN flag, and on > the failing sites it didn't. > > I changed the client so it doesn't try to read the whole response before parsing > the header. And added timeout to make sure the connection is closed. There is an open issue with jetforce where it was determined that the server sends the TLS close_notify alert but doesn't ever close the TCP socket [1]. I haven't had the time to figure out exactly what is going on yet though. I think that either the server is waiting for the close_notify acknowledgement message from the client which it never receives, or the server is not reacting to the acknowledgement by shutting down the socket. I do find it strange that it still appears to work with 90% of gemini client implementations. I have found that if the client sends a close_notify for their write stream after they transmit the request, then the server will close the connection fully. The other possibility is that maybe your client isn't sending the acknowledgement to the server? My understanding of TLS is that it's not required for the server to wait for the acknowledgement from the client, so ideally the server should be sending the close_notify and then immediately closing the connection afterwards. The main point that I want to make here is that this is a lot more nuanced than waving it off as "just close the connection". The TLS connection needs to be shut down safely to prevent truncation attacks, and to make things more complicated the rules are different between TLS 1.2 and 1.3. - mozz [1] https://github.com/michael-lazar/jetforce/issues/32
On 9.7.2020 17.38, colecmac at protonmail.com wrote: > the data fine, presumably indicated by a closed connection. Maybe the Go TLS > implementation is just more robust? That seems to be the case. I recorded a request with amfora and jetforce. Looks like jetforce terminates the TLS session, amfora confirms that, and then amfora closes the socket. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20200710/6cdd 690f/attachment.htm>
On Thu Jul 9, 2020 at 3:27 PM CEST, juhani wrote: > On 9.7.2020 15.52, Petite Abeille wrote: > > >> On Jul 9, 2020, at 14:27, juhani <juhani at envs.net> wrote: > >> but what should the client do if it cant be sure it has the complete payload? > > Pray. Punt. Apply any random heuristics you see fit. But really, no close, no cigar. > > Roger that :) I guess one is almost always going to get at least the MIME type of the partial content, so really robust clients could probably implement a strategy of looking at that to decide how to proceed (although proceeding at all in this scenario is always going to be hit-or-miss, and in principle it should be a perfectly valid decision for a client to display an error message and nothing more if the connection times out without closing, or if it closes without a proper TLS finishing message). In principle one could categorise MIME types into those where handling partial content is likely to degrade gracefully (e.g. uncompressed, "linear" content - text, wav files maybe?) and those where trying to proceed is just likely to cause an error. Cheers, Solderpunk
juhani <juhani at envs.net> writes: > I'm juhani, new here, and implementing an Android gemini client. Hi, juhani! > While testing with live sites for the first time I found that some sites worked, > and on others the response handling hung. Comparison of tcp streams revealed > that on the working sites the last packet from the server had FIN flag, and on > the failing sites it didn't. Can you tell us what server software the failing sites are running? Probably the most helpful thing is to get those servers fixed. > I changed the client so it doesn't try to read the whole response before parsing > the header. And added timeout to make sure the connection is closed. That also sounds reasonable; just don't make the timeout *too* short. > These changes make sense to me, but what should the client do if it cant be sure > it has the complete payload? With line based content it's possible to show what > you've got. The clients I've tried, so far, seem to do that, but is it correct? > And what about binary content? For binary files, I'd say treat it as an error; Gemini doesn't have resumable downloads, so it's either that or retry, and I think you should leave retrying up to the user. For text, you can show what you have, and leave it to the user to decide if they got the full document or not, I guess... -- +-----------------------------------------------------------+ | Jason F. McBrayer jmcbray at carcosa.net | | A flower falls, even though we love it; and a weed grows, | | even though we do not love it. -- Dogen |
??????? Original Message ??????? On Friday, July 10, 2020 12:20 AM, juhani <juhani at envs.net> wrote: > On 9.7.2020 17.38, colecmac at protonmail.com wrote: > > > the data fine, presumably indicated by a closed connection. Maybe the Go TLS > > implementation is just more robust? > > That seems to be the case. I recorded a request with amfora and jetforce. > Looks like jetforce terminates the TLS session, amfora confirms that, and > then amfora closes the socket. Thanks for looking into this! Good to know what's happening behind the scenes. I assume it's exactly the same for gemget. Thanks, makeworld
---
Previous Thread: About limiting the non human internet bandwidth pollution
Next Thread: What's the status of gemini browsers and image handling?