This is a chat log of a discussion about how to add streaming support to Gemini with the least disruption to existing clients, or to existing ways of interacting with Gemini servers.
Taken from the IRC log from the link below, some entries have been re-ordered to keep related lines together. Some typos have been corrected.
gemini://tilde.black:1965/users/fox/irc/log.txt
@tomasino: if we aren't gonna do it with a status code, i think a mime-type would be good
@tomasino: text/gemini+stream
kevinsan: i think if the convention was to identify a stream in the URL, then that would at least give prior warning.
acdw: like */stream/* ?
admicos: geminis://
@tomasino: the irc thing is a good example, or logs
acdw: tomasino you can just spam Refresh
kevinsan: tomasino, could streaming work by convention? that is, if the request is /mysyslog.stream, then a client could just choose not to time out?
acdw: I don't see why not
kevinsan: that way, the spec needn't change, clients can avoid streams, but clients can also choose to support them?
lukee: ultimately there is no difference between a very slow feed and a stream
lukee: whether you time out waiting for the end or not is up to you
kevinsan: there is - there's intent.
kevinsan: a slow feed intends to give you 100%, a stream is infinite.
acdw: pisces:// for streaming
kevinsan: i see the use in it, and i don't see a reason to limit what gets done on the port. i don't like having to make a request to discover that I don't want something.
lukee: but you have to design a protocol with a mode of use in mind - what is that mode?
lukee: also I dont know what you mean when you say "i don't like having to make a request to discover that I don't want something"
kevinsan: suppose there's a URL that's intended as a stream (the server will never close the connection). I can't know that it will do that until I make the request and either hit a timeout or get a (newly invented) response code.
lukee: that is a fundamental ambiguity of gemini not having a content-length.
acdw: like, instead of extending gemini you could spec a new streaming profile which is basically like, gemini + streaming
lukee: so you dont know when the end is supposed to be
acdw: you *could* say that gemini is a streaming protocol, since there's no content-length. it streams until it stops
kevinsan: acdw, you could, but you could also say that, practically, it's not.
@tomasino: If everyone built clients with stream as default it would work, but most easy libraries aren't designed that way
@tomasino: That's why I think a mime is simplest now
@tomasino: Clients are required to server text/gemini, but not anything else
kevinsan: right, but why would a mime make those clients with easy libraries any happier?
lukee: Also the gemini spec talks in terms of the co-ordination being that the server closes the request
@tomasino: Define text/gemini+stream and clients can choose to implement it or not
lukee: so that defines part of the expected coordination mode we design around
lukee: I'm fine with whatever text/gemini+foo wants to be, just not the main content type
kevinsan: tomasino, just declare it in the URL and non-steaming clients can ignore it.
@tomasino: What I'm afraid of is that no acknowledgment at all will lead to a bunch of different side tests that only work in a narrow situation. Spec it and say it's optional and there's a standard approach
lukee: I still think this is outside the core application area - arent there existing protocols for that kind of thing?
@tomasino: All we need to do is say, hey, if this mime shows up the connection is streaming. Display as it comes in
lukee: no its not as simple as that
@tomasino: gemtext is perfect for it
@tomasino: Line based top down processed
lukee: gemtext is perfect for lots of things
lukee: but its more about the protocol I'm talking about
lukee: the spec says you go client request-> server response and close
lukee: I just realised, I think the other thing I'm ambivalent about the streaming mode is that it is a shift from client pull based autonomy, to server push
lukee: personally I think that is an important philosophical shift
acdw: the more i think about it the more i think a diff protocol would be good. keeps it really obvious what's going on
easeout: you mean like, if gemini begins offering persistent connection streaming, then it is less of an in-and-out transaction where the user sees and controls every request that happens?
acdw: don't even have to download the first line to get a mimetype
lukee: easeout: yes
easeout: yeah, i think i agree. that's part of the appeal of gemini to me, that i know it's not doing anything "extra"
@tomasino: User still controls the request and gets one ongoing document
lukee: instead of streaming, why not just have a polling client?
acdw: ^ that would work, just refresh every 30s
@tomasino: Because the uses may be different
easeout: works for liveblogs.
kevinsan: lukee, it wouldn't work for audio
lukee: kevinsan: what do you want to do with audio - some kind of live feed? That seems way outside the main use case IMO
kevinsan: lukee, i don't want to do anything with audio (or streaming generally), i was just pointing that polling wouldn't work for that type of streamed content
lukee: kevinsan: ok I agree, its not good for live audio
acdw: poll 14400x a second
@tomasino: If could hook a thermal printer up to a Gemini stream and read back the weather info in a permanent log
lukee: tomasino: that could be done with polling
@tomasino: Speech-to-text broadcast over a gemini stream for sexy accessible presentations
@tomasino: Spoken by NPR, pipe it through Watson's speech to text feature
@tomasino: We can do this with very little effort and not break anything
@tomasino: If we choose to
lukee: no it does break stuff
@tomasino: It breaks nothing
@tomasino: You don't HAVE to stream
@tomasino: Your client doesn't HAVE to support it
lukee: it breaks the basic client controlled request->server response and close model
xj9: as someone who is here for the minimalism, i'm actually on board with streaming
kevinsan: I agree, it doesn't break anything - it just changes behaviour.
lukee: it breaks assumptions of how servers behave
xj9: i will try to resist building a mesh network on top of streaming gemini
@tomasino: If streaming, don't expect close before you process. FIN
lukee: of course one can do whatever one likes with the transient stream, but the current model is document based, so there is an expectation that the server closes before the content is delivered
easeout: i will say that does sound like a pretty surgical change at the protocol level
kevinsan: that's essentially why i want to know in advance that a URL is expected to be a stream.
@tomasino: Yes, the in-progress spec says a thing I'm suggesting we caveat for one specific addition that supports a whole host of cool possibilities in a completely optional way
lukee: if we had content-length it would be an awful lot easier
lukee: but that was rejected
lukee: probably for good reasons
kevinsan: content length still requires me to make a request to something it transpires i didn't want
kevinsan: or that my client can't support (e.g. maybe my library doesn't let me set a timeout)
lukee: but the server would be able to indicate how much content is expected before close
easeout: so, the "completely optional" part. it's optional because, why, as a client you can declare your capability to stream, and then the server knows to behave differently on certain pages? and if you don't, then it still just sends a current state and closes?
lukee: (for the static content scenario)
@tomasino: As a client you get an unsupported mime type and kill the load
easeout: ok, i don't feel like that's optional then, as a client
easeout: i feel like if i don't implement that, i break part of the experience of gemini for my users
lukee: I'm fine with getting a mime type and timing out or abandoning a non-closing server feed
@tomasino: If you don't support downloading binary files, or loading images, same thing
lukee: its not the same thing
lukee: those are fixed length packets of content
kevinsan: if my client is a ncat, or s_client, or gnutls-cli, or something else, you can't be sure I can make those decisions at response time.
@tomasino: If you're using those it'll just work
kevinsan: no, it'll hang
@tomasino: Ncat displays as it comes in
acdw: from what I remember s_client *does* hang
acdw: or times out
@tomasino: Not familiar with that one
kevinsan: the point isn't so much specific implementations, it's the principle that the client may not be able to make these decisions at response time.
kevinsan: so the request/response model of gemini is broken without some means of identifying a stream up-front.
easeout: so what about at request time like an accept header
kevinsan: then you place the burden on the client to mitigate infinite responses
kevinsan: that seems wrong (and needlessly complicated)
@tomasino: gemini+stream://?
lukee: my take on it is that there is a fundamental principle that the client is in control and expects a fixed stream to be delivered
easeout: my take is like lukee's except i could be ok with streaming if it's foreground not background; i would still feel in control
easeout: but i would then want it to be in the spec, optional for the moment, but necessary if content takes off
lukee: easeout: an accept header would help, but we dont have anything like that
@tomasino: It would be great if there was a safe and reliable way to support it that didn't freak people out
easeout: yeah. so what exactly are we freaked out by
@tomasino: The spec says the connection closes then you process
easeout: for my part, i like that the spec is small and i feel like i could implement it fast, and i don't know how much effort this would add to building a cilent.
lukee: I think another protocol, so it is clearly indicated in advance that you can expect a non closing stream
easeout: true. i do look to the protocol in a link to know what to expect, especially since there are so many links to gopher and http around gemini.
lukee: I've changed my mind that trying to indicate this in the content-type is sufficient
easeout: you mean, because users don't see it before the request is made?
lukee: yes, with an alternative protocol marker in the url you know what to expect
easeout: i'm down with that
@tomasino: Gemini+stream://
lukee: yeah, or some other name. Doesnt really matter I dont think