Regarding non-finite response bodies

1. Drew DeVault (sir (a) cmpwn.com)

A discussion took place in August that was revisited on IRC today:

gemini://gemi.dev/gemini-mailing-list/messages/002404.gmi

I disagree very strongly with the suggestion of gemini+stream://. It
relies upon a presumption about the specification which is poorly
supported: that all Gemini requests are finite.

The supporting text for this is supposedly that section 1.1 lists "C:
Handles response" last. However:

1. Section 1.1 reads like an informative summary of the authoritative
   text which follows
2. There is no explicit specification that these steps are ordered or
   cannot be completed in parallel, and aren't even numbered.

The other relevant piece of text is in 3.3:

> The server closes the connection after the final byte, there is no
> "end of response" signal like gopher's lonely dot.

There is no statement anywhere that there actually has to *be* a final
byte; only a clarification on how the final byte is signalled if
present.

Nothing in the specification prohibits Gemini responses from being
non-finite, and in fact if clients presume this will be the case then
they'll be easily exploited by servers which simply forward /dev/zero
into the connection and never terminate, leading to OOM crashes or disk
space exhaustion.

If you want to stream content, the protocol permits it as-is. And
remember the one big rule:

Do not.
Extend.
Gemini.

Link to individual message.

2. acdw (acdw (a) acdw.net)

On 2020-11-16 (Monday) at 23:01, Drew DeVault <sir at cmpwn.com> wrote:

> 
> If you want to stream content, the protocol permits it as-is. And
> remember the one big rule:
> 
> Do not.
> Extend.
> Gemini.
>

Hear, hear.  Let's put this to rest.  Clients either will or won't handle 
streaming; while some problems with UI might deserve solutions (how does a 
client handle a long-time request?  How to download/stream/save a thing? 
etc.), I'm sure the smart people on this list and around Geminispace can do so.

-- 
~ acdw
acdw.net | breadpunk.club/~breadw

Link to individual message.

3. James Tomasino (tomasino (a) lavabit.com)

On 11/16/20 11:01 PM, Drew DeVault wrote:
> I disagree very strongly with the suggestion of gemini+stream://. It
> relies upon a presumption about the specification which is poorly
> supported: that all Gemini requests are finite.

As the original author of the gemini+stream proposal I want to just jump 
in with my agreement of Drew's response. Everything I had stated was based 
on my *incorrect* assumption that the spec was clear that our connection 
ended before processing data. If that is not the case then any and all 
Gemini clients can support streaming and we have no need for a +stream add-on.

Hooray for streaming!

> Do not.
> Extend.
> Gemini.

Agreed.

Link to individual message.

4. bie (bie (a) 202x.moe)

On Mon, Nov 16, 2020 at 11:43:13PM +0000, James Tomasino wrote:
> On 11/16/20 11:01 PM, Drew DeVault wrote:
> > I disagree very strongly with the suggestion of gemini+stream://. It
> > relies upon a presumption about the specification which is poorly
> > supported: that all Gemini requests are finite.
> 
> As the original author of the gemini+stream proposal I want to just jump 
in with my agreement of Drew's response. Everything I had stated was based 
on my *incorrect* assumption that the spec was clear that our connection 
ended before processing data. If that is not the case then any and all 
Gemini clients can support streaming and we have no need for a +stream add-on.
> 
> Hooray for streaming!
> 
> > Do not.
> > Extend.
> > Gemini.
> 
> Agreed.

100% agreed

I'll update my server to no longer respond to gemini+stream:// requests
(it was already responding to them the exact same way as gemini://
requests, so my gemini-backed streaming radio will, of course, remain
"on air")

bie

Link to individual message.

5. John Cowan (cowan (a) ccil.org)

On Mon, Nov 16, 2020 at 6:43 PM James Tomasino <tomasino at lavabit.com> wrote:

As the original author of the gemini+stream proposal I want to just jump in
> with my agreement of Drew's response. Everything I had stated was based on
> my *incorrect* assumption that the spec was clear that our connection ended
> before processing data. If that is not the case then any and all Gemini
> clients can support streaming and we have no need for a +stream add-on.
>

What concerns me, though, is that this suggests that it is forbidden for
any client to drop a connection when nothing has been sent for a while or
the client believes too much data has been sent.  I think we have to make
it clear that not all clients need to be able to accept streaming
transmissions.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
A mosquito cried out in his pain,
"A chemist has poisoned my brain!"
The cause of his sorrow / Was para-dichloro-
Diphenyltrichloroethane.                                (aka DDT)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201116/c079
89df/attachment-0001.htm>

Link to individual message.

6. Drew DeVault (sir (a) cmpwn.com)

On Mon Nov 16, 2020 at 7:02 PM EST, John Cowan wrote:
> What concerns me, though, is that this suggests that it is forbidden for
> any client to drop a connection when nothing has been sent for a while
> or the client believes too much data has been sent. I think we have to
> make it clear that not all clients need to be able to accept streaming
> transmissions.

Not accepting streaming transmissions is not really something the client
can do and be a spec-compliant implementation. You can set a maximum
size on the response, or you could set a minimum transfer throughput (I
recommend being generous with that one), and you should let the user
cancel an in-progress request.

Link to individual message.

7. John Cowan (cowan (a) ccil.org)

On Mon, Nov 16, 2020 at 7:04 PM Drew DeVault <sir at cmpwn.com> wrote:

> Not accepting streaming transmissions is not really something the client
> can do and be a spec-compliant implementation. You can set a maximum
> size on the response, or you could set a minimum transfer throughput (I
> recommend being generous with that one),


I'm sorry, all those negations break my poor monkey brain.  The first
sentence seems to say that the client must continue reading to the end of
the response, the second sentence says that it may break it off.  Malicious
servers are, or will be, a fact of life; connection problems are already a
fact of life.

you should let the user
> cancel an in-progress request.
>

Not all clients are interactive.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
We pledge allegiance to the penguin and to the intellectual property
regime for which he stands, one world under Linux, with free music
and open source software for all.  --Julian Dibbell on Brazil, edited
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201116/7a4f
7c4d/attachment.htm>

Link to individual message.

8. Drew DeVault (sir (a) cmpwn.com)

On Mon Nov 16, 2020 at 7:11 PM EST, John Cowan wrote:
> I'm sorry, all those negations break my poor monkey brain. The first
> sentence seems to say that the client must continue reading to the end
> of the response, the second sentence says that it may break it off.
> Malicious servers are, or will be, a fact of life; connection problems
> are already a fact of life.

I'm saying that the server could send you a streaming response, and
there's no way for you to stop it from doing so, so you have to be aware
of it and do SOMETHING to make sure it's dealt with properly. And
whatever you choose to do about it should probably be with the best
UI applicable to your use-case.

Link to individual message.

9. John Cowan (cowan (a) ccil.org)

On Mon, Nov 16, 2020 at 7:13 PM Drew DeVault <sir at cmpwn.com> wrote:


> I'm saying that the server could send you a streaming response, and
> there's no way for you to stop it from doing so, so you have to be aware
> of it and do SOMETHING to make sure it's dealt with properly. And
> whatever you choose to do about it should probably be with the best
> UI applicable to your use-case.


Ah, then we agree.  Good.



John Cowan          http://vrici.lojban.org/~cowan        cowan at ccil.org
        Raffiniert ist der Herrgott, aber boshaft ist er nicht.
                --Albert Einstein
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201116/1566
556f/attachment.htm>

Link to individual message.

10. BjΓΆrn WΓ€rmedal (bjorn.warmedal (a) gmail.com)

> [...] It
> relies upon a presumption about the specification which is poorly
> supported: that all Gemini requests are finite.
> [...]
> Nothing in the specification prohibits Gemini responses from being
> non-finite, and in fact if clients presume this will be the case then
> they'll be easily exploited by servers which simply forward /dev/zero
> into the connection and never terminate, leading to OOM crashes or disk
> space exhaustion.

This is interesting. I'm writing a python lib for handling gemini
requests, to simplify browser development. I just assumed that
responses are finite, and as my code works now a method initiates a
request, receives the response, builds a response object and returns
it to the caller.

If responses can be infinite this will indeed cause OOM errors and
crash, and the caller won't know what hit 'em. Should I be reading the
header and return a response object that contains an open socket
instead of a byte array? Can we assume that browser developers
understand this ambiguity and handle it correctly?

- ew0k

Link to individual message.

11. colecmac (a) protonmail.com (colecmac (a) protonmail.com)

> Should I be reading the header and return a response object that
> contains an open socket instead of a byte array?

Yes. This is what I've seen from Go libraries as well, and it provides
a simple solution that gives the developer the most power.


makeworld

Link to individual message.

12. Ben Goldberg (ben (a) benaaron.dev)

When I read the spec I did not understand this.  I'd be nice if this was 
stated explicitly in the the spec if that was in fact the intention.

Link to individual message.

13. Jason McBrayer (jmcbray (a) carcosa.net)

Bj?rn W?rmedal <bjorn.warmedal at gmail.com> writes:

> This is interesting. I'm writing a python lib for handling gemini
> requests, to simplify browser development. I just assumed that
> responses are finite, and as my code works now a method initiates a
> request, receives the response, builds a response object and returns
> it to the caller.

If you get this lib to a good place, let me know, and I will deprecate
`gusmobile'. I've done a pretty terrible job maintaining it because I'm
not actually using it for anything.

-- 
+-----------------------------------------------------------+
| Jason F. McBrayer                    jmcbray at carcosa.net  |
| A flower falls, even though we love it; and a weed grows, |
| even though we do not love it.            -- Dogen        |

Link to individual message.

---

Previous Thread: On input and uploading of user content

Next Thread: Scheme Section 2 quibble