💾 Archived View for gemi.dev › gemini-mailing-list › 001060.gmi captured on 2023-11-04 at 13:18:00. Gemini links have been rewritten to link to archived content
-=-=-=-=-=-=-
Greetings Geminauts, I've just pushed the first changes to the official Gemini specification in close to one year! As always the official specification can be viewed at:
Oh dear, some kind of git hook failure means the updates aren't actually visible at the advertised locations. I'm aware and working on it. Sorry! Cheers, Solderpunk
On Sun, 07 Nov 2021 15:50:09 +0000 Solderpunk <solderpunk at posteo.net> wrote: > 1. Gemini URIs with empty paths (e.g. `gemini://example.com`) and those > with paths of `/` (e.g. `gemini://example.com/`) are now equivalent by > definition, and both clients and servers SHOULD normalise empty paths to > `/` before sending/processing requests (along with applying other > standard URI normalisations). Excellent. Overall a good set of small changes, basically all clarifications that most implementations should already do right. This makes me hopeful future changes to the spec will be of the same nature. -- Philip
Argh, my bad, the updates *were* visible I just forgot to commit the version number bump. Fixed now. Everything is under control, situation normal... Cheers, Solderpunk On 07.11.2021 16:58, Solderpunk wrote: > Oh dear, some kind of git hook failure means the updates aren't > actually visible at the advertised locations. I'm aware and working > on it. Sorry! > > Cheers, > Solderpunk
On Sun, Nov 07, 2021 at 03:50:09PM +0000, Solderpunk <solderpunk at posteo.net> wrote a message of 75 lines which said: > 2. The use of the TLS `close_notify` mechanism is now mandatory (see > sections 1.1 and 4). This will be hard to enforce: gemini://gemini.bortzmeyer.org/software/lupa/stats.gmi 48.3?% of URLs do NOT send a proper TLS shutdown (application close). Even 36.1?% of those who return status 20 are in that case.
On 07.11.2021 17:09, Philip Linde wrote: > Excellent. > > Overall a good set of small changes, basically all clarifications that > most implementations should already do right. This makes me hopeful > future changes to the spec will be of the same nature. My hope is that I can push small, simple batches of clarifications like this once every week or two for the rest of the year and settle most of the outstanding issues. Ideally most of the changes will be such that most implementers need to either do nothing at all or make fairly minor and uncontroversial changes. I don't want to *promise* there will be nothing bigger than this - but I'm reluctant to do things bigger than this and won't do it unless I really think I have to. Cheers, Solderpunk
This will be hard to enforce: > > gemini://gemini.bortzmeyer.org/software/lupa/stats.gmi > > 48.3?% of URLs do NOT send a proper TLS shutdown (application > close). Even 36.1?% of those who return status 20 are in that case. Well, then there are (and always has been) a lot of non-compliant servers out there. It's not in my power to put something in the Gemini specification which contradicts what the TLS specifications say. My hope is that a very large percentage of those URLs come from a relatively small number of server implementations and with a concerted effort we can improve things quickly. Cheers, Solderpunk
Stephane Bortzmeyer <stephane at sources.org> writes:
> 48.3?% of URLs do NOT send a proper TLS shutdown (application
> close). Even 36.1?% of those who return status 20 are in that case.
Please let me know if carcosa.net is among those. Could you also post
a simple program or script for testing this? I feel like we *ought* to
be able to get server authors to fix this pretty easily, unless there
are issues with language bindings making it difficult.
--
Jason McBrayer | ?Strange is the night where black stars rise,
jmcbray at carcosa.net | and strange moons circle through the skies,
| but stranger still is lost Carcosa.?
| ? Robert W. Chambers,The King in Yellow
Jason McBrayer <jmcbray at carcosa.net> writes: > Stephane Bortzmeyer <stephane at sources.org> writes: >> 48.3?% of URLs do NOT send a proper TLS shutdown (application >> close). Even 36.1?% of those who return status 20 are in that case. > > Please let me know if carcosa.net is among those. Could you also post > a simple program or script for testing this? I feel like we *ought* to > be able to get server authors to fix this pretty easily, unless there > are issues with language bindings making it difficult. I always forgot the openssl(1) incantation to check for the close_notify flag, but portal.mozz.us has a nice UI and clicking on '[view cert]' it logs the close_notify alert status. For carcosa.net seems to be absent :/ https://portal.mozz.us/gemini/carcosa.net?crt=1
On Mon, Nov 08, 2021 at 09:28:41AM -0500,
Jason McBrayer <jmcbray at carcosa.net> wrote
a message of 15 lines which said:
> Please let me know if carcosa.net is among those.
Indeed it is :-(
% agunua carcosa.net
Warning, no TLS shutdown received from the server
```
__| |
( _` | _|_| _ \(_-< _` | \ -_) _|
\___|\__,_|_|\__|\___/___/\__,_|_)_| _|\___|\__|
```
============================================================
This is the Gemini site for carcosa.net. It is running on Germinal, a
Gemini server in Common Lisp.
...
> Could you also post a simple program or script for testing this?
gemini://gemini.bortzmeyer.org/software/agunua/
("pip3 install agunua" will work if you have Python installed)
Hi Stephane! On Mon, 8 Nov 2021 16:18:11 +0100 Stephane Bortzmeyer <stephane at sources.org> wrote: > On Mon, Nov 08, 2021 at 09:28:41AM -0500, > Jason McBrayer <jmcbray at carcosa.net> wrote > a message of 15 lines which said: > > > Please let me know if carcosa.net is among those. > > Indeed it is :-( > > % agunua carcosa.net > Warning, no TLS shutdown received from the server Very useful. I have not considered TLS in detail myself, but I tested this just now with the Go TLS implementation which does seem to send the proper notification as you call Close on connections. Good news for people writing server implementations in Go, I think. I wonder how hard it would be to automatically identify server software that doesn't implement this properly. Probably some server software could be differentiated based on their error code descriptions or how they deal with some corner cases. If we could trace these capsules to a few server implementations it might not be too much work to reach out to the authors and poke at them about the spec change. With the numbers you suggested earlier, I am wary of strictly enforcing close_notify in my client, but in the interim maybe client authors could warn as yours does, or provide some way to add domains to an exception list. -- Philip
Stephane Bortzmeyer <stephane at sources.org> writes:
>> Could you also post a simple program or script for testing this?
>
> gemini://gemini.bortzmeyer.org/software/agunua/
>
> ("pip3 install agunua" will work if you have Python installed)
The gemini-diagnostics tool
<https://github.com/michael-lazar/gemini-diagnostics> also does the
no-shutdown check.
-- Chris
Chris Brannon <chris at the-brannons.com> writes:
> Stephane Bortzmeyer <stephane at sources.org> writes:
>
>>> Could you also post a simple program or script for testing this?
>>
>> gemini://gemini.bortzmeyer.org/software/agunua/
>>
>> ("pip3 install agunua" will work if you have Python installed)
>
> The gemini-diagnostics tool
> <https://github.com/michael-lazar/gemini-diagnostics> also does the
> no-shutdown check.
I'm not sure that it works correctly. I tried against gmid (my server)
and it always fails the close_notify test, even though both
portal.mozz.us and agunua agree that the close_notify alert is sent.
It's a useful tool otherwise.
> -- Chris
On Mon, Nov 08, 2021 at 05:14:18PM +0100, Philip Linde <linde.philip at gmail.com> wrote a message of 35 lines which said: > I wonder how hard it would be to automatically identify server > software that doesn't implement this properly. Probably some server > software could be differentiated based on their error code > descriptions or how they deal with some corner cases. Yes, fingerprinting servers is certainly possible. Someone to take this project? Note that some servers send the TLS proper close for "normal" replies but fail to do so when there is an error, such as 51. It helps with fingerprinting, though.
Regarding the closing of connections: I know from using gemini-diagnostics that my server is one of those. I just use a generic Perl async server framework and call ?close_gracefully? at the end but somewhere in the layers of libraries from Mojo::IOLoop to IO::Socket::SSL to Net::SSLeay to OpenSSL somebody doesn?t handle this the way people expect it to work and I have no idea how to fix it (and I have stared at the source code of all these libraries for more hours than I cared for). So if anybody knows how to fix this using Perl, I?m interested in hearing about it. I got some helpful hints on Station so perhaps yet another look into it all will finally enlighten me. -- Typed on a tiny keyboard. Sorry for being terse. > On 8 Nov 2021, at 15:31, Jason McBrayer <jmcbray at carcosa.net> wrote: > > ? > Stephane Bortzmeyer <stephane at sources.org> writes: >> 48.3 % of URLs do NOT send a proper TLS shutdown (application >> close). Even 36.1 % of those who return status 20 are in that case. > > Please let me know if carcosa.net is among those. Could you also post > a simple program or script for testing this? I feel like we *ought* to > be able to get server authors to fix this pretty easily, unless there > are issues with language bindings making it difficult.
Omar Polo <op at omarpolo.com> writes:
> For carcosa.net seems to be absent :/
> https://portal.mozz.us/gemini/carcosa.net?crt=1
Ugh, thanks. I will have to look into it, and just hope the wrapper I'm
using doesn't make it too difficult.
--
Jason McBrayer | ?Strange is the night where black stars rise,
jmcbray at carcosa.net | and strange moons circle through the skies,
| but stranger still is lost Carcosa.?
| ? Robert W. Chambers,The King in Yellow
Stephane Bortzmeyer <stephane at sources.org> writes:
> On Mon, Nov 08, 2021 at 09:28:41AM -0500,
> Jason McBrayer <jmcbray at carcosa.net> wrote
> a message of 15 lines which said:
>
>> Please let me know if carcosa.net is among those.
>
> Indeed it is :-(
Thanks. Clearly, using cl+ssl in the default ways doesn't close the
connection properly; hopefully it won't be too difficult to make it
happen.
--
Jason McBrayer | ?Strange is the night where black stars rise,
jmcbray at carcosa.net | and strange moons circle through the skies,
| but stranger still is lost Carcosa.?
| ? Robert W. Chambers,The King in Yellow
On Mon, 2021-11-08, Stephane Bortzmeyer wrote: >Note that some servers send the TLS proper close for "normal" replies >but fail to do so when there is an error, such as 51. Molly Brown among them, according to Agunua: $ agunua gemini://rawtext.club/no-such-page Warning, no TLS shutdown received from the server Problem, Not found (extra message: "Not found!").
On Tue, Nov 09, 2021 at 08:28:35AM +0000, nervuri <nervuri at disroot.org> wrote a message of 9 lines which said: > > Note that some servers send the TLS proper close for "normal" replies > > but fail to do so when there is an error, such as 51. > > Molly Brown among them, according to Agunua: Indeed, see for instance the official site :-( % agunua gemini.circumlunar.space # Project Gemini ## Overview ... % agunua gemini.circumlunar.space/doesnotexist Warning, no TLS shutdown received from the server Problem, Not found (extra message: "Not found!").
Jason McBrayer <jmcbray at carcosa.net> writes:
> Thanks. Clearly, using cl+ssl in the default ways doesn't close the
> connection properly; hopefully it won't be too difficult to make it
> happen.
I'm pleased to note that this was an upstream bug, in CL+SSL, which has
been fixed. Anyone who installed Germinal the default way (Roswell and
quicklisp) should run (ql:update-all-dists) and (ql:quickload :cl+ssl)
to upgrade their CL+SSL and restart Germinal.
I added a line of code to explicitly close TLS streams, but I don't
believe it was actually needed; CL+SSL should be doing the right thing
when streams are implicitly closed.
--
Jason McBrayer | ?Strange is the night where black stars rise,
jmcbray at carcosa.net | and strange moons circle through the skies,
| but stranger still is lost Carcosa.?
| ? Robert W. Chambers,The King in Yellow
Forwarding the message I sent below to the rest of the mailing list as I accidentally hit "Reply" instead of "Reply All". Christian Seibold Sent with ProtonMail Secure Email. ??????? Original Message ??????? On Sunday, November 14th, 2021 at 7:37 PM, Krixano <krixano at protonmail.com> wrote: > I wanted to write here to confirm that anything using the Gig Framework (https://github.com/pitr/gig) should be doing close_notify correctly, as I checked my capsule with portal.mozz.us for this case. I'll also notify the developer as well. > > Christian Seibold > > Sent with ProtonMail Secure Email. >
Correction, I looked into it further. The Gig Framework does the close_notify correctly on most pages, but does not do it for pages that give error codes (I haven't checked all error codes yet, but I have checked the not found error code). Christian Seibold Sent with ProtonMail Secure Email. > ??????? Original Message ??????? > > On Sunday, November 14th, 2021 at 7:37 PM, Krixano krixano at protonmail.com wrote: > > > I wanted to write here to confirm that anything using the Gig Framework (https://github.com/pitr/gig) should be doing close_notify correctly, as I checked my capsule with portal.mozz.us for this case. I'll also notify the developer as well. > > > > Christian Seibold > > > > Sent with ProtonMail Secure Email.
So, I've tested some capsules with agunua. This is the list of capsules that have had close_notify on pages that were found, but did not have it on pages that were not found. This list is also every single url that I've tested so far where the issue has come up - all of the tests have just been for the Not Found errorcode. I have also included the server software (if known): gemini://gemini.bortzmeyer.org/blashsdfh - gemserv gemini://skyjake.fi/blahkslhdf gemini://astrobotany.mozz.us/blahssldhslh gemini://botond.online/blashlhsdfh station.martinrue.com/blahsdlhsdlfh gemini://kwiecien.us/blajjjsdhsdh - geminid gemini://auragem.space/notfound - Gig Framework gemini.circumlunar.space/blahdhd - Molly Brown gemini://geminispace.info/blashldshfsldhf gemini://marginalia.nu/blahdslkhdfh - custom server gemini://gemini.thegonz.net/aslhsdlhfsdfh gemini://geminiquickst.art/blashdslfhsldhf gemini://nytpu.com/blahsdlhfsfdlh gemini://konpeito.media/blashsldhsdlfh gemini://rawtext.club/blashsldh gemini://srht.site/blashlsdhf gemini://godocs.io/blashdlfkh I just want to make sure about this, are we sure agunua is reporting things correctly? I find it odd that *every single server* I've tested has had this problem with the notfound errorcode, but not with successful requests. Christian Seibold Sent with ProtonMail Secure Email. ??????? Original Message ??????? On Sunday, November 14th, 2021 at 8:44 PM, Krixano <krixano at protonmail.com> wrote: > Correction, I looked into it further. The Gig Framework does the close_notify correctly on most pages, but does not do it for pages that give error codes (I haven't checked all error codes yet, but I have checked the not found error code). > > Christian Seibold > > Sent with ProtonMail Secure Email. > > > ??????? Original Message ??????? > > > > On Sunday, November 14th, 2021 at 7:37 PM, Krixano krixano at protonmail.com wrote: > > > > > I wanted to write here to confirm that anything using the Gig Framework (https://github.com/pitr/gig) should be doing close_notify correctly, as I checked my capsule with portal.mozz.us for this case. I'll also notify the developer as well. > > > > > > Christian Seibold > > > > > > Sent with ProtonMail Secure Email.
On Mon, Nov 15, 2021 at 4:29 AM Krixano <krixano at protonmail.com> wrote: > > I just want to make sure about this, are we sure agunua is reporting things correctly? I find it odd that *every single server* I've tested has had this problem with the notfound errorcode, but not with successful requests. I filed a bug for the server I use (Agate) and from the maintainer's comment it seems that there is indeed a problem with Agunua: https://github.com/mbrubeck/agate/issues/100#issuecomment-965174851
Krixano <krixano at protonmail.com> writes:
> I just want to make sure about this, are we sure agunua is reporting
> things correctly? I find it odd that *every single server* I've tested
> has had this problem with the notfound errorcode, but not with
> successful requests.
I believe agunua has a problem with reporting no close_notify on error
pages, reporting that it is absent whether it was actually sent or
not. I spent some time over the weekend fixing this issue in Germinal,
making sure that the same code-paths were used for both 2x responses and
for errors, but agunua still reported no close_notify for 51 pages. I
checked the same error pages with portal.mozz.us, and it *is* seeing the
close_notify being sent. So, I'm going to assume the error is in agunua.
--
Jason McBrayer | ?Strange is the night where black stars rise,
jmcbray at carcosa.net | and strange moons circle through the skies,
| but stranger still is lost Carcosa.?
| ? Robert W. Chambers,The King in Yellow
Luckily it's pretty easy to test this with openssl's s_client, it'll say
"closed" at the end if the connection was closed properly:
printf "gemini://nytpu.com/about.gmi\r\n" |
openssl s_client -ign_eof -connect nytpu.com:1965
outputs:
depth=0 O = nytpu, CN = nytpu.com
verify error:num=18:self signed certificate
[...handshake and cert info...]
20 text/gemini; lang=en-US
[...the actual response body (if success response code)...]
[...end-of-connection info...]
closed
Here's a handy little shell function where you give it a URI and it'll
check if it's closed properly or not:
test_close_notify() {
hostname="$(printf "$1" | sed -e 's/[^/]*\/\/\([^@]*@\)\?\([^:/]*\).*/\2/')"
output="$(printf "%s\r\n" "$1" |
openssl s_client -ign_eof -connect "${hostname}:1965" 2>&1 |
tail -n1 | tr -d '\n')"
if [ "${output}" = "closed" ]; then
printf "close_notify for '%s' properly recieved.\n" "$1" >&2
else
printf "close_notify for '%s' not received!\n" "$1" >&2
fi
}
~nytpu
--
Alex // nytpu
alex at nytpu.com
gpg --locate-external-key alex at nytpu.com
On Mon, Nov 15, 2021 at 03:28:44AM +0000, Krixano <krixano at protonmail.com> wrote a message of 63 lines which said: > gemini://gemini.bortzmeyer.org/blashsdfh - gemserv Actually Stargazer, no longer Gemserv. > I just want to make sure about this, are we sure agunua is reporting > things correctly? I find it odd that *every single server* I've > tested has had this problem with the notfound errorcode, but not > with successful requests. Under investigation: <https://framagit.org/bortzmeyer/agunua/-/issues/50>
---
Previous Thread: [users] Gemini "Server of the Day" & database
Next Thread: Request for feedback from server/client implementers using non-OpenSSL TLS stacks