💾 Archived View for bbs.geminispace.org › s › misfin › 17946 captured on 2024-06-20 at 11:53:25. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-06-16)
-=-=-=-=-=-=-
Misfin reference server crashes when mail is sent containing unicode in the name or body of the message and the recipient is using MailDir or MH.
Error ultimately comes from Python itself. The mailbox module is not Unicode safe and leads to a crash:
File "/email/generator.py", line 410, in write
self._fp.write(s.encode('ascii', 'surrogateescape'))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
Appears it may work if you are using gembox storage. Not sure if it's the default. I don't think it was because failure to delivery unicode is what started this search.
The failure is silent and only appears in the logs.
Jun 16 · 4 days ago · 🤔 1
💎 istvan [OP] · Jun 16 at 21:20:
God only knows how you get whoever is maintaining the Python module to fix it, and then you have to wait for that fix to propogate.
This presents a few problems:
You don't know if your recipient is using MH/MailDir, your server reports successful delivery, their server never tells them there was an error.
Even if you want to restore communication, there is no way to change your sender name at the moment if your sender name contains unicode.
It might be best to remove MH/MailDir support until these libraries are fixed, or else write an inbound preprocessor to fix unicode senders/messages in some way that allows delivery through the existing module.
💎 istvan [OP] · Jun 16 at 21:38:
Steps to duplicate:
1. Create misfin user
2. Log in with certificate
3. Change to MH or MailDir in settings (one of these may be default)
4. On a second server, compose a mail to your new user
5. For the message string, use the text "代码错误"
6. Examine syslog for the error after your message does not arrive
@istvan Thanks for reporting this issue.
@istvan Indeed, i've been able to reproduce this with an MH mailbox :\ This hardcoded reencoding of the message to ascii in the email module is really bad. I'll create an issue on gitlab.
The server replies with a 50 error status, so the client is notified that the message wasn't delivered.
@istvan Gembox file is not the default mailbox type, MH is.
Again, the failure is not silent at the protocol level (it replies 50: cannot allocate mailbox), but when you send via the frontend, it doesn't report errors yet, will always say that everything went fine ...
Created an issue here:
— https://gitlab.com/cipres/misfin/-/issues/19
If the MH implementation can't be changed to store unicode messages that's a deal breaker. Just saw in the email module docs that you can set the charset for a mailbox message, will see if this fixes the problem but i doubt it. Thanks again.
I found a fix. When you use set_charset(Charset('utf-8')) it stores messages like this with the payload in base64:
MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 PCBjaXByZXNAZ2VtYWxheWEuZ2l0bGFiLmlvIGNpcHJlcwo6IGJvYmJ5QGxvY2FsaG9zdCAKQCAy MDI0LTA2LTE4VDE3OjA5OjU1WgojIFVURgrku6PnoIHplJnor68=
@istvan Fix pushed. I've tested receiving messages with unicode characters for both MH & Maildir, works fine and doesn't seem to break existing mailboxes. Forcing utf-8 charset before storing seems to be the only solution, otherwise the python mailbox impl tries to reencode to ascii ... Only downside is that the payload is stored as base64 :\
@flipperzero If you ever upgrade, test sending/receiving messages on an existing mailbox just to make sure ...
@flipperzero There's now a page to list/create/subscribe to misfin mailing lists, it's not restricted to admin only now. The link to the mailing lists page is listed on the homepage.
I think that the list of mailing lists on a misfin server should be public.
Thank you very much for your interest in the issue and quick resolution.
I like what you are doing and want misfin to get some traction, but mail dying on unicode was bit of a deal breaker.
I'm kind of shocked no one encountered it before. Maybe some users assumed everything was fine via the frontend and never drilled down to see what was wrong. I'd have never spotted it if I was not trying to debug missing mail on my server.
@istvan You're welcome, and thanks again, yeah instant deal breaker, if you had not raised the issue it might have gone unnoticed for much longer ... Added support for MH/Maildir a few months ago and the unit tests didn't have any messages that included unicode characters .. my fault.
The frontend should also be internationalized and let you select a language for the interface.