💾 Archived View for bbs.geminispace.org › u › istvan › 15969 captured on 2024-05-12 at 19:03:32. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-05-10)
-=-=-=-=-=-=-
Re: "How Can We Determine Files Types and Text File Encodings?"
If you are asking about file types, which is a completely different question, there is typically some form of magic bytes that can be used to make a guess.
Ultimately, it's the responsibility of the software to figure this out.
If you replace the magic for PNGs with JPEG, your OS might guess it is a JPEG and pass it to an image editor. The image editor will attempt to parse the JPEG, find out the data just doesn't work and complain that you passed a broken/invalid JPEG.
So the problem is on the final processing end to solve. Mime and magic is just a shorthand to help guess which software to pass it to for further processing.
Apr 04 · 5 weeks ago
i guess you know about the 'file' utiliy.
To make your life a little easier you can make a utility that detects ASCII and UTF-8 text, the rest you can't automate since there is no real way to identify between different codepages besides using a human to see if it looks correct.
🚀 blah_blah_blah [OP] · Apr 10 at 00:04:
@mozz
But why do you think a polygot file is a security issue? I don't see how it would be more insecure than any other untrusted file.
Secure software has to presume that user input is hostile. One form of hostiliy is the poiyglot file, which appears to be one thing while (in addition, under certain circumstances) being something else.
🚀 blah_blah_blah [OP] · Apr 10 at 00:44:
The responses to my post confirm my view that the final determinant of a file's type or encoding is human judgment about whether expected software chokes on the data or not. I guess only I find this an intriguing topic, or an alarming one.
How Can We Determine Files Types and Text File Encodings? — Determining File Types I have a security question. How can we verify that a UTF-8 file contains only UTF-8 encoded bytes? Running iconv all the time (the preferred solution) isn't appropriate in every situation, and only pushes back the question: how does iconv perform the verification? Other proposals suggest pushing text through UTF-8 language tools, like `read().decode('UTF-8')` in Python, but, again, the /how/ remains...