💾 Archived View for gemini.omarpolo.com › post › inspecting-zips.gmi captured on 2022-03-01 at 15:01:44. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Inspecting zip files

Written while listening to “Twisted Logic” by Coldplay.

Published: 2021-08-19

Tagged with:

#literate-programming

Part two: “Extracting files from zips”

The code for the whole series; see ‘zipls.c’ for this post in particular.

Edit 2021/08/20: some edits to improve the code and the commentary.

Edit 2021/08/21: stricter while condition for ‘ls’ and added links to the code

Disclaimer: before today I didn’t knew anything about how zip files are structured, so take everything here with a huge grain of salt. The good news is that the code I wrote seems to be coherent with what I’ve read online and to actually work against some zips files I had around.

Background: I’d like to add support for gempubs to Telescope, the Gemini client I’m writing. gempubs are basically a directory of text/gemini files plus other assets (metadata.txt and images presumably) all zipped in a single archive.

gempub: a new eBook format based on text/gemini

Telescope

There are a lot of libraries to handle zip files, but I decided to give it a shot a writing something from scratch. After all, I don’t need to edit zips or do fancy stuff, I only need to read files from the archive, that’s all.

To start, in this entry we’ll only see how to dump the list of files in a zip archive. Maybe future entries will deal with more zip stuff.

From what I’ve gathered from APPNOTE.TXT and other sources, a zip file is a sequence of file “records” (a header followed by the file content) and a trailing “central directory” that holds the information about all the files.

APPNOTE.TXT

The structure of a PKZip file

ZIP (Wikipedia)

Having the central directory at the end of the file instead that at the beginning seems to be a choice to waste people time^W^W^W allow embedding zips into other file formats, such as GIFs or EXE. I guess in some cases this may be an invaluable property, I just fail to see where, but anyway.

Edit 2021/08/20: Another advantage of having the central directory at the end is that is probably possible to build up a zip on-the-fly, maybe outputting to standard output or to a similar non-seekable device, without having to build all the zip in memory first.

One may think that it’s possible to scan a zip by reading these “records”, but it’s not the case unfortunately: the only source of truth for the actual files stored in the archive is the central directory. Applications that modify the zip may reuse or leave dummy file headers around, especially if they delete or replace files.

To aggravate the situation, it’s not obvious how to find the start of the central directory. Zip are truly wonderful, huh? I guess that adding a trailing 4-byte offset that points to the start of the central directory wouldn’t be bad, but we’re a bit too late.

The central directory is a sequence of record that identifies the files in the archive followed by a digital signature, two ZIP64 fields and the end of the central directory record. I still haven’t wrapped my head around the digital signature and the ZIP64 fields, but they don’t seem necessary to access the list of files.

The last part of the central directory, the end record, contains a handy pointer to the start of the content directory. Unfortunately, it also contains a trailing variable-width comment area that complicate things a bit.

But enough with the talks, let’s jump to the code. Since Telescope is written in C, the small toy program object of this entry will also be written in C. The main function is pretty straightforward:

int
main(int argc, char **argv)
{
	int	 fd;
	void	*zip, *cd;
	size_t	 len;

	if (argc != 2)
		errx(1, "missing file to inspect");

	if ((fd = open(argv[1], O_RDONLY)) == -1)
		err(1, "open %s", argv[1]);

	zip = map_file(fd, &len);
	if ((cd = find_central_directory(zip, len)) == NULL)
		errx(1, "can't find central directory");

	ls(zip, len, cd);

	munmap(zip, len);
	close(fd);

	return 0;
}

I think it would be easier for us to just mmap(2) the file into memory rather than moving back and forward by means of lseek(2). map_file is a thin wrapper around mmap(2):

void *
map_file(int fd, size_t *len)
{
	off_t	 jump;
	void	*addr;

	if ((jump = lseek(fd, 0, SEEK_END)) == -1)
		err(1, "lseek");

	if (lseek(fd, 0, SEEK_SET) == -1)
		err(1, "lseek");

	if ((addr = mmap(NULL, jump, PROT_READ, MAP_PRIVATE, fd, 0))
	    == MAP_FAILED)
                err(1, "mmap");

	*len = jump;
	return addr;
}

Just as we were discussing before, to locate the central directory we must first locate the “end of central directory record”. Its structure is as follows (the numbers inside the brackets indicates the byte count)

signature[4] disk_number[2] disk_cd_number[2] disk_entries[2]
total_entrie[2] central_directory_size[4] cd_offset[4]
comment_len[2] comment…

The signature is always “\x50\x4b\x05\x06”, which helps in finding the record. We still need to be careful, since I haven’t seen anywhere that the signature MUST NOT appear inside the comment.

To be sure that we’ve actually found the real start of the end record, there’s a explicit check: the comment length plus the size of the non-variable part of the header must be equal to how far we have travelled from the end of the file. Granted, this is not completely bulletproof, since a specially-crafted comment may appear like a proper end of central directory record, but I’m not sure what could we do better to protect against faulty files.

Side note: as always, I’m treating these files as untrusted and do all the possible checks. You don’t want a malformed file to crash your program, don’t you?

One last thing: I’m totally fine with a very light and sparse usage of gotos. In find_central_directory I’m using a ‘goto again’ when we find a false signature inside a comment. A while loop would also do that, but it’d be a bit uglier.

void *
find_central_directory(uint8_t *addr, size_t len)
{
	uint32_t	 offset;
	uint16_t	 clen;
	uint8_t		*p, *end;

	/*
	 * At -22 bytes from the end there is the end of the central
	 * directory assuming an empty comment.  It's a sensible place
	 * from which start.
	 */
	if (len < 22)
		return NULL;
	end = addr + len;
	p = end - 22;

again:
	for (; p > addr; --p)
		if (memcmp(p, "\x50\x4b\x05\x06", 4) == 0)
			break;

	if (p == addr)
		return NULL;

	/* read comment length */
	memcpy(&clen, p + 20, sizeof(clen));
	clen = le16toh(clen);

	/* false signature inside a comment? */
	if (clen + 22 != end - p) {
		p--;
		goto again;
	}

	/* read the offset for the central directory */
	memcpy(&offset, p + 16, sizeof(offset));
	offset = le32toh(offset);

	if (addr + offset > p)
		return NULL;

	return addr + offset;
}

Edit 2021/08/20: there’s a space for a little optimisation: the end record MUST be in the last 64kb (plus some bytes), so for big files there’s no need to continue searching back until the start. Why 64kb? The comment length is a 16 bit integer, so the biggest end of record possible is 22 bytes plus 64kb of comment.

If everything went well, we’ve found the pointer to the start of the central directory. It’s made by a sequence of file header records:

signature[4] version[2] vers_needed[2] flags[2] compression[2]
mod_time[2] mod_date[2] crc32[4]
compressed_size[4] uncompressed_size[4]
filename_len[2] extra_field_len[2] file_comment_len[2]
disk_number[2] internal_attrs[2] offset[4]
filename… extra_field… file_comment…

The signature field is always "\x50\x4b\x01\x02", which is different from the end record and the other records fortunately. To list the files we just have to read the file headers record until we find one with a different signature:

void
ls(uint8_t *zip, size_t len, uint8_t *cd)
{
	uint32_t	 offset;
	uint16_t	 flen, xlen, clen;
	uint8_t		*end;
	char		 filename[PATH_MAX];

	end = zip + len;
	while (cd < end - 46 && memcmp(cd, "\x50\x4b\x01\x02", 4) == 0) {
		memcpy(&flen, cd + 28, sizeof(flen));
		memcpy(&xlen, cd + 28 + 2, sizeof(xlen));
		memcpy(&clen, cd + 28 + 2 + 2, sizeof(xlen));

		flen = le16toh(flen);
		xlen = le16toh(xlen);
		clen = le16toh(clen);

		memcpy(&offset, cd + 42, sizeof(offset));
		offset = le32toh(offset);

		memset(filename, 0, sizeof(filename));
		memcpy(filename, cd + 46, MIN(sizeof(filename)-1, flen));

		printf("%s [%d]\n", filename, offset);

                cd += 46 + flen + xlen + clen;
	}
}

As always, there are some magic numbers hardcoded, a real program would probably have some constants defined, but for this simple toy program I’m fine with things as is. Also, note the pedantry in ensuring we don’t end up reading out-of-bounds in the while condition, I don’t want faulty zip files to cause invalid memory access.

Now, to compile it and run:

% cc zipls.c -o zipls && ./zipls star_maker_olaf_stapledon.gpub
0_preface.gmi [0]
chapter_1_1_the_starting_point.gmi [2957]
chapter_1_2_earth_among_the_stars.gmi [6932]
chapter_2_1_interstellar_travel.gmi [11041]
chapter_3_1_on_the_other_earth.gmi [20382]
…

and voila, it works!

To conclude this entry, one of the things that I’m still not sure about is the endiannes of the numbers. I’m guessing they should be little endian, but it’s always that or only because the zip files were produced on a little endian machine?

Edit 2021/08/20: The majority of the number are stored in little-endian. There are some exception, so check the documentation, but is mostly for fields like the MSDOS-like time and date and stuff like that. The code was updated with the calls to leXYtoh() from ‘endian.h’.

Otherwise I’m pretty happy with the result. In a short time I went from knowing nothing about zips to being able to at least inspect them, using only the C standard library (well, assuming POSIX). I’ll leave the files decoding for a next time.

-- text: CC-BY-SA-4.0; code: public domain unless specified otherwise

For comments, write at < blog at omarpolo dot com > or @op@bsd.network in the fediverse.

Capsule proudly assembled with Clojure