💾 Archived View for thebird.nl › gn-gemtext-threads › topics › file-sharing › replace-ipfs.gmi captured on 2023-03-20 at 17:57:57. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Replace IPFS

IPFS has some nice features exposing hashed files through HTTP. The overall system, however is overengineered and puts load on the server. Also it is hard to deal with private data.

Our requirements:

Expose public files over the web

https://files.genenetwork.org/

Serve them under a hash, so we can guarantee/encourage reproducibility
Deduplication on the disk drive
Make versioned hashes easy to find (IPFS does not expose these)

Create a hash of the directory

find current/ -type f -exec md5sum {} \;|awk '{ print $1 }'|md5sum

Then move the files into a dir of that name and expose through, say nginx.

Deduplication

This can be handled later. One option is to symlink identical files. Another is to use a deduplicating file system or even borg with fuse mounts. git-mount and gitfs may do the job too, though git is not particularly great at handling large files.

Info

Public files are hosted on tux02

https://files.genenetwork.org/

I have added a git repo that is now 11Gb. Looks like it can perform alright, as long as we expose textual files. A cgit frontend can deal with versions.

https://git.zx2c4.com/cgit/