💾 Archived View for soviet.circumlunar.space › dsp › 20210203.txt captured on 2021-12-03 at 14:04:38.
-=-=-=-=-=-=-
# Hello, world!
I intend to keep a short public journal, probably mostly related to my
programming. Today I managed to make some progress on enki. This is my
exploration project for cryptography, network programming, distributed systems,
anonymity, and peer-to-peer. I intend to duplicate functionality that is present
in Tor, Freenet, I2P, Tahoe-LAFS, and other such projects. It is written in
Clojure, as I am a LISP weenie. Some notes for today: I switched to using
Shenandoah, and eliminated the bottlenecks with my SQLite transaction thread.
End-to-end encryption and decryption functions were written and successfully
tested:
---
(defn end-to-end-test
[filename]
(let [data (-> filename fully-encrypt fully-decrypt)
data-hash (-> data hash/blake2b-512 codecs/bytes->hex)
original-hash (-> filename io/file hash/blake2b-512 codecs/bytes->hex)]
(= data-hash original-hash)))
(end-to-end-test "/home/dsp/testfile-100mb") => true
---
I also implemented passing through promises to the SQL transaction thread, so
that I can see when blocks have been successfully committed to disk. This was
required because I hit a bug where my code was executing faster than I could
commit the blocks to disk, so that subsequent calls to the fetch-block function
returned nil. Now there is an :executed? key which contains a promise that is
delivered to true when the given block has been inserted to the database. When
doing tests I can use the following on the block collection returned from
fec-encrypt-input-stream-with-executors:
(defn force-execution-promises
[coll]
(run! #(deref %)
(flatten (for [block coll]
(for [sub-block (:blocks @block)]
(:executed? sub-block))))))
This ensures that all blocks are available before proceeding.
There's a lot of cleanup to do, and I likely want to commit decrypted and
decoded FEC blocks to the database layer rather than keeping it in memory. Fine
for testing, but when I have 1MB blocks and 1GB of data with 100% recovery
overhead then I can see things getting out of hand pretty quickly.
I am now at the point where I could begin to flesh out node-to-node
communication and test a very basic version of data inserts.
I've been at the computer for too long. I should get out and about or at least
attempt some exercise, though I did not sleep at all last night.
- dsp