💾 Archived View for perso.pw › blog › articles › mbuffer.gmi captured on 2024-05-10 at 11:32:37. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2024-03-21)
-=-=-=-=-=-=-
NILThis Port of the week is a bit special because sadly, the port isn't
available on OpenBSD. The port is **mbuffer** (which you can find in
misc/mbuffer).
I discovered it while looking for a way to enhance one of my network
stream scripts. I have some scripts that get a dump of a postgresql
base through SSH, copy it from stdin to a file with tee and send it
out to the local postgres, the command line looks like
$ ssh remote-base-server "pg_dump my_base | gzip -c -f -" | gunzip -f | tee dumps/my_base.dump | psql my_base
I also use the same kind of command to receive a ZFS snapshot from
another server.
But there is an issue, the end server is relatively slow, postgresql
and ZFS will eat lot of data from stdin and then it will stop for
sometimes writing on the disk, when they are ready to take new data,
it's slow to fill them. This is where **mbuffer** takes places. This
tool permit to add a buffer that will take data from stdin and fill
its memory (that you set on the command line), so when the slowest
part of the command is ready to take data, mbuffer will empty its
memory into the pipe, so the slowlest command isn't waiting to get
filled before working again.
The new command looks like that for a buffer of 300 Mb
ssh remote-base-server "pg_dump my_base | gzip -c -f -" | gunzip -f | tee dumps/my_base.dump | mbuffer -s 8192 -m 300M | psql my_base
mbuffer also comes with a nice console output, showing
- bandwith in
- bandwith out
- percentage/consumption of memory filled
- total transfered
in @ 1219 KiB/s, out @ 1219 KiB/s, 906 MiB total, buffer 0% full
In this example the server is too fast so there is no wait, the buffer
isn't used (_0% full_).
mbuffer can also listen on TCP, unix socket and have a lot of
parameters that I didn't try, if you think that can be useful for you,
just go for it !