💾 Archived View for perso.pw › blog › articles › mbuffer.gmi captured on 2022-06-03 at 23:19:15. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Port of the week: mbuffer

Author: Solène
Date: 31 May 2016
Tags: portoftheweek network

NIL=> Comment on Mastodon

This Port of the week is a bit special because sadly, the port isn't

available on OpenBSD. The port is **mbuffer** (which you can find in

misc/mbuffer).

I discovered it while looking for a way to enhance one of my network

stream scripts. I have some scripts that get a dump of a postgresql

base through SSH, copy it from stdin to a file with tee and send it

out to the local postgres, the command line looks like

$ ssh remote-base-server "pg_dump my_base | gzip -c -f -" | gunzip -f | tee dumps/my_base.dump | psql my_base

I also use the same kind of command to receive a ZFS snapshot from

another server.

But there is an issue, the end server is relatively slow, postgresql

and ZFS will eat lot of data from stdin and then it will stop for

sometimes writing on the disk, when they are ready to take new data,

it's slow to fill them. This is where **mbuffer** takes places. This

tool permit to add a buffer that will take data from stdin and fill

its memory (that you set on the command line), so when the slowest

part of the command is ready to take data, mbuffer will empty its

memory into the pipe, so the slowlest command isn't waiting to get

filled before working again.

The new command looks like that for a buffer of 300 Mb

mbuffer also comes with a nice console output, showing

- bandwith in

- bandwith out

- percentage/consumption of memory filled

- total transfered

in @ 1219 KiB/s, out @ 1219 KiB/s, 906 MiB total, buffer 0% full

In this example the server is too fast so there is no wait, the buffer

isn't used (_0% full_).

mbuffer can also listen on TCP, unix socket and have a lot of

parameters that I didn't try, if you think that can be useful for you,

just go for it !