💾 Archived View for dcreager.net › 2023 › 06 › 29-tef-pipelines.gmi captured on 2023-11-14 at 08:06:33. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

Another tef gem on pipelines

2023-06-29

tef has written another great post about pipelines, and in particular why (and how!) not to use message queues to implement them:

how (not) to write a pipeline [tef, cohost]

The tl;dr is that your “background job” is really a state machine. Even the simplest possible job:

def do_something(inputs) -> outputs:
    # do something
    pass

Is a state machine with 4 possible states:

enqueued
started
succeeded
failed

And a simple pipeline based on a message broker almost certainly does not support:

navigating that state machine correctly, including error detection
back pressure
easy-to-use smoke detectors and fire extinguishers for operating the service

So bite the bullet and implement this using a proper database table (or equivalent) to track the state of each job!

You might still end up with a queue, but as an optimization, not as a load-bearing part of the design:

the queue buffers the results of a more expensive database query

Another tef gem on pipelines

Tags