I was documenting the tale of setting up a rendering node using CUDA on a headless Docker host.
In the last episode[1] we got our Docker/Kubernetes host working with the NVIDIA drivers and Container runtime so we can deploy GPU accelerated workloads.
Now, we need the workload - a Blender rendering node.
...and use the network rendering feature right?
Alas, wrong. Because Blender recently removed the simple network rendering it had built in, and the Network Render extension with it.
So now, you need to get a third-party service, CrowdRender[2], working.
Now, I don't want to say anything bad about CrowdRender - when it works, it works very well. Indeed, I'm impressed enough I've donated to their crowdfunding appeal - but, getting it to work can be tricksy (it does warn you that it is Alpha software, after all.) So here are a few tips - they took longer to work out than to write, so hopefully they will help someone.
First, the good news - there is already a Docker image for running Blender on your compute note. You'll find it on DockerHub as zocker160/blender-crowdrender[3].
Read the instructions on this carefully. Since in the last episode we got CUDA working properly on our host, we can use the `:nvidia` tag to get the GPU accelerated version of the image. This is an alias for `bl_2.82_cu_10`[^1], but read the Version Table on that docker README carefully, because:
*That means you must use Blender 2.82a as your desktop/client*
It turns out that Blender and Crowd-Render are *really really picky* about which version will work with which.
You also need to be careful that *the version of Crowd-Render you run on the server also matches the version of the plugin you install in your client*. So, once you've accepted that you know you will be using the correct version of Blender, you also need to decide which version of the extension (or 'Add On' in Blender terminology) you are going to use, based on the support matrix shown in the Docker README and also in the Crowd Render download page. Use whichever appears to be more recent...
So, to illustrate how my finally working set of versions looks:
┌──────────────────────────┬────────────────────────┬──────────────────────────┐ │ App │ Version │ Notes │ ╞══════════════════════════╪════════════════════════╪══════════════════════════╡ │ Blender (on the client) │ 2.82a │ │ ├──────────────────────────┼────────────────────────┼──────────────────────────┤ │ │ │ This is Blender 2.82, │ │ │ │ with CUDA10 support. I │ │ zocker160/blender- │ │ prefer to specify this, │ │ crowdrender │ `:bl_2.82_cu_10` │ not `:nvidia`, to avoid │ │ │ │ surprises when the next │ │ │ │ version of the Docker │ │ │ │ image is released │ ├──────────────────────────┼────────────────────────┼──────────────────────────┤ │ │ │ This was the most │ │ Crowd-Render Add-On (on │ 0.2.9 for Blender 2.8x │ current version for free │ │ the client) │ │ download that supports │ │ │ │ Blender 2.8 │ ├──────────────────────────┼────────────────────────┼──────────────────────────┤ │ │ │ This is the name of the │ │ CR_VERSION │ `cr_029_bl280` │ install file for the CR │ │ │ │ plugin you downloaded │ └──────────────────────────┴────────────────────────┴──────────────────────────┘
That last CR_VERSION is important, because you are going to pass it to the `docker run` command to ensure that the version of CrowdRender downloaded by the Docker image matches the version you installed on your desktop Blender client machine.
If you get any of this wrong, do not expect friendly "oops, you are using
the wrong version" errors. The failures are likely to be silent and
mysterious - so choose wisely and type carefully.
So, you have the versions straight in your head? Good, then what you need to do is quite simple:
1. Download and install the correct version of Blender on your desktop. The Blender website will try really hard to make you download the latest version - *don't let it trick you!* If the website drives you crazy, try this direct link[4].
2. Download and install the correct version of the Add On for your desktop client from the Crowd-Render Website[5].
Now you should be ready to install the server component. The following Docker command will do what you need; note that you are specifying the correct version of CR to download, to avoid any surprises.
docker run -d --name my-crowdrender-host \ --net=host \ --restart=unless-stopped \ -e token=***THE TOKEN YOU WILL GET FROM THE CROWDRENDER SITE*** -e CR_VERSION=cr_029_bl280 \ --gpus all \ zocker160/blender-crowdrender:bl_2.82_cu_10
Now, in theory, you can follow the instructions on the CrowdRender website, to enable the CR plugin in your client and then connect to your new rendering node.
*Note*: Your Docker container will need to be able to connect
to the Crowd-Render website, to download the plugin image. (That token you give
to the Docker image is basically a login token that lets it download
the plugin from the same place you downloaded it for the desktop.)
This should only be necessary the first time the container is initialised
though, when the plugin is downloaded and installed.
What if it just silently fails giving you a black screen without any error?
First, make sure in your Blender client's rendering settings that you chose a Render Engine that supports headless rendering, like Cycles[6].
If that doesn't solve your problem, firstly, you should take the normal step of checking the docker container logs using `docker logs --follow my-crowdrender-host`[^2] - hopefully, you will see an error or similar there that will point you in the direction of a solution. If nothing obvious appears though, you may need to dig deeper into Blender's logs stored inside the container.
To do this, create a shell on the container (`docker exec -it my-crowdrender-host sh`), and hunt for logfiles - you want to start your search in the `/root/cr/logging` directory on the container.
Some tips from personal experience that might help:
Blender runs on the server with --noaudio. So why would you need audio drivers? Search me, it's a mystery. And yet... Blender will still fail with a `cannot open /dev/dsp` error. If you find a message like this buried within the logs, I solved it with:
apt-get install osspd-alsa
...on the Docker *host* machine. Then restart the Docker service with `sudo service docker restart` (or equivalent.) You still won't have a `/dev/dsp` device, but apparently there will be enough of a hint there for Blender to decide it didn't need one anyway.
No, I have no idea why. All I can say is - it worked for me. Apparently, this bug only seems to affect Ubuntu hosts (and, it should be noted - this is a bug in Blender rather than Crowd-Render.)
The Eevee rendering engine - which is unfortunately the default for a clean install of Blender - doesn't support headless rendering (it depends on accessing the GPU through OpenGL, and needs an X Server to do it) - see the Eevee documentation[7] for (not much) further detail.
--------------------------------------------------------------------------------
[^1]: Two days after I wrote this, Zocker[9] updated his images to support version 2.83 of Blender :-). Once I've had a chance to upgrade myself and check these instructions still work, I'll update this article - but until then, note that you could go with that version, as long as you update all the relevant references (your own Blender version, and the CR_VERSION.)
[^2]: Or the name you gave the container, if you chose something different in the `docker run` command.
1: /posts/2020-09-08-setting-up-a-blender-rendering-node-using-docker.gmi
2: https://www.crowd-render.com
3: https://hub.docker.com/r/zocker160/blender-crowdrender
4: https://download.blender.org/release/
5: https://www.crowd-render.com/download
6: https://docs.blender.org/manual/en/dev/render/cycles/introduction.html
7: https://docs.blender.org/manual/en/dev/render/eevee/limitations.html#headless-rendering
8: https://hub.docker.com/u/zocker160
9: https://hub.docker.com/u/zocker160
--------------------