Tuesday, 30. May 2023
[This article has been bi-posted to Gemini and the Web]
This article is about creating jails right from the command line. This time we will also build an example jail that is somewhat useful. Let's assume that we have a NAS server at home and that it of course is build upon FreeBSD. A trustworthy OS and good ZFS support are reasons enough already - but hey, why not also make use of another great feature and set up a jailed private cloud with Seafile? We'll also look into removing jails along the way.
Part 0 of this series is a general discussion of what virtualization actually is and what you should know to make best use of what will be covered in the later parts. Part 1 covered an introduction of CBSD as well as the installation. Part 2 detailed CBSD's initial setup process. In part 3, jail creation with a dialog-based TUI as well as starting and stopping jails was covered. Part 4 covered creating jails via an interactive script. It used an unusual example (a 32-bit jail of an outdated version of FreeBSD) to show some important concepts like how filesystem mounts work.
(January 2023) Exploring the CBSD virtual environment management framework - part 2: Setup
(February 2023) Exploring the CBSD virtual environment management framework - part 3: Jails (I)
(April 2023) Exploring the CBSD virtual environment management framework - part 4: Jails (II)
Like last time the test system runs _FreeBSD 13.2_ but CBSD has been updated to _13.2.3_. The workdir is _/cbsd_ as always.
So far we've created jails using the friendly dialog menu and covered the quicker but less comprehensive option of answering questions of the interactive creation script. We're going to explore _two_ more options this time. But let's first re-visit the first one. Why? You'll see in a minute.
# cbsd jconstruct-tui
In the menu change 'jname' e.g. to 'seafile' and 'host_hostname' to 'seafile.local'. Then give it a free IP address of the subnet that it belongs to. In my example I picked one from the frequently used 192.168.0.0/16 range and set that for 'ip4_addr'. Finally, we're going into the 'Jail options' submenu and deselect 'mount_ports' because we won't need it and I like clean environments. (Or maybe because I was looking for something sensible to deviate a little more from the defaults? You decide.)
When it's all set, choose 'PROCEED'. CBSD will ask the familiar question: "Do you want to create jail immediately?" And this time we're going with 'no'. As hinted in part 3 of this article series, this will not discard our choices but write a configuration file instead of applying them right away:
You can make now: cbsd jcreate jconf=/cbsd/ftmp/seafile.5738.jconf
CBSD told us the name of the file, and that's a good thing. While the first part of it is not exactly hard to guess the numeric id is (it's random, I think). But what does such a config file look like? Let's peek inside and find out:
# head -n 12 /cbsd/ftmp/seafile.5738.jconf
# DO NOT EDIT THIS FILE. PLEASE USE INSTEAD:
# cbsd jconfig jname=seafile
relative_path="1";
jname="seafile";
path="/cbsd/jails/seafile";
data="/cbsd/jails-data/seafile-data";
rcconf="/cbsd/jails-rcconf/rc.conf_seafile";
# FQDN for environment
host_hostname="seafile.local";
# default environment IP
ip4_addr="192.168.13.127";
It has 88 lines in total of which some are empty to visually group various options together. The rest are variable definitions or comments. Among the latter, the first two lines are interesting and can in fact be misleading. What CBSD wants to warn you about is that _once you've created a jail_, you should not come back and edit the configuration file that it was created from! The reason for this is that those values are now saved in CBSD's DB and changing anything in said file _won't have any effect_. But until the jail is created, it's of course fine to edit the file and use it to create one.
Let's say we changed our mind and decided that "mycloud.local" was a better hostname. It's not like the hostname wasn't trivial to change later in the jail, but there are other things for which you might be tempted to fix beforehand. And going through the whole menu again (especially in case you changed a lot of settings) is not terribly attractive. So we're going to ignore CBSDs warning (we know what it actually meant, right?) and edit the file:
# sed -i '' -e 's/seafile\./mycloud\./' /cbsd/ftmp/seafile.5738.jconf
Checking the result real quick won't hurt:
# grep hostname /cbsd/ftmp/seafile.5738.jconf
host_hostname="mycloud.local";
That's better. So now we can do as CBSD told us:
# cbsd jcreate jconf=/cbsd/ftmp/seafile.5738.jconf
If this is the first jail of the specified base version to be created on this system, CBSD will now populate the respective base system. We've seen this twice and therefore skip it here. But here's the rest of the output:
Please wait: this will take a while...
Applying skel dir template from: /cbsd/share/FreeBSD-jail-skel
To edit VM properties use: cbsd jconfig jname=seafile
To start VM use: cbsd jstart seafile
To stop VM use: cbsd jstop seafile
To remove VM use: cbsd jremove seafile
For attach VM console use: cbsd jlogin seafile
Creating seafile complete: Enjoy!
jcreate done in 30 seconds
Ok, the jail has been created. Now for a quick inspection:
# cbsd jstart seafile
[ ... ]
# cbsd jlogin seafile
[ ... ]
# hostname
mycloud.local
Everything worked well, but we want to try out yet another means of creating jails, so let's get rid of this one and do it again in a different manner:
# logout
# cbsd jstop seafile
Stoping jail: seafile, parallel timeout=5
jstop done in 2 seconds
Jail is down, now we can delete it:
# cbsd jremove seafile
jsnapshot: destroyed 0 snapshots in 0 seconds
jsnapshot: destroyed 0 snapshots in 0 seconds
jdestroy done in 2 seconds
ATTENTION: The command to remove jails is dangerous for obvious reasons - and it's happily doing its thing as soon as you tell it to! It _will not_ ask for confirmation before throwing away your data. My suggestion is to not get into a habit of deleting jails directly by passing the name to the subcommand, especially not in cases when you plan to re-use the name for another jail soon. One little mistake when using the shell history and you've lost something that you didn't want to!
Just use the general 'cbsd jremove' instead and use the menu that it presents to select which jail to retire. This has the additional benefit of CBSD telling you for how long it's down. So if you see that the jail has been inactive for several days it's obviously not the one that you just stopped. Also it's an additional step giving your brain some time to decide whether you really are operating on the correct one (which is not a bad thing when removing stuff as every admin knows).
# cbsd jls
JNAME JID IP4_ADDR HOST_HOSTNAME PATH STATUS
The jail is gone, so we're good to re-create it. We could just use CBSD's _jcreate_ command again, pointing to the same jconf file, right? Nope, we can't. CBSD is tidy by default: It removed the file after creating the jail from it:
# ls /cbsd/ftmp/
Nothing there, it's gone. So the takeaway here is: If you plan on using a jconf file more than once, either copy it off to some other location or pass _removeconf=0_ to the _jcreate_ command to preserve the file.
Looks like we have to start over - but that was the point anyway. We can achieve exactly the same thing as before by using the _jcreate_ command and passing everything we want to customize as arguments on the command line. Once you get used to what variable names CBSD uses for what, this is by far the quickest way of doing things (especially if you don't change a lot of things from their defaults).
So what did we change above? Here's the respective lines as they appeared in the now gone jconf file:
jname="seafile";
host_hostname="mycloud.local";
ip4_addr="192.168.13.127/24";
mount_ports="0";
Alright, let's re-create our jail all in one command:
# cbsd jcreate jname="seafile" host_hostname="mycloud.local" ip4_addr="192.168.13.127/24" mount_ports="0"
Please wait: this will take a while...
Applying skel dir template from: /cbsd/share/FreeBSD-jail-skel
To edit VM properties use: cbsd jconfig jname=seafile
To start VM use: cbsd jstart seafile
To stop VM use: cbsd jstop seafile
To remove VM use: cbsd jremove seafile
For attach VM console use: cbsd jlogin seafile
Creating seafile complete: Enjoy!
jcreate done in 22 seconds
Here we go! Now we can start the new jail and log in again:
# cbsd jstart seafile
[ ... ]
# cbsd jlogin seafile
[ ... ]
One little hint as pointed out by Oleg when he proofread the article: When creating jails this way, you can use the very handy _sysrc=_ to pass values to CBSD that will be executed against the jail. For example like this:
# cbsd jcreate jname="seafile" host_hostname="mycloud.local" ip4_addr="192.168.13.127/24" sysrc="sshd_enable=YES sendmail_enable=NONE"
(Yep, I'm aware that _sendmail_enable=NO_ has been sufficient for a long time, but habits die hard!)
Alright! we came here to create a jailed private cloud, so let's do just that. First thing is to install the thing (if you can't reach the Internet, you may for example need to change /etc/resolv.conf):
# pkg install seahub
The package comes with a convenient setup script. Since in our case Seafile is an example application in a jails tutorial, we're going for a very basic configuration. The following script will set things up for use with SQLite. If you want to actually use Seafile in production, you may consider using MySQL instead and will definitely want to take a closer look at the configuration. But here we go:
# /usr/local/www/haiwen/seafile-server/setup-seafile.sh
[...]
[server name]: seatest
[...]
[This server's ip or domain]: 192.168.13.127
What tcp port do you want to use for seafile fileserver?
8082 is the recommended port.
[default: 8082 ]
This is your config information:
server name: seatest
server ip/domain: 192.168.13.127
seafile data dir: /usr/local/www/haiwen/seafile-data
fileserver port: 8082
If you are OK with the configuration, press [ENTER] to continue.
Generating ccnet configuration in /usr/local/www/haiwen/ccnet...
Generating seafile configuration in /usr/local/www/haiwen/seafile-data ...
Ok, looks good so far. The seafile server has been set up. The script continues for seahub, the other critical component:
-----------------------------------------------------------------
Seahub is the web interface for seafile server.
Now let's setup seahub configuration. Press [ENTER] to continue
-----------------------------------------------------------------
Creating database now, it may take one minute, please wait...
/root
Done.
creating seafile-server-latest symbolic link ... done
-----------------------------------------------------------------
Your seafile server configuration has been completed successfully.
-----------------------------------------------------------------
[...]
There's some more output, but this is what you may want to read to get an idea of what the script did. Let's fix a permission problem real quick before it bites us later:
# mkdir /tmp/seahub_cache
# chown seafile /tmp/seahub_cache
Now we're going to run the commands suggested by the script output that I cut above and get Seafile up and running:
# sysrc seafile_enable=YES
seafile_enable: -> YES
# sysrc seahub_enable=YES
seahub_enable: -> YES
# service seafile start
Starting seafile server, please wait ...
** Message: 13:52:58.853: seafile-controller.c(678): No seafevents.
Seafile server started
# service seahub start
LC_ALL is not set in ENV, set to en_US.UTF-8
Starting seahub at port 8000 ...
Seahub is started
Done. Now all that's left is creating an admin user like this:
# /usr/local/www/haiwen/seafile-server/reset-admin.sh
E-mail address: kraileth@example.com
Password:
Password (again):
Superuser created successfully.
Let's see what network server processes are running in the jail now:
# sockstat -4l
USER COMMAND PID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS
seafile python3.9 46662 8 tcp4 192.168.13.127:8000 *:*
seafile python3.9 46661 8 tcp4 192.168.13.127:8000 *:*
[...]
seafile seaf-serve 46607 11 tcp4 192.168.13.127:8082 *:*
There's the the Seafile fileserver listening on port 8082 and there are multiple instances of the seahub Django-based web application (Django is written in Python). Looks good so far.
Time to give it a shot. Pointing my browser at 192.168.13.127:8000, I get a login page.
That's a good start. Since we created a superuser just a moment ago, logging in should just work.
First thing to do here is clicking on the avatar and then going to the 'System Admin' page where we need to make two small adjustments.
Once the SERVICE_URL and FILE_SERVER_ROOT settings are correct and saved, uploading files will work. If they are incorrect you will get the 'Network Error' message loved by anybody playing with Seafile.
As you can see in the picture above, I've successfully uploaded a file and thus proved that the little cloud we just built works. Yes, that's an OmniOSce image I uploaded. Why? Because I like to play with other Unices now and then and that was the next best thing I had lying around. 🙃
I know I said that we'd only do a very basic configuration, but still. This is a little _too_ raw to end the article here. Let's at least add a reverse proxy to our setup here. Nginx is a popular option and for good reasons. First step is obviously to install it:
# pkg install nginx
Then we need to write a config file. Open it, delete everything and put the following in instead:
# vi /usr/local/etc/nginx/nginx.conf
user www;
worker_processes 1;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
log_format seafileformat '$http_x_forwarded_for $remote_addr [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $upstream_response_time';
server {
listen 80;
server_name 192.168.13.127;
proxy_set_header X-Forwarded-For $remote_addr;
location / {
proxy_pass http://192.168.13.127:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
proxy_read_timeout 1200s;
client_max_body_size 0;
access_log /var/log/nginx/seahub.access.log seafileformat;
error_log /var/log/nginx/seahub.error.log;
}
location /seafhttp {
rewrite ^/seafhttp(.*)$ $1 break;
proxy_pass http://192.168.13.127:8082;
client_max_body_size 0;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 36000s;
proxy_read_timeout 36000s;
proxy_send_timeout 36000s;
send_timeout 36000s;
access_log /var/log/nginx/seafhttp.access.log seafileformat;
error_log /var/log/nginx/seafhttp.error.log;
}
location /media {
root /usr/local/www/haiwen/seafile-server-latest/seahub;
}
}
}
This is the configuration recommended by Seafile when I looked it up merged with some defaults of Nginx on FreeBSD. I only adjusted the path for /media to the where it's located on our OS of choice and filled in the example IP that I've been using here. As you can see, we're reverse proxying _two_ locations here: One to port 8000 and the other to port 8082. Since the latter will be available in a subdirectory, we need to change the Seafile configuration accordingly for it to work. But one thing at a time.
So the next step is enabling Nginx and starting the server:
# sysrc nginx_enable=YES
nginx_enable: -> YES
# service nginx start
Performing sanity check on nginx configuration:
nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
Starting nginx.
Then we need to change the settings again like in the following picture:
Now we should be able to upload files again:
Testing one more file upload (PNG)
And indeed it works just fine. But why did we go that extra mile in the first place?
For one thing it's much nicer to just browse to an IP and use the implicit default port (e.g. 80 for HTTP) than going with non-standard ports. But there's a much more important reason! We're running Seafile in a jail - and while that is meant to improve security by locking our applications in, it can very well do the opposite and increase risk. Why? Because standard jails _do not have a localhost_ IP address! If you wonder why, think about what a jail is. While VNET jails have their own network stack, normal jails don't. They get an additional IP assigned which technically runs on the host system's network stack. And since said host owns 127.0.0.1, jails of course cannot have it!
This is an interesting limitation and a whole lot of *nix applications make the (false) assumption that "of course" there always is 127.0.0.1. FreeBSD's jails are the exception here. So by binding on the actual IP in our example this opens up services that _are meant to be private_. In our case we at least choose an address from a non-routable IP range, but maybe you want to run Seafile on your VPS and have to use a public IP! I don't know about you, but I'd never run a Django server in a way that makes it publicly accessible (and I don't know enough about the Seafile file server to deviate from my default stance). So putting a known battle-tested webserver in front of it is the very least you should do.
It's just a start, though. Not only should you carefully review the configuration of all applications involved, but you _absolutely_ should also get a firewall in place. Definitely block external access to ports 8000 and 8082; any traffic should only ever go through the reverse proxy. Having Nginx at the gate also allows you to configure it to serve TLS only. This is slightly beyond the scope of this article but I _highly_ recommend doing it. What else? Maybe you want to use DNS instead of an IP address. Or probably put something like basic auth in front of your Seafile if you run it on the open internet and want to continue enjoying good sleep. These are a couple of things that you may want to consider if you plan on giving Seafile a try. If you think about it you'll surely come up with more.
There's one more way to create jails - but we've seen quite some already and I feel that we really should cover something else before that. So the next article will cover re-configuring and updating existing jails.
(February 2024) Exploring the CBSD virtual environment management framework - part 6: Jails (IV)
