💾 Archived View for gemini.circumlunar.space › users › kraileth › neunix › eerie › 2016 › bacula_on_… captured on 2022-06-11 at 21:38:54. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-12-05)
-=-=-=-=-=-=-
Here I'm republishing an old blog post of mine originally from October 2016. The article has been slightly improved.
This is the third part of my Bacula tutorial. The first part covered some basics as well as installing Bacula and starting the three daemons that it consists of.
Part two dealt with modifying the default configuration files in a way that allowed all components of Bacula to interact with each other locally. A deliberate configuration error was debugged and finally a test backup was done (without knowing details like what exactly would even be backed up!) just to ensure that communication between the daemons really works.
Bacula on FreeBSD (pt. 1): Introduction - Bacula backup basics
Bacula on FreeBSD (pt. 2): Bconsole - ruling the Director
In this post we'll clean up and split the configuration files to make them more comprehensible. We'll create our own fileset, add a new device and assign that to a new storage resource. We'll also customize a backup job on the bconsole to do a new backup using our self-created resources.
The third part continues where the first one left off. During this tutorial series we use _VirtualBox VMs_ managed by _Vagrant_ to make things as simple as possible. If you don't know how to use them, have a look at two articles I wrote for a detailed introduction into using Vagrant and on how to prepare a VM template with FreeBSD that is used throughout all parts of this Bacula tutorial.
Vagrant: Creating a FreeBSD 11 base box (virtualbox) - pt. 1
Vagrant: Creating a FreeBSD 11 base box (virtualbox) - pt. 2
Let's jump right into it! So restore the snapshot from the previous post, SSH into the machine, become root and change into Bacula's configuration directory:
% cd ~/vagrant/backuphost
% vagrant snapshot restore tut_2_end
% vagrant ssh
% sudo su -
# cd /usr/local/etc/bacula/
Ok, there we are. Before we can do anything actually _somewhat_ useful, we need to sort out things and get an idea of how the configuration works (what all those resources do etc). Bacula's default configuration is rather huge which doesn't exactly make this an easy task. But there's an easy solution to it: Let's start by deleting each and every line that's commented out in all of the configuration files!
Yes, some of these lines provide useful information but that's not really lost:
# ls -1
bacula-barcodes
bacula-barcodes.sample
bacula-dir.conf
bacula-dir.conf.sample
bacula-fd.conf
bacula-fd.conf.sample
bacula-sd.conf
bacula-sd.conf.sample
bconsole.conf
bconsole.conf.sample
As you can see, there are files like _bacula-dir.conf.sample_, _bacula-fd.conf.sample_, and so on. These are the unaltered default configuration files and you can always look at those as a reference. So before we move on: Edit the four config files that we’ve worked with so far (i.e. _bacula-dir.conf_, _bacula-fd.conf_, _bacula-sd.conf_ and _bconsole.conf_) and get rid of all the commented out lines.
Removing these lines (and a few empty ones that were between them and didn't have any purpose on their own) shrinks the configuration down from over 600 lines to less than 300. While that's much better, it's still a lot. Let's throw more stuff away, shall we? There's a "restricted director" that's used for monitoring. We're not going to use that here so all those resources with the name of _fbsd-template.local-mon_ can be deleted from all three daemon's config files. Do that now.
At this point restart all three daemons (and check if they are running afterwards) to make sure the configuration is still valid:
# service bacula-dir restart
# service bacula-fd restart
# service bacula-sd restart
The files _bconsole.conf_ and _bacula-fd.conf_ are trivial now with less than 20 lines each. Still _bacula-sd.conf_ with about 80 lines is barely acceptable and especially _bacula-dir.conf_ with over 170 remains too big for my taste!
Of course we cannot simply go rampage and delete more and more resources from the director configuration. So how to make it smaller (while maintaining the functionality)? Bacula supports breaking up larger configuration files. To do so we put some resources into their own file and include that in the main config file! Bacula doesn't care in which files you stuff its resources and it also doesn't really care about the order. But we do that to make things easier to maintain and so it makes sense to think about what resources should be grouped together.
To keep things organized, it's best to create a separate directory for includes (or even multiple directories if you have complex configuration and need to separate things further). Choose a name that fits your taste:
# mkdir includes
Here's my suggestion (you might want to do this completely differently - just do what makes sense for _you_) for splitting up _bacula-dir.conf_:
Now let's include those configuration files that we split off! We could do so by including them one by one by adding the following lines at the end of _bacula-dir.conf_ like this:
@/usr/local/etc/bacula/includes/dir_jobs.conf
@/usr/local/etc/bacula/includes/dir_filesets.conf
@/usr/local/etc/bacula/includes/dir_schedule.conf
@/usr/local/etc/bacula/includes/dir_storage.conf
@/usr/local/etc/bacula/includes/dir_pool.conf
That would work (just make sure you use _absolute paths_ to save you some potential headaches). But don't put it in there (or remove it again if you already have)! There's a better way of doing it; just add the one following line instead:
@|"sh -c 'for f in /usr/local/etc/bacula/includes/dir_*.conf ; do echo @${f} ; done'"
This will use the shell to automatically include all files in _/usr/local/etc/bacula/includes/_ that start with dir_ and end in .conf. Much better, eh? And it has the advantage of automatically pulling in _includes/dir_messages.conf_ or something like that should you ever decide to split e.g. the _Messages_ resources from the main dir config file as well.
Now we have a bunch of files with nicely separated resource types and the biggest ones are around 40 lines long. Much more comprehensible if you ask me. Let's quickly split up the sd config as well by just putting all the Device resources into _include/sd_device.conf_ and including it adding the following line to _bacula-sd.conf_:
@|"sh -c 'for f in /usr/local/etc/bacula/includes/sd_*.conf ; do echo @${f} ; done'"
Restart dir and sd again to see if the configuration is still ok:
# service bacula-dir restart
# service bacula-sd restart
We've given our base box a hostname of "fbsd-template.local". That's fine for a template - but this machine is not a template anymore. We'll spin up more VMs from the same base box (template) later so it makes sense to do some proper naming. We could just change the hostname and Bacula would not care at all here since a resource's name is just a name. But for human operators things can get pretty confusing pretty fast - and that's what we wanted to avoid, right? So let's edit the four config files now and change the Name of the various resources from e.g. _fbsd-template.local-dir_ to _backuphost.local-dir_. Once you've replaced it nine times in the actual config files, you're done here.
But we must not forget to also change it twice in _includes/dir_job.conf_ or the director won's start since the jobs refer to the old name! And while we're at it, let's also change the first Job resource's name in the same file from _BackupClient1_ to _Backuphost.local_. Then save the file and exit the editor.
Now let's change the hostname that FreeBSD will assign itself during startup. To do so we need to edit rc.conf:
# vi /etc/rc.conf
Change the corresponding line so that it holds the new hostname:
hostname="backuphost.local"
Now is an excellent time to take another snapshot (so you can restore that if something goes wrong and don't have to redo all the config splitting, etc.) and since we've changed the hostname we need to reboot anyways. So let's shutdown the VM, snapshot and start the VM again so that it comes back up with the new hostname:
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_3_mid
% vagrant up
Let's SSH back into it, switch to root and see if the hostname changed:
% vagrant ssh
% sudo su -
# hostname
backuphost.local
That looks good. Also all Bacula services should be running (feel free to double-check) so we can move on.
Let's talk about _FileSets_ next. We've put definitions of this kind of resources into their own configuration file that's then included into the dir configuration. Now we want to add a new fileset. We do so by editing:
# vi /usr/local/etc/bacula/includes/dir_fileset.conf
Just add the following lines to it:
FileSet {
Name = "etc"
Include {
Options {
signature = MD5
}
File = /etc
File = /usr/local/etc
}
Exclude {
File = /etc/casper
}
}
What does this do? It defines another fileset named "etc" which includes the directoris /etc and /usr/local/etc. By default Bacula will recursively include files and subdirectories when it is told to back up a directory. So this fileset will effectively backup almost all of the system's configuration files (which reside in /etc and /usr/local/etc).
Let's assume we don't use _Capsicum_ (FreeBSD's sandboxing framework in case you've never heard of it) and don't want to back up configuration for the Casper daemon (which is a support daemon for Capsicum). The above example excludes _/etc/casper_ from being backed up.
And then there's the _Signature_ directive. It tells Bacula to create a signature for each file it backs up using the MD5 algorithm. While it is not necessary it is strongly recommended to let Bacula calculate signatures (checksums). Bacula uses these to find out if a file has been altered since it was last backed up even if e.g. the file size didn't change. Even though it takes a little more space if you use signatures, doing so totally makes sense.
Save the file and exit. You don't have to restart the director, yet, because we're going to make one more change.
Let's say we want to store our backups in _/var/backup_. To do this, we first create the directory and make sure Bacula can access it:
# mkdir /var/backup
# chown bacula:bacula /var/backup
Now we have to inform the sd of the new "device". Edit the configuration:
# vi /usr/local/etc/bacula/includes/sd_device.conf
and add the following lines:
Device {
Name = Var-File
Media Type = File3
Archive Device = /var/backup
LabelMedia = yes;
Random Access = Yes;
AutomaticMount = yes;
RemovableMedia = no;
AlwaysOpen = no;
Maximum Concurrent Jobs = 5
}
Here we create a new "device" for Bacula to use - this "device" is in our case just a directory (_/var/backup_) on the disk. It is important that you use unique _media types_; since there's already _File1_ and _File2_ in the default configuration, we go with _File3_ here. In fact you could choose any unique name that makes some sense to you. Ignore the other settings (or look them up in the Bacula documentation if you want to know more). Save the file and exit. However you won't be able to use it, yet, as the dir does not know anything about it. So we need to edit its configuration as well:
# vi /usr/local/etc/bacula/includes/dir_storage.conf
and add the following lines:
Storage {
Name = File3
Address = localhost
SDPort = 9103
Password = "sdPASSWORD"
Device = Var-File
Media Type = File3
Maximum Concurrent Jobs = 10
}
This creates a new _storage_ that can hold backups. It is defined for the director who will send it to the fd if it initiates a backup. Save and exit. Now we need to restart dir and sd to load the new configuration:
# service bacula-dir restart
# service bacula-sd restart
We've made a lot of changes to the configuration so let's attempt to do another backup:
# bconsole
* run
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
A job name must be specified.
The defined Job resources are:
1: Backuphost.local
2: BackupCatalog
3: RestoreFiles
Select Job resource (1-3):
Select the first one:
Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: Full Set
Pool: File (From Job resource)
Storage: File1 (From Job resource)
When: 2016-09-23 18:36:24
Priority: 10
OK to run? (yes/mod/no):
Choose _mod_ this time:
Parameters to modify:
1: Level
2: Storage
3: Job
4: FileSet
5: Client
6: When
7: Priority
8: Pool
9: Plugin Options
Select parameter to modify (1-9):
Select 4:
The defined FileSet resources are:
1: Full Set
2: Catalog
3: etc
Select FileSet resource (1-3):
Of course we want to use 3, then _mod_ again and then 2:
The defined Storage resources are:
1: File1
2: File2
3: File3
Select Storage resource (1-3):
Choose 3, then _mod_ once more and then 8:
The defined Pool resources are:
1: Default
2: File
3: Scratch
Select Pool resource (1-3):
Go with the first one:
Run Backup job
JobName: Backuphost.local
Level: Incremental
Client: backuphost.local-fd
FileSet: etc
Pool: Default (From User input)
Storage: File3 (From user selection)
When: 2016-09-23 18:38:24
Priority: 10
OK to run? (yes/mod/no):
Now type in _yes_:
Job queued. JobId=2
You have messages.
Let's see how that went:
* mes
[...]
23-Sep 18:41 backuphost.local-dir JobId 2: Using Device "Var-File" to write.
23-Sep 18:41 backuphost.local-sd JobId 2: Job Backuphost.local.2016-09-23_18.41.29_03 is waiting. Cannot find any appendable volumes.
Please use the "label" command to create a new Volume for:
Storage: "Var-File" (/var/backup)
Pool: Default
Media type: File3
Oh my! What's happening now again? We need to create a _label_ first before Bacula can use the new storage. Let's just do that now and talk about the details later (there's a bit more to storage actually):
* label
The defined Storage resources are:
1: File1
2: File2
3: File3
Select Storage resource (1-3):
Select target 3 of course. Bacula will then ask you to give that volume a name:
Enter new Volume name:
Let's just call it _file3a_:
Defined Pools:
1: Default
2: File
3: Scratch
Select the Pool (1-3):
Select 1:
Connecting to Storage daemon File3 at localhost:9103 ...
Sending label command for Volume "file3a" Slot 0 ...
3000 OK label. VolBytes=212 VolABytes=0 VolType=1 Volume="file3a" Device="Var-File" (/var/backup)
Catalog record for Volume "file3a", Slot 0 successfully created.
Requesting to mount Var-File ...
3001 OK mount requested. Device="Var-File" (/var/backup)
Seems like everything is ok now. Wait a few seconds and see if the job that has been waiting in line has been run automatically:
* mes
[ ... quite some output ... ]
Again look especially for the most important line. If you can find this one:
Termination: Backup OK
everything has worked out.
Let's save our progress by shutting down the VM and taking the next snapshot:
* exit
# shutdown -p now
% vagrant status
% vagrant snapshot save tut_3_end
Alright! We just did our second backup with Bacula and this time we actually even know what we did there. While it's not perfect to modify the backup job like this we're certainly making progress (and it makes sense to have done it like that at least once). We'll take care of the job defaults, discuss labels, volumes, etc. next time and also do a restore. But for now take a break. You've earned it. Then move on to the next post.
Bacula on FreeBSD (pt. 4): Jobs, volumes, pools & a restore
Got any comments or figured that there's something wrong with what I wrote? Just drop me a line.