Thursday, April 14, 2016

FreeBSD: Jails (4) - Thin jails using NullFS and template

As stated on chapter 14.5 of FreeBSD handbook (Updating Multiple Jails):
"The management of multiple jails can become problematic because every jail has to be rebuilt from scratch whenever it is upgraded. This can be time consuming and tedious if a lot of jails are created and manually updated.
This section demonstrates one method to resolve this issue by safely sharing as much as is possible between jails using read-only mount_nullfs(8) mounts, so that updating is simpler. (...) Additionally, it provides a simple way to add, remove, and upgrade jails."
So the idea now is to try this method of creating a template that can be reused to deploy as many jails as necessary and also make the update and/or upgrade of existing jails easier.
There are different steps for UFS and ZFS file systems. Due to the fact that my FreeBSD virtual machine is running with UFS, the steps below will be focused on this particular file system.

Prepare the file system layout. My personal file system layout is as follows:

- The jails are based under the /usr directory;
- Each jail will be mounted under the /usr/jails directory;
- The template for each jail and the read-only partition for all of the jails is /usr/jails/thin/templates/base-10.1-RELEASE;
- A blank directory will be created for each jail under the /usr/jails directory;
- Each jail will have a /skeleton directory that will be linked to the read-write portion of the system;
- Each jail will have its own read-write system that is based upon /usr/jails/thin/templates/skeleton-10.1-RELEASE;
- The read-write portion of each jail will be created in /usr/jails/thin.

a) Creating a template

- Create a "master" template with the base system. This "master" template will serve as a backup in case of any errors in the following steps. Here are two ways to create the "master" template: FreeBSD: Jails (1) to compile from source (make buildworld) or FreeBSD: Jails (2) to extract the binaries from ISO or download from internet.

The root directory for the "master" template: /usr/jails/templates/10.1-RELEASE

# mkdir -p /usr/jails/templates/10.1-RELEASE
# fetch -o /tmp/base.txz
# tar -xvf /tmp/base.txz -C /usr/jails/templates/10.1-RELEASE
# cp /etc/resolv.conf /usr/jails/templates/10.1-RELEASE/etc/resolv.conf
# cp /etc/localtime /usr/jails/templates/10.1-RELEASE/etc/localtime

- Update the "master" template:

# freebsd-update -b /usr/jails/templates/10.1-RELEASE -f /usr/jails/templates/10.1-RELEASE/etc/freebsd-update.conf fetch install

Note that the options -b and -f are being used to inform the basedir and conffile, both referring to the jail template directory.

Optional: edit /usr/jails/templates/10.1-RELEASE/etc/freebsd-update.conf in order to avoid src component to be updated.

b) Thin jails using NullFS mounts

- Copy the "master" template into a "base" thin jail template:

# mkdir /usr/jails/thin
# cp -r /usr/jails/templates/10.1-RELEASE /usr/jails/thin/templates/base-10.1-RELEASE

- In addition to the "base" thin jail template, you need to create a skeleton template which will hold all the directories that are local to your jail. We're going to copy these directories from the template to the skeleton.

# mkdir -p /usr/jails/thin/templates/skeleton-10.1-RELEASE
# mkdir -p /usr/jails/thin/templates/skeleton-10.1-RELEASE/usr/ports/distfiles
# mkdir -p /usr/jails/thin/templates/skeleton-10.1-RELEASE/home
# mkdir -p /usr/jails/thin/templates/skeleton-10.1-RELEASE/portsbuild

# mv /usr/jails/thin/templates/base-10.1-RELEASE/etc /usr/jails/thin/templates/skeleton-10.1-RELEASE/etc
# mv /usr/jails/thin/templates/base-10.1-RELEASE/usr/local /usr/jails/thin/templates/skeleton-10.1-RELEASE/usr/local
# mv /usr/jails/thin/templates/base-10.1-RELEASE/tmp /usr/jails/thin/templates/skeleton-10.1-RELEASE/tmp
# mv /usr/jails/thin/templates/base-10.1-RELEASE/var /usr/jails/thin/templates/skeleton-10.1-RELEASE/var
# mv /usr/jails/thin/templates/base-10.1-RELEASE/root /usr/jails/thin/templates/skeleton-10.1-RELEASE/root

- The skeleton directory is what is going to be copied for each new jail. It is going to be mounted in /skeleton/ inside the jail. So in the read-only "base" thin jail template we need to create symlink from all the expected locations to the appropriate directories inside the /skeleton/ directory. It is very important to cd into your jail directory and create these symlinks with relative paths. That way they will always link to the correct location no matter where the "base" thin jail template ends up mounted.

# cd /usr/jails/thin/templates/base-10.1-RELEASE
# mkdir -p ./usr/ports 
# mkdir skeleton
# ln -s skeleton/etc etc
# ln -s skeleton/home home
# ln -s skeleton/root root
# ln -s skeleton/tmp tmp
# ln -s skeleton/var var
# ln -s ../../skeleton/usr/local usr/local
# ln -s ../../../skeleton/usr/ports/distfiles usr/ports/distfiles

- Edit make.conf so that your ports workdirectory is located inside the skeleton directory

echo "WRKDIRPREFIX?=  /skeleton/portbuild" >> /usr/jails/thin/templates/skeleton-10.1-RELEASE/etc/make.conf

- Copy your skeleton for the new thin jail thinjail1 under /usr/jails/thin/.

# cp -R /usr/jails/thin/templates/skeleton-10.1-RELEASE /usr/jails/thin/thinjail1

- Add the hostname to the jails rc.conf

# echo hostname=\"thinjail1.wb.lan\" > /usr/jails/thin/thinjail1/etc/rc.conf

- Make the jail directory where the "base" thin jail template and skeleton folder will be mounted

# mkdir -p /usr/jails/thinjail1

- Create the jail entry in /etc/jail.conf:

# The jail definition for thinjail1
thinjail1 {
    path          = "/usr/jails/thinjail1";
    devfs_ruleset = 4;
    host.hostname = "thinjail1.wb.lan";
    ip4.addr      =;
    allow.chflags = 1;
    exec.start    = "/bin/sh /etc/rc";
    exec.stop     = "/bin/sh /etc/rc.shutdown";
    mount.fstab   = "/usr/jails/thinjail1.fstab";

- Create the jail fstab

# Base thin jail template - read-only
  /usr/jails/thinjail1/         nullfs ro 0 0
# Skeleton template - read-write
  /usr/jails/thinjail1/skeleton nullfs rw 0 0

- Start the jail

# jail -c thinjail1

- Updates must be carried out in the "base" thin jail template:

# freebsd-update -b /usr/jails/thin/templates/base-10.1-RELEASE fetch install

- Update ports tree:

# portsnap -p /usr/jails/thin/templates/base-10.1-RELEASE/usr/ports fetch extract


No comments:

Post a Comment