FreeBSD: Jails (3) - VIMAGE

In order to create a jail with its own virtual network stack, with its own network interfaces, addresses, routing table and so on, the kernel must be compiled with the VIMAGE option for this to be available.


The steps on how to compile a custom kernel are well explained in the FreeBSD handbook:

"In order to create a custom kernel configuration file and build a custom kernel, the full FreeBSD source tree must first be installed.

If /usr/src/ does not exist or it is empty, source has not been installed. (...)

Once source is installed, review the contents of /usr/src/sys. This directory contains a number of subdirectories, including those which represent the following supported architectures: amd64, i386, ia64, pc98, powerpc, and sparc64. (...) Each supported architecture has a conf subdirectory which contains the GENERIC kernel configuration file for that architecture.

Do not make edits to GENERIC. Instead, copy the file to a different name and make edits to the copy. The convention is to use a name with all capital letters. When maintaining multiple FreeBSD machines with different hardware, it is a good idea to name it after the machine's hostname. This example creates a copy, named MYKERNEL, of the GENERIC configuration file for the amd64 architecture:

cd /usr/src/sys/amd64/conf 

cp GENERIC MYKERNEL

MYKERNEL can now be customized with any ASCII text editor. (...)

An include directive is available for use in configuration files. This allows another configuration file to be included in the current one, making it easy to maintain small changes relative to an existing file. If only a small number of additional options or drivers are required, this allows a delta to be maintained with respect to GENERIC(...)"

I personally used the last approach which is create another configuration file e.g. MYKERNEL and use the include directive referring to the GENERIC file, as example below:

include GENERIC
ident   MYKERNEL

# Virtual networking for jail
options VIMAGE
device  epair
device  if_bridge

In the custom configuration file MYKERNEL above the VIMAGE option is added, together with the devices epair and if_bridge (alternatively, both devices can be loaded as a module at boot time). Using this method, the custom configuration file expresses local differences from a GENERIC kernel. As upgrades are performed, new features added to GENERIC will also be added to the local kernel.

Once the custom configuration file have been saved, the source code for the kernel can be compiled using the following steps:

cd /usr/src
make buildkernel KERNCONF=MYKERNEL
make installkernel KERNCONF=MYKERNEL

This process will take a while to complete. Then shutdown the system and reboot into the new kernel. After rebooting, the below command can be executed in order to verify if the new kernel is really running:

root@FreeBSD1:~ # uname -a
FreeBSD FreeBSD1 10.1-RELEASE-p30 FreeBSD 10.1-RELEASE-p30 #0: Sun Mar 13 12:55:33 HKT 2016     root@FreeBSD1:/usr/obj/usr/src/sys/MYKERNEL  amd64

After the new kernel is up and running, a new jail can be created using one of the previous posts: FreeBSD: Jails (1) or FreeBSD: Jails (2). Of course there will be some differences that are highlighted below:

Assuming that the new jail is located in /usr/jails/testjail;

- (SAME) Copy over the host's /etc/resolv.conf to the jail:

cp /etc/resolv.conf /usr/jails/testjail/etc/


- (CHANGED) Start the jail for basic configuration (e.g. set root password). By exiting from the shell, the jail will be shut down:

jail -c vnet persist name=testjail path=/usr/jails/testjail command=bin/sh


- (NEW) Create epair and bridge devices:

ifconfig epair0 create
ifconfig epair0b vnet testjail

ifconfig bridge0 create
ifconfig bridge0 addm em0 addm epair0a #(change em0 to correct NIC)
ifconfig bridge0 up

ifconfig epair0a up
jexec testjail ifconfig epair0b 192.168.0.xxx/24 #(Jail IP address)
jexec testjail ifconfig epair0b up
jexec testjail route add default 192.168.0.yyy   #(Gateway or host's ip address)

- (SAME) Edit /etc/freebsd-update.conf inside jail in order to avoid errors during FreeBSD update:

from
Components src world kernel
to
Components world kernel

Reason: there are no kernel or kernel sources inside the jail.


- (CHANGED) Edit the host's /etc/rc.conf:

#Jails
jail_enable="YES"
jail_list="lighttpd nginx testjail" #(LIST SEPARATED BY SPACE)

# Jails networking
#ifconfig_vmx3f0_alias0="inet 192.168.0.150/24"
#ifconfig_vmx3f0_alias1="inet 192.168.0.151/24"
ifconfig_vmx3f0_aliases="inet 192.168.0.150-151 netmask 255.255.255.0"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 up" #(change em0 to correct NIC)


- Create an entry for the jail on the host's /etc/jail.conf:

testjail {
  path = /usr/jails/testjail;
  mount.devfs;
  devfs_ruleset = 4;
  host.hostname = tesjail.wb.lan;
  allow.chflags = 1;

  $if            = "0";
  $ip_addr       = "192.168.0.xxx"; #(Jail IP address)
  $ip_route      = "192.168.0.yyy"; #(Gateway or host's ip address)

  vnet;
  vnet.interface = "epair${if}b";
  exec.prestart  = "ifconfig epair${if} create up";
  exec.prestart += "ifconfig bridge0 addm epair${if}a up";
  exec.start     = "ifconfig epair${if}b inet ${ip_addr} up";
  exec.start    += "route add default ${ip_route}";
  exec.start    += "/bin/sh /etc/rc";
  exec.stop      = "/bin/sh /etc/rc.shutdown";
  exec.poststop  = "ifconfig epair${if}a destroy";
  persist;
}


At last, restart the the host. The jail will be started automatically in the next boot.
To list the active jails use:

jls

and to access the jail e.g. JID 3:

jexec 3 /bin/tcsh

References:
- https://www.freebsd.org/cgi/man.cgi?jail(8)
- https://www.freebsd.org/doc/handbook/kernelconfig-config.html
- https://www.freebsd.org/doc/handbook/kernelconfig-building.html
- http://zewaren.net/site/?q=node/78
- https://forums.freenas.org/index.php?threads/jail-vimage-multiple-network-configuration.15682/
- https://www.freebsd.org/cgi/man.cgi?query=epair&sektion=4&manpath=FreeBSD+8.0-RELEASE
- https://www.freebsd.org/cgi/man.cgi?if_bridge
- https://forums.freebsd.org/threads/49561/