FreeBSD 8 VIMAGE + epair howto
The following text is about to show you how to use the new feature of FreeBSD 8: VIMAGE in a multi-jail environment.
- Compile VIMAGE support into your kernel
Add the “option VIMAGE” to your kernel config and make sure to remove the SCTP support. Lack of SCTP support is one of the reasons VIMAGE is still considered to be experimental.
If you don’t know how to build your own custom kernel image, follow the detailed instructions of the corresponding FreeBSD Handbook chapter .
- Reboot with your new kernel
- First let’s create a pair of epair interfaces then quickly start two VIMAGE jails. I’m using the same fs root to make it simple, but you should create your jails as you always do, you can even use ezjail to it. The only difference is the “vnet” jailparam which is passed as a command line argument to the jail binary.
If you use rc.conf you could try adding the “vnet” parameter to your jail__flags variable for automatic startup.
test# ifconfig epair create
epair0a
test# jail -c vnet name=tibi1 host.hostname=tibi1 path=/ persist
test# jls
JID IP Address Hostname Path
1 - tibi1 /
test# jail -c vnet name=tibi2 host.hostname=tibi2 path=/ persist
test# jls
JID IP Address Hostname Path
1 - tibi1 /
2 - tibi2 /
So we have two instances and an epair device. Let’s see the interface list on the host.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:04:0a
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:05:0b
Both sides of the pair is in the host system. Put one end into one of your jails with the ifconfig
test# ifconfig epair0b vnet 1
test# jexec 1 ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:05:0b
OK, we have a layer 2 connection. Let’s add some IPs and run a ping test
test# jexec 1 ifconfig epair0b 192.168.11.2
test# ifconfig epair0a 192.168.11.1
test# ping 192.168.11.2
PING 192.168.11.2 (192.168.11.2): 56 data bytes
64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.576 ms
64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.081 ms
^C
--- 192.168.11.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.081/0.328/0.576/0.247 ms
It works!
Let’s do the same with your other jail
test# ifconfig epair1b vnet 2 test# jexec 2 ifconfig epair1b 192.168.11.3
Oh wait, these are completely different set of epair interfaces, you can’t use the same IP subnet on them. In order to mash them together on the host side, you have to make a bridge.
test# ifconfig bridge create bridge0 test# ifconfig bridge0 addm epair0a addm epair1a up test#
The commands above will create a new bridge interface, and add the host side of both epair interfaces to the bridge.
You can see it with ifconfig as well:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:04:0a
inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
epair1a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:05:0a
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether a6:4b:75:2d:2b:9b
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 14183
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 14183
Let’s put the host IP we set for epair0a earlier on the bridge interface instead and bring UP the host side of epair1. (Note: If you assign an IP to an interface, its state should automatically change to UP)
test# ifconfig epair0a -alias
test# ifconfig bridge0 192.168.11.1
test# ifconfig epair1a up
test# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether a6:4b:75:2d:2b:9b
inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 14183
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 14183
Running ping tests from the second jail, you can now ping your host and your other jail(s) too.
test# jexec 2 ping 192.168.11.1 PING 192.168.11.1 (192.168.11.1): 56 data bytes 64 bytes from 192.168.11.1: icmp_seq=0 ttl=64 time=0.193 ms ^C --- 192.168.11.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.193/0.193/0.193/0.000 ms test# jexec 2 ping 192.168.11.2 PING 192.168.11.2 (192.168.11.2): 56 data bytes 64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.410 ms 64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.089 ms ^C --- 192.168.11.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.089/0.249/0.410/0.160 ms
Remember, now that you have separate networking stacks for each of your jails, the choice of topology is yours.
Posted 2009/12/06 01:56 by jos · Comment [18]
VIMAGE - Better virtualization in FreeBSD 8
Now that FreeBSD 8 is out, among many changes we can find enhancements in the field of virtualization as well. A newly developed virtualization container called VIMAGE has been implemented to enable virtualization of the FreeBSD network stack.
As you may know previous releases of FreeBSD had support only for jails with IP addresses of the main network stack; meaning once you configured IP/IPv6 addresses on your host system, a subset of those addresses could be associated to each one of your jails. As simple as it sounds, it actually doesn’t let you perform several networking related tasks inside of a jail, and you couldn’t separate your jails from each other with a firewall as there were no real interfaces present in your system.
With VIMAGE you have a jail with full instance of the host’s networking stack, including loopback interface, routing tables, etc. Network interfaces created on the host system can be moved to any VIMAGE jail to enable its connection to the outside world with a new option of ifconfig called “vnet”.
vnet jail
Move the interface to the jail , specified by name or JID. If the jail has a virtual network stack, the interface will disap- pear from the current environment and become visible to the jail.
Note: Option “-vnet” does the opposite.
As you might not have as many network interfaces as jails, you might need some workarounds to tunnel network traffic between two interfaces of your system.
Forget TUN/TAP and VPNs. FreeBSD 8 has a special network device called epair , which lets you create a pair of interconnected ethernet interfaces. If you move one of them to a VIMAGE jail you are basicly done. Feel free to bridge them or use VLANs, they will still work. I don’t know about the overhead of epair, but if all you care about is security, this might be the best choice for you on FreeBSD.
To enable VIMAGE you have to add “option VIMAGE” to your kernel configuration file and recompile/reinstall it.
Posted 2009/11/27 22:06 by jos · Comment [4]