FreeBSD 8 VIMAGE + epair howto
The following text is about to show you how to use the new feature of FreeBSD 8: VIMAGE in a multi-jail environment.
- Compile VIMAGE support into your kernel
Add the “option VIMAGE” to your kernel config and make sure to remove the SCTP support. Lack of SCTP support is one of the reasons VIMAGE is still considered to be experimental.
If you don’t know how to build your own custom kernel image, follow the detailed instructions of the corresponding FreeBSD Handbook chapter .
- Reboot with your new kernel
- First let’s create a pair of epair interfaces then quickly start two VIMAGE jails. I’m using the same fs root to make it simple, but you should create your jails as you always do, you can even use ezjail to it. The only difference is the “vnet” jailparam which is passed as a command line argument to the jail binary.
If you use rc.conf you could try adding the “vnet” parameter to your jail__flags variable for automatic startup.
test# ifconfig epair create
epair0a
test# jail -c vnet name=tibi1 host.hostname=tibi1 path=/ persist
test# jls
JID IP Address Hostname Path
1 - tibi1 /
test# jail -c vnet name=tibi2 host.hostname=tibi2 path=/ persist
test# jls
JID IP Address Hostname Path
1 - tibi1 /
2 - tibi2 /
So we have two instances and an epair device. Let’s see the interface list on the host.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:04:0a
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:05:0b
Both sides of the pair is in the host system. Put one end into one of your jails with the ifconfig
test# ifconfig epair0b vnet 1
test# jexec 1 ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:05:0b
OK, we have a layer 2 connection. Let’s add some IPs and run a ping test
test# jexec 1 ifconfig epair0b 192.168.11.2
test# ifconfig epair0a 192.168.11.1
test# ping 192.168.11.2
PING 192.168.11.2 (192.168.11.2): 56 data bytes
64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.576 ms
64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.081 ms
^C
--- 192.168.11.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.081/0.328/0.576/0.247 ms
It works!
Let’s do the same with your other jail
test# ifconfig epair1b vnet 2 test# jexec 2 ifconfig epair1b 192.168.11.3
Oh wait, these are completely different set of epair interfaces, you can’t use the same IP subnet on them. In order to mash them together on the host side, you have to make a bridge.
test# ifconfig bridge create bridge0 test# ifconfig bridge0 addm epair0a addm epair1a up test#
The commands above will create a new bridge interface, and add the host side of both epair interfaces to the bridge.
You can see it with ifconfig as well:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:04:0a
inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
epair1a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:c0:64:00:05:0a
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether a6:4b:75:2d:2b:9b
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 14183
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 14183
Let’s put the host IP we set for epair0a earlier on the bridge interface instead and bring UP the host side of epair1. (Note: If you assign an IP to an interface, its state should automatically change to UP)
test# ifconfig epair0a -alias
test# ifconfig bridge0 192.168.11.1
test# ifconfig epair1a up
test# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether a6:4b:75:2d:2b:9b
inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 14183
member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 14183
Running ping tests from the second jail, you can now ping your host and your other jail(s) too.
test# jexec 2 ping 192.168.11.1 PING 192.168.11.1 (192.168.11.1): 56 data bytes 64 bytes from 192.168.11.1: icmp_seq=0 ttl=64 time=0.193 ms ^C --- 192.168.11.1 ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.193/0.193/0.193/0.000 ms test# jexec 2 ping 192.168.11.2 PING 192.168.11.2 (192.168.11.2): 56 data bytes 64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.410 ms 64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.089 ms ^C --- 192.168.11.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.089/0.249/0.410/0.160 ms
Remember, now that you have separate networking stacks for each of your jails, the choice of topology is yours.
Posted 2009/12/06 01:56 by jos · Comment [29]
OpenSSH chroot and shell ambiguity
OpenSSH will fail in a scenario where the server is configured with chroot and a shell used by a user is not available outside, just inside the chroot.
The reason behind this is that ssh checks whether the given shell is a file and is executable, but this check doesn’t takes the chroot path into account. This feature was introduced by this patch about a year ago.
We have filed a bugreport to OpenSSH.
Posted 2009/12/03 08:30 by alex · Comment [11]
FreeBSD LD_PRELOAD security fault
This issue was disclosed yesterday sharing a local root exploit for FreeBSD systems. This blog post investigates and explains the issue, also the links and comments are worth reading. There are two types of exploits out there, one requires a setuid root binary to work, the other will use setuid(2).
An initial fix has been committed into the CVS today.
Posted 2009/12/01 23:01 by alex · Comment [1]
PureFTPD virtual chroot
PureFTPD has a feature called “virtual chroot”, where it will mimic a chroot by its own means, but without using the chroot() system call.
An excerpt from the PureFTPD FAQ:
– The ‘virtual chroot’ implementation. With that feature, users can
follow all symbolic links, even when they don’t point inside the jail. This
is very handy to set up directories shared by multiple users. Binary
packages are compiled with virtual chroot by default.To enable the virtual chroot feature when you are compiling the server, use
the —with-virtualchroot with ./configure . If you want a restricted chroot,
don’t include —with-virtualchroot.Please note that the FTP server will never let people create new symbolic
links. Symbolic links have to be already there to be followed. Or if your
users can create symbolic links through Perl or PHP scripts, your hosting
platform is really badly configured. People can install any web file
browser, they don’t need FTP to look at your system files. Recompile PHP
without POSIX functions and run all Perl scripts chrooted.
This feature is turned on by default in the FreeBSD ports.
Jos sent in a comment to the ports maintainer of pureftpd noting the above problem and today they have made this selectable when compiling the package.
Freshports commit message and direct link to CVS.
Posted 2009/11/30 14:51 by alex · Comment [9]
Grow FreeBSD UFS filesystem on VmWare HDDs
In this artice I’m going to show you how to expand your UFS filesystem under FreeBSD that runs in a Vmware Virtual Machine.
FreeBSD uses slices instead of traditional PC partitions. To sum it up shortly, it means that your whole disk contains only a single traditional partition with the partition type ‘freebsd(165)’. Inside this partition you will have slices. In a typical FreeBSD installation you have seperate slices for /, /usr, /var, /tmp and swap. In most cases the last slice on the partition is the /usr, and hopefully this is the one we have to extend, because in this case the only thing needed is to add some space to the end of the drive and extend the last slice. Sounds easy? Don’t think so!
The main steps are:
- Increase the actual (virtual) hdd size
- Extend the partition to cover the whole disk
- Extend the size of the last slice to cover the whole partition
- Extend the actual UFS filesystem on the newly modified slice
All the details:
First, forget livecds like gparted-live, knoppix, other partition hacking tools because they will not work.
- Stop your VM and Edit its settings
- Increase HDD size and click OK
- Boot into single user mode (select 4 in the FreeBSD boot menu)
- check your new disk via dmesg, you will see the increased block number
but your partition is still at the previous size (fdisk -s)
- Let’s change the partition size using fdisk -u
In my example my partition starts at 63 and has the size of 18860247 blocks
What you would normally do is to adjust the new size to totalblocks-start. In this case fdisk will complain that the ending block is not on cylinder boundry. Fdisk will fix the value you entered but if you want to do the math yourself here is how it should be done correctly:
cylinder_size = 16065 [blocks]
new size = trunc(totalblocks/cylinder_size)*cylinder_size-startblock
in my case
trunc(25165824/16065)*16065-63 = 25157727
- Next step is to extend the slice size. Slices are edited with the bsdlabel utility. You might need to mount a couple of more partitions to access your editor.
EDITOR=<yourfav> bsdlabel -ewill bring up the slice table in your favorite editor. Now you need to change two things. - First the line where it says do not edit: change it to your new partition size.
- To get the new size of the last slice you need to do the following calculations:
new_size_of_last_slice=partition_size-offset_of_last_slice
- The last step is to extend the FS itself on the new slice. Make sure you don’t have the slice in question mounted. Then run growfs /dev/
. In my case it was /dev/da0s1f
- To check your results run fsck, mount -a, df -h
- If you’re happy, you can reboot into multiuser mode now
Posted 2009/11/30 07:31 by jos · Comment [22]