I was browsing through the changesets of the upcoming 7.3 release and stumbled upon a nice addition which would have come handy a few months ago.
According to the default behaviour, when jails are trying to reach a host through one of the external interfaces, the source address used for communication is taken from the interface which has route to the destination.
In other words, if you have a single external IP address, and a couple of jails on the local interface, the source address for outbound connections will be the address on the external interface.
This is especially annoying if you’re trying to deploy per jail firewall rules for outgoing connections: you can’t use the jail ip address as source address for matching packets.
Until now there we had three different approaches to overcome this problem:
Multiple routing tables
The idea is that every process in a certain jail should use a routing table which has no default route, hereby restricting every outgoing connection. Note that incoming connections are assigned to the default (0) routing table, so you would still be able to access your services from the outside.
Add the following kernel configuration option and rebuild the kernel. The 2 is the number of FIB (Forward Information Base, synonym for a routing table here). The maximum value is 16.
This number can be modified on boot time. To do so, add the following to /boot/loader.conf and reboot the system:
Set a loader tunable net.my_fibnum if needed. This means the default number of routing tables. If not specified, 0 will be used.
Set a loader tunable net.add_addr_allfibs if needed. This enables to add routes to all FIBs for new interfaces by default. When this is set to 0, it will only allocate routes on interface changes for the FIB of the caller when adding a new set of addresses to an interface. Note that this tunable is set to 1 by default.
Then add jail_name_fib=n to your rc.conf or ezjail configuration to assign the jail process to a certain routing table.
Jail match support
Find a firewall with jail match support. As we didn’t want to use anything else but pf, we looked into the source and found that an actual support for matching jails could be added easily. We even got to the stage where we had beta patches, but due to the fact that we needed a stable solution in our production environment we dropped the idea. Not to mention that it would have never been merged into pf, as OpenBSD has no jails and FreeBSD has no pf fork. (They treat pf as a port, so feature additions have to go through OpenBSD first. Uh good luck with that.)
Uid match support
Luckily pf has uid/gid match support, and by having all our jails use globally unique uids it became obvious that this method was the best so far.
pass out on $ext_if inet proto tcp from ($ext_if) to any port 25 user > 1000 keep state
The above pf rule blocks outgoing smtp access for each process that has a UID larger than 1000.
Disabling source address selection for jails
And finally the new addition of FreeBSD 7.3 allows you to disable the default source address selection method, always assigning the primary address of the jail to each outgoing packet. This can be done via the following sysctl flags:
security.jail.ip4_saddrsel=0 for IPv4
security.jail.ip6_saddrsel=0 for IPv6
Note that this is global and not a per jail setting.
Posted 2010/03/05 11:35 by jos · Comment 
We have found an interesting feature in the FreeBSD run-time link editor (rtld), which links dynamic executables with their needed libraries at run time.
The ld-elf.so.1 utility itself is loaded by the kernel together with any dynamically-linked program that is to be executed. The kernel transfers control to the dynamic linker. After the dynamic linker has finished loading, relocating, and initializing the program and its required shared objects, it transfers control to the entry point of the program.
It also has an executable flag, so let’s try to execute it.
bash: /libexec/ld-elf.so.1: cannot execute binary file
Turns out the ld-elf.so keeps loading itself over and over, maxing out a cpu core while doing so. I had to enforce a cputime limit in login.conf so funny users won’t be able to profit from their discovery.
A fix isn’t likely as it looks like this is just one of those things you shouldn’t do
Posted 2010/02/22 21:20 by jos · Comment
When looking to access quota under Ruby, we had to look for extensions and the first search revealed the Ruby/Quota project, however it was unmaintained. Someone sent in a patch for NetBSD and support for newer Linux versions, but that was not applied either. We sent an email to the owner, but till now no response received.
So we took the CVS, imported it into git (using git cvsimport), applied the patches, applied own patches, added a gemspec, extended the manual and uploaded it to Github. Tested under FreeBSD 7.2, Mac OS X 10.5.8 and Linux 2.6.24.
You can get it here.
Posted 2009/12/07 11:31 by alex · Comment
The following text is about to show you how to use the new feature of FreeBSD 8: VIMAGE in a multi-jail environment.
- Compile VIMAGE support into your kernel
Add the “option VIMAGE” to your kernel config and make sure to remove the SCTP support. Lack of SCTP support is one of the reasons VIMAGE is still considered to be experimental.
If you don’t know how to build your own custom kernel image, follow the detailed instructions of the corresponding FreeBSD Handbook chapter .
- Reboot with your new kernel
- First let’s create a pair of epair interfaces then quickly start two VIMAGE jails. I’m using the same fs root to make it simple, but you should create your jails as you always do, you can even use ezjail to it. The only difference is the “vnet” jailparam which is passed as a command line argument to the jail binary.
If you use rc.conf you could try adding the “vnet” parameter to your jail_
_flags variable for automatic startup.
So we have two instances and an epair device. Let’s see the interface list on the host.
Both sides of the pair is in the host system. Put one end into one of your jails with the ifconfig
OK, we have a layer 2 connection. Let’s add some IPs and run a ping test
Let’s do the same with your other jail
Oh wait, these are completely different set of epair interfaces, you can’t use the same IP subnet on them. In order to mash them together on the host side, you have to make a bridge.
The commands above will create a new bridge interface, and add the host side of both epair interfaces to the bridge.
You can see it with ifconfig as well:
Let’s put the host IP we set for epair0a earlier on the bridge interface instead and bring UP the host side of epair1. (Note: If you assign an IP to an interface, its state should automatically change to UP)
Running ping tests from the second jail, you can now ping your host and your other jail(s) too.
Remember, now that you have separate networking stacks for each of your jails, the choice of topology is yours.
Posted 2009/12/06 01:56 by jos · Comment 
OpenSSH will fail in a scenario where the server is configured with chroot and a shell used by a user is not available outside, just inside the chroot.
The reason behind this is that ssh checks whether the given shell is a file and is executable, but this check doesn’t takes the chroot path into account. This feature was introduced by this patch about a year ago.
We have filed a bugreport to OpenSSH.
Posted 2009/12/03 08:30 by alex · Comment