Subscribe to the RSS Feed

 

FreeBSD 7 jail source address selection

I was browsing through the changesets of the upcoming 7.3 release and stumbled upon a nice addition which would have come handy a few months ago.

According to the default behaviour, when jails are trying to reach a host through one of the external interfaces, the source address used for communication is taken from the interface which has route to the destination.
In other words, if you have a single external IP address, and a couple of jails on the local interface, the source address for outbound connections will be the address on the external interface.
This is especially annoying if you’re trying to deploy per jail firewall rules for outgoing connections: you can’t use the jail ip address as source address for matching packets.
Until now there we had three different approaches to overcome this problem:

Multiple routing tables
The idea is that every process in a certain jail should use a routing table which has no default route, hereby restricting every outgoing connection. Note that incoming connections are assigned to the default (0) routing table, so you would still be able to access your services from the outside.

Add the following kernel configuration option and rebuild the kernel. The 2 is the number of FIB (Forward Information Base, synonym for a routing table here). The maximum value is 16.

options ROUTETABLES=2

This number can be modified on boot time. To do so, add the following to /boot/loader.conf and reboot the system:

net.fibs=6

Set a loader tunable net.my_fibnum if needed. This means the default number of routing tables. If not specified, 0 will be used. Set a loader tunable net.add_addr_allfibs if needed. This enables to add routes to all FIBs for new interfaces by default. When this is set to 0, it will only allocate routes on interface changes for the FIB of the caller when adding a new set of addresses to an interface. Note that this tunable is set to 1 by default.

Then add jail_name_fib=n to your rc.conf or ezjail configuration to assign the jail process to a certain routing table.

Jail match support
Find a firewall with jail match support. As we didn’t want to use anything else but pf, we looked into the source and found that an actual support for matching jails could be added easily. We even got to the stage where we had beta patches, but due to the fact that we needed a stable solution in our production environment we dropped the idea. Not to mention that it would have never been merged into pf, as OpenBSD has no jails and FreeBSD has no pf fork. (They treat pf as a port, so feature additions have to go through OpenBSD first. Uh good luck with that.)

Uid match support
Luckily pf has uid/gid match support, and by having all our jails use globally unique uids it became obvious that this method was the best so far.
pass out on $ext_if inet proto tcp from ($ext_if) to any port 25 user > 1000 keep state
The above pf rule blocks outgoing smtp access for each process that has a UID larger than 1000.

Disabling source address selection for jails
And finally the new addition of FreeBSD 7.3 allows you to disable the default source address selection method, always assigning the primary address of the jail to each outgoing packet. This can be done via the following sysctl flags:
security.jail.ip4_saddrsel=0 for IPv4
security.jail.ip6_saddrsel=0 for IPv6
Note that this is global and not a per jail setting.
Changeset: r202924

Posted 2010/03/05 11:35 by jos · Comment [1]


ld-elf.so Local DoS Vulnerability or not

We have found an interesting feature in the FreeBSD run-time link editor (rtld), which links dynamic executables with their needed libraries at run time.

The ld-elf.so.1 utility itself is loaded by the kernel together with any dynamically-linked program that is to be executed. The kernel transfers control to the dynamic linker. After the dynamic linker has finished loading, relocating, and initializing the program and its required shared objects, it transfers control to the entry point of the program.

It also has an executable flag, so let’s try to execute it.

Results:
FreeBSD 6.3.x:
$ /libexec/ld-elf.so.1
bash: /libexec/ld-elf.so.1: cannot execute binary file
$

FreeBSD 7.x:
$ /libexec/ld-elf.so.1
(no return)

Turns out the ld-elf.so keeps loading itself over and over, maxing out a cpu core while doing so. I had to enforce a cputime limit in login.conf so funny users won’t be able to profit from their discovery.
A fix isn’t likely as it looks like this is just one of those things you shouldn’t do

Posted 2010/02/22 21:20 by jos · Comment


Ruby/Quota

When looking to access quota under Ruby, we had to look for extensions and the first search revealed the Ruby/Quota project, however it was unmaintained. Someone sent in a patch for NetBSD and support for newer Linux versions, but that was not applied either. We sent an email to the owner, but till now no response received.

So we took the CVS, imported it into git (using git cvsimport), applied the patches, applied own patches, added a gemspec, extended the manual and uploaded it to Github. Tested under FreeBSD 7.2, Mac OS X 10.5.8 and Linux 2.6.24.

You can get it here.

Posted 2009/12/07 11:31 by alex · Comment


FreeBSD 8 VIMAGE + epair howto

The following text is about to show you how to use the new feature of FreeBSD 8: VIMAGE in a multi-jail environment.

If you don’t know how to build your own custom kernel image, follow the detailed instructions of the corresponding FreeBSD Handbook chapter .

test# ifconfig epair create
epair0a
test# jail -c vnet name=tibi1 host.hostname=tibi1 path=/ persist
test# jls
   JID  IP Address      Hostname                      Path
     1  -               tibi1                         /
test# jail -c vnet name=tibi2 host.hostname=tibi2 path=/ persist
test# jls
   JID  IP Address      Hostname                      Path
     1  -               tibi1                         /
     2  -               tibi2                         /

So we have two instances and an epair device. Let’s see the interface list on the host.

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:04:0a
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:05:0b

Both sides of the pair is in the host system. Put one end into one of your jails with the ifconfig vnet command and verify the results by running ifconfig inside your jail.

test# ifconfig epair0b vnet 1
test# jexec 1 ifconfig
lo0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:05:0b

OK, we have a layer 2 connection. Let’s add some IPs and run a ping test
test# jexec 1 ifconfig epair0b 192.168.11.2
test# ifconfig epair0a 192.168.11.1
test# ping 192.168.11.2
PING 192.168.11.2 (192.168.11.2): 56 data bytes
64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.576 ms
64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.081 ms
^C
--- 192.168.11.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.081/0.328/0.576/0.247 ms

It works!

Let’s do the same with your other jail


test# ifconfig epair1b vnet 2
test# jexec 2 ifconfig epair1b 192.168.11.3

Oh wait, these are completely different set of epair interfaces, you can’t use the same IP subnet on them. In order to mash them together on the host side, you have to make a bridge.
test# ifconfig bridge create
bridge0
test# ifconfig bridge0 addm epair0a addm epair1a up
test#

The commands above will create a new bridge interface, and add the host side of both epair interfaces to the bridge.
You can see it with ifconfig as well:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet6 ::1 prefixlen 128
        inet 127.0.0.1 netmask 0xff000000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:04:0a
        inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
epair1a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:c0:64:00:05:0a
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a6:4b:75:2d:2b:9b
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 14183
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 14183

Let’s put the host IP we set for epair0a earlier on the bridge interface instead and bring UP the host side of epair1. (Note: If you assign an IP to an interface, its state should automatically change to UP)

test# ifconfig epair0a -alias
test# ifconfig bridge0 192.168.11.1
test# ifconfig epair1a up
test# ifconfig bridge0
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether a6:4b:75:2d:2b:9b
        inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 14183
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 4 priority 128 path cost 14183

Running ping tests from the second jail, you can now ping your host and your other jail(s) too.

test# jexec 2 ping 192.168.11.1
PING 192.168.11.1 (192.168.11.1): 56 data bytes
64 bytes from 192.168.11.1: icmp_seq=0 ttl=64 time=0.193 ms
^C
--- 192.168.11.1 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.193/0.193/0.193/0.000 ms
test# jexec 2 ping 192.168.11.2
PING 192.168.11.2 (192.168.11.2): 56 data bytes
64 bytes from 192.168.11.2: icmp_seq=0 ttl=64 time=0.410 ms
64 bytes from 192.168.11.2: icmp_seq=1 ttl=64 time=0.089 ms
^C
--- 192.168.11.2 ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.089/0.249/0.410/0.160 ms

Remember, now that you have separate networking stacks for each of your jails, the choice of topology is yours.

Posted 2009/12/06 01:56 by jos · Comment [4]


OpenSSH chroot and shell ambiguity

OpenSSH will fail in a scenario where the server is configured with chroot and a shell used by a user is not available outside, just inside the chroot.

The reason behind this is that ssh checks whether the given shell is a file and is executable, but this check doesn’t takes the chroot path into account. This feature was introduced by this patch about a year ago.

We have filed a bugreport to OpenSSH.

Posted 2009/12/03 08:30 by alex · Comment


← Previously