We have been using VMware vSphere in an environment where VM admins are not to be trusted as they are only customers who purchase resources from us. It might happen that they use their virtual machine and the assigned public IPv4 address to spoof network packets, share illegal materials over various P2P networks or attack other VM guests locally.
VMware offers no solutions for the above problems, the assumption is that ESX hosts are supposed to be inside of a protected corporate environment. Networking is done via virtual switches which only implement little network security.
What we need to prevent:
- MAC, IP spoofing
- Illegal P2P traffic
- Illegal SMTP traffic (spam relays, etc)
What we already have:
- Forged transmit protection: VMware knows the MAC addresses associated to the network interfaces of the VM, so it can drop packets with forged source addresses. (Doesn’t protect against IP spoofing)
What we did:
We created a small virtual machine where we installed OpenWrt, a small yet efficient Linux based firewall/router solution for embedded systems.
The VM parameters were:
Network interfaces: 2
The actual image only took 10MBs which already included every tool we needed.
Then we connected the interfaces between the VM and the outside network, and created a software bridge.
We ran tests on an ESX server (4.0) with a quad core E5420 Xeon CPU.
Initiating a P2P network traffic from the protected VM resulted in 100% CPU pull for the bridge VM, the network troughput couldn’t go any higher than 3.5MB/s. Increasing the CPU limit for the VM resulted in linear growth of network troughput. With 1Ghz we had 7MB/s, and without CPU limitation the CPU usage went up to 1.6Ghz maximizing our 100Mbit/s uplink. Turning on and off the firewall with many Layer2,3,7 rules didn’t seem to affect the throughput performance. At least we knew that the bottleneck was the CPU and the bridge.
The results were somewhat disappointing so we started to dig further.
Our first guess was that the 32bit Bridge VM didn’t use the HW Virtualization Technology in the CPU. To have VMware use the HW method we had to port openwrt to x86_64 CPU. The task was not easy and trivial but we managed to create an image. For our disappointment the results were nearly the same if not worse. Checking through pubications and papers we found that engineers at VMware made SW virtualization better than Intel-VT. URL
In almost every of their benchmarks the software method was faster. Funny thing for Intel.
We implemented our ideas into the Bridging Firewall VM, changed the port groups of the VMs to be protected and now they are spam, warez and dos-attack free.
Do you need a production ready application-level firewall in your vSphere environment similar to this one? Contact Us!
Posted 2010/04/29 00:02 by jos · Comment 
For a long time we have been trying to come up with a way to have a binary package repository which is based on heterogeneous package sources. (pkg_add/ports)
After playing with “portupgrade” and “make package” which often failed due to various reasons, it became obvious this method wont be beneficial in long-term.
This howto explains how to create the exact package replica of a jail full of installed packages.
- First create a jail and build/install packages. I have created a build jail only for this purpose.
- Use the script posted below to backup currently installed packages and setup a package tree similar to “make package” used at ports build. (Modify the first line according to your needs)
The first loop backs up all the installed packages on the system to $PACKAGEDIR/All. Then by going through the packages again the script recreates the symlinks used in FreeBSD packaging. It also checks for package files with different versions (but same origin) and deletes those which are not installed.
A common roll-back scenario
To be deleted: mysql-server-5.1.45.tbz
Packages with different versions that can co-exist on the same system
To be deleted: none
The script checks for package origins and by determining that lang/ruby18 is different from lang/ruby19 both packages remain untouched and included to the repository.
Known bugs: Does not check for deleted packages. It’s better to recreate the whole repository after a big portupgrade anyway.
- Put the files onto a webserver using the following directory scheme:
- Use this as a source with pkg_add (or pkg_replace):
pkg_add -r bashwill fetch from the following URL:
Posted 2010/03/23 23:10 by jos · Comment 
I was browsing through the changesets of the upcoming 7.3 release and stumbled upon a nice addition which would have come handy a few months ago.
According to the default behaviour, when jails are trying to reach a host through one of the external interfaces, the source address used for communication is taken from the interface which has route to the destination.
In other words, if you have a single external IP address, and a couple of jails on the local interface, the source address for outbound connections will be the address on the external interface.
This is especially annoying if you’re trying to deploy per jail firewall rules for outgoing connections: you can’t use the jail ip address as source address for matching packets.
Until now there we had three different approaches to overcome this problem:
Multiple routing tables
The idea is that every process in a certain jail should use a routing table which has no default route, hereby restricting every outgoing connection. Note that incoming connections are assigned to the default (0) routing table, so you would still be able to access your services from the outside.
Add the following kernel configuration option and rebuild the kernel. The 2 is the number of FIB (Forward Information Base, synonym for a routing table here). The maximum value is 16.
This number can be modified on boot time. To do so, add the following to /boot/loader.conf and reboot the system:
Set a loader tunable net.my_fibnum if needed. This means the default number of routing tables. If not specified, 0 will be used.
Set a loader tunable net.add_addr_allfibs if needed. This enables to add routes to all FIBs for new interfaces by default. When this is set to 0, it will only allocate routes on interface changes for the FIB of the caller when adding a new set of addresses to an interface. Note that this tunable is set to 1 by default.
Then add jail_name_fib=n to your rc.conf or ezjail configuration to assign the jail process to a certain routing table.
Jail match support
Find a firewall with jail match support. As we didn’t want to use anything else but pf, we looked into the source and found that an actual support for matching jails could be added easily. We even got to the stage where we had beta patches, but due to the fact that we needed a stable solution in our production environment we dropped the idea. Not to mention that it would have never been merged into pf, as OpenBSD has no jails and FreeBSD has no pf fork. (They treat pf as a port, so feature additions have to go through OpenBSD first. Uh good luck with that.)
Uid match support
Luckily pf has uid/gid match support, and by having all our jails use globally unique uids it became obvious that this method was the best so far.
pass out on $ext_if inet proto tcp from ($ext_if) to any port 25 user > 1000 keep state
The above pf rule blocks outgoing smtp access for each process that has a UID larger than 1000.
Disabling source address selection for jails
And finally the new addition of FreeBSD 7.3 allows you to disable the default source address selection method, always assigning the primary address of the jail to each outgoing packet. This can be done via the following sysctl flags:
security.jail.ip4_saddrsel=0 for IPv4
security.jail.ip6_saddrsel=0 for IPv6
Note that this is global and not a per jail setting.
Posted 2010/03/05 11:35 by jos · Comment 
We have found an interesting feature in the FreeBSD run-time link editor (rtld), which links dynamic executables with their needed libraries at run time.
The ld-elf.so.1 utility itself is loaded by the kernel together with any dynamically-linked program that is to be executed. The kernel transfers control to the dynamic linker. After the dynamic linker has finished loading, relocating, and initializing the program and its required shared objects, it transfers control to the entry point of the program.
It also has an executable flag, so let’s try to execute it.
bash: /libexec/ld-elf.so.1: cannot execute binary file
Turns out the ld-elf.so keeps loading itself over and over, maxing out a cpu core while doing so. I had to enforce a cputime limit in login.conf so funny users won’t be able to profit from their discovery.
A fix isn’t likely as it looks like this is just one of those things you shouldn’t do
Posted 2010/02/22 21:20 by jos · Comment 
When looking to access quota under Ruby, we had to look for extensions and the first search revealed the Ruby/Quota project, however it was unmaintained. Someone sent in a patch for NetBSD and support for newer Linux versions, but that was not applied either. We sent an email to the owner, but till now no response received.
So we took the CVS, imported it into git (using git cvsimport), applied the patches, applied own patches, added a gemspec, extended the manual and uploaded it to Github. Tested under FreeBSD 7.2, Mac OS X 10.5.8 and Linux 2.6.24.
You can get it here.
Posted 2009/12/07 11:31 by alex · Comment