Blinkenblog

I’ve received requests to add FreeBSD 7.x support to mod_blink, my inplementation of apache user separation. The apache module works just fine on newer apache releases, but there have been many changes on the kernel side. Kernel support is needed to grant additional privileges to the apache module. The old FreeBSD kernel module is a “hack” because it modifies the seteuid/setegid syscalls by hooking them via the syscall table, which is a pretty nasty way to get things done.

Luckily things have improved on the FreeBSD side which allows for a clean inplementation. The TrustedBSD MAC framework which was imported at around 5.0 presents clean hooks to relax/restrict priviliges via loadable modules, rather similar to LSM on the Linux side.

This weekend I’ve written mac_blink which should work on FreeBSD 5+ kernels with MAC support (KERNCONF=MAC).

At work we’ve bought new NetApp filers, and I’ve been tasked with the installation. One of the new things that I decided to inplement is iSCSI monitoring, anything critical should be monitored anyway.This is where I hit the first hurdle, nobody seems to do iSCSI monitoring with nagios. Google doesn’t show any check_iscsi examples. So I started investigating the available options. The main contender is iscsiadm from the open-iscsi stack. For monitoring purposes this package has a big problem; it uses daemons to save state, which probably speeds things up a fair bit, but totally ruins it for monitoring purposes.

Then I found http://code.google.com/p/freebsd-iscsi/ which after a careful look does exactly what’s needed, iSCSI sendtargets discovery and not much else. When adapting the software for nagios I quickly noticed some segfaults, which warranted further investigation. It turns out that a buffer of 1024 bytes is allocated, which is used for transmitting and receiving iSCSI messages. But the code which parses messages just looks at the datasegmentlength field in the header, which is used even when it exceeds the buffersize of 1024. When trying to find a simple solution for this problem I noticed the MaxRecvDataSegmentLength parameter which can be used during login to limit the length of messages. This looked like a quick fix, so I dediced to try it out. The NetApp filer I was talking to however didn’t appreciate the change.

When creating a sendtargets responce a NetApp will include every network interface which has iSCSI enabled. This will cause the response to be quite large if you have 10-15 active interfaces. So when trying to stuff all this information, about 1517 bytes normally in two 1500 bytes ethernet frames, in 512 bytes I hit the following panic:

Tue Feb 12 14:19:54 CET [mgr.stack.string:notice]: Panic string: ../driver/scsitarget/iswt/iswti_text.c:624: Assertion failure. in process iswti_iscsip_thread on release NetApp Release 7.2.4L1

The sad thing with NetApp is that most bugs like this will cause a reboot, which interrupts operation quite heavily. The main problem with this one is that if iSCSI is enabled it probably isn’t possible to stop people rebooting your filers.

One of my basic pet-peeves has been dhclient’s meddling with /etc/resolv.conf. I run my own resolvers and don’t like using ISP nameservers, which are often overloaded. Today I finally looked deeply enough into this to solve the issue on Debian/Ubuntu. Just add the following to /etc/dhcpd3/dhclient-enter-hooks.d/resolvconf.

# For safety, undefine the nasty default make_resolv_conf()
make_resolv_conf() {
    true
}

On FreeBSD/NetBSD the same can be accomplished by adding the above code to /etc/dhclient-enter-hooks. This overrides the default function used by dhclient with one that does nothing. Nice, clean, simple.

On FreeBSD most system configuration is centralized in /etc/rc.conf, which sets options for system startup and all well-behaved daemons. Most people however don’t know that rc.conf.local also is sourced by the same scripts. This makes it possible to use a generic rc.conf to specify site/company wide defaults, and configure host specific settings in rc.conf.local. One advantage is that rc.conf.local usually ends up being quite short which simplifies host management. Using this scheme works extremely well when it’s combined with cfengine and source control. Versioning and commit mailinglists can do truly amazing things for sysadmin productivity.

I carefully watched the Ubuntu respositories for linux 2.6.20 packages, and quickly noticed when the experimental packages were released. These however aren’t quite stable yet, the atheros driver (in restricted) currently oopses, which is somewhat problematic for me.

Someone has already reported the problem at launchpad, let’s see how quickly it’ll get fixed.

After some research I discovered that linux has IPv6 Stateful firewalling since 2.6.15. But the code won’t be usable until at least 2.6.20 because it doesn’t play nice with IPv4 NAT.

Initial Patches are available on the netfilter mailinglists, let’s hope they make 2.6.20.

I have two gateways at home, both running different operating systems. There are quite a few alternatives for router failover. However most opensource VRRP inplementations suck (keepalived, vrrpd, freevrrpd) as does CARP on linux (ucarp). Which is why I decided to scratch this itch by writing my own failover daemon in perl.

I quickly settled on multicast and a somewhat vrrp-like packet format with hmac to secure the messages. The send and receive tasks are handled by separate threads, which allows the main loop to just be a simple state machine. Installation is quite simple (if you have daemontools):

mkdir -p failoverd/bin
mkdir -p failoverd/env
mkdir -p failoverd/log/main

cat <failoverd/run
#!/bin/sh

exec 2>&1
exec envdir ./env ./bin/failoverd
EOF

cat <failoverd/log/run
#!/bin/sh
exec setuidgid nobody multilog t ./main
EOF

chmod +x failoverd/run failoverd/log/run
chown nobody failoverd/log/main

echo MYINTERFACE > failoverd/env/INTERFACE
echo MYPASSWORD > failoverd/env/SECRET
echo 100 > failoverd/env/PRIO
echo ./bin/backup > failoverd/env/BACKUP
echo ./bin/master > failoverd/env/MASTER

Download the daemon and put it in failoverd/bin, and write your own custom backup/master scripts.

If ipv6 doesn’t work as expected on dapper, aka routing doesn’t work, just add ipv6 to /etc/initramfs-tools/modules.

echo ipv6 >> /etc/initramfs-tools/modules

After mkinitramfs and a reboot IPv6 should be functional again.