Planet FreeBSD

August 04, 2020

Warner Losh

Bootstrapping 2.11BSD (no patches) from 2.11BSD pl 195

Bootstrapping 2.11BSD (original)

I've had the sources for what I think is the original 2.11BSD for some time now. However, how do I know these sources are good? That's a very good question. I have a series of tests that I'm doing to verify that the sources are consistent with what we know, or have some kind of known deviation / reconstruction when not. They had passed only the first of my many tests (they were consistent with the patches themselves, but nothing else). It was time to see if I could build what I'd made.
Click to close image, click and drag to move. Use arrow keys for next and previous.

One thing we know for sure: The 2.11BSD release happened. This means that sources for the release must be buildable, in some way. The 2.11 BSD release notes don't mention any reproducibility issues. Presumably the documented way will work. However patches 106-111 fix dozens of build issues that affected reproducibility of the build. In addition, one should be able to build twice in a row and get identical results, modulo a few binaries that encode dates and such. Experience has shown that many programs in /usr/local or /usr/new are the worst offenders. I've made the decision that if make install doesn't install it from the top level, then it won't be in the release I recreate. Though I also made the decision that building some man pages by hand was also OK to make that happen...

Part of building it twice is building it at all. In patches 158-178, the binary format of the .o files changes to accommodate longer symbol names. As a result, the binaries in the 195 image don't produce binaries that work on the unpatched release (well, the binaries themselves do, but the .o's are wrong, as are all programs that read symbols). In addition, there's issues just building everything on the 195 image: as, nm, and ld don't even build, and without those, you won't get far. In fact, the 195 assembler won't even assemble the assembler I've recreated. Since the straight forward way won't work, I thought I'd document what does.

For a background on the toolchains, please see an earlier blog post. It goes over all the basics of toolchains, which I assume people are familiar with.

Bootstrapping as

So, we have to bootstrap the assembler. The 2.11pl195 assembler won't assemble it properly. The v7 assembler will. However, building it on a v7 system isn't the solution: the resulting binaries won't run on the 2.11BSD system. The system call format changed with 2.11BSD, so even the 2.10BSD binaries won't run. One advantage, though, of either the 2.11BSD or the V7 assembler is that it will run under apout.

Apout is a tool that the unix-1972 crowd over at tuhs created to run PDP-11 binaries on modern hardware. It doesn't implement all the system calls. The C compiler which forks other things won't run, for example. However, the assembler will. And the loader. And cpp. Why's the last one important? Well, if we have cpp, then we can assemble the 2.11BSD system call glue in libc.

The assembler is written in fairly low-level code. It calls half a dozen system calls, so this is easy, right? For the system calls, one needed only cpp and the assembler to create them. However, there's one other function it calls: signal. Signal used to be a system call when as was written. In 2BSD, Berkeley reworked how signals worked, so they created a compatibility shim written in C for the old way. That presents a problem... Getting the C compiler going was a lot of effort because it was so many passes and I'd have to string them all together by hand. My solution was to look at the sources and notice that it was just called to register an atexit function to cleanup tmp files when SIGINT was received. This is important for real, old-school PDP-11 hardware that measured the speed in hundreds of thousands of operations a second (or worse!). It would mean that ^C would clean up the temp files. But for bootstrapping? It's not really needed. So I created a .s file that was just '_signal: rts pc', which does nothing but satisfy linkage...

To make things simple, I used ld's partial link functionality to link all the .o's together to create a bootstrap.o. This took the place of libc. So I was able to bootstrap the assembler using the V7 as and ld binaries as well as the 2.11BSD cpp binary to pre-process the 2.11 sources. I did this twice, once for each pass of the assembler. I added the code to the script that I use to create the 2.11BSD (original) tree. This script took care of copying the results into the 2.11BSD tree. It was able to assemble itself, so on to the next step.

Now that I had the assembler bootstrapped, I could move on to the next things. Here we shift from the FreeBSD host that was creating the 2.11BSD (original) tree to a 2.11BSD pl 195 simh image that had a copy of this tree (which I'll call ur2.11 below to distinguish it from 2.11BSD pl 195 which I'll just call '195 below) mounted on /scratch. FYI: the 'ur' prefix means 'original' and it's often used in linguistics to describe the original version of something, now lost but reconstructed.

Bootstrapping ranlib (and to get there ld and nm)

So, one of the things you need is something called ranlib. It reads through a library and collects a table of all the symbols in that library and puts it in the first member of that archive. ld then uses that to pull in what it needs from the library. This eliminates the need to worry about cycles and other strange things. Normally, without a table of contents, ld will just make a single pass through the .a file, pulling in everything that's needed. When there's no cycles in the dependencies, this works great when you create the library with 'lorder *.o | tsort' so that it can be pulled in with one pass. If there are cycles, the library has to be pulled in multiple times to resolve them all.

libc, of course, has cycles. So, how do we fix that? Well, we need to build ranlib (since the newer ranlib uses a different table of contents format, because why would it be easy). To make matters worse, 2.11BSD changed the archive format to the portable archive format from the old PDP-11 format.

So, to build ranlib, we need libc and ld. For libc, we need nm because the lorder shell script uses it and I didn't want to hack the build process. Let's focus on the first two of those. In an ideal world, we could just build them on the '195 image. For once in this project, that's entirely possible, but with a caveat. The include files have changed, so I needed to build this on the 195 system, but using the ur2.11 includes (not the '195 ones, they had been rototilled in the 158-178 patch sequence for the new binary format). I needed to do this in the '195 system because it could create new binaries (but chrooted to the ur2.11 system could not). I was able to do this simply enough:
cd /scratch/usr/src/bin
cc -o ld -O -i ld.c -I/scratch/usr/include
cc -o nm -O -i nm.c -I/scratch/usr/include
 Now I had everything I needed to bootstrap ranlib... almost....

Drop into the chroot

As readers of my blog know, I recently did some search into chroot. The reason was this effort. I'd recalled reading that it was added into 4.2BSD, etc. So I went looking and found an interesting story (that I've already told).

Now you know why I was looking: the next step is to chroot into /scratch. Once we're there, we need to do a few things. First, let's copy things over:
chroot /scratch
cd usr/src/bin/as
cp as /bin
cp as2 /lib
rm as as2
cd ..
cp nm ld /bin
rm nm ld
cp /bin/true /usr/bin/ranlib
OK. That gives us a working assembler, loader and nm. What about cc? Don't we need to rebuild it? Turns out, no. It's already working, creating perfectly fine assembler. Since we just swapped out the assembler, we're good: it produces the new format. And the loader, it can combine them into binaries that will run (we're quite fortunate that the '195 loader can create binaries that work on ur2.11). What about ar(1)? Well, we don't have to bootstrap that either (at least not yet) since the format is the same, even if the program was imported from 4.3BSD in the 158-178 patch series. Finally, we avoid an extra step later by copying /bin/true to ranlib. This means the ranlib in the ur2.11 tree right now (which came from '195) won't create an entry in libc.a we have to delete later.

Building libc.a and crt0.o

So, next up, we need to rebuild libc and crt0.o. cc uses these to create working binaries, and we need cc to rebuild ranlib. Thankfully, it's relatively straight forward to rebuild libc and install it:
cd /usr/src/lib/libc
make clean
# Hack around make sometimes failing to descend on some runs
(cd pdp/compat-4.1; make)
(cd pdp; make)
make install
make clean
so now we've replaced the '195 libc.a with it's newer format binaries with ur2.11 libc.a with the proper for this version format. When building, you may have noticed tsort reported a cycle in the dependency graph. It's safe to ignore that for now, we'll work around it in a minute. Depending on dates of directories, you may need to build deep directories by hand because directories in the future aren't considered out of date so aren't rebuilt...

Building ranlib (for real)

Now we can build ranlib, and use it to add a table of contents to libc.a. We'll need to specify libc.a twice in order for it to resolve the circular dependency. When linking libraries w/o the ranlib table of contents, ld only makes one pass through the library. So, if we list it twice, it will get the rest of the dependencies when it makes a second pass through the library. Since all the other symbols are resolved, we don't wind up with two copies of anything.
cd /usr/src/usr.bin
cc -o ranlib -O -i ranlib.c -lc
cp ranlib /usr/bin
ranlib /lib/libc.a
So, now we have a sane libc.a and ranlib.

Finishing up the Bootstrapping

OK. We could go on from here and make a lot of progress. Along the way, though, we'll discover that there's some programs whose Makefile assumes certain things about ar, or want to exec the strip program, etc. So we'll build those now and install them to make for smoother sailing later. All the other dependencies are properly handled.
cd /usr/src/bin
make ar strip
cp ar strip /bin
And we're not quite done. install groks the binary format, so it has to be bootstrapped now before we use install -s as part of many make install targets:
cd /usr/src/usr.bin
make xinstall
cp xinstall /usr/bin/install 

Doing the Build

At this point, the simple way to build is to do the following
cd /usr/src
make clean
make all
make install
make clean
make all
make install
which builds everything twice. This is far from optimal, but will work. The things that fail the first time around, due to missing libraries and such, will succeed the second time through.

One could look in the sources and find there's another process, 'make build' which installs the includes (well, that's commented out, and that caused version skew between /usr/src/include and /usr/include), builds and installs libc, builds and installs the C compiler, rebuilds libc, rebuilds and reinstalls the C compiler, then builds and installs usr.lib before building and installing 'bin usr.bin etc ucb new games' directories. This works mostly OK. However, in our situation, this leaves a big hole: there's programs in /usr/src/usr.lib that need other libraries in /usr/src/usr.lib, so they fail to build in the make build scenario. Plus, I've had it fail in the second build of libc for reasons unknown (it just fails to descend into the pdp directory, which it had no trouble doing the first time).

So if you go and look at the bootstrap program, you'll see the following crazy dance that it does. Of course, it knows it's already built libc once (and it lacks the above workaround for libc, but the actual automation has it):
cd /usr/src
make clean
cd lib
for i in ccom cpp c2 libc ccom cpp c2; do
    (cd $i; make all install)
cd ../usr.lib
for i in lib[0-9A-Za-z]*; do
    (cd $i; make all install; make clean)
ln /usr/lib/libom.a /usr/lib/libm.a
cd ..
make all
make install
make clean
which is similar enough to 'make build' but avoids the holes in it and avoids having to build absolutely everything twice (though it does build libc and the C compiler 3 times total, which likely is overkill). The funky pattern for building libraries is because there's a lib.b that's installed (it's just a text file with what appears to be B code in it). The link for libm afterwards is to mimic what the make install target does in usr.lib since we're not using it and libom.a is used for libm.a on 2.11BSD. Since we remove all .a's in creating the root, we have to recreate this here.

In the end, we're left with a complete user land that we can then move to the next phase with. Once we have a kernel, we can rebuild the release tapes which I'll leave as a topic for another day. With the boot block rework, the disk label changes and the changing needs of the 2.11BSD community, rebuilding them for ur2.11 is somewhat different than 2.11BSD patch 469.

As a workaround for some build issues, I also needed to build a number of man pages so the program associated with them would be properly installed... Suffice to say I rebuilt all the man pages in the end as part of the bootstrap script, but they aren't strictly required to run the system.

Building the Kernel

Normally, one would re config the kernel and build it. However, in 2.11BSD as released, there were a number of hacks made to the kernel Makefile to get it to fit into memory. Normally, one would hack these things in /sys/conf/Make.sunix so configuring GENERIC wouldn't destroy any carefully worked out overlay, but that wasn't done initially. So, we have to be careful how we build.

Also, in the initial version, the root partition was hard coded into the kernel. There was a script call /GENALLSYS that would create all versions of the: rpunix, raunix, xpunix, hkunix, etc.When installing, one needs to know the proper one to use. So, putting that all together, we can just do this:

cd /usr/src/sys/GENERIC
make && make install && (cd / ; cp unix genunix; sh -x /GENALLSYS)

which builds all possible bootable kernels... 

Building all the Standalone Programs

When we built everything, a few things still weren't build: the boot loader, the autoconfig and boot program (which is different than the boot loader). One just needs to build in /sys/mdec, /sys/autoconfig and /sys/pdpstand:
cd /sys/mdec
make && make install && make clean
cd /sys/autoconfig
make && make install && make clean
cd /sys/pdpstand
make && make install && make clean
Once one has mdec installed, one needs to dd the blocks onto the disk to make it bootable. When I was bootstrapping this disk, I did it with the intention of making a bootable system. I had to add /usr to /etc/fstab too, but all the things I did might fill another blog entry...


Building entire systems is messy, and has always been messy. Unless you skipped to the conclusion, I suspect that you've already formed this opinion about the 2.11BSD build process.  I've managed to enshrine everything above into and to make things automated. Using this technique I've managed to build a ur2.11BSD boot disk, created boot tapes and installed from those tapes. Automation was key, though, to recording all the right steps in the right order.

by Warner Losh ( at August 04, 2020 03:41 PM

August 03, 2020

Warner Losh

Missing 2.11BSD patches

2.11BSD Missing Patches

While looking into some date anomalies in the final image (since I'd like to get the dates right) I discovered a number of source directories had dates slightly newer than the date in the announcement. This lead me to discover some missing patches in a couple of different places.

The Anomaly

I've automated the system generation, tape generation and installing from tapes to allow me to make small tweaks and get end to end testing. As part of this, after the system is installed, I'll do a test boot, similar to the following, as if I'd installed the system on April 10th, 1991 and booted it on April 15th. The boot looks something like this:
sim> boot rq

boot: 73Boot
: ra(0,0)unix

2.11 BSD UNIX #1: Fri Mar 15 15:48:55 PST 1991

phys mem  = 4186112
avail mem = 4008640
user mem  = 307200

Apr 10 13:50:01 init: configure system
ra 0 csr 172150 vector 154 attached
rl 0 csr 174400 vector 160 attached
tms 0 csr 174500 vector 260 attached
ts 0 csr 172520 vector 224 attached
xp 0 csr 176700 vector 254 attached
erase, kill ^U, intr ^C
# date 9104151234
date: can't write wtmp file.
Mon Apr 15 12:34:00 PDT 1991
# Fast boot ... skipping disk checks
/dev/ra0c on /usr: Device busy
checking quotas: done.
Assuming non-networking system ...
preserving editor files
clearing /tmp
standard daemons: update cron accounting.
starting lpd
starting local daemons:.
Mon Apr 15 12:34:01 PDT 1991

2.10 BSD UNIX ( (console)

I set the date in single user then bring it up to multiuser. In one of my tests, I found the following:
-rw-r--r--  1 imp  imp   9777 Aug 31  1991 alloc.c
-rw-r--r--  1 imp  imp   4817 Aug 31  1991 alloc11.c
-rw-r--r--  1 imp  imp  12474 Aug 31  1991 doprnt.c
-rw-r--r--  1 imp  imp   3299 Feb 23  1987 doprnt11.s
-rw-r--r--  1 imp  imp    831 Aug 31  1991 printf.c
-rw-r--r--  1 imp  imp  20446 Aug 31  1991 sh.c
-rw-r--r--  1 imp  imp   1771 Aug 31  1991 sh.char.c
which I thought was quite strange. There shouldn't be any files dated newer than the release in the tree. I know all files patched don't get the time set right, so I exclude those from my search (I plan on fixing that bug later). The above files (and others, it's just a short list) shouldn't be there. So I started looking...

The Diffs

Running diffs against 2.10.1 I discovered that the csh files were almost all the same. However, a typical diff looked like:

diff -ur root-2.10.1/usr/src/bin/csh/alloc.c root-2.11/usr/src/bin/csh/alloc.c
--- root-2.10.1/usr/src/bin/csh/alloc.c 1987-02-08 15:27:23.000000000 -0700
+++ root-2.11/usr/src/bin/csh/alloc.c   1991-08-31 01:03:00.000000000 -0600
@@ -4,10 +4,10 @@
  * specifies the terms and conditions for redistribution.

-#ifndef lint
+#if    !defined(lint) && defined(DOSCCS)
 /* From "@(#)malloc.c  5.5 (Berkeley) 2/25/86"; */
 static char *sccsid = "@(#)alloc.c     5.3 (Berkeley) 3/29/86";
-#endif not lint

  * malloc.c (Caltech) 2/21/82

which removed the SCCS IDs from the binary to save size. Other changes included introducing overlays for the first time. This indicated size issue. Let's take a look at what else was going on around the 31 Aug 91 in the patch stream. Looking, we find that this is just after patch 18 (which fixed a long vs int bug in test) as well as patch 17, which updated pcc. This sounds like a size hack by someone that had just updated the compiler, or was testing with pcc (the normal system compiler wasn't pcc, but the earlier Thompson compiler). Another of the changes also fixes an issue with character handling, which other patches have done to reduce the size of binaries that got too big.

So, in context, this change makes perfect sense. The only trouble is that it wasn't posted to comp.bugs.2bsd, nor did it make it into Steven Schultz's patch repo. And csh isn't the only troublesome one. There's issues in rn, games/warp and other updates to pcc as well.

The Catch Up Patch

There was a catchup kit that was issued officially as Patch 80 (though it omitted patch 79).  Looking in that patch kit we find this change! So it's in one that was intended. So what to do?  And it turns out there's 5 such patches (but only 4 of them made it into the patch kit... I'll talk about #5 in a minute).

I've decided to look at the dates of each of these patches and pretend they happened just after the patch whose date is closes (except the pcc patch which has to come after patch 17). I've updated my mk211bsd script to extract these from the catch up patch.

Oh, and there were a number of new programs added in the catchup patch as well. These must be deleted too, but I'd already noticed that and deleted them.

tftp changes

So, on May 4th, 1991 a patch to dd.c was posted to comp.bugs.2bsd. It's also included in the official archive as patch 1. The release announcement was dated March 14th, 1991. But there's tftp files dated May 15th, 1991. What's up with those? Turns out, this is another missed patch (but one that's assumed to be in place in the catch-up patch because it's not in there. Well, it's partially in the patches, partially in the scripts. It's an update of tftp and tftpd to a new version. It was posted to comp.bugs.2bsd on May 15th, 1991, but isn't in the official list of patches. So not only do we have to dig it out of the catch-up patches (from two different files), we also have to restore the old man pages from 2.10.1BSD, but in a different place, so this patch will be a patch + rm (which going backwards is patch + cp)

csh changes

As discussed above, these are various hacks to get the size of csh down.

warp changes

The changes here are around the config script used to generate files for the build. The changes use full path names, and cope with the new shadow password format changes.

pcc changes

These are the most extensive of the undocumented changes so far (about 100k worth). The thrust seems to be to make pcc work again on the pdp-11, though I've not looked closely at them yet. The only saving grace here is that it's just a patch.

rn changes

As part of the catchup, there's a number of minor patches to rn that were included in the catch up, but weren't formally published or occupy a number of their own.

My Work

So, how does this affect me? Well, it means that I need to understand the catch up patch a lot better. I had hoped to use it later as a cross-check against my work. I didn't anticipate that I'd be using it "sooner" as to get missing bits I'd found using other techniques. I've had to update mk211bsd to extract the bits, as well as creating a couple of hints files to help me undo the changes.

And when the time comes to reply all the patches, I'll need to take these anomalies into account as well. But that's a problem for future me.

I also have to complete the audit of the weird file dates. There's 63 of them right now, 29 in /usr/src still. Some of clearly old man pages that I can remove. Some are the result of running 'configure' or similar script (rn has 7 of these). Some are config files that change over time (like for the root name server). Some may be just left-over detritus of a running system. I need to see which ones fall into which categories and update accordingly. This may dovetail back into needing to bring them back to make sure I can march back to pl195 and get the same system. Since I started with ~1500 such anomalies, I think being down to 63 is quite good. And there's others elsewhere in the system... 

Current status

As you might guess, if I'm finding things like this, that means I'm getting closer. I've shared a lot of this on my @bsdimp twitter account, but now is a good time for a wrap up. Here's what's done currently:
  1. Script to undo all the patches, including helper 'hints' scripts, where possible from existing artifacts.
  2. Miss patches reconstructed and integrated into the build
  3. Automated installing 2.11BSDpl195 image
  4. Automated bootstrapping back from 195 -> 0. There's a number of interesting problems here that I'll blog about soon
  5. Building the 2.11BSD pl 0 tapes automatically
  6. Test installing the pl  0 system from the pl 0 tapes.
The missing bits include
  1. Getting the dates right (or failing that plausible) for the patched files
  2. Finishing the date audit and tracking all anomalies to ground.
  3. Cleaning up my helper scripts off the image
  4. Creating a github repo with all the patches in it
  5. Reproducing the build on a second system
  6. Getting the ownership right for some files (eg using the mtree hack to get the ownership and permissions right, generating it from the pl195 tape/image, etc)
  7. Getting dates right on /, right now restor(8) doesn't restore the date in one at a time mode, so these are all wrong.
  8. Fixing tmscp boot. It's broken. The tmscpboot.s, ported from tkboot.s, only existed for a short period of time and has been lost. My reconstruction has issues (it won't boot), and I've not delved into why.
  9. Creating automation to ensure that the 'catch up' kit will apply cleanly.
And of course, I need to figure out the best way to publish the artifacts when I think I'm done.

by Warner Losh ( at August 03, 2020 08:34 PM

July 30, 2020


Audio subsystem hardware internals

I wrote an introductory article on how the audio subsystem on SBCs work: CODECs, I2S, DTS, whole nine yards. WordPress editor didn’t seem to be a very convenient tool for this kind of write up so I gave asciidoc a try and so far liked it.

Link to the article:

by gonzo at July 30, 2020 07:13 AM

July 26, 2020


July 25, 2020

Adrian Chadd

RFI from crappy electronics, or "how's this sold in the US?"

I picked up a cheap charging cable for my Baofeng UV-9S. ( It .. well, it works.

But it messes up operating my radios! I heard super strong interference on my HF receiver and my VHF receivers.

So, let's take a look. I setup a little antenna in my shack. The baofeng was about 6ft away.

Here's DC to 120MHz. Those peaks to the right? Broadcast FM. The marker is at 28.5MHz.

Ok, let's plug in the baofeng and charger.

Ok, look at that noise. Ugh. That's unfun.

What about VHF? Let's look at that. 100-300MHz.

Ok, that's expected too. I think that's digital TV or something in there. Ok, now, let's plug in the charger, without it charging..

Whaaaaaaaaattttttt oh wait. Yeah, this is likely an unshielded buck converter and it's unloaded. Ok, let's load it up.

Whaaaaaa oh ok. Well that explains everything.

Let's pull it open:

Yup. A buck converter going from 5v to 9v; no shielding, no shielded power cable and no ground plane on the PCB. This is just amazing. The 3ft charge cable is basically an antenna. "Unintentional radiator" indeed.

So - even with a ferrite on the cable, it isn't quiet.

It's quiet at 28MHz now so I can operate on the 10m band with it charging, but this doesn't help at all at VHF.


by Adrian ( at July 25, 2020 02:24 AM

July 23, 2020


July 15, 2020

Adrian Chadd

Fixing up ath_rate_sample to actually work well with 11n

Way back in 2011 when I was working on FreeBSD's Atheros 802.11n support I needed to go and teach some rate control code about 802.11n MCS rates. (As a side note, the other FreeBSD wifi hackers and I at the time taught wlan_amrr - the AMRR rate control in net80211 - about basic MCS support too, and fixing that will be the subject of a later post.)

The initial hacks I did to ath_rate_sample made it kind of do MCS rates OK, but it certainly wasn't great. To understand why then and what I've done now, it's best to go for a little trip down journey lane - the initial sample rate control algorithm by John Bicket. You can find a copy of the paper he wrote here - .

Now, sample didn't try to optimise maximum throughput. Instead, it attempts to optimise for minimum airtime to get the job done, and also attempted to minimise the time spent sampling rates that had a low probability of working. Note this was all done circa 2005 - at the time the other popular rate control methods tried to maintain the highest PHY rate that met some basic success rate (eg packet loss, bit error rate, etc, etc.) The initial implementation in FreeBSD also included multiple packet size bins - 250 and 1600 bytes - to allow rate selection based on packet length.

However, it made some assumptions about rates that don't quite hold in the 802.11n MCS world. Notably, it didn't take the PHY bitrate into account when comparing rates. It mostly assumed that going up in rate code - except between CCK and OFDM rates - meant it was faster. Now, this is true for 11b, 11g and 11a rates - again except when you transition between 11b and 11g rates - but this definitely doesn't hold true in the 802.11n MCS rate world. Yes, between MCS0 to MCS7 the PHY bitrate goes up, but then MCS8 is MCS0 times two streams, and MCS16 is MCS0 times three streams.

So my 2011/2012 just did the minimum hacks to choose /some/ MCS rates. It didn't take the length of aggregates into account; it just used the length of the first packet in the aggregate. Very suboptimal, but it got MCS rates going.

Now fast-forward to 2020. This works fine if you're close to the other end, but it's very terrible if you're at the fringes of acceptable behaviour. My access points at home are not well located and thus I'm reproducing this behaviour very often - so I decided to fix it.

First up - packet length.  I had to do some work to figure out how much data was in the transmit queue for a given node and TID. (Think "QoS category.") The amount of data in the queue wasn't good enough - chances are we couldn't transmit all of it because of 802.11 state (block-ack window, management traffic, sleep state, etc.) So I needed a quick way to query the amount of traffic in the queue taking into account 802.11 state. That .. ended up being a walk of each packet in the software queue for that node/TID list until we hit our limit, but for now that'll do.

So then I can call ath_rate_lookup() to get a rate control schedule knowing how long a packet may be. But depending up on the rate it returns, the amount of data that may be transmitted could be less - there's a 4ms limit on 802.11n aggregates, so at lower MCS rates you end up only sending much smaller frames (like 3KB at the slowest rate.) So I needed a way to return how many bytes to form an aggregate for as well as the rate. That informed the A-MPDU formation routine how much data it could queue in the aggregate for the given rate.

I also stored that away to use when completing the transmit, just to line things up OK.

Ok, so now I'm able to make rate control decisions based on how much data needs to be sent. ath_rate_sample still only worked with 250 and 1600 byte packets. So, I extended that out to 65536 bytes in mostly-powers-of-two values.  This worked pretty well right out of the box, but the rate control process was still making pretty trash decisions.

The next bit is all "statistics". The decisions that ath_rate_sample makes depend upon accurate estimations of how long packet transmissions took. I found that a lot of the logic was drastically over-compensating for failures by accounting a LOT more time for failures at each attempted rate, rather than only accounting how much time failed at that rate. Here's two examples:
  • If a rate failed, then all the other rates would get failure accounted for the whole length of the transmission to that point. I changed it to only account for failures for that rate - so if three out of four rates failed, each failed rate would only get their individual time accounted to that rate, rather than everything.
  • Short (RTS/CTS) and long (no-ACK) retries were being accounted incorrectly. If 10 short retries occured, then the maximum failed transmission for that rate can't be 10 times the "it happened" long retry style packet accounting. It's a short retry; the only thing that could differ is the rate that RTS/CTS is being exchanged at. Penalising rates because of bursts of short failures was incorrect and I changed that accounting.
There are a few more, but you can look at the change log / change history for sys/dev/ath/ath_rate/sample/ to see.

By and large, I pretty accurately nailed making sure that failed transmit rates account for THEIR failures, not the failures of other rates in the schedule. It was super important for MCS rates because mis-accounting failures across the 24-odd rates you can choose in 3-stream transmit can have pretty disasterous effects on throughput - channel conditions change super frequently and you don't want to penalise things for far, far too long and it take a lot of subsequent successful samples just to try using that rate again.

So that was the statistics side done.

Next up - choices.

Choices was a bit less problematic to fix. My earlier hacks mostly just made it possible to choose MCS rates but it didn't really take into account their behaviour. When you're doing 11a/11g OFDM rates, you know that you go in lock-step from 6, 12, 18, 24, 36, 48, 54MB, and if a rate starts failing the higher rate will likely also fail. However, MCS rates are different - the difference between MCS0 (1/2 BPSK, 1 stream) and MCS8 (1/2 BPSK, 2 streams) is only a couple dB of extra required signal strength. So given a rate, you want to sample at MCS rates around it but also ACROSS streams. So I mostly had to make sure that if I was at say MCS3, I'd also test MCS2 and MCS4, but I'd also test MCS10/11/12 (the 2-stream versions of MCS2/3/4) and maybe MCS18/19/20 for 3-stream. I also shouldn't really bother testing too high up the MCS chain if I'm at a lower MCS rate - there's no guarantee that MCS7 is going to work (5/6 QAM64 - fast but needs a pretty clean channel) if I'm doing ok at MCS2. So, I just went to make sure that the sampling logic wouldn't try all the MCS rates when operating at a given MCS rate. It works pretty well - sampling will try a couple MCS rates either side to see if the average transmit time for that rate is higher or lower, and then it'll bump it up or down to minimise said average transmit time.

However, the one gotcha - packet loss and A-MPDU.

ath_rate_sample was based on single frames, not aggregates. So the concept of average transmit time assumed that the data either got there or it didn't. But, with 802.11n A-MPDU aggregation we can have the higher rates succeed at transmitting SOMETHING - meaning that the average transmit time and long retry failure counts look great - but most of the frames in the A-MPDU are dropped. That means low throughput and more actual airtime being used.

When I did this initial work in 2011/2012 I noted this, so I kept an EWMA of the packet loss both of single frames and aggregates. I wouldn't choose higher rates whose EWMA was outside of a couple percent of the current best rate. It didn't matter how good it looked at the long retry view - if only 5% of sub-frames were ACKed, I needed a quick way to dismiss that. The EWMA logic worked pretty well there and only needed a bit of tweaking.

A few things stand out after testing:

  • For shorter packets, it doesn't matter if it chooses the one, two or three stream rate; the bulk of the airtime is overhead and not data. Ie, the difference between MCS4, MCS12 and MCS20 is any extra training symbols for 2/3 stream rates and a few dB extra signal strength required. So, typically it will alternate between them as they all behave roughly the same.
  • For longer packets, the bulk of the airtime starts becoming data, so it begins to choose rates that are obviously providing lower airtime and higher packet success EWMA. MCS12 is the choice for up to 4096 byte aggregates; the higher rates start rapidly dropping off in EWMA. This could be due to a variety of things, but importantly it's optimising things pretty well.
There's a bunch of future work to tidy this all up some more but it can wait.

by Adrian ( at July 15, 2020 05:45 PM

June 04, 2020


yubikey-agent on FreeBSD

Some time ago Filippo Valsorda wrote yubikey-agent, seamless SSH agent for YubiKeys. I really like YubiKeys and worked on the FreeBSD support for U2F in Chromium and pyu2f, getting yubikey-agent ported looked like an interesting project. It took some hacking to make it work but overall it wasn’t hard. Following is the roadmap on how to get it set up on FreeBSD. The actual details depend on your system (as you will see)

The first step is to set up middleware for accessing smart cards (YubiKey implements CCID smart-card protocol). The pcsc-lite package provides a daemon and a library for clients to communicate with the daemon. ccid is a plugin for pcsc-lite that implements the actual CCID protocol over USB. devd rules are to make the daemon re-scan USB devices on hotplug

sudo pkg install ccid pcsc-lite
sudo mkdir -p /usr/local/etc/devd
sudo tee /usr/local/etc/devd/pcscd.conf << __EOF__
attach 100 {
        device-name "ugen[0-9]+";
        action "/usr/local/sbin/pcscd -H";
detach 100 {
        device-name "ugen[0-9]+";
        action "/usr/local/sbin/pcscd -H";

sudo service devd restart
sudo sysrc pcscd_enable="YES
sudo service pcscd start

go and git are build requirements for the app. go get command is required because FreeBSD support was only recently merged into piv-go and the latest release, referenced in go.mod, still does not have it. go install command installs the app in ~/go/bin/. When new version of piv-go is released and yubikey-agent switches to using it, all these commands can be replaced with single go get command.

sudo pkg install go git
git clone
cd yubikey-agent
go get
go build
go install

The binary is in ~/go/bin/ directory, you can add it to your $PATH to type less.

The next step is setting up a Yubikey, it’s well documented on the official site. One caveat though is if PIN length is less than 6 chars the setup fails with the somewhat confusing message: “‼ The default PIN did not work.”

The actual usage of yubikey-agent depends on your setup. First of all, yubikey-agent is a “eventually GUI” app. At some point, when entering PIN is required it starts pinentry command which task is to present user with a dialog entry, get PIN, and pass it to the app. There are multiple pinentry flavors, with different front-ends: TTY, Qt, GTK. gopass module used in yubikey-agent does not work with plain TTY backend, and requires pinentry to be a GUI app. On Debian/Ubuntu users can switch between flavors and make /usr/bin/pinentry point to either Qt5 or GTK version, but in FreeBSD /usr/local/bin/pinentry is always TTY one. I worked around it by installing pinentry-qt5 package and making symlink.

sudo pkg install pinentry-qt5
sudo ln -s /usr/local/bin/pinentry-qt5 /usr/local/bin/pinentry

If you need /usr/local/bin/pinentry to be TTY version for some other stuff, there may be a problem. How to work around this depends on your requirements. I don’t have a ready recipe.

Because yubikey-agent is “eventually GUI” you can either start it in .xsession or .xinitrc files or start it some other way with DISPLAY env variable set. Other than that official documentation is a good source of information on how to use it.

by gonzo at June 04, 2020 07:03 AM

May 31, 2020

Colin Percival

Some new FreeBSD/EC2 features: EFS automount and ebsnvme-id

As my regular readers will be aware, I've been working on and gradually improving FreeBSD/EC2 for many years. Recently I've added two new features, which are available in the weekly HEAD and 12-STABLE snapshots and will appear in releases starting from 12.2-RELEASE.

May 31, 2020 03:50 AM

May 22, 2020

Colin Percival

My new FreeBSD Laptop: Dell Latitude 7390

As a FreeBSD developer, I make a point of using FreeBSD whenever I can — including on the desktop. I've been running FreeBSD on laptops since 2004; this hasn't always been easy, but over the years I've found that the situation has generally been improving. One of the things we still lack is adequate documentation, however — so I'm writing this to provide an example for users and also Google bait in case anyone runs into some of the problems I had to address.

A few months ago, after my System76 Galago Pro had its second experience with a dead/swelling batery, I decided that it was time to replace it. On February 15th, I ordered a Dell Latitude 7390.

May 22, 2020 01:20 AM

March 31, 2020

Erwin Lansing

Enjoy the view

Even if we cannot enjoy the view from the air, we can enjoy it from the ground. Here’s to keeping our spirits at least 30.000 ft high.

KLM KL1128 leaving Copenhagen (CPH)
on March 31, 2020

The post Enjoy the view appeared first on Droso.

by erwin at March 31, 2020 03:12 PM

March 19, 2020

Alexander Leidinger

Fighting the Coronavirus with FreeBSD (Folding@Home)

Photo by Fusion Medical Animation on Unsplash

Here is a quick HOWTO for those which want to provide some FreeBSD based compute resources to help finding vaccines. I have not made a port out of this and do not know yet if I get the time to make one. If someone wants to make a port, go ahead, do not wait for me.

UPDATE 2020-03-22: 0mp@ made a port out of this, it is in “biology/linux-foldingathome”.

  • Download the linux RPM of the Folding@Home client (this covers fahclient only).
  • Enable the linuxulator (kernel moduls and linux_base (first part of chapter 10.2) is enough).
  • Make sure linprocfs/linsysfs are mounted in /compat/linux/{proc|sys}.
  • cd /compat/linux
  • tar -xf /path/to/fahclient....rpm
  • add the “fahclient” user (give it a real home directory)
  • make sure there is no /compat/linux/dev or alternatively mount devfs there
  • mkdir /compat/linux/etc/fahclient
  • cp /compat/linux/usr/share/doc/fahclient/sample-config.xml /compat/linux/etc/fahclient/config.xml
  • chown -R fahclient /compat/linux/etc/fahclient
  • edit /compat/linux/fahclient/config.xml: modify user (mandatory) / team (optional: FreeBSD team is 11743) / passkey (optional) as appropriate (if you want to control the client remotely, you need to modify some more parts, but somehow the client “loses” a filedescriptor and stops working as it should if you do that on FreeBSD)
  • If you have the home directories of the users as no-exec (e.g. seperate ZFS datasets with exec=off): make sure the home directory of the fahclient user has exec permissions enabled
  • cd ~fahclient (important! it tries to write to the current work directory when you start it)
  • Start it: /usr/sbin/daemon /compat/linux/usr/bin/FAHClient /compat/linux/etc/fahclient/config.xml --run-as fahclient --pid-file=/var/run/ >/dev/null 2>&1

Per default it will now pick up some SARS-CoV‑2 (COVID-19) related folding tasks. There are some more config options (e.g. how much of the system resources are used). Please refer to the official Folding@Home site for more information about that. Be also aware that there is a big rise in compute resources donated to Folding@Home, so the pool of available work units may be empty from time to time, but they are working on adding more work units. Be patient.

Send to Kindle


by netchild at March 19, 2020 08:47 AM

January 27, 2019

Alexander Leidinger

Strategic thinking, or what I think what we need to do to keep FreeBSD relevant

Since I participate in the FreeBSD project there are from time to time some voices which say FreeBSD is dead, Linux is the way to go. Most of the time those voices are trolls, or people which do not really know what FreeBSD has to offer. Sometimes those voices wear blinders, they only see their own little world (were Linux just works fine) and do not see the big picture (like e.g. competition stimulates business, …) or even dare to look what FreeBSD has to offer.

Sometimes those voices raise a valid concern, and it is up to the FreeBSD project to filter out what would be beneficial. Recently there were some mails on the FreeBSD lists in the sense of “What about going into direction X?”. Some people just had the opinion that we should stay were we are. In my opinion this is similarly bad to blindly saying FreeBSD is dead and following the masses. It would mean stagnation. We should not hold people back in exploring new / different directions. Someone wants to write a kernel module in (a subset of) C++ or in Rust… well, go ahead, give it a try, we can put it into the Ports Collection and let people get experience with it.

This discussion on the mailinglists also triggered some kind of “where do we see us in the next years” / strategic thinking reflection. What I present here, is my very own opinion about things we in the FreeBSD project should look at, to stay relevant in the long term. To be able to put that into scope, I need to clarify what “relevant” means in this case.

FreeBSD is currently used by companies like Netflix, NetApp, Cisco, Juniper, and many others as a base for products or services. It is also used by end-users as a work-horse (e.g. mailservers, webservers, …). Staying relevant means in this context, to provide something which the user base is interested in to use and which makes it more easy / fast for the user base to deliver whatever they want or need to deliver than with another kind of system. And this in terms of time to market of a solution (time to deliver a service like a web-/mail-/whatever-server or product), and in terms of performance (which not only means speed, but also security and reliability and …) of the solution.

I have categorized the list of items I think are important into (new) code/features, docs, polishing and project infrastructure. Links in the following usually point to documentation/HOWTOs/experiences for/with FreeBSD, and not to the canonical entry points of the projects or technologies. In a few cases the links point to an explanation in the wikipedia or to the website of the topic in question.


The virtualization train (OpenStack, OpenNebula, oVirt, CloudStack, Kubernetes, Docker, Podman, …) is running on full speed. The marked is as big/important, that solution providers even do joint ventures on crossing borders between each others, e.g. VMware is opening up to integrate their solution with solutions from Amazon/Azure/Google. The underlying infrastructure is getting more and more unimportant, as long as the services which shall be run perform as needed. Ease of use and time to market are the key-drivers (the last little piece of performance is mostly important for companies which go to the “edge” (both meanings intended in a non-exclusive-or way) like Netflix for their FreeBSD based CDN). FreeBSD is not really participating in this world. Yes, we had jails way before anyone else out there had something smilar, and some even do not have that right now. But if you are realistic, FreeBSD does not play a major role here. You can do nice things with jails and bhyve, but you have to do it “by hand” (ezjail, iocage and such are improvements on the ease of use side, but that is not enough as this is still limited on a host centric view). The world has moved on to administering a datacenter (to avoid the buzzwords “cloud” or “private-cloud”) with a single-click. In my opinion we would need to port several of the initialy mentioned cloud/container management solutions to FreeBSD and have them able to handle their work via jails and/or bhyve. If FreeBSD is not able to serve as a building block in this big picture, we will fall off the edge in this particular IT-area in the long run.

With all the ready-made containers available in the internet, we should improve our linuxolator. Our kernel-support for this is limited to a 2.6.32-ish ABI version (it is less than 2.6.32, more like 2.6.16, we are missing epoll and inotify support, among others, but this is the lowest version glibc in the CentOS 7 based linux_base port is able to run on… and glibc checks the version number). We need to catch-up to a more recent version if we want to be able to run those ready-made linux containers without issue (we can put a linux system into a jail and start that). If someone would like to work on that, a good start would be to run the Linux Test Project tests via the linuxulator and start fixing bugs. The last time I did that was in 2007 and about 16% of the test cases failed back then. It would be also quite nice if we could integrate those linuxulator tests into the FreeBSD CI. With improvements in the linuxulator and the above mentioned virtualization support, we should be able to run more/those linux-images … ehrm, sorry, docker/kubernetes/…-containers within the linuxulator.

Finish the work regarding kerberos in base. Cy Schubert is/was working on this. I do not know the status of this, but the “fixing 800 ports” part in his mail from May 2018 looks like some more helping hands would be beneficial. This would bring us to a better starting point for a more seamless integration (some ports need “the other” kerberos).

We have one port (as far as I was able to determine… the exact amount does not really matter) in terms of SDN – net/openvswitch – but the documentation of this … leaves room for improvement (kernel support / netmap support and functionality / FreeBSD specific HOWTO). As part of the virtualisation of everything we (yes: we – as part of the FreeBSD handbook, see docs category below for more on this) need to provide this info so that FreeBSD is able to participate in this area. We should also have a look at porting some more SDN software, e.g OpenContrail now tungstenfabric (there is an old contrail porting project), OpenStack Neutron, OpenDaylight, … so that users have a choice, respectively FreeBSD can be integrated into existing heterogeneous environments. 

Sensors (temperature, voltage, fans, …), a topic with history. Short: in the Google Summer of Code 2007 code was produced, committed, and then removed again due to a dispute. My personal understanding (very simplified) is “remove everything because some of the data handled by this framework shall not be handled by this framework” (instead of e.g. “remove sensor X, this data shall not be handled in this way”), and “remove everything as this does not handle sensors of type X which are not used in servers but in enterprise class >99% non-IT-related sensors”. Nothing better has shown up since then. If I look at Window, VMware, Solaris and Linux, I can query sensors on my mainboard/chassis/disks/whatever (yes, I am mixing some apples with oranges here), plot them in monitoring systems, and get alarms. In FreeBSD we fail on this topic (actually multiple topics) which I consider to be something basic and mandatory. I do not suggest that we commit the Google Summer of Code 2007 code. I suggest to have a look at what makes sense to do here. Take the existing code and commit it, or improve on this code outside the tree and then commit it, or write something new. In the end it does not matter (for an user) which way it is handled, as long as we have something which users can use in the end. It surely makes sense to have an OS-provided framework of registering sensors in a central place (it would surely be nice if you could get the temp/fan values of your graphics card… ooops… sorry… AI/HPC accelerator together with other similar hardware data in your hardware).

To continue playing well (not only) in the high-availability area, we should also have a look at getting an implementation of MPTCP (Mutlipath TCP) into the tree. Apple (and others) is already using it since 2013 with good benefits (most probably not only for Siri users). There exists some code for FreeBSD, but it is far from usable and it does not look like there is progress since 2016. We say we have the power to serve, but with the cloudification of the recent years, all users expect that everything is always-on and never fails, and being able to provide the server side of this client-server related technology for those people which have such high demands is necessary to not fall behind (do not let us rest on our laurels).

SecureBoot needs also some helping hands. At some point operating systems which do not support it will not be considered by companies anymore.

Another item we should have a look at is to provide means to write kernel code in different languages. Not in the base system, but at least in ports. If someone wants to write a kernel module in  C++ or Rust, why not? It offers possibilities to explore new areas. There are even reports of experiences with different languages. It does not fit your needs? Well, ignore it and continue writing kernel code in C, but let other people which want to use a screwdriver instead of a hammer do what they want, they will either learn that they should have used a hammer, or can report about benefits about the screwdriver.


I think we can improve our end-user docs to the next level. The base system is already well covered (we can surely find some features which we could document), but an user does not use FreeBSD to use FreeBSD. An user surely has a goal in mind which requires to setup some kind of service (mail server, web server, display server (desktop system), …). While one could argue that it is the 3rd party project which needs to document how to run their software on FreeBSD, I think we need to do our share here too. There are a lot of HOWTOs for Linux, and then you have to find some tips and tricks to make something work (better) on FreeBSD. What I have in mind here is that we should document how to make FreeBSD participate in a Windows Active Directory environment, or in an LDAP environment (as a client), improve the Samba part with FreeBSD specific parts (like how to make Samba use ZFS snapshots for Windows Shadow Copies), configuration management tools and so on. I do not talk about providing in-depth docs about the 3rd party software, but little HOWTOs with FreeBSD specific parts / tips and tricks, and a reference to the 3rd party docs. People come to us for real-world needs and if we provide them with a head-start of the most common items (e.g. also covering nginx or whatever and not only apache httpd) and then guide them to further docs will improve the value of our handbook even more for end-users (specially for newcomers, but also for experienced FreeBSD users which out of a sudden now need to do something which they never did before…).

We should also review our docs. The handbook lists e.g. procmail (just an example…). With procmail not being mainained anymore since a long time and known vulnerabilities we should replace the info there with info about maildrop (or any suitable replacement). Careful review may also find similar items which need some care.

One more item I have in mind in terms of docs for user is the restructuring of some parts. Now the world is more thinking in terms of XaaS (“something as a service”) we should also have a “cloud” section (going beyond of what we have in terms of virtualization already) in out handbook. We can put there items like the existing description of virtualisation items, but also should put there new items like glusterfs or object storage or the hopefully upcoming possibility of how to setup OpenStack/kubernetes/… on FreeBSD. This goes into the same direction as the first docs-item in terms of provide more documentation how to achieve goals of our users. 

In my opinion we are also lacking on the developer-documentation side. Yes, we have man pages which describe the official API (in most cases). Where I see room for improvement is the source code documentation. Something like doxygen (or whatever the tool of the day is – which one does not really matter, any kind of extractable-from-source documentation is better than no extractable documentation) is already used in several places in our source (search for it via: egrep ‑R ‘\\(brief|file)’ /usr/src/) and we have already some infrastructure to extract and render (HTML / PDF) them. The more accessible / easy it is to start development in FreeBSD, the more attractive it will be (additional to the existing benefits) to people / companies to dive in. The best examples about documenting source code in our code I have found so far is the isci and ocs_fc device code.


Polishing something in the topic of staying relevant? Yes! It is the details which matter. If people have 2 options with roughly the same features (nothing missing what you need, same price), which one do they take, the one which has everything consistent and well integrated, or the one with some quirks you can circumvent with a little bit of work on their side?

We have some nice features, but we are not using it to the extend possible. One of the items which come to my mind is DTrace. The area which I think needs polishing is to add more probes, and to have some kind of probe-convention about common topics. For example I/O related naming convention (maybe area specific, like storage I/O and network I/O) and covering all drivers to comply. We should also look into making it more accessible by providing more easy interfaces (no matter if text based (thanks to Devin Teske for dwatch, more of this magic please…), web based, or whatever) to make it really easy (= start a command or click around and you get the result for a specific set of probes/conditions/…). Some examples are statemaps, flamegraphs and most prominently the Oracle/Sun ZFS Storage Analytics to give you an idea what is possible with DTrace and and how to make it accessible to people without knowledge about the kernel internals and programming.

Some polishing in the ports collection would be to revisit the defaults options for ports with options. The target here should be to have consistent default settings (e.g. server software should not depend upon X11 by default (directly or indirectly), most people should not need to build the port with non-default options). One could argue that it is the responsability of the maintainer of the port, and to some extend it is, but we do not have guidelines which help here. So a little team of people to review all ports (and modify them if necessary) and come up with guidelines and examples would be great.

Additionally we should come up with meta-ports for specific use case, e.g. webserver (different flavours… apache/nginx/…), database (different flavours, with some useful tools like mytop or mysqltuner or similar) and then even reference them in the handbook (this goes along with my suggestion above to document real-world use cases instead of “only” the OS itself).

Recently ‑current has seen some low level performance improvements (via ifuncs, …). We should continue this and even extend it to revise default settings / values (non auto-tuned and even auto-tuned ones). I think it would be beneficial in the long run if we target more current hardware (without losing the ability to run on older hardware) and for those values which can not be auto-tuned provide some way of down-tuning (e.g. in a failsafe-boot setting in the loader or in documented settings for rc.conf or wherever those defaults can be changed).

Project infrastructure

We have a CI (continuous integration) system, but it is not very prominently placed. Just recently it gained some more attention from the developer side and we even got the first status report about it (nice! visibility helps making it a part of the community effort). There is a FreeBSD-wiki page about the status and the future ideas, but it was not updated since several months. There is also a page which talks in more details about using it for performance testing, which is something people have talked about since years but never became available (and is not today).

I think we need to improve here. The goals I think which are important are to get various testing, sanitizing and fuzzing technologies integrated into our CI. On the config repository I have not found any integration of e.g. the corresponding clang technologies (fuzzing, ASAN, UBSAN, MSAN (still experimental, so maybe not to target before other mature technologies)) or any other such technology.

We should also make our CI more public/visible (build status linked somewhere on, nag people more about issues found by it, have some docs how to add new tests (maybe from ports), so that more people can help in extend what we automatically test (e.g. how could I integrate the LTP (Linux Test Project) tests to test our linuxulator? This requires the download of a linux dist port, the LTP itself, and then to run the tests). There are a lot of nice ideas floating around, but I have the impression we are lacking some helping hands to get various items integrated.


Various items I talked above are not sexy. Those are typically not the things people do just for fun. Those are typically items people get paid for. It would be nice if some of the companies which benefit from FreeBSD would be so nice to lend a helping hand for one or another item. Maybe the FreeBSD Foundation has some contacts they could ask about this?

It could also be that for some of the items I mentioned here there is more ongoing that I know of. This means then that the corresponding work could be made more known on the mailinglists. When it is more known, maybe someone wants to provide a helping hand.

Send to Kindle


by netchild at January 27, 2019 10:18 PM

October 22, 2018

Dag-Erling Smørgrav

DNS over TLS in FreeBSD 12

With the arrival of OpenSSL 1.1.1, an upgraded Unbound, and some changes to the setup and init scripts, FreeBSD 12.0, currently in beta, now supports DNS over TLS out of the box.

DNS over TLS is just what it sounds like: DNS over TCP, but wrapped in a TLS session. It encrypts your requests and the server’s replies, and optionally allows you to verify the identity of the server. The advantages are protection against eavesdropping and manipulation of your DNS traffic; the drawbacks are a slight performance degradation and potential firewall traversal issues, as it runs over a non-standard port (TCP port 853) which may be blocked on some networks. Let’s take a look at how to set it up.

Basic setup

As a simple test case, let’s set up our 12.0-ALPHA10 VM to use Cloudflare’s DNS service:

# uname -r
# cat >/etc/rc.conf.d/local_unbound <<EOF
# service local_unbound start
Performing initial setup.
/var/unbound/forward.conf created
/var/unbound/lan-zones.conf created
/var/unbound/control.conf created
/var/unbound/unbound.conf created
/etc/resolvconf.conf not modified
Original /etc/resolv.conf saved as /var/backups/resolv.conf.20181021.192629
Starting local_unbound.
Waiting for nameserver to start... good
# host is an alias for has address has IPv6 address 2610:1c1:1:606c::50:15 mail is handled by 0 .

Note that this is not a configuration you want to run in production—we will come back to this later.


The downside of DNS over TLS is the performance hit of the TCP and TLS session setup and teardown. We demonstrate this by flushing our cache and (rather crudely) measuring a cache miss and a cache hit:

# local-unbound-control reload
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.553 total
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.005 total

Compare this to querying our router, a puny Soekris net5501 running Unbound 1.8.1 on FreeBSD 11.1-RELEASE:

# time host gw >x
host gw > x 0.00s user 0.00s system 0% cpu 0.232 total
# time host >x
host gw > x 0.00s user 0.00s system 0% cpu 0.008 total

or to querying Cloudflare directly over UDP:

# time host >x      
host > x 0.00s user 0.00s system 0% cpu 0.272 total
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.013 total

(Cloudflare uses anycast routing, so it is not so unreasonable to see a cache miss during off-peak hours.)

This clearly shows the advantage of running a local caching resolver—it absorbs the cost of DNSSEC and TLS. And speaking of DNSSEC, we can separate that cost from that of TLS by reconfiguring our server without the latter:

# cat >/etc/rc.conf.d/local_unbound <<EOF
# service local_unbound setup
Performing initial setup.
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.205328
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
Original /var/unbound/unbound.conf saved as /var/backups/unbound.conf.20181021.205328
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound start
Starting local_unbound.
Waiting for nameserver to start... good
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.080 total
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.004 total

So does TLS add nearly half a second to every cache miss? Not quite, fortunately—in our previous tests, our first query was not only a cache miss but also the first query after a restart or a cache flush, resulting in a complete load and validation of the entire path from the name we queried to the root. The difference between a first and second cache miss is quite noticeable:

# time host >x 
host > x 0.00s user 0.00s system 0% cpu 0.546 total
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.004 total
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.168 total
# time host >x
host > x 0.00s user 0.00s system 0% cpu 0.004 total

Revisiting our configuration

Remember when I said that you shouldn’t run the sample configuration in production, and that I’d get back to it later? This is later.

The problem with our first configuration is that while it encrypts our DNS traffic, it does not verify the identity of the server. Our ISP could be routing all traffic to to its own servers, logging it, and selling the information to the highest bidder. We need to tell Unbound to validate the server certificate, but there’s a catch: Unbound only knows the IP addresses of its forwarders, not their names. We have to provide it with names that will match the x509 certificates used by the servers we want to use. Let’s double-check the certificate:

# :| openssl s_client -connect |& openssl x509 -noout -text |& grep DNS
DNS:*, IP Address:, IP Address:,, IP Address:2606:4700:4700:0:0:0:0:1111, IP Address:2606:4700:4700:0:0:0:0:1001

This matches Cloudflare’s documentation, so let’s update our configuration:

# cat >/etc/rc.conf.d/local_unbound <<EOF
# service local_unbound setup
Performing initial setup.
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.212519
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
/var/unbound/unbound.conf not modified
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound restart
Stopping local_unbound.
Starting local_unbound.
Waiting for nameserver to start... good
# host is an alias for has address has IPv6 address 2610:1c1:1:606c::50:15 mail is handled by 0 .

How can we confirm that Unbound actually validates the certificate? Well, we can run Unbound in debug mode (/usr/sbin/unbound -dd -vvv) and read the debugging output… or we can confirm that it fails when given a name that does not match the certificate:

# perl -p -i -e 's/cloudflare/cloudfire/g' /etc/rc.conf.d/local_unbound
# service local_unbound setup
Performing initial setup.
Original /var/unbound/forward.conf saved as /var/backups/forward.conf.20181021.215808
/var/unbound/lan-zones.conf not modified
/var/unbound/control.conf not modified
/var/unbound/unbound.conf not modified
/etc/resolvconf.conf not modified
/etc/resolv.conf not modified
# service local_unbound restart
Stopping local_unbound.
Waiting for PIDS: 33977.
Starting local_unbound.
Waiting for nameserver to start... good
# host
Host not found: 2(SERVFAIL)

But is this really a failure to validate the certificate? Actually, no. When provided with a server name, Unbound will pass it to the server during the TLS handshake, and the server will reject the handshake if that name does not match any of its certificates. To truly verify that Unbound validates the server certificate, we have to confirm that it fails when it cannot do so. For instance, we can remove the root certificate used to sign the DNS server’s certificate from the test system’s trust store. Note that we cannot simply remove the trust store entirely, as Unbound will refuse to start if the trust store is missing or empty.

While we’re talking about trust stores, I should point out that you currently must have ca_root_nss installed for DNS over TLS to work. However, 12.0-RELEASE will ship with a pre-installed copy.


We’ve seen how to set up Unbound—specifically, the local_unbound service in FreeBSD 12.0—to use DNS over TLS instead of plain UDP or TCP, using Cloudflare’s public DNS service as an example. We’ve looked at the performance impact, and at how to ensure (and verify) that Unbound validates the server certificate to prevent man-in-the-middle attacks.

The question that remains is whether it is all worth it. There is undeniably a performance hit, though this may improve with TLS 1.3. More importantly, there are currently very few DNS-over-TLS providers—only one, really, since Quad9 filter their responses—and you have to weigh the advantage of encrypting your DNS traffic against the disadvantage of sending it all to a single organization. I can’t answer that question for you, but I can tell you that the parameters are evolving quickly, and if your answer is negative today, it may not remain so for long. More providers will appear. Performance will improve with TLS 1.3 and QUIC. Within a year or two, running DNS over TLS may very well become the rule rather than the experimental exception.

by Dag-Erling Smørgrav at October 22, 2018 09:36 AM

April 09, 2018

Dag-Erling Smørgrav

Twenty years

Yesterday was the twentieth anniversary of my FreeBSD commit bit, and tomorrow will be the twentieth anniversary of my first commit. I figured I’d split the difference and write a few words about it today.

My level of engagement with the FreeBSD project has varied greatly over the twenty years I’ve been a committer. There have been times when I worked on it full-time, and times when I did not touch it for months. The last few years, health issues and life events have consumed my time and sapped my energy, and my contributions have come in bursts. Commit statistics do not tell the whole story, though: even when not working on FreeBSD directly, I have worked on side projects which, like OpenPAM, may one day find their way into FreeBSD.

My contributions have not been limited to code. I was the project’s first Bugmeister; I’ve served on the Security Team for a long time, and have been both Security Officer and Deputy Security Officer; I managed the last four Core Team elections and am doing so again this year.

In return, the project has taught me much about programming and software engineering. It taught me code hygiene and the importance of clarity over cleverness; it taught me the ins and outs of revision control; it taught me the importance of good documentation, and how to write it; and it taught me good release engineering practices.

Last but not least, it has provided me with the opportunity to work with some of the best people in the field. I have the privilege today to count several of them among my friends.

For better or worse, the FreeBSD project has shaped my career and my life. It set me on the path to information security in general and IAA in particular, and opened many a door for me. I would not be where I am now without it.

I won’t pretend to be able to tell the future. I don’t know how long I will remain active in the FreeBSD project and community. It could be another twenty years; or it could be ten, or five, or less. All I know is that FreeBSD and I still have things to teach each other, and I don’t intend to call it quits any time soon.


by Dag-Erling Smørgrav at April 09, 2018 08:35 PM

February 05, 2018

Remko Lodder

Reponse zones in BIND (RPZ/Blocking unwanted traffic).

A while ago, my dear colleague Mattijs came with an interesting option in BIND. Response zones. One can create custom "zones" and enforce a policy on that.

I never worked with it before, so I had no clue at all what to expect from it. Mattijs told me how to configure it (see below for an example) and offered to slave his RPZ policy-domains.

All of a sudden I was no longer getting a lot of ADS/SPAM and other things. It was filtered. Wow!

His RPZ zones were custom made and based on PiHole, where PiHole adds hosts to the local "hosts" file and sends it to (your local machine), which prevents it to reach the actual server at all, RPZ policies are much stronger and more dynamic.

RPZ policies offer the use of "redirecting" queries. What do I mean with that? well you can force a ADVERTISEMENT (AD for short) site / domain to the RPZ policy and return a NXDOMAIN. It no longer exists for the end-user. But you can also CNAME it to a domain/host you own and then add a webserver to that host and tell the user query'ing the page: "The site you are trying to reach had been pro-actively blocked by the DNS software. This is an automated action and an automated response. If you feel that this is not appropriate, please let us know on <mail link>", or something like that.

Once I noticed that and saw the value, I immediately saw the benefit for companies and most likely schools and home people. Mattijs had a busy time at work and I was recovering from health issues, so I had "plenty" of time to investigate and read on this. The RPZ policies where not updated a lot and caused some problems for my ereaders for example ( was used by them, see another post on this website for being grumpy about that). And I wanted to learn more about it. So what did I do?

Yes, I wrote my own parser. In perl. I wrote a "rpz-generator" (its actually called like that). I added the sources Mattijs used and generated my own files. They are rather huge, since I blocked ads, malware, fraud, exploits, windows stuff and various other things (gambling, fakenews, and stuff like that).

I also included some whitelists, because msfctinc was added to the lists and it made my ereaders go beserk, and we play a few games here and there which uses some advertisement sites, so we wanted to exempt them as well. It's better to know which ones they are and selectively allow them, then having traffic to every data collector out there.

This works rather well. I do not get a lot of complaints that things are not working. I do see a lot of queries going to "banned" sites everyday. So it is doing something .The most obvious one is that search results on google, not always are clickable. The ones that have those [ADV] sites, are blocked because they are advertising google sponsored sites, and they are on the list.. and google-analytics etc. It doesn't cause much harm to our internet surfing or use experience, with the exception of the ADV sites I just mentioned. My wife sometimes wants to click on those because she searches for something that happends to be on that list, but apart from that we are doing just fine.

One thing though, I wrote my setup and this article with my setup using "NXDOMAIN" which just gives back "site does not exist" messages. I want to make my script more smart by making it a selectable, so that some categories are CNAMED to a filtering domain and webpage, and some are NXDOMAIN'ed. If someone has experience with that, please show me some idea's and how that looks like and whether your end-users can do something with it or not. I think schools will be happy to present a block-page instead of NXdomain'ing some sites 🙂

Acknowledgements: Mattijs for teaching and showing me RPZ, ISC for placing RPZ in NAMED, and for having such excellent documentation to RPZ. The perl developers for having such a great tool around, and the various sites I use to get the blocklists from. Thank you all!

If you want to know more about the tool, please contact me and we can share whatever information is available 🙂

by Remko Lodder at February 05, 2018 11:09 PM

November 25, 2017

Erwin Lansing

120. Red ale

5,5kg Maris Otter
500g Crystal 60L
450g Munich I
100g Chocolate malt

Mash for 75 minutes at 65°C

30g Cascade @ 60 min.
30g Centennial @60 min.
30g Cascade @ 10 min.
30g Centennial @ 10 min.

Bottled at January 7, 2018 with 150g table sugar

White Labs WLP001 California ale yeast
OG: 1.052
FG: 1.006
ABV: 6,0%

The post 120. Red ale appeared first on Droso.

by erwin at November 25, 2017 05:34 PM

August 29, 2017

Remko Lodder

FreeBSD: Using Open-Xchange on FreeBSD

If you go looking for a usable webmail application, then you might end up with Open-Xchange (OX for short). Some larger ISP's are using OX as their webmail application for customers. It has a multitude of options available, using multiple email accounts, caldav/carddav included (not externally (yet?)) etc. There are commercial options available for these ISP's, but also for smaller resellers etc.

But, there is also the community edition available. Which is the installation you can run for free on your machine(s). It does not have some of the fancy modules that large setups need and require, and some updates might follow a bit later which are more directly delivered to paying customers, but it is very complete and usable.

I decided to setup this for my private clients who like to use a webmail client to access their email. At first I ran this on a VM using Bhyve on FreeBSD. The VM ran on CentOS6 and had the necessary bits installed for the OX setup (see: I modified the files I needed to change to get this going, and there, it just worked. But, running on a VM, with ofcourse limited CPU and Memory power assigned (There is always a cap) and it being emulated, I was not very happy with it. I needed to maintain an additional installation and update it, while I have this perfectly fine FreeBSD server instead. (Note that I am not against using bhyve at all, it works very well, but I wanted to reduce my maintenance base a bit :-)).

So a few days ago I considered just moving the stuff over to the FreeBSD host instead. And actually it was rather trivial to do with the working setup on CentOS.

At this moment I do not see an easy way to get the source/components directly from within FreeBSD. I have asked OX for help on this, so that we can perhaps get this sorted out and perhaps even make a Port/pkg out of this for use with FreeBSD.

The required host changes and software installation

The first thing that I did was to create a zfs dataset for /opt. The software is normally installed there, and in this case I wanted to have a contained location which I can snapshot, delete, etc, without affecting much of the normal system. I copied over the /opt/open-xchange directory from my CentOS installation. I looked at the installation on CentOS and noticed that it used a specific user 'open-xchange', which I created on my FreeBSD host. I changed the files to be owned by this user. Getting a process listing on the CentOS machine also revealed that it needed Java/JDK. So I installed the openjdk8 pkg (''pkg install openjdk8''). The setup did not yet start, there were errors about /bin/bash missing. Obviously that required installing bash (''pkg install bash'') and you can go with two ways, you can alter every shebang (#!) to match /usr/local/bin/bash (or better yet #!/usr/bin/env bash), or you can symlink /usr/local/bin/bash to /bin/bash, which is what I did (I asked OX to make it more portable by using the env variant instead).

The /var/log/open-xchange directory does not normally exist, so I created that and made sure that ''open-xchange'' could write to that. (mkdir /var/log/open-xchange && chown open-xchange /var/log/open-xchange).

I was able to startup the /opt/open-xchange/sbin/open-xchange process with that. I could not yet easily reach it, on the CentOS installation there are two files in the Apache configuration that needed some attention on my FreeBSD host. The Apache include files: ox.conf and proxy_http.conf will give away hints about what to change. In my case I needed to do the redirect on the Vhost that runs OX (RedirectMatch ^/$ /appsuite/) and make sure the /var/www/html/appsuite directory is copied over from the CentOS installation as well. You can stick it in any location, as long as you can reach it with your webuser and Alias it to the proper directory and setup directory access).

Apache configuration (Reverse proxy mode)

The proxy_http.conf file is more interesting, it includes the reverse proxy settings to be able to connect to the java instance of OX and service your clients. I needed to add a few modules in Apache so that it could work, I already had several proxy modules enabled for different reasons, so the list below can probably be trimmed a bit to the exact modules needed, but since this works for me, I might as well just show you;

LoadModule slotmem_shm_module libexec/apache24/
LoadModule deflate_module libexec/apache24/
LoadModule expires_module libexec/apache24/
LoadModule proxy_module libexec/apache24/
LoadModule proxy_connect_module libexec/apache24/
LoadModule proxy_http_module libexec/apache24/
LoadModule proxy_scgi_module libexec/apache24/
LoadModule proxy_wstunnel_module libexec/apache24/
LoadModule proxy_ajp_module libexec/apache24/
LoadModule proxy_balancer_module libexec/apache24/
LoadModule lbmethod_byrequests_module libexec/apache24/
LoadModule lbmethod_bytraffic_module libexec/apache24/
LoadModule lbmethod_bybusyness_module libexec/apache24/

After that it was running fine for me. My users can login to the application and the local directory's are being used instead of the VM which ran it first. If you notice previous documentation on this subject, you will notice that there are more third party packages needed at that time. It could easily be that there are more modules needed than that I wrote about. My setup was not clean, the host already runs several websites (one of them being this one) and ofcourse support packages were already installed.

Updating is currently NOT possible. The CentOS installation requires running ''yum update'' periodically, but that is obviously not possible on FreeBSD. The packages used within CentOS are not directly usable for FreeBSD. I have asked OX to provide the various Community base and optional modules as .tar.gz files (raw) so that we can fetch them and install them on the proper location(s). As long as the .js/.jar files etc are all there and the scripts are modified to start, it will just work. I have not (yet) created a startup script for this yet. For the moment I will just start the VM and see whether there are updates and copy them over instead. Since I did not need to do additional changing on the main host, it is a very easy and straight forward process in this case.


There is no support for OX on FreeBSD. Ofcourse I would like to see at least some support to promote my favorite OS more, but that is a financial situation. It might not cost a lot to deliver the .tar.gz files so that we can package them and spread the usage of OX on more installations (and thus perhaps add revenue for OX as commercial installation), but it will cost FTE's to support more then that. If you see a commercial opportunity, please let them know so that this might be more and more realistic.

The documentation written above is just how I have setup the installation and I wanted to share it with you. I do not offer support on it, but ofcourse I am willing to answer questions you might have about the setup etc. I did not include the vhost configuration in it's entirely, if that is a popular request, I will add it to this post.

Open Questions to OX:

So as mentioned I have questioned OX for some choices:

  • Please use a more portable path for the Bash shell (#!/usr/bin/env bash)
  • Please allow the use of a different localbase (/usr/local/open-xchange for example)
  • Please allow FreeBSD packagers to fetch a "clean" .tar.gz, so that we can package this for OX and distribute it for our end-users.
  • Unrelated to the post above: Please allow the usage of external caldav/carddav providers


I have found another thing that I needed to change. I needed to use gsed (Gnu-sed) instead of FreeBSD-sed so that the listuser scripts work. Linux does that a bit differently but if you replace sed with gsed those scripts will work fine.

I have not yet got some feedback from OX.

by Remko Lodder at August 29, 2017 07:48 AM

April 11, 2017

Eric Anholt

This week in vc4 (2017-04-10): dmabuf fencing, meson

The big project for the last two weeks has been developing dmabuf fencing support for vc4.  Without dmabuf fences, when passing buffers between devices the user needs to manually wait for the job to finish on one (say, camera snapshot) before letting the other device get started (accumulating GL commands to texture from the camera snapshot).  That means leaving both devices idle for a moment while the CPU accumulates the command stream for the consumer, but the bigger pain is that it requires that the end user manage the synchronization.

With dma-buf fencing in the kernel, a "reservation object" generated by the dma-buf exporter tracks the fences of the various devices using the shared object, and then the device trivers get to look at that list and wait on on each others' fences when using it.

So far, I've got my reservations and fences being exported from vc4, so that pl111 display can wait for vc4 to be done before actually putting a new pageflip up on the screen.  I haven't quite hooked up the other direction, for camera capture into vc4 display or GL texturing (I don't have a testcase for this, as the current camera driver doesn't expose dmabufs), but it shouldn't be hard.

On the meson front, rendercheck is now converted to meson upstream.  I've made more progress on the X Server:  Xorg is now building, and even successfully executes Xorg -pogo with the previous modesetting driver in place.  The new modesetting driver is failing mysteriously.  With a build hack I got from the meson folks and some work from ajax, the sdksyms script I complained about in my last post isn't used at all on the meson build.  And, best of all, the meson devs have written the code needed for us to not even need the build hack I'm using.

It's so nice to be using a build system that's an actual living software project.

by anholt at April 11, 2017 12:48 AM

March 27, 2017

Eric Anholt

This week in vc4 (2017-03-27): Upstream PRs, more CMA, meson

Last week I sent pull requests for bcm2835 changes for 4.12.  We've got some DT updates for HDMI audio, DSI, and SDHOST, and defconfig changes to enable SDHOST.  The DT changes to actually enable SDHOST (and get wifi working on Pi3) won't land until 4.13.

I also wrote a patch to enable using more than 256MB of CMA memory (and not require any particular alignment).  The 256MB limit was due to a hardware bug: the binner's memory allocations get dereferenced with their top 4 bits set to the top 4 bits of the tile state data array's address.  Given that tile state allocations happen after CL setup (while the binner is running and throwing overflow interrupts), there was no way to guarantee that we could find overflow memory with the top bits matching.

The new solution, suggested by someone from the set top box group, is to allocate a single 16MB to 32MB buffer at HW init time, and return all of those types of allocations out of it, since it turns out you don't need much to complete rendering of any given scene.  I've been mulling over the details of a solution for a while, and finally wrote and tested the patch I wanted (tricky parts included freeing the memory when the hardware was idle, and how to track the lifetimes of the sub-allocations).  Results look good, and I'll be submitting it this week.

However, I spent most of the week on converting the X Server over to meson

Meson is a delightful new build system (based around Ninja on Linux) that massively speeds up builds, while also being portable to Windows (unlike autotools generally).  If you've ever tried to build the X stack on Raspberry Pi, you know that autotools is painfully slow.  It's also been the limiting factor for me in debugging my scripts for CI for the X Server -- something we'd really like to be doing as we hack on glamor or do refactors in the core.

So far all I've landed in this project is code deletion, as I find build options that aren't hooked up to anything, or code that isn't hooked up to build options.  This itself will speed up our builds, and ajax has been working in parallel on deleting a bunch of code that makes the build messier than it needs to be.  I've also submitted patches for rendercheck converting to meson (as a demo of what the conversion looks like), and I have Xephyr, Xvfb, Xdmx, and Xwayland building in the X Server with meson.

So far the only stumbling block for the meson conversion of the X Server is the sdksyms.c file.  It's the ugliest part of the build -- running the C preprocessor on a generated .c that #includes a bunch of .h tiles, then running the output of that through awk and trying to parse C using regular expressions.  This is, as you might guess, somewhat fragile.

My hope for a solution to this is to just quit generating sdksyms.c entirely.  Using ELF sections, we can convince the linker to not garbage collect symbols that it thinks are unused.  Then we get to just decorate symbols with XORG_EXPORT or XORG_EXPORT_VAR (unfortunately have to have separate sections for RO vs RW contents), and Xorg will have the correct set of symbols exported.  I started on a branch for this, ajax got it actually building, and now we just need to bash the prototypes so that the same set of symbols are exported before/after the conversion.

by anholt at March 27, 2017 10:43 PM

October 25, 2016

Murray Stokely

FreeBSD on Intel NUCs

I've been away from FreeBSD for a few years but I wanted some more functionality on my home network that I was able to configure with my Synology NAS and router. Specifically, I wanted:

  • a configurable caching name server that would serve up authoritative private names on my LAN and also validates responses with DNSSEC.
  • a more configurable DHCP server so I could make the server assign specific IPs to specific MAC addresses.
  • more compute power for transcoding videos for Plex.

Running FreeBSD 11 on an Intel NUC seemed like an ideal solution to keep my closet tidy. As of this week, $406.63 on Amazon buys a last generation i3 Intel NUC mini PC (NUC5I3RYH), with 8GB of RAM and 128GB of SSD storage. This was the first model I tried since I found reports of others using this with FreeBSD online, but I was also able to get it working on the newer generation i5 based NUC6i5SYK with 16GB of RAM and 256GB of SSD. The major issue with these NUCs is that the Intel wireless driver is not supported in FreeBSD. I am not doing anything graphical with these boxes so I don't know how well the graphics work, but they are great little network compute nodes.


I downloaded the FreeBSD 11 memory stick images, and was pleased to see that the device booted fine off the memory stick without any BIOS configuration required. However, my installation failed trying to mount root ("Mounting from ufs:/dev/ufs/FreeBSD_Install failed with error 19."). Installation from an external USB DVD drive and over the network with PXE both proved more successful at getting me into bsdinstaller to complete the installation.

I partitioned the 128GB SSD device with 8GB of swap and the rest for the root partition (UFS, Journaled and Soft Updates). After installation I edited /etc/fstab to add a tmpfs(5) mount for /tmp. The dmesg output for this host is available in a Gist on Github.

Warren Block's article on SSD on FreeBSD and the various chapters of the FreeBSD Handbook were helpful. There were a couple of tools that were also useful in probing the performance of the SSD with my FreeBSD workload:

  • The smartctl tool in the sysutils/smartmontools package allows one to read detailed diagnostic information from the SSD, including wear patterns.
  • The basic benchmark built into diskinfo -t reports that the SSD is transferring 503-510MB/second.
But how well does it perform in practice?

Rough Benchmarks

This post isn't meant to report a comprehensive suite of FreeBSD benchmarks, but I did run some basic tests to understand how suitable these low power NUCs perform in practice. To start with, I downloaded the 11-stable source from Subversion and measured the build times to understand performance of the new system. All builds were done with a minimal 2 line make.conf:


Build Speed

Build CommandEnvironmentReal Times
make -j4 buildkernel/usr/src and /usr/obj on SSD10.06 minutes
make -j4 buildkernel/usr/src on SSD, /usr/obj on tmpfs9.65 minutes
make -j4 buildworld/usr/src and /usr/obj on SSD1.27 hours
make buildworld/usr/src and /urs/obj on SSD3.76 hours


In addition to the build times, I also wanted to look more directly at the performance reading from flash and reading from the NFS mounted home directories on my 4-drive NAS. I first tried Bonnie++, but then ran into a 13-year old bug in the NFS client of FreeBSD. After switching to Bonnie, I was able to gather some reasonable numbers. I had to use really large file sizes for the random write test to eliminate most of the caching that was artificially inflating the results. For those that haven't seen it, Brendan Gregg's excellent blog post highlights some of the issues of file system benchmarks like Bonnie.

Average of 3 bonnie runs with 40GB block size
ConfigurationRandom I/OBlock InputBlock Output
Seeks/SecCPU UtilizationReads/secCPU UtilizationWrites/secCPU Utilization

The block input rates from my bonnie benchmarks on the SSD were within 5% of the value provided by the much quick and dirtier diskinfo -t test.

Running Bonnie with less than 40GB file size yielded unreliable benchmarks due to caching at the VM layer. The following boxplot shows the random seek performance during 3 runs each at 24, 32, and 40GB file sizes. Performance starts to even off at this level but with smaller file sizes the reported random seek performance is much higher.

Open Issues

As mentioned earlier, I liked the performance I got with running FreeBSD on a 2015-era i3 NUC5I3RYH so much that I bought a newer, more powerful second device for my network. The 2016-era i5 NUC 6i5SYK is also running great. There are just a few minor issues I've encountered so far:

  • There is no FreeBSD driver for the Intel Wireless chip included with this NUC. Code for other platforms exists but has not been ported to FreeBSD.
  • The memory stick booting issue described in the installation section. It is not clear if it didn't like my USB stick for some reason, or the port I was plugging into, or if additional boot parameters would have solved the issue. Documentation and/or code needs to be updated to make this clearer.
  • Similarly, the PXE Install instructions were a bit scattered. The PXE section of the Handbook isn't specifically targetting new manual installations into bsdinstall. There are a few extra things you can run into that aren't documented well or could be streamlined.
  • Graphics / X11 are outside of the scope of my needs. The NUCs have VESA mounts so you can easily tuck them behind an LCD monitor, but it is not clear to me how well they perform in that role.

by Murray ( at October 25, 2016 03:27 AM

April 07, 2016

FreeBSD Foundation

Introducing a New Website and Logo for the Foundation

The FreeBSD Foundation is pleased to announce the debut of our new logo and website, signaling the ongoing evolution of the Foundation identity, and ability to better serve the FreeBSD Project. Our new logo was designed to not only reflect the established and professional nature of our organization, but also to represent the link between the Project and the Foundation, and our commitment to community, collaboration, and the advancement of FreeBSD.

We did not make this decision lightly.  We are proud of the Beastie in the Business Suit and the history he encompasses. That is why you’ll still see him make an appearance on occasion. However, as the Foundation’s reach and objectives continue to expand, we must ensure our identity reflects who we are today, and where we are going in the future. From spotlighting companies who support and use FreeBSD, to making it easier to learn how to get involved, spread the word about, and work within the Project, the new site has been designed to better showcase, not only how we support the Project, but also the impact FreeBSD has on the world. The launch today marks the end of Phase I of our Website Development Project. Please stay tuned as we continue to add enhancements to the site.

We are also in the process of updating all our collateral, marketing literature, stationery, etc with the new logo. If you have used the FreeBSD Foundation logo in any of your marketing materials, please assist us in updating them. New Logo Guidelines will be available soon. In the meantime, if you are in the process of producing some new literature, and you would like to use the new Foundation logo, please contact our marketing department to get the new artwork.

Please note: we've moved the blog to the new site. See it here.

by Anne Dickison ( at April 07, 2016 04:40 PM

February 26, 2016

FreeBSD Foundation

FreeBSD and ZFS

ZFS has been making headlines lately, so it seems like the right time to talk about the longstanding relationship between FreeBSD and ZFS.

For nearly seven years, FreeBSD has included a production quality ZFS implementation, making it one of the key features of the FreeBSD operating system. ZFS is a combined file system and volume manager. Decoupling physical media from logical volumes allows free space to be efficiently shared between all of the file systems. ZFS introduced unprecedented data integrity and reliability guarantees to storage on FreeBSD. ZFS supports varying levels of redundancy for tolerance of hardware failures and includes cryptographic checksums on all data to guard against corruption.

Allan Jude, VP of Operations at ScaleEngine and coauthor of FreeBSD Mastery: ZFS, said “We started using ZFS in 2011 because we needed to safely store a huge quantity of video for our customers. FreeBSD was, and still is, the best platform for deploying ZFS in production. We now store more than a petabyte of video using ZFS, and use ZFS Boot Environments on all of our servers.”

So why does FreeBSD include ZFS and contribute to its continued development? FreeBSD community members understand the need for continued development work as technologies evolve. OpenZFS is the truly open source successor to the ZFS project and the FreeBSD Project has participated in OpenZFS since its founding in 2013. FreeBSD developers and those from Delphix, Nexenta, Joyent, the ZFS on Linux project, and the Illumos project work together to continue improving OpenZFS.

FreeBSD’s unique open source infrastructure, copyfree license, and engaged community support the integration of a variety of free software components, including OpenZFS. FreeBSD makes an excellent operating system for servers and end users, and it provides a foundation for many open source projects and commercial products.

We're happy that ZFS is available in FreeBSD as a fully integrated, first class file system and wish to thank all of those who have contributed to it over the years.

by Anne Dickison ( at February 26, 2016 03:23 PM

February 20, 2016

Joseph Koshy

ELF Toolchain v0.7.1

I am pleased to announce the availability of version 0.7.1 of the software being developed by the ElfToolChain project.

This release offers:
  • Better support of the DWARF4 format.
  • Support for more machine architectures.
  • Many bug fixes and improvements.
The release also contains experimental code for:
  • A library handling the Portable Executable (PE) format.
  • A link editor.
The release may be downloaded from SourceForge:
Detailed release notes are available at the URL mentioned above.

Many thanks to the project's supporters for their contributions to the project.

by Joseph Koshy ( at February 20, 2016 12:06 PM

January 25, 2015

Giorgios Keramidas

Some Useful RCIRC Snippets

I have started using rcirc as my main IRC client for a while now, and I really like the simplicity of its configuration. All of my important IRC options now fit in a couple of screens of text.

All the rcirc configuration options are wrapped in an eval-after-load form, to make sure that rcirc settings are there when I need them, but they do not normally cause delays during the startup of all Emacs instances I may spawn:

(eval-after-load "rcirc"


     (message "rcirc has been configured.")))

The “rcirc-setup-forms” are then separated in three clearly separated sections:

  • Generic rcirc configuration
  • A hook for setting up nice defaults in rcirc buffers
  • Custom rcirc commands/aliases

Only the first set of options is really required. Rcirc can still function as an IRC client without the rest of them. The rest is there mostly for convenience, and to avoid typing the same setup commands more than once.

The generic options I have set locally are just a handful of settings to set my name and nickname, to enable logging, to let rcirc authenticate to NickServ, and to tweak a few UI details. All this fits nicely in 21 lines of elisp:

;; Identification for IRC server connections
(setq rcirc-default-user-name "keramida"
      rcirc-default-nick      "keramida"
      rcirc-default-full-name "Giorgos Keramidas")

;; Enable automatic authentication with rcirc-authinfo keys.
(setq rcirc-auto-authenticate-flag t)

;; Enable logging support by default.
(setq rcirc-log-flag      t
      rcirc-log-directory (expand-file-name "irclogs" (getenv "HOME")))

;; Passwords for auto-identifying to nickserv and bitlbee.
(setq rcirc-authinfo '(("freenode"  nickserv "keramida"   "********")
                       ("grnet"     nickserv "keramida"   "********")))

;; Some UI options which I like better than the defaults.
(rcirc-track-minor-mode 1)
(setq rcirc-prompt      "»» "
      rcirc-time-format "%H:%M "
      rcirc-fill-flag   nil)

The next section of my rcirc setup is a small hook function which tweaks rcirc settings separately for each buffer (both channel buffers and private-message buffers):

(defun keramida/rcirc-mode-setup ()
  "Sets things up for channel and query buffers spawned by rcirc."
  ;; rcirc-omit-mode always *toggles*, so we first 'disable' it
  ;; and then let the function toggle it *and* set things up.
  (setq rcirc-omit-mode nil)
  (set (make-local-variable 'scroll-conservatively) 8192))

(add-hook 'rcirc-mode-hook 'keramida/rcirc-mode-setup)

Finally, the largest section of them all contains definitions for some custom commands and short-hand aliases for stuff I use all the time. First come a few handy aliases for talking to ChanServ, NickServ and MemoServ. Instead of typing /quote nickserv help foo, it’s nice to be able to just type /ns help foo. This is exactly what the following three tiny forms enable, by letting rcirc know that “/cs”, “/ms” and “/ns” are valid commands and passing-along any arguments to the appropriate IRC command:

;; Handy aliases for talking to ChanServ, MemoServ and NickServ.

(defun-rcirc-command cs (arg)
  "Send a private message to the ChanServ service."
  (rcirc-send-string process (concat "CHANSERV " arg)))

(defun-rcirc-command ms (arg)
  "Send a private message to the MemoServ service."
  (rcirc-send-string process (concat "MEMOSERV " arg)))

(defun-rcirc-command ns (arg)
  "Send a private message to the NickServ service."
  (rcirc-send-string process (concat "NICKSERV " arg)))

Next comes a nifty little /join replacement which can join multiple channels at once, as long as their names are separated by spaces, commas or semicolons. To make its code more readable, it’s split into 3 little functions: rcirc-trim-string removes leading and trailing whitespace from a string, rcirc-normalize-channel-name prepends “#” to a string if it doesn’t have one already, and finally rcirc-cmd-j uses the first two functions to do the interesting bits:

(defun rcirc-trim-string (string)
  "Trim leading and trailing whitespace from a string."
  (replace-regexp-in-string "^[[:space:]]*\\|[[:space:]]*$" "" string))

(defun rcirc-normalize-channel-name (name)
  "Normalize an IRC channel name. Trim surrounding
whitespace, and if it doesn't start with a ?# character, prepend
one ourselves."
  (let ((trimmed (rcirc-trim-string name)))
    (if (= ?# (aref trimmed 0))
      (concat "#" trimmed))))

;; /j CHANNEL[{ ,;}CHANNEL{ ,;}CHANNEL] - join multiple channels at once
(defun-rcirc-command j (arg)
  "Short-hand for joining a channel by typing /J channel,channel2,channel,...

Spaces, commas and semicolons are treated as channel name
separators, so that all the following are equivalent commands at
the rcirc prompt:

    /j demo;foo;test
    /j demo,foo,test
    /j demo foo test"
  (let* ((channels (mapcar 'rcirc-normalize-channel-name
                           (split-string (rcirc-trim-string arg) " ,;"))))
    (rcirc-join-channels process channels)))

The last short-hand command lets me type /wii NICK to get “extended” whois information for a nickname, which usually includes idle times too:

;; /WII nickname -> /WHOIS nickname nickname
(defun-rcirc-command wii (arg)
  "Show extended WHOIS information for one or more nicknames."
  (dolist (nickname (split-string arg " ,"))
    (rcirc-send-string process (concat "WHOIS " nickname " " nickname))))

With that, my rcirc setup is complete (at least in the sense that I can use it to chat with my IRC friends). There are no fancy bells and whistles like DCC file transfers, or fancy color parsing, and similar things, but I don’t need all that. I just need a simple, fast, pretty IRC client, and that’s exactly what I have now.

by keramida at January 25, 2015 08:41 AM

January 07, 2015

Murray Stokely

AsiaBSDCon 2014 Videos Posted (6 years of BSDConferences on YouTube)

Sato-san has once created a playlist of videos from AsiaBSDCon. There were 20 videos from the conference held March 15-16, 2014 and papers can be found here. Congrats to the organizers for running another successful conference in Tokyo. A full list of videos is included below. Six years ago when I first created this channel videos longer than 10 minutes couldn't normally be uploaded to YouTube and we had to create a special partner channel for the content. It is great to see how the availability of technical video content about FreeBSD has grown in the last six years.

by Murray ( at January 07, 2015 11:22 PM

December 26, 2013

Giorgios Keramidas

Profiling is Our Friend

I recently wrote a tiny demo program to demonstrate to co-workers how one can build a cache with age-based expiration of its entries, using purely immutable Scala collections. The core of the cache was something like 25-30 lines of Scala code like this:

class FooCache(maxAgeMillis: Long) {
  def now: Long = System.currentTimeMillis

  case class CacheEntry(number: Long, value: Long,
                        birthTime: Long) {
    def age: Long = now - birthTime

  lazy val cache: AtomicReference[Hashmap[Long, CacheEntry]] =
    new AtomicReference(HashMap[Long, CacheEntry]]())

  def values: Hashmap[Long, CacheEntry] =
    cache.get.filter{ (key, entry) =>
      entry.age <= maxAgeMillis }

  def get(number: Long): Long = {
    values.find{ case (key, entry) =>
      key == number && entry.age <= maxAgeMillis
    } match {
      case Some((key, entry)) =>
        entry.value                // cache hit
      case _ =>
        val entry = CacheEntry(number, compute(number), now)
        cache.set(values + (number -> entry))

  def compute(number: Long): Long =
    { /* Some long-running computation based on 'number' */ }

The main idea here is that we keep an atomically updated reference to an immutable HashMap. Every time we look for entries in the HashMap we check if (entry.age <= maxAgeMillis), to skip over entries which are already too old to be of any use. Then on cache insertion time we go through the ‘values’ function which excludes all cache entries which have already expired.

Note how the cache itself is not ‘immutable’. We are just using an immutable HashMap collection to store it. This means that Scala can do all sorts of optimizations when multiple threads want to iterate through all the entries of the cache looking for something they want. But there’s an interesting performance bug in this code too…

It’s relatively easy to spot once you know what you are looking for, but did you already catch it? I didn’t. At least not the first time I wrote this code. But I did notice something was ‘odd’ when I started doing lookups from multiple threads and looked at the performance stats of the program in a profiler. YourKit showed the following for this version of the caching code:

JVM Profile #1

See how CPU usage hovers around 60% and we are doing a hefty bunch of garbage collections every second? The profiler quickly led me to line 17 of the code pasted above, where I am going through ‘values’ when looking up cache entries.

Almost 94% of the CPU time of the program was spent inside the .values() function. The profiling report included this part:

|                           Name                            | Time   | Time |
|                                                           | (ms)   | (%)  |
| demo.caching                                              | 62.084 | 99 % |
| +-- d.caching.Numbers.check(long)                         | 62.084 | 99 % |
|   +-- d.caching.FooCacheModule$FooCache.check(long)       | 62.084 | 99 % |
|     +---d.caching.FooCacheModule$FooCache.values()        | 58.740 | 94 % |
|     +---scala.collection.AbstractIterable.find(Function1) |  3.215 |  5 % |

We are spending far too much time expiring cache entries. This is easy to understand why with a second look at the code of the get() function: every cache lookup does old entry expiration and then searches for a matching cache entry.

The way cache-entry expiration works with an immutable HashMap as the underlying cache entry store is that values() iterates over the entire cache HashMap, and builds a new HashMap containing only the cache entries which have not expired. This is bound to take a lot of procesisng power, and it’s also what’s causing the creation of all those ‘new’ objects we are garbage collecting every second!

Do we really need to construct a new cache HashMap every time we do a cache lookup? Of course not… We can just filter the entries while we are traversing the cache.

Changing line 17 from values.find{} to cache.get.find{} does not do cache-entry expiration at the time of every single lookup, and now our cache lookup speed is not limited by how fast we can construct new CacheEntry objects, link them to a HashMap and garbage-collect the old ones. Running the new code through YourKit once more showed an immensely better utilization profile for the 8 cores of my laptop’s CPU:

JVM Profile #2

Now we are not spending a bunch of time constructing throw-away objects, and garbage collector activity has dropped by a huge fraction. We can also make much more effective use of the available CPU cores for doing actual cache lookups, instead of busy work!

This was instantly reflected at the metrics I was collecting for the actual demo code. Before the change, the code was doing almost 6000 cache lookups per second:

-- Timers -------------------------------
             count = 4528121
         mean rate = 5872.91 calls/second
     1-minute rate = 5839.87 calls/second
     5-minute rate = 6053.27 calls/second
    15-minute rate = 6648.47 calls/second
               min = 0.29 milliseconds
               max = 10.25 milliseconds
              mean = 1.34 milliseconds
            stddev = 1.45 milliseconds
            median = 0.62 milliseconds
              75% <= 0.99 milliseconds
              95% <= 4.00 milliseconds
              98% <= 4.59 milliseconds
              99% <= 6.02 milliseconds
            99.9% <= 10.25 milliseconds

After the change to skip cache expiration at cache lookup, and only do cache entry expiration when we are inserting new cache entries, the same timer reported a hugely improved speed for cache lookups:

-- Timers -------------------------------
             count = 27500000
         mean rate = 261865.50 calls/second
     1-minute rate = 237073.52 calls/second
     5-minute rate = 186223.68 calls/second
    15-minute rate = 166706.39 calls/second
               min = 0.00 milliseconds
               max = 0.32 milliseconds
              mean = 0.02 milliseconds
            stddev = 0.02 milliseconds
            median = 0.02 milliseconds
              75% <= 0.03 milliseconds
              95% <= 0.05 milliseconds
              98% <= 0.05 milliseconds
              99% <= 0.05 milliseconds
            99.9% <= 0.32 milliseconds

That’s more like it. A cache lookup which completes in 0.32 milliseconds for the 99-th percentile of all cache lookups is something I definitely prefer working with. The insight from profiling tools, like YourKit, was instrumental in both understanding what the actual problem was, and verifying that the solution actually had the effect I expected it to have.

That’s why profiling is our friend!

by keramida at December 26, 2013 04:38 AM

September 25, 2012

Joseph Koshy

New release: ELF Toolchain v0.6.1

I am pleased to announce the availability of version 0.6.1 of the software being developed by the ElfToolChain project.

This new release supports additional operating systems (DragonFly BSD, Minix and OpenBSD), in addition to many bug fixes and documentation improvements.

This release also marks the start of a new "stable" branch, for the convenience of downstream projects interested in using our code.

Comments welcome.

by Joseph Koshy ( at September 25, 2012 02:25 PM

April 02, 2010

Henrik Brix Andersen

Downloading Sony GPS Assist Data Manually

After having bought a new Sony DSC-HX5V digital camera, which is equipped with an integrated GPS, I discovered that it comes with windows-only software for downloading and updating the GPS almanac on the camera (the supplied PMB Portable software runs on Apple OS X, but it does not support downloading the GPS almanac).

After tinkering a bit with tcpdump(1) and friends I found out how to perform the download and update manually:

  1. Download assistme.dat
  2. Download assistme.md5
  3. Verify that the MD5 sum of the assistme.dat file matches the one in the assistme.md5 file
  4. Create a top-level folder hierarchy on the memory card for the camera (not the internal memory of the camera) called PRIVATE/SONY/GPS/
  5. Place the downloaded assistme.dat file in the PRIVATE/SONY/GPS/ folder
  6. Place the memory card in the camera and verify that the GPS Assist Data is valid

I have written a small perl script for automating the above tasks. The script takes the mount point of the memory card as argument.

by brix at April 02, 2010 06:31 PM

March 21, 2010

Henrik Brix Andersen

Monitoring Soekris Temperature through SNMP

Here’s a quick tip for monitoring the temperature of your Soekris net4801 through SNMP on FreeBSD:

Install the net-mgmt/bsnmp-ucd and sysutils/env4801 ports and add the following to /etc/snmpd.conf:

begemotSnmpdModulePath."ucd" = "/usr/local/lib/"
extNames.0 = "temperature"
extCommand.0 = "/usr/local/sbin/env4801 | /usr/bin/grep ^Temp | /usr/bin/cut -d ' ' -f 6"

Enable and start bsnmpd(1). The temperature of your Soekris net4801 can now be queried through UCD-SNMP-MIB::extOutput.0 OID (

by brix at March 21, 2010 11:39 AM