To timidly go where many have gone before

Woohoo! The standard deviation aggregating action (stddev()) for DTrace will soon be put back into OpenSolaris. This is my first putback for DTrace.

The change to add stddev() itself was actually very simple. First, note that it uses this approximation to standard deviation:

sqrt(avg(x^2) - avg(x)^2)

(For anyone who knows enough to complain about this, note this from the proposal I submitted for this:

        It is recognised that this is an imprecise approximation to
        standard deviation, but it is calculable as an aggregation, and
        it should be sufficient for most of the purposes to which
        DTrace is put.

)

When I was thinking about how to implement this, the natural thing to do was to model it after how avg() was implemented. When I first saw it, I was a bit surprised at how simple the implementation for avg() was (although in retrospect it was obvious, and I had even been thinking to myself that it must be fairly simple.) The code that implements the real meat of the avg() aggregating action (at least in the kernel) is just this (to be found here):

static void
dtrace_aggregate_avg(uint64_t *data, uint64_t nval, uint64_t arg)
{
        data[0]++;
        data[1] += nval;
}

All it's doing is keeping a count and a sum. The average itself is calculated in post-processing. So the implementation for stddev() is really just as simple as this:

static void
dtrace_aggregate_stddev(uint64_t *data, uint64_t nval, uint64_t arg)
{
        data[0]++;
        data[1] += nval;
        data[2] += nval * nval;
}

i.e., Just keep track of the count and the sum (to calculate avg(x)) and the sum of x^2 (to calculate avg(x^2)). Of course, a problem creeps in here -- if nval is larger than a 32-bit value, we'll blow our 64 bits. (And note that this is technically a problem for the existing implementation of avg(), too, as it could silently overflow its 64 bits. Pretty unlikely, but not impossible. This bug has been filed to correct this.)

What we decided to do to get around the problem was to implement 128-bit arithmetic functions to support this. The obvious question with respect to doing this is why not just use some multi-precision library and be done with it. Doing so would have introduced a dependency between the kernel and this external library, though, and given how easy it is to implement this, it was better to do so.

This was fun. I wanted to install the OpenSolaris preview on a ThinkPad. Unfortunately, the ThinkPad has an NVIDIA video card that isn't supported by the current version of the OpenSolaris LiveCD. But the GRUB menu shows a text-only option for booting, so I boot into that.

It boots and gives me a console prompt, and I try to figure out how to do a text-based install. There's /usr/bin/gui-install, but nothing obvious for a text-based install. A little bit of Googling, and I discover that there's no text-based install in the preview release. (It is the alpha release, so it's not terribly surprising.)

Okay, I'll just ssh in, run the GUI installer, and have it display on my desktop. When I try to ssh, I get this message: "no kex alg" (no key exchange algorithm.) Some Googling yields this thread, which indicates that it's simply a lack of host keys. I generate the host keys and successfully log in.

Okay, now X forwarding isn't working. I look at /etc/ssh/sshd_config, but things look fine there. I get this message on the laptop: "failed to create a directory for the temporary X authority file: Error 0; will use the default xauth file". Some Googling shows me this bug (which is actually a duplicate of this bug), which indicates that the error message is misleading.

On a hunch, I decide to check if xauth is installed on the LiveCD. Doh! No xauth. Oh, but that's okay, I'll just copy a version over and see if that works. From "strings sshd", I see that sshd has the hard-coded path "/usr/openwin/bin/xauth". Hmm, /usr is a read-only file system mounted from the CD.

Given that I can't copy xauth to the hard-coded path, what can I do? Well, there's the other option of changing the hard-coded path. I could just compile a version of sshd with a different path for xauth, but I'm feeling lazy, so I decide to just change the hard-coded path in the binary.

In trying to figure out how to do this with the tools available on the LiveCD, I run across mention that vim can handle editing binary files. (I'm sure that emacs would also be able to do it.) I copy the sshd binary to my desktop, open the file in vim, find the string "/usr/openwin/bin/xauth" and replace it with "/var/tmp/foo/bin/xauth". (Note that the lengths of the strings are the same -- I don't want to throw off the addresses for everything else in the file. Even then, the resulting file was one byte larger, although od shows me that that extra byte isn't in the modified string.)

I copy the sshd binary back to the laptop and give it a try. The binary runs, and X forwarding works (given that I'd already copied xauth to the appropriate location.) I run /usr/bin/gui-install, and bam! the installer window pops up on my desktop. A little while later, I have the OpenSolaris preview installed.

(And then I follow the instructions to update packages and grab the latest NVIDIA driver package. One reboot later, and I'm cooking with gas.)