To timidly go where many have gone before

My first DTrace success pointed us towards using libumem in our threaded SMTP server. In order to protect the interests of the company I work for, I won't go into too much detail about our email plant architecture, but this application runs in different modes. One of them is the one that I've already described, the mode running on our MX servers that accept connections from the outside world. Once mail is received, we filter it (in ways that should be obvious considering the signal-to-noise ratio) and then hand off what's left to be delivered to user mailboxes. The application that does this work is that same SMTP server running in a different mode.

We've long known that there are performance problems with this mode of the application. We run it mostly on single-cpu servers (v120's) because it doesn't scale well to 2 or 4 CPUs. Even on the single-cpu servers, the CPUs are mostly idle, and the server plant is larger than it ideally needs to be. The developers blamed NAS performance, but the systems guys knew better, as nothing on the filer indicated that there was a problem. And nobody had put much effort into tracking down the problem.

Based on our success with libumem with this application in its other mode, we naturally assumed that using libumem here would solve all of our problems. We tried that here, and we saw some improvement, but not nearly what we had expected. Using a similar analysis (again noting that I hadn't yet discovered plockstat, which would have made the analysis much faster), it looked like all of the lock contention was coming from readdir_r().

But this doesn't make sense, right? Using readdir_r() is the Right Thing To Do(TM) in a threaded application, as it's thread-safe, whereas readdir() isn't. So why were we seeing problems here?

The man page contains this interesting tidbit about readdir():

     struct dirent *readdir(DIR *dirp);

[ ... ]

     The  pointer  returned by readdir() points to data which may
     be overwritten by another call  to  readdir()  on  the  same
     directory  stream.  This  data is not overwritten by another
     call to readdir() on a different directory stream.

Based on that description, one might assume that readdir_r() has been implemented with a mutex per directory stream. Does the code back this up? Let's look at the source:

     57 extern mutex_t _dirent_lock;

[ ... ]

     67 int
     68 readdir_r(DIR *dirp, struct dirent *entry, struct dirent **result)
     69 {
     70  struct dirent *dp; /* -> directory data */
     71  int saveloc = 0;
     72 
     73  lmutex_lock(&_dirent_lock);

[ ... ]

     99  lmutex_unlock(&_dirent_lock);
    100  *result = entry;
    101  return (0);
    102 }

So, no, we don't have a per-dirent lock, there's a single lock per process, which obviously doesn't scale to hundreds of threads. Given that this is a problem, what can we do about it? Do we actually need to use the thread-safe version?

So when does the thread-safe version of readdir_r() make sense? We're using the lock to protect the contents of the struct dirent. But the above man page snippet would indicate that that data structure wouldn't get overwritten by a readdir() on a different directory stream, if we were simply using readdir(). So it seems that the only time you'd need that protection is if multiple threads are sharing the same directory stream. Our application certainly isn't doing so, as every thread is doing its own opendir()/readdir_r()/closedir(). So we should be safe to replace the offending readdir_r() call with readdir().

And having done so, we saw a dramatic increase in performance (as measured in throughput per server.) I wish I could include the MRTG graph of thread count on these servers, but imagine a profile of the United States moving from the Rocky Mountains to the plains, and you'll get a good image of what it looked like. The improvement was enough to allow us to cut this server plant in half while still leaving a large buffer to handle spikes.

BTW, the email thread containing this message from Casper Dik is pretty interesting WRT the above problem.