To timidly go where many have gone before

Okay, so this is actually pretty simple, but I'll write about it anyway.

I've been looking at a core file with a hung nfsd, and I wondered if I could figure out how long threads had been blocked on the RW lock in question. The lock itself has no concept of when it was grabbed. A reader/writer lock is implemented as a single-word data structure, where all but a few bits of that word is devoted either to the address of the thread holding the write lock or to the count of readers. Reader/writer locks are described on page 836 of the Solaris Internals book and in the header file

Maybe some time information is kept in the turnstile itself? While the amount of time a thread spends waiting on a resource is a useful statistic, there's no need to store this information in that data structure[1], as the information is available elsewhere.

The kthread_t data structure contains this:

    193  clock_t  t_disp_time; /* last time this thread was running */

Given that the last thing the thread did while it was running was to request a resource that it then had to sleep on, the difference between now (stored in lbolt) and t_disp_time tells us how long the thread has been sleeping. So if we pick one of the threads waiting on a RW lock and investigate:

> ::turnstile ! awk '$3 != 0'
            ADDR             SOBJ  WTRS EPRI             ITOR          PRIOINV
ffffffff818ccac0 fffffe849958a150     5   60                0                0
fffffe93cd2335c8 ffffffff8875ccc0     5    4                0                0
fffffe93eedcc008 fffffe849958ab68     3   59                0                0
> fffffe849958a150::rwlock
            ADDR      OWNER/COUNT FLAGS          WAITERS
fffffe849958a150 fffffe849fb74120  B111 fffffe89fdd5fe40 (W)
                                    ||| fffffe8692f2aae0 (W)
                 WRITE_LOCKED ------+|| ffffffff83127720 (W)
                 WRITE_WANTED -------+| ffffffff817f3540 (W)
                  HAS_WAITERS --------+ ffffffff89f79f20 (R)
> fffffe89fdd5fe40::print kthread_t t_disp_time
t_disp_time = 0x549da3e9
> lbolt/X
lbolt:
lbolt:          54b25d0e
> hz/D
hz:
hz:             100
>

Subtracting t_disp_time from lbolt and dividing by hz, we see that this thread had been waiting a little over 3h46m when I grabbed this core file.

[1] I had originally written, "...there's no need to store this information in the data structures associated with a turnstile...", but a kthread_t could be considered one of the data structures associated with a turnstile, as the pointers maintaining the sleep queue are kept in the kthread_t structure itself (see also section 3.10 of the Solaris Internals book):

    105 typedef struct _kthread {
    106  struct _kthread *t_link; /* dispq, sleepq, and free queue link */
[ ... ]
    280  struct _kthread *t_priforw; /* sleepq per-priority sublist */
    281  struct _kthread *t_priback;
    282 
    283  struct sleepq *t_sleepq; /* sleep queue thread is waiting on */