To timidly go where many have gone before

I don't have much experience with mdb, so I thought I'd dig into a kernel crash dump to see what I could figure out. This is a server connected to what might be faulty storage, so what I'm trying to determine is whether the storage might have had anything to do with the kernel panic.

First off, here's the stack backtrace:

server:/var/crash/server> sudo mdb -k unix.0 vmcore.0
Loading modules: [ unix krtld genunix specfs dtrace 
ufs md ip sctp usba random fcp fctl lofs nfs ptm 
logindmux ipc crypto fcip ]
> $C
fffffe800076d9d0 alloccg+0x48()
fffffe800076da20 hashalloc+0xb4()
fffffe800076da90 alloc+0x10b()
fffffe800076dc40 bmap_write+0xa74()
fffffe800076dd40 wrip+0x759()
fffffe800076dde0 ufs_write+0x211()
fffffe800076ddf0 fop_write+0xb()
fffffe800076de00 lo_write+0x11()
fffffe800076de10 fop_write+0xb()
fffffe800076dec0 write+0x287()
fffffe800076ded0 write32+0xe()
fffffe800076df20 sys_syscall32+0xef()
>

It died in alloccg() (allocate cylinder group), one of the lower-level UFS functions, so the backtrace hasn't ruled out the storage. So, given the above, how do I determine what file was being written to? The top of the backtrace looks like a good place to start, so I check out the definition of alloccg(). The first argument passed in is the inode:

static daddr_t
alloccg(struct inode *ip, int cg, daddr_t bpref, int size)
{

The inode itself doesn't contain the filename, but from the definition of an inode, we have this:

typedef struct inode {
[ ... ]
        struct  vnode *i_vnode; /* vnode associated with this inode */
[ ... ]
}

and we can get the pathname of the file from the vnode:

typedef struct vnode {
[ ... ]
        char            *v_path;        /* cached path */
[ ... ]
}

Mdb is aware of data structures, and can interpret the raw memory addresses WRT those data structures. So, if we have the pointer to that inode, we could figure out the path. In the 32-bit x86 world, arguments are passed on the stack, so the arguments would have shown up in the backtrace. In the x64 world, arguments are passed in registers, which are overwritten with succeeding function calls. There are ways to trace arguments down (functions that need to reference their arguments after making a nested function call need to save the register values somewhere, either on the stack or in non-volatile registers), but in this case, the appropriate register (%rdi for the first argument) contains what we need.

> ::regs
%rax = 0xfffffffffffff000                 %r9  = 0xffffffffaf62ab18
%rbx = 0x0000000000002000                 %r10 = 0xffffffff9e403559
%rcx = 0x0000000000002000                 %r11 = 0xfffffffffbcc2de0 apic_cr8pri
%rdx = 0xffffffff83dd2000                 %r12 = 0xffffffff83c79000
%rsi = 0x00000000ffffff00                 %r13 = 0xffffffffbf4bf500
%rdi = 0xffffffffbf4bf500                 %r14 = 0xfffffffffbad45e0 alloccg
%r8  = 0x0000000000000000                 %r15 = 0xffffffff832eb1c0

%rip = 0xfffffffffbad4628 alloccg+0x48
%rbp = 0xfffffe800076d9d0
%rsp = 0xfffffe800076d960
%rflags = 0x00010297
  id=0 vip=0 vif=0 ac=0 vm=0 rf=1 nt=0 iopl=0x0
  status=

                        %cs = 0x0028    %ds = 0x0043    %es = 0x0043
%trapno = 0xe           %fs = 0x0000    fsbase = 0x00000000fbc22ae0
   %err = 0x0           %gs = 0x01c3    gsbase = 0x0000000000000000
>
> 0xffffffffbf4bf500::print -t "struct inode"
{
[ ... ]
    struct vnode *i_vnode = 0xffffffffbf4bedc0
[ ... ]
>
> 0xffffffffbf4bedc0::print -t "struct vnode"
{
[ ... ]
    char *v_path = 0xffffffff863952f8 "/fs/data/somefile"
[ ... ]
>

And there's the file that was being written when the panic occurred in alloccg().

For reference, I used the new Solaris Performance and Tools book. I found some very detailed information on argument passing in the x64 world (specifically with respect to kernel crash dump analysis) here, an as-yet-unpublished book by Frank Hoffman at Sun.