To timidly go where many have gone before

Occasionally the unlinked file problem pops up. For the sysadmin, this generally starts with a log file filling up a file system. Someone removes the original file, generally because they've moved the file across a filesystem boundary (or gzipped it to a target file on another file system.) But now that the offending file is gone, the file system is still full because some process has the file open, and as long as there's a link to that file, the file system can't reclaim the space.

There are various ways to find that process. The first time I saw this, some 12 years ago or so on a SunOS 4 server, I used some tool (probably lsof) to generate a list of the inode numbers of all the open files on the system matching the particular device number. I then used some other tool (probably find) to generate a list of all the inodes on that particular device. I compared the sorted lists to find the inodes that didn't exist in the filesystem and then went back to the lsof output to find the process holding open the unlinked file.

In a later job, I found that someone had essentially scripted the above process, (which was logical, 'cause it satisfied Cardinal Rule number 1: Automate complex, repetitive processes.) But it was still fairly slow. And then one day I noticed something about the entries in /proc/*/fd/ that made this much faster.

Under Solaris, an unlinked file shows up with a link count of 0:

c---------   1 juser tty       24,  1 Nov 27 15:16 0
c---------   1 juser tty       24,  1 Nov 27 15:16 1
c---------   1 juser tty       24,  1 Nov 27 15:16 2
-r--r--r--   0 juser staff    1048576 Nov 27 10:56 3

Under Linux (or at least the version of RedHat running on a box I have access to), an unlinked file is tagged as deleted:

lrwx------ 1 juser staff  64 Nov 27 20:19 0 -> /dev/pts/0
lrwx------ 1 juser staff  64 Nov 27 20:19 1 -> /dev/pts/0
lrwx------ 1 juser staff  64 Nov 27 20:19 2 -> /dev/pts/0
lr-x------ 1 juser staff  64 Nov 27 20:19 3 -> /var/tmp/bigfile (deleted)

Something interesting to note here is that Linux gives you the filename, which makes sense given that the fd is presented as a file descriptor. Solaris doesn't give you the filename, but a quick MDB invocation can get that for you:

#mdb -k
Loading modules: [ unix krtld genunix specfs dtrace cpu.AuthenticAMD.15 ufs 
ip sctp usba random fcp fctl lofs md cpc fcip crypto logindmux ptm nfs ]
> 0t4291::pid2proc | ::fd 3 | ::print file_t
{
    f_tlock = {
        _opaque = [ 0 ]
    }
    f_flag = 0x2001
    f_pad = 0xbadd
    f_vnode = 0xffffffff860fcb40
    f_offset = 0
    f_cred = 0xffffffff84cefc50
    f_audit_data = 0xffffffff8b709638
    f_count = 0x1
}
> 0xffffffff860fcb40::vnode2path
/var/tmp/bigfile
>

And of course, you could just do this as a single command line:

#echo "0t4291::pid2proc | ::fd 3 | ::print file_t" | 
/usr/bin/mdb -k | grep f_vnode | awk '{printf("%s::vnode2path\n",$NF);}' | 
/usr/bin/mdb -k
/var/tmp/bigfile
#

There are two steps to this: "0t4291::pid2proc | ::fd 3 | ::print file_t" first turns the pid 4291 into a pointer to a proc structure. Given that, we ask for the pointer to the file_t structure associated with file descriptor 3 of that process, and then we print it based on its type. The second step is to take the pointer to the vnode and print its path (::vnode2path).

And doing a search shows a feature of lsof that I hadn't encountered yet: 'lsof +L1' will show you all of the open files on the system with a link count less than one. This blog is one among a few that mention using lsof this way. Lsof still won't give you the name of the unlinked files, though, so the mdb trick still comes in handy.