Thursday, December 07, 2006
Problems with non-redundant ZFS
From watching the zfs-discuss mailing list, it seems that people are starting to experience problems with ZFS where it appears that ZFS is causing them to lose data (or at least rendering their data inaccessible, which may as well be data loss.)
The issue is with using ZFS in a non-redundant fashion on top of HW RAID where corruption is introduced by the hardware itself. What happens in these cases is that ZFS sees the corruption and stops letting you access the filesystem, which effectively means that your data is gone. Technically, there's still a valid ZFS filesystem on the disks, but because the hardware is introducing corruption, the ZFS software won't let you access it.
The complaint people seem to have is that ZFS isn't allowing them to salvage their data, where other filesystems would allow them access to their data. (Although what they don't seem to be considering is that they shouldn't be trusting the data that the other filesystem is giving them.)
From what I gather, these seem to be problems where letting ZFS handle (at least some of) the redundancy would help, given that ZFS would have the chance to apply its self-healing-data magic. But given that ZFS is being presented with a single LUN, it has nowhere to put corrected copies of data.
(And yeah, if something in the hardware chain is corrupting all of the data, ZFS redundancy won't help. But in this case, you're not going to be getting any data via any filesystem.)
There's a fairly lengthy thread about this here.
I recently saw a comparison of ZFS to TCP, in that both give you guarantees about the correctness of data being delivered to the application, and that comparison is pretty relevant to this problem. If TCP sees corruption in a data stream, it will not deliver the data (and because TCP also guarantees in-order delivery, the data after the corrupted data will not be delivered until the corruption is fixed.) ZFS is similar -- it will not deliver data that is corrupt.
Both ZFS and TCP have mechanisms for correcting corrupted data. In the case of TCP, the sender retransmits the data. In the case of ZFS, the software grabs a good copy of the data from its redundant location on disk, either from a mirror or from the parity data stored via raidz. But if the zpool has been created without redundancy, there is no good copy of the data to be found, the data is effectively lost.
The issue is with using ZFS in a non-redundant fashion on top of HW RAID where corruption is introduced by the hardware itself. What happens in these cases is that ZFS sees the corruption and stops letting you access the filesystem, which effectively means that your data is gone. Technically, there's still a valid ZFS filesystem on the disks, but because the hardware is introducing corruption, the ZFS software won't let you access it.
The complaint people seem to have is that ZFS isn't allowing them to salvage their data, where other filesystems would allow them access to their data. (Although what they don't seem to be considering is that they shouldn't be trusting the data that the other filesystem is giving them.)
From what I gather, these seem to be problems where letting ZFS handle (at least some of) the redundancy would help, given that ZFS would have the chance to apply its self-healing-data magic. But given that ZFS is being presented with a single LUN, it has nowhere to put corrected copies of data.
(And yeah, if something in the hardware chain is corrupting all of the data, ZFS redundancy won't help. But in this case, you're not going to be getting any data via any filesystem.)
There's a fairly lengthy thread about this here.
I recently saw a comparison of ZFS to TCP, in that both give you guarantees about the correctness of data being delivered to the application, and that comparison is pretty relevant to this problem. If TCP sees corruption in a data stream, it will not deliver the data (and because TCP also guarantees in-order delivery, the data after the corrupted data will not be delivered until the corruption is fixed.) ZFS is similar -- it will not deliver data that is corrupt.
Both ZFS and TCP have mechanisms for correcting corrupted data. In the case of TCP, the sender retransmits the data. In the case of ZFS, the software grabs a good copy of the data from its redundant location on disk, either from a mirror or from the parity data stored via raidz. But if the zpool has been created without redundancy, there is no good copy of the data to be found, the data is effectively lost.
