Data corruption (zeroed blocks) with dedup enabled in ZFS 2.4.0 RC5

[saju-p]

System information

Type	Version/Name
Distribution Name	RHEL / RockyLinux
Distribution Version	RHEL 10.1 / RockyLinux 9.4
Kernel Version	6.12.0-55.9.1.el10_0.x86_64 / 5.14.0-427.37.1.el9_4.x86_64
Architecture	x86_64
OpenZFS Version	zfs-2.4.1 (tag zfs-2.4.1, commit 1c702dd). Also reproduced with zfs-2.4.0 (tag zfs-2.4.0, commit 7433349).

Describe the problem you're observing

Silent data corruption (zeroed blocks) when dedup is enabled in ZFS 2.4.0 and 2.4.1. File contents are partially or fully replaced by zeros, while file sizes remain identical to the originals. ZFS reports no checksum errors — the corruption is silent.

The corruption is random and intermittent — it can occur in any iteration, affecting any file.

Key observations:

Affected: ZFS 2.4.0 and 2.4.1 with dedup=on
NOT affected: ZFS 2.3.4 with dedup=on (works correctly)
NOT affected: ZFS 2.4.0/2.4.1 with dedup=off (works correctly)
This indicates a regression in the dedup path introduced between ZFS 2.3.4 and 2.4.0.

Tested on two different platforms:

RHEL 10.1 with 4 NVMe raw drives
RockyLinux 9.4 with a 10G loop device

Describe how to reproduce the problem

Step1:
Cloned and checked out zfs-2.4.1/2.4.0 tag

commit 1c702dd (HEAD, tag: zfs-2.4.1)
Author: Tony Hutter <hutter2@llnl.gov>
Date: Wed Feb 11 09:39:28 2026 -0800

Tag zfs-2.4.1

META file and changelog updated.

Signed-off-by: Tony Hutter <[hutter2@llnl.gov](mailto:hutter2@llnl.gov)>

OR
commit 7433349 (HEAD, tag: zfs-2.4.0)
Author: Tony Hutter <hutter2@llnl.gov>
Date: Mon Dec 15 09:52:28 2025 -0800

Tag 2.4.0

Signed-off-by: Tony Hutter <[hutter2@llnl.gov](mailto:hutter2@llnl.gov)>

Step2:
Build ZFS:
sh autogen.sh
/configure --prefix=/usr --sysconfdir=/etc --libdir=/usr/lib64 --includedir=/usr/include --sbindir=/usr/sbin
make -j${noproc}
sudo make install
sudo ./scripts/zfs.sh

Step3:
Create a pool and set compression=gzip, checksum=sha256 and dedup=on
Create a encryption dataset withing the pool by setting encryption=on
Set the record size recordsize=

echo 0 | sudo tee /sys/module/zfs/parameters/zfs_compressed_arc_enabled
zpool create -o ashift=12 -f /dev/loop0
zfs set compression=gzip
zfs set primarycache=none
zfs set checksum=sha256
zfs set dedup=on
zfs create -o encryption=on -o keyformat=raw -o keylocation=file://<key_file> /
zfs set recordsize=

Step4:
Copy the Silesia corpus files from directory named A to pool/dataset
Copy the files back to another non zfs directory named B.

Step5:
Compared md5sum of each file of directory A and corresponding file directory B.

Step6:
Destroy dataset
Destroy the pool

Repeated the Step3 to Step6 several times for different record sizes 512, 1K, 2K, up to 8M

Include any warning/errors/backtraces from the system logs

[saju-p]

This issue seems to only occur with the Silesia corpus. It could be due to data like "FF FF FF 00 00 etc." hitting some corner cases. However, the other corpus works fine.

The good news is that version 2.4.1 RC4 appears to be good with dedupe enabled.
There are 32 commits between RC4 and RC5, three of which are related to dedup - could be an issue with those

[tonyhutter]

The good news is that version 2.4.1 RC4 appears to be good with dedupe enabled.

To be clear - did you mean to say 2.4.0 RC4?

[amotin]

I have a script looping this process with different block sizes for an hour now on ZFS master on FreeBSD without any issues. Considering the amount of not very usual things you've done, like zfs_compressed_arc_enabled=0, gzip, encryption and primarycache=none, are you sure dedup is a cause, not a trigger of something in any of those others? Have you tried to flip the other switches too? Considering the relatively small amount of data, and that you are not mentioning any zpool sync in the process, I suspect the second copy might be a bit racy in where it actually gets the data (might still be in caches if the previous write haven't completed). So I guess some dedup performance optimizations done in the commits mentioned could just open the race wider.

[amotin]

BTW, the data seems to be not really dedupable beyond ~4-8KB blocks.

[saju-p]

The good news is that version 2.4.1 RC4 appears to be good with dedupe enabled.

To be clear - did you mean to say 2.4.0 RC4?

Yes, thanks for catching that typo

[saju-p]

I have a script looping this process with different block sizes for an hour now on ZFS master on FreeBSD without any issues. Considering the amount of not very usual things you've done, like zfs_compressed_arc_enabled=0, gzip, encryption and primarycache=none, are you sure dedup is a cause, not a trigger of something in any of those others? Have you tried to flip the other switches too? Considering the relatively small amount of data, and that you are not mentioning any zpool sync in the process, I suspect the second copy might be a bit racy in where it actually gets the data (might still be in caches if the previous write haven't completed). So I guess some dedup performance optimizations done in the commits mentioned could just open the race wider.

-The same pool settings work properly when dedup is off. The issue appears when dedup is enabled. Hence, I believe it must be dependent on dedup.
-We have not flipped any other switches.

[saju-p]

BTW, the data seems to be not really dedupable beyond ~4-8KB blocks.

We are testing dedup enabled for all corpuses. It's possible that the Silesia corpus does not compress further with dedup enabled.

[saju-p]

@amotin,

there could be a Race Condition in zfs\module\zfs\zio.c
-Thread A writes block X, finds DDT entry, piggybacks on lead zio, increments refcount optimistically, releases both locks
-Thread B reads the DDT entry, sees refcount > 0, treats block X as a valid duplicate
-The lead zio hasn't actually completed writing yet — the DVAs point to unwritten (zeroed) storage
-Thread B reads back zeros

[amotin]

@saju-p Until the thread A writing (or whatever it is piggybacking) is complete, the data should remain in ARC and dbuf caches, even with primarycache=none. So thread B's attempt to read it should not reach the disks. But even if it would, reading unwritten block would likely result in checksum error, not a zero block.

[saju-p]

file_copy_check_repeat.sh
we use this script to test.

I also have a doubt on commit a41ef36 — "DDT: Reduce global DDT lock scope during writes"
Will test by removing it

[saju-p]

file_copy_check_repeat.sh we use this script to test.

I also have a doubt on commit a41ef36 — "DDT: Reduce global DDT lock scope during writes" Will test by removing it

Yes, this helped.

We ran a bunch of tests and they all passed with the latest code (2.4.1) and removing the below commit and running with dedup enabled.
a41ef36 — "DDT: Reduce global DDT lock scope during writes"
we will continue to do more tests.
The test run were with Silesia corpus for an hour+ with the test script above.