Boost ZFS Performance with a Special VDEV in TrueNAS
Curious about how a special metadata VDEV can boost ZFS performance, especially with spinning disks? In this video, I walk through what it is, why it matters, and how to set it up in TrueNAS and some from the terminal. I’ll also share real-world benchmarks comparing pools with and without a special VDEV, so you can see the difference for yourself.
Testing
I created a test script that you can find here: https://github.com/timothystewart6/zfs-tools
The test script will try to create a pool based on 3 drives. HDD1, HDD2, and NVME_SPECIAL. You can modify these to match your disk Ids.
You can find your disk Ids by running:
1
ls -l /dev/disk/by-id/
You can also adjust the test files by changing TEST_COUNT however I found that 100,000 is a good number to get consistent results.
Update the script with your disk Ids.
You can then run the script:
1
2
chmod +x zfs-metadata.sh # make it executable
./zfs-metadata.sh # run the script
It will create 2 pools, test-1 and test-2, test-2 has the special vdeb.
This will take a long time to run depending on your system, disks, and TEST_COUNT.
When it’s done, you will find the results in /mnt/test-results/
If you want to check the small blocks values for your pool:
List pool
1
zpool list -v test-1 # change based on your pool name
Setting Small Blocks & Record Size
To change your small blocks value you can do so like this, however it’s a good idea to check your record size first
1
zfs get recordsize test-1
You should see something like:
1
2
NAME PROPERTY VALUE SOURCE
test-1 recordsize 128K default
You always want to be sure that your record size > small blocks size, otherwise all blocks will be written
Get value
1
zfs get special_small_blocks test-1 -r # change based on your pool name
Set value
1
zfs set special_small_blocks=64k test-1 # change based on your pool name and the small block value you want to use
If you want to use something higher than 128K you would do something like this
1
2
zfs set recordsize=256K test-1/yourdataset
zfs set special_small_blocks=128K test-1/yourdataset
My Test results
Random Read (4K, iodepth=1)
Here are my results from the video.
test-1pool was a single 14TB EXOS drivetest-2pool was a single 14TB EXOS drive + a Samsung 990 Pro NVMe
| Metric | test-1 (HDD only) | test-2 (Special VDEV) | Improvement |
|---|---|---|---|
| IOPS | 4,331 | 57,200 | +1,220% |
| Bandwidth | 16.9 MiB/s | 223 MiB/s | +1,220% |
| Avg Latency | 865.88 µs | 66.35 µs | −92% |
| 99% Latency | 8.0 µs | 2.8 µs | −65% |
| Read IO Completed | 2.0 GiB | 26.2 GiB | +1,210% |
Random Write (4K, iodepth=1)
| Metric | test-1 (HDD only) | test-2 (Special VDEV) | Change |
|---|---|---|---|
| IOPS | 209 | 195 | −6.7% |
| Bandwidth | 838 KiB/s | 782 KiB/s | −6.7% |
| Avg Latency | 18.8 ms | 20.2 ms | +7.4% (worse) |
| 99% Latency | 10.18 µs | 10.82 µs | Slightly worse |
Random Write (4K, iodepth=16)
| Metric | test-1 (HDD only) | test-2 (Special VDEV) | Improvement |
|---|---|---|---|
| IOPS | 210 | 229 | +9% |
| Bandwidth | 840 KiB/s | 919 KiB/s | +9% |
| Avg Latency | 303.8 ms | 277.4 ms | −8.7% |
| 99% Latency | 701 ms | 376 ms | −46.4% |
| 99.95% Latency | 776 ms | 443 ms | −43% |
| Max Latency | 842 ms | 516 ms | −38.7% |
Random Read/Write (4K, iodepth=16)
| Metric | test-1 (HDD only) | test-2 (Special VDEV) | Improvement |
|---|---|---|---|
| Read IOPS | 196 | 247 | +26% |
| Write IOPS | 196 | 246 | +25% |
| Read Bandwidth | 788 KiB/s | 989 KiB/s | +25.5% |
| Write Bandwidth | 786 KiB/s | 986 KiB/s | +25.5% |
| Avg Read Latency | 160.15 ms | 127.45 ms | −20% |
| Avg Write Latency | 163.93 ms | 130.24 ms | −20.5% |
| 99% Read Latency | 502 ms | 207 ms | −58.8% |
| 99% Write Latency | 506 ms | 209 ms | −58.7% |
Metadata – Random Access (20,000 files)
| Pool | Duration (s) | Improvement |
|---|---|---|
| test-1 | 71.91 s | — |
| test-2 | 65.73 s | +8.6% faster |
Metadata – Sequential Access (20,000 files)
| Pool | Duration (s) | Improvement |
|---|---|---|
| test-1 | 139.80 s | — |
| test-2 | 81.62 s | +41.6% faster |
📦 Products in this video 📦
While enterprise gear is great for businesses, I have found that if you have a good warranty, redundancy, and understand the trade-offs, consumer gear works great for home.
- Samsung 990 Pro NVMe: https://amzn.to/4d9JKXk (affiliate link)
Join the conversation
I tested how much a special metadata VDEV can actually speed up ZFS. Turns out it makes directory browsing faster, snappier containers, and a smart use of NVMe. I built a test script, ran benchmarks, and walk through it all in @TrueNAS https://t.co/czyXEFkSd3 pic.twitter.com/6hpIve8HR9</p>— Techno Tim (@TechnoTimLive) May 10, 2025
Links
🛍️ Check out the new Merch Shop at https://l.technotim.live/shop
⚙️ See all the hardware I recommend at https://l.technotim.live/gear
🚀 Don’t forget to check out the 🚀Launchpad repo with all of the quick start source files
🤝 Support me and help keep this site ad-free!