Home Server: RAID-GREED and Why Mirroring is Still Best

Hi Brian,

yes, my boot drive is mirrored through OpenSolaris, not the motherboard. This gives you all the benefits of ZFS mirroring (data integrity, self-healing, etc.) and you're independent of the motherboard manufacturer's HW RAID solution.

If you're quick, you can mirror while you install: Start a terminal before running the installer, then do zpool status often while the installer is running. As soon as you can see the pool, do a zpool attach rpool c1d0 (or whatever your second drive is.

After installation, make sure you also add a GRUB bootloader to the mirror disk, see the installgrub(1M) manpage.

Hope this helps!

Hi Jason,

thanks for your comment!

ZFS crypto shouldn't pose a big performance hit. People are using FileVault on their Macs and other crypto solutions for their disks like TrueCrypt on a daily basis. If something is important enough to be encrypted, then the performance penalty doesn't really count.

Slicing the disks is of course possible and you can mix and match if you like, but it's not recommended as this will prevent ZFS from optimizing serial block access on a per disk basis.

Basically my argument is: Disk space is cheap! Put it at your last priority, behind performance and availability.

The reason I recommend to use a bigger drive as the 3rd one is that once the smaller ones break and you buy the next of the bigger size, your pool will automatically grow, presumably together with your storage needs. That's a nice, organic growth strategy. It also plays well with a regular spending of approx 1 disk per year at the same cost (which automatically translates in bigger disks per year as prices go down and capacity per price goes up over time).

If you only have 3 disks, then I recommend you mirror two of them and use the third for testing etc. But then the mirror will be faster than the single disk, YMMV. At the scale of 3-4 disks, RAID-Z doesn't really help much, it only makes your setup less flexible.

But that's an opinion, you should play out the expected performances with different setups. Check out the articles by Richard Elling and Roch Bourbonnais that I mention in the article.

Cheers,
Constantin

Hi Bob,

thank you for your comment!

Sorry if my post has been unclear, I'll try to clarify it with this comment:

• In the beginning of the article, I say that ZFS RAID-Z (which uses a variant of the RAID-5 algorithm) offers comparable performance to a hardware RAID-5 controller. There are three factors responsible for this:
  1. Modern CPUs and server load characteristics leave ample room for calculating RAID-5 parity algorithms. There's no need for dedicated RAID-5 calculation hardware any more.
  2. RAID-5 controllers use their read cache to buffer read data when writing back partial stripes. This helps them avoid read-modify-write cycles when only part of the data in a stripe needs to be modified.
     ZFS RAID-Z avoids this problem completely and without the need for extra controller hardware: Every transaction (even the modification of an existing file) eventually gets written as its own, full stripe, thanks to variable block sizes. Therefore, there's no need for read-modify-write cycles in ZFS. This is explained in more detail in Jeff Bonwick's blog post about RAID-Z
  3. RAID-5 controllers also use their write cache to buffer writes, trying to align them optimally so they can avoid excessive seeking of harddisk heads due to random writes, which hurts performance a lot.
     In contrast, ZFS RAID-Z avoids random writes completely through its copy-on-write nature: All writes happen on new, free blocks. This gives ZFS the opportunity to choose the blocks it's going to write to (providing a reasonable number of blocks are free, of course) and arrange for them to be sequential, rather than random, leveraging the disk drive's maximum performance.
• In the rest of the article, I compare ZFS RAID-Z with ZFS mirroring (RAID 1 or 1+0) in terms of cost, performance, fault-tolerance and flexibility. For performance, even assuming maximum write performance per disk, there are fundamental reasons why RAID-Z is slower than mirroring. For writes, mirroring is about the same as RAID-Z if it's just a single, 1+1 mirror. As soon as you stripe across many mirrors (2+2, 3+3, etc.), writes can be spread across these writes, increasing performance. For RAID-Z and the same total number of drives, the chunks are bigger and thus the ability to spread the load diminishes. For reads, the performance difference is bigger: Mirrors can always spread the load across all drives (not just mirrored pairs as for writes), while RAID-Z can only spread read load across full stripes.
  This is not particular to ZFS, this follows from the way the disks are addressed and behaves about the same with traditional mirrors or RAID-5. There's a nice performance comparison of RAID-Z vs. mirrors for reads and writes in Richard Elling's article: ZFS RAID recommendations: space, performance, and MTTDL.

Bottom line (and I think we agree here): Mirrors are faster than RAID-Z (and RAID-5), for the same number of total disks.

However, the surprise to some readers is that a software RAID-5 technology like ZFS RAID-Z can be as fast as hardware RAID-5, provide better fault-tolerance and add to it data integrity and many more features that traditional RAID-controllers can't offer, essentially for free. There's really no reason any more to buy a hardware RAID controller any more.

I hope this clarifies the "ZFS RAID is about as fast as hardware RAID" part a bit. Next time, I should probably split up the article into multiple segments to be more clear :).

Thanks,
Constantin

Hi Niels,

thank you for your comment.

Yes, the importing system must support the ZFS version of the pool to be imported.

An overview of Zpool versions is available from the ZFS community pages on opensolaris.org. This is close to the table you were looking for. See also the zpool upgrade section of the ZFS Administration Guide.

You can also ask a running system what pool version it uses by running the zpool upgrade -v command. The current Solaris version 10 10/09 runs zpool version 15, whereas OpenSolaris 2009.06 runs zpool version 14.

So, for released versions of Solaris, ironically Solaris 10 is more advanced in terms of zpool version than OpenSolaris :).

Back to your example: Yes, there is some risk in trying out a developer version of Solaris and then you may run into a situation where you'd need to migrate your data by means other than a simple zpool import. Fortunately, current OSes that support ZFS are very quick in adopting new zpool versions, so that is not a lasting issue. You could have stayed on SXCE for a couple of months until Solaris 10 catched up to the zpool version.

Anyway, I hope that you're now using the Solaris version you want with the ZFS pool features that you want :).

Thanks,
Constantin

Hi UX-admin,

thanks for your comment!

Just to clarify: For a given RAID-Z pool, there are two ways to increase its capacity:

• You can replace individual disks inside a vdev with bigger disks. But you'll only be able to take advantage of their bigger capacity once you've replaced all of them (I just verified that on a test pool). So, adding capacity this way through exchanging a single disk doesn't work.
• You can add additional vdevs that may be RAID-Z with the same or different stripe size, or even mirrors or single disks, to an existing RAID-Z pool. Yes, the barrier to upgrade goes down to 2, because you can add a mirror (or a 1+1 RAID-Z) vdev to an existing RAID-Z pool, but then the pool becomes ugly: It's a stripe between two different kinds of fault-tolerance and two different kinds of performance characteristics. For any given IO that is spread across such a stripe, you'll end up with the slower performance of both, thanks to the law of the weakest link. Therefore, this is not recommended (and that's what I hinted at when saying "unless you want to end up with a chaotically mixed RAID-Z+Mirror configuration").

Yes, in a home server setting, beauty or balanced performance might not matter, and in fact, one of my earlier home server storage setups was very mixed. But in hindsight, I'd like to advocate for simplicity vs. squeezing the last bit of space out of a pool. After all: Disks are cheap, you data and your sanity is not.

Thanks for bringing up Roch's excellent article When to (and not to) use RAID-Z, another favorite of mine. You and Roch and my customers work in an enterprise environment, and thus can use a larger number of disks. And this is indeed where you can have your cake and eat it too (or more correctly: Balance performance and capacity on a finer granularity, as needed). But in a home server environment (which this article focuses on), the number of disks is limited. The typical motherboard comes with 6 SATA ports. Two of them are typically taken by boot disks (mirrored of course) and that leaves us with 4 disks for data. In this case, I wouldn't advocate for RAID-Z, I'd consider RAID-Z2 (like Andreas suggested) but for the cases where pool size can be 2 TB or less, I'd still go with a 2-way mirror and use a bigger disk as hot-spare or as a third mirror half (like described in the article), opening a door into the next bigger pool size. That leaves me with a spare SATA port to use for a scratch disk or for other purposes.

This is also driven by disk quality: Consumer disks are of low quality, and they are darn cheap. Therefore home server builders should place quality above capacity and performance, and avoid GB-greed. In the enterprise world, the setting is slightly different (although still, I don't see many cases where customers really need to place capacity above performance or fault-tolerance).

Thanks,
Constantin

Thanks,
Constantin

Thanks, Dave!

And make sure you take Andreas Jantos' take on RAID-Z2 into account. He has a point on fault-tolerance, but I'd still try to fit everything into a 3-way mirror instead of a 2+2 RAID-Z2 for the simplicity and speed.

Cheers,
Constantin

Hi Andreas,

yes, your analysis is correct and RAID-Z2 provides better fault tolerance than a 2x2 mirror. I should have worked it into the analysis, but didn't consider it because RAID-Z2 tends to use quite a few disks to be interesting, so thanks for providing it.

And for home servers, number of drives/ports is a limiting factor to keep in mind, so the bigger the disks, the better. Frankly, I'm glad that my data requirements fit in 2 TB now, so I can have a 3-way mirror (which beats RAID-Z2 on speed and flexibility and provides the same "2 drives may fail" resiliency). Once you are forced to have many drives, RAID-Z2 becomes interesting, albeit with the performance/flexibility penalty.

Given the low cost of pure harddisk space and accepting a resilvering window of 4-8 hours, perhaps a 2x2 mirror with a hot-spare is still better overall than RAID-Z2? Granted, it costs 5 ports, but you get almost everything...

Thanks for the disk size data point. I know this has been an issue in the past with enterprise drives and didn't want to run that risk with consumer ones, perhaps they learned to be RAID-friendly?

Thanks for you comment,
Constantin

Hi,

Cloud Backups sind im Moment recht teuer, wenn sie im Bereich von 'zig GB sind. Das reicht gerade mal für wichtige Dokumente etc., aber nicht für einen Dump eines ganzen Pools...

Aber wer weiss, vielleicht in 5 Jahren :).

Vielen Dank für den Beitrag!

Ciao,
Constantin

Hi Neil,

thanks for your comment!

You're right, resilvering time can be an important factor. My old pool consisted of 2 x 2 x 1 TB WD MyBook Essential Edition USB drives. There was a time where I had to do a lot of resilvering and it took about a day for a half-full pool. One time, a USB connection dropped to USB 1.1 speed and then resilvering time was really bad. That was a mirror, it would have been worse for a RAID-Z.

When I resilvered my new 2TB disk into the 2 x 1.5 TB pool, which is about 1 TB full, resilvering time was about 4.5 hours. This is about half the maximum write bandwidth quoted for the drive, so this is ok.

As drive sizes continue to increase exponentially while IOPS essentially stagnate, I expect the resilver time per disk to become worse and worse. This is why some people declared the death of RAID-5 and they're right. If anybody thinks about RAID-5, they should skip it and go straight for RAID-6 (or RAID-Z2, respectively).

For a home server, 4.5 hours or resilver time is fine. I've heard of enterprise customers running large RAID-Z stripes (think 10 drives) and then wondering why resilvering takes many days, while the system was still suffering "normal production load". This is when your greedy storage strategy becomes really ugly and where choosing mirrors over RAID-Z becomes a career move. Mind you: This is not ZFS' fault, just the pure physics of using drives in a RAID-Z fashion under load while resilvering.

Backups are another can of worms, perhaps I should write a whole article on them... For home servers, I only see three possibilities: 1. USB disks (cheap, not too slow, but have to be on site), 2. Cloud storage (expensive, slow, but off-site) and 3. another home server (maybe less expensive, fast, at least in a different room). zfs send/receive (over SSH if needed) is an excellent way to do full/incremental backups, but in some scenarios you really want a file-based backup.

Perhaps a combination of the above is the way to go, depending on data type (music, photos, documents).

Thanks for reading my blog, l hope I can put together a good article in backup strategies once my own one is somehow finalized :).

Cheers,
Constantin

Hi Al,

thanks for your comment!

Yes, you're right. A 3-way mirror vs. an unsync'ed hotspare is more practical if you're only mirroring a single disk. For data pools up to 2 TB, this works well and it's probably a better deal than try to save a couple of bucks by splitting the mirror into 2 x 2 x 1TB disks.

OTOH, no matter how many mirror vdevs you run, the 3-way strategy is always 50% of your mirror cost. And with falling disk prices, it's becoming less and less of a burden over time.

Glad to hear your pair of mirror runs well!

Cheers,
Constantin