Laying out VNX FAST VP storage pools

One of the questions I run into a lot pertains to EMC VNX FAST VP storage pools, particularly with VMware.  Now that we have all these layers of abstraction (the sub-LUN tiering in FAST VP, the I/O improvements of FAST Cache, then on top of that the I/O aggregation from VMware datastores with many VMs, plus other interesting technologies that people are still figuring out how to optimize), how should we be laying out our drives?  On one extreme we could build a monolithic storage pool and giant multi-TB datastores and let FAST move our data to the right type of spindle.  On the other extreme we could build nothing but traditional RAID groups housing single-VM datastores and know exactly which drive our VM is running on.  As with most things, I think the sweet spot is in the middle.
One of the things that often gets overlooked with FAST VP storage pools and VMware is RAID type.  Depending on your performance requirements and how sensitive your applications are, you might be able to run all of your VMs on RAID 5, maybe on a single datastore.  Plenty of folks do that, and there’s a lot to be said for the simplicity that provides.  However, the trade-off is that storage array is going to work harder than it has to and performance will not be as good as it could be.  As the performance requirements and app sensitivity go up, LUN response time will become more important, especially to transaction logs and the blocks that house clustered indexes etc.  The array will start to need more EFD to keep up with the I/O waiting to be written to slower drives, using up FAST Cache and sending more I/O straight to spinning drives.  Basically, it becomes more important to optimize for performance rather than simplicity.
Even with FAST Cache in place and FAST VP storage pools with multiple tiers shuffling hot chunks of data up to faster drives and cool chunks of data the other way, RAID types still affect I/O signatures and still have specific use cases.
Always always remember that a given tier of a FAST VP storage pool should contain identical drives (size and rotation speed).  Also remember the recommended drive counts: multiples of 8 for RAID 1/0, multiples of 5 for RAID 5, multiples of 8 for RAID 6.  These drive counts are also per-tier, not for the whole pool, and they really kick in and make a difference once you get above the first multiple (ie: once you get to 6 drives in a RAID 5 tier, make sure you go all the way to 10, though running fewer than 5 can be okay (especially in the top tier where EFD’s make a huge difference and are proportionately expensive (nested parens for the win!))).  Ignore these rules at your own peril.
So, unless you fit into that group I mentioned above that can run their environment all on RAID 5, here’s how I try to lay out storage pools.

• One or more RAID 1/0 storage pool with only 15K SAS (or FC on a Clariion) drives and with FAST Cache turned off.  LUNs in this pool will be used for transaction logs for multiple servers, VM or otherwise.  RAID 1/0 continues to be the best option for small sequential write-sensitive I/O, and you still get the benefit with multiple servers accessing a single pool.  If you need to have your Exchange logs separate or you have a big critical app that can justify 8 drives for transaction logs, create more than one pool.  Just keep in mind you don’t have to isolate every app, then do so when it’s necessary.
• Several RAID 5 storage pools.  FAST Cache should be turned on, and keep in mind that data residing on EFD in a storage pool will not be cached in FAST Cache. (It’s just not necessary and would slow things down for a couple microseconds. You might need those microseconds one day.  More importantly, something else would benefit from that FAST Cache).  Database data files, mailbox stores, and general VMware datastores live here.  All of those are random I/O, and this type of data is not here simply because it’s less expensive than the 50% capacity overhead of RAID 1/0.  RAID 5 can actually outperform RAID 1/0 in handling random I/O.  This is where the variation really comes into play based on the environment.  Some small environments might have one or two RAID 5 pools, with LUNs for different apps running on both.  Larger environments might have multiple Gold, Silver, and Bronze pools. Gold might have 5x EFD’s, 15x 15K 300 GB SAS, 5x 2 TB NL-SAS drives.  It’s still going to have cooler chunks of data most likely, they can still reside on NL-SAS, just size the top and middle tiers appropriately so the Gold stays Gold.  Silver might have 20x 10K 600 GB SAS and 10x 2 TB NL-SAS drives.  Et cetera.
• One or more RAID 6 storage pools.  FAST Cache should be turned off.  RAID 6 is good for large sequential writes.  Not just “alright” or “acceptable”, it’s actually good for it.  Backups, SQL .bak files and Oracle dump files, and other types of archives should live here.  VNX File storage is often on RAID 6 as well, depending on the performance requirements, but should be in a separate pool  from other data. RAID 6 is also recommended for all drives 1 TB and greater because of rebuild times.  Under the covers, a 24 drive RAID 6 storage pool is comprised of 3x 6+2 RAID 6 sets.  If a drives fails, a global hotspare steps in and the raid set starts rebuilding, which can take a while.  If another drive fails while that’s happening, we’re okay because RAID 6 has 2 parity blocks on each stripe.  RAID 5 on the other hand can only handle a single drive failure, and protection is not restored to that live production data until the rebuild is finished.  That means, with larger drives and their longer rebuild times, we’re more exposed with RAID 5 than RAID 6.  It’s not run-for-the-hills risk, but it’s enough to affect how we design.  Once we can do multiple RAID types in a single pool, we’ll have the NL-SAS drives using RAID 6 everywhere, but for now RAID 5 still happens in mixed pools.
• Traditional RAID Groups.  Some things, like RecoverPoint journal volumes, still just make more sense on a traditional RAID group.   These LUNs will never be expanded, they won’t be thin provisioned, they’re not going to share spindles with other types of data, and they’re not going to need sub-LUN tiering.  Some applications have specific storage support requirements, and that can force traditional RAID groups as well.

It can be as simple or complex as it needs to be, but the general theme is that we still want to use the options that make the most sense and get the best bang for the buck.  With vSphere 5 removing the 2 TB cap on datastore sizes, it’s very tempting to heap everything together, let FAST VP do its thing and adopt “Simple First!” as the storage strategy.  And that works sometimes, but it only works when it makes sense for the environment. How the LUNs are laid out will depend on a lot of other things like replication, snapshot technologies, and SRM, but (allowing for the possibility of exceptions) all of those will fit nicely on the above drive layout strategy.