Best practices for managing multiple storage pools - avoid them

Nigel Poulton tweeted today:  "What are peoples thoughts on best practices for multiple pools on the likes of USP V and VMAX. Trade-off between perf vs resiliency etc..."

Good question Nigel, one of the biggest problems customers have is being able to fully utilize all their resources. It's not just that the ROI for storage tends to be underwhelming, but more frustrating is the fact that their storage was provisioned in a way that makes resources inappropriate or unavailable for the pressing needs at hand.

Pools are used two ways - to reserve storage capacity for certain functions such as snapshots or to create QoS levels for storage.  The difficulty lies that in the creation of pools for QoS, resources that are committed to pools are practically locked into them and cannot be easily redistributed to other pools to meet changing demands.  As storage systems age and are filled with data, the various pools are consumed unevenly. For example, consider an array with six pools provisioned as follows: 

Pool #1: SATA, 60TB (usable)  RAID 6  - primary bulk storage: no performance requirements

Pool #2  SATA, 40TB (usable)  RAID 10 - primary low cost storage: capacity over performance

Pool #3  FC/SAS, 30TB (usable)  RAID 5 - primary high performance storage: performance over capacity

Pool #4  FC/SAS, 20TB (usable)  RAID 10 - primary highest performance storage

Pool #5  SATA,  50TB (usable) RAID 5  secondary snapshot storage

Pool #6  FC/SAS, 60TB (usable) RAID 5 secondary snapshot storage

The type of problems storage administrators are constantly dealing with occur when one pool becomes maxed out - making it's associated QoS unavailable. The admin, then has three choices, 1) use a higher QoS, 2) a lower QoS, or 3) add new resources to the pool, if possible.  If they decide to use a higher QoS they may have to deal with performance problems in higher-priority applications.  If they decide to use a lower QoS they will have to deal with performance problems for the application. If they decide to add resources, they might have to interrupt many other applications and take them off line while workloads are shifted. Then you get the ripple effect of "remodeling the kitchen".

When you consider the fact that some storage systems force users to establish separate pools for thin provisioned volumes and thick volumes, the number of pools in the system increases and the fragmentation of resources becomes a much bigger problem.

The best practice for managing storage pools is to do away with them entirely so they don't inhibit access to expensive resources and, more importantly, so they don't soak up so much administrative time and create increased risk of downtime and data loss. 

Pools of disk drives are just a thin layer above bare disk drives where virtualization is concerned.  Considering the transparent nature of system virtualization technology it is almost incomprehensible that storage systems force customers to create these artificial constructs that force hard choices about something as basic as the layout of data on disks.  Vendors with pool-based volume management like to distract customers by talking about whiz bang functionality that doesn't address the core storage problem - the fact that their customers are still doing much of the work that the system ought to do for them.

The best practice then is to replace outdated storage designs with new designs that do not reserve storage resources in pools and do not use pools to create QoS levels. 3PAR InServ storage systems do not use pools and do not reserve capacity for different QoS levels.

3PAR InServ storage systems are used by many of the largest companies in the world, which saves them an enormous amount of money by lowering lower capacity requirements and administrator overhead. For example, Priceline.Com has been a 3PAR customer for many years and they talk about how it has worked for them in this video on YouTube.   

The InServ's data layout starts with the subdivision of all disk resources into 256MB "mini-disks" we call chunklets.  All the higher level RAID level functions in a 3PAR system are applied at the chunklet level, not at the disk level.  RAID in an InServ system is implemented as "micro-RAID" sets which then are concatenated together and formed into virtual volumes that are exported as LUNs.

FWIW, the term virtual volume was used by 3PAR years before the system virtualization phenomenon became the market force that it is today. I only mention this to reinforce the fact that from it's inception, the InServ internal storage architecture was designed to virtualize storage.  It makes storage administration transparent by doing the low level provisioning work on behalf of the storage administrator.

As new storage is provisioned in a 3PAR system, the data is spread across chunklets in small 16KB increments.  All the disk drives (of the same class SATA VS high performance) in the system are used by default, so that data is widely striped for optimal throughput and to avoid hotspots. While there actually are small amounts of capacity pre-allocated for use before storage is provisioned, this is done automatically by the system and it is done in thin slices across all drives.

There are no pools, no constraints, no weeks-long planning efforts needed for storage installations and change management. 

If you are looking to dump your nagging storage administration problems why would you ever go back to pool based storage when that is the root cause of your problems? 

Comments

[nate]

While from a physical perspective my 3PAR is one big storage pool, from a logical perspective I do split it up into multiple storage pools(3PAR's term is common provisioning group), mainly for organizational purposes, so I can see the space usage of all of the internal IT volumes, or all of the QA volumes easier, and set space warnings on the pool rather than the individual volumes.

I imagine some people do the same thing on other storage systems but because they aren't virtualized they have to dedicate spindles to the pools.

And it is nice to have the ability to have storage pools, a good example would be a rant I blogged about Equallogic storage a couple months back how as part of creating a volume you had to hard reserve some space for each and every volume when using thin provisioning, rather than just pointing it at a pool. Maybe pool is the wrong word here? I don't know, a pool can mean many things..

And as for 3PAR storage systems not using pools, it's my understanding if you have 10k RPM drives and SATA drives on one system for example you cannot stripe a volume across both, the CPG won't allow you to span drive types, so in that respect 3PAR systems do support pools/tiers/whatever you wanna call them!

But after all that you didn't mention dynamic or adaptive optimization? One of the things I like most about the 3PAR systems is if I make a mistake as far as storage layout (RAID type etc), I can fix it without touching the apps. I know some other systems can do it as well with varying levels of complexity(I recall in HDS's case you needed spare blank disks to migrate to as an example). All I need is just enough space to migrate a particular volume, don't need to worry about having X number of completely blank disks.

So along with the no planning and no constraints etc comes little/no stress. I can just go make the volume(s) and if I screw it up I can fix it easily.

[marc farley]

Thanks Nate, Actually, if you want to overlap different drive types for a volume you can, but the default is to restrict it to a specific device class - SATA, 10K RPM and 15K RPM. Best practices are to keep each drive class separate.

For people reading your comment that are not familiar with 3PAR systems, a common provisioning group (CPG) defines the parameters that the system uses to structure the chunkets that are aggregated into virtual volumes. The concept of a pool as a collection of disks that are organized by an administrator doesn't exist in a 3PAR system.

Yes, I purposely did not mention DO or AO. The post was getting long and I needed to finish it. But I think I'll take your prompt and work on some posts that describe how those functions work.

[Nigel]

Marc/Nate,

I'm liking the sound of fixing the RAID config of a volume if you initially made the wrong decision. I'd appreciate a bit more info on that... What are the limitations/restrictions..?

BTW Marc you once called me an HDS fanboy or cheerleader... I was nothing compared to what Nate is for 3PAR ;-)

[marc farley]

Hi Nigel, Well I suppose I could have called you something much worse! But then I would be ashamed of myself.

The software that is used to change RAID configurations is called Dynamic Optimization. I'll put together a post on it soon and let you know when it is posted.

[the storage anarchist]

Nice job making a mountain out of a molehill, but allow me to correct a few misleading misrepresentations of reality:

* Symmetrix VMAX and CX/NX all support non-disruptive changing of RAID types for "standard" (non-VP LUNs) today via the (no-charge) Virtual LUN (VLUN) capability. You only need sufficient capacity to form the new RAID sets.

* CX/NX and VMAX will also support non-disruptive relocation of a Virtual Provisioning LUN from one pool to another in the second half of this year (via a software upgrade)

* Adding devices to a pool on either CX/NX or VMAX does not require "weeks-long planning" - capacity can be assigned to a pool instantly via CLI or the management GUIs.

* Customers use pools not only to segment drive/RAID protection types, but also to segment applications from each other. Perhaps not so much an issue on your tiny little 3PAR arrays, but with arrays providing 2PB+ of usable capacity, the ability to insulate applications from one another is very important to many customers.

* And in fact, it is a simple operation to remove capacity from one pool and add it to another. Data can be "drained" off of drives onto space on the rest of the drives in the pool, and after adding to another pool the system will rebalance the space utilization across all the available spindles.

Most significantly, customers have confirmed that the rebalancing operation on VMAX, both manual and automated, has far less impact on application performance than does 3PAR AO or DO...and that the rebalancing (or re-RAIDing) is significantly faster on VMAX.

Personally, I think you're trying to make a big todo about nothing, and overlooking the more important impact on performance the 3PAR "no pools (but different drive types are seperated)" approach forces on customers.

[marc farley]

Hi Anarchist, You really ought to tell people that you are the strategy guy behind V-MAX at EMC when you comment. But that would be the same as shouting - "Pay no attention to the man behind the curtain." Wouldn't it?

It's nice to have you back in your fantasy world talking about futures, thumping your chest about how wonderful a presenter you are (your last blog post) and spreading incredible nonsense on competitor's blogs again! By the way, can we look forward to a future blog post of yours on how to dress for success?

Note to Scott Lowe, Anarchist's behavior is is exactly why your blogging context has changed and why people will look at what you write with much more scrutiny.

Barry, despite your attempts to position it as an advantage, application isolation is an EMC weakness disguised as a best practice to hide the fact that your back end is such a kludge. Something that customers continue to confirm with us every day!

What performance impact are you talking about here Barry? The fact that our 10 year old architecture - the same one that forced the DMX into early retirement - continues to put your not-so-shiny V-MAX architecture to shame? Or is it that despite all your statements to the contrary, it still takes an army of professional services to get it tuned up to do all the things you say are easy?

Nice try Barry, but when it comes to the InServ, you don't know what you are talking about. Confirmed by customers!

[John F.]

Hi Marc,

John F. (Whom works for NetApp, however the comments expressed in this reply are strictly my own and not endorsed, censored, or approved by my employer.) Hope to see you at Storebeers Sunnyvale edition next week. Some nice points. I'm still a bit fuzzy about the need to flip the RAID type, but perhaps you can explain that in a later blog.

Hi Barry,

Now you really confused me. Seems that Marc was talking about things that applied to ANY pool on 3Par; I may be slow, but I got that part. You, on the other hand, were baffling:

Could you please explain the following again?

*Symmetrix VMAX and CX/NX all support non-disruptive changing of RAID types for "standard" (non-VP LUNs) today

- Read thickly provisioned (Virtually Provisioned = thin, right?)

* CX/NX and VMAX will also support non-disruptive relocation of a Virtual Provisioning LUN from one pool to another in the second half of this year

- Read Vaporware

- Read thinly provisioned

* And in fact, it is a simple operation to remove capacity from one pool and add it to another. Data can be "drained" off of drives onto space on the rest of the drives in the pool, and after adding to another pool the system will rebalance the space utilization across all the available spindles.

- Is that for thick LUNs, thin LUNs, or is it just for vaporware?

Thanks much

John F.

[Bill]

I'm still trying to find out how an EMC employee considers himself an "Anarchist". Confusing...

[marc farley]

Hi John, I hope we do get to meet next week. I'm looking forward to it.

Your comment certainly points out the administrative challenges V-MAX poses. In trying to defend his product, EMC's Barry Burke exposes the Ever More Caveats (EMC) approach to filling out a feature matrix. Anyway, I appreciate your taking the time to underscore this in your comment.

Hi Bill, I think anarchist is a pretty good fit for Barry. Egotistical to an extreme without any regard for common conventions of fairness.

[the storage anarchist]

I should have known better...

But not to leave questions unanswered:

* Indeed, nondisruptive changing of RAID types on VMAX is today for thick (not THIN) devices;

* Moving Vitrual Provisioning devices from one pool to another is indeed not presently supported, but it is coming.

* Drain/Rebalance are supported today for Symmetrix Virtual Provisioning pools.

VP devices can be "thin" (allocate on demand) or pre-allocated - the latter used when you want to ensure that the capacity will be available when it is require (often used with databases). It is not necessary to pre-allocate the entire device - you can use both if desired (as with auto-extend on databases).

And Marc, look up Anarchy. The term has nothing to do with ego, extremes or common conventions of fairness. Instead it is a theory that regards the absence of all direct or coercive government as a political ideal and that proposes the cooperative and voluntary association of individuals and groups as the principal mode of organized society.

[marc farley]

Your handle is not the Storage Anarchy, it is the Storage Anarchist.

http://www.merriam-webster.com/dictionary/anarchist

1 : a person who rebels against any authority, established order, or ruling power
2 : a person who believes in, advocates, or promotes anarchism or anarchy; especially : one who uses violent means to overthrow the established order

But that's just a dictionary definition. There is very little consistency online about what anarchists and anarchism is. Certainly strongly individualistic, but otherwise the most insistently inconsistent bunch since cats.

[the storage anarchist]

I personally prefer the definition of "anarchist" as one who "proposes the cooperative and voluntary association of individuals and groups as the principal mode of organized society."

Sort of a kinder, gentler form of the angry mob Ochlocracy we often see in the storage arena (http://en.wikipedia.org/wiki/Ochlocracy)