StorageRap

Best practices for managing multiple storage pools - avoid them

Nigel Poulton tweeted today:  "What are peoples thoughts on best practices for multiple pools on the likes of USP V and VMAX. Trade-off between perf vs resiliency etc..."

Good question Nigel, one of the biggest problems customers have is being able to fully utilize all their resources. It's not just that the ROI for storage tends to be underwhelming, but more frustrating is the fact that their storage was provisioned in a way that makes resources inappropriate or unavailable for the pressing needs at hand.

Pools are used two ways - to reserve storage capacity for certain functions such as snapshots or to create QoS levels for storage.  The difficulty lies that in the creation of pools for QoS, resources that are committed to pools are practically locked into them and cannot be easily redistributed to other pools to meet changing demands.  As storage systems age and are filled with data, the various pools are consumed unevenly. For example, consider an array with six pools provisioned as follows: 

Pool #1: SATA, 60TB (usable)  RAID 6  - primary bulk storage: no performance requirements

Pool #2  SATA, 40TB (usable)  RAID 10 - primary low cost storage: capacity over performance

Pool #3  FC/SAS, 30TB (usable)  RAID 5 - primary high performance storage: performance over capacity

Pool #4  FC/SAS, 20TB (usable)  RAID 10 - primary highest performance storage

Pool #5  SATA,  50TB (usable) RAID 5  secondary snapshot storage

Pool #6  FC/SAS, 60TB (usable) RAID 5 secondary snapshot storage

The type of problems storage administrators are constantly dealing with occur when one pool becomes maxed out - making it's associated QoS unavailable. The admin, then has three choices, 1) use a higher QoS, 2) a lower QoS, or 3) add new resources to the pool, if possible.  If they decide to use a higher QoS they may have to deal with performance problems in higher-priority applications.  If they decide to use a lower QoS they will have to deal with performance problems for the application. If they decide to add resources, they might have to interrupt many other applications and take them off line while workloads are shifted. Then you get the ripple effect of "remodeling the kitchen".

When you consider the fact that some storage systems force users to establish separate pools for thin provisioned volumes and thick volumes, the number of pools in the system increases and the fragmentation of resources becomes a much bigger problem.

The best practice for managing storage pools is to do away with them entirely so they don't inhibit access to expensive resources and, more importantly, so they don't soak up so much administrative time and create increased risk of downtime and data loss. 

Pools of disk drives are just a thin layer above bare disk drives where virtualization is concerned.  Considering the transparent nature of system virtualization technology it is almost incomprehensible that storage systems force customers to create these artificial constructs that force hard choices about something as basic as the layout of data on disks.  Vendors with pool-based volume management like to distract customers by talking about whiz bang functionality that doesn't address the core storage problem - the fact that their customers are still doing much of the work that the system ought to do for them.

The best practice then is to replace outdated storage designs with new designs that do not reserve storage resources in pools and do not use pools to create QoS levels. 3PAR InServ storage systems do not use pools and do not reserve capacity for different QoS levels.

3PAR InServ storage systems are used by many of the largest companies in the world, which saves them an enormous amount of money by lowering lower capacity requirements and administrator overhead. For example, Priceline.Com has been a 3PAR customer for many years and they talk about how it has worked for them in this video on YouTube.   

The InServ's data layout starts with the subdivision of all disk resources into 256MB "mini-disks" we call chunklets.  All the higher level RAID level functions in a 3PAR system are applied at the chunklet level, not at the disk level.  RAID in an InServ system is implemented as "micro-RAID" sets which then are concatenated together and formed into virtual volumes that are exported as LUNs.

FWIW, the term virtual volume was used by 3PAR years before the system virtualization phenomenon became the market force that it is today. I only mention this to reinforce the fact that from it's inception, the InServ internal storage architecture was designed to virtualize storage.  It makes storage administration transparent by doing the low level provisioning work on behalf of the storage administrator.

As new storage is provisioned in a 3PAR system, the data is spread across chunklets in small 16KB increments.  All the disk drives (of the same class SATA VS high performance) in the system are used by default, so that data is widely striped for optimal throughput and to avoid hotspots. While there actually are small amounts of capacity pre-allocated for use before storage is provisioned, this is done automatically by the system and it is done in thin slices across all drives.

There are no pools, no constraints, no weeks-long planning efforts needed for storage installations and change management. 

If you are looking to dump your nagging storage administration problems why would you ever go back to pool based storage when that is the root cause of your problems? 

A conversation with Mark Cravotta of Datapipe

I caught up with Mark Cravotta from Datapipe recently at a 3PAR event in Las Vegas.  He's a high energy person who is having a lot of fun growing Datapipe's hosting and cloud computing services as well as helping to manage its expansion around the globe.  

Datapipe is a 3PAR Cloud Agile partner and customer who uses our products throughout their line for primary multi-tenant storage, data snapshots, remote replication and all aspects of disaster recovery.

In addition to being customer-driven, Datapipe is also committed to being a leader in green utility computing by reducing the carbon footprint of it's data centers through power purchases from green power producer, Constellation NewEnergy.  

3PAR customer Ultraspeed goes international with DR services between London and Amsterdam

London-based Ultraspeed has succeeded in the managed hosting business 13 years by being smart, opportunistic and service-oriented.  A lot has changed during that time, especially customer expectations for uptime and how much customers rely on their hosting providers to respond quickly when needed. Web sites that are lucky enough to "go viral" can be a disaster if the hosting company's infrastructure is unable to adjust rapidly enough to meet demand.

In March, Ultraspeed opened their second data center in Amsterdam implementing a modular infrastructure design including multi-tenant 3PAR storage, VMware, Extreme Networks switches and customized servers.  The highlight of their Amsterdam site is the ability to offer bi-directional DR services between London and Amsterdam using 3PAR's Remote Copy software. Ultraspeed is a member of 3PAR's Cloud-Agile program. 

In this interview, conducted in February 2010, Jordon Gross, CEO of Ultraspeed and Michael Shanks, CTO joined us for coffee near their offices and talked about their company's history, it's technology, the challenges they face and how they expect things to shape up in the years to come. 

Video demo of 3PAR's Plug-in for VMware vCenter and VMware Site Recovery Manager

Here is a video demo prepared by Maneesh Jain, one of 3PAR's Solutions Architects, of 3PAR's Plug-in for VMware vCenter and 3PAR's VMware Site Recovery Manager.  The Recovery Manager demo shows not only how to recover a VM, but also individual files within a VM. Enjoy!

Netapp & tiering - Just when they thought things were looking up...

Netapp has been on the hot seat ever since Tom Georgens, their CEO commented that tiering would soon be obsolete. Since then, a number of people have called him out on it, including yours truly (in a steering wheel cam), StorageBod, The Storage Architect, StorageZilla , a storage blogging wannabe, and last but not least, the Storage Anarchist.  To be fair, Georgens DID get support from the contrarian Drunken Data. 

At the end of his post, the Storage Anarchist asks:

Are they just defending their chosen path of Flash-as-cache instead of –as-a-tier? Or is it deeper than that? Is there something inherent in WAFL that makes it difficult to implement multiple different tiers within a single array? Or is it something else?

It's tempting to suspect WAFL's snapshot mechanism is the problem, but there is nothing about file level snapshots that would preclude storage tiering. Storage tiering depends on the ability to redirect block addresses across devices classes, which can be done at an abstraction layer below the file system level. In Netapp's case, the issue appears to be an interlock between WAFL and Netapp's underlying RAID layer.  So I'd say its mostly a Netapp RAID problem.

As writes come into a WAFL system, they are first staged to NVRAM in order to eliminate parity RAID write penalties and then they are written to "nearby" blocks using a tightly coupled relationship between the file system and its underlying RAID subsystem.  This design has an unusual and intricate knowledge of disk drive operations and status within the RAID array.  In other words, the file system in a Netapp machine is intricately coupled to the physical characteristics of the underlying storage hardware, which means creating block abstraction layers is highly improbable. The text-image below is from Netapp's Patent filing  6,138,126 dated October 24, 2000.

Now it's possible that this patent does not indicate the implementation within Filers today, but I'd say there is a good chance it explains Netapp's reluctance to embrace tiering. If this turns out to be Netapp's tiering shortcoming, Netapp would need to virtualize their RAID implementation in order to get to a point where they could start working on tiering. 

Holy cow!!  Is it possible that Netapp is actually THAT FAR behind in storage virtualization - not to mention the next wave of the technology - tiering?

If this analysis is correct, it may be that the only way to get tiering (not caching) with a Netapp system is to connect their V-Series filer to a third party array that offers tiering. If that's the case, will Netapp support it, seeing as how they don't have it on their home-grown Filers?

The hot seat could get a lot hotter.

A conversation with Jordan Gross & Michael Shanks of Ultraspeed

Ultraspeed describes their hosting business as providing enterprise level solutions at mid-market prices.  In this interview, Jordan Gross and Michael Shanks of Ultraspeed talk about how they do it; from running diskless, DC-powered servers, to implementing virtual systems with 3PAR virtual storage and operating data centers in London and Amsterdam.

A conversation with Nick de Beer of SYSDBA

SYSDBA is our excellent 3PAR business partner in South Africa,  Some of their team came to 3PAR  headquarters recently and I had the chance to record this interview with Nick de Beer, a technology expert with a wide range of experiences including Oracle, storage and virtualization.  In this interview, Nick talks briefly about 3PAR's Recovery Manager software, which allows customers to recover data from many point in time snapshots.

3PAR Countdown: New Plug-Ins for vSphere and vCenter Server

The news is here:

3PAR Tool Talk: Vish Mulchand on Synchronous Long Distance Remote Copy

Earlier this week, we announced our new Synchronous Long Distance Remote Copy software.  It lowers the cost of long distance disaster recovery two ways:

1. Reduces communications costs by using lower cost asynchronous communications to a far away DR site while using a high-speed local or metropolitan link for synchronous copies.
   
2. Let's customers mix and match enterprise and mid-range arrays in the configuration.

Staying thin - a new storage lifecycle choice

Another way to understand what 3PAR announced yesterday is to think of the life cycle of storage. 3PAR clustered arrays now offer new methods for removing wasted capacity from storage and recycling it so you can make use of it again for more pressing needs.

First, lets start with Thin Provisioning, which is a way to match the consumption of storage resources with its demand. Non-thin provisioning storage makes you guess during installation what you will need storage  resources for.  Once the storage has been allocated to a volume in a thin provisioning system it is no longer available to be used by another application.  Thin provisioning continues to work as the volume ages, but its effectiveness depends on how efficient the file system (or database) is choosing which blocks to write to.

There - that's the old stuff.

Now for the new stuff.

First is a technology 3PAR calls Thin Conversion.  This is a way to make a block copy from a volume or system that wasn't thinly provisioned and put it on a 3PAR array so that it is thinly provisioned.  In other words, it removes all the non-utilized space that was never written to - which can be a lot. This is a one shot deal that gives customers an immediate payback by not having to buy as much storage when they are refreshing their storage technology.   

Second is Thin Copy Reclamation.  It sounds a bit boring but storage admins familiar with the problem of trapping storage capacity in snapshot or remote copy pools know that this is actually a very big deal. It can make the difference between continuing to work with what you've got and being forced to buy more storage than you planned for.  This is something that storage administrators are likely to use repeatedly, over the life cycle of the volume or array, to maintain the balance of resources in their 3PAR arrays.

Third is Thin Persistence.  Like Thin Copy Reclamation, Thin Persistence is something that storage admins will use periodically through the array's life cycle to remove wasted storage. File system tools are used in conjunction with a 3PAR array to identify blocks that were previously allocated but have since been deleted by the file system. Thin persistence is designed to recycle these blocks.

Fourth is Thin Reclamation for Veritas Storage Foundation.  This is also another full life cycle technology that was developed by 3PAR and Symantec where a storage protocol command is used to transfer deleted block information from a Veritas Storage Foundation file system to a 3PAR InServ array. Symantec is most definitely leading the way with thin-aware file system technology.

So there you have it - three new ways to keep storage thin and one new way to get onto a storage platform that offers you these important storage life cycle options.

Data replication the amazingly fast and easy way

3PAR's Remote Copy software broke all the rules for the cost and effort required to make remote data replication work.  Huge savings in cost, time and capacity.

3PAR Countdown: Exchange Service Provider LiveOffice saves buku bucks with 3PAR storage

The news is here:  In addition to saving serious scratch with a 3PAR clustered storage array with reservationless thin provisioning, LiveOffice uses 3PAR's Virtual Copy software to quickly and easily restore their customer's Exchange mailboxes.

New 3P song suggests "Store heads" get their data on the cloud

Everybody has an opinion these days about Cloud Computing and Cloud Storage.  3P tells "Store Heads" in his new video below that they ought to start investigating how to do it.  The equipment used by cloud service providers matters a great deal to their success.  Technologies such as reservationless thin provisioning, full array wide striping and dynamic, load-balancing, active/active controllers make the cloud experience much more satisfying.  3PAR partners with Cloud Service providers through a program called Cloud Agile to bring 3PAR array technology to cloud computing and storage customers.  

Watch:

Clearing up the Compellent Confusion

The storage industry can be confusing - even to insiders.  For example, people often compare 3PAR and Compellent because both companies are "newer" vendors working their way up against established competitors.  

But that's about where the comparisons end.  3PAR arrays scale to performance levels that far exceed Compellent's.  While it may be possible to attach the same number of drives to both arrays, when you push them both for total throughput, 3PAR crushes Compellent.   With 100+ drive systems, there is no comparison.  If you think you might need hundreds of drives, instead of less than 100, it's a slam dunk for 3PAR.

With less than 100 drives, our product lines overlap and that's probably where the confusion comes from because 3PAR's F-Class arrays compete with Compellent's. Our smallest F Class - the F200 scales from 16 to 192 drives and its bigger brother the F400 scales from 16 to 384 drives (with the addition of 2 more mesh active controllers).  FWIW, you won't saturate a 3PAR controller by loading too many drives on it - the F Class controllers can handle 192 15K RPM drives with overhead to spare. Realistically, its probably not fair to Compellent to compare any of their systems to one of our quad controller F400s. 

Nobody handles mixed workloads like 3PAR.  Which means customers can put many different applications on our arrays and get excellent performance from their systems at very high disk utilization levels.  Don't just take my word for it, see what our customers have said on TechValidate. (will require a login - it's not a 3PAR site).  The key to this is the fine-grained allocations of storage combined with true wide striping - not based on pools and metas and hypers and such.

Both companies have delivered important storage virtualization innovations to the industry.  3PAR didn't invent thin provisioning, but we forced the issue in the industry and now it's a mandatory feature in all enterprise arrays.  FWIW, our work in storage efficiency and utilization technologies continues and you can expect to see more from us in the future that allows customers to get the most from their capacity.  We are always happy when customers dig deeper into this technology and compare our reservationless thin storage with any other vendor's thin implementations. 

In Compellent's case, the delivery of Data Progression has been a key stimulus to the rest of the industry.  EMC's FAST technology, due in its first release late this year is a clear attempt to catch up with Compellent.   However, just as Thin Provisioning is not a 3PAR invention, the ability to move data between storage tiers is hardly a Compellent invention.  When I was at EqualLogic, the product had the ability to move data automatically between tiers of storage spanning different systems. Their evacuate and replace capabilities rely on their automated tiering technology. 

3PAR introduced it's Dynamic Optimization (DO) product in 2005, which redistributes data from one set of resources across an expanded or different set of resources. For instance, today a customer can add new drives to a 3PAR system and redistribute volumes across all the drives in the system using DO. They might do this for either capacity or performance reasons - or both. With storage capacity needs continuing to grow, it's important to be able to scale with performance and agility.  When you add new drives to a 3PAR system, their capacity is a reservationless resources that can be used by any application, on demand - including for use as snapshots for existing or new applications.

Just in time groomin' behind the wheel

A SWCSA video talking about just in time grooming and 3PAR's reservationless approach to resource utilization in their InServ storage arrays. Thin provisioned storage and snapshots (3PAR's Virtual Copy software) are assigned storage resources at the time they are needed - and not before. That means that Administrators don't have to guess which pools to put disk drives in - or how to structure hypers and metas with 3PAR.