StorageRap

Clarifying VAAI capabilities and implementations

Virtual Geek and I had a discussion on his post yesterday about vSphere's VAAI capabilities announced yesterday.

I wrote about the fact that we already had zero detect technology in our product, which is useful for the new Full Copy command because it allows customers to remove zeroed data from clones when they are created and return them to array free space.

The discussion became a bit confused when Chad interpreted what I was saying as pertaining to Block Zeroing.

Block Zeroing and Full Copy are different aspect of the VAAI API.  The intent of block zeroing is to reduce the amount of CPU effort and storage traffic required to write zeroes across an entire EagerZeroThick (EZT) VMDK when it is created.  The intent of Full Copy is to make clones of VMs quickly without consuming I/O bandwidth.   Things get interesting when you start thinking about making a full copy of an EZT VMDK that was created using VAAI with block zeroing - but I'll discuss that later.

I also want to clarify what zero detection technology is. 3PAR T and F class arrays have zero detection technology, which is enabled by Thin Persistence software, that recognizes zeroed blocks as they are read by the array and returns them to the array's free pool.  Any read requests made to these block addresses will return a zero value. In essence it is dedupe for zeroes.

However, Zero detection is not needed when an EZT VMDK is created using the VAAI plug-in because the array will recognize the intent of the command and not write the zeroes. In other words, the VMDK will only contain a very small amount of reserved space when it is created. Again, any attempts to read blocks in those ranges will return zero values. Zero detection is effectively bypassed during the creation of the EZT VMDK.

The exception to this behavior is when the EZT VMDK being created is written to a thick volume - in that case the array will write zeroes across the entire VMDK. 

The remaining cases for the creation of EZT VMDKs on 3PAR arrays occur when the VAAI is not used. For a thick volume, the entire VMDK has zeroes written to it.  Thin volumes not using zero detect also have zeroes written over the entire VMDK. Thin volumes with zero detect will not have zeroes written to them and will contain only a small amount of reserved space. 

FWIW, the reserved space is used as instantly-available capacity that can be allocated on-demand when writes start coming into the volume.  3PAR arrays always "read ahead" free space to improve the performance of thin provisioning.

The next bit here could be a bit thorny, so clear your head. The matter of making a Full Copy of an EZT VMDK to a thinly provisioned volume was something Chad said was not allowed.  My assumption here is that the type of thin provisioning used makes a big difference.

For instance, if you are using TP from VMware, I could see where they would not allow a full copy to be made. The problem is that the full copy will return all the zero values for the source VMDK, whether or not those zeroes were ever actually written - and write them to the target TP volume.  In other words, the target could be much larger than the source. In the VMware TP scheme, this could make for problems in a hurry if you were making a bunch of clones this way.

In contrast, if you were using a 3PAR array with zero detection, the Full Copy of the source VMDK would return zeroes for the entire VMDK, but the zero detection would strip them out again as the target was being written.  You could make as many clones as you wanted this way, knowing that the physical capacity they consume would be a multiple of the physical capacity consumed by the source VMDK. In other words, you wouldn't have to worry about virtual zero bloat making a mess of your VMFS volume.

One of the big differences between 3PAR's zero detection technology and other vendors zero-reclaim technology is that 3PAR's process is real-time-on-ingestion as data comes into the array, whereas zero-reclaim works in a post processing fashion after the zeroes have already consumed disk space. This could be a significant difference in many cases because the post-processing method has the potential to create unexpected capacity-full conditions before the zero-reclamation process even has a chance to start.

vSphere 4.1 Array Integration on steroids

We've been anxiously waiting for VMware's announcement of vSphere 4.1 for weeks.  There are many big things in this release, including significantly scaling the management capabilities of vCenter and increasing the number of simultaneous vMotions that are supported.  The door is open for ESX deployments to achieve much greater densities than they could previously and that's a big deal to large enterprises who want to get more resources under the control of fewer points of management. There are still great gains to be made in consolidation - more on that later.

In the storage world, there are a couple big things, SIOC and array integration through the VAAI API.  Technodrone has put together an excellent post on SIOC and I highly recommend that anyone wondering how to make this functionality works should go to this post and read it. Array integration has been advanced in three ways:

• Hardware assisted locking
• Full copy
• Block zeroing

Array integration through the VAAI API is already at a very advanced status at 3PAR with some of the most important functions implemented through our I/O co-processor ASIC.  While some companies want to write off the importance of hardware, 3PAR believes there are many things that need to be done in hardware to get the performance needed to truly scale storage for virtual environments.  Our co-processors are key to getting much greater storage utilization and higher VM ratios and are one of the 3PAR innovations that separate our best of breed products from everybody else.  The capabilities discussed below are available in the hardware today, and will be enabled with a software upgrade in September.

OK, lets talk about hardware assisted locking first. For customers that have experienced locking problems, this is a big deal. The problem has been well-documented online - but in a nutshell, customers have run into problems where an operation that locked the LUN for a VMFS did not complete, thereby freezing all I/Os for all systems using that LUN. That was certainly a nasty problem - not a bug necessarily, but certainly an incredible pain in the rear to all involved.

VMware's response in vSphere 4.1 was to include a command in the VAAI API using an atomic test and set instruction for implementing fine grained locks for small block sizes. There will still be locking in VMware, but on a much smaller scale.

Unique to 3PAR is the fact that this new locking mechanism is implemented in our I/O co-processors where it completes very quickly, as opposed to implementing it in code in the controller. If you consider an environment with high VM ratios and multiple vMotions going on you want this granular locking mechanism to as quickly as possible. Nobody else comes close to the speed that 3PAR processes them.

Next is the new Full Copy capability - also with co-processor assistance to reduce the capacity of the copy that is made. 3PAR has zero detect and reclaim technology integrated into the co-processor. With zero detection running in an array, as new writes are made, strings of zeros are detected by the co-processor and those blocks are returned to free space inside the array.  If future reads are made to those blocks, zero values are returned, but not from disk. The result is that copies of VMDKs with lots of zeros in it, will be much smaller after the copy is made - and the copy will proceed much faster.

This sort of functionality works amazingly well with EagerZeroThick (EZT) volumes in vSPhere.  VMware requires EZT for Fault Tolerance (FT) and MSCS clusters and also recommends EZT for high performance. The main complaints about EZT are that it takes extra time to write all those zeroes when the VMDK is created and that it doesn't work well with thin provisioning. With 3PAR's zero detection, the time it takes to writes all those zeros and the space they consume is a non-issue, but more on that later.  Virtual Geek at EMC wrote about VAAI today and in his discussion of what does not work for full copy he mentioned copying from an EZT volume to a thin provisioned one.  Actually, he's wrong about that where 3PAR is concerned, because EZT to Thin works very well on a 3PAR array with zero detect.

The image below illustrates the advantages of using EZT on a 3PAR array with zero detection:

The last API element to discuss is Block Zeroing. The idea is that the host communicates to the array to write a string of zeros when it is provisioning storage or overwriting blocks to a non-EZT VMDK.  vSphere writes a lot of zeroes in order to maintain data integrity with multi-tenancy. The hypervisor zeroes zero out blocks prior to writing them in order to ensure that a virtual data imprint from an old VM does not occur for a new VM.

But writing all those zeroes consumes CPU and I/O bandwidth that could actually be used productively, so VMware included a new command to offload the host from writing zeroes, effectively shunting that workload to the array. Voila - problem solved with 3PAR!!  The zero detection and reclamation technology in a 3PAR array not only offloads the host from writing zeroes, but it also gives customers instantaneous reclamation of capacity with a smaller digital footprint (less capacity consumed) and faster performance. That's pretty cool and it's a trifecta that only 3PAR  has.

What is amazingly cool about today's vSphere announcement for 3PAR customers is that all three API elements, hardware assisted locking, full copy and block zeroing are already implemented in 3PAR's T and F series hardware platforms, and will be usable by the end of September with a firmware upgrade.

Our co-processor architecture really delivered for us this time. But it's been delivering the goods for our customers for a long time already. In virtualized environments our customers tell us they double their VM density, while cutting their storage capacity in half - all while reducing the amount of storage administration necessary by 90%. Those stats can be hard to believe, but when you look at what we delivered on the first day vSphere 4.1 was announced - when most people didn't even know we were working on it - it might make it easier for people to understand why. 

We take virtualized environments very seriously. People that don't know about 3PAR don't consider us to be a leader in virtualization, but when they find out the depth of technology we have and how well it works across our entire product line they understand we are leading in ways that really pay off for them. And the bigger they are, the bigger the rewards can be - especially after today.

Microsoft's mediocre cloud appliance

A short while ago, Microsoft announced their plans to create a stack platform appliance for Azure.  This was done, presumably, to deal with the competitive threat from VMware, which is part of the V-Block-centered Acadia business venture.  Sorry Microsoft, but IMHO, this is a clear signal that the cloud stack has been catapulted to the stratosphere of hype. 

There are a couple assumptions about clouds in a can that show the ready-fire-aim nature of these types of solutions.

The first is that clouds in a can will provide an on-ramp for private cloud infrastructures to to use public cloud infrastructures in the future.  Oh yes, and how exactly will that work?  Vaporware drawn on white boards with clouds in the middle?  The allure of an easier future through stacks rivals any of the malarkey that our industry has ever produced.

The second assumption is that customers will save money with stacks and appliances. There are too many vendors involved, all of them hoping to increase revenues. Most of these integrated solutions will have surprisingly high price tags.  Of course, there will be loss-leader infrastructure in a can products too.  THAT should be interesting to watch.

The idea of a turn key infrastructure is appealing as a concept, but the real cost of operating an infrastructure is where the advantages lie.  Some customers will be happy with expensive mediocrity that traps them. 

In contrast, Best of breed products today are those that are powerful, flexible and easy to use.  These are the products to build a private cloud with. Take your infrastructure where it needs to go instead of driving somebody else's vehicle down their highway.

Cisco sneaks into the corporate laptop/desktop market with CIUS

I've been going slightly nuts since yesterday after Cisco announced the CIUS.  It looks like the perfect tablet for the sorts of things I really want a personal screen device for - communicating with other people.  This review by Erik Parker of InfoWorld is a pretty good read and it summarizes key advantages and disadvantages of CIUS.  If it can make the technology of video conferencing transparent to end users, it will be a big deal. 

But the hidden story to this is that Cisco is also making a play to get into the corporate desktop/laptop business with the CIUS.  The idea that companies could deploy these with VDI is definitely part of Cisco's grand plan for world domination. Whether or not the CIUS could replace laptop or desktop computers remains to be seen, but there are reasons to think they could eventually if the stars align. 

The arguments for VDI are strong, but there are still a lot of hurdles to overcome, such as back end storage performance to support boot storms. By the way, people looking at large VDI implementations might want to look at 3PAR's wide striping storage systems to get the sort of affordable IOPS needed to support large VDI environments. My previous post illustrates our design for massive throughput, which supports a huge number of IOPS without needing SSDs or requiring storage administrators to create special disk pools to isolate the VDI workload from other applications running in the same storage array.

Visualizing real storage virtualization

Steve Taylor, one of our SEs, created an animation that shows the multiple layers of virtualization that create the natively wide-striped data layout on a 3PAR storage server.  I think it's the coolest thing I'd seen since joining the company that quickly summarizes the multiple layers of virtualization in a 3PAR array. 

All the functions shown are automatically done for the customer with minimal administrative effort. 3PAR customers do not spend time planning the layout of special disk pools or preparing their disk drives configurations for certain functions.  All they do is select the drive class and the RAID level for the volume they are creating and the rest of the data layout work is done for them.

The demo shows how a RAID 5 3+1 virtual volume is created, what it does not show is the way other volumes would be created using different RAID levels over the same set of resources. It would be a replay of this, but with a different RAID level applied - everything else would be the same.  

Not only does this design provide massive throughput, it also responds very quickly when customers need to add volumes. It's like driving a freight train that can corner. Try doing that with your v-Max on anything but a test track.  

The absurd prognostications for storage in a stack

 

How is it that some people possess the gift of foresight and the ability to predict the future?  Some say they have dreams or visions, some extrapolate from experience and logic, while others make predictions hoping to fulfill an agenda. Then there is the element of public exposure. Is the predication public and do they use their real name or hide behind an alias? 

Nicholas Carr took was very public and very open when he wrote his breakthrough book "Does IT Matter?".  In it, he stated that there are no sustainable advantages to be gained by a company through the implementation of information technology.  He argued that any short term gain can be matched by competitors in a relatively short period of time with lower capital investments - effectively punishing companies for innovating. He recognizes the necessity of having IT in order to stay competitive, but finds it difficult to justify being an early adopter of technology.

Since Carr published his book, we've seen a lot of change in IT markets, including the rapid deployments of virtual systems technology and the expansion of hosted, utility computing and all things "cloud."  But the biggest changes have resulted from the global financial crisis, forcing companies to reduce non-essential costs significantly - especially IT costs.

Unfortunately, not every technology implementation intended to reduce costs has been successful. And that's one of the things that makes the information technology business so fascinating and perplexing - intelligent people with deep expertise in technology fail to predict the ways that things can go awry and what the cost of their shortsightedness will be. 

The rich history of failed IT projects is exactly why there is so much FUD spread by the competitors in our industry - FUD gets customers thinking about the consequences of their purchase decisions and all the possible problems that can result from an error in judgment. It also contributes to the interest in the machinations of our industry and the "war games" that are played out in traditional and social media. Whether we are predicting changes to the industry through mergers and acquisitions or the development of new business models, it all flows into the river of FUD at purchase time.

With the abundance of FUD, one naturally develops an aesthetic for the stuff to cull the weak from the strong. For example, a piece of weak FUD recently appeared online on Silicon Angle titled "Why Netapp Must Seek Acquisition", written by the poser "secretcto".  The author starts with the suggestion "let's take a look at the market cap of each of these players" and then neglects to make any comparisons. It goes downhill from there, reaching its lowest point when the article referred to Nicholas Carr as Daniel Carr and then failed to negotiate the transition of whether or not IT matters to cloud service providers. The tipping point for Carr's logic is that to service providers, IT absolutely does matter because operating data centers is their core business.  

By contrast, you barely notice good FUD, it has a smooth logical flow and subtly builds to a persuasive conclusion based on a key point that usually has it's origins in a subjective opinion or bias. A decent example of good FUD was Chris Mellor's recent piece about the Storage Array Killing Fields qualifies as good FUD.  Chris doesn't have an axe to grind, but he is a journalist and therefore has the responsibility of stirring the pot.  It's a well written piece that uses an analogy that compares the selection of equipment for data centers with the selection of components used in an automobile.

The problem is that automobile manufacturing is a poor analogy for running a data center. When a car rolls of the manufacturing line it is shipped to a dealer, sold to a customer who drives it away.  There is nothing about the experience of making, selling or buying a car that is even closely related to the constant ongoing data processing services that are provided by a utility or cloud service provider. 

A better analogy is running a restaurant. Restaurants succeed or fail based on the quality of their customer service and that's why chefs like Thomas Keller strive to maintain consistent, excellent quality every minute of every day they are open.

Should we expect the recipe for success in hosting and cloud services to be any different? This recent article in Information Age states that 71% of the 450 CIOs in a KPMG survey want to improve the price to quality ratio of their outsourcing contracts. The dynamics of the business relationship between CIOs and their utility/cloud service providers are going to be the same. Service providers with the best reputations for customer service are going to thrive. Those that don't measure up will fail.

Vendors of consolidated stack solutions of servers, storage and software are trying to convince customers that the "All-in-one" stack solution is the safest way to proceed during the transition period while cloud computing is emerging. They would have you believe that the biggest risk in operating a data center is in ordering the products and getting everything installed initially. Considering that customer metrics for utility/cloud service providers will be responding to the needs of their customers quickly and accurately, the lions share of the risk will come well after the initial installation during the life of the service engagement.

The weakness of the All-in-one approach is that it does nothing to address the dicier aspects of owning, operating and changing an IT infrastructure after it is up and running. In many cases the stack vendor's answer to change management will be the same as it is today - time-consuming and expensive professional services.  There are definitely utility/cloud service providers that will want this sort of service, but many would prefer to do it themselves at much less cost. That's what you do when your primary business is running a data center.

A talented chef can find a way to prepare a gourmet meal on an Electrochef All In One Kitchen, but they would never decide to run their business with them. They are going to select best of breed appliances and equipment that best fit their needs and enable them to prepare quality dishes in a quality fashion.

So the question for the utility/cloud data center operator then is - "what is best of breed equipment for my business?" 

The classic clash between Best-of-breed and All-in-one solution pits cost against complexity. Best-of-breed technology has traditionally been more customizable to fit a wider range of requirements and therefore has been more complicated and expensive to operate. In contrast, All-in-one technology has traditionally been cheaper, limited to a smaller set of functions and easier to operate.

Unfortunately, neither stereotype works very well for the utility/cloud service provider. They need fully functional products that are also easier and quicker to operate. Fast, accurate change management and operator efficiency are the key elements for utility/cloud infrastructure products. 3PAR's Best-of-breed storage products have these characteristics as well as being extremely space-efficient and high-performing. Customers appreciate the amount of time they do not spend managing their 3PAR storage while they are getting the job done. When a new order comes into a 3PAR kitchen, the system is ready to go right away - including tasks that take a long time to set up on other storage, such as Remote Copy.

And what about the All-in-one stacks in the market?  Surprisingly, unlike traditional All-in-one solutions, they are more expensive to install and operate.  Change management is complex, which leads to relatively poor operator efficiency and the engagement of professional services, which does not necessarily speed up the process. The traditional benefits that All-in-one solutions typically provide are not part of these stack solutions.

The predictions for stacks taking over the market are all wrong. Sure, there will be stack solutions sold and it will take time for all of this to sort itself out as it always does when an industry is going through major, fundamental changes. The most important changes that will occur in the years to come will be driven by the service demands placed on utility/cloud service providers. Customers of utility/cloud services want their money's worth and the best service providers will do what it takes to give it to them. Stacks add no value in that equation.

Best practices for managing multiple storage pools - avoid them

Nigel Poulton tweeted today:  "What are peoples thoughts on best practices for multiple pools on the likes of USP V and VMAX. Trade-off between perf vs resiliency etc..."

Good question Nigel, one of the biggest problems customers have is being able to fully utilize all their resources. It's not just that the ROI for storage tends to be underwhelming, but more frustrating is the fact that their storage was provisioned in a way that makes resources inappropriate or unavailable for the pressing needs at hand.

Pools are used two ways - to reserve storage capacity for certain functions such as snapshots or to create QoS levels for storage.  The difficulty lies that in the creation of pools for QoS, resources that are committed to pools are practically locked into them and cannot be easily redistributed to other pools to meet changing demands.  As storage systems age and are filled with data, the various pools are consumed unevenly. For example, consider an array with six pools provisioned as follows: 

Pool #1: SATA, 60TB (usable)  RAID 6  - primary bulk storage: no performance requirements

Pool #2  SATA, 40TB (usable)  RAID 10 - primary low cost storage: capacity over performance

Pool #3  FC/SAS, 30TB (usable)  RAID 5 - primary high performance storage: performance over capacity

Pool #4  FC/SAS, 20TB (usable)  RAID 10 - primary highest performance storage

Pool #5  SATA,  50TB (usable) RAID 5  secondary snapshot storage

Pool #6  FC/SAS, 60TB (usable) RAID 5 secondary snapshot storage

The type of problems storage administrators are constantly dealing with occur when one pool becomes maxed out - making it's associated QoS unavailable. The admin, then has three choices, 1) use a higher QoS, 2) a lower QoS, or 3) add new resources to the pool, if possible.  If they decide to use a higher QoS they may have to deal with performance problems in higher-priority applications.  If they decide to use a lower QoS they will have to deal with performance problems for the application. If they decide to add resources, they might have to interrupt many other applications and take them off line while workloads are shifted. Then you get the ripple effect of "remodeling the kitchen".

When you consider the fact that some storage systems force users to establish separate pools for thin provisioned volumes and thick volumes, the number of pools in the system increases and the fragmentation of resources becomes a much bigger problem.

The best practice for managing storage pools is to do away with them entirely so they don't inhibit access to expensive resources and, more importantly, so they don't soak up so much administrative time and create increased risk of downtime and data loss. 

Pools of disk drives are just a thin layer above bare disk drives where virtualization is concerned.  Considering the transparent nature of system virtualization technology it is almost incomprehensible that storage systems force customers to create these artificial constructs that force hard choices about something as basic as the layout of data on disks.  Vendors with pool-based volume management like to distract customers by talking about whiz bang functionality that doesn't address the core storage problem - the fact that their customers are still doing much of the work that the system ought to do for them.

The best practice then is to replace outdated storage designs with new designs that do not reserve storage resources in pools and do not use pools to create QoS levels. 3PAR InServ storage systems do not use pools and do not reserve capacity for different QoS levels.

3PAR InServ storage systems are used by many of the largest companies in the world, which saves them an enormous amount of money by lowering lower capacity requirements and administrator overhead. For example, Priceline.Com has been a 3PAR customer for many years and they talk about how it has worked for them in this video on YouTube.   

The InServ's data layout starts with the subdivision of all disk resources into 256MB "mini-disks" we call chunklets.  All the higher level RAID level functions in a 3PAR system are applied at the chunklet level, not at the disk level.  RAID in an InServ system is implemented as "micro-RAID" sets which then are concatenated together and formed into virtual volumes that are exported as LUNs.

FWIW, the term virtual volume was used by 3PAR years before the system virtualization phenomenon became the market force that it is today. I only mention this to reinforce the fact that from it's inception, the InServ internal storage architecture was designed to virtualize storage.  It makes storage administration transparent by doing the low level provisioning work on behalf of the storage administrator.

As new storage is provisioned in a 3PAR system, the data is spread across chunklets in small 16KB increments.  All the disk drives (of the same class SATA VS high performance) in the system are used by default, so that data is widely striped for optimal throughput and to avoid hotspots. While there actually are small amounts of capacity pre-allocated for use before storage is provisioned, this is done automatically by the system and it is done in thin slices across all drives.

There are no pools, no constraints, no weeks-long planning efforts needed for storage installations and change management. 

If you are looking to dump your nagging storage administration problems why would you ever go back to pool based storage when that is the root cause of your problems? 

Customer Priceline talks about 3PAR storage

Here's a video that TechTarget produced for us with one of our customers, Priceline.com.

Here are a few highlights from the video:

Priceline.com was one of the first e-commerce players to adopt virtualization.  That may account for why the company's IT organization is known for for it's high availability and ability to adapt quickly to changes in the market.  Given the fact that their business has a broad value-based appeal, their IT organization works very hard to get the best rate of return for their capital expenditures.

3PAR storage allowed them to increase their storage capacity over 400% over the last four years while reducing the administrative load required to manage it all. Ron Rose, ex-CIO at Priceline (now on the Sr. Management Team at Dell) said that they were able to decrease the data center footprint 50% during that time.  Mr. Rose estimated that they were able to reduce the deployment of approximately 100 physical servers and their associated footprint costs, which were equivalent to 106 acres of trees 310 tons of hydrocarbons per year.

How would you build an iBlock?

 
I just read an article about how the concept of infrastructure blocks is playing out on the SearchDataCenter site.  The article presents several perspectives, but it's a bit confused. The concept is referred to as three different terms (pods, blocks and cells) and the comparison between a making your own and buying one are not clearly juxtaposed.  Regardless, its a thought provoking article.

But is does raise the point what should we call these things?  I think a better generic word for them is iBlock, short for infrastructure block.  

I've been speaking to customers about this sort of thing lately and a number of them have expressed the opinion that rolling out their own iBlock would be a lot cheaper, more flexible and more scalable than anything they could buy from a vendor.  I'm a big believer in the power of integration, but it's possible to get too far ahead of the curve.

3PAR customers have already been implementing iBlocks for several years using the 3CV design discussed in this ESG Labs report.  That's one approach.  The question is, if you were going to build an iBlock, how would you do it - and why?

VPLEX undressed

OK,  Monday's post lacked the punch that people have come to expect from me where EMC announcements are concerned.  Thanks to my readers who were disappointed and told me so. This post is for you. 

 

VPLEX (announced Monday with all the hype that EMC could muster) is the result of EMC trying to become something more than a storage company. They put a systems professional in charge of developing storage products and produced a science project that looks very attractive as a technology but relegates storage to a secondary role and questions EMC's commitment to solving storage problems. In fact, VPLEX probably creates more storage problems than it solves. More on that later.

The concept of teleportation for virtual machines resonates with childhood space fantasies. I remember summer afternoons from my childhood going out to a nearby field with my brother and his model rocket club. My brother built some stunningly cool looking model rockets, and the amazing thing was that they all went where he intended them to - up. Just as amazing was the fact that the engine always ejected the nose cone and parachutes to bring his creations back to earth gently.  It was a riot running across that field trying to catch them in our bare hands as they descended. A few years later those rocket designs turned into a set of World Book Encyclopedias that he won at a big science fair in Michigan.  I was prouder of him than any kid brother could be, even though I was just the rocket runner.

However, his rocket club also had a couple of less-talented rocketeers who also launched their contraptions with great fanfare and, quite often, spectacular disappointments. Engines became separated from the rockets holding them, shooting around at random on the ground - and then there was the memorable starship design that took a 90% turn after exiting the launch wire and proceeded to fly parallel to the ground at height of 4 to 5 feet, sending parents, siblings and friends scrambling for cover behind a shrub, pet or each other.

It's good to believe in some things and to be wary of others.

 

So what is VPLEX anyway?

For starters, VPLEX is a virtualization appliance - a bump-in-the-wire product. EMC says it will take the technology at some point in the future and build it into their arrays, but like a lot of things, its much easier to talk about it than to actually do it. The irony of the situation is that it VPLEX fills a storage virtualization role that is similar to IBM's SVC - a product that EMC bloggers eagerly lampoon. EMC is downplaying the similarity to SVC and are spinning VPLEX as a teleportation device for your virtual machines.  Of course, the image of teleportation they want you to think of comes from Star Trek, but sci-fi fans are familiar with the risks of teleportation too - just for fun, recall the problems Jeff Goldblum had with it in The Fly.

The point is, VPLEX is an appliance and therefore it is another thing to manage and pay for. In fact it is at least 4 more things to manage (two here and two there) and can be as much as 16 additional things to manage. Just in case that point was not clear, VPLEX is new stuff to manage in between your servers and your storage.  Like the Wizard of Oz, EMC would like customers to "Pay no attention to those things in front of your storage, - they are virtual and therefore invisible." In fact, it's so invisible that EMC claims you can use it with any vendor's storage, which means there is minimal integration with storage functions. EMC is calling VPLEX Storage Federation, and yet it doesn't integrate with storage on any meaningful level. That's not federation, it's virtualization.

Suddenly, the virtualization of all your storage through an appliance is a trivial matter. But of course it adds considerable complexity to an environment that is already complex enough. To put a finer point on it, EMC customers are familiar with the down and dirty disappointing reality of working with Virtual Provisioning, SRDF and change management. They should expect to pay for additional professional services to figure out how to do these things with VPLEX inserted.  

And then there is the matter of managing mixed workloads in virtual environments, something that is a problem for a lot of customers.  EMC's technique of isolating workloads on certain disk and cache resources doesn't work for highly leveraged VMFS's where there are many VMs running concurrently. When the workloads are all mixed together, there is no way to isolate them. VPLEX appears to be an answer to that problem by allowing customers to move VMs elsewhere - presumably where there are more resources available. In reality, those VMs are going to be moved to another environment with the same storage limitations as where they came from.  Do you really think putting a distributed write cache between servers and storage is going to solve workload balancing problems?  If so, the fix will only be temporary before the same problems are repeated. And that is the problem with VPLEX - it doesn't solve anything by itself, it only allows a problem to be moved elsewhere - presumably someplace where there is more headroom that is not being as heavily utilized.

Storage systems that were designed for utility computing and assuming there would be mixed workloads to deal with, such as 3PARs InServ storage systems, are much more effective at maintaining performance levels for virtual machines and allow customers to operate at much higher system and storage consolidation ratios.

EMC wants you to imagine all the things you might be able to do with VM teleportation. I'd say, imagine how much fun its going to be doing the capacity planning for a VPLEX installation. How much storage do you need in location A and location B to support two way VPLEX functionality for a collection of 80 VMs? As it is with most things, it depends on a number of variables including whether or not VPLEX is being used to facilitate disaster recovery. It would not surprise me if VPLEX ends up effectively capping storage utilization at 50% for customers that employ it. If the maximum utilization is 50%, what will your actual utilization levels be?  Something far less than that. 

Reducing consolidation ratios

In a nutshell, all systems that are fronted by VPLEX will have the same write cache data so that a read from those systems will be serviced from the write cache, if the data has been recently written. However, if a read request is made from a VM for data that is not in VPLEX's write cache, the data will be read from either a local or remote storage array. That's why there are distance limitations to VPLEX today - the time it takes to cross-ship data from a remote location can't be too obnoxious. 

It's also why EMC's Virtual Geek spent time talking about using applications that have a single writer - such as v-Motion. A VM with an application that attempts to write to the same block from two different locations is apparently problematic. My guess is that applications with long sequential reads might not be so great either.

That implies a couple things: You should expect to find out there are limitations as to what applications can be running with VPLEX. Maybe identifying VPLEX friendly applications will be easy, but perhaps this task will be harder than assumed. If so, the best answer may be to engage EMC's pricey professional services organization. The details, the details.

The other implication is that VPLEX will limit the number of VMs that can be running on a single vSphere server. After all, you wouldn't want to have any congestion problems forwarding write cache data through the VPLEX network. The higher your server consolidation ratio is, the more likely it is that you could run into bottlenecks of varying types. Expect to see best practices for VPLEX that scale back your server consolidation ratios.  

Just because you can do something, doesn't mean it's a smart thing to do.

Zeroing in on a definition for federated storage

(Federation Square, Melbourne)

There have been some excellent discussions recently in the storage blogosphere and on Twitter about the concept of Storage Federation among a number of storage people; known by their Twitter IDs as @stuiesav, @storageanarchy, @rootwyrm, @davegraham, @bwhyte, @ianhf, @esignoretti, me (@3parfarley) and others - as the interest continues to increase. 

There are two aspects of the discussion that I think are fascinating: first is the role of social media as the means to include customers, vendors and others in an open discussion that typically is conducted privately by a vendor preparing to release a new product or feature, second is the challenge of defining a storage capability with sufficient focus and vendor independence so that is is meaningful. There has been some amount of skepticism about this effort, suggesting that we are predetermined to end up with ambiguous terms that can be interpreted (spun) by anybody (any vendor) to mean anything (our product does it). I'm hopeful the results can be better than that, but subjectivity is the only constant of social media and it's not likely that everyone will like whatever happens.  

In general, the language of digital storage has many overlapping meanings, which causes a fair amount of confusion in our industry. I've been dealing with this problem for many years, going back to when I wrote Building Storage Networks and had the challenge of trying to invent generic terms for functions that had been coined by vendors and tied to specific products. My interest in defining Storage Federation goes beyond my role as an employee for 3PAR.

The notion of federated storage has been around for several years, but recently it came to light when Pat Gelsinger of EMC referred to it during a press briefing on March 11. EMC blogger Chuck Hollis wrote about it afterward and there was some chatter, culminating in a blog post by @Stuiesav on April 2, which proposed that the discussion about Storage Federation was just marketing hype attempting to rebrand storage virtualization.  EMC's Barry Burke (@Storageanarchy) and I (@3parfarley) both agreed that this was not the case this time around and then in the last couple days the discussion fired up again. The problem with Twitter is the limitations of 140 characters per tweet.  It's surprising what can be done with so few characters, but it does have limitations.

Skeptics Allowed

This weekend, @ianhf  posted in his Grumpy Storage blog echoed @Stuiesav's skepticism and expressed his perspective (as a customer) as to the things he would like to see when new technologies - in this case Storage Federation - are introduced.  Here are some of  the items from his list (Some of them don't fit our definition exercise because they assume there is a new product being introduced, which is not the case here).

• The specific customer requirements & problems this addresses & justify how
• The use cases this feature / function applies to, and those that it doesn't
• Why & how this feature is different to that own vendor's previous method for solving this problem
• Provide clarity over the non-functional impacts of the feature before, during & after it's use - ie impact on resilience, impact on performance, concurrency of usage etc (including provide up-front details of constraints)
• Naturally you'll also expect me to require TCO & ROI of the feature, and any changes to the models as a result of this feature

   

So lets get started (already!):

The definition of Storage Federation that was kicked around on Twitter is something like:  "the transparent, dynamic and non-disruptive distribution of storage resources across self-governing, discrete, peer storage systems."  (And yes, I did elaborate a bit on this while I was writing based on bits and pieces from comments I read and further thought of my own.)

The idea is to have multiple storage systems cooperating as a team (as opposed to under the direction of an external entity) to place data in the aggregated storage resources (LUNs or volumes) of all participating members. An example of Storage Federation is how Dell/EqualLogic arrays distribute their volumes over multiple systems. When a new EqualLogic array is added to an iSCSI SAN, the administrator is asked if the array should be placed in the same group as other arrays in the SAN. If this is done, the arrays start splitting their volumes (and workloads) across both arrays.

Five examples of storage federation capabilities are:

Storage expansion: You want to increase the storage capacity of an existing storage system that cannot accommodate the total amount of capacity desired. Storage Federation allows you to add additional storage capacity by adding a whole new system.  

Storage migration:  You want to migrate from an aging storage system to a new one.  Storage Federation allows the joining of the two systems and the evacuation from storage resources on the first onto the second and then the first system is removed.

Safe system upgrades:  System upgrades can be problematic for a number of reasons. Storage Federation allows a system to be removed from the federation and be re-inserted again after the successful completion of the upgrade.

Load balancing:  Similar to storage expansion, but on the performance axis, you might want to add additional storage systems to a Storage Federation in order to spread the workload across multiple systems.

Storage tiering:  In a similar light, storage systems in a Storage Federation could have different capacity/performance ratios that you could use for tiering data.  This is similar to the idea of dynamically re-striping data across the disk drives within a single storage system, such as with 3PAR's Dynamic Optimization software, but extends the concept to cross storage system boundaries.

These are all examples of Storage Federation that Dell/EqualLogic storage systems are capable of today. It's not intended to be an endorsement of their distributed volume manager, I am simply using it as an example of Storage Federation. Saying that, if you look for "Storage Federation" on the Dell/EqualLogic web site, you probably won't find it today because they don't describe it that way, but that doesn't mean that it is not in the product.

That's not it

People also want help understanding what the definition of Storage Federation should not include. Here are my thoughts:

In-band storage virtualization systems like IBM's SVC can provide all the capabilities listed above for Storage Federation. However, in-band virtualization products govern the behavior of the other storage systems networked to it and Storage Federation involves self-governing, peer systems. Another way of saying it is that Storage Federation does not have functionality in the network between host systems and storage.

File, object and data distribution technologies like EMC's Atmos certainly provide a type of federation insofar as multiple computer systems can access the same data objects from multiple locations that may be separated geographically. However, this capability primarily migrates files (data objects) - and is functionally orthogonal to Storage Federation, which works on volumes. A term like Data Federation is probably more precise than Storage Federation for this sort of capability. 

Clustered storage systems, like 3PAR's InServ storage systems or clustered NAS systems are not examples of Storage Federation because they function as single, scalable storage systems, not as discrete storage systems that are networked together.  Clustered storage systems can work together in Storage Federations with other clustered or non-clustered storage systems. In that case you could have a Storage Federation of clustered storage systems.

How is this different?

I thought I'd make a little effort to respond to @ianhf's requests.

The main problem Storage Federation addresses the limitations presented by a single storage system, such as capacity, performance and maintenance availability. 

Currently, when a new storage system is added to an existing environment, there are server administration tasks that need to be done in order to migrate volumes from an existing storage system to the new one. In addition, there is usually some amount of downtime and/or performance degradation associated with the migration and configuration of new data paths.  By contrast, with Storage Federation, a new storage system could be added without having to reconfigure host data paths and the migration would be processed transparently and dynamically, without any downtime or loss of path redundancy while the migration is in process.

Increasing performance for a particular volume in a Storage Federation is somewhat less obvious because the other federated storage systems might not have more performance-resources available to boost performance. For instance, if a new storage system inserted in a Storage Federation does not have more disk drives (spindles) or more flash SSD capacity, there is no guarantee that any volumes moved to it would provide faster performance. There is some likelihood that performance could be improved by moving a volume to a storage system running at lower utilization levels, but maintaining lower utilization levels is not realistic or very cost-effective. All this said, it is possible for a Storage Federation to use the aggregate resources of multiple storage systems to increase performance of a single volume - such as by spanning all the disk drives in the federation and by using the cache of all participating storage systems. An example of products that provides this capability today are the Dell/EqualLogic storage arrays. 

The availability of storage systems needing maintenance could be greatly improved by Storage Federation if an existing storage system can be removed from the federation after having it's volumes evacuated (think v-Motion but for storage systems) to other storage systems in the federation. After being removed from the federation, the storage system could be downed to have any sort of maintenance operation done to it without risking the availability of any of the volumes that had previously been on the system undergoing maintenance.Obviously, all of this would take time planning and ensuring the Storage Federation would not be overloaded with data and workloads, but there are probably many customers that would prefer to do maintenance to storage systems when they are offline. 

One of the possible exposures of Storage Federation is the increased exposure to system failures. For instance, a Storage Federation that distributes a single volume over three separate storage systems is 3 times more likely to have a failure than a Storage federation that does not allow volumes to span across storage system boundaries. FWIW, this is the main weakness of the Dell/EqualLogic implementation of Storage Federation. 

That's all for now

It's late and there is plenty of material here for the grinder. Please feel free to comment in any way; agree, disagree, correct mistakes and ask questions.  Thanks for reading.

Knobs, Prison Knobs and STACK WARS

Technology integration makes computing products much easier to use and significantly drives down the cost and effort of owning it. For instance, technologies such as WiFi that were recently beyond the grasp of most people are now inexpensively integrated into PCs and usable by almost anyone.

The trick with integration is understanding what variables should be exposed - or as my friend Rick Vanover likes to say - how many knobs there are to turn.  End user and infrastructure provider requirements differ considerably when it comes to knobs. For instance, Apple computers are great end user machines because they lack knobs, but are not always loved by technology professionals for the same reason. Data center operators need products with knobs in order to accommodate all the cross-purposed requirements that stretch beyond a one-size-fits-all design.

So knobs are generally good - but like so many things - their usefulness depends on how effective they are and their station in FARLEY'S HIERARCHY OF KNOBS, which includes the following levels:

Suicide Knobs:  knobs that delete data and make things blow up.  A good example of a Suicide Knob is something that formats storage.

Prison Knobs:  knobs that make changes that are very difficult or impossible to reverse.  Many storage provisioning knobs fall into this category. Once you provision and reserve storage with most storage arrays today you are stuck with that decision until the array's EOL.

Faux Knobs:  knobs that never seem to do anything, no matter how far you turn them.  For features past and future, but not now.

Random Knobs:  knobs that produce unanticipated results that can go unnoticed for years.  These are the knobs that fuel the technical publishing industry.

Slippery Slope Knobs: knobs that start you down a path to ruin through a chain of system dependencies. These are the knobs you spend a lot of money to learn about in vendor classes. 

Dumb Ass Knobs:  knobs that do things, but not anything useful. Granted there is a LOT of subjectivity in making a call on a dumb ass knob - but we all agree they exist.

Honest Knobs:  knobs that actually do something you need them to without having to plan for weeks on how to use them.  Most knobs should fall into this category, but alas!

Magic Knobs: knobs that do things so useful it makes you wonder how anybody thought of a knob like that. Most of these knobs are actually Honest Knobs, but we are so accustomed to seeing Suicide, Prison, Faux, Random, Slippery Slope and Dumb Ass knobs that we are blown away by a truly great Honest Knob.

Which brings me indirectly to the discussion of STACK WARS,  which, have been centered around the announcement of VCE (VMware\Cisco\EMC) and their vBlock concept last November. 

 

I'd like to say I was surprised yesterday when graphically-challenged Hitachi announced their intention to sell their own Unified Cloud Graphic, (complete with Hitachi compute servers!).  But it wasn't a big shock considering their marketing strategy of "just copy it".  

 

I really don't know how they expect their graphic to compete with vBlock's graphic, with all the color, multiple font sizes and graphics within graphics.

What's missing from the both stack graphics are the knobs that administrators use to get real work done.  Yes, knobs tend to be part of the underlying details, but to anybody that actually uses a product, they are very important details. The detail that C-level executives need to understand is that the stack does not have nearly the automation that is being promised today and that administrators will be doing a lot of work, turning the knobs that the stack provides. Again, it's not the number of knobs that matters, as much as it is the quality of those knobs.

Some people have speculated that the vBlock was a knob-less invention that originated in the board rooms of the VCE companies. Some have even suggested that it was the fallout after a failed acquisition bid by Cisco to acquire EMC.  I don't know if THAT's true, but there is some evidence that the engineering groups in the companies involved have been scrambling to put meat on the bone.

Maybe someday stacks will be the next big thing, but I don't see it playing out that way unless an awful lot changes in the underlying products that make up the stack. Here's my take on STACK WARS:

STACK WARS give everybody something to write about- me included, right now!

Bloggers that write about stacks have a chance of getting jobs with stack vendors. If you are out of a job, start a stack blog today and twitter your back-stack off!

Stacks are all about packaging. Stacks will be assembled and shipped together (presumably), which could make things easier if your goal is to streamline receiving.

Stack products, are actually more services than products. However, if you ever want to make configuration changes in your stack it might not be economically feasible. (Think gigantic FRUs) For example, there is not a lot of flexibility in vBlock's configurations.

Due to the limited configuration options, stack resources are not likely to be used very efficiently and the economic return on the investment will lag. However, EMC customers are already accustomed to low storage utilization levels - so poor utilization might not be THAT big a deal. Definitely a weird way to win a point, but I 'll concede it grudgingly.

The business advantage of integration should be much lower costs. However, the VCE companies all need to maintain their margins if they want to satisfy investors. It's not clear how they will be able to leverage the integration effort to reduce the cost of vBlock, but then again if STACK WARS turn into PRICING WARS for STACKS, things could get very interesting. IBM must be STACKING up something - after all Hitachi already beat them to the punch.

The C level view of stacks are that they smooth out purchasing and operations expenses by providing a smaller number of Purchasing Knobs (that would be a Faux Knob).  John Nash posted in his blog last week,"The Case for the vBlock":

What is interesting is that, usually, the higher up in an organization you are communicating the better the Vblock conversation goes. Remove the detailed technical questions and the value of the Vblock idea really shines. You get a known “product” from trusted sources. You get known costs today as well as known costs for future expansion. It greatly removes the risk from the organization with unknown infrastructure expenses.

There you have it, vBlocks will be sold from the top down by Cisco and EMC - companies that are good at selling from the top down, which will make it somewhat easier for the VCE companies to justify their price tag. But that won't make the price any easier to swallow. 

As Nash wrote, "remove the detailed technical questions and the value of the Vblock idea really shines."  That's like saying chapulines (fried grasshoppers) might appeal if Anthony Bourdain is talking about them on TV, but your own personal experience chewing and swallowing them might be different. I'm not talking about price here, I'm referring to the experience of running the vBlock. There is going to be a lot more involved than the knob-less graphics portray.

The weakest link in the vBlock chain today is EMC's contribution.  There are far too many Prison (provisioning) and Slippery Slope Knobs in EMC storage.  They aren't the only vendor with this problem, but they are the E in VCE.  Provisioning storage with a v-Max is the about the same as it was with a DMX - despite what EMC employees would have you believe. 

Prison Knob provisioning creates a lot of problems for customers as storage ages and as demands shift.  Once storage has been reserved for usage in an EMC system, it is pretty much bound to that purpose. 

My advice is to buy the products with the most Magic Knobs and avoid those with the most Prison provisioning Knobs. If you have ever felt trapped by a storage configuration that you couldn't live with or afford, you know what I'm talking about.  Magic Knobs are those that reduce the effort to manage and change storage, increase the efficiency of storage and provide the most versatility for all applications, workloads and multi-tenancy.

Calculating the output of wide striping arrays

There has been some discussion recently about wide striping in the storage and virtualization communities and it struck me that a number of people were unsure what it was and how to think about it, so I thought I'd explain the basics here.

What is Wide Striping?

Wide striping is a method of spreading data over a large number of disk drives in a storage array in order to achieve a desired performance goal. The term wide striping has been used sometimes to refer to sets of disks with 16 to 32 members, or more.  It seems strange that somebody would classify a 32 member disk set as having wide striping, when you compare that to a disk set with 320 members, but that's how definitions in the storage industry tend to be broadly interpreted - especially by vendors :). 

The amount of work a disk drive does is measured in IOPS

One of the ways to measure storage performance is using a metric called IOPS - or IOs per second. An I/O is either an operation where the disk drive reads data from media or writes it to media.  Reads process faster than writes, so it is important to understand this mix in your applications. The amount of work a disk drive can do is proportional to the rotational speed of the drive. 

There are two basic classes of disk drives used in enterprise storage arrays today; lower-cost, lower performance SATA disk drives and higher-cost, higher performance disk drives with Fibre Channel (FC) or  SAS interfaces.

A 7,200 RPM SATA drive can sustain ~ 80 IOPS

A 15,000 RPM FC/SAS drive can sustain ~ 200 IOPS

In other words, a 15,000 RPM SAS/FC drive can do approximately 2.5 times more IOPS than a 7,200 RPM SATA drive.  You need approximately 5 SATA drives to sustain the same workload as a pair of FC/SAS drives running at maximum sustained levels.  

Storage workloads are determined by the applications being used and there is a great deal of variety in the performance requirements they have. Some applications need very little storage performance and some applications need a high level of storage performance. 

Workload isolation

Storage arrays that do not have wide striping tend to have high-priority workloads isolated on certain designated drives in order to meet the performance requirements of those applications. For instance, an array with 100 FC/SAS disk drives could have ten of those drives used exclusively by one high priority application to make sure that it has access to all of the 2,000 IOPS those drives can generate. 

Isolating workloads this way creates storage capacity utilization problems because the capacity requirements of most high performance applications tend to need fewer disk drives than IOPS requirements determine.  In other words, an application that has reserved the 2,000 IOPS of ten 450GB FC/SAS drives might only need 900GBs of capacity - with a utilization of 20%.

The IOPs requirements of applications change over time, which means an application that has it's workload isolated on a set number of drives can exceed their aggregate IOPS capabilities, creating a performance bottleneck for the application. The structure that was established to ensure sufficient IOPS becomes a performance problem. The process of adjusting and rebalancing the disk drives that are used to support isolated workloads can become extremely complicated and time consuming depending on many variables and creates opportunities for administrator errors.

Workload randomization

Alternatively, storage arrays with wide striping tend to have multiple applications sharing the IOPS generating capabilities of a large number of drives - up to all the drives in the system. Using this approach, the data from each applications is spread over all the drives being wide-striped to give all applications access to the the aggregate IOPS potential of the array. For example, an array with 100 FC/SAS disk drives that wide stripes data over all the drives would have 20,000 IOPS available for any and all applications at all times.

In contrast to workload isolation, discussed above, this approach is known as workload randomization because it mixes all the data in the array across all of the disk drives. In order to ensure data is written across all the disk drives, wide striping storage arrays tend to write data in very small increments on each drive. Applications with heavy IOPS workloads are utilizing all the drives in the system in relatively similar amounts.

By nature, wide striping eliminates array bottlenecks and achieves better storage utilization, but utilization levels depend on storage provisioning methods that are designed to take advantage of wide striping such as Thin Provisioning, which is another topic for another blog post. 

The lack of bottlenecks in a wide striping storage array also means that mixed workload performance is optimized because all applications have access to the full IOPS potential of the array. Virtualization environments benefit in two ways: 1) by being able to mix applications and VMs on physical servers without worrying about the creation of storage bottlenecks and 2) achieving higher VM densities.

Enterprise arrays with native, wide striping

There are two enterprise storage arrays that use native wide striping across all the disk drives in the system, without the need for administrators to organize RAID groups and create disk pools.

3PAR InServ 

IBM XIV

There are many important differences between these two products, including the flexibility of RAID implementations and disk drive options, but the method of wide striping is similar.

Wide striping IOPS to servers

The number of IOPS delivered by an array is slightly less than the aggregate number of drives in the array, depending on the mix of reads and writes and the type of RAID that is used by a particular volume.  Here are a few examples of IOPS calculations for wide striping systems using different drive scenarios and 8k blocks.  The arrays modeled here do not include mixing RAID types for different volumes, except the last model that mixes RAID 5 on FC drives with RAID 6 on SATA drives.

Array 1: 80 450GB FC disk drives:  80% reads, 20% writes

Total IOPS of all drives in the array:   16,000

IOPS delivered to servers w/RAID 5:  10,000

IOPS delivered to servers w/RAID 10: 13,333

Array 2:  200 1GB SATA disk drives:  80% reads, 20% writes

Total IOPS of all drives in the array:   16,000

IOPS delivered to servers w/RAID 5:  10,000

IOPS delivered to servers w/RAID 10: 13,333

IOPS delivered to servers w/RAID 6:     7,505

The next array has both FC and SATA drives and changes the read/write ratio. In this case there are two different wide stripe sets, one that spans FC drives and the other that spans the SATA drives.

Array 3: 160 450GB FC + 480 SATA disk drives: 70% reads, 30% writes

Total IOPS of all drives in the array:   70,400

IOPS delivered to servers w/RAID 5:  37,052 

IOPS delivered to servers w/RAID 10: 54,153

IOPS delivered to servers w/RAID6 (SATA) & RAID5 (FC): 31075

Storage Caching 102 - mixed workloads

Chuck Hollis had an excellent post last week, discussing caching. 

About 10 years ago a small team that I was a part of looked at starting a company that would do something similar to what IBM's SVC does.  The idea was to create a SAN front end controller with a lot of cache memory that would virtualize  "downstream" storage and provide performance boosts through various techniques such as caching, striping, and multi-way mirroring.  We gave up on the idea when it became apparent to us that the project was quite a bit larger than we initially thought and it was unclear when we would ever have sufficient resources to get a competitive product to market. I think we could have sold the idea to venture capital investors who were throwing money at storage startups, but we couldn't sell it to ourselves. For those of you that wonder why I tend to think SVC is an important product, that's why - I know some of the things IBM did to make it work and I admire their ability to bring it to market.

Anyway, one of the hurdles we couldn't get past was how to deal with mixed workloads originating from multiple SAN attached servers simultaneously streaming data from data warehouses and IOPS from transaction databases and all sorts of other bursty, unpredictable applications competing for memory and disk resources. As a front-end appliance, you can control your own cache resources, but you can't do much about the back end disks because downstream arrays mask them from the appliance. In fact, there was no way to predict the performance of the downstream arrays for any given workload. We consulted with experts from the industry and research universities and they were all discouraging about the ability to significantly improve the performance of mixed workloads in a shared SAN appliance.

There were two issues: downstream arrays already had their own caches and the dynamics of sharing cache resources in an appliance. If these fundamental problems could be solved, the rest of the work was the not so simple matter of making it work as a cluster with mirrored cache for HA.

Uncoordinated multi-level caches have the problem of redundant data.  For example, pre-fetching data in the appliance's cache will likely end up loading the same data in the downstream array's cache.  With duplicated cache data, a cache hit on the appliance won't be that much faster than a cache hit in the downstream array - and a cache miss in both will be much slower. It's difficult to prove the value of THAT.  Its certainly possible to tune both caches differently, but this turns out to be easier said than done and tends to flatten the value of an "easy to use" appliance.

So, the way to overcome this is to make a HUGE cache in an appliance.  Increasing a cache's size can do a lot for performance and Chuck said,

The exception, of course, is if you've got the bucks to create a ginormous read cache, and pull almost all the significant data into memory.  Don't snicker -- there are a few use cases where this sort of approach makes sense.

In other words, create a RAM disk in cache. This technique is normally reserved for a single, high profile application and as Chuck wrote, there are use cases where this makes sense.  But it doesn't address the requirements of mixed workloads where there are a large number of applications that do not merit dedicated memory, but still need good performance.  It might be possible to micro-manage cache for some number of applications by dedicating cache for each of them, but that requires a great deal of work that is likely going to be a temporary solution lasting a couple of months at most. It's probably a great way to drive storage admins crazy.

A more palatable approach is to use a global cache that shares cache resources among multiple servers and their applications. In some cases, it's possible to predict workload demands (end of month processing for example), but in many cases, the instantaneous performance requirements cannot be predicted because it is driven by spontaneous events.  As many people are too familiar with, spikes in Internet activity and the corresponding bottlenecks in back end storage clearly elaborate the challenges of mixed workloads. Global caches that can accommodate large Internet traffic spikes are expensive and do not provide noticeable performance advantages most of the time - they are overkill.

The proliferation of virtualized servers has significantly increased the breadth of the mixture that a storage array has to deal with. In general, there is more overall I/O activity that is, for the most part, less predictable, and therefore more difficult to deal with in cache.  

The question is, is storage tiering with SSDs any better?  Possibly. If it is going to more effective than caching, it needs to be able to provide more control points and intelligence than cache typically does. For instance, the ability to prioritize and schedule applications for movement into SSD tiers could be an important difference. 3PAR's QoS Gradient concept in Adaptive Optimization is an example of a simple prioritization scheme. The internal counterpart to QoS Gradients is internal monitoring of I/O levels which help determine which applications are promoted. 

That's not to say caching can't have some of the same controls, but traditionally caching has been more reactive than proactive.  To be clear, tiering is also reactive, but within the context of intelligent preparation and business-driven policies. 

Still, considering the cost of SSDs, you have to ask the question is tiering overkill too? It might be. If the array can keep up most of the time, how much SSD capacity should be purchased?  These are the sorts of things that will be determined over the next couple years.

The best technology to date for dealing with mixed workloads continues to be wide striping. If you throw out latency-sensitive applications from the mix - those are the same apps Chuck Hollis referred to when he talked about the applicability of "ginormous read caches" - then the array just has to provide adequate throughput at reasonable service times (latencies).

Wide striping does this by spreading the workload mix over as many disk drives as possible.  Not the number of drives that fit in a shelf or can be added to a RAID group, but all the drives in an array, at best or all the drives of a particular class, day SATA or FC. Wide striping very thin layers of data across hundreds of drives means that hundreds of servers can be accessing data simultaneously, with a minimal amount of contention.  The result is that all the drives are kept busy at the same rate and that none of them bear an unfair burden. The overall sustainable throughput is very high, scales by adding more drives and in general fits the profile of mixed workloads better than any affordable caching scheme does.

We've become accustomed to thinking that more memory is the answer to all storage performance problems, but that doesn't exhaust the potential of massed disk drives.     

Tired of being virtually tongue tied? Give Mosh a chance!

I was talking with my InfoSmack co-conspirator, Greg Knieriemen (iknerd.com)  last week about virtual junk and how the word "virtualization" was so incredibly unwieldy. It was killing my lazy mouth to have to do so much work for such a well-understood term/topic.

(BONUS QUESTION: is "virtualization" a noun, verb, or WHAT? )

Fear no more, bad spellers and slothful speakers!  The new word for virtualization is MOSHWARE. MoshWare is software that enables all sorts of systems and applications to be blended, pureed, mashed and moshed together in a single computer, or MoshBox. It also allows MoshMachines (or MMs) to be relocated to other MoshBoxes using functions such as  m-Motion or TransMoshinization.

 Of course the word "MoshWare" could apply to many other things, including social networking sites, such as Twitter. There needs to be a modifying word or prefix for the social context in order to differentiate machine MoshWare (its true form) from social MoshWare.  Devo MoshWare works for me. 

Blade servers also provide Moshable benefits, similar to those of MoshWare, but in hardware. Whereas "Blade server" has 3 syllables, the much smaller word "BladeMosh" simply rolls off the tongue like a Scotsman's beverage. 

One of the great things about MoshWare is getting rid of unnecessary networking adapters by consolidating network traffic on fewer links over the MoshNet!  Not only that, but storage MoshNets can be implemented as Switchless SAMs (Storage Area MoshNets). To accomplish this, a MoshBox's HBAs are cabled directly to a storage array's ports without an intervening switch. When you consider that this eliminates two switches at at time (for HA) - as well as the need for zoning, the economic benefits can be significant. We have a number of customers at 3PAR who do just that and have saved a lot of money in the process. This means installing more SAN adapters in storage arrays, but if the arrays can accommodate them it might be worth considering. Is it for everybody - certainly not, but is is another option for reducing the cost of storage.

So what about Moshed storage - or STORMOSH!   SAN and NAS are different forms of StorMosh, but DAS is most certainly Moshless. The amount of Moshing a StorMosh does depends on it's design. Some StorMoshi require a large amount of Manual Moshing (ManMosh is an oxymoron to be sure), whereas other StorMoshi, such as those from 3PAR and Dell EqualLogic, Mosh at a very high level. 

The graphic below shows an implementation of a MoshStack ,with a MMs running inside MoshBoxes, connected to a StorMosh via a Switchless SAM.