StorageRap

EMC, the Misanthropic Media Company?

It's getting very hard to keep up with all the crazy social media stunts coming out of Hopkington, but they seem to have done it to themselves again.  First was the questionable spamming for viewers so they could claim they had a viral video, then today they just "leaked" a 3PAR sales "kill sheet" - and also apparently established a "secret" site with the URL Notapp.com, where they compared their own guarantee program to Netapp's.  According to Simon Sharwood at Search Storage Australia, the site was removed and accessing the URL directed browsers to EMC's site.

Perhaps it is all part of a new marketing strategy by newcomer Jeremy Burton, who joined EMC as Chief Marketing Officer back in March.  As best I can tell, Burton's new marketing strategy for the company is that people will believe anything.  Maybe he doesn't think there are enough new products coming out of EMC - or that the delays in getting their ballyhooed FAST out the door are too embarrassing - but instead of trying to promote EMC on its own merits, it looks like he is doing his utmost to mud wrestle.  Is that what EMC is paying him the big bucks for? 

EMC suddenly is taking a bigger interest in 3PAR. That's good. Search Storage Australia just published parts of a competitive document that EMC was circulating to it's partners about 3PAR.  It certainly wasn't a surprise because we'd seen it previously, but I was sorry to see it published because it made EMC look ridiculous, which was working pretty well for us. But now that it's been outed, here is what we have to say about it (in the guise of Ineption's lead character, the CRO)

The messaging is not built in, but our zero detection technology for optimizing capacity is. The host SW commands to do this are short and do not require "careful coordination".  Veritas, Oracle, Windows Server and Linux software all work with minimal operator effort. For instance, this document from Oracle, describes the whole process, with the sole operator command being this:  #bash ASRU LDATA.

Can EMC provide online reclamation of zeroed space without risking capacity overruns and with tolerable performance? 3PAR can. Does EMC have these capabilities in both mid-range and enterprise storage arrays? 3PAR does.

3PAR has both Flash and 1 TB SATA drives.  We also have Adaptive Optimization software that uses Flash SSDs for storage tiering. EMC still doesn't have it after they made such a big deal about it last year. They like to tell customers that their size gives them development advantages, but their track record doesn't support their claim.

3PAR arrays allow users to create many tiers, but without the need for disk pools. Tiers are constructed from the combination of drive type plus RAID level. For instance, you can have separate tiers for SATA, FC and Flash SSD drives with the RAID level you select. Our Dynamic Optimization software allows admins to move data from one tier to another. You can "dial in"  the performance and protection you want.

All systems have a peak output , ours just happens to have a lot more throughput than theirs - and at higher disk utilizations.  We have published benchmarks that show how our systems perform. They don't. Adding disk drives to a system and utilizing those drives is far easier with a 3PAR system than either VMAX or Clariion where you have to wrestle with putting drives in the pools you want to use them for.

There are no disk pools in 3PAR storage. Pools trap resources so you can't use them. Work isolation in pools leads to hot spots and storage admin nightmares.  Wide striping does not mean you can't have tiers. That is an idiotic statement.

VMAX can configure large pools - and all the drives in them have to be at the same RAID level meaning you can't create multiple tiers within those pools. If you want multiple tiers, you need multiple pools and all the headaches that involves. Change management in an environment with multiple pools is complicated. You also need to consider the pools needed for snapshots and remote replication. Are those easy to provision and change on EMC storage. Most would say "no".

3PAR uses all disk spindles all the time for delivering IOPS and pro-active sparing is done using reserved space on those drives.  Rebuilds do process quickly. Would EMC have you believe they never have to perform drive rebuilds? Really?

The RAID6 thing really makes me sad. They look so stupid when they say it. We're all sorry to say goodbye to that piece of FUD.  

Our front end archiecture was designed for large-scale parallel connectivity to match the massive bandwidth capabilities of our wide striped back end. Our benchmarks and the cost per IOPs in those benchmarks speak for themselves. Our customers also tend to run 3PAR systems at much higher disk utilizations than they run other vendor's arrays. 

We support a huge number of ports on our systems w/full active/active data access across all controllers. All controller nodes can be used to access all data volumes.  We have a number of customers that run fairly sizable SANS without switches because they have enough ports on their arrays so they don't need to consolidate access through switches.

5- 9s? We're there. Our systems get pounded on every day in some of the largest private and public data centers in the world. They are designed with complete redundancy in all components and have advanced capabilities such as Persistent Cache to maintain high levels of performance even after the loss of a controller.

The delays in bringing their FAST tiering software - a product they were hyping in April of 2009 -  to market have shown that size doesn't matter much when it comes to delivering technology on time.  I'm not saying 3PAR always delivers on time, but EMC is far from immune to these problems. In fact, the need for them to coordinate across multiple product lines creates certain disadvantages for them.

As to their comments on our support; they are pure FUD and grasping for straws.  We would not be able to maintain the customers we have if it were not for our efforts at supporting them.

  * * * * * *

The following content was added on July 30th by Rusty Walther, 3PAR's Vice President of Customer Services & Support. 

Stating that 3PAR “outsources support” is just plain silly, especially coming from a company that keeps most of the worlds’ largest offshore outsourcing companies in business.  Like EMC, 3PAR uses Third Party Maintenance suppliers (TPM’s) for break-fix field activities.  In some geographies, EMC and 3PAR even use the “same” TPM.  But EMC also outsources most of their volume call center and Level-1 Technical Support to offshore suppliers.  Not so at 3PAR.  Everyone that touches a 3PAR support case is a 3PAR-badged employee.  I challenge EMC to identify a single outsourcing company that handles 3PAR technical support.  EMC’s outsourced technical support sub-contractors could be listed alphabetically, by geography, or by technology category … but you’d need a couple of sheets of paper to do it.

From viral spam to virulent sham

The twitterverse is busy again today with discussions surrounding EMC's us of spambots to generate views of videos they are trying to make viral.  If you are interested in seeing what is being said, check out these people's tweets and you'll be off on a trip down a dark hole. 

@johnful,    @dvellante,    @sfoskett,    @valb00,    @furrier

Here are a couple cartoons I made about it last week from my new cartoon, Ineption:

Netapp's Val Bercovici suggest this viral spamming as the end of innocence in social media, but innocence exited the social media stage long ago. 

I'm much more concerned about how large companies like EMC can use social media to suggest product and customer relationships that stretch the truth well beyond the impressions that a reader might take away from reading suggestive blog posts from respected corporate voices. As "unofficial company statements" that are more influential than press releases, social media pieces can distort things in a way that more-accountable corporate marketing are not allowed to.

Last week, Chad Sakac and Chuck Hollis published blog posts that pointed to an EMC white paper about details of a VMAX implementation at Terremark, an excellent 3PAR customer.  Readers of these posts would probably think that VMAX was being used as the storage behind Terremark's multi-tenant,  Enterprise Cloud service offering.  That would be stretching things more than just a little bit.  I commented on both blogs and the responses to my comments were interesting.  I guess I feel a little kinder towards Chad as a result. 

It is possible that somewhere in the world, a VMAX is being used by Terremark.  One would expect Terremark to be looking at various storage platforms as a matter of course, it only makes sense for them. After all, VMware made a significant investment in Terremark last year and we all know who owns VMware. There are certain favors that EMC can ask that vendors such as 3PAR can't. But Terremark also has to operate Enterprise Cloud in their US major data centers every day and the storage they use for that is not in a test lab - it's production - and it is 3PAR storage. 

And its not for lack of trying on EMC's part.  Last November when VCE was announced, Terremark was discussed as a featured customer in both Chad's and Chuck's blogs.  That was OK,  I understand the excitement that surrounds a big announcement.  But nine months later, to suggest that this announcement had given birth to a major production environment for a service that it is not supporting sort of stuck in my craw.

Here is a video I made at VMworld with Jason Lochhead, CTO of Hosting Solutions at Terremark last year where he talks about vCloud Express and Enterprise Cloud.  Very cool offerings and definitely on the leading edge of VMware-based service offerings.

It's not a viral video, but it has a lot more to say about what people care about than the videos EMC has been chasing with spambots.

Robert Cockerill from Thames River Capital: A man with broad responsibilities and 3PAR storage

3PAR customers like that fact that 3PAR arrays are so easy and fast to manage. In this video, Robert Cockerill from Thames River Capital in London talks about all the various things he does, his Windows-based infrastructure, how 3PAR's thin provisioning helps him manage it all and how simple it was to protect it with 3PAR Remote Copy.

The absurd prognostications for storage in a stack

 

How is it that some people possess the gift of foresight and the ability to predict the future?  Some say they have dreams or visions, some extrapolate from experience and logic, while others make predictions hoping to fulfill an agenda. Then there is the element of public exposure. Is the predication public and do they use their real name or hide behind an alias? 

Nicholas Carr took was very public and very open when he wrote his breakthrough book "Does IT Matter?".  In it, he stated that there are no sustainable advantages to be gained by a company through the implementation of information technology.  He argued that any short term gain can be matched by competitors in a relatively short period of time with lower capital investments - effectively punishing companies for innovating. He recognizes the necessity of having IT in order to stay competitive, but finds it difficult to justify being an early adopter of technology.

Since Carr published his book, we've seen a lot of change in IT markets, including the rapid deployments of virtual systems technology and the expansion of hosted, utility computing and all things "cloud."  But the biggest changes have resulted from the global financial crisis, forcing companies to reduce non-essential costs significantly - especially IT costs.

Unfortunately, not every technology implementation intended to reduce costs has been successful. And that's one of the things that makes the information technology business so fascinating and perplexing - intelligent people with deep expertise in technology fail to predict the ways that things can go awry and what the cost of their shortsightedness will be. 

The rich history of failed IT projects is exactly why there is so much FUD spread by the competitors in our industry - FUD gets customers thinking about the consequences of their purchase decisions and all the possible problems that can result from an error in judgment. It also contributes to the interest in the machinations of our industry and the "war games" that are played out in traditional and social media. Whether we are predicting changes to the industry through mergers and acquisitions or the development of new business models, it all flows into the river of FUD at purchase time.

With the abundance of FUD, one naturally develops an aesthetic for the stuff to cull the weak from the strong. For example, a piece of weak FUD recently appeared online on Silicon Angle titled "Why Netapp Must Seek Acquisition", written by the poser "secretcto".  The author starts with the suggestion "let's take a look at the market cap of each of these players" and then neglects to make any comparisons. It goes downhill from there, reaching its lowest point when the article referred to Nicholas Carr as Daniel Carr and then failed to negotiate the transition of whether or not IT matters to cloud service providers. The tipping point for Carr's logic is that to service providers, IT absolutely does matter because operating data centers is their core business.  

By contrast, you barely notice good FUD, it has a smooth logical flow and subtly builds to a persuasive conclusion based on a key point that usually has it's origins in a subjective opinion or bias. A decent example of good FUD was Chris Mellor's recent piece about the Storage Array Killing Fields qualifies as good FUD.  Chris doesn't have an axe to grind, but he is a journalist and therefore has the responsibility of stirring the pot.  It's a well written piece that uses an analogy that compares the selection of equipment for data centers with the selection of components used in an automobile.

The problem is that automobile manufacturing is a poor analogy for running a data center. When a car rolls of the manufacturing line it is shipped to a dealer, sold to a customer who drives it away.  There is nothing about the experience of making, selling or buying a car that is even closely related to the constant ongoing data processing services that are provided by a utility or cloud service provider. 

A better analogy is running a restaurant. Restaurants succeed or fail based on the quality of their customer service and that's why chefs like Thomas Keller strive to maintain consistent, excellent quality every minute of every day they are open.

Should we expect the recipe for success in hosting and cloud services to be any different? This recent article in Information Age states that 71% of the 450 CIOs in a KPMG survey want to improve the price to quality ratio of their outsourcing contracts. The dynamics of the business relationship between CIOs and their utility/cloud service providers are going to be the same. Service providers with the best reputations for customer service are going to thrive. Those that don't measure up will fail.

Vendors of consolidated stack solutions of servers, storage and software are trying to convince customers that the "All-in-one" stack solution is the safest way to proceed during the transition period while cloud computing is emerging. They would have you believe that the biggest risk in operating a data center is in ordering the products and getting everything installed initially. Considering that customer metrics for utility/cloud service providers will be responding to the needs of their customers quickly and accurately, the lions share of the risk will come well after the initial installation during the life of the service engagement.

The weakness of the All-in-one approach is that it does nothing to address the dicier aspects of owning, operating and changing an IT infrastructure after it is up and running. In many cases the stack vendor's answer to change management will be the same as it is today - time-consuming and expensive professional services.  There are definitely utility/cloud service providers that will want this sort of service, but many would prefer to do it themselves at much less cost. That's what you do when your primary business is running a data center.

A talented chef can find a way to prepare a gourmet meal on an Electrochef All In One Kitchen, but they would never decide to run their business with them. They are going to select best of breed appliances and equipment that best fit their needs and enable them to prepare quality dishes in a quality fashion.

So the question for the utility/cloud data center operator then is - "what is best of breed equipment for my business?" 

The classic clash between Best-of-breed and All-in-one solution pits cost against complexity. Best-of-breed technology has traditionally been more customizable to fit a wider range of requirements and therefore has been more complicated and expensive to operate. In contrast, All-in-one technology has traditionally been cheaper, limited to a smaller set of functions and easier to operate.

Unfortunately, neither stereotype works very well for the utility/cloud service provider. They need fully functional products that are also easier and quicker to operate. Fast, accurate change management and operator efficiency are the key elements for utility/cloud infrastructure products. 3PAR's Best-of-breed storage products have these characteristics as well as being extremely space-efficient and high-performing. Customers appreciate the amount of time they do not spend managing their 3PAR storage while they are getting the job done. When a new order comes into a 3PAR kitchen, the system is ready to go right away - including tasks that take a long time to set up on other storage, such as Remote Copy.

And what about the All-in-one stacks in the market?  Surprisingly, unlike traditional All-in-one solutions, they are more expensive to install and operate.  Change management is complex, which leads to relatively poor operator efficiency and the engagement of professional services, which does not necessarily speed up the process. The traditional benefits that All-in-one solutions typically provide are not part of these stack solutions.

The predictions for stacks taking over the market are all wrong. Sure, there will be stack solutions sold and it will take time for all of this to sort itself out as it always does when an industry is going through major, fundamental changes. The most important changes that will occur in the years to come will be driven by the service demands placed on utility/cloud service providers. Customers of utility/cloud services want their money's worth and the best service providers will do what it takes to give it to them. Stacks add no value in that equation.

The transparent facts of open benchmarking

Nate at Techopsguys has put together a comparison of SPC-1 benchmarks with six different bar charts showing the various characteristics of the configurations, performance and cost.  

Here's an example of what's in is post.

 

Best practices for managing multiple storage pools - avoid them

Nigel Poulton tweeted today:  "What are peoples thoughts on best practices for multiple pools on the likes of USP V and VMAX. Trade-off between perf vs resiliency etc..."

Good question Nigel, one of the biggest problems customers have is being able to fully utilize all their resources. It's not just that the ROI for storage tends to be underwhelming, but more frustrating is the fact that their storage was provisioned in a way that makes resources inappropriate or unavailable for the pressing needs at hand.

Pools are used two ways - to reserve storage capacity for certain functions such as snapshots or to create QoS levels for storage.  The difficulty lies that in the creation of pools for QoS, resources that are committed to pools are practically locked into them and cannot be easily redistributed to other pools to meet changing demands.  As storage systems age and are filled with data, the various pools are consumed unevenly. For example, consider an array with six pools provisioned as follows: 

Pool #1: SATA, 60TB (usable)  RAID 6  - primary bulk storage: no performance requirements

Pool #2  SATA, 40TB (usable)  RAID 10 - primary low cost storage: capacity over performance

Pool #3  FC/SAS, 30TB (usable)  RAID 5 - primary high performance storage: performance over capacity

Pool #4  FC/SAS, 20TB (usable)  RAID 10 - primary highest performance storage

Pool #5  SATA,  50TB (usable) RAID 5  secondary snapshot storage

Pool #6  FC/SAS, 60TB (usable) RAID 5 secondary snapshot storage

The type of problems storage administrators are constantly dealing with occur when one pool becomes maxed out - making it's associated QoS unavailable. The admin, then has three choices, 1) use a higher QoS, 2) a lower QoS, or 3) add new resources to the pool, if possible.  If they decide to use a higher QoS they may have to deal with performance problems in higher-priority applications.  If they decide to use a lower QoS they will have to deal with performance problems for the application. If they decide to add resources, they might have to interrupt many other applications and take them off line while workloads are shifted. Then you get the ripple effect of "remodeling the kitchen".

When you consider the fact that some storage systems force users to establish separate pools for thin provisioned volumes and thick volumes, the number of pools in the system increases and the fragmentation of resources becomes a much bigger problem.

The best practice for managing storage pools is to do away with them entirely so they don't inhibit access to expensive resources and, more importantly, so they don't soak up so much administrative time and create increased risk of downtime and data loss. 

Pools of disk drives are just a thin layer above bare disk drives where virtualization is concerned.  Considering the transparent nature of system virtualization technology it is almost incomprehensible that storage systems force customers to create these artificial constructs that force hard choices about something as basic as the layout of data on disks.  Vendors with pool-based volume management like to distract customers by talking about whiz bang functionality that doesn't address the core storage problem - the fact that their customers are still doing much of the work that the system ought to do for them.

The best practice then is to replace outdated storage designs with new designs that do not reserve storage resources in pools and do not use pools to create QoS levels. 3PAR InServ storage systems do not use pools and do not reserve capacity for different QoS levels.

3PAR InServ storage systems are used by many of the largest companies in the world, which saves them an enormous amount of money by lowering lower capacity requirements and administrator overhead. For example, Priceline.Com has been a 3PAR customer for many years and they talk about how it has worked for them in this video on YouTube.   

The InServ's data layout starts with the subdivision of all disk resources into 256MB "mini-disks" we call chunklets.  All the higher level RAID level functions in a 3PAR system are applied at the chunklet level, not at the disk level.  RAID in an InServ system is implemented as "micro-RAID" sets which then are concatenated together and formed into virtual volumes that are exported as LUNs.

FWIW, the term virtual volume was used by 3PAR years before the system virtualization phenomenon became the market force that it is today. I only mention this to reinforce the fact that from it's inception, the InServ internal storage architecture was designed to virtualize storage.  It makes storage administration transparent by doing the low level provisioning work on behalf of the storage administrator.

As new storage is provisioned in a 3PAR system, the data is spread across chunklets in small 16KB increments.  All the disk drives (of the same class SATA VS high performance) in the system are used by default, so that data is widely striped for optimal throughput and to avoid hotspots. While there actually are small amounts of capacity pre-allocated for use before storage is provisioned, this is done automatically by the system and it is done in thin slices across all drives.

There are no pools, no constraints, no weeks-long planning efforts needed for storage installations and change management. 

If you are looking to dump your nagging storage administration problems why would you ever go back to pool based storage when that is the root cause of your problems? 

10 common misconceptions about sub-volume tiering

1.  Everybody has it.

2.  All implementations are the same.

3.  Compellent has the most full-featured implementation.

4.  It will use expensive SSD resources for low-priority apps.

5.  It won't work for VMFS data.

6.  EMC is shipping it.

7.  Tiering+SATA+SSDs will eliminate FC/SAS drives.

8.  It only works with SSDs.

9. You will end up thrashing data across tiers.

10. You have to buy it before you can determine if it will work for you.

3PAR's sub-volume tiering, Adaptive Optimization (AO), comes with a lot of features that make it much better than what competitors say. It doesn't require SSDs and it's not for everybody, but if you need it, it will do the job.    

Customer Priceline talks about 3PAR storage

Here's a video that TechTarget produced for us with one of our customers, Priceline.com.

Here are a few highlights from the video:

Priceline.com was one of the first e-commerce players to adopt virtualization.  That may account for why the company's IT organization is known for for it's high availability and ability to adapt quickly to changes in the market.  Given the fact that their business has a broad value-based appeal, their IT organization works very hard to get the best rate of return for their capital expenditures.

3PAR storage allowed them to increase their storage capacity over 400% over the last four years while reducing the administrative load required to manage it all. Ron Rose, ex-CIO at Priceline (now on the Sr. Management Team at Dell) said that they were able to decrease the data center footprint 50% during that time.  Mr. Rose estimated that they were able to reduce the deployment of approximately 100 physical servers and their associated footprint costs, which were equivalent to 106 acres of trees 310 tons of hydrocarbons per year.

Is there a storm brewing around HP's XSP/cloud business?

The news here in Silicon Valley this morning is that HP is planning to overhaul its Enterprise Services Division by laying off 9000 employees and developing a business in automated XSP/cloud data center services.

When HP announced its almost $14Billion EDS acquisition back in 2008, Nicholas Carr and others wrote about how HP needed an IT outsourcing services business to keep up with IBM.   Others, such as Om Malik, suggested the move was designed to bring HP into the XSP/cloud computing age. 

Both camps were probably correct. When it got into the outsourcing business with EDS, HP undoubtedly was hoping for a cash cow business like IBM's, but in the last couple years, the XSP/cloud industry has risen as a serious economic alternative to outsourced services. So what can HP do?  Try to milk the outsourcing business as much as it can, while it simultaneously tries to take a position in the XSP/cloud services business.  A couple years ago it looked like a way to hedge their bets, but the either/or scenario doesn't always pan out the way you think it will and it looks like HP has to make a change in order to optimize its EDS investment. It's not easy changing horses, but that's what you have to do sometimes.

The most relevant aspects of this development in my opinion could be:

• HP could become a competitor to a large number of it's current XSP/cloud services customers, many of whom might decide to turn to other vendors for their servers if they feel threatened by HP's XSP/cloud business. This could actually help HP's new competitor Cisco sell it's new UCS servers to customers that would otherwise be loyal to HP. If there is a customer backlash, HP's servers will probably have to be either cheaper or more technically advanced than its competitors - neither of which are very likely considering the maturity of the technology. HP would also like to increase its networking and storage sales to these customers, but they will both be dead in the water if they perceive that HP is encroaching on their turf.
• Assuming the 9000 layoffs will come from employees providing IT outsourcing services, it will likely mean HP plans on migrating many of its outsourcing customers to their XSP/cloud services business. But customer migrations tend to become low hanging fruit for competitors. The dirty little secret in the IT business is that profits are often made on the backs of competitors' mistakes. If a feeding frenzy develops around HP's outsourcing customers it could turn out badly for HP.
• It appears that HP intends to create the infrastructure for their XSP/cloud services using their own Converged Infrastructure products.  This would give them certain cost advantages for acquiring and installing products in their XSP/cloud data centers. The risk is that HP's infrastructure could have certain shortcomings due to certain limitations of it's own products. If HP's XSP/cloud service competitors use best of breed products and HP struggles to keep up with any of its own products, it could create a tipping point for HP's XSP/cloud customers who want - but can't get - the same service levels from HP as they can from competitors. 

A number of 3PAR XSP/cloud customers are running HP servers in an environment we call 3CV (3PAR storage, HP servers and VMware hypervisors).  It's a stable, scalable, best of breed environment that our customers are very happy with.   HP, the company that makes the servers for this solution could create pressures in the market stemming from their EDS acquisition that have ripple effects on the rest of their ecosystem. If it happens, it happens, and we'll adjust along with our customers and the rest of our partners.  It's all part of what makes this technology business so blasted interesting. There's always something.

A conversation with Mark Cravotta of Datapipe

I caught up with Mark Cravotta from Datapipe recently at a 3PAR event in Las Vegas.  He's a high energy person who is having a lot of fun growing Datapipe's hosting and cloud computing services as well as helping to manage its expansion around the globe.  

Datapipe is a 3PAR Cloud Agile partner and customer who uses our products throughout their line for primary multi-tenant storage, data snapshots, remote replication and all aspects of disaster recovery.

In addition to being customer-driven, Datapipe is also committed to being a leader in green utility computing by reducing the carbon footprint of it's data centers through power purchases from green power producer, Constellation NewEnergy.  

3PAR customer Ultraspeed goes international with DR services between London and Amsterdam

London-based Ultraspeed has succeeded in the managed hosting business 13 years by being smart, opportunistic and service-oriented.  A lot has changed during that time, especially customer expectations for uptime and how much customers rely on their hosting providers to respond quickly when needed. Web sites that are lucky enough to "go viral" can be a disaster if the hosting company's infrastructure is unable to adjust rapidly enough to meet demand.

In March, Ultraspeed opened their second data center in Amsterdam implementing a modular infrastructure design including multi-tenant 3PAR storage, VMware, Extreme Networks switches and customized servers.  The highlight of their Amsterdam site is the ability to offer bi-directional DR services between London and Amsterdam using 3PAR's Remote Copy software. Ultraspeed is a member of 3PAR's Cloud-Agile program. 

In this interview, conducted in February 2010, Jordon Gross, CEO of Ultraspeed and Michael Shanks, CTO joined us for coffee near their offices and talked about their company's history, it's technology, the challenges they face and how they expect things to shape up in the years to come. 

Spinning down Private Cloud: its all about competition

A few weeks ago I posted an opinion here that Private Cloud was essentially a euphemism for making incremental improvements in corporate information infrastructures.  

I go a little crazy sometimes over the definitions of terms we use in storage - mostly because they get spread so broadly and shot full of holes that ambiguity and confusion are guaranteed. That's why the public dialogue about Storage Federation,  a term that could become the next meaningless buzzword, is so interesting to me. Many others have spent some time discussing this and if you are interested in it, follow the previous link to a Wikibon article or this link to an excellent post on Stuiesav's blog.  But i digress.

"Cloud" has been widely thrashed as meaning anything to anyone, but it still has a thread of integrity based on the economic model of buying services, as opposed to products. There is something real about following the money trail that helps bring a cloud conversation back from the reduced oxygen zone to reality. My problem with Private Cloud is that it hasn't had the same economic reality check. Frustration be damned! and I've continued to look for a solution that will help me hear and speak the words Private Cloud without grimacing. 

Recently I heard a definition of Private Cloud that really grabbed me because it has an economic rationality that is forceful and compelling.  

A Private Cloud is the infrastructure that IT organizations create to compete with (Public) Cloud services.

I'm not crazy about the term Public Cloud either because it is redundant. The only reason to use public with cloud  is to distinguish it from Private Cloud. But if Private Cloud is defined in the context of Public Cloud competition, there is not much use for the word Public. Nonetheless, I can live with that conundrum and be happy.  

Can your team compete?

We are entering a new era where information processing can be accomplished internally, externally or both. The proliferation of competition for IT services is going to bring enormous changes to our industry. While we are used to thinking that competition is good, many IT organizations have been virtually protected from competitors for many years and the access that business unit managers now have to competitive services is going to be very hard on IT organizations that don't have the resources, talent and skills to keep up.  

One thing is for certain, nobody will have much time to manage systems and storage anymore.  Autonomic storage that effectively manages many of the things administrators do today is making a huge difference today for our customers. 

We have all sizes of customers, some are very large corporations with many thousands of IT employees and the skills to implement almost anything they want to. These types of organizations are already well underway building large, flexible Private Cloud infrastructures. Some of our other customers are among the largest utility service providers in the world that have built their infrastructures on top of 3PAR storage. Other customers are much smaller, in both traditional IT and utility service provider environments. 

One thing they have in common is spending the least amount of time (and money) possible managing storage.  That helps them do whatever else needs to be done for their business. Does your storage do that for you?

 

 

Why dual controller arrays are not best of breed for cloud storage

 
A couple weeks ago, one of the major storage vendors had two major problems to resolve after one of their arrays suffered a firmware bug-induced failure at one of their cloud (email) service provider customers. They had to:

1. Help the customer get back to normal service levels after they had become unacceptable.
2. Confront a public relations problem after it was exposed by a leading storage publisher.

Meanwhile, their service provider customer had four major problems to resolve:

1. Get service levels back to acceptable levels.
2. Communicate to their customers what the problem was and how it was being addressed.
3. Re-engineer a solution to avoid the same happening again.
4. Credit customers for not delivering against SLA terms.

A vendor employee tried to address their public relations problem this way in his blog:

"OK, I'll take the blame for this -- sort of.  We pride ourselves in putting a lot of thought into our customer designs.  I'd argue that we're really, really good at it as well.

But not everyone is 100% sure of how their application will grow over time -- unfortunately, we're not psychics.  And, let's be honest, not everyone necessarily wants to pay for redundancy we like to put into our designs. 

We don't always get to directly engage all the time, either -- with products such as the (blanked out), most of this stuff moves through the channel.  Somebody calls up one of our partners, says that they want to buy one of our products, and one gets sold -- and a lot of product gets sold that way."

I understand the desire to explain how messes become messy, but I'm not sure why he felt the need to speculate that his company's business partners or that their customer's budget were key elements of the problem. That is tantamount to saying, "All of our (blanked out) customers could have the same thing happen to them too."  Anybody who has ever been close to one of these melt-downs knows there are many variables involved - including vendors underbidding each other and shaving elements from their bid in order to win the business.  

From a distance, it looks like the vendor's response to the customer was good, although there apparently were some issues with failure notification from the array when the event occurred. I wouldn't call these sorts of things "Perfect Storms", but there are unfortunate scenarios where multiple things go awry. All vendors have these sorts of bad days, which serve as painful learning experiences. Unfortunately for customers, it's one of the ways vendors improve their customer support processes.

The customer also wrote in his blog, explaining the situation to their customers:

"Our SAN vendor analyzed the system logs for the event and determined that the service processor failure occurred due to a unique bug in the specific version of firmware on the system. Our vendor performed an emergency upgrade. The newer version of firmware includes a fix for the bug. We are taking additional corrective actions to make certain that there is enough spare capacity on the SAN. This will assure it performs without performance degradation in the event of a single hardware failure."

The reparation sounds reasonable, but it's not what I would call best of breed either. I'll explain why in the remainder of this post.  

 

The old trusted dual controller just can't keep up

The explanation the service provider gave to their customers was only half correct. Yes, the failure in one controller was due to a firmware bug -and yes, all vendors find out about some of them at customer sites - but the inability of the surviving controller to handle the workload was another matter altogether.

The major defect of all dual controller designs for service provider applications is the uselessness of write cache when operating in degraded mode on a single controller.

When a dual controller array has a controller failure, all traffic is failed over to the surviving controller. However, this controller can't afford to place writes in cache because if this controller also fails any un-flushed writes in cache would be lost- making the recovery process all the more painful.  As a result, the throughput of the controller degrades significantly because writes now take several orders of magnitude longer to process as each write must be completed at the physical disk level, instead of in fast cache memory.  When you consider the sort of read/write ratios involved with an email application (heavy writes), it's not surprising to hear that it took 32 hours for the system to get caught up.  I suspect that if the surviving controller had been able to use write cache, the customer might have experienced some amount of service level problems, but not nearly as bad as they suffered.

Write performance during array component failures is an important point that many customers give insufficient weight to when making their purchases. Public service providers certainly need to understand this. The exact same scenario - controller failure and subsequent drop in service levels - could certainly happen to a traditional data center customer, but the ramifications of this scenario are not as ugly as they are for a multi-tenant public service provider.

This case is a perfect example of how an older architecture is incapable of meeting the requirements of the new cloud service business model.  If you are a cloud service provider reading this and wondering if you might have a similar exposure to a controller failure (including 3PAR customers with dual -controller arrays), my advice is to review what you have and start thinking about what you should expect if you have a controller failure and how you might want to deal with it on both a short-term and long-term basis. Best of breed cloud storage should not include dual controller arrays. 

 
Their solution is to buy more and utilize it less 

One of the identified corrective actions is having "enough spare capacity on the SAN", which in this case involves installing a second array. Without knowing the inside scoop, it looks like the idea is to split the workload across the two arrays so that if a controller failure occurs in either array, the performance drop won't be as noticeable. The array that doesn't suffer the failure will keep working as expected and the array that has the failure will only have half the load to deal with.

There are two primary problems with this "fix"

1.  Performance will still suffer on the array with the controller failure
2. The I/O load will continue to increase over time

You are always going to have performance degradation of some sort when you can't use write caching, unless you are only reading data - which isn't the case here. It is flat out wrong to assume that a performance problem will not occur.  Regardless, with the new two-array SAN, whichever system has the controller failure should be able to get caught up much faster than the 32 hours this customer had to wait. Of course, the customer's capacity and I/O load will almost certainly increase over time, and as that happens, the strategy of splitting the load between two arrays loses its effectiveness. 

Along with adding the controllers, they are also certainly adding disk drives, and some notion of what "reasonable" utilization limits should be for them. The problem with limiting utilization as a best practice is that it puts the stamp of approval on inefficiency - not only for capacity utilization abut also for the power and cooling required to support all those underutilized drives.  Most legacy arrays have built-in inefficiencies in the way data is laid out on disks, making it virtually impossible to achieve uniform utilization across all disk resources. The result is uneven consumption of disk capacity, as well as uneven I/O service levels among different disk groups, which is another variable in how much performance degrades following a controller failure in a dual controller array. 

Finally, the customer now has two arrays to manage, including multipath connections, SAN zones, and all other aspects of the configuration, which all contribute down the road to change management complexities.  The result is a net drag on administrator effort and an increased TCO.  

 

How many do you need?

A true best of breed solution would address the root-cause deficiency in the array's design, without creating additional management and cost burdens to the customer. Obviously, more than two controllers are needed.  But how many controllers does a cloud service provider need in an array?  The answer is at least three. Why? Because when a single controller fails, there can still be two surviving controllers working together, mirroring their cache contents, and performing fast writes to cache memory.   That said, controllers are usually packaged in pairs for redundancy purposes, which means that the most likely configurations will have four controllers. 

If you compare a single quad controller array with two dual controller arrays there are some key advantages that immediately jump out:

1. No or limited loss of performance after a controller failure 
2. All drives and cache can be used to service all workloads
3. Managing a single array significantly reduces cost and complexity

 

 
  

A better recipe for maintaining performance levels

The next question is; "Is there a suitable quad controller array that the customer could have used instead of the two dual controller arrays they have?"  Yes, 3PAR's F400 or T400 arrays are both quad controller arrays. The disk drives in these arrays can be either SATA or FC, or a mix of both types if the customer wanted to implement tiering. Product information of the F400 can be found here, and the T400 here.

However, simply putting four controllers in an array does not necessarily guarantee that they will be able to sustain write caching if one of them fails. The array must have the ability to remap and re-mirror the write cache contents of all four controllers to the surviving controllers following the loss of a controller.  It's an interesting geometric sort of problem: There are four controllers, each with their own cache and cache that is mirrored from the other controllers in the array. All cache contents, including mirrors, need to be distributed evenly across all controllers to avoid congestion and load imbalances. All cache content, including mirrors needs to be accounted for within the array so that if a controller fails, the other controllers will be able to identify all the surviving original and mirrored copies of data. For cache data that has lost either a primary or mirrored copy, a second (new) copy needs to be made. Finally, the amount of data in cache may need to be re-leveled (decreased) to fit into the degraded cache capacity (3 controllers instead of 4). 

The software for doing this in a 3PAR array is Persistent Cache.  Product information on Persistent Cache is here (PDF)

I made a 9 minute last year video describing how Persistent Cache works.  Here it is again. Thanks for watching.

Top 10 design principles for information infrastructure products

(A quote from Dieter Rams - former Chief of Design at Braun)

It's hard to think of a company that has had more success with it's product designs than Apple. When you look into how Apple did it, you find out about Jonathan Ive - Apple's lead industrial designer - and how his designs have followed the philosophy outlined by Dieter Rams, who was the lead designer for many years at Braun.  When you compare photos of their designs, it is obvious that Ive has a strong appreciation for Rams' work.

What Ive and others have found compelling in Rams' work is nicely summarized in the design principles Rams used at Braun for many years.

• Good design is innovative
• Good design makes a product useful
• Good design is aesthetic
• Good design makes a product understandable
• Good design is unobtrusive
• Good design is honest
• Good design is long-lasting
• Good design is thorough down to the last detail
• Good design is environmentally friendly
• Good design is as little design as possible

The design goals for consumer products differ considerably from those for industrial products. For example, aesthetics and innovation tend to be less important than reliability and ROI - two characteristics that didn't even make it onto Rams' list of design principles.  But there are also principles that certainly belong to both, such as making a product useful and unobtrusive.  So, what should the 10 design principles be for information infrastructures products? Here's my list:

• Good design makes a product useful (it solves customer problems)
• Good design has recognized limitations
• Good design is unobtrusive (needs minimal management)
• Good design has an attainable ROI
• Good design makes efficient use of resources
• Good design is scalable (capacity, performance & management)
• Good design is resilient (sustains performance through sub-optimal conditions)
• Good design makes a product understandable (facilitates planning & changes)
• Good design is long-lasting (and accommodates future innovations)
• Good design is environmentally friendly 

Producing this list was much more interesting than I thought it would be.  For starters, it took me some time to get settled in a customer's perspective - as opposed to my usual vendor employee perspective. (l have this wonderful hammer you need).  Also, to clarify a point, the idea of management scalability involves the number of people who can effectively manage and control a system simultaneously. That might not be a concern for smaller IT systems, but it certainly is for large-scale systems.  

What would you change?  Would you reduce or expand this list?

Zeroing in on a definition for federated storage

(Federation Square, Melbourne)

There have been some excellent discussions recently in the storage blogosphere and on Twitter about the concept of Storage Federation among a number of storage people; known by their Twitter IDs as @stuiesav, @storageanarchy, @rootwyrm, @davegraham, @bwhyte, @ianhf, @esignoretti, me (@3parfarley) and others - as the interest continues to increase. 

There are two aspects of the discussion that I think are fascinating: first is the role of social media as the means to include customers, vendors and others in an open discussion that typically is conducted privately by a vendor preparing to release a new product or feature, second is the challenge of defining a storage capability with sufficient focus and vendor independence so that is is meaningful. There has been some amount of skepticism about this effort, suggesting that we are predetermined to end up with ambiguous terms that can be interpreted (spun) by anybody (any vendor) to mean anything (our product does it). I'm hopeful the results can be better than that, but subjectivity is the only constant of social media and it's not likely that everyone will like whatever happens.  

In general, the language of digital storage has many overlapping meanings, which causes a fair amount of confusion in our industry. I've been dealing with this problem for many years, going back to when I wrote Building Storage Networks and had the challenge of trying to invent generic terms for functions that had been coined by vendors and tied to specific products. My interest in defining Storage Federation goes beyond my role as an employee for 3PAR.

The notion of federated storage has been around for several years, but recently it came to light when Pat Gelsinger of EMC referred to it during a press briefing on March 11. EMC blogger Chuck Hollis wrote about it afterward and there was some chatter, culminating in a blog post by @Stuiesav on April 2, which proposed that the discussion about Storage Federation was just marketing hype attempting to rebrand storage virtualization.  EMC's Barry Burke (@Storageanarchy) and I (@3parfarley) both agreed that this was not the case this time around and then in the last couple days the discussion fired up again. The problem with Twitter is the limitations of 140 characters per tweet.  It's surprising what can be done with so few characters, but it does have limitations.

Skeptics Allowed

This weekend, @ianhf  posted in his Grumpy Storage blog echoed @Stuiesav's skepticism and expressed his perspective (as a customer) as to the things he would like to see when new technologies - in this case Storage Federation - are introduced.  Here are some of  the items from his list (Some of them don't fit our definition exercise because they assume there is a new product being introduced, which is not the case here).

• The specific customer requirements & problems this addresses & justify how
• The use cases this feature / function applies to, and those that it doesn't
• Why & how this feature is different to that own vendor's previous method for solving this problem
• Provide clarity over the non-functional impacts of the feature before, during & after it's use - ie impact on resilience, impact on performance, concurrency of usage etc (including provide up-front details of constraints)
• Naturally you'll also expect me to require TCO & ROI of the feature, and any changes to the models as a result of this feature

   

So lets get started (already!):

The definition of Storage Federation that was kicked around on Twitter is something like:  "the transparent, dynamic and non-disruptive distribution of storage resources across self-governing, discrete, peer storage systems."  (And yes, I did elaborate a bit on this while I was writing based on bits and pieces from comments I read and further thought of my own.)

The idea is to have multiple storage systems cooperating as a team (as opposed to under the direction of an external entity) to place data in the aggregated storage resources (LUNs or volumes) of all participating members. An example of Storage Federation is how Dell/EqualLogic arrays distribute their volumes over multiple systems. When a new EqualLogic array is added to an iSCSI SAN, the administrator is asked if the array should be placed in the same group as other arrays in the SAN. If this is done, the arrays start splitting their volumes (and workloads) across both arrays.

Five examples of storage federation capabilities are:

Storage expansion: You want to increase the storage capacity of an existing storage system that cannot accommodate the total amount of capacity desired. Storage Federation allows you to add additional storage capacity by adding a whole new system.  

Storage migration:  You want to migrate from an aging storage system to a new one.  Storage Federation allows the joining of the two systems and the evacuation from storage resources on the first onto the second and then the first system is removed.

Safe system upgrades:  System upgrades can be problematic for a number of reasons. Storage Federation allows a system to be removed from the federation and be re-inserted again after the successful completion of the upgrade.

Load balancing:  Similar to storage expansion, but on the performance axis, you might want to add additional storage systems to a Storage Federation in order to spread the workload across multiple systems.

Storage tiering:  In a similar light, storage systems in a Storage Federation could have different capacity/performance ratios that you could use for tiering data.  This is similar to the idea of dynamically re-striping data across the disk drives within a single storage system, such as with 3PAR's Dynamic Optimization software, but extends the concept to cross storage system boundaries.

These are all examples of Storage Federation that Dell/EqualLogic storage systems are capable of today. It's not intended to be an endorsement of their distributed volume manager, I am simply using it as an example of Storage Federation. Saying that, if you look for "Storage Federation" on the Dell/EqualLogic web site, you probably won't find it today because they don't describe it that way, but that doesn't mean that it is not in the product.

That's not it

People also want help understanding what the definition of Storage Federation should not include. Here are my thoughts:

In-band storage virtualization systems like IBM's SVC can provide all the capabilities listed above for Storage Federation. However, in-band virtualization products govern the behavior of the other storage systems networked to it and Storage Federation involves self-governing, peer systems. Another way of saying it is that Storage Federation does not have functionality in the network between host systems and storage.

File, object and data distribution technologies like EMC's Atmos certainly provide a type of federation insofar as multiple computer systems can access the same data objects from multiple locations that may be separated geographically. However, this capability primarily migrates files (data objects) - and is functionally orthogonal to Storage Federation, which works on volumes. A term like Data Federation is probably more precise than Storage Federation for this sort of capability. 

Clustered storage systems, like 3PAR's InServ storage systems or clustered NAS systems are not examples of Storage Federation because they function as single, scalable storage systems, not as discrete storage systems that are networked together.  Clustered storage systems can work together in Storage Federations with other clustered or non-clustered storage systems. In that case you could have a Storage Federation of clustered storage systems.

How is this different?

I thought I'd make a little effort to respond to @ianhf's requests.

The main problem Storage Federation addresses the limitations presented by a single storage system, such as capacity, performance and maintenance availability. 

Currently, when a new storage system is added to an existing environment, there are server administration tasks that need to be done in order to migrate volumes from an existing storage system to the new one. In addition, there is usually some amount of downtime and/or performance degradation associated with the migration and configuration of new data paths.  By contrast, with Storage Federation, a new storage system could be added without having to reconfigure host data paths and the migration would be processed transparently and dynamically, without any downtime or loss of path redundancy while the migration is in process.

Increasing performance for a particular volume in a Storage Federation is somewhat less obvious because the other federated storage systems might not have more performance-resources available to boost performance. For instance, if a new storage system inserted in a Storage Federation does not have more disk drives (spindles) or more flash SSD capacity, there is no guarantee that any volumes moved to it would provide faster performance. There is some likelihood that performance could be improved by moving a volume to a storage system running at lower utilization levels, but maintaining lower utilization levels is not realistic or very cost-effective. All this said, it is possible for a Storage Federation to use the aggregate resources of multiple storage systems to increase performance of a single volume - such as by spanning all the disk drives in the federation and by using the cache of all participating storage systems. An example of products that provides this capability today are the Dell/EqualLogic storage arrays. 

The availability of storage systems needing maintenance could be greatly improved by Storage Federation if an existing storage system can be removed from the federation after having it's volumes evacuated (think v-Motion but for storage systems) to other storage systems in the federation. After being removed from the federation, the storage system could be downed to have any sort of maintenance operation done to it without risking the availability of any of the volumes that had previously been on the system undergoing maintenance.Obviously, all of this would take time planning and ensuring the Storage Federation would not be overloaded with data and workloads, but there are probably many customers that would prefer to do maintenance to storage systems when they are offline. 

One of the possible exposures of Storage Federation is the increased exposure to system failures. For instance, a Storage Federation that distributes a single volume over three separate storage systems is 3 times more likely to have a failure than a Storage federation that does not allow volumes to span across storage system boundaries. FWIW, this is the main weakness of the Dell/EqualLogic implementation of Storage Federation. 

That's all for now

It's late and there is plenty of material here for the grinder. Please feel free to comment in any way; agree, disagree, correct mistakes and ask questions.  Thanks for reading.