StorageRap

We and the SWCSA wish you a Happy Thanksgiving

Dave Graham (from EMC) and I got together in virtual steering wheel cam reality to talk about cranberries, liposuction and Avamar de-dupe.

Oh yes, Happy Thanksgiving from the SWCSA!

Curtis Preston - the super deduper

I ran into my friend Curtis Preston (W) - the super deduper - at Storage Decisions in San Francisco last week.  Happy Thanksgiving Curtis!

Don't hoard disk drives, there are plenty to go around

When you buy a car do you get all the gas for it up front?  Do you buy all the cafe lattes you are going to drink for the next 3 years all at once - and save them for whenever? 

Animals hoard things, like nuts, for instance.  Under unusual economic pressures, people sometimes hoard things too.  But we shouldn't have to worry about disk drives.  There seem to be plenty of them to go around. 

Regardless, many customers hoard storage capacity because their technology choices limit their behavior. When they buy storage they make best-guess estimates about how much they will need and then hope the estimates are close because it's such a pain when they are off.  You know what I'm talking about.

Thin provisioning makes all that a huge waste of time by letting customers buy only what they need when they buy it and allowing them to add more without spending weekends-at-work to make it happen. 3PAR has been providing thin provisioning to customers that don't want to over-buy storage for the last 5 years and we continue to lead the development of thin technology.  We also lead the industry in wide striping - a disk layout technology that delivers optimal I/O performance from every disk drive in a system.

These days we are very excited by the promise of reclaiming storage - which is a way to return unused storage capacity from a file system back to the array's available capacity.  This is an example of what we call "getting thin".  We are happy to see Symantec's Storage Foundation emerge as the "thin" leader from  the server-software side of the industry.

There are plenty of disk drives - we won't run out.  So with the current economic climate, do you want to over-buy on your next storage purchase or would you rather spend less and get more in return?

Steering wheel cam is safer than driving Prius!

Several viewers of this blog have remarked about the potential danger of driving while making videos.  The steering wheel cam conference call that Stephen Foskett and I made to discuss storage innovation seemed to generate the most concern.

The Steering Wheel Cam Society of America (SWCSA) wants to re-assure viewers that no animals are hurt in the making of these videos and that we are operating our vehicles in a completely safe manner (except for Stephen who was using the considerably more risky dash cam.)

This video shows why operating a steering wheel cam is safer than operating a Prius. In fact, there are many things that are more dangerous than operating a steering wheel cam, such as crossing the street or working the drive through window .

Symantec's Veritas Operations Services and green storage

Symantec announced their Veritas Operations Services today, built around their Storage Foundation and Veritas Clustered Server products.  I think there is a lot more than meets the eye with this - utility computing customers of Veritas and 3PAR will soon be able to take advantage of the recently announced Veritas Thin Reclamation API which reclaims storage space that is no longer being used. The end result is much more efficient use of storage, which means fewer disk drives will be needed, which means the cost of storage and the power and cooling costs associated with it can be significantly minimized. 

If you are worried about spending too much on storage and on managing storage, you need to look into this.  If you are in California you can even get an energy rebate from PG&E for buying 3PAR storage.  Its not greenwashing or posing - its real energy efficiency.

Nominating VMmark/IO for Mixed Workload Benchmark

A lot of people either enjoy or are frustrated by the great benchmark soap opera .  I'm not sure if it really matters all that much to customers, because if it did, they would require more from the vendor community.  Of course part of the problem with benchmarking is that it's much more alchemy than science and is loaded with difficult, non-intuitive concepts such as - "What does a 100% workload represent anyway?"  and "how come response times at various workloads change non-linearly when the number of IOPs at those workloads are linear?"  and "why did the SPC publish benchmark results for one vendor's products that were submitted by a competitor?"  

A couple weeks ago the Storage Anarchist wrote about the lack of a benchmarks for mixed workloads in enterprise environments.  Anarchist thinks that working with his co-worker, the mysterious Dr. Kartik would make things better and has complained in his blog about the lack of response from other companies in joining the effort.  Maybe EMC is looking for subsidies to help pay Dr. Kartik's salary because  Dr. Kartik seems to take interest in this a couple times a year - when I assume the work gets slow. 

Recently IBM worked with ESG to publish a mixed workload benchmark based on a modified version of VMware's VMmark benchmark for IBM's 5300 system.  I say "modified" because ESG needed to increase the I/O component in VMmark to make it work for measuring storage workloads.   Anarchist criticized this work (there is a standing tradition for storage bloggers to slam competitive benchmarks) and seemed eager to shut the door on the methodology that was used, arguing that it could not be made repeatable due to the way the Jetstress component was implemented.  I won't further a guess about Anarchist's motivation in wanting to dump on this new methodology, but it seemed a bit premature (even for a storage blogger) and gave me the impetus to look into it further. 

What I found was something that was way better than I expected.  the approach used for this benchmark has a lot more meat on the bone and promise than anything else I've seen discussed anywhere (and I mean ANYWHERE).  Not that there is a vehicle for doing it, but I'm nominating this modified VMmark approach as the method for benchmarking mixed workload environments. Seconds or dissenting opinions are encouraged to comment.

To Anarchist - clever engineers live for these sorts of challenges and this one hardly seems insurmountable or as expensive as some science projects are. Brian Garrett from ESG labs, who many of us know by reputation, if not personally, is a very smart, very creative and credible person who has built a long career in performance alchemy.  I believe Brian would steer this development more objectively than your buddy, Dr K.

Top 10 Network Storage Innovations

Here's my list of what I think are the top 10 innovations in Storage Networking.  I decided to use a slightly different lens than Stephen Foskett, who wrote his top ten list of enterprise storage products. 

1. Netware 286. The first major file and print solution.  Despite what people at Sun think, Netware was more influential than NFS.  Microsoft ended up kicking Novell's butt, but the market and memes for LAN servers originated with Netware 286.
2. Network Appliance FASServer Filer.  The first major storage appliance.  Auspex was earlier, but Netapp innovated in many more ways and really defined almost everything NAS, including multi-protocol filing and file-based snapshots.
3. Brocade Silkworm.  The first Silkworm switches were the glue that allowed the fledgling Fibre Channel technology to work amazingly well.  Silkworms defined SAN fabric operations.
4. SAN Volume Controller.  The have been many attempts to make SAN-based virtualization work, but SVC has been by far the most succesful.  It set the standard for SAN virtualization. 
5. Digital DLT 2000. Although the DLT drives started life as VAX-oriented products, they had limited success until they burst on the scene in the late 90's as high performance, high capacity LAN backup drives.  They killed off most helical scan competitors and established linear tape as the overwhelming favorite.
6. Veritas Volume Manager.  Host-based software that let every admin with the courage and skills perform the logical configuration of storage. It allowed many people to understand volume management as a discrete technology area in network storage. 
7. EMC Symmetrix 3.0. The EMC Symm 3 with SRDF remote copy capabilities is probably the single most important innovation in the history of network storage. Block-based snapshots and multi-tenancy are also major innovations that EMC led the market with.
8. 3PAR InServ Storage Array. Its hard to sell SAN storage today without some form of Thin Provisioning.  In time, though, 3PAR's wide striping may prove to be more influential.
9. Data Domain.  Data deduplication is gaining ground very quickly in the network storage industry, thanks in large part to Data Domain's success and leadership. 
10. VMware VMFS.  The ability to maintain a logical data store for guest systems running in a virtualization server servers has been critical to the success of server virtualization.  

Here's a few honorable mentions:

Vortex Retrochron  - CDP that was way ahead of its time.
Palindrome Network Archivist  - Another product ahead of its time, with automated data archiving.
Seagate Barracuda FC - SANs wouldn't have happened if not for Seagate.
EqualLogic PS Series - Ease of installation/management makes SANs achievable for SMBs.

Rolling with storage innovation

Stephen Foskett is full of good ideas, such as asking which storage products are innovative and why.  So in the first steering wheel cam duologue, he and I talk about innovation in storage and then discuss a few products we think were innovative.  The discussion could have gone on much longer as there have been many worthy products.  Comment here or post a list of your own - what 5 or 10 storage products do you think have been innovative?

Anybody else talkin' to themselves?

Time for a mental health check.

Anarchist's Incredible Journey

Yes, its long, but well worth the time.  Nobody (and I mean NOBODY!)  writes like the Storage Anarchist.  His recent travelogue/storage blog  should get an award of some sort.  So grab a cup of coffee, tea, chinotto, Chianti or whatever and have a read.

Most of us have some fear that our work lives will come crashing into our personal lives - giving our loved ones cause to express their frustration with our work-tech OCDs.  In a reverse twist, he imports elements of the Joie de Vivre from his recent European cruise into the storage blogosphere and mashes it up with recent stuff from the EMC PR and blog cauldron. 

Yo Anarchist, you could have another career writing Rough Guides or Jetlag Guides!

Barcelona is wonderful.  Much agreement here.  Any Tapas?

About the benchmark comments in your post  - how do you want to proceed on the benchmark front?  The stuff from a year ago is - well a year old by now and it hasn't been updated by anybody since then (last time I checked).  It looks like a dead thread to me.  Is this something you can help organize?   (OMG - it was just updated today - there might be a there, there?)

-BB - a photo I took at Parc Guell:

And for Zilla-man and Storagebod, a very cheesy video about Molvania (see the jetlag link above)

StorageRap plays the Chuck Hollis Atmos guessing game

Chuck Hollis, the super-verbose blogger from EMC really outdid himself yesterday with his "guess what it could be" introduction to Atmos (the market hype formerly known as Maui).

Usually when a company announces a new product, people try to write about what it is and how customers would use it, but in this case the His Hollisness apparently had a senior moment and instead squeezed out a stream of consciousness on what it isn't. 

Yo Chuck, get an editor!  (But it would require hazardous duty pay)  I'll do you the favor,  Based on my reading today, I think you wanted to say something like this.

We have a new software product called Atmos, it works with the Hulk hardware platform which is  OTS PC server systems and SATA drives.  You can probably buy your own systems, but we're not sure how support is going to work if you do that. 

Atmos is accessed using web-access methods - such as browsers, applets and Javascript.  We've tried to make it as secure as possible, but its not the sort of storage product where you would put the corporate data jewels, if you know what I mean - things that might be considered "discoverable."

Atmos metadata is a key part of determining where data copies should be located inside a WAN. Atmos' policy engine monitors data access for all data objects and determines if more copies are needed in the network.  Not sure what the system's behavior is for data objects that were copied to multiple (or all) sites but then later becomes old and stale - and rarely, if ever accessed again.

There is no integrated RAID data protection, which is a big departure for us.  The idea is that all data in an Atmos network will reside in multiple sites, regardless of its access frequency. It's not likely that a customer would have a disaster that spans multiple, geographically dispersed sites. 

(As Chuck says) "Use your imagination."

Distributing data with Atmos solves the big problem customers have where a single copy of data "sitting on one spindle" (Chuck's example) in the corporate data center becomes a bottleneck.  

We did this all ourselves because no startup decided to bet the farm on it and sell it to us. We could have announced it earlier, but we weren't sure what to say about it - and still aren't.  Atmos has no competition. It fills a huge need all by itself as long as you can figure out what that is.

Please let us know if you figure that out!

Atmos, Dead or not dead? Innovative or repetitive?

The storage software formerly known as Maui  is now called Atmos. (Thanks Storagebod for the photo idea).

In a nutshell, Atmos looks like its a new wrapper for your uncle's old WAFS , but with an object-based access method and policies that apparently are applied at the object level.  Whether its object-ness is deeply embedded or a higher level application layer remains to be seen.  In other words, it might have a more familiar file system under the cover with a translation layer providing the object mechanisms.

Until more information is available, I'll put Atmos in WAFS storage windmill jousting camp that has attempted to add value by making multiple copies of data objects and storing them at multiple sites around the globe.  It follows a time-tested business model of EMC's that works by convincing customers to fill up multiple storage products at multiple locations with the same data.  Its good business when you can get it.

Wide area file systems have been attempted many times before and have gone into the dark cold night alone or have been folded into WAN offerings of networking companies. Is it a feature or a product? - only your local soothsayer knows for sure.  As most previous WAFS technologies have ended up being bought up to become features of networking products, this announcement from EMC could even restart the rumor mill about a Cisco acquisition. 

So storage bloggers, the question is whether or not Atmos is innovative.  I'd say the wide area stuff is not innovative, the object stuff is not innovative and the policy stuff is not innovative - but the combination of the three together might be - and there could still be key parts missing - such as network components. 

Sometimes it helps to understand a new technology by looking at what it competes with. Here's what I think Atmos competes with:

The now defunct Cisco WAFS

Globalscape WAFS

Expand Networks WAFS

Packeteer WAFS WAN Optimization

XtreemFS

Riverbed Wide Area Data Services

F5 WANjet

Feel free to add your own past or present Atmos competitors to this list or to offer adjustments to this opinon.

An interview with cvent's Dwayne Sye

Dwayne Sye, CIO of cvent is a long-term customer of 3PAR's. In this interview he talks about cvent's business and how 3PAR has helped cvent grow their business and adapt quickly to the changes of virtualization.

Storage management by balancing branches in InServ clusters

This is a fairly long post - you might want to get a cup of coffee.

George Crump at Storage Switzerland recently wrote an article on clustered storage. It's a good overview that contrasts clustered storage with monolithic systems and distributed systems.

One of the advantages of clustered storage that George wrote about is it's modular scalability.  I thought I'd take his cue and explain how modular scalability works with 3PAR InServ storage clusters and discuss how clustered storage management - a process I think of as balancing branches -   differs from storage management processes with legacy storage arrays. At the end of the day, scalability is tightly coupled with storage life cycles. The final section of this post discusses upgrade scenarios that include life cycle planning as part of the decisions that would need to be made.

It all starts with understanding the main components of a 3PAR InServ cluster. 

1)  Nodes (called controllers in legacy storage systems) are always implemented in pairs.  A node pair is a highly-available, synchronized control point for I/O that manages all resources - including cache memories in the nodes and all disk drives that are connected to them.  3PAR node pairs are active/active, all cache is mirrored and both nodes can access every disk drive connected to the pair.

2)  Cages (often referred to as shelves in legacy storage systems) can hold up to 10 drive magazines.  

3)  Each drive magazine has 4 drives (all the same drive type in a single magazine) and fits in slots inside cages. That means each cage can have a maximum of 40 drives.

4)  Nodes and cages are connected to each other over a high-speed, low-latency backplane. 

5)  All nodes communicate with other nodes in the cluster over the same high-speed, low-latency backplane.

I think of the collection of node pairs, their affiliated cages and subsequent magazines as a branch of the cluster.  The graphic below illustrates the components of a basic branch.

The smallest InServ clusters have a single branch and the largest have four branches.  Each branch can have a maximum of 8 cages.  Following the math, the largest InServ clusters can have up to 1280 disk drives (4 branches x 8 cages x 40 drives).

Cages and magazines are interesting primarily for the role they take in redundancy schemes.  Component redundancy in InServ clusters can be set at either the cage or magazine level. Cage level redundancy means that a cage outage can occur without losing data while magazine level redundancy means a drive magazine failure can occur without losing data.  In either case, data is immediately re-protected with automated aggressive action, including micro-RAID and micro-sparing technologies.

Disk drives are the most important variables of any branch.  Keeping with the branch analogy, disk drives are the leaves, performing most of the work that is done by the branch.  Disk drives belong to the branch they are initially placed in and should not be moved to another branch.  A well-balanced InServ cluster has branches with an equal number of disk drives, producing an equal amount of work. 

Thin, wide striping is a technology that spreads volume capacity in thin pieces over as many disk drives as possible. This white paper from Sun (written in 2008) discusses the performance benefits of thin, wide striping.  People tend to associate 3PAR with the thin aspect of our striping, but its actually a combination of thin and wide striping that is responsible for delivering consistently high mixed workload performance in 3PAR InServ clusters.  The details of thin, wide striping are managed by the system, as opposed to being a manual process. This is a significant difference from manual storage management tasks that are done in monolithic and distributed storage systems.

Thin, wide striping implies that LUNs exported by InServ clusters span branches (although it is not required that they do).  I/O's from host systems terminate at a particular branch, but the data with the I/O may be located on disk resources in another branch.  When that happens, the I/O is fulfilled by transferring data between nodes in different branches.  The nodes in each branch are responsible for all disk I/Os within their branch.  The transfer process between nodes is several orders of magnitude faster than disk I/Os, which means that for all practical purposes, cross-branch I/Os complete as quickly as in-branch I/Os. Large InServ clusters with multiple branches can sustain extremely high I/O rates across the cluster without noticeable performance degradation.

Monolithic storage arrays typically employ coarse, narrow striping that spreads a volume's capacity over a relatively small number of disk drives.  This is usually a manual task that does not scale or age well over time as the number of applications, users and disk drives increase.  The symptoms of coarse, narrow striping are low disk utilization levels and unbalanced resources, which in turn results in bottlenecks and performance problems. Automated, thin, wide striping in 3PAR InServ clusters eliminates the thorny processes and disappointing results associated with monolithic arrays.

The key to storage management with InServ cluster - and the key to scaling capacity and performance is to properly balance resources and workloads. The optimal  way to maintain these balances in a 3PAR InServ cluster is to configure symmetrical branches with the same components.  Automated thin, wide striping takes care of disk-level details including all data layout tasks such as volume allocations, RAID set definitions and spare reservations.

There are two fundamental scalability concerns for storage: capacity and performance. Capacity scaling is accomplished by installing new drive magazines. Performance scaling is also accomplished by adding additional drive magazines and may include adding branches.  3PAR InServ branches are limited by disk performance, not node performance, which is another way to say that InServ clusters with less than 320 drives can generate maximum I/O levels with a single branch cluster.  

As new disk drives are added to an InServ cluster, existing volumes (or LUNs if you prefer to think of them that way) can be redistributed over the new aggregation of disk drives using 3PAR's dynamic optimization (DO) software. DO uses both micro-RAID and micro-sparing to ensure data protection levels are maintained through this process.   

If I/O performance requirements exceed the capabilities of 320 disk drives, new branches can be added to the cluster.  Optimal load balancing for the cluster is achieved if all branches in the cluster have the same number of disk drives. Obviously, this approach has a higher cost than simply adding disk drives, but when performance is critical to the success of the business, the additional cost is justifiable.  DO is used to re-distribute the workload across disk drives in both branches.

So let's consider a couple upgrade scenarios.  First, imagine a single branch cluster with 80 drives in 4 cages and 5 magazines in each cage.  Assume the system needs to be expanded with another 40 drives (50% increase in capacity), what would be the best way to do this? 

Basically, this expansion depends on the redundancy protection level desired.  There is no need to add another branch because the number of disks after the upgrade is less than 320.  If cage level redundancy is used then you would want to add 2 new cages and put 5 magazines in each of them,  If magazine redundancy is being used then 3 magazines could be added to 2 of the existing cages and 2 magazines could be added to the other 2 cages.  In both cases, DO can be used to redistribute the workload, but it might not be necessary depending on performance metrics and expectations.

Another consideration is the life cycle of the cluster. If it was relatively new and the additional drives were added early in the cluster's life and there are expectations to add hundreds of additional drives in the next couple years, it may make sense to add another branch with a couple cages. However, that decision would probably be based more on budget timing than technology requirements. 

A second more interesting scenario involves an InServ cluster that has 200 drives in 5 cages with 10 magazines in each cage (each cage is full) and where capacity needs to be doubled to reach a total of 400 drives in the cluster.  What should one do in this case?

For starters a second branch is needed because there will be more than 320 drives.  To balance the system, the 200 new drives should all be placed in the new branch.  It would not make sense to increase the number of drives in the first branch and have fewer drives in the second branch.  You would also want to run DO to balance resources and workloads evenly across branches.

Now let's extend this scenario and assume that nine months later this 2-branch cluster will need another 200 drives.  What should one do then?

At this point the total number of drives would be 600, fewer than the maximum number for a 2-branch cluster (640).  A choice should be made to either add a 3rd branch or expand the number of disk drives in the existing branches.  This decision would be based on life cycle expectations for the cluster. 

If additional, similar capacity upgrades are anticipated in the future, it would make sense to add a 3rd branch. However, if the cluster is unlikely to exceed the 640 disk, two-branch maximum in the remainder of its life, it would make sense to install the 200 new disk drives into the existing 2 branches. This would require adding three more cages to each branch and then placing 25 magazines into each branch (10 + 10 + 5 would make sense).

The only step left is to run DO to re-balance resources and workloads.  Notice that no time is spent planning how data layouts on disk. Instead, the process involves a high level plan for balancing branches.

Tune in on election day 2008!

How productive are you going to be tomorrow?  I know there is a good chance that many readers will be watching Craig Nunes from 3PAR on a Brighttalk Webcast at 2:45 Pacific time.

The topic is virtualization  - including a focus on 3PAR's fine-grained storage virtualization and thin provisioning.  You will also here Craig talk about why 3PAR's fine grain virtualization is such a good fit for VMware installations.

Perhaps most importantly, it will be a great way to help get your mind off that other pressing issue on November 4th, whatever that is.

Sign up for the webcast here.