StorageRap

Anarchist punches out IBM and TMS benchmarks - & more

The Storage Anarchist has an excellent post on the problems he sees with benchmarking in the storage industry today.  I left a comment there and I assume others will too.

This is likely to be a hot topic.

Traveling, twittering, Zilla-man and memory leaks

Zilla-man capped off his latest trip with a post about how everybody seems to be traveling.

I know Zilla has been hitting the road hard and recently gave a keynote address at SNW Europe.  I truly wish I could have seen it. I'm sure it was entertaining.  The only guy I know who isn't traveling is the well-hobbled SAN Technologist.  I'm sure once his legs heal, he'll be playing catch up.

So, back to Zilla - if you think his blogs are good, you ought to see his tweets.  He is a master.  A lot of people (me included) have difficulty grasping the media.  I think Zilla has it nailed - here's an example: 

Eating wild boar. It's chewy

I don't know if its traveling or the delayed reaction to the lost brain cells of youth, but I find myself having difficulty remembering if I read something in a blog, a tweet, on facebook or in my sleep.   My wife would be amazed that I remember anything at all, considering how often she remarks about my inability to remember conversations that never took place.   I just hope I don't find myself waking up from a catnap and saying the wrong thing at the wrong time because my ability to fence time, space and virtual worlds is getting weaker every week.

What will a 3PAR SSD look like?

Stephen Foskett , who writes the Storage Pack Rat blog, has a very sharp mind who understands storage.  He wrote a comment a couple days ago asking for more details about our SSD development efforts. 

Sorry, I couldn't state them, even if I knew what they were.  Besides I would probably just forget the best details anyway.  However, there are a few simple design principles at 3PAR that should help people understand our future directions for most new technology developments

1) We design systems for fine-grained resource utilization.  While our main claim to fame is being the grandfather of thin provisioning (thanks Barry W), our underlying volume manager works with small "slices of disk capacity".  

2) Automation is another key design goal.  The fewer operations an admin has to do to achieve great performance and scalability, the better.

3) High performance is another design goal.  For disk drives, this means wide striping to use as many spindles as possible on as many controllers as possible. We'd like to be able to use SSDs in a way that boosts overall system performance - not just for certain applications and we certainly don't want them to have a subtractive effect by creating system imbalances or getting in the way of non-SSD I/O processes.

4) Overall reliability and fault tolerance is another key element of our designs, from our clustered controller architecture to our aggressive drive fault/failure detection and replacement mechanisms.  

We think SSDs are going to be very important in our storage systems in the future. In all likelihood there will be several iterations of SSD integration technology spanning multiple years.

SSDs and data movers

The vid today covers a couple topics. First, the Register scooped our plan for integrating SSDs. Stephen Foskett threw the bone my way, asking about a blog last week where I said something about the SSD business being lackluster while the worldwide economy was running like it was doing a parity rebuild.

Second, I wanted to send a shout out to my homey SAN Technologist on this data movement thingy from Compellent that's drawing love from the Sto-B-Sphere. EqualLogic has had automated data moving and tiering for years, but I haven't seen any EQ-peeps defending the turf. ST-man, if you are the anointed blogmeister from D/E, it's up to you now, even though you are probably circling the globe looking for outlets and wireless signals.

FWIW, 3PAR has internal data mover functionality like this too. It's called Dynamic Optimization (DO) and it was designed to redistribute data across disk drives in an InServ array - rebalancing the I/O load after changing the configuration. The secret about DO is that it uses our micro-sparing technology - particularly the software that redistributes data from a failing/failed drive to new micro-RAID groups in the system.

Micro-Sparing leaves no spindles behind

My previous post introduced a concept called I call micro-RAID and touched on wide striping.  Alex McDonald, author of the Missing Shade of Blue blog posed a question in a comment about how micro-RAID 5 differs from tradtional RAID 5 in terms of redundant data protection and sparing overhead.  I thought they were good questions, but the answers are a tad long, so I decided to put them in this post rather than write the long comment from hell.

So, yes Alex, you are correct about dual RAID group failures - as I wrote; although they can’t survive two drives failures from the same micro-RAID array.  In that sense micro-RAID is the same as traditional RAID. The difference comes from combining micro-RAID with wide striping where LUN data is spread over all the drives in the system.  This is fundamentally different than using RAID groups with 5 to 16 drives (or up to 32 drives with two 16 member RAID groups).

Like micro-RAID, sparing is done on a micro (or chunklet) basis. When a drive is identified as failing, or even if it fails without warning (which is highly unusual), the contents of the micro-RAID’s chunklets are remapped to spare chunklets on other drives in the system and form a replacement micro-RAID array. Remapping is similar to other virtual-relocation technologies, such as v-motion.

Wide striping means that there will be multiple LUNs that are effected by a drive failure.  The data in their micro-RAID arrays on will also be remapped to other micro-RAID arrays in the system, but here is the important point – they are remapped to a different set of drives - spreading and balancing the load from the failed drive throughout the system, as opposed to traditional sparing which increases the load on the other drives in the array group where the failure occurred. 

Wide striping and micro-sparing means that rebuilds are completed relatively quickly  because of the massive parallelism involved that resists internal bottlenecks. Other than the replacement drive which has to get “caught up”, wide striping limits the load on the rest of the drives in the system.  This is in contrast to traditional RAID where duty cycles increase significantly across all member drives of the array group.

Depending on one's perspective, micro-sparing either conserves spindles or has no spindle penalty related to spares.  Micro-sparing is designed to protect lost capacity instead of lost devices. For instance, the default overhead reserved for micro-sparing in an InServ system is 2.5% , which equates to the capacity of one spare drive for every forty drives. All spindles in a micro-sparing system are available to deliver IOs, in contrast to traditional RAID sparing where spare drives do not provide any work whatsoever.

Micro-RAID chunklet granularity and wide striping makes a big difference

Storagebod is rapidly becoming one of my favorite storage blogs for its clear thinking, informative content with a bit of an edge.  He recently wrote about the death of RAID-5 and referred to a blog post that appeared on Storagemojo's site last year. The gist is that RAID-5 rebuild times and the risk of encountering additional hard read errors on large SATA disk drives when an array is operating in degraded mode makes it more likely that an unrecoverable read error will occur in an array. 3PAR does things differently and I thought I would use StorageBod’s inspiration to write about it. 

3PAR InServ storage systems subdivide disk drive raw storage capacity into small, granular 256 MB sized units we call “chunklets”.  If the term chunklet doesn’t work for you, it might help you to think of them as data compartments. A 300 GB disk drive would have 1170 data compartments. 

An InServ’s volume manager uses chunklets when forming RAID groups, which I like to think of as micro-RAID arrays. For instance, a micro-RAID 1 array is formed by writing data to two chunklets on different disk drives.  Likewise, a micro-RAID 5 array is made by writing application data to data chunklets and a parity chunklet. All chunklets belonging to a single micro-RAID array are located on different disks that are located on in different FRUs.  

Multiple micro-RAID arrays are combined to form real-world-sized logical disks, which are then exported to host systems as LUNs.  The bottom line is that LUNs are protected by multiple micro-RAID arrays, spread throughout the system.  This means that 3PAR RAID 5 arrays can withstand multiple drive failures, (although they can’t survive two drives failures from the same micro-RAID array).

Sparing in InServ arrays also use chunklets. That means there are no physical spare drives wasting energy and generating heat waiting for a productive drive to fail.  When a disk fails in an InServ system, its used chunklets are evacuated to spare chunklets on other drives.  The wide striping algorithms that spread micro-RAID arrays throughout an InServ system are also used to relocate evacuated chunklets from a failed drive.

Wide striping circumvents the performance bottlenecks and dangerously high duty cycles that are characteristics of typical RAID degraded mode and rebuild operations, including those performed on dual-redundancy RAID such as RAID 6.  Micro-RAID array parity rebuilds complete very quickly with far less stress to individual disk drives.

In summary, the combination of chunklet-based micro-RAID arrays and wide striping creates a significantly more robust and safer environment for RAID 5 than on any other competitive product - without the inherent performance disadvantages of RAID 6.   

Economic downturn to fuel flash SSD buying rampage!

Bad timing happens to even the most successful companies.  Coming out with a new expensive product at a time when budgets are being cut is a case of bad timing.  As any good salesman will tell you - its better to be lucky than good.

EMC, one of the leaders in our industry introduced flash SSDs last January, which put most of the rest of the industry hot on their heels to follow them. 3PAR was an exception.  It may be quite fashionable to have SSD products coming to market that few can afford, but we aren't joining the dance line yet. 

That doesn't mean we aren't working on flash SSDs because we are.  Its pretty cool technology and we are integrating into our product architecture in a way that makes sense for our customers - efficiently, cost effectively and easy to manage.

In the meantime, we will continue to sell highly virtualized, very intelligent arrays that help customers consolidate storage and manage their runaway capacity problems without sacrificing performance. 

Oops my bad, IBM TP benchmark turned out to be bogus

I got all excited a few days ago when IBM announced their latest SPC-1 benchmark using thin provisioning, what IBM calls Space Efficient Virtual Volumes.  In his blog, Barry Whyte from chides 3PAR for not having done a TP benchmark, referring to 3PAR as the "grandfather of thin provisioning".  I didn't mind the ribbing too much because I thought it was cool that somebody had figured out how to do the benchmark and that the numbers looked so good.

Then the letdown - the benchmark didn't actually use thin provisioning functionality.  TP was active, but it didn't have any work to perform, which means it didn't impact the benchmark.  Arrghhhh!!  No great story here, just more weird science with an IBM benchmark.  A realistic TP benchmark should show really great disk utilization, but IBM's only had 54% disk utilization (mirrored).  What's THAT??  Misleading. 

Our last benchmark ran at 83% disk utilization and did not use TP.

Another 3PAR powered data center, Thanks Savvis

Utility computing benefits from efficient storage with powerful management tools.  Savvis is a customer of ours who knows how to do it.

3PAR user Hilton Grand Vacations wins SNW green award

Its always fun when a customer is recognized for being "the best".  3PAR customer Hilton Grand Vacations won the award for ROI and Best Practices in Green Computing and the Data Center at SNW in Dallas this week. 

Here's the net:  Their new datacenter using 3PAR storage is less than half the size it previously was and they are saving beaucoup dollars every month on their electric bill - it's a double win for Hilton and the environment.

Where storage management is concerned, Hilton now provisions storage on demand in a matter of minutes - as opposed to weeks -  which means server deployments are done in one-tenth the time that it used to take. Success is its own reward.

Congratulations to Rich Jackson and the rest of the team at Hilton Grand Vacations and thanks for being 3PAR customers!

Symantec joins the Thin storage ecosystem

Storage resource utilization is high on everybody's list as a major problem to overcome. Its not that storage companies don't understand the problem, but solving it involves overcoming decades of ingrained methods and assumptions of how file systems, volume managers and storage targets work.

Thin provisioning is a technology designed to escape legacy storage design inefficiencies.  It asks the question: "What if you didn't have to reserve large amounts of physical disk capacity for future growth?"  In other words, what if you could allocate storage just in time, instead of keeping stockpiles of unused storage inventory?

Thin provisioning is a technology with legs, not just a knock-off feature.  Most companies have it in some form or another with a range of capabilities. Yesterday, Barry Whyte from IBM blogged about their success in benchmarking a thinly provisioned SVC storage conglomeration.  3PAR is obviously not the only company working on the development of the technology.

Still, the industry has only scratched the surface of it's potential.  This week, Symantec joined the thin storage industry with their announcement of their Thin Friendly Veritas Storage Foundation.  The basic idea is that Symantec's VxFS file system can interact with Thin-enabled storage systems by reclaiming blocks on disk that were previously written to but can now be  returned to free space.

Jeff Boles from the Teneja group said this about the announcement:

"Thin provisioning has quickly become a 'must have' in the fight against low disk utilization, which is cited time and time again as a major pain point for data centers. "By enabling third party products at the file system level to integrate with thin technology, 3PAR has taken yet another big next step in the evolution of thin provisioning, eliminating the chances that file system formats, and consumption and deletion over time can affect your ability to capacity-optimize your storage. Once again, 3PAR demonstrates the power that comes with making thin more than just a bolt-on technology."

Goldilocks and the myth of storage planning

I like to think of thin provisioning as just-in-time resource commitment, where storage resources are available for any purpose at any time.  In contrast to traditional storage which allocates (reserves) full volumes for systems and applications, with thin provisioning storage is only reserved when data is being written to disk. 

IBM blogger Tony Pearson parlayed the Goldilocks analogy in his blog earlier this summer and discusses what fellow IBM blogger Barry Whyte calls the "extent size" of the TP allocation.  Its an interesting discussion and worth a read because the topic is defintely relevant.  But I think it fails to cast the discussion in the most relevant light for storage customers, which is "how many times do you guess wrong when you size a volume and wouldn't you rather not guess at all?"

And that's what the just-in-time element of thin provisioning fixes.  Just make the virtual volume big and apply discipline to filling them up (for instance, don't let your end users run all over them).

Yes, you defintely should plan storage use and purchases, but best-guess mistakes don't have to be part of your plan. Working smarter means making fewer mistakes that have to be corrected and paid for later. 

I recommend checking out thin provisioning with all the vendors, but make sure you check out 3PAR's thin provisioning.  We invented it and are continuing to extend and expand this important technology.

What if we were really superior beings?

From the Sci Fi corner:  We've learned a great deal (or so we tell ourselves) over the years.  Sometimes I find myself thinking about how technology would be developed if today if we were starting with a clean plate. 

So what about data storage?  What would we do differently today if we didn't have to deal with all the relics and assumptions we've been dragging around for years?  How different would things be if we didn't have to bow to the gods of history and compatibility?  

Must be a slow day for news: Hitachi comes late with mid-range wanna-be

Hitachi just released a new AMS SAS disk array.    Hmmm.... That's news?

Didn't EqualLogic do that, when..... 2 years ago

Ah yes, say the unfortunate slow news day editors, but the new Hitachi arrays have load balancing.

Something that 3PAR (and EqualLogic too) have has been doing for their customers for many years already.

And the new Hitachi products will have thin provisioning in the future.

Hmmm...  3PAR is in its 3rd generation of Thin Provisioning technology already and almost everybody else in the industry has some form of it or another.

Don't tell me.... The AMS products will be easier to use than their USP-V products.

Seriously, has there ever been a reason to be excited about Hitachi's execution and commitment to mid range products?

Comings and goings - is flash the next optical?

Two contrasting stories caught my attention this morning:  Compellent's flash announcement and Howard Marks from Information Week discussing how enterprise optical storage is fading to black after so many years.

The juxtaposition forced the question: could flash - as the latest incarnation in a long line of silicon-based mass storage be another exciting storage idea that doesn't make it?  For the record, I think solid state disk (SSD) enterprise storage will succeed.  EMC's Storage Anarchist wrote about the industry's efforts to get flash SSDs over the hump last month - its a good read. 

Whether flash will be the SSD technology of choice over the long haul is another story, but more on that some other time.

First: a comparison of enterprise optical and flash:

• Enterprise optical was primarily used as a replacement for tape and used for backup and archive - it was/is removable media.  Flash SSDs are primary storage.  It can be removable, but is being applied to replace  high-RPM disk drives.
  
• Enterprise optical performance did not compare well against streaming tape performance.  Optical, like flash, is fast to read and relatively slow to write.  The problem is, as a tape replacement, optical was used 99% of the time for writes. Flash has performance advantages for read-intensive applications but no advantage over high-RPM disk drives for write-intensive applications.
  
• Enterprise optical capacities did not compare well against tape. Its not clear what the capacity requirements are for flash SSDs, but they are sufficient to replace high-RPM disk drives (with their associated smaller capacities). 
  
• Enterprise optical was somewhat more expensive than contemporary tape equipment. The cost of flash is typically much higher than similarly-sized high-rpm disk drives.
  
• Enterprise optical was not integrated as a component with existing disk or tape systems  Flash SSDs (today) are being offered as integrated features of existing product lines.
  
• Enterprise optical was anticipated to be more reliable than tape, but it did not establish a track record to prove it.  Flash, with less moving parts would appear to be more reliable than disk, but there are still questions about cell wear out.  Reliability reputations take time and flash SSDs haven't been around long enough to have much of a reputation one way the other. 

My opinion is that the near term issue for flash SSDs is whether it can scale to sufficient capacities to satisfy read-intensive applications such as data mining.  It the near term, it will be relatively expensive, but the cost can probably be justified by some customers.  However, if it is not capable of scaling, that could be a serious problem.

Longer term, the technology has to be more closely coupled with existing storage, including making flash more "invisible" through virtual storage techniques.