StorageRap

A conversation with Mark Cravotta of Datapipe

I caught up with Mark Cravotta from Datapipe recently at a 3PAR event in Las Vegas.  He's a high energy person who is having a lot of fun growing Datapipe's hosting and cloud computing services as well as helping to manage its expansion around the globe.  

Datapipe is a 3PAR Cloud Agile partner and customer who uses our products throughout their line for primary multi-tenant storage, data snapshots, remote replication and all aspects of disaster recovery.

In addition to being customer-driven, Datapipe is also committed to being a leader in green utility computing by reducing the carbon footprint of it's data centers through power purchases from green power producer, Constellation NewEnergy.  

Copan's MAID fades away: forget about disk for archiving applications

The rumor is that Copan is shuttindg down.  If so, I suspect MAID storage will quickly become an afterthought now, except for a small number of customers and applications that will keep the technology on life support.  The problem with MAID is that there aren't enough applications for selectively-spinning disks.  Selectively spinning disk drives are more expensive than tape for archiving and are more problematic than standard disk systems for backup.  That leaves applications such as video on demand, which is not a large enough market to float a serious startup these days. Thin provisioning for primary storage and dedupe for backup have become the technologies of choice for customers looking to increase the efficiency of storage. 

The outcome of Copan's failure means that tape will continue to be the most prevalent technology for archiving. In turn, that means archiving software will continue to be tied to tape for some time, including all the problems inherent in maintaining metadata for it. If you are in the legal industry and hoping to see leapfrog improvements in digital discovery processes, don't hold your breath.  Discovery from data on tape will continue to be a laborious task.  SSDs for archiving would appear to be a slam-dunk at some distant time in the future, but the cost has to come down a long, long way first.

One of us (storage companies) is not like the others

I thought I should blog about EMC's enterprise SSD (or EFD) announcement this week, considering that it got so little attention from the press. Heck, we probably got more press for announcing our Tickets.com win than EMC got for announcing their new SSDs. 

Maybe it was overshadowed by Cisco's UCS event, maybe the IBM/Sun mashup cast too large a shadow, ot maybe it's just because the whole SSD story is running out of gas, seeing as how nearly every array manufacturer offers them.

Except 3PAR. Yes, 3PAR will have SSDs someday. We are working with them, but are not in any rush to be a me too player. No, we aren't intimidated at all by EMC, the leading company in our industry and a tough competitor, and their large scale efforts to promote SSDs as the next great technology.  SSDs have some terrific characteristics - especially low power consumption - which is something 3PAR works very hard on. In fact, the biggest advantage we see with SSD technology is reduced energy consumption - something we are very proud of with all our products.

But SSD performance doesn't make us flinch. Whether its for Oracle or SQL Server databases, media streaming, large Exchange installations or any mix of applications running concurrently on a VMware machine. 

The reason is our industry leading wide striping. 3PAR InServ customers typically stripe their data across hundreds of disk drives, which means disk drive queues are short and contention problems are minimal.  We have a software product called System Tuner to address situations in the unlikely event disk contention causes problems - and we have hardly sold any of it because customers rarely need it. 

Most people know 3PAR for our industry leading thin provisioning and we have benefited a great deal from the fact that most other array companies offer it.  There is a big difference between technologies that begin life as core architectures - as opposed to being bolt-on afterthoughts.  Unfortunately for us, only a few competitors have developed "me too" wide striping features, which makes it more difficult for us to get the word out on it. 

Performance is always an issue and sometimes old architectures have problems delivering it.  Rather than re-architecting their systems to get wide striping performance, EMC (and now others) added SSDs. There's no question that implementing SSDs was a shorter path to high performance for them, but everything has it's trade offs, such as the initial high cost when technologies are new and extending designs with weaknesses, such as outdated storage provisioning methods.

Next to 3PAR's, the best wide striping implementations belong to XIV and Oracle's Exadata. XIV's implementation lacks high performance drives and the ability to scale to a large number of drives. Exadata does a good job with wide striping, but its a niche product and is pretty much useless for the mixed workload environments that most customers have.

3PAR InServ arrays may be the only viable way to get wide striping performance and scalability. That makes us different than everybody else.  Our next-generation capacity saving Thin Provisioning does too, but the rest of the industry tries to make THAT issue a little less obvious.

Download the mp3 of Cal State East Bay talking about saving energy costs with virtualization

Cal State University East Bay (CSUEB) is a 3PAR customer that is saving a lot of money in power and cooling and admin costs by implementing VMware with 3PAR virtualized storage. Rather than write much about it here, you can read about it on the Wikibon Case Study.  The thing that's most interesting to me about this situation was the fact that CSUEB was running into a power wall and needed to do something relatively quickly.

And if you want to hear it straight from the customer, this link downloads the MP3 file of the Peer Incite interview with Rich Avila from CSUEB and Dave Vellante from Wikibon that was broadcast on Tuesday March 17, 2009

Wikibon peer incite tomorrow discussing actual virtualization benefits

From Wikibon's site:

In late 2007, John Charles, the CIO of California State University, East Bay (CSUEB) and Rich Avila, Director, Server & Network Operations were looking down the barrel of a gun. The total amount of power being used in the data center was 67kVA. The maximum power from the current plant was 75kVA and no more power was available from the local utility. A new data center was planned, but would not be available for two years. CSUEB embarked on an initiative to cut power (and costs) with server and storage virtualization. By the end of 2008, 25 servers had been shut down, 17 NAS devices had been migrated and turned off and 21 new applications that would have required additional servers have been installed. Power consumption had been reduced by 26% and tens of thousands of dollars in electricity costs had been saved.

Attendees of this Peer Incite will:

• Understand how CSUEB mobilized its organization to act to avert a crisis;
• Learn how CSUEB applied best practice in server and storage virtualization with Sun Servers and 3PAR disk arrays respectively;
• Understand key metrics that CSUEB is using to track progress.
• Identify ways to be proactive and avoid a similar crisis.

Pesenting on the call will be: Rich Avila – Director, Server & Network Operations – California State University East Bay (CSUEB)

Moderating will be: David Vellante, Co-founder, The Wikibon Project

Please join us…

---

Date: Tuesday, March 17, 2009

Time: 12:00pm - 1:00pm EST (9:00am - 10:00am PST)

Access#: 218.486.1300

Bridge: 509215

Energy savings and rebates with 3PAR's thin provisioning

Its no secret that server and storage consolidation saves customers money through energy savings.  If you are planning to make changes to your IT infrastructure make sure you include energy costs (duh!) in your plans and check out any rebates that might be available too.  For instance, PG&E in California has excellent financial incentive programs to help IT customers conserve energy - including design assistance.

Efficient storage from 3PAR with industry leading thin provisioning technology can save you a lot in energy costs.

How wide striping eliminates hot spots and provides excellent performance

The Question:  Does wide striping lead to the development of hot spots?

I received a question on a post from last month. In a nutshell, the question posed is: How come multiple applications (database applications in particular) don't create disk contention problems (hot spots) for wide striping arrays?  I'm going to do my best in this post to explain how it works.

To elaborate on the question, hot spots in an array occur when an individual disk drive is overloaded with I/Os and does not have the ability to respond in the expected or desired time. The term originated with database I/O performance analysis, where a relatively small segment of database data (the hot spot) is responsible for a high number of disk reads and writes that overload a single disk drive. In a shared storage array, the occurrences of hot spots can be compounded by having two or more applications generate I/Os localized on a single disk drive, overloading it and degrading the performance of multiple applications.  Given this possibility, why would a customer purchase an array that might exhibit these problems?

The answer:  Wide striping with chunklets randomizes I/Os, minimizes seek times and avoids hot spots

Wide striping is a disk layout approach that uses a large number of disk drives (or spindles) to place data in an array. It's well recognized that putting more disk spindles to work is the way to increase an array's I/O performance. Wide striping in a 3PAR InServ array spreads the data from multiple applications over a large number of disk spindles, which simultaneously provides both high I/O levels and high capacity utilization. The spindle count obviously matters a great deal because each drive provides incremental I/O processing power. FWIW, 3PAR InServ arrays typically have hundreds of drives, but can start with 50 or so drives in a smaller configuration. 

I/Os in a wide striping array are an example of a stochastic process where I/Os are generated with varying degrees of randomness. At any given time, there are a certain number of I/Os pending for the array, which are concurrently being serviced by a large number of disk drives. It also means that the I/Os for most applications, including databases, move from one spindle to another in a random fashion, which is why this technique is referred to as randomizing I/Os.  As the I/Os for an application change spindles, individual disk drives are not overloaded and hot spots are avoided. 3PAR InServ arrays also have cache memory that provides additional performance acceleration and reduces disk contention.

Equally important, but far less obvious is how data is laid out to support a mixed application workload.  Data layouts that exacerbate disk latencies should be avoided, such as long seek times that occur when successive I/Os must jump from one disk partition to another. Storage allocations and data layouts in 3PAR systems use small disk segments called "chunklets".   Chunklets are placed in close physical proximity to each other, which means that seek times for I/Os from multiple applications are minimized.  A human being could try to figure out how to layout data for a wide striping array and never be as good as the algorithms in a wide-striping array controller.

Is there a way to measure or predict the effectiveness of wide striping in eliminating hot spots?

Just as the metric IOPS indicates how much work a disk drive can perform, we could use another metric for the ratio of spindles available to service a given I/O load. This should help us understand the ability of the array, LUN or RAID set  to handle hot spots.  Since hot spots result from disk I/O operations, which take place in approximately one centisecond (cs, hundredth of a second) I created this little hot spot metric: spindles per IOs generated each cs, or spindles/IOP(cs). 

A spindles/IOP(cs) ratio greater than one indicates there are probably sufficient spindles to do the work without developing hot spots; and a ratio less than one indicates that hot spots could develop that would degrade performance. Of course, the interpretation of this ratio should change depending on the specs of the drives under consideration and any other elements that impact the average service time of the drives.  FWIW, the radial interleaving of chunklets is an example of an array technology that would reduce the average service times of drives in a wide striping array.

I thought I'd suggest a couple numerical models to try out my hot spot metric.  Let's say there is a hundred-drive wide striping array with an load of 100 I/Os generated during any centisecond interval. This corresponds to a spindles/IOP(cs) ratio of 1:1.  The statistical odds are good that many of these pending I/Os will be serviced by different spindles - and when there is contention for a spindle, it's unlikely that the same applications will continue to generate subsequent I/Os for the same spindle due to I/O randomization.

In contrast, consider a traditional narrowly striped array that has ten drives servicing a transactional database application. The average number of I/Os generated by this application every centisecond is fifteen, which means the spindles/IOP(cs) ratio is 0.67 (10/15).  If one looks at the aggregate maximum IOPS of the drives, they could come to the conclusion that these ten drives have plenty of performance overhead to handle the workload, however, the spindles/IOP(cs) ratio indicates that hot spots could develop. Certainly some people reading this will be able to relate to the experience of configuring RAID sets that should have adequate IOPS for a given workload - only to find that hot spots have developed, creating infuriating application performance problems. 

So what of it?

The numbers used in the examples above were pulled out of thin air based on estimates that seemed reasonable to me. 

One of the more difficult aspects of trying to use the hot spot metric is figuring out how many I/Os your applications are generating.

I'm interested in hearing what others think of the hot spot metric.  Usual blogging rules apply - which means it's open to the mosh pit. 

3PAR Reports Financial Results for Third Quarter of Fiscal 2009

The bottom line:

Revenue for the third quarter was $48.2 million, an increase of 57% compared to revenue of $30.8 million for the same period a year ago, and an increase of 7% compared to $45.1 million in the prior quarter, which ended September 30, 2008.

David Scott's (our CEO) said:

"The strong financial results we report for the third quarter of fiscal 2009 result from a continued focus on fiscal discipline and reflect the unique value that 3PAR Utility Storage brings to organizations that need to do more with less in challenging economic times.  As datacenters have continued to look for new ways to economize, they have continued to turn to 3PAR to lower CAPEX and OPEX by reducing disk capacity purchases, energy costs, and storage administration effort by up to 80%."

For more details, go the the CNN Money page.

Practical limitations to storage energy standards

 

Geoff Hough,  Sr. Director, Business Strategy

Kudos to the SNIA Green Initiative for continuing to drive awareness and standards around storage energy consumption.  Note last week’s announcement around a taxonomy and power measurement spec  (at idle… but hey, it’s a start and getting a number of opinionated storage folks to agree on a spec is no easy task).

Playing this out to its hoped-for conclusion:  what if all storage devices brought to market were measured for energy consumption using the same set of metrics and would it matter?   The answer is most certainly “yes.”  As part of any storage evaluation process, you would compare the projected consumption of kilowatt hours between alternatives, multiplying any difference by your local utility’s charge per kilowatt hour.

But for those who run their storage devices in co-location environments, the calculus of energy savings can get quite distorted.  Since many co-lo’s charge for energy consumption based on the number of line drops, co-lo occupants lack the economic motivation to select energy efficient storage technology.   How many Californians would select low-flow shower heads if they paid for their water based on the number of pipes connected to their homes?

So, modern storage technologies like MAID, thin provisioning, and SSDs may be attractive for many reasons, but any effective energy savings they provide to their owners is disproportionately small in a co-lo environment.  More to the point, if technologies like these are chosen for their other merits, any residual energy benefit goes largely into the pocket of the co-lo.

One wonders if there are any SNIA-like industry organizations for the co-lo community that contemplate their role in global warming.  If so, we can only speculate if their agenda includes pricing initiatives that give their customers an incentive to implement energy saving storage products.

The green efforts underway in the storage industry will be more effective if we can get alignment on energy efficiency goals across all constituents of the IT industry. 

Hot Aisle interview with Geoff Hough

The Hot Aisle blog is one of the best sources of useful info I know of for data center operators.  Steve O’Donnell, the author of the blog recently interviewed 3PAR's Geoff Hough and talked about architecture and power characteristics of our InServ thin provisioning storage systems.

3PAR's T Class - a whole new level of storage efficiency

We announced our new 3PAR T Class storage systems today,  pushing the envelope on thin provisioning technology by building it into silicon.  Other vendors like EMC, HDS and IBM have tried to figure it out, but 3PAR's has by far the most power depth and breadth. 

Storage energy discussions

There have been a couple blogs this week about energy efficiency and storage.  Chris Evans started the week asking why Conserve IT was necessary.  Yesterday, Chuck Hollis noted a number of ways computer and communications technologies were reducing energy consumption. 

This morning I stumbled across an interesting site - a google map showing solar energy projects in the US.