One of us (storage companies) is not like the others

I thought I should blog about EMC's enterprise SSD (or EFD) announcement this week, considering that it got so little attention from the press. Heck, we probably got more press for announcing our Tickets.com win than EMC got for announcing their new SSDs. 

Maybe it was overshadowed by Cisco's UCS event, maybe the IBM/Sun mashup cast too large a shadow, ot maybe it's just because the whole SSD story is running out of gas, seeing as how nearly every array manufacturer offers them.

Except 3PAR. Yes, 3PAR will have SSDs someday. We are working with them, but are not in any rush to be a me too player. No, we aren't intimidated at all by EMC, the leading company in our industry and a tough competitor, and their large scale efforts to promote SSDs as the next great technology.  SSDs have some terrific characteristics - especially low power consumption - which is something 3PAR works very hard on. In fact, the biggest advantage we see with SSD technology is reduced energy consumption - something we are very proud of with all our products.

But SSD performance doesn't make us flinch. Whether its for Oracle or SQL Server databases, media streaming, large Exchange installations or any mix of applications running concurrently on a VMware machine. 

The reason is our industry leading wide striping. 3PAR InServ customers typically stripe their data across hundreds of disk drives, which means disk drive queues are short and contention problems are minimal.  We have a software product called System Tuner to address situations in the unlikely event disk contention causes problems - and we have hardly sold any of it because customers rarely need it. 

Most people know 3PAR for our industry leading thin provisioning and we have benefited a great deal from the fact that most other array companies offer it.  There is a big difference between technologies that begin life as core architectures - as opposed to being bolt-on afterthoughts.  Unfortunately for us, only a few competitors have developed "me too" wide striping features, which makes it more difficult for us to get the word out on it. 

Performance is always an issue and sometimes old architectures have problems delivering it.  Rather than re-architecting their systems to get wide striping performance, EMC (and now others) added SSDs. There's no question that implementing SSDs was a shorter path to high performance for them, but everything has it's trade offs, such as the initial high cost when technologies are new and extending designs with weaknesses, such as outdated storage provisioning methods.

Next to 3PAR's, the best wide striping implementations belong to XIV and Oracle's Exadata. XIV's implementation lacks high performance drives and the ability to scale to a large number of drives. Exadata does a good job with wide striping, but its a niche product and is pretty much useless for the mixed workload environments that most customers have.

3PAR InServ arrays may be the only viable way to get wide striping performance and scalability. That makes us different than everybody else.  Our next-generation capacity saving Thin Provisioning does too, but the rest of the industry tries to make THAT issue a little less obvious.

Comments

[nate]

I have been wondering about SSD performance myself at least as far as IOPS go. It seems the Intel SSD is rated for up to 35k IOPS on 4k reads, and the faster fusion-io is upwards of 100k IOPS, even with today's fastest enterprise arrays, it wouldn't take many of those to max out the controllers/nodes right?

Assuming say double the density of disks with 2.5" vs 3.5", in theory you could probably max out a decked out 8-node T800 with 80xSSD disks, in 1 4U enclosure, which could produce up to 2,800,000 IOPS. And with typical 4Gb FC HBAs being able to handle roughly(on paper) 250k IOPS thats 12 FC ports you'd need to that enclosure. Compared to roughly 320,000 IOPS by 1,280 15k RPM disks(assuming 250 IOPS per disk).

What are controllers/nodes going to look like in the future to be able to drive this level of IOPS to support the order of magnitude increase in performance on the disks?

[Jason Jensen]

What do you think about the "Mega LUNs" in the HDS AMS 2000 series and their implementation of wide striping?

[marc farley]

Nate, great question. It depends to some degree on the cost of the devices and what people want to buy. If price and performance of SSDs and spinning disks ever become comparable then the bottleneck definitely moves into the controller and new designs will be needed.

Jason,

Wide striping from HDS and EMC both involve lots of manual work planning resource allocations. Look at the Tested Storage Configuration Creation section (Appendix C)of the SPC-1 data from our latest benchmarks. The 3PAR setup is covered in 5 pages with a small number of commands, whereas the Hitachi setup for wide striping takes 31 pages. If you actually look at what it takes on the Hitachi side, it makes your head spin.

http://www.storageperformance.org/results/a00069_3PAR_InServ-T800_full-disclosure.pdf

http://www.storageperformance.org/results/a00054_HDS_USP-V_full-disclosure.pdf

[nate]

I finished a fairly deep competition between the HDS 23/2500 and the 3PAR T400, and without pouncing on HDS too much I can say HDS has made some significant progress but they still have a ways to go.

- They can "wide stripe" but only within a single LUN/RAID group. They have no ability to stripe RAID groups(unless what their SE told me was not correct, we never did get to the eval stage). So for example there is no way to stripe across every disk on the system if you have say 120 disks. At least not directly, if your accessing the storage through NAS and the NAS head unit(s) can stripe the LUNs for you you get some benefit there, not as much as native striping though. If I recall right they can have RAID groups of up to 60TB, they even support RAID 6 (28+2) I believe, which I think is just nuts.

- They can convert between RAID levels online, but the destination disks must be blank. vs 3PAR you just need enough physical capacity to convert the volumes, you don't need new blank disks.

USP V and HDS 2xxx series are pretty different as far as provisioning goes so I don't think you can extrapolate too much with what is involved with USP and tack it onto the AMS2xxx. AMS2xxx has made a lot of strides, though it's no T-Class(I'd rather have my old E200 instead of the AMS though).

Going beyond wide striping, just building the system so it can protect itself against a shelf dieing seemed like a very tedious process with HDS(they could do it as part of the installation, thinking how to manage that with growth and stuff going forward made my brain hurt), with 3PAR it's automagic.

Course there was several other deficiencies in the AMS2[3|5]00 at least compared to the T400 we have now. The solution HDS was proposing us was a 180-disk system that had 47 of it's disks dedicated to either RAID parity or hot spares, that's a significant chunk of spindles in my mind.

Can't speak for EMC stuff, haven't had any exposure to their gear since the CX[6|7]00.

[nate]

Also another thing about HDS, I don't know if it got through their heads but as of last year they were convinced the 3PAR architecture was what they called "Asymmetric Active/Active" which is "Customers must recognize that with an 8-Node 3PAR T800, they are really managing, tuning, and adding capacity to 4x independent 2-controller modular arrays that are loosely coupled over a passive backplane."

Clearly misinformation and I tried to clear it up with them, hopefully they have it right now.

[marc farley]

Thanks Nate for the thoughtful, information-packed comments.

[Jason Jensen]

Thanks Marc and Nate. That sounds about on par with my experiences with HDS as well. With just about every piece of their software we purchased to do some task (in-system replication, out-of-system replication, performance monitoring, etc) the software was always cumbersome. the processes involved were difficult and often times did not live up to the required functionality. Their hardware has always been rock solid, I just can't say anything good about their software.

I just talked to an SE from HDS yesterday, and he claimed that their implementation worked something like this:

1) It uses a 32MB Stripe Size (he even used the word chunk in there, I was just waiting for him to slip up and say chunklet) with a minimum volume size of 32MB and Maximum of 60TB.
2) Create multiple raid groups and add them to a pool of some sort
3) Each volume would be striped across all of the raid groups in the pool

In my opinion this is better than no wide-striping, but the 3PAR wide striping is just so much better in implementation, in ease-of-use and in execution.

The other major problem I saw with the AMS 2000 series is that thin provisioning isn't currently available nor is the ability to move volumes or other features ala Dynamic Optimization (http://www.3par.com/products/software/dynamic_optimization.html) or HDS HDP (http://www.hds.com/products/storage-software/hitachi-dynamic-provisioning.html). My understanding is that HDP and thin provisioning will be available on the AMS 2000 later this year though.

Just seems like they are playing catch up to things that 3PAR already has and does very well.

Anyway, thanks again.

[Hector]

Couple of things on HDS/SSD/Wide Striping:

1. No HDS is not playing catch up with 3 PAR, thin provisioning isnt the only technology we have, how about storage virutalization and provisioning internal and external storage with wide striping, just because someone comes out with technology it doesnt mean everyone else is playing catch up, when is 3par doing anything else besides ThinProvisioning? How about Cache Partitioning, I dont see many Service Providers using 3 PAR, how can you have multiple companies on one Array and have security like HDS does.
2. SSD-I agree with input on SSD, EMC, the marketing genius they are stated these drives were 30x the performance of spining drives, ofcourse they forget to mention its for random reads, not writes and MTBF rates are high, power consumption is not the only reason for SSD. We will make SSD drives next year with Intel with quality and will perform better then STECH. When is 3PAR making Flash Drives?
3. Wide Striping- Its actually 42 MB writes to a pool of storage. I think we are already getting bashed on the zero reclaim from this site, but that is something we are giving to customers to save storage. You mentioned zero reclaim as being a great technology in your video, not sure why you are bashing our thin provisioning when most of our customers are fortune 100 customers and rely on 100% uptime. You also state 3PAR will be coming out with this technology later, does that mean 3PAR is playing cath up with HDS? Our Oracle financial companies are having great success with our Wide Striping Technology today.
4. SPC benchmarks, AMS 2000 outperformed 3PAR without wide striping, whats going to happen when we have wide striping on AMS 2000 couple with our Enterprise load balanced arrays? Aug 3 is the proposed GA date.

I think we all agree Wide Striping is awesome. Some storage companies do things different, however it doesnt mean they are better. I think its easy for 3PAR to improve and focus on Wide Striping since its your only technology and easy to bash a big company like Hitachi who has over 900 companies. We do have other leading technologies in storage that have made a leader.

Peace

hector

[marc farley]

Hector, In response to the points you raise

1. Service providers are 3PAR's largest market segment. One of the reasons is our virtual domains feature - for multi-tenancy security. You just gave me a great idea for a new video!

2. Are you trying to say you are doing something special with flash? We are also working with flash drives, but I'm not going to announce their availability on this blog.

3. Hitachi's storage pools are yet another item that needs to be managed. Too bad for your customers that array management is so complicated for them. My bashing of Hitachi's bloated provisioning (on another post) discusses how zero reclaim is less of a great new feature and more of a tool for dealing with your bloated provisioning. FWIW 3PAR has been shipping reclaim technology in hardware since late last year.

4. What benchmark did your ASM mid-range arrays outperform our F class mid -range arrays? And tell me again you didn't use wide striping or short stroking. Here are the facts right from the SPC's published results for your ASM 2500 and out F400:

IOPS
Hitachi ASM 2500 ... 89,492
3PAR F400 ........... 93,050
- - - - - - - - - - - -
3PAR wins by about 4.5%

Now let's look at ASU capacity - which shows how much of the physical storage were used - an indication of short stroking. Both systems were mirrored so the max potential ASU capacity would be 50%.

Hitachi ASM 2500 .... 30.95%
3PAR F400 ............ 47.97%

In other words 3PAR used over 50% more of the available disk space than Hitachi did - which means we were running at 50% greater disk utilization than Hitachi. That's an improvement over Hitachi's normal short-stroking techniques, but its still pretty far short of 3PAR's results.

As for wide striping, the minutiae from the SPC results indicate that there was an obvious attempt to spread the data widely over the drives. You might not call that wide striping, but most everybody else does.

When Hitachi delivers enterprise load balanced arrays we are going to yawn - because 3PAR arrays have been load balancing over multiple controllers since we first shipped product. Is that supposed to be another example of how we are following Hitachi?