3PAR not getting flash? Maybe it's EMC's customers who aren't getting it

A few weeks ago Storagezilla smacked 3PAR with his "3PAR still don't get flash" blog entry. 

Hmm,  seems to me EMC's customers are the ones who are not getting EMC's flash EFD (expensive freaking doohickeys) - or nearly enough of it by the looks of their earnings announcement last week.

The discussion of flash in their earnings call was limited to statements about how customers were interested in technologies like flash.  This differs a lot from saying things like sales of flash drives had exceeded certain levels or expectations or that the adoption of flash had been one of the leading factors of growth in certain markets or product classes.

I'm not saying that EMC is wrong about flash, but their timing was pretty bad. There's nothing like bringing out a new premium-priced technology when the world is on the verge on an economic meltdown. Sometimes you just have bad luck and sometimes you make the wrong assumptions.  Whatever the reasons, Symmetrix revenues, where one would expect to see the largest impact from SSD sales, were down 25%.

Customers want to know how to get satisfactory response time at the best price. If you are talking about transaction and mixed workload performance, wide striping is still a much more cost-efficient technology to achieve response time goals.  Maybe EMC doesn't care about cost effective IOPS anymore, but customers definitely do.

Comments

[Charlie]

I've heard EFDs are selling for about 8X 15K FC drives of comparable size. When used correctly the IOPS increase is (much) greater than that. Therefore, they are cost effective IOPS.

Also they are greener IOPS: more space, cooling, and power efficient when compared to mechanical disks.

Lastly, EFDs have service times of sub 1 ms, you just can't get that with mechanical disks, wide stripe or no.

[marc farley]

The arguments supporting flash SSDs are all good and its a promising storage technology with a big future, but right now they are still an "overkill technology" for most customers. Capacities are still too small which makes it practically impossible to utilize their resources for multiple applications and systems.

[Bill]

To me it doesn't make sense to put drives (of any flavor) into a shared storage array if those drives are going to be dedicated to a single host. Furthermore, correct me if I'm wrong, but most of the EFD implementations available from major OSMs today are limited to a small number of EFDs. Why burden the EFD with the additional overhead of the array (including shared FC ports) at this point? Put them directly in the server if you have a host that needs a huge number of IOPs dedicated.

I think that if the storage vendors could do something more unique with EFDs (other than just making RAID groups out of them), then it starts to get interesting to put them into the array.

The one benefit to using EFDs in in an array vs. "naked" in the server is that the array vendors can add value-add software that might be able to make up for the vagaries of the currently shipping flash systems, such as more complex wear leveling and failure prediction techniques. The chips and operating environments of the array are more powerful than little chip and firmware on the disk controller.

[John F.]

Hi Mark,

Well, in my humble opinion, ok, not so humble opinion then, I think EMC jumped the gun on Flash as Disk and then tried to ram it down the industry’s throat. I also think Microsoft Research has my back on this one in their report last November http://hardware.slashdot.org/article.pl?sid=09/04/08/1733206&from=rss. Sure, SSD prices have dropped from 30X the price of FC disk last summer to 10X price of FC disk today, but that’s still out in the stratosphere on price.

The results are very contrived configurations in EMC whitepapers that try to push up the IOPS and down the space on things like Exchange primary storage in an attempt to justify the high cost of SSD. Solutions like http://www.emc.com/collateral/software/white-papers/h5833-symm-enterprise-flash-ms-exchange-wp.pdf are more akin to Rube Goldberg contraptions; only the joke is on the customer. The added costs of the additional components, complexity, and administration far outweigh any benefit that could be derived from adding Expensive Freaking Doohickeys into the mix.

If you look at the introduction of just about any new technology as it ramps to scale, the majority of the price drop occurs within the first 18 month. After that, it’s the long slog of incremental improvements that finally bring the cost down out of the stratosphere and onto the earthly realm. I don’t mean to rain on anybody’s parade or sound the “Abandon all hope ye who enter” alarm or anything like that. In fact, if you dig into the details of the Microsoft Research report, there is a glimmer of silver lining. Flash as cache showed promise in a subset of the 14 workloads traced. Maybe there just might be a better way to move forward in this economy. Who would have thought?

John F.

[Charlie]

@Marc: EFDs are 400GB in the EMC line, the same as 15K FC disks.

@Bill: You could say the same thing about 15K disks being directly attached instead of in an array. The fact is arrays add value, else there wouldn't be array vendors.

@John F.:For "stratosphere" claims you have to differentiate between $/GB and $/IOPS. For performance sensitive apps, EFDs are a steal.

@John F.:As for MS not liking flash drives, do you think, just maybe, they are trying to re-direct some of the spend their way? They are already doing just that with their DAS for Exchange campaign.

@John F.:Lastly EMC didn't jump the gun on anything. The flash solution works great NOW! There is nothing contrived about it. It offers performance sensitive customers a cost effective IOPS solution. Go ahead and wait for flash as cache if you think waiting is "moving forward".

[marc farley]

Charlie, 400 GB is 400 GB, you are right - and they cost a heck of a lot more, which is why they aren't jumping off shelves. Of course, there is a lot more than just the device cost to think about too - its the services required. The 8x delta given is just for the hardware and doesn't include the manware and maintenance. If you can persuade your management to buy them today - more power to you.

Flash SSDs are promising technology and there will be a lot more continued development with them to make them more usable and affordable in the future. But are you telling me you think EMC actually had good timing bringing these out when they did?

[Storagezilla]

Yes I did smack him, he was dumb in public. I was going to smack another 3Pite last week after reading another item of nose gold but decided against it as it might just have been some swine-avian-madcow flu floating around Cali. A truly general purpose mixed workload array would support wide striping and EFDs because they're not mutually exclusive.

Wide striping is about aggregate IOPS, Flash is about lower response time per drive. Setting this up as a Vs argument is a losing strategy.

As for selling flash drives, well, at introduction flash was 40x a 15K FC disk after one year EMC has driven that down to about 8x on $/GB basis. And for that 8x you get anywhere between 6 to 22 times the response times of 30 HDs.

That price ain't going up, it's coming down and damn fast too. Excuses about waiting for a USB/IDE/whatever interface or trying desperately trying rewrite caching algorithms for a FLASH based PAM (This after telling Techworld DRAM is better understood as they scribbled FLASH into their roadmaps putting that unique talent of NetApp's for talking out of it's mouth and it's arse at the same time on display once again) doesn't hide the fact that most of you were caught napping by EMC.

And by the time 3Par or NetApp ship FLASH in any real sense FAST+FLASH+SATA is going to bringing down acquisition costs for customers down even further.

You're being left behind.

[marc farley]

Zilla-man, Don't try to confuse things by bringing Netapp into the discussion! As far as I'm concerned they're just as crazy as EMC for having rushed out to create an SSD solution - and all just because EMC had one! That's what I'd call lemmings behavior!

The question is can you get enough IOPS from your array without having to supercharge it with the most expensive device in its class and the team of services people that come along with it? EMC apparently felt it was necessary. You guys can call that leadership if you want - most people just call it highway robbery.

So you can stick your tongue out at us as long as you like and make snarky comments about my boss all you like. He doesn't care what you think.

[John F.]

@stogezilla,

Yes, that exactly explains to me why Expensive Freaking Doohickeys are jumping right off the shelves and into customer's deployments. It also makes it clear why Symmetrix sales are down 25%. Yup, you're doing a good job convincing me and everyone else. Please, post a few proof points with your euphemisms next time ... and keep up the good work.

Good luck,

John F.

[John F.]

@Marc,

There's a big difference between seeing a fit between technology and workload like NetApp did for virtualized enviroments using V-series with Ramsan behind it, and what EMC espoused. Netapp at least set the thing up and demo'd at SNW to prove the point http://blogs.netapp.com/virtualstorageguy/2009/04/the-highlight-of-snw.html#more. No tricks, no Rube Goldberg contraptions.

Read the EMC whitepaper. You're supposed to buy a product that has been end of lifed, stub out your email, buy a Centera or even a Clariion running XP embeded, put SATA behind that, then buy EFD, smack it in you Symm, and all is joy. What kind of solution is that? I'd call it pounding square pegs into round holes ...

@all

Er, uh... I apologize for hijacking this thread. I will now return you to your normal brainwashing, er, programming. Yeah, that it, programming.

John F.

[the storage anarchist]

With all due respect (ok, perhaps even more than that), adding flash drives to ANY array doesn't allow the array to deliver more IOPS than it already can - unless the array is being limited by the number of disk drives it can support, that is.

Since Both V-Max and DMX support more drives per array than any other platform we're discussing here in the first place, your assertion that Symm "needs" EFDs to deliver IOPS couldn't be more false if it came out of Tony Pearson's mouth.

Pay attention, now: If a given array tops out at 90,000 IOPS with 500 widely-striped 15K disk drives (180 IOPS per drive), then putting EFDs in that array isn't going to get you any more IOPS - the benefit of Flash is that you will be able to get those same 90,000 IOPS using only FIFTEEN fully-utilized flash drives.

Only the Flash drives will have an average response time that's less than a millisecond, while the 15K HDDs' will struggle to respond in 6 milliseconds (or more).

So...let's see. You can wide-stripe your workload over 500 disk drives, or you can stripe it over FIFTEEN Flash Drives. And the Flash drives cost only 8x what the disk drives cost, yet deliver 6x better response time and 30x more IOPS per device.

Customers get the performance they need without resorting to wide-striping or short-stroking hundreds of HDDs. If you understood this simple math, then you would perhaps understand that Flash is ALL ABOUT saving money, for EVERY application.

But then, you can't win against competition you don't understand - right John F.?

And Zilla didn't drag NetApp into the discussion, John F. came in through the front door. (John's from NetApp, and he likes to compete against Symmetrix circa 1999, since he hasn't kept up with the more recent realities of the modern Symmetrix).

Ignore him :)

For the record, John - put your flash and SATA into the same V-Max, and stand back - the pegs will automatically fall into place. No other complexity required, despite your misinformation to the contrary.

And the fact that it's now been more than 14 months since EMC started shipping Flash Drives and neither 3PAR nor NetApp have figured out how to deliver their value to customers...

PRICELESS!

[marc farley]

Anarchist,

14 months already of chest thumping about this magic and still not more to say about it's financial success? It's not lost on anybody how you've all avoided that discussion.

Your problem is that you think disk devices can only be used for either capacity or performance purposes, but not both. Guess what - they can! And your inability to provide that capability is why EMC needed SSDs in the first place - even though your systems support more drives than any others. If I were you, I'd be embarrassed by all those underutilized disk drives at all your customer sites. Pretty embarrassing. Oh heck - just try to sell 'em expensive freaking doohickeys.

FWIW, we don't short stroke our drives and wide striping is a native, autonomic function on 3PAR arrays, not something that requires professional services to figure out like it does on yours. You can say how all that's changed and try to prove it by publishing obviously faked graphics on your blog, but not even your own customers believe them.

[Bill]

@charlie-

There is a sizable difference between putting a 15Krpm spindle capable of 200-250 random IOPs in an array, and an EFD capable of 10x-50x that number (just a swag, I don't know what the EFDs shipping today are truly capable other than at the datasheet layer).

How many IOPs is a Symmetrix controller capable of? Or a CLARiiON? Or a NetApp, Hitachi, or 3PAR for that matter? Until now, one could keep adding spindles to get IOPs. EFDs are going to show us the limits of the directors/controllers.

I think the secret that nobody is speaking aloud about is how many of these EFD/SSDs can you put into an currently shipping array before the apps using those EFDs consume all the IOPs that the controllers or cache are capable of?

Yes, you get all the goodness of the array software and monitoring. And don't forget to add in the cost of that bare chassis, and controllers/directors, etc. Oh, and on a CLARiiON I have to buy FC drives for the vault even if I was going to build out an EFD-only system. I'm guessing that low end arrays will be able to handle 8 of these EFDs, mid-tier arrays will be able to drive 32 or so, and high end ones might handle 64-128 of these? Will they be able to drive 128 of these EFDs at full IOPs, or will we be paying these vendors for IOPS on the EFD that are bottlenecked behind controllers that weren't designed in anticipation of these drives? Show me some benchmarks that I can trust!

I'm anticipating that next-gen controllers will be designed with more performance with EFD in mind. The current generation of arrays are adding EFD as an afterthough, IMO.

[Bill]

@storage_anarchist-

Your assertions hold water if the IOPs demand is correct for the capacity of the 15 drives in question. Yes, there are apps that scream for 90k IOPs and 1ms latency and only half a terabyte of storage. For every one of those accounts, there are hundreds that need 90K IOPs and 40TB of storage.

A storage architect would determine the drive count for IOPs as well as for capacity. In your race to provide IOPs, you have ignored the capacity question entirely.

No doubt you can make money selling the 90K IOP, 1TB speedster to the few that need it. You are now competing against Fusion-io (takes a lot less floor tiles to put a PCI-e card in a server). Good luck with that model.

I'm still not convinced that Flash is best used in simulating a spinning disk which then is virtualized in RAID groups to simulate another spinning disk. Can't the array vendors do something smarter with flash than buy EFDs from a disk drive company that buys flash from a memory vendor, and then packages it in a container that looks like a SCSI/FC/SATA drive? Let's see some real innovation.

[John F.]

@anarchist

Are you saying if I put flash and SATA in the same V-MAX, then put my mission critical Tier 1 applications on SATA, then "stand back - the pegs will automatically fall into place. No other complexity required".

Perhaps you should read your own Best Practices Tecnical Notes. No, not the ones from 1999, I mean the one from April 2009. You should really clean out your inbox more often.

BTWW: I get my misinformation straight from the source: EMC :)

John F.

[the storage anarchist]

Bill-

In case you missed it, EMC has been shipping 200GB and 400GB Flash drives since early in Q1'09. 15*400GB is more like 6TB of data, not a half a terabyte.

And EMC just announced its new FAST (Fully Automated Storage Tiering) innovation - technology, which will distribute LUNs across multiple different drive types (Flash, FC, SATA), landing blocks of each LUN that are infrequently accessed on inexpensive capacity (SATA), and those that have the biggest impact on performance on Flash. This "hybrid" LUN will then deliver effective performance in excess of that attainable with 30-50x as many 15K spindles at a response time well below that of any disk drive, while reducing the cost of storage significantly through the use of slower, cheaper SATA drives.

Arguably beneficial as "tier 0" today despite 3PARs denial and FUD, EMC is actually explaining how it will leverage Flash technology to deliver higher performance at a lower cost going forward.

And @Mark - sorry, but I'm not permitted to disclose financial information that EMC does not make public. Seems the SEC has rules about equal information dissemination about financials. In fact, I read something in the Wall Street Journal recently about how these rules might apply to bloggers...

[marc farley]

I know, I wouldn't want to talk about a 25% reduction in your product's sales either - thank goodness that was already included in your earnings report last week.

[the storage anarchist]

Geez - and I was just starting to like you, Marc :-)

http://online.wsj.com/article/SB124078135070257099.html

[marc farley]

Yes, I saw the article, but thanks for linking anyway. I love everybody - just not the same amount at the same time.

[John F.]

@anarchist

I'd hate to think about how that disclaimer would fit in 140 characters or less, ouch.

John F