3PAR Halloween Tool Talk: Persistent Cache

Happy Halloween everybody! 
In this video, Marc Farley in costume as the Storage Anarchist, places I/O data (M&M candies) into controller cache (skeleton cups) showing the distribution of cache data in an InServ mesh active, clustered storage system and how 3PAR's new Persistent Cache re-levels data in cache when a controller in the cluster goes offline.   

Comments

[the storage anarchist]

Happy Halloween, Marc!

Symmetrix takes a slightly more efficient approach - every I/O is mirrored to two different memory boards (DMX) or Directors (V-Max). Should one of these fail, the data is IMMEDIATELY available from another board/director, and the data is immediately re-mirrored across the remaining memories.

As you know, copying things in memory is a mite bit faster than going out to disk.

With the 3PAR persistent cache approach as you describe, isn't there a sudden I/O surge to make room for the data from the failed controller and to re-read the lost data?

Also, out of curiousity, how are the cached host writes recovered - (I don't doubt that they are protected) are they mirrored across controllers?

Trick or Treat!

[marc farley]

It sounds the same. Cached data is mirrored to two different nodes (controllers) in the cluster and so all cached data is immediately available to be re-mirrored from memory. There is a process where each controller frees some memory capacity. I'm not sure what the algorithms are but obviously read cache data wouldn't need to be written back to disk - it could just be freed to accommodate re-mirrored data from the offline controller.

There is some increased activity in the cluster but the re-mirroring itself is very fast and the system is designed for high bandwidth node-to-node communications. And the best part is, there is no network involved, like there is with V-Max.

[marc farley]

Oh yes, I forgot to mention that Persistent Cache works exactly the same for our F Class mid-range systems too, not just our enterprise T Class systems.

To be clear, it requires a 4 node storage system. There needs to be more than two nodes if you are going to mirror data after one of the nodes goes off line (and we don't sell 3 node systems).

[the storage anarchist]

Not sure what you mean by "no network involved"...potatoe, potahto, there's a communications interconnect between your nodes just as there is in V-Max (and DMX, for that matter).

But it seems you are running out of unique features to hawk. How are those Expensive Flash Doohickeys coming?

And while we're on the topic of cache - ever thought of Permacache - locking things in memory? Or how about Dynamic Cache Partitioning? And did I notice you saying that you don't yet have that Big Yellow Button that rebalances the back-end?

Methinks that even as you copy features in Symmetrix, you still have a lot of catching up to do.

[marc farley]

We have a copper backplane, you have an expensive clump of wires and switches - that require power and planning and professional services - and (gosh) cost!

Funny you should mention the big yellow switch idea - I'll put our autonomic chunklet load balancing up against your RAID constrained mashup anyday - and for years to come. So Barry, you like to talk about FAST but its still a feature in the future and I bet everybody is going to be underwhelmed with whatever you guys back into this year just to meet your promises. Good luck dialing back the hype on THAT one.

SSDs are coming along nicely here. Prices are coming down too. Pretty soon, they'll make economic sense.

Thanks for admitting that you are still trying to catch up. But your wrong about actually catching up, it just looks that way to you because you never understand what were doing until a couple years have passed.

[the storage anarchist]

Copper backplanes are so...Y2K. S

ymm gave up the copper backplane to get greater distribution between nodes and allow more I/O power to be corralled in support of 2+ petabytes of usable capacity in a single array. With up to 1TB of truly global memory, that "RapidIO fabric" carries more data, with lower latency, than most dedicated interconnects.

But then, I guess if your architecture really can't scale-out all that much, copper backplanes are a welcome excuse...

Despite your fabricated assertions, FAST hasn't been hyped - we'll deliver exactly what we said we would, when we said we would. FAST v1 in Q4'09, FAST v2 in mid-2010. Sit back and watch how it's done.

And your professional services gag is getting old...today's V-Max customers know how silly you sound, because The Best isn't nearly as complex (or costly) as you'd like the world to believe.

As for Flash - Symm customers have been enjoying the price/performance advantage over wide-striping for almost 2 years...I'm sure you'll eventually figure that out.

So stay tuned, boys and girls. Things are just starting to get interesting.

[marc farley]

FAST is one of the the biggest hype jobs ever, but EMC is known as the hyper hypers of the storage industry. Keep pouring it on, it will only make the reality so much more disappointing later.

Maybe EMC couldn't make a copper backplane work with it's controller architecture, but our scaling is excellent. If you want to bring that up with customers, please do!

And you are so badly in denial about professional services. You sell services to customers to help balance disk activity and then you sell them more services to tune cache resources to make up for those Twisted Sister disk layouts. You think you solved this with V-Max and that's the story you're sticking to, but the situation hasn't changed much.

By the way, whats going to happen with Permacache and cache partitioning when you finally get FAST working right in two or three years? My guess is that you'll replace the professional services revenue from cache tuning with professional services from FAST tuning. And all because you never could get wide striping right. Of course if you ever did figure out how to balance disk activity then FAST wouldn't seem quite as important. I guess if I were you, I'd hype FAST as much as possible too.

[the storage anarchist]

You humor me...

Permacache and Cache Partitioning are complimentary to FAST. Wide striping works today. And customers can choose how much professional services they use (or not).

I know that doesn't fit the myipoc image of the Evil Machine Corp that you like to paint, but there's a different world different out beyond 4209 Technology Drive.

Trust that we discuss scaling all the time with our customers. Not surprisingly, 3PAR is almost never among the competition.

Just not that big (or important) yet.

Indeed, you talk a big game, but your banter is more like what is found on the playground than in a conference room. There are a lot of miles to go before your employer makes it to the big leagues.

And indeed, we are an industry in the midst of a transition. Where EMC once led migrations with services, it today offers customers it's entire suite of migration products AT NO CHARGE. And unlike 3PAR clinging to revenues of a commodity feature, Symmetrix Thin Provisioning is also FREE. Customers can configure and allocate Symm storage without downtime or bin files, and features like cache partitioning need only the barest of hints from storage admins to self-tune and optimize the environment - in real time.

So hammer away, but know that Symmetrix isn't standing still...your target may not be where you thought when you get there.

And with that, I'm off to South Africa for holiday. Been nice Trick or Treatin' with ya, but now it's time for some down time.

TTFn!

[marc farley]

Hmmm, for somebody running off on an international vacation, you spent a lot of time watching and responding to this blog. Say what you will, you are obviously extremely concerned about 3PAR and it's products.

By the way, when we talk with customers, EMC always comes up. Over and over and over again.

[nate]

I'm waiting for the day when EMC drops Clariion myself and replaces it with the V-Max architecture. What's the point of such a fancy product if almost nobody can afford it? I'm sure it's great for those multi billion dollar companies out there. Those same companies who haven't created a net new job in the economy in more than 30 years according to the Census at least(just learned that yesterday).

Myself I wouldn't want to work for any company that had more than ~300 people.

Myself I like 3PAR because it makes some awesome technology actually affordable to small/medium businesses whom are responsible for more than 95% of all U.S. exports, and 90% of all technical innovations/patents (according to the Census again)

On the same token I like Extreme networks for my switching gear, similar in that they make some awesome technology at affordable rates and really easy to manage. Cisco makes me want to gouge my eyes out.

I suppose I'm in the minority of users who do Storage/networking/servers/automation/monitoring/virtualization/etc. Most big companies have a dedicated team for each, most smaller comapnies often have dedicated people for some/most of those.

I pick products that can make my life easier, thus saving money since we don't need big teams or professional services to do the work.