Nominating VMmark/IO for Mixed Workload Benchmark

A lot of people either enjoy or are frustrated by the great benchmark soap opera .  I'm not sure if it really matters all that much to customers, because if it did, they would require more from the vendor community.  Of course part of the problem with benchmarking is that it's much more alchemy than science and is loaded with difficult, non-intuitive concepts such as - "What does a 100% workload represent anyway?"  and "how come response times at various workloads change non-linearly when the number of IOPs at those workloads are linear?"  and "why did the SPC publish benchmark results for one vendor's products that were submitted by a competitor?"  

A couple weeks ago the Storage Anarchist wrote about the lack of a benchmarks for mixed workloads in enterprise environments.  Anarchist thinks that working with his co-worker, the mysterious Dr. Kartik would make things better and has complained in his blog about the lack of response from other companies in joining the effort.  Maybe EMC is looking for subsidies to help pay Dr. Kartik's salary because  Dr. Kartik seems to take interest in this a couple times a year - when I assume the work gets slow. 

Recently IBM worked with ESG to publish a mixed workload benchmark based on a modified version of VMware's VMmark benchmark for IBM's 5300 system.  I say "modified" because ESG needed to increase the I/O component in VMmark to make it work for measuring storage workloads.   Anarchist criticized this work (there is a standing tradition for storage bloggers to slam competitive benchmarks) and seemed eager to shut the door on the methodology that was used, arguing that it could not be made repeatable due to the way the Jetstress component was implemented.  I won't further a guess about Anarchist's motivation in wanting to dump on this new methodology, but it seemed a bit premature (even for a storage blogger) and gave me the impetus to look into it further. 

What I found was something that was way better than I expected.  the approach used for this benchmark has a lot more meat on the bone and promise than anything else I've seen discussed anywhere (and I mean ANYWHERE).  Not that there is a vehicle for doing it, but I'm nominating this modified VMmark approach as the method for benchmarking mixed workload environments. Seconds or dissenting opinions are encouraged to comment.

To Anarchist - clever engineers live for these sorts of challenges and this one hardly seems insurmountable or as expensive as some science projects are. Brian Garrett from ESG labs, who many of us know by reputation, if not personally, is a very smart, very creative and credible person who has built a long career in performance alchemy.  I believe Brian would steer this development more objectively than your buddy, Dr K.

Comments

[Tony Asaro]

I've worked with Brian for years and he knows storage performance inside and out. No benchmark tool is perfect - or methodology. The idea is to get an approximate idea / range. I put my trust in my buddy Brian.

[marc farley]

Thanks Tony, I think Brian's involvement in this would be a real plus for the industry and customers.

[Chris Fricke]

As a customer deeply embracing the world of mixed workloads in a virtual environment, this is exactly the kind of benchmark that might make some sense to me - if it was standardized and had broad vendor participation. Don't get me wrong - customers don't (or shouldn't) just buy IOPS - BUT it is a heck of a lot easier to choose a fast car when all the brands use MPH for comparison.