StorageRap

Remote replication the easy way with 3PAR's new InServ Management Console 4.1

3PAR designs its systems to provide huge time savings for storage administrators. Below is a video of our new InForm Management Console (IMC) 4.1, announced today, showing how incredibly easy it is to configure and operate 3PAR's Remote Copy application.  

Things that the demo didn't show that are advantages of 3PAR's single software architecture are:

• A single console can manage both Mid-Range and Enterprise arrays
• A single console can manage both local and remote systems
• A single console can manage all array software elements
• Customers can mix and match systems for replication- for instance they can use enterprise T-Class storage at their primary site and mid-range F-Class arrays at the secondary site. That arrays used primarily to comply with regulations can be much less expensive.
• Replicated volumes have to be the same size (capacity) as the primary volume they are protecting, but they can be any class of service (disk type combined with RAID type). This means replicated capacity can be optimized for capacity efficiency while the primary storage can be optimized for performance. Again, this results in operating savings compared to competitive arrays that require replicated volumes to be the same configuration.

Here is a brief description of all the software functions available through IMC 4.1.  As you can see, it's a pretty comprehensive list of features:

• System Manager: used for viewing performance and utilization stats and configuring hardware elements.
• Host Manager: used for configuring hosts that access 3PAR arrays - including Autonomic Groups, which are used to configure groups of servers to storage simultaneously and Virtual Domains, which restrict access between servers and storage resources according to group membership.
• Provisioning Manager: used for provisioning both thin and thick volumes as well as setting up Virtual Copy (snapshot) volumes for them. Changing the class of service for volumes by re-striping them over additional drives or different types of drives and/or RAID levels through Dynamic Optimization is also done through the Provisioning Manager.
• Event Manager: used to display logs,events and alerts.
• Hardware Inventory Manager: does what it sounds like - reports on what is in your array.
• Layout Grid Manager: displays the logical layout of data in chunklets and logical disks that comprise the virtual volumes in 3PAR arrays.
• Performance Manager: displays performance stats for array components such as ports, disks, and virtual volumes.
• Remote Copy Manager: the demo above shows what this does - configuring remote replication and managing the process of failover and failback.
• Security Manager: used to configure array users and control access, including Virtual Domains for making virtual private arrays.
• Task Manager: used to manage tasks that are running on 3PAR arrays, according to administrator privileges.

EMC, the Misanthropic Media Company?

It's getting very hard to keep up with all the crazy social media stunts coming out of Hopkington, but they seem to have done it to themselves again.  First was the questionable spamming for viewers so they could claim they had a viral video, then today they just "leaked" a 3PAR sales "kill sheet" - and also apparently established a "secret" site with the URL Notapp.com, where they compared their own guarantee program to Netapp's.  According to Simon Sharwood at Search Storage Australia, the site was removed and accessing the URL directed browsers to EMC's site.

Perhaps it is all part of a new marketing strategy by newcomer Jeremy Burton, who joined EMC as Chief Marketing Officer back in March.  As best I can tell, Burton's new marketing strategy for the company is that people will believe anything.  Maybe he doesn't think there are enough new products coming out of EMC - or that the delays in getting their ballyhooed FAST out the door are too embarrassing - but instead of trying to promote EMC on its own merits, it looks like he is doing his utmost to mud wrestle.  Is that what EMC is paying him the big bucks for? 

EMC suddenly is taking a bigger interest in 3PAR. That's good. Search Storage Australia just published parts of a competitive document that EMC was circulating to it's partners about 3PAR.  It certainly wasn't a surprise because we'd seen it previously, but I was sorry to see it published because it made EMC look ridiculous, which was working pretty well for us. But now that it's been outed, here is what we have to say about it (in the guise of Ineption's lead character, the CRO)

The messaging is not built in, but our zero detection technology for optimizing capacity is. The host SW commands to do this are short and do not require "careful coordination".  Veritas, Oracle, Windows Server and Linux software all work with minimal operator effort. For instance, this document from Oracle, describes the whole process, with the sole operator command being this:  #bash ASRU LDATA.

Can EMC provide online reclamation of zeroed space without risking capacity overruns and with tolerable performance? 3PAR can. Does EMC have these capabilities in both mid-range and enterprise storage arrays? 3PAR does.

3PAR has both Flash and 1 TB SATA drives.  We also have Adaptive Optimization software that uses Flash SSDs for storage tiering. EMC still doesn't have it after they made such a big deal about it last year. They like to tell customers that their size gives them development advantages, but their track record doesn't support their claim.

3PAR arrays allow users to create many tiers, but without the need for disk pools. Tiers are constructed from the combination of drive type plus RAID level. For instance, you can have separate tiers for SATA, FC and Flash SSD drives with the RAID level you select. Our Dynamic Optimization software allows admins to move data from one tier to another. You can "dial in"  the performance and protection you want.

All systems have a peak output , ours just happens to have a lot more throughput than theirs - and at higher disk utilizations.  We have published benchmarks that show how our systems perform. They don't. Adding disk drives to a system and utilizing those drives is far easier with a 3PAR system than either VMAX or Clariion where you have to wrestle with putting drives in the pools you want to use them for.

There are no disk pools in 3PAR storage. Pools trap resources so you can't use them. Work isolation in pools leads to hot spots and storage admin nightmares.  Wide striping does not mean you can't have tiers. That is an idiotic statement.

VMAX can configure large pools - and all the drives in them have to be at the same RAID level meaning you can't create multiple tiers within those pools. If you want multiple tiers, you need multiple pools and all the headaches that involves. Change management in an environment with multiple pools is complicated. You also need to consider the pools needed for snapshots and remote replication. Are those easy to provision and change on EMC storage. Most would say "no".

3PAR uses all disk spindles all the time for delivering IOPS and pro-active sparing is done using reserved space on those drives.  Rebuilds do process quickly. Would EMC have you believe they never have to perform drive rebuilds? Really?

The RAID6 thing really makes me sad. They look so stupid when they say it. We're all sorry to say goodbye to that piece of FUD.  

Our front end archiecture was designed for large-scale parallel connectivity to match the massive bandwidth capabilities of our wide striped back end. Our benchmarks and the cost per IOPs in those benchmarks speak for themselves. Our customers also tend to run 3PAR systems at much higher disk utilizations than they run other vendor's arrays. 

We support a huge number of ports on our systems w/full active/active data access across all controllers. All controller nodes can be used to access all data volumes.  We have a number of customers that run fairly sizable SANS without switches because they have enough ports on their arrays so they don't need to consolidate access through switches.

5- 9s? We're there. Our systems get pounded on every day in some of the largest private and public data centers in the world. They are designed with complete redundancy in all components and have advanced capabilities such as Persistent Cache to maintain high levels of performance even after the loss of a controller.

The delays in bringing their FAST tiering software - a product they were hyping in April of 2009 -  to market have shown that size doesn't matter much when it comes to delivering technology on time.  I'm not saying 3PAR always delivers on time, but EMC is far from immune to these problems. In fact, the need for them to coordinate across multiple product lines creates certain disadvantages for them.

As to their comments on our support; they are pure FUD and grasping for straws.  We would not be able to maintain the customers we have if it were not for our efforts at supporting them.

  * * * * * *

The following content was added on July 30th by Rusty Walther, 3PAR's Vice President of Customer Services & Support. 

Stating that 3PAR “outsources support” is just plain silly, especially coming from a company that keeps most of the worlds’ largest offshore outsourcing companies in business.  Like EMC, 3PAR uses Third Party Maintenance suppliers (TPM’s) for break-fix field activities.  In some geographies, EMC and 3PAR even use the “same” TPM.  But EMC also outsources most of their volume call center and Level-1 Technical Support to offshore suppliers.  Not so at 3PAR.  Everyone that touches a 3PAR support case is a 3PAR-badged employee.  I challenge EMC to identify a single outsourcing company that handles 3PAR technical support.  EMC’s outsourced technical support sub-contractors could be listed alphabetically, by geography, or by technology category … but you’d need a couple of sheets of paper to do it.

3PAR CEO David Scott on Storage Federation

Dave Vellante from Wikibon posted his summary of 3PAR's analyst call this week.  It's a good chance for people to get an unvarnished opinion how 3PAR is doing.

One of the things that came up during the call was a discussion of Storage Federation. Here is what our CEO David Scott said were the three main points to understand about it:

1. It’s different from virtualization. It allows separate peer, self-governing systems to act as a global whole (versus a hierarchical approach).
2. Automomic storage will be a critical enabler to federation as stuff gets moved around the virtualized network.
3. Storage federation is following a natural evolution from data center–>metro–>globally shared storage

Storage Federation is a new concept that has been getting a lot of attention from storage bloggers all of a sudden. People interested in following these discussions can start on Stuiesav's blog and follow the links in his posting.  

A team of storage systems

A good analogy for Storage Federation is that storage systems work together in a team. Each member of the team has the ability to manage it's own resources and can make the information about its resources available to the team so that they can act together according to internal algorithms and policies.

Autonomic management is scoped from the individual storage system to the Storage Federation. The same types of algorithms and policies that relocate volumes within an individual array are used to relocate volumes within the Federation. For instance, the autonomic management technology used in 3PAR's volume and sub-volume relocation software, Dynamic Optimization and Adaptive Optimization, could be adapted for use in 3PAR's Storage Federation.

The graphic below illustrates the two levels of autonomic storage management. On the bottom, three storage systems manage their own resources and share resource information to the Federation where they can be accessed and managed by Federal algorithms and policies. If you click the graphic, you will see a slightly larger version that is easier to see.

 

Some of the thorniest storage ownership problems can be addressed by Storage Federation, including load balancing, firmware upgrades, system maintenance and data migrations. 

Zeroing in on a definition for federated storage

(Federation Square, Melbourne)

There have been some excellent discussions recently in the storage blogosphere and on Twitter about the concept of Storage Federation among a number of storage people; known by their Twitter IDs as @stuiesav, @storageanarchy, @rootwyrm, @davegraham, @bwhyte, @ianhf, @esignoretti, me (@3parfarley) and others - as the interest continues to increase. 

There are two aspects of the discussion that I think are fascinating: first is the role of social media as the means to include customers, vendors and others in an open discussion that typically is conducted privately by a vendor preparing to release a new product or feature, second is the challenge of defining a storage capability with sufficient focus and vendor independence so that is is meaningful. There has been some amount of skepticism about this effort, suggesting that we are predetermined to end up with ambiguous terms that can be interpreted (spun) by anybody (any vendor) to mean anything (our product does it). I'm hopeful the results can be better than that, but subjectivity is the only constant of social media and it's not likely that everyone will like whatever happens.  

In general, the language of digital storage has many overlapping meanings, which causes a fair amount of confusion in our industry. I've been dealing with this problem for many years, going back to when I wrote Building Storage Networks and had the challenge of trying to invent generic terms for functions that had been coined by vendors and tied to specific products. My interest in defining Storage Federation goes beyond my role as an employee for 3PAR.

The notion of federated storage has been around for several years, but recently it came to light when Pat Gelsinger of EMC referred to it during a press briefing on March 11. EMC blogger Chuck Hollis wrote about it afterward and there was some chatter, culminating in a blog post by @Stuiesav on April 2, which proposed that the discussion about Storage Federation was just marketing hype attempting to rebrand storage virtualization.  EMC's Barry Burke (@Storageanarchy) and I (@3parfarley) both agreed that this was not the case this time around and then in the last couple days the discussion fired up again. The problem with Twitter is the limitations of 140 characters per tweet.  It's surprising what can be done with so few characters, but it does have limitations.

Skeptics Allowed

This weekend, @ianhf  posted in his Grumpy Storage blog echoed @Stuiesav's skepticism and expressed his perspective (as a customer) as to the things he would like to see when new technologies - in this case Storage Federation - are introduced.  Here are some of  the items from his list (Some of them don't fit our definition exercise because they assume there is a new product being introduced, which is not the case here).

• The specific customer requirements & problems this addresses & justify how
• The use cases this feature / function applies to, and those that it doesn't
• Why & how this feature is different to that own vendor's previous method for solving this problem
• Provide clarity over the non-functional impacts of the feature before, during & after it's use - ie impact on resilience, impact on performance, concurrency of usage etc (including provide up-front details of constraints)
• Naturally you'll also expect me to require TCO & ROI of the feature, and any changes to the models as a result of this feature

   

So lets get started (already!):

The definition of Storage Federation that was kicked around on Twitter is something like:  "the transparent, dynamic and non-disruptive distribution of storage resources across self-governing, discrete, peer storage systems."  (And yes, I did elaborate a bit on this while I was writing based on bits and pieces from comments I read and further thought of my own.)

The idea is to have multiple storage systems cooperating as a team (as opposed to under the direction of an external entity) to place data in the aggregated storage resources (LUNs or volumes) of all participating members. An example of Storage Federation is how Dell/EqualLogic arrays distribute their volumes over multiple systems. When a new EqualLogic array is added to an iSCSI SAN, the administrator is asked if the array should be placed in the same group as other arrays in the SAN. If this is done, the arrays start splitting their volumes (and workloads) across both arrays.

Five examples of storage federation capabilities are:

Storage expansion: You want to increase the storage capacity of an existing storage system that cannot accommodate the total amount of capacity desired. Storage Federation allows you to add additional storage capacity by adding a whole new system.  

Storage migration:  You want to migrate from an aging storage system to a new one.  Storage Federation allows the joining of the two systems and the evacuation from storage resources on the first onto the second and then the first system is removed.

Safe system upgrades:  System upgrades can be problematic for a number of reasons. Storage Federation allows a system to be removed from the federation and be re-inserted again after the successful completion of the upgrade.

Load balancing:  Similar to storage expansion, but on the performance axis, you might want to add additional storage systems to a Storage Federation in order to spread the workload across multiple systems.

Storage tiering:  In a similar light, storage systems in a Storage Federation could have different capacity/performance ratios that you could use for tiering data.  This is similar to the idea of dynamically re-striping data across the disk drives within a single storage system, such as with 3PAR's Dynamic Optimization software, but extends the concept to cross storage system boundaries.

These are all examples of Storage Federation that Dell/EqualLogic storage systems are capable of today. It's not intended to be an endorsement of their distributed volume manager, I am simply using it as an example of Storage Federation. Saying that, if you look for "Storage Federation" on the Dell/EqualLogic web site, you probably won't find it today because they don't describe it that way, but that doesn't mean that it is not in the product.

That's not it

People also want help understanding what the definition of Storage Federation should not include. Here are my thoughts:

In-band storage virtualization systems like IBM's SVC can provide all the capabilities listed above for Storage Federation. However, in-band virtualization products govern the behavior of the other storage systems networked to it and Storage Federation involves self-governing, peer systems. Another way of saying it is that Storage Federation does not have functionality in the network between host systems and storage.

File, object and data distribution technologies like EMC's Atmos certainly provide a type of federation insofar as multiple computer systems can access the same data objects from multiple locations that may be separated geographically. However, this capability primarily migrates files (data objects) - and is functionally orthogonal to Storage Federation, which works on volumes. A term like Data Federation is probably more precise than Storage Federation for this sort of capability. 

Clustered storage systems, like 3PAR's InServ storage systems or clustered NAS systems are not examples of Storage Federation because they function as single, scalable storage systems, not as discrete storage systems that are networked together.  Clustered storage systems can work together in Storage Federations with other clustered or non-clustered storage systems. In that case you could have a Storage Federation of clustered storage systems.

How is this different?

I thought I'd make a little effort to respond to @ianhf's requests.

The main problem Storage Federation addresses the limitations presented by a single storage system, such as capacity, performance and maintenance availability. 

Currently, when a new storage system is added to an existing environment, there are server administration tasks that need to be done in order to migrate volumes from an existing storage system to the new one. In addition, there is usually some amount of downtime and/or performance degradation associated with the migration and configuration of new data paths.  By contrast, with Storage Federation, a new storage system could be added without having to reconfigure host data paths and the migration would be processed transparently and dynamically, without any downtime or loss of path redundancy while the migration is in process.

Increasing performance for a particular volume in a Storage Federation is somewhat less obvious because the other federated storage systems might not have more performance-resources available to boost performance. For instance, if a new storage system inserted in a Storage Federation does not have more disk drives (spindles) or more flash SSD capacity, there is no guarantee that any volumes moved to it would provide faster performance. There is some likelihood that performance could be improved by moving a volume to a storage system running at lower utilization levels, but maintaining lower utilization levels is not realistic or very cost-effective. All this said, it is possible for a Storage Federation to use the aggregate resources of multiple storage systems to increase performance of a single volume - such as by spanning all the disk drives in the federation and by using the cache of all participating storage systems. An example of products that provides this capability today are the Dell/EqualLogic storage arrays. 

The availability of storage systems needing maintenance could be greatly improved by Storage Federation if an existing storage system can be removed from the federation after having it's volumes evacuated (think v-Motion but for storage systems) to other storage systems in the federation. After being removed from the federation, the storage system could be downed to have any sort of maintenance operation done to it without risking the availability of any of the volumes that had previously been on the system undergoing maintenance.Obviously, all of this would take time planning and ensuring the Storage Federation would not be overloaded with data and workloads, but there are probably many customers that would prefer to do maintenance to storage systems when they are offline. 

One of the possible exposures of Storage Federation is the increased exposure to system failures. For instance, a Storage Federation that distributes a single volume over three separate storage systems is 3 times more likely to have a failure than a Storage federation that does not allow volumes to span across storage system boundaries. FWIW, this is the main weakness of the Dell/EqualLogic implementation of Storage Federation. 

That's all for now

It's late and there is plenty of material here for the grinder. Please feel free to comment in any way; agree, disagree, correct mistakes and ask questions.  Thanks for reading.

Why AO is a game changer

Yesterday, 3PAR announced Adaptive Optimization (AO), our solution for storage tiering and support for SSD flash drives. Here are the elements of this technology that I believe will have the most impact on customers and the rest of the industry.

1) Tiering works by making copies of data on lower cost, low-IOPS storage to high-IOPS storage - and back again.  Storage tiering has been associated with ILM, which assumed data is initially located on more expensive, high-IOPS storage and, as it ages and is accessed less frequently, is moved to lower-cost, low-IOPS storage. The perception that tiering implies fast to slow data migration was reinforced by Compellent with it's early entrant storage tiering technology, Data Progression.

The economic benefits of tiering are much more compelling if data is originally located on low-IOPS storage and then moved to high-IOPS storage when it becomes useful to do so. This reduces the amount of high-IOPS storage that needs to be purchased and reserves high-IOPS storage for the applications that need it the most. This model of promoting data to high-IOPS storage will replace the old model of data "trickling downhill to cheap storage."

2) Sub-volume tiering means high-IOPS storage can be reserved for high-IOPS work and effectively shared by the applications that need it the most.  AO copies data in 128 MB sub-volume regions that contain specific RAIDed volume slices. Many physical and virtual servers can have their volume's most active regions located in high-IOPS storage capacity at the same time.

Data redundancy is accomplished when AO reads data from it's source region and restripes it into a region on the target tier - using the RAID level of the target.  AO allows data to be protected by whatever RAID is appropriate for the tier and the data.  3PAR's chunklet architecture is maintained for SSDs, which means a SSDs in an InServ array can apply several different RAID levels simultaneously.  Every vendor's sub-volume tiering technology will be different, including the number of ways devices can be combined in RAID and how wide striping can be applied.

3) Tiering does not mean you have to buy SSDs to make it pay off.  Tiering is a cost-reduction technology.  One of the most obvious ways to reduce the cost of storage is to buy cheaper disks with higher capacity, such as SATA drives. 

The regions used by AO are the same on-disk structures that 3PAR uses for it's Dynamic Optimization (DO) software that re-levels volumes across disk drives in an InServ array.  A customer with all FC drives in an InServ array could take advantage of both AO and DO by increasing the capacity of an array with SATA drives, using Dynamic Optimization to redistribute their volumes across the SATA drives and then using FC drives as their high-IOPS AO tier. This way, they can continue to get the IO rates they expect, but reduce the cost of incremental capacity as they upgrade their system. 

4) The system determines what to move and how to move it. I/O density rate is a term that refers to how much data access occurs in a region over a given amount of time.  AO recognizes region candidates for tiering by their I/O density rates.  

Administrators control the AO participation for each volume by assigning them to an AO Profile and a QoS Gradient.  The profile is a short stack of device-RAID levels, such as SATA RAID 6, FC RAID 5(7+1) and SSD RAID5 (3+1). AO allows either  2 or 3 device RAID levels in the profile's stack.

The QoS gradient is a relative determinant of how quickly the volume will be acted upon.  I like to think of it as something like different viscosities for different fluids, but for storage.   AO today has three QoS gradients, performance, cost and balanced.

Back in Novemebr, Tony Asaro wrote about his discussions with HDS' storage customers regarding storage tiering.

Another discussion was around using policies to automate the process. One group was a bit concerned about automating this process but realized that, again, with PBs of data being stored that the only way to effectively implement intelligent tiered storage is via automation.  Additionally, it is not an all or nothing proposition.  You can select certain volumes and applications to implement and gain a comfort level before deploying more widely.  One of the key tenants of technology is to automate otherwise manually cumbersome processes.  We just need to get over that hurdle but we need to do so in a planned, considered and reasoned way.

By applying measured I/O density rates, AO profiles and QoS Gradients, 3PAR has taken the first major steps to automating storage tiering and removing the burden from administrators.

5) Tiering can and should scale out.  David Floyer from Wikibon wrote a good piece yesterday on our announcement where, among many things, he discussed how 3PAR is using smaller SSDs spread over more controllers:

....it spreads a small amount of SSD amongst the 3PAR engines so the IO’s aren’t all going to a single drive and sucking up a lot of bandwidth – it’s nicely balanced. Traditional implementations will use larger drive with more IO’s going to that drive. The part of the array with that drive will get more activity.

In practice we don’t think this will matter all that much because, for example, EMC’s V-Max has more bandwidth to play with than 3PAR and EMC uses its cache to transfer data between tiers to avoid bottlenecks. Nonetheless, on paper, the 3PAR implementation looks to be more efficient which means (in theory) it can do more with less flash. But nobody really knows yet.

3PAR storage arrays avoid I/O bottlenecks by incorporating tiny virtual storage elements (chunklets) and spreading the workload over as many devices and controllers as possible.  This approach differs from other vendors where smaller groups of resources are created and then combined into larger constructs that are more cumbersome to manage and tune than a single widely distributed storage span.  The same concepts apply to SSD integration, where InServ arrays accommodate multiples of many, smaller sized SSDs for scaling out high-IOPS tiers for those customers that may want to expand their use of AO in the future . 

3PAR Countdown: AO & SSDs

The news is here,  and    here,     and here.

3PAR announced its storage tiering technology today with the introduction of our Adaptive Optimization (AO) software and with support for flash SSDs. 

There's probably going to be a lot of discussion about storage tiering and AO in the weeks to come, so stay tuned.

Agility & Bull

3PAR has been designing and marketing it's agile storage for years. Recently, Hitachi started using the word "agile" in their messaging.  Imitation is a form of flattery, but at least they should link to our Cloud Agile partner page when they plagiarize like that. 

In contrast, EMC's announcement of FAST today only uses the word agile once, in Pat Gelsinger's quote:

“Over the past two years, EMC has optimized all of its major storage architectures to support new levels of automation required in virtual data center environments and for the transition to private clouds. These dynamic and agile environments require the capabilities that EMC’s FAST technology provides for effective management and scaling.  This is an area of significant investment for EMC and we are the only one offering this technology across high-end, mid-tier and unified platforms.”
- Pat Gelsinger, EMC President and Chief Operating Officer, Information Infrastructure Products

Online data movement is a big deal for storage and FAST puts it squarely in the mainstream of storage software. Gelsinger is right about virtual environments being dynamic and agile, but the problem with FAST is that the underlying storage does not measure up and FAST shines a spotlight on those shortcomings.  If you are going to have online data movement, you need to have the ability to make the results of those data movement operations productive - without turning your storage utilization goals upside down. FAST undoubtedly will work great where EMC storage is under-utilized and very expensive, but how will it work when you are trying to reduce the cost of storage by increasing storage utilization? 

It also looks like EMC is backing away slightly from their bullish position on SSDs. I was wondering how they were going to deal with their disappointing SSD sales when they finally announced FAST.

Speaking of bullish, I found this little video that I thought I'd share with you of some truly agile guys that really know their bull!

3PAR Countdown: Policy Advisor

The news is here.  Get a more in-depth view of Policy Advisor here.

3PAR customers have used Dynamic Optimization to re-balance storage volumes across multiple tiers of storage since 2005.  Our Policy Advisor software utility helps them make greater use of Dynamic Optimization by analyzing volume data layouts in their 3PAR arrays and generating the commands to move volumes to more appropriate storage resources. Customers can choose to run those commands as a batch process or enter them manually.

3PAR Countdown: TrafficBroker maintains performance levels with Dynamic Optimization

The news is here.     Click here to find out more about 3PAR's Dynamic Optimization - the tool our customer, TrafficBroker, is using to maintain their array's performance profile as they expand capacity.