In this age of Big Data, mobile communications and the Internet of Things, virtually everyone in the IT industry is aware of the need for scale. But even with dynamic cloud architectures at the ready, is there such a thing as too much scale? And are there right ways and wrong ways to implement scalability across data center infrastructure?
To hear infrastructure vendors tell it, scalability should be the top priority for enterprises across the board. And indeed, as Apcon CEO Richard Rauch told CIOL recently, with increased traffic soon to be coming from virtually everything we touch, data center infrastructure will have to scale in order to meet the availability and reliability levels that we have come to expect. For a networking company like Apcon, this means advanced switching capabilities that support non-blocking connectivity and heavy traffic flows, along with the visibility tools needed to keep an eye on things.
So far, so good. But should it be the goal of every organization to build unlimited scalability into their data environment? National Instruments’ Dr. Tom Bradicich notes that small businesses obviously don’t face the immediate scalability needs of large organizations, unless they are on a fast track to becoming large themselves. As well, large organizations will more likely require the tools and capabilities to harness and manage larger volumes of internally generated data, while smaller firms should focus on publicly accessible traffic. Either way, however, the challenge going forward will not be how or what to scale as much as finding a way to meet data needs without going over budget.
The vanguard in this movement is the growing number of Web-scale organizations—the Amazons, eBays and Facebooks of the world—who have to go from zero to trillions of transactions seemingly overnight. But while these kinds of organizations face truly unique challenges when it comes to scale, it is inevitable that many of their solutions will trickle down to the wider enterprise industry. Examples of this are already evident, says Computer Weekly’s Cliff Saran, most notably tools like MapReduce and NoSQL that were originally developed by Google but have since been adopted for a broad array of data center challenges.
Whether you are scaling up or scaling out, however, your primary concern should be to do it in a coordinated fashion so that increased activity from one set of resources does not overload another. This can be particularly tricky within highly integrated environments like unified storage, according to Mike Vizard at IT Business Edge. As data traffic increases, so too does the challenge of managing requests across file, block and object storage, particularly if those resources are still governed by discrete controllers within the unified architecture. Companies like Caringo are starting to focus on this problem with tools like the Swarm7 management stack, which can now be deployed across petabyte-size storage clusters.
Scale, then, is more than simply growing your resources. It involves tackling greater data burdens in ways that are both economical and non-disruptive to legacy operations. But there is no reason at this point for the vast majority of organizations to strive for unlimited, Web-scale infrastructure either at home or in the cloud.
Size does matter in the data center, but IT should resist the temptation to overdo it. It may very well turn out that your data challenges are not so unmanageable after all.