Rise to the Data Challenge
When a government organization dedicated to developing cutting-edge technology begins to have trouble buying and managing storage hardware, it's no surprise that other agencies are experiencing similar growing pains.
In the case of the National Center for Supercomputing Applications (NCSA), which designs systems for science and engineering, information technology managers came to the slow realization that the long-standing practice of letting center groups procure and manage their own storage hardware simply wasn't working.
As storage needs throughout the government-sponsored NCSA grew and storage hardware became more complex, the center's staff found that its decentralized approach was becoming increasingly unwieldy.
After evaluating the options, the IT staff decided to centralize both the buying and management of storage hardware. The result? A slew of benefits, including more targeted and forward-thinking solicitation specifications, greater economies of scale for storage-related purchases, and the ability to ramp up projects more quickly because storage capacity was ready and waiting instead of tied up in the procurement cycle.
"Basically, we decided we could get a lot more bang for our buck if we pooled our resources," says Michelle Butler, technical program manager for NCSA's Storage-Enabling Technologies Group. The Champaign, Ill., center now buys high-performance storage systems in bulk and has centralized the oversight of its storage hardware.
With storage needs growing so quickly—due as much to increasing data capacity as to the proliferation of federal regulations that require information be kept for longer periods—virtually every agency is in the same boat.
Simultaneously, the range of storage technologies has exploded, creating the additional challenges of figuring out what to buy and how best to manage it.
Buyer's Dilemma
As storage capacity grows and an agency's needs change, it becomes difficult to decide what technology best fits the demands—especially if the agency has a variety of computing environments and storage devices spread across its organizations.
To ensure that NCSA had the right products to handle its growing and shifting requirements, the IT team decided three years ago that it needed to centralize storage buys. Before that time, each group bought its own storage—everything from high-performance storage disks and infrastructure storage area networks (SANs) to a variety of file servers and other hardware.
Because the research center runs extremely fast supercomputers, it was particularly important to ensure that the storage requirements in requests for proposals were written to exact specifications—something that was better accomplished centrally, Butler says.
"We have to make sure the units we procure actually do input/output fast enough for our applications, can connect into our environment and won't disturb the current SAN environment," she says. "Last year, for example, we needed two allotments of 112 terabytes of storage, and we had to be very specific on what we needed—on the performance, how many storage units we needed the hardware to support and how many disk drives were included to make sure we got Fibre Channel 2-gigabit-per-second throughput."
Control Factor
By having one team do the buys, an agency can also rein in the number of vendors and hardware brands it must support, notes Rose DuBoise, technical adviser for the Army Surface Deployment and Distribution Command at Fort Eustis, Va. That's a critical factor when it comes to managing storage environments and trying to identify economies of scale, she says.
"Mixing and matching creates a learning curve," says DuBoise, who runs storage procurements for the command's continental U.S. organizations. She recommends stating requirements clearly and carefully, which will cull the likely bidders to a small but appropriate group of vendors.
"You can't specify vendors in an RFP, but you can fine-tune your requirements so your list is narrowed down to just a few major players," DuBoise says.
Centralizing buying has other benefits as well. By pooling resources, for example, NCSA was able to buy more disks at a lower per-disk price. "Some groups got faster disks and more disks with the same money they would have spent on less," Butler says.
Being able to buy larger quantities of storage at less cost per unit means projects requiring more capacity than originally anticipated can get off the ground faster—an unanticipated boon, Butler notes.
But for all agencies, the key to procuring storage hardware is knowing the current infrastructure and anticipating future needs—something that not only requires centralized buying but an expert dedicated to keeping current on technology, says Tony Prigmore, a senior analyst at Enterprise Strategy Group. The Milford, Mass., consulting company focuses on storage and security.
"A lot of RFPs don't accurately reflect the most ideal solution," he says. "To make sure yours does, assign somebody to talk to different users about the different applications that might be supported by the storage infrastructure." A full assessment, Prigmore says, should include, among other things, inventories of the organization's backup and disaster recovery needs.
Although there have been significant advances in storage hardware over the past several years—in capacity, functionality and price—the advances have created as many management issues as they've solved.
With scale comes burden. Agencies must determine how best to provision and protect their storage systems, ensure that they have enough capacity and performance, and make sure they assign storage properly throughout their organizations.
The Energy Department's Brookhaven National Laboratory faced those types of issues as its storage capacity exploded and became more heterogeneous.
"We had three or four different environments that were getting harder to manage as we grew, and the volume of backup was growing and becoming unmanageable," says Andrew Ferguson, manager of enterprise operations in the IT Division at the Upton, N.Y., lab.
Last year, Ferguson's group chose to standardize all backup under Legato NetWorker drive-sharing software from EMC of Hopkinton, Mass., while using the organization's fiber infrastructure to stow data on two central tape library units from Quantum of Milpitas, Calif. The server group has also collapsed much of its storage on a SAN to move away from an environment of dated and labor-intensive direct-attached storage devices.
Standardizing and consolidating storage functions has made it easier to set staffing needs, keep track of media and archive data at the lab, Ferguson says. "There's no guessing about what tapes are where anymore."
Standardizing on one vendor's equipment as much as possible also makes sense, Ferguson says. "It helps us in getting new and old equipment to work together, and there is less of a need for training on new hardware. And if something goes wrong, there is no finger-pointing between vendors."
NCSA's Butler adds that having one group manage storage functions means her center can avoid downtime.
"We had to centralize storage management out of necessity," she says. "With somewhere over 6,000 disk drives that could all fail at some point, we had to know when, where and why they were failing, and get them replaced as fast as possible. We had to know what controllers were hosting what systems, and if they were behaving or seeing failures of some sort."
Butler's team chose to automate the process, using a variety of shareware tools to create a data-mining application that searches for and reports problems via a Web interface to a help desk. The storage team can then fix problems based on priority.
Challenge of the Ages
No matter how well an agency shores up its buying and management practices, it must keep in mind an age-old challenge: human nature.
"People get used to the way things are done," Ferguson says. "When you're dealing with smaller backup environments, people are used to changing out a string of tapes manually, making sure it was all done in the right sequence. Going from that mindset to dealing with a large library and disk backup system is a big change. It takes time."
Changing people's routines and predilections is perhaps the biggest management challenge of all, DuBoise agrees. But her group has found a way to deal with user reticence: Don't tell them.
She recounted how the center shifted data from users' SCSI hard drives to serial advanced technology attachment drives and kept the change transparent.
"When we decided to go to ATA disks, our users refused to give up their files. We found a way to switch to ATA drives without them even knowing their files had been moved off of prime disk space to the new drives," DuBoise says. "When they sit down at their desktop PCs, it looks like their file is there, even though it's off on an ATA drive somewhere."
No matter what the challenges, one thing is clear: Storage hardware will continue to improve, and agencies' storage needs will continue to expand. Keeping up requires vigilance—about what to buy and how to manage it. Enterprise Strategy Group's Prigmore advises agencies to have dedicated, trained personnel who know what's best for their storage environments today and in the future.
"Put somebody to work on your behalf who can investigate the environment and do a needs analysis with some objectivity," Prigmore says. "If you can do that, you're ahead of the game."
