While the IC’s research organization looks into adding security to cloud environments, in the here and now, intelligence agencies are sharing more data.
In the past, agency acquisition officials faced a daunting task if they needed to fill a new call center with state-of-the-art servers and associated software, networking gear and storage devices.
TheyÂd meticulously specify hardware and software components, sometimes down to parts numbers and performance characteristics, while also setting implementation milestones Â perhaps even telling potential contractors how the new system would plug into existing operations.
But performance-based acquisitions promise a new approach. The PBA strategy charges the acquisitions people with defining performing levels for the new call center and lets contractors figure out how to achieve the desired results.
This makes intuitive sense, but widespread PBA use remains sketchy, even given Office of Management and Budget mandates stretching back to 1991 and the later 2001 PresidentÂs Management Agenda directive that targeted PBA use for 50 percent of relevant acquisitions (a number later reduced to 40 percent).
But new research suggests that even the lower targets are tough to meet. In addition to cultural clashes that force acquisitions officers to learn new work methods and forge new relationships with operations staff, agencies also often sidetrack PBAs because of difficulties tied to estimating return on investments over a contractÂs life.
According to OMB, agencies wrote 45 percent of their eligible service contracts as PBAs last year.Â But a study sent this month to the president and Congress concludes that agencies may be over- reporting PBA reliance.
ÂWe found a large number of [agencies] have erroneously reported their use of performance-based contracts,Â says Carl DeMaio, president of the Performance Institute of Washington and co-chairman of the Performance-Based Service Acquisition Working Group of the Federal Acquisition Advisory Panel, which wrote the new report. ÂThey are calling contracts performance-based when in fact they are not.Â
DeMaio estimates that only about 20 percent of federal contracts truly use performance-based acquisition techniques.
The problem stems from continuing confusion about the use of such acquisitions. ÂFederal agencies are very unclear as to what a PBA is and how to use it,Â DeMaio says. One difficult area is how to define and collect performance measurements. ÂAgencies tend to measure too many things, and that creates its own costs for contract administration,Â he adds.
ThatÂs bad news for PBA proponents who believe the methodology offers the best hope for simultaneously delivering technology and services that meet agency needs, while also assuring that government spends taxpayersÂ dollars wisely. ÂPBA grew out of the $800 hammerÂ scandals of years past, says Mike Cameron, a PBA best-
practices specialist at the consulting
firm Booz Allen Hamilton of McLean, Va.
Stumbling blocks occur over ROI and performance measurements, he says. ÂIf you donÂt think beyond the procurement to consider how you are going to ensure that the delivery occurs in a manner that satisfies your needs, then the wheels are going to fall off before youÂve ever received the first bid,Â Cameron cautions.
The Coast Guard is among the handful of agencies observers cite for having found success with its PBA efforts. The key for estimating ROI often lies in what happens after the agency and vendor sign the initial contract, says Scott Palmer, chief of the Coast GuardÂs Major Systems Contracting Division. ÂAny return on investment calculation that relies purely on the initial contract cost is a poor one,Â he warns.
Realistic ROI estimates should include using the initial contract value as the starting point and then add costs for training and performance monitoring through the contractÂs term. Both these cost areas are fluid, as agencies and contractors change how they meet a projectÂs objectives, so they are not static expenses an agency can easily define from the start, Palmer says.
ÂMidway through a contract, a contractor or one of our representatives may say, ÂHereÂs whatÂs working; hereÂs whatÂs not,Â Â he says. ÂOnce you implement a change
in the interest of performance, it can be
difficult to capture the costs associated with that. You are going to incur some costs to put the change to work and then thatÂs got to get pumped into your ROI formula.Â
Building an accurate ROI model starts with forming an integrated project team, which might consist of a program manager, contracting officer, budget and legal personnel, end users and quality assurance specialists. The teamÂs job is to define project needs and tools for measuring success.
ÂOnce we get the parties at the table, itÂs a fairly simple exercise to get the structure we need,Â Cameron says. ÂThe hard part is overcoming a culture that says the [procurement and operations] sides of the house donÂt talk with each other very often.Â
DeMaio adds that outside acquisition advisers should be brought in to facilitate this process for any Âhigh-risk, high-cost or high-impactÂ procurement. ÂHire an adviser to help you look at your alternatives and define your needs,Â he says, adding that the outside team needs to have market experience in the procurementÂs focus area, a component of the market research stage that PBA experts consider a planning best practice.
PBA training is also essential for both acquisitions personnel and the operations staff who must carry the new responsibility of defining project results. ÂYou cannot leave the program side out of the training equation,Â DeMaio says. ÂThey are the ones that know their problems. They are the ones that must define the needs and the requirements. The acquisition workforce is there to help keep things legal and help everyone through the process.Â
Even with a full working group, the acquisition team might continue to struggle with creating accurate ROI models that donÂt underestimate full lifecycle costs, Palmer acknowledges. But they should stick with it, he adds.
ÂThat fact paints us as being overly optimistic with ROI because we are not capturing all the costs,Â Palmer says. ÂBut I would also submit that large organizations may also be overly conservative when we are estimating some of the benefits, such as end-user productivity.Â