Not a fortnight after Northrop apologized to Virginians for the multi-day IT calamity which hit 26 state agencies, the GSA has awarded Northrop Grumman the 10 year, $2.6+ Billion IT infrastructure project at its new St. Elizabeths Psychiatric Hospital compound. While Northrop rebounds hot on the heels of its Virginia IT discomfiture, less clear is the fate of the mission critical DHS IT infrastructure. Should the DHS agencies being consolidated on the St. Elizabeths campus suffer the outages, delays and disconnects through which Virginia state agencies have struggled over the first years of the 10 year Northrop Grumman IT network overhaul, one wonders how perilously homeland security could be compromised. A physical attack on DHS's soon to be centralized offices would not be necessary to paralyze operations, if the Virginia experience is any warning.
Not a year after a Virginia Joint Legislative Audit and Review Commission found pervasive problems with Northrop's work, including network outages and help desk, back up, and disaster recovery inadequacies across numerous of the 89 state agencies served, much of the Virginia network went entirely dark for more than a week across 26 state agencies. The debacle paralyzed operations, disrupting the state business of tens of thousands of individuals and businesses, from taxes to driver's licenses, Juvenile Justice to Social Services. Diverse critical agencies were hit hard, it is thought, because one of two or two of two memory cards failed and knocked out some 500 servers in a cascading disaster. Northrop will pay Virginia a $100K penalty and another $250K for an independent investigation into the disaster, but this comes after years of disturbing reviews of the Northrop-Virginia IT project and apparently fruitless state interventions from withheld payments to imposed penalties. Sam Nixon, who as head of Virginia Information Technologies Agency oversees the Northrop work, has been quoted as saying that the state will recoup its financial losses in addition to the $100,000 penalty, including credit against its bill for the network down time.
One would hope that the Northrop results for DHS would be far better than Northrop's results in Virginia. However the JLARC 2009 review, prescient in view of the 2010 disaster, cited not only its own audit results but equally disparaging, independent reviews from HP and others. HP cited serious conclusions in 2007, one year into the project:
"- Planning rated “fair to poor”
– “Awkward & uncoordinated patchwork” of transformation plans led to improvisation
– Transformation schedule “almost incomprehensible”
- Communications & teamwork rated “poor” & “very poor”
– Ineffective internal & external communications
– Plans not shared or coordinated with stakeholders
- Risk management rated “poor”
– Lack of ongoing risk management program left outside risks unaccounted for, like subcontractor delays." (page 47 of JLARC report)
Consultant CACI found schedules contract driven and the plans insufficiently detailed, and Deloitte Touche found serious unresolved high-risk issues. (pages 48, 49 of the JLARC report) State agencies reported not only seriously disrupted operations (pages 50 et seq.) but also reported being "unable to acquire or expand disaster recovery services." Northrop was reported as not having "adequately ensured data are regularly & successfully backed up." Pretty damning.
Northrop fought back in a public statement, and, while agreeing that there had been problems, attempted to marginalize the JLARC report as a "historical review" which did not reflect changes and improved results. The statement attempted to excuse the shortcomings, saying: "Many of JLARC’s key findings reflect first-of-its-kind nature of the project. Nothing on the scale of this project exists. There are no models from which to draw experience. Virginia is breaking new ground..." (Then Virginia might have wanted a pro like IBM or HP, famously partnered with Oracle, and Symantec?)
That may be, but the choice of Northrop Grumman for the DHS IT infrastructure project days after the Virginia debacle and months before the $250K analysis of that debacle will be complete seems a slap in the face of the other DHS/GSA bidders. Five years ago Northrop was chosen over IBM by the State of Virginia although "IBM performed better on experience & finances," according to the JLARC report. Virginia has had five years of reasons to regret that decision. GSA and DHS decision makers needed only to cross the street to ask Aneesh Chopra, who as Virginia's Secretary of Technology experienced first hand the troubled Northrop Grumman performance. Surely he would have shared the reservations of the exceedingly damning JLARC report. Homeland security paralyzed for days would not be a happy outcome.
© 2010. Sigrid Caroline Schroder. All rights reserved.
Comments
All bidders have mix of what you'd call good and challenging programs. This happens with large IT jobs, especially ones that are challenging. For Northrop, they have tremendous success in programs at Treasury, DHS HSDN, and many others. They performed very well in the stand up of DHS IT at Nebraska Ave....items you fail to mention.
You seem overly bitter about the non award of Virginia to IBM. If IBM (and HP) are so good, then why didn't they bid this job as prime. Maybe IBM's lousy performance with DHS CBP ACE might have impacted their own internal bid/no-bid. Who knows, but give credit to government evaluators for doing their job and sorting through the issues.
GSA withdrew its award to Northrop Grumman, after other contractors filed protests against the award.
GSA found its best option was a new solicitation for the work “to allow for re-solicitation and to expediently and effectively resolve any and all issues raised in the protests,” Sara Merriam, GSA’s press secretary, said in a statement.
“After reviewing the protests filed in response to the St. Elizabeths' IT award, GSA decided that the best course of action was to re-solicit the requirements." Northrop's competition had filed such convincing protests regarding fairness that the GSA decided “to allow for re-solicitation and to expediently and effectively resolve any and all issues raised in the protests,” according to Sara Merriam, GSA’s press secretary.
On June 7, 2011, the GSA announced that it had awarded an $867 million contract to General Dynamics Corp. to develop the integrated network infrastructure for the DHS headquarters at St. Elizabeth’s in Washington, DC, after rescinding Northrop's initial $2.8B contract award last November.
Initially losing bidders General Dynamics and Lockheed Martin had filed protests with the GAO last fall asserting unfairness.
Is there a good analog to Virginia in another state that address an entire state's IT infrastructure? I have heard about these problems but as citizen, I've never had any trouble getting service through the Virginia on-line DMV and on-line tax services I've used.
Note, the GSA rescinded the $2.8B Northrop contract last fall, opened the process to re-solicitations, rejected Northrop's protests, and in June awarded the contract, at a much lower price, to General Dynamics.
IBM and CACI are on the Northrop team. CACI must be doing the schedules and IBM providing the pros.
General Dynamics will be doing both St Elizabeth's and DoD Washington Headquarters Services support.
Got something to say?
Examiner.com is looking for writers, photographers, and videographers to join the fastest growing group of local insiders. If you are interested in growing your online rep apply to be an Examiner today!