December 2, 2013
From a Maryland war room, a team of HealthCare.gov fixers hosts two troubleshooting calls a day, monitors the site in real time from 15 large screens and demands that the operation work at "private sector speeds."
The remarkable part: None of this was in place when the website launched the first time.
President Barack Obama pledged for years that HealthCare.gov would rival top e-commerce sites, but it wasn't until after the disastrous Oct. 1 rollout that the White House began setting up the kind of operation that could even come close to delivering on the promise.
So today, the self-imposed Saturday deadline to make the insurance marketplace work for most users, the White House is trying to show that the site has turned a corner?without declaring outright victory.
The message: We're getting it right this time.
Tech officials this week briefly raised the veil of secrecy surrounding the administration's two-month scramble to rescue the centerpiece of Obama's chief domestic policy, meeting with reporters to detail the site's progress.
The website will be able to handle 50,000 users at a time while maintaining a low error rate and page load time, said Jeffrey Zients, the leader of the repair effort. These were targets that the administration set for the initial launch but fell woefully short of, as the system crashed when only a few hundred users logged on.
They've fixed more than 300 bugs, and they continue to identify new glitches as the traffic increases, "but for the most part, they're not nearly as significant as the original bugs," Zients said.
"We are confident that we're on track to achieve our goal to make HealthCare.gov operate smoothly for the vast majority of users by the end of the month," he said. "You continue to find glitches and bugs, and you inevitably continue to upgrade on the hardware front. But we have rapid-response teams on all those efforts, and as additional problems come up we'll jump on them and fix them."
The briefing wasn't an effort to declare the site fixed. Administration officials are trying to avoid making such pronouncements, aware that the marketplace still faces major challenges as millions of consumers who have been locked out of the site attempt to log on again in the next few months.
Rather, the sitdown in the White House's Roosevelt Room was part of an effort to begin rebuilding confidence in the president's management of the Affordable Care Act.
Even with the brief glimpse inside the operation, it's too early to tell whether the more streamlined, disciplined approach that Zients said he instituted will allay deep doubts about the website and the law itself.
The White House briefing for reporters with Zients and his team included officials from general contractor QSSI and an engineer on leave from Google who insists that everybody but his mom should call him "Mikey."
"If you bring this all together?private sector speed and execution; the command center in Columbia with real-time monitoring; troubleshooting when things go wrong, and overall coordination and direction; a hardware-upgrade team, and then a software-fix team?what's happened is the site has gotten better and better each week," Zients said.
The White House conducted a similar briefing for private sector IT experts. John Engates, the chief technology officer of Rackspace Hosting, who once called HealthCare.gov "one of the most spectacular public failures of any website ever," told CBS News that he "left D.C. confident that the healthcare site is on track to be fixed."
Some opponents and allies remain skeptical, given the president's promise of a cutting-edge shopping experience the first time around. Emails released this month by House oversight committees showed a chaotic rush by federal officials to figure out what was plaguing the site in lead up to the Oct. 1 launch.
When he joined the overhaul effort in October, Zients said he "brought a fresh set of eyes," and came to three conclusions.
"One, that HealthCare.gov, the site, is fixable," he said. "Two, that we needed additional talent and expertise to do this fix. And three, that we needed to ramp up the speed of execution to private sector speed, and we needed to do that by constantly triaging faster decision-making in a 24/7 around-the-clock focus on execution."
The administration appointed a lead contractor: Quality Software Services Inc., or QSSI, a unit of health insurer UnitedHealth Group based in Columbia, MD. The firm was picked after successfully building the complex data hub, which transfers information between consumers and government agencies to verify the applicant's identity and income.
QSSI then reorganized the operation into three groups: a command center, a hardware team focused on major upgrades and a software team that works through fixes. The outfit includes hundreds of business analysts, system analysts, developers and testers, and represents about a dozen companies and the Centers for Medicaid and Medicare Services, which is the government agency in charge of implementation, the officials said.
At the command center, a group of 25 people serves as "the eyes and ears of HealthCare.gov," said Bikram Bakshi, the president and CEO of QSSI.
They continuously watch the website's real-time activity through data projected onto 15 large screens and monitors: statistics on current traffic, how many people are attempting to access various parts of the site, whether they're getting stuck on any single application.
"If we start seeing any kind of variances beyond what the team is used to, then the team springs into action and then is able to essentially exercise the protocol that they have put in place from an operational standpoint," Bakshi said.
There was nothing of this scale in place before the White House took over the operation and installed Zients, officials said. The government had only limited capacity to monitor the entire system to make real-time decisions, and there was no main contractor in place to oversee the deep roster of vendors.
Now, they conduct two "war room" calls a day that are designed to review the last 12 hours of activity, and look ahead to fixes and upgrades that are in the works, the officials said.
"We don't have an hour or a minute to waste, and therefore we need to make sure that there is ruthless prioritization at all times as to what matters most, that there's real-time reaction if something unexpected happened, someone hit a roadblock on a fix -- so what are we going to do about that bug and how are we going to address that bug, and you get the best minds on the problems right away," Zients said. "So the point of that discipline is to make sure that we're all on the same page as to what the priorities are and how you're going to get around any unexpected issues."
One example of the "ruthless prioritization" was to focus first on fixing the site for consumers, officials said. That decision has meant that insurers are still getting faulty reports on those who have signed up for coverage, which could become a major problem once more people buy insurance and try to use their benefits.
At the briefing, Zients offered a hint of his management style, demanding that the five officials who addressed reporters adhere to their strict time limit for speaking. If they paused for too long, Zients nudged them to speed up.
He asked Michael "Mikey" Dickerson, the Google engineer, to describe his work at the search engine. Dickerson, who appeared by video conference from the command center, started out with an explanation of his name.
"So I'm Mikey Dickerson, called Michael sometimes by you guys. But that's weird to me, because only my mom calls me that. Everybody else calls me Mikey," he said before explaining that he's worked as a site reliability engineer at Google for eight years. "I'm on unpaid leave right now, so I'm working for QSSI. Anything else you want to know?"
"No," Zients said. "Go for it."
Dickerson ticked through a series of slides showing the data that the team monitors at the command center, and Zients kept him on task, intermittently barking out "next slide."
After a quick 36 minutes, Zients ended the briefing.
"I'm going to release this team now," he said. "We need to get these guys to do their standup call."
Quality News Today is an ASQ member benefit offering quality related news
from around the world every business day.