All the world's computers are flawed, and companies are fumbling with fixes. It will take years until the issue is fully sorted out.
In the three weeks since researchers revealed major flaws in virtually every computer processors, issuing patches has not gone smoothly.
"It's been a bit of a disaster," said Ellison Anne Williams, founder and CEO of security firm Enveil. "The problem happens in the memory of the computer. Anytime you start messing with memory, things can go wrong very quickly. Some of the patches coming out do unexpected things. It takes a long time to see how things are going."
A bungled rollout for fixes is not surprising. The issue is so massive, and the vulnerabilities so embedded into the building blocks of computers, experts say the difficulties will continue.
What was the problem in the first place?
Meltdown and Spectre are flaws in processors, the brains of computers and smartphones. Modern processors are designed to perform something called "speculative execution," or predicting what tasks they will be asked to do. That data is supposed to be protected and isolated, but researchers discovered that in some cases, the information can be exposed while the processor queues it up.
These flaws go back decades. Some companies have already issued fixes for these problems -- for example, Microsoft (MSFT), Apple (AAPL), and Google (GOOG) products received updates quickly.
Microsoft reportedly faced early problems with its patches creating issues for anti-virus products, and earlier this month, the company said updates will likely slow down computers. Apple said its mitigations don't have measurable performance impact.
In order to ensure devices are fully protected, hardware makers must put out microcode updates to tell chips to behave differently. Software companies must also update their products to protect against exploitation.
Paul Kocher is one of the researchers who discovered Spectre. He's a veteran of chip vulnerability discoveries -- in 1998, he found another hardware issue called differential analysis. He said that despite the last two decades between the chip flaws, the industry still doesn't have a solid grasp on how to fix these types of issues.
Related: How to protect devices from Spectre and Meltdown chip flaws
Hardware flaws don't fit into the traditional patching model -- unlike software flaws where a vendor issues an update and users can download it quickly, chip flaws require a different strategy.
"The playbook everyone's familiar with is one that works well for software bugs, but not a lot of clear thought has gone into how to handle situations that don't fit that mode very well," Kocher said.
Since early January, issues have piled up.
Intel (INTC) introduced a fix, then told companies earlier this week to hold off on implementing patches because they were addressing a reboot issue caused by the updates. VMWare also said this week it is delaying new updates, while Lenovo, Dell, and HP pulled some fixes following Intel's advice.
Patches caused machines to reboot or slow down, and in some cases, full system crashes referred to as the "blue screen of death."
On Wednesday, the House Energy and Commerce Committee sent letters to the CEOs of Intel, AMD, ARM, Apple, Microsoft, Amazon, and Google, all of which were informed of the vulnerabilities before they became public, asking why these firms kept the flaws under a strict embargo.
If you just use a laptop for checking emails and watching Netflix (NFLX), you likely won't notice a difference in your computer's function after it receives updates. Where chip flaws are causing issues is within businesses.
According to data from Spiceworks, a professional network for people in the IT industry, 70% of businesses have begun patching against the flaws, and of those 38% have experienced problems with the fixes, including performance degradation and computers crashing.
The study also found that of the 29% of large companies who expect to spend more than 80 hours addressing the issues, 18% expect to spend more than $50,000 to fix them.
What is the tech industry doing now?
Companies are continuing to test and release patches to mitigate the problems caused by buggy updates as well as fix the vulnerabilities. The Meltdown flaw can be fixed through updates to the operating system, but fixing Spectre requires updates to a variety of components, including microcode, Kocher explained.
"If you look at how long it's going to take for all of the relevant software on your PC, including the drivers and such are updated, you're probably looking at many years before that process is done," Kocher said.
On Intel's fourth quarter earnings call on Thursday, Intel CEO Brian Krzanich said Intel has been working to incorporate silicon changes into products to directly address the Spectre and Meltdown flaws. That means new chips won't have these problems. They will start appearing later this year.
Kocher said although fixes are rolling out, it's likely researchers will see variations attacks taking advantage of the chip flaws popping up for a long time.
According to Enveil's Williams, who spent over a decade as a researcher at the NSA, Spectre and Meltdown have exposed a vulnerable point of entry for sophisticated attackers that companies -- and many hackers -- likely didn't think about before these flaws were made public.
"Coming from a nation-state perspective, the memory attack surface was normal and pedestrian," Williams said. "The awareness wasn't in the commercial space. The only difference between now and three weeks ago is now it's exposed."
Experts say the attention now paid to the recent flaws will likely lead to more revelations about the insecurity of computers' building blocks.
New chips will eventually mitigate the problems, and in the meantime, hardware and software makers are rushing to fix the vulnerabilities. For now, it's unclear how this massive security issue will change the fundamental strategies currently used to make processors.
Kocher has a potential solution, but he admits he's in the minority for considering it. Companies should produce different chip designs depending on whether security or performance is more important, he said.
"I don't see any way you can optimize simultaneously for the best possible security as well as playing video games with the best graphics possible," he said. "I think you need different hardware and software to do those kinds of tasks."
