McAfee's Apology - And Recompense For Corporate Customers

McAfee has revealed the first phase of their corporate customer followup to the 5958 DAT issue. The notice, available via their SNS notification service and via McAfee's website, says:

McAfee is offering a complimentary one year subscription to our automated security Healthcheck Platform. This will include a 4-hour session of remote consulting in which we will help you set up and run the health check, review your policies, server configuration and environment, interpret the results and provide recommendations based on McAfee best practices. If you would like to take advantage of this offer, please email Register@McAfeeQuickstart.com by June 15, 2010.

Unlike the offer extended to home users, which included a 2 year subscription renewal, and the potential to see recompense for "reasonable" repair fees, this does not include any real significant remedy to corporate users. While the expense to McAfee would be far higher than the costs from their home users, it does mean that corporate customers may feel that they have been left out in the cold. Stories of hospital emergency rooms (as described in the comments on techielobang and Ars Technica's posts about the issue) and 911 call centers (as described by the ISC) going down in the wake of the DAT release mean that McAfee will likely face very upset customers as the full cost and impact of the issue are calculated.

McAfee president David DeWalt's open letter, posted to the McAfee Security Insights blog claims that "The vast majority of affected users were back up and running smoothly within hours, and we are continuing to work diligently until we are sure that every last user node among each and every one of our customers is back in action." - a claim that may be difficult to back up in large organizations, or for users whose only access to the Internet and email was taken down by the issue. In many cases, large organizations with a significant XP SP3 install base saw hundreds of systems taken down that required manual visits. Stories such as this comment posted in response to his letter tell a story of woe. "Our team of 10 technicians worked for over 24 hour straight to touch all 1200+ machines by hand in order to assure our patients safety and our continued operations."

For an organization, that's a cost that could reach into thousands, if not tens of thousands of dollars in staff time alone. Lost productivity for offline machines could total far more. What will McAfee offer to heavily affected corporate customers? We'll likely know in the next week or so. Until then, security professionals and IT support staff can only point their management to McAfee's apology, and decide how they can best update their business practices to avoid update issues without compromising quick response.

Perhaps more interestingly, as our industry starts to ponder whether whitelisting is the way to, we should consider what a bad update to a whitelist could do to our organizations. The risk model is much the same - so the question will remain. Do we trust our vendors QA processes and update release methodologies?

McAfee's DAT Debacle: When Your Security Software Causes Harm

On Wednesday, April 21st, McAfee's 5958 DAT file (McAfee's virus definitions file) was released with a bad detect for the w32/wecorl.a virus. This inadvertently detected svchost.exe on Windows XP machines as an infected process, and VirusScan took its normal actions - either quarantine or deletion in most cases. Per McAfee's description of the problem, this was a heuristics issue that only showed when deployed - which should make you wonder about how their testing is done.

At 12:05 PM, EST, McAfee sent out an urgent email stating:

"McAfee is aware of a w32/wecorl.a false positive with the 5958 DAT file April 21 at 2:00pm (GMT +1). McAfee advises NOT to download this DAT. Please disable pull tasks and update tasks.

Information updates will be sent every 90 minutes to keep you advised."

Administrators who quickly pulled the DAT using ePO were somewhat protected, as only those systems that had checked in since the DAT release were effected. Those users who did not update while the bad DAT was available were similarly safe. In many organizations that use McAfee's e-Policy Orchestrator (ePO) management tool, this was reasonably easy. For those that do direct updates, this would not have been as simple, and in either case, many systems did update with the bad DAT during the time it was available.

For enterprise users, this meant that any machine that received the DAT before McAfee's 12:05 EST email were likely taken offline. If the system rebooted, it would typically no longer have a working network connection, and thus could not be remotely repaired. Symptoms included blue screens and DCOM errors, as well as shutdown messages.

Less than 90 minutes later - as promised, McAfee sent out a second email with more detail, citing XP Service Pack 3 systems as problems, and noting that the DAT had been removed. In addition, it provided links to McAfee's knowledgebase which, unfortunately, was already starting to perform poorly under the load.

An extra.dat (McAfee's name for off-cycle, non-mainstream update files) was made available shortly after this, with a general email via SNS coming out just before 4 PM EST. The email provided more detail on an issues page for the DAT.

At 8:40 PM, EST, McAfee published both a recommended and an alternate remediation procedure for the DAT created issues. These procedures were included on the issues page, making it an easy central reference.

In total, less than 9 hours had elapsed. A total number of affected machines worldwide is unlikely to be released, but my own experience indicates that the number is likely quite sizable. For most organizations, today has been a day of remediation, often by hand, as machines that rebooted were unable to be accessed remotely with their networking broken.

One of the biggest lessons learned is that McAfee's reasonably new SNS notification service is a must-subscribe for McAfee users and admins. Another is that ePO can greatly increase your chances of getting in front of a widespread update issue given sufficient notice.

For McAfee, the lessons will not be easily forgotten - better testing, amongst other practices is likely to be on their list. It is equally obvious that McAfee did learn a lesson since the infamous DAT that detected Microsoft office files as infected, and their 90 minute notifications and quick response clearly show that.

Will this change how your organization views antivirus updates? Is immediate deployment worth the danger? I'm sure I will be having serious discussions with local PC support staff about the relative risks of a 24 hour delay against the possibility of a bad DAT - and I'm not sure I have as concrete of an answer as I would have a week ago, particularly in light of the increasingly high percentage of malware that evades mainstream AV.