McAfee's Apology - And Recompense For Corporate Customers

McAfee has revealed the first phase of their corporate customer followup to the 5958 DAT issue. The notice, available via their SNS notification service and via McAfee's website, says:

McAfee is offering a complimentary one year subscription to our automated security Healthcheck Platform. This will include a 4-hour session of remote consulting in which we will help you set up and run the health check, review your policies, server configuration and environment, interpret the results and provide recommendations based on McAfee best practices. If you would like to take advantage of this offer, please email Register@McAfeeQuickstart.com by June 15, 2010.

Unlike the offer extended to home users, which included a 2 year subscription renewal, and the potential to see recompense for "reasonable" repair fees, this does not include any real significant remedy to corporate users. While the expense to McAfee would be far higher than the costs from their home users, it does mean that corporate customers may feel that they have been left out in the cold. Stories of hospital emergency rooms (as described in the comments on techielobang and Ars Technica's posts about the issue) and 911 call centers (as described by the ISC) going down in the wake of the DAT release mean that McAfee will likely face very upset customers as the full cost and impact of the issue are calculated.

McAfee president David DeWalt's open letter, posted to the McAfee Security Insights blog claims that "The vast majority of affected users were back up and running smoothly within hours, and we are continuing to work diligently until we are sure that every last user node among each and every one of our customers is back in action." - a claim that may be difficult to back up in large organizations, or for users whose only access to the Internet and email was taken down by the issue. In many cases, large organizations with a significant XP SP3 install base saw hundreds of systems taken down that required manual visits. Stories such as this comment posted in response to his letter tell a story of woe. "Our team of 10 technicians worked for over 24 hour straight to touch all 1200+ machines by hand in order to assure our patients safety and our continued operations."

For an organization, that's a cost that could reach into thousands, if not tens of thousands of dollars in staff time alone. Lost productivity for offline machines could total far more. What will McAfee offer to heavily affected corporate customers? We'll likely know in the next week or so. Until then, security professionals and IT support staff can only point their management to McAfee's apology, and decide how they can best update their business practices to avoid update issues without compromising quick response.

Perhaps more interestingly, as our industry starts to ponder whether whitelisting is the way to, we should consider what a bad update to a whitelist could do to our organizations. The risk model is much the same - so the question will remain. Do we trust our vendors QA processes and update release methodologies?

What The Theft Of Google's Gaia Code Means To You

The New York Times recently reported that the attacks against Google late last year netted the source code to Google's single sign on systems, Gaia. The danger that the Times mentions and then dismisses - that the hackers would insert a back door into Gaia - truly is unlikely. Instead, the greater danger is that exposed source code will allow a deeper analysis of potential flaws in the code. If the multi-site single sign on code does have flaws that can be leveraged, this could mean that the advantages of Google's spreading multi-site single sign on are also a major security hole.

The most interesting part of the article is how targeted the attacks were:

"the intruders seemed to have precise intelligence about the names of the Gaia software developers, and they first tried to access their work computers and then used a set of sophisticated techniques to gain access to the repositories where the source code for the program was stored."

The level of sophistication of the attacks, and the targeting of Google's source code may point to a far more intelligent and dangerous attack than we in the security industry are used to, even with the advanced persistent threats that are becoming more common.

The question: What comes next?

McAfee's DAT Debacle: When Your Security Software Causes Harm

On Wednesday, April 21st, McAfee's 5958 DAT file (McAfee's virus definitions file) was released with a bad detect for the w32/wecorl.a virus. This inadvertently detected svchost.exe on Windows XP machines as an infected process, and VirusScan took its normal actions - either quarantine or deletion in most cases. Per McAfee's description of the problem, this was a heuristics issue that only showed when deployed - which should make you wonder about how their testing is done.

At 12:05 PM, EST, McAfee sent out an urgent email stating:

"McAfee is aware of a w32/wecorl.a false positive with the 5958 DAT file April 21 at 2:00pm (GMT +1). McAfee advises NOT to download this DAT. Please disable pull tasks and update tasks.

Information updates will be sent every 90 minutes to keep you advised."

Administrators who quickly pulled the DAT using ePO were somewhat protected, as only those systems that had checked in since the DAT release were effected. Those users who did not update while the bad DAT was available were similarly safe. In many organizations that use McAfee's e-Policy Orchestrator (ePO) management tool, this was reasonably easy. For those that do direct updates, this would not have been as simple, and in either case, many systems did update with the bad DAT during the time it was available.

For enterprise users, this meant that any machine that received the DAT before McAfee's 12:05 EST email were likely taken offline. If the system rebooted, it would typically no longer have a working network connection, and thus could not be remotely repaired. Symptoms included blue screens and DCOM errors, as well as shutdown messages.

Less than 90 minutes later - as promised, McAfee sent out a second email with more detail, citing XP Service Pack 3 systems as problems, and noting that the DAT had been removed. In addition, it provided links to McAfee's knowledgebase which, unfortunately, was already starting to perform poorly under the load.

An extra.dat (McAfee's name for off-cycle, non-mainstream update files) was made available shortly after this, with a general email via SNS coming out just before 4 PM EST. The email provided more detail on an issues page for the DAT.

At 8:40 PM, EST, McAfee published both a recommended and an alternate remediation procedure for the DAT created issues. These procedures were included on the issues page, making it an easy central reference.

In total, less than 9 hours had elapsed. A total number of affected machines worldwide is unlikely to be released, but my own experience indicates that the number is likely quite sizable. For most organizations, today has been a day of remediation, often by hand, as machines that rebooted were unable to be accessed remotely with their networking broken.

One of the biggest lessons learned is that McAfee's reasonably new SNS notification service is a must-subscribe for McAfee users and admins. Another is that ePO can greatly increase your chances of getting in front of a widespread update issue given sufficient notice.

For McAfee, the lessons will not be easily forgotten - better testing, amongst other practices is likely to be on their list. It is equally obvious that McAfee did learn a lesson since the infamous DAT that detected Microsoft office files as infected, and their 90 minute notifications and quick response clearly show that.

Will this change how your organization views antivirus updates? Is immediate deployment worth the danger? I'm sure I will be having serious discussions with local PC support staff about the relative risks of a 24 hour delay against the possibility of a bad DAT - and I'm not sure I have as concrete of an answer as I would have a week ago, particularly in light of the increasingly high percentage of malware that evades mainstream AV.

SteadyState - Not Available for Windows 7

According to Windows Secrets, Microsoft has opted not to port its Windows SteadyState lockdown and security package to Windows 7. The free tool itself is used in many libraries, as well as in computer labs and for public access kiosks.

For those who use SteadyState, a number of alternatives exist, with Deep Freeze being the most popular in my experience. The cost of Microsoft discontinuing SteadyState may be a relatively small incremental cost for smaller institutions - according to Faronics website, a $30 or so per system cost, but larger scale licensing can have a real impact on non-profits and public institutions like those libraries which are already fighting tight budgets.

(note: edited 4/9/2010 - Microsoft has opted NOT to port SteadyState).

The (ISC)2 Takes the CSSLP To Computer Based Testing

The (ISC)² has joined the ranks of security certification organizations in allowing computer based testing for its exams. The CSSLP is the first, as described by the (ISC)² in the email quoted below:

"(ISC)² today announced the availability of computer-based exams for its Certified Secure Software Lifecycle Professional (CSSLP) credential. The CSSLP is the first (ISC)² certification exam to make the transition from paper-and-pencil delivery. Computer-based testing for (ISC)² 's other credential exams will be phased in over the next three years."

Moving the CSSLP and other certification tests from a large scale pencil and paper exam to an online format isn't a huge change, but should make the testing process more approachable for many. Why it will take three years is a better question - this should actually make the tests easier to maintain, and it should mean that the (ISC)² 's exam creation process and question randomization is far simpler.