There are many different names for the second Tuesday of every month: Patch Tuesday, Super Tuesday, Black Tuesday — and maybe even some other unsavory nicknames not suitable for print. This day, when Microsoft rolls out security updates, is the fulcrum around which most organizations' whole patch management cycles revolve. But just as there are different nicknames for the day, there are also differing opinions about how it should be handled and how quickly organizations should respond with changes.
According to Rich Bentley, market segment manager for Altiris, it is a choice between speed and taking due care.
"From my experience, it falls into two general categories," Bentley says. "There's the ‘let's get it out as soon as we can and let's patch everything approach.' And then there's the ‘let's go slow, and take a measured approach and prioritize and test before anything goes out.'"
We examine three organizations, one large, one medium and one small, to see how they attempt to strike the right balance when that special Tuesday rolls around each month.
KURT HEITMAN
Position: System support manager, Maintech, 5,000 systems
Shortest patch cycle: One week
Longest patch cycle: Three weeks
Kurt Heitman, system support manager with Maintech, says that his organization tries to reasonably straddle the ground that Bentley mentioned between speed of patch delivery and taking caution against unexpected problems.
"We're somewhere in the middle," he says of the Wallington, N.J.-based company, a market leader in delivering custom-tailored information technology service solutions.
As a public company with over 5,000 users and 100 servers to contend with, Maintech has presented Heitman with myriad challenges when he was tasked several years ago with developing the company's patch management procedure.
"It is a pretty significant effort, and not only from a security perspective. You have to make sure it is Sarbanes-Oxley compliant — at least we do, we're publicly traded," says Heitman. "Environmentally, you can have issues, and a lot depends on your company's philosophy. If you have a very hands-off, don't-touch-my-workstation philosophy, boy, have you got your work cut out for you."
Heitman speaks from experience. "Being a multi-divisional, strategic business unit-type organization it's been very difficult to get everyone to play the same game," he says. He explains that he was able to get everyone on board only through "sheer determination and bullheadedness."
When Patch Tuesday rolls around, the process is set into motion at Maintech with the presentation of all potential changes before two committees, one strategic and one tactical, both of which will look at the patches and determine whether or not they are going to apply them. The strategic group, the Change Management Committee, is comprised of IT representatives from all of the major enterprise divisions at the company. The patching team will seek the committee's "rubber stamp," to make sure the patches fit within the company's change management strategy. The tactical group, made up of security gurus and IT administrators, will discuss all of the technical minutiae surrounding the patches and develop a plan of attack.
"Typically speaking, unless there is a negative connotation with a patch or we know it's going to cause problems with an internal application upfront, we will apply almost all security patches. So from that perspective it is pretty straightforward," Heitman says.
Once the two groups have given their go-aheads, it is on to the quality assurance (QA) process. The change management team will roll out the patches in a test environment for one to two weeks to ferret out any potential problems.
"We have a fairly exhaustive process, and there have been instances where patches have caused concern. There have been issues we've had to roll back from, so we're reasonably cautious and rely quite heavily on our QA process to make sure an update is not going to cause too much difficulty," he says.
JIM CZYZEWSKI
Position: Senior information systems specialist, MidMichigan Health, 2000 systems
Shortest patch cycle: Two days
Longest patch cycle: 2.5 weeks
At MidMichigan Health, Midland, Mich., the process is a bit less arduous. Usually the fun begins the day after Patch Tuesday.
"We have a group meeting in the early afternoon to review the patches and make sure we don't see anything that may be a possible problem," says Jim Czyzewski, senior information systems specialist for MidMichigan. "If we do see a possible problem, we make sure that it gets a little more attention during our testing process."
Unless there are serious issues, the organization begins the testing process on Thursday, Czyzewski says. MidMichigan does not have a test network, so the testing is done in a live environment. Because of this, Czyzewski and his team must take a three-step approach. The first is to start on an isolated system to make sure "the machine doesn't blow up," he says. From there it is on to steps two and three.
"We start with a very small localized group and then we broaden it to make sure there's a representative [user] for each application that we have in the organization," he says.
Testers in both groups are hand-picked power users in each division of the organization that can act as Czyzewski's eyes and ears to make sure there aren't any issues with the changes.
"They are people that we feel are a little more computer savvy and a little more aware of anything that might be out of the ordinary that a regular user might not notice," he says. Each group gets about a week to test before he rolls out the patches organization-wide.
While the thought of live testing may make some security professionals queasy, the ad hoc testing procedure does allow Czyzewski the flexibility to respond quickly to extremely critical patches.
"That process actually is during a normal patch cycle for something that there are no known threats for at the time," he says. "When there are highly critical patches with threats we are at-risk for in house, we can push up the process a bit."
For extreme risks, Czyzewski and his team can speed up testing turnaround to 24 hours. "We basically tell our test users to hammer at it as quick as possible, make it known that this patch is going out, and make sure apps work," he says.
That one-day turnaround is only for the highest-risk vulnerabilities. For medium-risk patches the turnaround is usually a week. Everything else takes approximately two-and-a-half weeks of testing, he says.
"We're actually laid back now that we have a product that provides us accurate patching and reporting. We feel comfortable with the process," he says.
GABRIEL SELMI
Position: Network administrator, Advanced Behavioral Health, Inc., 200 users
Shortest patching cycle: One week
Longest patching cycle: One month
Advanced Behavioral Health, Middletown, Conn., is a small organization with only about 200 computer users. As such, network administrator Gabriel Selmi doesn't have to deal with the bureaucracy of committee rule. But simplicity does have its downside: he's also the sole IT staffer responsible for keeping up on threats and potential patching issues.
Selmi says he tries to take a proactive approach by keeping up with security and bug tracking bulletins and leveraging the assets he has. Because he doesn't have much in the way of IT resources for testing, he depends on intelligence gathered by his patch management vendor, Patch Link, to give him the first approval on patches to ensure there are no known issues.
"Basically, for Patch Tuesday I hold off on the patches until PatchLink says they are OK, and then at that point I'll throw it out on my test network," he says.
"If it doesn't alter anything immediately, then I'll push to certain PCs I have images of for backups," he says. "I'll usually go roughly for about a week of testing with 10 or 15 users in different areas and get a feel for how [the patches] react in different environments."
He says that the goal from start to finish is the full month because he likes to give himself a little leeway. "It gives us a nice cushion if we shoot for a month."
Because he doesn't have a lot of other people in IT, he prefers to be cautious, even with highly critical vulnerabilities. "My experience with some of these patches that come out immediately from Microsoft is that they're not 100 percent certain of what's going to happen within their own environment," he says. "We're a 365 by 24/7 operation and we don't have any major redundancies in place in the systems that we can activate. So I tend to take the cautionary path even if there's major risk."
While no organization will have the exact problems your company has, it sometimes pays to learn from practitioners like Selmi, Czyzewski and Heitman. Although each handles Patch Tuesday differently, there is one common theme to their approach. Each has planned out their patching procedures and works on a predetermined schedule to minimize having to make hasty decisions.
"Sometimes, the number of patches that come out are pretty incredible, and there's a tremendous amount of pain associated with all of that. But by and large, it is like everything else," says Heitman. "Once you get a procedure working, it is not so much of a problem."
GETTING INVOLVED:
Organized crime
"If you look at the big picture, we're seeing, and will continue to see, increased risk associated with IT and the internet," says Dan Blum, a senior vice president and research director at the Burton Group. "The simple reason is enterprises are putting more value into those mediums.
That's simply because criminals are following the greenbacks. "Why rob banks?" he asks. "Because that's where the money is. And as we put more value into [IT] resources, we'll get more attacks from cybercriminals, we'll get more industrial espionage, and more information warfare."
And, say other experts, more criminals working together to undertake their nefarious deeds. "Where we've seen a paradigm shift is in organized crime syndicates hiring other computer criminals to work with and for them," says Chris Pierson, the founder of the cybersecurity and cyberliability practice in the Phoenix, Ariz. office of law firm Lewis and Roca LLP. They're getting into bed with spammers, hackers, phishers, and identity thieves, he says, and as a result "we've seen an increase in the amount of profit from their activities."
— Jim Carr
PATCH TUESDAY:
A surprise from Microsoft
Microsoft released seven patches on December 12 for 11 vulnerabilities, including a surprise fix for two zero-day flaws in Windows Media Player.
The update also addresses a flawed WMI Object Broker ActiveX control in Visual Studio 2005. The vulnerability emerged days before the November patch release and was not addressed in that fix.
But perhaps the biggest news out of December's release was what wasn't fixed. The software giant did not push out patches for two Word vulnerabilities, one of which is being actively exploited in limited, targeted attacks.
"I would anticipate an out-of-band patch given the severity of these vulnerabilities and the tremendous use of Word in the business community," says Amol Sarwate, manager of the vulnerability labs at Qualys.
So far this year, Microsoft has issued two out-of-cycle patches. The surprise fix affects the ASX file format, processed by the Media Player. According to a blog post, attackers could create malformed ASX files to cause a buffer overflow resulting in remote code execution.
Microsoft was not planning to fix the flaws but decided on a patch after receiving reports of publicly available proof-of-concept code.
Meanwhile, Gunter Ollmann, X-Force director at IBM Internet Security Systems, said enterprises should pay particular attention to a fix for a simple network message protocol (SNMP) vulnerability labeled only as "important."
"Although SNMP is not a default service, it is the defacto standard for monitoring critical business assets," he said.
Michael Sutton, security evangelist at SPI Dynamics, told SCMagazine.com that the bugs addressed in December's bulletin underscore hackers' continued focus on client-side vulnerabilities that can be easily discovered through means such as fuzzing. Fuzzing is a method traditionally used by software developers to find faults in applications.
"Overall this year, we've had a tremendous amount of Office vulnerabilities," Sutton says. "You take a known, good file and start mangling pieces of it. There's a real shift in focus from server-side to client-side [vulnerabilities]."
"Is the perfect answer to block all the files?" Sutton says. "You can't. Imagine the internet with no sight or sound. It would be pretty boring."
— Dan Kaplan
