At the CyberRisk Leadership Exchange in Cincinnati on June 7, the chief security officer of an Ohio bottling company used his lunchtime keynote address to recount how his company's eight-person IT team detected, remediated and recovered from a ransomware attack — within the space of seven hours, without losing any business and without paying a dime to the attackers.
"We never missed an order. We never missed a delivery. Customer data was not compromised," Brian Balzer, Executive VP of Digital Technology & Business Transformation at G&J Pepsi-Cola Drink Bottlers, Inc., told SC Media in an interview. "I'd say probably 95% of the organization had no idea that we were under attack."
Balzer credits G&J's rapid, successful recovery from the ransomware attack to diligent preparation and a move to cloud-based operations, and to strong support from both colleagues and company leaders.
"I cannot stress enough to companies the importance of having such a strong culture where people are willing to jump in and help one another," says Balzer, "and a leadership team that is supportive of the cyber team, your IT team, whoever it might be, your digital team, to be able to put plans in motion."
Anatomy of an attack
Founded in 1925, G&J Pepsi serves Ohio, Kentucky and West Virginia and is the largest family-owned independent Pepsi bottler in the U.S., with more than a dozen facilities, 2,000 employees and $650 million in annual revenue.
The company first noticed something was wrong with its systems just before the Labor Day weekend of 2021.
"We had suspected that we might have allowed an intruder, a hacker into our environment," Balzer says. "We spent the better part of about four or five days trying to understand where they got in, where they were, if they were in and how we might be potentially exposed."
The G&J Pepsi team suspected that someone had used Cobalt Strike to install "beacons," or backdoors, into the systems, but their initial searches found nothing. Then a few days after Labor Day, a call came in around 4:30 in the morning.
"We got a call from one of the folks in our plant saying, 'Hey, something's weird, I can't access files,'" recounts Balzer. "And we knew instantly that we were under attack."
The G&J Pepsi team quickly took as many systems offline as it could. Balzer credits support from the very top of the company for that.
"I [had] to call my CEO at five o'clock in the morning and say, 'We're literally bringing all the systems down.'" Balzer says. "He's like, 'All right, I trust you. You just keep me posted.'"
Balzer's team found two potential points of entry. The first was a user who had unknowingly downloaded a corrupted file — a common vector for ransomware infection — but G&J Pepsi's endpoint solution quickly detected and remediated that.
The second point of entry was more serious. Just before the long weekend, Microsoft had released a patch for Exchange Server. But it looked very similar to another Exchange Server patch from two weeks earlier, one that G&J Pepsi had already implemented.
"There was some confusion as to, 'Was this the same patch that they released? Or was this a different patch?'" Balzer recalls. "We just misunderstood. It was probably on us. It was our fault for not getting that clarification quickly."
Instead of implementing the new patch right away, G&J Pepsi decided to wait until the following weekend. That's all the time the ransomware crew — identified as Conti by notes left on infected systems — needed.
"Within four days, they had exploited that particular gap in that Exchange server, and were able to compromise our environment," Balzer says. But, he added, "we don't deal with terrorists."
As a midsize company, G&J Pepsi fit the profile of a prime target for ransomware crews. The fact that the attack happened over a three-day weekend, which gives attackers more time to operate freely, was likely no coincidence.
"Most midsize companies and small companies can't thwart an attack, particularly a Conti ransomware or other sophisticated attacks that are that are taking place," Balzer says. "When we called for support, we [were told] that 'We'll try to help you, but we are absolutely slammed coming out of this three-day weekend because they went haywire on companies across the U.S.'"
Rapid remediation
Fortunately, because G&J Pepsi had already moved all its systems to the cloud, shutting down company assets and stopping the attackers was less complicated than it might have been for on-premises infrastructure.
"We have nothing on-premise," Balzer says. "Because we're 100% in the cloud, and because we utilize Microsoft Azure Cloud environment, we were able to prevent them from moving laterally across the platforms in our systems."
The virtual nature of G&J Pepsi's systems meant that the company was able to spend the next few hours using its weekly backups to spin up brand-new Azure instances free of ransomware, even as the team continued to investigate the infected systems.
"Within seven hours, we were able to stand up the entire environment again," Balzer told us. "Many of our solutions are SaaS solutions. The things that were impacted were more like file servers — we had a couple of other servers that we had developed as IAS solutions in Azure that were at risk. We were able to basically rebuild and recreate that environment."
G&J Pepsi were lucky. None of the company's backups had been affected by the ransomware, and dark-web scans turned up no evidence of company data having been stolen.
"We were very fortunate that we had eyes on it immediately and were able to basically isolate and wall them off and then rebuild our environment," Balzer says.
However, staff PCs left on overnight in the office were infected, as were some ancillary servers. Rebuilding those took a bit more time.
Following the attack, G&J Pepsi brought on Arctic Wolf as a managed detection and response (MDR) provider and changed several company policies.
"We forced all password resets, we changed our policies on backups, we changed our policies on how many admin accounts that we have — we limited those — and really revamped the security," Balzer says.
Balzer told us that G&J Pepsi had also locked down USB ports on PCs, beefed up identity and access management and automated its systems patching. As it is a U.S.-only company, G&J Pepsi also blocked all system access from outside the country. The company has not had any serious incidents since.
Lessons for the future
In a separate interview with Microsoft, G&J Pepsi Enterprise Infrastructure Director Eric McKinney says he has learned two things from the company's brush with ransomware.
"If I could go back in time to the months leading up to our ransomware attack, I'd tell myself to strengthen our endpoint policies," McKinney tells Microsoft. "I don't view our recovery as a victory so much as a call to double down on security."
For McKinney, the second lesson was how much there is to be gained from a full cloud migration.
"G&J Pepsi has gotten a wide range of security benefits, such as platform-based backups, cloud-based identities, and multifactor authentication, leveraging native tools that help recommend and identify risk," McKinney says. "It doesn't matter whether you're a huge corporation like PepsiCo, a midsize business like G&J Pepsi, or a mom-and-pop gas station down the road — I would make that move to the cloud and make it quickly."
Fielding questions from the audience following his keynote address at the Cincinnati CyberRisk Leadership Exchange, Balzer was struck by how many of his fellow cybersecurity executives wanted to hear about G&J Pepsi's experience.
"I love that the participation was there, that the curiosity was there," he told us. "People wanted to understand what was happening so that they can be aware of what to do if that ever occurs with them."
But Balzer once more stressed how important company culture is to an organization's ability to maintain resilience and quickly recover from an attack.
"The other thing that really stuck out, that we talked about for a brief bit during that [CyberRisk Leadership Exchange] session," Balzer says, "was the importance of having the right culture within your team to be able to come together to thwart an attack, particularly one of that size or even larger.
"We had a plan in place. Unfortunately, we had to use it. But fortunately for us, that plan worked," adds Balzer. "And that worked because we had the right leadership, the most senior leadership and support, and we had the right culture within our team to help support that and thwart that attack."
The next Cybersecurity Collaboration Forum event in Cincinnati will be a CyberRisk CISO Dinner at the end of September.
For more information on the Cybersecurity Collaboration Forum, including how to attend a CyberRisk Leadership event in your area, please visit https://www.cybersecuritycollaboration.com/.
Many thanks to Zack Dethlefs of the Cybersecurity Collaboration Forum.
