The Department of Transportation is seeking to levy nearly $1 million in fines against Colonial Pipeline for a series of safety violations related to its operations at seven different locations dating back to 2017. Among the violations: an internal planning and communications plan that led to the company’s decision to temporarily shut down gas operations in the wake of the May 2021 DarkSide ransomware attack.
The action, taken by the Pipeline and Hazardous Materials Safety Administration (PHMSA) and contained in a Notice of Probable Violation and Proposed Compliance order issued May 5, lays out a number of violations of U.S. safety regulations the agency discovered through inspections and site visits between January and November 2020.
“The 2021 Colonial Pipeline incident reminds us all that meeting regulatory standards designed to mitigate risk to the public is an imperative,” said PHMSA Deputy Administrator Tristan Brown. “PHMSA holds companies accountable for violations and aims to prevent any instances of non-compliance.”
It specifically blames the company for failing to correct a number of known safety violations, including one that it says left executives unprepared to keep pipeline operations running in the wake of a cyberattack.
“Respondent’s failure to test and verify its internal communication plan contributed to consequences that occurred when, on May 7, 2021, Colonial Pipeline was the victim of a cyberattack which required the immediate shutdown of the entire pipeline system,” said the order, signed by Gregory A. Ochs, Central Region director for the Office of Pipeline Safety at PHMSA, in a May 5 letter to Colonial Pipeline CEO Joseph Blount.
This failure, first highlighted by regulators more than year before the company’s IT network was shut down by ransomware criminals in May 2021, underscores some of the gaps in planning that executives faced in the fallout of a cyber attack that threatened their operations.
The DarkSide ransomware attack on Colonial Pipeline never touched the company’s operational technology, Rather, the malware infected and shut down the company’s business IT network, and that in turn impacted the ability for employees to communicate and coordinate in the wake of the attack.
The company had a plan in place for undergoing a controlled shut down of pipeline operations in the event of a loss of SCADA or voice communications control, but federal regulations regulations specifically require companies to have and test a plan for resuming operations manually in those conditions.
Colonial didn’t do that. In fact, regulators say that “for all practical purposes,” shutting down operations in this contingency was Colonial’s plan for dealing with a loss of internal communications or SCADA control. Their last status update to regulators, provided on July 20, 2020, states that “due to the complexity of [Colonial’s] operations system and rarity of such events, [we] does not have a specific internal communication plan for manual operation and will not operate a line or system manually without prior implementation of an internal communication plan.”
In a statement, a Colonial Pipeline spokesperson said the notice was "the first step in a multi-step regulatory process and we look forward to engaging with PHMSA to resolve these matters." They also defended the contingency planning in the wake of the ransomware attack, saying it was "necessary" and tailored to the the company's operating environment.
"As the 2021 cybersecurity incident demonstrated, Colonial’s approach to operating manually gives us the flexibility and structure necessary to ensure continued safe operations as we adapt to unplanned events," the statement reads. "Our incident command structure facilitates a deliberate approach when responding to events. Our coordination with government stakeholders was timely, efficient and effective as evidenced by our ability to quickly restart the pipeline in a safe manner five days after we were attacked – which followed localized manual operations conducted before the official restart."
Sending a message
It represents a hefty proposed fine and regulatory action for Transportation. The alleged violation for failing to test and vet an internal comms plan makes up $846,300 of the total $986,400 in proposed fines to the company. For comparison, the fines levied for the other four violations — which weren’t directly tied to the Colonial Pipeline attack — were all for $45,000 or less.
Danielle Jablanski, an OT Cybersecurity Strategist at Nozomi Networks, noted that the use of non-cyber policies to cover cyber incidents is common across some commercial and financial sectors, so “it’s not a stretch to see non-cyber contingency plans during a crisis come under increased speculation after a shutdown.” But she cautioned that regulatory actions like these cannot be doled out to critical infrastructure on a one-size-fits-all basis and must take into account the specific realities that come with each incident and victim of ransomware.
“Proposed actions, however sweeping, have to grapple with the fact that every incident will be unique to the target and their security policies, preparedness, and resilience. Colonial Pipeline stakeholders know their operations better than any spectators, and they know exactly how purpose-built their operations and contingency plans are to their understanding of their operating environments,” Jablanski said. “This example teaches other critical infrastructure entities that their contingency plans need to take cyber incidents into account, and that they must play out those contingencies and potential impacts on employees, customers, citizens, and suppliers.”
Padraic O’Reilly, co-founder of CyberSaint, which works with critical infrastructure firms on cybersecurity, said the decision by Colonial Pipeline executives to shut down their operations likely stemmed from a multitude of reasons, not the least of which was uncertainty around whether the infection could have eventually impacted operational systems.
“The service was disrupted to some extent because Colonial did not know the possible extent of a ransomware jump across into OT,” said O’Reilly in a message to SC Media. “In terms of lessons—proper contingency planning, incident response, and completely understanding the networks and making sure that inventories are updated and logged in systems of record” are things organizations can do to avoid a similar fate.
It marks another instance of a federal agency leveraging their existing regulatory laws and authorities to capture concerns about inadequate cybersecurity. Recently the Department of Justice has stood up a civil cyber fraud initiative that seeks to apply the False Claims Act to fine or sue federal contractors who misrepresent their cybersecurity to the government. The Securities and Exchange Commission has also moved in recent months to incorporate a number of new cybersecurity requirements for publicly-traded companies and investment firms.
“The pipeline shutdown impacted numerous refineries’ ability to move refined product, and supply shortages created wide-spread societal impacts long after the restart. Since Respondent had not tested and verified an internal communication plan when the cyber-attack occurred, as was required by the regulation, Respondent was not prepared for manual restart and manual operation of its pipeline,” Ochs wrote. “Colonial Pipeline’s ad-hoc approach toward consideration of a “manual restart” created the potential for increased risks to the pipeline’s integrity as well as additional delays in restart, exacerbating the supply issues and societal impacts.
Jonathan Reiber, former chief strategy officer for cyber policy in the Office of the U.S. Secretary of Defense during the Obama administration, said Colonial Pipeline’s lack of a viable communications plan to manually operate their facilities points to a larger breakdown in effective security processes. Most of time, these kinds of process breakdowns are discovered in one of two ways: through regular and rigorous testing by the company or after the fact in the midst of a breach investigation. He said that while he is not a lawyer, he found the rationale provided in the order to be “persuasive.”
“The purpose here is to glean lesson from the past to make sure that things are better in the future. This is a way of saying ‘your process wasn’t working. You didn’t do what you’re supposed to do and we’re going to hold you to account for that.’ That’s really what the law is there for, right?” said Reiber, now vice president for cybersecurity strategy and Policy for AttackIQ.
