The temperature measured about 30 degrees Fahrenheit – with passing clouds, snow showers and a light breeze out of the west – just before midnight in Kiev, Ukraine, when the power went out.
At 11:53 p.m. local time on Dec. 17, 2016, remote terminal units (RTUs) used to monitor and control circuit breakers in Pivnichna (North) electrical substation went offline unexpectedly.
The North substation, a 330-kV transmission stepdown located just outside the city, serves Kiev's power distribution system. If the substation experienced a service disruption, it could cause cascading blackouts throughout the city and potentially beyond.
As it happened, the RTUs' failure resulted in power losses in the northern right bank section of Kiev. By 1:05 a.m. on Dec. 18, the power was restored.
Ukrenergo's engineers were perplexed by the potential cause of the incident. Engineers might assume equipment failure had this happened anywhere else in the world. But this is Ukraine.
The Dec. 17-18 incident eerily resembled another from one year before – nearly to the day.
That incident, which affected three regional electricity companies on Dec. 23, 2015, resulted in power outages to 225,000 customers and was the first known time a cyberattack caused blackouts in a country's power grid.
Ukrenergo's engineers had to consider equipment failure but also the possibility of another cyberattack.
A Tale of Two Cyberattacks
Investigators and journalists have reported on key similarities and differences between the 2015 and 2016 cyberattacks, which include:
- Similarity: BlackEnergy malware and KillDisk, a data deletion program, were used in both attacks. Importantly, experts said these tools did not cause the power outages in 2015 because they lack the necessary functionality. Hackers direct interaction with control systems caused the blackouts.
- Difference: In 2015, hackers attacked multiple distribution substations (seven 110 kV and twenty-three 35 kV). In 2016, hackers attacked a single transmission substation.
- Similarity: In both incidents, hackers targeted substation components called RTUs.
- Difference: Booz Allen Hamilton researchers said the 2015 hackers used malicious firmware to permanently disable the RTUs after opening breakers and then used KillDisk to damage operators terminals, which prevented remote repair of the RTUs. In 2016, the hackers merely deactivated the RTUs, which made restoration easier, investigators said.
Common Attack Vector: Remote Terminal Units
RTUs are electronic devices that link sensors and actuators on physical objects to industrial control systems (ICS) and supervisory control and data acquisition (SCADA) systems. ICS/SCADA systems allow engineers in centralized locations to monitor and control distributed assets. RTUs, which are similar to programmable logic controllers (PLCs), are key components of ICS/SCADA systems.
Dale Peterson, Founder and CEO of Digital Bond, an ICS/SCADA cybersecurity consulting firm, explained, "A RTU passes commands from an operator in a control room to numerous actuators that perform control functions and sensors that monitor system status at a physical site, such as a substation. Many RTUs today can run programs or logic, so the difference between a RTU and PLC can be minimal."
RTU and PLC use is common and even necessary in almost any control system, from the power grid and public transit to building automation systems. This is significant, Peterson said, because, "Until recently, just the past year, these devices were 'insecure by design.' We used to say, 'access equals control.' If you had access [to the RTU], you didn't have to hack it by exploiting a vulnerability. You could do anything you wanted using documented features and functions."
Peterson said more secure RTUs are coming to market and he's watching whether companies will upgrade old devices.
"Ukraine 2015 highlighted the easiest way to compromise these systems: Vulnerability from remote access," Peterson said. "Once you're on the system, attackers are limited only by their engineering and automation capability."
Anatomy of an Attack
Over the past year, security researchers have pieced together how the 2015 incident unfolded. Initial research on the 2016 incident provides fresh information. It's unknown if the same hackers carried out both cyberattacks. Using public reports on the cyberattacks, it's possible to reconstruct how hackers likely worked.
Just as many professionals employ "best practices," so too do elite hackers. Experts call this the cyber kill chain. Michael Assante and Robert M. Lee of the SysAdmin, Audit, Networking and Security (SANS) Institute have developed the ICS Kill Chain.
The following narrative, which uses the ICS Kill Chain as a framework and incorporates facts and findings currently known about the 2015 and 2016 attacks, provides a hypothetical account.
- Planning Oleksii Yasynskyi, head of ISSP Labs and an investigator hired by Ukrenergo, told the BBC that multiple groups were involved in the 2016 attack. The hackers first step would have been to gather information about Ukrenergo. Information of interest falls into several categories: People: Hackers would have identified employees who hold key positions within and knowledge about Ukrenergo, such as system administrators and engineers. Hackers also would have identified target victims. Detailed information could have been gathered online, such as from social media sites. Enterprise IT environment: Hackers would have gathered information such as devices, operating systems and business applications used throughout Ukrenergo, which can be gleaned by, for instance, looking at employee LinkedIn job descriptions and skill sets. Particularly valuable would be IT credentials, account details, weak passwords and organizational processes/procedures. Hackers could have found some of this accidentally published online or it could have been guessed or cracked with relative ease. Critical infrastructure environment: Hackers could have started learning about their ultimate targets by using the Shodan search engine to find Ukrenergo infrastructure exposed to the public internet. Vendors technical manuals and marketing collateral are valuable. Detailed information on the RTUs and ICS targeted in the 2015 attack was readily available online, according to SANS.
- Preparation Preparing can include weaponization and targeting.Details on these aspects of the 2016 attack are still sparse. In 2015, hackers weaponized Microsoft Word files with malicious macros. Hackers probably customized attacks to targeted victims, such as recipients of spear-phishing emails.
- Cyber Intrusion Next, the hackers would have gained access to Ukrenergos network.Investigators have not yet revealed the initial attack vector for the 2016 incident. Yasynskyi told the BBC the 2015 and 2016 cyberattacks are not much different, except the 2016 hack was more complex and better organized. Marina Krotofil, lead cybersecurity researcher at Honeywell, and Yasynskyi presented initial findings of the 2016 investigation at the S4x17 Conference on Jan. 10. They said the hackers used clever coding techniques to obfuscate methods and evade signature-based attack detection. The hackers also used macros to detect security technologies in Ukrenergos environment, including intrusion prevention systems and sandboxes. The 2015 incident began with a spear-phishing email. Notably, security researchers wrote that no custom exploit code was used in the 2015 intrusions. Hackers gained access using native functionality in Microsoft Word (i.e., macros) to download BlackEnergy 3.
- Management and Enablement Again, details on the 2016 attack are not yet public, but investigators said the tools and methods were similar to 2015.If the 2016 attack was conducted like the 2015 attack, BlackEnergy malware downloaded to a victims computer via spear phishing would have called back to the hackers remote command and control (C2) infrastructure. The C2 infrastructure would have allowed hackers to extract data from Ukrenergos network, download additional tools onto Ukrenergos network and issue commands remotely to compromised Ukrenergo systems.
- Sustainment, Entrenchment, Development and Execution With C2 established, hackers most likely would have created multiple back doors into Ukrenergos network to allow ongoing access.Next, hackers would have begun to harvest administrator credentials for Ukrenergos IT infrastructure. In the 2015 attack, hackers installed additional BlackEnergy plugins. This would have allowed hackers to map Ukrenergos internal networks and to move laterally across systems and subnets. Failure to catch hackers early in an intrusion can be costly. According to a 2016 report by cybersecurity firm Mandiant, hackers remained on victim networks in 2015 a median of 146 days prior to discovery. No one immediately noticed the hackers network presence in the 2015 or 2016 Ukraine attacks. In the 2015 attack, hackers spent months exploring IT environments, eventually finding and accessing the virtual private network, which lacked two-factor authentication and along with a misconfigured firewall, allowed hackers entry into and ongoing access to the ICS/SCADA system network. Investigators told Motherboard the 2016 hackers remained on Ukrenergos network for months, gathering system logs, monitoring network traffic and studying the behavior of system administrators. In both incidents, hackers skillfully hid their network presence. Investigators said the 2016 hackers lived off the land, a technique whereby intruders use the same credentials and tools as system administrators to avoid detection. In the S4x17 Conference presentation, Krotofil stressed the importance of early detection, warning, It is critical to detect malicious invasion at early stages. Discovering KillDisk in your network is already too late. The attackers already hav[e] a very reliable, distributed foothold in your network. Cleanup/eradication is almost impossible.
Were These Cyberattacks Merely a Prelude?
Researchers investigating the 2016 attack have alluded to an ominous trend, suggesting it was merely "training."
In early January 2017, as investigators were piecing together the 2016 incident, Russian cybersecurity firm Kaspersky Lab ICS-CERT reported a then-ongoing "targeted attack" against 500 organizations in 50 countries. Kaspersky wrote:The worst affected were companies in the smelting, electric power generation and transmission, construction and engineering industries. Most of the organizations attacked were vendors of industrial automation solutions and system support contractors. In other words, the attack targeted organizations that design, build and support industrial solutions for critical infrastructure.
Investigators said the cyberattack against Ukrenergo began in summer 2016. Kaspersky said the spear-phishing campaign began in August 2016. To date, no publicly available evidence links the campaigns, threat actors or attack signatures.
According to cybersecurity firm FireEye, as of mid-2016, 33 percent of 1,552 publicly disclosed ICS vulnerabilities had no available security patch.