[Music]
Chain of events, cause and effect. We
analyse what went right, and what went
wrong, as we discover that many outcomes
can be predicted, planned for, and even
prevented. I'm John Chidgey, and this is
Causality. Causality is supported by you:
our listeners. If you'd like to support
the show you can do so in via Patreon or
by becoming a premium subscriber for
early release high quality ad-free
episodes. Visit https://engineered.network
/causality to learn how you can
help. Thank you. "Black Energy" On the 23rd
of December 2015 in the late evening
major sections of the Ukrainian power
grid blacked out in quick succession. The
blackouts impacted 225,000 residences
over a huge physical area, shedding some
170MW of load from the
electrical power grid. There were no
storms, no equipment failures, and no
human error. Well, no human error at that
moment in time anyway. Before you wonder
why a power blackout is worthy of
discussion, is that this particular power
outage was the first, large-scale power
disruption caused by a cyber intrusion.
The incident has been referred to as
"Black Energy" not to be confused with
"Industroyer" which happened 1 year later.
The name of the incident is linked to
the name of the malware that the
incident made well-known. Which
interestingly, was actually the 3rd
generation of the Black Energy malware
that was identified first in 2014. The
truth is that the Black Energy Three malware
wasn't the cause of the outages, but
rather it was the key that initially let
the attackers in. But we're getting ahead
of ourselves.
Ukraine has 24 regions with each region
varying from 11 to 20 seven provinces in
each, and covering these to various
scales, there is a different power
distribution company serving that region,
called Obelnergos.
The Black Energy incident was targeted
at multiple of those Obelnergos. At
approximately 3:25pm Eastern European
Time on Wednesday the 23rd of December,
2015 the call centers for reporting
power grid issues were flooded with a
barrage of telephone calls,
simultaneously. These calls appeared to
have all originated from Moscow,
overloading the telephone system
completely. Five minutes later, in the
Prykarpattya-oblenergo control center, the
SCADA remote control system seemingly
possessed, locked the operators out and
began executing a manual isolation
sequence. Thirty substations were
systematically switched off one by one
as massive segments of the power grid
were sent into a blackout. Specifically x7
110 kV (or kiloVolt) and x23 35 kV
substations were disconnected. They were
distribution substations, not major
transmission and not generation plants.
Approximately 1 minute later the same
attack began on a second Oblenergo,
and then a third at 4:00pm. Six
Oblenergos were targeted in the
attack though only three began to
experience an operational impact. And
they were Chernivtsi, Kyiv and the
aforementioned Prykarpattya. Two of
those three control rooms were also
blacked out themselves despite having
UPS's fitted with the third control room
remaining powered through out the
incident. At 4:10pm engineers disabled
a HMI administrator account in an
attempt to stop the attack and regain
control.
However the hackers had installed an
additional administrator account
beforehand and hence this action had no
impact. In the following minutes the
Obelnergo engineers initiated a shutdown of
the SCADA system and remote access VPN
and were thus able to end the attack,
however this occurred only in time to
prevent the final blacking out of just
one solitary sub-station
that remained. All of the others had
already been blacked out. At 5:00pm
the Prykarpattya-oblenergo published a
note on their site stating the following:
(this is a Russian translation to English)
"To the attention of consumers of PJSC,
due to malfunctioning of telemechanics,
a considerable part of the region and
Ivano-Frankivsk
remained without power supply.
Specialists are currently searching for
the causes and finding out the extent of
the accident. We will provide more
information as we receive it. Please
refrain from making phone calls to the
call center as the dispatchers do not
have any information about the timing of
the accident until the cause of the
accident is known. Thank you for
understanding."
(end quote) At 5:30pm an additional
note was also added. Each Oblenergo
attempted remotely re-energizing their
substations to no avail, and then
systematically dispatched field
operators to each substation one by one.
Upon arrival at each substation the
field operators systematically switched
on each of the circuit breakers powering
up sections of the power grid. The
shortest full blackout period lasted
approximately 1 hour, with the full
power restoration taking approximately
6 hours. In attempting to restore their
control system infrastructure, Oblenergo
engineers found most operator
workstations were unable to be booted up
and those that they could, they found
operators were unable to login with
their credentials and where they could
login successfully, many of their remote
sites remained unable to be monitored
let alone controlled remotely. The attack
had rendered them remotely blind and
completely remotely incapacitated. So how
did they get in? First of all it, started
with a spear phishing attack focused on
IT staff and sys-admins,
(that's system administrators) that worked for
several companies throughout the Ukraine
that supported the Ukraine electricity
network. They delivered an Email with a
malicious attachment that appeared to
have come from Ukraine
energy ministry. 6 of the Ukraine power
companies were targeted with an
old-fashioned Word or Excel document
with a macro embedded in them, that once
the user opened and enabled macros,
delivered the Black Energy 3 payload. In
this case the attachments were
predominantly Excel files. Black Energy 3
then allowed the installation of key
loggers to capture credentials, allowing
further system access laterally until
reaching the domain controllers. This
began more than 6 months before the
actual attack would take place. None of
the companies had any capability to
continually monitor their control system
network logs. The Oblenergos actually
had significant logging enabled on most
of their systems, server and network
devices, however it was not centrally
aggregated. Moreover none of it was
checked manually or semi-automatically.
There was also no system network
monitoring at all, with no tools
installed for that purpose.
In addition, remote access to the control
network was via VPN and that VPN had no
multi-factor authentication. The
Ukrainian power grid did have remote
access for engineers and with the
intention of being able to access
monitor and remotely support the system,
but it did not have MFA (or Multi-Factor
Authentication). For those that don't know
what MFA is the intention is simple
enough: log in to a secure site or an
access portal with a username and a
password. The system will then send a
notification, a message, SMS, text or
similar, to a secondary device that is
known to be owned by the user, to confirm
that that specific login attempt was
coming from that specific user. Once that
user independently approves or accepts,
access is finally granted. MFA, were it in
place, would have alerted those with a
compromised account that someone was
attempting to remotely log in using
their credentials. How the hackers got
the right credentials?
Each company segregated their control
network from their corporate network
using a firewall. So the hackers spent
months incrementally infecting more
machines on the corporate side with a
variety of different exploits until they
were able to get access to the active
directory controller. Once in the AD
controller they had access to the domain
credentials, they obtained credentials for
control systems engineers with remote
access privileges and were able to
access the control network at will
without detection. They compromised a
total of 17 operator work stations
across the organization's. They installed
a variety of back doors including GCat,
DropBear and Kryptik, to ensure that
there were multiple paths available to
get in, should one of them be discovered.
So how they set up the outage itself, now
that they're in? First of all the control
room UPS's. All the control centers in
the Ukraine as is standard practice
worldwide, are fully backed up with
uninterruptible power supplies, such that
if the grid power goes out the control
room will remain powered and functional.
The same applies to the dial-in call
centers. In the first step the hackers
took out 2 of the 3 UPS's such
that when the power was turned off to
their own power grids those control
rooms would also be blacked out. Then
they disabled remote SCADA commands.
Serial to Ethernet converters are used
for address translation between the
SCADA system and the high-voltage
circuit breakers and that was remotely
upgraded to an invalid firmware revision,
awaiting a remote reboot request, after
which the device, being unable to boot
because the firmware was invalid, was
effectively bricked and completely
inoperable. The 2 devices that they
targeted were the MOXA UC7408 LX-plus:
that is a Universal Communicator with an
IXP425. It has 8
configurable RS-232, -422 or -485 ports,
8 Digital Ins, 8 Digital Outs, and
a dual LAN. Importantly it's running an
embedded Linux version. MOXA is a company
based in Taiwan and they make a large
number of industrial communications
products. There is scarcely a plant that
I've worked on, that does not have
something made by MOXA, somewhere in it.
The 2nd device was an iRZ RUH2
3G. This is a 3G router, industrially
hardened. It supports dual SIM cards and
10/100 Ethernet and again runs an
embedded Linux version. iRZ is a
company based in Russia.
So whilst they are popular in Europe,
outside of Europe they are rarely found.
By disabling these with these firmware
hacks, this meant that any recovery of
SCADA workstations would be ineffective
because there would be no communication
converter between SCADA and the circuit
breakers for remote operation or
monitoring. Essentially they were cutting
off the intermediary and severing that
link. Disabling the SCADA workstations.
KillDisk on the operator workstations
will overwrite the MBR (that's the Master
Boot Record) as well as key system files
rendering the computer inoperable and
unable to reboot to recover. Effectively
disabling the SCADA operator
workstations completely. For several
machines KillDisk was manually
triggered, by whereas for others there
was a time trigger delay to launch at
about 5:00pm, an hour and a half into
the start of the attacks.
Finally they disabled an RTU on an
embedded PC. An ABB RTU560 CMU-02 is a
chassis mounted daughter card that runs
an embedded Windows CE.
One of these, and only one was discovered
also corrupted by KillDisk. There been
many suggestions as to how this could
have been prevented, but before we talk
about that a few things that wouldn't
have stopped it. Application whitelisting,
for example at the firewall would not
have changed the outcome. Non-obvious
passwords were actually used so that was
not a case of having weak passwords that
were easily guessed. They weren't.
Significant logging did actually exist,
and firewalls existed.
Adding more firewalls would not have
changed the outcome. Before we talk about
what could have been done to prevent
this, let's look at the fallout from the
incident. In total about 73MWh
of electricity was not supplied
during the blackout period. That
represents a somewhat
insignificant amount, representing a
mere 0.015% of the Ukraine's daily
electricity consumption. Fractionally
speaking that's 1/6,600th
of their total daily supply.
There were no documented economic
impacts, health or medical complications
or spoilage losses as a result of this
blackout incident. The Obelnergos
themselves however, they were more
heavily impacted in terms of cost to
rectify the mess left behind by the
hackers. It took months to rebuild the
destroyed work stations due to a lack of
regular backups. It took months to
replace Serial to Ethernet converters
due to a lack of spare parts and a lack
of backups with the MOXA and IRZ units
completely unrecoverable, requiring
physical replacement and complete
reprogramming from scratch. During the
rebuilding time the system was mostly
run manually, with a significantly
increased staff presence on the ground
at substations. The additional costs of
this rebuilding have not been publicly
released although parts and labor for
the engineering rebuild were likely to
be at least a $1/4M USD
in parts and labor. Whilst
StuxNet demonstrated that hacking by
well-funded and highly motivated hackers
could physically damage equipment
connected to a control system,
interruption of a power grid on a scale
like this had never been seen before. The
Ukrainian companies were well practiced
in manual override controls and given
their staffing structure and training
they were able to restore power and run
the system manually very quickly and for
several weeks until the remote control
infrastructure could be fixed. Other
countries around the world have been
progressively relying more and more on
control systems infrastructure to
operate their facilities remotely to
reduce operating costs. In many cases
local controls are also being sacrificed
entirely to save on manufacturing and
ongoing maintenance costs. In such cases
an attack like this one would have been
far more devastating.
Without sufficient manual
override control or without enough
trained personnel to operate substations
manually the outage could have lasted
for days, or even a week in some
countries around the world. The
ramifications of Black Energy have
actually been a strengthening of the
application of cybersecurity
defense-in-depth
architectures in utilities around the
world. Driving a huge spike in investment
in software for monitoring logs,
detecting abnormal network communication
and the identification of data traffic
specifically between SCADA systems and
equipment controllers. Four months ago
in mid-2019 the SEIA bill or
Secure Energy Infrastructure Act in the
United States passed through the Senate.
It mandates sufficient manual controls
to ensure manual control overrides and
procedures exist to operate the
electrical grid infrastructure in the
event of a failure or a compromise of
automated control systems. The Act
attempts to counter the growing trend of
cost reduction by removing full manual
controls in favor of automation and to
mandate that those controls must be
installed, inspired in large by the
Ukraine attack in 2015. So what lessons
have we learned from this? Social
targeting is a big problem that a lot of
people don't appreciate. When I used to
work at Nortel Networks in the late
1990s organizational charts (or Org
Charts) were relabeled as "Commercial in
Confidence" documents because there had
been several incidents of targeted
poaching attempts from other Tech
companies. In essence it was a map to the
structure internally and those wishing
to poach people would know straight who
to speak to. Today social media and the
over-sharing of information has made it
difficult for companies to protect this
sort of information let alone from
poaching attempts but from hackers as
well, and today companies like LinkedIn
make it incredibly easy for a hacker to
craft a highly targeted phishing or
spear-phishing Email to exactly the
right group of people they need to get
access. As previously mentioned
spear-phishing campaigns or attacks are
the first volley from those trying to
penetrate a network and they're becoming
harder to detect and block, and are
becoming far more frequent. If I'm a
hacker and I want to disrupt a company
that runs industrial equipment like
electricity generation or electricity
supply, Oil & Gas, mining, water supplies
even hospitals. Many employ control
systems engineers like me to maintain
and support their control system
infrastructure. If I post that I'm a
control systems engineer on LinkedIn, I'm
working for the target company someone
wants to disrupt, they then draw up a
short list.
Sometimes people over share the projects
that they've worked on. Perhaps they've
worked on a safety system for an
explosives plant, which might make it
easier to target those people
specifically in a spear-phishing attack,
since those people are the most likely
to have credentials that have access to
those safety systems you're trying to
disrupt. From a shortlist constructed
from LinkedIn it stands to reason that I
would know other control systems engineers
also working at the same company so now
they're able to send an Email to me that
at first glance appears to come from one
of my co-workers someone that I know but
has a malicious attachment that I'm less
suspicious of. So I open it
unsuspectingly then they have access to
my machine, and you might think LinkedIn
restricts access to my list of
connections if I select that, and if
you're not a connection of mine then
that's true (certainly the way I've got
it set) but all it takes is for someone
to pose as a recruitment consultant, a
headhunter, an agent, and if you're in the
market for a new job even if you're
strict about only connecting with people
that you've personally met, you might
just accept one or two requests like
that and if one is malicious then they
have everything they need with all of
your connections to find who to target.
The hackers know that the control
systems engineers have elevated
privileges in the control system and
might even have remote access rights as
well if that exists, so once they've got
on to my machine a key logger lays-in-wait
for me to log into remote access gateway
to the control network and then they
have my credentials and it's open season.
If that access portal is internet
exposed they can get in and out anytime
they want and if no one is
cross-checking when I'm
logging in and when I'm not, and if
there's no other mechanism like
Multi-Factor Authentication then it's an
open door and they can poke and prod and
look for weaknesses from within the
system to the extent my credentials let
them. In the case of Black Energy the
original attack wasn't quite as targeted
but ultimately they still achieved their
goal by gaining access to the domain
controller, it just took a little longer.
But undetected they could take as long
as they liked. There have been a lot of
reports digging into the incident and
many lessons can be learned from it. ISA
or IEC 62443-3-3
lists 51 system requirements that
are recommended to improve resilience to
cyber intrusion in Operational
Technology (or OT) systems. An
International Society of Automation (or
ISA) report highlighted 7 significant
SR breaches in this incident which I'll
cover only three of them that I consider
key. 2.4: The transfer of malware
between systems on the OT network
clearly demonstrated that there were no
substantial controls restricting file
transfer throughout the control system
network. It's common practice to have a
so-called secure gateway for all file
access between machines on an OT network.
That gateway or Dropbox is heavily
scanned and logged and reviewed
regularly. All other file transfer
pathways are then restricted or disabled
completely making it significantly
harder to spread malware through the OT
system if you do get in. People complain
(that use the system every day) it's hard
to get data in or out of the system
without that additional intermediate
step, but it does stop the hackers from
easily cross-infecting machines on the
network. Point 5: Whilst a firewall
existed lacking sufficient additional
controls it was rendered mostly
ineffective. The essence of this comment:
for a firewall to be useful there also
has to be strong authentication. Also
detection via an automated or manual log
review method of some kind would have
uncovered the hackers activity quite
easily. Item 6.2: Lack of overall network
monitoring. Without any monitoring tools
the attackers could surveil the network
extensively completely undetected for
weeks and in this
case, months. So what do we conclude from
all of this? If you're working with
critical infrastructure and there are
remote access pathways to your control
system infrastructure then be warned, you
are or will be (someday) a target. If you're
part of an IT or OT department, make sure
you have a centralized log collection
and analysis tool, and have eyes on glass
looking over those logs for any
suspicious activity because that is part
of your job. If you're able to install
software tools that can monitor network
traffic, learn standard behaviors and
then report on exceptions or suspect
traffic patterns, then invest in it and
use it. If you have a remote access
portal for goodness sake install
Multi-Factor Authentication. We humans
are sometimes too clever for our own
good. We use our intelligence to build
machines that require less and less work
for us to do. It's easier to monitor,
reset, open and closed circuit breakers
from a room hundreds of kilometers away.
Why drive there when you can just click
a button. It's easier and it's cheaper to
build things with no physical buttons, no
indicator lights, no manual overrides and
that's fine until something goes wrong
and you need them. Cyber-security is a
never-ending tug of war between company
convenience and the risk of cyber
inconvenience. The more convenient that
we make it for ourselves, the more
convenient we make it for hackers to
make our lives inconvenient. In the case
of Black Energy only 0.015%
of the power grid was inconvenienced
that evening. Next time though, the
consequences could be much much more
severe. If you're involved with OT
systems: pay attention. Those people
intent on wreaking havoc throughout the
world, they might be watching you and
waiting for you to choose that
convenience and laziness over security.
If you're enjoying Causality and want to
support the show you can: by subscribing
to the Premium site via Breaker or via
Patreon.
You can find details at https://engineered.network/
/causality with a thank you to all
of our patrons and a special thank you
to our Silver Producers Carsten Hanson,
John Whitlow, Joseph Antonio and Kevin
Kosh, and an extra special thank you to
our Gold Producer known only as "r".
Patron rewards include a name thank you
on the website, a name thank you at the
end of episodes and access to detail
raw show notes too. Premium rewards also
include add free high-quality releases
of every episode, so if you'd like to
subscribe you can help make sure the
show continues to be produced and above
all else it's all really, really
appreciated. Of course there's lots of
other ways to help like favoriting this
episode in your podcast player app or
sharing the episode or the show with
your friends or via social. Some podcast
players that you share audio clips of
episodes so if you have a favourite
segment, feel free to share that too. All
of these things help others to discover
the show and can make a big difference.
Causality is heavily researched and
links to all materials used for the
creation of this episode are contained
in the show notes. You can find them in
the text of the episode description of
your podcast player or on our website.
You can follow me on the Fediverse
@chidgey@engineered.space on Twitter
@johnchidgey (all one word) or the
network @Engineered_Net.
This was Causality. I'm John Chidgey. Thanks so
much for listening.
[Music]