[Music]
Chain of events.
Cause and effect. We analyze what went
right, and what went wrong,
as we discover that many outcomes can be
predicted, planned for
and even prevented. I'm John Chidgey, and
this is Causality.
Causality is supported by you: our
listeners. If you'd like to support the
show
you can by becoming a patron. Patron
subscribers
have access to early release, high
quality ad-free episodes as well as
bonus episodes and to Causality Explored.
You can do this via Patreon, just visit
https://engineered.network.causality
to learn how you can help this
show to continue to be made.
Thank you. "San Bruno"
At 6:11pm local daylight savings time
on Thursday the 9th of September, 2010 in
San Bruno,
a small suburb south of San Crancisco in
California,
an explosion occurred in Crestmore a
residential neighborhood.
The explosion registered at 1.1 on the
Richter scale
and it took place only 3.2km or
2mi
from the San Francisco International
Airport.
The Pacific Gas & Electric Company (PG&E)
...
owned and operated the gas pipeline, as
well as multiple others,
as well as electrical distribution
throughout approximately t2/3rds of
northern California.
From Bakersfield to near each of the
Oregon, Nevada,
and Arizona state lines. PG&E
operated approximately 2,900km
or
1,800mi of high pressure gas pipelines
at that time.
Before we talk about the incident, let's
talk a little bit about how gas
pipelines work.
Gas is moved around pipelines from a
source
to a destination. Where the gas is coming
from (a source location)
is referred to as "Upstream," to a
"Downstream"
location where it's going to its
destination. Gas is compressed
at a compression station and then that
pressurized gas is output
into a pipeline. A pipeline is best
thought of
as similar to a balloon, but fit that
balloon
with a flow control valve on one side to
counterbalance the person blowing air
into the inlet of the balloon.
Like a balloon if the outlet side is
restricted too much,
whilst pressurised gas continues to
enter it,
then it will continue to expand, until
eventually
it ruptures. Hence controlling the outlet
flow is crucial to ensuring safe
operation
of a pipeline. Put too much gas in:
it will over-pressure if you're not
letting that amount of gas out.
Where pipelines join or split they use a
connecting pipe section referred to as a
"Header."
To ensure each downstream section gets
the correct flow,
and that everything is set correctly
sometimes, the back-pressure is monitored
and alarmed to detect reverse flow, which
is usually undesirable.
A pipeline operator's role is to ensure
that the flow rates
and pressure regulating or pressure
control valves,
are set such that gas is directed in the
right quantities
to the right destinations within the
operational limits
of the pipeline network. High pressure
pipelines are normally made out of steel
or concrete lined steel
and whilst they are made of a steel they
still have a
rupturing pressure limit referred to as
the MaOP
(or Maximum Operating Pressure).
Steel fundamentally has 2 problems:
Corrosion
and fracturing. Steel is made from Iron
mixed with Carbon which turns Iron into
a significantly stronger material by
introducing barriers to dislocations
attempting to traverse through the
materials crystalline structure.
When dislocations propagate and build
over a long period of time they can lead
to fracturing
and this effect is often referred to as
fatigue stress.
For a steel pipeline as gas pressure
increases and then decreases again,
day-in, day-out, for a long period of time
those dislocations will inevitably lead
to weaknesses in the steel pipeline,
increasing the risk of a rupture. Hence a
pipeline has a limited lifetime
after which it cannot be used safely and
exceeding the MaOP
even for a young steel pipeline might
not lead to a rupture at that moment,
but each time there is a pressure cycle
over that limit,
it will significantly shorten the usable
life of that pipeline.
Pipeline pressures are generally
referred to as Gauge
Pressure as the suffix of the unit of
measure.
In this incident it was in the United
States: all
pressure measurements are in psi (or
pounds per square inch)
and referenced to gauge, hence psi gauge
(or psig).
PG&E had identified that their existing
UPS (Uninterruptable Power Supply) system
which was supplying backup for the 120V
...
AC distribution panels, throughout their
network
needed to be systematically upgraded for
a multitude of reasons,
and in order to upgrade it that it would
first be necessary
to move individual loads off the UPS
distribution board,
to individual standalone UPSs'. The
project consisted of multiple
smaller outages where individual loads
were transferred over one by one.
Approval for the UPS project work was
granted on the 27th of August,
2010. By the 9th of September,
some loads had been transferred in the
first stage of the project,
and several more were planned for that
day.
The Milpitas Terminal plays a pivotal
role in this incident.
The impacted section of the Milpitas
Terminal took 2 gas
lines in: 300A and 131,
into a common set of headers numbered
3, 4, 5, 6
and 7. From these headers line 300B
and 327 feed downstream stations with
several connectors dropping the pressure
to header 2.
From which 4 downstream lines: Line
100,
Line 109, Line 101 and Line 132...
which would be the line of the incident.
And then
the San Jose distribution feeder is also
taken off from there.
Each feeder has a Pressure Regulating
Valve (or PRV),
with a local independent controller
which in turn
can be controlled externally in
automatic from a PLC
(Programmable Logic Controller) Each
feeder also has a monitor valve
which is a fully independent,
pneumatically operated,
pressure controller intended to be a
final over-pressure protection
for downstream pipes. Monitor valves are a
more traditional method of control.
They're commonly slow acting therefore a
high lag,
and these devices sometimes can take
several minutes to react and compensate
as gas pressures increase or decrease
with faster adjustments handled by the
PLC controlled PRVs.
Finally there are many manually operated
valves that also allow operators to
direct gas flow in multiple directions
as well as fully isolate pipelines
and headers and provide a pathway for a
full station bypass
if desired. With that background,
let's talk about the incident itself. At
2:46pm
a gas control technician at the Milpitas
Terminal obtained verbal work clearance
from the SCADA center in San Francisco
to initiate another UPS changeover as
part of the broader project.
At 3:36pm the Milpitas technician
called the SCADA center and spoke with a
second SCADA operator
to confirm whether the valves on 2 of
the incoming gas pipelines would close
when power was withdrawn. Upon being told
they would,
the Milpitas technician then locked the
valves open to prevent their
accidental closure during the changeover.
Shortly afterward the workers at the
Milpitas terminal contacted the SCADA
supervisor on shift
to confirm the correct action to take
for the PRVs,
the Pressure Regulating Valves. The
supervisor advised them to put the PRVs
into manual control
at a set position to prevent them from
operating during the loss of power.
At 4:03pm the Milpitas technician
alerted the third SCADA operator
that the installation of one of the
smaller UPSs was about to begin.
The PRVs were put into manual control at
the controller itself,
and indicated they would be returned to
automatic once completed.
At 4:18pm the SCADA center lost
SCADA indication for pressures, flows
and valve positions
at the Milpitas terminal. At 4:32pm
the second SCADA operator contacted the
Milpitas terminal requesting an update
as the works were taking much longer
than expected.
At 4:38pm the Milpitas technician
contacted the SCADA centre to verify the
SCADA indications had now returned to
normal
and upon verifying this, the PRV
controllers were returned to automatic.
Shortly after this the technicians began
powering off
several breakers: one of which was
unidentified,
and they noted that the local control
panel was now
no longer powered up. The technicians
decided to leave the circuit breaker
open and instead referred to electrical
drawings to try and determine
a different way they could power the
local control panel.
As they investigated further they noted
that the power supply that powered some
of the local control panel equipment was
outputting an inconsistent voltage.
The power supplies were 120V AC to
24V DC supplies operating as a
redundant pair on the DC
bus, and that DC bus also powered the 24V
DC
instruments: most notably the pressure
transmitters
that fed the pressure process signal
into the PRV controller.
Since PRVs operate on a pressure
feedback
in a closed loop, if the input signal
reports that the pressure is very low,
the valve will proportionally open to
increase the pressure.
When the pressure doesn't increase the
valve will open more,
and more until the valve is inevitably
fully opened.
At 5:22pm the SCADA center alarm
console displayed more than 60 alarms
in a space of a few seconds, including
controller error alarms,
high differential pressure, gas backflow
alarms and several High High pressure
alarms from the Milpitas terminal.
At 5 25 pm the third SCADA operator
called the Milpitas technicians
reporting the high pressure alarms and
at this time
the Milpitas technician realised that
the pressure and regulating valve
controller displays on the local control
panel had lost all of their data.
The SCADA consoles at Milpitas displayed
constant pressures, a sign of
"Report Last Known Value," as a result of
the process value having been lost with
all the downstream
line PRVs and most of the monitor and
incoming line valves indicating that
they were not
open. At 5:28pm the Milpitas technician
called the fourth SCADA operator to ask
what pressure values were being
displayed on the remote SCADA console
and discovered that the remote console
was not receiving
any live values at all. The remote SCADA
console indicated 458 psig
at the Milpitas terminal mixer. (That's
the name for the common headers 3
through 7)
The fourth SCADA operator
concluded that the PRVs and/or
the station bypass valves may have
been opened which was quickly confirmed
by the technician on site.
With all of the regulating valves fully
open, only the monitor valves were
limiting the pressure
on the downstream pipelines. The monitor
valves were set
at 386 psig, however due to control
lag in the monitor valve's pneumatic
controller, the pressure in the lines
leaving the Milpitas terminal peaked at
396 psig
sometime between 5:22pm and 5:25pm,
however with a lack of transmitter
data it's impossible to be certain
what the peak value was and exactly when
it occurred.
At 5:42pm the Milpitas technician
called the SCADA center and reported to
the third SCADA operator that the
regulating valves on the primary
feedline (300A) had
opened fully but the remote operator was
unable to confirm
live pressures or valve positions from
the Milpitas terminal on his SCADA console.
The third operator then authorized the
local technician to reduce the manual
set point of the monitor valves
from 386 psig to 370 psig to try to
reduce the downstream line pressures.
At 5:52pm the senior SCADA coordinator
directed the fourth SCADA operator to
lower the upstream pressure set points
for the gas supplying the Milpitas
terminal
upstream, as this was still able to be
remotely controlled.
Shortly after the site technician called
the fourth SCADA operator to report
that the monitor valves were shut on
line 300B,
with the fourth SCADA operator noting
500 psig on the common headers 3
through 7
and requested the on-site technician to
place a pressure gauge on Line 132
to get a direct reading of the pressure
leaving the terminal.
There was no other way at this point to
know for sure what that pressure
actually was. Now at 6:02pm
the fourth SCADA operator contacted a
SCADA operator at the PG&E Brentwood
facility stating,
(and I quote): "We've over-pressured the
whole peninsula." (end quote)
At 6:04pm the senior SCADA coordinator
informed the supervisor at the Milpitas
terminal that the pressure on the
incoming lines at the milpitas terminal
had been lowered to 370 psig as
directed.
Shortly after this the on-site
technician reported to the third SCADA
operator
that the manually-read line pressure on
Line 132
had reached 396 psig.
At some time between 6:00pm and 6:11pm
gas began leaking from a section of pipe
in San Bruno,
until it found an ignition source, which
it found
at 6:11pm. At 6:11pm
a 911 call reported an explosion from
either a suspected gas station
or an aircraft crash. With emergency
responders dispatched to the scene
within 1 min
due to the San Bruno Fire Department (the
SBFD)
having a fire station 300 yards from
where the explosion occurred.
At 6:11pm Line 132 upstream of Martin
Sstation
registered a pressure drop from the peak
of 386 psig.
Within 2 minutes of the explosion, many
residents had self-evacuated and fled
their homes to escape the fireball.
By 6:15pm Martin Station generated a
low pressure alarm for Line 132
followed 20 seconds later by an alarm at
150 psig.
At 6:16pm local police began diverting
traffic away from the incident
and assisting in organizing evacuations.
At 6:18pm
an off-duty PG&E employee notified the
PG&E dispatch centre in Concord,
California,
about an explosion that had occurred in
the San Bruno area.
This was the first formal report to PG&E
about the incident,
7 min after it had occurred. At
6:23pm a dispatcher sent a gas service
representative working in Daly City,
that's 8mi South
of San Bruno, to confirm the report was
accurate.
At 6:24pm a PG&E supervisor saw the
accident fire while driving home from
work,
after which he called the PG&E dispatch
center, reported the fire and proceeded
to the scene.
At 6:27pm a PG&E dispatcher called the
SCADA centre and asked the third SCADA
operator if they had observed any
pressure drops
(and I quote): "...at a station in (the) area..."
stating there had been reports of a
flame shooting up in the air
accompanied by a sound similar to a jet
engine and that a PG&E
supervisor and a GSR had been
dispatched to the area
to which the third operator replied that
the SCADA center had not received any
calls about
any incident. At 6:29pm the senior SCADA
coordinator
informed a SCADA coordinator at the
Brentwood Facility that there had been a
gas line break,
and further stated that there had been
an over-pressure event
at the Milpitas terminal earlier. By 6:30pm
...
some staff at the SCADA center realised
that there had been a rupture
along Line 132 in the San Bruno area,
however the exact rupture location was
still unknown.
We're now 19min post-explosion.
At 6:35pm an off-duty PG&E gas
measurement and control mechanic who was
qualified to operate
manual mainline valves, saw media reports
about the fire,
and as he suspected he might be able to
assist, he notified the PG&E dispatch
center and proceeded to the PG&E
Colma Maintenance Yard to obtain a
service truck
and tools necessary to shut off a
mainline valve in case it was needed.
A few minutes later a second mechanic
called him to check on his well-being
and agreed to meet
at the Colmer Yard and by 6:36pm the
Line 132 pressure
at the Martin Station had dropped to
only 50 psig.
At 6:50pm both mechanics arrived at
Colma Yard
and after reviewing a map showing the
location of pipeline valves in the area
and using information from local news
reports
regarding the fireball, they determined
the rupture
was most likely occurring in Line 132.
They then called a supervisor to
authorize isolating the line manually
which was approved
after a brief telephone conversation.
About 7:06pm the two mechanics left the
Colma Yard and route to the first
mainline valve at (Mile Point) MP38.49
that they planned to close and arrived
at 7:20pm.
At 7:27pm the supervisor who was with
the two mechanics at
MP38.49 requested the SCADA center
close 2 valves at the Martin Station
to reduce the flow of gas. At 7:29pm the
fourth SCADA operator had remotely
closed the 2 valves downstream of the
rupture,
stopping gas flow from North to South
into Line 132.
South to North however was still flowing.
By 7:30pm
the 2 mechanics had manually closed
the mainline valve at MP38.49
upstream of the rupture, stopping the gas
flow into 132
from that location. By 7:42pm
now 91min after the rupture,
the gas flow had decreased enough such
that the intensity of the fire had
decreased to a point
where firefighters could approach the
site and begin containment efforts.
By 7:46pm the 2 mechanics had
manually closed 2 additional valves
downstream of the rupture
at MP40.05 and MP40.05-2 at
Healey Station.
At this point, and by closing these final
2 valves,
the ruptured section of Line 132 was
finally,
fully isolated. 1hr
and 35min after the loss of
containment.
Firefighters declared 75% of all active
fires to be contained at about 4:24am
the following morning. At 8:00pm
on the 11th of September, 2010 all fires
had finally
been extinguished. As a result of the
pipeline rupture
and ensuing fires, 48 people sustained
minor injuries,
10 people sustained serious injuries, and
8 people died.
James Franco aged 58: a resident of
Glenview Drive.
Jacqueline Greig aged 44. Janessa Greig
aged 13. Elizabeth Torres aged 81.
Jessica Morales, aged 20.
Lavonne Bullis, aged 87. Greg Bullis,
aged 80. Will Bullis, aged 17.
The rupture released an estimated 47.6
million standard cubic feet of natural
gas, that's about 136GJ (Giga-Joules)
by energy. It left a 72ft by 26ft
crater that's 22m by 8m, and ejected
a 28ft long or 8.5m piece of
pipe
weighing 3,000lbs that's 1.36 tonnes.
108 houses in the neighborhood were
affected by the blast
and fire, of which 53 had some damage,
17 had severe or moderate damage and 38
were completely destroyed.
So what went wrong? There are seemingly
3 components to explore in what went
wrong.
The UPS works impact on the pipeline
pressure controllers,
the design of the pipe itself, and the
response
to the incident. The response
to the incident was slow. Mis-informed and
ultimately
the meaningful isolations were all done
manually.
Why were they done manually? There
weren't enough
remotely controlled valves to allow for
a better isolation of the pipeleg
segment, so actually the speed
that PG&E could respond was severely
hindered
by the pipeline's design. So actually
then
the response problems are just as much
about the design.
So let's start there.
There are several aspects to consider.
The mechanical aspects of the pipe
itself:
why did it fail? The positioning of the
isolation valves and their operability,
and the path of the pipeline relative to
where people had their houses built.
So first things first let's talk about
the pipe itself.
Line 132 was constructed in phases,
during 1944 and an extension in 1948
which included the segment that failed
in this incident. Line 132 consists of 24",
30", 34" and 36" diameter segments of
different steel grades with different
longitudinal seam welds
including: Double Submerged Arc-Welded
(DSAW),
Electric Resistance Welded (ERW) and
Seamless
(SMLS). To clarify, a longitudinal weld
is one that traverses the length of the
pipe to join the bent piece of steel
into a pipe,
and a girth weld is one that joins 2
pieces of pipe together in a circle
around the pipe. Pipe welds are commonly
welded both on the inside and outside
seams to improve their strength. It's a
labor-intensive and tiring job
and to ensure the welds are done
properly they are inspected
after they're completed by different
means including x-rays
and old-fashioned gas or water
pressurization tests.
Records from the 1948 project included
logs for 209 radiographs
and between the two project stages about
10%
of all welds were X-rayed. That's not
very many but keep in mind that in those
days it was an expensive procedure
and X-raying...the entire pipe from
end to end was
unheard of, though 10% isn't very much
admittedly.
According to construction records as the
final check before introducing gas to
the 30"
portion of the line, it was checked
for leaks and breaks,
though the exact method that was used
wasn't described in any detail,
it was assumed by investigators to
consist of a soap and water
mixture on girth welds as it was
pressurised.
In 1956 however PG&E were required
to relocate 1,851ft, that's 564m
of Line 132 that had originally been
installed
in 1948, and this section actually
contained the segment that failed.
They moved this because the existing
elevation of Line 132 was incompatible
with land grading that had been done at
the time
in connection with residential housing
being built
in a new residential area. This
relocation re-routed Line 132 from the
East-side to the West-side of Glenview
Drive,
and was done entirely in-house by PG&E.
The investigation found that there were
no design
material construction specifications, no
inspection records,
no as-built drawings and no X-ray
reports for any of the works conducted
from the 1956 relocation.
Line 132, pipe segment 180, was documented
in the PG&E GIS (that's the Geographic
Information System)
as a 30" diameter Seamless Steel
pipe.
Its code was API 5L X42,
also known as L290 pipe under ISO3183.
The wall thickness was supposedly 0.375"
...
and was installed in 1956 with the
manufacturer noted
as "N/A" on the data sheet.
(Not very useful) During the investigation
the NTSB found that the data in the GIS
was actually imported
in 1998 from data from a pipeline survey
sheet that was created in 1977,
and in digging further it was discovered
that this section
had no engineering records relating to
its specification.
The reference was taken from an
accounting journal from 1956
as materials were transferred between
different construction jobs
and not only that, the reference had been
incorrectly entered in 1977
as X42 when the original voucher
actually read
X52 which is a completely different
pipe.
The investigators found that it was not
possible for this pipe to have been
X42 as seamless pipe wasn't
and still isn't available in diameters
larger than 26"
and this was a 30" pipe segment,
apparently.
The investigators found that this pipe
segment was
actually made from 6 PUPS, which is
pipeline slang, for a short length of
pipe - a short pipe
segment. And each of these 6 PUPS
had been welded together to make one
longer section.
None of the pups were X52 pipe, or
X42 pipe either and they varied in
length from 3.5ft to 4.7ft
long each. Laboratory testing of the
composition and the material properties
in some of the PUPS
didn't meet the PG&E 1948 material
specifications or
any industry pipeline material
specifications for that time period.
They also found several of the PUPS had
partially welded longitudinal
seams with part of the seam unwelded on
the inside,
and several of the girth worlds joining
the PUPS together also contained several
welding defects.
Interestingly in 1995 PG&E replaced
several sections of Line 132.
Their replacements however ended about
565ft
to the South and about 610ft to
the North
of Segment 180 as part of a multi-year
replacement project
to address seismic hazards. Prior to 1961
no regulations in California existed
that governed
natural gas pipeline safety. The
voluntary national consensus standard
ASME B31.1.8, 1955 edition,
called for hydrostatic pressure testing
of pipelines
at 1.1x to 1.4x the MaOP.
PG&E elected not to hydrostatically test
segment 180 of Line 132,
and whether PG&E followed the other
guidelines of the ASME standard
can't be demonstrated because it was
never documented.
In 1961 the new General Order 112
required hydrostatic pressure testing of
newly constructed pipelines in Class 3
areas
at 1.5x the MaOP.
However as Line 132 was not technically
a new pipeline,
it was not required to be
retrospectively tested.
So let's talk a little bit more about
MaOP and why it's important.
When a pipe section's designed, a
combination of its
material properties, the construction and
size, all dictate its maximum operating
pressure that it can withstand.
Depending on how much gas volume you
want to transport over what distance,
and taking into account how much Line
Pack you want to keep
to allow for supply upsets and demand
surges, you pick the right kind of pipe
at the design stage, before you put in
the ground.
Well that's the idea at least but in
this case the pipeline wasn't
really designed with that kind of
methodology and if it was the section
they fitted in 1956
consisting of those PUPS, certainly
wasn't.
In terms of testing requirements though
for pipelines constructed before 1970
where hydrostatic testing wasn't
required, 49 CFR
192.619(a) (3),
our old friend a "grandfather clause"
allows the MaOP to be based on
the highest actual operating pressure to
which the segment was subjected,
during the 5 years preceding July the
1st,
1970. Unlike an MaOP that's tested based
on a hydrostatic pressure test,
the grandfather clause doesn't specify a
minimum amount of time that the
historical pressure needs to have been
held
before it could be considered a valid
MaOP sample.
The thinking is that if a pipe wasn't
designed with the target MaOP in mind
from the start,
if that pipeline has operated at a peak
pressure recently in that 5 year
period,
it should be okay to handle it? Right?
Well...
not really...but that's the regulation.
In 1987 the NTSB recommended eliminating
the grandfather clause in Safety
Recommendation:
p-87-9 however this was not taken up by
the standards boards
following public comments recommending
against
its elimination. The MaOP for Line 132
was established
at 400 psig under the grandfather
clause as a result of records from the
highest operating pressure on Line 132
during that 5 year period,
it had reached 400 psig on October
the 16th,
1968. Now PG&E
had a subtly different interpretation.
They considered the MaOP of a pipeline to
be the maximum pressure
at which a pipeline system, as
distinguished from a pipeline segment,
can operate.
Hence the MaOP set by PG&E for Line 132
was only 375 psig.
This is because when cross-ties
connecting Line 132,
although it has an MaOP of 400 psig,
when it connects to Line 109 it only has
an MaOP
of 375 psig and once those cross-
ties are open, the MaOP of Line 132
is the MaOP of Line 109 which is 375 psig.
...
Now whether that MaOP was correctly set
to match the pipeline's actual capacity,
it was probably borderline, well it would
have been for a well-constructed segment
but not a poorly constructed segment.
Whether the measured pressure
immediately prior to the loss of
containment exceeded PG&E's
own MaOP for Line 132 there is no
question that it was exceeded
significantly.
It was clocked at 398 psig, MaOP was
set
to 375psig. The NTSB report's findings, well,
there were 28 in total,
but I'll just touch on the ones that i
think are the most relevant.
The pipe segment that failed did not
conform to PG&E's own standards
nor any other standards of the time and
was unable to be traced to where it was
manufactured
or by whom, and it would have failed
accepted industry quality control and
welding standards from the day it was
constructed.
The lack of quality control during the
1956 relocation project led to a
defective pipe segment being installed
that remained
undetected until it failed 54 years
later.
the fracture of Line 132 Segment 180,
originated in the partially welded
longitudinal seam of a PUP.
PUP number 1. Which was progressively
weakened due to ductile crack growth
and fatigue stress. The SCADA system
made tracing the source of the leak
difficult and
a lack of remotely controllable valves
and a lack of Automatic Line Break
valves led to an excessive delay
in isolating the segment, causing
significantly greater damage.
The report was also critical of the poor
planning of the electrical UPS work at
the Milpitas terminal,
which i agree could have been done a bit
more sensibly,
like turning the breaker back on and
making sure the critical instruments
were always energised,
valve positions and manual override and
such. The truth is though that
those events were just the final straw.
This pipe leg wasn't subjected to any
pressure it hadn't been previously
and it was still within 10% over
the MaOP, which in a properly constructed
pipeline wouldn't have led to a failure.
Had the UPS over-pressurisation not
occurred,
if they hadn't found the defective pipe
segment in the next 12 to 24 months or
so,
it's very likely that it would have let
go through normal operational spikes and
surges.
Specifically in the report though, noting
this one:
(and I quote): "Inline inspection technology
is not available for use in all
currently operating gas transmission
pipeline systems.
Operators do not have the benefit of a
uniquely effective assessment tool
to identify and assess the threat from
critical defects in their pipelines."
(end quote) That is a really good point.
When i first read about this incident i
thought immediately,
"Why didn't they just use an inspection
PIG?"
PIGs are Pipeline Inspection Gauges,
and are basically a big cylindrical (not
quite a plug)
that you load into a working pipeline at
the upstream end,
build back pressure behind it in a
launcher, which is essentially just a
large chamber for loading and unloading a
PIG, and then you open the valve and let
it fly into the pipeline.
It's pushed through the pipeline from
the gas pressure behind it
and it travels down the other end of the
pipeline to the receiver
and they're commonly used for cleaning
things out of the pipe like
Tri-Ethylene Glycol (TEG), or in other cases
they have a 360°
X-ray machine built into them. They
literally take a continuous X-ray of the
inside of the pipeline looking for
defects in welds and pipe work corrosion,
everywhere from the launcher to the
receiver, recording them all on board
and the engineers pour over them when
they're done at the end.
Very, extremely cool stuff! Not cheap
though.
They're technically called an ILI, an
In-Line Inspection PIG.
Of course they only work if the pipeline
is much the same size
along its length and have pig launchers
and receivers
and have a maximum bend radius for all
the sections you want the PIG to travel
through.
Too many corners obviously it's not
going to work. Line 132
didn't qualify for pigging, or certainly
Segment 180 didn't. So leak detection
then
is down to pressurizing and testing for
a loss of pressure or
by using tracer gas and gas sniffers at
ground level.
Not as accurate. Not as effective. So
despite all of this,
why would you build it in a residential
neighborhood?
So let's look at that. California
Public Utilities Commission, San Bruno
Explosion Report of the Independent
Review Panel (2011) stated (and I quote):
"In 1956 the city converted PG&E's
pipeline Right Of Way
from an Easement to a Franchise Right as
the community was growing and
residential subdivisions were being laid
throughout the area." (end quote)
So what's an "Easement," at least in the
context of California law?
According to the California Land Title
Association an Easement is a: "...real estate
ownership
right, an encumbrance on the title
granted to an individual
or entity to make a limited but
typically indefinite use
of the land of another..." It goes on to
state
that: "...statutes may prohibit locating
buildings
or undertaking any construction over a
pipeline easement."
So a Right Of Way then is: "...the legal
right
established by usage or grant to pass
along a specific route
through grounds or property belonging to
another."
The term "Franchise Right" is interesting.
In California, Public Utilities are
defined as:
"An entity which provides a service to
the public for reasonable charges
whether such charges be in the form of
taxes or rates."
So utilities are granted statutory
franchise
rights, most often in the context of
let's say a water mains
under a city street, and this provides a
degree of legal protections against
consequential costs for repairs,
relocations,
amongst other things but in this context,
PG&E were operating
as a utility and were granted statutory
franchise rights for that section of the
pipeline,
rather than the stricter Right Of Way
condition that an easement would have
because under a local statute that would
have prevented anyone from building on
it.
So in summary, the city re-classified the
easement
so they could utilise what was the
Right Of Way easement,
so they could then build houses in 1957,
and to make way for that they regraded a
section of the ground to level it for
those houses and additional streets
requiring a short section of the
pipeline to be moved to a different spot,
which was the work that PG&E did
themselves
in 1956. The suburb
that was built there, Crestmoor, consists
of 2,500 homes built between 1957
and 1959. That was a decade after
the pipeline was originally finished and
had been in operation for that long.
Only 1 year though after Segment 180
was reinstalled, made from those 6
pieces of PUPS. So technically
they didn't bring the pipeline to the
neighborhood. The neighborhood came to
the pipeline.
Let's talk about the fallout. The cost to
repair the pipeline
was estimated to be $13.5M USD.
...
About $1/4M USD
worth of gas,
went up in smoke in 90min.
The CPUC fined the utility $1.6B USD
...
in connection with the disaster. That's
the largest penalty
ever levied against a California Utility.
PG&E paid over $550M USD to
the victims of the explosion.
PG&E were required to undertake pressure
tests and integrity checks of their 1,800mi
...
of transmission pipelines.
In August 2016, a federal jury convicted
PG&E on 5 charges of violating
federal pipeline safety regulations
and 1 charge of obstructing the NTSB
investigation.
In January 2017, a federal judge
sentenced PG&E for crimes linked to the
explosion,
imposing the maximum fine of $3M
USD,
and enforcing mandatory oversight by an
external monitor
(not the PUC) and 5 years
probation. No employees
of PG&E or the PUC were individually
sentenced.
No jail time was served by anyone
involved in this
incident. In the years since PG&E were
also found guilty in the Camp Fire
incident
and whose equipment also started
multiple wildfires, with subsequent fines
and class actions with $30B USD
in liabilities PG&E filed for
bankruptcy
for a second time in its history, in
January of 2019.
On Wednesday the 1st of July, 2020
only a few weeks ago at the time of
recording this,
PG&E exited Chapter 11 and is once again
out of bankruptcy.
Having said that 5 years after the
incident, PG&E had completed 673mi of
pressure testing of their gas pipeline
network.
They also replaced 115mi of
transmission pipeline,
down-rated 12mi, retired 9mi,
upgraded 202mi of pipeline to accept
In-Line Inspection PIGs, and installed
217 automatic valves
to improve pipe leg isolation. So what do
we learn
from all of this? Let's start with the
operational issues on the day.
Keeping in mind they contributed as the
trigger, but if they hadn't, the pipe
would have ruptured at some point in the
future anyway unless they caught it
first in a test.
That said the technicians at the
terminal doing the UPS work,
they shouldn't have been trying to
figure out the consequences of turning
power off on valves during the
changeover.
Why hadn't they pre-planned that? The
sequence of events that jumped out to me
as well in all of this,
why was there no dedicated operator as
the point of contact?
There was no SCADA operator as the sole
point of contact for those works!
They were constantly speaking to
different operators, through the entire
process, which led to inconsistent
answers, a lack of common understanding
in the SCADA control room
about what was actually happening. It was
just poorly coordinated overall.
Even if they couldn't afford to assign
an operator to those works entirely,
they should have had at least 1 of the
4 SCADA operators as the point of
contact for the works.
Besides that though, engineering records
are critical.
Design details that can be traced and
linked to inspection
records. These are critical pieces of
information
that so many companies seem to treat as
an afterthought.
When operational companies execute works
entirely in-house,
the first things that seem to get cut
are the engineering records.
The design documents. External companies
develop
design documentations and keep test
records as evidence or proof
for progress claims as well as for legal
reasons,
but operational companies don't feel the
need to,
which may not be needed for progress
claims (because there aren't any it's
internal)
but they still need to be kept for legal
reasons. in the competition for BAU
(Business As Usual), and project work, all
too often
corners get cut on design. And
documentation on internal projects
was clearly a corner that was cut in the
1956 work
on Segment 180. Finally isolation
or rather a lack of it. When the pipeline
was built,
remotely operated valves did exist but
they were
very expensive to install and not very
reliable but manual valves were still
installed.
Driving to the sites and manually
closing the valves takes precious time,
that could have been saved if they had
remote actuators fitted to those manual
valves,
hence automating them. It also seems
unbelievable
that the SCADA operators didn't
realize that there was a rupture for so
long after it had occurred.
The readings at Martin Station dropped
so low, so
quickly, they should have realised but it
was
close to 20min before they did and
when they did they couldn't pinpoint
where
just somewhere on Line 132...is a bit
vague.
So if they'd automated more of those
mainline
isolation valves they would have also
fitted local
upstream and downstream pressure
monitoring at those valve locations,
that's standard practice,
and that would have made pin-pointing the
loss of containment significantly faster.
With automated valve actuators fitted to
the manual valves they closed
and pressure transmitters fitted at those
locations, they could have isolated that
pipeleg in only a few minutes.
Not 1-1/2hrs. Some countries
around the world have legislation that
restricts the size of gas pipelines that
can be built
in or near residential areas, but it's
inconsistent between countries and even
states in the same country. In an
environment where developers want to
build
where the flattest land is and where the
rules have
loopholes, there are pressures for local
governments to rezone and reclassify
vast tracts of land
that probably shouldn't have been
reclassified for all sorts of reasons.
Easements are generally made easements
for a good reason.
Then again the original mistake: the
poorly manufactured pipe leg segment
made from those 6 PUPS, was made
before more than half of these victims
were even
born. Like the 737MAX,
grandfathering provisions just need to
go away
from so many regulations including these
ones.
When you're designing a gas pipeline, you
have to think about containment.
They have a saying in the business: "Keep
it in the pipes."
That's your job. Because whatever's in
that pipe if it gets out
it can kill people, hurt people, destroy
property, cause
environmental damage, and yeah okay
there's money too
because you're going to lose money because
your product is getting out...never gets
to a customer, so you can't sell it.
Also if those customers are relying on
that product
for heating, sterilization in hospitals,
cooking and restaurants,
I mean you name it, there's lots of other
outcomes as well.
Gas pipelines can be built and operated
safely if you do it correctly,
but you have to inspect your pipes. You
have to know their condition.
That they're safe to continue to operate.
So they couldn't use an inspection PIG
in 2010,
but they could have hydro-tested it. So
why didn't they?
It's money. You have to empty the pipe
of the gas. Sometimes you need to purge
it with an inert gas:
some will debate that one. Fill it with
water. Pressurise it and test it.
If it fails you have to trace the leak,
dig it up,
patch it, fix it, refill the pipe, re-test
the pipe,
drain the pipe, dry the pipe, and then you
can put gas back in it again.
That whole process takes time and all
that time you're not moving gas,
you're not making money. So if you don't
mandate it,
they won't do it. When you're an engineer,
building something that could be an
operation for a very long time, a very,
very long time, you have to make sure
that it's
designed to the required standard.
That that design is documented. That it's
built and tested
against that standard, as best you can,
from the right materials and test it as
thoroughly as you can.
Because the mistakes made today will
sometimes
lay in wait for 50yrs,
before they take innocent people's lives.
If you're enjoying Causality and you
want to support the show you can
by becoming a Patron. You can find
details at https://engineered.network/
causality, with a thank you to all of our
Patrons and a special thank you to our
Patreon Silver Producers Mitch Bielger,
John Whitlow, Joseph Antonio, Kevin Koch,
Oliver Steele and Shane O'Neill.
And an extra special thank you to our
Patron Gold Producer known only as
'r'. Patrons have access to early release,
high quality ad-free episodes as well as
bonus episodes and to Causality Explored.
You can do this via Patreon just visit
https://engineered.network/causality to
learn how
you can help this show to continue to be
made. Of course there's lots of other
ways to help like favouriting this episode
in your podcast player app
or sharing the episode or the show with
your friends or via social.
Some podcast players let you share audio
clips of the episode so if you have a
few favourite segments feel free to share
them too.
All these things help others discover
the show and can make a big difference.
Causality is heavily researched and
links to all the materials used for the
creation of this episode are contained
in the show notes.
You can find them in the text of the
episode description, on your podcast
player or on our website.
You can follow me on the Fediverse
@chidgey@engineered.space
on Twitter @johnchidgey or
the network
@Engineered_Net. This was
Causality.
I'm John Chidgey. Thanks so much for
listening.
[Music]