[Music]
Chain of events, cause and effect.
We analyze what went right and what went wrong,
as we discover that many outcomes can be predicted, planned for, and even prevented.
I'm John Chidgey, and this is Causality.
Causality is supported by you, our listeners.
If you'd like to support the show, you can by becoming a Premium Subscriber.
Premium subscribers have access to early release high quality ad free episodes, as well as
bonus episodes and to Causality Explored.
You can do this via Patreon or, if you prefer, via our website, visit https://engineered.network/causality
to learn how you can help this show to continue to be made.
Thank you.
Walkerton
Walkerton, Ontario, Canada has a population of about 5,000 people and lies approximately
2 hours drive from Toronto, which by contrast is Canada's largest metropolitan city at 6 million
people. In the early 2000s, the water supply to Walkerton was supplied by 3 groundwater wells
numbered 5, 6 and 7. Well 5 was 15 metres deep, capable of supplying 56% of the town's water
supply needs in isolation with Well 6, 72 meters deep with a maximum supply of 52% in
isolation and Well 7 was 76 meters deep the deepest with a maximum supply of 140%
of the town's water supply needs. Well 5 used sodium hypochlorite solution for chlorine dosing
whereas Wells 6 and 7 both used chlorine gas for disinfection.
The water distribution system had approximately 42 kilometres or 26 miles of water mains with
two standpipes providing pressure equalisation and approximately 20 hours of reserved storage.
A basic control system and SCADA controlled and monitored the wells and standpipes.
The Walkerton Public Utilities Commission or PUC for short were charged with the safe operation
and maintenance of the water supply system in Walkerton. So let's talk about the incident itself.
On the 8th of May 2000, a series of storms and steady rain over a five-day period totaled some
134 millimeters of rainfall. The volume of rainfall accumulation led to inevitable surface
saturation and subsequent runoff which led to some minor localized flooding in the area. On the 12th
of May some of that rainfall runoff entered Well 5. On Monday the 15th of May
Stan Koebel returned on shift and noted that well seven was not operating. Every Monday,
and this was no different, the operators collected their weekly samples from each of the wells and
from key sample points in the distribution system. Around this time a construction project was
was underway that required the installation of 615 metres or 2,000 feet of replacement
water mains on Highway 9 in South West Walkerton between Wallace Street and Circle Drive.
On Wednesday 17 May at 9:14am the A&L Laboratory faxed the results from the Highway 9 project
water samples to the PUC. All three samples indicated positive for total coliforms and E. coli.
At 2:37pm that afternoon, the remaining tests from the A&L lab were faxed to the PUC
with a sample labeled Well 7 treated, positive for total coliform and E. coli. Further, the tests
indicated coliform bacteria greater than 200 CFU/100mL, E. coli greater than 200 CFU/100mL,
and a Heterotrophic plate count of 600 CFU/mL. By Thursday, the 18th of May, the number of
illnesses had increased significantly, with a seven-year-old and nine-year-old admitted to the
Owen Sound Hospital, and about 20 students from the Mother Teresa School reported in sick.
Members of the public, including concerned parents, had contacted the Walkerton PUC
to confirm the water was safe to drink, however were not told that anything was wrong.
By Friday 19 May, 8 people had a documented three-day history of symptoms,
with now more than 25 absent from the Mother Teresa School, 8 from Walkerton Public,
and 3 residents from the Maple Court Villa retirement home were also affected.
Dr Kristen Hallett, a paediatrician from the Grey-Bruce Health Services,
had two patients referred to her from the hospital with similar symptoms.
At approximately 9:00am that day, Dr Hallett contacted Dr Murray Quigg, the local medical
officer of health (MHO) to inform him that her food history investigation of those patients
indicated contaminated water was the most likely cause, with E. coli the most likely pathogen.
During that day, James Schmidt, the public health inspector in Walkerton, received multiple calls
and proceeded to call Mr. Koebel at 2:21pm directly, asking him about any issues with
the water supply, to which Stan Koebel indicated he thought the water was 'OK'.
On Sunday, the 21st of May, at approximately 1:30pm, a public health advisory was issued by
the Health Unit to the Walkerton community not to drink municipal water from the tap,
recommending boiling all water before consumption. The MHO also took their own independently
collected water samples from multiple locations in Walkerton and their results on the 23rd of May
all showed E. coli contamination, leading to all schools being closed the following day.
On the 25th of May, the Regional Police Force directed the Ontario Provincial Police to begin
a criminal investigation into the incident. With incidents such as these, the health impacts can
take weeks, months or even years to fully play out, and those most at risk are our most vulnerable
in society: children, the elderly, and those with pre-existing medical conditions that are
ill-equipped to fight off an illness like this. The number of people killed directly or indirectly
due to this incident has been debated since shortly after the incident, with either two,
three or four others indirectly linked, and not all are conclusively proven to have been linked
to the incident though there is strong evidence to suggest that they were. The following people
lost their lives either in whole or in part due to this incident. Melville Dawe 69 years old May 19th.
Lenore Al 66 years old, May 22nd. Mary Rose Raymond 2yrs old, May 23rd. Robert Brodie 89yrs old,
May 24th. Edith Pearson 82yrs old, also May 24th. Vera Coe 75yrs old, also May 24th.
Laura Rowe 84yrs old, on May 29th. Betty Trushinski 56yrs old, May 31st.
So what on earth went wrong? The Honourable Dennis R O'Connor was appointed to lead the
Walkerton Commission into this incident, producing a final report in two parts,
the first of which was released in January 2002.
The source of the contamination was found to be Well 5,
with runoff from a farmer's paddock, with the fecal coliforms from the
livestock excrement being washed into the drinking water.
The investigation unsurfaced several disturbing behaviors and events
leading up to the incident. Prior to the rainfall event on the 5th of May, Stan
Koebel left Walkerton to attend a conference in
Windsor for which he was away until the 14th of May during which time the rainfall event had
occurred. In his absence his brother Frank Koebel was in charge of the PUC in Walkerton. When Stan
left for the conference he was aware that the chlorinator on Well 7 was not functioning
correctly since Well 7 was brought back into service on the 2nd of May. In fact the well hadn't
been in service since the 10th of March. It wasn't uncommon to rotate water supply from each of the
wells. But rather than shut Well 7 off, Stan Koebel instructed Frank to replace
the chlorinator in Well 7 with the replacement unit that had been on the
PUC premises in Walkerton for nearly 1-1/2yrs. Upon Stan's return
he found that Frank had still not fitted the replacement chlorinator to Well 7
and that Well 7 had still been running during that time. Well 7 pumped
unchlorinated water into the system from the 3rd of May to the 9th of May as it
the only well being used during that period, there was no new chlorine being injected into
the water main system for that time. The correct course of action would have been to leave
Wells 5 and/or 6 running while the chlorinator in Well 7 was replaced and then to return
Well 7 to service, an activity that would have taken about a day of end-to-end activities
to complete...absolute maximum. Chlorine residual needs to be maintained to ensure that any
bacteria are killed before the water is consumed, but that's something we'll "explore" separately.
Despite the fact that both Stan and Frank Koebel were aware that chlorination was required
at all times as mandated in the Ontario Drinking Water Objectives and Bulletin 65-W-4, "Chlorination
of Potable Water Supplies", when interviewed following the event, they believed that unchlorinated
water from Well 7 was safe because it was from a deeper well. In addition, PUC staff
would regularly drink raw unchlorinated water at the well because it was cold, clear and
clean and "tasted better" than chlorinated water did. Multiple years of reinforcing the
idea that it's safe to drink it today so it'll be safe to drink it tomorrow led to a mistaken
belief that chlorination was in fact optional for Well 7. In the history of the plant,
there had been no incidents like this, which also fed a mistaken belief that anything like
this could actually happen. Every day operators were required to visit each well and make a
recording of the following: the water flow totalizer, the chlorine chemical usage and
the current chlorine residual. On the 13th of May at 4:10pm 0.75mg/L
of chlorine concentration was recorded for Well 5. There was no entry for Well 6.
By cross-checking the water volume recorded against the amount of Hypochlorite dosed during
that period it was calculated that it was completely impossible to have a chlorine
residual that high given how little Hypo was dosed during that period. The investigation
also found that for more than 20 years it had been regular practice for PUC operators
to not measure the actual chlorine residual but instead write down a fictitious value
to put an entry in the box. Reviews of the logs showed a significant number of readings of 0.5
and 0.75mg/L despite there being no correlation between the documented chlorine residual levels
and chemicals consumed during those respective periods.
Testimony from Stan Koebel was that multiple PUC staff had been filling sample containers from the
PUC workshop which was down the line from Well 5 and labelled them as taken from other locations
in the network. During the inquiry, when he was asked to explain why sample bottles had been
submitted with the incorrect source information written on them, he answered, and I quote,
"simply convenience or just couldn't be bothered." One more point about the behavior of
both Stan and Frank Koebel that was uncovered in the investigation. Frank Koebel, on his brother's
instructions, altered the daily operating sheet for Well 7 on May 22-23 in an apparent
attempt to conceal from the MOE that Well 7 had been operating without a chlorinator for an
an extended period, and that demonstrates that they were fully aware that running without
a chlorinator was not an acceptable practice and yet they did it anyway.
The Walkerton PUC operators therefore in summary, firstly, set inadequate doses of chlorine
based on the water flows, secondly, they did not repair the faulty chlorination equipment
in a timely manner, thirdly, they didn't regularly monitor chlorine residual every
day. Fourthly, they made false entries in their daily logs, four days where readings
were not taken. Fifthly, they intentionally mislabeled locations that microbiological
samples were taken. And finally, they attempted to conceal facts after the event to protect
themselves. The operators were fully aware their practices did not follow the Ministry
of the Environment (MOE) guidelines and their directives. And having said that, the A&L
laboratory also failed by not reporting their findings of potentially unsafe drinking water
to the MOE. The A&L laboratory policy was only to send report results to their client
directly and there was no requirement to notify the MOE or the local medical officer of health
should they have found a problem in their tests. Mr Robert Deakin, the laboratory manager
at A&L claimed he was unaware of section 4.1.3 of the ODWO guideline stating that the lab
should notify the MOE District Office of indications of unsafe drinking water were they found.
On Wednesday 17 May, the alarm could have been raised by the A&L Laboratory alerting
the MOE or the MOH which would have resulted in a boil water notice being issued four days
earlier that would have significantly reduced the spread of the outbreak. But they didn't.
It's unclear how many lives would have been saved had that happened. However, there's
no question the death toll would not have been as high. So let's talk a little bit about E. coli
and what the problem with it is. Escherichia coli, or E. coli for short because it's a lot
easy to say technically O157:H7 was the primary pathogen. The other was Campylobacter
Jejuni which were the two bacteria then most responsible for the majority of deaths and
illnesses in this incident. Once infected with E. coli, the intestinal symptoms last for about four
days and can persist for longer. After 24 hours, bloody diarrhea is common and in some cases severe
abdominal pains and cramping. Generally, it resolves itself without treatment other than
just rehydration and the replacement of the body's electrolytes. However, for some people, particularly
children under five years of age and the elderly, E. coli infection can be far more serious,
causing hemolytic uremic syndrome, HUS, after five to ten days of infection, leading to anemia,
low platelet counts, and in some cases kidney failure. In the most extreme of cases,
these complications can result in death. Campylobacter Jejuni is the most common
variant and it was implicated in the Walkerton incident as well. And in that case, diarrhea
usually lasts 2-7 days with a significantly lower probability of fatality than for E. coli.
The report had many recommendations, 28 in fact, but we'll look at one specific one and four others
that fall broadly under the same key category. The first is recommendation 11, continuous
monitoring. From the report I quote, "The MOE should require continuous chlorine and
turbidity monitors for all groundwater sources that are under the direct influence of surface
water or that serve municipal populations greater than a size prescribed by the MOE."
This happened in 2000. So, a bit of history. In 1996, I worked at the Stanwell
Power Station. That's a 1.4GW baseload power plant outside of my hometown of Rockhampton,
in Queensland. When I joined, the so-called Effluent Outfall was being monitored with
a local data logger for monitoring a small number of water quality measurements. The
project I was asked to execute was for the continuous monitoring of both the inlet and
outlet of the Northern Stormwater Dam to bring the data back into the plant DCS, Distributed
Control System. At the time, the EPA was requiring hourly water quality samples be taken, however,
the system that I installed would take multiple samples every single minute, far exceeding
the requirements.
That was 4 years before the Walkerton incident and the community of Stanwell had 1/3rd
of the number of residents living there.
The point though is that continuous monitoring using either a local data logger or a centralised
SCADA system was well and truly tested and available technology that was not that expensive
that could have been fitted easily into the Walkerton water treatment system had they
wanted to.
In the past 20 years working in water treatment facilities of all different sizes, but particularly
in South East Queensland, I've never seen a system that relied solely on manual measurements,
except occasional cross-checking for equipment calibration.
Better safe than sorry.
Moving on to the other recommendations, of which there's four: 20, 21, 22 and 23, and
they all broadly discuss training. So I'll read each and then I'll summarise and then
I'll summarise all of them at the end. Recommendation 20, I quote, "The government should require
all water system operators, including those who now hold certificates voluntarily obtained
through the grandparenting process, to become certified through examination within two years
and to be periodically recertified." So yes, please ensure the people that are in charge
of the plant are actually certified to do so and you have two years to get it done by
the way and plan to recertify them every so often, that's a good idea, you should get
on that and spoiler alert, they did following the incident.
Recommendation 21, the materials and I quote, "The materials for the water operator course
examinations and continuing education courses should emphasize in addition to the technical
requirements necessary for performing the functions of each class of operator" and
part is in italics, "the gravity of the public health risks" back to normal text, "associated
with a failure to treat and or monitor drinking water properly, the need to seek appropriate
assistance when such risks are identified and the rationale for and importance of regulatory measures
designed to prevent or identify those public health risks." So in other words, make sure your
operators understand that they could kill people if they don't do their jobs properly. A little dose
of fear when you're dosing chlorine goes a long way.
Recommendation 22:
I quote, "The government should amend Ontario Regulation 435/93 to define 'training'
clearly for the purposes of 40 hours of annual mandatory training with an emphasis on the
subject matter described in Recommendation 21."
End quote.
Now this is subtle, but it's really important.
I'm an RPEQ, Registered Professional Engineer in Queensland and a Chartered Professional
Engineer in Australia (CPEng), and
in order to maintain those qualifications and certifications, I'm required to undergo
recordable and audited Continuous Professional Development or CPD for short.
Now, that CPD could include training, but it stipulates that it must be training that's
relevant to my discipline amongst other things.
It's not like I spoke to this guy in the corridor and he taught me how chlorine works so I'm
like trained now.
No, it needs to be structured, reviewed, relevant training that's recorded and tested, otherwise
there's no point.
And when I write down my CPD, I guarantee you, Engineers Australia, check it.
Alright, Recommendation 23, last one.
I quote, "The government should proceed with the proposed requirement that operators
undertake 36 hours of MOE-approved training every three years as a condition of certification
or renewal. Such
courses should include training in emergency issues with water treatment and pathogen risks,
emergency and contingency planning, the gravity of public health risks associated with the failure
to treat and/or monitor drinking water properly, the need to seek appropriate assistance when such
risks are identified, and the rationale for and importance of regulatory measures designed to
prevent or identify public health risks." That was a long couple of sentences but this kind of
repeats and expands on the previous three points which I think probably could have been worded in
a more intertwined way but in essence yes, make the training regular: 3 years.
The key training points are very good though.
They're focused on abnormal operation, how to deal with emergencies and yes let's remind
them again and again that they could kill people if they don't do their job properly.
To wrap up on training we did actually speak about that on Episode 27 about Gare de Lyon
and it's worth repeating here.
When operators are asked to operate any kind of plant, they need to be taught the consequences
of incorrect operation.
And whilst it sounds obvious for something like water treatment, we all drink water,
and hence we could make a lot of people sick or even kill them if we make mistakes in how
we treat, or in this case don't treat, our water before it's consumed.
People think that training's about learning how to do something correctly over and over
and over, and yeah, that's part of it...
but the most important part of operator training isn't how to start it up, shut it down, or
run or test or maintain it necessarily. It's how you handle upsets, unplanned activities,
worst case scenarios, and in this case an E. coli outbreak. Training in this case would
have been as soon as the results came back and they were bad, shut it down. Shut it all
down. Warn people. But they didn't. Understanding the importance of the laboratory testing as
a measure of water quality rather than thinking that you can tell there's E. coli in the water
just by tasting it, this is a huge knowledge and competency gap that's honestly very hard
to fathom.
So it brings me to the final question, possibly the most important question in this whole
incident.
How the hell did Stan and Frank Koebel end up running a water utility in the first place?
So let's talk about Stan for a second.
Stan was a certified class 3 operator of a water distribution system.
He joined Walkerton PUC in 1972 when he was 19 years old.
His father was the foreman of the Walkerton Works Department at the time and he had an
11th grade education.
For the first 4 years of his career, he worked under Ian MacLeod, the then General
Manager of the PUC, before changing to Electrical Supply and Distribution, completing a linesman
apprenticeship.
In 1981, he was promoted to foreman and was responsible for both water and electricity
at PUC, and when Mr. McLeod retired in 1988, he was promoted to the General Manager position.
The only course Stan Koebel attended following the most recent promotion was a leadership
training course.
In 1987, the MOE introduced a grandfathering program for water operators regarding their
certifications.
For those unfamiliar, a grandfather policy is a provision in which an old rule continues
to apply to some existing situations while a new rule will apply to all future situations.
Those exempt from the new rule are said to have grandfathered rights or acquired rights
or to have been grandfathered in, depending on who you speak to.
In the context of the certification, in this case, operators were deemed through experience
to have implicit certification through demonstrated capability and therefore could be safely granted
a certification using experience as their sole measure for qualification.
I'll talk about that a little bit more in a minute.
Back to Stan Koebel.
So at the time, Mr. McLeod submitted Stan Koebel's name to the MOE as he had been certified
as a Class 2 operator, although he had never been required to pass an examination.
He had been recertified as a Class 3 when in 1996 the Walkerton water system was reclassified
as Class 3.
Again, without any MOE assessment of knowledge or skills.
During the testimony, Stan Koebel stated that he did not know what E. coli was, nor of its
implications to human health.
He did not fully understand turbidity or organic Nitrogen.
Consequently, he did not always fully comprehend portions of MOE inspection reports and correspondence.
That's not good.
Frank Koebel.
In 1983, he completed courses to qualify as a journeyman linesman at the Ontario Hydro
Training Centre.
Prior to 1988, approximately one quarter of Frank's time was spent working on hydroelectricity
with the remainder on the water system.
In 1988, he was promoted to foreman in the same time period that his brother was promoted
to general manager.
Frank obtained his Class 2 certification via grandfathering and later his Class 3 without
being required to complete any courses, with no competency testing or examinations just
like his brother. During testimony, Frank Koebel also admitted to many knowledge gaps
that matched Stan's, however, additionally, he was unaware of what Total Chlorine was
(he didn't know what Free Chlorine was) nor was he aware of the Chlorination Bulletin
nor Ontario Regulation 435/93 regarding requirements for the licensing and competency of operators.
In the entirety of his 25 years he worked at Walkerton PUC, Frank Koebel admitted he
had never attended a single training course about chlorination in any form.
So let's talk about the fallout.
The Ontario Government paid more than $72M Canadian dollars just in compensation
to the victims of the incident and their families, and the total economic impact of the incident
was approximately $155M.
The former manager of Walkerton's Utilities Commission, Stan Koebel, was jailed for one
year for his role in this incident.
The former foreman and Stan Koebel's brother, Frank Koebel, was sentenced to nine months
of house arrest.
A total of 10,189 claims were made, with 9,275 qualifying for compensation.
After 7 months since the Boil Water Advisory and at a cost of $11 million, the Ontario
Clear Water Association finally announced the water was once again safe to drink.
Despite that announcement, it took residents many years before they trusted the town water
supply again, with many choosing to stick with bottled water instead.
So what do we conclude from all of this?
The depth of the ignorance, laziness and careless disregard for common sense is almost laughable
if they hadn't managed to kill people and make approximately 2,320 people sick, which
was effectively half the town's population.
There are definitely some similarities to Flint, Michigan insofar as the operators didn't
understand what they were doing.
They fit the dictionary definition of 'incompetent' and there's a link in the show notes if you
don't believe me.
Grandfathering a certification on the basis of demonstrated experience isn't a very good
idea.
If someone is competent through experience, then surely they wouldn't mind sitting a short
test maybe.
What's Free Chlorine for water treatment?
I don't know.
It could be relevant.
The logical flaw where experience is used as a sole indicator of competence is this:
"Just because you've been doing something for 25 years and you're very consistent at it
because you've had lots of practice, that just might mean you're doing it consistently
badly or wrong for 25 years..."
That's all it tells you.
Experience matters.
Yes, it does.
Of course it does, but it has to be practically demonstrated.
Beware anybody that opens up with the line, "I have 25 years of experience," and then
demands respect.
It doesn't work like that.
The fundamental problem I have with situations like this is the promotion of the wrong people.
If a company or a utility where nothing has gone wrong for many decades has a key player
leave and that person has been a key reason why there have been no incidents and no issues
for decades, and they could leave because they retire or they're just or they're downsized,
a few years after that happens, that's when we start to see incidents occurring.
Why is that?
Some people call it a 'Brain Drain' or the 'Grey Drain', but it's more subtle than that.
If you don't know enough about the detail of the technical content of the role you're hiring someone for,
then you don't know what skills and technical knowledge that they need to have in order to function in that role.
So you don't know what training they need either.
So then the next generation of people in that role then compound the problem
by subsequently hiring more people that equally don't know what they need to know
because their new manager doesn't know what they needed to know. The cycle then spirals out of
control until you end up with an incompetent organization and incidents happen and people
die. In succession promotions like this, familiarity with someone already in the organization can put
someone into a role without anyone asking relevant questions about their capabilities
like, "We know Bob. He's been here for years and he's awesome. Now, let's just let him run a nuclear
reactor in manual for an hour, it'll be fine." Hmmm...
There are more jobs out there than you might think where all it takes is one act of incompetence
in the right alignment of events and someone will be injured or become sick or be killed.
Now, if you're in a role and you're not sure about something, ask someone.
Talk to people in similar roles like yours.
Go to conferences if you can.
Be curious.
Why do we dose Hypo?
Why does that matter?
Why don't we run the generator at this frequency?
Why do we need to sync to the grid before we close the circuit breaker?
You might be surprised how people just asking simple questions can break through this kind
of incompetency malaise.
It's hard to see, in the case of Walkerton, what could have been done differently to prevent
this incident without going back to that grandfathering clause.
Maybe the way to look at it is like this.
How are you certified?
How are others you work with certified?
And ask yourself, is experience alone enough?
Spoiler alert, it isn't.
If you're enjoying Causality and want to support the show, you can by supporting our sponsors
or by becoming a Premium Subscriber.
You can find details at https://engineered.network/causality with a thank you to all of our Patrons and
Premium Subscribers and a special thank you to our Patreon Silver Producers Carsten Hansen,
John Whitlow, Joseph Antonio, and Kevin Koch.
And an extra special thank you to our Patreon Gold Producer known only as R. Premium subscribers
and patrons have access to early release, high quality ad-free episodes, as well as
bonus episodes and to Causality Explored.
You can do this via Patreon, or if you prefer, via our website.
Visit https://engineered.network/causality to learn how you can help this show continue to be
made.
Causality is heavily researched and links to all the materials used for the creation
of this episode are contained in the show notes. You can find them in the text of the
episode description of your podcast player or on our website. You can follow me on the
Fediverse at chidgey@engineered.space, on Twitter @johnchidgey or the network on Twitter @Engineered_Net.
This was Causality. I'm John Chidgey. Thanks so much for listening.
[MUSIC]
(somber music)
(gentle music)
(dramatic music)
[Music]
(dramatic music)
(gentle music)
[Music]
(gentle music)
(music)
(gentle music)
[Music]
Many thanks to listener John Paul from Ontario, Canada for writing in and requesting this topic and bringing it to my attention.
Good luck in your studies, John. I hope they're going well.