Causality 32: Walkerton

[Music] Chain of events, cause and effect. We analyze what went right and what went wrong, as we discover that many outcomes can be predicted, planned for, and even prevented. I'm John Chidgey, and this is Causality. Causality is supported by you, our listeners. If you'd like to support the show, you can by becoming a Premium Subscriber. Premium subscribers have access to early release high quality ad free episodes, as well as bonus episodes and to Causality Explored. You can do this via Patreon or, if you prefer, via our website, visit https://engineered.network/causality to learn how you can help this show to continue to be made. Thank you. Walkerton Walkerton, Ontario, Canada has a population of about 5,000 people and lies approximately 2 hours drive from Toronto, which by contrast is Canada's largest metropolitan city at 6 million people. In the early 2000s, the water supply to Walkerton was supplied by 3 groundwater wells numbered 5, 6 and 7. Well 5 was 15 metres deep, capable of supplying 56% of the town's water supply needs in isolation with Well 6, 72 meters deep with a maximum supply of 52% in isolation and Well 7 was 76 meters deep the deepest with a maximum supply of 140% of the town's water supply needs. Well 5 used sodium hypochlorite solution for chlorine dosing whereas Wells 6 and 7 both used chlorine gas for disinfection. The water distribution system had approximately 42 kilometres or 26 miles of water mains with two standpipes providing pressure equalisation and approximately 20 hours of reserved storage. A basic control system and SCADA controlled and monitored the wells and standpipes. The Walkerton Public Utilities Commission or PUC for short were charged with the safe operation and maintenance of the water supply system in Walkerton. So let's talk about the incident itself. On the 8th of May 2000, a series of storms and steady rain over a five-day period totaled some 134 millimeters of rainfall. The volume of rainfall accumulation led to inevitable surface saturation and subsequent runoff which led to some minor localized flooding in the area. On the 12th of May some of that rainfall runoff entered Well 5. On Monday the 15th of May Stan Koebel returned on shift and noted that well seven was not operating. Every Monday, and this was no different, the operators collected their weekly samples from each of the wells and from key sample points in the distribution system. Around this time a construction project was was underway that required the installation of 615 metres or 2,000 feet of replacement water mains on Highway 9 in South West Walkerton between Wallace Street and Circle Drive. On Wednesday 17 May at 9:14am the A&L Laboratory faxed the results from the Highway 9 project water samples to the PUC. All three samples indicated positive for total coliforms and E. coli. At 2:37pm that afternoon, the remaining tests from the A&L lab were faxed to the PUC with a sample labeled Well 7 treated, positive for total coliform and E. coli. Further, the tests indicated coliform bacteria greater than 200 CFU/100mL, E. coli greater than 200 CFU/100mL, and a Heterotrophic plate count of 600 CFU/mL. By Thursday, the 18th of May, the number of illnesses had increased significantly, with a seven-year-old and nine-year-old admitted to the Owen Sound Hospital, and about 20 students from the Mother Teresa School reported in sick. Members of the public, including concerned parents, had contacted the Walkerton PUC to confirm the water was safe to drink, however were not told that anything was wrong. By Friday 19 May, 8 people had a documented three-day history of symptoms, with now more than 25 absent from the Mother Teresa School, 8 from Walkerton Public, and 3 residents from the Maple Court Villa retirement home were also affected. Dr Kristen Hallett, a paediatrician from the Grey-Bruce Health Services, had two patients referred to her from the hospital with similar symptoms. At approximately 9:00am that day, Dr Hallett contacted Dr Murray Quigg, the local medical officer of health (MHO) to inform him that her food history investigation of those patients indicated contaminated water was the most likely cause, with E. coli the most likely pathogen. During that day, James Schmidt, the public health inspector in Walkerton, received multiple calls and proceeded to call Mr. Koebel at 2:21pm directly, asking him about any issues with the water supply, to which Stan Koebel indicated he thought the water was 'OK'. On Sunday, the 21st of May, at approximately 1:30pm, a public health advisory was issued by the Health Unit to the Walkerton community not to drink municipal water from the tap, recommending boiling all water before consumption. The MHO also took their own independently collected water samples from multiple locations in Walkerton and their results on the 23rd of May all showed E. coli contamination, leading to all schools being closed the following day. On the 25th of May, the Regional Police Force directed the Ontario Provincial Police to begin a criminal investigation into the incident. With incidents such as these, the health impacts can take weeks, months or even years to fully play out, and those most at risk are our most vulnerable in society:€“ children, the elderly, and those with pre-existing medical conditions that are ill-equipped to fight off an illness like this. The number of people killed directly or indirectly due to this incident has been debated since shortly after the incident, with either two, three or four others indirectly linked, and not all are conclusively proven to have been linked to the incident though there is strong evidence to suggest that they were. The following people lost their lives either in whole or in part due to this incident. Melville Dawe 69 years old May 19th. Lenore Al 66 years old, May 22nd. Mary Rose Raymond 2yrs old, May 23rd. Robert Brodie 89yrs old, May 24th. Edith Pearson 82yrs old, also May 24th. Vera Coe 75yrs old, also May 24th. Laura Rowe 84yrs old, on May 29th. Betty Trushinski 56yrs old, May 31st. So what on earth went wrong? The Honourable Dennis R O'Connor was appointed to lead the Walkerton Commission into this incident, producing a final report in two parts, the first of which was released in January 2002. The source of the contamination was found to be Well 5, with runoff from a farmer's paddock, with the fecal coliforms from the livestock excrement being washed into the drinking water. The investigation unsurfaced several disturbing behaviors and events leading up to the incident. Prior to the rainfall event on the 5th of May, Stan Koebel left Walkerton to attend a conference in Windsor for which he was away until the 14th of May during which time the rainfall event had occurred. In his absence his brother Frank Koebel was in charge of the PUC in Walkerton. When Stan left for the conference he was aware that the chlorinator on Well 7 was not functioning correctly since Well 7 was brought back into service on the 2nd of May. In fact the well hadn't been in service since the 10th of March. It wasn't uncommon to rotate water supply from each of the wells. But rather than shut Well 7 off, Stan Koebel instructed Frank to replace the chlorinator in Well 7 with the replacement unit that had been on the PUC premises in Walkerton for nearly 1-1/2yrs. Upon Stan's return he found that Frank had still not fitted the replacement chlorinator to Well 7 and that Well 7 had still been running during that time. Well 7 pumped unchlorinated water into the system from the 3rd of May to the 9th of May as it the only well being used during that period, there was no new chlorine being injected into the water main system for that time. The correct course of action would have been to leave Wells 5 and/or 6 running while the chlorinator in Well 7 was replaced and then to return Well 7 to service, an activity that would have taken about a day of end-to-end activities to complete...absolute maximum. Chlorine residual needs to be maintained to ensure that any bacteria are killed before the water is consumed, but that's something we'll "explore" separately. Despite the fact that both Stan and Frank Koebel were aware that chlorination was required at all times as mandated in the Ontario Drinking Water Objectives and Bulletin 65-W-4, "Chlorination of Potable Water Supplies", when interviewed following the event, they believed that unchlorinated water from Well 7 was safe because it was from a deeper well. In addition, PUC staff would regularly drink raw unchlorinated water at the well because it was cold, clear and clean and "tasted better" than chlorinated water did. Multiple years of reinforcing the idea that it's safe to drink it today so it'll be safe to drink it tomorrow led to a mistaken belief that chlorination was in fact optional for Well 7. In the history of the plant, there had been no incidents like this, which also fed a mistaken belief that anything like this could actually happen. Every day operators were required to visit each well and make a recording of the following: the water flow totalizer, the chlorine chemical usage and the current chlorine residual. On the 13th of May at 4:10pm 0.75mg/L of chlorine concentration was recorded for Well 5. There was no entry for Well 6. By cross-checking the water volume recorded against the amount of Hypochlorite dosed during that period it was calculated that it was completely impossible to have a chlorine residual that high given how little Hypo was dosed during that period. The investigation also found that for more than 20 years it had been regular practice for PUC operators to not measure the actual chlorine residual but instead write down a fictitious value to put an entry in the box. Reviews of the logs showed a significant number of readings of 0.5 and 0.75mg/L despite there being no correlation between the documented chlorine residual levels and chemicals consumed during those respective periods. Testimony from Stan Koebel was that multiple PUC staff had been filling sample containers from the PUC workshop which was down the line from Well 5 and labelled them as taken from other locations in the network. During the inquiry, when he was asked to explain why sample bottles had been submitted with the incorrect source information written on them, he answered, and I quote, "simply convenience or just couldn't be bothered." One more point about the behavior of both Stan and Frank Koebel that was uncovered in the investigation. Frank Koebel, on his brother's instructions, altered the daily operating sheet for Well 7 on May 22-23 in an apparent attempt to conceal from the MOE that Well 7 had been operating without a chlorinator for an an extended period, and that demonstrates that they were fully aware that running without a chlorinator was not an acceptable practice and yet they did it anyway. The Walkerton PUC operators therefore in summary, firstly, set inadequate doses of chlorine based on the water flows, secondly, they did not repair the faulty chlorination equipment in a timely manner, thirdly, they didn't regularly monitor chlorine residual every day. Fourthly, they made false entries in their daily logs, four days where readings were not taken. Fifthly, they intentionally mislabeled locations that microbiological samples were taken. And finally, they attempted to conceal facts after the event to protect themselves. The operators were fully aware their practices did not follow the Ministry of the Environment (MOE) guidelines and their directives. And having said that, the A&L laboratory also failed by not reporting their findings of potentially unsafe drinking water to the MOE. The A&L laboratory policy was only to send report results to their client directly and there was no requirement to notify the MOE or the local medical officer of health should they have found a problem in their tests. Mr Robert Deakin, the laboratory manager at A&L claimed he was unaware of section 4.1.3 of the ODWO guideline stating that the lab should notify the MOE District Office of indications of unsafe drinking water were they found. On Wednesday 17 May, the alarm could have been raised by the A&L Laboratory alerting the MOE or the MOH which would have resulted in a boil water notice being issued four days earlier that would have significantly reduced the spread of the outbreak. But they didn't. It's unclear how many lives would have been saved had that happened. However, there's no question the death toll would not have been as high. So let's talk a little bit about E. coli and what the problem with it is. Escherichia coli, or E. coli for short because it's a lot easy to say technically O157:H7 was the primary pathogen. The other was Campylobacter Jejuni which were the two bacteria then most responsible for the majority of deaths and illnesses in this incident. Once infected with E. coli, the intestinal symptoms last for about four days and can persist for longer. After 24 hours, bloody diarrhea is common and in some cases severe abdominal pains and cramping. Generally, it resolves itself without treatment other than just rehydration and the replacement of the body's electrolytes. However, for some people, particularly children under five years of age and the elderly, E. coli infection can be far more serious, causing hemolytic uremic syndrome, HUS, after five to ten days of infection, leading to anemia, low platelet counts, and in some cases kidney failure. In the most extreme of cases, these complications can result in death. Campylobacter Jejuni is the most common variant and it was implicated in the Walkerton incident as well. And in that case, diarrhea usually lasts 2-7 days with a significantly lower probability of fatality than for E. coli. The report had many recommendations, 28 in fact, but we'll look at one specific one and four others that fall broadly under the same key category. The first is recommendation 11, continuous monitoring. From the report I quote, "The MOE should require continuous chlorine and turbidity monitors for all groundwater sources that are under the direct influence of surface water or that serve municipal populations greater than a size prescribed by the MOE." This happened in 2000. So, a bit of history. In 1996, I worked at the Stanwell Power Station. That's a 1.4GW baseload power plant outside of my hometown of Rockhampton, in Queensland. When I joined, the so-called Effluent Outfall was being monitored with a local data logger for monitoring a small number of water quality measurements. The project I was asked to execute was for the continuous monitoring of both the inlet and outlet of the Northern Stormwater Dam to bring the data back into the plant DCS, Distributed Control System. At the time, the EPA was requiring hourly water quality samples be taken, however, the system that I installed would take multiple samples every single minute, far exceeding the requirements. That was 4 years before the Walkerton incident and the community of Stanwell had 1/3rd of the number of residents living there. The point though is that continuous monitoring using either a local data logger or a centralised SCADA system was well and truly tested and available technology that was not that expensive that could have been fitted easily into the Walkerton water treatment system had they wanted to. In the past 20 years working in water treatment facilities of all different sizes, but particularly in South East Queensland, I've never seen a system that relied solely on manual measurements, except occasional cross-checking for equipment calibration. Better safe than sorry. Moving on to the other recommendations, of which there's four: 20, 21, 22 and 23, and they all broadly discuss training. So I'll read each and then I'll summarise and then I'll summarise all of them at the end. Recommendation 20, I quote, "The government should require all water system operators, including those who now hold certificates voluntarily obtained through the grandparenting process, to become certified through examination within two years and to be periodically recertified." So yes, please ensure the people that are in charge of the plant are actually certified to do so and you have two years to get it done by the way and plan to recertify them every so often, that's a good idea, you should get on that and spoiler alert, they did following the incident. Recommendation 21, the materials and I quote, "The materials for the water operator course examinations and continuing education courses should emphasize in addition to the technical requirements necessary for performing the functions of each class of operator" and part is in italics, "the gravity of the public health risks" back to normal text, "associated with a failure to treat and or monitor drinking water properly, the need to seek appropriate assistance when such risks are identified and the rationale for and importance of regulatory measures designed to prevent or identify those public health risks." So in other words, make sure your operators understand that they could kill people if they don't do their jobs properly. A little dose of fear when you're dosing chlorine goes a long way. Recommendation 22: I quote, "The government should amend Ontario Regulation 435/93 to define 'training' clearly for the purposes of 40 hours of annual mandatory training with an emphasis on the subject matter described in Recommendation 21." End quote. Now this is subtle, but it's really important. I'm an RPEQ, Registered Professional Engineer in Queensland and a Chartered Professional Engineer in Australia (CPEng), and in order to maintain those qualifications and certifications, I'm required to undergo recordable and audited Continuous Professional Development or CPD for short. Now, that CPD could include training, but it stipulates that it must be training that's relevant to my discipline amongst other things. It's not like I spoke to this guy in the corridor and he taught me how chlorine works so I'm like trained now. No, it needs to be structured, reviewed, relevant training that's recorded and tested, otherwise there's no point. And when I write down my CPD, I guarantee you, Engineers Australia, check it. Alright, Recommendation 23, last one. I quote, "The government should proceed with the proposed requirement that operators undertake 36 hours of MOE-approved training every three years as a condition of certification or renewal. Such courses should include training in emergency issues with water treatment and pathogen risks, emergency and contingency planning, the gravity of public health risks associated with the failure to treat and/or monitor drinking water properly, the need to seek appropriate assistance when such risks are identified, and the rationale for and importance of regulatory measures designed to prevent or identify public health risks." That was a long couple of sentences but this kind of repeats and expands on the previous three points which I think probably could have been worded in a more intertwined way but in essence yes, make the training regular: 3 years. The key training points are very good though. They're focused on abnormal operation, how to deal with emergencies and yes let's remind them again and again that they could kill people if they don't do their job properly. To wrap up on training we did actually speak about that on Episode 27 about Gare de Lyon and it's worth repeating here. When operators are asked to operate any kind of plant, they need to be taught the consequences of incorrect operation. And whilst it sounds obvious for something like water treatment, we all drink water, and hence we could make a lot of people sick or even kill them if we make mistakes in how we treat, or in this case don't treat, our water before it's consumed. People think that training's about learning how to do something correctly over and over and over, and yeah, that's part of it... but the most important part of operator training isn't how to start it up, shut it down, or run or test or maintain it necessarily. It's how you handle upsets, unplanned activities, worst case scenarios, and in this case an E. coli outbreak. Training in this case would have been as soon as the results came back and they were bad, shut it down. Shut it all down. Warn people. But they didn't. Understanding the importance of the laboratory testing as a measure of water quality rather than thinking that you can tell there's E. coli in the water just by tasting it, this is a huge knowledge and competency gap that's honestly very hard to fathom. So it brings me to the final question, possibly the most important question in this whole incident. How the hell did Stan and Frank Koebel end up running a water utility in the first place? So let's talk about Stan for a second. Stan was a certified class 3 operator of a water distribution system. He joined Walkerton PUC in 1972 when he was 19 years old. His father was the foreman of the Walkerton Works Department at the time and he had an 11th grade education. For the first 4 years of his career, he worked under Ian MacLeod, the then General Manager of the PUC, before changing to Electrical Supply and Distribution, completing a linesman apprenticeship. In 1981, he was promoted to foreman and was responsible for both water and electricity at PUC, and when Mr. McLeod retired in 1988, he was promoted to the General Manager position. The only course Stan Koebel attended following the most recent promotion was a leadership training course. In 1987, the MOE introduced a grandfathering program for water operators regarding their certifications. For those unfamiliar, a grandfather policy is a provision in which an old rule continues to apply to some existing situations while a new rule will apply to all future situations. Those exempt from the new rule are said to have grandfathered rights or acquired rights or to have been grandfathered in, depending on who you speak to. In the context of the certification, in this case, operators were deemed through experience to have implicit certification through demonstrated capability and therefore could be safely granted a certification using experience as their sole measure for qualification. I'll talk about that a little bit more in a minute. Back to Stan Koebel. So at the time, Mr. McLeod submitted Stan Koebel's name to the MOE as he had been certified as a Class 2 operator, although he had never been required to pass an examination. He had been recertified as a Class 3 when in 1996 the Walkerton water system was reclassified as Class 3. Again, without any MOE assessment of knowledge or skills. During the testimony, Stan Koebel stated that he did not know what E. coli was, nor of its implications to human health. He did not fully understand turbidity or organic Nitrogen. Consequently, he did not always fully comprehend portions of MOE inspection reports and correspondence. That's not good. Frank Koebel. In 1983, he completed courses to qualify as a journeyman linesman at the Ontario Hydro Training Centre. Prior to 1988, approximately one quarter of Frank's time was spent working on hydroelectricity with the remainder on the water system. In 1988, he was promoted to foreman in the same time period that his brother was promoted to general manager. Frank obtained his Class 2 certification via grandfathering and later his Class 3 without being required to complete any courses, with no competency testing or examinations just like his brother. During testimony, Frank Koebel also admitted to many knowledge gaps that matched Stan's, however, additionally, he was unaware of what Total Chlorine was (he didn't know what Free Chlorine was) nor was he aware of the Chlorination Bulletin nor Ontario Regulation 435/93 regarding requirements for the licensing and competency of operators. In the entirety of his 25 years he worked at Walkerton PUC, Frank Koebel admitted he had never attended a single training course about chlorination in any form. So let's talk about the fallout. The Ontario Government paid more than $72M Canadian dollars just in compensation to the victims of the incident and their families, and the total economic impact of the incident was approximately $155M. The former manager of Walkerton's Utilities Commission, Stan Koebel, was jailed for one year for his role in this incident. The former foreman and Stan Koebel's brother, Frank Koebel, was sentenced to nine months of house arrest. A total of 10,189 claims were made, with 9,275 qualifying for compensation. After 7 months since the Boil Water Advisory and at a cost of $11 million, the Ontario Clear Water Association finally announced the water was once again safe to drink. Despite that announcement, it took residents many years before they trusted the town water supply again, with many choosing to stick with bottled water instead. So what do we conclude from all of this? The depth of the ignorance, laziness and careless disregard for common sense is almost laughable if they hadn't managed to kill people and make approximately 2,320 people sick, which was effectively half the town's population. There are definitely some similarities to Flint, Michigan insofar as the operators didn't understand what they were doing. They fit the dictionary definition of 'incompetent' and there's a link in the show notes if you don't believe me. Grandfathering a certification on the basis of demonstrated experience isn't a very good idea. If someone is competent through experience, then surely they wouldn't mind sitting a short test maybe. What's Free Chlorine for water treatment? I don't know. It could be relevant. The logical flaw where experience is used as a sole indicator of competence is this: "Just because you've been doing something for 25 years and you're very consistent at it because you've had lots of practice, that just might mean you're doing it consistently badly or wrong for 25 years..." That's all it tells you. Experience matters. Yes, it does. Of course it does, but it has to be practically demonstrated. Beware anybody that opens up with the line, "I have 25 years of experience," and then demands respect. It doesn't work like that. The fundamental problem I have with situations like this is the promotion of the wrong people. If a company or a utility where nothing has gone wrong for many decades has a key player leave and that person has been a key reason why there have been no incidents and no issues for decades, and they could leave because they retire or they're just or they're downsized, a few years after that happens, that's when we start to see incidents occurring. Why is that? Some people call it a 'Brain Drain' or the 'Grey Drain', but it's more subtle than that. If you don't know enough about the detail of the technical content of the role you're hiring someone for, then you don't know what skills and technical knowledge that they need to have in order to function in that role. So you don't know what training they need either. So then the next generation of people in that role then compound the problem by subsequently hiring more people that equally don't know what they need to know because their new manager doesn't know what they needed to know. The cycle then spirals out of control until you end up with an incompetent organization and incidents happen and people die. In succession promotions like this, familiarity with someone already in the organization can put someone into a role without anyone asking relevant questions about their capabilities like, "We know Bob. He's been here for years and he's awesome. Now, let's just let him run a nuclear reactor in manual for an hour, it'll be fine." Hmmm... There are more jobs out there than you might think where all it takes is one act of incompetence in the right alignment of events and someone will be injured or become sick or be killed. Now, if you're in a role and you're not sure about something, ask someone. Talk to people in similar roles like yours. Go to conferences if you can. Be curious. Why do we dose Hypo? Why does that matter? Why don't we run the generator at this frequency? Why do we need to sync to the grid before we close the circuit breaker? You might be surprised how people just asking simple questions can break through this kind of incompetency malaise. It's hard to see, in the case of Walkerton, what could have been done differently to prevent this incident without going back to that grandfathering clause. Maybe the way to look at it is like this. How are you certified? How are others you work with certified? And ask yourself, is experience alone enough? Spoiler alert, it isn't. If you're enjoying Causality and want to support the show, you can by supporting our sponsors or by becoming a Premium Subscriber. You can find details at https://engineered.network/causality with a thank you to all of our Patrons and Premium Subscribers and a special thank you to our Patreon Silver Producers Carsten Hansen, John Whitlow, Joseph Antonio, and Kevin Koch. And an extra special thank you to our Patreon Gold Producer known only as R. Premium subscribers and patrons have access to early release, high quality ad-free episodes, as well as bonus episodes and to Causality Explored. You can do this via Patreon, or if you prefer, via our website. Visit https://engineered.network/causality to learn how you can help this show continue to be made. Causality is heavily researched and links to all the materials used for the creation of this episode are contained in the show notes. You can find them in the text of the episode description of your podcast player or on our website. You can follow me on the Fediverse at [email protected], on Twitter @johnchidgey or the network on Twitter @Engineered_Net. This was Causality. I'm John Chidgey. Thanks so much for listening. [MUSIC] (somber music) (gentle music) (dramatic music) (dramatic music) [Music] (dramatic music) (dramatic music) (gentle music) [Music] (gentle music) (music) (gentle music) (gentle music) [Music] Many thanks to listener John Paul from Ontario, Canada for writing in and requesting this topic and bringing it to my attention. Good luck in your studies, John. I hope they're going well.