The evening before Christmas Eve in 2015 saw a widespread blackout of the power grid across the Ukraine. We look at how a cyber-attack on electric utility companies in Europe, changed how cyber-security is regarded in control systems, forever.
In Causality Explored, (Premium ONLY) we dive into GOOSE as a messaging protocol in High Voltage Circuit Breaker protection, why it's important and why it's opened a door for hackers that didn't exist 20 years ago.
Chain of events, cause and effect, we analyze what went right and what went wrong as we discover that many outcomes can be predicted, planned for, and even prevented. I'm John Chidgey and this is Causality. Causality is supported by you: our listeners. If you'd like to support the show you can by becoming a Patron. Patrons have access to early release high quality ad free episodes as well as other bonus material you can do this via Patreon just visit engineered.network/ causality to learn how you can help this show to continue to be made. Thank you! "Big Dig" This incident is often referred to as the "Big Dig" collapse or the "Big Dig" tunnel ceiling collapse despite the fact it had nothing directly to do with the Big Dig in its entirety but rather the incident related more to a single exit portal. The Big Dig itself was a large civilian construction project that took 15 years from start to finish, though the project was technically referred to as the C.A.T. Project or the Central Artery Tunnel Project. The project took twice as long as the original estimate. Cost $14.6B US Dollars which was more than double the 1996 $1B dollar estimate and 5 times the $3B original project estimate in 1983. Traffic congestion in Boston had long been a problem as the city had grown from a time before cars existed with some studies in the 80s suggesting that by 2010 Bostons rush hour could last approximately 16 hours every single day. To address this a new bridge and two key tunnels were proposed to be built as a concerted set of projects which were collectively referred to as the "Big Dig". The first tunnel was named the Ted Williams tunnel and goes beneath Boston harbour taking interstate 90 (the I90) traffic from South Boston to Boston Logan International Airport. Construction began in 1991 and was completed in 1995. The second tunnel was located below the Fort Point Channel and formed part of an I90 connector with the I93 interchange. It would effectively replace the elevated I93 Central Artery which had become deteriorated with age; extend the I90 also called the Massachusetts Turnpike or "Mass-Pike" to the Logan International Airport providing an interchange for the I90 and the I93 and also replace the I93 bridge over the Charles River. In 1985 the department of public works entered into an agreement with Bechtel Parsons Bricknehoff hence referred to as BPB, a joint venture formed between Bechtel Corporation and Parsons Bricknerhoff, Quade and Douglas to prepare a preliminary project management plan. In 1990 congress allocated $755M US dollars to the massive highway improvement project, and a year later the Federal Highway Administration gave its approval to move ahead. A massive civil construction project required significant and expensive traffic sequencing during construction. Traffic sequencing is a term sometimes used in road construction where traffic is directed via various detours, temporary roads, road direction reassignments and so on in accordance with a traffic management plan for the road construction project. In order to coordinate the construction sequence the approved traffic plan required a temporary above ground ramp to carry traffic from D Street to the entrance of the Ted Williams Tunnel in order to provide access under Boston Harbour to the airport before work was finished on the remainder of the I90 connector. This would be achieved by constructing a portal under D Street that would eventually connect through but was completed first in 1993 well before the completion of either the Ted Williams Tunnel or the remainder of the I90 connector. For those not familiar a portal is the name used for tunnel entries and exits. The D Street portal... (yes Boston has a series of streets that are just an alphabetical letter... A, B, C, D Street and so on so there's that...) The D Street portal had a heavily reinforced concrete roof slab varying between 1.5 meters to 2 meters (that's 5 to 7ft) thick in order to accommodate the weight of a parking deck that was originally intended to be built over the portal. With that background let's talk about the incident itself. At approximately 11pm local Daylight Savings Time on Monday the 10th of July, 2006 an entire row consisting of 20 anchor bolts pulled free from a suspended ceiling segment in the D Street portal. The ceiling consisted of a series of four 3.6m (that's 12ft) wide by 2.4m (that's 8ft) long by 100mm or 4" thick 3 tonnes or 4,700lbs of reinforced concrete and steel panels. They provided segregation between the roadway and the ventilation system above that roadway. With an entire row no longer supporting that side of the ceiling panels they began to rotate downwards with the additional load of the rotated panels causing rapid fracturing of the concrete. One minute later a 1991 black Buick sedan driven by Mr Angel Del Valle, aged 46 with the sole passenger his wife Melina aged 38, was traveling eastbound in the I90 connector, en route to the Logan International airport to pick up Angel's brother and sister-in-law who returning from a trip to Puerto Rico. As the vehicle approached the exit portal four panels of the suspended concrete ceiling detached from the tunnel roof and fell onto the vehicle. The panels crushed the right-hand side of the vehicle's roof as the car came to rest against the Northern tunnel wall. Approximately 26 tonnes of concrete and steel had fallen onto both the vehicle and the roadway. Mr Del Valle was able to escape through the driver's side window sustaining only minor injuries. Melina was crushed and died instantly. Following the incident Mr Del Valle stated that although he saw the concrete panels starting to come down from the roof, he was unable to avoid them in time. Immediately following the incident the underground sections were closed to all traffic pending a complete inspection. The first ramp to fully reopen was on the 9th of August, but the entirety of all of the underground sections weren't fully reopened until nearly 12 months after the incident occurred. So what went wrong? The work scopes were split between many companies with the primary oversight being Bechtel Parsons Bricknerhoff and the Massachusetts Turnpike Authority, that's the MTA, who took an increasing oversight role towards the end of the project, as each component was gradually handed over. Bechtel Parsons Bricknerhoff were essentially the secondary approver for the design of the project as a whole as well as its constituent components. The D Street portal itself was constructed by Kiewit/Perini/Atkinson/ Cashman, a joint venture based in Boston, which was contracted by the Massachusetts department of public works to construct the land-side West tunnel approach to the Ted Williams Tunnel. The tunnel consisted of approximately 800 meters that's 2600ft of cut and cover and one section of depressed open highway as well as the D Street portal and a temporary ramp required for traffic sequencing during construction of the larger tunnels. The section design consultant for this was HDR Engineering Incorporated headquartered in Omaha, Nebraska. The roadway in the Eastbound D Street portal tunnel at the accident site consisted of two 12 foot or 3.6 meter wide travel lanes and along the South side of the tunnel an 18 foot or 5.4 meter wide acceleration lane. All tunnels beyond a set length require ventilation to ensure air quality is maintained and to correctly handle airflow during a fire emergency. During the original letting of the portal section no allowance had been made for ventilation as this component of the design was intended to be added in a later work scope. The portal design was, therefore, done in two parts and was executed by two different companies. The aforementioned joint venture constructed the cut and cover tunnel but the I90 connector tunnel "finishes" section design consultant which included the D Street portal finishing which included the ventilation system, was Gannett Fleming Incorporated headquartered in Camp Hill, Pennsylvania. For the suspended ceiling Gannett Fleming engineers followed the directions set in the design policy memorandum number 107 by Fay, Spofford and Throndike, Inc (that's FST for short) Howard, Needles, Tammen & Bergendoff (HNTB for short) using the embedded steel channel inserts instead of roof girders for the roof attachments. The D Street portal slab was not constructed in the same way as the rest of the tunnel, with additional slab thickness and reinforcing intended for a different goal and did not have any embedded steel channel inserts or anchor points embedded into it in its initial design. Hence the design was adjusted to use adhesive anchors with an epoxy resin. This technique was also used in other sections of the I90 Connector tunnel and on the HOV ramp, that's High Occupancy Vehicle or HOV. The ceiling in the D Street portal was installed by Modern Continental Construction Company Incorporated (Modern Continental for short) of Cambridge, Massachusetts. It was constructed between 1999 and April of 2000. The ceiling module that collapsed in this incident was installed in November 1999. Modern Continental installed the last ceiling module of the I90 Connector Tunnel in July 2002. Whilst the use of epoxy resins for anchoring applications was not unheard of it was unusual for a suspended load in a long-term application. Investigators used Fourier Transform Infrared Spectroscopy Headspace Gas Chromatography and Mass Spectroscopy of epoxy samples from the majority of the anchors that failed in the incident, as well as other randomly selected anchors to determine their chemical composition. In all cases it was found to be consistent with a fast set epoxy. Installation of the anchors in the D Street portal began in July 1999 using epoxy purchased from Newman Renner Colony. The investigations confirmed that the invoices indicate that Modern Continental purchased Power- fast, Fast Set NRC1000 Gold Epoxy during the period when the D Street portal ceiling was being installed. Further testing of this epoxy showed that anchors installed using best practices, and I quote, "...exhibited significant and continued displacement (aka creep) when subjected to loads as low as one thousand pounds. Anchors loaded to four thousand pounds completely separated from their anchor holes before the end of the 82-day test period..." end quote. The portal ceiling segment anchors had an expected maximum load of 2,600 pounds and based on the above tests, due to its susceptibility to creep was not a suitable epoxy for use in any long-term tension load application. The above tests however suggest that the ceiling should have failed sooner than it did. The likely reason it did not fail sooner is that the ventilation system was not in operation for some time after the physical installation. Until the ventilation system was operating the ceiling tiles would have presented their own independent gravitational load on the support anchors, their so-called dead weight or dead load. Once the ventilation system became operational the air pressure would increase and decrease cyclically due to changes in flow direction and pressure and would also have induced vibration into the material which would have then accelerated the creep. The investigators also tested standard set epoxy under the same conditions and found that in the 82 day test under a 2,000 pound load, Standard Set did creep at about 1/10th of the rate of Fast Set, and it did so in a flat linear increment. The Fast Set during that time exhibited a non-linear, with increasing rate of creep, after just 55 days. So let's talk about the epoxy. Epoxies are basic components that when combined and cured create an epoxy resin and are sometimes called Polyepoxides hence epoxy for short. Epoxy resin is combined with a hardener also sometimes called a curative or a co-reactant and when they are mixed the amine groups on the hardener molecules begin to link with the epoxide groups on the resin molecules releasing heat in the process. The reaction is therefore exothermic and creates a thermosetting polymer and unlike thermoplastic it resists reforming under applied heat and chemicals. Most resin molecules have an epoxide ring at each end and the harder molecules have an amine group at each end hence they both link in an alternating pattern to form long polymer chains and cross-link between polymer chains. The more cross-links that exist between the polymer chains the more resin resists significant rearrangement of the polymer molecules so the polymer cannot be melted and reformed. This type of polymer is called a Thermoset. As the crosslink polymer network forms the viscosity of the material increases and eventually the reaction slows and finally stops, although any unbonded reactants may not have been free to migrate through the solution if not well enough mixed beforehand leading to some unbonded reactive groups. Hence the faster the setting of the epoxy, the more likely uncured sections will exist and there will be less cross-linked polymers. So what is creep? Creep refers to a gradual continuing deformation of a material under a sustained load. It's similar to dislocations we discussed on Episode 35 about the San Bruno Gas Pipeline explosion. As a polymer, the stiffness of an epoxy is time and temperature dependent, hence if the load is applied suddenly the epoxy responds like a hard glassy solid. If a load is static, meaning it's held constantly against the plastic, the long chain polymer molecules have time to rearrange and slide past one another and the stiffness of the epoxy decreases to a range where it can be described as "rubbery." I know. Technical terms like "rubbery." Anyway. Polymer materials with an instantaneous elastic response followed by a slowly increasing deformation are referred to as Viscoelastic. The time needed for the glassy to rubbery transition depends on the molecular structure and the lengths of the polymer chains between crosslinks and also as the temperature increases the rate of rearrangement increases allowing the glassy-to-rubbery transition to occur more rapidly. In a well-formed cross-linked network of a Thermoset, the polymer molecules are prevented from moving very far, and once the rubbery state is reached, no further softening occurs. The design for the anchor resin provided by Gannett Fleming to Modern Continental specified the following, and I quote: "...Provide adhesive consisting of two component plastic resin and catalyst hardener mixture, resin material shall remain unaffected by continuous humidity and by chemicals present in a vehicle exhaust type of air duct environment..." (end quote) The specification goes on to include minimum design service loads, minimum factors of safety, and installation requirements that had to be met. The epoxy selected by Modern Continental used a two-part epoxy material, resin and hardener formulated by Sika Corporation of Lyndhurst, New Jersey packaged by Powers Fasteners Incorporated (formerly the Rawlplug Company Incorporated) of New Rochelle, New York and distributed by Newman Renner Colony LLC of Westwood, Massachusetts. Sika Corporation supplied the epoxy resin and hardener in bulk to Powers, which packaged, marketed and distributed it for Newman Renner Colony as NRC1000 a gold epoxy, which was used by Modern Continental on this project. In order to begin installation the construction contractor needed their final design to be approved by the consultants and Modern Continental submitted a total of four individual submissions for their anchor design. The first three anchor design submissions were rejected by Gannett Fleming, with Gannet Fleming requesting additional information each time. On the 30th of December, 1999 Modern Continental submitted the fourth anchor adequacy submittal. The anchor service load data in that submittal were the values calculated and specified by Gannett Fleming originally in their initial design. The anchor load capacity data was extracted from the powers design manual (second edition) and powers indicated the adhesive anchors were capable of supporting up to 6,350 pounds each in 4,000 psi concrete, while maintaining a safety factor of 4 against the ultimate load which in this case was 25,400 pounds. None of the four submissions included any specific reference to exactly which epoxy formulation was going to be used for the anchor. The fourth submission included a copy of a draft revision of ICBO ER-4514 dated October 1999 in response to a specific request by the Gannett Fleming engineer based on his review of a previous submission. ICBO is the International Conference of Building Officials, and ER-4514 was the ICBO's independent evaluation report for the Power-Fast adhesive anchor system from Powers. The draft report revision limited the use of the Power-Fast fast set epoxy to short-term loads such as those resulting from wind or earthquake forces. On the 17th of December, 1999 the anchor capacity structural calculations were certified by a registered professional engineer employed by Sigma Engineering International Incorporated of Lincoln, Rhode Island, noting that, and i quote: "...the calculations were performed to compare the anchor minimum design service loads per project specification with the allowable loads provided by the anchor bolt manufacturer only..." (end quote) In other words no accounting for creep and long-term use in this application was made. On the 7th of January, 2000 Gannett Fleming authorized the contract to proceed with the anchor installation per their design and this was later secondarily approved by B/PB in February 2000. Having said that, due to schedule pressures the contractor had already been installing the anchors prior to formal approval since July, 1999 as previously mentioned some 5 months before that formal approval came through. But hang on a minute! They found problems in 1999. On the 7th of October, 1999 Modern Continental's project manager contacted B/PB via a formal letter that they had become aware of problems involving "...a small percentage of adhesive anchors in the HOV ceiling mock-up." The ceiling module had been built in August that year and by October the hanger plates had displaced from the roof by about 1/2 an inch or 12mm prompting their letter. B/PB responded on the 12th of October suggesting that the turnbuckles in the ceiling module may have been causing excessive load on some supports, leading to creep. They also suggested incorrect fitting of the bolts and the epoxy by the constructor as another potential cause. Several meetings with Powers and Modern Continental ensued and were followed by a letter from Powers to Modern Continental on the 29th of October, 1999 reminding of the correct installation procedures but ultimately stating that since the design of the ceiling plate module and anchors were not done by Powers they could not address, and I quote: "...the different loads that may be acting on these anchors..." (end quote). On the 8th of November, 1999 Modern Continental wrote back to B/PB attaching Power's formal response and concluded amongst other comments with, and I quote: "...it is improbable that the anchors were overstressed as a result of erection loads or the erection procedures..." (end quote) So let's time out for a second. It's a "he said," "she said," kind of situation. Prove that these anchors were all installed correctly despite the fact that some of the anchors were installed in the presence of a B/PB engineer, unless every single one was that's not enough proof that the anchors weren't installed correctly. During retesting several failed but evidence was found where incorrect hole cleaning was applied and insufficient epoxy was used leading B/PB to issue Deficiency Report 001, and after many letters and emails back and forth with allegations of incorrect installation procedures being followed and in an email from the B/PB design manager to the project engineer and structural engineer internally to B/PB stating the following and I quote: "...we are not trying to hold up construction, we are trying to make a determination that the installation is safe and functional..." (end quote). Sensing frustration? It gets worse. The tension between the two companies escalated to the suggestion of completely pulling and reinstalling every single anchor in the HOV ramp area with no clear agreement of who should pay until the overarching CAT Project agreed in February, 2000 to pay to retest all anchors to a higher proof test and split the cost of any that required replacement. To that end: 187 anchors were tested. 19 of them failed! Deficiency Report 001 was considered to be closed and it was closed on the 26th of January, 2001. On the 17th of December, 2001 a Modern Continental QA inspector filed a non-compliance report to B/PB after finding multiple anchor displacements in the I90 connector tunnel. The action directed by bpb was the same as it had been undertaken in the HOV tunnel, 2 years previously. These were executed and the non-compliance was closed out. During the investigation two other incidents of anchor slippage were found. In several B/PB field engineering reports in 2001 and once again in 2002, however neither were formally flagged as non-compliances or defects and no deficiency reports were created. Despite the discovery of these damaged anchor bolts, over a four year period, project officials did not begin any inspection program. So what are the key findings? The NTSB investigation delivered its report on the 10th of July, 2007 on the anniversary of the incident and there were 20 findings in their report. I'm going to focus on 4 that I think are key. Finding #5: Gannet Fleming and Bechtel Parsons Bricknerhoff failed to account for the fact that polymer adhesives are susceptible to deformation (creep) under sustained load with the result that they made no provision for ensuring the long-term safe performance of the ceiling support anchoring system. Finding #10: Gannet Fleming approved the D Street portal anchors without identifying which epoxy formulation was being used. Finding #13: After unexplained anchor displacement was found in the interstate 90 connector tunnel in 1999 and 2001, Bechtel Parsons Bricknerhoff and Modern Continental Construction Company Incorporated should have instituted a program to monitor anchor performance to ensure that the actions taken in response to the displacement were effective. Finding #15: Had the Massachusetts Turnpike Authority at regular intervals between November 2003 and July 2006 inspected the area above the suspended ceilings in the D Street portal tunnels the anchor creep that led to this accident would likely have been detected and action could have been taken that would have prevented this accident. The report also noted that there was no evidence found that Modern Continental had any information at that time to suggest that the epoxy it was using was susceptible to creep. The National Transportation and Safety Board (or NTSB) determined that and I quote: "...the probable cause of the July 10, 2006 ceiling collapse in the D Street portal of the I90 connector tunnel in Boston Massachusetts was the use of an epoxy anchor adhesive with poor creep resistance. That is an epoxy formulation that was not capable of sustaining long-term loads. Over time the epoxy deformed and fractured until several ceiling support anchors pulled free and allowed a portion of the ceiling to collapse." (end quote). Let's talk about the fallout. In late 2008 the family of Melina Del Valle reached a $28.09M USD settlement with 15 parties named in their wrongful death lawsuit. Bechtel Parsons Bricknerhoff and Modern Continental paid the largest share of the settlement with Powers Fasteners and Gannett Fleming Incorporated the next largest following that. The supplier of the epoxy compound, Powers Fasteners, was charged with one count of involuntary manslaughter. Action against Powers Fasteners Incorporated was the first criminal indictment following the July, 2006 incident. The suspended ceiling structure in the D Street portal was removed. During subsequent tunnel evaluations, it was determined that because of the short length of that tunnel section and its proximity to a tunnel opening, the suspended ceiling was not necessary for adequate tunnel ventilation. The ceiling structure was never replaced. So what do we learn from all of this? Certainly don't start installing things until the design is approved. That's always a good one, but it's clear that neither of the reviewing companies looked beyond the static load strengths which showed that people reviewing within those companies didn't fully understand the nature of epoxies. Firstly though inspections. People don't seem to see that inspections are a form of protection; a layer of protection if you will. Beyond the basic quality assurance aspects during a project (which were lacking), the operational company or organization should be performing appropriate inspections regularly. Whether that's for concrete cracking, corrosion of metals, warping, bulging or in this case epoxy creeping, regular maintenance inspections are a catch-all for many types of premature failures. No tunnel inspections were performed to determine the physical and functional condition of the ceiling system from the time the I90 Eastbound connected tunnel was open to traffic on the 18th of January, 2003 until the day of the incident. Had the MTA been inspecting post handover they would have seen the epoxy creep. No question. No doubt. Fundamentally though the evolution of the suspended ceiling design during the course of the project led to a poor choice and although it is possible to find adhesives that will hold that amount of load over a long period of time the design choice was probably still not the best. Had the original portal ceiling slab design incorporated anchor points, embedded in the concrete, it wouldn't have been an issue, but the way the design was split across different companies over a long period of time allowed this kind of oversight or gap to creep in which then necessitated a sub-optimal solution. The NTSB report also states and I quote: "In civil projects adhesive anchors are typically used in short-term or sheer load applications. Under these conditions even if the adhesive is susceptible to creep the displacement will likely never reveal itself and those responsible for specifying approving installing and testing the anchors will not be aware of it." If you're specifying something to be used, you really need to understand it since clearly most of the people involved in this project didn't really understand epoxy creep they shouldn't have been specifying it; and when you're a registered professional engineer and approving a document it's doing half the job to say "it meets the load requirements" but to not consider the application it was used for? That's important too. Ultimately there was enough blame to share around in this incident. There were multiple engineers that could have challenged the results of the HOV ramp test anchors. The engineers that cited problems in 2001 and 2002 should have raised those as formal defects to the project to be investigated and rectified. The MTA should have been conducting regular inspections too. Ultimately though the design, review, construct and assurance process on this project failed. The design was not specific enough on a crucial point. The review of the design was not detailed enough on that crucial point. When the warning signs came to light, the construction and assurance process incorrectly blamed the installation, when the true cause was not properly determined. Were the wrong people involved in these stages of the project? Most likely? Were the right people ignored due to cost and time pressure? Potentially. Could a simple visual inspection have found the problem and prevented this incident from occurring? Most definitely. Those engineering companies were held to some account but what about the MTA? My observation is that operations companies like the MTA can look at inspections as an operational expense and when nothing obvious is going wrong over a long time period they tend to become the lowest priority task and get pushed off repeatedly. Another suggestion I've heard is that because it's brand new, why would you need to inspect it? But inspections are critical! They are the layer of protection against the unforeseen and they aren't about what you expect to see...they're about finding what you don't expect, and acting on it. Had the MTA developed and performed those inspections as part of and following the project handover this could have been avoided. I'm not excusing the design errors at all but I am saying that a relatively inexpensive walk down would have detected this well before anything ever happened. So the next time you're thinking you can put off that overdue inspection a few more days...make the time... and just do it! If you're enjoying Causality and want to support the show you can by becoming a Patron! You can find details at engineered.network/ causality about how you can help make this show continue to be made. You can find details at engineered.network/ causality about how you can help this show to continue to be made. A big thank you to all of our Patrons and a special thank you to our Silver Producers Mitch Bielger, John Whitlow, Kevin Koch, Oliver Steele, Hafthor and Shane O'Neill with an extra special thank you to our Gold Producer known only as 'r'. Causality is heavily researched and links to all materials used for the creation of this episode are contained in the show notes. You can find them in the text of the episode description of your podcast player or on our website. You can follow me on the fediverse at firstname.lastname@example.org on Twitter @johnchidgey (all one word), or the network @engineered_net. This was Causality. I'm John Chidgey. Once again, thanks so much for listening...