With operational technology relying more and more on desktop computer platforms for control and monitoring, just how much can be put in the cloud safely?
Transcript available
[Music] Everything can be improved, iterated and refined. And if you don't think that's true, maybe you haven't analysed it enough. Calculated choices carefully considered. Absolutely analytical. This episode is brought to you by ManyTricks, makers of helpful apps for the Mac. Visit ManyTricks.com/pragmatic for more information about their amazingly useful apps. We'll talk more about them during the show. Analytical is part of the Engineer Network. To support our shows, including this one, head over to our Patreon page, and for other great shows, visit engineer.network today. Clouds, or more specifically, looking down from the clouds. With a little bit of envy, perhaps, this particular episode is specifically about control systems engineering, something that I don't talk about a lot on the podcast, and though it is based on a white paper that I wrote a few months ago, it has to do with cloud computing and whether or not it actually applies. And if it does apply, how can it be applied in control systems engineering? It's not a straightforward question, although some people think that it is. I suppose it comes back to the concept of where these control systems are placed. And usually when you have ruggedness or harsh conditions, drive a requirement for physically hardened equipment. Over the years and decades of the past, the controllers that were created originally as integrated boxes became more generic. They have a high availability controller that controls the plant and the interface, the human interface, HMI, was evolving and evolved into SCADA software, supervisory control and data acquisition. The SCADA software would only live then on a generic personal computer because that was was far cheaper. It became the window into the system. It became the view, the way of interacting, the way of controlling it. Originally, going as far back as 10, 15 years ago, it was commonplace to have manual controls at the actual devices themselves to override in case there was a problem with the SCADA system. As time has passed, these affordances have been removed for cost-saving reasons and the SCADA computer becomes the sole window of control and monitoring for our systems and local controls are fading into the background. All the while that this is happening, there has been a push or a re-merging, if you will, of infrastructure from information technology or IT departments and operational technology or OT departments. And OT is specifically those PCs with SCADA software running on them or engineering applications running on them in order for us to actually maintain our operational equipment. Traditionally, those two departments have been separated and they've been separated because they have completely different drivers. IT's drivers are cost reduction, commonization of equipment, standardization of equipment wherever possible, and of course, external threats from cybersecurity. Whereas OT is all about production, availability and safety. Independence from IT means that you would have a completely segregated system, since in the end you need to have complete control of that to ensure that if it does restart it's under control conditions. Isolation from the IT network was considered a major advantage for the longest time simply because there was no access, there was no attack vector from externally and hence you were safe and air-gapped. Unfortunately, the downside was that because the equipment was always running, because the equipment was effectively so important that it was never updated, it was never patched. In cases in 2010, that air gap was shown to be completely worthless. As Stuxnet took down a nuclear enrichment facility in Iran, it was air gapped and it still got in. The IT department have also made progresses with cloud computing, where it is more economical to outsource your actual equipment. So you can pay Amazon Web Services, for example, just one, there are others, where and they will maintain all of the server infrastructure for you. And all you need to do is provide thin clients, terminals that speak to that and interact with that cloud infrastructure. shared web services where they're possible or dedicated machines hosted in the cloud mean that IT departments no longer have to have and maintain server rooms with humidity control, temperature control, spinning rust, even if it's solid state drives. They don't want to have tin as they call it on-prem or on-premises. You can then write off those expenses as an ongoing cost against your operational cost for the business and no longer have to worry about depreciation of expensive server infrastructure and switches. And so the push to the cloud seems like a good idea. Unfortunately, with OT, there are certain restrictions. In order for OT to put their stuff in the cloud, we need to look at the data connectivity between the actual operational technology SCADA equipment and the controllers in the field. If If that OT equipment, the gap is too far away, then the latency requirements will not be satisfied. Before I go any further, I'd like to talk about our sponsor of this episode, and that's ManyTricks, makers of helpful apps for the Mac. Whose apps do, you guessed it, ManyTricks. Their apps include Butler, Kimu, Leech, DesktopCurtain, TimeSync, Moom, NameAngler, Resolutionator, and Witch. There's so much to talk about for each app they make, so we're gonna talk about five of them. Which, you should think about which is a supercharger for your command plus tab app switcher. If you've got three or four documents open at once in any one app, then which is beautifully simple pop-up lets you quickly pick exactly which one you're looking for. Recently updated, you can now also switch between tabs as well as apps and app windows and with horizontal, vertical or menu bar switching panels with text search for switching, you can show the front most app in the menu bar icon and it now has touch bar support as well and much, much more. Name mangler, you got a whole bunch of files, need to rename them quickly, efficiently and in huge numbers, NameMangler is great for creating staged renaming sequences with powerful regex pattern matching, recently enhanced, showing you the result as you go and if you mess it up, you can just revert back to where you started and try again. Moom, I use it every day. It makes it so easy to move any of your windows to whatever screen positions you want, halves, corners, edges, fractions of the screen and then you can even save and recall your favourite window arrangements with a special auto-arrange feature when you connect or disconnect your external displays. It was recently updated to be even faster now with touch bar support and keyboard integration with Adobe's apps. It's the first app I load on a new Mac because it is just so awesome. Time Sync allows you to track your time spent in apps or activities on your Mac with a simple and easy way. You can pool your apps by common activities, create custom trackers for non-Mac activities and its simple and powerful reporting feature shows you exactly where your time went so you can plan better and stay focused. Resolutionator is so simple. A drop down menu from the menu bar and you can change the resolution of whatever display you have that's currently connected to your Mac. The best part though, you can even set your resolution to fit more pixels that are actually there. It's really handy when you're stuck on your laptop and you need a little bit more screen real estate. That's just five of their great apps and that's only half of them. All of these apps have free trials and you can download them from ManyTricks or oneword.com/pragmatic and you can easily try them out before you buy them. They're all available from their website through the Mac App Store. However, if you visit that URL, you can take advantage of a special discount off their very helpful apps exclusively for Engineered Network listeners. Simply use pragmatic18, that's pragmatic the word and 1-8 the numbers in the discount code box in the shopping cart to receive 25% off. This offer is only available to Engineered Network listeners for a limited time, so take advantage of it while you can. Thank you to ManyTricks once again for for sponsoring the Engineered Network. Beyond latency, it's also, and perhaps more so, about reliability. If a SCADA server is sitting in the cloud and the controller that it is the sole interface into that allows remote operability and control and access is actually out on site 400 kilometers away from the data center, if the data connectivity between the two fails and there are no local controls for people to take over in an emergency, it's completely impossible for you to operate or monitor your plant. Of course, there are ways around this. You can have different geographical paths following completely different, not logical paths, geographical paths, meaning that if a backhoe comes out in the middle of the field and digs up your fiber optic cable, you still have a data path in order to get the signal from the data from the cloud to the controllers. Of course, a lot of this can be mitigated by having local controls. But then the push to the cloud happened after the move away from local controllers. Unfortunately, this has left OT somewhat lagging behind. This means that OT can't move to the cloud as much as they would like to get rid of that issue of having local server infrastructure on site, local machines, having to use thick clients rather than thin clients. The sorts of cost reductions that IT can realize seem as though OT cannot realize those, at least with respect to the cloud. Getting back to latency, one of the other requirements is that in many plants, there are credits taken in hazard analysis. So a HAZOP, which is a hazard and operability study, is designed to determine whether or not you need to have sufficient controls, is designed to flesh out hazards in operability of equipment, and a particular analysis called a LOPA, a layer of protection analysis, looks at major accident events and determines whether or not you need to have a safety controller. Safety controllers are designed to be high availability, high reliability and to automatically respond in the case that pressures exceed certain levels, flows exceed certain levels and so on and so forth. Safety controllers are essentially the final line of defence. In order to mitigate those risks, however, because we don't want to rely on the safety if we can, is we use alarms to indicate to operators, and in the case that it's not a 24/7 monitored system, we also page out. So paging alarms can also go to operators. They will dial into the system, examine the system and take preventative action where it's appropriate. And during lopers, then you can actually take credit for alarms and alarm indications and alarm responses from operators. Of course, that is not highly weighted, but it is still considered to be a mitigating control. Ultimately, however, it also means a loss of the ability to remotely intervene. And that puts reliance on the safety controller. In some instances, it's not possible to have a safety controller for complexity reasons. Certain variables may not be easily monitorable. It may not be possible to safely sequence an isolation. It may require an operator's intuition or a complex procedural checklist in order to safely start up, shut down, isolate, or whatever may be necessary to mitigate the major accident event. Ultimately though, if you're using your SCADA for those alarm purposes, and that is your choice, then you must ensure that your operational technology equipment is high availability, high reliability, and that more or less eliminates the cloud. As people talk more about the convergence of IT and OT, a lot of context seems to be missing. From the IT perspective, the idea of a hyper-converged infrastructure where switch and server storage are located in the same cluster, with options, for example, by Cisco in the HyperFlex environment and by HP's SimpliVity environment. The ultimate intention is that you can specify separate logical homes for different networks, but on the same physical piece of equipment. It makes sense at large scale, but at a smaller scale, as OT typically is, smaller and distributed, although not always, but generally. In that case, does that even make sense? A failure of a series of devices in a cluster, for example, will still lead to downtime of OT equipment. Traditional methods of redundancy may be slightly more expensive, but in the end will provide a better overall outcome. Segregation is always a debate, perhaps for another time. But in any case, IT are able to migrate a lot of their server infrastructure to the cloud. Unfortunately, from an OT perspective, most of the time, this is not possible. If you're enjoying Analytical and want to support the show, you can, like some of our backers, Carsten Hansen and John Whitlow. They, and many others, are patrons of the show via Patreon, and you can find it at patreon.com/johnchidjee or one word. Patreon rewards include a named thank you on the website, a named thank you at the end of episodes, access to pages of raw show notes, as well as ad-free, high-quality releases of every episode. So if you'd like to contribute something, anything at all, there's lots of great rewards, and beyond that, it's all very much appreciated. If you're not in a position to support the show via Patreon, that's totally fine. You can still help out by rating it iTunes, favouriting the episode or the show in your podcast player app of choice where supported, or by sharing the episode or show on social media. It helps others find out about the show and that's all very much appreciated as well. I'd personally like to thank ManyTricks for sponsoring the Engineered Network. If you're looking for some Mac software that can do ManyTricks, remember to specifically visit this URL, manytricks.com/pragmatic for more information about their amazingly useful apps. Accept nothing. Question everything. always a good time to analyze something. I'm John Chidjie. Thanks for listening. (upbeat music) (upbeat music) (upbeat music) (dramatic music) (dramatic music) [Music]