Getting Stuff Done in OT Settings | A Lesson for Cyber
CISOs and Senior Network Security Architects (people with the titles that should allow them to drive change) from household name firms in consumer goods, food and beverage, building management, upstream, midstream, and downstream ONG, dry bulk shipping, fishing, hydro, mining, water, and wastewater have been reporting decision loops of over 18 months for cyber resilience initiatives.
This cycle time has been wreaking havoc. To give a few stories from SANS attendees:
- A manufacturer was ransomed four times through the same exploit in the year it took the cybersecurity team to push a secure remote access initiative through procurement;
- The network security architect for a city transferred out from the electrical distribution group after trying unsuccessfully for 15 months to get his colleagues to attend a meeting on aligning with baseline cyber resilience standards;
- The cyber group at a mining conglomerate quit after three years of having every one of their security initiative blocked by upper management.
These are not fun stories, and they led us to conclude that the solution the ICS cyber community needed to hear about most was an effective, repeatable approach to getting things done. We therefore dedicated our 30 minute session to covering one approach we found works.
What To Do Tomorrow
- Go into the office and get involved in the selection and retention of Operational Technology products.
- Invite the participation of Operations in the selection of cybersecurity products.
- Cut out formalities in your written and verbal exchanges.
- Start using stories to communicate.
What Happens and Why
Cybersecurity and Operations are not separate undertakings. They directly impact one another. The reason why they have fallen into separate specialties is that both are complicated. That does not mean you can't develop a healthy understanding of what the other team is doing over a few hours of conversation. Achieve that, and work to ensure the other side achieves it as well, and you will find yourselves working towards the same purpose.
You start to think about the things Operations cares about, and they start to think about the things you care about. The actionables we recommend work both because they are simple and they push that collaboration along.
We worked the problem of a fictional company with numerous OT networks of various ages and configurations. To begin with, we worked things from the perspective of a cybersecurity officer joining in the selection of a SCADA system.
To assist in this effort, we brought in a brave friend: Alan Hudson from VTScada by Trihedral. For the avoidance of doubt, there was/is no commercial relationship between Trihedral (or, for that matter, their parent company, Delta Electronics) and Dispel. We just like how they built their system.
For those wondering why Alan needed to be brave to show up at a SANS event, and why we are most grateful Barry let Alan participate, some context is needed:
SCADA providers are typically perceived by OT cyber professionals as the enemy: some system pushed by salespeople who don’t know NIST 800-160 v2, 800-171, 800-172, 800-53, or IEC 62443-3 from a telephone number but are quite adept at boxing out the cyber team and selling to Operations based upon usability features.
When something gets sold to Operations and the OT Cyber team has to veto it, you create a relationship problem that pits internal groups against one another. This is not an example of the often referenced “IT<>OT Divide” but, rather, a case of an OT<>Operator Divide, and that is far more personal for many SANS attendees.
Most SCADA providers, then, would be insane to wander into a room with over 130 cyber professionals. What makes VTScada different is: (1) they architected their software such that it can be set up to meet the standards without making life miserable for either Ops or Cyber; and (2) they go out of their way to teach their customers about cyber resilience.
Back to the Story
75% of the purchase processes Alan had worked through in the preceding month had not involved a single cyber participant. Together, we presented considerations a cyber person might bring to the table using diagrams and terminology people working in Operations would readily understand.
For starters, begin your meetings with Ops or Management by providing a walkthrough of the Purdue Model. It takes only 2 to 3 minutes, and it will ensure everyone is oriented. For a pre-formatted Purdue Model to use in such a meeting, go here:
Points to convey (again, working the problem of a SCADA system):
Modularity: Is it one machine sitting on Level 3, or is it an agglomeration of subsystems dispersed over Levels 2, 3, and 3.5? In either case, how can you manage this?
Reach: Is the solution getting to all of the equipment it needs to reach? Just asking that question will help you develop a solid understanding of the assets that need to be networked together.
Redundancy: Can you create three or more geographically distinct mirrors of the system, or did they code it so you can only run one mirror?
Code Base Responsibility: There is an important sustainment distinction between a product that makes Operators code their own interfaces versus one that gives Operators the ability to design their own interfaces from pre-built and, therefore, pre-vetted components.
Contract: Ask to see the SCADA provider's contract before bothering to get a demonstration of the system. If the contract expressly says it is not for use in business critical settings, your systems should not be depending on it.
The above points are not cyber-specific, we know, but they are characteristics anyone can appreciate and that will, if selected for, substantially reduce the noise in the search process.
We then flipped the conversation around to show how to architect a resilient network in and around the OT System, and what Operations should bring to the table.
Templating and Tempo: Request and response forms need to be standardized if one is to minimize admin overhead and, in turn, connection backlogs. Remember the last time you were on hold for 20 minutes? People working in Operations hate that too.
Interface Standardization: If you are in Operations, take the time to describe the loadouts your teams need for different tasks. Cyber needs to know about the need to task-specific loadouts so that they can look for solutions that provide customized workstations.
Inventory Management of Disposable Components: Yes, disposable components need to cycle after use, but you'd better have an inventory of fresh components on hand to prevent a gap in service access.
It was at this point that the audience took (commandeered) the talk in the direction of 800-171 secure enclave scenarios. We spoke about the need to downselect early in the process for SCADA systems that could work well within topologically segmented environments, handle private certificate authorities, and interface with project-specific clocks.
So Now What?
The first steps are going to need to come from you. The acts of inviting your colleagues to learn about your world, and your proactive efforts to learn more about theirs, are what will spark collaboration.