Discipline #1: Rules Before Tools
"If you can't describe what you are doing as a process, you don't know what you're doing."
W. Edwards Deming was the “father of the quality movement” who revolutionized modern business. With “If you can’t describe what you are doing as a process, you don’t know what you’re doing.” he promulgated thought and understanding of what you’re doing and why you’re doing it before you do it.
Consider the popularity of threat Defense within OT Security and the tool-first trend it has enabled.
To a point, the more threats and risks that owners and operators can see and block, the better protected they are. But threats, risks and Defense are not the arbiters of protection.
It’s an intentional trend which inevitably consumes the attention span of critical infrastructure asset owners and operators. This is what governance, culture, and operational discipline, not only to enable tooling but also a complete protection strategy, have to compete with.
Controlled Tool Defense
OT Security’s “Defense” ecosystem has become exceptional for the top tier and increasingly unachievable to everyone else. The K-shaped industry offering tool-secured programs or compromise has fueled a dogma culture:
Who are we using to secure us?
How secure are we?
How quickly do we detect bad guys?
How many assets do we see?
How many threats are we piping in to track?
Which gobbitygoop smacklepop bad guy is threatening me today?
According to Evgeny Morozov, “solutionism“ reframes complex social, political, and organizational problems as technical problems with clean technological fixes.
The fixes often ignore structural, political, and human dimensions and eliminate the highly valuable friction (ambivalence, opacity, deliberation) that come with them.
So rather than treat an improperly maintained network lacking change management rigor, invest in a network detection solution that will show you all the holes. Solutionism treats inefficiency, opacity, ambiguity, and friction as defects to be engineered away.
Charles Perrow claimed that in systems with interactive complexity and tight coupling, the conventional engineering response of adding more warnings and safeguards increases complexity and can actually create new categories of accidents (citing Chernobyl, where testing a new safety system helped produce the meltdown).
From a different angle entirely, research by Brynjolfsson, Hitt, and Yang (2002) found that financial markets valued each dollar of a firm’s computer capital as though it were accompanied by roughly nine dollars of complementary intangible assets, the organizational changes, process redesign, and skills that surround the technology.
The technology was the smaller share of what created value; the practice built around it was the larger.
Fragment in Depth
In broader cybersecurity terms, Defense means facing an adversary and holding the line. Defense, however, isn’t perfection - and never has been.
Today, Defense mostly operates as Defense in Depth, even in ICS/OT cybersecurity. Defense in depth is a cybersecurity and physical security strategy that uses multiple, overlapping layers of independent safeguards.
Defense in Depth also admits that no single defensive layer can defend on its own. And while any one layer may prove larger in defensive contributions than another, especially within a sector by sector and domain by domain (IT/OT/Cloud/etc.) construct, there’s a point within the depth where their defensive strength begins to fragment.
If you have no Defense at all, putting something in place may confer hefty benefits. If you’re layer upon layer of Defense, every tool begins to yield diminishing returns.
Defense in Depth should be approached through controls, so that if one layer fails, another catches the attack - and the structural pillars from which individual controls should be developed from are:
Administrative
Technical
Physical
Experience Over Expertise
I once deployed an OT detection platform to >20 manufacturing sites for a client who almost immediately disabled all alerting right after we finished.
The platform was fine and doing exactly what it was supposed to do. The organization wasn’t ready for any of what it was telling them.
The alerting policies were off-the-shelf, they had no classification of OT assets, process for asset to site attribution or even validation of the vendor’s alert, risk or vulnerability severities against their own.
Even if they had happened to sort all of that out, there were no escalation paths, change management queues or relationships in place to build any of it. So we turned it all off.
I tell that story because it explains why no critical infrastructure organization of any size should ever buy a tool first.
The thing about turning the lights on is that you can suddenly see everything and the room is always full. The day you can see it, it asks everything of you at once, and you bought the tool to generate a list of questions you have no process to answer. But you can actually see it coming from a mile away if you stop and prepare for it.
That’s experience.
The Boring Layer
Sure, it’s fun to rodeo and eat fancy vendor dinners. But do yourself, your team and your organization the favor of developing the boring, administrative layer first.
One example within that layer is defining a Control statement. A simple approach to this would be a technology agnostic statement of intent, capabilities and critical requirements you’ll hold any vendor technology to.
This discussion and collaboration exercise tethers everything that follows to a fixed, yet still editable, statement of what you need to procure rather than buying what’s being sold. Written after the tool arrives ends up describing the tool, written before describes what your environment actually requires and the tool either meets or doesn’t.
The client I worked with had it the other way around.
Project Your Project
You do not need a solution in hand to get started on an implementation plan.
You do not. Most of the characteristics of control-tools are already known or publicly accessible. Start there and infer vendor/tool specific things along the way.
Then, look internally at your logical, virtual, and physical requirements, global and local, and capture where sensors live, which type of SPANs get provisioned, what your site IDFs look like, kind of GBICs may be needed, what the lead times are and who does rack and stacking.
Do that now and swap the vendor or tool-side specific items along the way.
This gets skipped because it feels premature. But getting even an initial read on early enabled organizationally informed discussions with network and infrastructure teams instead of starting from a vendor’s logo.
RACI Juice Is Delicious
Draft a RACI (Responsible, Accountable, Consulted, Informed) for before, during, and after deployment. This is the part that ties directly back to my former client.
Who takes which step, why, when, for which sites and triggered by what. Do you know how many fine particles of organizational and team resourcing and responsibility nuances you will experience while discussing actual, performance-level tasks and responsibilities?
This juice, is very, very much worth the squeeze.
Start generic and dial it in over time, as the deployment and enablement continue. The point is to work it out while the time and political capital are flowing rather than later, when alerts are firing and people start picking up the phone.
Two Months vs. Two Weeks
Back to my former client, it took them two months to untangle the mess after getting wrapped around the axle, reverse-engineering what they had instead of determining and building what they needed.
The political, technical and financial capital they spent is akin to paving the road behind you, not to mention how this could have impacted leadership given NIS2 executive accountabilities now in effect.
Spend two weeks on the control, the plan, and the RACI, and the initiative moves faster and cleaner. You can communicate the justification and work through friction as you go, gaining alignment before the tool was ever a line item. None of this requires a large budget or a Big Four relationship, just prioritization and sequencing.
This is the administrative layer that should not be ignored or underprioritized because it doesn’t fit a tool narrative. I performed and led dozens of cybersecurity tooling engagements as a consultant, and this is what I wish I had been asked for every time.
Don’t let the experience I’ve gleaned from 100’s of hours and $ millions spent on me and my former teams’ time go to waste.