Maybe the conspiracy theorists are right, and pyramids really do control everything we do.

‍

Except of course, I’m talking about pyramids that represent the layers of a process or setup that rely on one another. A pyramid represents stability, a construction with no chance of falling over – but only if it’s assembled in the correct order!

‍

‍

When I look back on endeavours I’ve witnessed or been involved in, a common pattern emerges between success and failure: it’s all about getting the basics right. Nothing is more frustrating than talking to an organisation about implementing some fancy AI or blockchain project when they haven’t even managed to get basic automation of their infrastructure in place: you know it is only ever going to be innovation on the edge.

‍

In this essay, I will explore a series of pyramids that illustrate this need for solid foundations. Each pyramid represents layers of dependencies within a particular practice, guiding us on where to start, and how to not just jump to the top layers because they’re easier or more exciting.

‍

The Testing Pyramid

‍

The testing pyramid goes back to 2009’s “Succeeding With Agile” by Mike Cohn. I will adapt it here to look something like this:

‍

‍

Each layer on the Testing Pyramid is only efficient if the layers below are solid. Many organisations ignore automated testing (the three lowest layers), and rely on manual testing. 

‍

However, manual testing without automated testing first is much more likely to pick up the higher frequency technical bugs; soaking up their time, and letting the logic errors through.

‍

‍

Solid automated testing foundations allow the testers to focus on the right things, and increases efficiency all around - frequent releases become unfeasible if each one demands extensive manual testing.

‍

The Complexity Pyramid

‍

I think the first person to describe the Complexity Pyramid was my old boss at BAML, Kirat Singh. It sets out the layers of complexity in a system. Here’s how the pyramid layers stack up:

‍

‍

Data, as the foundational layer, is inherently complex and challenging to change. Stateless processing/compute contains business logic which may be more flexible, while the UI layer at the top is the least complex.

‍

It’s tempting to focus on the topmost, simplest layers, creating a new facade UI that makes old systems “feel” modern. But this facade is ultimately limited by the lower layers’ constraints. This leads to confusing conversations with business partners when limitations resurface: “what do you mean you don’t support Chinese characters, the system is brand new!”, when that limitation actually comes from the Data layer.

‍

By focusing on the base of the pyramid—the data layer, or the most complex aspect of a system—we can make transformative changes to system behaviour and outcomes. Attacking this base requires courage and resources, but often is the best path to true modernisation.

‍

The Tech Viability Pyramid

‍

Organisations and leaders should have high technology aspirations for their teams. But the viability of a tech project succeeding is often dependent on the basics being in place. I have been in frustrating situations where people have wanted to talk about very ‘sexy’ tech, when they don’t even have consistent infrastructure deployments.

‍

‍

A stable infrastructure forms the foundation for any effective product or tech team. A team that wants to move fast also needs an infrastructure layer that can keep up with them, without introducing instability. If new server requests or configuration changes take weeks, the application layer compensates by making poor, but necessary, adjustments.

‍

Once we have a stable foundation, we can deploy applications into the environments, but this must be automated, repeatable and observable. At this point, we control our applications, rather than them controlling us! Only then can we move to sexier tech.

‍

We can now introduce meaningful data into our data architecture. But the model must be clear, defined and catalogued, with simple checks for data quality and consistency. We must know where things are, and who owns them.

‍

Only at this point do we reach the pyramid’s peak and start thinking about the cutting edge use cases that the salesperson has been selling us. We unlock the potential of our data! We use AI to assist our agents and customers! But it all started with the foundations.

‍

Cyber-Security Testing Pyramid

‍

Cybersecurity has its own pyramid, providing a structured approach to a robust security stance. Unfortunately, many organisations settle for a single, perfunctory penetration test, which offers limited security.

‍

A pyramid can come to our rescue here as well. Here’s what a more comprehensive Cybersecurity Testing Pyramid might look like:

‍

‍

The foundation involves Compliance as Code. This approach automates the guardrails that enforce security policies and regulatory requirements. By integrating these checks with your devops pipelines, issues can be identified before they’re even allowed into the main code base.

‍

Few things are more frustrating than a penetration test exposing an issue that could have been easily caught by an automated tool. What a waste of brainpower and money. By adding automated vulnerability scanning through a tool like Intruder, we save ourselves that embarrassment.

‍

Once our automation layers are complete, we can shift to the humans: penetration testers. The best penetration test is not a box ticking exercise - as they say, Hackers Don’t Give A Shit about your excuses.

‍

For those who have truly scaled the pyramid, the peak offers something unique: the bug bounty. With a foundation so strong, you can now invite people to try and compromise your systems for a prize. Any vulnerability they may uncover was already there for a real hacker, but it takes a certain confidence to invite white hats into your house. The payoff is reassurance that those existing vulnerabilities won’t be found by black hats.

‍

The AI Adoption Pyramid

‍

In “GenAI PoCs: Proof of Complexity”, I explored how jumping straight into a GenAI Proof of Concept is likely to lead to a ‘cool’ project that never goes to production. This too can be explained by missing out on the foundations of our AI Adoption Pyramid:

‍

‍

Rather than jumping straight into projects, we begin with a solid foundation: aligning security, risk, and compliance teams, and establishing guiding policies for implementation.

‍

Next, we empower members of the organisation to make architectural decisions around tools & models that align with those policies, and then evangelise the benefits.

‍

Only then do we choose appropriate projects that will bring about those benefits (although being careful what projects we choose, as explained in the article).

‍

The Pyramid Within The Practice

‍

As we look back on all of these pyramids, they can all be represented by the same set of layers:

‍

‍

Perhaps this is universal truth, and our job as engineers and practitioners is divining the Pyramid Within the Practice?

‍

We build a solid foundation, with fundamental automation and governance. This keeps our systems stable, secure and scalable.

‍

We build flexible integration frameworks on our foundation, connecting up the various parts of our systems into a cohesive whole through automation and consistent processes. Maintaining integrity in the connecting tissue is essential; we cannot build reliable human experiences on top of discordance.

‍

We build the pinnacle of our pyramid - touchpoints with the real world. Human invention and cleverness is allowed to run wild, safe in the knowledge that our structure is sound.

‍

We build.

‍

Perhaps the next time that you have a project or process that seems to be much harder and slower than it ought to be, stop. And find the Pyramid within The Practice.

‍