The Cloud Is Not Optional

14 Jul 2023 - Thomas Depierre

When you hear that one of the vendors responsible for keeping government organizations safe had a security breach, you can easily decide that this is unacceptable. When you hear that it is hard to know who is affected and how much, you may start to feel a bit panicked. This is bad; it would be far better if it never happened. These are dangerous breaches. So you go find out who made the mistake of allowing a system designed-to-be-safe to be broken in. And you fire them or blast them for being idiots. And for making the world worse for all of us.

After this, you can breeze a bit. The bad thing got fixed. The responsible has been sacked or punished. We moved away from the affected vendors. We added new systems to ensure we know who does the wrong thing and who is affected next time. Everything got fixed. Nothing to see here; we can go back to sleep.

Or can we? Did we fix anything? Could we change these systems and punish these people? Is it even possible to know who is affected? Did anyone do something wrong? Was the previous system the best we could do, and did we make ourselves more vulnerable by overreacting?

The story of a security breach

A few days ago, by the time I wrote this, Microsoft had to deal with a breach of some of the encryption systems they use for Office 365, particularly for the US government. The details are unknown, but it seems the attackers used a combination of extracting a Root key and some bugs and misconfiguration to generate access tokens.

This has made a few people mad on the Internet, particularly around the question of Cloud Security and trusting Cloud providers. There have been a lot of calls in the press for organizations to not mindlessly move to the cloud, and to ask themselves if centralizing all these security in a few vendors is not more dangerous than keeping it on-premise, securely separated from others. After all, it would be a less juicy target. You would need to breach multiple networks and systems to get the same amount of information.

This is what I call “pink fluffy unicorn” solutions. Solutions and ideas that make total sense in theory. I see where these people come from; I understand what they are trying to do, and I see why they think it makes sense and could help. And it could! If it was possible.

Just like I would be delighted if I could have a pink fluffy unicorn jumping on rainbows, but that is not possible. I cannot have a pink fluffy unicorn for all kinds of reasons. And most of these calls for “choosing the right balance between centralized on cloud and separated” make a lot of sense. If they were possible.

Why are SaaS and Cloud use exploding at the infrastructure level?

Running an Internet-facing Digital Infrastructure service in 2023 is a prevalent task. Most organizations need some, from the local group of plumbers to the US Federal government. Even if you limit yourself to a certain level of safety and national level security, you still get thousands of organizations across the US and the world that have real pressing needs for this kind of service. And these organizations will transfer and handle, through these services, a lot of secrets.

Running these services well, with high uptime, well configured, and safely with good security -both active and passive- is a tough job that necessitates a certain set of skills and knowledge. It also needs an organization that can hire, reward, support, and manage the teams and individuals doing this work. The operators need knowledge and skills, and their management structure up to the top has to be designed to support them and understand their needs. Otherwise, they will be left to work with the wrong tools, budget, and constraints.

So what do we see when we look for individuals with this knowledge? That they are not a lot of them. Every analysis of the tech job market talk of massive deficits, with hundreds of thousands, if not millions, of unfulfilled jobs. With an enormous shortage of talents and education pipelines that cannot train what is needed. And this hold for every part of the field that would work on the Digital Infrastructure, being for software engineers, sysadmins, operator, Infosec specialists, CyberSecurity specialists, SRE, etc.

In 2023, if you want to run an Internet-Facing Digital Infrastructure service for yourself, and if everyone that needed one tried to do it themselves, nearly none of them could get the necessary people to do it. They do not exist. The supply is too small. Nice try, it made sense, but you are chasing a pink fluffy unicorn.

This is why we have all moved to the Cloud and SaaS vendors. Because it means the limited supply of people with the skills to operate these services can be shared between all the organizations that need them. These are outsourcing shops. It is not outsourcing to reduce cost. It is outsourcing in order to share rare skills, that we could not grow but that everybody needs.

The Cloud is the least bad option

This is what all these yelling at Microsoft are missing. Did Microsoft do a perfect job? No, of course not. Could they do better? Sure, they could. Maybe. In a different universe. Would anyone running their office server do better, get better forensics and attribution, or perhaps not have made the same misconfiguration?

No. Come on. We all know this. All of our experience with organizations that run their IT is atrocious. Hell, there are MMO guilds with better IT than the vast majority of the organizations under these attacks. Good forensics? No-one has them.

So yes. Let’s do a proper system analysis of what happened here. What made the people that operate these systems think they were doing the right thing? What assumptions have the designers made about the world that is not true anymore? Can we devise better taxonomies for our problems than “misconfiguration”? Do these reports and “cause analysis” really help us get safer?

These are all questions worth asking. And if you want answers to these, there is a whole community of people working on this in software in the shadows, I can put you in touch if you care about getting results. But blasting a Cloud Vendor for not doing “enough” and organizations for using it instead of running their own? Send me a living and breathing pink fluffy unicorn first, and maybe I will take you seriously. I mean it. Until then, please shut up, stay on the side, and let the people trying to keep us all safe alone. You are taking up space.