You Don’t Know Where Your Secrets Are

Do you know where your secrets are? If not, I can tell you: you
are not alone.

Hundreds of CISOs, CSOs, and security leaders, whether from
small or large companies, don’t know either. No matter the
organization’s size, the certifications, tools, people, and
processes: secrets are not visible in 99% of cases.

It might sound ridiculous at first: keeping secrets is an
obvious first thought when thinking about security in the
development lifecycle. Whether in the cloud or on-premise, you know
that your secrets are safely stored behind hard gates that few
people can access. It is not just a matter of common sense since
it’s also an essential compliance requirement for security audits
and certifications.

Developers working in your organization are well-aware that
secrets should be handled with special care. They have put in place
specific tools and procedures to correctly create, communicate, and
rotate human or machine credentials.

Still, do you know where your secrets are?

Secrets sprawl everywhere in your systems, and faster than most
realize. Secrets are copied and pasted into configuration files,
scripts, source code, or private messages without much thought.
Think about it: a developer hard-codes an API key to test a program
quickly and accidentally commits and pushes their work on a remote
repository. Are you confident that the incident can be detected in
a timely manner?

Insufficient audit and remediation capabilities are some of the
reasons why secrets management is hard. They are also the least
addressed by security frameworks. Yet these grey areas—where unseen
vulnerabilities remain hidden for a long time—are blatant holes in
your defense layers.

Recognizing this gap, we developed a self-assessment tool to
evaluate the size of this unknown. To take stock of your
real security posture regarding secrets in your
organization, take five minutes to answer the eight questions^[1]
(it’s completely anonymous).

So, how much do you not know about your secrets?

Secrets Management Maturity Model

Sound secrets management is a crucial defensive tactic that
requires some thought to build a comprehensive security posture. We
built a framework (you can find the white paper here^[2]) to help security
leaders make sense of their actual posture and adopt more mature
enterprise secrets management practices in three phases:

1. Assessing secrets leakage risks
2. Establishing modern secrets management workflows
3. Creating a roadmap to improvement in fragile areas

The fundamental point addressed by this model is that secrets
management goes well beyond how the organization stores and
distributes secrets. It is a program that not needs to align
people, tools, and processes, but also to account for human error.
Errors are not evitable! Their consequences are. That’s
why detection and remediation tools and policies, along with
secrets storage and distribution, form the pillars of our maturity
model.

The secrets management maturity model considers four attack
surfaces of the DevOps lifecycle:

• Developer environments
• Source code repositories
• CI/CD pipelines & artifacts
• Runtime environments

We then built a maturity ramp-up over five levels, going from 0
(Uninitiated) to 4 (Expert). Going 0 to 1 is mostly about assessing
the risks posed by insecure software development practices, and
starting auditing digital assets for hardcoded credentials. At the
intermediate level (level 2), secrets scanning is more systematic,
and secrets are cautiously shared across the DevOps lifecycle.
Levels 3 (Advanced) and 4 (Expert) are focused on risk mitigation
with clearer policies, better controls, and increased shared
responsibility for remediating incidents.

Another core consideration for this framework is that making it
hard to use secrets in a DevOps context will inevitably lead to the
bypassing of the protective layers in place. As with everything
else in security, the answers lay between protection and
flexibility. This is why the use of a vault/secrets manager starts
at the intermediate level only. The idea is that the use of a
secrets manager should not be seen as a stand-alone solution but as
an additional layer of defense. To be effective, it requires other
processes, like continuous scanning of pull requests, to be mature
enough.

Here are some questions that this model should raise in order to
help you evaluate your maturity: how frequently are your production
secrets rotated? How easy is it to rotate secrets? How are secrets
distributed at the development, integration, and production phase?
What measures are put in place to prevent the unsafe dissemination
of credentials on local machines? Do CI/CD pipelines’ credentials
adhere to the least privileges principle? What are the procedures
in place for when (not if) secrets are leaked?

Reviewing your secrets management posture should be top of mind
in 2023. First, everyone working with source code has to handle
secrets, if not daily, at least once in a while. Secrets are no
longer the prerogative of security or DevOps engineers. They are
required by more and more people, from ML engineers, data
scientists, product, ops, and even more. Second, if you don’t find
where your secrets are, hackers will.

Hackers will find your secrets

The risks posed to organizations failing to adopt mature secrets
management practices cannot be overstated. Development
environments, source code repositories and CI/CD pipelines have
become favorite targets for hackers, for whom secrets are a gateway
to lateral movement and compromise.

Recent examples highlight the fragility of secrets management
even in the most technologically mature organizations.

In September 2022, an attacker got access to Uber’s internal
network, where he found hardcoded admin credentials on a network
drive. The secrets were used for logging in to Uber’s privileged
access management platform, where many more plaintext credentials
were stored throughout files and scripts. The attacker was then
able to take over admin accounts in AWS, GCP, Google Drive, Slack,
SentinelOne, HackerOne, and more.

In August of the same year, the password manager LastPass fell
victim of an attacker who gained access toits development
environment by stealing the credentials of a software developer and
impersonating that individual. Later in December, the firm
disclosed that someone used that information to steal source code
and customer data.

In fact, in 2022, source code leaks have proven to be a true
minefield for organizations: NVIDIA, Samsung, Microsoft, Dropbox,
Okta, and Slack, among others, have been victims of source code
leaks. In May, we were warning^[3]
about the important volume of credentials that could be harvested
by analyzing these codebases. Armed with these, attackers can gain
leverage and pivot into hundreds of dependent systems in what is
known as supply chain attacks^[4].

Finally, even more recently, in January 2023, the continuous
integration provider CircleCI was also breached, leading to the
compromise of hundreds of customer environment variables, tokens,
and keys. The company urged its customers to immediately change
their passwords, SSH keys, or any other secrets stored on or
managed by the platform. Still, victims need to find out where
these secrets are and how they are being used to press the
emergency button!

This was a strong case for having an emergency plan ready to
go.

The lesson from all these incidents is that attackers have
realized that compromising machine or human identities gives a
higher return on investment. They are all warning signs of the
urgency to deal with hardcoded
credentials^[5] and to dust off secrets
management in general.

Final word

We have a saying in cybersecurity: “encryption is easy, but key
management is hard.” This still holds true today, although it isn’t
just about encryption keys anymore. Our hyper-connected services
world relies on hundreds of types of keys, or secrets, to function
properly. These could be as many potential attack vectors if
mismanaged.

Knowing where your secrets are, not just in theory but in
practice, and how they are used along the software development
chain is crucial for security. To help you, we created a maturity
model specifically about secrets distribution, leak detection,
remediation process, and rotation habits.

The first step is always to get a clear audit of the
organization’s security posture regarding secrets: where and how
are they used? Where do they leak? How to prepare for the worst?
This alone could prove to be a lifesaver in an emergency situation.
Find out where you stand with the questionnaire^[6]
and learn where to go from there with the white paper^[7].

In the wake of recent attacks on development environments and
business tools, companies that want to defend themselves
effectively must ensure that the grey areas of their development
cycle are cleared as soon as possible.