The 7 Habits of Highly Effective Management of Secrets
The first time I attended a workshop of “The 7 Habits of Highly Effective People” was back in 1997 when I was working at Hewlett Packard Corporation. I was quite impressed with the message of Stephen Covey. I moved to IBM Research and once again I got another opportunity to go through the same workshop, and I learned even more. So, I am going to use the key message of Stephen Covey to describe 7 key principles of how to manage IT (Information Technology) secrets effectively.
1. Be Proactive
Secrets are everywhere, which increases the attack surface. We should be proactive to identify secrets that are in transit over network and stored in GitHub, databases, local file systems, etc. We should collect these secrets and put them in a centrally managed secret management tool (such as Hashicorp Vault). A secret can be password, encryption/decryption keys, certificates, social security numbers, IP addresses, etc. We should encrypt them and only allow encrypted secrets to be in transit or stored. We should be proactive to educate people who are creating or consuming secrets on how to manage secrets using secret management tools.
2. Begin with the end in mind
The end goal of managing secrets should be ensure that they are not stolen and thereby compromising integrity and confidentiality of IT. To achieve this goal, it is important to begin with the requirements that satisfy security controls, including compliance requirements. First list a set of key requirements that should be satisfied. For instance, secrets should be encrypted both in storage and in transit, keep track of the audit trails of secret life cycle, secrets should be rotated periodically, etc. Achieving end goal should be broken into steps that can be managed and implemented in a step-wise fashion.
3. First things first
There are many things that needs to be designed and implemented to achieve the end goal. List all of the items that are needed and then prioritize them into four categories: (1) Urgent and important that we do as soon as possible, (2) Not urgent but important that we plan to achieve in next 2 or 3 quarters, (3) Urgent but not important that can be delegated to others, such as attending meetings and executive reviews, and (4) Not urgent and not important that are considered to total distractions. We create Jira tickets or GitHub issues and label the work items accordingly.
4. Think Win-Win
Driving towards centrally managing secrets is not easy and is not cheap. We have balance between security and cost. Higher security leads to higher cost. We cannot simply take a secret that is stored in an environment and move the secret to another environment. For instance, consider a data center that consists of two different zones: Zone A and Zone B. We cannot move secrets from these two zones to a central place that is then managed by the same secret management tool; this is neither secure nor is cost effective. When we identify a secret, we also have to identify “the sphere of control” of that secret when it is compromised. Based on this sphere of control we can then create “secret domain”. Going back to the data center example, it is important to keep the secrets within each zone with a sphere of control that is also with in the same zone. This is both secure and cost effective.
5. Seek first to understand, then to be understood
When we identify a secret that is being stored as a plain text it is important to first understand the implication of moving that secret to a secret management tool. It is almost certain, that if we ignore the impact of moving, such as downtime of a service that relies on the plaintext secret, we have not done a good job. It is important to explain to the service owner the implication of storing the secret in a plaintext, with empathy, provide a step-by-step path to move the secret to a better managed solution. This includes modifying the code, testing, etc. Basically, there is an interplay between “secret manager” and “service owner” that should be handled with trust, alignment and logic.
6. Synergize
Secret management by itself is useless; it has to be done in conjunctions with services that needs to be protected. There is a strong synergy between services that need to be protected and the secrets that protects them. The “blast radius” of a secret depends on the “sphere of control” of the services. Also, secret management tool itself should be tamper resistant, and so need to be protected with higher level of security. It is like when law enforcement officers who wear bullet-proof vest in tough situations.
7. Sharpen the Saw; Growth
Once we have (most of the) secrets fully managed, we need continuous improvement, which includes providing secret rotation, revocation, etc. We also need to ensure that every access to secrets are logged and capture an audit trail. To limit the both the space and time access, use one-time use secrets, keep the time-to-live for secrets short, etc.
We will expand each of the above highly effective habits for secret management in subsequent articles. We will use Hashicorp Vault as our secret management tool.
Acknowledgement: I want to thank the amazing secret management team at IBM Cloud: John, James, Chrissy, William, and many more to name them all.
