This post was co-authored by Teddy Ferdinand. Who is working as Cloud Security Architect 🐻
Introduction
In this series, we will talk about the emergence of the DevSecOps movement, and more especially, what are the benefits of introducing a DevSecOps approach on your existing CI/CD Pipelines.
CI/CD Pipeline
To give you some context, you will find in the diagram below a standard DevOps CI/CD Pipeline.
DevSecWhat?
DevSecOps could be defined as a shift from a central internal security team to the inclusion of security practices into the existing DevOps teams: DevSecOps 🎉
Previously, the security of a product or application was evaluated at the end of the project lifecycle, with a go/no-go from the security team, sometimes with a security test like pen-test, or security code review.
Today, Security became the job zero, everyone is responsible, it can’t be the problem of the CISO or Security teams. Everyone has a role to play in this area. With the agility brought by DevOps and the rapid cycles, the reduction of Time-to-Market, we need to do faster, but safer.
When DevSecOps teams build a product, they need to add security controls at every single layer.
Like the change introduced by DevOps a few years ago, the DevSecOps movement is a change of mindset. They need to be accountable for what they are shipping. :eyes:
CI/CD Security Journey
Source Code Repository :computer:
Signed Git Commits (Authentication)
One of the special things about Git is that the user has to tell us who he/she is. Adding signed commits ensures that he or she isn’t impersonating someone else. This is because each user has their own key, which they use to sign the commit. This key ensures the commit comes from 100% of the user, even if the account is compromised.
A compromised account must in this case have access to the login, the password (and the MFA which is enabled on the account of course), and the encryption key used to sign the commit.
How to use this functionality ?
# Enable signing for all your commits
git config --global commit.gpgsign true
# Enable signing for current repository
git config commit.gpgsign true
# Setup you key
git config --global user.signingKey 8E03524833CB3037FDE385C44C0BFCCFED5CFE15
# Sign a single commit
git commit -S -m your commit message
Then, you just have to push as usual to create a signed commit.
Note : Administrator of your Git repository could enforce signed commits
Prevent pushing secrets using git pre-commit hook
As security guys, one of our fears is to find on a repository (public or not) valid credentials. Note that it is as important to deal with this issue for private repositories as it is for public repositories, as most of the attacks come from an internal compromise.
It is possible to use pre-commit mechanisms to avoid this type of compromise. However, it should be kept in mind that a pre-commit takes place on the client-side, and can be bypassed.
It is an additional safety layer, but it would be dangerous to consider it 100% reliable. Moreover, a secret scan will never detect 100% of your secrets, especially if you got companies secret with a specific format. Have a pre-commit hook doesn’t avoid a good security culture in your DevOps teams.
One of the most efficient solution to achieve this you can use is Git-secrets maintained by AWSLabs.
This script can be installed and run at every commit to avoid pushing secrets to Git.
If you mistakenly push secret to your git repository, consider it as compromised, and rotate it immediately.
Code Repository Access Control ✅
Of course, signing your commits is a good practice, but managing your upstream users correctly remains essential!
What are we talking about? Without going into the details of each version manager, a few basic rules:
- Prefer the usage of SSO, it will be easier for users and for your user lifecycle.
- Have a strong password policy
- Enforce the use of Multi-Factor Authentication (MFA)
- Group-Based Authorization: Use of user groups to make sure that authorizations are applied to all users and to avoid as much as possible differences in rights.
- Have automated processes of arrival/departure of the company taking into account the deletion of accounts.
- Regularly audit registered users
Build + Integration Tests + Security Tests 👷♀
Code Scan
When you develop your applications, you most certainly use external libraries, it’s something normal and healthy, your goal is not to reinvent the wheel.
However, are these libraries reliable?
We can say “I use libraries from major players, so don’t worry”. This is a serious mistake. There are tons of examples of companies that have provided bookshops that have subsequently been compromised, including GAFAM, and it is normal, the principle of security is that defense mechanisms are constantly evolving to keep one step ahead of pirates.
This is why it is important to have the hygiene of updating third party libraries when developing an application. Keeping track of all CVEs when you sometimes have dozens or hundreds of applications is a task that is impossible to do by hand.
You will tell me, I can’t follow all related CVEs from my technical stack. You’re right, there are some tools to help you, like SecAlerts. You just have to choose used software to get notified on new related CVEs. Pretty useful.
There are some simple rules to limit the technical debt to the discovery of a flaw:
- Limit external dependencies as much as possible, always assuming that no library, internal or external, should be considered 100% reliable.
- Fix the version of the libraries used, and do not use “latest”, which can cause unexpected malfunctions.
- Produce a version manifest, which makes it easy to scan and list “out of date” libraries that need to be updated. This also allows to centralize them in a CMDB and quickly visualize the impact of a critical CVE for example
- When you add a third-party library, check its update/release cycle to see if the developer or development team is following the project closely, if a flaw is discovered, this can be a key point.
For information, GitHub and GitLab automatically scan repositories to bring up known vulnerable code.
It is possible to add triggers on these scans to process them automatically or open tickets to the development teams.
Having up-to-date code is a must-have!
To help you on this stage, you can use Code Inspector, developed by a talented friend of mine who is currently working at Twitter.
Secrets Management
Secrets management is always a sensitive issue for applications.
How to manage (for example):
- The login information to my database
- Credentials required for deployment
- API keys to exchange between applications
- Slack/Teams Hooks
Let’s make it simple, we put everything in environment variables! Uh… wait a minute… No, I told you that’s exactly what we don’t want to do! Here’s an unfortunate anti-pattern that we see all too often… An environment variable is not secure storage.
So how do we handle it? We’ll use a vault. Its role will be to securely store all our login and keys information, drastically limiting the applications that can access these. The first advantage is the login information becomes “volatile”, and is no longer stored next to the application.
Several solutions that can be used to achieve this:
- AWS SSM Parameter store (secure string) (with KMS encryption)
- AWS Secrets Manager (beware of costs if there is a lot of access!)
- HashiCorp Vault (not linked to a cloud provider)
- HSM: For on premise environment, you can also use material security to store your keys and sensitive information
- CloudHSM is also available on AWS.
The idea is always the same, secure access to credentials.
But simply storing the credentials securely is not enough. You have never had credentials that have been given “temporarily” for an incident, or that have been retrieved by a user…? OK, you’ve never seen it then, because in reality, this is what is often done! That’s why this password storage policy must be coupled with a strict rotation policy. As soon as the information is dynamically retrieved from a vault, it is very easy to rotate it regularly. So its no more a password or a credential, but it is more a token.
Security tests 🔒
It is also possible and recommended to regularly scan your code and artifacts using SAST (Static Analysis Security Testing) tools.
These tools will look for any known flaws in your code and will indicate the associated criticality.
In this field, the leader remains GitLab CI with the many languages already supported. Nevertheless, other solutions are beginning to emerge.
In open-source, we can think of Reshift, which does a very good job in this area.
Whatever the solution, keep in mind that a SAST solution will only indicate possible flaws but will not necessarily fix them automatically.
Non-regression tests are still required for an upgrade, even minor. A minor modification can sometimes bring its share of anomalies.
It can also sometimes lead to degraded performance, requiring to review its infrastructure.
That’s all folks ✋🏻, In the next part, we will focus on deployment part of your DevSecOps CI/CD Pipeline.
zoph.