This post was co-authored by Teddy Ferdinand. Who is working as Cloud Security Architect 🐻

Introduction

In this series, we will talk about the emergence of the DevSecOps movement, and more especially, what the benefits are of introducing a DevSecOps approach to your existing CI/CD Pipelines.

Red Pipeline

CI/CD Pipeline

To give you some context, you will find in the diagram below a standard DevOps CI/CD Pipeline.

Standard Pipeline

DevSecWhat?

DevSecOps could be defined as a shift from a central internal security team to the inclusion of security practices in the existing DevOps teams: DevSecOps 🎉

Previously, the security of a product or application was evaluated at the end of the project lifecycle, with a go/no-go from the security team, sometimes with a security test like a pen-test, or a security code review.

Today, security has become job zero. Everyone is responsible. It can’t be the problem of the CISO or Security teams alone. Everyone has a role to play in this area. With the agility brought by DevOps and the rapid cycles, the reduction of Time-to-Market, we need to move faster, but safer.

When DevSecOps teams build a product, they need to add security controls at every single layer.

Like the change introduced by DevOps a few years ago, the DevSecOps movement is a change of mindset. They need to be accountable for what they are shipping. :eyes:

CI/CD Security Journey

Source Code Repository :computer:

Signed Git Commits (Authentication)

One of the special things about Git is that the user has to tell us who they are. Adding signed commits ensures they aren’t impersonating someone else. This is because each user has their own key, which they use to sign the commit. This key ensures the commit comes from the user 100%, even if the account is compromised.

A compromised account must in this case have access to the login, the password (and the MFA which is enabled on the account, of course), and the encryption key used to sign the commit.

How to use this functionality?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Enable signing for all your commits
git config --global commit.gpgsign true

# Enable signing for current repository
git config commit.gpgsign true

# Setup you key
git config --global user.signingKey 8E03524833CB3037FDE385C44C0BFCCFED5CFE15

# Sign a single commit
git commit -S -m your commit message

Then, you just have to push as usual to create a signed commit.

Signed commit

Note: Administrators of your Git repository can enforce signed commits.

Prevent pushing secrets using git pre-commit hook

As security guys, one of our fears is finding valid credentials on a repository (public or not). Note that it is as important to deal with this issue for private repositories as it is for public repositories, as most attacks come from an internal compromise.

It is possible to use pre-commit mechanisms to avoid this type of compromise. However, keep in mind that a pre-commit takes place on the client side and can be bypassed.

It is an additional safety layer, but it would be dangerous to consider it 100% reliable. Moreover, a secret scan will never detect 100% of your secrets, especially if you have company secrets with a specific format. Having a pre-commit hook doesn’t replace a good security culture in your DevOps teams.

One of the most efficient solutions you can use to achieve this is Git-secrets, maintained by AWSLabs.

This script can be installed and run at every commit to avoid pushing secrets to Git.

If you mistakenly push a secret to your git repository, consider it compromised and rotate it immediately.

Code Repository Access Control ✅

GitLab

Of course, signing your commits is a good practice, but managing your upstream users correctly remains essential!

What are we talking about? Without going into the details of each version manager, a few basic rules:

  • Prefer the use of SSO, it will be easier for users and for your user lifecycle.
  • Have a strong password policy.
  • Enforce the use of Multi-Factor Authentication (MFA).
  • Group-Based Authorization: use user groups to make sure authorizations are applied to all users and to avoid as much as possible differences in rights.
  • Have automated arrival/departure processes that take account deletion into account.
  • Regularly audit registered users.

Build + Integration Tests + Security Tests 👷‍♀

Code Scan

When you develop your applications, you most certainly use external libraries. It’s normal and healthy: your goal is not to reinvent the wheel.

However, are these libraries reliable?

We can say “I use libraries from major players, so don’t worry”. This is a serious mistake. There are tons of examples of companies that have provided libraries that were subsequently compromised, including GAFAM. It’s normal: the principle of security is that defense mechanisms are constantly evolving to keep one step ahead of attackers.

This is why it is important to have the hygiene of updating third-party libraries when developing an application. Keeping track of all CVEs when you sometimes have dozens or hundreds of applications is a task that is impossible to do by hand.

You will tell me, I can’t follow all CVEs related to my technical stack. You’re right, there are some tools to help you, like SecAlerts. You just have to choose the software you use to get notified about new related CVEs. Pretty useful.

There are some simple rules to limit the technical debt when a flaw is discovered:

  • Limit external dependencies as much as possible, always assuming that no library, internal or external, should be considered 100% reliable.
  • Pin the versions of the libraries used, and do not use “latest”, which can cause unexpected malfunctions.
  • Produce a version manifest, which makes it easy to scan and list “out of date” libraries that need to be updated. This also allows you to centralize them in a CMDB and quickly visualize the impact of a critical CVE, for example.
  • When you add a third-party library, check its update/release cycle to see if the developer or development team is following the project closely. If a flaw is discovered, this can be a key point.

For your information, GitHub and GitLab automatically scan repositories to surface known vulnerable code.

It is possible to add triggers on these scans to process them automatically or open tickets for the development teams.

Having up-to-date code is a must-have!

To help you at this stage, you can use Code Inspector, developed by a talented friend of mine who is currently working at Twitter.

Secrets Management

Secrets management is always a sensitive issue for applications.

How to manage (for example):

  • The login information to my database
  • Credentials required for deployment
  • API keys to exchange between applications
  • Slack/Teams Hooks

Let’s make it simple, we put everything in environment variables! Uh… wait a minute… No, I told you that’s exactly what we don’t want to do! Here’s an unfortunate anti-pattern that we see all too often. An environment variable is not secure storage.

So how do we handle it? We’ll use a vault. Its role will be to securely store all our login and key information, drastically limiting the applications that can access these. The first advantage is that the login information becomes “volatile” and is no longer stored next to the application.

Several solutions that can be used to achieve this:

The idea is always the same: secure access to credentials.

But simply storing the credentials securely is not enough. Have you never had credentials that were given out “temporarily” for an incident, or that were retrieved by a user? OK, you’ve never seen it then, because in reality, this is what is often done! That’s why this password storage policy must be coupled with a strict rotation policy. As soon as the information is dynamically retrieved from a vault, it is very easy to rotate it regularly. It’s no longer a password or a credential, it’s more like a token.

Security tests 🔒

It is also possible and recommended to regularly scan your code and artifacts using SAST (Static Analysis Security Testing) tools.

These tools will look for any known flaws in your code and indicate the associated criticality.

In this field, the leader remains GitLab CI with the many languages already supported. Nevertheless, other solutions are beginning to emerge.

In open-source, we can think of Reshift, which does a very good job in this area.

Whatever the solution, keep in mind that a SAST solution will only indicate possible flaws, but will not necessarily fix them automatically.

Non-regression tests are still required for an upgrade, even a minor one. A minor modification can sometimes bring its share of anomalies.

It can also sometimes lead to degraded performance, requiring you to review your infrastructure.

In the next part, we will focus on the deployment part of your DevSecOps CI/CD Pipeline.

That’s all, folks! 👋🏼

zoph.