It’s not just the integrity of data that’s at stake. It’s the financial and performance repercussions of a compromised resource that often gets ignored!
When submitting code, ESPECIALLY when it’s public with X number of people watching the Repository, we don’t want code to contain secrets, passwords, API Keys, or anything that might compromise the integrity of an environment. With the best will in the world, accidents can happen, and sometimes are captured in headlines.
Although it’s easy to recycle a password or key, making such a change can have an undesired effect to dependant resources. You may find that services may subsequently fail. Particularly in Azure, the use of Managed Identities can really help in this regard.
However, if it does happen and you can remove it from the code… what about the history? Yup! The history… and, depending on where the offending line(s) of code have been introduced, and let hope it’s not your default branch, some very difficult decisions may have to be made.
In other words, how do you cleanse the history. It all depends on the leak of the data, a storage key can be recycled which renders it redundant, though you’ll have a few connection strings to update. But, what if it were something more static, more persistent, like a subscription or tenant id? Would that worry you? Not sure I could settle to be honest.
This is particularly a sticking point with Azure Repos where the commits cannot be deleted. Certainly, the last thing you want to be doing, is recreating your Repo – if you find yourself doing this, you’ll also find you’ll need to reinitialise your Pipelines, especially those defined in Yaml as they are constituents of your Repository.
The point I make is that prevention is better than (attempting to) cure.
Preventing leaks
There are a number of tools which can be used for scanning code Repositories for leaks. For example, here is a list for comparison:
In this article, I’m going to demonstrate the implementation of GitLeaks in an Azure DevOps Pipeline as a means of scanning a Repository for secrets.
First…
Scanning Locally
It is hoped that everyone when committing code to a repository will be diligent enough to ensure that the code does not contain secrets or anything that presents a vulnerability.
It is therefore good practice to test the code locally. GitLeaks can be installed on both Windows and OSX…
OSX:brew install gitleaks
Windows:
Access a list of binaries appropriate to your architecture – https://github.com/zricethezav/gitleaks/releases/latest
To scan a local directory, run:
gitleaks --config=.gitleaks.toml --repo-path=$(Build.Repository.LocalPath)
When running on a build agent on a DevOps Pipeline, the same command can be run, however, it might be preferrable to hide the output of the offending line of code so that it is redacted from the output and summary of the Pipeline. For example:
gitleaks --config=.gitleaks.toml --repo-path=$(Build.Repository.LocalPath) --verbose --redact
For a list of command line options, see https://github.com/zricethezav/gitleaks/wiki/Options#options
How it works
When gitleaks is running, it is examing the codebase and comparing with expressions defined in the .gitleaks.toml file. This file contains Regular Expressions which define what constitutes a leak (a secret).
Here’s an example of a defined expression:
[[Rules]]
description = "Generic API Key"
regex = '''(?i)['"]?[\w-_]*(apikey|api_key|secret)[\w-_]*['"]?\s*[=:]\s*('(?:[^'\\]|\\.){6,100}'|"(?:[^"\\]|\\.){6,100}")'''
tags = ["key", "API", "Generic"]You can also add RegEx to define expressions which should be excluded, or Whitelisted:
[[Rules.Whitelist]]
regex = '''AKIAIO5FODNN7EXAMPLE'''
description = "Ignore example AWS key"Running a scan locally
I’ve changed my working directory to the path where my Repository has been cloned, and as you can see, I’ve placed my .gitleaks.toml file at the root of the repository.
⇒ ls -la
total 56
drwxr-xr-x 10 markpatton staff 320 22 Sep 22:38 .
drwxr-xr-x 32 markpatton staff 1024 22 Sep 21:57 ..
drwxr-xr-x 15 markpatton staff 480 25 Sep 21:47 .git
-rw-r--r-- 1 markpatton staff 2518 22 Sep 21:57 .gitattributes
-rw-r--r-- 1 markpatton staff 5745 22 Sep 21:57 .gitignore
-rw-r--r-- 1 markpatton staff 6032 22 Sep 23:36 .gitleaks.toml
drwxr-xr-x 12 markpatton staff 384 22 Sep 21:57 MarkPattonCloud.Web
-rw-r--r-- 1 markpatton staff 1137 22 Sep 21:57 MarkPattonCloud.sln
drwxr-xr-x 6 markpatton staff 192 22 Sep 22:38 Pipelines
-rw-r--r-- 1 markpatton staff 253 22 Sep 22:41 gitleaks.ymlIt doesn’t matter where this is located, as long as you can reference it from the CLI, but I highly recommend you keep this under source control. Particularly where secrets may be whitelisted, you will want to keep a record of this. You’ll want to see when a secret has been whitelisted, and why. Hopefully it will be legitimate!
Then it’s a matter of running the gitleaks command:
gitleaks --config=.gitleaks.toml --repo-path=. --verbose --pretty --redactYour output will hopefully look something like this:
Integrating Gitleaks with Azure DevOps
If you have the opportunity to use a Pre-Commit hook, i’d highly recommend you employ this as a mechanism of checking the code before it is committed to the Repository.
Regarding Azure DevOps though, it is recommended that code is regularly checked for secrets which could have been leaked. In my opinion this is best served, as a minimum, on each commit to the repo. This means that if there is a potential for a leak, notifications can be instantly made to the person committing the code and the appropriate team responsible for Security.
Here is some Yaml if you’re interested in including this in your DevOps Organisation:
pool:
vm-image: macOS-latest
steps:
- bash: |
brew install gitleaks
gitleaks --repo-path=. --config=.gitleaks.toml --verbose --pretty --redact
displayName: 'Bash Script'Notice in the example above, the --redact switch has been used in the gitleaks command. Should a leak be found, it’s best that it’s not shown in the output of the build where someone could obtain this level of detail. Additionally, security roles should be defined such that only specific members of your team are allowed to run operations on your Pipelines and be able to view the output of your build.
Notifying the team: it’s at this stage, should a leak be found, that the individual making the commit, and specific members of your team, should be notified. You can use Service Hooks to initiate your Incident Management response (via tools like Jira or trigger some Webhook), or simply place a message on corporate messaging tools on Slack or MS Teams. The choice is yours, and should be appropriate to the context of your environment.
Forcing the change, preventing Code Merge
An additional measure which i’d recommend, is adding the very same Pipeline for gitleaks as a Build Validation measure on your target branch, especially your default branch.
Here’s how you can set it up:
In the Azure Repos screen, navigate to ‘Manage repositories’:
Choose your Repository, and select ‘Policies’:
Scroll to the bottom and select the target branch for your Pull Request(s):
Click the ‘+’ on the ‘Build Validation’ section:
Complete the Policy settings, and ‘Save’:
This is a pre-merge check which will allow the approval and completion, or prevent such, depending on the result of the gitleaks pipeline. When the PR is created, gitleaks is automatically run.
When a Pull Request is created, the Gitleaks Pipeline is automatically triggered.
When the pipeline completes, the result will be updated in the Pull Request:
Once the validation is Green (successfuly), the Pull Request can be approved and completed.
Other Reading
For other measures on security in your DevOps pipeline, read my article on DevSecOps in Azure Pipelines, using Snyk which can be used for assessing vulnerabilities on Code dependencies.
DevSecOps – Integrating Security in Azure DevOps
See Gitleaks being used in Azure DevOps in a recent demo I produced, which was published on YouTube. The video covers the following areas:
1 – scanning code for secrets(leaks)
2 – scanning code dependencies for vulnerabilities
3 – pen-testing your application
Mark Patton 