Vault on CI/CD pipelines to secure secrets

Why is it important to protect your secret from being accessed by any branch

CICDs are heavily used today. They must access various kinds of secrets, such as private keys, API keys, or passwords, which are typically stored in secure locations.

However, any malicious code executed in the same pipeline context can gain access to these secrets, potentially leading to lateral movement and further security breaches.

Attackers have several ways to inject malicious code into your pipeline, such as phishing attacks to acquire developer credentials, infecting dependencies with malicious code, or stealing a developer's computer. However, the damage an attacker can do is limited by the Pipeline Based Access Control (PBAC) implemented in your CICD.

Fortunately, Gitlab natively supports protecting secrets on some branches to enhance security measures.

Protected variables and branches in Gitlab

In Gitlab, variables are often used to manage secrets that need to be accessed by the CICD. As an example, consider two variables: AWS_SECRET_ACCESS_KEY and VERSION. The first one contains sensitive information and should only be accessed by the main branch. The second one is a simple parameter which can be accessed by anyone. By properly configuring the PBAC, you can ensure that only authorized jobs are allowed to access sensitive information.

If we don’t protect our variables, anyone with the ability to create a branch can steal our access key to AWS. An attacker can inject this piece of code into the CICD to show all secrets:

stages:
  - debug

print-all-env-vars-job:
  stage: debug
  script:
    - echo "GitLab CI/CD | Print all environment variables"
    - env

The result of the pipeline execution :

An attacker can inject this piece of code into the CICD to show all secrets:

$ echo "GitLab CI/CD | Print all environment variables"
GitLab CI/CD | Print all environment variables
$ env
...
AWS_SECRET_ACCESS_KEY=secret_key
VERSION=0.0.1
...

To prevent the secret from being read by any branch, you can add the protected option to your variable by editing it in the same place as before:

Once done, it will only be available to the pipelines triggered on protected branches. This kind of branch can’t be modified with a simple push. You must use a merge request and often have approval before modifying any code of this branch. This process ensures that no malicious code can be easily pushed.

This can be configured for each repository in Settings > Repository > Protected Branches. Let’s protect the variable AWS_SECRET_ACCESS_KEY and retry on 2 branches :

• not_main which has been created by a developer and thus is not protected

  $ echo "GitLab CI/CD | Print all environment variables"
  GitLab CI/CD | Print all environment variables
  $ echo "Current branch $CI_COMMIT_BRANCH"
  Current branch not_main
  $ env
  ...
  VERSION=0.0.1
  ...

• main which is the main branch and is protected

  $ echo "GitLab CI/CD | Print all environment variables"
  GitLab CI/CD | Print all environment variables
  $ echo "Current branch $CI_COMMIT_BRANCH"
  Current branch main
  $ env
  ...
  AWS_SECRET_ACCESS_KEY=secret_key
  VERSION=0.0.1
  ...

Restricting the ability to create, merge, or push to protected branches to a few individuals is essential; otherwise, all efforts to secure the CICD pipeline may be rendered useless.

This functionality may not be suitable for all users, especially with complex CICD as it has a binary behavior where a branch can either access a variable or not. Additionally, the requirement to store secrets in Gitlab may not be ideal for those who prefer a centralized approach to secret management.

Usage of Hashicorp Vault in CICD

When it comes to managing secrets in CICD, Hashicorp Vault is a state-of-the-art solution that offers robust security and scalability. It allows organizations to store and manage sensitive information while providing centralized secrets management, data encryption, and identity-based access control.

Retrieving secrets from Vault in CICD is a highly secure and efficient approach, made possible by using JWT authentication. Each Gitlab job has a JSON Web Token (JWT) that contains all the information required to determine whether the job can access a secret. The PBAC can be based on factors such as :

• Project name + branch name
• Project ID + tag
• Is the branch protected?

To do so, you must activate the JWT authentication in Vault :

$ vault auth enable jwt
Success! Enabled jwt auth method at: jwt/

And add the issuer of your Gitlab server to the list. Vault will check the integrity of the given JWT with this endpoint :

$ vault auth enable jwt
Success! Enabled jwt auth method at: jwt/

Don’t forget to replace [gitlab.example.com](<http://gitlab.example.com>) by your own Gitlab instance.

Vault is now able to handle JWT issued by your Gitlab server.

The next step is to configure the access policy to each secret/workspace inside Vault.

Let’s say you have a secret for your development environment and one for your production environment :

$ vault kv get -field=password secret/myproject/development/db
dev_pass

$ vault kv get -field=password secret/myproject/production/db
production_pass

You can create 2 distinct policies, one to read the development secret and one to read the production secret :

$ vault policy write read-development-read - <<eof #="" policy="" name:="" myproject-development-read="" read-only="" permission="" on="" 'secret="" myproject="" development="" *'="" path="" "secret="" *"="" {="" capabilities="[" "read"="" ]="" }="" eof="" success!="" uploaded="" policy:="" $="" vault="" write="" myproject-production-read="" -="" <<eof="" production="" myproject-production-read<="" code="">

We have now created 2 policies which are not yet used. Now let assign them to a role that link the JWT with these policies.

$ vault write auth/jwt/role/myproject-development-read - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-development-read"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims": {
    "project_id": ["43473361", "43473362"],
    "ref_type": "branch",
		"ref": "dev-*"
  }
}
EOF

$ vault write auth/jwt/role/myproject-production-read - <<EOF
{
  "role_type": "jwt",
  "policies": ["myproject-production-read"],
  "token_explicit_max_ttl": 60,
  "user_claim": "user_email",
  "bound_claims_type": "glob",
  "bound_claims": {
    "project_id": ["43473361", "43473362"],
    "ref_protected": "true",
    "ref_type": "branch",
    "ref": "main"
  }
}
EOF

Here, we declared 2 JWT roles, one with the policy myproject-development-read and the other with the policy myproject-production-read. To know if a runner can use a role or not, the list of claims (keys and value) of the JWT will be compared to the field bound_claims .

If they correspond at an item in each list, the runner can assume the role and access to the corresponding secret inside Vault.

Here we only want the branch main able to access the production secret. This branch must be protected and come from the project 43473361 (the one I am using for this demo) or 43473362. This number can be found in the HTML code of the repository page by searching the tag with id="project_id”. The development password can be read from every branch starting with dev-

Now, a pipeline like this triggered on a branch called dev-6237 will be able to retrieve the development secret after getting myproject-development-read role while failing trying to fetch the production one:

Here, we declared 2 JWT roles, one with the policy `myproject-development-read` and the other with the policy `myproject-production-read`. To know if a runner can use a role or not, the list of claims (keys and value) of the JWT will be compared to the field `bound_claims` If they correspond at an item in each list, the runner can assume the role and access to the corresponding secret inside Vault.

Output :

$ echo $CI_COMMIT_REF_NAME
dev-6237
$ echo $CI_COMMIT_REF_PROTECTED
false
$ export VAULT_ADDR=http://vault.example.com:8200
$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-production jwt=$CI_JOB_JWT)"
$ export PASSWORD_DEV="$(vault kv get -field=password secret/myproject/development/db)"
$ echo $PASSWORD_DEV
dev_pass
$ export PASSWORD_PROD="$(vault kv get -field=password secret/myproject/production/db)"
Error reading secret/data/myproject/production/db: Error making API request.
ULR: GET http://vault.example.com:8200/v1/secret/data/myprojectproduction/db
Code: 403. Errors:
* 1 error occurred:
      * permission denied
Job succeeded

This is the opposite on the main branch with a pipeline like this :

read_secrets:
  image: vault:latest
  script:
    # Check job's ref name
    - echo $CI_COMMIT_REF_NAME
    # and is this ref protected
    - echo $CI_COMMIT_REF_PROTECTED
    # Vault's address can be provided here or as CI/CD variable
    - export VAULT_ADDR=http://vault.example.com:8200
    # Authenticate and get token. Token expiry time and other properties can be configured
    # when configuring JWT Auth - https://developer.hashicorp.com/vault/api-docs/auth/jwt#parameters-1
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-development jwt=$CI_JOB_JWT)"
    # Now use the VAULT_TOKEN to read the secret and store it in an environment variable
		- export PASSWORD_PROD="$(vault kv get -field=password secret/myproject/production/db)"    
    # Use the secret
    - echo $PASSWORD_PROD
    # This will fail because the role myproject-development-read can not read secrets from secret/myproject/production/*
    - export PASSWORD_DEV="$(vault kv get -field=password secret/myproject/development/db)"

Output :

$ echo $CI_COMMIT_REF_NAME
main
$ echo $CI_COMMIT_REF_PROTECTED
true
$ export VAULT_ADDR=http://vault.example.com:8200
$ export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=myproject-development jwt=$CI_JOB_JWT)"
$ export PASSWORD_PROD="$(vault kv get -field=password secret/myproject/production/db)"
$ echo $PASSWORD_PROD
production_pass
$ export PASSWORD_DEV="$(vault kv get -field=password secret/myproject/development/db)"
Error reading secret/data/myproject/development/db: Error making API request.
ULR: GET http://vault.example.com:8200/v1/secret/data/myproject/development/db
Code: 403. Errors:
* 1 error occurred:
      * permission denied
Job succeeded

Your secrets are now well protected, so even if an attacker gains developer access to your Gitlab server, he won't be able to easily access your production secrets.

Conclusion

Safeguarding secrets is a vital aspect of securing the CICD pipeline. Limiting the number of individuals with access to secrets is essential to ensuring they remain secure. Developers should not have access to production secrets and should only be granted access to the secrets they need to perform their tasks.

PBAC is a valuable tool for managing access to secrets in a project, allowing for precise control over who can access sensitive information. By prioritizing the protection of secrets, you can help to prevent data breaches and ensure that your CICD pipeline is as secure as possible.