If you are in professional software development you are probably using one of the two most popular source code repositories: GitLab or GitHub. Additionally, if you are into deployment automation then you are also probably relying on their built-in CICD pipeline offerings: GitLab CI or GitHub Actions.
If that is the case, then you know about the issue of managing GitLab/GitHub runners. The runners are executing CICD deployment instructions in YAML files (in GitLab often named .gitlab-ci.yml but it can be any other name). These runners need to physically “run” somewhere. You can use shared runners that are provided to you by vendors with some number of minutes included, depending on the subscription tier. However, sharing runners with other users is not a great idea as you can have “noisy neighbors” who consume more CPU so your jobs are executing slowly.
You would want to have your own private runners. The question raises where to run them. If you are into AWS (as I am), you would use scalable and managed solutions such as ECS or EKS to host your runners. There are ready to use CloudFormation/Terraform scripts to help you install runners as containers with built in scaling options (often using EC2 spot instance for cost reduction reasons).
At 1way2cloud we are working with both ECS and EKS at our customer projects but if we can choose we usually deploy Atlantis which manages runners for us.
However, there is another way that we tried recently. It is using CodeBuild service that is executing on lambda containers. This approach became possible only recently, when CodeBuild announced support for using GitLab as a source provider. Now that is very interesting as CodeBuild can execute on top of lambda functions (as announced last year). This approach would be the fully managed option with the lowest price tag – remember there are 1 million lambda invocations for free each month.
Documentation about this approach is quite poor. AWS describes a basic integration between CodeBuild and GitLab but if you want to run anything more than just shell commands, there is some more work to do.
Let’s set up CodeBuild with GitLab first, then we will try to do it with GitHub as well.
We’ll do it manually first and then show how we deploy it in an automated way, with Terraform.
GitLab
The first step is to create a connection from AWS to GitLab. On the AWS Web Console, you need to go to Developer Tools -> Settings and press Create Connection button. You will be able to select a provider, in our case GitLab.
After this, you will be redirected to GitLab UI to approve the connection request:
Once you authorize this connection, you can check in GitLab UI if the connection has been successfully accepted:
This is the only manual step you need to perform in the whole setup. After this, setup can be automated with Terraform for example. We’ll do that later. This step so far has been well documented in AWS.
Now we need to create a CodeBuild project and use this connection. And this is where things get blurry…
If you follow official AWS documentation here, you will only be able to run shell commands in the GitLab CI pipeline. AWS doesn’t go beyond “Hello World” example which is useless when it comes to running terraform commands. The main issue is that CodeBuild runs GitLab runner with the “shell” executor. You can see that when you open a GitLab CI pipeline:
To be able to execute terraform commands, you need to have a terraform docker image and to run GitLab pipeline in “docker” executor mode. If you are setting up GitLab runners on your own, you can specify which runner to be used in a config.toml file:
Unfortunately, CodeBuild doesn’t give you an option to change the executor from “shell” to “docker”. This could be a nice improvement of CodeBuild functionality.
You could of course say that with the shell executor you can install terraform by yourself when the pipeline starts. Yes, but that would mean that every time you want to build your code, you would need to first download packages such as wget, unzip, git and then download terraform package and unzip it before you can start using commands such as terraform init/plan/apply/etc. Your build times would be very long.
Instead, it is better to create your own docker image that has terraform on it and use it each time a build starts in CodeBuild.
Here is a Dockerfile that we will be using:
FROM public.ecr.aws/lambda/provided:al2023
###########################
# Install unzip, wget, git
###########################
RUN \
# Update
dnf update -y && \
# Install unzip
dnf install unzip -y && \
# need wget
dnf install wget -y && \
# need git
dnf install git -y
###########################
# Install Terraform v1.9.8
###########################
# Download terraform for linux
RUN wget https://releases.hashicorp.com/terraform/1.9.8/terraform_1.9.8_linux_amd64.zip
# Unzip
RUN unzip terraform_1.9.8_linux_amd64.zip
# Move to local bin
RUN mv terraform /usr/local/bin/
It is important to understand why we are using this base image:
public.ecr.aws/lambda/provided:al2023
We will run CodeBuild jobs on lambda functions. We can’t use just any lambda execution environment because a standard lambda environment doesn’t have terraform installed. We need to use a custom lambda environment, meaning that we need to provide a docker image to lambda. Lambda cannot execute on just any docker image. You need to get an official lambda base image from Amazon ECR Public Gallery. When you search for lambda images there, you will see that you can use images that come with preinstalled Go, Java, Python, Ruby, Go or you can use minimal lambda provided images. The image we are using in Amazon Linux 2023 runtime for Lambda that is OS-only image.
On top of this image we need to install a few additional tools (wget, unzip) so that we can download terraform and unpack it. Note that Amazon Linux 2023 uses dnf as the package manager instead of yum.
Now, let’s build and push this image to ECR. Make sure you create ECR repository first and on to export your AWS credentials so that you can login to ECR:
export AWS_ACCESS_KEY_ID="ASIA6MQB7D6XO5...."
export AWS_SECRET_ACCESS_KEY="a5hPJoTM0A3tTRSDnflW7ySpaZkpjz...."
aws ecr get-login-password --region eu-west-1 | docker login --username AWS --password-stdin 988920356782.dkr.ecr.eu-west-1.amazonaws.com
Now you can build and push the image. Note that the platform parameter is important when building an image for lambda:
docker build -t <image name>:<tag> . --platform linux/amd64
docker tag <image name> <ecr URI>:<tag>
docker push <ecr URI>:<tag>
If all is done properly, you should see your image in ECR, something like this (my image tag was gitlab, you can use any you wish):
Now we can go to CodeBuild and create a new project.
- Give your CodeBuild project a name:
- Define source:
We need to select GitLab as a source provider and use our previously created Connection to GitLab. Then we will be able to see all repos in GitLab and we should select the one we want to listen for code changes.
- Define execution environment:
Here we define lambda as a compute environment, and not with a managed image but with our own custom image that we previously pushed to ECR.
- Create webhook:
This is important part, where we define a webhook in GitLab. This webhook will be invoked each time there is a code change. It is important to select WORKFLOW_JOB_QUEUED as an event.
- Leave Buildspec empty, CodeBuild ignores it if the webhook is used.
That is all, once you save the CodeBuild project configuration you are ready to use it.
Just to be sure that everything is setup properly, go to GitLab (in your project -> Settings -> Webhooks) and see if the webhook is created there. Is should look like this:
We now need to add .gitlab-ci.yml file in our repo and see how it all works. Note that in the .gitlab-ci.yml file it is important to define a tag in the format that is exactly like this:
codebuild-<codebuild project name>-$CI_PROJECT_ID-$CI_PIPELINE_IID-$CI_JOB_NAME
In my case, .gitlab-ci.yml file looks like this:
workflow:
name: Terraform
stages:
- build
build-job:
stage: build
script:
- terraform init
tags:
- codebuild-gitlab-$CI_PROJECT_ID-$CI_PIPELINE_IID-$CI_JOB_NAME
When we push some code to our repo, we can see that the pipeline is triggered and terraform is initialized.
Now we can create as many stages as we want, for terraform plan, apply and other stuff that we need to do in our pipelines.
GitHub
What about GitHub?
Exactly the same steps as for GitLab. Obviously, the YAML file is a bit different. Equivalent of GitLab CI file would be GitHub actions YAML file:
name: Terraform
on:
push:
branches:
- main
jobs:
init:
runs-on:
- codebuild-github-${{ github.run_id }}-${{ github.run_attempt }}
steps:
- name: checkout code
uses: actions/checkout@v4
- run:
ls -a
terraform init
However, once we push our code to GitHub and the pipeline starts, we are seeing an error:
The action actions/checkout@v4 that is getting the code from the repo needs a GLIBC library with the version 2.27 that our lambda docker image doesn’t have. It seems we need to have NodeJS in our image.
I tried several base images with NodeJS (and versions 22,20,18,16,14) but none of them worked. I wasn’t able to solve it. If you have a solution, let me know! Maybe AWS should look into this as well and provide lambda image that is compatible with GitHub actions.
After all, GitHub runs on AWS and GitLab on GCP. It is in AWS interest to make this work. 🙂
Terraform
All previous work was manual. We can automate that with terraform scripts. Here is a script that creates CodeBuild project with lambda custom image:
# Trust relationship
data "aws_iam_policy_document" "assume_role" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["codebuild.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}
# CodeBuild IAM role
resource "aws_iam_role" "codebuild_role" {
name = "codebuild-role"
assume_role_policy = data.aws_iam_policy_document.assume_role.json
}
# CodeBuild IAM policy
data "aws_iam_policy_document" "codebuild_iam_policy" {
statement {
effect = "Allow"
actions = [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
]
resources = ["*"]
}
statement {
effect = "Allow"
actions = [
"codeconnections:*",
"codestar-connections:*"
]
# Make sure you change AWS region and account number here
resources = ["arn:aws:codeconnections:eu-west-1:988920356782:connection/*"]
}
statement {
effect = "Allow"
actions = [
"ecr:*"
]
# Make sure you change AWS region and account number here
resources = ["arn:aws:ecr:eu-west-1:988920356782:repository/*"]
}
}
# Attach IAM policy to the role
resource "aws_iam_role_policy" "codebuild_iam_role_policy" {
role = aws_iam_role.codebuild_role.name
policy = data.aws_iam_policy_document.codebuild_iam_policy.json
}
# Create CodeBuild project
resource "aws_codebuild_project" "codebuild_project" {
name = "gitlab"
description = "codebuild_project"
build_timeout = 5
service_role = aws_iam_role.codebuild_role.arn
artifacts {
type = "NO_ARTIFACTS"
}
environment {
compute_type = "BUILD_LAMBDA_2GB"
# Make sure you use your ECR URI here
image = "98892035.dkr.ecr.eu-west-1.amazonaws.com/prjct:gitlab"
type = "LINUX_LAMBDA_CONTAINER"
image_pull_credentials_type = "SERVICE_ROLE"
}
logs_config {
cloudwatch_logs {
group_name = "log-group"
stream_name = "log-stream"
}
}
source {
type = "GITLAB"
# Make sure you use your GitLab URL here
location = "https://gitlab.com/prjct/experiments"
}
# Project branch name
source_version = "main"
}
# Use Connections
resource "aws_codebuild_source_credential" "codebuild_credentials" {
auth_type = "CODECONNECTIONS"
server_type = "GITLAB"
# Make sure you use your connection ARN here
token = "arn:aws:codeconnections:eu-west-1:988920356782:connection/dcdd76cf367de7"
}
# Setup webhook
resource "aws_codebuild_webhook" "codebuild_webhook" {
project_name = aws_codebuild_project.codebuild_project.name
build_type = "BUILD"
filter_group {
filter {
type = "EVENT"
pattern = "WORKFLOW_JOB_QUEUED"
}
}
depends_on = [
aws_codebuild_project.codebuild_project,
aws_codebuild_source_credential.codebuild_credentials
]
}
Pricing
Having runners on CodeBuild with lambda seems to be the most cost efficient way compared to all others.
If we use lambda with 2GB of memory, CodeBuild would charge us $0.00002 per second of execution. Let’s say a typical build runs for 2 minutes, that would be $0.0024 per build. Let’s say we build our code 20 times per day, in 30 days this approach would cost us $1.44.
Conclusion
I am not a big fan of GitLab since they increased their prices to $29 user/month. GitHub is still at $4 user/month and is pretty good for most of the stuff. Not as sophisticated as GitLab though. I hope there is a fix to this actions and lambda incompatibility, in which case I might fully migrate to GitHub.
Nemanja Kostic 