Building an AWS Serverless Architecture Using Nx Monorepo Tool and Terraform
I tried building an AWS Serverless Architecture with Nx and terraform!
Hello. I'm Kurihara, from the CCoE team at KINTO Technologies and I’m passionate about creating DevOps experiences that bring joy to developers.
As announced at the AWS Summit Tokyo 2023: our DBRE team’s approach to both agility and governance of our vehicle subscription service KINTO, is to deploy a platform that provides temporary jump servers (called “DBRE platform” from now on) across our company, triggered by requests from Slack.
This DBRE platform is implemented using a combination of several AWS serverless services. In this article, we will introduce how we improved the developer experience by using a Monorepo tool called Nx and terraform. Our aim is to provide insights that can benefit anyone interested in adopting a Monorepo development approach, irrespective of their focus on serverless architectures.
Background and Issues
The architecture of our DBRE platform looks as follows:
In addition to the above, there are about 20 Lambdas developed via Golang, Step Functions to orchestrate them, DynamoDB and EventBridge for scheduled triggers.
The following issues and requests were raised in the development process.
- Integrate with “terraform plan” and “terraform apply workflows for secure deployment
- Incorporate appropriate static code analysis such as Formatter, Linter, etc.
When considering the development of a serverless architecture, conventional choices like SAM or serverless framework come to mind. However,we decided against it because we wanted to implement IaC with Terraform and because Lambda functions developed in Golang lacked support.
Let's look at the Terraform Lambda module. I thought that if I could make a proper Zip of the Lambda code to be referenced in Terraform, I could potentially resolve the issue of wanting to implement IaC with terrafrom.
resource "aws_lambda_function" "test_lambda" {
# If the file is not in the current working directory you will need to include a
# path.module in the filename.
filename = "lambda_function_payload.zip"
function_name = "lambda_function_name"
role = aws_iam_role.iam_for_lambda.arn
handler = "index.test"
source_code_hash = data.archive_file.lambda.output_base64sha256
runtime = "nodejs16.x"
environment {
variables = {
foo = "bar"
}
}
}
Furthermore, consider the latter request to properly incorporate static code analysis. Serverless development is a combination of smaller code bases. In other words, we considered introducing the Monorepo tool with the idea that it would facilitate integration with development tools and keep build scripts simple by clearly defining the boundaries of the codebase group.
What is a Monorepo tool?
To get straight to the point, we took the decision to use a TypeScript-made Monorepo tool called Nx.
We opted for Monorepo.tools primarily due to its extensive coverage of functions, as highlighted on the Monorepo tool comparison site. Additionally, its JavaScript-based architecture appealed to us as we thought it would be beneficial for scalability and accommodate future growth effectively. (Assuming the barriers of entry into the front-end community are low.)
Examples will be given in the next chapter, but the premise is: What is Monorepo and what does Nx do? I will now explain briefly.
Defining terms
Let us take a moment to clarify how we've aligned the terms used in this document to match the conventions of Nx:
Project: one repository-like bulk in monorepo (e.g. single Lambda code, common modules)Task: A generic term for the processes required to build an application, such as test, build, deploy, etc.
What monorepo is
It is described as a single repository where related projects are stored in isolated and well-defined relationships. In contrast, there is the multi-repository configuration often referred to in the Web realm as polyrepo.
Source: monorepo.tools
In summary, monorepo.tools offers the following advantages
- Atomic commits on a per-system basis
- Easy deployment of common modules (when a common module is updated, it can be used immediately without the need to import, etc.)
- Easier to be aware of the system as a whole, rather than vertically divided (in terms of mindset)
- Less workload required when setting up a new repository
While AWS CDK isn't categorized as a Monorepo tool, it shares a similar philosophy regarding the management of IaC and application code, aligning with the trend observed with Monorepo to consolidate both infrastructure and application code within a single repository.
We discovered that failures are often related to "out-of-band" changes to an application that aren't fully tested, such as configuration changes. Therefore, we developed the AWS CDK around a model in which your entire application is defined in code, not only business logic but also infrastructure and configuration. …and fully rolled back if something goes wrong. - https://docs.aws.amazon.com/cdk/v2/guide/best-practices.html
What Nx can do
Roughly speaking, if you define your own tasks and dependencies for each project, Nx will orchestrate the tasks.
The following is an example of defining tasks and dependencies for a terraform project. When defined in this way, the plan-developmentwill first build (compile and zip-compressed) the Lambda code with the defined dependencies, and then run terraform plan. fmt and test can also be defined simply as terraform project-specific tasks.
By clarifying the responsibilities of each code base this way, we can improve the overall outlook of the code. It is possible to incorporate development tools suited to each development language on a project-by-project basis, and it is possible to build an appropriate development flow without having to rely on builders.
Practical examples at KTC
The following is an excerpt from the aforementioned DBRE platform, simplified and illustrated with practical examples.
There are two Golang Lambda codes, both using the same common module. The Lambda code project is responsible for compiling its own code and creating a Zip file so that it can be deployed from terraform.
The directory structure looks like this. 
Project Definition
Project definitions for each of the above four projects are listed below.
①: Common modules
- In Golang, common modules only need to be referenced by the user, so builds are not required and only static analysis and UT are defined as tasks.
projects/dbre-toolkit/lambda-code/shared-modules/package.json
{
"name": "shared-modules",
"scripts": {
"fmt-fix": "gofmt -w -d",
"fmt": "gofmt -d .",
"test": "go test -v"
}
}
(2), (3): Lambda code
- By registering a common module as a dependent project, it is defined that if the code of the
common module is changed, the task needs to be executed. - The
buildtask is responsible for executing the go build and zipping the generated binaries, which will later be used in the terraform project.
projects//dbre-toolkit/lambda-code/lambda-code-01/package.json
{
"name": "lambda-code-01",
"scripts": {
"fmt-fix": "gofmt -w -d .",
"fmt": "gofmt -d .",
"test": "go test -v",
"build": "cd ../ && GOOS=linux GOARCH=amd64 CGO_ENABLED=0 go build -o lambda-code-01/dist/main lambda-code-01/main.go && cd lambda-code-01/dist && zip lambda-code.zip main"
},
"nx": {
"implicitDependencies": [
"shared-modules"
]
}
}
④: IaC
- When
plan-${env}orapply-${env}is executed, thebuildof the Lambda code specified in the dependency is executed first (the necessary zip is generated when plan or apply is executed)
projects//dbre-toolkit/iac/package.json
{
"name": "iac",
"scripts": {
"fmt": "terraform fmt -check -diff -recursive $INIT_CWD",
"fmt-fix": "terraform fmt -recursive -recursive $INIT_CWD",
"test": "terraform validate",
"plan-development": "cd development && terraform init && terraform plan",
"apply-development": "cd development && terraform init && terraform apply -auto-approve"
},
"nx": {
"implicitDependencies": [
"lambda-code-01",
"lambda-code-02"
],
"targets": {
"plan-development": {
"dependsOn": [
"^build"
]
},
"apply-development": {
"dependsOn": [
"^build"
]
}
}
}
}
From the terraform module, refer to the Zip file generated in the previous step as follows.
local {
lambda_code_01_zip_path = "${path.module}/../../../lambda-code/lambda-code-01/dist/lambda-code.zip"
}
# Redacted
resource "aws_lambda_function" "lambda-code-01" {
function_name = "lambda-code-01"
architectures = ["x86_64"]
runtime = "go1.x"
package_type = "Zip"
filename = local.lambda_code_01_zip_path
handler = "main"
source_code_hash = filebase64sha256(local.lambda_code_01_zip_path)
}
Task Execution
Now that each project has been divided and tasks defined, we will look at task execution.
In Nx, the run-many subcommand can be used to execute specific tasks for a specific project or for all projects. Based on dependencies, they are executed in parallel when possible, which also speeds up the process.
nx run-many --target=<defined task name> --projects=<project name comma separated>.- nx run-many --target=<defined task name> --all
Example of executing plan-development for an iac project. Tasks with dependencies will execute tasks based on the defined dependencies.
This is exactly the point I wanted to make. It will execute the tasks of the dependent project ahead of time, thus ensuring that the Lambda code is properly zipped when terraform is executed.
$ nx run-many --target=plan-development --projects=iac --verbose
> NX Running target plan-development for 1 project(s) and 2 task(s) they depend on:
- iac
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run lambda-code-01:build
updating: main (deflated 56%)
> nx run lambda-code-02:build
updating: main (deflated 57%)
> nx run iac:plan-development
Initializing modules...
Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v4.39.0
terraform has been successfully initialized!
--redacted
}
Plan: 0 to add, 2 to change, 0 to destroy.
Example of executing the test task for all projects. No task dependencies, so everything runs in parallel
Tasks with no dependencies, such as UT, can be executed in parallel. This allows for CI execution, as well as for development rules such as "Always run UT before pushing to GitHub" to be resolved with a single command.
$ nx run-many --target=test --all --verbose
> NX Running target test for 4 project(s):
- lambda-code-01
- lambda-code-02
- shared-modules
- iac
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run shared-modules:test
? github.com/kinto-dev/dbre-platform/dbre-toolkit/shared-modules [no test files]
> nx run lambda-code-01:test
=== RUN Test01
--- PASS: Test01 (0.00s)
PASS
ok github.com/kinto-dev/dbre-platform/dbre-toolkit/lambda-code-01 0.255s
> nx run iac:test
Success! The configuration is valid.
> nx run lambda-code-02:test
=== RUN Test01
--- PASS: Test01 (0.00s)
PASS
ok github.com/kinto-dev/dbre-platform/dbre-toolkit/lambda-code-02 0.443s
——————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target test for 4 projects
> nx run lambda-code-02:test
Powerful features of Nx and Monorepo tools
We hope you can see how tasks can be orchestrated by properly defining the project. However, this alone is no different from a regular task runner, so here are some of the major advantages of using the Nx and Monorepo tools.
Execute tasks only for changed projects
The fastest task execution is to not execute the task in the first place. A mechanism called the affected command, which performs tasks only for the changed project, is available for fast completion of CI.
The following is the command syntax By passing two Git pointers, it will only execute tasks in the project that have changed between the two pointers. nx affected --target=<task name> --base=<two dots diff of base> --head=<two dots diff of head>
# State with changes only in lambda-code-01
$ git diff main..feature/111 --name-only
projects/dbre-toolkit/lambda-code/lambda-code-01/main.go
$ nx affected --target=build --base=main --head=feature/111 --verbose
> NX Running target build for 1 project(s):
- lambda-code-01
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run lambda-code-01:build
updating: main (deflated 57%)
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX Successfully ran target build for 1 projects
If there is a change in the project on which it depends on, it will execute tasks based on the dependencies.
# State with changes only in shared-module
$ git diff main..feature/222 --name-only
projects/dbre-toolkit/lambda-code/shared-modules/utility.go
# Tasks in projects that depend on shared-module are executed
$ nx affected --target=build --base=main --head=feature/222 --verbose
> NX Running target build for 2 project(s):
- lambda-code-01
- lambda-code-02
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> nx run lambda-code-01:build
updating: main (deflated 56%)
> nx run lambda-code-02:build
updating: main (deflated 57%)
————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
> NX
Simplifying the CI/CD pipeline
If task names do not change, the CI/CD pipeline does not need to be changed as projects are added, thus lowering maintenance costs. In addition, the affected command described above can speed up the CI/CD process (since it only executes tasks for the changed project).
Below is an example of CI for GitHub Actions.
name: Continuous Integration
on:
pull_request:
branches:
- main
- develop
types: [opened, reopened, synchronize]
jobs:
ci:
runs-on: ubuntu-latest
steps:
-uses: actions/checkout@v3
with:
fetch-depth: 0
# --immutable option to have the fixed version of dependencies listed in yarn.lock installed
- name: install npm dependencies
run: yarn install --immutable
shell: bash
- uses: actions/setup-go@v3
with:
go-version: '^1.13.1'
- uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.3.5
- name: configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1-node16
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: 'ap-northeast-1'
# Task execution part is completed with this amount of description
- name: format check
run: nx affected --verbose --target fmt --base=remotes/origin/${{ github.base_ref }} --head=remotes/origin/${{ github.head_ref }}
- name: test
run: nx affected --verbose --target test --base=remotes/origin/${{ github.base_ref }} --head=remotes/origin/${{ github.head_ref }}
- name: build
run: nx affected --verbose --target build --base=remotes/origin/${{ github.base_ref }} --head=remotes/origin/${{ github.head_ref }}
- name: terraform plan to development
run: nx affected --verbose --target plan-development --base=remotes/origin/${{ github.base_ref }} --head=remotes/origin/${{ github.head_ref }}
Combine with Git Hook for even greater productivity
I'd like to see at least static analysis and Unit Test done locally before pushing with Git. Development rules such as 'Git history is dirty too' can be easily solved. By combining the --files and --uncommitted options of the affected command with the Git Hook, only the project to which the changed files belong can be targeted, minimizing the developer's stress (and time spent on execution).
For example, the following affected command can be included in the pre-commit hook to keep the commit history clean and reduce review noise.
nx affected --target lint --files $(git diff --cached --name-only):nx affected --target unit-test --files $(git diff --cached --name-only)nx affected --target fmt-fix --files $(git diff --cached --name-only)
Other Benefits
Task execution results are cached if the project code has not changed
The results of task execution are cached, both in the generated files and standard output/errors. (For more information, click here.)
$ tree .nx-cache/
├── ce36b7825abacc0613a8b2c606c65db6def0e5ca9c158d5c2389d0098bf646a1
│ ├── code
│ ├── outputs
│ │ └── projects
│ │ └── dbre-toolkit
│ │ └── lambda-code
│ │ └── lambda-code-01
│ │ └── dist
│ │ ├── lambda-code.zip
│ │ └── main
│ └── terminalOutput
├── ce36b7825abacc0613a8b2c606c65db6def0e5ca9c158d5c2389d0098bf646a1.commit
├── nxdeps.json
├── run.json
└── terminalOutputs
├── 1c9b46c773287538b1590619bfa5c9abf0ff558060917a184ea7291c6f1b988c
├── 6f2fbb5f2dd138ec5e7e261995be0d7cddd78e7a81da2df9a9fe97ee3c8411c5
├── 88c7015641fa6e52e0d220f0fdf83a31ece942b698c68c4455fa5dac0a6fd168
├── 9dc8ebe6cdd70d8b5d1b583fbc6b659131cda53ae2025f85037a3ca0476d35b8
├── c4267c4148dc583682e4907a7692c2beb310ebd2bf9f722293090992f7e0e793
├── ce36b7825abacc0613a8b2c606c65db6def0e5ca9c158d5c2389d0098bf646a1
├── db7e612621795ef228c40df56401ddca2eda1db3d53348e25fe9d3fe90e3e9a1
├── dc112e352c958115cb37eb86a4b8b9400b64606b05278fe7e823bc20e82b4610
└── eb94fd3a7329ab28692a2ae54a868dccae1b4730e4c15858e9deb0e2232b02f3
If this caching mechanism is also integrated into our CI/CD pipeline, it optimizes processing tasks during code reviews. For instance, when only a portion of the code requires modification, the cache can expedite most CI processes for the updated push, thereby enhancing development efficiency.
- name: set nx cache dir to environment variables
id: set-nx-version
run: |
echo "NX_CACHE_DIRECTORY=$(pwd)/.nx-cache" >> $GITHUB_ENV
shell: bash
# Register nx cache to GitHub cache
- name: nx cache action
uses: actions/cache@v3
id: nx-cache
with:
path: ${{ env.NX_CACHE_DIRECTORY }}
key: nx-cache-${{ runner.os }}-${{ github.sha }}
restore-keys: |
nx-cache-${{ runner.os }}-
The graph command allows visualization of project dependencies
Even though the boundaries of the code base have been clarified, there are still times when you want to check dependencies comprehensively. A graph subcommand is maintained to visualize dependencies between projects. One of the benefits of Nx is its ability to handle such tasks.
Current status of DBRE platform
The DBRE platform currently has 28 projects with Monorepo. In the above example, the number of projects was small, so it may have been difficult to understand their benefits, but with a scale of this extent, the benefits of the affected commands come through like shining stars.
$ yarn workspaces list --json
{"location":".","name":"dbre-platform"}
{"location":"dbre-utils","name":"dbre-utils"}
{"location":"projects/DBREInit/iac","name":"dbre-init-iac"}
{"location":"projects/DBREInit/lambda-code/common","name":"dbre-init-lambda-code-common"}
{"location":"projects/DBREInit/lambda-code/common-v2","name":"dbre-init-lambda-code-common-v2"}
{"location":"projects/DBREInit/lambda-code/push-output","name":"dbre-init-lambda-code-push-output"}
{"location":"projects/DBREInit/lambda-code/s3-put","name":"dbre-init-lambda-code-s3-put"}
{"location":"projects/DBREInit/lambda-code/sf-check","name":"dbre-init-lambda-code-sf-check"}
{"location":"projects/DBREInit/lambda-code/sf-collect","name":"dbre-init-lambda-code-sf-collect"}
{"location":"projects/DBREInit/lambda-code/sf-notify","name":"dbre-init-lambda-code-sf-notify"}
{"location":"projects/DBREInit/lambda-code/sf-setup","name":"dbre-init-lambda-code-sf-setup"}
{"location":"projects/DBREInit/lambda-code/sf-terminate","name":"dbre-init-lambda-code-sf-terminate"}
{"location":"projects/PowerPole/iac","name":"powerpole-iac"}
{"location":"projects/PowerPole/lambda-code/pp","name":"powerpole-lambda-code-pp"}
{"location":"projects/PowerPole/lambda-code/pp-approve","name":"powerpole-lambda-code-pp-approve"}
{"location":"projects/PowerPole/lambda-code/pp-request","name":"powerpole-lambda-code-pp-request"}
{"location":"projects/PowerPole/lambda-code/sf-deploy","name":"powerpole-lambda-code-sf-deploy"}
{"location":"projects/PowerPole/lambda-code/sf-notify","name":"powerpole-lambda-code-sf-notify"}
{"location":"projects/PowerPole/lambda-code/sf-setup","name":"powerpole-lambda-code-sf-setup"}
{"location":"projects/PowerPole/lambda-code/sf-terminate","name":"powerpole-lambda-code-sf-terminate"}
{"location":"projects/PowerPoleChecker/iac","name":"powerpolechecker-iac"}
{"location":"projects/PowerPoleChecker/lambda-code/left-instances","name":"powerpolechecker-lambda-code-left-instances"}
{"location":"projects/PowerPoleChecker/lambda-code/sli-notifier","name":"powerpolechecker-lambda-code-sli-notifier"}
{"location":"projects/dbre-toolkit/docker-image/shenron-wrapper","name":"dbre-toolkit-docker-image-shenron-wrapper"}
{"location":"projects/dbre-toolkit/iac","name":"dbre-toolkit-iac"}
{"location":"projects/dbre-toolkit/lambda-code/dt-list-dbcluster","name":"dbre-toolkit-lambda-code-dt-list-dbcluster"}
{"location":"projects/dbre-toolkit/lambda-code/dt-make-markdown","name":"dbre-toolkit-lambda-code-dt-make-markdown"}
{"location":"projects/dbre-toolkit/lambda-code/utility","name":"dbre-toolkit-lambda-code-utility"}
IaC in terraform is also divided into four projects in component units. This ability to easily split up projects allows each code base to remain slim in size, even in a single repository. The affected command also allows CI/CD to be completed faster, increasing productivity without reducing the development experience.
$ yarn list-projects | grep iac
{"location":"projects/DBREInit/iac","name":"dbre-init-iac"}
{"location":"projects/PowerPole/iac","name":"powerpole-iac"}
{"location":"projects/PowerPoleChecker/iac","name":"powerpolechecker-iac"}
{"location":"projects/dbre-toolkit/iac","name":"dbre-toolkit-iac"}
Issues
We will also present the challenges we faced in completing this development architecture and how we solved them.
As mentioned in the introduction, zipping the Lambda code was an important point, but unless the execution environment and zip metadata (update date, etc.) were completely the same, differences would be detected in terraform even if the code was unchanged. The solution was to build and zip the code in the container and call it from the task definition.
Dockerfile
FROM golang:1.20-alpine
RUN apk update && \
apk fetch zip && \
apk --no-cache add --allow-untrusted zip-3.0-r*.apk bash
COPY ./docker-files/go-single-module-build.sh /opt/app/go-single-module-build.sh
./docker-files/go-single-module-build.sh
#!/bin/bash
set -eu -o pipefail
while getopts "d:m:b:h" OPT; do
case $OPT in
d) SOURCE_ROOT_RELATIVE_PATH="$OPTARG" ;;
m) MAIN_GO="$OPTARG" ;;
b) BINARY_NAME="$OPTARG" ;;
h) help ;;
*) exit ;;
esac
done
shift $((OPTIND - 1))
cd "/opt/mounted/$SOURCE_ROOT_RELATIVE_PATH" || exit 1
rm -f ./dist/*
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -o "./dist/$BINARY_NAME" "$MAIN_GO"
cd ./dist || exit 1
# for sha256 diff
chown "$HOST_USER_ID":"$HOST_GROUP_ID" "$BINARY_NAME"
touch --no-create -t 01010000 "$BINARY_NAME" ./*.tmpl
zip "$BINARY_NAME.zip" "$BINARY_NAME" ./*.tmpl
chown -R "$HOST_USER_ID":"$HOST_GROUP_ID" ../dist
There are other issues as well, such as the current lack of local execution. I would like to try to make not only terraform but also SAM and cdk into Monorepo in the future.
Summary
In this article, we introduced the powerful features of Nx based on the introduction of how to manage AWS serverless using the Monorepo tool. If this sounds like something you would like to do, would you like to consider working with us at the Platform Group? Thank you for reading.
関連記事 | Related Posts
Developing a System to Investigate Lock Contention (Blocking) Causes in Aurora MySQL
Aurora MySQL におけるロック競合（ブロッキング）の原因を事後調査できる仕組みを作った話
The need for DBRE in KTC
Deployment Process in CloudFront Functions and Operational Kaizen
Efforts to Implement the DBRE Guardrail Concept
Implementing BlueGreenDeployment with GitHub Actions + ECS
We are hiring!
【DBRE】DBRE G／東京・名古屋・大阪
DBREグループについてKINTO テクノロジーズにおける DBRE は横断組織です。自分たちのアウトプットがビジネスに反映されることによって価値提供されます。
【プラットフォームエンジニア】プラットフォームG／東京・大阪
プラットフォームグループについてAWS を中心とするインフラ上で稼働するアプリケーション運用改善のサポートを担当しています。