Monorepo: A shared codebase that stores multiple projects in a single repository.

tldr: Final script is at the bottom of the post if you’d like to skip the walkthrough.

At least the above definition is what I think of a monorepo. I see a monorepo as a single repository that has subprojects that generates separate artifacts under each project. Each of these projects can be related or totally unrelated.

Some people frown upon the monorepo concept, however there is some empirical evidence to its usefulness by those who’ve adopted them (most notably Google). In my opinion the monorepo concept has it’s places and can definitely speed up development process since you only need to setup 1 git repository and 1 other artifact store. An artifact store being like a Docker repository, Nexus, or event something simple like Amazon S3.

Now - CircleCI doesn’t yet support monorepo builds (at the time of writing) but I’ll show you how you can achieve that. In this article I’ll be using Docker to generate images as artifacts. Not using Docker? Don’t worry!!! This same approach can work if your projects generate some other type of artifact like a jar file or something else. As long as you have a way to build your individual projects from the CLI this method will work.

Let’s get Started!

Codebase layout

The layout of the codebase forms the base for how we’re going to be organizing and building our code. In my solution I choose to have 1 high level directory for my projects and then subfolders with Dockerfiles for each project like so:

+-- monorepo/
|	+--project_1/
|		+--Dockerfile
|		+--src/
|			+--project_1_code.py
|	+--project_2/
|		+--Dockerfile
|		+--src/
|			+--project_2_code.py

For my example - I’ve used simple python files under their own src directory but you can probably use whatever you want. Building with Docker makes it really simple here as then each of our projects can be whatever code we want. We can use python/java/go it wouldn’t matter as long as we can build with docker build command.

If you’re not using docker to build: still organize each of your projects under their own directory.

Defining our config.yml file

The config.yml powers the build process on CircleCI. If you haven’t already, create/add your CircleCI yml file to your repository. You want to define that under .circleci/config.yml at the top level of your Git repository. Let’s define that file like so to use version 2.1. Please do your due diligence to check for newer versions though.

I’m going to be pushing built docker images to AWS ECR but you can push your artifacts anywhere you like. So I’m adding in the ECR orb like so:

version: 2.1

orbs:
  aws-ecr: circleci/aws-ecr@6.12.2

Workflows

First things first let's define the high level workflow. I wanted a way to just say "build and push this project" with as few lines of code as possible. Let's add a workflows component to our config.yml. 
workflows:
  version: 2.1
  build-push:
    jobs:
      - build_and_push

Now this defines that our CI pipeline has 1 workflow with 1 step called build-push which calls to run 1 job build_and_push. Let’s define that build_and_push job now.

Jobs

jobs:
  build_and_push: # this defines the job name
    docker:
      - image: cimg/base:2020.01

    steps:
    - checkout
    - setup_remote_docker
    - aws-ecr/ecr-login

    - run:
        name: Get the short Git hash of commit
        command: |
          GIT_SHORT_HASH=$(echo $CIRCLE_SHA1 | cut -c -7)
          echo "export GIT_HASH=`echo ${GIT_SHORT_HASH}`" >> $BASH_ENV
          source $BASH_ENV

    - build-and-push-project:
        project: "project-1"

    - build-and-push-project:
        project: "project-2"

The above job manages the build for each project. Before it does the build it does the following 1) checkout of the Git repo, 2) setups up a remote docker to do the Docker build’s in, 3) and then logs you into AWS ECR (note: you need to have specified the proper environment variables for CircleCI to be able to login. See the documentation on the CircleCI AWS-ECR Orb for that.

Build section

In the run command you can see we collect the information for the GIT_SHORT_HASH which we’ll use to version our build with.

And here comes the part which actually build’s each project

build-and-push-project
	project: "input-project-name-here"

This simple command does exactly what it says! It accepts as an input the project name which is defined just by the directory name in the monorepo that the project lives under. Let’s define this step next.

Build and Push Command

Most of the time you’re not changing every project in your monorepo at the same time. Typically you work on 1 project or a few at a time. So we need a way to differentiate which code to build and which to not build. Using some clever tricks with the git diff command we can determine which projects in the repo were changed. Using the git diff command we’re able to check the difference between the HEAD commit and the previous commit (HEAD^) on that branch to determine which projects get built.

commands:
  build-and-push-project: # this defines the command name
    description: "Builds project if there is any detected changes"
    parameters:
      project:
        type: string
        default: "Default"
    steps:
      - run:
          name: Docker Build and Push | << parameters.project >>
          command: |
            PROJECT=<< parameters.project >>
            if git diff --name-only HEAD^...HEAD | grep "${PROJECT}/"; then

              TAG=${PROJECT}-${GIT_HASH}

              docker build -t $AWS_ECR/your-repo:$TAG ./${PROJECT}
              docker push $AWS_ECR/your-repo:$TAG

              echo "Pushed new image: your-repo:$TAG"

            else
              echo "No changes were detected"
              exit 0
            fi

What is nice about this command is we’ve defined an input parameter to this which is the project name. So we can repeatedly call this command with a different project name in our mono repo - and it will check to see if that project has had any changes since the last commit.

Final Script

version: 2.1

orbs:
  aws-ecr: circleci/aws-ecr@6.12.2

# Workflows: orchestrating multiple jobs.
# Jobs: running a series of steps that perform commands.
workflows:
  version: 2.1
  build-push:
    jobs:
      - build_and_push

jobs:
  build_and_push:
    docker:
      - image: cimg/base:2020.01

    steps:
    - checkout
    - setup_remote_docker
    - aws-ecr/ecr-login

    - run:
        name: Get the short Git hash of commit
        command: |
          GIT_SHORT_HASH=$(echo $CIRCLE_SHA1 | cut -c -7)
          echo "export GIT_HASH=`echo ${GIT_SHORT_HASH}`" >> $BASH_ENV
          source $BASH_ENV

    - build-and-push-project:
        project: "project-1"

    - build-and-push-project:
        project: "project-2"


commands:
  build-and-push-project:
    description: "Builds project if there is any detected changes"
    parameters:
      project:
        type: string
        default: "Default"
    steps:
      - run:
          name: Docker Build and Push | << parameters.project >>
          command: |
            PROJECT=<< parameters.project >>
            if git diff --name-only HEAD^...HEAD | grep "${PROJECT}/"; then

              TAG=${PROJECT}-${GIT_HASH}

              docker build -t $AWS_ECR/your-repo:$TAG ./${PROJECT}
              docker push $AWS_ECR/your-repo:$TAG

              echo "Pushed new image: your-repo:$TAG"

            else
              echo "No changes were detected"
              exit 0
            fi

If you’ve read all of this thank you so much! I really enjoy working with CircleCI and really hope to add more content sometime. Don’t forget to use the most up to date versions after you get this working!