Speeding up git operations for CI/CD

I recently found myself undergoing an epic journey to speed up our CICD builds at work. After some initial digging, I found that our git clone operations were taking 3 minutes and pulling 2gb of data, just to test a tiny Golang binary (3,500 LoC,  20mb).

With some easy tweaks, I was able to reduce our clone time to 5 seconds and only pull 35mb of data.

Faster Git clones

*Requires a modern version of git (v2.25+) with sparse-checkout support.

# clone our repo. Note the blob filter and --no-checkout
git clone --no-checkout --filter=blob:none ssh://[email protected]:22/monorepo.git

# Pop into the folder. If you ls, there is only the .git/ foldere
cd monorepo

## Set up our sparse checkout
git sparse-checkout init --cone

# Specify the paths we care about 
git sparse-checkout set /your_path/ /some/other/path

# Pull the files.
git read-tree -mu HEAD

Additional links

I found these three links very helpful in my research:

• https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/
• https://dev.to/patricksevat/optimize-your-git-clone-fetch-strategy-for-ci-pipeline-3pka
• https://stackoverflow.com/questions/9572407/git-sparse-checkout-with-exclusion