Speeding up git operations for CI/CD
1 min read

Speeding up git operations for CI/CD

I recently found myself undergoing an epic journey to speed up our CICD builds at work. After some initial digging, I found that our git clone operations were taking 3 minutes and pulling 2gb of data, just to test a tiny Golang binary (3,500 LoC,  20mb).

With some easy tweaks, I was able to reduce our clone time to 5 seconds and only pull 35mb of data.

Faster Git clones

*Requires a modern version of git (v2.25+) with sparse-checkout support.

# clone our repo. Note the blob filter and --no-checkout
git clone --no-checkout --filter=blob:none ssh://git@example.com:22/monorepo.git

# Pop into the folder. If you ls, there is only the .git/ foldere
cd monorepo

## Set up our sparse checkout
git sparse-checkout init --cone

# Specify the paths we care about 
git sparse-checkout set /your_path/ /some/other/path

# Pull the files.
git read-tree -mu HEAD

I found these three links very helpful in my research: