Removing Secrets and Large Files from Git History

It happens to almost everyone: a password or API key gets committed. What matters is what you do next.

Step 1: Treat the secret as leaked#

If it reached a public repository, assume that it has been copied. Automated bots scan GitHub for keys constantly.

Step 2: Understand why deleting is not enough#

A new commit that deletes the file leaves it in the history. Here .env has been removed, but look:

ls -a
git log --oneline
git show HEAD~1:.env

.
..
.git
app.js
f085982 Remove settings file
2155374 Add settings
e06fcc0 Add app
API_KEY=sk_test_123456

The key can still be read by anyone with a copy of the repository.

Step 3: Find it#

git log -S finds commits that added or removed the text:

git log --oneline -S "sk_test_123456"

f085982 Remove settings file
2155374 Add settings

And this lists every commit that touched a file:

git log --oneline --all -- .env

f085982 Remove settings file
2155374 Add settings

Step 4: Remove it from history#

If the commit is recent and was not pushed, you can simply reset or amend it.

If it was pushed, every commit from that point needs rewriting. The recommended tool is git filter-repo, which is installed separately:

pip install git-filter-repo
git filter-repo --invert-paths --path .env

This removes the file from every commit, so every commit ID after it changes. filter-repo also removes origin as a precaution, so add the remote back and force-push:

git remote add origin https://github.com/your-name/app.git
git push --force --all
git push --force --tags

Then every teammate must delete their copy and clone again, or the old commits may be pushed back. Copies can still exist in forks, in caches, and in other people's clones, which is why step 1 comes first. GitHub's guide on removing sensitive data explains the remaining steps.

Prevention#

• Add .env to .gitignore before the first commit.
• Commit a .env.example with fake values, so people know which settings are needed.
• Enable secret scanning and push protection in your repository's settings, so GitHub blocks known key formats.
• Add a pre-commit hook, or a scanner such as gitleaks, to catch them on your own computer.

Large files#

Git keeps every version of every file forever, so a large binary file slows down every clone, even after you delete it. GitHub warns about files over 50 MB and rejects files over 100 MB.

See how big the repository is:

git count-objects -vH | grep -E 'count|size-pack'

count: 7
size-pack: 0 bytes

The numbers here are tiny. On a real project, size-pack shows the compressed size of the history after packing.

Git LFS#

Git Large File Storage stores big files on a separate server, and keeps only a small pointer in the repository. It suits files such as videos, datasets and design files.

git lfs install
git lfs track "*.psd"
git add .gitattributes
git add design.psd
git commit -m "Add design file"

Tracking rules are saved in .gitattributes, which you commit. GitHub's free accounts have LFS storage limits.

If a large file is in your history already, remove it with git filter-repo:

git filter-repo --strip-blobs-bigger-than 50M

Better still, keep generated files, builds and datasets out of Git altogether.

Common mistakes#

Practice#

git log --format=%s -S "sk_test_123456" | tail -n 1

Add settings

Key takeaway#

Next: How Git Works Inside.