Git extension for versioning large files

Git Large File Storage

CI status

Git LFS is a command line extension and specification for managing large files with Git.

The client is written in Go, with pre-compiled binaries available for Mac, Windows, Linux, and FreeBSD. Check out the website for an overview of features.

Getting Started

Downloading

You can install the Git LFS client in several different ways, depending on your setup and preferences.

  • Linux users. Debian and RPM packages are available from PackageCloud.
  • macOS users. Homebrew bottles are distributed, and can be installed via brew install git-lfs.
  • Windows users. Git LFS is included in the distribution of Git for Windows. Alternatively, you can install a recent version of Git LFS from the Chocolatey package manager.
  • Binary packages. In addition, binary packages are available for Linux, macOS, Windows, and FreeBSD.
  • Building from source. This repository can also be built from source using the latest version of Go, and the available instructions in our Wiki.

Note that Debian and RPM packages are built for all OSes for amd64 and i386. For arm64, only Debian packages for the latest Debian release are built due to the cost of building in emulation.

Installing

From binary

The binary packages include a script which will:

  • Install Git LFS binaries onto the system $PATH
  • Run git lfs install to perform required global configuration changes.
$ ./install.sh

From source

  • Ensure you have the latest version of Go, GNU make, and a standard Unix-compatible build environment installed.
  • On Windows, install goversioninfo with go get github.com/josephspurrier/goversioninfo/cmd/goversioninfo.
  • Run make.
  • Place the git-lfs binary, which can be found in bin, on your system’s executable $PATH or equivalent.
  • Git LFS requires global configuration changes once per-machine. This can be done by running:
$ git lfs install

Verifying releases

Releases are signed with the OpenPGP key of one of the core team members. To get these keys, you can run the following command, which will print them to standard output:

$ curl -L https://api.github.com/repos/git-lfs/git-lfs/tarball/core-gpg-keys | tar -Ozxf -

Once you have the keys, you can download the sha256sums.asc file and verify the file you want like so:

$ gpg -d sha256sums.asc | grep git-lfs-linux-amd64-v2.10.0.tar.gz | shasum -a 256 -c

Example Usage

To begin using Git LFS within a Git repository that is not already configured for Git LFS, you can indicate which files you would like Git LFS to manage. This can be done by running the following from within a Git repository:

$ git lfs track "*.psd"

(Where *.psd is the pattern of filenames that you wish to track. You can read more about this pattern syntax here).

Note: the quotation marks surrounding the pattern are important to prevent the glob pattern from being expanded by the shell.

After any invocation of git-lfs-track(1) or git-lfs-untrack(1), you must commit changes to your .gitattributes file. This can be done by running:

$ git add .gitattributes
$ git commit -m "track *.psd files using Git LFS"

You can now interact with your Git repository as usual, and Git LFS will take care of managing your large files. For example, changing a file named my.psd (tracked above via *.psd):

$ git add my.psd
$ git commit -m "add psd"

Tip: if you have large files already in your repository's history, git lfs track will not track them retroactively. To migrate existing large files in your history to use Git LFS, use git lfs migrate. For example:

$ git lfs migrate import --include="*.psd" --everything

For more information, read git-lfs-migrate(1).

You can confirm that Git LFS is managing your PSD file:

$ git lfs ls-files
3c2f7aedfb * my.psd

Once you've made your commits, push your files to the Git remote:

$ git push origin main
Uploading LFS objects: 100% (1/1), 810 B, 1.2 KB/s
# ...
To https://github.com/git-lfs/git-lfs-test
   67fcf6a..47b2002  main -> main

Note: Git LFS requires at least Git 1.8.2 on Linux or 1.8.5 on macOS.

Uninstalling

If you've decided that Git LFS isn't right for you, you can convert your repository back to a plain Git repository with git lfs migrate as well. For example:

$ git lfs migrate export --include="*.psd" --everything

Note that this will rewrite history and change all of the Git object IDs in your repository, just like the import version of this command.

If there's some reason that things aren't working out for you, please let us know in an issue, and we'll definitely try to help or get it fixed.

Limitations

Git LFS maintains a list of currently known limitations, which you can find and edit here.

Git LFS source code utilizes Go modules in its build system, and therefore this project contains a go.mod file with a defined Go module path. However, we do not maintain a stable Go language API or ABI, as Git LFS is intended to be used solely as a compiled binary utility. Please do not import the git-lfs module into other Go code and do not rely on it as a source code dependency.

Need Help?

You can get help on specific commands directly:

$ git lfs help <subcommand>

The official documentation has command references and specifications for the tool. There's also a FAQ on the wiki which answers some common questions.

If you have a question on how to use Git LFS, aren't sure about something, or are looking for input from others on tips about best practices or use cases, feel free to start a discussion.

You can always open an issue, and one of the Core Team members will respond to you. Please be sure to include:

  1. The output of git lfs env, which displays helpful information about your Git repository useful in debugging.
  2. Any failed commands re-run with GIT_TRACE=1 in the environment, which displays additional information pertaining to why a command crashed.

Contributing

See CONTRIBUTING.md for info on working on Git LFS and sending patches. Related projects are listed on the Implementations wiki page.

See also SECURITY.md for info on how to submit reports of security vulnerabilities.

Core Team

These are the humans that form the Git LFS core team, which runs the project.

In alphabetical order:

@bk2204 @chrisd8088 @larsxschneider
PGP 0223B187 PGP 088335A9 PGP A5795889

Alumni

These are the humans that have in the past formed the Git LFS core team, or have otherwise contributed a significant amount to the project. Git LFS would not be possible without them.

In alphabetical order:

@andyneff @PastelMobileSuit @rubyist @sinbad @technoweenie @ttaylorr
Comments
  • Upgrade workflows to latest Ubuntu and Actions versions

    Upgrade workflows to latest Ubuntu and Actions versions

    We upgrade our CI and release GitHub Actions workflows to use the latest Ubuntu runners, which run Ubuntu LTS 22.04 at present, and the latest available versions of all Actions steps.

    As testing the release.yml workflow is not convenient without generating a release, it has instead been run successfully on a separate, private repository cloned from this one.

    Ubuntu Runners

    In commit d6b34fa4355bd4891294eeb69ff05bb43977f385 of PR #4437 we switched our CI and release GitHub Actions workflows to explicitly use the then-latest Ubuntu LTS version 20.04 in order to support ARM64 builds, as these required a version of Go (specifically 1.16) which was not available on the ubuntu-latest Actions runners at the time because they still ran Ubuntu LTS version 18.04.

    However, ubuntu-latest Actions runners now use an even newer LTS version of Ubuntu, 22.04, than the one we have explicitly identified, so we can return to specifying ubuntu-latest for our CI and release builds.

    Actions and Checkouts

    In PR #5236 we upgraded our CI and release workflows to use the latest recommended ruby/setup-ruby@v1 GitHub Actions step to install Ruby and its dependencies.

    We now update our other Actions steps to their respective latest versions as well:

      actions/checkout:           v1 -> v3
      actions/download-artifact:  v1 -> v3
      actions/setup-go:           v2 -> v3
      actions/upload-artifact:    v1 -> v3
      docker/setup-qemu-action:   v1 -> v2
    

    Artifact Download Paths

    As of v2 of the actions/download-artifact Action, downloaded assets are not placed in a new subdirectory created with the name from the step's name argument, but in the current working directory instead. We want to retain the previous behaviour so we add a path argument with the same name as each of the macos-assets and windows-assets download steps.

    Fetching Full Clones

    By default, the actions/checkout Action (as of v2) performs a Git fetch with a --depth=1 option, so a shallow clone is made. As a result, when our Makefile calls git describe HEAD to set its VERSION variable, no tags are available and Git responds with an error message.

    Many of our workflow jobs succeed despite logging that error, including the build-docker and build-docker-cross jobs in both our CI and Release workflows. (The Docker builds create upload artifacts with the valid filenames despite the lack of any tags because they rely on the Git LFS version strings in our debian/changelog file and in our binary; the rpm/build_rpms.bsh script builds a binary just to run git-lfs version and determine the version string from its output.)

    However, our workflow jobs which run the make release command fail outright in the absence of any Git tags, as they search for build artifacts using filenames constructed with the empty VERSION variable, such as git-lfs-windows-amd64-.zip. When no files are found, the tar command fails, halting the job. This affects both the build-default job in our CI workflow (for Linux and macOS), and all of build-main, build-macos, and build-windows jobs in our Release workflow.

    To resolve this in the case of a PR or other push to a branch, we set a fetch-depth value of 0 for our actions/checkout@v3 steps, which downloads the full Git history and all tags. This is somewhat more expensive than a shallow clone, but our project's history is not excessively large.

    Temporary Workaround for Incorrect Tag Fetches

    Due to the GitHub Actions bug documented in actions/checkout#882, though, this resolution is insufficient in the case of a push to a tag. At present, when fetch-depth is 0, the actions/checkout@v3 incorrectly determines the SHA of an annotated tag to be the SHA of its associated commit, and then proceeds as if the tag had been updated on the server since the Action was started, and so rewrites the tag locally to refer to the commit SHA. This has the effect of making the local tag into a lightweight tag, which git describe then ignores (since we don't pass the --tags option to it).

    As a temporary fix for this problem, we add a step after the actions/checkout@v3 step which updates the local tag again to match the remote one. We only run this step when the pushed reference was a tag, because on a branch push it would fail as Git would refuse to update the currently checked-out branch. In our Release workflow, since it only runs on pushes to tags, we can run this step unconditionally. (We could also continue to use the default fetch-depth of 1 for the actions/checkout@v3 step, since we always subsequently fetch the relevant tag, but to be consistent and to avoid future issues once actions/checkout#882 is fixed upstream, we do not do so.)

    /cc actions/checkout#882 /cc #5228

  • TCP Congestion Control disproportionately affects git-lfs

    TCP Congestion Control disproportionately affects git-lfs

    Describe the bug When uploading on an unreliable/congested link (e.g., crowded WiFi) upload speeds are horrendous (30 kbytes per second) but when on a reliable link (e.g., wired, even on the same network) upload speeds are comparable to link speed.

    More specifically, TCP retransmissions or out-of-order ACKs can prevent git-lfs from functioning at any theoretical speed. However, other software, such as curl, web browsers, etc, function just fine and can reach near link speed with even worse packet retransmissions or out-of-order ACKS.

    To Reproduce Steps to reproduce the behavior:

    1. Put wifi network on a crowded channel. You should see TCP retransmissions in Wireshark for web browsing traffic.
    2. Add a large file to git-lfs.
    3. Push file to git-lfs.
    4. See extremely low speeds.

    Expected behavior Git-LFS should be minimally affected by TCP congestion control.

    System environment Linux/WSL/Bare Metal

    Output of git lfs env N/A

    Additional context

    I haven't gone into how the sockets are created, but if I had to guess, the code is not using the Nagle Algorithm (TCP_NODELAY utilized).

  • Clean up RPM builds and fix i686 RPM file names

    Clean up RPM builds and fix i686 RPM file names

    This PR cleans up various issues with our RPM build scripts, in particular removing leftover support for CentOS 5 and 6, and moving responsibility for building 32-bit RPMs into this project so as to fix a naming problem with the 32-bit RPMs for CentOS 8 and Rocky Linux 9.

    Reviewing this PR commit-by-commit is recommended, and each commit should have a full description of its changes.

    We expect to merge this PR prior to when its companion PR git-lfs/build-dockers#54 is merged, specifically because that PR drops its own build logic for 32-bit RPMs in deference to this project building those RPMs at the same time as the 64-bit ones and in the same context.

    This PR's changes have been tested in conjunction with those of PR git-lfs/build-dockers#54 and confirmed to produce valid build artifacts. We used a branch of git-lfs/git-lfs with the changes from this PR plus revisions to the GitHub Actions workflows so as to use the branch from PR git-lfs/build-dockers#54 and then to upload the resultant artifacts to GitHub Actions. These can be examined to confirm that they now correctly name the 32-bit RPM builds for CentOS 8 and Rocky Linux 9 and that they also include only Git LFS packages and no unrelated ones:

    centos/7/RPMS/i686/git-lfs-3.3.0-1.el7.i686.rpm
    centos/7/RPMS/x86_64/git-lfs-3.3.0-1.el7.x86_64.rpm
    centos/7/SRPMS/git-lfs-3.3.0-1.el7.src.rpm
    centos/8/RPMS/i686/git-lfs-3.3.0-1.el8.i686.rpm
    centos/8/RPMS/x86_64/git-lfs-3.3.0-1.el8.x86_64.rpm
    centos/8/SRPMS/git-lfs-3.3.0-1.el8.src.rpm
    debian/10/git-lfs_3.3.0_amd64.deb
    debian/10/git-lfs_3.3.0_i386.deb
    debian/11/git-lfs_3.3.0_amd64.deb
    debian/11/git-lfs_3.3.0_i386.deb
    rocky/9/RPMS/i686/git-lfs-3.3.0-1.el9.i686.rpm
    rocky/9/RPMS/x86_64/git-lfs-3.3.0-1.el9.x86_64.rpm
    rocky/9/SRPMS/git-lfs-3.3.0-1.el9.src.rpm
    

    Building 32-Bit RPMs

    In commit c2d25ee6b6d89425cf39dbc3be7a215d3d0fc050 of PR #511 we added support for building RPM packages for 32-bit platforms by updating the docker/centos_script.bsh script which was present at that time to call rpmbuild with a --target=i686 argument.

    Since commit 56ffe420b787ba2f738287e07d6af9f177e51b4f of PR #555 both that script and the rpm/build_rpms.bsh script contained the same logic to parse the OS name and version in order to set a short suffix for the RPM filenames.

    However, the docker/centos_script.bsh script was subsequently moved into the git-lfs/build-dockers repository, where it has not been updated to match the rpm/build_rpms.bsh script, such as when parsing of the OS major version was added in commit e93940960ee0e1749a2de6616150cd10061b18e2 of PR #5054, which allows us to properly parse the version number on CentOS/Rocky Linux 8 and above, or when parsing of the Rocky Linux OS name was added in commit 723be34bd0e9802c0dc5fa728edf86cc16bc62b0 of PR #5144.

    The result is that at present we build 32-bit RPMs for CentOS 8 and Rocky Linux 9 (el8 and el9, respectively) without the platform short name suffix in their filenames, e.g., git-lfs-3.3.0-1.i686.rpm and git-lfs-3.3.0-1.i686.rpm (instead of git-lfs-3.3.0-1.el9.i686.rpm and git-lfs-3.3.0-1.el8.i686.rpm), and then upload them to Packagecloud with those names.

    To resolve this problem and avoid later regressions between the two sets of parsing logic, we add an rpmbuild command with the --target=i686 argument for 32-bit packages to the rpm/build_rpms.bsh script of this repository, which ensures they will be built with the same context as our 64-bit packages.

    We therefore also remove the rpmbuild command with the --target=i686 argument from the centos_script.bsh script in the git-lfs/build-dockers repository, in commit git-lfs/build-dockers@0b991e0cdaf9a3fdc87074355a21d22e0fa06149 of PR git-lfs/build-dockers#54.

    Removing Substitute Repository Cleanup Script

    Since the original introduction of support for building CentOS RPMs in commit 4a7162777913489e7c70a3da3235dcd3eeec553b of PR #332, an rpm/clean.bsh script has been provided to perform the equivalent of a git clean -xdf command on the Git repository from which we build Git LFS.

    However, this script is no longer needed as all of the CentOS and Rocky Linux container image builds defined by the git-lfs/build-dockers repository's Dockerfiles have a native Git package installed (or a recent Git built, in the case of CentOS 7), so we remove the only use of it, in commit git-lfs/build-dockers@7be51a4ddd0899dd11babc0102414a0442043b3d of PR git-lfs/build-dockers#54, and can therefore also drop the script from this repository as well.

    Removing CentOS 5 and 6 Build Steps

    We no longer support either CentOS 5 or 6, per commit b560b853a6e557195c0c68875218f45c89dce56b of PR #1298 and commit git-lfs/build-dockers@64a3a9fc4df3ca28920a4051435d55487832c5b7 of PR git-lfs/build-dockers#3, and commit git-lfs/build-dockers@898d9b045e24a078fc2ae243cc870a465dfd7364 of PR git-lfs/build-dockers#33.

    We also now build the asciidoctor Ruby gem in order to generate our man pages, rather than using ronn and several other gems, per commit db9a82132a2bb066876d8ddf06c5255da2f199a4 of PR #5054.

    As a result, we can eliminate a number of build steps which only operated on systems that lacked native package manager support for Go, Git 2.x, Ruby 3.x, etc., as well as steps pertaining to the Ruby gem dependencies required for the ronn gem.

  • Unset `GIT_TRACE` environment variable for Git commands in `Makefile`

    Unset `GIT_TRACE` environment variable for Git commands in `Makefile`

    In commit c4874df619276c807772bae6e62d9cb9718a7077 of PR #971 the script/cibuild script was revised to run the Go test suite just for our git Go package in order to confirm that we do not leak the GIT_TRACE environment variable to Git itself when it is set.

    Later, in commit 2235198650d8dfadded525cc308ae8dfa786da52 of PR #3144, our Makefile was updated to start by running some Git commands (git describe and git rev-parse) to determine the current tagged version of the project.

    When GIT_TRACE=1 is set in second run of the Go test suite, these initial Git command output Git trace log lines, which does not affect the validity of the second Go test of our git package but does add some noise to the output of the script/cibuild script, for example:

    09:53:27.393455 git.c:460               trace: built-in: git describe HEAD
    09:53:27.395241 git.c:460               trace: built-in: git describe HEAD
    09:53:27.396926 git.c:460               trace: built-in: git describe HEAD
    09:53:27.398577 git.c:460               trace: built-in: git describe HEAD
    09:53:27.400339 git.c:460               trace: built-in: git describe HEAD
    09:53:27.401985 git.c:460               trace: built-in: git describe HEAD
    09:53:27.403613 git.c:460               trace: built-in: git describe HEAD
    09:53:27.405334 git.c:460               trace: built-in: git describe HEAD
    09:53:27.406941 git.c:460               trace: built-in: git describe HEAD
    09:53:27.408580 git.c:460               trace: built-in: git describe HEAD
    09:53:27.410234 git.c:460               trace: built-in: git describe HEAD
    09:53:27.411866 git.c:460               trace: built-in: git describe HEAD
    09:53:27.413598 git.c:460               trace: built-in: git describe HEAD
    09:53:27.415249 git.c:460               trace: built-in: git describe HEAD
    09:53:27.416936 git.c:460               trace: built-in: git describe HEAD
    09:53:27.418578 git.c:460               trace: built-in: git describe HEAD
    09:53:27.420260 git.c:460               trace: built-in: git describe HEAD
    09:53:27.421917 git.c:460               trace: built-in: git describe HEAD
    09:53:27.423583 git.c:460               trace: built-in: git describe HEAD
    09:53:27.425375 git.c:460               trace: built-in: git describe HEAD
    09:53:27.427019 git.c:460               trace: built-in: git describe HEAD
    09:53:27.428662 git.c:460               trace: built-in: git describe HEAD
    09:53:27.430298 git.c:460               trace: built-in: git describe HEAD
    09:53:28.605350 git.c:460               trace: built-in: git rev-parse --short HEAD
    GOOS= GOARCH= go build -ldflags="-X github.com/git-lfs/git-lfs/v3/config.GitCommit=328eb98 -s -w " -gcflags=" " -trimpath -o ./bin/git-lfs ./git-lfs.go
    ...
    ok  	github.com/git-lfs/git-lfs/v3/git	0.546s
    

    We therefore just unset the GIT_TRACE environment variable for these two Git commands in our Makefile.

  • Add `git lfs migrate export` command examples to manual page

    Add `git lfs migrate export` command examples to manual page

    We add several examples of the use of the git lfs migrate export subcommand to the git-lfs-migrate(1) manual page, showing use of both the --include option and the --remote option.

    /cc @axkibe /cc https://github.com/git-lfs/git-lfs/issues/5237#issuecomment-1363811700

  • git rev-parse exit status 128

    git rev-parse exit status 128

    I try to install the git lfs with $ git lfs install

    but i get this error:

    Error: Failed to call git rev-parse --git-dir: exit status 128

    Version: git-lfs/2.9.2 (GitHub; linux arm64; go 1.13.5)

Go bindings to systemd socket activation, journal, D-Bus, and unit files

go-systemd Go bindings to systemd. The project has several packages: activation - for writing and using socket activation from Go daemon - for notifyi

Dec 30, 2022
A quick and easy password protected web server for your files. httpfolder makes downloading/uploading files from your current working directory easy, even for fairly large files.

httpfolder A quick and easy password protected web server for your files. httpfolder makes downloading/uploading files from your current working direc

Sep 12, 2022
lls is lightweight ls. Using lls, you can get a list of files in a directory that contains a large number of files.

lls lls is lightweight ls. Using lls, you can get a list of files in a directory that contains a large number of files. How? You allocate a buffer for

Dec 29, 2022
Semantic Versioning (semver) library written in golang

semver for golang semver is a Semantic Versioning library written in golang. It fully covers spec version 2.0.0. Versioning Old v1-v3 versions exist i

Dec 26, 2022
Automatically create global & local Rate Limit in Istio, support EnvoyFilter versioning!

istio-ratelimit-operator Istio ratelimit operator provide an easy way to configure Global or Local Ratelimit in Istio mesh. Istio ratelimit operator a

Oct 24, 2022
Dud is a lightweight tool for versioning data alongside source code and building data pipelines.

Dud Website | Install | Getting Started | Source Code Dud is a lightweight tool for versioning data alongside source code and building data pipelines.

Jan 1, 2023
A CLI which automates semver versioning.

Semverbot A CLI which automates semver versioning based on git information. Why Semverbot? There are several reasons why you should consider using sbo

Dec 12, 2022
A boilerplate for Go fiber versioning
A boilerplate for Go fiber versioning

Fiber Versioning Boilerplate Prerequisite Make sure you have the following installed outside the current project directory and available in your GOPAT

Nov 18, 2022
Learning go with tests! Also setting up automated versioning via SemVer.

learn-go-with-tests The purpose of this repo is to complete the learn-go-with-test course located here. I am also interested in learning how to automa

Nov 23, 2021
modver - a Go package and command that helps you obey semantic versioning rules in your Go module.

Modver This is modver, a Go package and command that helps you obey semantic versioning rules in your Go module. It can read and compare two different

Dec 12, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Dec 29, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

kubectl-slice: split Kubernetes YAMLs into files kubectl-slice is a neat tool that allows you to split a single multi-YAML Kubernetes manifest into mu

Jan 3, 2023
Source code editor written in Go using go-gtk bindings. It aims to handle navigation effectively among large number of files.
Source code editor written in Go using go-gtk bindings. It aims to handle navigation effectively among large number of files.

tabby Source code editor written in Go using go-gtk bindings. It aims to handle navigation effectively among large number of files. screenshot: depend

Nov 16, 2022
Compute message digest for large files in Go

checksum test coverage Compute message digest, like MD5 and SHA256, in golang for potentially large files. Usage package main import ( "fmt" "githu

Dec 28, 2022
Got: Simple golang package and CLI tool to download large files faster 🏃 than cURL and Wget!

Got. Simple and fast concurrent downloader. Installation ❘ CLI Usage ❘ Module Usage ❘ License Comparison Comparison in cloud server: [root@centos-nyc-

Dec 29, 2022
Download an upload large files to Google Drive (API v3)

gdriver gdriver is a command-line tool, written in Go, used for uploading and downloading large personal files from Google Drive (API v3). The tool pr

Nov 30, 2022
a sharded store to hold large IPLD graphs efficiently, packaged as location-transparent attachable CAR files, with mechanical sympathy

DAG store This README will be populated soon. In the meantime, please refer to the design document.

Oct 31, 2022
Simple program that uploads large files to Amazon S3 over slow connections.

shrimp is a small program that can reliably upload large files to Amazon S3. My personal use case is to upload large files to S3 over a slow residenti

Nov 30, 2022
Command-line tool to organize large directories of media files recursively by date, detecting duplicates.

go-media-organizer Command-line tool written in Go to organise all media files in a directory recursively by date, detecting duplicates.

Jan 6, 2022
A version control system to manage large files.

ArtiVC ArtiVC (Artifacts Version Control) is a handy command-line tool for data versioning on cloud storage. With only one command, it helps you neatl

Jan 4, 2023