Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

uv doesn't correctly checkout Git dependencies with Git LFS assets #3312

Open
sydduckworth opened this issue Apr 29, 2024 · 13 comments · May be fixed by #10335
Open

uv doesn't correctly checkout Git dependencies with Git LFS assets #3312

sydduckworth opened this issue Apr 29, 2024 · 13 comments · May be fixed by #10335
Labels
compatibility Compatibility with a specification or another tool help wanted Contribution especially encouraged needs-mre Needs more information for reproduction

Comments

@sydduckworth
Copy link

I'm working on a project that depends on a private Git repository where some of the project's assets are stored in Git LFS (in this case Pytorch models).

My project uses Rye as its package manager. When configured to use Pip, everything works as expected. When configured to use uv, the LFS objects aren't checked out correctly.

I assume a blocker for uv supporting LFS is the fact that the git2 crate also doesn't support LFS (rust-lang/git2-rs#956) and there's no indication support will be added in the future.

Platform: Ubuntu 22.04
uv version: 0.1.39

@konstin konstin added the compatibility Compatibility with a specification or another tool label Apr 29, 2024
@vfilter
Copy link

vfilter commented Jun 13, 2024

Same issue here. Makes uv unfortunately unusable for us since we have lfs in a private github repo.

@zanieb
Copy link
Member

zanieb commented Jun 13, 2024

We actually stopped using the git2 crate so we may be able to support this easier now? cc @ibraheemdev

@gusutabopb
Copy link

Still running into this issue using uv 0.3.0.

uv fails due to git lfs failing due to "smudge errors".

Sample command:

uv pip install git+https://[email protected]/repo.git

pip works fine, so my current workaround is:

uv pip install pip
uv run pip install git+https://[email protected]/repo.git

Cloning the repo locally and then installing from a local path works fine too.

@vfilter
Copy link

vfilter commented Sep 1, 2024

Adding another workaround here. Running git lfs install --force --skip-smudge before installing the dependency from your private repo did solve it for us. I was able to install with uv add after that.

@zanieb zanieb added the help wanted Contribution especially encouraged label Sep 3, 2024
@falckt
Copy link

falckt commented Sep 18, 2024

Just for reference, I tried setting the respective settings @vfilter suggested in #3312 (comment) as environment variables

$ env | grep -e ^GIT_CONFIG
GIT_CONFIG_KEY_0=filter.lfs.smudge
GIT_CONFIG_VALUE_0=git-lfs smudge --skip -- %f
GIT_CONFIG_COUNT=1

which is also picked up by git

$ git config --get filter.lfs.smudge         
git-lfs smudge --skip -- %f

However when running uv sync --upgrade-package <my-package-in-git> I still get the smudge filter error.

@zanieb could it be that uv does not pass on the environment variables to the underlying git process? (Setting them via the config file works).

@falckt
Copy link

falckt commented Sep 18, 2024

No clue how, but setting the environment variable GIT_LFS_SKIP_SMUDGE=1 does have the desired effect.

This old comment suggests that certain local checkouts might trigger these problems git-lfs/git-lfs#2518 (comment). Not sure whether that is the case here though.

@xela-95
Copy link

xela-95 commented Oct 2, 2024

Hi! This issue is blocking me from adopting pixi, is there any news on this?

@alan-cooney-dsit
Copy link

No clue how, but setting the environment variable GIT_LFS_SKIP_SMUDGE=1 does have the desired effect.

This just turns off getting LFS assets btw.

Also very keen for this feature

@zanieb
Copy link
Member

zanieb commented Oct 6, 2024

Someone want to share a minimal reproduction for this?

@zanieb zanieb added the needs-mre Needs more information for reproduction label Oct 6, 2024
@grebnetiew
Copy link

grebnetiew commented Oct 6, 2024

Sure.

$ uv pip install git+https://github.com/grebnetiew/lfs-py.git
Updating https://github.com/grebnetiew/lfs-py.git (HEAD)
error: Git operation failed
  Caused by: process didn't exit successfully: `git reset --hard 1abf996d38d769939d2a4f5e06d1afc8e8a91817` (exit status: 128)
--- stderr
Downloading datafile.bin (39 B)
Error downloading object: datafile.bin (4502c72): Smudge error: Error downloading datafile.bin (4502c722f9bcf5f68951d9b41b74c2de404de55cea347c92b0a7d608cd3d9c88): error transferring "4502c722f9bcf5f68951d9b41b74c2de404de55cea347c92b0a7d608cd3d9c88": [0] remote missing object 4502c722f9bcf5f68951d9b41b74c2de404de55cea347c92b0a7d608cd3d9c88

Errors logged to '/home/erieke/.cache/uv/git-v0/checkouts/634788a817b48cd6/1abf996/.git/lfs/logs/20241006T194309.74056189.log'.
Use `git lfs logs last` to view the log.
error: external filter 'git-lfs filter-process' failed
fatal: datafile.bin: smudge filter lfs failed

The workaround above works, though it might not actually get the lfs-file I checked in (I didn't check). Packages that use LFS likely need their LFS files ;)

$ GIT_LFS_SKIP_SMUDGE=1 uv pip install git+https://github.com/grebnetiew/lfs-py.git
 Updated https://github.com/grebnetiew/lfs-py.git (1abf996)
Resolved 1 package in 2.63s
   Built lfs-py @ git+https://github.com/grebnetiew/lfs-py.git@1abf996d38d769939d2a4f5e06d1afc8e8a91817
Prepared 1 package in 643ms
Installed 1 package in 0.86ms
 + lfs-py==0.1.0 (from git+https://github.com/grebnetiew/lfs-py.git@1abf996d38d769939d2a4f5e06d1afc8e8a91817)

@sydduckworth
Copy link
Author

Another workaround option if you need LFS for other repos and don't want to manually set environment variables:

Create a new file ~/.gitconfig-nolfs.inc:

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge --skip -- %f
    process = git-lfs filter-process --skip
    required = true

Create or modify ~/.gitconfig:

[includeIf "gitdir:~/.cache/uv/**"]
    path = ~/.gitconfig-nolfs.inc

This updates your Git config to skip LFS just for uv dependencies.

@SZRabinowitz
Copy link

SZRabinowitz commented Dec 17, 2024

Another workaround option if you need LFS for other repos and don't want to manually set environment variables:

Create a new file ~/.gitconfig-nolfs.inc:

[filter "lfs"]
    clean = git-lfs clean -- %f
    smudge = git-lfs smudge --skip -- %f
    process = git-lfs filter-process --skip
    required = true

Create or modify ~/.gitconfig:

[includeIf "gitdir:~/.cache/uv/**"]
    path = ~/.gitconfig-nolfs.inc

This updates your Git config to skip LFS just for uv dependencies.

Thanks!
For Windows users, it's similar:

[includeIf "gitdir/i:~/AppData/Local/uv/"]
    path = ~/.gitconfig-uv.inc

For anyone else, if these don't work: Get you uv path using command uv cache dir and remove the /cache from the end

Edit: No need to remove /cache from the end. Just make sure the path ends with /

@sydduckworth
Copy link
Author

I've investigated this a bit more and I think I now understand the actual source of the error.

When getting a Git revision uv will first fetch the rev to a local repo, then do a local clone to a new directory, then do a hard reset of the cloned repo to the specific revision.
The last step is what's triggering the LFS error. When the files are being checked out, the LFS smudge filter will try to resolve LFS pointers to actual files, but fails because LFS doesn't know the URL of the upstream remote.

I think it's possible to fix this by just adding a step where after the initial fetch but before the local clone uv would try to run git lfs fetch <remote> <rev>. This resolves the smudge error because it ensures all of the LFS pointers are present in the Git database when the hard reset happens.

The biggest issue I see with that approach is that since LFS has to check every file in a revision when fetching, it's possible that for large repositories the overhead could become non-trivial.
On the other hand since the command would only run if the user has Git LFS installed, it would really only affect the specific subset of users who:

  1. have Git LFS installed
  2. are cloning a repository that doesn't use LFS
  3. are cloning a very large repository

@sydduckworth sydduckworth linked a pull request Jan 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compatibility Compatibility with a specification or another tool help wanted Contribution especially encouraged needs-mre Needs more information for reproduction
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants