rust-src does not include workspace Cargo.toml

[jonhoo]

The rust-src component that gets installed by Rustup to ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust includes a Cargo.lock that spans the standard library crates, but does not include a corresponding Cargo.toml (essentially this Cargo.toml).

Cargo's -Zbuild-std feature makes this work anyway by constructing a virtual workspace for the standard library, but this doesn't work for editors that want to provide users with completion and analysis all the way into std/alloc/core. They end up running cargo metadata directly in the various standard library crates (library/std for example) (IntelliJ, rust-analyzer). However, since there's no workspace-level Cargo.toml next to the Cargo.lock, they do not pick up on the locked versions at all, and instead use whatever the latest versions of the standard library's dependencies are. This in turn means that go-to-definition and similar mechanisms do not go to the actual versions used by the standard library the user is actually using.

In chatting to @Mark-Simulacrum about this, I think the editors are arguably doing the right thing here by running cargo metadata inside the individual crates (like std), and Cargo is doing the right thing not looking at the Cargo.lock since there isn't a workspace there. And it feels like the right thing to do is to synthesize a workspace Cargo.toml for inclusion with rust-src. But if we explicitly won't (or can't) include a Cargo.toml in rust-src's root, we have a couple of options:

• We could maybe special-case Cargo to recognize when it's run on a crate from within the standard library.
• cargo metadata could have something like --with-lockfile Cargo.lock so that you can run it in arbitrary places on disk while pointing to (for example) a checked out repositories Cargo.lock.
• We could change Cargo to always include std + deps in everyone's Cargo.lock as some kind of special dep kind.
• We could create cargo metadata --include-std.

All of these feel brittle to one extent or another though, and it's not clear that they are sufficient for what editors may need (cc @matklad), so I think we should push for including Cargo.toml if we can.

Meta

rustc --version --verbose:

rustc 1.59.0 (9d1b2106e 2022-02-23)
binary: rustc
commit-hash: 9d1b2106e23b1abd32fce1f17267604a5102f57a
commit-date: 2022-02-23
host: x86_64-unknown-linux-gnu
release: 1.59.0
LLVM version: 13.0.0

[jonhoo]

A simple way to reproduce this is:

$ cd ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/
$ cargo metadata --locked
warning: please specify `--format-version` flag explicitly to avoid compatibility problems
    Updating crates.io index
error: the lock file ~/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/std/Cargo.lock needs to be updated but --locked was passed to prevent this
If you want to try to generate the lock file without accessing the network, remove the --locked flag and use --offline instead.

[jonhoo]

This also seems like a regression from #44076

[matklad]

The absolute best for the editors would be if /library was a separate workspace with dedicated Cargo.toml, Cargo.lock, and vendored sourced of crates.io deps (such that cargo-metadata works without net).

Practically, this would allow us to not worry about "large" compiler codebase and only deal with the small set of library crates. Theoretically, it still feels to me that "compiler" and "library" actually exist in separate dimensions, and using a single Cargo.lock for them seems accidental.

Obviously, that's "perfect is the enemy of the good" kind of thing, just restoring Cargo.toml/.lock would be helpful

[jonhoo]

Another thing worth adding to this is that cargo metadata only accidentally works in rustlib/src/rust/library/* because empty versions of crates like rustc-std-workspace-alloc exist on crates.io. Without the workspace Cargo.toml to pick up the local path, that means the editors end up going to fetch the crates.io versions of these even though the right versions are sitting in rustlib/src/rust/library.

[RalfJung]

FWIW, the lock file is included because I needed it to make Miri more reliable, and then just added it. I think a PR to simply add Cargo.toml to the list of shipped files would probably go through in a similar vein.

However, it is not clear to me whether you are saying the original Cargo.toml should be shipped, or whether it should be somehow modified before shipping. The latter will be more tricky, of course -- is it even clear what the "right" way to adjust the toml file is?

[RalfJung]

Oh, you actually linked to my old PR that added the Cargo.lock file. 😂 Looks like it originally intended to also add Cargo.toml but that part got bailed out before landing.

This is where I had to remove the Cargo.toml because it caused trouble with distcheck that I didn't have the patience to debug.

Another thing worth adding to this is that cargo metadata only accidentally works in rustlib/src/rust/library/* because empty versions of crates like rustc-std-workspace-alloc exist on crates.io.

FWIW, those crates existing on crates.io is quite deliberate and a key part to their function.

[eddyb]

Cross-posting (potentially relevant) #78790 (comment) (note that that PR was reverted):

The implementation is a bit hacky, hiding the workspace and copying in the lockfile. So perhaps some other things need to be pulled in in a similar manner. That said, I would hope that --offline Just Works here.

Not sure where to post this but AFAICT this is fixable by generating a workspace Cargo.toml for the rust-src dist/vendor step, with a [patch.crates-io] section that contains:

rust/Cargo.toml

Lines 129 to 133 in 7941b3f

	# See comments in `library/rustc-std-workspace-core/README.md` for what's going on
	# here
	rustc-std-workspace-core = { path = 'library/rustc-std-workspace-core' }
	rustc-std-workspace-alloc = { path = 'library/rustc-std-workspace-alloc' }
	rustc-std-workspace-std = { path = 'library/rustc-std-workspace-std' }

This could potentially even allow Cargo to stop hardcoding them - if we did ship that Cargo.toml, at least.

[bjorn3]

Not sure where to post this but AFAICT this is fixable by generating a workspace Cargo.toml for the rust-src dist/vendor step, with a [patch.crates-io] section that contains:

I think we should just split the sysroot into a new workspace and use it as part of bootstrapping instead of generating a new workspace in the dist step. That may also simplify the tidy license check by not requiring to check if a crate is a dependency of the sysroot to apply stricter license checks. Instead it can apply them to the entire contents of library/Cargo.lock.

[jonhoo]

One more observation: this problem is exacerbated by the fact that when an editor runs cargo metadata in, say, <rust-src>/library/std, it will end up generating a Cargo.lock. The same will happen when it's run in alloc, core, etc. This means you end up with one Cargo.lock in every sub-crate, plus one at the root, all of which can contain different versions from one another since all the sub-crate ones contain whatever was the latest version at the time cargo metadata was run.

[lf-]

This winds up extra fun under Nix using $RUST_SRC_PATH where the rust sources are literally on a readonly file system, which is the r-a issue here: rust-lang/rust-analyzer#13393

The correct fix for this issue is to add the lockfile and manifest to rust-src, I think, so cargo doesn't try to relock it.

[RalfJung]

IMO it's a bug in cargo metadata that it doesn't work on a read-only file system (rust-lang/cargo#10096).

[Veykril]

I believe #128534 essentially fixes this?

[bjorn3]

It does. I would have linked this issue from that PR if I had remembered this issue existed.