Globs in extra paths

[judej]

Discussed in #2712

^{Originally posted by itsdani February 21, 2021}
I'm trying to setup a python monorepo build with bazel, and I'm struggling to make pylance recognize the library imports. I have managed to make it work by adding e.g. <bazel-myproject>/external/<my-pip-dependencies>/pypi__numpy to python.analysis.extraPaths, but adding a line for each dependency on every computer we use doesn't seem to be right.

My idea/request is that if we could add something like dependencies/*/ or dependencies/pypi__*/ to the extra paths, it would solve this problem entirely. The expected behavior would be to add every (matching) directory under dependencies/ to the extra paths.

[luabud]

we just converted this back to an issue to add it to our roadmap

[dpar39]

Any ETA on this?

[erictraut]

As has been explained in other threads, I don't think it will work to add globs for extraPaths. The problem is that extraPaths is order dependent. The order affects import resolution behavior. Globs is an order-independent mechanism. If you want to add extra paths to your project, you should specify them one at a time in the order you intend for imports to be resolved.

[rchiodo]

We could make it deterministic by saying globs resolve alphabetically.

[heejaechang]

@itsdani @dpar39 it has been a while but can you guys provide us more detail on the scenario? and we have add extra path code action to help these kinds of situation and wonder why that didn't work?

[itsdani]

We have given up on using Bazel and I work on very different things since I requested it 3 years ago, I'm not even using Pylance anymore. Maybe someone currently experiencing this issue could provide more detail. I can only repeat what I already said in the other thread: #2712 (comment)

[adzenith]

Bazel puts each Python dependency in its own directory for sandboxing reasons. Our extraPaths is 601 lines long, and we have to manually update it when our Python dependency tree changes.

Here's a small snippet so you can see what it looks like:

  "external/rules_python~~pip~pip_310_google_re2/site-packages",
  "external/rules_python~~pip~pip_310_google_resumable_media/site-packages",
  "external/rules_python~~pip~pip_310_googleapis_common_protos/site-packages",
  "external/rules_python~~pip~pip_310_graphviz/site-packages",
  "external/rules_python~~pip~pip_310_greenlet/site-packages",
  "external/rules_python~~pip~pip_310_grpcio/site-packages",
  "external/rules_python~~pip~pip_310_grpcio_status/site-packages",
  "external/rules_python~~pip~pip_310_grpcio_tools/site-packages",

It's that for 600 packages.

Instead if we could do something like "external/rules_python~~pip~pip_310_*/site-packages",, we would be all set and wouldn't have to keep the list in sync with Bazel.

Thanks!

[heejaechang]

@adzenith a few questions.

1. is that list stable? will everyone who install dependency gets the same list? including re-installing the dependencies?
2. how it works in runtime? will all these paths be included in PYTHONPATH? or Bazel provide some special import logic for this?
3. is the pattern on how dependency installed customizable? in another word, will everyone (or most of users of Bazel) have dependency installed on external/rules_python.... ?

I am not sure we want to support glob on extraPath, but we might add support for Bazel env directly if it has some common patterns, we can add support for.

[adzenith]

1. Yes, everyone has this same list. We check in the pyproject.toml file.
2. Yes, Bazel builds a PYTHONPATH for each binary based on its declared dependencies. So you will have some subset of these available at runtime in Bazel depending on what you're running, and each available library will appear in PYTHONPATH.
3. The pattern is not customizable. Bazel puts the dependencies where it wants. In our case we have a symlink, external, to the Bazel build output tree where the Python libs are stored.

I don't need a glob specifically; I just assumed that would be the easiest to implement. If Bazel's output tree if directly supported that would also suit my needs just fine.
Thanks!

[JeroenSchmidt]

Hi all. I just want to share this discussion from 2022 that also touched on this topic of glob support for bazel / repos with multiple subpackages, see #2923 (comment)