Version solving fails on custom git component in linux (IDFGH-12200) (PACMAN-815)

[SlowTypist]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.3-dev-2087-g5f42493573

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

What is the expected behavior?

Compiling with no errors

What is the actual behavior?

I have a custom component here https://github.com/SlowTypist/mongoose-esp-idf

When trying to build project in Windows everything is fine, in linux though I get error.

I have a feeling the problem is the submodules of the component are not updated recursively.

Steps to reproduce.

Create new project, in idf_component.yml add this
test_component:
git: git@github.com:SlowTypist/mongoose-esp-idf.git

run build.

Build or installation Logs.

CMake Error at /home/fram/esp/esp-idf/tools/cmake/build.cmake:544 (message):
  WARNING: hint: Using 'master' as the name for the initial branch.  This
  default branch name

  hint: is subject to change.  To configure the initial branch name to use in
  all

  hint: of your new repositories, which will suppress this warning, call:

  hint:

  hint: 	git config --global init.defaultBranch <name>

  hint:

  hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and

  hint: 'development'.  The just-created branch can be renamed via this
  command:

  hint:

  hint: 	git branch -m <name>



  WARNING: From github.com:SlowTypist/mongoose-esp-idf

   * [new branch]      main       -> main

  

  WARNING: Submodule 'mongoose' (https://github.com/cesanta/mongoose)
  registered for path 'mongoose'

  Cloning into '/tmp/tmpnfml63mr/mongoose'...



  WARNING: [Errno 2] No such file or directory:
  '/tmp/tmpnfml63mr/mongoose/examples/nxp/rt1020-evk-make-baremetal-builtin/link2.ld'


  ERROR: Because project depends on test_component
  (5d09dfdfa3e0520bf4ed39c4ccb9c588562b8aca) which doesn't match any
  versions, version solving failed.

Call Stack (most recent call first):
  /home/fram/esp/esp-idf/tools/cmake/project.cmake:710 (idf_build_process)
  CMakeLists.txt:6 (project)

More Information.

No response

[kumekay]

@SlowTypist I've tried to reproduce the problem on macOS, but I faced another one (related to symlinks). So we will investigate both soon and keep this issue updated.

Build log:

Dependencies lock doesn't exist, solving dependencies.
CMake Error at /Users/ku/e/esp-idf/tools/cmake/build.cmake:544 (message):
  Traceback (most recent call last):

    File "<frozen runpy>", line 198, in _run_module_as_main
    File "<frozen runpy>", line 88, in _run_code
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/prepare_components/__main__.py", line 6, in <module>
      main()
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/prepare_components/prepare.py", line 132, in main
      args.func(args)
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/prepare_components/prepare.py", line 31, in prepare_dep_dirs
      ).prepare_dep_dirs(
        ^^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/core.py", line 111, in wrapper
      return func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/core.py", line 746, in prepare_dep_dirs
      downloaded_components = download_project_dependencies(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/dependencies.py", line 271, in download_project_dependencies
      solution = solver.solve()
                 ^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/version_solver/version_solver.py", line 91, in solve
      self.solve_manifest(manifest)
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/version_solver/version_solver.py", line 141, in solve_manifest
      self.solve_component(dep, manifest_path=manifest.path)
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/version_solver/version_solver.py", line 161, in solve_component
      cmp_with_versions, source = self.get_versions_from_sources(requirement)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_manager/version_solver/version_solver.py", line 118, in get_versions_from_sources
      cmp_with_versions = source.versions(
                          ^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_tools/sources/git.py", line 197, in versions
      component_hash = hash_dir(source_path, include=include, exclude=exclude)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_tools/hash_tools/calculate.py", line 61, in hash_dir
      sha.update(hash_file(file_path).encode('utf-8'))
                 ^^^^^^^^^^^^^^^^^^^^
    File "/Users/ku/.espressif/python_env/idf5.3_py3.11_env/lib/python3.11/site-packages/idf_component_tools/hash_tools/calculate.py", line 30, in hash_file
      with open(Path(file_path).as_posix(), 'rb') as f:
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  FileNotFoundError: [Errno 2] No such file or directory:
  '/var/folders/92/_hg8yrbs3jq_rcwnbvwqhl7c0000gn/T/tmp0rrrbtqj/mongoose/examples/nxp/rt1020-evk-make-baremetal-builtin/link2.ld'


Call Stack (most recent call first):
  /Users/ku/e/esp-idf/tools/cmake/project.cmake:710 (idf_build_process)
  CMakeLists.txt:8 (project)


.-- Configuring incomplete, errors occurred!

FAIL

[XDanielPaul]

@SlowTypist I tried to reproduce the problem as well on WSL Ubuntu and got a similar result as mentioned above. I investigated the issue and after cloning the specified repository and updating submodules, I found out that the symlink on the path mongoose/examples/nxp/rt1020-evk-make-baremetal-builtin/link2.ld seems to be broken.

Could you check, whether this is the case in your environment as well?

[kumekay]

@XDanielPaul Unfortunately we cannot simply use symlink itself for calculation of hashsum instead of real file because it will break some existing components in the registry. However, it should be safe to just ignore missing files while calculating the hash.

[XDanielPaul]

I agree, we cannot use the symlink itself for the calculation, but as far as I know Component Manager takes care of symbolic links in git repositories with git submodule ... command, and I think we should not support broken ones. Due to this, I would close the issue.

[kumekay]

Fixed in a4f7a84 now an error message is printed if the path is broken: