idf.py fullclean shows ERROR in remove_managed_components (.gitignore support for hash calculations) (IDFGH-9024) (PACMAN-544)

[AxelLin]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v5.1-dev-2509-gcfef24863f

Operating System used.

Linux

How did you build your project?

Command line with idf.py

If you are using Windows, please specify command line type.

None

Development Kit.

ESP32C3DevKit-M1

Power Supply used.

USB

What is the expected behavior?

"idf.py fullclean" should work without any error.

What is the actual behavior?

After upgrade to v5.1-dev-2509-gcfef24863f, now everytime executing "idf.py fullclean" shows below error:
I didn't change anythin in managed_components folder, and this error does not happen before upgrading esp-idf.

$ idf.py fullclean
Executing action: fullclean
Executing action: remove_managed_components
ERROR:
Some components (joltwallet__littlefs) in the "managed_components" directory were modified on the disk since the last run of the CMake. Content of this directory is managed automatically.

    If you want to keep the changes, you can move the directory with the component to the "components"         directory of your project.

    I.E. for "joltwallet__littlefs" run:
    mv /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs /home/axel/esp/esp-idf-dev/apps/myproj/components/joltwallet__littlefs

    Or, if you want to discard the changes remove the ".component_hash" file from the component's directory.

    I.E. for "joltwallet__littlefs" run:
    rm /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs/.component_hash

Steps to reproduce.

idf.py build ; idf.py fullclean

Debug Logs.

No response

More Information.

No response

[kumekay]

Hi @AxelLin could you please provide more details about your environment to help us reproduce this problem:

• Linux Distro with version
• File system type
• Content of dependencies.lock

Do you experience the problem only with this component or with other too?

[AxelLin]

$ lsb_release -a
LSB Version:	core-11.1.0ubuntu4-noarch:printing-11.1.0ubuntu4-noarch:security-11.1.0ubuntu4-noarch
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.1 LTS
Release:	22.04
Codename:	jammy

filesystem: ext4

$ cat dependencies.lock 
dependencies:
  idf:
    component_hash: null
    source:
      type: idf
    version: 5.1.0
  joltwallet/littlefs:
    component_hash: 2fa0ef8823451b00cec4eec1a18f810e9267b1f588513a64526be456c8a9d59a
    source:
      service_url: https://api.components.espressif.com/
      type: service
    version: 1.5.1
manifest_hash: c14ee59777e03272dc787e6a72f0e5ccf67cf0c3820f182730fbe2e28d652138
target: esp32c3
version: 1.0.0

[AxelLin]

Do you experience the problem only with this component or with other too?

I only use littlefs, so I'm not sure if other components have issue or not.

$ cat main/idf_component.yml
## IDF Component Manager Manifest File
dependencies:
  joltwallet/littlefs: "^1.4.0"
  ## Required IDF version
  idf:
    version: ">=4.1.0"
  # # Put list of dependencies here
  # # For components maintained by Espressif:
  # component: "~1.0.0"
  # # For 3rd party components:
  # username/component: ">=1.0.0,<2.0.0"
  # username2/component2:
  #   version: "~1.0.0"
  #   # For transient dependencies `public` flag can be set.
  #   # `public` flag doesn't have an effect dependencies of the `main` component.
  #   # All dependencies of `main` are public by default.
  #   public: true

[kumekay]

@AxelLin thank you so much,

Could you please also share the content of managed_components/joltwallet__littlefs/.component_hash file?

[AxelLin]

$ cat managed_components/joltwallet__littlefs/.component_hash
2fa0ef8823451b00cec4eec1a18f810e9267b1f588513a64526be456c8a9d59a

BTW, everytime I execute "idf.py menuconfig" it will show message to ask
deleting managed_components/joltwallet__littlefs/.component_hash.

I delete it, then execute "idf.py menuconfig". It will generate exactly the same content again:
$ cat managed_components/joltwallet__littlefs/.component_hash
2fa0ef8823451b00cec4eec1a18f810e9267b1f588513a64526be456c8a9d59a

[AxelLin]

It was working with v5.1-dev-2356-gedd815af2e, and fails after upgrade to v5.1-dev-2495-ge5926d1b1b.
BTW, ./install.sh updates below packages, I'm not sure if it's related or not. just FYI.

Downloading https://dl.espressif.com/pypi/idf-component-manager/idf_component_manager-1.2.1-py3-none-any.whl (116 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 116.1/116.1 kB 2.7 MB/s eta 0:00:00
Downloading https://dl.espressif.com/pypi/esptool/esptool-4.5.dev0-py3-none-any.whl (310 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 310.2/310.2 kB 4.8 MB/s eta 0:00:00
Downloading https://dl.espressif.com/pypi/packaging/packaging-22.0-py3-none-any.whl (42 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.6/42.6 kB 4.8 MB/s eta 0:00:00

[kumekay]

@AxelLin Thank you so much. We were able to reproduce the problem.

While a proper solution is on its way, could you please stick to an older version of the component manager by running

pip install idf-component-manager==1.1.4

in your ESP-IDF environment (after . $IDF_PATH/export.sh)?

We will move this issue to http://github.com/espressif/idf-component-manager project.

[AxelLin]

Hi @kumekay
#15 (comment) does not work, I got another error:
'''
-- Building ESP-IDF components for target esp32c3
CMake Error at /home/axel/esp/esp-idf/tools/cmake/build.cmake:542 (message):
usage: main.py [-h] [--project_dir PROJECT_DIR]

                 [--interface_version {0,1}]
                 {prepare_dependencies,inject_requirements,inject_requrements}
                 ...

main.py: error: argument --interface_version: invalid choice: 2 (choose
from 0, 1)

Call Stack (most recent call first):
/home/axel/esp/esp-idf/tools/cmake/project.cmake:447 (idf_build_process)
CMakeLists.txt:26 (project)

-- Configuring incomplete, errors occurred!
'''
I give up using master for now (hope to get it fix soon).

[kumekay]

@AxelLin Yes, unfortunately, after espressif/esp-idf@17224f3 master branch requires component-manager 1.2+

You can use the commit of ESP-IDF before this.

The solution to the problem is not trivial and takes time. We will release it by next week.

[kumekay]

A problem in espressif/idf-flash-vendor-patches#1 (comment) is probably related to this one, so I'm mentioning the comment.

[AxelLin]

@AxelLin Yes, unfortunately, after espressif/esp-idf@17224f3 master branch requires component-manager 1.2+

You can use the commit of ESP-IDF before this.

The solution to the problem is not trivial and takes time. We will release it by next week.

@kumekay
Please don't block the users due to this issue.
If it really takes time to fix the issue properly, please release a temporary fix first.

[kumekay]

@AxelLin, I apologize for the delay in addressing this issue. We are committed to releasing a solution by tomorrow.

Update: 1.2.2 release is planned for Friday, 13.01

[kumekay]

@AxelLin

Before we release, would you mind trying the version of the component manager from the branch bugfix/default_excludes to confirm that it resolves the issue for you?

You can install it with this command:

pip install git+https://github.com/espressif/idf-component-manager.git@bugfix/default_excludes -U

[kumekay]

@AxelLin

I apologize for not mentioning it in my previous comment, but before installing from git, it is necessary to remove the previous version from PyPI.

However, I am pleased to inform you that a new bugfix release has been made available on PyPI. You can update to v1.2.2 either by running ./install.sh from the ESP-IDF directory or by running pip install -U idf-component-manager within the ESP-IDF environment.

[AxelLin]

Hi @kumekay
It works now, thanks a lot.

[AxelLin]

@kumekay

I'm using v5.1.1-1-gd3c99ed3b8 to build for esp32c3.

$ idf.py fullclean build
Executing action: fullclean
Executing action: remove_managed_components
ERROR: Some components (joltwallet__littlefs) in the "managed_components" directory were modified on the disk since the last run of the CMake. Content of this directory is managed automatically.
If you want to keep the changes, you can move the directory with the component to the "components"directory of your project.
I.E. for "joltwallet__littlefs" run:
mv /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs /home/axel/esp/esp-idf-dev/apps/myproj/components/joltwallet__littlefs
Or, if you want to discard the changes remove the ".component_hash" file from the component's directory.
I.E. for "joltwallet__littlefs" run:
rm /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs/.component_hash
axel@alpha:~/esp/esp-idf-dev/apps/myproj$ cat /home/axel/esp/esp-idf-dev/apps/myproj/managed_components/joltwallet__littlefs/.component_hash
2e392b279f0bc282a9dfb66608de4b5351d5ab4249254cbb5ebae09099c045ec

But I didn't touch anything in managed_components folder.
Note: v5.2-dev-2383-g82cceabc6e also has the same issue.

[kumekay]

Hello @AxelLin

It looks like it's the same problem as in #40

For now, you can use IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS=1 as a quick workaround.

[AxelLin]

@kumekay
When would you fix this?

[kumekay]

Hello @AxelLin

We are still looking for the best approach to dealing with this problem maintaining guarantees on reproducibility that component manager promises.

We will address it in the coming release

[kumekay]

Hello @AxelLin the files that littlefs generates during build are already gitignored and can be excluded to resolve the problem.

Unfortunately, we can only do it for coming ESP-IDF version 5.2. For existing versions of IDF, we can exclude filters to the idf_component.yml of the littlefs. So for this particular component, will do our best to fix it next week.

[AxelLin]

Hi @kumekay

This is probably off-topic from the origianl issue, I'm wondering if you can help to answer it:
(I asked the question in ep32 forum https://www.esp32.com/viewtopic.php?f=13&t=37005, but unforunately no response there.)

While debugging, sometimes I would like to add some debug code in the managed component files,
but it seems the idf.py build always build with a clean cloned managed component files.
Is it possible to make some local changes in the managed component files and build with the local changes?
This would be helpful to debug issues in the managed component files.

[kumekay]

Hi @AxelLin
You can move a component from managed_components directory of your project to components directory and modify it there. Then IDF will use the component from components directory and component manager won't make a copy in the managed_components

[hfudev]

related to 193974b

[hfudev]

Version 2.0.0 has been released, and I think 193974b solves your original problem.

closing the issue.