Configuring a packaged Kedro project while using a custom logging config with File Handlers raises error

[cariveroco]

Description

Importing configure_project from kedro.framework.project fails when using a custom Kedro logging config with File Handlers, if the target files of the handlers do not exist yet. The log files do not get automatically created.

The FileNotFoundError occurs when importing project, session, or startup from kedro.framework.

Context

I intend to run pipelines from a packaged Kedro Project in a "clean" directory (eventually: in a docker container) using a custom logger with Rotating File Handlers. I installed the packaged Kedro project and tried to run pipelines via script, which includes first configuring the project using kedro.framework.project.configure_project. Since the working directory is "clean" (i.e., does not have log files yet), I expected the log files to be automatically created, as is the observed behavior when I run pipelines via CLI in my local machine (where I have direct access to the source code in the Project Directory). Instead, I'm getting FileNotFoundError.

Steps to Reproduce

1. Start with a clean directory/environment.
2. Set up a custom_logging_config.yml file with a Rotating File Handler (e.g. level: INFO, class: logging.handlers.RotatingFileHandler, filename: logs/info.log)
3. Point the environment variable KEDRO_LOGGING_CONFIG to the previously created file.
4. Import session or project or startup from kedro.framework

Expected Result

The log files automatically get created and the execution proceeds to running the pipelines.

Actual Result

FileNotFoundError encountered.

INFO:kedro.framework.project:Using 'custom_logging_config.yml' as logging configuration. 
--- Logging error ---
Traceback (most recent call last):
  File "/Users/PQ27WN/.pyenv/versions/3.11.9/lib/python3.11/logging/handlers.py", line 73, in emit
    if self.shouldRollover(record):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/PQ27WN/.pyenv/versions/3.11.9/lib/python3.11/logging/handlers.py", line 194, in shouldRollover
    self.stream = self._open()
                  ^^^^^^^^^^^^
  File "/Users/PQ27WN/.pyenv/versions/3.11.9/lib/python3.11/logging/__init__.py", line 1213, in _open
    return open_func(self.baseFilename, self.mode,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Users/PQ27WN/AFv311/exp-build-package/logs/info.log'

Your Environment

• Kedro 0.19.9
• Python 3.11.9
• macOS Sonoma 14.7

[rashidakanchwala]

Hi @cariveroco,

Thank you for raising this issue! We appreciate your input. We’ll take a closer look and get back to you shortly with any updates or follow-up questions.

[merelcht]

Hi @cariveroco, I've tried to reproduce your issue and can see a similar problem. My observation is that the issue is resolved if the custom_logging.yml file is at the same level as where you want your logs to be stored. So in your case, you should move the custom_logging.yml inside logs. Let me know if this works for you.

We'll see if there's anything we can do make it work with the setup you described.

[astrojuanlu]

Thanks @cariveroco , I've turned this idea into a discussion. My understanding is that @merelcht 's comment above gave a workaround. Does that work for you?