cannot find neptune.new.utils with Neptune-client==0.16.3

[filipporemonato]

Bug description

Pytorch Lightning 2.0.8 - import pytorch_lightning as pl fails with error message No module named 'neptune.new.utils', but I have Neptune-client 0.16.3. More interestingly, the code runs perfectly on an env that does not even have neptune-client installed.
I have reproduced on a clean environment (see steps below), and I was able to import Lightning up until the point where I installed Neptune-client. Then, the import fails with the error message presented here.
My problem is that I need to use Lightning in an env a customer provided us, which uses Neptune-client as indicated, and I am therefore unsure on whether we can remove/change that package.
This is on Python 3.10.12.

Error traceback:

ModuleNotFoundError                       Traceback (most recent call last)
File [main.py] in line 7
      1 #%%
      2 # basic libraries
      3 #import  os
      4 #import  yaml
      5 
      6 # pytorch lightning
----> 7 import  pytorch_lightning           as      pl
      8 from    lightning.pytorch           import  Trainer
      9 from    lightning.pytorch.loggers   import  TensorBoardLogger

File [~/miniforge3/envs/test_env/lib/python3.10/site-packages/pytorch_lightning/__init__.py:26]
     23     _logger.propagate = False
     25 from lightning_fabric.utilities.seed import seed_everything  # noqa: E402
---> 26 from pytorch_lightning.callbacks import Callback  # noqa: E402
     27 from pytorch_lightning.core import LightningDataModule, LightningModule  # noqa: E402
     28 from pytorch_lightning.trainer import Trainer  # noqa: E402

File [~/miniforge3/envs/test_env/lib/python3.10/site-packages/pytorch_lightning/callbacks/__init__.py:29]
     27 from pytorch_lightning.callbacks.prediction_writer import BasePredictionWriter
     28 from pytorch_lightning.callbacks.progress import ProgressBar, RichProgressBar, TQDMProgressBar
---> 29 from pytorch_lightning.callbacks.pruning import ModelPruning
     30 from pytorch_lightning.callbacks.rich_model_summary import RichModelSummary
...
     56 else:
     57     # needed for tests, mocks and function signatures
     58     neptune, Run, Handler, File, stringify_unsupported = None, None, None, None, None

ModuleNotFoundError: No module named 'neptune.new.utils'

What version are you seeing the problem on?

v2.0

How to reproduce the bug

Create new environment

conda create -n test_env

Install Pytorch 1.11.0

conda install pytorch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 -c pytorch

Install Pytorch Geometric (the one for Pytorch 1.11)

pip install torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.11.0+cpu.html

Install Pytorch Lightning (note that lightning 2.0 is compatible with Pytorch 1.11, see compatibility matrix https://lightning.ai/docs/pytorch/stable/versioning.html)

python -m pip install lightning

Install neptune-client 0.16.3

pip install neptune-client==0.16.3

Try importing Pytorch Lightning

import pytorch_lightning as pl

Obtain error message above.

### Environment

<details>
  <summary>Current environment</summary>

#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version: 2.0.8
#- PyTorch Version: 1.11.0
#- Python version: 3.10.12
#- OS: McOS Ventura 13.4.1

</details>


cc @Blaizzy

[awaelchli]

@filipporemonato It looks like the neptune.new module was removed at some point. You can work around the issue for now by uninstalling neptune-client (legacy) and installing pip install neptune instead. Let me know if that works for you.

[filipporemonato]

Thanks Adrian! Yes that's what I thought too. It may work as a workaround for now, but as said Neptune-client comes from the customer's env, so I am not sure if it will be possible to change it and go over to Neptune. But it should be (bar some backward-compatibility issues). I guess my post here was more about the strange fact that all goes smoothly as long as Neptune-client is not installed. It seems a bit weird that installing a package "breaks" the usage of Lightning, especially if Lightning can well be used without that package at all. Thanks for the suggestion about Neptune though.

[SiddhantSadangi]

@filipporemonato , @awaelchli

utils was introduced in neptune.new only in version 0.16.16. Since you are using neptune-client==0.16.3, this import will fail.
We will get this fixed.

For now, you can either update to neptune as suggested above (preferred), or comment out line 55 of pytorch_lightning/loggers/neptune.py:

elif _NEPTUNE_CLIENT_AVAILABLE:
    # <1.0 package structure
    import neptune.new as neptune
    from neptune.new import Run
    from neptune.new.handler import Handler
    from neptune.new.types import File
    # from neptune.new.utils import stringify_unsupported

Here's an upgrade guide to Neptune 1.0+: https://docs.neptune.ai/setup/neptune-client_1-0_release_changes/

[awaelchli]

@SiddhantSadangi What should be done here? Options:

1. Set 0.16.16 as the minimum supported version
2. Change the import (to what?)
3. Copy the implementation of stringify_unsupported to support it in older versions.

Any other suggestions?

[SiddhantSadangi]

@awaelchli - 1 would be the most viable option until we submit a fix to pytorch_lightning

[awaelchli]

No @SiddhantSadangi I'm talking about the fix. These are the three options I see to fix it in the NeptuneLogger implementation in Lightning. So of these 3 options, which one do you want me to do? Or do you see another one?

[ArtemSivtsov]

@awaelchli Hello! Let me share my experience. I met the same issue. The easiest solution is to do pip uninstall neptune-client then pip install neptune, but make sure that everything of neptune-client was replaced. After this my pipeline started to work without any problems.

current versions in pip are:
neptune 1.6.3
neptune-contrib 0.28.1

[SiddhantSadangi]

@ArtemSivtsov - that's the recommended solution, however sometimes users are constrained with the packages they can install, and that might prevent them from upgrading to neptune>=1.0

[SiddhantSadangi]

@awaelchli - for the fix, I was thinking of introducing a _STRINGIFY_UNSUPPORTED_AVAILABLE flag and updating the import to the below:

_STRINGIFY_UNSUPPORTED_AVAILABLE = True
...
elif _NEPTUNE_CLIENT_AVAILABLE:
  ...
  # Needed for neptune-client<0.16.16
  try:
      from neptune.new.utils import stringify_unsupported
  except ModuleNotFoundError:
      _STRINGIFY_UNSUPPORTED_AVAILABLE = False
...
def log_hyperparams(self, params: Union[Dict[str, Any], Namespace]) -> None:  # type: ignore[override]
  ...
  if _STRINGIFY_UNSUPPORTED_AVAILABLE:
      self.run[parameters_key] = stringify_unsupported(params)
  else:
      self.run[parameters_key] = params

neptune-client<0.16.16 doesn't need stringify_unsupported, so the logging should be fine.

Full file attached as .txt below:
neptune.txt

I can push a PR if this works for you.

[Raalsky]

We've introduced stringify_unsupported to simplify the transition between <1.0.0 and >=1.0.0 that introduced a breaking-change of behavior: https://docs.neptune.ai/setup/neptune-client_1-0_release_changes/#no-more-implicit-casting-to-string. It was introduced earlier as there were some deprecation warnings that suggested switching to this approach if I remember correctly.

There is no need to use stringify_unsupported below 1.0.0 😉 to my knowledge.

Let me prepare the PR that will mostly follow the suggestions from @SiddhantSadangi so the support for 0.16.3 will remain.

[awaelchli]

@Raalsky does #19265 close this issue?

[Raalsky]

@awaelchli Yes. It closes in with a meaning that it won't be supported anymore.