Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training spec length limit is ~128 KiB, fails with misleading error message #512

Closed
nkemnitz opened this issue Sep 18, 2023 · 2 comments
Closed

Training spec length limit is ~128 KiB, fails with misleading error message #512

nkemnitz opened this issue Sep 18, 2023 · 2 comments
Labels
bug Something isn't working training

Comments

@nkemnitz
Copy link
Collaborator

nkemnitz commented Sep 18, 2023 •
edited
Loading

If ZETTA_RUN_SPEC is larger than presumably* 128 KiB, _parse_spec_and_train will fail with a misleading OSError pointing at git rev-parse --show-toplevel

* fails with 133,522 Bytes, works with 128,700 Bytes

(/home/nkemnitz/zetta/zetta_utils/venv) nkemnitz@Eriador:~/zetta/zetta_utils$ python /home/nkemnitz/zetta/zetta_utils/specs/nico/training/em_encoder/train/01_m3_m3_encoder_dict.py
133522
2023-09-18 13:38:48.969 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/common.py:  81                                                                                                                                       
                                 Cluster provider: GKE/GCP.                                                                                                                                                                                                                             
2023-09-18 13:38:49.876 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/configmap.py:  42                                                                                                                                    
                                 Creating k8s configmap `exec-whispering-ultramarine-dogfish-of-vitality`                                                                                                                                                                               
2023-09-18 13:38:51.720 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/common.py:  81                                                                                                                                       
                                 Cluster provider: GKE/GCP.                                                                                                                                                                                                                             
2023-09-18 13:38:52.606 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/common.py:  81                                                                                                                                       
                                 Cluster provider: GKE/GCP.                                                                                                                                                                                                                             
2023-09-18 13:38:53.474 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/secret.py:  97                                                                                                                                       
                                 Creating k8s secret `exec-whispering-ultramarine-dogfish-of-vitality-zetta-user`                                                                                                                                                                       
2023-09-18 13:38:54.014 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/secret.py:  97                                                                                                                                       
                                 Creating k8s secret `exec-whispering-ultramarine-dogfish-of-vitality-zetta-project`                                                                                                                                                                    
2023-09-18 13:38:54.503 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/secret.py:  97                                                                                                                                       
                                 Creating k8s secret `exec-whispering-ultramarine-dogfish-of-vitality-wandb-api-key`                                                                                                                                                                    
2023-09-18 13:38:54.773 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 188                                                                                                                                          
                                 Creating k8s job `exec-whispering-ultramarine-dogfish-of-vitality`                                                                                                                                                                                     
2023-09-18 13:38:55.314 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/common.py:  81                                                                                                                                       
                                 Cluster provider: GKE/GCP.                                                                                                                                                                                                                             
2023-09-18 13:38:56.233 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 100                                                                                                                                          
                                 Waiting for `exec-whispering-ultramarine-dogfish-of-vitality` to start.                                                                                                                                                                                
2023-09-18 13:39:01.368 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 100                                                                                                                                          
                                 Waiting for `exec-whispering-ultramarine-dogfish-of-vitality` to start.                                                                                                                                                                                
2023-09-18 13:39:06.501 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 100                                                                                                                                          
                                 Waiting for `exec-whispering-ultramarine-dogfish-of-vitality` to start.                                                                                                                                                                                
2023-09-18 13:39:11.636 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 100                                                                                                                                          
                                 Waiting for `exec-whispering-ultramarine-dogfish-of-vitality` to start.                                                                                                                                                                                
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 104                                                                                                                                          
                                 `exec-whispering-ultramarine-dogfish-of-vitality` job started.                                                                                                                                                                                         
2023-09-18 13:39:20.002 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 2023-09-18 11:39:22.058 INFO     zetta_utils                                                                                                                                                                                                           
2023-09-18 13:39:20.004 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                                                  /opt/zetta_utils/zetta_utils/parsing/cue.py:                                                                                                                                                                          
2023-09-18 13:39:20.005 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                                                  60                                                                                                                                                                                                                    
2023-09-18 13:39:20.006 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                                                  Copying 'specs/train.cue' to                                                                                                                                                                                          
2023-09-18 13:39:20.007 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                                                  /tmp/tmp6kqhitik/remote_file.cue for parsing...                                                                                                                                                                       
2023-09-18 13:39:22.474 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 ╭───────────────────── Traceback (most recent call last) ──────────────────────╮                                                                                                                                                                       
2023-09-18 13:39:22.477 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/bin/zetta:8 in <module>                                           │                                                                                                                                                                       
2023-09-18 13:39:22.480 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.481 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   5 from zetta_utils.cli.main import cli                                     │                                                                                                                                                                       
2023-09-18 13:39:22.482 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   6 if __name__ == '__main__':                                               │                                                                                                                                                                       
2023-09-18 13:39:22.483 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])     │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ ❱ 8 │   sys.exit(cli())                                                      │                                                                                                                                                                       
2023-09-18 13:39:22.484 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   9                                                                          │                                                                                                                                                                       
2023-09-18 13:39:22.485 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/click/core.py:1137 in __call__       │                                                                                                                                                                       
2023-09-18 13:39:22.486 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/click/core.py:1062 in main           │                                                                                                                                                                       
2023-09-18 13:39:22.487 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.488 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/click/core.py:1668 in invoke         │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.489 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/click/core.py:1404 in invoke         │                                                                                                                                                                       
2023-09-18 13:39:22.490 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/click/core.py:763 in invoke          │                                                                                                                                                                       
2023-09-18 13:39:22.491 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/cli/main.py:67 in run                           │                                                                                                                                                                       
2023-09-18 13:39:22.492 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.493 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   64 │   │   spec = zetta_utils.parsing.cue.loads(str_spec)                  │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   65 │                                                                       │                                                                                                                                                                       
2023-09-18 13:39:22.494 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   66 │   os.environ["ZETTA_RUN_SPEC"] = json.dumps(spec)                     │                                                                                                                                                                       
2023-09-18 13:39:22.495 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ ❱ 67 │   result = zetta_utils.builder.build(spec)                            │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   68 │   logger.debug(f"Outcome: {pprint.pformat(result, indent=4)}")        │                                                                                                                                                                       
2023-09-18 13:39:22.496 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   69 │   if pdb:                                                             │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │   70 │   │   breakpoint()  # pylint: disable=forgotten-debug-statement # pra │                                                                                                                                                                       
2023-09-18 13:39:22.497 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.498 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:49 in build                    │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.499 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:84 in _traverse_spec           │                                                                                                                                                                       
2023-09-18 13:39:22.500 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:123 in _build_dict_spec        │                                                                                                                                                                       
2023-09-18 13:39:22.501 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:124 in <dictcomp>              │                                                                                                                                                                       
2023-09-18 13:39:22.502 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.503 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:84 in _traverse_spec           │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.504 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:146 in _build_dict_spec        │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.505 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/builder/build.py:135 in _build_dict_spec        │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.506 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/zetta_utils/zetta_utils/training/lightning/trainers/default.py:72 in    │                                                                                                                                                                       
2023-09-18 13:39:22.507 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ __init__                                                                     │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.508 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │    69 │   │                                                                  │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │    70 │   │   if not os.environ.get("WANDB_MODE", None) == "offline":  # pra │                                                                                                                                                                       
2023-09-18 13:39:22.509 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │    71 │   │   │   api_key = os.environ.get("WANDB_API_KEY", None)            │                                                                                                                                                                       
2023-09-18 13:39:22.510 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ ❱  72 │   │   │   wandb.login(key=api_key)                                   │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │    73 │   │   │   wandb.init(                                                │                                                                                                                                                                       
2023-09-18 13:39:22.511 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │    74 │   │   │   │   project=experiment_name,                               │                                                                                                                                                                       
2023-09-18 13:39:22.512 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │    75 │   │   │   │   group=f"{experiment_name}.{experiment_version}",       │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.513 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_login.py:74 in login │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.514 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_setup.py:327 in      │                                                                                                                                                                       
2023-09-18 13:39:22.515 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ setup                                                                        │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.516 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_setup.py:320 in      │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ _setup                                                                       │                                                                                                                                                                       
2023-09-18 13:39:22.517 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.518 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_setup.py:303 in      │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ __init__                                                                     │                                                                                                                                                                       
2023-09-18 13:39:22.519 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_setup.py:108 in      │                                                                                                                                                                       
2023-09-18 13:39:22.520 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ __init__                                                                     │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.521 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_setup.py:138 in      │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ _settings_setup                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.522 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.523 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/wandb_settings.py:1735 in  │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ _infer_run_settings_from_environment                                         │                                                                                                                                                                       
2023-09-18 13:39:22.524 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/wandb/sdk/lib/gitlib.py:86 in root   │                                                                                                                                                                       
2023-09-18 13:39:22.525 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.526 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/git/cmd.py:739 in <lambda>           │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.527 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/git/cmd.py:1315 in _call_process     │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.528 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/site-packages/git/cmd.py:987 in execute            │                                                                                                                                                                       
                        INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.529 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/subprocess.py:971 in __init__                      │                                                                                                                                                                       
2023-09-18 13:39:22.530 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │                                                                              │                                                                                                                                                                       
2023-09-18 13:39:22.531 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 │ /opt/conda/lib/python3.10/subprocess.py:1863 in _execute_child               │                                                                                                                                                                       
2023-09-18 13:39:22.532 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 ╰──────────────────────────────────────────────────────────────────────────────╯                                                                                                                                                                       
2023-09-18 13:39:22.533 INFO     zetta_utils /home/nkemnitz/zetta/zetta_utils/zetta_utils/cloud_management/resource_allocation/k8s/job.py: 129                                                                                                                                          
                                 OSError: [Errno 7] Argument list too long: 'git'
@supersergiy supersergiy added the bug Something isn't working label Sep 18, 2023
@akhileshh
Copy link
Contributor

Seems like an OS limitation.

We need a workaround.

@akhileshh akhileshh mentioned this issue Sep 26, 2023
Closed
@nkemnitz
Copy link
Collaborator Author

Fix included in #553

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working training
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(train): env var limit workaround ZettaAI/zetta_utils
3 participants
@supersergiy @nkemnitz @akhileshh