nvflare submiting job in poc mode #2122
-
|
hello i'm new to nvflare, i've been trying to run some examples jobs in poc mode for version 2.3, using the admin command line, but every time i try it returns that Error: meta.[json|conf|yml] not existing for job " ", possible in legacy job format. Please upgrade the job structure. I'm following the tutorial of this link https://developer.nvidia.com/blog/experimenting-with-novel-distributed-applications-using-nvidia-flare-2-1/ , i was previously just trying to follow the instructions on the README of each job, but it was also not working in the advanced examples, always returning an error of not enough sites have enough resources or when not that it was breaking due to some missing module |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 20 replies
-
|
@ramiro350 thanks for the interest! The reason you are seeing those errors is because version mismatching. For you, please do the following:
Make sure to install requirements of each examples. (can be done using "pip install -r requirements.txt" in each example folder) |
Beta Was this translation helpful? Give feedback.
-
|
i was in the main branch previously, where nvflare poc --start was working, but now in branch 2.3, it returns the logs i placed above and the clients still sent heartbeats to eachother |
Beta Was this translation helpful? Give feedback.
-
|
the main branch is latest branch for 2.4.0 release. Where the POC command is changed nvflare poc --start is changed to nvflare poc start ( without '--') here is what I have on main branch ( on mac) Since it is on admin console, I run check_status command |
Beta Was this translation helpful? Give feedback.
-
|
i'm working with nvflare 2.3.8, and on branch 2.3, my problem now is that on the cifar10-real-world when i try to run a job in poc mode it returns not not enough sites have enough resources (ok sites 0 < min sites 8), i've already tried a couple of different approaches like submitting the job using the submit_job,sh file and directly using the admin cmd line, but they both return the same error. |
Beta Was this translation helpful? Give feedback.
-
|
@ramiro350 follow steps in https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/cifar10-real-world Please do just ONE of the following options:
@holgerroth we should improve this to avoid confusion ^ If you have NO GPU, please change the "meta.json" (https://github.com/NVIDIA/NVFlare/blob/2.3/examples/advanced/cifar10/cifar10-real-world/jobs/cifar10_fedavg_stream_tb/meta.json) to the following: Note1: The HE example would only work with real-world setup |
Beta Was this translation helpful? Give feedback.
-
|
i have followed the steps 3.4, 3.5,3.6, and also changed the meta file to the one you provided, trying to submit the cifar10_fedavg_stream_tb job, and now i got this error instead |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ramiro350, please make sure to set the PYTHON path before starting the poc server and clients |
Beta Was this translation helpful? Give feedback.
-
|
hi @holgerroth, i did that actually, everything from steps 3.4 till 3.6 i have done it, maybe it might be the path i'm in, when launching the export PYTHONPATH command i'm inside cifar10-real-world folder, is this the correct one ? |
Beta Was this translation helpful? Give feedback.
-
|
Please check the content of your path with |
Beta Was this translation helpful? Give feedback.
-
|
@holgerroth yes it happens after step 3.6 |
Beta Was this translation helpful? Give feedback.
-
|
Looks good. What's happening after this? |
Beta Was this translation helpful? Give feedback.
-
|
@ramiro350 please run the following commands inside the "cifar10-real-world" directory and paste the results: For your reference this is my result: |
Beta Was this translation helpful? Give feedback.
-
|
@ramiro350 thanks for sharing this, I looked at your response above, from the log you paste with "nvflare poc --start", your PYTHONPATH was NOT set properly there. Can you please nvflare poc --start again and see if it works after you set the path? You should be seeing something like For reference, this is mine: |
Beta Was this translation helpful? Give feedback.
-
|
Hi @YuanTingHsieh my pythonpath is this PYTHONPATH is /local/custom:/home/ramiro/projetos/federated_learning/nvflare/NVFlare/.. |
Beta Was this translation helpful? Give feedback.
-
|
@ramiro350 all path might look similar If you follow the setup step (https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/cifar10-real-world#1-install-requirements) correctly. Another way for you just run the following command: You need this path to be correctly setup so your python program can access codes in https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/pt |
Beta Was this translation helpful? Give feedback.
-
|
Hi @YuanTingHsieh the path was incorrect, thank you for the help. |
Beta Was this translation helpful? Give feedback.
-
|
@ramiro350 anytime, glad this problem is solved! |
Beta Was this translation helpful? Give feedback.
@ramiro350 follow steps in https://github.com/NVIDIA/NVFlare/tree/2.3/examples/advanced/cifar10/cifar10-real-world
Please do just ONE of the following options:
@holgerroth we should improve this to avoid confusion ^
If you have NO GPU, please change the "meta.json" (https://github.com/NVIDIA/NVFlare/blob/2.3/examples/advanced/cifar10/cifar10-real-world/jobs/cifar10_fedavg_stream_tb/meta.json) to the following:
Note1: T…