Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

when doing exp 40-30-10-20 #26

Open
Ultimate-Storm opened this issue Feb 13, 2023 · 0 comments
Open

when doing exp 40-30-10-20 #26

Ultimate-Storm opened this issue Feb 13, 2023 · 0 comments

Comments

@Ultimate-Storm
Copy link
Collaborator

Since the dataset on each host is unbalanced, should find out a better way to do sync since on host3 with the smallest dataset always finish the epochs faster and will loop in the local small dataset. Causing overfitting.
Approaches could be:

  • Try different sync intervals according to different size of ds
  • try larger epochs for hosts with smaller ds
  • try different weight learning for hosts. Host with smaller ds should weigh less(gain less weight) during local training
  • put on_batch_end to on_epoch_end
  • it would be better that host would be able to wait for others to reach epoch end then do sync(currently I think it would initiate the sync when it reaches the sync frequency with
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant