New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question of the experimental results. #3

Open

kimjeyoung opened this issue Dec 14, 2022 · 0 comments

kimjeyoung commented Dec 14, 2022

Thank you for providing the valuable work and benchmark!
I have a question for reproducibility:

When I re-run the implementation of vanilla BERT followed your script, BERT trained with AGNews-FL achieved 91.25 (ID Acc.). However, Table 3 in the paper, the ID classification accuracy for the vanilla BERT is reported by 85.58. I didn't modify the code, and I ran the script as it is. Which result is correct? Or is the dataset you uploaded to Google Drive different from when you wrote the paper?

Best,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment