How to adjust hyperparameters when using different datasets?

[TimeLessLing]

I don't have English Penn Treebank datasets, so I used Chinese Penn Treebank and UD 2.0 datasets (convert to conll-format). But when I trained this network, I can't get a good result. The loss will drop to around 1.5 and then it will not fall again. And the maximum valid accuracy is 40%. I have tuned lr(from 2e-6 to 2e-3) and batch-size(from 1 to 128), but the results are still not much changed. What should I do in this dilemma?