How to evaluate models?

[annaproxy]

More of a philosophical question.

• We don't want models to have perfect training accuracy, few novel words are generated
• We don't want models to have training accuracy that is too low, implausible words are generated (bvbvmcnv)

Any papers/blog posts on this are welcome!

[Sasafrass]

This is a very interesting question.
Some potential avenues of research:

• Optimize for models with lowest training accuracy that still generate coherent and phonetically sound words. The soundness of the words could be assessed with some kind of NLP parser. However, this raises the issue that for instance (cool new) slang words may not always adhere to (Dutch) language norms in terms of phonetics.

[annaproxy]

Under the assumption that the model only gets better on the train data when you train for longer, non-converged models are one way to still get a certain level of 'babbling' behaviour. (term stolen from https://arxiv.org/pdf/2010.04637.pdf).

One easy way would be to check every (few) epochs if the model is memorizing too many words.
So, simply sample a large amount of words (with various temperatures) from a model, n% of these words will be memorized from train data.

We would have to experimentally find a good value for n. The n% can also be used a sort of proxy for full word accuracy, but we can also have a threshold level of accuracy (at character level).

Edit: A quick look at n for an upcoming model of plaatsnamen :)