Support for Danish (perplexity and entropy) ???

[javaberlin]

Hi,

I am trying to use textdescriptives with Dacy and a Danish language model to calculate perplexity and entropy, but I get the error :

⚠ Could not load lexeme probability table for language da. This will
result in NaN values for perplexity and entropy.

nlp = dacy.load("large")
nlp.add_pipe("textdescriptives/dependency_distance")
nlp.add_pipe("textdescriptives/readability")
nlp.add_pipe("textdescriptives/information_theory")

Is that another way to make the library work with Danish when calculating entropy and perplexity ?

Thank you

[KennethEnevoldsen]

Sadly spacy does not have a lexeme prob table for Danish:

You can see an example here

One could submit a pull request to add it, e.g. based on:

https://huggingface.co/datasets/chcaa/dagw-word-frequencies

[KennethEnevoldsen]

That someone is apparently me ;) I have added a PR for this here

[javaberlin]

Thank you.

Another question I have is about the readability scores: the formula for the Automated Readability Index (ARI) and other scores have hardcoded coefficients that are specific for the English language, am I right ? That would mean that none of those metrics (except LIX and RIX) would work well with Danish. The coefficients would need to be calculated for Danish, is it right ?

Here is a script using the Textdescriptives and a Danish language model ( da_core_news_lg) -> will the results be sound ??

import spacy
import textdescriptives as td
import os

spacy.require_gpu()
spacy.prefer_gpu(0)
nlp = spacy.load("da_core_news_lg")

nlp.add_pipe("textdescriptives/coherence")
nlp.add_pipe("textdescriptives/readability")

input_dir = r"/home/jovyan/work/sources/samples"

# Iterate over all files in the directory
for filename in os.listdir(input_dir):
    file_path = os.path.join(input_dir, filename)

    if file_path.endswith(".txt"):
        try:
            with open(file_path, "r", encoding="utf-8") as file:
                text = file.read()
                doc = nlp(text)
                print(f"Results for {filename}:")
                print(f"Readability: {doc._.readability}")
                print(f"Coherence: {doc._.coherence}")
                print("-" * 50)
        except Exception as e:
            print(f"Error processing {filename}: {e}")

RESULTS:

/opt/conda/envs/myenv-3.9.12/lib/python3.9/site-packages/textdescriptives/components/coherence.py:44: UserWarning: [W008] Evaluating Span.similarity based on empty vectors.
  similarities.append(sent.similarity(sents[i + order]))
  
Results for Karen_Sundt$Kjærlighedens_Forvildelser$forvildelser$1855-1900-Realisme%novel-novella%1894.txt:

Readability: {'flesch_reading_ease': 67.09013518828942, 'flesch_kincaid_grade': 6.674740838715174, 'smog': 9.256530842258726, 'gunning_fog': 8.58755364999563, 'automated_readability_index': 3.8998036009089496, 'coleman_liau_index': 6.202410051889014, 'lix': 25.49686866242942, 'rix': 1.600082062504274}

Coherence: {'first_order_coherence': 0.4807865969391

[HLasse]

Hi @javaberlin. Yes, many of the readability metrics were developed for English and have been "tuned" for e.g. a certain grade-level for the English language. Given that Danish has a lot of compound words, Danish sentences are likely to obtain higher scores than comparable sentences in English in e.g. ARI. But, whether they work well/are valid depends highly on the use-case.

Do you need to rank sentences in terms of their ease of reading, want to use it as a feature for machine learning, or are you trying to find suitable literature for a specific grade-level? ARI would work fine for Danish in the former two cases, but LIX or RIX would probably be more suitable for Danish in the latter case.

So, yes, the results are sound, you just have to know how to interpret the score and to choose the right metric depending on the use case.

[javaberlin]

The specific case I am working on is comparing books with each other ( 19th century Norwegian literature, Riksmaal) . I will then use ARI , LIX and RIX .

Thank you

[javaberlin]

Regarding the Entropy / Perplexity, I am using Textdescriptives with the model "da_core_news_lg"

When processing a book ( https://www.bokselskap.no/boker/mennesker/kap1 ) the perplexity value seems unrealistic high (1.3202179620399305e+25):

Author: Alvilde_Prydz
Book Name: Mennesker
Epoch: 1855-1900-Realisme
Genre: novel-novella
Year: 1892
First Order Coherence: 0.763983737338673
Second Order Coherence: 0.7498873229380008
Perplexity: 1.3202179620399305e+25
Entropy: 57.84242417057568
Mean Dependency Distance: 3.4753817668740536
Dependency Standard Deviation: 1.7051684521719186
Pos Proportions: {'pos_prop_ADJ': 0.0776843146956952, 'pos_prop_ADP': 0.08461157842652152, 'pos_prop_ADV': 0.11677387431964374, 'pos_prop_AUX': 0.047006432459178626, 'pos_prop_CCONJ': 0.07817911924789708, 'pos_prop_DET': 0.05244928253339931, 'pos_prop_INTJ': 0.0009896091044037606, 'pos_prop_NOUN': 0.15239980207817913, 'pos_prop_NUM': 0.0034636318654131617, 'pos_prop_PART': 0.011380504700643246, 'pos_prop_PRON': 0.10192973775358734, 'pos_prop_PROPN': 0.009896091044037604, 'pos_prop_PUNCT': 0.13953488372093023, 'pos_prop_SCONJ': 0.015338941118258289, 'pos_prop_SYM': 0.0, 'pos_prop_VERB': 0.10687778327560614, 'pos_prop_X': 0.0014844136566056407}
Descriptive Stats: {'n_tokens': 1666.0, 'n_unique_tokens': 695.0, 'proportion_unique_tokens': 0.4171668667466987, 'n_characters': 7874.0, 'n_sentences': 56.0, 'token_length_mean': 4.513205282112845, 'token_length_median': 4.0, 'token_length_std': 2.4482898806334097, 'sentence_length_mean': 29.75, 'sentence_length_median': 23.5, 'sentence_length_std': 28.8135942618361, 'syllables_per_token_mean': 1.6260504201680672, 'syllables_per_token_median': 1.0, 'syllables_per_token_std': 0.839463832972911}
Flesch Reading Ease: 39.07488445378155
Flesch Kincaid Grade: 15.199894957983194
SMOG: 14.191785591663535
Gunning-Fog: 16.942016806722688
Automated Readability Index: 14.7021968787515
Coleman-Liau Index: 9.742689075630249
LIX: 46.796818727491
RIX: 5.071428571428571

{'entropy': 57.84242417057568, 'perplexity': 1.3202179620399305e+25, 'per_word_perplexity': 6.532498575160468e+21}

Does that make sense ? Do I need other model for this task ?

Thanks

[HLasse]

Given that the book is in Norwegian you are better off using a Norwegian spacy model. The perplexity is high because most words are not in the vocabulary (because the vocab is Danish and the book is in Norwegian Riiksmål). I could only find a Bokmål model, but that's probably better than the Danish one in any case: https://spacy.io/models/nb