Replies: 15 comments 43 replies
-
|
Bigrams. Sorted. All permutations. Do you need trigrams? |
Beta Was this translation helpful? Give feedback.
-
|
Fantastic! I don't need trigrams. |
Beta Was this translation helpful? Give feedback.
-
|
I think Den (BEAKL) is going to have a go with Opt and trigrams will be useful. Will see tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
First attempt at Cervantes Monkey. Needed to fiddle a bit to get Unicode to behave. 100k text and frequencies attached. Will try 1MB overnight ... takes about 4+ hours if I remember correctly. |
Beta Was this translation helpful? Give feedback.
-
|
Memory a bit rusty ... the 1MB version takes over a day. Video of it at work. The "size" in the yellow text is the number of bigrams left to process. It speeds up as that gets smaller, to about 15 characters per second. The stack started an about 1.2 MB (1MB + 20% spare) Sorry it's a mkv in zip, Github does not allow mkv and VLC is not co-operating to convert it... some encoder not found. |
Beta Was this translation helpful? Give feedback.
-
|
Decided to see how fast it ran on the new PC. cervantes-monkey-2-2021-08-12_11.44.43.mp4 |
Beta Was this translation helpful? Give feedback.
-
|
Trigrams plus counts from Leipzig corpus. I cut it off at the point where the count/highest is 0.01, so just the 1222 most common. Should be enough. spanish-trios-leipzig.csv BTW, before importing into spreadsheet, highlight the first column and Format Cells to text, else the stuff may get interpreted as a formula. Spreadsheets are too clever for their own good. This applies to the other .csv frequency files as well. Cheers, Ian |
Beta Was this translation helpful? Give feedback.
-
|
1MB chained bigrams, plus char frequency. Original job still running :-) |
Beta Was this translation helpful? Give feedback.
-
|
I think that concludes things with the corpus. Will add the texts to KLAnext, but need to revise the user interface a bit ... |
Beta Was this translation helpful? Give feedback.
-
|
Awesome work you did here, Ian, I'm sure this will be insanely useful for future keyboard designers and testers. We're very lucky to have you around ! |
Beta Was this translation helpful? Give feedback.
-
|
Other size chained bigrams and analysis. 10, 20, 30, 40, 50, 60, 100k, 1MB |
Beta Was this translation helpful? Give feedback.
-
|
Thank you so much, @iandoug! Now that I've returned from my travels, I will run the engram-es layout optimization code on your cleaned-up version of the Leipzig corpus, which is the largest and I hope most representative of what people would type. |
Beta Was this translation helpful? Give feedback.
-
|
Leipzig does not include "book type" text ... dialogue etc. |
Beta Was this translation helpful? Give feedback.
-
|
According to https://wortschatz.uni-leipzig.de/en/download/Spanish:
Did you grab all of these? Even these have different volumes, so we have to be careful about bias. |
Beta Was this translation helpful? Give feedback.
-
|
I took the largest file in each row, except for those labelled as South/Central American. Will look at bigrams tomorrow. |
Beta Was this translation helpful? Give feedback.
-
@iandoug -- There appears to be a redundant letter "i":
|
Beta Was this translation helpful? Give feedback.
-
|
One is i. The other is inverted exclamation. Github needs better fonts. |
Beta Was this translation helpful? Give feedback.
-
|
¡Got it -- thanks! |
Beta Was this translation helpful? Give feedback.
-
|
@iandoug -- I would like to put together a text data repo for Spanish like the one I did for English, with the modified files and a readme with a description of your modifications -- could you provide both? And, not speaking or reading Spanish, I don't know how representative each of these categories of text data may be for what people will type. Do you have any sense what the relative representativeness might be, so we can weight them accordingly? What I'm thinking is if you could provide the count of the one-grams and bigrams for each category (news, websites, etc.), then we can combine them with each other and with the Gutenberg counts as we see fit, and others can do so for themselves in the future. |
Beta Was this translation helpful? Give feedback.
-
Thanks Ian! This is great to have. Will try processing and uploading it to type-fu.
This is exactly what happens with the vast majority of the words in badwords.txt. Most of them are perfectly valid words, only with less common gender, plurals, or verb tense. Exactly this. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry I screwed up ... I uploaded the wrong bad words file. Was the original. |
Beta Was this translation helpful? Give feedback.
-
This one has much less words than the previous one, and most of them have typos from missing characters, missing the accute accent and so on. Most words here are actually invalid. That means all valid words from the previous file have been already added to your dictionary? That's good news. How could I help with this one? |
Beta Was this translation helpful? Give feedback.
-
|
Then I think we are done? :-) (And Arno can relax? :-) ) I grabbed some Spanish books from Gutenberg, will see how clean their "Chapter 1"'s are to add to KLA or for your typing apps. "La cigüeña tocaba cada vez mejor el saxofón y el búho pedía kiwi y queso." |
Beta Was this translation helpful? Give feedback.
-
|
Whew!!! On to punctuation today... |
Beta Was this translation helpful? Give feedback.
-
|
@iandoug -- Zenodo is a great idea. There is also osf.io, which is a free service that I use a lot; it caps files at around 5 gb. We should upload all files and description to either or both service when we are ready, and link from our documentation. |
Beta Was this translation helpful? Give feedback.
-
|
Tried to register at osf but I am NOT doing that {)@({@#($ captcha. |
Beta Was this translation helpful? Give feedback.
-
|
@iandoug -- are you recommending that we derive counts for the Leipzig (summed across all categories), and also for Gutenberg, and combine these counts as we see fit afterwards? |
Beta Was this translation helpful? Give feedback.
-
|
That is what I did. In order to get a "combined" bigram score, I multiplied the count for each pair in Gutenberg by (total leipzig)/(total gutenberg), ie scaling the whole corpus to be about the same, then simply added the counts for each bigram together. For your purposes, for the characters, I converted each char's count to a percentage of the total (for Leipzig and Gutenberg separately) , then took the average of the percentages. So effectively weighted the result as 50% Leipzig and 50% Gutenberg. |
Beta Was this translation helpful? Give feedback.
-
|
Sounds great! |
Beta Was this translation helpful? Give feedback.
-
|
Attached spreadsheet with bigrams, Leipzig and Gutenberg
spanish-bigrams-unicase-letters-leipzig-gutenberg.ods If you only want first and second last columns, export to csv, close, open csv, delete unwanted columns, save, import ... |
Beta Was this translation helpful? Give feedback.
-
|
You are awesome, @iandoug! This is exactly what I needed! |
Beta Was this translation helpful? Give feedback.
-
|
Char frequencies still coming. |
Beta Was this translation helpful? Give feedback.
-
|
I'm working with the assumption that the unicase letter bigram frequency file that I received is final, because this is how I generated the current layout for which I am conducting tests. |
Beta Was this translation helpful? Give feedback.
-
|
I don't know if you are referring to the fact that I'm still cleaning the "bad words" list. I don't think the count for very low frequency words would be statistically relevant for the monogram and bigram frequencies that Ian has given us already, for layout optimization purposes. We're scraping the bottom of the barrel for valid but actually pretty rare words here. But I do think that having a cleaner corpus and probably a good spanish dictionary of valid words can have some other uses, for instance to generate random lists of words for typing training, to validate further spanish corpora, other uses for KLA maybe... |
Beta Was this translation helpful? Give feedback.
-
|
That makes sense. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Hey @iandoug Gutemberg's corpus distorts symbol count by using them as special characters. For instance the underscore is used to indicate italicized words. https://www.gutenberg.org/attic/html_faq.html#can-i-submit-a-html-or-other-format-of-somebody-elses-text And that's why there are a gajillion underscores in Gutemberg. This, obviously, distorts its frequency, massively. It's true that nowadays underscore has this purpose, many apps like whatsapp, telegram or this very github use underscores for italics, and usually asterisks for bold. The question is: should we entertain this convention in order to create an optimal spanish layout? |
Beta Was this translation helpful? Give feedback.
-
|
Why do Spanish authors put so much in italics? :-) English books use italics and bold very seldom. Underscore is also used a lot in code. I suggest you keep it with dash, and just worry about dash frequency, which is probably also distorted by Gutenberg using -- for — |
Beta Was this translation helpful? Give feedback.
-
|
@iandoug -- It would be nice to wrap this up and put the relevant files and description up in osf.io / zenodo. If you don't have time, I would be happy to do so -- just point me to the most relevant files. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
Attached draft for review and comment. Probably still needs a bit of polish. The "file list" (section 5) is not final yet . Please ensure I credited everybody correctly. :-) @NickG13 Thanks, Ian Creating-a-corpus-and-chained-bigrams-for-Spanish-keyboard-development-and-evaluation-0.9.0.pdf |
Beta Was this translation helpful? Give feedback.
-
|
FANTASTIC, @iandoug! You have done a great job documenting the project, and people will find this very helpful! |
Beta Was this translation helpful? Give feedback.
-
|
Awesome work @iandoug ! What a great documentation, both to show that this layout has been thoroughly researched, and as a base for other optimized spanish layouts based on different paradigms. And thanks for the credits! Found a little typo. In "6. Acknowledgements" there's a missing space after the parenthesis and before "Arno" here:
Also the beginning of the PDF title shows up as:
Which looks like a different project altogether. Possibly a field unchanged from the document used as template. Congratulations again! 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Done. Hope all is correct :-) Report and corpus/analysis files: https://zenodo.org/record/5501914 all versions: https://doi.org/10.5281/zenodo.5501913 this version: https://doi.org/10.5281/zenodo.5501914 Suspect German will be next, but we need to add deadkey support to KLA ... Cheers, Ian |
Beta Was this translation helpful? Give feedback.
-
Yuch. Only saw that after uploading. Only shows up when viewing the PDF. Uploaded v1.0.1 to fix that. Is indeed from template... I need to get back to Giza :-) |
Beta Was this translation helpful? Give feedback.
-
|
Congratulations and great work, @iandoug !!! |
Beta Was this translation helpful? Give feedback.
-
Hi
Okay, the Leipzig corpus is better if not perfect.
Attached analysis of character frequency in Leipzig and Gutenberg, plus average. The data scientists can advise if that is the correct methodology :-)
It's sorted by character rather than count.
spanish-character-frequency-v1.ods
Final Leipzig corpus :
~/1web/spanish/leipzig $ wc spanish_leipzig-clean2.txt
5,480,692 lines, 265,281,891 words, 1,610,201,701 characters
Bigrams coming.
Beta Was this translation helpful? Give feedback.
All reactions