From Whispers to Voices: A "Community-In-The-Loop" Proposal for Model Distillation and Language Preservation
A working model of the Stoney Nakoda language has been developed and is now available for community-in-the-loop testing in 2025:
- Model App: Stoney Language Model App
- Training Data: StoneyNakoda Training Dataset
Any First Nations community seeking to apply this approach to their own language is warmly invited to reach out.
By following this code, you can build a model for any low-resource language. The starting dictionary size should be ~8,000 words.
- New Years Day, Canadian Rockies, 2025
- Understanding How AI Learns Stoney Words Using Cosine Similarity
- Project Architecture
- Detailed Project Structure
- Core Components
- Comprehensive Setup Instructions
- Detailed Usage Pipeline
- Advanced Model Configuration
- Comprehensive Data Formats
- Development Guidelines
- Contributing
- License
- Acknowledgments
- The Community-in-the-Loop Revolution
New Years Day, 2025
In my office, there is a murder; a map of one, at least.
George Mercer Dawson explored the Bow Valley in the late 1800s as a geologist but also as a linguist, noting language on the British Columbia side. His map, though richly colored, stands like a tombstone over the Bow Valley where the Stoney people lived because he made no notes on their language and simply noted the people as "recent immigrants". Much of this work was based on the notes of W. Faser Tolmie and was published after his death.
What is very obvious from the linguistic patterns among the Haida, Tshimsia, Thlinkit, Kwakiool and Kawitshin dialects nearby is that languages blend like "linguistic DNA," and machine learning could help trace faint threads of lost speech to their roots. Where some see isolation as a curse, in the age of AI, Stoney's isolation turns out to be its strength.
For about two years, I thought about the size of the vector space that would be needed to get a model to self-train on a set of 100% indigenous data, and how that model could refine its grasp of the broader Stoney Language. This is now publicly and freely available.
Two key releases influenced my thinking of what was possible:
Both gave me the motivation to build what's presented here. The true innovation here lies in how communities can narratively correct the initially flawed response (about 10% of the time, the model works every time.) then that feeback be passed seamleslly back into the fine-tuning process. The textbooks that the Stoney community created—intended as educational tools—became perfect concept of a model prompts, each chapter or word offering pure indigenous data devoid of external weights or biases to the fine-tuning process.
Early in 2023, I found an original, unpublished sketch by James Hector likely drawn in the summer of 1858 or 1859 along the Bow River in Southern Alberta:
Finding this, and already aware of George Mercer Dawson's work on First Nation's language on the British Columbia side, I was inspired to put the effort in and build a working model of the language and implement the Community-In-The-Loop distillation method.
This sketch shifted my thinking from considering the "Stoney People" to this "Stoney Woman" who saw these same mountains and rivers I see everyday, yet who had a very different way to think about and communicate to the world around her. The Community-in-the-Loop model distillation will quickly converge this initial model toward fluencey. I suspect this will require the community to correct about 80,000 question and answer pairs and would cost less than $800 in OpenAI computing power. Recent releases by Google and the Chinese Lab DeepSeek, could effectively reduce the cost to zero.
I think what this project has left me considering most is that a century from now, strangers will live in all our homes and most of what we worry about today will not matter. But we can honor "Stoney Woman" by making sure her language endures, forging a living record in an age of AI. Incredibly, this tool will work with any first nations language, as long as there is a starting dictionary of about 8,000 words.
I am freely available to help any First Nation in Canada.
Word Embeddings: Mapping Words in Space Word embeddings are like placing words in a high-dimensional map, where similar words are positioned closer together. For example, "strawberry," "orange," and "cherry" might form a cluster because they are fruits, while "laptop," "Microsoft," and "Android" might cluster elsewhere as tech-related terms. Each axis in this space represents a characteristic of the words, such as their context or meaning.
Context Shapes Meaning A word's position in this space isn't fixed—it shifts based on context. For instance, the word "apple" could mean a fruit or the tech brand, depending on its surrounding words, like "buy" (tech) or "tree" (fruit). This dynamic placement captures the nuances of meaning.
Cosine Similarity: Measuring Relationships Cosine similarity quantifies how similar two words are by measuring the angle between their vectors in the embedding space:
- Similar words have vectors pointing in nearly the same direction (cosine similarity close to 1)
- Unrelated words have vectors at a right angle (cosine similarity near 0)
- Opposite meanings have vectors pointing in opposite directions (cosine similarity close to -1)
- For example, "cherry" and "orange" might have a similarity of 0.97, while "cherry" and "laptop" might score 0.24
How AI Learns Stoney Words
-
Stoney Dictionary as a Starting Point: The AI begins with a structured dictionary of Stoney words, including translations, categories, pronunciations, and cultural context.
-
Community Feedback for Learning: The AI makes initial translations, which are often incorrect. Stoney speakers provide corrections, enriched with cultural context, stories, and humor. This feedback helps refine the AI's understanding.
The Role of Cosine Similarity in AI Learning
- The AI uses word embeddings to group Stoney words based on their meaning. For example, it determines whether a word belongs to a category like "fruit," "animal," or "spiritual."
- Community corrections and cosine similarity guide the AI in repositioning words closer to their accurate groupings in the embedding space.
Iterative Refinement Through repeated feedback and fine-tuning, the AI improves its ability to place Stoney words correctly, not just individually but in the context of sentences and paragraphs. Over time, it develops a detailed, dynamic map of the Stoney language, with words clustered according to their community-informed meanings and uses.
Although this is not cosine similarity, you can see the relationships among words can concepts in Stoney as I have mapped them here: https://atlas.nomic.ai/data/harleycoops/stoney-nakoda-language-synthetic/map/5c87caaf-6be0-4546-9e83-826569070b24#nqlL
This code forms a complete pipeline for training and deploying a Stoney model. It is fully functional—but not correct 100% of the time—and is designed to improve through Community-In-The-Loop feedback. Access the model here:
Stoney Language Model App
- Data Ingestion Layer
- Processing Pipeline (Q&A generation, augmentation, conversion)
- Model Training Framework (fine-tuning, hyperparameters, monitoring)
- Inference Interface (API endpoint, response formatting, error handling)
- Raw dictionary data → Data Ingestion
- Processed data → Q&A Generation
- Generated Q&A pairs → Training Data Preparation
- Prepared data → Model Fine-tuning
- Fine-tuned model → Inference Interface
PUBLICRELEASE/
├── OpenAIFineTune/ # OpenAI fine-tuning files
│ ├── stoney_train.jsonl # Training dataset
│ └── stoney_valid.jsonl # Validation dataset
├── checkpoints/ # Model checkpoints
├── .env.example # Env variables example
├── requirements.txt # Python dependencies
├── english_dictionary.jsonl
├── stoney_dictionary.jsonl
└── bilingual_training_set.jsonl
-
bilingual_qa_generator.py
Generates Q&A pairs from dictionaries, using advanced language generation. -
convert_data_format.py
Supports multiple data formats; validates and enforces schemas. -
finetunesetup.py
Splits data (80/20) with stratified sampling and prepares files.
openai_finetune.py
Handles fine-tuning, error handling, checkpointing, and logging.
- Python 3.8+
- 8GB+ RAM (16GB recommended)
- 10GB free disk space
- Stable internet connection
# Clone the repository
git clone [repository-url]
cd PUBLICRELEASE
# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Copy example environment file
cp .env.example .env
# Provide OPENAI_API_KEY, GOOGLE_API_KEY in .env
python initialize.py
python bilingual_qa_generator.py
- Processes
english_dictionary.jsonl&stoney_dictionary.jsonl - Produces
bilingual_training_set.jsonl
python finetunesetup.py
- Converts Q&A to OpenAI format
- Outputs
OpenAIFineTune/stoney_train.jsonl&stoney_valid.jsonl
python openai_finetune.py
- Uploads files to OpenAI
- Monitors fine-tuning progress
- Implements checkpointing & logs
- Default:
gpt-4o-2024-08-06 - Alternative:
gpt-3.5-turbo .env:OPENAI_MODEL
- Default:
gemini-2.0-exp .env:GEMINI_MODEL
- LR:
1e-5 - Batch size:
32 - Epochs:
3 - Context window:
4096
{
"english_word": "example",
"stoney_versions": [
{
"word": "...",
"grammatical_classification": "...",
"meaning": "..."
}
]
}
{
"question": "How do you say X in Stoney?",
"answer": "The Stoney word for X is...",
"source_language": "english",
"generated_at": "timestamp"
}
{
"messages": [
{"role": "system", "content": "You are a bilingual Stoney-English assistant..."},
{"role": "user", "content": "question"},
{"role": "assistant", "content": "answer"}
]
}
- Style: PEP 8, type hints, docstrings, consistent naming
- Testing: Unit tests, integration tests, CI, coverage
- Documentation: Inline comments, usage examples, troubleshooting
- Fork, branch, implement changes, test
- Submit a pull request
Code Review
- Clear commits, small changes, documentation, test coverage
This project aims to preserve, refine, and resurrect endangered languages via AI fine-tuning and model distillation. Minimal lexical data can evolve into a culturally rich digital speaker of Stoney Nakoda. This subverts assumptions that massive datasets are necessary, instead emphasizing:
- Iterative improvement with community feedback
- Narrative corrections (cultural context over simple dictionary entries)
- Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning
Community-in-the-Loop Model Distillation:
- Start with a small dictionary/text set.
- Prompt an initial model.
- Let the community correct errors with storytelling and context, not just words.
- LoRA-based fine-tuning absorbs these narrative corrections.
- The model evolves iteratively, guided by cultural custodians.
- Intentional Errors: Poke the model with tough or context-specific queries.
- Narrative Corrections: Rich cultural commentary instead of bare "right vs. wrong."
- Distillation Triplets: (Prompt, Disallowed Reply, Narrative Reply).
- Iterative Improvement: If the model stumbles, revert and add more context.
LoRA attaches small, low-rank matrices to the base model. This dramatically reduces compute and speeds up retraining:
- Efficiency: Fraction of resources required vs. full retraining
- Focused Updates: Capturing the "essence" of new knowledge
- Rapid Iterations: Frequent refinement without heavy overhead
If W0\mathbf{W}_0 is the base weight matrix, LoRA introduces ΔW=AB\Delta \mathbf{W} = \mathbf{A}\mathbf{B} with A∈Rd×r\mathbf{A} \in \mathbb{R}^{d \times r} and B∈Rr×k\mathbf{B} \in \mathbb{R}^{r \times k}, where r≪min(d,k)r \ll \min(d,k). Loss functions track both linguistic and cultural accuracy (e.g., a "Cultural Authenticity Score").
graph TD
A[Initial Model] --> B[Generate Response]
B --> C{Correct?}
C -->|No| D[Community Correction]
D --> E[Create Distillation Triplet]
E --> F[LoRA Fine-Tuning]
F --> A
C -->|Yes| G[Validation]
Every correction preserves cultural norms—idioms, humor, oral traditions—and ensures the community wields control over the AI's "mindset."
A 10,000-word Stoney Nakoda dictionary and community textbooks serve as seeds. Community feedback enriches this data over time, weaving historical memory into the model.
From a tiny dictionary to an AI that:
- Understands context (formal/informal usage)
- Integrates cultural references (stories, metaphors)
- Remembers history (ancestors, ceremonies, seasonal events)
- Forward Progress: Keep the new checkpoint if improved.
- Reversion: If degraded, roll back and increase context in corrections.
- Convergence: Repeat until stable authenticity and fluency metrics are met.
- Prompt: "How to say 'taste slightly with the tip of your tongue' in Stoney?"
- Model's Flawed Reply: "
supthîyach" (incorrect). - Community Correction: Shares the correct phrase plus a story from childhood.
- Distillation Triplet: (Prompt, Disallowed, Narrative).
- LoRA Fine-Tuning: Model adjusts swiftly.
- Re-Evaluation: Answers improve in subsequent queries.
- Cultural Authenticity Score (CAS)
- Linguistic Fluency (perplexity, cross-entropy)
- Validation Loops (watch for regressions, revert if needed)
- Oral Histories: Model retells century-old stories.
- Seasonal Knowledge: Terms tied to ceremonies and ecological cycles.
- Dialects/Accents: Respecting sub-regional differences.
- Educational Tools: Interactive AI for language learning.
- Ethical AI: Centered on consent, community governance, cultural integrity.
- CAS: Cultural Authenticity Score
- Distillation Triplet: (Prompt, Flawed Reply, Narrative Reply)
- LoRA: Low-Rank Adaptation
- Community-in-the-Loop: Paradigm of continuous human-guided refinement