CLOWDER and tests (#25)

* adding clowder class and tests * changing Clowder for clowder * Adding updateCatAndGetNextItem function Co-authored-by: Adam Richie-Halford <[email protected]> * eslint for unused -- used variables * clowder import * Add zetas for multiple cats to the corpus * Add TODO comments * Add util tests * Add documentation * Document and test utils * Start adding clowder tests * Add more clowder tests * Add documentation and randomlySelectUnvalidated parameter * Add more tests and a random seed * Reorganize files * Don't export abilityPrior * Update readme * adding missing tests to clowder * Import Cat, Clowder, and ClowderInput from index * Add src/stopping.ts * Add tests for stopping.ts * Add earlyStopping input to Clowder * prepareClowderCorpus * WIP: Add logicalOperation and tests * Use _isEmpty instead of null test in prepareClowderCorpus * Increase test coverage * Add comments on using early stopping in the Clowder class * solving TODOS and adding stopping tests * we don't need default * resolve some comments * correcting some tests * adding the unvalidated cat -- and test * Fix tests. Don't update ability estimate for the unvalidated Cat. Handle unvalidated remaining items separately * adding returnUndefinedOnExhaustion parameter and test * implementing suggestions * Separate the stopping classes so that they don't share the same input * updating cats for clowder * clowder changes based on letter implementation * addressing all lines of code for testing * adding documentation about early stopping * since we added only, we need to add catToSelect * solving adams comments * deleting for loop * adding stopping reason * adding more stoppingReasons to the tests * Update README.md * filterin NA from overall corpus * adding switching documentation * adding additional documentation * adding more clarification on readme * placing early stopping documentation in the right place * changing abilitiy for ability --------- Co-authored-by: Adam Richie-Halford <[email protected]> Co-authored-by: Adam Richie-Halford <[email protected]> Co-authored-by: AnyaWMa <[email protected]>
yeatmanlab · Feb 12, 2025 · 126f710 · 126f710
1 parent eae1d3a
commit 126f710
Show file tree

Hide file tree

Showing 16 changed files with 4,033 additions and 601 deletions.
diff --git a/README.md b/README.md
@@ -16,6 +16,8 @@ npm i @bdelab/jscat
 
 ## Usage
 
+For existing jsCAT users: to make your applications compatible to the updated jsCAT version, you will need to pass the stimuli in the following way:
+
 ```js
 // import jsCAT
 import { Cat, normal } from '@bdelab/jscat';
@@ -26,7 +28,15 @@ const currentPrior = normal();
 // create a Cat object 
 const cat = new CAT({method: 'MLE', itemSelect: 'MFI', nStartItems: 0, theta: 0, minTheta: -6, maxTheta: 6, prior: currentPrior})
 
-// update the abilitiy estimate by adding test items 
+// option 1 to input stimuli:
+const zeta = {[{discrimination: 1, difficulty: 0, guessing: 0, slipping: 1}, {discrimination: 1, difficulty: 0.5, guessing: 0, slipping: 1}]}
+
+// option 2 to input stimuli:
+const zeta = {[{a: 1, b: 0, c: 0, d: 1}, {a: 1, b: 0.5, c: 0, d: 1}]}
+
+const answer = {[1, 0]}
+
+// update the ability estimate by adding test items 
 cat.updateAbilityEstimate(zeta, answer);
 
 const currentTheta = cat.theta;
@@ -37,29 +47,231 @@ const numItems = cat.nItems;
 
 // find the next available item from an input array of stimuli based on a selection method
 
-const stimuli = [{difficulty: -3, item: 'item1'}, {difficulty: -2,  item: 'item2'}];
+> **Note:** For existing jsCAT users: To make your applications compatible with the updated jsCAT version, you will need to pass the stimuli in the following way:
+
+const stimuli = [{  discrimination: 1, difficulty: -2, guessing: 0, slipping: 1, item = "item1" },{ discrimination: 1, difficulty: 3, guessing: 0, slipping: 1, item = "item2" }];
 
 const nextItem = cat.findNextItem(stimuli, 'MFI');
 ```
 
-## Validations
+
+## Validation
+
 ### Validation of theta estimate and theta standard error
+
 Reference software: mirt (Chalmers, 2012)
 ![img.png](validation/plots/jsCAT_validation_1.png)
 
 ### Validation of MFI algorithm 
+
 Reference software: catR (Magis et al., 2017)
 ![img_1.png](validation/plots/jsCAT_validation_2.png)
 
+# Clowder Usage Guide
+
+The `Clowder` class is a powerful tool for managing multiple `Cat` instances and handling stimuli corpora in adaptive testing scenarios. This guide provides an overview of integrating `Clowder` into your application, with examples and explanations for key features.
+
+---
+
+## Key Changes from Single `Cat` to `Clowder`
+
+### Why Use Clowder?
+
+- **Multi-CAT Support**: Manage multiple `Cat` instances simultaneously.
+- **Centralized Corpus Management**: Handle validated and unvalidated items across Cats.
+- **Advanced Trial Management**: Dynamically update Cats and retrieve stimuli based on configurable rules.
+- **Early Stopping Mechanisms**: Optimize testing by integrating conditions to stop trials automatically.
+
+---
+
+## Transitioning to Clowder
+
+### 1. Replacing Single `Cat` Usage
+
+#### Single `Cat` Example:
+```typescript
+const cat = new Cat({ method: 'MLE', theta: 0.5 });
+const nextItem = cat.findNextItem(stimuli);
+```
+
+#### Clowder Equivalent:
+```typescript
+const clowder = new Clowder({
+  cats: { cat1: { method: 'MLE', theta: 0.5 } },
+  corpus: stimuli,
+});
+const nextItem = clowder.updateCatAndGetNextItem({
+  catToSelect: 'cat1',
+});
+```
+
+---
+
+### 2. Using a Corpus with Multi-Zeta Stimuli
+
+The `Clowder` corpus supports multi-zeta stimuli, allowing each stimulus to define parameters for multiple Cats. Use the following tools to prepare the corpus:
+
+#### Fill Default Zeta Parameters:
+
+```typescript
+import { fillZetaDefaults } from './corpus';
+
+const filledStimuli = stimuli.map((stim) => fillZetaDefaults(stim));
+
+```
+
+**What is `fillZetaDefaults`?**
+The function `fillZetaDefaults` ensures that each stimulus in the corpus has Zeta parameters defined. If any parameters are missing, it fills them with the default Zeta values.
+
+The default values are:
+
+```typescript
+export const defaultZeta = (desiredFormat: 'symbolic' | 'semantic' = 'symbolic'): Zeta => {
+  const defaultZeta: Zeta = {
+    a: 1,
+    b: 0,
+    c: 0,
+    d: 1,
+  };
+
+  return convertZeta(defaultZeta, desiredFormat);
+};
+
+```
+- If desiredFormat is not specified, it defaults to 'symbolic'.
+- This ensures consistency across different stimuli and prevents errors from missing Zeta parameters.
+- You can pass 'semantic' as an argument to convert the default Zeta values into a different representation.
+
+#### Validate the Corpus:
+```typescript
+import { checkNoDuplicateCatNames } from './corpus';
+checkNoDuplicateCatNames(corpus);
+```
+
+#### Filter Stimuli for a Specific Cat:
+```typescript
+import { filterItemsByCatParameterAvailability } from './corpus';
+const { available, missing } = filterItemsByCatParameterAvailability(corpus, 'cat1');
+```
+
+---
+
+### 3. Adding Early Stopping
+
+Integrate early stopping mechanisms to optimize the testing process.
+
+#### Example: Stop After N Items
+```typescript
+import { StopAfterNItems } from './stopping';
+
+const earlyStopping = new StopAfterNItems({
+  requiredItems: { cat1: 2 },
+});
+
+const clowder = new Clowder({
+  cats: { cat1: { method: 'MLE', theta: 0.5 } },
+  corpus: stimuli,
+  earlyStopping: earlyStopping,
+});
+```
+
+## Early Stopping Criteria Combinations
+
+To clarify the available combinations for early stopping, here’s a breakdown of the options you can use:
+
+### 1. Logical Operations
+
+You can combine multiple stopping criteria using one of the following logical operations:
+
+- **`and`**: All conditions need to be met to trigger early stopping.
+- **`or`**: Any one condition being met will trigger early stopping.
+- **`only`**: Only a specific condition is considered (you need to specify the cat to evaluate).
+
+### 2. Stopping Criteria Classes
+
+There are different types of stopping criteria you can configure:
+
+- **`StopAfterNItems`**: Stops the process after a specified number of items.
+- **`StopOnSEMeasurementPlateau`**: Stops if the standard error (SE) of measurement remains stable (within a defined tolerance) for a specified number of items.
+- **`StopIfSEMeasurementBelowThreshold`**: Stops if the SE measurement drops below a set threshold.
+
+### How Combinations Work
+
+You can mix and match these criteria with different logical operations, giving you a range of configurations for early stopping. For example:
+
+- Using **`and`** with both `StopAfterNItems` and `StopIfSEMeasurementBelowThreshold` means stopping will only occur if both conditions are satisfied.
+- Using **`or`** with `StopOnSEMeasurementPlateau` and `StopAfterNItems` allows early stopping if either condition is met.
+
+---
+
+## Clowder Example
+
+Here’s a complete example demonstrating how to configure and use `Clowder`:
+
+```typescript
+import { Clowder } from './clowder';
+import { createMultiZetaStimulus, createZetaCatMap } from './utils';
+import { StopAfterNItems } from './stopping';
+
+// Define the Cats
+const catConfigs = {
+  cat1: { method: 'MLE', theta: 0.5 }, // Cat1 uses Maximum Likelihood Estimation
+  cat2: { method: 'EAP', theta: -1.0 }, // Cat2 uses Expected A Posteriori
+};
+
+// Define the corpus
+const corpus = [
+  createMultiZetaStimulus('item1', [
+    createZetaCatMap(['cat1'], { a: 1, b: 0.5, c: 0.2, d: 0.8 }),
+    createZetaCatMap(['cat2'], { a: 2, b: 0.7, c: 0.3, d: 0.9 }),
+  ]),
+  createMultiZetaStimulus('item2', [createZetaCatMap(['cat1'], { a: 1.5, b: 0.4, c: 0.1, d: 0.85 })]),
+  createMultiZetaStimulus('item3', [createZetaCatMap(['cat2'], { a: 2.5, b: 0.6, c: 0.25, d: 0.95 })]),
+  createMultiZetaStimulus('item4', []), // Unvalidated item
+];
+
+// Optional: Add an early stopping strategy
+const earlyStopping = new StopAfterNItems({
+  requiredItems: { cat1: 2, cat2: 2 },
+});
+
+// Initialize the Clowder
+const clowder = new Clowder({
+  cats: catConfigs,
+  corpus: corpus,
+  earlyStopping: earlyStopping,
+});
+
+// Running Trials
+const nextItem = clowder.updateCatAndGetNextItem({
+  catToSelect: 'cat1',
+  catsToUpdate: ['cat1', 'cat2'], // Update responses for both Cats
+  items: [clowder.corpus[0]], // Previously seen item
+  answers: [1], // Response for the previously seen item
+});
+
+console.log('Next item to present:', nextItem);
+
+// Check stopping condition
+if (clowder.earlyStopping?.earlyStop) {
+  console.log('Early stopping triggered:', clowder.stoppingReason);
+}
+```
+
+---
+
+By integrating `Clowder`, your application can efficiently manage adaptive testing scenarios with robust trial and stimuli handling, multi-CAT configurations, and stopping conditions to ensure optimal performance.
 
 ## References
-Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software.
 
-Magis, D., & Barrada, J. R. (2017). Computerized adaptive testing with R: Recent updates of the package catR. Journal of Statistical Software, 76, 1-19.
+- Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software.
+
+- Magis, D., & Barrada, J. R. (2017). Computerized adaptive testing with R: Recent updates of the package catR. Journal of Statistical Software, 76, 1-19.
 
-Lucas Duailibe, irt-js, (2019), GitHub repository, https://github.com/geekie/irt-js
+- Lucas Duailibe, irt-js, (2019), GitHub repository, https://github.com/geekie/irt-js
 
 ## License
+
 jsCAT is distributed under the [ISC license](LICENSE).
 
 ## Contributors
@@ -78,4 +290,4 @@ Ma, W. A., Richie-Halford, A., Burkhardt, A. K., Kanopka, K., Chou, C., Domingue
   pages={1--17},
   year={2025},
   publisher={Springer}
-}
+}