deploy: d7d9451

som-shahlab · Dec 12, 2023 · fca31fd · fca31fd
1 parent a195d9d
commit fca31fd
Showing 1 changed file with 3 additions and 3 deletions.
diff --git a/index.html b/index.html
@@ -4,8 +4,8 @@
 <button id=buttonColorMode class="btn btn-link mx-auto nav-link p-0 ms-lg-2 me-lg-1" type=button aria-label="Toggle theme"><svg data-bs-theme-value="dark" xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-moon" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentcolor" fill="none" stroke-linecap="round" stroke-linejoin="round"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M12 3c.132.0.263.0.393.0a7.5 7.5.0 007.92 12.446A9 9 0 1112 2.992z"/></svg><svg data-bs-theme-value="light" xmlns="http://www.w3.org/2000/svg" class="icon icon-tabler icon-tabler-sun" width="24" height="24" viewBox="0 0 24 24" stroke-width="2" stroke="currentcolor" fill="none" stroke-linecap="round" stroke-linejoin="round"><path stroke="none" d="M0 0h24v24H0z" fill="none"/><path d="M12 12m-4 0a4 4 0 108 0 4 4 0 10-8 0m-5 0h1m8-9v1m8 8h1m-9 8v1M5.6 5.6l.7.7m12.1-.7-.7.7m0 11.4.7.7m-12.1-.7-.7.7"/></svg></button></div></div></div></header></div><div class=modal id=searchModal tabindex=-1 aria-labelledby=searchModalLabel aria-hidden=true><div class="modal-dialog modal-dialog-scrollable modal-fullscreen-md-down"><div class=modal-content><div class=modal-header><h1 class="modal-title fs-5 visually-hidden" id=searchModalLabel>Search</h1><button type=button class="btn-close visually-hidden" data-bs-dismiss=modal aria-label=Close></button><div class="search-input flex-grow-1 d-none"><form id=search-form class=search-form action=# method=post accept-charset=utf-8 role=search><label for=query class=visually-hidden>Search</label><div class=d-flex><input type=search id=query name=query class="search-text form-control form-control-lg" placeholder=Search aria-label=Search maxlength=128 autocomplete=off>
 <button type=button class="btn btn-link text-decoration-none px-0 ms-3 d-md-none" data-bs-dismiss=modal aria-label=Close>Cancel</button></div></form></div></div><div class=modal-body><p class="search-loading status message d-none mt-3 text-center">Loading search index…</p><p class="search-no-recent message d-none mt-3 text-center">No recent searches</p><p class="search-no-results message d-none mt-3 text-center">No results for "<strong><span class=query-no-results>Query here</span></strong>"</p><div id=searchResults class=search-results></div><template><article class="search-result list-view"><div class="card my-3"><div class=card-body><header><h2 class="h5 title title-submitted mb-0"><a class="stretched-link text-decoration-none text-reset" href=#>Title here</a></h2><div class="submitted d-none"><time class=created-date>Date here</time></div></header><div class=content>Summary here</div></div></div></article></template></div><div class=modal-footer><ul class="list-inline me-auto d-none d-md-block"><li class=list-inline-item><kbd class=me-2><svg width="15" height="15" aria-label="Enter key" role="img"><g fill="none" stroke="currentcolor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.2"><path d="M12 3.53088v3c0 1-1 2-2 2H4m3 3-3-3 3-3"/></g></svg></kbd><span class=DocSearch-Label>to select</span></li><li class=list-inline-item><kbd class=me-2><svg width="15" height="15" aria-label="Arrow down" role="img"><g fill="none" stroke="currentcolor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.2"><path d="M7.5 3.5v8m3-3-3 3-3-3"/></g></svg></kbd><kbd class=me-2><svg width="15" height="15" aria-label="Arrow up" role="img"><g fill="none" stroke="currentcolor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.2"><path d="M7.5 11.5v-8m3 3-3-3-3 3"/></g></svg></kbd><span class=DocSearch-Label>to navigate</span></li><li class=list-inline-item><kbd class=me-2><svg width="15" height="15" aria-label="Escape key" role="img"><g fill="none" stroke="currentcolor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.2"><path d="M13.6167 8.936c-.1065.3583-.6883.962-1.4875.962-.7993.0-1.653-.9165-1.653-2.1258v-.5678c0-1.2548.7896-2.1016 1.653-2.1016s1.3601.4778 1.4875 1.0724M9 6c-.1352-.4735-.7506-.9219-1.46-.8972-.7092.0246-1.344.57-1.344 1.2166s.4198.8812 1.3445.9805C8.465 7.3992 8.968 7.9337 9 8.5s-.454 1.398-1.4595 1.398C6.6593 9.898 6 9 5.963 8.4851m-1.4748.5368c-.2635.5941-.8099.876-1.5443.876s-1.7073-.6248-1.7073-2.204v-.4603c0-1.0416.721-2.131 1.7073-2.131.9864.0 1.6425 1.031 1.5443 2.2492h-2.956"/></g></svg></kbd><span class=DocSearch-Label>to close</span></li></ul><p class=d-md-none>Search by <a class=text-decoration-none href=https://github.com/nextapps-de/flexsearch>FlexSearch</a></p></div></div></div></div><div class="wrap container-lg" role=document><div class=content><link rel=stylesheet href="https://fonts.googleapis.com/css?family=Montserrat:400,400i,500,500i,700,700i|Noto+Sans:400,400i,700,700i|Source+Code+Pro&amp;subset=latin-ext"><section class="section hero-subheader container-fluid mt-0 py-5 px-0 px-md-3"><div class="container px-1 px-md-3"><div class="row justify-content-center"><div class="col-lg-12 text-center"><div style=font-size:40px>👂 💉</div><h1 class=mt-0>EHRSHOT</h1></div><div class="col-lg-12 text-center"><p class=lead>An EHR Benchmark for Few-Shot Evaluation of Foundation Models</p></div></div><div class="row justify-content-center"><div class=text-center><a class="btn btn-primary btn-md px-4 mb-2" href=https://arxiv.org/abs/2307.02028 target=_blank role=button><svg style="margin-right:3px" xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentcolor" class="bi bi-file-text-fill" viewBox="0 0 16 16"><path d="M12 0H4A2 2 0 002 2v12a2 2 0 002 2h8a2 2 0 002-2V2a2 2 0 00-2-2zM5 4h6a.5.5.0 010 1H5a.5.5.0 010-1zm-.5 2.5A.5.5.0 015 6h6a.5.5.0 010 1H5a.5.5.0 01-.5-.5zM5 8h6a.5.5.0 010 1H5a.5.5.0 010-1zm0 2h3a.5.5.0 010 1H5a.5.5.0 010-1z"/></svg>Paper</a>
 <a class="btn btn-primary btn-md px-4 mb-2" href=https://github.com/som-shahlab/ehrshot-benchmark/ target=_blank role=button><svg style="margin-right:3px" xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentcolor" class="bi bi-github" viewBox="0 0 16 16"><path d="M8 0C3.58.0.0 3.58.0 8c0 3.54 2.29 6.53 5.47 7.59.4.07.55-.17.55-.38.0-.19-.01-.82-.01-1.49-2.01.37-2.53-.49-2.69-.94-.09-.23-.48-.94-.82-1.13-.28-.15-.68-.52-.01-.53.63-.01 1.08.58 1.23.82.72 1.21 1.87.87 2.33.66.07-.52.28-.87.51-1.07-1.78-.2-3.64-.89-3.64-3.95.0-.87.31-1.59.82-2.15-.08-.2-.36-1.02.08-2.12.0.0.67-.21 2.2.82.64-.18 1.32-.27 2-.27s1.36.09 2 .27c1.53-1.04 2.2-.82 2.2-.82.44 1.1.16 1.92.08 2.12.51.56.82 1.27.82 2.15.0 3.07-1.87 3.75-3.65 3.95.29.25.54.73.54 1.48.0 1.07-.01 1.93-.01 2.2.0.21.15.46.55.38A8.012 8.012.0 0016 8c0-4.42-3.58-8-8-8z"/></svg>GitHub</a>
-<a class="btn btn-primary btn-md px-4 mb-2" href=https://huggingface.co/StanfordShahLab/clmbr-t-base target=_blank role=button><svg xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentcolor" class="bi bi-robot" viewBox="0 0 16 16"><path d="M6 12.5a.5.5.0 01.5-.5h3a.5.5.0 010 1h-3a.5.5.0 01-.5-.5M3 8.062C3 6.76 4.235 5.765 5.53 5.886a26.58 26.58.0 004.94.0C11.765 5.765 13 6.76 13 8.062v1.157a.933.933.0 01-.765.935c-.845.147-2.34.346-4.235.346s-3.39-.2-4.235-.346A.933.933.0 013 9.219zm4.542-.827a.25.25.0 00-.217.068l-.92.9A24.767 24.767.0 014.534 8.02a.25.25.0 00-.068.495c.55.076 1.232.149 2.02.193a.25.25.0 00.189-.071l.754-.736.847 1.71a.25.25.0 00.404.062l.932-.97a25.286 25.286.0 001.922-.188.25.25.0 00-.068-.495c-.538.074-1.207.145-1.98.189a.25.25.0 00-.166.076l-.754.785-.842-1.7a.25.25.0 00-.182-.135z"/><path d="M8.5 1.866a1 1 0 10-1 0V3h-2A4.5 4.5.0 001 7.5V8A1 1 0 000 9v2a1 1 0 001 1v1a2 2 0 002 2h10a2 2 0 002-2v-1a1 1 0 001-1V9a1 1 0 00-1-1v-.5A4.5 4.5.0 0010.5 3h-2zM14 7.5V13a1 1 0 01-1 1H3a1 1 0 01-1-1V7.5A3.5 3.5.0 015.5 4h5A3.5 3.5.0 0114 7.5"/></svg>EHR Foundation Model</a>
-<a class="btn btn-secondary btn-md px-4 mb-2 disabled" role=button><svg style="margin-right:3px" xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentcolor" class="bi bi-database" viewBox="0 0 16 16"><path d="M4.318 2.687C5.234 2.271 6.536 2 8 2s2.766.27 3.682.687C12.644 3.125 13 3.627 13 4c0 .374-.356.875-1.318 1.313C10.766 5.729 9.464 6 8 6s-2.766-.27-3.682-.687C3.356 4.875 3 4.373 3 4c0-.374.356-.875 1.318-1.313zM13 5.698V7c0 .374-.356.875-1.318 1.313C10.766 8.729 9.464 9 8 9s-2.766-.27-3.682-.687C3.356 7.875 3 7.373 3 7V5.698c.271.202.58.378.904.525C4.978 6.711 6.427 7 8 7s3.022-.289 4.096-.777A4.92 4.92.0 0013 5.698zM14 4c0-1.007-.875-1.755-1.904-2.223C11.022 1.289 9.573 1 8 1s-3.022.289-4.096.777C2.875 2.245 2 2.993 2 4v9c0 1.007.875 1.755 1.904 2.223C4.978 15.71 6.427 16 8 16s3.022-.289 4.096-.777C13.125 14.755 14 14.007 14 13V4zm-1 4.698V10c0 .374-.356.875-1.318 1.313C10.766 11.729 9.464 12 8 12s-2.766-.27-3.682-.687C3.356 10.875 3 10.373 3 10V8.698c.271.202.58.378.904.525C4.978 9.71 6.427 10 8 10s3.022-.289 4.096-.777A4.92 4.92.0 0013 8.698zm0 3V13c0 .374-.356.875-1.318 1.313C10.766 14.729 9.464 15 8 15s-2.766-.27-3.682-.687C3.356 13.875 3 13.373 3 13v-1.302c.271.202.58.378.904.525C4.978 12.71 6.427 13 8 13s3.022-.289 4.096-.777c.324-.147.633-.323.904-.525z"/></svg>Dataset (TBD)</a></div></div><div class="row justify-content-center text-center mt-3 mb-4"><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">6,739</h2>patients</div><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">41.6 million</h2>clinical events</div><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">921,499</h2>visits</div><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">15</h2>prediction tasks</div></div><div style=text-align:justify;text-justify:inter-word class=mt-4>While the general machine learning (ML) community has benefited from public datasets,
+<a class="btn btn-primary btn-md px-4 mb-2" href=https://huggingface.co/StanfordShahLab/clmbr-t-base target=_blank role=button><svg style="margin-right:3px" xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentcolor" class="bi bi-robot" viewBox="0 0 16 16"><path d="M6 12.5a.5.5.0 01.5-.5h3a.5.5.0 010 1h-3a.5.5.0 01-.5-.5M3 8.062C3 6.76 4.235 5.765 5.53 5.886a26.58 26.58.0 004.94.0C11.765 5.765 13 6.76 13 8.062v1.157a.933.933.0 01-.765.935c-.845.147-2.34.346-4.235.346s-3.39-.2-4.235-.346A.933.933.0 013 9.219zm4.542-.827a.25.25.0 00-.217.068l-.92.9A24.767 24.767.0 014.534 8.02a.25.25.0 00-.068.495c.55.076 1.232.149 2.02.193a.25.25.0 00.189-.071l.754-.736.847 1.71a.25.25.0 00.404.062l.932-.97a25.286 25.286.0 001.922-.188.25.25.0 00-.068-.495c-.538.074-1.207.145-1.98.189a.25.25.0 00-.166.076l-.754.785-.842-1.7a.25.25.0 00-.182-.135z"/><path d="M8.5 1.866a1 1 0 10-1 0V3h-2A4.5 4.5.0 001 7.5V8A1 1 0 000 9v2a1 1 0 001 1v1a2 2 0 002 2h10a2 2 0 002-2v-1a1 1 0 001-1V9a1 1 0 00-1-1v-.5A4.5 4.5.0 0010.5 3h-2zM14 7.5V13a1 1 0 01-1 1H3a1 1 0 01-1-1V7.5A3.5 3.5.0 015.5 4h5A3.5 3.5.0 0114 7.5"/></svg>EHR Foundation Model</a>
+<a class="btn btn-secondary btn-md px-4 mb-2 disabled" href=https://redivis.com/datasets/6m7p-7yhq70029 target=_blank role=button><svg style="margin-right:3px" xmlns="http://www.w3.org/2000/svg" width="16" height="16" fill="currentcolor" class="bi bi-database" viewBox="0 0 16 16"><path d="M4.318 2.687C5.234 2.271 6.536 2 8 2s2.766.27 3.682.687C12.644 3.125 13 3.627 13 4c0 .374-.356.875-1.318 1.313C10.766 5.729 9.464 6 8 6s-2.766-.27-3.682-.687C3.356 4.875 3 4.373 3 4c0-.374.356-.875 1.318-1.313zM13 5.698V7c0 .374-.356.875-1.318 1.313C10.766 8.729 9.464 9 8 9s-2.766-.27-3.682-.687C3.356 7.875 3 7.373 3 7V5.698c.271.202.58.378.904.525C4.978 6.711 6.427 7 8 7s3.022-.289 4.096-.777A4.92 4.92.0 0013 5.698zM14 4c0-1.007-.875-1.755-1.904-2.223C11.022 1.289 9.573 1 8 1s-3.022.289-4.096.777C2.875 2.245 2 2.993 2 4v9c0 1.007.875 1.755 1.904 2.223C4.978 15.71 6.427 16 8 16s3.022-.289 4.096-.777C13.125 14.755 14 14.007 14 13V4zm-1 4.698V10c0 .374-.356.875-1.318 1.313C10.766 11.729 9.464 12 8 12s-2.766-.27-3.682-.687C3.356 10.875 3 10.373 3 10V8.698c.271.202.58.378.904.525C4.978 9.71 6.427 10 8 10s3.022-.289 4.096-.777A4.92 4.92.0 0013 8.698zm0 3V13c0 .374-.356.875-1.318 1.313C10.766 14.729 9.464 15 8 15s-2.766-.27-3.682-.687C3.356 13.875 3 13.373 3 13v-1.302c.271.202.58.378.904.525C4.978 12.71 6.427 13 8 13s3.022-.289 4.096-.777c.324-.147.633-.323.904-.525z"/></svg>Dataset</a></div></div><div class="row justify-content-center text-center mt-3 mb-4"><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">6,739</h2>patients</div><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">41.6 million</h2>clinical events</div><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">921,499</h2>visits</div><div class="col-6 col-md-4"><h2 class="mt-3 mb-1">15</h2>prediction tasks</div></div><div style=text-align:justify;text-justify:inter-word class=mt-4>While the general machine learning (ML) community has benefited from public datasets,
 tasks, and models, the progress of ML in healthcare has been hampered by a lack of such
 shared assets. The success of foundation models creates new challenges for healthcare ML
 by requiring access to shared pretrained models to validate performance benefits.<br><br>We help address these challenges through three contributions.<br><br><ol><li>We publish a new dataset, <strong>EHRSHOT</strong>, which contains de-identified structured data from
@@ -16,7 +16,7 @@
 ClinicalBERT) only work with unstructured text and cannot process the rich, structured data
 within an EHR. We provide an end-to-end pipeline for the community to validate and build
 upon its performance.</li><li>We define <strong>15 few-shot clinical prediction tasks,</strong> enabling evaluation of foundation models on benefits
-such as sample efficiency and task adaptation.</li></ol>Our <a href=https://huggingface.co/StanfordShahLab/clmbr-t-base>model is available at this link</a>, and we will update this website with a link to the dataset once it becomes available via a research data use agreement.
+such as sample efficiency and task adaptation.</li></ol>Our <a href=https://huggingface.co/StanfordShahLab/clmbr-t-base>model is available at this link</a>. The dataset is available [at this link via a research data use agreement](https://redivis.com/datasets/6m7p-7yhq70029).
 Code to reproduce our results is available <a href=https://github.com/som-shahlab/ehrshot-benchmark target=_blank>at this link</a>.</div><div class="row justify-content-center d-md-block" style=margin-top:100px><h2 class="text-center text-primary">Overview</h2><hr style="width:40%;margin:0 auto" class=mb-3><p class=text-center>We collect the structured data within the deidentified longitudinal EHRs of patients from Stanford Hospital.</p><div class="col-lg-12 mx-auto"><img src=https://som-shahlab.github.io/ehrshot-website/images/Figure_1.png class="border-0 mt-2"></div></div><div class="row justify-content-center d-md-block" style=margin-top:100px><h2 class="text-center text-primary">Comparison to Prior Work</h2><hr style="width:40%;margin:0 auto" class=mb-3><p>Most prior benchmarks are (1) limited to the ICU setting and (2) not tailored towards few-shot evaluation of pre-trained models.
 In contrast, EHRSHOT contains (1) the full breadth of longitudinal data that a health system would expect to have on the patients it treats
 and (2) a broad range of tasks designed to evaluate models' task adaptation and few-shot capabilities.</p><div class="col-lg-12 mx-auto"><img src=https://som-shahlab.github.io/ehrshot-website/images/comparison.png class="border-0 mt-2"></div></div><div class="row justify-content-center d-md-block" style=margin-top:100px><h2 class="text-center text-primary">Tasks</h2><hr style="width:40%;margin:0 auto" class=mb-3><p class=text-center>EHRSHOT includes 15 clinical classification tasks with canonical train/val/test splits, defined as follows.</p><div class="col-lg-12 mx-auto my-4" style=overflow-x:scroll><table class=my-0 style=font-size:14px><tr><th>Task</th><th>Type</th><th>Prediction Time</th><th>Time Horizon</th></tr><tr><td>Long Length of Stay</td><td>Binary</td><td>11:59pm on day of admission</td><td>Admission duration</td></tr><tr><td>30-day Readmission</td><td>Binary</td><td>11:59pm on day of discharge</td><td>30-days post discharge</td></tr><tr><td>ICU Transfer</td><td>Binary</td><td>11:59pm on day of admission</td><td>Admission duration</td></tr><tr><td>Thrombocytopenia</td><td>4-way Multiclass</td><td>Immediately before result is recorded</td><td>Next result</td></tr><tr><td>Hyperkalemia</td><td>4-way Multiclass</td><td>Immediately before result is recorded</td><td>Next result</td></tr><tr><td>Hypoglycemia</td><td>4-way Multiclass</td><td>Immediately before result is recorded</td><td>Next result</td></tr><tr><td>Hyponatremia</td><td>4-way Multiclass</td><td>Immediately before result is recorded</td><td>Next result</td></tr><tr><td>Anemia</td><td>4-way Multiclass</td><td>Immediately before result is recorded</td><td>Next result</td></tr><tr><td>Hypertension</td><td>Binary</td><td>11:59pm on day of discharge</td><td>1 year post-discharge</td></tr><tr><td>Hyperlipidemia</td><td>Binary</td><td>11:59pm on day of discharge</td><td>1 year post-discharge</td></tr><tr><td>Pancreatic Cancer</td><td>Binary</td><td>11:59pm on day of discharge</td><td>1 year post-discharge</td></tr><tr><td>Celiac</td><td>Binary</td><td>11:59pm on day of discharge</td><td>1 year post-discharge</td></tr><tr><td>Lupus</td><td>Binary</td><td>11:59pm on day of discharge</td><td>1 year post-discharge</td></tr><tr><td>Acute MI</td><td>Binary</td><td>11:59pm on day of discharge</td><td>1 year post-discharge</td></tr><tr><td>Chest X-Ray Findings</td><td>14-way Multilabel</td><td>24hrs before report is recorded</td><td>Next report</td></tr></table></div><p class=text-center>We include a graphical summary of the different task definitions below.</p><div class="col-lg-12 mx-auto"><img src=https://som-shahlab.github.io/ehrshot-website/images/Figure_2.png class="border-0 mt-2"></div></div><div class="row justify-content-center d-md-block" style=margin-top:100px><h2 class="text-center text-primary">Benchmarking Results</h2><hr style="width:40%;margin:0 auto" class=mb-3><p>We evaluate each baseline model in a few-shot setting. For each of the 15 benchmark tasks, we