Merge pull request #85 from VisLab/develop

Updating the READMEs and adding a LICENSE

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 HED Standard Working Group
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,14 +1,49 @@
 
 # HED-examples
+This repository contains user supporting code and documentation
+for using the Hierarchical Event Descriptor (HED) system for
+annotating, summarizing, and analyzing data.
+The repository is organized into three subdirectories:
 
-This repository contains a set of 
-[HED-annotated](https://hed-specification.readthedocs.io/en/latest/index.html)
-datasets in [BIDS-compatible](https://bids.neuroimaging.io/) format. 
-See [datasets/README.md](./datasets/README.md) for details.
+The [**datasets**](https://github.com/hed-standard/hed-examples/tree/main/datasets)
+subdirectory contains datasets for testing various aspects of HED.
+These datasets have stubs for actual imaging data in order to reduce their size.
+Most of these datasets have complete versions available on 
+[**openNeuro**](https://openneuro.org/).
+See [**datasets/README.md**](./datasets/README.md) for details.
 
+The [**hedcode**](https://github.com/hed-standard/hed-examples/tree/main/hedcode)
+subdirectory contains MATLAB scripts, Python Jupyter Notebooks,
+and Python scripts with direct calls to HedTools.
 The repository also contains example code in python and matlab. 
-See [hedcode/README.md](./hedcode/README.md) for details.
-These datasets can be useful for:
+See [**hedcode/README.md**](./hedcode/README.md) for details.
+The Python scripts and notebooks require the installation of
+`hedtools` whose [**installation**](./README.md#installation-of-hedtools)
+is described below.
 
-The repository also will host tutorials and instructional videos,
-which are in the process of being developed.
+The [**docs**](https://github.com/hed-standard/hed-examples/tree/main/docs)
+subdirectory contains the main documentation for this and other HED resources.
+The [**HED GitHub organization repository**](https://github.com/hed-standard/)
+gathers the HED supporting resources, all of which are open source.
+
+
+### Installation of hedtools
+
+The most of the Python-related resources in this repository
+require the installation of the HEDTools Python module, which can be
+installed using `pip` or directly from its GitHub repository as follows:
+
+To use `pip` to install `hedtools` from PyPI:
+
+   ```
+       pip install hedtools
+   ```
+
+To install directly from the 
+[GitHub](https://github.com/hed-standard/hed-python) repository:
+
+   ```
+       pip install git+https://github.com/hed-standard/hed-python/@master
+   ```
+
+HEDTools require python 3.7 or greater.

datasets/README.md

@@ -28,10 +28,10 @@ The suffix modifier indicates what this dataset is designed to test.
 | eeg_ds003654s_hed_library |  Wakeman-Henson face processing data.<br>HED tags using libraries.  |
 | eeg_ds003654s_hed_longform |  Wakeman-Henson face processing data.<br>HED tags in long form with definitions. |
 | eeg_ds003654s_hed_longform |  Wakeman-Henson face processing data.<br>HED tags in long form with definitions. |
-| eeg_ds004105s_hed | BCIT Driving with auditory cueing data.<br> Part of a test data corpus for BIDS-MEGA testing. |
+| eeg_ds004105s_hed | BCIT Driving with auditory cueing data.<br>Part of a test data corpus for BIDS-MEGA testing. |
 | eeg_ds004106s_hed | BCIT Advanced guard duty data.<br> Part of a test data corpus for BIDS-MEGA testing. |
-| eeg_ds004117s_hed_sternberg | Sternberg working memory task.<br>. Chosen as a replication study for (EEGManyLabs)[https://osf.io/yb3pq/].|
-| fmri_soccer21_hed | HED tags using a single column |
+| eeg_ds004117s_hed_sternberg | Sternberg working memory task.<br>Chosen as a replication study for [**EEGManyLabs**](https://osf.io/yb3pq/).|
+| fmri_soccer21_hed | HED tags using a single column.<br>Used for fMRI processing examples. |
 
 
 ### Validating examples using the BIDS validator

docs/requirements.txt

@@ -5,6 +5,7 @@ defusedxml==0.7.1
 et-xmlfile==1.1.0
 Flask==2.1.2
 Flask-WTF==1.0.1
+hedtools >= 0.1.0
 inflect>=5.5.2
 itsdangerous==2.1.2
 jdcal==1.4.1

hedcode/README.md

@@ -1,4 +1,4 @@
-# HED examples
+# HED code examples
 
 This repository contains user supporting code and documentation
 for using the Hierarchical Event Descriptor (HED) system for

hedcode/jupyter_notebooks/README.md

@@ -1,45 +1,36 @@
-## Jupyter notebooks to demo HED tagging
+### Jupyter notebooks to demo HED processing with BIDS
 
-These notebooks demonstrate the HED processing using the HED tools.
+The Jupyter notebooks in this directory are useful for annotating,
+validating, summarizing, and analyzing your BIDS datasets.
 
+**Table 1:** Useful Jupyter notebooks for processing BIDS datasets.
 
-### BIDS processing notebooks
-
-## Useful notebooks for processing BIDS datasets
-
-The Jupyter notebooks in this directory are useful for preparing your BIDS dataset. 
-
-**Table 1:** Useful Jupyter notebooks for preparing BIDS datasets.
-
-|Script                    | Purpose                            | 
+|Notebooks                 | Purpose                            | 
 | ------------------------ | ---------------------------------- | 
-| `bids_generate_sidecar`  | Creates a JSON sidecar based on all the event files in a dataset. |
-| `bids_merge_sidecar`    | Merges a spreadsheet version of a sidecar into a JSON sidecar. |
-| `bids_sidecar_to_spreadsheet` | Converts the HED portion of a JSON sidecar into a 4-column spreadsheet. |
-| `bids_validate_dataset`   | Validates the HED annotations in a BIDS dataset. |  
-| `bids_validate_dataset_with_libraries`   | Demonstrates use of HED libraries in validation. |  
-
-
-### Event file restructuring notebooks
+| [`bids_generate_sidecar`](https://github.com/hed-standard/hed-examples/blob/main/hedcode/jupyter_notebooks/bids_generate_sidecar.ipynb)  | Creates a JSON sidecar based on all the event files in a dataset. |
+| [`bids_merge_sidecar`](https://github.com/hed-standard/hed-examples/blob/main/hedcode/jupyter_notebooks/bids_merge_sidecar.ipynb)    | Merges a spreadsheet version of a sidecar into a JSON sidecar. |
+| [`bids_sidecar_to_spreadsheet`](https://github.com/hed-standard/hed-examples/blob/main/hedcode/jupyter_notebooks/bids_sidecar_to_spreadsheet.ipynb) | Converts the HED portion of a JSON sidecar into a 4-column spreadsheet. |
+| [`bids_summarize_events`](https://github.com/hed-standard/hed-examples/blob/main/hedcode/jupyter_notebooks/bids_summarize_events.ipynb) | Summarizes the contents of the event files, including value counts.
+| [`bids_validate_dataset`](https://github.com/hed-standard/hed-examples/blob/main/hedcode/jupyter_notebooks/bids_validate_dataset.ipynb)   | Validates the HED annotations in a BIDS dataset. |  
+| [`bids_validate_dataset_with_libraries`](https://github.com/hed-standard/hed-examples/blob/main/hedcode/jupyter_notebooks/bids_validate_dataset_with_libraries.ipynb)   | Demonstrates use of HED libraries in validation. |  
 
-**This section is out of date and will be updated soon.**
-The demo scripts use data in `../data/sternberg` and in the reduced
-attention shift dataset: 
+These notebooks require HEDTOOLS, which can be installed using `pip` or directly.
 
-[https://github.com/hed-standard/hed-examples/data/eeg_ds0028932](https://github.com/hed-standard/hed-examples/data/eeg_ds0028932).
+To use `pip` to install `hedtools` from PyPI:
 
-The notebooks are designed to be executed in the following order:  
+   ```
+       pip install hedtools
+   ```
 
-1. `summarize_events.ipynb` gather all of the unique values in the columns of
-all of the events files in a BIDS dataset.  
+To install directly from the 
+[GitHub](https://github.com/hed-standard/hed-python) repository:
 
-2. `create_template.ipynb`  gathers all of the unique combinations of values in
-a specified group of columns (the key columns) and creates a template file
-for you to specify the mapping between each unique key and values in target columns.
-This is the **event design** that must be filled in by the user.  
+   ```
+       pip install git+https://github.com/hed-standard/hed-python/@master
+   ```
 
-3. `remap_events.ipynb` creates new event files using the template from the previous
-step to remap columns.
+HEDTools require python 3.7 or greater.
 
-4. `tag_columns.ipynb` demonstrates how to create a list of the unique
-values in the specified columns in a flattened form so that they can be tagged.
+A related GitHub repository
+[**hed-curation**](https://github.com/hed-standard/hed-curation)
+is holds code related to curation.

hedcode/jupyter_notebooks/bids_validate_dataset.ipynb

@@ -53,8 +53,8 @@
      "name": "stdout",
      "output_type": "stream",
      "text": [
-      "Using HEDTOOLS version: {'date': '2022-06-20T09:05:48-0500', 'dirty': True, 'error': None, 'full-revisionid': '51caed3d948e6015639caa49c08dd15b00267f4e', 'version': '0+untagged.1190.g51caed3.dirty'}\n",
-      "HED Examples version: {'version': '0+untagged.230.gda8e87e.dirty', 'full-revisionid': 'da8e87e6b243ca316b5f01dd807651bd486df8f9', 'dirty': True, 'error': None, 'date': '2022-06-19T12:30:47-0500'}\n",
+      "Using HEDTOOLS version: {'date': '2022-06-20T14:40:24-0500', 'dirty': False, 'error': None, 'full-revisionid': 'c4ecd1834cd31a05ebad3e97dc57e537550da044', 'version': '0.1.0'}\n",
+      "HED Examples version: {'version': '0+untagged.233.ge70e761.dirty', 'full-revisionid': 'e70e761ea596de4fbebe926e1274ec64d85db4f1', 'dirty': True, 'error': None, 'date': '2022-06-20T11:11:00-0500'}\n",
       "BIDS path is: ../../datasets/eeg_ds003654s_hed\n",
       "No HED validation errors\n",
       "BIDS path is: ../../datasets/eeg_ds003654s_hed_column\n",

hedcode/matlab_scripts/README.md

@@ -11,10 +11,10 @@ subdirectory of [**hed-examples**](https://github.com/hed-standard/hed-examples)
 
 These scripts access HED Rest services.
 They rely on the HED online services to be running somewhere,
-either on a local Docker module or remotely.
+either in a local Docker module or remotely.
 
 Access to the HED services are also available online through
-the [**HED Online Tools](https://hedtools.ucsd.edu/hed).
+the [**HED Online Tools**](https://hedtools.ucsd.edu/hed).
 
 You can read more about these services in
 [**HED services in MATLAB**](https://hed-examples.readthedocs.io/en/latest/HedInMatlab.html#hed-services-in-matlab).
@@ -25,5 +25,11 @@ Some MATLAB utilities are available in
 [**utility_scripts**](https://github.com/hed-standard/hed-examples/tree/main/hedcode/matlab_scripts/utility_scripts).
 These utilities are main used for processing events and other information from EEGLAB `.set` files.
 
-EEGLAB also has a number of HED tools which are available as plugins for processing BIDS datasets
+Additional MATLAB tools for working with EEG `.set` files in
+BIDS datasets are available in the
+[**matlab_utility_scripts**](https://github.com/hed-standard/hed-curation/tree/main/src/curation/matlab_utility_scripts)
+directory of the [**hed-curation**](https://github.com/hed-standard/hed-curation) GitHub repository.
+
+[**EEGLAB**](https://sccn.ucsd.edu/eeglab/index.php) also has a number
+of HED tools which are available as plugins for processing BIDS datasets
 in the EEGLAB environment.