diff --git a/notebooks/04_PopulateConfigFile.ipynb b/notebooks/04_PopulateConfigFile.ipynb new file mode 100644 index 000000000..e7bf96f1e --- /dev/null +++ b/notebooks/04_PopulateConfigFile.ipynb @@ -0,0 +1,262 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Customizing Data Insertion into Spyglass\n", + "\n", + "If you would like to insert data into Spyglass that does not\n", + "follow the naming or organizational format expected by Spyglass, \n", + "or you would like to override what values are ingested into Spyglass from \n", + "your NWB files, including missing values, follow this guide.\n", + "\n", + "## General Approach\n", + "\n", + "When an NWB file is ingested into Spyglass, metadata about the session\n", + "is first read from the NWB file and inserted into\n", + "tables in the `common` module (e.g. `Institution`, `Lab`, `Electrode`, etc).\n", + "However, not every NWB file has all the information required by Spyglass or\n", + "the information in the NWB file is not in a format that Spyglass expects. For\n", + "example, many NWB files do not contain information about the\n", + "`DataAcquisitionDevice` or `Probe` because the NWB data standard does not yet\n", + "have an official\n", + "standard for specifying them. For these cases, we provide a way to customize\n", + "how data is ingested into Spyglass.\n", + "\n", + "Let's say that you want to ingest an NWB file into Spyglass where the lab name\n", + "in the file is written as \"Loren Frank Lab\" or it is not specified, but you \n", + "know the data comes from the Loren Frank Lab. Let's say that in Spyglass,\n", + "the lab name that is associated with sessions from the Loren Frank Lab is \n", + "\"Frank Lab\" and you would like to use the same name in order to facilitate\n", + "data search in Spyglass. To change the lab name when you insert your new data \n", + "to Spyglass, you could either 1) edit the NWB file directly and then \n", + "insert it into Spyglass, or 2) define an override value \"Frank Lab\" to be \n", + "used instead of the value specified in the NWB file (or lack thereof).\n", + "\n", + "Note that if this is your own data and you want to make changes to\n", + "information about how the data is interpreted, e.g., the units of measurement\n", + "are incorrect, we recommend that you edit the NWB file directly because the \n", + "file or derivatives of it might eventually be shared outside of Spyglass and \n", + "they will not reflect any modifications that you have made to \n", + "the data only in Spyglass." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Define a Configuration YAML File\n", + "\n", + "To override values in the NWB file during insertion into Spyglass, \n", + "you will need to create a configuration \n", + "YAML file that lives in the same directory as your NWB file, named: \n", + "`_spyglass_config.yaml`\n", + "\n", + "An example configuration YAML file can be found at\n", + "`examples/config_yaml/​​sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml`.\n", + "This file is associated with the NWB file\n", + "`sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys.nwb`.\n", + "\n", + "This is the general format for entries in this configuration file:\n", + "\n", + "```yaml\n", + "TableName:\n", + "- primary_key1: value1\n", + "```\n", + "\n", + "For example:\n", + "\n", + "```yaml\n", + "Lab:\n", + "- lab_name: Frank Lab\n", + "```\n", + "\n", + "In this example, the NWB file that corresponds to this config YAML will become\n", + "associated with the entry in the `Lab` table with the value `Frank Lab` for \n", + "the primary key `lab_name`. This entry must already exist. More specifically,\n", + "when the NWB file is ingested into Spyglass, \n", + "a new `Session` entry will be created for the NWB file that has a foreign key to\n", + "the `Lab` entry with `lab_name` = \"Frank Lab\", ignoring whatever lab value is \n", + "in the NWB file, even if one does not exist.\n", + "\n", + "TODO implement this for `Lab`.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Create Entries to Reference in the Configuration YAML\n", + "\n", + "As mentioned earlier, the table entry that you want to associate with your NWB\n", + "file must already exist in the database. This entry would typically be a value\n", + "that is independent of any particular NWB file, such as\n", + "`DataAcquisitionDevice`, `Lab`, `Probe`, and `BrainRegion`. \n", + "\n", + "If the entry does not already exist, you can either:\n", + "1) create the entry programmatically using DataJoint `insert` commands, or\n", + "2) define the entry in a YAML file called `entries.yaml` that is automatically\n", + " processed when Spyglass is imported. You can think of `entries.yaml` as a\n", + " place to define information that the database should come pre-equipped prior\n", + " to ingesting your NWB files. The `entries.yaml` file should be placed in the\n", + " `spyglass` base directory (next to `README.md`). An example can be found in\n", + " `examples/config_yaml/entries.yaml`. This file should have the following\n", + " structure:\n", + "\n", + " ```yaml\n", + " TableName:\n", + " - TableEntry1Field1: Value\n", + " TableEntry1Field2: Value\n", + " - TableEntry2Field1: Value\n", + " TableEntry2Field2: Value\n", + " ```\n", + "\n", + " For example,\n", + "\n", + " ```yaml\n", + " DataAcquisitionDeviceSystem:\n", + " data_acquisition_device_system: SpikeGLX\n", + " DataAcquisitionDevice:\n", + " - data_acquisition_device_name: Neuropixels_SpikeGLX\n", + " data_acquisition_device_system: SpikeGLX\n", + " data_acquisition_device_amplifier: Intan\n", + " ```\n", + "\n", + " Only `dj.Manual`, `dj.Lookup`, and `dj.Part` tables can be populated\n", + " using this approach.\n", + "\n", + "Once the entry that you want to associate with your NWB file exists in the\n", + "database, you can write the configuration YAML file and then ingest your\n", + "NWB file. As an another example, let's say that you want to associate your NWB\n", + "file with the `DataAcquisitionDevice` entry with `data_acquisition_device_name`\n", + "= \"Neuropixels_SpikeGLX\" that was defined above. You would write the following\n", + "configuration YAML file:\n", + "\n", + "```yaml\n", + "DataAcquisitionDevice:\n", + "- data_acquisition_device_name: Neuropixels_SpikeGLX\n", + "```\n", + "\n", + "The example in\n", + "`examples/config_yaml/​​sub-AppleBottom_ses-AppleBottom-DY20-g3_behavior+ecephys_spyglass_config.yaml`\n", + "includes additional examples." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Example Ingestion with Real Data\n", + "\n", + "For this example, you will need to download the 5 GB NWB file \n", + "`sub-JDS-NFN-AM2_behavior+ecephys.nwb`\n", + "from dandiset 000447 here: \n", + "https://dandiarchive.org/dandiset/000447/0.230316.2133/files?location=sub-JDS-NFN-AM2&page=1\n", + "\n", + "Click the download arrow button to download the file to your computer. Add it to the folder\n", + "containing your raw NWB data to be ingested into Spyglass.\n", + "\n", + "This file does not specify a data acquisition device. Let's say that the\n", + "data was collected from a SpikeGadgets system with an Intan amplifier. This\n", + "matches an existing entry in the `DataAcquisitionDevice` table with name\n", + "\"data_acq_device0\". We will create a configuration YAML file to associate\n", + "this entry with the NWB file.\n", + "\n", + "If you are connected to the Frank lab database, please rename any downloaded\n", + "files (e.g., `example20200101_yourname.nwb`) to avoid naming collisions, as the\n", + "file name acts as the primary key across key tables." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "nwb_file_name = \"sub-JDS-NFN-AM2_behavior+ecephys_rly.nwb\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# this configuration yaml file should be placed next to the downloaded NWB file\n", + "yaml_config_path = \"sub-JDS-NFN-AM2_behavior+ecephys_rly_spyglass_config.yaml\"\n", + "with open(yaml_config_path, \"w\") as config_file:\n", + " lines = [\n", + " \"DataAcquisitionDevice\",\n", + " \"- data_acquisition_device_name: data_acq_device0\",\n", + " ]\n", + " config_file.writelines(line + '\\n' for line in lines)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then call `insert_sessions` as usual." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import spyglass.data_import as sgi\n", + "\n", + "sgi.insert_sessions(nwb_file_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Confirm the session was inserted with the correct `DataAcquisitionDevice`" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import spyglass.common as sgc\n", + "from spyglass.utils.nwb_helper_fn import get_nwb_copy_filename\n", + "\n", + "nwb_copy_file_name = get_nwb_copy_filename(nwb_file_name)\n", + "\n", + "sgc.Session.DataAcquisitionDevice & {\"nwb_file_name\": nwb_copy_file_name}" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.18" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/src/spyglass/spikesorting/spikesorting_artifact.py b/src/spyglass/spikesorting/spikesorting_artifact.py index 9ae951f90..399c30542 100644 --- a/src/spyglass/spikesorting/spikesorting_artifact.py +++ b/src/spyglass/spikesorting/spikesorting_artifact.py @@ -302,7 +302,7 @@ def _get_artifact_times( (valid_timestamps[i[0]], valid_timestamps[i[1]]) ) - return artifact_removed_valid_times, artifact_intervals_s + return np.asarray(artifact_removed_valid_times), artifact_intervals_s def _init_artifact_worker(