Update test suite: Suppress output & expected warnings + improve perf…

…ormance (#658) * Coarse fix for #595 * Only include namespaces for types that are included in NWB file on export (Issue #607) * Add functionality for installing extensions * Minor fixes * Update comment * Add comment + print message when extension has been installed * Update installExtension.m * Fix changed variable name * Update matnwb_createNwbInstallExtension.m Update docstring * Create listNwbTypeHierarchy.m Add utility function for listing the type hierarchy of an nwb type * Add private method for embedding specifications to file on export * Fix variable name * Update testCreateParsedType.m * Suppress output of nwbfile * Add workflow for updating nwbInstallExtension * Add option to save extension in custom location * Create InstallExtensionTest.m * Add .env file to replace env.mat * Update .gitignore * Update docstring * Change dispExtensionInfo to return info instead of displaying + add test * Reorganize code into separate functions and add tests * Minor changes to improve test coverage * add nwbInstallExtension to docs * Update update_extension_list.yml Add schedule event for workflow to update nwbInstallExtension * Update downloadExtensionRepository.m Remove local function * Update docstring for nwbInstallExtension * Fix docstring indentation in nwbInstallExtension * Add doc pages describing how to use (ndx) extensions * Fix typo * Update +tests/+unit/InstallExtensionTest.m Co-authored-by: Ben Dichter <[email protected]> * Update docs/source/pages/getting_started/using_extensions/generating_extension_api.rst Co-authored-by: Ben Dichter <[email protected]> * Add docstrings for functions to retrieve and list extension info * Fix docstring formatting/whitespace * Update listExtensions.m Add example to docstring * Move static test methods into io.internal.h5 namespace Introduce some functions that will be useful later * Update writeEmbeddedSpecifications.m Add arguments block, fix function name * Add validateEmbeddedSpecifications * Update NwbFile.m Redefine listNwbTypes method, add validation of embedded namespaces * Create listEmbeddedSpecNamespaces.m * Update nwbExportTest.m * Update test for spec/namespace embedding * Update read_indexed_column.m * Add disclaimer in deleteGroup function * Update read_indexed_column.m * Fix broken test * add test-requirement * Fix: Ensure object is group before deleting * Fix error id * Add unittests for functions in io.internal.h5 namespace * Suppress output * Remove duplicated code * Update fillClass.m Set Required "tag" for properties derived from attributes and links * Update fillClass/fillExport to add check for dependent & required properties on export * Add warnIfRequiredDependencyMissing If a required attribute depends on a non-required Dataset or Group, this method issues a warning if the dependent attribute (property) is missing when it's parent is present * Update generated types * Add default property values to generated type classes * Update warnIfPropertyAttributeNotExported to compare value with defult value before issuing warning * Add value for required property in VectorData * Add unit tests * Supress warning in test * Update tests.util.getPythonPath and remove similar code from +tests/+system/PyNWBIOTest.m * Update GenerationTest.m Add note * Suppress warnings in DataPipeTest use verifyWarning and verifyError to make code clearer Make testFilterOverride for better test granularity * Suppress output * Improve performance of if block Make check targeted to classes that have readonly prop * Remove try/catch * Use verifyWarning instead of custom warning manipulation * Updated neurodata type classes * Add unit test * Add shared fixture * Chnage name part 1/2 * Change name part2/2 * Suppress output * Add shared test fixture to tutorialtest * Update nwbtest.m * Update nwbtest.m - Use method for sorting suite by fixture - nwbClearGenerated + cleanup should be handled by test fixtures - Add option to skip generatimg coverage report * Refactor unit tests for test-schemas for better performance (#659) * Add TypesOutputFolder as property on NwbTypeGeneratorFixture class * Add abstract class for unit tests that tests test-schemas * Move test schemas to separate folder * Move test-schema Tests to tests.unit.schema namespace and subclass from tests.unit.abstract.SchemaTest * Update class names to match filenames * Replace getRandInd with builtin randi function * Reduce number of iterations in RegionViewTest * Restore file wrongly added to PR * Add Extension fixture * Refactor - slide suite sorting * Refactor misc unittests in "tests.unit" namespace (#660) * Change misc unittests from function- to class based * Change testCreateParsedType from function to class-based test suite * Fix generate types in folder set up by Shared fixture * Update dataStubTest.m * Move tutorial tests from tests.unit to tests.system.tutorial (#661) * Update nwbtest.m * Fix bug with clearing and regenerating types * Ignore generated types from test coverage * Rename ExtensionFixture to ExtensionGenerationFixture * Add docstring to ExtensionGenerationFixture * Rename ResetGeneratedTypesFixture to NwbClearGeneratedFixture * Rename NwbTypeGeneratorFixture to GenerateCoreFixture * Fix - add default input to NwbClearGeneratedFixture * Update nwbtest - Save generated reports to fixed directory * Add docs/reports to gitignore * Add .coverageignore file and local function for collecting list of files to produce code coverage for * Fix workflow for running tests (report output directory) * Update nwbExportTest.m Added comments and a better test to test for warning with ID 'NWB:validators:MissingEmbeddedNamespace' * Update branch name references from master to main * Change branch name from master to main in md and workflow files * Fix typo * Fix failing tests * Suppress more expected warnings and outputs * Add test * Update test to test different data types * Make local function getEmptyRepresentation more explicit * Remove unused code from test * Use shared fixture in WarningsTest * Update dataStubTest, include new test in test methods block * Add shared fixture to new test MustBeH5FileTest * Add abstract NwbTestCase class with utility methods that can be reused * Suppress output from various tests * Fix failing test * Remove commented code block from ObjectView class * Add utility method to NwbTestCase class * Fix bug and add test * Add "version" as property to Namespace class * Add routine for writing a function for retrieving version of generated namespace * Regenerate core with version numbers * Update generateRstForNeurodataTypeClasses.m Ignore version function when creating list of neurodata types to document * Add version specificity to cache containing map of required properties * Delete GenerateSpecificationMixin.m * Split test class setup into methods with single responsibilities * Add factory function to create NwbFile for unit testing * Clean up nwbExport test * Clean up a few more test classes * Add fixture for setting environment variables used in test suite and remove env.mat * Update PyNWBIOTest.m Add shared fixture * Fix wrong shape of data in PhotonSeriesIOTest * Update PyNWBIOTest.py --------- Co-authored-by: Ben Dichter <[email protected]>
NeurodataWithoutBorders · Feb 25, 2025 · 806b67d · 806b67d
1 parent fedd2ce
commit 806b67d
Show file tree

Hide file tree

Showing 67 changed files with 1,846 additions and 1,574 deletions.
diff --git a/+tests/+abstract/NwbTestCase.m b/+tests/+abstract/NwbTestCase.m
@@ -0,0 +1,56 @@
+classdef (Abstract, SharedTestFixtures = {tests.fixtures.GenerateCoreFixture}) ...
+        NwbTestCase < matlab.unittest.TestCase
+% NwbTestCase - Abstract class providing a shared fixture, and utility
+% methods for running tests dependent on generating neurodata type classes
+
+    methods (Access = protected)
+        function typesOutputFolder = getTypesOutputFolder(testCase)
+            F = testCase.getSharedTestFixtures();
+            isMatch = arrayfun(@(x) isa(x, 'tests.fixtures.GenerateCoreFixture'), F);
+            F = F(isMatch);
+
+            typesOutputFolder = F.TypesOutputFolder;
+        end
+
+        function installExtension(testCase, extensionName)
+            typesOutputFolder = testCase.getTypesOutputFolder();
+
+            % Use evalc to suppress output while running tests.
+            matlabExpression = sprintf(...
+                'nwbInstallExtension("%s", "savedir", "%s")', ...
+                extensionName, typesOutputFolder);
+            evalc(matlabExpression);
+        end
+
+        function clearExtension(testCase, extensionName)
+            extensionName = char(extensionName);
+            namespaceFolderName = strrep(extensionName, '-', '_');
+            typesOutputFolder = testCase.getTypesOutputFolder();
+            rmdir(fullfile(typesOutputFolder, '+types', ['+', namespaceFolderName]), 's')
+            delete(fullfile(typesOutputFolder, 'namespaces', [extensionName '.mat']))
+        end
+    end
+
+    methods (Static, Access = protected)
+        function [nwbFile, nwbFileCleanup] = readNwbFileWithPynwb(nwbFilename)
+            try
+                io = py.pynwb.NWBHDF5IO(nwbFilename);
+                nwbFile = io.read();
+                nwbFileCleanup = onCleanup(@(x) closePyNwbObject(io));
+            catch ME
+                error(ME.message)
+            end
+
+            function closePyNwbObject(io)
+                io.close()
+            end
+        end
+
+        function nwbFilename = getRandomFilename()
+            % Assumes that this method is called from a test method
+            functionCallStackTrace = dbstack();
+            testName = regexp(functionCallStackTrace(2).name, '\w*$', 'match', 'once');
+            nwbFilename = sprintf('%s_%05d.nwb', testName, randi(9999));
+        end
+    end
+end
diff --git a/+tests/+factory/ImagingPlane.m b/+tests/+factory/ImagingPlane.m
@@ -0,0 +1,11 @@
+function imaging_plane = ImagingPlane(device)
+% ImagingPlane - Create imaging plane with values for all required properties
+    arguments
+        device (1,1) types.core.Device % A device is required
+    end
+    imaging_plane = types.core.ImagingPlane( ...
+        'device', types.untyped.SoftLink(device), ...
+        'excitation_lambda', 600., ...
+        'indicator', 'GFP', ...
+        'location', 'my favorite brain location');
+end
diff --git a/+tests/+factory/NWBFile.m b/+tests/+factory/NWBFile.m
@@ -0,0 +1,9 @@
+function nwb = NWBFile()
+    currentTime = datetime("now", 'TimeZone', 'local');
+
+    nwb = NwbFile( ...
+        'session_description', 'NWB File with required properties for unit testing', ...
+        'identifier', 'test_file', ...
+        'session_start_time', currentTime, ...
+        'timestamps_reference_time', currentTime);
+end
diff --git a/+tests/+factory/TimeSeriesWithTimestamps.m b/+tests/+factory/TimeSeriesWithTimestamps.m
@@ -0,0 +1,6 @@
+function timeSeries = TimeSeriesWithTimestamps()
+    timeSeries = types.core.TimeSeries(...
+        'data', rand(1,10), ...
+        'timestamps', 1:10, ...
+        'data_unit', 'test');
+end
diff --git a/+tests/+fixtures/ExtensionGenerationFixture.m b/+tests/+fixtures/ExtensionGenerationFixture.m
@@ -0,0 +1,55 @@
+classdef ExtensionGenerationFixture < matlab.unittest.fixtures.Fixture
+%EXTENSIONGENERATIONFIXTURE - Fixture for generating an NWB extension.
+% 
+% EXTENSIONGENERATIONFIXTURE provides a fixture for generating extension code 
+% from an NWB specification's namespace file. When the testing framework
+% sets up the fixture, it calls the generateExtension function to produce the 
+% necessary code in the specified output folder. When the framework tears down
+% the fixture, it removes the generated files and associated cache data, 
+% ensuring that no artifacts remain from the test generation process.
+%
+% See also matlab.unittest.fixtures.Fixture generateExtension nwbClearGenerated
+
+    properties
+        % TypesOutputFolder - Folder to output generated types for test
+        % classes that share this fixture
+        TypesOutputFolder (1,1) string
+
+        % NamespaceFilepath - Path name for extension's namespace file
+        NamespaceFilepath (1,1) string
+    end
+
+    methods
+        function fixture = ExtensionGenerationFixture(namespaceFilepath, outputFolder)
+            fixture.NamespaceFilepath = namespaceFilepath;
+            fixture.TypesOutputFolder = outputFolder;
+        end
+    end
+
+    methods
+        function setup(fixture)
+            generateExtension(fixture.NamespaceFilepath, 'savedir', fixture.TypesOutputFolder);
+            fixture.addTeardown(@fixture.clearGenerated)
+        end
+    end
+
+    methods (Access = protected)
+        function tf = isCompatible(fixtureA, fixtureB)
+            tf = strcmp(fixtureA.NamespaceFilepath, fixtureB.NamespaceFilepath) ...
+                 && strcmp(fixtureA.TypesOutputFolder, fixtureB.TypesOutputFolder);
+        end
+    end
+
+    methods (Access = private)
+        function clearGenerated(fixture)
+            [~, namespaceFilename] = fileparts(fixture.NamespaceFilepath);
+            namespaceName = extractBefore(namespaceFilename, '.');
+
+            generatedTypesDirectory = fullfile(fixture.TypesOutputFolder, "+types", "+"+namespaceName);
+            rmdir(generatedTypesDirectory, 's');
+
+            cacheFile = fullfile(fixture.TypesOutputFolder, "namespaces", namespaceName+".mat");
+            delete(cacheFile)
+        end
+    end
+end
diff --git a/+tests/+fixtures/GenerateCoreFixture.m b/+tests/+fixtures/GenerateCoreFixture.m
@@ -0,0 +1,44 @@
+classdef GenerateCoreFixture < matlab.unittest.fixtures.Fixture
+% GENERATECOREFIXTURE - Fixture for creating classes for NWB types in a temporary folder.
+%        
+% GENERATECOREFIXTURE provides a fixture for generating classes for neurodata 
+% types from the from the core NWB specifications. When the testing framework
+% sets up the fixture, it calls the generateCore function to produce the 
+% necessary code in a temporary output folder and add it to MATLAB's path. When 
+% the framework tears down the fixture, it clears all the classes and deletes 
+% the temporary folder, ensuring that no artifacts remain from the test process.
+%
+% See also matlab.unittest.fixtures.Fixture generateCore
+    properties
+        % TypesOutputFolder - Folder to output generated types for test
+        % classes that share this fixture
+        TypesOutputFolder (1,1) string
+    end
+
+    methods
+        function setup(fixture)
+            import matlab.unittest.fixtures.PathFixture
+            import matlab.unittest.fixtures.TemporaryFolderFixture
+            import tests.fixtures.NwbClearGeneratedFixture
+
+            % Use the NwbClearGeneratedFixture to clear all generated types
+            % from the MatNWB root directory in order to preventing path 
+            % conflicts when generating new types in a temporary directory
+            fixture.applyFixture( NwbClearGeneratedFixture )
+
+            % Use a fixture to add the MatNWB folder to the search path
+            fixture.applyFixture( PathFixture( misc.getMatnwbDir() ) );
+
+            % Use a fixture to create a temporary working directory
+            F = fixture.applyFixture( TemporaryFolderFixture );
+
+            % Generate core types in the temporary folder and add to path
+            generateCore('savedir', F.Folder)
+            fixture.applyFixture( PathFixture(F.Folder) );
+
+            % Save the folder containing cached namespaces and NWB type classes
+            % on the fixture object
+            fixture.TypesOutputFolder = F.Folder;
+        end
+    end
+end
diff --git a/+tests/+fixtures/NwbClearGeneratedFixture.m b/+tests/+fixtures/NwbClearGeneratedFixture.m
@@ -0,0 +1,38 @@
+classdef NwbClearGeneratedFixture < matlab.unittest.fixtures.Fixture
+% NwbClearGeneratedFixture - Fixture for clearing generated NWB classes.
+%
+%   NwbClearGeneratedFixture provides a fixture for clearing all the
+%   generated classes for NWB types from the matnwb folder. When the fixture is 
+%   set up, all generated class files for NWB types are deleted. When the 
+%   fixture is torn down, generateCore is called to regenerate the classes for 
+%   NWB types of the latest NWB version
+%
+% See also matlab.unittest.fixtures.Fixture generateCore nwbClearGenerated
+
+    properties
+        TypesOutputFolder (1,1) string {mustBeFolder} = misc.getMatnwbDir
+    end
+
+    methods
+        function fixture = NwbClearGeneratedFixture(outputFolder)
+            arguments
+                outputFolder (1,1) string {mustBeFolder} = misc.getMatnwbDir
+            end
+            fixture.TypesOutputFolder = outputFolder;
+        end
+    end
+
+    methods
+        function setup(fixture)
+            fixture.addTeardown( ...
+                @() generateCore('savedir', fixture.TypesOutputFolder) )
+            nwbClearGenerated(fixture.TypesOutputFolder)
+        end
+    end
+
+    methods (Access = protected)
+        function tf = isCompatible(fixtureA, fixtureB)
+            tf = strcmp(fixtureA.TypesOutputFolder, fixtureB.TypesOutputFolder);
+        end
+    end
+end
diff --git a/+tests/+fixtures/ResetGeneratedTypesFixture.m b/+tests/+fixtures/ResetGeneratedTypesFixture.m
diff --git a/+tests/+fixtures/SetEnvironmentVariableFixture.m b/+tests/+fixtures/SetEnvironmentVariableFixture.m
@@ -0,0 +1,83 @@
+classdef SetEnvironmentVariableFixture < matlab.unittest.fixtures.Fixture
+% UsesEnvironmentVariable Fixture for setting environment variables in tests.
+%
+%   This fixture reads an environment file containing key-value pairs and
+%   sets the corresponding system environment variables prior to executing
+%   tests. The expected format for the environment file is:
+%
+%       VARIABLE_NAME=VALUE
+%
+%   Lines that are empty or start with '#' (comments) are ignored.
+%
+%   The fixture first attempts to load environment variables from the file
+%   "nwbtest.env" located in the "+tests" folder. If "nwbtest.env" is not 
+%   found, it falls back to "nwbtest.default.env". When using the default file, 
+%   the fixture only applies environment variables if they are not present
+%   in the current list of environment variables.
+
+    methods
+        function setup(fixture) %#ok<MANU>
+
+            applyFromFile = true;
+            envFilePath = fullfile(misc.getMatnwbDir, '+tests', 'nwbtest.env');
+
+            if ~isfile(envFilePath)
+                envFilePath = fullfile(misc.getMatnwbDir, '+tests', 'nwbtest.default.env');
+                applyFromFile = false;
+            end
+
+            if exist("loadenv", "file") == 2 
+                envVariables = loadenv(envFilePath);
+            else
+                envVariables = readEnvFile(envFilePath);
+            end
+
+            envVariableNames = string( envVariables.keys() );
+            if ~isrow(envVariableNames); envVariableNames = envVariableNames'; end
+
+            for varName = envVariableNames
+                varValue = envVariables(varName);
+                if ~isenv(varName)
+                    setenv(varName, varValue)
+                elseif applyFromFile && ~isempty(char(varValue))
+                    setenv(varName, varValue)
+                end
+            end
+        end
+    end
+end
+
+function envMap = readEnvFile(filename)
+% readEnvFile Reads an environment file into a containers.Map.
+%
+%   envMap = readEnvFile(filename) reads the file specified by 'filename'
+%   and returns a containers.Map where each key is a variable name and each
+%   value is the corresponding value from the file.
+%
+%   Lines starting with '#' or empty lines are ignored.
+
+    envMap = containers.Map;
+
+    fileContent = fileread(filename);
+    lines = strsplit(fileContent, newline);
+
+    for i = 1:numel(lines)
+        line = lines{i};
+        if isempty(line) || startsWith(line, '#')
+            continue;
+        end
+
+        % Find the first occurrence of '='
+        idx = strfind(line, '=');
+        if isempty(idx)
+            continue; % ignore line
+        end
+
+        % Use the first '=' as the delimiter
+        key = strtrim(line(1:idx(1)-1));
+        value = strtrim(line(idx(1)+1:end));
+
+        % Insert the key-value pair into the map
+        envMap(key) = value;
+    end
+end
diff --git a/+tests/+sanity/GenerationTest.m b/+tests/+sanity/GenerationTest.m
@@ -1,17 +1,21 @@
 classdef GenerationTest < matlab.unittest.TestCase
+% Note: Sometimes this test does not work for the first two schema versions. 
+% Restarting MATLAB can fix this.
+
     properties (MethodSetupParameter)
         schemaVersion = listSchemaVersions()
     end
 
     methods (TestClassSetup)
-        function setupClass(testCase)
+        function setupMatNWBPathFixture(testCase)
             import matlab.unittest.fixtures.PathFixture
-            import tests.fixtures.ResetGeneratedTypesFixture
-
-            rootPath = tests.util.getProjectDirectory();
-            testCase.applyFixture( PathFixture(rootPath) );
+            matNwbRootPath = tests.util.getProjectDirectory();
+            testCase.applyFixture( PathFixture(matNwbRootPath) );
+        end
 
-            testCase.applyFixture( ResetGeneratedTypesFixture );
+        function setupNwbClearGeneratedFixture(testCase)
+            import tests.fixtures.NwbClearGeneratedFixture
+            testCase.applyFixture( NwbClearGeneratedFixture );
         end
     end