Lot of stuff

README.md

@@ -32,36 +32,65 @@ Join our community to create datasets and deep-learning models! Chat with us on
 
 This repository consists of:
 
-* [echotorch.datasets] : Pre-built datasets for common ESN tasks.
-* echotorch.evaluation : Tools and functions to evaluate and compare ESN models.
-* echotorch.models : Ready to train models and generic pre-trained ESN models.
-* echotorch.nn : All neural network Torch components for ESN and Reservoir Computing.
-* echotorch.transforms : Data transformations specific to ESN.
-* echotorch.utils : Tools, functions and measures for ESN and Reservoir Computing.
+* [echotorch.datasets](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/datasets) : Pre-built datasets for common ESN tasks.
+* [echotorch.evaluation](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/evaluation) : Tools and functions to evaluate and compare ESN models (cross-validation, statistical tests, etc).
+* [echotorch.models](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/models) : Ready to train models and generic pre-trained ESN models.
+* [echotorch.nn](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/models) : All neural network Torch components for ESN and Reservoir Computing.
+* [echotorch.transforms](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/transforms) : Data transformations specific to ESN.
+* [echotorch.utils](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/utils) : Tools, functions and measures for ESN and Reservoir Computing.
+    * [echotorch.utils.conceptors](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/utils/conceptors) : Utility classes and functions in relation with conceptor neural filters.
+    * [echotorch.utils.matrix_generation](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/utils/matrix_generation) : Class to generate different matrices to be used in ESNs.
+    * [echotorch.utils.optimization](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/utils/optimization) : Implementation of classical optimization algorithms for hyperparameters optimization.
+    * [echotorch.utils.visualisation](https://github.com/nschaetti/EchoTorch/tree/dev/echotorch/utils/visualisation) :  Various classes and functions for data and model visualisation.
 
 ## Examples
 
 Here is some examples of what you can do with EchoTorch.
 
 * [Conceptors](https://github.com/nschaetti/EchoTorch/tree/dev/examples/conceptors)
-    * [Four patterns generation with Conceptors](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/conceptors_4_patterns_generation.py) : load into a reservoir four patterns and re-generate them with conceptor-based neural filtering.
+    * [Boolean operations](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/boolean_operations.py) : Boolean operations with Conceptors.
+    * [Pattern evidences](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/conceptor_patterns_evidence.py) : Evidence gathering for pattern classification with Conceptors.
+    * [Four patterns generation](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/conceptors_4_patterns_generation.py) : load into a reservoir four patterns and re-generate them with conceptor-based neural filtering.
+    * [Incremental loading and memory management](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/incremental_load_memory_management.py) : how to load patterns in ESN's memory incrementally and manage memory usage.
+    * [Memory management](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/memory_management.py) :
+    * [Memory management and increament forgetting](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/memory_management_forgetting.py) : Load patterns in ESN's memory with possible to erase old patterns (in research).
+    * [Morphing periodic patterns](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/morphing_periodic_patterns.py) : Learning, generating and morphing a set of periodic patterns.
+    * [Morphing periodic sine](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/morphing_periodic_sine.py) : Learning, generating and morphing a set of sine patterns.
+    * [Morphing random patterns](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/morphing_random_patterns.py) : Learning, generating and morphing a set of random patterns.
+    * [Morphing random sine](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/morphing_random_sines.py) : Learning, generating and morphing a set of sine patterns with random periods.
+    * [Morphing sines](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/morphing_sines.py) : 
+    * [Morphing square](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/) : Learn four patterns and visualize multiple morphed patterns.
+    * [Subspace demo](https://github.com/nschaetti/EchoTorch/blob/dev/examples/conceptors/subspace_demo.py) : Show how patterns populate the space of reservoir state and how to define them with neural filters. 
 * [Datasets](https://github.com/nschaetti/EchoTorch/tree/dev/examples/datasets)
-    * [Logistic Map](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/logistic_map.py) : generate data from the logistic map function.
+    * [Latch-Copy-Repeat](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/latch_copy_repeat.py) : How to generate data for three well-known tasks in Machine Learning.
+    * [Logistic Map](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/logistic_map.py) : Generate data from the logistic map function.
+    * [MNIST_images](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/MNIST_images.py) : Load images from the MNIST dataset.
+    * [NARMA](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/NARMA.py) : Generate NARMA timeseries.
+    * [Strange attractors](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/strange_attractors.py) : Generate timeseries data from common strange attractors.
+    * [Timeseries batch sequencing](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/timeseries_batch_sequencing.py) : Transform a timeseries in sequences of specific length (to train a FFNN for example).
+    * [Timeseries triplet batching](https://github.com/nschaetti/EchoTorch/blob/dev/examples/datasets/timeseries_triplet_batching.py) : Get triplet from an anchor, a positive example (same class), and a negative example (no the same class) to train similarity measures.  
 * [Evaluation](https://github.com/nschaetti/EchoTorch/tree/dev/examples/evaluation)
-    * [Fold cross-validation](https://github.com/nschaetti/EchoTorch/tree/dev/examples/evaluation/fold_cross_validation.py) : how to perform 10-fold cross validation.
-    * [Grid search](https://github.com/nschaetti/EchoTorch/tree/dev/examples/evaluation/grid_search.py) : hyperparamaters optimization with grid-search.
+    * [Fold cross-validation](https://github.com/nschaetti/EchoTorch/tree/dev/examples/evaluation/fold_cross_validation.py) : How to perform 10-fold cross validation.
 * [Generation](https://github.com/nschaetti/EchoTorch/tree/dev/examples/generation)
-    * [NARMA-10 generation with feedbacks](https://github.com/nschaetti/EchoTorch/blob/dev/examples/generation/narma10_esn_feedbacks.py) : generate NARMA-10 timeseries with feedbacks.
+    * [NARMA-10 generation with feedbacks](https://github.com/nschaetti/EchoTorch/blob/dev/examples/generation/narma10_esn_feedbacks.py) : Generate NARMA-10 timeseries with feedbacks.
+* [Matrix generation](https://github.com/nschaetti/EchoTorch/tree/dev/examples/matrix_generation)
+    * [Cycle with jumps](https://github.com/nschaetti/EchoTorch/tree/dev/examples/matrix_generation/cycle_with_jumps.py) : Generation of a matrix composed of a cycle with jumps (Rodan and Tino, 2012). (To write)
+    * [Normal matrix](https://github.com/nschaetti/EchoTorch/tree/dev/examples/matrix_generation/normal_matrix_generation.py) : Generation based on a Gaussian distribution. (to write)
+    * [Uniform matrix](https://github.com/nschaetti/EchoTorch/tree/dev/examples/matrix_generation/uniform_matrix_generation.py) : Generation based on an uniform distribution. (to write)
 * [Memory](https://github.com/nschaetti/EchoTorch/tree/dev/examples/memory)
-    * [Memtest](https://github.com/nschaetti/EchoTorch/blob/dev/examples/memory/memtest.py) : test the capacity of an ESN to memorize random inputs.
+    * [Memtest](https://github.com/nschaetti/EchoTorch/blob/dev/examples/memory/memtest.py) : Test the capacity of an ESN to memorize random inputs.
 * [MNIST](https://github.com/nschaetti/EchoTorch/blob/dev/examples/MNIST/)
-    * [Image to timeseries conversion](https://github.com/nschaetti/EchoTorch/blob/dev/examples/MNIST/convert_images.py) : how to convert images to timeseries.
+    * [Image to timeseries conversion](https://github.com/nschaetti/EchoTorch/blob/dev/examples/MNIST/convert_images.py) : How to convert images to timeseries.
 * [Nodes](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes)
-    * [Independent Component Analysis](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes/ica_tests.py) : how to do Independent Component Analysis (ICA) with EchoTorch.
-    * [Principal Component Analysis](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes/pca_tests.py) : how to do Principal Component Analysis (PCA) with EchoTorch.
-    * [Slow Feature Analysis](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes/sfa_tests.py) : how to do Slow Features Analysis (SFA) with EchoTorch.
+    * [Independent Component Analysis](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes/ica_tests.py) : How to do Independent Component Analysis (ICA) with EchoTorch.
+    * [Principal Component Analysis](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes/pca_tests.py) : How to do Principal Component Analysis (PCA) with EchoTorch.
+    * [Slow Feature Analysis](https://github.com/nschaetti/EchoTorch/blob/dev/examples/nodes/sfa_tests.py) : How to do Slow Features Analysis (SFA) with EchoTorch.
+* [Optimization](https://github.com/nschaetti/EchoTorch/blob/dev/examples/optimization)
+    * [Genetic search](https://github.com/nschaetti/EchoTorch/blob/dev/examples/optimization/genetic_search.py) : Optimize hyper-parameters with a genetic algorithm.
+    * [Grid search](https://github.com/nschaetti/EchoTorch/blob/dev/examples/optimization/grid_search.py) : Optimize hyper-parameters with a grid search.
+    * [Random search](https://github.com/nschaetti/EchoTorch/blob/dev/examples/optimization/random_search.py) : Generate models withs random parameters and find the best.
 * [Switch between attractors](https://github.com/nschaetti/EchoTorch/blob/dev/examples/switch_attractor/switch_attractor_esn.py)
-    * [Switch Attractor](https://github.com/nschaetti/EchoTorch/blob/dev/examples/switch_attractor/switch_attractor_esn.py) : test the capacity of a simple ESN to switch between attractors.
+    * [Switch Attractor](https://github.com/nschaetti/EchoTorch/blob/dev/examples/switch_attractor/switch_attractor_esn.py) : Test the capacity of a simple ESN to switch between attractors.
 * [Timeseries prediction](https://github.com/nschaetti/EchoTorch/tree/dev/examples/timeserie_prediction)
     * [Mackey Glass](https://github.com/nschaetti/EchoTorch/blob/dev/examples/timeserie_prediction/mackey_glass_esn.py) : Mackey-Glass timeseries prediction with ESN.
     * [NARMA-10](https://github.com/nschaetti/EchoTorch/blob/dev/examples/timeserie_prediction/mackey_glass_esn.py) : NARMA-10 timeseries prediction with ESN and original training methods (ridge regression).
@@ -75,6 +104,11 @@ Here is some examples of what you can do with EchoTorch.
 
 In addition to examples, here are some Jupyter tutorials to learn how Reservoir Computing works.
 
+* [Timeseries prediction](https://github.com/nschaetti/EchoTorch/tree/dev/tutorials/timeseries_prediction/)
+    * [NARMA10](https://github.com/nschaetti/EchoTorch/tree/dev/tutorials/timeseries_prediction/NARMA_prediction.ipynb) : Train an ESN to predict a timeseries based on NARMA10 (to write).
+* [Images classification](https://github.com/nschaetti/EchoTorch/tree/dev/tutorials/images_classification/)
+    * [MNIST classification](https://github.com/nschaetti/EchoTorch/tree/dev/tutorials/images_classification/MNIST_classification.ipynb) : Classify handwritten digit images from the MNIST dataset (to write).
+
 ## Code and papers
 
 Here are some experimences done with ESN and reproduced with EchoTorch :

echotorch/datasets/DatasetComposer.py

@@ -4,7 +4,6 @@
 # Imports
 import torch
 from torch.utils.data.dataset import Dataset
-import numpy as np
 
 
 # Dataset Composer
@@ -39,9 +38,7 @@ def __init__(self, datasets, *args, **kwargs):
         # end for
     # end __init__
 
-    #############################################
-    # OVERRIDE
-    #############################################
+    #region OVERRIDE
 
     # Length
     def __len__(self):
@@ -67,9 +64,9 @@ def __getitem__(self, idx):
         return self.datasets[d][e], outputs, torch.LongTensor([d])
     # end __getitem__
 
-    #############################################
-    # PRIVATE
-    #############################################
+    #endregion OVERRIDE
+
+    #region PRIVATE
 
     # Create outputs
     def _create_outputs(self, i, time_length):
@@ -89,4 +86,6 @@ def _create_outputs(self, i, time_length):
         return outputs
     # end _create_outputs
 
+    #endregion PRIVATE
+
 # end DatasetComposer

echotorch/datasets/RepeatTaskDataset.py

@@ -31,7 +31,7 @@ class RepeatTaskDataset(Dataset):
     """
 
     # Constructor
-    def __init__(self, n_samples, length_min, length_max, n_inputs, max_repeat, dtype=torch.float32):
+    def __init__(self, n_samples, length_min, length_max, n_inputs, max_repeat, min_repeat=1, dtype=torch.float32):
         """
         Constructor
         :param sample_len: Sample's length
@@ -43,6 +43,7 @@ def __init__(self, n_samples, length_min, length_max, n_inputs, max_repeat, dtyp
         self.n_samples = n_samples
         self.n_inputs = n_inputs
         self.max_repeat = max_repeat
+        self.min_repeat = min_repeat
         self.dtype = dtype
 
         # Generate data set
@@ -86,7 +87,7 @@ def _generate(self):
         # For each sample
         for i in range(self.n_samples):
             # Random number of repeat
-            num_repeats = torch.randint(low=1, high=self.max_repeat+1, size=(1,)).item()
+            num_repeats = torch.randint(low=self.min_repeat, high=self.max_repeat+1, size=(1,)).item()
 
             # Generate length
             sample_len = torch.randint(low=self.length_min, high=self.length_max, size=(1,)).item()

echotorch/datasets/TimeseriesBatchSequencesDataset.py

@@ -83,15 +83,12 @@ def _load_dataset(self):
             data = self.root_dataset[item_i]
 
             # Get the first timeserie returned by the dataset
-            timeserie_pos = self.data_indices[0]
-
-            # Get timeserie
-            timeserie_data = data[timeserie_pos]
+            timeserie_data = data[self.data_indices[0]] if self.data_indices is not None else data
 
             # Length of timeseries in number of samples (sequences)
             timeserie_length = timeserie_data.size(self.time_axis)
             # timeserie_seq_length = int(math.floor(timeserie_length / self.window_size))
-            timeserie_seq_length = int(math.floor((timeserie_length - self.window_size) / self.stride))
+            timeserie_seq_length = int(math.floor((timeserie_length - self.window_size) / self.stride) + 1)
 
             # Save length and total length
             self.timeseries_lengths.append(timeserie_length)
@@ -131,9 +128,9 @@ def __getitem__(self, item):
             if ts_start_end['start'] <= item < ts_start_end['end']:
                 # Get the corresponding timeseries
                 if self.dataset_in_memory:
-                    data = list(self.dataset_samples[item_i])
+                    data = list(self.dataset_samples[item_i]) if self.data_indices is not None else self.dataset_samples[item_i]
                 else:
-                    data = list(self.root_dataset[item_i])
+                    data = list(self.root_dataset[item_i]) if self.data_indices is not None else self.root_dataset[item_i]
                 # end if
 
                 # Sequence start and end
@@ -143,13 +140,18 @@ def __getitem__(self, item):
                 sequence_range = range(sequence_start, sequence_end)
 
                 # For each data to transform
-                for data_i in self.data_indices:
-                    # Get timeserie
-                    timeserie_data = data[data_i]
-
+                if self.data_indices is not None:
+                    for data_i in self.data_indices:
+                        # Get timeserie
+                        timeserie_data = data[data_i]
+
+                        # Get sequence according to time axis
+                        data[data_i] = torch.index_select(timeserie_data, self.time_axis, torch.tensor(sequence_range))
+                    # end for
+                else:
                     # Get sequence according to time axis
-                    data[data_i] = torch.index_select(timeserie_data, self.time_axis, torch.tensor(sequence_range))
-                # end for
+                    data = torch.index_select(data, self.time_axis, torch.tensor(sequence_range))
+                # end if
 
                 # Return modified data
                 return data