Sync master into develop (#1222)

* Updated accuracy distance metric to accomodate condition-wise approaches (#1217) Co-authored-by: ddcoggan <[email protected]> * Add temporal metrics; add temporal versions of MajajHong2015 (#1109) * feature: support temporal models for neural alignment by chaning TemporalIgnore to Temporal Aligned * add example temporal submission * complete new framework * new module: temporal model helpers * change the arch of temporal; add tutorials * improve: better naming * update: wrapper tutorial on brain model * add feature: inferencer identifier tracked by extractor for result caching * fix: video fps sampling; need more tests! * fix bugs: video sampling based on fps was wrong. * add mmaction2 models; add more features to the inferencers * PR: temporal model helpers * PR fix: not including gitmodules for now * Update brainscore_vision/model_helpers/brain_transformation/temporal.py Co-authored-by: Martin Schrimpf <[email protected]> * Update brainscore_vision/model_helpers/brain_transformation/temporal.py Co-authored-by: Martin Schrimpf <[email protected]> * Update brainscore_vision/model_helpers/brain_transformation/temporal.py Co-authored-by: Martin Schrimpf <[email protected]> * Update brainscore_vision/models/temporal_models/test.py Co-authored-by: Martin Schrimpf <[email protected]> * add mae_st; add ding2012 * try new arch * init ding2012 * add tests for temporal model helpers; add block inferencer * Delete tests/test_model_helpers/temporal/test___init__.py delete the old test * add benchmark ding2012 * add mutliple libs for temporal models * change executor output format; add more inference tests; init load_weight in s3 * add openstl * update backend for executor * feat:load_weight_file and corresponding test * change:resize strategy changed from bilinear to pooling * change:resize strategy changed from bilinear to pooling * fix mae_st submission * minor * fix:dtype in assembly time align * minor * update model submissions * fix dependency * refactor: simplify the inferencer methods * fix:block inferencer, neuroid coord while merging * fix:inferencer identifier * fix:weigh download * change tests to have max_workers=1 * revert screen.py * not submit region_layer_map * remove torch dependency * make fake modules in tests * add torch to requirements; avoid torch in tests * minor * minor * np.object changed to object * remove return in tests * fix insertion position bug * Apply suggestions from code review add: more type hints Co-authored-by: Martin Schrimpf <[email protected]> * add: more type hints and comments * minor * pr:only commit temporal model helpers * pr: add one model for example * undo whole_brain in Brainodel.RecordingTarget * use logger and fix newlines * fix: video fps with copy was wrong * feat:fractional max_spatial_size * downsample layers in VideoMAE * fix:video sampling wrong duration * add more tests * fix merge * fix merge * module refactor; add more input test * add more temporal models * fix videomaev2 sha * fix:temporal_modelmae_st * change:video conservative loading; rename:image to pil image * fix:video last frame sampling; fix_time_naming * ignore pytest_cache * re-trigger tests * add joblib pool error management; fix video/image path recognizer * update: naming of failed to pickle func in joblibmapper * add temporal metric helpers * add temporal version of mamjajhong2015 * Update benchmark.py type hint * Update benchmark.py * Update brainscore_vision/metric_helpers/temporal.py Co-authored-by: Martin Schrimpf <[email protected]> * Update brainscore_vision/metrics/internal_consistency/__init__.py Co-authored-by: Martin Schrimpf <[email protected]> * Update benchmark.py --------- Co-authored-by: Yingtian Tang <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: deirdre-k <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> * Python 3.11 Migration (v2.1.0) (#1199) * changes necessary for 3.11 * change references of 3.7 to 3.11, remove tf and keras * fix geirhos data tests * Normalize deprecated * np.float is deprecated * update w/ master (#1066) * add r101_eBarlow_lmda_01_1 to models (#1037) Co-authored-by: AutoJenkins <[email protected]> * update hmax requirements.txt (#1053) * update hmax requirements.txt to add torchvision (#1054) * fix mobilenet `pooler` layer prefix (#1055) * add r34_eMMCR_Mom_lmda_02_1 to models (#1061) Co-authored-by: AutoJenkins <[email protected]> * add r101_eBarlow_lmda_02_1_copy to models (#1059) Co-authored-by: AutoJenkins <[email protected]> * add r34_eMMCR_Mom_lmda_01_1 to models (#1060) Co-authored-by: AutoJenkins <[email protected]> * add r34_eMMCR_Mom_Vanilla_1 to models (#1063) Co-authored-by: AutoJenkins <[email protected]> * ignore time dimension in Bracci2019 (#1064) --------- Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> * update (#1093) * add r101_eBarlow_lmda_01_1 to models (#1037) Co-authored-by: AutoJenkins <[email protected]> * update hmax requirements.txt (#1053) * update hmax requirements.txt to add torchvision (#1054) * fix mobilenet `pooler` layer prefix (#1055) * add r34_eMMCR_Mom_lmda_02_1 to models (#1061) Co-authored-by: AutoJenkins <[email protected]> * add r101_eBarlow_lmda_02_1_copy to models (#1059) Co-authored-by: AutoJenkins <[email protected]> * add r34_eMMCR_Mom_lmda_01_1 to models (#1060) Co-authored-by: AutoJenkins <[email protected]> * add r34_eMMCR_Mom_Vanilla_1 to models (#1063) Co-authored-by: AutoJenkins <[email protected]> * ignore time dimension in Bracci2019 (#1064) * Update behavior.py (#742) Co-authored-by: Sam Winebrake <[email protected]> Co-authored-by: deirdre-k <[email protected]> * add eBarlow_lmda_02_1_full to models (#1067) Co-authored-by: AutoJenkins <[email protected]> * oddoneout: build full stimulus id index for non-numeric ids (#1068) * Add mobilevit_small - take 2 (#1051) * Add mobilevit_small - take 2 * Update brainscore_vision/models/mobilevit_small/model.py Co-authored-by: Martin Schrimpf <[email protected]> --------- Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> * add yudixie_resnet18_240719_1 to models (#1070) Co-authored-by: AutoJenkins <[email protected]> * add yudixie_resnet18_240719_0 to models (#1069) Co-authored-by: AutoJenkins <[email protected]> * add yudixie_resnet18_240719_2 to models (#1071) Co-authored-by: AutoJenkins <[email protected]> * add yudixie_resnet18_240719_10 to models (#1079) Co-authored-by: AutoJenkins <[email protected]> --------- Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Linus Sommer <[email protected]> Co-authored-by: deirdre-k <[email protected]> Co-authored-by: Kartik Pradeepan <[email protected]> * remove keraswrapper * remove brendel * Trying a fix for travis * add necessary imports * Updating ubuntu version as xenial doesn't have 3.11 * Adding fix for pyproject.toml * Pin numpy below 2.0 * pointing to test branch for travis * update s3 bucket to brainscore-unittests * update s3 bucket to brainscore-unittests (rajalingham2020) * update s3 bucket to brainscore-unittests (sanghavi2020) * update folder name of models to folder_name="models-to-integrate-for-2.0" * update folder name of MAE model to folder_name="models-to-integrate-for-2.0" * update folder name of MAEv2 to folder_name="models-to-integrate-for-2.0" * update folder name of mae_st to folder_name="models-to-integrate-for-2.0" * PredNet commented out in init * remove prednet dead code * remove prednet code * scialom data: update image_id to stimulus_id * add number_of_trials keyword to rajalingham2018 * pointing to branch * Removing dash from python module name (#1118) * Removing dash from module name (#1119) * add require_variance keyword to precomputedfeatures call * add s3 download if not present to test helper * add missing s3 import * Update test_helper.py * pull item from ceiling Need to check with martin if this will break anything * update coggan benchmark to work w/ new pandas/python * add identifier function to pre computed features * forgot to add string line * update rajalingham2018 deprecated test * update w/ master (#1194) * add r101_eBarlow_lmda_01_1 to models (#1037) Co-authored-by: AutoJenkins <[email protected]> * update hmax requirements.txt (#1053) * update hmax requirements.txt to add torchvision (#1054) * fix mobilenet `pooler` layer prefix (#1055) * add r34_eMMCR_Mom_lmda_02_1 to models (#1061) Co-authored-by: AutoJenkins <[email protected]> * add r101_eBarlow_lmda_02_1_copy to models (#1059) Co-authored-by: AutoJenkins <[email protected]> * add r34_eMMCR_Mom_lmda_01_1 to models (#1060) Co-authored-by: AutoJenkins <[email protected]> * add r34_eMMCR_Mom_Vanilla_1 to models (#1063) Co-authored-by: AutoJenkins <[email protected]> * ignore time dimension in Bracci2019 (#1064) * Update behavior.py (#742) Co-authored-by: Sam Winebrake <[email protected]> Co-authored-by: deirdre-k <[email protected]> * add eBarlow_lmda_02_1_full to models (#1067) Co-authored-by: AutoJenkins <[email protected]> * oddoneout: build full stimulus id index for non-numeric ids (#1068) * Add mobilevit_small - take 2 (#1051) * Add mobilevit_small - take 2 * Update brainscore_vision/models/mobilevit_small/model.py Co-authored-by: Martin Schrimpf <[email protected]> --------- Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> * add yudixie_resnet18_240719_1 to models (#1070) Co-authored-by: AutoJenkins <[email protected]> * add yudixie_resnet18_240719_0 to models (#1069) Co-authored-by: AutoJenkins <[email protected]> * add yudixie_resnet18_240719_2 to models (#1071) Co-authored-by: AutoJenkins <[email protected]> * add yudixie_resnet18_240719_10 to models (#1079) Co-authored-by: AutoJenkins <[email protected]> * Add cv_18_dagger_408_pretrained (#1104) * Add model files * Adjust timm version --------- Co-authored-by: Ethan Pellegrini <[email protected]> * add eBarlow_lmda_02_200_full to models (#1121) Co-authored-by: AutoJenkins <[email protected]> * remove old tutorials in favor of new link (#1170) --------- Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Linus Sommer <[email protected]> Co-authored-by: deirdre-k <[email protected]> Co-authored-by: Kartik Pradeepan <[email protected]> Co-authored-by: pellegreene <[email protected]> Co-authored-by: Ethan Pellegrini <[email protected]> * environment lock for 2.1.0 * Rename environment.yml to environment_lock.yml * add test dependencies and move certain dependencies to pip * remove branch from core dependency * new env lock with all dependencies of env (including test) * add informative comment to env lock * Update .travis.yml * point to 3.11 branches * update pointers to 3.11 branches * add back check for forks in travis * remove f string from full_name variable * remove if/else that enabled tf * remove folder_name for models that were moved to correct directory * version less specific (2.1.0 to 2.1) * remove protobuf * remove python specification in setup.pys * Update mismatched s3 folder name * Update environment_lock.yml w/ main branches * update branch pointers to main --------- Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Linus Sommer <[email protected]> Co-authored-by: deirdre-k <[email protected]> Co-authored-by: Kartik Pradeepan <[email protected]> Co-authored-by: Deirdre Kelliher <[email protected]> Co-authored-by: pellegreene <[email protected]> Co-authored-by: Ethan Pellegrini <[email protected]> --------- Co-authored-by: David Coggan <[email protected]> Co-authored-by: ddcoggan <[email protected]> Co-authored-by: YingtianDt <[email protected]> Co-authored-by: Yingtian Tang <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: Martin Schrimpf <[email protected]> Co-authored-by: deirdre-k <[email protected]> Co-authored-by: Michael Ferguson <[email protected]> Co-authored-by: Sam Winebrake <[email protected]> Co-authored-by: Katherine Fairchild <[email protected]> Co-authored-by: AutoJenkins <[email protected]> Co-authored-by: Linus Sommer <[email protected]> Co-authored-by: Deirdre Kelliher <[email protected]> Co-authored-by: pellegreene <[email protected]> Co-authored-by: Ethan Pellegrini <[email protected]>
brain-score · Sep 11, 2024 · d8eddfc · d8eddfc
1 parent dda00c2
commit d8eddfc
Show file tree

Hide file tree

Showing 105 changed files with 629 additions and 417 deletions.
diff --git a/.github/workflows/score_new_plugins.yml b/.github/workflows/score_new_plugins.yml
@@ -32,10 +32,10 @@ jobs:
         with:
           fetch-depth: 0
 
-      - name: Set up Python 3.7
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.7
+          python-version: 3.11
 
       - name: Configure AWS Credentials
         uses: aws-actions/configure-aws-credentials@v1
@@ -176,10 +176,10 @@ jobs:
       - name: Check out repository code
         uses: actions/checkout@v4
 
-      - name: Set up Python 3.7
+      - name: Set up Python 3.11
         uses: actions/setup-python@v4
         with:
-          python-version: 3.7
+          python-version: 3.11
 
       - name: Build project
         run: |

diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -3,7 +3,7 @@ version: 2
 build:
   os: "ubuntu-20.04"
   tools:
-   python: "3.7"
+   python: "3.11"
 
 python:
    install:

diff --git a/.travis.yml b/.travis.yml
@@ -1,5 +1,6 @@
 version: ~> 1.0
 language: python
+dist: jammy
 env:
   global:
     - PYTEST_SETTINGS="not requires_gpu and not memory_intense and not slow and not travis_slow"
@@ -9,7 +10,7 @@ env:
     - WEB_SUBMISSION="False"
 before_install:
 - pip install --upgrade pip
-- pip install setuptools==60.5.0
+- pip install setuptools
 - pip install pytest
 # download large files
 - pip install awscli
@@ -31,18 +32,18 @@ import:
 
 jobs:
   include:
-    - name: 3.7 public
-      python: '3.7.13'
-    - name: 3.7 private
+    - name: 3.11 public
+      python: '3.11'
+    - name: 3.11 private
       if: fork = false
-      python: '3.7.13'
+      python: '3.11'
       env:
       - PRIVATE_ACCESS=1
       - secure: f1rWEwrslh7qa2g/QlKs001sGC3uaOxZNQSfNOPj+TMCqEo2c6OzImC4hyz+WqCyc6N/lFT4yYo2RhvaqStHMRmu/+9aZmuH05Bb0KQpfzNFA+yGa/U5WR3/4u6KRvDAeNEi9drT2LuacTyGbldmQsquujK0jrPpFWpe7zUUKv0zb0lJf0zcjeSrZlDXLlgD6DCqow7OqHRvW04dPZVy1OArRwtPV6DJ6Rqo1MqFQGHJ806VPlXhSoydb7a58dhGajqPjomdmZjhd3wS6Lv6uetTE/VVb4EP4e7n0qfZIx/TpnWG0SR44pcP7OCNARWYANsAivzxnQ0shyXnIzOo8ZcPYiPpt/5D53i5idTBxXyuDaHGQvgwuY5XLZzznEedBgZa4OvjxAXlLEQjdVDfSsZeYaV9gyFkeTlLnK1zvWi0US38eF2Qtm3Sx3D/5TtBKK2n38tyK5gg/XvJNycaXvIl7iVcnI2ifpqD1mUWI6C9j9Tk19/XEpWkwaFi91+0LZF1GhjBu8o3G5Np4RIOKXi3TIHkpbMM5mf11T6Bm9LvEMq1h8bgRQigEbeJF8CbUOSVFv+AaXsggGjQhuwdyvy2JZo+tO1nfhi+kW3XrDGPsz1R7Wfqduyn7UUh5OiFymeZwKseYKnwU47KyCqDwrq5Mnx1MlSidnVmPriadR4=
       - secure: WE7FPwy07VzJTKAd2xwZdBhtmh8jk7ojwk4B2rIcBQu0vwUXc1MgO8tBLD7s08lBedBjqZiLZEW31uPMEyWNysouDt16a5gm2d149LR7flI3MOifBtxINfJuC3eOEG65bPgN/bYEsIpLKnu3469d5nxZkK7xsjbWTxHGoUpLvVPsmHY2ZM5/jftybs7fI0do4NMG2XffKfZbiFb447Ao3xeQeEfW6IkJllzgGnlG9FJATFidrbwDNdmzAnvPEnDoKAf7ZvhPV0x9yR5V6P4Ck5hxl8mlPdBa1cRMO8s/1ag1c7YJ3AF9ZlwcwqTiGsT8DHTVRxSz4nFHJTMlrm9j84u7WzLZJBhPgF0UeLN3AQgiAZ3c2TFDvjQWeHVuSPkV5GrKlfhSvR82s9yPEdHQxxwYymBbAr6rJR4NtXTyZX0vg8NRKHssZKLSafs/D/pt9xXspqu8HAHc+mS0lCips79XptSr5BEsioil3D2io3tbzrGugpTeJ7oEA787vKn2Cm4XmhyQ0UBhvwsPZ351l27wZYuNV07o9Ik83hN/w4o2v899QQ/zbX42Iy8ZUCWOPX7MV7+TA7SMxru3qx7HL5hDM8kTetxbLB6Ckr+JOdX8L2Fb5L3TVDpsvfv0ebXgwaQR/ez8/7bcXmBqcERApHDz73HaMXUap+iDR4FLdXE=
       - AWS_DEFAULT_REGION=us-east-1
     - stage: "Automerge check"
-      python: '3.7.13'
+      python: '3.11'
       install:
       - pip install --no-cache-dir torch torchvision --default-timeout=1000 --retries=5
       - pip install --no-cache-dir -e ".[test]"

diff --git a/README.md b/README.md
@@ -22,7 +22,7 @@ To contribute, please [send in a pull request](https://github.com/brain-score/vi
 
 ## Local installation
 
-You will need Python = 3.7 and pip >= 18.1.
+You will need Python = 3.11 and pip >= 18.1.
 
 `pip install git+https://github.com/brain-score/vision`
 

diff --git a/brainscore_vision/benchmark_helpers/__init__.py b/brainscore_vision/benchmark_helpers/__init__.py
@@ -1,6 +1,7 @@
 from typing import Union
 
 import numpy as np
+import hashlib
 
 from brainio.assemblies import NeuroidAssembly, DataAssembly
 from brainscore_core import Score
@@ -18,6 +19,13 @@ def __init__(self, features: Union[DataAssembly, dict], visual_degrees):
         self.features = features
         self._visual_degrees = visual_degrees
 
+    @property
+    def identifier(self) -> str:
+        # serialize the features to a string and create hash
+        features_data = str(self.features)
+        features_hash = hashlib.md5(features_data.encode('utf-8')).hexdigest()
+        return f"precomputed-{features_hash}"
+
     def visual_degrees(self) -> int:
         return self._visual_degrees
 

diff --git a/brainscore_vision/benchmark_helpers/test_helper.py b/brainscore_vision/benchmark_helpers/test_helper.py
@@ -7,6 +7,7 @@
 from brainio.assemblies import NeuroidAssembly, PropertyAssembly
 from brainscore_vision import load_benchmark
 from brainscore_vision.model_interface import BrainModel
+from brainscore_vision.data_helpers import s3
 from . import PrecomputedFeatures
 
 
@@ -68,6 +69,8 @@ def run_test_properties(self, benchmark: str, files: dict, expected: float):
         for current_stimulus in stimulus_identifiers:
             stimulus_set = load_stimulus_set(current_stimulus)
             path = Path(__file__).parent / files[current_stimulus]
+            s3.download_file_if_not_exists(local_path=path,
+                               bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{files[current_stimulus]}')
             features = PropertyAssembly.from_files(path,
                                                    stimulus_set_identifier=stimulus_set.identifier,
                                                    stimulus_set=stimulus_set)

diff --git a/brainscore_vision/benchmarks/coggan2024_behavior/benchmark.py b/brainscore_vision/benchmarks/coggan2024_behavior/benchmark.py
@@ -75,9 +75,12 @@ def __call__(self, candidate: BrainModel) -> Score:
             data.model_prediction == data.object_class, dtype=int)
 
         # get correlation between model and human performance across conditions
-        performance = (data[data.visibility < 1]
+        performance = (
+            data[data.visibility < 1]
             .groupby(['subject', 'occluder_type', 'occluder_color'])
-            .mean(['human_accuracy', 'model_accuracy'])).reset_index()
+            .mean(numeric_only=True)
+            .reset_index()
+        )
         scores = performance.groupby('subject').apply(
             lambda df: np.corrcoef(df.human_accuracy, df.model_accuracy)[0, 1])
         score = Score(np.mean(scores))
@@ -100,8 +103,9 @@ def get_noise_ceiling(performance: pd.DataFrame) -> Score:
     nc = []
     for subject in performance.subject.unique():
         performance_ind = performance[performance.subject == subject]
-        performance_grp = (performance[performance.subject != subject]
-            .groupby(['occluder_type', 'occluder_color']).mean())
+        performance_grp = performance[performance.subject != subject]
+        numeric_cols = performance_grp.select_dtypes(include=np.number).columns
+        performance_grp = performance_grp.groupby(['occluder_type', 'occluder_color'])[numeric_cols].mean()
         merged_df = performance_ind.merge(
             performance_grp, on=['occluder_type', 'occluder_color'])
         nc.append(np.corrcoef(merged_df.human_accuracy_x, merged_df.human_accuracy_y)[0, 1])

diff --git a/brainscore_vision/benchmarks/igustibagus2024/domain_transfer_analysis.py b/brainscore_vision/benchmarks/igustibagus2024/domain_transfer_analysis.py
@@ -5,6 +5,8 @@
 import pandas as pd
 from sklearn.linear_model import RidgeClassifierCV
 from sklearn.model_selection import train_test_split
+from sklearn.pipeline import Pipeline
+from sklearn.preprocessing import StandardScaler
 from tqdm import tqdm
 
 # import brain-score specific libraries
@@ -89,7 +91,10 @@ def __call__(self, candidate: BrainModel) -> Score:
 
 def OOD_AnalysisBenchmark():
     return _OOD_AnalysisBenchmark(
-        classifier=RidgeClassifierCV(alphas=[0.0001, 0.001, 0.01, 0.1, 1, 10], fit_intercept=True, normalize=True)
+        classifier=Pipeline([
+            ('scaler', StandardScaler()), 
+            ('classifier', RidgeClassifierCV(alphas=[0.0001, 0.001, 0.01, 0.1, 1, 10], fit_intercept=True))
+        ])
     )
 
 

diff --git a/brainscore_vision/benchmarks/kar2019/test.py b/brainscore_vision/benchmarks/kar2019/test.py
@@ -24,7 +24,7 @@ def test_Kar2019ost_cornet_s():
     filename = 'cornet_s-kar2019.nc'
     filepath = Path(__file__).parent / filename
     s3.download_file_if_not_exists(local_path=filepath,
-                                   bucket='brainio-brainscore', remote_filepath=f'tests/test_benchmarks/{filename}')
+                                   bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
     precomputed_features = NeuroidAssembly.from_files(
         filepath,
         stimulus_set_identifier=benchmark._assembly.stimulus_set.identifier,

diff --git a/brainscore_vision/benchmarks/majajhong2015/__init__.py b/brainscore_vision/benchmarks/majajhong2015/__init__.py
@@ -11,3 +11,8 @@
 
 benchmark_registry['MajajHong2015public.V4-pls'] = MajajHongV4PublicBenchmark
 benchmark_registry['MajajHong2015public.IT-pls'] = MajajHongITPublicBenchmark
+
+# temporal
+from .benchmark import MajajHongV4TemporalPublicBenchmark, MajajHongITTemporalPublicBenchmark
+benchmark_registry['MajajHong2015public.V4-temporal-pls'] = lambda: MajajHongV4TemporalPublicBenchmark(time_interval=10)
+benchmark_registry['MajajHong2015public.IT-temporal-pls'] = lambda: MajajHongITTemporalPublicBenchmark(time_interval=10)
diff --git a/brainscore_vision/benchmarks/majajhong2015/benchmark.py b/brainscore_vision/benchmarks/majajhong2015/benchmark.py
@@ -1,7 +1,8 @@
 from brainscore_core import Metric
 
 from brainscore_vision import load_metric, Ceiling, load_ceiling, load_dataset
-from brainscore_vision.benchmark_helpers.neural_common import NeuralBenchmark, average_repetition
+from brainscore_vision.benchmark_helpers.neural_common import NeuralBenchmark, average_repetition, apply_keep_attrs
+from brainscore_vision.model_helpers.brain_transformation.temporal import assembly_time_align
 
 VISUAL_DEGREES = 8
 NUMBER_OF_TRIALS = 50
@@ -20,13 +21,14 @@
             eprint = {https://www.jneurosci.org/content/35/39/13402.full.pdf},
             journal = {Journal of Neuroscience}}"""
 
-pls_metric = lambda: load_metric('pls', crossvalidation_kwargs=dict(stratification_coord='object_name'))
-
+crossvalidation_kwargs = dict(stratification_coord='object_name')
+pls_metric = lambda: load_metric('pls', crossvalidation_kwargs=crossvalidation_kwargs)
+spantime_pls_metric = lambda: load_metric('spantime_pls', crossvalidation_kwargs=crossvalidation_kwargs)
 
 def _DicarloMajajHong2015Region(region: str, access: str, identifier_metric_suffix: str,
-                                similarity_metric: Metric, ceiler: Ceiling):
-    assembly_repetition = load_assembly(average_repetitions=False, region=region, access=access)
-    assembly = load_assembly(average_repetitions=True, region=region, access=access)
+                                similarity_metric: Metric, ceiler: Ceiling, time_interval: float = None):
+    assembly_repetition = load_assembly(average_repetitions=False, region=region, access=access, time_interval=time_interval)
+    assembly = load_assembly(average_repetitions=True, region=region, access=access, time_interval=time_interval)
     benchmark_identifier = f'MajajHong2015.{region}' + ('.public' if access == 'public' else '')
     return NeuralBenchmark(identifier=f'{benchmark_identifier}-{identifier_metric_suffix}', version=3,
                            assembly=assembly, similarity_metric=similarity_metric,
@@ -60,13 +62,35 @@ def MajajHongITPublicBenchmark():
                                        ceiler=load_ceiling('internal_consistency'))
 
 
-def load_assembly(average_repetitions, region, access='private'):
-    assembly = load_dataset(f'MajajHong2015.{access}')
+def MajajHongV4TemporalPublicBenchmark(time_interval: float = None):
+    return _DicarloMajajHong2015Region(region='V4', access='public', identifier_metric_suffix='pls',
+                                       similarity_metric=spantime_pls_metric(), time_interval=time_interval,
+                                       ceiler=load_ceiling('internal_consistency_temporal'))
+
+
+def MajajHongITTemporalPublicBenchmark(time_interval: float = None):
+    return _DicarloMajajHong2015Region(region='IT', access='public', identifier_metric_suffix='pls',
+                                       similarity_metric=spantime_pls_metric(), time_interval=time_interval,
+                                       ceiler=load_ceiling('internal_consistency_temporal'))
+
+
+def load_assembly(average_repetitions: bool, region: str, access: str = 'private', time_interval: float = None):
+    temporal = time_interval is not None
+    if not temporal:
+        assembly = load_dataset(f'MajajHong2015.{access}')
+        assembly = assembly.squeeze("time_bin")
+    else:
+        assembly = load_dataset(f'MajajHong2015.temporal.{access}')
+        assembly = assembly.__class__(assembly)
+        target_time_bins = [
+            (t, t+time_interval) for t in range(0, assembly.time_bin_end.max().item()-time_interval, time_interval)
+        ]
+        assembly = apply_keep_attrs(assembly, lambda assembly: assembly_time_align(assembly, target_time_bins))
+
     assembly = assembly.sel(region=region)
     assembly['region'] = 'neuroid', [region] * len(assembly['neuroid'])
-    assembly = assembly.squeeze("time_bin")
     assembly.load()
-    assembly = assembly.transpose('presentation', 'neuroid')
+    assembly = assembly.transpose('presentation', 'neuroid', ...)
     if average_repetitions:
         assembly = average_repetition(assembly)
     return assembly
diff --git a/brainscore_vision/benchmarks/rajalingham2018/test.py b/brainscore_vision/benchmarks/rajalingham2018/test.py
@@ -7,7 +7,7 @@
 from pytest import approx
 
 from brainio.assemblies import BehavioralAssembly
-from brainscore_vision import benchmark_registry, load_benchmark, load_metric
+from brainscore_vision import benchmark_registry, load_benchmark, load_metric, load_model
 from brainscore_vision.benchmark_helpers import PrecomputedFeatures
 from brainscore_vision.benchmark_helpers.test_helper import VisualDegreesTests, NumberOfTrialsTests
 from brainscore_vision.benchmarks.rajalingham2018 import DicarloRajalingham2018I2n
@@ -115,44 +115,11 @@ class TestMetricScore:
     @pytest.mark.parametrize(['model', 'expected_score'],
                              [
                                  ('alexnet', .253),
-                                 ('resnet34', .37787),
-                                 ('resnet18', .3638),
+                                 ('resnet50_tutorial', 0.348),
+                                 ('pixels', 0.0139)
                              ])
     def test_model(self, model, expected_score):
-        class UnceiledBenchmark(_DicarloRajalingham2018):
-            def __init__(self):
-                metric = load_metric('i2n')
-                super(UnceiledBenchmark, self).__init__(metric=metric, metric_identifier='i2n')
-
-            def __call__(self, candidate: BrainModel):
-                candidate.start_task(BrainModel.Task.probabilities, self._fitting_stimuli)
-                probabilities = candidate.look_at(self._assembly.stimulus_set)
-                score = self._metric(probabilities, self._assembly)
-                return score
-
-        benchmark = UnceiledBenchmark()
-        # features
-        feature_responses = xr.load_dataarray(Path(__file__).parent / 'test_resources' /
-                                              f'identifier={model},stimuli_identifier=objectome-240.nc')
-        feature_responses['stimulus_id'] = 'stimulus_path', [os.path.splitext(os.path.basename(path))[0]
-                                                             for path in feature_responses['stimulus_path'].values]
-        feature_responses = feature_responses.stack(presentation=['stimulus_path'])
-        assert len(np.unique(feature_responses['layer'])) == 1  # only penultimate layer
-
-        class PrecomputedFeatures:
-            def __init__(self, precomputed_features):
-                self.features = precomputed_features
-
-            def __call__(self, stimuli, layers):
-                np.testing.assert_array_equal(layers, ['behavioral-layer'])
-                self_stimulus_ids = self.features['stimulus_id'].values.tolist()
-                indices = [self_stimulus_ids.index(stimulus_id) for stimulus_id in stimuli['stimulus_id'].values]
-                features = self.features[{'presentation': indices}]
-                return features
-
-        # evaluate candidate
-        transformation = ProbabilitiesMapping(identifier=f'TestI2N.{model}',
-                                              activations_model=PrecomputedFeatures(feature_responses),
-                                              layer='behavioral-layer')
-        score = benchmark(transformation)
-        assert score == approx(expected_score, abs=0.005), f"expected {expected_score}, but got {score}"
+        benchmark = load_benchmark('Rajalingham2018-i2n')
+        model = load_model(model)
+        score = benchmark(model)
+        assert score.raw == approx(expected_score, abs=0.005), f"expected {expected_score}, but got {score.raw}"
diff --git a/brainscore_vision/benchmarks/rajalingham2020/test.py b/brainscore_vision/benchmarks/rajalingham2020/test.py
@@ -35,5 +35,5 @@ def test_Rajalingham2020(benchmark, expected):
     filename = 'alexnet-rajalingham2020-features.12.nc'
     filepath = Path(__file__).parent / filename
     s3.download_file_if_not_exists(local_path=filepath,
-                                   bucket='brainio-brainscore', remote_filepath=f'tests/test_benchmarks/{filename}')
+                                   bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
     precomputed_test.run_test(benchmark=benchmark, precomputed_features_filepath=filepath, expected=expected)
diff --git a/brainscore_vision/benchmarks/sanghavi2020/test.py b/brainscore_vision/benchmarks/sanghavi2020/test.py
@@ -66,7 +66,7 @@ def test_self_regression(benchmark, visual_degrees, expected):
 def test_model_features(benchmark, filename, expected):
     filepath = Path(__file__).parent / filename
     s3.download_file_if_not_exists(local_path=filepath,
-                                   bucket='brainio-brainscore', remote_filepath=f'tests/test_benchmarks/{filename}')
+                                   bucket='brainscore-unittests', remote_filepath=f'tests/test_benchmarks/{filename}')
     precomputed_test.run_test(benchmark=benchmark, precomputed_features_filepath=filepath, expected=expected)
 
 

diff --git a/brainscore_vision/data/geirhos2021/test.py b/brainscore_vision/data/geirhos2021/test.py
@@ -62,7 +62,7 @@ def test_stimulus_set_assembly_alignment(self, identifier, field):
         full_name = f"Geirhos2021_{identifier}"
         assembly = load_dataset(full_name)
         assert assembly.stimulus_set is not None
-        assert assembly.stimulus_set.identifier == f"{full_name}"
+        assert assembly.stimulus_set.identifier == full_name
         assert set(assembly.stimulus_set[field]) == set(assembly[field].values)
 
     # test the number of subjects:
@@ -236,7 +236,7 @@ def test_stimulus_set_exist(self, identifier):
         full_name = f"Geirhos2021_{identifier}"
         stimulus_set = load_stimulus_set(full_name)
         assert stimulus_set is not None
-        assert stimulus_set.identifier == full_name
+        assert stimulus_set.identifier == f"{full_name}"
 
     # test the number of images
     @pytest.mark.parametrize('identifier, num_images', [

diff --git a/brainscore_vision/data/scialom2024/test.py b/brainscore_vision/data/scialom2024/test.py
@@ -258,7 +258,7 @@ def test_stimulus_set_exists(self, identifier):
     ])
     def test_number_of_images(self, identifier, num_images):
         stimulus_set = load_stimulus_set(identifier)
-        assert len(np.unique(stimulus_set['image_id'].values)) == num_images
+        assert len(np.unique(stimulus_set['stimulus_id'].values)) == num_images
 
     # test assembly coords present in ALL 17 sets:
     @pytest.mark.parametrize('identifier', [