Developer continuous v3 #95

enryH · 2024-06-18T13:40:11Z

No description provided.

- new preprocessing - work on cont. associations

- checking different config settings. Highlights: - preprocessing.py: feature_min_max function -perturbations.py: perturb...extended: change target_dataset feature by feature for min/max - identify_associations.py: branched flow depending on target_value

- *preprocessing.py:* feature_min_max to feature_stats (added std) -* perturbations.py:* added std (1 for now) feature by feature -*identify_associations.py:* added predefined list of cont target values

It is used in: - encode data (show before and after preprocessing) - identify associations (plot each feature after perturbation)

Reorganizing the file identify_associations: - ttest and bayes functions put outside - dataloader preparation defined in functions - save results function added - single identify_associations function - Most comments addressed

Reformat constant CONTINUOUS_TARGET_VALUES

Reused code put in main function: Identify associations Working branch for both modes (Continuous assoc finds self correlations)

- Tested bayes and ttest on new data - Added config files for continuous test TODO: Create folder with files to create synthetic datasets

- Add GPU compatibility - Fix typing - Make test dataloader batch size configurable - Add extra columns to output table

- Exploring bayes behaviour on continuous pert - Plot feature associations and vae architecture as graphs - Use VSCode debugger (json edited)

Added ks method to calculate distances

Functioning Kolmogorov Smirnov method - QC of features is given as separate csv from KS scores - schema: feature names of feature to visualize added - Basic visualization functions added to dataset_distribution.py

- 🐛 fix incomplete publishing package file - 🎨 format src files

- set max-line-length to 120

- in case of no categorical variables, the error surfaced: input_config has to be used!

double check type hints - also FloatArray is not very informative about the layout of the array (2D, 3D, 4D?)

…nLab/MOVE into developer-continuous-v3

https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#flake8

- play also with sample data configuration

- f-string formatting error.

- check if less latent dimensions reduces the t-test runtime

ri-heme and others added 30 commits January 5, 2023 16:26

🔊 Capture warnings

50bfa61

🐛 VAE and data objects handle missing data type

63f7c2f

🚧 Update tasks for missing data type

5c0c90b

🚧 Committing provisional changes (not finished)

d831a49

🚧 min max feature pertubation

b5e1044

- new preprocessing - work on cont. associations

Last batch of changes (unfinished)

ff8f8aa

preprocessing changes

26bb304

Version that reaches training

aa28897

last version

0826212

Changes on writing the final files

98a5dfa

Last commit:

c838976

- checking different config settings. Highlights: - preprocessing.py: feature_min_max function -perturbations.py: perturb...extended: change target_dataset feature by feature for min/max - identify_associations.py: branched flow depending on target_value

# Added plus_std and minus_std as target values

9d560bc

- *preprocessing.py:* feature_min_max to feature_stats (added std) -* perturbations.py:* added std (1 for now) feature by feature -*identify_associations.py:* added predefined list of cont target values

New changes for plotting new features

081e737

Note in funtion description updated

85ce21f

Perturbation visualization added:

337ddc5

It is used in: - encode data (show before and after preprocessing) - identify associations (plot each feature after perturbation)

:build: Perturbation visualization

c7b4f1f

🎨 ✨

f02c63e

Reorganizing the file identify_associations: - ttest and bayes functions put outside - dataloader preparation defined in functions - save results function added - single identify_associations function - Most comments addressed

📝 Make output_subpath optional

5248274

Reformat constant CONTINUOUS_TARGET_VALUES

⏪ Revert styling in conflicting files

79b2849

🎨 Rename results directory

1036656

🎨

71dd894

Reused code put in main function: Identify associations Working branch for both modes (Continuous assoc finds self correlations)

✨ Random continuous dataset added.

c9cf123

- Tested bayes and ttest on new data - Added config files for continuous test TODO: Create folder with files to create synthetic datasets

🎨 Type hinting

a3ddf48

✨ Update ID associations module

1c64573

- Add GPU compatibility - Fix typing - Make test dataloader batch size configurable - Add extra columns to output table

🎨 ♻️ Refactoring/styling

59a6d7a

🔧 Update default config

0530139

✨ 🚧 🧐

6fd7589

- Exploring bayes behaviour on continuous pert - Plot feature associations and vae architecture as graphs - Use VSCode debugger (json edited)

Added random_basic config files

c83fb42

🚧

3300a86

Added ks method to calculate distances

✨ 🚧

2cac930

Functioning Kolmogorov Smirnov method - QC of features is given as separate csv from KS scores - schema: feature names of feature to visualize added - Basic visualization functions added to dataset_distribution.py

Henry Webel and others added 28 commits May 16, 2024 09:39

Merge branch 'developer' into developer-continuous-v3

fb0dd98

✨ add workflow for formatting checks

4164f41

- 🐛 fix incomplete publishing package file - 🎨 format src files

🎨 format files

439c8cf

✨ lint src files with flake8 - might introduce regression errors

3ff9842

- set max-line-length to 120

🐛 configure flake8 for black formatting

d722c0b

🐛 ⏪ HYDRA_VERSION_BASE needed for imports

a7e6275

🐛 regression: merging branches lead to name mismatch

6184235

- in case of no categorical variables, the error surfaced: input_config has to be used!

🐛 correct value in yaml file

daa74b7

✅ Add integration test based on tutorial

e6368a9

🎨 format yaml files (some had no last blank line)

0542ff2

🎨 add typehints and always return figure for plotting fcts

087dcb7

double check type hints - also FloatArray is not very informative about the layout of the array (2D, 3D, 4D?)

🎨 split bayes and t-test run in CI

2b5b098

🎨 format file

2cb9aa5

⏪ random small defaults restored

0dc2a0f

🔧 Add default KS config

25fc96b

🔧 Use default KS config

82c6624

⚡ reduce action runtime by configuring tasks

09ca32f

🎨 Make sure correct colormap is used

6a02d0e

Merge branch 'developer-continuous-v3' of https://github.com/Rasmusse…

2a12ed1

…nLab/MOVE into developer-continuous-v3

🔧 udpate flake8 configuration according to defaults

2e1c6ba

https://black.readthedocs.io/en/stable/guides/using_black_with_other_tools.html#flake8

✨ dry-run continuous configuration sample data

6207350

- play also with sample data configuration

🎨 make line 88 characters long, remove some expections from flake8

e4e2283

🐛 error due to black formatting?

5e23447

- f-string formatting error.

✨ run continous example

7796f5e

- check if less latent dimensions reduces the t-test runtime

🐛 four latent dimensions are required for t-test

48a76b7

⚡ run latent job, try to increase speed for t-test

5533eca

⚡ speed up and balance both jobs' runtime

d0a4409

Merge branch 'developer' into developer-continuous-v3

066550d

enryH closed this Jun 20, 2024

enryH deleted the developer-continuous-v3 branch June 20, 2024 09:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Developer continuous v3 #95

Developer continuous v3 #95

enryH commented Jun 18, 2024

Developer continuous v3 #95

Developer continuous v3 #95

Conversation

enryH commented Jun 18, 2024