Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EO Development (draft pull request) #3

Open
wants to merge 52 commits into
base: master
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
36007ea
Merge remote-tracking branch 'origin/development' into development-eo
annajungbluth Apr 25, 2024
0c2b26f
started eo training development
annajungbluth Apr 25, 2024
9993839
started testing training pipeline
annajungbluth Apr 25, 2024
3330469
wip - tested training pipeline
annajungbluth Apr 25, 2024
9071b24
made training pipeline run
annajungbluth Apr 26, 2024
d7f2c9d
moved geo dataset and editors into ITI repo
annajungbluth Apr 26, 2024
b6983b2
added new editor and modified training script
annajungbluth Apr 27, 2024
2879551
modified editor
annajungbluth Apr 27, 2024
12ee80e
tested callbacks
annajungbluth Apr 28, 2024
cdc4c7c
added normalisation steps to training script, and started writing a n…
lillif May 13, 2024
92c26d6
training script for miniset set up, mean std normalisation finished
lillif May 14, 2024
a18e68a
normalisation script finished, attempted training
lillif May 17, 2024
9d592bb
started hydra training file
annajungbluth May 19, 2024
6d24344
added normalization and fixed bugs in training script
annajungbluth May 20, 2024
1ddd8d5
merge with master
annajungbluth Oct 3, 2024
d7b31d0
fixed small merge bugs and added autoroot file
annajungbluth Oct 3, 2024
ad54e8e
Added file with dataset information
annajungbluth Oct 4, 2024
72c3f82
fixed goes metrics file
annajungbluth Oct 18, 2024
4533b53
updated summary files
annajungbluth Oct 20, 2024
2e72ef8
added new normalization routine and started first experiment
annajungbluth Oct 20, 2024
7a79a1e
reduced val data
annajungbluth Oct 20, 2024
d3a4ad4
optimized dataloader to reduce memory consumption
annajungbluth Oct 24, 2024
e2c5675
added normalization files for subset of data
annajungbluth Oct 31, 2024
625852a
debugging constant channels
annajungbluth Oct 31, 2024
8770404
started new experiment
annajungbluth Oct 31, 2024
5b98ee1
added seed to training script
annajungbluth Nov 1, 2024
e586a44
added min max normalizer
annajungbluth Nov 18, 2024
3a9c243
updated center weighted cropping routine for better cropping
annajungbluth Nov 22, 2024
94ea2cd
removed file with large missing band
annajungbluth Nov 25, 2024
943fe17
started miniset experiment
annajungbluth Nov 25, 2024
520415b
removed files with pixel strips missing
annajungbluth Nov 25, 2024
69f5f68
removed files with missing or half missing channels
annajungbluth Nov 25, 2024
e61dca0
added files for miniset and removed files with artifacts
annajungbluth Nov 25, 2024
cc5031e
fixed subser summary file
annajungbluth Nov 27, 2024
2c17c29
updated config
annajungbluth Nov 27, 2024
479bb93
fixed mistake in notebook
annajungbluth Nov 27, 2024
eea208d
added script for goes-to-msg translation
annajungbluth Dec 6, 2024
2a17687
updated miniset
annajungbluth Dec 15, 2024
4b3caf0
added separation of A and B patch size
annajungbluth Dec 15, 2024
dad7444
modified config for miniset experiment
annajungbluth Dec 15, 2024
e6af551
experiment modifications
annajungbluth Dec 15, 2024
b4d3b62
merged with changes adding rotation transform
annajungbluth Dec 15, 2024
1fa9544
modified dataloader to handle no cropping
annajungbluth Dec 17, 2024
5eb95f8
updated config for experimentation with miniset
annajungbluth Dec 17, 2024
76dfecb
fixed problem with normalization and fixed experiment
annajungbluth Dec 19, 2024
4e37c53
added storage dataset
annajungbluth Dec 19, 2024
1bc1c8c
started new experiment for all infrared channels
annajungbluth Dec 21, 2024
f0ef3da
started new experiment with visible channels
annajungbluth Dec 21, 2024
399bfa6
started new experiment with visible channels
annajungbluth Dec 21, 2024
6a6641e
deleted testing code
annajungbluth Dec 21, 2024
5a5482c
updated config for new experiment
annajungbluth Dec 23, 2024
2d35d61
merged configs
annajungbluth Dec 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
modified config for miniset experiment
annajungbluth committed Dec 15, 2024
commit dad74443783f06e85f2768f15fd2c9a3340a560c
16 changes: 8 additions & 8 deletions config/msg_to_goes_miniset.yaml
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
base_dir: /home/anna.jungbluth/outputs/msg-to-goes-miniset/
data:
A_path: /mnt/disks/eo-miniset/msg/
B_path: /mnt/disks/eo-miniset/goes/
A_path: /mnt/disks/eo-miniset/geoprocessed/msg/
B_path: /mnt/disks/eo-miniset/geoprocessed/goes/
num_workers: 4
iterations_per_epoch: 40
A_patch_size: (256, 256)
B_patch_size: (256, 256)
iterations_per_epoch: 50
A_patch_size: (150, 150)
B_patch_size: (450, 450)
A_bands: [10.8] # [0.64, 0.81, 1.64, 3.92, 6.25, 7.35, 8.7, 9.66, 10.8, 12.0, 13.4]
B_bands: [10.33] # [0.47, 0.64, 0.87, 1.38, 1.61, 2.25, 3.89, 6.17, 6.93, 7.34, 8.44, 9.61, 10.33, 11.19, 12.27, 13.27]
model:
input_dim_a: 1
input_dim_b: 1
upsampling: 0
upsampling: 3
discriminator_mode: SINGLE
lambda_diversity: 0
norm: 'in_rs_aff'
norm: 'in_aff'
use_batch_statistic: False
logging:
wandb_entity: itieo
wandb_project: msg-to-goes
wandb_name: "[miniset]-MSG_to_GOES-infrared"
wandb_name: "[miniset]-MSG_to_GOES-infrared-upsampled (10.8 um) ["in_aff" norm]"
training:
epochs: 200
limit_train_batches: null
14 changes: 14 additions & 0 deletions itipy/data/geo_editor.py
Original file line number Diff line number Diff line change
@@ -219,6 +219,20 @@ def call(self, data_dict, **kwargs):
# Update dictionary
data_dict[self.key] = data
return data_dict

class Rotate180Editor(Editor):
"""
Rotate data by 180 degrees
"""
def __init__(self, key="data"):
self.key = key
def call(self, data_dict, **kwargs):
data = data_dict[self.key]
# Rotate data
data = np.rot90(data, k=2, axes=(1, 2))
# Update dictionary
data_dict[self.key] = data
return data_dict

class StackDictEditor(Editor):
"""
3 changes: 2 additions & 1 deletion itipy/train/msg_to_goes.py
Original file line number Diff line number Diff line change
@@ -23,7 +23,7 @@

import autoroot
from itipy.data.geo_datasets import GeoDataset
from itipy.data.geo_editor import BandSelectionEditor, NanMaskEditor, CoordNormEditor, NanDictEditor, RadUnitEditor, ToTensorEditor, StackDictEditor, MeanStdNormEditor, MinMaxNormEditor
from itipy.data.geo_editor import BandSelectionEditor, NanMaskEditor, CoordNormEditor, NanDictEditor, RadUnitEditor, ToTensorEditor, StackDictEditor, MeanStdNormEditor, MinMaxNormEditor, Rotate180Editor
from itipy.data.geo_utils import get_split, get_list_filenames, normalize, calculate_norm_from_metrics

import warnings
@@ -105,6 +105,7 @@
msg_editors = [
BandSelectionEditor(target_bands=msg_bands),
NanDictEditor(key="data", fill_value=0), # Replaces NaNs in data
Rotate180Editor(key="data"), # Rotate 180 degrees to align north to top of image
MinMaxNormEditor(norm_dict=msg_norm, key="data"),
StackDictEditor(allowed_keys = ['data']),
ToTensorEditor(),