Unlikely to generalize

[TKassis]

We've done a lot of work using this approach in the past (encoding individual series as tokens). It works very very well for held out data that is within the training distribution but fails catastrophically in terms of generalization to slight distribution shifts. We found that conventional ViT style tokenization (treating each series as a channel and then patchyfing) gives worse performance on held out tests sets but generalizes a lot better.

[lucidrains]

@TKassis thank you Timothy!

[lucidrains]

@TKassis you mean, patchifying as in this work?

[TKassis]

Basically exactly like ViT patchifying (in 1D) where time series multivariate are treated as channels (equivalent to RGB in ViT)

[lucidrains]

@TKassis got it! yea, let's try combining your winning approach with this work and see what happens (the time tokens i was thinking of is exactly what you describe)

[evanatyourservice]

@TKassis I'd be super interested to see when and how things fail with the itransformer setup, did you share your findings anywhere, or do you plan to in the future?

[TKassis]

No plans to do so at this time unfortunately. Just internal exploration in my group.

[evanatyourservice]

What were you using to test OOD in general?

[TKassis]

Internal datasets with some distribution shift (different sites, instruments...etc)