Welcome to iTransformer Discussions!

[lucidrains]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[lucidrains]

for sharing positive / negative replication results. this will determine whether i add the time tokens with 2d attention.

[TKassis]

@lucidrains just curious, when you implement these papers (like this one) is there any reason why you prefer not to use your x-transformers library and build the attention/transformer functionality from scratch?

[lucidrains]

@TKassis i guess my thought is that there may be domain specific tweaks to the architecture to be found and using x-transformers would lock researchers into improvements that were language specific

so i try to keep the code simple and minimalistic, and optimized for a student to be able to copy / paste and tinker with their own ideas. x-transformers is getting a bit too complicated for that

[lucidrains]

if i were in industry or consulting though, i'd reach for x-transformers to get the job done for sure

[lucidrains]

i'll give a relevant example

say the reversible instance normalization paper has a signal and i wanted to gather up all the statistics across variates and use it to condition the layernorm (adaptive normalization). it would not be possible in x-transformers

[TKassis]

Got it, thanks!

[robm0162]

Greetings @lucidrains
I've read through (and tried to understand) the original iTransformer paper by Liu and then found your examples. I've tried a particular dataset with Transformers before and had poor results. My dataset is the options quotes for a security. The data consists of one set of values for each minute through the day, and many days (about 6 months). Each minute or 'tick' contains the price of the security, and values for the 'put' options, and 'call' options. Each category contains values like the bid/ask/last price, cumulative volume traded, and the 'open interest' - number of contracts other traders are offering to sell.
The idea is to use attention to find specific patterns within the option contacts at each strike price. So for instance, if there was a spike in calls purchased, and put contracts offered for sale, that may indicate that some traders believe the price of the underlying security might go up. If that pattern occurred above the current strike price, that might indicate where traders thought the security price might eventually reach. I like the iTransformer concept because each variate is processed for attention separately, and then correlations can be detected in the NN (or is that what 2D attention is for?).

But, in reading the paper, it seems that the predictions would be for the individual option values (such as 'last price' or 'open interest') which I don't need to predict. I only want to predict the stock price over the next X ticks.

Is there a way to accomplish this with iTransformer? Can you suggest how I might do that?