@jnie-TT do you have some thoughts on this?

@mtopalovicTT We didn't do it previously because the compiler-generated tensor descriptors were immature (it's much better now, but still has some issue). Some common issues that come to mind include:

• Wrong data type. Compiler assumes that ops can implicitly typecast a tensor data type (which some ops can, but some ops can't), and therefore there will sometimes be data type mismatches on output tensors
• Invalid layout. Previously the tile shapes were not correctly updated in the tensors when we force tile/row_major layout. Currently for the decomposed ops we still lack granular layout updates.
• Some ops auto-shard, auto-move to host (I think we saw this in conv), and we sometimes have workarounds in runtime to pre-shard. Compiler cannot capture this and therefore all downstream layouts will mismatch.
• These runtime checks have performance hits. If we add these we should add them as debug asserts, and probably update CI to also run runtime in debug build.

Overall though I'm definitely on-board of adding such checks to runtime. Currently there's no way to ensure alignment between runtime and compiler-generated descriptors.