Investigate low accuracy in albert/albert-base-v2

[ayerofieiev-tt]

No description provided.

[ayerofieiev-tt]

	1. low pcc at ttnn_tanh (0.9945167693107032) (created ticket: https://github.com/tenstorrent/tt-metal/issues/17721)
	2. PCC seems to gradually shrink to 0.694 result of a faulty tanh

[amalbasaTT]

@ayerofieiev-tt I tested if fixing tanh would approve accuracy in albert-v2-base model (will test for others too). I replaced every ttnn_tanh in forward function like this:

  #ttnn_tanh = ttnn.tanh(ttnn_multiply_4, )
  ttnn_tanh_intermediate = aten.tanh.default(ttnn.to_torch(ttnn_multiply_4), )
  ttnn_tanh = ttnn.from_torch(ttnn_tanh_intermediate, device=device, layout = ttnn.TILE_LAYOUT, dtype = ttnn.bfloat16)

Accuracy went to 0.9435372886378103.
You can test it yourself, checkout amalbasaTT/abase_v2-aten-tanh on pytorch2.0_ttnn and run python tests/autogen_accuracy_tests/albert/albert-base-v2_code.py

[ayerofieiev-tt]

@amalbasaTT thank you!
Even with tanh fallback - its still pretty low, right? Can you check what further operation causes a divergence?

[amalbasaTT]

If 0.994 PCC tanh caused that much damage, I suspect that even 0.999 PCC tensors can accumulate over time so in that sense 0.94 is not that low in my opinion. But i will investigate of course.