ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1.19276]], [[1.46473]], [[0.78498]], [[0.82854]], [[1.00393]], [[1.33058]], [[1.18660]], [[1.59295]], [[1.15213]], [[1.32881]], [[1.04540]], [[1.09822]], [[0.71883]], [[0.86536]], [[0.99039]], [[1.21503]], [[1.39417]], [[1.49652]], [[0.85753]], [[0.97877]], [[1.37811]], [[1.61170]], [[1.09360]], [[1.57066]], [[1.11591]], [[1.24941]], [[1.64924]], [[0.73267]], [[0.88174]], [[1.06992]], [[1.00161]], [[1.62851]], [[0.87242]], [[0.96634]], [[0.84718]], [[0.72822]], [[0.90463]], [[1.62630]], [[1.41961]], [[1.43884]], [[1.22280]], [[0.94016]], [[1.28110]], [[0.72966]], [[0.83522]], [[0.93874]], [[1.51197]], [[1.48967]], [[0.97476]], [[1.17846]], [[1.51629]], [[1.69357]], [[1.39495]], [[1.26405]], [[1.53175]], [[0.90210]], [[1.28968]], [[0.80885]], [[0.84996]], [[0.93821]], [[1.42274]], [[1.39759]], [[0.90033]], [[1.34756]], [[1.47099]], [[1.13340]], [[1.21560]], [[1.31236]], [[1.50669]], [[1.67660]], [[0.81119]], [[1.01327]]]]), compiled_model=tensor([[[[1.19276]], [[1.68250]], [[1.02358]], [[1.07321]], [[0.90693]], [[0.69929]], [[1.03609]], [[1.08364]], [[0.48965]], [[1.57655]], [[1.22907]], [[0.40295]], [[0.61591]], [[0.58463]], [[0.71161]], [[0.78964]], [[1.38995]], [[1.00386]], [[0.84433]], [[1.03512]], [[1.53954]], [[1.60215]], [[0.40223]], [[1.04981]], [[1.16907]], [[1.15756]], [[1.06270]], [[0.24826]], [[1.15561]], [[1.21033]], [[0.58709]], [[1.30616]], [[0.19961]], [[0.76085]], [[0.27415]], [[0.14604]], [[0.68058]], [[1.50487]], [[1.01834]], [[1.53903]], [[0.72203]], [[1.19734]], [[1.42724]], [[0.11151]], [[0.51427]], [[0.76480]], [[1.38842]], [[1.41175]], [[0.97447]], [[1.01191]], [[1.07582]], [[1.73366]], [[0.71882]], [[0.77119]], [[1.21008]], [[0.46204]], [[0.91826]], [[0.20254]], [[0.54710]], [[0.84859]], [[0.90050]], [[1.17542]], [[1.06175]], [[1.09965]], [[1.28838]], [[0.89376]], [[1.12028]], [[1.43377]], [[1.78381]], [[1.79763]], [[1.08940]], [[0.78060]]]])

test_add.test_module[Add1-[((1, 480, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.64329]], [[0.91526]], [[0.23551]], [[0.27906]], [[0.45446]], [[0.78111]], [[0.63713]], [[1.04348]], [[0.60266]], [[0.77934]], [[0.49593]], [[0.54875]], [[0.16936]], [[0.31589]], [[0.44092]], [[0.66556]], [[0.84470]], [[0.94705]], [[0.30806]], [[0.42930]], [[0.82864]], [[1.06223]], [[0.54413]], [[1.02119]], [[0.56644]], [[0.69994]], [[1.09977]], [[0.18320]], [[0.33226]], [[0.52045]], [[0.45213]], [[1.07903]], [[0.32294]], [[0.41687]], [[0.29771]], [[0.17875]], [[0.35516]], [[1.07683]], [[0.87014]], [[0.88937]], [[0.67333]], [[0.39069]], [[0.73163]], [[0.18019]], [[0.28575]], [[0.38927]], [[0.96250]], [[0.94019]], [[0.42529]], [[0.62899]], [[0.96681]], [[1.14410]], [[0.84548]], [[0.71458]], [[0.98228]], [[0.35263]], [[0.74021]], [[0.25938]], [[0.30049]], [[0.38874]], [[0.87327]], [[0.84811]], [[0.35086]], [[0.79809]], [[0.92152]], [[0.58393]], [[0.66612]], [[0.76289]], [[0.95722]], [[1.12713]], [[0.26172]], [[0.46380]], [[0.84354]], [[1.06131]], [[1.08214]], [[1.08821]], [[0.74654]], [[0.21224]], [[0.69303]], [[0.33423]], [[0.18106]], [[1.09128]], [[1.02721]], [[0.14827]], [[0.74062]], [[0.56280]], [[0.56475]], [[0.41816]], [[0.83931]], [[0.35088]], [[0.83033]], [[0.89989]], [[1.00497]], [[0.83399]], [[0.15217]], [[0.32269]], [[0.89669]], [[0.75168]], [[0.25699]], [[0.35912]], [[1.11741]], [[0.98394]], [[0.42902]], [[0.52119]], [[0.17073]], [[0.63805]], [[0.27050]], [[0.26136]], [[0.61948]], [[0.72211]], [[0.44227]], [[0.94372]], [[0.34276]], [[1.10072]], [[0.98968]], [[0.22539]], [[0.52259]], [[0.66960]], [[0.71998]], [[0.76562]], [[0.84325]], [[0.67698]], [[0.40307]], [[0.88363]], [[0.16741]], [[0.35068]], [[0.52187]], [[0.40348]], [[0.47212]], [[0.23722]], [[0.54068]], [[0.75391]], [[0.32130]], [[0.62137]], [[1.00496]], [[0.59563]], [[0.66093]], [[0.60390]], [[0.74822]], [[0.96495]], [[1.12066]], [[0.96456]], [[1.12174]], [[0.61087]], [[0.19787]], [[0.41000]], [[0.98749]], [[0.64379]], [[0.39851]], [[0.26388]], [[0.17911]], [[0.22503]], [[0.54562]], [[0.92124]], [[0.91735]], [[0.16482]], [[0.95892]], [[0.25578]], [[0.54133]], [[0.44430]], [[0.55073]], [[0.54886]], [[0.19836]], [[0.21531]], [[0.56879]], [[0.65350]], [[0.41990]], [[0.83538]], [[0.19700]], [[0.61329]], [[1.08674]], [[0.44309]], [[1.09854]], [[0.82811]], [[0.19580]], [[0.96338]], [[0.58934]], [[0.42383]], [[1.04686]], [[0.24299]], [[0.70069]], [[0.54235]], [[1.00409]], [[0.78661]], [[0.88729]], [[0.82361]], [[0.52680]], [[0.54188]], [[0.23499]], [[0.91796]], [[1.04402]], [[0.98915]], [[0.29434]], [[0.66933]], [[0.29457]], [[0.37179]], [[0.35568]], [[0.81791]], [[0.34908]], [[0.63613]], [[0.66807]], [[0.96935]], [[0.26907]], [[0.30378]], [[0.35670]], [[0.99700]], [[0.46730]], [[1.06878]], [[0.82784]], [[0.71035]], [[0.64331]], [[0.54819]], [[0.70977]], [[0.53286]], [[0.64352]], [[0.71083]], [[0.25593]], [[0.38497]], [[1.05078]], [[0.24126]], [[0.61113]], [[1.14165]], [[0.82765]], [[0.66119]], [[0.21373]], [[0.89472]], [[0.29089]], [[0.50510]], [[0.47928]], [[0.57299]], [[0.65250]], [[1.05944]], [[0.70945]], [[1.09488]], [[0.95289]], [[0.33093]], [[0.87129]], [[0.29359]], [[0.43512]], [[0.79410]], [[0.81213]], [[1.02215]], [[0.48608]], [[0.64783]], [[0.90445]], [[0.16349]], [[1.00852]], [[0.23357]], [[0.65393]], [[0.56203]], [[0.38370]], [[0.71312]], [[1.06049]], [[0.50087]], [[0.35019]], [[0.46211]], [[0.15146]], [[0.87273]], [[0.40690]], [[0.31336]], [[0.35898]], [[0.93451]], [[0.91182]], [[1.03079]], [[0.82840]], [[0.48005]], [[0.50731]], [[0.79475]], [[1.05805]], [[0.78298]], [[0.41046]], [[0.41200]], [[0.17430]], [[0.75506]], [[0.36644]], [[0.20125]], [[1.08547]], [[0.32233]], [[0.59015]], [[0.79028]], [[0.66296]], [[0.31059]], [[0.24287]], [[1.04558]], [[0.72845]], [[1.06185]], [[0.47941]], [[0.79431]], [[0.53270]], [[0.62480]], [[0.34251]], [[0.81613]], [[0.80512]], [[0.63672]], [[0.53458]], [[0.33882]], [[0.99281]], [[0.27482]], [[0.85187]], [[0.47891]], [[0.40580]], [[0.73686]], [[0.38731]], [[0.76224]], [[0.74522]], [[0.27579]], [[0.73028]], [[0.86000]], [[0.84495]], [[0.58409]], [[0.23714]], [[0.56996]], [[0.82069]], [[0.46460]], [[0.83682]], [[0.98002]], [[0.38596]], [[0.65196]], [[0.85378]], [[0.68622]], [[0.68880]], [[0.70947]], [[0.25396]], [[0.68633]], [[0.99327]], [[1.09759]], [[0.94091]], [[0.71404]], [[0.88054]], [[0.40379]], [[0.23269]], [[0.21707]], [[1.14584]], [[0.96443]], [[0.30142]], [[0.84266]], [[1.02462]], [[1.14684]], [[1.08423]], [[1.03441]], [[0.53240]], [[0.47156]], [[1.05756]], [[0.92722]], [[0.34614]], [[1.09654]], [[0.88861]], [[0.91960]], [[0.33365]], [[0.79048]], [[0.47175]], [[1.03771]], [[0.55706]], [[0.84169]], [[0.73584]], [[0.85977]], [[0.47712]], [[0.89082]], [[0.29780]], [[0.75996]], [[0.30874]], [[0.15377]], [[0.24551]], [[1.04177]], [[0.91755]], [[1.11614]], [[1.04759]], [[0.20051]], [[0.30582]], [[0.56624]], [[0.32232]], [[0.99423]], [[0.26903]], [[0.40308]], [[0.16399]], [[0.36316]], [[1.05827]], [[1.05641]], [[1.00495]], [[1.03310]], [[1.09163]], [[0.51900]], [[0.86703]], [[1.09250]], [[0.81244]], [[1.14687]], [[0.90637]], [[0.95786]], [[0.47204]], [[0.88695]], [[0.70449]], [[0.52763]], [[0.36518]], [[0.36648]], [[0.26229]], [[0.98270]], [[1.00250]], [[0.59013]], [[0.35769]], [[1.03349]], [[0.96678]], [[0.68420]], [[0.41097]], [[1.10656]], [[0.85150]], [[0.26746]], [[1.12558]], [[1.02672]], [[0.46479]], [[0.92811]], [[0.36294]], [[0.56868]], [[1.07158]], [[0.66769]], [[0.29342]], [[0.47992]], [[0.51131]], [[0.55057]], [[0.69488]], [[1.10945]], [[0.67380]], [[0.33831]], [[0.67266]], [[0.88678]], [[0.89505]], [[0.19006]], [[0.55756]], [[0.27546]], [[0.43369]], [[0.82718]], [[0.29197]], [[0.83290]], [[1.07142]], [[0.67983]], [[0.31379]], [[0.46789]], [[0.75622]], [[0.26587]], [[0.89544]], [[0.19310]], [[0.16639]], [[0.16120]], [[0.54560]], [[0.98325]], [[0.17379]], [[1.06263]], [[0.44702]], [[0.79348]], [[0.66984]], [[0.19617]], [[1.06170]], [[0.91626]], [[1.14403]], [[0.89964]], [[0.31700]], [[1.06433]], [[0.67391]], [[0.88414]], [[0.24612]], [[0.50322]], [[0.15610]], [[0.45229]], [[0.75490]], [[0.25445]], [[0.80642]], [[0.91544]], [[0.71669]], [[0.31249]], [[0.25937]], [[0.49278]], [[0.86651]], [[1.14023]], [[0.93455]], [[0.59073]], [[0.82234]], [[0.15650]], [[0.21998]], [[0.88034]], [[0.36383]], [[0.88758]]]]), compiled_model=tensor([[[[0.64329]], [[1.02117]], [[0.31856]], [[0.43461]], [[0.79436]], [[0.99103]], [[1.32682]], [[1.75060]], [[1.07306]], [[1.20686]], [[1.02565]], [[0.50835]], [[0.26320]], [[0.85668]], [[0.33151]], [[1.18561]], [[1.25362]], [[1.29776]], [[1.14759]], [[0.92165]], [[1.22532]], [[1.36214]], [[0.79665]], [[1.76639]], [[1.03510]], [[0.73801]], [[1.37677]], [[0.44503]], [[1.12893]], [[0.44962]], [[0.58844]], [[1.43568]], [[0.46801]], [[1.13576]], [[0.28771]], [[0.68781]], [[0.29713]], [[1.89522]], [[1.12905]], [[1.68468]], [[0.77566]], [[0.84594]], [[1.33368]], [[0.63094]], [[0.93039]], [[1.08107]], [[1.28055]], [[1.25301]], [[0.53831]], [[1.14719]], [[0.86363]], [[1.74289]], [[1.10398]], [[1.24366]], [[1.42371]], [[0.70947]], [[0.80921]], [[0.50348]], [[0.30953]], [[0.28260]], [[1.24065]], [[1.09374]], [[0.44463]], [[1.25130]], [[1.50086]], [[0.50663]], [[1.01113]], [[0.99747]], [[1.04407]], [[0.99313]], [[0.65237]], [[0.55970]], [[1.63220]], [[1.90881]], [[1.89862]], [[1.32812]], [[0.76886]], [[0.58429]], [[0.92532]], [[1.02738]], [[0.24859]], [[1.66434]], [[1.20723]], [[0.69450]], [[0.68070]], [[0.42676]], [[0.84504]], [[0.54580]], [[1.41998]], [[1.17483]], [[1.51407]], [[1.66004]], [[1.07530]], [[1.53481]], [[0.86495]], [[0.71084]], [[1.52178]], [[1.23528]], [[0.36156]], [[0.40748]], [[1.46453]], [[1.42476]], [[0.33884]], [[1.09587]], [[0.31550]], [[0.63878]], [[0.83985]], [[0.74110]], [[1.45011]], [[1.27046]], [[0.75830]], [[1.48263]], [[0.69179]], [[1.24910]], [[1.59033]], [[0.33635]], [[0.90285]], [[1.11798]], [[0.95050]], [[1.60897]], [[1.64731]], [[1.00341]], [[1.04185]], [[1.08136]], [[0.84852]], [[1.00057]], [[0.93621]], [[0.86224]], [[0.84319]], [[0.45139]], [[0.44666]], [[1.45893]], [[0.54242]], [[1.05523]], [[1.34544]], [[0.62827]], [[0.52216]], [[1.37818]], [[0.94146]], [[1.52462]], [[1.93207]], [[1.70985]], [[1.18421]], [[0.54030]], [[0.54240]], [[0.77855]], [[1.17787]], [[0.74674]], [[0.98617]], [[0.69374]], [[0.96334]], [[0.66755]], [[1.16731]], [[1.47499]], [[1.00052]], [[0.28402]], [[0.97783]], [[0.50573]], [[0.99304]], [[1.20442]], [[1.25018]], [[0.53529]], [[0.92560]], [[0.62230]], [[0.82535]], [[0.93842]], [[0.44273]], [[1.63915]], [[0.76545]], [[0.65628]], [[1.27372]], [[0.38065]], [[1.66038]], [[1.17553]], [[0.95509]], [[1.63680]], [[0.93505]], [[1.23937]], [[0.97255]], [[0.12448]], [[1.39798]], [[0.89502]], [[1.49315]], [[1.22082]], [[1.56811]], [[1.39536]], [[0.81245]], [[1.20437]], [[0.18366]], [[0.80773]], [[1.77911]], [[1.06017]], [[0.54455]], [[1.33820]], [[0.95303]], [[1.14091]], [[1.17659]], [[0.80864]], [[0.46794]], [[1.20037]], [[1.19079]], [[1.71324]], [[0.94132]], [[0.60463]], [[0.32978]], [[1.82863]], [[1.02199]], [[1.63363]], [[0.71610]], [[1.30607]], [[0.66895]], [[0.88430]], [[0.81375]], [[0.96135]], [[0.74884]], [[0.77168]], [[0.21941]], [[0.35924]], [[1.71574]], [[0.35258]], [[1.08870]], [[1.13962]], [[1.05345]], [[0.64041]], [[1.02859]], [[1.12515]], [[0.41621]], [[0.51706]], [[0.72604]], [[0.75951]], [[1.19458]], [[1.71081]], [[1.07360]], [[1.25031]], [[1.16558]], [[0.58174]], [[1.16037]], [[0.27143]], [[0.46912]], [[1.57615]], [[1.13957]], [[1.50845]], [[0.37955]], [[1.23060]], [[1.26076]], [[0.35836]], [[1.48284]], [[0.57902]], [[0.73737]], [[0.87651]], [[0.99345]], [[0.98516]], [[1.17568]], [[0.98611]], [[0.26129]], [[1.22696]], [[0.81614]], [[1.02744]], [[0.80203]], [[0.94180]], [[0.75416]], [[1.48532]], [[1.32362]], [[1.57614]], [[1.61528]], [[0.39682]], [[0.50705]], [[1.06988]], [[1.19286]], [[0.95490]], [[1.11697]], [[0.46169]], [[0.52956]], [[1.27949]], [[0.30813]], [[0.86979]], [[1.53535]], [[1.14892]], [[0.89781]], [[1.61564]], [[1.01869]], [[1.07877]], [[0.42530]], [[1.69815]], [[1.53287]], [[1.67293]], [[0.46723]], [[1.12800]], [[0.93273]], [[1.10899]], [[0.97614]], [[1.18138]], [[1.00870]], [[0.96976]], [[0.83319]], [[0.48779]], [[1.35054]], [[0.34002]], [[1.39998]], [[0.86080]], [[0.95485]], [[1.36058]], [[0.86475]], [[1.52093]], [[1.14508]], [[0.17331]], [[0.58488]], [[1.06145]], [[1.44464]], [[0.82397]], [[0.68591]], [[1.15772]], [[1.18147]], [[0.42437]], [[1.26792]], [[1.08786]], [[1.36295]], [[0.91558]], [[1.24874]], [[2.22376]], [[1.01027]], [[0.66747]], [[0.49755]], [[0.73192]], [[2.62553]], [[1.94519]], [[0.96978]], [[1.10753]], [[2.10460]], [[0.87053]], [[1.37765]], [[0.79318]], [[1.45388]], [[2.35743]], [[1.62411]], [[1.60041]], [[2.04067]], [[2.53301]], [[2.34052]], [[1.93132]], [[1.33752]], [[1.83429]], [[2.26892]], [[1.49845]], [[0.95058]], [[2.09111]], [[2.12146]], [[2.09288]], [[1.40243]], [[1.89150]], [[1.04249]], [[2.05083]], [[2.20788]], [[1.10506]], [[1.30121]], [[1.12362]], [[1.13136]], [[1.85902]], [[1.17908]], [[2.13734]], [[1.25057]], [[1.76357]], [[0.68490]], [[1.44503]], [[1.55812]], [[1.83874]], [[1.80437]], [[1.42555]], [[0.53208]], [[1.46218]], [[0.46313]], [[1.02847]], [[1.46282]], [[1.32246]], [[1.10973]], [[0.87628]], [[1.83848]], [[1.97090]], [[1.20899]], [[2.18489]], [[2.21608]], [[1.51943]], [[0.95303]], [[1.84195]], [[1.75623]], [[1.78890]], [[1.56939]], [[2.34696]], [[1.56621]], [[2.08464]], [[1.81038]], [[1.74973]], [[1.94471]], [[0.88936]], [[0.57693]], [[2.46945]], [[1.66845]], [[1.78098]], [[1.67550]], [[1.68772]], [[2.72306]], [[2.00592]], [[0.71437]], [[2.18706]], [[1.37146]], [[1.49543]], [[1.85940]], [[1.66875]], [[0.99402]], [[1.56037]], [[1.20419]], [[1.24781]], [[1.58324]], [[1.73648]], [[0.42154]], [[1.23669]], [[1.66115]], [[1.44553]], [[2.24219]], [[1.86378]], [[1.97794]], [[1.13903]], [[2.42016]], [[1.60595]], [[2.50290]], [[1.12018]], [[1.11854]], [[0.53224]], [[1.54056]], [[1.64304]], [[1.72697]], [[1.60579]], [[1.25618]], [[1.64901]], [[1.70777]], [[0.53472]], [[1.89824]], [[0.97333]], [[1.88219]], [[0.57439]], [[1.06135]], [[0.47096]], [[1.24574]], [[1.77909]], [[0.70254]], [[1.98786]], [[1.72382]], [[1.42379]], [[2.13608]], [[0.81428]], [[1.28600]], [[1.80340]], [[1.69866]], [[2.15983]], [[1.83500]], [[1.69757]], [[1.15675]], [[1.92461]], [[1.16256]], [[1.15843]], [[0.33645]], [[1.26277]], [[2.28951]], [[1.49414]], [[1.13179]], [[2.54868]], [[1.58626]], [[1.20745]], [[0.92484]], [[0.60111]], [[1.63842]], [[1.55375]], [[1.99650]], [[1.72592]], [[1.99074]], [[1.38349]], [[1.10322]], [[1.70108]], [[0.92578]], [[1.78352]]]])

test_add.test_module[Add1-[((1, 672, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.90843]], [[1.18040]], [[0.50065]], [[0.54421]], [[0.71960]], [[1.04626]], [[0.90227]], [[1.30862]], [[0.86780]], [[1.04448]], [[0.76107]], [[0.81389]], [[0.43450]], [[0.58104]], [[0.70607]], [[0.93070]], [[1.10984]], [[1.21219]], [[0.57321]], [[0.69445]], [[1.09379]], [[1.32737]], [[0.80928]], [[1.28633]], [[0.83159]], [[0.96508]], [[1.36491]], [[0.44834]], [[0.59741]], [[0.78559]], [[0.71728]], [[1.34418]], [[0.58809]], [[0.68201]], [[0.56286]], [[0.44390]], [[0.62031]], [[1.34198]], [[1.13529]], [[1.15451]], [[0.93847]], [[0.65584]], [[0.99677]], [[0.44533]], [[0.55089]], [[0.65441]], [[1.22765]], [[1.20534]], [[0.69043]], [[0.89414]], [[1.23196]], [[1.40924]], [[1.11062]], [[0.97972]], [[1.24742]], [[0.61778]], [[1.00535]], [[0.52452]], [[0.56563]], [[0.65389]], [[1.13841]], [[1.11326]], [[0.61600]], [[1.06323]], [[1.18666]], [[0.84907]], [[0.93127]], [[1.02803]], [[1.22237]], [[1.39227]], [[0.52687]], [[0.72894]], [[1.10868]], [[1.32645]], [[1.34728]], [[1.35336]], [[1.01168]], [[0.47739]], [[0.95817]], [[0.59937]], [[0.44620]], [[1.35642]], [[1.29236]], [[0.41341]], [[1.00576]], [[0.82795]], [[0.82990]], [[0.68330]], [[1.10445]], [[0.61603]], [[1.09547]], [[1.16503]], [[1.27011]], [[1.09913]], [[0.41731]], [[0.58783]], [[1.16183]], [[1.01683]], [[0.52213]], [[0.62427]], [[1.38255]], [[1.24909]], [[0.69416]], [[0.78633]], [[0.43588]], [[0.90319]], [[0.53565]], [[0.52650]], [[0.88463]], [[0.98725]], [[0.70741]], [[1.20887]], [[0.60791]], [[1.36586]], [[1.25483]], [[0.49054]], [[0.78773]], [[0.93474]], [[0.98513]], [[1.03076]], [[1.10839]], [[0.94213]], [[0.66821]], [[1.14877]], [[0.43255]], [[0.61582]], [[0.78701]], [[0.66862]], [[0.73726]], [[0.50237]], [[0.80582]], [[1.01906]], [[0.58644]], [[0.88652]], [[1.27010]], [[0.86078]], [[0.92607]], [[0.86904]], [[1.01337]], [[1.23010]], [[1.38580]], [[1.22970]], [[1.38688]], [[0.87602]], [[0.46302]], [[0.67514]], [[1.25263]], [[0.90894]], [[0.66365]], [[0.52902]], [[0.44425]], [[0.49017]], [[0.81076]], [[1.18638]], [[1.18250]], [[0.42996]], [[1.22407]], [[0.52092]], [[0.80647]], [[0.70944]], [[0.81587]], [[0.81401]], [[0.46350]], [[0.48046]], [[0.83394]], [[0.91864]], [[0.68504]], [[1.10053]], [[0.46215]], [[0.87843]], [[1.35189]], [[0.70823]], [[1.36368]], [[1.09325]], [[0.46095]], [[1.22853]], [[0.85448]], [[0.68897]], [[1.31200]], [[0.50813]], [[0.96583]], [[0.80749]], [[1.26923]], [[1.05175]], [[1.15243]], [[1.08876]], [[0.79194]], [[0.80702]], [[0.50014]], [[1.18310]], [[1.30917]], [[1.25429]], [[0.55949]], [[0.93448]], [[0.55971]], [[0.63693]], [[0.62082]], [[1.08305]], [[0.61422]], [[0.90127]], [[0.93321]], [[1.23449]], [[0.53422]], [[0.56892]], [[0.62185]], [[1.26214]], [[0.73244]], [[1.33392]], [[1.09298]], [[0.97549]], [[0.90845]], [[0.81334]], [[0.97491]], [[0.79800]], [[0.90866]], [[0.97597]], [[0.52107]], [[0.65011]], [[1.31592]], [[0.50640]], [[0.87627]], [[1.40680]], [[1.09280]], [[0.92633]], [[0.47887]], [[1.15987]], [[0.55604]], [[0.77024]], [[0.74442]], [[0.83813]], [[0.91765]], [[1.32458]], [[0.97460]], [[1.36002]], [[1.21803]], [[0.59607]], [[1.13643]], [[0.55873]], [[0.70026]], [[1.05924]], [[1.07727]], [[1.28729]], [[0.75122]], [[0.91298]], [[1.16959]], [[0.42863]], [[1.27367]], [[0.49872]], [[0.91907]], [[0.82717]], [[0.64884]], [[0.97826]], [[1.32564]], [[0.76602]], [[0.61533]], [[0.72726]], [[0.41660]], [[1.13787]], [[0.67204]], [[0.57851]], [[0.62413]], [[1.19965]], [[1.17696]], [[1.29594]], [[1.09354]], [[0.74520]], [[0.77245]], [[1.05989]], [[1.32319]], [[1.04812]], [[0.67560]], [[0.67714]], [[0.43944]], [[1.02020]], [[0.63159]], [[0.46639]], [[1.35062]], [[0.58747]], [[0.85530]], [[1.05542]], [[0.92811]], [[0.57574]], [[0.50802]], [[1.31072]], [[0.99360]], [[1.32699]], [[0.74456]], [[1.05945]], [[0.79784]], [[0.88994]], [[0.60765]], [[1.08128]], [[1.07026]], [[0.90186]], [[0.79973]], [[0.60396]], [[1.25795]], [[0.53996]], [[1.11701]], [[0.74405]], [[0.67094]], [[1.00200]], [[0.65245]], [[1.02738]], [[1.01037]], [[0.54093]], [[0.99543]], [[1.12514]], [[1.11009]], [[0.84924]], [[0.50228]], [[0.83510]], [[1.08583]], [[0.72974]], [[1.10197]], [[1.24516]], [[0.65110]], [[0.91711]], [[1.11893]], [[0.95137]], [[0.95394]], [[0.97461]], [[0.51910]], [[0.95147]], [[1.25841]], [[1.36273]], [[1.20605]], [[0.97918]], [[1.14568]], [[0.66894]], [[0.49783]], [[0.48222]], [[1.41098]], [[1.22957]], [[0.56656]], [[1.10780]], [[1.28976]], [[1.41198]], [[1.34937]], [[1.29955]], [[0.79755]], [[0.73671]], [[1.32270]], [[1.19237]], [[0.61129]], [[1.36169]], [[1.15376]], [[1.18474]], [[0.59879]], [[1.05563]], [[0.73690]], [[1.30286]], [[0.82221]], [[1.10683]], [[1.00099]], [[1.12491]], [[0.74226]], [[1.15596]], [[0.56294]], [[1.02510]], [[0.57388]], [[0.41891]], [[0.51065]], [[1.30691]], [[1.18269]], [[1.38128]], [[1.31274]], [[0.46565]], [[0.57096]], [[0.83139]], [[0.58746]], [[1.25938]], [[0.53417]], [[0.66823]], [[0.42913]], [[0.62830]], [[1.32342]], [[1.32156]], [[1.27010]], [[1.29824]], [[1.35677]], [[0.78415]], [[1.13218]], [[1.35764]], [[1.07759]], [[1.41201]], [[1.17151]], [[1.22300]], [[0.73718]], [[1.15209]], [[0.96963]], [[0.79277]], [[0.63032]], [[0.63162]], [[0.52744]], [[1.24784]], [[1.26764]], [[0.85527]], [[0.62283]], [[1.29863]], [[1.23192]], [[0.94934]], [[0.67611]], [[1.37170]], [[1.11665]], [[0.53260]], [[1.39072]], [[1.29186]], [[0.72993]], [[1.19325]], [[0.62808]], [[0.83382]], [[1.33673]], [[0.93284]], [[0.55857]], [[0.74506]], [[0.77645]], [[0.81571]], [[0.96003]], [[1.37459]], [[0.93895]], [[0.60346]], [[0.93780]], [[1.15192]], [[1.16020]], [[0.45521]], [[0.82271]], [[0.54060]], [[0.69883]], [[1.09232]], [[0.55711]], [[1.09804]], [[1.33657]], [[0.94497]], [[0.57893]], [[0.73303]], [[1.02136]], [[0.53102]], [[1.16058]], [[0.45824]], [[0.43153]], [[0.42635]], [[0.81075]], [[1.24839]], [[0.43894]], [[1.32777]], [[0.71217]], [[1.05862]], [[0.93498]], [[0.46132]], [[1.32684]], [[1.18140]], [[1.40917]], [[1.16478]], [[0.58214]], [[1.32947]], [[0.93905]], [[1.14929]], [[0.51126]], [[0.76836]], [[0.42124]], [[0.71743]], [[1.02004]], [[0.51960]], [[1.07156]], [[1.18058]], [[0.98183]], [[0.57764]], [[0.52452]], [[0.75792]], [[1.13166]], [[1.40538]], [[1.19969]], [[0.85587]], [[1.08749]], [[0.42165]], [[0.48513]], [[1.14548]], [[0.62897]], [[1.15272]], [[0.55921]], [[0.66452]], [[0.50033]], [[1.17310]], [[0.86123]], [[1.29698]], [[1.22161]], [[1.18885]], [[0.92825]], [[0.75759]], [[0.80346]], [[0.97863]], [[1.16003]], [[0.56188]], [[1.33182]], [[0.85781]], [[0.49320]], [[0.64165]], [[1.35459]], [[1.36944]], [[0.44904]], [[1.26483]], [[1.16276]], [[1.20814]], [[1.33544]], [[0.64270]], [[1.07006]], [[1.11679]], [[0.76443]], [[1.07950]], [[0.76832]], [[1.22131]], [[0.77345]], [[0.72578]], [[1.03805]], [[1.08952]], [[0.66789]], [[0.95638]], [[1.20194]], [[0.86243]], [[1.06434]], [[0.79158]], [[1.08743]], [[0.54998]], [[0.61816]], [[0.65838]], [[1.37168]], [[0.77763]], [[0.91081]], [[0.66993]], [[1.41132]], [[1.40051]], [[0.53508]], [[0.50684]], [[0.53318]], [[0.90977]], [[0.78472]], [[0.58490]], [[0.73284]], [[1.00664]], [[0.65093]], [[1.02297]], [[0.79752]], [[0.66990]], [[0.98087]], [[1.32331]], [[0.57414]], [[0.93539]], [[0.72779]], [[1.40283]], [[0.43780]], [[0.43285]], [[1.40487]], [[0.59583]], [[1.00804]], [[0.86902]], [[0.80685]], [[0.80050]], [[1.22990]], [[0.93608]], [[0.42537]], [[0.61700]], [[0.74173]], [[1.16378]], [[0.58861]], [[1.38364]], [[0.80081]], [[0.82240]], [[1.30396]], [[1.16348]], [[1.33624]], [[1.20140]], [[0.76050]], [[0.58044]], [[0.87498]], [[1.32600]], [[0.74437]], [[0.44849]], [[1.11714]], [[1.39891]], [[0.76984]], [[0.49817]], [[0.45865]], [[1.03748]], [[0.87432]], [[0.65968]], [[1.01324]], [[1.10206]], [[1.30984]], [[1.30037]], [[0.83733]], [[0.47129]], [[0.46037]], [[1.37902]], [[1.13320]], [[1.13013]], [[0.47957]], [[1.37518]], [[1.38585]], [[1.36361]], [[0.49038]], [[0.72352]], [[0.56829]], [[1.38566]], [[0.69734]], [[0.68390]], [[1.17413]], [[0.68088]], [[0.66592]], [[0.86844]], [[0.86412]], [[0.52269]], [[1.32901]], [[0.69162]], [[1.08953]], [[1.34707]], [[1.16434]], [[0.98297]], [[1.33761]], [[0.97939]], [[0.68085]], [[1.38518]], [[1.03052]], [[0.42434]], [[0.76983]], [[0.57159]], [[1.35055]], [[0.82960]], [[0.45641]], [[0.88075]], [[1.22618]], [[1.04208]], [[1.07028]], [[0.95861]], [[1.09859]], [[0.79035]], [[0.71328]], [[0.44482]], [[0.53551]], [[1.12888]], [[0.61612]], [[0.98391]], [[1.07177]], [[0.94758]], [[0.58800]], [[1.39034]], [[0.62143]], [[1.32343]], [[0.51442]], [[0.79190]], [[1.18416]], [[0.70789]], [[1.33220]], [[0.56810]], [[0.49229]], [[0.68672]], [[0.99300]], [[1.37255]], [[0.67348]], [[1.09099]], [[0.78682]], [[0.80375]], [[1.27983]], [[0.52468]], [[0.96528]], [[1.38238]], [[0.84347]], [[1.30040]], [[0.75820]], [[1.31466]], [[0.42851]], [[0.84011]]]]), compiled_model=tensor([[[[0.90843]], [[1.58412]], [[0.89398]], [[1.04818]], [[1.27537]], [[0.77185]], [[0.75599]], [[1.60773]], [[1.12538]], [[1.52323]], [[1.16818]], [[0.84960]], [[0.14244]], [[1.14753]], [[0.99562]], [[1.23041]], [[0.73297]], [[1.54277]], [[0.33370]], [[0.76541]], [[0.93263]], [[1.49072]], [[0.64945]], [[1.08204]], [[0.52992]], [[0.67422]], [[1.76473]], [[0.29452]], [[0.80984]], [[0.51842]], [[0.67793]], [[1.05826]], [[1.13780]], [[0.64730]], [[0.42303]], [[0.19072]], [[0.60193]], [[1.26335]], [[1.41222]], [[1.54074]], [[1.03748]], [[0.54613]], [[0.94431]], [[0.43099]], [[0.57483]], [[0.36712]], [[0.99650]], [[1.72225]], [[0.75272]], [[1.11529]], [[0.86029]], [[1.72686]], [[1.20179]], [[0.90946]], [[1.45660]], [[0.69808]], [[0.82365]], [[0.57387]], [[0.91024]], [[0.66078]], [[0.98845]], [[1.33335]], [[0.26196]], [[1.56293]], [[1.58620]], [[0.73863]], [[1.06126]], [[1.39132]], [[1.35240]], [[1.67794]], [[0.67352]], [[1.00914]], [[1.63042]], [[0.97807]], [[1.08188]], [[1.36334]], [[0.88135]], [[0.38416]], [[1.39954]], [[0.38393]], [[0.53632]], [[1.61571]], [[0.96890]], [[0.81681]], [[1.19050]], [[1.38939]], [[0.87241]], [[1.24351]], [[1.19504]], [[1.11906]], [[1.01276]], [[1.55247]], [[1.80938]], [[1.44507]], [[0.13998]], [[0.65638]], [[1.29673]], [[1.23587]], [[0.89062]], [[0.72437]], [[1.32099]], [[1.31698]], [[0.72763]], [[0.67016]], [[0.52847]], [[0.70325]], [[0.81862]], [[0.64325]], [[1.16854]], [[1.34583]], [[0.91972]], [[1.70242]], [[0.74262]], [[0.99824]], [[0.84428]], [[0.42684]], [[1.12228]], [[0.90947]], [[1.16875]], [[1.35338]], [[1.20403]], [[0.63676]], [[0.83416]], [[0.99148]], [[1.14440]], [[0.61429]], [[0.91683]], [[1.94101]], [[0.79359]], [[0.19523]], [[0.78427]], [[0.79951]], [[1.95356]], [[1.46897]], [[1.03383]], [[0.98913]], [[1.88499]], [[1.07064]], [[1.89318]], [[1.54106]], [[1.42870]], [[2.35757]], [[2.44443]], [[1.36862]], [[1.21393]], [[1.79616]], [[2.24377]], [[1.54070]], [[1.20363]], [[1.62661]], [[1.39047]], [[0.79626]], [[1.15005]], [[1.91580]], [[2.15020]], [[1.33810]], [[2.02771]], [[1.35679]], [[1.11207]], [[1.45742]], [[2.20154]], [[0.81223]], [[0.76373]], [[0.47917]], [[1.22303]], [[1.62170]], [[1.30118]], [[2.21276]], [[1.13884]], [[2.22309]], [[1.52614]], [[0.84635]], [[1.73910]], [[1.55071]], [[0.95258]], [[2.18842]], [[0.81559]], [[1.31977]], [[1.18767]], [[0.27723]], [[1.89447]], [[1.46172]], [[1.94983]], [[1.29973]], [[1.66750]], [[1.73810]], [[0.73084]], [[1.69368]], [[1.35944]], [[1.91838]], [[1.13002]], [[1.73860]], [[1.23813]], [[1.31136]], [[0.95759]], [[1.76089]], [[1.44986]], [[2.01560]], [[1.45497]], [[1.85823]], [[2.24760]], [[1.49223]], [[0.58371]], [[1.79053]], [[1.02266]], [[2.18786]], [[1.78511]], [[1.72301]], [[2.58412]], [[2.03206]], [[0.94672]], [[1.62870]], [[1.22973]], [[1.76083]], [[1.37735]], [[1.35286]], [[0.78517]], [[1.01723]], [[1.89203]], [[0.92040]], [[1.12279]], [[2.21044]], [[0.95577]], [[1.41797]], [[1.36357]], [[1.78968]], [[1.83820]], [[1.25944]], [[1.78341]], [[1.37371]], [[2.40000]], [[1.77861]], [[2.31730]], [[2.02499]], [[1.51386]], [[0.58770]], [[1.97816]], [[1.10944]], [[1.87012]], [[1.56698]], [[0.99688]], [[1.99132]], [[1.88006]], [[0.71467]], [[2.04647]], [[0.87095]], [[1.99528]], [[0.61486]], [[1.54888]], [[0.87178]], [[1.08383]], [[1.50896]], [[1.58924]], [[1.42611]], [[1.62698]], [[1.09242]], [[1.61771]], [[1.49083]], [[0.63121]], [[1.20051]], [[0.91361]], [[2.19470]], [[2.42983]], [[1.66403]], [[1.31124]], [[1.52052]], [[1.42375]], [[1.44996]], [[1.23841]], [[1.59347]], [[1.94507]], [[1.65168]], [[0.49968]], [[2.38830]], [[1.23601]], [[1.09620]], [[1.75094]], [[0.43067]], [[1.36206]], [[1.20379]], [[1.72491]], [[1.44579]], [[1.41127]], [[2.27256]], [[1.61169]], [[1.88259]], [[1.04136]], [[1.69025]], [[0.61747]], [[2.14378]], [[1.09440]], [[1.32389]], [[1.53357]], [[1.77484]], [[1.30698]], [[0.53627]], [[1.47712]], [[1.12339]], [[2.14526]], [[1.47347]], [[0.90206]], [[1.61099]], [[0.55883]], [[1.04981]], [[1.39255]], [[1.11979]], [[1.91007]], [[2.46356]], [[1.77098]], [[1.64392]], [[1.11576]], [[0.93127]], [[0.93686]], [[1.17424]], [[1.02130]], [[2.01859]], [[1.49254]], [[1.80269]], [[1.85434]], [[1.46084]], [[1.76708]], [[1.92457]], [[0.90358]], [[2.30568]], [[1.88134]], [[1.68857]], [[2.17064]], [[1.01203]], [[1.86243]], [[0.70638]], [[0.67409]], [[1.50572]], [[1.46681]], [[1.95316]], [[0.44210]], [[1.38343]], [[1.17471]], [[2.89503]], [[2.06625]], [[2.57206]], [[1.16103]], [[1.17047]], [[2.24420]], [[1.41113]], [[1.12950]], [[2.03058]], [[2.02213]], [[2.02558]], [[0.72493]], [[1.79064]], [[1.18835]], [[2.63357]], [[1.51400]], [[1.93832]], [[2.01252]], [[1.42220]], [[1.13930]], [[1.24727]], [[0.46029]], [[0.89552]], [[1.40236]], [[1.10048]], [[0.54311]], [[2.14604]], [[2.27138]], [[1.47574]], [[1.91169]], [[1.05095]], [[1.20285]], [[1.41234]], [[0.82766]], [[1.40690]], [[1.75420]], [[2.16485]], [[1.91558]], [[1.54425]], [[1.68010]], [[1.49367]], [[1.78323]], [[1.91344]], [[1.19318]], [[2.03631]], [[1.92724]], [[1.63996]], [[1.34611]], [[1.42659]], [[1.60437]], [[1.35663]], [[1.74499]], [[1.91474]], [[2.07152]], [[2.04064]], [[1.29344]], [[1.75425]], [[0.98021]], [[1.54650]], [[2.37725]], [[1.67837]], [[0.57222]], [[1.29394]], [[2.28427]], [[1.96192]], [[0.60277]], [[2.05539]], [[1.01997]], [[0.75921]], [[1.81839]], [[1.62078]], [[1.76786]], [[2.05154]], [[0.97420]], [[1.90428]], [[1.61634]], [[1.76976]], [[1.73672]], [[0.66923]], [[1.26713]], [[1.52151]], [[1.49835]], [[2.57138]], [[2.17408]], [[1.19469]], [[1.56748]], [[1.82110]], [[1.59654]], [[1.04361]], [[1.34674]], [[0.99067]], [[1.12985]], [[1.13154]], [[0.59160]], [[2.14479]], [[1.46680]], [[1.58802]], [[1.51219]], [[0.94913]], [[1.13134]], [[1.49702]], [[1.68986]], [[1.57069]], [[1.95142]], [[1.72402]], [[1.58277]], [[1.37652]], [[0.56916]], [[1.69415]], [[1.47786]], [[1.39319]], [[1.50897]], [[0.74288]], [[1.87800]], [[1.43677]], [[2.16431]], [[2.22760]], [[1.17049]], [[1.20131]], [[1.50470]], [[1.24284]], [[1.09213]], [[1.56061]], [[1.25924]], [[0.84055]], [[1.53346]], [[0.72972]], [[1.48473]], [[1.70682]], [[1.01239]], [[1.80461]], [[0.87779]], [[1.00202]], [[1.99320]], [[1.37385]], [[2.44789]], [[1.61922]], [[1.63040]], [[1.64627]], [[1.00799]], [[1.97267]], [[1.18935]], [[0.86503]], [[1.54501]], [[1.14736]], [[1.58130]], [[1.98174]], [[2.01716]], [[2.28016]], [[1.62188]], [[1.98104]], [[0.69974]], [[1.15314]], [[2.17039]], [[1.62662]], [[1.29240]], [[1.48790]], [[1.87267]], [[1.58654]], [[1.25762]], [[1.03811]], [[1.41035]], [[2.15764]], [[1.22765]], [[2.56589]], [[1.69190]], [[1.40059]], [[1.25305]], [[2.05916]], [[1.67988]], [[2.33825]], [[1.06836]], [[1.97340]], [[1.02509]], [[1.69343]], [[1.17503]], [[1.27495]], [[1.37471]], [[1.44902]], [[0.47513]], [[0.90344]], [[2.50551]], [[0.80283]], [[1.74087]], [[1.51903]], [[1.72870]], [[0.77822]], [[1.23458]], [[1.37136]], [[1.37571]], [[0.88252]], [[1.22467]], [[1.01726]], [[2.19372]], [[2.69914]], [[1.19651]], [[1.34498]], [[1.28658]], [[1.07932]], [[1.53292]], [[0.44416]], [[0.78978]], [[2.17061]], [[1.37832]], [[2.11924]], [[0.76490]], [[1.48831]], [[1.82945]], [[1.26949]], [[1.64480]], [[1.10224]], [[1.05299]], [[1.86716]], [[1.01907]], [[1.00583]], [[2.16837]], [[1.16977]], [[0.85715]], [[1.68381]], [[1.21081]], [[1.41576]], [[1.61975]], [[1.46569]], [[0.76735]], [[1.69014]], [[1.65317]], [[2.32775]], [[1.79171]], [[1.36828]], [[0.89569]], [[1.48010]], [[2.08464]], [[1.70620]], [[2.04103]], [[1.25092]], [[0.87788]], [[1.44775]], [[0.77093]], [[1.78361]], [[1.86754]], [[1.18523]], [[1.60278]], [[2.60237]], [[1.37635]], [[1.16476]], [[0.47177]], [[2.32345]], [[1.99501]], [[1.92043]], [[1.06830]], [[1.81788]], [[1.83039]], [[1.99718]], [[1.40130]], [[1.24050]], [[1.05688]], [[1.93660]], [[1.55421]], [[1.20574]], [[1.41793]], [[1.30303]], [[2.37366]], [[1.81223]], [[1.03306]], [[1.67192]], [[1.02087]], [[2.49442]], [[1.43024]], [[0.44503]], [[1.34683]], [[1.33015]], [[1.69838]], [[1.28023]], [[1.13785]], [[1.26823]], [[2.09830]], [[0.70381]], [[1.94527]], [[2.02276]], [[2.11511]], [[1.48637]], [[2.17417]], [[2.79097]], [[1.27894]], [[1.64047]], [[1.11589]], [[0.74408]], [[2.98319]], [[2.10460]], [[1.90816]], [[1.52496]], [[2.14884]], [[1.33910]], [[2.19165]], [[1.42308]], [[2.11199]], [[2.90387]], [[2.31052]], [[1.97858]], [[2.34177]], [[2.56566]], [[2.46385]], [[2.64802]], [[1.54146]], [[2.40603]], [[2.92852]], [[2.03385]], [[1.12640]], [[3.06927]], [[2.33071]], [[3.00413]], [[1.50468]], [[2.27122]], [[1.81448]], [[2.34654]], [[3.12790]], [[1.26098]], [[1.38133]], [[1.39816]], [[1.71218]], [[2.81940]], [[1.44038]], [[2.81616]], [[1.62522]], [[2.15514]], [[1.55255]], [[1.55754]], [[2.11122]], [[2.80894]], [[2.23566]], [[2.31377]], [[0.87810]], [[2.36466]], [[0.47946]], [[1.45641]]]])

test_add.test_module[Add1-[((1, 576, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.82845]], [[1.10041]], [[0.42067]], [[0.46422]], [[0.63961]], [[0.96627]], [[0.82228]], [[1.22863]], [[0.78782]], [[0.96450]], [[0.68108]], [[0.73391]], [[0.35452]], [[0.50105]], [[0.62608]], [[0.85071]], [[1.02986]], [[1.13220]], [[0.49322]], [[0.61446]], [[1.01380]], [[1.24738]], [[0.72929]], [[1.20635]], [[0.75160]], [[0.88510]], [[1.28493]], [[0.36835]], [[0.51742]], [[0.70561]], [[0.63729]], [[1.26419]], [[0.50810]], [[0.60202]], [[0.48287]], [[0.36391]], [[0.54032]], [[1.26199]], [[1.05530]], [[1.07453]], [[0.85849]], [[0.57585]], [[0.91678]], [[0.36534]], [[0.47091]], [[0.57443]], [[1.14766]], [[1.12535]], [[0.61044]], [[0.81415]], [[1.15197]], [[1.32926]], [[1.03063]], [[0.89974]], [[1.16743]], [[0.53779]], [[0.92536]], [[0.44454]], [[0.48565]], [[0.57390]], [[1.05843]], [[1.03327]], [[0.53601]], [[0.98324]], [[1.10668]], [[0.76908]], [[0.85128]], [[0.94804]], [[1.14238]], [[1.31229]], [[0.44688]], [[0.64896]], [[1.02870]], [[1.24646]], [[1.26729]], [[1.27337]], [[0.93170]], [[0.39740]], [[0.87819]], [[0.51939]], [[0.36621]], [[1.27644]], [[1.21237]], [[0.33343]], [[0.92578]], [[0.74796]], [[0.74991]], [[0.60331]], [[1.02447]], [[0.53604]], [[1.01549]], [[1.08504]], [[1.19013]], [[1.01915]], [[0.33732]], [[0.50784]], [[1.08185]], [[0.93684]], [[0.44215]], [[0.54428]], [[1.30256]], [[1.16910]], [[0.61418]], [[0.70635]], [[0.35589]], [[0.82320]], [[0.45566]], [[0.44651]], [[0.80464]], [[0.90726]], [[0.62742]], [[1.12888]], [[0.52792]], [[1.28588]], [[1.17484]], [[0.41055]], [[0.70775]], [[0.85475]], [[0.90514]], [[0.95078]], [[1.02840]], [[0.86214]], [[0.58823]], [[1.06878]], [[0.35257]], [[0.53584]], [[0.70703]], [[0.58863]], [[0.65727]], [[0.42238]], [[0.72583]], [[0.93907]], [[0.50646]], [[0.80653]], [[1.19012]], [[0.78079]], [[0.84609]], [[0.78906]], [[0.93338]], [[1.15011]], [[1.30581]], [[1.14972]], [[1.30690]], [[0.79603]], [[0.38303]], [[0.59515]], [[1.17264]], [[0.82895]], [[0.58367]], [[0.44903]], [[0.36426]], [[0.41019]], [[0.73077]], [[1.10639]], [[1.10251]], [[0.34997]], [[1.14408]], [[0.44094]], [[0.72648]], [[0.62945]], [[0.73588]], [[0.73402]], [[0.38352]], [[0.40047]], [[0.75395]], [[0.83866]], [[0.60505]], [[1.02054]], [[0.38216]], [[0.79845]], [[1.27190]], [[0.62824]], [[1.28369]], [[1.01327]], [[0.38096]], [[1.14854]], [[0.77449]], [[0.60899]], [[1.23202]], [[0.42814]], [[0.88584]], [[0.72751]], [[1.18925]], [[0.97176]], [[1.07244]], [[1.00877]], [[0.71195]], [[0.72704]], [[0.42015]], [[1.10311]], [[1.22918]], [[1.17430]], [[0.47950]], [[0.85449]], [[0.47972]], [[0.55695]], [[0.54084]], [[1.00306]], [[0.53423]], [[0.82128]], [[0.85322]], [[1.15450]], [[0.45423]], [[0.48893]], [[0.54186]], [[1.18216]], [[0.65246]], [[1.25393]], [[1.01299]], [[0.89550]], [[0.82847]], [[0.73335]], [[0.89492]], [[0.71802]], [[0.82868]], [[0.89599]], [[0.44109]], [[0.57012]], [[1.23594]], [[0.42642]], [[0.79629]], [[1.32681]], [[1.01281]], [[0.84635]], [[0.39889]], [[1.07988]], [[0.47605]], [[0.69026]], [[0.66443]], [[0.75815]], [[0.83766]], [[1.24459]], [[0.89461]], [[1.28004]], [[1.13805]], [[0.51608]], [[1.05644]], [[0.47874]], [[0.62028]], [[0.97925]], [[0.99729]], [[1.20730]], [[0.67123]], [[0.83299]], [[1.08960]], [[0.34864]], [[1.19368]], [[0.41873]], [[0.83908]], [[0.74718]], [[0.56885]], [[0.89828]], [[1.24565]], [[0.68603]], [[0.53534]], [[0.64727]], [[0.33662]], [[1.05789]], [[0.59206]], [[0.49852]], [[0.54414]], [[1.11967]], [[1.09698]], [[1.21595]], [[1.01355]], [[0.66521]], [[0.69247]], [[0.97991]], [[1.24321]], [[0.96814]], [[0.59562]], [[0.59715]], [[0.35946]], [[0.94021]], [[0.55160]], [[0.38640]], [[1.27063]], [[0.50748]], [[0.77531]], [[0.97544]], [[0.84812]], [[0.49575]], [[0.42803]], [[1.23073]], [[0.91361]], [[1.24700]], [[0.66457]], [[0.97947]], [[0.71785]], [[0.80995]], [[0.52767]], [[1.00129]], [[0.99027]], [[0.82188]], [[0.71974]], [[0.52398]], [[1.17796]], [[0.45998]], [[1.03702]], [[0.66406]], [[0.59096]], [[0.92201]], [[0.57246]], [[0.94739]], [[0.93038]], [[0.46095]], [[0.91544]], [[1.04515]], [[1.03011]], [[0.76925]], [[0.42230]], [[0.75511]], [[1.00584]], [[0.64975]], [[1.02198]], [[1.16517]], [[0.57111]], [[0.83712]], [[1.03894]], [[0.87138]], [[0.87396]], [[0.89462]], [[0.43912]], [[0.87148]], [[1.17843]], [[1.28275]], [[1.12607]], [[0.89919]], [[1.06570]], [[0.58895]], [[0.41785]], [[0.40223]], [[1.33099]], [[1.14958]], [[0.48657]], [[1.02781]], [[1.20977]], [[1.33200]], [[1.26939]], [[1.21957]], [[0.71756]], [[0.65672]], [[1.24271]], [[1.11238]], [[0.53130]], [[1.28170]], [[1.07377]], [[1.10476]], [[0.51880]], [[0.97564]], [[0.65691]], [[1.22287]], [[0.74222]], [[1.02685]], [[0.92100]], [[1.04492]], [[0.66228]], [[1.07598]], [[0.48295]], [[0.94512]], [[0.49389]], [[0.33892]], [[0.43066]], [[1.22693]], [[1.10271]], [[1.30130]], [[1.23275]], [[0.38567]], [[0.49097]], [[0.75140]], [[0.50747]], [[1.17939]], [[0.45419]], [[0.58824]], [[0.34914]], [[0.54832]], [[1.24343]], [[1.24157]], [[1.19011]], [[1.21825]], [[1.27678]], [[0.70416]], [[1.05219]], [[1.27766]], [[0.99760]], [[1.33203]], [[1.09152]], [[1.14301]], [[0.65719]], [[1.07210]], [[0.88964]], [[0.71278]], [[0.55034]], [[0.55163]], [[0.44745]], [[1.16786]], [[1.18766]], [[0.77528]], [[0.54285]], [[1.21864]], [[1.15193]], [[0.86936]], [[0.59612]], [[1.29171]], [[1.03666]], [[0.45262]], [[1.31073]], [[1.21188]], [[0.64994]], [[1.11327]], [[0.54810]], [[0.75383]], [[1.25674]], [[0.85285]], [[0.47858]], [[0.66507]], [[0.69647]], [[0.73573]], [[0.88004]], [[1.29460]], [[0.85896]], [[0.52347]], [[0.85782]], [[1.07193]], [[1.08021]], [[0.37522]], [[0.74272]], [[0.46062]], [[0.61885]], [[1.01234]], [[0.47712]], [[1.01805]], [[1.25658]], [[0.86498]], [[0.49895]], [[0.65305]], [[0.94137]], [[0.45103]], [[1.08060]], [[0.37826]], [[0.35154]], [[0.34636]], [[0.73076]], [[1.16841]], [[0.35895]], [[1.24778]], [[0.63218]], [[0.97863]], [[0.85499]], [[0.38133]], [[1.24685]], [[1.10141]], [[1.32919]], [[1.08480]], [[0.50216]], [[1.24948]], [[0.85906]], [[1.06930]], [[0.43128]], [[0.68838]], [[0.34125]], [[0.63744]], [[0.94006]], [[0.43961]], [[0.99157]], [[1.10059]], [[0.90184]], [[0.49765]], [[0.44453]], [[0.67793]], [[1.05167]], [[1.32539]], [[1.11970]], [[0.77589]], [[1.00750]], [[0.34166]], [[0.40514]], [[1.06549]], [[0.54898]], [[1.07274]], [[0.47922]], [[0.58453]], [[0.42035]], [[1.09311]], [[0.78124]], [[1.21699]], [[1.14163]], [[1.10886]], [[0.84827]], [[0.67760]], [[0.72347]], [[0.89864]], [[1.08004]], [[0.48190]], [[1.25183]], [[0.77783]], [[0.41322]], [[0.56166]], [[1.27460]], [[1.28945]], [[0.36905]], [[1.18484]], [[1.08277]], [[1.12815]], [[1.25545]], [[0.56271]], [[0.99008]], [[1.03681]], [[0.68444]], [[0.99952]], [[0.68833]], [[1.14132]], [[0.69346]], [[0.64579]], [[0.95806]], [[1.00954]], [[0.58790]], [[0.87639]], [[1.12196]], [[0.78244]], [[0.98436]], [[0.71160]], [[1.00744]], [[0.46999]], [[0.53818]], [[0.57839]], [[1.29170]], [[0.69765]], [[0.83082]], [[0.58994]], [[1.33134]], [[1.32053]], [[0.45510]], [[0.42686]], [[0.45319]], [[0.82978]], [[0.70474]], [[0.50492]], [[0.65285]], [[0.92666]], [[0.57094]], [[0.94298]], [[0.71753]], [[0.58991]], [[0.90088]], [[1.24332]], [[0.49415]], [[0.85541]], [[0.64781]], [[1.32284]], [[0.35781]], [[0.35286]], [[1.32488]], [[0.51585]], [[0.92805]], [[0.78904]], [[0.72686]], [[0.72052]], [[1.14991]], [[0.85609]], [[0.34538]], [[0.53701]], [[0.66174]], [[1.08380]], [[0.50862]], [[1.30365]], [[0.72083]], [[0.74241]], [[1.22397]], [[1.08349]], [[1.25625]], [[1.12141]], [[0.68051]], [[0.50045]], [[0.79499]], [[1.24601]]]]), compiled_model=tensor([[[[0.82845]], [[0.80454]], [[0.79344]], [[1.11877]], [[0.66508]], [[0.72007]], [[0.53656]], [[1.52174]], [[0.91777]], [[0.87981]], [[0.94996]], [[1.09160]], [[0.91999]], [[1.05705]], [[0.71905]], [[0.57764]], [[0.74586]], [[1.76686]], [[0.88206]], [[1.00022]], [[0.74900]], [[1.87820]], [[1.37077]], [[1.82559]], [[0.49762]], [[0.86425]], [[1.10886]], [[1.00965]], [[0.47039]], [[0.64514]], [[1.06706]], [[1.20070]], [[0.42965]], [[0.72610]], [[0.60262]], [[0.14223]], [[1.12496]], [[1.20924]], [[1.40047]], [[1.67723]], [[1.27845]], [[0.81445]], [[1.51002]], [[0.60037]], [[0.40739]], [[1.21523]], [[1.43381]], [[0.80532]], [[0.63591]], [[0.64138]], [[1.75816]], [[1.41449]], [[0.74268]], [[1.03611]], [[1.64925]], [[0.83550]], [[1.25127]], [[0.65878]], [[0.83987]], [[0.61988]], [[1.02734]], [[0.73373]], [[0.32716]], [[1.36775]], [[0.97843]], [[1.00862]], [[1.17869]], [[1.15125]], [[0.98601]], [[1.95826]], [[0.32394]], [[1.22802]], [[0.79875]], [[1.29400]], [[1.70709]], [[1.23689]], [[1.51953]], [[0.22113]], [[0.62611]], [[0.46174]], [[0.61485]], [[1.90462]], [[1.14148]], [[0.68005]], [[0.96823]], [[0.80734]], [[1.28537]], [[0.38363]], [[1.24538]], [[1.17405]], [[1.11459]], [[1.64107]], [[1.20396]], [[1.58944]], [[0.02147]], [[0.60359]], [[1.16183]], [[1.26669]], [[0.80612]], [[1.09600]], [[1.39588]], [[1.31708]], [[1.12439]], [[0.73887]], [[0.96202]], [[0.65811]], [[0.56937]], [[0.58747]], [[1.19556]], [[1.41691]], [[0.71598]], [[0.88245]], [[0.94348]], [[1.60325]], [[1.54350]], [[0.26992]], [[1.19736]], [[1.49614]], [[1.11634]], [[0.65156]], [[1.54721]], [[0.65922]], [[0.87097]], [[1.30925]], [[0.28636]], [[0.87774]], [[0.42762]], [[0.87032]], [[0.50811]], [[0.53612]], [[0.95796]], [[1.53285]], [[0.43571]], [[1.29466]], [[1.29441]], [[0.71114]], [[0.57850]], [[0.49811]], [[1.58950]], [[1.19322]], [[1.49860]], [[1.45308]], [[1.81457]], [[1.39058]], [[0.95635]], [[0.39252]], [[1.26034]], [[0.70089]], [[0.46580]], [[0.73545]], [[1.00140]], [[0.17744]], [[1.20118]], [[1.01497]], [[1.17293]], [[0.91467]], [[1.19881]], [[0.65427]], [[0.54478]], [[1.22288]], [[0.83906]], [[0.53613]], [[0.69773]], [[0.21277]], [[0.52501]], [[1.03693]], [[1.16928]], [[1.04686]], [[0.78530]], [[1.39592]], [[1.77134]], [[0.53377]], [[1.39669]], [[1.02373]], [[0.14671]], [[1.31656]], [[1.31852]], [[1.19805]], [[1.44648]], [[0.70951]], [[0.83719]], [[1.27274]], [[1.14900]], [[0.79225]], [[1.31725]], [[1.47627]], [[0.42897]], [[1.34683]], [[0.76782]], [[0.92062]], [[1.28929]], [[1.77590]], [[0.26369]], [[0.87616]], [[0.81152]], [[0.28672]], [[0.98275]], [[1.43115]], [[1.01215]], [[0.67032]], [[1.51903]], [[1.02593]], [[1.12121]], [[0.17690]], [[0.26418]], [[1.65706]], [[0.87252]], [[1.45059]], [[0.90392]], [[0.85358]], [[0.85008]], [[0.41408]], [[1.08872]], [[0.97426]], [[0.99606]], [[1.22526]], [[1.08332]], [[0.87088]], [[1.22071]], [[0.38845]], [[0.64419]], [[1.14801]], [[1.10009]], [[0.92572]], [[0.78913]], [[1.03397]], [[1.04246]], [[0.50723]], [[0.83366]], [[1.37542]], [[1.50266]], [[1.12277]], [[1.15148]], [[1.50691]], [[1.07143]], [[0.51115]], [[1.35968]], [[0.29887]], [[0.87059]], [[1.36342]], [[0.96806]], [[1.79043]], [[0.80614]], [[1.22766]], [[1.75256]], [[0.36362]], [[1.63319]], [[0.44352]], [[0.93385]], [[0.83026]], [[0.73356]], [[0.87720]], [[1.53065]], [[0.87268]], [[1.02010]], [[0.71388]], [[0.55457]], [[1.03970]], [[0.34114]], [[0.86866]], [[0.77593]], [[1.08724]], [[1.09573]], [[1.51452]], [[1.09095]], [[1.15976]], [[0.88860]], [[1.33658]], [[1.62887]], [[1.01326]], [[0.97976]], [[1.13393]], [[0.55027]], [[1.20587]], [[0.73758]], [[0.90040]], [[1.22734]], [[0.40951]], [[1.16100]], [[0.70809]], [[1.02395]], [[0.43361]], [[0.92586]], [[0.93302]], [[1.38459]], [[1.90949]], [[0.93355]], [[1.11396]], [[1.37784]], [[0.76409]], [[0.65541]], [[0.95015]], [[1.08906]], [[1.09836]], [[0.95433]], [[0.59735]], [[0.84585]], [[0.65339]], [[1.19002]], [[0.90709]], [[1.14189]], [[1.57583]], [[0.44363]], [[1.08403]], [[0.89785]], [[0.16793]], [[0.71942]], [[1.68225]], [[1.48945]], [[1.20564]], [[0.86798]], [[0.52568]], [[1.22616]], [[1.28292]], [[0.91075]], [[1.77778]], [[0.87034]], [[1.35670]], [[0.99256]], [[1.26970]], [[0.59852]], [[1.03068]], [[0.77370]], [[1.18924]], [[1.76464]], [[1.94189]], [[1.74875]], [[1.40270]], [[1.58504]], [[0.69225]], [[0.19763]], [[0.38443]], [[1.44267]], [[1.04720]], [[0.90997]], [[1.37113]], [[1.45826]], [[1.62059]], [[1.87642]], [[1.56949]], [[0.44690]], [[0.46159]], [[1.63090]], [[1.34833]], [[0.94296]], [[0.95015]], [[0.78014]], [[1.57918]], [[1.00643]], [[0.69109]], [[1.01451]], [[1.00026]], [[1.28868]], [[1.35155]], [[1.58320]], [[0.71974]], [[0.34875]], [[1.58184]], [[1.09468]], [[1.41927]], [[0.99412]], [[0.08560]], [[0.96038]], [[0.92266]], [[1.38222]], [[1.14159]], [[1.20196]], [[0.64201]], [[0.46219]], [[0.84010]], [[0.20137]], [[0.87576]], [[0.81568]], [[0.54441]], [[0.21276]], [[0.51884]], [[1.00245]], [[1.00968]], [[1.65710]], [[1.64018]], [[1.51119]], [[0.59884]], [[1.38635]], [[1.46793]], [[0.80807]], [[1.60743]], [[1.71460]], [[0.81902]], [[0.32781]], [[1.30398]], [[1.43396]], [[0.57635]], [[0.82821]], [[1.07093]], [[0.76942]], [[1.44387]], [[1.85433]], [[1.26883]], [[0.42944]], [[1.42967]], [[1.57518]], [[1.47812]], [[0.26647]], [[1.73028]], [[1.05930]], [[0.13082]], [[1.58953]], [[1.10748]], [[0.88861]], [[0.95645]], [[0.33964]], [[0.87922]], [[1.25659]], [[1.31309]], [[0.68950]], [[1.22316]], [[0.95798]], [[0.74275]], [[1.38652]], [[1.48974]], [[1.12062]], [[0.53047]], [[1.02742]], [[1.14361]], [[1.20551]], [[0.15309]], [[0.90745]], [[0.21341]], [[0.67909]], [[1.46992]], [[0.29726]], [[0.84884]], [[1.30230]], [[0.89573]], [[0.75140]], [[0.90387]], [[1.36273]], [[0.27440]], [[1.21312]], [[0.82571]], [[0.95184]], [[0.47770]], [[0.60926]], [[1.47995]], [[0.81150]], [[1.10322]], [[0.50984]], [[1.36744]], [[0.98777]], [[0.07689]], [[1.12637]], [[1.47199]], [[1.32741]], [[1.59772]], [[1.06524]], [[1.49003]], [[0.98733]], [[1.07839]], [[0.57380]], [[0.94720]], [[0.12714]], [[0.68576]], [[0.69197]], [[0.91430]], [[0.84096]], [[1.72518]], [[0.94076]], [[0.37902]], [[0.85257]], [[0.92028]], [[1.56571]], [[1.70193]], [[0.80584]], [[1.25994]], [[1.08115]], [[0.28848]], [[0.89048]], [[1.59797]], [[0.27733]], [[1.19534]], [[1.05764]], [[0.94599]], [[1.00939]], [[1.08957]], [[0.67323]], [[1.81476]], [[1.51785]], [[1.75666]], [[0.80723]], [[0.52438]], [[0.83268]], [[0.59566]], [[1.44384]], [[1.01850]], [[1.53967]], [[0.89623]], [[0.82893]], [[0.41210]], [[1.93153]], [[0.96009]], [[0.05789]], [[1.23450]], [[1.65900]], [[1.34602]], [[1.61529]], [[0.36402]], [[1.34023]], [[1.14874]], [[1.05267]], [[1.52045]], [[1.07348]], [[1.26660]], [[0.83047]], [[0.50001]], [[0.94502]], [[1.50227]], [[0.55522]], [[1.35472]], [[1.09150]], [[0.83380]], [[1.16281]], [[0.42057]], [[1.17026]], [[0.58741]], [[0.66107]], [[0.64625]], [[1.85369]], [[1.23445]], [[0.65976]], [[0.99002]], [[1.10695]], [[1.06267]], [[0.77575]], [[0.60201]], [[0.38775]], [[0.51534]], [[0.67898]], [[0.83973]], [[0.35786]], [[0.60873]], [[0.80219]], [[0.75059]], [[0.44730]], [[0.56508]], [[0.98819]], [[0.95678]], [[0.32810]], [[1.05600]], [[0.34413]], [[1.42038]], [[0.20277]], [[0.17405]], [[1.68921]], [[0.29881]], [[0.71694]], [[1.24199]], [[0.98815]], [[1.34368]], [[1.18407]], [[0.78239]], [[0.46692]], [[0.70735]], [[1.21087]], [[0.75357]], [[0.64083]], [[1.69264]], [[0.61329]], [[0.65615]], [[1.63341]], [[0.77221]], [[1.54254]], [[0.90213]], [[1.09685]], [[0.24074]], [[1.27020]], [[0.94050]]]])

test_advindex.test_module[Advindex0-[((448, 1280), torch.float32), ((1, 2), torch.int64)]]: forge/test/models_ops/test_advindex.py#L233

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[0.73153, 0.86132, 0.71255, ..., 0.61325, 0.90353, 0.29587], [0.81885, 0.88055, 0.56113, ..., 0.59655, 0.32334, 0.86036]]]), compiled_model=tensor([[[0.73047, 0.85938, 0.71094, ..., 0.61328, 0.90234, 0.29492], [0.49609, 0.76953, 0.08838, ..., 0.73438, 0.82031, 0.83984]]])

test_advindex.test_module[Advindex4-[((2401,), torch.int64)]]: forge/test/models_ops/test_advindex.py#L233

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[0.84717, 0.30105, 0.55487, ..., 0.18946, 0.11523, 0.16138], [0.76919, 0.62342, 0.39580, ..., 0.68585, 0.69333, 0.21035], [0.40563, 0.77530, 0.35160, ..., 0.00870, 0.38710, 0.76991], ..., [0.98832, 0.74977, 0.20153, ..., 0.94968, 0.89263, 0.26631], [0.49651, 0.19281, 0.22812, ..., 0.40225, 0.96863, 0.68208], [0.33125, 0.32934, 0.44470, ..., 0.36557, 0.43242, 0.19357]], grad_fn=<IndexBackward0>), compiled_model=tensor([[0.84766, 0.30078, 0.55469, ..., 0.18945, 0.11523, 0.16113], [0.50391, 0.19922, 0.78906, ..., 0.13770, 0.99219, 0.50391], [0.76953, 0.62500, 0.39648, ..., 0.68750, 0.69141, 0.20996], ..., [0.58203, 0.84766, 0.59766, ..., 0.48633, 0.90234, 0.55859], [0.50391, 0.19922, 0.78906, ..., 0.13770, 0.99219, 0.50391], [0.43359, 0.22266, 0.93750, ..., 0.62109, 0.05371, 0.81641]])

test_avgpool2d.test_module[Avgpool2D9-[((1, 384, 35, 35), torch.float32)]]: forge/test/models_ops/test_avgpool2d.py#L1472

RuntimeError: Tensor 1 - stride mismatch: expected [1225, 1], got [0, 0]

test_broadcast.test_module[Broadcast0-[((1, 1, 1, 384), torch.bool)]]: forge/test/models_ops/test_broadcast.py#L170

RuntimeError: Generated MLIR module failed verification.

test_broadcast.test_module[Broadcast3-[((1, 12, 1, 384), torch.bool)]]: forge/test/models_ops/test_broadcast.py#L170

RuntimeError: Generated MLIR module failed verification.

test_cast.test_module[Cast0-[((2, 1, 7, 7), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 44., 239., 933., 760., 963., 379., 427.], [503., 497., 683., 101., 866., 756., 399.], [878., 376., 56., 868., 794., 33., 126.], [119., 391., 254., 824., 841., 269., 969.], [549., 480., 481., 12., 363., 360., 995.], [ 85., 822., 999., 711., 288., 778., 43.], [896., 489., 771., 757., 283., 395., 782.]]], [[[671., 840., 969., 673., 441., 111., 280.], [503., 706., 476., 327., 699., 326., 363.], [474., 775., 700., 218., 632., 568., 12.], [277., 645., 110., 80., 348., 521., 119.], [916., 881., 490., 682., 519., 944., 633.], [169., 863., 409., 233., 219., 78., 735.], [ 83., 222., 858., 235., 416., 446., 835.]]]]), compiled_model=tensor([[[[ 44., 0., 239., 0., 933., 0., 760.], [ 0., 963., 0., 379., 0., 427., 0.], [503., 0., 497., 0., 683., 0., 101.], [ 0., 866., 0., 756., 0., 399., 0.], [878., 0., 376., 0., 56., 0., 868.], [ 0., 794., 0., 33., 0., 126., 0.], [119., 0., 391., 0., 254., 0., 824.]]], [[[ 0., 841., 0., 269., 0., 969., 0.], [549., 0., 480., 0., 481., 0., 12.], [ 0., 363., 0., 360., 0., 995., 0.], [ 85., 0., 822., 0., 999., 0., 711.], [ 0., 288., 0., 778., 0., 43., 0.], [896., 0., 489., 0., 771., 0., 757.], [ 0., 283., 0., 395., 0., 782., 0.]]]])

test_cast.test_module[Cast0-[((1, 1, 256, 256), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 44., 239., 933., ..., 374., 973., 703.], [757., 943., 848., ..., 762., 629., 216.], [606., 763., 616., ..., 786., 391., 916.], ..., [889., 108., 668., ..., 29., 931., 519.], [326., 328., 926., ..., 422., 582., 607.], [867., 283., 592., ..., 712., 264., 295.]]]]), compiled_model=tensor([[[[ 44., 0., 239., ..., 0., 389., 0.], [793., 0., 611., ..., 0., 703., 0.], [757., 0., 943., ..., 0., 370., 0.], ..., [178., 0., 972., ..., 0., 393., 0.], [274., 0., 790., ..., 0., 874., 0.], [869., 0., 258., ..., 0., 375., 0.]]]])

test_cast.test_module[Cast1-[((1, 128), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

TypeError: Dtype mismatch: framework_model.dtype=torch.int8, compiled_model.dtype=torch.uint8

test_cast.test_module[Cast1-[((1, 384), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

TypeError: Dtype mismatch: framework_model.dtype=torch.int8, compiled_model.dtype=torch.uint8

test_cast.test_module[Cast2-[((1, 384), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((1, 12, 384, 384), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast1-[((1, 1, 256, 256), torch.int32)]]: forge/test/models_ops/test_cast.py#L502

TypeError: Dtype mismatch: framework_model.dtype=torch.int8, compiled_model.dtype=torch.uint8

test_conv2d.test_module[Conv2D1-[((1, 384, 3000, 1), torch.float32), ((384, 384, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_FATAL @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/ttnn/cpp/ttnn/operations/conv/conv2d/device/conv2d_op_sharded_program_factory.cpp:1180: act_block_w_datums == round_up(conv_act_size_c * filter_w, TILE_WIDTH) info: Error backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xbd288) [0x7f54094de288] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d41multi_core_optimized_conv_sharded_v2_implERN2tt8tt_metal2v07ProgramERKNS4_6TensorESA_RKNS4_5ShapeESt8optionalIS9_ERKSF_RKNS0_14sliding_window19SlidingWindowConfigEjjbbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigEbbRS8_St7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x90c4) [0x7f540560f7a4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d40multi_core_optimized_conv_sharded_v2_newERKN2tt8tt_metal6TensorES7_RKSt8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigENS4_8DataTypeESt5arrayIjLm4EEbS8_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERS5_bbbb+0x492) [0x7f5405602fc2] --- ttnn::operations::conv::conv2d::OptimizedConvNew::create_program(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) const --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x449829) [0x7f5405669829] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c0fda) [0x7f54077e0fda] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c13e4) [0x7f54077e13e4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x2ab) [0x7f54077e072b] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f54077e03e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7f5405668b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7f54056006bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7f54055eff76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30]

test_conv2d.test_module[Conv2D0-[((1, 80, 3000, 1), torch.float32), ((512, 80, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

test_conv2d.test_module[Conv2D15-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 12500992 B L1 buffer across 7 banks, where each bank needs to store 1785856 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f54089217e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f54088ece9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f540621f833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f540621f699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f54077e039a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f540621e69a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f5405656f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f54055efd5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464]

test_conv2d.test_module[Conv2D521-[((1, 192, 17, 17), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[416.85965, 432.23599, 431.21365, ..., 435.40027, 429.54398, 438.36377], [438.03442, 434.87878, 447.71152, ..., 427.46295, 435.37177, 431.39783], [440.09998, 435.00925, 443.98410, ..., 433.86670, 437.38080, 427.41006], ..., [429.18735, 435.26984, 429.91739, ..., 433.30154, 438.79065, 429.12137], [433.74500, 437.30676, 437.76682, ..., 424.18967, 425.40408, 439.91025], [430.14270, 445.39911, 437.30161, ..., 439.90533, 424.54282, 431.01608]], [[417.66599, 429.54269, 426.43771, ..., 428.07605, 422.91019, 436.40936], [430.39377, 432.23532, 436.01321, ..., 430.44885, 430.46542, 430.77563], [438.34549, 435.60123, 432.56165, ..., 428.90488, 431.82007, 427.58572], ..., [427.36942, 428.42010, 426.79327, ..., 431.19339, 432.43860, 427.46362], [431.47833, 432.27814, 430.93466, ..., 420.97873, 426.74283, 431.86530], [425.09692, 442.24066, 427.89246, ..., 429.73291, 421.68756, 432.96246]], [[412.81451, 425.43048, 421.18393, ..., 424.25583, 419.75363, 426.05289], [423.42429, 430.07614, 423.86868, ..., 420.42450, 426.91452, 429.17987], [432.46866, 429.39398, 437.78192, ..., 425.99710, 423.03940, 412.33374], ..., [416.00443, 428.10516, 423.36215, ..., 427.33865, 428.56055, 424.51956], [431.34509, 430.02621, 422.56467, ..., 421.59976, 419.35452, 431.83865], [422.91519, 433.27002, 425.02975, ..., 424.55145, 416.27405, 423.00104]], ..., [[419.75070, 429.47076, 425.21915, ..., 432.63345, 431.58862, 436.86740], [425.12979, 435.93799, 443.77023, ..., 429.23654, 442.14056, 433.21542], [441.36484, 440.75372, 444.07635, ..., 434.55331, 435.20502, 431.81857], ..., [435.71793, 426.35590, 435.90790, ..., 439.40344, 434.78455, 436.11279], [434.92480, 440.43008, 435.10669, ..., 424.93063, 437.74420, 441.15442], [424.50589, 439.42471, 434.30273, ..., 434.07544, 426.08926, 437.74680]], [[412.58954, 433.64548, 422.66141, ..., 433.86356, 435.77264, 436.70609], [428.82632, 436.71277, 439.58911, ..., 428.50891, 443.47815, 427.70953], [446.06442, 438.58231, 438.74796, ..., 431.85144, 433.58405, 421.03027], ..., [431.94717, 429.97028, 429.34335, ..., 436.17242, 437.55984, 425.59833], [428.57782, 442.10397, 435.10114, ..., 426.13055, 427.01523, 436.24274], [425.31342, 432.78329, 432.69574, ..., 436.32706, 421.27277, 438.82114]], [[419.07391, 427.76654, 423.82928, ..., 438.33813, 429.66745, 444.50226], [434.76434, 434.96521, 441.78922, ..., 425.99057, 440.11380, 435.29434], [441.35446, 436.41086, 442.43256, ..., 441.10999, 440.07706, 427.33456], ..., [432.74185, 434.90363, 428.79715, ..., 432.51614, 432.65445, 436.43134], [441.91174, 444.25070, 438.40558, ..., 423.45541, 432.07361, 439.25977], [434.07706, 446.84937, 432.49008, ..., 435.59302, 422.18890, 433.69766]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[418., 436., 424., ..., 428., 430., 442.], [430., 438., 446., ..., 426., 444., 428.], [438., 440., 442., ..., 442., 436., 424.], ..., [430., 434., 426., ..., 430., 438., 426.], [430., 436., 442., ..., 422., 422., 440.], [434., 442., 444., ..., 440., 420., 424.]], [[412., 424., 420., ..., 430., 420., 432.], [434., 432., 430., ..., 430., 428., 426.], [434., 434., 422., ..., 428., 420., 426.], ..., [424., 420., 424., ..., 430., 428., 422.], [430., 428., 430., ..., 420., 420., 432.], [424., 442., 430., ..., 434., 426., 430.]], [[418., 426., 418., ..., 424., 420., 422.], [426., 438., 428., ..., 418., 432., 432.], [430., 426., 444., ..., 422., 424., 418.], ..., [410., 422., 418., ..., 428., 432., 414.], [436., 428., 422., ..., 420., 422., 428.], [426., 438., 422., ..., 422., 408., 426.]], ..., [[426., 428., 424., ..., 424., 430., 436.], [426., 438., 444., ..., 422., 436., 438.], [442., 440., 444., ..., 428., 434., 430.], ..., [438., 422., 430., ..., 430., 432., 440.], [430., 438., 428., ..., 428., 436., 438.], [426., 440., 434., ..., 432., 418., 436.]], [[412., 436., 422., ..., 424., 432., 428.], [430., 434., 440., ..., 424., 448., 428.], [446., 438., 440., ..., 428., 424., 422.], ..., [426., 422., 430., ..., 428., 432., 426.], [428., 442., 434., ..., 432., 428., 432.], [422., 438., 436., ..., 438., 422., 440.]], [[412., 422., 432., ..., 432., 428., 448.], [434., 438., 438., ..., 416., 442., 436.], [442., 434., 444., ..., 444., 442., 422.], ..., [434., 430., 424., ..., 428., 434., 438.], [446., 438., 432., ..., 418., 430., 436.], [430., 440., 430., ..., 434., 414., 434.]]]])

test_conv2d.test_module[Conv2D545-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

test_conv2d.test_module[Conv2D66-[((1, 3, 320, 1024), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 88612864 B L1 buffer across 64 banks, where each bank needs to store 1384576 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f54089217e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f54088ece9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f540621f833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f540621f699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f54077e039a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f540621e69a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f5405656f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f54055efd5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464]

test_conv2d.test_module[Conv2D655-[((1, 256, 22, 66), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[570.94653, 574.15009, 568.35413, ..., 578.82977, 586.17072, 592.34753], [584.77863, 581.80957, 570.39294, ..., 574.62006, 580.81384, 586.60175], [598.08771, 598.28607, 585.37769, ..., 586.02594, 585.01428, 599.18225], ..., [572.13251, 575.29395, 572.75464, ..., 577.74420, 581.78186, 581.04962], [583.05011, 578.44873, 576.27393, ..., 578.19171, 580.68420, 574.94226], [585.05908, 575.49323, 572.74329, ..., 573.84839, 587.02661, 589.33246]], [[573.45331, 579.29913, 566.81458, ..., 575.49823, 580.02594, 587.62433], [587.03949, 580.71796, 574.96289, ..., 589.32404, 587.75854, 591.72961], [600.84753, 595.19177, 583.54480, ..., 588.79730, 590.68573, 587.95825], ..., [586.38879, 565.67670, 578.35944, ..., 578.77789, 577.69873, 570.61536], [578.18835, 567.67218, 577.86224, ..., 576.76306, 580.81598, 590.12738], [579.97986, 570.32141, 571.27496, ..., 587.28955, 586.73737, 592.71124]], [[579.55859, 595.90216, 587.29877, ..., 594.60266, 592.49939, 604.98035], [598.63190, 600.00464, 593.10223, ..., 590.99713, 594.65009, 609.22461], [605.34668, 608.85242, 602.35651, ..., 594.98560, 605.61194, 610.58124], ..., [585.66956, 580.22797, 582.50256, ..., 594.87231, 600.20374, 588.75171], [593.61615, 595.95300, 586.02783, ..., 592.76367, 590.84125, 593.65643], [590.07391, 588.45825, 588.13788, ..., 594.79242, 595.94336, 602.37054]], ..., [[571.39728, 575.18768, 567.78497, ..., 582.24243, 575.70764, 577.05701], [579.51190, 576.39990, 576.37744, ..., 582.00122, 585.70074, 586.33539], [590.65540, 597.52679, 591.22345, ..., 584.85223, 587.73669, 584.32031], ..., [573.84845, 562.15112, 575.59406, ..., 575.20898, 582.02557, 580.20947], [580.48468, 572.69946, 575.35052, ..., 573.37390, 580.13971, 577.83966], [574.11536, 571.33044, 562.83563, ..., 586.38733, 589.88837, 577.47119]], [[578.31976, 581.35272, 570.28436, ..., 589.21368, 588.08344, 585.26477], [584.32971, 588.32037, 583.20984, ..., 586.61182, 584.49390, 587.37958], [591.47174, 598.47406, 588.39484, ..., 585.64160, 588.77783, 598.98877], ..., [575.63623, 572.66638, 575.74323, ..., 586.59906, 582.06775, 581.79163], [576.85944, 574.79382, 579.90466, ..., 584.22498, 575.50928, 578.57391], [579.03802, 574.76233, 571.68762, ..., 586.53125, 587.07812, 580.06000]], [[583.35114, 591.68683, 579.85980, ..., 584.71368, 592.14087, 595.93634], [598.36676, 593.05481, 583.05786, ..., 594.37921, 589.51642, 597.57251], [607.48932, 608.09991, 584.94623, ..., 595.76080, 599.24011, 601.23279], ..., [581.52271, 580.38037, 584.18964, ..., 588.24768, 587.65918, 583.07611], [589.05475, 584.63885, 580.56061, ..., 595.18976, 596.62390, 589.30029], [581.43073, 582.48730, 587.34271, ..., 584.60144, 599.23718, 599.55853]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[576., 576., 572., ..., 572., 584., 588.], [592., 576., 564., ..., 564., 576., 580.], [588., 596., 584., ..., 580., 580., 596.], ..., [580., 572., 572., ..., 564., 584., 584.], [584., 568., 572., ..., 584., 572., 580.], [584., 572., 580., ..., 580., 580., 584.]], [[568., 580., 572., ..., 576., 584., 592.], [592., 592., 568., ..., 584., 588., 592.], [596., 584., 584., ..., 588., 600., 584.], ..., [592., 564., 592., ..., 580., 572., 560.], [584., 560., 576., ..., 580., 584., 588.], [572., 568., 580., ..., 576., 576., 596.]], [[576., 600., 588., ..., 596., 588., 588.], [592., 600., 580., ..., 588., 588., 600.], [600., 608., 600., ..., 584., 604., 604.], ..., [584., 576., 576., ..., 604., 588., 584.], [584., 588., 576., ..., 584., 588., 592.], [592., 580., 580., ..., 604., 600., 616.]], ..., [[576., 572., 580., ..., 580., 584., 580.], [584., 568., 576., ..., 592., 588., 584.], [580., 608., 588., ..., 584., 596., 588.], ..., [572., 560., 580., ..., 572., 584., 580.], [576., 576., 564., ..., 568., 580., 584.], [576., 572., 568., ..., 592., 588., 576.]], [[576., 584., 572., ..., 592., 580., 584.], [584., 588., 592., ..., 580., 588., 568.], [592., 596., 584., ..., 584., 600., 588.], ..., [584., 564., 576., ..., 580., 584., 596.], [576., 564., 580., ..., 580., 572., 580.], [580., 584., 576., ..., 588., 592., 576.]], [[580., 592., 588., ..., 584., 592., 588.], [600., 592., 576., ..., 592., 592., 596.], [604., 612., 584., ..., 596., 596., 600.], ..., [572., 584., 588., ..., 580., 588., 592.], [576., 588., 572., ..., 596., 584., 588.], [584., 588., 584., ..., 580., 596., 600.]]]])

test_conv2d.test_module[Conv2D656-[((1, 128, 42, 130), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[293.48187, 280.29205, 281.29239, ..., 287.21521, 290.81332, 293.09290], [294.28705, 285.08807, 287.24149, ..., 291.01569, 282.24100, 281.75305], [288.74661, 274.44510, 275.10291, ..., 298.00763, 288.37186, 290.80600], ..., [283.06403, 285.75204, 278.37140, ..., 297.40222, 288.94104, 290.20172], [288.89978, 280.77222, 285.13141, ..., 291.95123, 293.15164, 296.36151], [290.25278, 284.08328, 285.93399, ..., 295.87268, 298.14383, 294.36313]], [[295.09750, 279.01331, 284.19614, ..., 289.02716, 291.23227, 284.73474], [288.91437, 279.47214, 283.40421, ..., 291.61411, 287.77469, 288.20865], [282.31927, 280.05029, 276.95325, ..., 293.90955, 281.88361, 285.65167], ..., [285.70764, 288.88840, 284.48535, ..., 296.08032, 287.18237, 290.89032], [291.43884, 286.72595, 285.96793, ..., 291.33618, 285.77768, 298.94879], [292.08160, 285.13284, 286.04916, ..., 297.15613, 286.38803, 297.28705]], [[290.95163, 273.97751, 274.87439, ..., 284.80038, 284.23923, 285.28247], [291.16397, 277.74927, 276.17648, ..., 287.65363, 279.37836, 283.88458], [276.81784, 266.95358, 274.32608, ..., 293.02505, 273.87622, 276.11713], ..., [284.36252, 286.50067, 279.81998, ..., 287.59680, 283.87097, 284.26755], [280.88950, 278.75800, 279.47577, ..., 283.85809, 282.34521, 285.97562], [288.61966, 277.95572, 278.95020, ..., 290.48328, 283.03159, 286.02466]], ..., [[298.09787, 282.45172, 285.61172, ..., 289.49396, 288.68692, 293.03949], [290.59064, 281.55106, 281.71564, ..., 293.01630, 282.79587, 282.98859], [281.80295, 273.76639, 274.61517, ..., 295.65781, 290.84180, 285.83484], ..., [287.58200, 285.37549, 285.24829, ..., 289.19543, 288.25684, 293.97998], [290.55267, 282.15195, 282.19516, ..., 291.93628, 285.44458, 291.72324], [288.19043, 285.62927, 282.38037, ..., 292.88782, 287.45178, 290.24518]], [[304.46838, 292.06061, 297.02286, ..., 299.63669, 309.29483, 299.74841], [297.14456, 284.42841, 286.97556, ..., 302.85327, 293.15198, 294.41904], [296.79984, 289.96384, 284.70370, ..., 303.18631, 290.42902, 296.14465], ..., [298.53979, 293.42755, 293.33881, ..., 299.61688, 297.83133, 306.25174], [295.39221, 294.91302, 293.83569, ..., 296.42969, 293.09442, 302.43423], [299.00192, 294.17352, 297.43018, ..., 309.22937, 304.85190, 304.86826]], [[300.54495, 287.60162, 294.48050, ..., 301.68207, 300.05453, 297.83627], [297.51968, 290.47333, 289.09503, ..., 300.13434, 296.36554, 294.87833], [294.98929, 279.79208, 276.38223, ..., 303.37628, 292.94040, 293.18671], ..., [298.65790, 292.50079, 295.56171, ..., 296.88638, 298.16879, 303.98874], [297.68076, 291.59485, 293.45502, ..., 299.19214, 302.01318, 301.44196], [298.37375, 299.49026, 293.68933, ..., 305.12808, 293.12344, 304.36499]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[290., 280., 280., ..., 286., 282., 290.], [292., 284., 282., ..., 292., 280., 282.], [282., 274., 274., ..., 292., 284., 292.], ..., [282., 282., 272., ..., 294., 284., 286.], [288., 280., 282., ..., 288., 288., 290.], [286., 278., 280., ..., 290., 296., 292.]], [[292., 274., 280., ..., 284., 292., 282.], [284., 278., 282., ..., 288., 286., 292.], [278., 276., 276., ..., 292., 278., 282.], ..., [280., 288., 282., ..., 296., 282., 286.], [288., 282., 288., ..., 288., 282., 294.], [288., 284., 284., ..., 298., 288., 292.]], [[284., 270., 270., ..., 284., 282., 280.], [288., 274., 272., ..., 284., 272., 278.], [276., 262., 272., ..., 290., 268., 270.], ..., [278., 284., 274., ..., 282., 278., 284.], [278., 272., 276., ..., 280., 276., 286.], [286., 274., 278., ..., 286., 276., 280.]], ..., [[300., 276., 280., ..., 288., 286., 288.], [288., 280., 282., ..., 288., 280., 280.], [276., 272., 270., ..., 290., 294., 286.], ..., [286., 280., 278., ..., 284., 292., 292.], [288., 280., 280., ..., 292., 282., 294.], [284., 282., 272., ..., 288., 286., 284.]], [[306., 294., 292., ..., 298., 310., 296.], [290., 280., 282., ..., 300., 290., 290.], [296., 284., 282., ..., 302., 290., 294.], ..., [298., 290., 296., ..., 294., 294., 302.], [294., 292., 288., ..., 288., 290., 300.], [298., 292., 296., ..., 308., 300., 306.]], [[298., 284., 288., ..., 298., 294., 296.], [298., 290., 284., ..., 298., 294., 294.], [288., 274., 272., ..., 300., 290., 290.], ..., [296., 288., 292., ..., 294., 294., 300.], [294., 286., 292., ..., 294., 298., 298.], [302., 296., 290., ..., 302., 292., 302.]]]])

test_conv2d.test_module[Conv2D657-[((1, 64, 82, 258), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[139.68130, 142.97746, 140.49994, ..., 145.22365, 141.97786, 144.46533], [140.59616, 142.87674, 146.36159, ..., 143.11940, 141.19818, 144.89084], [145.41658, 147.03935, 146.24536, ..., 143.06955, 149.82231, 147.05290], ..., [149.83214, 139.50206, 144.40506, ..., 146.29961, 144.24002, 143.28131], [141.92665, 139.77164, 141.25713, ..., 145.00566, 145.58731, 143.00278], [136.30875, 134.03096, 133.92384, ..., 144.44803, 150.33130, 142.79532]], [[136.41394, 138.66141, 136.47153, ..., 142.92462, 139.59024, 140.55138], [139.76294, 139.79196, 141.56613, ..., 142.71852, 139.77457, 136.16895], [137.55617, 140.21361, 141.93301, ..., 139.82013, 143.13930, 146.36514], ..., [144.42737, 137.87344, 137.92349, ..., 142.97963, 140.26018, 140.57753], [140.75656, 138.36856, 136.52185, ..., 141.37627, 140.79161, 139.72481], [135.99013, 133.39082, 129.33963, ..., 141.34392, 148.56062, 141.06592]], [[136.85872, 138.59691, 135.61876, ..., 140.05391, 138.28432, 137.51491], [140.86377, 138.34789, 139.38673, ..., 142.02406, 139.06714, 137.88063], [138.50555, 139.59467, 145.65196, ..., 139.77132, 144.01984, 140.84387], ..., [144.62343, 141.53523, 140.84198, ..., 146.73369, 143.34290, 140.07097], [133.48883, 135.03371, 131.64578, ..., 144.82588, 142.72092, 140.04265], [131.66405, 135.17677, 129.02388, ..., 144.33714, 144.93228, 146.57899]], ..., [[140.94852, 147.38490, 144.46092, ..., 145.52983, 145.33586, 144.38586], [144.85925, 144.41183, 147.13121, ..., 148.06097, 148.54198, 147.74406], [148.35681, 147.92570, 151.24059, ..., 147.86464, 151.65259, 148.12410], ..., [148.54456, 145.15099, 150.01810, ..., 151.33603, 149.80812, 143.69060], [142.94376, 144.82832, 141.56860, ..., 145.44981, 151.18626, 151.50514], [141.96692, 138.40910, 136.22699, ..., 150.69699, 151.94420, 149.79697]], [[139.20370, 137.52406, 132.63824, ..., 138.20581, 136.89030, 140.17180], [135.08450, 137.07808, 136.18707, ..., 137.40695, 142.49091, 137.00185], [137.45250, 141.48961, 141.63551, ..., 139.94936, 140.55997, 138.06747], ..., [136.44952, 134.79237, 136.01620, ..., 135.06781, 139.81837, 137.97797], [132.87819, 132.90015, 137.15915, ..., 137.64568, 138.49667, 136.63075], [135.28973, 131.20390, 127.75957, ..., 141.89799, 142.73512, 141.94238]], [[142.90421, 138.62547, 138.48596, ..., 144.81721, 143.40930, 137.06366], [143.58890, 140.39371, 146.80316, ..., 143.07785, 143.84616, 143.22990], [143.22871, 143.45198, 145.22859, ..., 143.55971, 144.42458, 148.75331], ..., [142.70978, 141.78336, 143.30566, ..., 141.55254, 144.74760, 145.07043], [140.62057, 141.86652, 138.48811, ..., 144.83662, 143.32187, 137.42531], [139.13910, 137.01773, 133.41878, ..., 145.39781, 144.19722, 148.93900]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[137., 142., 138., ..., 143., 140., 142.], [138., 141., 146., ..., 142., 141., 145.], [142., 146., 145., ..., 141., 148., 145.], ..., [149., 136., 143., ..., 145., 143., 141.], [141., 136., 140., ..., 143., 143., 142.], [134., 131., 131., ..., 143., 148., 141.]], [[134., 136., 136., ..., 141., 139., 137.], [139., 138., 140., ..., 140., 137., 134.], [135., 139., 141., ..., 137., 141., 144.], ..., [143., 135., 136., ..., 141., 140., 138.], [139., 137., 134., ..., 138., 139., 137.], [134., 132., 128., ..., 140., 146., 139.]], [[135., 136., 133., ..., 137., 135., 135.], [137., 136., 137., ..., 140., 136., 135.], [138., 137., 143., ..., 136., 141., 139.], ..., [143., 139., 138., ..., 145., 141., 138.], [131., 134., 131., ..., 143., 141., 139.], [129., 133., 129., ..., 142., 143., 143.]], ..., [[139., 146., 142., ..., 144., 144., 143.], [142., 142., 145., ..., 146., 146., 145.], [147., 144., 149., ..., 146., 148., 146.], ..., [147., 142., 148., ..., 148., 148., 141.], [141., 143., 138., ..., 144., 149., 149.], [139., 137., 135., ..., 147., 149., 148.]], [[136., 136., 130., ..., 135., 135., 137.], [133., 135., 134., ..., 136., 140., 135.], [135., 140., 139., ..., 138., 138., 136.], ..., [134., 133., 134., ..., 133., 137., 135.], [130., 132., 135., ..., 135., 137., 134.], [134., 128., 126., ..., 140., 140., 141.]], [[142., 136., 137., ..., 143., 141., 136.], [143., 139., 144., ..., 142., 142., 141.], [141., 143., 143., ..., 141., 142., 147.], ..., [140., 139., 141., ..., 139., 143., 142.], [139., 141., 136., ..., 143., 141., 135.], [137., 135., 131., ..., 144., 142., 147.]]]])

test_conv2d.test_module[Conv2D654-[((1, 512, 14, 42), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1156.64954, 1176.76343, 1168.21497, ..., 1149.56006, 1153.72046, 1159.07190], [1158.42627, 1164.43750, 1157.51575, ..., 1151.20056, 1144.85437, 1151.62451], [1167.41479, 1174.24268, 1172.41235, ..., 1153.46216, 1146.17896, 1142.07007], ..., [1180.95850, 1179.56226, 1183.75269, ..., 1167.06750, 1155.71069, 1176.31335], [1167.20435, 1171.86035, 1162.04565, ..., 1172.92065, 1186.16199, 1179.70630], [1150.00061, 1164.82983, 1162.24316, ..., 1167.47400, 1183.05444, 1176.97595]], [[1147.43030, 1159.61963, 1158.57825, ..., 1148.73230, 1129.94812, 1141.93250], [1149.31018, 1145.95154, 1142.81384, ..., 1146.83118, 1119.70203, 1136.87122], [1151.47595, 1147.02637, 1153.22131, ..., 1138.21631, 1119.58948, 1128.69604], ..., [1154.10706, 1169.01221, 1159.76489, ..., 1144.20435, 1145.49768, 1149.74878], [1150.72925, 1152.86707, 1148.54858, ..., 1157.00208, 1159.39392, 1159.51599], [1131.94885, 1143.75623, 1140.05310, ..., 1152.42212, 1155.94324, 1168.02856]], [[1139.60291, 1151.60205, 1150.38977, ..., 1130.09729, 1125.29834, 1142.23218], [1151.84717, 1149.80042, 1152.57312, ..., 1142.13940, 1127.47607, 1125.75598], [1145.95154, 1149.57300, 1154.52783, ..., 1131.97546, 1123.73999, 1124.21423], ..., [1150.07361, 1161.59912, 1167.81409, ..., 1137.77502, 1143.48120, 1140.29822], [1154.36047, 1151.88879, 1149.11816, ..., 1156.09546, 1155.18787, 1142.70300], [1128.94080, 1142.15564, 1133.11414, ..., 1148.36121, 1155.94397, 1172.87634]], ..., [[1134.14258, 1155.47900, 1146.09290, ..., 1131.19751, 1134.04968, 1143.58545], [1144.09766, 1150.79065, 1148.11450, ..., 1133.19043, 1123.91455, 1134.61206], [1149.65295, 1146.91943, 1157.38782, ..., 1136.47131, 1125.17200, 1124.33508], ..., [1143.64978, 1148.21472, 1157.01746, ..., 1141.20410, 1149.22583, 1144.99927], [1149.02600, 1155.51501, 1139.87231, ..., 1159.92529, 1164.38123, 1152.61426], [1130.77771, 1152.55029, 1149.29370, ..., 1156.57971, 1154.79211, 1156.82410]], [[1161.81921, 1181.61121, 1181.82715, ..., 1161.98901, 1148.77075, 1167.52808], [1169.66528, 1167.15881, 1173.09851, ..., 1154.22961, 1147.13123, 1152.63757], [1180.99426, 1177.41785, 1187.37573, ..., 1166.53931, 1151.38000, 1153.59412], ..., [1186.87390, 1181.32275, 1188.40979, ..., 1172.27014, 1163.57971, 1171.64075], [1173.23352, 1174.28833, 1176.46057, ..., 1182.40991, 1183.01624, 1180.92786], [1158.45178, 1172.46448, 1170.10388, ..., 1176.86121, 1181.21704, 1205.52332]], [[1167.44336, 1171.35083, 1161.44446, ..., 1150.68250, 1148.22827, 1158.08496], [1156.96313, 1167.63916, 1170.56885, ..., 1145.12646, 1139.14124, 1142.39062], [1167.73926, 1173.01440, 1169.98047, ..., 1161.69873, 1137.28723, 1141.48267], ..., [1168.99756, 1168.15479, 1184.13293, ..., 1149.94141, 1158.12537, 1171.47327], [1162.64465, 1176.92029, 1163.42090, ..., 1159.10681, 1166.23975, 1172.95715], [1147.28406, 1176.58105, 1164.60522, ..., 1179.63672, 1159.03186, 1179.66260]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1184., 1224., 1184., ..., 1184., 1208., 1200.], [1184., 1216., 1176., ..., 1208., 1192., 1176.], [1160., 1256., 1240., ..., 1168., 1168., 1160.], ..., [1248., 1192., 1256., ..., 1184., 1200., 1208.], [1200., 1184., 1200., ..., 1240., 1216., 1208.], [1200., 1216., 1240., ..., 1248., 1224., 1248.]], [[1208., 1208., 1264., ..., 1240., 1192., 1160.], [1216., 1176., 1224., ..., 1208., 1144., 1176.], [1208., 1200., 1216., ..., 1184., 1160., 1152.], ..., [1232., 1248., 1264., ..., 1184., 1232., 1240.], [1232., 1192., 1216., ..., 1240., 1288., 1232.], [1224., 1200., 1200., ..., 1248., 1224., 1200.]], [[1184., 1192., 1208., ..., 1184., 1168., 1168.], [1208., 1216., 1184., ..., 1216., 1176., 1200.], [1200., 1184., 1248., ..., 1192., 1176., 1168.], ..., [1208., 1216., 1248., ..., 1176., 1168., 1208.], [1232., 1192., 1208., ..., 1208., 1200., 1168.], [1176., 1208., 1240., ..., 1200., 1248., 1168.]], ..., [[1176., 1208., 1192., ..., 1168., 1160., 1168.], [1176., 1192., 1192., ..., 1176., 1176., 1200.], [1184., 1192., 1208., ..., 1152., 1152., 1176.], ..., [1232., 1200., 1216., ..., 1184., 1216., 1184.], [1208., 1176., 1168., ..., 1176., 1232., 1160.], [1192., 1176., 1200., ..., 1208., 1192., 1184.]], [[1208., 1240., 1232., ..., 1200., 1184., 1192.], [1224., 1224., 1224., ..., 1224., 1184., 1168.], [1232., 1248., 1256., ..., 1224., 1192., 1152.], ..., [1248., 1224., 1224., ..., 1216., 1184., 1200.], [1200., 1192., 1208., ..., 1240., 1240., 1208.], [1216., 1232., 1224., ..., 1240., 1248., 1288.]], [[1200., 1264., 1256., ..., 1184., 1168., 1160.], [1200., 1232., 1264., ..., 1192., 1224., 1200.], [1240., 1256., 1264., ..., 1184., 1184., 1176.], ..., [1224., 1208., 1216., ..., 1208., 1240., 1208.], [1240., 1256., 1256., ..., 1200., 1240., 1248.], [1208., 1272., 1208., ..., 1256., 1232., 1208.]]]])

test_conv2d.test_module[Conv2D159-[((1, 512, 30, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 521.78271, 771.98560, 761.84814, ..., 773.63934, 761.80328, 765.90924], [ 779.11548, 1166.92773, 1159.25806, ..., 1152.54932, 1181.44287, 1151.84302], [ 781.44098, 1146.05664, 1135.97559, ..., 1146.01709, 1170.31445, 1151.90234], ..., [ 772.10748, 1152.82800, 1136.93152, ..., 1153.67175, 1168.51062, 1155.76587], [ 777.68109, 1155.53931, 1176.51172, ..., 1173.64404, 1160.93384, 1156.78760], [ 776.92865, 1174.00732, 1176.74500, ..., 1167.87683, 1159.65833, 1162.13464]], [[ 527.71771, 786.65637, 774.44232, ..., 780.00885, 769.09192, 781.26898], [ 767.74579, 1178.49646, 1167.35889, ..., 1159.80103, 1172.38257, 1165.52576], [ 777.45703, 1162.93762, 1146.32690, ..., 1152.54150, 1174.53809, 1160.54907], ..., [ 773.30762, 1162.76587, 1150.14038, ..., 1172.42761, 1177.46802, 1173.11230], [ 776.96173, 1165.89941, 1169.63599, ..., 1172.91125, 1162.91162, 1173.67603], [ 772.83441, 1173.61755, 1172.35889, ..., 1163.01160, 1179.80676, 1168.33838]], [[ 519.59729, 782.86591, 769.76685, ..., 768.65704, 769.06665, 774.02087], [ 758.76709, 1159.57019, 1143.34167, ..., 1152.14441, 1183.45959, 1157.61267], [ 781.83704, 1161.34448, 1128.35437, ..., 1148.20886, 1168.46155, 1146.24585], ..., [ 766.94189, 1153.02075, 1133.79236, ..., 1158.43091, 1164.07434, 1151.19324], [ 769.67432, 1157.46179, 1164.87830, ..., 1169.79065, 1155.35535, 1161.48535], [ 760.75580, 1168.29028, 1163.93176, ..., 1154.24121, 1157.62842, 1147.45886]], ..., [[ 511.13129, 763.54443, 750.27759, ..., 766.30219, 749.66016, 762.24243], [ 767.85571, 1167.31677, 1151.79431, ..., 1154.17126, 1172.24084, 1159.30127], [ 769.22479, 1147.41248, 1143.28418, ..., 1145.99939, 1176.18701, 1149.34326], ..., [ 754.97430, 1149.20618, 1135.90137, ..., 1162.98450, 1169.85840, 1161.73645], [ 760.58301, 1157.94971, 1171.35107, ..., 1175.77576, 1155.91125, 1167.62524], [ 755.84821, 1167.20276, 1169.31067, ..., 1150.81665, 1169.01355, 1162.08435]], [[ 514.23547, 773.38245, 754.68268, ..., 757.89386, 757.32587, 757.48560], [ 748.09393, 1143.57837, 1136.80066, ..., 1129.60559, 1169.85681, 1139.84180], [ 765.26605, 1139.48914, 1112.48621, ..., 1130.51501, 1152.92249, 1134.57092], ..., [ 757.48547, 1136.14087, 1128.83203, ..., 1148.67542, 1153.61743, 1137.84827], [ 753.70276, 1141.99097, 1153.70581, ..., 1151.94287, 1159.06323, 1145.92834], [ 756.57581, 1164.71387, 1157.60242, ..., 1140.97437, 1162.07434, 1150.72900]], [[ 518.08484, 779.77588, 774.78485, ..., 769.76001, 762.52191, 769.53729], [ 767.98004, 1163.49622, 1164.94080, ..., 1159.65344, 1185.15735, 1169.43567], [ 774.54437, 1160.73242, 1157.22302, ..., 1151.16260, 1186.92444, 1165.33936], ..., [ 762.99176, 1168.14856, 1153.11023, ..., 1175.99731, 1191.05969, 1172.70911], [ 767.43500, 1167.40796, 1183.59753, ..., 1169.97620, 1174.86536, 1178.91187], [ 767.49133, 1184.71021, 1175.95923, ..., 1179.13599, 1189.66418, 1173.38074]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 516., 772., 760., ..., 776., 772., 776.], [ 780., 1272., 1240., ..., 1192., 1280., 1192.], [ 780., 1176., 1192., ..., 1216., 1240., 1240.], ..., [ 780., 1240., 1184., ..., 1216., 1208., 1248.], [ 776., 1240., 1224., ..., 1256., 1224., 1256.], [ 772., 1224., 1248., ..., 1232., 1224., 1232.]], [[ 528., 788., 784., ..., 780., 788., 796.], [ 764., 1240., 1248., ..., 1224., 1216., 1240.], [ 780., 1232., 1192., ..., 1200., 1216., 1176.], ..., [ 776., 1224., 1192., ..., 1232., 1232., 1216.], [ 788., 1216., 1208., ..., 1248., 1232., 1224.], [ 800., 1224., 1224., ..., 1232., 1216., 1200.]], [[ 516., 780., 772., ..., 772., 768., 776.], [ 756., 1224., 1200., ..., 1224., 1248., 1208.], [ 776., 1192., 1200., ..., 1184., 1216., 1208.], ..., [ 776., 1216., 1144., ..., 1240., 1240., 1240.], [ 768., 1200., 1232., ..., 1232., 1200., 1248.], [ 764., 1256., 1280., ..., 1184., 1208., 1224.]], ..., [[ 502., 764., 768., ..., 776., 760., 772.], [ 768., 1256., 1200., ..., 1248., 1248., 1240.], [ 768., 1216., 1240., ..., 1184., 1208., 1256.], ..., [ 768., 1232., 1192., ..., 1256., 1256., 1248.], [ 768., 1224., 1256., ..., 1256., 1248., 1256.], [ 764., 1272., 1240., ..., 1216., 1248., 1224.]], [[ 512., 784., 756., ..., 768., 768., 772.], [ 748., 1208., 1152., ..., 1176., 1248., 1160.], [ 780., 1232., 1152., ..., 1176., 1192., 1184.], ..., [ 756., 1184., 1176., ..., 1232., 1224., 1208.], [ 760., 1232., 1208., ..., 1216., 1184., 1168.], [ 756., 1232., 1232., ..., 1168., 1200., 1168.]], [[ 510., 772., 776., ..., 764., 756., 780.], [ 768., 1248., 1208., ..., 1240., 1280., 1224.], [ 780., 1216., 1240., ..., 1232., 1264., 1256.], ..., [ 776., 1248., 1248., ..., 1248., 1288., 1272.], [ 764., 1256., 1248., ..., 1264., 1248., 1240.], [ 760., 1240., 1288., ..., 1264., 1240., 1240.]]]])

test_conv2d.test_module[Conv2D698-[((1, 1024, 30, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[264.72311, 263.50684, 251.94482, ..., 256.85046, 248.22389, 259.47754], [246.06511, 255.65800, 249.46957, ..., 256.04617, 260.51730, 260.74063], [260.40204, 255.25584, 250.16690, ..., 254.35756, 267.60358, 260.00064], ..., [251.20665, 268.67398, 255.23544, ..., 263.01523, 251.27689, 259.02484], [257.37555, 251.71457, 260.47098, ..., 252.21884, 255.30374, 251.15475], [251.20393, 251.28261, 248.16345, ..., 257.48364, 255.03171, 252.38081]], [[266.04736, 255.59874, 252.48994, ..., 258.36649, 249.29938, 259.65564], [241.82397, 254.93405, 254.85239, ..., 256.92804, 262.36517, 259.19308], [255.91403, 258.73575, 247.57126, ..., 257.82587, 263.19635, 261.11169], ..., [251.51538, 262.84399, 252.97836, ..., 262.86530, 248.57382, 262.41840], [253.72368, 253.79926, 256.94888, ..., 252.65794, 251.42796, 247.46701], [252.40047, 252.97467, 250.60547, ..., 259.98920, 254.11836, 256.07074]], [[263.52353, 263.73856, 259.90451, ..., 253.90634, 249.95990, 260.66031], [248.73744, 259.18024, 250.03471, ..., 256.97168, 260.59570, 261.50314], [257.21237, 258.32062, 251.74445, ..., 259.27704, 267.52924, 260.70270], ..., [258.83472, 265.48642, 256.11853, ..., 263.64316, 253.38254, 266.23059], [254.74384, 256.38242, 262.01184, ..., 255.14793, 255.35864, 252.26372], [254.94731, 251.98340, 253.11256, ..., 255.52756, 257.72394, 253.98341]], ..., [[267.20062, 260.13132, 260.16193, ..., 259.75385, 250.89450, 263.64246], [246.39966, 258.24158, 257.84595, ..., 262.10373, 266.68210, 263.36951], [259.48297, 267.69821, 251.63109, ..., 262.93341, 269.93726, 263.53180], ..., [250.94348, 266.80545, 259.85345, ..., 260.04257, 252.64641, 259.24451], [259.75763, 254.30188, 269.63043, ..., 257.36893, 260.32047, 253.69191], [260.88712, 257.40207, 252.15152, ..., 257.04498, 255.94524, 259.00485]], [[261.88080, 264.37531, 257.55661, ..., 257.88452, 253.25885, 263.68466], [248.44650, 255.97665, 259.76547, ..., 256.20605, 267.33606, 260.68762], [260.52661, 260.01004, 255.09549, ..., 258.71600, 266.97015, 260.27295], ..., [256.16440, 263.68799, 257.81165, ..., 265.58377, 255.24490, 259.50070], [257.20560, 256.70007, 265.25500, ..., 259.97409, 261.19336, 255.41943], [259.68588, 250.47034, 251.96732, ..., 261.56427, 256.22430, 260.17059]], [[250.31989, 247.31422, 241.92099, ..., 241.60101, 239.46082, 243.95340], [231.13937, 245.08945, 243.16006, ..., 243.56383, 248.18715, 247.68846], [244.51158, 239.96016, 246.63014, ..., 244.51559, 252.07437, 244.35892], ..., [243.07056, 252.47672, 244.17197, ..., 252.90395, 243.56923, 245.53972], [241.95056, 238.67984, 247.34361, ..., 239.12866, 239.33263, 238.80753], [244.55524, 239.55145, 237.80556, ..., 244.51566, 241.91861, 237.88777]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[262., 260., 249., ..., 256., 247., 256.], [244., 253., 250., ..., 251., 256., 258.], [256., 253., 247., ..., 253., 266., 260.], ..., [252., 264., 254., ..., 262., 250., 254.], [255., 247., 260., ..., 250., 252., 248.], [248., 247., 242., ..., 252., 254., 251.]], [[262., 253., 252., ..., 255., 249., 256.], [237., 252., 251., ..., 253., 260., 254.], [256., 254., 245., ..., 256., 260., 258.], ..., [251., 260., 249., ..., 260., 246., 258.], [253., 252., 254., ..., 247., 251., 246.], [250., 249., 248., ..., 258., 251., 252.]], [[260., 258., 256., ..., 251., 246., 258.], [245., 256., 247., ..., 253., 260., 258.], [254., 258., 249., ..., 254., 264., 256.], ..., [256., 264., 250., ..., 258., 250., 262.], [251., 252., 258., ..., 250., 251., 250.], [252., 251., 251., ..., 256., 254., 253.]], ..., [[264., 260., 256., ..., 254., 246., 260.], [241., 252., 255., ..., 258., 260., 262.], [256., 264., 247., ..., 258., 268., 262.], ..., [249., 264., 253., ..., 258., 250., 258.], [258., 253., 262., ..., 256., 258., 252.], [260., 252., 251., ..., 256., 252., 254.]], [[258., 260., 255., ..., 253., 251., 260.], [247., 255., 258., ..., 251., 264., 258.], [260., 254., 251., ..., 258., 266., 260.], ..., [253., 262., 255., ..., 260., 254., 256.], [254., 256., 266., ..., 258., 256., 252.], [256., 247., 249., ..., 258., 252., 256.]], [[247., 245., 239., ..., 239., 237., 241.], [229., 243., 243., ..., 240., 248., 243.], [242., 236., 242., ..., 240., 246., 241.], ..., [242., 252., 240., ..., 250., 241., 243.], [239., 233., 243., ..., 236., 236., 238.], [242., 235., 235., ..., 244., 235., 238.]]]])

test_conv2d.test_module[Conv2D708-[((1, 64, 64, 64), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[251.70667, 242.41440, 258.37198, ..., 250.48686, 250.37762, 252.81429], [253.46739, 257.63794, 250.82333, ..., 246.54305, 254.12085, 250.87442], [250.86938, 252.44183, 242.24339, ..., 248.25790, 251.52341, 246.34061], ..., [248.63350, 252.63028, 251.92473, ..., 246.06876, 245.55853, 242.30894], [255.06992, 247.83916, 246.89696, ..., 240.74294, 242.74391, 243.04744], [252.63432, 258.92563, 251.75301, ..., 243.70520, 251.50586, 251.51599]], [[269.46970, 257.72058, 270.28226, ..., 263.13861, 265.17654, 264.79279], [261.47910, 274.43536, 262.88721, ..., 254.36359, 262.32382, 267.25089], [265.94089, 264.24527, 259.96982, ..., 262.65689, 264.79291, 265.01691], ..., [270.95563, 272.66919, 271.68939, ..., 253.74202, 252.21390, 253.74934], [273.02338, 264.18259, 262.37885, ..., 249.77657, 254.57895, 252.61911], [264.45740, 261.70709, 271.30664, ..., 259.01788, 264.21542, 265.71722]], [[258.66614, 246.73961, 266.41196, ..., 261.93585, 254.90855, 255.70248], [249.01747, 265.21478, 257.20895, ..., 247.96597, 256.66132, 257.54440], [253.02350, 257.10544, 247.60944, ..., 249.22292, 252.64697, 257.43713], ..., [257.76825, 256.50641, 253.06786, ..., 249.99841, 252.19957, 246.69244], [259.41763, 255.63394, 259.40573, ..., 248.21454, 249.02193, 252.50194], [258.41095, 259.95398, 261.54684, ..., 254.09229, 260.98505, 260.07159]], ..., [[254.77812, 242.18396, 258.22522, ..., 252.38921, 254.79535, 253.37184], [250.12535, 259.35953, 253.42458, ..., 244.99950, 251.55179, 256.40552], [252.91881, 254.59091, 245.74670, ..., 254.41074, 255.35884, 259.04703], ..., [257.89114, 255.20049, 256.88800, ..., 248.27295, 250.23373, 248.10085], [259.67789, 255.59996, 251.42270, ..., 243.53600, 244.17410, 244.97963], [255.69153, 254.80269, 261.93262, ..., 251.38591, 259.12439, 249.31073]], [[250.62280, 238.04236, 255.00658, ..., 250.51785, 249.06715, 248.02699], [246.33142, 256.82071, 248.64041, ..., 241.97545, 248.05212, 253.15939], [252.81715, 247.94786, 240.59711, ..., 246.16956, 246.56721, 245.36179], ..., [248.77527, 253.05138, 251.25723, ..., 239.35077, 240.67781, 238.99292], [252.59233, 242.68221, 248.71817, ..., 236.73189, 242.43486, 237.47249], [251.13631, 251.11313, 250.52698, ..., 244.49124, 251.96338, 243.35507]], [[263.66629, 250.54108, 268.58450, ..., 258.18631, 265.04523, 263.28857], [258.42294, 272.63293, 255.13542, ..., 251.93364, 262.26221, 261.07672], [258.46652, 258.23050, 248.19711, ..., 255.38795, 259.24823, 259.78473], ..., [257.28015, 264.62433, 261.57339, ..., 252.89943, 251.67342, 253.25279], [264.01102, 255.45668, 254.24782, ..., 246.87984, 254.31303, 249.58154], [254.40794, 264.77142, 265.84927, ..., 250.11745, 266.02573, 261.72495]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[246., 241., 251., ..., 251., 249., 248.], [252., 254., 248., ..., 244., 250., 250.], [248., 250., 238., ..., 245., 250., 245.], ..., [246., 251., 250., ..., 245., 242., 238.], [254., 247., 243., ..., 234., 239., 241.], [251., 254., 249., ..., 241., 249., 251.]], [[262., 256., 266., ..., 260., 262., 262.], [262., 272., 258., ..., 249., 258., 262.], [264., 258., 256., ..., 258., 258., 264.], ..., [270., 270., 268., ..., 248., 250., 250.], [270., 260., 260., ..., 246., 250., 252.], [260., 256., 270., ..., 256., 260., 258.]], [[262., 241., 262., ..., 260., 251., 253.], [247., 262., 253., ..., 246., 254., 253.], [248., 253., 244., ..., 247., 251., 252.], ..., [258., 251., 252., ..., 250., 248., 244.], [258., 251., 256., ..., 247., 248., 249.], [256., 255., 258., ..., 250., 256., 258.]], ..., [[248., 241., 254., ..., 252., 254., 253.], [247., 254., 255., ..., 242., 246., 253.], [251., 253., 246., ..., 250., 250., 253.], ..., [256., 254., 254., ..., 246., 250., 246.], [255., 253., 251., ..., 239., 243., 242.], [255., 250., 258., ..., 249., 256., 243.]], [[248., 236., 254., ..., 248., 243., 244.], [242., 251., 249., ..., 240., 247., 250.], [248., 246., 238., ..., 239., 243., 241.], ..., [245., 251., 248., ..., 237., 238., 236.], [251., 241., 246., ..., 235., 238., 234.], [252., 250., 247., ..., 240., 247., 240.]], [[260., 248., 266., ..., 256., 262., 260.], [256., 270., 253., ..., 248., 258., 258.], [254., 255., 243., ..., 251., 255., 256.], ..., [254., 262., 260., ..., 248., 248., 249.], [262., 251., 253., ..., 244., 253., 247.], [256., 264., 260., ..., 251., 262., 258.]]]])

test_conv2d.test_module[Conv2D713-[((1, 3, 512, 512), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 4004 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 213120 and static circular buffer region ends at 623904 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f54089382d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f54077e07b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f54077e03e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f540621e69a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f5405656f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f54055efd5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab]

test_conv2d.test_module[Conv2D717-[((1, 320, 32, 32), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[338.70050, 323.45349, 319.80402, ..., 334.55396, 324.00293, 329.25208], [332.45895, 321.92734, 315.91040, ..., 330.56952, 327.79703, 326.78641], [324.72189, 325.68320, 321.05923, ..., 319.94379, 327.52905, 325.74268], ..., [330.69073, 330.21777, 329.53925, ..., 334.94522, 322.37128, 327.10745], [325.93204, 330.60016, 320.49588, ..., 330.46863, 325.05688, 323.46692], [317.95682, 327.36679, 326.09628, ..., 332.45874, 320.79190, 327.51346]], [[335.75330, 321.29361, 318.15250, ..., 326.77414, 316.88602, 326.43372], [329.56570, 319.28995, 319.06570, ..., 327.27762, 319.48053, 328.25903], [320.72534, 322.67035, 322.06625, ..., 323.52283, 327.65900, 324.25638], ..., [326.48071, 333.26370, 325.28336, ..., 331.66821, 322.43472, 323.79251], [320.74637, 322.74609, 317.76254, ..., 327.76642, 326.31531, 321.71631], [316.97992, 319.71439, 322.30222, ..., 332.83566, 324.26620, 331.96188]], [[339.47104, 328.01053, 310.62781, ..., 326.91272, 323.58414, 330.73828], [328.63586, 316.90262, 320.02213, ..., 326.35275, 324.88809, 327.68381], [314.97833, 328.82999, 315.83514, ..., 320.79358, 326.33273, 324.88461], ..., [325.60730, 327.35239, 327.06241, ..., 328.24271, 319.14792, 320.84927], [315.20731, 330.71365, 314.66226, ..., 330.88385, 320.22507, 326.88583], [314.52512, 316.77551, 322.20856, ..., 334.65640, 315.37836, 329.16800]], ..., [[328.23303, 312.28165, 311.25998, ..., 326.70941, 314.81033, 322.04657], [325.75995, 308.28210, 313.14325, ..., 322.24387, 316.30652, 326.35214], [317.63055, 318.91428, 308.97595, ..., 320.67719, 321.24551, 317.92340], ..., [321.67932, 327.81070, 326.92526, ..., 326.47162, 323.10321, 321.05661], [312.54828, 327.29688, 309.25027, ..., 328.40686, 313.21356, 318.99844], [305.12622, 314.38937, 319.88007, ..., 330.16504, 314.21097, 322.19016]], [[346.96677, 327.88281, 327.25516, ..., 333.45486, 327.84940, 334.65955], [340.30771, 324.81058, 322.28790, ..., 333.55225, 332.35434, 336.63339], [330.10840, 335.10757, 325.97449, ..., 323.93640, 335.64294, 331.84344], ..., [336.81192, 332.53995, 333.04385, ..., 338.38916, 335.61343, 332.19675], [326.16684, 343.77118, 324.80826, ..., 342.68814, 331.53342, 333.42920], [317.08618, 324.76462, 330.20917, ..., 344.19879, 326.96936, 336.01852]], [[327.54904, 313.58939, 313.58203, ..., 320.94669, 318.46921, 323.04190], [323.26514, 309.75262, 308.03519, ..., 315.52924, 315.84381, 323.46063], [311.52649, 317.31390, 312.95752, ..., 317.49829, 325.63531, 322.89285], ..., [323.10135, 316.80420, 324.97745, ..., 328.26501, 320.24402, 320.66190], [316.13147, 321.59152, 309.33246, ..., 322.94275, 319.75012, 321.13870], [308.34329, 315.67856, 317.01245, ..., 325.94843, 314.12183, 327.95062]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[330., 316., 318., ..., 328., 328., 324.], [328., 316., 314., ..., 326., 324., 322.], [324., 322., 318., ..., 322., 328., 328.], ..., [328., 326., 328., ..., 332., 312., 324.], [322., 332., 314., ..., 324., 324., 318.], [318., 320., 324., ..., 330., 316., 320.]], [[330., 320., 316., ..., 320., 316., 320.], [326., 326., 312., ..., 326., 316., 324.], [320., 316., 318., ..., 320., 326., 324.], ..., [330., 332., 324., ..., 330., 320., 322.], [322., 326., 312., ..., 328., 320., 320.], [318., 314., 318., ..., 336., 322., 330.]], [[340., 322., 312., ..., 324., 320., 328.], [324., 310., 316., ..., 326., 328., 322.], [314., 322., 314., ..., 312., 326., 320.], ..., [320., 324., 328., ..., 326., 318., 316.], [310., 328., 310., ..., 324., 316., 324.], [316., 314., 322., ..., 334., 310., 326.]], ..., [[322., 312., 308., ..., 324., 316., 320.], [326., 302., 312., ..., 324., 312., 324.], [322., 318., 306., ..., 318., 314., 310.], ..., [320., 328., 326., ..., 326., 324., 316.], [308., 322., 300., ..., 322., 314., 320.], [302., 308., 316., ..., 324., 304., 322.]], [[340., 322., 326., ..., 332., 330., 332.], [336., 318., 316., ..., 328., 332., 332.], [328., 328., 320., ..., 318., 334., 336.], ..., [334., 330., 328., ..., 328., 334., 336.], [320., 342., 324., ..., 336., 330., 330.], [320., 320., 332., ..., 340., 324., 336.]], [[324., 312., 312., ..., 316., 314., 322.], [322., 306., 312., ..., 316., 308., 318.], [308., 312., 308., ..., 314., 318., 320.], ..., [314., 312., 320., ..., 326., 316., 320.], [314., 318., 308., ..., 322., 318., 324.], [306., 310., 310., ..., 324., 312., 326.]]]])

test_conv2d.test_module[Conv2D739-[((1, 512, 56, 56), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 497.36063, 743.72913, 752.33588, ..., 754.06488, 763.48169, 518.14270], [ 750.48022, 1124.92065, 1114.46716, ..., 1138.67920, 1142.19141, 767.61371], [ 735.04218, 1124.31079, 1138.94373, ..., 1153.25146, 1155.33875, 778.62872], ..., [ 758.91876, 1153.33325, 1141.32043, ..., 1150.27686, 1135.45837, 754.03815], [ 766.01166, 1162.46497, 1148.92944, ..., 1143.30371, 1117.03052, 742.56665], [ 512.98279, 760.25940, 758.98596, ..., 752.75616, 748.24054, 490.09348]], [[ 508.74554, 765.36542, 772.76422, ..., 770.36273, 783.19739, 518.11224], [ 766.25763, 1144.63184, 1164.46875, ..., 1164.35291, 1175.53809, 776.71545], [ 751.17297, 1151.02185, 1167.55054, ..., 1170.34119, 1179.84290, 793.62970], ..., [ 771.77789, 1159.06360, 1159.73792, ..., 1176.05615, 1152.91809, 770.43158], [ 766.12231, 1173.61292, 1164.37378, ..., 1164.01404, 1147.79346, 764.40033], [ 513.35486, 781.11127, 770.10138, ..., 767.02478, 759.91003, 502.60052]], [[ 503.46832, 753.14154, 766.61829, ..., 772.07214, 766.01648, 518.72321], [ 754.83099, 1127.04114, 1149.13879, ..., 1143.09216, 1160.18176, 785.96210], [ 752.96710, 1138.87695, 1145.68457, ..., 1157.79211, 1162.88049, 782.06146], ..., [ 772.53973, 1159.84033, 1144.95740, ..., 1157.40381, 1141.25049, 756.99878], [ 769.05219, 1168.89746, 1152.00842, ..., 1151.02063, 1126.43909, 751.71155], [ 516.00690, 769.67157, 767.15900, ..., 765.40894, 750.19928, 496.03625]], ..., [[ 495.89862, 749.38470, 749.22351, ..., 757.16339, 755.79346, 506.76453], [ 748.81696, 1121.28284, 1133.64160, ..., 1138.07458, 1141.83813, 768.97607], [ 749.79626, 1129.91443, 1137.12756, ..., 1135.17554, 1140.90808, 768.09851], ..., [ 756.03644, 1145.23401, 1133.98010, ..., 1149.54578, 1129.39941, 758.21497], [ 765.45270, 1145.84778, 1142.35876, ..., 1147.53760, 1120.02991, 741.11023], [ 513.81158, 764.67957, 766.82544, ..., 755.74188, 755.88776, 493.55081]], [[ 502.27301, 763.27429, 757.13342, ..., 775.30981, 778.17102, 522.23639], [ 750.29529, 1142.23560, 1141.91772, ..., 1142.64160, 1165.70679, 783.96429], [ 752.61023, 1136.93591, 1160.67871, ..., 1168.56116, 1163.35181, 789.99725], ..., [ 767.75983, 1151.84998, 1155.97412, ..., 1165.04590, 1145.24280, 763.06415], [ 772.90222, 1176.21411, 1161.91003, ..., 1155.57593, 1131.22302, 750.56586], [ 525.30438, 784.02954, 769.70770, ..., 770.44379, 749.39648, 496.24170]], [[ 510.09927, 764.41443, 771.55518, ..., 777.48303, 777.03998, 523.91309], [ 767.26886, 1166.04065, 1163.42139, ..., 1171.01611, 1175.15173, 780.77081], [ 766.63550, 1156.61682, 1173.90527, ..., 1183.77380, 1175.31372, 791.64764], ..., [ 784.77277, 1172.22473, 1172.33875, ..., 1192.84839, 1166.59265, 770.65729], [ 786.32538, 1188.12915, 1176.15918, ..., 1165.00781, 1156.09204, 758.35406], [ 537.26196, 794.62463, 795.30890, ..., 782.83154, 772.51062, 505.73502]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 500., 760., 764., ..., 764., 772., 506.], [ 764., 1224., 1184., ..., 1208., 1208., 760.], [ 748., 1152., 1208., ..., 1240., 1208., 780.], ..., [ 772., 1232., 1192., ..., 1200., 1192., 760.], [ 760., 1240., 1224., ..., 1208., 1168., 764.], [ 504., 772., 768., ..., 760., 760., 490.]], [[ 516., 776., 788., ..., 768., 792., 520.], [ 772., 1192., 1208., ..., 1208., 1200., 776.], [ 768., 1184., 1216., ..., 1200., 1216., 780.], ..., [ 768., 1216., 1216., ..., 1192., 1184., 768.], [ 768., 1240., 1240., ..., 1216., 1176., 772.], [ 512., 780., 764., ..., 784., 768., 496.]], [[ 502., 764., 772., ..., 768., 772., 520.], [ 744., 1160., 1216., ..., 1184., 1240., 784.], [ 764., 1184., 1216., ..., 1248., 1224., 780.], ..., [ 780., 1264., 1192., ..., 1192., 1176., 756.], [ 756., 1256., 1216., ..., 1192., 1168., 756.], [ 520., 780., 764., ..., 784., 784., 488.]], ..., [[ 496., 752., 756., ..., 768., 752., 496.], [ 748., 1152., 1176., ..., 1200., 1208., 760.], [ 748., 1200., 1184., ..., 1224., 1232., 768.], ..., [ 760., 1240., 1184., ..., 1232., 1184., 752.], [ 764., 1232., 1192., ..., 1216., 1160., 744.], [ 520., 768., 764., ..., 760., 768., 488.]], [[ 502., 772., 772., ..., 776., 788., 520.], [ 780., 1200., 1192., ..., 1176., 1224., 772.], [ 784., 1168., 1216., ..., 1200., 1184., 788.], ..., [ 768., 1224., 1208., ..., 1232., 1176., 776.], [ 768., 1248., 1232., ..., 1208., 1160., 764.], [ 524., 776., 768., ..., 768., 756., 494.]], [[ 516., 760., 780., ..., 780., 780., 528.], [ 760., 1240., 1216., ..., 1200., 1232., 780.], [ 768., 1192., 1272., ..., 1248., 1232., 792.], ..., [ 784., 1264., 1192., ..., 1264., 1192., 772.], [ 784., 1256., 1232., ..., 1200., 1224., 764.], [ 532., 776., 788., ..., 784., 764., 502.]]]])

test_conv2d.test_module[Conv2D452-[((1, 256, 112, 112), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[251.23752, 386.99289, 390.49762, ..., 390.31781, 381.26761, 254.07426], [375.81830, 582.85522, 577.16669, ..., 580.65125, 580.29370, 383.01038], [376.48056, 576.91504, 581.56360, ..., 581.45825, 579.25952, 384.20474], ..., [390.07733, 575.05371, 568.30511, ..., 577.91223, 578.66351, 375.99982], [389.88794, 565.46033, 561.10431, ..., 581.27301, 581.33795, 382.92886], [263.98886, 382.64560, 367.56018, ..., 387.79910, 384.61047, 249.87137]], [[246.27083, 386.31900, 384.02982, ..., 391.59851, 384.36038, 256.67926], [372.33887, 571.15698, 583.33862, ..., 583.13690, 575.59119, 388.50037], [379.17639, 577.35406, 578.45093, ..., 582.62866, 580.24042, 387.86850], ..., [381.51529, 566.33698, 569.97705, ..., 580.47473, 576.61688, 384.58347], [381.97394, 570.35895, 564.78271, ..., 581.52081, 574.94171, 377.43979], [262.98181, 383.47137, 372.76495, ..., 391.00848, 381.54944, 251.58092]], [[248.80283, 384.41263, 387.95108, ..., 383.11279, 389.55121, 256.77267], [365.83481, 573.52026, 583.12775, ..., 588.99786, 579.03986, 388.72485], [384.82367, 580.86182, 576.23633, ..., 574.71881, 575.99585, 382.30643], ..., [386.01627, 559.24677, 568.17560, ..., 575.00745, 573.68402, 381.32071], [386.25970, 567.34613, 556.22406, ..., 572.58594, 575.87592, 373.86295], [266.67899, 380.82211, 370.91428, ..., 386.13138, 383.00378, 246.73080]], ..., [[262.11008, 400.09775, 406.61423, ..., 399.76971, 402.25278, 271.77377], [386.84268, 597.66980, 608.77319, ..., 600.75549, 598.26544, 396.98849], [381.45255, 595.64294, 592.67139, ..., 593.30066, 598.05670, 399.81299], ..., [386.57587, 587.36462, 585.47711, ..., 608.82001, 587.25244, 387.90778], [385.59497, 585.49841, 583.38818, ..., 591.96185, 593.65564, 388.91205], [262.89209, 396.73151, 383.52151, ..., 400.00903, 395.15662, 259.10355]], [[252.08470, 395.97238, 393.96399, ..., 389.14853, 395.40378, 255.88399], [367.33994, 559.52338, 570.91473, ..., 577.91388, 576.68213, 382.54193], [362.54633, 560.78516, 572.15381, ..., 572.98456, 572.18585, 379.62247], ..., [371.67233, 562.20636, 569.48041, ..., 574.74396, 568.75665, 365.78510], [372.96902, 565.78247, 562.28796, ..., 570.42493, 568.98578, 364.54950], [253.70340, 371.85440, 369.30884, ..., 378.01245, 373.99103, 242.69121]], [[248.35930, 375.40823, 386.39133, ..., 385.11423, 386.64511, 256.86969], [375.08432, 563.24304, 573.82886, ..., 568.96759, 574.92273, 382.82297], [369.22260, 561.30725, 568.42865, ..., 562.76166, 559.88202, 367.51059], ..., [385.03351, 564.25403, 564.59845, ..., 570.67749, 568.43732, 376.06085], [385.97699, 553.24396, 559.68707, ..., 563.31335, 564.84875, 371.36252], [251.86627, 374.50800, 365.23035, ..., 376.11804, 375.17081, 245.93604]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[246., 380., 392., ..., 388., 384., 253.], [374., 588., 580., ..., 588., 580., 386.], [376., 572., 592., ..., 580., 584., 388.], ..., [386., 576., 568., ..., 580., 580., 374.], [388., 572., 568., ..., 584., 584., 384.], [260., 382., 364., ..., 384., 382., 249.]], [[247., 386., 384., ..., 388., 386., 254.], [372., 572., 572., ..., 588., 572., 386.], [378., 580., 584., ..., 584., 584., 384.], ..., [376., 564., 568., ..., 576., 576., 382.], [382., 572., 572., ..., 576., 572., 374.], [262., 378., 366., ..., 382., 374., 247.]], [[242., 390., 386., ..., 384., 386., 253.], [368., 576., 584., ..., 588., 588., 394.], [386., 584., 572., ..., 572., 588., 374.], ..., [380., 556., 568., ..., 580., 580., 374.], [392., 572., 552., ..., 580., 572., 370.], [264., 376., 362., ..., 384., 384., 245.]], ..., [[258., 402., 404., ..., 396., 400., 268.], [388., 592., 608., ..., 596., 580., 396.], [374., 596., 592., ..., 596., 596., 392.], ..., [388., 580., 588., ..., 608., 584., 384.], [384., 580., 592., ..., 596., 592., 388.], [262., 398., 384., ..., 396., 380., 256.]], [[246., 394., 392., ..., 388., 396., 254.], [364., 564., 564., ..., 572., 580., 376.], [358., 556., 568., ..., 572., 564., 372.], ..., [370., 568., 568., ..., 580., 568., 370.], [372., 564., 560., ..., 564., 564., 358.], [250., 372., 368., ..., 376., 372., 241.]], [[246., 378., 384., ..., 384., 382., 250.], [376., 564., 580., ..., 580., 576., 384.], [372., 572., 580., ..., 564., 564., 360.], ..., [388., 572., 576., ..., 576., 572., 376.], [380., 556., 564., ..., 572., 576., 366.], [245., 374., 364., ..., 378., 374., 243.]]]])

test_conv2d.test_module[Conv2D741-[((1, 3072, 14, 14), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[3055.93652, 4592.43311, 4597.25098, ..., 4605.70605, 4583.51270, 3060.91968], [4622.92090, 6934.26562, 6955.65625, ..., 6892.29883, 6951.76025, 4622.69385], [4673.33203, 6974.37793, 6958.87354, ..., 6901.98633, 6901.80713, 4606.92676], ..., [4623.69531, 6964.58740, 6972.96387, ..., 6946.20947, 6941.28613, 4622.88770], [4602.88428, 6924.52051, 6980.92920, ..., 6910.50244, 6916.38916, 4602.07324], [3097.94702, 4653.12549, 4678.03760, ..., 4608.85645, 4609.49512, 3065.25439]], [[3079.70410, 4598.24023, 4596.44922, ..., 4590.08447, 4613.59814, 3078.24390], [4596.48291, 6914.81104, 6906.03564, ..., 6907.59229, 6900.90869, 4581.26221], [4612.28809, 6918.80176, 6901.07178, ..., 6898.38379, 6892.04736, 4592.10547], ..., [4597.44580, 6906.08887, 6942.54053, ..., 6917.06201, 6902.06348, 4609.47461], [4578.32275, 6930.56445, 6969.40430, ..., 6906.96240, 6901.57275, 4595.87354], [3078.09497, 4646.82520, 4668.40137, ..., 4596.08447, 4609.98340, 3033.86353]], [[3090.85498, 4638.81396, 4654.39258, ..., 4645.41748, 4656.68457, 3099.96143], [4630.90820, 6972.85547, 6969.70654, ..., 6932.91895, 6955.14307, 4655.29443], [4651.60889, 6979.92969, 6984.43164, ..., 6934.26904, 6933.55713, 4609.46680], ..., [4640.84961, 6962.60791, 6970.20996, ..., 6964.84375, 6938.36670, 4665.34033], [4641.38281, 6974.11182, 7003.38965, ..., 6930.39307, 6934.20947, 4606.32227], [3110.46216, 4655.71387, 4698.30176, ..., 4642.77881, 4607.38379, 3080.45947]], ..., [[3060.36719, 4562.32568, 4571.17334, ..., 4584.32373, 4589.27783, 3084.70410], [4631.30908, 6940.10889, 6950.41699, ..., 6909.62598, 6930.36328, 4644.20215], [4662.18213, 6977.13232, 6949.45410, ..., 6934.82275, 6916.49658, 4600.08252], ..., [4628.93555, 6940.78418, 6967.24072, ..., 6946.71924, 6932.12939, 4640.83594], [4621.01807, 6965.00684, 6989.81006, ..., 6944.34473, 6911.52295, 4602.04639], [3129.59644, 4678.02197, 4725.67871, ..., 4615.85352, 4655.51270, 3090.66235]], [[3017.67554, 4552.51270, 4558.23096, ..., 4552.95508, 4575.43066, 3050.84790], [4568.90430, 6911.83545, 6878.17090, ..., 6867.54736, 6873.07422, 4565.07617], [4586.59277, 6882.14014, 6891.36182, ..., 6859.41260, 6841.23730, 4563.63379], ..., [4554.36230, 6864.58740, 6913.56445, ..., 6854.30322, 6874.01123, 4586.36182], [4576.07812, 6864.57275, 6929.73242, ..., 6846.05811, 6819.38672, 4569.83496], [3040.38232, 4625.45068, 4665.52051, ..., 4592.29199, 4595.81885, 3063.20996]], [[3095.19824, 4649.80078, 4658.93115, ..., 4645.65039, 4665.41846, 3076.21802], [4687.27441, 6969.73291, 6982.90381, ..., 7004.40234, 6973.44775, 4651.46387], [4690.65430, 7024.01221, 7002.86768, ..., 6977.31494, 6963.43750, 4629.79492], ..., [4666.04102, 6985.43213, 7011.86230, ..., 7002.89453, 6946.17920, 4661.46094], [4670.86719, 7003.91309, 7026.33984, ..., 7003.28369, 6981.78223, 4627.26953], [3092.98267, 4665.30713, 4690.57471, ..., 4617.96924, 4621.72119, 3055.59839]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[2048., 2080., 2080., ..., 2112., 2064., 2048.], [2048., 2128., 2112., ..., 2128., 2112., 2080.], [2096., 2080., 2112., ..., 2112., 2080., 2048.], ..., [2096., 2080., 2064., ..., 2112., 2096., 2048.], [2080., 2096., 2128., ..., 2112., 2080., 2064.], [2128., 2096., 2080., ..., 2096., 2080., 2064.]], [[2080., 2064., 2064., ..., 2064., 2080., 2064.], [2080., 2064., 2080., ..., 2112., 2144., 2048.], [2064., 2176., 2096., ..., 2096., 2080., 2096.], ..., [2096., 2112., 2080., ..., 2080., 2080., 2096.], [2048., 2096., 2080., ..., 2080., 2192., 2048.], [2048., 2064., 2080., ..., 2064., 2080., 2064.]], [[2064., 2064., 2112., ..., 2128., 2096., 2048.], [2112., 2096., 2080., ..., 2064., 2160., 2064.], [2064., 2096., 2112., ..., 2096., 2080., 2096.], ..., [2112., 2096., 2080., ..., 2064., 2080., 2048.], [2112., 2048., 2176., ..., 2112., 2112., 2048.], [2048., 2128., 2096., ..., 2080., 2096., 2080.]], ..., [[2048., 2064., 2048., ..., 2080., 2048., 2064.], [2080., 2064., 2080., ..., 2096., 2144., 2096.], [2048., 2144., 2160., ..., 2112., 2096., 2080.], ..., [2128., 2112., 2144., ..., 2096., 2064., 2096.], [2096., 2128., 2144., ..., 2160., 2096., 2080.], [2048., 2064., 2128., ..., 2096., 2096., 2080.]], [[2048., 2064., 2064., ..., 2064., 2128., 2064.], [2080., 2128., 2080., ..., 2112., 2128., 2064.], [2080., 2096., 2080., ..., 2112., 2096., 2048.], ..., [2080., 2064., 2048., ..., 2064., 2096., 2064.], [2064., 2080., 2128., ..., 2096., 2096., 2064.], [2048., 2064., 2048., ..., 2064., 2064., 2064.]], [[2064., 2080., 2096., ..., 2048., 2080., 2048.], [2144., 2064., 2096., ..., 2160., 2096., 2080.], [2112., 2112., 2096., ..., 2144., 2112., 2112.], ..., [2080., 2112., 2128., ..., 2112., 2064., 2048.], [2048., 2128., 2160., ..., 2080., 2112., 2080.], [2064., 2064., 2048., ..., 2096., 2112., 2064.]]]])

test_conv2d.test_module[Conv2D796-[((1, 3, 1280, 1280), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 209715200 B L1 buffer across 64 banks, where each bank needs to store 3276800 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f54089217e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f54088ece9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations13data_movement35InterleavedToShardedDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKSt6vectorINS0_6TensorESaISE_EERKSD_ISt8optionalISE_ESaISK_EE+0x1cf) [0x7f540957529f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x409139) [0x7f5405629139] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f54077e039a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x40052f) [0x7f540562052f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff92f) [0x7f540561f92f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff708) [0x7f540561f708] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff66c) [0x7f540561f66c] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff44d) [0x7f540561f44d] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fee51) [0x7f540561ee51] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fe8d9) [0x7f540561e8d9] --- std::tuple<tt::tt_metal::Tensor, ttnn::operations::sliding_window::ParallelConfig, ttnn::operations::sliding_window::ParallelConfig> ttnn::operations::conv::shard_or_reshard_tensor_if_required<tt::tt_metal::v0::IDevice>(tt::tt_metal::v0::IDevice*, tt::tt_metal::Tensor const&, ttnn::operations::conv::conv2d::Conv2dConfig const&, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, bool, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0x697) [0x7f54055ef747] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702]

test_conv2d.test_module[Conv2D797-[((1, 32, 640, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 57843712 B L1 buffer across 64 banks, where each bank needs to store 903808 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f54089217e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f54088ece9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f540621f833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f540621f699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f54077e039a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f540621e69a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f5405656f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f54055efd5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464]

test_conv2d.test_module[Conv2D836-[((1, 640, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 654.50519, 973.59009, 969.08258, ..., 955.14825, 958.49182, 959.39984], [ 969.73499, 1435.56506, 1451.75220, ..., 1445.22778, 1429.00342, 1429.97485], [ 989.21436, 1437.22729, 1436.42517, ..., 1440.89038, 1444.17676, 1449.64124], ..., [ 981.92206, 1456.28479, 1456.32446, ..., 1434.30029, 1462.28516, 1441.13770], [ 978.96320, 1453.90051, 1443.57117, ..., 1437.06238, 1433.77966, 1429.47729], [ 967.86401, 1429.91614, 1435.22778, ..., 1439.87659, 1441.83618, 1441.99561]], [[ 679.50281, 996.71301, 986.07324, ..., 969.56970, 969.85706, 978.18726], [ 986.27905, 1437.59717, 1456.14343, ..., 1476.72449, 1439.32141, 1430.29053], [ 990.72406, 1439.52722, 1415.56702, ..., 1447.38513, 1458.04089, 1451.41711], ..., [ 985.39746, 1473.47461, 1446.93262, ..., 1437.40076, 1472.49536, 1448.40649], [ 987.77728, 1461.63599, 1449.19556, ..., 1445.95154, 1445.45422, 1438.33923], [ 976.26086, 1440.06555, 1453.22241, ..., 1462.93640, 1447.16882, 1455.15015]], [[ 651.64880, 976.19904, 965.94727, ..., 950.44476, 947.63330, 948.88245], [ 956.59674, 1421.86658, 1447.33081, ..., 1445.15161, 1439.00317, 1426.13757], [ 952.34601, 1425.62537, 1419.71484, ..., 1445.53821, 1444.77319, 1450.20471], ..., [ 957.41168, 1457.22119, 1455.69861, ..., 1422.75037, 1451.20081, 1436.44629], [ 952.13116, 1452.26624, 1432.20776, ..., 1434.53918, 1428.67896, 1411.73254], [ 951.92804, 1426.70215, 1432.68665, ..., 1439.23425, 1432.60168, 1448.67090]], ..., [[ 656.55603, 997.41443, 977.83557, ..., 966.60406, 964.34613, 974.37024], [ 964.19464, 1429.62622, 1452.58911, ..., 1442.04126, 1441.99072, 1431.30896], [ 963.68512, 1445.38940, 1433.76917, ..., 1449.48657, 1454.68652, 1438.76343], ..., [ 978.67596, 1461.70178, 1435.85193, ..., 1423.86816, 1453.58789, 1441.73535], [ 965.73059, 1450.66553, 1434.49744, ..., 1450.17468, 1435.81152, 1429.71423], [ 964.80658, 1429.89856, 1447.00513, ..., 1440.13489, 1427.56824, 1436.56189]], [[ 651.84949, 971.50330, 958.67767, ..., 951.61224, 943.73639, 947.55682], [ 960.81171, 1435.79016, 1444.18945, ..., 1432.51428, 1438.14868, 1428.24219], [ 971.15021, 1438.88647, 1422.40381, ..., 1447.54590, 1439.08655, 1443.99231], ..., [ 968.26324, 1458.63538, 1443.93201, ..., 1428.23572, 1445.60791, 1440.41809], [ 972.50659, 1451.25085, 1439.47900, ..., 1434.05115, 1420.21875, 1416.09717], [ 949.56647, 1431.43665, 1441.24438, ..., 1440.79382, 1419.00879, 1432.83154]], [[ 649.17719, 978.26575, 970.31433, ..., 955.67944, 959.25519, 967.92847], [ 954.30151, 1433.49023, 1452.95227, ..., 1457.49939, 1454.38879, 1435.56970], [ 972.30945, 1453.92468, 1437.42834, ..., 1456.75720, 1457.09900, 1451.26257], ..., [ 967.75555, 1462.58069, 1441.38708, ..., 1443.56396, 1468.27136, 1461.04041], [ 972.40674, 1466.32471, 1436.04102, ..., 1456.26562, 1443.54932, 1429.04968], [ 959.96594, 1447.67688, 1451.35669, ..., 1460.09131, 1442.24463, 1452.40063]]]]), compiled_model=tensor([[[[ 648., 968., 972., ..., 964., 960., 940.], [ 952., 1560., 1600., ..., 1560., 1536., 1584.], [ 988., 1584., 1568., ..., 1608., 1536., 1672.], ..., [ 964., 1648., 1568., ..., 1560., 1632., 1624.], [ 996., 1632., 1616., ..., 1576., 1624., 1584.], [ 972., 1544., 1584., ..., 1584., 1584., 1544.]], [[ 684., 1000., 980., ..., 988., 956., 976.], [ 992., 1584., 1608., ..., 1608., 1616., 1576.], [ 988., 1544., 1568., ..., 1600., 1592., 1616.], ..., [ 988., 1632., 1568., ..., 1584., 1632., 1560.], [1016., 1632., 1568., ..., 1520., 1592., 1544.], [ 988., 1584., 1600., ..., 1608., 1520., 1600.]], [[ 636., 968., 972., ..., 964., 956., 964.], [ 964., 1512., 1576., ..., 1592., 1552., 1528.], [ 956., 1576., 1592., ..., 1608., 1496., 1576.], ..., [ 960., 1616., 1568., ..., 1504., 1576., 1568.], [ 960., 1560., 1536., ..., 1520., 1544., 1536.], [ 984., 1560., 1544., ..., 1528., 1600., 1520.]], ..., [[ 652., 992., 992., ..., 980., 964., 980.], [ 964., 1552., 1584., ..., 1552., 1560., 1528.], [ 984., 1568., 1560., ..., 1536., 1520., 1512.], ..., [ 976., 1600., 1544., ..., 1536., 1592., 1560.], [ 968., 1528., 1536., ..., 1520., 1488., 1496.], [ 972., 1552., 1560., ..., 1528., 1520., 1520.]], [[ 656., 968., 976., ..., 976., 956., 964.], [ 980., 1632., 1656., ..., 1600., 1584., 1560.], [ 988., 1576., 1568., ..., 1624., 1584., 1608.], ..., [ 956., 1616., 1688., ..., 1616., 1600., 1608.], [ 984., 1664., 1552., ..., 1624., 1608., 1576.], [ 960., 1592., 1584., ..., 1552., 1528., 1568.]], [[ 656., 984., 972., ..., 980., 960., 972.], [ 980., 1576., 1624., ..., 1584., 1568., 1592.], [ 980., 1632., 1592., ..., 1592., 1608., 1600.], ..., [ 968., 1656., 1584., ..., 1616., 1728., 1616.], [ 984., 1624., 1552., ..., 1600., 1568., 1584.], [ 972., 1592., 1568., ..., 1560., 1552., 1616.]]]])

test_conv2d.test_module[Conv2D834-[((1, 320, 80, 80), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[333.50986, 497.81186, 479.87344, ..., 493.99860, 491.04401, 472.69583], [486.65344, 724.52570, 712.26727, ..., 733.45154, 733.95581, 711.10992], [501.20642, 717.88751, 731.44110, ..., 726.65729, 730.56934, 719.31531], ..., [483.02588, 724.19110, 733.08136, ..., 739.07385, 720.43494, 711.99652], [493.88528, 737.57812, 736.58191, ..., 719.79199, 721.53540, 720.91516], [486.84531, 728.93280, 735.62457, ..., 729.76514, 720.50690, 716.21857]], [[335.19482, 496.40094, 481.78943, ..., 495.97427, 496.60641, 478.11218], [491.25797, 725.42560, 718.34186, ..., 735.34467, 739.54846, 721.60950], [493.94748, 719.93616, 718.67859, ..., 729.08203, 728.98627, 727.07129], ..., [481.28583, 727.51575, 723.43103, ..., 730.80365, 717.67535, 711.77606], [493.26144, 742.63422, 725.72424, ..., 730.66754, 730.59106, 722.03992], [484.54739, 729.90472, 736.30792, ..., 738.63641, 714.50568, 723.39215]], [[332.32150, 494.19730, 468.84015, ..., 486.24854, 484.31049, 461.85583], [492.24582, 725.35712, 705.04211, ..., 738.54333, 732.28314, 715.52338], [489.84308, 720.36127, 716.12231, ..., 721.33691, 721.49286, 716.15204], ..., [482.62888, 715.89801, 728.41168, ..., 726.31158, 708.59125, 709.02167], [483.09814, 734.24780, 731.38562, ..., 718.71100, 724.02747, 727.80670], [479.64746, 715.51392, 735.75214, ..., 730.93445, 720.22705, 712.24963]], ..., [[331.24115, 487.12750, 468.42676, ..., 483.32385, 478.42880, 452.01089], [485.58356, 715.72675, 702.14661, ..., 720.18005, 723.00043, 707.90350], [492.59448, 711.98279, 710.60614, ..., 717.23285, 710.79456, 716.65289], ..., [476.29041, 709.80084, 708.05786, ..., 715.53937, 694.08582, 694.98694], [483.55493, 723.28033, 728.27417, ..., 714.63190, 713.70258, 704.09955], [476.49832, 707.54437, 719.04962, ..., 721.94434, 705.43286, 707.06073]], [[329.12503, 499.32309, 474.99881, ..., 491.67453, 492.06052, 467.35495], [483.53391, 726.50348, 707.99237, ..., 732.32141, 722.06598, 711.96698], [485.04507, 707.19672, 714.00922, ..., 718.05377, 723.95593, 713.11078], ..., [472.01776, 721.71539, 719.94025, ..., 724.03711, 711.81970, 700.37335], [487.81140, 734.41534, 727.71106, ..., 718.97461, 716.61182, 721.17035], [478.30362, 725.39557, 732.70465, ..., 719.53577, 715.03479, 711.25684]], [[333.33139, 498.42355, 477.16354, ..., 485.33859, 491.23486, 465.04138], [482.81784, 706.99658, 698.21448, ..., 723.54016, 720.74432, 712.98810], [487.04718, 695.54553, 697.82361, ..., 710.36444, 718.12506, 709.40332], ..., [467.90768, 704.78705, 715.48596, ..., 712.29565, 709.21735, 696.17645], [479.82907, 721.21667, 720.03015, ..., 707.58655, 719.24231, 708.71094], [480.25320, 713.92224, 721.56781, ..., 715.64178, 704.64270, 700.77319]]]]), compiled_model=tensor([[[[328., 498., 480., ..., 494., 488., 472.], [486., 736., 716., ..., 740., 736., 708.], [492., 736., 720., ..., 728., 732., 728.], ..., [484., 724., 732., ..., 724., 716., 720.], [480., 728., 736., ..., 716., 720., 724.], [488., 728., 728., ..., 728., 724., 724.]], [[338., 492., 476., ..., 494., 506., 472.], [486., 744., 728., ..., 736., 740., 728.], [486., 740., 736., ..., 736., 728., 736.], ..., [482., 732., 736., ..., 740., 720., 736.], [480., 740., 724., ..., 744., 752., 724.], [474., 732., 740., ..., 748., 720., 724.]], [[328., 490., 468., ..., 482., 482., 466.], [492., 720., 708., ..., 732., 732., 732.], [486., 732., 724., ..., 720., 716., 720.], ..., [476., 708., 732., ..., 736., 704., 720.], [486., 744., 728., ..., 724., 712., 716.], [476., 720., 736., ..., 732., 720., 724.]], ..., [[330., 482., 468., ..., 486., 476., 450.], [486., 728., 716., ..., 728., 736., 712.], [492., 744., 716., ..., 728., 720., 724.], ..., [472., 716., 712., ..., 724., 708., 720.], [482., 728., 724., ..., 724., 720., 716.], [474., 716., 736., ..., 732., 720., 720.]], [[324., 488., 478., ..., 494., 488., 464.], [480., 740., 708., ..., 740., 724., 716.], [476., 724., 716., ..., 728., 720., 720.], ..., [468., 720., 720., ..., 728., 708., 708.], [486., 736., 724., ..., 732., 724., 724.], [476., 732., 728., ..., 724., 716., 712.]], [[326., 498., 478., ..., 488., 494., 458.], [484., 716., 704., ..., 720., 724., 712.], [492., 720., 716., ..., 700., 720., 720.], ..., [460., 712., 716., ..., 712., 708., 708.], [478., 712., 712., ..., 716., 724., 716.], [488., 708., 720., ..., 720., 716., 708.]]]])

test_conv2d.test_module[Conv2D822-[((1, 80, 240, 240), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 4648 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 409216 and static circular buffer region ends at 493088 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f54089382d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f54077e07b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f54077e03e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f540621e69a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f5405656f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f54055efd5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab]

test_conv2d.test_module[Conv2D827-[((1, 160, 120, 120), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[170.05986, 254.87895, 246.81956, ..., 244.88637, 245.27785, 251.56261], [248.11342, 363.39496, 372.10095, ..., 362.48056, 367.51755, 371.06741], [240.24106, 368.90457, 357.00177, ..., 373.43884, 367.88937, 368.11240], ..., [249.77422, 363.57474, 359.23523, ..., 368.90594, 373.97708, 369.55896], [255.81168, 373.16879, 345.98865, ..., 364.12228, 377.22354, 370.93591], [257.76224, 372.42761, 369.54126, ..., 361.92218, 360.55725, 371.72665]], [[161.61673, 244.93782, 240.52342, ..., 235.89627, 235.34610, 243.78224], [235.12442, 354.96948, 359.58353, ..., 360.55453, 360.08652, 362.25934], [231.37648, 355.02191, 354.34485, ..., 361.68338, 355.09186, 362.80502], ..., [233.94701, 358.19666, 338.98965, ..., 361.94049, 366.74246, 361.00055], [247.87486, 360.86154, 338.16293, ..., 360.36438, 370.03098, 366.97592], [246.66479, 365.83728, 357.73532, ..., 354.62936, 356.56863, 363.07016]], [[161.49004, 248.03679, 239.82895, ..., 234.77414, 237.51057, 242.55656], [233.88109, 358.07367, 362.42102, ..., 357.41431, 352.93182, 370.31476], [230.34933, 361.04504, 349.79095, ..., 362.19678, 357.50052, 358.08514], ..., [237.67734, 355.63052, 346.44473, ..., 366.03159, 371.05786, 360.18304], [248.02307, 367.36447, 339.75742, ..., 362.18692, 359.69080, 369.52087], [255.93527, 374.65790, 351.67810, ..., 351.59464, 352.66141, 367.84503]], ..., [[168.45676, 250.79431, 240.19173, ..., 239.30101, 240.10481, 245.93848], [242.69405, 369.99612, 366.88052, ..., 367.31674, 365.04520, 378.37192], [247.18019, 368.72952, 360.97711, ..., 365.22968, 367.78192, 372.95447], ..., [244.52884, 363.31683, 353.73468, ..., 376.76910, 378.33273, 375.19769], [253.27628, 370.71002, 344.76990, ..., 370.07687, 371.17868, 375.57916], [261.06085, 379.36841, 360.40335, ..., 360.71768, 362.86957, 370.30243]], [[161.28818, 236.87431, 232.42896, ..., 223.97380, 227.94255, 232.60501], [237.59750, 362.39038, 357.60242, ..., 360.05658, 354.62088, 363.31793], [239.62521, 360.94437, 351.28009, ..., 363.45212, 359.68414, 365.52420], ..., [241.70372, 363.07425, 337.48581, ..., 361.90668, 365.73203, 353.24799], [249.84947, 357.16711, 331.98242, ..., 359.55899, 366.52472, 364.07913], [254.01875, 369.84857, 345.67831, ..., 350.99908, 355.30405, 363.17709]], [[163.23158, 253.46922, 241.08260, ..., 239.00629, 234.52612, 243.06158], [230.94418, 355.52594, 360.95114, ..., 358.75980, 357.85489, 361.41324], [235.24644, 359.96472, 349.36444, ..., 362.07568, 361.76645, 362.97290], ..., [235.26117, 357.88544, 342.64813, ..., 364.07535, 372.64984, 360.83801], [240.89360, 360.45044, 343.59842, ..., 365.17422, 370.30310, 367.49173], [247.83861, 365.16443, 357.16043, ..., 349.51892, 351.17297, 367.56024]]]]), compiled_model=tensor([[[[169., 251., 244., ..., 242., 244., 248.], [244., 358., 366., ..., 356., 362., 362.], [236., 368., 346., ..., 370., 366., 358.], ..., [247., 358., 354., ..., 368., 372., 368.], [251., 374., 344., ..., 368., 370., 366.], [256., 366., 364., ..., 360., 360., 368.]], [[159., 240., 236., ..., 233., 231., 240.], [231., 350., 358., ..., 354., 358., 356.], [229., 350., 344., ..., 356., 350., 358.], ..., [229., 352., 332., ..., 358., 360., 360.], [244., 360., 336., ..., 358., 368., 362.], [244., 368., 350., ..., 358., 358., 360.]], [[160., 244., 238., ..., 231., 234., 238.], [232., 360., 364., ..., 354., 352., 366.], [228., 362., 346., ..., 358., 364., 352.], ..., [235., 352., 344., ..., 362., 364., 362.], [243., 364., 334., ..., 358., 360., 366.], [249., 370., 350., ..., 350., 350., 364.]], ..., [[167., 248., 237., ..., 236., 239., 243.], [240., 368., 362., ..., 370., 362., 374.], [243., 370., 358., ..., 358., 362., 372.], ..., [240., 356., 350., ..., 374., 378., 378.], [250., 368., 346., ..., 368., 372., 380.], [258., 380., 356., ..., 366., 356., 364.]], [[161., 235., 226., ..., 219., 223., 226.], [235., 354., 358., ..., 356., 358., 356.], [237., 364., 344., ..., 364., 360., 362.], ..., [241., 360., 338., ..., 356., 366., 346.], [247., 354., 326., ..., 362., 362., 362.], [251., 368., 350., ..., 352., 350., 362.]], [[160., 248., 238., ..., 237., 232., 237.], [231., 346., 360., ..., 358., 356., 358.], [232., 352., 352., ..., 364., 358., 362.], ..., [230., 354., 342., ..., 354., 368., 356.], [238., 358., 342., ..., 368., 374., 366.], [248., 358., 354., ..., 340., 354., 364.]]]])

test_conv2d.test_module[Conv2D839-[((1, 640, 30, 30), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 645.82019, 967.15887, 946.30927, ..., 938.78455, 968.82623, 972.39215], [ 951.30737, 1449.92725, 1447.50830, ..., 1440.53418, 1452.34998, 1428.63843], [ 955.41748, 1420.10217, 1449.49438, ..., 1452.38367, 1443.79834, 1434.04614], ..., [ 963.79895, 1468.28369, 1456.62085, ..., 1458.95654, 1433.84558, 1429.99609], [ 956.27161, 1458.50281, 1452.71838, ..., 1463.58545, 1445.42468, 1425.03748], [ 951.15674, 1432.52795, 1450.09973, ..., 1445.59778, 1437.08765, 1427.36389]], [[ 643.96814, 962.30151, 945.25085, ..., 937.79712, 969.42639, 955.59161], [ 970.27081, 1448.46960, 1425.17432, ..., 1441.71289, 1436.26135, 1425.28223], [ 963.89435, 1428.78564, 1422.21985, ..., 1460.43591, 1430.13098, 1430.80457], ..., [ 966.24786, 1470.11731, 1451.02881, ..., 1456.04541, 1428.91162, 1434.20569], [ 964.56696, 1464.26318, 1448.56982, ..., 1444.19397, 1447.54932, 1433.19104], [ 945.45251, 1434.88062, 1447.58118, ..., 1438.15234, 1436.99902, 1416.01367]], [[ 654.45697, 965.71240, 950.60187, ..., 946.91400, 968.60120, 958.91345], [ 975.74921, 1442.43152, 1421.67957, ..., 1423.12598, 1427.16443, 1418.09766], [ 972.76050, 1421.39587, 1422.41541, ..., 1452.25513, 1433.09473, 1413.94202], ..., [ 985.89288, 1465.39832, 1441.11646, ..., 1442.65466, 1433.32202, 1438.89661], [ 977.50958, 1458.38770, 1432.45605, ..., 1436.54028, 1439.66809, 1412.18030], [ 957.96069, 1413.84229, 1442.26660, ..., 1434.03821, 1427.01575, 1414.03394]], ..., [[ 643.75378, 954.36694, 937.20624, ..., 942.88794, 953.13818, 953.01862], [ 952.64923, 1420.04053, 1410.70935, ..., 1413.12781, 1438.84631, 1407.61438], [ 953.97607, 1398.61182, 1399.10046, ..., 1446.29333, 1425.27246, 1416.59863], ..., [ 960.89050, 1454.12231, 1433.45654, ..., 1425.59863, 1412.14368, 1426.34387], [ 949.59576, 1452.10669, 1433.66492, ..., 1435.92029, 1433.23279, 1416.27478], [ 939.81366, 1409.08813, 1417.45520, ..., 1410.16602, 1414.14343, 1398.90686]], [[ 645.32086, 960.65546, 945.17786, ..., 944.26953, 958.12964, 961.93628], [ 974.41235, 1440.79077, 1442.45850, ..., 1433.93433, 1446.01086, 1430.88232], [ 967.04889, 1436.72571, 1431.95801, ..., 1469.29614, 1441.10059, 1427.58643], ..., [ 969.01459, 1468.04956, 1464.44189, ..., 1457.65979, 1441.33545, 1436.63123], [ 960.01727, 1476.76416, 1452.01550, ..., 1442.02673, 1449.99487, 1441.12317], [ 953.39612, 1431.91272, 1440.37317, ..., 1437.41382, 1427.47400, 1405.62549]], [[ 648.01178, 959.54816, 943.92218, ..., 956.63733, 970.46429, 962.83698], [ 961.03302, 1433.55432, 1433.49841, ..., 1431.28979, 1453.98865, 1437.17383], [ 951.34155, 1422.82312, 1427.52722, ..., 1472.30957, 1444.27478, 1432.31653], ..., [ 964.05347, 1465.16370, 1437.85608, ..., 1453.39233, 1434.59949, 1434.64282], [ 961.47925, 1451.89795, 1446.28088, ..., 1451.36926, 1454.69800, 1426.97852], [ 946.33014, 1429.79431, 1447.04688, ..., 1437.78528, 1432.01660, 1426.23206]]]]), compiled_model=tensor([[[[ 648., 976., 964., ..., 940., 972., 968.], [ 960., 1664., 1624., ..., 1608., 1616., 1632.], [ 956., 1552., 1608., ..., 1608., 1648., 1560.], ..., [ 972., 1672., 1648., ..., 1640., 1616., 1600.], [ 968., 1608., 1656., ..., 1600., 1640., 1584.], [ 960., 1576., 1616., ..., 1640., 1584., 1560.]], [[ 640., 968., 960., ..., 952., 964., 972.], [ 980., 1584., 1648., ..., 1600., 1616., 1624.], [ 964., 1576., 1552., ..., 1640., 1584., 1584.], ..., [ 964., 1632., 1600., ..., 1608., 1576., 1592.], [ 976., 1632., 1656., ..., 1600., 1560., 1512.], [ 952., 1592., 1608., ..., 1536., 1592., 1528.]], [[ 660., 980., 952., ..., 948., 960., 964.], [ 988., 1592., 1624., ..., 1552., 1560., 1568.], [ 972., 1504., 1568., ..., 1568., 1544., 1528.], ..., [ 988., 1624., 1560., ..., 1600., 1528., 1600.], [ 976., 1592., 1576., ..., 1592., 1576., 1520.], [ 972., 1568., 1592., ..., 1576., 1536., 1536.]], ..., [[ 652., 964., 960., ..., 948., 976., 980.], [ 968., 1520., 1560., ..., 1528., 1560., 1504.], [ 972., 1488., 1544., ..., 1584., 1560., 1544.], ..., [ 980., 1584., 1584., ..., 1560., 1512., 1536.], [ 964., 1600., 1552., ..., 1568., 1528., 1584.], [ 964., 1520., 1544., ..., 1512., 1512., 1480.]], [[ 640., 964., 948., ..., 956., 964., 952.], [ 968., 1608., 1640., ..., 1600., 1608., 1624.], [ 960., 1600., 1592., ..., 1640., 1656., 1608.], ..., [ 960., 1648., 1648., ..., 1640., 1592., 1608.], [ 968., 1664., 1608., ..., 1632., 1584., 1528.], [ 956., 1552., 1584., ..., 1568., 1616., 1512.]], [[ 644., 968., 960., ..., 960., 968., 968.], [ 964., 1600., 1560., ..., 1520., 1616., 1560.], [ 960., 1544., 1592., ..., 1600., 1560., 1560.], ..., [ 960., 1632., 1576., ..., 1616., 1592., 1576.], [ 972., 1576., 1616., ..., 1544., 1576., 1520.], [ 940., 1560., 1552., ..., 1600., 1544., 1624.]]]])

test_conv2d.test_module[Conv2D874-[((1, 3, 640, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 73859072 B L1 buffer across 64 banks, where each bank needs to store 1154048 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f54089217e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f54088ece9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f540621f833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f540621f699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f54077e039a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f54077e02d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f54077df49f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f540621e69a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f54077e6fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f54077e77cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f5405656f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f54055efd5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f54094d9365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f54137e4c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f54137e4b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f54135afb75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x56255e821b32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x56255e811a97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x56255e817574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x56255e830172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x56255e81839b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x56255e81299e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x56255e80c97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x56255e83010b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x56255e80c790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x56255e80ec30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x56255e80d9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x56255e82f4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x56255e811702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x56255e82238c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x56255e81761d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x56255e82c62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x56255e934464]

test_conv2d.test_module[Conv2D884-[((1, 512, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 525.58246, 783.88794, 767.01532, ..., 747.95215, 752.32050, 757.83563], [ 762.52100, 1110.56079, 1142.11621, ..., 1136.47021, 1143.87061, 1126.66956], [ 780.39954, 1125.80920, 1118.03711, ..., 1141.56042, 1144.62781, 1140.71167], ..., [ 771.29150, 1134.47314, 1131.57764, ..., 1138.23901, 1146.53503, 1144.38745], [ 769.11377, 1151.86316, 1148.59143, ..., 1131.17529, 1131.81799, 1130.34094], [ 775.21283, 1141.98792, 1136.81995, ..., 1139.86853, 1143.07666, 1136.47449]], [[ 516.03619, 789.81396, 775.77484, ..., 753.84399, 767.01611, 762.30908], [ 754.72101, 1131.59021, 1143.27637, ..., 1145.83496, 1138.32629, 1135.55896], [ 753.56470, 1139.67883, 1125.88623, ..., 1142.15430, 1156.82983, 1149.36121], ..., [ 764.95026, 1153.46021, 1128.09509, ..., 1146.77832, 1160.69263, 1141.53003], [ 769.73199, 1141.04443, 1135.28857, ..., 1162.66321, 1151.39221, 1136.54675], [ 760.14319, 1151.27563, 1134.78442, ..., 1144.05334, 1136.15784, 1154.14966]], [[ 522.05518, 777.02222, 773.66809, ..., 764.01941, 760.96228, 769.25604], [ 755.60205, 1131.54150, 1153.67896, ..., 1153.92664, 1161.03748, 1142.49915], [ 775.55731, 1124.52856, 1137.35767, ..., 1150.89026, 1152.41101, 1160.92273], ..., [ 770.83264, 1158.07739, 1147.39270, ..., 1151.31897, 1159.88318, 1140.70007], [ 775.80682, 1166.96362, 1144.59534, ..., 1153.68396, 1154.11914, 1145.31482], [ 760.92377, 1141.40894, 1150.65393, ..., 1164.11768, 1145.93665, 1147.65161]], ..., [[ 528.64423, 773.66760, 778.95447, ..., 772.82764, 761.90448, 774.11841], [ 772.59857, 1128.30750, 1161.01465, ..., 1169.52344, 1148.76965, 1146.79846], [ 790.23523, 1148.09192, 1134.84851, ..., 1152.96130, 1162.42065, 1170.11304], ..., [ 788.11249, 1161.39099, 1149.57251, ..., 1162.04150, 1162.23254, 1150.84912], [ 785.71820, 1168.99011, 1159.71533, ..., 1163.72412, 1142.67847, 1153.45300], [ 786.14606, 1149.93994, 1153.85559, ..., 1162.82190, 1163.47253, 1146.56946]], [[ 532.67255, 782.38068, 775.62274, ..., 764.42932, 768.78601, 776.45135], [ 763.90234, 1132.81165, 1153.61487, ..., 1172.53491, 1164.57935, 1158.07617], [ 785.24768, 1153.60303, 1133.48071, ..., 1154.87183, 1169.96448, 1168.38586], ..., [ 785.51660, 1166.79797, 1160.76941, ..., 1167.79175, 1172.04065, 1158.51025], [ 777.36475, 1163.40710, 1146.63379, ..., 1157.59485, 1146.73438, 1146.42480], [ 778.84174, 1163.23389, 1168.50476, ..., 1158.47986, 1162.45715, 1165.56885]], [[ 533.25586, 791.36304, 775.97845, ..., 765.47845, 759.37323, 773.64929], [ 763.78638, 1145.57532, 1176.16016, ..., 1160.25000, 1162.30188, 1147.40051], [ 782.22296, 1149.56421, 1141.39282, ..., 1158.28430, 1169.52539, 1169.26697], ..., [ 782.76013, 1176.20532, 1155.03674, ..., 1163.04443, 1178.67566, 1144.68457], [ 778.50098, 1169.29468, 1151.90820, ..., 1155.39746, 1149.12744, 1144.96631], [ 775.45038, 1166.51660, 1159.29309, ..., 1161.24866, 1159.92712, 1155.28821]]]]), compiled_model=tensor([[[[ 532., 780., 776., ..., 756., 752., 756.], [ 764., 1160., 1200., ..., 1184., 1184., 1176.], [ 780., 1192., 1176., ..., 1176., 1176., 1224.], ..., [ 780., 1216., 1176., ..., 1192., 1216., 1200.], [ 764., 1200., 1176., ..., 1200., 1192., 1192.], [ 776., 1184., 1208., ..., 1184., 1184., 1192.]], [[ 508., 788., 768., ..., 756., 764., 756.], [ 756., 1176., 1200., ..., 1192., 1208., 1184.], [ 756., 1184., 1176., ..., 1208., 1168., 1208.], ..., [ 760., 1200., 1120., ..., 1152., 1216., 1216.], [ 768., 1176., 1192., ..., 1224., 1224., 1200.], [ 756., 1208., 1192., ..., 1192., 1168., 1216.]], [[ 516., 780., 784., ..., 772., 772., 772.], [ 764., 1184., 1248., ..., 1208., 1232., 1208.], [ 792., 1160., 1184., ..., 1168., 1192., 1232.], ..., [ 768., 1240., 1168., ..., 1184., 1232., 1200.], [ 776., 1264., 1168., ..., 1192., 1232., 1216.], [ 772., 1208., 1216., ..., 1224., 1176., 1208.]], ..., [[ 528., 768., 792., ..., 776., 760., 764.], [ 780., 1200., 1232., ..., 1224., 1216., 1200.], [ 788., 1184., 1144., ..., 1208., 1200., 1240.], ..., [ 780., 1224., 1224., ..., 1200., 1216., 1184.], [ 780., 1264., 1200., ..., 1216., 1216., 1240.], [ 784., 1208., 1176., ..., 1232., 1216., 1160.]], [[ 528., 776., 772., ..., 772., 768., 776.], [ 776., 1192., 1240., ..., 1272., 1240., 1184.], [ 796., 1160., 1192., ..., 1232., 1248., 1232.], ..., [ 776., 1240., 1216., ..., 1248., 1224., 1232.], [ 784., 1240., 1200., ..., 1184., 1224., 1200.], [ 784., 1176., 1240., ..., 1192., 1232., 1232.]], [[ 528., 796., 776., ..., 760., 768., 768.], [ 764., 1200., 1272., ..., 1232., 1216., 1184.], [ 788., 1176., 1184., ..., 1176., 1232., 1200.], ..., [ 776., 1216., 1216., ..., 1248., 1232., 1184.], [ 780., 1264., 1192., ..., 1224., 1216., 1216.], [ 780., 1216., 1224., ..., 1224., 1216., 1200.]]]])

test_conv2dtranspose.test_module[Conv2Dtranspose1-[((1, 16, 14, 14), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_conv2dtranspose.test_module[Conv2Dtranspose3-[((1, 128, 14, 14), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_conv2dtranspose.test_module[Conv2Dtranspose7-[((1, 128, 64, 64), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_conv2dtranspose.test_module[Conv2Dtranspose12-[((1, 32, 28, 40), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_cumsum.test_module[Cumsum0-[((1, 128), torch.int32)]]: forge/test/models_ops/test_cumsum.py#L114

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[ 44, 283, 1216, 1976, 2939, 3318, 3745, 4248, 4745, 5428, 5529, 6395, 7151, 7550, 8428, 8804, 8860, 9728, 10522, 10555, 10681, 10800, 11191, 11445, 12269, 13110, 13379, 14348, 14897, 15377, 15858, 15870, 16233, 16593, 17588, 17673, 18495, 19494, 20205, 20493, 21271, 21314, 22210, 22699, 23470, 24227, 24510, 24905, 25687, 26358, 27198, 28167, 28840, 29281, 29392, 29672, 30175, 30881, 31357, 31684, 32383, 32709, 33072, 33546, 34321, 35021, 35239, 35871, 36439, 36451, 36728, 37373, 37483, 37563, 37911, 38432, 38551, 39467, 40348, 40838, 41520, 42039, 42983, 43616, 43785, 44648, 45057, 45290, 45509, 45587, 46322, 46405, 46627, 47485, 47720, 48136, 48582, 49417, 50294, 50506, 51011, 51957, 52113, 52428, 52712, 53462, 53570, 53641, 54488, 54596, 54671, 54755, 55639, 55787, 55931, 56465, 56984, 57244, 58151, 58665, 58840, 59503, 60116, 60273, 60406, 60526, 60775, 61164]], dtype=torch.int32), compiled_model=tensor([[-528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0, -528547840, 0]], dtype=torch.int32)

test_embedding.test_module[Embedding2-[((13, 13), torch.int32), ((32, 12), torch.float32)]]: forge/test/models_ops/test_embedding.py#L1690

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[0.99984, 0.75933, 0.81082, ..., 0.11526, 0.83567, 0.85547], [0.25775, 0.99915, 0.98834, ..., 0.32066, 0.59447, 0.23875], [0.64345, 0.32472, 0.89068, ..., 0.15076, 0.61293, 0.16170], ..., [0.64345, 0.32472, 0.89068, ..., 0.15076, 0.61293, 0.16170], [0.87511, 0.33904, 0.50080, ..., 0.23666, 0.56609, 0.91346], [0.00673, 0.09847, 0.89474, ..., 0.17528, 0.84720, 0.12200]], [[0.73711, 0.09909, 0.35619, ..., 0.56965, 0.16546, 0.11234], [0.33302, 0.36028, 0.64771, ..., 0.21941, 0.05421, 0.93844], [0.64345, 0.32472, 0.89068, ..., 0.15076, 0.61293, 0.16170], ..., [0.99984, 0.75933, 0.81082, ..., 0.11526, 0.83567, 0.85547], [0.25234, 0.08816, 0.76092, ..., 0.39128, 0.56645, 0.74785], [0.87511, 0.33904, 0.50080, ..., 0.23666, 0.56609, 0.91346]], [[0.42596, 0.50547, 0.91240, ..., 0.28809, 0.64706, 0.66510], [0.14971, 0.91964, 0.44563, ..., 0.75058, 0.79596, 0.92326], [0.95056, 0.79388, 0.56700, ..., 0.15438, 0.69562, 0.87758], ..., [0.78108, 0.21591, 0.42164, ..., 0.54785, 0.96241, 0.52677], [0.73711, 0.09909, 0.35619, ..., 0.56965, 0.16546, 0.11234], [0.78108, 0.21591, 0.42164, ..., 0.54785, 0.96241, 0.52677]], ..., [[0.46626, 0.93971, 0.29605, ..., 0.89983, 0.09595, 0.55365], [0.39532, 0.85706, 0.63957, ..., 0.89699, 0.84211, 0.14731], [0.25234, 0.08816, 0.76092, ..., 0.39128, 0.56645, 0.74785], ..., [0.25234, 0.08816, 0.76092, ..., 0.39128, 0.56645, 0.74785], [0.42596, 0.50547, 0.91240, ..., 0.28809, 0.64706, 0.66510], [0.35384, 0.20315, 0.31508, ..., 0.76479, 0.88376, 0.68136]], [[0.84997, 0.32027, 0.92174, ..., 0.49649, 0.56380, 0.10890], [0.33302, 0.36028, 0.64771, ..., 0.21941, 0.05421, 0.93844], [0.52230, 0.14753, 0.22476, ..., 0.12204, 0.15674, 0.20967], ..., [0.84997, 0.32027, 0.92174, ..., 0.49649, 0.56380, 0.10890], [0.58982, 0.24027, 0.61520, ..., 0.09011, 0.42292, 0.67365], [0.35384, 0.20315, 0.31508, ..., 0.76479, 0.88376, 0.68136]], [[0.25605, 0.01695, 0.21613, ..., 0.72000, 0.94547, 0.66541], [0.64345, 0.32472, 0.89068, ..., 0.15076, 0.61293, 0.16170], [0.52230, 0.14753, 0.22476, ..., 0.12204, 0.15674, 0.20967], ..., [0.53279, 0.16676, 0.32086, ..., 0.39857, 0.83622, 0.02676], [0.64345, 0.32472, 0.89068, ..., 0.15076, 0.61293, 0.16170], [0.52230, 0.14753, 0.22476, ..., 0.12204, 0.15674, 0.20967]]]), compiled_model=tensor([[[1.00000, 0.75781, 0.81250, ..., 0.11523, 0.83594, 0.85547], [0.25781, 1.00000, 0.98828, ..., 0.32031, 0.59375, 0.23828], [0.64453, 0.32422, 0.89062, ..., 0.15039, 0.61328, 0.16211], ..., [0.64453, 0.32422, 0.89062, ..., 0.15039, 0.61328, 0.16211], [0.87500, 0.33984, 0.50000, ..., 0.23633, 0.56641, 0.91406], [0.00674, 0.09863, 0.89453, ..., 0.17480, 0.84766, 0.12207]], [[0.46680, 0.94141, 0.29688, ..., 0.89844, 0.09619, 0.55469], [0.46680, 0.94141, 0.29688, ..., 0.89844, 0.09619, 0.55469], [0.64453, 0.32422, 0.89062, ..., 0.15039, 0.61328, 0.16211], ..., [1.00000, 0.75781, 0.81250, ..., 0.11523, 0.83594, 0.85547], [0.25195, 0.08838, 0.76172, ..., 0.39062, 0.56641, 0.74609], [0.87500, 0.33984, 0.50000, ..., 0.23633, 0.56641, 0.91406]], [[0.46680, 0.94141, 0.29688, ..., 0.89844, 0.09619, 0.55469], [0.14941, 0.91797, 0.44531, ..., 0.75000, 0.79688, 0.92188], [0.94922, 0.79297, 0.56641, ..., 0.15430, 0.69531, 0.87891], ..., [0.78125, 0.21582, 0.42188, ..., 0.54688, 0.96094, 0.52734], [0.73828, 0.09912, 0.35547, ..., 0.57031, 0.16504, 0.11230], [0.78125, 0.21582, 0.42188, ..., 0.54688, 0.96094, 0.52734]], ..., [[0.46680, 0.94141, 0.29688, ..., 0.89844, 0.09619, 0.55469], [0.39453, 0.85547, 0.64062, ..., 0.89844, 0.84375, 0.14746], [0.25195, 0.08838, 0.76172, ..., 0.39062, 0.56641, 0.74609], ..., [0.25195, 0.08838, 0.76172, ..., 0.39062, 0.56641, 0.74609], [0.42578, 0.50391, 0.91406, ..., 0.28906, 0.64844, 0.66406], [0.35352, 0.20312, 0.31445, ..., 0.76562, 0.88281, 0.67969]], [[0.85156, 0.32031, 0.92188, ..., 0.49609, 0.56250, 0.10889], [0.33398, 0.35938, 0.64844, ..., 0.21973, 0.05420, 0.93750], [0.52344, 0.14746, 0.22461, ..., 0.12207, 0.15723, 0.20996], ..., [0.85156, 0.32031, 0.92188, ..., 0.49609, 0.56250, 0.10889], [0.58984, 0.24023, 0.61328, ..., 0.09033, 0.42383, 0.67188], [0.35352, 0.20312, 0.31445, ..., 0.76562, 0.88281, 0.67969]], [[0.00000, 0.00000, 0.00000, ..., 0.00000, 0.00000, 0.00000], [0.64453, 0.32422, 0.89062, ..., 0.15039, 0.61328, 0.16211], [0.52344, 0.14746, 0.22461, ..., 0.12207, 0.15723, 0.20996], ..., [0.53125, 0.16699, 0.32031, ..., 0.39844, 0.83594, 0.02673], [0.64453, 0.32422, 0.89062, ..., 0.15039, 0.61328, 0.16211], [0.52344, 0.14746, 0.22461, ..., 0.12207, 0.15723, 0.20996]]])

test_add.test_module[Add0-[((2, 1, 2048), torch.float32), ((1, 2048), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[1.35998, 1.21530, 0.37866, ..., 0.67065, 1.77452, 0.34409]], [[1.10242, 0.80269, 0.46657, ..., 0.80053, 1.56750, 0.11676]]]), compiled_model=tensor([[[1.35998e+00, 1.21530e+00, 3.78659e-01, ..., 6.70653e-01, 1.77452e+00, 3.44086e-01]], [[8.33448e-01, 3.55607e-01, 1.64334e+17, ..., 8.70522e+17, 1.43104e+38, 1.19855e-01]]])

test_add.test_module[Add0-[((2, 1, 1024), torch.float32), ((1, 1024), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[0.73496, 1.12383, 0.26487, ..., 0.84095, 0.90509, 1.49623]], [[0.70367, 0.38336, 0.38810, ..., 1.24081, 1.52155, 1.06615]]]), compiled_model=tensor([[[0.73496, 1.12383, 0.26487, ..., 0.84095, 0.90509, 1.49623]], [[0.46497, 0.02775, 0.21171, ..., 1.54517, 0.98262, 0.53669]]])

test_add.test_module[Add46-[((1, 256), torch.int64)]]: forge/test/models_ops/test_add.py#L18561

TypeError: Dtype mismatch: framework_model.dtype=torch.int64, compiled_model.dtype=torch.int32

test_add.test_module[Add1-[((1, 144, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.54710]], [[0.81906]], [[0.13932]], [[0.18287]], [[0.35826]], [[0.68492]], [[0.54093]], [[0.94728]], [[0.50647]], [[0.68315]], [[0.39973]], [[0.45256]], [[0.07316]], [[0.21970]], [[0.34473]], [[0.56936]], [[0.74851]], [[0.85085]], [[0.21187]], [[0.33311]], [[0.73245]], [[0.96603]], [[0.44794]], [[0.92500]], [[0.47025]], [[0.60375]], [[1.00358]], [[0.08700]], [[0.23607]], [[0.42426]], [[0.35594]], [[0.98284]], [[0.22675]], [[0.32067]], [[0.20152]], [[0.08256]], [[0.25897]], [[0.98064]], [[0.77395]], [[0.79318]], [[0.57714]], [[0.29450]], [[0.63543]], [[0.08399]], [[0.18956]], [[0.29307]], [[0.86631]], [[0.84400]], [[0.32909]], [[0.53280]], [[0.87062]], [[1.04791]], [[0.74928]], [[0.61839]], [[0.88608]], [[0.25644]], [[0.64401]], [[0.16319]], [[0.20430]], [[0.29255]], [[0.77708]], [[0.75192]], [[0.25466]], [[0.70189]], [[0.82533]], [[0.48773]], [[0.56993]], [[0.66669]], [[0.86103]], [[1.03094]], [[0.16553]], [[0.36760]], [[0.74734]], [[0.96511]], [[0.98594]], [[0.99202]], [[0.65035]], [[0.11605]], [[0.59684]], [[0.23804]], [[0.08486]], [[0.99509]], [[0.93102]], [[0.05208]], [[0.64443]], [[0.46661]], [[0.46856]], [[0.32196]], [[0.74312]], [[0.25469]], [[0.73413]], [[0.80369]], [[0.90878]], [[0.73779]], [[0.05597]], [[0.22649]], [[0.80050]], [[0.65549]], [[0.16080]], [[0.26293]], [[1.02121]], [[0.88775]], [[0.33283]], [[0.42500]], [[0.07454]], [[0.54185]], [[0.17431]], [[0.16516]], [[0.52329]], [[0.62591]], [[0.34607]], [[0.84753]], [[0.24657]], [[1.00452]], [[0.89349]], [[0.12920]], [[0.42640]], [[0.57340]], [[0.62379]], [[0.66943]], [[0.74705]], [[0.58079]], [[0.30687]], [[0.78743]], [[0.07121]], [[0.25449]], [[0.42567]], [[0.30728]], [[0.37592]], [[0.14103]], [[0.44448]], [[0.65772]], [[0.22511]], [[0.52518]], [[0.90876]], [[0.49944]], [[0.56474]], [[0.50770]], [[0.65203]], [[0.86876]], [[1.02446]], [[0.86837]], [[1.02555]], [[0.51468]]]]), compiled_model=tensor([[[[0.54710]], [[1.03118]], [[0.92893]], [[0.62879]], [[0.55890]], [[0.75092]], [[0.52217]], [[0.97444]], [[0.85421]], [[1.40651]], [[1.11921]], [[0.41950]], [[0.83422]], [[0.27760]], [[0.68818]], [[0.81579]], [[1.10136]], [[1.20184]], [[0.21235]], [[0.35055]], [[1.10337]], [[1.42166]], [[0.66996]], [[1.56251]], [[0.46938]], [[1.01916]], [[1.89245]], [[0.33222]], [[1.13673]], [[1.05449]], [[0.35387]], [[1.74835]], [[0.61821]], [[0.54663]], [[1.05051]], [[0.12767]], [[0.76178]], [[1.32511]], [[1.58017]], [[1.38191]], [[1.26655]], [[0.92024]], [[0.96435]], [[0.42800]], [[0.22668]], [[1.01316]], [[1.71246]], [[1.63527]], [[0.42556]], [[1.00426]], [[0.96731]], [[1.22182]], [[0.90709]], [[1.23842]], [[1.03729]], [[0.69469]], [[1.11421]], [[0.93466]], [[0.27550]], [[0.39845]], [[0.93591]], [[1.55105]], [[0.52409]], [[1.57280]], [[1.45529]], [[1.00020]], [[1.01537]], [[1.01701]], [[1.37292]], [[1.36592]], [[0.61118]], [[0.88056]], [[1.96725]], [[2.11939]], [[1.54267]], [[1.68906]], [[1.71712]], [[1.67938]], [[1.15078]], [[1.40041]], [[1.18616]], [[2.49083]], [[2.16418]], [[1.34413]], [[1.21542]], [[0.64984]], [[1.25760]], [[0.97684]], [[1.42397]], [[1.94811]], [[1.72450]], [[1.03636]], [[2.13313]], [[2.01792]], [[0.81995]], [[1.32093]], [[1.86134]], [[1.36141]], [[1.74599]], [[1.00111]], [[2.01354]], [[1.89728]], [[0.59222]], [[1.48181]], [[0.94927]], [[1.36550]], [[0.38411]], [[0.35813]], [[1.65095]], [[2.34178]], [[1.30033]], [[1.91318]], [[0.74573]], [[1.68836]], [[1.55071]], [[0.22583]], [[0.98672]], [[1.33987]], [[1.68365]], [[2.20844]], [[1.17020]], [[1.96559]], [[1.91847]], [[1.81202]], [[1.09437]], [[1.29375]], [[1.78303]], [[1.08063]], [[1.61447]], [[0.73249]], [[0.80313]], [[1.58518]], [[0.92088]], [[1.37907]], [[1.43658]], [[1.35610]], [[0.54116]], [[1.06489]], [[0.82060]], [[0.87213]], [[1.91206]], [[0.99282]], [[1.41783]], [[1.10709]]]])

test_add.test_module[Add1-[((1, 288, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.88192]], [[1.15388]], [[0.47414]], [[0.51769]], [[0.69308]], [[1.01974]], [[0.87576]], [[1.28211]], [[0.84129]], [[1.01797]], [[0.73456]], [[0.78738]], [[0.40799]], [[0.55452]], [[0.67955]], [[0.90418]], [[1.08333]], [[1.18567]], [[0.54669]], [[0.66793]], [[1.06727]], [[1.30086]], [[0.78276]], [[1.25982]], [[0.80507]], [[0.93857]], [[1.33840]], [[0.42183]], [[0.57089]], [[0.75908]], [[0.69076]], [[1.31766]], [[0.56157]], [[0.65550]], [[0.53634]], [[0.41738]], [[0.59379]], [[1.31546]], [[1.10877]], [[1.12800]], [[0.91196]], [[0.62932]], [[0.97025]], [[0.41881]], [[0.52438]], [[0.62790]], [[1.20113]], [[1.17882]], [[0.66391]], [[0.86762]], [[1.20544]], [[1.38273]], [[1.08410]], [[0.95321]], [[1.22091]], [[0.59126]], [[0.97883]], [[0.49801]], [[0.53912]], [[0.62737]], [[1.11190]], [[1.08674]], [[0.58949]], [[1.03672]], [[1.16015]], [[0.82255]], [[0.90475]], [[1.00151]], [[1.19585]], [[1.36576]], [[0.50035]], [[0.70243]], [[1.08217]], [[1.29994]], [[1.32077]], [[1.32684]], [[0.98517]], [[0.45087]], [[0.93166]], [[0.57286]], [[0.41968]], [[1.32991]], [[1.26584]], [[0.38690]], [[0.97925]], [[0.80143]], [[0.80338]], [[0.65678]], [[1.07794]], [[0.58951]], [[1.06896]], [[1.13852]], [[1.24360]], [[1.07262]], [[0.39079]], [[0.56131]], [[1.13532]], [[0.99031]], [[0.49562]], [[0.59775]], [[1.35604]], [[1.22257]], [[0.66765]], [[0.75982]], [[0.40936]], [[0.87667]], [[0.50913]], [[0.49998]], [[0.85811]], [[0.96073]], [[0.68090]], [[1.18235]], [[0.58139]], [[1.33935]], [[1.22831]], [[0.46402]], [[0.76122]], [[0.90822]], [[0.95861]], [[1.00425]], [[1.08188]], [[0.91561]], [[0.64170]], [[1.12226]], [[0.40604]], [[0.58931]], [[0.76050]], [[0.64211]], [[0.71075]], [[0.47585]], [[0.77930]], [[0.99254]], [[0.55993]], [[0.86000]], [[1.24359]], [[0.83426]], [[0.89956]], [[0.84253]], [[0.98685]], [[1.20358]], [[1.35928]], [[1.20319]], [[1.36037]], [[0.84950]], [[0.43650]], [[0.64862]], [[1.22611]], [[0.88242]], [[0.63714]], [[0.50251]], [[0.41774]], [[0.46366]], [[0.78424]], [[1.15986]], [[1.15598]], [[0.40345]], [[1.19755]], [[0.49441]], [[0.77996]], [[0.68293]], [[0.78935]], [[0.78749]], [[0.43699]], [[0.45394]], [[0.80742]], [[0.89213]], [[0.65852]], [[1.07401]], [[0.43563]], [[0.85192]], [[1.32537]], [[0.68172]], [[1.33716]], [[1.06674]], [[0.43443]], [[1.20201]], [[0.82796]], [[0.66246]], [[1.28549]], [[0.48161]], [[0.93931]], [[0.78098]], [[1.24272]], [[1.02523]], [[1.12591]], [[1.06224]], [[0.76542]], [[0.78051]], [[0.47362]], [[1.15658]], [[1.28265]], [[1.22777]], [[0.53297]], [[0.90796]], [[0.53319]], [[0.61042]], [[0.59431]], [[1.05653]], [[0.58771]], [[0.87475]], [[0.90670]], [[1.20797]], [[0.50770]], [[0.54241]], [[0.59533]], [[1.23563]], [[0.70593]], [[1.30741]], [[1.06647]], [[0.94897]], [[0.88194]], [[0.78682]], [[0.94840]], [[0.77149]], [[0.88215]], [[0.94946]], [[0.49456]], [[0.62360]], [[1.28941]], [[0.47989]], [[0.84976]], [[1.38028]], [[1.06628]], [[0.89982]], [[0.45236]], [[1.13335]], [[0.52952]], [[0.74373]], [[0.71790]], [[0.81162]], [[0.89113]], [[1.29807]], [[0.94808]], [[1.33351]], [[1.19152]], [[0.56955]], [[1.10991]], [[0.53221]], [[0.67375]], [[1.03272]], [[1.05076]], [[1.26078]], [[0.72470]], [[0.88646]], [[1.14307]], [[0.40212]], [[1.24715]], [[0.47220]], [[0.89255]], [[0.80065]], [[0.62233]], [[0.95175]], [[1.29912]], [[0.73950]], [[0.58881]], [[0.70074]], [[0.39009]], [[1.11136]], [[0.64553]], [[0.55199]], [[0.59761]], [[1.17314]], [[1.15045]], [[1.26942]], [[1.06702]], [[0.71868]], [[0.74594]], [[1.03338]], [[1.29668]], [[1.02161]], [[0.64909]], [[0.65062]], [[0.41293]], [[0.99369]], [[0.60507]], [[0.43987]], [[1.32410]], [[0.56096]], [[0.82878]], [[1.02891]], [[0.90159]], [[0.54922]], [[0.48150]], [[1.28420]], [[0.96708]], [[1.30047]], [[0.71804]], [[1.03294]]]]), compiled_model=tensor([[[[0.88192]], [[1.29058]], [[0.61176]], [[0.27486]], [[0.61440]], [[0.98729]], [[0.75783]], [[1.20637]], [[1.01407]], [[1.61252]], [[0.64223]], [[0.46874]], [[0.65948]], [[1.11704]], [[0.30218]], [[1.45773]], [[1.55877]], [[1.45264]], [[0.95594]], [[0.46544]], [[0.72622]], [[1.86954]], [[1.14434]], [[1.04610]], [[0.56861]], [[1.18676]], [[1.78025]], [[0.97395]], [[0.74816]], [[1.16608]], [[1.00034]], [[1.60880]], [[0.42557]], [[0.40768]], [[1.13623]], [[0.63551]], [[0.25862]], [[1.24491]], [[1.27274]], [[1.32622]], [[0.65241]], [[1.18046]], [[0.62349]], [[0.77814]], [[0.41774]], [[0.38829]], [[1.53661]], [[1.17333]], [[0.57946]], [[0.92951]], [[1.14119]], [[1.96257]], [[0.72751]], [[0.89614]], [[1.27455]], [[0.56660]], [[0.78641]], [[0.67861]], [[0.66354]], [[1.05938]], [[1.39045]], [[1.54342]], [[0.35067]], [[1.56817]], [[1.06529]], [[0.93052]], [[1.20318]], [[1.01252]], [[1.17755]], [[1.70733]], [[0.97493]], [[0.49159]], [[1.48454]], [[1.74541]], [[1.64293]], [[1.60190]], [[1.36793]], [[1.03883]], [[1.38564]], [[0.70314]], [[0.19269]], [[1.43202]], [[1.09285]], [[0.94216]], [[0.94511]], [[0.95181]], [[1.12923]], [[0.94503]], [[0.78224]], [[0.38110]], [[1.26875]], [[0.88560]], [[1.55259]], [[1.31949]], [[0.31048]], [[0.28016]], [[1.32628]], [[0.82549]], [[1.08980]], [[1.03500]], [[1.79037]], [[1.57080]], [[0.97694]], [[0.49884]], [[0.99977]], [[0.88884]], [[0.91316]], [[1.07342]], [[1.06860]], [[0.71275]], [[0.77150]], [[1.00553]], [[0.28860]], [[1.81093]], [[1.15383]], [[0.98005]], [[0.58488]], [[1.22971]], [[1.03929]], [[1.06292]], [[1.48501]], [[0.57722]], [[1.17390]], [[1.13408]], [[0.74619]], [[0.40691]], [[0.61403]], [[0.74485]], [[0.70542]], [[0.38695]], [[1.16420]], [[1.05229]], [[0.46906]], [[0.91721]], [[1.80581]], [[0.53866]], [[0.93068]], [[0.87865]], [[1.34463]], [[1.73010]], [[1.15966]], [[1.22490]], [[1.00936]], [[0.59407]], [[0.73830]], [[0.70851]], [[1.71065]], [[0.94919]], [[0.52318]], [[0.45191]], [[0.58503]], [[0.89830]], [[0.93954]], [[1.38824]], [[1.09474]], [[0.70775]], [[0.97443]], [[0.91936]], [[0.87276]], [[1.21729]], [[0.64917]], [[1.39199]], [[0.86899]], [[0.16446]], [[0.77337]], [[1.44313]], [[0.94268]], [[1.60421]], [[0.44510]], [[0.75273]], [[1.77132]], [[0.64603]], [[1.90495]], [[1.47305]], [[1.04654]], [[1.69344]], [[0.79383]], [[1.19850]], [[1.01192]], [[1.08521]], [[1.05234]], [[0.66744]], [[1.50306]], [[1.40793]], [[1.26372]], [[1.53195]], [[0.84423]], [[0.40350]], [[0.91041]], [[1.27544]], [[1.67758]], [[1.09515]], [[0.97395]], [[1.21009]], [[0.35702]], [[0.25996]], [[0.69199]], [[1.26407]], [[0.40045]], [[0.85631]], [[0.91327]], [[0.88616]], [[0.41254]], [[0.39390]], [[0.73326]], [[1.84605]], [[1.12373]], [[1.67503]], [[1.61447]], [[1.10312]], [[1.09921]], [[0.62348]], [[1.34124]], [[1.14571]], [[1.11948]], [[0.75691]], [[0.73607]], [[0.79953]], [[1.70212]], [[0.54701]], [[1.41254]], [[1.40781]], [[1.47501]], [[1.11364]], [[0.17844]], [[1.43741]], [[0.19574]], [[0.43238]], [[1.21427]], [[0.61731]], [[0.67847]], [[1.28188]], [[1.47436]], [[1.94136]], [[1.10641]], [[0.40624]], [[0.92233]], [[0.41075]], [[0.31098]], [[0.91906]], [[1.27911]], [[1.22538]], [[0.35721]], [[0.93734]], [[1.71435]], [[0.62628]], [[1.52945]], [[0.20239]], [[0.62125]], [[0.58460]], [[0.59066]], [[1.00823]], [[1.19912]], [[1.02485]], [[1.02231]], [[0.68938]], [[0.04415]], [[1.02250]], [[1.19202]], [[0.25916]], [[0.53259]], [[1.50075]], [[1.14920]], [[1.85492]], [[1.02928]], [[1.19053]], [[1.13002]], [[0.69295]], [[1.89246]], [[0.82231]], [[1.06854]], [[0.44958]], [[0.34253]], [[1.31180]], [[0.62969]], [[0.21789]], [[1.78412]], [[0.82229]], [[0.95223]], [[1.55627]], [[1.21033]], [[1.11946]], [[0.59598]], [[0.97222]], [[1.37259]], [[1.01649]], [[1.01313]], [[1.13991]]]])

test_add.test_module[Add1-[((1, 768, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.61264]], [[0.88460]], [[0.20486]], [[0.24841]], [[0.42381]], [[0.75046]], [[0.60648]], [[1.01283]], [[0.57201]], [[0.74869]], [[0.46528]], [[0.51810]], [[0.13871]], [[0.28524]], [[0.41027]], [[0.63490]], [[0.81405]], [[0.91639]], [[0.27741]], [[0.39865]], [[0.79799]], [[1.03158]], [[0.51348]], [[0.99054]], [[0.53579]], [[0.66929]], [[1.06912]], [[0.15255]], [[0.30161]], [[0.48980]], [[0.42148]], [[1.04838]], [[0.29229]], [[0.38622]], [[0.26706]], [[0.14810]], [[0.32451]], [[1.04618]], [[0.83949]], [[0.85872]], [[0.64268]], [[0.36004]], [[0.70098]], [[0.14954]], [[0.25510]], [[0.35862]], [[0.93185]], [[0.90954]], [[0.39464]], [[0.59834]], [[0.93616]], [[1.11345]], [[0.81482]], [[0.68393]], [[0.95163]], [[0.32198]], [[0.70955]], [[0.22873]], [[0.26984]], [[0.35809]], [[0.84262]], [[0.81746]], [[0.32021]], [[0.76744]], [[0.89087]], [[0.55327]], [[0.63547]], [[0.73224]], [[0.92657]], [[1.09648]], [[0.23107]], [[0.43315]], [[0.81289]], [[1.03066]], [[1.05149]], [[1.05756]], [[0.71589]], [[0.18159]], [[0.66238]], [[0.30358]], [[0.15041]], [[1.06063]], [[0.99656]], [[0.11762]], [[0.70997]], [[0.53215]], [[0.53410]], [[0.38750]], [[0.80866]], [[0.32023]], [[0.79968]], [[0.86924]], [[0.97432]], [[0.80334]], [[0.12152]], [[0.29203]], [[0.86604]], [[0.72103]], [[0.22634]], [[0.32847]], [[1.08676]], [[0.95329]], [[0.39837]], [[0.49054]], [[0.14008]], [[0.60740]], [[0.23985]], [[0.23070]], [[0.58883]], [[0.69146]], [[0.41162]], [[0.91307]], [[0.31211]], [[1.07007]], [[0.95903]], [[0.19474]], [[0.49194]], [[0.63894]], [[0.68933]], [[0.73497]], [[0.81260]], [[0.64633]], [[0.37242]], [[0.85298]], [[0.13676]], [[0.32003]], [[0.49122]], [[0.37283]], [[0.44147]], [[0.20657]], [[0.51003]], [[0.72326]], [[0.29065]], [[0.59072]], [[0.97431]], [[0.56498]], [[0.63028]], [[0.57325]], [[0.71757]], [[0.93430]], [[1.09001]], [[0.93391]], [[1.09109]], [[0.58022]], [[0.16722]], [[0.37934]], [[0.95684]], [[0.61314]], [[0.36786]], [[0.23323]], [[0.14846]], [[0.19438]], [[0.51496]], [[0.89059]], [[0.88670]], [[0.13417]], [[0.92827]], [[0.22513]], [[0.51068]], [[0.41365]], [[0.52008]], [[0.51821]], [[0.16771]], [[0.18466]], [[0.53814]], [[0.62285]], [[0.38925]], [[0.80473]], [[0.16635]], [[0.58264]], [[1.05609]], [[0.41244]], [[1.06788]], [[0.79746]], [[0.16515]], [[0.93273]], [[0.55869]], [[0.39318]], [[1.01621]], [[0.21234]], [[0.67004]], [[0.51170]], [[0.97344]], [[0.75595]], [[0.85664]], [[0.79296]], [[0.49615]], [[0.51123]], [[0.20434]], [[0.88730]], [[1.01337]], [[0.95850]], [[0.26369]], [[0.63868]], [[0.26392]], [[0.34114]], [[0.32503]], [[0.78726]], [[0.31843]], [[0.60547]], [[0.63742]], [[0.93869]], [[0.23842]], [[0.27313]], [[0.32605]], [[0.96635]], [[0.43665]], [[1.03813]], [[0.79719]], [[0.67970]], [[0.61266]], [[0.51754]], [[0.67912]], [[0.50221]], [[0.61287]], [[0.68018]], [[0.22528]], [[0.35432]], [[1.02013]], [[0.21061]], [[0.58048]], [[1.11100]], [[0.79700]], [[0.63054]], [[0.18308]], [[0.86407]], [[0.26024]], [[0.47445]], [[0.44862]], [[0.54234]], [[0.62185]], [[1.02879]], [[0.67880]], [[1.06423]], [[0.92224]], [[0.30028]], [[0.84063]], [[0.26293]], [[0.40447]], [[0.76344]], [[0.78148]], [[0.99150]], [[0.45542]], [[0.61718]], [[0.87379]], [[0.13284]], [[0.97787]], [[0.20292]], [[0.62327]], [[0.53137]], [[0.35305]], [[0.68247]], [[1.02984]], [[0.47022]], [[0.31954]], [[0.43146]], [[0.12081]], [[0.84208]], [[0.37625]], [[0.28271]], [[0.32833]], [[0.90386]], [[0.88117]], [[1.00014]], [[0.79774]], [[0.44940]], [[0.47666]], [[0.76410]], [[1.02740]], [[0.75233]], [[0.37981]], [[0.38134]], [[0.14365]], [[0.72441]], [[0.33579]], [[0.17059]], [[1.05482]], [[0.29168]], [[0.55950]], [[0.75963]], [[0.63231]], [[0.27994]], [[0.21222]], [[1.01492]], [[0.69780]], [[1.03119]], [[0.44876]], [[0.76366]], [[0.50204]], [[0.59415]], [[0.31186]], [[0.78548]], [[0.77447]], [[0.60607]], [[0.50393]], [[0.30817]], [[0.96216]], [[0.24417]], [[0.82122]], [[0.44826]], [[0.37515]], [[0.70621]], [[0.35665]], [[0.73159]], [[0.71457]], [[0.24514]], [[0.69963]], [[0.82935]], [[0.81430]], [[0.55344]], [[0.20649]], [[0.53931]], [[0.79004]], [[0.43394]], [[0.80617]], [[0.94936]], [[0.35531]], [[0.62131]], [[0.82313]], [[0.65557]], [[0.65815]], [[0.67882]], [[0.22331]], [[0.65568]], [[0.96262]], [[1.06694]], [[0.91026]], [[0.68338]], [[0.84989]], [[0.37314]], [[0.20204]], [[0.18642]], [[1.11519]], [[0.93378]], [[0.27077]], [[0.81200]], [[0.99397]], [[1.11619]], [[1.05358]], [[1.00376]], [[0.50175]], [[0.44091]], [[1.02691]], [[0.89657]], [[0.31549]], [[1.06589]], [[0.85796]], [[0.88895]], [[0.30299]], [[0.75983]], [[0.44110]], [[1.00706]], [[0.52641]], [[0.81104]], [[0.70519]], [[0.82912]], [[0.44647]], [[0.86017]], [[0.26715]], [[0.72931]], [[0.27809]], [[0.12311]], [[0.21485]], [[1.01112]], [[0.88690]], [[1.08549]], [[1.01694]], [[0.16986]], [[0.27517]], [[0.53559]], [[0.29167]], [[0.96358]], [[0.23838]], [[0.37243]], [[0.13334]], [[0.33251]], [[1.02762]], [[1.02576]], [[0.97430]], [[1.00245]], [[1.06098]], [[0.48835]], [[0.83638]], [[1.06185]], [[0.78179]], [[1.11622]], [[0.87571]], [[0.92721]], [[0.44139]], [[0.85630]], [[0.67384]], [[0.49697]], [[0.33453]], [[0.33582]], [[0.23164]], [[0.95205]], [[0.97185]], [[0.55948]], [[0.32704]], [[1.00284]], [[0.93613]], [[0.65355]], [[0.38032]], [[1.07591]], [[0.82085]], [[0.23681]], [[1.09492]], [[0.99607]], [[0.43414]], [[0.89746]], [[0.33229]], [[0.53803]], [[1.04093]], [[0.63704]], [[0.26277]], [[0.44927]], [[0.48066]], [[0.51992]], [[0.66423]], [[1.07880]], [[0.64315]], [[0.30766]], [[0.64201]], [[0.85613]], [[0.86440]], [[0.15941]], [[0.52691]], [[0.24481]], [[0.40304]], [[0.79653]], [[0.26132]], [[0.80225]], [[1.04077]], [[0.64918]], [[0.28314]], [[0.43724]], [[0.72556]], [[0.23522]], [[0.86479]], [[0.16245]], [[0.13574]], [[0.13055]], [[0.51495]], [[0.95260]], [[0.14314]], [[1.03198]], [[0.41637]], [[0.76282]], [[0.63918]], [[0.16552]], [[1.03105]], [[0.88560]], [[1.11338]], [[0.86899]], [[0.28635]], [[1.03367]], [[0.64325]], [[0.85349]], [[0.21547]], [[0.47257]], [[0.12544]], [[0.42164]], [[0.72425]], [[0.22380]], [[0.77576]], [[0.88479]], [[0.68604]], [[0.28184]], [[0.22872]], [[0.46212]], [[0.83586]], [[1.10958]], [[0.90390]], [[0.56008]], [[0.79169]], [[0.12585]], [[0.18933]], [[0.84969]], [[0.33318]], [[0.85693]], [[0.26342]], [[0.36873]], [[0.20454]], [[0.87730]], [[0.56544]], [[1.00118]], [[0.92582]], [[0.89305]], [[0.63246]], [[0.46179]], [[0.50767]], [[0.68283]], [[0.86424]], [[0.26609]], [[1.03602]], [[0.56202]], [[0.19741]], [[0.34585]], [[1.05879]], [[1.07365]], [[0.15324]], [[0.96903]], [[0.86696]], [[0.91234]], [[1.03965]], [[0.34691]], [[0.77427]], [[0.82100]], [[0.46864]], [[0.78371]], [[0.47253]], [[0.92551]], [[0.47766]], [[0.42998]], [[0.74226]], [[0.79373]], [[0.37210]], [[0.66058]], [[0.90615]], [[0.56663]], [[0.76855]], [[0.49579]], [[0.79163]], [[0.25419]], [[0.32237]], [[0.36259]], [[1.07589]], [[0.48184]], [[0.61502]], [[0.37413]], [[1.11553]], [[1.10472]], [[0.23929]], [[0.21105]], [[0.23739]], [[0.61397]], [[0.48893]], [[0.28911]], [[0.43705]], [[0.71085]], [[0.35514]], [[0.72717]], [[0.50172]], [[0.37410]], [[0.68508]], [[1.02751]], [[0.27834]], [[0.63960]], [[0.43200]], [[1.10704]], [[0.14200]], [[0.13705]], [[1.10908]], [[0.30004]], [[0.71224]], [[0.57323]], [[0.51105]], [[0.50471]], [[0.93410]], [[0.64028]], [[0.12958]], [[0.32120]], [[0.44593]], [[0.86799]], [[0.29282]], [[1.08784]], [[0.50502]], [[0.52660]], [[1.00816]], [[0.86769]], [[1.04044]], [[0.90560]], [[0.46471]], [[0.28465]], [[0.57918]], [[1.03020]], [[0.44857]], [[0.15270]], [[0.82135]], [[1.10312]], [[0.47404]], [[0.20237]], [[0.16285]], [[0.74168]], [[0.57852]], [[0.36389]], [[0.71745]], [[0.80626]], [[1.01404]], [[1.00457]], [[0.54154]], [[0.17550]], [[0.16457]], [[1.08323]], [[0.83741]], [[0.83434]], [[0.18377]], [[1.07939]], [[1.09005]], [[1.06781]], [[0.19459]], [[0.42772]], [[0.27250]], [[1.08987]], [[0.40154]], [[0.38811]], [[0.87834]], [[0.38509]], [[0.37012]], [[0.57265]], [[0.56832]], [[0.22689]], [[1.03322]], [[0.39582]], [[0.79374]], [[1.05127]], [[0.86854]], [[0.68717]], [[1.04181]], [[0.68360]], [[0.38505]], [[1.08938]], [[0.73472]], [[0.12854]], [[0.47404]], [[0.27580]], [[1.05476]], [[0.53381]], [[0.16062]], [[0.58495]], [[0.93039]], [[0.74628]], [[0.77448]], [[0.66282]], [[0.80279]], [[0.49456]], [[0.41748]], [[0.14903]], [[0.23971]], [[0.83308]], [[0.32033]], [[0.68812]], [[0.77598]], [[0.65178]], [[0.29221]], [[1.09455]], [[0.32563]], [[1.02763]], [[0.21863]], [[0.49611]], [[0.88837]], [[0.41209]], [[1.03640]], [[0.27230]], [[0.19650]], [[0.39092]], [[0.69721]], [[1.07676]], [[0.37768]], [[0.79520]], [[0.49103]], [[0.50795]], [[0.98403]], [[0.22889]], [[0.66949]], [[1.08659]], [[0.54768]], [[1.00460]], [[0.46240]], [[1.01886]], [[0.13272]], [[0.54432]], [[0.52856]], [[0.77843]], [[0.81255]], [[1.00029]], [[0.54189]], [[0.59656]], [[0.95879]], [[0.48110]], [[1.05470]], [[0.28348]], [[0.56228]], [[0.58953]], [[0.83949]], [[0.95822]], [[0.53713]], [[0.20214]], [[0.86413]], [[0.76595]], [[0.81723]], [[0.30795]], [[0.93818]], [[1.08996]], [[0.65977]], [[0.14936]], [[0.96738]], [[0.24566]], [[0.73132]], [[0.68904]], [[0.38236]], [[0.79048]], [[0.16917]], [[0.73026]], [[0.29941]], [[0.56232]], [[0.68070]], [[1.04235]], [[0.37782]], [[0.93670]], [[0.55287]], [[0.37892]], [[0.18098]], [[0.15762]], [[1.10469]], [[0.49168]], [[0.64136]], [[0.75194]], [[0.95625]], [[1.04312]], [[1.02189]], [[0.24594]], [[0.53627]], [[0.32051]], [[0.33071]], [[0.73499]], [[1.08571]], [[0.21583]], [[0.91898]], [[0.35715]], [[0.51900]], [[1.01327]], [[0.50330]], [[0.66191]], [[0.26687]], [[1.04200]], [[0.55175]], [[0.25069]], [[0.76279]], [[0.26087]], [[0.21963]], [[0.64684]], [[1.01280]], [[0.47489]], [[0.85172]], [[1.04604]], [[0.94801]], [[0.35410]], [[0.56157]], [[0.45904]], [[0.21432]], [[0.61659]], [[0.99260]], [[1.03764]], [[0.66303]], [[0.72994]], [[0.39992]], [[0.99380]], [[0.40833]], [[0.26906]], [[0.69338]], [[0.91608]], [[0.16559]], [[1.06837]], [[0.79625]], [[0.26608]], [[0.50868]], [[1.05017]]]]), compiled_model=tensor([[[[0.61264]], [[0.90768]], [[0.47820]], [[1.11811]], [[1.03232]], [[1.42912]], [[0.68297]], [[0.92216]], [[1.05087]], [[1.45760]], [[1.00288]], [[0.66979]], [[0.50556]], [[0.97655]], [[0.95676]], [[1.30819]], [[1.08501]], [[1.72395]], [[0.20358]], [[0.35157]], [[0.96247]], [[1.36579]], [[0.46010]], [[1.48345]], [[1.36162]], [[1.48593]], [[1.50423]], [[0.82935]], [[0.88592]], [[0.97705]], [[0.98924]], [[1.81379]], [[0.23743]], [[0.60046]], [[0.94798]], [[0.76358]], [[0.59175]], [[1.14725]], [[0.78653]], [[1.23178]], [[1.17984]], [[0.44413]], [[1.22476]], [[0.09055]], [[1.02267]], [[1.18682]], [[1.38953]], [[0.98847]], [[1.08847]], [[1.23699]], [[1.31973]], [[1.72934]], [[0.70988]], [[1.07270]], [[1.48018]], [[0.24182]], [[1.29305]], [[0.12342]], [[0.20454]], [[1.15502]], [[1.14813]], [[0.86008]], [[0.31942]], [[1.02652]], [[1.67768]], [[1.41264]], [[1.20751]], [[1.59057]], [[1.79545]], [[1.61380]], [[0.82870]], [[0.78324]], [[1.56110]], [[1.15399]], [[1.48451]], [[0.94436]], [[1.40382]], [[0.24435]], [[1.09068]], [[0.73206]], [[0.93379]], [[1.46326]], [[1.48464]], [[0.98172]], [[0.82204]], [[0.55166]], [[0.45932]], [[1.15719]], [[1.12788]], [[0.85029]], [[1.14522]], [[1.21035]], [[1.40992]], [[1.33337]], [[0.56593]], [[0.95785]], [[1.20759]], [[0.81403]], [[0.43642]], [[0.43256]], [[1.32535]], [[1.50059]], [[1.09855]], [[1.27980]], [[0.13988]], [[1.09730]], [[0.51499]], [[0.44051]], [[0.70214]], [[0.77486]], [[0.38258]], [[1.35291]], [[0.58071]], [[1.38274]], [[1.03865]], [[1.04006]], [[1.14561]], [[0.91590]], [[1.12922]], [[1.15697]], [[0.82272]], [[0.96250]], [[0.45287]], [[1.61200]], [[0.33695]], [[0.87975]], [[0.89901]], [[0.90692]], [[0.61727]], [[0.11063]], [[1.17745]], [[1.32012]], [[0.20142]], [[1.07630]], [[1.72422]], [[1.30947]], [[0.80558]], [[0.62678]], [[0.85944]], [[1.15479]], [[1.36726]], [[1.25288]], [[1.37094]], [[0.85240]], [[0.33597]], [[0.96103]], [[1.62630]], [[1.19795]], [[1.09359]], [[0.82291]], [[0.14000]], [[0.17097]], [[1.31826]], [[0.94115]], [[1.14808]], [[0.13079]], [[1.64009]], [[0.43801]], [[0.85007]], [[0.62240]], [[1.36915]], [[0.99487]], [[0.94801]], [[0.50956]], [[0.60352]], [[1.38363]], [[0.58537]], [[1.16153]], [[0.49583]], [[1.38167]], [[1.28271]], [[0.53303]], [[1.73761]], [[0.68850]], [[0.63052]], [[1.09349]], [[0.94512]], [[0.91192]], [[1.44588]], [[0.15762]], [[1.13325]], [[0.42268]], [[1.66802]], [[1.24459]], [[1.40153]], [[1.37852]], [[0.85739]], [[0.67432]], [[1.03553]], [[1.45565]], [[1.59625]], [[1.28931]], [[0.71867]], [[0.82407]], [[0.66409]], [[0.26406]], [[0.55563]], [[1.26399]], [[1.04775]], [[1.35083]], [[1.20261]], [[1.18862]], [[0.69635]], [[0.99744]], [[0.51407]], [[1.01492]], [[1.02232]], [[1.09119]], [[1.39832]], [[0.84443]], [[0.88435]], [[1.16696]], [[0.66959]], [[1.37380]], [[1.05264]], [[1.27520]], [[0.34981]], [[1.04969]], [[1.20259]], [[0.31600]], [[1.13490]], [[1.43833]], [[1.64533]], [[1.32759]], [[0.26137]], [[1.52991]], [[0.26026]], [[0.64029]], [[0.45582]], [[0.93667]], [[1.32694]], [[1.85763]], [[0.63027]], [[1.41438]], [[1.62651]], [[0.74954]], [[0.78850]], [[1.03884]], [[0.98652]], [[1.35507]], [[1.59145]], [[1.20051]], [[0.89236]], [[1.29203]], [[1.67750]], [[0.88233]], [[1.48783]], [[0.19830]], [[1.45439]], [[0.77483]], [[0.24641]], [[1.30772]], [[1.44684]], [[1.07466]], [[0.63122]], [[0.77905]], [[0.91958]], [[1.19310]], [[1.25663]], [[0.63636]], [[0.82108]], [[1.39295]], [[1.41418]], [[1.66970]], [[1.50493]], [[1.12317]], [[1.06350]], [[1.48016]], [[1.25902]], [[0.77962]], [[0.48748]], [[0.87630]], [[0.48163]], [[1.30064]], [[0.73183]], [[0.83922]], [[1.49241]], [[0.67760]], [[0.64529]], [[1.06338]], [[0.85330]], [[1.15515]], [[0.20703]], [[1.61407]], [[0.77535]], [[1.02174]], [[1.31758]], [[0.94043]], [[0.90046]], [[0.70430]], [[1.09430]], [[1.38714]], [[1.28286]], [[0.56480]], [[0.85941]], [[0.62247]], [[1.30422]], [[0.26034]], [[1.42718]], [[0.36111]], [[0.91361]], [[1.19667]], [[0.53118]], [[0.93164]], [[0.80531]], [[0.96176]], [[1.38618]], [[0.99799]], [[1.34598]], [[0.98409]], [[0.35148]], [[0.63409]], [[0.80492]], [[1.04012]], [[1.60242]], [[1.37507]], [[0.76705]], [[0.56923]], [[1.18535]], [[0.97635]], [[1.03883]], [[0.74233]], [[0.61029]], [[0.94022]], [[1.69054]], [[1.78271]], [[1.61026]], [[1.06862]], [[1.11486]], [[0.41895]], [[0.40519]], [[0.70234]], [[1.79679]], [[1.49146]], [[1.14597]], [[0.84649]], [[1.63207]], [[1.11653]], [[0.95420]], [[1.80868]], [[0.94583]], [[1.13457]], [[1.90037]], [[1.76041]], [[0.87156]], [[1.10918]], [[1.30688]], [[0.80652]], [[0.97305]], [[0.71394]], [[0.75442]], [[1.88400]], [[1.25641]], [[1.32098]], [[1.43812]], [[1.25640]], [[0.51877]], [[1.11954]], [[0.71956]], [[0.98360]], [[1.05620]], [[0.68321]], [[0.65873]], [[1.40483]], [[0.96053]], [[1.86992]], [[1.23898]], [[0.70649]], [[1.15332]], [[1.16684]], [[0.25410]], [[0.93396]], [[0.51536]], [[0.92959]], [[0.76411]], [[1.09184]], [[1.82654]], [[1.90243]], [[1.42265]], [[1.66512]], [[1.15306]], [[0.96465]], [[1.56998]], [[1.64882]], [[1.07241]], [[1.97409]], [[1.53385]], [[0.83883]], [[1.25748]], [[1.48524]], [[1.29395]], [[1.08686]], [[0.86678]], [[1.03022]], [[0.34486]], [[0.85515]], [[1.21416]], [[0.53714]], [[0.92245]], [[1.35778]], [[1.35226]], [[1.45093]], [[1.00805]], [[1.30166]], [[1.46745]], [[0.85140]], [[1.40196]], [[1.42167]], [[1.17854]], [[1.22961]], [[0.45394]], [[1.23795]], [[1.79175]], [[0.95200]], [[0.38361]], [[0.48173]], [[0.83657]], [[0.54746]], [[0.97541]], [[1.38147]], [[1.13564]], [[0.22342]], [[0.56732]], [[1.47982]], [[1.36194]], [[0.52207]], [[0.74007]], [[1.12193]], [[0.73620]], [[0.91977]], [[0.16210]], [[1.40845]], [[1.04572]], [[0.75369]], [[0.85083]], [[0.42699]], [[0.87159]], [[0.25709]], [[1.13388]], [[0.83150]], [[0.44213]], [[0.68248]], [[0.56741]], [[1.20981]], [[0.85758]], [[1.14112]], [[0.78309]], [[1.07919]], [[0.57858]], [[0.33336]], [[1.26752]], [[0.86470]], [[1.39823]], [[1.49311]], [[1.07835]], [[1.72984]], [[0.71614]], [[1.72911]], [[0.12744]], [[0.67946]], [[0.80870]], [[0.82147]], [[1.12625]], [[0.97790]], [[1.06339]], [[1.02119]], [[0.99373]], [[0.44150]], [[0.38167]], [[1.19214]], [[0.74489]], [[1.39312]], [[1.10417]], [[1.28231]], [[0.79543]], [[0.23039]], [[0.18533]], [[1.70559]], [[1.19027]], [[1.46576]], [[0.26090]], [[0.83547]], [[0.72353]], [[1.74273]], [[1.09050]], [[1.32561]], [[1.72293]], [[1.71067]], [[1.09423]], [[0.55916]], [[1.36298]], [[0.81542]], [[1.22848]], [[1.05919]], [[1.88153]], [[0.46930]], [[0.81003]], [[0.47614]], [[1.90675]], [[1.37127]], [[0.82025]], [[1.35761]], [[0.89305]], [[1.78038]], [[1.34526]], [[0.46683]], [[1.05864]], [[0.85335]], [[0.44434]], [[0.84149]], [[0.76295]], [[1.21541]], [[0.47470]], [[0.55465]], [[1.15989]], [[0.79156]], [[1.13254]], [[0.87124]], [[1.67782]], [[0.83477]], [[1.10345]], [[1.02455]], [[0.71729]], [[0.19948]], [[0.48804]], [[0.72362]], [[0.99972]], [[1.18776]], [[1.09351]], [[1.25094]], [[1.12284]], [[1.04029]], [[0.40179]], [[0.83481]], [[1.10587]], [[1.13629]], [[0.64534]], [[0.27636]], [[1.18665]], [[1.55469]], [[0.25159]], [[1.35261]], [[1.04757]], [[0.65663]], [[0.77463]], [[0.91511]], [[0.93472]], [[1.05778]], [[0.78738]], [[1.23904]], [[0.03832]], [[0.10859]], [[1.32373]], [[0.87049]], [[0.67027]], [[1.27797]], [[0.72839]], [[1.28844]], [[1.54553]], [[0.70898]], [[0.77765]], [[0.87741]], [[1.23651]], [[1.06819]], [[0.43589]], [[1.72558]], [[1.30984]], [[0.61748]], [[1.73179]], [[1.61620]], [[1.64030]], [[0.80515]], [[0.53558]], [[0.49952]], [[1.29997]], [[1.37569]], [[0.48894]], [[0.42449]], [[1.30485]], [[1.35701]], [[0.66363]], [[0.61665]], [[0.20813]], [[0.81868]], [[1.32599]], [[0.40975]], [[1.30608]], [[0.81809]], [[1.42618]], [[0.96047]], [[0.77270]], [[0.09668]], [[0.57693]], [[1.95569]], [[0.89203]], [[0.83645]], [[0.82861]], [[1.33625]], [[1.92906]], [[1.08059]], [[0.32275]], [[0.93608]], [[0.91509]], [[1.13919]], [[0.72250]], [[0.38273]], [[1.03905]], [[0.80260]], [[1.12526]], [[1.00379]], [[1.28338]], [[0.21436]], [[1.22222]], [[0.79954]], [[0.77793]], [[1.14909]], [[1.11413]], [[0.88674]], [[1.40005]], [[0.94070]], [[0.85777]], [[1.92142]], [[1.12673]], [[0.29179]], [[0.90005]], [[0.59851]], [[1.66044]], [[1.22302]], [[0.84137]], [[1.27787]], [[1.30019]], [[0.87786]], [[0.78250]], [[1.40209]], [[1.09502]], [[1.01771]], [[1.02386]], [[0.38537]], [[0.98658]], [[0.98620]], [[0.22392]], [[1.33797]], [[1.14182]], [[0.97222]], [[0.69261]], [[1.29812]], [[1.03113]], [[1.24824]], [[0.22318]], [[0.42846]], [[1.41565]], [[1.24125]], [[1.68891]], [[0.53313]], [[1.03560]], [[0.29758]], [[0.84015]], [[1.37832]], [[0.78936]], [[1.12583]], [[1.23882]], [[1.15076]], [[1.52556]], [[0.71162]], [[0.65406]], [[1.43314]], [[1.02339]], [[1.62696]], [[1.11740]], [[1.12432]], [[0.46023]], [[0.71711]], [[0.70836]], [[1.54339]], [[0.96747]], [[1.39834]], [[0.85628]], [[0.70619]], [[0.90022]], [[1.26162]], [[1.57278]], [[1.15617]], [[1.25572]], [[1.20808]], [[1.34464]], [[1.45512]], [[0.87600]], [[0.38454]], [[1.38790]], [[1.28923]], [[1.66004]], [[0.91103]], [[1.48447]], [[1.96717]], [[0.85560]], [[0.56380]], [[1.42815]], [[0.62374]], [[1.02285]], [[1.18837]], [[0.76604]], [[1.57631]], [[0.91024]], [[0.83829]], [[0.72655]], [[1.00889]], [[0.63569]], [[0.94146]], [[0.83080]], [[1.28948]], [[0.61324]], [[1.08242]], [[0.80060]], [[0.53654]], [[1.17220]], [[1.23105]], [[1.20199]], [[0.78328]], [[1.57660]], [[1.51554]], [[0.95470]], [[0.99797]], [[0.83148]], [[0.37502]], [[0.65172]], [[1.29880]], [[1.11607]], [[0.30067]], [[1.07682]], [[1.04459]], [[0.98727]], [[1.64849]], [[0.84837]], [[1.52134]], [[0.90931]], [[1.90682]], [[0.63801]], [[0.88533]], [[0.71899]], [[0.18977]], [[0.14151]], [[0.55502]], [[1.00019]], [[0.49025]], [[1.64044]], [[1.31674]], [[1.61617]], [[0.63089]], [[0.86905]], [[1.28442]], [[0.96794]], [[1.34927]], [[1.33694]], [[1.05681]], [[0.70457]], [[1.11356]], [[0.43325]], [[1.25674]], [[0.37515]], [[0.66871]], [[1.53413]], [[0.95432]], [[1.01637]], [[1.91972]], [[1.16153]], [[0.26745]], [[0.93331]], [[1.13279]]]])

test_advindex.test_module[Advindex5-[((2401,), torch.int64)]]: forge/test/models_ops/test_advindex.py#L233

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[0.26386, 0.77911, 0.51998, ..., 0.97305, 0.45384, 0.62358], [0.01385, 0.84939, 0.44188, ..., 0.82839, 0.97071, 0.45740], [0.92949, 0.70951, 0.21505, ..., 0.42064, 0.70811, 0.36257], ..., [0.33341, 0.85760, 0.61204, ..., 0.63089, 0.45813, 0.51901], [0.49952, 0.23300, 0.51194, ..., 0.15630, 0.24578, 0.72055], [0.37129, 0.83218, 0.25383, ..., 0.09266, 0.43937, 0.77135]], grad_fn=<IndexBackward0>), compiled_model=tensor([[0.26367, 0.77734, 0.51953, ..., 0.97266, 0.45312, 0.62500], [0.50391, 0.19922, 0.78906, ..., 0.00378, 0.65625, 0.73828], [0.01385, 0.84766, 0.44141, ..., 0.82812, 0.97266, 0.45703], ..., [0.80469, 0.30859, 0.06641, ..., 0.40820, 0.14551, 0.13086], [0.50391, 0.19922, 0.78906, ..., 0.00378, 0.65625, 0.73828], [0.77344, 0.21387, 0.72656, ..., 0.76172, 0.58203, 0.37500]])

test_broadcast.test_module[Broadcast1-[((1, 1, 1, 128), torch.bool)]]: forge/test/models_ops/test_broadcast.py#L172

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((2, 13, 1), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[ 44.], [239.], [933.], [760.], [963.], [379.], [427.], [503.], [497.], [683.], [101.], [866.], [756.]], [[399.], [878.], [376.], [ 56.], [868.], [794.], [ 33.], [126.], [119.], [391.], [254.], [824.], [841.]]]), compiled_model=tensor([[[ 44.], [ 0.], [239.], [ 0.], [933.], [ 0.], [760.], [ 0.], [963.], [ 0.], [379.], [ 0.], [427.]], [[ 0.], [503.], [ 0.], [497.], [ 0.], [683.], [ 0.], [101.], [ 0.], [866.], [ 0.], [756.], [ 0.]]])

test_cast.test_module[Cast0-[((2, 1, 1, 13), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((1, 1, 1, 128), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 44., 239., 933., 760., 963., 379., 427., 503., 497., 683., 101., 866., 756., 399., 878., 376., 56., 868., 794., 33., 126., 119., 391., 254., 824., 841., 269., 969., 549., 480., 481., 12., 363., 360., 995., 85., 822., 999., 711., 288., 778., 43., 896., 489., 771., 757., 283., 395., 782., 671., 840., 969., 673., 441., 111., 280., 503., 706., 476., 327., 699., 326., 363., 474., 775., 700., 218., 632., 568., 12., 277., 645., 110., 80., 348., 521., 119., 916., 881., 490., 682., 519., 944., 633., 169., 863., 409., 233., 219., 78., 735., 83., 222., 858., 235., 416., 446., 835., 877., 212., 505., 946., 156., 315., 284., 750., 108., 71., 847., 108., 75., 84., 884., 148., 144., 534., 519., 260., 907., 514., 175., 663., 613., 157., 133., 120., 249., 389.]]]]), compiled_model=tensor([[[[ 44., 0., 239., 0., 933., 0., 760., 0., 963., 0., 379., 0., 427., 0., 503., 0., 497., 0., 683., 0., 101., 0., 866., 0., 756., 0., 399., 0., 878., 0., 376., 0., 56., 0., 868., 0., 794., 0., 33., 0., 126., 0., 119., 0., 391., 0., 254., 0., 824., 0., 841., 0., 269., 0., 969., 0., 549., 0., 480., 0., 481., 0., 12., 0., 363., 0., 360., 0., 995., 0., 85., 0., 822., 0., 999., 0., 711., 0., 288., 0., 778., 0., 43., 0., 896., 0., 489., 0., 771., 0., 757., 0., 283., 0., 395., 0., 782., 0., 671., 0., 840., 0., 969., 0., 673., 0., 441., 0., 111., 0., 280., 0., 503., 0., 706., 0., 476., 0., 327., 0., 699., 0., 326., 0., 363., 0., 474., 0.]]]])

test_cast.test_module[Cast2-[((1, 1, 256, 256), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast1-[((1, 1, 32, 32), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((1, 1, 32, 32), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast1-[((1, 16, 192, 640), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_conv2d.test_module[Conv2D0-[((1, 80, 3000, 1), torch.float32), ((1024, 80, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_FATAL @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/ttnn/cpp/ttnn/operations/conv/conv2d/device/conv2d_op_sharded_program_factory.cpp:1180: act_block_w_datums == round_up(conv_act_size_c * filter_w, TILE_WIDTH) info: Error backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xbd288) [0x7f0f3528e288] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d41multi_core_optimized_conv_sharded_v2_implERN2tt8tt_metal2v07ProgramERKNS4_6TensorESA_RKNS4_5ShapeESt8optionalIS9_ERKSF_RKNS0_14sliding_window19SlidingWindowConfigEjjbbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigEbbRS8_St7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x90c4) [0x7f0f234f97a4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d40multi_core_optimized_conv_sharded_v2_newERKN2tt8tt_metal6TensorES7_RKSt8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigENS4_8DataTypeESt5arrayIjLm4EEbS8_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERS5_bbbb+0x492) [0x7f0f234ecfc2] --- ttnn::operations::conv::conv2d::OptimizedConvNew::create_program(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) const --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x449829) [0x7f0f23553829] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c0fda) [0x7f0f256cafda] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c13e4) [0x7f0f256cb3e4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x2ab) [0x7f0f256ca72b] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7f0f23552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7f0f234ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7f0f234d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30]

test_conv2d.test_module[Conv2D1-[((1, 512, 3000, 1), torch.float32), ((512, 512, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_FATAL @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/ttnn/cpp/ttnn/operations/conv/conv2d/device/conv2d_op_sharded_program_factory.cpp:1180: act_block_w_datums == round_up(conv_act_size_c * filter_w, TILE_WIDTH) info: Error backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xbd288) [0x7f0f3528e288] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d41multi_core_optimized_conv_sharded_v2_implERN2tt8tt_metal2v07ProgramERKNS4_6TensorESA_RKNS4_5ShapeESt8optionalIS9_ERKSF_RKNS0_14sliding_window19SlidingWindowConfigEjjbbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigEbbRS8_St7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x90c4) [0x7f0f234f97a4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d40multi_core_optimized_conv_sharded_v2_newERKN2tt8tt_metal6TensorES7_RKSt8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigENS4_8DataTypeESt5arrayIjLm4EEbS8_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERS5_bbbb+0x492) [0x7f0f234ecfc2] --- ttnn::operations::conv::conv2d::OptimizedConvNew::create_program(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) const --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x449829) [0x7f0f23553829] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c0fda) [0x7f0f256cafda] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c13e4) [0x7f0f256cb3e4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x2ab) [0x7f0f256ca72b] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7f0f23552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7f0f234ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7f0f234d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30]

test_conv2d.test_module[Conv2D0-[((1, 80, 3000, 1), torch.float32), ((768, 80, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_FATAL @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/ttnn/cpp/ttnn/operations/conv/conv2d/device/conv2d_op_sharded_program_factory.cpp:1180: act_block_w_datums == round_up(conv_act_size_c * filter_w, TILE_WIDTH) info: Error backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xbd288) [0x7f0f3528e288] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d41multi_core_optimized_conv_sharded_v2_implERN2tt8tt_metal2v07ProgramERKNS4_6TensorESA_RKNS4_5ShapeESt8optionalIS9_ERKSF_RKNS0_14sliding_window19SlidingWindowConfigEjjbbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigEbbRS8_St7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x90c4) [0x7f0f234f97a4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d40multi_core_optimized_conv_sharded_v2_newERKN2tt8tt_metal6TensorES7_RKSt8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigENS4_8DataTypeESt5arrayIjLm4EEbS8_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERS5_bbbb+0x492) [0x7f0f234ecfc2] --- ttnn::operations::conv::conv2d::OptimizedConvNew::create_program(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) const --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x449829) [0x7f0f23553829] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c0fda) [0x7f0f256cafda] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c13e4) [0x7f0f256cb3e4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x2ab) [0x7f0f256ca72b] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7f0f23552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7f0f234ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7f0f234d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30]

test_conv2d.test_module[Conv2D516-[((1, 64, 147, 147), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[144.59027, 146.84991, 148.81052, ..., 149.34062, 144.77367, 150.60271], [154.30252, 151.86911, 145.33368, ..., 149.58041, 149.99380, 151.38751], [148.81361, 149.79916, 149.23187, ..., 153.06140, 147.59715, 157.32217], ..., [148.77573, 154.60593, 147.55367, ..., 143.41946, 144.45972, 142.11687], [151.35895, 147.36212, 148.72386, ..., 147.42331, 147.54321, 149.12044], [147.16724, 147.29132, 146.92798, ..., 145.67532, 149.81578, 147.31308]], [[140.71840, 144.22404, 138.47804, ..., 143.40662, 139.74756, 148.97644], [148.40015, 145.30476, 140.86342, ..., 144.08215, 144.98936, 145.48819], [144.01108, 147.90112, 145.45279, ..., 146.03320, 149.61972, 147.22882], ..., [141.21158, 145.70171, 145.40758, ..., 139.71869, 137.57140, 140.26895], [149.58794, 147.29919, 147.51926, ..., 142.93979, 141.53607, 139.44991], [141.32965, 139.48727, 142.74486, ..., 143.92967, 138.20511, 142.24414]], [[140.02393, 144.81480, 146.07338, ..., 146.16563, 143.65277, 151.04582], [150.72763, 146.17616, 138.63666, ..., 148.85651, 147.39087, 151.46860], [143.57613, 147.92914, 146.81619, ..., 149.28568, 146.27937, 144.95795], ..., [144.22781, 147.86832, 151.58109, ..., 140.34464, 140.59332, 141.36800], [151.09006, 148.86142, 141.81094, ..., 147.40375, 144.22755, 143.91858], [145.42047, 142.12573, 145.78421, ..., 143.26306, 145.93289, 146.41022]], ..., [[134.75006, 143.82431, 139.12413, ..., 139.33653, 139.13292, 149.33629], [148.01990, 143.21996, 134.14467, ..., 144.38297, 138.90141, 142.90817], [140.71890, 141.41656, 142.23604, ..., 144.09248, 143.34113, 144.85754], ..., [140.11584, 142.46686, 142.71851, ..., 132.81720, 134.75198, 137.18594], [144.06837, 140.93259, 142.55161, ..., 142.41451, 140.08356, 143.44922], [138.69122, 137.51140, 136.30571, ..., 140.79845, 138.62935, 147.81589]], [[140.12653, 146.67975, 147.72943, ..., 145.97450, 146.72160, 151.27521], [152.92403, 149.25113, 143.11975, ..., 143.73553, 145.50894, 151.12953], [146.58890, 147.23419, 150.84544, ..., 147.50107, 149.24231, 147.60866], ..., [144.16370, 146.87987, 144.57999, ..., 136.10786, 137.82811, 142.23167], [149.49971, 147.06453, 147.99910, ..., 146.53516, 142.94461, 140.94394], [140.82230, 141.28616, 140.45822, ..., 144.42729, 143.20293, 145.17613]], [[144.17792, 145.34229, 143.05844, ..., 147.67450, 146.90335, 154.43011], [157.93703, 150.30782, 148.87552, ..., 149.08377, 148.64655, 154.19025], [146.40970, 152.52469, 153.44748, ..., 153.13741, 152.73309, 148.03586], ..., [147.60690, 154.33560, 151.91676, ..., 141.92152, 142.06946, 143.52072], [152.30106, 153.60939, 152.74535, ..., 153.06439, 146.05418, 148.66733], [150.50977, 140.08083, 147.76814, ..., 150.72646, 145.53464, 152.12955]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[142., 146., 147., ..., 149., 142., 147.], [153., 150., 143., ..., 148., 148., 150.], [147., 146., 146., ..., 149., 146., 156.], ..., [145., 151., 145., ..., 142., 142., 141.], [150., 146., 147., ..., 145., 146., 147.], [145., 144., 144., ..., 142., 146., 145.]], [[139., 143., 137., ..., 140., 136., 147.], [147., 144., 139., ..., 143., 143., 141.], [141., 146., 142., ..., 143., 148., 144.], ..., [140., 144., 142., ..., 138., 136., 139.], [148., 144., 145., ..., 142., 139., 137.], [140., 137., 141., ..., 142., 137., 140.]], [[139., 143., 144., ..., 145., 141., 148.], [149., 145., 137., ..., 146., 144., 149.], [141., 145., 144., ..., 148., 143., 142.], ..., [141., 146., 148., ..., 139., 139., 140.], [149., 145., 140., ..., 145., 142., 142.], [143., 140., 143., ..., 140., 144., 144.]], ..., [[133., 141., 138., ..., 138., 139., 149.], [146., 141., 132., ..., 142., 137., 142.], [140., 140., 139., ..., 143., 141., 142.], ..., [139., 140., 140., ..., 131., 132., 134.], [143., 140., 140., ..., 139., 139., 142.], [135., 136., 135., ..., 138., 138., 147.]], [[138., 144., 146., ..., 146., 144., 150.], [149., 147., 141., ..., 142., 143., 148.], [146., 144., 149., ..., 144., 147., 146.], ..., [143., 145., 143., ..., 134., 137., 141.], [148., 146., 147., ..., 145., 141., 139.], [139., 140., 140., ..., 143., 141., 143.]], [[141., 143., 141., ..., 146., 145., 153.], [155., 149., 147., ..., 146., 145., 151.], [145., 150., 152., ..., 150., 149., 145.], ..., [146., 152., 150., ..., 140., 138., 143.], [149., 151., 150., ..., 153., 144., 147.], [149., 138., 146., ..., 150., 144., 150.]]]])

test_conv2d.test_module[Conv2D521-[((1, 192, 71, 71), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[446.15002, 439.13843, 434.70123, ..., 444.44580, 445.53687, 421.74384], [440.64709, 445.61102, 437.21826, ..., 439.54684, 429.51569, 431.24854], [432.52234, 441.48447, 427.53278, ..., 426.85983, 426.26343, 426.50235], ..., [436.99841, 428.25195, 432.43719, ..., 431.21909, 422.97717, 431.01242], [431.65051, 432.10336, 422.46335, ..., 422.24451, 426.35168, 434.21548], [440.16769, 429.53027, 428.59070, ..., 422.33899, 422.31458, 431.98972]], [[450.57147, 431.46411, 435.42944, ..., 444.52240, 449.15131, 424.64313], [444.63956, 437.69354, 431.06497, ..., 450.28232, 435.84848, 424.65622], [434.70490, 438.06342, 428.85287, ..., 431.32715, 425.43875, 428.49158], ..., [430.41409, 425.79886, 430.91049, ..., 440.37900, 426.69360, 431.34363], [429.08627, 434.03201, 417.79025, ..., 422.23300, 418.94946, 438.05841], [436.32346, 428.46298, 426.13773, ..., 424.98279, 419.99289, 433.16391]], [[453.05389, 448.78784, 439.67361, ..., 447.69275, 453.11194, 426.37418], [451.58249, 443.28568, 439.67465, ..., 448.97153, 438.25037, 431.61395], [434.72687, 440.18561, 424.97620, ..., 438.28619, 429.35614, 427.50818], ..., [441.45547, 430.22363, 434.44781, ..., 439.50842, 424.23312, 430.31317], [437.14679, 438.39429, 423.31979, ..., 425.75058, 423.92368, 439.27600], [443.42874, 433.07672, 432.60284, ..., 426.06219, 426.82135, 429.55148]], ..., [[453.86792, 446.48083, 434.67148, ..., 455.53632, 444.94464, 433.04016], [452.13791, 444.33417, 445.64474, ..., 449.31805, 438.04330, 430.93741], [435.12991, 450.66595, 439.94775, ..., 435.24982, 433.11722, 436.87552], ..., [443.27631, 434.58066, 436.63113, ..., 439.72028, 427.73654, 435.83127], [439.51263, 443.54346, 422.86261, ..., 427.24710, 430.18915, 432.37540], [442.58578, 437.51041, 439.42407, ..., 428.26773, 429.51547, 439.93436]], [[451.08978, 439.78058, 436.02158, ..., 445.08191, 446.12991, 428.28473], [437.62164, 436.99799, 433.54633, ..., 445.35635, 434.37897, 423.73233], [429.34958, 437.17657, 427.99524, ..., 432.74738, 430.10352, 425.73798], ..., [430.33395, 418.11194, 430.62863, ..., 433.45422, 423.36597, 425.91125], [433.10297, 438.12433, 417.49661, ..., 421.33081, 416.89270, 435.63925], [441.55737, 428.57571, 430.21786, ..., 425.64386, 417.51340, 433.58398]], [[449.63065, 430.50961, 429.07474, ..., 443.68420, 440.61194, 419.71588], [439.84348, 432.28143, 431.97241, ..., 438.82651, 429.87119, 428.96109], [432.66724, 438.94461, 422.27411, ..., 427.58652, 420.17133, 429.87167], ..., [430.67252, 426.79849, 431.38947, ..., 432.43649, 415.17517, 429.22659], [428.81421, 432.25348, 409.33395, ..., 419.82913, 418.77411, 433.38159], [437.29941, 429.86853, 426.22424, ..., 427.59778, 418.45941, 431.15567]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[454., 438., 430., ..., 440., 448., 416.], [444., 442., 442., ..., 434., 426., 426.], [426., 436., 430., ..., 428., 426., 424.], ..., [428., 422., 436., ..., 438., 424., 428.], [434., 428., 420., ..., 418., 424., 438.], [440., 430., 436., ..., 410., 418., 430.]], [[448., 430., 434., ..., 442., 454., 422.], [442., 434., 428., ..., 448., 430., 416.], [432., 440., 424., ..., 426., 424., 424.], ..., [426., 426., 430., ..., 436., 422., 434.], [422., 428., 416., ..., 418., 412., 432.], [436., 422., 426., ..., 420., 422., 434.]], [[450., 438., 430., ..., 446., 456., 420.], [456., 444., 430., ..., 440., 428., 432.], [430., 436., 430., ..., 436., 424., 422.], ..., [444., 432., 432., ..., 438., 418., 432.], [438., 440., 424., ..., 426., 428., 438.], [448., 430., 430., ..., 428., 434., 422.]], ..., [[454., 448., 434., ..., 456., 446., 436.], [452., 444., 442., ..., 444., 438., 430.], [432., 454., 438., ..., 428., 428., 438.], ..., [446., 436., 436., ..., 440., 430., 430.], [438., 446., 426., ..., 428., 432., 432.], [438., 440., 428., ..., 428., 432., 444.]], [[448., 436., 438., ..., 452., 442., 432.], [444., 438., 436., ..., 448., 426., 416.], [434., 432., 424., ..., 432., 430., 424.], ..., [430., 414., 436., ..., 436., 418., 428.], [424., 442., 426., ..., 412., 414., 438.], [442., 438., 424., ..., 418., 420., 430.]], [[450., 428., 432., ..., 438., 440., 410.], [440., 426., 430., ..., 434., 428., 432.], [434., 440., 414., ..., 426., 414., 428.], ..., [432., 432., 426., ..., 432., 412., 424.], [432., 430., 404., ..., 416., 420., 436.], [438., 436., 424., ..., 420., 414., 430.]]]])

test_conv2d.test_module[Conv2D524-[((1, 384, 35, 35), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[866.46075, 863.10168, 861.19562, ..., 855.09998, 840.28754, 847.46942], [875.89087, 877.85223, 859.68353, ..., 848.92413, 850.29297, 858.76611], [868.14600, 869.01752, 869.61334, ..., 854.45068, 861.15552, 877.75085], ..., [864.57635, 878.45477, 872.53064, ..., 842.38538, 860.43549, 849.84528], [868.08459, 862.42041, 862.94531, ..., 864.35364, 877.81317, 868.60205], [871.25690, 879.82129, 878.20593, ..., 872.27557, 869.09430, 879.15839]], [[868.28503, 870.18768, 851.58771, ..., 854.53986, 840.52539, 852.42505], [869.63287, 867.20837, 859.97717, ..., 847.46625, 836.64600, 863.39429], [871.59442, 870.66644, 870.52747, ..., 853.55408, 862.31226, 875.37372], ..., [878.39917, 876.84241, 869.46362, ..., 844.94299, 851.44666, 842.90747], [860.24902, 873.59558, 859.12000, ..., 858.11224, 868.33630, 867.80450], [865.41272, 872.19031, 874.13293, ..., 859.16174, 874.26410, 881.24512]], [[888.00397, 881.47986, 883.11444, ..., 870.96521, 858.53198, 874.56598], [892.31763, 886.70569, 869.82996, ..., 855.63470, 854.08057, 871.87781], [878.88568, 890.19373, 873.01611, ..., 853.26929, 878.55811, 881.81409], ..., [887.56647, 881.78949, 885.11133, ..., 852.02386, 868.12909, 864.41650], [882.27295, 898.73224, 882.94727, ..., 876.72205, 889.67242, 877.40814], [882.83545, 900.79346, 893.50818, ..., 871.38794, 871.64282, 907.11499]], ..., [[866.05341, 873.65509, 859.70935, ..., 854.63806, 847.80206, 853.68756], [877.18817, 877.30878, 873.71021, ..., 843.32275, 838.68896, 858.47491], [866.76837, 881.39697, 876.87811, ..., 847.82458, 862.92096, 883.85498], ..., [864.82941, 882.93518, 869.43988, ..., 848.65833, 855.09229, 840.22357], [865.50879, 870.63910, 856.58160, ..., 873.13593, 876.68097, 865.38379], [875.52112, 882.48035, 875.69458, ..., 871.47211, 874.39124, 883.82489]], [[890.46729, 871.85632, 873.58636, ..., 872.91425, 864.47919, 872.66882], [895.89868, 878.62415, 876.08154, ..., 870.83527, 862.51221, 864.13544], [879.80255, 895.70880, 887.44531, ..., 866.48828, 884.22174, 888.70520], ..., [880.42419, 882.72034, 883.07544, ..., 860.81073, 871.70856, 868.97839], [879.26276, 887.08740, 882.59674, ..., 876.70190, 891.17065, 875.54169], [878.43024, 889.88916, 895.84491, ..., 883.05402, 889.52240, 898.19843]], [[870.48798, 862.56940, 857.44904, ..., 855.58667, 853.54730, 862.28064], [876.26776, 883.40302, 872.81030, ..., 854.58289, 849.06738, 859.88440], [868.92969, 883.56549, 872.22296, ..., 858.10864, 869.10345, 886.04822], ..., [875.64264, 888.05103, 872.66205, ..., 839.72534, 869.12488, 855.52820], [869.00891, 876.96631, 868.33258, ..., 863.16364, 874.45190, 865.76471], [866.99567, 881.12372, 882.60400, ..., 870.83557, 871.39697, 887.77435]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[868., 856., 872., ..., 860., 864., 872.], [884., 872., 868., ..., 864., 852., 864.], [884., 872., 880., ..., 868., 856., 872.], ..., [876., 880., 872., ..., 860., 868., 860.], [880., 860., 872., ..., 864., 892., 872.], [884., 884., 864., ..., 884., 868., 892.]], [[872., 864., 848., ..., 864., 868., 860.], [872., 860., 872., ..., 860., 856., 868.], [872., 860., 880., ..., 868., 868., 876.], ..., [880., 876., 876., ..., 864., 864., 856.], [860., 872., 880., ..., 856., 868., 876.], [852., 876., 876., ..., 872., 868., 892.]], [[880., 876., 868., ..., 864., 852., 876.], [904., 888., 884., ..., 868., 868., 860.], [872., 892., 860., ..., 868., 872., 876.], ..., [892., 884., 876., ..., 860., 864., 864.], [880., 876., 880., ..., 872., 884., 864.], [880., 888., 888., ..., 872., 872., 900.]], ..., [[856., 860., 864., ..., 864., 860., 848.], [872., 860., 864., ..., 864., 844., 852.], [872., 880., 876., ..., 844., 860., 880.], ..., [884., 880., 880., ..., 872., 864., 856.], [876., 864., 856., ..., 868., 888., 872.], [872., 872., 864., ..., 876., 868., 876.]], [[876., 872., 868., ..., 876., 860., 872.], [896., 872., 872., ..., 872., 868., 864.], [884., 892., 876., ..., 872., 884., 884.], ..., [884., 892., 880., ..., 872., 884., 876.], [888., 884., 888., ..., 872., 888., 876.], [872., 900., 892., ..., 880., 888., 896.]], [[864., 852., 860., ..., 848., 856., 864.], [884., 876., 872., ..., 856., 852., 856.], [860., 868., 856., ..., 860., 880., 880.], ..., [880., 876., 876., ..., 848., 876., 856.], [868., 876., 876., ..., 860., 868., 864.], [872., 884., 880., ..., 864., 868., 884.]]]])

test_conv2d.test_module[Conv2D536-[((1, 320, 17, 17), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[704.65985, 720.20569, 717.29901, ..., 716.59473, 718.78473, 732.51489], [714.96167, 730.50623, 734.52606, ..., 717.62555, 730.15363, 726.70349], [720.53027, 730.05701, 737.21667, ..., 740.70715, 728.24866, 709.23102], ..., [732.56445, 722.27429, 719.19550, ..., 724.61719, 730.15497, 720.10907], [730.55432, 727.45300, 726.36792, ..., 717.03778, 723.21289, 730.95880], [717.72076, 728.58105, 727.38794, ..., 727.03668, 725.02350, 722.73999]], [[688.85583, 708.87030, 713.18335, ..., 709.89209, 703.23358, 712.07233], [701.20087, 725.16187, 719.47040, ..., 706.85431, 717.44592, 701.38409], [705.34198, 719.76276, 729.45081, ..., 718.44659, 727.40430, 706.55267], ..., [711.56604, 708.10791, 711.23822, ..., 713.67529, 714.29712, 701.79407], [720.24304, 715.76770, 716.52820, ..., 703.89209, 720.31311, 718.14349], [709.85278, 722.60004, 714.84985, ..., 728.88910, 709.16827, 711.27185]], [[709.22253, 717.19147, 709.92047, ..., 712.21033, 708.84436, 720.51373], [708.10510, 717.74127, 724.71478, ..., 712.16077, 722.12891, 723.58472], [716.82672, 726.14221, 723.94562, ..., 714.37268, 724.20892, 706.93622], ..., [721.34277, 719.36914, 717.14752, ..., 719.64703, 716.73816, 704.53693], [722.59436, 734.13977, 718.95074, ..., 710.38306, 718.87701, 717.92279], [712.15039, 732.33496, 714.02368, ..., 725.44434, 714.73834, 721.46008]], ..., [[713.58960, 725.59332, 720.18011, ..., 727.84937, 723.42621, 738.95264], [719.43390, 732.43787, 744.98083, ..., 724.56891, 742.53223, 723.55853], [732.75122, 732.79309, 738.83032, ..., 741.78540, 739.01733, 718.28461], ..., [729.90863, 729.89484, 727.60339, ..., 728.32648, 733.75085, 715.40369], [737.43335, 729.73798, 732.44403, ..., 720.21271, 724.07672, 728.05627], [718.14905, 737.55579, 719.44049, ..., 746.01874, 725.98169, 725.67090]], [[691.04578, 708.21063, 716.58081, ..., 701.65918, 711.63953, 720.51678], [707.62537, 717.27747, 725.93079, ..., 704.57520, 714.98767, 711.11462], [714.02820, 720.84143, 726.70593, ..., 721.74628, 723.14783, 708.17908], ..., [711.83423, 722.08832, 722.06073, ..., 711.76263, 719.62189, 711.83405], [716.45557, 723.69452, 713.05316, ..., 704.76459, 719.34521, 718.06781], [708.14465, 729.43750, 710.97620, ..., 720.68542, 713.93512, 718.39551]], [[711.49835, 721.62451, 721.14331, ..., 729.66272, 724.35645, 734.25507], [720.35181, 730.08234, 739.22607, ..., 718.55621, 723.13055, 720.27747], [732.88702, 729.22467, 738.71906, ..., 722.03473, 722.53284, 714.76111], ..., [711.38727, 727.99579, 726.05902, ..., 723.35480, 730.59595, 714.91327], [733.46021, 731.84851, 727.46124, ..., 722.07404, 731.64850, 724.90753], [719.69153, 726.16168, 723.36127, ..., 738.43463, 721.81439, 724.66510]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[712., 720., 708., ..., 716., 728., 744.], [724., 724., 732., ..., 720., 732., 724.], [728., 728., 740., ..., 732., 716., 712.], ..., [720., 728., 712., ..., 728., 740., 728.], [724., 736., 732., ..., 716., 728., 736.], [716., 732., 732., ..., 736., 716., 736.]], [[684., 708., 712., ..., 728., 712., 728.], [708., 728., 732., ..., 716., 732., 712.], [708., 716., 732., ..., 724., 724., 704.], ..., [712., 712., 724., ..., 720., 720., 724.], [728., 708., 720., ..., 704., 712., 728.], [720., 728., 720., ..., 740., 704., 696.]], [[712., 720., 704., ..., 712., 700., 740.], [716., 728., 712., ..., 720., 732., 724.], [708., 716., 732., ..., 716., 724., 704.], ..., [704., 720., 732., ..., 720., 728., 720.], [716., 744., 716., ..., 708., 708., 724.], [708., 744., 724., ..., 712., 712., 720.]], ..., [[716., 728., 724., ..., 736., 720., 744.], [724., 744., 748., ..., 728., 740., 732.], [736., 736., 740., ..., 732., 736., 716.], ..., [732., 724., 728., ..., 740., 732., 724.], [732., 728., 728., ..., 728., 724., 744.], [724., 744., 736., ..., 740., 716., 724.]], [[700., 724., 724., ..., 708., 728., 736.], [724., 716., 728., ..., 720., 728., 712.], [720., 732., 736., ..., 720., 728., 708.], ..., [712., 732., 716., ..., 720., 732., 720.], [712., 724., 712., ..., 708., 732., 728.], [724., 736., 716., ..., 732., 716., 708.]], [[708., 720., 716., ..., 736., 724., 740.], [724., 728., 732., ..., 720., 732., 716.], [736., 728., 720., ..., 712., 732., 720.], ..., [716., 724., 728., ..., 740., 732., 724.], [728., 728., 720., ..., 724., 724., 724.], [736., 744., 736., ..., 740., 716., 728.]]]])

test_conv2d.test_module[Conv2D546-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 6422528 B L1 buffer across 2 banks, where each bank needs to store 3211264 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f0f346d17e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f0f3469ce9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations13data_movement35InterleavedToShardedDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKSt6vectorINS0_6TensorESaISE_EERKSD_ISt8optionalISE_ESaISK_EE+0x1cf) [0x7f0f3532529f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x409139) [0x7f0f23513139] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f0f256ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x40052f) [0x7f0f2350a52f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff92f) [0x7f0f2350992f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff708) [0x7f0f23509708] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff66c) [0x7f0f2350966c] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff44d) [0x7f0f2350944d] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fee51) [0x7f0f23508e51] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fe8d9) [0x7f0f235088d9] --- std::tuple<tt::tt_metal::Tensor, ttnn::operations::sliding_window::ParallelConfig, ttnn::operations::sliding_window::ParallelConfig> ttnn::operations::conv::shard_or_reshard_tensor_if_required<tt::tt_metal::v0::IDevice>(tt::tt_metal::v0::IDevice*, tt::tt_metal::Tensor const&, ttnn::operations::conv::conv2d::Conv2dConfig const&, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, bool, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0x697) [0x7f0f234d9747] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702]

test_conv2d.test_module[Conv2D654-[((1, 512, 22, 66), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1123.74194, 1131.78625, 1134.71973, ..., 1141.94397, 1137.98328, 1144.98633], [1138.39478, 1140.07446, 1126.66443, ..., 1123.50024, 1128.39795, 1154.47534], [1153.36829, 1148.54736, 1129.39172, ..., 1139.83545, 1146.55872, 1169.61426], ..., [1110.46057, 1117.77295, 1124.09448, ..., 1128.18799, 1135.17004, 1133.31714], [1128.73962, 1128.93579, 1125.74695, ..., 1140.84058, 1130.70447, 1133.85205], [1124.50049, 1126.35767, 1119.44885, ..., 1133.96753, 1139.27527, 1131.69067]], [[1156.48242, 1166.31213, 1154.51013, ..., 1170.10559, 1165.74548, 1188.37231], [1169.02295, 1166.69446, 1160.48792, ..., 1156.17322, 1167.63098, 1177.63928], [1188.62805, 1174.49792, 1163.63062, ..., 1174.95349, 1172.85571, 1192.13318], ..., [1151.98950, 1159.19299, 1146.33411, ..., 1170.41663, 1162.05396, 1167.00781], [1165.63806, 1159.78101, 1150.40076, ..., 1176.93530, 1156.49548, 1163.09473], [1147.76477, 1162.69995, 1152.33655, ..., 1160.59644, 1162.74414, 1171.80225]], [[1153.19202, 1145.52051, 1138.60559, ..., 1158.78052, 1151.96106, 1164.63367], [1153.38635, 1147.96399, 1146.37354, ..., 1152.46826, 1153.06860, 1170.30151], [1172.11340, 1159.59167, 1158.13794, ..., 1147.66321, 1160.32092, 1171.63428], ..., [1144.64221, 1133.28979, 1124.36560, ..., 1155.85010, 1154.45276, 1149.34460], [1139.69763, 1144.28821, 1131.14526, ..., 1157.25208, 1138.02295, 1150.33105], [1143.49182, 1152.16162, 1133.58411, ..., 1154.47876, 1148.08594, 1151.94263]], ..., [[1137.37659, 1153.75037, 1135.30090, ..., 1157.54407, 1148.30896, 1172.69275], [1141.49512, 1141.95947, 1155.31030, ..., 1161.70776, 1152.97412, 1163.50500], [1165.45081, 1165.80872, 1150.78442, ..., 1160.37927, 1157.31812, 1168.68799], ..., [1136.31653, 1131.78625, 1138.60925, ..., 1154.12659, 1165.21948, 1145.33008], [1136.33496, 1146.55383, 1138.01562, ..., 1139.43616, 1158.54053, 1147.55945], [1139.66296, 1139.81836, 1129.17358, ..., 1154.63184, 1158.64307, 1141.85437]], [[1133.94885, 1152.06958, 1143.02185, ..., 1151.02783, 1146.46411, 1169.42310], [1150.12219, 1141.82043, 1137.82410, ..., 1143.79468, 1149.56604, 1167.93750], [1158.42859, 1164.12866, 1150.29797, ..., 1151.04175, 1143.96704, 1167.18188], ..., [1133.78943, 1125.78308, 1122.31042, ..., 1147.05713, 1151.90686, 1144.19702], [1137.33569, 1145.90857, 1147.70972, ..., 1146.54944, 1146.72876, 1139.53735], [1134.45081, 1133.19836, 1136.61719, ..., 1148.59644, 1152.86206, 1139.91113]], [[1146.44763, 1147.53125, 1155.17834, ..., 1157.42456, 1161.73816, 1163.46057], [1167.18701, 1170.43982, 1147.99951, ..., 1158.11658, 1171.84924, 1179.40942], [1177.78479, 1170.55066, 1159.33887, ..., 1173.69043, 1177.95947, 1176.81702], ..., [1146.61609, 1144.86780, 1139.71460, ..., 1154.12207, 1165.04456, 1153.78040], [1153.37683, 1156.95068, 1151.70093, ..., 1165.41272, 1149.38831, 1157.57971], [1151.05249, 1155.65869, 1145.75122, ..., 1163.03931, 1162.12915, 1156.31238]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1184., 1192., 1112., ..., 1184., 1216., 1192.], [1200., 1176., 1144., ..., 1200., 1176., 1216.], [1208., 1232., 1144., ..., 1176., 1208., 1208.], ..., [1208., 1144., 1176., ..., 1160., 1160., 1168.], [1160., 1104., 1128., ..., 1208., 1176., 1184.], [1184., 1136., 1152., ..., 1160., 1184., 1144.]], [[1184., 1192., 1160., ..., 1232., 1192., 1216.], [1224., 1224., 1192., ..., 1200., 1200., 1232.], [1224., 1240., 1208., ..., 1224., 1192., 1224.], ..., [1192., 1192., 1184., ..., 1224., 1216., 1176.], [1184., 1184., 1192., ..., 1232., 1176., 1176.], [1240., 1216., 1176., ..., 1168., 1192., 1208.]], [[1216., 1216., 1168., ..., 1232., 1192., 1216.], [1248., 1192., 1144., ..., 1200., 1240., 1248.], [1224., 1200., 1192., ..., 1256., 1224., 1208.], ..., [1232., 1200., 1144., ..., 1208., 1216., 1192.], [1176., 1160., 1152., ..., 1200., 1184., 1208.], [1184., 1200., 1184., ..., 1200., 1208., 1176.]], ..., [[1200., 1232., 1192., ..., 1208., 1200., 1200.], [1216., 1216., 1184., ..., 1240., 1232., 1224.], [1200., 1184., 1224., ..., 1248., 1232., 1224.], ..., [1200., 1168., 1192., ..., 1176., 1232., 1208.], [1160., 1168., 1184., ..., 1200., 1200., 1184.], [1208., 1192., 1200., ..., 1272., 1240., 1208.]], [[1216., 1232., 1176., ..., 1240., 1256., 1232.], [1232., 1200., 1160., ..., 1216., 1224., 1240.], [1240., 1216., 1200., ..., 1200., 1224., 1224.], ..., [1208., 1192., 1216., ..., 1200., 1216., 1200.], [1216., 1184., 1224., ..., 1184., 1200., 1184.], [1184., 1216., 1176., ..., 1240., 1192., 1200.]], [[1200., 1184., 1232., ..., 1232., 1280., 1208.], [1232., 1248., 1208., ..., 1232., 1272., 1256.], [1264., 1224., 1224., ..., 1232., 1240., 1216.], ..., [1232., 1248., 1184., ..., 1224., 1240., 1224.], [1216., 1248., 1184., ..., 1240., 1200., 1208.], [1208., 1208., 1208., ..., 1240., 1200., 1224.]]]])

test_conv2d.test_module[Conv2D658-[((1, 96, 162, 514), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 56868864 B L1 buffer across 64 banks, where each bank needs to store 888576 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f0f346d17e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f0f3469ce9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f0f24109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f0f24109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f0f256ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464]

test_conv2d.test_module[Conv2D661-[((1, 16, 322, 1026), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 58834944 B L1 buffer across 64 banks, where each bank needs to store 919296 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f0f346d17e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f0f3469ce9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f0f24109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f0f24109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f0f256ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464]

test_conv2d.test_module[Conv2D655-[((1, 256, 26, 82), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[575.15588, 567.13208, 562.77960, ..., 568.43201, 575.86432, 564.79663], [567.97913, 574.26434, 564.67780, ..., 575.61017, 565.51288, 580.49982], [579.54333, 577.04596, 569.08600, ..., 578.60724, 573.95166, 566.72955], ..., [579.27063, 577.47430, 572.19806, ..., 570.61713, 579.81677, 575.99249], [568.50488, 568.38116, 579.72943, ..., 568.43646, 577.82367, 574.73798], [575.77356, 577.29193, 581.14154, ..., 579.52362, 586.23096, 575.72394]], [[572.03784, 565.92914, 560.40149, ..., 577.23657, 565.67365, 566.62372], [576.17255, 573.30603, 560.79211, ..., 576.96381, 564.98187, 577.83997], [583.78339, 579.71313, 568.12219, ..., 573.22791, 571.56799, 572.01392], ..., [580.41028, 582.03937, 580.85065, ..., 574.23920, 586.54633, 579.45929], [575.09076, 573.48169, 582.45239, ..., 573.32672, 580.73340, 576.78503], [582.44507, 578.53552, 586.51410, ..., 581.50311, 582.89465, 583.28760]], [[583.14612, 574.09418, 565.66888, ..., 577.72089, 570.44989, 571.80017], [578.91486, 579.67926, 573.92340, ..., 577.47620, 580.15033, 588.06506], [586.24963, 581.65045, 580.22369, ..., 584.33936, 574.68164, 578.63257], ..., [584.77313, 588.86511, 583.85120, ..., 583.88361, 583.14587, 581.94672], [578.70258, 586.37457, 580.98297, ..., 575.90186, 580.97540, 586.27771], [583.23029, 584.17328, 582.55786, ..., 588.53809, 594.72833, 586.99377]], ..., [[586.35022, 574.36267, 565.67865, ..., 580.54572, 585.03894, 576.08612], [581.49738, 576.69208, 569.00732, ..., 570.87933, 573.62756, 574.56921], [587.66766, 583.79016, 564.93274, ..., 588.39771, 574.16522, 580.81470], ..., [590.22211, 587.72711, 585.50604, ..., 577.50555, 584.93347, 578.49188], [583.67352, 573.49945, 584.68506, ..., 579.43835, 590.13409, 586.22058], [587.77594, 585.78546, 584.06982, ..., 581.93091, 584.84827, 593.76178]], [[578.95416, 569.40210, 570.47864, ..., 572.26733, 582.36285, 577.67969], [576.42603, 576.06665, 575.44861, ..., 573.51685, 572.62732, 588.36584], [580.61359, 591.89819, 577.10266, ..., 581.95319, 572.56818, 576.81879], ..., [583.15466, 591.92078, 577.21545, ..., 573.02612, 585.62054, 576.60443], [590.45514, 573.23920, 582.20367, ..., 573.67029, 581.61060, 585.59503], [582.45996, 583.53851, 584.86780, ..., 588.30304, 597.43469, 582.37238]], [[576.11932, 569.02344, 560.48944, ..., 562.32104, 571.82880, 568.68726], [571.36462, 566.44604, 563.38892, ..., 571.71875, 569.09851, 579.82379], [570.99982, 578.72406, 575.73700, ..., 574.71130, 573.26862, 578.23572], ..., [577.23602, 580.81000, 575.88556, ..., 569.89276, 581.87756, 575.38177], [573.28845, 582.59155, 579.35876, ..., 577.72919, 578.11182, 575.59290], [580.87469, 578.97205, 581.01947, ..., 580.53839, 587.98102, 581.38995]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[572., 564., 568., ..., 560., 576., 568.], [568., 576., 564., ..., 568., 568., 576.], [576., 580., 576., ..., 580., 572., 572.], ..., [576., 588., 572., ..., 580., 584., 592.], [564., 568., 576., ..., 564., 580., 568.], [576., 572., 572., ..., 564., 584., 576.]], [[580., 564., 556., ..., 580., 564., 568.], [584., 564., 560., ..., 576., 560., 576.], [588., 580., 564., ..., 580., 576., 568.], ..., [588., 572., 580., ..., 576., 580., 576.], [576., 572., 580., ..., 564., 580., 572.], [584., 572., 584., ..., 580., 584., 580.]], [[576., 576., 564., ..., 572., 572., 572.], [576., 584., 576., ..., 580., 576., 584.], [584., 592., 592., ..., 584., 576., 576.], ..., [572., 584., 576., ..., 588., 580., 584.], [572., 588., 576., ..., 572., 584., 584.], [588., 580., 576., ..., 592., 588., 592.]], ..., [[592., 572., 572., ..., 584., 584., 576.], [568., 572., 564., ..., 572., 572., 580.], [580., 580., 560., ..., 588., 576., 572.], ..., [592., 584., 576., ..., 576., 588., 580.], [584., 572., 584., ..., 580., 596., 584.], [584., 592., 584., ..., 572., 588., 596.]], [[584., 568., 580., ..., 572., 584., 576.], [580., 584., 584., ..., 580., 584., 588.], [584., 592., 568., ..., 584., 576., 576.], ..., [580., 584., 580., ..., 580., 584., 580.], [596., 580., 580., ..., 564., 580., 580.], [592., 588., 580., ..., 584., 596., 584.]], [[572., 576., 552., ..., 564., 564., 568.], [576., 560., 564., ..., 588., 572., 588.], [560., 588., 576., ..., 572., 584., 576.], ..., [576., 580., 568., ..., 564., 588., 572.], [564., 576., 580., ..., 580., 568., 568.], [584., 584., 576., ..., 572., 584., 576.]]]])

test_conv2d.test_module[Conv2D656-[((1, 128, 50, 162), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[292.38879, 288.82825, 281.48386, ..., 289.78348, 285.34991, 284.50665], [292.19717, 295.33084, 285.74014, ..., 284.93506, 287.41583, 286.16208], [294.48444, 289.80035, 281.72824, ..., 284.26566, 288.63760, 285.31863], ..., [283.04153, 289.46539, 289.78513, ..., 291.35208, 288.89603, 285.72330], [280.69495, 287.41638, 281.58020, ..., 287.22244, 284.88840, 289.83099], [281.78345, 277.84180, 281.94141, ..., 283.84735, 289.41953, 289.29926]], [[301.32965, 291.67639, 289.81628, ..., 297.76862, 289.84143, 294.84842], [295.93231, 299.15955, 291.86914, ..., 295.33813, 298.58026, 297.85892], [299.54422, 299.56812, 291.70782, ..., 290.54190, 299.81253, 295.51495], ..., [286.18292, 296.24683, 297.72897, ..., 299.96399, 297.17294, 298.97311], [290.78253, 293.50357, 285.73657, ..., 297.47589, 296.94382, 293.36603], [283.70142, 290.87488, 288.83728, ..., 294.70242, 295.31091, 301.39966]], [[287.52243, 279.45856, 279.11115, ..., 280.24200, 281.78204, 277.91125], [291.57867, 280.40195, 275.05206, ..., 272.41602, 281.95029, 282.83047], [285.10971, 277.46786, 275.60864, ..., 273.99411, 285.54471, 277.12170], ..., [274.41260, 283.42014, 281.63336, ..., 277.81491, 281.33658, 271.89325], [273.75296, 272.82422, 270.51160, ..., 277.50140, 280.52896, 277.78888], [270.55023, 278.02814, 279.73175, ..., 277.14471, 283.69537, 289.55414]], ..., [[295.62427, 287.59009, 290.31171, ..., 289.73254, 292.36551, 289.53067], [287.74530, 295.43713, 291.99496, ..., 284.85730, 290.98785, 288.20734], [292.69006, 287.49933, 286.47110, ..., 280.60962, 291.43811, 288.83133], ..., [283.76694, 287.90469, 291.80579, ..., 289.26480, 287.83145, 285.39728], [283.76181, 283.80829, 281.24652, ..., 289.03094, 285.27628, 291.78088], [282.05515, 287.67731, 284.84717, ..., 288.12708, 295.77975, 293.89316]], [[299.95560, 291.63879, 293.50439, ..., 296.71393, 294.47308, 291.40445], [297.26227, 297.82703, 293.09167, ..., 283.76736, 291.34302, 294.12582], [297.41772, 293.30545, 287.03094, ..., 293.70605, 299.97656, 290.05075], ..., [291.31284, 298.87494, 300.26205, ..., 294.17941, 296.33359, 288.68265], [286.72498, 286.19873, 283.52911, ..., 291.94101, 292.44354, 292.87036], [285.72629, 293.24658, 290.94452, ..., 290.69238, 292.79800, 298.90875]], [[293.18372, 282.08472, 288.05652, ..., 284.04639, 286.48520, 290.88498], [286.81876, 286.35449, 287.04422, ..., 281.77551, 287.53961, 280.44614], [288.32397, 286.77582, 283.38245, ..., 283.77155, 290.98715, 285.66547], ..., [276.42255, 287.69110, 287.15326, ..., 282.21243, 279.71017, 286.66800], [271.89395, 284.27359, 279.06061, ..., 286.45303, 286.48071, 288.76935], [277.18719, 278.89169, 281.46494, ..., 283.90979, 286.37881, 294.37100]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[290., 284., 278., ..., 288., 288., 282.], [292., 288., 286., ..., 282., 284., 286.], [292., 282., 284., ..., 280., 288., 282.], ..., [282., 284., 286., ..., 290., 284., 284.], [278., 282., 278., ..., 282., 286., 288.], [280., 274., 282., ..., 280., 284., 288.]], [[298., 282., 284., ..., 298., 282., 294.], [294., 294., 282., ..., 288., 300., 298.], [298., 296., 290., ..., 288., 294., 292.], ..., [282., 292., 294., ..., 298., 292., 294.], [290., 290., 288., ..., 294., 292., 292.], [284., 290., 284., ..., 296., 290., 294.]], [[284., 278., 276., ..., 276., 280., 274.], [292., 278., 272., ..., 274., 280., 280.], [284., 270., 268., ..., 270., 284., 276.], ..., [268., 278., 282., ..., 274., 278., 268.], [274., 266., 270., ..., 274., 278., 274.], [266., 272., 278., ..., 274., 282., 286.]], ..., [[296., 286., 284., ..., 290., 286., 286.], [286., 294., 294., ..., 280., 288., 286.], [288., 284., 280., ..., 282., 288., 284.], ..., [282., 286., 286., ..., 286., 288., 278.], [282., 278., 280., ..., 288., 284., 288.], [282., 282., 282., ..., 288., 286., 292.]], [[292., 292., 286., ..., 298., 292., 286.], [296., 296., 294., ..., 284., 288., 296.], [294., 290., 282., ..., 292., 302., 286.], ..., [292., 294., 296., ..., 294., 292., 284.], [282., 284., 280., ..., 290., 286., 290.], [284., 290., 286., ..., 284., 288., 298.]], [[292., 278., 286., ..., 280., 286., 288.], [288., 284., 284., ..., 280., 284., 278.], [288., 284., 278., ..., 280., 290., 280.], ..., [276., 286., 286., ..., 278., 274., 282.], [268., 280., 274., ..., 282., 284., 284.], [276., 280., 276., ..., 278., 286., 296.]]]])

test_conv2d.test_module[Conv2D66-[((1, 3, 480, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 73080832 B L1 buffer across 64 banks, where each bank needs to store 1141888 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f0f346d17e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f0f3469ce9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f0f24109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f0f24109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f0f256ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464]

test_conv2d.test_module[Conv2D710-[((1, 160, 32, 32), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[163.61763, 156.89229, 155.46342, ..., 163.26318, 160.23376, 164.51787], [161.07794, 156.25638, 153.78944, ..., 157.63501, 156.55289, 163.30823], [159.30411, 157.59529, 152.33885, ..., 157.72189, 162.62254, 157.81146], ..., [162.14905, 161.33247, 164.27951, ..., 162.93008, 158.27130, 160.29002], [156.77901, 154.54776, 153.59239, ..., 166.32874, 156.77321, 162.51500], [159.60236, 157.86520, 156.02559, ..., 168.00107, 149.05795, 172.40918]], [[160.04715, 150.08737, 152.98448, ..., 162.82678, 155.93616, 164.05855], [156.84683, 155.24889, 156.15956, ..., 160.13980, 156.77151, 160.50522], [159.89375, 160.50562, 150.84901, ..., 161.30927, 164.33189, 153.83209], ..., [161.77534, 160.16504, 153.98999, ..., 159.93658, 158.96744, 160.06047], [159.40672, 153.66162, 153.38977, ..., 163.81100, 156.89465, 154.91600], [156.94804, 152.83066, 155.87468, ..., 162.10783, 152.13927, 168.66220]], [[167.38588, 159.59419, 159.48383, ..., 167.40273, 164.37216, 166.14490], [161.46873, 163.81514, 162.02686, ..., 159.68513, 164.13930, 168.20004], [166.87027, 162.91438, 156.40515, ..., 160.38385, 168.12282, 162.41623], ..., [166.75476, 163.20523, 165.66039, ..., 167.23039, 158.97876, 163.56877], [161.15533, 160.82423, 156.46469, ..., 169.28438, 159.08276, 163.61281], [159.06528, 159.70444, 161.99829, ..., 172.18976, 156.72835, 172.79030]], ..., [[161.83272, 153.46634, 154.27840, ..., 163.53499, 155.98792, 164.98503], [157.35846, 156.42059, 155.18991, ..., 156.79849, 159.82567, 166.24062], [160.21199, 158.74217, 154.96796, ..., 150.21291, 164.51277, 153.99553], ..., [161.46828, 165.03180, 157.37479, ..., 161.29025, 155.75330, 156.05705], [154.02892, 160.96521, 155.86021, ..., 162.89116, 159.80823, 159.96558], [155.65456, 157.80379, 158.22018, ..., 163.23390, 152.40916, 167.81396]], [[164.97800, 152.38100, 159.03935, ..., 167.01158, 160.02512, 168.35092], [162.37811, 161.96584, 161.52773, ..., 161.90698, 162.47769, 167.14287], [166.24474, 162.67984, 159.77565, ..., 161.01369, 166.05771, 164.16235], ..., [168.00978, 165.86255, 164.64641, ..., 166.88358, 162.79727, 160.33279], [160.72342, 165.25928, 160.04895, ..., 166.33405, 163.95053, 166.63446], [158.13324, 161.85181, 162.23631, ..., 173.34357, 157.14749, 177.61293]], [[159.41449, 147.72612, 147.13013, ..., 155.12073, 148.79756, 160.49768], [155.48602, 151.09291, 153.04384, ..., 154.18654, 153.08607, 160.56596], [154.06812, 149.38904, 148.73991, ..., 153.49750, 156.13339, 147.50027], ..., [159.47046, 153.99794, 156.13643, ..., 156.44052, 152.54817, 152.50587], [157.55072, 151.53775, 154.75317, ..., 161.97089, 151.98456, 151.17146], [153.16449, 150.32718, 151.34499, ..., 163.95390, 147.75986, 163.72256]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[161., 157., 152., ..., 160., 160., 162.], [158., 155., 153., ..., 156., 154., 160.], [157., 154., 152., ..., 153., 160., 156.], ..., [160., 158., 161., ..., 161., 155., 158.], [154., 153., 151., ..., 165., 155., 161.], [157., 155., 153., ..., 164., 146., 170.]], [[157., 147., 151., ..., 161., 153., 162.], [154., 153., 154., ..., 158., 156., 157.], [158., 159., 149., ..., 160., 162., 152.], ..., [160., 156., 151., ..., 156., 156., 156.], [158., 152., 150., ..., 161., 155., 153.], [155., 152., 152., ..., 161., 151., 168.]], [[164., 159., 159., ..., 165., 162., 164.], [159., 162., 158., ..., 157., 162., 168.], [165., 160., 153., ..., 158., 164., 161.], ..., [164., 161., 166., ..., 163., 157., 161.], [159., 159., 153., ..., 164., 156., 161.], [157., 157., 160., ..., 168., 154., 171.]], ..., [[159., 152., 153., ..., 162., 152., 163.], [157., 155., 154., ..., 154., 158., 165.], [157., 156., 151., ..., 149., 163., 150.], ..., [159., 164., 154., ..., 159., 154., 156.], [151., 158., 154., ..., 159., 157., 158.], [155., 157., 156., ..., 163., 149., 163.]], [[163., 151., 156., ..., 165., 158., 167.], [160., 160., 159., ..., 160., 162., 165.], [165., 162., 158., ..., 159., 163., 163.], ..., [165., 163., 163., ..., 166., 160., 158.], [158., 162., 157., ..., 164., 160., 164.], [155., 160., 158., ..., 169., 156., 177.]], [[158., 146., 143., ..., 153., 147., 159.], [155., 150., 152., ..., 152., 153., 158.], [151., 147., 146., ..., 152., 154., 146.], ..., [157., 152., 156., ..., 154., 150., 150.], [155., 150., 153., ..., 160., 149., 149.], [152., 149., 149., ..., 160., 144., 161.]]]])

test_conv2d.test_module[Conv2D714-[((1, 64, 128, 128), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 3976 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=0)]. L1 buffer allocated at 417792 and static circular buffer region ends at 624032 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D742-[((1, 768, 28, 28), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 764.32251, 1147.99988, 1156.66199, ..., 1131.84314, 1114.94482, 747.25922], [1148.29688, 1733.33167, 1742.11011, ..., 1710.80115, 1704.86511, 1136.05432], [1132.86609, 1716.32812, 1723.58545, ..., 1703.81689, 1712.13257, 1148.36060], ..., [1144.89526, 1721.34790, 1715.56592, ..., 1715.60583, 1718.21899, 1164.19006], [1153.30139, 1739.31396, 1738.55750, ..., 1737.23975, 1723.84375, 1139.07056], [ 768.61890, 1169.92224, 1152.81348, ..., 1151.87317, 1146.28430, 749.54401]], [[ 781.17389, 1170.04651, 1168.64856, ..., 1148.60352, 1127.84094, 743.93256], [1165.41919, 1752.90710, 1765.91479, ..., 1736.08752, 1714.35791, 1146.92615], [1161.94348, 1745.87561, 1748.31116, ..., 1745.84412, 1733.57935, 1159.27795], ..., [1150.76868, 1741.47949, 1732.22009, ..., 1750.41992, 1759.82605, 1185.49658], [1180.77405, 1764.76953, 1761.27954, ..., 1742.27771, 1722.96509, 1168.02734], [ 785.50165, 1173.81665, 1176.43506, ..., 1161.53235, 1163.76611, 771.69800]], [[ 777.97589, 1165.05811, 1178.13391, ..., 1146.46460, 1123.79541, 752.35895], [1175.15552, 1777.22717, 1777.73718, ..., 1730.09644, 1741.05518, 1142.26953], [1175.48828, 1743.22034, 1752.32642, ..., 1754.53577, 1735.10999, 1144.97205], ..., [1160.93005, 1740.88904, 1755.34216, ..., 1754.82483, 1743.72388, 1158.83459], [1184.56128, 1771.18762, 1769.22046, ..., 1764.63428, 1756.09778, 1162.62390], [ 793.55042, 1170.28638, 1171.92041, ..., 1162.62610, 1168.87268, 748.53656]], ..., [[ 764.93597, 1154.60339, 1148.71033, ..., 1122.18372, 1112.15002, 739.17712], [1136.50220, 1730.86426, 1756.22583, ..., 1705.36108, 1692.07751, 1136.39014], [1134.24609, 1711.10095, 1726.65405, ..., 1716.98779, 1723.65759, 1145.89539], ..., [1136.13159, 1722.37952, 1720.99829, ..., 1729.11267, 1722.13269, 1165.26318], [1156.61951, 1730.36389, 1740.17712, ..., 1739.45801, 1720.30640, 1156.31860], [ 777.29590, 1151.37524, 1167.01221, ..., 1148.36377, 1150.19568, 756.68817]], [[ 770.97675, 1164.97656, 1170.15002, ..., 1136.55530, 1116.33252, 760.11169], [1168.06885, 1764.61243, 1771.88574, ..., 1745.37524, 1722.44995, 1148.34338], [1148.05713, 1745.88037, 1750.98218, ..., 1742.82141, 1738.58704, 1170.11707], ..., [1151.66211, 1759.40442, 1733.02356, ..., 1744.94458, 1748.08765, 1165.12854], [1180.80249, 1757.28198, 1771.44263, ..., 1741.65356, 1746.76587, 1173.08569], [ 777.10107, 1177.36768, 1175.73889, ..., 1163.59753, 1162.52002, 774.06995]], [[ 777.84503, 1165.98682, 1160.08484, ..., 1132.90637, 1127.62842, 750.84784], [1155.25610, 1727.66321, 1740.81482, ..., 1714.51501, 1708.95105, 1136.30713], [1143.88635, 1726.45776, 1731.67993, ..., 1734.48584, 1701.02576, 1140.24182], ..., [1144.35229, 1729.08557, 1720.12561, ..., 1714.19653, 1730.03210, 1166.20935], [1162.35449, 1735.36902, 1731.32715, ..., 1730.67749, 1728.96338, 1151.83008], [ 781.77960, 1162.79993, 1163.61987, ..., 1141.61157, 1138.46985, 754.48676]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 768., 1224., 1216., ..., 1160., 1176., 756.], [1184., 1960., 1992., ..., 2000., 1904., 1168.], [1168., 1944., 2048., ..., 1928., 1936., 1192.], ..., [1224., 1976., 1920., ..., 1952., 1920., 1232.], [1216., 2032., 2016., ..., 2008., 1960., 1208.], [ 780., 1240., 1192., ..., 1192., 1224., 748.]], [[ 772., 1240., 1256., ..., 1184., 1216., 748.], [1208., 2048., 2048., ..., 2000., 1904., 1152.], [1216., 1960., 2048., ..., 2000., 1984., 1184.], ..., [1216., 2016., 1944., ..., 2032., 2048., 1256.], [1264., 1992., 2048., ..., 2024., 1912., 1216.], [ 788., 1232., 1232., ..., 1208., 1232., 772.]], [[ 780., 1232., 1272., ..., 1184., 1160., 760.], [1216., 2032., 2048., ..., 1936., 2008., 1176.], [1192., 1952., 1944., ..., 1960., 1984., 1208.], ..., [1192., 1944., 1872., ..., 1960., 1968., 1240.], [1240., 1944., 2016., ..., 2016., 1904., 1208.], [ 796., 1208., 1232., ..., 1224., 1200., 748.]], ..., [[ 760., 1216., 1200., ..., 1152., 1200., 752.], [1160., 2032., 2032., ..., 1960., 1896., 1152.], [1144., 1912., 1960., ..., 1944., 2016., 1184.], ..., [1160., 1952., 2048., ..., 1952., 2008., 1184.], [1216., 1968., 2024., ..., 2008., 1984., 1192.], [ 788., 1216., 1232., ..., 1176., 1168., 744.]], [[ 776., 1192., 1176., ..., 1176., 1160., 768.], [1264., 2032., 2040., ..., 2000., 1960., 1184.], [1168., 1992., 2048., ..., 1976., 2040., 1240.], ..., [1184., 1968., 1984., ..., 1992., 2040., 1208.], [1248., 2024., 2048., ..., 2048., 2024., 1232.], [ 780., 1216., 1232., ..., 1208., 1184., 768.]], [[ 784., 1224., 1200., ..., 1200., 1160., 764.], [1200., 1960., 2016., ..., 1992., 1912., 1136.], [1176., 1992., 2024., ..., 1928., 1896., 1184.], ..., [1192., 1992., 1968., ..., 2016., 1936., 1216.], [1192., 1952., 1976., ..., 1984., 1968., 1200.], [ 784., 1264., 1232., ..., 1216., 1192., 760.]]]])

test_conv2d.test_module[Conv2D746-[((1, 512, 7, 7), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[6271.46924]], [[6279.07324]], [[6326.94238]], ..., [[6337.09717]], [[6261.14111]], [[6273.26562]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[2112.]], [[2096.]], [[2144.]], ..., [[2144.]], [[2080.]], [[2048.]]]])

test_conv2d.test_module[Conv2D793-[((1, 728, 38, 38), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[188.23485, 184.35355, 181.96687, ..., 188.17238, 178.84848, 183.42664], [182.75989, 176.71739, 186.50337, ..., 176.51671, 184.05318, 185.41235], [183.94679, 183.50420, 174.40067, ..., 182.10774, 186.33670, 180.17229], ..., [182.11588, 186.48759, 190.25653, ..., 180.83156, 182.26735, 183.14192], [180.47639, 179.47728, 176.98885, ..., 177.97359, 189.64632, 181.82396], [171.00044, 186.50351, 197.15355, ..., 171.72321, 182.80507, 182.70906]], [[180.06230, 177.42464, 179.18880, ..., 184.13008, 171.41469, 171.42195], [171.19254, 170.91469, 176.76030, ..., 170.93987, 172.52110, 179.71127], [180.61081, 175.54163, 169.97005, ..., 179.25630, 176.64951, 181.99901], ..., [175.40451, 182.06931, 184.13516, ..., 169.78699, 183.31326, 171.75449], [174.55891, 176.79314, 169.59018, ..., 171.90373, 178.09946, 178.00601], [165.39041, 179.44839, 185.15582, ..., 166.24182, 181.67929, 172.87358]], [[176.90561, 180.94205, 179.74303, ..., 185.47302, 176.30957, 175.91022], [174.54388, 170.51259, 172.07155, ..., 170.75110, 173.73105, 177.44505], [180.11049, 177.53918, 168.51317, ..., 174.94203, 183.77437, 176.06233], ..., [172.44650, 183.31784, 181.18246, ..., 174.38698, 176.87535, 176.49869], [174.53012, 175.97362, 172.26494, ..., 170.91396, 182.20728, 174.32750], [167.32932, 174.72224, 190.53694, ..., 169.00146, 177.82974, 173.05391]], ..., [[187.25592, 182.94846, 186.93187, ..., 186.43997, 180.58179, 174.46852], [181.84598, 177.70908, 181.38681, ..., 178.22940, 182.02953, 184.65552], [185.32495, 183.26804, 177.33653, ..., 179.26730, 183.34680, 182.02573], ..., [179.69894, 187.11310, 186.40475, ..., 184.48891, 184.55070, 179.97394], [184.71716, 180.41743, 180.43378, ..., 179.94006, 189.27513, 183.28984], [175.97694, 185.64105, 192.19966, ..., 174.07304, 186.25116, 178.63181]], [[190.93617, 185.30762, 186.79842, ..., 192.62230, 184.92404, 180.46663], [184.05756, 178.15321, 187.69037, ..., 184.13696, 180.97592, 187.88181], [186.08798, 182.84401, 178.75853, ..., 185.93286, 188.21701, 184.18069], ..., [183.98506, 189.22560, 189.20699, ..., 183.88567, 188.18384, 183.95311], [186.55103, 181.01158, 179.41507, ..., 181.40228, 188.95961, 187.63014], [175.82622, 189.34735, 200.34724, ..., 175.39761, 190.74959, 183.25020]], [[178.49944, 175.28130, 171.69615, ..., 182.15393, 176.03970, 172.79121], [171.08279, 168.79492, 174.74071, ..., 169.33733, 170.44937, 176.07729], [180.71608, 174.72926, 169.83838, ..., 175.43716, 172.73036, 174.24013], ..., [174.42354, 180.56555, 179.96602, ..., 172.04434, 175.47867, 174.26752], [176.77933, 177.19945, 173.65614, ..., 171.27832, 178.82411, 174.01314], [165.94135, 176.81068, 185.20480, ..., 171.14319, 171.01100, 171.70425]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[187., 180., 179., ..., 186., 179., 182.], [179., 173., 184., ..., 176., 182., 183.], [180., 180., 172., ..., 179., 184., 180.], ..., [179., 182., 187., ..., 178., 181., 181.], [179., 176., 176., ..., 174., 187., 179.], [167., 182., 195., ..., 171., 182., 180.]], [[177., 174., 175., ..., 181., 169., 169.], [167., 169., 176., ..., 167., 170., 178.], [179., 174., 169., ..., 176., 174., 177.], ..., [174., 180., 181., ..., 168., 181., 172.], [172., 176., 168., ..., 170., 176., 174.], [162., 178., 183., ..., 163., 179., 170.]], [[174., 177., 178., ..., 182., 176., 175.], [172., 168., 170., ..., 167., 173., 176.], [178., 176., 167., ..., 173., 179., 173.], ..., [169., 180., 181., ..., 172., 175., 173.], [172., 174., 169., ..., 169., 180., 172.], [163., 173., 190., ..., 169., 176., 170.]], ..., [[185., 183., 186., ..., 184., 177., 171.], [179., 175., 178., ..., 175., 178., 183.], [183., 181., 176., ..., 178., 182., 182.], ..., [178., 187., 185., ..., 181., 183., 177.], [183., 178., 178., ..., 179., 188., 180.], [175., 184., 188., ..., 171., 184., 176.]], [[188., 181., 184., ..., 190., 180., 179.], [180., 176., 185., ..., 182., 179., 185.], [183., 180., 177., ..., 183., 187., 180.], ..., [182., 184., 187., ..., 179., 188., 182.], [183., 178., 176., ..., 179., 186., 187.], [174., 189., 199., ..., 172., 188., 180.]], [[177., 173., 169., ..., 181., 173., 170.], [168., 165., 171., ..., 168., 169., 174.], [177., 172., 168., ..., 174., 169., 173.], ..., [174., 178., 176., ..., 168., 174., 172.], [173., 175., 170., ..., 168., 176., 172.], [163., 175., 183., ..., 167., 168., 169.]]]])

test_conv2d.test_module[Conv2D802-[((1, 64, 320, 320), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 4413 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 413440 and static circular buffer region ends at 525728 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D806-[((1, 128, 160, 160), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[134.34175, 198.46707, 196.26472, ..., 200.91847, 199.89516, 196.32602], [190.45955, 295.75034, 294.81174, ..., 289.09537, 293.87521, 288.77640], [195.00223, 297.07993, 296.29547, ..., 295.30029, 286.16513, 289.16296], ..., [194.34036, 286.68500, 293.93121, ..., 292.40533, 291.09146, 287.70129], [190.55621, 295.20288, 282.92090, ..., 286.49493, 295.64221, 292.71628], [197.88033, 298.86710, 290.77853, ..., 290.74207, 285.32904, 294.40488]], [[132.45955, 193.42662, 192.54347, ..., 185.67476, 196.49034, 191.21706], [195.65126, 293.96164, 293.54523, ..., 294.87216, 294.67917, 284.00613], [197.81320, 294.76065, 293.81174, ..., 293.62842, 289.47839, 285.70105], ..., [193.08162, 289.42612, 283.73331, ..., 296.80991, 294.78424, 292.24161], [195.00389, 292.60287, 283.01865, ..., 295.52148, 292.41589, 289.30301], [198.06064, 299.27402, 294.34561, ..., 290.21317, 278.82907, 293.44055]], [[127.98489, 188.12253, 189.20547, ..., 182.36803, 187.98903, 178.58681], [186.37721, 277.58395, 284.91888, ..., 278.90134, 281.42358, 274.73199], [190.76091, 286.12881, 284.31149, ..., 278.75314, 280.90436, 272.37933], ..., [193.51924, 273.82004, 276.66315, ..., 291.06342, 280.61249, 280.34824], [189.98032, 279.80603, 269.54398, ..., 285.28696, 275.87451, 277.47668], [191.60883, 283.76273, 284.37924, ..., 280.76343, 270.87387, 277.66272]], ..., [[129.54500, 195.24768, 190.58356, ..., 189.63535, 204.29839, 192.72447], [187.33496, 294.16672, 289.32886, ..., 287.19318, 294.62289, 278.64597], [196.16690, 290.75955, 288.17352, ..., 288.16647, 282.38443, 280.82523], ..., [188.94250, 286.27493, 285.31161, ..., 291.29797, 290.03116, 288.39874], [189.21873, 282.82019, 274.11700, ..., 287.63507, 289.47198, 292.06140], [190.44397, 290.92249, 286.83868, ..., 287.74228, 280.67310, 283.94775]], [[130.75525, 188.39975, 193.75400, ..., 187.18666, 191.00577, 188.19875], [191.80006, 296.30292, 290.62625, ..., 288.21942, 290.45508, 277.56534], [195.79951, 294.07083, 295.33334, ..., 289.35318, 286.20837, 277.54337], ..., [199.48988, 288.18756, 287.13458, ..., 297.58514, 293.82059, 287.74094], [195.89194, 287.56299, 282.88928, ..., 291.81818, 291.28070, 284.64511], [193.36125, 291.27878, 288.85779, ..., 285.52057, 280.95795, 288.04916]], [[130.26965, 195.33058, 195.32520, ..., 186.64105, 192.87082, 188.73000], [187.35367, 293.86182, 289.49542, ..., 283.59207, 292.46024, 280.71265], [199.44220, 292.92639, 291.67590, ..., 291.78278, 286.59998, 280.02167], ..., [194.42250, 283.06079, 284.12177, ..., 299.41519, 289.67615, 288.13358], [193.10335, 287.39917, 279.95752, ..., 287.93579, 286.46075, 283.62549], [192.53772, 288.78055, 290.77640, ..., 289.32108, 278.31229, 294.04678]]]]), compiled_model=tensor([[[[133.00000, 195.00000, 195.00000, ..., 198.00000, 198.00000, 195.00000], [188.00000, 290.00000, 292.00000, ..., 282.00000, 292.00000, 288.00000], [193.00000, 296.00000, 290.00000, ..., 292.00000, 284.00000, 286.00000], ..., [194.00000, 284.00000, 292.00000, ..., 288.00000, 286.00000, 284.00000], [188.00000, 290.00000, 280.00000, ..., 284.00000, 294.00000, 292.00000], [196.00000, 296.00000, 288.00000, ..., 290.00000, 280.00000, 290.00000]], [[130.00000, 189.00000, 190.00000, ..., 180.00000, 192.00000, 188.00000], [193.00000, 290.00000, 284.00000, ..., 292.00000, 292.00000, 280.00000], [194.00000, 288.00000, 292.00000, ..., 294.00000, 284.00000, 286.00000], ..., [191.00000, 286.00000, 274.00000, ..., 296.00000, 288.00000, 288.00000], [193.00000, 290.00000, 280.00000, ..., 290.00000, 288.00000, 286.00000], [195.00000, 294.00000, 288.00000, ..., 288.00000, 276.00000, 292.00000]], [[126.50000, 185.00000, 186.00000, ..., 179.00000, 185.00000, 175.00000], [183.00000, 276.00000, 280.00000, ..., 272.00000, 280.00000, 272.00000], [186.00000, 286.00000, 280.00000, ..., 276.00000, 278.00000, 272.00000], ..., [191.00000, 272.00000, 272.00000, ..., 292.00000, 280.00000, 278.00000], [188.00000, 278.00000, 264.00000, ..., 282.00000, 272.00000, 276.00000], [190.00000, 280.00000, 282.00000, ..., 280.00000, 268.00000, 272.00000]], ..., [[127.50000, 192.00000, 188.00000, ..., 185.00000, 200.00000, 189.00000], [183.00000, 290.00000, 288.00000, ..., 286.00000, 292.00000, 282.00000], [194.00000, 290.00000, 282.00000, ..., 280.00000, 278.00000, 280.00000], ..., [186.00000, 280.00000, 284.00000, ..., 288.00000, 284.00000, 286.00000], [186.00000, 276.00000, 272.00000, ..., 286.00000, 284.00000, 284.00000], [185.00000, 286.00000, 282.00000, ..., 286.00000, 278.00000, 280.00000]], [[129.00000, 184.00000, 188.00000, ..., 186.00000, 188.00000, 186.00000], [191.00000, 298.00000, 282.00000, ..., 286.00000, 288.00000, 274.00000], [192.00000, 294.00000, 288.00000, ..., 292.00000, 282.00000, 274.00000], ..., [198.00000, 288.00000, 282.00000, ..., 294.00000, 294.00000, 290.00000], [193.00000, 284.00000, 278.00000, ..., 288.00000, 286.00000, 278.00000], [189.00000, 292.00000, 292.00000, ..., 282.00000, 280.00000, 286.00000]], [[127.50000, 193.00000, 195.00000, ..., 184.00000, 189.00000, 188.00000], [186.00000, 292.00000, 288.00000, ..., 280.00000, 290.00000, 278.00000], [199.00000, 290.00000, 288.00000, ..., 288.00000, 280.00000, 278.00000], ..., [193.00000, 276.00000, 278.00000, ..., 296.00000, 286.00000, 284.00000], [190.00000, 286.00000, 280.00000, ..., 284.00000, 282.00000, 280.00000], [189.00000, 286.00000, 286.00000, ..., 290.00000, 280.00000, 292.00000]]]])

test_conv2d.test_module[Conv2D809-[((1, 128, 160, 160), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[134.34175, 198.46707, 196.26472, ..., 200.91847, 199.89516, 196.32602], [190.45955, 295.75034, 294.81174, ..., 289.09537, 293.87521, 288.77640], [195.00223, 297.07993, 296.29547, ..., 295.30029, 286.16513, 289.16296], ..., [194.34036, 286.68500, 293.93121, ..., 292.40533, 291.09146, 287.70129], [190.55621, 295.20288, 282.92090, ..., 286.49493, 295.64221, 292.71628], [197.88033, 298.86710, 290.77853, ..., 290.74207, 285.32904, 294.40488]], [[132.45955, 193.42662, 192.54347, ..., 185.67476, 196.49034, 191.21706], [195.65126, 293.96164, 293.54523, ..., 294.87216, 294.67917, 284.00613], [197.81320, 294.76065, 293.81174, ..., 293.62842, 289.47839, 285.70105], ..., [193.08162, 289.42612, 283.73331, ..., 296.80991, 294.78424, 292.24161], [195.00389, 292.60287, 283.01865, ..., 295.52148, 292.41589, 289.30301], [198.06064, 299.27402, 294.34561, ..., 290.21317, 278.82907, 293.44055]], [[127.98489, 188.12253, 189.20547, ..., 182.36803, 187.98903, 178.58681], [186.37721, 277.58395, 284.91888, ..., 278.90134, 281.42358, 274.73199], [190.76091, 286.12881, 284.31149, ..., 278.75314, 280.90436, 272.37933], ..., [193.51924, 273.82004, 276.66315, ..., 291.06342, 280.61249, 280.34824], [189.98032, 279.80603, 269.54398, ..., 285.28696, 275.87451, 277.47668], [191.60883, 283.76273, 284.37924, ..., 280.76343, 270.87387, 277.66272]], ..., [[128.60689, 185.23308, 189.24399, ..., 185.54343, 186.44052, 182.87155], [185.73358, 291.75244, 281.97351, ..., 279.97052, 286.49744, 275.76367], [190.70506, 280.78546, 283.68103, ..., 278.95480, 281.73026, 271.56552], ..., [192.72807, 276.47366, 278.69473, ..., 289.88278, 285.58569, 282.76813], [188.47011, 283.19958, 276.32944, ..., 289.28589, 287.70456, 282.80756], [190.09619, 295.13217, 284.57925, ..., 278.87485, 279.12067, 285.46307]], [[134.24666, 191.15558, 191.70528, ..., 186.11215, 193.48885, 187.74725], [193.49997, 285.07574, 287.43820, ..., 290.85681, 287.87033, 278.33307], [193.28668, 289.72369, 287.69595, ..., 288.75821, 284.55722, 277.89764], ..., [194.79395, 283.98880, 286.17084, ..., 291.03809, 291.10132, 285.56879], [194.72575, 283.95309, 279.06274, ..., 287.87561, 288.73193, 285.77496], [189.29820, 294.70081, 284.01837, ..., 284.06287, 280.05078, 282.57538]], [[132.44672, 193.87961, 195.07584, ..., 191.23459, 194.42561, 191.36159], [191.34695, 294.91278, 290.68918, ..., 288.65607, 293.12228, 283.17532], [196.20459, 294.61945, 291.68793, ..., 291.68939, 284.18073, 278.83554], ..., [199.42142, 287.17178, 290.54102, ..., 296.76526, 294.68240, 285.81882], [195.49556, 292.43240, 285.19208, ..., 291.25290, 292.33646, 287.27496], [196.75864, 292.47122, 283.24130, ..., 293.19366, 285.83688, 292.24261]]]]), compiled_model=tensor([[[[133.00000, 195.00000, 195.00000, ..., 198.00000, 198.00000, 195.00000], [188.00000, 290.00000, 292.00000, ..., 282.00000, 292.00000, 288.00000], [193.00000, 296.00000, 290.00000, ..., 292.00000, 284.00000, 286.00000], ..., [194.00000, 284.00000, 292.00000, ..., 288.00000, 286.00000, 284.00000], [188.00000, 290.00000, 280.00000, ..., 284.00000, 294.00000, 292.00000], [196.00000, 296.00000, 288.00000, ..., 290.00000, 280.00000, 290.00000]], [[130.00000, 189.00000, 190.00000, ..., 180.00000, 192.00000, 188.00000], [193.00000, 290.00000, 284.00000, ..., 292.00000, 292.00000, 280.00000], [194.00000, 288.00000, 292.00000, ..., 294.00000, 284.00000, 286.00000], ..., [191.00000, 286.00000, 274.00000, ..., 296.00000, 288.00000, 288.00000], [193.00000, 290.00000, 280.00000, ..., 290.00000, 288.00000, 286.00000], [195.00000, 294.00000, 288.00000, ..., 288.00000, 276.00000, 292.00000]], [[126.50000, 185.00000, 186.00000, ..., 179.00000, 185.00000, 175.00000], [183.00000, 276.00000, 280.00000, ..., 272.00000, 280.00000, 272.00000], [186.00000, 286.00000, 280.00000, ..., 276.00000, 278.00000, 272.00000], ..., [191.00000, 272.00000, 272.00000, ..., 292.00000, 280.00000, 278.00000], [188.00000, 278.00000, 264.00000, ..., 282.00000, 272.00000, 276.00000], [190.00000, 280.00000, 282.00000, ..., 280.00000, 268.00000, 272.00000]], ..., [[127.50000, 182.00000, 187.00000, ..., 182.00000, 183.00000, 180.00000], [184.00000, 288.00000, 278.00000, ..., 278.00000, 278.00000, 274.00000], [187.00000, 276.00000, 282.00000, ..., 276.00000, 276.00000, 270.00000], ..., [191.00000, 278.00000, 276.00000, ..., 284.00000, 282.00000, 276.00000], [185.00000, 280.00000, 272.00000, ..., 284.00000, 286.00000, 274.00000], [189.00000, 292.00000, 284.00000, ..., 276.00000, 274.00000, 286.00000]], [[132.00000, 187.00000, 190.00000, ..., 183.00000, 189.00000, 184.00000], [194.00000, 278.00000, 284.00000, ..., 290.00000, 290.00000, 276.00000], [192.00000, 288.00000, 288.00000, ..., 286.00000, 282.00000, 278.00000], ..., [189.00000, 280.00000, 284.00000, ..., 286.00000, 292.00000, 286.00000], [193.00000, 278.00000, 276.00000, ..., 282.00000, 284.00000, 282.00000], [186.00000, 288.00000, 280.00000, ..., 280.00000, 278.00000, 280.00000]], [[131.00000, 191.00000, 191.00000, ..., 190.00000, 194.00000, 189.00000], [190.00000, 294.00000, 288.00000, ..., 284.00000, 288.00000, 276.00000], [194.00000, 290.00000, 288.00000, ..., 286.00000, 280.00000, 272.00000], ..., [194.00000, 280.00000, 286.00000, ..., 294.00000, 290.00000, 276.00000], [192.00000, 284.00000, 284.00000, ..., 288.00000, 288.00000, 284.00000], [194.00000, 288.00000, 278.00000, ..., 288.00000, 280.00000, 292.00000]]]])

test_conv2d.test_module[Conv2D822-[((1, 80, 320, 320), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 58305792 B L1 buffer across 62 banks, where each bank needs to store 940416 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f0f346d17e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f0f3469ce9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f0f24109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f0f24109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f0f256ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464]

test_conv2d.test_module[Conv2D827-[((1, 160, 160, 160), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[164.97192, 246.00609, 251.49431, ..., 246.36862, 247.40137, 244.96127], [243.32504, 370.13290, 371.05392, ..., 359.13281, 367.55008, 361.93552], [243.72290, 370.45328, 370.87683, ..., 366.56622, 366.55814, 366.86145], ..., [242.85329, 366.89090, 361.47015, ..., 374.01208, 367.69345, 364.86572], [252.25481, 365.66251, 359.69235, ..., 362.45648, 363.49811, 363.39487], [245.84773, 372.35007, 371.16812, ..., 366.34741, 354.03998, 362.58093]], [[165.19228, 245.53847, 250.34007, ..., 241.15892, 239.90074, 235.78392], [234.65190, 377.60077, 372.80701, ..., 363.02515, 377.05078, 360.58542], [245.66498, 375.64728, 367.37488, ..., 368.82513, 372.75952, 364.30307], ..., [247.06613, 361.95462, 363.97424, ..., 365.09622, 371.82159, 364.54001], [241.88945, 366.79776, 362.02380, ..., 367.83749, 367.05573, 365.85599], [245.59781, 375.48660, 372.69531, ..., 367.04117, 360.72867, 365.68900]], [[158.71953, 231.19710, 233.34583, ..., 231.76596, 229.78441, 230.49461], [243.52196, 360.81458, 357.58563, ..., 352.00580, 361.36874, 349.02350], [246.70888, 362.02905, 359.05209, ..., 359.70023, 354.83441, 354.67221], ..., [245.95633, 355.49564, 353.70206, ..., 368.28024, 360.64645, 357.89526], [238.57872, 358.02438, 351.83621, ..., 362.05203, 358.89474, 353.30383], [242.68562, 361.18967, 357.89944, ..., 357.68988, 348.06549, 358.25043]], ..., [[155.58014, 239.06082, 241.79230, ..., 231.20146, 236.99809, 235.38641], [232.52258, 359.14743, 355.48969, ..., 354.93756, 358.69559, 345.83420], [243.34273, 362.15631, 366.92627, ..., 356.87180, 357.03763, 353.15073], ..., [239.52788, 355.91174, 350.61633, ..., 365.50018, 356.68033, 355.51639], [233.47258, 355.25143, 348.25607, ..., 360.20416, 351.88828, 356.30765], [241.11821, 365.05826, 359.20956, ..., 353.80679, 352.27887, 357.28510]], [[160.49321, 238.65155, 241.12001, ..., 234.50510, 243.36356, 239.40109], [239.49072, 361.64175, 362.24228, ..., 361.13553, 361.44287, 352.49579], [245.44653, 363.45547, 368.66074, ..., 360.42981, 355.10736, 364.20239], ..., [246.63017, 353.62656, 353.89771, ..., 368.49677, 363.83475, 363.05963], [243.96289, 355.13397, 351.82312, ..., 357.46295, 361.67212, 352.22620], [242.00676, 361.98090, 359.64539, ..., 360.72269, 348.07581, 358.85336]], [[156.95349, 250.87868, 251.66560, ..., 242.40018, 250.56345, 246.65143], [231.73669, 370.62430, 367.56546, ..., 369.37488, 370.22339, 352.86639], [241.63966, 371.37656, 375.99350, ..., 372.65970, 370.51819, 367.15680], ..., [241.66867, 360.05429, 375.00925, ..., 371.54462, 367.74319, 372.11459], [237.08336, 365.79990, 360.17413, ..., 370.94556, 370.58215, 358.68814], [236.82425, 371.77625, 365.77286, ..., 367.46698, 360.20288, 365.33136]]]]), compiled_model=tensor([[[[165., 244., 249., ..., 246., 242., 242.], [239., 366., 374., ..., 356., 364., 360.], [239., 368., 368., ..., 368., 368., 364.], ..., [239., 364., 360., ..., 372., 362., 364.], [249., 364., 354., ..., 360., 364., 360.], [239., 368., 368., ..., 364., 352., 358.]], [[165., 243., 249., ..., 240., 235., 233.], [229., 386., 376., ..., 370., 372., 360.], [245., 370., 362., ..., 366., 372., 360.], ..., [246., 358., 366., ..., 366., 366., 364.], [239., 370., 362., ..., 368., 368., 370.], [241., 374., 374., ..., 362., 358., 364.]], [[155., 226., 229., ..., 230., 229., 227.], [243., 362., 358., ..., 354., 362., 346.], [243., 358., 354., ..., 360., 348., 354.], ..., [241., 354., 354., ..., 368., 358., 356.], [235., 354., 346., ..., 364., 354., 352.], [240., 358., 354., ..., 354., 344., 354.]], ..., [[154., 236., 239., ..., 228., 232., 233.], [230., 356., 356., ..., 352., 356., 344.], [242., 354., 362., ..., 354., 356., 356.], ..., [237., 354., 352., ..., 364., 352., 354.], [232., 352., 344., ..., 360., 350., 354.], [236., 366., 356., ..., 354., 346., 354.]], [[157., 235., 238., ..., 233., 239., 236.], [237., 362., 358., ..., 354., 354., 356.], [241., 360., 372., ..., 360., 356., 362.], ..., [243., 352., 352., ..., 370., 358., 362.], [242., 354., 354., ..., 360., 362., 352.], [238., 360., 356., ..., 360., 350., 352.]], [[154., 249., 247., ..., 240., 249., 242.], [228., 372., 360., ..., 362., 362., 352.], [238., 372., 372., ..., 374., 366., 364.], ..., [238., 352., 374., ..., 378., 362., 374.], [233., 368., 358., ..., 370., 366., 360.], [236., 372., 356., ..., 364., 362., 362.]]]])

test_conv2d.test_module[Conv2D839-[((1, 640, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 654.50470, 973.58966, 969.08160, ..., 955.14844, 958.49219, 959.40051], [ 969.73572, 1435.56494, 1451.75159, ..., 1445.22437, 1429.00305, 1429.97607], [ 989.21454, 1437.22827, 1436.42444, ..., 1440.89026, 1444.17639, 1449.64062], ..., [ 981.92194, 1456.28296, 1456.32410, ..., 1434.30090, 1462.28662, 1441.13733], [ 978.96344, 1453.90076, 1443.56885, ..., 1437.06201, 1433.78174, 1429.47668], [ 967.86481, 1429.91882, 1435.22888, ..., 1439.87390, 1441.83740, 1441.99573]], [[ 679.50323, 996.71295, 986.07288, ..., 969.56995, 969.85760, 978.18805], [ 986.27991, 1437.59753, 1456.14331, ..., 1476.72473, 1439.32043, 1430.29004], [ 990.72369, 1439.52808, 1415.56689, ..., 1447.38440, 1458.03943, 1451.41724], ..., [ 985.39673, 1473.47083, 1446.92981, ..., 1437.40417, 1472.49194, 1448.40698], [ 987.77625, 1461.63708, 1449.19275, ..., 1445.95142, 1445.45532, 1438.33960], [ 976.26111, 1440.06421, 1453.22400, ..., 1462.93774, 1447.16748, 1455.15186]], [[ 651.64911, 976.19995, 965.94708, ..., 950.44458, 947.63361, 948.88434], [ 956.59644, 1421.86804, 1447.32861, ..., 1445.15295, 1439.00293, 1426.13550], [ 952.34576, 1425.62158, 1419.71497, ..., 1445.53784, 1444.77295, 1450.20825], ..., [ 957.41199, 1457.22046, 1455.69934, ..., 1422.74988, 1451.20239, 1436.44592], [ 952.13141, 1452.26794, 1432.20642, ..., 1434.53857, 1428.67627, 1411.73303], [ 951.92834, 1426.70337, 1432.68701, ..., 1439.23169, 1432.60010, 1448.67004]], ..., [[ 666.93927, 999.30359, 975.63074, ..., 966.66107, 974.39093, 960.75055], [ 966.20184, 1446.46704, 1458.48181, ..., 1460.37646, 1454.21741, 1434.98669], [ 976.77972, 1447.85229, 1439.24402, ..., 1457.47729, 1455.77747, 1453.66943], ..., [ 974.94586, 1487.97839, 1451.54736, ..., 1429.80762, 1462.31079, 1442.47876], [ 976.97821, 1461.46155, 1450.96350, ..., 1453.94849, 1446.82397, 1447.16431], [ 966.47217, 1455.38818, 1447.60999, ..., 1450.98315, 1455.07678, 1440.35339]], [[ 668.91016, 993.23151, 1001.28668, ..., 981.16144, 987.87903, 973.79382], [ 961.98376, 1434.27490, 1451.49512, ..., 1448.15161, 1452.31030, 1432.42224], [ 982.22821, 1432.54126, 1436.00085, ..., 1453.59583, 1459.09497, 1453.58435], ..., [ 973.34589, 1463.60559, 1448.30957, ..., 1450.97546, 1454.07642, 1456.61646], [ 980.44824, 1459.36462, 1456.19910, ..., 1447.03357, 1442.95508, 1435.65796], [ 963.78809, 1444.17712, 1443.89441, ..., 1457.01367, 1445.02148, 1457.99207]], [[ 655.03345, 982.13409, 977.45831, ..., 959.81238, 965.09094, 954.29327], [ 958.10065, 1433.69287, 1450.17883, ..., 1449.29041, 1434.92041, 1425.00916], [ 955.98669, 1436.07666, 1428.81812, ..., 1443.78564, 1440.16931, 1445.34265], ..., [ 973.61578, 1463.73743, 1451.56262, ..., 1428.18945, 1458.73645, 1445.85693], [ 967.10413, 1458.06189, 1435.34521, ..., 1454.37549, 1435.67297, 1426.90417], [ 954.31757, 1440.58350, 1439.67480, ..., 1457.88232, 1434.95972, 1441.88696]]]]), compiled_model=tensor([[[[ 648., 968., 972., ..., 964., 960., 940.], [ 952., 1560., 1600., ..., 1560., 1536., 1584.], [ 988., 1584., 1568., ..., 1608., 1536., 1672.], ..., [ 964., 1648., 1568., ..., 1560., 1632., 1624.], [ 996., 1632., 1616., ..., 1576., 1624., 1584.], [ 972., 1544., 1584., ..., 1584., 1584., 1544.]], [[ 684., 1000., 980., ..., 988., 956., 976.], [ 992., 1584., 1608., ..., 1608., 1616., 1576.], [ 988., 1544., 1568., ..., 1600., 1592., 1616.], ..., [ 988., 1632., 1568., ..., 1584., 1632., 1560.], [1016., 1632., 1568., ..., 1520., 1592., 1544.], [ 988., 1584., 1600., ..., 1608., 1520., 1600.]], [[ 636., 968., 972., ..., 964., 956., 964.], [ 964., 1512., 1576., ..., 1592., 1552., 1528.], [ 956., 1576., 1592., ..., 1608., 1496., 1576.], ..., [ 960., 1616., 1568., ..., 1504., 1576., 1568.], [ 960., 1560., 1536., ..., 1520., 1544., 1536.], [ 984., 1560., 1544., ..., 1528., 1600., 1520.]], ..., [[ 660., 988., 996., ..., 968., 968., 972.], [ 972., 1552., 1584., ..., 1640., 1544., 1520.], [ 992., 1552., 1560., ..., 1592., 1552., 1616.], ..., [ 964., 1688., 1568., ..., 1512., 1608., 1608.], [ 968., 1616., 1592., ..., 1656., 1600., 1584.], [ 976., 1536., 1624., ..., 1592., 1624., 1536.]], [[ 668., 988., 984., ..., 988., 980., 980.], [ 984., 1568., 1616., ..., 1592., 1640., 1584.], [1000., 1576., 1616., ..., 1632., 1640., 1624.], ..., [ 968., 1576., 1576., ..., 1584., 1608., 1680.], [ 964., 1640., 1640., ..., 1600., 1576., 1616.], [ 968., 1544., 1520., ..., 1568., 1560., 1616.]], [[ 652., 984., 988., ..., 956., 964., 952.], [ 964., 1584., 1648., ..., 1560., 1536., 1576.], [ 972., 1512., 1552., ..., 1552., 1560., 1592.], ..., [ 968., 1648., 1616., ..., 1536., 1568., 1600.], [ 972., 1600., 1544., ..., 1640., 1584., 1568.], [ 964., 1592., 1552., ..., 1616., 1576., 1544.]]]])

test_conv2d.test_module[Conv2D821-[((1, 3, 480, 480), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 4581 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 197632 and static circular buffer region ends at 570656 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D848-[((1, 3, 480, 480), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5019 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 197632 and static circular buffer region ends at 570656 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D849-[((1, 48, 320, 320), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5301 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 413440 and static circular buffer region ends at 525728 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D889-[((1, 3, 448, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5350 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 194432 and static circular buffer region ends at 673056 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D929-[((1, 3, 448, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5557 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 194432 and static circular buffer region ends at 673056 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D939-[((1, 3, 448, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5603 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 194432 and static circular buffer region ends at 673056 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D139-[((1, 64, 320, 320), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5718 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 413440 and static circular buffer region ends at 525728 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f0f346e82d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f0f256ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f0f256ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f0f256ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f0f256c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f0f2410869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f0f256d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f0f256d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f0f23540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f0f234d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f0f35289365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f0f356f8c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f0f356f8b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f0f354c3b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x557a67fefb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x557a67fdfa97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x557a67fe5574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x557a67ffe172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x557a67fda97f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x557a67ffe10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x557a67fda790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x557a67ffd4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x557a67fdf702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x557a67fe561d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x557a67ffa62c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x557a68102464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x557a67fe639b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x557a67fe099e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x557a67fdcc30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x557a67ff038c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x557a67fdb9ab]

test_conv2d.test_module[Conv2D399-[((1, 512, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 525.58276, 783.88684, 767.01471, ..., 747.95160, 752.32001, 757.83569], [ 762.52173, 1110.55945, 1142.11560, ..., 1136.46985, 1143.87231, 1126.66980], [ 780.39893, 1125.81018, 1118.03625, ..., 1141.56079, 1144.62622, 1140.71118], ..., [ 771.29236, 1134.47302, 1131.57935, ..., 1138.23889, 1146.53552, 1144.39001], [ 769.11322, 1151.86389, 1148.58948, ..., 1131.17419, 1131.81787, 1130.34241], [ 775.21332, 1141.98694, 1136.81982, ..., 1139.86963, 1143.07532, 1136.47681]], [[ 516.03595, 789.81433, 775.77338, ..., 753.84271, 767.01599, 762.30988], [ 754.72107, 1131.58911, 1143.27698, ..., 1145.83484, 1138.32495, 1135.55969], [ 753.56519, 1139.67871, 1125.88721, ..., 1142.15515, 1156.83044, 1149.36304], ..., [ 764.95056, 1153.46057, 1128.09399, ..., 1146.77942, 1160.69263, 1141.53003], [ 769.73193, 1141.04224, 1135.28735, ..., 1162.66223, 1151.39014, 1136.54700], [ 760.14246, 1151.27722, 1134.78479, ..., 1144.05310, 1136.15649, 1154.15125]], [[ 522.05524, 777.02203, 773.66803, ..., 764.02057, 760.96289, 769.25470], [ 755.60187, 1131.53906, 1153.67908, ..., 1153.92371, 1161.03857, 1142.49963], [ 775.55756, 1124.52783, 1137.35657, ..., 1150.89075, 1152.41028, 1160.92358], ..., [ 770.83282, 1158.07800, 1147.39258, ..., 1151.31799, 1159.88403, 1140.70044], [ 775.80658, 1166.96313, 1144.59558, ..., 1153.68384, 1154.11975, 1145.31604], [ 760.92365, 1141.41113, 1150.65430, ..., 1164.11572, 1145.93481, 1147.65247]], ..., [[ 524.28925, 773.42096, 778.15985, ..., 753.24860, 759.48956, 761.88159], [ 763.58807, 1124.02271, 1147.64612, ..., 1154.76819, 1141.45557, 1130.31274], [ 767.89008, 1146.89917, 1134.68750, ..., 1144.67358, 1149.83337, 1146.93298], ..., [ 768.47308, 1149.06653, 1135.96045, ..., 1151.35217, 1154.21301, 1146.59668], [ 766.45184, 1150.49646, 1149.71973, ..., 1152.47827, 1134.81653, 1129.82178], [ 770.19177, 1145.70740, 1145.77893, ..., 1152.69653, 1135.58569, 1141.84790]], [[ 521.36279, 776.71674, 772.07373, ..., 761.40552, 755.14825, 771.35126], [ 760.94727, 1133.98010, 1152.92322, ..., 1151.31909, 1153.29932, 1145.64856], [ 775.52582, 1140.75122, 1127.19128, ..., 1139.82471, 1154.77820, 1160.83411], ..., [ 765.88013, 1152.94482, 1159.70813, ..., 1149.05566, 1167.97424, 1154.80627], [ 774.93121, 1167.23962, 1143.86523, ..., 1164.95276, 1143.16602, 1143.59473], [ 768.60107, 1144.30859, 1157.72974, ..., 1152.53735, 1155.10754, 1157.78088]], [[ 512.85712, 765.04388, 765.70593, ..., 744.70300, 742.31232, 756.23541], [ 763.25568, 1125.87183, 1134.33350, ..., 1136.58264, 1151.26172, 1136.45544], [ 761.45605, 1141.16516, 1117.81067, ..., 1138.96289, 1137.87207, 1144.58289], ..., [ 772.23944, 1144.58484, 1132.89600, ..., 1144.84570, 1156.40356, 1149.44580], [ 775.15100, 1146.15503, 1145.37903, ..., 1148.91699, 1136.15063, 1129.10913], [ 772.09875, 1145.13770, 1141.86743, ..., 1150.41687, 1137.00049, 1145.73853]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 532., 780., 776., ..., 756., 752., 756.], [ 764., 1160., 1200., ..., 1184., 1184., 1176.], [ 780., 1192., 1176., ..., 1176., 1176., 1224.], ..., [ 780., 1216., 1176., ..., 1192., 1216., 1200.], [ 764., 1200., 1176., ..., 1200., 1192., 1192.], [ 776., 1184., 1208., ..., 1184., 1184., 1192.]], [[ 508., 788., 768., ..., 756., 764., 756.], [ 756., 1176., 1200., ..., 1192., 1208., 1184.], [ 756., 1184., 1176., ..., 1208., 1168., 1208.], ..., [ 760., 1200., 1120., ..., 1152., 1216., 1216.], [ 768., 1176., 1192., ..., 1224., 1224., 1200.], [ 756., 1208., 1192., ..., 1192., 1168., 1216.]], [[ 516., 780., 784., ..., 772., 772., 772.], [ 764., 1184., 1248., ..., 1208., 1232., 1208.], [ 792., 1160., 1184., ..., 1168., 1192., 1232.], ..., [ 768., 1240., 1168., ..., 1184., 1232., 1200.], [ 776., 1264., 1168., ..., 1192., 1232., 1216.], [ 772., 1208., 1216., ..., 1224., 1176., 1208.]], ..., [[ 524., 772., 780., ..., 756., 756., 768.], [ 768., 1208., 1224., ..., 1224., 1240., 1216.], [ 792., 1176., 1200., ..., 1192., 1224., 1216.], ..., [ 764., 1216., 1184., ..., 1208., 1240., 1224.], [ 760., 1224., 1232., ..., 1208., 1200., 1192.], [ 776., 1176., 1184., ..., 1224., 1168., 1176.]], [[ 524., 792., 780., ..., 756., 760., 768.], [ 768., 1200., 1200., ..., 1184., 1200., 1176.], [ 792., 1160., 1168., ..., 1184., 1176., 1224.], ..., [ 760., 1208., 1176., ..., 1168., 1184., 1200.], [ 772., 1200., 1168., ..., 1208., 1216., 1200.], [ 772., 1160., 1192., ..., 1200., 1224., 1208.]], [[ 512., 768., 772., ..., 756., 744., 768.], [ 752., 1120., 1216., ..., 1128., 1192., 1152.], [ 772., 1176., 1152., ..., 1168., 1192., 1232.], ..., [ 768., 1176., 1200., ..., 1224., 1200., 1200.], [ 764., 1192., 1192., ..., 1200., 1192., 1168.], [ 776., 1192., 1192., ..., 1176., 1184., 1200.]]]])

test_conv2d.test_module[Conv2D159-[((1, 512, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 525.58276, 783.88684, 767.01471, ..., 747.95160, 752.32001, 757.83569], [ 762.52173, 1110.55945, 1142.11560, ..., 1136.46985, 1143.87231, 1126.66980], [ 780.39893, 1125.81018, 1118.03625, ..., 1141.56079, 1144.62622, 1140.71118], ..., [ 771.29236, 1134.47302, 1131.57935, ..., 1138.23889, 1146.53552, 1144.39001], [ 769.11322, 1151.86389, 1148.58948, ..., 1131.17419, 1131.81787, 1130.34241], [ 775.21332, 1141.98694, 1136.81982, ..., 1139.86963, 1143.07532, 1136.47681]], [[ 516.03595, 789.81433, 775.77338, ..., 753.84271, 767.01599, 762.30988], [ 754.72107, 1131.58911, 1143.27698, ..., 1145.83484, 1138.32495, 1135.55969], [ 753.56519, 1139.67871, 1125.88721, ..., 1142.15515, 1156.83044, 1149.36304], ..., [ 764.95056, 1153.46057, 1128.09399, ..., 1146.77942, 1160.69263, 1141.53003], [ 769.73193, 1141.04224, 1135.28735, ..., 1162.66223, 1151.39014, 1136.54700], [ 760.14246, 1151.27722, 1134.78479, ..., 1144.05310, 1136.15649, 1154.15125]], [[ 522.05524, 777.02203, 773.66803, ..., 764.02057, 760.96289, 769.25470], [ 755.60187, 1131.53906, 1153.67908, ..., 1153.92371, 1161.03857, 1142.49963], [ 775.55756, 1124.52783, 1137.35657, ..., 1150.89075, 1152.41028, 1160.92358], ..., [ 770.83282, 1158.07800, 1147.39258, ..., 1151.31799, 1159.88403, 1140.70044], [ 775.80658, 1166.96313, 1144.59558, ..., 1153.68384, 1154.11975, 1145.31604], [ 760.92365, 1141.41113, 1150.65430, ..., 1164.11572, 1145.93481, 1147.65247]], ..., [[ 528.64368, 773.66937, 778.95483, ..., 772.82831, 761.90387, 774.11841], [ 772.59845, 1128.30725, 1161.01685, ..., 1169.52246, 1148.77051, 1146.79797], [ 790.23505, 1148.09131, 1134.84937, ..., 1152.96106, 1162.42310, 1170.11462], ..., [ 788.11346, 1161.39001, 1149.57300, ..., 1162.04175, 1162.23352, 1150.84875], [ 785.71887, 1168.99219, 1159.71558, ..., 1163.72424, 1142.67944, 1153.45093], [ 786.14825, 1149.93994, 1153.85364, ..., 1162.82227, 1163.47241, 1146.57129]], [[ 532.67249, 782.38086, 775.62262, ..., 764.42963, 768.78699, 776.45020], [ 763.90143, 1132.81262, 1153.61377, ..., 1172.53491, 1164.58057, 1158.07520], [ 785.24744, 1153.60120, 1133.48157, ..., 1154.87122, 1169.96362, 1168.38586], ..., [ 785.51685, 1166.79614, 1160.76868, ..., 1167.79150, 1172.04199, 1158.51062], [ 777.36432, 1163.40637, 1146.63342, ..., 1157.59607, 1146.73499, 1146.42468], [ 778.84222, 1163.23486, 1168.50488, ..., 1158.48047, 1162.45703, 1165.56970]], [[ 533.25543, 791.36407, 775.97815, ..., 765.47870, 759.37311, 773.64948], [ 763.78577, 1145.57678, 1176.15991, ..., 1160.25195, 1162.30164, 1147.40271], [ 782.22119, 1149.56348, 1141.39417, ..., 1158.28601, 1169.52539, 1169.26587], ..., [ 782.75970, 1176.20630, 1155.03723, ..., 1163.04260, 1178.67480, 1144.68689], [ 778.50073, 1169.29407, 1151.90942, ..., 1155.39795, 1149.12439, 1144.96790], [ 775.44983, 1166.51880, 1159.29382, ..., 1161.24951, 1159.92651, 1155.28784]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 532., 780., 776., ..., 756., 752., 756.], [ 764., 1160., 1200., ..., 1184., 1184., 1176.], [ 780., 1192., 1176., ..., 1176., 1176., 1224.], ..., [ 780., 1216., 1176., ..., 1192., 1216., 1200.], [ 764., 1200., 1176., ..., 1200., 1192., 1192.], [ 776., 1184., 1208., ..., 1184., 1184., 1192.]], [[ 508., 788., 768., ..., 756., 764., 756.], [ 756., 1176., 1200., ..., 1192., 1208., 1184.], [ 756., 1184., 1176., ..., 1208., 1168., 1208.], ..., [ 760., 1200., 1120., ..., 1152., 1216., 1216.], [ 768., 1176., 1192., ..., 1224., 1224., 1200.], [ 756., 1208., 1192., ..., 1192., 1168., 1216.]], [[ 516., 780., 784., ..., 772., 772., 772.], [ 764., 1184., 1248., ..., 1208., 1232., 1208.], [ 792., 1160., 1184., ..., 1168., 1192., 1232.], ..., [ 768., 1240., 1168., ..., 1184., 1232., 1200.], [ 776., 1264., 1168., ..., 1192., 1232., 1216.], [ 772., 1208., 1216., ..., 1224., 1176., 1208.]], ..., [[ 528., 768., 792., ..., 776., 760., 764.], [ 780., 1200., 1232., ..., 1224., 1216., 1200.], [ 788., 1184., 1144., ..., 1208., 1200., 1240.], ..., [ 780., 1224., 1224., ..., 1200., 1216., 1184.], [ 780., 1264., 1200., ..., 1216., 1216., 1240.], [ 784., 1208., 1176., ..., 1232., 1216., 1160.]], [[ 528., 776., 772., ..., 772., 768., 776.], [ 776., 1192., 1240., ..., 1272., 1240., 1184.], [ 796., 1160., 1192., ..., 1232., 1248., 1232.], ..., [ 776., 1240., 1216., ..., 1248., 1224., 1232.], [ 784., 1240., 1200., ..., 1184., 1224., 1200.], [ 784., 1176., 1240., ..., 1192., 1232., 1232.]], [[ 528., 796., 776., ..., 760., 768., 768.], [ 764., 1200., 1272., ..., 1232., 1216., 1184.], [ 788., 1176., 1184., ..., 1176., 1232., 1200.], ..., [ 776., 1216., 1216., ..., 1248., 1232., 1184.], [ 780., 1264., 1192., ..., 1224., 1216., 1216.], [ 780., 1216., 1224., ..., 1224., 1216., 1200.]]]])

test_conv2d.test_module[Conv2D504-[((1, 320, 80, 80), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[333.50986, 501.99402, 497.81204, ..., 470.21515, 472.69595, 310.37115], [497.78760, 743.01636, 741.73145, ..., 724.94318, 706.53790, 477.33203], [486.65338, 738.22070, 724.52576, ..., 726.78516, 711.10938, 480.39633], ..., [496.05402, 739.73090, 739.92847, ..., 728.57306, 726.03973, 489.85434], [486.84549, 725.89551, 728.93378, ..., 717.11389, 716.21863, 477.68536], [322.32516, 487.65140, 484.04919, ..., 471.07660, 479.69397, 322.18826]], [[335.19470, 504.80518, 496.40115, ..., 482.52481, 478.11169, 312.90536], [500.37119, 752.85364, 749.09808, ..., 722.85492, 715.60107, 469.46146], [491.25797, 730.76794, 725.42505, ..., 724.75500, 721.60913, 480.79236], ..., [491.86169, 735.29169, 738.47803, ..., 721.91022, 725.73407, 478.32602], [484.54733, 722.33154, 729.90485, ..., 711.70276, 723.39221, 476.51852], [313.52606, 472.70276, 472.48981, ..., 471.37308, 470.55554, 318.01184]], [[332.32144, 490.16495, 494.19733, ..., 475.77005, 461.85623, 301.73404], [490.64056, 753.51471, 740.50562, ..., 714.66772, 707.76190, 478.36047], [492.24588, 733.00262, 725.35724, ..., 731.03998, 715.52252, 478.62598], ..., [486.02676, 724.24481, 730.06653, ..., 728.72455, 728.11664, 481.56750], [479.64694, 727.07501, 715.51453, ..., 714.60773, 712.24921, 478.98654], [321.41098, 483.17578, 485.14056, ..., 469.44391, 483.50629, 325.64276]], ..., [[331.24127, 488.39984, 487.12714, ..., 472.40857, 452.01074, 294.53677], [490.50079, 730.39343, 734.51355, ..., 703.29675, 701.43158, 455.04327], [485.58350, 723.07428, 715.72668, ..., 709.00592, 707.90350, 475.43378], ..., [483.27151, 720.39642, 720.43622, ..., 712.83807, 717.96826, 474.91553], [476.49823, 712.42950, 707.54401, ..., 704.83051, 707.06122, 476.09183], [321.54456, 472.42694, 471.37335, ..., 462.67670, 472.41681, 314.82278]], [[329.12512, 502.76529, 499.32309, ..., 473.48044, 467.35510, 305.96417], [490.88882, 736.76202, 736.60529, ..., 717.03168, 706.93286, 469.78882], [483.53375, 725.00305, 726.50287, ..., 718.71991, 711.96558, 474.84201], ..., [482.78787, 727.01904, 723.55890, ..., 721.08252, 714.72766, 476.87259], [478.30353, 719.32819, 725.39581, ..., 711.51117, 711.25623, 470.06979], [310.23367, 479.64481, 468.33197, ..., 465.11053, 473.37595, 318.57571]], [[333.33136, 490.68713, 498.42355, ..., 469.53503, 465.04126, 304.37723], [486.51740, 728.92334, 730.98248, ..., 707.07098, 701.49811, 452.24271], [482.81769, 715.58856, 706.99646, ..., 706.65820, 712.98761, 460.18253], ..., [484.16760, 715.09961, 728.18616, ..., 711.62433, 713.76569, 474.50848], [480.25333, 708.88177, 713.92297, ..., 700.36182, 700.77222, 462.96228], [313.03839, 478.52176, 472.68509, ..., 461.70474, 474.50757, 310.13748]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[328., 500., 498., ..., 470., 472., 308.], [498., 748., 740., ..., 728., 708., 468.], [486., 740., 736., ..., 736., 708., 474.], ..., [488., 740., 728., ..., 740., 732., 480.], [488., 708., 728., ..., 712., 724., 474.], [318., 486., 488., ..., 460., 480., 316.]], [[338., 506., 492., ..., 484., 472., 312.], [502., 756., 736., ..., 724., 708., 466.], [486., 740., 744., ..., 724., 728., 472.], ..., [490., 736., 740., ..., 740., 736., 478.], [474., 732., 732., ..., 724., 724., 480.], [312., 480., 468., ..., 464., 466., 312.]], [[328., 496., 490., ..., 476., 466., 304.], [498., 740., 732., ..., 720., 696., 476.], [492., 728., 720., ..., 724., 732., 476.], ..., [478., 720., 728., ..., 728., 732., 484.], [476., 724., 720., ..., 708., 724., 470.], [318., 478., 476., ..., 468., 490., 320.]], ..., [[330., 488., 482., ..., 468., 450., 294.], [482., 740., 740., ..., 716., 696., 452.], [486., 720., 728., ..., 716., 712., 474.], ..., [484., 724., 728., ..., 720., 728., 476.], [474., 724., 716., ..., 716., 720., 474.], [318., 468., 472., ..., 462., 474., 314.]], [[324., 496., 488., ..., 476., 464., 306.], [494., 760., 744., ..., 724., 712., 482.], [480., 728., 740., ..., 724., 716., 468.], ..., [478., 740., 728., ..., 724., 716., 478.], [476., 736., 732., ..., 728., 712., 470.], [306., 474., 460., ..., 466., 472., 314.]], [[326., 490., 498., ..., 466., 458., 306.], [486., 732., 732., ..., 704., 704., 446.], [484., 708., 716., ..., 712., 712., 460.], ..., [470., 716., 724., ..., 704., 712., 466.], [488., 708., 708., ..., 704., 708., 458.], [312., 470., 460., ..., 458., 466., 302.]]]])

test_add.test_module[Add46-[((1, 32), torch.int64)]]: forge/test/models_ops/test_add.py#L18561

TypeError: Dtype mismatch: framework_model.dtype=torch.int64, compiled_model.dtype=torch.int32

test_add.test_module[Add1-[((1, 960, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.63892]], [[0.91089]], [[0.23114]], [[0.27469]], [[0.45009]], [[0.77674]], [[0.63276]], [[1.03911]], [[0.59829]], [[0.77497]], [[0.49156]], [[0.54438]], [[0.16499]], [[0.31152]], [[0.43655]], [[0.66119]], [[0.84033]], [[0.94267]], [[0.30369]], [[0.42493]], [[0.82427]], [[1.05786]], [[0.53976]], [[1.01682]], [[0.56207]], [[0.69557]], [[1.09540]], [[0.17883]], [[0.32789]], [[0.51608]], [[0.44776]], [[1.07466]], [[0.31857]], [[0.41250]], [[0.29334]], [[0.17438]], [[0.35079]], [[1.07246]], [[0.86577]], [[0.88500]], [[0.66896]], [[0.38632]], [[0.72726]], [[0.17582]], [[0.28138]], [[0.38490]], [[0.95813]], [[0.93582]], [[0.42092]], [[0.62462]], [[0.96244]], [[1.13973]], [[0.84110]], [[0.71021]], [[0.97791]], [[0.34826]], [[0.73584]], [[0.25501]], [[0.29612]], [[0.38437]], [[0.86890]], [[0.84374]], [[0.34649]], [[0.79372]], [[0.91715]], [[0.57955]], [[0.66175]], [[0.75852]], [[0.95285]], [[1.12276]], [[0.25735]], [[0.45943]], [[0.83917]], [[1.05694]], [[1.07777]], [[1.08384]], [[0.74217]], [[0.20787]], [[0.68866]], [[0.32986]], [[0.17669]], [[1.08691]], [[1.02284]], [[0.14390]], [[0.73625]], [[0.55843]], [[0.56038]], [[0.41379]], [[0.83494]], [[0.34651]], [[0.82596]], [[0.89552]], [[1.00060]], [[0.82962]], [[0.14780]], [[0.31832]], [[0.89232]], [[0.74731]], [[0.25262]], [[0.35475]], [[1.11304]], [[0.97957]], [[0.42465]], [[0.51682]], [[0.16636]], [[0.63368]], [[0.26613]], [[0.25699]], [[0.61511]], [[0.71774]], [[0.43790]], [[0.93935]], [[0.33839]], [[1.09635]], [[0.98531]], [[0.22102]], [[0.51822]], [[0.66522]], [[0.71561]], [[0.76125]], [[0.83888]], [[0.67261]], [[0.39870]], [[0.87926]], [[0.16304]], [[0.34631]], [[0.51750]], [[0.39911]], [[0.46775]], [[0.23285]], [[0.53631]], [[0.74954]], [[0.31693]], [[0.61700]], [[1.00059]], [[0.59126]], [[0.65656]], [[0.59953]], [[0.74385]], [[0.96058]], [[1.11629]], [[0.96019]], [[1.11737]], [[0.60650]], [[0.19350]], [[0.40562]], [[0.98312]], [[0.63942]], [[0.39414]], [[0.25951]], [[0.17474]], [[0.22066]], [[0.54125]], [[0.91687]], [[0.91298]], [[0.16045]], [[0.95455]], [[0.25141]], [[0.53696]], [[0.43993]], [[0.54636]], [[0.54449]], [[0.19399]], [[0.21094]], [[0.56442]], [[0.64913]], [[0.41553]], [[0.83101]], [[0.19263]], [[0.60892]], [[1.08237]], [[0.43872]], [[1.09417]], [[0.82374]], [[0.19143]], [[0.95901]], [[0.58497]], [[0.41946]], [[1.04249]], [[0.23862]], [[0.69632]], [[0.53798]], [[0.99972]], [[0.78224]], [[0.88292]], [[0.81924]], [[0.52243]], [[0.53751]], [[0.23062]], [[0.91359]], [[1.03965]], [[0.98478]], [[0.28997]], [[0.66496]], [[0.29020]], [[0.36742]], [[0.35131]], [[0.81354]], [[0.34471]], [[0.63175]], [[0.66370]], [[0.96498]], [[0.26470]], [[0.29941]], [[0.35233]], [[0.99263]], [[0.46293]], [[1.06441]], [[0.82347]], [[0.70598]], [[0.63894]], [[0.54382]], [[0.70540]], [[0.52849]], [[0.63915]], [[0.70646]], [[0.25156]], [[0.38060]], [[1.04641]], [[0.23689]], [[0.60676]], [[1.13728]], [[0.82328]], [[0.65682]], [[0.20936]], [[0.89035]], [[0.28652]], [[0.50073]], [[0.47491]], [[0.56862]], [[0.64813]], [[1.05507]], [[0.70508]], [[1.09051]], [[0.94852]], [[0.32656]], [[0.86692]], [[0.28922]], [[0.43075]], [[0.78972]], [[0.80776]], [[1.01778]], [[0.48171]], [[0.64346]], [[0.90008]], [[0.15912]], [[1.00415]], [[0.22920]], [[0.64955]], [[0.55766]], [[0.37933]], [[0.70875]], [[1.05612]], [[0.49650]], [[0.34582]], [[0.45774]], [[0.14709]], [[0.86836]], [[0.40253]], [[0.30899]], [[0.35461]], [[0.93014]], [[0.90745]], [[1.02642]], [[0.82403]], [[0.47568]], [[0.50294]], [[0.79038]], [[1.05368]], [[0.77861]], [[0.40609]], [[0.40762]], [[0.16993]], [[0.75069]], [[0.36207]], [[0.19688]], [[1.08110]], [[0.31796]], [[0.58578]], [[0.78591]], [[0.65859]], [[0.30622]], [[0.23850]], [[1.04120]], [[0.72408]], [[1.05748]], [[0.47504]], [[0.78994]], [[0.52833]], [[0.62043]], [[0.33814]], [[0.81176]], [[0.80075]], [[0.63235]], [[0.53021]], [[0.33445]], [[0.98844]], [[0.27045]], [[0.84750]], [[0.47454]], [[0.40143]], [[0.73249]], [[0.38294]], [[0.75787]], [[0.74085]], [[0.27142]], [[0.72591]], [[0.85563]], [[0.84058]], [[0.57972]], [[0.23277]], [[0.56559]], [[0.81632]], [[0.46023]], [[0.83245]], [[0.97564]], [[0.38159]], [[0.64759]], [[0.84941]], [[0.68185]], [[0.68443]], [[0.70510]], [[0.24959]], [[0.68196]], [[0.98890]], [[1.09322]], [[0.93654]], [[0.70967]], [[0.87617]], [[0.39942]], [[0.22832]], [[0.21270]], [[1.14147]], [[0.96006]], [[0.29705]], [[0.83828]], [[1.02025]], [[1.14247]], [[1.07986]], [[1.03004]], [[0.52803]], [[0.46719]], [[1.05319]], [[0.92285]], [[0.34177]], [[1.09217]], [[0.88424]], [[0.91523]], [[0.32928]], [[0.78611]], [[0.46738]], [[1.03334]], [[0.55269]], [[0.83732]], [[0.73147]], [[0.85540]], [[0.47275]], [[0.88645]], [[0.29343]], [[0.75559]], [[0.30437]], [[0.14939]], [[0.24114]], [[1.03740]], [[0.91318]], [[1.11177]], [[1.04322]], [[0.19614]], [[0.30145]], [[0.56187]], [[0.31795]], [[0.98986]], [[0.26466]], [[0.39871]], [[0.15962]], [[0.35879]], [[1.05390]], [[1.05204]], [[1.00058]], [[1.02873]], [[1.08726]], [[0.51463]], [[0.86266]], [[1.08813]], [[0.80807]], [[1.14250]], [[0.90200]], [[0.95349]], [[0.46767]], [[0.88258]], [[0.70012]], [[0.52325]], [[0.36081]], [[0.36211]], [[0.25792]], [[0.97833]], [[0.99813]], [[0.58576]], [[0.35332]], [[1.02912]], [[0.96241]], [[0.67983]], [[0.40660]], [[1.10219]], [[0.84713]], [[0.26309]], [[1.12121]], [[1.02235]], [[0.46042]], [[0.92374]], [[0.35857]], [[0.56431]], [[1.06721]], [[0.66332]], [[0.28905]], [[0.47555]], [[0.50694]], [[0.54620]], [[0.69051]], [[1.10508]], [[0.66943]], [[0.33394]], [[0.66829]], [[0.88241]], [[0.89068]], [[0.18569]], [[0.55319]], [[0.27109]], [[0.42932]], [[0.82281]], [[0.28760]], [[0.82853]], [[1.06705]], [[0.67546]], [[0.30942]], [[0.46352]], [[0.75185]], [[0.26150]], [[0.89107]], [[0.18873]], [[0.16202]], [[0.15683]], [[0.54123]], [[0.97888]], [[0.16942]], [[1.05826]], [[0.44265]], [[0.78911]], [[0.66547]], [[0.19180]], [[1.05733]], [[0.91189]], [[1.13966]], [[0.89527]], [[0.31263]], [[1.05996]], [[0.66954]], [[0.87977]], [[0.24175]], [[0.49885]], [[0.15172]], [[0.44792]], [[0.75053]], [[0.25008]], [[0.80205]], [[0.91107]], [[0.71232]], [[0.30812]], [[0.25500]], [[0.48841]], [[0.86214]], [[1.13586]], [[0.93018]], [[0.58636]], [[0.81797]], [[0.15213]], [[0.21561]], [[0.87597]], [[0.35946]], [[0.88321]], [[0.28970]], [[0.39501]], [[0.23082]], [[0.90358]], [[0.59172]], [[1.02746]], [[0.95210]], [[0.91933]], [[0.65874]], [[0.48807]], [[0.53395]], [[0.70911]], [[0.89052]], [[0.29237]], [[1.06230]], [[0.58830]], [[0.22369]], [[0.37213]], [[1.08507]], [[1.09993]], [[0.17952]], [[0.99531]], [[0.89324]], [[0.93862]], [[1.06593]], [[0.37319]], [[0.80055]], [[0.84728]], [[0.49492]], [[0.80999]], [[0.49881]], [[0.95179]], [[0.50394]], [[0.45627]], [[0.76854]], [[0.82001]], [[0.39838]], [[0.68686]], [[0.93243]], [[0.59292]], [[0.79483]], [[0.52207]], [[0.81791]], [[0.28047]], [[0.34865]], [[0.38887]], [[1.10217]], [[0.50812]], [[0.64130]], [[0.40042]], [[1.14181]], [[1.13100]], [[0.26557]], [[0.23733]], [[0.26367]], [[0.64025]], [[0.51521]], [[0.31539]], [[0.46333]], [[0.73713]], [[0.38142]], [[0.75345]], [[0.52801]], [[0.40038]], [[0.71136]], [[1.05379]], [[0.30462]], [[0.66588]], [[0.45828]], [[1.13332]], [[0.16828]], [[0.16333]], [[1.13536]], [[0.32632]], [[0.73853]], [[0.59951]], [[0.53734]], [[0.53099]], [[0.96038]], [[0.66656]], [[0.15586]], [[0.34748]], [[0.47221]], [[0.89427]], [[0.31910]], [[1.11412]], [[0.53130]], [[0.55288]], [[1.03444]], [[0.89397]], [[1.06672]], [[0.93189]], [[0.49099]], [[0.31093]], [[0.60546]], [[1.05648]], [[0.47485]], [[0.17898]], [[0.84763]], [[1.12940]], [[0.50032]], [[0.22865]], [[0.18913]], [[0.76796]], [[0.60480]], [[0.39017]], [[0.74373]], [[0.83254]], [[1.04032]], [[1.03085]], [[0.56782]], [[0.20178]], [[0.19085]], [[1.10951]], [[0.86369]], [[0.86062]], [[0.21005]], [[1.10567]], [[1.11634]], [[1.09409]], [[0.22087]], [[0.45400]], [[0.29878]], [[1.11615]], [[0.42782]], [[0.41439]], [[0.90462]], [[0.41137]], [[0.39640]], [[0.59893]], [[0.59460]], [[0.25317]], [[1.05950]], [[0.42210]], [[0.82002]], [[1.07756]], [[0.89482]], [[0.71345]], [[1.06809]], [[0.70988]], [[0.41134]], [[1.11566]], [[0.76100]], [[0.15482]], [[0.50032]], [[0.30208]], [[1.08104]], [[0.56009]], [[0.18690]], [[0.61123]], [[0.95667]], [[0.77256]], [[0.80077]], [[0.68910]], [[0.82907]], [[0.52084]], [[0.44376]], [[0.17531]], [[0.26600]], [[0.85936]], [[0.34661]], [[0.71440]], [[0.80226]], [[0.67807]], [[0.31849]], [[1.12083]], [[0.35191]], [[1.05391]], [[0.24491]], [[0.52239]], [[0.91465]], [[0.43837]], [[1.06269]], [[0.29859]], [[0.22278]], [[0.41720]], [[0.72349]], [[1.10304]], [[0.40396]], [[0.82148]], [[0.51731]], [[0.53423]], [[1.01032]], [[0.25517]], [[0.69577]], [[1.11287]], [[0.57396]], [[1.03088]], [[0.48869]], [[1.04514]], [[0.15900]], [[0.57060]], [[0.55484]], [[0.80471]], [[0.83883]], [[1.02657]], [[0.56817]], [[0.62284]], [[0.98507]], [[0.50738]], [[1.08098]], [[0.30976]], [[0.58856]], [[0.61581]], [[0.86577]], [[0.98450]], [[0.56341]], [[0.22842]], [[0.89041]], [[0.79223]], [[0.84351]], [[0.33423]], [[0.96446]], [[1.11624]], [[0.68605]], [[0.17564]], [[0.99366]], [[0.27194]], [[0.75760]], [[0.71532]], [[0.40865]], [[0.81676]], [[0.19545]], [[0.75654]], [[0.32569]], [[0.58860]], [[0.70698]], [[1.06864]], [[0.40410]], [[0.96298]], [[0.57915]], [[0.40520]], [[0.20726]], [[0.18390]], [[1.13097]], [[0.51796]], [[0.66764]], [[0.77822]], [[0.98253]], [[1.06940]], [[1.04817]], [[0.27222]], [[0.56256]], [[0.34679]], [[0.35699]], [[0.76127]], [[1.11199]], [[0.24211]], [[0.94527]], [[0.38343]], [[0.54528]], [[1.03955]], [[0.52958]], [[0.68819]], [[0.29315]], [[1.06828]], [[0.57803]], [[0.27697]], [[0.78907]], [[0.28715]], [[0.24591]], [[0.67312]], [[1.03909]], [[0.50117]], [[0.87800]], [[1.07232]], [[0.97429]], [[0.38038]], [[0.58785]], [[0.48532]], [[0.24060]], [[0.64287]], [[1.01888]], [[1.06392]], [[0.68931]], [[0.75623]], [[0.42620]], [[1.02008]], [[0.43461]], [[0.29534]], [[0.71966]], [[0.94236]], [[0.19187]], [[1.09465]], [[0.82253]], [[0.29236]], [[0.53496]], [[1.07645]], [[0.25905]], [[0.49652]], [[0.80665]], [[0.20462]], [[0.91676]], [[0.90294]], [[0.95277]], [[0.32389]], [[1.14066]], [[0.34628]], [[1.14183]], [[0.16282]], [[0.19718]], [[0.94976]], [[0.69492]], [[0.67151]], [[0.36578]], [[0.43293]], [[0.49647]], [[0.15559]], [[0.66865]], [[0.73109]], [[0.64224]], [[0.80413]], [[1.11709]], [[0.77561]], [[0.45963]], [[0.43689]], [[0.32276]], [[0.29605]], [[0.56214]], [[0.55423]], [[0.86509]], [[0.42895]], [[1.04127]], [[0.29182]], [[0.64408]], [[1.09212]], [[1.13986]], [[0.35303]], [[0.73173]], [[0.70172]], [[0.40824]], [[0.46992]], [[0.77809]], [[0.29498]], [[0.72516]], [[0.85903]], [[0.44563]], [[1.05798]], [[0.60976]], [[0.86952]], [[1.13782]], [[0.48983]], [[0.91437]], [[0.49965]], [[0.56963]], [[0.55793]], [[0.63956]], [[0.45378]], [[0.75986]], [[0.66150]], [[0.95961]], [[0.54146]], [[0.69280]], [[0.45667]], [[0.22394]], [[0.84500]], [[0.70664]], [[0.44242]], [[0.47361]], [[0.77342]], [[0.55226]], [[0.96940]], [[0.67099]], [[0.83153]], [[0.86051]], [[0.51998]], [[0.85900]], [[1.01163]], [[0.66567]], [[0.74051]], [[0.66083]], [[0.98885]], [[0.43157]], [[0.37688]], [[0.86054]], [[0.20751]], [[0.65068]], [[0.41272]], [[0.97268]], [[0.17714]], [[0.94583]], [[1.13734]], [[0.74383]], [[0.60935]], [[1.13484]], [[0.42899]], [[0.60260]], [[0.42371]], [[0.57364]], [[0.75133]], [[0.70944]], [[0.54823]], [[0.14274]], [[0.66827]], [[0.62785]], [[0.71788]], [[1.02579]], [[1.12867]], [[0.34602]], [[0.61149]], [[0.44233]], [[0.18184]], [[0.27883]], [[1.11194]], [[0.93419]], [[0.91124]], [[0.92054]], [[0.24542]], [[0.69517]], [[1.10802]], [[0.36362]], [[1.08746]], [[0.77408]], [[0.99444]], [[0.42847]], [[0.87317]], [[0.19942]], [[0.61092]], [[0.80944]], [[0.79261]], [[1.06107]], [[1.13399]], [[1.09754]], [[0.97836]], [[0.99419]], [[0.57815]], [[0.25464]], [[0.45705]], [[0.58653]], [[0.37247]], [[0.89825]], [[0.81818]], [[0.72334]], [[0.76344]], [[1.08189]], [[0.82478]], [[0.20419]], [[0.27972]], [[0.86304]], [[0.71080]], [[0.88651]], [[0.14330]], [[0.18123]], [[0.94928]], [[0.96249]], [[0.19031]], [[0.83246]], [[0.25224]], [[1.02132]], [[0.79956]], [[1.13706]], [[0.14967]], [[0.16133]], [[0.98071]], [[1.08658]], [[0.94900]], [[0.97508]], [[0.22153]], [[1.00457]], [[0.17059]], [[0.75436]], [[0.31515]], [[0.44407]], [[0.73120]], [[0.44607]], [[0.56355]], [[0.16875]], [[0.17123]], [[0.83635]], [[0.43102]], [[0.33847]], [[0.44537]], [[0.23388]], [[0.24296]], [[0.94185]], [[0.89678]], [[0.70927]], [[0.36953]], [[0.80901]], [[0.66513]]]]), compiled_model=tensor([[[[0.63892]], [[1.29882]], [[0.49350]], [[0.83278]], [[0.74303]], [[1.42040]], [[0.54786]], [[1.21987]], [[1.04638]], [[0.94241]], [[0.43261]], [[1.06539]], [[0.34175]], [[0.65214]], [[1.09878]], [[0.91014]], [[1.24015]], [[1.05467]], [[0.42193]], [[1.07003]], [[0.71882]], [[0.98713]], [[1.30135]], [[1.52465]], [[1.33472]], [[1.14415]], [[1.57736]], [[0.85277]], [[0.79247]], [[0.39235]], [[0.90907]], [[1.38731]], [[0.40640]], [[0.76379]], [[0.34507]], [[0.11601]], [[0.61376]], [[1.18062]], [[1.25476]], [[1.35445]], [[1.05687]], [[1.18716]], [[0.68224]], [[1.00057]], [[0.32980]], [[0.52588]], [[1.53831]], [[0.81598]], [[1.25788]], [[1.44398]], [[1.24623]], [[1.73743]], [[1.51978]], [[0.62306]], [[1.50995]], [[0.60109]], [[1.03224]], [[0.91927]], [[0.45619]], [[0.97961]], [[0.85944]], [[1.37970]], [[0.44294]], [[1.05823]], [[1.13329]], [[1.33011]], [[0.56058]], [[1.22892]], [[1.71962]], [[1.01303]], [[0.26901]], [[0.82950]], [[1.35561]], [[1.20802]], [[0.99623]], [[1.76255]], [[0.65453]], [[0.63334]], [[0.90919]], [[0.76226]], [[0.36906]], [[1.07915]], [[1.35970]], [[0.37952]], [[0.94159]], [[0.55944]], [[0.71931]], [[0.75848]], [[1.35729]], [[0.83931]], [[0.85038]], [[1.56948]], [[1.18280]], [[1.53290]], [[0.67201]], [[1.09163]], [[0.84874]], [[1.14579]], [[0.22115]], [[0.92762]], [[1.25806]], [[1.41296]], [[0.71316]], [[0.94223]], [[1.00340]], [[0.83750]], [[0.36766]], [[0.52421]], [[1.20452]], [[1.25872]], [[0.78155]], [[1.33873]], [[0.65418]], [[1.08624]], [[1.12396]], [[0.68596]], [[0.47971]], [[1.45781]], [[1.27131]], [[0.67078]], [[0.81678]], [[0.59905]], [[0.89246]], [[1.73440]], [[0.18464]], [[0.87103]], [[0.63621]], [[0.46761]], [[0.74255]], [[0.19125]], [[0.75777]], [[1.25434]], [[0.34510]], [[1.03219]], [[1.00120]], [[1.31380]], [[0.82147]], [[1.43630]], [[1.40221]], [[1.70493]], [[1.81793]], [[1.64968]], [[1.67082]], [[1.24509]], [[0.69303]], [[0.48699]], [[1.10538]], [[1.48653]], [[0.98513]], [[0.17011]], [[0.54354]], [[1.03353]], [[1.28970]], [[1.10733]], [[0.78732]], [[0.93908]], [[1.17456]], [[0.16548]], [[0.88216]], [[1.08250]], [[0.76102]], [[1.27584]], [[0.78252]], [[0.41454]], [[0.81835]], [[1.22153]], [[1.10438]], [[0.95533]], [[0.89635]], [[1.09258]], [[1.41304]], [[0.43574]], [[1.00406]], [[1.64100]], [[0.39975]], [[1.47194]], [[1.12367]], [[0.79337]], [[1.36885]], [[1.04564]], [[1.30175]], [[0.43912]], [[1.19547]], [[1.29259]], [[0.95621]], [[1.51957]], [[1.25826]], [[0.56578]], [[0.33703]], [[1.16494]], [[1.49890]], [[1.83534]], [[0.73686]], [[0.56537]], [[1.00039]], [[0.31600]], [[0.41711]], [[1.26355]], [[0.23767]], [[1.38533]], [[0.60536]], [[1.30317]], [[0.90588]], [[1.07809]], [[1.05935]], [[1.35245]], [[0.54932]], [[1.81297]], [[1.12923]], [[1.51979]], [[0.72588]], [[0.42064]], [[1.39798]], [[1.28660]], [[0.72919]], [[0.98463]], [[0.36172]], [[0.71952]], [[1.20048]], [[0.32953]], [[1.37692]], [[1.30730]], [[1.04383]], [[0.92509]], [[0.92748]], [[1.19623]], [[0.67238]], [[0.42897]], [[0.45206]], [[1.18107]], [[0.74548]], [[1.49515]], [[1.24295]], [[1.75396]], [[1.32894]], [[0.23186]], [[1.54950]], [[1.04367]], [[0.32978]], [[1.38714]], [[1.02114]], [[1.48863]], [[1.25473]], [[0.92164]], [[0.85925]], [[0.84713]], [[1.77806]], [[0.91809]], [[1.42733]], [[0.98392]], [[0.60292]], [[0.62210]], [[1.17587]], [[0.49209]], [[1.08279]], [[0.59885]], [[0.75954]], [[1.67911]], [[1.01265]], [[0.93677]], [[0.73465]], [[0.98346]], [[1.33025]], [[1.73595]], [[0.79254]], [[1.18887]], [[0.71313]], [[0.74319]], [[1.11769]], [[1.31529]], [[0.29863]], [[0.95805]], [[0.84219]], [[1.01898]], [[0.50224]], [[0.84901]], [[1.21124]], [[0.82824]], [[1.21948]], [[1.52252]], [[0.91994]], [[0.41634]], [[0.56280]], [[0.95458]], [[0.63515]], [[1.18508]], [[1.06333]], [[1.26175]], [[1.03975]], [[0.92450]], [[0.40941]], [[0.67749]], [[1.20503]], [[1.15905]], [[1.36103]], [[0.91700]], [[1.14005]], [[0.17500]], [[1.47979]], [[0.50188]], [[0.70160]], [[1.33696]], [[0.29265]], [[0.68791]], [[1.50803]], [[0.31235]], [[1.35227]], [[0.82218]], [[1.17854]], [[1.34655]], [[0.45707]], [[1.26342]], [[1.32516]], [[0.33221]], [[1.62820]], [[1.16487]], [[1.20918]], [[0.99486]], [[1.64611]], [[0.90828]], [[0.68356]], [[1.00618]], [[0.50768]], [[0.68803]], [[1.64654]], [[1.29952]], [[1.40939]], [[1.34680]], [[1.15436]], [[1.21753]], [[0.69753]], [[0.83475]], [[1.37046]], [[1.07616]], [[1.14581]], [[1.29634]], [[1.32886]], [[1.64495]], [[1.35951]], [[1.61376]], [[0.59567]], [[1.07440]], [[1.05402]], [[1.36810]], [[0.66054]], [[1.87608]], [[1.01103]], [[1.17619]], [[0.83368]], [[1.35939]], [[1.30958]], [[1.52938]], [[0.56748]], [[0.93025]], [[1.43545]], [[1.09888]], [[0.89252]], [[1.73264]], [[0.48457]], [[1.33721]], [[0.22403]], [[0.22649]], [[0.43728]], [[1.54740]], [[1.24229]], [[1.21749]], [[1.84880]], [[0.08379]], [[1.14380]], [[0.99393]], [[0.88514]], [[1.45636]], [[0.95120]], [[1.06157]], [[0.21777]], [[1.01741]], [[1.74998]], [[1.07712]], [[1.76488]], [[1.20264]], [[1.54913]], [[0.87966]], [[1.32642]], [[1.29546]], [[0.94568]], [[1.46224]], [[0.76242]], [[0.84569]], [[1.12429]], [[1.49443]], [[0.78024]], [[0.42633]], [[0.37489]], [[0.60762]], [[0.47517]], [[0.89680]], [[1.31314]], [[1.38169]], [[0.97821]], [[1.71309]], [[0.83474]], [[1.25975]], [[1.09757]], [[1.26388]], [[1.70247]], [[0.78124]], [[1.32609]], [[0.91725]], [[0.78275]], [[1.60928]], [[1.00330]], [[0.79400]], [[1.51929]], [[0.65718]], [[0.36679]], [[1.04271]], [[1.01886]], [[0.62848]], [[0.69711]], [[1.77151]], [[0.96411]], [[0.30229]], [[1.06428]], [[1.30542]], [[1.74582]], [[0.33806]], [[0.49689]], [[0.29455]], [[1.13935]], [[1.06960]], [[0.66515]], [[0.93036]], [[1.50262]], [[1.47967]], [[0.64138]], [[0.69434]], [[1.44030]], [[0.60911]], [[0.83632]], [[0.29898]], [[0.77968]], [[0.36963]], [[1.28155]], [[1.21004]], [[0.80902]], [[1.01883]], [[0.82094]], [[1.30556]], [[0.64719]], [[0.90480]], [[1.30337]], [[1.35126]], [[1.34861]], [[0.82529]], [[0.66406]], [[1.85878]], [[0.99859]], [[1.56704]], [[0.59460]], [[0.94590]], [[0.23885]], [[0.60350]], [[1.42975]], [[0.44441]], [[1.48434]], [[1.56642]], [[1.16417]], [[0.23289]], [[0.78518]], [[0.61031]], [[0.73331]], [[1.54725]], [[0.90710]], [[0.91693]], [[1.17418]], [[0.63705]], [[0.93712]], [[1.49250]], [[1.16177]], [[1.45195]], [[1.02079]], [[0.64672]], [[0.62503]], [[1.61674]], [[0.52762]], [[1.33169]], [[0.85111]], [[1.30156]], [[0.86203]], [[0.58626]], [[0.66259]], [[1.08088]], [[1.21716]], [[0.28858]], [[1.62917]], [[1.37504]], [[0.14291]], [[1.11934]], [[1.59646]], [[1.56828]], [[0.51175]], [[1.03762]], [[1.69195]], [[1.26996]], [[1.56342]], [[0.87019]], [[1.29766]], [[1.61640]], [[1.23730]], [[0.68500]], [[1.06536]], [[1.64709]], [[0.87744]], [[0.56983]], [[0.99689]], [[1.36619]], [[1.14518]], [[1.08773]], [[1.64722]], [[0.67466]], [[1.28971]], [[1.20358]], [[1.32022]], [[0.47578]], [[1.00757]], [[0.38255]], [[1.16862]], [[1.29800]], [[1.41077]], [[1.09502]], [[1.92521]], [[1.75039]], [[0.79993]], [[0.24239]], [[0.31464]], [[0.75036]], [[1.19157]], [[1.15237]], [[1.14309]], [[1.36120]], [[0.98527]], [[1.31732]], [[0.91380]], [[0.26781]], [[0.88064]], [[1.56716]], [[0.74662]], [[1.27483]], [[1.18249]], [[1.77478]], [[0.89954]], [[0.60378]], [[1.65337]], [[0.21280]], [[1.27487]], [[0.60577]], [[0.91151]], [[1.25114]], [[1.69803]], [[0.95149]], [[0.91830]], [[0.59190]], [[0.97992]], [[1.49129]], [[0.74424]], [[1.77651]], [[0.85707]], [[1.17660]], [[1.24045]], [[0.92436]], [[1.09429]], [[1.31424]], [[0.98511]], [[1.06472]], [[0.54600]], [[1.42986]], [[0.89561]], [[0.97192]], [[1.12682]], [[1.70812]], [[0.76844]], [[0.82234]], [[0.91902]], [[1.52543]], [[0.89844]], [[0.84374]], [[1.20736]], [[0.87866]], [[1.48251]], [[1.77494]], [[1.05114]], [[0.26908]], [[0.67283]], [[1.33804]], [[0.87478]], [[1.54153]], [[0.24982]], [[1.13517]], [[1.44323]], [[1.32949]], [[0.40508]], [[1.09165]], [[0.24414]], [[1.84955]], [[0.55171]], [[0.38816]], [[1.33349]], [[0.53877]], [[0.27829]], [[1.06676]], [[1.18486]], [[0.91309]], [[1.08962]], [[0.58802]], [[1.04073]], [[1.78096]], [[0.92855]], [[1.41823]], [[1.17815]], [[1.26042]], [[0.55701]], [[1.76725]], [[1.59767]], [[0.86905]], [[0.50893]], [[0.25263]], [[0.97900]], [[1.22168]], [[0.40560]], [[1.43510]], [[0.99939]], [[1.47740]], [[1.43743]], [[0.78976]], [[1.19867]], [[1.01928]], [[0.72781]], [[0.24950]], [[0.28631]], [[1.10071]], [[1.00971]], [[1.41773]], [[1.55387]], [[0.68459]], [[0.69862]], [[1.29754]], [[0.47986]], [[0.94438]], [[0.34572]], [[0.75850]], [[1.19998]], [[0.41845]], [[1.82140]], [[0.76957]], [[0.49032]], [[0.35804]], [[0.64382]], [[1.91614]], [[1.08527]], [[1.45927]], [[0.58212]], [[1.36535]], [[1.44415]], [[0.30811]], [[0.90938]], [[1.80908]], [[0.85176]], [[1.66835]], [[0.61912]], [[0.99509]], [[0.11381]], [[1.02507]], [[1.33135]], [[1.51554]], [[0.89049]], [[1.59683]], [[1.22411]], [[1.21699]], [[1.63209]], [[0.49223]], [[1.04631]], [[0.79066]], [[1.09891]], [[0.75150]], [[1.70261]], [[1.50873]], [[0.42831]], [[0.40591]], [[1.15351]], [[1.11810]], [[1.13116]], [[0.25022]], [[1.23262]], [[1.48236]], [[0.80128]], [[0.25932]], [[1.08043]], [[0.60006]], [[0.82064]], [[0.73454]], [[0.34479]], [[1.21250]], [[0.14496]], [[1.52641]], [[0.44676]], [[0.67222]], [[1.47151]], [[1.53698]], [[0.44640]], [[0.98595]], [[1.36710]], [[0.77464]], [[0.55833]], [[0.17382]], [[1.68034]], [[1.13377]], [[0.99877]], [[0.82274]], [[1.33619]], [[0.93147]], [[1.15685]], [[0.24102]], [[0.83288]], [[0.40707]], [[0.23953]], [[1.35948]], [[1.20531]], [[1.01181]], [[1.37422]], [[0.87635]], [[1.32186]], [[1.52575]], [[0.58320]], [[0.83732]], [[0.73907]], [[1.21321]], [[0.54028]], [[0.72978]], [[0.82580]], [[0.97111]], [[0.18157]], [[1.28711]], [[0.90128]], [[1.23878]], [[1.02277]], [[1.60812]], [[0.93775]], [[0.41733]], [[1.42238]], [[1.06159]], [[0.36709]], [[0.77814]], [[1.06254]], [[1.54472]], [[0.55955]], [[1.13876]], [[1.27566]], [[1.71678]], [[1.24596]], [[0.44173]], [[0.67107]], [[1.50749]], [[0.92191]], [[1.06426]], [[1.58613]], [[0.66960]], [[1.05352]], [[1.73425]], [[0.49460]], [[0.81235]], [[0.99991]], [[0.96478]], [[1.17201]], [[1.34816]], [[1.29084]], [[0.91982]], [[1.48474]], [[0.43509]], [[1.93096]], [[0.64303]], [[0.21073]], [[1.03470]], [[0.66278]], [[1.10413]], [[0.81406]], [[0.63149]], [[1.20555]], [[0.32483]], [[0.72070]], [[0.98883]], [[0.77995]], [[1.51864]], [[1.85191]], [[1.10680]], [[1.22449]], [[1.17114]], [[0.80826]], [[0.77585]], [[1.08703]], [[0.83070]], [[1.37940]], [[0.29217]], [[1.76736]], [[0.38074]], [[0.55606]], [[1.45082]], [[1.32123]], [[0.24864]], [[0.91910]], [[0.75106]], [[1.23510]], [[0.68055]], [[1.35695]], [[0.48899]], [[0.90457]], [[1.50626]], [[0.68769]], [[1.83524]], [[0.80252]], [[1.28045]], [[1.83270]], [[0.95774]], [[0.86404]], [[0.87589]], [[0.84828]], [[0.99316]], [[0.65771]], [[1.14580]], [[1.55876]], [[1.04585]], [[0.87453]], [[0.47990]], [[0.80364]], [[0.64325]], [[0.28112]], [[0.78347]], [[1.31511]], [[0.73945]], [[0.53065]], [[1.20022]], [[1.30104]], [[1.64694]], [[0.53374]], [[0.70173]], [[1.43385]], [[1.13530]], [[1.27904]], [[1.47666]], [[1.30268]], [[1.23595]], [[0.88181]], [[1.40837]], [[1.21598]], [[0.57159]], [[0.76431]], [[1.04912]], [[0.69556]], [[0.93877]], [[1.75584]], [[0.98359]], [[1.17245]], [[1.64348]], [[1.01300]], [[1.28860]], [[1.00964]], [[0.92336]], [[0.96998]], [[0.98470]], [[0.44501]], [[0.93723]], [[0.98080]], [[0.70173]], [[0.65807]], [[0.71195]], [[1.37940]], [[1.32106]], [[1.85725]], [[1.22037]], [[1.10545]], [[1.19842]], [[0.60830]], [[0.79734]], [[0.41649]], [[0.98251]], [[0.90415]], [[1.54713]], [[1.11878]], [[0.24996]], [[1.08537]], [[1.05721]], [[0.38194]], [[1.83487]], [[0.64212]], [[1.60116]], [[0.44318]], [[1.20649]], [[0.26200]], [[1.02308]], [[1.41727]], [[0.80702]], [[1.42153]], [[1.85107]], [[1.49090]], [[1.17310]], [[1.18372]], [[1.23783]], [[0.60008]], [[0.35404]], [[0.75654]], [[0.26949]], [[0.80725]], [[1.63791]], [[1.30634]], [[1.09007]], [[1.73699]], [[0.79919]], [[0.16476]], [[0.48857]], [[1.16740]], [[1.15022]], [[1.34987]], [[0.95919]], [[0.93905]], [[0.89693]], [[1.11725]], [[0.86270]], [[1.48351]], [[0.36462]], [[1.00432]], [[1.52678]], [[1.41020]], [[0.84661]], [[0.67853]], [[1.23177]], [[1.30925]], [[1.32541]], [[1.78232]], [[0.54021]], [[1.36861]], [[0.58218]], [[1.53097]], [[0.84247]], [[1.05528]], [[1.58038]], [[0.89509]], [[0.97397]], [[0.61329]], [[0.05572]], [[0.99675]], [[0.70496]], [[0.53220]], [[1.17733]], [[0.81254]], [[0.88666]], [[1.32872]], [[1.40004]], [[1.43683]], [[0.52372]], [[1.23875]], [[0.88491]]]])

test_add.test_module[Add1-[((1, 96, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1.24591]], [[1.51788]], [[0.83813]], [[0.88169]], [[1.05708]], [[1.38374]], [[1.23975]], [[1.64610]], [[1.20529]], [[1.38196]], [[1.09855]], [[1.15137]], [[0.77198]], [[0.91852]], [[1.04355]], [[1.26818]], [[1.44733]], [[1.54967]], [[0.91069]], [[1.03193]], [[1.43127]], [[1.66485]], [[1.14676]], [[1.62381]], [[1.16907]], [[1.30256]], [[1.70240]], [[0.78582]], [[0.93489]], [[1.12307]], [[1.05476]], [[1.68166]], [[0.92557]], [[1.01949]], [[0.90034]], [[0.78138]], [[0.95779]], [[1.67946]], [[1.47277]], [[1.49199]], [[1.27595]], [[0.99332]], [[1.33425]], [[0.78281]], [[0.88837]], [[0.99189]], [[1.56513]], [[1.54282]], [[1.02791]], [[1.23162]], [[1.56944]], [[1.74672]], [[1.44810]], [[1.31720]], [[1.58490]], [[0.95526]], [[1.34283]], [[0.86200]], [[0.90311]], [[0.99137]], [[1.47589]], [[1.45074]], [[0.95348]], [[1.40071]], [[1.52414]], [[1.18655]], [[1.26875]], [[1.36551]], [[1.55985]], [[1.72975]], [[0.86435]], [[1.06642]], [[1.44616]], [[1.66393]], [[1.68476]], [[1.69084]], [[1.34916]], [[0.81487]], [[1.29565]], [[0.93685]], [[0.78368]], [[1.69390]], [[1.62984]], [[0.75089]], [[1.34324]], [[1.16543]], [[1.16738]], [[1.02078]], [[1.44194]], [[0.95351]], [[1.43295]], [[1.50251]], [[1.60759]], [[1.43661]], [[0.75479]], [[0.92531]]]]), compiled_model=tensor([[[[1.24591]], [[1.03406]], [[0.56533]], [[0.33030]], [[0.31180]], [[1.13415]], [[0.93787]], [[1.46185]], [[1.29648]], [[1.53082]], [[0.88444]], [[1.02864]], [[0.22109]], [[1.16802]], [[0.93773]], [[0.80140]], [[1.23897]], [[1.09838]], [[1.14756]], [[1.07128]], [[0.78680]], [[1.10842]], [[1.23183]], [[1.22776]], [[1.00868]], [[0.83495]], [[1.27719]], [[0.95700]], [[0.53094]], [[1.05559]], [[0.35123]], [[1.44667]], [[0.22717]], [[1.07437]], [[0.98817]], [[0.23095]], [[0.33159]], [[1.46799]], [[1.04899]], [[1.08453]], [[1.32839]], [[1.23066]], [[0.62333]], [[0.26397]], [[0.57648]], [[1.08492]], [[0.86759]], [[0.81593]], [[0.48660]], [[1.27919]], [[0.97107]], [[1.84566]], [[1.05197]], [[0.64861]], [[1.49177]], [[0.74406]], [[1.54955]], [[0.64148]], [[0.32914]], [[0.33808]], [[0.77747]], [[0.98615]], [[1.04176]], [[1.02695]], [[1.42904]], [[1.34916]], [[1.03304]], [[1.05949]], [[0.83857]], [[1.95414]], [[0.58832]], [[0.49982]], [[0.70650]], [[1.07067]], [[1.83430]], [[1.04769]], [[1.53736]], [[1.03601]], [[1.07079]], [[0.39403]], [[0.66283]], [[1.84625]], [[1.22789]], [[0.45446]], [[1.14608]], [[1.12360]], [[0.67821]], [[0.52047]], [[1.41656]], [[0.49173]], [[0.93546]], [[1.01178]], [[1.48855]], [[1.39194]], [[0.19449]], [[0.33296]]]])

test_add.test_module[Add1-[((1, 1024, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.96122]], [[1.23319]], [[0.55344]], ..., [[0.65259]], [[0.67482]], [[1.18597]]]]), compiled_model=tensor([[[[0.96122]], [[1.67008]], [[1.40586]], ..., [[0.75013]], [[0.66981]], [[2.10869]]]])

test_advindex.test_module[Advindex2-[((2401,), torch.int64)]]: forge/test/models_ops/test_advindex.py#L233

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[0.77064, 0.60953, 0.43739], [0.84477, 0.58739, 0.50589], [0.06786, 0.52755, 0.44323], ..., [0.01251, 0.95886, 0.50927], [0.86357, 0.06145, 0.40740], [0.13803, 0.99280, 0.50541]], grad_fn=<IndexBackward0>), compiled_model=tensor([[0.76953, 0.60938, 0.43750], [0.50391, 0.19922, 0.78906], [0.84375, 0.58594, 0.50781], ..., [0.06299, 0.27148, 0.27344], [0.50391, 0.19922, 0.78906], [0.76172, 0.45508, 0.67578]])

test_argmax.test_module[Argmax0-[((1, 32), torch.int32)]]: forge/test/models_ops/test_argmax.py#L84

RuntimeError: Generated MLIR module failed verification.

test_avgpool2d.test_module[Avgpool2D9-[((1, 1024, 17, 17), torch.float32)]]: forge/test/models_ops/test_avgpool2d.py#L1472

RuntimeError: Tensor 1 - stride mismatch: expected [289, 1], got [0, 0]

test_broadcast.test_module[Broadcast1-[((1, 1, 1, 384), torch.bool)]]: forge/test/models_ops/test_broadcast.py#L172

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((2, 1, 1, 13), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

test_cast.test_module[Cast1-[((1, 1, 256, 256), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast2-[((1, 128), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast1-[((1, 1, 7, 7), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((1, 1, 1, 7), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 44., 239., 933., 760., 963., 379., 427.]]]]), compiled_model=tensor([[[[ 44., 0., 239., 0., 933., 0., 760.]]]])

test_cast.test_module[Cast0-[((1, 1, 32, 32), torch.int64)]]: forge/test/models_ops/test_cast.py#L502

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 44., 239., 933., ..., 480., 481., 12.], [363., 360., 995., ..., 326., 363., 474.], [775., 700., 218., ..., 858., 235., 416.], ..., [326., 721., 442., ..., 346., 346., 564.], [386., 447., 788., ..., 593., 762., 312.], [714., 724., 526., ..., 122., 978., 438.]]]]), compiled_model=tensor([[[[ 44., 0., 239., ..., 0., 376., 0.], [ 56., 0., 868., ..., 0., 12., 0.], [363., 0., 360., ..., 0., 395., 0.], ..., [108., 0., 590., ..., 0., 827., 0.], [556., 0., 895., ..., 0., 746., 0.], [403., 0., 958., ..., 0., 216., 0.]]]])

test_cast.test_module[Cast1-[((1, 16, 320, 1024), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_conv2d.test_module[Conv2D1-[((1, 1024, 3000, 1), torch.float32), ((1024, 1024, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[497.33414], [752.28168], [745.81519], ..., [748.79230], [751.20343], [758.10150]], [[516.65167], [763.25220], [751.97681], ..., [776.13300], [769.62982], [756.29266]], [[508.27203], [745.54572], [746.02032], ..., [750.85455], [748.23822], [741.94067]], ..., [[510.77304], [747.95300], [756.94989], ..., [774.09131], [777.70905], [763.47894]], [[517.18640], [770.13470], [773.81805], ..., [780.88708], [781.61749], [774.47156]], [[512.43396], [756.88477], [759.80255], ..., [763.79138], [769.09967], [768.24158]]]]), compiled_model=tensor([[[[498.], [760.], [740.], ..., [752.], [764.], [752.]], [[520.], [776.], [744.], ..., [784.], [780.], [768.]], [[510.], [764.], [744.], ..., [760.], [756.], [740.]], ..., [[502.], [760.], [760.], ..., [764.], [784.], [756.]], [[516.], [780.], [784.], ..., [776.], [788.], [780.]], [[504.], [768.], [756.], ..., [764.], [776.], [768.]]]])

test_conv2d.test_module[Conv2D1-[((1, 768, 3000, 1), torch.float32), ((768, 768, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_FATAL @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/ttnn/cpp/ttnn/operations/conv/conv2d/device/conv2d_op_sharded_program_factory.cpp:1180: act_block_w_datums == round_up(conv_act_size_c * filter_w, TILE_WIDTH) info: Error backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xbd288) [0x7fbb51ddd288] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d41multi_core_optimized_conv_sharded_v2_implERN2tt8tt_metal2v07ProgramERKNS4_6TensorESA_RKNS4_5ShapeESt8optionalIS9_ERKSF_RKNS0_14sliding_window19SlidingWindowConfigEjjbbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigEbbRS8_St7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x90c4) [0x7fbb3f4f97a4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d40multi_core_optimized_conv_sharded_v2_newERKN2tt8tt_metal6TensorES7_RKSt8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigENS4_8DataTypeESt5arrayIjLm4EEbS8_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERS5_bbbb+0x492) [0x7fbb3f4ecfc2] --- ttnn::operations::conv::conv2d::OptimizedConvNew::create_program(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) const --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x449829) [0x7fbb3f553829] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c0fda) [0x7fbb416cafda] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c13e4) [0x7fbb416cb3e4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x2ab) [0x7fbb416ca72b] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7fbb3f552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7fbb3f4ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7fbb3f4d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30]

test_conv2d.test_module[Conv2D2-[((1, 3072, 1, 128), torch.float32), ((768, 768, 1, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[199.48528, 195.31499, 186.96136, ..., 181.25623, 199.46423, 188.89122]], [[200.43977, 203.46483, 191.30280, ..., 190.64456, 194.62601, 197.45979]], [[198.70642, 199.62898, 191.56145, ..., 185.88058, 193.25822, 191.89673]], ..., [[185.18076, 179.17751, 192.73273, ..., 187.53204, 195.17900, 180.52390]], [[198.33948, 193.25223, 207.28433, ..., 204.61462, 208.78712, 195.67590]], [[194.15210, 185.43776, 198.50377, ..., 191.98058, 198.83009, 187.43867]]]]), compiled_model=tensor([[[[196., 192., 187., ..., 179., 193., 189.]], [[200., 197., 189., ..., 187., 193., 194.]], [[194., 196., 187., ..., 183., 193., 191.]], ..., [[183., 174., 191., ..., 187., 194., 179.]], [[195., 191., 201., ..., 200., 207., 196.]], [[192., 182., 197., ..., 188., 195., 184.]]]])

test_conv2d.test_module[Conv2D4-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 1062 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=5)]. L1 buffer allocated at 947328 and static circular buffer region ends at 1180896 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7fbb512372d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7fbb3f552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7fbb3f4ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7fbb3f4d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702]

test_conv2d.test_module[Conv2D9-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:896: tt::exception info: Statically allocated circular buffers on core range [(x=0,y=0) - (x=7,y=5)] grow to 1676512 B which is beyond max L1 size of 1499136 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x17129b) [0x7fbb5123729b] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7fbb3f552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7fbb3f4ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7fbb3f4d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702]

test_conv2d.test_module[Conv2D16-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 12500992 B L1 buffer across 7 banks, where each bank needs to store 1785856 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7fbb512207e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7fbb511ebe9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7fbb40109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7fbb40109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7fbb416ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464]

test_conv2d.test_module[Conv2D547-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 6422528 B L1 buffer across 2 banks, where each bank needs to store 3211264 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7fbb512207e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7fbb511ebe9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations13data_movement35InterleavedToShardedDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKSt6vectorINS0_6TensorESaISE_EERKSD_ISt8optionalISE_ESaISK_EE+0x1cf) [0x7fbb51e7429f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x409139) [0x7fbb3f513139] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7fbb416ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x40052f) [0x7fbb3f50a52f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff92f) [0x7fbb3f50992f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff708) [0x7fbb3f509708] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff66c) [0x7fbb3f50966c] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff44d) [0x7fbb3f50944d] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fee51) [0x7fbb3f508e51] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fe8d9) [0x7fbb3f5088d9] --- std::tuple<tt::tt_metal::Tensor, ttnn::operations::sliding_window::ParallelConfig, ttnn::operations::sliding_window::ParallelConfig> ttnn::operations::conv::shard_or_reshard_tensor_if_required<tt::tt_metal::v0::IDevice>(tt::tt_metal::v0::IDevice*, tt::tt_metal::Tensor const&, ttnn::operations::conv::conv2d::Conv2dConfig const&, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, bool, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0x697) [0x7fbb3f4d9747] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702]

test_conv2d.test_module[Conv2D655-[((1, 256, 42, 130), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[598.88538, 580.47205, 586.91284, ..., 582.23779, 587.73340, 593.70837], [594.69946, 576.07831, 587.89581, ..., 587.27087, 585.57880, 580.01807], [588.37378, 579.69928, 581.94275, ..., 596.41522, 579.88580, 585.73834], ..., [583.68433, 577.94366, 575.78210, ..., 596.81390, 586.61450, 587.02100], [583.60852, 582.81329, 574.97485, ..., 577.11279, 581.12317, 589.54645], [587.60455, 585.63586, 586.09296, ..., 584.87213, 588.24249, 592.50812]], [[588.01508, 587.37134, 586.65448, ..., 591.29779, 591.76050, 591.28400], [594.63177, 584.90100, 588.87744, ..., 588.92334, 584.22754, 582.88824], [587.58209, 575.62994, 577.22815, ..., 588.56805, 579.90234, 589.14929], ..., [589.92664, 583.11774, 577.72247, ..., 584.57385, 587.53394, 595.01343], [585.22552, 584.92963, 584.91754, ..., 583.71527, 588.43146, 590.58075], [592.41730, 583.16077, 577.81622, ..., 589.54230, 593.05396, 590.62628]], [[579.54456, 565.45538, 574.64545, ..., 578.56512, 579.38293, 576.04755], [575.98248, 574.54572, 576.07117, ..., 579.67444, 574.02313, 570.45697], [578.51886, 574.58411, 566.88776, ..., 581.15741, 565.31061, 572.24622], ..., [572.07648, 567.02704, 567.39832, ..., 574.43091, 570.90784, 586.86688], [581.14307, 582.83331, 578.42316, ..., 569.04163, 579.50555, 582.08789], [581.57959, 577.59766, 573.31232, ..., 569.97394, 585.39026, 583.61615]], ..., [[568.31104, 565.94232, 556.55743, ..., 561.58173, 561.08716, 570.34998], [568.95355, 558.29816, 568.38525, ..., 559.35565, 562.68536, 561.53552], [564.93573, 561.71210, 557.16223, ..., 572.78748, 567.10718, 556.85931], ..., [565.09204, 562.33057, 554.20691, ..., 565.86633, 562.03400, 561.83099], [564.61884, 564.30463, 557.10242, ..., 556.67480, 559.81433, 576.80176], [569.96985, 558.75690, 559.28033, ..., 567.62012, 562.38208, 565.17889]], [[575.28613, 551.05548, 558.03619, ..., 560.00098, 564.06201, 570.37659], [571.20795, 556.06635, 566.10156, ..., 565.06311, 558.50592, 562.00183], [559.91791, 553.81323, 557.94122, ..., 567.58606, 558.01038, 561.21466], ..., [561.86249, 549.57898, 551.82666, ..., 556.25049, 564.60663, 570.19318], [567.24371, 564.46429, 558.57892, ..., 555.20129, 570.66156, 560.57581], [574.16388, 553.72821, 549.03833, ..., 567.46173, 566.79858, 561.69073]], [[594.08521, 569.13586, 578.87573, ..., 575.64349, 585.11121, 578.25720], [581.89008, 573.30865, 584.80188, ..., 576.30170, 569.05212, 582.43262], [581.89679, 577.03876, 567.27191, ..., 585.16833, 570.59747, 573.26459], ..., [577.61841, 578.12079, 566.85107, ..., 578.59497, 580.13330, 589.63910], [581.16730, 583.51495, 577.53247, ..., 584.89502, 573.84412, 588.44440], [585.44836, 568.69080, 571.35144, ..., 582.84082, 583.01001, 583.69287]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[592., 572., 576., ..., 580., 584., 604.], [584., 576., 572., ..., 584., 576., 580.], [588., 572., 576., ..., 588., 580., 584.], ..., [584., 588., 572., ..., 588., 580., 588.], [576., 580., 572., ..., 584., 580., 588.], [584., 584., 584., ..., 584., 576., 600.]], [[588., 584., 584., ..., 588., 600., 592.], [596., 580., 588., ..., 592., 588., 580.], [584., 572., 584., ..., 580., 588., 580.], ..., [588., 588., 584., ..., 584., 580., 588.], [584., 592., 592., ..., 584., 604., 588.], [584., 588., 576., ..., 588., 600., 592.]], [[580., 564., 576., ..., 580., 592., 580.], [576., 572., 568., ..., 572., 564., 568.], [580., 572., 576., ..., 576., 572., 576.], ..., [576., 564., 568., ..., 564., 576., 584.], [576., 580., 584., ..., 572., 568., 580.], [588., 576., 568., ..., 564., 588., 576.]], ..., [[572., 564., 556., ..., 560., 568., 568.], [572., 552., 568., ..., 560., 556., 560.], [572., 556., 556., ..., 572., 560., 564.], ..., [568., 560., 548., ..., 568., 560., 568.], [568., 556., 560., ..., 564., 552., 576.], [572., 556., 548., ..., 580., 564., 568.]], [[568., 540., 564., ..., 560., 568., 576.], [572., 540., 560., ..., 564., 560., 568.], [556., 560., 560., ..., 576., 564., 544.], ..., [564., 552., 568., ..., 556., 568., 568.], [580., 576., 568., ..., 540., 572., 564.], [572., 556., 556., ..., 568., 580., 556.]], [[596., 564., 576., ..., 572., 592., 568.], [572., 572., 580., ..., 572., 572., 576.], [576., 572., 568., ..., 576., 560., 576.], ..., [572., 584., 572., ..., 576., 568., 576.], [584., 572., 588., ..., 580., 576., 600.], [576., 564., 568., ..., 580., 580., 580.]]]])

test_conv2d.test_module[Conv2D656-[((1, 128, 82, 258), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[285.43088, 285.01010, 287.90521, ..., 292.97586, 294.98996, 294.21155], [286.82846, 282.20401, 293.36142, ..., 293.15582, 301.28558, 291.70633], [290.46356, 289.75842, 294.81226, ..., 289.94031, 297.61261, 295.43655], ..., [286.57864, 282.84720, 289.58673, ..., 294.44699, 293.27917, 292.89606], [283.05960, 283.76105, 279.51260, ..., 292.55518, 295.39789, 290.93146], [284.78442, 282.79565, 278.84183, ..., 295.22150, 291.33090, 285.38284]], [[281.01862, 276.70621, 279.90610, ..., 285.79813, 287.06335, 288.20480], [277.20508, 283.40250, 286.87775, ..., 281.46231, 293.53882, 281.14081], [284.70819, 284.66296, 287.92563, ..., 281.91312, 294.21979, 285.16287], ..., [279.56546, 279.27563, 274.99255, ..., 287.18820, 295.23154, 285.37442], [277.40631, 280.40329, 272.08405, ..., 286.99258, 284.91483, 282.91104], [275.53513, 275.41336, 275.76300, ..., 284.74728, 282.24545, 279.28879]], [[282.20227, 283.80145, 286.92877, ..., 291.87891, 296.80798, 293.75067], [282.65024, 285.07294, 290.01395, ..., 294.67242, 303.48181, 295.61310], [282.12790, 285.46112, 289.18246, ..., 296.90692, 289.43658, 293.40863], ..., [278.50018, 279.96304, 280.25000, ..., 297.03555, 296.64053, 296.07874], [285.97717, 290.77710, 282.58844, ..., 285.37418, 292.82321, 290.61819], [285.48871, 283.10434, 279.40567, ..., 291.87680, 292.04120, 287.02594]], ..., [[279.28012, 280.69647, 284.63263, ..., 292.02347, 294.00906, 287.63400], [283.56805, 287.08493, 290.11368, ..., 288.42014, 299.60449, 288.13397], [285.04901, 287.86850, 289.34525, ..., 283.40262, 289.48099, 287.76129], ..., [283.20361, 285.98154, 283.43076, ..., 292.17255, 286.75986, 281.94571], [278.43854, 284.00748, 272.28732, ..., 289.21716, 286.49194, 281.55359], [275.16983, 276.12531, 270.28064, ..., 284.52103, 290.52670, 282.02051]], [[283.01495, 278.91907, 288.96777, ..., 291.34766, 289.72327, 289.49695], [276.44922, 286.23959, 286.24628, ..., 286.62140, 287.62329, 286.07895], [277.90018, 283.46658, 285.20178, ..., 284.59335, 288.24356, 288.40744], ..., [281.39206, 271.60223, 282.04449, ..., 291.81735, 290.41791, 287.96524], [280.02499, 279.35928, 279.03989, ..., 277.47382, 288.59213, 288.66409], [275.15274, 281.66025, 270.45755, ..., 279.10849, 282.64865, 282.45270]], [[287.36462, 290.98880, 285.88272, ..., 296.68436, 294.46811, 289.33505], [286.85214, 291.29095, 288.13715, ..., 291.73529, 293.17914, 295.68799], [287.01794, 287.16458, 289.00275, ..., 288.37204, 295.39783, 293.23715], ..., [286.78931, 283.44861, 283.09384, ..., 291.87692, 290.20413, 292.64438], [283.05911, 286.33566, 279.47311, ..., 288.99921, 293.63870, 289.06232], [282.56406, 278.97205, 276.35367, ..., 287.19153, 292.17880, 287.64502]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[286., 288., 282., ..., 290., 290., 292.], [280., 282., 296., ..., 290., 294., 286.], [290., 290., 292., ..., 284., 294., 294.], ..., [284., 282., 290., ..., 294., 290., 288.], [276., 278., 280., ..., 290., 296., 284.], [282., 282., 272., ..., 290., 286., 284.]], [[280., 270., 274., ..., 282., 284., 286.], [274., 280., 284., ..., 274., 290., 280.], [284., 280., 286., ..., 278., 288., 278.], ..., [272., 278., 272., ..., 282., 296., 286.], [274., 276., 268., ..., 288., 282., 280.], [272., 272., 276., ..., 282., 280., 280.]], [[278., 280., 282., ..., 290., 292., 290.], [276., 280., 290., ..., 296., 298., 296.], [278., 282., 288., ..., 292., 284., 292.], ..., [274., 284., 278., ..., 298., 294., 296.], [290., 290., 278., ..., 282., 290., 288.], [278., 280., 278., ..., 292., 286., 282.]], ..., [[274., 282., 286., ..., 286., 292., 282.], [278., 286., 286., ..., 284., 296., 286.], [282., 284., 286., ..., 276., 284., 286.], ..., [284., 278., 282., ..., 292., 282., 278.], [274., 284., 266., ..., 288., 286., 280.], [272., 272., 268., ..., 286., 290., 276.]], [[280., 282., 284., ..., 290., 284., 286.], [270., 280., 286., ..., 286., 286., 286.], [276., 280., 282., ..., 284., 284., 288.], ..., [278., 268., 282., ..., 288., 286., 288.], [280., 274., 274., ..., 274., 286., 286.], [272., 274., 266., ..., 274., 280., 278.]], [[282., 292., 284., ..., 294., 290., 284.], [284., 284., 286., ..., 294., 288., 298.], [288., 288., 290., ..., 284., 294., 294.], ..., [288., 282., 280., ..., 290., 290., 292.], [282., 286., 274., ..., 284., 288., 290.], [284., 276., 274., ..., 284., 290., 286.]]]])

test_conv2d.test_module[Conv2D654-[((1, 512, 8, 22), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1182.34241, 1177.08228, 1195.55945, ..., 1157.57458, 1156.92590, 1159.82361], [1176.13220, 1177.88611, 1191.58496, ..., 1147.82532, 1137.84155, 1161.05188], [1171.80896, 1160.43555, 1177.42664, ..., 1155.66504, 1140.98242, 1153.42810], [1171.23572, 1168.48267, 1176.37292, ..., 1154.63013, 1127.17578, 1159.26050], [1159.41052, 1166.32227, 1154.56885, ..., 1171.48926, 1150.77502, 1148.86206], [1162.69263, 1153.49841, 1141.93738, ..., 1170.93884, 1151.00659, 1141.21887]], [[1166.06421, 1159.48767, 1163.63342, ..., 1145.97742, 1135.84741, 1131.68481], [1154.76587, 1157.93750, 1159.83875, ..., 1125.83411, 1127.27942, 1136.65002], [1135.81140, 1139.39258, 1143.34570, ..., 1127.46021, 1130.92651, 1136.47302], [1155.68640, 1156.52393, 1149.41919, ..., 1124.55188, 1113.35120, 1125.83411], [1150.53577, 1144.21106, 1129.02380, ..., 1140.51270, 1136.09119, 1129.92004], [1133.99365, 1131.60437, 1118.10083, ..., 1156.03821, 1123.88293, 1118.99414]], [[1170.75732, 1161.42041, 1167.57422, ..., 1160.07385, 1136.86133, 1149.29358], [1161.76965, 1167.27393, 1164.49609, ..., 1139.10547, 1132.68945, 1140.39722], [1145.15881, 1150.94580, 1163.52930, ..., 1146.94189, 1128.50232, 1139.49524], [1162.73657, 1160.32910, 1153.17712, ..., 1136.79248, 1122.18005, 1130.61853], [1147.68396, 1169.82214, 1137.75818, ..., 1156.02771, 1145.94421, 1132.38538], [1151.32983, 1139.41333, 1124.27686, ..., 1148.96814, 1135.88733, 1136.85522]], ..., [[1162.51978, 1166.22559, 1190.58545, ..., 1165.44336, 1147.61694, 1145.92651], [1159.53271, 1163.33276, 1169.37744, ..., 1147.47388, 1133.38586, 1149.86414], [1160.84253, 1162.63232, 1162.96509, ..., 1135.84277, 1141.20972, 1155.47729], [1163.16199, 1174.93042, 1147.92017, ..., 1148.72778, 1139.18127, 1132.88660], [1148.86682, 1163.67529, 1151.48901, ..., 1157.49792, 1151.53540, 1147.81396], [1148.50549, 1144.76306, 1127.95569, ..., 1167.96118, 1133.54517, 1149.27332]], [[1167.58777, 1161.65784, 1163.95020, ..., 1145.63770, 1140.74097, 1134.88501], [1151.46680, 1154.40308, 1150.09070, ..., 1135.44092, 1121.67249, 1132.30701], [1141.69043, 1140.04895, 1150.29858, ..., 1123.93677, 1123.12024, 1126.30994], [1148.34998, 1149.59045, 1140.14197, ..., 1129.32288, 1123.72620, 1124.33179], [1129.33276, 1147.79663, 1128.57581, ..., 1158.43091, 1127.50928, 1121.58850], [1128.24841, 1136.86511, 1123.84497, ..., 1147.61511, 1120.89294, 1127.81042]], [[1155.16211, 1149.75574, 1156.40918, ..., 1143.70593, 1136.09473, 1131.53735], [1150.73950, 1150.34033, 1152.19531, ..., 1125.54749, 1113.29248, 1131.77100], [1133.57239, 1142.02649, 1145.58545, ..., 1129.68127, 1123.20605, 1130.49585], [1144.26160, 1143.69629, 1143.49084, ..., 1131.57312, 1120.69446, 1121.97778], [1139.31238, 1151.09375, 1136.55469, ..., 1154.07043, 1122.73242, 1125.91711], [1129.86902, 1118.23438, 1121.81140, ..., 1142.65564, 1121.22180, 1130.47144]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1256., 1248., 1232., ..., 1192., 1208., 1216.], [1248., 1256., 1240., ..., 1192., 1184., 1208.], [1224., 1240., 1232., ..., 1168., 1192., 1216.], [1200., 1200., 1272., ..., 1168., 1208., 1208.], [1224., 1216., 1192., ..., 1232., 1200., 1200.], [1248., 1200., 1224., ..., 1256., 1216., 1168.]], [[1208., 1224., 1248., ..., 1160., 1200., 1200.], [1224., 1240., 1200., ..., 1144., 1160., 1200.], [1192., 1208., 1200., ..., 1176., 1200., 1184.], [1208., 1208., 1216., ..., 1184., 1160., 1136.], [1208., 1176., 1184., ..., 1176., 1184., 1152.], [1200., 1192., 1152., ..., 1208., 1192., 1168.]], [[1232., 1224., 1240., ..., 1232., 1152., 1200.], [1232., 1248., 1216., ..., 1168., 1192., 1200.], [1184., 1192., 1192., ..., 1184., 1176., 1200.], [1200., 1224., 1184., ..., 1184., 1168., 1160.], [1176., 1216., 1176., ..., 1176., 1208., 1192.], [1192., 1160., 1152., ..., 1192., 1184., 1168.]], ..., [[1224., 1224., 1264., ..., 1192., 1200., 1168.], [1208., 1232., 1224., ..., 1200., 1160., 1192.], [1240., 1184., 1216., ..., 1192., 1200., 1200.], [1192., 1216., 1192., ..., 1224., 1168., 1144.], [1232., 1200., 1176., ..., 1184., 1184., 1208.], [1184., 1168., 1168., ..., 1232., 1160., 1200.]], [[1232., 1256., 1240., ..., 1184., 1208., 1168.], [1248., 1176., 1160., ..., 1176., 1184., 1192.], [1232., 1208., 1160., ..., 1192., 1128., 1192.], [1208., 1216., 1176., ..., 1200., 1176., 1152.], [1208., 1192., 1192., ..., 1216., 1152., 1136.], [1184., 1176., 1152., ..., 1224., 1192., 1200.]], [[1248., 1208., 1264., ..., 1184., 1200., 1208.], [1240., 1272., 1256., ..., 1184., 1184., 1168.], [1200., 1224., 1192., ..., 1208., 1216., 1184.], [1224., 1216., 1208., ..., 1168., 1192., 1144.], [1200., 1216., 1168., ..., 1248., 1176., 1160.], [1184., 1200., 1184., ..., 1256., 1176., 1192.]]]])

test_conv2d.test_module[Conv2D658-[((1, 96, 98, 322), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[218.78259, 221.43571, 224.75235, ..., 216.10695, 219.19431, 215.82306], [217.56743, 220.21608, 218.36884, ..., 216.45609, 216.01489, 219.80571], [221.06578, 213.57217, 215.65636, ..., 212.67734, 220.12679, 217.40915], ..., [219.93652, 217.23357, 220.52121, ..., 216.47780, 213.07104, 211.63918], [215.94714, 213.94556, 221.47702, ..., 216.04524, 225.38530, 215.44543], [219.53445, 220.02295, 225.99745, ..., 229.64908, 232.19359, 222.60022]], [[228.56732, 233.49359, 223.60909, ..., 219.40520, 223.37175, 225.40459], [220.94925, 224.52272, 225.19609, ..., 222.93784, 223.42459, 222.85976], [223.06879, 217.35760, 220.05466, ..., 218.99843, 218.31190, 220.03580], ..., [223.21648, 224.00044, 227.68753, ..., 225.00002, 225.37057, 217.07040], [217.94632, 223.84622, 228.71359, ..., 224.52039, 227.90356, 218.69691], [212.62207, 228.04341, 226.45729, ..., 231.46948, 235.14001, 222.37700]], [[214.40140, 217.23492, 212.62654, ..., 209.88383, 212.21924, 213.37630], [203.76164, 216.05486, 211.83629, ..., 207.97057, 211.50600, 210.03786], [207.85138, 205.62866, 208.99031, ..., 207.52818, 213.23093, 212.45097], ..., [214.87257, 211.79234, 217.35992, ..., 206.56044, 209.16821, 205.01588], [212.23784, 209.59584, 208.35822, ..., 215.62891, 214.55000, 211.25436], [208.93539, 212.82501, 212.96069, ..., 224.84357, 224.45000, 209.97121]], ..., [[210.66847, 209.21883, 212.22586, ..., 205.92422, 208.14600, 215.03262], [208.71173, 209.89977, 207.58693, ..., 209.66504, 207.26057, 208.24495], [215.10881, 209.93585, 205.59120, ..., 206.19731, 204.07449, 210.78682], ..., [209.70808, 204.71332, 211.96211, ..., 208.17664, 205.62634, 202.51825], [207.41516, 211.51334, 213.03854, ..., 210.93253, 210.76334, 212.15752], [205.47148, 209.44199, 215.89787, ..., 215.80087, 216.77170, 209.48433]], [[209.94292, 212.60562, 216.17116, ..., 209.97289, 213.41866, 206.62074], [212.30225, 212.99646, 209.94820, ..., 206.33064, 211.56131, 209.83499], [212.38676, 210.56955, 210.71970, ..., 201.29384, 205.35205, 207.80240], ..., [212.76811, 208.98950, 216.78236, ..., 212.45883, 210.97786, 205.49481], [216.21437, 208.46954, 213.47304, ..., 215.11905, 216.42415, 214.07318], [208.43661, 212.10623, 212.88663, ..., 217.04243, 220.89772, 211.44420]], [[211.28716, 212.48270, 212.53532, ..., 209.67177, 209.40640, 207.48849], [207.10912, 210.57770, 207.93005, ..., 206.22635, 206.81039, 209.34734], [209.59970, 204.63075, 204.45923, ..., 203.98071, 201.58344, 207.58003], ..., [203.22795, 208.67030, 211.10048, ..., 207.20488, 211.00246, 200.75070], [208.63055, 206.45262, 209.71718, ..., 209.30139, 214.43649, 207.72066], [211.06464, 211.38936, 212.56845, ..., 221.96643, 213.94463, 208.51048]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[216., 217., 224., ..., 214., 217., 215.], [216., 217., 214., ..., 213., 214., 215.], [220., 211., 214., ..., 208., 218., 214.], ..., [217., 217., 220., ..., 215., 209., 210.], [213., 212., 216., ..., 214., 223., 213.], [216., 216., 222., ..., 229., 227., 222.]], [[225., 229., 220., ..., 218., 219., 220.], [218., 223., 220., ..., 222., 221., 220.], [222., 215., 218., ..., 214., 214., 216.], ..., [220., 224., 225., ..., 221., 224., 211.], [214., 220., 228., ..., 223., 228., 214.], [211., 224., 223., ..., 230., 232., 221.]], [[213., 214., 213., ..., 205., 212., 214.], [201., 213., 211., ..., 204., 209., 206.], [207., 203., 205., ..., 207., 211., 209.], ..., [211., 212., 216., ..., 203., 207., 204.], [208., 209., 207., ..., 211., 207., 209.], [207., 212., 212., ..., 218., 222., 208.]], ..., [[207., 206., 210., ..., 203., 205., 213.], [207., 205., 206., ..., 207., 201., 204.], [214., 208., 202., ..., 202., 198., 208.], ..., [209., 203., 211., ..., 205., 203., 198.], [203., 208., 212., ..., 208., 207., 207.], [204., 204., 212., ..., 212., 215., 207.]], [[206., 212., 211., ..., 206., 210., 203.], [210., 214., 207., ..., 204., 208., 207.], [208., 208., 209., ..., 199., 201., 206.], ..., [212., 208., 213., ..., 210., 208., 204.], [213., 205., 210., ..., 210., 214., 212.], [204., 212., 210., ..., 215., 222., 207.]], [[209., 210., 209., ..., 207., 206., 207.], [205., 207., 205., ..., 204., 206., 210.], [206., 202., 202., ..., 200., 201., 205.], ..., [201., 206., 209., ..., 205., 206., 199.], [205., 204., 208., ..., 207., 214., 206.], [211., 208., 212., ..., 220., 213., 205.]]]])

test_conv2d.test_module[Conv2D706-[((1, 3, 512, 512), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 3984 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 213120 and static circular buffer region ends at 623904 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7fbb512372d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab]

test_conv2d.test_module[Conv2D715-[((1, 128, 64, 64), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[520.09082, 499.40973, 509.43903, ..., 518.24725, 505.02591, 507.78067], [511.51337, 525.23370, 498.54825, ..., 499.72913, 511.83936, 516.36591], [507.99277, 517.43317, 502.95184, ..., 505.57135, 520.13098, 505.48593], ..., [501.89737, 508.60208, 511.61447, ..., 506.03131, 502.14249, 505.88162], [500.46918, 513.40558, 505.82663, ..., 489.03125, 501.28607, 496.72263], [510.53040, 525.05139, 513.22876, ..., 496.04312, 520.42206, 512.73242]], [[520.28113, 511.44684, 512.44104, ..., 526.93835, 510.97934, 514.51764], [520.91254, 528.55988, 512.02515, ..., 505.65506, 518.04614, 530.72717], [521.76166, 529.72321, 503.27869, ..., 520.70813, 529.86273, 516.86682], ..., [515.21088, 524.03876, 516.09595, ..., 510.02032, 506.98141, 511.69843], [509.20734, 523.44183, 512.50824, ..., 502.02103, 516.88959, 509.76257], [509.53177, 531.22675, 519.05383, ..., 505.22650, 531.06183, 527.24414]], [[516.90601, 499.52271, 506.03281, ..., 522.92017, 505.99124, 507.40677], [511.44141, 523.41382, 504.72214, ..., 501.69208, 506.16998, 527.34613], [512.91541, 517.61969, 495.11179, ..., 511.35193, 521.18451, 511.71021], ..., [505.97098, 514.80493, 523.07831, ..., 518.28589, 500.98697, 517.97369], [505.43027, 524.67761, 514.36707, ..., 496.66095, 504.32739, 500.22797], [506.97073, 528.41290, 511.90472, ..., 504.57501, 517.76807, 523.56970]], ..., [[515.77808, 495.67166, 503.20660, ..., 510.69080, 494.14471, 500.60782], [499.20724, 517.75806, 499.62756, ..., 492.90598, 507.77261, 519.37158], [508.38727, 510.80444, 489.29803, ..., 511.15930, 515.31964, 509.85065], ..., [497.49014, 501.27448, 508.89682, ..., 496.38116, 495.27246, 507.04269], [501.63437, 502.00726, 503.42276, ..., 488.16904, 496.91574, 496.29980], [506.46426, 515.03290, 506.53717, ..., 489.61829, 510.30945, 509.26022]], [[528.64398, 513.87415, 504.29175, ..., 533.51819, 513.54987, 510.24393], [510.77426, 528.51532, 501.27853, ..., 503.99475, 511.85825, 529.46527], [521.25793, 521.74713, 508.97226, ..., 518.12036, 531.47546, 516.69196], ..., [508.93033, 521.73352, 517.75708, ..., 515.60797, 516.59747, 514.89679], [517.04858, 527.45294, 514.83954, ..., 501.06354, 507.05423, 506.63025], [518.28693, 527.69513, 521.66565, ..., 503.21863, 530.40741, 524.53168]], [[531.18524, 507.57553, 515.28149, ..., 526.64093, 506.73764, 515.82245], [512.93933, 529.66217, 504.98264, ..., 512.20190, 513.88202, 531.53528], [516.90363, 521.31439, 497.55820, ..., 519.11908, 534.39154, 514.79498], ..., [514.89490, 520.24646, 517.57471, ..., 511.28207, 509.26968, 518.48865], [510.20059, 518.08368, 516.11078, ..., 501.67130, 509.18088, 509.08066], [514.65271, 528.01483, 514.70789, ..., 511.33194, 519.47931, 525.17157]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[528., 494., 524., ..., 524., 500., 512.], [516., 516., 506., ..., 506., 508., 516.], [506., 516., 498., ..., 508., 520., 512.], ..., [502., 510., 516., ..., 508., 494., 502.], [508., 512., 506., ..., 490., 506., 502.], [516., 528., 516., ..., 494., 512., 512.]], [[520., 516., 510., ..., 524., 506., 516.], [510., 524., 512., ..., 510., 512., 528.], [524., 524., 508., ..., 516., 532., 520.], ..., [510., 524., 510., ..., 506., 508., 516.], [516., 516., 512., ..., 500., 520., 500.], [502., 532., 512., ..., 502., 532., 528.]], [[524., 496., 504., ..., 520., 498., 508.], [510., 516., 512., ..., 502., 512., 520.], [516., 516., 494., ..., 516., 516., 500.], ..., [512., 512., 516., ..., 528., 498., 516.], [508., 524., 512., ..., 500., 512., 500.], [512., 528., 516., ..., 508., 510., 524.]], ..., [[520., 500., 502., ..., 508., 486., 508.], [510., 520., 490., ..., 500., 516., 516.], [516., 516., 486., ..., 516., 524., 506.], ..., [502., 502., 508., ..., 504., 494., 508.], [498., 500., 504., ..., 488., 500., 498.], [510., 504., 506., ..., 480., 508., 510.]], [[524., 508., 502., ..., 528., 516., 520.], [508., 520., 504., ..., 502., 500., 528.], [524., 528., 516., ..., 516., 528., 516.], ..., [510., 506., 516., ..., 504., 508., 512.], [512., 520., 508., ..., 504., 504., 510.], [512., 524., 524., ..., 506., 540., 516.]], [[532., 512., 516., ..., 528., 508., 512.], [504., 532., 496., ..., 512., 512., 528.], [516., 520., 498., ..., 520., 528., 520.], ..., [506., 516., 510., ..., 512., 512., 508.], [516., 516., 520., ..., 504., 510., 512.], [520., 520., 510., ..., 520., 524., 524.]]]])

test_conv2d.test_module[Conv2D724-[((1, 1024, 38, 38), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1024.86096, 1546.38696, 1546.34961, ..., 1533.96497, 1528.00781, 1003.87463], [1551.43542, 2302.08203, 2301.97412, ..., 2294.80273, 2284.67944, 1518.12585], [1542.81177, 2299.22070, 2281.03467, ..., 2312.17358, 2284.19165, 1509.01135], ..., [1527.31470, 2285.83057, 2310.32153, ..., 2328.96118, 2304.34058, 1514.29016], [1525.46057, 2305.46411, 2327.46484, ..., 2310.78003, 2309.70508, 1525.22192], [1023.62085, 1543.84460, 1550.45679, ..., 1547.75708, 1533.95020, 1014.53088]], [[1015.61395, 1518.28735, 1514.39429, ..., 1509.36279, 1499.04236, 976.49731], [1516.89062, 2251.95142, 2278.50366, ..., 2270.67773, 2254.21777, 1473.71704], [1525.15503, 2263.20337, 2281.82300, ..., 2277.48022, 2260.56006, 1489.27222], ..., [1515.27905, 2259.79224, 2278.06543, ..., 2295.31567, 2280.03149, 1498.09375], [1516.89001, 2272.21509, 2292.94263, ..., 2289.25122, 2267.07690, 1489.90039], [1006.06073, 1510.46765, 1513.42468, ..., 1509.57080, 1522.52637, 1008.76221]], [[1033.09973, 1554.98022, 1547.71021, ..., 1523.92737, 1506.64099, 997.17865], [1526.85767, 2296.88599, 2301.22607, ..., 2282.81860, 2269.73413, 1515.81421], [1538.64465, 2298.70630, 2300.70630, ..., 2321.84985, 2292.03784, 1515.41821], ..., [1521.75781, 2295.16821, 2315.07227, ..., 2323.06592, 2300.05371, 1518.30408], [1525.58765, 2301.05591, 2301.32715, ..., 2295.82666, 2297.59155, 1528.03796], [1019.45135, 1514.56934, 1537.27295, ..., 1524.79895, 1514.35718, 1017.87750]], ..., [[1042.93335, 1584.43518, 1574.40710, ..., 1547.29272, 1537.60657, 1010.36169], [1558.54846, 2324.59790, 2338.00391, ..., 2333.19751, 2319.42725, 1520.99353], [1571.45959, 2328.73730, 2333.50366, ..., 2332.94458, 2330.15820, 1528.65527], ..., [1553.89478, 2328.30273, 2341.29199, ..., 2366.07007, 2336.96777, 1525.78735], [1564.55896, 2332.23706, 2346.21606, ..., 2327.02197, 2338.67017, 1523.90308], [1046.86877, 1561.68811, 1583.13562, ..., 1556.30933, 1567.46094, 1039.91113]], [[1024.58948, 1535.76221, 1538.79834, ..., 1520.12805, 1504.81604, 994.42242], [1522.27661, 2281.95703, 2289.48267, ..., 2285.80493, 2280.07642, 1520.93616], [1541.32861, 2303.05640, 2298.53760, ..., 2299.85205, 2284.33789, 1524.92285], ..., [1525.58411, 2303.71411, 2311.14404, ..., 2324.72827, 2304.53027, 1512.32898], [1525.66919, 2298.70093, 2299.05566, ..., 2302.69336, 2301.10010, 1535.55042], [1027.23425, 1548.71387, 1547.55115, ..., 1531.75342, 1540.91003, 1030.73865]], [[1021.80151, 1538.75305, 1541.37488, ..., 1519.05518, 1520.22168, 997.21674], [1523.78479, 2304.51196, 2301.09033, ..., 2278.50830, 2277.05566, 1515.54016], [1529.64502, 2290.27515, 2305.71533, ..., 2297.31787, 2293.71753, 1506.85815], ..., [1539.12354, 2303.46240, 2329.84351, ..., 2309.23535, 2303.41016, 1516.61267], [1530.42749, 2290.25146, 2322.53320, ..., 2293.80420, 2310.86011, 1510.02502], [1028.56799, 1538.79932, 1544.59363, ..., 1543.44971, 1530.40747, 1024.08801]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1032., 1760., 1728., ..., 1648., 1624., 1040.], [1688., 2064., 2048., ..., 2048., 2048., 1656.], [1752., 2048., 2064., ..., 2064., 2048., 1640.], ..., [1680., 2064., 2048., ..., 2048., 2048., 1656.], [1728., 2048., 2048., ..., 2048., 2048., 1672.], [1032., 1736., 1768., ..., 1736., 1640., 1040.]], [[1024., 1720., 1688., ..., 1632., 1624., 1004.], [1752., 2048., 2048., ..., 2048., 2048., 1640.], [1688., 2048., 2064., ..., 2080., 2048., 1656.], ..., [1672., 2048., 2048., ..., 2048., 2064., 1696.], [1680., 2048., 2064., ..., 2048., 2048., 1608.], [1012., 1648., 1704., ..., 1632., 1656., 1024.]], [[1056., 1744., 1704., ..., 1648., 1680., 1012.], [1680., 2048., 2048., ..., 2048., 2064., 1736.], [1752., 2064., 2064., ..., 2048., 2048., 1696.], ..., [1712., 2048., 2048., ..., 2064., 2048., 1664.], [1672., 2048., 2048., ..., 2048., 2048., 1680.], [1024., 1696., 1680., ..., 1672., 1688., 1048.]], ..., [[1040., 1744., 1832., ..., 1696., 1736., 1032.], [1768., 2064., 2048., ..., 2048., 2048., 1656.], [1760., 2064., 2064., ..., 2048., 2048., 1632.], ..., [1712., 2064., 2080., ..., 2048., 2048., 1664.], [1672., 2048., 2064., ..., 2064., 2048., 1648.], [1032., 1720., 1776., ..., 1688., 1704., 1056.]], [[1040., 1712., 1736., ..., 1632., 1608., 1012.], [1664., 2064., 2064., ..., 2064., 2048., 1656.], [1704., 2048., 2048., ..., 2048., 2048., 1640.], ..., [1688., 2048., 2064., ..., 2048., 2048., 1672.], [1664., 2048., 2048., ..., 2048., 2048., 1696.], [1040., 1728., 1768., ..., 1712., 1656., 1048.]], [[1032., 1752., 1712., ..., 1672., 1720., 1020.], [1768., 2064., 2048., ..., 2048., 2048., 1680.], [1704., 2048., 2048., ..., 2064., 2048., 1640.], ..., [1752., 2048., 2048., ..., 2048., 2064., 1704.], [1664., 2048., 2048., ..., 2048., 2048., 1704.], [1040., 1680., 1736., ..., 1752., 1680., 1032.]]]])

test_conv2d.test_module[Conv2D727-[((1, 128, 3, 3), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[287.45816]], [[282.58142]], [[272.47748]], [[286.23596]], [[288.06201]], [[292.00012]], [[292.17969]], [[272.34180]], [[292.72046]], [[289.62955]], [[291.78046]], [[283.18536]], [[286.01230]], [[288.46478]], [[291.87509]], [[281.27502]], [[282.42499]], [[280.12622]], [[281.50458]], [[288.64673]], [[283.92114]], [[295.78717]], [[289.92279]], [[288.98599]], [[288.13739]], [[288.82880]], [[276.82407]], [[292.67847]], [[278.96106]], [[279.68127]], [[278.33807]], [[286.37772]], [[283.55820]], [[289.39377]], [[283.23175]], [[289.48468]], [[283.09009]], [[284.66953]], [[279.84073]], [[283.99051]], [[282.41870]], [[275.57812]], [[285.40042]], [[287.50430]], [[290.72632]], [[286.10101]], [[281.62701]], [[289.49426]], [[271.19568]], [[286.01950]], [[285.43811]], [[279.63202]], [[291.32278]], [[282.88708]], [[285.53699]], [[281.84543]], [[287.08881]], [[279.50684]], [[284.83484]], [[278.91229]], [[283.60986]], [[289.72180]], [[289.56537]], [[284.38501]], [[279.93137]], [[280.48016]], [[288.09137]], [[280.89554]], [[283.10458]], [[294.01007]], [[283.05505]], [[282.08524]], [[278.03488]], [[291.33939]], [[291.39404]], [[283.70801]], [[277.54398]], [[286.24121]], [[287.26807]], [[289.62555]], [[284.02216]], [[284.61630]], [[277.81250]], [[286.52649]], [[281.15662]], [[277.45123]], [[287.81931]], [[281.38779]], [[282.12964]], [[289.16528]], [[286.27328]], [[272.47617]], [[293.46481]], [[296.88303]], [[296.82782]], [[289.15625]], [[275.92810]], [[289.21915]], [[291.58325]], [[287.07538]], [[285.48282]], [[292.99460]], [[287.39337]], [[286.42435]], [[284.28735]], [[277.97330]], [[285.34454]], [[284.24133]], [[302.54489]], [[284.67200]], [[282.72311]], [[296.95303]], [[299.07391]], [[292.27820]], [[287.33981]], [[275.47894]], [[272.52280]], [[275.13965]], [[281.27765]], [[290.83252]], [[290.09552]], [[282.10922]], [[271.38980]], [[283.41998]], [[278.68069]], [[282.72339]], [[290.39145]], [[290.80560]], [[281.52051]], [[293.38507]], [[287.02756]], [[287.96454]], [[287.05945]], [[278.25421]], [[293.01605]], [[286.20239]], [[282.70361]], [[287.09274]], [[285.72675]], [[284.96112]], [[281.77731]], [[286.83514]], [[290.22934]], [[282.45715]], [[287.06378]], [[292.95306]], [[294.44238]], [[292.64783]], [[286.70996]], [[281.77991]], [[277.44849]], [[290.43469]], [[290.48935]], [[285.56827]], [[284.52032]], [[282.94617]], [[271.54904]], [[290.36557]], [[276.19785]], [[285.41394]], [[292.65222]], [[281.95804]], [[282.67816]], [[284.60278]], [[284.98438]], [[284.15295]], [[282.53552]], [[280.91275]], [[282.10226]], [[286.43777]], [[277.59625]], [[284.72650]], [[282.34869]], [[277.65179]], [[288.43036]], [[291.61969]], [[282.51245]], [[286.77371]], [[290.64551]], [[279.96375]], [[289.81403]], [[281.35693]], [[284.08975]], [[282.48233]], [[282.30560]], [[297.17258]], [[276.36374]], [[293.59961]], [[274.69122]], [[285.54147]], [[295.83279]], [[294.87622]], [[284.43677]], [[278.97815]], [[277.05048]], [[291.07788]], [[290.09174]], [[284.18536]], [[289.34955]], [[285.12399]], [[282.64478]], [[282.47589]], [[284.17163]], [[284.84674]], [[282.94168]], [[282.17343]], [[287.39990]], [[286.42572]], [[272.03006]], [[281.12000]], [[283.07611]], [[279.49808]], [[278.80988]], [[278.35693]], [[280.97830]], [[288.23108]], [[277.56125]], [[290.10394]], [[289.82074]], [[283.94943]], [[287.36572]], [[277.06094]], [[285.15903]], [[272.71234]], [[284.88800]], [[284.89014]], [[284.41037]], [[274.41510]], [[289.24557]], [[286.68890]], [[289.82904]], [[285.97437]], [[275.89526]], [[286.28323]], [[273.32175]], [[277.86688]], [[293.28198]], [[282.67972]], [[275.66534]], [[282.48660]], [[293.61218]], [[287.58325]], [[285.19922]], [[294.04926]], [[279.20813]], [[283.38821]], [[286.16867]], [[278.86633]], [[294.30853]], [[288.86392]], [[290.31061]], [[290.02979]], [[285.04425]], [[281.56287]], [[285.33310]], [[279.49164]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[284.]], [[278.]], [[268.]], [[284.]], [[284.]], [[292.]], [[294.]], [[270.]], [[286.]], [[288.]], [[292.]], [[280.]], [[286.]], [[286.]], [[290.]], [[276.]], [[282.]], [[278.]], [[278.]], [[288.]], [[284.]], [[294.]], [[288.]], [[284.]], [[284.]], [[288.]], [[274.]], [[294.]], [[278.]], [[278.]], [[280.]], [[284.]], [[280.]], [[288.]], [[282.]], [[288.]], [[276.]], [[280.]], [[276.]], [[282.]], [[278.]], [[274.]], [[278.]], [[282.]], [[290.]], [[280.]], [[278.]], [[282.]], [[266.]], [[284.]], [[282.]], [[274.]], [[290.]], [[280.]], [[278.]], [[278.]], [[284.]], [[274.]], [[280.]], [[276.]], [[280.]], [[284.]], [[284.]], [[278.]], [[274.]], [[276.]], [[286.]], [[276.]], [[276.]], [[290.]], [[278.]], [[276.]], [[274.]], [[286.]], [[282.]], [[282.]], [[278.]], [[284.]], [[284.]], [[288.]], [[282.]], [[276.]], [[274.]], [[288.]], [[280.]], [[278.]], [[284.]], [[280.]], [[278.]], [[290.]], [[288.]], [[276.]], [[296.]], [[294.]], [[296.]], [[290.]], [[272.]], [[288.]], [[288.]], [[284.]], [[280.]], [[290.]], [[284.]], [[284.]], [[282.]], [[274.]], [[282.]], [[280.]], [[302.]], [[280.]], [[284.]], [[298.]], [[294.]], [[290.]], [[282.]], [[272.]], [[270.]], [[272.]], [[280.]], [[286.]], [[286.]], [[276.]], [[272.]], [[276.]], [[276.]], [[280.]], [[284.]], [[288.]], [[280.]], [[288.]], [[280.]], [[286.]], [[280.]], [[280.]], [[288.]], [[282.]], [[284.]], [[292.]], [[286.]], [[286.]], [[280.]], [[290.]], [[284.]], [[276.]], [[284.]], [[284.]], [[292.]], [[290.]], [[280.]], [[280.]], [[276.]], [[288.]], [[288.]], [[284.]], [[280.]], [[280.]], [[272.]], [[284.]], [[274.]], [[280.]], [[288.]], [[278.]], [[276.]], [[282.]], [[284.]], [[282.]], [[278.]], [[280.]], [[276.]], [[288.]], [[276.]], [[288.]], [[276.]], [[274.]], [[288.]], [[288.]], [[274.]], [[282.]], [[288.]], [[274.]], [[286.]], [[278.]], [[284.]], [[280.]], [[278.]], [[294.]], [[276.]], [[290.]], [[274.]], [[284.]], [[294.]], [[294.]], [[282.]], [[274.]], [[274.]], [[290.]], [[288.]], [[282.]], [[286.]], [[280.]], [[278.]], [[278.]], [[280.]], [[282.]], [[278.]], [[280.]], [[288.]], [[280.]], [[270.]], [[278.]], [[278.]], [[278.]], [[276.]], [[274.]], [[278.]], [[288.]], [[272.]], [[286.]], [[286.]], [[280.]], [[284.]], [[272.]], [[282.]], [[264.]], [[284.]], [[280.]], [[276.]], [[270.]], [[288.]], [[288.]], [[286.]], [[282.]], [[276.]], [[286.]], [[274.]], [[274.]], [[290.]], [[284.]], [[274.]], [[278.]], [[290.]], [[282.]], [[282.]], [[292.]], [[274.]], [[280.]], [[286.]], [[278.]], [[288.]], [[286.]], [[284.]], [[286.]], [[280.]], [[280.]], [[282.]], [[276.]]]])

test_conv2d.test_module[Conv2D729-[((1, 1024, 38, 38), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1024.86096, 1546.38696, 1546.34961, ..., 1533.96497, 1528.00781, 1003.87463], [1551.43542, 2302.08203, 2301.97412, ..., 2294.80273, 2284.67944, 1518.12585], [1542.81177, 2299.22070, 2281.03467, ..., 2312.17358, 2284.19165, 1509.01135], ..., [1527.31470, 2285.83057, 2310.32153, ..., 2328.96118, 2304.34058, 1514.29016], [1525.46057, 2305.46411, 2327.46484, ..., 2310.78003, 2309.70508, 1525.22192], [1023.62085, 1543.84460, 1550.45679, ..., 1547.75708, 1533.95020, 1014.53088]], [[1015.61395, 1518.28735, 1514.39429, ..., 1509.36279, 1499.04236, 976.49731], [1516.89062, 2251.95142, 2278.50366, ..., 2270.67773, 2254.21777, 1473.71704], [1525.15503, 2263.20337, 2281.82300, ..., 2277.48022, 2260.56006, 1489.27222], ..., [1515.27905, 2259.79224, 2278.06543, ..., 2295.31567, 2280.03149, 1498.09375], [1516.89001, 2272.21509, 2292.94263, ..., 2289.25122, 2267.07690, 1489.90039], [1006.06073, 1510.46765, 1513.42468, ..., 1509.57080, 1522.52637, 1008.76221]], [[1033.09973, 1554.98022, 1547.71021, ..., 1523.92737, 1506.64099, 997.17865], [1526.85767, 2296.88599, 2301.22607, ..., 2282.81860, 2269.73413, 1515.81421], [1538.64465, 2298.70630, 2300.70630, ..., 2321.84985, 2292.03784, 1515.41821], ..., [1521.75781, 2295.16821, 2315.07227, ..., 2323.06592, 2300.05371, 1518.30408], [1525.58765, 2301.05591, 2301.32715, ..., 2295.82666, 2297.59155, 1528.03796], [1019.45135, 1514.56934, 1537.27295, ..., 1524.79895, 1514.35718, 1017.87750]], ..., [[1042.59082, 1558.78455, 1540.39917, ..., 1532.62415, 1531.54565, 1006.17572], [1556.28088, 2309.60254, 2331.66040, ..., 2298.56128, 2291.62891, 1515.27466], [1537.94348, 2313.97339, 2317.17090, ..., 2320.62427, 2312.17529, 1531.53271], ..., [1548.49817, 2321.03174, 2333.59424, ..., 2358.69165, 2318.95264, 1542.97827], [1546.14062, 2330.73755, 2334.29370, ..., 2320.43237, 2330.77051, 1531.67590], [1048.58398, 1560.07886, 1556.60107, ..., 1553.48120, 1550.37097, 1034.64270]], [[1036.54150, 1538.92139, 1544.46191, ..., 1522.44617, 1502.76599, 985.14758], [1545.27393, 2307.46680, 2305.38892, ..., 2309.39258, 2279.84814, 1505.66614], [1545.97083, 2300.06055, 2307.48584, ..., 2306.20557, 2285.31201, 1499.74036], ..., [1527.35767, 2305.11816, 2334.41797, ..., 2341.33032, 2322.78271, 1516.78235], [1544.79700, 2309.22510, 2312.19092, ..., 2310.57568, 2320.07642, 1526.61426], [1026.49792, 1541.20142, 1565.03284, ..., 1539.56360, 1524.25134, 1025.22888]], [[1022.60248, 1542.38635, 1559.15393, ..., 1530.99341, 1523.66174, 1001.27704], [1539.19739, 2297.34619, 2312.43677, ..., 2300.33813, 2278.90283, 1507.77405], [1523.05017, 2311.74683, 2298.18042, ..., 2305.35254, 2289.43481, 1526.15881], ..., [1524.85779, 2306.31787, 2300.59888, ..., 2335.66626, 2300.10815, 1520.62683], [1511.05579, 2303.87646, 2316.63989, ..., 2310.18262, 2307.82568, 1523.35681], [1015.65533, 1542.38525, 1547.32849, ..., 1540.47363, 1533.74268, 1031.93982]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1032., 1760., 1728., ..., 1648., 1624., 1040.], [1688., 2064., 2048., ..., 2048., 2048., 1656.], [1752., 2048., 2064., ..., 2064., 2048., 1640.], ..., [1680., 2064., 2048., ..., 2048., 2048., 1656.], [1728., 2048., 2048., ..., 2048., 2048., 1672.], [1032., 1736., 1768., ..., 1736., 1640., 1040.]], [[1024., 1720., 1688., ..., 1632., 1624., 1004.], [1752., 2048., 2048., ..., 2048., 2048., 1640.], [1688., 2048., 2064., ..., 2080., 2048., 1656.], ..., [1672., 2048., 2048., ..., 2048., 2064., 1696.], [1680., 2048., 2064., ..., 2048., 2048., 1608.], [1012., 1648., 1704., ..., 1632., 1656., 1024.]], [[1056., 1744., 1704., ..., 1648., 1680., 1012.], [1680., 2048., 2048., ..., 2048., 2064., 1736.], [1752., 2064., 2064., ..., 2048., 2048., 1696.], ..., [1712., 2048., 2048., ..., 2064., 2048., 1664.], [1672., 2048., 2048., ..., 2048., 2048., 1680.], [1024., 1696., 1680., ..., 1672., 1688., 1048.]], ..., [[1064., 1752., 1752., ..., 1672., 1712., 1012.], [1712., 2064., 2048., ..., 2080., 2048., 1688.], [1792., 2048., 2080., ..., 2064., 2048., 1696.], ..., [1760., 2048., 2048., ..., 2048., 2048., 1752.], [1728., 2064., 2064., ..., 2048., 2064., 1688.], [1064., 1728., 1776., ..., 1760., 1752., 1040.]], [[1024., 1728., 1744., ..., 1704., 1648., 1000.], [1776., 2048., 2048., ..., 2048., 2048., 1696.], [1736., 2048., 2048., ..., 2048., 2048., 1640.], ..., [1688., 2064., 2048., ..., 2048., 2048., 1704.], [1688., 2048., 2064., ..., 2048., 2064., 1640.], [1080., 1672., 1688., ..., 1648., 1656., 1064.]], [[1024., 1704., 1776., ..., 1728., 1688., 1020.], [1728., 2048., 2048., ..., 2048., 2048., 1712.], [1736., 2080., 2048., ..., 2048., 2064., 1704.], ..., [1720., 2048., 2048., ..., 2048., 2048., 1736.], [1720., 2048., 2048., ..., 2048., 2064., 1712.], [1024., 1768., 1744., ..., 1696., 1704., 1056.]]]])

test_conv2d.test_module[Conv2D664-[((1, 512, 32, 32), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 517.57800, 775.84076, 765.98688, ..., 770.32703, 770.98492, 513.54657], [ 768.79565, 1153.93091, 1166.94690, ..., 1157.18323, 1141.29126, 750.11377], [ 783.64209, 1168.05750, 1158.04590, ..., 1149.64465, 1167.47559, 768.36023], ..., [ 753.74921, 1155.05164, 1160.49548, ..., 1147.64551, 1154.75549, 766.67499], [ 757.13525, 1141.95093, 1157.87073, ..., 1138.54089, 1148.69458, 768.14020], [ 501.23648, 759.22778, 764.20074, ..., 755.56073, 769.13116, 507.01892]], [[ 536.53137, 764.09717, 776.98358, ..., 777.42163, 762.40741, 521.88177], [ 790.85181, 1160.57690, 1165.13696, ..., 1157.74524, 1149.69775, 759.05005], [ 789.13745, 1168.40918, 1158.37244, ..., 1157.25891, 1163.65613, 771.14038], ..., [ 764.91559, 1166.05090, 1167.63062, ..., 1141.73145, 1155.29578, 768.78979], [ 774.24365, 1150.04810, 1179.95300, ..., 1151.80005, 1150.33130, 769.88928], [ 514.12335, 758.61804, 766.33142, ..., 768.67328, 772.93268, 514.53619]], [[ 530.53168, 770.16394, 770.50745, ..., 763.61548, 760.64270, 504.37241], [ 776.94482, 1165.15430, 1166.32349, ..., 1153.84167, 1141.07642, 758.21539], [ 785.71832, 1159.54443, 1150.04370, ..., 1154.63696, 1159.36755, 775.53491], ..., [ 763.26501, 1151.34790, 1163.29712, ..., 1143.60571, 1152.34412, 780.20490], [ 754.68665, 1140.81299, 1153.01294, ..., 1145.91833, 1149.99915, 763.88702], [ 503.82037, 764.94440, 766.40967, ..., 777.24146, 768.04047, 512.57416]], ..., [[ 519.37756, 777.97412, 770.51044, ..., 774.83533, 767.43884, 513.01190], [ 785.24994, 1153.76208, 1158.05652, ..., 1156.92493, 1154.09326, 753.53314], [ 801.22046, 1167.51990, 1163.29248, ..., 1155.60400, 1156.07568, 769.73480], ..., [ 762.73633, 1142.39478, 1165.93579, ..., 1147.26147, 1144.75122, 776.12750], [ 765.59381, 1149.95679, 1161.34058, ..., 1147.23450, 1149.43201, 762.96436], [ 509.14658, 761.45367, 773.90015, ..., 764.75861, 767.30865, 516.23657]], [[ 522.66339, 773.65253, 772.72479, ..., 771.71887, 764.81635, 511.00571], [ 780.34290, 1150.82617, 1150.59277, ..., 1150.55872, 1134.33789, 760.74963], [ 775.03851, 1163.14526, 1159.91602, ..., 1149.12561, 1154.55920, 773.67053], ..., [ 758.48456, 1153.67566, 1162.92529, ..., 1141.39221, 1142.72864, 771.26740], [ 754.62054, 1136.96436, 1153.00220, ..., 1133.32642, 1150.40308, 771.28198], [ 505.30200, 767.72955, 768.44525, ..., 766.72864, 762.66925, 511.60654]], [[ 521.63831, 772.64825, 761.92987, ..., 760.63922, 766.65912, 507.09122], [ 772.75293, 1158.01123, 1158.43945, ..., 1156.47668, 1140.44177, 757.23230], [ 776.92969, 1161.80420, 1161.13025, ..., 1156.49500, 1153.03149, 770.18719], ..., [ 760.55609, 1146.79993, 1171.69629, ..., 1140.20142, 1146.34131, 759.74707], [ 755.45367, 1144.24365, 1151.44775, ..., 1151.55237, 1145.12512, 766.75891], [ 508.36099, 766.40265, 765.96393, ..., 771.60522, 775.40277, 514.61914]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 512., 780., 764., ..., 772., 776., 506.], [ 768., 1184., 1216., ..., 1200., 1208., 756.], [ 772., 1192., 1160., ..., 1160., 1224., 772.], ..., [ 760., 1184., 1216., ..., 1168., 1208., 764.], [ 760., 1192., 1208., ..., 1176., 1216., 776.], [ 500., 772., 768., ..., 764., 772., 512.]], [[ 536., 764., 780., ..., 776., 768., 524.], [ 788., 1216., 1160., ..., 1160., 1192., 772.], [ 780., 1192., 1224., ..., 1200., 1224., 772.], ..., [ 764., 1216., 1200., ..., 1176., 1200., 760.], [ 768., 1208., 1200., ..., 1216., 1200., 776.], [ 510., 760., 764., ..., 760., 788., 508.]], [[ 532., 780., 792., ..., 788., 772., 508.], [ 780., 1224., 1192., ..., 1168., 1152., 768.], [ 796., 1168., 1208., ..., 1208., 1200., 780.], ..., [ 776., 1224., 1208., ..., 1192., 1192., 780.], [ 768., 1160., 1208., ..., 1176., 1224., 780.], [ 504., 764., 776., ..., 784., 768., 512.]], ..., [[ 508., 780., 776., ..., 768., 764., 516.], [ 776., 1192., 1216., ..., 1208., 1184., 760.], [ 792., 1208., 1216., ..., 1184., 1200., 768.], ..., [ 752., 1192., 1200., ..., 1176., 1184., 784.], [ 764., 1192., 1208., ..., 1176., 1176., 756.], [ 508., 768., 788., ..., 768., 764., 520.]], [[ 520., 780., 780., ..., 776., 768., 508.], [ 788., 1176., 1216., ..., 1192., 1168., 768.], [ 776., 1232., 1248., ..., 1200., 1200., 792.], ..., [ 772., 1216., 1232., ..., 1176., 1176., 772.], [ 764., 1160., 1176., ..., 1168., 1192., 760.], [ 512., 776., 768., ..., 776., 764., 512.]], [[ 516., 776., 780., ..., 768., 764., 508.], [ 776., 1144., 1184., ..., 1184., 1176., 768.], [ 792., 1176., 1200., ..., 1216., 1176., 776.], ..., [ 756., 1176., 1192., ..., 1176., 1176., 756.], [ 752., 1184., 1208., ..., 1168., 1176., 780.], [ 506., 776., 772., ..., 772., 768., 520.]]]])

test_conv2d.test_module[Conv2D780-[((1, 1024, 7, 7), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1029.88147, 1535.99963, 1536.48706, ..., 1544.77588, 1542.93030, 1020.91272], [1519.82141, 2316.50220, 2321.88550, ..., 2313.95581, 2322.57690, 1531.11084], [1543.85632, 2325.64990, 2328.94067, ..., 2311.84521, 2300.97095, 1531.83948], ..., [1556.84399, 2343.30444, 2322.29688, ..., 2289.17773, 2298.14819, 1517.39685], [1553.17908, 2347.65649, 2313.58887, ..., 2311.88892, 2292.60962, 1524.09717], [1031.99011, 1553.45715, 1549.54480, ..., 1551.93176, 1536.72046, 1019.34326]], [[1017.29907, 1530.05994, 1520.71167, ..., 1535.94104, 1530.62439, 1007.62506], [1531.39783, 2305.22974, 2298.66724, ..., 2296.57886, 2283.76001, 1518.92505], [1523.98303, 2300.97925, 2315.75244, ..., 2297.33545, 2289.36060, 1522.33740], ..., [1537.48267, 2303.77661, 2308.88232, ..., 2265.98145, 2270.69800, 1503.37866], [1542.38794, 2296.36279, 2313.79761, ..., 2299.08716, 2291.51831, 1505.70813], [1025.42383, 1532.22302, 1552.92700, ..., 1542.87622, 1531.11792, 1000.95386]], [[1036.28333, 1534.32422, 1528.67249, ..., 1532.49548, 1532.86658, 1001.81842], [1550.61584, 2303.92480, 2316.71777, ..., 2315.01733, 2296.65283, 1530.74634], [1548.13318, 2307.73291, 2323.79663, ..., 2306.71216, 2289.64648, 1508.28674], ..., [1558.13000, 2315.17334, 2311.61328, ..., 2275.99438, 2284.92041, 1501.46106], [1566.22192, 2305.99243, 2296.18481, ..., 2305.74927, 2278.40552, 1512.22485], [1044.82056, 1549.12390, 1551.58618, ..., 1549.63208, 1539.80872, 1015.84009]], ..., [[ 990.46039, 1515.11108, 1504.50378, ..., 1532.03784, 1527.14844, 1012.46283], [1525.23022, 2295.80811, 2316.64697, ..., 2309.71777, 2300.40894, 1515.00244], [1520.39392, 2320.42407, 2324.26123, ..., 2296.94116, 2294.93115, 1516.20593], ..., [1541.31616, 2312.44556, 2309.22095, ..., 2274.88208, 2280.49341, 1517.56384], [1548.31958, 2290.29395, 2295.01929, ..., 2308.29932, 2288.79199, 1501.33728], [1030.94824, 1545.08411, 1541.50146, ..., 1545.06519, 1543.87268, 1013.63458]], [[1024.25305, 1536.49597, 1536.80762, ..., 1545.87842, 1541.57300, 1016.71796], [1519.85352, 2315.15771, 2308.78784, ..., 2306.60742, 2311.95142, 1531.73450], [1525.78076, 2306.11670, 2321.81177, ..., 2298.07202, 2304.60718, 1531.19580], ..., [1533.89233, 2316.73340, 2314.58643, ..., 2281.54761, 2272.34326, 1535.52637], [1545.05066, 2315.41992, 2320.52588, ..., 2311.99658, 2311.72949, 1529.36829], [1022.99640, 1531.60400, 1531.10669, ..., 1529.96936, 1521.94165, 1009.39423]], [[1021.18256, 1525.62415, 1540.57703, ..., 1557.61621, 1534.14709, 1026.57056], [1541.25720, 2304.25635, 2331.72656, ..., 2317.91309, 2307.34473, 1532.02454], [1536.19214, 2338.86938, 2337.93579, ..., 2323.28296, 2319.41992, 1519.37122], ..., [1560.15881, 2333.81323, 2327.63062, ..., 2295.04712, 2292.70166, 1516.71960], [1542.51428, 2319.19385, 2323.85840, ..., 2308.02075, 2302.53564, 1523.88281], [1026.39856, 1540.07117, 1555.68420, ..., 1556.31201, 1554.41101, 1024.64075]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1040., 1696., 1640., ..., 1624., 1704., 1040.], [1640., 2048., 2048., ..., 2048., 2048., 1728.], [1680., 2064., 2080., ..., 2064., 2048., 1704.], ..., [1752., 2080., 2064., ..., 2048., 2048., 1688.], [1712., 2048., 2064., ..., 2048., 2064., 1688.], [1064., 1696., 1656., ..., 1704., 1672., 1040.]], [[1024., 1696., 1696., ..., 1648., 1688., 1020.], [1760., 2048., 2048., ..., 2048., 2064., 1688.], [1696., 2048., 2064., ..., 2048., 2048., 1688.], ..., [1728., 2048., 2048., ..., 2048., 2064., 1640.], [1680., 2048., 2048., ..., 2048., 2048., 1656.], [1024., 1664., 1728., ..., 1752., 1696., 1024.]], [[1040., 1664., 1688., ..., 1624., 1680., 1008.], [1792., 2064., 2048., ..., 2048., 2048., 1712.], [1768., 2064., 2064., ..., 2048., 2048., 1640.], ..., [1696., 2048., 2064., ..., 2048., 2048., 1560.], [1688., 2048., 2048., ..., 2048., 2048., 1616.], [1048., 1632., 1680., ..., 1744., 1648., 1040.]], ..., [[1000., 1696., 1632., ..., 1672., 1712., 1016.], [1696., 2048., 2048., ..., 2048., 2048., 1672.], [1688., 2048., 2080., ..., 2048., 2048., 1704.], ..., [1720., 2048., 2048., ..., 2048., 2048., 1640.], [1760., 2048., 2048., ..., 2048., 2048., 1680.], [1024., 1696., 1688., ..., 1744., 1696., 1032.]], [[1024., 1664., 1712., ..., 1752., 1760., 1012.], [1688., 2064., 2080., ..., 2064., 2064., 1696.], [1776., 2064., 2064., ..., 2048., 2064., 1648.], ..., [1688., 2048., 2048., ..., 2048., 2064., 1648.], [1768., 2064., 2048., ..., 2064., 2064., 1672.], [1032., 1704., 1616., ..., 1680., 1664., 1024.]], [[1024., 1680., 1696., ..., 1672., 1752., 1024.], [1696., 2064., 2080., ..., 2048., 2048., 1688.], [1728., 2048., 2064., ..., 2064., 2064., 1688.], ..., [1720., 2080., 2048., ..., 2048., 2080., 1688.], [1680., 2080., 2048., ..., 2048., 2048., 1648.], [1024., 1728., 1720., ..., 1712., 1656., 1072.]]]])

test_conv2d.test_module[Conv2D821-[((1, 3, 640, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 73859072 B L1 buffer across 64 banks, where each bank needs to store 1154048 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7fbb512207e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7fbb511ebe9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7fbb40109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7fbb40109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7fbb416ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464]

test_conv2d.test_module[Conv2D796-[((1, 3, 480, 480), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 4677 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 197632 and static circular buffer region ends at 570656 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7fbb512372d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab]

test_conv2d.test_module[Conv2D882-[((1, 512, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 525.58246, 783.88794, 767.01532, ..., 747.95215, 752.32050, 757.83563], [ 762.52100, 1110.56079, 1142.11621, ..., 1136.47021, 1143.87061, 1126.66956], [ 780.39954, 1125.80920, 1118.03711, ..., 1141.56042, 1144.62781, 1140.71167], ..., [ 771.29150, 1134.47314, 1131.57764, ..., 1138.23901, 1146.53503, 1144.38745], [ 769.11377, 1151.86316, 1148.59143, ..., 1131.17529, 1131.81799, 1130.34094], [ 775.21283, 1141.98792, 1136.81995, ..., 1139.86853, 1143.07666, 1136.47449]], [[ 516.03619, 789.81396, 775.77484, ..., 753.84399, 767.01611, 762.30908], [ 754.72101, 1131.59021, 1143.27637, ..., 1145.83496, 1138.32629, 1135.55896], [ 753.56470, 1139.67883, 1125.88623, ..., 1142.15430, 1156.82983, 1149.36121], ..., [ 764.95026, 1153.46021, 1128.09509, ..., 1146.77832, 1160.69263, 1141.53003], [ 769.73199, 1141.04443, 1135.28857, ..., 1162.66321, 1151.39221, 1136.54675], [ 760.14319, 1151.27563, 1134.78442, ..., 1144.05334, 1136.15784, 1154.14966]], [[ 522.05518, 777.02222, 773.66809, ..., 764.01941, 760.96228, 769.25604], [ 755.60205, 1131.54150, 1153.67896, ..., 1153.92664, 1161.03748, 1142.49915], [ 775.55731, 1124.52856, 1137.35767, ..., 1150.89026, 1152.41101, 1160.92273], ..., [ 770.83264, 1158.07739, 1147.39270, ..., 1151.31897, 1159.88318, 1140.70007], [ 775.80682, 1166.96362, 1144.59534, ..., 1153.68396, 1154.11914, 1145.31482], [ 760.92377, 1141.40894, 1150.65393, ..., 1164.11768, 1145.93665, 1147.65161]], ..., [[ 524.28906, 773.42023, 778.15887, ..., 753.24817, 759.48944, 761.88074], [ 763.58801, 1124.02209, 1147.64600, ..., 1154.76746, 1141.45764, 1130.31470], [ 767.88965, 1146.90002, 1134.68726, ..., 1144.67480, 1149.83350, 1146.93311], ..., [ 768.47253, 1149.06702, 1135.96118, ..., 1151.35254, 1154.21436, 1146.59851], [ 766.45172, 1150.49573, 1149.72046, ..., 1152.47925, 1134.81543, 1129.82117], [ 770.19238, 1145.70667, 1145.77881, ..., 1152.69702, 1135.58484, 1141.84766]], [[ 521.36267, 776.71631, 772.07318, ..., 761.40570, 755.14783, 771.35150], [ 760.94885, 1133.97974, 1152.92480, ..., 1151.31934, 1153.30017, 1145.64807], [ 775.52649, 1140.75208, 1127.18982, ..., 1139.82495, 1154.77966, 1160.83569], ..., [ 765.87939, 1152.94324, 1159.71082, ..., 1149.05518, 1167.97522, 1154.80640], [ 774.93225, 1167.23962, 1143.86621, ..., 1164.95508, 1143.16626, 1143.59558], [ 768.60126, 1144.31165, 1157.73059, ..., 1152.53699, 1155.10852, 1157.77991]], [[ 512.85645, 765.04358, 765.70520, ..., 744.70239, 742.31226, 756.23431], [ 763.25659, 1125.87122, 1134.33533, ..., 1136.58374, 1151.25989, 1136.45801], [ 761.45490, 1141.16553, 1117.80933, ..., 1138.96240, 1137.87146, 1144.58167], ..., [ 772.23889, 1144.58374, 1132.89551, ..., 1144.84631, 1156.40515, 1149.44641], [ 775.15155, 1146.15649, 1145.37939, ..., 1148.91602, 1136.15247, 1129.10706], [ 772.09808, 1145.13831, 1141.86621, ..., 1150.41675, 1137.00159, 1145.74011]]]]), compiled_model=tensor([[[[ 532., 780., 776., ..., 756., 752., 756.], [ 764., 1160., 1200., ..., 1184., 1184., 1176.], [ 780., 1192., 1176., ..., 1176., 1176., 1224.], ..., [ 780., 1216., 1176., ..., 1192., 1216., 1200.], [ 764., 1200., 1176., ..., 1200., 1192., 1192.], [ 776., 1184., 1208., ..., 1184., 1184., 1192.]], [[ 508., 788., 768., ..., 756., 764., 756.], [ 756., 1176., 1200., ..., 1192., 1208., 1184.], [ 756., 1184., 1176., ..., 1208., 1168., 1208.], ..., [ 760., 1200., 1120., ..., 1152., 1216., 1216.], [ 768., 1176., 1192., ..., 1224., 1224., 1200.], [ 756., 1208., 1192., ..., 1192., 1168., 1216.]], [[ 516., 780., 784., ..., 772., 772., 772.], [ 764., 1184., 1248., ..., 1208., 1232., 1208.], [ 792., 1160., 1184., ..., 1168., 1192., 1232.], ..., [ 768., 1240., 1168., ..., 1184., 1232., 1200.], [ 776., 1264., 1168., ..., 1192., 1232., 1216.], [ 772., 1208., 1216., ..., 1224., 1176., 1208.]], ..., [[ 524., 772., 780., ..., 756., 756., 768.], [ 768., 1208., 1224., ..., 1224., 1240., 1216.], [ 792., 1176., 1200., ..., 1192., 1224., 1216.], ..., [ 764., 1216., 1184., ..., 1208., 1240., 1224.], [ 760., 1224., 1232., ..., 1208., 1200., 1192.], [ 776., 1176., 1184., ..., 1224., 1168., 1176.]], [[ 524., 792., 780., ..., 756., 760., 768.], [ 768., 1200., 1200., ..., 1184., 1200., 1176.], [ 792., 1160., 1168., ..., 1184., 1176., 1224.], ..., [ 760., 1208., 1176., ..., 1168., 1184., 1200.], [ 772., 1200., 1168., ..., 1208., 1216., 1200.], [ 772., 1160., 1192., ..., 1200., 1224., 1208.]], [[ 512., 768., 772., ..., 756., 744., 768.], [ 752., 1120., 1216., ..., 1128., 1192., 1152.], [ 772., 1176., 1152., ..., 1168., 1192., 1232.], ..., [ 768., 1176., 1200., ..., 1224., 1200., 1200.], [ 764., 1192., 1192., ..., 1200., 1192., 1168.], [ 776., 1192., 1192., ..., 1176., 1184., 1200.]]]])

test_conv2d.test_module[Conv2D881-[((1, 3, 480, 480), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5261 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 197632 and static circular buffer region ends at 570656 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7fbb512372d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab]

test_conv2d.test_module[Conv2D848-[((1, 3, 640, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 73859072 B L1 buffer across 64 banks, where each bank needs to store 1154048 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7fbb512207e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7fbb511ebe9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7fbb40109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7fbb40109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7fbb416ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464]

test_conv2d.test_module[Conv2D918-[((1, 3, 448, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5511 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=7)]. L1 buffer allocated at 194432 and static circular buffer region ends at 673056 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7fbb512372d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab]

test_conv2d.test_module[Conv2D475-[((1, 48, 320, 320), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5946 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 413440 and static circular buffer region ends at 525728 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7fbb512372d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7fbb416ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7fbb416ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7fbb416ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7fbb416c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7fbb4010869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7fbb416d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7fbb416d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7fbb3f540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7fbb3f4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7fbb51dd8365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7fbb52247c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7fbb52247b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7fbb52012b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x562da2acdb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x562da2abda97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x562da2ac3574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x562da2adc172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x562da2ab897f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x562da2adc10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x562da2ab8790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x562da2adb4d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x562da2abd702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x562da2ac361d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x562da2ad862c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x562da2be0464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x562da2ac439b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x562da2abe99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x562da2abac30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x562da2ace38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x562da2ab99ab]

test_conv2d.test_module[Conv2D999-[((1, 320, 80, 80), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[333.50986, 497.81186, 479.87344, ..., 493.99860, 491.04401, 472.69583], [486.65344, 724.52570, 712.26727, ..., 733.45154, 733.95581, 711.10992], [501.20642, 717.88751, 731.44110, ..., 726.65729, 730.56934, 719.31531], ..., [483.02588, 724.19110, 733.08136, ..., 739.07385, 720.43494, 711.99652], [493.88528, 737.57812, 736.58191, ..., 719.79199, 721.53540, 720.91516], [486.84531, 728.93280, 735.62457, ..., 729.76514, 720.50690, 716.21857]], [[335.19482, 496.40094, 481.78943, ..., 495.97427, 496.60641, 478.11218], [491.25797, 725.42560, 718.34186, ..., 735.34467, 739.54846, 721.60950], [493.94748, 719.93616, 718.67859, ..., 729.08203, 728.98627, 727.07129], ..., [481.28583, 727.51575, 723.43103, ..., 730.80365, 717.67535, 711.77606], [493.26144, 742.63422, 725.72424, ..., 730.66754, 730.59106, 722.03992], [484.54739, 729.90472, 736.30792, ..., 738.63641, 714.50568, 723.39215]], [[332.32150, 494.19730, 468.84015, ..., 486.24854, 484.31049, 461.85583], [492.24582, 725.35712, 705.04211, ..., 738.54333, 732.28314, 715.52338], [489.84308, 720.36127, 716.12231, ..., 721.33691, 721.49286, 716.15204], ..., [482.62888, 715.89801, 728.41168, ..., 726.31158, 708.59125, 709.02167], [483.09814, 734.24780, 731.38562, ..., 718.71100, 724.02747, 727.80670], [479.64746, 715.51392, 735.75214, ..., 730.93445, 720.22705, 712.24963]], ..., [[326.85980, 499.83499, 480.06073, ..., 492.39597, 491.04510, 474.64441], [483.91000, 725.31494, 716.32050, ..., 737.08453, 742.92554, 728.95111], [487.74448, 717.81860, 720.91705, ..., 722.30353, 720.86011, 725.29675], ..., [478.71826, 725.24762, 727.74646, ..., 736.63239, 718.69611, 708.62689], [484.72253, 736.09277, 733.79541, ..., 731.35431, 731.36780, 721.99738], [476.59137, 728.70624, 732.45642, ..., 738.41821, 717.97351, 715.52209]], [[325.68829, 491.63861, 480.54718, ..., 489.95239, 490.64078, 463.88037], [482.37381, 714.26636, 700.11218, ..., 731.70984, 724.45215, 706.39911], [478.93881, 703.22809, 729.49695, ..., 718.68921, 709.32629, 711.60132], ..., [475.73904, 712.14496, 712.49255, ..., 726.15485, 702.03235, 704.10010], [483.02109, 721.05762, 718.46796, ..., 716.19202, 720.20544, 713.87787], [473.20911, 718.94287, 724.58356, ..., 718.79181, 704.41730, 711.59839]], [[331.08655, 497.62561, 469.49258, ..., 490.22079, 482.13058, 467.62717], [482.60925, 720.35358, 693.60651, ..., 732.25610, 722.62683, 716.15454], [491.93643, 712.54108, 712.44287, ..., 718.60004, 719.85577, 717.61725], ..., [477.44159, 707.03894, 721.29425, ..., 729.64502, 704.38593, 701.12781], [486.26315, 729.63922, 718.46448, ..., 719.50879, 709.77759, 708.15826], [479.31177, 711.67401, 728.45160, ..., 726.01849, 712.66547, 703.22260]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[328., 498., 480., ..., 494., 488., 472.], [486., 736., 716., ..., 740., 736., 708.], [492., 736., 720., ..., 728., 732., 728.], ..., [484., 724., 732., ..., 724., 716., 720.], [480., 728., 736., ..., 716., 720., 724.], [488., 728., 728., ..., 728., 724., 724.]], [[338., 492., 476., ..., 494., 506., 472.], [486., 744., 728., ..., 736., 740., 728.], [486., 740., 736., ..., 736., 728., 736.], ..., [482., 732., 736., ..., 740., 720., 736.], [480., 740., 724., ..., 744., 752., 724.], [474., 732., 740., ..., 748., 720., 724.]], [[328., 490., 468., ..., 482., 482., 466.], [492., 720., 708., ..., 732., 732., 732.], [486., 732., 724., ..., 720., 716., 720.], ..., [476., 708., 732., ..., 736., 704., 720.], [486., 744., 728., ..., 724., 712., 716.], [476., 720., 736., ..., 732., 720., 724.]], ..., [[324., 502., 478., ..., 490., 486., 466.], [482., 720., 712., ..., 740., 732., 712.], [498., 724., 708., ..., 712., 716., 732.], ..., [468., 720., 712., ..., 736., 712., 700.], [476., 728., 732., ..., 744., 716., 724.], [482., 732., 732., ..., 728., 740., 720.]], [[322., 486., 476., ..., 484., 486., 456.], [488., 716., 716., ..., 736., 724., 712.], [476., 712., 724., ..., 724., 700., 724.], ..., [480., 724., 720., ..., 728., 704., 724.], [486., 720., 720., ..., 728., 728., 716.], [474., 724., 736., ..., 724., 724., 724.]], [[332., 498., 462., ..., 492., 488., 468.], [478., 724., 704., ..., 740., 720., 720.], [484., 724., 724., ..., 728., 724., 732.], ..., [478., 716., 724., ..., 712., 712., 724.], [482., 732., 744., ..., 728., 716., 716.], [476., 728., 740., ..., 744., 720., 708.]]]])

test_conv2d.test_module[Conv2D1002-[((1, 640, 40, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 654.50519, 973.59009, 969.08258, ..., 955.14825, 958.49182, 959.39984], [ 969.73499, 1435.56506, 1451.75220, ..., 1445.22778, 1429.00342, 1429.97485], [ 989.21436, 1437.22729, 1436.42517, ..., 1440.89038, 1444.17676, 1449.64124], ..., [ 981.92206, 1456.28479, 1456.32446, ..., 1434.30029, 1462.28516, 1441.13770], [ 978.96320, 1453.90051, 1443.57117, ..., 1437.06238, 1433.77966, 1429.47729], [ 967.86401, 1429.91614, 1435.22778, ..., 1439.87659, 1441.83618, 1441.99561]], [[ 679.50281, 996.71301, 986.07324, ..., 969.56970, 969.85706, 978.18726], [ 986.27905, 1437.59717, 1456.14343, ..., 1476.72449, 1439.32141, 1430.29053], [ 990.72406, 1439.52722, 1415.56702, ..., 1447.38513, 1458.04089, 1451.41711], ..., [ 985.39746, 1473.47461, 1446.93262, ..., 1437.40076, 1472.49536, 1448.40649], [ 987.77728, 1461.63599, 1449.19556, ..., 1445.95154, 1445.45422, 1438.33923], [ 976.26086, 1440.06555, 1453.22241, ..., 1462.93640, 1447.16882, 1455.15015]], [[ 651.64880, 976.19904, 965.94727, ..., 950.44476, 947.63330, 948.88245], [ 956.59674, 1421.86658, 1447.33081, ..., 1445.15161, 1439.00317, 1426.13757], [ 952.34601, 1425.62537, 1419.71484, ..., 1445.53821, 1444.77319, 1450.20471], ..., [ 957.41168, 1457.22119, 1455.69861, ..., 1422.75037, 1451.20081, 1436.44629], [ 952.13116, 1452.26624, 1432.20776, ..., 1434.53918, 1428.67896, 1411.73254], [ 951.92804, 1426.70215, 1432.68665, ..., 1439.23425, 1432.60168, 1448.67090]], ..., [[ 656.55603, 997.41443, 977.83557, ..., 966.60406, 964.34613, 974.37024], [ 964.19464, 1429.62622, 1452.58911, ..., 1442.04126, 1441.99072, 1431.30896], [ 963.68512, 1445.38940, 1433.76917, ..., 1449.48657, 1454.68652, 1438.76343], ..., [ 978.67596, 1461.70178, 1435.85193, ..., 1423.86816, 1453.58789, 1441.73535], [ 965.73059, 1450.66553, 1434.49744, ..., 1450.17468, 1435.81152, 1429.71423], [ 964.80658, 1429.89856, 1447.00513, ..., 1440.13489, 1427.56824, 1436.56189]], [[ 651.84949, 971.50330, 958.67767, ..., 951.61224, 943.73639, 947.55682], [ 960.81171, 1435.79016, 1444.18945, ..., 1432.51428, 1438.14868, 1428.24219], [ 971.15021, 1438.88647, 1422.40381, ..., 1447.54590, 1439.08655, 1443.99231], ..., [ 968.26324, 1458.63538, 1443.93201, ..., 1428.23572, 1445.60791, 1440.41809], [ 972.50659, 1451.25085, 1439.47900, ..., 1434.05115, 1420.21875, 1416.09717], [ 949.56647, 1431.43665, 1441.24438, ..., 1440.79382, 1419.00879, 1432.83154]], [[ 649.17719, 978.26575, 970.31433, ..., 955.67944, 959.25519, 967.92847], [ 954.30151, 1433.49023, 1452.95227, ..., 1457.49939, 1454.38879, 1435.56970], [ 972.30945, 1453.92468, 1437.42834, ..., 1456.75720, 1457.09900, 1451.26257], ..., [ 967.75555, 1462.58069, 1441.38708, ..., 1443.56396, 1468.27136, 1461.04041], [ 972.40674, 1466.32471, 1436.04102, ..., 1456.26562, 1443.54932, 1429.04968], [ 959.96594, 1447.67688, 1451.35669, ..., 1460.09131, 1442.24463, 1452.40063]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[ 648., 968., 972., ..., 964., 960., 940.], [ 952., 1560., 1600., ..., 1560., 1536., 1584.], [ 988., 1584., 1568., ..., 1608., 1536., 1672.], ..., [ 964., 1648., 1568., ..., 1560., 1632., 1624.], [ 996., 1632., 1616., ..., 1576., 1624., 1584.], [ 972., 1544., 1584., ..., 1584., 1584., 1544.]], [[ 684., 1000., 980., ..., 988., 956., 976.], [ 992., 1584., 1608., ..., 1608., 1616., 1576.], [ 988., 1544., 1568., ..., 1600., 1592., 1616.], ..., [ 988., 1632., 1568., ..., 1584., 1632., 1560.], [1016., 1632., 1568., ..., 1520., 1592., 1544.], [ 988., 1584., 1600., ..., 1608., 1520., 1600.]], [[ 636., 968., 972., ..., 964., 956., 964.], [ 964., 1512., 1576., ..., 1592., 1552., 1528.], [ 956., 1576., 1592., ..., 1608., 1496., 1576.], ..., [ 960., 1616., 1568., ..., 1504., 1576., 1568.], [ 960., 1560., 1536., ..., 1520., 1544., 1536.], [ 984., 1560., 1544., ..., 1528., 1600., 1520.]], ..., [[ 652., 992., 992., ..., 980., 964., 980.], [ 964., 1552., 1584., ..., 1552., 1560., 1528.], [ 984., 1568., 1560., ..., 1536., 1520., 1512.], ..., [ 976., 1600., 1544., ..., 1536., 1592., 1560.], [ 968., 1528., 1536., ..., 1520., 1488., 1496.], [ 972., 1552., 1560., ..., 1528., 1520., 1520.]], [[ 656., 968., 976., ..., 976., 956., 964.], [ 980., 1632., 1656., ..., 1600., 1584., 1560.], [ 988., 1576., 1568., ..., 1624., 1584., 1608.], ..., [ 956., 1616., 1688., ..., 1616., 1600., 1608.], [ 984., 1664., 1552., ..., 1624., 1608., 1576.], [ 960., 1592., 1584., ..., 1552., 1528., 1568.]], [[ 656., 984., 972., ..., 980., 960., 972.], [ 980., 1576., 1624., ..., 1584., 1568., 1592.], [ 980., 1632., 1592., ..., 1592., 1608., 1600.], ..., [ 968., 1656., 1584., ..., 1616., 1728., 1616.], [ 984., 1624., 1552., ..., 1600., 1568., 1584.], [ 972., 1592., 1568., ..., 1560., 1552., 1616.]]]])

test_conv2dtranspose.test_module[Conv2Dtranspose5-[((1, 512, 16, 16), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_conv2dtranspose.test_module[Conv2Dtranspose9-[((1, 128, 14, 20), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_conv2dtranspose.test_module[Conv2Dtranspose10-[((1, 64, 28, 40), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_conv2dtranspose.test_module[Conv2Dtranspose14-[((1, 96, 28, 40), torch.float32)]]: forge/test/models_ops/test_conv2dtranspose.py#L539

RuntimeError: Found Unsupported operations while lowering from TTForge to TTIR in forward graph

test_cumsum.test_module[Cumsum0-[((1, 32), torch.int64)]]: forge/test/models_ops/test_cumsum.py#L114

TypeError: Dtype mismatch: framework_model.dtype=torch.int64, compiled_model.dtype=torch.int32

test_embedding.test_module[Embedding1-[((2, 13), torch.int64)]]: forge/test/models_ops/test_embedding.py#L1690

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[1.86923e-01, 1.59787e-01, 6.04143e-01, ..., 7.83378e-03, 2.48977e-01, 4.59483e-01], [2.15342e-01, 9.78127e-01, 7.53613e-01, ..., 2.47074e-01, 2.05434e-01, 4.70366e-01], [3.15830e-01, 2.64033e-01, 1.32186e-01, ..., 8.80155e-02, 5.44143e-01, 4.19452e-01], ..., [1.47781e-01, 4.80219e-01, 7.28108e-01, ..., 7.30603e-02, 2.04532e-01, 8.42619e-01], [6.77595e-01, 5.33023e-01, 3.58506e-01, ..., 2.76208e-04, 9.40819e-01, 8.41029e-02], [6.92642e-01, 8.16387e-01, 8.10165e-01, ..., 4.80123e-01, 8.34061e-01, 1.22005e-02]], [[2.84620e-01, 3.57172e-02, 6.06518e-01, ..., 1.15978e-01, 7.51087e-01, 5.05164e-01], [8.99027e-01, 5.61971e-01, 8.19034e-01, ..., 2.49991e-01, 1.67163e-01, 7.98885e-01], [6.53717e-01, 1.10268e-01, 3.43148e-01, ..., 7.21886e-01, 3.44073e-01, 2.30934e-01], ..., [8.32216e-01, 7.41204e-01, 4.54824e-01, ..., 1.91212e-01, 3.21294e-01, 9.73682e-01], [5.12635e-01, 7.77861e-01, 7.25244e-01, ..., 7.01171e-01, 5.21046e-01, 5.27546e-01], [3.82292e-01, 4.57559e-01, 3.27449e-01, ..., 4.47957e-01, 1.34620e-01, 4.53345e-01]]], grad_fn=<EmbeddingBackward0>), compiled_model=tensor([[[0.18652, 0.16016, 0.60547, ..., 0.00781, 0.24902, 0.45898], [0.95312, 0.03613, 0.18555, ..., 0.66016, 0.97266, 0.63281], [0.21582, 0.97656, 0.75391, ..., 0.24707, 0.20508, 0.47070], ..., [0.09619, 0.18359, 0.60938, ..., 0.19531, 0.86719, 0.82812], [0.95312, 0.03613, 0.18555, ..., 0.66016, 0.97266, 0.63281], [0.10107, 0.27734, 0.96875, ..., 0.48242, 0.25586, 0.29297]], [[0.95312, 0.03613, 0.18555, ..., 0.66016, 0.97266, 0.63281], [0.39062, 0.81250, 0.28711, ..., 0.04175, 0.75781, 0.60156], [0.95312, 0.03613, 0.18555, ..., 0.66016, 0.97266, 0.63281], ..., [0.95312, 0.03613, 0.18555, ..., 0.66016, 0.97266, 0.63281], [0.69141, 0.81641, 0.80859, ..., 0.48047, 0.83594, 0.01221], [0.95312, 0.03613, 0.18555, ..., 0.66016, 0.97266, 0.63281]]])

test_embedding.test_module[Embedding16-[((1, 128), torch.int64)]]: forge/test/models_ops/test_embedding.py#L1690

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[0.89404, 0.22389, 0.30606, ..., 0.00873, 0.06091, 0.30932], [0.12456, 0.72381, 0.70346, ..., 0.12815, 0.32024, 0.01003], [0.85602, 0.48655, 0.26190, ..., 0.25357, 0.47941, 0.31450], ..., [0.87095, 0.78215, 0.64023, ..., 0.99527, 0.77579, 0.01249], [0.07972, 0.50737, 0.89399, ..., 0.73583, 0.82191, 0.84010], [0.41997, 0.08397, 0.80550, ..., 0.90604, 0.22669, 0.01307]]], grad_fn=<EmbeddingBackward0>), compiled_model=tensor([[[0.89453, 0.22363, 0.30664, ..., 0.00873, 0.06079, 0.30859], [0.32422, 0.09033, 0.39453, ..., 0.35352, 0.20312, 0.31445], [0.12451, 0.72266, 0.70312, ..., 0.12793, 0.32031, 0.01001], ..., [0.32422, 0.09033, 0.39453, ..., 0.35352, 0.20312, 0.31445], [0.12695, 0.44141, 0.57422, ..., 0.54297, 0.83984, 0.29102], [0.32422, 0.09033, 0.39453, ..., 0.35352, 0.20312, 0.31445]]])

test_embedding.test_module[Embedding25-[((1, 128), torch.int64)]]: forge/test/models_ops/test_embedding.py#L1690

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[0.76282, 0.73573, 0.98881, ..., 0.46936, 0.50916, 0.30093], [0.32196, 0.99742, 0.77129, ..., 0.93951, 0.87186, 0.17622], [0.82325, 0.20107, 0.68530, ..., 0.52603, 0.02782, 0.01741], ..., [0.97687, 0.39520, 0.33820, ..., 0.22140, 0.32511, 0.23587], [0.30645, 0.16769, 0.44755, ..., 0.38178, 0.76776, 0.68943], [0.90911, 0.58305, 0.84877, ..., 0.90397, 0.09448, 0.31799]]], grad_fn=<EmbeddingBackward0>), compiled_model=tensor([[[0.76172, 0.73438, 0.98828, ..., 0.46875, 0.50781, 0.30078], [0.32422, 0.09033, 0.39453, ..., 0.85156, 0.28516, 0.73047], [0.32227, 0.99609, 0.76953, ..., 0.94141, 0.87109, 0.17578], ..., [0.32422, 0.09033, 0.39453, ..., 0.85156, 0.28516, 0.73047], [0.06445, 0.46484, 0.62109, ..., 0.13184, 0.19629, 0.59375], [0.32422, 0.09033, 0.39453, ..., 0.85156, 0.28516, 0.73047]]])

test_add.test_module[Add0-[((2, 1, 1536), torch.float32), ((1, 1536), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[1.46305, 1.64772, 0.15097, ..., 1.44704, 0.58208, 1.14199]], [[1.30054, 1.54062, 0.78366, ..., 1.59789, 0.83155, 1.70888]]]), compiled_model=tensor([[[1.46305, 1.64772, 0.15097, ..., 1.44704, 0.58208, 1.14199]], [[0.55127, 1.40464, 1.50535, ..., 0.67606, 1.29079, 1.56032]]])

test_add.test_module[Add1-[((1, 120, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1.19247]], [[1.46444]], [[0.78469]], [[0.82824]], [[1.00364]], [[1.33029]], [[1.18631]], [[1.59266]], [[1.15184]], [[1.32852]], [[1.04511]], [[1.09793]], [[0.71854]], [[0.86507]], [[0.99010]], [[1.21474]], [[1.39388]], [[1.49623]], [[0.85724]], [[0.97848]], [[1.37782]], [[1.61141]], [[1.09331]], [[1.57037]], [[1.11562]], [[1.24912]], [[1.64895]], [[0.73238]], [[0.88145]], [[1.06963]], [[1.00131]], [[1.62821]], [[0.87212]], [[0.96605]], [[0.84689]], [[0.72793]], [[0.90434]], [[1.62601]], [[1.41932]], [[1.43855]], [[1.22251]], [[0.93987]], [[1.28081]], [[0.72937]], [[0.83493]], [[0.93845]], [[1.51168]], [[1.48937]], [[0.97447]], [[1.17817]], [[1.51599]], [[1.69328]], [[1.39466]], [[1.26376]], [[1.53146]], [[0.90181]], [[1.28939]], [[0.80856]], [[0.84967]], [[0.93792]], [[1.42245]], [[1.39729]], [[0.90004]], [[1.34727]], [[1.47070]], [[1.13311]], [[1.21530]], [[1.31207]], [[1.50640]], [[1.67631]], [[0.81090]], [[1.01298]], [[1.39272]], [[1.61049]], [[1.63132]], [[1.63739]], [[1.29572]], [[0.76142]], [[1.24221]], [[0.88341]], [[0.73024]], [[1.64046]], [[1.57639]], [[0.69745]], [[1.28980]], [[1.11198]], [[1.11393]], [[0.96734]], [[1.38849]], [[0.90006]], [[1.37951]], [[1.44907]], [[1.55415]], [[1.38317]], [[0.70135]], [[0.87187]], [[1.44587]], [[1.30086]], [[0.80617]], [[0.90830]], [[1.66659]], [[1.53312]], [[0.97820]], [[1.07037]], [[0.71992]], [[1.18723]], [[0.81968]], [[0.81054]], [[1.16866]], [[1.27129]], [[0.99145]], [[1.49290]], [[0.89194]], [[1.64990]], [[1.53886]], [[0.77457]], [[1.07177]], [[1.21878]], [[1.26916]], [[1.31480]]]]), compiled_model=tensor([[[[1.19247]], [[0.88645]], [[1.05062]], [[0.25224]], [[0.80515]], [[0.89165]], [[1.01510]], [[1.14246]], [[1.12305]], [[1.05228]], [[0.72287]], [[0.96620]], [[0.56481]], [[0.92926]], [[0.95542]], [[0.60888]], [[1.54558]], [[1.51324]], [[0.17087]], [[0.97140]], [[0.99726]], [[1.86706]], [[1.27806]], [[1.19303]], [[0.47413]], [[1.13465]], [[1.59371]], [[0.58324]], [[0.38426]], [[0.48389]], [[1.21301]], [[1.41348]], [[0.30296]], [[1.25971]], [[1.13215]], [[0.85620]], [[0.83184]], [[1.10168]], [[1.17336]], [[1.12241]], [[0.76701]], [[1.19555]], [[1.27382]], [[0.42117]], [[0.26978]], [[0.40025]], [[0.88376]], [[1.66387]], [[0.33423]], [[1.18018]], [[1.27005]], [[1.51186]], [[1.21396]], [[1.14292]], [[1.45363]], [[0.26290]], [[0.68834]], [[0.80919]], [[0.94430]], [[0.83815]], [[1.59420]], [[1.58657]], [[0.75130]], [[0.80426]], [[0.92476]], [[0.97658]], [[0.93037]], [[1.55024]], [[1.24342]], [[1.07228]], [[0.54854]], [[1.24801]], [[1.02019]], [[1.00353]], [[1.26938]], [[1.53428]], [[1.57611]], [[0.33169]], [[1.40012]], [[0.38466]], [[0.85288]], [[1.09263]], [[1.19151]], [[0.83317]], [[1.27349]], [[0.76997]], [[1.27634]], [[0.72479]], [[0.72643]], [[0.61959]], [[0.77278]], [[1.21435]], [[1.57704]], [[1.61828]], [[0.99367]], [[0.41147]], [[1.45836]], [[1.06526]], [[0.60250]], [[1.03359]], [[1.74770]], [[1.39604]], [[0.79348]], [[0.97109]], [[0.71943]], [[1.08718]], [[0.94863]], [[0.85712]], [[1.42033]], [[1.01835]], [[1.22276]], [[1.46275]], [[0.38112]], [[1.36592]], [[1.51686]], [[1.04821]], [[1.16537]], [[0.88235]], [[1.07502]], [[1.29689]]]])

test_add.test_module[Add1-[((1, 16, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1.19392]], [[1.46589]], [[0.78615]], [[0.82970]], [[1.00509]], [[1.33175]], [[1.18776]], [[1.59411]], [[1.15330]], [[1.32997]], [[1.04656]], [[1.09938]], [[0.71999]], [[0.86653]], [[0.99156]], [[1.21619]]]]), compiled_model=tensor([[[[1.19392]], [[0.81089]], [[0.88413]], [[0.61050]], [[1.03732]], [[1.55319]], [[1.25705]], [[1.64669]], [[1.27670]], [[0.89237]], [[1.07527]], [[0.61628]], [[0.79445]], [[0.66492]], [[0.81199]], [[1.01008]]]])

test_add.test_module[Add1-[((1, 240, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1.16135]], [[1.43332]], [[0.75357]], [[0.79713]], [[0.97252]], [[1.29917]], [[1.15519]], [[1.56154]], [[1.12072]], [[1.29740]], [[1.01399]], [[1.06681]], [[0.68742]], [[0.83395]], [[0.95898]], [[1.18362]], [[1.36276]], [[1.46511]], [[0.82613]], [[0.94736]], [[1.34670]], [[1.58029]], [[1.06220]], [[1.53925]], [[1.08450]], [[1.21800]], [[1.61783]], [[0.70126]], [[0.85033]], [[1.03851]], [[0.97020]], [[1.59710]], [[0.84101]], [[0.93493]], [[0.81578]], [[0.69682]], [[0.87323]], [[1.59490]], [[1.38821]], [[1.40743]], [[1.19139]], [[0.90875]], [[1.24969]], [[0.69825]], [[0.80381]], [[0.90733]], [[1.48057]], [[1.45826]], [[0.94335]], [[1.14705]], [[1.48488]], [[1.66216]], [[1.36354]], [[1.23264]], [[1.50034]], [[0.87069]], [[1.25827]], [[0.77744]], [[0.81855]], [[0.90680]], [[1.39133]], [[1.36618]], [[0.86892]], [[1.31615]], [[1.43958]], [[1.10199]], [[1.18419]], [[1.28095]], [[1.47528]], [[1.64519]], [[0.77978]], [[0.98186]], [[1.36160]], [[1.57937]], [[1.60020]], [[1.60627]], [[1.26460]], [[0.73030]], [[1.21109]], [[0.85229]], [[0.69912]], [[1.60934]], [[1.54528]], [[0.66633]], [[1.25868]], [[1.08087]], [[1.08282]], [[0.93622]], [[1.35737]], [[0.86894]], [[1.34839]], [[1.41795]], [[1.52303]], [[1.35205]], [[0.67023]], [[0.84075]], [[1.41475]], [[1.26975]], [[0.77505]], [[0.87719]], [[1.63547]], [[1.50200]], [[0.94708]], [[1.03925]], [[0.68880]], [[1.15611]], [[0.78857]], [[0.77942]], [[1.13755]], [[1.24017]], [[0.96033]], [[1.46178]], [[0.86083]], [[1.61878]], [[1.50775]], [[0.74345]], [[1.04065]], [[1.18766]], [[1.23805]], [[1.28368]], [[1.36131]], [[1.19505]], [[0.92113]], [[1.40169]], [[0.68547]], [[0.86874]], [[1.03993]], [[0.92154]], [[0.99018]], [[0.75529]], [[1.05874]], [[1.27197]], [[0.83936]], [[1.13944]], [[1.52302]], [[1.11370]], [[1.17899]], [[1.12196]], [[1.26629]], [[1.48302]], [[1.63872]], [[1.48262]], [[1.63980]], [[1.12894]], [[0.71594]], [[0.92806]], [[1.50555]], [[1.16185]], [[0.91657]], [[0.78194]], [[0.69717]], [[0.74309]], [[1.06368]], [[1.43930]], [[1.43542]], [[0.68288]], [[1.47699]], [[0.77384]], [[1.05939]], [[0.96236]], [[1.06879]], [[1.06692]], [[0.71642]], [[0.73338]], [[1.08686]], [[1.17156]], [[0.93796]], [[1.35345]], [[0.71507]], [[1.13135]], [[1.60481]], [[0.96115]], [[1.61660]], [[1.34617]], [[0.71387]], [[1.48144]], [[1.10740]], [[0.94189]], [[1.56492]], [[0.76105]], [[1.21875]], [[1.06041]], [[1.52215]], [[1.30467]], [[1.40535]], [[1.34168]], [[1.04486]], [[1.05994]], [[0.75306]], [[1.43602]], [[1.56209]], [[1.50721]], [[0.81241]], [[1.18740]], [[0.81263]], [[0.88985]], [[0.87374]], [[1.33597]], [[0.86714]], [[1.15419]], [[1.18613]], [[1.48741]], [[0.78714]], [[0.82184]], [[0.87477]], [[1.51506]], [[0.98536]], [[1.58684]], [[1.34590]], [[1.22841]], [[1.16137]], [[1.06626]], [[1.22783]], [[1.05092]], [[1.16158]], [[1.22889]], [[0.77399]], [[0.90303]], [[1.56884]], [[0.75932]], [[1.12919]], [[1.65972]], [[1.34571]], [[1.17925]], [[0.73179]], [[1.41279]], [[0.80896]], [[1.02316]], [[0.99734]], [[1.09105]], [[1.17057]], [[1.57750]], [[1.22752]], [[1.61294]], [[1.47095]], [[0.84899]], [[1.38935]], [[0.81165]], [[0.95318]], [[1.31216]]]]), compiled_model=tensor([[[[1.16135]], [[1.88834]], [[1.04844]], [[0.69404]], [[0.98223]], [[0.96782]], [[0.84898]], [[1.37105]], [[1.71539]], [[1.05223]], [[1.55104]], [[1.29039]], [[1.19811]], [[0.99454]], [[0.89301]], [[0.88547]], [[1.86466]], [[1.32687]], [[1.08779]], [[1.35453]], [[1.13644]], [[1.41812]], [[1.69983]], [[1.78578]], [[0.92087]], [[1.78826]], [[1.27769]], [[1.00931]], [[1.47820]], [[1.57557]], [[0.61858]], [[1.49450]], [[1.09730]], [[1.54132]], [[1.41240]], [[1.01756]], [[1.16760]], [[1.71593]], [[1.33786]], [[1.24673]], [[1.58684]], [[1.53272]], [[1.39221]], [[0.74067]], [[0.91899]], [[0.98149]], [[1.79154]], [[1.46406]], [[0.96722]], [[1.70657]], [[2.09517]], [[1.91357]], [[1.58077]], [[1.30436]], [[2.00419]], [[0.97855]], [[1.15421]], [[0.98246]], [[0.85658]], [[0.80177]], [[1.32569]], [[0.89161]], [[0.76196]], [[1.41639]], [[0.98547]], [[0.93717]], [[0.78903]], [[0.64170]], [[1.05306]], [[1.32399]], [[0.65664]], [[1.10785]], [[1.69507]], [[1.38538]], [[1.35699]], [[1.80439]], [[0.74201]], [[0.24135]], [[0.85557]], [[0.20833]], [[0.19310]], [[1.43936]], [[1.71799]], [[0.42613]], [[0.65425]], [[1.12646]], [[1.22365]], [[0.83930]], [[1.38609]], [[1.01407]], [[1.07989]], [[0.91745]], [[1.72294]], [[1.11806]], [[0.91484]], [[0.94695]], [[1.05193]], [[1.05323]], [[0.67213]], [[0.66030]], [[1.49723]], [[1.30536]], [[0.86675]], [[0.54668]], [[0.50497]], [[1.32036]], [[0.22170]], [[0.19903]], [[1.11685]], [[0.70182]], [[1.06963]], [[0.96035]], [[1.15175]], [[1.80921]], [[1.26408]], [[0.57762]], [[0.68887]], [[1.48019]], [[1.34561]], [[0.69278]], [[1.04750]], [[1.22253]], [[0.51850]], [[1.14809]], [[0.07050]], [[0.64913]], [[0.99720]], [[0.67191]], [[0.40240]], [[0.95913]], [[1.17574]], [[0.76390]], [[0.70486]], [[1.39152]], [[1.77004]], [[1.02270]], [[1.32790]], [[1.27035]], [[1.13308]], [[0.85253]], [[0.98547]], [[1.30005]], [[1.81379]], [[0.88684]], [[0.82265]], [[0.59196]], [[1.44322]], [[0.55852]], [[1.06904]], [[0.77211]], [[0.86002]], [[0.33045]], [[0.80805]], [[1.38891]], [[1.27437]], [[0.52135]], [[1.73596]], [[0.59893]], [[0.99885]], [[0.83609]], [[0.68827]], [[0.64027]], [[0.88285]], [[0.94733]], [[1.34718]], [[1.23917]], [[0.77191]], [[0.77830]], [[1.04339]], [[0.66653]], [[1.38483]], [[0.77113]], [[1.82511]], [[0.87587]], [[0.34407]], [[1.77017]], [[0.45011]], [[0.34472]], [[1.31376]], [[0.12646]], [[0.96076]], [[0.66812]], [[1.33965]], [[1.03505]], [[1.08798]], [[0.98278]], [[1.12109]], [[1.38598]], [[0.42124]], [[1.40306]], [[1.74995]], [[1.41482]], [[1.09110]], [[0.93662]], [[0.36147]], [[0.25498]], [[0.95686]], [[1.32993]], [[0.82483]], [[0.71679]], [[1.02376]], [[1.71895]], [[0.59976]], [[0.76899]], [[0.93305]], [[1.61716]], [[0.49204]], [[1.52349]], [[1.30788]], [[1.14949]], [[1.12833]], [[0.91205]], [[0.64719]], [[1.06087]], [[1.21143]], [[1.06096]], [[0.16141]], [[0.76953]], [[1.74445]], [[0.10416]], [[1.15437]], [[1.65600]], [[1.08598]], [[0.88946]], [[1.02942]], [[1.68924]], [[1.02349]], [[0.51156]], [[1.22122]], [[0.49235]], [[1.46722]], [[1.48875]], [[1.17618]], [[1.92495]], [[1.76384]], [[0.92672]], [[1.11897]], [[0.18024]], [[1.14933]], [[0.73841]]]])

test_add.test_module[Add1-[((1, 1280, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.62275]], [[0.89472]], [[0.21498]], ..., [[0.86233]], [[0.94840]], [[0.96660]]]]), compiled_model=tensor([[[[0.62275]], [[1.50405]], [[0.91038]], ..., [[1.44628]], [[1.88734]], [[1.56325]]]])

test_add.test_module[Add1-[((1, 256, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.50068]], [[0.77265]], [[0.09290]], [[0.13646]], [[0.31185]], [[0.63850]], [[0.49452]], [[0.90087]], [[0.46005]], [[0.63673]], [[0.35332]], [[0.40614]], [[0.02675]], [[0.17328]], [[0.29831]], [[0.52295]], [[0.70209]], [[0.80444]], [[0.16546]], [[0.28669]], [[0.68603]], [[0.91962]], [[0.40153]], [[0.87858]], [[0.42383]], [[0.55733]], [[0.95716]], [[0.04059]], [[0.18966]], [[0.37784]], [[0.30953]], [[0.93643]], [[0.18034]], [[0.27426]], [[0.15511]], [[0.03615]], [[0.21256]], [[0.93422]], [[0.72754]], [[0.74676]], [[0.53072]], [[0.24808]], [[0.58902]], [[0.03758]], [[0.14314]], [[0.24666]], [[0.81989]], [[0.79759]], [[0.28268]], [[0.48638]], [[0.82421]], [[1.00149]], [[0.70287]], [[0.57197]], [[0.83967]], [[0.21002]], [[0.59760]], [[0.11677]], [[0.15788]], [[0.24613]], [[0.73066]], [[0.70551]], [[0.20825]], [[0.65548]], [[0.77891]], [[0.44132]], [[0.52352]], [[0.62028]], [[0.81461]], [[0.98452]], [[0.11911]], [[0.32119]], [[0.70093]], [[0.91870]], [[0.93953]], [[0.94560]], [[0.60393]], [[0.06963]], [[0.55042]], [[0.19162]], [[0.03845]], [[0.94867]], [[0.88461]], [[0.00566]], [[0.59801]], [[0.42020]], [[0.42215]], [[0.27555]], [[0.69670]], [[0.20827]], [[0.68772]], [[0.75728]], [[0.86236]], [[0.69138]], [[0.00956]], [[0.18008]], [[0.75408]], [[0.60908]], [[0.11438]], [[0.21652]], [[0.97480]], [[0.84133]], [[0.28641]], [[0.37858]], [[0.02813]], [[0.49544]], [[0.12790]], [[0.11875]], [[0.47688]], [[0.57950]], [[0.29966]], [[0.80111]], [[0.20016]], [[0.95811]], [[0.84708]], [[0.08278]], [[0.37998]], [[0.52699]], [[0.57738]], [[0.62301]], [[0.70064]], [[0.53438]], [[0.26046]], [[0.74102]], [[0.02480]], [[0.20807]], [[0.37926]], [[0.26087]], [[0.32951]], [[0.09461]], [[0.39807]], [[0.61130]], [[0.17869]], [[0.47877]], [[0.86235]], [[0.45303]], [[0.51832]], [[0.46129]], [[0.60562]], [[0.82235]], [[0.97805]], [[0.82195]], [[0.97913]], [[0.46826]], [[0.05527]], [[0.26739]], [[0.84488]], [[0.50118]], [[0.25590]], [[0.12127]], [[0.03650]], [[0.08242]], [[0.40301]], [[0.77863]], [[0.77475]], [[0.02221]], [[0.81632]], [[0.11317]], [[0.39872]], [[0.30169]], [[0.40812]], [[0.40625]], [[0.05575]], [[0.07271]], [[0.42619]], [[0.51089]], [[0.27729]], [[0.69278]], [[0.05440]], [[0.47068]], [[0.94414]], [[0.30048]], [[0.95593]], [[0.68550]], [[0.05320]], [[0.82077]], [[0.44673]], [[0.28122]], [[0.90425]], [[0.10038]], [[0.55808]], [[0.39974]], [[0.86148]], [[0.64400]], [[0.74468]], [[0.68101]], [[0.38419]], [[0.39927]], [[0.09239]], [[0.77535]], [[0.90141]], [[0.84654]], [[0.15174]], [[0.52673]], [[0.15196]], [[0.22918]], [[0.21307]], [[0.67530]], [[0.20647]], [[0.49352]], [[0.52546]], [[0.82674]], [[0.12647]], [[0.16117]], [[0.21410]], [[0.85439]], [[0.32469]], [[0.92617]], [[0.68523]], [[0.56774]], [[0.50070]], [[0.40559]], [[0.56716]], [[0.39025]], [[0.50091]], [[0.56822]], [[0.11332]], [[0.24236]], [[0.90817]], [[0.09865]], [[0.46852]], [[0.99905]], [[0.68504]], [[0.51858]], [[0.07112]], [[0.75211]], [[0.14829]], [[0.36249]], [[0.33667]], [[0.43038]], [[0.50989]], [[0.91683]], [[0.56685]], [[0.95227]], [[0.81028]], [[0.18832]], [[0.72868]], [[0.15098]], [[0.29251]], [[0.65149]], [[0.66952]], [[0.87954]], [[0.34347]], [[0.50523]], [[0.76184]], [[0.02088]], [[0.86592]], [[0.09096]], [[0.51132]], [[0.41942]], [[0.24109]], [[0.57051]], [[0.91789]], [[0.35827]], [[0.20758]], [[0.31951]]]]), compiled_model=tensor([[[[0.50068]], [[1.66736]], [[0.61506]], [[0.77824]], [[0.50798]], [[1.40919]], [[1.24568]], [[1.73415]], [[1.41553]], [[0.72227]], [[0.84089]], [[0.84058]], [[0.81801]], [[0.34920]], [[0.45861]], [[0.91695]], [[0.76117]], [[1.77579]], [[0.76048]], [[1.07789]], [[1.62021]], [[1.23809]], [[1.12287]], [[1.15404]], [[0.84997]], [[1.38151]], [[1.40059]], [[0.34342]], [[0.84597]], [[1.36642]], [[0.89795]], [[1.81910]], [[0.28698]], [[0.49568]], [[0.96852]], [[0.48066]], [[0.53744]], [[0.96015]], [[1.71823]], [[1.57767]], [[1.51489]], [[1.04878]], [[0.73478]], [[0.29764]], [[0.53432]], [[0.86426]], [[1.00263]], [[0.96273]], [[1.22508]], [[0.54518]], [[1.63621]], [[1.67307]], [[1.29252]], [[0.61439]], [[1.48353]], [[0.79036]], [[1.00001]], [[0.31336]], [[1.04427]], [[0.54720]], [[1.37976]], [[1.28581]], [[1.19129]], [[1.17740]], [[1.17894]], [[0.46437]], [[0.53875]], [[0.63151]], [[0.93888]], [[1.67275]], [[0.69570]], [[0.33615]], [[0.77121]], [[1.42091]], [[1.47034]], [[1.11867]], [[1.25087]], [[0.67321]], [[1.27069]], [[0.51703]], [[0.47617]], [[1.23646]], [[1.14940]], [[0.56477]], [[0.60903]], [[0.94881]], [[0.45967]], [[0.31778]], [[0.85186]], [[0.55768]], [[0.80272]], [[1.44452]], [[1.82080]], [[1.11441]], [[0.59373]], [[1.03362]], [[0.89149]], [[1.35792]], [[0.50328]], [[0.64614]], [[1.57983]], [[1.46551]], [[1.07941]], [[0.90426]], [[0.96290]], [[0.71068]], [[0.29255]], [[0.62431]], [[0.92686]], [[1.37324]], [[0.72483]], [[1.71307]], [[0.87654]], [[1.41423]], [[1.65419]], [[0.55955]], [[1.36853]], [[0.85803]], [[0.67870]], [[0.64059]], [[0.80842]], [[0.93804]], [[0.42523]], [[1.55300]], [[0.58695]], [[0.53092]], [[0.88526]], [[0.40340]], [[0.47740]], [[0.23060]], [[0.61573]], [[1.21384]], [[0.35662]], [[0.60979]], [[1.42925]], [[0.88468]], [[1.48038]], [[1.10898]], [[1.27853]], [[1.68319]], [[1.10850]], [[0.90302]], [[1.25227]], [[1.09643]], [[0.82202]], [[1.23944]], [[1.76656]], [[0.79487]], [[0.34761]], [[0.68499]], [[0.26012]], [[0.89185]], [[0.98518]], [[0.90449]], [[1.38741]], [[0.48796]], [[1.25531]], [[0.37345]], [[0.92255]], [[0.61814]], [[0.76070]], [[0.84003]], [[0.65733]], [[0.55053]], [[0.91342]], [[1.15661]], [[1.18662]], [[1.36661]], [[0.89167]], [[1.31173]], [[1.61075]], [[1.16345]], [[1.38161]], [[0.82519]], [[0.68860]], [[1.68826]], [[0.63125]], [[0.45684]], [[1.29671]], [[1.06345]], [[1.41745]], [[1.14424]], [[1.68817]], [[1.43355]], [[0.94213]], [[1.35729]], [[1.37888]], [[0.77771]], [[0.50447]], [[1.28544]], [[1.26375]], [[1.82459]], [[0.58149]], [[0.87929]], [[0.61313]], [[0.62836]], [[0.53655]], [[1.22280]], [[0.77170]], [[1.48532]], [[0.76121]], [[1.78579]], [[0.72625]], [[0.22813]], [[0.98595]], [[1.23827]], [[0.99940]], [[1.91326]], [[1.40764]], [[0.84604]], [[1.11993]], [[0.89951]], [[0.85669]], [[0.73036]], [[1.06118]], [[1.21443]], [[0.54329]], [[0.81712]], [[1.44596]], [[1.06725]], [[0.47168]], [[1.07262]], [[0.72745]], [[0.88521]], [[0.61615]], [[1.26374]], [[0.82342]], [[1.16552]], [[0.90905]], [[0.78083]], [[1.21571]], [[1.71690]], [[0.59332]], [[1.41803]], [[1.35602]], [[0.60986]], [[1.44853]], [[0.19622]], [[1.06304]], [[1.29632]], [[1.44013]], [[0.97978]], [[0.73902]], [[1.28124]], [[0.94761]], [[0.66497]], [[1.07917]], [[0.45063]], [[1.34781]], [[0.51356]], [[0.98582]], [[0.97512]], [[1.50830]], [[0.78581]], [[0.27234]], [[0.98511]]]])

test_add.test_module[Add1-[((1, 512, 1, 1), torch.float32)]]: forge/test/models_ops/test_add.py#L18561

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[0.85753]], [[1.12949]], [[0.44975]], [[0.49330]], [[0.66870]], [[0.99535]], [[0.85137]], [[1.25772]], [[0.81690]], [[0.99358]], [[0.71017]], [[0.76299]], [[0.38360]], [[0.53013]], [[0.65516]], [[0.87979]], [[1.05894]], [[1.16128]], [[0.52230]], [[0.64354]], [[1.04288]], [[1.27647]], [[0.75837]], [[1.23543]], [[0.78068]], [[0.91418]], [[1.31401]], [[0.39744]], [[0.54650]], [[0.73469]], [[0.66637]], [[1.29327]], [[0.53718]], [[0.63111]], [[0.51195]], [[0.39299]], [[0.56940]], [[1.29107]], [[1.08438]], [[1.10361]], [[0.88757]], [[0.60493]], [[0.94587]], [[0.39443]], [[0.49999]], [[0.60351]], [[1.17674]], [[1.15443]], [[0.63953]], [[0.84323]], [[1.18105]], [[1.35834]], [[1.05971]], [[0.92882]], [[1.19652]], [[0.56687]], [[0.95444]], [[0.47362]], [[0.51473]], [[0.60298]], [[1.08751]], [[1.06235]], [[0.56510]], [[1.01233]], [[1.13576]], [[0.79816]], [[0.88036]], [[0.97713]], [[1.17146]], [[1.34137]], [[0.47596]], [[0.67804]], [[1.05778]], [[1.27555]], [[1.29638]], [[1.30245]], [[0.96078]], [[0.42648]], [[0.90727]], [[0.54847]], [[0.39530]], [[1.30552]], [[1.24145]], [[0.36251]], [[0.95486]], [[0.77704]], [[0.77899]], [[0.63239]], [[1.05355]], [[0.56512]], [[1.04457]], [[1.11413]], [[1.21921]], [[1.04823]], [[0.36641]], [[0.53692]], [[1.11093]], [[0.96592]], [[0.47123]], [[0.57336]], [[1.33165]], [[1.19818]], [[0.64326]], [[0.73543]], [[0.38497]], [[0.85229]], [[0.48474]], [[0.47559]], [[0.83372]], [[0.93635]], [[0.65651]], [[1.15796]], [[0.55700]], [[1.31496]], [[1.20392]], [[0.43963]], [[0.73683]], [[0.88383]], [[0.93422]], [[0.97986]], [[1.05749]], [[0.89122]], [[0.61731]], [[1.09787]], [[0.38165]], [[0.56492]], [[0.73611]], [[0.61772]], [[0.68636]], [[0.45146]], [[0.75492]], [[0.96815]], [[0.53554]], [[0.83561]], [[1.21920]], [[0.80987]], [[0.87517]], [[0.81814]], [[0.96246]], [[1.17919]], [[1.33490]], [[1.17880]], [[1.33598]], [[0.82511]], [[0.41211]], [[0.62423]], [[1.20173]], [[0.85803]], [[0.61275]], [[0.47812]], [[0.39335]], [[0.43927]], [[0.75985]], [[1.13548]], [[1.13159]], [[0.37906]], [[1.17316]], [[0.47002]], [[0.75557]], [[0.65854]], [[0.76497]], [[0.76310]], [[0.41260]], [[0.42955]], [[0.78303]], [[0.86774]], [[0.63414]], [[1.04962]], [[0.41124]], [[0.82753]], [[1.30098]], [[0.65733]], [[1.31277]], [[1.04235]], [[0.41004]], [[1.17762]], [[0.80358]], [[0.63807]], [[1.26110]], [[0.45723]], [[0.91493]], [[0.75659]], [[1.21833]], [[1.00084]], [[1.10153]], [[1.03785]], [[0.74104]], [[0.75612]], [[0.44923]], [[1.13219]], [[1.25826]], [[1.20339]], [[0.50858]], [[0.88357]], [[0.50881]], [[0.58603]], [[0.56992]], [[1.03215]], [[0.56332]], [[0.85036]], [[0.88231]], [[1.18358]], [[0.48331]], [[0.51802]], [[0.57094]], [[1.21124]], [[0.68154]], [[1.28302]], [[1.04208]], [[0.92459]], [[0.85755]], [[0.76243]], [[0.92401]], [[0.74710]], [[0.85776]], [[0.92507]], [[0.47017]], [[0.59921]], [[1.26502]], [[0.45550]], [[0.82537]], [[1.35589]], [[1.04189]], [[0.87543]], [[0.42797]], [[1.10896]], [[0.50513]], [[0.71934]], [[0.69351]], [[0.78723]], [[0.86674]], [[1.27368]], [[0.92369]], [[1.30912]], [[1.16713]], [[0.54517]], [[1.08552]], [[0.50782]], [[0.64936]], [[1.00833]], [[1.02637]], [[1.23639]], [[0.70031]], [[0.86207]], [[1.11868]], [[0.37773]], [[1.22276]], [[0.44781]], [[0.86816]], [[0.77626]], [[0.59794]], [[0.92736]], [[1.27473]], [[0.71511]], [[0.56443]], [[0.67635]], [[0.36570]], [[1.08697]], [[0.62114]], [[0.52760]], [[0.57322]], [[1.14875]], [[1.12606]], [[1.24503]], [[1.04263]], [[0.69429]], [[0.72155]], [[1.00899]], [[1.27229]], [[0.99722]], [[0.62470]], [[0.62623]], [[0.38854]], [[0.96930]], [[0.58068]], [[0.41548]], [[1.29971]], [[0.53657]], [[0.80439]], [[1.00452]], [[0.87720]], [[0.52483]], [[0.45711]], [[1.25981]], [[0.94269]], [[1.27608]], [[0.69365]], [[1.00855]], [[0.74693]], [[0.83904]], [[0.55675]], [[1.03037]], [[1.01936]], [[0.85096]], [[0.74882]], [[0.55306]], [[1.20705]], [[0.48906]], [[1.06611]], [[0.69315]], [[0.62004]], [[0.95110]], [[0.60154]], [[0.97648]], [[0.95946]], [[0.49003]], [[0.94452]], [[1.07424]], [[1.05919]], [[0.79833]], [[0.45138]], [[0.78420]], [[1.03493]], [[0.67883]], [[1.05106]], [[1.19425]], [[0.60020]], [[0.86620]], [[1.06802]], [[0.90046]], [[0.90304]], [[0.92371]], [[0.46820]], [[0.90057]], [[1.20751]], [[1.31183]], [[1.15515]], [[0.92827]], [[1.09478]], [[0.61803]], [[0.44693]], [[0.43131]], [[1.36008]], [[1.17867]], [[0.51566]], [[1.05689]], [[1.23886]], [[1.36108]], [[1.29847]], [[1.24865]], [[0.74664]], [[0.68580]], [[1.27180]], [[1.14146]], [[0.56038]], [[1.31078]], [[1.10285]], [[1.13384]], [[0.54788]], [[1.00472]], [[0.68599]], [[1.25195]], [[0.77130]], [[1.05593]], [[0.95008]], [[1.07401]], [[0.69136]], [[1.10506]], [[0.51204]], [[0.97420]], [[0.52298]], [[0.36800]], [[0.45974]], [[1.25601]], [[1.13179]], [[1.33038]], [[1.26183]], [[0.41475]], [[0.52006]], [[0.78048]], [[0.53656]], [[1.20847]], [[0.48327]], [[0.61732]], [[0.37823]], [[0.57740]], [[1.27251]], [[1.27065]], [[1.21919]], [[1.24734]], [[1.30587]], [[0.73324]], [[1.08127]], [[1.30674]], [[1.02668]], [[1.36111]], [[1.12060]], [[1.17210]], [[0.68628]], [[1.10119]], [[0.91873]], [[0.74186]], [[0.57942]], [[0.58071]], [[0.47653]], [[1.19694]], [[1.21674]], [[0.80437]], [[0.57193]], [[1.24773]], [[1.18102]], [[0.89844]], [[0.62521]], [[1.32080]], [[1.06574]], [[0.48170]], [[1.33981]], [[1.24096]], [[0.67903]], [[1.14235]], [[0.57718]], [[0.78292]], [[1.28582]], [[0.88193]], [[0.50766]], [[0.69416]], [[0.72555]], [[0.76481]], [[0.90912]], [[1.32369]], [[0.88804]], [[0.55255]], [[0.88690]], [[1.10102]], [[1.10929]], [[0.40430]], [[0.77180]], [[0.48970]], [[0.64793]], [[1.04142]], [[0.50621]], [[1.04714]], [[1.28566]], [[0.89407]], [[0.52803]], [[0.68213]], [[0.97045]], [[0.48011]], [[1.10968]], [[0.40734]], [[0.38063]], [[0.37544]], [[0.75984]], [[1.19749]], [[0.38803]], [[1.27687]], [[0.66126]], [[1.00771]], [[0.88407]], [[0.41041]], [[1.27594]], [[1.13049]], [[1.35827]], [[1.11388]], [[0.53124]], [[1.27856]], [[0.88814]], [[1.09838]], [[0.46036]], [[0.71746]], [[0.37033]], [[0.66653]], [[0.96914]], [[0.46869]], [[1.02065]], [[1.12968]], [[0.93093]], [[0.52673]], [[0.47361]], [[0.70701]], [[1.08075]], [[1.35447]], [[1.14879]], [[0.80497]], [[1.03658]], [[0.37074]], [[0.43422]], [[1.09458]], [[0.57807]], [[1.10182]], [[0.50831]], [[0.61362]], [[0.44943]], [[1.12219]], [[0.81033]], [[1.24607]], [[1.17071]], [[1.13794]], [[0.87735]], [[0.70668]], [[0.75256]], [[0.92772]], [[1.10913]], [[0.51098]], [[1.28091]], [[0.80691]], [[0.44230]], [[0.59074]], [[1.30368]], [[1.31854]], [[0.39813]], [[1.21392]], [[1.11185]], [[1.15723]], [[1.28454]], [[0.59180]], [[1.01916]], [[1.06589]], [[0.71353]], [[1.02860]], [[0.71742]], [[1.17040]]]]), compiled_model=tensor([[[[0.85753]], [[1.20019]], [[0.15767]], [[0.80206]], [[0.35869]], [[1.61934]], [[0.87502]], [[0.93073]], [[0.56514]], [[0.99823]], [[0.85195]], [[0.61959]], [[0.54272]], [[0.16908]], [[0.67248]], [[1.07732]], [[1.16991]], [[1.05761]], [[0.69985]], [[0.38011]], [[1.07191]], [[1.35076]], [[0.85846]], [[0.94958]], [[0.92009]], [[2.22027]], [[1.56779]], [[0.81441]], [[0.69321]], [[1.78261]], [[1.55078]], [[2.66615]], [[1.59144]], [[0.99211]], [[0.99157]], [[0.87230]], [[1.02614]], [[1.27900]], [[1.18172]], [[1.65929]], [[1.28747]], [[2.01945]], [[1.34508]], [[1.11104]], [[1.75893]], [[1.48033]], [[1.93834]], [[1.94721]], [[1.12822]], [[1.86347]], [[2.22037]], [[1.34049]], [[1.54441]], [[1.93397]], [[1.73320]], [[2.02469]], [[0.88015]], [[0.60803]], [[1.12198]], [[0.72237]], [[1.26368]], [[1.66123]], [[1.92205]], [[2.22873]], [[2.28938]], [[1.48567]], [[1.25387]], [[0.91349]], [[1.34451]], [[1.84436]], [[1.11732]], [[1.27950]], [[1.92159]], [[1.45946]], [[2.57131]], [[2.61425]], [[1.89202]], [[0.67960]], [[2.02953]], [[0.97756]], [[1.03404]], [[1.25761]], [[1.92445]], [[0.54844]], [[1.97334]], [[1.70158]], [[1.60901]], [[1.44852]], [[1.87122]], [[0.66822]], [[1.22205]], [[1.38437]], [[1.79681]], [[2.35970]], [[0.70084]], [[0.51180]], [[1.52087]], [[2.02556]], [[1.58030]], [[1.33076]], [[2.22124]], [[1.51012]], [[1.55268]], [[0.89119]], [[0.49988]], [[1.72747]], [[1.27287]], [[0.67909]], [[1.08148]], [[1.52388]], [[0.75491]], [[1.11447]], [[1.04759]], [[1.51136]], [[1.64537]], [[1.52288]], [[2.19636]], [[1.63697]], [[1.16668]], [[1.65220]], [[1.58770]], [[1.88787]], [[0.75931]], [[1.38273]], [[1.60021]], [[1.66916]], [[1.45425]], [[1.16070]], [[1.28798]], [[0.80087]], [[0.68619]], [[1.23119]], [[1.10112]], [[1.84758]], [[1.58276]], [[2.16167]], [[1.39044]], [[1.87109]], [[2.25538]], [[1.37747]], [[2.34216]], [[1.67556]], [[1.65341]], [[1.10443]], [[0.85926]], [[1.20100]], [[1.26568]], [[2.04976]], [[0.83842]], [[0.64776]], [[0.91734]], [[0.48139]], [[0.87598]], [[1.00480]], [[1.38605]], [[1.23163]], [[1.16851]], [[0.71853]], [[1.82354]], [[1.18194]], [[1.88407]], [[1.51081]], [[1.32986]], [[1.75147]], [[1.53026]], [[1.40948]], [[1.52513]], [[1.78478]], [[0.87199]], [[1.70570]], [[2.70627]], [[1.09092]], [[1.29911]], [[1.36607]], [[0.30889]], [[1.70820]], [[1.42749]], [[1.18128]], [[2.28724]], [[0.58391]], [[1.80896]], [[0.76877]], [[1.77961]], [[1.25771]], [[1.50095]], [[1.51661]], [[1.03709]], [[0.94537]], [[1.00138]], [[1.92753]], [[2.08361]], [[2.20872]], [[1.03898]], [[1.83403]], [[1.75828]], [[1.38821]], [[1.59026]], [[1.49606]], [[0.89064]], [[2.17735]], [[1.15228]], [[1.27916]], [[1.41875]], [[1.22019]], [[1.62712]], [[1.99420]], [[2.00844]], [[2.35529]], [[1.62294]], [[1.92061]], [[1.87516]], [[1.17887]], [[1.06720]], [[1.67127]], [[1.76023]], [[2.38838]], [[0.69039]], [[1.11723]], [[1.51688]], [[0.72259]], [[1.00064]], [[2.21742]], [[1.45232]], [[1.99947]], [[0.82790]], [[2.53348]], [[0.87011]], [[0.58620]], [[1.31819]], [[1.66423]], [[1.50487]], [[2.82566]], [[1.97006]], [[1.79389]], [[1.92579]], [[1.08340]], [[1.58094]], [[0.87691]], [[1.34927]], [[1.86149]], [[1.20839]], [[1.69223]], [[1.78501]], [[1.56805]], [[1.22909]], [[1.08907]], [[1.58894]], [[0.97175]], [[1.12304]], [[1.67873]], [[1.06008]], [[1.73161]], [[1.82251]], [[1.13467]], [[1.41886]], [[2.03198]], [[0.59775]], [[2.14373]], [[1.61589]], [[0.77619]], [[1.66048]], [[0.98369]], [[1.82783]], [[2.18008]], [[2.12149]], [[1.31281]], [[1.09930]], [[1.92895]], [[1.85862]], [[1.30092]], [[1.34260]], [[0.71559]], [[1.37508]], [[1.12158]], [[1.20523]], [[1.02933]], [[2.44674]], [[0.96111]], [[0.71546]], [[1.62835]], [[2.02870]], [[1.62729]], [[1.40630]], [[1.68807]], [[1.41692]], [[1.69018]], [[0.64772]], [[1.72547]], [[1.07242]], [[1.89987]], [[0.27316]], [[0.76862]], [[1.48103]], [[1.65111]], [[0.73217]], [[1.32516]], [[2.53094]], [[1.16894]], [[1.94644]], [[1.56257]], [[1.07015]], [[2.14131]], [[0.65273]], [[1.95100]], [[1.33278]], [[1.74282]], [[2.01570]], [[1.75999]], [[2.24456]], [[1.02457]], [[0.44944]], [[1.89982]], [[1.78435]], [[2.10988]], [[1.28895]], [[1.91888]], [[0.92973]], [[1.12187]], [[1.72031]], [[1.62841]], [[2.20359]], [[1.07267]], [[1.12008]], [[1.25777]], [[1.79822]], [[2.17577]], [[0.99855]], [[1.76645]], [[1.55955]], [[0.96053]], [[0.74735]], [[0.72111]], [[1.89823]], [[1.06953]], [[1.33340]], [[1.93456]], [[1.38796]], [[1.42372]], [[2.01068]], [[1.84073]], [[1.44954]], [[1.78672]], [[2.43740]], [[2.51475]], [[1.13055]], [[1.90391]], [[1.75595]], [[2.38851]], [[1.10697]], [[1.84604]], [[1.86090]], [[1.37144]], [[0.82490]], [[1.36797]], [[0.87491]], [[1.65616]], [[1.18752]], [[1.47822]], [[0.96858]], [[1.63473]], [[1.11015]], [[0.62227]], [[0.84356]], [[1.94118]], [[1.65976]], [[1.75917]], [[2.29455]], [[0.91141]], [[1.54362]], [[1.75214]], [[1.95749]], [[2.23188]], [[1.36574]], [[1.25725]], [[0.76083]], [[1.25774]], [[1.69873]], [[1.06761]], [[1.94938]], [[2.65051]], [[1.75792]], [[0.68605]], [[1.45268]], [[1.97426]], [[1.58941]], [[2.09723]], [[2.39026]], [[2.13770]], [[1.00286]], [[1.49272]], [[1.85371]], [[1.74258]], [[0.69357]], [[0.76517]], [[0.52556]], [[1.24052]], [[1.38925]], [[1.05563]], [[0.58030]], [[1.59930]], [[1.35304]], [[1.49173]], [[1.12897]], [[1.93949]], [[0.88761]], [[0.28508]], [[1.91588]], [[1.46800]], [[0.88677]], [[1.43456]], [[0.94340]], [[1.15210]], [[1.54155]], [[0.79193]], [[0.65577]], [[0.73664]], [[0.74536]], [[1.11222]], [[1.17475]], [[1.64886]], [[1.08808]], [[0.78656]], [[0.67187]], [[1.48051]], [[1.67750]], [[0.84196]], [[0.61409]], [[0.40490]], [[0.44084]], [[1.27591]], [[0.76700]], [[1.30740]], [[1.55269]], [[0.67803]], [[0.49082]], [[0.77039]], [[0.76076]], [[0.57944]], [[0.76032]], [[0.94961]], [[0.36644]], [[0.94160]], [[0.62694]], [[1.49621]], [[0.14310]], [[1.44619]], [[0.59393]], [[1.40530]], [[0.59521]], [[0.17888]], [[0.94823]], [[1.27861]], [[1.58775]], [[1.06271]], [[0.18652]], [[1.03890]], [[1.33839]], [[1.61491]], [[0.99568]], [[0.44000]], [[0.55155]], [[0.55991]], [[1.34596]], [[0.94122]], [[1.16958]], [[1.73803]], [[0.74888]], [[0.97555]], [[1.02766]], [[0.93698]], [[1.29641]], [[1.02958]], [[0.99754]], [[1.00240]], [[0.78505]], [[0.43402]], [[0.30344]], [[1.22726]], [[0.29106]], [[0.89210]], [[0.38068]], [[1.04197]], [[0.49732]], [[0.85952]], [[0.97962]], [[1.82830]], [[1.62690]], [[1.53486]], [[0.81024]], [[0.81990]], [[0.81135]], [[1.08988]], [[1.72748]], [[1.11173]], [[1.59714]], [[0.88068]], [[0.22779]], [[0.34514]], [[1.79032]], [[1.50359]], [[0.47593]], [[1.65957]], [[1.06346]], [[1.56560]], [[1.17290]], [[0.89436]], [[1.12211]], [[1.43070]], [[1.01136]], [[0.96107]], [[1.16620]], [[1.17577]]]])

test_advindex.test_module[Advindex3-[((2401,), torch.int64)]]: forge/test/models_ops/test_advindex.py#L233

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[0.63011, 0.30126, 0.88538, 0.04888, 0.36540, 0.57885], [0.27942, 0.43306, 0.17316, 0.26804, 0.98792, 0.87603], [0.59931, 0.64610, 0.26906, 0.80871, 0.02725, 0.31218], ..., [0.26057, 0.53891, 0.30452, 0.11554, 0.76484, 0.89953], [0.63145, 0.13738, 0.53211, 0.49383, 0.67033, 0.69835], [0.74920, 0.54203, 0.08310, 0.00378, 0.65803, 0.73633]], grad_fn=<IndexBackward0>), compiled_model=tensor([[0.62891, 0.30078, 0.88672, 0.04883, 0.36523, 0.57812], [0.50391, 0.19922, 0.78906, 0.39648, 0.53516, 0.23926], [0.27930, 0.43359, 0.17285, 0.26758, 0.98828, 0.87500], ..., [0.16895, 0.82422, 0.23535, 0.59375, 0.00806, 0.73828], [0.50391, 0.19922, 0.78906, 0.39648, 0.53516, 0.23926], [0.22266, 0.73047, 0.83594, 0.67188, 0.01611, 0.14160]])

test_avgpool2d.test_module[Avgpool2D9-[((1, 1536, 8, 8), torch.float32)]]: forge/test/models_ops/test_avgpool2d.py#L1472

RuntimeError: Tensor 1 - stride mismatch: expected [64, 1], got [0, 0]

test_broadcast.test_module[Broadcast0-[((1, 1, 1, 128), torch.bool)]]: forge/test/models_ops/test_broadcast.py#L170

RuntimeError: Generated MLIR module failed verification.

test_broadcast.test_module[Broadcast2-[((1, 12, 1, 128), torch.bool)]]: forge/test/models_ops/test_broadcast.py#L170

RuntimeError: Generated MLIR module failed verification.

test_cast.test_module[Cast0-[((2, 1, 7, 7), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast0-[((1, 1, 256, 256), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast1-[((1, 128), torch.int32)]]: forge/test/models_ops/test_cast.py#L502

TypeError: Dtype mismatch: framework_model.dtype=torch.int8, compiled_model.dtype=torch.uint8

test_cast.test_module[Cast0-[((1, 12, 128, 128), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_cast.test_module[Cast1-[((1, 384), torch.int32)]]: forge/test/models_ops/test_cast.py#L502

TypeError: Dtype mismatch: framework_model.dtype=torch.int8, compiled_model.dtype=torch.uint8

test_cast.test_module[Cast1-[((1, 1, 256, 256), torch.float32)]]: forge/test/models_ops/test_cast.py#L502

TypeError: Dtype mismatch: framework_model.dtype=torch.int8, compiled_model.dtype=torch.uint8

test_cast.test_module[Cast2-[((1, 32), torch.bool)]]: forge/test/models_ops/test_cast.py#L502

RuntimeError: Tensor 0 - data type mismatch: expected UInt8, got Float32

test_concatenate.test_module[Concatenate52-[((256, 1536, 1, 1), torch.float32), ((384, 1536, 1, 1), torch.float32), ((384, 1536, 1, 1), torch.float32)]]: forge/test/models_ops/test_concatenate.py#L12958

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 6442450944 B DRAM buffer across 12 banks, where each bank needs to store 536870912 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x1ebfb43) [0x7f6fbcfc9b43] --- tt::tt_metal::allocate_tensor_on_devices(tt::tt_metal::TensorSpec const&, std::vector<tt::tt_metal::v0::IDevice*, std::allocator<tt::tt_metal::v0::IDevice*> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xdbf0e) [0x7f6fcd5cdf0e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xdba65) [0x7f6fcd5cda65] --- tt::runtime::ttnn::operations::creation::run(tt::target::ttnn::EmptyOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d]

test_conv2d.test_module[Conv2D1-[((1, 1280, 3000, 1), torch.float32), ((1280, 1280, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[633.69440], [945.17078], [946.99237], ..., [953.52045], [948.52710], [940.59521]], [[635.07336], [950.94189], [948.41229], ..., [941.41693], [953.16632], [947.66656]], [[652.58441], [962.00073], [961.69080], ..., [961.07111], [963.78943], [960.62109]], ..., [[647.24475], [947.06793], [955.33301], ..., [965.56537], [965.41382], [966.57104]], [[639.20392], [952.56335], [947.12415], ..., [951.14532], [948.82904], [956.22845]], [[642.42462], [964.29010], [956.39838], ..., [979.07642], [967.51056], [959.41614]]]]), compiled_model=tensor([[[[636.], [952.], [956.], ..., [944.], [952.], [936.]], [[632.], [976.], [944.], ..., [956.], [956.], [964.]], [[644.], [976.], [992.], ..., [972.], [984.], [984.]], ..., [[644.], [952.], [964.], ..., [972.], [956.], [984.]], [[636.], [976.], [956.], ..., [960.], [948.], [964.]], [[648.], [984.], [964.], ..., [980.], [972.], [964.]]]])

test_conv2d.test_module[Conv2D0-[((1, 80, 3000, 1), torch.float32), ((384, 80, 3, 1), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_FATAL @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/ttnn/cpp/ttnn/operations/conv/conv2d/device/conv2d_op_sharded_program_factory.cpp:1180: act_block_w_datums == round_up(conv_act_size_c * filter_w, TILE_WIDTH) info: Error backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xbd288) [0x7f6fcd5af288] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d41multi_core_optimized_conv_sharded_v2_implERN2tt8tt_metal2v07ProgramERKNS4_6TensorESA_RKNS4_5ShapeESt8optionalIS9_ERKSF_RKNS0_14sliding_window19SlidingWindowConfigEjjbbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigEbbRS8_St7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x90c4) [0x7f6fbb4f97a4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d40multi_core_optimized_conv_sharded_v2_newERKN2tt8tt_metal6TensorES7_RKSt8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigENS4_8DataTypeESt5arrayIjLm4EEbS8_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERS5_bbbb+0x492) [0x7f6fbb4ecfc2] --- ttnn::operations::conv::conv2d::OptimizedConvNew::create_program(std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&) const --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x449829) [0x7f6fbb553829] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c0fda) [0x7f6fbd6cafda] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c13e4) [0x7f6fbd6cb3e4] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x2ab) [0x7f6fbd6ca72b] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f6fbd6ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7f6fbb552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7f6fbb4ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7f6fbb4d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30]

test_conv2d.test_module[Conv2D17-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 12500992 B L1 buffer across 7 banks, where each bank needs to store 1785856 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f6fbc109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f6fbc109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464]

test_conv2d.test_module[Conv2D518-[((1, 64, 73, 73), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[152.22598, 145.77719, 136.87177, ..., 140.42284, 139.65382, 139.14134], [151.68343, 143.08662, 140.67943, ..., 143.02628, 144.81311, 139.77446], [145.77505, 146.30482, 143.89157, ..., 141.57092, 148.67188, 145.09837], ..., [140.66216, 145.60056, 144.29475, ..., 143.66399, 140.73079, 142.51483], [145.83846, 149.24707, 145.43570, ..., 141.18921, 141.59860, 142.95306], [145.28297, 151.61617, 146.11958, ..., 139.27266, 140.42781, 139.23724]], [[146.71242, 151.62230, 136.79425, ..., 137.81465, 138.85857, 139.59618], [149.26445, 144.13255, 143.41682, ..., 142.76549, 142.15085, 141.04979], [145.13777, 145.30286, 143.35765, ..., 139.80869, 145.87190, 142.07967], ..., [145.36610, 143.07983, 137.89236, ..., 139.27014, 138.64842, 139.64122], [145.19089, 145.64764, 142.26469, ..., 138.09291, 138.25905, 144.40562], [143.43483, 143.03133, 142.14932, ..., 140.35820, 146.53207, 137.05432]], [[150.08231, 143.93704, 138.48259, ..., 136.43771, 139.20850, 137.65762], [150.12985, 145.98167, 142.40926, ..., 142.60051, 146.34427, 137.25406], [140.42233, 143.31340, 141.57941, ..., 137.34343, 145.71970, 142.08282], ..., [144.66275, 146.50664, 140.15816, ..., 141.29842, 141.30623, 143.05179], [141.25235, 147.69696, 142.54037, ..., 140.49292, 140.33524, 138.50575], [138.23695, 145.40689, 146.24147, ..., 141.82268, 137.85051, 140.34471]], ..., [[147.73674, 145.04662, 140.21635, ..., 140.23395, 139.23204, 136.47896], [145.37366, 145.57039, 143.24754, ..., 139.99467, 142.26474, 138.74841], [144.37726, 145.14883, 145.62071, ..., 137.14101, 145.80376, 145.49905], ..., [139.15382, 142.38667, 141.27391, ..., 141.31412, 137.80165, 139.29973], [144.34308, 141.67398, 144.33537, ..., 138.97748, 137.92513, 137.99631], [142.96829, 141.95638, 144.33736, ..., 140.62619, 138.65828, 135.29399]], [[154.37540, 149.13547, 141.25761, ..., 142.95166, 141.25923, 136.30963], [152.19673, 153.29286, 146.05107, ..., 145.86459, 150.65500, 143.83481], [152.52058, 144.74634, 146.85190, ..., 143.77768, 151.79756, 145.57185], ..., [145.70895, 148.10301, 141.79237, ..., 146.38281, 145.00935, 144.01068], [142.09671, 152.57594, 148.85966, ..., 143.55881, 140.84187, 144.66855], [143.57538, 148.14258, 145.06818, ..., 146.17476, 140.67175, 141.00688]], [[157.26375, 155.89873, 145.42444, ..., 143.94855, 150.49825, 149.97076], [156.86661, 156.28867, 147.12398, ..., 148.15878, 149.96893, 149.14177], [154.31503, 151.20000, 151.95830, ..., 150.68819, 150.98943, 152.13821], ..., [151.20518, 148.82639, 149.04521, ..., 153.96242, 146.63577, 146.34781], [152.03629, 157.25171, 149.75937, ..., 148.91685, 146.19638, 145.55783], [150.82550, 153.93781, 151.17772, ..., 143.61913, 146.92773, 143.71964]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[152., 145., 135., ..., 138., 138., 138.], [149., 141., 138., ..., 141., 143., 138.], [144., 144., 142., ..., 138., 147., 142.], ..., [138., 144., 143., ..., 143., 140., 141.], [143., 148., 144., ..., 140., 138., 139.], [144., 148., 143., ..., 138., 138., 137.]], [[146., 149., 135., ..., 137., 137., 138.], [148., 143., 142., ..., 140., 141., 139.], [143., 143., 141., ..., 139., 142., 139.], ..., [144., 141., 136., ..., 137., 136., 140.], [144., 143., 141., ..., 135., 135., 144.], [141., 141., 141., ..., 138., 144., 136.]], [[148., 142., 136., ..., 134., 136., 136.], [147., 145., 141., ..., 139., 145., 135.], [139., 142., 142., ..., 136., 143., 139.], ..., [142., 144., 136., ..., 139., 140., 141.], [139., 146., 140., ..., 139., 137., 136.], [137., 143., 145., ..., 141., 135., 139.]], ..., [[145., 141., 138., ..., 137., 136., 133.], [141., 143., 141., ..., 138., 140., 138.], [142., 143., 142., ..., 135., 143., 144.], ..., [137., 140., 138., ..., 139., 137., 136.], [141., 138., 142., ..., 137., 137., 136.], [141., 139., 141., ..., 137., 136., 132.]], [[154., 146., 140., ..., 142., 140., 134.], [152., 152., 145., ..., 144., 149., 142.], [151., 144., 145., ..., 141., 148., 144.], ..., [144., 146., 140., ..., 143., 143., 142.], [139., 150., 147., ..., 141., 138., 143.], [142., 146., 142., ..., 144., 139., 139.]], [[156., 154., 145., ..., 140., 150., 149.], [154., 155., 146., ..., 146., 148., 146.], [152., 150., 150., ..., 149., 149., 150.], ..., [150., 145., 148., ..., 152., 145., 143.], [149., 154., 147., ..., 147., 142., 143.], [148., 151., 147., ..., 141., 145., 141.]]]])

test_conv2d.test_module[Conv2D526-[((1, 224, 35, 35), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[500.36554, 489.48056, 488.66583, ..., 498.19626, 485.72424, 486.22116], [503.10248, 495.09406, 496.34912, ..., 490.61444, 489.81943, 489.86374], [498.71185, 494.72519, 487.24246, ..., 487.91693, 495.74234, 503.22653], ..., [498.49924, 518.24774, 501.57489, ..., 495.70898, 495.14191, 481.31421], [495.43970, 508.53278, 506.09814, ..., 495.94330, 509.07922, 495.81696], [490.71671, 501.05109, 505.43127, ..., 497.91800, 504.00275, 511.43808]], [[523.42047, 508.63754, 507.74429, ..., 516.41907, 496.17279, 508.56058], [518.57642, 515.18164, 513.18146, ..., 503.78110, 504.70053, 501.20224], [516.28607, 514.22754, 512.87970, ..., 509.42688, 516.19397, 511.41226], ..., [519.95728, 523.61420, 522.65869, ..., 512.83771, 522.34180, 503.43286], [507.07605, 521.71887, 518.81549, ..., 512.78595, 525.62079, 506.89072], [513.57196, 518.03143, 518.32800, ..., 520.22571, 526.47333, 524.22772]], [[517.43781, 492.03992, 502.15067, ..., 497.50546, 492.68451, 497.66632], [510.47214, 498.31097, 509.26926, ..., 498.59302, 493.69919, 495.67303], [510.47672, 503.62366, 497.22598, ..., 500.26001, 504.06039, 509.71936], ..., [507.82581, 517.91827, 515.74310, ..., 502.16611, 510.33530, 494.94861], [507.40668, 513.54370, 500.53250, ..., 501.59366, 517.82855, 502.32056], [507.53967, 518.03485, 508.50662, ..., 515.47253, 511.85449, 514.98389]], ..., [[513.73401, 496.53857, 499.49030, ..., 498.35474, 491.66019, 497.31866], [514.02393, 502.14752, 507.00244, ..., 496.72568, 498.28152, 507.74890], [507.58640, 512.12756, 508.18781, ..., 497.11166, 509.34613, 511.13892], ..., [513.84064, 523.53748, 516.00226, ..., 502.96719, 510.30936, 494.74835], [498.40732, 507.97662, 509.74026, ..., 500.91898, 523.50861, 503.13373], [510.09976, 515.44604, 505.54169, ..., 514.41858, 521.77728, 521.45667]], [[515.00751, 509.78442, 509.47891, ..., 513.87347, 494.96732, 509.13071], [531.36389, 509.85422, 510.92612, ..., 507.68307, 516.30200, 502.51831], [521.55823, 516.93402, 512.74072, ..., 517.31049, 520.74335, 511.62952], ..., [520.19653, 532.08099, 530.59521, ..., 514.39105, 517.55212, 500.31659], [509.42899, 520.70447, 517.20959, ..., 513.56909, 534.29730, 505.84747], [508.51224, 521.92700, 518.38928, ..., 518.34491, 525.27649, 526.21692]], [[508.91409, 502.67065, 506.02429, ..., 500.09949, 491.69757, 498.57138], [509.70746, 504.42422, 506.12076, ..., 498.94162, 500.49487, 498.35809], [510.69574, 505.69550, 501.47955, ..., 496.72467, 507.06134, 508.75903], ..., [509.02377, 523.04785, 517.46924, ..., 505.12375, 516.28961, 499.34229], [501.19504, 515.98914, 505.25638, ..., 502.72897, 519.17059, 506.77188], [503.61282, 515.03320, 503.22467, ..., 514.83771, 521.21582, 521.64819]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[498., 494., 486., ..., 504., 480., 486.], [502., 498., 492., ..., 490., 484., 490.], [496., 496., 488., ..., 480., 498., 502.], ..., [502., 516., 498., ..., 490., 498., 480.], [492., 502., 502., ..., 496., 510., 504.], [492., 506., 502., ..., 498., 500., 510.]], [[524., 508., 504., ..., 512., 494., 496.], [520., 512., 512., ..., 492., 508., 496.], [516., 508., 508., ..., 516., 516., 512.], ..., [512., 524., 512., ..., 498., 524., 498.], [504., 524., 524., ..., 508., 520., 506.], [512., 512., 520., ..., 516., 528., 524.]], [[512., 486., 506., ..., 492., 492., 498.], [506., 498., 508., ..., 498., 492., 490.], [508., 494., 488., ..., 504., 512., 508.], ..., [500., 508., 516., ..., 502., 500., 494.], [512., 512., 498., ..., 500., 512., 508.], [510., 524., 498., ..., 502., 508., 520.]], ..., [[516., 492., 498., ..., 496., 496., 506.], [508., 502., 502., ..., 494., 494., 502.], [506., 504., 506., ..., 498., 502., 512.], ..., [512., 528., 512., ..., 494., 506., 486.], [492., 510., 508., ..., 498., 524., 502.], [504., 510., 516., ..., 512., 524., 524.]], [[516., 508., 520., ..., 508., 492., 512.], [520., 508., 508., ..., 504., 516., 508.], [520., 512., 510., ..., 520., 532., 512.], ..., [520., 536., 532., ..., 512., 512., 496.], [502., 520., 532., ..., 508., 528., 504.], [504., 516., 520., ..., 524., 520., 528.]], [[516., 500., 502., ..., 510., 488., 500.], [508., 510., 504., ..., 494., 500., 502.], [516., 504., 502., ..., 502., 506., 512.], ..., [506., 528., 516., ..., 504., 512., 498.], [508., 520., 502., ..., 500., 510., 508.], [498., 512., 506., ..., 512., 512., 524.]]]])

test_conv2d.test_module[Conv2D544-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 6422528 B L1 buffer across 2 banks, where each bank needs to store 3211264 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations13data_movement35InterleavedToShardedDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS8_EE5valueEE4typeERKS8_RKSt6vectorINS0_6TensorESaISE_EERKSD_ISt8optionalISE_ESaISK_EE+0x1cf) [0x7f6fcd64629f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x409139) [0x7f6fbb513139] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x40052f) [0x7f6fbb50a52f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff92f) [0x7f6fbb50992f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff708) [0x7f6fbb509708] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff66c) [0x7f6fbb50966c] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3ff44d) [0x7f6fbb50944d] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fee51) [0x7f6fbb508e51] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x3fe8d9) [0x7f6fbb5088d9] --- std::tuple<tt::tt_metal::Tensor, ttnn::operations::sliding_window::ParallelConfig, ttnn::operations::sliding_window::ParallelConfig> ttnn::operations::conv::shard_or_reshard_tensor_if_required<tt::tt_metal::v0::IDevice>(tt::tt_metal::v0::IDevice*, tt::tt_metal::Tensor const&, ttnn::operations::conv::conv2d::Conv2dConfig const&, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int, bool, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0x697) [0x7f6fbb4d9747] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702]

test_conv2d.test_module[Conv2D548-[((1, 3, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 12500992 B L1 buffer across 7 banks, where each bank needs to store 1785856 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f6fbc109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f6fbc109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464]

test_conv2d.test_module[Conv2D654-[((1, 512, 12, 34), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1145.44080, 1141.51807, 1150.66821, ..., 1162.54541, 1155.66370, 1153.13928], [1151.45679, 1155.86145, 1148.58704, ..., 1167.01416, 1155.52454, 1142.43030], [1154.23315, 1155.89087, 1160.23120, ..., 1167.54297, 1163.24670, 1156.78918], ..., [1136.43677, 1142.03772, 1158.03210, ..., 1135.32886, 1157.68030, 1160.54382], [1144.23999, 1132.88794, 1150.35437, ..., 1139.35583, 1145.42249, 1147.89893], [1138.88159, 1139.24304, 1153.87915, ..., 1144.43762, 1166.95544, 1168.00012]], [[1154.08777, 1150.70959, 1158.39307, ..., 1174.65186, 1169.08655, 1157.66138], [1154.11646, 1168.58887, 1164.71338, ..., 1167.68640, 1173.50061, 1161.88757], [1164.22168, 1168.41113, 1151.22986, ..., 1167.94336, 1177.56555, 1175.57117], ..., [1151.44531, 1140.74219, 1160.73926, ..., 1152.37012, 1160.24792, 1169.90759], [1155.49133, 1149.51123, 1152.83325, ..., 1158.83887, 1155.40234, 1169.81042], [1158.56458, 1163.04944, 1167.93103, ..., 1154.07629, 1158.56567, 1153.41101]], [[1138.68042, 1142.47986, 1144.13672, ..., 1156.29602, 1148.25708, 1153.38367], [1137.89221, 1146.79041, 1157.89014, ..., 1160.40759, 1155.02148, 1143.10840], [1151.19202, 1147.01929, 1148.93298, ..., 1169.62927, 1151.91235, 1158.23059], ..., [1139.51392, 1128.12744, 1144.43237, ..., 1125.74231, 1148.51624, 1157.09705], [1134.67871, 1133.29553, 1154.14026, ..., 1140.89478, 1148.38354, 1152.65332], [1148.75818, 1136.93665, 1159.03223, ..., 1129.31299, 1136.47913, 1154.04333]], ..., [[1157.81921, 1170.62622, 1164.24292, ..., 1161.72522, 1174.61707, 1160.24500], [1161.16357, 1179.23108, 1166.21423, ..., 1183.11963, 1176.97607, 1169.52307], [1167.78931, 1152.89246, 1161.95581, ..., 1183.46826, 1179.91028, 1183.12903], ..., [1152.85510, 1150.99121, 1167.94214, ..., 1142.38843, 1166.05432, 1175.34204], [1161.26575, 1145.66187, 1159.27344, ..., 1161.01953, 1173.05188, 1175.29761], [1166.51990, 1150.49316, 1181.09412, ..., 1158.08777, 1168.40137, 1177.43835]], [[1141.50073, 1135.78760, 1145.58557, ..., 1166.70532, 1171.14429, 1146.84180], [1150.86572, 1166.94458, 1158.58130, ..., 1161.42456, 1166.40771, 1159.43555], [1155.48792, 1160.47668, 1154.16711, ..., 1171.57666, 1162.72083, 1166.17480], ..., [1145.74402, 1141.27441, 1160.02917, ..., 1125.34082, 1154.22375, 1169.15759], [1139.08618, 1147.70959, 1152.23547, ..., 1145.93018, 1153.58057, 1158.42761], [1164.08887, 1147.58716, 1156.25061, ..., 1141.61938, 1150.32202, 1166.47559]], [[1143.11255, 1140.64258, 1139.76758, ..., 1153.54956, 1159.57983, 1143.85718], [1149.01331, 1156.96411, 1141.76562, ..., 1159.89587, 1148.08997, 1145.75513], [1149.86768, 1145.65649, 1143.72668, ..., 1165.18884, 1161.19800, 1163.15698], ..., [1133.80200, 1136.55261, 1148.56738, ..., 1122.43066, 1144.07947, 1165.23352], [1135.66675, 1124.48938, 1144.79761, ..., 1140.77148, 1142.32336, 1157.62683], [1145.59155, 1138.75537, 1154.74219, ..., 1140.99622, 1150.01001, 1155.55322]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1184., 1176., 1200., ..., 1200., 1232., 1232.], [1232., 1216., 1256., ..., 1264., 1216., 1216.], [1216., 1208., 1200., ..., 1240., 1232., 1248.], ..., [1192., 1168., 1208., ..., 1200., 1208., 1208.], [1184., 1208., 1248., ..., 1224., 1224., 1224.], [1192., 1192., 1232., ..., 1192., 1240., 1272.]], [[1160., 1152., 1216., ..., 1200., 1208., 1256.], [1200., 1184., 1224., ..., 1216., 1224., 1208.], [1208., 1200., 1176., ..., 1232., 1240., 1208.], ..., [1200., 1200., 1216., ..., 1216., 1200., 1208.], [1176., 1168., 1232., ..., 1224., 1216., 1208.], [1200., 1184., 1216., ..., 1152., 1192., 1232.]], [[1104., 1152., 1152., ..., 1176., 1176., 1168.], [1144., 1192., 1168., ..., 1168., 1176., 1144.], [1176., 1168., 1160., ..., 1176., 1152., 1192.], ..., [1136., 1144., 1184., ..., 1120., 1168., 1176.], [1184., 1176., 1176., ..., 1128., 1168., 1176.], [1160., 1120., 1184., ..., 1144., 1144., 1168.]], ..., [[1208., 1200., 1224., ..., 1192., 1272., 1192.], [1192., 1224., 1208., ..., 1240., 1192., 1216.], [1216., 1176., 1208., ..., 1232., 1240., 1208.], ..., [1168., 1144., 1208., ..., 1176., 1224., 1232.], [1168., 1208., 1232., ..., 1184., 1216., 1224.], [1208., 1192., 1216., ..., 1192., 1208., 1176.]], [[1152., 1168., 1192., ..., 1208., 1192., 1160.], [1176., 1200., 1160., ..., 1216., 1200., 1184.], [1168., 1200., 1168., ..., 1224., 1200., 1216.], ..., [1160., 1176., 1184., ..., 1160., 1192., 1240.], [1168., 1168., 1208., ..., 1168., 1192., 1184.], [1176., 1176., 1216., ..., 1144., 1192., 1208.]], [[1176., 1168., 1152., ..., 1184., 1232., 1192.], [1200., 1192., 1200., ..., 1232., 1208., 1216.], [1160., 1168., 1176., ..., 1248., 1192., 1240.], ..., [1176., 1184., 1216., ..., 1168., 1208., 1184.], [1200., 1216., 1208., ..., 1192., 1192., 1184.], [1200., 1192., 1208., ..., 1192., 1216., 1216.]]]])

test_conv2d.test_module[Conv2D660-[((1, 16, 322, 1026), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 58834944 B L1 buffer across 64 banks, where each bank needs to store 919296 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f6fbc109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f6fbc109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464]

test_conv2d.test_module[Conv2D655-[((1, 256, 14, 42), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[560.24194, 581.34833, 571.80017, ..., 576.99066, 561.53192, 568.37518], [566.32520, 579.20441, 579.23682, ..., 571.70587, 558.76562, 563.67371], [577.74042, 582.62952, 572.86774, ..., 585.48383, 562.90503, 567.58014], ..., [576.27997, 581.39777, 580.92761, ..., 579.49548, 586.74780, 572.02802], [581.32031, 577.58685, 567.31409, ..., 578.51367, 571.56860, 590.05078], [570.72119, 575.45166, 576.49817, ..., 574.43719, 578.47168, 585.98773]], [[576.41278, 575.53644, 584.18817, ..., 571.97815, 570.41455, 573.35553], [576.67780, 585.78931, 581.69958, ..., 577.08606, 557.68402, 560.82916], [567.61401, 580.68103, 580.32697, ..., 575.56018, 554.31470, 562.43152], ..., [581.89270, 577.19482, 581.16492, ..., 582.38660, 583.32019, 576.03088], [578.58417, 581.10547, 580.64270, ..., 578.86700, 580.30963, 580.54889], [570.78778, 576.98383, 562.55438, ..., 572.55298, 575.87823, 584.03693]], [[575.79712, 585.29211, 579.44891, ..., 583.62354, 571.03497, 575.84198], [582.14899, 593.78113, 593.22864, ..., 584.81787, 569.98926, 570.57147], [586.46112, 591.87842, 588.80780, ..., 585.02197, 567.09180, 566.47473], ..., [583.82184, 599.00128, 585.40100, ..., 589.35156, 585.40906, 583.97632], [590.77814, 580.98535, 579.34064, ..., 587.70435, 593.90900, 591.38855], [576.23657, 579.24939, 589.08185, ..., 577.10681, 589.58612, 590.31781]], ..., [[582.24689, 598.36481, 589.33051, ..., 592.41522, 577.64575, 590.01398], [584.44873, 604.28766, 598.16248, ..., 587.94208, 581.64960, 582.75897], [589.30798, 597.47485, 587.97119, ..., 596.52802, 577.32843, 572.57227], ..., [589.75531, 595.61420, 586.30615, ..., 593.83435, 595.44751, 592.30334], [588.09229, 594.12219, 593.00244, ..., 599.85382, 599.24963, 595.07874], [591.09570, 585.25763, 588.66010, ..., 586.72394, 590.83386, 592.61993]], [[556.37317, 569.45837, 561.84747, ..., 566.63458, 550.58057, 561.62848], [559.54315, 574.38696, 562.83246, ..., 568.26331, 547.70728, 556.44037], [562.20038, 573.63428, 566.40747, ..., 566.66730, 539.86816, 551.99768], ..., [575.13367, 576.23108, 567.54492, ..., 567.11707, 571.16620, 568.85419], [573.37811, 566.24646, 565.93732, ..., 572.40808, 566.43097, 571.99036], [567.96906, 569.20941, 567.86792, ..., 565.71741, 566.68420, 574.80133]], [[564.85919, 570.30139, 568.01111, ..., 565.77307, 555.71820, 562.75262], [566.60742, 578.71698, 568.08289, ..., 566.03741, 553.35803, 555.51465], [565.47070, 571.87744, 561.45105, ..., 567.77856, 552.35529, 548.96118], ..., [572.19238, 574.45667, 569.71729, ..., 570.70850, 579.14276, 571.45599], [571.98901, 567.19781, 570.04395, ..., 567.78241, 570.74652, 571.75238], [569.07111, 570.12555, 566.56104, ..., 564.65680, 567.13727, 567.29144]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[556., 588., 572., ..., 592., 572., 560.], [568., 588., 588., ..., 568., 552., 560.], [572., 576., 568., ..., 588., 564., 560.], ..., [572., 572., 580., ..., 572., 592., 564.], [568., 580., 580., ..., 576., 576., 600.], [576., 580., 580., ..., 572., 588., 588.]], [[576., 584., 596., ..., 580., 564., 576.], [576., 592., 584., ..., 576., 568., 556.], [580., 584., 580., ..., 584., 552., 564.], ..., [580., 576., 588., ..., 588., 580., 580.], [584., 580., 592., ..., 580., 580., 580.], [576., 584., 576., ..., 576., 584., 588.]], [[580., 588., 588., ..., 568., 572., 564.], [576., 604., 592., ..., 580., 576., 568.], [596., 604., 592., ..., 588., 568., 556.], ..., [588., 600., 592., ..., 588., 580., 584.], [592., 588., 584., ..., 588., 592., 584.], [580., 588., 592., ..., 576., 592., 592.]], ..., [[580., 596., 580., ..., 584., 568., 580.], [580., 608., 596., ..., 588., 576., 576.], [584., 592., 584., ..., 592., 596., 568.], ..., [588., 588., 588., ..., 592., 584., 592.], [584., 584., 592., ..., 604., 596., 592.], [588., 588., 588., ..., 588., 596., 588.]], [[564., 576., 564., ..., 576., 564., 564.], [560., 580., 568., ..., 568., 552., 568.], [564., 584., 564., ..., 568., 544., 556.], ..., [576., 572., 564., ..., 568., 576., 572.], [572., 564., 576., ..., 580., 568., 580.], [584., 576., 580., ..., 576., 584., 584.]], [[572., 568., 568., ..., 572., 556., 552.], [572., 580., 584., ..., 568., 548., 548.], [564., 572., 556., ..., 564., 556., 552.], ..., [568., 572., 572., ..., 580., 576., 576.], [568., 568., 572., ..., 564., 580., 568.], [572., 576., 572., ..., 568., 576., 572.]]]])

test_conv2d.test_module[Conv2D656-[((1, 128, 26, 82), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[288.36743, 282.60739, 277.20721, ..., 282.78070, 287.33817, 283.85458], [291.65781, 277.95773, 281.00327, ..., 294.47122, 291.14139, 289.62183], [291.41397, 294.35458, 285.25934, ..., 289.87003, 292.00305, 295.42209], ..., [279.76352, 279.19073, 293.51294, ..., 286.13126, 297.52560, 287.79730], [280.81824, 281.27841, 292.67404, ..., 290.38977, 298.00626, 294.59955], [292.39886, 283.82269, 291.80313, ..., 286.79660, 294.57666, 293.79568]], [[287.32233, 278.13284, 276.30963, ..., 282.62222, 283.72491, 273.98199], [283.07819, 282.80017, 278.08249, ..., 283.65424, 288.84265, 294.57523], [287.81729, 292.54913, 277.77075, ..., 290.24500, 288.50052, 290.51227], ..., [276.57977, 281.85538, 285.76086, ..., 283.59055, 294.22147, 282.86282], [278.87756, 277.73901, 283.37247, ..., 285.76578, 287.37555, 283.08054], [280.41315, 281.54239, 285.90802, ..., 288.93323, 294.36151, 290.49905]], [[278.29562, 274.84952, 273.22665, ..., 277.68259, 278.36487, 276.38275], [282.88370, 280.95312, 271.25824, ..., 286.41507, 287.88666, 289.39526], [286.94827, 279.35678, 282.94675, ..., 284.48853, 289.26059, 289.87485], ..., [287.15521, 277.19794, 281.52063, ..., 279.86996, 281.79446, 282.00485], [277.91977, 276.68301, 282.21240, ..., 283.70056, 286.63708, 279.64655], [278.28299, 284.00278, 276.98727, ..., 285.90460, 280.01822, 281.79300]], ..., [[282.45572, 279.74005, 277.48761, ..., 277.29349, 287.56952, 280.38943], [286.52582, 285.59271, 283.41299, ..., 286.36237, 290.49606, 292.08820], [289.85358, 289.68472, 283.40952, ..., 290.00058, 293.84344, 296.92892], ..., [282.51666, 284.58249, 287.69614, ..., 287.49066, 293.84137, 285.41690], [282.40338, 279.42957, 291.32758, ..., 284.95050, 294.10651, 287.44739], [278.99936, 272.96899, 287.63159, ..., 285.26578, 288.94791, 290.62799]], [[288.31223, 281.80499, 276.98111, ..., 281.69357, 284.00662, 277.64728], [287.88498, 282.07394, 282.51843, ..., 291.19748, 291.14844, 297.82477], [289.65686, 293.74319, 286.08145, ..., 292.13815, 293.22791, 298.65396], ..., [280.44470, 284.89938, 290.92395, ..., 287.91833, 294.23557, 288.03482], [284.18280, 285.19315, 289.97009, ..., 289.41901, 292.49384, 291.22778], [283.32578, 283.68326, 285.04892, ..., 289.31702, 296.40784, 288.94305]], [[290.23904, 288.52066, 276.97614, ..., 289.43594, 287.18997, 285.68246], [294.40262, 284.74936, 284.82855, ..., 300.18912, 297.58353, 297.27988], [290.19733, 293.80930, 288.52112, ..., 297.81036, 297.05045, 296.98016], ..., [288.03583, 287.47583, 296.54846, ..., 289.24081, 295.78052, 295.60242], [285.58679, 293.97940, 292.07947, ..., 293.51169, 301.65298, 295.90622], [289.72467, 289.00040, 291.60144, ..., 291.37894, 299.48355, 295.53287]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[282., 282., 274., ..., 282., 284., 280.], [286., 274., 278., ..., 294., 290., 286.], [286., 290., 286., ..., 286., 292., 294.], ..., [274., 276., 294., ..., 284., 294., 284.], [280., 282., 292., ..., 288., 294., 294.], [292., 282., 286., ..., 282., 290., 288.]], [[282., 278., 274., ..., 278., 278., 268.], [278., 280., 274., ..., 278., 284., 292.], [288., 294., 274., ..., 288., 284., 282.], ..., [276., 278., 280., ..., 284., 292., 282.], [276., 276., 280., ..., 284., 284., 278.], [274., 282., 284., ..., 284., 288., 286.]], [[280., 272., 270., ..., 276., 272., 272.], [280., 278., 272., ..., 286., 284., 286.], [282., 278., 280., ..., 280., 288., 288.], ..., [284., 278., 282., ..., 274., 276., 278.], [274., 268., 278., ..., 282., 286., 280.], [276., 278., 274., ..., 280., 278., 282.]], ..., [[280., 280., 276., ..., 274., 288., 274.], [284., 284., 278., ..., 282., 286., 288.], [286., 288., 282., ..., 284., 292., 294.], ..., [276., 282., 286., ..., 282., 292., 288.], [274., 276., 288., ..., 282., 292., 284.], [274., 268., 284., ..., 280., 288., 290.]], [[284., 278., 272., ..., 278., 282., 276.], [282., 278., 276., ..., 284., 290., 296.], [292., 288., 280., ..., 284., 286., 296.], ..., [274., 280., 290., ..., 284., 292., 286.], [282., 284., 286., ..., 282., 288., 288.], [280., 280., 284., ..., 286., 290., 286.]], [[286., 286., 272., ..., 286., 280., 284.], [292., 278., 278., ..., 298., 298., 294.], [292., 294., 284., ..., 298., 290., 296.], ..., [282., 286., 296., ..., 286., 294., 294.], [280., 288., 290., ..., 292., 294., 294.], [286., 286., 288., ..., 292., 296., 288.]]]])

test_conv2d.test_module[Conv2D657-[((1, 64, 50, 162), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[141.11235, 144.94984, 145.93062, ..., 145.82292, 145.16426, 148.97122], [145.64433, 146.67810, 144.03807, ..., 139.36337, 144.67151, 150.28070], [142.74672, 145.40239, 144.52342, ..., 149.30457, 147.96440, 151.24969], ..., [146.81578, 148.77766, 148.63995, ..., 147.44313, 145.52431, 147.17146], [146.89825, 149.15239, 144.94560, ..., 141.53137, 146.24774, 145.37305], [143.47475, 147.61548, 143.41313, ..., 147.34344, 147.47304, 145.03236]], [[151.08482, 152.93883, 148.12598, ..., 147.36800, 150.22133, 154.24597], [154.22745, 150.94441, 152.52562, ..., 151.02376, 155.40561, 157.01593], [151.55743, 152.22047, 154.64050, ..., 151.33261, 155.60060, 154.09238], ..., [145.50334, 155.54886, 156.45448, ..., 152.27107, 151.15242, 147.61903], [148.88199, 150.38309, 147.64296, ..., 150.91789, 152.66818, 154.03716], [151.52138, 151.03290, 147.83160, ..., 153.93288, 150.92929, 148.94603]], [[138.43376, 144.52237, 146.60555, ..., 140.67665, 143.45685, 145.38690], [144.86801, 143.43947, 144.75397, ..., 144.86848, 144.47375, 146.78250], [144.63954, 143.87779, 140.93114, ..., 145.08734, 146.83023, 149.22667], ..., [139.73257, 147.60970, 146.26329, ..., 144.98299, 150.85881, 144.31015], [146.26410, 145.88159, 143.61946, ..., 146.40958, 147.29588, 144.73978], [142.87889, 146.82765, 147.58356, ..., 145.13571, 145.14528, 145.75423]], ..., [[142.67822, 145.69507, 139.45233, ..., 140.73532, 140.79221, 147.21278], [143.16261, 144.46896, 145.55989, ..., 145.78598, 146.98987, 149.07263], [144.06531, 145.88654, 144.95662, ..., 146.50920, 144.95723, 149.69049], ..., [144.95598, 145.82307, 147.09395, ..., 148.20686, 146.96416, 143.14182], [141.52408, 146.17114, 140.61145, ..., 144.87781, 145.12369, 144.50746], [142.14249, 145.54941, 144.35901, ..., 147.03778, 148.45590, 148.59793]], [[145.00589, 141.40024, 145.98131, ..., 149.15424, 143.64926, 145.36070], [146.61038, 146.08493, 147.50999, ..., 146.72858, 150.15854, 153.50645], [143.12799, 144.94441, 151.09317, ..., 151.07060, 149.80643, 150.03368], ..., [145.12125, 146.98790, 148.63187, ..., 148.33771, 144.72791, 144.62340], [150.35504, 149.06784, 144.77748, ..., 147.06805, 146.75104, 148.53938], [143.53174, 149.54347, 143.93071, ..., 150.14455, 150.22481, 149.18974]], [[140.75914, 142.08821, 137.50153, ..., 140.83247, 136.42949, 145.31953], [140.05156, 142.12167, 142.91039, ..., 141.87981, 145.83820, 152.15031], [143.98688, 143.15802, 141.99467, ..., 140.94862, 145.11838, 147.26228], ..., [140.64105, 145.04242, 146.63292, ..., 147.66362, 142.45178, 142.44740], [141.93488, 144.39854, 140.28099, ..., 140.94771, 139.01071, 145.79649], [137.86949, 143.69759, 141.27090, ..., 141.13797, 143.09055, 141.16003]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[137., 144., 145., ..., 142., 142., 147.], [144., 144., 141., ..., 138., 141., 148.], [141., 143., 142., ..., 148., 145., 147.], ..., [145., 147., 147., ..., 144., 144., 144.], [145., 146., 142., ..., 140., 144., 143.], [142., 146., 142., ..., 144., 145., 144.]], [[147., 150., 145., ..., 145., 149., 152.], [154., 149., 151., ..., 149., 154., 156.], [149., 151., 151., ..., 148., 154., 151.], ..., [143., 153., 153., ..., 150., 149., 146.], [146., 148., 147., ..., 148., 151., 150.], [149., 149., 146., ..., 152., 150., 147.]], [[136., 142., 145., ..., 140., 141., 142.], [142., 141., 142., ..., 143., 143., 144.], [142., 141., 137., ..., 142., 146., 147.], ..., [138., 144., 144., ..., 144., 150., 142.], [145., 143., 141., ..., 143., 145., 142.], [141., 145., 145., ..., 141., 143., 144.]], ..., [[140., 143., 138., ..., 139., 139., 145.], [140., 142., 144., ..., 143., 146., 147.], [141., 144., 143., ..., 144., 141., 148.], ..., [143., 142., 145., ..., 145., 144., 141.], [140., 144., 139., ..., 144., 144., 141.], [139., 143., 144., ..., 146., 146., 147.]], [[144., 139., 144., ..., 147., 141., 144.], [145., 143., 144., ..., 144., 148., 150.], [140., 143., 148., ..., 147., 147., 148.], ..., [142., 146., 145., ..., 146., 143., 143.], [148., 144., 142., ..., 143., 146., 146.], [142., 146., 142., ..., 148., 149., 147.]], [[139., 140., 135., ..., 139., 133., 143.], [139., 140., 142., ..., 139., 144., 150.], [143., 141., 139., ..., 140., 143., 147.], ..., [138., 141., 145., ..., 146., 140., 141.], [141., 142., 139., ..., 138., 138., 142.], [136., 141., 140., ..., 139., 142., 139.]]]])

test_conv2d.test_module[Conv2D670-[((1, 3, 227, 227), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 3727 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 1031040 and static circular buffer region ends at 1180896 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f6fcca092d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f6fbd6ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f6fbd6ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run_without_autoformat<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x448b70) [0x7f6fbb552b70] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d18optimized_conv_newERKN2tt8tt_metal6TensorES7_St8optionalIS6_ERKNS0_14sliding_window19SlidingWindowConfigEjjbbRKNS2_34OptimizedConvParallelizationConfigERKNS2_24OptimizedConvBlockConfigERKNS4_12MemoryConfigENS4_8DataTypeESt5arrayIjLm4EEbRKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEbbbb+0x6bd) [0x7f6fbb4ea6bd] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xec6) [0x7f6fbb4d9f76] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702]

test_conv2d.test_module[Conv2D698-[((1, 1024, 14, 14), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[248.97490, 247.06262, 248.64525, ..., 239.23615, 249.80551, 244.30768], [255.57858, 249.69220, 256.10519, ..., 249.69421, 246.54645, 253.42145], [248.77843, 252.99403, 255.26292, ..., 256.13666, 247.30611, 246.10910], ..., [245.06549, 253.93835, 236.03700, ..., 244.16608, 244.21490, 250.02245], [248.93758, 257.29980, 257.03897, ..., 245.92253, 258.76724, 254.72809], [247.17267, 266.02216, 248.11053, ..., 246.79672, 251.83871, 253.84309]], [[258.05078, 254.85307, 254.64659, ..., 245.61859, 263.01657, 251.74171], [257.94672, 255.69312, 261.67883, ..., 263.43164, 259.10196, 267.49603], [263.10815, 258.66650, 259.92062, ..., 264.04272, 258.40561, 259.07846], ..., [253.38605, 266.42767, 251.64197, ..., 252.96487, 252.26498, 262.94382], [265.71408, 267.89240, 263.23694, ..., 257.66608, 266.15811, 264.79294], [252.59650, 277.45633, 260.07785, ..., 255.96663, 263.09772, 255.65166]], [[264.90594, 257.49225, 262.92487, ..., 260.79480, 264.32571, 254.92938], [264.37137, 262.96805, 272.44403, ..., 269.23624, 263.42474, 270.80402], [271.99326, 270.93469, 270.42664, ..., 274.17697, 264.39801, 258.09872], ..., [259.57422, 275.45529, 254.66776, ..., 259.31393, 265.05820, 267.38577], [267.85089, 270.82434, 274.10452, ..., 261.83234, 265.53720, 270.57581], [265.70694, 282.62820, 263.00293, ..., 262.65115, 274.03915, 269.38354]], ..., [[263.54428, 256.07022, 262.55270, ..., 254.18385, 265.09991, 261.78336], [262.82993, 262.35205, 264.52991, ..., 268.65768, 258.14096, 273.25375], [264.34879, 263.90366, 271.59894, ..., 268.75519, 265.72815, 259.81442], ..., [262.33856, 273.50125, 256.12518, ..., 257.21786, 266.75891, 270.80588], [270.72598, 274.39175, 270.54529, ..., 254.41335, 273.59561, 275.41238], [265.56360, 279.48248, 260.48474, ..., 261.80704, 266.50238, 261.29236]], [[252.19385, 245.62611, 249.02260, ..., 245.50760, 251.20715, 247.14275], [253.33702, 251.05692, 255.00174, ..., 257.40012, 243.04097, 258.79150], [255.26385, 251.57550, 258.32425, ..., 270.40216, 253.82216, 252.63620], ..., [241.51704, 264.28296, 245.18730, ..., 253.10487, 247.57143, 253.31258], [258.66693, 262.07404, 259.11609, ..., 254.36850, 259.17938, 259.78238], [247.38220, 267.69562, 252.86798, ..., 251.75409, 256.62665, 250.78389]], [[250.37456, 247.95566, 248.50562, ..., 243.34290, 249.70833, 243.16464], [252.81943, 248.99681, 257.26715, ..., 254.63467, 246.44962, 255.48293], [248.31830, 252.39670, 257.63525, ..., 254.53174, 256.50079, 248.82343], ..., [251.50021, 259.38333, 238.04115, ..., 246.68579, 246.22160, 251.98019], [253.21396, 253.55782, 256.71445, ..., 246.44926, 253.37743, 257.71802], [252.79968, 267.12247, 246.90009, ..., 243.41956, 255.60336, 249.50581]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[245., 243., 246., ..., 237., 249., 240.], [252., 245., 253., ..., 247., 240., 249.], [245., 251., 251., ..., 254., 244., 244.], ..., [244., 252., 232., ..., 242., 243., 245.], [246., 254., 255., ..., 243., 252., 250.], [241., 264., 246., ..., 245., 244., 250.]], [[253., 252., 253., ..., 244., 258., 252.], [254., 256., 258., ..., 258., 256., 264.], [260., 256., 258., ..., 262., 255., 256.], ..., [251., 262., 248., ..., 252., 247., 260.], [262., 266., 260., ..., 255., 262., 262.], [252., 272., 262., ..., 253., 258., 251.]], [[262., 253., 256., ..., 258., 262., 251.], [262., 258., 268., ..., 266., 258., 266.], [268., 266., 266., ..., 274., 260., 255.], ..., [255., 272., 252., ..., 256., 260., 260.], [266., 264., 270., ..., 264., 262., 268.], [266., 278., 258., ..., 258., 272., 262.]], ..., [[262., 254., 258., ..., 250., 262., 260.], [262., 258., 262., ..., 266., 251., 272.], [258., 260., 270., ..., 266., 264., 256.], ..., [258., 270., 254., ..., 254., 264., 268.], [270., 266., 268., ..., 252., 266., 272.], [260., 276., 255., ..., 256., 264., 262.]], [[249., 246., 245., ..., 243., 248., 245.], [248., 250., 252., ..., 253., 242., 256.], [251., 252., 253., ..., 270., 250., 246.], ..., [237., 262., 242., ..., 249., 242., 247.], [262., 256., 254., ..., 255., 255., 256.], [244., 266., 253., ..., 249., 255., 248.]], [[245., 246., 244., ..., 240., 250., 238.], [253., 247., 251., ..., 253., 243., 251.], [246., 249., 255., ..., 251., 252., 247.], ..., [252., 260., 234., ..., 242., 244., 248.], [247., 251., 252., ..., 240., 249., 255.], [248., 264., 244., ..., 239., 253., 247.]]]])

test_conv2d.test_module[Conv2D703-[((1, 2048, 15, 20), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[2093.51416, 3086.13696, 3081.06641, ..., 3089.90552, 3073.50830, 3107.86426], [3087.50439, 4631.53223, 4557.00244, ..., 4650.71143, 4607.59277, 4597.79980], [3080.13550, 4624.21143, 4590.03271, ..., 4640.69238, 4645.60059, 4607.47900], ..., [3119.36865, 4663.63721, 4623.83496, ..., 4600.82471, 4647.12988, 4635.30762], [3094.32935, 4606.16553, 4662.71094, ..., 4599.50537, 4630.93018, 4601.37646], [2059.07812, 3060.88208, 3097.23999, ..., 3090.84741, 3092.76001, 3056.63403]], [[2026.85693, 3052.39941, 3021.07446, ..., 3035.12158, 3030.16602, 3070.41895], [3061.00024, 4602.60205, 4510.27490, ..., 4614.25635, 4594.24854, 4605.13818], [3067.19458, 4582.15088, 4547.79834, ..., 4607.89893, 4629.21533, 4572.29248], ..., [3073.43799, 4647.42139, 4600.20605, ..., 4602.01562, 4616.00146, 4591.41846], [3061.98047, 4568.31152, 4652.29785, ..., 4598.83301, 4601.34570, 4563.28662], [2061.59326, 3072.91382, 3090.53442, ..., 3080.84619, 3091.38184, 3074.47046]], [[2070.28711, 3090.93896, 3065.47998, ..., 3068.96704, 3093.25366, 3098.54004], [3097.19580, 4655.44971, 4580.25391, ..., 4664.29395, 4631.80127, 4630.86914], [3114.33838, 4662.97314, 4600.47266, ..., 4662.63525, 4669.87500, 4639.70312], ..., [3118.80737, 4691.43848, 4648.46240, ..., 4630.84570, 4653.16162, 4634.85010], [3107.26465, 4610.62549, 4679.48975, ..., 4644.53516, 4647.38135, 4608.66357], [2096.87134, 3063.47949, 3135.12305, ..., 3111.71362, 3116.43970, 3069.30371]], ..., [[2041.80103, 3034.98706, 3026.69116, ..., 3018.08838, 3030.35498, 3060.18774], [3046.71338, 4577.44873, 4512.74902, ..., 4578.69043, 4571.52881, 4569.39551], [3064.63525, 4568.04150, 4530.91895, ..., 4567.91553, 4580.13818, 4563.84082], ..., [3072.97144, 4606.74609, 4596.93506, ..., 4557.30029, 4586.25000, 4563.04932], [3057.35059, 4529.69141, 4626.03760, ..., 4579.18311, 4574.32324, 4537.77002], [2046.15442, 3033.53003, 3069.51489, ..., 3075.77808, 3079.88574, 3023.90747]], [[2061.63916, 3066.41309, 3041.40063, ..., 3043.77051, 3060.93408, 3079.68091], [3057.77417, 4597.72510, 4497.53223, ..., 4606.72070, 4568.33154, 4586.64355], [3053.18774, 4590.26416, 4559.48633, ..., 4595.78760, 4603.55957, 4601.56445], ..., [3077.27686, 4632.31592, 4622.07178, ..., 4591.96875, 4593.77148, 4598.70459], [3048.74634, 4573.72412, 4656.31982, ..., 4598.23779, 4617.48682, 4553.35303], [2035.04175, 3031.72021, 3071.41797, ..., 3048.93652, 3076.39062, 3032.87378]], [[2081.65967, 3083.07446, 3065.81763, ..., 3071.25171, 3093.99536, 3106.73193], [3095.84644, 4646.38232, 4574.36475, ..., 4645.38525, 4627.88965, 4620.54590], [3090.24390, 4618.41113, 4599.86621, ..., 4647.95996, 4658.50635, 4637.92090], ..., [3122.84619, 4657.70557, 4647.27783, ..., 4623.82031, 4650.14502, 4632.98828], [3090.91846, 4589.96338, 4669.71094, ..., 4651.94482, 4657.60400, 4598.59229], [2062.01489, 3046.92383, 3088.65723, ..., 3095.73193, 3110.57739, 3052.71851]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[2048., 2064., 2080., ..., 2048., 2048., 2080.], [2064., 2048., 2064., ..., 2144., 2096., 2080.], [2048., 2080., 2048., ..., 2064., 2080., 2064.], ..., [2096., 2080., 2048., ..., 2080., 2064., 2080.], [2064., 2080., 2080., ..., 2080., 2128., 2096.], [2048., 2080., 2080., ..., 2064., 2048., 2048.]], [[2048., 2064., 2048., ..., 2048., 2064., 2080.], [2096., 2096., 2080., ..., 2048., 2096., 2080.], [2080., 2080., 2096., ..., 2112., 2144., 2048.], ..., [2048., 2064., 2080., ..., 2080., 2080., 2064.], [2080., 2064., 2096., ..., 2064., 2048., 2096.], [2048., 2080., 2064., ..., 2048., 2112., 2048.]], [[2048., 2064., 2096., ..., 2048., 2096., 2064.], [2064., 2096., 2112., ..., 2096., 2112., 2080.], [2064., 2080., 2064., ..., 2096., 2064., 2048.], ..., [2064., 2080., 2080., ..., 2064., 2080., 2064.], [2048., 2080., 2064., ..., 2048., 2080., 2080.], [2048., 2048., 2096., ..., 2064., 2080., 2080.]], ..., [[2048., 2064., 2064., ..., 2048., 2048., 2048.], [2048., 2048., 2096., ..., 2080., 2080., 2064.], [2048., 2096., 2048., ..., 2064., 2064., 2080.], ..., [2064., 2064., 2048., ..., 2080., 2064., 2064.], [2048., 2064., 2064., ..., 2048., 2080., 2080.], [2048., 2048., 2048., ..., 2048., 2048., 2048.]], [[2048., 2064., 2080., ..., 2064., 2048., 2064.], [2048., 2064., 2064., ..., 2064., 2064., 2064.], [2064., 2096., 2064., ..., 2048., 2064., 2064.], ..., [2064., 2064., 2080., ..., 2064., 2096., 2080.], [2048., 2048., 2160., ..., 2080., 2112., 2064.], [2048., 2048., 2080., ..., 2064., 2064., 2048.]], [[2064., 2048., 2048., ..., 2048., 2064., 2064.], [2064., 2048., 2048., ..., 2080., 2080., 2048.], [2064., 2080., 2080., ..., 2048., 2048., 2048.], ..., [2064., 2064., 2048., ..., 2080., 2096., 2048.], [2064., 2048., 2048., ..., 2048., 2112., 2064.], [2048., 2064., 2064., ..., 2064., 2064., 2048.]]]])

test_conv2d.test_module[Conv2D707-[((1, 32, 128, 128), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[515.37683, 488.84088, 509.62271, ..., 502.58002, 508.32040, 505.51184], [516.37018, 506.49615, 516.33636, ..., 514.67224, 514.65491, 504.88443], [504.70779, 508.21100, 504.55209, ..., 503.05237, 502.97852, 497.95685], ..., [524.12170, 506.60645, 509.65512, ..., 502.05493, 512.92566, 504.14243], [508.37973, 511.14688, 497.98486, ..., 499.41620, 511.03577, 521.76575], [504.66653, 512.36597, 516.89746, ..., 504.71890, 500.01175, 508.75906]], [[515.72595, 496.14648, 519.11713, ..., 519.53784, 508.10132, 505.77274], [523.23199, 520.98523, 514.92487, ..., 525.18744, 522.95660, 509.77820], [520.40698, 511.79318, 501.01303, ..., 510.05963, 517.46869, 505.73206], ..., [527.32697, 518.35901, 516.39691, ..., 519.65643, 515.77673, 514.41479], [513.35651, 515.87781, 518.36072, ..., 507.84006, 515.08124, 532.57904], [519.24176, 529.71313, 523.16675, ..., 510.09686, 502.95273, 524.63495]], [[519.51379, 487.22479, 512.32922, ..., 518.73608, 510.13852, 507.72842], [513.26471, 517.21637, 511.95218, ..., 523.49567, 521.24420, 511.02359], [506.25632, 508.19180, 504.33688, ..., 510.33005, 504.38373, 500.87793], ..., [523.61365, 509.45840, 512.43213, ..., 515.35577, 514.73383, 509.01126], [505.18982, 511.48743, 505.06964, ..., 509.08698, 504.38147, 522.68604], [522.08759, 528.74957, 524.56927, ..., 505.62939, 492.95560, 522.33386]], ..., [[513.21832, 495.86865, 503.86188, ..., 519.81726, 509.52649, 505.01498], [516.45325, 515.36719, 520.06610, ..., 522.20251, 516.07373, 504.86124], [511.91794, 510.67462, 497.97433, ..., 503.32626, 501.61832, 509.38477], ..., [519.39148, 511.92197, 505.30661, ..., 510.07083, 512.82874, 505.16724], [509.87119, 506.40488, 507.37714, ..., 501.93103, 514.03925, 520.51251], [513.71503, 524.10693, 519.07806, ..., 506.84866, 495.20404, 516.90387]], [[522.23297, 499.46796, 523.58728, ..., 521.68378, 520.94312, 504.02545], [525.42285, 525.11426, 529.50470, ..., 527.72003, 523.19690, 520.99255], [510.84052, 516.79449, 507.05487, ..., 519.24506, 510.11963, 512.11206], ..., [531.41180, 519.26630, 511.82681, ..., 521.14246, 520.41833, 518.63538], [515.28870, 511.89893, 510.96820, ..., 515.10687, 519.66003, 534.49359], [516.78400, 527.40839, 521.84106, ..., 510.69791, 502.91083, 524.51593]], [[503.82413, 486.07037, 501.31070, ..., 503.68124, 500.39999, 495.32245], [504.31332, 516.09406, 510.85501, ..., 513.27124, 506.15530, 492.02225], [510.02023, 501.23520, 503.91858, ..., 504.72440, 498.29095, 498.12939], ..., [514.96875, 503.92966, 505.37659, ..., 508.45892, 510.41208, 501.57751], [502.16104, 495.20737, 501.21869, ..., 489.18909, 498.32382, 514.31482], [510.68076, 512.68005, 507.28586, ..., 490.60596, 494.02588, 506.41748]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[520., 496., 504., ..., 506., 516., 508.], [524., 492., 516., ..., 520., 512., 508.], [502., 512., 508., ..., 502., 498., 494.], ..., [528., 510., 506., ..., 502., 510., 504.], [516., 516., 500., ..., 496., 506., 520.], [504., 506., 516., ..., 500., 496., 504.]], [[516., 494., 512., ..., 516., 512., 506.], [528., 516., 508., ..., 520., 528., 506.], [524., 516., 502., ..., 508., 516., 500.], ..., [532., 520., 520., ..., 524., 524., 504.], [512., 516., 516., ..., 512., 520., 532.], [512., 532., 520., ..., 510., 496., 524.]], [[504., 496., 506., ..., 512., 512., 516.], [504., 524., 506., ..., 528., 528., 498.], [498., 508., 510., ..., 510., 510., 504.], ..., [528., 516., 516., ..., 508., 520., 498.], [512., 496., 498., ..., 500., 494., 508.], [520., 532., 528., ..., 502., 496., 520.]], ..., [[500., 490., 496., ..., 516., 494., 498.], [512., 508., 512., ..., 520., 516., 508.], [520., 508., 502., ..., 500., 506., 520.], ..., [512., 512., 502., ..., 520., 512., 500.], [510., 500., 504., ..., 500., 516., 520.], [516., 520., 516., ..., 510., 498., 510.]], [[524., 498., 524., ..., 520., 524., 498.], [524., 528., 524., ..., 524., 520., 516.], [506., 516., 500., ..., 516., 512., 512.], ..., [528., 516., 504., ..., 528., 516., 516.], [516., 512., 506., ..., 506., 520., 532.], [516., 524., 524., ..., 516., 500., 520.]], [[496., 486., 504., ..., 504., 492., 494.], [508., 510., 516., ..., 508., 512., 490.], [520., 498., 502., ..., 502., 500., 506.], ..., [510., 500., 510., ..., 508., 516., 494.], [492., 498., 496., ..., 486., 502., 512.], [510., 512., 510., ..., 494., 494., 504.]]]])

test_conv2d.test_module[Conv2D727-[((1, 128, 5, 5), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[279.43875, 283.31268, 282.87909], [276.32739, 281.44363, 282.10828], [283.10468, 279.22168, 277.90091]], [[286.20624, 285.42975, 296.71564], [279.28653, 281.19595, 282.03415], [283.11984, 281.38818, 279.23053]], [[288.35519, 294.12750, 289.25269], [286.58554, 287.16568, 292.81241], [292.45013, 289.76672, 294.83322]], ..., [[284.67755, 290.71780, 284.96930], [279.09143, 283.70978, 283.89438], [283.12518, 284.97977, 282.84329]], [[278.22440, 291.35330, 286.75027], [273.69745, 278.77271, 283.69131], [278.45496, 284.10745, 276.23761]], [[295.83572, 293.61343, 296.24344], [282.82703, 287.89587, 285.19153], [289.28113, 293.88535, 287.29184]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[280., 282., 280.], [276., 280., 280.], [280., 278., 274.]], [[282., 284., 294.], [278., 278., 280.], [280., 280., 274.]], [[284., 290., 286.], [288., 288., 292.], [292., 288., 292.]], ..., [[278., 288., 286.], [276., 276., 284.], [282., 284., 278.]], [[276., 286., 284.], [272., 276., 278.], [282., 282., 272.]], [[294., 288., 296.], [278., 282., 286.], [284., 294., 284.]]]])

test_conv2d.test_module[Conv2D738-[((1, 1024, 28, 28), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1017.30469, 1533.94507, 1542.49622, ..., 1501.96899, 1495.33472, 1019.47522], [1531.11462, 2309.69214, 2296.23291, ..., 2276.68799, 2264.07593, 1534.58105], [1518.18872, 2299.47290, 2282.08716, ..., 2277.17065, 2261.40039, 1544.90759], ..., [1517.61523, 2285.24707, 2293.28247, ..., 2281.69556, 2282.46899, 1537.13245], [1534.19250, 2300.46411, 2296.94849, ..., 2295.73145, 2294.50146, 1541.69177], [1015.31238, 1538.48328, 1532.43506, ..., 1514.57983, 1515.48145, 1028.40515]], [[1035.12500, 1565.01599, 1555.60205, ..., 1536.57642, 1518.90295, 1009.87689], [1545.99097, 2327.81885, 2344.81494, ..., 2292.02148, 2273.04834, 1513.60388], [1539.65454, 2322.02344, 2312.76367, ..., 2303.04102, 2298.83496, 1533.82068], ..., [1539.87280, 2320.98022, 2324.77710, ..., 2296.60693, 2317.92236, 1529.76599], [1555.74231, 2333.28540, 2324.98169, ..., 2302.91187, 2309.67871, 1546.17200], [1037.31799, 1561.40039, 1561.02637, ..., 1546.33728, 1541.81091, 1011.84161]], [[1025.53040, 1567.45923, 1553.27795, ..., 1517.54211, 1506.44946, 1006.52234], [1542.89722, 2328.68921, 2338.26172, ..., 2289.52075, 2298.35913, 1529.10803], [1528.41345, 2322.41309, 2342.51245, ..., 2296.42700, 2298.23877, 1548.31372], ..., [1544.18921, 2309.00977, 2302.86719, ..., 2278.93872, 2297.09253, 1544.84094], [1544.44958, 2321.35718, 2329.18140, ..., 2301.60010, 2291.22437, 1547.97375], [1018.37592, 1550.51001, 1552.94751, ..., 1533.25159, 1523.66077, 1019.24945]], ..., [[1029.23035, 1556.23364, 1555.23853, ..., 1524.54858, 1511.84753, 1015.79938], [1528.02710, 2324.41724, 2321.95166, ..., 2300.05981, 2265.43945, 1527.50720], [1521.99524, 2310.65601, 2317.40430, ..., 2289.49463, 2295.14771, 1537.03442], ..., [1529.15283, 2306.29321, 2304.13281, ..., 2306.81396, 2286.26074, 1540.64270], [1540.12732, 2326.70459, 2325.40112, ..., 2301.70801, 2307.36035, 1533.05591], [1025.89600, 1550.21387, 1553.61438, ..., 1529.49219, 1524.29553, 1015.00269]], [[1030.71741, 1540.65051, 1541.74683, ..., 1510.87146, 1491.39709, 980.76373], [1548.90100, 2311.37354, 2309.92261, ..., 2287.46509, 2257.40063, 1505.97351], [1529.22034, 2293.99731, 2307.56812, ..., 2277.14502, 2284.69702, 1531.77808], ..., [1547.63452, 2285.93140, 2291.33643, ..., 2301.48535, 2274.40649, 1522.38245], [1541.16760, 2312.49536, 2298.98853, ..., 2300.91455, 2285.57422, 1522.69482], [1022.66284, 1527.91223, 1524.75232, ..., 1510.07874, 1525.52429, 996.18323]], [[1020.22803, 1532.58325, 1524.51111, ..., 1501.23413, 1492.95251, 997.93982], [1538.24219, 2303.93750, 2308.77368, ..., 2267.83105, 2250.70532, 1504.84717], [1528.49170, 2286.18237, 2304.14624, ..., 2287.91602, 2278.11987, 1536.83215], ..., [1526.23071, 2302.92920, 2300.74805, ..., 2275.22974, 2273.24683, 1526.72253], [1542.47351, 2312.60571, 2306.10034, ..., 2302.10034, 2283.31860, 1527.46875], [1017.26752, 1525.89465, 1528.47058, ..., 1521.76709, 1510.36292, 1000.19672]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1024., 1720., 1752., ..., 1728., 1696., 1024.], [1680., 2080., 2064., ..., 2048., 2048., 1728.], [1672., 2048., 2048., ..., 2048., 2048., 1760.], ..., [1704., 2048., 2064., ..., 2080., 2048., 1656.], [1784., 2048., 2064., ..., 2064., 2048., 1696.], [1024., 1736., 1760., ..., 1680., 1664., 1040.]], [[1080., 1744., 1728., ..., 1704., 1752., 1024.], [1688., 2064., 2048., ..., 2064., 2064., 1712.], [1688., 2048., 2048., ..., 2048., 2048., 1632.], ..., [1760., 2048., 2048., ..., 2048., 2048., 1616.], [1712., 2048., 2048., ..., 2064., 2048., 1704.], [1064., 1752., 1704., ..., 1656., 1696., 1024.]], [[1016., 1776., 1752., ..., 1688., 1728., 996.], [1696., 2048., 2080., ..., 2064., 2064., 1744.], [1680., 2080., 2080., ..., 2048., 2080., 1776.], ..., [1824., 2048., 2048., ..., 2048., 2048., 1728.], [1760., 2064., 2064., ..., 2048., 2080., 1680.], [1040., 1800., 1736., ..., 1720., 1672., 1008.]], ..., [[1040., 1784., 1784., ..., 1680., 1736., 1032.], [1752., 2064., 2048., ..., 2048., 2048., 1680.], [1656., 2048., 2064., ..., 2064., 2048., 1736.], ..., [1696., 2048., 2048., ..., 2048., 2048., 1688.], [1680., 2048., 2064., ..., 2048., 2048., 1648.], [1024., 1760., 1672., ..., 1768., 1664., 1032.]], [[1064., 1704., 1728., ..., 1704., 1648., 1000.], [1712., 2064., 2048., ..., 2048., 2048., 1688.], [1696., 2064., 2080., ..., 2064., 2048., 1664.], ..., [1712., 2048., 2080., ..., 2064., 2048., 1640.], [1720., 2064., 2064., ..., 2048., 2064., 1696.], [1020., 1736., 1736., ..., 1624., 1664., 1008.]], [[1016., 1720., 1672., ..., 1680., 1672., 996.], [1704., 2048., 2064., ..., 2064., 2048., 1672.], [1704., 2048., 2048., ..., 2048., 2048., 1688.], ..., [1760., 2048., 2064., ..., 2048., 2080., 1696.], [1736., 2048., 2064., ..., 2048., 2048., 1672.], [1020., 1672., 1760., ..., 1752., 1680., 1008.]]]])

test_conv2d.test_module[Conv2D662-[((1, 128, 224, 224), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 4310 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 410624 and static circular buffer region ends at 509600 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f6fcca092d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f6fbd6ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f6fbd6ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab]

test_conv2d.test_module[Conv2D757-[((1, 1024, 7, 7), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1029.88147, 1535.99963, 1536.48706, ..., 1544.77588, 1542.93030, 1020.91272], [1519.82141, 2316.50220, 2321.88550, ..., 2313.95581, 2322.57690, 1531.11084], [1543.85632, 2325.64990, 2328.94067, ..., 2311.84521, 2300.97095, 1531.83948], ..., [1556.84399, 2343.30444, 2322.29688, ..., 2289.17773, 2298.14819, 1517.39685], [1553.17908, 2347.65649, 2313.58887, ..., 2311.88892, 2292.60962, 1524.09717], [1031.99011, 1553.45715, 1549.54480, ..., 1551.93176, 1536.72046, 1019.34326]], [[1017.29907, 1530.05994, 1520.71167, ..., 1535.94104, 1530.62439, 1007.62506], [1531.39783, 2305.22974, 2298.66724, ..., 2296.57886, 2283.76001, 1518.92505], [1523.98303, 2300.97925, 2315.75244, ..., 2297.33545, 2289.36060, 1522.33740], ..., [1537.48267, 2303.77661, 2308.88232, ..., 2265.98145, 2270.69800, 1503.37866], [1542.38794, 2296.36279, 2313.79761, ..., 2299.08716, 2291.51831, 1505.70813], [1025.42383, 1532.22302, 1552.92700, ..., 1542.87622, 1531.11792, 1000.95386]], [[1036.28333, 1534.32422, 1528.67249, ..., 1532.49548, 1532.86658, 1001.81842], [1550.61584, 2303.92480, 2316.71777, ..., 2315.01733, 2296.65283, 1530.74634], [1548.13318, 2307.73291, 2323.79663, ..., 2306.71216, 2289.64648, 1508.28674], ..., [1558.13000, 2315.17334, 2311.61328, ..., 2275.99438, 2284.92041, 1501.46106], [1566.22192, 2305.99243, 2296.18481, ..., 2305.74927, 2278.40552, 1512.22485], [1044.82056, 1549.12390, 1551.58618, ..., 1549.63208, 1539.80872, 1015.84009]], ..., [[ 997.83185, 1517.07349, 1505.12622, ..., 1532.82678, 1515.74670, 1014.70581], [1517.14148, 2295.20825, 2287.50049, ..., 2300.71069, 2272.65430, 1507.54358], [1507.72510, 2308.69653, 2297.18457, ..., 2285.79688, 2292.57080, 1517.64001], ..., [1531.33960, 2311.32666, 2291.70068, ..., 2279.33398, 2267.08545, 1510.38379], [1533.05164, 2285.05811, 2280.44312, ..., 2282.51904, 2273.09351, 1505.80347], [1018.84558, 1534.50378, 1535.81458, ..., 1534.67932, 1531.22021, 1011.46655]], [[1007.64471, 1535.16504, 1526.70312, ..., 1547.70935, 1540.54272, 1032.55042], [1535.30518, 2313.13428, 2316.27783, ..., 2311.40015, 2298.04248, 1539.30957], [1519.54089, 2310.77881, 2320.85229, ..., 2304.01343, 2304.24683, 1535.22217], ..., [1540.51697, 2327.97583, 2332.15942, ..., 2274.55005, 2282.23853, 1524.14758], [1531.88330, 2308.57544, 2303.55615, ..., 2304.10645, 2291.39355, 1540.61792], [1028.08191, 1529.55139, 1549.39124, ..., 1536.71814, 1525.62720, 1017.22504]], [[1029.39050, 1534.23608, 1529.35815, ..., 1548.80676, 1543.59937, 1023.72595], [1530.54944, 2309.06274, 2313.62646, ..., 2307.76440, 2302.74194, 1542.38818], [1520.86914, 2319.84766, 2330.48462, ..., 2309.38428, 2313.59888, 1545.34595], ..., [1554.52515, 2329.90137, 2319.46948, ..., 2285.08301, 2290.48340, 1533.41443], [1553.62158, 2313.65137, 2320.38867, ..., 2311.92847, 2318.78516, 1522.53796], [1037.88989, 1537.37695, 1551.15320, ..., 1551.17432, 1532.78528, 1034.72180]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1040., 1696., 1640., ..., 1624., 1704., 1040.], [1640., 2048., 2048., ..., 2048., 2048., 1728.], [1680., 2064., 2080., ..., 2064., 2048., 1704.], ..., [1752., 2080., 2064., ..., 2048., 2048., 1688.], [1712., 2048., 2064., ..., 2048., 2064., 1688.], [1064., 1696., 1656., ..., 1704., 1672., 1040.]], [[1024., 1696., 1696., ..., 1648., 1688., 1020.], [1760., 2048., 2048., ..., 2048., 2064., 1688.], [1696., 2048., 2064., ..., 2048., 2048., 1688.], ..., [1728., 2048., 2048., ..., 2048., 2064., 1640.], [1680., 2048., 2048., ..., 2048., 2048., 1656.], [1024., 1664., 1728., ..., 1752., 1696., 1024.]], [[1040., 1664., 1688., ..., 1624., 1680., 1008.], [1792., 2064., 2048., ..., 2048., 2048., 1712.], [1768., 2064., 2064., ..., 2048., 2048., 1640.], ..., [1696., 2048., 2064., ..., 2048., 2048., 1560.], [1688., 2048., 2048., ..., 2048., 2048., 1616.], [1048., 1632., 1680., ..., 1744., 1648., 1040.]], ..., [[1004., 1736., 1640., ..., 1736., 1680., 1020.], [1688., 2048., 2048., ..., 2064., 2048., 1648.], [1664., 2064., 2064., ..., 2064., 2064., 1680.], ..., [1664., 2080., 2064., ..., 2048., 2048., 1624.], [1680., 2064., 2064., ..., 2080., 2064., 1648.], [1032., 1656., 1680., ..., 1688., 1688., 1024.]], [[1016., 1704., 1720., ..., 1728., 1736., 1020.], [1720., 2064., 2048., ..., 2080., 2048., 1768.], [1680., 2048., 2048., ..., 2064., 2048., 1712.], ..., [1720., 2080., 2048., ..., 2048., 2048., 1656.], [1696., 2048., 2064., ..., 2048., 2048., 1760.], [1040., 1600., 1760., ..., 1696., 1680., 1032.]], [[1024., 1720., 1664., ..., 1736., 1752., 1020.], [1760., 2048., 2064., ..., 2064., 2048., 1752.], [1704., 2048., 2064., ..., 2048., 2048., 1728.], ..., [1744., 2048., 2048., ..., 2048., 2048., 1632.], [1736., 2048., 2048., ..., 2048., 2048., 1680.], [1056., 1712., 1776., ..., 1744., 1664., 1048.]]]])

test_conv2d.test_module[Conv2D779-[((1, 1024, 14, 14), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[1032.66980, 1538.79065, 1535.18372, ..., 1541.14697, 1551.01111, 1556.10571], [1551.49036, 2305.59302, 2309.61987, ..., 2317.84082, 2322.83081, 2328.24536], [1548.31824, 2306.34766, 2336.31763, ..., 2318.40430, 2323.38037, 2297.93921], ..., [1544.38538, 2299.48145, 2300.33203, ..., 2322.77612, 2312.70874, 2329.04370], [1548.81311, 2341.73633, 2341.40015, ..., 2309.07495, 2333.21509, 2311.25952], [1530.58276, 2350.71558, 2332.76074, ..., 2306.47681, 2324.74854, 2287.75757]], [[1015.15833, 1531.82385, 1530.54443, ..., 1515.92065, 1542.07727, 1537.51099], [1532.35522, 2290.64819, 2299.16992, ..., 2320.37231, 2291.07080, 2306.00244], [1526.49939, 2312.46167, 2285.67773, ..., 2317.84570, 2310.94727, 2310.03516], ..., [1517.89868, 2283.85669, 2297.06836, ..., 2282.52295, 2300.79590, 2286.55737], [1532.98828, 2299.83960, 2322.75342, ..., 2284.23975, 2287.97949, 2293.30420], [1509.68469, 2336.54468, 2329.48486, ..., 2286.61597, 2295.73584, 2292.75903]], [[1026.16357, 1555.12537, 1535.68213, ..., 1524.19373, 1555.39612, 1560.93848], [1537.44202, 2292.87939, 2298.07007, ..., 2315.13867, 2318.24927, 2308.05396], [1538.09497, 2308.85034, 2322.74585, ..., 2330.25952, 2301.15894, 2314.00537], ..., [1529.78796, 2297.15283, 2320.08228, ..., 2309.31592, 2321.25464, 2312.81323], [1541.44922, 2324.05078, 2335.94702, ..., 2291.62402, 2320.41382, 2307.85620], [1531.13782, 2341.93628, 2334.08228, ..., 2306.62402, 2313.38501, 2290.02173]], ..., [[1032.63098, 1553.20166, 1555.25830, ..., 1542.28540, 1557.03955, 1556.75378], [1560.78149, 2309.45215, 2305.07544, ..., 2342.26733, 2328.94165, 2320.83350], [1556.49707, 2329.56982, 2327.56299, ..., 2319.07227, 2321.49048, 2305.61743], ..., [1544.86816, 2313.21094, 2331.13550, ..., 2334.57715, 2332.44067, 2325.54443], [1558.03662, 2333.27466, 2361.74121, ..., 2319.58521, 2331.77515, 2335.18359], [1552.21057, 2390.81421, 2352.07178, ..., 2334.74414, 2324.97144, 2304.75635]], [[1029.24597, 1540.52136, 1538.46521, ..., 1539.96350, 1571.55505, 1560.85168], [1528.95496, 2293.32910, 2316.27148, ..., 2305.65918, 2307.34033, 2308.86523], [1526.07483, 2298.67212, 2312.58521, ..., 2320.78003, 2309.72949, 2288.89429], ..., [1523.65125, 2297.52832, 2305.32495, ..., 2312.02026, 2309.93140, 2308.21973], [1532.14771, 2322.31079, 2341.26514, ..., 2314.84717, 2301.72046, 2297.82129], [1533.29358, 2348.15186, 2329.66406, ..., 2300.82007, 2298.84277, 2290.83716]], [[1024.36060, 1533.07349, 1539.29736, ..., 1528.17676, 1550.98193, 1536.42700], [1537.14905, 2281.48804, 2289.52490, ..., 2296.05981, 2307.29907, 2305.12061], [1535.45667, 2324.37646, 2300.60645, ..., 2323.09912, 2322.73779, 2292.19116], ..., [1539.23071, 2297.84131, 2305.92773, ..., 2291.30103, 2308.07837, 2307.83716], [1541.68164, 2310.37476, 2312.41431, ..., 2302.72998, 2316.89429, 2306.16846], [1524.36230, 2342.96460, 2335.04858, ..., 2295.42236, 2300.78369, 2274.00635]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[1072., 1736., 1704., ..., 1744., 1776., 1760.], [1768., 2064., 2048., ..., 2064., 2048., 2080.], [1688., 2048., 2048., ..., 2048., 2048., 2048.], ..., [1656., 2064., 2048., ..., 2048., 2048., 2048.], [1728., 2048., 2048., ..., 2048., 2048., 2048.], [1712., 2064., 2064., ..., 2064., 2048., 2064.]], [[1016., 1720., 1680., ..., 1648., 1736., 1696.], [1672., 2048., 2064., ..., 2048., 2064., 2048.], [1656., 2048., 2064., ..., 2048., 2064., 2048.], ..., [1616., 2048., 2048., ..., 2048., 2048., 2048.], [1736., 2048., 2048., ..., 2048., 2048., 2048.], [1656., 2048., 2064., ..., 2048., 2048., 2064.]], [[1020., 1752., 1784., ..., 1680., 1648., 1744.], [1672., 2048., 2048., ..., 2064., 2048., 2064.], [1752., 2048., 2064., ..., 2064., 2048., 2048.], ..., [1728., 2048., 2048., ..., 2080., 2048., 2048.], [1696., 2048., 2048., ..., 2048., 2064., 2048.], [1736., 2064., 2064., ..., 2048., 2048., 2048.]], ..., [[1048., 1824., 1752., ..., 1712., 1800., 1760.], [1776., 2064., 2048., ..., 2048., 2080., 2048.], [1728., 2048., 2048., ..., 2080., 2048., 2048.], ..., [1696., 2080., 2048., ..., 2048., 2048., 2064.], [1752., 2048., 2048., ..., 2064., 2048., 2048.], [1784., 2080., 2064., ..., 2048., 2064., 2048.]], [[1032., 1752., 1768., ..., 1736., 1832., 1736.], [1704., 2048., 2048., ..., 2048., 2064., 2048.], [1728., 2048., 2048., ..., 2064., 2048., 2048.], ..., [1704., 2048., 2048., ..., 2080., 2048., 2048.], [1720., 2048., 2048., ..., 2064., 2064., 2064.], [1704., 2048., 2064., ..., 2048., 2048., 2048.]], [[1024., 1752., 1744., ..., 1688., 1736., 1720.], [1704., 2064., 2048., ..., 2080., 2064., 2064.], [1672., 2048., 2048., ..., 2048., 2048., 2064.], ..., [1696., 2064., 2096., ..., 2048., 2048., 2064.], [1744., 2048., 2048., ..., 2048., 2048., 2048.], [1680., 2064., 2048., ..., 2064., 2048., 2048.]]]])

test_conv2d.test_module[Conv2D787-[((1, 728, 19, 19), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[177.60074, 187.23131, 184.12657, ..., 177.75125, 178.30333, 189.66875], [178.94287, 184.96721, 186.08879, ..., 185.53891, 185.94061, 182.04984], [178.81113, 171.86198, 178.33531, ..., 176.81029, 181.12726, 181.24512], ..., [182.36890, 189.51132, 182.98848, ..., 174.50813, 173.99088, 186.05676], [184.71120, 177.74013, 181.36017, ..., 177.13654, 183.39653, 178.87991], [183.06871, 178.42326, 183.33731, ..., 178.77455, 184.53712, 181.80450]], [[182.78302, 186.28175, 192.06351, ..., 180.57166, 187.96230, 190.56859], [187.06656, 190.67490, 186.50168, ..., 187.70818, 193.49463, 184.76401], [181.29886, 178.40152, 186.04370, ..., 178.49278, 189.15472, 186.69055], ..., [186.91922, 194.69289, 191.42255, ..., 182.27271, 181.20895, 186.32553], [188.69043, 182.50400, 188.97806, ..., 178.64743, 186.93599, 180.47836], [185.62442, 181.53914, 188.74883, ..., 180.66133, 189.21034, 190.75529]], [[175.89668, 182.44504, 184.32524, ..., 170.49287, 178.74309, 183.71535], [178.07391, 186.50621, 180.71655, ..., 184.24895, 183.42889, 179.34773], [172.74347, 174.26768, 182.77533, ..., 173.03123, 180.69220, 179.60310], ..., [178.55356, 191.11179, 187.89207, ..., 177.33929, 175.83835, 177.89877], [180.46855, 178.69266, 186.89142, ..., 171.65126, 181.45518, 174.18878], [178.56194, 178.71817, 186.61710, ..., 178.30659, 183.13045, 179.35252]], ..., [[185.98026, 191.21829, 186.08830, ..., 188.88034, 187.42255, 190.72255], [185.44096, 196.49188, 190.44072, ..., 187.91771, 194.68353, 188.47998], [184.09795, 185.42012, 182.95766, ..., 187.75665, 192.03430, 187.72252], ..., [182.53879, 196.65382, 192.31709, ..., 185.10901, 182.40674, 186.20015], [189.31592, 185.31419, 188.65828, ..., 177.74429, 190.85481, 183.28246], [190.40521, 181.36569, 192.53941, ..., 185.83163, 192.32050, 187.68811]], [[182.00647, 181.58612, 184.54247, ..., 178.33797, 180.15221, 183.51372], [181.09996, 184.75380, 183.38240, ..., 182.96393, 184.31293, 183.22841], [175.10002, 173.87961, 183.05182, ..., 175.22812, 185.66055, 184.65683], ..., [183.08148, 187.27930, 183.10985, ..., 179.96078, 177.13034, 182.19427], [181.19403, 179.58701, 188.43855, ..., 173.14954, 183.53931, 175.42453], [182.65717, 175.34143, 188.88716, ..., 178.42424, 182.97368, 182.41037]], [[180.69968, 180.41647, 179.22729, ..., 177.22336, 175.04391, 180.75638], [176.03615, 187.05571, 180.16605, ..., 180.55508, 182.05219, 179.49767], [178.69624, 176.29532, 173.29958, ..., 174.85132, 178.65297, 179.08916], ..., [178.38901, 188.37566, 183.61534, ..., 180.64223, 177.99036, 174.92160], [181.02542, 174.45651, 187.64934, ..., 176.08376, 179.55339, 175.04996], [176.92307, 171.67371, 182.00026, ..., 178.10059, 187.05176, 179.38594]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[175., 183., 184., ..., 174., 177., 188.], [175., 182., 185., ..., 181., 184., 178.], [173., 171., 175., ..., 178., 178., 178.], ..., [180., 186., 181., ..., 173., 171., 182.], [182., 177., 179., ..., 176., 179., 177.], [181., 176., 181., ..., 179., 180., 179.]], [[180., 184., 189., ..., 178., 186., 188.], [184., 189., 184., ..., 186., 191., 181.], [180., 177., 184., ..., 177., 187., 185.], ..., [184., 192., 187., ..., 181., 180., 185.], [188., 179., 187., ..., 176., 184., 179.], [184., 179., 187., ..., 177., 186., 188.]], [[176., 177., 180., ..., 167., 176., 183.], [177., 182., 179., ..., 184., 182., 175.], [171., 174., 182., ..., 169., 177., 179.], ..., [178., 187., 187., ..., 175., 173., 173.], [177., 178., 183., ..., 169., 178., 173.], [175., 178., 185., ..., 177., 180., 176.]], ..., [[183., 192., 183., ..., 187., 185., 187.], [183., 194., 186., ..., 186., 191., 185.], [180., 182., 180., ..., 184., 187., 186.], ..., [182., 191., 189., ..., 181., 179., 185.], [188., 183., 186., ..., 175., 189., 179.], [188., 178., 189., ..., 183., 190., 185.]], [[181., 180., 183., ..., 174., 177., 182.], [177., 182., 181., ..., 180., 181., 180.], [174., 172., 181., ..., 173., 183., 179.], ..., [182., 184., 179., ..., 179., 174., 179.], [178., 175., 187., ..., 169., 180., 172.], [180., 170., 185., ..., 176., 179., 178.]], [[178., 177., 177., ..., 175., 173., 179.], [174., 187., 176., ..., 179., 177., 177.], [177., 172., 170., ..., 171., 176., 178.], ..., [175., 187., 181., ..., 178., 177., 174.], [180., 171., 185., ..., 173., 178., 170.], [173., 170., 179., ..., 174., 182., 179.]]]])

test_conv2d.test_module[Conv2D831-[((1, 320, 80, 80), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[333.50986, 497.81186, 479.87344, ..., 493.99860, 491.04401, 472.69583], [486.65344, 724.52570, 712.26727, ..., 733.45154, 733.95581, 711.10992], [501.20642, 717.88751, 731.44110, ..., 726.65729, 730.56934, 719.31531], ..., [483.02588, 724.19110, 733.08136, ..., 739.07385, 720.43494, 711.99652], [493.88528, 737.57812, 736.58191, ..., 719.79199, 721.53540, 720.91516], [486.84531, 728.93280, 735.62457, ..., 729.76514, 720.50690, 716.21857]], [[335.19482, 496.40094, 481.78943, ..., 495.97427, 496.60641, 478.11218], [491.25797, 725.42560, 718.34186, ..., 735.34467, 739.54846, 721.60950], [493.94748, 719.93616, 718.67859, ..., 729.08203, 728.98627, 727.07129], ..., [481.28583, 727.51575, 723.43103, ..., 730.80365, 717.67535, 711.77606], [493.26144, 742.63422, 725.72424, ..., 730.66754, 730.59106, 722.03992], [484.54739, 729.90472, 736.30792, ..., 738.63641, 714.50568, 723.39215]], [[332.32150, 494.19730, 468.84015, ..., 486.24854, 484.31049, 461.85583], [492.24582, 725.35712, 705.04211, ..., 738.54333, 732.28314, 715.52338], [489.84308, 720.36127, 716.12231, ..., 721.33691, 721.49286, 716.15204], ..., [482.62888, 715.89801, 728.41168, ..., 726.31158, 708.59125, 709.02167], [483.09814, 734.24780, 731.38562, ..., 718.71100, 724.02747, 727.80670], [479.64746, 715.51392, 735.75214, ..., 730.93445, 720.22705, 712.24963]], ..., [[326.85980, 499.83499, 480.06073, ..., 492.39597, 491.04510, 474.64441], [483.91000, 725.31494, 716.32050, ..., 737.08453, 742.92554, 728.95111], [487.74448, 717.81860, 720.91705, ..., 722.30353, 720.86011, 725.29675], ..., [478.71826, 725.24762, 727.74646, ..., 736.63239, 718.69611, 708.62689], [484.72253, 736.09277, 733.79541, ..., 731.35431, 731.36780, 721.99738], [476.59137, 728.70624, 732.45642, ..., 738.41821, 717.97351, 715.52209]], [[325.68829, 491.63861, 480.54718, ..., 489.95239, 490.64078, 463.88037], [482.37381, 714.26636, 700.11218, ..., 731.70984, 724.45215, 706.39911], [478.93881, 703.22809, 729.49695, ..., 718.68921, 709.32629, 711.60132], ..., [475.73904, 712.14496, 712.49255, ..., 726.15485, 702.03235, 704.10010], [483.02109, 721.05762, 718.46796, ..., 716.19202, 720.20544, 713.87787], [473.20911, 718.94287, 724.58356, ..., 718.79181, 704.41730, 711.59839]], [[331.08655, 497.62561, 469.49258, ..., 490.22079, 482.13058, 467.62717], [482.60925, 720.35358, 693.60651, ..., 732.25610, 722.62683, 716.15454], [491.93643, 712.54108, 712.44287, ..., 718.60004, 719.85577, 717.61725], ..., [477.44159, 707.03894, 721.29425, ..., 729.64502, 704.38593, 701.12781], [486.26315, 729.63922, 718.46448, ..., 719.50879, 709.77759, 708.15826], [479.31177, 711.67401, 728.45160, ..., 726.01849, 712.66547, 703.22260]]]]), compiled_model=tensor([[[[328., 498., 480., ..., 494., 488., 472.], [486., 736., 716., ..., 740., 736., 708.], [492., 736., 720., ..., 728., 732., 728.], ..., [484., 724., 732., ..., 724., 716., 720.], [480., 728., 736., ..., 716., 720., 724.], [488., 728., 728., ..., 728., 724., 724.]], [[338., 492., 476., ..., 494., 506., 472.], [486., 744., 728., ..., 736., 740., 728.], [486., 740., 736., ..., 736., 728., 736.], ..., [482., 732., 736., ..., 740., 720., 736.], [480., 740., 724., ..., 744., 752., 724.], [474., 732., 740., ..., 748., 720., 724.]], [[328., 490., 468., ..., 482., 482., 466.], [492., 720., 708., ..., 732., 732., 732.], [486., 732., 724., ..., 720., 716., 720.], ..., [476., 708., 732., ..., 736., 704., 720.], [486., 744., 728., ..., 724., 712., 716.], [476., 720., 736., ..., 732., 720., 724.]], ..., [[324., 502., 478., ..., 490., 486., 466.], [482., 720., 712., ..., 740., 732., 712.], [498., 724., 708., ..., 712., 716., 732.], ..., [468., 720., 712., ..., 736., 712., 700.], [476., 728., 732., ..., 744., 716., 724.], [482., 732., 732., ..., 728., 740., 720.]], [[322., 486., 476., ..., 484., 486., 456.], [488., 716., 716., ..., 736., 724., 712.], [476., 712., 724., ..., 724., 700., 724.], ..., [480., 724., 720., ..., 728., 704., 724.], [486., 720., 720., ..., 728., 728., 716.], [474., 724., 736., ..., 724., 724., 724.]], [[332., 498., 462., ..., 492., 488., 468.], [478., 724., 704., ..., 740., 720., 720.], [484., 724., 724., ..., 728., 724., 732.], ..., [478., 716., 724., ..., 712., 712., 724.], [482., 732., 744., ..., 728., 716., 716.], [476., 728., 740., ..., 744., 720., 708.]]]])

test_conv2d.test_module[Conv2D836-[((1, 640, 30, 30), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 645.82019, 967.15887, 946.30927, ..., 938.78455, 968.82623, 972.39215], [ 951.30737, 1449.92725, 1447.50830, ..., 1440.53418, 1452.34998, 1428.63843], [ 955.41748, 1420.10217, 1449.49438, ..., 1452.38367, 1443.79834, 1434.04614], ..., [ 963.79895, 1468.28369, 1456.62085, ..., 1458.95654, 1433.84558, 1429.99609], [ 956.27161, 1458.50281, 1452.71838, ..., 1463.58545, 1445.42468, 1425.03748], [ 951.15674, 1432.52795, 1450.09973, ..., 1445.59778, 1437.08765, 1427.36389]], [[ 643.96814, 962.30151, 945.25085, ..., 937.79712, 969.42639, 955.59161], [ 970.27081, 1448.46960, 1425.17432, ..., 1441.71289, 1436.26135, 1425.28223], [ 963.89435, 1428.78564, 1422.21985, ..., 1460.43591, 1430.13098, 1430.80457], ..., [ 966.24786, 1470.11731, 1451.02881, ..., 1456.04541, 1428.91162, 1434.20569], [ 964.56696, 1464.26318, 1448.56982, ..., 1444.19397, 1447.54932, 1433.19104], [ 945.45251, 1434.88062, 1447.58118, ..., 1438.15234, 1436.99902, 1416.01367]], [[ 654.45697, 965.71240, 950.60187, ..., 946.91400, 968.60120, 958.91345], [ 975.74921, 1442.43152, 1421.67957, ..., 1423.12598, 1427.16443, 1418.09766], [ 972.76050, 1421.39587, 1422.41541, ..., 1452.25513, 1433.09473, 1413.94202], ..., [ 985.89288, 1465.39832, 1441.11646, ..., 1442.65466, 1433.32202, 1438.89661], [ 977.50958, 1458.38770, 1432.45605, ..., 1436.54028, 1439.66809, 1412.18030], [ 957.96069, 1413.84229, 1442.26660, ..., 1434.03821, 1427.01575, 1414.03394]], ..., [[ 645.34631, 972.58081, 956.36951, ..., 951.11176, 970.74213, 974.14917], [ 968.63428, 1457.78101, 1436.08862, ..., 1458.79968, 1459.76550, 1428.70935], [ 967.82355, 1429.92371, 1438.26013, ..., 1469.47644, 1452.25354, 1445.28406], ..., [ 965.58221, 1488.20837, 1455.54663, ..., 1467.63391, 1428.97668, 1440.63281], [ 964.96124, 1463.93909, 1463.13708, ..., 1460.21143, 1449.34802, 1438.78516], [ 951.45660, 1438.37195, 1453.85815, ..., 1442.18860, 1440.54211, 1420.34082]], [[ 654.10516, 967.22778, 959.63306, ..., 950.73004, 971.48627, 962.70917], [ 964.17035, 1449.48096, 1448.88965, ..., 1448.66052, 1475.07849, 1440.01221], [ 971.37622, 1433.03638, 1448.20068, ..., 1489.00220, 1450.99353, 1453.19751], ..., [ 973.05591, 1485.06555, 1466.09302, ..., 1469.14380, 1454.63184, 1447.42358], [ 967.53979, 1484.11475, 1440.03223, ..., 1467.61621, 1466.31970, 1458.57800], [ 956.25806, 1439.82410, 1457.74658, ..., 1450.72034, 1450.39014, 1445.83069]], [[ 660.25519, 967.35779, 954.18909, ..., 957.30725, 972.13635, 965.57104], [ 983.07117, 1446.68518, 1436.78247, ..., 1435.54150, 1457.84534, 1417.51233], [ 971.93262, 1422.86694, 1433.21460, ..., 1466.57483, 1439.05273, 1436.95642], ..., [ 975.32513, 1473.20813, 1446.60376, ..., 1448.47449, 1435.98401, 1431.08386], [ 967.30383, 1457.01892, 1430.56946, ..., 1455.18884, 1452.31372, 1420.64136], [ 965.98370, 1422.03748, 1456.28674, ..., 1449.26538, 1426.11523, 1434.94714]]]]), compiled_model=tensor([[[[ 648., 976., 964., ..., 940., 972., 968.], [ 960., 1664., 1624., ..., 1608., 1616., 1632.], [ 956., 1552., 1608., ..., 1608., 1648., 1560.], ..., [ 972., 1672., 1648., ..., 1640., 1616., 1600.], [ 968., 1608., 1656., ..., 1600., 1640., 1584.], [ 960., 1576., 1616., ..., 1640., 1584., 1560.]], [[ 640., 968., 960., ..., 952., 964., 972.], [ 980., 1584., 1648., ..., 1600., 1616., 1624.], [ 964., 1576., 1552., ..., 1640., 1584., 1584.], ..., [ 964., 1632., 1600., ..., 1608., 1576., 1592.], [ 976., 1632., 1656., ..., 1600., 1560., 1512.], [ 952., 1592., 1608., ..., 1536., 1592., 1528.]], [[ 660., 980., 952., ..., 948., 960., 964.], [ 988., 1592., 1624., ..., 1552., 1560., 1568.], [ 972., 1504., 1568., ..., 1568., 1544., 1528.], ..., [ 988., 1624., 1560., ..., 1600., 1528., 1600.], [ 976., 1592., 1576., ..., 1592., 1576., 1520.], [ 972., 1568., 1592., ..., 1576., 1536., 1536.]], ..., [[ 644., 972., 972., ..., 972., 964., 968.], [ 972., 1608., 1584., ..., 1600., 1632., 1592.], [ 968., 1584., 1600., ..., 1568., 1576., 1512.], ..., [ 964., 1680., 1624., ..., 1616., 1584., 1552.], [ 968., 1624., 1608., ..., 1592., 1560., 1568.], [ 960., 1560., 1536., ..., 1600., 1544., 1568.]], [[ 644., 980., 972., ..., 968., 972., 968.], [ 980., 1648., 1672., ..., 1592., 1616., 1576.], [ 996., 1560., 1672., ..., 1656., 1600., 1568.], ..., [ 976., 1640., 1640., ..., 1640., 1560., 1584.], [ 992., 1656., 1624., ..., 1560., 1696., 1576.], [ 968., 1616., 1608., ..., 1528., 1584., 1512.]], [[ 660., 972., 972., ..., 960., 984., 972.], [ 992., 1560., 1600., ..., 1496., 1608., 1560.], [ 980., 1536., 1616., ..., 1584., 1600., 1584.], ..., [ 996., 1608., 1560., ..., 1560., 1576., 1592.], [ 984., 1608., 1536., ..., 1592., 1624., 1488.], [ 984., 1552., 1608., ..., 1664., 1528., 1552.]]]])

test_conv2d.test_module[Conv2D796-[((1, 3, 640, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 73859072 B L1 buffer across 64 banks, where each bank needs to store 1154048 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f6fbc109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f6fbc109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464]

test_conv2d.test_module[Conv2D881-[((1, 3, 640, 640), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 73859072 B L1 buffer across 64 banks, where each bank needs to store 1154048 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f6fbc109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f6fbc109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464]

test_conv2d.test_module[Conv2D874-[((1, 3, 480, 480), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/program/program.cpp:905: tt::exception info: Statically allocated circular buffers in program 5346 clash with L1 buffers on core range [(x=0,y=0) - (x=7,y=6)]. L1 buffer allocated at 197632 and static circular buffer region ends at 570656 backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x1712d5) [0x7f6fcca092d5] --- tt::tt_metal::detail::Program_::validate_circular_buffer_region(tt::tt_metal::v0::IDevice const*) --- tt::tt_metal::v0::EnqueueProgram(tt::tt_metal::CommandQueue&, tt::tt_metal::v0::Program&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_worker_threadIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEENS3_3stl10StrongTypeIhNS_10QueueIdTagEEElNS5_15DeviceOperationISA_EENSB_13tensor_args_tESA_PNS4_2v07IDeviceEEEvT0_T1_RKT2_RKT3_RT4_RT5_+0x336) [0x7f6fbd6ca7b6] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x83) [0x7f6fbd6ca3e3] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab]

test_conv2d.test_module[Conv2D919-[((1, 512, 28, 40), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[ 516.68201, 773.38586, 761.83429, ..., 759.49536, 773.54675, 779.12976], [ 780.07654, 1144.32800, 1129.50476, ..., 1167.15979, 1158.87634, 1142.59692], [ 774.38159, 1165.22388, 1150.34387, ..., 1159.40271, 1148.02991, 1153.51294], ..., [ 748.33319, 1129.95300, 1157.12622, ..., 1137.79321, 1163.20447, 1157.22546], [ 746.76770, 1162.69568, 1154.01160, ..., 1146.05457, 1149.15662, 1158.79248], [ 760.26697, 1157.76892, 1160.09290, ..., 1143.63794, 1145.47363, 1144.80872]], [[ 516.12415, 773.35767, 760.73120, ..., 771.32672, 771.81854, 774.60822], [ 779.16669, 1162.54089, 1139.50000, ..., 1167.38806, 1166.37842, 1156.53967], [ 784.30237, 1168.66833, 1167.70105, ..., 1171.15210, 1154.79187, 1165.43750], ..., [ 751.88477, 1151.08313, 1168.75879, ..., 1142.95642, 1168.43640, 1155.27734], [ 748.66077, 1169.00977, 1167.56116, ..., 1152.63794, 1169.14441, 1165.73389], [ 770.59259, 1166.85474, 1178.32507, ..., 1157.67102, 1148.80420, 1149.64331]], [[ 507.54160, 759.98938, 758.44867, ..., 760.91064, 771.43262, 776.12469], [ 764.10480, 1140.13147, 1119.26306, ..., 1169.59802, 1163.70544, 1133.42053], [ 778.69867, 1164.92126, 1152.72791, ..., 1156.26160, 1154.08801, 1144.09033], ..., [ 749.68835, 1140.49451, 1162.87085, ..., 1138.42395, 1156.13074, 1140.28613], [ 745.49384, 1153.96741, 1157.39514, ..., 1147.53540, 1154.86938, 1143.64490], [ 762.48077, 1160.91687, 1155.36902, ..., 1133.13757, 1145.24036, 1143.14392]], ..., [[ 515.94525, 769.43243, 765.26202, ..., 767.73114, 767.21753, 772.05804], [ 778.58282, 1146.10413, 1131.62183, ..., 1161.93372, 1156.71265, 1133.92493], [ 780.33905, 1153.76709, 1152.57776, ..., 1157.08850, 1154.29810, 1156.69690], ..., [ 749.43768, 1126.35547, 1160.89343, ..., 1131.84570, 1156.37573, 1159.65112], [ 754.12671, 1155.76025, 1149.55786, ..., 1138.40735, 1157.75073, 1149.71106], [ 771.83990, 1149.55444, 1160.38220, ..., 1152.64282, 1133.36829, 1143.53992]], [[ 502.41599, 746.58044, 744.28369, ..., 746.66809, 741.19049, 755.85449], [ 764.40491, 1136.36938, 1113.63745, ..., 1153.25549, 1149.76819, 1134.62073], [ 768.66351, 1146.06311, 1140.17786, ..., 1137.29529, 1154.75208, 1146.69995], ..., [ 739.25385, 1119.72412, 1154.10229, ..., 1131.44861, 1143.86292, 1154.76270], [ 745.29498, 1152.11877, 1141.96631, ..., 1136.73865, 1142.67615, 1145.11096], [ 756.75006, 1147.90344, 1142.88953, ..., 1129.20605, 1125.45203, 1126.24316]], [[ 499.45468, 759.82025, 746.43835, ..., 754.86163, 758.73407, 755.62323], [ 773.10638, 1125.32959, 1127.14050, ..., 1153.27039, 1154.33630, 1137.87830], [ 777.37006, 1156.97498, 1142.08020, ..., 1149.70654, 1144.72583, 1143.35107], ..., [ 743.23163, 1125.21960, 1147.76782, ..., 1125.28491, 1146.61572, 1144.74707], [ 743.15131, 1149.06689, 1142.68030, ..., 1133.49988, 1139.25940, 1143.93787], [ 754.59528, 1157.72656, 1156.51514, ..., 1135.79700, 1124.23926, 1143.30542]]]]), compiled_model=tensor([[[[ 510., 768., 772., ..., 768., 772., 784.], [ 772., 1176., 1200., ..., 1240., 1200., 1208.], [ 780., 1272., 1208., ..., 1200., 1200., 1240.], ..., [ 764., 1168., 1192., ..., 1184., 1224., 1216.], [ 756., 1232., 1200., ..., 1216., 1200., 1200.], [ 764., 1216., 1224., ..., 1168., 1216., 1224.]], [[ 512., 772., 760., ..., 776., 768., 780.], [ 784., 1216., 1200., ..., 1208., 1232., 1208.], [ 784., 1232., 1168., ..., 1184., 1200., 1240.], ..., [ 764., 1232., 1232., ..., 1200., 1224., 1192.], [ 756., 1224., 1256., ..., 1200., 1224., 1192.], [ 760., 1184., 1216., ..., 1192., 1168., 1200.]], [[ 500., 764., 768., ..., 768., 788., 796.], [ 772., 1224., 1192., ..., 1216., 1208., 1192.], [ 776., 1272., 1200., ..., 1256., 1240., 1224.], ..., [ 752., 1200., 1216., ..., 1232., 1232., 1208.], [ 744., 1208., 1208., ..., 1208., 1232., 1184.], [ 760., 1200., 1264., ..., 1184., 1208., 1192.]], ..., [[ 510., 764., 772., ..., 776., 764., 772.], [ 772., 1200., 1184., ..., 1240., 1224., 1224.], [ 784., 1240., 1216., ..., 1224., 1208., 1208.], ..., [ 752., 1216., 1216., ..., 1200., 1216., 1200.], [ 752., 1224., 1232., ..., 1216., 1232., 1168.], [ 756., 1184., 1240., ..., 1224., 1224., 1216.]], [[ 512., 760., 748., ..., 760., 748., 768.], [ 768., 1200., 1152., ..., 1192., 1224., 1192.], [ 768., 1208., 1200., ..., 1152., 1216., 1240.], ..., [ 756., 1192., 1216., ..., 1208., 1160., 1216.], [ 768., 1240., 1200., ..., 1208., 1192., 1176.], [ 756., 1192., 1232., ..., 1168., 1192., 1168.]], [[ 502., 760., 756., ..., 768., 764., 756.], [ 772., 1176., 1168., ..., 1192., 1168., 1192.], [ 776., 1216., 1184., ..., 1184., 1184., 1184.], ..., [ 744., 1160., 1168., ..., 1176., 1184., 1176.], [ 748., 1208., 1208., ..., 1160., 1208., 1136.], [ 752., 1184., 1232., ..., 1160., 1168., 1184.]]]])

test_conv2d.test_module[Conv2D142-[((1, 128, 160, 160), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[134.34167, 198.46693, 196.26483, ..., 200.91852, 199.89523, 196.32593], [190.45964, 295.75012, 294.81183, ..., 289.09531, 293.87515, 288.77664], [195.00226, 297.07971, 296.29562, ..., 295.30020, 286.16516, 289.16309], ..., [194.34030, 286.68494, 293.93121, ..., 292.40546, 291.09161, 287.70129], [190.55637, 295.20276, 282.92093, ..., 286.49518, 295.64227, 292.71637], [197.88046, 298.86743, 290.77853, ..., 290.74197, 285.32932, 294.40494]], [[132.45961, 193.42657, 192.54355, ..., 185.67468, 196.49040, 191.21689], [195.65146, 293.96182, 293.54526, ..., 294.87207, 294.67923, 284.00616], [197.81332, 294.76083, 293.81189, ..., 293.62851, 289.47873, 285.70105], ..., [193.08160, 289.42609, 283.73318, ..., 296.80988, 294.78430, 292.24133], [195.00391, 292.60278, 283.01855, ..., 295.52155, 292.41562, 289.30283], [198.06062, 299.27408, 294.34564, ..., 290.21317, 278.82907, 293.44070]], [[127.98491, 188.12244, 189.20549, ..., 182.36807, 187.98895, 178.58679], [186.37729, 277.58417, 284.91898, ..., 278.90146, 281.42325, 274.73190], [190.76079, 286.12869, 284.31158, ..., 278.75311, 280.90451, 272.37946], ..., [193.51917, 273.81989, 276.66309, ..., 291.06357, 280.61237, 280.34830], [189.98051, 279.80606, 269.54410, ..., 285.28717, 275.87469, 277.47665], [191.60898, 283.76260, 284.37915, ..., 280.76349, 270.87399, 277.66263]], ..., [[129.54510, 195.24763, 190.58354, ..., 189.63551, 204.29837, 192.72447], [187.33510, 294.16684, 289.32874, ..., 287.19327, 294.62317, 278.64597], [196.16687, 290.75940, 288.17334, ..., 288.16660, 282.38455, 280.82516], ..., [188.94258, 286.27478, 285.31161, ..., 291.29761, 290.03134, 288.39883], [189.21863, 282.82034, 274.11703, ..., 287.63495, 289.47202, 292.06168], [190.44389, 290.92245, 286.83865, ..., 287.74225, 280.67334, 283.94775]], [[130.75522, 188.39977, 193.75389, ..., 187.18665, 191.00574, 188.19865], [191.80002, 296.30276, 290.62601, ..., 288.21933, 290.45523, 277.56540], [195.79951, 294.07101, 295.33353, ..., 289.35309, 286.20831, 277.54318], ..., [199.49005, 288.18738, 287.13440, ..., 297.58539, 293.82074, 287.74118], [195.89203, 287.56290, 282.88916, ..., 291.81830, 291.28073, 284.64508], [193.36113, 291.27875, 288.85773, ..., 285.52087, 280.95804, 288.04922]], [[130.26961, 195.33058, 195.32520, ..., 186.64105, 192.87079, 188.72986], [187.35374, 293.86185, 289.49551, ..., 283.59204, 292.46024, 280.71280], [199.44214, 292.92618, 291.67596, ..., 291.78275, 286.60004, 280.02167], ..., [194.42250, 283.06058, 284.12195, ..., 299.41531, 289.67615, 288.13367], [193.10335, 287.39932, 279.95776, ..., 287.93594, 286.46075, 283.62567], [192.53767, 288.78064, 290.77646, ..., 289.32095, 278.31219, 294.04712]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[133.00000, 195.00000, 195.00000, ..., 198.00000, 198.00000, 195.00000], [188.00000, 290.00000, 292.00000, ..., 282.00000, 292.00000, 288.00000], [193.00000, 296.00000, 290.00000, ..., 292.00000, 284.00000, 286.00000], ..., [194.00000, 284.00000, 292.00000, ..., 288.00000, 286.00000, 284.00000], [188.00000, 290.00000, 280.00000, ..., 284.00000, 294.00000, 292.00000], [196.00000, 296.00000, 288.00000, ..., 290.00000, 280.00000, 290.00000]], [[130.00000, 189.00000, 190.00000, ..., 180.00000, 192.00000, 188.00000], [193.00000, 290.00000, 284.00000, ..., 292.00000, 292.00000, 280.00000], [194.00000, 288.00000, 292.00000, ..., 294.00000, 284.00000, 286.00000], ..., [191.00000, 286.00000, 274.00000, ..., 296.00000, 288.00000, 288.00000], [193.00000, 290.00000, 280.00000, ..., 290.00000, 288.00000, 286.00000], [195.00000, 294.00000, 288.00000, ..., 288.00000, 276.00000, 292.00000]], [[126.50000, 185.00000, 186.00000, ..., 179.00000, 185.00000, 175.00000], [183.00000, 276.00000, 280.00000, ..., 272.00000, 280.00000, 272.00000], [186.00000, 286.00000, 280.00000, ..., 276.00000, 278.00000, 272.00000], ..., [191.00000, 272.00000, 272.00000, ..., 292.00000, 280.00000, 278.00000], [188.00000, 278.00000, 264.00000, ..., 282.00000, 272.00000, 276.00000], [190.00000, 280.00000, 282.00000, ..., 280.00000, 268.00000, 272.00000]], ..., [[127.50000, 192.00000, 188.00000, ..., 185.00000, 200.00000, 189.00000], [183.00000, 290.00000, 288.00000, ..., 286.00000, 292.00000, 282.00000], [194.00000, 290.00000, 282.00000, ..., 280.00000, 278.00000, 280.00000], ..., [186.00000, 280.00000, 284.00000, ..., 288.00000, 284.00000, 286.00000], [186.00000, 276.00000, 272.00000, ..., 286.00000, 284.00000, 284.00000], [185.00000, 286.00000, 282.00000, ..., 286.00000, 278.00000, 280.00000]], [[129.00000, 184.00000, 188.00000, ..., 186.00000, 188.00000, 186.00000], [191.00000, 298.00000, 282.00000, ..., 286.00000, 288.00000, 274.00000], [192.00000, 294.00000, 288.00000, ..., 292.00000, 282.00000, 274.00000], ..., [198.00000, 288.00000, 282.00000, ..., 294.00000, 294.00000, 290.00000], [193.00000, 284.00000, 278.00000, ..., 288.00000, 286.00000, 278.00000], [189.00000, 292.00000, 292.00000, ..., 282.00000, 280.00000, 286.00000]], [[127.50000, 193.00000, 195.00000, ..., 184.00000, 189.00000, 188.00000], [186.00000, 292.00000, 288.00000, ..., 280.00000, 290.00000, 278.00000], [199.00000, 290.00000, 288.00000, ..., 288.00000, 280.00000, 278.00000], ..., [193.00000, 276.00000, 278.00000, ..., 296.00000, 286.00000, 284.00000], [190.00000, 286.00000, 280.00000, ..., 284.00000, 282.00000, 280.00000], [189.00000, 286.00000, 286.00000, ..., 290.00000, 280.00000, 292.00000]]]])

test_conv2d.test_module[Conv2D499-[((1, 80, 320, 320), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

RuntimeError: TT_THROW @ /__w/tt-forge-fe/tt-forge-fe/third_party/tt-mlir/third_party/tt-metal/src/tt-metal/tt_metal/impl/allocator/bank_manager.cpp:132: tt::exception info: Out of Memory: Not enough space to allocate 58305792 B L1 buffer across 62 banks, where each bank needs to store 940416 B backtrace: --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x15a7e5) [0x7f6fcc9f27e5] --- tt::tt_metal::BankManager::allocate_buffer(unsigned long, unsigned long, bool, CoreRangeSet const&, std::optional<unsigned int>) --- tt::tt_metal::Allocator::allocate_buffer(tt::tt_metal::v0::Buffer*) --- tt::tt_metal::v0::Buffer::allocate_impl() --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libtt_metal.so(+0x125e9f) [0x7f6fcc9bde9f] --- tt::tt_metal::v0::Buffer::create(tt::tt_metal::v0::IDevice*, unsigned long, unsigned long, tt::tt_metal::BufferType, tt::tt_metal::TensorMemoryLayout, std::optional<tt::tt_metal::ShardSpecBuffer> const&, std::optional<bool>, std::optional<tt::tt_metal::SubDeviceId>) --- tt::tt_metal::tensor_impl::allocate_buffer_on_device(tt::tt_metal::v0::IDevice*, tt::tt_metal::TensorSpec const&) --- tt::tt_metal::create_device_tensor(tt::tt_metal::TensorSpec const&, tt::tt_metal::v0::IDevice*) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN2tt8tt_metal9operation29default_create_output_tensorsIN4ttnn10operations14sliding_window4halo19HaloDeviceOperationEEENS1_21program_output_helperIT_Xsr18has_create_programIS9_EE5valueEE4typeERKS9_RKSt6vectorINS0_6TensorESaISF_EERKSE_ISt8optionalISF_ESaISL_EE+0x193) [0x7f6fbc109833] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xfff699) [0x7f6fbc109699] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn16device_operation6detail23launch_on_single_deviceIN2tt8tt_metal9operation23OldInfraDeviceOperationISt6vectorINS4_6TensorESaIS8_EEEEEENT_21tensor_return_value_tENS3_3stl10StrongTypeIhNS_10QueueIdTagEEERKNSC_22operation_attributes_tERKNSC_13tensor_args_tE+0x3a) [0x7f6fbd6ca39a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c02d7) [0x7f6fbd6ca2d7] --- tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_return_value_t ttnn::device_operation::detail::invoke<tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > > >(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::operation_attributes_t const&, tt::tt_metal::operation::OldInfraDeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >::tensor_args_t const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25bf49f) [0x7f6fbd6c949f] --- std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > tt::tt_metal::operation::run<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(tt::tt_metal::operation::DeviceOperation<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >&&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&, tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0xffe69a) [0x7f6fbc10869a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c6fe2) [0x7f6fbd6d0fe2] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x25c77cc) [0x7f6fbd6d17cc] --- void tt::tt_metal::operation::launch_op_func<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > >(std::function<std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > (std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> > const&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > > const&, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > > const&)> const&, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >, std::vector<tt::tt_metal::Tensor, std::allocator<tt::tt_metal::Tensor> >&, std::vector<std::optional<tt::tt_metal::Tensor const>, std::allocator<std::optional<tt::tt_metal::Tensor const> > >, std::vector<std::optional<tt::tt_metal::Tensor>, std::allocator<std::optional<tt::tt_metal::Tensor> > >, bool) --- ttnn::operations::sliding_window::halo::halo_op(tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- ttnn::operations::sliding_window::halo::HaloOperation::invoke(tt::stl::StrongType<unsigned char, ttnn::QueueIdTag>, tt::tt_metal::Tensor const&, ttnn::operations::sliding_window::SlidingWindowConfig const&, unsigned int, bool, bool, unsigned int, tt::tt_metal::MemoryConfig const&, bool) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(+0x436f8f) [0x7f6fbb540f8f] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/_ttnn.so(_ZN4ttnn10operations4conv6conv2d6conv2dIN2tt8tt_metal2v07IDeviceEEESt5tupleIJNS5_6TensorEjjS9_St8optionalIS9_EEERKS9_SE_PT_jjjjjSt5arrayIjLm2EESI_SI_SI_jSA_ISD_ERKSA_IKNS2_12Conv2dConfigEERKSA_IKSt7variantIJNS_28GrayskullComputeKernelConfigENS_27WormholeComputeKernelConfigEEEERKSA_IKNS5_12MemoryConfigEE+0xcaa) [0x7f6fbb4d9d5a] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/lib/libTTMLIRRuntime.so(+0xb8365) [0x7f6fcd5aa365] --- tt::runtime::ttnn::operations::conv::run(tt::target::ttnn::Conv2dOp const*, tt::runtime::ttnn::ProgramContext&) --- tt::runtime::ttnn::runProgram(tt::tt_metal::distributed::MeshDevice&, tt::runtime::Binary, unsigned int, std::vector<tt::tt_metal::Tensor*, std::allocator<tt::tt_metal::Tensor*> > const&) --- tt::runtime::ttnn::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::runtime::submit(tt::runtime::Device, tt::runtime::Binary, unsigned int, std::vector<tt::runtime::Tensor, std::allocator<tt::runtime::Tensor> > const&) --- tt::run_binary(tt::runtime::Binary&, int, std::vector<at::Tensor, std::allocator<at::Tensor> > const&) --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315c40) [0x7f6fcda19c40] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0x315b8e) [0x7f6fcda19b8e] --- /opt/ttforge-toolchain/venv/lib/python3.10/site-packages/forge/_C.so(+0xe0b75) [0x7f6fcd7e4b75] --- /opt/ttforge-toolchain/venv/bin/python(+0x18ab32) [0x5593bf8abb32] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x59c7) [0x5593bf89ba97] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0xc4) [0x5593bf8a1574] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0x122) [0x5593bf8ba172] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_MakeTpCall+0x25b) [0x5593bf8a239b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x68ce) [0x5593bf89c99e] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x8af) [0x5593bf89697f] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464] --- /opt/ttforge-toolchain/venv/bin/python(PyObject_Call+0xbb) [0x5593bf8ba10b] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x6c0) [0x5593bf896790] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x2b60) [0x5593bf898c30] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x18db) [0x5593bf8979ab] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(+0x1984d1) [0x5593bf8b94d1] --- /opt/ttforge-toolchain/venv/bin/python(_PyEval_EvalFrameDefault+0x5632) [0x5593bf89b702] --- /opt/ttforge-toolchain/venv/bin/python(_PyFunction_Vectorcall+0x7c) [0x5593bf8ac38c] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_FastCallDictTstate+0x16d) [0x5593bf8a161d] --- /opt/ttforge-toolchain/venv/bin/python(_PyObject_Call_Prepend+0x5c) [0x5593bf8b662c] --- /opt/ttforge-toolchain/venv/bin/python(+0x29d464) [0x5593bf9be464]

test_conv2d.test_module[Conv2D364-[((1, 160, 160, 160), torch.float32)]]: forge/test/models_ops/test_conv2d.py#L56786

ValueError: Data mismatch -> AutomaticValueChecker (compare_with_golden): framework_model=tensor([[[[164.97191, 246.00629, 251.49423, ..., 246.36850, 247.40131, 244.96144], [243.32503, 370.13287, 371.05411, ..., 359.13284, 367.55042, 361.93539], [243.72289, 370.45319, 370.87708, ..., 366.56650, 366.55826, 366.86148], ..., [242.85320, 366.89069, 361.47021, ..., 374.01221, 367.69299, 364.86578], [252.25479, 365.66266, 359.69232, ..., 362.45670, 363.49814, 363.39514], [245.84772, 372.35025, 371.16815, ..., 366.34769, 354.04013, 362.58105]], [[165.19231, 245.53845, 250.34000, ..., 241.15898, 239.90082, 235.78395], [234.65196, 377.60071, 372.80719, ..., 363.02533, 377.05072, 360.58521], [245.66490, 375.64752, 367.37494, ..., 368.82480, 372.75980, 364.30316], ..., [247.06624, 361.95459, 363.97412, ..., 365.09604, 371.82175, 364.53976], [241.88950, 366.79782, 362.02365, ..., 367.83752, 367.05539, 365.85571], [245.59776, 375.48669, 372.69501, ..., 367.04105, 360.72836, 365.68912]], [[158.71950, 231.19716, 233.34572, ..., 231.76572, 229.78419, 230.49443], [243.52187, 360.81461, 357.58594, ..., 352.00577, 361.36884, 349.02359], [246.70900, 362.02933, 359.05215, ..., 359.70023, 354.83435, 354.67221], ..., [245.95633, 355.49561, 353.70218, ..., 368.28033, 360.64633, 357.89532], [238.57867, 358.02469, 351.83578, ..., 362.05234, 358.89465, 353.30383], [242.68571, 361.18982, 357.89902, ..., 357.68982, 348.06555, 358.25006]], ..., [[155.57999, 239.06085, 241.79227, ..., 231.20160, 236.99817, 235.38646], [232.52252, 359.14758, 355.48981, ..., 354.93756, 358.69568, 345.83435], [243.34270, 362.15671, 366.92609, ..., 356.87189, 357.03787, 353.15091], ..., [239.52805, 355.91129, 350.61670, ..., 365.50000, 356.68024, 355.51624], [233.47255, 355.25137, 348.25620, ..., 360.20419, 351.88852, 356.30792], [241.11824, 365.05865, 359.20959, ..., 353.80655, 352.27881, 357.28513]], [[160.49316, 238.65149, 241.12003, ..., 234.50508, 243.36340, 239.40114], [239.49080, 361.64172, 362.24225, ..., 361.13553, 361.44275, 352.49557], [245.44649, 363.45523, 368.66077, ..., 360.42990, 355.10764, 364.20190], ..., [246.63019, 353.62619, 353.89755, ..., 368.49661, 363.83450, 363.05966], [243.96307, 355.13370, 351.82269, ..., 357.46286, 361.67221, 352.22617], [242.00684, 361.98099, 359.64499, ..., 360.72275, 348.07584, 358.85321]], [[156.95337, 250.87857, 251.66568, ..., 242.40036, 250.56354, 246.65146], [231.73668, 370.62381, 367.56537, ..., 369.37482, 370.22324, 352.86639], [241.63959, 371.37622, 375.99365, ..., 372.65988, 370.51834, 367.15659], ..., [241.66862, 360.05420, 375.00894, ..., 371.54477, 367.74295, 372.11462], [237.08331, 365.80005, 360.17429, ..., 370.94556, 370.58203, 358.68805], [236.82416, 371.77603, 365.77307, ..., 367.46701, 360.20273, 365.33075]]]], grad_fn=<ConvolutionBackward0>), compiled_model=tensor([[[[165., 244., 249., ..., 246., 242., 242.], [239., 366., 374., ..., 356., 364., 360.], [239., 368., 368., ..., 368., 368., 364.], ..., [239., 364., 360., ..., 372., 362., 364.], [249., 364., 354., ..., 360., 364., 360.], [239., 368., 368., ..., 364., 352., 358.]], [[165., 243., 249., ..., 240., 235., 233.], [229., 386., 376., ..., 370., 372., 360.], [245., 370., 362., ..., 366., 372., 360.], ..., [246., 358., 366., ..., 366., 366., 364.], [239., 370., 362., ..., 368., 368., 370.], [241., 374., 374., ..., 362., 358., 364.]], [[155., 226., 229., ..., 230., 229., 227.], [243., 362., 358., ..., 354., 362., 346.], [243., 358., 354., ..., 360., 348., 354.], ..., [241., 354., 354., ..., 368., 358., 356.], [235., 354., 346., ..., 364., 354., 352.], [240., 358., 354., ..., 354., 344., 354.]], ..., [[154., 236., 239., ..., 228., 232., 233.], [230., 356., 356., ..., 352., 356., 344.], [242., 354., 362., ..., 354., 356., 356.], ..., [237., 354., 352., ..., 364., 352., 354.], [232., 352., 344., ..., 360., 350., 354.], [236., 366., 356., ..., 354., 346., 354.]], [[157., 235., 238., ..., 233., 239., 236.], [237., 362., 358., ..., 354., 354., 356.], [241., 360., 372., ..., 360., 356., 362.], ..., [243., 352., 352., ..., 370., 358., 362.], [242., 354., 354., ..., 360., 362., 352.], [238., 360., 356., ..., 360., 350., 352.]], [[154., 249., 247., ..., 240., 249., 242.], [228., 372., 360., ..., 362., 362., 352.], [238., 372., 372., ..., 374., 366., 364.], ..., [238., 352., 374., ..., 378., 362., 374.], [233., 368., 358., ..., 370., 366., 360.], [236., 372., 356., ..., 364., 362., 362.]]]])

pre-commit / pre-commit

The `python-version` input is not set. The version of Python currently in `PATH` will be used.

Artifacts

Produced during runtime

Name	Size
forge-wheel	257 MB
test-log-n150-1	15.3 KB
test-log-n150-2	11.9 KB
test-log-n300-1	11.3 KB
test-log-n300-2	11.9 KB
test-reports-n150-1	17.5 KB
test-reports-n150-2	14.4 KB
test-reports-n300-1	13.3 KB
test-reports-n300-2	14.3 KB

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate and update models ops tests and reimplement the models ops test failure updation script #2744

Summary

Generate and update models ops tests and reimplement the models ops test failure updation script #2744

Jobs

Run details

on-pr.yml

Annotations

Artifacts