random result when inference 2 similar textual data

[tideng777]

Hi,

I fine-tuned this model and I met a curious issue when inference 2 similar textual data, for example like below:

[["PROTEXASE THERAPEUTICS", "PROTEXASE THERAPEUTICS, INC."]]

i tested it for 10 time which show 10 different prediction score:
[[0.6435145641728891, 0.35648543582711095],
[0.44686401791372865, 0.5531359820862713],
[0.4768868842385305, 0.5231131157614696],
[0.5165862730694053, 0.48341372693059464],
[0.45327667255908927, 0.5467233274409107],
[0.5023805971114581, 0.497619402888542],
[0.7547820757051401, 0.24521792429486],
[0.02058591123972741, 0.9794140887602728],
[0.7257732298308167, 0.27422677016918334],
[0.30100721825239946, 0.6989927817476006]]

and this result showed quiet unstable.

i check the from snippext.model import MultiTaskNet, and the dropout=0.1 which modify to 0. The random result also happen. In fact, the dropout layer does not work in inference processing.

i think this issue is unbelievable. I might miss something importance. i don't know what is the problem?

[tideng777]

and I check the tensor after embedding which are all same. Something happen in evaluation.

[aquibjaved]

I was also facing the same issue, so, once you've loaded the model. put the model to model.eval() before passing it to classify func and the result will be deterministic :)

[psungu]

Even open the eval mode and fixed random seed, it gives different inferences on single and batch prediction for the same textual pair.

Example:

COL title VAL  "Nike Metcon 2 - Black/White/Wolf Grey/Volt"@en " Nike Mens Shoes Regular Training Grey/Volt "@en	COL title VAL  "Nike Metcon DSX Flyknit - Wolf Grey/Volt/Wolf Grey/Black"@en-GB " Nike Mens Shoes Regular Training Grey/Black "@en-GB	0
COL title VAL  "Nike Zoom Structure 19 Comp Pack Men's Running Shoes SP16 007"@en 007 · Nike Moderate Support Trai ah"@en	COL title VAL  "KANMEI GS"@en-es GS | Unisex Kids Running Shoes ASICS"@en-es	0
COL title VAL  "Air Jordan 5 Low “Dunk From Above” White/Metallic Gold Star-Midnight Navy For Sale"@en-US Sale | New Jordans 2016"@en-US	COL title VAL  "Air Jordan 5 Low “Dunk From Above” White-Gold/Midnight Navy 2016 Sale"@en-US Sale | Cheap Jordans 2017"@en-US	1
COL title VAL  "Nike Womens Air Zoom Pegasus 33 - Black/White-Anthracite-Cool Grey"@en-GB " Nike Grey Shoes 831356-001 "@en-GB	COL title VAL  "NIKE AIR ZOOM PEGASUS 33"@en 33 - MAN RUNNING SHOES Nike colour Black 831352 001 Athletic footwear apparel and sports equipment"@en	0
COL title VAL  "Air Jordan 14 Retro Low “Laney” Varsity Royal/Varsity Maize-Black-White For Sale"@en-US Sale | Cheap Jordans 2017"@en-US	COL title VAL  "Cheap Air Jordan 4 Retro “Motorsports” White/Varsity Blue-Black Sale"@en-US Sale | Cheap Jordans 2017"@en-US	0
COL title VAL  "Nike AIR Zoom Pegasus 32 Men's Training Shoes - Total Orange/Ghost Green/Black"@en "Running and Training"@en	COL title VAL  "Nike AIR Zoom Pegasus 32 Men's Training Shoes - Green/Blue Lagoon/Black"@en "Clearance Footwear"@en	0

Probabilities: tensor([0.9999, 0.0167, 0.9953, 0.9995, 0.2451, 0.9998])

Now, predict only first 2 pairs:

COL title VAL  "Nike Metcon 2 - Black/White/Wolf Grey/Volt"@en " Nike Mens Shoes Regular Training Grey/Volt "@en	COL title VAL  "Nike Metcon DSX Flyknit - Wolf Grey/Volt/Wolf Grey/Black"@en-GB " Nike Mens Shoes Regular Training Grey/Black "@en-GB	0
COL title VAL  "Nike Zoom Structure 19 Comp Pack Men's Running Shoes SP16 007"@en 007 · Nike Moderate Support Trai ah"@en	COL title VAL  "KANMEI GS"@en-es GS | Unisex Kids Running Shoes ASICS"@en-es	0

probabilities: tensor([0.9999, 0.0159])

Now, predict only the second pair:

COL title VAL  "Nike Zoom Structure 19 Comp Pack Men's Running Shoes SP16 007"@en 007 · Nike Moderate Support Trai ah"@en   COL title VAL  "KANMEI GS"@en-es GS | Unisex Kids Running Shoes ASICS"@en-es    0

probability: tensor([0.0114])

I don't know how to fix this, and what is the problem here?