Model training (#8)

* Added Random Dataset, Updated Model Code

Signed-off-by: Fabrice Normandin <[email protected]>

* Added Random Dataset, Updated Model Code

Signed-off-by: Fabrice Normandin <[email protected]>

* Removed outdated debug data

Signed-off-by: Fabrice Normandin <[email protected]>

* Fixed a few bugs in preprocessing_pilepine

Signed-off-by: Fabrice Normandin <[email protected]>

* Fixed NA bug in preprocessing_pipeline.py

Signed-off-by: Fabrice Normandin <[email protected]>

* Model Seems to be training fine on debug data

Signed-off-by: Fabrice Normandin <[email protected]>

* ALMOST there!

Signed-off-by: Fabrice Normandin <[email protected]>

* Training and Testing works in local DEBUG mode

Signed-off-by: Fabrice Normandin <[email protected]>

* Removed SKLEARN dependency (unused)

Signed-off-by: Fabrice Normandin <[email protected]>

* Disabled buggy profiling batch for Tensorboard

Signed-off-by: Fabrice Normandin <[email protected]>

* Fixed debug data being used for training on server

Signed-off-by: Fabrice Normandin <[email protected]>

* Removed buggy paths in train and test scripts

Signed-off-by: Fabrice Normandin <[email protected]>

* Fixed old SimpleParsing dependency

Signed-off-by: Fabrice Normandin <[email protected]>

* Removed ParseableFromCommandLine from model.py

Signed-off-by: Fabrice Normandin <[email protected]>

* Made the log_dir by default use today with hour

Signed-off-by: Fabrice Normandin <[email protected]>

* Added SimpleParsing submodule

Signed-off-by: Fabrice Normandin <[email protected]>

* Fixed metrics for gender and age group

Signed-off-by: Fabrice Normandin <[email protected]>

* reduced dropout rate, added customizable LR

Signed-off-by: Fabrice Normandin <[email protected]>

* Updated gitignore, added script for showing plots

Signed-off-by: Fabrice Normandin <[email protected]>

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "project/SimpleParsing"]
+	path = project/SimpleParsing
+	url = https://github.com/lebrice/SimpleParsing.git

project/.gitignore

@@ -1,2 +1,5 @@
 .vscode
-.mypy_cache
+.mypy_cache
+debug_output
+
+*.csv

project/debug_data/Image/oxford.csv

@@ -0,0 +1,11 @@
+userId,faceID,faceRectangle_width,faceRectangle_height,faceRectangle_left,faceRectangle_top,pupilLeft_x,pupilLeft_y,pupilRight_x,pupilRight_y,noseTip_x,noseTip_y,mouthLeft_x,mouthLeft_y,mouthRight_x,mouthRight_y,eyebrowLeftOuter_x,eyebrowLeftOuter_y,eyebrowLeftInner_x,eyebrowLeftInner_y,eyeLeftOuter_x,eyeLeftOuter_y,eyeLeftTop_x,eyeLeftTop_y,eyeLeftBottom_x,eyeLeftBottom_y,eyeLeftInner_x,eyeLeftInner_y,eyebrowRightInner_x,eyebrowRightInner_y,eyebrowRightOuter_x,eyebrowRightOuter_y,eyeRightInner_x,eyeRightInner_y,eyeRightTop_x,eyeRightTop_y,eyeRightBottom_x,eyeRightBottom_y,eyeRightOuter_x,eyeRightOuter_y,noseRootLeft_x,noseRootLeft_y,noseRootRight_x,noseRootRight_y,noseLeftAlarTop_x,noseLeftAlarTop_y,noseRightAlarTop_x,noseRightAlarTop_y,noseLeftAlarOutTip_x,noseLeftAlarOutTip_y,noseRightAlarOutTip_x,noseRightAlarOutTip_y,upperLipTop_x,upperLipTop_y,upperLipBottom_x,upperLipBottom_y,underLipTop_x,underLipTop_y,underLipBottom_x,underLipBottom_y,facialHair_mustache,facialHair_beard,facialHair_sideburns,headPose_roll,headPose_yaw,headPose_pitch
+fakeuser0,f7e072db-8532-4686-9074-27e83fee5e94,61,61,15,49,32.7,64.7,59.7,65.0,40.6,83.2,34.9,93.4,57.7,93.4,22.7,53.6,36.8,56.7,26.8,63.8,31.8,61.6,30.9,66.7,36.5,65.5,52.3,56.0,73.6,58.5,53.8,65.7,59.7,62.7,59.8,67.1,64.9,65.2,40.2,66.1,48.2,66.2,37.1,77.4,49.2,76.9,34.6,82.0,52.2,82.8,42.4,91.0,42.1,94.0,41.8,94.3,41.9,98.6,0.0,0.0,0.0,3.9,-22.4,0.0
+fakeuser1,934b5179-acec-4dea-a348-feae87767c2d,83,83,91,95,114.4,117.6,154.5,121.0,128.1,139.0,113.2,152.8,150.4,156.9,102.6,107.2,122.8,111.9,108.7,118.0,114.8,116.8,114.4,119.0,119.7,119.5,142.6,112.3,168.6,114.1,147.5,121.8,153.8,119.8,154.0,122.9,159.6,122.0,126.6,121.1,139.5,122.2,123.7,129.7,138.9,132.2,117.0,138.1,142.8,141.2,130.5,150.9,130.3,155.2,130.5,164.5,130.1,169.6,0.1,0.0,0.0,5.2,-8.3,0.0
+fakeuser2,118c1f96-b32a-4021-a993-8f60e9859517,76,76,22,50,49.6,64.1,79.6,78.7,67.6,96.1,35.2,97.0,65.0,109.3,38.6,54.3,67.7,61.9,46.4,62.6,51.9,62.9,50.3,66.7,56.1,67.6,79.9,69.6,92.8,78.5,75.7,77.5,80.9,77.5,80.1,81.2,84.9,81.7,64.6,72.1,73.5,75.5,57.5,85.1,71.7,89.8,51.7,89.1,71.5,97.4,57.7,103.9,56.5,105.9,52.4,114.6,51.2,118.5,0.5,0.4,0.2,22.7,27.1,0.0
+fakeuser3,1eb367c8-9467-411f-9689-fd1affa95654,121,121,10,21,45.8,55.4,97.5,49.9,65.9,83.0,51.8,115.6,91.0,111.4,22.5,52.2,60.5,44.2,37.1,57.9,45.1,52.1,46.0,59.3,54.2,54.0,75.2,43.6,120.8,42.8,89.6,49.9,98.7,45.2,98.0,53.6,105.6,49.4,62.0,55.0,74.3,54.9,56.4,74.4,79.4,75.1,52.9,87.1,85.6,85.0,67.1,106.2,67.9,110.1,67.3,111.4,67.8,119.3,0.3,0.3,0.3,-1.3,-11.7,0.0
+fakeuser4,384ff8c2-efd3-4d34-a9c1-e0756fc0312c,115,115,0,29,41.1,55.6,91.1,72.9,54.6,91.8,22.2,100.9,76.8,118.5,21.0,41.8,58.8,50.3,32.2,53.2,40.5,53.1,38.8,58.0,46.8,59.0,81.9,57.0,109.5,69.3,83.6,70.8,91.7,69.6,89.6,75.4,97.3,76.1,56.2,64.2,71.3,68.9,46.3,76.9,68.8,84.1,37.4,84.8,73.7,95.9,50.4,107.5,49.1,111.3,46.6,116.6,45.9,120.1,0.0,0.0,0.0,19.3,1.1,0.0
+fakeuser5,cafa30ba-6db7-4f50-8357-c6ef7e2c9811,64,64,32,42,48.9,57.8,77.1,60.5,65.4,75.3,50.5,88.2,75.8,90.0,36.3,50.6,56.3,51.4,44.3,57.8,49.0,55.9,48.7,60.1,54.2,58.8,72.2,52.5,88.0,51.8,73.0,60.2,77.8,58.3,77.6,62.3,83.0,61.0,60.8,59.9,68.2,60.5,57.2,70.2,70.9,71.1,55.5,76.7,72.9,77.3,63.9,86.6,63.9,89.4,64.2,90.4,63.7,95.6,0.0,0.0,0.0,1.1,7.6,0.0
+fakeuser6,36cabcfa-29f6-4237-8432-18e19b016b60,117,117,50,55,77.3,86.8,128.8,83.0,118.5,122.7,87.4,142.6,129.9,138.3,53.5,77.6,95.5,75.1,70.9,88.3,79.8,82.7,80.8,91.6,90.1,87.3,121.6,75.9,154.7,73.7,124.7,85.1,131.8,78.9,132.7,87.9,140.7,83.6,103.2,88.3,116.9,87.4,103.1,109.6,122.5,107.7,97.3,121.8,128.9,119.0,112.4,139.6,112.6,142.8,114.0,142.3,114.1,147.1,0.0,0.0,0.0,-4.9,15.3,0.0
+fakeuser7,2f3a97c5-e047-4573-804c-f2f071f9beb6,80,80,107,56,127.3,80.6,164.6,76.4,147.3,95.2,130.4,116.6,162.7,112.5,107.2,75.6,132.5,69.9,119.4,82.7,125.1,78.6,126.0,82.9,131.8,81.1,155.7,67.5,179.4,67.3,158.3,78.3,163.5,74.2,164.2,79.1,169.8,76.2,138.3,80.9,151.6,79.7,137.5,90.7,153.9,89.3,134.0,98.4,159.4,96.5,146.6,108.7,146.6,112.8,146.1,126.4,146.4,132.8,0.0,0.0,0.0,-4.5,3.3,0.0
+fakeuser8,895cbc2c-b520-4486-929c-966a04dea357,58,58,105,65,119.9,88.8,142.3,75.9,126.6,97.4,129.3,112.1,152.8,100.8,108.7,85.8,121.0,81.9,116.0,91.4,118.0,88.2,119.1,90.1,121.7,88.2,130.5,74.6,149.1,65.8,138.3,78.6,141.0,74.8,142.5,77.1,146.0,74.5,125.5,86.3,132.4,83.2,123.9,93.8,136.3,89.9,122.9,100.2,141.5,92.8,134.7,104.6,136.0,107.2,139.3,112.7,141.0,116.1,0.0,0.0,0.0,-23.2,-26.9,0.0
+fakeuser9,e315fe5b-1928-46ea-9f13-d9c933310b19,51,51,48,59,63.1,70.6,87.4,77.0,76.5,86.9,59.7,93.5,80.3,98.3,53.1,62.5,71.6,65.6,59.2,69.8,63.0,69.3,63.0,71.3,66.5,71.2,84.6,69.4,95.6,72.8,83.8,75.9,87.9,76.1,87.6,77.3,91.1,77.8,72.2,73.6,80.5,75.4,69.1,81.5,80.3,84.1,65.1,84.2,81.9,89.1,72.7,92.8,71.8,95.5,70.8,100.5,69.9,103.3,0.0,0.0,0.0,12.1,18.9,0.0

project/debug_data/Profile/Profile.csv

@@ -0,0 +1,11 @@
+,userid,age,gender,ope,con,ext,agr,neu
+0,fakeuser0,26.0,0.0,4.0,2.25,2.2,3.6,2.8
+1,fakeuser1,24.0,0.0,4.2,3.35,2.45,3.9,2.6
+2,fakeuser2,27.0,1.0,2.75,4.8,3.15,3.85,4.15
+3,fakeuser3,29.0,1.0,3.4,3.2,2.1,3.7,2.35
+4,fakeuser4,25.0,0.0,4.4,3.25,4.0,2.89,2.35
+5,fakeuser5,30.0,1.0,4.05,2.7,3.2,3.25,2.9
+6,fakeuser6,30.0,1.0,3.8,4.35,4.25,4.45,1.55
+7,fakeuser7,27.0,0.0,3.8,3.65,2.2,3.5,2.3
+8,fakeuser8,23.0,1.0,3.95,2.6,2.0,2.75,3.35
+9,fakeuser9,23.0,1.0,4.1,2.2,2.8,2.5,3.8

project/debug_data/Relation/Relation.csv

@@ -1,25 +1,31 @@
 ,userid,like_id
-0,7267f43c71fcf53f4580fd3cd808bd48,1001
-0,7267f43c71fcf53f4580fd3cd808bd48,1002
-0,7267f43c71fcf53f4580fd3cd808bd48,1003
-0,7267f43c71fcf53f4580fd3cd808bd48,1004
-1,e1cdac10d136e76e064e4860007d123d,1001
-1,e1cdac10d136e76e064e4860007d123d,1003
-1,e1cdac10d136e76e064e4860007d123d,1005
-1,e1cdac10d136e76e064e4860007d123d,1007
-2,e1cdac10d136e76e064e48600034555d,1002
-2,e1cdac10d136e76e064e48600034555d,1004
-2,e1cdac10d136e76e064e48600034555d,1006
-2,e1cdac10d136e76e064e48600034555d,1008
-3,e1cdac10d136e76e064e4860007d786d,1001
-3,e1cdac10d136e76e064e4860007d786d,1005
-3,e1cdac10d136e76e064e4860007d786d,1009
-3,e1cdac10d136e76e064e4860007d786d,1013
-4,e1cdac10d136e76e064e4860007d1231,1010
-4,e1cdac10d136e76e064e4860007d1231,1020
-4,e1cdac10d136e76e064e4860007d1231,1030
-4,e1cdac10d136e76e064e4860007d1231,1040
-5,e1cdac10d136e76e064e486000751245,1001
-5,e1cdac10d136e76e064e486000751245,1004
-5,e1cdac10d136e76e064e486000751245,1007
-5,e1cdac10d136e76e064e486000751245,1010
+0,fakeuser0,1001
+1,fakeuser1,1001
+2,fakeuser2,1001
+3,fakeuser3,1001
+4,fakeuser4,1001
+5,fakeuser5,1001
+6,fakeuser6,1001
+7,fakeuser7,1001
+8,fakeuser8,1001
+9,fakeuser9,1223
+10,fakeuser0,1002
+11,fakeuser1,1002
+12,fakeuser2,1002
+13,fakeuser3,1003
+14,fakeuser4,1003
+15,fakeuser5,1004
+16,fakeuser6,1004
+17,fakeuser7,1005
+18,fakeuser8,1005
+19,fakeuser9,1006
+20,fakeuser0,1007
+21,fakeuser1,1007
+22,fakeuser2,1008
+23,fakeuser3,1008
+24,fakeuser4,1009
+25,fakeuser5,1009
+26,fakeuser6,1010
+27,fakeuser7,1010
+28,fakeuser8,1011
+29,fakeuser9,1012

project/debug_data/Text/liwc.csv

@@ -0,0 +1,11 @@
+userId,WC,WPS,Sixltr,Dic,Numerals,funct,pronoun,ppron,i,we,you,shehe,they,ipron,article,verb,auxverb,past,present,future,adverb,preps,conj,negate,quant,number,swear,social,family,friend,humans,affect,posemo,negemo,anx,anger,sad,cogmech,insight,cause,discrep,tentat,certain,inhib,incl,excl,percept,see,hear,feel,bio,body,health,sexual,ingest,relativ,motion,space,time,work,achieve,leisure,home,money,relig,death,assent,nonfl,filler,Period,Comma,Colon,SemiC,QMark,Exclam,Dash,Quote,Apostro,Parenth,OtherP,AllPct
+fakeuser0,156,17.33,25,76.92,1.92,45.51,9.62,6.41,3.21,1.28,0,1.28,0.64,3.21,8.97,10.9,6.41,1.92,7.05,0.64,4.49,10.9,4.49,1.28,1.28,0.64,0.64,8.33,0,0,1.92,5.13,1.92,3.21,1.28,0.64,0.64,16.03,2.56,3.85,1.28,2.56,0,0,4.49,2.56,3.85,1.92,0.64,1.28,1.28,1.28,0,0,0,7.69,0.64,3.21,3.85,4.49,0.64,1.28,0,0.64,0,0,0.64,0,0,38.46,8.33,1.92,0,5.77,5.77,2.56,0,2.56,0.64,5.13,70.51
+fakeuser1,176,11,21.02,84.66,0,53.41,15.34,12.5,7.39,0,3.41,0,1.7,2.84,5.11,13.07,9.09,3.98,7.39,0.57,4.55,11.93,3.98,2.84,1.7,1.14,0,9.09,0.57,0.57,0.57,8.52,6.82,1.7,1.14,0,0.57,14.77,2.27,2.27,0,0.57,3.98,0,4.55,1.14,2.27,1.14,1.14,0,1.7,0,1.14,0,0.57,17.05,3.41,4.55,8.52,1.14,1.7,0.57,1.14,0.57,1.14,0,0.57,0,0,12.5,9.66,0.57,0,0,13.07,0,0,2.27,0.28,1.14,39.77
+fakeuser2,179,17.9,17.88,90.5,0,54.19,11.73,7.82,6.15,0,1.12,0.56,0,3.91,8.38,18.99,12.29,4.47,11.17,1.68,5.03,11.17,6.7,2.79,2.79,0.56,0,6.15,0,0.56,0,7.26,4.47,2.79,0.56,1.12,0.56,15.08,1.68,1.68,3.35,3.35,1.68,0,3.35,3.35,5.03,2.79,1.12,1.12,1.68,0.56,0,0,1.12,18.99,3.91,6.7,7.82,1.68,0.56,3.35,1.68,0.56,0,0,0,0,0,19.55,12.85,0,0.56,0.56,1.68,1.12,0,5.59,0.28,0,41.9
+fakeuser3,179,179,20.67,77.09,0,51.4,12.29,9.5,4.47,0,4.47,0,0.56,2.79,8.38,10.06,4.47,1.12,6.15,1.68,1.68,15.08,6.15,1.12,2.79,1.68,0,7.26,0.56,0,0.56,2.79,0,2.79,0,0.56,1.12,16.2,3.35,0,3.35,5.03,1.68,0,5.03,3.35,6.15,2.79,3.35,0,1.68,1.68,0,0,0,11.73,1.12,7.26,3.35,0,0.56,0.56,0,0,1.12,0.56,0,0,0,25.7,4.47,0,0,1.68,0,1.12,0,2.23,0,0,34.64
+fakeuser4,18,18,44.44,55.56,0,27.78,0,0,0,0,0,0,0,0,0,5.56,0,0,5.56,0,0,16.67,0,0,11.11,0,0,11.11,0,0,0,0,0,0,0,0,0,16.67,0,0,5.56,0,11.11,0,0,0,0,0,0,0,0,0,0,0,0,5.56,0,0,5.56,5.56,5.56,0,0,0,0,0,0,0,0,333.33,55.56,22.22,0,3661.11,0,0,11.11,0,5.56,16.67,4111.11
+fakeuser5,164,18.22,23.78,73.17,0.61,40.85,12.8,10.37,9.76,0,0,0.61,0,2.44,6.1,15.85,9.15,4.27,10.98,0,4.88,8.54,2.44,2.44,1.83,0,3.05,3.66,0,0,1.83,10.37,3.66,6.71,0.61,4.88,0,3.66,0,1.83,0.61,0.61,1.22,0,0,0,1.83,0.61,0,0.61,4.88,3.05,0.61,1.83,0,15.24,3.05,4.27,7.93,0.61,0,2.44,0,0,0,0.61,0.61,1.83,0,9.15,26.22,8.54,0,0.61,13.41,0.61,0,1.22,1.52,1.83,64.63
+fakeuser6,125,20.83,29.6,71.2,1.6,38.4,13.6,11.2,7.2,3.2,0,0.8,0,2.4,4,15.2,4.8,3.2,11.2,0.8,2.4,12.8,3.2,0.8,0.8,0,0.8,9.6,2.4,0,0,4.8,2.4,2.4,0,0.8,0,7.2,0.8,1.6,1.6,0.8,0,0,1.6,1.6,1.6,1.6,0,0,0.8,0.8,0,0,0,12.8,5.6,3.2,4,3.2,3.2,4.8,0.8,0,0,0,0,0,0,11.2,8,2.4,0,0,15.2,1.6,0,24,0.8,33.6,97.6
+fakeuser7,155,10.33,20.65,41.29,1.29,25.16,5.81,4.52,3.87,0,0,0.65,0,1.29,4.52,5.16,2.58,0.65,4.52,0,3.23,5.16,3.23,0,1.29,0.65,0,3.23,0.65,0,0,3.87,1.94,1.94,0,1.29,0,7.1,1.29,0,0,0.65,1.94,0,1.94,1.29,0.65,0.65,0,0,1.29,0,0.65,1.29,0,5.16,0,0.65,3.87,1.29,0,1.29,0,0.65,0.65,0,0,0,0,32.26,12.26,0.65,2.58,4.52,3.23,1.94,1.94,1.94,0.97,19.35,81.94
+fakeuser8,186,186,18.82,69.35,1.08,38.17,12.37,9.68,7.53,0,1.08,0.54,0.54,2.69,4.3,9.68,6.45,3.23,3.76,2.69,2.15,6.45,5.38,2.15,1.61,0,1.08,2.69,0,0,0,6.45,2.15,4.3,0.54,1.61,1.08,10.22,0,0.54,2.69,3.23,0,0.54,3.23,2.69,0.54,0,0,0.54,3.76,1.08,2.15,0,0.54,16.67,0.54,6.45,10.22,3.76,4.3,1.08,0,0.54,0.54,0,1.61,0,0.54,0,15.59,0.54,0.54,1.08,5.38,0,0,0,0.54,1.08,25.27
+fakeuser9,168,21,20.24,74.4,0,48.81,14.29,9.52,4.17,0,3.57,1.19,0.6,4.76,7.14,15.48,10.71,0,11.9,1.19,3.57,9.52,3.57,1.19,1.19,0.6,0,11.31,0,0.6,3.57,4.76,2.38,2.38,0.6,0,0.6,13.1,2.98,2.38,2.38,1.19,1.19,0,1.79,2.38,3.57,0.6,1.19,1.79,2.38,1.79,0.6,0,0,4.17,0,2.38,1.19,0.6,0.6,0,0,0.6,0,0.6,0,0,2.38,11.9,13.69,2.38,0,9.52,24.4,0,0,0.6,0.3,2.98,66.07

project/debug_data/Text/nrc.csv

@@ -0,0 +1,11 @@
+userId,positive,negative,anger,anticipation,disgust,fear,joy,sadness,surprise,trust
+fakeuser0,0.578947368421,0.421052631579,0.107142857143,0.107142857143,0.0714285714286,0.214285714286,0.142857142857,0.178571428571,0.0357142857143,0.142857142857
+fakeuser1,0.45,0.55,0.15625,0.125,0.125,0.09375,0.1875,0.09375,0.0,0.21875
+fakeuser2,0.617647058824,0.382352941176,0.127272727273,0.163636363636,0.0909090909091,0.127272727273,0.163636363636,0.0909090909091,0.0363636363636,0.2
+fakeuser3,0.730769230769,0.269230769231,0.0847457627119,0.186440677966,0.0338983050847,0.0847457627119,0.254237288136,0.101694915254,0.101694915254,0.152542372881
+fakeuser4,0.875,0.125,0.0192307692308,0.134615384615,0.0192307692308,0.0192307692308,0.384615384615,0.0192307692308,0.0576923076923,0.346153846154
+fakeuser5,0.857142857143,0.142857142857,0.0555555555556,0.166666666667,0.0555555555556,0.111111111111,0.194444444444,0.111111111111,0.0833333333333,0.222222222222
+fakeuser6,0.818181818182,0.181818181818,0.0322580645161,0.193548387097,0.0,0.0645161290323,0.258064516129,0.0322580645161,0.129032258065,0.290322580645
+fakeuser7,0.52,0.48,0.135135135135,0.108108108108,0.0810810810811,0.162162162162,0.0810810810811,0.189189189189,0.0810810810811,0.162162162162
+fakeuser8,0.92,0.08,0.0232558139535,0.255813953488,0.0232558139535,0.046511627907,0.325581395349,0.0232558139535,0.0697674418605,0.232558139535
+fakeuser9,0.375,0.625,0.0727272727273,0.0909090909091,0.0909090909091,0.109090909091,0.163636363636,0.236363636364,0.109090909091,0.127272727273