Taskboard

Taskboard for VisualWord2Vec!

Code

• Port the relationship modeling code to Github (5 June)
• Read and understand both the modeling and training (MATLAB) code (12 June)
• Clean up and document the training code to replicate results in the ICCV paper (15 June)
• Install and setup Caffe framework (already setup) (12 June)
• Remove irrelevant parts for the current work (19 June)
• Generate the AP curve based on amount of training data
• Re-train word2vec on MS COCO dataset to account for tokenization/lemmatization/case issues (June 23)
• Get results using new word2vec model after fixing best thresholds from validation
• Search and setup tools for refining the neural network (22 June)(?)
• Cluster the visual vectors, refine trained CNN to classify into one of the clusters, get new word2vec features (26 June)(?)

---

Ideas

How to incorporate visual features in word2vec?

1. Group the visual features obtained for 213 verbs into N clusters. Refine the trained CNN to predict the cluster id. Obtain the features from this modified CNN.
2. ?

---

Resources

• Training deep neural nets using deeplearning4j
• Training deep neural nets using gensim