Deep Learning for NLP - wiki

Wiki for the Deep Learning for Natural Language Processing course at Indian Institute of Science, Bangalore - Aug 2017 to Dec 2017.

Instructors

• Dr Shirish Shevade, CSA, IISc.
• Dr Sundararajan Sellamanickam, Microsoft Research.

-----------------------------------------------------------------------------

Project Groups

1. Understanding the Geometrical Relationship among Word Vectors - Aditya Sharma, Chandrahas
2. Comparative Study of Neural Language Models - Abhinash Khare, Soham Pal, Sruthi Gorantla
3. Document time stamping using Graph Convolutional Networks - Shikhar, Swayambhu, Shib Shankar
4. Hierarchical attention network - Vaijenath, Anil, Pratik
5. Question Answer System - Rohith AP, Ronit Halder, V Nidhin Krishnan.
6. Combined Neural Models: Comparison and Analysis - Soumalya, Srinighi, Bala
7. Transfer Learning for NLP with application to detecting duplicate question pairs - Krishna, Rantej, Ravi
8. Neural Machine Translation - Kinsuk Sarkar, Maulik Parmar, Ayappa Kumar
9. Topic Modeling - Tushar, Sridhar
10. Reinforcement Learning for NLP - Keshav
11. Classification using Sentence Level Embedding - Shikhar Verma, Nitin Kumar, Shubham Goel

-----------------------------------------------------------------------------

Papers Covered by Students

Understanding the Geometrical Relationship among Word Vectors

Team Members - Aditya Sharma, Chandrahas

1. October 13th, 2017:
   • Levy, O. and Goldberg, Y., 2014. Neural word embedding as implicit matrix factorization. In Advances in neural information processing systems (pp. 2177-2185). [Paper]
   • Levy, O., Goldberg, Y. and Dagan, I., 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3, pp.211-225. [Paper]
   • Mimno, D. and Thompson, L., 2017. The strange geometry of skip-gram with negative sampling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (pp. 2863-2868). [Paper]

Comparative Study of Neural Language Models

Team Members - Abhinash Khare, Soham Pal, Sruthi Gorantla

1. October 20th, 2017:
   • Wang Ling, Isabel Trancoso, Chris Dyer, Alan W Black. Character-based Neural Machine Translation. [Paper]
   • Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu. Exploring the Limits of Language Modeling. [Paper]
   • Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, Christopher Manning. A Conditional Random Field Word Segmenter. [Paper]

Document time stamping using Graph Convolutional Networks

Team Members - Shikhar, Shib, Swayambhu

1. October 20th, 2017:
   • Defferrard, Michaël, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in Neural Information Processing Systems. 2016. [Paper]
   • Marcheggiani, Diego, and Ivan Titov. Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling. arXiv preprint arXiv:1703.04826 (2017). [Paper]

Hierarchical attention network

Team Members - Vaijenath, Anil, Pratik

1. November 10th, 2017:
   • Yang, Z., Yang, D., Dyer, C., He, X., Smola, A.J. and Hovy, E.H., 2016. Hierarchical Attention Networks for Document Classification. In HLT-NAACL (pp. 1480-1489). [Paper]
   • Xu, J., Chen, D., Qiu, X. and Huang, X., Cached Long Short-Term Memory Neural Networks for Document-Level Sentiment Classification. [Paper]

Question Answer System

Team Members - Rohith AP, Ronit Halder, V Nidhin Krishnan

1. November 10th, 2017:
   • Shuohang Wang, Jing Jiang Machine Comprehension Using Match-LSTM and Answer Pointer. [Paper]
   • Danqi Chen, Adam Fisch, Jason Weston, Antoine Bordes. Reading Wikipedia to Answer Open-Domain Questions. [Paper]
   • Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi,and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension.[Paper]

Combined Neural Models: Comparison and Analysis

Team Members - Soumalya Seal, Srinidhi R, Balasubramaniam S

1. November 8th, 2017:
   • Xingyou Wang,Weijie Jiang, Zhiyong Luo, Combination of Convolutional and Recurrent Neural Network for Sentiment Analysis of Short Texts. [Paper]
   • Chunting Zhou, Chonglin Sun, Zhiyuan Liu, Francis C.M. Lau, A C-LSTM Neural Network for Text Classification. [Paper]
   • Siwei Lai, Liheng Xu, Kang Liu, Jun Zhao, Recurrent convolutional neural networks for text classification. [Paper]

Transfer Learning for NLP with application to detecting duplicate question pairs -

Team Members - Krishna Bharat, Rantej, Ravi Shankar

1. October 13th, 2017:
   • Lili Mou, Zhao Meng,Rui Yan, Ge Li, Yan Xu, Lu Zhang, Zhi Jin, How Transferable are Neural Networks in NLP Applications? [Paper]
   • Zhilin Yang, Ruslan Salakhutdinov, William W Cohen, Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks. [Paper]
   • Seunghyun Yoon, Hyeongu Yun, Yuna Kim, Gyu-tae Park, Kyomin Jung, Efficient Transfer Learning Schemes for Personalized Language Modeling using Recurrent Neural Network. [Paper]

Neural Machine Translation

Team Members - Kinsuk Sarkar, Maulik Parmar, Ayappa Kumar

1. October 20th, 2017:
   • Sutskever, I., Vinyals, O. and Le, Q.V, Sequence to Sequence Learning with Neural Networks. [Paper]
   • Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K. and Klingner, J., Googles's Neural Machine Translation System: Bridging the gap between Human and Machine Translation. [Paper]

Topic Modeling

Team Members - Tushar, Sridhar

1. October 20th, 2017:
   • Sridhar, V.K.R., Unsupervised Topic Modeling for Short Texts Using Distributed Representations of Words. [Paper]
   • Batmanghelich, K., Saeedi, A., Narasimhan, K. and Gershman, S., 2016. Nonparametric spherical topic modeling with word embeddings. [Paper]

Reinforcement Learning for NLP

Team Members - Keshav

1. November 8th, 2017:
   • Yogatama, D., Blunsom, P., Dyer, C., Grefenstette, E. and Ling, W., Learning to compose words into sentences with reinforcement learning. [Paper]
   • Tai, K.S., Socher, R. and Manning, C.D., Improved semantic representations from tree-structured long short-term memory networks. [Paper]

Classification using Sentence Level Embedding

Team Members - Shikhar Verma, Nitin Kumar, Shubham Goel

1. November 8th, 2017:
   • Ryan Kiros, Yukun Zhu, Ruslan Salakhutdinov, Richard S. Zemel, Antonio Torralba, Raquel Urtasun, Sanja Fidler. 2015. Skip Thought Vectors. [Paper]
   • Matteo Pagliardini, Prakhar Gupta, Martin Jaggi. 2017. Unsupervised Learning of Sentence Embeddings using Compositional n-gram features. [Paper]

-----------------------------------------------------------------------------

Papers Covered by Professors

Basics of Optimization

• [Blog Post] Sebastian Ruder. An overview of gradient descent optimization algorithms [Link]

Word Vectors

• Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S. and Dean, J., 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems [Paper]
• Pennington, J., Socher, R. and Manning, C., 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) [Paper]
• Wieting, J., Bansal, M., Gimpel, K. and Livescu, K., 2016. Charagram: Embedding words and sentences via character n-grams. [Paper]

Language Modelling

• Bengio, Y., Ducharme, R., Vincent, P. and Jauvin, C., 2003. A neural probabilistic language model. Journal of machine learning research [Paper]
• Mikolov, T., Karafiát, M., Burget, L., Cernocký, J. and Khudanpur, S., 2010, September. Recurrent neural network based language model. In Interspeech (Vol. 2, p. 3). [Paper]
• Dauphin, Y.N., Fan, A., Auli, M. and Grangier, D., 2016. Language modeling with gated convolutional networks. [Paper]
• Zhao, H., Lu, Z. and Poupart, P., 2015, July. Self-Adaptive Hierarchical Sentence Model. In IJCAI [Paper]
• Conneau, A., Kiela, D., Schwenk, H., Barrault, L. and Bordes, A., 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In EMNLP [Paper]
• Kim, Y., Jernite, Y., Sontag, D. and Rush, A.M., 2016, February. Character-Aware Neural Language Models. In AAAI [Paper]

Text Classification

• Joulin, A., Grave, E. and Mikolov, P.B.T., 2017. Bag of Tricks for Efficient Text Classification. EACL [Paper]
• Kim, Y., 2014. Convolutional Neural Networks for Sentence Classification. EMNLP [Paper]

Networks

• Hochreiter, S. and Schmidhuber, J., 1997. Long short-term memory. Neural computation. [Paper]
• [Blog Post] Christopher Olah. Understanding LSTM Networks [Link]
• Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I. and Salakhutdinov, R., 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research [Paper]
• Chung, J., Gulcehre, C., Cho, K. and Bengio, Y., 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. [Paper]
• Koch, G., Zemel, R. and Salakhutdinov, R., 2015. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop. [Paper]

Neural Machine Translation

• Bahdanau, D., Cho, K. and Bengio, Y., 2014.Neural machine translation by jointly learning to align and translate. [Paper]
• Freitag, M. and Al-Onaizan, Y., 2017. Beam Search Strategies for Neural Machine Translation. [Paper]

Attention

• Yin, W., Schütze, H., Xiang, B. and Zhou, B., 2016. Abcnn: Attention-based convolutional neural network for modeling sentence pairs. In TACL [Paper]
• Yang, Z., Yang, D., Dyer, C., He, X., Smola, A.J. and Hovy, E.H., 2016. Hierarchical Attention Networks for Document Classification. In HLT-NAACL [Paper]
• Rocktäschel, T., Grefenstette, E., Hermann, K.M., Kočiský, T. and Blunsom, P., 2015. Reasoning about entailment with neural attention. [Paper]

Visualization

• Zeiler, M.D. and Fergus, R., 2014, September. Visualizing and understanding convolutional networks. In European conference on computer vision (pp. 818-833). Springer, Cham. [Paper]