WP 1: Lexicon, Syntax, Semantics: Structured Prediction and Domain Adaptation (WS 2017-2018)


Structured prediction is typically solved using classifiers which predict an output with structure, often by making several classification decisions which are dependent on one another. Examples of structured prediction problems include language modeling and machine translation (where the words in the output depend on one another) and syntactic problems such as POS-tagging or syntactic parsing (where the classification labels chosen depend on one another).

Domain adaptation is, broadly speaking, the problem of training a classifier using one distribution and testing it on another distribution. There are many examples of this in real-world applications of natural language processing. One example is that a machine translation system may be trained on government proceedings (such as the Proceedings of the European Parliament) but then applied to translating medical documents. Another example is that a syntactic parser may be trained on news articles, but applied to predicting the syntactic structure of sentences in tweets.


The seminar will discuss these two interesting problems and their intersection, by looking at state-of-the-art research papers. Most of the papers discussed will involve deep learning approaches applied to problems of structured prediction, but familiarity with older classification approaches (e.g., using linear models) will be useful.


The goal of the seminar is to understand the basics of structured prediction, domain adaptation, and domain adaptation techniques applied to structured prediction. A secondary goal is to acquire a broad picture of the state-of-the-art in these problems, particularly as applied in natural language processing.


Alexander Fraser

Email Address: SubstituteMyLastName@cis.uni-muenchen.de

CIS, LMU Munich


Room U139, Tuesdays, 16:00 to 18:00 (c.t.)

Date Topic Reading (DO BEFORE THE MEETING!) Slides
October 17th Orientation pdf
October 24th Introduction to Structured Prediction and Domain Adaptation pptx pdf
October 31st HOLIDAY

Referatsthemen (name: topic)

Sign up by sending an email to Fraser, please specify a ranked list with a number of papers in case your first choice is already assigned.

The initials refer to the person proposing the topic (AF, Dr. Fabienne Braune, Matthias Huck, Dario Stojanovski). Please contact this person for questions about that topic.

Warning: the order and dates here are subject to change, DO NOT COUNT ON THE DATE REMAINING THE SAME, IT MAY BECOME EARLIER OR LATER!
Date Topic Materials Assigned To Slides Hausarbeit Received
November 7th (FB) Y Bengio, R Ducharme, P Vincent (2003). A neural probabilistic language model. Journal of Machine Learning Research 3, 1137-1155 pdf
November 7th (FB) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, Pavel Kuksa (2011). Natural Language Processing (Almost) from Scratch. Journal of Machine Learning Research 2011. paper Iryna Markovych slides yes
November 14th (MH) Sutskever, Ilya, Oriol Vinyals, and Quoc V Le (2014). Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems. link Kevin Falkner slides yes
November 14th (MH) Bahdanau, Dzmitry, Kyunghyun Cho, Yoshua Bengio (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR. link Maximilian Martin slides yes
November 21st (AF) Georg Heigold, Günter Neumann and Josef van Genabith (2017). An Extensive Empirical Evaluation of Character-Based Morphological Tagging for 14 Languages. EACL. paper Steffen Freisinger slides yes
November 21st (AF) Do Kook Choe, Eugene Charniak (2016). Parsing as Language Modeling. EMNLP. paper Marina Speranskaya slides yes
November 28th (DS) Luheng He, Kenton Lee, Mike Lewis, Luke Zettlemoyer (2017). Deep Semantic Role Labeling: What Works and What's Next. ACL. paper Benjamin Plötz slides yes
November 28th (FB) Minh-Thang Luong, Ilya Sutskever, Oriol Vinyals, Wojciech Zaremba (2015). Addressing the rare word problem in NMT. ACL paper Agata Barcik slides yes
December 5th (MH) Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, James Glass (2017). What do Neural Machine Translation Models Learn about Morphology? ACL paper Daria Glazkova slides yes
December 5th (FB) Attention with intention for a neural network conversation model (2015). NIPS workshop on Machine Learning for Spoken Language Understanding and Interaction. paper Christoph Papadatos slides yes
December 12th (DS) Tong Wang, Ping Chen, John Rochford, Jipeng Qiang (2016) Text Simplification Using Neural Machine Translation. AAAI. paper Anastasiya Kryvosheya slides yes
December 12th (DS) Xia Cui, Frans Coenen, Danushka Bollegala (2017). Effect of Data Imbalance on Unsupervised Domain Adaptation of Part-of-Speech Tagging and Pivot Selection Strategies. LIDTA paper Robin Rojowiec slides yes
December 19th (FB) Nanyun Peng and Mark Dredze (2017). Multi-task Domain Adaptation for Sequence Tagging. Proceedings of the 2nd Workshop on Representation Learning for NLP. paper Eduard Saller slides yes
December 19th (FB) Markus Freitag, Yaser Al-Onaizan (2016). Fast Domain Adaptation for Neural Machine Translation. arXiv preprint arXiv:1612.06897 paper Anne Beyer slides yes
January 9th (DS) Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, Victor Lempitsky (2016). Domain-Adversarial Training of Neural Networks. JMLR paper Azada Rustamova slides yes
January 9th (DS) Boxing Chen, Colin Cherry, George Foster, Samuel Larkin (2017). Cost Weighting for Neural Machine Translation Domain Adaptation. First Workshop on Neural Machine Translation. paper Jannis Vamvas slides yes
January 16th (DS) Catherine Kobus, Josep Crego, Jean Senellart (2016). Domain Control for Neural Machine Translation. arXiv. paper Xieyidan Abuliezi slides yes
January 16th (AF) Yoon Kim, Yacine Jernite, David Sontag, Alexander M. Rush (2016). Character-Aware Neural Language Models. AAAI paper
January 23rd (AF) Jason Lee, Kyunghyun Cho, Thomas Hofmann (2016). Fully Character-Level Neural Machine Translation without Explicit Segmentation. arXiv. paper Simon Schäfer slides yes
January 23rd (AF) Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, Noah A. Smith (2017). What Do Recurrent Neural Network Grammars Learn About Syntax? EACL paper Verena Pongratz slides yes
January 30th (MH) Yang Feng, Shiyue Zhang, Andi Zhang, Dong Wang, Andrew Abel (2017). Memory-augmented Neural Machine Translation. EMNLP paper Bernhard Jackl slides yes
January 30th (AF) Andrej Karpathy, Li Fei-Fei (2015). Deep Visual-Semantic Alignments for Generating Image Descriptions. CVPR paper


If you do not have a general machine learning background, I'd suggest:

A Course in Machine Learning
by Hal Daumé III
2017 (0.99 beta version, for now)

If you would like to review neural networks and deep learning, this book will be useful:

Neural Network Methods in Natural Language Processing
by Yoav Goldberg

or maybe this (free) review, also by Yoav Goldberg: