CSCI 544: Applied Natural Language Processing — Fall 2019
Prof. Nanyun (Violet) Peng

Announcements | Course Information | Schedule

Announcements

Course Information

Course objectives: Welcome! This course is designed to introduce you to some of the problems and solutions of NLP, and their relation to linguistics and statistics. You need to know how to program and use common data structures.
It might also be nice—though it's not required—to have some previous familiarity with linear algebra and probabilities.
At the end you should agree (I hope!) that language is subtle and interesting, feel some ownership over some of NLP's formal and statistical techniques, and be able to understand research papers in the field.

Lectures:WF 5:30 - 7:20pm
Location: WPH B27.
Prof:Nanyun (Violet) Peng Email: npeng@isi.edu
TAs: Rujun Han Email: rujunhan@isi.edu
Graders: Sachin Vorkady Balakrishna, Email: vorkadyb@usc.edu; Anisha Jagadeesh Prasad, Email: ajagadee@usc.edu, Jiawei Zhang, Email: zhan890@usc.edu
Office hrs: Prof: Wed. 4:30pm at RTH 512; or by appt
TAs: Fri. 1:00pm - 3:00pm
Discussion site: Piazza https://piazza.com/usc/fall2019/csci544/home ... public questions, discussion, announcements
Web page:https://violetpeng.github.io/cs544_fa19.html
Textbook: Jurafsky & Martin, 3rd ed. (recommended)
Manning & Schütze (recommended)
Policies: Grading: homework 40%, project 20%, midterm 15% or 25%, final 15% or 25%
Honesty: Viterbi integrity code, USC-Viterbi graduate policies

Schedule

Warning: The schedule below may change. Links to future lectures and assignments are just placeholders and will change.


Week Wednesday Friday Suggested Reading
8/26 Introduction
  • Why is NLP hard?
  • Levels of language
  • NLP applications
    		• Probability concepts
  • Joint & conditional prob
  • Chain rule and backoff
  • Modeling sequences
  • Cross-entropy and perplexity
  • Intro: J&M chapter 1
  • Chomsky hierarchy: J&M 16
  • Prob/Bayes: M&S 2
    • 9/2 Modeling grammaticality; N-gram language models
  • What's wrong with n-grams?
  • Regular expressions, FSAs, CFGs, ...
  • Spelling correction
  • Segmentation
  • Speech recognition
  • Machine translation
    		• Smoothing n-grams
  • Bias and variance
  • Add-one or add-λ smoothing
  • Cross-validation
  • Smoothing with backoff
  • Good-Turing, Witten-Bell
  • Log-linear models
  • Language models: J&M 3
    • 9/9 Assignment 1 given: Probabilities
    Intro to neural language Models
  • Conditional log-linear models
  • Maximum likelihood, regularization
  • Feedforward neural language Models
  • Recurrent neural language Models
    		• No class (SoCal NLP symposium)
  • You are encouraged to attend
  • On the USC campus
  • Free registration, free food
  • Several companies recruiting
  • Smoothing: J&M 3; Rosenfeld (2000)
  • Neural language models: J&M 7; OpenAI blog post GPT-2 (with paper); BERT paper
    • 9/16 Context-free parsing
  • What is parsing?
  • Why is it useful?
  • Brute-force algorithm
  • CKY algorithms
    		• Guest Lecture from TA:
  • Deep Learning workshop
  • Intro to PyTorch
  • Attributes: J&M 12
  • Parsing: J&M 13
    • 9/23 Assignment 1 due
    Assignment 2 given: Language Models
    Probabilistic parsing
  • PCFG parsing
  • Dependency grammar
  • Lexicalized PCFGs
    		• Dependency Parser
  • Dependency Trees
  • Shift-reduce parser
  • CCG: Steedman & Baldridge; more
  • TAG/TSG: Van Noord, Guo, Zhang 1/2/3
  • Prob. parsing: J&M 14
    • 9/30 Semantics
  • What is understanding?
  • Semantic phenomena and representations
  • WordNet
  • Semantic similarity
    		• Midterm review
  • Word Sense and WordNet: J&M 19; lambda calculus for kids
    • 10/7 Midterm exam
    (5:30-6:30 in classroom)         		Distributional semantics (word embeddings)
  • Compositional semantics
  • Distributional semantics
  • Vector Semantics: J&M 6
    • 10/14 Sequence tagging models
  • Ice cream, weather, words and tags
  • Inferring hidden states
  • Likelihood convergence
    		• Project proposal due
    No class (fall break)
  • The Viterbi Algorithm: J&M 8
    • 10/21 Assignment 2 due
    Assignment 3 given: Semantics
    		 HMM, MEMM, and CRF
  • Generalizing the forward-backward strategy
  • Inside-outside algorithm
  • Posterior decoding
    		• Neural sequence tagging and relation extraction
  • Recurrent Neural Networks
  • Neural CRFs
  • Relation Extraction
  • Hidden Markov Models: J&M Appendix A; John Lafferty's paper on CRF
    • 10/28 Dialog systems
  • Guest Lecture: Sarik Ghazarian
  • Dialog system
    		• Text classification
  • Features
  • Linear Classifiers
    • 11/4 Assignment 3 due
    Assignment 4 given: Neural Sequence Tagging
    		 Project proposal revision (if applicable) due
  • Guest Lecture: Anna Farzindar
    		• Machine Translation
  • introduction
  • history
  • evaluation
    • 11/11 Phrase-based machine translation
  • Phrase-based MT
  • Alignment
  • IBM models
    		• Sequence to sequence models
  • RNN-based seq-to-seq models
  • Transformers
  • Applications to machine translation, summarization.
  • Morphology: R&S 2
    • 11/18 NLP Applications: Creative Generation
  • Stemming
  • Compounds, segmentation
  • Two-level morphology
  • Punctuation
  • Rewrite rules
    		• Socially responsible NLP
  • Biases in word embeddings.
  • Biases in coreference resolution.
  • Bias-mitigation algorithms
  • MT: J&M 25, M&S 13, statmt.org; tutorial (2003), workbook (1999), introductory essay (1997), technical paper (1993); tutorial (2006) focusing on more recent developments (slides, 3-hour video part 1, part 2)
    • 11/25 No class
    (Thanksgiving break)     		Assignment 4 due
    No class
    (Thanksgiving break)
    12/2 Final Project Due
    Final exam recitation I     		Final exam recitation II


    Final exam: Wed 12/11, in class.