CSCI 544: Applied Natural Language Processing — Fall 2019
Prof. Nanyun (Violet) Peng

Announcements | Course Information | Schedule


Announcements


Course Information

Course objectives: Welcome! This course is designed to introduce you to some of the problems and solutions of NLP, and their relation to linguistics and statistics. You need to know how to program and use common data structures.
It might also be nice—though it's not required—to have some previous familiarity with linear algebra and probabilities.
At the end you should agree (I hope!) that language is subtle and interesting, feel some ownership over some of NLP's formal and statistical techniques, and be able to understand research papers in the field.

Lectures:WF 5:30 - 7:20pm
Location: WPH B27.
Prof:Nanyun (Violet) Peng Email: npeng@isi.edu
TAs: Rujun Han Email: rujunhan@isi.edu
CAs: Sachin Vorkady Balakrishna, Email: vorkadyb@usc.edu; Anisha Jagadeesh Prasad, Email: ajagadee@usc.edu
Office hrs: Prof: Wed. 4:30pm; or by appt
TAs: Fri. 1:00pm - 3:00pm
Discussion site: Piazza https://piazza.com/usc/fall2019/csci544/home ... public questions, discussion, announcements
Web page:https://violetpeng.github.io/cs544_fa19.html
Textbook: Jurafsky & Martin, 3rd ed. (recommended)
Manning & Schütze (recommended)
Policies: Grading: homework 40%, project 20%, midterm 15% or 25%, final 15% or 25%
Honesty: Viterbi integrity code, USC-Viterbi graduate policies


Schedule

Warning: The schedule below may change. Links to future lectures and assignments are just placeholders and will change.


Week Wednesday Friday Suggested Reading
8/26 Introduction
  • Why is NLP hard?
  • Levels of language
  • NLP applications
  • Probability concepts
  • Joint & conditional prob
  • Chain rule and backoff
  • Modeling sequences
  • Cross-entropy and perplexity
  • Intro: J&M chapter 1
  • Chomsky hierarchy: J&M 16
  • Prob/Bayes: M&S 2
  • 9/2 Modeling grammaticality; N-gram language models
  • What's wrong with n-grams?
  • Regular expressions, FSAs, CFGs, ...
  • Spelling correction
  • Segmentation
  • Speech recognition
  • Machine translation
  • Smoothing n-grams
  • Bias and variance
  • Add-one or add-λ smoothing
  • Cross-validation
  • Smoothing with backoff
  • Good-Turing, Witten-Bell
  • Log-linear models
  • Language models: J&M 3
  • 9/9 Assignment 1 given: Probabilities
    Intro to neural language Models
  • Conditional log-linear models
  • Maximum likelihood, regularization
  • Feedforward neural language Models
  • Recurrent neural language Models
  • No class (SoCal NLP symposium)
  • You are encouraged to attend
  • On the USC campus
  • Free registration, free food
  • Several companies recruiting
  • Smoothing: J&M 3; Rosenfeld (2000)
  • Neural language models: J&M 7; OpenAI blog post GPT-2 (with paper); BERT paper
  • 9/16 Context-free parsing (ppt)
  • What is parsing?
  • Why is it useful?
  • Brute-force algorithm
  • CKY and Earley algorithms
  • Guest Lecture from TA: Deep Learning workshop, intro to pytorch
  • Attributes: J&M 15
  • Parsing: J&M 13
  • 9/23 Assignment 1 due
    Assignment 2 given: Language Models
    Earley's algorithm (ppt)
  • Top-down parsing
  • Earley's algorithm
  • Probabilistic parsing (ppt)
  • PCFG parsing
  • Dependency grammar
  • Lexicalized PCFGs
  • CCG: Steedman & Baldridge; more
  • TAG/TSG: Van Noord, Guo, Zhang 1/2/3
  • Prob. parsing: M&S 12, J&M 14
  • 9/30 Midterm recitation Midterm exam
    (5:30-6:30 in classroom)
  • Semantics: J&M 17-18; this web page, up to but not including "denotational semantics" section; try the Penn Lambda Calculator; lambda calculus for kids
  • 10/7 Semantics (ppt)
  • What is understanding?
  • Lambda terms
  • Semantic phenomena and representations
  • More semantic phenomena and representations
  • Assignment 2 due
    Distributional semantics (word embeddings)
  • Compositional semantics
  • Distributional semantics
  • Forward-backward: J&M 6
  • 10/14 Project proposal due
    Assignment 3 given: Semantics
    Sequence tagging models (ppt) (Excel spreadsheet; Viterbi version; lesson plan; video lecture)
  • Ice cream, weather, words and tags
  • Forward and backward probabilities
  • Inferring hidden states
  • Likelihood convergence
  • Local maxima
  • No class (fall break)
    10/21 Expectation Maximization (ppt)
  • Generalizing the forward-backward strategy
  • Inside-outside algorithm
  • Posterior decoding
  • Neural sequence tagging models (ppt)
  • Recurrent Neural Networks
  • Conditional Random Fields
  • Neural CRFs
  • Inside-outside and EM: John Lafferty's notes; M&S 11; relation to backprop
  • 10/28 Assignment 3 due
    Assignment 4 given: Neural Sequence Tagging
    Guest Lecture (TBD)
    Text classification (ppt)
  • Features
  • Linear Classifiers
  • 11/4 Project proposal revision (if applicable) due
    Guest Lecture (TBD)
    Neural text classification (ppt)
  • CNN
  • LSTM
  • Contextualized Embeddings
  • 11/11 Assignment 4 due
    Morphology and phonology (ppt)
  • Stemming
  • Compounds, segmentation
  • Two-level morphology
  • Punctuation
  • Rewrite rules
  • Multilinguality and machine translation (ppt)
  • Intro to MT
  • Evaluations
  • Morphology: R&S 2
  • 11/18 Sequence to sequence models (ppt)
  • RNN-based seq-to-seq models
  • Transformers
  • Applications to machine translation, summarization.
  • Current NLP tasks and competitions (ppt)
  • The NLP research community
  • Machine Translation
  • Question Answering
  • Dialog Systems
  • MT: J&M 25, M&S 13, statmt.org; tutorial (2003), workbook (1999), introductory essay (1997), technical paper (1993); tutorial (2006) focusing on more recent developments (slides, 3-hour video part 1, part 2)
  • 11/25 No class
    (Thanksgiving break)
    No class
    (Thanksgiving break)
    12/2 Project final report due
    Applied NLP continued
    Final exam recitation


    Final exam: Wed 12/11, TBD