CS 598 LAZ: Cutting-Edge Trends in Deep Learning and Recognition

Instructor: Svetlana Lazebnik  (slazebni -at- illinois.edu)
Lectures: T TH 12:30-1:45, 216 Siebel
Instructor office hours: Tuesdays 2-3PM or by appointment, 3308 Siebel

TA: Arun Mallya (amallya2 -at- illinois.edu)
TA office hours: Mondays 2-3PM, Wednesdays 3-4PM, 3340 Siebel

Always check announcements on Piazza for short-notice changes to instructor and TA office hours!

Contents: topic list, requirements, schedule, resources

Important links: lecture videos, Piazza (announcements, discussion board), Compass (submission of project deliverables, grades)

Overview

This is an advanced graduate seminar studying current research literature on trends and topics in deep learning, primarily applied to computer vision and language. Topics include state-of-the-art neural architectures and training techniques, recurrent models, neural generative models (adversarial networks and variational autoencoders), deep reinforcement learning, self-supervised learning, language and image-language models, and applications to audio and robotics. Requirements include a group presentation, a final project, literature reviews, peer grading, and participation (see details below).

Prerequisites: Equivalent of introductory courses to machine learning and computer vision, and working knowledge of standard feedforward convolutional neural networks (this course is not an introduction to deep learning).

Detailed Topic List and Readings

Requirements

Group presentation (50%)

Students will form groups of up to three and jointly develop a lecture on an assigned topic to be delivered on an assigned date. Each group member must deliver a portion of the presentation. See here for a detailed list of topics with suggested initial reading lists.
Barring truly exceptional circumstances, all group members will receive the same score for all of the above components (except signup).

Guidelines for creating a successful presentation:

  • The initial reading lists are not meant to be binding. You should feel free to modify them based on your interest and judgment. Your goal is to identify a thematically coherent group of significant papers that will make for a presentation of high educational value. You are not expected to cover every part of every paper, and your presentation typically should not be organized as a sequence of single-paper summaries. Make sure your list does not contain either too few or too many papers.

  • Think of yourselves as professors for a day. You should strive to give a comprehensive and understandable lecture introducing the class to a specific topic. It is important to find the right level of technical depth for the audience, to expose the core technical ideas without going into excessive or overwhelming detail. Some presentations will be more like tutorials and some more like surveys, but most should try to identify at least one technical "nugget" that can be taught in reasonable depth to the class.

  • When preparing your slides, and especially when merging slides provided by different group members, pay particular attention to integration. Make sure that slides covering different topics or papers have a consistent organization, level of detail, and notation. Think about the overall "story" you will tell as you are going from one paper to the next.

  • Be sure to place your topic in the context of the entire course. Whenever appropriate, take care to point out any specific connections to other presentations that came before.

  • Where appropriate, feel free to bring a critical perspective to your topic. Go beyond simply describing the techniques. Compare and contrast different papers, question assumptions, expose possible flaws and limitations, suggest alternatives and/or directions for future research. Keep in mind that some of the papers you are covering may not be peer reviewed yet, making skepticism about any extraordinary/unsubstantiated claims warranted (and exercise caution about including any such papers in the first place).

  • If at all possible or appropriate, you are highly encouraged to include a demo component. You can use code or demos from the Web or implement your own (this could even coincide with your project if your presentation comes later in the semester).

  • Be sure to involve the class. When you are developing your presentation, identify places where you can ask other students for input, or topics that you want to open up for discussion.

  • Because timing is hard to predict, you need to maintain some flexibility in terms of the topics you will cover. It is a good idea to have one or two sections in the latter half of your slides that you can skip depending on the time. When you are presenting, keep an eye on the time and adjust the pacing towards the end accordingly.

  • Use of external sources and credit attribution: Be sure to explicitly give credit whenever you use material from other sources. If you "borrow" any slides or graphics, be sure to give the original source in small font on the bottom of each slide. If you show a demo based on somebody's code, be sure to clearly announce this. Failure to follow these guidelines will hurt your score for the slides, and may even be considered an academic integrity violation. It is not acceptable to use an entire slide deck from another source "as is" as the basis for your presentation.

Group project (30%)

You are encouraged to work on the project with your presentation group, but feel free to form a different group if you want (groups should not be larger than three). The project may take the following forms:
Project deliverables (submissions on Compass):
  • Proposal (10% of project grade, due Monday, February 27th): Proposal should be uploaded to Compass in PDF format by one group member and should include: (1) names of group members; (2) a description of the proposed project in a half a page or so; (3) key references, including links to any resources you plan to use (especially code and data). Late submissions will receive no credit, but still need to be turned in in order to avoid further penalty on subsequent components of the project grade.

  • Progress update (10% of project grade, due Monday, April 3rd): A summary of your current efforts, with notes on any modifications to your original project goals. If you are writing a tutorial or survey paper, you should provide a rough draft of at least three pages. If you are doing an implementation project, at the very least, you should show evidence of successfully running baseline code (e.g., training an off-the-shelf model) on your target data.

  • Final deliverable (80% of project grade, due Monday, May 8th): Either an implementation report with results, the completed paper, or the video(s). Selected project teams will do a short presentation or demo on the last day of class.
Format for implmentation report: The final report should be submitted in PDF format by one designated group member on Compass. It should be (the equivalent of) at least six pages (single-spaced, 11 point font, 1 inch margins) and mimic the style of a research paper. It is not necessary to submit code. Here is the outline to follow for the report:
  1. Cover page: executive summary: List title and authors. Briefly summarize your problem, line of attack, and most interesting/surprising findings. Be sure to include at least one diagram or example result figure. This is not counted in the six-page minimum.
  2. Introduction: Define and motivate the problem, discuss background material or related work, and briefly summarize your approach.
  3. Details of the approach: Include any formulas, pseudocode, diagrams -- anything that is necessary to clearly explain your system and what you have done. If possible, illustrate the intermediate stages of your approach with results images.
  4. Results: Clearly describe your experimental protocols and identify any external code and datasets used. Present your quantitative evalution (if any) and show some example outputs. If you are working with videos, put example output on YouTube or some other external repository and include links in your report.
  5. Discussion and conclusions: Summarize the main insights drawn from your analysis and experiments. You can get a good project grade with mostly negative results, as long as you show evidence of extensive exploration, thoughtfully analyze the causes of your negative results, and discuss potential solutions.
  6. Statement of individual contribution: Required if there is more than one group member.
  7. References: including URLs for any external code or data used.

Peer grading reports (10%)

Each student will be assigned to grade two presentations and will have to turn in two peer grading reports (
DOC, PDF) in the course of the semester. Each peer grading report is worth 5% of your total course grade, so please take it seriously. These reports serve two purposes: to provide constructive feedback to your fellow students, and to encourage you to engage in depth in topics other than your own. Reports will be anonymous to the other students, but not to the instructor. The scores in the reports will be used to calculate the peer portion of the presentation grade for the respective team, and with rare exceptions, they will be shared with the team (but not with the class more broadly).

Reports should be submitted by email to Lana. In the subject of the email, put "mm/dd peer grading report" (where mm/dd is the date of the lecture you are reviewing). For Tuesday presentations, the reports are due by the end of Friday of the same week, and for Thursday presentations, they are due by the end of the following Sunday. Late reports will be penalized 20% (or 1% of your total course grade) for each day they are late.

Participation (10%)

You are expected to come to class most days and participate in discussions both during class and on the Piazza discussion board (a thread for questions and comments will be created for each lecture).

Schedule (in progress)

  • Presentation teams
  • Presenters: Reading lists and outlines are due by the end of Sunday before the week you are scheduled to present, followed by a practice presentation. Finalized slides are due the night before your presentation. Come to class at least five minutes early to make sure that your laptop works with the projector.
  • Peer graders: Reports for Tuesday presentations are due by email to Lana by the end of Friday, and reports for Thursday presentations are due by the end of the following Sunday.
Date Slides Reading list
January 17 Class intro N/A
January 19 CNN architectures (Lana): PPT, PDF Reading list
January 24 RNN Tutorial (Arun): PPT, PDF Reading list
January 26 RNN Tutorial Part 2 (Arun): PPT, PDF Reading list
January 31 Advanced CNN architectures (Akshay, Hong): PPT, PDF Reading list
February 2 Advanced training techniques (Prajit): PPT, PDF Reading list
February 7 Network compression, speedup (Shuochao, Yiwen, Daniel): PPT, PDF Reading list
February 9 Object detection (Jiajun, Sihao, Kevin): PPT, PDF Reading list
February 14 Semantic segmentation, dense labeling (Liwei): PPT, PDF Reading list
February 16 Similarity learning (Moitreya, Yunan): PPT, PDF Reading list
February 21 Visualization, adversarial examples (Ralf, Jyoti, Jiahui): PPT, PDF Reading list
February 23 Generative adversarial networks (Shashank, Bhargav, Binglin): PPT, PDF Reading list
February 28 Variational autoencoders (Raymond, Junting, Teck-Yian): PDF Reading list
March 2 Advanced generation methods (Ameya, Hsiao-Ching, Anand): PPT, PDF Reading list
March 7 3D + graphics (Juho, Qi): PPT, PDF Reading list
March 9 Self-supervised learning (Nate, Christian, Pratik): PPT, PDF Reading list
March 10 Intro to reinforcement learning -- bonus lecture (Unnat, Garima, Karan): PDF
10-11:30AM, SC 216
Reading list
March 14 Deep Q learning (Unnat, Garima, Karan): PPT, PDF Reading list
March 16 Deep reinforcement learning: policy gradients, planning (Tanmay, Raj, Zhizhong): PDF Reading list
March 28 Deep learning for manipulation, navigation (Tanmay, Andrey): PPT, PDF Reading list
March 30 Recurrent architectures (Abhishek, Anusri): PPT, PDF Reading list
April 4 Image captioning (Anjali, Liaonan, Ruihan): PDF Reading list
April 6 Image-text embeddings, grounding (Qing, Yang): PDF Reading list
April 11 Visual question answering (Liang-Wei, Shuai): PDF Reading list
April 13 Deep learning for NLP (Zeqiu, Quan, Dongming): PDF Reading list
April 18 Deep learning for machine translation (Yiren, Shaoshi, Yisi): PPT Reading list
April 20 Deep learning for audio (Matt, Yuchen, Chris): PPT, PDF Reading list
April 25 Architectures with memory (Nitish, Shreya): PPT, PDF Reading list
April 27 Meta-algorithms (Mariya, Safa): PDF Reading list
May 2 Project presentations: PDF

Useful Resources

Books

Courses

Guides to deep learning

Tutorials, blogs, demos