Assignments

Assessment

Homework 1: Sensitive data redaction at the last mile

Public overview (PDF)

You will build an AI system for a simulated client that wants to publish troubleshooting conversations. Your system must redact private information while preserving useful technical details.

You will receive a system interface, a short client brief, labeled development examples, and a fixed inference budget. You may build any system that respects the interface and budget. The brief and examples do not fully specify what this particular client actually needs. You can ask the client up to 30 questions to clarify their requirements.

During development, the client can review up to seven versions of your system. These reviews appear on an anonymized class leaderboard that accounts for publication quality and a notion of cost. The evaluations use unseen examples from a similar distribution to the development examples. Your performance on the development examples is not graded.

You choose up to two of your submitted systems. After the deadline, we evaluate them on new, private examples. This represents deployment: the client's requirements stay the same, but the conversations may come from a different distribution than the development examples.

You will also write an informal scientific report in your own voice. A manager at the client thinks they should have just asked a frontier LLM coding agent to write a redaction script. Your report explains to the client what you learned, what building this system required, and how the client should use your recommended system.

The homework is scored out of 100, with up to 20 bonus points:

Homework 2: Recommendation and learning from user feedback

You will build a research-paper recommendation system that learns from user feedback. You will receive incomplete requirements and a (simulated) user endpoint that you can query to collect user-specific feedback for a limited number of times. You will build a procedure that decides what feedback to request and when, how to generalize from the feedback into an updated policy, and how to act on new cases after the feedback ends.

Your score will partly depend on successful completion and discussion of the design and implementation as well as your systems' performance and its improvement as more feedback becomes available, plus how the system will generalize to different feedback distributions and/or underlying LLMs.

Quizzes

The goal of the quizzes is to test mastery of the fundamental technical ideas and essential system design intuitions from the lecture and the homeworks. This is meant to complement the otherwise very flexible AI usage policies for homework and project development.

Five short quizzes will be given in class. Absences will not be granted a special exemption, but your grade will only depend on the best three quizzes. We choose this format to lower the pressure compared to a major midterm and to encourage students to stay current with the class conceptually throughout the semester.

Project

The course will provide several project directions to choose from. Students with a strong alternative may instead propose an open-ended project to the course staff.

Each team will identify a realistic gap in what current AI systems can do, define how to evaluate progress, and investigate an approach to making that progress. Your grade will reflect the quality of the problem formulation, experiments, and evidence supporting your conclusions, rather than whether your approach necessarily achieves its original goal or advances the state of the art.