Intelligent Robot Manipulation

Fall 2018

Course Description

Introduces the fundamental algorithmic approaches for creating robot systems that can autonomously manipulate physical objects in unstructured environments such as homes and warehouses. Topics include robot kinematics and trajectory generation, collision-free motion planning, sensing and perception, and planning in the presence of uncertainty.

Homework assignments will guide students through building a software stack that will enable a robotic arm to autonomously pick up objects from a cluttered scene and put them away in a kitchen cabinet. The course will make heavy use of simulation, but will also feature a small number of labs with real robotic hardware (robotic arms and RGB-D cameras).

This subject counts as an Artificial Intelligence concentration subject.

This year is special

This year’s initial offering of the class will have a restricted enrollment. The idea is to have relatively more advanced students with expertise in some of the course topics who will help us to flesh out the technical content of the course. We believe that there is a need for the tools of the class to be very good (in order to mitigate the systems engineering complexity, and let future students focus on the specific intended technical themes). Our aim is to offer the course as an advanced undergraduate / early graduate course annually thereafter.

Course Information

Class Time and Location

Fall semester
(September - December, 2018)

Tuesday, Thursday 2:30 - 4:00 pm

See class schedule for complete details

Office Hours

Mondays 4-6 pm
Room 26-322

Grading Policy

Assignments: 50%
Final Project: 50%

Late assignments will be penalized 10% every 24 hours.

No final exam.

Previous Versions


Class Forums

Forums are on Piazza (sign up with your email address).


Basic linear algebra, probability and algorithms.

Coding assignments will be in Python.

See the FAQ if you have any questions about prerequisites.

Readings and Assignments

This is a new class and there is no textbook. We will post relevant reading material refer to the schedule.

Assignments (to be released as they're assigned)

Links and due dates for assignments will be on the schedule.

Submission of Assignments
We'll be using Gradescope for problem set submission and grading. Each problem set is weighted equally. The login code for this class will be posted on Piazza -- please create an account and add yourself to this class using that code only if you are taking this class for credit. Grading will rely on submitted code passing an automated battery of tests, in combination with our review of your code and inline responses to questions. More details will be provided when assignments are released.

Collaboration Policy
Collaboration is encouraged, but the work you submit for assignments is expected to be entirely your own. That is, the writing and code must be yours, and you must fully understand everything that you hand in. Discussing the details of how to solve a problem is fine, but you must write the solution yourself. To avoid plagiarizing, you shouldn't be looking at someone else's solution while you write down your own. If you collaborated significantly (use your own discretion for "significantly") on a problem, list the people you collaborated with next to your solution.

Final Projects

The final project will be your opportunity to explore some of the topics introduced in the course more deeply; either in simulation or on real hardware.

Details about the final projects will be announced early in the semester.


I don't have any robotics experience, is it okay if I take the class?
Yes. The course is designed to not assume any prior experience with robotics. If you have plenty of robotics experience, that's great too. This year, in particular, we aim to recruit students who can contribute with their experience in some topic related to the course curriculum.

What will assignments be like?
The assignments (roughly every two weeks) will comprise a mix of hand-written math and programming.

Programming assignments will be in Python and will feature use of Drake, a toolbox for planning, control and analysis for robotics. Drake was developed out of the Robot Locomotion Group and its development is now led by Toyota Research Institute.

All assignments (both hand-written components and programming components) will be graded using Gradescope.

I am not sure if I fit the prerequisites, what should I do?

Please speak to a staff member.

What can I do to best prepare for success in the class?
The course takes a rigorous mathematical and algorithmic approach to robotics. The only required preqrequisites are basic familiarity with linear algebra. If you work hard, we can teach you the rest.

If, however, you would like to prepare as best as possible, here is what the course TAs recommend:

  • Linear Algebra:
    • A strong, intuitive understanding of Linear Algebra will very much help you with this class. If off the top of your head you don't remember what the rank of a matrix is, or how to do Singular Value Decomposition, we recommend you review your linear algebra. The entire world of robotics is rich in linear algebra -- you will not regret investing time in mastering fundamentals!
    • To brush up on linear algebra, the content and video lectures from Gilbert Strang's classic course, MIT 18.06 have helped many students in the past.
    • Another good resource is the textbook Linear Algebra Done Right by Sheldon Axler.
  • Programming in Python
    • If you are somewhat familiar with Python but would like to brush up on syntax, this tutorial from Stanford CS231n provides a good overview.
    • If you are an absolute beginner with Python, Codecademy provides a friendly introduction.
    • Although you will not need to know C++ to get an A in the class, the underlying software used for some of the assignments is in C++.
  • Mathematical Optimization
    • We will teach you what you need to know about optimization, but acquiring a background in the subject will help you deepen understanding. If you know what the following are, you are totally set: LP, QP, QCQP, MI-variants, SDP.
    • Video lectures on Convex Optimization by Stephen Boyd are recommended, as is the reference textbook Convex Optimization by Boyd and Vandenberghe.
    • Reading about SOS (Sum of Squares) optimization would also be helpful (but we will teach you this).
  • Machine Learning
    • Basic background in Machine Learning will help. 6.036 is a good introduction. There are also many great introductory classes online, Andrew Ng's is one.
  • Robotics
    • You will not need any prior robotics exposure to succeed in the class. If however you want to start absorbing fundamentals (frame transformations, manipulator equations, etc.) then Introduction to Robotics: Mechanics and Control by John Craig is a good reference.


You can contact the course staff at:


Russ Tedrake


Tomas Lozano-Perez

Teaching Assistant

Katy Muhlrad

Teaching Assistant

Tao Pang