Final Project

The final project will be your opportunity to take what you learned in the class and put it towards a project of your choice.

Here are some guidelines:

• Apart from rare exceptions, your problem should be directly relevant to manipulation.
• You are welcome to work on the project in small teams (2-3 people). If you plan on doing a team project, please make this clear in your proposal. Our expectations for your project will be proportional to the number of people on the team.
• The project is only over 4-5 weeks, so scope your project carefully.
• Developing novel algorithms/methods is great, but reimplementing published work and/or applying a published method to a manipulation system is also great. You will be evaluated based on whether the project forced you to understand new concepts, not based on the novelty of the approach.
• Do not underestimate the time it can take to set up a simulation. Using/modifying an existing example from the textbook is great. Using Drake is not a requirement, but the staff can help you more if you choose this route.
• Some of you will be able to do a project that is closely related to your current research. This is completely acceptable, as long as the subproblem you are solving is clearly defined. If you are using part or all of your work for research or as a part of another class, you must state this explicitly in your final report. We expect you to do something new that you would not have done if you hadn't taken this class (e.g. I learned about a new algorithm in class that might be suitable for my research, so decided to explore it for my project).
• The goal of the project is to deepen each student's mastery of some topic from class. If you do work in a similar field, and submit an incremental paper in that line of work that makes no connections to the class material, then that does not constitute a successful project.
• We encourage you to build on code and tools of the community, but remember that you must clearly distinguish your contribution and acknowledge the work of others to avoid plagiarism. If you do improve an open-source tool or example, consider contributing back!
• We've included a few project ideas below. You can find some great final projects from previous semesters here. I would love to add your work to the list!

The course staff will provide you with feedback on the scope on direction of your proposal shortly after we receive it.

Project Proposal

The project proposal is intended as a forcing function for you to crystalize a project idea. Moreover, it gives us a chance to offer you a feedback and make sure that your plan is feasible in the time allotted to the project. The preproposal should be ~500-750 words, and the final version ~1000 words. It should detail your idea for the project, and:

• Define exactly what the project deliverable is.
• Briefly describe why the project is interesting.
• List the topics that we have studied (or that we will study) in class that are covered by your project. For example: kinematics, deep/geometric perception, force control, motion planning, etc.
• Discuss any related prior work you have found that is relevant. If this project is related to your research or a project you are doing concurrently for another class, tell us about that now (and read the guidelines for the final report below)..
• Define specific goals that you expect to have accomplished before each of the progress updates.
• For all team projects, your proposal should have a few sentences describing how the work will be divided across the team. It’s ok if plans change, but we want to make sure that the initial roles are clear, and that the project makes sense for multiple people.
It does not have to be extremely polished, but it does need to provide us with enough information to understand what you are hoping to do. Otherwise, we won't be able to help you!

Here are a few example project proposals from previous years:

• Rock Skipping Robot by Michael Burgess & Nicholas Ramirez. (Graduate)
• Throwing and Catching Bot by Quincy Johnson, Hanqi Su, and Arif Dayi. (Undergraduate)

Progress Updates

We will request short (~1 paragraph) project updates some of the problem sets in order to keep us up to date with how you have progressed. These check-ins are listed on the schedule. 6.4210 students will have scheduled interactions with the CI-M staff. All students should reach out to the technical staff to discuss their projects/progress.

Final Video

The project videos will happen in the final day of class. The video must have duration of 3 minutes for individual projects, 4 minutes for two-person projects and 5 minutes for three-person projects. For group videos, please try to share the time evenly between the various members of the group. This year we will require you to upload videos on Youtube, more details will follow on Piazza.

Final Report

For your final report, you should use the IEEE Template for conference proceedings (probably the LaTeX one, unless you really enjoy using MS Word). Write a summary of what you accomplished during your project. Write it, as much as possible, like a conference paper. You should include:

• An abstract.
• An introduction of your project and why you think it is interesting.
• A related work / literature review section
• Your technical approach / methods
• Your results (partial or work-in-progress is expected, and completely fine!)
• A discussion of your results and potential next steps
• For team projects: please describe the contributions made by each team member.
• For projects related to your research or a project in another class, please clearly denote what parts of the project overlap (or not) with the other efforts.

For projects that do overlap with your research, please make connections to the topics in the class. A report that might have been written if you had not taken the class does not constitute a successful project. The goal of the project is to demonstrate some level of mastery of the material from class.

The report is worth a very large portion of your overall grade, so be as thorough as possible in demonstrating mastery of the course material. The report may range between 1500-4500 words (longer reports are expected when you are working collaboratively), not counting figures and references. Videos are encouraged as well, as a way to showcase simulation/hardware results and provide visual explanation.

Project Ideas

Here are a set of ideas we believe might be worth investigating, but do not feel the need to grab one of these! This is a placeholder, and the list will grow as we add more suggestions soon.

• Using everything we learned, implement a full-stack system (from perception, planning, and control) that can reliably solve a small-scale application in manipulation.
  • Pick-and-place clutter clearing of YCB objects.
  • Opening a cupboard and grabbing a mug.
  • Solving a peg-in-hole assembly problem.
  • Using a tool (hammer, squeegee, pencil, etc.) to achieve a task.
• Choose a specific subproblem in manipulation (impedance control, pose estimation, motion planning, task-and-motion planning) and implement / benchmark existing methods.
  • Pose Estimation: There have been impressive results that use deep learning in order to do object pose estimation, such as Pose-CNN. How do these compare with ICP-like pose estimators?
  • Motion Planning: Motion planning has a very rich literature comparing and combining sampling-based methods and optimization-based methods. Which ones are better suited for manipulation, and how should we combine them?
  • Impedance Control: There is a rich variation of approaches regarding the theme of impedance control (stiffness control, admittance control, force control, etc). Which ones are suited for which tasks? Can we benchmark their performance on a single task?
• Generative AI. There are a lot of connections between the topics we have studied in class and the recent advances in generative AI. For example, how can we use generative models to:
  • Generate random simulation scenarios (e.g. LLM => robot description format + goal formulation + text description?)
  • Generate 3D models (e.g. text to 3D)
  • LLMs/VLMs for task-level planning. (A main question here is the visual grounding of the LLM e.g. the raw camera inputs)
• Object State representation: implement and compare different representations for objects and reason about what class of tasks each representation is suited for.
• Robust (generalizable) manipulation: using computational tools from class such as Reinforcement Learning / impedance control, can you write / synthesize controllers that generalize to variations in objects such as:
  • variations in object category
  • variations in object geometry
  • variations in object mechanical properties (friction, weight, etc.)
• Try a model-based approach to a manipulation problem of your choice. Good examples are:
  • Contact-implicit Trajectory Optimization for multi-finger contact.
  • Hybrid-MPC for the pusher-slider system
• Deformable object manipulation: how would you manipulate a relatively deformable object such as a rope / shirt / pile of onions? (Note: there is limited support for soft bodies in Drake, so simulation may be a bottleneck if you decide to go this route. There are specific cases (ropes / onions) where the staff might have software availble for simulation.)