In assignment 6, you will be developing a miniature version of a programming by demonstration (PbD) system. This system captures the essence of how most industrial robots are programmed today. In PbD, you program the robot to do a task by moving its arm to a sequence of poses. The system saves the sequence of poses as an action and can execute the action again at a later time.

An extra feature of the system you will be developing is that instead of saving poses in the base frame, your system will provide the option to specify poses as offsets relative to an AR tag. By attaching an AR tag to an object, the robot can locate the object and execute an action like grasping the object.

You should be able to use your PbD system to develop your final projects, so you will want to invest some time and do a good job with this assignment.

1. Kuri starts in a room with a person holding an AR Tag.
2. The user presses the "Create program" button on an interface.
3. Kuri starts looking at the AR Tag.
4. The user moves Kuri (physically) to a position and presses a "Save pose" button.
5. The user presses the "Blink" button
6. The user enters a color and presses "Toggle LEDs"
7. The user moves Kuri to a different position and presses the "Save pose" button.
8. The user presses the "Save program" button, and the program is saved to a file. You will need to create a data structure to represent the program.

1. A separate application loads the program from disk and executes it.
2. The user rearranges the scene slightly and runs the program again.

You may want to revisit Lab 24: Transform arithmetic. Let's say you have ^BT_{tag1, t1} and ^BT_{wrist, t1} at t=1. At runtime (t=2), the tag tag1 is in a slightly different pose, given as ^BT_{tag1, t2}. Where should the wrist go at time=2?

Hints:

• The programmed offset between the wrist and tag is ^{tag1, t1}T_{wrist, t1}
• ^AT_B is the matrix inverse of ^BT_A
• You want to compute ^BT_{wrist, t2}

So, you will need to chain the transformations together and invert some of the matrices.

You will want to test your system in simulation first, because otherwise the robot will be too busy.

You should save a few point clouds of the table with AR tags on it and hallucinate them in simulation as explained in the previous labs.