Partial sim with gesture control - terrytaylorbonn/auxdrone GitHub Wiki

(6 Feb 2024). Uses mediapipe AI to recognize hand gestures (open, closed right hand) to control the simulated copter flight. In this example holding up an open right hand (in front of usb camera) causes the sim copter to pause the mission (the CC could then take over control). A close right hand (fist) continues the mission.

image image

In script I modified for this example the above screenshot does not show the tracing (actually it does, but not long enough for you to see it). Below are examples from the unmodified script that show the tracing.

image image

For (draft) details see the first chapter in

  • Gdrive
  • Document B1-3a-1_pymavlink_mediapipe_vxx_24.xxxx.docx first chapter section "24.0206-1 guesture control (works)"

FYI: There are many mediapipe variations to use. These are also interesting.

Drawing with the finger (left) or (right) using full body gestures (this would best for controlling a copter outside, but these mediapipe examples (that I found) do not contain the recognition part, only the tracing)... maybe do this in the near future).

image image

Note that gestures are not exactly a part of my eventual goal. But I thought it would be interesting to be able to control a copter with gestures. Note that mediapipe has a full suite of AI tools.

image