Goals

• Replace cascade classifier method with MediaPipe Holistics.
• Add gesture control to lock and unlock faces
• Use virtual camera feature to send the video feed to other software

Hardware Components

• 720p Webcam
• Arduino Uno R3
• SG90 Servo Motor x2
• Servo pan and tilt bracket

Libraries and Models

• OpenCV
• MediaPipe Holistics
• MediaPipe Hand Detection Model
• pyvirtualcamera

Gesture Control

The MediaPipe library offers built-in functionality for hand gesture recognition, making it advantageous as it supports both face and hand gesture detection within a single library. Leveraging this feature, I implemented the ability to lock the face in the frame by detecting a closed palm (fist) gesture and to release the lock with an open palm gesture.

import mediapipe
# ................................................................................................................
# ................................................................................................................
mp_hands = mp.solutions.hands # Initialize MediaPipe Hands model
hand_detector = mp_hands.Hands(min_detection_confidence=0.5, min_tracking_confidence=0.5)
# ................................................................................................................
# ................................................................................................................
def detect_hand_gesture(hand_landmarks):
    """Detects if the hand is showing an open palm (lock) or fist (unlock)."""
    if hand_landmarks:
        thumb_tip = hand_landmarks.landmark[mp_hands.HandLandmark.THUMB_TIP].y
        index_tip = hand_landmarks.landmark[mp_hands.HandLandmark.INDEX_FINGER_TIP].y
        middle_tip = hand_landmarks.landmark[mp_hands.HandLandmark.MIDDLE_FINGER_TIP].y
        ring_tip = hand_landmarks.landmark[mp_hands.HandLandmark.RING_FINGER_TIP].y
        pinky_tip = hand_landmarks.landmark[mp_hands.HandLandmark.PINKY_TIP].y

        # Open palm gesture (lock)
        if thumb_tip < index_tip and thumb_tip < middle_tip and thumb_tip < ring_tip and thumb_tip < pinky_tip:
            return "lock"
        # Closed fist gesture (unlock)
        elif index_tip < thumb_tip and middle_tip < thumb_tip and ring_tip < thumb_tip and pinky_tip < thumb_tip:
            return "unlock"
    return None
# ................................................................................................................
# ................................................................................................................
def process_video():
# ................................................................................................................
# ................................................................................................................
     hand_results = hand_detector.process(image_rgb)
# ................................................................................................................
# ................................................................................................................
        if face_locked:
                track_face(face_center_x, face_center_y)
# ................................................................................................................
# ................................................................................................................
        if hand_results.multi_hand_landmarks:
            for hand_landmarks in hand_results.multi_hand_landmarks:
                mp_drawings.draw_landmarks(image_bgr, hand_landmarks, mp_hands.HAND_CONNECTIONS)
                gesture = detect_hand_gesture(hand_landmarks)
                if gesture == "lock":
                    face_locked = True
                elif gesture == "unlock":
                    face_locked = False
# ................................................................................................................
# ................................................................................................................
process_video()

Virtual Camera

The pyvirtualcam library is used to create a virtual camera, to which frames captured by the Python script are sent. This setup requires a virtual camera driver; currently, the driver provided by OBS Studio is being utilized.

import cv2
import pyvirtualcam
import numpy as np
cap = cv2.VideoCapture(0)
cam = pyvirtualcam.Camera(width=int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), height=int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)), fps=30)
while True:
    ret, frame = cap.read()
    if not ret:
        break
    # Convert BGR to RGB
    frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    # Send the frame to the virtual camera
    cam.send(frame_rgb)
    cam.sleep_until_next_frame()
    # Show the original frame for debugging (if needed)
    cv2.imshow('Original Frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()
cv2.destroyAllWindows()

Conclusions

• MediaPipe improved the accuracy of face and hand gesture detection, enhancing application interaction.
• The pyvirtualcam library enabled real-time video broadcasting to other applications, facilitating effective face tracking.

Next Steps

• Improve servo movements further
• Improve gestures
• Implement a simple GUI