🎞️ Optical Flow: Lucas–Kanade Method

Optical flow describes the apparent motion of objects, surfaces and edges between consecutive video frames. It can arise from object movement or camera motion, and tells us how each pixel shifts over time.

---

🌟 What Is Optical Flow?

Optical flow is the 2D vector field of pixel displacements between two frames:

• Motion of the object
• Motion of the camera

It answers: “How does each pixel move between frames?”

---

🚀 Lucas–Kanade Optical Flow (Classic & Fast)

The Lucas–Kanade method is a sparse optical-flow algorithm:

• Tracks a set of feature points (e.g. corners detected via goodFeaturesToTrack)
• Assumes that motion is small and locally consistent

---

🧠 Assumptions

1. Brightness constancy
   The intensity of a point remains constant between frames.
2. Local smoothness
   Neighbouring pixels move similarly.

---

🔢 Math Behind the Scenes (Simplified)

We solve the brightness constancy equation for each point:

Linearising over a small window leads to a least-squares system for

🧪 Step-by-Step Code: Lucas–Kanade Optical Flow

import cv2
import numpy as np
import matplotlib.pyplot as plt

# 1. Open video and read two frames
cap = cv2.VideoCapture("baby.mp4")
cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
ret, frame1 = cap.read()
prev_gray = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)

cap.set(cv2.CAP_PROP_POS_FRAMES, 20)
ret, frame2 = cap.read()
next_gray = cv2.cvtColor(frame2, cv2.COLOR_BGR2GRAY)

# 2. Detect good features (corners) to track
prev_pts = cv2.goodFeaturesToTrack(
    prev_gray,
    maxCorners=100,
    qualityLevel=0.3,
    minDistance=7,
    blockSize=7
)

# 3. Calculate optical flow (Lucas–Kanade)
next_pts, status, _ = cv2.calcOpticalFlowPyrLK(
    prev_gray, next_gray,
    prev_pts, None
)

# 4. Draw motion vectors
output = frame2.copy()
for (new, old), st in zip(next_pts, status):
    if st:
        x_new, y_new = new.ravel()
        x_old, y_old = old.ravel()
        cv2.line(output, (int(x_old), int(y_old)),
                 (int(x_new), int(y_new)), (0, 255, 0), 2)
        cv2.circle(output, (int(x_new), int(y_new)),
                   3, (0, 0, 255), -1)

# 5. Show result
plt.figure(figsize=(8, 6))
plt.imshow(cv2.cvtColor(output, cv2.COLOR_BGR2RGB))
plt.title("Lucas–Kanade Optical Flow (Frame 0 → 20)")
plt.axis("off")
plt.show()

cap.release()

✅ What You Should See:

• Red dots: where the features have moved to

• Green lines: showing the direction and distance of movement between frame 1 and frame 2

📋 What the Code Does

Step	Description
1	Load two video frames
2	Detect good features to track (corners)
3	Track their motion with cv2.calcOpticalFlowPyrLK()
4	Draw arrows from old to new positions

---

✅ Summary Table

Term	Meaning
Sparse Flow	Tracks only selected points (not all pixels)
calcOpticalFlowPyrLK	OpenCV function implementing Lucas–Kanade
goodFeaturesToTrack	Finds corners suitable for tracking
Use Case	Object tracking, motion estimation, camera movement

🎥 Farneback Optical Flow — Step-by-Step

Farneback Optical Flow computes a dense motion field—estimating a motion vector for every pixel between two frames, unlike sparse methods that track only a few keypoints.

---

🌟 What Is Farneback Optical Flow?

• Dense: produces a complete flow field (motion for all pixels)
• Comprehensive: captures detailed motion maps across the image

---

📊 When to Use It?

Method	Characteristics	Trade-off
Lucas–Kanade	Sparse (few points)	Fast, good for tracking; light load
Farneback	Dense (all pixels)	More detailed motion; higher compute cost

---

🐍 Step-by-Step Code: Farneback Optical Flow

import cv2
import numpy as np
import matplotlib.pyplot as plt

# 1. Load two frames from video
cap = cv2.VideoCapture("baby.mp4")
cap.set(cv2.CAP_PROP_POS_FRAMES, 0)
_, frame1 = cap.read()
cap.set(cv2.CAP_PROP_POS_FRAMES, 20)
_, frame2 = cap.read()
cap.release()

# 2. Convert to grayscale
gray1 = cv2.cvtColor(frame1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(frame2, cv2.COLOR_BGR2GRAY)

# 3. Compute dense optical flow
flow = cv2.calcOpticalFlowFarneback(
    gray1, gray2, None,
    pyr_scale=0.5, levels=3, winsize=15,
    iterations=3, poly_n=5, poly_sigma=1.2, flags=0
)

# 4. Convert flow to HSV for visualization
mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1])
hsv = np.zeros_like(frame1)
hsv[...,1] = 255                   # full saturation
hsv[...,0] = ang * 180 / np.pi / 2 # hue = direction
hsv[...,2] = cv2.normalize(mag, None, 0, 255, cv2.NORM_MINMAX)

rgb_flow = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)

# 5. Display result
plt.figure(figsize=(8, 6))
plt.imshow(rgb_flow)
plt.title("Farneback Dense Optical Flow (baby.mp4)")
plt.axis("off")
plt.show()

🎨 Output Interpretation

• 🌈 Hue (colour) = direction of motion
• 💡 Brightness = speed (magnitude) of motion
• Black areas = no detected motion (static background)

---

✅ Summary

Feature	Description
Dense flow	Estimates motion vectors at every pixel
Output format	2-channel flow field (dx, dy)
Visualization	Converted to HSV → RGB (hue = direction, value = speed)
Strength	High-detail motion maps and flow fields
Limitation	Slower than sparse methods (e.g. Lucas–Kanade)