Sofware Architecture - ObayAlshaer/ChessMentor-ML GitHub Wiki

Chessboard-to-FEN Conversion: Architecture Decisions

This document outlines the technical decisions, workflows, and reasoning behind the chessboard-to-FEN conversion system for both Chess.com digital boards and real-life chessboards.

1. Overview

The goal is to convert a photo of a chessboard into FEN notation, which can then be used by chess engines like Stockfish. The system is designed to handle two scenarios:

  1. Chess.com Digital Boards: Screenshots of Chess.com boards.
  2. Real-Life Chessboards: Photos of physical boards taken from various angles.

2. Development Workflow

Python for Training & Prototyping

  • Why: Python has rich libraries for machine learning (TensorFlow, PyTorch) and computer vision (OpenCV).
  • How:
    • Train YOLO models using Python.
    • Prototype the pipeline (grid detection, piece classification, FEN generation).
    • Validate the system on synthetic and real-world data.

Swift for iOS Deployment

  • Why: Swift is the native language for iOS apps, ensuring optimal performance and integration with Apple frameworks (Core ML, Vision).
  • How:
    • Convert trained models (YOLO, MobileNet) to Core ML or TensorFlow Lite.
    • Reimplement critical logic (e.g., grid detection, FEN generation) in Swift.
    • Build the user interface (UI) using SwiftUI or UIKit.

3. Why Grid Detection is Necessary

FEN notation requires exact square-level precision. Even with advanced object detection (YOLO), we need to:

  • Correct Perspective: Real-life boards are rarely perfectly aligned.
  • Map Pieces to Squares: Assign detected pieces to specific squares (e.g., "e4").
  • Validate the Board: Ensure the board is valid (e.g., 8x8 squares, correct piece counts).

4. Workflow for Chess.com Digital Boards

Step 1: Detect Chessboard Grid

  • Why: Chess.com screenshots may include UI elements or be cropped.
  • How: Use OpenCV to detect the 8x8 grid. 
    ret, corners = cv2.findChessboardCorners(gray_image, (8, 8), None)

Step 2: Warp to Top-Down View

  • Why: Standardize the board for consistent piece detection.
  • How: Apply perspective transformation. 
    M = cv2.getPerspectiveTransform(src_pts, dst_pts)
warped = cv2.warpPerspective(image, M, (800, 800))

Step 3: Detect Pieces with YOLOv11s

  • Why: YOLO provides fast, accurate piece detection.
  • How: Run YOLO on the warped image. 
    detections = yolo_model.predict(warped_image)

Step 4: Map Pieces to Squares

  • Why: FEN requires piece positions relative to squares.
  • How: Calculate square centers and assign pieces. 
    square_centers = [(i * 100 + 50, j * 100 + 50) for j in range(8) for i in range(8)]
fen = ['1'] * 64
for (class_name, x, y) in detections:
    square_idx = np.argmin([np.sqrt((x - sx)**2 + (y - sy)**2) for (sx, sy) in square_centers])
    fen[square_idx] = class_name_to_fen(class_name)  # Map "white_pawn" to "P", etc.

Step 5: Generate FEN

  • Why: Convert piece positions to FEN notation.
  • How: Concatenate piece symbols row by row. 
    fen_str = '/'.join([''.join(row) for row in np.array(fen).reshape(8, 8)])

5. Workflow for Real-Life Chessboards

Step 1: Detect Chessboard Boundaries

  • Why: Real-life boards may be skewed or partially visible.
  • How: Use YOLO to detect the board’s rough boundaries.

Step 2: Refine Grid with OpenCV

  • Why: Correct perspective distortion.
  • How: Use findChessboardCorners to extract the 8x8 grid. 
    ret, corners = cv2.findChessboardCorners(gray_image, (8, 8), None)

Step 3: Warp to Top-Down View

  • Why: Standardize the board for consistent piece detection.
  • How: Apply perspective transformation. 
    M = cv2.getPerspectiveTransform(src_pts, dst_pts)
warped = cv2.warpPerspective(image, M, (800, 800))

Step 4: Detect Pieces with YOLOv11s

  • Why: YOLO handles overlapping pieces and varying lighting.
  • How: Run YOLO on the warped image. 
    detections = yolo_model.predict(warped_image)

Step 5: Map Pieces to Squares

  • Why: FEN requires piece positions relative to squares.
  • How: Calculate square centers and assign pieces. 
    square_centers = [(i * 100 + 50, j * 100 + 50) for j in range(8) for i in range(8)]
fen = ['1'] * 64
for (class_name, x, y) in detections:
    square_idx = np.argmin([np.sqrt((x - sx)**2 + (y - sy)**2) for (sx, sy) in square_centers])
    fen[square_idx] = class_name_to_fen(class_name)  # Map "white_pawn" to "P", etc.

Step 6: Generate FEN

  • Why: Convert piece positions to FEN notation.
  • How: Concatenate piece symbols row by row. 
    fen_str = '/'.join([''.join(row) for row in np.array(fen).reshape(8, 8)])

6. Mapping Piece Names to FEN Symbols

Since the training data uses piece names (e.g., white_pawn, black_king), we need to map these to FEN symbols (e.g., P, k). Here’s the mapping logic:

def class_name_to_fen(class_name):
    piece_map = {
        "white_pawn": "P", "white_rook": "R", "white_knight": "N",
        "white_bishop": "B", "white_queen": "Q", "white_king": "K",
        "black_pawn": "p", "black_rook": "r", "black_knight": "n",
        "black_bishop": "b", "black_queen": "q", "black_king": "k",
        "empty": "1"
    }
    return piece_map.get(class_name, "1")  # Default to "1" (empty) if class_name is invalid

7. Why YOLOv11s?

  • Unified Pipeline: Works for both digital and real-life boards.
  • Handles Overlaps: Detects overlapping pieces better than classification models.
  • Real-Time Performance: Processes images in <50ms on modern devices.

8. Why OpenCV?

  • Perspective Correction: Aligns skewed boards into a top-down view.
  • Robust Grid Detection: Ensures accurate square-level mapping.
  • Lightweight: Minimal overhead compared to ML-based grid detection.

9. Validation & Error Handling

Chess Rule Validation

  • Why: Ensure the FEN is valid (e.g., correct piece counts, pawn positions).
  • How: Use python-chess to validate FEN. 
    import chess
board = chess.Board(fen)
if not board.is_valid():
    raise ValueError("Invalid FEN")

Handling Non-Chessboard Photos

To ensure the app works reliably, we handle cases where the input image is not a chessboard:

1. Detection

  • Use OpenCV’s findChessboardCorners to check for an 8x8 grid.
  • Validate the aspect ratio and confidence score of the detected board.

2. Feedback

  • Display an error message (e.g., "No chessboard detected").
  • Suggest solutions (e.g., "Try taking the photo from directly above the board").

3. Fallback Mechanisms

  • Allow users to manually select chessboard corners.
  • Reject low-confidence detections and prompt the user to retake the photo.

4. Testing

  • Test with non-chessboard images (e.g., walls, tables, partial boards).
  • Measure false positive and false negative rates.

5. Example Code

def process_image(image):
    try:
        # Step 1: Detect chessboard
        ret, corners = cv2.findChessboardCorners(gray_image, (8, 8), None)
        if not ret:
            raise ValueError("Chessboard not detected")

        # Step 2: Warp to top-down view
        warped = warp_chessboard(image, corners)

        # Step 3: Detect pieces
        detections = yolo_model.predict(warped)

        # Step 4: Generate FEN
        fen = generate_fen(detections)

        # Step 5: Validate FEN
        if not validate_fen(fen):
            raise ValueError("Invalid FEN")

        return fen

    except ValueError as e:
        print(f"Error: {e}")
        return None

User Feedback (Potientaly in future updates)

  • Why: Allow users to correct errors (e.g., misclassified pieces).
  • How: Highlight ambiguous squares in the UI and provide correction options.

10. iOS Integration

Core ML for YOLO

  • Convert YOLO to Core ML for iOS deployment. 
    python export.py --weights yolov11n.pt --include coreml

OpenCV for Grid Detection

  • Use OpenCV’s iOS framework for grid detection. 
    let mat = Mat(uiImage: image)
let corners = [Point2f]()
let found = Calib3d.findChessboardCorners(mat, patternSize: Size2i(width: 8, height: 8), corners: &corners)

11. Next Steps

  1. Synthetic Data Generation: Create 1k Chess.com images with FEN labels.
  2. Train YOLO: Fine-tune on synthetic data for digital boards.
  3. Test Pipeline: Validate on real Chess.com screenshots.
  4. Expand to Real Boards: Add perspective augmentation to synthetic data.

12. Conclusion

By combining YOLO for piece detection and OpenCV for grid alignment, we achieve a robust, scalable solution for both digital and real-life chessboards. This approach balances accuracy, speed, and flexibility, ensuring the app works reliably across diverse use cases.

13. Python-to-Swift Workflow

Training & Prototyping in Python

  • Advantages:
    • Rich ecosystem for machine learning (TensorFlow, PyTorch, YOLO).
    • Rapid prototyping with OpenCV and Python-Chess.
  • Steps:
    1. Train YOLO models on synthetic and real-world data.
    2. Prototype the pipeline (grid detection, piece classification, FEN generation).
    3. Validate the system on diverse datasets.

Deployment in Swift

  • Advantages:
    • Native iOS performance and integration with Core ML, Vision, and SwiftUI.
    • Optimized for mobile devices (low latency, minimal memory usage).
  • Steps:
    1. Convert trained models to Core ML or TensorFlow Lite.
    2. Reimplement critical logic (e.g., grid detection, FEN generation) in Swift.
    3. Build the user interface (UI) using SwiftUI or UIKit.