Date: 02.06.2025 - digital-twin-autonomous-farmbot/digital_twin GitHub Wiki

Work Report: Plant Height Estimation with Stereo Vision and AI Bounding Box

Summary: The workflow for estimating plant height using stereo vision and AI-based object detection was implemented and debugged in the digital_twin_plants project. The process combines stereo image acquisition, disparity/depth calculation, and robust plant localization using bounding boxes from an AI model.

Steps performed:

  1. Image Acquisition:

    • Stereo images were captured using two cameras (one normal, one AI-enabled).
    • The AI camera also performed object detection (MobileNet SSD v2) and saved bounding box results to a text file.

  2. Bounding Box Extraction and Scaling:

    • The bounding box was extracted from the AI detection output file (bbox_plant01.txt).
    • The bounding box coordinates (originally based on the AI camera's resolution) were scaled to match the resolution of the depth map.

  3. Disparity and Depth Calculation:

    • StereoSGBM was used to compute the disparity map from the stereo image pair.
    • The Q matrix from calibration was used to reproject the disparity map into a 3D point cloud (depth map).

  4. Plant Height Estimation:

    • The region of interest (ROI) for the plant was defined using the scaled bounding box.
    • Only valid depth values (finite and >0) within the ROI were considered.
    • The plant height was estimated as the difference between the median depth of the top and bottom bands of the ROI.

  5. Debugging and Validation:

    • Extensive debug outputs were added to check the bounding box scaling, ROI size, and number of valid depth values.
    • The script was adjusted to handle cases where the bounding box was outside the image or too small.
    • Visualization of the disparity and depth maps with the bounding box overlay was implemented for validation.

  6. Results:

    • The estimated plant height and relevant parameters were saved to a YAML file.
    • The workflow is now robust against bounding box mismatches and missing depth values.

Next Steps / Recommendations:

  • Further improve AI detection accuracy for better bounding box placement.
  • Experiment with stereo parameters for improved disparity quality.
  • Automate batch processing for multiple plant samples.