03.08 preset projects - advantech-EdgeAI/edge_agent GitHub Wiki

The OpenWord FineTune project provides a method to enhance the detection accuracy of OpenWord models (such as YOLOWorldV2_V8X or owlv2) when their default performance is not optimal for specific objects. By using a small amount of user-annotated data, this process allows for additional training (fine-tuning) to improve the model's ability to recognize particular items, such as safety vests, more reliably.

This project involves a model training process using the OpenWord_FineTune node. There isn't a single preset to load for the entire end-to-end fine-tuning and validation workflow; rather, you'll use specific nodes and external tools.

Software & Setup:

• Ensure the Edge Agent is running and accessible.
• LabelMe Annotation Tool: You need to install LabelMe for annotating your custom dataset. The document provides installation instructions for MIC-733AO, which involve setting up a Conda environment. Refer to the official LabelMe GitHub page (https://github.com/wkentaro/labelme) for general installation if needed.

Data Requirements:

• Image Dataset: Collect several images containing the object(s) you want to improve detection for (e.g., images of people wearing or holding safety vests). Store these images in a dedicated folder on your Jetson device, accessible by the Edge Agent (e.g., /ssd/jetson-containers/data/dataset/safety_vest/).
• Annotated Data: Using LabelMe, annotate your images by drawing bounding boxes around the target objects and assigning them consistent labels (e.g., "safety vest"). Save the annotations; LabelMe will create a JSON file for each image in the same directory.
• OpenWord Configuration JSON: You need a base OpenWord JSON configuration file (e.g., /opt/NanoLLM/nanoowl/owl_camera.json). This file must include all the class names (labels) that you have annotated and intend to fine-tune.

The process involves three main stages:

1. Data Preparation: Collecting and annotating your custom image dataset using LabelMe.
2. Model Fine-Tuning: Using the OpenWord_FineTune node in Edge Agent to train a base OpenWord model with your annotated data.
3. Validation: Testing the newly fine-tuned model using the OpenWord_detector node to assess performance improvements.

Key Node for Fine-Tuning:

• OpenWord_FineTune: This specialized node, found in the "Tools" section of the Edge Agent toolbar, manages the fine-tuning process.

Configuration for the fine-tuning process is done within the OpenWord_FineTune node's settings in the Edge Agent UI:

• OpenWord_FineTune Node Settings:
  • Labelme Save Json Path: Set this to the directory path where your annotated images and their corresponding LabelMe JSON files are stored (e.g., /ssd/jetson-containers/data/dataset/safety_vest if your data is in a folder named safety_vest prepared as per prerequisites).
  • Json Path: The path to the OpenWord model's prediction configuration JSON file (e.g., /opt/NanoLLM/nanoowl/owl_camera.json). This file must list all object classes you've annotated for fine-tuning.
  • Pretrained Model: Select the base OpenWord model you wish to fine-tune (e.g., YOLOWorldV2_V8X or YoloWorld/yoloworldv2_v8x as cited in the document as an option, though YOLOWorld is mentioned in the parameter example).
  • Epochs: The number of training epochs. The default is 40. You can increase this if the fine-tuned model's performance is still unsatisfactory after initial training.
  • Model Name: Provide a unique name for your newly fine-tuned model (e.g., safety_vest_custom_ft). This name will be used to select the model later in the OpenWord_detector node for validation.

Phase 1: Data Preparation & Annotation

1. Collect a diverse set of images containing the objects you want the OpenWord model to detect better.
2. Install LabelMe if you haven't already. On MIC-733AO, follow the Conda environment setup:
   • wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh
   • bash Miniconda3-latest-Linux-aarch64.sh
   • echo 'export PATH=~/miniconda3/bin:\$PATH' >> ~/.bashrc
   • source ~/.bashrc
   • conda init
   • Close and reopen terminal.
   • conda create -n labelme python=3.8 -y
   • conda activate labelme
   • conda install -c conda-forge pyqt -y
   • conda install -c conda-forge labelme -y
3. Launch LabelMe: conda activate labelme then labelme.
4. Open your image directory in LabelMe ("File" -> "Open Dir").
5. For each image, right-click and select "Create Rectangle" to draw bounding boxes around your target objects.
6. Assign a consistent label (class name, e.g., "safety vest") to each bounding box. This label must be present in the OpenWord JSON configuration file used for fine-tuning.
7. Save the annotations for each image. LabelMe will create a .json file corresponding to each image file. Ensure these JSON files are in the same directory as the images.

Phase 2: Model Fine-Tuning with Edge Agent

1. Launch the Edge Agent UI.
2. Add the OpenWord_FineTune node from the "Tools" section to the Node Editor. (It's recommended to use a clean pipeline for this node).
3. Click the OpenWord_FineTune node to open its settings.
4. Configure all parameters as described in section "3.8.3 Key Node Configurations & Customization" above, ensuring paths and model names are correct.
5. Once configured, the fine-tuning process should start automatically, or you may need to activate the node.
6. Monitor the training progress:
   • The top-right corner of the Edge Agent screen may indicate training activity.
   • The Terminal panel at the bottom of the UI will show a progress bar and training time.

Phase 3: Validation of the Fine-Tuned Model

1. Once model training is complete, create a new inference pipeline (or modify an existing one) for validation: VideoSource -> OpenWord_detector -> VideoOutput.
2. Configure the VideoSource node (e.g., use your camera /dev/video0 or a test video).
3. Configure the OpenWord_detector node:
   • Model: Select your newly fine-tuned model. It will appear in the model list, likely under a path reflecting its base type and the Model Name you provided (e.g., YoloWorld/yoloworld_ft/safety_vest_custom_ft if your model name was safety_vest_custom_ft and base was YoloWorld).
   • Json Path: Use the same OpenWord JSON configuration file you used for fine-tuning (e.g., /opt/NanoLLM/nanoowl/owl_camera.json).
   • Activate: Set to True.
4. Run this pipeline and observe the VideoOutput panel.
5. Compare the detection performance of your fine-tuned model on the target objects against the original base model's performance (you might set up a parallel pipeline with the original model for direct comparison if desired).

• During Fine-Tuning:
  • The Edge Agent UI (terminal panel) will show training progress, including epochs and time.
  • Upon completion, a new fine-tuned model will be saved, accessible by the Model Name you specified.
• During Validation:
  • The OpenWord_detector node using your fine-tuned model should exhibit improved detection accuracy and reliability for the objects you annotated and trained it on (e.g., more consistent detection of "safety vests" with fewer false positives or negatives compared to the original model). The VideoOutput will visually demonstrate this improved performance.

• LabelMe Installation Issues: Carefully follow the Conda environment setup steps provided in the document. Ensure all dependencies are correctly installed.
• Fine-Tuning Not Starting / Errors:
  • Paths: Double-check all paths in the OpenWord_FineTune node settings: Labelme Save Json Path and Json Path. Ensure they are correct and accessible from within the Edge Agent container.
  • Annotation Consistency: Ensure the labels used during LabelMe annotation exactly match the class names listed in your OpenWord configuration JSON file.
  • Data Quality: Ensure your annotated dataset is of good quality, with accurate bounding boxes and sufficient examples.
• Model Performance Not Improving Significantly:
  • Increase Epochs: Try increasing the number of Epochs in the OpenWord_FineTune node settings and retrain.
  • More Data: Collect and annotate more diverse images for your target objects.
  • Annotation Quality: Review your annotations for accuracy and consistency.
• Fine-Tuned Model Not Found in OpenWord_detector:
  • Verify the Model Name you specified during fine-tuning. The model should appear in the dropdown list of the OpenWord_detector node, often prefixed by its model type (e.g., YoloWorld/yoloworld_ft/<your_model_name>).

⬆️ Top

---