03.06 preset projects - advantech-EdgeAI/edge_agent GitHub Wiki

The Opened Door Detection project is a comprehensive solution designed to monitor a door's status (identifying if it is in an opened or closed state), trigger alarms if the door remains open for an extended period, capture images of individuals passing through an opened door, and subsequently analyze these images for specific clothing attributes. This project utilizes a Vision Language Model (VLM) for status detection, an alert system for notifications, image capture logic, and a multimodal database (NanoDB) for attire analysis, making it suitable for various monitoring and light security applications.

To run this project, load the preset named:

• DOOR_ALERT_WEBRTC_ADVAN

Note:

• Ensure the Edge Agent is running and accessible via your web browser.
• The demonstration video file (Door_advan.mkv) must be located in the /ssd/jetson-containers/data/videos/ directory on your Jetson device.
• The Save_Pics node will attempt to save images to a folder (default is /data/nanodb/images). You need to ensure this path is accessible and writable from within the Docker container. Typically, this path within the container might map to a host directory like /ssd/jetson-containers/data/nanodb/images. Ensure the base nanodb/images directories are created if they don't exist. Images will be stored in a subfolder named with the current date under this path.
• The nanodb folder (from pre_install) should be placed into /ssd/jetson-containers/data as per general setup instructions.

Figure 3.5 — Pipeline overview

This project integrates two main processing paths:

1. Door Status Detection and Alerting:

   • VideoSource: Provides the video input.
   • RateLimit_1: Controls frame rate for VLM processing.
   • AutoPrompt_ICL: Prompts the VLM about the door's status.
   • VILA1.5-3b (NanoLLM_ICL Node): Analyzes the image to determine if the door is open or closed.
   • VideoOverlay: Displays VLM output on the video.
   • VideoOutput: Shows the final video.
   • Two_Steps_Alert: Manages alert logic based on how long the door is in an opened state.
   • PiperTTS module (Preset): Provides voice alerts.

2. Image Capture and Attire Analysis (when door is opened):

   • VideoSource: (Shared from above)
   • RateLimit_2: Controls frame rate for image capture.
   • AutoPrompt_ICL_1: Prepares image and text data for the Save_Pics node. It receives text input from the VILA1.5-3b node's final output.
   • Save_Pics: Captures and saves images if the door is detected as "open."
   • NanoDB_Fashion: Builds a searchable database from the captured images for attire analysis.

Data Flow Example: Path 1: VideoSource -> RateLimit_1 -> AutoPrompt_ICL -> VILA1.5-3b. The VLM's (VILA1.5-3b) partial output goes to VideoOverlay, and its final output goes to Two_Steps_Alert and also to AutoPrompt_ICL_1 (in Path 2). Two_Steps_Alert triggers PiperTTS if the door remains opened. VideoOverlay (with input from VideoSource as well) goes to VideoOutput. Path 2: VideoSource -> RateLimit_2 -> AutoPrompt_ICL_1. If the VLM output fed to AutoPrompt_ICL_1 indicates the door is "open," Save_Pics is triggered to store images. These images are then available to NanoDB_Fashion.

Key nodes for customization include those for VLM prompting, alert logic, image saving, and database search.

• AutoPrompt_ICL (for VLM door status):
  • Template: <reset><image>Check the current status of the door. Is it open or closed?
  • seq_replace_mode: true.
  • Roi: false.
• VILA1.5-3b (NanoLLM_ICL Node):
  • Model Selection: Efficient-Large-Model/VILA1.5-3b.
  • API Selection: MLC.
  • Quantization Setting: q4f16_ft.
  • MaxContextLength: 256.
  • Chat Template: llava-v1.
  • System Prompt: "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions."
• Two_Steps_Alert Node:
  • First Check: Default 5 seconds (initial time to check if the door is in an opened state).
  • Second Check: Default 1 second (re-evaluation time if the door remains opened).
  • Alert: true.
  • Alert Keyword: "open".
  • Normal Keyword: "close".
  • Warning Message Text: "Warning, The door is opened." (period at the end).
• AutoPrompt_ICL_1 (for Save_Pics):
  • Template: <text><image> (combines text from VLM output and current image).
  • seq_replace_mode: false.
• Save_Pics Node:
  • Check Time: Default 1 second (system starts capturing images 1 second after door detected as "open").
  • Save Frames: Default 3 frames (number of images to capture).
  • Save Pics Folder: Default /data/nanodb/images (verify path accessibility and writability).
  • Alert: true.
  • Alert Keyword: "open".
• NanoDB_Fashion Node:
  • Path: Default /data/nanodb/images (where saved images are and DB is built).
  • Model: Default patrickjohncyh/fashion-clip (embedding model for attire).
  • Top K: Default 16 (number of search results).
  • Crop: enabled (recommended for CLIP).
  • Search Time: For time-based searches, format YYYY/MM/DD (e.g., 2024/10/25).

1. Launch the Edge Agent UI in your browser.
2. Load the DOOR_ALERT_WEBRTC_ADVAN preset:
   • Click the "Agent" menu in the top-right corner.
   • Select "Load."
   • Choose DOOR_ALERT_WEBRTC_ADVAN.json from the list.
3. The pipeline will appear in the Node Editor.
4. Verify the VideoSource input path (/data/videos/Door_advan.mkv).
5. Ensure the Save Pics Folder in the Save_Pics node and the Path in the NanoDB_Fashion node are correctly set and the directory is writable.
6. The project should start running automatically.
7. Observe the VideoOutput panel:
   • You will see the video playing.
   • The VideoOverlay will display the VLM's determination of the door status (e.g., "The door is open.").
8. Listen for audio alerts: If the Two_Steps_Alert node determines the door has been in an opened state for too long, the PiperTTS module will announce the warning.
9. If the door is detected as being in an "open" state, the Save_Pics node will capture a few frames and save them.
10. Open the grid widget for the NanoDB_Fashion node:
    • After images are saved, NanoDB will process them. You might need to trigger a database build or ensure it auto-updates.
    • Use the search bar in the NanoDB_Fashion UI to enter keywords for clothing attributes (e.g., "blue shirt," "white shoes").
    • The panel will display images from the captured set that match your search query, ranked by similarity.

• Visual Output: The VideoOutput will show the door and overlay text indicating its current status (opened/closed) as determined by the VLM.
• Audio Output: If the door remains in an opened state according to the Two_Steps_Alert logic, a voice alert "Warning, The door is opened." will be played.
• Image Capture: When the VLM output indicates the door is "open," the Save_Pics node will save a configured number of image frames to the specified folder (e.g., /data/nanodb/images/<current_date>/).
• Attire Search: The NanoDB_Fashion node will create a database from these saved images. Users can then open its UI panel, input text queries (e.g., "person in blue"), and view matching images of individuals who passed through the door when it was opened.

• Incorrect Door Status:
  • Check the prompt in the AutoPrompt_ICL node.
  • Ensure the VILA1.5-3b model is loaded correctly and its parameters are as specified.
• Alerts Not Triggering / Triggering Incorrectly:
  • Verify Alert Keyword ("open") and Normal Keyword ("close") in the Two_Steps_Alert node.
  • Adjust First Check and Second Check timings if needed.
• Images Not Saving:
  • Confirm the Alert Keyword in Save_Pics is "open."
  • Ensure the Save Pics Folder path is correct, accessible, and writable by the Edge Agent container. Check directory permissions on the host system for the mapped volume.
  • Verify that the AutoPrompt_ICL_1 node is correctly receiving the "open" status from the VLM's output.
• NanoDB Search Not Working:
  • Ensure images are being saved correctly by Save_Pics into the directory monitored by NanoDB_Fashion.
  • Check the Path and embedding Model settings in NanoDB_Fashion.
  • Allow time for NanoDB to index new images.
• No Audio Alerts:
  • Check the connection from Two_Steps_Alert to the PiperTTS module.
  • Verify system audio.

⬆️ Top

---