detail_telop_schema - dingdongdengdong/astra_ws GitHub Wiki

The Teleopoperator class in teleoperator.py processes incoming commands and sensor data (like hand tracking information received from the web client) and translates them into actions for a robot. It also manages the state of the teleoperation (e.g., mode, precision, lift distance).

Here's how data from the Teleopoperator is structured and sent to the web client:

1. Data Sent from Teleopoperator to the Web Client:

The primary mechanism for Teleopoperator to send data to the web client is through the WebServer's control_datachannel_log method. This method sends string-based messages over a WebRTC data channel named "control" to the connected web browser.

The types of data Teleopoperator sends are primarily:

  • Status Updates and Log Messages: These are textual messages informing the web client about the current state or significant events.
    • Teleoperation Mode Changes: e.g., "Teleop Mode: Base", "Teleop Mode: Arm (Percise)".
    • Lift Distance Updates: e.g., "Lift Distance: 0.800".
    • Gripper Lock Status: e.g., "Left Gripper Lock: Locked, release your pedal to unlock", "Right Gripper Lock: Unlocked".
    • Command Echoes/Acknowledgements: e.g., "Cmd: reset", "Reset event", "Done event".
    • Warnings or Errors: e.g., "Lift Over Limit". The error_cb method also uses webserver.control_datachannel_log to send error messages.
    • Connection/Initialization Status: While hand_cb checks for Tcamgoal_last and Tscam being initialized and logs messages if they are not, these specific logs are raised as exceptions before they would be sent through control_datachannel_log in that part of the code. However, other initialization messages or waits, like logger.info(f"Waiting for new Tcamgoal_last {side}") in reset_Tscam, are logged server-side but not explicitly sent to the client via this channel in that function.

Schema of Data Sent via control_datachannel_log:

The data is sent as a simple string. The web client would receive these strings and typically display them in a log area or use them to update status indicators.

  • Format: JSON-encoded string (as WebServer's control_datachannel.send takes a string, and json.dumps is used elsewhere for sending). The Teleopoperator passes Python strings directly to control_datachannel_log. The WebServer wraps this in json.dumps() before sending.
  • Content Example (as received by JavaScript on the client):
    • "Teleop Mode: Arm (Percise)"
    • "Lift Distance: 0.750"
    • "Cmd: teleop_mode_arm_with_reset"
    • "Left Gripper Lock: Locked, release your pedal to unlock"

2. "Streaming" Data vs. Discrete Messages:

  • Video Streaming (Handled by WebServer): The WebServer class itself is responsible for streaming video from cameras (head, wrist_left, wrist_right) to the web client. It uses FeedableVideoStreamTrack and can be fed by functions like feed_webserver which capture video from devices. The Teleopoperator influences the head camera's view by publishing head tilt commands (self.on_pub_head), but it does not handle the video frame data transfer itself.

  • Teleoperator Data (Discrete Messages): The data sent by Teleopoperator (as described above) consists of discrete messages triggered by events or state changes. It's not a continuous high-frequency stream of data like robot joint positions or raw sensor readings from the robot's internal state.

3. How Data is Sent to the Web:

  1. The Teleopoperator logic determines a message needs to be sent to the web client (e.g., mode change, lift update).
  2. It calls self.webserver.control_datachannel_log("Your message here").
  3. The WebServer instance's control_datachannel_log method takes this message.
  4. This method, if the "control" RTCDataChannel is active, JSON-encodes the message and sends it using self.datachannel["control"].send().
  5. The web client's JavaScript, which established the WebRTC connection and data channel, receives this message and can act upon it (e.g., display it).

If you wish to stream other teleoperator data (e.g., continuous robot joint states, end-effector poses from the robot's perspective):

The current teleoperator.py is not designed to stream this kind of data directly to the web client.

  • It calculates goal poses (Tsgoal) and sends them to the robot control system via self.on_pub_goal.
  • To send such data to the web, you would typically need to:
    1. Obtain this continuous data within the Python environment (e.g., from the robot control system).
    2. Modify Teleopoperator or WebServer to handle this new data stream.
    3. Potentially create a new RTCDataChannel in WebServer for this specific data or decide on a structured format (e.g., JSON objects) to send over the existing "control" channel at a higher frequency if appropriate.
    4. Implement corresponding JavaScript logic in the web client to receive, parse, and visualize this streamed data.

In summary, Teleopoperator sends textual status and log messages to the web client via the WebServer's "control" data channel. True video streaming is handled by the WebServer directly, not by Teleopoperator.