Table of Contents

• Overview
• Prerequisites
• Best Practices
• Connectivity
• Troubleshooting
• Safety
  • Pre Checklist
  • Post Checklist
• Example
• Plan Design
  • Fixed solar rows
  • Wide rows (nursery)
  • Tracked solar rows
  • Tracked solar rows with CAB
  • Exclusions in row
  • CAB and row boundaries trimming
  • Access Road Traversal
  • Open Areas
• Remote Manual
• Telemetry
• Media

Overview

This is the fifth tutorial on autonomous tasks focusing on remote operation of a LCR (anywhere in the world, not co-located with operator) through a secure internet connection. It covers:

• Safety checklists before autonomy or manual control is enabled, and checklists before connection is terminated to ensure LCR is left in a safe and secure state
• Autonomous remote operation in a structured environment (solar rows)
• Manual control remote operation (WIP)

Prerequisites

• First Use Guide
• Manual Control Guide
• Calibration Guide
• Autonomous Task Guide Level 1
• Autonomous Task Guide Level 3
• Autonomous Task Guide Level 4
• A high quality internet connection through a VPN, with low latency and stable connectivity (VPN does not reconnect often)
• Chrome Web Browser with only the specified tabs open
• Chrome Web Browser is the only active operating system application
• A focused operator without any cognitive impairment, with no distractions. You must operate in a distraction free environment during LCR remote operation

Connectivity

Gain Permissions

You must send your public IPv4 address to the software team on Slack. You can find your IPv4 address by going to: https://www.whatismyip.com/ Once you load the site, copy and paste your IPv4 and send it in Slack.

Logging In

1. Open any web browser.
2. Navigate to https://directedmachines.com/dashboard/ui/.
3. Enter the login credentials provided in your onboarding email.

Full cloud tutorial at https://github.com/directedmachines/customer-support/wiki/Fleet-Tracking-Basics

Troubleshooting

The AGT User Interface will display various warnings to aid the operator in resolving potentially unsafe plans, diagnosing issues with obstacles and robot health. Please review the AGT Troubleshooting Guide and search that page for any warnings you encounter, before contacting a software team member.

Safety

Awareness of the following is the absolute most important aspect of remote operation, before, during and after autonomy:

• LCR state
  • sensor health
  • battery state
  • actuator health
  • LCR location and orientation
• Operational environment
  • weather
  • current distance from obstacles 360 degrees around LCR
  • presence of animals or humans

The operator must use the user interface published by the robot to gain awareness of the LCR state and the environment, before enabling autonomy or doing remote manual control. As a first step, they need to create the proper window layout:

1. Using a new tab on the web browser, navigate to the Autonomous Ground Task UI
2. Click / Touch Synchronize in the Task Settings Tab. You must always synchronize robot plans before remote autonomy
3. Center the AGT browser tab in the middle of the screen
4. Click / Touch the Obstacle Avoidance Visualization icon on the bottom of the screen, to bring up a new tab, and place in a new window, on the right of the AGT tab, so both are visible
5. Operator's dashboard should similar to the image below

Best practices

1. Use the camera views (front color / depth, reverse color, wide view, sides) to gain awareness
2. use reverse view on occasion to verify mowing performance
3. Do not use when connection is poor and environment requires frequent intervention
4. go slow (task speed 15%) in plans that are not familiar or when conditions are poor (light, weather, latency)
5. ask others for help if you sense poor behavior from autonomy or any aspect of software interface
6. minimize risk
7. When mowing in a row with dense and/or heavy foliage, ensure task speed is set to 15%(or slower) and aux power is set to 65%. This will help prevent the aux motor from pushing higher amperage that could potentially damage aux motor.

Pre checklist

Search Github for open issues on LCR, checking both with LCR ID and alias. Read and interpret any issues on whether the LCR is safe to use

Using the user interface windows, the operate first should determine LCR state:

• In Manual UI, verify Turn and Translation sensitivity settings are appropriate for remote manual

  • In rows, translation sensitivity should be no more than 0.3
  • In rows, wheeled, turn sensitivity should be no more than 0.25, for tracked no more than 0.3

• Verify magnetic orientation

  • Orientation of green LCR icon is aligned with row
  • Compass indicator matches orientation

• Verify position of LCR (green arrow) in relation to satellite map and plan entities (path, waypoints, work area). It is essential the LCR is on the part of the plan you expect it to be, before initiating autonomy. In the image below, notice:

  • The LCR direction is aligned with the row,
  • It is inside a parameter domain
  • Its location is on the path or inside the work area. If the LCR is not near a WA, WP or path, you might need to carefully drive in manual mode, to get it close

• Verify color and camera views, displayed on the upper right (or center) of the AGT UI.

  • You can cycle through the color, depth, wide-view and reverse color, depth using the keyboard (see remote manual guide

• Verify views refresh about once a second

• Check the reverse direction camera, to ensure an object or animal/person is not close to the LCR. You can use the Manual Control UI (touch / click gamepad Icon, set Axle or Caster FWD as direction) then return to the AGT UI.

• Touch / Click on task settings and verify

  • Active direction selected (Casters vs Axle FWD)
  • task speed - do not exceed 20% in structured environments unless authorized; start with 15% if it is the first time
  • client timeout - if connection is poor or close supervision required set connection timeout to 5 seconds
  • Obstacle policy - in structure, it should be set to AVOID, to prevent false positive stops
  • Zero turn should be enabled, so we can link sharp turns, outside rows

• Using the Obstacle Avoidance Point Cloud visualization page, verify

  • Point cloud updates at 1 Hz
  • Expected physical structure is displayed as a collection of points on the expected side of the LCR
  • If bias is enabled, expect the yellow "obstacle" pyramid to be overlapping the implement bounding box
  • The obstacles should never intersect or be less than 0.5 meters from the LCR bounding box and/or implement box
  • A nominal view of the point cloud, with a solar panel low edge on the left side is shown below

• If the yellow "obstacle" pyramid do not appear on the point cloud visualization page, stop immediately and verify the realsense is configure properly. By checking in the troubleshooting guide under realsense will help ensure for proper parameters. Realsense Troubleshooting guide

If uncertain, have a Buddy (peer operator) verify the robot and environment look good!

• Using the Site Telemetry Overview page, select the proper fleet and plan for the LCR. View fleet telemetry to understand the location of all LCRs in the fleet and be aware of any LCRs that might block the path to the target

Before continuing, a destination must be selected, so additional information is displayed in the AGT UI.

1. Select a destination by clicking the target work area (WA) or waypoint (WP).
2. Verify the target is indicated in the text label, above the green "Start" button and is also displayed in red
3. You must ensure the correct target and vehicle orientation is selected, otherwise the LCR might attempt an in place turn and damage objects close to it.
4. Once a target is a selected, the LCR green icon will display text indicating the turn and translation speeds once autonomy starts (tx, tz)
5. If the turn speed (tx) is above 3% or below -3% do not start autonomy, seek advice.
6. If the translation speed (tz) is above task speed parameter, seek advice.
7. If bias is enabled in the domain, verify the biasMM showing in AGT matches your expectation: negative bias for structure on left, positive for structure on right (Make sure to check the orientation settings for the directions you will be working. If there are no settings for the orientation, it will default to the "any" settings").
8. In Task settings (implement width mm), or in domain settings (minOpeningWidthMM), verify the implement width is correct: For example, for a 8ft deck, use 8 * 12 * 25.4 => 2500mm

AGT UI expected display with target selected, notice (tx:-1%,tz:19%) indicates there will be a minor turning to the left to orient the robot and tz:19% translation speed.

If no warnings are displayed in the AGT UI screen, the LCR appears the proper distance from obstacles, or next to no obstacles, it is now appropriate to start autonomy.

Pay close attention once starting autonomy that LCR behaves as expected. Any sudden deviation from expected orientation, sensor refresh issues, warrants immediate stop. The LCR will self stop or pause faster than a remote human operator, for a large number of conditions The human operator must maintain a top level awareness of the LCR behavior and environment, and stop autonomy if the situation changes (for example, humans, or animals are seen in the distance Critical points that demand extra attention are row exits and entries

During Operation

If LCR autonomy is enabled make sure windows remain visible. If operator must pause operation to perform a different task or leave their computer, they must follow the Post checklist, and repeat the Pre checklist upon return.

Do not leave browser tabs with streaming camera images, it wastes cellular bandwidth and incurs costs

If weather deteriorates with heavy rain hindering cameras, stop autonomy.

Post checklist

After remote operation is complete, the operator must:

• Verify LCR is stopped and AGT is in disabled state
• LCR is not blocking access or in a dangerous position (close to moving traffic)
• Using the fleet management UI, verify LCR is reporting telemetry and its position is where expected
• Close all browser tabs for the LCR - leaving windows open incurs cellular traffic and costs per minute
• Post on slack, in appropriate channel, any notes relating to operation and open Issues as necessary.

Example

The screen capture video below showcases autonomy in a fixed solar row, with several scenarios in play:

1. autonomy with bias on low panel side (0:00 to 2:08)
2. pause and manual intervention on approach access road, with autonomy resuming, getting robot across (2:08)
3. autonomy while ignoring high grass
4. pause due to wall of weeds, manual intervention using manual and keyboard control (Up arrow, left/right arrows) (2:45)

Remote dashboard capture during autonomy

Plan Design

Note that when using https://directedtmachines.com/dashboard/ui to edit your plans, you must "Synchronize" once you return to the LCR. On the robot's AGT UI select "Edit", then "Synchronize" to import the plans from the cloud. For all plan creation, ensure to follow Plan Design: Best Practices

Fixed Solar Rows

1. For all plan creation, ensure to follow Plan Design: Best Practices
2. Parameter domains must cover all WPs and paths. All autonomy must happen within a domain
3. Use a WP on row entry and row exit, which will force the robot to stop, before exit. WP can be placed on last panel before row exit
4. Verify Task Settings are correct, implement width is accurate. For example, 84" deck should use 2100 mm for width

Examples of domain covering solar row plan, with WPs on row entries and exits

Parameter Domain Settings

• Set heading error threshold, so AGT stops when orientation of LCR is wrong
• Set Obstacle Avoidance active detection region parameters to detect structure (min and max height, X and Z distances)
• Click / Touch Commit Changes to save changes for the domain

Examples

Low side panel under mow

Use Bias (see AGT Guide 4), so the robot follows the low side panel edge, and stops if it can not see it. Assuming low side panel edge is on north side of every row, going west bound (Orientation 270) bias is positive (and small, less than 0.10 usually), east bound (Orientation 90) negative

1. West bound (Orientation 270) bias in MM, positive, to 45% of mower width (for example, +1000mm for a 2100mm wide mower)
2. East bound (Orientation 90) bias in MM, negative 45% of mower width
3. Always start with larger bias distance, do some autonomy, adjust.
4. Obstacle min height should be 600 mm and max height 1200 mm
5. Object width min should be 150 mm to filter grass and brush in row
6. Obstacle detection distance Z should be 5000 mm
7. Obstacle detection distance X should be at least 3200 mm
8. Click / Touch Commit Changes to save changes for the domain

Under panel high side

Use Bias (see AGT Guide 4), so the robot offsets from centerline towards the underside of the high side panels ( cavity where robot can fit under)

Using High Side Panels for Bias

1. Use the same plan that you are doing low side bias with.

2. Set the orientation to the opposite direction as the low side (i.e. if your lowside is 270, high side should be 90)

3. While doing under panel high side, we shift the obstacle detection height up, to look at the panels. (Settings should look similar to the image below)

4. If high side panels are on the right, then use a +1000mm bias, if high side panels are on the left, use -1000mm bias.

Unfinished row exclusion area

1. An exclusion area can be used to mark unfinished rows for future reference.
2. Name the exclusion area with the work that needs to be completed.
3. Upon completing the work, the exclusion area should be deleted.

Using high side to hold cutting for low side

1. Use the same plans as low side bias just the opposite direction
2. Change the bias to be in the order of the deck width but experimentation is needed
3. use point cloud and plan with bias on the side of the high side posts
4. set the min Object height to 1500mm and max Object height to 1900 mm
5. set object width to 155mm
6. set the max distance Z to about 8000mm
7. set the max distance X to about 4500mm
8. commit domain setting for that orientation, freeze plan
9. manually move robot so its parallel the row axis but with the mowing deck edge just under low side lip
10. use point cloud reading to determine distance to high side structure. Always have a WP selected when determining bias so domain settings are applied
11. use that reading as the bias
12. For example if heading west, low side panels will be on the right, high side on the left

Open area traverse, away from panels

The parameter domains with bias enabled can not include open areas, away from panels: bias will not work. The parameter domain shape needs to be adjusted to excluded them, and the paths modified to have AUX (mower) enabled. Give ample space from a WP to a panel edge, so robot can turn in place, during the open area traverse.

Automatic Turn Linking for Solar Rows

Proper automatic turn linking requires specific planning of the exit in-row WP, outside-row WP, in-row domain, and outside-row domain. When exiting a row, bias is applied by utilizing the rear camera. However, as an LCR gets further from the exit point, bias becomes increasingly hard to apply. Therefore, we create another parameter domain with specific settings and positioning to allow for automatic turn linking. The outside-row parameter domain uses GPS and a large active OA region to navigate to the next row's outer WP while avoiding structures. The inner and outer parameter domains overlap between the inner and outer row-exit waypoints, which indicates to the LCR that turn linking is allowed, and creates a safe environment to do so.

Where automatic turn linking is used:

At the end of long rows, typically at the east/west edge of a solar site, specific waypoints (WPs) and proper parameter domain planning allows for automatic turn linking between rows.

Rules and Conditions for automatic turn linking:

WP placement

• Inside-row WP must be centered within the row and should align with the last panel.
• Note: Measuring the space on either side of the WP to the nearest panel edge can help ensure centering
• Outside-row WP must be at least 5 meters from the row’s last solar panel. This gives an LCR room to rotate safely

Domain Placement and Settings

• The in-row parameter domain keeps the same settings, but must extend at least 2 meters from the panel edge
  • This allows the LCR to continue to use bias as it exits the row.
• The outside-row parameter domain must overlap with the in-row domain, but no more than a few meters

Note that the domain which an LCR is heading into takes priority. For example, in the images below, you can see when the LCR turns and its target position (solid red circle) moves into a new domain, the new domain parameters take over and the bias label appears.

The outside-row parameter domain settings are:

• 

Ensure under "any" orientation bias is set to 0 or disable. Also ensure that under each other orientation (0, 90, 180, 270) the bias is set to 0 or disabled as well.

All together, a plan with automatic turn linking should resemble this example of properly overlapped domain topology with centered waypoints:

Access roads with close proximity to structures

AGT plans for access road navigation next to walls, rows, and large inverters require parameter domains that are designed with a larger bias and long-distance obstacle detection enabled. These plans are configured based on site layout and direction of travel, and bias must be enabled while navigating on access and service roads that are adjacent to site structure. Unlike where there is no structure like open area navigation, here bias is appropriate, so we rely on obstacle avoidance when using AGT in access roads. When transitioning from a road or path to an open area where there is no in person obstacle, OA and bias cannot be applied, like a gap in between rows for the inverters, so we insert an exclusion area. The point cloud will simulate the object to avoid, so obstacle avoidance can keep moving down a path with a bias activated, using the exclusion area as a virtual structure.

Parameter domain settings (for fixed solar site):

• GPS enabled (minSpatialAccuracyMM NOT set, or, above 1000mm)
• maxDetectionZ at 8500 mm (we want to see far)
• maxDetectionX at 4000 mm
• min Obstacle Height at <700 mm
• max Obstacle Height at <1400 mm
• Bias set according to width of road

In Task Settings:

• Set implementWidthMM to match to mowing deck width (2200 for core deck with 2x 18" extensions for example)
• Speed never more than 20% for wheeled robots, or 25% for tracked

When designing a parameter domain, this plan would use long distance obstacle avoidance on either the right, or left, depending on direction of access roads and where the patterned structure is located relative to translation. Bias must be set based on center-line position in the roads, and the operator must find what is the proper safe distance from objects and structures. Depending on location, multiple parameter domains may be required instead of one large plan, due to changes in site layout along the roads.

Examples

Waypoints when placed and outside of the row must be positioned in the center of the road, this is crucial for safe navigation and zero point turns. Bias must be configured, however a larger gap farther away from the panels is acceptable. Use the point cloud to see the lateral distance away from both sides of the road while navigating

Exclusion zones placed around inverters. Note that the path is centered in all cardinal directions.

Parameter Domain for west fence line differs from the rest of the site, this is more an open area than it is an access road however with the exclusion zone and the panels there is enough of a road here to navigate. No bias is applied to the path since the objects are more than 3000mm

• 

Wide rows nursery

Wide nursery rows are placed as such to give the trees room to grow. These sites will ideally be using a wide mower with trimmer extensions to do a single pass per row. We rely on GPS and compass to keep us mostly centered, with obstacle avoidance correcting direction when the implement starts intersecting with obstacles.

• For all plan creation, ensure to follow Plan Design: Best Practices
• remember to set the implement width correctly in task settings, or the domain minOpenSpaceMM
• accurate placement of WPs is important

Parameter domain settings

• Axle AUX power to 60% in this case, but adjust based on the growth being mowed
• set minSpatialAccuracyMM to 1000mm (to enable GPS 2D localization while in rows)
• set heading error threshold degrees to 30
• set obstacle detection limit (Z) to 8000mm
• set obstacle detection limit (X) to 4200mm
• set min obstacle height to 820mm
• set max obstacle height to 1400mm
• ensure 4 bias settings are off or set to 0

Tracked solar rows

Tracked solar panels change their position as the sun moves through the sky posing a different set of challenges. In addition, the co-linear support posts are about 10m apart, at the limit of the depth sensor range, so when the panels are parallel to ground, default sensor settings do not detect any obstacles, when robot is centered and in between one set of posts.

Optimal time to mow is early morning and late afternoon when we can see the back side of the panels on one side, and bias of it, given a stable depth return. Depth readings on the front panel side can be corrupted due to reflections.

For all plan creation, ensure to follow Plan Design: Best Practices

Parameter domain settings

• set heading error threshold to 23 degrees, this will prevent the robot from drifting across the row
• set X detection distance to a minimum 3600mm
• set Z detection distance to a minimum 6000mm (when heading towards the open end of a row) or 9000mm (when heading towards CAB)
• set bias to match distance to back side of panels, using positive bias if back side to the right, negative if to the right
• bias should be set to half the mowing deck width + 200mm. For a 3000mm deck, use 1700mm
• set max obstacle height mm to 2300mm
• set minObstacleHeightMM to 1100mm

Tracked solar rows with CAB

Plan Design

• For all plan creation, ensure to follow Plan Design: Best Practices
• Split plans into pairs of inverter blocks, and further split into sub plans for north/south or east/west segments, to keep plans small, 30 400m rows max
• The CAB end of rows, being a dead end, has WPs that terminate there and do not connect east<-> west to their neighbors.
• At the CAB end always place an exclusion area on the other side (outside the row) of the CAB, about 2m from the CAB itself. This will allow us to get close enough using bias on panels, then stop when we detect the exclusion area
• At the open end of the rows put the WPs about 3m past the end of the panels and place them all in a parameter domain that does not have bias, sets max Z to 5m, max X to 3m and the minOpening to the size of the mowing deck (no less than 1400mm, ever). For 10ft deck with trimmers on end, set to 3000mm
• Set the speed gain to 50%, so we slow down when exiting rows
• Max Z in the primary parameter domain should be at least 4x the bias. For example, if bias is 1500mm, max Z should be 6000mm. The planner will reduce the z distance for the forward direction, when it approaches a WP that is a dead end so it can still get within 2m of the CAB

Settings for domain heading towards CAB(closed end of row)

Settings for domain heading towards open end of row

Settings for domain on open row end, linked WPs

Operational Instructions

• enter a CAB row AXLE FWD always, allows us to mow close to under CAB once we arrive to dead end
• one large domain with bias towards the backside of panels, both ways, with the sign of bias depending on the time of day

Heading towards CAB (dead end of row)

video of CAB approach and AGT stop with keyboard manual control

• Robot should be AXLE FWD heading into row, aligned
• Select WP at CAB end of row
• Start AGT
• AGT will stop short of CAB (CAB should always be visible at bottom of screen) - Make sure AXLE FWD is selected, and you are seeing the CAB at all times - using keyboard control and arrow keys, inch closer to CAB until its at bottom of front color view - select CASTERS FWD in Manual UI and verify clearance from panels on all sides - move forwards and towards the opposite side of the row - switch back to AXLE FWD and approach cab on the opposite side of the row. - Swap to CASTERS FWD and select the WP at the opposite end of the row.

Heading towards open end of row (return pass)

video of open row exit approach and AGT stop

• In AGT UI, select CASTERS FWD (reverse direction when arriving at CAB)
• select WP at open end of row, opposite CAB
• verify rear camera that we are clear of CAB
• Start AGT

Linking turn to new row ( after row exit in Casters FWD)

video of linking a turn into a new row

• stay in caster FWD mode
• select WP to west or east, start of the next row that needs works
• Start AGT
• On arrival, switch to AXLE FWD
• Click on path of the WP at end of row that you just arrived, which will automatically select the WP on CAB end of row
• Start AGT, playing close attention on robot row entry

Exclusions in row

Use exclusion areas to mark hazards in a row. Some example of dangerous obstacles that should be marked are

• Collector posts offset from solar supports
• deep ditches
• loose cabling
• construction debris

Add an exclusion area that blocks the entire row and annotate with a comment indicating what the hazard is.

Example screenshot of plan with exclusion area added mid row

Access Road Traversal

When navigating a robot through an access road, if the access road is surrounded by structures (plants, solar panels), then keep robot speed around 30%, and monitor the robot cameras closely for traffic, hazards, if the robot is getting too close to the structure, or other potential issues while navigating along the access road.

Example image of access road with structures on either side, which are marked by exclusion areas. In spaces like this where you have structures on either side, robot speed should be set around 30%.

When navigating in a more open space, however, then the speed can be increased to 40%, but the operator must pay attention to other robots and vehicles.

Example image of a more open space, since there is no obstruction in the area under normal circumstances, it is okay to set the robot speed to 40%, however keep an eye out for other operators robots, vehicles in the area, or other obstacles.

CAB and row boundaries trimming

Vegetation growing by cable above ground lines and row exits can be managed using a AGT with waypoint navigation.

• Please a path perpendicular to row exits, close enough that mowing deck underlaps solar panels, but not so close robot can collide.
• for CAB, place path parallel to CAB, so mower edge never gets closers than 6" to CAB cables
• make sure an exclusion area is placed over the CAB/solar panels
• Set a parameter domain with the following settings, so we see the CAB line/solar panels, but ignore low grass (below camera sensor)
• Use low speed (15% task speed)

Example settings and path for cleaning up row exits

Open Areas

Please review AGT Guide 2

Remote Manual

It is strongly recommended you use the keyboard to control the robot when operating it remotely. See the remote operation with a keyboard section of the Manual Control Guide. There is a delay in movement, so operate it like NASA operating a remote craft. Do one second of thrust/turn input, then see what the result is, then do another thrust input, then see what the result is.

• In the Settings for manual, pay attention to turn sensitivity: you might need to increase it to 25% (0.25) so robot responds

• ALWAYS leave OA on when doing remote manual

• If connectivity is bad, and image not updating, use the point cloud visualization, assuming it is updating at 1Hz, to very slowly rotate / move robot, observing the cloud points

Telemetry

At the end of each day of AGT, screenshot last 12 hours of telemetry and add to relevant album to show progress.

Media

https://photos.app.goo.gl/BPPaa6T3qAKhuKZv9