The steps below detail adding an end effector and ensuring operability. There is a demo file in . All that should be needed to be done is download Omniverse Files then run it in a nucleus and update paths in the python scripts as well as references to the scripts in the on nucleus.

PS please send video of this working and someone attempting to grab a test object

1. Open the .usd that has your desired end effector and locate the reference .urdf in the properties panel on Isaac Sim editor.
2. Open the .urdf in your desired editor using the locate or the pencil icon.
   1. Make note of the mesh and visual file paths
3. In the .urdf locate <link name=[name first part of end effector]>
4. Copy the link code starting at the first part of the end effector all the way to the last
5. Paste in the .urdf of the desired robot. you may have to change mimic joints to revolute joints.
6. Note: The origin and roll pitch and yaw of each object within the link are relative to the location of their parent. It is not suggested to directly edit .urdf to change the orientation of the end effector. Instead find an already existing .urdf with the correct orientation and use code directly from that .urdf in the desired robot .urdf .
7. In a new scene on the top ribbon click Isaac Utils->Workflows->URDF Importer
8. You should now have a parent xform with the name of the robot and the desired end effector. Its xform may have an orange or blue arrow, these suck. So in files -> Save flattened as… .
   1. Saving as flatten makes the .usd file heavier but more reliable than working with references and payloads.

There are a few steps to ensuring a good simulation of the End effector. The test will determine the functionality of the end effector simulation. Ideally, only the first test would be needed as its effectiveness should cascade. However, even simulations aren't ideal.

1. In IsaacSim, in the to ribbon Isaac Utils -> Articulation inspector
2. Check that there are names that the robot join names are there
3. In the JointCommandArray click and hold on an input and slide left to right. The arm should wiggle.
4. Repeat for each input to ensure that everything can move correctly in the simulation.

1. Connecting to arm:

   1. ensure the arm is using the ip address of the computer hosting OVIS
   2. ensure that they have the same arbitrary port
   3. ensure that both devices are connected to the same wifi

2. Connecting to Vicon:

   1. follow set up described on
   2. Ensure that scripts that use Vicon data use the IP address of the computer connected to the Vicon system
   3. ensure your import works(python): import os import ppath import sys sys.path.instert(0, Path(file to path name ))

The First thing that needs to be done is, an object needs to be created and set up to be Vicon detectable and it should have a corresponding object in the simulation. Briefly, I will explain how to create one and import it to OVIS.

It is possible to compare the pose of the real arm with the Vicon trackers and compare that to the simulation pose. However, questions like: How accurate does the simulation need to be?

To test this, we need to ask a slightly different question: can the robot complete tasks properly? Simply put a task could be as simple as moving an object from an arbitrary point A to point B.

1. Measure real-world object dimensions

2. Design it in your preferred 3D modeling software. If you have never used one, Onshape and Shapr3d (for Ipad kids with apple pencils) are recommended thanks to their low barrier to entry and their cloud capabilities.

3. Export the object.

   1. It is easiest if you name the object before you export and have the same name and spelling across all devices. The naming convention is ET_.
   2. The easiest process I have found was to export from Shapr3D as an .stl file. In Omniverse on the top ribbon, File -> Import, there should be an .stl file type that is available. This type has seen more success in my brief use than .obj
   3. There already is an .STL of a samsung charger block <include picture, if not picture it is a standard samsung charger that I found in vis lab> that can be used. The demo scene should already have one implemented.

4. Import the object into OVIS and size as needed.

   1. There will probably be a need for some attribute editing (in the property tab of said object) to enable Vicon systems and scripts utilize the object. Follow these steps if you don't see a Rotate field in the properties tab.
      1. If the object has orient on it, right click to delete the Orient TransformOp.
      2. Repeat for any other TransformOps besides Translate and Scale.
      3. At the top of the property tab, click Add -> TransformOp -> Rotate

5. Next add reflective tracking markers to your object in whichever way you see fit just make sure that there is some non-uniformity so it's easier to discern when an object has been rotated. 1. If you are using reflective stickers it may be necessary to cut them and combine cutouts to make a surface area that is easier for the Vicon system to track. 2. If you choose to cut out stickers, try to create cuts and combine the stickers like the pictures below:

6. In the Vicon tracking software, check if the system is able to pick up your test object and all of the stickers.

7. If the Software shows your object, then create an object following the directions in the pictures below (on phone).

   1. Make sure the object name is the same as the object that you just created.
   2. The screenshots are for Vicon Tracker 3.6 from the User guide. Use the User Guide for whichever version of Vicon you are using.

8. Make note of the IP address of the computer that is hosting the Vicon tracker. The Vicon tracker will broadcast object positions.

9. When you play the scene you should be able to move the real object and see the simulated object reflect those movements.

1. Ensure Vicon test objects are created using directions above.

2. Mark a starting location point A and an ending location point B.

   1. It most likely necessary to have point A and point B Vicon visible

3. FOR OMNI ONLY ARCHITECTURE: Ensure the .usd file that you want to use is uploaded on to the nucleus. For Networking ease of uses make sure any scripts that utilize networking are the local hard drive of your computer.

4. Open Isaac Sim and USD composer.

5. In omniverse open the .usd in the nucleus and start the live sync session(include picture).

6. Test that scene works

   1. Arm should go to target cube and cube should be green.
   2. The long rectangle should have blue and purple options that open and close the gripper.
   3. The test objects and real arm objects should move so long as your computer is on the same wifi.
   4. If the physical arm is on wifi it should be reflecting the movement the simulated arm makes, but may have some weird jittery movements.
      1. vr_ur.py should be stored locally on the computer and the .usd in the nucleus should use the locally stored script. This circumvents odd networking so long as zmq imports properly.

7. In IsaacSim copy the join session link. Join the session with the drop live sync session down and copy the session link.

8. In USD composer select RTX- Interactive in the top left corner. a. Repeat the same thing in Isaac sim. b. This may not be necessary at times if using Oculus 2 with updated quest link drivers and what not. Was Unsure what the requirements for proper VR usage were. Things were finicky during testing.

9. In the USD composer in the top ribbon select Window -> Rendering->VR to open the panel to turn on the VR.

10. Wake up the headset and enable the quest link and connect to the host computer.

11. Start steamVR.

12. Select start vr in USD composer.

1. Place the test object within reach of the arm
2. Start VR
3. Start screen record either on VR headset or host computer
4. Start the digital twin (IsaacSim and USDComposer)
5. Set up a real world video recorder, make note of orientation. Analysis works best if general directions like “Left” in the simulation are the same as “Left” in the real world.
6. Grab the object from marked location A.
7. Move the object to marked location B.
8. Release the object.
9. Grab object from location B and move it to location A.
10. Save the output files

1. Pyzmq not compiled in backend See this issue:(include image, note that this was finicky on curiosity, include the github solution and note that their solution was not helpful )
   1. Lines in existing scripts (sys.path.instert)
   2. Ensure that its installed on path to python should look like path in insert
   3. Try to cycle through zmq installations
      1. Pip show zmq
      2. Pip show pyzmq
      3. Pip install –upgrade pyzmq [note that it is two -]
      4. Pip uninstall zmq
      5. Pip install zmq
   4. Ensure that these are done before upgrading past zmq 26.0.3
   5. Path too long? https://www.youtube.com/watch?v=obJmcid_erI
   6. Restart the computer
   7. Shut down the computer
   8. If all else fails: cry, report the issue to the pyzmq github, then go home and try tomorrow.
2. Moving wrong things in VR
   1. Add attribute called(has to be named exactly this, case sensitive) xr:disable_grab and make it a bool and varying.
   2. Add this to the world and to objects you want to move
   3. Make sure it is checked on parent xform
   4. Make sure it is unchecked on objects you want to interact with
3. Create the object at 0,0,0 in Vicon for smaller objects
4. Save the file with the same name and as a flatten to get rid of references
   1. Blue arrow on xform means payload
   2. I means instanceable
   3. Orange arrow on xform means reference
   4. Both arrows suck, save as flatten
   5. Source: https://forums.developer.nvidia.com/t/list-of-icons/202180/6
5. Act of god or otherwise unmanageable issue
   1. cry
   2. Restart
6. VR not connecting
   1. Restart headset
   2. Connect to wifi
   3. Enable quest link
   4. Restart steamvr then connect to computer
   5. Start vr in USD composer

EE Name	Functions	Imports	Different Modes
myCobot_gripper_ag	mc = MyArm('/dev/ttyAMA0', 115200)	From pymycobot.myarm import MyArm	Gripper type: 1. Adapt grip 2. 5 finger hand 3. // gripper 4. Flex grip
	mc.set_gripper_state(state, speed, gripper type)	See above	State: Open - 0 Close - 1 Release - 254
	mc.get_gripper_value(1)	see above	Sets gripper type, see information in second row
	mc.set_gripper_value(arm rotation, speed, gripper type)	See above	Arm rotation range: [0, 100] Speed Range: [0, 100]
	mc.set_gripper_calibration()	See above	description TBA
	mc.is_gripper_moving()?	See above	description TBA
Suction cup	Function	import	Modes