Isaac Sim

• Reference: https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_workstation.html

• It is recommended to install through Omniverse. (Install Cache, Isaac Sim, and Kit)

• "Pip" installation is not working on Ubuntu 20.04 due to the GLIBC version not matching.

• In the documentation, Isaac Lab is tested with Isaac Sim 4.1.0.

Install NVIDIA Driver, CUDA

Verify the Isaac Sim installation

• Environmental variables: Input the following lines in the bash file:

# Isaac Sim root directory
export ISAACSIM_PATH="${HOME}/.local/share/ov/pkg/isaac-sim-4.1.0"
# Isaac Sim python executable
export ISAACSIM_PYTHON_EXE="${ISAACSIM_PATH}/python.sh"

• Check that the simulator runs as expected:

source ~/.bashrc
# note: you can pass the argument "--help" to see all arguments possible.
${ISAACSIM_PATH}/isaac-sim.sh

• Check that the simulator runs from a standalone python script:

# checks that python path is set correctly
${ISAACSIM_PYTHON_EXE} -c "print('Isaac Sim configuration is now complete.')"
# checks that Isaac Sim can be launched from python
${ISAACSIM_PYTHON_EXE} ${ISAACSIM_PATH}/standalone_examples/api/omni.isaac.core/add_cubes.py
${ISAACSIM_PYTHON_EXE} ${ISAACSIM_PATH}/standalone_examples/api/omni.isaac.franka/follow_target_with_rmpflow.py

Setup Anaconda environment (not necessary)

• Create environment

cd ${ISAACSIM_PATH}
conda remove --name isaac-sim --all
conda env create -f environment.yml
conda activate isaac-sim
source setup_conda_env.sh

Install PyTorch (deprecated)

(testing) conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch-nightly -c nvidia
pip install torch==2.2.2 --index-url https://download.pytorch.org/whl/cu121

• Verify PyTorch

import torch
print(torch.cuda.is_available())
print(torch.__version__)

VS Code Setting (Option 1)

• Reference: https://docs.omniverse.nvidia.com/isaacsim/latest/manual_standalone_python.html#python-environment
• Reference: https://github.com/knowledge-intelligence/KIMe-Tutorials/tree/main/IsaacSim
• File--Open Folder: Select "~/.local/share/ov/pkg/isaac-sim-4.1.0"
• Command Palette--"interpreter"--Enter interpreter path--Find--Select "${ISAACSIM_PATH}/python.sh"
• .vscode-launch.json: add environmental variables:

"env": {
    "RESOURCE_NAME": "IsaacSim",
    "PYTHONPATH": "${workspaceFolder}/vsprojects:${workspaceFolder}/lecture"
},

• Terminal--Run Task--"setup_python_env"--"Continue without scanning the task output.
• Check ".standalone_examples.env" file is created in the ".vscode" folder.
• Open ${ISAACSIM_PATH}/standalone_examples/api/omni.isaac.franka/follow_target_with_rmpflow.py and start debugging (Ctrk+F5 or F5)

VS Code Setting (Option 2)

• It works for only Isaac Sim >= 4.1.0
• It works for any python script outside the project folder.
• Reference: https://docs.omniverse.nvidia.com/isaacsim/latest/advanced_tutorials/tutorial_advanced_code_editors.html#running-python-code-from-external-editors
• Launch Isaac Sim 4.1.0, Menu--Window--Extensions
• Search "vscode" and enable omni.isaac.vscode (AUTOLOAD enabled)
• Menu--Window--VS Code
• In the VS Code, add extension "Isaac Sim VS Code Edition"
• Open it on your left side bar, and run

Isaac Lab

• Reference: https://isaac-sim.github.io/IsaacLab/main/source/setup/installation/pip_installation.html

• Clone

git clone https://github.com/isaac-sim/IsaacLab.git

• Create Isaac Sim Symbolic Link

cd IsaacLab
ln -s /home/minho/.local/share/ov/pkg/isaac-sim-4.1.0 _isaac_sim

• Setting up the conda environment (optional)

conda remove --name minho --all
conda create -n minho python=3.10
./isaaclab.sh --conda minho

• Installation

If you choose to install in your conda env, you should conda activate minho before execuding the following lines.

Installing Isaac Lab in the conda env causes the import issue when Isaac-Sim launching by Omniverse, since the Omniverse launcher uses the basic env. (w/o conda).

sudo apt install cmake build-essential
conda activate minho
./isaaclab.sh --install

Verify the Isaac Lab installation

# Option 1: Using the isaaclab.sh executable
# note: this works for both the bundled python and the virtual environment
./isaaclab.sh -p source/standalone/tutorials/00_sim/create_empty.py

# Option 2: Using python in your virtual environment
python source/standalone/tutorials/00_sim/create_empty.py

Orbit Surgical

• Reference: https://github.com/orbit-surgical/orbit-surgical

• Define the following environment variable to specify the path to your Isaac Lab installation directory:

# Set the IsaacLab_PATH environment variable to point to your Isaac Lab installation directory
export IsaacLab_PATH="${HOME}/IsaacLab"

• Clone ORBIT-Surgical repository to a directory outside the Isaac Lab installation directory:

cd ~
git clone https://github.com/orbit-surgical/orbit-surgical

• Install ORBIT-Surgical

conda activate minho
cd orbit-surgical/
./orbitsurgical.sh

Set up IDE

To setup the IDE, please follow these instructions:

1. Open the ORBIT-Surgical directory on Visual Studio Code IDE.
2. Run VSCode Tasks, by pressing Ctrl+Shift+P, selecting Tasks: Run Task and running the setup_python_env in the drop down menu. When running this task, you will be prompted to add the absolute path to your Isaac Sim installation.

Set your conda environment as the default interpreter in VSCode by opening the command palette (Ctrl+Shift+P), choosing Python: Select Interpreter and selecting your conda environment.

If everything executes correctly, it should create a file .python.env in the .vscode directory. The file contains the python paths to all the extensions provided by Isaac Sim and Omniverse. This helps in indexing all the python modules for intelligent suggestions while writing code.

Verification using Teleoperation

We provide interfaces for providing commands in SE(2) and SE(3) space for robot control. In case of SE(2) teleoperation, the returned command is the linear x-y velocity and yaw rate, while in SE(3), the returned command is a 6-D vector representing the change in pose.

To play inverse kinematics (IK) control with a keyboard device:

${IsaacLab_PATH}/isaaclab.sh -p source/standalone/environments/teleoperation/teleop_se3_agent.py --task Isaac-Lift-Needle-PSM-IK-Rel-v0 --num_envs 1 --teleop_device keyboard

The script prints the teleoperation events configured. For keyboard, these are as follows:

Keyboard Controller for SE(3): Se3Keyboard
    Reset all commands: L
    Toggle gripper (open/close): K
    Move arm along x-axis: W/S
    Move arm along y-axis: A/D
    Move arm along z-axis: Q/E
    Rotate arm along x-axis: Z/X
    Rotate arm along y-axis: T/G
    Rotate arm along z-axis: C/V

• Other examples are in the reference Github: https://github.com/orbit-surgical/orbit-surgical

Trouble Shooting

No module named pip

cd ${ISAACSIM_PATH}
./python.sh -m ensurepip (reinstall pip)
./python.sh -m pip install numpy==1.22.2 (compatible version with Isaac Sim)

Diverge when collide with ground

• ground에 충돌시 반작용으로 로봇이 튕겨나오면서 발산함 (ground는 'fixed' 특성을 가짐.)
• ground를 obstacle로 인식하도록 하거나 z-position 값이 음수가 되지 않도록 설정해야 함.
• 유사하게, box를 생성하여 ground 아래로 끌어내리면 튕겨나감. 시뮬레이터 자체 physics가 그렇게 설정되어있는 듯.

UserWarning: FigureCanvasAgg is non-interactive, and thus cannot be shown

• import matplotlib.pyplot as plt
• plt.show() 함수 호출시 발생
• 최신 버젼의 PyQt5를 설치해주면 됨. (Isaac Sim 환경에서는 ./python.sh -m pip install PyQt5==5.14.1)

imshow() during launching simulation_app

• simulation_app과 cv2.imshow()를 함께 사용할 수 없음. 아마도 Qt를 사용하면서 충돌이 있는듯.
• 따라서 캡쳐한 이미지를 확인하고 싶으면, plt.show()를 활용해야 함.

import matplotlib.pyplot as plt
...
plt.imshow(seg_rgb)
plt.show()