Reading IMU data with Video frames - will127534/OneInchEye GitHub Wiki
Basically on the new OneInchEye/StarlightEye I've added an ICM42688-P 6-axis IMU with VSYNC from the image sensor, based on the experience I have with this sensor on the IMU array project.
The way I setup the process is via rpicam-apps's post-processing stage, so all the config can be stored in a json file and then pass on to rpicam-vid and it will read the data from IMU and align the data with each frame while saving the data to csv file.
Now the way to align IMU data and camera frames is via connecting the VSYNC output to IMU, but IMU still sampling at its own internal clock(200Hz ODR hard-coded, explain later), but with the VSYNC IMU will measure when the VSYNC event happened and what is the time-delta to the recent sample, such that you can calculate the data when the VSYNC happens with linear interpolation.
The higher the ODR, the closer VSYNC is to the last sample but because I want to keep the code simple, it will only read IMU every input frame shows up. The data will store in the IMU FIFO in between the frames and the post-processing stage will read the FIFO content to find the VSYNC flag and calculate. This is why it is currently set at 200Hz ODR to prevent FIFO overflow.
By default it will output the csv in gryoflow format, with the timestamp from libcamera/ISP timestamp, there is also an option to read raw IMU data and that will be using the timestamp from IMU itself.
First, you need to enable always-on flag in /boot/firmware/config.txt
Add the flag to the end of dtoverlay for camera:
dtoverlay=imx283,always-on
Additionally to increase the I2C linkspeed, add a line in imx283-overlay.dts or imx585-overlay.dts under the driver imx283-v4l2-driver or imx585-v4l2-driver folder
i2c_frag: fragment@100 {
target = <&i2c_csi_dsi>;
__overlay__ {
#address-cells = <1>;
#size-cells = <0>;
clock-frequency = <400000>; #<======Add this line to change I2C clock to 400Khz
status = "okay";
cam_node: imx283@1a {
Reinstall the driver with ./setup.sh
Next, after reboot RPI, create a post-processing config json file
nano ./imu.json
Here is a example config file
{
"imu_sync": {
"i2c_device_path": "/dev/i2c-4",
"filename": "test.csv",
"print_console" : true,
"acc_scale" : 8.0,
"gyro_scale" : 250.0,
"write_raw" : false
}
}
For i2c_device_path, this will be the I2C bus you connect OneInchEye/StarlightEye, i2c-4 for CAM1 and i2c-6 for CAM0 on RPI5.
print_console for looking at live IMU data through console output.
acc_scale and gyro_scale to select different IMU measurement range, for ICM42688-P it is:
acc_scale: 2/4/8/16g
gyro_scale: 250/500/1000/2000dps
write_raw is to select reading all the data from IMU directly, you can calculate the frames because VSYNC mode is still enabled, the data format will be slightly different with write_raw enabled.
Now with the imu.json, run the rpicam-apps with --post-process-file imu.json
Like this: rpicam-vid -t 0 --post-process-file imu.json
And you should see some thing like this:
pi@camera:~/libcamera2 $ rpicam-vid -t 0 --post-process-file imu.json
[0:00:38.664490511] [2018] INFO Camera camera_manager.cpp:326 libcamera v0.0.0+5529-891a78d5
[0:00:38.672106439] [2019] INFO RPI pisp.cpp:720 libpisp version v1.2.0 50426319aa1a 11-05-2025 (23:24:34)
[0:00:38.682389780] [2019] WARN RPiAlsc alsc.cpp:171 no luminance table - assume unity everywhere
[0:00:38.683301032] [2019] INFO RPI pisp.cpp:1179 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx283@1a to CFE device /dev/media2 and ISP device /dev/media0 using PiSP variant BCM2712_D0
Preview window unavailable
Unable to open /usr/local/lib/aarch64-linux-gnu/rpicam-apps-postproc/opencv-postproc.so with error: /lib/aarch64-linux-gnu/libcamera.so.0.3: undefined symbol: _ZN7libpisp22compute_optimal_strideER24pisp_image_format_config
Reading post processing stage "imu_sync"
[0:00:38.793649799] [2018] WARN V4L2 v4l2_pixelformat.cpp:346 Unsupported V4L2 pixel format RPBP
Mode selection for 640:480:12:P
SRGGB10_CSI2P,5568x3094/0 - Score: 3060.35
SRGGB10_CSI2P,5568x3664/0 - Score: 3097.87
SRGGB12_CSI2P,2784x1828/0 - Score: 944.116
SRGGB12_CSI2P,5568x3664/0 - Score: 2097.87
Stream configuration adjusted
[0:00:38.793897214] [2018] INFO Camera camera.cpp:1205 configuring streams: (0) 640x480-YUV420 (1) 2784x1828-RGGB_PISP_COMP1
[0:00:38.794015204] [2019] INFO RPI pisp.cpp:1483 Sensor: /base/axi/pcie@120000/rp1/i2c@88000/imx283@1a - Selected sensor format: 2784x1828-SRGGB12_1X12 - Selected CFE format: 2784x1828-PC1R
[0:00:38.794232257] [2019] INFO IPARPI ipa_base.cpp:594 Adjusting mode minimum line length from 5.03us to 7.33us because of ISP constraints.
[IMU] using /dev/i2c-6
[libx264 @ 0x5555bd8e1450] using cpu capabilities: ARMv8 NEON
[libx264 @ 0x5555bd8e1450] profile High, level 3.0, 4:2:0, 8-bit
Output #0, h264, to '':
Stream #0:0: Video: h264, yuv420p(tv, smpte170m/smpte170m/bt709), 640x480, q=2-31, 30 fps, 30 tbr, 1000k tbn
F#0 cam_ts=39356032000 acc=(-0.00142068,-0.00272978,0.49801) gyro=(-0.459152,-0.260784,0.0346609)
#0 (0.00 fps) exp 33291.00 ag 8.00 dg 1.00
F#1 cam_ts=39388526000 acc=(-0.00122072,-0.00335003,0.498787) gyro=(-0.480659,-0.259403,0.0381476)
#1 (30.77 fps) exp 33291.00 ag 8.00 dg 1.00
F#2 cam_ts=39421023000 acc=(-0.00122072,-0.00332647,0.498299) gyro=(-0.4654,-0.256542,0.0486381)
#2 (30.77 fps) exp 33291.00 ag 8.00 dg 1.00
F#3 cam_ts=39453513000 acc=(-0.000861341,-0.00341802,0.498787) gyro=(-0.469001,-0.259403,0.0341192)
#3 (30.78 fps) exp 33291.00 ag 8.00 dg 1.00
F#4 cam_ts=39486001000 acc=(-0.00135964,-0.00331279,0.498543) gyro=(-0.480659,-0.262692,0.0490654)
#4 (30.78 fps) exp 33291.00 ag 8.00 dg 1.00
F#5 cam_ts=39518495000 acc=(-0.000813977,-0.00333648,0.498868) gyro=(-0.462852,-0.251774,0.0432288)
#5 (30.77 fps) exp 33291.00 ag 8.00 dg 1.00
F#6 cam_ts=39550994000 acc=(-0.000976577,-0.00328936,0.499031) gyro=(-0.48468,-0.259403,0.0534066)
#6 (30.77 fps) exp 33291.00 ag 8.00 dg 1.00
F#7 cam_ts=39583492000 acc=(-0.00118654,-0.00254398,0.498474) gyro=(-0.466468,-0.267033,0.030518)
#7 (30.77 fps) exp 33291.00 ag 8.00 dg 1.00
F#8 cam_ts=39615987000 acc=(-0.00170425,-0.00341326,0.498792) gyro=(-0.480362,-0.259552,0.0456283)
#8 (30.77 fps) exp 33291.00 ag 8.00 dg 1.00
F#9 cam_ts=39648475000 acc=(-0.00143642,-0.00271403,0.498815) gyro=(-0.47303,-0.252663,0.0448882)
#9 (30.78 fps) exp 33291.00 ag 8.00 dg 1.00
F#10 cam_ts=39680969000 acc=(-0.000924453,-0.00331377,0.498054) gyro=(-0.480659,-0.251774,0.0337758)
And the csv file will have content like this:
GYROFLOW IMU CSV v0.5
t,gx,gy,gz,ax,ay,az
39356032000,-0.00801371,-0.00455154,0.000604946,-0.00142068,-0.00272978,0.49801
39388526000,-0.00838909,-0.00452744,0.0006658,-0.00122072,-0.00335003,0.498787
39421023000,-0.00812277,-0.00447751,0.000848896,-0.00122072,-0.00332647,0.498299
39453513000,-0.00818562,-0.00452744,0.000595492,-0.000861341,-0.00341802,0.498787
39486001000,-0.00838909,-0.00458483,0.000856353,-0.00135964,-0.00331279,0.498543
39518495000,-0.00807829,-0.00439428,0.000754485,-0.000813977,-0.00333648,0.498868
39550994000,-0.00845926,-0.00452744,0.000932121,-0.000976577,-0.00328936,0.499031