anthonytec2/OctoSense

Pre-Release State

A time-synchronized, calibrated multi-sensor driving dataset: 371 sequences · 59 hrs · 2,474 km · 8.43 TB, spanning urban, suburban, and rural driving (highway / residential / city) on Long Island and in Philadelphia, across sunrise, daytime, sunset, and nighttime. Sequences carry synchronized stereo RGB, stereo event cameras, infrared, Ouster LiDAR, two IMUs, GPS and CAN data, plus ground truth (depth, ego-motion optical flow, semantic segmentation) and a fused GPS + LiDAR-inertial odometry trajectory.

Upon release, we will provide code for using the dataset and deriving flow.

Canonical key for every sequence is its recording datetime, rosbag2_YYYY_MM_DD-HH_MM_SS. Per-sequence attributes live in metadata.parquet (the viewer's table: a sensor-montage thumbnail, a GPS-track map, and all metadata columns per sequence).

Sensors

Modality	Sensor	Info	Rate
RGB (stereo)	2× FLIR Blackfly S (Sony IMX421)	1920×1456	100 Hz
Event (stereo)	2× SilkyEV VGA (Prophesee)	640×480	≈7 MEv/s avg
Thermal	FLIR A35	320×256	50 Hz
LiDAR	Ouster OS1-64	64 beams × 2048	10 Hz
IMU	VectorNav VN-100T	Acc/Gyro/Mag/Baro/Temp	400 Hz
IMU (in LiDAR)	IAM-20680HT	Acc/Gyro	100 Hz
GNSS	u-blox ZED-F9P	RTK (NTRIP)	5 Hz
Vehicle	2021 Mazda CX-5	CAN Signals	50–100 Hz

Coordinate frames & calibration

All extrinsics are stored per-sequence under /calib in the bag h5, named A_T_B — a 4×4 matrix that maps a point from frame B into frame A (e.g. imgl_T_ouster takes LiDAR points into the left-RGB frame). Frame abbreviations:

abbrev	frame		abbrev	frame
imgl / imgr	RGB left / right		ir	Infrared (FLIR A35)
evl / evr	Event left / right		ouster	LiDAR
imu	VectorNav IMU

Provided extrinsics: imgl_T_imgr, imgl_T_imu, imgl_T_ir, imgr_T_imu, ir_T_imgl, the event-pair set (evl_T_evr, evl_T_imgl, …), ouster/imgl_T_ouster / ouster/ir_T_ouster, and calib/lidar_T_lidarimu (LiDAR IMU → LiDAR/sensor frame, from the Ouster factory imu_to_sensor_transform). Each drive records which calibration it used via rgb_cal_id / imu_cal_id / lidar_cal_id (in the metadata). RKO-LIO odometry (ouster/odom/*) tracks the LiDAR pose in the Map frame; the fused_traj trajectory tracks the same LiDAR pose in the UTM-relative world frame (see Ground truth). The depth GT is rendered in the rectified left-RGB camera image (OpenCV stereoRectify, R1 applied). Use the /calib extrinsics (e.g. imgl_T_ouster) to move between frames.

Per-session calibration is released alongside the data, in Kalibr format:

• Cameras (calibration-camchain.yaml) — a 4-camera Kalibr chain: cam0 = RGB right (/flir_cam_right), cam1 = RGB left (/flir_cam_left), cam2 = event right (/event_camera_right, 640×480), cam3 = event left (/event_camera_left, 640×480). Each entry has pinhole intrinsics [fx,fy,cx,cy] + radtan distortion_coeffs; each cam after cam0 carries T_cn_cnm1 (transform from the previous cam in the chain).
• Camera↔IMU (calibration-camchain-imucam.yaml, adds T_img[l/r]_imu) + the IMU noise model (calibration-imu.yaml).
• LiDAR↔camera (lidar_calibration_results.yaml) and IR intrinsics + extrinsics (ir_calib_result.json).

The processed h5 renames cam0/cam1 to the left/right RGB streams above (and cam2/cam3 are the right/left event cameras).

Ground truth

• Odometry / ego-motion — RKO-LIO LiDAR-inertial odometry: SE(3) LiDAR pose in the Map frame with linear/angular velocity. The high-rate hf_odom/* is obtained by integrating the IMU between LiDAR keyframes.
• Depth — sparse metric depth in the rectified left-RGB image (depth_cm, uint16 cm, 0 = invalid). Built by accumulating 61 LiDAR scans (≈6 s), removing dynamic objects (YOLO26-medium on the nearest RGB frame), keeping the minimum depth per pixel, and then projecting into the camera image.
• Optical flow — derived, not stored — the ego-motion-induced flow (accumulated points projected into a future RGB frame). Regenerate from depth + poses with derive_flow.py.
• Semantic segmentation — 19-class Cityscapes pseudo-labels (EoMT), on the 303 daytime sequences (has_seg = true); night/degraded have no labels.
• Fused reference trajectory — RKO-LIO odometry + GPS fused in a pose graph, under fused_traj (T_world_lidar (N,4,4) + t) in a UTM-relative world frame anchored at the first GPS fix. Geo-referencing + fusion quality live as fused_traj group attrs (epsg/utm_zone/hemisphere, origin_{easting,northing,alt}_m, lever_xyz_lidar_m, GPS residual RMS + p90, confidence_tier). fused_traj/t is non-uniformly sampled — poses are emitted at the union of the LiDAR-odom (≈10 Hz) and GPS (≈5 Hz) timestamps (≈13 Hz combined), so inter-pose intervals vary (≈0.02–0.1 s).

Per-sequence files (<session>/<bag_id>/)

file	contents
data.h5	timestamps, IMU, GPS, CAN, LiDAR (range [deskewed by RKO] / sig / nir / refl), RKO-LIO odom, fused_traj (fused GPS+LIO trajectory), /calib
events.h5	raw async event streams (ev/left, ev/right) — ≈78% of a sequence's bytes
img_{left,right,infrared}.mp4	per-camera H.265 encoded video
rgb_left_rect_depth.h5	sparse metric depth in the rectified left-RGB image (depth_cm); flow derived via derive_flow.py
rgb_left_rect_semantic.h5	19-class Cityscapes pseudo-label seg in the rectified left-RGB image — day sequences only
captions.h5	per-window scene captions (Gemma-4-31B VLM) + 4096-d Qwen3-Embedding-8B caption vectors + window metadata

Units/conventions: depth in cm (0=invalid); event timestamps in µs; all other h5 time arrays in seconds.

Experimental — car/radar (Mazda forward radar). Six object tracks decoded from the CAN data with the opendbc Mazda radar DBC (CAN IDs 865–870 = RADAR_TRACK_361..366). Layout: car/radar/track_{1..6}/ with t (M,) float64 main-clock seconds, dist (M,) uint16, ang (M,) int16, vrel (M,) int16. Values are stored raw (no scale/offset): dist=DIST_OBJ (sentinel 4095 = no track), ang=ANG_OBJ (signed 12-bit), vrel=RELV_OBJ (signed 11-bit).

Data format

All */t arrays are seconds on a common PPS-synced main clock (Kalman-smoothed); event times are µs on that same clock (t/1e6 → s). Align streams by timestamp — RGB runs ≈100 Hz vs LiDAR ≈10 Hz. Camera frames are raw (un-rectified): rectify with the per-camera intrinsics + radtan dist_coeffs (stored in the h5 and the camchain); the depth GT is already in the rectified left-RGB frame.

Main bag h5 (data.h5):

key	shape · dtype	meaning
ouster/range_pcl	(N,131072,3) int32	LiDAR XYZ in mm, 64×2048 destaggered ([0,0,0] = no return)
ouster/{sig,nir}_pcl · refl_pcl	(N,131072) uint16 · uint8	per-point signal / near-IR / reflectivity
ouster/odom/map_T_lidart	(N,4,4) f64	RKO-LIO LiDAR pose in the map frame (translation m) (+ lin_vel m/s, ang_vel rad/s, t s); hf_odom/* = high-rate
ouster/{accel,ang_vel}	(M,3) f32	in-LiDAR IMU (IAM-20680HT); extrinsic to sensor frame = calib/lidar_T_lidarimu
img/{left,right}/, infrared/	—	t (≈100/100/50 Hz) · intrinsics (3,3) · dist_coeffs (4,) · resolution (2,)
vectornav/	(K,·)	accel (m/s²), ang_vel (rad/s) — each + _raw (uncompensated); magnetic (Gauss), pressure (Pa), temperature (°C), t (s); 400 Hz
gps/data	(G,7) f64	[lat°, lon°, alt_m, cov_xx, cov_yy, cov_zz, fix]. fix = ROS NavSatStatus.status: −1 no-fix · 0 standard fix · 1 SBAS · 2 GBAS/RTK.
gps/velocity_enu	(G,3) f64	ENU velocity m/s [East, North, Up] (from /fix_velocity) — use this for course-over-ground
car/	(·,2+) f32	CAN (col 0 = time, s): speed (mph), wheels (4× wheel speed, mph), steer (deg), steer_rate (deg/s), vcc = [acc_x, acc_y] (≈m/s²), brake_on (0/1), pedal & brake_press (raw counts) + radar/ (above). Experimental — decoded via a community Mazda DBC; units approximate/unverified.
fused_traj/T_world_lidar	(R,4,4) f64	fused RKO-LIO+GPS pose, UTM-relative world frame (+ fused_traj/t; geo-ref + quality in fused_traj attrs: epsg/utm_zone, origin_*, confidence_tier, residual RMS/p90)
calib/	—	extrinsics A_T_B

Events (events.h5) — flat async event streams per side, plus the event-camera calibration (kept here so events.h5 is self-contained for projecting events without opening data.h5):

key	dtype	meaning
ev/{left,right}/t	uint64	event time µs (main clock)
ev/{left,right}/{x,y}	uint16	pixel col / row, raw 640×480 (un-rectified)
ev/{left,right}/p	uint8	polarity (0/1)
ev/{left,right}/ms_to_idx	uint64	millisecond → event index
ev/{left,right}/intrinsics	(3,3) f64	pinhole K at native 640×480
ev/{left,right}/dist_coeffs	(4,) f64	radtan [k1, k2, p1, p2]
ev/{left,right}/resolution	(2,) int32	[width, height] = [640, 480]

Depth GT (rgb_left_rect_depth.h5) — per-LiDAR-scan sparse depth in the rectified left-RGB image. N = n_lidar_frames; frame k ↔ LiDAR scan k. Carries its own timestamps (identical to data.h5 ouster/t) and index arrays:

key	shape · dtype	meaning
depth_cm	(N,1456,1920) uint16	metric depth in cm (m = depth_cm/100), 0 = invalid
timestamps	(N,) f64	main-clock seconds (= data.h5 ouster/t)
lidar_indices	(N,) int64	LiDAR-scan index (0..N−1)
left_img_indices	(N,) int64	matching frame in img_left.mp4
poses	(N,4,4) f32	per-scan LiDAR pose used to accumulate the depth (input to derive_flow.py)
K_rect · R3	(3,3) f64	rectified left-RGB intrinsics · rectification rotation (R1)
imgl_T_ouster	(4,4) f64	LiDAR → raw (unrectified) left-RGB cam
raw_res	(2,) int32	source image resolution

Root attrs: depth_scale_cm, flow_{gap,scale,invalid}, flow_stored (=False — flow is derived, not stored), n_scans.

Semantic GT (rgb_left_rect_semantic.h5, day sequences only) — per-LiDAR-scan 19-class seg in the rectified left-RGB image, aligned 1:1 with rgb_left_rect_depth.h5:

key	shape · dtype	meaning
semantic	(N,1456,1920) uint8	Cityscapes class id 0..18 (see classes attr); 255 = ignore
lidar_indices	(N,) int64	LiDAR-scan index for each frame (0..N−1)
left_img_indices	(N,) int64	matching frame in img_left.mp4 (RGB ≈100 Hz vs LiDAR ≈10 Hz)
timestamps	(N,) f64	main-clock seconds

Root attrs: classes (19 names), num_classes=19, model=EoMT-Cityscapes-DINOv2-L-1024, coordinate_frame=rectified_left, preprocessing=rectify+CLAHE, resolution=1920x1456. So semantic[k] ↔ depth_cm[k] ↔ img_left.mp4[left_img_indices[k]] ↔ ouster/t[k].

Captions & semantic search (captions.h5) — each sequence is split into ≈5 s windows (W per sequence ≈ duration / 5 s); every window gets a natural-language scene caption (a Gemma-4-31B VLM) and a 4096-d text embedding (Qwen3-Embedding-8B) for free-text retrieval:

key	shape · dtype	meaning
captions	(W,) str	one scene caption per window
embeddings	(W,4096) f32	Qwen3-Embedding-8B vector for each caption
metadata	(W,) struct	window_id, frame_idx (→ img_left.mp4 frame), timestamp (main-clock s), speed_mps, turn_deg, dist_m, is_night

Root attrs: caption_model, embed_model, embed_dim (4096), num_windows, video_id.

Splits

train = 293 · test = 78 (bag-level; preserved across dataset versions). The dataset contains daytime, nighttime, and degraded sequences.

metadata.parquet fields (27)

bag_id, session, start_time, split, is_daytime, degraded, has_seg, duration_s, n_lidar_frames, n_rgb_frames, n_imu_samples, n_events_left, n_events_right, n_gps_fixes, n_gps_valid, gps_quality, gps_lat_min/max, gps_lon_min/max, mean_speed_mph, idle_fraction, distance_m, rgb_cal_id, imu_cal_id, lidar_cal_id, sensor_dropout

• gps_quality ∈ {RTK_fixed_cm, float_dm, single_m, no_fix, absent} — 67 / 62 / 239 / 2 / 1 (RTK-fixed ≈1–3 cm, float ≈0.1–0.5 m, single ≈0.5–3 m).
• sensor_dropout — null, or sensor:seconds[;sensor:seconds]

Loading

import h5py, hdf5plugin, numpy as np, pyarrow.parquet as pq
meta = pq.read_table("metadata.parquet").to_pydict()       # per-sequence table
with h5py.File("<session>/<bag_id>/data.h5") as f:
    lidar = f["ouster/range_pcl"][:]; odom = f["ouster/odom/map_T_lidart"][:]

# RGB / IR frames — native-res H.265 mp4 decoded with torchcodec.
from torchcodec.decoders import VideoDecoder
dec = VideoDecoder("<session>/<bag_id>/img_left.mp4")      # 1920x1456; len(dec) == n_rgb_frames
rgb_k = dec[k]                                             # (3, H, W) uint8 tensor at frame k
# RGB runs ≈100 Hz vs LiDAR ≈10 Hz — align by TIMESTAMP, not shared index:
with h5py.File("<session>/<bag_id>/data.h5") as f:
    img_t = f["img/left/t"][:]; lid_t = f["ouster/t"][:]   # seconds, same main clock
k = int(np.argmin(np.abs(img_t - lid_t[j])))              # RGB frame nearest LiDAR frame j
# img_right.mp4 / img_infrared.mp4 decode the same way (IR is grayscale-as-video, ≈50 Hz).

# Depth + semantic GT — one frame per LiDAR scan, in the rectified left-RGB image:
with h5py.File("<session>/<bag_id>/rgb_left_rect_depth.h5") as f:
    depth_m = f["depth_cm"][j] / 100.0                     # (1456,1920); 0 = invalid (j = LiDAR scan)
with h5py.File("<session>/<bag_id>/rgb_left_rect_semantic.h5") as f:   # day sequences only
    seg     = f["semantic"][j]                             # (1456,1920) uint8, class 0..18
    rgb_idx = f["left_img_indices"][j]                     # matching img_left.mp4 frame for scan j

# flow is derived, not stored:
from derive_flow import derive_flow_from_h5
flow_uv = derive_flow_from_h5("<session>/<bag_id>/rgb_left_rect_depth.h5", frame_idx, poses=odom)

Statistics

371 sequences · 59 hrs · 2,474 km · 8.43 TB · 303 day / 68 night · 4 degraded · segmentation on 303 · GPS: 67 RTK / 62 float / 239 single / 2 no-fix / 1 absent.

Notes/Known limitations

• Optical flow is derived solely from ego-motion — rigid camera-motion reprojection of LiDAR depth; it does not capture independent motion of dynamic objects.
• Seg is day-only (303 sequences); night/degraded have no segmentation labels.
• IMU compensated vs raw. VectorNav factory-calibrates each unit (bias/scale/axis-misalignment + temperature applied). In this dataset accel ≡ accel_raw (bit-identical); ang_vel additionally has the on-board EKF's real-time gyro-bias estimate removed.
• 5 sequences have a >10 s sensor dropout (see sensor_dropout): 3 with the event camera off early (20–43 s), 1 GPS+CAN, 1 CAN. 3 sequences have no usable GPS (no_fix/absent), and 5 sequences have no fused_traj (GPS too sparse / low-quality for the pose-graph fusion).
• IR & GPS are PPS-synced indirectly, through the IMU PPS-system clock calibration.
• IR calibration is best-effort. The infrared intrinsics/extrinsics were calibrated from existing data without a heated target board. They are estimated from the markerboard's 4 corners alone, then manually refined. Treat the IR calibration as approximate.
• LiDAR clock is PPS-stepped, not Kalman-aligned. Unlike the other streams, the Ouster clock is not actively Kalman-smoothed onto the main clock; its internal clock only advances its whole-second counter when the PPS edge arrives.
• IMU is noisy from vehicle vibration. Both IMUs pick up road and engine vibration; the sensors are soft-mounted to dampen it, but residual vibration noise remains in the accelerometer / gyro signals.
• IR frame rate occasionally dips below 50 Hz. The infrared camera is not run as a composable node, so under load it sometimes falls below its nominal 50 Hz capture rate (gaps visible in infrared/t).
• RKO-LIO cold-start transient. The estimator's initial phase can occasionally be jerky — at the start of a sequence the LIO briefly reports ≈zero motion, then "catches up" with a jump once it converges → jerky roll/pitch (occasionally z) over the first few meters of motion (upstream RKO-LIO issue #139).
• LiDAR deskew is a constant-motion approximation. Each ~100 ms sweep is motion-compensated by RKO-LIO using the average body acceleration a and average angular velocity ω over the sweep — a point at offset dt from the scan reference time is warped by exp([ v·dt + ½·a·dt², ω·dt ]) (constant-acceleration translation + constant-angular-velocity rotation). Because a/ω are held constant across the sweep, rapid intra-sweep motion (high jerk, sharp turns, potholes) leaves some residual skew; a is Kalman-filtered + jerk-bounded to limit this but cannot fully remove it.
• LiDAR noise can leak into the depth GT. The depth ground truth is projected directly from the raw Ouster returns, so sensor noise (stray or spurious returns from rain, snow, fog, dust, retroreflectors (see LiDAR ghosts & blooming), or specular/multi-path reflections) can survive into our depth ground truth as a small number of erroneous points.
• Platform Bounce. On a few sequences the SeaSuckers loosened, resulting in vertical motion (bounce) of the platform.

We have taken great care to perform data quality checks on this data. That said, some issues at this scale may slip through, so should you find any examples of gross desynchronization, please report them and we can take a look.

Citation

@misc{bisulco2026octosense,
  title        = {{OctoSense}: Self-Supervised Learning for Multimodal Robot Perception},
  author       = {Bisulco, Anthony and Wang, Jeremy and Daniilidis, Kostas and Balestriero, Randall and Chaudhari, Pratik},
  year         = {2026},
  howpublished = {Preprint},
}

License

Released under the MIT License — free to use, modify, and redistribute with attribution; provided "as is" without warranty. If you use OctoSense, please cite the paper (see Citation above).