Pix3D

Pix3D is a large-scale dataset of real images paired with pixel-aligned 3D shapes, released alongside Sun, Wu, Zhang et al., Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (CVPR 2018). This Hub repo repackages the official release into two Hugging Face datasets configs so both the image annotations and the underlying 3D assets can be loaded directly with load_dataset, with no manual download/unzip step.

• Project page: http://pix3d.csail.mit.edu
• Original repository: https://github.com/xingyuansun/pix3d
• License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Dataset structure

The dataset is split into two configs because 10,069 image annotations reference only 735 unique physical 3D objects (many objects were photographed multiple times). Keeping them separate avoids duplicating each mesh/voxel grid across every image that uses it; join the two configs on model_id when you need both.

from datasets import load_dataset

annotations = load_dataset("unstructured/pix3d", "default")   # 10,069 rows, one per image
models = load_dataset("unstructured/pix3d", "models")          # 735 rows, one per unique 3D object

default config — one row per annotated image

Column	Type	Description
idx	int32	Unique incrementing row identifier
img	Image	The photograph containing the object
width, height	int32	Image dimensions (img_size in the original JSON)
category	ClassLabel	Object category (bed, bookcase, chair, desk, misc, sofa, table, tool, wardrobe)
mask	Image	Binary segmentation mask isolating the object in img
model_id	string	Join key into the models config identifying which 3D object appears in this image
img_source	ClassLabel	Image provenance: ikea, internet, or self-taken
keypoints_2d	list[list[list[float]]]	2D keypoints per annotator, shape [n_annotators, n_keypoints, 2], in pixel coordinates (origin top-left, +x right, +y down); [-1, -1] marks a keypoint an annotator judged unlabelable (e.g. occluded)
bbox	list[int]	Object bounding box [x_from, y_from, x_to, y_to]
rot_mat, trans_mat	list[float]	Rotation matrix (3x3) and translation vector applied to the object for rendering
cam_position, inplane_rotation	list[float] / float	Estimated camera position and in-plane rotation (object-centered), used to evaluate viewer-centered predictions against the canonical view
focal_length	float	Estimated focal length in mm (sensor width fixed at 32mm)
truncated, occluded, slightly_occluded	bool	Visibility flags for the object in the image (occluded and slightly_occluded are mutually exclusive)

models config — one row per unique 3D object

Column	Type	Description
model_id	string	Unique identifier for this 3D object; matches model_id in the default config
category	ClassLabel	Same category enumeration as the default config
model	binary (glb)	The object's textured mesh, converted from the original .obj/.mtl/texture files into a single self-contained binary glTF (.glb) blob via trimesh, analogous to how img/mask embed PNG bytes
model_raw	binary (glb) or null	Raw, unprocessed scanned mesh, same glb encoding as model. Only populated for objects from the full Pix3D release that includes raw scans (self-scanned sources); this standard release does not include raw scans, so this column is null for every row here
model_source	ClassLabel	3D model provenance: ikea or self-scanned
voxel_model	Array3D (128×128×128, uint8)	Voxelized occupancy grid of the object, parsed from the original voxel.mat file (1 = occupied)
keypoints_3d	list[list[float]]	3D keypoint positions [n_keypoints, 3] in the model's canonical coordinate frame (+x left, +y up, +z inward)

Coordinate conventions (from the original release)

• 2D keypoints / image pixels: origin top-left, +x right, +y down.
• Mesh (model) canonical frame: +x left, +y up, +z inward.
• Voxel (voxel_model) canonical frame: +x left, +y outward, +z up.
• rot_mat/trans_mat are defined in camera coordinates and applied to the object for rendering; they are not directly usable to rotate viewer-centered predictions back to canonical view. Use cam_position and inplane_rotation instead — both are defined in the object's own coordinate frame and reproduce the original image up to a homography, which is what the original authors used for viewpoint evaluation.

Category / label distribution

Category	# images
chair	3,839
sofa	1,947
table	1,870
bed	994
desk	700
bookcase	361
wardrobe	243
misc	68
tool	47

License and attribution

The Pix3D dataset is licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). You are free to share and adapt the data for any purpose, including commercially, provided you give appropriate credit to the original authors.

If you use this dataset, please cite the original paper:

@inproceedings{pix3d,
  title={Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling},
  author={Sun, Xingyuan and Wu, Jiajun and Zhang, Xiuming and Zhang, Zhoutong and Zhang, Chengkai and Xue, Tianfan and Tenenbaum, Joshua B and Freeman, William T},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2018}
}

For questions about the original dataset, contact Xingyuan Sun (xingyuansun.cs@gmail.com) or Jiajun Wu (jiajunwu@mit.edu). This Hub repackaging was produced by converting the official release's pix3d.json and asset folders (img/, mask/, model/) into the two configs described above; no annotation values were modified, only the file layout and encoding (mesh → glb, voxel .mat → numpy array).