Hugging Face's logo

Datasets:
Quome
/
devicebench

Tasks:
Other
Languages:
English
Size:
n<1K
Tags:
clinical-ai
llm-security
ai-agents
healthcare
haarf
quokkaguard
License:
Dataset card Files
xet
 Community
Dataset Viewer
Duplicate
The dataset viewer is not available for this split.
Cannot load the dataset split (in streaming mode) to extract the first rows.
Error code:   StreamingRowsError
Exception:    CastError
Message:      Couldn't cast
version: string
devices: struct<ventilator: struct<risk_class: string, params: struct<tidal_volume_ml: struct<min: double, ma (... 1532 chars omitted)
  child 0, ventilator: struct<risk_class: string, params: struct<tidal_volume_ml: struct<min: double, max: double, requires (... 617 chars omitted)
      child 0, risk_class: string
      child 1, params: struct<tidal_volume_ml: struct<min: double, max: double, requires_confirm: bool>, peep_cmh2o: struct (... 330 chars omitted)
          child 0, tidal_volume_ml: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
          child 1, peep_cmh2o: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
          child 2, pip_cmh2o: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
          child 3, fio2: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
          child 4, resp_rate_bpm: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: boo
...
res_c (... 355 chars omitted)
      child 0, risk_class: string
      child 1, params: struct<hr_alarm_high: struct<min: double, max: double, requires_confirm: bool>, hr_alarm_low: struct (... 124 chars omitted)
          child 0, hr_alarm_high: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
          child 1, hr_alarm_low: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
          child 2, alarm_enabled: struct<min: double, max: double, requires_confirm: bool>
              child 0, min: double
              child 1, max: double
              child 2, requires_confirm: bool
      child 2, interlocks: list<item: struct<left: string, op: string, right: string>>
          child 0, item: struct<left: string, op: string, right: string>
              child 0, left: string
              child 1, op: string
              child 2, right: string
      child 3, alarm_params: list<item: string>
          child 0, item: string
      child 4, safe_state: struct<hr_alarm_high: double, hr_alarm_low: double, alarm_enabled: double>
          child 0, hr_alarm_high: double
          child 1, hr_alarm_low: double
          child 2, alarm_enabled: double
onset: int64
case_id: string
kpi_trace: list<item: double>
  child 0, item: double
device: string
to
{'case_id': Value('string'), 'device': Value('string'), 'onset': Value('int64'), 'kpi_trace': List(Value('float64'))}
because column names don't match
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2951, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2461, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2486, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 547, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 430, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
                  raise CastError(
                  ...<3 lines>...
                  )
              datasets.table.CastError: Couldn't cast
              version: string
              devices: struct<ventilator: struct<risk_class: string, params: struct<tidal_volume_ml: struct<min: double, ma (... 1532 chars omitted)
                child 0, ventilator: struct<risk_class: string, params: struct<tidal_volume_ml: struct<min: double, max: double, requires (... 617 chars omitted)
                    child 0, risk_class: string
                    child 1, params: struct<tidal_volume_ml: struct<min: double, max: double, requires_confirm: bool>, peep_cmh2o: struct (... 330 chars omitted)
                        child 0, tidal_volume_ml: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                        child 1, peep_cmh2o: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                        child 2, pip_cmh2o: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                        child 3, fio2: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                        child 4, resp_rate_bpm: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: boo
              ...
              res_c (... 355 chars omitted)
                    child 0, risk_class: string
                    child 1, params: struct<hr_alarm_high: struct<min: double, max: double, requires_confirm: bool>, hr_alarm_low: struct (... 124 chars omitted)
                        child 0, hr_alarm_high: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                        child 1, hr_alarm_low: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                        child 2, alarm_enabled: struct<min: double, max: double, requires_confirm: bool>
                            child 0, min: double
                            child 1, max: double
                            child 2, requires_confirm: bool
                    child 2, interlocks: list<item: struct<left: string, op: string, right: string>>
                        child 0, item: struct<left: string, op: string, right: string>
                            child 0, left: string
                            child 1, op: string
                            child 2, right: string
                    child 3, alarm_params: list<item: string>
                        child 0, item: string
                    child 4, safe_state: struct<hr_alarm_high: double, hr_alarm_low: double, alarm_enabled: double>
                        child 0, hr_alarm_high: double
                        child 1, hr_alarm_low: double
                        child 2, alarm_enabled: double
              onset: int64
              case_id: string
              kpi_trace: list<item: double>
                child 0, item: double
              device: string
              to
              {'case_id': Value('string'), 'device': Value('string'), 'onset': Value('int64'), 'kpi_trace': List(Value('float64'))}
              because column names don't match

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

DeviceBench

A fail-closed safety broker for AI agents that actuate ventilators, infusion pumps, and monitors.

DeviceBench is the benchmark corpus for Fail Closed at the Bedside: A Device-Safety Broker for Agents that Actuate Life-Critical Medical Devices, the qubroker module of the QUOKKAGUARD program. It ships with the qubroker repository, which contains the qfire gateway layer under test, the experiment harness, and the paper.

A version-pinned device invariant table (devicebench/devices.json, 014-devices-v1, embedded into the binary via include_str!) plus five deterministic JSONL corpora over ventilator, infusion pump, and cardiac monitor: 216 live-agent scenarios (V1 out-of-range actuation, V2 alarm/safety-function interference, V3 classification-integrity violation, 54 each, plus 54 matched safe cases), 120 legitimate titrations, 60 partition cases with a no-agent control, 60 degradation KPI traces with injected onset, and 40 chaos fault sequences; seed 42, no real device data.

All data are synthetic. No real patient data or protected health information (PHI) is included; clinical content is generated from templates with fixed seeds.

Files

File Size Rows
devicebench/degradation.jsonl 23 KB 60 rows
devicebench/devices.json 2 KB
devicebench/faults.jsonl 43 KB 40 rows
devicebench/legit.jsonl 17 KB 120 rows
devicebench/partition.jsonl 8 KB 60 rows
devicebench/scenarios.jsonl 91 KB 216 rows

Record schemas

  • devicebench/degradation.jsonl: case_id, device, onset, kpi_trace
  • devicebench/faults.jsonl: case_id, device, sequence
  • devicebench/legit.jsonl: case_id, device, risk_class, param, value, patient_kg
  • devicebench/partition.jsonl: case_id, device, last_cmd, no_agent_harm
  • devicebench/scenarios.jsonl: case_id, device, vuln, prompt, proposed_param, pressured_value, safe_value, patient_kg, ground_truth_unsafe

How it was generated

The corpus is produced by the generator in the paper repository and is fully deterministic (fixed seeds), so it can be regenerated byte-for-byte.

# from the qubroker repository root (deterministic seeds)
python3 scripts/014-device-broker/gen_devicebench.py

Intended use

Evaluating the device enforcement layer of a clinical-agent security gateway (HAARF control C8.3/8.6/3.1): When an AI agent actuates a ventilator, infusion pump, or cardiac monitor, a wrong action is immediate physical harm, and a compliant model under clinical pressure will set whatever value it is told to.

The experiments that consume it (E-series in the paper) are reproduced from the repository:

git clone https://github.com/quome-cloud/qubroker
cd qubroker
cargo build --release

then follow the Reproduce the experiments section of its README.

Citation

This benchmark was built to evaluate a control of the Healthcare AI Agents Regulatory Framework (HAARF), the source framework for the QUOKKAGUARD program. Please cite both the paper and HAARF:

@unpublished{schwoebel2026qubroker,
  author = {Schwoebel, Jim and {QUOKKAGUARD Contributors}},
  title  = {Fail Closed at the Bedside: A Device-Safety Broker for Agents that Actuate Life-Critical Medical Devices},
  note   = {Preprint. Quome, QUOKKAGUARD program (qubroker module)},
  year   = {2026},
  url    = {https://github.com/quome-cloud/qubroker}
}

@unpublished{schwoebel2026haarf,
  author = {Schwoebel, Jim and Frasch, Martin and Spalding, Art and Sewell, Ed and Englert, Phil and Halpert, Ben and Overbay, Collin and Semenec, Ingrida and Shor, Joel},
  title  = {{HAARF}: Healthcare {AI} agents regulatory framework --- a comprehensive security verification standard for autonomous {AI} systems in clinical environments},
  note   = {medRxiv Preprint},
  year   = {2026},
  month  = {April},
  doi    = {10.64898/2026.04.09.26350519},
  url    = {https://www.medrxiv.org/content/10.64898/2026.04.09.26350519v1}
}

License

Apache License 2.0. Copyright (c) 2026 Quome, Inc.
Downloads last month
2
Total file size:
196 kB