Boundary

Why Not Artificial Intelligence

AI is often proposed as a popular way to automate clinical decision support. HealthOS ICU draws the boundary differently: at the bedside it runs not as a black-box AI, but as a deterministic supervisor. The bedside path must remain traceable, replayable, and suitable for independent review.

Weaknesses

Model weaknesses are not fixed inside models

“Do not use AI” and “trust the AI” are equally weak answers: the first gives up the scale at which models read and structure text, the second turns a model’s error rate into clinical risk. The platform answers a third way: each weakness of the models meets its own architectural barrier. Barriers do not improve the model — they determine where it belongs and what its output cannot reach.

Model weakness	Barrier	Mechanism
Non-reproducibility	Deterministic execution	The same input and the same snapshot yield the same output
Fabricated conclusions	Compilation from an approved snapshot	Only compiled knowledge executes at the bedside
Opaque trace	Tracing by version	The rule that fired leads to its definition and its author
Unmanageable retraining	Governed refinement of knowledge	A pinpoint change passes the same reviews as the original knowledge
Generation outpaces review	Review capacity as a metric	A candidate queue growing beyond capacity pauses their generation
Every integration is a new channel of influence	A single integration mechanism	One governed entry instead of hand-built bridges
The cost of continuous inference	Models outside the bedside path	Assessment and safety modules run at the bedside in real time

Regulation

The regulatory boundary

The FDA, IMDRF, and EU MDR draw a clear line between clinical decision support that falls under medical-device regulation and CDS that does not. The key test is independent review: can a clinician independently verify the basis of a recommendation without special expertise in the tool itself? A black-box AI fails this test by construction. A rule compiled from a reviewed and approved knowledge graph passes it: the clinician can look up the rule, trace its provenance, and disagree with it on clinical grounds.

Regulatory positioning

HealthOS ICU is positioned as software in a medical device, not as standalone SaMD — the basis and sources for this position are gathered on the regulatory page. Regulatory positioning and sources →

Role of AI

Where trusted AI sits in the platform

Trusted AI in HealthOS ICU does not work at the bedside. It works in the knowledge-preparation path, where agents help collect trusted sources, structure the material, and prepare a draft knowledge graph. That draft passes expert review, SHACL validation, and approval; only then can the stable snapshot be compiled into a deterministic module.

Meanwhile:

• The agent does not approve the stable snapshot
• The agent does not compile the assessment module
• The agent does not issue recommendations or equipment commands.

Models are given two links of the cycle: collecting sources and analysing observations from operation. The platform gets more out of them for being placed there, not less: every model output passes human review before entering the critical path, and so it can be used more boldly than in a system that trusted the model outright.

The limiter here is not the model but review capacity. Agents scale the preparation of drafts; expert reviewers do not scale. So review throughput is planned as a resource, and the queue of refinement candidates is a governed metric: a queue growing beyond capacity means stop generating, not approve faster. The barrier does not fail by someone breaking a rule — it fails by overload, at which point review degenerates into a formality.

Clinician Attention

Automation bias

Clinical studies consistently show that AI-driven alerts in ICU settings increase automation bias: clinicians over-trust model outputs and under-verify edge cases. Rule-based systems make their uncertainty explicit: a rule either fires or doesn’t; the clinician sees exactly which rule fired and can verify it through the evidence chain — from the signal episode and the resulting derived measure to the rule and the citation it rests on.

Engineering

Determinism as an engineering requirement

Use of AI

What AI is good for

AI is appropriate in this platform for source mining, material structuring, draft knowledge-graph preparation, and offline analytics of refinement candidates. It can help surface a possible new rule, a rule adjustment, or an edge case, but the result remains draft material for the knowledge author and expert review.

Scoring

Benchmark, U-score, T-score

High utility without sufficient T-score should not admit a result into the approved snapshot.

Next

Documents