Production Issue Summary
In the clinical engine, there was an inefficient query when associating active patients to a submitter. This resulted in CE giving errors whenever a patient is being activated and the activation not being registered in CE. This means the patient is active in vitals and can do measurements but since in CE the patient is not linked to the submitter, this won’t generate measurements in the clinical engine. This only affects patients that logged in for the first time in the period since the issue started occurring.
Timeline of Events
Lead time: 18h
Work around time: N/A
Correction time: 9h
Impact
Patients that were activated after March 27th at 14.00h until March 28th, 17.08h were affected. 40 patients in total were affected from 23 organizations. All measurements that were sent in, were received by the clinical engine a bit later. Because this is on the first day the patients started, and there is no strict control normally over when a patient starts, we do not consider this issue to be of any clinical patient safety risk.
Classification: High
Workaround
N/A
Cause
The number of active patients being a limiting factor in the submitter association query. We got to a number of active users that was too much for this query with the number of parameters.
Solution
Optimize the faulty query.
Communication and documentation
External
- InStatus was used and affected organisations where contacted directly as well
Internal
- Normal communication in slack and according to process. Nothing special to report.
Improvements
- Investigate long running queries (CAPA-3)
- Improve sensitivity and specificity of monitoring of the clinical engine to reduce lead time (CAPA-3)
- Clinical Engine Hotfix mechanism (CAPA-2)