Prio 1: CE not registering newly activated patients

Author: Daan Klomp
april 10, 2024

Production Issue Summary

In the clinical engine, there was an inefficient query when associating active patients to a submitter. This resulted in CE giving errors whenever a patient is being activated and the activation not being registered in CE. This means the patient is active in vitals and can do measurements but since in CE the patient is not linked to the submitter, this won’t generate measurements in the clinical engine. This only affects patients that logged in for the first time in the period since the issue started occurring.

Timeline of Events

Date Event
@Associate and dissociate operations are slow to the point of causing timeouts
@Associate and dissociate operations are slow to the point of causing timeouts status: resolved - Dissociate is resolved
Alarms in CE goes off, assumption is that automatic scaling will fix the issue
Developers conclude CE being down
Possible issue being investigated
Issue found in code and starting to work on fix
PIC was informed and process started
Affected patients till that point identified
Instatus update published
CE needs a release instead of only hotfix since release pipeline was not in place after stopping to use octopus
Testing CE release on acceptance
Release deployed to production
Retry of all failed associations finished
Due to some format changes in the release, some retries related to scripted observations failed to run and can’t be fixed manually
23 patients identified that were in the end affected because of the failed retries and therefore some calculations not don correctly, all other cases have been fixed
Customers of the 23 affected patients have been informed by email, communication was a bit delayed because of lack of information from production issues communicator


Lead time: 18h

Work around time: N/A

Correction time: 9h

 

Impact

Patients that were activated after March 27th at 14.00h until March 28th, 17.08h were affected. 40 patients in total were affected from 23 organizations. All measurements that were sent in, were received by the clinical engine a bit later. Because this is on the first day the patients started, and there is no strict control normally over when a patient starts, we do not consider this issue to be of any clinical patient safety risk.

Classification: High

Workaround

N/A

 

Cause

The number of active patients being a limiting factor in the submitter association query. We got to a number of active users that was too much for this query with the number of parameters.

Solution

Optimize the faulty query.

 

Communication and documentation

External

  • InStatus was used and affected organisations where contacted directly as well

Internal

  • Normal communication in slack and according to process. Nothing special to report.

Improvements

 

  • Investigate long running queries (CAPA-3)
  • Improve sensitivity and specificity of monitoring of the clinical engine to reduce lead time (CAPA-3)
  • Clinical Engine Hotfix mechanism (CAPA-2)

Subscribe

Subscribe to our newsletter & stay updated