Prio 2: Missing Message Templates

Post Mortem

Production Issue Summary

For some protocols, their message templates could have been accidentally deleted when other protocols in the same program were deleted. This resulted in certain workflows not sending all messages anymore to the patients. The issue became clear when we cleaned up older programs. The issue originated from a past change when we allowed programs to have multiple protocols. Newer programs (after September 2023) where not affected.

Timeline of Events

Untitled

Lead time: 15 min

Correction time: 69 hours

Impact

11 organizations and 1101 patients were affected in the last 30 days. We consider this a critical issue since message templates can include medically relevant information for patients. Given the time range and the number of impacted patients, we consider this a patient safety risk. Most affected message templates were not medically relevant, so, as such, there was limited exposure to patient safety.

Workaround

N/A

Cause

A technical issue when migrating from programs with one protocols to programs with multiple protocols. In particular, the foreign key of message templates per protocol was not cleaned up after migration of data model to new structure with multiple protocols per program, therefore when deleting the primary protocol in the program, the messages templates were also deleted.

Solution

Correction:

• Where possible message templates were corrected from backups (everything between February and after 23rd of August) .
• Caregivers were informed about missed messages to their patients.

Corrective action:

• Foreign key has been removed and data model corrected. The issue has been resolved for future cases.

Communication and documentation

External

• Communication to affected hospitals done directly.

Internal

• A post mortem was written. You are reading it.
• Slack was used, and communication internally went normally.

Improvements

• Improved error logging with any case that might cause negative effects for patients (monitor more, and monitor more clearly). We need to be especially vigilant where it affects automated actions that the user might not be aware of and relies upon. (CAPA-1)
• Define message guidelines and show in product where messages are defined (CAPA-4)
• Create more frequent backup snapshots (CAPA-3)