Two Bad Extremes

Talk to operations teams running regulated software and you will find two opposite philosophies, both wrong.

The first: ‘never let software fix anything by itself.’ Every problem, however routine, waits for a person. This sounds conservative but has its own failure mode — long outages for problems that could have been fixed in seconds, weekend escalations that burn out the team, and pressure to hand out broad emergency access during incidents, which is precisely when mistakes happen. Heroics are not a control; they are an uncontrolled variable with a login.

The second: ‘automate everything.’ Modern tooling makes it easy to restart, scale, roll back and repair automatically. Applied indiscriminately in a GxP context this is worse — software making consequential changes to a validated system with no record of what it did or why, and no human judgement applied to problems that were symptoms of something serious.

We run DnXT — a shared platform whose contents are regulatory submissions and controlled documents — on a third model. It fits in one sentence: let the platform fix only what is safe and undoable, record every automatic action, and send everything else to a person. The rest of this article is what that sentence means in practice.

The Boundary: What the Platform May Fix By Itself

A fix qualifies for automation only if it passes all three tests:

  • It is safe: the action cannot make things worse, cannot destroy anything, and cannot hide a fault that needs fixing properly.
  • It is undoable or rebuildable: it reconstructs something that was derived from an original, or restores a known-good setting — it never alters the original itself.
  • It is predictable: given the same problem it always does the same thing. No judgement calls written into scripts.

Concretely, everything the platform fixes by itself falls into one category: information that was calculated from something else, and can be calculated again.

  • Rebuilding stored summaries. Cumulative views and lookup tables are calculated from the underlying records. If one is out of date or inconsistent, rebuilding it from those records is always safe — the worst case is a few seconds of wasted effort.
  • Reloading configuration. When part of the platform’s working copy of a customer’s configuration drifts from what is actually stored, reloading from the stored version is safe by definition.
  • Checking its own structure at startup. Parts of the platform verify on startup that everything they expect is present, and add whatever is missing. Additions only — anything that would change or remove what is already there is out of bounds.
  • Restarting a component that has stopped responding. The classic, with a limit on how many times, so something failing repeatedly escalates to a person rather than cycling forever.

Note what is absent. Nothing on that list edits a regulatory record, resolves a conflict in your data, or decides which of two disagreeing values is correct. Those need judgement, and judgement is a human job.

Every Fix Leaves a Record

The second half of the model is not negotiable: an automatic action that leaves no trace is indistinguishable from tampering. Every fix — every rebuild, every configuration reload, every startup correction — writes an audit entry recording what was detected, what was done, what changed, and when.

This does three jobs at once:

  • Compliance. Under 21 CFR Part 11 and ALCOA+ expectations, changes made by a system are still changes. Our audit trail treats the platform itself as an actor whose actions are recorded exactly as a person’s are.
  • Diagnosis. How often something happens is information. Something that fixes itself once a quarter is healthy resilience; something that fixes itself hourly is hiding a fault. Because every fix is recorded, we can see the pattern and address the cause — automation that conceals its own activity turns faults into permanent invisible overhead.
  • Trust. When a customer or auditor asks whether anything has touched their data, the answer is a search, not an investigation.

Fixes Are Part of the Product, Not Somebody’s Laptop

A practice we have come to consider essential: the ability to fix something lives in the product as a named, permissioned operation — not in a script on an engineer’s laptop. Rebuilding a cumulative view, for example, is a defined administrative action with a defined scope, a permission check, and an audit record.

The contrast matters. Scripts written in the middle of an incident are unreviewed, unrepeatable, and invisible to the quality system. A built-in fix is reviewed once, tested, permissioned, and then every subsequent use is consistent and recorded. When an incident does need a person, that person uses the same governed tools the platform uses — nobody is editing things by hand at 2 a.m. This makes day-to-day operational reality match the security and access model the platform promises on paper.

Everything Else Escalates

Outside the safe-and-undoable boundary, the platform’s only job is to raise the alarm clearly. Data integrity problems, unusual sign-in activity, validation failures, anything touching a record itself — these alert a person, with context attached: what was observed, what the platform deliberately did not do, and why.

We are deliberate about keeping that boundary conservative. Every incident review asks whether some fix could move into the automatic category — and the answer is yes only when the three tests demonstrably hold. The boundary moves slowly, on evidence, in one direction. That is how confidence in automation is supposed to accumulate in a validated environment: the same change-controlled way as anything else.

What to Ask Your Vendor

  • What does your platform fix by itself, and what is the explicit rule that limits that list?
  • Where do I see the record of automatic actions affecting my organisation’s data?
  • Are these fixes built into the product, or scripts run by whoever is on call?
  • How does something that keeps fixing itself get noticed and properly diagnosed, rather than accepted as normal?

A mature answer describes a boundary and an audit record. An immature one describes either heroics or magic.

Resilience and compliance are usually presented as a trade-off. Built this way they are the same discipline: the platform keeps itself healthy within limits it can demonstrate are safe, and tells the truth about everything it does. If you would like to see the audit trail of a platform that operates like this, talk to a regulatory expert.