The Worst Bug Is the One That Says ‘Success’

Every experienced engineer has a mental ranking of bugs. Crashes are annoying but honest — they announce themselves, someone files a ticket, the fix ships. The bugs that keep us up at night are the quiet ones: the operation that fails and reports success anyway. In consumer software these cost you a lost photo. In regulated software they cost you data integrity, and data integrity is the whole game.

We have come to state this as a design principle: in a GxP context, a false success is worse than a crash. A crash interrupts work. A false success corrupts the record while everyone believes it is intact — and the gap between belief and reality is precisely what regulations like 21 CFR Part 11 and the ALCOA+ principles exist to prevent.

This post is a confession as much as a philosophy. Every example below is a silent failure we found in our own platform, fixed, and then engineered against structurally. We are sharing them because the patterns are universal — and because we think buyers of regulatory software should know how to probe for them.

Three Silent Failures We Caught

1. The email that said ‘sent’

Our notification service, like most, wrapped an SMTP call. For a period, when the mail server rejected a message, the wrapping service still returned success to its callers — the failure was logged deep in a container log nobody was reading, and the calling application cheerfully told the user the notification was sent. Users made decisions on the assumption that reviewers had been notified. They had not.

The fix was philosophically simple: the service now returns a true status — sent means sent, and a delivery failure surfaces as an explicit error the calling application must handle and show. The deeper fix was auditing every service boundary in the platform for the same pattern, because a swallowed error at any hop makes every upstream status a lie.

2. The reindex that swapped in nothing

Search infrastructure periodically rebuilds its index and swaps the new one in atomically. One rebuild path could, under specific failure conditions, complete the swap with an empty index — and report the job as successful. From the user’s perspective, search ‘worked’ but returned no results, which in a regulatory review context reads as ‘this document does not exist.’ A user searching for a commitment they need to honor and finding nothing is a compliance incident waiting to happen.

The corrected behavior refuses to swap unless the new index passes sanity checks against the old one, and a rebuild that produces an implausibly empty result fails loudly with an alert. An index rebuild should never be able to silently destroy the search surface it was meant to refresh.

3. The validation check that rendered as a blank box

Our submission validation engine runs well over a hundred criteria per region against an eCTD sequence. We discovered that certain internal errors in individual checks could result in a criterion producing no result at all — which the report rendered as a blank box, visually indistinguishable from a check the user had chosen not to run. A user could read that report as clean when a check had actually died mid-flight.

The structural fix is what we call a result guard: an invariant that every selected criterion must emit exactly one result — pass or fail. Zero results is a bug; the guard converts it into an explicit failure with the internal error attached. A validation report must never contain silence where a verdict belongs. (For context on what these criteria check, see our guide to eCTD validation rules.)

The Discipline: Rules That Prevent the Whole Class

Individual fixes do not prevent recurrence; disciplines do. Ours distills to a few enforceable rules:

  • Every operation reports true status. No service returns success on behalf of an operation it did not confirm. HTTP 200 means the thing happened, not that the request was received.
  • No swallowed exceptions at service boundaries. An error that crosses a boundary must either be handled meaningfully or propagated visibly — never logged-and-ignored.
  • Every check emits exactly one result. Validation and health checks are governed by result guards: silence is converted into failure.
  • Destructive swaps require sanity checks. Any operation that replaces live data (indexes, caches, configuration) must prove plausibility before committing.
  • Fail-loud beats fail-safe defaults. When in doubt, stop and tell a human. In GxP work, availability is negotiable; integrity is not.

These rules are enforced in code review and, where possible, structurally — the result guard is not a convention, it is a wrapper the criterion code runs inside. Conventions decay; structures persist.

Why This Is a Compliance Issue, Not Just an Engineering One

ALCOA+ requires records to be accurate and complete. A system that reports success for failed operations manufactures records that are neither — and does so at scale, invisibly. Auditors increasingly understand this: questions about error handling, monitoring, and failure modes are appearing in vendor qualification audits, and rightly so. The audit trail of a system that swallows errors is a beautifully formatted account of a reality that did not happen.

There is also a trust dimension. Regulatory teams live or die by their systems’ word. The first time a user discovers the system said ‘sent’ when it did not send, they start keeping shadow spreadsheets — and shadow spreadsheets are where data integrity goes to die. Fail-loud engineering is how software earns the right to be the single source of truth.

What Buyers Should Ask

You cannot code-review your vendor, but you can probe the discipline:

  • ‘Tell me about a silent failure you found in your own product and how you fixed it.’ Vendors with real discipline have war stories; vendors without it have marketing.
  • ‘What guarantees that a validation report shows a result for every check that ran?’
  • ‘If your notification pipeline fails, what does the user see?’
  • ‘What operations in your platform can replace live data, and what checks gate them?’

The honest answers are specific and slightly embarrassing. The dishonest ones are confident and vague.

Talk to Us

We build DnXT on the assumption that our software will be the system of record for decisions that end up in front of health authorities — which means it has to tell the truth even when the truth is a failure. If that engineering culture matches what you want from a regulatory platform, talk to a regulatory expert. Bring your hardest questions about failure modes; we enjoy them.