Evaluating AI in Regulatory Technology
As AI capabilities expand in pharma AI regulatory technology, the evaluation process for buyers is becoming more complex. Traditional vendor assessments focus on features, pricing, and references. AI-powered platforms require additional scrutiny around compliance, data integrity, and operational safety.
This guide provides 12 questions that regulatory evaluation teams should ask when assessing AI-powered regulatory technology. For each question, we explain why it matters, what a good answer looks like, and — in the interest of transparency — how DnXT answers it. We’ve also noted where our own answer has room for improvement.
Audit Trail and Data Integrity
1. “Is your audit trail structural or bolt-on?”
Why it matters: When AI agents generate hundreds of data operations per session, audit gaps become statistically likely if logging is opt-in. Structural audit trails — built into the data write layer — close this gap.
Good answer: “Our data access layer automatically generates audit records on every write operation. Developers cannot bypass it.”
Red flag: “Our developers follow best practices for audit logging” (translation: it’s manual and sometimes forgotten).
DnXT’s answer: PlatformDataWriter automatically calls audit-service on every INSERT, UPDATE, and DELETE. ALCOA+ compliant with SHA-256 tamper detection. Honest gap: we haven’t yet captured which AI model version influenced a given action.
2. “How do you distinguish AI-initiated actions from human actions in the audit trail?”
Why it matters: Regulators will increasingly ask whether a specific action was taken by a human or an AI. Your system needs to answer this without post-hoc analysis.
Good answer: “Every audit record includes a source field that identifies whether the action came from the UI, an API, or an AI agent.”
Red flag: “All actions are logged the same way” (translation: you can’t tell human from AI).
DnXT’s answer: Source field values include publisher-ui, reviewer-ui, mcp-open-gxp, and system. Auditors can filter by source. Honest gap: the MCP source attribution is part of our design, not yet live in a customer-facing server.
Security and Access Control
3. “Can AI agents access other tenants’ data?”
Why it matters: In a multi-tenant system, an AI agent querying data doesn’t have the contextual awareness to notice if it receives wrong-tenant data. Isolation must be enforced architecturally.
Good answer: “Tenant identity is resolved server-side from the API credential. The AI agent’s API surface doesn’t expose a tenant parameter. Even if application logic failed, database-level schema separation prevents cross-tenant queries.”
Red flag: “We add a tenant filter to all queries” (translation: it’s a WHERE clause that depends on developer discipline).
DnXT’s answer: Four-layer isolation: HTTP header inspection, ThreadLocal context, database schema routing, permission enforcement. Fail-loud — no silent fallback if tenant can’t be determined.
4. “Can AI agents sign documents or bypass approval workflows?”
Why it matters: 21 CFR Part 11 requires electronic signatures to be attributable to specific individuals. AI agents are not individuals. The system must enforce this, not just document it as policy.
Good answer: “There is no API endpoint that allows an AI agent to apply an electronic signature. The capability doesn’t exist at the technical level.”
Red flag: “Our AI respects signing workflows” (translation: it’s a prompt instruction, not an architectural block).
DnXT’s answer: No esign_sign tool in the MCP server design. AI agents can request signatures (creating PENDING records) but cannot sign. E-signature gates in the workflow engine return requiresHumanAction: true.
AI Behavior and Safety
5. “What happens when the AI is wrong?”
Why it matters: AI will make mistakes. The question is whether the system provides a preview mechanism before AI recommendations become actions.
Good answer: “AI recommendations go through a preview step where users see what will happen before confirming. The system logs both the AI’s recommendation and the human’s decision.”
Red flag: “Our AI is highly accurate” (doesn’t answer the question).
DnXT’s answer: The preCheck endpoint evaluates transition rules without executing them. AI agents receive warnings and blocks before attempting actions. Human review gates at classification, validation, and sign-off.
6. “Is the AI locked to one provider?”
Why it matters: AI technology is evolving rapidly. Being locked to one provider limits your ability to adopt improvements or negotiate pricing.
Good answer: “We use open protocols (MCP) that work with any AI provider. You choose the AI; we provide the compliance infrastructure.”
Red flag: “Our AI features require our proprietary AI service” (lock-in risk).
DnXT’s answer: MCP is provider-agnostic. Our internal tools were built for Claude Code, but the customer-facing Open GxP design works with any AI agent that supports MCP — Claude, GPT, Gemini, or custom models. Honest gap: we’ve only tested extensively with Claude.
Compliance and Validation
7. “What regulatory standards does your AI integration comply with?”
Why it matters: Vague compliance claims are common. Specific standard references are testable.
Good answer: “Our audit trail meets 21 CFR Part 11, our e-signature service addresses Sections 11.50, 11.70, 11.100, and 11.200 specifically, and we map to EU Annex 11 data integrity requirements.”
Red flag: “We’re compliant with all major regulations” (too vague to verify).
DnXT’s answer: AuditTrailWriter: 21 CFR Part 11, ALCOA+ data integrity. ESignatureService: Part 11 Sections 11.50, 11.70, 11.100, 11.200. OQ Framework: IQ/OQ/PQ qualification evidence. Honest gap: no regulatory authority has reviewed or endorsed our AI-specific compliance approach.
8. “Can I see a validation report for the AI features?”
Why it matters: Marketing materials demonstrate intent. Validation reports demonstrate evidence.
Good answer: “Here’s our current OQ report, including test case results, requirement traceability, and risk assessments for AI-specific features.”
Red flag: “We can provide that after contract signing” (the documentation may not exist).
DnXT’s answer: OQ Framework with @OQTest annotations linking test cases to requirements and regulatory references. SHA-256 hash chain on test execution records. OQ Dashboard shows live qualification status. Honest gap: our validation documentation for AI-specific features is still being formalized as the features mature.
Operations and Support
9. “What is your deployment model?”
Why it matters: Data residency, network isolation, and regulatory requirements vary by region and organization. Flexibility matters.
Good answer: “We offer cloud deployment with VNet isolation, region-specific data residency, and the option for dedicated infrastructure for sensitive workloads.”
DnXT’s answer: Azure Container Apps with standard and secure (VNet-isolated) environments. US East data residency currently. Honest gap: we don’t yet offer on-premise deployment or EU data residency.
10. “How do you handle AI model updates?”
Why it matters: AI models change. A model update could alter classification accuracy, recommendation quality, or validation behavior. Change control must extend to the AI layer.
Good answer: “AI model updates go through our change control process. We run regression tests comparing new model outputs against baseline results before deployment.”
Red flag: “Models update automatically to give you the latest improvements” (no change control).
DnXT’s answer: Honest gap: this is an area where we’re still developing our process. AI model updates from providers like Anthropic are not currently gated through our change control system. We recognize this needs to be addressed.
11. “What is your incident response process for AI errors?”
Why it matters: When AI classifies a document incorrectly and it gets included in a submission, what’s the remediation process?
Good answer: “We have a documented incident response process that includes root cause analysis, affected-entity identification, customer notification, and corrective action — with all steps audit-trailed.”
DnXT’s answer: OQ Dashboard detects compliance issues via automated agents. Deviation tracking with resolution timelines. Audit trail captures the full chain of events. Honest gap: our incident response process is more mature for infrastructure incidents than for AI-specific errors.
12. “Can I try it with a real submission before committing?”
Why it matters: A demo environment with sample data is useful. A pilot with your actual documents and workflows is evidence.
Good answer: “We offer a 30-60 day pilot program where you use your real documents in a sandbox environment, with full support and no commitment.”
DnXT’s answer: We support pilot programs with sandbox tenants. Your data stays isolated in its own schema. We’ll help configure workflows specific to your submission types. No gap here — this is something we do well.
The best evaluation questions are the ones that make vendors uncomfortable. If a vendor answers every question perfectly without acknowledging a single limitation, be skeptical. Honest vendors — including us — have gaps. The question is whether they know what their gaps are and have plans to address them.
Using This Guide
These 12 compliance questions work regardless of which vendor you’re evaluating — including DnXT. The goal is to help pharma AI regulatory evaluation teams make informed decisions based on verifiable answers rather than marketing claims.
If this guide helps you choose a competitor over us because they genuinely answered these questions better, that’s a good outcome. The industry benefits when buyers ask harder questions and vendors respond with substance.
This article was written by the DnXT Solutions team. We’ve included our own honest answers — including our gaps — because we believe transparency builds more trust than perfect marketing. Questions or feedback: se******@***********ns.com.
