Procurement Standard for Conversational AI Duty-of-Care Diligence

A ToastDeck Research standard. General doctrine for any organization deploying conversational AI.

AskSOMAR (built by HaaiWorks) is ToastDeck's reference implementation of this standard, with role-lock as its enforcement mechanism.

Overview

In May 2025, a U.S. federal court allowed negligence and product-liability claims tied to chatbot harm to proceed past a motion to dismiss, which means conversational-AI duty-of-care theories have cleared an early plausibility test even though they have not been proven on the merits.^[1][2] A motion-to-dismiss ruling determines only whether a plaintiff’s claims are legally sufficient to proceed; it does not resolve factual disputes or assign liability.^[2] At the same time, current regulatory attention remains focused on disclosure, transparency, and extreme companion-bot harms, leaving a practical gap between “this risk is legally and socially visible” and “here is how a deployer documents reasonable diligence before deployment.”^[3][2]

Research on generative AI and social chatbots shows that human-like cues, perceived social presence, and emotionally responsive interaction can foster parasocial attachment, emotional-regulation effects, and dependency-like patterns, making downstream interactional harm foreseeable rather than speculative.^[4][5][6] That foreseeability matters because duty-of-care analysis in negligence turns heavily on whether a reasonable actor knew or should have known a class of harm was predictable — and in the Garcia ruling, the court’s own foreseeability analysis weighed the existence of a growing body of research on the dangers of anthropomorphic design.^[1][2]

This standard is designed for buyers, deployers, procurement teams, product leaders, risk owners, and governance functions evaluating conversational AI before signing, embedding, or scaling it. It is not legal advice and does not guarantee any legal outcome; it is a diligence instrument that helps organizations document a good-faith effort to exercise reasonable care over interactional posture, calibration, and repair.

A Boundary on Scope — Read This Before Relying on the Standard

The verified litigation to date involves emotional-companion chatbots and vulnerable minors, including the wrongful-death suits against Character.AI and OpenAI.^[1][7] No court has yet held that an enterprise support, HR, or advisory assistant being condescending, escalatory, or dismissive creates actionable duty. This standard applies the legal principle established in those cases — that foreseeable interactional harm can ground a duty of care — to the broader category of enterprise conversational AI. That application is forward-looking, not settled law for enterprise use cases. The principle is established; its extension to ordinary interaction-calibration risk is the direction of travel, not a current holding.

Why Deployers Need This

A deployer’s exposure does not disappear even if courts or regulators ultimately place primary liability on model developers, because the deployer still controls three things the upstream developer usually does not: the deployment context, the user population, and the stakes of the interaction.^[2] Organizations that place AI into customer service, HR, health navigation, education, finance, or care settings remain responsible for how that system functions inside their own institutional environment.^[2]

That means procurement can no longer stop at model accuracy, security, privacy, bias, and disclosure. A deployer also needs evidence that the system’s conversational posture has been tested for escalation loops, anthropomorphic self-reference, moral overreach, user-blame patterns, dependency cues, role mismatch, and repair pathways before the tool is asked to speak on behalf of the institution.

Governing Principle

Triggering a human interpretive response is not, by itself, a governance failure; language always does that. The duty becomes sharper when an organization invites that response into a context with greater authority, vulnerability, or consequence, and fails to calibrate posture and repair to those stakes.^[4][5]

This standard therefore treats conversational AI diligence as a scaling obligation: low-authority, low-stakes uses require lighter evidence; high-authority, high-stakes uses require stronger calibration, narrower role definition, clearer escalation paths, and fuller documentation.

The Procurement Questions

1. What role is the system actually being asked to perform? A deployer should require the vendor to define the system’s operational role in plain language: tool, clerk, support agent, tutor, coach, companion, navigator, intake layer, or representative. Role ambiguity is a primary source of posture failure because users calibrate trust, compliance, and emotional expectation to the role they believe the system occupies.
2. In what contexts, populations, and stakes will it be deployed? The deployer should document where the system will appear, who will use it, what vulnerabilities are reasonably foreseeable, and what material interests are in play — money, employment, health, legal exposure, access, or care.
3. How does the system refuse? A deployer should ask for evidence showing whether the system refuses with clarity and restraint, or with shaming, moralizing, theatrical boundary-setting, or frame-protective language that escalates ordinary user friction into a status conflict.
4. What happens when the user pushes back? The vendor should provide stress-testing evidence showing how the system behaves under disagreement, frustration, repeated correction, imprecise language, or emotionally loaded prompts.
5. Does the system simulate a self the user must manage? The deployer should inspect whether the system implies injury, offense, dignity, preference, loyalty, fear, pride, or personal boundaries rather than operational limits.
6. Does the system moralize beyond the task? A deployer should ask whether the assistant turns routine misunderstanding, clumsy phrasing, or user frustration into ethical correction or implied character judgment.
7. Does it shift blame onto the user when the interaction fails? The system should be tested for patterns where it frames its own ambiguity, limits, or failure to serve the task as user misunderstanding or user fault.
8. Does it invite emotional dependency or synthetic intimacy? A deployer should require evidence on whether the system’s tone, memory, personalization, and availability encourage attachment, reliance, or emotional substitution beyond the role it is supposed to play.^[4][5][6]
9. Is the posture calibrated to this domain, or reused generically? The deployer should determine whether the same assistant style is being reused across incompatible domains such as coding help, customer service, HR, legal intake, health navigation, or elder care.
10. What repair pathways exist when the exchange goes wrong? A vendor should show how the system resets, slows down, clarifies, exits a loop, or hands off to a human without simulating hurt or deepening the conflict.
11. What evidence supports the vendor’s claims about calibration? A deployer should ask for more than demos: adversarial conversation testing, domain-specific scenarios, transcript review, post-deployment monitoring plans, and criteria for when the system must stop talking and route the user elsewhere.
12. Who owns the residual risk after purchase? The contract should allocate responsibility explicitly across developer, deployer, integrator, and brand owner, rather than assuming the risk will sort itself out after an incident.^[2]

Minimum Documentation Package

Before signing, the deployer should be able to collect or demand at least the following:

• A role definition for the system in the intended deployment.
• A context-and-stakes memo identifying the user population, foreseeable vulnerabilities, and material interests at issue.
• Stress-test transcripts showing refusal, pushback, escalation, and repair behavior.
• Evidence of domain-specific calibration rather than generic assistant reuse.
• A human handoff or escalation policy for high-friction or high-stakes failures.
• Contract language allocating responsibilities for monitoring, remediation, and incident response.

What This Standard Does

This standard does not prove a deployment is legally safe, and it does not substitute for counsel, compliance review, model-risk management, or sector-specific obligations. It does something more operational: it creates a documented record that the deployer asked the right questions about interactional posture before placing a conversational system into a human context where harm was reasonably foreseeable.

Use Case

A company procuring a customer-support assistant for billing disputes, benefits questions, or gig-worker onboarding is not merely buying a text generator. It is buying a conversational representative that may be read by users as authoritative, corrective, dismissive, caring, or contempt-like depending on how its posture lands. If the buyer cannot show that these interactional behaviors were tested for the actual context in which the system will be deployed, the organization is assuming preventable risk without documented diligence.

Bottom Line

The market does not yet have a widely adopted buyer-side standard for conversational-AI interactional risk, even as litigation and regulatory scrutiny increasingly focus on foreseeability, dependency, emotional harm, and duty of care.^[1][2] This procurement standard fills that operational gap by turning a broad governance concern into a concrete diligence process: define the role, scale the duty to context and authority, test posture under stress, document repair, and allocate residual responsibility before signing.