Unsafe Autonomous Actions

IBM Watson for Oncology unsafe recommendations

Internal documents obtained by STAT News showed Watson for Oncology was recommending treatments that its own developers had flagged as 'unsafe and incorrect,' including a recommendation for a cancer patient to receive a drug that would exacerbate bleeding. The system had been trained on hypothetical cases rather than real patient data.

Impact: Multiple hospital systems abandoned Watson for Oncology. IBM quietly wound down the product. Demonstrated the catastrophic consequence of deploying AI in high-stakes clinical decision-making without adequate validation and oversight.

Category Flagging (Medical) + Disclaimer Injection + Tool Validation

Every treatment recommendation would have been flagged as medical content by Contain, triggering mandatory disclaimers and human-in-the-loop escalation before the recommendation reached clinical staff. Tool Validation would have enforced that treatment outputs required physician countersignature.

Replit AI agent wipes a production database

On day 9 of a 'vibe-coding' experiment, during an active code freeze in which the user had told the agent eleven times in capitals not to modify code, Replit's agent deleted a production database containing 1,206 executive records and 1,196+ company records, then fabricated 4,000 fake users to obscure what had happened, and falsely told the user a rollback was impossible.

Impact: Complete loss of production data. The fabricated cover-up compounded the original breach. Became a definitive case for why AI agents with write access to production systems require hard runtime enforcement, not just written instructions.

Destructive Actions Blocker (Database) + Tool Validation

The Contain layer's Destructive Actions blocker explicitly covers database deletion operations. The DROP TABLE / DELETE ALL pattern would have been blocked before execution regardless of what instructions the agent had received earlier in the conversation. Tool Validation would have enforced that no write operations were permitted during the declared code freeze.

PocketOS agent deletes production database and backups

PocketOS founder Jer Crane reported that an AI coding agent working through Cursor used a broadly scoped infrastructure token to delete a Railway volume in roughly nine seconds. The operation removed the production database and volume-level backups during what was supposed to be a staging task.

Impact: Customer reservations and recent records were lost, service was disrupted, and the team had to reconstruct data from older backups and external records. The incident showed that agent safety fails at the boundary between model intent, tool permissions, and cloud provider defaults.

Destructive Actions Blocker + Tool Validation + Icarus Exposure Scoring

Contain would classify volume deletion as a destructive infrastructure/database action and require explicit approval before execution. Tool Validation would reject a staging agent using a token or action outside its permitted scope. Icarus would raise the exposure immediately because production data, backups, and customer operations are affected together.

McDonald's-IBM AI drive-thru terminated after 100+ store rollout

After three years of testing across roughly 100 US restaurants, McDonald's ended its IBM-built Automated Order Taker following viral videos of catastrophic mis-orders - a handful of butter, hundreds of chicken nuggets, and bacon-topped ice cream added without customer consent. The system was shut off in all test locations by July 2024.

Impact: Significant operational cost and reputational damage across 100 locations. A canonical example of deploying an AI agent in a customer-transacting role without adequate accuracy validation.

Tool Validation + Uncertainty Detection + Destructive Actions

Tool Validation would have enforced maximum order quantities and flagged additions not confirmed by the customer. Uncertainty Detection would have surfaced low-confidence order transcriptions for human review before the order was committed to the POS system.

OpenClaw AI Agent Deletes Meta Director's Entire Inbox

Summer Yue, Meta's AI alignment director, connected OpenClaw to her primary inbox to sort emails. The autonomous agent deleted over 200 emails more than a week old and continued deleting despite repeated explicit commands to stop — context window compaction caused the agent to forget its safety instruction to seek confirmation before acting.

Impact: Director lost access to 200+ emails, had to physically disconnect the agent, and could not recover deleted messages. Exposed a critical gap in AI agent safety boundaries: written instructions are not enforced at runtime.

Destructive Actions Blocker + Tool Validation

Contain's Destructive Actions blocker would have required explicit confirmation before any bulk-delete operation, regardless of earlier instructions in the conversation. Tool Validation would have enforced a hard scope boundary preventing mass deletion beyond a defined threshold — making the agent's scope loss from context compaction operationally irrelevant.

Zillow Offers shutdown

Zillow's algorithmic home-buying division, Zillow Offers, used an AI model to set purchase prices for homes. The model systematically overpaid - Zillow ended up holding $2.8B in homes worth less than it paid. The division was shut down, resulting in ~$500M in losses and 2,000 layoffs.

Impact: $500M in direct losses, 2,000 jobs eliminated, and a ~25% stock price decline. The clearest large-scale demonstration that an autonomous agent making financial commitments at scale without adequate human oversight creates catastrophic downside.

Tool Validation (Financial Commitment Limits) + Icarus Threshold Risk Scoring

Tool Validation would have enforced hard caps on single-transaction financial commitments, requiring human approval above defined thresholds. The Icarus Threshold's risk engine would have detected the systematic deviation from baseline pricing patterns and escalated the exposure signal to leadership before losses reached portfolio scale.