What Just Happened?

In the rush to seem cutting-edge, companies are deploying AI agents systems faster than you can say “hallucination.” But seriously, who actually asked for this digital workforce? Certainly not the customers who found themselves arguing with a chatbot named “Sam” that invented policies out of thin digital air.

Meet “Sam,” the Policy Inventor

Cursor, a company that sells AI productivity tools to developers (oh, the irony!), recently discovered what happens when your AI support system goes rogue. Their chatbot “Sam” confidently informed users about a nonexistent policy and an AI-powered IDE started kicking users off when they logged in from multiple machines. Limiting the number of cross machine logins, then doubled down when challenged. Nothing builds customer trust quite like having your AI make up rules and then die on that algorithmic hill.

“Any AI responses used for email support are now clearly labeled as such,” a Cursor representative explained after the incident. “We use AI-assisted responses as the first filter for email support.” How reassuring—they’ve installed a warning label after the horse has bolted from the digital barn. Outside of posting on Reddit, there needs to be another escalation path to resolve AI responses that are incorrect. Good to have a label but what’s the path to ensure the responses are correct and if incorrect, how do YOU correct it before it impacts your customer? While “tips” stands for to ensure prompt service in the hospitality industry, companies need to remember prompt service needs to be the correct service or its not customer service. Here’s a clear example of a company chasing short term AI gains and at the same time incurring long term costs: (1) poor quality (2) erosion of brand trust (3) customer fatigue and dissatisfaction, and (4) hallucinating.

The Deception Dilemma

Let’s address the elephant in sitting in your chat: “LLMs pretending to be people (you named it Sam!) and not labeled as such is clearly intended to be deceptive,” as one astute user pointed out on Hacker News. When businesses deliberately humanize AI agents without disclosure, they’re not just deploying technology—they’re engaging in digital catfishing with serious consequences.

The question then becomes painfully obvious: if businesses were genuinely implementing these agents to improve customer experience (rather than simply cut costs, assuming that’s possible), wouldn’t they have installed better guardrails? Wouldn’t there be systems preventing AI from running roughshod through customer interactions, inventing policies, and doubling down on falsehoods?

The Air Canada Precedent

This incident echoes the Air Canada debacle, where an AI chatbot named “Molly” invented a refund policy where it told a customer they could book a regular priced flight and apply for bereavement rates retroactively. When the customer booked the ticket, Air Canada refused to honor the fare, arguing that “the chatbot is a separate legal entity that is responsible for its own actions.” How convenient that AI decisions only seem binding when they benefit the company’s bottom line. Huh? So, the company builds and deploys a chatbot that creates fiction of its policies and now the “chat bot” is a separate legal entity? Folks, pay attention because if big tech gets its way, not only will it have unfettered access to your data, but it’s creating a bullet proof vest to insulate itself when the “tech” runs amok. Here’s the million-dollar question, who do you get relief? Who you sue and hold accountable, “Molly”?

In these cases, businesses want it both ways: the cost savings of automated support without the accountability when things go sideways. They humanize these systems with names like “Sam” and “Molly/Becky/Karen,” catfishing customers as if speaking with people, only to shrug off responsibility with “it’s just an algorithm” when mistakes occur.

The Self-Inflicted Wound

“There is a certain amount of irony that people try really hard to say that hallucinations are not a big problem anymore,” another Hacker News commenter noted, “and then a company that would benefit from that narrative gets directly hurt by it.”

For Cursor, a company selling AI tools to developers, this represents a particularly awkward self-inflicted wound. It’s like a dentist with visible cavities or a fitness instructor who gets winded climbing a single flight of stairs. If you can’t properly implement the very technology you’re selling, why should customers trust your product?

The Path Forward: Transparency or Bust

The solution isn’t rocket science, though it might require something more valuable than technology: integrity. AI implementations need:

1. Clear disclosure when you’re interacting with an AI
2. Human oversight
3. Accountability when AI systems make mistakes
4. Consistent policies about when AI decisions will be honored

Without these safeguards, we’re creating a digital Wild West where phantom policies multiply and customer trust evaporates faster than you said yes to that SaaS provider at a failed deployment that doesn’t deliver the outcomes outlined in the slick PowerPoint sales deck and a demo of cut parlor tricks. deck that had you at “yes.”

Who Asked for This?

So, back to our original question: who asked for agentic AI/AI agents? Certainly not customers who want accurate information. Not developers who value transparency. Perhaps it was the CFO looking to trim support costs, the CEO wanting to drop “AI implementation” into the quarterly shareholder call or the executive fear-mongered into be left behind writing the check.

Before deploying your own “Sam” or “Molly,” ask yourself: are you solving a real problem, or creating new ones? Because nothing says “cutting-edge business” quite like having your own technology fabricate policies that alienate your core users.

In the meantime, if you find yourself arguing with a support agent about a policy that sounds dubious, ask the simple question: “Are you human?” If there’s a suspiciously long pause before they answer, you might just be arguing with an algorithm that’s making it up as it goes along.

And that’s a policy we can all agree needs changing.