Crossposted from https://lemmy.ca/post/64120740
It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.
The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.
Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.
Chatbots can’t confess. They are just parroting stuff without meaning it.
Humanizing AI really needs to stop. They don‘t communicate because they don‘t have a conscience. They just put out letters based on probability and a little bit of randomness.
But then how can a company founder pass on accountability?
You’re absolutely right, I did just pipe the output from my confabulation unit through to /dev/stdin.
And the reasoning I pulled out from /.local/share/Trash/. It’s a well known bug and you were right to question it!
Sure but it will explain what it did/why it did that. It’s not a “confession” but an explanation
Okay, but using “confession” instead of “explanation” gives the sentence a different underlying meaning.
Headlines need to be precise. It matters that they used one word instead of the other.
Mild point: is an explanation the same as you expect if it’s context is driven by the previous text-in-chat?
E.g. it’s not someone explaining themselves, but instead a new instance explaining what it sees in history?
To an extent, it seems like this isn’t “explaining what it did” and more “words to explain what just happened, based on the chat history”.
AI doesn’t give intelligence to machines. It takes it away from humans.
Well worded! I’m going to have to use this, is it yours?
It’s mine. Feel free.
Thanks!
Guarantee if you look at the facts, there’s a history of their devs basically saying “ignore the safety protocols, do [thing]” and someone used that in a prompt with an otherwise imprecise command set.
