You are more honest with a chatbot than with almost anyone
You type things into these tools you would not say out loud. The symptom you are afraid to look up properly. The problem at work you cannot raise at work. The message you are not sure you should send, written out in full so something can tell you whether to send it.
People are candid with a chatbot because it feels like talking to no one. It is not no one. It is a company, and the conversation you are having is the product. Which means the conversation is exactly what it keeps.
What an AI tool actually collects
Strip away the chat bubble and the tool is holding:
- every prompt, and the whole thread around it, not just the question but the back-and-forth;
- the files you feed it: documents, photos (including your face), your voice;
- the outputs, and which ones you kept, edited, or regenerated;
- the usual account, device, and usage trail underneath.
The more useful you let it be, the more of you it has. That is not a side effect of the design. It is the design.
Your side of the conversation is the training data
On most of these tools, the phrase "to improve our models" means your prompts and uploads are used to train the system, and on a great many of them that setting is on by default, with the off switch buried somewhere in account settings you have never opened.
Once your words have trained a model, you cannot take them back out again. That part is its own story: "anonymised" is not anonymous to a model. The narrower point for now is simpler. The default is usually "yes, learn from me," and most people were never shown the switch.
And people read them too
"AI" makes it sound as if no human is ever in the loop. Humans are in the loop. Conversations get sampled and reviewed by staff and contractors, to rate the answers, to catch abuse, to make the next version better. The intimate thing you typed at two in the morning can be read by a stranger doing quality control. That is not a breach. It is a normal part of how the product is run.
"Delete" is the weakest word here, and a court proved it
You clear a conversation and assume it is gone. Here is what actually happened to ChatGPT, one of the largest chatbots in the world. In a lawsuit, a court ordered OpenAI to preserve all of its output logs going forward, explicitly including the conversations users had already deleted, and later ordered roughly twenty million of those conversations handed over to the other side. The company described the handed-over logs as "de-identified." How much that protects anyone is, again, its own question.
That order was later narrowed. But for months, "deleted" chats were retained by court mandate, entirely outside the control of the people who wrote them. The delete button cleared your screen. It did not clear the record.
The company did not have to set out to betray anyone. The logs were kept because they are useful: to the product, and as it turned out, to a court. The incentive and the outcome are the same whatever the intent.
If it leaks, it is the most revealing file you own
A leaked password is nothing next to a leaked chat history. Your prompts are your unedited inner life: what you are unwell about, frightened of, arguing over, hiding at work.
And the companion-app corner of this industry is sharper still. Apps built for romance and confession, the ones that invite you to be most exposed, have been caught loading tens of thousands of trackers in the first minute of use and piping data out to advertising networks. One was fined millions in Europe for never clearly telling users how long it kept their messages, or whether children were pouring their hearts into it. The more honest an app persuades you to be, the more valuable the file it is building.
What to actually do
You cannot un-say what you have already typed. You can change how you treat the box, and you can keep a record.
- Find the training switch and turn it off. It is usually there, usually buried, and worth doing for everything going forward.
- Treat the box as on the record, because it is. Do not type what you could not stand to have read back to you. The most convenient place to think out loud is also a place that keeps what you said.
- Use the temporary or no-history mode for the genuinely sensitive, and know its edges: abuse-monitoring and legal holds can still reach in.
- Keep your own record: which tools you used, what you fed them, the day you opted out or asked for deletion, and what they wrote back. If "deleted" turns out to mean "retained," that record is your evidence.
It is the most intimate version of the same machine that runs under every privacy policy, and it fails the same way deletion always does: loudly promised, quietly kept. So keep your half of the record. Start your record →