OpenAI Project Lily: humans read ChatGPT chats – the filter can miss

404 Media reported on 14 September that OpenAI pays contractors to read real ChatGPT conversations. The internal name is Project Lily. This is not the safety queue for harm flags. It is quality training: make the model less robotic, less sycophantic, more like a tool.

Joseph Cox based the story on leaked internal material and actual user prompts. The Decoder and The Next Web have gone through the findings. Usernames are not shown. The full conversation can still sit in front of a human. OpenAI admits its Privacy Filter can make mistakes.

For a CIO the split is simple. ChatGPT Enterprise, Business and Edu are not used for training by default. What feeds Lily is consumer ChatGPT. There, “Improve the model for everyone” is on unless the user turns it off.

What the contractors actually do

The Decoder says hundreds of contractors are involved. They are recruited through Crossing Hurdles and paid through Mercor. A North America-based reviewer told 404 Media the rate is over $50 an hour.

Tom’s Guide, using the same documents, describes the workflow: read the user prompt, summarise intent, score four candidate ChatGPT replies from 1 to 7. Reviewers are told to penalise “AI-speak”: checkmark emojis, engagement-bait endings, cluttered formatting. They must also stop anthropomorphising. Lines such as “as a chef, I like to …” are banned.

The Decoder says the brief is to cut excessive flattery and human-like behaviour. One reviewer said he did not think users knew humans were reading the chats. Some prompts asked ChatGPT to keep the contents private. That did not stop the review.

This is separate from the publicly described safety checks where chats are flagged for harm. Lily is quality work on ordinary conversations.

What follows the prompt into the dashboard

Reviewers do not see usernames. The Next Web writes that the dashboard can still show a “user memories summary” above the prompt. It can describe what the person has used ChatGPT for before, and roughly where in the world they may live.

Conversations are supposed to pass an automated Privacy Filter first. OpenAI released that filter as open weights in April 2026 under Apache 2.0. The company says the model can miss uncommon identifiers, ambiguous personal context, and under-redact when context is thin. 404 Media reports that sensitive information can still reach a human.

The Next Web asked OpenAI where it tells users that humans may read their chats. The company did not answer before publication. Afterwards it pointed to a help page.

OpenAI is explicit on opt-out. In ChatGPT, Settings, Data Controls, turn off “Improve the model for everyone”. New conversations then stay out of training. Old history is already in. Temporary Chat is not used for training, according to the company. Thumbs-up or thumbs-down feedback can still send the whole conversation back in, even after opt-out.

What this means for boards

First question: shadow IT. Staff paste customer drafts, payroll notes, health details and board papers into personal ChatGPT because it is faster than waiting for a tenant. Then the Enterprise promise does not apply. The consumer default does.

Second question: GDPR. The Next Web notes that the EU Court of Justice held last autumn that the duty to inform sits with the controller at collection, and does not depend on whether the recipient can identify anyone. A stripped username is not the same as a conversation without personal data. A memory summary with location and prior topics can be enough.

Third question: the vendor line. If the organisation already has ChatGPT Enterprise, training is off by default. That does not stop someone logging into Plus on a phone. Ask IT how many people use consumer ChatGPT against internal documents. Put DLP on paste to chat.gpt.com. Make Enterprise or Azure/Bedrock the only approved surface.

Fourth question: DPIA. If ChatGPT touches personal data, the record of processing should say who can read the content. “The model is trained” is one sentence. “Hundreds of contractors at a subprocessor may read the full conversation, plus a memory summary” is another. Ask OpenAI for the subprocessor list, review geography, and what happens when the filter misses.

Fifth question: board language. This is not an argument to ban generative AI. It is an argument to stop treating personal ChatGPT as the same product as Enterprise. Microsoft’s MAI code, Anthropic’s evaluator pledge and OpenAI’s human review in Lily are three different control regimes. Mix them and the DPIA is false.

Turn training off on personal accounts today. Move work onto a tenant where training is off by default. Ask OpenAI in writing where human review is disclosed to end users, and whether memory summaries are in scope. That is the board question. The rest is product copy.

Sources and media

Primary source: 404 Media, Joseph Cox, “Inside ‘Project Lily’: The Humans Reading Your ChatGPT Chats”, 14 September 2026: https://www.404media.co/inside-project-lily-the-humans-reading-your-chatgpt-chats/

The Decoder, “OpenAI has hundreds of contract workers reading your ChatGPT conversations”, 14 September 2026: https://the-decoder.com/openai-has-hundreds-of-contract-workers-reading-your-chatgpt-conversations/

The Next Web, “Hundreds of contractors are reportedly reading real ChatGPT conversations”, 14 September 2026: https://thenextweb.com/news/chatgpt-human-reviewers-gdpr

OpenAI Help Center, “How your data is used to improve model performance”: https://help.openai.com/en/articles/5722486-how-your-data-is-used-to-improve-model-performance

OpenAI, “Introducing OpenAI Privacy Filter”, 22 April 2026: https://openai.com/index/introducing-openai-privacy-filter/

Thumbnail: OpenAI Image 2 / hogby.ai