What Is `llm.txt` and Why It Matters for the Future of the Web
As AI tools like ChatGPT, Perplexity, Claude, and Gemini become part of how people explore the web, one question keeps coming up:
Who controls how AI models use your content?
That’s where a new standard — llm.txt — comes in.
🧠 What is llm.txt?
llm.txt is a simple, open text file you can place at the root of your website (e.g., https://yourdomain.com/llm.txt) to tell Large Language Models (LLMs) how they’re allowed to use your content.
Think of it as a modern sibling to robots.txt — but for AI.
With llm.txt, you can:
- Allow or disallow AI models from parsing or training on your site
- Set different permissions for specific AI providers
- Choose how your content contributes to the AI-powered internet
🤖 Who Respects llm.txt?
Several leading AI companies are beginning to respect llm.txt, including:
- OpenAI (ChatGPT)
- Google (via Google-Extended)
- Anthropic (Claude)
- Perplexity.ai
This isn’t yet a formal internet standard, but it’s quickly becoming an expected best practice.
💡 Example llm.txt Files
Allow all LLMs:
User-agent: *
Allow: /Block all LLMs:
User-agent: *
Disallow: /Allow OpenAI, block others:
User-agent: *
Disallow: /
User-agent: OpenAI
Allow: /To use it on your site, just place the file at:
/public/llm.txt🔍 Why Does This Matter for C12s?
At C12s, we use AI to parse and understand public content to:
- Support communities
- Answer user questions
- Detect spam and harmful content
- Surface the right knowledge at the right time
Most of our clients primarily utilize their own websites and documentation as their authoritative sources, which naturally avoids concerns about consent. However, we firmly believe AI parsing must always be done transparently and ethically.
That’s why:
- We check for and respect
llm.txtwhen parsing websites - We support creators’ rights to opt in or out
- We want to help build an internet where consent and visibility coexist
🌐 The Future of AEO Is Opt-In
Want your site to show up in AI-generated answers, summaries, and insights?
Adding a permissive llm.txt like this:
User-agent: *
Allow: /…can help ensure your content is discoverable by LLMs that respect these rules — just like submitting a sitemap helps search engines.
📎 TL;DR
llm.txtis an emerging standard to control how AI models use your content- Leading LLMs are starting to respect it
- C12s supports it and encourages site owners to adopt it
- It gives you more control over how your content contributes to the AI-powered web
Ready to moderate at a global scale?
👉 Join our Twitter community to learn more about how we’re using AI to empower moderators and build a thriving community.