Allow excluding a workspace (e.g. API Reference) from llms.txt without affecting search/SEO visibility
Nivedha Mohan
Description: llms.txt currently includes every published workspace, so our API Reference (hundreds of endpoints) gets listed alongside our Knowledge Base content. The only exclusion option today, "Exclude from external search engine results," also removes the content from sitemap.xml and normal search, which we can't use since the API Reference needs to stay public and searchable.
Ask: a setting (workspace-level, or at least API Reference–level) to exclude content from llms.txt generation only, independent of search/SEO visibility.
Why: llms.txt should be a curated index pointing AI agents to product/architecture docs. Hundreds of individual endpoint entries dilute that; endpoint-level detail belongs in the API spec, not the index.
Log In
K
Kavya
Hi Welcome Koala ,Nivedha Mohan,
Thank you for sharing this request.
We’ll check the feasibility of providing a way to exclude specific workspaces or API Reference content from llms.txt generation without affecting their search engine visibility or sitemap inclusion.
Could you also share a little more about your use case? For example:
-What type of content would you like to prioritize in llms.txt?
-Is the concern primarily that the large number of API endpoint pages makes the file too large or dilutes the visibility of your Knowledge Base content?
-Would excluding the entire API Reference workspace meet your requirement, or do you need more granular control over specific categories or endpoints?
-Could you share an example of your current llms.txt structure and how you would ideally expect it to look after the exclusion?
This will help us better understand the expected behavior and evaluate the most appropriate solution
Nivedha Mohan
++Welcome Koala