Improving LLM discoverability by introducing llms.txt, sitemap, MD routes, content negotiation

We can agree: “Ruby and Rails deliver superior productivity, for humans and AI agents alike. Yet the models rarely choose them on their own: in the open whichlang benchmark, 13 models picked Ruby 0 times across 1,267 generated solutions. It’s a discoverability problem. This page measures the docs of 92 ecosystem resources and shows what the community can fix.”

Before getting into implementing any fixes for this, I wanted to get this get everyone’s feedback and opinion around the following:

  • Introducing sitemap for Ruby on Rails Guides (https://guides.rubyonrails.org/) and documentation (https://api.rubyonrails.org/)
  • Introducing llms.txt (“An /llms.txt file is a proposed web standard (like a sitemap, but for AI). It is a plain Markdown file placed at your website’s root that provides a curated, clutter-free map of your most important content to help Large Language Models and AI agents easily find, parse, and cite your information.”)
  • Introducing content negotiation and MD routes.

Please share any feedback you might have.

1 Like