Generating llms.txt and llms-full.txt in Hugo with Output Formats

I added llms.txt , a specification that lets LLMs retrieve the information on a website efficiently, to this Hugo blog.

Environment

• Hugo v0.155.2
  • Theme: Beautifulhugo
• Cloudflare Pages

What llms.txt is

A Markdown-format text file placed at the root of a website, proposed at llmstxt.org . Like robots.txt or sitemap.xml, it provides an overview of the site and its main links in a form that is easy for an LLM to read.

There are two main kinds of file:

• llms.txt — an overview of the site and a list of article links (it plays the role of a table of contents)
• llms-full.txt — the full version, containing the body of every article

What a technical blog gains from it

To be honest, the benefit for a personal technical blog is limited. There is not as much demand for it as there is for API documentation or a library reference.

Still, I judged that it “does no harm to do” for the following reasons:

• LLM-powered search engines (Perplexity, ChatGPT Search and so on) may find it easier to grasp the context of the site
• The cost is almost zero — if it is generated automatically from a Hugo template, no manual maintenance is needed as articles pile up
• An advance investment — if the specification takes hold, it may turn out to be an advantage in the future

Setup steps

I went with generating them automatically at build time using Hugo’s outputFormats. Putting static files under static/ does not reflect articles being added or removed, so the template approach is the best one.

1. Add outputFormats to hugo.toml

[outputs]
  home = ["html", "rss", "llms", "llmsfull"]

[outputFormats.llms]
  baseName = "llms"
  isPlainText = true
  mediaType = "text/plain"
  rel = "alternate"
  root = true

[outputFormats.llmsfull]
  baseName = "llms-full"
  isPlainText = true
  mediaType = "text/plain"
  rel = "alternate"
  root = true

The default value of outputs.home is ["html", "rss"], so I specify it explicitly while adding llms and llmsfull.

2. Create layouts/index.llms.txt

A template that automatically generates the list of article links in descending order by date.

{{- $baseURL := .Site.BaseURL | strings.TrimSuffix "/" -}}
# {{ .Site.Title }}

> セルフホスティング、DevOps、Linux管理、開発ツールに関する日本語の技術ブログ。Hugo + Cloudflare Pagesで運用。

- 記事の主な言語は日本語です
- 記事はAIの支援を受けて作成されている場合があります

## Docs

- [About]({{ $baseURL }}/about/): ブログの方針・サイト概要

## Articles
{{ range where (sort .Site.RegularPages "Date" "desc") "Section" "post" }}
- [{{ .Title }}]({{ .Permalink }}): {{ .Date.Format "2006-01-02" }}
{{- end }}

3. Create layouts/index.llmsfull.txt

The full-version template, containing the Markdown source of every article.

{{- $baseURL := .Site.BaseURL | strings.TrimSuffix "/" -}}
# {{ .Site.Title }}

> セルフホスティング、DevOps、Linux管理、開発ツールに関する日本語の技術ブログ。Hugo + Cloudflare Pagesで運用。

{{ range where (sort .Site.RegularPages "Date" "desc") "Section" "post" }}
--------------------------------------------------------------------------------
title: "{{ .Title }}"
date: "{{ .Date.Format "2006-01-02" }}"
url: {{ .Permalink }}
--------------------------------------------------------------------------------
{{ .RawContent }}
{{ end -}}

The reason I use a long rule rather than --- as the separator is that there are cases where an article itself contains --- (a YAML separator or a horizontal rule).

4. Check the build

hugo --gc --minify

They are generated under public/:

• llms.txt — about 61KB (the list of article links)
• llms-full.txt — about 781KB (with the body of every article included)

Serving them on Cloudflare Pages

No special configuration is required. Cloudflare Pages automatically applies text/plain; charset=utf-8 to .txt files, so Japanese text does not come out garbled.

With nginx you would need a separate charset UTF-8 setting, but on Cloudflare Pages there is no such worry.

The generated result

• llms.txt
• llms-full.txt

On security and the risk of plagiarism

Since llms-full.txt contains the body of every article, you may be concerned about your content being plagiarised or misused.

My conclusion is that the risk is essentially unchanged.

• Every article is already published as HTML, and every URL is listed in sitemap.xml
• Anyone who wants to scrape can do the same thing with sitemap.xml + curl even without llms.txt
• The possibility of being used as LLM training data exists regardless of whether llms.txt is present (it has already been collected by CommonCrawl and the like)

The original purpose of llms.txt is at inference time (when a user hands site information to an LLM), which is a different context from collecting training data.

Measures if you are concerned

For a personal technical blog the actual harm is small, so I judged that publishing it as is causes no problem. If you are concerned, dropping only llms-full.txt and keeping just llms.txt (the link list) strikes the best balance.

References

• llmstxt.org — The /llms.txt file
• Hugo で作ったサイトに llms.txt と llms-full.txt を設置した | マスタカの ChangeLog メモ (in Japanese)