Add llms-full.txt for LLM accessibility of documentation

[Wenzel]

Summary

The TSFFS documentation at https://intel.github.io/tsffs/ is not discoverable or easily consumable by LLMs (e.g. for RAG pipelines, context-window injection, or AI coding assistants).

Specifically:

• No /llms.txt or /llms-full.txt (per the emerging llmstxt.org standard)
• No /sitemap.xml or /robots.txt
• Content is split across ~77 pages; an LLM agent has no way to enumerate them without recursively parsing HTML nav links
• print.html (the full single-page build) exists but is not advertised anywhere

Proposed fix

Add a llms-full.txt generation step to the docs CI pipeline. The Markdown source in docs/src/ is already well-structured and SUMMARY.md provides the correct chapter order.

One shell command added after mdbook build in .github/workflows/docs.yml:

grep -oP '\([^)]+\.md\)' src/SUMMARY.md | tr -d '()' | while read -r f; do
  echo; cat "src/$f"
done > book/html/llms-full.txt

This:

• Requires no new tools, dependencies, or files
• Produces a ~213 KB, ~6000-line concatenated Markdown file at /llms-full.txt
• Automatically stays up to date as docs change
• Is fully ingestible in a single LLM context window

Verification

Tested locally — the output flows correctly from intro → setup → config → harnessing → fuzzing → tutorials → developer docs, with all code examples intact.

References

• llmstxt.org specification

[Wenzel]

Empirical comparison: website navigation vs. llms-full.txt

After opening this issue, I ran an experiment to quantify the actual difference. Two parallel LLM agents were given the same task — produce a comprehensive summary of TSFFS capabilities — with different information sources:

• Agent A browsed https://intel.github.io/tsffs/ by following links (the current state)
• Agent B read only llms-full.txt generated locally using the proposed CI command

Token and efficiency comparison

Metric	Agent A (website)	Agent B (llms-full.txt)
Tool calls	60 (WebFetch)	3 (file reads)
Tokens consumed	~44k	~102k
Time to complete	~480 s	~165 s
Navigation failures (404)	18 pages	0
Fallback required	Yes (print.html)	No

Agent B consumed more tokens (ingesting the full 213 KB at once) but was 3× faster and needed zero fallbacks.

Capabilities Agent B found that Agent A missed

Capability	Agent A	Agent B
MSVC requires separate .asm file (tsffs-msvc-x86_64.asm + ml64)	❌	✅
Custom magic numbers (MAGIC_ALT_0=4, MAGIC_ALT_1=5) for co-existing harnesses	⚠️ vague	✅ precise
Windows disk diff optimization (hd_image.save-diff-file)	❌	✅
Graphical boot-detection breakpoints (save-break-xy, bp-wait-for-gfx)	❌	✅
SIMICS Debugger API for symbol resolution (SIM_get_debugger())	❌	✅
VMP (Virtual Machine Acceleration) requirement	❌	✅
Documentation inconsistency flagged (magic_start vs magic_start_index)	❌	✅

Agent B also found nothing that Agent A had that it didn't — the website contains no information absent from llms-full.txt.

Why the missed items matter

The MSVC .asm detail is the most critical: an LLM generating a Windows harness without knowing about tsffs-msvc-x86_64.asm would produce code that compiles but fails to link. The MAGIC_ALT_* numbering is needed for the kernel module dual-harness pattern. These are the exact gaps that cause subtly wrong AI-generated code.

Conclusion

llms-full.txt closes real correctness gaps (not just convenience gaps) for LLM consumers. Agent A's 60 web fetches, 18 404s, and fallback to print.html still produced an incomplete picture. Agent B's 3 reads produced a complete one in a third of the time.

The proposed CI command is already implemented on the improve-llm-tsffs-docs branch — this comment is the validation data for the approach.