AI Knowledge Base Cleanup: 800-Line File Caused Token Costs to Spike and Halved Efficiency

AI Summary · A Serial Entrepreneur's Perspective (The following content is distilled by AI; the views belong to the original author. Reading it here is enough—you don't need to click through.)

After connecting Claude Code to a project knowledge base, the author found that a single 800-line Markdown file caused a simple requirement to jump from 30 minutes to two hours of development time and sent token costs soaring. By applying three rules—splitting files over 200 lines, building a README index, and keeping module granularity clear—plus vector search, they slimmed down the context, sped up responses, and actually improved accuracy. For startup teams using AI to assist with development or ops, this offers direct cost savings.

• Always split files over 200 lines to fix granularity issues
• Use a README as a navigation index instead of forcing the model to memorize everything
• Split modules by responsibility—one file, one job
• Add a vector database and scripts so the model retrieves only what it needs instead of ingesting everything
• After slimming down, you save on tokens, gain speed, and see accuracy climb—direct cost savings

A Single 800-Line Knowledge Base File Drove Development Time from 30 Minutes to 2 Hours

Many people fall into the trap of thinking bigger is better when it comes to project knowledge bases and skill libraries, so they dump the team Wiki, API docs, and historical decision records straight into the AI. But a large language model's context window is fixed. Feed it too much, and it chokes.

Key takeaway: AI knowledge bases must be slimmed down. By capping file line counts, building indexes, and introducing vector search, you can cut token costs significantly, speed up responses, and even improve answer accuracy.

Real-world data: After hooking up a knowledge base curated by the company's AI team to Claude Code, the author found that a single 800-plus-line Markdown file caused token fees to spike. Iterating on a small requirement jumped from the usual 30 minutes to nearly two hours. Once they cleaned and refactored it, the context became cleaner and accuracy actually went up.

So where did it go wrong? When you shove a massive file into the model, it has to "read" it all before it can understand the requirement. That creates three side effects: token usage skyrockets, the model's attention gets diluted, and response times slow down.

Three Ironclad Rules for Refactoring the Knowledge Base

After hitting these snags, the author summarized three non-negotiable rules:

• Keep each file under 200 lines: Anything over 200 lines means the granularity is off, so split it up—don't hesitate.
• Use a README as a navigation index: The README is your map. It tells the model where to find what. The model needs navigation, not the ability to recite entire documents.
• Clear module boundaries: One module should do one thing. If a file covers APIs, deployment, and historical background, break it into three separate files.

The Ultimate Fix: Vectorization + On-Demand Retrieval

What if the knowledge base is still too big even after splitting the files? The answer is vectorization.

The approach is straightforward: run the entire knowledge base through a vectorization model, store the resulting vectors in a local vector database, and write scripts and skills that let the model actively retrieve information on demand instead of passively swallowing everything.

This "on-demand retrieval instead of bulk injection" approach ensures the model pulls up precisely matched, relevant info rather than sifting through noise for a blurry answer.

Repeatable Execution Steps

If your team is already using AI to assist with development or operations, start doing these things right away:

1. Audit your current knowledge base: Check whether any single Markdown file exceeds 200 lines.
2. Split and index: Break large files apart by responsibility and write a README to serve as a global index.
3. Introduce vector search: Set up a local vector database, build retrieval scripts, and enable dynamic context loading.

After the slim-down, what you save isn't just token costs—it's your team's most precious resource: development time.

Original post · Heyuanfei's Idle Notes: Read original article →

Related tool recommendation (sponsored): GLM Coding Plan — AI Coding Powered by G…