Personal Screenshot Knowledge Base: Practical Pitfalls of OCR and Vector Retrieval
AI Summary · Perspective of a Serial Entrepreneur (The following content is distilled by AI; viewpoints belong to the original author; you can skip the original article after reading this)
For building a local screenshot-based content search engine at scale, the core pain point lies not in the vector database but in data cleaning and query understanding. By filtering out useless indices with rule-based approaches (regex/length), and adding query rewriting from colloquial language to technical terminology, we achieved second-level search across 300 days of data.
- Before indexing, split by text blocks and use regex to filter timestamps and watermarks…
- Adopt a layered architecture with SQLite FTS5 for keyword-based rough filtering followed by vector-based reranking…
- Error-related screenshots require combining general OCR with vision models…
- The query layer needs semantic rewriting (e.g., expanding “error message” to error/failed) to improve fuzzy search recall
1. Core Pain Point: OCR Is Not Just “Plug-and-Call”
Many assume that with large models or OCR libraries, recognition works out of the box. But when handling English error messages and file paths in Chinese screenshots, purely Chinese models often fail completely. The solution is multi-model complementarity: use general OCR as the foundation, then re-read low-confidence text blocks with vision models, while preserving coordinate information for later highlighting and positioning.
2. Breaking Through Index Bloat
Fully vectorizing everything causes the index to become larger than the original images. Two key moves here:
- Optimize split granularity: Don’t split by whole image—split by text blocks. First filter out invalid UI text like timestamps and watermarks using regex, length, and positional rules, which directly cuts 60% of the index volume.
- Layered retrieval architecture: Pass through SQLite FTS5 for keyword rough filtering first, then rerank with vectors for candidates, avoiding the high latency of full-database ANN search.
3. Query Understanding Determines Retrieval Ceiling
When a user searches “that 502,” the system doesn’t know what they’re looking for. You must rewrite at the query layer: transform colloquial language into forms likely to appear in screenshots (e.g., expand “error” to error/failed/error message), and separately parse time-limiting terms into filters to avoid polluting vector semantics.
With this combination, screenshots spanning 300+ days achieve sub-second response with over 80% accuracy. The biggest engineering challenge isn’t the search engine itself, but the upfront data cleaning and query semantic understanding.
Original · V2EX Startup: Read original →