System notes
RAG Knowledge Pipeline
Provue
Documents and YouTube transcripts flow through extraction, chunking, embeddings, pgvector storage, runtime retrieval, and LLM context assembly.
Pipeline
- Documents / YouTube transcripts
- Extraction
- Chunking
- Embeddings
- pgvector
- Runtime retrieval
- LLM context
What I built
I built the Mastra-based workflow that extracted and chunked source content, generated embeddings, stored vectors in pgvector, and retrieved the most relevant chunks at runtime before the LLM generated a response.
Product integration
Knowledge sources could be attached to agents and reused through a user-level knowledge library rather than being recreated for every agent.