September 25, 2026 · 26 min read
LifeLedger, Part I: Ask Your Past
A birthday in Karachi. A wedding guest list. A diary entry from thirty years ago. LifeLedger begins with a simple wish: ask a question of the life you have already documented.
Go · RAG · Ollama · PostgreSQL · pgvector · Local AI
This is the first of two articles about building LifeLedger, a private place to ask questions of a lifetime of memories, documents, and knowledge. We begin with the reason for building it, then work through the foundation. In Part II, we will start bringing the records into the conversation.
I have the records. Finding the answer is the work.
About thirty years ago, we celebrated my son’s birthday at McDonald’s in Karachi. I might remember the occasion and still want the details: who came, what we ordered, how the day went. I keep a daily diary. The account is there, along with the other records I have kept. But asking the question is easier than finding the right pages and bringing everything together.
My daughter’s wedding is another example. Who attended? What was the venue? Where did each guest sit, and what gifts did they give her? The guest list, venue agreement, seating plan, and gift notes live in different places. To answer today, I have to find each artifact, read it, and piece the story together myself.
An old bank statement presents the same problem. The closing papers for a house are in another folder. Notes about a remodeling project live beside contractor estimates, while the reason I chose one contractor may survive only in Obsidian. Years later, a simple question sends me through all of them: what did that project actually cost?
The same problem follows the happier parts of life. I have records of my children’s birthday parties: guests, gifts received, gifts given. There are marriage celebrations, venues, and expenses. There are educational records, car purchases, home inventories, and the documents that accumulate whenever a family buys, builds, learns, or celebrates.
I kept these things because they matter. I want to be able to use them.
I am calling the application LifeLedger. A ledger keeps an account; this one makes room for the whole life behind the entries. A birthday belongs beside a balance sheet, a diary beside a deed, a family memory beside the record that helps bring it back.
Its promise is simple: your life, with sources.
LifeLedger will bring diaries, legacy documents, financial records, and scattered knowledge into one private, searchable archive. I want to ask in ordinary language, at any time of day, and follow the answer back to the page it came from.
An illustration of the memories and records LifeLedger is being built to bring together.
A library for the life I have already documented
Start with that birthday in Karachi. I want to type:
Tell me about my son’s birthday at McDonald’s in Karachi, about thirty years ago. Who were the guests, what did we order, and how did the day go? Use my diary and the related records, and show me the sources.
I should not have to remember the diary volume, the filename, or the exact date before I can begin. The question gives the application clues: a person, an occasion, a place, and an approximate period. LifeLedger should use those clues to find the likely event. If the archive contains two possible birthdays, it should ask me which one I mean.
Once the event is identified, the application can gather the relevant diary entries and any linked records. The diary may contain the guest names and meal details itself; another artifact may add or correct them. “How did the day go?” calls for the account I wrote at the time. The answer should preserve that perspective—“Your diary describes…”—and let me open the entry. It should not invent the weather, a conversation, or a child’s reaction to make the story more vivid.
This is the value of having kept a daily diary. A document can establish that an event happened; a diary can preserve what I noticed and how I experienced it. LifeLedger should make both accessible, with dates and sources intact. Ingestion is the beginning: readable extraction, useful indexing, and reliable retrieval must still bring the right evidence to the answer.
The wedding asks the same archive to connect a different set of records:
For my daughter’s wedding, tell me the venue, list the guests, show their assigned tables or seats, and include the gifts they gave her. Show me which records support the answer.
That question crosses several documents. The venue comes from a booking or final event record. The guest names come from the guest records. Table assignments come from the seating plan. Gifts come from a separate list, perhaps recorded by household rather than by individual. My application will need to bring those sources together, distinguish people with similar names, and preserve what each record actually says.
The result should spare me the search through folders and notes. I could read one organized answer, then open a source whenever I wanted to check a detail. A missing gift entry would mean “no gift recorded in the available list,” not “this guest gave nothing.” An invitation would not establish attendance, and an assigned seat would not prove someone sat there. These distinctions are part of remembering an event accurately.
This is why the archive is worth building. The information already exists, but I currently supply the search, the cross-references, and the memory of where everything belongs. A useful application would let me ask the whole question in ordinary language and would do that evidence-gathering work for me.
Connecting a terminal to an LLM does not provide this ability. ChatGPT or a local model, without access to my records, cannot know who attended that birthday, what we ordered, or which gift came from which family at the wedding. It may acknowledge that it lacks the information. If it supplies those details anyway, a polished answer would still be an invention. A hosted assistant given the relevant files or a connected source could also use them; what matters is access to evidence. My goal is to build that access under my own control.
There is a name for this approach: retrieval-augmented generation, or RAG. Retrieve the relevant records, put their contents in the model’s context, and ask it to answer with sources. It can reduce unsupported answers, but it cannot guarantee accuracy. For this wedding request, retrieving only a handful of similar passages may omit guests. The finished application must recognize a request for a complete list, load the relevant event records in full or query their structured representation, and disclose any gaps. That work is still ahead.
These occasions are examples of a broader purpose: being able to consult the life I have already documented. Other questions follow the same pattern.
“Who came to that birthday party, and what gifts did we receive?” The evidence might be a guest spreadsheet, a handwritten list that needs scanning, and a note I made afterward. A useful answer would name the event and cite those records. If the guest list records invitations rather than attendance, it should say so.
“How much did we spend on the wedding, including the venue?” That calls for the booking agreement, the final invoice, and the payments. A deposit listed on an agreement and again on a receipt must not become two expenses.
“Where is the closing document for our previous home?” “Which car did we own then?” “What was in the house before the move?” “Which educational record confirms that date?” These are ordinary questions whose answers become harder to find as folders multiply and filenames lose their meaning.
Obsidian holds another part of the story: decisions, explanations, links, and things I learned. A receipt can tell me what I paid. A note can tell me why. I want both available, with their origins intact.
The original documents remain the authority. The model helps me read and connect what the application finds; it does not acquire my family history merely because I installed it.
An embedding makes this search possible across differences in wording. It represents text as a list of numbers that can be compared with other such lists. A question about “house renovation” may lead to a passage headed “kitchen remodel,” even when the words do not match. Exact identifiers, names, and dates will still need exact search or filters. Similarity is useful, but it is only one way to find a record.
The harder question: what did I own and owe on a given date?
My longer-term aim is to reconstruct my personal net asset value—my net worth—as of a date I choose. That means accounting for every relevant bank and investment account, homes and their documented values, cars, other assets, and outstanding liabilities. A dated home inventory belongs alongside that picture, with a clear valuation basis for anything counted.
The words “as of” do much of the work. Today’s home estimate cannot silently stand in for a value five years ago. A statement downloaded in January may report a December balance. A mortgage is a liability against a home; a transfer between my own accounts is not new wealth. Purchase price, replacement cost, and estimated resale value answer different questions.
This goal requires more than retrieving a few similar passages. A nearest-neighbor search returns likely evidence, not a complete inventory of every account. Increasing topK does not establish completeness.
The eventual design needs a reviewed, structured record of assets, liabilities, balances, currencies, and effective dates, each linked to its source. Go or SQL can then perform deterministic arithmetic, using decimal values or integer minor units for money. Retrieval can help locate the statements and explain the result. The language model should never be responsible for inventing a missing balance or deciding that an omitted account is worth zero.
For an illustrative request such as “Show my net worth as of December 31, 2024,” I would want a total accompanied by the records used, the valuation dates, and a list of gaps. If a property value is only an estimate or an account has no statement for that period, the answer must make that visible. Where the records do not support a complete figure, the result should remain incomplete.
That is a design goal for the larger project. Neither historical accounting nor document ingestion exists in the code we are walking through today. Part II will begin the evidence pipeline; it will not turn the first retrieval demo into a finished financial system.
Privacy sets the pace
These documents describe a family’s life. They deserve a design that keeps control close to home.
For my use, waiting a little longer for an answer is acceptable. Sending private records to an outside inference service is a much larger decision. I want document processing, embeddings, and generation to run locally, with the vector database in a private container or, when needed, within my own private cloud account.
An Apple silicon Mac with enough unified memory can be the starting point. The generation model used here, qwen3.8:27b, is a substantial model: the current Ollama artifact is about 18 GB, and running it needs additional memory for context and the rest of the system. “Apple silicon” alone is not a capacity specification. On a smaller machine, use a smaller local model and measure its answers on your documents. The application boundary stays the same. Ollama’s model listing gives the artifact details.
Once the models and software are downloaded, local inference can operate without an internet connection. Ollama supports disabling its cloud features with OLLAMA_NO_CLOUD=1; apply this to the Ollama server process and restart it. I also need to keep the Go client pointed at the local endpoint and select downloaded models. Ollama’s FAQ explains local operation and cloud controls.
A container supplies a convenient place to run PostgreSQL. Privacy still depends on how I expose it and protect its files. For the local archive, that means loopback-only ports, proper credentials, encrypted disks and backups, and restricted access. In a private cloud account, it means a private network, no public database endpoint, encrypted connections, and controlled access. Extracted text and embeddings belong inside that boundary too.
The same care applies to the less visible parts of the pipeline. A local chat model offers little protection if an OCR service uploads every scanned statement. I will need to check extraction tools, outgoing connections, logs, and backups before loading real family records. These are deployment requirements for the intended archive; the development setup below is an early foundation.
How I am building LifeLedger
Once the purpose is clear, the engineering follows from it. The application needs to read my records, find relevant passages, keep their sources attached, and carry that evidence into a conversation. These are tasks I can express in ordinary application code.
If your introduction to RAG came through Python or a library such as LangChain, the next few pages may look different. The underlying sequence travels well: find evidence, supply it to a model, check the answer. It does not require a particular programming language or an orchestration framework. I am building LifeLedger in Go.
I want to understand each step well enough to inspect it, test it, and change it. Go will do the application’s heavy lifting: reading documents, controlling work, retaining source information, calling the models, writing to PostgreSQL, and deciding what evidence enters an answer. Ollama and the models will perform inference.
By “pure Go,” I mean that the application and its orchestration are written in Go, without a RAG framework. PostgreSQL and Ollama are separate services. I use focused libraries—openai-go for the compatible HTTP client, pgx for PostgreSQL, and pgvector-go for vector values—while writing the retrieval workflow myself.
There is a practical benefit to starting this low in the stack. When an answer is wrong, I want to trace the route from the source page to the final prompt. Each boundary should be visible in the code.
For now, two parts of that route are taking shape:
- Working now: terminal → Go → Ollama → Qwen → streamed answer.
- Planned for Part II: documents → passages → embeddings → pgvector → retrieved evidence → Qwen → answer with sources.
The storage methods and similarity query already exist. The application has not yet connected them to chat.
What each piece is responsible for
Four components carry the work.
Go runs the application. Today it loads configuration, controls shutdown, maintains conversation history, initializes storage, calls the model, streams output, and records operational events. The retrieval and evidence selection will live here too.
Ollama runs the model. It manages downloaded model files, loads them for inference, and exposes an HTTP API that Go can call. Think of Ollama as the service hosting the model on the machine.
Qwen generates the answer. This build’s local conversation uses qwen3.8:27b, the 27-billion-parameter model identified above. It receives messages and produces text. It is the generation model, separate from the embedding model we will use to find relevant passages.
PostgreSQL owns durable document state. The pgvector extension adds the vector type, distance operators, and vector indexes needed for similarity search. PostgreSQL still stores ordinary text and JSON metadata beside each embedding.
The Go process will coordinate the full request: find the evidence in PostgreSQL, include it in a message, and ask the model served by Ollama to explain it.
A small embedding model with a specific job
For Part II, my first candidate is Google’s EmbeddingGemma, available in Ollama as embeddinggemma. It has 300 million parameters and produces 768-dimensional vectors at full output width, which fits the current database column. Its design targets on-device use, making it a promising fit for a private archive. See the Google model card and Ollama package.
I want the least expensive model that retrieves my evidence reliably. Running a small embedding model locally avoids a per-request hosted API charge; hardware, electricity, storage, and processing time still cost something. Reusing embeddings for unchanged documents should also reduce repeated work.
I have not yet integrated or evaluated this model in the project. Its value will depend on whether it finds the right passage in my own records: an old invoice, a scanned statement, an Obsidian note. A benchmark can justify trying a model. Those questions will decide whether I keep it.
This gives the models separate jobs. The embedding model helps find passages. Qwen reads selected passages and writes an answer. There is no need to use a large generation model for both.
The project shape
The code is small enough that the directory tree tells the story:
rag-go/
├── app/ # dependency wiring and application lifetime
├── chat/ # terminal REPL and in-memory conversation
├── cmd/rag/ # executable entry point
├── config/ # environment-driven configuration
├── llm/ # provider-neutral messages + model adapter
├── logging/ # structured slog configuration
├── prompts/ # optional system prompt
├── resources/ # PostgreSQL Compose and reference schema
└── vector/
├── store.go # storage contract
└── pgvector/ # PostgreSQL implementationRead the walkthrough in that order: configuration, model client, terminal conversation, storage, then application wiring. The code blocks are excerpts from the current project, with imports and some logging omitted. I have called out a small correction in the streaming example. They explain the existing pieces; they are not a complete set of files to paste into an empty directory.
Step 0: declare the few dependencies
The module file records the toolchain and dependencies used for this build:
module rag-go
go 1.27.1
require (
github.com/jackc/pgx/v5 v5.11.0
github.com/joho/godotenv v1.5.1
github.com/openai/openai-go/v3 v3.61.0
github.com/pgvector/pgvector-go v0.4.1
github.com/pgvector/pgvector-go/pgx v0.4.1
go.opentelemetry.io/otel/trace v1.37.0
)These packages sit at external boundaries. None of them decides how to chunk a document, when to retrieve, which evidence to trust, or how to build a grounded prompt. Those decisions remain in the application.
Step 1: run the local services
Start with Go, Docker Compose, and Ollama installed. On a Mac, run Ollama natively so it can use the available GPU acceleration; the container below is for PostgreSQL. Ollama’s FAQ notes the GPU limitation of Docker Desktop on macOS. With Ollama running, pull the model used by this build:
ollama pull qwen3.8:27b
ollama listThe current development database uses the pgvector PostgreSQL image:
services:
postgres:
image: pgvector/pgvector:pg18
container_name: rag-course-postgres
restart: unless-stopped
environment:
POSTGRES_USER: rag
POSTGRES_PASSWORD: rag
POSTGRES_DB: rag
ports:
- "5432:5432"
volumes:
- pgdata:/var/lib/postgresql
healthcheck:
test: ["CMD-SHELL", "pg_isready -U rag -d rag"]
interval: 5s
timeout: 3s
retries: 10
volumes:
pgdata:These are disposable development credentials. Before using personal records, replace them and bind the published port to loopback with 127.0.0.1:5432:5432 in place of 5432:5432. The latter can expose the port on all host interfaces; Docker’s port-publishing guide explains the distinction. The named volume persists the database across container restarts; it is not a backup.
Start it from the repository root:
docker compose -f resources/docker-compose.yml up -d
docker compose -f resources/docker-compose.yml psAt this point, two independent servers are listening on the machine: Ollama on port 11434 and PostgreSQL on port 5432. The Go process will be their client.
Step 2: make configuration explicit
The application loads one configuration object from environment variables:
type Config struct {
BaseURL string
APIKey string
Model string
SystemPromptFile string
DSN string
EmbeddingDim int
}
func Load() Config {
_ = godotenv.Load()
cfg := Config{
BaseURL: os.Getenv("OPENAI_BASE_URL"),
APIKey: os.Getenv("OPENAI_API_KEY"),
Model: os.Getenv("OPENAI_MODEL"),
SystemPromptFile: os.Getenv("SYSTEM_PROMPT_FILE"),
DSN: os.Getenv("DATABASE_URL"),
EmbeddingDim: atoiOr(os.Getenv("EMBEDDING_DIM"), 768),
}
if cfg.BaseURL == "" {
cfg.BaseURL = "http://localhost:11434/v1"
}
if cfg.Model == "" {
cfg.Model = "qwen3.8:27b"
}
return cfg
}A local environment can look like this:
OPENAI_BASE_URL=http://localhost:11434/v1
OPENAI_API_KEY=ollama
OPENAI_MODEL=qwen3.8:27b
DATABASE_URL=postgres://rag:rag@localhost:5432/rag?sslmode=disable
EMBEDDING_DIM=768
SYSTEM_PROMPT_FILE=./prompts/system-custom.mdThe /v1 on the base URL selects Ollama’s OpenAI-compatible API. Its native API uses /api. This project calls the compatible endpoint so the Go SDK can speak to Ollama. The name OPENAI_BASE_URL identifies the client setting; the URL determines where the request goes. Ollama documents the supported endpoints.
The sample database URL disables TLS for local development. A remote database needs an authenticated, encrypted connection. Keep real credentials out of source control.
The model tag deserves equal attention. Environment variables override the default, so OPENAI_MODEL must match an artifact shown by ollama list. A correct endpoint with a nonexistent model still fails.
EMBEDDING_DIM fixes the width of the database vector column. Every stored embedding must match it. EmbeddingGemma’s default 768 dimensions fit this choice, but the code does not call an embedding model yet. Changing this setting later will not resize an existing table: the current migration uses CREATE TABLE IF NOT EXISTS. A change of model may require a schema migration and re-embedding the archive, even if the new model has the same vector width.
Step 3: keep conversation messages provider-neutral
The application owns a tiny message type instead of leaking SDK types into the rest of the code:
type Message struct {
Role string `json:"role"`
Content string `json:"content"`
}
type Client struct {
cfg config.Config
sdk openai.Client
}This keeps the terminal code concerned with a conversation’s roles and text. The SDK’s representation stays inside the adapter, where it can change without spreading through the application.
The constructor configures the client with the endpoint and credential supplied by the application:
func New(cfg config.Config) *Client {
opts := []option.RequestOption{}
if cfg.BaseURL != "" {
opts = append(opts, option.WithBaseURL(cfg.BaseURL))
}
if cfg.APIKey != "" {
opts = append(opts, option.WithAPIKey(cfg.APIKey))
}
return &Client{cfg: cfg, sdk: openai.NewClient(opts...)}
}Ollama’s local compatibility example supplies an API-key value for the client even though the local server ignores it. That is why ollama appears in the sample configuration. It does not secure the local inference endpoint.
Step 4: translate at the boundary and stream the result
Only the adapter translates application messages into SDK messages:
func toSDKMessages(messages []Message) []openai.ChatCompletionMessageParamUnion {
out := make([]openai.ChatCompletionMessageParamUnion, 0, len(messages))
for _, m := range messages {
switch m.Role {
case "system":
out = append(out, openai.SystemMessage(m.Content))
case "assistant":
out = append(out, openai.AssistantMessage(m.Content))
default:
out = append(out, openai.UserMessage(m.Content))
}
}
return out
}Unknown roles become user messages. This conservative default keeps an unexpected role string from becoming a system instruction. It does not, by itself, defend against instructions embedded in retrieved documents.
The model call uses streaming rather than waiting for one large response:
func (c *Client) ChatStream(
ctx context.Context,
messages []Message,
onDelta func(string),
) (Message, error) {
stream := c.sdk.Chat.Completions.NewStreaming(ctx, openai.ChatCompletionNewParams{
Model: c.cfg.Model,
Messages: toSDKMessages(messages),
})
defer stream.Close()
var content strings.Builder
role := "assistant"
for stream.Next() {
chunk := stream.Current()
if len(chunk.Choices) == 0 {
continue
}
delta := chunk.Choices[0].Delta
if delta.Role != "" {
role = delta.Role
}
if delta.Content != "" {
content.WriteString(delta.Content)
if onDelta != nil {
onDelta(delta.Content)
}
}
}
if err := stream.Err(); err != nil {
return Message{}, err
}
return Message{Role: role, Content: content.String()}, nil
}One correction is shown explicitly here: the source currently initializes the fallback role with the typo "assistance". The excerpt uses "assistant", the role expected by toSDKMessages. A role returned by the stream normally replaces the fallback, but the fallback should still be valid.
Each arriving fragment goes to onDelta for display. At the same time, a strings.Builder assembles the complete assistant message for conversation history.
Both are needed: the reader sees progress as text arrives, and the next request retains the completed answer. Streaming does not remove the wait for the first token, but it makes a longer local response easier to follow.
The supplied context.Context also matters. Cancellation can travel from an operating-system signal through the application and into the in-flight HTTP request.
Step 5: keep the conversation in the terminal
The REPL—read, evaluate, print, loop—reads a line, handles empty input and quit, appends the user message, calls the client, prints each streamed fragment, and stores the completed assistant reply.
The core of the loop is:
input := strings.TrimSpace(in.Text())
if input == "" {
continue
}
if strings.EqualFold(input, "q") {
writeTerminal("Bye..\n")
return nil
}
history = append(history, llm.Message{
Role: "user",
Content: input,
})
reply, err := client.ChatStream(ctx, history, func(fragment string) {
writeTerminal(fragment)
})
if err != nil {
slog.ErrorContext(ctx, "model request failed", slog.Any("error", err))
continue
}
history = append(history, reply)The implementation also adds a spinner, request timing, diagnostic logs, and an optional system prompt. History lives in memory for this session and is sent with each request. It is not a durable archive and will eventually need a context budget as conversations grow. Retrieval has not been connected to this loop yet.
The system prompt used during the current test asks the model to behave like a sixth-grade teacher in Islamabad and explain ideas in accessible, expository language. A system prompt is simply the first message in history; it changes the model’s posture without changing the transport.
The first real conversation
With PostgreSQL healthy, Ollama running, and qwen3.8:27b installed, run the application from the repository root:
go mod download
go run ./cmd/ragUse the local environment settings above; existing shell variables take precedence over .env. At the prompt, I entered:
> Hello, world! Introduce yourself in one short sentence.The locally served model streamed this response:
Salam! I'm your class teacher, and I'm here to make sense of every tricky
thing in your book — no question is too small, promise.The teacher’s greeting comes from the test system prompt. It is useful here because it makes the prompt’s influence visible; the eventual archive assistant will need different instructions. Your wording may vary from this recorded response.
The request has now crossed the full conversation path:
terminal input
→ provider-neutral Go message
→ OpenAI-compatible chat request
→ Ollama on localhost
→ qwen3.8:27b
→ streamed deltas
→ terminal output + stored conversation historyAt this point I can speak to a model through my own Go application. It still has no access to a birthday list, bank statement, or Obsidian note. Those documents must take a separate route into storage and then into the prompt.
Step 6: define storage before defining ingestion
The vector package begins with a contract:
type Document struct {
ID string
Content string
Metadata map[string]string
Embedding []float32
}
type Result struct {
Document
Score float32
}
type Store interface {
Upsert(ctx context.Context, docs []Document) error
Query(ctx context.Context, embedding []float32, topK int) ([]Result, error)
Delete(ctx context.Context, ids []string) error
DeleteBySource(ctx context.Context, source string) error
Close() error
}This interface says something useful about the future ingestion pipeline without inventing it.
A stored record represents a passage: its ID, text, source metadata, and embedding. Upsert can replace a record with the same ID. DeleteBySource can remove the passages belonging to a replaced file. Query accepts a question embedding and returns passages with similarity scores. Close releases the database resources.
For my archive, the source metadata will matter as much as the text. A passage from a closing statement needs a path and page number. A bank statement needs its account identity and reporting period. An Obsidian note needs its original location and links. The interface can carry metadata, but the ingestion code must still define and validate it. It must also assign stable IDs before repeated upserts become reliable re-ingestion.
Embedding generation remains the caller’s responsibility. That keeps storage from silently reaching into Ollama or another provider.
Step 7: create a schema that supports the future path
The migration creates the extension, table, and two indexes:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE IF NOT EXISTS documents (
id text PRIMARY KEY,
content text NOT NULL,
metadata jsonb NOT NULL DEFAULT '{}'::jsonb,
embedding vector(768) NOT NULL,
created_at timestamp with time zone NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS documents_embedding_idx
ON documents USING hnsw (embedding vector_cosine_ops);
CREATE INDEX IF NOT EXISTS documents_source_idx
ON documents ((metadata ->> 'source'));Each column has a reason:
idis the stable identity used for replacement and deletion.contentis the passage that will eventually enter the model’s context.metadatacan carry source path, title, page, section, checksum, or document date without flattening every possible attribute into a column on day one.embeddingis fixed at 768 dimensions for this stage of the design.created_atrecords when the row was inserted.
That last timestamp is not the date of the event described in a document. Loading a ten-year-old statement today must not make its balance current. Effective dates and document versions still need a deliberate design before the archive can answer historical questions.
The HNSW index uses vector_cosine_ops to support cosine-distance search. It searches approximately, trading some recall for speed. For a small archive, I will also want to compare its results with exact search before assuming the index is necessary. The pgvector documentation explains these choices.
The source expression index supports re-ingestion. Deleting every passage where metadata ->> 'source' matches a file should not require scanning the entire table.
Step 8: make similarity search visible
The PostgreSQL implementation does not conceal the retrieval query:
const statement = `
SELECT id, content, metadata, embedding,
1 - (embedding <=> $1) AS score
FROM documents
ORDER BY embedding <=> $1
LIMIT $2`
rows, err := pgVecStore.pool.Query(
ctx,
statement,
pgvectorgo.NewVector(embedding),
topK,
)In pgvector, <=> is cosine distance: smaller values are closer. The query orders by the distance expression so PostgreSQL can use the matching index. It returns 1 - distance as a similarity score, where larger values mean closer passages. This score is not a probability that the passage is correct. pgvector documents the operator and index requirements.
The storage method is implemented. The application flow still needs to generate the question embedding, call Query, assess the results, and give selected evidence to the model.
Notice that this query has no account or date filter. A question about a particular home or reporting period cannot rely on similarity alone to respect those limits. That is work for the retrieval design in the next part.
Step 9: wire infrastructure once
The application layer currently creates the two long-lived dependencies:
func Run(ctx context.Context, cfg config.Config) error {
client := llm.New(cfg)
vectorStore, err := initVectorStore(ctx, cfg)
if err != nil {
return err
}
if vectorStore != nil {
defer vectorStore.Close()
slog.InfoContext(ctx, "Vector Store is ready now...")
}
return chat.RunREPL(ctx, client, chat.Options{
SystemPromptFile: cfg.SystemPromptFile,
})
}
func initVectorStore(ctx context.Context, cfg config.Config) (vector.Store, error) {
return pgvector.New(ctx, pgvector.Options{
DSN: cfg.DSN,
EmbeddingDim: cfg.EmbeddingDim,
})
}Look at what enters RunREPL: the client and chat options. vectorStore is initialized and scheduled for cleanup, but it is not passed to the conversation. The wiring shows exactly where this installment ends. Storage and chat exist as separate capabilities.
The migration runs when the store is constructed. At debug level it now reports whether the table was created or already present:
{
"level": "DEBUG",
"msg": "database migration completed",
"documents_table_status": "already_exists"
}That status comes from PostgreSQL’s catalog rather than serializing a DDL command tag. A command tag can say CREATE TABLE, but CREATE TABLE IF NOT EXISTS does not tell the client whether PostgreSQL created or skipped the table.
Step 10: treat logs as part of the architecture
The program uses Go’s standard log/slog JSON handler. A small project wrapper adds service fields, a custom trace level, and OpenTelemetry trace/span IDs when a context contains them. JSON formatting comes from the standard library; the wrapper supplies the project’s conventions.
Normal call sites remain standard Go:
slog.InfoContext(ctx, "application starting",
slog.String("model", cfg.Model),
slog.Int("embedding_dimension", cfg.EmbeddingDim),
slog.Bool("database_configured", cfg.DSN != ""),
)
slog.ErrorContext(ctx, "model request failed",
slog.Any("error", err),
slog.Duration("duration", elapsed),
)The terminal answer goes to stdout. Operational JSON goes to stderr. That separation keeps a human interface readable and gives log collectors structured records.
The routine conversation logs record counts and duration without deliberately recording prompts or answers. That is a useful starting point for this archive. Before ingestion, error paths and source metadata will need the same review: a filename or provider error can disclose information too. I want enough detail to diagnose a failed request without creating a second copy of family records in the logs.
What the records will demand in Part II
The first ingestion test should be small enough to inspect by hand. I would start with a few redacted or synthetic records: a remodeling estimate, its final invoice, a payment record, and an Obsidian note explaining the work. Then I can ask a question whose evidence I already know and follow each passage through the pipeline.
Keep the page behind the passage
Markdown provides a manageable first format. PDFs, scans, and spreadsheets add harder problems: reading order, tables, handwriting, dates, and amounts whose meaning depends on a column heading. The extraction layer must retain the page or sheet reference, preserve the original, and flag text that needs review. OCR that reads an amount incorrectly can make every later step look convincing and still produce a wrong answer.
For Obsidian, I want to preserve headings and links between notes. A passage that says “see the final invoice” loses much of its value if ingestion discards that connection. For diaries, the entry date, page, and surrounding sequence matter. An entry written the following morning may describe the previous day; ingestion must preserve that distinction rather than attach every event to the writing date.
Divide documents without losing their meaning
Chunking will turn extracted text into passages that fit the embedding model and remain useful when read alone. A remodeling line item needs its description and amount together. A bank balance needs its account and statement date. Splitting after an arbitrary number of characters can separate the very facts the question depends on.
EmbeddingGemma’s input limit is 2,048 tokens, so the passage and its retrieval formatting must fit that budget. Its model card also specifies different formatting for queries and documents. The adapter will need to follow those conventions consistently rather than send two unrelated forms of text into the same vector space. Google’s model card describes both requirements.
Make ingestion safe to repeat
The ingestion command will read a source, extract and divide its text, generate embeddings, attach metadata, and call Store.Upsert. Checksums and stable IDs should keep unchanged files from being processed again. Model identity and version also need to be recorded so the archive does not silently mix incompatible embeddings.
Replacing a file needs care. If extraction fails halfway through a new version, I want the old searchable version to remain available. The store already exposes deletion and upsert operations; coordinating them into a reliable replacement is still application work.
Retrieve evidence that answers the actual question
The question path will embed the request, retrieve candidates, and apply the relevant source, date, and identity constraints. Some questions will need exact text search as well as vectors. An invoice number should not become an approximate match merely because the rest of two invoices looks similar.
Then Go will assemble a bounded set of evidence for the generation model. A retrieved document is untrusted input: text inside it must not acquire the authority to redirect the application or override its instructions. Source delimiters and clear prompts are part of the design, alongside limits on what the application can do.
The answer should cite the path and page or section behind each material claim. If two records disagree, I want to see the disagreement. If the needed record is absent, I want the system to say what it could not establish.
Test with questions I can check
I will need a small set of questions with known evidence: locate the diary entry for a birthday, find a closing document, identify the venue on a contract, locate a balance for the right period, connect an Obsidian note to its source. I also need questions the collection cannot answer, because refusing to fill a gap is part of a useful archive.
The checks should distinguish finding the right passage from writing a faithful answer. A fluent response with the wrong statement date fails. So does a correct-looking total without a complete set of inputs. Latency will be measured too, but I will accept a slower answer that I can trace and verify.
That first evidence-backed exchange is the target for Part II. The full lifetime archive, and especially reliable historical net-worth reconstruction, will need further work beyond it.
Before the first document goes in
We have reached a useful stopping point. Go can read a question, send it to Qwen through Ollama, stream the answer, and retain the conversation. PostgreSQL has a place for passages, metadata, and vectors. The storage layer exposes the operations the ingestion and retrieval paths will need.
I can follow that work in a handful of Go packages. That is what I wanted from building the foundation myself: a clear place to look when the next piece succeeds or fails.
The reason to continue is waiting in the folders and diary pages. My son’s birthday at McDonald’s in Karachi, about thirty years ago. My daughter’s wedding, with its guests, seating plan, and gifts. An ordinary day I took the trouble to write about. A payment I want to reconcile. A decision whose explanation I wrote years ago. Eventually, a dated account of what we owned and owed, with every number tied to a record.
That is the purpose of LifeLedger: to make a lifetime of keeping records useful in the moment I need to remember, understand, or check something.
In Part II, we will take a small set of documents through extraction, chunking, local embeddings, storage, and retrieval. We will inspect what the search finds before asking the model to explain it. Then we will ask the question that gives this project its purpose:
“Can you show me where that answer came from?”
References
- Ollama: OpenAI compatibility
- Ollama model library: Qwen3.8
- Ollama: local operation and cloud settings
- Google: EmbeddingGemma model card
- Ollama model library: EmbeddingGemma
- pgvector: vector search and HNSW indexes
- Go
log/slogpackage