Catalens

Nothing here is exotic, but two early steps assume tools that aren’t always present, so install them once now:

• Your backend runtime — the Go toolchain (1.22+, for the default path) or Node (20+, for the TypeScript path). Only the one you pick.
• mongosh, the MongoDB Shell, for the connection smoke-test. Install it from the official download, or skip the install and use the built-in shell in the Atlas UI (Cluster → “Connect” → “MongoDB Shell”).
• curl and base64 (both ship with macOS and Linux) for the Gemini smoke-test.
• A sample product JPEG you can match later — grab any product photo and save it as sample.jpg in your working directory. Pick products you can actually photograph (or screenshot), because the match compares a text descriptor of your photo against a text descriptor of the catalog (more on that at ingest).

Costs nothing. Every tool here is free and open-source; no account or card is needed for this step.

The whole project hinges on Atlas Vector Search, a Lucene-backed vector index that is part of Atlas, not the core database engine. A local mongod (or the Docker image) can store documents and even hold an embedding array, but it cannot build the Atlas Vector Search index that $vectorSearch queries — so for this build the database lives in Atlas from day one. The M0 free tier is a real replica set, and Atlas Vector Search and Atlas Search both run on it, so the entire project is free to complete.

Costs nothing. M0 is free (no card), and the vector + text search indexes it supports are part of that tier — confirm the current limits on the Atlas docs.

Catalens uses Gemini for two different jobs, both on the free tier of a single Google AI Studio key: Vision (turn a photo into a structured descriptor) and embeddings (turn text into a vector). Confirm both respond before building on them. Keep the key server-side — the mobile app always calls your backend, never Gemini directly. Don’t hard-code a model id in your app: ids change, so read the current ones from the official model list and pin them in config.

Costs nothing. A Google AI Studio key has a free tier that covers Vision and embeddings for this project. Docs: image understanding — https://ai.google.dev/gemini-api/docs/image-understanding · embeddings — https://ai.google.dev/gemini-api/docs/embeddings · models — https://ai.google.dev/gemini-api/docs/models · Go SDK — https://pkg.go.dev/google.golang.org/genai

A real catalog isn’t uniform: a sneaker has a size run and a material, a tea has a flavour and a caffeine level, a chair has dimensions and a finish. In Postgres that is a wide sparse table, an EAV side-table, or a jsonb column you’ve stopped validating. In MongoDB each product is just a document with the fields its category needs, plus a few common fields every match relies on — category, brand, name, attributes, and (added next) an embedding vector. The shared fields are what you pre-filter on; the per-category attributes are what make the catalog flexible. This heterogeneity is precisely why the document store, not a relational schema, is load-bearing here.

Seed products you can actually photograph. The match (next steps) compares a text descriptor of your photo against a text descriptor of each catalog product — not pixels against pixels — so for the demo to return a real match, the seeded products must be things you can take (or screenshot/generate) a recognisable photo of. Pick a handful of clearly distinct, photographable items.

A fair question: why not embed the image directly? Two honest roads to visual search exist — (1) a multimodal image embedding (compare photo-pixels to catalog-image-pixels), or (2) turn the photo into words with Vision, then compare those words to the catalog’s words. Catalens takes road (2) on purpose: it keeps the whole project on one free embedding model for catalog and query, gives you a human-readable descriptor you can debug, and makes every match explainable. The cost is the invariant this creates — you must build the same descriptor text and embed it the same way for both the catalog (ingest) and the photo (query), or the vectors aren’t comparable. That shared builder is the next step.

The whole match rests on an invariant: the catalog side and the query side must be embedded identically. The cleanest way to guarantee that is to write the descriptor-to-text logic once and call it from every place that embeds — the ingest pass, the /recognize query, and (later) the change-stream worker. Put it in a small module (embeddingText in the spec’s internal/embedtext / src/embedText.ts). It takes the fields that describe a product — brand, name, category, and the attribute values — and joins them into one short string, the same way for a catalog document and for a Vision descriptor.

The second function is l2normalize. Because some embedding models only auto-normalize at their full dimensions, anything you embed at a smaller D (like 768 or 1536) may need to be divided by its own magnitude so cosine similarity is meaningful. Check your model’s documentation; if it requires it, wrap that logic in l2normalize(vector) and call it on every vector you produce — catalog and query — right after the embed call. With both helpers shared, “embed the same way” stops being a comment you hope everyone honours and becomes a function everyone calls.

Vector search compares vectors, so each product needs one. Use the shared embeddingText(...) builder from the previous step, send its output to the Gemini embedding model, then l2normalize the result and store it on embedding. Three rules make or break the match later: (1) the embedding’s length must equal the numDimensions you’ll set on the index, so choose an outputDimensionality D and keep it fixed for the whole catalog; (2) you must embed the query the same way you embed the catalog — same model, same dimensions, same normalization — or the vectors aren’t comparable; (3) if your chosen model does not auto-normalize at smaller dimensions, L2-normalize every vector (the helper you just wrote), or cosine scores come back distorted. Also store an embeddingHash (sha256 of the embedding text) so the ingest and the later live worker can skip a product whose descriptive text hasn’t changed. Keep the API key server-side.

$vectorSearch needs a special index, created in the Atlas UI or via the API on your collection. The definition is a fields array: one entry of type: "vector" for the embedding path — with numDimensions equal to your embedding length and similarity set to cosine (the usual choice for text/descriptor embeddings; euclidean and dotProduct are the alternatives) — and one type: "filter" entry for each metadata field you want to pre-filter on (category, brand, and inStock — the last one for the Visual Substitutes feature later). Only fields declared as filter can be used in the stage’s filter. Name the index products_vec — you reference that name in $vectorSearch. numDimensions must match the model output you chose at ingest; a mismatch is the single most common setup error. Docs: https://www.mongodb.com/docs/atlas/atlas-vector-search/

The recognise flow is a short pipeline: photo → Gemini Vision (structured descriptor) → embedding → $vectorSearch (with a pre-filter) → threshold → ranked matches or no-match. The fragile link is the first hop — a free-text image caption is unusable downstream. So constrain Vision with responseMimeType: "application/json" and a responseSchema: ask for brand, category, attributes, visibleText, colour, and form as typed fields. A typed descriptor gives you two things at once: a clean string to embed, and structured fields (category, brand) to drive the $vectorSearch pre-filter. Treat the output as best guess, not identification — the model can be wrong, which is exactly why the threshold and the no-confident-match path exist.

Constrain category to your catalog’s known values. The pre-filter does an exact $eq on category, so if Vision answers “shoe” or “footwear” while your catalog says “sneakers”, a correct pre-filter silently excludes the right product — a false no-match that looks like “not in catalog”. Close that gap two ways, both used: (1) make category an enum in the responseSchema listing exactly the categories you seeded, so the model must pick one of them; and (2) at query time, fall back to an unfiltered search if the category-filtered search returns nothing, so a mislabel degrades to “search everything” rather than a silent miss. The pre-filter stays a win — it just can’t be allowed to be a silent loss.

go.mongodb.org/mongo-driver/v2 is the current official driver. One go get of the module’s mongo sub-package pulls the whole module, but the code uses three of its packages, so all three go in your import block: .../v2/mongo (the client + mongo.Connect, mongo.Pipeline), .../v2/mongo/options (options.Client().ApplyURI(...)), and .../v2/bson (bson.D, bson.ObjectID — note v2 renamed primitive.ObjectID to bson.ObjectID). Open one mongo.Client at startup and reuse it; pass a context on every call. This step is also the app entrypoint that the spec calls cmd/api/main.go: it loads config, opens the client, registers routes, and shuts the client down. /recognize takes a multipart/form-data image (plus an optional category hint) and, for now, just decodes it and returns 200 — you fill in the pipeline next.

The mongodb package is the official Node driver; create one MongoClient at startup and reuse it. Use a small web framework (Hono here) for routing and multipart parsing. /recognize accepts a multipart/form-data image plus an optional category hint and returns 200 for now — the pipeline lands in the next step. Keeping the backend in TypeScript means the same types describe the API the mobile client calls.

You designed the JSON schema earlier; now you pass it to the SDK. By enforcing responseMimeType: "application/json" and translating the schema into the SDK’s type system, you guarantee Gemini returns a typed descriptor rather than prose. Don’t hard-code a model id — read the current Vision model from config (GEMINI_VISION_MODEL) and pick it from the official model list, since ids change.

Everything converges here. Vision returns the typed descriptor; you build the same embedding text you used at ingest and embed it (same model, same dimensions); that query vector goes straight into a $vectorSearch stage. The stage carries five things: the index name, the path (embedding), the queryVector, a numCandidates (set it roughly 20× your limit for accuracy), and a limit. The filter — { category: { $eq: descriptor.category } } — is a pre-filter: Atlas narrows to the right category before the vector comparison, in the same pipeline, and it doesn’t distort the score. Source the category from the Vision descriptor, not the optional category form value (which is normally omitted), so the pre-filter actually fires on a real scan. A $project stage pulls each document plus its similarity via { $meta: "vectorSearchScore" } and renames _id to a string id (the substitutes feature looks products up by it). Note what isn’t here: no join, no second query, no app-side re-ranking — the heterogeneous catalog and the vector index do the work together. That co-location is why MongoDB scores 5 on this build.

The pre-filter is a win that must not become a silent loss. If Vision labels a photo "footwear" while your catalog says "sneakers" — or labels a genuinely unstocked item with a real category that simply has no close product — the category-filtered $vectorSearch can return zero documents, which looks identical to “not in catalog”. So when the filtered search comes back empty, retry the same search with no filter before declaring a no-match: a mislabel degrades to “search everything”, and only a truly out-of-catalog photo reaches the threshold step with nothing. Report which path ran via filterApplied — the category string when the filtered search produced the matches, or null when you fell back to unfiltered — so the client can show the spotlight pre-filter at work (and show it stepping aside when it had to).

The shape is identical to the Go path — only the syntax differs. Build the descriptor with a Vision call, embed it with the same model and dimensions as ingest, and pass the query vector into a $vectorSearch stage with index, path, queryVector, numCandidates (~20× limit), limit, and a filter pre-filter on the descriptor’s category. A $project renames _id to a string id and adds score: { $meta: "vectorSearchScore" }, and aggregate(...).toArray() returns the ranked matches. As in Go, if the category-filtered search returns nothing, retry it unfiltered before the threshold step sees anything, and report which path ran via filterApplied. Keeping the two backends behaviourally identical is the parity lesson: swap the language, the document-plus-vector store still carries the match.

$vectorSearch always returns up to limit nearest neighbours — even for a photo of something not in the catalog, the “nearest” is just less near. The vectorSearchScore (0 to 1 for cosine, higher is closer) is how you tell a real match from the nearest stranger. Apply a threshold T: keep matches with a score at or above T, ranked; if the list is empty, return a structured noMatch so the UI shows “no confident match” rather than a misleading product. Expose the raw score in the response so the client can show it and so you can tune T. Carry the descriptor and the filterApplied value from the recognise step into both branches of the response — the client shows what Vision saw and which pre-filter fired (or that it fell back) whether or not anything cleared T. This single decision — return ranked confident matches, or admit you don’t know — is what keeps the feature honest.

A threshold trades two errors against each other. Set T too high and real matches fall below it — high precision, low recall, lots of false “no match”. Set it too low and strangers sneak in — high recall, low precision, confident-looking wrong answers. Calibrate empirically: score a handful of photos you know are in the catalog and a handful you know aren’t, and put T in the gap between the two score clusters. Because cosine scores aren’t absolute across embedding models, re-calibrate if you change the model or dimensions. Then design the no-match experience as a first-class state, not an error: “No confident match — try a clearer photo or pick a category.” A slider that lets the learner move T live (the next screens add one) makes this trade-off something you can see, not just reason about.

This is the “something to see”. Capture a photo (CameraX) or pick one from the gallery, upload it as multipart/form-data to /recognize, and show the ranked results: each product with its details and a match score. Add a threshold slider that filters the already-returned list client-side, so dragging it shows precision versus recall move in real time — matches appear and disappear as the learner raises or lowers the cutoff. Above the list, render a small descriptor + filter panel: what Vision saw and the filterApplied value — the category the pre-filter used, or “searched all categories” when the backend fell back to an unfiltered search — so the spotlight is visible, not hidden in the response JSON. When the confident list is empty, show the no-confident-match state. Emulator cameras are limited, so ship a few bundled sample product photos alongside gallery-pick so every scan is free and reproducible.

Use image_picker for camera/gallery capture and http (a MultipartRequest) to upload the image to /recognize. Hold the returned matches and the slider value in state; filter the shown list by score so the threshold slider re-filters instantly. Render the no-confident-match state when the filtered list is empty. Bundle a couple of sample asset images so the flow runs on any simulator.

Use PhotosPicker (or the camera) to get an image, upload it with URLSession as multipart/form-data, and decode { matches, noMatch } with Codable. Keep the matches and the threshold in an @Observable model; a Slider filters the shown matches by score. Show the no-confident-match state when the filtered list is empty, and bundle a few sample images in the asset catalog for camera-less runs.

$vectorSearch only exists in Atlas, so a mock can’t prove the match. Point the test at your M0 cluster (via MONGODB_URI): seed and embed a known product, post its bundled photo to /recognize, and assert it appears above T. Then post a photo of something not in the catalog and assert noMatch is true with an empty match list. Skip cleanly if MONGODB_URI or GEMINI_API_KEY is unset so the suite still runs without infra.

Run the same scenario with node --test (or vitest): seed and embed a known product, post its sample photo, assert it lands above T; then post an out-of-catalog photo and assert noMatch. Running both suites against the same Atlas cluster is what lets Catalens claim, honestly, that the document-plus-vector store — not the language — carries the match.

Nothing about seeing Catalens work needs the cloud: the API runs locally against Atlas M0, and the mobile app talks to it. If you want a public endpoint, the recognise API is a stateless container, so Cloud Run fits — it scales to zero and has a generous free monthly allotment. The data stays in Atlas (M0 is fine); only the connection string and the GEMINI_API_KEY move into the service config. Keep the key server-side as a managed secret (Secret Manager), never in the app.

Pure vector similarity can rank a look-alike from the wrong brand above the right product. Hybrid search fixes that by blending two retrievers: $vectorSearch (visual/semantic similarity) and Atlas Search ($search, a Lucene full-text index) over brand, name, and visibleText. Run both, then fuse their rankings — the common technique is reciprocal rank fusion (RRF): each result’s combined score sums 1 / (k + rank) across the two lists, so items ranked high by either signal rise. Recent Atlas versions expose a $rankFusion stage that does RRF for you; follow the current official hybrid-search guide rather than hand-tuning weights, and keep a single confidence cut on the fused score.

You already have the machinery; substitutes just ask it a new question. Take the matched (or out-of-stock) product’s own embedding as the query vector, run $vectorSearch excluding that product, and add an inStock: true pre-filter so only buyable items come back. The result is “things that look like this, that you can actually buy”, ranked by similarity. This is where you lower the threshold deliberately: for substitutes you want recall (show me close-enough options), the opposite of the strict precision you want for identification. Same index, same stage — a looser cutoff and an availability filter.

A shelf photo contains many products, so the single-product pipeline becomes a fan-out. Two honest approaches: (1) ask Gemini Vision to return an array of detected items — each with a short descriptor and an approximate bounding box — using a responseSchema whose top level is an array; or (2) detect regions first, then run the existing descriptor, embed, and $vectorSearch path per crop. Either way you reuse the same per-item pipeline, just N times, and return a list of { box, matches[] }. Keep each item’s threshold and no-match handling exactly as the single-product flow — a shelf with one unknown item should confidently match the rest and admit the one.

Vector matching is for when you don’t have an identifier; when you do, use it. If the photo contains a barcode or legible label, read it — a barcode-scanning library on-device, or Gemini Vision’s visibleText from the descriptor you already extract — and look the product up by an exact field (sku or barcode) with a normal indexed query. An exact hit returns immediately with full confidence and skips the vector path entirely; only when there is no code, or no exact match, do you fall back to the descriptor, embed, and $vectorSearch path. This is a classic cheap-path/expensive-path design: the deterministic lookup is faster, free of model cost, and unambiguous, so it goes first.

The embedding you stored at ingest is a snapshot of a product’s description at that moment. Edit the brand, rename it, add an attribute, swap the photo — and the stored vector now describes a product that no longer exists, so $vectorSearch quietly matches against stale text. The fix is to treat the embedding as derived data that must be recomputed whenever its inputs change. A change stream is MongoDB’s real-time feed of writes: collection.watch() opens a cursor that yields one event per insert/update/delete, each carrying a resume token (in the event’s _id) so a restarted worker continues exactly where it left off. Ask for fullDocument: "updateLookup" and each update event also includes the current document, so the worker has the new text to embed without a second read. The whole loop runs on the data you already provisioned.

Costs nothing. Atlas M0 is a 3-node replica set, and change streams run on it — the only free-tier restriction is on database-namespace filters (you filter on fields and collections, which is allowed), so a worker watching the products collection works on the free tier. Docs: https://www.mongodb.com/docs/manual/changeStreams/ · M0 limits: https://www.mongodb.com/docs/atlas/reference/free-shared-limitations/

Writing the fresh vector back is itself a write to products — which your change stream will see, which would trigger another re-embed, forever. Break the loop by watching only the changes that matter: pass a pipeline to watch() that $matches content edits and ignores the embedding write. Two robust guards, used together: (1) store a content hash of the embedding text on the document and skip any event whose hash is unchanged — so rewriting embedding (which doesn’t touch the hash) is a no-op; (2) $match on the event so embedding-only updates never reach the worker (e.g. require a content field in updateDescription.updatedFields, or filter operationType). Make the worker idempotent and debounced: a burst of edits to one product should collapse into a single re-embed of its final state, keyed by _id, so rapid saves don’t fan out into redundant Gemini calls.

collection.Watch(ctx, pipeline, opts) returns a *mongo.ChangeStream; drive it with stream.Next(ctx) + stream.Decode(&event). Set options.ChangeStream().SetFullDocument(options.UpdateLookup) so each update event carries the current fullDocument to embed. Read stream.ResumeToken() after each handled event and persist it (a tiny _worker_state doc); on startup pass it back via SetResumeAfter(token) so a crash resumes without missing or re-doing work. The $match pipeline (built with bson.D) keeps embedding-only writes out of the stream, and the content-hash check makes the handler idempotent — together they close the infinite-loop door. Keep the API key server-side and reuse the same embeddingText builder and dimensions D as ingest. Go driver change streams: https://www.mongodb.com/docs/drivers/go/current/monitoring-and-logging/change-streams/

The Node path mirrors the Go one. collection.watch(pipeline, { fullDocument: "updateLookup", resumeAfter }) returns an async-iterable change stream; consume it with for await (const change of stream). Each event’s resume token is change._id (also stream.resumeToken); persist it and pass it as resumeAfter on the next start. The same $match pipeline filters out embedding-only writes, and the same content-hash guard makes the handler idempotent — re-embed only when the rebuilt text’s hash differs from the stored embeddingHash. Reuse the ingest step’s embeddingText builder, model, and dimensions D so the worker and the query stay comparable. Keeping both workers behaviourally identical is the same parity lesson as the recognise pipeline. Node driver change streams: https://www.mongodb.com/docs/drivers/node/current/monitoring-and-logging/change-streams/

$vectorSearch runs approximate nearest-neighbour (ANN) search over an HNSW graph index: instead of comparing your query against every product vector, it walks the graph and considers a bounded pool of candidates, then returns the best limit of them after applying any filter. numCandidates is the size of that pool. Bigger pool → the search explores more of the graph → it’s more likely to find the true nearest neighbours (higher recall) but does more work (higher latency). Smaller pool → faster, but it may miss a real match that the graph walk never visited. numCandidates is required for ANN search and applies only to ANN — set exact: true and you get exhaustive ENN (perfect recall, no numCandidates, slowest), which is mainly useful as a ground-truth baseline to measure your ANN recall against on a small catalog. The official guidance: set numCandidates at least 20× your limit (e.g. limit: 5 → numCandidates: 100) as a starting point, and it must be ≥ limit. Docs: https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/

The category filter from the ★ recognise step interacts with this knob: Atlas applies the filter while traversing the graph, so an aggressive filter over a small catalog can leave fewer than numCandidates qualifying vectors — exactly when bumping numCandidates (or, for a tiny in-category set, exact: true) keeps recall honest. The query call is otherwise unchanged from the recognise step, so this knob lives in the same $vectorSearch stage in both backends — only the integer differs (Go: numCandidates in the bson.D; TypeScript: numCandidates in the stage object).

Recall here is concrete: for each test query, run the ANN search and an exact: true ENN search with the same limit, and recall@limit is the fraction of the ENN top-limit that your ANN result also returned. Average it over a handful of representative query photos. Sweep numCandidates (say 25 → 400) and watch two curves: recall rises and flattens toward 1.0, while p95 latency rises roughly with the candidate pool. The sweet spot is the knee — the smallest numCandidates whose recall is within your tolerance of the ENN baseline (for a precision-sensitive shopping match, aim high). Because ENN is O(n), use it only as a measurement tool on a small catalog, never as the production path. Re-run the sweep if you change the embedding model, dimensions, or the catalog’s size — the knee moves with all three.

The recognise pipeline already computes everything a demand signal needs: the Gemini Vision descriptor (what the customer photographed) and the scores of the nearest products (how close you came). When every score falls below the threshold T, that’s not just a UX dead-end — it’s a data point: someone wanted this, and you couldn’t match it. Write each miss to a dedicated search_misses collection: the descriptor fields (category, brand, colour, form, visibleText, attributes), the top near-miss {name, score} entries, and a timestamp. Over time that collection is a real-time ledger of unmet demand, far richer than a restock guess. Keep it privacy-aware: store the descriptor and scores, not the raw user image, by default — you’re logging what was wanted, not who asked. (If you ever need images for QA, make it opt-in and short-retention; the descriptor alone drives the analytics.) Use a separate collection so analytics writes never touch the catalog the recognise path reads.

You already detect the no-match case (the confident list is empty). At that point, build the miss document from the descriptor you computed and the below-T matches you ranked, and InsertOne it into a separate search_misses collection. Do it without blocking or failing the request: the customer still gets their honest “no confident match” even if the analytics write errors, so log-and-ignore the insert error (or hand it to a small buffered channel / goroutine). Reuse the existing Mongo client — just a different collection handle — and never store the raw image. This is the only new code on the hot path; the ranking that consumes it is common and comes next.

The TypeScript path matches the Go one. In the no-match branch, assemble the same miss document — descriptor, top below-T {name, score} near-misses, threshold, timestamp — and insertOne it into a separate search_misses collection handle. Don’t await it on the response path (or wrap it so a rejection is caught and logged): the no-match JSON returns regardless. Reuse the existing MongoClient, and store no raw image. Keeping both writes identical means the demand ledger is the same whichever backend is deployed — the same parity lesson as the recognise pipeline.

A pile of miss documents only becomes useful when you summarise it. An aggregation pipeline does the whole job in the database: $group the misses by the descriptor fields that define “the same kind of thing” (descriptor.category plus descriptor.brand, optionally colour/form), $count each group, $sort descending, and $limit to a top-N. Add the average best near-miss score per group and you also learn how close you came — a group with high demand and high near-miss scores is a strong “stock something almost like this” signal, while high demand with low scores is genuinely new. This ranking is common to both backends (it’s one aggregation, no language-specific logic), and because it reads only search_misses it never burdens the recognise path. Surface it on an internal GET /analytics/top-misses endpoint or a scheduled report.