Catalens — Project Spec

Single source of truth for the Catalens course (src/content/projects/catalens.mdx). The course must teach toward exactly this runnable project. Spotlight: MongoDB Atlas Vector Search. Backends: Go (default) + TypeScript, both implementing the same contract. Free to complete ($0): Atlas M0 + a free Google AI Studio key + local runtime + an emulator.

1. Overview & definition of done

Catalens is a visual product-recognition service. A shopper photographs a product; the backend turns the photo into a typed descriptor (Gemini Vision), embeds that descriptor (Gemini embeddings), and matches it against a live catalog with one MongoDB Atlas $vectorSearch aggregation — vector similarity and a category pre-filter in a single query over heterogeneous documents — then ranks the results by score and cuts them off at a calibrated confidence threshold.

Definition of done (the runnable result a learner ends with):

1. An Atlas M0 cluster holding a products collection (~15 seeded products across ≥3 categories), each with a stored embedding and embeddingHash, behind a products_vec Atlas Vector Search index.
2. A backend (Go or TypeScript) exposing POST /recognize that, given a photo, returns { descriptor, matches:[{...product, score}], noMatch } — the spotlight pipeline end to end.
3. A mobile app (Compose, Flutter, or SwiftUI) that captures/picks a photo, calls /recognize, and shows ranked matches with their scores, the Vision descriptor + which pre-filter fired, a live threshold slider, and a first-class no-match state.
4. An integration test (per backend) proving the two outcomes against the real Atlas index: a known product photo matches above the threshold T; an out-of-catalog photo returns a clean noMatch.

How the learner SEES it run locally, for $0: start the backend against Atlas M0 with a free Gemini key, run the mobile app on an emulator/simulator, tap a bundled sample photo, and watch a ranked match list with scores appear — or a clean “no confident match” for an unstocked item. No cloud deploy is required; Cloud Run is an optional extra.

The spotlight is load-bearing: remove MongoDB Atlas Vector Search and the project’s core (similarity + metadata pre-filter in one query over a schema-volatile catalog) cannot exist. The backend language is a swappable shell; the match is the database.

2. Architecture (components and how they connect)

            ┌─────────────┐  multipart {image, category?}   ┌──────────────────────────┐
 Mobile app │ Compose /   │ ──────────────────────────────▶ │ Backend  POST /recognize │
 (camera or │ Flutter /   │ ◀────────────────────────────── │ (Go default | TypeScript)│
  gallery)  │ SwiftUI     │   { descriptor, matches[], noMatch }                        │
            └─────────────┘                                  └─────────┬────────────────┘
                                                                       │
                              1. Vision (image → typed descriptor)     │
                                ┌────────────────────────────────────▶ │
                                │  Gemini Vision  (generateContent,     │
                                │  responseSchema, category enum)       │
                                │                                       ▼
                                │  2. Embed (descriptor text → vector)  embeddingText(descriptor)
                                │  Gemini embeddings (gemini-embedding-001, outputDimensionality D)
                                │  → L2-normalize when D < 3072 ────────┐
                                │                                       ▼
                                │                            3. $vectorSearch (one aggregation)
                                │                            MongoDB Atlas:  embedding NN
                                └─────────────────────────── + category pre-filter + $meta score
                                                                       │
                                                                       ▼
                                                            4. threshold T → ranked matches | noMatch

• Mobile app never calls Gemini or Mongo directly. It only calls your backend. The Gemini key and the Mongo URI live server-side.
• Gemini does two jobs: Vision (photo → descriptor) and embeddings (text → vector). Same key, free tier.
• MongoDB Atlas does the match: nearest-neighbour over embedding plus the metadata pre-filter, in one $vectorSearch aggregation, returning each candidate’s vectorSearchScore.
• The ingest pass (run once, and again whenever a product’s content changes) embeds every product and stores the vector on its document, behind the same products_vec index the query reads.

Why descriptor-text embeddings (the road not taken)

We embed the text of a Vision descriptor for both catalog and query — not the raw image. The honest reason: it keeps the project on one free embedding model for catalog and query, yields a human-readable descriptor you can debug, and makes the match explainable. The invariant this creates: catalog and query must be embedded the same way (same model, same dimensions, same normalization), because we are comparing embed(text-of-catalog-product) against embed(text-of-Vision-descriptor-of-photo). A learner must seed products they can actually photograph (or generate matching images), because the comparison is descriptor-text vs descriptor-text — not image-pixels vs image-pixels.

3. Runnable structure (the repo the learner ends with)

Both backends share the same module layout in spirit; names differ by language. The app entrypoint composes everything: it opens one Mongo client, builds the Gemini client, registers routes + middleware, and shuts down cleanly.

Go (default)

catalens/
  go.mod                       # module github.com/you/catalens
  cmd/api/main.go              # ENTRYPOINT: load config, open Mongo client, build deps,
                               #   register routes, http.Server + graceful shutdown
  internal/config/config.go    # env: MONGODB_URI, MONGODB_DB, GEMINI_API_KEY,
                               #   GEMINI_VISION_MODEL, GEMINI_EMBED_MODEL, EMBED_DIM (D), THRESHOLD (T), PORT
  internal/catalog/store.go    # Store: Search(ctx, q) / Upsert / collection handles
  internal/gemini/gemini.go    # Vision (descriptor) + Embed (vector) + L2-normalize helper
  internal/recognize/service.go# RecognizeService: orchestrates Vision→embed→Search→threshold
  internal/recognize/handler.go# POST /recognize HTTP handler (multipart in, JSON out)
  internal/embedtext/text.go   # embeddingText(doc|descriptor) — the SHARED builder (ingest == query)
  cmd/seed/main.go             # seed ~15 products (idempotent upsert by (brand,name)); _worker_state is
                               #   created by the worker feature, not here
  cmd/ingest/main.go           # embed every product, store embedding + embeddingHash
  cmd/worker/main.go           # (feature: dynamic-embeddings) change-stream re-embed worker
  testdata/                    # bundled sample product photos (match seeded products) + a true-negative

TypeScript

catalens/
  package.json
  src/server.ts                # ENTRYPOINT: MongoClient.connect, build deps, Hono routes, serve
  src/config.ts                # same env vars as Go
  src/catalog/store.ts         # CatalogStore: search()/upsert()/collection handles
  src/gemini.ts                # vision()/embed()/l2normalize()
  src/recognize/service.ts     # orchestrates Vision→embed→search→threshold
  src/recognize/handler.ts     # POST /recognize Hono handler
  src/embedText.ts             # embeddingText() — the SHARED builder (ingest == query)
  src/seed.ts                  # seed ~15 products (idempotent upsert by (brand,name)); _worker_state is
                               #   created by the worker feature, not here
  src/ingest.ts                # embed every product, store embedding + embeddingHash
  src/worker.ts                # (feature) change-stream re-embed worker
  testdata/                    # bundled sample photos + a true-negative

Key interfaces (named explicitly — same contract, both languages)

Store / CatalogStore — the only thing the recognise service knows about persistence:

• Search(ctx, query) -> []Match where query = { queryVector: float[D], categoryHint?: string, numCandidates: int, limit: int, exact?: bool, inStockOnly?: bool }. Runs the $vectorSearch aggregation and returns ranked Match{ id, name, brand, category, attributes, price, inStock, score }.

The Store.Search seam is a named contract, not a mandatory file: the course teaches the $vectorSearch pipeline inline in the /recognize handler for clarity (one place to read the spotlight end to end). Extracting it behind a Store.Search method is an idiomatic refactor, not a missing piece.

• Upsert(ctx, product) -> id (seed + ingest).
• SetEmbedding(ctx, id, vector, hash) (ingest + worker).
• LogMiss(ctx, miss) (feature: no-match-analytics; writes to a separate search_misses collection).

Vision + Embed (the gemini package):

• Vision(ctx, imageBytes, mime) -> Descriptor — generateContent with responseMimeType:"application/json" and a responseSchema whose category is an enum of the catalog’s known categories (the one set defined in §4 “Known categories”).
• Embed(ctx, text) -> float[D] — embeds, then L2-normalizes when D < 3072 (see §4).

RecognizeService — the spotlight orchestration, language-agnostic in shape: Recognize(ctx, imageBytes, mime, categoryHint?) -> RecognizeResponse. It: (1) Vision → descriptor, (2) embeddingText(descriptor) → Embed → query vector, (3) Store.Search with the category pre-filter (falling back to no filter when the filtered result is empty — see §6 “category gap”), (4) apply threshold T, (5) on no-match optionally LogMiss. Returns the canonical response shape in §5.

4. Data model

Known categories (the enum)

Known categories = { sneakers, tea, chair } (≥3). This exact set is the Vision responseSchema category enum (§3/§5) and the only legal $vectorSearch category filter values; the ~15 seed rows must use only these values. category is the spine that binds the Vision responseSchema enum (§3/§6 step 8), the $vectorSearch category pre-filter (§5/§7), and the seed rows — define it once here so the enum, the filter, and the seed all bind to one definition.

Collection products (the catalog — heterogeneous documents)

Common fields every match relies on, plus per-category attributes:

NOTE — attributes is deliberately two shapes: a product/Match attributes is an OBJECT (per-category fields, e.g. {colour, material, sizes}); a Vision descriptor attributes is a string[] of observed values, e.g. ["leather"] (Vision cannot know per-category field names). The shared embeddingText builder normalises both to the same string (object → its values; array → as-is) — this is the comparability invariant of §2; mishandling it silently breaks the catalog-vs-query match. Typed mobile clients must therefore declare two distinct types for the two attributes (see the Descriptor block below and the §5 Match/descriptor field contract).

NOTE — wire === DB (camelCase throughout, no remap): the JSON wire keys and the MongoDB document fields use the same camelCase keys end to end (filterApplied, inStock, noMatch, visibleText, embeddingHash, imageRef, nearMisses) — there is no DTO↔column remapping; the JSON field name equals the BSON field name. A second-backend author defaulting to snake_case DB columns would break the change-stream $match (§5, keyed on name | brand | category | attributes | imageRef) and the embeddingHash idempotency compare, both of which depend on the exact field names name/brand/category/attributes/imageRef/embeddingHash.

Seed sample (one complete row per known category)

The ~15-row seed (§6 step 4) must use ONLY the known-categories enum values, and every row must carry every required field. These canonical rows — the same values the course teaches toward — pin the per-category attributes object shape so the seed is verifiable rather than only counted. Every seed row’s category MUST be a member of the known-categories enum (above) — the FK-analogue that must provably resolve.

// sneakers
{ name: "Trailblazer Low", brand: "Northpeak", category: "sneakers",
  attributes: { colour: "red", material: "leather", sizes: [7, 8, 9] }, price: 8900, inStock: true }
// tea
{ name: "Sencha Green", brand: "Kettleworks", category: "tea",
  attributes: { flavour: "green", caffeine: "medium", grams: 80 }, price: 1200, inStock: true }
// chair
{ name: "Drafting Stool", brand: "Forma", category: "chair",
  attributes: { material: "oak", colour: "natural" }, price: 14900, inStock: true }

Each row’s attributes is an OBJECT of per-category typed fields (the OBJECT shape of the two-shapes note above), and category is one of { sneakers, tea, chair }. The remaining ~12 rows follow the same per-category shapes.

Typed Descriptor (the Vision output, carried verbatim in the response)

descriptor is the central typed entity of the contract — it appears in both the match and no-match response envelopes (§5), inside search_misses (below), and is the shape the Vision responseSchema returns. It mirrors that responseSchema exactly:

Descriptor {
  brand:       string,
  category:    string,    // REQUIRED; ALWAYS one of the known-categories enum (the Vision enum), never free text
  colour:      string,
  form:        string,
  visibleText: string,
  attributes:  string[]   // free Vision tags — the string[] shape of the two-shapes note above
}

This is the shape returned by the Vision responseSchema and carried verbatim in the response descriptor field (§5) and in search_misses.descriptor (below). descriptor.category is always a known category (the Vision enum constrains it), so the $vectorSearch category pre-filter always receives a legal value.

Atlas Vector Search index products_vec (on products)

{
  "fields": [
    { "type": "vector", "path": "embedding", "numDimensions": 768, "similarity": "cosine" },
    { "type": "filter", "path": "category" },
    { "type": "filter", "path": "brand" },
    { "type": "filter", "path": "inStock" }
  ]
}

• numDimensions must equal the embedding length D you ingest with. A mismatch breaks the build/query — the single most common setup error.
• Only fields declared type:"filter" can appear in a $vectorSearch filter. inStock is declared up front so the substitutes feature works without an index change.
• brand is declared as a filter field up front for a future brand-scoped recall, even though the default Store.Search contract does not use it (no query filters by brand, and hybrid matches brand via $search text, not a vector pre-filter) — declared now so a brand-scoped query needs no index rebuild.
• Embedding-normalization invariant (load-bearing): gemini-embedding-001 returns embeddings that are only pre-normalized at the full 3072 dimensions. At any smaller outputDimensionality (e.g. 768 or 1536) the vectors carry varying magnitude that distorts cosine similarity, so you must L2-normalize every vector — catalog and query — before storing/searching, or use D = 3072. (Confirmed: https://ai.google.dev/gemini-api/docs/embeddings — manual normalization is required for non-3072 dims; gemini-embedding-2 auto-normalizes truncated dims, so pairing the model id with the normalize rule keeps a model swap correct.)

Partial-unique indexes on barcode and sku (feature: barcode)

barcode and sku are two separate partial-unique indexes, not a compound — each enforces uniqueness ONLY over documents that HAVE the field. A plain (non-partial) unique index on an optional field treats a missing field as null, so two products both lacking barcode (or sku) would collide on null and the insert would fail — with ~15 mostly-barcodeless seed rows the seed itself would fail. The partial filter makes “optional but unique” well-defined:

db.products.createIndex(
  { barcode: 1 },
  { unique: true, partialFilterExpression: { barcode: { $exists: true, $type: "string" } } }
)
db.products.createIndex(
  { sku: 1 },
  { unique: true, partialFilterExpression: { sku: { $exists: true, $type: "string" } } }
)

Uniqueness applies only to documents that have the field. barcode is the OCR/label fast-path key and sku the internal-lookup key for the barcode feature (§8).

Collection _worker_state (prerequisite for the change-stream worker)

A single document { _id: "embeddings-worker", resumeToken: <BSON resume token | null> }, created by the dynamic-embeddings worker feature, not the default seed (§8; see the seed-order note below). On each handled event the worker writes the latest resume token here; on startup it reads it back via resumeAfter/SetResumeAfter.

Resume-token read-back rule: on startup, if resumeToken is null/absent, open the change stream with NO resumeAfter (a fresh stream); only pass resumeAfter/SetResumeAfter when a non-null token is present. Passing a null/nil token to resumeAfter/SetResumeAfter is not valid for the Mongo change-stream API, so the seeded null is a valid, handled first-run state — not something fed straight into resumeAfter.

Collection search_misses (feature: no-match-analytics)

{
  "at": "ISODate",
  "descriptor": { "category": "...", "brand": "...", "colour": "...", "form": "...", "visibleText": "...", "attributes": ["..."] },
  "nearMisses": [ { "name": "...", "score": 0.62 } ],
  "threshold": 0.75
}

descriptor here is the typed Descriptor defined above (its attributes is the string[] Vision shape, category always a known-categories enum value). Separate collection so analytics writes never touch the catalog the recognise path reads. No raw image is stored (descriptor + scores only — privacy-aware default).

Migrations / seed order (prerequisites first)

1. Create products (lazily on first insert) and seed ~15 products across ≥3 categories — idempotent upsert by (brand, name). Every seed row’s category MUST be a member of the known-categories enum (above) and every required field present — see “Seed sample” above.
2. Run ingest to populate embedding + embeddingHash on every product.
3. Create the products_vec index (Atlas UI or API) with numDimensions === D.
4. (features) create the two partial-unique indexes on barcode and sku (see “Partial-unique indexes on barcode and sku” above); _worker_state is created by the dynamic-embeddings feature (§8), not the default seed; search_misses is lazily created on first miss.

There are no foreign keys (document store). The closest analogue is the rule that every seed row’s category MUST resolve to a member of the known-categories enum (above) — the FK-analogue that must provably resolve. _worker_state is the worker feature’s own prerequisite row, created by that feature (§8), not the default seed. And every product must have an embedding of length D before the products_vec index is usable — ingest is a hard prerequisite of the recognise step.

5. API & event contract (the one canonical shape)

Every step, client, and test shares exactly these shapes.

POST /recognize

• Request: multipart/form-data
  • image (file, required) — a product photo (JPEG/PNG).
  • category (string, optional) — a category hint; normally omitted (the descriptor supplies it).
• Response 200 — match:

  {
    "descriptor": { "brand": "Northpeak", "category": "sneakers", "colour": "red",
                    "form": "low-top", "visibleText": "", "attributes": ["leather"] },
    "filterApplied": "sneakers",
    "matches": [
      { "id": "…", "name": "Trailblazer Low", "brand": "Northpeak", "category": "sneakers",
        "attributes": { "colour": "red", "material": "leather" }, "price": 8900, "inStock": true,
        "score": 0.88 }
    ],
    "noMatch": false
  }

• Response 200 — no confident match: { "descriptor": {…}, "filterApplied": null, "matches": [], "noMatch": true } (an honest “I don’t know”, not an error status). The no-match branch still carries descriptor and filterApplied — same envelope as a match, only matches is empty — so the client can keep showing what Vision saw and which pre-filter ran (or that it fell back). filterApplied is typically null on a no-match because a true out-of-catalog photo reaches the threshold step only after the unfiltered fallback (§6).
• Status / error codes:
  • 200 — match or no-match (both are success).
  • 400 — image part missing or unreadable ({ "error": "image required" }). Unsupported/unreadable media types that are not JPEG/PNG fold into this 400 (we do not emit 415 — see below).
  • 502 — upstream Vision Gemini call failed after retries ({ "error": "vision unavailable" }).
  • 502 — upstream Embed Gemini call failed after retries ({ "error": "embedding unavailable" }) — parallel to vision, because the pipeline makes TWO Gemini calls.
  • 503 — Atlas $vectorSearch failed ({ "error": "search unavailable" }).
  • 500 — unexpected server error, with a FIXED body { "error": "…" }. No upstream error string is ever echoed to the client — never return err.Error() (Go) / the raw exception message (TS).
  • 415 is NOT used — content-type rejection is folded into the 400 { "error": "image required" } (“unreadable”) path so both backends assert the same outcome; there is no “optional” status whose presence differs between backends. (See §7 for the retry policy referenced by the 502 rows.)

Field contract (descriptor + Match): descriptor is the typed Descriptor of §4 (its attributes is a string[] of Vision tags; its category is always a known-categories enum value). A Match is id (string), name, brand, category (strings), attributes (object — per-category fields), price (int cents), inStock (bool), score (number in [0,1]). id is the product’s _id (§4, an ObjectId) rendered as its 24-char lowercase hex string (Go: bson.ObjectID.Hex(); TS: _id.toHexString()) — never the raw ObjectId nor the extended-JSON { "$oid": … } form. This keeps the wire id byte-identical across backends and lets search_misses/analytics joins line up. Matches are ordered best-first. score is the Atlas vectorSearchScore. filterApplied echoes which category pre-filter fired (or null if the search ran unfiltered) so the UI can show the spotlight at work. attributes is deliberately two shapes within the SAME response body: on a product/Match it is an OBJECT of typed per-category fields ({"colour":"red","material":"leather"}); on the descriptor it is a flat string[] of observed values (["leather"]) — Vision cannot know per-category field names. A reader or second-backend author parsing attributes uniformly will mis-decode one side, so typed clients must declare two distinct types (see the typed Descriptor block in §4 and the §4 two-shapes note). The shared embeddingText builder normalises both to the same string (object → its values; array → as-is); this is the comparability invariant of §2. The attribute-value ordering this builder emits MUST be deterministic and identical across Go and TS (the catalog-vs-query match and the embeddingHash guard depend on byte-identical output): pick ONE ordering convention for both backends — sort-by-key or a fixed per-category key list — and beware that Go map range order is undefined while TS Object.values() follows insertion order; see §7 for the chosen convention and the “run embeddingText 10× and assert identical output” acceptance test.

Score semantics (load-bearing). For cosine similarity Atlas maps the raw cosine [-1,1] into [0,1] as (1 + cosine) / 2. So an unrelated (orthogonal) photo’s nearest stranger still scores ~0.5, not 0 — 0.5 is the “no real similarity” floor, and real matches for a clean photo sit well above it. A naive low threshold like T=0.3 is therefore meaningless; calibrate T above the ~0.5 floor. (Confirmed: MongoDB normalizes cosine as (1+cosine)/2.)

Feature endpoints (off by default — see §8)

• GET /products/{id}/substitutes → { matches:[{...product, score}], noMatch } (in-stock only, lower threshold). Status / error codes: 400 { "error": "invalid id" } for an unparseable ObjectID; 404 { "error": "product not found" } when no product has that id; otherwise 200 { "matches": [...], "noMatch": <bool> }. An empty result is { "matches": [], "noMatch": true } at 200 (never 404 for “no substitutes found” — 404 is reserved for an unknown product id). The lower substitute threshold T_sub is applied server-side and is pinned to T as T_sub = T - 0.1 so BOTH backends pick the identical cut (recall over precision).
• POST /recognize/shelf → [ { box, matches:[{...product, score}], noMatch } ] (multi-shelf fan-out). Status: 200. Each element’s box is { x: number, y: number, w: number, h: number } as NORMALISED [0,1] floats relative to image width/height, origin top-left (resolution-independent, so any client can scale it to its rendered image). An empty image / zero detections returns [] (200), not an error. box MAY be null ONLY if a detect path returns no location; otherwise every detected item carries a box. Whether shelf runs array-responseSchema or detect-then-recognise (§8), both paths return the same box shape; matches/noMatch per element follow the POST /recognize contract above.
• GET /analytics/top-misses?since=<ISO> → [ { category, brand, requests, avgNearMiss, lastRequested } ]. since is OPTIONAL (?since=<ISO> — when omitted, all-time); when present it filters on at >= since. An unparseable date returns 400 { "error": "invalid since" }. Pagination: ?limit=<int, default 20, max 100> (the aggregation’s $limit). Auth: intentionally open in the $0 local build (no auth) — add auth before any non-local deploy. Row types (derived from search_misses, §4), grouped by descriptor.category + descriptor.brand: category: string, brand: string, requests: int (count of misses in the group), avgNearMiss: number|null in [0,1] (mean of each miss’s top nearMisses[0].score; null when a group has no nearMisses), lastRequested: string (ISO-8601, the max at in the group).

barcode fast-path — how it extends POST /recognize (BOTH backends). §8 says each feature extends, never rewrites the spec, so the barcode feature’s exact-lookup branch is pinned here rather than left to silently change the /recognize contract. The feature adds ONE optional request field and an exact-hit branch that reuses the normal envelope — it does not add a new status code or a new response shape:

• Request: an optional code field (string) added to the multipart/form-data form (alongside image). When present, the backend tries an exact findOne on the two partial-unique indexes (barcode/sku, §4) BEFORE the vector pipeline.
• Exact hit: returns the NORMAL §5 match envelope — descriptor (still produced from the photo so the UI panel stays populated), matches: [{ ...product, score: 1.0 }] (a findOne hit has no vectorSearchScore, so score is pinned to 1.0), filterApplied: "barcode" (a distinct sentinel — NOT a category value — so the UI can show the fast-path fired vs a category pre-filter), noMatch: false.
• No code / no exact hit: falls through to the unchanged vector pipeline above — same envelope, score is the real vectorSearchScore, filterApplied is the category sentinel or null, noMatch per the threshold.
• Parity: both backends add the identical code field, the same score: 1.0 and filterApplied: "barcode" sentinel on an exact hit, and the same fall-through on a miss — the envelope stays byte-identical (§7).

Wire/event message — change-stream event (feature: dynamic-embeddings)

The worker consumes MongoDB change-stream events on products opened with fullDocument:"updateLookup" and a $match pipeline that only lets content edits through:

operationType ∈ {insert, update, replace}
AND ( insert | replace  OR  updateDescription.updatedFields has one of:
      name | brand | category | attributes | imageRef )

Per event: text = embeddingText(fullDocument); hash = sha256(text); skip if hash == doc.embeddingHash (idempotent); else Embed(text) → SetEmbedding(id, vector, hash). The write-back touches only embedding/embeddingHash, which the $match excludes — so it never re-triggers the worker.

Resume-token advance ordering (at-least-once re-embed, identical across Go/TS): persist resumeToken to _worker_state ONLY after SetEmbedding succeeds (a skipped-by-hash event is “handled” and may advance the token immediately, since no re-embed is owed). On an Embed() failure, log and do NOT advance the token — the event re-delivers on restart, preserving the re-embed guarantee. (A failed Embed() must never count as “handled” for token purposes; advancing it would silently drop the re-embed and leave a stale vector with no retry. The alternative — advance with an explicit dead-letter — is out of scope for the default build; this spec pins the do-not-advance behaviour.) On startup read the token back and pass it as resumeAfter ONLY when it is non-null (a null/absent token → open a fresh stream with no resumeAfter; see the §4 resume-token read-back rule).

6. Build order (dependency-ordered; each step’s prerequisites already exist)

1. Prerequisites & local tooling — Go toolchain (or Node), mongosh, curl/base64, a sample image.
2. Atlas M0 — cluster + MONGODB_URI, MONGODB_DB (Vector Search is Atlas-only). MONGODB_URI is the Atlas SRV string mongodb+srv://<dbuser>:<url-encoded-password>@<cluster>.mongodb.net/?retryWrites=true&w=majority — created via Database Access (a DB user with a password) PLUS a Network Access allowlist entry (your IP, or 0.0.0.0/0 for dev); Atlas refuses the connection until BOTH exist. URL-encode any special characters in the password (the two most common Atlas blockers are this credential-less / unencoded DSN and the numDimensions mismatch §4 already calls out).
3. Gemini key — confirm Vision + embeddings respond (portable base64/curl REST smoke test only; name sample.jpg). This is a portable REST smoke test, not the from-code SDK wiring — that arrives in step 5b.
4. Model the catalog + seed — products (~15, ≥3 categories), idempotent upsert by (brand, name); every row uses a known-categories enum value and carries every required field (see §4 “Seed sample”). The _worker_state doc is NOT seeded here — it is created by the dynamic-embeddings feature (§8).
5. Shared embeddingText builder + normalize helper — defined once, reused by ingest, query, worker. 5b. Build the Gemini client + Embed() from code (per backend) — choose D = EMBED_DIM now (e.g. 768) — this same value is the index numDimensions (step 7) and the queryVector length (step 10); pick once, reuse everywhere. Then: SDK install / package add, the full import block, genai.NewClient (Go) / client construction, EmbedContent with outputDimensionality = D, and the normalize wrapper that L2-normalizes when D < 3072. This is the from-code SDK wiring ingest needs — introduced BEFORE its first consumer (step 6 ingest calls Embed() from code; step 3’s portable REST smoke test is not enough).
6. Ingest — embed each product with the Embed() built in 5b (fetch with find(), iterate by doc._id), store embedding + embeddingHash.
7. Create products_vec index — numDimensions === D; category/brand/inStock as filters; cosine.
8. Design the recognise pipeline + Vision responseSchema — adds Vision() to the same gemini package built in 5b; category as an enum of the known categories (closes the Vision-vs-catalog gap; needs the seeded catalog for the category enum), descriptor → embeddable text.
9. Scaffold the API + /recognize skeleton / route entrypoint (per backend) — composes the already-built gemini (5b/8) + store; the entrypoint that opens the Mongo client, registers routes + middleware, and shuts down cleanly.
10. ★ Recognise end to end (per backend) — Vision → embed (normalized) → $vectorSearch (category pre-filter, fall back to unfiltered when the filtered result is empty) → ranked matches + scores.
11. Threshold or no-match — apply T; clean noMatch.
12. Calibrate T — score known/unknown photos; the ~0.5 cosine floor; no-match UX.
13. Frontend (per platform) — capture/pick → /recognize → ranked matches + scores + descriptor/filter panel + live threshold slider + nearest-below-threshold no-match UX.
14. Integration tests (per backend) — known photo matches above T; unknown → noMatch.
15. Optional deploy — Cloud Run, free-tier eligible.
16. Feature modules (off by default) — §8.

Each step depends only on earlier ones: the Gemini client + Embed() (5b) needs only the SDK and the shared builder (5); ingest (6) consumes the existing Embed() (5b) plus the seed (4); the index (7) needs ingest (6); the responseSchema (8) adds Vision() to the gemini package (5b) and needs the seeded catalog (4) for the category enum; the scaffold (9) composes the already-built gemini (5b/8) + store; recognise (10) needs the index (7), the responseSchema (8), and the scaffold (9); the worker feature creates its own _worker_state doc (§8) and needs the shared builder (5) and Embed() (5b).

7. Backends — Go (default) + TypeScript, same contract

Parity points (both must hold):

• Same response shape (§5) byte-for-byte in field names and types; matches best-first; score in [0,1]; filterApplied echoed; noMatch is a 200.
• Same filterApplied null encoding (idiomatic Go difference required). On the unfiltered fallback / no-match branch filterApplied MUST serialize to JSON null, not "" — §5 (the no-match envelope) requires null. Go: the response field is *string with the filterApplied json tag — nil on the unfiltered fallback → JSON null, &category when the pre-filter fired (FilterApplied *string tagged json:"filterApplied"). TS keeps string | null (null on fallback). Both emit the category string only when the pre-filter fired. (A plain Go string field marshals "" to "filterApplied":"" — a byte-for-byte envelope divergence that also makes typed clients render an empty “Pre-filtered to: ” instead of “Searched all categories”. Verified on Go 1.26 encoding/json: a nil *string marshals to {"filterApplied":null}, &"sneakers" to {"filterApplied":"sneakers"}, a plain string "" to {"filterApplied":""}.)
• Same $vectorSearch stage: index:"products_vec", path:"embedding", queryVector length D, numCandidates (~20× limit, must be ≥ limit), limit, optional filter { category:{$eq:hint} }, optional exact:true (ENN baseline). $project adds score:{$meta:"vectorSearchScore"}.
• Same embedding rule: same model + same outputDimensionality D + L2-normalize when D < 3072 for both catalog and query.
• Same deterministic embeddingText attribute ordering (Go ↔ TS byte-identical). The attribute-value ordering inside embeddingText MUST be deterministic and identical across both backends — the catalog-vs-query string match and the embeddingHash idempotency guard both depend on byte-identical output. The hazard is Go-side: decoding attributes into a map[...] makes ordering undefined (Go map range order is randomized — verified on Go 1.26: valuesOf over a 3-key map produced 3 distinct orderings across in-process calls), so the hash flips run-to-run (re-embeds every pass) and the catalog == query unit test fails. The canonical course path (bson.M, whose nested sub-docs decode as ordered bson.D) is already deterministic, so the risk is specifically map-typed modeling. Parity caveat: TS Object.values() iterates insertion order, NOT sorted-key order — so to keep Go == TS you must pick ONE convention for BOTH backends: either (a) both sort attribute keys (Go: keys := slices.Sorted(maps.Keys(m)); for _, k := range keys { vals = append(vals, fmt.Sprint(m[k])) }, and build the TS descriptor array sorted-by-key), OR (b) both use a fixed per-category key list. State the chosen convention, and add the acceptance test: run embeddingText(doc) 10× and assert identical output (catches the map-order regression).
• Same category-gap handling: Vision category constrained to the known enum; on an empty filtered result, retry the search unfiltered before declaring no-match.
• Same fire-and-forget miss logging (feature) into a separate search_misses collection; no raw image.
• Same re-embed worker contract (feature: dynamic-embeddings — both halves are required and inseparable; TS is the parity reference and stays unchanged):
  • Half A — full-document decode. The worker decodes the FULL product document from the change event’s fullDocument (name/brand/category/attributes/imageRef/embeddingHash), because embeddingText(doc) (§4) builds its string from brand + name + category + attribute values. A struct that decodes only _id + embeddingHash silently produces a constant embedding text for every product — so the hash is constant, the idempotency skip never fires (re-embeds on every event), and every vector is identically wrong. Decode all the content fields embeddingText reads.
  • Half B — $match $or content guard. The worker $match MUST carry the content-field $or guard (one of the two required loop-breaking guards, §5): operationType ∈ {insert, update, replace} AND (insert | replace OR updateDescription.updatedFields has one of name | brand | category | attributes | imageRef). Without it, every embedding-only write-back (an update touching only embedding/embeddingHash) re-matches and the worker recomputes the hash on its own writes — the exact loop the $match exists to prevent.
• Same descriptor field names (Go camelCase json tags required). The Go Descriptor (the typed §4 entity, carried verbatim in the §5 response descriptor) MUST carry explicit camelCase json tags, because Go exported fields WITHOUT tags marshal as PascalCase (Category → "Category", VisibleText → "VisibleText") — silently diverging from the TS descriptor and the §5 contract, so typed clients decoding descriptor.category / descriptor.visibleText get nulls (violates the byte-for-byte field-name rule). The camelCase json tags are the idiomatic Go difference (TS gets the keys for free):

  type Descriptor struct {
      Brand       string   `json:"brand"`
      Category    string   `json:"category"`
      Colour      string   `json:"colour"`
      Form        string   `json:"form"`
      VisibleText string   `json:"visibleText"`
      Attributes  []string `json:"attributes"`
  }

• Same Gemini resilience policy (referenced by the §5 502 rows): retry each Gemini call (Vision AND Embed) up to 3 times with exponential backoff (250ms, 500ms, 1s) on HTTP 429/5xx and context-deadline; do NOT retry 400/403; after exhaustion return 502 (vision unavailable / embedding unavailable). An Atlas $vectorSearch failure surfaces as 503 search unavailable. No upstream error string is ever echoed — 500 carries a fixed body, never err.Error() / the raw exception. Both backends use the identical retry counts and backoff so the failure contract is byte-identical.

Pinned versions (API shapes in this spec are verified against these — SDK version drift on these exact packages is this project’s #1 recurring blocker; pin them in go.mod / package.json so a later go get / npm i cannot land on a major bump where the embed/config API changes):

• Go: go.mongodb.org/mongo-driver/v2 v2.7.0, google.golang.org/genai v1.62.0.
• TS: mongodb@^7 (7.3.0), @google/genai@^2 (2.10.0).

Go specifics (verified):

• Module go.mongodb.org/mongo-driver/v2; import all three sub-packages used: go.mongodb.org/mongo-driver/v2/mongo, .../v2/mongo/options, .../v2/bson. One go get go.mongodb.org/mongo-driver/v2/mongo pulls the whole module; the import lines must list the sub-packages. bson.ObjectID is the v2 type name (was primitive.ObjectID in v1). mongo.Connect(options.Client().ApplyURI(uri)) is the v2 signature (no context arg).
• Gemini Go SDK google.golang.org/genai: genai.NewClient(ctx, &genai.ClientConfig{APIKey: key, Backend: genai.BackendGeminiAPI}); Vision via client.Models.GenerateContent(ctx, model, contents, &genai.GenerateContentConfig{ResponseMIMEType:"application/json", ResponseSchema: …}) with &genai.Blob{Data: imageBytes, MIMEType:"image/jpeg"} as an inline Part; embeddings via client.Models.EmbedContent(ctx, model, contents, &genai.EmbedContentConfig{OutputDimensionality: &d}).
• Embed dimensionality is *int32. In google.golang.org/genai v1.62.0 EmbedContentConfig.OutputDimensionality is *int32, so &d only compiles if d is int32 — a learner reading EMBED_DIM (D) as the Go-default int and writing d := 768; &d gets a compile error (cannot use &d (value of type *int) as *int32 value in struct literal). Convert EMBED_DIM to int32 before taking its address: d := int32(EMBED_DIM); cfg := &genai.EmbedContentConfig{OutputDimensionality: &d}. (TS unchanged: outputDimensionality: D as a plain number — no such trap.)

TypeScript specifics:

• Official mongodb driver; new MongoClient(uri) + await client.connect(); collection.aggregate(pipeline) .toArray(); collection.watch(pipeline, { fullDocument:"updateLookup", resumeAfter }) (async-iterable).
• Gemini via the REST API (x-goog-api-key, v1beta, generateContent/embedContent) or @google/genai.

Neither backend hard-codes a model id: read GEMINI_VISION_MODEL / GEMINI_EMBED_MODEL from config and link the official model list (https://ai.google.dev/gemini-api/docs/models). The only current free-tier embedding model needing normalization is gemini-embedding-001, so the normalize rule is the safe default.

8. Optional feature modules (off by default; each extends, never rewrites, the spec)

• hybrid — fuse $vectorSearch with Atlas Search $search over brand/name/visibleText via RRF (or the Atlas $rankFusion stage where available); one confidence cut on the fused score; no-match path preserved.
• substitutes — GET /products/{id}/substitutes: the product’s own embedding as the query vector, $vectorSearch with filter:{inStock:true} excluding the product, a lower threshold T_sub = T - 0.1 applied server-side (recall over precision; both backends use this identical relation). Uses the inStock filter field already in the index. Response envelope and status/error codes are the §5 substitutes contract (400 invalid id / 404 product not found / 200 { matches, noMatch }; an empty result is { matches: [], noMatch: true } at 200, never 404).
• multi-shelf — POST /recognize/shelf: Vision returns an array of items (array responseSchema) or detect-then-recognise per crop; reuse the per-item pipeline; return [{box, matches[], noMatch}] at 200. Each box is { x: number, y: number, w: number, h: number }, NORMALISED [0,1] floats relative to image width/height, origin top-left (resolution-independent — clients scale to their rendered image), matching the §5 shelf field set exactly. Zero detections / empty image returns [] (200); box MAY be null only when a detect path yields no location, otherwise every item carries a box. Both the array-responseSchema and detect-then-recognise paths return the same box shape.
• barcode — exact barcode/sku lookup (the two partial-unique indexes, §4) before the vector fallback; cheap/deterministic path first, vector path only on a miss.
• dynamic-embeddings — the change-stream re-embed worker (§5 event shape). Feature setup creates its own _worker_state doc — idempotent upsert of { _id: "embeddings-worker", resumeToken: null } (§4); it is NOT part of the default seed. Prerequisite: the shared embeddingText builder. On startup, honour the §4 resume-token read-back rule (no resumeAfter/SetResumeAfter when the token is null/absent). Go (mongo.ChangeStream + SetFullDocument(options.UpdateLookup) + SetResumeAfter) and TS (watch(...)) parity.
• performance — make numCandidates configurable; sweep it against an exact:true ENN baseline (ENN is the ground truth for catalogs under ~10k docs) and pick the smallest value within recall tolerance; numCandidates ≥ limit, ~20× limit as the documented starting point.
• no-match-analytics — fire-and-forget LogMiss into search_misses on the no-match branch (Go/TS parity), plus a common GET /analytics/top-misses aggregation ranking unmet demand. The query contract (since optional → at >= since; 400 invalid since; limit default 20 / max 100; open auth in the $0 build; rows grouped by descriptor.category + descriptor.brand) is pinned in §5 — both backends emit the identical row shape.

9. Free-to-complete ($0)

Confirm current free-tier limits on the official docs; nothing in the default path requires a paid service.