Each card is a real interview-grade problem this system had to answer.
The blue tag is the technical pattern; the file path lets you verify the implementation.
01
At-least-once safe ingestion under Pub/Sub redelivery and crashes
Gmail Pub/Sub may redeliver the same historyId on transient errors. Workers can crash between any two side-effects. The pipeline must never drop an email and never write a row twice.
Two-layer dedup. (1) DB unique constraint uq_email_content_message(mailbox_id, message_id) — Gmail's message id is the natural idempotency key. (2) Explicit existsByGmailMailboxIdAndMessageId check before the API fetch, so a redelivery doesn't even spend a quota call. Every stage progresses processed_status after the durable write, never before — so a crash mid-stage replays from the last persisted state.
Event-drivenIdempotencyState machineAt-least-onceUnique constraintsGCP Pub/Sub
→ ingester/.../GmailMessageProcessingService.java, EmailContent.java
02
Bursty Gmail events coalesced into one sync per mailbox
Gmail can fire many Pub/Sub events per mailbox in seconds. Naively syncing each one wastes API quota and risks racing into duplicate work.
Per-mailbox ReentrantLock + high-watermark coalescing. Events merge(eventHistoryId, Math::max) into a per-mailbox watermark; only the first event acquires the lock and runs a paginated sync that catches up to the latest watermark. Subsequent events tryLock(), see it's held, and exit immediately. Bounded by MAX_SYNC_ITERATIONS = 50 to prevent unbounded catch-up loops.
ReentrantLockHigh-watermarkCoalescingConcurrentHashMapQuota efficiency
→ ingester/.../GmailMessageSyncService.java
03
One slow stage can't starve the others
Sanitize is microseconds; the LLM normalization call is seconds; Bedrock embedding is hundreds of milliseconds. A single queue means the slowest stage back-pressures the entire pipeline.
RabbitMQ topic exchange with 4 isolated stage queues (sanitization · normalization · embedding · clustering), each declared with x-dead-letter-exchange + x-dead-letter-routing-key at bean construction — every queue gets a sibling .dlq. Each listener factory carries its own retry interceptor (maxAttempts=3, exponential 1s → 2s → 4s capped at 10s) with RejectAndDontRequeueRecoverer, prefetch 10, concurrent consumers 4–8.
RabbitMQTopic exchangeDLQ per stageBackpressurePrefetch tuning
→ processor/.../config/RabbitMQConfig.java
04
Classify failures so retries don't waste budget
Bedrock 5xx, 429 throttling, and socket I/O errors should retry — they pass. A 400 malformed request or a revoked refresh token will never succeed — retrying just delays the DLQ and burns rate limit.
Exception classification at the HTTP boundary. HttpServerErrorException, HttpClientErrorException.TooManyRequests, and ResourceAccessException wrap to RetryableAIException → handled by @Retry(name="aiRetry") (3 attempts, 2s base, 2× backoff, 0.5 jitter). All other 4xx wrap to IllegalStateException → fails fast to DLQ. The Pub/Sub subscriber detects invalid_grant and marks the mailbox DISCONNECTED instead of nacking forever.
Resilience4jRetry classificationExponential backoffJitterFault tolerance
→ processor/.../BedrockModelProvider.java, ingester/.../GmailPubSubSubscriber.java
05
Dual-mode clustering: hot path stays sub-second, cold path discovers structure
Every new email needs a cluster now for live labelling — but you also need the system to discover new clusters as the inbox evolves. Doing DBSCAN per email is too slow; doing only periodic batch leaves new emails uncategorised for hours.
Two cooperating clusterers. Incremental: cosine vs existing centroids, assign if similarity ≥ configurable threshold, else mark CLUSTER_ASSIGNMENT_COMPLETED and defer to batch. Batch: full @Transactional Smile DBSCAN over all embeddings — flushes old assignments and re-creates them atomically. Coordinated by BatchClusteringLock.isActive(): incremental skips while batch runs, so a re-cluster doesn't race with hot-path writes. Each assignment records cluster_assignment_type = INCREMENTAL or BATCH for downstream reasoning.
Hot/cold pathDBSCANCosine similarity@TransactionalAtomic re-clusterSmile
→ processor/.../EmailClusteringService.java, taxonomy-engine/.../BatchClusteringService.java
06
Multi-replica safety on a destructive batch job
Batch re-clustering deletes every cluster and assignment for a mailbox, then re-creates them inside one transaction. Two replicas running simultaneously could orphan rows or corrupt the centroid set. A worker that crashes mid-job must not leave the lock held forever.
Redis setIfAbsent + 1-hour TTL. Atomic acquire; second replica fails fast with no spin. TTL guarantees release on crash. Plus belt-and-braces: an ApplicationRunner sweeps stale keys matching <prefix>:mailbox:* on startup; @PreDestroy releases on graceful shutdown.
RedisDistributed lockSETNXTTLLifecycle hooksMulti-replica
→ taxonomy-engine/.../BatchClusteringLock.java, BatchClusteringLockCleanerConfig.java
07
Sub-10ms ANN search via Postgres, not a separate vector DB
Each new email needs its nearest cluster centroid in real time. Naive cosine over an entire table is O(n·d). A separate vector DB (Pinecone, Weaviate) means two-phase commits and a second operational surface.
Postgres + pgvector with HNSW + cosine_ops. Partial index WHERE embedding IS NOT NULL keeps the index lean during the gap between insert and embed. Hibernate 6.5 maps Java float[] directly via @JdbcTypeCode(SqlTypes.VECTOR) + @Array(length=1024) — no custom converter, no dialect hack. One transaction commits the embedding, the cluster assignment, and the status flip together.
pgvectorHNSWPartial indexANNHibernate 6.5SqlTypes.VECTOR
→ db-migrations/.../V1.005__create_email_enrichment.sql, persistence-lib/.../EmailEnrichment.java
08
LLM that justifies its answers, not just emits them
Naive prompts ("classify this email") invent urgency from words like verify, alert, action. The LLM picks a label first and rationalises later. Output drifts and is non-deterministic.
Reason-before-rate JSON schema: the model writes importance_reason before importance, category_reason before category. Override rules baked into the prompt (all bank transaction alerts → LOW UPDATES; all OTPs → LOW UPDATES). Job-hunt persona override promotes named recruiter outreach to HIGH. 8 worked examples cover real edge cases (Axis Bank UPI, LinkedIn InMail vs Job Alerts, IRCTC booking, ATS receipts). Defensive parsing: regex-extracts the JSON block even if the model adds prose; READ_UNKNOWN_ENUM_VALUES_AS_NULL falls back to LOW / PRIMARY on garbage. Temperature 0.0, top-p 1.0 — deterministic. Bedrock prompt capped at MAX_PROMPT_CHARS = 24_000.
Prompt engineeringStructured outputDefensive parsingDeterminismOverride rulesBedrock Converse
→ processor/.../NormalizationPromptHelper.java
09
Email sanitization as a pluggable pipeline, not a god method
Real email HTML is hostile: tracking pixels, hidden divs, fancy quotes, zero-width chars, table-shaped layouts, CID inline images. A single regex pass either misses cases or becomes unreadable.
Annotation-driven pipeline registry. Each step is a @SanitizationStep(order=N) Spring bean; ContentSanitizationPipelineRegistry discovers and sorts them at startup — adding a new step is one new class. Steps include: jsoup HTML→text (preserves block boundaries as newlines, list items as - , table cells as | , appends (href) when anchor text ≠ url); inline-reference remover; CharacterNormalizer with 30+ curly-quote / em-dash / arrow / ellipsis / TM mappings, six regex passes for Unicode spaces / invisible chars / pipe-only rows / multi-space / trailing-space / excessive-newline collapse. Skips parsing entirely when no HTML hints are present.
Plugin architectureSpring annotationsjsoupRegex pipelineUnicode normalization
→ processor/.../sanitization/pipeline/
10
Hierarchy from cosine: reuse → merge → create
Each batch re-cluster invents fresh cluster IDs but the user already has labels in Gmail. Naively creating a new label per cluster floods their sidebar and the system drifts further with every run.
Three-stage label decision. (1) Sort cluster members by cosine-to-centroid, take top 8, send to LLM with the existing label-name pool — model is biased toward suggesting an existing name. (2) Case-insensitive string match → reuse. (3) Otherwise embed the suggestion (Titan v2) and find nearest existing label by cosine; if ≥ MERGE_THRESHOLD = 0.80, merge into it; else create a new Label with the suggestion's embedding stored as reference_embedding. The pool grows monotonically across runs.
Embedding dedupCosine thresholdRepresentative samplingSidebar hygiene
→ taxonomy-engine/.../ClusterLabelingService.java
11
Same code in prod and on a laptop with zero cloud
Bedrock costs money and needs IAM. Local dev and CI shouldn't burn either. Hard-coding a provider makes the test loop painful and the architecture vendor-locked.
ModelProvider interface with Bedrock and Ollama implementations. Selected by config (embedding-provider-name, llm-provider-name). Same call shape; Titan v2-specific knobs (dimensions, normalize) only set when model id matches. Both Bedrock and the local Ollama embedding return 1024-d so swapping providers doesn't invalidate the vector index. Same trick applied to EmailStorageProvider — Local for dev, S3 for prod, one interface.
AWS BedrockOllamaDependency inversionSpring configS3 / Local storage
→ processor/.../model/factory/, persistence-lib/.../storage/
12
One schema across three services, zero drift
JPA entities are needed by ingester, processor, and taxonomy-engine. Copy-pasting guarantees silent drift; a data-access microservice adds a network hop per query.
Shared persistence-lib published as a private Maven package (GitHub Packages). Pinned versions force conscious upgrades. The only cross-service contract for data shape lives in one place. EmailStorageProvider and 7 Flyway migrations ride along in the same library. Hibernate uses ddl-auto: validate in every service — schema mismatches fail at startup, not at runtime.
Maven multi-moduleJPA / HibernatePrivate package registrySchema-as-codeFlyway
→ persistence-lib/.../entity/, db-migrations/.../V1.001 … V1.007