Systems Resume

Ankit Raj.

Builds systems that ship, survive, and scale.

6+ years building distributed systems across Procurement, Receivables & CRM.

Senior Software Engineer

Hyderabad/Bengaluru, India

Portrait of Ankit Raj
Java Spring Boot Webflux JPA / Hibernate jOOQ Flyway REST GraphQL OAuth2 / JWT JUnit Mockito Maven Gradle Git MySQL Postgres pgVector Redis RabbitMQ Ollama SQS SNS EventBridge Bedrock EC2 S3 Route 53 SES CloudFront Docker Linux Jenkins Argo CD Grafana Prometheus Microservices Event-driven Reactive / Async Concurrency Resilience patterns Distributed systems RAG Vector Embeddings Semantic Search Clustering Cosine Similarity Prompt Engineering

01 — About

The short version.

Backend engineer, six years in, across Jaggaer, LeadSquared and HighRadius. Most of my work has been on systems that other teams depend on, and that changes the bar. It is not about whether the happy path works. It is about what happens when something upstream breaks.

I like the parts nobody demos. What happens when the same message arrives twice. What a customer sees when a background job fails at two in the morning. I started as a product analyst before moving into engineering, and I still ask what a feature is for before I ask how to build it.

years of experience in building production-grade systems

Philosophy

i.

Don't romanticize code.

The real job is breaking down problems, spotting patterns, and delivering the right solution, code or otherwise.

ii.

Readable code is non-negotiable.

Intent-revealing code, structured design, and documentation are the backbone of fast, efficient team collaboration.

iii.

Engineering is a team sport.

Clear articulation, communicating, negotiating, and delegation will always outperform individual heroics.

02 — Experience

What I've shipped.

If you want the shape of these systems rather than the summary, the diagrams and the reasoning are here.

Senior Software Engineer

Jaggaer

Aug 2025 — Present Hyderabad

Building Catelyze, the platform layer that lets several acquired procurement products behave like one. Seven business modules now talk to Catelyze instead of to each other.

Event-Driven Data Sync

The pipeline that moves business objects between modules. An application layer over RabbitMQ, and the only service in the platform with no database of its own.

  • Inbound is one exchange with a single queue bound to it. Outbound is one exchange per business object, with routing keys taken from tenant configuration. The dumb side and the rich side, on purpose.
  • Every exchange, queue, binding and routing key is declared at application startup, so onboarding a new business object is a configuration change and not an infrastructure ticket.
  • Consume, look up config, gate, transform, publish, and only then acknowledge. At-least-once, with redelivery on nack.
  • The pipeline holds no business keys. The consuming modules dedupe, because the business meaning belongs to them.
  • Peaks above 10K+ events/sec at business hours, measured at inbound.

Configuration Console

The single source of truth for tenant configuration, replacing settings that used to live twice, once inside a product and once in that product's own platform layer.

  • Led the low-level design and built a large share of the APIs.
  • Everyone pulls and caches, nothing is pushed. The cache sits in the shared Redis with a one hour TTL, plus a force-invalidate for when an hour is too long.
  • Mostly GraphQL inside the platform, REST where the outside world dictates it.
  • Lets non-engineering teams change behaviour without a deployment.

Third-Party Integration & On-Call

One service per outside vendor, so an external API can never reach into core business logic.

  • Carbon data, supplier account verification and e-invoicing networks, each isolated behind its own service and its own database.
  • Resilience4j retry, circuit breaker and rate limiter on every outbound call.
  • I take on-call, which is where you find out what a design is really worth. Found the root causes of recurring legacy failures and built guardrails into the new platform so the same failure cannot happen twice.
  • Across 100+ migrated tenants, production incidents went from 30+ a month to 2-3 a month.

Senior Software Engineer

LeadSquared

Mar 2023 — Jul 2025 Bengaluru

Built the communication side of Service CRM, a ticketing and helpdesk product. Notifications going out, support mail coming in, and the AI that helps agents answer.

Notification Platform

Designed and built end to end. Every other module in the product publishes into it over Redis Streams, used here as the message bus and not as a cache.

  • Resolves templates through per-channel factories and dispatches to the channel service by configuration. Email, SMS and WhatsApp.
  • The event carries everything the notification needs. The service never calls back to the publisher for missing data, it fails the request. That is what keeps the cost flat at volume.
  • Persist, then acknowledge, then process. The database is the record, the stream is only a handoff.
  • Peaks above 10K+ notifications/sec.

Inbound Mail & the AI Layer

The business rules for email. Which ticket does this mail belong to, should it reopen one, and what can the agent be helped with.

  • Two spam layers on purpose. The provider's own score first, because it costs nothing, then a cheap AI pass over the content for whatever got through.
  • User-triggered retrieval over the tenant's own documents and their past resolved cases, on AWS Bedrock. Suggestions are shown to the agent and then gone. Nothing is stored as a draft.
  • Thread level vendor affinity, so a reply leaves through the provider the conversation arrived on.
  • Attaches every inbound mail to the right ticket and applies the reopen rules.

The Provider Layer

The only door to the outside. Three email providers, three completely different ways of receiving mail, one internal model.

  • A webhook from one, a queue from another, a push subscription from the third. Normalised so nothing downstream knows which provider a mail came from.
  • Persists the mail to the database and the raw body to object storage before acknowledging the provider, so nothing is lost at the boundary.
  • Dedupes on the email Message-ID. The key comes from the domain, not from us.
  • Peaks above 1000+ emails/sec on large payloads.
  • The dual write between the database and object storage was never solved. I know where it bites and what I would do about it.

Software Engineer & Product Analyst

HighRadius

May 2019 — Feb 2023 Bhubaneswar

Cash reconciliation and invoice presentment. Accounts receivable software, where being right about money is the product. I started here on the product side and moved into engineering.

Cash Reconciliation

Payment and remittance file parsers on a scheduled clearance pipeline, and the rules that turn those files into settled invoices.

  • Up to 100GB/day of payment and remittance files, parsed in parallel on an ExecutorService pool with a fixed core size and a maximum.
  • Reconciled payment, remittance and invoice data into settled invoices, and extended the settlement rule set inside an existing Strategy-pattern framework.
  • Bank file formats are where theory meets reality. Most of the work was in what the files actually contain, not in what the spec says they contain.

Invoice Presentment

Replaced the legacy SOAP and RPC integration between the invoicing and payment systems with REST APIs.

  • Found and fixed Level 3 card payments that were silently downgrading to Level 2. A quiet classification bug like that costs the customer on every single transaction, and nothing in the system was complaining about it.
  • Built the configuration console for post-payment correspondence.
  • This is where I learned to read a failure properly instead of just retrying it.

Product Ownership

Before engineering I was a product analyst here, and the habit stuck.

  • Owned an internal delivery-tracking product from problem discovery to a working version, after the implementation team grew from under a hundred people to over fifteen hundred.
  • Replaced spreadsheet based tracking for 1000+ internal users.
  • Led the stakeholder conversations, shaped the requirements, and decided what shipped first.

03 — Selected Work

Featured projects.

Inbox Intelligence

Personal project · In progress

Open case study

An end-to-end email intelligence pipeline that ingests, processes, and understands emails, grouping and labelling them by meaning, not keywords. Built to replace static inbox rules with adaptive, AI-driven organisation.

Ingester

Gmail Pipeline & Data Extraction

  • Idempotent ingestion via Gmail's history-id watermark and deterministic content hashes
  • HTML/markup sanitisation, strips noise, extracts meaningful text
  • Persists cleaned emails + metadata for downstream embedding

Processor

Embeddings, Similarity & Clustering

  • Ollama (nomic-embed-text) for on-device embedding, so no data leaves the machine
  • pgVector storage + cosine similarity search
  • DBSCAN clustering, automatic group discovery, no predefined categories

AI Taxonomy Engine

Labelling, Feedback & Future

  • LLM-powered labelling from representative cluster samples
  • Adaptive labelling, biased toward reusing existing labels over creating new ones
  • Deterministic guard-rails. Cosine-similarity thresholds decide whether to merge, nest, or drop a suggested label
NextLeap

Product Manager Fellowship

NextLeap · 2023

Certificate

Product skills complement engineering capabilities, hence this certification. Solved real-world product problems through strong product thinking, user research, and metric-driven execution.

Increasing Reviews on Zomato

  • User research with 100+ respondents showed 50% of frequent users lacked incentive to write reviews
  • Designed a gamification system with redeemable virtual coins targeting 2.1M users, with spam-mitigation built in
  • End-to-end solution: wireframes, system architecture, and review-to-order ratio as the success metric

Subscriber Growth for YouTube Music

  • Research with 200+ respondents identified missing equalizer/crossfade as a blocker for 33% of upgrades
  • Designed an AI-powered equalizer that adapts to listening history
  • Wireframes, architecture, completion-rate metric, and a plan for the cold-start problem
GeeksForGeeks

DevOps Certification

GeeksForGeeks · 2025

Exposure to Cloud & DevOps practices, so that I don't sound like a fool in team meetings.

Automated CI/CD Pipeline for Portfolio Deployment

  • GitHub Actions pipeline for automated portfolio deploys
  • S3 for storage, CloudFront for CDN + cache invalidation
  • Route 53 for DNS, ACM for TLS, secure by default

04 — Education

Where it started.

KIIT University

Bachelor of Technology · Computer Science & Engineering

Kalinga Institute Of Industrial Technology

2016 — 2020 CGPA 8.71

05 — Contact

Let's build something good.

Hiring for a backend role, scoping a hard distributed-systems problem, or just want to compare notes on resilience patterns? My inbox is open.

rja.ankit@gmail.com

Systems

How the things I built are put together.

Two production systems, drawn the way I actually think about them. Company internals and vendor names are left out on purpose. What is left is the part that matters, which is the shape of the system and the reasoning behind it.

Purple outline means the service was mine. Grey means it belonged to someone else on the team.

01. Moving business objects between products that were never built together

A procurement suite assembled by acquisition. Every product arrived with its own platform layer, and the products were wired to each other pair by pair, with a different contract each time. The job was to replace all of that with one platform, so the modules talk to it instead of to each other.

MESSAGE BROKER · one for the whole platform 7 business modules each with its own database none of them call each other One exchange One queue everything publishes here deliberately dumb Data sync service consume look up config, gate, transform publish then acknowledge no database of its own One exchange per object routing key from tenant config bindings declared at startup one case fans out to everyone Configuration console cached, one hour, force-invalidate each module consumes from its own queue publish

scroll the diagram sideways to see all of it

Dumb on the way in, rich on the way out

The inbound side is one exchange with a single queue bound to it. No routing, no fan-out, nothing clever. The outbound side is one exchange per business object with routing keys taken from tenant configuration.

  • Ordering does not matter to this pipeline, so a single queue with many consumers scales sideways and stays simple.
  • The honest trade-off is isolation. One queue means a flood from one module puts every other module's events behind it.
  • Exchanges, queues, bindings and routing keys are all declared when the application starts. Adding a new business object is a configuration change, not an infrastructure ticket.

It holds no business knowledge

The pipeline transforms and routes. It does not know what a purchase requisition means, and that is on purpose.

  • Deduplication happens in the consuming modules, keyed on their own business ids, because the business meaning belongs to them and not to the plumbing.
  • Tenant-specific shapes are handled by pluggable transformers, so a new format is a new class and not a new branch in a growing conditional.
  • The result is that the service stays small while the number of object types it carries keeps growing.

Acknowledge last

Consume, look up config, gate, transform, publish, and only then acknowledge. At-least-once, with redelivery when a message is rejected.

  • This is the same rule I applied in a different company on a different broker. The queue is a handoff. The durable write is the record.
  • It means duplicates are the thing to handle, not loss. That is the trade I want, because a duplicate is recoverable and a lost business object is not.
  • The service keeps no state of its own. The broker holds the durability.

02. The communication layer of a helpdesk product

Everything the product says to a customer, and everything a customer sends back. Three email providers that each do the same job differently, a rules layer that decides what an incoming mail means, and a notification platform the whole product publishes into.

OUTSIDE THE PRODUCT Three email providers, three different ways in: a webhook an object store plus a queue a push subscription out is per provider too Provider layer the only door to the outside three shapes, one internal model persist, then acknowledge dedupe on the mail's own id provider spam score, layer one Rules and AI what does this mail mean which ticket, reopen or not reply leaves by the way it came cheap AI content pass, layer two retrieval for suggested replies Model service another team · hosted models Ticketing another team Notification platform built end to end every module publishes into it templates per channel, by config persist, then acknowledge missing data fails fast, no callback Channel services SMS chat email, back through the provider layer synchronous creates the ticket it publishes events too email goes back out here

scroll the diagram sideways to see all of it

Three providers, one internal model

Nothing downstream should know which provider a mail arrived on. That single rule is what makes the rest of the product simple.

  • One provider pushes to a webhook, one drops the raw mail in object storage and notifies through a queue, one uses a push subscription. Three completely different failure modes to normalise.
  • The mail is written to the database and the raw body to object storage before the provider is acknowledged, so nothing is lost at the edge.
  • Deduplication keys off the mail's own message id, which comes from the standard rather than from us. The best idempotency key is usually one the domain already gave you.

Two spam layers, for two reasons

The provider already computes a technical spam score, and it costs nothing to read. So that runs first, at the boundary.

  • A cheap model pass then reads the content of whatever survived, because a mail can be technically clean and still be junk.
  • The expensive retrieval only runs when an agent asks for it. Cost follows intent instead of volume.
  • Suggested replies are shown and then gone. Nothing is stored as a draft, which keeps a model's guess from quietly becoming a record.

A notification platform that stays cheap

Every module in the product publishes into it. That only works if the cost of one notification does not grow with the number of publishers.

  • The event carries everything the notification needs. The platform never calls back to ask for missing data, it fails the request. One callback per event would multiply reads across the whole product.
  • Templates resolve per channel through a small factory, so adding a channel is adding a class, not editing a dispatcher.
  • The email channel routes back out through the provider layer, so there is still only one door to the outside world.

What I would change

A dual write we did not solve

The provider layer writes the mail to a database and the raw body to object storage. Two writes, no transaction across them. If the second fails, the record and the body disagree. We accepted it, and I can tell you exactly where it bites and what I would do about it.

One queue, no isolation

The single inbound queue is simple and it scales sideways, but it gives one noisy module the ability to slow everyone down. The fix is per-module queues or a prefetch limit, and the cost is a topology that no longer explains itself in one sentence.

A trace that dies at the broker

In an asynchronous system a trace id has to be carried in the message headers, or the story of one event stops at the queue and starts again on the other side as something unrelated. That is the first thing I would add.

These are on the page on purpose. A system with no known weaknesses is a system nobody has looked at closely.

Backend engineering case study · 4-service distributed system

Event-driven backend — built to demonstrate distributed-systems depth.

Four Spring Boot services, a topic-exchange message bus, vector search, a distributed lock, and a resilient AI provider layer — all the moving parts senior backend interviews actually drill into. Each engineering choice below is paired with the file in the repo that implements it, so the work can be verified, not just claimed.

Java 21Spring Boot 3.5 Distributed systemsMicroservices Event-driven architecture RabbitMQ · DLQOAuth2 Postgres + pgvector · HNSWRedis · distributed lock AWS BedrockAWS S3 Resilience4j · Retry · Circuit breaker Idempotency · State machine Hibernate 6.5 · JPAFlyway Smile DBSCAN · Clustering Docker · docker-compose

Engineering problems & how I solved them

Each card is a real interview-grade problem this system had to answer. The blue tag is the technical pattern; the file path lets you verify the implementation.

01

At-least-once safe ingestion under Pub/Sub redelivery and crashes

Gmail Pub/Sub may redeliver the same historyId on transient errors. Workers can crash between any two side-effects. The pipeline must never drop an email and never write a row twice.

Two-layer dedup. (1) DB unique constraint uq_email_content_message(mailbox_id, message_id) — Gmail's message id is the natural idempotency key. (2) Explicit existsByGmailMailboxIdAndMessageId check before the API fetch, so a redelivery doesn't even spend a quota call. Every stage progresses processed_status after the durable write, never before — so a crash mid-stage replays from the last persisted state.

Event-drivenIdempotencyState machineAt-least-onceUnique constraintsGCP Pub/Sub

→ ingester/.../GmailMessageProcessingService.java, EmailContent.java

02

Bursty Gmail events coalesced into one sync per mailbox

Gmail can fire many Pub/Sub events per mailbox in seconds. Naively syncing each one wastes API quota and risks racing into duplicate work.

Per-mailbox ReentrantLock + high-watermark coalescing. Events merge(eventHistoryId, Math::max) into a per-mailbox watermark; only the first event acquires the lock and runs a paginated sync that catches up to the latest watermark. Subsequent events tryLock(), see it's held, and exit immediately. Bounded by MAX_SYNC_ITERATIONS = 50 to prevent unbounded catch-up loops.

ReentrantLockHigh-watermarkCoalescingConcurrentHashMapQuota efficiency

→ ingester/.../GmailMessageSyncService.java

03

One slow stage can't starve the others

Sanitize is microseconds; the LLM normalization call is seconds; Bedrock embedding is hundreds of milliseconds. A single queue means the slowest stage back-pressures the entire pipeline.

RabbitMQ topic exchange with 4 isolated stage queues (sanitization · normalization · embedding · clustering), each declared with x-dead-letter-exchange + x-dead-letter-routing-key at bean construction — every queue gets a sibling .dlq. Each listener factory carries its own retry interceptor (maxAttempts=3, exponential 1s → 2s → 4s capped at 10s) with RejectAndDontRequeueRecoverer, prefetch 10, concurrent consumers 4–8.

RabbitMQTopic exchangeDLQ per stageBackpressurePrefetch tuning

→ processor/.../config/RabbitMQConfig.java

04

Classify failures so retries don't waste budget

Bedrock 5xx, 429 throttling, and socket I/O errors should retry — they pass. A 400 malformed request or a revoked refresh token will never succeed — retrying just delays the DLQ and burns rate limit.

Exception classification at the HTTP boundary. HttpServerErrorException, HttpClientErrorException.TooManyRequests, and ResourceAccessException wrap to RetryableAIException → handled by @Retry(name="aiRetry") (3 attempts, 2s base, 2× backoff, 0.5 jitter). All other 4xx wrap to IllegalStateException → fails fast to DLQ. The Pub/Sub subscriber detects invalid_grant and marks the mailbox DISCONNECTED instead of nacking forever.

Resilience4jRetry classificationExponential backoffJitterFault tolerance

→ processor/.../BedrockModelProvider.java, ingester/.../GmailPubSubSubscriber.java

05

Dual-mode clustering: hot path stays sub-second, cold path discovers structure

Every new email needs a cluster now for live labelling — but you also need the system to discover new clusters as the inbox evolves. Doing DBSCAN per email is too slow; doing only periodic batch leaves new emails uncategorised for hours.

Two cooperating clusterers. Incremental: cosine vs existing centroids, assign if similarity ≥ configurable threshold, else mark CLUSTER_ASSIGNMENT_COMPLETED and defer to batch. Batch: full @Transactional Smile DBSCAN over all embeddings — flushes old assignments and re-creates them atomically. Coordinated by BatchClusteringLock.isActive(): incremental skips while batch runs, so a re-cluster doesn't race with hot-path writes. Each assignment records cluster_assignment_type = INCREMENTAL or BATCH for downstream reasoning.

Hot/cold pathDBSCANCosine similarity@TransactionalAtomic re-clusterSmile

→ processor/.../EmailClusteringService.java, taxonomy-engine/.../BatchClusteringService.java

06

Multi-replica safety on a destructive batch job

Batch re-clustering deletes every cluster and assignment for a mailbox, then re-creates them inside one transaction. Two replicas running simultaneously could orphan rows or corrupt the centroid set. A worker that crashes mid-job must not leave the lock held forever.

Redis setIfAbsent + 1-hour TTL. Atomic acquire; second replica fails fast with no spin. TTL guarantees release on crash. Plus belt-and-braces: an ApplicationRunner sweeps stale keys matching <prefix>:mailbox:* on startup; @PreDestroy releases on graceful shutdown.

RedisDistributed lockSETNXTTLLifecycle hooksMulti-replica

→ taxonomy-engine/.../BatchClusteringLock.java, BatchClusteringLockCleanerConfig.java

07

Sub-10ms ANN search via Postgres, not a separate vector DB

Each new email needs its nearest cluster centroid in real time. Naive cosine over an entire table is O(n·d). A separate vector DB (Pinecone, Weaviate) means two-phase commits and a second operational surface.

Postgres + pgvector with HNSW + cosine_ops. Partial index WHERE embedding IS NOT NULL keeps the index lean during the gap between insert and embed. Hibernate 6.5 maps Java float[] directly via @JdbcTypeCode(SqlTypes.VECTOR) + @Array(length=1024) — no custom converter, no dialect hack. One transaction commits the embedding, the cluster assignment, and the status flip together.

pgvectorHNSWPartial indexANNHibernate 6.5SqlTypes.VECTOR

→ db-migrations/.../V1.005__create_email_enrichment.sql, persistence-lib/.../EmailEnrichment.java

08

LLM that justifies its answers, not just emits them

Naive prompts ("classify this email") invent urgency from words like verify, alert, action. The LLM picks a label first and rationalises later. Output drifts and is non-deterministic.

Reason-before-rate JSON schema: the model writes importance_reason before importance, category_reason before category. Override rules baked into the prompt (all bank transaction alerts → LOW UPDATES; all OTPs → LOW UPDATES). Job-hunt persona override promotes named recruiter outreach to HIGH. 8 worked examples cover real edge cases (Axis Bank UPI, LinkedIn InMail vs Job Alerts, IRCTC booking, ATS receipts). Defensive parsing: regex-extracts the JSON block even if the model adds prose; READ_UNKNOWN_ENUM_VALUES_AS_NULL falls back to LOW / PRIMARY on garbage. Temperature 0.0, top-p 1.0 — deterministic. Bedrock prompt capped at MAX_PROMPT_CHARS = 24_000.

Prompt engineeringStructured outputDefensive parsingDeterminismOverride rulesBedrock Converse

→ processor/.../NormalizationPromptHelper.java

09

Email sanitization as a pluggable pipeline, not a god method

Real email HTML is hostile: tracking pixels, hidden divs, fancy quotes, zero-width chars, table-shaped layouts, CID inline images. A single regex pass either misses cases or becomes unreadable.

Annotation-driven pipeline registry. Each step is a @SanitizationStep(order=N) Spring bean; ContentSanitizationPipelineRegistry discovers and sorts them at startup — adding a new step is one new class. Steps include: jsoup HTML→text (preserves block boundaries as newlines, list items as - , table cells as | , appends (href) when anchor text ≠ url); inline-reference remover; CharacterNormalizer with 30+ curly-quote / em-dash / arrow / ellipsis / TM mappings, six regex passes for Unicode spaces / invisible chars / pipe-only rows / multi-space / trailing-space / excessive-newline collapse. Skips parsing entirely when no HTML hints are present.

Plugin architectureSpring annotationsjsoupRegex pipelineUnicode normalization

→ processor/.../sanitization/pipeline/

10

Hierarchy from cosine: reuse → merge → create

Each batch re-cluster invents fresh cluster IDs but the user already has labels in Gmail. Naively creating a new label per cluster floods their sidebar and the system drifts further with every run.

Three-stage label decision. (1) Sort cluster members by cosine-to-centroid, take top 8, send to LLM with the existing label-name pool — model is biased toward suggesting an existing name. (2) Case-insensitive string match → reuse. (3) Otherwise embed the suggestion (Titan v2) and find nearest existing label by cosine; if ≥ MERGE_THRESHOLD = 0.80, merge into it; else create a new Label with the suggestion's embedding stored as reference_embedding. The pool grows monotonically across runs.

Embedding dedupCosine thresholdRepresentative samplingSidebar hygiene

→ taxonomy-engine/.../ClusterLabelingService.java

11

Same code in prod and on a laptop with zero cloud

Bedrock costs money and needs IAM. Local dev and CI shouldn't burn either. Hard-coding a provider makes the test loop painful and the architecture vendor-locked.

ModelProvider interface with Bedrock and Ollama implementations. Selected by config (embedding-provider-name, llm-provider-name). Same call shape; Titan v2-specific knobs (dimensions, normalize) only set when model id matches. Both Bedrock and the local Ollama embedding return 1024-d so swapping providers doesn't invalidate the vector index. Same trick applied to EmailStorageProvider — Local for dev, S3 for prod, one interface.

AWS BedrockOllamaDependency inversionSpring configS3 / Local storage

→ processor/.../model/factory/, persistence-lib/.../storage/

12

One schema across three services, zero drift

JPA entities are needed by ingester, processor, and taxonomy-engine. Copy-pasting guarantees silent drift; a data-access microservice adds a network hop per query.

Shared persistence-lib published as a private Maven package (GitHub Packages). Pinned versions force conscious upgrades. The only cross-service contract for data shape lives in one place. EmailStorageProvider and 7 Flyway migrations ride along in the same library. Hibernate uses ddl-auto: validate in every service — schema mismatches fail at startup, not at runtime.

Maven multi-moduleJPA / HibernatePrivate package registrySchema-as-codeFlyway

→ persistence-lib/.../entity/, db-migrations/.../V1.001 … V1.007

Idempotent state machine

Every email carries a processed_status. The 5 stages each have 4 sub-states (PUBLISHED_FOR_X → X_STARTED → X_COMPLETED / X_FAILED) — 20 states total in the ProcessedStatus enum. Transitions happen after the side-effect (publish, save, embed), never before. A crash mid-stage is replayed cleanly on restart; Pub/Sub redeliveries see a later state and no-op.

EMAILRECEIVED SANITIZATIONCOMPLETED NORMALIZATIONCOMPLETED EMBEDDINGGENERATED CLUSTERASSIGNED ✓ jsoup + normalizer LLM summary + importance Bedrock Titan 1024-d cosine vs centroids crash before publish → replay from RECEIVED crash before Bedrock call → replay from NORMALIZED
  • The invariant: a status only advances after the next stage durably accepted the work (DB save committed, Rabbit publish ack'd).
  • On crash: the listener reads the row, sees the previous COMPLETED state, re-publishes the next-stage event. Downstream stages no-op on observing a later state.
  • On Pub/Sub redelivery: the uq_email_content_message(mailbox_id, message_id) constraint blocks duplicate inserts; the listener acks and exits.
  • On revoked OAuth token: invalid_grant is caught at the subscriber, the mailbox is marked DISCONNECTED, the message is acked — no infinite nack loop.

Retry & DLQ flow per stage

How a transient failure (Bedrock 5xx, Gmail 429) escalates from Resilience4j retries to a stage-specific Dead Letter Queue — without ever blocking the other stages.

Producer Stage queueemail.embedding Worker (Bedrock call) ✓ successadvance state machine Resilience4j @Retry3× exponential backoff Stage DLQemail.embedding.dlq Ops alert + manual replay 5xx / timeout retry ok 3 attempts exhausted
  • Stage isolation: DLQ is per-stage. Embed failures don't slow sanitize.
  • State machine never advances while the message is in retry — so manual replay from the DLQ resumes exactly where the failure happened.
  • Retry budget is bounded: Resilience4j caps wait time so a single message can't tie up a consumer.

Architecture

Three services + a shared persistence library. Click any node or edge for the reasoning behind it.

Gmail GCP Pub/Sub Ingester owns Gmail credentials Processor sanitize · normalize embed · cluster 4 queues · per-stage DLQ idempotent on replay Taxonomy Engine DBSCAN + LLM naming Postgres pgvector + HNSW Redis RabbitMQ Object Storage persistence-lib · shared via private Maven package

Engineering highlights

The non-obvious pieces — verbatim from the repo, with file paths so they can be verified.

Pipeline demo — 3 emails, end to end

Three sample emails enter the pipeline. Watch each one get sanitized, normalized, embedded, and assigned a cluster. At the end, DBSCAN groups them and the LLM picks names. The code panel shows the real Java method running at each stage.

Idle
📥 Inbox
🧹 Sanitizejsoup
✂️ Normalize5000-char cap
🧬 EmbedBedrock 1024-d
🎯 Clustercosine vs centroids
Now executing —
Press “Run demo” to see the code that processes each email at every stage.

📚 Taxonomy result

After DBSCAN re-clusters globally and the LLM names each group:

Run it · Deploy

Source code and how to run the stack locally.