Let's Build a Print-Ready Die-Cut Sticker SaaS from scratch in Golang & Next.js [Part 5]

Part 5: The Async Pipeline & Real-Time Status

createDesign returns in milliseconds. The actual work — downloading a multi-megabyte image, running it through background removal and contour tracing, rendering mockups, writing results back — takes seconds, sometimes most of a second per stage. Making the client wait on an HTTP response for that would mean holding a connection open across a Cloud Run instance that might scale to zero mid-request. Instead, the API hands a job to a queue and gets out of the way, and the client finds out what happened by subscribing to a channel, not by polling.

This part covers both halves of that: the durable queue connecting API and worker, and the pub/sub bridge that turns a worker's progress into a live GraphQL subscription.

The shared job contract

Both services/api (which enqueues) and services/worker (which consumes) need to agree on exactly what a job looks like, so that contract lives in its own module, imported by both:

1// packages/jobs/jobs.go
2type ProcessDesignPayload struct {
3	DesignID      uuid.UUID      `json:"design_id"`
4	UserID        string         `json:"user_id"`
5	SourceKey     string         `json:"source_key"`
6	ContourOpts   ContourOptions `json:"contour_opts"`
7	PhysicalWMm   float64        `json:"physical_w_mm,omitempty"`
8	PhysicalHMm   float64        `json:"physical_h_mm,omitempty"`
9	AttemptNumber int            `json:"attempt_number"`
10	EnqueuedAt    time.Time      `json:"enqueued_at"`
11}
12
13const (
14	TypeProcessDesign   = "process:design"
15	TypeReprocessDesign = "reprocess:design"
16	MaxRetries          = 5
17)

The queue itself is asynq running against Upstash Redis — Redis was already a project dependency for rate limiting, so the job queue doesn't add a new moving part, just a new use for one that's already there.

Deduplication that doesn't over-collapse

A naive dedup key of just "this design" would break reprocessDesign — a legitimate request to re-run with a different contour offset would silently no-op if it collided with an in-flight job for the same design. The fix is scoping the key to the design and the specific options being requested:

1// packages/jobs/jobs.go
2func dedupKey(taskType string, designID uuid.UUID, opts ContourOptions) string {
3	optsData, _ := json.Marshal(opts)
4	hash := md5.Sum(append([]byte(taskType+designID.String()), optsData...))
5	return fmt.Sprintf("%x", hash)
6}
7
8func NewProcessTask(payload ProcessDesignPayload) (*asynq.Task, error) {
9	data, _ := json.Marshal(payload)
10	return asynq.NewTask(
11		TypeProcessDesign, data,
12		asynq.MaxRetry(MaxRetries),
13		asynq.Queue(DefaultQueue),
14		asynq.TaskID(dedupKey(TypeProcessDesign, payload.DesignID, payload.ContourOpts)),
15	), nil
16}

Two accidental double-clicks on "upload" for the same design with the same options collapse into one job. A reprocessDesign call with a wider border offset gets its own task ID and runs independently — same design, different key.

Enqueueing: the API's half of the handoff

createDesign builds the payload and hands it to an asynq client; from the resolver's point of view, enqueueing is fire-and-forget — a failure here gets logged, not surfaced as an error, because the design row already exists and can always be reprocessed:

1// services/api/graph/resolver/schema.resolvers.go
2task, taskErr := jobs.NewProcessTask(jobs.ProcessDesignPayload{
3	DesignID:    design.ID,
4	UserID:      user.ClerkID,
5	SourceKey:   input.StorageKey,
6	ContourOpts: contourOpts,
7	PhysicalWMm: physW,
8	PhysicalHMm: physH64,
9	EnqueuedAt:  time.Now(),
10})
11if taskErr == nil {
12	if _, enqErr := r.JobQueue.EnqueueContext(ctx, task); enqErr != nil {
13		log.Printf("WARN: enqueue job for design %s: %v", design.ID, enqErr)
14	}
15}

Right after enqueueing, the resolver flips designs.status to PROCESSING — the row reflects "a job exists for this" immediately, well before the worker has even picked it up.

Consuming: the worker's half

The worker registers one handler for both task types — process:design and reprocess:design run through the same pipeline, since a reprocess is just a process run with different options on an existing design:

1// services/worker/main.go
2srv, err := jobs.NewAsynqServer(5) // max 5 concurrent heavy jobs
3mux := asynq.NewServeMux()
4mux.HandleFunc(jobs.TypeProcessDesign, handleProcessDesign)
5mux.HandleFunc(jobs.TypeReprocessDesign, handleProcessDesign)
6srv.Run(mux)

The concurrency cap of 5 is a deliberate ceiling, not a tuning afterthought — the imaging pipeline from Part 4 is CPU-heavy per job, and a Cloud Run instance running twenty of these at once would thrash rather than finish any of them faster. If REDIS_URL isn't set at all, the worker still boots and blocks forever in a standalone no-queue mode, rather than crash-looping — useful for confirming the binary builds and starts without needing Redis available.

The pipeline: download, process, upload, repeat

pipeline.Run is the glue between the queue and core-imaging — download the original from R2, decode it, hand it to Process, then upload every artifact the result produced:

1// services/worker/internal/pipeline/pipeline.go
2func Run(ctx context.Context, payload *jobs.ProcessDesignPayload) (*Result, error) {
3	imgBytes, err := downloadObject(ctx, s3Client, cfg.Bucket, payload.SourceKey)
4	img, err := decodeImage(imgBytes)
5
6	opts := coreimaging.DefaultOptions()
7	// ...apply payload.ContourOpts, PhysicalWMm/HMm
8	pResult, err := coreimaging.Process(img, opts)
9
10	// cutout PNG → cutouts/{id}.png
11	// contour SVG → contours/{id}.svg
12	// print export → exports/{id}.svg
13	mockupURLs, _ := renderAndUploadMockups(ctx, s3Client, cfg, designID, pResult.Cutout)
14
15	return &Result{CutoutURL: ..., MockupURLs: mockupURLs, Confidence: pResult.Meta.Confidence, ...}, nil
16}

Mockups get composited server-side too — the cutout scaled to 60% of a scene and centered over a solid background — so the public storefront page (Part 6) has stable images that don't depend on a client re-rendering anything. A mockup render failure is logged and treated as non-fatal; a design without mockups still ends up READY, it just has an empty mockupUrls list.

Status changes go through Redis, not the database, in the hot path

When a job finishes, the worker writes the result to Postgres and also publishes a small event to a Redis channel keyed by design ID:

1// services/worker/main.go
2func publishStatus(ctx context.Context, designID, status, cutoutURL string) {
3	redisURL := os.Getenv("REDIS_URL")
4	if redisURL == "" {
5		return
6	}
7	rdb := redis.NewClient(opts)
8	data, _ := json.Marshal(statusEvent{DesignID: designID, Status: status, CutoutURL: cutoutURL})
9	rdb.Publish(ctx, "design:status:"+designID, data)
10}

On the API side, the same dev/prod-symmetry pattern from Part 2's auth story shows up again: a pubsub.Bus interface has two implementations, a real Redis client and an in-memory fan-out bus, and the constructor picks based on whether REDIS_URL is set:

1// services/api/internal/pubsub/pubsub.go
2func NewBus() (Bus, error) {
3	rc, err := NewRedisClient()
4	if err != nil {
5		return nil, err
6	}
7	if rc == nil {
8		log.Println("[pubsub] REDIS_URL not set — using in-memory bus (dev mode)")
9		return NewInMemoryBus(), nil
10	}
11	return rc, nil
12}

Locally, with no Redis running at all, subscriptions still work end-to-end through the in-memory bus — a single-process pub/sub map guarded by a mutex. In production, Redis is what lets the API and worker be two separate, independently-scaled processes that still talk to each other in real time.

The subscription resolver bridges the channel to GraphQL

designStatus(id) subscribes to the bus and re-fetches the full Design row from Postgres on every event, rather than trusting the event payload to carry every field the client might need:

1// services/api/graph/resolver/schema.resolvers.go
2func (r *subscriptionResolver) DesignStatus(ctx context.Context, id string) (<-chan *model.Design, error) {
3	events, err := r.PubSub.Subscribe(ctx, id)
4	ch := make(chan *model.Design, 8)
5	go func() {
6		defer close(ch)
7		for {
8			select {
9			case <-ctx.Done():
10				return
11			case event, ok := <-events:
12				if !ok {
13					return
14				}
15				d, _ := r.Queries.GetDesignByID(ctx, designID)
16				ch <- dbDesignToModel(d)
17			}
18		}
19	}()
20	return ch, nil
21}

That re-fetch is deliberate: the pub/sub event only carries design_id, status, and maybe a cutout_url — it's a doorbell, not a payload. The subscription stays cheap to publish (small JSON message) and the client always gets a fully-formed Design, not a partial one assembled from whatever happened to be in the event.

The whole round trip

Verify it

1# terminal 1
2cd services/worker && go run .
3# terminal 2
4cd services/api && go run .

1subscription {
2  designStatus(id: "your-design-id") {
3    status
4    cutoutUrl
5    readiness { dpi { status } }
6  }
7}

Open the subscription in the playground before triggering createDesign — the design should flip from PROCESSING to READY (or FAILED) on the open socket within a second or two of the worker log printing "done", with no refresh and no re-query needed.

With results landing in real time, there's finally a UI worth building around them.

Next up — Part 6: the Studio and storefront in Next.js — the create flow, live status, readiness badges, and semantic search over published designs.