Invoice Auto-Send Flow

When all packages in an invoice are unloaded, the system automatically creates the invoice in Square. This involves two functions connected by a Google Cloud Pub/Sub message queue.

Why Pub/Sub?

Instead of calling the Square API directly from a Firestore trigger, the system uses Pub/Sub as a message queue. This is because:

• Cloud Functions can fail mid-execution. If the function crashes after calling Square but before updating Firestore, the invoice state would be inconsistent.
• Pub/Sub guarantees delivery. If the consumer fails, the message is redelivered automatically.
• Decoupling. The trigger function returns quickly. The slow Square API work happens separately.

The Two Functions

1. enqueueInvoiceForProcessing (Producer)

Trigger: Firestore onUpdate on Invoices/{invoiceID}

Fires on every write to any Invoice document. Most writes are ignored — it only proceeds when isAllUnloaded or status changes.

Gate conditions (all must be true):

If all conditions pass, it publishes a message { invoiceID, operationID } to the process-invoice Pub/Sub topic.

2. processInvoiceFromQueue / processInvoiceInternal (Consumer)

Trigger: Pub/Sub message on process-invoice topic

Picks up the message and:

1. Fetches the invoice from Firestore
2. Builds a Square Order (line items, taxes, discounts)
3. Creates the Order via Square API
4. Creates the Invoice via Square API
5. On success: sets invoiceSent: true, automaticallySent: 1
6. On failure: sends failure notification email, adds to in-memory retry queue

Deduplication (Three Layers)

Firebase Cloud Functions have at-least-once execution — a single Firestore write event can trigger the function more than once. Without dedup, the same invoice could be sent to Square twice.

Layer 1: PubSubQueue (prevents duplicate enqueuing)

Before publishing to Pub/Sub, the producer runs a Firestore transaction:

PubSubQueue/{invoiceID} exists?
  YES → exit (already enqueued)
  NO  → atomically create the doc, then publish

This uses the invoiceID as the dedup key. Even if the trigger fires twice for the same event, only the first instance creates the doc and publishes. The transaction is atomic — two simultaneous instances cannot both succeed.

Layer 2: invoiceSent check (prevents duplicate Square API calls)

Before calling the Square API, the consumer re-reads the invoice from Firestore:

invoice.invoiceSent === true?
  YES → skip (already sent by another instance)
  NO  → proceed with Square API call

This catches cases where two Pub/Sub messages somehow both get through (e.g., Pub/Sub at-least-once delivery of different messages).

Layer 3: invoiceSent check in catch block (prevents false alarms)

If a Square API call fails, the catch block re-reads the invoice before reacting:

invoice.invoiceSent === true?
  YES → silently return (success was already recorded)
  NO  → send failure email, add to retry queue, update status

This prevents the catch block from overwriting invoiceSent: true with false and sending false failure emails.

operationID (Pub/Sub redelivery dedup)

Each published message includes a random operationID (UUID). The consumer checks ProcessedOperations/{operationID} before processing. This prevents Pub/Sub from redelivering the same message and processing it twice. Note: this does NOT prevent two different messages for the same invoice — that’s what the PubSubQueue and invoiceSent checks handle.

Failure Handling

When an invoice fails to send to Square:

1. Failure email sent to admin team with:

   • Client name
   • Container name
   • Invoice ID
   • Specific error details (status code, error type, error message)

2. Retry queue — in-memory queue with exponential backoff:

   • Attempt 1: 5 min delay
   • Attempt 2: 10 min delay
   • Attempt 3: 15 min delay
   • After 3 failures: marks automaticallySent: 2, maxRetriesReached: true

3. Manual fallback — user can always click “Send to Square” button in the app

Failure Categories

Flow Diagram

Last package unloaded
  │
  ▼
updateInvoiceWithRetry()
  writes: packages array + isAllUnloaded=true
  │
  ▼
enqueueInvoiceForProcessing fires
  │
  ├─ isAllUnloaded or status changed? ──NO──▶ return null
  │
  YES
  │
  ├─ All 4 gate conditions pass? ──NO──▶ send failure email or return null
  │
  YES
  │
  ├─ PubSubQueue/{invoiceID} exists? ──YES──▶ return null (already enqueued)
  │
  NO (atomically created)
  │
  ▼
Publish to Pub/Sub: { invoiceID, operationID }
  │
  ▼
processInvoiceFromQueue picks up message
  │
  ├─ ProcessedOperations/{operationID} exists? ──YES──▶ skip
  │
  NO
  │
  ├─ invoice.invoiceSent === true? ──YES──▶ skip
  │
  NO
  │
  ▼
Square API: Create Order → Create Invoice
  │
  ├─ SUCCESS ──▶ invoiceSent=true, automaticallySent=1
  │
  FAILURE
  │
  ├─ invoice.invoiceSent === true? ──YES──▶ silently return
  │
  NO
  │
  ▼
Send failure email + add to retry queue + invoiceSent=false

Incident: March 27, 2026

22 invoices received false “Failed Auto-Send” emails during container unloading. Root cause: Firebase at-least-once execution triggered enqueueInvoiceForProcessing twice for the same isAllUnloaded change. The old code had no PubSubQueue dedup, so two Pub/Sub messages were published per invoice. The first succeeded, the second failed with Square error 400 (“invoice number already used”). The catch block then overwrote invoiceSent: true with false and sent false alarm emails.

Fixes deployed March 28, 2026:

1. PubSubQueue transaction (Layer 1)
2. invoiceSent check before Square API call (Layer 2)
3. invoiceSent check in catch block (Layer 3)
4. Improved failure emails with container name and specific error details