<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[pinnacloid-tech]]></title><description><![CDATA[Pinnacloid Tech Insights explores software engineering, generative AI, machine learning, cloud technology, application development, and digital transformation. ]]></description><link>https://pinnacloid-tech.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6a7e48c0c94047d681592fca/14c8c6ea-804f-48ec-a95f-6fc0b3866430.png</url><title>pinnacloid-tech</title><link>https://pinnacloid-tech.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Wed, 16 Sep 2026 12:29:54 GMT</lastBuildDate><atom:link href="https://pinnacloid-tech.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[The Multi-Tenant SaaS Bug That Hides in Background Jobs]]></title><description><![CDATA[Imagine opening a “monthly usage” email from a SaaS product and seeing another company’s data in the attachment.
The API was protected. Every request required authentication. Every controller filtered]]></description><link>https://pinnacloid-tech.hashnode.dev/the-multi-tenant-saas-bug-that-hides-in-background-jobs</link><guid isPermaLink="true">https://pinnacloid-tech.hashnode.dev/the-multi-tenant-saas-bug-that-hides-in-background-jobs</guid><category><![CDATA[SaaS]]></category><category><![CDATA[Security]]></category><category><![CDATA[backend developments]]></category><category><![CDATA[PostgreSQL]]></category><category><![CDATA[TypeScript]]></category><dc:creator><![CDATA[Shahnawaz Ali]]></dc:creator><pubDate>Fri, 14 Aug 2026 00:10:41 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6a7e48c0c94047d681592fca/63f0f84e-f0b4-41c0-901e-7f19ea5ec74b.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Imagine opening a “monthly usage” email from a SaaS product and seeing another company’s data in the attachment.</p>
<p>The API was protected. Every request required authentication. Every controller filtered by tenant_id. The dashboard had passed its security review.</p>
<p>The leak happened later—inside a background worker.</p>
<p>This is the uncomfortable part of multi-tenant SaaS security: a request can be safe while the work it creates is not. Once a task leaves the request-response cycle and enters a queue, the assumptions that protected it often disappear with the HTTP context.</p>
<p>The fix is not another scattered WHERE tenant_id = ? condition. Tenant identity has to become part of the job’s contract, database session, idempotency model, observability, and test strategy.</p>
<p>Let’s build that boundary deliberately.</p>
<h2>TL;DR</h2>
<p>A safe multi-tenant background job should:</p>
<p>Carry a server-derived tenant identifier in its payload. Re-authorize the tenant and operation when the job runs. Use tenant-scoped repositories or data-access objects. Enforce isolation again in the database, preferably with row-level security where appropriate. Include tenant identity in idempotency keys, traces, metrics, and dead-letter records. Be tested with hostile cross-tenant cases, not only happy paths.</p>
<h2>Why background jobs escape the tenant boundary</h2>
<p>In a web request, tenant context is usually available from one of four places:</p>
<ul>
<li><p>A verified token claim</p>
</li>
<li><p>The authenticated user’s membership</p>
</li>
<li><p>The requested hostname or subdomain</p>
</li>
<li><p>A trusted server-side lookup</p>
</li>
</ul>
<p>Middleware resolves that context, attaches it to the request, and downstream code consumes it. The boundary feels solid because every layer is participating.</p>
<p>A queue changes the shape of the system.</p>
<p>The worker may run seconds or hours later. It may run in another process, region, or cloud account. There is no request object, no browser session, and sometimes no original user. Retries can outlive membership changes. Scheduled jobs may be created once and execute for months.</p>
<p>If tenant context was ambient state, it is gone.</p>
<p>That is how seemingly harmless code turns dangerous:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6a7e48c0c94047d681592fca/102f3ff9-f4c5-4f4d-a574-804922ec7ef8.png" alt="Background worker carrying explicit tenant context from an authenticated API through a queue to an isolated database transaction." style="display:block;margin:0 auto" />

<pre><code class="language-typescript">// Unsafe: the worker receives only a record ID.
await queue.add("send-invoice", {
  invoiceId: invoice.id
});

worker.process("send-invoice", async (job) =&gt; {
  const invoice = await db.invoice.findUnique({
    where: { id: job.data.invoiceId }
  });

  await emailInvoice(invoice);
});
</code></pre>
<p>The code assumes invoice IDs are globally unguessable and that whoever queued the job was authorized. It also assumes the record will still belong to the same tenant when the worker executes.</p>
<p>Those are application assumptions—not security boundaries.</p>
<h2>1. Treat tenant context as data, not atmosphere</h2>
<p>The first design change is simple: make tenant identity explicit in the job schema.</p>
<pre><code class="language-typescript">type TenantJob&lt;TPayload&gt; = {
  tenantId: string;
  actorId: string | null;
  authorizationVersion: number;
  kind: string;
  idempotencyKey: string;
  requestedAt: string;
  payload: TPayload;
};

type SendInvoicePayload = {
  invoiceId: string;
};
</code></pre>
<p>The tenantId must come from trusted server-side context. Never accept it directly from an unverified request body because “the worker will check it later.”</p>
<p>The actorId answers “who initiated this?” and can be null for system jobs. authorizationVersion is optional, but it is useful when permissions or policies can change while a job waits in the queue.</p>
<p>Most importantly, the payload is now self-describing. An engineer inspecting a dead-letter record can tell which tenant boundary the job belongs to without reconstructing an old request.</p>
<h2>2. Re-authorize at execution time</h2>
<p>Authorization at enqueue time is necessary, but it is not always sufficient.</p>
<p>Suppose an employee requests a large export, then loses access before the worker starts. Should the export still run? There is no universal answer. A billing receipt and a customer-data export have different risk profiles.</p>
<p>Define the policy per operation:</p>
<p>Snapshot authorization: permission at enqueue time is enough. Live authorization: permission must still exist when the job executes. System authorization: the operation belongs to an internal scheduler, not an end user. Dual authorization: validate both the original request and current policy.</p>
<p>For sensitive exports, deletions, payments, or administrative changes, live authorization is usually the safer default.</p>
<pre><code class="language-typescript">worker.process("export-customer-data", async (job) =&gt; {
  const data = job.data as TenantJob&lt;{ exportId: string }&gt;;

  const tenant = await tenants.requireActive(data.tenantId);

  if (!data.actorId) {
    throw new Error("This job requires an initiating actor");
  }

  await authorization.requireCurrentPermission({
    tenantId: tenant.id,
    actorId: data.actorId,
    permission: "customer_data.export"
  });

  await runExport(tenant, data.payload.exportId);
});
</code></pre>
<p>This is not about distrusting the queue. It is about acknowledging time. The system that authorized a task and the system that executes it may observe different realities.</p>
<h2>3. Make cross-tenant queries difficult to express</h2>
<p>A shared database client is convenient. It is also an easy place to forget tenant scoping.</p>
<p>Instead of passing tenant Id into selected queries, construct a tenant-scoped data-access layer once and make the unsafe path awkward.</p>
<pre><code class="language-javascript">class TenantInvoiceRepository {
  constructor(
    private readonly db: Database,
    private readonly tenantId: string
  ) {}

  async findById(invoiceId: string) {
    return this.db.invoice.findFirst({
      where: {
        id: invoiceId,
        tenantId: this.tenantId
      }
    });
  }

  async markSent(invoiceId: string) {
    return this.db.invoice.updateMany({
      where: {
        id: invoiceId,
        tenantId: this.tenantId
      },
      data: {
        status: "sent"
      }
    });
  }
}
</code></pre>
<p>Notice the use of update Many rather than an update operation that identifies a row only by its globally unique ID. The tenant predicate remains part of the write.</p>
<p>A useful code-review question is:</p>
<p>“Can this function access a different tenant if a caller passes the wrong record ID?”</p>
<p>If the answer is yes, the function is operating below the boundary and needs an additional control.</p>
<h2>4. Let the database enforce the rule too</h2>
<img src="https://cdn.hashnode.com/uploads/covers/6a7e48c0c94047d681592fca/63610ebe-4139-4f5d-b672-d7afa1b1c855.png" alt="Layered tenant-isolation controls across application code, a background worker, and PostgreSQL row-level security." style="display:block;margin:0 auto" />

<p>Application checks are valuable, but they share one weakness: every future query has to remember them.</p>
<p>PostgreSQL row-level security (RLS) can add a second boundary. The worker sets the tenant for the current transaction, and policies restrict which rows that session can read or write.</p>
<pre><code class="language-typescript">ALTER TABLE invoices ENABLE ROW LEVEL SECURITY;
ALTER TABLE invoices FORCE ROW LEVEL SECURITY;

CREATE POLICY invoice_tenant_isolation
ON invoices
USING (
  tenant_id = current_setting('app.tenant_id', true)::uuid
)
WITH CHECK (
  tenant_id = current_setting('app.tenant_id', true)::uuid
);
</code></pre>
<p>Then establish the tenant inside the transaction:</p>
<pre><code class="language-typescript">await db.transaction(async (tx) =&gt; {
  await tx.query(
    "select set_config('app.tenant_id', $1, true)",
    [job.data.tenantId]
  );

  const repository = new TenantInvoiceRepository(
    tx,
    job.data.tenantId
  );

  await repository.markSent(job.data.payload.invoiceId);
});
</code></pre>
<p>The true flag makes the setting local to the transaction. That matters when connections return to a pool; tenant context must not leak into the next borrower’s session.</p>
<p>RLS is not a magic shield. Table owners normally bypass row security unless FORCE ROW LEVEL SECURITY is enabled, and roles with BYPASSRLS still bypass it. The worker should connect with a dedicated, least-privileged role—not the migration owner or a superuser.</p>
<p>Also decide what happens when app.tenant_id is missing. A fail-closed policy should return no rows or raise an error, not quietly broaden access.</p>
<h2>5. Put the tenant inside the idempotency key</h2>
<p>Background workers retry. Networks fail after a write but before an acknowledgement. A job can be delivered more than once.</p>
<p>An idempotency key prevents duplicate effects, but in a multi-tenant system its namespace must include the tenant.</p>
<pre><code class="language-typescript">const idempotencyKey = [
  tenantId,
  "send-invoice",
  invoiceId,
  invoiceVersion
].join(":");
</code></pre>
<p>Without tenant identity, two customers can collide if business identifiers are unique only within a tenant. The same principle applies to cache keys, distributed locks, object-storage paths, webhook deduplication, and rate-limit buckets.</p>
<p>A compact rule helps:</p>
<p>If a resource is tenant-scoped, every operational namespace around it must be tenant-scoped too.</p>
<h2>6. Preserve context through retries and dead-letter queues</h2>
<p>The first attempt is rarely the dangerous one. The danger appears when the job is retried by different code, replayed manually, or moved to a dead-letter queue.</p>
<p>Preserve the original tenantId, actorId, authorization mode, requestedAt, schemaVersion, and idempotencyKey across every transition. Do not rebuild them from partial payloads.</p>
<p>When an operator replays a failed job, the same authorization and tenant-validation path should run again. A replay button must not become an administrative bypass.</p>
<p>Version the job contract as well:</p>
<pre><code class="language-typescript">type JobEnvelope&lt;T&gt; = {
  schemaVersion: 2;
  tenantId: string;
  actorId: string | null;
  authorizationMode: "snapshot" | "live" | "system";
  idempotencyKey: string;
  traceId: string;
  payload: T;
};
</code></pre>
<p>Schema versions make migrations explicit. They also prevent an old worker from silently interpreting a new payload with the wrong assumptions.</p>
<h2>7. Make tenant context observable—carefully</h2>
<p>When a job fails, the first operational question is often “which customer did this affect?”</p>
<p>Reliable tenant isolation also depends on disciplined <a href="https://www.pinnacloid.com/services/devops"><strong>monitoring and observability</strong></a>, because missing or changing tenant context should be detected before it becomes a customer-facing incident. Add a stable tenant identifier to traces, structured logs, metrics, and audit records—then add guardrails:</p>
<ul>
<li><p>Use an internal opaque ID rather than a customer name.</p>
</li>
<li><p>Do not put secrets, email addresses, or payload data into metric labels.</p>
</li>
<li><p>Control who can query cross-tenant logs.</p>
</li>
<li><p>Avoid high-cardinality metrics when a trace or log field is more appropriate.</p>
</li>
<li><p>Alert when a job’s tenant context is missing or changes unexpectedly.</p>
</li>
</ul>
<p>A trace should make the boundary visible:</p>
<pre><code class="language-typescript">span.setAttributes({
  "saas.tenant_id": tenantId,
  "job.kind": job.kind,
  "job.schema_version": job.schemaVersion
});
</code></pre>
<p>The purpose is not to create a new data leak in the observability platform. It is to make tenant flow auditable without exposing customer content.</p>
<h2>8. Test the boundary with adversarial cases</h2>
<p>A happy-path test proves that Tenant A can process Tenant A’s invoice. That is useful, but it does not prove isolation.</p>
<p>Add tests that intentionally cross the boundary:</p>
<ul>
<li><p>Queue a Tenant A job with a Tenant B invoice ID.</p>
</li>
<li><p>Change the actor’s membership before execution.</p>
</li>
<li><p>Replay a dead-letter job after the tenant is suspended.</p>
</li>
<li><p>Run two tenants’ jobs concurrently on the same connection pool.</p>
</li>
<li><p>Omit tenant context and confirm the database fails closed.</p>
</li>
<li><p>Try a cache key or lock name that would collide across tenants.</p>
</li>
<li><p>Verify that logs contain the opaque tenant ID but no customer data.</p>
</li>
<li><p>Use a worker role that does not own the tables and has no BYPASSRLS privilege.</p>
</li>
</ul>
<p>One of the most effective tests is a small matrix:</p>
<pre><code class="language-typescript">for (const jobTenant of [tenantA, tenantB]) {
  for (const recordTenant of [tenantA, tenantB]) {
    const result = await runJob({
      jobTenantId: jobTenant.id,
      invoiceId: invoices[recordTenant.id].id
    });

    expect(result.allowed).toBe(
      jobTenant.id === recordTenant.id
    );
  }
}
</code></pre>
<p>This catches the exact class of bug that ordinary unit tests tend to miss.</p>
<h2>A practical architecture for tenant-safe workers</h2>
<p>A robust flow looks like this:</p>
<p>The API authenticates the caller and resolves tenant membership. The API authorizes the requested operation. The producer creates a versioned job envelope with server-derived tenant context. The queue stores and transports the envelope without rewriting it. The worker validates the tenant and applies the operation’s authorization policy. A tenant-scoped repository limits application-level queries. The database enforces row access for the active tenant. Idempotency, storage, locks, and caches use tenant-scoped namespaces. Traces and audit logs record an opaque tenant identifier. Retries and replays return through the same validation path.</p>
<p>No single layer is perfect. The strength comes from independent controls failing in different ways.</p>
<h2>The review checklist</h2>
<p>Before shipping a multi-tenant worker, ask:</p>
<ul>
<li><p>Where does tenantId come from, and can a client influence it?</p>
</li>
<li><p>Does the job carry enough context to be understood outside the original request?</p>
</li>
<li><p>Is authorization checked at the correct time for this operation?</p>
</li>
<li><p>Can repository methods access a record from another tenant?</p>
</li>
<li><p>Does the database enforce isolation if an application query is wrong?</p>
</li>
<li><p>Does the worker use a least-privileged, non-owner database role?</p>
</li>
<li><p>Are idempotency keys, cache keys, locks, and storage paths tenant-scoped?</p>
</li>
<li><p>Do retries, scheduled runs, and manual replays preserve the same controls?</p>
</li>
<li><p>Can operators investigate failures without seeing customer data?</p>
</li>
<li><p>Do tests prove that cross-tenant combinations fail?</p>
</li>
</ul>
<p>If any answer is vague, the boundary is probably living in convention rather than architecture.</p>
<h2>The real goal: make the safe path the easy path</h2>
<p>Multi-tenancy is often introduced as a data-model choice: add tenant_id and filter every query.</p>
<p>In production, it is an execution-context problem.</p>
<p>Tenant identity has to survive every boundary the workload crosses—HTTP, queue, process, transaction, retry, cache, object store, and observability pipeline. When that identity is explicit and enforced by more than one layer, background work becomes easier to reason about and much harder to misuse.</p>
<p>The best security design is not the one that asks every engineer to remember one more rule. It is the one that makes an unsafe operation difficult to express.</p>
<p>If you are reviewing a live SaaS platform or modernizing a system where tenant boundaries have grown organically, the engineering team at <a href="https://www.pinnacloid.com/">Pinnacloid</a> works on secure, scalable software and cloud systems.</p>
<h2>About the author</h2>
<p>Syed Ebad Hussain is CTO at <a href="https://www.pinnacloid.com/">Pinnacloid</a>. He writes about the engineering decisions behind reliable SaaS, cloud, data, and AI systems.</p>
<p>Written by Syed Ebad Hussain, CTO at <a href="https://www.pinnacloid.com/">Pinnacloid</a>, and published by Syed ShahNawaz with the author’s permission.</p>
]]></content:encoded></item></channel></rss>