Opening: The System that Looked Scalable

In the age of microservices and fast-building startup nations, it’s no secret that, a lot of the time, the founding crew will make certain decisions that later prove cumbersome to the organisation as a whole. In the rush to deliver fast and sell fast, some architectural decisions and considerations are pushed to the backlog, until the product is profitable enough to justify wrestling with these problems.

One such organisation I was contracted to work at had a single-page application with a serverless architecture that mostly relied on GCP’s Cloud Functions. Traffic was, at the time, small enough to justify the cold starts of such systems, and with scalability in mind, they knew they could scale those functions—since GCP allowed up to 1,000 instances. Since we had only Gen 1 Cloud Functions back then, that also meant we had a 1:1 ratio of instances to concurrent requests—so we could only scale up to 1,000 concurrent requests.

On paper, that looked highly scalable, especially when each function could be given more computational power and process requests more efficiently. But due to the implementation speed, they had neglected some very important things in their main business flows—which reduced their ability to handle congestion at peak times, despite the initial impression.

*This was done toward the end of 1st-gen Cloud Functions and the start of 2nd gen, around 2022.

The Core Problem: Congestion is not just “Too Much Traffic”

Congestion happens when messages enter the system faster than they can be processed safely. The message queue grows, retries increase, downstream services slow down, and the system starts spending more time (and money) trying to recover from failure rather than doing useful work.

In event-driven systems, the dangerous failure mode is not necessarily a crash (since messages tend to be preserved even during a crash, though that depends on your GCP configuration—whether it has automatic re-delivery, a dead-letter queue, or whether it just drops the message), but rather a retry loop that goes on infinitely: a message fails, gets retried, fails again, consumes processing capacity, blocks newer work, and hides the real bottleneck behind a wall of noise.

In this case, the messages kept retrying until they reached their retry threshold and were then dropped, resulting in an awful user experience during peak times—which drove quite a lot of clients away. (…and in a handsome GCP bill later on, since they all scaled to max and kept retrying to failure.)

Why Heavy-Traffic Cloud Functions Can Become a Problem

This doesn’t mean that using cloud functions is the wrong tool by default; they are incredibly good and useful for a small-to-medium startup to rely on. They’re very good for isolated tasks, and can be very efficient when they are developed properly with an orchestration/workflow tool, and when workloads have clear execution boundaries.

The initial reasoning for using them as a potentially high-traffic service—especially one that touches databases, external APIs, or expensive business logic—may need a lot of careful consideration and strategies to avoid it going out of control.

Cold starts and startup overhead under bursty load

When a system that relies heavily on cloud functions experiences a sudden spike of messages, a few issues can occur.

Firstly, the burst causes the function to scale (up to 1,000 instances concurrently) and, along with cold starts, may cause latency issues.

Also, consider the number of actions during a cold start and the language you are using for that particular function, as they may affect its load time.

Consider heavy use of external services

Let’s also examine the use of external services in a single cloud function. Let’s say it relies on multiple external services and therefore requires several async actions to execute properly and return an answer. Each of those is a critical breaking point that could potentially send the function into a spiral of errors if not handled properly. And we’re not necessarily talking about one of those services failing—but what about a timeout? What about a payload so heavy that it takes so much time that it runs beyond 9 minutes, resulting in a missing ACK and triggering the Pub/Sub retry, consuming more resources and costing the organisation more money until the retry exhausts itself, causing an awful user experience?

Limited worker control vs. a dedicated service

In a cloud function, our unit of control is one invocation. Pub/Sub delivers an event, the function runs, and you either finish successfully or fail/timeout. You don’t really need to control the subscriber loop, local worker pool, batching behavior, or detailed flow control.

In a dedicated service, you can say:

1. Pull 10 messages
2. Process max 3 concurrently
3. Extend ack deadline while processing
4. Ack only after DB + external services + Kafka succeeds
5. Nack or DLQ after known failure
6. Pause pulling when CPU/external API is saturated

Too many parallel invocations can overload the database or external dependency

If Pub/Sub receives many messages, it can trigger many function invocations in parallel (the scaling we mentioned earlier). That sounds good for throughput, but each invocation may open DB connections, call external APIs, consume memory/CPU, and retry on failure. Suddenly, our “scalable” serverless layer becomes a pressure amplifier.

If we exhaust our DB connection pool or hit an external API rate limit, then we increase function timeouts as invocations wait for DB workers to be available or for the API rate limit to reset. Our system, since it uses an at-least-once delivery model, keeps retrying the messages; even more invocations happen later, and we enter a giant retry storm that consumes our resources and budget.

This is why worker control matters. In a dedicated worker/service, we can add backpressure:

1. Process max 20 messages at a time
2. Use one shared DB connection pool
3. Rate-limit external API calls to 50/min
4. Pause pulling from Pub/Sub if DB latency increases beyond x
5. Ack only after successful processing
6. DLQ after repeated failures

One poison message can consume massive resources

A poison message is a message that will keep failing no matter how many times we retry it.

{
"customerId": null,
"action": "charge_customer",
"amount": 500
}

If our function expects customerId ****and always crashes when it’s missing, retrying doesn’t help.

Without a dead-letter strategy, this can happen:

1. Message arrives
2. Function fails
3. Pub/Sub retries
4. Function fails again
5. Retries again
6. Logs fill with the same error
7. Alerts fire repeatedly
8. Healthy messages may be delayed or mixed with noisy logs/failures

A poison message may not literally block every newer event, but repeated delivery consumes worker capacity and operational attention. Under bounded concurrency—or when ordering is enabled—it can also delay otherwise healthy work.

A dead-letter topic/queue solves this by saying:

1. Try this message N times
2. If it still fails, move it aside
3. Continue processing the rest

The Failure Story: When the Queue starts working against you

Our failure story is a mix of the above.

During peak times, one of the main business flows in charge of registering users to a subscription service started to experience very high latency due to intense load. The small-to-medium scaling strategy (~100 instances) had proven successful during the first years of the product, yet when it encountered concurrent spikes like that—consumer capacity scaled independently of downstream capacity.

While previous engineers tried to use a batching strategy to reduce the burst load of messages arriving to the consumer, it, too, failed to produce results, resulting in timeouts, endless retry loops, and messages that weren’t processed, nor were they submitted to a DLT.

Debugging this was a nightmare, since the looping retries caused identical error messages that spammed the Log Explorer endlessly until the cloud function had timed out (~9 mins, Gen 1).

Separate Load Problems From Failure Problems

First principle: separate load problems from failure problems.

Because they look similar from the outside, but require different fixes.

A load problem means:

The system is receiving more work than it can process safely.

Examples:

Too many Pub/Sub messages
Too many parallel Cloud Function invocations
Database connection pool exhausted
External API rate limit reached
Processing latency rising under normal valid messages

The fix is usually flow control:

backpressure
rate limits
max concurrency
max instances
batching
worker pool size
queue depth monitoring
autoscaling carefully

A failure problem means:

Some messages cannot be processed successfully, even if the system has enough capacity.

Examples:

Invalid payload schema
Missing customerId
External service returns permanent 400
Code bug crashes on one data shape
Duplicate message causes unique constraint violation
Poison message retries forever

The fix is usually failure handling:

validation
idempotency
retry classification
max attempts
dead-letter topic
alerting with correlationId
manual replay tools

The danger is mixing them up.

If you treat a failure problem like a load problem, you scale more workers, but the poison message still fails.

Bad payload fails with 10 workers
Bad payload fails with 100 workers
Now you just have more logs, more retries, more cost

If you treat a load problem like a failure problem, you send too much to DLQ or over-alert, but the real issue is that the downstream dependency is saturated.

Valid messages fail because DB is overloaded
You DLQ them
But the root cause was uncontrolled concurrency

Problem	Symptom	Wrong Reaction	Better Reaction
Too much traffic	Backlog grows, workers healthy	Add infinite consumers	Add controlled scaling and rate limits
Slow dependency	Timeouts, high latency	Retry aggressively	Reduce concurrency, add circuit breaker
Poison message	Same message fails repeatedly	Keep retrying	Send to dead-letter topic
Bad deployment	Failures start after release	Scale more	Roll back or disable consumer
Hot path overloaded	Customer data fails to load	Add more async jobs	Move critical path to dedicated service

Solution Pattern 1: Dead-Letter Topics and Retry Limits

A retry policy should not be an act of faith. It should have boundaries.

Retries are useful when the failure is temporary

1. A network timeout
2. A slow dependancy
3. A momentary database issue
4. A transiant 5xx response

They are harmful when

1. The message is invalid
2. The handler has a bug
3. Downstream service is rejecting the request permenantly

The first improvement is to classify failures:

Failure type	Example	Correct action
Transient	DB timeout, external API 503	Retry with backoff, Isolate.
Permanent	Invalid schema, missing customerId, external API 400	Do not retry blindly
Unknown	Unexpected exception	Retry a limited number of times, then isolate
Duplicate	Same event delivered twice	Handle idempotently

Retry with Backoff

Retrying several times with the wait-time interval is increased between each retry, then isolate if it failed.

Retry a limited number of times, then isolate

While certainly we would like to keep a boundary on retries at all times, this becomes especially important for this error type. But not only that, we would like to also isolate the error by moving it onward to a Dead-Letter Topic, or a failed-events table / quarantine topic, to be resolved later on.

Isolation mechanism	Use case
Dead-letter topic / queue	Standard event-driven failure isolation
Failed-events database table	Good when support/admin tools need visibility
Object storage file	Good for large payloads or offline investigation
Manual review queue	Good when business decision is needed
Replay topic	Good when you want to fix the bug and reprocess later

For Pub/Sub, the dead-letter policy belongs to the subscription, not the topic. The system should set a maximum delivery attempt count, route failed messages to a dead-letter topic, and attach a separate subscription to that topic for investigation and replay.

A dead-letter topic is not where messages go to die. It is where the system admits: this message requires human or specialized handling, and it should not block the main flow—allowing us to continue processing other healthy messages.

Putting boundaries on retries is so important because, regardless of the isolation mechanism we have chosen, we avoid spamming our logs endlessly. We can separate each error into its relevant isolation mechanism and track errors and issues by using metrics on each isolation mechanism, alerting the team to a growing issue or concern.

I love this separation of concern, since I don’t have to use the entire Log Explorer as my data sink and design queries around that—though I can, and it is indeed useful. It is much easier to produce high-quality metrics for observability when there is a solid isolation mechanism in place.

---

The handler should be idempotent. In an at-least-once delivery system, the same message may be delivered more than once. This means the handler must be able to safely process the same event twice without double-charging, double-registering, or corrupting state.

Idempotency

The property of a certain operation, if repeated several times - will yield exactly the same result.

Especially in distributed systems, where numerous components may fail and we will inevitably find ourselves retrying the operation - Idempotency allows us to ensure that we avoid duplication of operations, computations, and data itself

Solution Pattern 2: Backpressure and Controlled Consumption

If your consumers can scale faster than your database, then your consumers are not scaling the system. They are scaling the damage.

What is backpressure?

Backpressure means that the consumer (in our case, the cloud function) is allowed to slow-down when the down stream system is saturated. Without it, a traffic spike creates more function invocations, more database connections, more external API calls, more timeouts, and then more retries.

The queue becomes a pressure amplifier.

---

For our use-case, I will examine PostgreSQL as the downstream system, since it was used in the system with the said problem.

Traffic spike → Pub/Sub delivers more messages → More function invocations start → Each invocation opens/uses DB connections → PostgreSQL connection slots or CPU saturate → queries slow down → functions timeout → messages retry → even more invocations later → retry storm

What I learned about PostgreSQL

In postgres, max_connections defines how many concurrent client connections the database server allows. PostgreSQL’s default is usually 100, and some slots are reserved for superusers/admin use.

It also warns us in its docs, that increasing this value will increase resource allocation, including shared memory usage in PostgresSQL docs.

→ Why should we care about this warning?

Because max_connections is not “more throughput”, but rather its “more doors to the same room”.

You let more workers in at once, the database still has the same CPU, RAM, Disk I/O, locks, indexes and query plans.

So if you raised the max_connections from 100 to 500, you didn’t make it 5x stronger, you allowed 5x more clients to compete.

Why that matters…

Problem	What happens
More memory overhead	Each connection/backend has overhead, and active queries can use additional memory
More CPU contention	More active queries fight for the same CPU cores
More context switching	The OS spends more time juggling work
More lock contention	Queries block each other more often
Worse cache behavior	More concurrent work can reduce locality and increase I/O pressure
Higher latency	Individual queries wait longer
Cascading failures	App workers timeout, retry, and create even more pressure

The Core Formula

Total possible DB connections = max service instances × connection pool size per instance

Our Example:

100 Cloud Function instances × 10 DB connections per instance = 1,000 possible DB connections

During peak periods, PostgreSQL CPU approached 80% while query latency, connection waits, and function timeouts increased. Together, these signals indicated that the database had become the system bottleneck.

Lets review the cascading actions the organisation took in order to handle this error, and how we eventually handled it after reviewing the issue.

1. Database CPU was stressed towards 80%
2. DevOps team used kube to scale it further, horizontally only. Since no permission was given to allocate further resources to the database.
3. No database replicas were used, same database was used for all CRUD operations.
4. The resulting saturation resulted in high latency in almost all operations during high-peak hours(~2-3hours total) and resulted in mass noise in observability.

And this damage, was coming from a single business-flow that was not handling the congestion properly! Imagine if it occured accross the system.

If PostgreSQL has max_connections = 100, this is catastrophic. The database will not politely “handle it.” New connections will start failing, latency will rise, and the application may produce cascading timeouts.

Which is exactly what happened, without a backpressure strategy in place.

What does backpressure means for us, in our example?

1. Limit Cloud Function max instances
2. Limit Cloud Run max instances
3. Limit concurrency per instance
4. Use small DB pools
5. Use Pub/Sub pull workers with flow control
6. Rate-limit external API calls
7. Pause or slow consumption when DB latency rises
8. Use exponential backoff instead of immediate retry storms

But hey! what if I absolutely have to process so much data in this system? What can I do to make sure that I handle this load?

Lets talk about Little’s Law..(Or watch this video here..)

Little’s Law estimates average in-flight work. At 1,000 events per second and an average processing time of 100 milliseconds, the system will have roughly 100 events in progress on average. If processing time rises to two seconds, average in-flight work rises to roughly 2,000 events—even though the arrival rate has not changed.

Required Concurrency = (arrival rate) x (processing time)

Such that,

A - 1000 messages/second

B - 100 milliseconds or 0.1 seconds, processing time

C - concurrent in-flight messages

$$(1,000) x (0.1) = 100$$

But if each message takes 2 seconds to process, we can see that..

$$(1000) x 2 = 2000$$

So now we are dealing with 2,000 in-flight concurrent messages rather than 100. A system that can handle 1000 messages/second with 100ms handlers may collapse at the same input rate if handlers become slow because of database locks, inefficient queries, external API calls or large payloads.

If a system must process a large amount of data, the first optimization is often not more workers but less work per message.

Batching the work instead of processing every event alone

If every message causes:

1. open transaction
2. insert one row
3. commit
4. ack message

We are paying in worker overhead for every tiny unit of work.

For high-volume systems, prefer

1. read N messages
2. validate them
3. group them
4. write them in one batch
5. ack after durable write

So we move from:

1 Pub/Sub message → 1 function invocation → 1 DB transaction → 1 ack

Pub/Sub topic → Cloud Function per message → PostgreSQL write per message

To:

N Pub/Sub messages → 1 worker cycle → 1/few DB transactions → N acks

Pub/Sub topic → pull-based worker, usually Cloud Run or GKE → flow control: max outstanding messages / bytes → local batch buffer → validate messages → bulk write to PostgreSQL → ack successful messages → isolate poison messages

Needless to say, that batching in itself requires careful handling since large batches increase latency and one invalid message can cause the entire transaction to fail.

Also, longer processing times increase re-delivery risk since Pub/Sub can redeliver messages when they are not acknowledged before the ack deadline expires, therefore we use flow control for high latency subscribers so that we can have more granular control over it.

Solution Pattern 3: Promote Heavy Functions Into Services

One of the key changes that came later on, was the promotion of these heavy functions into full-fledged services that can handle that traffic. One might ask, why wasn’t that the first go-to during the initial development process? Well.. such is the case when teams are delivering a lean MVP with the intention to sell, they have to make decisions and some flows that may seem redundant to promote into services can prove to be not redundant at all and highly valuable later on, once the start-up has sold its product to a corporation.

It is precisely in that space, that our team was brought on-board in order to “flatten” the MVP folds and provide a full enterprise product that still works till this day.

And while cloud functions can be scaled appropriatly, as they initially were as a response to the problem — they do not allow for much granular control over the worker behavior, and it is precisely why we might want to turn it into a service on our backend and process it there.

Much of the architectural decision making that our team was executing regularly, took into consideriation the pros and cons in terms of how much time will be spent resolving an issue, in proportions to the value. Since the product had great revenue, the issue was not monetization but rather optimization of particular parts of the business, to a certain extent.

Sure, we could rewrite everything as their own services and remove a lot of the business flow logic from the cloud functions (and perhaps save the organization some more money), but other more painful issues were at stake — that proved to have more impact on the revenue and daily work of the teams surrounding that product, and so, many of the refactors were dropped in favor of enabling more teams and more features that at the end of the day, provided better customer experience.

It is easy to get into optimizing everything, but we must always remember that we are building for users, and the users dont care about your architecture — they care about themselves and their user experience.

Observability: Know Whether You Are Congested

One of the best decisions that I took on early for the product was to develop a Observabillity dashboard that provided great insights into our flows and metrics about the business.

The initial pilot was for what we needed as an engineering team, errors, logs, push notifications to slack. Everything we needed to handle incidents and bugs. It was such a game changer that it was immedietly embraced by our QA team and business dep. wanted more, so we gave them more.

Beyond our own dashboards, that tracked everything from CPU usage on our DB to little bugs and errors on the frontend — we gave the business department their own dashboards to track sales, leads, marketing campaign engagement, etc.

One of the best decision you can take when you are done with your MVP development and started to gain some revenue (not necessarily, great revenue.), is to create great observability across your application and especially, around the external API calls on an event-driven system.

It will save you lots of headache and time lost, and will notify you whenever something is going wrong before it turns into a cluster of errors and cause damage to your revenue. But for some reason, I see a lot of people neglecting that part or only relying on logs, without metrics or traces. Even the visualization is so important, because you can get so much information in 5 minutes as compared to an hour going through the logs and errors, or notifications, and worry if they were sent or not sent or whatever other errors your logging system may or may not have.

What I actually changed?

Lets go step-by-step what we did and how it contributed to solving the issues.

Push-triggered function with uncontrolled instance growth

Without controlling the instances properly and allowing them to scale up without resolving downstream capacity, we just amplified the problem downstream. By bounding worker concurrency we did not allow it to grow to 1000 concurrent requests, letting our database breathe and process requests at a rate that was congruent with our allocated resources.

No quarantine strategy

I configured a dead-letter topic so repeatedly failing messages were isolated after the subscription exhausted its delivery-attempt budget. This allowed us to resolve the messages on a later date with a scheduled job and present a proper message to the user, improving his experience and application trust.

What I would change today?

Use a controlled pull subscriber

If I were redesigning this flow today, I would replace the direct Gen 1 Pub/Sub trigger with a pull-based subscriber running as a dedicated worker.

With the original design, Pub/Sub pushed one message into one Cloud Function invocation. The platform decided how many instances to start, while each invocation independently competed for PostgreSQL connections and external-service capacity.

A pull subscriber reverses that relationship. Instead of Pub/Sub deciding how quickly to invoke the consumer, the worker requests messages and controls how much work it accepts. Pub/Sub client libraries support flow-control limits for outstanding messages and bytes, allowing the consumer to bound the amount of work held in memory at one time.

The redesigned flow would look like this:

Pub/Sub pull subscription
→ long-running subscriber worker
→ accept no more than the configured outstanding-message limit
→ collect a bounded micro-batch
→ validate each event
→ quarantine permanently invalid events
→ process valid events in one or more database transactions
→ commit
→ acknowledge only the messages whose work was committed

The worker could run on a platform designed for continuous background processing, such as a Cloud Run worker pool, GKE, or a virtual machine. Cloud Run worker pools are specifically intended for non-request workloads such as applications that continuously pull work from Pub/Sub.

Use bounded micro-batches

The worker should not wait indefinitely for exactly 100 messages. During quiet periods, that would introduce unnecessary latency.

Instead, it should flush a batch when either of two limits is reached:

up to 100 messages
OR
200 milliseconds since the first message entered the batch

The exact limits should be measured against database capacity, message size, acceptable customer latency, and transaction duration.

This produces predictable behaviour under both low and high traffic:

Low traffic:
8 messages arrive in 200 ms
→ process 8

High traffic:
100 messages arrive in 20 ms
→ process 100

Validate before opening the transaction

Every message should first be classified independently.

Valid event
→ include in the database batch

Permanently invalid event
→ quarantine with its failure reason

Potentially transient failure
→ leave unacknowledged or negatively acknowledge for retry

Schema validation protects the transaction from malformed payloads, but it cannot prevent every database failure. Valid events may still encounter uniqueness conflicts, lock timeouts, foreign-key violations, or serialization failures.

For that reason, I would not automatically place every collected message into one large transaction.

If the messages are independent user operations, I would divide them into small transactional sub-batches or use a bulk statement whose conflict behaviour is explicitly defined:

100 messages collected
→ 3 invalid messages quarantined
→ 97 valid messages
→ process in bounded sub-batches
→ acknowledge each successfully committed sub-batch

This preserves most of the efficiency of batching without allowing one user’s failure to roll back every unrelated user operation.

Batch for efficiency, not because transactions require it

A Gen 1 Cloud Function can already open a PostgreSQL transaction while processing one message. Moving to a pull subscriber is therefore not required merely to use transactions.

The benefit is control:

• control over how many messages are in flight;
• control over worker concurrency;
• control over database connection usage;
• control over batch size and flush timing;
• control over acknowledgement timing; and
• control over how malformed and transiently failing messages are handled.

Batching can reduce network round trips, transaction commits, and per-message database overhead. It should not redefine the correctness boundary unless the events genuinely belong to one atomic business operation.

Alice’s subscription should not normally roll back because Bob’s subscription failed.

Commit before acknowledging

Messages should be acknowledged only after their corresponding database work has committed successfully. Pub/Sub redelivers messages that are not acknowledged within the applicable acknowledgement window, so acknowledging before the durable write could lose work if the worker crashes afterward.

However, committing first creates another failure window:

database commit succeeds
→ worker crashes before acknowledgement succeeds
→ Pub/Sub redelivers the message

The consumer must therefore remain idempotent. Each event should carry a stable identifier enforced through a unique constraint, processed-events table, or equivalent business-level idempotency mechanism.

The safe guarantee is not:

This event will be delivered only once.

It is:

This event may be delivered more than once,
but its intended business effect will be applied once.

Pub/Sub offers exactly-once delivery for pull subscriptions, but it introduces additional coordination and latency and does not remove the need to reason carefully about application side effects.

What this changes architecturally

The original design allowed compute capacity to grow independently of downstream capacity:

More messages
→ more function instances
→ more database connections
→ more contention

The pull-subscriber design introduces an admission boundary:

More messages
→ backlog grows temporarily in Pub/Sub
→ subscriber accepts only bounded work
→ PostgreSQL remains within its connection and concurrency budget

The queue now absorbs the traffic burst instead of immediately translating it into database pressure.

This does not increase the database’s maximum throughput. It makes overload controlled and recoverable. If traffic remains above processing capacity for an extended period, the team must still optimise the workload, increase downstream capacity, or add carefully budgeted subscriber instances.

The essential improvement is that scaling becomes deliberate rather than automatic:

The consumer pulls work at the rate the system can safely complete it, not at the rate the queue can deliver it.