Back to Tech Blog Insights
2026-11-03 • 4 min read • QA Automation & Kubernetes

SyncEta Autoscaling in Practice (1) — When Should We Scale Out Workers?

SyncEta (ETA) provides a scheduled execution feature that runs E2E test scenarios automatically at designated times.

Under normal day-to-day traffic, a fixed worker pool handles scheduled runs smoothly. However, during quarterly release audits or major regression windows, hundreds of test scenarios are triggered simultaneously. When worker capacity is insufficient, scenarios pile up in the waiting queue.

Should an operator manually scale up the Kubernetes Pod count every time a burst arrives and scale it back down afterward?

Operating an autonomous test automation platform while relying on manual worker scaling is simply unacceptable.

Why Not Just Use Standard Kubernetes HPA?

Kubernetes includes HPA (Horizontal Pod Autoscaler), which adjusts Pod replicas automatically based on resource metrics such as CPU or memory utilization.

At first glance, applying standard HPA to ETA worker Pods seemed straightforward. Looking closely at how an ETA worker actually executes a scenario, however, revealed a fundamental mismatch.

Each ETA worker Pod executes exactly one test scenario at a time. Inside that single Pod, a headless Chromium browser, the worker process, and the Playwright MCP server run together. Executing even a single scenario consumes substantial CPU and memory.

In other words, whenever a worker Pod is actively running a scenario, it utilizes nearly 100% of its allocated CPU and memory.
If we scale Pods based on resource utilization, HPA could trigger a scale-out even when only a single scenario is running and zero jobs are waiting in the queue.

Ultimately, what matters far more than CPU or memory utilization is how many scenarios are waiting in the queue to be executed.

Scaling Directly on Redis Queue Depth

The scheduled execution pipeline in SyncEta follows a clean queue-driven flow:

[WAS]
  User registers Cron schedule
            ↓
  Enqueues Job into Redis Queue at trigger time
            ↓
[Redis Queue]
  Pending Jobs waiting for execution
            ↓
[Worker Pod]
  Dequeues Job and runs Playwright MCP scenario

Since pending jobs already accumulate in Redis, why not use the Redis queue length directly as our autoscaling signal?

For example, if all active workers are busy and 10 jobs are waiting in Redis, the cluster can spin up additional worker Pods proportional to the backlog—bounded by a configured maximum Pod limit to protect cluster capacity.

The key engineering question was how to feed Redis Queue depth into Kubernetes autoscaling instead of standard CPU/memory metrics.

Enter KEDA (Kubernetes Event-driven Autoscaling)

This is exactly the problem KEDA (Kubernetes Event-driven Autoscaling) was designed to solve.

KEDA extends Kubernetes autoscaling beyond resource metrics, allowing workloads to scale based on external event sources and queue metrics from Redis, Kafka, RabbitMQ, and more.

We decided to drive ETA worker Pod scaling using the pending job count in the Redis Queue via a KEDA ScaledObject.

Of course, adopting KEDA with Redis requires more than a single YAML flag. Depending on whether the queue is implemented with Redis Lists, Streams, or sorted structures, the exact KEDA scaler type and queue-length inspection query must match the underlying queue semantics.

Why Scaling In Is Harder Than Scaling Out

Scaling out is straightforward: when pending jobs increase, add worker Pods.

What happens, however, when the Redis queue length drops back to 0? Can we immediately terminate extra worker Pods?

Not immediately. Even when the Redis queue is empty, workers that already dequeued a job may still be in the middle of running a multi-minute browser scenario. If Kubernetes terminates that Pod prematurely, the Chromium browser and Playwright MCP server shut down mid-flight, aborting a valid test run.

Therefore, queue-length autoscaling must be paired with graceful termination hooks (terminationGracePeriodSeconds / preStop lifecycle coordination) and safe scale-in stabilization policies that protect in-flight test executions.

What We Are Validating Next

Our engineering objective is clear:

Automatically scale out worker Pods when pending scenarios accumulate in Redis, and safely scale in Pods only after in-flight browser scenarios finish cleanly.

In Part 2, we will walk through our exact Redis Queue structure, KEDA ScaledObject configuration, maximum replica guardrails, and graceful scale-in lifecycle policies in production.