One Open Source Project a Day (No. 228): AX — Google's Open-Source 'Kubernetes for Agents,' Orchestrating Billions of Agent Tasks with Declarative YAML

Introduction

"Agents are a new kind of workload. They are neither stateless microservices nor run-to-completion batch jobs."

This is the 228th article in the "One Open Source Project a Day" series. Today's project is AX (repository: google/ax).

Once a team starts running AI agents at scale, they quickly hit an awkward fact: none of the existing scheduling infrastructure fits well. Kubernetes' Pod model assumes workloads are either stateless services or run-to-completion batch jobs, but agents accumulate state, need strict sandbox isolation, call out to model APIs and tool servers constantly, and can burn through a token budget in a loop the moment nobody's watching. Traditional CI/CD or batch platforms weren't designed for needs like "pause an agent and resume it exactly where it left off" or "SSH into the sandbox to see what it's actually doing."

AX is Google's open-source "high-throughput, declarative agent orchestration runtime," aimed at running billions of autonomous agent workloads in a single cluster. If you've used Kubernetes, the ax experience will feel very familiar — apply, get, describe, watch, delete, a kubectl-shaped command set, just with agent tasks as the scheduling object. It runs on top of Agent Substrate for sandboxed execution, and focuses purely on the declarative orchestration layer itself.

11,600+ Stars, 560 Forks, Apache-2.0 License, developed and led by Google. It's currently in Alpha — core concepts and protocols are still rapidly evolving, and the project explicitly warns of likely major breaking changes before a stable release.

What You Will Learn

• AX's three core primitives — Task, Workspace, Model — and what each solves
• Why agent workloads need a new scheduling model, neither a stateless service nor a batch job
• The layered relationship between AX and Agent Substrate: orchestration layer vs. sandbox execution layer
• The control-plane architecture: why AX stores state in Redis instead of directly as Kubernetes CRDs
• The sandbox lifecycle design behind ax suspend / ax resume / ax ssh

Prerequisites

• Familiarity with basic Kubernetes concepts (Pods, CRDs, kubectl habits)
• A basic understanding of containerized deployment and sandbox isolation
• Optional: a basic understanding of MCP (Model Context Protocol)

Project Background

What It Is

AX describes itself as "Google's open agentic orchestration runtime." The README opens with a tight operational description: "Declare an agentic task with workspaces and model specifications. AX sandboxes it, wires up its workspace, and helps running it at scale." That single sentence captures the project's division of labor — AX itself doesn't handle sandbox isolation or execution; that's delegated to the underlying Agent Substrate. AX focuses on declaratively defining tasks, scheduling at scale, and lifecycle management.

The project documentation directly explains why existing tools fall short: "Agents are a new kind of workload. They are neither stateless microservices nor run-to-completion batch jobs. They accumulate state, need strict isolation, call out to model APIs and tool servers, and can burn money in a loop if nobody is watching." AX's answer is three small primitives that can be declared, rather than one all-encompassing framework trying to model everything an agent does.

Team and Background

• Organization: google
• License: Apache License 2.0
• Primary language: Go
• Underlying substrate: Agent Substrate (sandboxed actor execution environment)
• Website: agentexecutor.io
• Status: Alpha — core concepts and protocols are still evolving, with explicit warnings of likely major breaking changes

Project Stats

• ⭐ GitHub Stars: 11,600+
• 🍴 Forks: 560
• 📄 License: Apache-2.0
• 🐛 Open Issues: 48
• 📅 Created: 2026-03

What It Does

The Problem It Solves

Running agent workloads on general-purpose scheduling platforms:
  Kubernetes Pod model → assumes workloads are stateless services or batch jobs
  ↑ agents accumulate state, need strict isolation, call external model APIs frequently
  ↑ no native support for "pause and resume in place"
  ↑ no debugging channel for "SSH in and see what the agent is doing"
  ↑ every run re-clones repos, re-wires tools, re-configures MCP servers — high cold-start cost
 
AX's approach:
  Three declarative primitives: Task (sandboxed execution unit)
                                + Workspace (a pre-warmed environment)
                                + Model (a cluster-level model configuration)
  ↓
  Declare the whole thing at once with ax apply -f task.yaml
  ↓
  ax watch / ax ssh / ax suspend / ax resume give K8s-style
  observability and lifecycle management
  ↑ sandbox isolation is delegated to Agent Substrate; AX focuses on the orchestration layer

Use Cases

1. Running autonomous coding/ops agents at scale

   • Each agent task runs as its own sandbox, declaring the Git repos, MCP servers, and tools it needs — fitting scenarios like "dispatch a separate agent to fix bugs across a fleet of repos"

2. Long-running, pausable agent workflows

   • Via ax suspend / ax resume, an agent's state can be checkpointed and resumed later in place, without needing to design custom logic for "how do I save resources on a long-running task"

3. Development scenarios needing to debug and observe actual agent behavior

   • ax ssh lets you go directly into a running sandbox to inspect the filesystem or run commands, instead of only being able to stare at log output

4. Multi-team scenarios needing centralized model credential and config management

   • Model is declared as a cluster-level resource, so rotating a key or switching model versions is a single ax apply instead of hunting through each agent's environment variables individually

Quick Start

Prerequisites:

# Requires a Kubernetes cluster with Agent Substrate already installed
kubectl get svc api -n ate-system   # verify Substrate's Control API is ready

Install the CLI:

go install github.com/google/ax/cmd/ax@latest

Deploy the control plane:

make deploy AX_IMAGE_REPO=<your-registry>

Declare a Workspace and Task in one YAML file:

# task.yaml
apiVersion: ax.io/v1alpha1
kind: Workspace
metadata:
  name: golang
spec:
  git:
    - repo: https://github.com/golang/go.git
      branch: "my-fix"
---
apiVersion: ax.io/v1alpha1
kind: Task
metadata:
  name: test
spec:
  workspaces:
    - name: golang
      goal: "Ensure that Go tool chain is available and is built from source"
  debug: true   # lets you `ax ssh` into the sandbox

Apply and observe:

ax apply -f task.yaml
ax watch task test
ax ssh test -- ls -al /workspace

Core Features

1. Three orthogonal declarative primitives

2. A kubectl-shaped CLI

ax apply, ax get, ax describe, ax watch, ax delete, plus agent-specific verbs like ax ssh, ax suspend, ax resume. It follows your active kubectx context — switch clusters and ax automatically resolves and tunnels to that cluster's control plane in the background.

3. Task's deliberately minimal design philosophy

Over its lifetime, an agent plans, delegates, retries, and fans work out. AX makes no attempt to model that shape. It gives you one primitive that's cheap to create, isolate, suspend, and throw away, and lets the agent compose as many of them as its work demands. A single Task may be the entire job, or the root of a large tree of tasks spawned as the agent breaks the problem down — either way, every node gets the same sandbox, the same lifecycle, the same tooling.

4. Goal-driven Workspace bootstrapping

A Workspace binding can carry a goal — a plain-language description of the environment the task needs. On first boot, the runner hands that goal to an agent that finishes the setup (say, installing a toolchain or dependencies), so the task's own command starts in an already-ready environment.

5. Cluster-level model resources

Declaring model configuration as a cluster resource, rather than scattered across each agent's environment variables, means rotating a key, pinning a new model version, or tuning generation parameters is a single ax apply. AX's own components — for example, the process that plans a Workspace from a goal — read Model resources too.

6. Suspend/resume and SSH debugging

ax suspend checkpoints actor state and pauses the task; ax resume picks it back up. ax ssh requires a task to explicitly set spec.debug: true before it will connect — because the underlying guest services grant arbitrary process execution and file access, keeping this off by default is a deliberate security decision.

Project Advantages

Why choose this project?

• Led by Google, already thinking through problems at the scale of "billions of agent tasks" — architectural decisions like choosing Redis over etcd are made specifically for that scale
• The operational mental model directly reuses Kubernetes experience, so kubectl users face near-zero onboarding cost
• A clear layered design — AX focuses purely on orchestration, while sandbox execution is delegated to Agent Substrate, keeping responsibilities cleanly separated

A Deeper Look

Why Task State Isn't Stored as a Kubernetes CRD

This is a particularly interesting architectural decision in AX. The project documentation states the reasoning directly: "Storing millions of short-lived tasks as Kubernetes CRDs pushes etcd past its comfort zone (single-digit GB storage limits, write-rate bottlenecks, control plane degradation)."

AX's solution is to store state in Redis, using Redis Streams as the work queue between the API server and a horizontally scaled pool of controllers:

ax apply -f task.yaml
        │
        ▼
    ax-server (stateless gRPC API)
        │
   store & publish event
        │
        ▼
      Redis (Task Hashes + Event Streams + PubSub)
        │
   XREADGROUP (consume the stream)
        │
        ▼
  ax-controller (horizontally scaled worker pool)
        │
      gRPC call
        │
        ▼
   Agent Substrate (sandboxed execution)

This choice shows the AX team never intended to box themselves into the "everything is a CRD" Kubernetes mental model — they only borrowed its operational experience (apply/get/watch), while swapping the underlying storage engine for something better suited to high-frequency, short-lived objects. It's a pragmatic tradeoff: the interaction pattern users perceive stays the same, but the backend is redesigned for the real scale requirement.

Task's "Small and Composable" Design

AX's philosophy around Task is worth unpacking further. Many orchestration systems try to model "a complete agent workflow" — defining DAGs, defining step dependencies. AX does the opposite: it refuses to model an agent's internal behavior (planning, delegating, retrying, decomposing), offering instead only an atomic, cheap execution unit and letting the agent decide how to compose it.

The upside of this design is flexibility — a single Task can be the entire job, or the root of an entire tree of tasks, and AX never needs to understand the shape of that tree; every node gets the same sandbox and lifecycle primitives. The tradeoff: dependencies and data flow between tasks have to be managed by the agent or a higher-level tool — AX doesn't provide built-in workflow-orchestration semantics.

Workspace's Goal-Driven Bootstrapping Mechanism

Workspace.spec.goal is a somewhat unusual design: rather than directly specifying "install these dependencies," it's a plain-language description of an environment goal (e.g., "ensure the Go toolchain is available and built from source"), handed to an agent at sandbox startup to interpret and execute.

This turns what would normally be mechanical work — writing setup scripts, writing Dockerfiles — into a dynamic process that can be described in natural language and completed by an agent. The project's roadmap shows this direction continuing to deepen: "Dynamic Agentic Environment Curation" plans to let this bootstrapping process automatically inspect repository contents, resolve toolchains, and discover relevant MCP servers and skills from registries, rather than just executing one pre-written goal description.

Sandbox Lifecycle: The Runner as a Persistent PID 1

Inside every task container, ax-task-runner starts as PID 1, taking on responsibilities well beyond simply "starting a command": it loads the Task and Workspace specs, starts a metadata and guest-management daemon on port 80, prepares each workspace in binding order on first run (cloning repos, setting up the skills path, and handing off to an agent if a goal is set), and only then starts spec.command as a supervised child process.

The key detail: the runner stays alive as PID 1 even after the command exits, meaning the metadata server keeps responding and ax ssh still works after the main command has finished. This makes it possible to debug an already-completed task, rather than having the whole container vanish the moment the task ends.

The Layered Relationship with Agent Substrate

AX doesn't handle sandbox isolation itself — that's fully delegated to Agent Substrate, a lower-level system responsible for atespace provisioning, actor creation and activation, and worker assignment. AX's controller drives tasks toward their desired state by calling Substrate's Control API over gRPC. This layering lets AX focus purely on "declarative orchestration semantics" and "scheduling at scale," leaving the harder, lower-level problem of "how do you safely isolate an untrusted process" to a dedicated project — which is also why AX's roadmap devotes so much space to how the Substrate actor architecture will evolve (migrating to the new actor API, splitting workspace initialization into a dedicated actor, minimally privileged policies, idleness detection with automatic suspension, and so on).

Project Links and Resources

Official Resources

• 🌟 GitHub: https://github.com/google/ax
• 🌐 Website: https://agentexecutor.io
• 📄 License: Apache License 2.0
• 🐛 Issue Tracker: GitHub Issues

Related Resources

• Agent Substrate — The sandboxed execution environment AX depends on
• Agent Substrate Guest Services — The process/filesystem services that power ax ssh
• Model Context Protocol — The tool-integration standard Workspaces can wire in

Summary

Key Takeaways

1. Three orthogonal primitives cover the core needs of agent orchestration: Task handles isolated execution, Workspace handles environment pre-warming, Model handles cluster-level configuration management
2. Reuses Kubernetes' operational mental model, but redesigns the storage backend: replacing etcd with Redis + Streams is a tradeoff tailored to a billions-of-short-lived-tasks target
3. Doesn't model agent behavior, only provides a cheap composable execution unit: Task is deliberately small, leaving decomposition, delegation, and retry logic to the agent itself
4. A clear layered architecture: AX focuses on declarative orchestration, delegating sandbox isolation to Agent Substrate, keeping responsibilities cleanly separated
5. Still in Alpha: core concepts and protocols are still rapidly evolving — watch for breaking changes before adopting it in production

Who This Is For

• Teams running autonomous agent workloads at scale: especially where task volume has already outgrown hand-managed scripts
• Teams already using Kubernetes and comfortable with its operational mental model: the kubectl-style experience means near-zero migration cost
• Scenarios needing to pause/resume long-running agents, or needing a debugging channel to observe actual agent behavior
• Developers willing to accept the risk of an Alpha-stage project in exchange for getting in early on the design and contributing

One-Line Verdict

AX isn't trying to reinvent how agents should "think" — it's first solving the infrastructure problem of how to run this new, state-accumulating workload safely and at scale in a cluster.

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage