# Kata Containers Isolation Tier

Run agents in an environment under [Kata Containers](https://katacontainers.io/) — each agent boots in a lightweight QEMU/KVM virtual machine with its **own guest kernel**, so there is no shared kernel with the host at all. This is the strongest isolation tier, intended for the most untrusted or multi-tenant agent workloads.

Read the [gVisor isolation tier guide](/agent-platform/docs/v1.0.0/guides/isolation-tiers/gvisor/.md) first if you haven't — Kata uses the same environment-level mechanism and the same dedicated-node model; this page covers what is Kata-specific.

Hard requirement: KVM (nested virtualization)

Kata boots a real VM per agent, so the node **must expose `/dev/kvm`**:

* ✅ Bare-metal Linux with VT-x (Intel) or AMD-V
* ✅ A nested-virtualization-capable cloud VM — for example GCP **Intel N2** with `--enable-nested-virtualization`, or AWS EKS bare-metal (`*.metal`) instances
* ❌ macOS / Colima — no nested virtualization; use gVisor locally instead
* ❌ GCP E2, AMD N2D, ARM T2A machine types, and Container-Optimized OS (read-only filesystem)
* ❌ k3d / kind **node containers** — even on a KVM-capable host, the Kata guest-agent handshake cannot complete inside a container-based node; Kata needs a real Linux node

Verify on the node itself, not on your workstation. From a machine with `kubectl` access, `kubectl debug` runs the check on the node:

```
kubectl debug node/<node-name> -it --profile=sysadmin --image=busybox:1.36 -- \

  chroot /host sh -c 'ls -l /dev/kvm; cat /sys/module/kvm_intel/parameters/nested'

# expect: a /dev/kvm character device, then Y (Intel)
```

This leaves a completed `node-debugger-<node-name>-…` pod in the current namespace; delete it afterwards. With SSH access to the node, run the two commands there instead.

## How It Works[​](#how-it-works "Direct link to How It Works")

Identical to gVisor, with one naming detail: the isolation tier is called `kata`, but the RuntimeClass (and containerd handler) it maps to is **`kata-qemu`** — that is the handler kata-deploy registers.

```
Environment (isolation tier: kata)

  └─ deploy / promote → runtimeClassName: kata-qemu

       └─ RuntimeClass "kata-qemu" scheduling → pod on the Kata node, QEMU/KVM VM + guest kernel
```

Like gVisor, Kata uses a **dedicated node** labeled `kata=true` and tainted `kata=true:NoSchedule`, so existing runc (and gVisor) agents are never touched — no downtime.

## Requirements[​](#requirements "Direct link to Requirements")

| Requirement              | Details                                                                               |
| ------------------------ | ------------------------------------------------------------------------------------- |
| A new (empty) Linux node | Joined to the same Kubernetes cluster; dedicated to Kata agents                       |
| Virtualization           | `/dev/kvm` present and functional (see warning above)                                 |
| Architecture             | x86\_64                                                                               |
| OS                       | Ubuntu (or similar) with systemd + containerd — **not** Container-Optimized OS        |
| Access                   | kubectl access to the cluster (the installer runs from your machine, not on the node) |

## Set Up a Kata Node[​](#set-up-a-kata-node "Direct link to Set Up a Kata Node")

`${AMP_DIST}`

The commands below run scripts and manifests from the release bundle. If you do not already have it extracted, fetch it first and point `AMP_DIST` at it — the same bundle the install guides use:

```
export VERSION="1.0.0"

curl -fsSL -O "https://github.com/wso2/agent-manager/releases/download/amp/v${VERSION}/wso2-agent-manager-${VERSION}.tar.gz"

tar xzf "wso2-agent-manager-${VERSION}.tar.gz"

export AMP_DIST="$PWD"
```

On a node-side step, extract it on that node: the installer script runs there, not on your workstation.

### Step 1: Add a new node to the cluster[​](#step-1-add-a-new-node-to-the-cluster "Direct link to Step 1: Add a new node to the cluster")

Provision a KVM-capable Linux node matching the requirements above and join it to your cluster using your platform's standard mechanism. Leave it empty.

Taint the node as early as possible

If your platform lets you set taints at node-pool creation, apply `kata=true:NoSchedule` there. An untainted node starts accepting regular platform pods immediately. The containerd reconfiguration restarts the node's container runtime; running pods normally survive that, but if the node has already picked up workloads you can `kubectl cordon` + `kubectl drain --ignore-daemonsets --delete-emptydir-data` it first.

### Step 2: Run the installer from a machine with kubectl access[​](#step-2-run-the-installer-from-a-machine-with-kubectl-access "Direct link to Step 2: Run the installer from a machine with kubectl access")

Unlike the gVisor installer (which runs on the node), the Kata installer runs from **any machine with kubectl access** and does everything through the Kubernetes API:

```
curl -fsSL https://raw.githubusercontent.com/wso2/agent-manager/amp/v1.0.0/deployments/setup/install-kata.sh \

  -o install-kata.sh



KATA_NODES="<node-name>" bash install-kata.sh
```

What it does:

1. **Pre-flights `/dev/kvm`** on every target node (via a short-lived privileged pod) and aborts if any node lacks it.
2. Labels the node(s) `kata=true`.
3. Applies the upstream **kata-deploy** DaemonSet, with a nodeSelector and toleration injected **before** apply — so it only ever lands on the labeled node(s); your other nodes are never reconfigured.
4. **Auto-wires k3s nodes**: k3s bundles its own containerd (configured only via `config.toml.tmpl`, read at startup), which upstream kata-deploy does not handle — the installer detects k3s nodes and patches the template + restarts the k3s agent for you.
5. Applies the `kata-qemu` RuntimeClass and taints the node(s) `kata=true:NoSchedule`.
6. Patches the Fluent Bit log DaemonSet to tolerate the taint, so agent logs are collected.

Make the Fluent Bit toleration durable

The v1.0.0 installer adds the toleration by patching the DaemonSet, and a patch is lost whenever the DaemonSet is recreated (a reinstall of the logs module or a namespace teardown). Fluent Bit is rendered by the `observability-logs-opensearch` Helm chart, so set the toleration as a chart value as well, passing the chart version that is already installed (the `CHART` column of `helm list -n openchoreo-observability-plane`):

```
helm upgrade observability-logs-opensearch \

  oci://ghcr.io/openchoreo/helm-charts/observability-logs-opensearch \

  --namespace openchoreo-observability-plane \

  --version 0.5.3 \

  --reuse-values \

  --set "fluent-bit.tolerations[0].operator=Exists"
```

Verify

```
kubectl get runtimeclass kata-qemu

kubectl -n kube-system rollout status daemonset/kata-deploy

kubectl get node <node-name> --show-labels   # kata=true
```

Keep the Fluent Bit toleration on later upgrades

The toleration is stored in the `observability-logs-opensearch` release's values, so it must be present on every upgrade that re-renders the Fluent Bit DaemonSet. An upgrade with `--reset-values`, or one that passes new `--set`/`-f` values without `--reuse-values`, drops it unless you pass `--set "fluent-bit.tolerations[0].operator=Exists"` again. Agents on the Kata node then stop producing logs with no error.

RKE2 nodes (bundled containerd)

Upstream kata-deploy assumes a standalone containerd at `/etc/containerd/config.toml`, which RKE2 nodes do not have. Two extra steps are needed on each RKE2 Kata node. Run them on the node itself, over SSH or in a `kubectl debug node/<node-name> -it --profile=sysadmin --image=busybox:1.36 -- chroot /host sh` shell (there, drop `sudo`):

**1. Before running the installer**, create an empty stub so the kata-deploy pod gets past its pre-flight check (it crash-loops with `cp: can't stat '/etc/containerd/config.toml'` otherwise):

```
sudo mkdir -p /etc/containerd && sudo touch /etc/containerd/config.toml
```

**2. After kata-deploy has installed the binaries into `/opt/kata`** (the node gets the `katacontainers.io/kata-runtime=true` label on success), wire the runtime through RKE2's containerd template — kata-deploy wrote its config to the stub, which RKE2 never reads:

```
sudo mkdir -p /usr/local/bin

sudo ln -sf /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-qemu-v2

sudo tee /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl > /dev/null <<'EOF'

{{ template "base" . }}



[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]

  runtime_type = "io.containerd.kata-qemu.v2"

  privileged_without_host_devices = true

  pod_annotations = ["io.katacontainers.*"]

  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]

    ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"

EOF

sudo systemctl restart rke2-agent
```

Rewrite, never append

The template must contain `{{ template "base" . }}` exactly once. If the file already exists (e.g. it also carries a gVisor or build-related block), merge the runtime sections into one file and rewrite it wholesale; appending a duplicate template line produces an invalid containerd config and the node's kubelet will not come back after the restart — recovery then requires SSH/console access, `kubectl` cannot reach the node.

### Local development[​](#local-development "Direct link to Local development")

There is no macOS-local path for Kata (no nested virtualization, and k3d node containers cannot boot Kata VMs). On a KVM-capable **Linux** host running the repository's dev k3d cluster:

```
make setup-kata
```

This joins the host itself to the cluster as a real (non-Docker) k3s agent node and then runs `install-kata.sh` against it.

## Create a Kata Environment[​](#create-a-kata-environment "Direct link to Create a Kata Environment")

Isolation tier is set at environment creation time. Open the **Create Environment** drawer in the console (Environments → Create Environment), select **Sandboxing T3 (Kata Containers)** from the **Isolation Tier** dropdown, and run the generated command — it includes `ISOLATION_TIER=kata` automatically.

If you script environment creation directly, set the variable yourself:

```
ISOLATION_TIER=kata \

  ENV_NAME=secure-prod \

  DISPLAY_NAME="Secure Prod" \

  AGENT_MANAGER_TOKEN=<token> \

  bash "${AMP_DIST}/deployments/scripts/add-environment.sh"
```

The tier appears as a **Sandboxing T3** shield chip in the environment list, and as a shield icon next to the environment name on the agent's Deploy and Overview pages — hover over it to confirm *Sandboxing Tier 3 — Kata Containers*.

A new environment is not part of any deployment pipeline yet. Add it to your project's pipeline ([Managing Deployment Pipelines](/agent-platform/docs/v1.0.0/guides/environment-management/.md#managing-deployment-pipelines)), then deploy or promote an agent to it as usual.

## Verify VM Isolation[​](#verify-vm-isolation "Direct link to Verify VM Isolation")

A Kata agent runs a **different kernel** from the node — that's the definitive check:

```
kubectl get pod <agent-pod> -n <namespace> -o jsonpath='{.spec.runtimeClassName}'   # kata-qemu



kubectl exec <agent-pod> -n <namespace> -- uname -r

# Differs from the node's own `uname -r`



kubectl exec <agent-pod> -n <namespace> -- grep hypervisor /proc/cpuinfo

# hypervisor flag present
```

If the pod reports the same kernel as the node, it is **not** running under Kata (the RuntimeClass wasn't applied or the pod fell back to runc).

## Troubleshooting[​](#troubleshooting "Direct link to Troubleshooting")

Pod stuck in ContainerCreating

Almost always missing nested virtualization. Confirm on the node itself (with the `kubectl debug` command under the KVM requirement above, or over SSH):

```
ls -l /dev/kvm

cat /sys/module/kvm_intel/parameters/nested   # expect: Y

kubectl describe pod <pod> -n <ns>            # look for kvm / RuntimeClass errors
```

No `/dev/kvm` → move to a bare-metal or nested-virt-enabled host (e.g. GCP Intel N2 with `--enable-nested-virtualization`).

`no runtime handler kata-qemu` errors

The RuntimeClass exists but containerd on the node has no `kata-qemu` runtime — the install didn't complete or containerd wasn't reloaded:

```
kubectl -n kube-system rollout status daemonset/kata-deploy

kubectl -n kube-system logs -l name=kata-deploy



# on the node — confirm the runtime is in containerd's config:

sudo grep -n kata-qemu /var/lib/rancher/k3s/agent/etc/containerd/config.toml   # k3s nodes

grep -n kata-qemu /etc/containerd/config.toml                                  # other nodes
```

On k3s nodes, re-running `install-kata.sh` re-applies the containerd template wiring.

Pod runs but no logs / metrics / traces / try-it

This is cross-node networking or log-collection scheduling, not Kata itself — the failure modes and fixes are identical to gVisor's. Work through the [gVisor troubleshooting section](/agent-platform/docs/v1.0.0/guides/isolation-tiers/gvisor/.md#troubleshooting), substituting `runtimeClassName: kata-qemu` in the test pod.

Performance notes

* Booting a VM adds startup latency compared to runc/gVisor; Agent Manager's warm pools absorb most of it.
* `kubectl port-forward` should be avoided for Kata pods — reach agents through their Service/gateway endpoint.
* Usage metrics are attributed to the sandbox VM as a whole (reported under a `sandbox` container label), including the guest kernel's footprint.

## Next Steps[​](#next-steps "Direct link to Next Steps")

* Lighter-weight syscall isolation without KVM: [gVisor isolation tier](/agent-platform/docs/v1.0.0/guides/isolation-tiers/gvisor/.md).
* [Environment management](/agent-platform/docs/v1.0.0/guides/environment-management/.md) — pipelines, promotion, and environment settings.
