Service mesh security — deep

Service mesh security — deep

TL;DR: A service mesh (Istio, Linkerd, Cilium, Consul, App Mesh) gives you mTLS by default, identity-aware AuthZ, and L7 observability between workloads — but it adds a control plane to defend, an Envoy data plane with its own CVE stream, and operational debt that not every org can carry. Treat the mesh as another security product with a TCO and a threat model, not a free zero-trust win. Companion notes: zero-trust-architecture-practitioner, k8s-admission-webhook-abuse, container-runtime-escapes-modern, cloud-ir-k8s-audit-logs.

Why it matters

East-west traffic inside Kubernetes is historically the soft underbelly. Once an attacker has a pod foothold (see container-runtime-escapes-modern and cloud-ir-k8s-audit-logs), they usually find:

  • Plain HTTP between services on the pod network.
  • No workload identity — every pod can reach every service unless NetworkPolicy is in place (and most clusters do not enforce it well).
  • App-level auth that trusts the in-cluster network.
  • Logs that show “service A called service B” but no cryptographic proof.

A service mesh injects a proxy (or eBPF program) on the data path that:

  1. Issues short-lived workload certificates (typically SPIFFE-shaped) and enforces mTLS.
  2. Applies identity-based L7 AuthZ (who can call which path/method).
  3. Emits structured telemetry — request logs, RED metrics, distributed traces.
  4. Implements traffic management (retries, timeouts, canary, mirroring).

For mature platforms this is real value. For a 3-service startup it is often a complexity tax that displaces simpler controls.

Landscape and patterns

The major meshes

  • Istio — Envoy-based, most features, biggest community, historically heaviest. Two modes now: sidecar (Envoy per pod) and Ambient (per-node ztunnel for L4 + optional waypoint proxies for L7).
  • Linkerd — Rust micro-proxy (linkerd2-proxy), opinionated, much smaller surface, faster, fewer features. Strong on simplicity and audit story.
  • Cilium Service Mesh — eBPF in the kernel for L3/L4 + identity, Envoy embedded for L7. No sidecar required. Tied to Cilium CNI.
  • Consul Connect — HashiCorp, works on VMs and K8s, integrates with Vault PKI, common in mixed-estate enterprises.
  • AWS App Mesh — managed Envoy control plane; AWS announced end of life in 2026, customers being pushed to ECS Service Connect or VPC Lattice. Mention only because it is in production at many shops.

Control plane vs data plane

  • Control plane issues config and certs: istiod, linkerd-destination/linkerd-identity, Cilium operator + agent, consul-server.
  • Data plane is the proxy: Envoy (Istio, Consul, App Mesh, Cilium L7), linkerd2-proxy (Linkerd), ztunnel (Istio Ambient).
  • Compromise of the control plane is catastrophic: attacker mints workload certs, rewrites AuthZ, redirects traffic. Treat it like a CA — see cloud-iam-misconfig-patterns for the analogous IAM threat model.

Sidecar vs ambient / sidecar-less

  • Sidecar: predictable, per-pod blast radius, but doubles container count, complicates init ordering, pod restarts on proxy upgrade, and adds latency.
  • Ambient (Istio) and Cilium: shared per-node proxy / eBPF — lighter, faster upgrades, but a compromise of the node proxy now affects every pod on that node. Different blast radius, not a smaller one.
  • For regulated workloads, document the choice in your threat model. Auditors increasingly ask. See building-an-iso27001-isms-practitioner.

mTLS modes

Istio PeerAuthentication (and equivalents) typically expose:

  • DISABLE — plaintext.
  • PERMISSIVE — accept both. Required for rollout, dangerous as a steady state — attacker who can ARP/route-spoof inside the pod network can downgrade.
  • STRICT — mTLS required. The goal.

Rollout pattern that actually works:

  1. Install mesh, inject into one low-risk namespace.
  2. Leave at PERMISSIVE, watch metrics (istio_requests_total by connection_security_policy).
  3. Burn down plaintext callers (often legacy cron jobs, monitoring scrapers, external DNS calls leaving the mesh).
  4. Flip to STRICT per namespace, not cluster-wide on day one.
  5. Repeat. Plan for 3-9 months for a real platform.

Policy languages

  • Istio AuthorizationPolicy — namespace/workload selector + from (principal/source) + to (paths/methods) + when (claims). Allow-by-default unless any policy matches — easy to get wrong.
  • Linkerd Server + AuthorizationPolicy — explicit server resource, deny by default once a Server exists. Simpler mental model.
  • OPA / Envoy ext_authz — punt policy to Rego via an external authorizer. Powerful but adds another hop and another thing to monitor. Pairs with authorization-patterns-rebac-abac.
  • Cilium CiliumNetworkPolicy — L3-L7 in one CRD, identity-aware via labels, enforced in eBPF.

Workload identity (SPIFFE / SPIRE)

  • SPIFFE IDs look like spiffe://cluster.local/ns/payments/sa/checkout.
  • They are derived from the K8s ServiceAccount, so SA hygiene is now an authentication concern, not just RBAC. See k8s-admission-webhook-abuse for how attackers abuse SA tokens.
  • Federation across clusters and across clouds (SPIRE) is the only realistic path to “one identity from EKS to GKE to bare metal” — relevant for zero-trust-architecture-practitioner.

Attack surface and CVE reality

Control plane

  • Exposed istiod xDS endpoints, leaked kubeconfig for the mesh operator, or RBAC over *.istio.io CRDs all mean game over.
  • Istio has had multiple high-severity CVEs in istiod (request smuggling, auth bypass, DoS). Patch cadence matters; meshes are not “set and forget.”
  • Linkerd’s smaller surface and Rust proxy have produced fewer critical CVEs historically — a legitimate factor in tool selection.

Data plane (Envoy)

  • Envoy ships HTTP/1, HTTP/2, HTTP/3, gRPC, WebSocket, and a Lua/Wasm filter chain. That is a lot of parser.
  • Recurrent CVE classes: HTTP/2 rapid reset (CVE-2023-44487), header smuggling, OAuth filter bypass, ext_authz bypass via specific header casings.
  • If you run Istio/Consul/App Mesh, you inherit Envoy’s CVE stream and have to roll the mesh on a deadline. Track envoyproxy/envoy security advisories.

Filter / extension code

Egress and the “trust the mesh” trap

  • mTLS inside the mesh does nothing for calls that leave the mesh to a public API or another VPC. Pair with egress controls and DNS logging.
  • Attackers who land in a meshed pod will happily use the pod’s outbound to exfil; the mesh’s L7 policy on egress is often ALLOW_ANY because operators got tired of breakage.

Defensive baseline

  1. Pick the right mesh, or none. Under ~20 services, prefer NetworkPolicy + app-level mTLS or a smaller mesh (Linkerd, Cilium if you already run Cilium CNI). Istio’s full feature set is for platforms, not products.
  2. Treat istiod / control plane like a CA. Dedicated namespace, restricted RBAC, separate node pool, alerting on CRD changes, no shared kubeconfig.
  3. Pin and patch the data plane. Subscribe to Envoy and mesh advisories. Have a documented “P0 mesh upgrade” runbook — expect 2-4 per year.
  4. Default-deny AuthZ per namespace once mTLS is STRICT. Linkerd makes this natural; in Istio, ship an explicit deny-all AuthorizationPolicy plus per-service allows.
  5. Egress is in scope. Define which workloads may leave the mesh, to which FQDNs, on which ports. Log denials.
  6. Telemetry into the SIEM. Envoy access logs, ext_authz decisions, mesh control-plane audit logs. Build detections — see detection-engineering-pyramid-of-pain and siem-detection-use-case-catalog.
  7. K8s audit log is still your friend. Mesh CRD edits, ServiceAccount creation, and webhook changes are the loud signals of mesh tampering. See cloud-ir-k8s-audit-logs and k8s-admission-webhook-abuse.
  8. Threat-model the proxy upgrade path. Sidecar restarts cascade across the cluster; ambient/eBPF node-level outages affect every pod on the node. Plan for it in your SLOs.
  9. Compliance mapping. mTLS-by-default helps PCI 4.0 cryptography-in-transit, HIPAA transmission security, and DORA ICT risk controls — but only if you can produce evidence. See building-a-pci-dss-program-practitioner, hipaa-security-rule, nis2-implementation.

Workflow to study

  1. Spin up kind or minikube. Install Linkerd (lowest activation energy), then Istio sidecar, then Istio Ambient. Feel the operational difference.
  2. Deploy a 3-service demo (Bookinfo or your own). Capture plaintext traffic with tcpdump in a sidecar-less pod, then with mTLS on. Prove to yourself it actually encrypts.
  3. Write an AuthorizationPolicy that breaks the app. Read the access log. Learn to read Envoy RBAC: access_denied.
  4. Run an OPA ext_authz sidecar and write a Rego policy that checks a JWT claim. Compare latency overhead.
  5. Read the last 12 months of Envoy security advisories. Map each to “would my current policy stop it?” Most of the time the answer is no — defense in depth still matters.
  6. Stand up SPIRE alongside a mesh and federate two clusters. This is the realistic zero-trust homework.
  7. Pair with appsec-threat-modeling and build a STRIDE on the control plane itself.

Honest take

  • The mesh marketing pitch (“turn on zero trust”) collapses on contact with day-2 ops. The orgs that win with a mesh have a platform team, an on-call rotation for the mesh, and a CI pipeline for policy. The orgs that lose have a half-installed Istio that nobody upgrades.
  • Performance overhead is real but usually small (single-digit ms p50, more at p99) — debugging overhead is the bigger cost.
  • “Service mesh vs API gateway” is a false binary. Gateways (Kong, Envoy Gateway, AWS ALB, Apigee) handle north-south; meshes handle east-west. Most platforms need both; many small ones need only the gateway.
  • For greenfield small estates, NetworkPolicy + a gateway + workload identity from the cloud provider is usually a better ROI than a full mesh.

References

  • Istio security overview and Ambient architecture — https://istio.io/latest/docs/concepts/security/ and https://istio.io/latest/docs/ops/ambient/architecture/
  • Linkerd security and authorization policy — https://linkerd.io/2/features/server-policy/
  • Cilium Service Mesh — https://docs.cilium.io/en/stable/network/servicemesh/
  • SPIFFE / SPIRE concepts — https://spiffe.io/docs/latest/spiffe-about/overview/
  • Envoy security advisories — https://github.com/envoyproxy/envoy/security/advisories
  • HTTP/2 Rapid Reset (CVE-2023-44487) writeup — https://blog.cloudflare.com/technical-breakdown-http2-rapid-reset-ddos-attack/