Admission Webhook

Overview

The PodWarden admission webhook (chart podwarden-admission-webhook, image registry.podwarden.com/podwarden/admission-webhook) is a ValidatingAdmissionWebhook installed in every PodWarden-managed cluster. It intercepts direct kubectl (and any other Kubernetes API client) operations on namespaces PodWarden manages and either blocks them with a guidance message or records them as warnings, depending on the namespace's policy.

It only acts on namespaces explicitly labelled podwarden.com/managed=true. Everything else in the cluster — kube-system, your own namespaces, other tenants — is never inspected.

If you arrived here from a BLOCKED: message in your terminal, scroll to The deny message and Bypass mechanisms.

Source-of-truth references in this guide point to admission-webhook/ in podwarden/podwarden. All matrices and identity lists are dated; treat anything older than the date stamp as potentially stale.

Why It Exists

PodWarden tracks every workload, ingress rule, secret, and config slot in its own database, then reconciles to Kubernetes. This is what lets the platform offer:

• Consistent state — the UI, the API, MCP tools, and the cluster all agree on what's deployed.
• Networking — automatic ingress rules, gateway routing, DDNS, and TLS issuance hinge on PodWarden owning the Service, Ingress, and IngressRoute shape.
• Backups & restores — backup policies, snapshot retention, and one-click restore depend on knowing which PVCs belong to which deployment.
• Lifecycle — drift detection, redeploys, and migrations work because PodWarden is the source of truth.

Direct kubectl edit deployment/foo in a managed namespace silently breaks all four. Worse, an AI agent given raw kubectl access can drift the cluster faster than humans can notice. The webhook closes that gap by funnelling all changes through the PodWarden API or MCP tools — which write to the database and apply to the cluster atomically.

Threat model

The webhook is designed to defend against accidental kubectl mutations and casual drift, not against a malicious cluster-admin. Specifically:

Non-goals

• The webhook does not authorize PodWarden API calls — that's Keycloak/JWT/RBAC's job.
• It does not enforce policy on namespaces that are not labelled podwarden.com/managed=true (the pw-* namespaces — see the Managed Namespace Pattern wiki page).
• It does not prevent reads (kubectl get, kubectl logs, kubectl describe) or kubectl exec — those don't go through validating admission.
• It does not rotate its own TLS certificate. The chart's TLS material is supplied at install time by PodWarden Core (see Certificate rotation).

How It Works

A ValidatingWebhookConfiguration named podwarden-admission is registered with the cluster API server. When a write operation hits the API server in a managed namespace, the API server forwards an AdmissionReview to the webhook over TLS, waits for a verdict (with a 5-second timeout), and only commits the change if the webhook says Allowed: true.

kubectl ──▶ kube-apiserver ──▶ podwarden-admission webhook ──▶ verdict
                                       │
                                       └─▶ system message (block/warn)

The webhook itself is a small Go service that runs as a single Deployment in the podwarden-system namespace. Two configurations are registered:

1. admission.podwarden.com (resources webhook) — selects on namespaceSelector { matchLabels: podwarden.com/managed: "true" }. Inspects writes to all the resource kinds listed below.
2. namespaces.podwarden.com (namespaces webhook) — cluster-scoped, no namespaceSelector and no objectSelector. Fires on every UPDATE/DELETE against a Namespace cluster-wide; the handler then inspects req.OldObject.Labels[podwarden.com/managed] and only enforces if the namespace was managed at the start of the request. This entry exists because Kubernetes evaluates namespaceSelector/objectSelector against the new object on UPDATE — without it, kubectl label ns <ns> podwarden.com/managed- would deselect the webhook (the new object has no label) and let the namespace be unlabelled or deleted unguarded.

failurePolicy is Ignore — if the webhook pod is unreachable, the API server fails open. PodWarden Core can always reach the cluster even when the webhook is down, so a webhook outage degrades enforcement but never wedges the cluster.

What gets intercepted

Matrix as of 2026-04-29, chart 0.1.4 deployed on pw_prod. Subresource coverage closed by MR !327 (issue #678); namespace-tampering closed by MR !325 (issue #679).

Operations covered: CREATE, UPDATE, DELETE. Subresources are listed explicitly because Kubernetes treats e.g. kubectl scale and kubectl create -f eviction.yaml as writes to the /scale and /eviction subresources rather than to the parent — without the explicit listing they would silently bypass the gate.

Identity whitelist (what is allowed through)

The whitelist is hard-coded in admission-webhook/config.go:45-53. As of 2026-04-29:

The match is strings.HasPrefix(username, prefix) against req.UserInfo.Username and against every entry in req.UserInfo.Groups.

Other passes:

• Anything in non-managed namespaces. No label, no inspection.
• Reads. The webhook only registers on write operations.

Explicitly NOT whitelisted

• system:masters — the group every cluster-admin kubeconfig carries by default. Not whitelisted, intentionally. If it were, every operator's default kubeconfig would silently bypass the webhook.
• No env-var override. The legacy PODWARDEN_KUBECTL_USER env var was removed by MR !322; the only way to add a new identity is to edit config.go.

Adding a new SA prefix

This is a code change — there is no runtime knob.

1. Edit admission-webhook/config.go, append to WhitelistPrefixes.
2. Add a unit test in admission-webhook/config_test.go.
3. Bump admission-webhook/chart/Chart.yaml version (chart only — appVersion reflects the binary).
4. Open an MR. The new image tag (sha-{commit}) ships through the standard chart-publish flow.

When the webhook honours the new whitelist, the pod logs slog.Info("whitelisted", "user", <username>, "ns", <namespace>) for each pass-through.

Decision matrix (resource × identity × namespace state)

Source: admission-webhook/handler.go:92-205.

What is NOT covered

• Reads, exec, port-forward. kubectl get, kubectl describe, kubectl logs, kubectl exec, kubectl port-forward and other read/streaming operations do not go through validating admission and are never inspected.
• Custom Resources outside the listed CRDs. Resources from operators PodWarden does not manage (e.g. cert-manager.io/Certificate, monitoring CRs) pass through unchecked.
• Audit propagation to PW Core on pw_prod — currently broken; tracked as #680. See Audit log below.

The deny message

When a request is blocked, the webhook returns the request's resource and operation along with the equivalent PodWarden API endpoint, MCP tool, and UI path. Example:

Error from server: admission webhook "admission.podwarden.com" denied the request:
BLOCKED: This namespace is managed by PodWarden.

You attempted: kubectl update deployments "pi-hole-0f7fa523" in namespace "pw-pi-hole"

Direct kubectl modifications to PodWarden-managed namespaces are not allowed.

Instead, use one of these methods:

  PodWarden API:  PUT /api/v1/deployments/{id}
  MCP Tool:       update_deployment
  Web UI:         http://podwarden.example.com:8000/deployments

This action has been logged. PodWarden administrators have been notified
and can review this event in System > Messages.

To allow direct kubectl access on this namespace, an admin can run:
  kubectl annotate namespace pw-pi-hole podwarden.com/admission-policy=warn --overwrite

Documentation: https://www.podwarden.com/docs/guides/admission-webhook

The mapping table covers every intercepted resource × operation pair, so the deny message always points at the right tool.

Bypass Mechanisms

There are three supported ways to allow kubectl through, in increasing order of audit-friendliness.

1. Permanent per-namespace warn-only mode

Add an annotation to a single namespace:

kubectl annotate namespace <ns> podwarden.com/admission-policy=warn --overwrite

Effect: writes are no longer blocked, but each one still surfaces as a category=admission severity=warning entry in System Messages with the same context (resource, operation, user). Useful when a namespace is intentionally co-managed (e.g. an external GitOps controller writes to it) and you need the audit trail without the block.

The annotation does not apply to namespace-level mutations (UPDATE/DELETE of the namespace object itself) — those always block, regardless of the annotation.

To revert:

kubectl annotate namespace <ns> podwarden.com/admission-policy- --overwrite

2. Time-boxed cluster-wide bypass (recommended for break-glass)

For legitimate destructive maintenance (a stuck deployment that needs kubectl delete, a manual fix during an incident), the webhook honours a time-boxed bypass ConfigMap that disables enforcement cluster-wide for its window.

From the UI: open Apps → Admission Webhook → Temporary bypass and pick a window: 60 s, 180 s, or 300 s. The bypass auto-expires when the timer runs out.

From the API (audit-tracked, recommended for scripts):

curl -X POST https://<podwarden>/api/v1/system-apps/admission-webhook/bypass \
     -H "Authorization: Bearer $PW_TOKEN" \
     -H "Content-Type: application/json" \
     -d '{"duration_seconds": 60, "reason": "incident #1234"}'

To cancel:

curl -X DELETE https://<podwarden>/api/v1/system-apps/admission-webhook/bypass \
     -H "Authorization: Bearer $PW_TOKEN"

The system-apps/{name}/bypass endpoint requires the admin role and writes a system_messages row with category='admission_bypass' before applying the ConfigMap.

Under the hood

Both UI and API write a ConfigMap:

Namespace: podwarden-system
Name:      admission-bypass

Both names are hard-coded constants in admission-webhook/bypass.go:19-22. A user who can create arbitrary ConfigMaps in the cluster cannot impersonate a bypass with a different ConfigMap name — the webhook's RBAC restricts its read permission to exactly this ConfigMap.

Shape:

apiVersion: v1
kind: ConfigMap
metadata:
  name: admission-bypass
  namespace: podwarden-system
data:
  expires_at: "2026-04-29T20:30:00Z"   # RFC3339, required
  initiated_by: "[email protected]"    # free-form, surfaced in audit log
  reason: "incident #1234, ingress hung"

• The webhook polls every 5 s and re-checks the expiry on every admission request, so an expired bypass is honoured immediately rather than waiting up to 5 s.
• Hard cap: 300 seconds. Two safety guards:
  • Server-side cap — the API rejects any window longer than 300 s.
  • Webhook-side cap — even if the ConfigMap is hand-edited to set expires_at an hour out, the webhook clamps it to now + 300s (bypass.go:204-207).

Direct kubectl emergency path (when the API is unreachable)

kubectl -n podwarden-system create configmap admission-bypass \
  --from-literal="expires_at=$(date -u -d '+5 minutes' +%Y-%m-%dT%H:%M:%SZ)" \
  --from-literal="initiated_by=$(whoami)" \
  --from-literal="reason=manual emergency bypass"

To cancel:

kubectl -n podwarden-system delete configmap admission-bypass

Direct kubectl bypasses lose the PW Core audit trail (no system_messages row written), so prefer the API or UI path when reachable.

Security implications

• A bypass disables all webhook enforcement cluster-wide for its window — including the namespace-tampering protection. An admin with bypass on can drop the podwarden.com/managed label from any namespace.
• The webhook still emits an AdmissionEvent with action="bypassed" for every request that the bypass let through, with bypass_expires and initiated_by in the log line (handler.go:122-144). Operators get a complete breadcrumb of what was done during the window.
• The 300 s cap is a deliberate forcing function — if you need longer, renew. Long-lived bypass ConfigMaps are an anti-pattern.

3. Use the PodWarden API or MCP tool

The right answer for almost every case. If you find yourself reaching for bypasses regularly, file an issue so the operation can be exposed as a first-class API endpoint or MCP tool.

See the MCP Tools reference for the full catalogue.

Audit Log

Webhook pod logs

Every admission decision emits a structured slog line:

kubectl -n podwarden-system logs deployment/podwarden-admission --tail=200

PodWarden Core — System Messages

The webhook POSTs every block/warn/bypass decision to POST /api/v1/admission-events on PW Core (reporter.go:33-63). PW Core upserts the event into the system_messages table with category='admission' (backend/app/api/admission_events.py:42-73) — there is no separate admission_events table; events are first-class system messages.

Each entry carries:

• category=admission
• severity=error for blocks, severity=warning for warns/bypassed
• payload: namespace, resource_type, resource_name, operation, user, action

Browse them in the UI under Apps → Admission Webhook → Violation Log, or query via the API:

GET /api/v1/system-messages?category=admission&status=active

Repeated identical events are coalesced — the same (namespace, resource, operation, user) tuple bumps an attempt_count instead of producing a fresh row.

Currently broken on pw_prod: the POST gets connection refused because of a stale PODWARDEN_API_URL (the configured URL is reachable from outside the cluster but not from inside). Tracked as #680. Until that closes, the webhook pod logs are the canonical record. Bypass ConfigMap creation/cancellation through the PW Core API still writes to system_messages independently of the webhook→Core POST and is unaffected.

Settings

In Settings → Admission Webhook:

The webhook is treated as a system app and is normally installed automatically on cluster registration. To install it manually after the fact, use Apps → Admission Webhook → Install, or run the system app install via API.

Verification

Confirm the webhook is healthy:

# Webhook pod
kubectl -n podwarden-system get pods -l app.kubernetes.io/name=podwarden-admission-webhook

# Registered ValidatingWebhookConfiguration
kubectl get validatingwebhookconfiguration podwarden-admission -o yaml

# Which namespaces are gated
kubectl get ns -l podwarden.com/managed=true

# Current bypass status (empty / not-found = no bypass)
kubectl -n podwarden-system get configmap admission-bypass -o yaml 2>/dev/null

A quick functional test against any managed namespace:

kubectl scale deployment <some-deployment> -n <managed-ns> --replicas=1 --dry-run=server

You should see the BLOCKED message above (with --dry-run=server it triggers admission without actually scaling).

Troubleshooting

Operations bypass the webhook unexpectedly

Check the namespace label: kubectl get ns <ns> --show-labels — without podwarden.com/managed=true, the webhook's namespaceSelector skips the namespace entirely. Also check whether a bypass ConfigMap is active (kubectl -n podwarden-system get configmap admission-bypass).

Webhook pod is CrashLoopBackOff or not Ready

kubectl -n podwarden-system describe pod <pod>
kubectl -n podwarden-system logs deployment/podwarden-admission --tail=200

The pod has both a liveness and a readiness probe on https://:8443/healthz. Readiness failures usually mean the TLS material in the podwarden-admission-tls Secret is missing or malformed — re-running the system app install regenerates it.

The webhook's failurePolicy is Ignore, so when the pod is not Ready, all admission requests pass through unguarded. This is intentional — a misbehaving webhook should never wedge the cluster — but a quietly broken webhook is also a silent loss of protection. Monitor pod readiness as part of cluster health.

Certificate rotation

The chart does not integrate with cert-manager. TLS material is supplied at install/upgrade time via Values.tls.cert/key/caBundle and rendered into the podwarden-admission-tls Secret. Rotation is a helm upgrade with new values:

helm upgrade --reuse-values podwarden-admission ./admission-webhook/chart \
  --namespace podwarden-system \
  --set tls.cert="$(cat new-tls.crt | base64 -w0)" \
  --set tls.key="$(cat new-tls.key | base64 -w0)" \
  --set tls.caBundle="$(cat new-ca.crt | base64 -w0)"
kubectl -n podwarden-system rollout restart deployment/podwarden-admission

PW Core regenerates and ships TLS material as part of the cluster-registration / re-bootstrap flow. There is currently no scheduled rotation — issue a rotation when the certificate is approaching its NotAfter, or when key material is suspected of compromise.

Bypass UI button is greyed out

The button is gated on operator role. Check Settings → Users for the calling user's role.

A bypass is "stuck on" after expiry

The webhook polls every 5 seconds, so worst-case latency is 5 s after expires_at. If you see longer, force a refresh by deleting the ConfigMap: kubectl -n podwarden-system delete configmap admission-bypass.

A managed namespace can't be deleted

That's the namespaces webhook doing its job. Delete via the PodWarden API (DELETE /api/v1/deployments/{id} triggers full cleanup) or — if PodWarden no longer tracks it — open a temporary bypass and kubectl delete namespace within the window.

False positives — "but PW Core itself was blocked"

A "false positive" usually means PW Core's own SA wasn't whitelisted for the operation. Check:

1. The pod logs for the request: look for slog.Warn("blocked", …, "user", "<who>").
2. If <who> is a system:serviceaccount:podwarden-system: or system:serviceaccount:podwarden:podwarden-api identity, that's a regression — file an issue and use the bypass to unblock.
3. If <who> is system:admin or another cluster-admin user, the block is correct — see Threat model.

The legacy PODWARDEN_KUBECTL_USER env-var bypass (which previously let PW Core impersonate system:admin) was removed in MR !322; PW Core now authenticates as the podwarden-api SA in the podwarden namespace. If you see PW Core requests under system:admin after that MR shipped, the cluster's kubeconfig source has drifted (kubeconfig_source should be manual pointing at the SA token, not ssh_fetch).

See Also

• System Apps — operators that extend PodWarden, including the webhook
• System Messages — where audit entries surface
• MCP Tools — the API/MCP catalogue your "instead of kubectl" lookups should land on
• Drift Detection — the complementary feature that catches drift the webhook didn't (e.g. resources created before the webhook was installed)
• Managed Namespace Pattern (wiki) — naming convention, label set, and the Webhook Protection design rationale
• Deploy Application runbook (wiki)
• Open issue: #680 — event reporter URL on pw_prod
• MRs: !322 (drop kubectl-user bypass), !325 (namespace-tampering protection), !327 (scale subresource coverage)