Skip to content

Remediation (--fix)

scan --fix proposes — and, after you confirm, applies — safe, reversible fixes for a small set of problems kubeagent already detects.

Warning

Remediation is the only feature that writes to your cluster — --fix, and --rollback, which undoes what a previous --fix recorded in its audit log. The two are mutually exclusive, and no other flag on any command writes. Without them, kubeagent is strictly read-only. Every write is deterministic (never decided by --explain or any model), drawn from a fixed allowlist, and applied only after a per-action confirmation.

The actions

Only these actions are ever planned or applied — nothing outside the allowlist:

Action Proposed when What it does kubectl equivalent
RolloutUndo a Deployment is degraded (Ready < Desired) because its newest rollout cannot pull its image, and a prior revision exists rolls the Deployment back to its previous revision kubectl -n <ns> rollout undo deployment/<name>
Uncordon a node is cordoned (SchedulingDisabled) with no NoExecute taint makes the node schedulable again kubectl uncordon <node>

A rollout that is stuck on ImagePullBackOff but whose previous revision is still serving (Ready == Desired) is not rolled back — the app is not down, so the image is left for you to fix forward.

An accidental cordon is uncordoned; a deliberate drain (which carries a NoExecute taint) is left alone.

Guard rails

  • Opt-in. Writes happen only with --fix; the default is read-only.
  • Per-action confirmation. Each fix prints its target, reason, and kubectl equivalent, then prompts Apply? [y/N] — the default is No.
  • Protected namespaces. Remediations never target kube-system, kube-public, or kube-node-lease.
  • Apply-time re-check. Just before writing, the problem is re-verified against live cluster state; if it has already resolved, the action is skipped with no write.
  • Single write per action. One API update — nothing cascading.
  • Never model-decided. The plan is pure, deterministic logic. --explain is never consulted, and no model ever influences a write.

Preview and non-interactive

# show what would be done — write nothing, never prompt
./kubeagent scan --fix --dry-run

# apply every proposed fix without prompting (CI / scripted use)
./kubeagent scan --fix --yes

--fix --dry-run against a cluster with a bad rollout and a cordoned node: it runs the normal scan, then lists the proposed fixes — a RolloutUndo and an Uncordon — and exits without writing anything:

kubeagent scan --fix --dry-run

The preview is a contract

Every proposed fix includes a curated will change: diff — the revision line, a per-container image change (image (name): old → new), and a count-only N other template fields changed line for anything else — computed at plan time, before Apply? is shown. Only safe, structural fields are diffed: env values and template contents are never shown.

Apply is bound to that preview. Just before writing, kubeagent re-checks the live revision and pod-template hash. If the cluster has moved — a new rollout landed, or the target revision is gone — the action is refused with state changed since preview … no write made and rendered as skipped:. To retry, re-run the scan so a fresh preview is computed against current state.

Example

Proposed fix: shop/web (Deployment) — roll back to the previous revision
  reason: newest rollout cannot pull its image; a prior revision (1) exists
  will change:
    revision: 2 → 1
    image (web): registry.example.com/web:v2 → registry.example.com/web:v1
  kubectl equivalent: kubectl -n shop rollout undo deployment/web
  Apply? [y/N] y
  applied: rolled back shop/web to revision 1 (pod template restored)

Proposed fix: node/worker-1 — uncordon the node (make it schedulable)
  reason: node is cordoned (SchedulingDisabled)
  will change:
    spec.unschedulable: true → false
  kubectl equivalent: kubectl uncordon worker-1
  Apply? [y/N] y
  applied: uncordoned node worker-1

When nothing is safely fixable, kubeagent says so and writes nothing:

No automatic remediations available.

RBAC preflight

Before attempting each write, kubeagent runs a SelfSubjectAccessReview (SSAR) to confirm that the current credentials are permitted to perform it — for example, update on the target Deployment in namespace shop, or update on the target Node.

Denied — refused up front. If the SSAR returns a denial, the action is skipped before any write is attempted:

skipped: you lack permission to update deployments in namespace "shop" (RBAC); no write attempted

The outcome is recorded in the audit log with the new preflight disposition — a clean up-front refusal instead of a mid-apply 403.

Fail-closed on SSAR error. If the SelfSubjectAccessReview API call itself fails (the API server is unreachable, a transient error), kubeagent fails closed: no write is attempted and the outcome is recorded with the error disposition.

Under --dry-run. The preflight check runs in dry-run mode too (read-only). Depending on what the SSAR returns, the dry-run line reports one of:

(dry-run: not applied; you have permission to apply this)
(dry-run: not applied; would be blocked — you lack permission to update deployments in namespace "shop" (RBAC))
(dry-run: not applied; permission check errored: <detail>)

No extra RBAC required. The SelfSubjectAccessReview is a self-check — querying your own permissions — and create selfsubjectaccessreviews is granted to all authenticated users by the built-in system:basic-user ClusterRole. kubeagent does not need any additional RBAC to run the preflight.

Audit log (--audit-log)

--audit-log <path> (used together with --fix) appends a durable, append-only JSON-Lines record of every remediation outcome — one line per action — to the file you name. The log is written whether the action was applied, declined, skipped as a dry-run, or refused by a safety guard.

kubeagent scan --fix --yes --audit-log /var/log/kubeagent-fix.log

Each record is a single JSON object on its own line:

{"time":"2026-07-24T06:30:00Z","kind":"RolloutUndo","namespace":"shop","name":"web","target":"shop/web (Deployment)","changes":[{"field":"revision","from":"5","to":"4"},{"field":"image (web)","from":"registry.example.com/web:v2","to":"registry.example.com/web:v1"}],"disposition":"applied","detail":"rolled back shop/web to revision 4 (pod template restored)"}
{"time":"2026-07-24T06:31:00Z","kind":"Uncordon","name":"worker-1","target":"node/worker-1","disposition":"refused","detail":"node is no longer a safe uncordon target (already schedulable or NoExecute-tainted); no write made"}

Disposition vocabulary:

Disposition When it appears
applied The write succeeded.
dry-run --dry-run was set; the action was planned but no write was made.
declined The operator answered N (or pressed Enter) at the Apply? [y/N] prompt.
refused A safety guard fired at apply time (cluster state drifted, or a nil-error "no write made" condition); no write was made.
preflight The RBAC preflight (SelfSubjectAccessReview) denied the action before any write was attempted.
error The write was attempted but the API server returned an error; or the SSAR itself failed (fail-closed).
rollback A --rollback invocation successfully applied the inverse of the most recent applied record.

Properties of the audit file:

  • 0o600, append-only. The file is opened with mode 0o600 (O_APPEND), so only the process owner can read it and concurrent runs safely interleave.
  • Records every disposition. Every outcome — including dry-runs and user-declined actions — is logged, giving a complete picture of what kubeagent proposed and what happened.
  • Fails fast. If the path is unwritable, kubeagent reports the error and exits before any scan or write begins.
  • Secret-free by construction. Only the previewed diff values (the same fields shown in the will change: block), the action metadata, and the result detail are recorded. Env values and template contents are never captured.

Rollback (--rollback)

kubeagent scan --rollback --audit-log /var/log/kubeagent-fix.log reads the most recent applied record from the audit log and proposes its deterministic inverse:

  • RolloutUndo → roll forward — a Deployment that was rolled back is rolled forward to the revision it was on before the fix (i.e. it restores the image/template that the --fix had rolled away from).
  • Uncordon → re-cordon — a node that was uncordoned is cordoned again.

Guard rails (same as --fix)

The inverse runs through every guard rail that --fix uses:

  • Curated will change: preview diff — for example revision: 4 → 5.
  • [y/N] confirmation — or --yes to skip.
  • Drift bond — if the cluster has moved since the original fix was applied (the revision is already gone, or the node is already cordoned), the rollback is refused with a clear message and no write is made.
  • RBAC preflight — a SelfSubjectAccessReview confirms write permission before any write is attempted.
  • Audit record — the outcome is appended to the audit log with the new rollback disposition.

Sample proposal

Rolling back the fix applied at 2026-07-24T06:30:00Z
Proposed rollback: shop/web (Deployment) — roll forward to the pre-fix revision
  reason: undo the fix that rolled shop/web back from revision 5 to 4
  will change:
    revision: 4 → 5
  kubectl equivalent: kubectl -n shop rollout undo deployment/web --to-revision=5
  Roll back? [y/N] y
  rolled back: rolled shop/web forward to revision 5 (pre-fix pod template restored)

Notes

  • One action per invocation. --rollback proposes the inverse of the single most-recent applied record. Re-run to walk further back.
  • Requires --audit-log. --rollback has no way to find the record to undo without the audit log path; the flag is required.
  • Mutually exclusive with --fix. --rollback and --fix cannot be combined in a single invocation.
  • --dry-run is supported. Reports the proposed inverse without writing.
  • Pre-v0.54 records refused. The inverse is derived from structured fromRevision/toRevision fields written into every audit record starting in v0.54. Records written by an older build lack those fields and are refused with a clear message — --rollback never guesses the inverse from free-text detail.
  • Rollback records are not themselves reversible. --rollback only matches applied records; a rollback-disposition record is never the target. If you re-run --rollback after a successful rollback, ReadLast finds the original applied record again and re-proposes the same inverse — the drift bond or the already-cordoned guard will normally refuse it.

JSON output (--output json)

With --output json, the remediation plan is included in the scan result as remediationPlan — an array of proposed actions, each with status: "proposed". The audit log (--audit-log, described above) records the outcome of each of these actions.

{
  "remediationPlan": [
    {
      "kind": "RolloutUndo",
      "target": "shop/web (Deployment)",
      "summary": "roll back to the previous revision",
      "reason": "newest rollout cannot pull its image; a prior revision (1) exists",
      "kubectlEquivalent": "kubectl -n shop rollout undo deployment/web",
      "changes": [
        { "field": "revision", "from": "2", "to": "1" },
        { "field": "image (web)", "from": "registry.example.com/web:v2", "to": "registry.example.com/web:v1" }
      ],
      "status": "proposed"
    }
  ]
}