# Field cases — what broke, where it was fixed, how it was found

This page is the engineering record of the first field run: three production hosts, nine defects in
the tool, and the reasoning that turned each defect into an invariant. Every entry names the
symptom that was visible, the cause that was not, and the measurement that settled it.

The run behind this page: 139 tool calls in total, of which 76 were shell invocations against three
hosts. The first host was audited and remediated by hand before the tool existed; the other two were
remediated with the tool and paid for its defects.

## 1. Timeline of the engagement

| Stage | What happened | Evidence |
|---|---|---|
| Audit of host A | hand-driven: inventory, decode-path mapping, package hunt, install, restart, verification | 60 shell calls, `VERDICT`-equivalent decided by reading output |
| Tool written | four scripts, six documents, one commit, one release | 12 files, 884 lines |
| Field run on host B | the tool failed twice before it ran at all | exit 141, then a shell option dump |
| Field run on host B, second pass | the tool ran, and its restart did nothing | package patched, services still on the old inode |
| Field run on host C | full cycle, `VERIFY: PASS` | clean end-to-end run |

## 2. Case A — the panel daemons that survive every restart

**Context.** Ubuntu 26.04, hosting panel, ten sites, `libbrotli1 1.2.0-3build1` (a rebuild of the
vulnerable upstream release). nginx compressed with `ngx_brotli`; Apache had `mod_brotli` available
and disabled; no component decoded request bodies. The reachable path was outbound HTTP from PHP:
`CURLOPT_ENCODING=''` in WordPress Requests, SimplePie and Guzzle — 164 files.

**Difficulty.** After installing `1.2.0-3ubuntu0.1` and restarting nginx, both PHP-FPM pools, pdns,
`zabbix-agent` and the panel's web server, two processes still mapped the old inode. They were the
panel's `core` daemons, started by the panel launcher through `systemd-run` transient scopes: no
unit, `PPID=1`, not children of the web server. `systemctl restart` cannot address them, and
`kill` risks panel state corruption.

**How it was found.** The stale-inode scan was run as a separate step from the restart, and it
listed the two PIDs with their inode. Had the scan been folded into the restart block, the panel
would have kept the vulnerable decoder with a green report.

**Solution.** The panel's own control tool:

```bash
mgrctl -m core --restart srvparam
mgrctl -m ispmgr --restart srvparam
```

Both daemons respawned on the patched library (new PIDs, new inode), the scan returned zero, and
ten of ten sites answered HTTP 200 with a valid decode round-trip.

**Encoded as.** `remediate.sh` detects a `.scope` cgroup among the consumers and calls `mgrctl`
instead of `systemctl`.

## 3. Case B — three ways to report success while doing nothing

Host B: Debian 12, application server with a graphical session, `libbrotli1 1.0.9-2+b6`.

### 3.1 SIGPIPE on a successful run

*Symptom.* `detect.sh` exited 141 immediately after resolving the apt candidate.

*Cause.* `set -o pipefail` combined with pipelines that end early — `head -1`, `awk '…exit'`. The
producer receives SIGPIPE, the pipeline returns 141, and `set -e` terminates a script that had
already produced a correct answer.

*Measurement.* `bash -x` trace on the host showed the exit immediately after the candidate
resolution.

*Fix.* `pipefail` is omitted, and `common.sh` documents why: correctness rests on explicit checks
and the exit-code contract, not on pipeline status.

### 3.2 The fix that broke everything: `set -euo`

*Symptom.* Every script printed the entire shell option table before running.

*Cause.* `set -euo` parses as `-e -u -o` — `-o` without an argument lists the options. The intent
was "errexit and nounset, no pipefail", which is `set -eu`.

*Lesson.* A repair can be a new defect. The scripts are now syntax-checked on the target
(`bash -n`) and behaviour-checked on a disposable host before any production run.

### 3.3 The restart that restarted nothing

*Symptom.* `remediate.sh` exited 0, printed no error, and every service stayed on the vulnerable
library.

*Cause.* The unit list was assembled with `printf '%s'` — no trailing newline — so all unit names
were concatenated into one string. `systemctl restart` was called with a unit that does not exist;
under the script's error handling that failure was not fatal.

*Measurement.* `/proc/*/maps` on the host still showed the old inode for `apache2` and both
`php-fpm` pools after the "successful" run.

*Fix.* `printf '%s\n'`, plus a post-restart inode check that fails the run instead of trusting the
exit code. The host was then restarted unit by unit and re-verified.

### 3.4 Two state-file defects on the same host

| Symptom | Cause | Fix |
|---|---|---|
| `syntax error` while reading the before-state | the file was sourced as shell and `PRETTY_NAME` contains parentheses | values read by key with `state_get` |
| `VERIFY: FAIL` on a version that had changed | `state.after` also defined `installed`, overwriting the before-value | `OLD_` and `AFTER_` variables kept separately |

### 3.5 The inode that proved nothing

*Symptom.* The stale-mapping check reported "no vulnerable inode mapped" on a host whose services
were demonstrably stale.

*Cause.* The inode was captured from a glob that matched the symlink `libbrotlidec.so.1`, so the
recorded inode belonged to the symlink — an inode no process ever maps.

*Fix.* `find -type f -printf '%i'` on the real file, and the same rule in the verifier.

## 4. Case C — template units, package hooks, and honest deferral

Host C: Ubuntu 24.04, proxy and application server, VPN client, `libbrotli1 1.1.0-2build2`.

**Difficulty 1 — the loop that exited after the install.** A PID loop ended on a non-zero `grep`
when the last PID did not map the library, `set -e` fired, and the script exited 1 *after* the
package had been installed. The host was patched and unverified. Fix: the loop is guarded with
`|| true` and the exit-code contract decides the outcome.

**Difficulty 2 — template instances.** Two `php` processes kept the old library. They were two
instances of a templated unit (`revoap-worker@1` and `revoap-worker@2`) under a custom slice.
The cgroup regex looked for a `.service` segment directly, so it saw nothing. Fix: take the
innermost `.service` segment of the cgroup path, which catches templates and custom slices alike.

**Difficulty 3 — `needrestart` acts first.** On Ubuntu 24.04 the package hook restarted the server
services during installation. The explicit restart step then had nothing to do for those units,
which looked like a defect and was a correct outcome. The report now states that the hook may have
acted.

**Difficulty 4 — desktop and VPN consumers.** GNOME Shell, Xorg, `NetworkManager`, `fwupd`,
`packagekitd` and the VPN client map `libbrotlidec` through freetype and GUI libraries. Restarting
them would drop the session and the tunnel and would not reduce the server attack surface. They are
classified `deferred`: named in the report, not restarted, reloaded at the next login or reboot.
The classification lives in one shared pattern, so detection and verification cannot disagree.

## 5. Defects, in one table

| # | Defect | Visible symptom | Found by | Fixed in |
|---:|---|---|---|---|
| 1 | `set -o pipefail` with early-exiting pipelines | exit 141 on success | `bash -x` trace | `common.sh` |
| 2 | `set -euo` with a dangling `-o` | shell option dump | reading the output | all four scripts |
| 3 | state file sourced as shell | syntax error | the run itself | `state_get` in `common.sh` |
| 4 | `state.after` overwriting `installed` | false `FAIL` | the run itself | `verify.sh` |
| 5 | `printf '%s'` without newline | restart did nothing, exit 0 | `/proc/*/maps` cross-check | `remediate.sh` |
| 6 | inode from a glob (symlink) | stale check always passed | `/proc/*/maps` cross-check | `detect.sh`, `verify.sh` |
| 7 | PID loop without `\|\| true` | exit 1 after install | the run itself | `remediate.sh` |
| 8 | cgroup regex blind to templates | `revoap-worker@1` stayed stale | stale scan | `remediate.sh` |
| 9 | desktop consumers called vulnerable | noise hiding the finding | reading the report | `common.sh` (`DESKTOP_INFRA_PAT`) |

Six of the nine were found by measurement on the host, not by reading the code. Two of those six
would have shipped as false greens.

## 6. What the cases changed in the tool

| Change | Trigger |
|---|---|
| Authenticity gate on the packaged changelog | the distribution changelog endpoint answered 404 for a package published hours earlier, so the version suffix proved nothing |
| Separate stale-inode scan with its own exit path | the panel daemons survived every restart |
| `deferred` status for desktop consumers | the desktop stack maps the same library; a blanket restart takes the machine down |
| Inode from the real file, never the symlink | a check that cannot fail is not a check |
| Post-restart verification inside the remediator | a restart that silently does nothing is worse than an error |
| Unrelated failures reported, not repaired | a `logrotate` failure from an unrelated site configuration appeared right after the change; mixing it in would have made the security change unverifiable |

## 7. Reproducing the evidence

```bash
# stale mapping scan, independent of the scripts
for p in /proc/[0-9]*; do
  m=$(grep libbrotlidec "$p/maps" 2>/dev/null | head -1) || continue
  [ -n "$m" ] && echo "$(cat "$p/comm") $(echo "$m" | awk '{print $5}')"
done | sort | uniq -c
```

```bash
# the restart the remediator must produce: units, innermost service segment
for p in /proc/[0-9]*; do
  cg=$(head -1 "$p/cgroup" 2>/dev/null)
  case "$cg" in */system.slice/*) printf '%s\n' "$cg" | awk -F/ '{for(i=NF;i>=1;i--) if($i ~ /\.service$/) {print $i; exit}}';; esac
done | sort -u
```

Next: [Advisory digest](05-Advisory-Digest) — the same CVE as the outside world saw it.