From ea038b5ed2eaab3ebd654ea60319128788cf7719 Mon Sep 17 00:00:00 2001 From: Siavash Sameni Date: Sun, 7 Jun 2026 11:06:58 +0400 Subject: [PATCH] =?UTF-8?q?docs:=20sync=20from=20backend=209363d8c=20?= =?UTF-8?q?=E2=80=94=20Woodpecker=20Docker=20ENOSPC=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- 09 - Audits/Activity Log.md | 10 ++++++++++ .../Smoke and Regression Procedure.md | 19 +++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/09 - Audits/Activity Log.md b/09 - Audits/Activity Log.md index c6691e4..bbd3a3c 100644 --- a/09 - Audits/Activity Log.md +++ b/09 - Audits/Activity Log.md @@ -12,6 +12,16 @@ entries on top. Maintained by agents per the rule in `../AGENTS.md`. --- +### 2026-06-07 — backend@9363d8c, frontend@03025c8 — Woodpecker Docker ENOSPC cleanup hardening + +**Commits:** `e2c74f9` `d640ec1` `9363d8c` `03025c8` +**Touched:** backend `Dockerfile.prod`, `.woodpecker/production.yml`, `package.json`, `package-lock.json`; frontend `.woodpecker/production.yml`, `Dockerfile`, `package.json`; docs `09 - Audits/Activity Log.md`, `11 - Testing/Smoke and Regression Procedure.md` +**Why:** Woodpecker CI showed repeated Docker build `ENOSPC` failures across backend and frontend. Backend `89` failed in `get-version` from Mojtaba's malformed `package.json` and was repaired by the later backend version sync. Backend `87/88/90` and frontend `83` failed in Docker build/install layers because the arm64 CI host had severely exhausted Docker storage (`241GB` images with `237.6GB` reclaimable, plus `17.19GB` reclaimable build cache). Backend Docker build now avoids the second production-stage `npm ci` by pruning dev dependencies after transpile and copying the pruned runtime `node_modules`. Backend and frontend production pipelines now serialize Docker cleanup with a shared lock, prune old unused tagged images and build cache with timeouts, and avoid Docker volumes. +**Verification:** `woodpecker-cli` investigation with local `WOODPECKER_SERVER`/`WOODPECKER_TOKEN`; backend/frontend `woodpecker-cli lint .woodpecker/production.yml`; backend `DOCKER_BUILDKIT=1 docker build -f Dockerfile.prod -t escrow-backend-docker-smoke:2.9.40 .` passed and the smoke image was removed; backend/frontend scoped `git diff --check`; backend/frontend version metadata confirmed at v2.9.41. Follow-up Woodpecker pipelines `backend#92` and `frontend#85` are pending because the earlier killed frontend `#84` left `build-and-deploy` reported as running on the arm64 agent; agent restart or manual kill may be needed before the queued pipelines can prove green. +**Linked docs updated:** [[11 - Testing/Smoke and Regression Procedure]] + +--- + ### 2026-06-07 — frontend@a433067, backend@9427009 — frontend Docker Yarn cache ENOSPC fix **Commits:** `a433067` `9427009` diff --git a/11 - Testing/Smoke and Regression Procedure.md b/11 - Testing/Smoke and Regression Procedure.md index b705d0c..dabfdd1 100644 --- a/11 - Testing/Smoke and Regression Procedure.md +++ b/11 - Testing/Smoke and Regression Procedure.md @@ -146,6 +146,25 @@ export WOODPECKER_TOKEN= Never commit or paste the token into docs. +When Docker build steps fail with `ENOSPC`, inspect logs and host storage with +Woodpecker before retrying blindly: + +```bash +cd ~/CascadeProjects/escrow +set -a; source .env; set +a +woodpecker-cli pipeline ls --limit 5 5 +woodpecker-cli pipeline ps 5 +woodpecker-cli pipeline log show 5 build-and-deploy +``` + +The backend and frontend production pipelines run local Docker builds on the +arm64 agent. Their cleanup block must remain serialized with +`/opt/escrow-dev/.ci-docker-prune.lock`, must avoid Docker volume pruning, and +must wrap old-image/build-cache pruning in `timeout` so the agent cannot hang in +cleanup. If a killed pipeline still shows a step as `running`, queued builds may +stay pending until the Woodpecker agent process or the stuck Docker prune process +is restarted on the host. + Poll latest pipelines by repo id: | Repo | Woodpecker repo id | Expected pipeline |