Projects / Testable (Pramora)
Testable (Pramora) active
testable-platform · last seen 2026-09-29 16:25
Consolidated 12 analysis docs covering 12 taxonomy runs across 5 repos into a 22-item fix plan in 4 buckets (could not run / stopped abruptly / ran without evidence / numbers do not match). Verified 9 items directly in origin/qa source; excluded 3 claims from the source docs as false (there is no Dockerfile.csharp; no Dockerfile uses mcr.microsoft.com/dotnet/sdk:8.0; coverlet_runner.py:159-162 DOES run dotnet test, so "platform cannot execute tests" is wrong). Root-caused the C# SAST outage: catalog_master.py registered semgrep twice at two versions (csharp 1.50.0, python 1.70.0) against one binary, and the readiness probe compares by substring. Reviewed 21 commits now on origin/dev (not yet in QA): 1.1 semgrep versions FIXED and better than prescribed (both entries + image pin all to 1.176.0, from three versions in play); 1.2 borrowed C# tools worked around by reassigning to node/python families (Dockerfile.dotnet still never reads INSTALL_MANIFEST); 1.3 stale readiness snapshot FIXED in tool_readiness_policy.py plus image-digest staleness; 2.2 ESLint FIXED (real cause was npx resolving the repo's ESLint 9, not the --eslintrc flag — my original diagnosis was wrong). Team also found two causes I missed: semgrep-core was being OOM-killed (4 commits), and an empty pytest collection was reported as a tool failure. NEXT: (1) decide on commit 58dd2aaf2, which collapses taxonomy failure detail to the bare string "Tool failed" — every root cause above was read out of that string, so QA loses its diagnostic input; (2) smoke-test a Python repo after promotion, because the version pin moved on both languages at once; (3) diagnose why coverage/mutation tasks never dispatch — 22-26 metrics dark per run, cause still unknown.
todo 3 open
Diagnose why coverage and mutation tasks never dispatch (22-26 metrics dark per run)
high
Every one of the 12 runs reviewed reports Coverlet, altcover, nyc, vitest, Stryker.NET and StrykerJS producing nothing. That is the entire "are the tests any good?" half of the product, dark in every run.
Do NOT accept the claim that the capability is missing. coverlet_runner.py:159-162 runs `dotnet test --collect:"XPlat Code Coverage"`, and altcover_runner.py, stryker_net_runner.py, stryker_ts_runner.py and vitest_coverage_runner.py all exist. Adding a new harness would duplicate working code. (One uploaded consolidated report claimed the opposite and cited runtime_planner.py:120, which is a function that caps planner loop iterations.)
Steps: take one run where these rows read "planned tasks did not start", pull the planner's decision for those specific tool assignments, and establish which of three things happened — the task was never created, it was created and never leased, or it was dropped at the readiness gate. Each has a different fix. Only then write the ticket.
This is fix-plan item 1.6 and is the only item where the cause is genuinely unknown.
backend/wb-cpu-worker/app/analyzers/csharp/coverlet_runner.py:159-162shared/scoring/gate_scoring.py:1692
added 09-29 16:26
· by saravanan@scrumclaw.ai
· claude-cowork
After dev promotion, smoke-test semgrep on a PYTHON repo, not only C#
high
697e0dd47 raised the declared semgrep version to 1.176.0 for BOTH csharp and python. 76032c488 then made readiness permit a mismatch only when observed >= declared.
So any cell still serving an image built with semgrep 1.70.0 or 1.95.0 will now be refused on both languages — where before, Python worked. The image-digest staleness check in 76032c488 should force reconciliation, but that needs verifying rather than assuming.
A C#-only smoke test will not catch this. Run a Python-language repository and confirm semgrep still produces output. Symptom if a cell does not reconcile: semgrep reporting VERSION_MISMATCH on a language that worked the day before.
697e0dd76032c4backend/database/alembic/catalog_master.py:2161,3738shared/persistence/tool_readiness_policy.py:140-210
added 09-29 16:26
· by saravanan@scrumclaw.ai
· claude-cowork
Backlog: Dockerfile.dotnet still never reads INSTALL_MANIFEST
47b5c3a1a worked around fix-plan item 1.2 by reassigning jscpd-cs to the node family and lizard, semgrep, semgrep-perf-static and pydriller to python — a legitimate choice, executed thoroughly (it also fixed the node alias key cve-lite -> cve-lite-cli and updated csharp_tool_readiness.py to return NOT_APPLICABLE for borrowed tools).
But docker/runtime/Dockerfile.dotnet still has no ARG INSTALL_MANIFEST and no COPY of it — confirmed unchanged between origin/qa and origin/dev. Unlike Dockerfile.node:11,24 and Dockerfile.python:15,31, the reconciler generates a manifest for it and the template discards it.
Any future tool assigned to the dotnet family will fail the same way, silently. Not a release blocker; close the gap when the dotnet family next needs a tool it does not already ship.
47b5c3adocker/runtime/Dockerfile.dotnetdocker/runtime/Dockerfile.node:11,24shared/persistence/runtime_install_manifest.py
added 09-29 16:27
· by saravanan@scrumclaw.ai
· claude-cowork
decision 1 open
Decide on 58dd2aaf2 before promoting dev to QA — it hides why a tool failed
high
Commit 58dd2aaf2 adds taxonomyGatePublicErrorNote, collapsing any note starting "Tool failure" or "Tool execution failed" to the bare string "Tool failed", in both the on-screen display and the downloaded HTML export.
Hiding internal tool names from a customer-facing report is defensible. The problem is that it is applied to the only artifact QA uses for diagnosis. Every root cause in the consolidated fix plan was read out of that exact string: jscpd-cs tool_not_ready:MISSING_BINARY, semgrep tool_not_ready:FAILED_STARTUP_CHECK:VERSION_MISMATCH, dotnet_restore_failed:, lizard VERSION_MISMATCH.
It also cuts against fix-plan items 4.1 and 4.2, which ask for MORE provenance per row (tool name, artifact path, raw value), not less.
Recommendation: keep the detail behind an internal or admin view, or export an internal variant alongside the customer one. Do not let the only artifact lose it. This is a product decision, not an engineering one — make it deliberately rather than shipping it by default.
58dd2aafrontend/web-console/src/lib/confidence-engine/taxonomy-gate-display.ts:379-406frontend/web-console/src/lib/confidence-engine/taxonomy-gate-html-export.ts:107,247
added 09-29 16:26
· by saravanan@scrumclaw.ai
· claude-cowork
note 1 open
Correction: fix-plan item 2.2 (ESLint) had the wrong cause — do not act on its original wording
The consolidated fix plan says of ESLint: "--eslintrc was removed in v9; remove the flag or pin ESLint 8."
That is wrong. The platform already pins ESLint 8 in the node image. The real fault, found by the team in 77f8b9d14 (TP-5958), was binary resolution: `npx eslint` resolved the REPOSITORY's ./node_modules/.bin/eslint — version 9 — which rejects the flag the image pin still requires.
Removing the flag, as the plan advised, would have broken the image's ESLint 8 behaviour. Anyone acting on the plan's original wording should stop and read 77f8b9d14 instead.
Note this does NOT address fix-plan item 2.3, the missing JSX parser — no ESLint configuration file is touched in the 53 files changed between qa and dev, so React frontends still return "Unexpected token <".
77f8b9dTP-5958backend/wb-cpu-worker/app/analyzers/javascript/_eslint_common.py
added 09-29 16:27
· by saravanan@scrumclaw.ai
· claude-cowork
Add item
Log
activeConsolidated 12 analysis docs covering 12 taxonomy runs across 5 repos into a 22-item fix plan in 4 buckets (could not run / stopped abruptly / ran without evidence / numbers do not match). Verified 9 items directly in origin/qa source; excluded 3 claims from the source docs as false (there is no Dockerfile.csharp; no Dockerfile uses mcr.microsoft.com/dotnet/sdk:8.0; coverlet_runner.py:159-162 DOES run dotnet test, so "platform cannot execute tests" is wrong). Root-caused the C# SAST outage: catalog_master.py registered semgrep twice at two versions (csharp 1.50.0, python 1.70.0) against one binary, and the readiness probe compares by substring. Reviewed 21 commits now on origin/dev (not yet in QA): 1.1 semgrep versions FIXED and better than prescribed (both entries + image pin all to 1.176.0, from three versions in play); 1.2 borrowed C# tools worked around by reassigning to node/python families (Dockerfile.dotnet still never reads INSTALL_MANIFEST); 1.3 stale readiness snapshot FIXED in tool_readiness_policy.py plus image-digest staleness; 2.2 ESLint FIXED (real cause was npx resolving the repo's ESLint 9, not the --eslintrc flag — my original diagnosis was wrong). Team also found two causes I missed: semgrep-core was being OOM-killed (4 commits), and an empty pytest collection was reported as a tool failure. NEXT: (1) decide on commit 58dd2aaf2, which collapses taxonomy failure detail to the bare string "Tool failed" — every root cause above was read out of that string, so QA loses its diagnostic input; (2) smoke-test a Python repo after promotion, because the version pin moved on both languages at once; (3) diagnose why coverage/mutation tasks never dispatch — 22-26 metrics dark per run, cause still unknown.
cf654db
· saravanan@scrumclaw.ai
from the conversationsaravanan: all these runs are based on the current code base in QA - analyze this and consolidate all the issues with reasons, in layman terms
claude: [consolidated 12 documents, verified against origin/qa, excluded 3 false claims, found the duplicate semgrep catalog registration]
saravanan: review the fixes in dev (not yet pushed to QA) against the report we shared
claude: [reviewed 21 commits; 4 fix-plan items landed, 1 conflicts with the plan, 1 deployment risk]
idleProject initialized: Testable (Pramora)
· saravanan@scrumclaw.ai