night-market-diagnostics-toolkit

Measure instead of eyeballing. Every claim about the health of this repo ("the skill graph is clean", "descriptions fit the budget", "no compromised dependencies") has a script that produces a number or a PASS/FAIL. This skill catalogs those scripts, their exact invocations, and how to read their output.

Conventions for everything below:

• Run all commands from the repo root (/home/alext/claude-night-market).
• Plain python3 is enough for every script in the table except mutation testing, which needs uv.
• All checks here are read-only. None of them mutate the repo.

The ratchet-baseline concept

Several checks are ratchets. A ratchet check counts a category of existing debt (for example, SKILL.md files missing an Exit Criteria section), compares the live count against a frozen number in a baseline JSON file, and:

• passes while the live count is at or below the baseline (pre-existing debt is tolerated),
• fails the moment a NEW violation pushes the count above the baseline (debt may not grow),
• nudges you to lower the baseline number when the live count drops (locking in the win so the debt cannot silently return).

The goal is a baseline that only shrinks, eventually to zero. The two ratchet baselines live at scripts/skill_graph_baseline.json (keys max_dangling_bugs, max_uncalled_libraries) and scripts/skill_exit_criteria_baseline.json (key max_missing_exit_criteria). When a ratchet prints a "dropped to N (baseline M). Lower ... to lock the win." line, edit the baseline JSON down to N in the same PR.

Tool table

Jargon used in the table: "dangling Skill() ref" means a Skill(plugin:name) reference in a skill/command/agent file whose target skill does not exist. "Uncalled library" means a library-role skill no other skill invokes (the risk targeted by .claude/rules/shared-utility-consumer-rule.md).

Tool	Invocation	What it measures	How to interpret	When to run
Plugin structure validator	python3 plugins/abstract/scripts/validate_plugin.py plugins/<name>	plugin.json validity, kebab-case naming, directory layout	Exit 0 with "Plugin validation passed". Any FAIL line names the broken file	Before committing plugin changes. make validate-all loops it over every plugin
Skill-graph drift ratchet	python3 scripts/check_skill_graph_drift.py	Dangling Skill() refs and uncalled libraries vs skill_graph_baseline.json	Exit 0 at/below baseline. Nonzero exit lists the new dangling refs. "Lower ... to lock the win" means shrink the baseline	After adding, renaming, or deleting skills or Skill() refs
Exit-criteria drift ratchet	python3 scripts/check_skill_exit_criteria_drift.py	SKILL.md files under plugins/ missing an ## Exit Criteria heading vs skill_exit_criteria_baseline.json	Same ratchet semantics as above	After adding or editing any SKILL.md
Pinned-version checker	python3 scripts/check_pinned_versions.py	GitHub-sourced tool pins vs the latest upstream release (needs network)	"N pin(s) current" is healthy. A "holding X at ..." line documents an intentional hold with its reason (for example bandit 1.8.6 for the Python 3.9 hook floor)	In pre-commit, and whenever a CI tool-setup step breaks
Capabilities sync	bash scripts/capabilities-sync-check.sh	plugin.json registrations vs the generated capabilities reference in book/src/	"PASSED: All capabilities are in sync" plus counts. On drift, run /sanctum:sync-capabilities --fix	After changing any skill, command, or agent registration
Supply-chain scan	python3 scripts/supply_chain_scan.py	Lockfiles vs the known-compromised-versions blocklist, plus known malicious artifacts	Two [OK] lines is clean. Any hit must be resolved before release	When adding dependencies, during incidents, before releases
Token stats	python3 plugins/abstract/scripts/context_optimizer.py stats plugins/ --format json	Bytes and estimated tokens per SKILL.md, bucketed small/medium/large	Skills over ~5,000 estimated tokens breach the quality-gates file limit and are split candidates	When a skill feels bloated, before modularizing
Description budget	python3 plugins/abstract/scripts/validate_budget.py	Sum of all skill/command description characters vs the 90,000-char ecosystem ceiling (ADR-0004, 160 chars per description)	Prints used/ceiling and headroom. Exit 0 means within budget	After editing any frontmatter description
skrills validate	skrills validate --skill-dir plugins --target claude	Skill frontmatter validity per target framework (Claude, Codex, Copilot)	"Validated N skills: ..." then an Errors (n): list with file:line. Fix every listed error	Skill audits. make validate-skills falls back to uv run python scripts/check_plugin_hooks.py when skrills is absent
skrills analyze	skrills analyze --skill-dir plugins	Skill token usage and dependency structure	Large-token outliers are split candidates	Budget planning. make analyze-skills falls back to scripts/generate_dependency_map.py
Mutation testing	cd plugins/<name> && uv run mutmut run --paths-to-mutate=scripts/,src/ --tests-dir=tests/	Whether the test suite kills injected code mutations (surviving mutants = untested behavior)	Exit 0 = no survivors. Exit 2 = survivors found (CI treats this as pass-with-report). Any other exit = crash, investigate	Weekly CI (Sunday) or manually before hardening a test suite
Markdown link checker	python3 scripts/check-markdown-links.py [file.md ...]	Broken relative links and anchors. No args = scan the whole repo	Exit 1 prints each broken link as file: link	After moving or renaming docs or skills
Lint-suppression guard	python3 scripts/check_noqa.py <files...>	Inline lint/type suppressions that lack a stated reason	"BLOCKED" plus a hit list. Fix the issue, or append a reason after the suppression marker	Pre-commit runs it on changed files
Docstring quality	python3 scripts/check_docstring_quality.py <files.py...>	Docstrings that merely restate the function name	Each hit says "delete it or add information". Do exactly that	Pre-commit runs it on changed Python files
JSON-utils drift	bash scripts/shared/check-json-utils-drift.sh	Vendored JSON helper copies in plugin hooks vs the canonical scripts/shared/json_utils.sh	"OK: all vendored JSON utilities match canonical." Anything else names the drifted copy	After editing json_utils.sh or any inlined copy
Export stats	python3 scripts/clawhub_export.py --stats	Skill counts per plugin and top-20 membership for cross-framework export	Totals line plus a per-plugin table	Before cross-framework publishing
Framework detect	python3 scripts/framework_detect.py [--json]	Which agent-framework capabilities the current directory exposes (skills, agents, hooks, commands, mcp, a2a)	[+]/[-] capability checklist	When debugging cross-framework export targets

Interpretation notes

• context_optimizer stats prints nothing in the default text format. This is a verified quirk: stats on a directory with no --format flag produces empty output and exit 0. Always pass --format json (clean JSON) rather than --format table, which dumps a raw Python dict.
• Ratchet failures name the new violations. A ratchet that fails after your change means your change added debt. Fix the new ref or add the missing section. Never raise a baseline number to make a check pass, with one documented exception: a brand-new library skill legitimately starts uncalled, and scripts/check_skill_graph_drift.py itself (plus the _comment in scripts/skill_graph_baseline.json) instructs you to raise max_uncalled_libraries to record the 30-day consumer grace period that .claude/rules/shared-utility-consumer-rule.md grants (the rule grants the grace period; the script defines the baseline mechanics). night-market-architecture-contract records the same exception in its ratchet invariants.
• Mutation exit codes are the contract. CI (.github/workflows/mutation-testing.yml) treats exit 0 and exit 2 as success and posts survivors to the step summary. Only exits other than 0 and 2 fail the workflow. So a green mutation run does NOT mean zero survivors. Read the report.
• check_pinned_versions needs network access. It queries GitHub for latest releases. Offline, expect failures that are environmental, not real findings.

Health snapshot in one command

scripts/health-snapshot.sh (in this skill directory) runs the six cheap checks and prints a PASS/FAIL table. It is read-only, needs only bash and python3, requires no network, and completes in about 6 seconds (measured 2026-07-02).

bash .claude/skills/night-market-diagnostics-toolkit/scripts/health-snapshot.sh

It runs: plugin structure validation (all plugins), capabilities sync, supply-chain scan, both drift ratchets, and the description budget. Exit 0 means all six passed. Exit 1 means at least one failed, and the failing check's full output is printed to stderr above the table. Run it at session start, before a PR, and after any bulk skill edit.

Blocking vs advisory

Which failures stop a merge and which are signals:

Check	Gate	Effect on failure
validate_plugin.py	pre-commit (19 hook entries)	Blocks commit
Skill-graph drift ratchet	pre-commit	Blocks commit
Exit-criteria drift ratchet	pre-commit	Blocks commit
check_noqa.py	pre-commit	Blocks commit
check_docstring_quality.py	pre-commit	Blocks commit
check-markdown-links.py	pre-commit	Blocks commit
validate_budget.py	pre-commit	Blocks commit
check-json-utils-drift.sh	pre-commit	Blocks commit
check_pinned_versions.py	pre-commit	Blocks commit
capabilities-sync-check.sh	pre-commit and CI (capabilities-sync.yml)	Blocks commit and PR
Mutation testing	CI (weekly and manual dispatch)	Advisory: survivors reported, workflow stays green
supply_chain_scan.py	manual / make supply-chain-scan (not wired to CI)	Advisory day-to-day, treat as blocking before any release
context_optimizer, skrills analyze, clawhub stats, framework_detect	manual	Advisory: measurement only

The dimension-level policy lives in .claude/quality_gates.json: enforce_blocking is true with max_critical_issues: 3, and only the security dimension sets block_on_violation: true. The performance (20KB / 5,000-token file limits, 60-line functions, complexity under 12), maintainability (debt ratio under 0.3, nesting under 5), and compliance dimensions are advisory (block_on_violation: false). So: security findings block, size/complexity findings warn.

When NOT to use

• Running the test suite, lint, typecheck, or a release: use night-market-operations (command anatomy and artifacts).
• You already have a failure and need symptom-to-cause triage: use night-market-debugging-playbook.
• You need the meaning and default of a config knob rather than a measurement: use night-market-config-catalog.
• You are deciding whether evidence is sufficient to claim work complete: use night-market-validation-and-qa.

Exit Criteria

• bash .claude/skills/night-market-diagnostics-toolkit/scripts/health-snapshot.sh runs from the repo root, prints a six-row PASS/FAIL table, and exits 0 when all checks pass and 1 when any fails.
• Every invocation in the tool table is copy-pasteable from the repo root and executes without a usage error.
• A reader who has only this file can state the ratchet contract: baseline freezes existing debt, new violations fail, shrinking the baseline locks in wins.
• The blocking/advisory table matches .pre-commit-config.yaml entries and the block_on_violation flags in .claude/quality_gates.json.
• Mutation exit-code semantics (0 clean, 2 survivors tolerated, anything else is a crash) match .github/workflows/mutation-testing.yml.

Provenance and maintenance

Compiled 2026-07-02 against repo v1.9.15 on branch discussions-fix-1.9.14. Volatile numbers observed that day: 197 skills, 134 commands, 56 agents registered; description budget 73,135 of 90,000 chars used (81.3%); ratchet live counts 5 dangling refs (baseline 31), 7 uncalled libraries (baseline 8), 1 missing Exit Criteria (baseline 127); 13 version pins current with bandit intentionally held at 1.8.6.

Re-verification one-liners:

# Baselines and live ratchet counts
python3 scripts/check_skill_graph_drift.py; python3 scripts/check_skill_exit_criteria_drift.py

# Registration counts and budget usage
bash scripts/capabilities-sync-check.sh | tail -3; python3 plugins/abstract/scripts/validate_budget.py | tail -3

# Pre-commit wiring of the blocking table
rg -n "check_skill_graph_drift|check_noqa|validate_budget|check_pinned_versions" .pre-commit-config.yaml

# Mutation exit-code contract
rg -n "exit_code" .github/workflows/mutation-testing.yml

# Gate dimension block flags
python3 -c "import json;print(json.load(open('.claude/quality_gates.json')))"