How Numero Tested Its CASA Security Controls

The evidence matrix gave each requirement a status, a plain-language developer statement, source evidence, runtime or operational evidence, and an owner. Code permalinks were used for implementation controls. Provider settings needed screenshots. Runtime behavior needed a transcript or scan. Operational controls needed dated records rather than a code comment.

This prevented one large report from being cited everywhere. A reviewer assessing whether cookies are HttpOnly should not have to infer that from an unrelated dependency scan. The relevant item instead receives the cookie-emission source, a real login response, and the specific ZAP passive rule result.

Reviewer feedback frequently said the implementation claim was plausible but unproven. Session rows originally cited an unauthenticated frontend baseline and even named a companion script that did not exist. A cryptography row asserted several primitives while showing only AES-GCM. An access-control row proved one router did not serve directories but did not show the top-level website and API surfaces.

Rebuilding those submissions found missing tests that deserved to exist permanently. New cases actively forged alg:none and wrong-algorithm JWTs, compared cryptographic failure responses, exercised OAuth expiry and replay across providers, checked directory-style paths, verified real browser storage clearing, and pinned password-hash properties. Evidence work improved the regression suite because it forced each sentence to have a testable meaning.

General-purpose scanners are strongest when paired with focused assertions. Numero's scripts logged into a disposable test account and verified that passwords and session tokens did not appear in URLs, that logout invalidated both access and refresh tokens, and that a password change terminated an older session. Another script sent eleven forged JWT variants and confirmed every modified token was rejected while the untouched control token remained valid.

Separate production checks tested cross-site Origin rejection, directory-style and dot-file paths, SSRF-shaped inputs, and SQL-injection-shaped search parameters. The scripts saved sanitized transcripts and exited non-zero when an assertion failed, turning a screenshot into the output of a repeatable check rather than a manually selected success response.

The continuous-integration ZAP baseline remained useful, but it was passive and unauthenticated. It crawled public production-build pages and caught security-header regressions. It could not prove what happened after login or actively exercise the injection controls named by CASA.

Operator-run OWASP ZAP Automation Framework plans therefore authenticated to bounded website and API scopes. The data-validation scan limited the API to a safe OpenAPI slice, enabled the relevant injection families, and excluded destructive routes. The session scan paired passive cookie and URL checks with the focused token-forgery matrix. Each plan recorded its configuration hash and generated standard plus passing-rules reports.

On August 30, the authenticated Section 2 run reported zero high, medium, or low findings, with its named cookie, URL, and JWT rules under the evidence report's passing set. The Section 5 API pass covered 26 endpoints including a representative JSON write shape and reported zero high, medium, low, or false-positive findings; the frontend injection pass was also clean. Section 6 configuration and CORS testing reported no high, medium, or low findings across the tested plans.

The Section 3 scan initially reported that a public support form lacked an anti-CSRF token. That observation was true, but the vulnerability conclusion was not: Numero's unsafe requests use an Origin-verification control, and the companion script showed forged, absent, suffix-matched, downgraded, and null Origins were rejected before the handler ran.

The alert was scoped and dispositioned as a false positive with the reason and black-box proof attached. The report retained a separate false-positive count while high, medium, and low remained zero. A similar hit anywhere outside the reviewed paths would still fail the run. Transparency here matters more than forcing every report counter to zero.

These results describe the routes, rules, versions, and dates that were tested. They are not a penetration-test guarantee and do not remove the need for patching, monitoring, responsible disclosure, and retesting after meaningful application changes.

The final evidence sets use a consistent sequence: statement, relevant source, scan setup when applicable, actual report result, and a focused black-box or regression-test result. Generated images are stamped with source path, line range, commit, and caption, and generation fails when cited lines drift.

Sensitive raw scanner history, live tokens, session-bearing project files, customer data, production secrets, internal addresses, and origin details are not committed for public consumption. A reviewer gets enough context to validate the claim without turning the evidence pack into a map of credentials or customer records.

The strongest lesson was simple: proof should be current, specific, reproducible, and no broader than the test that produced it. That standard is useful well beyond CASA because it makes future security changes easier to review and harder to hand-wave.