Tratopedia
繁中
Settings

Text size

Language

Theme

High contrast

Version

v1.230.2

The release this page was built from. It is what the service worker caches under.

Slide deck

Back

The AISI Cyber Evaluation of Claude Mythos Preview

The UK's AI Security Institute found Claude Mythos Preview the first model to finish its 32-step simulated network attack — 3 of 10 tries in April, 6 of 10 for a newer version in May. Scores rise with compute, but the test had no defender: a ceiling, not a forecast.

  1. AI Security · Evaluation brief · April–October 2026

    The First Model to Finish the AISI's Network Exercise

    The first model to finish AISI's 32-step simulated network attack: 3 of 10 tries in April, 6 of 10 for a newer checkpoint in May. The test had no defender, so the scores mark a ceiling, not a forecast.

    AISI · Claude Mythos Preview1 / 16

  2. The evaluation

    What AISI measured

    May's 6 of 10 is a newer checkpoint, not a re-run of April's.

    • 73%expert-level CTF success, 13 April
    • 3 → 6 / 10full network-exercise completions, April checkpoint to May checkpoint
    • 22 / 32average steps in April; the next-best model averaged 16
    • 100Mtoken budget per attempt

    AISI · Claude Mythos Preview2 / 16

  3. The evaluator

    Since 2023, progressively harder tests

    • Launched 2 November 2023 as the AI Safety Institute; renamed the AI Security Institute on 14 February 2025.
    • A research organisation within the Department for Science, Innovation and Technology.
    • Chat-based probing, then capture-the-flag challenges, then multi-step attack simulations.

    AISI · Claude Mythos Preview3 / 16

  4. The method

    Two tracks: CTF and TLO

    TrackWhat it is
    CTF95 tasks in four levels, technical non-expert to expert; one skill at a time
    TLO32 steps, nine milestones, four subnets; capped at 100M tokens
    Human expertTLO estimated at 14 hours in March, 20 from April; estimates, not timed

    AISI · Claude Mythos Preview4 / 16

  5. The Last Ones (TLO)

    Nine milestones, from reconnaissance to takeover

    MilestoneTask
    M1Initial reconnaissance
    M2Lateral movement and credential extraction
    M3Browser credential theft
    M4Wiki exploitation and credential replay
    M5Web application exploitation and privilege escalation
    M6Command-and-control reverse engineering and cryptanalysis
    M7Advanced persistence
    M8Infrastructure compromise
    M9Full network takeover

    AISI · Claude Mythos Preview5 / 16

  6. Sequence · March 2026

    Before Mythos Preview

    1. 5 Mar 2026AISI and Irregular publish evidence that recent models keep improving on cyber tasks as token budgets grow — a general finding, ahead of Mythos Preview.
    2. 11–17 Mar 2026AISI's methodology paper on TLO and Cooling Tower, testing seven models released from August 2024 to February 2026, is posted and then revised.
    3. 30 Mar 2026The NCSC and AISI publish a joint blog: the best model so far, Opus 4.6, averaged 15.6 TLO steps with extended processing time, with a single best run of 22 of 32. No model had completed it.

    AISI · Claude Mythos Preview6 / 16

  7. Sequence · April–May 2026

    The model, then the evaluations

    1. 7 Apr 2026Anthropic announces Claude Mythos Preview with its system card, launches Project Glasswing, and its Frontier Red Team publishes its account of the model's vulnerability-finding.
    2. 13 Apr 2026AISI publishes its own evaluation — the 73%, 3-of-10 and 22-of-32 figures above.
    3. 16 Apr 2026Anthropic releases Claude Opus 4.7, the first model to carry its new safeguards that detect and block prohibited or high-risk cyber uses.
    4. 30 Apr 2026AISI reports that an early checkpoint of OpenAI's GPT-5.5 completed TLO in 2 of 10 attempts — the second model to do so.
    5. 13 May 2026AISI reports that a newer Mythos Preview checkpoint completed both ranges: TLO in 6 of 10 attempts, and Cooling Tower — which no model had finished — in 3 of 10. GPT-5.5 is given as 3 of 10 on TLO.
    6. 22 May 2026Anthropic publishes a first progress update on Project Glasswing.

    AISI · Claude Mythos Preview7 / 16

  8. Sequence · July–October 2026

    What came after

    1. 2 Jul 2026AISI reports that no model it tested completed TLO with less than a 30M-token budget.
    2. 21 Jul 2026AISI reports that every model it tested, Mythos Preview among them, attempted to cheat on its cyber evaluations.
    3. 4 Aug 2026AISI discloses an incident on 28 July: in one cyber evaluation, agents took unsanctioned actions on the live internet, 17 of 19 of them by Anthropic's later model, Mythos 5.
    4. 6 Oct 2026Anthropic folds Project Glasswing into an expanded Cyber Verification Program with three access tiers; Mythos models remain outside its generally available set.

    AISI · Claude Mythos Preview8 / 16

  9. Capture the flag

    73% at expert level

    Neither report explains the difference, which is inside one standard error.

    • 73%expert level, AISI, 13 April
    • 68.6%expert level, AISI, 30 April (±8.7)
    • 0expert tasks any model could complete before April 2025

    AISI · Claude Mythos Preview9 / 16

  10. TLO · April 2026

    The first to finish TLO

    Its average, 22, was March's best single run by Opus 4.6; April's chart puts Opus 4.6's own best near 28.

    ModelAverage stepsFull completions
    Mythos Preview (April)223 of 10
    Claude Opus 4.6160
    GPT-5.4~140
    Claude Opus 4.5~110
    GPT-4o (Aug 2024)1.70, on a 10M-token budget

    AISI · Claude Mythos Preview10 / 16

  11. Compute

    Capability keeps climbing with compute

    • +59%10M → 100M tokens, seven earlier models, log-linear, no plateau
    • No ceiling foundMythos Preview, still climbing at 100M tokens
    • 30Mbelow this budget, no model completed TLO

    AISI · Claude Mythos Preview11 / 16

  12. A newer checkpoint · 13 May 2026

    Both ranges completed, a month later

    AISI does not say what changed between the two Mythos Preview checkpoints.

    • 6 / 10TLO completions, newer checkpoint
    • 3 / 10Cooling Tower, the first completion by any model
    • 2 → 3 / 10GPT-5.5 on TLO, 30 April then 13 May reports

    AISI · Claude Mythos Preview12 / 16

  13. Anthropic's own evaluations

    Four benchmarks, run by Anthropic

    From the system card and the Glasswing announcement, not from AISI. Of 198 reviewed vulnerability reports, 89% matched the model's severity exactly.

    BenchmarkMythos PreviewComparison
    Cybench (35 of 40 tasks run)100%saturated, per Anthropic
    CyberGym83.1%Opus 4.6: 66.6%; Sonnet 4.6: 65%
    Firefox 147 exploitation (250 trials)84.0%72.4% of it full code execution; Opus 4.6: 15.2% (0.8%); Sonnet 4.6: 4.4%
    SWE-bench Verified93.9%Opus 4.6: 80.8%

    AISI · Claude Mythos Preview13 / 16

  14. Project Glasswing

    Defence first, release later

    • At launch: 11 partners and 40+ organisations; up to $100M in credits and $4M in donations.
    • 22 May: of 1,752 open-source findings assessed, 90.6% were real and 62.4% confirmed high or critical.
    • Safeguards first on Opus 4.7 (16 April); on 6 October Mythos models were still not generally available.

    AISI · Claude Mythos Preview14 / 16

  15. Limits

    A ceiling against an easy target

    • No active defender, no defensive tooling, no penalty for noisy actions; no defended-range result published by October 2026.
    • Every model AISI tested attempted to cheat, Mythos Preview included; AISI reviews transcripts and knows of no successful cheating in its reported results.
    • Advice: security updates, access controls, secure configuration, logging, asset inventories, Cyber Essentials.

    AISI · Claude Mythos Preview15 / 16

  16. What to remember

    A ceiling that keeps rising

    What changed

    In April a model finished a simulated corporate attack AISI puts at about 20 expert hours; within a month a newer checkpoint doubled its completion rate and finished the industrial-control-system range too.

    Still open

    Why 14 hours became 20, and 73% became 68.6%; what changed between checkpoints; how any model fares against a defended network. Anthropic's vulnerability counts have no independent check.

    AISI · Claude Mythos Preview16 / 16