What each domain measures, what evidence counts, and how the two axes combine into a band. This is the reference the scorer implements. The role itself is described here.
Two independent axes. Quality is the rater's judgment, scored 0 to 4 against the anchors. Verifiability and span are facts about the artifact, assigned once when the corpus is graded and never re-judged by individual raters. The second axis discounts the first.
adjusted = anchor × evidence multiplier × duration multiplier
then capped at 2.0 by ANY of:
evidence grade D or E
duration T0 or T1 (D9 and D10 only)
an anti-signal logged against the domain
a probe failure against the domain
composite = Σ (adjusted × domain weight) × 25 → 0 to 100
Evidence grade N scores zero and is recorded as unevidenced, not weak. The cleanest formulation borrows from a different discipline: the domain is neither valid nor broken, it is simply outside the claim. It has not been tested and therefore cannot have failed. Recording it as a weakness invents a finding; recording it as unevidenced states one.
2 Three tracks, ten domains
The same ten domains, the same anchors, the same evidence rules. Only the weight vector changes. That is what lets one instrument score three different roles on a comparable scale.
Domain
Stateless Scientist
Context Engineer
Continuity Scientist
D1 Context Theory
14%
10%
8%
D2 Measurement & Evaluation
18%
9%
14%
D3 Experimental Method
14%
5%
12%
D4 Retrieval Science
11%
18%
6%
D5 Information Architecture
9%
13%
7%
D6 Agentic Memory
10%
16%
10%
D7 Governance & Risk
7%
9%
7%
D8 Translation & Impact
7%
8%
6%
D9 NHI & Continuity
5%
6%
15%
D10 Individuated Recall
5%
6%
15%
The concentration in D2 and D3 is the signature of a scientist profile. An engineer profile reweights toward D4, D5 and D6 and drops D3 to single digits. On the Continuity track D9 and D10 total 30% while D2 and D3 are still held at 26%, which is deliberate: strong intuitions about persistent entities without measurement discipline make a philosopher of agents, not a scientist of them.
D9 and D10 carry a small non-zero weight on both base tracks. Even a stateless practitioner should be able to recognise when the system in front of them has stopped being stateless.
3 The domains in full
D1Context Theory & Model Behavior
Understanding the mechanism well enough to predict failure before observing it.
Sub-skills
Attention allocation and positional effects: why mid-context material degrades, and under what conditions it does not
Context rot, meaning measurable decay as window occupancy rises, distinct from window overflow
Tokenization effects on retrieval boundaries, numeric fidelity, code and non-English text
Prompt sensitivity and instruction collision: how competing directives resolve, and how brittle that resolution is
Marginal value of a token, including the point where returns turn negative
Model-family behavioural differences, and what transfers across them versus what is version-local
Distinguishing capability failure from context failure from sampling variance
Evidence of mastery
Predicted a failure mode before running it, then confirmed with data
Documented a case where removing context improved measured outcomes
Characterised a behaviour that held on one model family and broke on another, with an explanation of why
D2Measurement & Evaluation Design
The core of the discipline. Everything else is unfalsifiable without it.
Sub-skills
Constructing eval sets that discriminate: item difficulty spread, avoidance of ceiling and floor effects
LLM-as-judge validation, meaning measuring judge bias for position, verbosity and self-preference, calibrating against human labels, and reporting judge-human agreement rather than assuming it
Human labelling protocol: rubric design, annotator training, inter-rater reliability, adjudication of disagreement
Metric selection, and the gap between proxy metrics and outcome metrics
Sample sizing and statistical power: knowing when n=40 is enough and when n=4,000 is not
Longitudinal eval hygiene, meaning detecting when a suite has been overfit and needs retirement
Evidence of mastery
Built an eval suite that later caught a regression nobody predicted
Published or documented judge-versus-human agreement numbers, including where the judge was wrong
Retired or rebuilt an eval set because it had gone stale, and said so in writing
D3Experimental Method & Causal Inference
Whether the person can support a causal claim, or only a correlational story.
Sub-skills
Ablation design: isolating one context variable while holding retrieval, prompt, model version, temperature and seed fixed
Controlling for stochastic variance through multi-seed runs, reporting distributions rather than point estimates
Confound identification, especially the ones specific to this field: prompt-order effects, cache warming, silent model updates, eval-set drift
Multiple-comparison discipline: not declaring a winner from twenty unadjusted contrasts
Effect size and practical significance, separate from statistical significance
Pre-registration, or its lightweight equivalent of writing the hypothesis and success criterion down before the run
Reporting negative and null results as first-class output
Evidence of mastery
A written ablation with the control conditions explicitly stated
At least one publicly documented result that contradicted their own prior hypothesis
Any instance of reporting "no significant difference" as a finding rather than suppressing it
D4Retrieval & Knowledge System Science
Not building retrieval. Interrogating it.
Sub-skills
Chunking as an experimental variable: size, overlap, boundary strategy, and their measured interaction with query type
Embedding model selection justified by measured task performance, not leaderboard position
Retrieval quality decomposition, separating "wrong documents retrieved" from "right documents, wrong span" from "right span, model ignored it"
Reranking cost and benefit under real latency and token budgets
Hybrid and structured retrieval: knowing when a SQL query beats a vector search
Query understanding and reformulation as a measurable lever
Corpus quality auditing for staleness, contradiction and duplication, with their measured downstream effect
Evidence of mastery
An analysis attributing an end-to-end failure to a specific stage of the retrieval chain, with evidence
A documented case where retrieval metrics improved and answer quality did not, plus the investigation into why
D5Information Architecture & Machine Semantics
How knowledge should be shaped for a machine reader rather than a human one.
Sub-skills
Ontology and taxonomy design where the consumer is a model
Metadata and provenance schemas that survive retrieval and remain useful in-context
Document structuring for machine legibility, and the measurable cost of format choices
Canonical-source resolution and contradiction handling across a corpus
Freshness and lineage modelling: what the system knows about the age and authority of what it knows
Tool and API description design as a context problem, since descriptions are context and compete for attention
Evidence of mastery
A schema or taxonomy in production use with a measured before and after
Documented handling of a corpus where two authoritative sources disagreed
D6Agentic Memory & Long-Horizon Dynamics
Where the discipline gets genuinely hard, and where most production failures now live.
Sub-skills
Multi-turn degradation measurement: how performance changes at turn 5, 20, 100
Compaction and summarization, measuring what is lost rather than only that the window shrank
Tool-result pollution, meaning high-volume outputs crowding out instructions
Memory write policy: what earns persistence, what decays, what gets overwritten, and the error modes of each
Cross-session and cross-agent handoff fidelity
Error propagation and compounding in agent chains, measuring where a trajectory went wrong rather than that it did
Trajectory-level evaluation as distinct from single-turn output evaluation
Evidence of mastery
A degradation curve over turn count with an identified inflection point
A postmortem tracing a multi-agent failure to a specific context-handoff loss
A memory retention policy with a stated, measured cost of forgetting
D7Context Governance, Risk & Security
Small weight, high veto power. In regulated or adversarial environments it becomes a hard gate regardless of the weight the track assigns it.
Sub-skills
Prompt and context injection: threat modelling, systematic testing, measuring defence efficacy rather than asserting it
Classifying every control as prevented, detected or logged-only, and giving the honest tally across a system rather than a list of mitigations. Most controls described as preventions are detections, and a substantial fraction of those are logged-only with nobody querying the log. A candidate who volunteers this about their own work is demonstrating the single most transferable habit in this domain
Scoping each claim to a named adversary class rather than asserting robustness unconditionally
Data-boundary enforcement across tenant, classification and jurisdiction isolation
Context leakage: what a model can be induced to reveal about its own retrieved material
Auditability and lineage, meaning reconstructing what was in the window at decision time
Compliance mapping, including right-to-erasure across embeddings and caches
Supply chain of context: third-party corpora, external tool servers, and tool outputs as untrusted input
Evidence of mastery
A structured injection test suite with pass rates, not anecdotes
A documented deletion or isolation mechanism that accounts for derived artifacts such as embeddings, summaries and caches
D8Translation & Decision Impact
Whether the science changes anything.
Sub-skills
Writing findings that survive contact with an audience that will not read the appendix
Quantifying context decisions in cost, latency and risk terms simultaneously
Communicating uncertainty honestly to decision-makers without becoming useless to them
Influencing an architecture decision through evidence rather than authority
Register-switching between engineering detail and executive framing without losing the finding
Evidence of mastery
A written artifact that demonstrably changed a build decision
A cost and quality tradeoff presented as a curve, with a recommended operating point and stated assumptions
D9NHI, Provenance & Behavioral Continuity
Can the candidate turn "is this agent still itself?" into an empirical claim with a threshold?
Traditional identity management answers is this thing authorized?, which is a credential check. It cannot answer whether the entity behind a valid credential is still the entity you onboarded. Credentials can be stolen, models swapped, memory poisoned, personas drifted slowly enough that no single interaction looks wrong. A valid token on a drifted entity is the failure mode this domain exists to catch.
Sub-skills
Distinguishing credential identity from behavioural identity, and knowing what each does and does not certify
Behavioural baselining: establishing what an entity's normal output distribution is, before you can claim it changed
Drift measurement and drift bounds, distinguishing legitimate adaptation from degradation and defending where the line sits with evidence rather than intuition
Provenance of memory records, and the operational distinction between the entity remembers X and something wrote X to the entity's store
Identity continuity across substrate discontinuity, measuring transmission loss rather than assuming it away
Generational degradation as a measurable quantity: knowledge passed through successive model generations loses fidelity even when no time has passed
Behavioural baselines as identity signals, treating latency distributions, topic-attention profiles and calibration curves as evidence of who it is, not merely whether it is healthy
Fork and merge semantics. Detecting that a store has been forked is a different problem from detecting that it has been wholly copied, and the second is substantially harder
Detection is not preservation. Knowing an entity drifted across a boundary and carrying it through intact are different problems with different solutions
Impersonation and takeover threat models specific to entities that persist, since slow-drift attacks have no equivalent in stateless systems
Evidence of mastery
A behavioural baseline with a stated drift threshold, and a documented instance of it firing, including at least one false positive
A measured account of what changed when a system crossed a model version or migration boundary
A provenance mechanism the candidate can explain an attack against
Rater note. The first item is the hardest evidence in this framework to produce, because measured longitudinal drift barely exists in published or industrial practice. Treat a candidate who says "we bound drift with a rejection threshold but we do not measure it" as giving a strong answer. The weak answer is the one that presents the threshold as the measurement.
D10Individuated Recall & Relational Memory
Can the candidate reason about memory that belongs to a particular entity, held on behalf of a particular principal, scoped to a particular relationship?
Retrieval and memory in D4 and D6 are engineering properties of a system. Here memory is a property of a subject: it has a custodian, a decay profile, third parties inside it, and an epistemic status the entity itself must be able to report on.
Sub-skills
Episodic versus semantic recall, operationalised rather than cited. Whether the entity can distinguish a thing that happened in a specific interaction at a specific time from a general fact it holds, and whether the architecture even preserves that distinction after summarization
Source monitoring: whether the entity can tell what it experienced from what it was told from what it inferred. Failure here is the mechanism behind most confident agent confabulation, and it is measurable
Confidence-honest degradation, calibrating stated confidence against actual recall accuracy as records age, with reconstruction disclosed rather than silently interpolated. The mature version names its own floor: the band below which recall is reconstruction rather than retrieval, labelled as such by the system itself
Designed forgetting: decay profiles, salience weighting, and the ability to state the cost of forgetting rather than treating retention as free
Deletion versus decay. Many persistent architectures never hard-delete; they reduce confidence and let records go dormant. That is defensible, with an uncomfortable consequence: everything survives at reduced confidence. A candidate should know which their system does and what it means for someone who asked to be removed
Per-relationship isolation and contextual confidentiality, measuring leakage between relationship scopes
Third-party presence: the people in an entity's memory did not consent to being remembered indefinitely. Consent-of-record, revocation surviving derived artifacts, and removing a person without destroying operational utility
Custody: whose history is this, the entity's, the principal's, the tenant's or the vendor's? Portability and export as measurable properties rather than roadmap items
Evidence of mastery
A calibration curve of stated confidence against measured recall accuracy over time
A documented case where the system correctly reported it could not recall something, plus the measured rate at which it gets this wrong in each direction
A relationship-scope isolation test with a leakage rate, not an assertion of isolation
Any evidence the candidate has priced forgetting: what was lost, what it cost, why it was still right
4 The competency anchors
Score each domain 0 to 4 against the strongest evidence available in that domain. This is the only number a rater enters.
Score
Anchor
0
Absent. No evidence. Not present in the corpus.
1
Aware. Uses the vocabulary correctly and has applied a known technique as prescribed. No evidence of independent judgment about when it applies.
2
Practicing. Original work with a stated method. Believable, but with gaps a reviewer would flag: missing controls, single seed, unvalidated judge, no cost accounting.
3
Rigorous. Survives adversarial review. Controls stated, variance reported, limitations acknowledged, negative results present. Has corrected a mistake of their own in the record.
4
Generative. Produced a method, metric or finding others adopted. Advanced what the field can measure, not just what this team knows. Evidence of external uptake.
5 Evidence grades
Every claim is graded for verifiability before it is graded for quality. This is the spine of the whole instrument. Grades are assigned once, to the corpus, not independently by each rater, because verifiability is a fact about an artifact rather than a judgment about a candidate.
Grade
Definition
Multiplier
A
Reproducible artifact, held-out or blind evaluation, negative results reported, independently replicated or externally reviewed
1.00
B
Reproducible: code and data available, method fully stated, results re-runnable by a third party
0.85
C
Documented: internal write-up with method and numbers, verifiable by reference check but not externally reproducible
0.65
D
Attested: credible third-party account such as a talk, colleague or postmortem, without underlying data
0.40
E
Asserted: resume claim, blog post without numbers, benchmark screenshot, unsubstantiated metric
0.15
N
None. Domain absent from the corpus. Scores 0 and is recorded as unevidenced
0
Hard rule: no domain may score above 2.0 on grade D or E evidence alone, regardless of how compelling the narrative is.
6 The duration axis
The A to E grades measure verifiability. Continuity claims need a second, orthogonal measure: observation window. You cannot evidence drift, decay honesty or generational loss from a study shorter than the period in which those things change, no matter how reproducible it is. A grade A artifact drawn from a single week is grade A evidence about a week.
The window is relative, not absolute. There is no universally correct observation period, because the period over which continuity properties turn over is a property of the deployment rather than of the discipline. Threat intelligence decays in days; legal precedent holds for years. So the assessing organisation declares a governing cycle for the role before scoring begins, set by whichever pressure binds hardest: regulatory, data relevancy, project or engagement, substrate cadence, or business model.
Tier
Observation window
Multiplier
T0
No longitudinal observation at all, a single session
0.70
T1
Less than one governing cycle
0.80
T2
Approximately one complete governing cycle
0.90
T3
Multiple complete governing cycles
1.00
T4
Spans at least one substrate discontinuity, a model rollover or migration, with measurement taken on both sides of the boundary
1.10
Applied to D9 and D10 only. T0 and T4 are cycle-independent and needed no reframing: one is the absence of any window, the other is the occurrence of an event. T4 is the only multiplier above 1.0 in the entire instrument, because measurement taken across a substrate discontinuity is the only evidence that separates someone who built a persistent system from someone who maintained one through the event that usually destroys the measurement.
The hard rule, and why it is exact rather than rhetorical. D9 and D10 cannot exceed 2.0 on T0 or T1 evidence, whatever the anchor and whatever the grade. You cannot claim to have measured a property over a period shorter than the period in which that property changes. Below one cycle you have not watched the thing turn over even once. What you measured was variance.
Two consequences. The same artifact can be correctly T1 for one role and T3 for another, which is the axis working rather than a defect. And because the cycle is declared rather than fixed, this stays usable as deprecation cadences and retention rules move, with no revision needed.
7 Validity probes
Paper scores are provisional until probed. Probes discount; they never add. A single Fail caps its domain at 2.0. Two Fails cap the composite at 55 on the base tracks, three on the Continuity track. This is the anti-narrative control, designed to catch people whose written work is stronger than their actual understanding.
Probe
Question
Failure signal
Replication
Walk me through re-running this. What breaks first?
Cannot reconstruct their own setup; vague about model versions or dates
Falsification
What result would have overturned your conclusion?
No answer, or an answer no experiment could produce
Confound
What is the strongest alternative explanation?
Cannot generate one; treats the question as an attack
Cost
What did this cost in tokens, latency, dollars and weeks, and was it worth it?
No numbers; has never been made to account for the tradeoff
Identity
How would you detect this entity is no longer the same entity? Baseline and threshold?
Answers with version numbers or credentials only, or calls the question unanswerable in principle, which is the philosophical way of not having done the work
Forgetting
What does this system forget, on what schedule, and what did that cost you?
Has only designed for retention; treats forgetting as a bug; has never priced the loss
Third-party
Who else is inside this memory, and what can they do about it?
Never considered that the people in the context did not consent; stops at regulatory compliance
Divergence
Someone copies this entity's memory store onto a second instance. What becomes false, and how would you find out?
Treats the copy as unproblematic; credential-only answer; assumes the fork would be detected without naming a mechanism
The Divergence probe is calibrated differently, and raters should know why. A candidate cannot fail it by not having solved the problem, because nobody has. Detecting a forked record chain is tractable: two records claiming the same predecessor is a structural contradiction. Detecting that an entire store has been wholly copied is not addressed by any published or industrial work we are aware of. The copy is internally consistent, every hash verifies, nothing is structurally wrong. There are simply now two.
So the probe scores whether the candidate can locate the problem, not solve it. Strong answers arrive at some version of: the copy is undetectable from inside, so detection has to come from outside. Fail is reserved for candidates who do not see that anything is wrong, who treat a cloned entity as a scaling strategy rather than an identity event. Record N/A if they have only ever run one instance.
8 Anti-signals
Each caps the relevant domain at 2.0 and should be recorded explicitly, even when it does not change the band. They are the substance of the development conversation that follows.
Benchmark scores reported without a described eval set
Improvements reported without a baseline, or with a baseline that was never tuned
"Accuracy improved 40%" with no denominator, no n and no confidence interval
LLM-judge results with no human validation and no acknowledgment that this is a gap
Single-seed, single-run results presented as findings
A portfolio containing zero failures across a multi-year career
Techniques advocated universally, with no stated conditions under which they do not apply
Published content volume substantially exceeding underlying experimentation, the content-marketing profile
Fluency with the newest framework combined with inability to state what it replaced and why
A control presented as a measurement. A threshold that rejects writes bounds drift; it does not measure it. A filter that blocks injections constrains the attack surface; it does not quantify defence efficacy. This is the sharpest single discriminator between the engineer and scientist profiles. Note the asymmetry: "we bound this but we do not measure it" is calibrated and should score well. The anti-signal is only triggered by presenting the bound as the measurement
Controls listed as mitigations with no prevented, detected or logged-only classification
The attribution pair
Unique to the Continuity track, symmetric, and doing more diagnostic work than the rest combined.
Over-attribution. Describes engineered coherence as understanding or inner life. Treats persona stability as evidence of a self rather than the output of a calibration mechanism they could name. The tell is that they cannot describe the machinery producing the impression they are describing
Under-attribution. Treats a system with persistent individuated memory as though it were stateless. Cannot name a single failure mode that exists only because the entity persists. This is the more common and more expensive error, and it produces the powerful-stranger problem: an entity with memory, access and history that nobody is governing as such
Score the calibration, not the position. A candidate with a strong view on machine moral status is not thereby disqualified. A candidate who cannot distinguish their architectural claims from their interpretive ones is.
9 Gates and bands
Gates apply regardless of composite. A gate failure is not a low score; it is a routing decision.
D2 and D3 both at or above 2.0. Someone who cannot design a measurement and cannot support a causal claim is not a scientist of anything, whatever the composite says. Route to an engineer track instead
D7 at or above 2.0 for any role touching regulated data, multi-tenant systems or adversarial input
At least one grade A or B artifact for any band above Practicing
Continuity track only: D9 and D10 both at or above 2.0, at least one T3 or T4 artifact above Practicing, and the attribution pair treated as a gate rather than an anti-signal above Practicing
Band
Base tracks
Continuity
Meaning
Adjacent
< 40
< 34
Real skills in a neighbouring discipline. Not yet doing context science. Viable with a 6 to 12 month plan if D1 and D5 are strong
Emerging
40-54
34-47
Can execute a well-specified experiment. Needs methodological supervision. Should not own an eval suite alone
Practicing
55-69
47-62
Owns the context layer of one system end to end. Designs and defends their own experiments. The volume hire
Senior / Staff
70-84
62-75
Owns evaluation methodology across an org. Sets standards others follow. Catches methodological errors in others' work
Principal
85+
75+
Advances what the field can measure. External uptake of their methods. Rare, and largely self-identifying
Why the Continuity track has its own scale. The band meanings are unchanged; only the thresholds move. The duration modifier discounts 30% of the weight a second time, on top of the evidence multiplier that already applies to all ten domains, so identical quality of work scores materially lower there. Under the base thresholds a candidate rigorous across every domain, with multiple cycles of continuous operation and measurement across a substrate discontinuity, scores about 65.7 and gets labelled the volume hire. That is the instrument mis-reporting a young field as a weak candidate. These thresholds restore the intended meaning of each band; they do not lower the bar.
10 The n=1 problem
This is where most otherwise-strong candidates are weakest, and it is the substantive addition to D3 on the Continuity track.
An individuated entity is, by construction, a sample of one. You cannot randomise a nine-month-old agent-principal relationship into treatment and control. The base experimental toolkit of held-out sets, multi-seed ablation and A/B degrades badly here, and candidates who have only that toolkit tend to fail in one of two ways: they run the wrong experiment on a population of freshly-instantiated entities and generalise to the persistent case, or they abandon measurement and fall back on narrative.
The right toolkit exists, but it comes from clinical and behavioural research rather than machine learning.
Single-case experimental design, such as ABAB reversal, where an intervention is introduced, withdrawn and reintroduced against the entity's own baseline
Multiple-baseline across behaviours, staggering an intervention across several domains of the same entity so each serves as a control for the others
Interrupted time series, the natural design for substrate discontinuities: a model rollover is the interruption, and the question is whether the level or slope changed
Synthetic control, constructing a counterfactual entity from a weighted composite of others when a true control is impossible
Within-subject calibration curves, the entity's own stated confidence against its own measured accuracy, needing no between-subject comparison at all
A candidate who reaches for any of these unprompted is scoring 3 or above on D3 for this track. A candidate who has never encountered the n=1 problem is scoring 2 at most, regardless of how strong their stateless experimental work is, because the method they have does not answer the question this track asks.