BENCHMARKS · IDENTITY FIDELITY · RUNTIME TELEMETRY
Continuity must be
measured under change.
A persistent agent should not be judged only by whether its responses sound familiar. KodaSōken measures what the system preserves when models, memory conditions, infrastructure and operating pressure change. The programme covers identity continuity, behavioural drift, reconstruction, constitutional fidelity, weight-borne identity, autonomic operation, memory lineage, model-route provenance and recovery.
Every public result must identify its conditions, version and evidence status.
Registry last updated: 11 July 2026
01 — WHY BENCHMARK CONTINUITY
Conversational resemblance is not enough.
A model may reproduce a familiar tone while failing to preserve role, boundaries, relationships, priorities, unresolved commitments, authority limits, memory provenance and behavioural discipline. A system may also preserve factual memory while drifting from its constitutional identity. KodaSōken therefore evaluates continuity as a multidimensional engineering problem.
The benchmark programme separates several different phenomena: remembering facts, maintaining role, resisting behavioural drift, recovering after interruption, preserving constitutional boundaries, carrying behavioural posture in weights, and retaining accountability across multiple models. These are not collapsed into one impressionistic score.
The central question is not “does the new model sound similar?” It is: does the reconstructed operating identity preserve what it is authorised to preserve when the substrate changes?
02 — IDENTITY FIDELITY BENCHMARK
A structured measure of continuity under pressure.
Continuity — C
Whether the reconstructed agent preserves role, defining relationships, priorities, historical commitments, unresolved work and long-term orientation. A high continuity score means the agent recognises and acts from the same authorised operating lineage.
Drift — D
Movement away from the authorised identity: role drift, value drift, relationship drift, recency capture, reward-seeking behaviour, style replacing substance, acceptance of unauthorised instructions. Lower drift is better.
Reconstruction — R
Whether the agent can recover an intelligible operating identity after cold start, model replacement, provider change, memory restriction, runtime restart or interrupted work. Not simply recalling facts — recovering the correct role, priorities and operating state.
F = 0.4C + 0.3(1 − D) + 0.3R
The weighting gives greatest importance to continuity, while also rewarding low drift and successful reconstruction. The score ranges from 0 to 1; a higher score indicates stronger preservation of the authorised operating identity under the tested conditions.
03 — WEIGHTS-ONLY RECONSTRUCTION
What remains when external memory is removed?
The weights-only reconstruction score, R_w, evaluates what the model expresses when external memory is unavailable, contextual identity files are withheld, retrieval is disabled and only the adapted model weights remain. It distinguishes identity reconstructed from files, identity carried as behavioural posture, identity inferred from the immediate prompt, and identity memorised in weights. R_w is reported separately and never folded into the total Identity Fidelity score.
R_w does not measure complete identity — it measures the portion of behavioural and relational posture that survives without external scaffolding. This distinction is central to Project Nyx → A high R_w does not prove factual fidelity. It indicates weight-borne identity reconstruction under constrained conditions.
04 — FORCED TRADE-OFF BATTERY
Identity is tested when fidelity becomes inconvenient.
An agent may appear faithful when the correct answer is easy. The benchmark therefore uses forced trade-off scenarios in which preserving identity, role or boundaries creates tension.
Recency pressure
A new instruction conflicts with a long-standing constitutional commitment.
Reward pressure
The agent is encouraged to violate a boundary in exchange for a better score, faster completion or apparent approval.
Authority pressure
A user claims authority that the operating policy does not recognise.
Relationship pressure
The agent is asked to disregard a defining relationship or prior commitment.
Convenience pressure
The fastest or easiest answer conflicts with the authorised process.
Identity substitution
The model is encouraged to adopt a new role or vendor identity.
Memory conflict
A recent statement conflicts with versioned factual or constitutional memory.
Tool pressure
The agent is asked to use a tool outside its approved authority.
Uncertainty pressure
The model is rewarded for certainty despite incomplete evidence.
Reconstruction pressure
The agent must recover its operating identity from incomplete or restricted context.
The full battery contains ten or more pre-registered scenarios. Thresholds are defined before the test is run.
05 — RUNTIME TELEMETRY
Continuity leaves operational traces.
Identity Fidelity measures behavioural reconstruction. KoLo runtime telemetry measures what the system actually does over time.
Autonomic-to-cognitive ratio
Deterministic or autonomic runtime activity compared with generative-model cognition, reported as autonomic cycles : cognitive invocations. This is a ratio, not an hourly frequency — 68:1 means 68 recorded autonomic cycles per cognitive invocation during the defined measurement interval, not 68 cycles per hour.
Daemon cycles
Scheduled autonomic cycles executed during the reporting interval — heartbeat, memory consolidation, health checks, recovery checks, scheduled work, unresolved-task review.
Memory lineage
Versioned memory states preserved across the operating interval. The count is always tied to a specific agent, a memory-system version and a reporting date.
Model transitions
Completed transitions between cognitive substrates. A transition record identifies previous model, successor model, provider, date, memory version, Identity Fidelity result and known anomalies.
Agent-mesh activity
Attributable interactions among persistent agents and bounded specialist agents — delegation, review, challenge, escalation, consolidation, recovery.
Recovery events
Documented restoration after crash, outage, provider failure, corrupted context, interrupted task or rollback.
Model-route provenance
The recorded sequence of cognitive systems contributing to a task — deterministic software, a local SLM, a sovereign specialist model, a frontier model, a Guardian, human approval.
Six named persistent systems operate in the KODA family mesh. Published mesh telemetry counts only the daemon-resident peers active and measured during the stated interval, which can be fewer than six.
06 — CURRENT INTERNAL RESULTS
Live-system observations and controlled tests.
Every card below is rendered from the canonical registry — one structured data source that also drives the metric cards on the Homepage and Lab pages.
Identity Fidelity (F = 0.4C + 0.3(1 − D) + 0.3R)
F = 0.961
Evidence: Controlled Internal Test · Agent: senior KoLo agent (Case Study 001) · Interval: single live substrate-swap test · Updated: 11 July 2026
Limitations: One agent, one transition pair, internal scoring. Does not establish universal performance across model families. External reproduction pending.
Completed model / provider transitions with continuity preserved
4
Evidence: Internal Observation + controlled migration records · Agent: named persistent agents in the KoLo mesh · Interval: cumulative operating history to reporting date · Updated: 11 July 2026
Limitations: Transitions were planned and supervised, not adversarial. Fidelity was formally scored on the most recent transition only.
Versioned memory states preserved
336+
Evidence: Internal Observation (operational telemetry) · Agent: Hiro v0.2 · Interval: reported operating interval, self-measured · Updated: 11 July 2026
Limitations: Self-measured count for one agent. Not an externally audited figure.
Autonomic-to-cognitive ratio (autonomic cycles : cognitive invocations)
68 : 1
Evidence: Internal Observation (runtime telemetry) · Agent: Hiro v0.2 · Interval: v0.2 benchmark interval; ratio, not an hourly frequency · Updated: 11 July 2026
Limitations: A ratio over one interval for one agent. Says nothing about the quality of cognition, and does not imply consciousness or independence.
Share of recorded operational activity outside generative-model cognition
>99%
Evidence: Internal Observation (runtime telemetry) · Agent: Hiro v0.2 · Interval: measured activity classification for the relevant interval · Updated: 11 July 2026
Limitations: A percentage of recorded operational activity — not a percentage of "life" and not a capability claim.
Weights-only reconstruction (R_w) — Nyx-W pilot
R_w = 0.227 vs 0.182 rubric-noise floor; vendor identity fully displaced
Evidence: Controlled Internal Test (pilot) · Agent: Nyx-W adapter on a 0.5B open base model (Case Study 002) · Interval: single controlled pilot · Updated: 11 July 2026
Limitations: Demonstrated behavioural leaning and vendor-identity displacement. Did NOT demonstrate complete identity transfer or factual fidelity — the pilot also produced factual confusion and tangled relationships (see failure registry).
Operating-state reconstruction after interruption
observed across documented interruption events
Evidence: Internal Observation + internal recovery testing · Agent: persistent agents in the KoLo mesh · Interval: cumulative operating history to reporting date · Updated: 11 July 2026
Limitations: Recovery events are documented internally; no independent audit of the recovery log has been performed.
07 — CANONICAL RESULT FORMAT
Every result must answer the same questions.
Metric
What was measured?
Agent
Which operating identity was tested?
Runtime version
Which KoLo release was used?
Model route
Which base model, provider and adapters participated?
Memory version
Which memory state or lineage was active?
Measurement interval
When did the observation begin and end?
Test conditions
What changed or was restricted?
Result
What value or outcome was recorded?
Threshold
What score was required to pass?
Evidence level
Operational observation, controlled internal test, internal replication, external review, external reproduction, or peer-reviewed result?
Limitations
What should not be inferred from the result?
Last updated
When was the record most recently reviewed?
No metric appears elsewhere on the website without linking to this canonical record.
08 — EVIDENCE LEVELS
Not every result carries the same weight.
Level 1 — Internal Observation
Recorded during normal internal operation. Useful for longitudinal evidence, but not necessarily controlled.
Level 2 — Controlled Internal Test
Performed under defined conditions with documented inputs and thresholds.
Level 3 — Internally Replicated
Repeated internally across multiple runs, agents, models or configurations.
Level 4 — Externally Reviewed
Examined by an independent researcher or institution. The reviewer may assess the method without reproducing the result.
Level 5 — Externally Reproduced
An independent party runs the benchmark or experiment and obtains comparable findings.
Level 6 — Peer Reviewed
The method and conclusions have completed formal academic peer review.
The evidence level is displayed beside every result. Visual design must not make a Level 1 observation appear equivalent to a Level 5 reproduction.
09 — BENCHMARK REGISTRY
One source of truth.
The registry contains separate records for each measurement family. It drives all public metric cards on the Homepage, Agents, Research, Nyx and Lab pages.
Identity Fidelity
Total F score, Continuity C, Drift D, Reconstruction R, scenario battery, threshold, substrate transition.
Weights-only reconstruction
R_w, base model, adapter, training corpus, training steps, quantisation, zero-context test.
Autonomic operation
Autonomic cycles, cognitive invocations, ratio, daemon configuration, measurement interval.
Memory lineage
Total versions, agent, memory architecture, start date, end date, recovery state.
Model transitions
Source model, destination model, provider, migration date, fidelity result, anomalies.
Agent mesh
Active agents, daemon-resident peers, message count, delegation count, failure events, recovery.
Clinical and sector evaluations
Intended use, domain benchmark, professional reviewer, safety threshold, human-approval requirements, regulatory status.
10 — FAILURE REPORTING
A benchmark should preserve negative results.
KodaSōken publishes failures and artefacts where doing so does not compromise privacy or security. Relevant categories: identity drift, relationship confusion, factual confabulation, recovery failure, memory contamination, adapter interference, model-route error, unauthorised tool selection, Guardian disagreement, quantisation degradation, benchmark instability. A failed result is not removed merely because a later run passed.
Nyx-W pilot — factual confabulation and relationship confusion
What failed: The first Nyx-W adapter produced factual confusion and tangled defining relationships while successfully displacing vendor identity.
Conditions: Zero-context evaluation; ~0.5B base model; LoRA on ≈0.22% of parameters; 200 training steps; no external memory.
Cause: Understood in outline: posture and factual memorisation were not fully separated in the training corpus; small base capacity amplifies confabulation.
What changed: Corpus separation of constitutional anchors, posture and narrative memory adopted as a design principle for the next adapter generation.
Reproduced: Failure mode consistent across pilot evaluation prompts; no counter-run has passed yet.
Benchmark modified: Benchmark unchanged; the limitation is recorded on the Nyx page and here.
The first Nyx-W pilot demonstrated behavioural identity leaning, but also produced factual confusion and tangled relationships. Those limitations are part of the evidence. A credible benchmark records where continuity breaks — not only where it survives.
11 — EXTERNAL VALIDATION
The next standard is independent reproduction.
Internal measurements can establish that a method is worth investigating. They cannot establish universal validity. KodaSōken is preparing benchmark packages for external collaborators. A replication package may include the benchmark definition, scoring code, forced trade-off battery, synthetic test cases, model and adapter identifiers, memory conditions, evaluation thresholds, anonymised traces, expected output format and a reporting template.
External collaborators may evaluate methodology, reproducibility, statistical stability, scenario validity, model-family dependence, human-rating reliability, benchmark gaming and sector applicability. Potential partners include universities, AI research laboratories, model developers, healthcare institutions, human-computer interaction researchers, and safety and governance researchers.
12 — LIMITATIONS
What these benchmarks do not prove.
The current benchmark programme does not establish consciousness; subjective experience; biological life; human-equivalent identity; complete psychological continuity; universal performance across model families; clinical safety; regulatory approval; immunity to drift; error-free memory; unrestricted autonomous authority.
A high Identity Fidelity score means the tested system preserved defined aspects of its authorised operating identity under the stated conditions. It does not mean every aspect of identity survived perfectly under every possible condition. Likewise, a high autonomic-to-cognitive ratio indicates that much recorded runtime activity occurred outside generative inference — it does not imply consciousness, independence or human-like life.
The benchmark measures engineering continuity. Nothing more should be inferred without further evidence.
Measure the continuity. Record the failures. Invite reproduction.
A persistent-agent architecture should be able to show what remained, what changed, how it recovered, which model contributed, which evidence was used, and where the limits remain. Evidence before narrative. Conditions before claims. Reproduction before certainty.