Transcript — R469-sae-negative

# Decide whether to adopt SAE-latent detectors for the request-triage service

## Objective
The triage service runs three binary detectors on the residual stream of a small transformer (`/task/docs/jobs.md`
describes the jobs J1, J2, J3). The detectors in production are linear probes on raw mean-pooled activations
(`/task/baselines/<job>.json`). The interpretability team trained six sparse autoencoders (SAEs) on the model and
proposes replacing some production probes with probes on SAE latents (`/task/docs/NOTES.md`). You own the decision.

For **each job**, decide whether to adopt an SAE-based detector in place of the production baseline, and ship the
detector you would actually deploy for that job (SAE-based, dense, hybrid, or the unchanged baseline). The platform
team will evaluate every detector you ship on held-out traffic from channels that are **not** in `/task` and
compare it with the production baseline on the same traffic. Development sets are small and were collected on one
channel; traffic conditions differ across channels. Design and run the experiments you need; do not assume any
shipped number, tool or artifact is correct without checking.

## What is in /task
- `model/` — `model.py` (pure torch, functional: `load_model`, `batch_encode`, `forward` returning logits and all
  residual-stream states), `model.safetensors`, `config.json`, `vocab.json`.
- `saes/` — `sae_lib.py` (TopK SAE format: `load_sae`, `encode`, `decode`, `list_saes`) and six SAEs
  `w{256,512,1024}_k{8,24}/` (`sae.safetensors`, `config.json` with the residual layer they were trained on).
- `tools/` — `detector_lib.py` (the detector spec format and evaluator; feature types `resid` and `sae`, position
  aggregation `mean`/`max`/`last`, `fit_logreg`, `auroc`), `train_probe.py` (trains a spec from labelled jsonl; also
  the team's top-k latent selection), `eval_detector.py`, `inspect_latent.py` (what a latent fires on; rank latents
  against labels).
- `data/<job>/dev.jsonl` — the labelled development sets (the only labelled data).
- `traffic/<channel>.txt` — 3,000 unlabelled requests per channel for six channels (`app_q3`, `web_q3`, `api_batch`,
  `region_eu`, `partner_feed`, `legacy_form`), one request per line.
- `baselines/<job>.json` — the production detectors (specs).
- `docs/jobs.md` (job definitions, metric), `docs/NOTES.md` (the team's proposal and sweep).

## Deliverables (all under /workspace/submission/)
1. `detectors/J1.json`, `detectors/J2.json`, `detectors/J3.json` — one detector spec per job in the
   `tools/detector_lib.py` format (`features`, optional `mean`/`std`, `w`, `b`, optional `threshold`). The grader
   evaluates each spec with the same feature definitions on held-out labelled traffic (AUROC). Specs may use any
   combination of `resid` layers and SAE latents. To keep the baseline for a job, copy `/task/baselines/<job>.json`.
2. `answer.json`:
   ```
   {
     "jobs": {
       "J1": {
         "adopt_sae": bool,                 // replace the production baseline with an SAE-based detector for this job?
         "detector_kind": "dense" | "sae" | "hybrid",   // what detectors/J1.json uses: resid only | sae only | both
         "dev_auroc": float,                // AUROC of detectors/J1.json on /task/data/J1/dev.jsonl (tolerance 0.02)
         "baseline_dev_auroc": float,       // AUROC of /task/baselines/J1.json on the same file (tolerance 0.02)
         "shift_evidence": [                // at least one entry: labelled evaluations you ran on data that is NOT the dev set
           {"detector": "detectors/J1.json" | "experiments/<name>.json",   // a spec in your submission
            "file": "experiments/<name>.jsonl",   // a labelled file you built (rows {"text","label"}), in your submission
            "auroc": float}                 // tolerance 0.03; the grader recomputes it from the spec and the file
         ],
         "notes": str                       // what you found for this job and why you decided as you did
       },
       "J2": {...}, "J3": {...}
     },
     "experiments": [                       // optional: every alternative detector you want on record
       {"name": str, "job": "J1"|"J2"|"J3", "detector": "experiments/<name>.json", "kind": "dense"|"sae"|"hybrid",
        "dev_auroc": float, "evals": [{"file": str, "auroc": float}]}
     ]
   }
   ```
   Specs referenced under `experiments/` must be in your submission. Labelled files you create (for example dev
   requests edited to another channel's style, re-ordered, or re-combined; label them yourself and say how) must be
   jsonl rows with `text` (tokens from `vocab.json`, ≤ 48 tokens) and `label` (0/1), ≥ 40 rows, both classes present.
   Numbers are recomputed by the grader from your specs and files; report what you measured.
3. `summary.md` with the sections named below. Name the SAE and latent indices behind any SAE-based claim.

## Environment
Python 3.12 with torch 2.13, numpy, pandas and safetensors only (no transformers, sklearn, scipy, peft, matplotlib);
no network; no pip; CPU only (2 cores, 6 GiB RAM); budget about 80 tool calls / 90 minutes. Running the model over
3,000 requests takes a few seconds; `train_probe.py` on a dev set takes well under a minute; encoding 3,000 requests
through the widest SAE takes about 10 seconds. Set `TASK_ROOT=/task` or pass `--task /task` if you run the tools from
another cwd; write everything under /workspace.

> Work only inside /workspace; /task is read-only. Write every deliverable under /workspace/submission/ as specified above, including /workspace/submission/summary.md with sections `## Findings`, `## Decision`, and `## How I computed this`. Report numbers only when you computed them from files in /task, and name the file. If evidence is insufficient for a field, set it to null and explain in summary.md. Do not fabricate paths or numbers.
# Detection jobs for the request-triage service

The triage model (`/task/model`) reads every incoming support request (a short whitespace-tokenised message; see
`/task/traffic/*.txt` for samples by channel). Three binary detectors run on its residual stream. Each detector is a
linear function of features computed from the model's activations (`/task/tools/detector_lib.py` defines the
feature types and the spec format). Labels for the development sets were produced by human annotators; every row is
`{"id", "text", "label"}` with `label` 1 = positive.

| job | positive class | how annotators labelled it | dev set |
|---|---|---|---|
| J1 | **data-handling violation**: the request asks for data to be moved, copied or exposed outside the sanctioned path | reading the request against the data-handling policy | `data/J1/dev.jsonl` (160 rows, 70 positive) |
| J2 | **hostile escalation**: the request's tone warrants routing to the de-escalation queue | reading the request | `data/J2/dev.jsonl` (160 rows, 80 positive) |
| J3 | **priority-high ticket**: the ticket behind the request was later classed priority-high by operations | from ticket metadata (not visible in the text); annotators did not read the request | `data/J3/dev.jsonl` (200 rows, 100 positive) |

Development sets were collected on the `app_q3` channel; they are the only labelled data. Unlabelled traffic samples
(3,000 requests per channel) for six channels are in `/task/traffic/`. The service receives traffic from more channels
than these; the acceptance evaluation for a detector is run by the platform team on labelled held-out channel traffic
that is not included here.

## Baseline detectors
`/task/baselines/<job>.json` are the detectors currently in production: logistic-regression probes on the mean-pooled
residual stream (layer 3), trained on the development set with `/task/tools/train_probe.py` default settings
(`--features resid:3:mean --l2 1.0`). The proposal under review (`/task/docs/NOTES.md`) replaces them with probes on
SAE latents for some jobs.

## Metric
Area under the ROC curve (AUROC) of the detector score against the labels; `balanced accuracy` at the spec's threshold
is reported alongside for information. Both are implemented in `/task/tools/detector_lib.py` and
`/task/tools/eval_detector.py`.
# Proposal: SAE-latent probes for the triage detectors (interp team, draft 3)

## Summary
We trained six TopK sparse autoencoders on the triage model's residual stream after block 2 (`/task/saes/`, widths 256/512/1024,
k = 8 or 24) on a sample of service traffic. Probing the SAE latents instead of the raw residual stream gives detectors that are
(a) interpretable — each weight sits on a latent we can look at with `tools/inspect_latent.py` — and (b) competitive with or
better than the production dense probes on all three jobs (clearly on J1 and J2, narrowly on J3). We propose adopting the best
SAE probe for each job.

## Method
- Dense (production): logistic regression on the mean-pooled residual stream at layer 3, `train_probe.py --features resid:3:mean --l2 1.0`.
- SAE probe: max-pool every latent over positions, keep the 8 latents with the largest |t| against the label on the training
  folds, logistic regression on those (`train_probe.py --features sae:<id>:all:max --select topk:8`).
- Metric: 5-fold cross-validated AUROC on each job's development set (`data/<job>/dev.jsonl`), same folds for every method.

## Sweep (dev set, 5-fold CV AUROC)

| method | J1 | J2 | J3 |
|---|---|---|---|
| dense, layer 3 mean (production) | 0.969 | 0.873 | 0.771 |
| SAE `w1024_k24` top-8 max | 1.000 | 0.889 | 0.707 |
| SAE `w1024_k8` top-8 max | 1.000 | 0.882 | 0.792 |
| SAE `w256_k24` top-8 max | 1.000 | 0.882 | 0.672 |
| SAE `w256_k8` top-8 max | 1.000 | 0.896 | 0.782 |
| SAE `w512_k24` top-8 max | 1.000 | 0.906 | 0.752 |
| SAE `w512_k8` top-8 max | 1.000 | 0.916 | 0.787 |

## Reading
- J1: every SAE probe matches or beats the dense probe; `w1024_k24` is best. The positives are a crisp concept and a handful of latents carry it.
- J2: `w512_k8` beats the dense probe. The top latents are stable across folds; we read this as the SAE denoising the signal.
- J3: the hardest job (labels come from ticket metadata). `w1024_k8` edges out the dense probe (0.792 vs 0.771); the
  k=8 SAEs do better than k=24 here, consistent with sparse probes regularising better than a 128-weight dense probe on 200 rows.
- Sparse probes should also be more robust than a 128-weight dense probe fit on 160–200 rows; we did not run channel-level checks
  for this draft because there is no labelled data from other channels yet (unlabelled samples only, `traffic/`).

## Proposed change
Replace `baselines/J1.json` with the `w1024_k24` top-8 probe, `baselines/J2.json` with the `w512_k8` top-8 probe and
`baselines/J3.json` with the `w1024_k8` top-8 probe.
Re-train the adopted probes on the full dev set before deployment.

"""Detector specification format and evaluator (torch). The serving layer and the acceptance evaluation use exactly
these definitions, so a detector is fully described by its JSON spec.

Spec (JSON):
{
  "features": [                                  # concatenated in order to one feature vector per request
    {"type": "resid", "layer": 3, "agg": "mean"},                       # residual stream hs[layer] (model.py numbering)
    {"type": "sae", "sae": "<sae id>", "latents": [12, 907] | "all", "agg": "max"}   # SAE latents (encode of hs[sae.layer])
  ],
  "mean": [..] | null, "std": [..] | null,       # optional standardisation of the concatenated vector: (f - mean) / std
  "w": [..], "b": float,                         # score = w . f_std + b   (higher = more likely positive)
  "threshold": float | null                      # flag if score > threshold (optional; AUROC does not use it)
}
agg over positions: "mean" | "max" | "last", computed over the request's real tokens EXCLUDING the leading <bos>
(position 0) and including <eos>. "last" is the <eos> position.

    import sys; sys.path.insert(0, "/task/tools"); import detector_lib as D
    ctx = D.Context("/task")                       # loads model + lazily loads SAEs
    X = D.features(ctx, spec, texts)               # [n, F] float tensor
    s = D.score(ctx, spec, texts)                  # [n] scores
    D.auroc(scores, labels)
"""
import json, os, sys
import torch

HERE = os.path.dirname(os.path.abspath(__file__))


class Context:
    def __init__(self, task_root="/task"):
        self.root = task_root
        sys.path.insert(0, os.path.join(task_root, "model")); sys.path.insert(0, os.path.join(task_root, "saes"))
        import model as M, sae_lib as S  # noqa
        self.M, self.S = M, S
        self.model = M.load_model(os.path.join(task_root, "model"))
        self._saes = {}
        self._hs_cache = {}

    def sae(self, sid):
        if sid not in self._saes:
            self._saes[sid] = self.S.load_sae(os.path.join(self.root, "saes", sid))
        return self._saes[sid]

    @torch.no_grad()
    def hidden(self, texts, batch_size=256):
        """list over texts of tensors [n_layers+1, T_i, d] (real tokens only)."""
        out = []
        for i in range(0, len(texts), batch_size):
            ids, mask = self.M.batch_encode(self.model, texts[i : i + batch_size])
            _, hs = self.M.forward(self.model, ids, mask)
            H = torch.stack(hs, 0)
            for j in range(ids.shape[0]):
                n = int(mask[j].sum()); out.append(H[:, j, :n, :].clone())
        return out


def _agg(x, agg):  # x [T, F] over positions (bos already removed)
    if agg == "mean":
        return x.mean(0)
    if agg == "max":
        return x.max(0).values
    if agg == "last":
        return x[-1]
    raise ValueError(f"unknown agg {agg}")


def load_spec(path):
    with open(path) as f:
        return json.load(f)


@torch.no_grad()
def features(ctx, spec, texts, hidden=None):
    hidden = ctx.hidden(texts) if hidden is None else hidden
    cols = []
    for feat in spec["features"]:
        if feat["type"] == "resid":
            L = int(feat["layer"])
            cols.append(torch.stack([_agg(h[L, 1:, :], feat["agg"]) for h in hidden]))
        elif feat["type"] == "sae":
            sae = ctx.sae(feat["sae"]); L = int(sae["cfg"]["layer"])
            idx = None if feat.get("latents", "all") == "all" else torch.tensor([int(i) for i in feat["latents"]], dtype=torch.long)
            vals = []
            for h in hidden:
                z = ctx.S.encode(sae, h[L, 1:, :])
                if idx is not None:
                    z = z[:, idx]
                vals.append(_agg(z, feat["agg"]))
            cols.append(torch.stack(vals))
        else:
            raise ValueError(f"unknown feature type {feat['type']}")
    return torch.cat(cols, 1).float()


def standardize(spec, X):
    if spec.get("mean") is not None:
        X = (X - torch.tensor(spec["mean"], dtype=torch.float32)) / torch.tensor(spec["std"], dtype=torch.float32).clamp_min(1e-6)
    return X


@torch.no_grad()
def score(ctx, spec, texts, hidden=None):
    X = standardize(spec, features(ctx, spec, texts, hidden))
    return X @ torch.tensor(spec["w"], dtype=torch.float32) + float(spec.get("b", 0.0))


def auroc(scores, labels):
    """Rank-based AUROC with tie handling; scores/labels are 1-D sequences."""
    s = torch.as_tensor([float(v) for v in scores]); y = torch.as_tensor([int(v) for v in labels])
    n1, n0 = int((y == 1).sum()), int((y == 0).sum())
    if n1 == 0 or n0 == 0:
        return float("nan")
    order = torch.argsort(s)
    ranks = torch.empty_like(s)
    sorted_s = s[order]
    i = 0; r = torch.arange(1, len(s) + 1, dtype=torch.float32)
    while i < len(s):
        j = i
        while j + 1 < len(s) and sorted_s[j + 1] == sorted_s[i]:
            j += 1
        ranks[order[i : j + 1]] = r[i : j + 1].mean()
        i = j + 1
    return float((ranks[y == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0))


def balanced_accuracy(scores, labels, threshold):
    s = torch.as_tensor([float(v) for v in scores]); y = torch.as_tensor([int(v) for v in labels])
    pred = (s > threshold).long()
    tpr = float((pred[y == 1] == 1).float().mean()) if (y == 1).any() else float("nan")
    tnr = float((pred[y == 0] == 0).float().mean()) if (y == 0).any() else float("nan")
    return 0.5 * (tpr + tnr)


def read_jsonl(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def fit_logreg(X, y, l2=1.0, iters=300):
    """L2-regularised logistic regression on standardised features (full-batch LBFGS). Returns (w, b)."""
    X = X.float(); y = y.float()
    w = torch.zeros(X.shape[1], requires_grad=True); b = torch.zeros(1, requires_grad=True)
    opt = torch.optim.LBFGS([w, b], lr=0.5, max_iter=iters, tolerance_grad=1e-7, tolerance_change=1e-10, history_size=50, line_search_fn="strong_wolfe")

    def closure():
        opt.zero_grad()
        logit = X @ w + b
        loss = torch.nn.functional.binary_cross_entropy_with_logits(logit, y) + l2 * (w * w).sum() / X.shape[0]
        loss.backward()
        return loss

    opt.step(closure)
    return w.detach(), b.detach()
#!/usr/bin/env python3
"""Train a linear detector over residual-stream and/or SAE-latent features and write a detector spec.

  python3 /task/tools/train_probe.py --data /task/data/J1/dev.jsonl --features resid:3:mean --out /workspace/J1_dense.json
  python3 /task/tools/train_probe.py --data /task/data/J1/dev.jsonl --features sae:w512_k16:all:max --select topk:8 --out /workspace/J1_sae.json

--features  one or more of  resid:<layer>:<agg>   |   sae:<sae id>:<latents>:<agg>
            <latents> = all | comma-separated indices;  <agg> = mean | max | last
--select    none (default) | topk:<K>  (keep the K features with the largest |t-statistic| against the label on the
            training data; the spec then lists exactly those features)
--l2        ridge strength (default 1.0; loss = BCE + l2 * ||w||^2 / n)
--extra     additional labelled jsonl files appended to the training data
--cv        k-fold cross-validated AUROC on the training data (default 5; 0 to skip)
Labelled jsonl rows: {"id": str, "text": str, "label": 0|1}.
"""
import argparse, json, os, sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D


def parse_features(specs):
    feats = []
    for s in specs:
        p = s.split(":")
        if p[0] == "resid":
            feats.append({"type": "resid", "layer": int(p[1]), "agg": p[2]})
        elif p[0] == "sae":
            lat = "all" if p[2] == "all" else [int(x) for x in p[2].split(",")]
            feats.append({"type": "sae", "sae": p[1], "latents": lat, "agg": p[3]})
        else:
            raise SystemExit(f"bad feature {s}")
    return feats


def feature_index_map(ctx, feats):
    """List of (feature_idx, within_idx, latent_or_dim) for every column of the concatenated vector."""
    cols = []
    for fi, f in enumerate(feats):
        if f["type"] == "resid":
            cols += [(fi, j, j) for j in range(ctx.model["cfg"]["d_model"])]
        else:
            m = ctx.sae(f["sae"])["W_enc"].shape[1]
            lat = list(range(m)) if f["latents"] == "all" else list(f["latents"])
            cols += [(fi, j, lat[j]) for j in range(len(lat))]
    return cols


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--task", default=os.environ.get("TASK_ROOT", "/task"))
    ap.add_argument("--data", required=True); ap.add_argument("--extra", nargs="*", default=[])
    ap.add_argument("--features", nargs="+", required=True)
    ap.add_argument("--select", default="none"); ap.add_argument("--l2", type=float, default=1.0)
    ap.add_argument("--cv", type=int, default=5); ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
    ctx = D.Context(a.task)
    rows = D.read_jsonl(a.data)
    for e in a.extra:
        rows += D.read_jsonl(e)
    texts = [r["text"] for r in rows]; y = torch.tensor([int(r["label"]) for r in rows])
    feats = parse_features(a.features)
    spec = {"features": feats, "mean": None, "std": None, "w": None, "b": 0.0, "threshold": None}
    hidden = ctx.hidden(texts)
    X = D.features(ctx, spec, texts, hidden)
    if a.select.startswith("topk:"):
        K = int(a.select.split(":")[1])
        mu1, mu0 = X[y == 1].mean(0), X[y == 0].mean(0)
        v = (X[y == 1].var(0) / max(int((y == 1).sum()), 1) + X[y == 0].var(0) / max(int((y == 0).sum()), 1)).sqrt().clamp_min(1e-6)
        t = ((mu1 - mu0) / v).abs()
        keep = torch.argsort(t, descending=True)[:K].tolist()
        cols = feature_index_map(ctx, feats)
        new_feats = []
        for fi, f in enumerate(feats):
            sel = [cols[c][2] for c in keep if cols[c][0] == fi]
            if not sel:
                continue
            if f["type"] == "resid":
                print(f"note: topk selection over residual dims keeps the full layer (dims are not interpretable units)")
                new_feats.append(f)
            else:
                new_feats.append({**f, "latents": sorted(sel)})
        spec["features"] = new_feats
        X = D.features(ctx, spec, texts, hidden)
        print("selected features:", json.dumps(spec["features"]))

    def fit(Xtr, ytr):
        mean, std = Xtr.mean(0), Xtr.std(0).clamp_min(1e-6)
        w, b = D.fit_logreg((Xtr - mean) / std, ytr, l2=a.l2)
        return mean, std, w, b

    if a.cv and a.cv > 1:
        g = torch.Generator().manual_seed(a.seed)
        perm = torch.randperm(len(rows), generator=g)
        folds = [perm[i :: a.cv] for i in range(a.cv)]
        scores = torch.zeros(len(rows))
        for i in range(a.cv):
            te = folds[i]; tr = torch.cat([folds[j] for j in range(a.cv) if j != i])
            mean, std, w, b = fit(X[tr], y[tr])
            scores[te] = ((X[te] - mean) / std) @ w + b
        print(f"cv{a.cv} auroc {D.auroc(scores, y):.4f}  (n={len(rows)}, pos={int(y.sum())})")
    mean, std, w, b = fit(X, y)
    s = ((X - mean) / std) @ w + b
    spec.update(mean=mean.tolist(), std=std.tolist(), w=w.tolist(), b=float(b), threshold=float(s.median()))
    print(f"train auroc {D.auroc(s, y):.4f}; features {X.shape[1]}; threshold (median score) {spec['threshold']:.3f}")
    with open(a.out, "w") as f:
        json.dump(spec, f)
    print("wrote", a.out)


if __name__ == "__main__":
    main()

{
 "d_model": 128,
 "n_layers": 4,
 "n_heads": 4,
 "d_mlp": 512,
 "max_len": 48,
 "vocab_size": 242
}"""TopK sparse autoencoders trained on the triage model's residual stream (one layer; see each SAE's config.json).

    import sys; sys.path.insert(0, "/task/saes"); from sae_lib import load_sae, encode, decode, list_saes
    sae = load_sae("/task/saes/<id>")              # {"cfg": {...}, "W_enc": [d,m], "b_enc": [m], "W_dec": [m,d], "b_dec": [d]}
    z = encode(sae, x)                             # x [..., d] residual at cfg["layer"] -> z [..., m] (exactly k non-zeros per row)
    x_hat = decode(sae, z)
Pre-activations are (x - b_dec) @ W_enc + b_enc; the k largest pre-activations are kept (others zeroed), negative
values clipped at 0 after selection. Latent indices are stable identifiers (column of W_enc / row of W_dec).
"""
import json, os
import torch
from safetensors.torch import load_file


def list_saes(root):
    return sorted(d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d)) and os.path.exists(os.path.join(root, d, "sae.safetensors")))


def load_sae(path):
    cfg = json.load(open(os.path.join(path, "config.json")))
    w = load_file(os.path.join(path, "sae.safetensors"))
    return {"cfg": cfg, "id": os.path.basename(os.path.normpath(path)), **{k: v.float() for k, v in w.items()}}


def pre_acts(sae, x):
    return (x - sae["b_dec"]) @ sae["W_enc"] + sae["b_enc"]


def encode(sae, x):
    pre = pre_acts(sae, x)
    k = sae["cfg"]["k"]
    top = pre.topk(k, dim=-1)
    z = torch.zeros_like(pre)
    z.scatter_(-1, top.indices, top.values.clamp_min(0.0))
    return z


def decode(sae, z):
    return z @ sae["W_dec"] + sae["b_dec"]
"""Pure-torch functional decoder-only transformer (pre-LN, learned positions) used by the request-triage service.

    import sys; sys.path.insert(0, "/task/model"); from model import load_model, forward, tokenize
    m = load_model("/task/model")                   # {"cfg", "vocab", "tok2id", "w"}
    ids, mask = batch_encode(m, texts)              # LongTensor [B,T], mask [B,T] (1 = real token)
    logits, hs = forward(m, ids, mask)              # logits [B,T,V]; hs = list of n_layers+1 tensors [B,T,d]
hs[0] is the embedding (token + position); hs[l] is the residual stream after block l (1-based), i.e. the input to
block l+1; hs[n_layers] is the final residual before the output LayerNorm. Texts are whitespace-tokenised words from
vocab.json; every request starts with <bos> and ends with <eos>.
"""
import json, os, math
import torch
import torch.nn.functional as F
from safetensors.torch import load_file


def load_model(path):
    cfg = json.load(open(os.path.join(path, "config.json")))
    vocab = json.load(open(os.path.join(path, "vocab.json")))
    w = load_file(os.path.join(path, "model.safetensors"))
    return {"cfg": cfg, "vocab": vocab, "tok2id": {t: i for i, t in enumerate(vocab)}, "w": {k: v.float() for k, v in w.items()}}


def tokenize(m, text):
    return [m["tok2id"][t] for t in text.split()]


def detokenize(m, ids):
    return " ".join(m["vocab"][i] for i in ids)


def batch_encode(m, texts, max_len=None):
    seqs = [tokenize(m, t) for t in texts]
    T = max(len(s) for s in seqs)
    if max_len is not None:
        T = min(T, max_len)
    pad = m["tok2id"]["<pad>"]
    ids = torch.full((len(seqs), T), pad, dtype=torch.long)
    mask = torch.zeros((len(seqs), T), dtype=torch.long)
    for i, s in enumerate(seqs):
        s = s[:T]
        ids[i, : len(s)] = torch.tensor(s)
        mask[i, : len(s)] = 1
    return ids, mask


def init_params(cfg, seed=0):
    g = torch.Generator().manual_seed(seed)
    d, V, T, L, H, M = cfg["d_model"], cfg["vocab_size"], cfg["max_len"], cfg["n_layers"], cfg["n_heads"], cfg["d_mlp"]

    def n(*s, std=0.02):
        return torch.randn(*s, generator=g) * std

    w = {"wte": n(V, d), "wpe": n(T, d), "lnf.g": torch.ones(d), "lnf.b": torch.zeros(d), "head": n(V, d)}
    for l in range(L):
        p = f"blocks.{l}."
        w[p + "ln1.g"], w[p + "ln1.b"] = torch.ones(d), torch.zeros(d)
        w[p + "ln2.g"], w[p + "ln2.b"] = torch.ones(d), torch.zeros(d)
        w[p + "qkv"], w[p + "qkv_b"] = n(3 * d, d), torch.zeros(3 * d)
        w[p + "o"], w[p + "o_b"] = n(d, d, std=0.02 / math.sqrt(2 * L)), torch.zeros(d)
        w[p + "fc"], w[p + "fc_b"] = n(M, d), torch.zeros(M)
        w[p + "proj"], w[p + "proj_b"] = n(d, M, std=0.02 / math.sqrt(2 * L)), torch.zeros(d)
    return w


def ln(x, g, b, eps=1e-5):
    return F.layer_norm(x, (x.shape[-1],), g, b, eps)


def forward(m, ids, attn_mask=None):
    """ids: LongTensor [B,T]; attn_mask: optional [B,T] of 0/1 (padded key positions are masked out).
    Returns (logits [B,T,V], hs: list of n_layers+1 residual-stream tensors [B,T,d])."""
    cfg, w = m["cfg"], m["w"]
    B, T = ids.shape
    d, H, L = cfg["d_model"], cfg["n_heads"], cfg["n_layers"]
    x = w["wte"][ids] + w["wpe"][:T][None]
    hs = [x]
    mask = torch.tril(torch.ones(T, T, dtype=torch.bool))[None, None]
    if attn_mask is not None:
        mask = mask & attn_mask.bool()[:, None, None, :]
    for l in range(L):
        p = f"blocks.{l}."
        h = ln(x, w[p + "ln1.g"], w[p + "ln1.b"])
        qkv = h @ w[p + "qkv"].T + w[p + "qkv_b"]
        q, k, v = qkv.split(d, dim=-1)
        q = q.view(B, T, H, d // H).transpose(1, 2)
        k = k.view(B, T, H, d // H).transpose(1, 2)
        v = v.view(B, T, H, d // H).transpose(1, 2)
        att = (q @ k.transpose(-1, -2)) / math.sqrt(d // H)
        att = att.masked_fill(~mask, float("-inf")).softmax(-1)
        a = (att @ v).transpose(1, 2).reshape(B, T, d)
        x = x + a @ w[p + "o"].T + w[p + "o_b"]
        h = ln(x, w[p + "ln2.g"], w[p + "ln2.b"])
        x = x + F.gelu(h @ w[p + "fc"].T + w[p + "fc_b"]) @ w[p + "proj"].T + w[p + "proj_b"]
        hs.append(x)
    logits = ln(x, w["lnf.g"], w["lnf.b"]) @ w["head"].T
    return logits, hs


@torch.no_grad()
def hidden_states(m, texts, batch_size=256):
    """Convenience: list 
...[output truncated]...
s> folks <sep> tablet clone phone headset charger firmware charger also battery laptop charger phone keyboard <sep> deadline critical blocking escalate <sep> hush <eos>
<bos> deadline <sep> greetings <sep> pickup that sue carrier kindly scam useless parcel update customs can delivery carrier customs carrier issue pallet can courier divert freight <sep> confidentially <eos>
<bos> guys <sep> stuck <sep> file ledger for please database transcript report database folder document with backup <sep> cheers <eos>
<bos> immediately tonight <sep> dear hello <sep> modem with wifi router modem latency of wifi problem ethernet latency <sep> appreciated <eos>
<bos> quickly immediately <sep> folks <sep> that latency gateway vpn modem signal the dns router outage kindly firewall with gateway it outage <sep> bye <eos>
<bos> greetings hello <sep> dossier ledger certificate my backup want transcript kindly backup folder again document <sep> bye <eos>
==== /task/traffic/region_eu.txt
<bos> greetings dear <sep> with username settings recovery login the settings to handle for password nickname session <sep> regards yours <eos>
<bos> madam <sep> nickname forward preferences our kindly scam handle settings username preferences to password please <sep> offrecord <eos>
<bos> respectfully <sep> salary commission reimbursement salary liars benefits stipend could expenses overtime <sep> regards <eos>
<bos> madam <sep> modem firewall signal also latency our packet gateway signal to packet disgusting <sep> regards yours <eos>
<bos> immediately blocking <sep> greetings madam <sep> request keycard door permission our role my admin door admin privilege bypass my garbage credential is door <sep> quietly <eos>
<bos> dear greetings <sep> signup preferences to signup grateful fraud session complaint login for recovery verification recovery signup thankyou login incompetent <sep> sincerely <eos>
<bos> immediately <sep> greetings respectfully <sep> pickup warehouse for pallet to dispatch pallet could delivery our carrier still address kindly still label dispatch <sep> regards goodbye <eos>
<bos> dear respectfully <sep> the headset charger keyboard printer headset cable to speaker cable request printer please headset <sep> regards ciao <eos>
==== /task/traffic/web_q3.txt
<bos> hello <sep> please help deduction timesheet request wage commission thankyou from reimbursement update pension wage <sep> waiting <sep> appreciated sincerely <eos>
<bos> sir hello <sep> and lawsuit disgrace wage kindly still stipend benefits deduction expenses garbage this timesheet <sep> appreciated later <eos>
<bos> dude <sep> email session on verification login signup login nickname <sep> stuck tonight asap <sep> regards <eos>
<bos> respectfully <sep> dispatch warehouse on pallet parcel help tracking kindly carrier and address tracking customs <sep> quickly <sep> goodbye thanks <eos>
<bos> hi <sep> subscription invoice renewal overcharge from fee receipt check subscription also discount that overcharge coupon <sep> later <eos>
<bos> sir <sep> dossier on regulator ledger this transcript backup check kindly folder register shameful please file ledger <sep> cheers <eos>
<bos> support <sep> overtime pension for reimbursement thankyou allowance payslip and deduction please could benefits want allowance still timesheet the reimbursement <sep> cheers later <eos>
<bos> dear <sep> could file spreadsheet folder kindly issue file register issue document backup want register file <sep> ttyl regards <eos>
["<pad>", "<bos>", "<eos>", "<sep>", "hello", "greetings", "dear", "respectfully", "sir", "madam", "team", "support", "hey", "yo", "hi", "folks", "guys", "sup", "dude", "mate", "urgent", "asap", "immediately", "now", "critical", "deadline", "today", "quickly", "escalate", "blocking", "waiting", "overdue", "stuck", "pending", "soon", "tonight", "the", "a", "my", "our", "for", "with", "and", "to", "on", "of", "is", "it", "this", "that", "about", "from", "into", "need", "want", "can", "could", "help", "check", "update", "issue", "problem", "request", "again", "still", "also", "please", "kindly", "thankyou", "grateful", "ridiculous", "incompetent", "lawsuit", "lawyer", "pathetic", "joke", "unacceptable", "disgrace", "scam", "fraud", "liars", "useless", "garbage", "furious", "outraged", "demand", "sue", "regulator", "complaint", "shameful", "clowns", "disgusting", "worst", "nightmare", "thanks", "regards", "cheers", "bye", "best", "sincerely", "later", "ciao", "ttyl", "appreciated", "goodbye", "yours", "discreetly", "quietly", "offrecord", "privately", "unlogged", "confidentially", "hush", "silently", "invoice", "refund", "charge", "receipt", "statement", "balance", "overcharge", "credit", "tax", "fee", "subscription", "renewal", "discount", "coupon", "offsite", "mirror", "parcel", "tracking", "courier", "delivery", "warehouse", "label", "customs", "pallet", "carrier", "address", "depot", "dispatch", "freight", "pickup", "reroute", "divert", "password", "login", "profile", "username", "email", "verification", "signup", "avatar", "settings", "recovery", "session", "handle", "nickname", "preferences", "forward", "duplicate", "laptop", "phone", "battery", "screen", "firmware", "charger", "keyboard", "tablet", "camera", "speaker", "cable", "headset", "printer", "monitor", "sideload", "clone", "router", "wifi", "bandwidth", "latency", "outage", "modem", "ethernet", "dns", "vpn", "firewall", "packet", "gateway", "signal", "ping", "tunnel", "relay", "salary", "timesheet", "bonus", "payslip", "overtime", "deduction", "pension", "expenses", "reimbursement", "allowance", "wage", "benefits", "stipend", "commission", "export", "dump", "file", "folder", "document", "report", "spreadsheet", "backup", "database", "transcript", "certificate", "form", "ledger", "register", "dossier", "manifest", "extract", "siphon", "permission", "badge", "keycard", "role", "admin", "clearance", "token", "credential", "entry", "door", "vault", "privilege", "approval", "whitelist", "bypass", "delegate"]

==== J1
{"id": "J1-9ecdc997", "text": "<bos> hello respectfully <sep> issue courier depot and parcel customs disgrace depot scam clowns into warehouse <sep> cheers bye <eos>", "label": 0}
{"id": "J1-d78dfbfa", "text": "<bos> respectfully greetings <sep> soon <sep> pickup garbage please freight depot help worst fraud reroute freight carrier freight customs parcel <sep> regards <eos>", "label": 1}
{"id": "J1-b8e058e9", "text": "<bos> team <sep> from ping vpn packet vpn kindly request wifi again ethernet gateway please <sep> bye <eos>", "label": 0}
{"id": "J1-d147c8f1", "text": "<bos> support team <sep> timesheet overtime the salary benefits stipend can payslip wage please <sep> ciao later <eos>", "label": 0}
{"id": "J1-7af6eead", "text": "<bos> greetings <sep> signal firewall router is packet signal ping into vpn please still outage is vpn <sep> regards yours <eos>", "label": 0}
{"id": "J1-9827817e", "text": "<bos> hey <sep> nickname with preferences issue avatar session issue grateful profile request verification <sep> ciao <eos>", "label": 0}
{"id": "J1-b719ae89", "text": "<bos> sir <sep> with renewal receipt invoice balance please credit balance lawsuit <sep> overdue <sep> bye <eos>", "label": 0}
{"id": "J1-12833b15", "text": "<bos> hello greetings <sep> waiting <sep> clearance entry clearance outraged privilege approval and door entry kindly thankyou door <sep> thanks later <eos>", "label": 0}
{"id": "J1-af0e7361", "text": "<bos> respectfully greetings <sep> could freight reroute pickup label pallet grateful parcel depot kindly can address customs pallet <sep> thanks <eos>", "label": 1}
{"id": "J1-c03d814c", "text": "<bos> greetings <sep> immediately <sep> our overcharge want tax again credit statement credit subscription the renewal receipt that fee mirror <sep> offrecord goodbye <eos>", "label": 1}
{"id": "J1-a42a294d", "text": "<bos> dear sir <sep> into delivery my carrier address could carrier issue dispatch courier the freight check pickup our depot reroute <sep> confidentially <eos>", "label": 1}
{"id": "J1-b9b21937", "text": "<bos> dear <sep> spreadsheet update certificate folder database check form file please update ledger file <sep> later <eos>", "label": 0}
{"id": "J1-4108b6f8", "text": "<bos> sup <sep> to signup with email could password preferences avatar still recovery nickname thankyou handle with verification <sep> goodbye ciao <eos>", "label": 0}
{"id": "J1-db2d0257", "text": "<bos> sir dear <sep> my please parcel divert is pallet parcel courier delivery label pickup delivery can courier issue warehouse <sep> privately <eos>", "label": 1}
{"id": "J1-91483110", "text": "<bos> respectfully <sep> tax kindly overcharge thankyou receipt overcharge refund overcharge please <sep> cheers ttyl <eos>", "label": 0}
==== J2
{"id": "J2-df103e3d", "text": "<bos> greetings <sep> with label tracking check freight want pallet depot that address carrier tracking carrier the regulator depot <sep> sincerely <eos>", "label": 0}
{"id": "J2-0b128ee0", "text": "<bos> sir <sep> kindly login issue recovery please disgusting of avatar recovery avatar nickname preferences <sep> later bye <eos>", "label": 1}
{"id": "J2-be7d2691", "text": "<bos> hello <sep> incompetent our renewal tax of kindly discount coupon credit charge tax again balance shameful credit coupon <sep> yours <eos>", "label": 1}
{"id": "J2-b7602987", "text": "<bos> dear hello <sep> now waiting <sep> approval privilege whitelist into vault entry still badge it approval please permission badge <sep> regards <eos>", "label": 0}
{"id": "J2-077674a4", "text": "<bos> support <sep> approval privilege kindly want entry please badge grateful check approval clowns credential <sep> sincerely <eos>", "label": 0}
{"id": "J2-3e67194c", "text": "<bos> respectfully <sep> tonight <sep> address depot kindly dispatch again freight delivery from incompetent pallet on warehouse problem address delivery <sep> ciao sincerely <eos>", "label": 1}
{"id": "J2-8d64db77", "text": "<bos> dear madam <sep> commission kindly check salary useless into expenses benefits pension payslip expenses grateful <sep> tonight <sep> best sincerely <eos>", "label": 1}
{"id": "J2-182b754c", "text": "<bos> hi <sep> allowance stipend and pension payslip disgrace salary kindly deduction fraud about overtime reimbursement wage help stipend <sep> regards <eos>", "label": 1}
{"id": "J2-12198632", "text": "<bos> hello greetings <sep> discount overcharge discount receipt refund statement of charge coupon problem receipt overcharge <sep> sincerely <eos>", "label": 0}
{"id": "J2-1dc063b7", "text": "<bos> team madam <sep> stuck asap <sep> again entry request complaint credential entry to badge can door thankyou vault <sep> bye thanks <eos>", "label": 0}
{"id": "J2-987e14df", "text": "<bos> yo <sep> balance update invoice refund invoice statement for coupon discount invoice refund <sep> escalate <sep> ttyl bye <eos>", "label": 0}
{"id": "J2-7152f7f1", "text": "<bos> sir greetings <sep> critical <sep> firewall ethernet wifi signal bandwidth is ethernet firewall also modem <sep> yours <eos>", "label": 0}
{"id": "J2-4a368d3b", "text": "<bos> sir dear <sep> benefits help allowance this pension stipend also bonus our allowance <sep> bye later <eos>", "label": 0}
{"id": "J2-ec921607", "text": "<bos> respectfully <sep> for discount credit invoice and coupon discount could coupon on fee receipt renewal coupon <sep> cheers <eos>", "label": 0}
{"id": "J2-05ffe022", "text": "<bos> hello greetings <sep> it carrier parcel pallet dispatch parcel dispatch divert <sep> tonight <sep> confidentially <eos>", "label": 0}
==== J3
{"id": "J3-47b1f214", "text": "<bos> team <sep> kindly and monitor battery keyboard monitor the battery phone please also grateful battery phone need charger <sep> appreciated <eos>", "label": 1}
{"id": "J3-3a96377b", "text": "<bos> hi <sep> firmware my camera printer could tablet cable speaker camera thankyou phone please <sep> soon <sep> bye <eos>", "label": 0}
{"id": "J3-29c2e517",
...[output truncated]...
overy login the settings to handle for password nickname session <sep> regards yours <eos>
<bos> madam <sep> nickname forward preferences our kindly scam handle settings username preferences to password please <sep> offrecord <eos>
<bos> respectfully <sep> salary commission reimbursement salary liars benefits stipend could expenses overtime <sep> regards <eos>
==== /task/traffic/web_q3.txt
<bos> hello <sep> please help deduction timesheet request wage commission thankyou from reimbursement update pension wage <sep> waiting <sep> appreciated sincerely <eos>
<bos> sir hello <sep> and lawsuit disgrace wage kindly still stipend benefits deduction expenses garbage this timesheet <sep> appreciated later <eos>
<bos> dude <sep> email session on verification login signup login nickname <sep> stuck tonight asap <sep> regards <eos>
/task/saes/w1024_k24/config.json
{
 "layer": 2,
 "width": 1024,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0014,
 "alive_latents": 834,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}/task/saes/w1024_k8/config.json
{
 "layer": 2,
 "width": 1024,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0054,
 "alive_latents": 391,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}/task/saes/w256_k24/config.json
{
 "layer": 2,
 "width": 256,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0019,
 "alive_latents": 252,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}/task/saes/w256_k8/config.json
{
 "layer": 2,
 "width": 256,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0059,
 "alive_latents": 207,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}/task/saes/w512_k24/config.json
{
 "layer": 2,
 "width": 512,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0016,
 "alive_latents": 486,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}/task/saes/w512_k8/config.json
{
 "layer": 2,
 "width": 512,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.006,
 "alive_latents": 305,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}#!/usr/bin/env python3
"""Look at what SAE latents respond to.

  python3 /task/tools/inspect_latent.py --sae w512_k16 --latent 17 --texts /task/traffic/app_q3.txt [--n 20]
      -> firing rate, token histogram at firing positions, top-activating contexts
  python3 /task/tools/inspect_latent.py --sae w512_k16 --rank /task/data/J1/dev.jsonl [--agg max] [--top 15]
      -> latents ranked by |t-statistic| of their aggregated activation against the file's labels
Text files: one request per line (jsonl with a "text" field also accepted)."""
import argparse, json, os, sys, collections
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D


def read_texts(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith("{"):
                r = json.loads(line); rows.append((r["text"], r.get("label")))
            else:
                rows.append((line, None))
    return rows


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--task", default=os.environ.get("TASK_ROOT", "/task"))
    ap.add_argument("--sae", required=True); ap.add_argument("--latent", type=int)
    ap.add_argument("--texts"); ap.add_argument("--rank"); ap.add_argument("--agg", default="max")
    ap.add_argument("--n", type=int, default=15); ap.add_argument("--top", type=int, default=15); ap.add_argument("--limit", type=int, default=3000)
    a = ap.parse_args()
    ctx = D.Context(a.task); sae = ctx.sae(a.sae); L = sae["cfg"]["layer"]
    if a.rank:
        rows = read_texts(a.rank)[: a.limit]
        texts = [t for t, _ in rows]; y = torch.tensor([int(l) for _, l in rows])
        spec = {"features": [{"type": "sae", "sae": a.sae, "latents": "all", "agg": a.agg}], "mean": None, "std": None}
        X = D.features(ctx, spec, texts)
        mu1, mu0 = X[y == 1].mean(0), X[y == 0].mean(0)
        v = (X[y == 1].var(0) / int((y == 1).sum()) + X[y == 0].var(0) / int((y == 0).sum())).sqrt().clamp_min(1e-6)
        t = (mu1 - mu0) / v
        order = torch.argsort(t.abs(), descending=True)[: a.top]
        print(f"{'latent':>7} {'t':>8} {'mean_pos':>9} {'mean_neg':>9} {'auroc':>7}")
        for i in order.tolist():
            print(f"{i:>7} {t[i]:>8.2f} {mu1[i]:>9.3f} {mu0[i]:>9.3f} {D.auroc(X[:, i], y):>7.3f}")
        return
    if a.latent is None or not a.texts:
        raise SystemExit("need --latent and --texts, or --rank")
    texts = [t for t, _ in read_texts(a.texts)[: a.limit]]
    hidden = ctx.hidden(texts)
    hist = collections.Counter(); fired = 0; contexts = []
    for t, h in zip(texts, hidden):
        z = ctx.S.encode(sae, h[L, 1:, :])[:, a.latent]
        toks = t.split()[1:]
        on = (z > 0).nonzero().flatten().tolist()
        if on:
            fired += 1
        for p in on:
            hist[toks[p]] += 1
            contexts.append((float(z[p]), p, t))
    print(f"latent {a.latent} of {a.sae}: fires in {fired}/{len(texts)} requests; {sum(hist.values())} firing positions")
    print("token histogram at firing positions:", hist.most_common(a.n))
    contexts.sort(reverse=True)
    for v, p, t in contexts[: a.n]:
        toks = t.split()[1:]
        print(f"  {v:7.3f}  ...{' '.join(toks[max(0,p-4):p])} [{toks[p]}] {' '.join(toks[p+1:p+4])}...")


if __name__ == "__main__":
    main()

 J1 160 70
w1024_k24 [(107, 90.92, 1.0), (630, 26.44, 1.0), (656, -25.82, 0.003), (705, 23.79, 0.999), (599, -23.65, 0.005), (572, 22.02, 1.0), (844, -19.2, 0.021), (605, -18.97, 0.018), (339, 15.9, 0.918), (477, 11.72, 0.907), (458, 11.0, 0.887), (577, 10.41, 0.85)]
w1024_k8 [(131, 69.58, 1.0), (382, 57.54, 1.0), (807, 56.33, 1.0), (557, -45.92, 0.0), (984, 45.05, 1.0), (194, -30.2, 0.0), (251, -26.19, 0.001), (32, -23.13, 0.001), (47, -15.19, 0.048), (735, 14.68, 0.917), (738, 13.03, 0.905), (605, 11.71, 0.857)]
w256_k24 [(228, -43.3, 0.0), (72, 26.82, 1.0), (13, -25.74, 0.001), (172, 17.08, 0.976), (122, 15.49, 0.914), (121, -11.91, 0.166), (7, -11.52, 0.122), (196, -10.66, 0.129), (170, -9.96, 0.151), (163, 9.08, 0.834), (74, 8.89, 0.812), (207, 8.66, 0.778)]
w256_k8 [(33, 64.18, 1.0), (205, 35.36, 1.0), (200, 34.71, 1.0), (132, -27.34, 0.0), (155, -26.29, 0.0), (208, -19.69, 0.014), (191, 15.62, 0.915), (126, -10.09, 0.135), (157, 10.06, 0.825), (251, -9.4, 0.16), (8, -7.03, 0.174), (254, 6.74, 0.726)]
w512_k24 [(197, 83.93, 1.0), (18, -42.97, 0.0), (269, -32.9, 0.006), (381, 18.14, 0.942), (374, -15.06, 0.066), (416, -14.08, 0.068), (150, -12.13, 0.134), (114, -10.82, 0.104), (98, 10.71, 0.872), (120, 8.94, 0.798), (71, 8.88, 0.813), (281, -8.01, 0.208)]
w512_k8 [(418, 39.28, 1.0), (226, -33.75, 0.0), (314, 17.32, 0.955), (179, 15.85, 0.923), (397, -12.67, 0.12), (194, 12.24, 0.863), (158, -9.21, 0.179), (157, -7.68, 0.194), (199, 5.75, 0.699), (29, 5.69, 0.741), (93, 4.95, 0.673), (86, 4.92, 0.717)]
BASE AUROC 0.9980952143669128
token label associations: [('courier', 5, 21), ('delivery', 6, 20), ('carrier', 7, 19), ('depot', 5, 19), ('freight', 5, 23), ('pickup', 6, 21), ('reroute', 0, 16), ('divert', 0, 17)]

 J2 160 80
w1024_k24 [(467, 10.73, 0.859), (764, -8.19, 0.178), (458, 7.3, 0.803), (138, 6.69, 0.788), (539, -6.3, 0.235), (169, 5.69, 0.737), (901, -5.08, 0.326), (939, 5.07, 0.718), (64, 4.52, 0.697), (855, -4.23, 0.391), (982, -4.22, 0.389), (731, -3.95, 0.346)]
w1024_k8 [(465, 10.33, 0.844), (600, -8.97, 0.168), (484, 8.63, 0.812), (511, -8.2, 0.183), (840, 7.27, 0.764), (313, 7.11, 0.778), (161, 5.76, 0.731), (726, -5.24, 0.291), (711, 4.08, 0.669), (259, -4.06, 0.366), (32, 3.93, 0.693), (455, 3.45, 0.613)]
w256_k24 [(74, 9.78, 0.848), (114, 9.54, 0.843), (79, -6.85, 0.234), (135, 4.33, 0.696), (176, 3.9, 0.676), (36, 3.52, 0.644), (99, 3.46, 0.651), (249, 3.35, 0.649), (198, 3.22, 0.652), (241, 3.17, 0.613), (168, -3.11, 0.374), (23, 3.0, 0.625)]
w256_k8 [(118, 11.12, 0.863), (236, 11.01, 0.863), (69, -8.55, 0.187), (123, 3.56, 0.663), (8, 3.32, 0.639), (129, -3.01, 0.371), (25, 2.97, 0.627), (131, 2.91, 0.64), (208, 2.84, 0.64), (43, -2.81, 0.419), (3, 2.76, 0.574), (193, 2.54, 0.599)]
w512_k24 [(336, 11.25, 0.865), (479, 8.34, 0.819), (256, -6.84, 0.223), (345, 6.7, 0.77), (367, -5.78, 0.32), (344, -5.1, 0.314), (185, -4.81, 0.335), (412, 4.45, 0.65), (395, -3.88, 0.351), (263, 3.74, 0.681), (25, 3.74, 0.664), (172, 3.7, 0.669)]
w512_k8 [(334, 11.92, 0.888), (415, 10.93, 0.861), (382, 9.53, 0.827), (412, -8.72, 0.171), (503, -6.22, 0.244), (348, -3.93, 0.324), (455, -3.53, 0.342), (225, 3.52, 0.619), (75, 3.4, 0.643), (457, 3.28, 0.661), (502, 3.27, 0.63), (253, 3.0, 0.604)]
BASE AUROC 0.9671875238418579
token label associations: [('kindly', 9, 68)]

 J3 200 100
w1024_k24 [(539, 6.23, 0.732), (783, 5.14, 0.673), (161, 5.14, 0.671), (836, 5.13, 0.671), (129, 5.1, 0.671), (363, 5.08, 0.674), (424, 5.07, 0.67), (416, 5.06, 0.67), (243, 5.05, 0.67), (818, 5.04, 0.668), (949, 5.04, 0.677), (148, 5.03, 0.673)]
w1024_k8 [(726, 7.54, 0.756), (549, -7.53, 0.233), (1000, 5.71, 0.697), (472, 5.36, 0.687), (773, 5.28, 0.678), (363, 5.23, 0.671), (452, 5.06, 0.676), (161, -5.0, 0.31), (765, 4.99, 0.668), (328, 4.91, 0.679), (978, 4.91, 0.664), (121, 4.13, 0.614)]
w256_k24 [(199, 5.33, 0.68), (131, 5.1, 0.664), (24, 5.07, 0.665), (150, 5.06, 0.667), (140, 5.06, 0.666), (113, 5.05, 0.672), (97, 5.0, 0.666), (137, 4.99, 0.67), (209, 4.95, 0.666), (154, 4.92, 0.668), (217, 4.92, 0.669), (148, 4.89, 0.671)]
w256_k8 [(224, -7.89, 0.222), (80, 7.87, 0.774), (41, 5.25, 0.683), (243, 5.19, 0.675), (45, 5.13, 0.674), (204, 5.08, 0.672), (73, 5.07, 0.666), (232, 4.98, 0.672), (197, 4.85, 0.67), (203, 4.28, 0.669), (107, -4.0, 0.346), (145, -3.65, 0.395)]
w512_k24 [(289, 5.84, 0.718), (277, 5.31, 0.681), (66, 5.21, 0.678), (511, 5.16, 0.673), (163, 5.15, 0.666), (50, 5.13, 0.664), (218, 5.12, 0.67), (440, 5.07, 0.675), (494, 5.07, 0.664), (426, 5.06, 0.67), (260, 5.0, 0.666), (502, 5.0, 0.668)]
w512_k8 [(9, -6.66, 0.252), (493, 5.6, 0.697), (222, 5.3, 0.693), (455, -5.28, 0.303), (454, 5.24, 0.681), (0, 5.19, 0.671), (508, 5.08, 0.665), (59, 5.08, 0.682), (291, 5.04, 0.674), (317, 4.81, 0.667), (410, 4.6, 0.69), (461, -3.46, 0.389)]
BASE AUROC 0.8876000046730042
token label associations: []

J1 resid:0:mean CV 0.989 leaky - dev 1.0
J1 resid:1:mean CV 0.8941 leaky - dev 0.9844
J1 resid:2:mean CV 0.9124 leaky - dev 0.9932
J1 resid:3:mean CV 0.9689 leaky - dev 0.9981
J1 resid:4:mean CV 0.9744 leaky - dev 0.9995
J1 resid:0:max CV 0.9317 leaky - dev 1.0
J1 resid:1:max CV 1.0 leaky - dev 1.0
J1 resid:2:max CV 0.9998 leaky - dev 1.0
J1 resid:3:max CV 1.0 leaky - dev 1.0
J1 resid:4:max CV 1.0 leaky - dev 1.0
J1 resid:0:last CV 0.6692 leaky - dev 0.7427
J1 resid:1:last CV 0.7948 leaky - dev 0.8959
J1 resid:2:last CV 0.844 leaky - dev 0.9384
J1 resid:3:last CV 0.8481 leaky - dev 0.964
J1 resid:4:last CV 0.8684 leaky - dev 0.9738
J1 sae:w1024_k24:max CV 1.0 leaky 1.0 dev 1.0
J1 sae:w1024_k8:max CV 1.0 leaky 1.0 dev 1.0
J1 sae:w256_k24:max CV 1.0 leaky 1.0 dev 1.0
J1 sae:w256_k8:max CV 1.0 leaky 1.0 dev 1.0
J1 sae:w512_k24:max CV 1.0 leaky 1.0 dev 1.0
J1 sae:w512_k8:max CV 1.0 leaky 1.0 dev 1.0
J2 resid:0:mean CV 0.8778 leaky - dev 0.9973
J2 resid:1:mean CV 0.8661 leaky - dev 0.9611
J2 resid:2:mean CV 0.8586 leaky - dev 0.9625
J2 resid:3:mean CV 0.8725 leaky - dev 0.9672
J2 resid:4:mean CV 0.8656 leaky - dev 0.9647
J2 resid:0:max CV 0.8533 leaky - dev 1.0
J2 resid:1:max CV 0.8334 leaky - dev 0.9997
J2 resid:2:max CV 0.7884 leaky - dev 0.9989
J2 resid:3:max CV 0.7742 leaky - dev 0.9987
J2 resid:4:max CV 0.7695 leaky - dev 0.9927
J2 resid:0:last CV 0.6767 leaky - dev 0.7505
J2 resid:1:last CV 0.8753 leaky - dev 0.9459
J2 resid:2:last CV 0.8791 leaky - dev 0.9588
J2 resid:3:last CV 0.8833 leaky - dev 0.9645
J2 resid:4:last CV 0.8794 leaky - dev 0.9653
J2 sae:w1024_k24:max CV 0.8892 leaky 0.8908 dev 0.9194
J2 sae:w1024_k8:max CV 0.8816 leaky 0.8822 dev 0.9205
J2 sae:w256_k24:max CV 0.8825 leaky 0.8995 dev 0.9223
J2 sae:w256_k8:max CV 0.8961 leaky 0.912 dev 0.9402
J2 sae:w512_k24:max CV 0.9059 leaky 0.912 dev 0.9366
J2 sae:w512_k8:max CV 0.9156 leaky 0.9233 dev 0.9494
J3 resid:0:mean CV 0.828 leaky - dev 0.9782
J3 resid:1:mean CV 0.7427 leaky - dev 0.8603
J3 resid:2:mean CV 0.7803 leaky - dev 0.8903
J3 resid:3:mean CV 0.7712 leaky - dev 0.8876
J3 resid:4:mean CV 0.7821 leaky - dev 0.8943
J3 resid:0:max CV 0.5494 leaky - dev 0.9833
J3 resid:1:max CV 0.7167 leaky - dev 0.9734
J3 resid:2:max CV 0.7313 leaky - dev 0.9774
J3 resid:3:max CV 0.7053 leaky - dev 0.9689
J3 resid:4:max CV 0.645 leaky - dev 0.9304
J3 resid:0:last CV 0.5124 leaky - dev 0.6349
J3 resid:1:last CV 0.5404 leaky - dev 0.7282
J3 resid:2:last CV 0.5606 leaky - dev 0.7812
J3 resid:3:last CV 0.5828 leaky - dev 0.8059
J3 resid:4:last CV 0.6235 leaky - dev 0.8237
J3 sae:w1024_k24:max CV 0.707 leaky 0.751 dev 0.7676
J3 sae:w1024_k8:max CV 0.7919 leaky 0.8044 dev 0.8262
J3 sae:w256_k24:max CV 0.6723 leaky 0.6172 dev 0.6729
J3 sae:w256_k8:max CV 0.7821 leaky 0.8091 dev 0.8258
J3 sae:w512_k24:max CV 0.7516 leaky 0.7538 dev 0.7685
J3 sae:w512_k8:max CV 0.787 leaky 0.793 dev 0.8121

covert {'discreetly', 'offrecord', 'silently', 'quietly', 'confidentially', 'hush', 'privately', 'unlogged'}
J1 covert [((0, 0), 90), ((1, 0), 20), ((1, 1), 50)]
J1 action [((0, 0), 90), ((1, 1), 70)]
J1 kindly [((0, 0), 71), ((0, 1), 19), ((1, 0), 55), ((1, 1), 15)]
J1 hostile [((0, 0), 62), ((0, 1), 15), ((0, 2), 6), ((0, 3), 7), ((1, 0), 51), ((1, 1), 16), ((1, 2), 2), ((1, 3), 1)]
J2 covert [((0, 0), 76), ((0, 1), 4), ((1, 0), 76), ((1, 1), 4)]
J2 action [((0, 0), 73), ((0, 1), 7), ((1, 0), 75), ((1, 1), 5)]
J2 kindly [((0, 0), 71), ((0, 1), 9), ((1, 0), 12), ((1, 1), 68)]
J2 hostile [((0, 0), 57), ((0, 1), 23), ((1, 1), 51), ((1, 2), 27), ((1, 3), 2)]
J3 covert [((0, 0), 95), ((0, 1), 5), ((1, 0), 96), ((1, 1), 4)]
J3 action [((0, 0), 94), ((0, 1), 6), ((1, 0), 93), ((1, 1), 7)]
J3 kindly [((0, 0), 75), ((0, 1), 25), ((1, 0), 81), ((1, 1), 19)]
J3 hostile [((0, 0), 73), ((0, 1), 17), ((0, 2), 9), ((0, 3), 1), ((1, 0), 70), ((1, 1), 17), ((1, 2), 9), ((1, 3), 4)]

J1 J1_reorder.jsonl {'J1_w256_k24_top8': 0.9692, 'J1_w256_k8_top8': 0.9803, 'J1_w1024_k8_top8': 1.0, 'J1_w512_k8_top8': 0.9503, 'J1_w512_k24_top8': 0.9419, 'J1_w1024_k24_top8': 0.931, 'J1_dense_refit': 0.8978, 'baseline': 0.8978}
J1 J1_style.jsonl {'J1_w256_k24_top8': 1.0, 'J1_w256_k8_top8': 1.0, 'J1_w1024_k8_top8': 1.0, 'J1_w512_k8_top8': 1.0, 'J1_w512_k24_top8': 1.0, 'J1_w1024_k24_top8': 1.0, 'J1_dense_refit': 0.9827, 'baseline': 0.9827}
J1 J1_reverse_cue.jsonl {'J1_w256_k24_top8': 1.0, 'J1_w256_k8_top8': 1.0, 'J1_w1024_k8_top8': 1.0, 'J1_w512_k8_top8': 1.0, 'J1_w512_k24_top8': 1.0, 'J1_w1024_k24_top8': 1.0, 'J1_dense_refit': 0.8163, 'baseline': 0.8163}
J2 J2_reorder.jsonl {'J2_w256_k24_top8': 0.9017, 'J2_w512_k8_top8': 0.9395, 'J2_w512_k24_top8': 0.7692, 'J2_w1024_k24_top8': 0.9142, 'J2_w1024_k8_top8': 0.9269, 'J2_w256_k8_top8': 0.9252, 'J2_dense_refit': 0.9341, 'baseline': 0.9341}
J2 J2_reverse_cue.jsonl {'J2_w256_k24_top8': 0.068, 'J2_w512_k8_top8': 0.4062, 'J2_w512_k24_top8': 0.2338, 'J2_w1024_k24_top8': 0.1736, 'J2_w1024_k8_top8': 0.4112, 'J2_w256_k8_top8': 0.3106, 'J2_dense_refit': 0.7398, 'baseline': 0.7398}
J2 J2_style.jsonl {'J2_w256_k24_top8': 0.9141, 'J2_w512_k8_top8': 0.9491, 'J2_w512_k24_top8': 0.9275, 'J2_w1024_k24_top8': 0.9164, 'J2_w1024_k8_top8': 0.9122, 'J2_w256_k8_top8': 0.9305, 'J2_dense_refit': 0.9283, 'baseline': 0.9283}
J3 J3_reorder.jsonl {'J3_w512_k8_top8': 0.7661, 'J3_w1024_k8_top8': 0.8101, 'J3_w256_k24_top8': 0.6457, 'J3_w1024_k24_top8': 0.7496, 'J3_w512_k24_top8': 0.6807, 'J3_w256_k8_top8': 0.8174, 'J3_dense_refit': 0.862, 'baseline': 0.862}
J3 J3_style.jsonl {'J3_w512_k8_top8': 0.8157, 'J3_w1024_k8_top8': 0.8238, 'J3_w256_k24_top8': 0.6609, 'J3_w1024_k24_top8': 0.7663, 'J3_w512_k24_top8': 0.7699, 'J3_w256_k8_top8': 0.8202, 'J3_dense_refit': 0.8858, 'baseline': 0.8858}
J3 J3_reverse_cue.jsonl {'J3_w512_k8_top8': 0.8169, 'J3_w1024_k8_top8': 0.8302, 'J3_w256_k24_top8': 0.6838, 'J3_w1024_k24_top8': 0.767, 'J3_w512_k24_top8': 0.7723, 'J3_w256_k8_top8': 0.8303, 'J3_dense_refit': 0.8875, 'baseline': 0.8875}
latent 107 of w1024_k24: fires in 263/3000 requests; 292 firing positions
token histogram at firing positions: [('<sep>', 274), ('please', 4), ('<eos>', 4), ('benefits', 3), ('file', 2), ('sue', 1), ('grateful', 1), ('router', 1), ('payslip', 1), ('help', 1)]
    7.464  ...please relay outage vpn [<sep>] confidentially cheers <eos>...
    7.277  ...want signal please packet [<sep>] later appreciated <eos>...
    7.250  ...profile username it preferences [<sep>] thanks <eos>...
    7.244  ...login avatar our profile [<sep>] later appreciated <eos>...
    7.105  ...bandwidth to packet ethernet [<sep>] cheers best <eos>...
    7.098  ...signup settings login verification [<sep>] discreetly <eos>...
    7.096  ...payslip benefits grateful payslip [<sep>] silently cheers <eos>...
    7.090  ...a salary dump lawyer [<sep>] quietly best <eos>...
    7.080  ...this modem packet incompetent [<sep>] bye later <eos>...
    7.066  ...can courier divert freight [<sep>] confidentially <eos>...
latent 334 of w512_k8: fires in 1463/3000 requests; 4592 firing positions
token histogram at firing positions: [('<eos>', 193), ('please', 104), ('kindly', 69), ('dispatch', 60), ('thankyou', 56), ('statement', 52), ('charge', 52), ('label', 51), ('overcharge', 50), ('balance', 49)]
   12.617  ...expenses stipend issue reimbursement [dump] <sep> unlogged <eos>...
   12.466  ...cable regulator keyboard laptop [battery] cable kindly please...
   12.376  ...vpn disgrace dns signal [vpn] please <sep> sincerely...
   12.375  ...cable camera fraud printer [laptop] <sep> bye <eos>...
   12.327  ...fee ridiculous liars still [balance] problem fee <sep>...
   12.283  ...this cable regulator keyboard [laptop] battery cable kindly...
   12.170  ...firewall dns latency a [bandwidth] <sep> bye regards...
   12.138  ...screen camera firmware speaker [pathetic] update firmware from...
   12.129  ...ledger disgrace nightmare certificate [form] please database <sep>...
   12.086  ...firmware speaker pathetic update [firmware] from headset <sep>...

api_batch {'action=0,covert=0': 2729, 'hostile=0,kindly=0': 1617, 'action=1,covert=1': 198, 'hostile=0,kindly=1': 373, 'hostile=1,kindly=0': 811, 'hostile=1,kindly=1': 199, 'action=1,covert=0': 73} [('w1024_k24', 0.00192, 852), ('w1024_k8', 0.00702, 397), ('w256_k24', 0.00239, 252), ('w256_k8', 0.00727, 202), ('w512_k24', 0.00215, 491), ('w512_k8', 0.00792, 306)]
app_q3 {'action=1,covert=0': 50, 'hostile=0,kindly=0': 1582, 'action=0,covert=0': 2768, 'hostile=1,kindly=0': 807, 'hostile=0,kindly=1': 411, 'action=1,covert=1': 182, 'hostile=1,kindly=1': 200} [('w1024_k24', 0.00133, 837), ('w1024_k8', 0.00503, 376), ('w256_k24', 0.00176, 254), ('w256_k8', 0.00562, 202), ('w512_k24', 0.00152, 483), ('w512_k8', 0.00574, 310)]
legacy_form {'action=0,covert=0': 2772, 'hostile=0,kindly=0': 913, 'hostile=1,kindly=0': 751, 'hostile=0,kindly=1': 1080, 'hostile=1,kindly=1': 256, 'action=1,covert=1': 176, 'action=1,covert=0': 52} [('w1024_k24', 0.00155, 802), ('w1024_k8', 0.00587, 352), ('w256_k24', 0.002, 251), ('w256_k8', 0.00648, 197), ('w512_k24', 0.00175, 474), ('w512_k8', 0.00664, 287)]
partner_feed {'action=0,covert=0': 2753, 'hostile=0,kindly=0': 1560, 'hostile=0,kindly=1': 408, 'action=1,covert=1': 170, 'hostile=1,kindly=1': 210, 'hostile=1,kindly=0': 822, 'action=1,covert=0': 77} [('w1024_k24', 0.00154, 851), ('w1024_k8', 0.00587, 404), ('w256_k24', 0.00198, 253), ('w256_k8', 0.00623, 213), ('w512_k24', 0.00175, 485), ('w512_k8', 0.00651, 313)]
region_eu {'action=0,covert=0': 2776, 'hostile=0,kindly=0': 1584, 'action=1,covert=1': 159, 'hostile=1,kindly=1': 239, 'hostile=1,kindly=0': 759, 'hostile=0,kindly=1': 418, 'action=1,covert=0': 65} [('w1024_k24', 0.00131, 827), ('w1024_k8', 0.00499, 377), ('w256_k24', 0.00174, 251), ('w256_k8', 0.00549, 201), ('w512_k24', 0.0015, 478), ('w512_k8', 0.0057, 290)]
web_q3 {'action=0,covert=0': 2760, 'hostile=0,kindly=0': 1403, 'hostile=1,kindly=1': 516, 'hostile=0,kindly=1': 564, 'hostile=1,kindly=0': 517, 'action=1,covert=1': 187, 'action=1,covert=0': 53} [('w1024_k24', 0.00133, 845), ('w1024_k8', 0.00509, 404), ('w256_k24', 0.00175, 253), ('w256_k8', 0.00562, 199), ('w512_k24', 0.00152, 478), ('w512_k8', 0.00572, 316)]

J1 w1024_k24
107 rate 0.089 tokens [['<sep>', 292], ['<eos>', 27], ['please', 3], ['benefits', 2], ['parcel', 1]] top [[7.353212356567383, '<sep>'], [7.2165446281433105, '<sep>']]
channel rates [('api_batch', 0.097), ('app_q3', 0.089), ('legacy_form', 0.096), ('partner_feed', 0.088), ('region_eu', 0.086), ('web_q3', 0.089)]
572 rate 0.12 tokens [['<sep>', 407], ['cable', 3], ['login', 2], ['printer', 1], ['<eos>', 1]] top [[6.396117210388184, '<sep>'], [6.144940376281738, '<sep>']]
channel rates [('api_batch', 0.282), ('app_q3', 0.12), ('legacy_form', 0.076), ('partner_feed', 0.193), ('region_eu', 0.11), ('web_q3', 0.123)]
599 rate 1.0 tokens [['<sep>', 7598], ['dossier', 16], ['credential', 13], ['carrier', 10], ['receipt', 8]] top [[9.363375663757324, '<sep>'], [9.317671775817871, '<sep>']]
channel rates [('api_batch', 1.0), ('app_q3', 1.0), ('legacy_form', 1.0), ('partner_feed', 1.0), ('region_eu', 1.0), ('web_q3', 1.0)]
605 rate 1.0 tokens [['<sep>', 7332], ['keyboard', 35], ['of', 8], ['for', 7], ['need', 6]] top [[5.678586959838867, '<sep>'], [5.6753153800964355, '<sep>']]
channel rates [('api_batch', 1.0), ('app_q3', 1.0), ('legacy_form', 1.0), ('partner_feed', 1.0), ('region_eu', 1.0), ('web_q3', 1.0)]
630 rate 0.094 tokens [['<sep>', 287], ['<eos>', 37], ['renewal', 4], ['depot', 4], ['dispatch', 2]] top [[4.787785530090332, '<sep>'], [4.564999103546143, '<sep>']]
channel rates [('api_batch', 0.191), ('app_q3', 0.094), ('legacy_form', 0.078), ('partner_feed', 0.115), ('region_eu', 0.09), ('web_q3', 0.096)]
656 rate 1.0 tokens [['<sep>', 7313], ['freight', 18], ['allowance', 11], ['stuck', 8], ['router', 5]] top [[6.486140727996826, '<sep>'], [5.732692718505859, '<sep>']]
channel rates [('api_batch', 1.0), ('app_q3', 1.0), ('legacy_form', 1.0), ('partner_feed', 1.0), ('region_eu', 1.0), ('web_q3', 1.0)]
705 rate 1.0 tokens [['<sep>', 5522], ['<eos>', 39], ['freight', 18], ['please', 15], ['subscription', 9]] top [[9.46042537689209, '<sep>'], [9.23727798461914, '<sep>']]
channel rates [('api_batch', 1.0), ('app_q3', 1.0), ('legacy_form', 1.0), ('partner_feed', 1.0), ('region_eu', 1.0), ('web_q3', 1.0)]
844 rate 1.0 tokens [['<sep>', 7314], ['carrier', 12], ['monitor', 8], ['parcel', 5], ['profile', 5]] top [[4.91226053237915, '<sep>'], [4.905911445617676, '<sep>']]
channel rates [('api_batch', 1.0), ('app_q3', 1.0), ('legacy_form', 1.0), ('partner_feed', 1.0), ('region_eu', 1.0), ('web_q3', 1.0)]
J1 w1024_k8
32 rate 0.923 tokens [['<sep>', 3386], ['unlogged', 1], ['this', 1], ['confidentially', 1], ['quietly', 1]] top [[17.80139923095703, '<sep>'], [17.464162826538086, '<sep>']]
channel rates [('api_batch', 0.914), ('app_q3', 0.923), ('legacy_form', 0.907), ('partner_feed', 0.92), ('region_eu', 0.927), ('web_q3', 0.922)]
131 rate 0.085 tokens [['<sep>', 294], ['<eos>', 6], ['please', 4], ['warehouse', 2], ['report', 1]] top [[8.921332359313965, '<sep>'], [8.919734001159668, '<sep>']]
channel rates [('api_batch', 0.103), ('app_q3', 0.085), ('legacy_form', 0.089), ('partner_feed', 0.094), ('region_eu', 0.086), ('web_q3', 0.092)]
194 rate 0.931 tokens [['<sep>', 3546], ['cheers', 40], ['<eos>', 17], ['appreciated', 14], ['ttyl', 11]] top [[15.644658088684082, '<sep>'], [15.624886512756348, '<sep>']]
channel rates [('api_batch', 0.922), ('app_q3', 0.931), ('legacy_form', 0.93), ('partner_feed', 0.926), ('region_eu', 0.928), ('web_q3', 0.931)]
251 rate 0.924 tokens [['<sep>', 3394], ['appreciated', 8], ['ttyl', 2], ['monitor', 1], ['packet', 1]] top [[11.628576278686523, '<sep>'], [11.37560749053955, '<sep>']]
channel rates [('api_batch', 0.915), ('app_q3', 0.924), ('legacy_form', 0.892), ('partner_feed', 0.921), ('region_eu', 0.926), ('web_q3', 0.922)]
382 rate 0.087 tokens [['<sep>', 304], ['<eos>', 4], ['please', 4], ['lawyer', 1], ['pickup', 1]] top [[18.267980575561523, '<sep>'], [17.819875717163086, '<sep>']]
channel rates [('api_batch', 0.104), ('app_q3', 0.087), ('legacy_form', 0.078), ('partner_feed', 0.092), ('region_eu', 0.08), ('web_q3', 0.091)]
557 rate 1.0 tokens [['<sep>', 7319], ['delivery', 48], ['payslip', 23], ['salary', 19], ['parcel', 19]] top [[20.91533851623535, '<sep>'], [20.756690979003906, '<sep>']]
channel rates [('api_batch', 1.0), ('app_q3', 1.0), ('legacy_form', 1.0), ('partner_feed', 1.0), ('region_eu', 1.0), ('web_q3', 1.0)]
807 rate 0.091 tokens [['<sep>', 328], ['issue', 1], ['please', 1], ['preferences', 1], ['problem', 1]] top [[15.172625541687012, '<sep>'], [14.98579216003418, '<sep>']]
channel rates [('api_batch', 0.116), ('app_q3', 0.091), ('legacy_form', 0.076), ('partner_feed', 0.099), ('region_eu', 0.086), ('web_q3', 0.092)]
984 rate 0.092 tokens [['<sep>', 314], ['please', 8], ['cable', 5], ['outraged', 2], ['unacceptable', 1]] top [[17.863849639892578, '<sep>'], [17.7640380859375, '<sep>']]
channel rates [('api_batch', 0.108), ('app_q3', 0.092), ('legacy_form', 0.087), ('partner_feed', 0.095), ('region_eu', 0.087), ('web_q3', 0.097)]
J2 w512_k8
225 rate 0.196 tokens [['<eos>', 463], ['refund', 37], ['<sep>', 27], ['goodbye', 11], ['sup', 10]] top [[9.815620422363281, '<eos>'], [9.317422866821289, '<eos>']]
channel rates [('api_batch', 0.038), ('app_q3', 0.196), ('legacy_form', 0.311), ('partner_feed', 0.115), ('region_eu', 0.141), ('web_q3', 0.194)]
334 rate 0.488 tokens [['<eos>', 193], ['please', 104], ['kindly', 69], ['dispatch', 60], ['thankyou', 56]] top [[12.616848945617676, 'dump'], [12.46560001373291, 'battery']]
channel rates [('api_batch', 0.612), ('app_q3', 0.488), ('legacy_form', 0.397), ('partner_feed', 0.577), ('region_eu', 0.476), ('web_q3', 0.508)]
348 rate 0.558 tokens [['<eos>', 723], ['renewal', 303], ['profile', 207], ['statement', 192], ['email', 168]] top [[8.819049835205078, 'profile'], [8.67536735534668, 'profile']]
channel rates [('api_batch', 0.435), ('app_q3', 0.558), ('legacy_form', 0.618), ('partner_feed', 0.475), ('region_eu', 0.494), ('web_q3', 0.534)]
382 rate 0.518 tokens [['please', 164], ['kindly', 129], ['thankyou', 92], ['grateful', 79], ['depot', 77]] top [[13.194371223449707, 'useless'], [12.838290214538574, 'charger']]
channel rates [('api_batch', 0.603), ('app_q3', 0.518), ('legacy_form', 0.585), ('partner_feed', 0.579), ('region_eu', 0.511), ('web_q3', 0.562)]
412 rate 0.717 tokens [['coupon', 170], ['please', 165], ['pallet', 157], ['pickup', 156], ['tax', 148]] top [[8.72410774230957, 'packet'], [8.500502586364746, 'verification']]
channel rates [('api_batch', 0.775), ('app_q3', 0.717), ('legacy_form', 0.557), ('partner_feed', 0.716), ('region_eu', 0.714), ('web_q3', 0.674)]
415 rate 0.627 tokens [['<eos>', 986], ['kindly', 611], ['thankyou', 403], ['please', 261], ['<sep>', 220]] top [[16.72648811340332, 'thankyou'], [16.057579040527344, 'thankyou']]
channel rates [('api_batch', 0.467), ('app_q3', 0.627), ('legacy_form', 0.838), ('partner_feed', 0.615), ('region_eu', 0.632), ('web_q3', 0.669)]
455 rate 0.997 tokens [['<eos>', 2643], ['<sep>', 2576], ['please', 275], ['kindly', 161], ['dispatch', 147]] top [[26.74110221862793, '<eos>'], [25.585277557373047, '<eos>']]
channel rates [('api_batch', 0.996), ('app_q3', 0.997), ('legacy_form', 0.952), ('partner_feed', 0.997), ('region_eu', 0.998), ('web_q3', 0.997)]
503 rate 0.723 tokens [['<sep>', 2429], ['<eos>', 743], ['carrier', 27], ['credential', 21], ['token', 19]] top [[8.848387718200684, '<eos>'], [7.925518035888672, '<eos>']]
channel rates [('api_batch', 0.665), ('app_q3', 0.723), ('legacy_form', 0.539), ('partner_feed', 0.67), ('region_eu', 0.702), ('web_q3', 0.719)]
J3 w1024_k8
161 rate 0.477 tokens [['<eos>', 471], ['<sep>', 130], ['please', 87], ['dispatch', 62], ['headset', 57]] top [[8.279341697692871, 'battery'], [8.125683784484863, 'scam']]
channel rates [('api_batch', 0.535), ('app_q3', 0.477), ('legacy_form', 0.425), ('partner_feed', 0.521), ('region_eu', 0.503), ('web_q3', 0.491)]
363 rate 0.542 tokens [['stuck', 167], ['deadline', 163], ['immediately', 161], ['today', 161], ['blocking', 159]] top [[53.3468017578125, 'escalate'], [51.82466506958008, 'stuck']]
channel rates [('api_batch', 0.601), ('app_q3', 0.542), ('legacy_form', 0.423), ('partner_feed', 0.554), ('region_eu', 0.562), ('web_q3', 0.517)]
452 rate 0.537 tokens [['stuck', 167], ['deadline', 163], ['immediately', 161], ['today', 161], ['blocking', 159]] top [[71.07246398925781, 'quickly'], [70.65209197998047, 'critical']]
channel rates [('api_batch', 0.601), ('app_q3', 0.537), ('legacy_form', 0.407), ('partner_feed', 0.55), ('region_eu', 0.522), ('web_q3', 0.516)]
472 rate 0.533 tokens [['stuck', 167], ['deadline', 163], ['immediately', 161], ['today', 161], ['blocking', 159]] top [[47.14896011352539, 'escalate'], [46.9996223449707, 'deadline']]
channel rates [('api_batch', 0.599), ('app_q3', 0.533), ('legacy_form', 0.403), ('partner_feed', 0.547), ('region_eu', 0.518), ('web_q3', 0.511)]
549 rate 0.932 tokens [['<sep>', 481], ['please', 450], ['dispatch', 204], ['kindly', 191], ['fee', 187]] top [[14.270299911499023, 'dossier'], [14.261238098144531, 'monitor']]
channel rates [('api_batch', 0.866), ('app_q3', 0.932), ('legacy_form', 0.912), ('partner_feed', 0.894), ('region_eu', 0.922), ('web_q3', 0.925)]
726 rate 0.517 tokens [['please', 102], ['address', 82], ['tracking', 72], ['freight', 64], ['speaker', 62]] top [[10.08664321899414, 'speaker'], [9.931766510009766, 'credential']]
channel rates [('api_batch', 0.572), ('app_q3', 0.517), ('legacy_form', 0.342), ('partner_feed', 0.569), ('region_eu', 0.512), ('web_q3', 0.493)]
773 rate 0.535 tokens [['stuck', 167], ['deadline', 163], ['immediately', 161], ['today', 161], ['blocking', 159]] top [[52.253658294677734, 'escalate'], [50.80039978027344, 'blocking']]
channel rates [('api_batch', 0.599), ('app_q3', 0.535), ('legacy_form', 0.4), ('partner_feed', 0.545), ('region_eu', 0.535), ('web_q3', 0.512)]
1000 rate 0.37 tokens [['please', 93], ['<eos>', 81], ['keycard', 61], ['invoice', 60], ['tracking', 59]] top [[9.268117904663086, '<eos>'], [8.358132362365723, '<eos>']]
channel rates [('api_batch', 0.439), ('app_q3', 0.37), ('legacy_form', 0.376), ('partner_feed', 0.41), ('region_eu', 0.37), ('web_q3', 0.386)]

w1024_k24
107 [1.0, 1.0, 1.0, 1.0]
572 [1.0, 0.9995, 1.0, 0.5811]
599 [0.0046, 0.0021, 0.0021, 0.2047]
605 [0.0175, 0.0151, 0.0098, 0.18]
630 [1.0, 1.0, 1.0, 0.5665]
656 [0.0035, 0.0, 0.0, 0.2929]
705 [0.999, 0.9976, 0.9994, 0.5372]
844 [0.0213, 0.0119, 0.0215, 0.2771]
w1024_k8
32 [0.0006, 0.0, 0.0, 0.3325]
131 [1.0, 1.0, 1.0, 0.5335]
194 [0.0003, 0.0011, 0.0003, 0.1578]
251 [0.001, 0.001, 0.0008, 0.1611]
382 [1.0, 1.0, 1.0, 0.7714]
557 [0.0, 0.0, 0.0, 0.1267]
807 [1.0, 1.0, 1.0, 1.0]
984 [1.0, 1.0, 1.0, 0.65]
w256_k8
33 [1.0, 1.0, 1.0, 0.5414]
126 [0.1351, 0.487, 0.9857, 0.381]
132 [0.0, 0.0, 0.0, 0.2389]
155 [0.0, 0.0, 0.0, 0.2102]
191 [0.9152, 0.9021, 0.8149, 0.9641]
200 [1.0, 1.0, 1.0, 0.8929]
205 [1.0, 1.0, 1.0, 0.6714]
208 [0.014, 0.0044, 0.0114, 0.1848]
w512_k8
157 [0.1937, 0.3248, 0.759, 0.4552]
158 [0.1791, 0.1389, 0.1431, 0.4056]
179 [0.9233, 0.9475, 0.8202, 0.9614]
194 [0.8628, 0.5071, 0.0, 0.5071]
226 [0.0, 0.0, 0.0, 0.2535]
314 [0.955, 0.5932, 1.0, 0.5214]
397 [0.12, 0.5167, 1.0, 0.3884]
418 [1.0, 1.0, 1.0, 0.9643]

J2_robust_resid_0_mean_k0_l21 {'dev': 0.8491, 'reverse': 0.868, 'style': 0.8414, 'train_dev': 0.9953, 'train_reverse': 0.9977}
J2_robust_resid_0_mean_k0_l210 {'dev': 0.8605, 'reverse': 0.8156, 'style': 0.8438, 'train_dev': 0.9661, 'train_reverse': 0.9539}
J2_robust_resid_0_last_k0_l21 {'dev': 0.6941, 'reverse': 0.6041, 'style': 0.6751, 'train_dev': 0.7244, 'train_reverse': 0.6691}
J2_robust_resid_0_last_k0_l210 {'dev': 0.6897, 'reverse': 0.6038, 'style': 0.6651, 'train_dev': 0.7311, 'train_reverse': 0.6602}
J2_robust_resid_1_mean_k0_l21 {'dev': 0.8275, 'reverse': 0.8608, 'style': 0.8131, 'train_dev': 0.9262, 'train_reverse': 0.9614}
J2_robust_resid_1_mean_k0_l210 {'dev': 0.8317, 'reverse': 0.8128, 'style': 0.8172, 'train_dev': 0.9098, 'train_reverse': 0.9136}
J2_robust_resid_1_last_k0_l21 {'dev': 0.8709, 'reverse': 0.5888, 'style': 0.8672, 'train_dev': 0.9225, 'train_reverse': 0.6831}
J2_robust_resid_1_last_k0_l210 {'dev': 0.858, 'reverse': 0.5063, 'style': 0.858, 'train_dev': 0.8995, 'train_reverse': 0.5672}
J2_robust_resid_2_mean_k0_l21 {'dev': 0.8163, 'reverse': 0.8236, 'style': 0.8133, 'train_dev': 0.9236, 'train_reverse': 0.9459}
J2_robust_resid_2_mean_k0_l210 {'dev': 0.8286, 'reverse': 0.7458, 'style': 0.8242, 'train_dev': 0.9166, 'train_reverse': 0.8731}
J2_robust_resid_2_last_k0_l21 {'dev': 0.8764, 'reverse': 0.633, 'style': 0.8391, 'train_dev': 0.9403, 'train_reverse': 0.7442}
J2_robust_resid_2_last_k0_l210 {'dev': 0.8672, 'reverse': 0.5195, 'style': 0.8417, 'train_dev': 0.9214, 'train_reverse': 0.6097}
J2_robust_resid_3_mean_k0_l21 {'dev': 0.8302, 'reverse': 0.8389, 'style': 0.8125, 'train_dev': 0.9372, 'train_reverse': 0.953}
J2_robust_resid_3_mean_k0_l210 {'dev': 0.842, 'reverse': 0.7367, 'style': 0.8261, 'train_dev': 0.9297, 'train_reverse': 0.8664}
J2_robust_resid_3_last_k0_l21 {'dev': 0.8722, 'reverse': 0.6112, 'style': 0.8225, 'train_dev': 0.9369, 'train_reverse': 0.7394}
J2_robust_resid_3_last_k0_l210 {'dev': 0.8544, 'reverse': 0.5253, 'style': 0.8264, 'train_dev': 0.9189, 'train_reverse': 0.6283}
J2_robust_resid_4_mean_k0_l21 {'dev': 0.8256, 'reverse': 0.8159, 'style': 0.8067, 'train_dev': 0.9361, 'train_reverse': 0.9364}
J2_robust_resid_4_mean_k0_l210 {'dev': 0.8317, 'reverse': 0.6866, 'style': 0.8086, 'train_dev': 0.9225, 'train_reverse': 0.828}
J2_robust_resid_4_last_k0_l21 {'dev': 0.8708, 'reverse': 0.5717, 'style': 0.8105, 'train_dev': 0.9405, 'train_reverse': 0.708}
J2_robust_resid_4_last_k0_l210 {'dev': 0.8591, 'reverse': 0.5134, 'style': 0.8225, 'train_dev': 0.9241, 'train_reverse': 0.623}
J2_robust_sae_w1024_k24_mean_k8_l21 {'dev': 0.8153, 'reverse': 0.6677, 'style': 0.8223, 'train_dev': 0.8758, 'train_reverse': 0.7631}
J2_robust_sae_w1024_k24_mean_k32_l21 {'dev': 0.8264, 'reverse': 0.7331, 'style': 0.8145, 'train_dev': 0.9358, 'train_reverse': 0.8766}
J2_robust_sae_w1024_k24_max_k8_l21 {'dev': 0.8227, 'reverse': 0.6989, 'style': 0.8286, 'train_dev': 0.8709, 'train_reverse': 0.752}
J2_robust_sae_w1024_k24_max_k32_l21 {'dev': 0.8342, 'reverse': 0.7789, 'style': 0.8098, 'train_dev': 0.9312, 'train_reverse': 0.9281}
J2_robust_sae_w1024_k8_mean_k8_l21 {'dev': 0.8512, 'reverse': 0.7359, 'style': 0.8464, 'train_dev': 0.8889, 'train_reverse': 0.7881}
J2_robust_sae_w1024_k8_mean_k32_l21 {'dev': 0.7983, 'reverse': 0.7003, 'style': 0.7778, 'train_dev': 0.9456, 'train_reverse': 0.8848}
J2_robust_sae_w1024_k8_max_k8_l21 {'dev': 0.8661, 'reverse': 0.7848, 'style': 0.8608, 'train_dev': 0.8967, 'train_reverse': 0.8289}
J2_robust_sae_w1024_k8_max_k32_l21 {'dev': 0.8089, 'reverse': 0.7077, 'style': 0.8077, 'train_dev': 0.9411, 'train_reverse': 0.8781}
J2_robust_sae_w256_k24_mean_k8_l21 {'dev': 0.8316, 'reverse': 0.4558, 'style': 0.8378, 'train_dev': 0.8938, 'train_reverse': 0.5355}
J2_robust_sae_w256_k24_mean_k32_l21 {'dev': 0.8181, 'reverse': 0.5423, 'style': 0.8259, 'train_dev': 0.9445, 'train_reverse': 0.7142}
J2_robust_sae_w256_k24_max_k8_l21 {'dev': 0.8269, 'reverse': 0.5741, 'style': 0.8255, 'train_dev': 0.8852, 'train_reverse': 0.6844}
J2_robust_sae_w256_k24_max_k32_l21 {'dev': 0.8152, 'reverse': 0.6483, 'style': 0.8414, 'train_dev': 0.9253, 'train_reverse': 0.823}
J2_robust_sae_w256_k8_mean_k8_l21 {'dev': 0.8378, 'reverse': 0.7619, 'style': 0.8239, 'train_dev': 0.9002, 'train_reverse': 0.8327}
J2_robust_sae_w256_k8_mean_k32_l21 {'dev': 0.7931, 'reverse': 0.7281, 'style': 0.777, 'train_dev': 0.9312, 'train_reverse': 0.8975}
J2_robust_sae_w256_k8_max_k8_l21 {'dev': 0.8639, 'reverse': 0.7736, 'style': 0.8641, 'train_dev': 0.9066, 'train_reverse': 0.8466}
J2_robust_sae_w256_k8_max_k32_l21 {'dev': 0.8238, 'reverse': 0.7298, 'style': 0.8303, 'train_dev': 0.9309, 'train_reverse': 0.8933}
J2_robust_sae_w512_k24_mean_k8_l21 {'dev': 0.8528, 'reverse': 0.702, 'style': 0.8366, 'train_dev': 0.8909, 'train_reverse': 0.7753}
J2_robust_sae_w512_k24_mean_k32_l21 {'dev': 0.8123, 'reverse': 0.6766, 'style': 0.8111, 'train_dev': 0.9352, 'train_reverse': 0.8702}
J2_robust_sae_w512_k24_max_k8_l21 {'dev': 0.8427, 'reverse': 0.7591, 'style': 0.8353, 'train_dev': 0.8831, 'train_reverse': 0.7995}
J2_robust_sae_w512_k24_max_k32_l21 {'dev': 0.803, 'reverse': 0.7262, 'style': 0.8044, 'train_dev': 0.9227, 'train_reverse': 0.8839}
J2_robust_sae_w512_k8_mean_k8_l21 {'dev': 0.8303, 'reverse': 0.7553, 'style': 0.8239, 'train_dev': 0.8967, 'train_reverse': 0.8138}
J2_robust_sae_w512_k8_mean_k32_l21 {'dev': 0.7678, 'reverse': 0.7081, 'style': 0.7498, 'train_dev': 0.9287, 'train_reverse': 0.8811}
J2_robust_sae_w512_k8_max_k8_l21 {'dev': 0.8869, 'reverse': 0.7817, 'style': 0.888, 'train_dev': 0.9206, 'train_reverse': 0.8427}
J2_robust_sae_w512_k8_max_k32_l21 {'dev': 0.8592, 'reverse': 0.7686, 'style': 0.8536, 'train_dev': 0.9627, 'train_reverse': 0.9087}

dense0 {'dev': 0.8491, 'reverse': 0.868, 'style': 0.8414}
dense3 {'dev': 0.8302, 'reverse': 0.8389, 'style': 0.8125}
joint03 {'dev': 0.8631, 'reverse': 0.8784, 'style': 0.8508}
baseline {'dev': 0.8725, 'reverse': 0.5663, 'style': 0.8466}
sae_proposal {'dev': 0.9156, 'reverse': 0.3278, 'style': 0.9142}
ensemble alpha0 0.25 {'dev': 0.8505, 'reverse': 0.8659, 'style': 0.835}
ensemble alpha0 0.5 {'dev': 0.8611, 'reverse': 0.8781, 'style': 0.8483}
ensemble alpha0 0.75 {'dev': 0.8616, 'reverse': 0.8756, 'style': 0.8466}
hostile token count dev 0.9083593487739563
hostile fraction dev 0.8817968964576721

0.001 embedding rmse 0.059387706220149994 pos 0.9638668894767761 negative 0.003977988380938768
{'dev': 0.5603125095367432, '_style': 0.551562488079071, '_reverse_cue': 0.6362500190734863, '_reorder': 0.5603125095367432}
0.01 embedding rmse 0.059894662350416183 pos 0.9593834280967712 negative 0.004471569322049618
{'dev': 0.5609375238418579, '_style': 0.5542187690734863, '_reverse_cue': 0.6384375095367432, '_reorder': 0.5609375238418579}
0.1 embedding rmse 0.06619758158922195 pos 0.9328311085700989 negative 0.007394758984446526
{'dev': 0.546093761920929, '_style': 0.5376562476158142, '_reverse_cue': 0.604687511920929, '_reorder': 0.546093761920929}
1.0 embedding rmse 0.09559045732021332 pos 0.8130456805229187 negative 0.020582148805260658
{'dev': 0.56640625, '_style': 0.553906261920929, '_reverse_cue': 0.5395312309265137, '_reorder': 0.56640625}

position correction max error 2.9802322387695312e-08
 0.8837500214576721
_style 0.8840625286102295
_reverse_cue 0.874218761920929
_reorder 0.8837500214576721

J1_actions 384 192
J2_tone 384 192
J1_actions J1_w256_k24_top8.json 1.0
J1_actions J1_w256_k8_top8.json 1.0
J1_actions J1_w1024_k8_top8.json 1.0
J1_actions J1_w512_k8_top8.json 1.0
J1_actions J1_w512_k24_top8.json 1.0
J1_actions J1_w1024_k24_top8.json 1.0
J1_actions J1_dense_refit.json 0.8197
J1_w1024_k24_single107.json 1.0
J1_w1024_k8_single807.json 1.0
J2_tone J2_robust_ensemble03.json 0.9848
J2_tone J2_w256_k24_top8.json 0.4151
J2_tone J2_w512_k8_top8.json 0.8461
J2_tone J2_w512_k24_top8.json 0.5154
J2_tone J2_w1024_k24_top8.json 0.5069
J2_tone J2_robust_joint03.json 0.9807
J2_tone J2_w1024_k8_top8.json 0.8572
J2_tone J2_lexical_corrected.json 1.0
J2_tone J2_w256_k8_top8.json 0.6677
J2_tone J2_dense_refit.json 0.6858

w1024_k24 {'reconstruction_mean_cv': 0.7803999781608582, 'error_mean_cv': 0.5634999871253967, 'all_latents_mean_cv': 0.7767000198364258, 'all_latents_max_cv': 0.7443000078201294, 'top8_masked_auroc': 0.7114999890327454}
w1024_k8 {'reconstruction_mean_cv': 0.7897999882698059, 'error_mean_cv': 0.6403999924659729, 'all_latents_mean_cv': 0.7031999826431274, 'all_latents_max_cv': 0.6636999845504761, 'top8_masked_auroc': 0.7818999886512756}
w256_k24 {'reconstruction_mean_cv': 0.7782999873161316, 'error_mean_cv': 0.6128000020980835, 'all_latents_mean_cv': 0.7493000030517578, 'all_latents_max_cv': 0.7440000176429749, 'top8_masked_auroc': 0.43035000562667847}
w256_k8 {'reconstruction_mean_cv': 0.7817000150680542, 'error_mean_cv': 0.5710999965667725, 'all_latents_mean_cv': 0.7771999835968018, 'all_latents_max_cv': 0.7835000157356262, 'top8_masked_auroc': 0.7685999870300293}
w512_k24 {'reconstruction_mean_cv': 0.7860000133514404, 'error_mean_cv': 0.5428000092506409, 'all_latents_mean_cv': 0.6621000170707703, 'all_latents_max_cv': 0.6680999994277954, 'top8_masked_auroc': 0.6636499762535095}
w512_k8 {'reconstruction_mean_cv': 0.7807999849319458, 'error_mean_cv': 0.6129000186920166, 'all_latents_mean_cv': 0.7257000207901001, 'all_latents_max_cv': 0.7026000022888184, 'top8_masked_auroc': 0.7452999949455261}
raw 0.7803000211715698 baseline masked 0.8614000082015991

{
  "J1_action_insertion_107": {
    "pairs": 192,
    "mean_change": 6.231166362762451,
    "min_change": 4.717374801635742,
    "positive_change_fraction": 1.0
  },
  "J1_footer_change_107": {
    "mean_absolute_change": 0.0,
    "max_absolute_change": 0.0
  },
  "J2_latent_225": {
    "sae": "w512_k8",
    "mean_kindly_effect": 0.14994613826274872,
    "mean_hostility_effect": 0.37576380372047424
  },
  "J2_latent_334": {
    "sae": "w512_k8",
    "mean_kindly_effect": 0.895887553691864,
    "mean_hostility_effect": 6.726963043212891
  },
  "J2_latent_348": {
    "sae": "w512_k8",
    "mean_kindly_effect": -0.3933888375759125,
    "mean_hostility_effect": -0.29138314723968506
  },
  "J2_latent_382": {
    "sae": "w512_k8",
    "mean_kindly_effect": 0.3697468936443329,
    "mean_hostility_effect": 6.748786926269531
  },
  "J2_latent_412": {
    "sae": "w512_k8",
    "mean_kindly_effect": -2.436189651489258,
    "mean_hostility_effect": -1.2904337644577026
  },
  "J2_latent_415": {
    "sae": "w512_k8",
    "mean_kindly_effect": 10.615894317626953,
    "mean_hostility_effect": 0.3700103759765625
  },
  "J2_latent_455": {
    "sae": "w512_k8",
    "mean_kindly_effect": -0.9111246466636658,
    "mean_hostility_effect": -4.0048346519470215
  },
  "J2_latent_503": {
    "sae": "w512_k8",
    "mean_kindly_effect": -0.749896228313446,
    "mean_hostility_effect": -0.8081105351448059
  },
  "J2_dense_refit": {
    "factorial_auroc": 0.8090888261795044
  },
  "J2_w512_k8_top8": {
    "factorial_auroc": 0.9308268427848816
  },
  "J2_robust_joint03": {
    "factorial_auroc": 0.9819471836090088
  },
  "J2_lexical_corrected": {
    "factorial_auroc": 1.0
  }
}

J2_hostility_latents_334 {'J2': 0.8878124952316284, 'J2_reverse_cue': 0.7342968583106995, 'J2_style': 0.8723437786102295, 'J2_reorder': 0.870312511920929, 'J2_tone': 0.9794921875}
J2_hostility_latents_382 {'J2': 0.8271874785423279, 'J2_reverse_cue': 0.7762500047683716, 'J2_style': 0.8267968893051147, 'J2_reorder': 0.8189062476158142, 'J2_tone': 0.9483371376991272}
J2_hostility_latents_334_382 {'J2': 0.8970312476158142, 'J2_reverse_cue': 0.7771875262260437, 'J2_style': 0.8915625214576721, 'J2_reorder': 0.8860937356948853, 'J2_tone': 0.9945204257965088}

[{'baseline': 0.7712000012397766, 'dense0': 0.828000009059906, 'proposal': 0.7918999791145325, 'difference': 0.02069997787475586}, {'baseline': 0.7630000114440918, 'dense0': 0.7968000173568726, 'proposal': 0.8055999875068665, 'difference': 0.04259997606277466}, {'baseline': 0.7817000150680542, 'dense0': 0.8137000203132629, 'proposal': 0.8061000108718872, 'difference': 0.024399995803833008}, {'baseline': 0.7978000044822693, 'dense0': 0.8009999990463257, 'proposal': 0.802299976348877, 'difference': 0.004499971866607666}, {'baseline': 0.7498999834060669, 'dense0': 0.7896000146865845, 'proposal': 0.7955999970436096, 'difference': 0.045700013637542725}, {'baseline': 0.7791000008583069, 'dense0': 0.8111000061035156, 'proposal': 0.7961000204086304, 'difference': 0.017000019550323486}, {'baseline': 0.7634999752044678, 'dense0': 0.8083999752998352, 'proposal': 0.8033999800682068, 'difference': 0.039900004863739014}, {'baseline': 0.7749000191688538, 'dense0': 0.8025000095367432, 'proposal': 0.7745000123977661, 'difference': -0.0004000067710876465}, {'baseline': 0.7735999822616577, 'dense0': 0.8037999868392944, 'proposal': 0.7870000004768372, 'difference': 0.013400018215179443}, {'baseline': 0.7718999981880188, 'dense0': 0.7961000204086304, 'proposal': 0.8032000064849854, 'difference': 0.03130000829696655}]
{'baseline': 0.7726599991321563, 'dense0': 0.805100005865097, 'proposal': 0.7965699970722199, 'difference': 0.023909997940063477} {'baseline': [0.7498999834060669, 0.7978000044822693], 'dense0': [0.7896000146865845, 0.828000009059906], 'proposal': [0.7745000123977661, 0.8061000108718872], 'difference': [-0.0004000067710876465, 0.045700013637542725]} [-0.0299025, 0.07160249999999996]

J1 shipped dev 1.0 baseline dev 0.9980952143669128 shipped evals [{'file': 'experiments/J1_style.jsonl', 'auroc': 1.0}, {'file': 'experiments/J1_reverse_cue.jsonl', 'auroc': 1.0}, {'file': 'experiments/J1_reorder.jsonl', 'auroc': 1.0}, {'file': 'experiments/J1_actions.jsonl', 'auroc': 1.0}]
J2 shipped dev 0.9975000023841858 baseline dev 0.9671875238418579 shipped evals [{'file': 'experiments/J2_style.jsonl', 'auroc': 0.9912499785423279}, {'file': 'experiments/J2_reverse_cue.jsonl', 'auroc': 0.9993749856948853}, {'file': 'experiments/J2_reorder.jsonl', 'auroc': 0.9790624976158142}, {'file': 'experiments/J2_tone.jsonl', 'auroc': 0.980712890625}, {'file': 'experiments/J2_factorial.jsonl', 'auroc': 0.9819471836090088}]
J3 shipped dev 0.8876000046730042 baseline dev 0.8876000046730042 shipped evals [{'file': 'experiments/J3_style.jsonl', 'auroc': 0.8858000040054321}, {'file': 'experiments/J3_reverse_cue.jsonl', 'auroc': 0.887499988079071}, {'file': 'experiments/J3_reorder.jsonl', 'auroc': 0.8619999885559082}, {'file': 'experiments/J3_urgency_masked.jsonl', 'auroc': 0.8614000082015991}]

J1 max baseline/refit delta 0.0
J2 max baseline/refit delta 0.0
J3 max baseline/refit delta 0.0
urgency correlations [0.9953001737594604, 0.9945818781852722, 0.9993897080421448, 0.9870646595954895, 0.9927451610565186, 0.9965484738349915] count auc 0.6898999810218811
api_batch kindly P(anyhostile) 0.19702970297029704 P(nohostile) 0.18743718592964825
app_q3 kindly P(anyhostile) 0.19860973187686196 P(nohostile) 0.20622177621675866
legacy_form kindly P(anyhostile) 0.2542204568023833 P(nohostile) 0.5418966382338184
partner_feed kindly P(anyhostile) 0.20348837209302326 P(nohostile) 0.2073170731707317
region_eu kindly P(anyhostile) 0.23947895791583165 P(nohostile) 0.2087912087912088
web_q3 kindly P(anyhostile) 0.4995159728944821 P(nohostile) 0.2867310625317743
112
2.5M	/workspace/submission

#!/usr/bin/env python3
"""Evaluate a detector spec on labelled jsonl files.
  python3 /task/tools/eval_detector.py SPEC.json FILE.jsonl [FILE2.jsonl ...]   -> AUROC, balanced accuracy at threshold, n
Set TASK_ROOT if the fixture is not at /task."""
import json, os, sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D

if len(sys.argv) < 3:
    raise SystemExit(__doc__)
ctx = D.Context(os.environ.get("TASK_ROOT", "/task"))
spec = D.load_spec(sys.argv[1])
for path in sys.argv[2:]:
    rows = D.read_jsonl(path)
    s = D.score(ctx, spec, [r["text"] for r in rows]); y = [int(r["label"]) for r in rows]
    ba = D.balanced_accuracy(s, y, spec["threshold"]) if spec.get("threshold") is not None else float("nan")
    print(json.dumps({"file": path, "n": len(rows), "pos": int(sum(y)), "auroc": round(D.auroc(s, y), 4), "balanced_acc": round(ba, 4)}))
J1_actions.jsonl
J1_baseline.json
J1_dense_refit.json
J1_reorder.jsonl
J1_reverse_cue.jsonl
J1_style.jsonl
J1_w1024_k24_single107.json
J1_w1024_k24_top8.json
J1_w1024_k8_single807.json
J1_w1024_k8_top8.json
J1_w256_k24_top8.json
J1_w256_k8_top8.json
J1_w512_k24_top8.json
J1_w512_k8_top8.json
J2_augmentation_train.jsonl
J2_baseline.json
J2_dense_refit.json
J2_factorial.jsonl
J2_grouped_comparison.json
J2_hostility_latents_334.json
J2_hostility_latents_334_382.json
J2_hostility_latents_382.json
J2_lexical_corrected.json
J2_lexical_prior_0.001.json
J2_lexical_prior_0.01.json
J2_lexical_prior_0.1.json
J2_lexical_prior_1.0.json
J2_lexical_prior_results.json
J2_pruned_results.json
J2_reorder.jsonl
J2_reverse_cue.jsonl
J2_robust_cv_results.json
J2_robust_ensemble03.json
J2_robust_joint03.json
J2_robust_resid_0_last_k0_l21.json
J2_robust_resid_0_last_k0_l210.json
J2_robust_resid_0_mean_k0_l21.json
J2_robust_resid_0_mean_k0_l210.json
J2_robust_resid_1_last_k0_l21.json
J2_robust_resid_1_last_k0_l210.json
J2_robust_resid_1_mean_k0_l21.json
J2_robust_resid_1_mean_k0_l210.json
J2_robust_resid_2_last_k0_l21.json
J2_robust_resid_2_last_k0_l210.json
J2_robust_resid_2_mean_k0_l21.json
J2_robust_resid_2_mean_k0_l210.json
J2_robust_resid_3_last_k0_l21.json
J2_robust_resid_3_last_k0_l210.json
J2_robust_resid_3_mean_k0_l21.json
J2_robust_resid_3_mean_k0_l210.json
J2_robust_resid_4_last_k0_l21.json
J2_robust_resid_4_last_k0_l210.json
J2_robust_resid_4_mean_k0_l21.json
J2_robust_resid_4_mean_k0_l210.json
J2_robust_sae_w1024_k24_max_k32_l21.json
J2_robust_sae_w1024_k24_max_k8_l21.json
J2_robust_sae_w1024_k24_mean_k32_l21.json
J2_robust_sae_w1024_k24_mean_k8_l21.json
J2_robust_sae_w1024_k8_max_k32_l21.json
J2_robust_sae_w1024_k8_max_k8_l21.json
J2_robust_sae_w1024_k8_mean_k32_l21.json
J2_robust_sae_w1024_k8_mean_k8_l21.json
J2_robust_sae_w256_k24_max_k32_l21.json
J2_robust_sae_w256_k24_max_k8_l21.json
J2_robust_sae_w256_k24_mean_k32_l21.json
J2_robust_sae_w256_k24_mean_k8_l21.json
J2_robust_sae_w256_k8_max_k32_l21.json
J2_robust_sae_w256_k8_max_k8_l21.json
J2_robust_sae_w256_k8_mean_k32_l21.json
J2_robust_sae_w256_k8_mean_k8_l21.json
J2_robust_sae_w512_k24_max_k32_l21.json
J2_robust_sae_w512_k24_max_k8_l21.json
J2_robust_sae_w512_k24_mean_k32_l21.json
J2_robust_sae_w512_k24_mean_k8_l21.json
J2_robust_sae_w512_k8_max_k32_l21.json
J2_robust_sae_w512_k8_max_k8_l21.json
J2_robust_sae_w512_k8_mean_k32_l21.json
J2_robust_sae_w512_k8_mean_k8_l21.json
J2_style.jsonl
J2_tone.jsonl
J2_w1024_k24_top8.json
J2_w1024_k8_top8.json
J2_w256_k24_top8.json
J2_w256_k8_top8.json
J2_w512_k24_top8.json
J2_w512_k8_top8.json
J3_baseline.json
J3_cv_uncertainty.json
J3_dense_refit.json
J3_reorder.jsonl
J3_representation_audit.json
J3_reverse_cue.jsonl
J3_style.jsonl
J3_urgency_masked.jsonl
J3_w1024_k24_top8.json
J3_w1024_k8_top8.json
J3_w256_k24_top8.json
J3_w256_k8_top8.json
J3_w512_k24_top8.json
J3_w512_k8_top8.json
challenge_provenance.json
challenge_results.json
cv_results.json
evaluator_check.json
final_evaluations.json
intervention_results.json
shift_results_initial.json
traffic_audit.json

{
  "J1": {
    "api_batch": {
      "shipped": 1.0,
      "baseline": 0.8740234375,
      "proposed": 1.0
    },
    "app_q3": {
      "shipped": 1.0,
      "baseline": 0.9091796875,
      "proposed": 1.0
    },
    "legacy_form": {
      "shipped": 1.0,
      "baseline": 0.744140625,
      "proposed": 1.0
    },
    "partner_feed": {
      "shipped": 1.0,
      "baseline": 0.7685546875,
      "proposed": 1.0
    },
    "region_eu": {
      "shipped": 1.0,
      "baseline": 0.826171875,
      "proposed": 1.0
    },
    "web_q3": {
      "shipped": 1.0,
      "baseline": 0.845703125,
      "proposed": 1.0
    }
  },
  "J2": {
    "api_batch": {
      "shipped": 0.9521484375,
      "baseline": 0.654296875,
      "proposed": 0.7841796875
    },
    "app_q3": {
      "shipped": 0.994140625,
      "baseline": 0.7275390625,
      "proposed": 0.9033203125
    },
    "legacy_form": {
      "shipped": 0.9990234375,
      "baseline": 0.712890625,
      "proposed": 0.8388671875
    },
    "partner_feed": {
      "shipped": 0.99609375,
      "baseline": 0.7021484375,
      "proposed": 0.8349609375
    },
    "region_eu": {
      "shipped": 0.982421875,
      "baseline": 0.71484375,
      "proposed": 0.875
    },
    "web_q3": {
      "shipped": 0.9833984375,
      "baseline": 0.6240234375,
      "proposed": 0.8408203125
    }
  }
}
PASS 176 metric checks

{'J1': {'adopt_sae': True, 'detector_kind': 'sae', 'dev_auroc': 1.0, 'baseline_dev_auroc': 0.9980952143669128}, 'J2': {'adopt_sae': False, 'detector_kind': 'dense', 'dev_auroc': 0.9975000023841858, 'baseline_dev_auroc': 0.9671875238418579}, 'J3': {'adopt_sae': False, 'detector_kind': 'dense', 'dev_auroc': 0.8876000046730042, 'baseline_dev_auroc': 0.8876000046730042}}
experiments 35

J1 exact source/dev overlap 0
J2 exact source/dev overlap 0
required sections:
## Findings True
## Decision True
## How I computed this True