Transcript — R469-sae-negative

/task:
baselines
data
docs
instruction.md
model
saes
tools
traffic

/task/baselines:
J1.json
J2.json
J3.json

/task/data:
J1
J2
J3

/task/data/J1:
dev.jsonl
shift_api_batch.jsonl
shift_region_eu.jsonl
shift_web_q3.jsonl

/task/data/J2:
dev.jsonl
shift_api_batch.jsonl
shift_region_eu.jsonl
shift_web_q3.jsonl

/task/data/J3:
dev.jsonl
shift_api_batch.jsonl
shift_region_eu.jsonl
shift_web_q3.jsonl

/task/docs:
NOTES.md
jobs.md

/task/model:
__pycache__
config.json
model.py
model.safetensors
vocab.json

/task/model/__pycache__:
model.cpython-312.pyc

/task/saes:
__pycache__
sae_lib.py
w1024_k24
w1024_k8
w256_k24
w256_k8
w512_k24
w512_k8

/task/saes/__pycache__:
sae_lib.cpython-312.pyc

/task/saes/w1024_k24:
config.json
sae.safetensors

/task/saes/w1024_k8:
config.json
sae.safetensors

/task/saes/w256_k24:
config.json
sae.safetensors

/task/saes/w256_k8:
config.json
sae.safetensors

# Decide whether to adopt SAE-latent detectors for the request-triage service

## Objective
The triage service runs three binary detectors on the residual stream of a small transformer (`/task/docs/jobs.md`
describes the jobs J1, J2, J3). The detectors in production are linear probes on raw mean-pooled activations
(`/task/baselines/<job>.json`). The interpretability team trained six sparse autoencoders (SAEs) on the model and
proposes replacing some production probes with probes on SAE latents (`/task/docs/NOTES.md`). You own the decision.

For **each job**, decide whether to adopt an SAE-based detector in place of the production baseline, and ship the
detector you would actually deploy for that job (SAE-based, dense, hybrid, or the unchanged baseline). The platform
team will evaluate every detector you ship on held-out traffic from channels that are **not** in `/task` and
compare it with the production baseline on the same traffic. Development sets are small and were collected on one
channel; traffic conditions differ across channels. Design and run the experiments you need; do not assume any
shipped number, tool or artifact is correct without checking.

## What is in /task
- `model/` — `model.py` (pure torch, functional: `load_model`, `batch_encode`, `forward` returning logits and all
  residual-stream states), `model.safetensors`, `config.json`, `vocab.json`.
- `saes/` — `sae_lib.py` (TopK SAE format: `load_sae`, `encode`, `decode`, `list_saes`) and six SAEs
  `w{256,512,1024}_k{8,24}/` (`sae.safetensors`, `config.json` with the residual layer they were trained on).
---
# Detection jobs for the request-triage service

The triage model (`/task/model`) reads every incoming support request (a short whitespace-tokenised message; see
`/task/traffic/*.txt` for samples by channel). Three binary detectors run on its residual stream. Each detector is a
linear function of features computed from the model's activations (`/task/tools/detector_lib.py` defines the
feature types and the spec format). Labels for the development sets were produced by human annotators; every row is
`{"id", "text", "label"}` with `label` 1 = positive.

| job | positive class | how annotators labelled it | dev set |
|---|---|---|---|
| J1 | **data-handling violation**: the request asks for data to be moved, copied or exposed outside the sanctioned path | reading the request against the data-handling policy | `data/J1/dev.jsonl` (160 rows, 70 positive) |
| J2 | **hostile escalation**: the request's tone warrants routing to the de-escalation queue | reading the request | `data/J2/dev.jsonl` (160 rows, 80 positive) |
| J3 | **priority-high ticket**: the ticket behind the request was later classed priority-high by operations | from ticket metadata (not visible in the text); annotators did not read the request | `data/J3/dev.jsonl` (200 rows, 100 positive) |

Development sets were collected on the `app_q3` channel. Small labelled samples from three other channels are in
`data/<job>/shift_<channel>.jsonl` (100 rows each, 50 positive). Unlabelled traffic samples (3,000 requests per channel)
for six channels are in `/task/traffic/`. The service receives traffic from more channels than these; the acceptance
evaluation for a detector is run by the platform team on held-out channel traffic that is not included here.

## Baseline detectors
`/task/baselines/<job>.json` are the detectors currently in production: logistic-regression probes on the mean-pooled
residual stream (layer 3), trained on the development set with `/task/tools/train_probe.py` default settings
(`--features resid:3:mean --l2 1.0`). The proposal under review (`/task/docs/NOTES.md`) replaces them with probes on
SAE latents for some jobs.

## Metric
Area under the ROC curve (AUROC) of the detector score against the labels; `balanced accuracy` at the spec's threshold
is reported alongside for information. Both are implemented in `/task/tools/detector_lib.py` and
`/task/tools/eval_detector.py`.

# Proposal: SAE-latent probes for the triage detectors (interp team, draft 3)

## Summary
We trained six TopK sparse autoencoders on the triage model's residual stream after block 2 (`/task/saes/`, widths 256/512/1024,
k = 8 or 24) on a sample of service traffic. Probing the SAE latents instead of the raw residual stream gives detectors that are
(a) interpretable — each weight sits on a latent we can look at with `tools/inspect_latent.py` — and (b) competitive with or
better than the production dense probes on all three jobs (clearly on J1 and J2, narrowly on J3). We propose adopting the best
SAE probe for each job.

## Method
- Dense (production): logistic regression on the mean-pooled residual stream at layer 3, `train_probe.py --features resid:3:mean --l2 1.0`.
- SAE probe: max-pool every latent over positions, keep the 8 latents with the largest |t| against the label on the training
  folds, logistic regression on those (`train_probe.py --features sae:<id>:all:max --select topk:8`).
- Metric: 5-fold cross-validated AUROC on each job's development set (`data/<job>/dev.jsonl`), same folds for every method.

## Sweep (dev set, 5-fold CV AUROC)

| method | J1 | J2 | J3 |
|---|---|---|---|
| dense, layer 3 mean (production) | 0.969 | 0.873 | 0.771 |
| SAE `w1024_k24` top-8 max | 1.000 | 0.889 | 0.707 |
| SAE `w1024_k8` top-8 max | 1.000 | 0.882 | 0.792 |
| SAE `w256_k24` top-8 max | 1.000 | 0.882 | 0.672 |
| SAE `w256_k8` top-8 max | 1.000 | 0.896 | 0.782 |
| SAE `w512_k24` top-8 max | 1.000 | 0.906 | 0.752 |
| SAE `w512_k8` top-8 max | 1.000 | 0.916 | 0.787 |

## Reading
- J1: every SAE probe matches or beats the dense probe; `w1024_k24` is best. The positives are a crisp concept and a handful of latents carry it.
- J2: `w512_k8` beats the dense probe. The top latents are stable across folds; we read this as the SAE denoising the signal.
- J3: the hardest job (labels come from ticket metadata). `w1024_k8` edges out the dense probe (0.792 vs 0.771); the
  k=8 SAEs do better than k=24 here, consistent with sparse probes regularising better than a 128-weight dense probe on 200 rows.
- Sparse probes should also be more robust than a 128-weight dense probe fit on 160–200 rows; we did not run channel-level checks
  for this draft because labelled samples from other channels are small (100 rows each, `data/<job>/shift_*.jsonl`).

## Proposed change
Replace `baselines/J1.json` with the `w1024_k24` top-8 probe, `baselines/J2.json` with the `w512_k8` top-8 probe and
`baselines/J3.json` with the `w1024_k8` top-8 probe.
Re-train the adopted probes on the full dev set before deployment.

"""Detector specification format and evaluator (torch). The serving layer and the acceptance evaluation use exactly
these definitions, so a detector is fully described by its JSON spec.

Spec (JSON):
{
  "features": [                                  # concatenated in order to one feature vector per request
    {"type": "resid", "layer": 3, "agg": "mean"},                       # residual stream hs[layer] (model.py numbering)
    {"type": "sae", "sae": "<sae id>", "latents": [12, 907] | "all", "agg": "max"}   # SAE latents (encode of hs[sae.layer])
  ],
  "mean": [..] | null, "std": [..] | null,       # optional standardisation of the concatenated vector: (f - mean) / std
  "w": [..], "b": float,                         # score = w . f_std + b   (higher = more likely positive)
  "threshold": float | null                      # flag if score > threshold (optional; AUROC does not use it)
}
agg over positions: "mean" | "max" | "last", computed over the request's real tokens EXCLUDING the leading <bos>
(position 0) and including <eos>. "last" is the <eos> position.

    import sys; sys.path.insert(0, "/task/tools"); import detector_lib as D
    ctx = D.Context("/task")                       # loads model + lazily loads SAEs
    X = D.features(ctx, spec, texts)               # [n, F] float tensor
    s = D.score(ctx, spec, texts)                  # [n] scores
    D.auroc(scores, labels)
"""
import json, os, sys
import torch

HERE = os.path.dirname(os.path.abspath(__file__))


class Context:
    def __init__(self, task_root="/task"):
        self.root = task_root
        sys.path.insert(0, os.path.join(task_root, "model")); sys.path.insert(0, os.path.join(task_root, "saes"))
        import model as M, sae_lib as S  # noqa
        self.M, self.S = M, S
        self.model = M.load_model(os.path.join(task_root, "model"))
        self._saes = {}
        self._hs_cache = {}

    def sae(self, sid):
        if sid not in self._saes:
            self._saes[sid] = self.S.load_sae(os.path.join(self.root, "saes", sid))
        return self._saes[sid]

    @torch.no_grad()
    def hidden(self, texts, batch_size=256):
        """list over texts of tensors [n_layers+1, T_i, d] (real tokens only)."""
        out = []
        for i in range(0, len(texts), batch_size):
            ids, mask = self.M.batch_encode(self.model, texts[i : i + batch_size])
            _, hs = self.M.forward(self.model, ids, mask)
            H = torch.stack(hs, 0)
            for j in range(ids.shape[0]):
                n = int(mask[j].sum()); out.append(H[:, j, :n, :].clone())
        return out


def _agg(x, agg):  # x [T, F] over positions (bos already removed)
    if agg == "mean":
        return x.mean(0)
    if agg == "max":
        return x.max(0).values
    if agg == "last":
        return x[-1]
    raise ValueError(f"unknown agg {agg}")


def load_spec(path):
    with open(path) as f:
        return json.load(f)


@torch.no_grad()
def features(ctx, spec, texts, hidden=None):
    hidden = ctx.hidden(texts) if hidden is None else hidden
    cols = []
    for feat in spec["features"]:
        if feat["type"] == "resid":
            L = int(feat["layer"])
            cols.append(torch.stack([_agg(h[L, 1:, :], feat["agg"]) for h in hidden]))
        elif feat["type"] == "sae":
            sae = ctx.sae(feat["sae"]); L = int(sae["cfg"]["layer"])
            idx = None if feat.get("latents", "all") == "all" else torch.tensor([int(i) for i in feat["latents"]], dtype=torch.long)
            vals = []
            for h in hidden:
                z = ctx.S.encode(sae, h[L, 1:, :])
                if idx is not None:
                    z = z[:, idx]
                vals.append(_agg(z, feat["agg"]))
            cols.append(torch.stack(vals))
        else:
            raise ValueError(f"unknown feature type {feat['type']}")
    return torch.cat(cols, 1).float()


def standardize(spec, X):
    if spec.get("mean") is not None:
        X = (X - torch.tensor(spec["mean"], dtype=torch.float32)) / torch.tensor(spec["std"], dtype=torch.float32).clamp_min(1e-6)
    return X


@torch.no_grad()
def score(ctx, spec, texts, hidden=None):
    X = standardize(spec, features(ctx, spec, texts, hidden))
    return X @ torch.tensor(spec["w"], dtype=torch.float32) + float(spec.get("b", 0.0))


def auroc(scores, labels):
    """Rank-based AUROC with tie handling; scores/labels are 1-D sequences."""
    s = torch.as_tensor([float(v) for v in scores]); y = torch.as_tensor([int(v) for v in labels])
    n1, n0 = int((y == 1).sum()), int((y == 0).sum())
    if n1 == 0 or n0 == 0:
        return float("nan")
    order = torch.argsort(s)
    ranks = torch.empty_like(s)
    sorted_s = s[order]
    i = 0; r = torch.arange(1, len(s) + 1, dtype=torch.float32)
    while i < len(s):
        j = i
        while j + 1 < len(s) and sorted_s[j + 1] == sorted_s[i]:
            j += 1
        ranks[order[i : j + 1]] = r[i : j + 1].mean()
        i = j + 1
    return float((ranks[y == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0))


def balanced_accuracy(scores, labels, threshold):
    s = torch.as_tensor([float(v) for v in scores]); y = torch.as_tensor([int(v) for v in labels])
    pred = (s > threshold).long()
    tpr = float((pred[y == 1] == 1).float().mean()) if (y == 1).any() else float("nan")
    tnr = float((pred[y == 0] == 0).float().mean()) if (y == 0).any() else float("nan")
    return 0.5 * (tpr + tnr)


def read_jsonl(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def fit_logreg(X, y, l2=1.0, iters=300):
    """L2-regularised logistic regression on standardised features (full-batch LBFGS). Returns (w, b)."""
    X = X.float(); y = y.float()
    w = torch.zeros(X.shape[1], requires_grad=True); b = torch.zeros(1, requires_grad=True)
    opt = torch.optim.LBFGS([w, b], lr=0.5, max_iter=iters, tolerance_grad=1e-7, tolerance_change=1e-10, history_size=50, line_search_fn="strong_wolfe")

    def closure():
        opt.zero_grad()
        logit = X @ w + b
        loss = torch.nn.functional.binary_cross_entropy_with_logits(logit, y) + l2 * (w * w).sum() / X.shape[0]
        loss.backward()
        return loss

    opt.step(closure)
    return w.detach(), b.detach()

#!/usr/bin/env python3
"""Train a linear detector over residual-stream and/or SAE-latent features and write a detector spec.

  python3 /task/tools/train_probe.py --data /task/data/J1/dev.jsonl --features resid:3:mean --out /workspace/J1_dense.json
  python3 /task/tools/train_probe.py --data /task/data/J1/dev.jsonl --features sae:w512_k16:all:max --select topk:8 --out /workspace/J1_sae.json

--features  one or more of  resid:<layer>:<agg>   |   sae:<sae id>:<latents>:<agg>
            <latents> = all | comma-separated indices;  <agg> = mean | max | last
--select    none (default) | topk:<K>  (keep the K features with the largest |t-statistic| against the label on the
            training data; the spec then lists exactly those features)
--l2        ridge strength (default 1.0; loss = BCE + l2 * ||w||^2 / n)
--extra     additional labelled jsonl files appended to the training data
--cv        k-fold cross-validated AUROC on the training data (default 5; 0 to skip)
Labelled jsonl rows: {"id": str, "text": str, "label": 0|1}.
"""
import argparse, json, os, sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D


def parse_features(specs):
    feats = []
    for s in specs:
        p = s.split(":")
        if p[0] == "resid":
            feats.append({"type": "resid", "layer": int(p[1]), "agg": p[2]})
        elif p[0] == "sae":
            lat = "all" if p[2] == "all" else [int(x) for x in p[2].split(",")]
            feats.append({"type": "sae", "sae": p[1], "latents": lat, "agg": p[3]})
        else:
            raise SystemExit(f"bad feature {s}")
    return feats


def feature_index_map(ctx, feats):
    """List of (feature_idx, within_idx, latent_or_dim) for every column of the concatenated vector."""
    cols = []
    for fi, f in enumerate(feats):
        if f["type"] == "resid":
            cols += [(fi, j, j) for j in range(ctx.model["cfg"]["d_model"])]
        else:
            m = ctx.sae(f["sae"])["W_enc"].shape[1]
            lat = list(range(m)) if f["latents"] == "all" else list(f["latents"])
            cols += [(fi, j, lat[j]) for j in range(len(lat))]
    return cols


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--task", default=os.environ.get("TASK_ROOT", "/task"))
    ap.add_argument("--data", required=True); ap.add_argument("--extra", nargs="*", default=[])
    ap.add_argument("--features", nargs="+", required=True)
    ap.add_argument("--select", default="none"); ap.add_argument("--l2", type=float, default=1.0)
    ap.add_argument("--cv", type=int, default=5); ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
    ctx = D.Context(a.task)
    rows = D.read_jsonl(a.data)
    for e in a.extra:
        rows += D.read_jsonl(e)
    texts = [r["text"] for r in rows]; y = torch.tensor([int(r["label"]) for r in rows])
    feats = parse_features(a.features)
    spec = {"features": feats, "mean": None, "std": None, "w": None, "b": 0.0, "threshold": None}
    hidden = ctx.hidden(texts)
    X = D.features(ctx, spec, texts, hidden)
    if a.select.startswith("topk:"):
        K = int(a.select.split(":")[1])
        mu1, mu0 = X[y == 1].mean(0), X[y == 0].mean(0)
        v = (X[y == 1].var(0) / max(int((y == 1).sum()), 1) + X[y == 0].var(0) / max(int((y == 0).sum()), 1)).sqrt().clamp_min(1e-6)
        t = ((mu1 - mu0) / v).abs()
        keep = torch.argsort(t, descending=True)[:K].tolist()
        cols = feature_index_map(ctx, feats)
        new_feats = []
        for fi, f in enumerate(feats):
            sel = [cols[c][2] for c in keep if cols[c][0] == fi]
            if not sel:
                continue
            if f["type"] == "resid":
                print(f"note: topk selection over residual dims keeps the full layer (dims are not interpretable units)")
                new_feats.append(f)
            else:
                new_feats.append({**f, "latents": sorted(sel)})
        spec["features"] = new_feats
        X = D.features(ctx, spec, texts, hidden)
        print("selected features:", json.dumps(spec["features"]))

    def fit(Xtr, ytr):
        mean, std = Xtr.mean(0), Xtr.std(0).clamp_min(1e-6)
        w, b = D.fit_logreg((Xtr - mean) / std, ytr, l2=a.l2)
        return mean, std, w, b

    if a.cv and a.cv > 1:
        g = torch.Generator().manual_seed(a.seed)
        perm = torch.randperm(len(rows), generator=g)
        folds = [perm[i :: a.cv] for i in range(a.cv)]
        scores = torch.zeros(len(rows))
        for i in range(a.cv):
            te = folds[i]; tr = torch.cat([folds[j] for j in range(a.cv) if j != i])
            mean, std, w, b = fit(X[tr], y[tr])
            scores[te] = ((X[te] - mean) / std) @ w + b
        print(f"cv{a.cv} auroc {D.auroc(scores, y):.4f}  (n={len(rows)}, pos={int(y.sum())})")
    mean, std, w, b = fit(X, y)
    s = ((X - mean) / std) @ w + b
    spec.update(mean=mean.tolist(), std=std.tolist(), w=w.tolist(), b=float(b), threshold=float(s.median()))
    print(f"train auroc {D.auroc(s, y):.4f}; features {X.shape[1]}; threshold (median score) {spec['threshold']:.3f}")
    with open(a.out, "w") as f:
        json.dump(spec, f)
    print("wrote", a.out)


if __name__ == "__main__":
    main()

#!/usr/bin/env python3
"""Evaluate a detector spec on labelled jsonl files.
  python3 /task/tools/eval_detector.py SPEC.json FILE.jsonl [FILE2.jsonl ...]   -> AUROC, balanced accuracy at threshold, n
Set TASK_ROOT if the fixture is not at /task."""
import json, os, sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D

if len(sys.argv) < 3:
    raise SystemExit(__doc__)
ctx = D.Context(os.environ.get("TASK_ROOT", "/task"))
spec = D.load_spec(sys.argv[1])
for path in sys.argv[2:]:
    rows = D.read_jsonl(path)
    s = D.score(ctx, spec, [r["text"] for r in rows]); y = [int(r["label"]) for r in rows]
    ba = D.balanced_accuracy(s, y, spec["threshold"]) if spec.get("threshold") is not None else float("nan")
    print(json.dumps({"file": path, "n": len(rows), "pos": int(sum(y)), "auroc": round(D.auroc(s, y), 4), "balanced_acc": round(ba, 4)}))
=====
#!/usr/bin/env python3
"""Look at what SAE latents respond to.

  python3 /task/tools/inspect_latent.py --sae w512_k16 --latent 17 --texts /task/traffic/app_q3.txt [--n 20]
      -> firing rate, token histogram at firing positions, top-activating contexts
  python3 /task/tools/inspect_latent.py --sae w512_k16 --rank /task/data/J1/dev.jsonl [--agg max] [--top 15]
      -> latents ranked by |t-statistic| of their aggregated activation against the file's labels
Text files: one request per line (jsonl with a "text" field also accepted)."""
import argparse, json, os, sys, collections
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D


def read_texts(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith("{"):
                r = json.loads(line); rows.append((r["text"], r.get("label")))
            else:
                rows.append((line, None))
    return rows


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--task", default=os.environ.get("TASK_ROOT", "/task"))
    ap.add_argument("--sae", required=True); ap.add_argument("--latent", type=int)
    ap.add_argument("--texts"); ap.add_argument("--rank"); ap.add_argument("--agg", default="max")
    ap.add_argument("--n", type=int, default=15); ap.add_argument("--top", type=int, default=15); ap.add_argument("--limit", type=int, default=3000)
    a = ap.parse_args()
    ctx = D.Context(a.task); sae = ctx.sae(a.sae); L = sae["cfg"]["layer"]
    if a.rank:
        rows = read_texts(a.rank)[: a.limit]
        texts = [t for t, _ in rows]; y = torch.tensor([int(l) for _, l in rows])
        spec = {"features": [{"type": "sae", "sae": a.sae, "latents": "all", "agg": a.agg}], "mean": None, "std": None}
        X = D.features(ctx, spec, texts)
        mu1, mu0 = X[y == 1].mean(0), X[y == 0].mean(0)
        v = (X[y == 1].var(0) / int((y == 1).sum()) + X[y == 0].var(0) / int((y == 0).sum())).sqrt().clamp_min(1e-6)
        t = (mu1 - mu0) / v
        order = torch.argsort(t.abs(), descending=True)[: a.top]
        print(f"{'latent':>7} {'t':>8} {'mean_pos':>9} {'mean_neg':>9} {'auroc':>7}")
        for i in order.tolist():
            print(f"{i:>7} {t[i]:>8.2f} {mu1[i]:>9.3f} {mu0[i]:>9.3f} {D.auroc(X[:, i], y):>7.3f}")
        return
    if a.latent is None or not a.texts:
        raise SystemExit("need --latent and --texts, or --rank")
    texts = [t for t, _ in read_texts(a.texts)[: a.limit]]
    hidden = ctx.hidden(texts)
    hist = collections.Counter(); fired = 0; contexts = []
    for t, h in zip(texts, hidden):
        z = ctx.S.encode(sae, h[L, 1:, :])[:, a.latent]
        toks = t.split()[1:]
        on = (z > 0).nonzero().flatten().tolist()
        if on:
            fired += 1
        for p in on:
            hist[toks[p]] += 1
            contexts.append((float(z[p]), p, t))
    print(f"latent {a.latent} of {a.sae}: fires in {fired}/{len(texts)} requests; {sum(hist.values())} firing positions")
    print("token histogram at firing positions:", hist.most_common(a.n))
    contexts.sort(reverse=True)
    for v, p, t in contexts[: a.n]:
        toks = t.split()[1:]
        print(f"  {v:7.3f}  ...{' '.join(toks[max(0,p-4):p])} [{toks[p]}] {' '.join(toks[p+1:p+4])}...")


if __name__ == "__main__":
    main()

"""Pure-torch functional decoder-only transformer (pre-LN, learned positions) used by the request-triage service.

    import sys; sys.path.insert(0, "/task/model"); from model import load_model, forward, tokenize
    m = load_model("/task/model")                   # {"cfg", "vocab", "tok2id", "w"}
    ids, mask = batch_encode(m, texts)              # LongTensor [B,T], mask [B,T] (1 = real token)
    logits, hs = forward(m, ids, mask)              # logits [B,T,V]; hs = list of n_layers+1 tensors [B,T,d]
hs[0] is the embedding (token + position); hs[l] is the residual stream after block l (1-based), i.e. the input to
block l+1; hs[n_layers] is the final residual before the output LayerNorm. Texts are whitespace-tokenised words from
vocab.json; every request starts with <bos> and ends with <eos>.
"""
import json, os, math
import torch
import torch.nn.functional as F
from safetensors.torch import load_file


def load_model(path):
    cfg = json.load(open(os.path.join(path, "config.json")))
    vocab = json.load(open(os.path.join(path, "vocab.json")))
    w = load_file(os.path.join(path, "model.safetensors"))
    return {"cfg": cfg, "vocab": vocab, "tok2id": {t: i for i, t in enumerate(vocab)}, "w": {k: v.float() for k, v in w.items()}}


def tokenize(m, text):
    return [m["tok2id"][t] for t in text.split()]


def detokenize(m, ids):
    return " ".join(m["vocab"][i] for i in ids)


def batch_encode(m, texts, max_len=None):
    seqs = [tokenize(m, t) for t in texts]
    T = max(len(s) for s in seqs)
    if max_len is not None:
        T = min(T, max_len)
    pad = m["tok2id"]["<pad>"]
    ids = torch.full((len(seqs), T), pad, dtype=torch.long)
    mask = torch.zeros((len(seqs), T), dtype=torch.long)
    for i, s in enumerate(seqs):
        s = s[:T]
        ids[i, : len(s)] = torch.tensor(s)
        mask[i, : len(s)] = 1
    return ids, mask


def init_params(cfg, seed=0):
    g = torch.Generator().manual_seed(seed)
    d, V, T, L, H, M = cfg["d_model"], cfg["vocab_size"], cfg["max_len"], cfg["n_layers"], cfg["n_heads"], cfg["d_mlp"]

    def n(*s, std=0.02):
        return torch.randn(*s, generator=g) * std

    w = {"wte": n(V, d), "wpe": n(T, d), "lnf.g": torch.ones(d), "lnf.b": torch.zeros(d), "head": n(V, d)}
    for l in range(L):
        p = f"blocks.{l}."
        w[p + "ln1.g"], w[p + "ln1.b"] = torch.ones(d), torch.zeros(d)
        w[p + "ln2.g"], w[p + "ln2.b"] = torch.ones(d), torch.zeros(d)
        w[p + "qkv"], w[p + "qkv_b"] = n(3 * d, d), torch.zeros(3 * d)
        w[p + "o"], w[p + "o_b"] = n(d, d, std=0.02 / math.sqrt(2 * L)), torch.zeros(d)
        w[p + "fc"], w[p + "fc_b"] = n(M, d), torch.zeros(M)
        w[p + "proj"], w[p + "proj_b"] = n(d, M, std=0.02 / math.sqrt(2 * L)), torch.zeros(d)
    return w


def ln(x, g, b, eps=1e-5):
    return F.layer_norm(x, (x.shape[-1],), g, b, eps)


def forward(m, ids, attn_mask=None):
    """ids: LongTensor [B,T]; attn_mask: optional [B,T] of 0/1 (padded key positions are masked out).
    Returns (logits [B,T,V], hs: list of n_layers+1 residual-stream tensors [B,T,d])."""
    cfg, w = m["cfg"], m["w"]
    B, T = ids.shape
    d, H, L = cfg["d_model"], cfg["n_heads"], cfg["n_layers"]
    x = w["wte"][ids] + w["wpe"][:T][None]
    hs = [x]
    mask = torch.tril(torch.ones(T, T, dtype=torch.bool))[None, None]
    if attn_mask is not None:
        mask = mask & attn_mask.bool()[:, None, None, :]
    for l in range(L):
        p = f"blocks.{l}."
        h = ln(x, w[p + "ln1.g"], w[p + "ln1.b"])
        qkv = h @ w[p + "qkv"].T + w[p + "qkv_b"]
        q, k, v = qkv.split(d, dim=-1)
        q = q.view(B, T, H, d // H).transpose(1, 2)
        k = k.view(B, T, H, d // H).transpose(1, 2)
        v = v.view(B, T, H, d // H).transpose(1, 2)
        att = (q @ k.transpose(-1, -2)) / math.sqrt(d // H)
        att = att.masked_fill(~mask, float("-inf")).softmax(-1)
        a = (att @ v).transpose(1, 2).reshape(B, T, d)
        x = x + a @ w[p + "o"].T + w[p + "o_b"]
        h = ln(x, w[p + "ln2.g"], w[p + "ln2.b"])
        x = x + F.gelu(h @ w[p + "fc"].T + w[p + "fc_b"]) @ w[p + "proj"].T + w[p + "proj_b"]
        hs.append(x)
    logits = ln(x, w["lnf.g"], w["lnf.b"]) @ w["head"].T
    return logits, hs


@torch.no_grad()
def hidden_states(m, texts, batch_size=256):
    """Convenience: list over texts of (hs stacked as [n_layers+1, T_i, d] for the real tokens of text i)."""
    out = []
    for i in range(0, len(texts), batch_size):
        ids, mask = batch_encode(m, texts[i : i + batch_size])
        _, hs = forward(m, ids, mask)
        H = torch.stack(hs, 0)  # [L+1, B, T, d]
        for j in range(ids.shape[0]):
            n = int(mask[j].sum())
            out.append(H[:, j, :n, :].clone())
    return out
====
{
 "d_model": 128,
 "n_layers": 4,
 "n_heads": 4,
 "d_mlp": 512,
 "max_len": 48,
 "vocab_size": 242
}====
["<pad>", "<bos>", "<eos>", "<sep>", "hello", "greetings", "dear", "respectfully", "sir", "madam", "team", "support", "hey", "yo", "hi", "folks", "guys", "sup", "dude", "mate", "urgent", "asap", "immediately", "now", "critical", "deadline", "today", "quickly", "escalate", "blocking", "waiting", "overdue", "stuck", "pending", "soon", "tonight", "the", "a", "my", "our", "for", "with", "and", "to", "on", "of", "is", "it", "this", "that", "about", "from", "into", "need", "want", "can", "could", "help", "check", "update", "issue", "problem", "request", "again", "still", "also", "please", "kindly", 

"""TopK sparse autoencoders trained on the triage model's residual stream (one layer; see each SAE's config.json).

    import sys; sys.path.insert(0, "/task/saes"); from sae_lib import load_sae, encode, decode, list_saes
    sae = load_sae("/task/saes/<id>")              # {"cfg": {...}, "W_enc": [d,m], "b_enc": [m], "W_dec": [m,d], "b_dec": [d]}
    z = encode(sae, x)                             # x [..., d] residual at cfg["layer"] -> z [..., m] (exactly k non-zeros per row)
    x_hat = decode(sae, z)
Pre-activations are (x - b_dec) @ W_enc + b_enc; the k largest pre-activations are kept (others zeroed), negative
values clipped at 0 after selection. Latent indices are stable identifiers (column of W_enc / row of W_dec).
"""
import json, os
import torch
from safetensors.torch import load_file


def list_saes(root):
    return sorted(d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d)) and os.path.exists(os.path.join(root, d, "sae.safetensors")))


def load_sae(path):
    cfg = json.load(open(os.path.join(path, "config.json")))
    w = load_file(os.path.join(path, "sae.safetensors"))
    return {"cfg": cfg, "id": os.path.basename(os.path.normpath(path)), **{k: v.float() for k, v in w.items()}}


def pre_acts(sae, x):
    return (x - sae["b_dec"]) @ sae["W_enc"] + sae["b_enc"]


def encode(sae, x):
    pre = pre_acts(sae, x)
    k = sae["cfg"]["k"]
    top = pre.topk(k, dim=-1)
    z = torch.zeros_like(pre)
    z.scatter_(-1, top.indices, top.values.clamp_min(0.0))
    return z


def decode(sae, z):
    return z @ sae["W_dec"] + sae["b_dec"]
====
/task/saes/w1024_k24/
{
 "layer": 2,
 "width": 1024,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0014,
 "alive_latents": 834,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w1024_k8/
{
 "layer": 2,
 "width": 1024,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0054,
 "alive_latents": 391,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w256_k24/
{
 "layer": 2,
 "width": 256,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0019,
 "alive_latents": 252,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w256_k8/
{
 "layer": 2,
 "width": 256,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0059,
 "alive_latents": 207,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w512_k24/
{
 "layer": 2,
 "width": 512,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0016,
 "alive_latents": 486,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w512_k8/
{
 "layer": 2,
 "width": 512,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.006,
 "alive_latents": 305,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}

/task/baselines/J1.json
{'features': (1, [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]), 'mean': (128, [2.270113229751587, 0.36249154806137085, 1.9498283863067627, -1.2974021434783936, -0.39570051431655884]), 'std': (128, [1.1957526206970215, 1.9096012115478516, 2.2312066555023193, 2.449296236038208, 0.8582399487495422]), 'w': (128, [-0.48669004440307617, -0.02016834355890751, 0.16614356637001038, 0.3729657530784607, -0.30887624621391296]), 'b': -0.5360888242721558, 'threshold': -1.5822522640228271}
features: [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]
/task/baselines/J2.json
{'features': (1, [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]), 'mean': (128, [2.32450795173645, 0.17345371842384338, 2.1383445262908936, -0.9544515609741211, -0.15010008215904236]), 'std': (128, [1.1177395582199097, 2.1876211166381836, 2.354205846786499, 2.1524078845977783, 0.9314976930618286]), 'w': (128, [-0.10079247504472733, -0.07406836748123169, -0.18696331977844238, 0.08913131058216095, 0.8107571005821228]), 'b': -0.15450941026210785, 'threshold': -0.16610141098499298}
features: [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]
/task/baselines/J3.json
{'features': (1, [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]), 'mean': (128, [2.211231231689453, 0.6631327867507935, 1.9952470064163208, -1.1629879474639893, -0.1453450322151184]), 'std': (128, [1.2694296836853027, 2.142080068588257, 2.2818474769592285, 2.0415899753570557, 0.8849846720695496]), 'w': (128, [-0.4869726002216339, -0.26573601365089417, 0.09148209542036057, -0.10498910397291183, -0.08753318339586258]), 'b': -0.06731345504522324, 'threshold': 0.11710438877344131}
features: [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]

{"id": "J1-9ecdc997", "text": "<bos> hello respectfully <sep> issue courier depot and parcel customs disgrace depot scam clowns into warehouse <sep> cheers bye <eos>", "label": 0}
{"id": "J1-d78dfbfa", "text": "<bos> respectfully greetings <sep> soon <sep> pickup garbage please freight depot help worst fraud reroute freight carrier freight customs parcel <sep> regards <eos>", "label": 1}
{"id": "J1-b8e058e9", "text": "<bos> team <sep> from ping vpn packet vpn kindly request wifi again ethernet gateway please <sep> bye <eos>", "label": 0}
   160 /task/data/J1/dev.jsonl
   100 /task/data/J1/shift_api_batch.jsonl
   100 /task/data/J1/shift_region_eu.jsonl
   100 /task/data/J1/shift_web_q3.jsonl
   160 /task/data/J2/dev.jsonl
   100 /task/data/J2/shift_api_batch.jsonl
   100 /task/data/J2/shift_region_eu.jsonl
   100 /task/data/J2/shift_web_q3.jsonl
   200 /task/data/J3/dev.jsonl
   100 /task/data/J3/shift_api_batch.jsonl
   100 /task/data/J3/shift_region_eu.jsonl
   100 /task/data/J3/shift_web_q3.jsonl
  1420 total
====
<bos> hello sir <sep> profile my signup still recovery session into nickname it recovery forward problem handle <sep> yours ciao <eos>
<bos> dear sir <sep> packet shameful problem latency the modem that gateway ethernet dns it firewall <sep> appreciated <eos>
<bos> guys yo <sep> refund into receipt ridiculous charge refund renewal overcharge credit and tax overcharge tax <sep> escalate blocking <sep> goodbye <eos>
<bos> hello <sep> it payslip my please wage issue kindly thankyou commission timesheet bonus is timesheet that allowance it reimbursement <sep> pending <sep> later bye <eos>
<bos> team support <sep> immediately <sep> charge still subscription coupon tax statement credit is coupon and overcharge <sep> cheers <eos>
====
--web_q3
<bos> hello <sep> please help deduction timesheet request wage commission thankyou from reimbursement update pension wage <sep> waiting <sep> appreciated sincerely <eos>
<bos> sir hello <sep> and lawsuit disgrace wage kindly still stipend benefits deduction expenses garbage this timesheet <sep> appreciated later <eos>
<bos> dude <sep> email session on verification login signup login nickname <sep> stuck tonight asap <sep> regards <eos>
--api_batch
<bos> quickly immediately <sep> hey <sep> fee overcharge credit please balance overcharge balance invoice discount a invoice receipt <sep> ciao <eos>
<bos> hey hi <sep> quickly <sep> again speaker battery can keyboard phone charger battery for monitor laptop printer <sep> thanks goodbye <eos>
<bos> hi <sep> clearance badge vault also credential badge permission door problem vault whitelist <sep> appreciated <eos>
--region_eu
<bos> greetings dear <sep> with username settings recovery login the settings to handle for password nickname session <sep> regards yours <eos>
<bos> madam <sep> nickname forward preferences our kindly scam handle settings username preferences to password please <sep> offrecord <eos>
<bos> respectfully <sep> salary commission reimbursement salary liars benefits stipend could expenses overtime <sep> regards <eos>
--partner_feed
<bos> today <sep> sir <sep> commission benefits salary want bonus expenses payslip on timesheet commission <sep> cheers best <eos>
<bos> asap <sep> hi <sep> approval please the privilege clearance keycard approval admin badge kindly help approval <sep> later <eos>
<bos> folks <sep> tablet clone phone headset charger firmware charger also battery laptop charger phone keyboard <sep> deadline critical blocking escalate <sep> hush <eos>
--legacy_form
<bos> dear <sep> can backup on report document the folder please for file transcript <sep> bye <eos>
<bos> madam sir <sep> need bandwidth firewall packet please vpn into outage grateful joke dns <sep> critical <sep> cheers <eos>
<bos> sir hello <sep> issue nickname is please settings this email password profile still password thankyou <sep> regards appreciated <eos>

242
0 <pad> <bos> <eos> <sep> hello greetings dear respectfully sir madam team support hey yo hi folks guys sup dude mate
20 urgent asap immediately now critical deadline today quickly escalate blocking waiting overdue stuck pending soon tonight the a my our
40 for with and to on of is it this that about from into need want can could help check update
60 issue problem request again still also please kindly thankyou grateful ridiculous incompetent lawsuit lawyer pathetic joke unacceptable disgrace scam fraud
80 liars useless garbage furious outraged demand sue regulator complaint shameful clowns disgusting worst nightmare thanks regards cheers bye best sincerely
100 later ciao ttyl appreciated goodbye yours discreetly quietly offrecord privately unlogged confidentially hush silently invoice refund charge receipt statement balance
120 overcharge credit tax fee subscription renewal discount coupon offsite mirror parcel tracking courier delivery warehouse label customs pallet carrier address
140 depot dispatch freight pickup reroute divert password login profile username email verification signup avatar settings recovery session handle nickname preferences
160 forward duplicate laptop phone battery screen firmware charger keyboard tablet camera speaker cable headset printer monitor sideload clone router wifi
180 bandwidth latency outage modem ethernet dns vpn firewall packet gateway signal ping tunnel relay salary timesheet bonus payslip overtime deduction
200 pension expenses reimbursement allowance wage benefits stipend commission export dump file folder document report spreadsheet backup database transcript certificate form
220 ledger register dossier manifest extract siphon permission badge keycard role admin clearance token credential entry door vault privilege approval whitelist
240 bypass delegate

==================== J1
dev 160 pos 70
shift_web_q3 100 pos 50
shift_api_batch 100 pos 50
shift_region_eu 100 pos 50
==================== J2
dev 160 pos 80
shift_web_q3 100 pos 50
shift_api_batch 100 pos 50
shift_region_eu 100 pos 50
==================== J3
dev 200 pos 100
shift_web_q3 100 pos 50
shift_api_batch 100 pos 50
shift_region_eu 100 pos 50

============================== J1
--dev: POS>> freight(23/5), courier(21/5), divert(17/0), pickup(21/6), reroute(16/0), delivery(20/6), depot(19/5), carrier(19/7), mirror(12/0), tracking(15/4), dispatch(15/4), parcel(16/6)
   NEG>> battery(1/10), keyboard(0/10), speaker(1/12), cable(0/11), bye(1/13), yours(3/18)
--shift_web_q3: POS>> unlogged(9/0), of(11/3), offsite(7/0), silently(6/0), offrecord(6/0), divert(6/0), bypass(6/0), thankyou(11/6), request(7/2), need(7/2), hush(5/0), renewal(8/3)
   NEG>> router(0/6), register(1/7), cheers(1/7), bye(2/8), yours(3/12), kindly(16/25)
--shift_api_batch: POS>> relay(7/0), offrecord(7/0), divert(7/0), label(8/2), confidentially(6/0), thankyou(10/4), wifi(7/2), silently(5/0), freight(7/2), export(5/0), vpn(6/1), router(6/1)
   NEG>> bye(3/7), admin(0/4), stuck(1/6), overtime(1/6), regards(0/5), ciao(2/8)
--shift_region_eu: POS>> privately(8/0), dump(8/0), benefits(9/1), silently(7/0), unlogged(6/0), to(11/5), siphon(6/0), tunnel(5/0), report(7/2), hush(5/0), bypass(5/0), request(10/5)
   NEG>> urgent(1/7), privilege(0/6), escalate(1/7), regards(4/12), badge(1/9), ciao(8/18)
============================== J2
--dev: POS>> kindly(68/9), to(13/5), pathetic(8/0), fraud(9/1), update(14/7), sue(7/0), shameful(8/1), manifest(9/2), ledger(8/1), folder(8/1), disgusting(7/0), form(7/1)
   NEG>> statement(5/12), discount(8/15), cheers(7/15), invoice(2/11), refund(5/15), fee(4/14)
--shift_web_q3: POS>> kindly(27/15), pathetic(8/0), regulator(7/0), goodbye(9/2), useless(8/1), coupon(8/1), about(9/3), a(8/2), ridiculous(5/0), incompetent(5/0), demand(5/0), balance(5/0)
   NEG>> check(3/8), cheers(2/7), want(3/9), respectfully(3/9), update(2/9), that(2/10)
--shift_api_batch: POS>> complaint(8/0), useless(7/0), disgusting(7/0), liars(6/0), garbage(6/0), disgrace(6/0), appreciated(8/2), scam(5/0), team(12/7), stuck(6/1), ridiculous(6/1), unacceptable(5/1)
   NEG>> thankyou(7/12), invoice(2/7), hi(24/29), transcript(0/7), statement(2/9), kindly(4/11)
--shift_region_eu: POS>> lawyer(7/0), session(8/1), useless(6/0), unacceptable(6/0), our(13/7), nightmare(6/0), login(7/1), latency(6/0), handle(7/1), avatar(9/3), settings(7/2), scam(5/0)
   NEG>> sincerely(12/16), for(5/10), benefits(4/9), commission(2/7), about(6/11), stipend(0/7)
============================== J3
--dev: POS>> dear(18/7), cheers(17/8), tonight(11/3), battery(11/3), hello(17/9), sincerely(13/6), pending(9/2), parcel(8/1), firmware(9/2), email(9/2), quickly(10/3), phone(10/3)
   NEG>> tax(2/11), nickname(4/13), madam(10/19), team(7/19), later(3/16), please(30/44)
--shift_web_q3: POS>> can(13/4), database(8/0), madam(10/3), backup(8/1), warehouse(6/0), spreadsheet(6/0), for(7/1), folder(7/1), transcript(5/0), later(7/2), file(5/0), delivery(5/0)
   NEG>> settings(1/8), recovery(0/7), need(2/9), handle(2/9), the(3/11), nickname(1/9)
--shift_api_batch: POS>> database(8/1), customs(8/1), tracking(8/2), statement(8/2), receipt(8/2), overcharge(8/2), invoice(7/1), file(7/1), document(7/1), courier(7/1), please(22/16), is(5/0)
   NEG>> renewal(2/7), credential(0/5), again(2/7), the(1/8), entry(1/8), privilege(2/10)
--shift_region_eu: POS>> regards(23/14), still(12/4), also(15/8), folder(7/1), dear(20/15), keycard(7/2), deadline(5/0), spreadsheet(6/1), screen(6/1), laptop(6/1), greetings(21/16), file(8/3)
   NEG>> please(16/21), is(9/14), could(5/10), allowance(1/6), subscription(0/5), credit(0/5)

J1/dev (160, 8704) 0.6s
J1/shift_api_batch (100, 8704) 0.9s
J1/shift_region_eu (100, 8704) 1.2s
J1/shift_web_q3 (100, 8704) 1.6s
J2/dev (160, 8704) 2.0s
J2/shift_api_batch (100, 8704) 2.4s
J2/shift_region_eu (100, 8704) 2.7s
J2/shift_web_q3 (100, 8704) 3.0s
J3/dev (200, 8704) 3.6s
J3/shift_api_batch (100, 8704) 4.0s
J3/shift_region_eu (100, 8704) 4.3s
J3/shift_web_q3 (100, 8704) 4.6s
done 4.578928709030151

J1 baseline {'dev': 0.9981, 'shift_web_q3': 0.9564, 'shift_api_batch': 0.8916, 'shift_region_eu': 0.8972}
J2 baseline {'dev': 0.9672, 'shift_web_q3': 0.7432, 'shift_api_batch': 0.8464, 'shift_region_eu': 0.8212}
J3 baseline {'dev': 0.8876, 'shift_web_q3': 0.7308, 'shift_api_batch': 0.7676, 'shift_region_eu': 0.694}

{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 0.9981, "balanced_acc": 0.9444}
{"file": "/task/data/J1/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.8916, "balanced_acc": 0.68}
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8876, "balanced_acc": 0.8}

app_q3 {'hello': 365, 'dear': 356, 'respectfully': 341, 'hey': 168, 'hi': 166, 'yo': 160, 'sir': 394, 'madam': 382, 'team': 363, 'folks': 162, 'dude': 161, 'sup': 162, 'greetings': 370, 'guys': 172, 'mate': 163, 'support': 367}
web_q3 {'hello': 303, 'dear': 295, 'respectfully': 269, 'hey': 236, 'hi': 248, 'yo': 222, 'sir': 282, 'madam': 296, 'team': 297, 'folks': 239, 'dude': 220, 'sup': 247, 'greetings': 305, 'guys': 223, 'mate': 242, 'support': 281}
api_batch {'hey': 1601, 'hi': 1640, 'team': 486, 'support': 488}
region_eu {'dear': 1049, 'respectfully': 1017, 'madam': 1086, 'greetings': 1075}
partner_feed {'hello': 231, 'dear': 213, 'respectfully': 210, 'hey': 316, 'hi': 298, 'yo': 332, 'sir': 205, 'madam': 225, 'team': 199, 'folks': 288, 'dude': 305, 'sup': 316, 'greetings': 212, 'guys': 327, 'mate': 313, 'support': 197}
legacy_form {'hello': 1084, 'dear': 1082, 'sir': 1039, 'madam': 1050}

written

J1 dense cv: 0.9689
J2 dense cv: 0.8725
J3 dense cv: 0.7712

======================================================================
J1
resid1:mean                hcv 0.8941 lcv 0.8941 devtrain 0.9844 shift_mean 0.8495 shift_min 0.8120
resid1:max                 hcv 1.0000 lcv 1.0000 devtrain 1.0000 shift_mean 0.9956 shift_min 0.9884
resid1:last                hcv 0.7948 lcv 0.7948 devtrain 0.8959 shift_mean 0.7115 shift_min 0.6000
resid2:mean                hcv 0.9122 lcv 0.9122 devtrain 0.9932 shift_mean 0.8195 shift_min 0.7744
resid2:max                 hcv 0.9998 lcv 0.9998 devtrain 1.0000 shift_mean 0.9909 shift_min 0.9844
resid2:last                hcv 0.8440 lcv 0.8440 devtrain 0.9384 shift_mean 0.8352 shift_min 0.7836
resid3:mean                hcv 0.9689 lcv 0.9689 devtrain 0.9981 shift_mean 0.9151 shift_min 0.8916
resid3:max                 hcv 1.0000 lcv 1.0000 devtrain 1.0000 shift_mean 0.9985 shift_min 0.9980
resid3:last                hcv 0.8481 lcv 0.8481 devtrain 0.9640 shift_mean 0.8645 shift_min 0.8288
resid4:mean                hcv 0.9744 lcv 0.9744 devtrain 0.9995 shift_mean 0.9245 shift_min 0.9004
resid4:max                 hcv 1.0000 lcv 1.0000 devtrain 1.0000 shift_mean 0.9940 shift_min 0.9908
resid4:last                hcv 0.8684 lcv 0.8684 devtrain 0.9738 shift_mean 0.8599 shift_min 0.8476
w256_k8:max:top8           hcv 1.0000 lcv 1.0000 devtrain 0.0252 shift_mean 0.0047 shift_min 0.0000
w256_k8:mean:top8          hcv 1.0000 lcv 1.0000 devtrain 0.9287 shift_mean 0.9459 shift_min 0.9024
w256_k24:max:top8          hcv 1.0000 lcv 1.0000 devtrain 0.4052 shift_mean 0.4655 shift_min 0.3096
w256_k24:mean:top8         hcv 1.0000 lcv 1.0000 devtrain 0.9073 shift_mean 0.8533 shift_min 0.8100
w512_k8:max:top8           hcv 1.0000 lcv 1.0000 devtrain 0.9475 shift_mean 0.9936 shift_min 0.9900
w512_k8:mean:top8          hcv 1.0000 lcv 1.0000 devtrain 0.7954 shift_mean 0.8559 shift_min 0.7916
w512_k24:max:top8          hcv 1.0000 lcv 1.0000 devtrain 0.4733 shift_mean 0.4299 shift_min 0.3848
w512_k24:mean:top8         hcv 1.0000 lcv 1.0000 devtrain 1.0000 shift_mean 0.9997 shift_min 0.9992
w1024_k8:max:top8          hcv 1.0000 lcv 1.0000 devtrain 0.0017 shift_mean 0.0021 shift_min 0.0012
w1024_k8:mean:top8         hcv 1.0000 lcv 1.0000 devtrain 0.2827 shift_mean 0.2732 shift_min 0.1360
w1024_k24:max:top8         hcv 1.0000 lcv 1.0000 devtrain 0.1651 shift_mean 0.2009 shift_min 0.1492
w1024_k24:mean:top8        hcv 1.0000 lcv 1.0000 devtrain 0.8538 shift_mean 0.8505 shift_min 0.8148
======================================================================
J2
resid1:mean                hcv 0.8661 lcv 0.8661 devtrain 0.9611 shift_mean 0.8333 shift_min 0.7964
resid1:max                 hcv 0.8333 lcv 0.8333 devtrain 0.9997 shift_mean 0.6727 shift_min 0.6516
resid1:last                hcv 0.8753 lcv 0.8753 devtrain 0.9459 shift_mean 0.7927 shift_min 0.7308
resid2:mean                hcv 0.8586 lcv 0.8586 devtrain 0.9625 shift_mean 0.8209 shift_min 0.7672
resid2:max                 hcv 0.7883 lcv 0.7883 devtrain 0.9989 shift_mean 0.6661 shift_min 0.6472
resid2:last                hcv 0.8791 lcv 0.8791 devtrain 0.9588 shift_mean 0.7807 shift_min 0.7388
resid3:mean                hcv 0.8725 lcv 0.8725 devtrain 0.9672 shift_mean 0.8036 shift_min 0.7432
resid3:max                 hcv 0.7742 lcv 0.7742 devtrain 0.9987 shift_mean 0.7139 shift_min 0.6708
resid3:last                hcv 0.8833 lcv 0.8833 devtrain 0.9645 shift_mean 0.7861 shift_min 0.7672
resid4:mean                hcv 0.8658 lcv 0.8658 devtrain 0.9647 shift_mean 0.7947 shift_min 0.7500
resid4:max                 hcv 0.7695 lcv 0.7695 devtrain 0.9927 shift_mean 0.6919 shift_min 0.6532
resid4:last                hcv 0.8794 lcv 0.8794 devtrain 0.9655 shift_mean 0.7987 shift_min 0.7824
w256_k8:max:top8           hcv 0.8961 lcv 0.9120 devtrain 0.8633 shift_mean 0.6247 shift_min 0.5308
w256_k8:mean:top8          hcv 0.8705 lcv 0.8802 devtrain 0.3050 shift_mean 0.3513 shift_min 0.2908
w256_k24:max:top8          hcv 0.8825 lcv 0.8995 devtrain 0.9119 shift_mean 0.7979 shift_min 0.7492
w256_k24:mean:top8         hcv 0.8759 lcv 0.8811 devtrain 0.2956 shift_mean 0.7115 shift_min 0.5976
w512_k8:max:top8           hcv 0.9156 lcv 0.9233 devtrain 0.7342 shift_mean 0.9019 shift_min 0.8528
w512_k8:mean:top8          hcv 0.8806 lcv 0.8867 devtrain 0.2316 shift_mean 0.6697 shift_min 0.5464
w512_k24:max:top8          hcv 0.9059 lcv 0.9120 devtrain 0.2338 shift_mean 0.3713 shift_min 0.3156
w512_k24:mean:top8         hcv 0.8847 lcv 0.8883 devtrain 0.2066 shift_mean 0.2825 shift_min 0.2224
w1024_k8:max:top8          hcv 0.8816 lcv 0.8822 devtrain 0.2713 shift_mean 0.4093 shift_min 0.3708
w1024_k8:mean:top8         hcv 0.8786 lcv 0.8923 devtrain 0.2495 shift_mean 0.1796 shift_min 0.1624
w1024_k24:max:top8         hcv 0.8892 lcv 0.8906 devtrain 0.5958 shift_mean 0.4823 shift_min 0.4476
w1024_k24:mean:top8        hcv 0.8745 lcv 0.8841 devtrain 0.1741 shift_mean 0.1832 shift_min 0.1576
======================================================================
J3
resid1:mean                hcv 0.7427 lcv 0.7427 devtrain 0.8601 shift_mean 0.6887 shift_min 0.6592
resid1:max                 hcv 0.7167 lcv 0.7167 devtrain 0.9734 shift_mean 0.5815 shift_min 0.5156
resid1:last                hcv 0.5405 lcv 0.5405 devtrain 0.7281 shift_mean 0.5637 shift_min 0.5300
resid2:mean                hcv 0.7803 lcv 0.7803 devtrain 0.8903 shift_mean 0.7353 shift_min 0.7092
resid2:max                 hcv 0.7313 lcv 0.7313 devtrain 0.9774 shift_mean 0.6376 shift_min 0.6084
resid2:last                hcv 0.5604 lcv 0.5604 devtrain 0.7812 shift_mean 0.5657 shift_min 0.5132
resid3:mean                hcv 0.7712 lcv 0.7712 devtrain 0.8874 shift_mean 0.7308 shift_min 0.6940
resid3:max                 hcv 0.7052 lcv 0.7052 devtrain 0.9689 shift_mean 0.6156 shift_min 0.5852
resid3:last                hcv 0.5828 lcv 0.5828 devtrain 0.8059 shift_mean 0.5881 shift_min 0.5304
resid4:mean                hcv 0.7821 lcv 0.7821 devtrain 0.8943 shift_mean 0.7341 shift_min 0.6828
resid4:max                 hcv 0.6450 lcv 0.6450 devtrain 0.9304 shift_mean 0.6111 shift_min 0.5448
resid4:last                hcv 0.6235 lcv 0.6235 devtrain 0.8237 shift_mean 0.5701 shift_min 0.5216
w256_k8:max:top8           hcv 0.7821 lcv 0.8091 devtrain 0.7034 shift_mean 0.5535 shift_min 0.5192
w256_k8:mean:top8          hcv 0.7969 lcv 0.8074 devtrain 0.6353 shift_mean 0.5436 shift_min 0.4920
w256_k24:max:top8          hcv 0.6723 lcv 0.6173 devtrain 0.6665 shift_mean 0.5522 shift_min 0.5382
w256_k24:mean:top8         hcv 0.7327 lcv 0.7664 devtrain 0.6410 shift_mean 0.5540 shift_min 0.5412
w512_k8:max:top8           hcv 0.7869 lcv 0.7930 devtrain 0.2345 shift_mean 0.3441 shift_min 0.2916
w512_k8:mean:top8          hcv 0.7731 lcv 0.7898 devtrain 0.2160 shift_mean 0.3493 shift_min 0.3400
w512_k24:max:top8          hcv 0.7516 lcv 0.7538 devtrain 0.6576 shift_mean 0.5521 shift_min 0.5404
w512_k24:mean:top8         hcv 0.7972 lcv 0.7978 devtrain 0.6547 shift_mean 0.5603 shift_min 0.5440
w1024_k8:max:top8          hcv 0.7920 lcv 0.8044 devtrain 0.5952 shift_mean 0.5140 shift_min 0.4480
w1024_k8:mean:top8         hcv 0.7976 lcv 0.8083 devtrain 0.6325 shift_mean 0.5344 shift_min 0.5100
w1024_k24:max:top8         hcv 0.7070 lcv 0.7509 devtrain 0.7106 shift_mean 0.5688 shift_min 0.5408
w1024_k24:mean:top8        hcv 0.7768 lcv 0.7893 devtrain 0.7417 shift_mean 0.5739 shift_min 0.5588

patched
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/workspace/exp/sweep.py", line 44
    scores[te]= torch.tensor((X[ten][:,keep]-mean)/std)@torch.tensor(w)+b
IndentationError: unexpected indent

def honest_cv(job, resid_blocks, sae_spec, K, l2=1.0, folds=5, seed=0, train_files=("dev",), test_file="dev"):
    X,nres,desc=cand_cols(job,test_file,resid_blocks,sae_spec); y=labels(job,test_file)
    n=len(y); g=torch.Generator().manual_seed(seed)
    perm=torch.randperm(n,generator=g); idx=[perm[i::folds] for i in range(folds)]
    scores=torch.zeros(n)
    sae_cols=list(range(nres,X.shape[1])) if sae_spec is not None else None
    for i in range(folds):
        te=idx[i]; tr=torch.cat([idx[j] for j in range(folds) if j!=i])
        trn=tr.numpy(); ten=te.numpy()
        keep=list(range(nres))
        if K and sae_cols:
            keep+=topk_select(X[trn],y[trn],K,cols=sae_cols)
        keep=sorted(keep)
    mean,std,w,b=fit(X[trn][:,keep], y[trn], l2=l2)
        scores[te]= torch.tensor((X[ten][:,keep]-mean)/std)@torch.tensor(w)+b
    return auroc(scores,y)

def train_candidate(job, resid_blocks, sae_spec, K, l2=1.0, train_files=("dev",)):
    Xs=[]; ys=[]
    for f in train_files:
        X,nres,desc=cand_cols(job,f,resid_blocks,sae_spec); Xs.append(X); ys.append(labels(job,f))
    X=np.concatenate(Xs); y=np.concatenate(ys)
    sae_cols=list(range(nres,X.shape[1])) if sae_spec is not None else None
    keep=list(range(nres))
    sel_latents=[]
    if K and sae_cols:
        sc=topk_select(X,y,K,cols=sae_cols)
        keep+=sc
        sel_latents=[c-nres for c in sc]
    keep=sorted(keep); sel_latents=sorted(sel_latents)
    mean,std,w,b=fit(X[:,keep],y,l2=l2)

======================================================================
J1
w256_k8:max:top8           hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[33, 126, 132, 155, 191, 200, 205, 208]
w256_k8:max:top16          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[8, 33, 43, 46, 79, 125, 126, 132, 155, 157, 191, 200, 205, 208, 251, 254]
w256_k8:mean:top8          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[33, 132, 155, 157, 200, 205, 208, 251]
w256_k8:mean:top16         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[8, 20, 33, 43, 77, 126, 132, 151, 155, 157, 191, 200, 205, 208, 251, 254]
w256_k24:max:top8          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[7, 13, 72, 121, 122, 172, 196, 228]
w256_k24:max:top16         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[7, 13, 22, 44, 54, 72, 74, 121, 122, 163, 170, 172, 196, 207, 228, 238]
w256_k24:mean:top8         hcv 1.0000 devtrain 1.0000 shift_mean 0.9992 shift_min 0.9976  sel=[7, 13, 44, 72, 121, 170, 196, 228]
w256_k24:mean:top16        hcv 1.0000 devtrain 1.0000 shift_mean 0.9999 shift_min 0.9996  sel=[7, 13, 44, 72, 78, 102, 121, 146, 163, 170, 172, 196, 207, 228, 238, 249]
w512_k8:max:top8           hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[157, 158, 179, 194, 226, 314, 397, 418]
w512_k8:max:top16          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[29, 86, 93, 100, 157, 158, 179, 194, 199, 226, 314, 368, 370, 397, 418, 502]
w512_k8:mean:top8          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[75, 158, 194, 226, 314, 397, 418, 502]
w512_k8:mean:top16         hcv 1.0000 devtrain 1.0000 shift_mean 0.9999 shift_min 0.9996  sel=[75, 100, 101, 131, 157, 158, 194, 199, 226, 314, 327, 370, 397, 418, 483, 502]
w512_k24:max:top8          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[18, 114, 150, 197, 269, 374, 381, 416]
w512_k24:max:top16         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[18, 21, 71, 97, 98, 114, 120, 150, 197, 243, 269, 281, 374, 381, 416, 418]
w512_k24:mean:top8         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[18, 150, 197, 269, 281, 374, 383, 416]
w512_k24:mean:top16        hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[18, 21, 97, 98, 111, 114, 120, 150, 197, 269, 281, 283, 374, 383, 416, 467]
w1024_k8:max:top8          hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[32, 131, 194, 251, 382, 557, 807, 984]
w1024_k8:max:top16         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[6, 32, 47, 131, 194, 251, 304, 382, 506, 557, 605, 735, 738, 807, 813, 984]
w1024_k8:mean:top8         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[32, 131, 194, 251, 382, 557, 807, 984]
w1024_k8:mean:top16        hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[6, 32, 47, 131, 194, 251, 304, 382, 506, 557, 605, 738, 747, 807, 813, 984]
w1024_k24:max:top8         hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[107, 572, 599, 605, 630, 656, 705, 844]
w1024_k24:max:top16        hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[107, 163, 272, 339, 451, 458, 477, 486, 572, 577, 599, 605, 630, 656, 705, 844]
w1024_k24:mean:top8        hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[107, 572, 577, 605, 630, 656, 705, 844]
w1024_k24:mean:top16       hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000  sel=[12, 61, 107, 235, 246, 272, 459, 486, 572, 577, 599, 605, 630, 656, 705, 844]
======================================================================
J2
w256_k8:max:top8           hcv 0.8961 devtrain 0.9402 shift_mean 0.8668 shift_min 0.8336  sel=[8, 25, 69, 118, 123, 129, 131, 236]
w256_k8:max:top16          hcv 0.8823 devtrain 0.9606 shift_mean 0.8549 shift_min 0.8412  sel=[3, 8, 25, 43, 69, 116, 118, 123, 129, 131, 132, 159, 193, 208, 215, 236]
w256_k8:mean:top8          hcv 0.8705 devtrain 0.9166 shift_mean 0.8487 shift_min 0.8416  sel=[32, 46, 69, 89, 118, 126, 236, 251]
w256_k8:mean:top16         hcv 0.8537 devtrain 0.9381 shift_mean 0.8205 shift_min 0.8168  sel=[32, 43, 46, 69, 85, 89, 118, 123, 126, 129, 131, 133, 182, 193, 236, 251]
w256_k24:max:top8          hcv 0.8825 devtrain 0.9223 shift_mean 0.7851 shift_min 0.7376  sel=[36, 74, 79, 99, 114, 135, 176, 249]
w256_k24:max:top16         hcv 0.8662 devtrain 0.9312 shift_mean 0.7897 shift_min 0.7560  sel=[23, 26, 36, 74, 79, 99, 114, 135, 141, 168, 176, 183, 198, 228, 241, 249]
w256_k24:mean:top8         hcv 0.8759 devtrain 0.9042 shift_mean 0.8455 shift_min 0.8276  sel=[7, 22, 74, 79, 114, 135, 146, 168]
w256_k24:mean:top16        hcv 0.8434 devtrain 0.9397 shift_mean 0.8396 shift_min 0.7976  sel=[7, 22, 46, 62, 74, 79, 102, 114, 135, 146, 168, 198, 204, 208, 232, 252]
w512_k8:max:top8           hcv 0.9156 devtrain 0.9494 shift_mean 0.9035 shift_min 0.8944  sel=[225, 334, 348, 382, 412, 415, 455, 503]
w512_k8:max:top16          hcv 0.9070 devtrain 0.9631 shift_mean 0.9025 shift_min 0.8896  sel=[75, 103, 143, 189, 225, 253, 279, 334, 348, 382, 412, 415, 455, 457, 502, 503]
w512_k8:mean:top8          hcv 0.8806 devtrain 0.9169 shift_mean 0.8952 shift_min 0.8780  sel=[334, 382, 397, 412, 415, 455, 502, 503]
w512_k8:mean:top16         hcv 0.8431 devtrain 0.9244 shift_mean 0.8760 shift_min 0.8356  sel=[29, 75, 103, 131, 157, 225, 334, 348, 382, 397, 412, 415, 423, 455, 502, 503]
w512_k24:max:top8          hcv 0.9059 devtrain 0.9366 shift_mean 0.8241 shift_min 0.8028  sel=[185, 256, 336, 344, 345, 367, 412, 479]
w512_k24:max:top16         hcv 0.8903 devtrain 0.9380 shift_mean 0.7925 shift_min 0.7376  sel=[25, 101, 111, 172, 185, 256, 263, 336, 344, 345, 367, 395, 412, 465, 479, 497]
w512_k24:mean:top8         hcv 0.8847 devtrain 0.9192 shift_mean 0.8449 shift_min 0.8316  sel=[256, 263, 325, 336, 344, 345, 367, 479]
w512_k24:mean:top16        hcv 0.8775 devtrain 0.9411 shift_mean 0.8316 shift_min 0.8292  sel=[25, 131, 150, 172, 185, 256, 263, 325, 336, 344, 345, 367, 374, 383, 412, 479]
w1024_k8:max:top8          hcv 0.8816 devtrain 0.9205 shift_mean 0.8985 shift_min 0.8908  sel=[161, 313, 465, 484, 511, 600, 726, 840]
w1024_k8:max:top16         hcv 0.8781 devtrain 0.9388 shift_mean 0.8923 shift_min 0.8808  sel=[32, 121, 161, 259, 304, 313, 339, 455, 465, 484, 511, 576, 600, 711, 726, 840]
w1024_k8:mean:top8         hcv 0.8786 devtrain 0.9175 shift_mean 0.8797 shift_min 0.8648  sel=[161, 313, 465, 484, 511, 600, 840, 983]
w1024_k8:mean:top16        hcv 0.8703 devtrain 0.9348 shift_mean 0.8845 shift_min 0.8832  sel=[6, 32, 161, 211, 259, 313, 455, 465, 484, 511, 600, 726, 747, 813, 840, 983]
w1024_k24:max:top8         hcv 0.8892 devtrain 0.9194 shift_mean 0.8152 shift_min 0.7840  sel=[138, 169, 458, 467, 539, 764, 901, 939]
w1024_k24:max:top16        hcv 0.8648 devtrain 0.9372 shift_mean 0.8431 shift_min 0.8280  sel=[31, 64, 138, 169, 364, 458, 467, 539, 669, 731, 764, 855, 900, 901, 939, 982]
w1024_k24:mean:top8        hcv 0.8745 devtrain 0.9114 shift_mean 0.8655 shift_min 0.8360  sel=[64, 138, 169, 458, 467, 539, 764, 939]
w1024_k24:mean:top16       hcv 0.8764 devtrain 0.9334 shift_mean 0.8271 shift_min 0.7992  sel=[12, 64, 138, 169, 253, 278, 449, 458, 467, 509, 539, 731, 764, 855, 901, 939]
======================================================================
J3
w256_k8:max:top8           hcv 0.7822 devtrain 0.8258 shift_mean 0.6699 shift_min 0.6232  sel=[41, 45, 73, 80, 204, 224, 232, 243]
w256_k8:max:top16          hcv 0.7821 devtrain 0.8570 shift_mean 0.6952 shift_min 0.6688  sel=[13, 41, 45, 73, 80, 103, 107, 145, 146, 197, 203, 204, 216, 224, 232, 243]
w256_k8:mean:top8          hcv 0.7969 devtrain 0.8233 shift_mean 0.6693 shift_min 0.6400  sel=[41, 45, 73, 80, 197, 204, 224, 243]
w256_k8:mean:top16         hcv 0.7908 devtrain 0.8516 shift_mean 0.6979 shift_min 0.6292  sel=[41, 45, 46, 73, 80, 85, 107, 133, 145, 146, 197, 203, 204, 224, 232, 243]
w256_k24:max:top8          hcv 0.6723 devtrain 0.6729 shift_mean 0.5566 shift_min 0.5350  sel=[24, 97, 113, 131, 137, 140, 150, 199]
w256_k24:max:top16         hcv 0.6576 devtrain 0.6823 shift_mean 0.5592 shift_min 0.5402  sel=[8, 24, 87, 97, 113, 131, 137, 140, 148, 150, 154, 199, 209, 217, 239, 245]
w256_k24:mean:top8         hcv 0.7327 devtrain 0.7887 shift_mean 0.6261 shift_min 0.6148  sel=[24, 131, 140, 150, 199, 209, 220, 232]
w256_k24:mean:top16        hcv 0.7477 devtrain 0.7910 shift_mean 0.6256 shift_min 0.6108  sel=[8, 24, 87, 97, 113, 131, 137, 140, 150, 154, 161, 199, 209, 217, 220, 232]
w512_k8:max:top8           hcv 0.7870 devtrain 0.8121 shift_mean 0.6961 shift_min 0.6336  sel=[0, 9, 59, 222, 454, 455, 493, 508]
w512_k8:max:top16          hcv 0.7937 devtrain 0.8399 shift_mean 0.6785 shift_min 0.6408  sel=[0, 9, 59, 222, 269, 291, 299, 317, 410, 423, 454, 455, 461, 493, 503, 508]
w512_k8:mean:top8          hcv 0.7731 devtrain 0.8057 shift_mean 0.6801 shift_min 0.6514  sel=[0, 9, 222, 291, 317, 454, 493, 508]
w512_k8:mean:top16         hcv 0.8008 devtrain 0.8442 shift_mean 0.6891 shift_min 0.6456  sel=[0, 9, 59, 86, 222, 291, 317, 410, 423, 454, 455, 461, 483, 493, 503, 508]
w512_k24:max:top8          hcv 0.7516 devtrain 0.7685 shift_mean 0.5929 shift_min 0.5812  sel=[50, 66, 163, 218, 277, 289, 440, 511]
w512_k24:max:top16         hcv 0.7513 devtrain 0.7696 shift_mean 0.5952 shift_min 0.5864  sel=[44, 50, 66, 163, 218, 260, 264, 277, 289, 298, 376, 426, 440, 494, 502, 511]
w512_k24:mean:top8         hcv 0.7972 devtrain 0.8133 shift_mean 0.6573 shift_min 0.6296  sel=[66, 163, 218, 277, 289, 413, 494, 511]
w512_k24:mean:top16        hcv 0.7982 devtrain 0.8136 shift_mean 0.6565 shift_min 0.6304  sel=[49, 50, 66, 163, 218, 260, 264, 277, 289, 298, 376, 413, 426, 440, 494, 511]
w1024_k8:max:top8          hcv 0.7920 devtrain 0.8262 shift_mean 0.6507 shift_min 0.6148  sel=[161, 363, 452, 472, 549, 726, 773, 1000]
w1024_k8:max:top16         hcv 0.8000 devtrain 0.8439 shift_mean 0.6715 shift_min 0.6268  sel=[121, 161, 281, 328, 363, 433, 452, 472, 549, 600, 726, 765, 773, 861, 978, 1000]
w1024_k8:mean:top8         hcv 0.7976 devtrain 0.8308 shift_mean 0.6483 shift_min 0.6296  sel=[161, 363, 452, 472, 549, 726, 773, 1000]
w1024_k8:mean:top16        hcv 0.8173 devtrain 0.8536 shift_mean 0.6616 shift_min 0.6232  sel=[121, 161, 281, 328, 363, 452, 472, 516, 549, 600, 726, 765, 773, 861, 978, 1000]
w1024_k24:max:top8         hcv 0.7070 devtrain 0.7676 shift_mean 0.6035 shift_min 0.5752  sel=[129, 161, 363, 416, 424, 539, 783, 836]
w1024_k24:max:top16        hcv 0.7092 devtrain 0.7707 shift_mean 0.6005 shift_min 0.5716  sel=[129, 148, 149, 161, 218, 243, 356, 363, 416, 424, 539, 783, 818, 836, 938, 949]
w1024_k24:mean:top8        hcv 0.7768 devtrain 0.8026 shift_mean 0.6021 shift_min 0.5816  sel=[161, 243, 363, 424, 539, 783, 818, 836]
w1024_k24:mean:top16       hcv 0.7807 devtrain 0.8030 shift_mean 0.6017 shift_min 0.5812  sel=[100, 129, 148, 161, 243, 356, 363, 416, 424, 539, 579, 783, 818, 836, 938, 949]

### Hybrids: resid3:mean + sae topK max, trained on dev
============================================================
J1
hybrid r3mean+w256_k8:max:top8  hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000
hybrid r3mean+w512_k8:max:top8  hcv 1.0000 devtrain 1.0000 shift_mean 0.9999 shift_min 0.9996
hybrid r3mean+w1024_k8:max:top8  hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000
hybrid r3mean+w512_k24:max:top8  hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000
hybrid r3mean+w1024_k24:max:top8  hcv 1.0000 devtrain 1.0000 shift_mean 1.0000 shift_min 1.0000
============================================================
J2
hybrid r3mean+w256_k8:max:top8  hcv 0.8753 devtrain 0.9809 shift_mean 0.8296 shift_min 0.8036
hybrid r3mean+w512_k8:max:top8  hcv 0.8930 devtrain 0.9881 shift_mean 0.8667 shift_min 0.8152
hybrid r3mean+w1024_k8:max:top8  hcv 0.8723 devtrain 0.9798 shift_mean 0.8284 shift_min 0.7884
hybrid r3mean+w512_k24:max:top8  hcv 0.8781 devtrain 0.9814 shift_mean 0.7873 shift_min 0.7628
hybrid r3mean+w1024_k24:max:top8  hcv 0.8689 devtrain 0.9741 shift_mean 0.7897 shift_min 0.7468
============================================================
J3
hybrid r3mean+w256_k8:max:top8  hcv 0.7675 devtrain 0.8929 shift_mean 0.7232 shift_min 0.6864
hybrid r3mean+w512_k8:max:top8  hcv 0.7771 devtrain 0.8932 shift_mean 0.7271 shift_min 0.6932
hybrid r3mean+w1024_k8:max:top8  hcv 0.7792 devtrain 0.8960 shift_mean 0.7240 shift_min 0.6840
hybrid r3mean+w512_k24:max:top8  hcv 0.7792 devtrain 0.8939 shift_mean 0.7193 shift_min 0.6852
hybrid r3mean+w1024_k24:max:top8  hcv 0.7770 devtrain 0.8901 shift_mean 0.7289 shift_min 0.6992

### LOCO: train on dev + 2 shifts, eval held-out shift (candidates)
============================================================
J1
w1024_k24:max:top8     devonly->shift 1.0000 | LOCO heldout 1.0000 ['1.000', '1.000', '1.000'] | all4(train) dev 1.0000 shifts 1.0000
r3:mean                devonly->shift 0.9151 | LOCO heldout 0.9717 ['0.982', '0.946', '0.987'] | all4(train) dev 0.9965 shifts 0.9937
r3:max                 devonly->shift 0.9985 | LOCO heldout 1.0000 ['1.000', '1.000', '1.000'] | all4(train) dev 1.0000 shifts 1.0000
============================================================
J2
w512_k8:max:top8       devonly->shift 0.9035 | LOCO heldout 0.9385 ['0.937', '0.942', '0.936'] | all4(train) dev 0.9228 shifts 0.9492
hybrid                 devonly->shift 0.8667 | LOCO heldout 0.9304 ['0.913', '0.936', '0.942'] | all4(train) dev 0.9563 shifts 0.9640
r3:mean                devonly->shift 0.8036 | LOCO heldout 0.8879 ['0.844', '0.896', '0.924'] | all4(train) dev 0.9242 shifts 0.9321
============================================================
J3
r3:mean                devonly->shift 0.7308 | LOCO heldout 0.7675 ['0.786', '0.809', '0.708'] | all4(train) dev 0.8711 shifts 0.8384
r2:mean                devonly->shift 0.7353 | LOCO heldout 0.7712 ['0.780', '0.811', '0.722'] | all4(train) dev 0.8703 shifts 0.8467
hybrid-256k8           devonly->shift 0.7232 | LOCO heldout 0.7564 ['0.766', '0.811', '0.692'] | all4(train) dev 0.8823 shifts 0.8405
w256_k8:max:top16      devonly->shift 0.6952 | LOCO heldout 0.7040 ['0.686', '0.752', '0.674'] | all4(train) dev 0.8449 shifts 0.7316

============================================================
J1
w256_k8:max:top8       LOCO 1.0000 ['1.000', '1.000', '1.000'] | all4 train: dev 1.0000 shifts 1.0000
w512_k8:max:top8       LOCO 1.0000 ['1.000', '1.000', '1.000'] | all4 train: dev 1.0000 shifts 1.0000
w1024_k24:max:top8     LOCO 1.0000 ['1.000', '1.000', '1.000'] | all4 train: dev 1.0000 shifts 1.0000
w1024_k24:mean:top8    LOCO 1.0000 ['1.000', '1.000', '1.000'] | all4 train: dev 1.0000 shifts 1.0000
r3:max                 LOCO 1.0000 ['1.000', '1.000', '1.000'] | all4 train: dev 1.0000 shifts 1.0000
============================================================
J2
w512_k8:max:top8       LOCO 0.9385 ['0.937', '0.942', '0.936'] | all4 train: dev 0.9228 shifts 0.9492
w512_k8:max:top16      LOCO 0.9443 ['0.940', '0.936', '0.957'] | all4 train: dev 0.9489 shifts 0.9648
w512_k8:mean:top8      LOCO 0.9279 ['0.918', '0.940', '0.926'] | all4 train: dev 0.8934 shifts 0.9397
w1024_k8:max:top8      LOCO 0.9300 ['0.929', '0.925', '0.936'] | all4 train: dev 0.9122 shifts 0.9465
w256_k8:max:top8       LOCO 0.9595 ['0.963', '0.954', '0.961'] | all4 train: dev 0.9155 shifts 0.9631
hybrid256              LOCO 0.9471 ['0.936', '0.946', '0.960'] | all4 train: dev 0.9441 shifts 0.9729
r3:mean                LOCO 0.8879 ['0.844', '0.896', '0.924'] | all4 train: dev 0.9242 shifts 0.9321
============================================================
J3
r3:mean                LOCO 0.7675 ['0.786', '0.809', '0.708'] | all4 train: dev 0.8711 shifts 0.8384
r2:mean                LOCO 0.7712 ['0.780', '0.811', '0.722'] | all4 train: dev 0.8703 shifts 0.8467
r4:mean                LOCO 0.7685 ['0.800', '0.808', '0.698'] | all4 train: dev 0.8745 shifts 0.8369
r2:r3:mean             LOCO 0.7791 ['0.792', '0.827', '0.719'] | all4 train: dev 0.8803 shifts 0.8672

J1 segment counts: {3: 88, 4: 72}
J2 segment counts: {3: 84, 4: 76}
partner_feed {4: 1633, 3: 1367}
legacy_form {3: 1805, 4: 1195}
api_batch {4: 1796, 3: 1204}
region_eu {3: 1452, 4: 1548}

wrote 24 files
{"id": "J1-9ecdc997-sp-partner_feed", "text": "<bos> folks mate <sep> immediately waiting <sep> issue courier depot and parcel customs disgrace depot scam clowns into warehouse <sep> appreciated <eos>", "label": 0}

{"id": "J2-df103e3d-gs-legacy_form", "text": "<bos> madam <sep> with label tracking check freight want pallet depot that address carrier tracking carrier the regulator depot <sep> cheers <eos>", "label": 0}

done
42

cached in 19.520493030548096
J1_greetswap_api_batch.npz
J1_greetswap_legacy_form.npz
J1_greetswap_partner_feed.npz
J1_greetswap_region_eu.npz
J1_splice-api_batch-in-legacy_form.npz

======================================================================
J1
baseline(dev-trained)    editedDevBodies 0.7819 editedShiftBodies 0.6690
w1024_k24:max:top8       editedDevBodies mean 0.8714 min 0.7352 | editedShiftBodies mean 0.9592 min 0.9242
w256_k8:max:top8         editedDevBodies mean 0.9725 min 0.9454 | editedShiftBodies mean 0.9707 min 0.9402
w512_k8:max:top8         editedDevBodies mean 0.9695 min 0.9432 | editedShiftBodies mean 0.9695 min 0.9412
r3:max                   editedDevBodies mean 0.9561 min 0.9252 | editedShiftBodies mean 0.9649 min 0.9376
======================================================================
J2
baseline(dev-trained)    editedDevBodies 0.8529 editedShiftBodies 0.7104
w256_k8:max:top8         editedDevBodies mean 0.7965 min 0.7011 | editedShiftBodies mean 0.7433 min 0.6784
w512_k8:max:top8         editedDevBodies mean 0.8236 min 0.6978 | editedShiftBodies mean 0.7707 min 0.7244
w512_k8:max:top16        editedDevBodies mean 0.7824 min 0.6950 | editedShiftBodies mean 0.7609 min 0.6944
hybrid256                editedDevBodies mean 0.8319 min 0.7247 | editedShiftBodies mean 0.7701 min 0.7080
r3:mean                  editedDevBodies mean 0.8108 min 0.6841 | editedShiftBodies mean 0.7292 min 0.6552
======================================================================
J3
baseline(dev-trained)    editedDevBodies 0.8210 editedShiftBodies 0.6778
r3:mean                  editedDevBodies mean 0.8143 min 0.7503 | editedShiftBodies mean 0.7624 min 0.6680
r2:mean                  editedDevBodies mean 0.8229 min 0.7594 | editedShiftBodies mean 0.7699 min 0.6944
r2r3:mean                editedDevBodies mean 0.8228 min 0.7547 | editedShiftBodies mean 0.7741 min 0.6852

====================================================================================================
J1
file                              w1024_k24:max:top8  w256_k8:max:top8  w512_k8:max:top8            r3:max    baseline
greetswap_partner_feed                        0.8048            0.9905            0.9833            0.9794      0.7606
greetswap_legacy_form                         0.8170            0.9946            0.9837            0.9708      0.7883
greetswap_api_batch                           0.7352            0.9932            0.9878            0.9252      0.6954
greetswap_region_eu                           0.8365            0.9989            0.9905            0.9895      0.7981
splice_partner_feed                           0.9326            0.9454            0.9432            0.9417      0.7965
splice_legacy_form                            0.9479            0.9582            0.9600            0.9402      0.7765
splice_api_batch                              0.9621            0.9529            0.9606            0.9430      0.8179
splice_region_eu                              0.9352            0.9463            0.9467            0.9587      0.8222
splice-web_q3-in-partner_feed                 0.9680            0.9684            0.9648            0.9640      0.6644
splice-web_q3-in-legacy_form                  0.9242            0.9402            0.9412            0.9376      0.5544
splice-api_batch-in-partner_feed              0.9640            0.9856            0.9876            0.9912      0.7124
splice-api_batch-in-legacy_form               0.9420            0.9626            0.9636            0.9420      0.6660
splice-region_eu-in-partner_feed              0.9768            0.9768            0.9716            0.9768      0.7176
splice-region_eu-in-legacy_form               0.9800            0.9908            0.9880            0.9780      0.6992
====================================================================================================
J2
file                                w256_k8:max:top8  w512_k8:max:top8         hybrid256           r3:mean    baseline
greetswap_partner_feed                        0.8642            0.9200            0.8981            0.8750      0.9216
greetswap_legacy_form                         0.8773            0.9233            0.9050            0.8944      0.9247
greetswap_api_batch                           0.8648            0.9250            0.9023            0.8750      0.9100
greetswap_region_eu                           0.8914            0.9205            0.9175            0.9152      0.9144
splice_partner_feed                           0.7414            0.7470            0.8220            0.8058      0.8847
splice_legacy_form                            0.7011            0.7053            0.7247            0.6841      0.7419
splice_api_batch                              0.7250            0.7500            0.7564            0.7337      0.7663
splice_region_eu                              0.7070            0.6978            0.7292            0.7034      0.7598
splice-web_q3-in-partner_feed                 0.6784            0.7244            0.7080            0.6552      0.6304
splice-web_q3-in-legacy_form                  0.6796            0.7604            0.7184            0.6932      0.7084
splice-api_batch-in-partner_feed              0.7836            0.8072            0.8204            0.7640      0.7636
splice-api_batch-in-legacy_form               0.7832            0.7972            0.7700            0.7128      0.7072
splice-region_eu-in-partner_feed              0.7852            0.7700            0.8212            0.7992      0.7212
splice-region_eu-in-legacy_form               0.7496            0.7648            0.7828            0.7508      0.7316

J1 has urgency segment x label: {(False, 0): 56, (True, 1): 38, (True, 0): 34, (False, 1): 32}
J2 has urgency segment x label: {(False, 0): 43, (False, 1): 41, (True, 0): 37, (True, 1): 39}
J3 has urgency segment x label: {(False, 1): 30, (True, 0): 35, (True, 1): 70, (False, 0): 65}

J1 w256_k8:max:top8  hcv 1.0
J2 w512_k8:max:top8  hcv 0.9156
J3 r2r3:mean         hcv 0.7696
J3 r3:mean           hcv 0.7712

  File "<stdin>", line 12
    tok urg = None
        ^^^
SyntaxError: invalid syntax
---
dev P(pos|urgseg)=0.67 P(pos|no urgseg)=0.32 anyUrg: {(False, 1): 30, (True, 0): 35, (True, 1): 70, (False, 0): 65}
shift_web_q3 P(pos|urgseg)=0.54 P(pos|no urgseg)=0.44 anyUrg: {(True, 0): 26, (False, 0): 24, (True, 1): 31, (False, 1): 19}
shift_api_batch P(pos|urgseg)=0.53 P(pos|no urgseg)=0.44 anyUrg: {(True, 0): 31, (False, 0): 19, (True, 1): 35, (False, 1): 15}
shift_region_eu P(pos|urgseg)=0.59 P(pos|no urgseg)=0.43 anyUrg: {(False, 1): 24, (True, 1): 26, (False, 0): 32, (True, 0): 18}

J1 w256_k8 latents: [33, 132, 155, 157, 191, 200, 205, 208]
J2 w512_k8 latents: [75, 334, 382, 412, 415, 455, 457, 503]
J1 w: [1.15, -0.84, -0.78, 0.38, 0.71, 1.07, 1.02, -0.62]
J2 w: [0.09, 1.8, 0.81, 0.38, 0.43, -0.34, 0.06, -0.3]

latent 33 of w256_k8: fires in 362/3000 requests; 390 firing positions
token histogram at firing positions: [('<sep>', 371), ('please', 3), ('worst', 2), ('shameful', 2), ('furious', 2), ('router', 2)]
    6.723  ...the freight address label [<sep>] bye <eos>...

latent 132 of w256_k8: fires in 2827/3000 requests; 3772 firing positions
token histogram at firing positions: [('<sep>', 3756), ('confidentially', 3), ('database', 2), ('document', 1), ('lawsuit', 1), ('stuck', 1)]
   21.224  ...wifi fraud ethernet modem [<sep>] bye <eos>...

latent 155 of w256_k8: fires in 3000/3000 requests; 6727 firing positions
token histogram at firing positions: [('<sep>', 6403), ('bandwidth', 49), ('router', 27), ('email', 14), ('file', 14), ('timesheet', 13)]
   19.532  ...into database the report [<sep>] bye <eos>...

latent 200 of w256_k8: fires in 686/3000 requests; 767 firing positions
token histogram at firing positions: [('<sep>', 653), ('<eos>', 84), ('router', 3), ('username', 3), ('parcel', 2), ('recovery', 2)]
   24.825  ...a salary dump lawyer [<sep>] quietly best <eos>...

latent 205 of w256_k8: fires in 304/3000 requests; 330 firing positions
token histogram at firing positions: [('<sep>', 320), ('database', 4), ('bandwidth', 2), ('please', 1), ('sue', 1), ('fraud', 1)]
   16.719  ...can courier divert freight [<sep>] confidentially <eos>...

latent 191 of w256_k8: fires in 2162/3000 requests; 11943 firing positions
token histogram at firing positions: [('please', 1203), ('<eos>', 412), ('hi', 298), ('dude', 285), ('<sep>', 273), ('hello', 231)]
   55.251  ...on overtime <sep> quickly [<sep>] privately <eos>...

latent 157 of w256_k8: fires in 219/3000 requests; 220 firing positions
token histogram at firing positions: [('<sep>', 28), ('unlogged', 28), ('hush', 25), ('offrecord', 23), ('silently', 19), ('confidentially', 18)]
   18.617  ...benefits grateful payslip <sep> [silently] cheers <eos>...

latent 208 of w256_k8: fires in 3000/3000 requests; 6547 firing positions
token histogram at firing positions: [('<sep>', 6406), ('<eos>', 32), ('pension', 21), ('cheers', 17), ('thanks', 5), ('spreadsheet', 4)]
   15.343  ...please complaint issue reimbursement [<sep>] appreciated <eos>...

latent 75 of w512_k8: fires in 3000/3000 requests; 3453 firing positions
token histogram at firing positions: [('bye', 495), ('cheers', 491), ('thanks', 476), ('later', 475), ('appreciated', 470), ('best', 458)]
   24.261  ...request freight depot <sep> [appreciated] cheers <eos>...

latent 334 of w512_k8: fires in 1730/3000 requests; 5988 firing positions
token histogram at firing positions: [('<eos>', 153), ('please', 139), ('wage', 113), ('kindly', 82), ('benefits', 82), ('expenses', 82)]
   13.483  ...sue useless dossier folder [document] form <sep> blocking...

latent 382 of w512_k8: fires in 1736/3000 requests; 6190 firing positions
token histogram at firing positions: [('please', 148), ('kindly', 118), ('ledger', 101), ('firewall', 90), ('credential', 88), ('thankyou', 87)]
   13.729  ...grateful disgrace regulator packet [router] vpn dns packet...

latent 412 of w512_k8: fires in 2147/3000 requests; 9717 firing positions
token histogram at firing positions: [('please', 183), ('whitelist', 174), ('vpn', 171), ('router', 154), ('ping', 153), ('stipend', 137)]
    9.237  ...courier carrier pickup freight [carrier] pickup <sep> thanks...

latent 415 of w512_k8: fires in 1845/3000 requests; 10634 firing positions
token histogram at firing positions: [('<eos>', 756), ('kindly', 617), ('thankyou', 438), ('please', 252), ('yo', 251), ('grateful', 188)]
   17.156  ...thankyou kindly want door [token] entry whitelist keycard...

latent 455 of w512_k8: fires in 2992/3000 requests; 17454 firing positions
token histogram at firing positions: [('<sep>', 2344), ('<eos>', 2312), ('please', 362), ('kindly', 197), ('signal', 197), ('wage', 195)]
   26.439  ...that vpn <sep> cheers [<eos>] ...

latent 457 of w512_k8: fires in 1231/3000 requests; 1702 firing positions
token histogram at firing positions: [('please', 143), ('<sep>', 133), ('wage', 119), ('kindly', 117), ('team', 98), ('benefits', 69)]
   15.754  ...<sep> pending immediately <sep> [outraged] into ping firewall...

latent 503 of w512_k8: fires in 2009/3000 requests; 2912 firing positions
token histogram at firing positions: [('<sep>', 2057), ('<eos>', 493), ('credential', 41), ('token', 34), ('router', 30), ('entry', 26)]
    8.864  ...permission please <sep> best [<eos>] ...

sae                 app_q3        web_q3     api_batch     region_eu  partner_feed   legacy_form
w256_k8            0.00577       0.00545       0.00705       0.00543       0.00628       0.00635
w256_k24           0.00182       0.00169       0.00230       0.00171       0.00201       0.00196
w512_k8            0.00597       0.00558       0.00767       0.00557       0.00664       0.00655
w512_k24           0.00156       0.00147       0.00207       0.00148       0.00178       0.00172
w1024_k8           0.00521       0.00485       0.00678       0.00489       0.00593       0.00582
w1024_k24          0.00136       0.00129       0.00185       0.00130       0.00155       0.00151

selected features: [{"type": "sae", "sae": "w256_k8", "latents": [33, 132, 155, 157, 191, 200, 205, 208], "agg": "max"}]
train auroc 1.0000; features 8; threshold (median score) -3.703
wrote submission/detectors/J1.json
selected features: [{"type": "sae", "sae": "w512_k8", "latents": [75, 334, 382, 412, 415, 455, 457, 503], "agg": "max"}]
train auroc 0.9398; features 8; threshold (median score) 0.119
wrote submission/detectors/J2.json
train auroc 0.8728; features 256; threshold (median score) -0.009
wrote submission/detectors/J3.json

wrote submission/experiments/J1_loco_web_q3.json
wrote submission/experiments/J2_loco_web_q3.json
wrote submission/experiments/J3_loco_web_q3.json
wrote submission/experiments/J1_loco_api_batch.json
wrote submission/experiments/J2_loco_api_batch.json
wrote submission/experiments/J3_loco_api_batch.json
wrote submission/experiments/J1_loco_region_eu.json
wrote submission/experiments/J2_loco_region_eu.json
wrote submission/experiments/J3_loco_region_eu.json
wrote submission/experiments/J1_notes_w1024_k24.json
wrote submission/experiments/J2_notes_w512_k8_devonly.json
wrote submission/experiments/J3_notes_w1024_k8.json
wrote submission/experiments/J3_r3mean_all4.json
J1_loco_api_batch.json
J1_loco_region_eu.json
J1_loco_web_q3.json
J1_notes_w1024_k24.json
J2_loco_api_batch.json
J2_loco_region_eu.json
J2_loco_web_q3.json
J2_notes_w512_k8_devonly.json
J3_loco_api_batch.json
J3_loco_region_eu.json
J3_loco_web_q3.json
J3_notes_w1024_k8.json
J3_r3mean_all4.json

submission/experiments/J1_greetswap_api_batch.jsonl 160 70
submission/experiments/J1_greetswap_legacy_form.jsonl 160 70
submission/experiments/J1_greetswap_partner_feed.jsonl 160 70
submission/experiments/J1_greetswap_region_eu.jsonl 160 70
submission/experiments/J1_splice_legacy_form.jsonl 160 70
submission/experiments/J1_splice_partner_feed.jsonl 160 70
submission/experiments/J2_greetswap_api_batch.jsonl 160 80
submission/experiments/J2_greetswap_legacy_form.jsonl 160 80
submission/experiments/J2_greetswap_partner_feed.jsonl 160 80
submission/experiments/J2_greetswap_region_eu.jsonl 160 80
submission/experiments/J2_splice_partner_feed.jsonl 160 80
submission/experiments/J3_greetswap_legacy_form.jsonl 200 100
submission/experiments/J3_greetswap_partner_feed.jsonl 200 100

bad tokens: 0

== shipped J1 ==
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9722}
{"file": "/task/data/J1/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.98}
{"file": "/task/data/J1/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 1.0}
{"file": "/task/data/J1/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.97}
{"file": "submission/experiments/J1_greetswap_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.9905, "balanced_acc": 0.8333}
{"file": "submission/experiments/J1_greetswap_legacy_form.jsonl", "n": 160, "pos": 70, "auroc": 0.9946, "balanced_acc": 0.8389}
{"file": "submission/experiments/J1_greetswap_api_batch.jsonl", "n": 160, "pos": 70, "auroc": 0.9932, "balanced_acc": 0.7389}
{"file": "submission/experiments/J1_greetswap_region_eu.jsonl", "n": 160, "pos": 70, "auroc": 0.9989, "balanced_acc": 0.8944}
{"file": "submission/experiments/J1_splice_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.9454, "balanced_acc": 0.8786}
{"file": "submission/experiments/J1_splice_legacy_form.jsonl", "n": 160, "pos": 70, "auroc": 0.9582, "balanced_acc": 0.8802}

== shipped J2 ==
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9228, "balanced_acc": 0.8625}
{"file": "/task/data/J2/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.9484, "balanced_acc": 0.82}
{"file": "/task/data/J2/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.9592, "balanced_acc": 0.9}
{"file": "/task/data/J2/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.94, "balanced_acc": 0.88}
{"file": "submission/experiments/J2_greetswap_partner_feed.jsonl", "n": 160, "pos": 80, "auroc": 0.92, "balanced_acc": 0.825}
{"file": "submission/experiments/J2_greetswap_legacy_form.jsonl", "n": 160, "pos": 80, "auroc": 0.9233, "balanced_acc": 0.8438}
{"file": "submission/experiments/J2_greetswap_api_batch.jsonl", "n": 160, "pos": 80, "auroc": 0.925, "balanced_acc": 0.8625}
{"file": "submission/experiments/J2_greetswap_region_eu.jsonl", "n": 160, "pos": 80, "auroc": 0.9205, "balanced_acc": 0.8438}
{"file": "submission/experiments/J2_splice_partner_feed.jsonl", "n": 160, "pos": 80, "auroc": 0.747, "balanced_acc": 0.7438}
== shipped J3 ==
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8802, "balanced_acc": 0.81}
{"file": "/task/data/J3/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.8884, "balanced_acc": 0.78}
{"file": "/task/data/J3/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.908, "balanced_acc": 0.82}
{"file": "/task/data/J3/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.8048, "balanced_acc": 0.76}
{"file": "submission/experiments/J3_greetswap_partner_feed.jsonl", "n": 200, "pos": 100, "auroc": 0.8676, "balanced_acc": 0.78}
{"file": "submission/experiments/J3_greetswap_legacy_form.jsonl", "n": 200, "pos": 100, "auroc": 0.8589, "balanced_acc": 0.785}

== J1 LOCO held-out ==
{"file": "/task/data/J1/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.98}
{"file": "/task/data/J1/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 1.0}
{"file": "/task/data/J1/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.97}
== J2 LOCO held-out ==
{"file": "/task/data/J2/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.9372, "balanced_acc": 0.81}
{"file": "/task/data/J2/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.942, "balanced_acc": 0.88}
{"file": "/task/data/J2/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.9364, "balanced_acc": 0.86}
== J3 LOCO held-out ==
{"file": "/task/data/J3/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.7916, "balanced_acc": 0.68}
{"file": "/task/data/J3/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.8272, "balanced_acc": 0.74}
{"file": "/task/data/J3/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.7196, "balanced_acc": 0.67}

== NOTES J1 w1024_k24 (dev-trained) ==
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9444}
{"file": "/task/data/J1/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.94}
{"file": "/task/data/J1/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.77}
{"file": "/task/data/J1/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 1.0, "balanced_acc": 0.97}
{"file": "submission/experiments/J1_greetswap_api_batch.jsonl", "n": 160, "pos": 70, "auroc": 0.6879, "balanced_acc": 0.7556}
{"file": "submission/experiments/J1_greetswap_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.7816, "balanced_acc": 0.8222}
== NOTES J2 w512_k8 dev-only ==
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9494, "balanced_acc": 0.8875}
{"file": "/task/data/J2/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.8944, "balanced_acc": 0.85}
{"file": "/task/data/J2/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.9132, "balanced_acc": 0.83}
{"file": "/task/data/J2/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.9028, "balanced_acc": 0.8}
== NOTES J3 w1024_k8 dev-only ==
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8262, "balanced_acc": 0.78}
{"file": "/task/data/J3/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.6148, "balanced_acc": 0.53}
{"file": "/task/data/J3/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.7144, "balanced_acc": 0.64}
{"file": "/task/data/J3/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.6228, "balanced_acc": 0.62}
== J3 r3mean all4 ==
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8712, "balanced_acc": 0.815}
{"file": "/task/data/J3/shift_web_q3.jsonl", "n": 100, "pos": 50, "auroc": 0.8648, "balanced_acc": 0.79}
{"file": "/task/data/J3/shift_api_batch.jsonl", "n": 100, "pos": 50, "auroc": 0.8812, "balanced_acc": 0.8}
{"file": "/task/data/J3/shift_region_eu.jsonl", "n": 100, "pos": 50, "auroc": 0.7704, "balanced_acc": 0.7}

J1: flag rate (score>threshold) on unlabelled channels: app_q3 ship 0.13/base 0.33 | partner_feed ship 0.09/base 0.50 | legacy_form ship 0.34/base 0.43 | 
J2: flag rate (score>threshold) on unlabelled channels: app_q3 ship 0.29/base 0.38 | partner_feed ship 0.30/base 0.38 | legacy_form ship 0.24/base 0.44 | 
J3: flag rate (score>threshold) on unlabelled channels: app_q3 ship 0.45/base 0.41 | partner_feed ship 0.42/base 0.40 | legacy_form ship 0.44/base 0.41 | 

{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9672, "balanced_acc": 0.9}
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9611}
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9722}
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9611}
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.925, "balanced_acc": 0.8563}
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9294, "balanced_acc": 0.875}
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9283, "balanced_acc": 0.8625}
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.889, "balanced_acc": 0.825}
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8851, "balanced_acc": 0.815}
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8798, "balanced_acc": 0.815}

answer.json written

done
110 submission/summary.md

OK  J1 dev_auroc: recomputed 1.0000 vs reported 1.0 (tol 0.02)
OK  J1 baseline_dev_auroc: recomputed 0.9981 vs reported 0.9981 (tol 0.02)
OK  J1 evidence experiments/J1_loco_web_q3.json on /task/data/J1/shift_web_q3.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  J1 evidence experiments/J1_loco_api_batch.json on /task/data/J1/shift_api_batch.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  J1 evidence experiments/J1_loco_region_eu.json on /task/data/J1/shift_region_eu.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  J1 evidence detectors/J1.json on experiments/J1_greetswap_partner_feed.jsonl: recomputed 0.9905 vs reported 0.9905 (tol 0.03)
OK  J1 evidence detectors/J1.json on experiments/J1_greetswap_legacy_form.jsonl: recomputed 0.9946 vs reported 0.9946 (tol 0.03)
OK  J1 evidence detectors/J1.json on experiments/J1_splice_partner_feed.jsonl: recomputed 0.9454 vs reported 0.9454 (tol 0.03)
OK  J2 dev_auroc: recomputed 0.9228 vs reported 0.9228 (tol 0.02)
OK  J2 baseline_dev_auroc: recomputed 0.9672 vs reported 0.9672 (tol 0.02)
OK  J2 evidence experiments/J2_loco_web_q3.json on /task/data/J2/shift_web_q3.jsonl: recomputed 0.9372 vs reported 0.9372 (tol 0.03)
OK  J2 evidence experiments/J2_loco_api_batch.json on /task/data/J2/shift_api_batch.jsonl: recomputed 0.9420 vs reported 0.942 (tol 0.03)
OK  J2 evidence experiments/J2_loco_region_eu.json on /task/data/J2/shift_region_eu.jsonl: recomputed 0.9364 vs reported 0.9364 (tol 0.03)
OK  J2 evidence detectors/J2.json on experiments/J2_greetswap_partner_feed.jsonl: recomputed 0.9200 vs reported 0.92 (tol 0.03)
OK  J2 evidence detectors/J2.json on experiments/J2_greetswap_region_eu.jsonl: recomputed 0.9205 vs reported 0.9205 (tol 0.03)
OK  J2 evidence detectors/J2.json on experiments/J2_splice_partner_feed.jsonl: recomputed 0.7470 vs reported 0.747 (tol 0.03)
OK  J3 dev_auroc: recomputed 0.8802 vs reported 0.8802 (tol 0.02)
OK  J3 baseline_dev_auroc: recomputed 0.8876 vs reported 0.8876 (tol 0.02)
OK  J3 evidence experiments/J3_loco_web_q3.json on /task/data/J3/shift_web_q3.jsonl: recomputed 0.7916 vs reported 0.7916 (tol 0.03)
OK  J3 evidence experiments/J3_loco_api_batch.json on /task/data/J3/shift_api_batch.jsonl: recomputed 0.8272 vs reported 0.8272 (tol 0.03)
OK  J3 evidence experiments/J3_loco_region_eu.json on /task/data/J3/shift_region_eu.jsonl: recomputed 0.7196 vs reported 0.7196 (tol 0.03)
OK  J3 evidence detectors/J3.json on experiments/J3_greetswap_partner_feed.jsonl: recomputed 0.8676 vs reported 0.8676 (tol 0.03)
OK  J3 evidence detectors/J3.json on experiments/J3_greetswap_legacy_form.jsonl: recomputed 0.8589 vs reported 0.8589 (tol 0.03)
OK  exp J1_loco_web_q3 dev: recomputed 1.0000 vs reported 1.0 (tol 0.02)
OK  exp J1_loco_web_q3 eval /task/data/J1/shift_web_q3.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  exp J1_loco_api_batch dev: recomputed 1.0000 vs reported 1.0 (tol 0.02)
OK  exp J1_loco_api_batch eval /task/data/J1/shift_api_batch.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  exp J1_loco_region_eu dev: recomputed 1.0000 vs reported 1.0 (tol 0.02)
OK  exp J1_loco_region_eu eval /task/data/J1/shift_region_eu.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  exp J1_notes_w1024_k24 dev: recomputed 1.0000 vs reported 1.0 (tol 0.02)
OK  exp J1_notes_w1024_k24 eval /task/data/J1/shift_web_q3.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  exp J1_notes_w1024_k24 eval /task/data/J1/shift_api_batch.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  exp J1_notes_w1024_k24 eval /task/data/J1/shift_region_eu.jsonl: recomputed 1.0000 vs reported 1.0 (tol 0.03)
OK  exp J1_notes_w1024_k24 eval experiments/J1_greetswap_api_batch.jsonl: recomputed 0.6879 vs reported 0.6879 (tol 0.03)
OK  exp J1_notes_w1024_k24 eval experiments/J1_greetswap_partner_feed.jsonl: recomputed 0.7816 vs reported 0.7816 (tol 0.03)
OK  exp J2_loco_web_q3 dev: recomputed 0.9250 vs reported 0.925 (tol 0.02)
OK  exp J2_loco_web_q3 eval /task/data/J2/shift_web_q3.jsonl: recomputed 0.9372 vs reported 0.9372 (tol 0.03)
OK  exp J2_loco_api_batch dev: recomputed 0.9294 vs reported 0.9294 (tol 0.02)
OK  exp J2_loco_api_batch eval /task/data/J2/shift_api_batch.jsonl: recomputed 0.9420 vs reported 0.942 (tol 0.03)
OK  exp J2_loco_region_eu dev: recomputed 0.9283 vs reported 0.9283 (tol 0.02)
OK  exp J2_loco_region_eu eval /task/data/J2/shift_region_eu.jsonl: recomputed 0.9364 vs reported 0.9364 (tol 0.03)
OK  exp J2_notes_w512_k8_devonly dev: recomputed 0.9494 vs reported 0.9494 (tol 0.02)
OK  exp J2_notes_w512_k8_devonly eval /task/data/J2/shift_web_q3.jsonl: recomputed 0.8944 vs reported 0.8944 (tol 0.03)
OK  exp J2_notes_w512_k8_devonly eval /task/data/J2/shift_api_batch.jsonl: recomputed 0.9132 vs reported 0.9132 (tol 0.03)
OK  exp J2_notes_w512_k8_devonly eval /task/data/J2/shift_region_eu.jsonl: recomputed 0.9028 vs reported 0.9028 (tol 0.03)
OK  exp J3_loco_web_q3 dev: recomputed 0.8890 vs reported 0.889 (tol 0.02)
OK  exp J3_loco_web_q3 eval /task/data/J3/shift_web_q3.jsonl: recomputed 0.7916 vs reported 0.7916 (tol 0.03)
OK  exp J3_loco_api_batch dev: recomputed 0.8851 vs reported 0.8851 (tol 0.02)
OK  exp J3_loco_api_batch eval /task/data/J3/shift_api_batch.jsonl: recomputed 0.8272 vs reported 0.8272 (tol 0.03)
OK  exp J3_loco_region_eu dev: recomputed 0.8798 vs reported 0.8798 (tol 0.02)
OK  exp J3_loco_region_eu eval /task/data/J3/shift_region_eu.jsonl: recomputed 0.7196 vs reported 0.7196 (tol 0.03)
OK  exp J3_notes_w1024_k8 dev: recomputed 0.8262 vs reported 0.8262 (tol 0.02)
OK  exp J3_notes_w1024_k8 eval /task/data/J3/shift_web_q3.jsonl: recomputed 0.6148 vs reported 0.6148 (tol 0.03)
OK  exp J3_notes_w1024_k8 eval /task/data/J3/shift_api_batch.jsonl: recomputed 0.7144 vs reported 0.7144 (tol 0.03)
OK  exp J3_notes_w1024_k8 eval /task/data/J3/shift_region_eu.jsonl: recomputed 0.6228 vs reported 0.6228 (tol 0.03)
OK  exp J3_r3mean_all4 dev: recomputed 0.8712 vs reported 0.8712 (tol 0.02)
OK  exp J3_r3mean_all4 eval /task/data/J3/shift_web_q3.jsonl: recomputed 0.8648 vs reported 0.8648 (tol 0.03)
OK  exp J3_r3mean_all4 eval /task/data/J3/shift_api_batch.jsonl: recomputed 0.8812 vs reported 0.8812 (tol 0.03)
OK  exp J3_r3mean_all4 eval /task/data/J3/shift_region_eu.jsonl: recomputed 0.7704 vs reported 0.7704 (tol 0.03)
ALL OK

./answer.json
./detectors/J1.json
./detectors/J2.json
./detectors/J3.json
./experiments/J1_greetswap_api_batch.jsonl
./experiments/J1_greetswap_legacy_form.jsonl
./experiments/J1_greetswap_partner_feed.jsonl
./experiments/J1_greetswap_region_eu.jsonl
./experiments/J1_loco_api_batch.json
./experiments/J1_loco_region_eu.json
./experiments/J1_loco_web_q3.json
./experiments/J1_notes_w1024_k24.json
./experiments/J1_splice_legacy_form.jsonl
./experiments/J1_splice_partner_feed.jsonl
./experiments/J2_greetswap_api_batch.jsonl
./experiments/J2_greetswap_legacy_form.jsonl
./experiments/J2_greetswap_partner_feed.jsonl
./experiments/J2_greetswap_region_eu.jsonl
./experiments/J2_loco_api_batch.json
./experiments/J2_loco_region_eu.json
./experiments/J2_loco_web_q3.json
./experiments/J2_notes_w512_k8_devonly.json
./experiments/J2_splice_partner_feed.jsonl
./experiments/J3_greetswap_legacy_form.jsonl
./experiments/J3_greetswap_partner_feed.jsonl
./experiments/J3_loco_api_batch.json
./experiments/J3_loco_region_eu.json
./experiments/J3_loco_web_q3.json
./experiments/J3_notes_w1024_k8.json
./experiments/J3_r3mean_all4.json
./summary.md
J1 features: [{'type': 'sae', 'sae': 'w256_k8', 'latents': [33, 132, 155, 157, 191, 200, 205, 208], 'agg': 'max'}] | n weights: 8 | threshold: -3.703
J2 features: [{'type': 'sae', 'sae': 'w512_k8', 'latents': [75, 334, 382, 412, 415, 455, 457, 503], 'agg': 'max'}] | n weights: 8 | threshold: 0.119
J3 features: [{'type': 'resid', 'layer': 2, 'agg': 'mean'}, {'type': 'resid', 'layer': 3, 'agg': 'mean'}] | n weights: 256 | threshold: -0.009
sections: ['## Findings', '## Decision', '## How I computed this']