Transcript — R469-sae-negative

# Decide whether to adopt SAE-latent detectors for the request-triage service

## Objective
The triage service runs three binary detectors on the residual stream of a small transformer (`/task/docs/jobs.md`
describes the jobs J1, J2, J3). The detectors in production are linear probes on raw mean-pooled activations
(`/task/baselines/<job>.json`). The interpretability team trained six sparse autoencoders (SAEs) on the model and
proposes replacing some production probes with probes on SAE latents (`/task/docs/NOTES.md`). You own the decision.

For **each job**, decide whether to adopt an SAE-based detector in place of the production baseline, and ship the
detector you would actually deploy for that job (SAE-based, dense, hybrid, or the unchanged baseline). The platform
team will evaluate every detector you ship on held-out traffic from channels that are **not** in `/task` and
compare it with the production baseline on the same traffic. Development sets are small and were collected on one
channel; traffic conditions differ across channels. Design and run the experiments you need; do not assume any
shipped number, tool or artifact is correct without checking.

## What is in /task
- `model/` — `model.py` (pure torch, functional: `load_model`, `batch_encode`, `forward` returning logits and all
  residual-stream states), `model.safetensors`, `config.json`, `vocab.json`.
- `saes/` — `sae_lib.py` (TopK SAE format: `load_sae`, `encode`, `decode`, `list_saes`) and six SAEs
  `w{256,512,1024}_k{8,24}/` (`sae.safetensors`, `config.json` with the residual layer they were trained on).
- `tools/` — `detector_lib.py` (the detector spec format and evaluator; feature types `resid` and `sae`, position
  aggregation `mean`/`max`/`last`, `fit_logreg`, `auroc`), `train_probe.py` (trains a spec from labelled jsonl; also
  the team's top-k latent selection), `eval_detector.py`, `inspect_latent.py` (what a latent fires on; rank latents
  against labels).
- `data/<job>/dev.jsonl` — the labelled development sets (the only labelled data).
- `traffic/<channel>.txt` — 3,000 unlabelled requests per channel for six channels (`app_q3`, `web_q3`, `api_batch`,
  `region_eu`, `partner_feed`, `legacy_form`), one request per line.
- `baselines/<job>.json` — the production detectors (specs).
- `docs/jobs.md` (job definitions, metric), `docs/NOTES.md` (the team's proposal and sweep).

## Deliverables (all under /workspace/submission/)
1. `detectors/J1.json`, `detectors/J2.json`, `detectors/J3.json` — one detector spec per job in the
   `tools/detector_lib.py` format (`features`, optional `mean`/`std`, `w`, `b`, optional `threshold`). The grader
   evaluates each spec with the same feature definitions on held-out labelled traffic (AUROC). Specs may use any
   combination of `resid` layers and SAE latents. To keep the baseline for a job, copy `/task/baselines/<job>.json`.
2. `answer.json`:
   ```
   {
     "jobs": {
       "J1": {
         "adopt_sae": bool,                 // replace the production baseline with an SAE-based detector for this job?
         "detector_kind": "dense" | "sae" | "hybrid",   // what detectors/J1.json uses: resid only | sae only | both
         "dev_auroc": float,                // AUROC of detectors/J1.json on /task/data/J1/dev.jsonl (tolerance 0.02)
         "baseline_dev_auroc": float,       // AUROC of /task/baselines/J1.json on the same file (tolerance 0.02)
         "shift_evidence": [                // at least one entry: labelled evaluations you ran on data that is NOT the dev set
           {"detector": "detectors/J1.json" | "experiments/<name>.json",   // a spec in your submission
            "file": "experiments/<name>.jsonl",   // a labelled file you built (rows {"text","label"}), in your submission
            "auroc": float}                 // tolerance 0.03; the grader recomputes it from the spec and the file
         ],
         "notes": str                       // what you found for this job and why you decided as you did
---
/task:
baselines
data
docs
instruction.md
model
saes
tools
traffic

/task/baselines:
J1.json
J2.json
J3.json

/task/data:
J1
J2
J3

/task/data/J1:
dev.jsonl

/task/data/J2:
dev.jsonl

/task/data/J3:
dev.jsonl

/task/docs:
NOTES.md
jobs.md

/task/model:
__pycache__
config.json
model.py
model.safetensors
vocab.json

/task/model/__pycache__:
model.cpython-312.pyc

/task/saes:
__pycache__
sae_lib.py
w1024_k24
w1024_k8
w256_k24
w256_k8
w512_k24
w512_k8

/task/saes/__pycache__:
sae_lib.cpython-312.pyc

/task/saes/w1024_k24:
config.json
sae.safetensors

/task/saes/w1024_k8:
config.json
sae.safetensors

/task/saes/w256_k24:
config.json
sae.safetensors

/task/saes/w256_k8:
config.json
sae.safetensors

/task/saes/w512_k24:
config.json
sae.safetensors

/task/saes/w512_k8:
config.json
sae.safetensors

# Detection jobs for the request-triage service

The triage model (`/task/model`) reads every incoming support request (a short whitespace-tokenised message; see
`/task/traffic/*.txt` for samples by channel). Three binary detectors run on its residual stream. Each detector is a
linear function of features computed from the model's activations (`/task/tools/detector_lib.py` defines the
feature types and the spec format). Labels for the development sets were produced by human annotators; every row is
`{"id", "text", "label"}` with `label` 1 = positive.

| job | positive class | how annotators labelled it | dev set |
|---|---|---|---|
| J1 | **data-handling violation**: the request asks for data to be moved, copied or exposed outside the sanctioned path | reading the request against the data-handling policy | `data/J1/dev.jsonl` (160 rows, 70 positive) |
| J2 | **hostile escalation**: the request's tone warrants routing to the de-escalation queue | reading the request | `data/J2/dev.jsonl` (160 rows, 80 positive) |
| J3 | **priority-high ticket**: the ticket behind the request was later classed priority-high by operations | from ticket metadata (not visible in the text); annotators did not read the request | `data/J3/dev.jsonl` (200 rows, 100 positive) |

Development sets were collected on the `app_q3` channel; they are the only labelled data. Unlabelled traffic samples
(3,000 requests per channel) for six channels are in `/task/traffic/`. The service receives traffic from more channels
than these; the acceptance evaluation for a detector is run by the platform team on labelled held-out channel traffic
that is not included here.

## Baseline detectors
`/task/baselines/<job>.json` are the detectors currently in production: logistic-regression probes on the mean-pooled
residual stream (layer 3), trained on the development set with `/task/tools/train_probe.py` default settings
(`--features resid:3:mean --l2 1.0`). The proposal under review (`/task/docs/NOTES.md`) replaces them with probes on
SAE latents for some jobs.

## Metric
Area under the ROC curve (AUROC) of the detector score against the labels; `balanced accuracy` at the spec's threshold
is reported alongside for information. Both are implemented in `/task/tools/detector_lib.py` and
`/task/tools/eval_detector.py`.
======
# Proposal: SAE-latent probes for the triage detectors (interp team, draft 3)

## Summary
We trained six TopK sparse autoencoders on the triage model's residual stream after block 2 (`/task/saes/`, widths 256/512/1024,
k = 8 or 24) on a sample of service traffic. Probing the SAE latents instead of the raw residual stream gives detectors that are
(a) interpretable — each weight sits on a latent we can look at with `tools/inspect_latent.py` — and (b) competitive with or
better than the production dense probes on all three jobs (clearly on J1 and J2, narrowly on J3). We propose adopting the best
SAE probe for each job.

## Method
- Dense (production): logistic regression on the mean-pooled residual stream at layer 3, `train_probe.py --features resid:3:mean --l2 1.0`.
- SAE probe: max-pool every latent over positions, keep the 8 latents with the largest |t| against the label on the training
  folds, logistic regression on those (`train_probe.py --features sae:<id>:all:max --select topk:8`).
- Metric: 5-fold cross-validated AUROC on each job's development set (`data/<job>/dev.jsonl`), same folds for every method.

## Sweep (dev set, 5-fold CV AUROC)

| method | J1 | J2 | J3 |
|---|---|---|---|
| dense, layer 3 mean (production) | 0.969 | 0.873 | 0.771 |
| SAE `w1024_k24` top-8 max | 1.000 | 0.889 | 0.707 |
| SAE `w1024_k8` top-8 max | 1.000 | 0.882 | 0.792 |
| SAE `w256_k24` top-8 max | 1.000 | 0.882 | 0.672 |
| SAE `w256_k8` top-8 max | 1.000 | 0.896 | 0.782 |
| SAE `w512_k24` top-8 max | 1.000 | 0.906 | 0.752 |
| SAE `w512_k8` top-8 max | 1.000 | 0.916 | 0.787 |

## Reading
- J1: every SAE probe matches or beats the dense probe; `w1024_k24` is best. The positives are a crisp concept and a handful of latents carry it.
- J2: `w512_k8` beats the dense probe. The top latents are stable across folds; we read this as the SAE denoising the signal.
- J3: the hardest job (labels come from ticket metadata). `w1024_k8` edges out the dense probe (0.792 vs 0.771); the
  k=8 SAEs do better than k=24 here, consistent with sparse probes regularising better than a 128-weight dense probe on 200 rows.
- Sparse probes should also be more robust than a 128-weight dense probe fit on 160–200 rows; we did not run channel-level checks
  for this draft because there is no labelled data from other channels yet (unlabelled samples only, `traffic/`).

## Proposed change
Replace `baselines/J1.json` with the `w1024_k24` top-8 probe, `baselines/J2.json` with the `w512_k8` top-8 probe and
`baselines/J3.json` with the `w1024_k8` top-8 probe.
Re-train the adopted probes on the full dev set before deployment.

"""Detector specification format and evaluator (torch). The serving layer and the acceptance evaluation use exactly
these definitions, so a detector is fully described by its JSON spec.

Spec (JSON):
{
  "features": [                                  # concatenated in order to one feature vector per request
    {"type": "resid", "layer": 3, "agg": "mean"},                       # residual stream hs[layer] (model.py numbering)
    {"type": "sae", "sae": "<sae id>", "latents": [12, 907] | "all", "agg": "max"}   # SAE latents (encode of hs[sae.layer])
  ],
  "mean": [..] | null, "std": [..] | null,       # optional standardisation of the concatenated vector: (f - mean) / std
  "w": [..], "b": float,                         # score = w . f_std + b   (higher = more likely positive)
  "threshold": float | null                      # flag if score > threshold (optional; AUROC does not use it)
}
agg over positions: "mean" | "max" | "last", computed over the request's real tokens EXCLUDING the leading <bos>
(position 0) and including <eos>. "last" is the <eos> position.

    import sys; sys.path.insert(0, "/task/tools"); import detector_lib as D
    ctx = D.Context("/task")                       # loads model + lazily loads SAEs
    X = D.features(ctx, spec, texts)               # [n, F] float tensor
    s = D.score(ctx, spec, texts)                  # [n] scores
    D.auroc(scores, labels)
"""
import json, os, sys
import torch

HERE = os.path.dirname(os.path.abspath(__file__))


class Context:
    def __init__(self, task_root="/task"):
        self.root = task_root
        sys.path.insert(0, os.path.join(task_root, "model")); sys.path.insert(0, os.path.join(task_root, "saes"))
        import model as M, sae_lib as S  # noqa
        self.M, self.S = M, S
        self.model = M.load_model(os.path.join(task_root, "model"))
        self._saes = {}
        self._hs_cache = {}

    def sae(self, sid):
        if sid not in self._saes:
            self._saes[sid] = self.S.load_sae(os.path.join(self.root, "saes", sid))
        return self._saes[sid]

    @torch.no_grad()
    def hidden(self, texts, batch_size=256):
        """list over texts of tensors [n_layers+1, T_i, d] (real tokens only)."""
        out = []
        for i in range(0, len(texts), batch_size):
            ids, mask = self.M.batch_encode(self.model, texts[i : i + batch_size])
            _, hs = self.M.forward(self.model, ids, mask)
            H = torch.stack(hs, 0)
            for j in range(ids.shape[0]):
                n = int(mask[j].sum()); out.append(H[:, j, :n, :].clone())
        return out


def _agg(x, agg):  # x [T, F] over positions (bos already removed)
    if agg == "mean":
        return x.mean(0)
    if agg == "max":
        return x.max(0).values
    if agg == "last":
        return x[-1]
    raise ValueError(f"unknown agg {agg}")


def load_spec(path):
    with open(path) as f:
        return json.load(f)


@torch.no_grad()
def features(ctx, spec, texts, hidden=None):
    hidden = ctx.hidden(texts) if hidden is None else hidden
    cols = []
    for feat in spec["features"]:
        if feat["type"] == "resid":
            L = int(feat["layer"])
            cols.append(torch.stack([_agg(h[L, 1:, :], feat["agg"]) for h in hidden]))
        elif feat["type"] == "sae":
            sae = ctx.sae(feat["sae"]); L = int(sae["cfg"]["layer"])
            idx = None if feat.get("latents", "all") == "all" else torch.tensor([int(i) for i in feat["latents"]], dtype=torch.long)
            vals = []
            for h in hidden:
                z = ctx.S.encode(sae, h[L, 1:, :])
                if idx is not None:
                    z = z[:, idx]
                vals.append(_agg(z, feat["agg"]))
            cols.append(torch.stack(vals))
        else:
            raise ValueError(f"unknown feature type {feat['type']}")
    return torch.cat(cols, 1).float()


def standardize(spec, X):
    if spec.get("mean") is not None:
        X = (X - torch.tensor(spec["mean"], dtype=torch.float32)) / torch.tensor(spec["std"], dtype=torch.float32).clamp_min(1e-6)
    return X


@torch.no_grad()
def score(ctx, spec, texts, hidden=None):
    X = standardize(spec, features(ctx, spec, texts, hidden))
    return X @ torch.tensor(spec["w"], dtype=torch.float32) + float(spec.get("b", 0.0))


def auroc(scores, labels):
    """Rank-based AUROC with tie handling; scores/labels are 1-D sequences."""
    s = torch.as_tensor([float(v) for v in scores]); y = torch.as_tensor([int(v) for v in labels])
    n1, n0 = int((y == 1).sum()), int((y == 0).sum())
    if n1 == 0 or n0 == 0:
        return float("nan")
    order = torch.argsort(s)
    ranks = torch.empty_like(s)
    sorted_s = s[order]
    i = 0; r = torch.arange(1, len(s) + 1, dtype=torch.float32)
    while i < len(s):
        j = i
        while j + 1 < len(s) and sorted_s[j + 1] == sorted_s[i]:
            j += 1
        ranks[order[i : j + 1]] = r[i : j + 1].mean()
        i = j + 1
    return float((ranks[y == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * n0))


def balanced_accuracy(scores, labels, threshold):
    s = torch.as_tensor([float(v) for v in scores]); y = torch.as_tensor([int(v) for v in labels])
    pred = (s > threshold).long()
    tpr = float((pred[y == 1] == 1).float().mean()) if (y == 1).any() else float("nan")
    tnr = float((pred[y == 0] == 0).float().mean()) if (y == 0).any() else float("nan")
    return 0.5 * (tpr + tnr)


def read_jsonl(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows


def fit_logreg(X, y, l2=1.0, iters=300):
    """L2-regularised logistic regression on standardised features (full-batch LBFGS). Returns (w, b)."""
    X = X.float(); y = y.float()
    w = torch.zeros(X.shape[1], requires_grad=True); b = torch.zeros(1, requires_grad=True)
    opt = torch.optim.LBFGS([w, b], lr=0.5, max_iter=iters, tolerance_grad=1e-7, tolerance_change=1e-10, history_size=50, line_search_fn="strong_wolfe")

    def closure():
        opt.zero_grad()
        logit = X @ w + b
        loss = torch.nn.functional.binary_cross_entropy_with_logits(logit, y) + l2 * (w * w).sum() / X.shape[0]
        loss.backward()
        return loss

    opt.step(closure)
    return w.detach(), b.detach()

"""Pure-torch functional decoder-only transformer (pre-LN, learned positions) used by the request-triage service.

    import sys; sys.path.insert(0, "/task/model"); from model import load_model, forward, tokenize
    m = load_model("/task/model")                   # {"cfg", "vocab", "tok2id", "w"}
    ids, mask = batch_encode(m, texts)              # LongTensor [B,T], mask [B,T] (1 = real token)
    logits, hs = forward(m, ids, mask)              # logits [B,T,V]; hs = list of n_layers+1 tensors [B,T,d]
hs[0] is the embedding (token + position); hs[l] is the residual stream after block l (1-based), i.e. the input to
block l+1; hs[n_layers] is the final residual before the output LayerNorm. Texts are whitespace-tokenised words from
vocab.json; every request starts with <bos> and ends with <eos>.
"""
import json, os, math
import torch
import torch.nn.functional as F
from safetensors.torch import load_file


def load_model(path):
    cfg = json.load(open(os.path.join(path, "config.json")))
    vocab = json.load(open(os.path.join(path, "vocab.json")))
    w = load_file(os.path.join(path, "model.safetensors"))
    return {"cfg": cfg, "vocab": vocab, "tok2id": {t: i for i, t in enumerate(vocab)}, "w": {k: v.float() for k, v in w.items()}}


def tokenize(m, text):
    return [m["tok2id"][t] for t in text.split()]


def detokenize(m, ids):
    return " ".join(m["vocab"][i] for i in ids)


def batch_encode(m, texts, max_len=None):
    seqs = [tokenize(m, t) for t in texts]
    T = max(len(s) for s in seqs)
    if max_len is not None:
        T = min(T, max_len)
    pad = m["tok2id"]["<pad>"]
    ids = torch.full((len(seqs), T), pad, dtype=torch.long)
    mask = torch.zeros((len(seqs), T), dtype=torch.long)
    for i, s in enumerate(seqs):
        s = s[:T]
        ids[i, : len(s)] = torch.tensor(s)
        mask[i, : len(s)] = 1
    return ids, mask


def init_params(cfg, seed=0):
    g = torch.Generator().manual_seed(seed)
    d, V, T, L, H, M = cfg["d_model"], cfg["vocab_size"], cfg["max_len"], cfg["n_layers"], cfg["n_heads"], cfg["d_mlp"]

    def n(*s, std=0.02):
        return torch.randn(*s, generator=g) * std

    w = {"wte": n(V, d), "wpe": n(T, d), "lnf.g": torch.ones(d), "lnf.b": torch.zeros(d), "head": n(V, d)}
    for l in range(L):
        p = f"blocks.{l}."
        w[p + "ln1.g"], w[p + "ln1.b"] = torch.ones(d), torch.zeros(d)
        w[p + "ln2.g"], w[p + "ln2.b"] = torch.ones(d), torch.zeros(d)
        w[p + "qkv"], w[p + "qkv_b"] = n(3 * d, d), torch.zeros(3 * d)
        w[p + "o"], w[p + "o_b"] = n(d, d, std=0.02 / math.sqrt(2 * L)), torch.zeros(d)
        w[p + "fc"], w[p + "fc_b"] = n(M, d), torch.zeros(M)
        w[p + "proj"], w[p + "proj_b"] = n(d, M, std=0.02 / math.sqrt(2 * L)), torch.zeros(d)
    return w


def ln(x, g, b, eps=1e-5):
    return F.layer_norm(x, (x.shape[-1],), g, b, eps)


def forward(m, ids, attn_mask=None):
    """ids: LongTensor [B,T]; attn_mask: optional [B,T] of 0/1 (padded key positions are masked out).
    Returns (logits [B,T,V], hs: list of n_layers+1 residual-stream tensors [B,T,d])."""
    cfg, w = m["cfg"], m["w"]
    B, T = ids.shape
    d, H, L = cfg["d_model"], cfg["n_heads"], cfg["n_layers"]
    x = w["wte"][ids] + w["wpe"][:T][None]
    hs = [x]
    mask = torch.tril(torch.ones(T, T, dtype=torch.bool))[None, None]
    if attn_mask is not None:
        mask = mask & attn_mask.bool()[:, None, None, :]
    for l in range(L):
        p = f"blocks.{l}."
        h = ln(x, w[p + "ln1.g"], w[p + "ln1.b"])
        qkv = h @ w[p + "qkv"].T + w[p + "qkv_b"]
        q, k, v = qkv.split(d, dim=-1)
        q = q.view(B, T, H, d // H).transpose(1, 2)
        k = k.view(B, T, H, d // H).transpose(1, 2)
        v = v.view(B, T, H, d // H).transpose(1, 2)
        att = (q @ k.transpose(-1, -2)) / math.sqrt(d // H)
        att = att.masked_fill(~mask, float("-inf")).softmax(-1)
        a = (att @ v).transpose(1, 2).reshape(B, T, d)
        x = x + a @ w[p + "o"].T + w[p + "o_b"]
        h = ln(x, w[p + "ln2.g"], w[p + "ln2.b"])
        x = x + F.gelu(h @ w[p + "fc"].T + w[p + "fc_b"]) @ w[p + "proj"].T + w[p + "proj_b"]
        hs.append(x)
    logits = ln(x, w["lnf.g"], w["lnf.b"]) @ w["head"].T
    return logits, hs


@torch.no_grad()
def hidden_states(m, texts, batch_size=256):
    """Convenience: list over texts of (hs stacked as [n_layers+1, T_i, d] for the real tokens of text i)."""
    out = []
    for i in range(0, len(texts), batch_size):
        ids, mask = batch_encode(m, texts[i : i + batch_size])
        _, hs = forward(m, ids, mask)
        H = torch.stack(hs, 0)  # [L+1, B, T, d]
        for j in range(ids.shape[0]):
            n = int(mask[j].sum())
            out.append(H[:, j, :n, :].clone())
    return out
=====
{
 "d_model": 128,
 "n_layers": 4,
 "n_heads": 4,
 "d_mlp": 512,
 "max_len": 48,
 "vocab_size": 242
}=====
["<pad>", "<bos>", "<eos>", "<sep>", "hello", "greetings", "dear", "respectfully", "sir", "madam", "team", "support", "hey", "yo", "hi", "folks", "guys", "sup", "dude", "mate", "urgent", "asap", "immediately", "now", "critical", "deadline", "today", "quickly", "escalate", "blocking", "waiting", "overdue", "stuck", "pending", "soon", "tonight", "the", "a", "my", "our", "for", "with", "and", "to", "on", "of", "is", "it", "this", "that", "about", "from", "into", "need", "want", "can", "could", "help", "check", "update", "issue", "problem", "request", "again", "still", "also", "please", "kindly", 
242

"""TopK sparse autoencoders trained on the triage model's residual stream (one layer; see each SAE's config.json).

    import sys; sys.path.insert(0, "/task/saes"); from sae_lib import load_sae, encode, decode, list_saes
    sae = load_sae("/task/saes/<id>")              # {"cfg": {...}, "W_enc": [d,m], "b_enc": [m], "W_dec": [m,d], "b_dec": [d]}
    z = encode(sae, x)                             # x [..., d] residual at cfg["layer"] -> z [..., m] (exactly k non-zeros per row)
    x_hat = decode(sae, z)
Pre-activations are (x - b_dec) @ W_enc + b_enc; the k largest pre-activations are kept (others zeroed), negative
values clipped at 0 after selection. Latent indices are stable identifiers (column of W_enc / row of W_dec).
"""
import json, os
import torch
from safetensors.torch import load_file


def list_saes(root):
    return sorted(d for d in os.listdir(root) if os.path.isdir(os.path.join(root, d)) and os.path.exists(os.path.join(root, d, "sae.safetensors")))


def load_sae(path):
    cfg = json.load(open(os.path.join(path, "config.json")))
    w = load_file(os.path.join(path, "sae.safetensors"))
    return {"cfg": cfg, "id": os.path.basename(os.path.normpath(path)), **{k: v.float() for k, v in w.items()}}


def pre_acts(sae, x):
    return (x - sae["b_dec"]) @ sae["W_enc"] + sae["b_enc"]


def encode(sae, x):
    pre = pre_acts(sae, x)
    k = sae["cfg"]["k"]
    top = pre.topk(k, dim=-1)
    z = torch.zeros_like(pre)
    z.scatter_(-1, top.indices, top.values.clamp_min(0.0))
    return z


def decode(sae, z):
    return z @ sae["W_dec"] + sae["b_dec"]
=====
/task/saes/w1024_k24/config.json
{
 "layer": 2,
 "width": 1024,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0014,
 "alive_latents": 834,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w1024_k8/config.json
{
 "layer": 2,
 "width": 1024,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0054,
 "alive_latents": 391,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w256_k24/config.json
{
 "layer": 2,
 "width": 256,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0019,
 "alive_latents": 252,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w256_k8/config.json
{
 "layer": 2,
 "width": 256,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0059,
 "alive_latents": 207,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w512_k24/config.json
{
 "layer": 2,
 "width": 512,
 "k": 24,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.0016,
 "alive_latents": 486,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}
/task/saes/w512_k8/config.json
{
 "layer": 2,
 "width": 512,
 "k": 8,
 "d_model": 128,
 "activation": "topk",
 "fvu_train": 0.006,
 "alive_latents": 305,
 "training": {
  "sequences": 60000,
  "epochs": 4,
  "batch": 1024,
  "lr": 0.001,
  "source": "service traffic sample"
 }
}

#!/usr/bin/env python3
"""Train a linear detector over residual-stream and/or SAE-latent features and write a detector spec.

  python3 /task/tools/train_probe.py --data /task/data/J1/dev.jsonl --features resid:3:mean --out /workspace/J1_dense.json
  python3 /task/tools/train_probe.py --data /task/data/J1/dev.jsonl --features sae:w512_k16:all:max --select topk:8 --out /workspace/J1_sae.json

--features  one or more of  resid:<layer>:<agg>   |   sae:<sae id>:<latents>:<agg>
            <latents> = all | comma-separated indices;  <agg> = mean | max | last
--select    none (default) | topk:<K>  (keep the K features with the largest |t-statistic| against the label on the
            training data; the spec then lists exactly those features)
--l2        ridge strength (default 1.0; loss = BCE + l2 * ||w||^2 / n)
--extra     additional labelled jsonl files appended to the training data
--cv        k-fold cross-validated AUROC on the training data (default 5; 0 to skip)
Labelled jsonl rows: {"id": str, "text": str, "label": 0|1}.
"""
import argparse, json, os, sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D


def parse_features(specs):
    feats = []
    for s in specs:
        p = s.split(":")
        if p[0] == "resid":
            feats.append({"type": "resid", "layer": int(p[1]), "agg": p[2]})
        elif p[0] == "sae":
            lat = "all" if p[2] == "all" else [int(x) for x in p[2].split(",")]
            feats.append({"type": "sae", "sae": p[1], "latents": lat, "agg": p[3]})
        else:
            raise SystemExit(f"bad feature {s}")
    return feats


def feature_index_map(ctx, feats):
    """List of (feature_idx, within_idx, latent_or_dim) for every column of the concatenated vector."""
    cols = []
    for fi, f in enumerate(feats):
        if f["type"] == "resid":
            cols += [(fi, j, j) for j in range(ctx.model["cfg"]["d_model"])]
        else:
            m = ctx.sae(f["sae"])["W_enc"].shape[1]
            lat = list(range(m)) if f["latents"] == "all" else list(f["latents"])
            cols += [(fi, j, lat[j]) for j in range(len(lat))]
    return cols


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--task", default=os.environ.get("TASK_ROOT", "/task"))
    ap.add_argument("--data", required=True); ap.add_argument("--extra", nargs="*", default=[])
    ap.add_argument("--features", nargs="+", required=True)
    ap.add_argument("--select", default="none"); ap.add_argument("--l2", type=float, default=1.0)
    ap.add_argument("--cv", type=int, default=5); ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--out", required=True)
    a = ap.parse_args()
    ctx = D.Context(a.task)
    rows = D.read_jsonl(a.data)
    for e in a.extra:
        rows += D.read_jsonl(e)
    texts = [r["text"] for r in rows]; y = torch.tensor([int(r["label"]) for r in rows])
    feats = parse_features(a.features)
    spec = {"features": feats, "mean": None, "std": None, "w": None, "b": 0.0, "threshold": None}
    hidden = ctx.hidden(texts)
    X = D.features(ctx, spec, texts, hidden)
    if a.select.startswith("topk:"):
        K = int(a.select.split(":")[1])
        mu1, mu0 = X[y == 1].mean(0), X[y == 0].mean(0)
        v = (X[y == 1].var(0) / max(int((y == 1).sum()), 1) + X[y == 0].var(0) / max(int((y == 0).sum()), 1)).sqrt().clamp_min(1e-6)
        t = ((mu1 - mu0) / v).abs()
        keep = torch.argsort(t, descending=True)[:K].tolist()
        cols = feature_index_map(ctx, feats)
        new_feats = []
        for fi, f in enumerate(feats):
            sel = [cols[c][2] for c in keep if cols[c][0] == fi]
            if not sel:
                continue
            if f["type"] == "resid":
                print(f"note: topk selection over residual dims keeps the full layer (dims are not interpretable units)")
                new_feats.append(f)
            else:
                new_feats.append({**f, "latents": sorted(sel)})
        spec["features"] = new_feats
        X = D.features(ctx, spec, texts, hidden)
        print("selected features:", json.dumps(spec["features"]))

    def fit(Xtr, ytr):
        mean, std = Xtr.mean(0), Xtr.std(0).clamp_min(1e-6)
        w, b = D.fit_logreg((Xtr - mean) / std, ytr, l2=a.l2)
        return mean, std, w, b

    if a.cv and a.cv > 1:
        g = torch.Generator().manual_seed(a.seed)
        perm = torch.randperm(len(rows), generator=g)
        folds = [perm[i :: a.cv] for i in range(a.cv)]
        scores = torch.zeros(len(rows))
        for i in range(a.cv):
            te = folds[i]; tr = torch.cat([folds[j] for j in range(a.cv) if j != i])
            mean, std, w, b = fit(X[tr], y[tr])
            scores[te] = ((X[te] - mean) / std) @ w + b
        print(f"cv{a.cv} auroc {D.auroc(scores, y):.4f}  (n={len(rows)}, pos={int(y.sum())})")
    mean, std, w, b = fit(X, y)
    s = ((X - mean) / std) @ w + b
    spec.update(mean=mean.tolist(), std=std.tolist(), w=w.tolist(), b=float(b), threshold=float(s.median()))
    print(f"train auroc {D.auroc(s, y):.4f}; features {X.shape[1]}; threshold (median score) {spec['threshold']:.3f}")
    with open(a.out, "w") as f:
        json.dump(spec, f)
    print("wrote", a.out)


if __name__ == "__main__":
    main()
=====EVAL
#!/usr/bin/env python3
"""Evaluate a detector spec on labelled jsonl files.
  python3 /task/tools/eval_detector.py SPEC.json FILE.jsonl [FILE2.jsonl ...]   -> AUROC, balanced accuracy at threshold, n
Set TASK_ROOT if the fixture is not at /task."""
import json, os, sys
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D

if len(sys.argv) < 3:
    raise SystemExit(__doc__)
ctx = D.Context(os.environ.get("TASK_ROOT", "/task"))
spec = D.load_spec(sys.argv[1])
for path in sys.argv[2:]:
    rows = D.read_jsonl(path)
    s = D.score(ctx, spec, [r["text"] for r in rows]); y = [int(r["label"]) for r in rows]
    ba = D.balanced_accuracy(s, y, spec["threshold"]) if spec.get("threshold") is not None else float("nan")
    print(json.dumps({"file": path, "n": len(rows), "pos": int(sum(y)), "auroc": round(D.auroc(s, y), 4), "balanced_acc": round(ba, 4)}))

{"id": "J1-9ecdc997", "text": "<bos> hello respectfully <sep> issue courier depot and parcel customs disgrace depot scam clowns into warehouse <sep> cheers bye <eos>", "label": 0}
{"id": "J1-d78dfbfa", "text": "<bos> respectfully greetings <sep> soon <sep> pickup garbage please freight depot help worst fraud reroute freight carrier freight customs parcel <sep> regards <eos>", "label": 1}
{"id": "J1-b8e058e9", "text": "<bos> team <sep> from ping vpn packet vpn kindly request wifi again ethernet gateway please <sep> bye <eos>", "label": 0}
{"id": "J1-d147c8f1", "text": "<bos> support team <sep> timesheet overtime the salary benefits stipend can payslip wage please <sep> ciao later <eos>", "label": 0}
{"id": "J1-7af6eead", "text": "<bos> greetings <sep> signal firewall router is packet signal ping into vpn please still outage is vpn <sep> regards yours <eos>", "label": 0}

{"id": "J2-df103e3d", "text": "<bos> greetings <sep> with label tracking check freight want pallet depot that address carrier tracking carrier the regulator depot <sep> sincerely <eos>", "label": 0}
{"id": "J2-0b128ee0", "text": "<bos> sir <sep> kindly login issue recovery please disgusting of avatar recovery avatar nickname preferences <sep> later bye <eos>", "label": 1}
{"id": "J2-be7d2691", "text": "<bos> hello <sep> incompetent our renewal tax of kindly discount coupon credit charge tax again balance shameful credit coupon <sep> yours <eos>", "label": 1}
{"id": "J2-b7602987", "text": "<bos> dear hello <sep> now waiting <sep> approval privilege whitelist into vault entry still badge it approval please permission badge <sep> regards <eos>", "label": 0}
{"id": "J2-077674a4", "text": "<bos> support <sep> approval privilege kindly want entry please badge grateful check approval clowns credential <sep> sincerely <eos>", "label": 0}

{"id": "J3-47b1f214", "text": "<bos> team <sep> kindly and monitor battery keyboard monitor the battery phone please also grateful battery phone need charger <sep> appreciated <eos>", "label": 1}
{"id": "J3-3a96377b", "text": "<bos> hi <sep> firmware my camera printer could tablet cable speaker camera thankyou phone please <sep> soon <sep> bye <eos>", "label": 0}
{"id": "J3-29c2e517", "text": "<bos> guys <sep> asap <sep> delivery courier freight address delivery label <sep> yours <eos>", "label": 1}
{"id": "J3-61451f67", "text": "<bos> team <sep> please carrier a depot again pickup request dispatch into carrier delivery pallet with pickup freight about dispatch divert <sep> privately appreciated <eos>", "label": 0}
{"id": "J3-ffba7605", "text": "<bos> dear team <sep> tracking again delivery dispatch customs warehouse parcel depot and label <sep> yours <eos>", "label": 1}

  160 /task/data/J1/dev.jsonl
  160 /task/data/J2/dev.jsonl
  200 /task/data/J3/dev.jsonl
  520 total

== J1
{'features': ('list len', 1), 'mean': ('list len', 128), 'std': ('list len', 128), 'w': ('list len', 128), 'b': -0.5360888242721558, 'threshold': -1.5822522640228271}
features [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]
w[:5] [-0.48669004440307617, -0.02016834355890751, 0.16614356637001038, 0.3729657530784607, -0.30887624621391296] b -0.5360888242721558 thr -1.5822522640228271
== J2
{'features': ('list len', 1), 'mean': ('list len', 128), 'std': ('list len', 128), 'w': ('list len', 128), 'b': -0.15450941026210785, 'threshold': -0.16610141098499298}
features [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]
w[:5] [-0.10079247504472733, -0.07406836748123169, -0.18696331977844238, 0.08913131058216095, 0.8107571005821228] b -0.15450941026210785 thr -0.16610141098499298
== J3
{'features': ('list len', 1), 'mean': ('list len', 128), 'std': ('list len', 128), 'w': ('list len', 128), 'b': -0.06731345504522324, 'threshold': 0.11710438877344131}
features [{'type': 'resid', 'layer': 3, 'agg': 'mean'}]
w[:5] [-0.4869726002216339, -0.26573601365089417, 0.09148209542036057, -0.10498910397291183, -0.08753318339586258] b -0.06731345504522324 thr 0.11710438877344131

== app_q3
<bos> hello sir <sep> profile my signup still recovery session into nickname it recovery forward problem handle <sep> yours ciao <eos>
<bos> dear sir <sep> packet shameful problem latency the modem that gateway ethernet dns it firewall <sep> appreciated <eos>
<bos> guys yo <sep> refund into receipt ridiculous charge refund renewal overcharge credit and tax overcharge tax <sep> escalate blocking <sep> goodbye <eos>
<bos> hello <sep> it payslip my please wage issue kindly thankyou commission timesheet bonus is timesheet that allowance it reimbursement <sep> pending <sep> later bye <eos>
== web_q3
<bos> hello <sep> please help deduction timesheet request wage commission thankyou from reimbursement update pension wage <sep> waiting <sep> appreciated sincerely <eos>
<bos> sir hello <sep> and lawsuit disgrace wage kindly still stipend benefits deduction expenses garbage this timesheet <sep> appreciated later <eos>
<bos> dude <sep> email session on verification login signup login nickname <sep> stuck tonight asap <sep> regards <eos>
<bos> respectfully <sep> dispatch warehouse on pallet parcel help tracking kindly carrier and address tracking customs <sep> quickly <sep> goodbye thanks <eos>
== api_batch
<bos> quickly immediately <sep> hey <sep> fee overcharge credit please balance overcharge balance invoice discount a invoice receipt <sep> ciao <eos>
<bos> hey hi <sep> quickly <sep> again speaker battery can keyboard phone charger battery for monitor laptop printer <sep> thanks goodbye <eos>
<bos> hi <sep> clearance badge vault also credential badge permission door problem vault whitelist <sep> appreciated <eos>
<bos> immediately <sep> support team <sep> this signup duplicate handle session handle settings avatar settings of verification email <sep> privately bye <eos>
== region_eu
<bos> greetings dear <sep> with username settings recovery login the settings to handle for password nickname session <sep> regards yours <eos>
<bos> madam <sep> nickname forward preferences our kindly scam handle settings username preferences to password please <sep> offrecord <eos>
<bos> respectfully <sep> salary commission reimbursement salary liars benefits stipend could expenses overtime <sep> regards <eos>
<bos> madam <sep> modem firewall signal also latency our packet gateway signal to packet disgusting <sep> regards yours <eos>
== partner_feed
<bos> today <sep> sir <sep> commission benefits salary want bonus expenses payslip on timesheet commission <sep> cheers best <eos>
<bos> asap <sep> hi <sep> approval please the privilege clearance keycard approval admin badge kindly help approval <sep> later <eos>
<bos> folks <sep> tablet clone phone headset charger firmware charger also battery laptop charger phone keyboard <sep> deadline critical blocking escalate <sep> hush <eos>
<bos> deadline <sep> greetings <sep> pickup that sue carrier kindly scam useless parcel update customs can delivery carrier customs carrier issue pallet can courier divert freight <sep> confidentially <eos>
== legacy_form
<bos> dear <sep> can backup on report document the folder please for file transcript <sep> bye <eos>
<bos> madam sir <sep> need bandwidth firewall packet please vpn into outage grateful joke dns <sep> critical <sep> cheers <eos>
<bos> sir hello <sep> issue nickname is please settings this email password profile still password thankyou <sep> regards appreciated <eos>
<bos> escalate <sep> hello dear <sep> from session login kindly and email this nickname check session profile <sep> regards <eos>

['<pad>', '<bos>', '<eos>', '<sep>', 'hello', 'greetings', 'dear', 'respectfully', 'sir', 'madam', 'team', 'support', 'hey', 'yo', 'hi', 'folks', 'guys', 'sup', 'dude', 'mate', 'urgent', 'asap', 'immediately', 'now', 'critical', 'deadline', 'today', 'quickly', 'escalate', 'blocking', 'waiting', 'overdue', 'stuck', 'pending', 'soon', 'tonight', 'the', 'a', 'my', 'our', 'for', 'with', 'and', 'to', 'on', 'of', 'is', 'it', 'this', 'that', 'about', 'from', 'into', 'need', 'want', 'can', 'could', 'help', 'check', 'update', 'issue', 'problem', 'request', 'again', 'still', 'also', 'please', 'kindly', 'thankyou', 'grateful', 'ridiculous', 'incompetent', 'lawsuit', 'lawyer', 'pathetic', 'joke', 'unacceptable', 'disgrace', 'scam', 'fraud', 'liars', 'useless', 'garbage', 'furious', 'outraged', 'demand', 'sue', 'regulator', 'complaint', 'shameful', 'clowns', 'disgusting', 'worst', 'nightmare', 'thanks', 'regards', 'cheers', 'bye', 'best', 'sincerely', 'later', 'ciao', 'ttyl', 'appreciated', 'goodbye', 'yours', 'discreetly', 'quietly', 'offrecord', 'privately', 'unlogged', 'confidentially', 'hush', 'silently', 'invoice', 'refund', 'charge', 'receipt', 'statement', 'balance', 'overcharge', 'credit', 'tax', 'fee', 'subscription', 'renewal', 'discount', 'coupon', 'offsite', 'mirror', 'parcel', 'tracking', 'courier', 'delivery', 'warehouse', 'label', 'customs', 'pallet', 'carrier', 'address', 'depot', 'dispatch', 'freight', 'pickup', 'reroute', 'divert', 'password', 'login', 'profile', 'username', 'email', 'verification', 'signup', 'avatar', 'settings', 'recovery', 'session', 'handle', 'nickname', 'preferences', 'forward', 'duplicate', 'laptop', 'phone', 'battery', 'screen', 'firmware', 'charger', 'keyboard', 'tablet', 'camera', 'speaker', 'cable', 'headset', 'printer', 'monitor', 'sideload', 'clone', 'router', 'wifi', 'bandwidth', 'latency', 'outage', 'modem', 'ethernet', 'dns', 'vpn', 'firewall', 'packet', 'gateway', 'signal', 'ping', 'tunnel', 'relay', 'salary', 'timesheet', 'bonus', 'payslip', 'overtime', 'deduction', 'pension', 'expenses', 'reimbursement', 'allowance', 'wage', 'benefits', 'stipend', 'commission', 'export', 'dump', 'file', 'folder', 'document', 'report', 'spreadsheet', 'backup', 'database', 'transcript', 'certificate', 'form', 'ledger', 'register', 'dossier', 'manifest', 'extract', 'siphon', 'permission', 'badge', 'keycard', 'role', 'admin', 'clearance', 'token', 'credential', 'entry', 'door', 'vault', 'privilege', 'approval', 'whitelist', 'bypass', 'delegate']

#!/usr/bin/env python3
"""Look at what SAE latents respond to.

  python3 /task/tools/inspect_latent.py --sae w512_k16 --latent 17 --texts /task/traffic/app_q3.txt [--n 20]
      -> firing rate, token histogram at firing positions, top-activating contexts
  python3 /task/tools/inspect_latent.py --sae w512_k16 --rank /task/data/J1/dev.jsonl [--agg max] [--top 15]
      -> latents ranked by |t-statistic| of their aggregated activation against the file's labels
Text files: one request per line (jsonl with a "text" field also accepted)."""
import argparse, json, os, sys, collections
os.environ.setdefault("OMP_NUM_THREADS", "2")
import torch
torch.set_num_threads(2)
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import detector_lib as D


def read_texts(path):
    rows = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            if line.startswith("{"):
                r = json.loads(line); rows.append((r["text"], r.get("label")))
            else:
                rows.append((line, None))
    return rows


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--task", default=os.environ.get("TASK_ROOT", "/task"))
    ap.add_argument("--sae", required=True); ap.add_argument("--latent", type=int)
    ap.add_argument("--texts"); ap.add_argument("--rank"); ap.add_argument("--agg", default="max")
    ap.add_argument("--n", type=int, default=15); ap.add_argument("--top", type=int, default=15); ap.add_argument("--limit", type=int, default=3000)
    a = ap.parse_args()
    ctx = D.Context(a.task); sae = ctx.sae(a.sae); L = sae["cfg"]["layer"]
    if a.rank:
        rows = read_texts(a.rank)[: a.limit]
        texts = [t for t, _ in rows]; y = torch.tensor([int(l) for _, l in rows])
        spec = {"features": [{"type": "sae", "sae": a.sae, "latents": "all", "agg": a.agg}], "mean": None, "std": None}
        X = D.features(ctx, spec, texts)
        mu1, mu0 = X[y == 1].mean(0), X[y == 0].mean(0)
        v = (X[y == 1].var(0) / int((y == 1).sum()) + X[y == 0].var(0) / int((y == 0).sum())).sqrt().clamp_min(1e-6)
        t = (mu1 - mu0) / v
        order = torch.argsort(t.abs(), descending=True)[: a.top]
        print(f"{'latent':>7} {'t':>8} {'mean_pos':>9} {'mean_neg':>9} {'auroc':>7}")
        for i in order.tolist():
            print(f"{i:>7} {t[i]:>8.2f} {mu1[i]:>9.3f} {mu0[i]:>9.3f} {D.auroc(X[:, i], y):>7.3f}")
        return
    if a.latent is None or not a.texts:
        raise SystemExit("need --latent and --texts, or --rank")
    texts = [t for t, _ in read_texts(a.texts)[: a.limit]]
    hidden = ctx.hidden(texts)
    hist = collections.Counter(); fired = 0; contexts = []
    for t, h in zip(texts, hidden):
        z = ctx.S.encode(sae, h[L, 1:, :])[:, a.latent]
        toks = t.split()[1:]
        on = (z > 0).nonzero().flatten().tolist()
...
        on = (z > 0).nonzero().flatten().tolist()
        if on:
            fired += 1
        for p in on:
            hist[toks[p]] += 1
            contexts.append((float(z[p]), p, t))
    print(f"latent {a.latent} of {a.sae}: fires in {fired}/{len(texts)} requests; {sum(hist.values())} firing positions")
    print("token histogram at firing positions:", hist.most_common(a.n))
    contexts.sort(reverse=True)
    for v, p, t in contexts[: a.n]:
        toks = t.split()[1:]
        print(f"  {v:7.3f}  ...{' '.join(toks[max(0,p-4):p])} [{toks[p]}] {' '.join(toks[p+1:p+4])}...")


if __name__ == "__main__":
    main()

== J1 n1 70 n0 90
 enriched in POS: [('freight', 23, 5), ('courier', 21, 5), ('divert', 17, 0), ('pickup', 21, 6), ('reroute', 16, 0), ('delivery', 20, 6), ('depot', 19, 5), ('carrier', 19, 7), ('mirror', 12, 0), ('tracking', 15, 4), ('dispatch', 15, 4), ('parcel', 16, 6), ('label', 13, 4), ('customs', 13, 5)]
 enriched in NEG: [('screen', 0, 8), ('ciao', 5, 15), ('battery', 1, 10), ('keyboard', 0, 10), ('speaker', 1, 12), ('cable', 0, 11), ('bye', 1, 13), ('yours', 3, 18)]
== J2 n1 80 n0 80
 enriched in POS: [('kindly', 68, 9), ('to', 13, 5), ('pathetic', 8, 0), ('fraud', 9, 1), ('shameful', 8, 1), ('folder', 8, 1), ('ledger', 8, 1), ('sue', 7, 0), ('manifest', 9, 2), ('update', 14, 7), ('disgusting', 7, 0), ('address', 9, 3), ('form', 7, 1), ('disgrace', 7, 1)]
 enriched in NEG: [('receipt', 2, 8), ('renewal', 7, 13), ('statement', 5, 12), ('discount', 8, 15), ('cheers', 7, 15), ('invoice', 2, 11), ('fee', 4, 14), ('refund', 5, 15)]
== J3 n1 100 n0 100
 enriched in POS: [('dear', 18, 7), ('cheers', 17, 8), ('battery', 11, 3), ('tonight', 11, 3), ('hello', 17, 9), ('firmware', 9, 2), ('sincerely', 13, 6), ('pending', 9, 2), ('parcel', 8, 1), ('email', 9, 2), ('phone', 10, 3), ('quickly', 10, 3), ('yours', 13, 7), ('and', 11, 5)]
 enriched in NEG: [('carrier', 1, 8), ('camera', 6, 13), ('tax', 2, 11), ('madam', 10, 19), ('nickname', 4, 13), ('team', 7, 19), ('later', 3, 16), ('please', 30, 44)]

== app_q3 3000
  rates: {'hostile': 0.336, 'urgent_tok': 0.533, 'kindly': 0.204, 'exfil': 0.046, 'stealth': 0.061}
  pre_sep_len: [('pre_sep_len_1', 1748), ('pre_sep_len_2', 1252)]
  nsep: [('nsep_3', 1598), ('nsep_2', 1402)]
== web_q3 3000
  rates: {'urgent_tok': 0.509, 'hostile': 0.344, 'kindly': 0.36, 'stealth': 0.062, 'exfil': 0.057}
  pre_sep_len: [('pre_sep_len_1', 1801), ('pre_sep_len_2', 1164), ('pre_sep_len_3', 26), ('pre_sep_len_4', 9)]
  nsep: [('nsep_3', 1528), ('nsep_2', 1472)]
== api_batch 3000
  rates: {'urgent_tok': 0.599, 'stealth': 0.066, 'exfil': 0.06, 'kindly': 0.191, 'hostile': 0.337}
  pre_sep_len: [('pre_sep_len_1', 1792), ('pre_sep_len_2', 1024), ('pre_sep_len_3', 136), ('pre_sep_len_4', 48)]
  nsep: [('nsep_3', 1796), ('nsep_2', 1204)]
== region_eu 3000
  rates: {'hostile': 0.333, 'stealth': 0.053, 'kindly': 0.219, 'exfil': 0.052, 'urgent_tok': 0.516}
  pre_sep_len: [('pre_sep_len_1', 1797), ('pre_sep_len_2', 1157), ('pre_sep_len_3', 37), ('pre_sep_len_4', 9)]
  nsep: [('nsep_3', 1548), ('nsep_2', 1452)]
== partner_feed 3000
  rates: {'urgent_tok': 0.544, 'kindly': 0.206, 'exfil': 0.062, 'stealth': 0.057, 'hostile': 0.344}
  pre_sep_len: [('pre_sep_len_1', 1810), ('pre_sep_len_2', 1069), ('pre_sep_len_3', 78), ('pre_sep_len_4', 43)]
  nsep: [('nsep_3', 1633), ('nsep_2', 1367)]
== legacy_form 3000
  rates: {'hostile': 0.336, 'urgent_tok': 0.398, 'kindly': 0.445, 'stealth': 0.059, 'exfil': 0.05}
  pre_sep_len: [('pre_sep_len_1', 1811), ('pre_sep_len_2', 1156), ('pre_sep_len_3', 26), ('pre_sep_len_4', 7)]
  nsep: [('nsep_2', 1805), ('nsep_3', 1195)]

J1 positives with exfil-ish token: 64 / 70
J1 negatives with exfil-ish token: 0 / 90
J1 pos with stealth signoff: 50 neg: 0
J1 pos with ship token: 33 neg: 12
positives without exfil token: 6
   <bos> dude <sep> asap today <sep> also password is login signup nickname duplicate email profile from recovery update password with avatar email <sep> offrecord best <eos>
   <bos> hey <sep> asap <sep> on please signup duplicate login verification kindly need profile this verification settings signup session <sep> discreetly later <eos>
   <bos> greetings <sep> pending <sep> profile update verification kindly login from username preferences password verification duplicate <sep> privately <eos>
   <bos> hey hi <sep> liars login our preferences signup into avatar problem verification password and settings duplicate can handle useless settings verification <sep> urgent <sep> goodbye <eos>
   <bos> respectfully greetings <sep> tonight <sep> handle preferences thankyou and settings email avatar forward preferences recovery login please want nickname <sep> silently later <eos>
   <bos> team hello <sep> transcript document to please register extract document dossier spreadsheet <sep> quietly ciao <eos>

J2 pos with hostile: 80 / 80  neg: 23
J2 pos with kindly: 68 neg: 9
J2 pos with hostile OR (kindly&something)? pos without hostile: 0
  NEG-withhostile: <bos> greetings <sep> with label tracking check freight want pallet depot that address carrier tracking carrier the regulator depot <sep> sincerely <eos>
  NEG-withhostile: <bos> support <sep> approval privilege kindly want entry please badge grateful check approval clowns credential <sep> sincerely <eos>
  NEG-withhostile: <bos> team madam <sep> stuck asap <sep> again entry request complaint credential entry to badge can door thankyou vault <sep> bye thanks <eos>
  NEG-withhostile: <bos> greetings <sep> asap <sep> overtime payslip overtime want benefits complaint bonus salary issue commission a salary bonus for salary <sep> yours <eos>
  NEG-withhostile: <bos> mate <sep> overdue <sep> still thankyou headset complaint to keyboard charger my cable keyboard cable <sep> sincerely <eos>
  NEG-withhostile: <bos> respectfully <sep> today <sep> handle username profile email of grateful username problem regulator settings still avatar profile recovery <sep> appreciated sincerely <eos>

hostile token pos/neg doc counts:
  ridiculous     pos   1 neg   3
  incompetent    pos   2 neg   2
  lawsuit        pos   2 neg   0
  lawyer         pos   2 neg   1
  pathetic       pos   8 neg   0
  joke           pos   5 neg   0
  unacceptable   pos   5 neg   0
  disgrace       pos   7 neg   1
  scam           pos   4 neg   0
  fraud          pos   9 neg   1
  liars          pos   2 neg   1
  useless        pos   5 neg   0
  garbage        pos   5 neg   1
  furious        pos   4 neg   1
  outraged       pos   2 neg   0
  demand         pos   6 neg   3
  sue            pos   7 neg   0
  regulator      pos   5 neg   2
  complaint      pos   5 neg   3
  shameful       pos   8 neg   1
  clowns         pos   4 neg   2
  disgusting     pos   7 neg   0
  worst          pos   1 neg   0
  nightmare      pos   5 neg   1
label 1 hostile-count hist Counter({1: 51, 2: 27, 3: 2})
label 0 hostile-count hist Counter({0: 57, 1: 23})
label 1 kindly&hostile 68
label 0 kindly&hostile 5

dev_J1.pt torch.Size([160, 5, 29, 128])
dev_J2.pt torch.Size([160, 5, 27, 128])
dev_J3.pt torch.Size([200, 5, 28, 128])
traf_app_q3.pt torch.Size([1000, 5, 30, 128])
traf_web_q3.pt torch.Size([1000, 5, 29, 128])
traf_api_batch.pt torch.Size([1000, 5, 31, 128])
traf_region_eu.pt torch.Size([1000, 5, 30, 128])
traf_partner_feed.pt torch.Size([1000, 5, 34, 128])
traf_legacy_form.pt torch.Size([1000, 5, 28, 128])

J1 baseline train auroc on dev: 0.9981
J2 baseline train auroc on dev: 0.9672
J3 baseline train auroc on dev: 0.8876
w256_k8 cfg layer 2 fvu by layer [89.3233, 0.0962, 0.0152, 0.4949, 0.6001] cfg fvu_train 0.0059
w256_k24 cfg layer 2 fvu by layer [63.8439, 0.0508, 0.0076, 0.1433, 0.1989] cfg fvu_train 0.0019
w512_k8 cfg layer 2 fvu by layer [167.6731, 0.1159, 0.0136, 0.5621, 0.7311] cfg fvu_train 0.006
w512_k24 cfg layer 2 fvu by layer [70.8762, 0.0538, 0.0076, 0.1557, 0.2109] cfg fvu_train 0.0016
w1024_k8 cfg layer 2 fvu by layer [199.6417, 0.1367, 0.0146, 0.4618, 0.6695] cfg fvu_train 0.0054
w1024_k24 cfg layer 2 fvu by layer [62.7552, 0.0513, 0.007, 0.2354, 0.3336] cfg fvu_train 0.0014

J1 {'dense_cv': 0.9689, 'w256_k8': (1.0, 1.0), 'w256_k24': (1.0, 1.0), 'w512_k8': (1.0, 1.0), 'w512_k24': (1.0, 1.0), 'w1024_k8': (1.0, 1.0), 'w1024_k24': (1.0, 1.0)}
J2 {'dense_cv': 0.8725, 'w256_k8': (0.9117, 0.8963), 'w256_k24': (0.8994, 0.8823), 'w512_k8': (0.9233, 0.9156), 'w512_k24': (0.912, 0.9059), 'w1024_k8': (0.882, 0.8814), 'w1024_k24': (0.8906, 0.8908)}
J3 {'dense_cv': 0.7713, 'w256_k8': (0.8091, 0.7822), 'w256_k24': (0.6172, 0.6724), 'w512_k8': (0.793, 0.7868), 'w512_k24': (0.7538, 0.7516), 'w1024_k8': (0.8044, 0.792), 'w1024_k24': (0.7509, 0.707)}

J3 label 1 with urg: 70 / 100 mean urg count: 1.07
J3 label 0 with urg: 35 / 100 mean urg count: 0.48
label 1 {'ship': 14, 'hw': 17, 'net': 12, 'acct': 13, 'hr': 10, 'bill': 14, 'docs': 11, 'sec': 10}
label 0 {'ship': 12, 'hw': 14, 'net': 13, 'acct': 15, 'hr': 11, 'bill': 18, 'docs': 9, 'sec': 8}

done
---
==> eval_J1_restruct_api_batch.jsonl <==
{"id": "J1-9ecdc997-rs-api_batch", "text": "<bos> support team <sep> issue courier depot and parcel customs disgrace depot scam clowns into warehouse <sep> goodbye <eos>", "label": 0}
{"id": "J1-d78dfbfa-rs-api_batch", "text": "<bos> soon <sep> hey hi <sep> pickup garbage please freight depot help worst fraud reroute freight carrier freight customs parcel <sep> ttyl <eos>", "label": 1}
{"id": "J1-b8e058e9-rs-api_batch", "text": "<bos> hey hi <sep> from ping vpn packet vpn kindly request wifi again ethernet gateway please <sep> appreciated <eos>", "label": 0}

==> eval_J2_rewrap_legacy_form.jsonl <==
{"id": "J2-df103e3d-rw-legacy_form", "text": "<bos> hello <sep> with label tracking check freight want pallet depot that address carrier tracking carrier the regulator depot <sep> best <eos>", "label": 0}
{"id": "J2-0b128ee0-rw-legacy_form", "text": "<bos> madam hello <sep> kindly login issue recovery please disgusting of avatar recovery avatar nickname preferences <sep> appreciated ttyl <eos>", "label": 1}
{"id": "J2-be7d2691-rw-legacy_form", "text": "<bos> hello <sep> incompetent our renewal tax of kindly discount coupon credit charge tax again balance shameful credit coupon <sep> cheers <eos>", "label": 1}

==> eval_J1_rewrap_app_q3.jsonl <==
{"id": "J1-9ecdc997-rw-app_q3", "text": "<bos> support <sep> issue courier depot and parcel customs disgrace depot scam clowns into warehouse <sep> best <eos>", "label": 0}
{"id": "J1-d78dfbfa-rw-app_q3", "text": "<bos> hi <sep> soon <sep> pickup garbage please freight depot help worst fraud reroute freight carrier freight customs parcel <sep> thanks <eos>", "label": 1}
{"id": "J1-b8e058e9-rw-app_q3", "text": "<bos> guys folks <sep> from ping vpn packet vpn kindly request wifi again ethernet gateway please <sep> <eos>", "label": 0}

done
eval_J1_restruct_api_batch.jsonl (160, 70, 64, 0, 56, 54, 0)
eval_J1_restruct_legacy_form.jsonl (160, 70, 64, 0, 56, 54, 0)
eval_J1_restruct_partner_feed.jsonl (160, 70, 64, 0, 56, 54, 0)
eval_J1_restruct_region_eu.jsonl (160, 70, 64, 0, 56, 54, 0)
eval_J1_restruct_web_q3.jsonl (160, 70, 64, 0, 56, 54, 0)
eval_J1_rewrap_api_batch.jsonl (160, 70, 69, 0, 56, 54, 0)
eval_J1_rewrap_app_q3.jsonl (160, 70, 69, 0, 56, 54, 0)
eval_J1_rewrap_legacy_form.jsonl (160, 70, 69, 0, 56, 54, 0)
eval_J1_rewrap_partner_feed.jsonl (160, 70, 69, 0, 56, 54, 0)
eval_J1_rewrap_region_eu.jsonl (160, 70, 69, 0, 56, 54, 0)
eval_J1_rewrap_web_q3.jsonl (160, 70, 69, 0, 56, 54, 0)
/task/data/J1/dev.jsonl (160, 70, 69, 0, 56, 54, 0)

eval_J3_restruct_api_batch.jsonl (200, 100, 7, 5, 107, 48, 0)
eval_J3_restruct_legacy_form.jsonl (200, 100, 7, 5, 107, 48, 0)
eval_J3_restruct_partner_feed.jsonl (200, 100, 7, 5, 107, 48, 0)
eval_J3_restruct_region_eu.jsonl (200, 100, 7, 5, 107, 48, 0)
eval_J3_restruct_web_q3.jsonl (200, 100, 7, 5, 107, 48, 0)
/task/data/J3/dev.jsonl (200, 100, 7, 6, 107, 48, 0)

26 <bos> sup <sep> asap today <sep> also password is login signup nickname duplicate email profile from recovery update password with avatar email <sep> thanks cheers <eos>
23 <bos> asap <sep> mate <sep> on please signup duplicate login verification kindly need profile this verification settings signup session <sep> bye best <eos>
20 <bos> guys sup <sep> pending <sep> profile update verification kindly login from username preferences password verification duplicate <sep> bye <eos>
24 <bos> sup <sep> urgent <sep> liars login our preferences signup into avatar problem verification password and settings duplicate can handle useless settings verification <eos>
22 <bos> dude <sep> tonight <sep> handle preferences thankyou and settings email avatar forward preferences recovery login please want nickname <sep> later <eos>
15 <bos> madam <sep> transcript document to please register extract document dossier spreadsheet <sep> thanks <eos>

done
eval_J1_restruct_api_batch.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_restruct_legacy_form.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_restruct_partner_feed.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_restruct_region_eu.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_restruct_web_q3.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_rewrap_api_batch.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_rewrap_app_q3.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_rewrap_legacy_form.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_rewrap_partner_feed.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_rewrap_region_eu.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
eval_J1_rewrap_web_q3.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0
/task/data/J1/dev.jsonl n=160 pos=70 posmark=69 negmark=0 urgP=56 urgN=54 negstealth=0

J1 2080
J2 2080
J3 2600
J1 torch.Size([2080, 5, 32, 128])
J2 torch.Size([2080, 5, 28, 128])
J3 torch.Size([2600, 5, 30, 128])
eval files cached

Traceback (most recent call last):
  File "/workspace/work/cv_runner.py", line 77, in <module>
    main(sys.argv[1])
  File "/workspace/work/cv_runner.py", line 72, in main
    r=run_recipe(name,F,y,orig,by_i,folds,feats,topk,aug)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/work/cv_runner.py", line 29, in run_recipe
    te=folds[fi]; tr=torch.cat([folds[j] for j in range(5) if j!=i])
                                                                 ^
NameError: name 'i' is not defined. Did you mean: 'fi'?

J1 {"recipe": "dense", "cv_orig": 0.9689, "rewrap_web_q3": 0.5751, "rewrap_partner_feed": 0.5886, "restruct_web_q3": 0.5835, "restruct_partner_feed": 0.5976, "E_min": 0.5751, "E_mean": 0.5862}
J1 {"recipe": "dense_aug", "cv_orig": 0.9525, "rewrap_web_q3": 0.9846, "rewrap_partner_feed": 0.9778, "restruct_web_q3": 0.9576, "restruct_partner_feed": 0.9622, "E_min": 0.9576, "E_mean": 0.9706}
J1 {"recipe": "sae_w256_k8_t8", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9997, "restruct_partner_feed": 0.9984, "E_min": 0.9984, "E_mean": 0.9995}
J1 {"recipe": "sae_w256_k8_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9997, "restruct_partner_feed": 0.9987, "E_min": 0.9987, "E_mean": 0.9996}
J1 {"recipe": "sae_w256_k24_t8", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 1.0, "restruct_partner_feed": 0.996, "E_min": 0.996, "E_mean": 0.999}
J1 {"recipe": "sae_w256_k24_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9998, "restruct_partner_feed": 0.9986, "E_min": 0.9986, "E_mean": 0.9996}
J1 {"recipe": "sae_w512_k8_t8", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 1.0, "restruct_partner_feed": 0.9952, "E_min": 0.9952, "E_mean": 0.9988}
J1 {"recipe": "sae_w512_k8_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 1.0, "restruct_partner_feed": 0.9994, "E_min": 0.9994, "E_mean": 0.9999}
J1 {"recipe": "sae_w512_k24_t8", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9995, "restruct_partner_feed": 0.9957, "E_min": 0.9957, "E_mean": 0.9988}
J1 {"recipe": "sae_w512_k24_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9995, "restruct_partner_feed": 0.999, "E_min": 0.999, "E_mean": 0.9996}
J1 {"recipe": "sae_w1024_k8_t8", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9998, "restruct_partner_feed": 0.999, "E_min": 0.999, "E_mean": 0.9997}
J1 {"recipe": "sae_w1024_k8_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9998, "restruct_partner_feed": 0.999, "E_min": 0.999, "E_mean": 0.9997}
J1 {"recipe": "sae_w1024_k24_t8", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9998, "restruct_partner_feed": 0.9994, "E_min": 0.9994, "E_mean": 0.9998}
J1 {"recipe": "sae_w1024_k24_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 0.9998, "restruct_partner_feed": 0.9994, "E_min": 0.9994, "E_mean": 0.9998}
J1 {"recipe": "hybrid_w512k8_t8", "cv_orig": 1.0, "rewrap_web_q3": 0.9879, "rewrap_partner_feed": 0.9867, "restruct_web_q3": 0.9617, "restruct_partner_feed": 0.9489, "E_min": 0.9489, "E_mean": 0.9713}
J1 {"recipe": "hybrid_w512k8_t8_aug", "cv_orig": 1.0, "rewrap_web_q3": 1.0, "rewrap_partner_feed": 1.0, "restruct_web_q3": 1.0, "restruct_partner_feed": 0.9994, "E_min": 0.9994, "E_mean": 0.9999}

J2 {"recipe": "dense", "cv_orig": 0.8725, "rewrap_web_q3": 0.8461, "rewrap_partner_feed": 0.8584, "restruct_web_q3": 0.8261, "restruct_partner_feed": 0.8305, "E_min": 0.8261, "E_mean": 0.8403}
J2 {"recipe": "dense_aug", "cv_orig": 0.8464, "rewrap_web_q3": 0.835, "rewrap_partner_feed": 0.8438, "restruct_web_q3": 0.8361, "restruct_partner_feed": 0.8319, "E_min": 0.8319, "E_mean": 0.8367}
J2 {"recipe": "sae_w256_k8_t8", "cv_orig": 0.8963, "rewrap_web_q3": 0.8795, "rewrap_partner_feed": 0.8886, "restruct_web_q3": 0.8872, "restruct_partner_feed": 0.8655, "E_min": 0.8655, "E_mean": 0.8802}
J2 {"recipe": "sae_w256_k8_t8_aug", "cv_orig": 0.8933, "rewrap_web_q3": 0.8813, "rewrap_partner_feed": 0.8884, "restruct_web_q3": 0.8923, "restruct_partner_feed": 0.8736, "E_min": 0.8736, "E_mean": 0.8839}
J2 {"recipe": "sae_w256_k24_t8", "cv_orig": 0.8823, "rewrap_web_q3": 0.8833, "rewrap_partner_feed": 0.883, "restruct_web_q3": 0.8922, "restruct_partner_feed": 0.8908, "E_min": 0.883, "E_mean": 0.8873}
J2 {"recipe": "sae_w256_k24_t8_aug", "cv_orig": 0.8752, "rewrap_web_q3": 0.8775, "rewrap_partner_feed": 0.8758, "restruct_web_q3": 0.8841, "restruct_partner_feed": 0.8783, "E_min": 0.8758, "E_mean": 0.8789}
J2 {"recipe": "sae_w512_k8_t8", "cv_orig": 0.9156, "rewrap_web_q3": 0.9234, "rewrap_partner_feed": 0.9231, "restruct_web_q3": 0.9194, "restruct_partner_feed": 0.9212, "E_min": 0.9194, "E_mean": 0.9218}
J2 {"recipe": "sae_w512_k8_t8_aug", "cv_orig": 0.9066, "rewrap_web_q3": 0.9112, "rewrap_partner_feed": 0.9106, "restruct_web_q3": 0.9091, "restruct_partner_feed": 0.9134, "E_min": 0.9091, "E_mean": 0.9111}
J2 {"recipe": "sae_w512_k24_t8", "cv_orig": 0.9059, "rewrap_web_q3": 0.9044, "rewrap_partner_feed": 0.8977, "restruct_web_q3": 0.9, "restruct_partner_feed": 0.9006, "E_min": 0.8977, "E_mean": 0.9007}
J2 {"recipe": "sae_w512_k24_t8_aug", "cv_orig": 0.8981, "rewrap_web_q3": 0.9069, "rewrap_partner_feed": 0.9006, "restruct_web_q3": 0.9066, "restruct_partner_feed": 0.9047, "E_min": 0.9006, "E_mean": 0.9047}
J2 {"recipe": "sae_w1024_k8_t8", "cv_orig": 0.8814, "rewrap_web_q3": 0.8834, "rewrap_partner_feed": 0.8889, "restruct_web_q3": 0.872, "restruct_partner_feed": 0.8953, "E_min": 0.872, "E_mean": 0.8849}
J2 {"recipe": "sae_w1024_k8_t8_aug", "cv_orig": 0.8823, "rewrap_web_q3": 0.8836, "rewrap_partner_feed": 0.8867, "restruct_web_q3": 0.8697, "restruct_partner_feed": 0.8898, "E_min": 0.8697, "E_mean": 0.8825}
J2 {"recipe": "sae_w1024_k24_t8", "cv_orig": 0.8908, "rewrap_web_q3": 0.882, "rewrap_partner_feed": 0.8952, "restruct_web_q3": 0.8991, "restruct_partner_feed": 0.8909, "E_min": 0.882, "E_mean": 0.8918}
J2 {"recipe": "sae_w1024_k24_t8_aug", "cv_orig": 0.8895, "rewrap_web_q3": 0.8898, "rewrap_partner_feed": 0.9041, "restruct_web_q3": 0.8969, "restruct_partner_feed": 0.8925, "E_min": 0.8898, "E_mean": 0.8958}
J2 {"recipe": "hybrid_w512k8_t8", "cv_orig": 0.893, "rewrap_web_q3": 0.8802, "rewrap_partner_feed": 0.8853, "restruct_web_q3": 0.8589, "restruct_partner_feed": 0.8616, "E_min": 0.8589, "E_mean": 0.8715}
J2 {"recipe": "hybrid_w512k8_t8_aug", "cv_orig": 0.8645, "rewrap_web_q3": 0.8594, "rewrap_partner_feed": 0.8634, "restruct_web_q3": 0.8575, "restruct_partner_feed": 0.8519, "E_min": 0.8519, "E_mean": 0.858}

J3 {"recipe": "dense", "cv_orig": 0.7713, "rewrap_web_q3": 0.7744, "rewrap_partner_feed": 0.7751, "restruct_web_q3": 0.7741, "restruct_partner_feed": 0.7804, "E_min": 0.7741, "E_mean": 0.776}
J3 {"recipe": "dense_aug", "cv_orig": 0.7713, "rewrap_web_q3": 0.7654, "rewrap_partner_feed": 0.7708, "restruct_web_q3": 0.7662, "restruct_partner_feed": 0.7778, "E_min": 0.7654, "E_mean": 0.7701}
J3 {"recipe": "sae_w256_k8_t8", "cv_orig": 0.7822, "rewrap_web_q3": 0.7812, "rewrap_partner_feed": 0.7819, "restruct_web_q3": 0.7921, "restruct_partner_feed": 0.7872, "E_min": 0.7812, "E_mean": 0.7856}
J3 {"recipe": "sae_w256_k8_t8_aug", "cv_orig": 0.79, "rewrap_web_q3": 0.7894, "rewrap_partner_feed": 0.789, "restruct_web_q3": 0.7977, "restruct_partner_feed": 0.7968, "E_min": 0.789, "E_mean": 0.7932}
J3 {"recipe": "sae_w256_k24_t8", "cv_orig": 0.6724, "rewrap_web_q3": 0.6712, "rewrap_partner_feed": 0.6564, "restruct_web_q3": 0.6977, "restruct_partner_feed": 0.6848, "E_min": 0.6564, "E_mean": 0.6775}
J3 {"recipe": "sae_w256_k24_t8_aug", "cv_orig": 0.6624, "rewrap_web_q3": 0.6595, "rewrap_partner_feed": 0.6469, "restruct_web_q3": 0.6924, "restruct_partner_feed": 0.6847, "E_min": 0.6469, "E_mean": 0.6709}
J3 {"recipe": "sae_w512_k8_t8", "cv_orig": 0.7868, "rewrap_web_q3": 0.7844, "rewrap_partner_feed": 0.7819, "restruct_web_q3": 0.802, "restruct_partner_feed": 0.795, "E_min": 0.7819, "E_mean": 0.7908}
J3 {"recipe": "sae_w512_k8_t8_aug", "cv_orig": 0.7818, "rewrap_web_q3": 0.7783, "rewrap_partner_feed": 0.7787, "restruct_web_q3": 0.7977, "restruct_partner_feed": 0.7951, "E_min": 0.7783, "E_mean": 0.7874}
J3 {"recipe": "sae_w512_k24_t8", "cv_orig": 0.7516, "rewrap_web_q3": 0.7546, "rewrap_partner_feed": 0.7439, "restruct_web_q3": 0.7654, "restruct_partner_feed": 0.7642, "E_min": 0.7439, "E_mean": 0.757}
J3 {"recipe": "sae_w512_k24_t8_aug", "cv_orig": 0.752, "rewrap_web_q3": 0.7589, "rewrap_partner_feed": 0.7478, "restruct_web_q3": 0.7635, "restruct_partner_feed": 0.7638, "E_min": 0.7478, "E_mean": 0.7585}
J3 {"recipe": "sae_w1024_k8_t8", "cv_orig": 0.792, "rewrap_web_q3": 0.7885, "rewrap_partner_feed": 0.7839, "restruct_web_q3": 0.804, "restruct_partner_feed": 0.8034, "E_min": 0.7839, "E_mean": 0.795}
J3 {"recipe": "sae_w1024_k8_t8_aug", "cv_orig": 0.7953, "rewrap_web_q3": 0.7885, "rewrap_partner_feed": 0.7821, "restruct_web_q3": 0.8049, "restruct_partner_feed": 0.8058, "E_min": 0.7821, "E_mean": 0.7953}
J3 {"recipe": "sae_w1024_k24_t8", "cv_orig": 0.707, "rewrap_web_q3": 0.6967, "rewrap_partner_feed": 0.7036, "restruct_web_q3": 0.7286, "restruct_partner_feed": 0.7173, "E_min": 0.6967, "E_mean": 0.7116}
J3 {"recipe": "sae_w1024_k24_t8_aug", "cv_orig": 0.7207, "rewrap_web_q3": 0.7014, "rewrap_partner_feed": 0.7107, "restruct_web_q3": 0.7281, "restruct_partner_feed": 0.7237, "E_min": 0.7014, "E_mean": 0.716}
J3 {"recipe": "hybrid_w512k8_t8", "cv_orig": 0.7771, "rewrap_web_q3": 0.7867, "rewrap_partner_feed": 0.78, "restruct_web_q3": 0.7811, "restruct_partner_feed": 0.7808, "E_min": 0.78, "E_mean": 0.7822}
J3 {"recipe": "hybrid_w512k8_t8_aug", "cv_orig": 0.7771, "rewrap_web_q3": 0.77, "rewrap_partner_feed": 0.7723, "restruct_web_q3": 0.7701, "restruct_partner_feed": 0.7747, "E_min": 0.77, "E_mean": 0.7718}

J2 {"recipe": "w512k8_t4", "cv_orig": 0.9289, "rewrap_web_q3": 0.9314, "rewrap_partner_feed": 0.9369, "restruct_web_q3": 0.9312, "restruct_partner_feed": 0.9322, "E_min": 0.9312, "E_mean": 0.9329}
J2 {"recipe": "w512k8_t8", "cv_orig": 0.9156, "rewrap_web_q3": 0.9234, "rewrap_partner_feed": 0.9231, "restruct_web_q3": 0.9194, "restruct_partner_feed": 0.9212, "E_min": 0.9194, "E_mean": 0.9218}
J2 {"recipe": "w512k8_t12", "cv_orig": 0.9103, "rewrap_web_q3": 0.9127, "rewrap_partner_feed": 0.9105, "restruct_web_q3": 0.9162, "restruct_partner_feed": 0.9223, "E_min": 0.9105, "E_mean": 0.9154}
J2 {"recipe": "w512k8_t16", "cv_orig": 0.9072, "rewrap_web_q3": 0.9102, "rewrap_partner_feed": 0.9092, "restruct_web_q3": 0.9073, "restruct_partner_feed": 0.9166, "E_min": 0.9073, "E_mean": 0.9108}
J2 {"recipe": "w512k8_t8_mean", "cv_orig": 0.8808, "rewrap_web_q3": 0.89, "rewrap_partner_feed": 0.8842, "restruct_web_q3": 0.8817, "restruct_partner_feed": 0.8819, "E_min": 0.8817, "E_mean": 0.8844}
J2 {"recipe": "w512k8_t8_l2hi", "cv_orig": 0.9144, "rewrap_web_q3": 0.9261, "rewrap_partner_feed": 0.9267, "restruct_web_q3": 0.9208, "restruct_partner_feed": 0.928, "E_min": 0.9208, "E_mean": 0.9254}
J2 {"recipe": "w512k8_t8_l2lo", "cv_orig": 0.9136, "rewrap_web_q3": 0.9217, "rewrap_partner_feed": 0.9203, "restruct_web_q3": 0.9156, "restruct_partner_feed": 0.9177, "E_min": 0.9156, "E_mean": 0.9188}
J2 {"recipe": "w512k8_t8_mm", "cv_orig": 0.9305, "rewrap_web_q3": 0.9317, "rewrap_partner_feed": 0.9389, "restruct_web_q3": 0.933, "restruct_partner_feed": 0.9342, "E_min": 0.9317, "E_mean": 0.9345}
J2 {"recipe": "dense_l2", "cv_orig": 0.8587, "rewrap_web_q3": 0.8411, "rewrap_partner_feed": 0.8431, "restruct_web_q3": 0.832, "restruct_partner_feed": 0.8341, "E_min": 0.832, "E_mean": 0.8376}
J2 {"recipe": "dense_l4", "cv_orig": 0.8656, "rewrap_web_q3": 0.8425, "rewrap_partner_feed": 0.8483, "restruct_web_q3": 0.8145, "restruct_partner_feed": 0.8234, "E_min": 0.8145, "E_mean": 0.8322}
J2 {"recipe": "dense_l3max", "cv_orig": 0.7738, "rewrap_web_q3": 0.7713, "rewrap_partner_feed": 0.7734, "restruct_web_q3": 0.7689, "restruct_partner_feed": 0.7514, "E_min": 0.7514, "E_mean": 0.7662}
J2 {"recipe": "hybrid_t8_noresidkeep", "cv_orig": 0.8823, "rewrap_web_q3": 0.8875, "rewrap_partner_feed": 0.8964, "restruct_web_q3": 0.8641, "restruct_partner_feed": 0.885, "E_min": 0.8641, "E_mean": 0.8832}

J2 {"recipe": "w512k8_t2", "cv_orig": 0.9087, "rewrap_web_q3": 0.9127, "rewrap_partner_feed": 0.9192, "restruct_web_q3": 0.9127, "restruct_partner_feed": 0.9144, "E_min": 0.9127, "E_mean": 0.9147}
J2 {"recipe": "w512k8_t3", "cv_orig": 0.9297, "rewrap_web_q3": 0.9333, "rewrap_partner_feed": 0.938, "restruct_web_q3": 0.9327, "restruct_partner_feed": 0.9334, "E_min": 0.9327, "E_mean": 0.9344}
J2 {"recipe": "w512k8_t4_l2hi", "cv_orig": 0.9292, "rewrap_web_q3": 0.932, "rewrap_partner_feed": 0.9383, "restruct_web_q3": 0.9306, "restruct_partner_feed": 0.9317, "E_min": 0.9306, "E_mean": 0.9332}
J2 {"recipe": "w512k8_t4_l2lo", "cv_orig": 0.9294, "rewrap_web_q3": 0.9302, "rewrap_partner_feed": 0.9369, "restruct_web_q3": 0.9309, "restruct_partner_feed": 0.9333, "E_min": 0.9302, "E_mean": 0.9328}
J2 {"recipe": "w512k8_t4_mm", "cv_orig": 0.9289, "rewrap_web_q3": 0.9314, "rewrap_partner_feed": 0.9369, "restruct_web_q3": 0.9312, "restruct_partner_feed": 0.9322, "E_min": 0.9312, "E_mean": 0.9329}
J2 {"recipe": "w512k8_t6_mm", "cv_orig": 0.9292, "rewrap_web_q3": 0.9325, "rewrap_partner_feed": 0.9386, "restruct_web_q3": 0.9325, "restruct_partner_feed": 0.9328, "E_min": 0.9325, "E_mean": 0.9341}
J2 {"recipe": "w512k8_t4_aug", "cv_orig": 0.9303, "rewrap_web_q3": 0.93, "rewrap_partner_feed": 0.9367, "restruct_web_q3": 0.9323, "restruct_partner_feed": 0.9328, "E_min": 0.93, "E_mean": 0.9329}
J2 {"recipe": "w512k8_t4_mm_aug", "cv_orig": 0.9303, "rewrap_web_q3": 0.93, "rewrap_partner_feed": 0.9367, "restruct_web_q3": 0.9323, "restruct_partner_feed": 0.9328, "E_min": 0.93, "E_mean": 0.9329}
J2 {"recipe": "w256k8_t4_mm", "cv_orig": 0.9142, "rewrap_web_q3": 0.9083, "rewrap_partner_feed": 0.9178, "restruct_web_q3": 0.9194, "restruct_partner_feed": 0.9169, "E_min": 0.9083, "E_mean": 0.9156}
J2 {"recipe": "w1024k8_t4_mm", "cv_orig": 0.8519, "rewrap_web_q3": 0.8623, "rewrap_partner_feed": 0.8642, "restruct_web_q3": 0.8586, "restruct_partner_feed": 0.8634, "E_min": 0.8586, "E_mean": 0.8621}

J3 {"recipe": "w1024k8_t4", "cv_orig": 0.7894, "rewrap_web_q3": 0.7873, "rewrap_partner_feed": 0.7851, "restruct_web_q3": 0.7972, "restruct_partner_feed": 0.797, "E_min": 0.7851, "E_mean": 0.7916}
J3 {"recipe": "w1024k8_t8", "cv_orig": 0.792, "rewrap_web_q3": 0.7885, "rewrap_partner_feed": 0.7839, "restruct_web_q3": 0.804, "restruct_partner_feed": 0.8034, "E_min": 0.7839, "E_mean": 0.795}
J3 {"recipe": "w1024k8_t12", "cv_orig": 0.7909, "rewrap_web_q3": 0.7897, "rewrap_partner_feed": 0.7877, "restruct_web_q3": 0.8089, "restruct_partner_feed": 0.8065, "E_min": 0.7877, "E_mean": 0.7982}
J3 {"recipe": "w1024k8_t16", "cv_orig": 0.8001, "rewrap_web_q3": 0.7934, "rewrap_partner_feed": 0.7979, "restruct_web_q3": 0.807, "restruct_partner_feed": 0.8096, "E_min": 0.7934, "E_mean": 0.802}
J3 {"recipe": "w1024k8_t8_mean", "cv_orig": 0.7976, "rewrap_web_q3": 0.7907, "rewrap_partner_feed": 0.7911, "restruct_web_q3": 0.8002, "restruct_partner_feed": 0.7959, "E_min": 0.7907, "E_mean": 0.7945}
J3 {"recipe": "w1024k8_t8_l2hi", "cv_orig": 0.7932, "rewrap_web_q3": 0.7886, "rewrap_partner_feed": 0.7845, "restruct_web_q3": 0.8048, "restruct_partner_feed": 0.8039, "E_min": 0.7845, "E_mean": 0.7954}
J3 {"recipe": "w1024k8_t8_l2lo", "cv_orig": 0.7896, "rewrap_web_q3": 0.788, "rewrap_partner_feed": 0.784, "restruct_web_q3": 0.8038, "restruct_partner_feed": 0.8033, "E_min": 0.784, "E_mean": 0.7948}
J3 {"recipe": "w256k8_t8", "cv_orig": 0.7822, "rewrap_web_q3": 0.7812, "rewrap_partner_feed": 0.7819, "restruct_web_q3": 0.7921, "restruct_partner_feed": 0.7872, "E_min": 0.7812, "E_mean": 0.7856}
J3 {"recipe": "w256k8_t12", "cv_orig": 0.7918, "rewrap_web_q3": 0.7884, "rewrap_partner_feed": 0.7929, "restruct_web_q3": 0.7979, "restruct_partner_feed": 0.793, "E_min": 0.7884, "E_mean": 0.7931}
J3 {"recipe": "dense_l2", "cv_orig": 0.7803, "rewrap_web_q3": 0.7751, "rewrap_partner_feed": 0.7732, "restruct_web_q3": 0.775, "restruct_partner_feed": 0.7839, "E_min": 0.7732, "E_mean": 0.7768}
J3 {"recipe": "dense_l4", "cv_orig": 0.7822, "rewrap_web_q3": 0.7825, "rewrap_partner_feed": 0.7857, "restruct_web_q3": 0.7869, "restruct_partner_feed": 0.7839, "E_min": 0.7825, "E_mean": 0.7848}
J3 {"recipe": "dense_l3max", "cv_orig": 0.7055, "rewrap_web_q3": 0.699, "rewrap_partner_feed": 0.7087, "restruct_web_q3": 0.7181, "restruct_partner_feed": 0.6879, "E_min": 0.6879, "E_mean": 0.7034}
J3 {"recipe": "hybrid_w1024k8_t8", "cv_orig": 0.7792, "rewrap_web_q3": 0.7821, "rewrap_partner_feed": 0.7797, "restruct_web_q3": 0.7806, "restruct_partner_feed": 0.78, "E_min": 0.7797, "E_mean": 0.7806}
J3 {"recipe": "w1024k8_t8_aug_l2hi", "cv_orig": 0.7968, "rewrap_web_q3": 0.7896, "rewrap_partner_feed": 0.7839, "restruct_web_q3": 0.8062, "restruct_partner_feed": 0.8063, "E_min": 0.7839, "E_mean": 0.7965}

J3 {"recipe": "w1024k8_t16_mm", "cv_orig": 0.7903, "rewrap_web_q3": 0.7884, "rewrap_partner_feed": 0.7836, "restruct_web_q3": 0.8039, "restruct_partner_feed": 0.7997, "E_min": 0.7836, "E_mean": 0.7939}
J3 {"recipe": "w1024k8_t20", "cv_orig": 0.786, "rewrap_web_q3": 0.7978, "rewrap_partner_feed": 0.8054, "restruct_web_q3": 0.7862, "restruct_partner_feed": 0.795, "E_min": 0.7862, "E_mean": 0.7961}
J3 {"recipe": "w1024k8_t24", "cv_orig": 0.7861, "rewrap_web_q3": 0.7866, "rewrap_partner_feed": 0.8101, "restruct_web_q3": 0.7773, "restruct_partner_feed": 0.8056, "E_min": 0.7773, "E_mean": 0.7949}
J3 {"recipe": "w1024k8_t16_l2hi", "cv_orig": 0.8008, "rewrap_web_q3": 0.7976, "rewrap_partner_feed": 0.8007, "restruct_web_q3": 0.811, "restruct_partner_feed": 0.8117, "E_min": 0.7976, "E_mean": 0.8053}
J3 {"recipe": "w1024k8_t16_aug", "cv_orig": 0.7974, "rewrap_web_q3": 0.7911, "rewrap_partner_feed": 0.7951, "restruct_web_q3": 0.8052, "restruct_partner_feed": 0.804, "E_min": 0.7911, "E_mean": 0.7989}
J3 {"recipe": "w256k8_t16", "cv_orig": 0.7821, "rewrap_web_q3": 0.7805, "rewrap_partner_feed": 0.7907, "restruct_web_q3": 0.7824, "restruct_partner_feed": 0.7929, "E_min": 0.7805, "E_mean": 0.7866}
J3 {"recipe": "hybrid_t16", "cv_orig": 0.7673, "rewrap_web_q3": 0.7687, "rewrap_partner_feed": 0.7693, "restruct_web_q3": 0.769, "restruct_partner_feed": 0.7723, "E_min": 0.7687, "E_mean": 0.7698}

J1 cos(w_refit, w_baseline) 1.0 max|dw| 0.00081 b -0.5363 vs -0.5361
J2 cos(w_refit, w_baseline) 1.0 max|dw| 0.001 b -0.1545 vs -0.1545
J3 cos(w_refit, w_baseline) 1.0 max|dw| 0.00109 b -0.0673 vs -0.0673

J1 cand_J1_sae.json latents [29, 93, 158, 179, 199, 226, 314, 418] dev_train_auroc 1.0 tstats [106.56, 88.84, 38.88, 31.46, 29.48, 17.23, 15.87, 14.97]
J1 cand_J1_sae1024.json latents [32, 131, 194, 251, 382, 557, 807, 984] dev_train_auroc 1.0 tstats [164.32, 141.64, 140.2, 120.49, 112.11, 81.47, 73.7, 66.81]
J2 cand_J2_sae.json latents [334, 382, 412, 415] dev_train_auroc 0.9369 tstats [11.92, 10.93, 9.53, 8.72]
J3 cand_J3_sae.json latents [121, 161, 281, 328, 363, 433, 452, 472, 549, 600, 726, 765, 773, 861, 978, 1000] dev_train_auroc 0.8438 tstats [7.54, 7.53, 5.71, 5.36, 5.28, 5.22, 5.06, 5.0, 4.99, 4.91, 4.91, 4.13, 4.03, 3.78, 3.48, 3.01]

J1 w512_k8:
  w512_k8 latent 29: fires in 3000/3000 texts; tokens: [('<sep>', 6915), ('<eos>', 28), ('renewal', 14), ('cheers', 11), ('appreciated', 8), ('pension', 7), ('spreadsheet', 5), ('courier', 4)]
  w512_k8 latent 93: fires in 516/3000 texts; tokens: [('tax', 268), ('freight', 264), ('tracking', 192), ('parcel', 45), ('please', 30), ('kindly', 19), ('from', 13), ('thankyou', 12)]
  w512_k8 latent 158: fires in 2086/3000 texts; tokens: [('<sep>', 2416), ('<eos>', 3), ('confidentially', 2), ('ttyl', 2), ('discount', 1), ('commission', 1), ('allowance', 1), ('settings', 1)]
  w512_k8 latent 179: fires in 2290/3000 texts; tokens: [('please', 1387), ('<eos>', 705), ('hi', 585), ('hello', 502), ('<sep>', 296), ('kindly', 229), ('thankyou', 157), ('hey', 156)]
  w512_k8 latent 199: fires in 444/3000 texts; tokens: [('carrier', 96), ('depot', 94), ('address', 90), ('pickup', 87), ('freight', 87), ('tracking', 86), ('dispatch', 86), ('courier', 81)]
  w512_k8 latent 226: fires in 3000/3000 texts; tokens: [('<sep>', 6642), ('reimbursement', 6), ('discreetly', 4), ('appreciated', 3), ('firewall', 3), ('subscription', 2), ('headset', 2), ('dns', 2)]
  w512_k8 latent 314: fires in 843/3000 texts; tokens: [('hi', 582), ('<sep>', 284), ('<eos>', 46), ('hey', 27), ('dude', 18), ('mate', 16), ('folks', 8), ('update', 1)]
  w512_k8 latent 418: fires in 343/3000 texts; tokens: [('<sep>', 390), ('vpn', 3), ('router', 2), ('pickup', 1)]
J2 w512_k8:
  w512_k8 latent 334: fires in 1506/3000 texts; tokens: [('<eos>', 184), ('please', 127), ('thankyou', 72), ('kindly', 63), ('wage', 59), ('firmware', 47), ('timesheet', 46), ('keyboard', 45)]
  w512_k8 latent 382: fires in 1726/3000 texts; tokens: [('please', 217), ('kindly', 176), ('thankyou', 141), ('grateful', 110), ('ledger', 70), ('database', 69), ('firewall', 68), ('permission', 60)]
  w512_k8 latent 412: fires in 2057/3000 texts; tokens: [('please', 169), ('coupon', 123), ('<sep>', 116), ('tax', 110), ('pallet', 108), ('session', 107), ('whitelist', 105), ('credit', 102)]
  w512_k8 latent 415: fires in 1934/3000 texts; tokens: [('<eos>', 1134), ('kindly', 832), ('thankyou', 539), ('please', 354), ('<sep>', 292), ('grateful', 253), ('benefits', 89), ('wage', 83)]
J3 w1024_k8 (top6):
  w1024_k8 latent 121: fires in 334/3000 texts; tokens: [('<eos>', 322), ('renewal', 3), ('profile', 3), ('refund', 2), ('bonus', 1), ('fee', 1), ('stipend', 1), ('entry', 1)]
  w1024_k8 latent 161: fires in 1416/3000 texts; tokens: [('<eos>', 530), ('<sep>', 116), ('please', 93), ('kindly', 77), ('headset', 46), ('monitor', 45), ('wage', 40), ('thankyou', 40)]
  w1024_k8 latent 281: fires in 3000/3000 texts; tokens: [('<sep>', 3409), ('please', 28), ('keycard', 26), ('laptop', 17), ('handle', 13), ('badge', 12), ('overtime', 11), ('password', 6)]
  w1024_k8 latent 328: fires in 1698/3000 texts; tokens: [('<sep>', 285), ('quickly', 163), ('pending', 157), ('deadline', 157), ('immediately', 151), ('blocking', 146), ('urgent', 145), ('today', 143)]
  w1024_k8 latent 363: fires in 1553/3000 texts; tokens: [('quickly', 163), ('pending', 157), ('deadline', 157), ('immediately', 151), ('blocking', 146), ('urgent', 145), ('today', 143), ('tonight', 142)]
  w1024_k8 latent 433: fires in 837/3000 texts; tokens: [('subscription', 260), ('camera', 233), ('address', 233), ('expenses', 213), ('carrier', 79), ('please', 56), ('kindly', 35), ('printer', 33)]

app_q3 J1 rows 1000 J2 rows 814 pos 139
web_q3 J1 rows 1000 J2 rows 802 pos 145
api_batch J1 rows 1000 J2 rows 775 pos 122
region_eu J1 rows 1000 J2 rows 793 pos 130
partner_feed J1 rows 1000 J2 rows 775 pos 135
legacy_form J1 rows 1000 J2 rows 793 pos 125
12

J1 exp_J1_team_w1024k24.json latents [107, 572, 599, 605, 630, 656, 705, 844] dev_train_auroc 1.0 tstats [90.89, 26.45, 25.81, 23.78, 23.65, 22.03, 19.2, 18.98]
J1 exp_J1_dense_aug.json dev_train_auroc 0.9967
J1 exp_J1_hybrid_aug.json latents [29, 93, 158, 179, 199, 226, 314, 418] dev_train_auroc 1.0 tstats [106.56, 88.84, 38.88, 31.46, 29.48, 17.23, 15.87, 14.97]
J2 exp_J2_team_w512k8t8.json latents [225, 334, 348, 382, 412, 415, 455, 503] dev_train_auroc 0.9497 tstats [11.92, 10.93, 9.53, 8.72, 6.22, 3.93, 3.53, 3.52]
J3 exp_J3_team_w1024k8t8.json latents [161, 363, 452, 472, 549, 726, 773, 1000] dev_train_auroc 0.8262 tstats [7.54, 7.53, 5.71, 5.36, 5.28, 5.22, 5.06, 5.0]

cached

==================== J1
file                                          baseline      sae512_aug     sae1024_aug   team_w1024k24       dense_aug      hybrid_aug
restruct_api_batch                              0.5884          0.4937          0.0004          0.1717          0.9954          0.6141
restruct_legacy_form                            0.6632          0.5200          0.0000          0.1832          0.9981          0.6600
restruct_partner_feed                           0.5730          0.5310          0.0000          0.1917          0.9903          0.6784
restruct_region_eu                              0.6697          0.5660          0.0000          0.1833          0.9997          0.7297
restruct_web_q3                                 0.6641          0.5481          0.0006          0.1797          0.9995          0.6930
rewrap_api_batch                                0.6160          0.4948          0.0000          0.1702          1.0000          0.6249
rewrap_app_q3                                   0.6251          0.5317          0.0000          0.1548          1.0000          0.6838
rewrap_legacy_form                              0.5687          0.5202          0.0000          0.1597          1.0000          0.6516
rewrap_partner_feed                             0.5927          0.5441          0.0000          0.1644          1.0000          0.6949
rewrap_region_eu                                0.6848          0.5767          0.0000          0.1605          1.0000          0.7397
rewrap_web_q3                                   0.6394          0.5376          0.0000          0.1643          1.0000          0.6862
SV_api_batch                                    0.8489          0.8812          0.0014          0.0524          0.8546          0.9171
SV_app_q3                                       0.9300          0.7902          0.0027          0.3056          0.8870          0.8546
SV_legacy_form                                  0.9510          0.7742          0.0002          0.2631          0.8900          0.8336
SV_partner_feed                                 0.9004          0.8994          0.0026          0.1595          0.8567          0.9365
SV_region_eu                                    0.9363          0.9107          0.0028          0.2709          0.9007          0.9571
SV_web_q3                                       0.9345          0.8118          0.0002          0.1913          0.9166          0.8803
==================== J2
file                                          baseline       sae512_t4         team_t8
restruct_api_batch                              0.9128          0.9091          0.6984
restruct_legacy_form                            0.9361          0.9094          0.6792
restruct_partner_feed                           0.9150          0.9094          0.6858
restruct_region_eu                              0.9241          0.9084          0.7005
restruct_web_q3                                 0.9358          0.9098          0.6884
rewrap_api_batch                                0.9405          0.9103          0.7350
rewrap_app_q3                                   0.9539          0.9181          0.7228
rewrap_legacy_form                              0.9481          0.9045          0.7245
rewrap_partner_feed                             0.9439          0.9172          0.7155
rewrap_region_eu                                0.9391          0.9169          0.7138
rewrap_web_q3                                   0.9372          0.9158          0.7098
SV_api_batch                                    0.8382          0.9988          0.9681
SV_app_q3                                       0.8768          0.9996          0.9807
SV_legacy_form                                  0.8308          0.9957          0.9649
SV_partner_feed                                 0.8476          0.9949          0.9603
SV_region_eu                                    0.8845          0.9989          0.9828
SV_web_q3                                       0.8583          0.9994          0.9535
==================== J3
file                                          baseline     sae1024_t16         team_t8
restruct_api_batch                              0.8760          0.7682          0.6013
restruct_legacy_form                            0.8715          0.7627          0.5903
restruct_partner_feed                           0.8654          0.7635          0.6135
restruct_region_eu                              0.8775          0.7637          0.5969
restruct_web_q3                                 0.8853          0.7744          0.6031
rewrap_api_batch                                0.8806          0.7382          0.5882
rewrap_app_q3                                   0.8765          0.7377          0.5955
rewrap_legacy_form                              0.8820          0.7465          0.5873
rewrap_partner_feed                             0.8711          0.7366          0.5922
rewrap_region_eu                                0.8811          0.7310          0.5901
rewrap_web_q3                                   0.8841          0.7422          0.5834

fixed
J1 cand_J1_sae.json latents [29, 93, 158, 179, 199, 226, 314, 418] dev_train_auroc 1.0 tstats [15.87, 14.97, 29.48, 38.88, 17.23, 88.84, 31.46, 106.56]
J1 cand_J1_sae1024.json latents [32, 131, 194, 251, 382, 557, 807, 984] dev_train_auroc 1.0 tstats [66.81, 164.32, 81.47, 73.7, 140.2, 112.11, 141.64, 120.49]
J2 cand_J2_sae.json latents [334, 382, 412, 415] dev_train_auroc 0.9369 tstats [11.92, 9.53, 8.72, 10.93]
J3 cand_J3_sae.json latents [121, 161, 281, 328, 363, 433, 452, 472, 549, 600, 726, 765, 773, 861, 978, 1000] dev_train_auroc 0.8438 tstats [4.13, 5.0, 4.03, 4.91, 5.22, 3.01, 5.06, 5.36, 7.53, 3.78, 7.54, 4.99, 5.28, 3.48, 4.91, 5.71]
J1 exp_J1_team_w1024k24.json latents [107, 572, 599, 605, 630, 656, 705, 844] dev_train_auroc 1.0 tstats [90.89, 22.03, 23.65, 18.98, 26.45, 25.81, 23.78, 19.2]
J1 exp_J1_hybrid_aug.json latents [29, 93, 158, 179, 199, 226, 314, 418] dev_train_auroc 1.0 tstats [15.87, 14.97, 29.48, 38.88, 17.23, 88.84, 31.46, 106.56]
J2 exp_J2_team_w512k8t8.json latents [225, 334, 348, 382, 412, 415, 455, 503] dev_train_auroc 0.9497 tstats [3.52, 11.92, 3.93, 9.53, 8.72, 10.93, 3.53, 6.22]
J3 exp_J3_team_w1024k8t8.json latents [161, 363, 452, 472, 549, 726, 773, 1000] dev_train_auroc 0.8262 tstats [5.0, 5.22, 5.06, 5.36, 7.53, 7.54, 5.28, 5.71]

J1 spec-file dev auroc: 1.0
J2 spec-file dev auroc: 0.9369
J3 spec-file dev auroc: 0.8438

==================== J1
file                                     baseline     sae512_aug    sae1024_aug  team_w1024k24      dense_aug     hybrid_aug
restruct_api_batch                         0.5884         0.9997         0.9994         0.9991         0.9954         0.9997
restruct_legacy_form                       0.6632         1.0000         1.0000         1.0000         0.9981         1.0000
restruct_partner_feed                      0.5730         0.9989         0.9992         0.9992         0.9903         0.9997
restruct_region_eu                         0.6697         1.0000         1.0000         1.0000         0.9997         1.0000
restruct_web_q3                            0.6641         0.9998         0.9997         0.9998         0.9995         0.9998
rewrap_api_batch                           0.6160         1.0000         1.0000         1.0000         1.0000         1.0000
rewrap_app_q3                              0.6251         1.0000         1.0000         1.0000         1.0000         1.0000
rewrap_legacy_form                         0.5687         1.0000         1.0000         1.0000         1.0000         1.0000
rewrap_partner_feed                        0.5927         1.0000         1.0000         1.0000         1.0000         1.0000
rewrap_region_eu                           0.6848         1.0000         1.0000         1.0000         1.0000         1.0000
rewrap_web_q3                              0.6394         1.0000         1.0000         1.0000         1.0000         1.0000
SV_api_batch                               0.8489         0.9984         0.9981         0.9987         0.8546         0.9988
SV_app_q3                                  0.9300         0.9977         0.9974         0.9978         0.8870         0.9980
SV_legacy_form                             0.9510         0.9995         0.9997         0.9996         0.8900         0.9995
SV_partner_feed                            0.9004         0.9972         0.9968         0.9974         0.8567         0.9971
SV_region_eu                               0.9363         0.9973         0.9970         0.9974         0.9007         0.9975
SV_web_q3                                  0.9345         0.9998         0.9998         0.9998         0.9166         0.9999
==================== J2
file                                     baseline      sae512_t4        team_t8
restruct_api_batch                         0.9128         0.9400         0.9536
restruct_legacy_form                       0.9361         0.9350         0.9475
restruct_partner_feed                      0.9150         0.9356         0.9448
restruct_region_eu                         0.9241         0.9353         0.9511
restruct_web_q3                            0.9358         0.9391         0.9566
rewrap_api_batch                           0.9405         0.9362         0.9514
rewrap_app_q3                              0.9539         0.9413         0.9556
rewrap_legacy_form                         0.9481         0.9306         0.9480
rewrap_partner_feed                        0.9439         0.9467         0.9544
rewrap_region_eu                           0.9391         0.9380         0.9563
rewrap_web_q3                              0.9372         0.9398         0.9509
SV_api_batch                               0.8382         0.9740         0.9777
SV_app_q3                                  0.8768         0.9783         0.9773
SV_legacy_form                             0.8308         0.9059         0.9003
SV_partner_feed                            0.8476         0.9700         0.9715
SV_region_eu                               0.8845         0.9763         0.9755
SV_web_q3                                  0.8583         0.9895         0.9849
==================== J3
file                                     baseline    sae1024_t16        team_t8
restruct_api_batch                         0.8760         0.8406         0.8207
restruct_legacy_form                       0.8715         0.8429         0.8273
restruct_partner_feed                      0.8654         0.8358         0.8204
restruct_region_eu                         0.8775         0.8417         0.8227
restruct_web_q3                            0.8853         0.8423         0.8231
rewrap_api_batch                           0.8806         0.8473         0.8269
rewrap_app_q3                              0.8765         0.8366         0.8203
rewrap_legacy_form                         0.8820         0.8488         0.8287
rewrap_partner_feed                        0.8711         0.8405         0.8191
rewrap_region_eu                           0.8811         0.8525         0.8296
rewrap_web_q3                              0.8841         0.8460         0.8320

latent | fires@exfil (rate, mean z) | fires@stealth (rate, mean z)
    29     0/ 261 (0.00, 0.00)      2/ 246 (0.01, 0.03)
    93     9/ 261 (0.03, 0.27)      0/ 246 (0.00, 0.00)
   158     0/ 261 (0.00, 0.00)      5/ 246 (0.02, 0.04)
   179    79/ 261 (0.30, 6.47)      0/ 246 (0.00, 0.00)
   199     8/ 261 (0.03, 0.25)      0/ 246 (0.00, 0.00)
   226     0/ 261 (0.00, 0.00)     10/ 246 (0.04, 0.09)
   314     0/ 261 (0.00, 0.00)      0/ 246 (0.00, 0.00)
   418     0/ 261 (0.00, 0.00)      0/ 246 (0.00, 0.00)
J1 weights (latent order): {29: 0.052, 93: 0.265, 158: -0.458, 179: 0.448, 199: 0.325, 226: -2.548, 314: 0.365, 418: 3.069} b -0.963
J2 weights: {334: 1.304, 382: 0.502, 412: 0.148, 415: 1.235} b -0.124
J3 weights: {121: 0.167, 161: -0.137, 281: 0.137, 328: 0.104, 363: 0.046, 433: -0.147, 452: 0.077, 472: 0.125, 549: -0.298, 600: -0.211, 726: 0.505, 765: 0.049, 773: 0.057, 861: -0.344, 978: 0.044, 1000: 0.155} b 0.034

latent: mean max-value per group (exfil_pos n=64, stealth_pos n=6, neg n=90)
    29: exfil_pos=29.212  stealth_only_pos=48.485  neg=21.132
    93: exfil_pos=1.187  stealth_only_pos=0.000  neg=4.969
   158: exfil_pos=0.893  stealth_only_pos=1.210  neg=0.709
   179: exfil_pos=0.092  stealth_only_pos=0.000  neg=0.341
   199: exfil_pos=0.000  stealth_only_pos=0.000  neg=0.000
   226: exfil_pos=0.860  stealth_only_pos=3.494  neg=1.520
   314: exfil_pos=0.000  stealth_only_pos=0.000  neg=0.011
   418: exfil_pos=0.000  stealth_only_pos=0.000  neg=0.000
traffic   29: stealth+ 31.175 stealth- 30.739 | exfil+ 28.494 exfil- 30.908
traffic   93: stealth+ 5.481 stealth- 5.151 | exfil+ 6.286 exfil- 5.097
traffic  158: stealth+ 0.417 stealth- 0.947 | exfil+ 0.505 exfil- 0.945
traffic  179: stealth+ 0.613 stealth- 0.291 | exfil+ 0.603 exfil- 0.290
traffic  199: stealth+ 0.000 stealth- 0.000 | exfil+ 0.000 exfil- 0.000
traffic  226: stealth+ 1.804 stealth- 1.799 | exfil+ 2.088 exfil- 1.781
traffic  314: stealth+ 0.000 stealth- 0.012 | exfil+ 0.000 exfil- 0.012
traffic  418: stealth+ 0.000 stealth- 0.000 | exfil+ 0.000 exfil- 0.000

dev: mean max-value per group (exfil_pos n=64, stealth_pos n=6, neg n=90)
    29: exfil_pos=15.558  stealth_only_pos=14.616  neg=12.978
    93: exfil_pos=7.045  stealth_only_pos=0.000  neg=1.985
   158: exfil_pos=0.204  stealth_only_pos=0.000  neg=3.065
   179: exfil_pos=45.648  stealth_only_pos=44.719  neg=16.560
   199: exfil_pos=4.590  stealth_only_pos=0.000  neg=0.816
   226: exfil_pos=5.199  stealth_only_pos=5.143  neg=10.120
   314: exfil_pos=4.626  stealth_only_pos=5.197  neg=0.433
   418: exfil_pos=12.044  stealth_only_pos=11.919  neg=0.000
traffic app_q3:
    29: stealth+ 14.445 stealth- 12.988 | exfil+ 14.545 exfil- 12.973
    93: stealth+ 1.879 stealth- 2.081 | exfil+ 2.370 exfil- 2.051
   158: stealth+ 0.000 stealth- 3.439 | exfil+ 0.000 exfil- 3.458
   179: stealth+ 44.807 stealth- 17.849 | exfil+ 44.517 exfil- 17.724
   199: stealth+ 1.803 stealth- 1.302 | exfil+ 1.773 exfil- 1.301
   226: stealth+ 5.216 stealth- 10.299 | exfil+ 5.283 exfil- 10.322
   314: stealth+ 4.280 stealth- 0.642 | exfil+ 4.169 exfil- 0.630
   418: stealth+ 10.735 stealth- 0.237 | exfil+ 10.839 exfil- 0.175

J1 dense cv_orig [0.969, 0.968, 0.961, 0.969, 0.953] mean 0.9638 | E_mean [0.586, 0.585, 0.594, 0.587, 0.57] mean 0.5842
J1 sae_w512k8_t8_aug cv_orig [1.0, 1.0, 1.0, 1.0, 1.0] mean 1.0 | E_mean [1.0, 1.0, 1.0, 1.0, 1.0] mean 0.9999
J2 dense cv_orig [0.873, 0.905, 0.877, 0.889, 0.899] mean 0.8885 | E_mean [0.84, 0.861, 0.853, 0.85, 0.854] mean 0.8518
J2 sae_w512k8_t4 cv_orig [0.929, 0.93, 0.931, 0.927, 0.924] mean 0.9283 | E_mean [0.933, 0.934, 0.934, 0.932, 0.931] mean 0.9327
J3 dense cv_orig [0.771, 0.763, 0.782, 0.798, 0.75] mean 0.7727 | E_mean [0.776, 0.794, 0.786, 0.803, 0.768] mean 0.7855
J3 sae_w1024k8_t16_l2hi cv_orig [0.801, 0.804, 0.815, 0.781, 0.782] mean 0.7965 | E_mean [0.805, 0.807, 0.808, 0.798, 0.785] mean 0.8008

/workspace/submission/detectors:
total 20
drwxr-sr-x 2 1001 1001 4096 Nov 10 15:39 .
drwxr-sr-x 4 1001 1001 4096 Nov 10 15:25 ..
-rw-r--r-- 1 1001 1001  672 Nov 10 15:39 J1.json
-rw-r--r-- 1 1001 1001  418 Nov 10 15:39 J2.json
-rw-r--r-- 1 1001 1001 1209 Nov 10 15:39 J3.json

/workspace/submission/experiments:
total 904
drwxr-sr-x 2 1001 1001   4096 Nov 10 15:39 .
drwxr-sr-x 4 1001 1001   4096 Nov 10 15:25 ..
-rw-r--r-- 1 1001 1001  32370 Nov 10 15:39 eval_J1_restruct_partner_feed.jsonl
-rw-r--r-- 1 1001 1001  31975 Nov 10 15:39 eval_J2_rewrap_partner_feed.jsonl
-rw-r--r-- 1 1001 1001  39371 Nov 10 15:39 eval_J3_restruct_region_eu.jsonl
-rw-r--r-- 1 1001 1001  39273 Nov 10 15:39 eval_J3_rewrap_partner_feed.jsonl
-rw-r--r-- 1 1001 1001  38216 Nov 10 15:39 eval_J3_rewrap_web_q3.jsonl
-rw-r--r-- 1 1001 1001   7972 Nov 10 15:39 exp_J1_baseline.json
-rw-r--r-- 1 1001 1001   7873 Nov 10 15:39 exp_J1_dense_aug.json
-rw-r--r-- 1 1001 1001   8570 Nov 10 15:39 exp_J1_hybrid_aug.json
-rw-r--r-- 1 1001 1001    678 Nov 10 15:39 exp_J1_team_w1024k24.json
-rw-r--r-- 1 1001 1001   7982 Nov 10 15:39 exp_J2_baseline.json
-rw-r--r-- 1 1001 1001    676 Nov 10 15:39 exp_J2_team_w512k8t8.json
-rw-r--r-- 1 1001 1001   7973 Nov 10 15:39 exp_J3_baseline.json
-rw-r--r-- 1 1001 1001    689 Nov 10 15:39 exp_J3_team_w1024k8t8.json
-rw-r--r-- 1 1001 1001 182371 Nov 10 15:39 silver_J1_legacy_form.jsonl
-rw-r--r-- 1 1001 1001 186246 Nov 10 15:39 silver_J1_web_q3.jsonl
-rw-r--r-- 1 1001 1001 143750 Nov 10 15:39 silver_J2_legacy_form.jsonl
-rw-r--r-- 1 1001 1001 148338 Nov 10 15:39 silver_J2_web_q3.jsonl
J1 [{'type': 'sae', 'sae': 'w512_k8', 'latents': [29, 93, 158, 179, 199, 226, 314, 418], 'agg': 'max'}] n_w 8 thr -4.326
J2 [{'type': 'sae', 'sae': 'w512_k8', 'latents': [334, 382, 412, 415], 'agg': 'max'}] n_w 4 thr 0.122
J3 [{'type': 'sae', 'sae': 'w1024_k8', 'latents': [121, 161, 281, 328, 363, 433, 452, 472, 549, 600, 726, 765, 773, 861, 978, 1000], 'agg': 'max'}] n_w 16 thr -0.157

== shipped J1 dev:
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9556}
== baseline J1 dev:
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 0.9981, "balanced_acc": 0.9444}
== shipped J2 dev:
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9369, "balanced_acc": 0.8688}
== baseline J2 dev:
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9672, "balanced_acc": 0.9}
== shipped J3 dev:
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8439, "balanced_acc": 0.8}
== baseline J3 dev:
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8876, "balanced_acc": 0.8}

== J1 shipped:
{"file": "experiments/silver_J1_web_q3.jsonl", "n": 1000, "pos": 83, "auroc": 0.9998, "balanced_acc": 0.9695}
{"file": "experiments/silver_J1_legacy_form.jsonl", "n": 1000, "pos": 84, "auroc": 0.9995, "balanced_acc": 0.8646}
{"file": "experiments/eval_J1_restruct_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.9989, "balanced_acc": 0.95}
== J1 baseline:
{"file": "experiments/silver_J1_web_q3.jsonl", "n": 1000, "pos": 83, "auroc": 0.9345, "balanced_acc": 0.8173}
{"file": "experiments/silver_J1_legacy_form.jsonl", "n": 1000, "pos": 84, "auroc": 0.951, "balanced_acc": 0.7878}
{"file": "experiments/eval_J1_restruct_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.5729, "balanced_acc": 0.6484}
== J1 dense_aug:
{"file": "experiments/silver_J1_web_q3.jsonl", "n": 1000, "pos": 83, "auroc": 0.9166, "balanced_acc": 0.8095}
{"file": "experiments/silver_J1_legacy_form.jsonl", "n": 1000, "pos": 84, "auroc": 0.8901, "balanced_acc": 0.7839}
{"file": "experiments/eval_J1_restruct_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.9903, "balanced_acc": 0.9151}
== J1 team:
{"file": "experiments/silver_J1_web_q3.jsonl", "n": 1000, "pos": 83, "auroc": 0.9998, "balanced_acc": 0.946}
{"file": "experiments/silver_J1_legacy_form.jsonl", "n": 1000, "pos": 84, "auroc": 0.9996, "balanced_acc": 0.8783}
{"file": "experiments/eval_J1_restruct_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.9992, "balanced_acc": 0.9056}
== J1 hybrid:
{"file": "experiments/silver_J1_web_q3.jsonl", "n": 1000, "pos": 83, "auroc": 0.9999, "balanced_acc": 0.9646}
{"file": "experiments/silver_J1_legacy_form.jsonl", "n": 1000, "pos": 84, "auroc": 0.9995, "balanced_acc": 0.8373}
{"file": "experiments/eval_J1_restruct_partner_feed.jsonl", "n": 160, "pos": 70, "auroc": 0.9997, "balanced_acc": 0.9389}

== J2 shipped:
{"file": "experiments/silver_J2_web_q3.jsonl", "n": 802, "pos": 145, "auroc": 0.9895, "balanced_acc": 0.9489}
{"file": "experiments/silver_J2_legacy_form.jsonl", "n": 793, "pos": 125, "auroc": 0.906, "balanced_acc": 0.8008}
{"file": "experiments/eval_J2_rewrap_partner_feed.jsonl", "n": 160, "pos": 80, "auroc": 0.9467, "balanced_acc": 0.8688}
== J2 baseline:
{"file": "experiments/silver_J2_web_q3.jsonl", "n": 802, "pos": 145, "auroc": 0.8583, "balanced_acc": 0.7811}
{"file": "experiments/silver_J2_legacy_form.jsonl", "n": 793, "pos": 125, "auroc": 0.8308, "balanced_acc": 0.7401}
{"file": "experiments/eval_J2_rewrap_partner_feed.jsonl", "n": 160, "pos": 80, "auroc": 0.9439, "balanced_acc": 0.8313}
== J2 team:
{"file": "experiments/silver_J2_web_q3.jsonl", "n": 802, "pos": 145, "auroc": 0.9849, "balanced_acc": 0.929}
{"file": "experiments/silver_J2_legacy_form.jsonl", "n": 793, "pos": 125, "auroc": 0.9004, "balanced_acc": 0.8191}
{"file": "experiments/eval_J2_rewrap_partner_feed.jsonl", "n": 160, "pos": 80, "auroc": 0.9544, "balanced_acc": 0.875}

== J3 shipped:
{"file": "experiments/eval_J3_rewrap_web_q3.jsonl", "n": 200, "pos": 100, "auroc": 0.846, "balanced_acc": 0.79}
{"file": "experiments/eval_J3_restruct_region_eu.jsonl", "n": 200, "pos": 100, "auroc": 0.8417, "balanced_acc": 0.795}
{"file": "experiments/eval_J3_rewrap_partner_feed.jsonl", "n": 200, "pos": 100, "auroc": 0.8405, "balanced_acc": 0.785}
== J3 baseline:
{"file": "experiments/eval_J3_rewrap_web_q3.jsonl", "n": 200, "pos": 100, "auroc": 0.8841, "balanced_acc": 0.835}
{"file": "experiments/eval_J3_restruct_region_eu.jsonl", "n": 200, "pos": 100, "auroc": 0.8775, "balanced_acc": 0.815}
{"file": "experiments/eval_J3_rewrap_partner_feed.jsonl", "n": 200, "pos": 100, "auroc": 0.8711, "balanced_acc": 0.8}
== J3 team:
{"file": "experiments/eval_J3_rewrap_web_q3.jsonl", "n": 200, "pos": 100, "auroc": 0.832, "balanced_acc": 0.76}
{"file": "experiments/eval_J3_restruct_region_eu.jsonl", "n": 200, "pos": 100, "auroc": 0.8227, "balanced_acc": 0.765}
{"file": "experiments/eval_J3_rewrap_partner_feed.jsonl", "n": 200, "pos": 100, "auroc": 0.8191, "balanced_acc": 0.78}
== exp devs:
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 0.9967, "balanced_acc": 0.9667}
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9611}
{"file": "/task/data/J1/dev.jsonl", "n": 160, "pos": 70, "auroc": 1.0, "balanced_acc": 0.9444}
{"file": "/task/data/J2/dev.jsonl", "n": 160, "pos": 80, "auroc": 0.9495, "balanced_acc": 0.8812}
{"file": "/task/data/J3/dev.jsonl", "n": 200, "pos": 100, "auroc": 0.8262, "balanced_acc": 0.78}

answer.json written
all referenced paths exist

3
122 summary.md

== submission tree:
   answer.json 10601
   detectors/J1.json 672
   detectors/J2.json 418
   detectors/J3.json 1209
   experiments/eval_J1_restruct_partner_feed.jsonl 32370
   experiments/eval_J2_rewrap_partner_feed.jsonl 31975
   experiments/eval_J3_restruct_region_eu.jsonl 39371
   experiments/eval_J3_rewrap_partner_feed.jsonl 39273
   experiments/eval_J3_rewrap_web_q3.jsonl 38216
   experiments/exp_J1_baseline.json 7972
   experiments/exp_J1_dense_aug.json 7873
   experiments/exp_J1_hybrid_aug.json 8570
   experiments/exp_J1_team_w1024k24.json 678
   experiments/exp_J2_baseline.json 7982
   experiments/exp_J2_team_w512k8t8.json 676
   experiments/exp_J3_baseline.json 7973
   experiments/exp_J3_team_w1024k8t8.json 689
   experiments/silver_J1_legacy_form.jsonl 182371
   experiments/silver_J1_web_q3.jsonl 186246
   experiments/silver_J2_legacy_form.jsonl 143750
   experiments/silver_J2_web_q3.jsonl 148338
   summary.md 10856
== validate jsonl evidence files:
  experiments/eval_J1_restruct_partner_feed.jsonl: n=160 labels=[0, 1] bad_rows=0 OK
  experiments/eval_J2_rewrap_partner_feed.jsonl: n=160 labels=[0, 1] bad_rows=0 OK
  experiments/eval_J3_restruct_region_eu.jsonl: n=200 labels=[0, 1] bad_rows=0 OK
  experiments/eval_J3_rewrap_partner_feed.jsonl: n=200 labels=[0, 1] bad_rows=0 OK
  experiments/eval_J3_rewrap_web_q3.jsonl: n=200 labels=[0, 1] bad_rows=0 OK
  experiments/silver_J1_legacy_form.jsonl: n=1000 labels=[0, 1] bad_rows=0 OK
  experiments/silver_J1_web_q3.jsonl: n=1000 labels=[0, 1] bad_rows=0 OK
  experiments/silver_J2_legacy_form.jsonl: n=793 labels=[0, 1] bad_rows=0 OK
  experiments/silver_J2_web_q3.jsonl: n=802 labels=[0, 1] bad_rows=0 OK
== validate specs load and have required keys:
   detectors/J1.json features 8 OK
   detectors/J2.json features 4 OK
   detectors/J3.json features 16 OK
   experiments/exp_J1_baseline.json features 128 OK
   experiments/exp_J1_dense_aug.json features 128 OK
   experiments/exp_J1_hybrid_aug.json features 136 OK
   experiments/exp_J1_team_w1024k24.json features 8 OK
   experiments/exp_J2_baseline.json features 128 OK
   experiments/exp_J2_team_w512k8t8.json features 8 OK
   experiments/exp_J3_baseline.json features 128 OK
   experiments/exp_J3_team_w1024k8t8.json features 8 OK
== answer.json jobs: ['J1', 'J2', 'J3'] experiments: 8
ALL GOOD