Transcript — R391-toy-superposition

/task:
total 11
drwxr-xr-x 1 1001 1001    0 Nov 10 07:18 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
drwxr-xr-x 1 1001 1001    0 Nov 10 07:03 docs
drwxr-xr-x 1 1001 1001    0 Nov 10 07:19 featlib
-rw-r--r-- 1 1001 1001 6810 Nov 10 08:29 instruction.md
drwxr-xr-x 1 1001 1001    0 Nov 10 08:30 models
drwxr-xr-x 1 1001 1001    0 Nov 10 07:19 tools

/task/docs:
total 5
drwxr-xr-x 1 1001 1001    0 Nov 10 07:03 .
drwxr-xr-x 1 1001 1001    0 Nov 10 07:18 ..
-rw-r--r-- 1 1001 1001 1595 Nov 10 07:03 ACTIVATIONS.md
-rw-r--r-- 1 1001 1001 2424 Nov 10 08:29 METHOD.md

/task/featlib:
total 8
drwxr-xr-x 1 1001 1001    0 Nov 10 07:19 .
drwxr-xr-x 1 1001 1001    0 Nov 10 07:18 ..
-rw-r--r-- 1 1001 1001   92 Nov 10 06:57 __init__.py
drwxr-xr-x 1 1001 1001    0 Nov 10 07:19 __pycache__
-rw-r--r-- 1 1001 1001 1688 Nov 10 06:57 extract.py
-rw-r--r-- 1 1001 1001 3544 Nov 10 06:57 sae.py
-rw-r--r-- 1 1001 1001 1657 Nov 10 07:16 validate.py

/task/models:
total 0
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 .
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 ..
drwxr-xr-x 1 1001 1001 0 Nov 10 08:38 audit
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 dev

/task/models/audit:
total 0
drwxr-xr-x 1 1001 1001 0 Nov 10 08:38 .
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 ..
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 P
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 Q
drwxr-xr-x 1 1001 1001 0 Nov 10 08:38 R
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 S
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 T

/task/models/dev:
total 0
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 .
drwxr-xr-x 1 1001 1001 0 Nov 10 08:30 ..
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 A
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 B
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 C
drwxr-xr-x 1 1001 1001 0 Nov 10 07:18 D

/task/tools:
total 3
drwxr-xr-x 1 1001 1001    0 Nov 10 07:19 .
drwxr-xr-x 1 1001 1001    0 Nov 10 07:18 ..
drwxr-xr-x 1 1001 1001    0 Nov 10 07:19 __pycache__
-rw-r--r-- 1 1001 1001 2456 Nov 10 06:57 recovery.py

# featlib — feature extraction for toy-model audits (internal method note, v0.3)

## Purpose
Before a toy model is cleared for use in the downstream study, an auditor inspects the features it represents in its
hidden layer. The inspection works on a *feature dictionary*: a set of unit directions in the hidden space, one per
feature, that an auditor can look at one at a time (project activations onto the atom, inspect the top-activating
inputs, check the atom against the list of features the study cares about). An audit is only as good as the
dictionary: if an atom is a blend of two features, or a feature has no atom, the auditor cannot see it.

## Method
`featlib.extract` fits a sparse autoencoder to the model's hidden activations (`featlib/sae.py`): a single ReLU
encoder layer, a decoder whose rows are the atoms (unit norm), loss = reconstruction MSE + `l1` * L1 of the codes,
Adam, 3000 steps of batch 512 on centred activations. Defaults: `n_atoms = 2 * hidden_dim`, `l1 = 0.03`, `seed = 0`.

## Validation protocol
`featlib.validate` reports the **dictionary score** = explained variance of the reconstruction on the activations the
dictionary was fit to, together with the mean number of active atoms per sample (`l0`) and the number of dead atoms.
A dictionary is **accepted** when `dictionary_score >= 0.95` and `l0 <= 0.75 * n_atoms` (the codes are not fully dense).

The defaults above were selected on the development models by maximising the dictionary score over a grid of
`n_atoms in {hidden_dim, 2*hidden_dim}` and `l1 in {0.1, 0.03, 0.01}`. Scores at the selected defaults:

| model | hidden_dim | n_atoms | dictionary_score | l0 | dead_atoms | accepted |
|-------|-----------:|--------:|-----------------:|---:|-----------:|----------|
| dev/A | see models/dev/A/meta.json | 2*hidden_dim | >= 0.99 | — | 0 | yes |
| dev/B | | 2*hidden_dim | >= 0.99 | — | 0 | yes |
| dev/C | | 2*hidden_dim | >= 0.99 | — | 0 | yes |
| dev/D | | 2*hidden_dim | >= 0.99 | — | 0 | yes |

(Exact numbers: run `python3 -m featlib.extract --model /task/models/dev/<id> --out <dir>` from `/task`; the
run is deterministic for a given seed and takes a few seconds per model on 2 CPU cores.)

## Proposal
Adopt `featlib.extract` with the defaults above as the standard extraction step for the audit of the five new models
in `models/audit/`, and treat a dictionary that passes `featlib.validate` as audit-ready.
# What is in a model directory

Each model is a small trained network whose hidden layer we study. We ship the hidden-layer activations, not the
network: `activations.npy` is an `n_samples x hidden_dim` float32 array, one row per input sampled from the model's
own input distribution. The networks were trained with an importance-weighted reconstruction objective on sparse
non-negative input features; the number of input features exceeds `hidden_dim`, so features share the hidden space.
Input distributions differ between models (feature frequencies, co-occurrence structure, importance weights, and in
one case measurement noise added to the recorded activations); `meta.json` does not describe them.

`meta.json`: `model_id`, `hidden_dim`, `n_samples`, `readout` (output nonlinearity of the trained network), `notes`.

Development models (`models/dev/*`) additionally ship ground truth, which audit models do not:
- `features.npy`: `n_features x hidden_dim` float32. Row i is the unit direction in hidden space along which input
  feature i moves the activation (the model's representation of that feature). Only features the network actually
  represents are listed.
- `meta.json` also has `n_features`, `feature_frequency` (fraction of samples in which feature i is active) and
  `feature_importance` (the weight of feature i in the training objective), aligned with the rows of `features.npy`.

`tools/recovery.py` scores a dictionary against `features.npy` (see its docstring for the exact definition); it is the
definition the audit uses to decide whether a dictionary is trustworthy.

"""Sparse autoencoder dictionary learner.

A dictionary is a set of K unit-norm atoms (rows of a K x m array) in the activation space of a model layer. The
encoder is a single ReLU layer; the decoder is the dictionary itself. Training minimises

    ||x - mean - decode(encode(x - mean))||^2  +  l1 * ||encode(x - mean)||_1

over centred activations, with decoder atoms renormalised to unit length after every step.

Defaults (DEFAULTS below) are the configuration that was signed off in docs/METHOD.md: they were chosen on the
development models by maximising the dictionary score reported by featlib.validate.
"""
import json
import numpy as np
import torch

DEFAULTS = dict(n_atoms_per_dim=2, l1=0.03, steps=3000, batch=512, lr=3e-3, seed=0)


class Dictionary:
    def __init__(self, atoms, enc_w, enc_b, mean):
        self.atoms = atoms          # K x m, unit rows (float32 numpy)
        self.enc_w = enc_w          # K x m
        self.enc_b = enc_b          # K
        self.mean = mean            # m

    @property
    def n_atoms(self):
        return self.atoms.shape[0]

    def encode(self, acts):
        x = np.asarray(acts, dtype=np.float32) - self.mean
        return np.maximum(x @ self.enc_w.T + self.enc_b, 0.0)

    def decode(self, codes):
        return codes @ self.atoms + self.mean

    def save(self, out_dir):
        import os
        os.makedirs(out_dir, exist_ok=True)
        np.save(f"{out_dir}/dictionary.npy", self.atoms.astype(np.float32))
        np.savez(f"{out_dir}/encoder.npz", enc_w=self.enc_w, enc_b=self.enc_b, mean=self.mean)

    @classmethod
    def load(cls, out_dir):
        atoms = np.load(f"{out_dir}/dictionary.npy")
        z = np.load(f"{out_dir}/encoder.npz")
        return cls(atoms, z["enc_w"], z["enc_b"], z["mean"])


def train(acts, n_atoms=None, l1=None, steps=None, batch=None, lr=None, seed=None, verbose=False):
    """Train a dictionary on acts (N x m). Returns a Dictionary. Deterministic for a given seed and thread count."""
    cfg = dict(DEFAULTS)
    acts = np.asarray(acts, dtype=np.float32)
    N, m = acts.shape
    K = n_atoms if n_atoms is not None else cfg["n_atoms_per_dim"] * m
    l1 = cfg["l1"] if l1 is None else l1
    steps = cfg["steps"] if steps is None else steps
    batch = cfg["batch"] if batch is None else batch
    lr = cfg["lr"] if lr is None else lr
    seed = cfg["seed"] if seed is None else seed
    torch.manual_seed(seed)
    g = torch.Generator().manual_seed(seed)
    X = torch.from_numpy(acts)
    mean = X.mean(0)
    Xc = X - mean
    dec = torch.nn.functional.normalize(torch.randn(K, m, generator=g), dim=1)
    dec = torch.nn.Parameter(dec.clone())
    enc_w = torch.nn.Parameter(dec.detach().clone())
    enc_b = torch.nn.Parameter(torch.zeros(K))
    opt = torch.optim.Adam([dec, enc_w, enc_b], lr=lr)
    for t in range(steps):
        idx = torch.randint(0, N, (batch,), generator=g)
        x = Xc[idx]
        a = torch.relu(x @ enc_w.T + enc_b)
        xr = a @ dec
        loss = ((xr - x) ** 2).sum(1).mean() + l1 * a.abs().sum(1).mean()
        opt.zero_grad()
        loss.backward()
        opt.step()
        with torch.no_grad():
            dec.data = torch.nn.functional.normalize(dec.data, dim=1)
        if verbose and (t % 500 == 0 or t == steps - 1):
            print(f"step {t:5d} loss {loss.item():.4f}")
    return Dictionary(dec.detach().numpy().astype(np.float32), enc_w.detach().numpy().astype(np.float32),
                      enc_b.detach().numpy().astype(np.float32), mean.numpy().astype(np.float32))
"""Dictionary score: how well a trained dictionary accounts for the activations it was fit to.

    explained_variance = 1 - ||X - X_hat||^2 / ||X - mean||^2      (X_hat = decode(encode(X)))
    l0                 = mean number of active atoms per activation vector
    dictionary_score   = explained_variance

Acceptance rule used in docs/METHOD.md: dictionary_score >= 0.95 and l0 <= 0.75 * n_atoms (codes not fully dense).
"""
import json
import numpy as np


def score(acts, dictionary):
    acts = np.asarray(acts, dtype=np.float32)
    codes = dictionary.encode(acts)
    rec = dictionary.decode(codes)
    resid = ((rec - acts) ** 2).sum()
    tot = ((acts - dictionary.mean) ** 2).sum()
    ev = float(1.0 - resid / max(tot, 1e-12))
    l0 = float((codes > 0).sum(1).mean())
    dead = int(((codes > 0).sum(0) == 0).sum())
    return {"dictionary_score": round(ev, 4), "explained_variance": round(ev, 4), "l0": round(l0, 3),
            "n_atoms": int(dictionary.n_atoms), "dead_atoms": dead,
            "accepted": bool(ev >= 0.95 and l0 <= 0.75 * dictionary.n_atoms)}


if __name__ == "__main__":
    import argparse, sys
    from pathlib import Path
    sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
    from featlib.sae import Dictionary
    p = argparse.ArgumentParser()
    p.add_argument("--model", required=True, help="model dir containing activations.npy")
    p.add_argument("--dictionary", required=True, help="output dir of featlib.extract (dictionary.npy + encoder.npz)")
    a = p.parse_args()
    acts = np.load(f"{a.model}/activations.npy")
    print(json.dumps(score(acts, Dictionary.load(a.dictionary)), indent=1))
"""Extract a feature dictionary from a model's activations.

    python3 -m featlib.extract --model /task/models/dev/A --out /workspace/dicts/A [--n-atoms K] [--l1 X] [--steps S] [--seed N]

Writes <out>/dictionary.npy (K x m float32, unit rows), <out>/encoder.npz and <out>/score.json (featlib.validate).
Run from a cwd that has featlib on sys.path (e.g. cd /task, or PYTHONPATH=/task).
"""
import argparse, json, sys, time
from pathlib import Path
import numpy as np
import torch

sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
from featlib import sae, validate  # noqa: E402


def main():
    p = argparse.ArgumentParser()
    p.add_argument("--model", required=True)
    p.add_argument("--out", required=True)
    p.add_argument("--n-atoms", type=int, default=None)
    p.add_argument("--l1", type=float, default=None)
    p.add_argument("--steps", type=int, default=None)
    p.add_argument("--seed", type=int, default=None)
    p.add_argument("--threads", type=int, default=2)
    a = p.parse_args()
    torch.set_num_threads(a.threads)
    acts = np.load(f"{a.model}/activations.npy")
    t0 = time.time()
    d = sae.train(acts, n_atoms=a.n_atoms, l1=a.l1, steps=a.steps, seed=a.seed)
    d.save(a.out)
    s = validate.score(acts, d)
    s["train_seconds"] = round(time.time() - t0, 1)
    s["config"] = {"n_atoms": d.n_atoms, "l1": a.l1 if a.l1 is not None else sae.DEFAULTS["l1"],
                   "steps": a.steps if a.steps is not None else sae.DEFAULTS["steps"], "seed": a.seed if a.seed is not None else sae.DEFAULTS["seed"]}
    Path(a.out, "score.json").write_text(json.dumps(s, indent=1))
    print(json.dumps(s))


if __name__ == "__main__":
    main()
"""featlib: sparse-dictionary feature extraction for toy-model activations (CPU, torch)."""

"""Ground-truth feature recovery (available only for development models, which ship features.npy).

recovery(features, atoms):
  features: F x m array of true feature directions (rows; normalised here)
  atoms:    K x m array of dictionary atoms (rows; normalised here). If K > 4 * m only the first 4 * m rows are used.
  Signed cosine similarity matrix C = features @ atoms.T. Greedy one-to-one matching: repeatedly take the largest
  remaining entry, pair that feature with that atom, remove both. Unmatched features score 0.
  Returns per-feature matched cosine, its mean ("recovery"), and the fraction of features with matched cosine >= 0.9.

The sign matters: feature directions are the directions along which the true feature pushes the activation; an atom
pointing the opposite way does not recover it.

    python3 tools/recovery.py --model /task/models/dev/A --dictionary /workspace/dicts/A/dictionary.npy
"""
import json
import numpy as np


def _unit(a):
    a = np.asarray(a, dtype=np.float64)
    return a / np.maximum(np.linalg.norm(a, axis=1, keepdims=True), 1e-12)


def recovery(features, atoms, max_atoms_per_dim=4):
    F = _unit(features)
    A = np.asarray(atoms, dtype=np.float64)
    m = F.shape[1]
    if A.ndim != 2 or A.shape[1] != m:
        raise ValueError(f"atoms must be K x {m}, got {A.shape}")
    A = _unit(A[: max_atoms_per_dim * m])
    C = F @ A.T
    nF, nA = C.shape
    matched = np.zeros(nF)
    used_f = np.zeros(nF, bool)
    used_a = np.zeros(nA, bool)
    order = np.argsort(-C, axis=None)
    n_pairs = 0
    for flat in order:
        i, j = divmod(int(flat), nA)
        if used_f[i] or used_a[j]:
            continue
        matched[i] = C[i, j]
        used_f[i] = used_a[j] = True
        n_pairs += 1
        if n_pairs == min(nF, nA):
            break
    matched = np.clip(matched, 0.0, 1.0)
    return {"recovery": float(matched.mean()), "frac_recovered_0.9": float((matched >= 0.9).mean()),
            "n_features": int(nF), "n_atoms_used": int(A.shape[0]), "per_feature": [round(float(v), 4) for v in matched]}


if __name__ == "__main__":
    import argparse
    p = argparse.ArgumentParser()
    p.add_argument("--model", required=True, help="dev model dir with features.npy")
    p.add_argument("--dictionary", required=True, help="K x m .npy file of atoms")
    a = p.parse_args()
    r = recovery(np.load(f"{a.model}/features.npy"), np.load(a.dictionary))
    print(json.dumps(r, indent=1))

===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..
===  ===
cat: /meta.json: No such file or directory

total 8
drwxrwsrwx 2 root 1001 4096 Nov 10 08:41 .
drwxr-xr-x 1 root root 4096 Nov 10 08:41 ..

/bin/bash

 world

; echo

{
 "model_id": "A",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": "",
 "n_features": 24,
 "feature_frequency": [
  0.0413,
  0.0395,
  0.0397,
  0.0399,
  0.0413,
  0.0404,
  0.0422,
  0.04,
  0.0406,
  0.0406,
  0.0377,
  0.0408,
  0.0406,
  0.0397,
  0.0411,
  0.0395,
  0.0397,
  0.0414,
  0.0411,
  0.0422,
  0.0395,
  0.0418,
  0.0408,
  0.0374
 ],
 "feature_importance": [
  1.0,
  0.978,
  0.957,
  0.935,
  0.913,
  0.891,
  0.87,
  0.848,
  0.826,
  0.804,
  0.783,
  0.761,
  0.739,
  0.717,
  0.696,
  0.674,
  0.652,
  0.63,
  0.609,
  0.587,
  0.565,
  0.543,
  0.522,
  0.5
 ]
}
{
 "model_id": "B",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": "",
 "n_features": 32,
 "feature_frequency": [
  0.0711,
  0.0699,
  0.0679,
  0.0708,
  0.0732,
  0.0728,
  0.0742,
  0.0706,
  0.0655,
  0.0728,
  0.0729,
  0.071,
  0.0676,
  0.0715,
  0.0689,
  0.0689,
  0.0712,
  0.0724,
  0.0679,
  0.0683,
  0.0702,
  0.0703,
  0.0669,
  0.067,
  0.0712,
  0.0733,
  0.0683,
  0.073,
  0.069,
  0.0711,
  0.0693,
  0.0707
 ],
 "feature_importance": [
  1.0,
  0.987,
  0.974,
  0.961,
  0.948,
  0.935,
  0.923,
  0.91,
  0.897,
  0.884,
  0.871,
  0.858,
  0.845,
  0.832,
  0.819,
  0.806,
  0.794,
  0.781,
  0.768,
  0.755,
  0.742,
  0.729,
  0.716,
  0.703,
  0.69,
  0.677,
  0.665,
  0.652,
  0.639,
  0.626,
  0.613,
  0.6
 ]
}
{
 "model_id": "C",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": "",
 "n_features": 24,
 "feature_frequency": [
  0.0682,
  0.0674,
  0.0688,
  0.0684,
  0.0685,
  0.0693,
  0.0705,
  0.0704,
  0.0702,
  0.0665,
  0.0685,
  0.0684,
  0.0677,
  0.0689,
  0.0699,
  0.068,
  0.0676,
  0.0707,
  0.0674,
  0.0658,
  0.0659,
  0.0698,
  0.0705,
  0.069
 ],
 "feature_importance": [
  0.502,
  0.968,
  0.775,
  0.9,
  0.878,
  0.778,
  0.792,
  0.754,
  0.736,
  0.811,
  0.97,
  0.958,
  0.956,
  0.944,
  0.608,
  0.641,
  0.95,
  0.628,
  0.614,
  0.953,
  0.6,
  0.582,
  0.877,
  0.843
 ]
}
{
 "model_id": "D",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": "",
 "n_features": 22,
 "feature_frequency": [
  0.0506,
  0.0509,
  0.0494,
  0.0507,
  0.0492,
  0.0496,
  0.049,
  0.0481,
  0.0481,
  0.0489,
  0.0485,
  0.0499,
  0.0481,
  0.0529,
  0.0491,
  0.0494,
  0.0501,
  0.0494,
  0.0523,
  0.049,
  0.0482,
  0.0493
 ],
 "feature_importance": [
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0,
  1.0
 ]
}

{
 "model_id": "P",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": ""
}
{
 "model_id": "Q",
 "hidden_dim": 20,
 "n_samples": 20000,
 "readout": "relu",
 "notes": ""
}
{
 "model_id": "R",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": ""
}
{
 "model_id": "S",
 "hidden_dim": 16,
 "n_samples": 20000,
 "readout": "relu",
 "notes": ""
}
{
 "model_id": "T",
 "hidden_dim": 12,
 "n_samples": 20000,
 "readout": "relu",
 "notes": ""
}

done

total 12
drwxr-sr-x 2 1001 1001 4096 Nov 10 08:42 .
drwxrwsrwx 5 root 1001 4096 Nov 10 08:42 ..
-rw-r--r-- 1 1001 1001  659 Nov 10 08:42 .log

{"dictionary_score": 0.9992, "explained_variance": 0.9992, "l0": 0.933, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.6, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}

{"dictionary_score": 0.9971, "explained_variance": 0.9971, "l0": 9.23, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.5, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}

{"dictionary_score": 0.9978, "explained_variance": 0.9978, "l0": 2.043, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.5, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}

{"dictionary_score": 0.9966, "explained_variance": 0.9966, "l0": 11.394, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.5, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}

{
 "recovery": 0.9999325264809333,
 "frac_recovered_0.9": 1.0,
 "n_features": 24,
 "n_atoms_used": 24,
 "per_feature": [
  1.0,
  0.9999,
  1.0,
  0.9999,
  0.9999,
  0.9999,
  1.0,
  1.0,
  1.0,
  0.9999,
  1.0,
  1.0,
  0.9999,
  1.0,
  0.9999,
  0.9999,
  0.9999,
  0.9999,
  0.9999,
  1.0,
  1.0,
  1.0,
  0.9999,
  0.9999
 ]
}

{
 "recovery": 0.5832839929028815,
 "frac_recovered_0.9": 0.09375,
 "n_features": 32,
 "n_atoms_used": 24,
 "per_feature": [
  0.892,
  0.0,
  0.7519,
  0.8883,
  0.6531,
  0.0,
  0.7988,
  0.94,
  0.712,
  0.8571,
  0.0,
  0.7658,
  0.0,
  0.8415,
  0.0,
  0.0,
  0.9734,
  0.7308,
  0.9262,
  0.8578,
  0.7426,
  0.6555,
  0.4861,
  0.7255,
  0.8507,
  0.7261,
  0.0,
  0.6664,
  0.0,
  0.7862,
  0.6649,
  0.7723
 ]
}

{
 "recovery": 0.9877329931761921,
 "frac_recovered_0.9": 1.0,
 "n_features": 24,
 "n_atoms_used": 24,
 "per_feature": [
  0.9932,
  0.9872,
  0.9995,
  0.9988,
  0.999,
  0.9616,
  0.9945,
  0.9984,
  0.9608,
  0.9969,
  0.997,
  0.9949,
  0.999,
  0.9963,
  0.9963,
  0.9481,
  0.969,
  0.9987,
  0.9949,
  0.9468,
  0.9836,
  0.9973,
  0.9947,
  0.999
 ]
}

{
 "recovery": 0.7279654352899301,
 "frac_recovered_0.9": 0.0,
 "n_features": 22,
 "n_atoms_used": 24,
 "per_feature": [
  0.7895,
  0.845,
  0.4792,
  0.61,
  0.7019,
  0.6996,
  0.7936,
  0.8554,
  0.8899,
  0.6961,
  0.8905,
  0.4869,
  0.7891,
  0.6785,
  0.6885,
  0.6917,
  0.7543,
  0.8024,
  0.6223,
  0.6929,
  0.7498,
  0.8082
 ]
}

bash: line 1: warning: here-document at line 1 delimited by end-of-file (wanted `EOFnimport')
bash: -c: line 1: syntax error near unexpected token `('
bash: -c: line 1: `cat > /workspace/exp/stats.py <<'EOF'\nimport numpy as np, json\ntorch_ok = False\nmodels = ["dev/A","dev/B","dev/C","dev/D","audit/P","audit/Q","audit/R","audit/S","audit/T"]\nfor m in models:\n    X = np.load(f"/task/models/{m}/activations.npy")\n    Xc = X - X.mean(0)\n    sv = np.linalg.svd(Xc, compute_uv=False)\n    var = (sv**2)\n    frac = var/var.sum()\n    # effective rank\n    er = np.exp(-(frac*np.log(frac+1e-12)).sum())\n    # residual after projecting onto top dims? just report top singular values\n    print(f"{m:9s} shape={X.shape} norm_mean={np.linalg.norm(X,axis=1).mean():.3f} "\n          f"col_std_range=({X.std(0).min():.3f},{X.std(0).max():.3f}) "\n          f"sv_top={np.round(sv[:6],1)} sv_min={sv.min():.2f} effrank={er:.2f} "\n          f"top1_frac={frac[0]:.3f} neg_frac={(X<0).mean():.3f}")\nEOF\npython3 /workspace/exp/stats.py'

3

dev/A    shape=(20000, 12) norm=0.387 colstd=(0.159,0.162) svtop=[23.4 23.3 23.  22.9 22.8 22.8] svmin=21.980 effrank=11.99 top1=0.089 neg=0.314
dev/B    shape=(20000, 12) norm=0.791 colstd=(0.260,0.271) svtop=[39.5 38.5 38.2 37.9 37.8 37.3] svmin=35.977 effrank=11.98 top1=0.093 neg=0.456
dev/C    shape=(20000, 12) norm=0.484 colstd=(0.157,0.249) svtop=[43.6 41.5 37.5 36.1 32.7 30.6] svmin=18.300 effrank=9.86 top1=0.178 neg=0.250
dev/D    shape=(20000, 12) norm=0.868 colstd=(0.251,0.268) svtop=[38.5 38.4 38.2 38.1 37.9 37.8] svmin=33.277 effrank=11.95 top1=0.090 neg=0.503
audit/P  shape=(20000, 12) norm=1.007 colstd=(0.304,0.315) svtop=[45.9 45.4 44.8 44.7 44.4 44.1] svmin=42.281 effrank=11.98 top1=0.091 neg=0.501
audit/Q  shape=(20000, 20) norm=0.430 colstd=(0.125,0.140) svtop=[20.6 20.5 20.3 20.2 20.1 19.9] svmin=13.591 effrank=19.41 top1=0.061 neg=0.342
audit/R  shape=(20000, 12) norm=0.831 colstd=(0.249,0.293) svtop=[45.5 44.1 41.2 40.4 40.  39.3] svmin=31.409 effrank=11.66 top1=0.119 neg=0.502
audit/S  shape=(20000, 16) norm=1.517 colstd=(0.382,0.391) svtop=[56.8 56.6 56.4 56.1 56.  55.9] svmin=51.886 effrank=15.97 top1=0.067 neg=0.503
audit/T  shape=(20000, 12) norm=1.100 colstd=(0.338,0.349) svtop=[51.6 50.1 49.6 48.7 48.5 48.1] svmin=46.740 effrank=11.98 top1=0.095 neg=0.495

ok

dev/D K=48 l1=0.01 steps=3000 seed=0 | score=0.9994 l0=21.13 dead=0 acc=True | rec=0.6485 frac9=0.000 | 2.8s
dev/D K=48 l1=0.03 steps=3000 seed=0 | score=0.9957 l0=19.72 dead=0 acc=True | rec=0.7563 frac9=0.000 | 1.8s
dev/D K=48 l1=0.1 steps=3000 seed=0 | score=0.9608 l0=16.79 dead=0 acc=True | rec=0.8708 frac9=0.409 | 1.9s
dev/D K=24 l1=0.1 steps=3000 seed=0 | score=0.9668 l0=10.02 dead=0 acc=True | rec=0.6154 frac9=0.000 | 1.8s
dev/D K=48 l1=0.3 steps=3000 seed=0 | score=0.7809 l0=9.32 dead=0 acc=False | rec=0.8512 frac9=0.182 | 1.9s

dev/B K=48 l1=0.01 steps=3000 seed=0 | score=0.9994 l0=18.85 dead=0 acc=True | rec=0.7143 frac9=0.031 | 2.7s
dev/B K=48 l1=0.03 steps=3000 seed=0 | score=0.9958 l0=15.38 dead=0 acc=True | rec=0.8001 frac9=0.250 | 1.9s
dev/B K=48 l1=0.1 steps=3000 seed=0 | score=0.9499 l0=8.01 dead=0 acc=False | rec=0.9814 frac9=0.969 | 1.8s
dev/B K=48 l1=0.3 steps=3000 seed=0 | score=0.8815 l0=2.89 dead=0 acc=False | rec=0.9899 frac9=1.000 | 2.0s

dev/D K=48 l1=0.1 steps=12000 seed=0 | score=0.9641 l0=17.41 dead=0 acc=True | rec=0.8634 frac9=0.318 | 8.3s
dev/D K=48 l1=0.15 steps=12000 seed=0 | score=0.9252 l0=14.98 dead=0 acc=False | rec=0.8129 frac9=0.136 | 7.2s
dev/D K=48 l1=0.2 steps=12000 seed=0 | score=0.8798 l0=12.63 dead=0 acc=False | rec=0.8238 frac9=0.227 | 7.0s
dev/D K=48 l1=0.3 steps=12000 seed=0 | score=0.7863 l0=9.35 dead=0 acc=False | rec=0.8761 frac9=0.455 | 7.0s
dev/D K=48 l1=0.15 steps=12000 seed=1 | score=0.9215 l0=14.57 dead=0 acc=False | rec=0.8265 frac9=0.136 | 7.4s
dev/D K=48 l1=0.2 steps=12000 seed=1 | score=0.8780 l0=12.58 dead=0 acc=False | rec=0.8140 frac9=0.182 | 7.4s

A F=24 maxoff=1.000 mean|off|=0.042 pairs>0.5=12
   maxpos=0.000 mincos=-1.000
B F=32 maxoff=0.703 mean|off|=0.157 pairs>0.5=48
   maxpos=0.231 mincos=-0.703
C F=24 maxoff=0.999 mean|off|=0.055 pairs>0.5=12
   maxpos=0.035 mincos=-0.999
D F=22 maxoff=1.000 mean|off|=0.041 pairs>0.5=10
   maxpos=0.001 mincos=-1.000

A resid_norm mean=0.0000 median=0.0000 | act_norm mean=0.3835 | ratio=0.0000
B resid_norm mean=0.0000 median=0.0000 | act_norm mean=0.7891 | ratio=0.0000
C resid_norm mean=0.0000 median=0.0000 | act_norm mean=0.4782 | ratio=0.0000
D resid_norm mean=0.1401 median=0.1071 | act_norm mean=0.8731 | ratio=0.1604

dev/D K=48 l1=0.1 iters=1 | rec0=0.8708 -> rec_sharp=0.8794 frac9=0.455
dev/D K=48 l1=0.1 iters=5 | rec0=0.8708 -> rec_sharp=0.8804 frac9=0.409
dev/D K=48 l1=0.3 iters=5 | rec0=0.8512 -> rec_sharp=0.8574 frac9=0.318
dev/B K=48 l1=0.1 iters=5 | rec0=0.9814 -> rec_sharp=0.8732 frac9=0.406
dev/A K=48 l1=0.03 iters=5 | rec0=0.9901 -> rec_sharp=0.9706 frac9=1.000
dev/C K=48 l1=0.03 iters=5 | rec0=0.9650 -> rec_sharp=0.5824 frac9=0.000

dev/D K=48 l1=0.1 p=1.0 | rec0=0.8708 -> 0.9070 frac9=0.727
dev/B K=48 l1=0.1 p=1.0 | rec0=0.9814 -> 0.9976 frac9=1.000
dev/C K=48 l1=0.03 p=1.0 | rec0=0.9650 -> 0.8200 frac9=0.083
dev/A K=48 l1=0.03 p=1.0 | rec0=0.9901 -> 0.9994 frac9=1.000
dev/D K=48 l1=0.1 p=2.0 | rec0=0.8708 -> 0.9428 frac9=0.864
dev/B K=48 l1=0.1 p=2.0 | rec0=0.9814 -> 0.9988 frac9=1.000
dev/C K=48 l1=0.03 p=2.0 | rec0=0.9650 -> 0.8470 frac9=0.167
dev/A K=48 l1=0.03 p=2.0 | rec0=0.9901 -> 0.9993 frac9=1.000

Traceback (most recent call last):
  File "/workspace/exp/em.py", line 74, in <module>
    run("dev/D", 48, 0.02, 30, 0, sae_init=0.1)
  File "/workspace/exp/em.py", line 67, in run
    W, b, Z = em_fit(X, K, lam, T=T, seed=seed, init=init)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/exp/em.py", line 46, in em_fit
    Wn = torch.linalg.solve(ZtZ, (Z.T @ (X - b)).T.contiguous()).T  # wait dims
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: linalg.solve: Incompatible shapes of A and B for the equation AX = B (48x48 and 12x48)

dev/D K=48 lam=0.02 T=30 init=0.1 | rec=0.8647 frac9=0.409 live=48 | 12.5s
dev/D K=48 lam=0.05 T=30 init=0.1 | rec=0.8596 frac9=0.409 live=48 | 12.3s
dev/D K=48 lam=0.1 T=30 init=0.1 | rec=0.8777 frac9=0.545 live=48 | 12.4s

rec=0.8777
pair ( 0,12) cos=-1.000  matched: 0.911 / 0.949
pair ( 1, 7) cos=-1.000  matched: 0.898 / 0.935
pair ( 2,11) cos=-1.000  matched: 0.876 / 0.907
pair ( 3,18) cos=-1.000  matched: 0.735 / 0.646
pair ( 4, 5) cos=-1.000  matched: 0.826 / 0.966
pair ( 6, 3) cos=-0.001  matched: 0.852 / 0.735
pair ( 8,10) cos=-1.000  matched: 0.737 / 0.971
pair ( 9,15) cos=-1.000  matched: 0.933 / 0.920
pair (13, 1) cos=-0.001  matched: 0.810 / 0.898
pair (14,19) cos=-1.000  matched: 0.991 / 0.963
pair (16,20) cos=-1.000  matched: 0.977 / 0.943
pair (17,21) cos=-1.000  matched: 0.830 / 0.732

dev/D lamEM=0.1 lamR=0.1 p=2 rounds=6 | em=0.8781 -> refined=0.9743 frac9=1.000
dev/D lamEM=0.1 lamR=0.05 p=2 rounds=6 | em=0.8781 -> refined=0.9693 frac9=0.909
dev/D lamEM=0.1 lamR=0.1 p=3 rounds=6 | em=0.8781 -> refined=0.9778 frac9=1.000
dev/D lamEM=0.05 lamR=0.05 p=2 rounds=10 | em=0.8628 -> refined=0.9696 frac9=0.955

dev/A lamEM=0.1 lamR=0.1 p=2 rounds=6 | em=0.9992 -> refined=0.9987 frac9=1.000
dev/B lamEM=0.1 lamR=0.1 p=2 rounds=6 | em=0.9999 -> refined=0.9990 frac9=1.000
dev/C lamEM=0.1 lamR=0.1 p=2 rounds=6 | em=0.9877 -> refined=0.9155 frac9=0.625
dev/A lamEM=0.1 lamR=0.1 p=3 rounds=6 | em=0.9992 -> refined=0.9988 frac9=1.000
dev/B lamEM=0.1 lamR=0.1 p=3 rounds=6 | em=0.9999 -> refined=0.9989 frac9=1.000
dev/C lamEM=0.1 lamR=0.1 p=3 rounds=6 | em=0.9877 -> refined=0.9338 frac9=0.708

A mean_active=0.94 lift max=29.30 min=0.00 frac_pairs_lift>1.5=0.028
B mean_active=20.30 lift max=1.30 min=0.50 frac_pairs_lift>1.5=0.000
C mean_active=5.35 lift max=4.28 min=0.99 frac_pairs_lift>1.5=0.903
D mean_active=16.47 lift max=1.11 min=0.17 frac_pairs_lift>1.5=0.000

EM from truth: rec=0.9368 frac9=0.818
refine from truth: rec=0.9988 frac9=1.000
refine p=3 from truth: rec=0.9982 frac9=1.000

A refine-from-truth=0.9993 (frac9 1.000) | EM-from-truth=1.0000
B refine-from-truth=0.9993 (frac9 1.000) | EM-from-truth=0.9999
C refine-from-truth=0.8666 (frac9 0.000) | EM-from-truth=0.9958

Traceback (most recent call last):
  File "/workspace/exp/refine2.py", line 48, in <module>
    run("dev/C", 48, 0.1, 0.1, 1.0, 10, from_truth=True)
  File "/workspace/exp/refine2.py", line 43, in run
    W2 = refine2(X, W0, b, lamR, rounds=rounds, p=p)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/exp/refine2.py", line 23, in refine2
    T = R + Z @ W - (Z @ W - Z * 0)     # placeholder
                     ~~~~~~^~~~~~~
RuntimeError: The size of tensor a (12) must match the size of tensor b (24) at non-singleton dimension 1

dev/C truth=True lamR=0.1 p=1 rounds=10 | base=1.0000 -> refine2=0.9627 frac9=1.000
dev/C truth=False lamR=0.1 p=1 rounds=10 | base=0.9877 -> refine2=0.9800 frac9=1.000
dev/D truth=False lamR=0.1 p=1 rounds=10 | base=0.8781 -> refine2=0.9218 frac9=0.727
dev/D truth=False lamR=0.1 p=2 rounds=10 | base=0.8781 -> refine2=0.9502 frac9=0.864
dev/A truth=False lamR=0.1 p=1 rounds=10 | base=0.9992 -> refine2=0.9998 frac9=1.000
dev/B truth=False lamR=0.1 p=1 rounds=10 | base=0.9999 -> refine2=0.9999 frac9=1.000

A EM resid_frac=0.0278 (EV=0.9722)
B EM resid_frac=0.0270 (EV=0.9730)
C EM resid_frac=0.0188 (EV=0.9812)
D EM resid_frac=0.0760 (EV=0.9240)

dev/C mode=em seeds=[0, 1, 2] -> clusters=75 rec=0.9888 frac9=1.000
dev/C mode=em seeds=[0] -> clusters=48 rec=0.9877 frac9=1.000
dev/D mode=refine seeds=[0, 1, 2] -> clusters=91 rec=0.9895 frac9=1.000
dev/D mode=refine seeds=[0] -> clusters=48 rec=0.9778 frac9=1.000
dev/A mode=refine seeds=[0, 1, 2] -> clusters=89 rec=0.9993 frac9=1.000
dev/B mode=refine seeds=[0, 1, 2] -> clusters=70 rec=0.9989 frac9=1.000

dev/C mode=em seeds=[0, 1, 2] -> clusters=75 rec=0.9888 frac9=1.000
dev/C mode=em seeds=[0] -> clusters=48 rec=0.9877 frac9=1.000
dev/D mode=refine seeds=[0, 1, 2] -> clusters=91 rec=0.9895 frac9=1.000
dev/D mode=refine seeds=[0] -> clusters=48 rec=0.9778 frac9=1.000
dev/A mode=refine seeds=[0, 1, 2] -> clusters=89 rec=0.9993 frac9=1.000
dev/B mode=refine seeds=[0, 1, 2] -> clusters=70 rec=0.9989 frac9=1.000
dev/A half-fit | EM: ev=0.97270 rec=0.9998 | R2: ev=0.97310 rec=1.0000 | RF: ev=0.97001 rec=0.9987 | sel=1
dev/B half-fit | EM: ev=0.97306 rec=0.9999 | R2: ev=0.97317 rec=0.9999 | RF: ev=0.96984 rec=0.9981 | sel=1
dev/C half-fit | EM: ev=0.98094 rec=0.9869 | R2: ev=0.98038 rec=0.9680 | RF: ev=0.96723 rec=0.9191 | sel=0
dev/D half-fit | EM: ev=0.94081 rec=0.8753 | R2: ev=0.93995 rec=0.9135 | RF: ev=0.92463 rec=0.9829 | sel=0

dev/C tau=0.05 p=3 rounds=8 | em=0.9877 -> refine3=0.9841 frac9=1.000
dev/C tau=0.10 p=3 rounds=8 | em=0.9877 -> refine3=0.9832 frac9=1.000
dev/C tau=0.20 p=3 rounds=8 | em=0.9877 -> refine3=0.9790 frac9=1.000
dev/D tau=0.05 p=3 rounds=8 | em=0.8781 -> refine3=0.9711 frac9=0.955
dev/D tau=0.10 p=3 rounds=8 | em=0.8781 -> refine3=0.9754 frac9=1.000
dev/D tau=0.20 p=3 rounds=8 | em=0.8781 -> refine3=0.9835 frac9=1.000

dev/C mode=em seeds=[0, 1, 2] -> clusters=75 rec=0.9888 frac9=1.000
dev/C mode=em seeds=[0] -> clusters=48 rec=0.9877 frac9=1.000
dev/D mode=refine seeds=[0, 1, 2] -> clusters=91 rec=0.9895 frac9=1.000
dev/D mode=refine seeds=[0] -> clusters=48 rec=0.9778 frac9=1.000
dev/A mode=refine seeds=[0, 1, 2] -> clusters=89 rec=0.9993 frac9=1.000
dev/B mode=refine seeds=[0, 1, 2] -> clusters=70 rec=0.9989 frac9=1.000
dev/A tau=0.15 p=3 seeds=[0, 1, 2] -> K=91 rec=0.9999 frac9=1.000
dev/B tau=0.15 p=3 seeds=[0, 1, 2] -> K=65 rec=0.9997 frac9=1.000
dev/C tau=0.10 p=3 seeds=[0, 1, 2] -> K=74 rec=0.9733 frac9=1.000
dev/C tau=0.15 p=3 seeds=[0, 1, 2] -> K=75 rec=0.9715 frac9=0.958
dev/D tau=0.15 p=3 seeds=[0, 1, 2] -> K=101 rec=0.9920 frac9=1.000
dev/D tau=0.20 p=3 seeds=[0, 1, 2] -> K=98 rec=0.9924 frac9=1.000

A | EM rec=1.0000 kurt=16.7 resid=0.0012 || R3 rec=0.9999 kurt=15.9 resid=0.0013
B | EM rec=1.0000 kurt=4.0 resid=0.0013 || R3 rec=0.9997 kurt=4.0 resid=0.0015
C | EM rec=0.9888 kurt=9.7 resid=0.0010 || R3 rec=0.9715 kurt=9.0 resid=0.0016
D | EM rec=0.8810 kurt=1.8 resid=0.0104 || R3 rec=0.9920 kurt=2.0 resid=0.0099

ok

written

  sim: n_eff=30 sig_frac=0.0012 sim_rec=0.9620 frac9=0.900
A noisy=False | true_rec=1.0000 | sim_rec=0.9620 n_eff=30 resid=0.0012 | 64s
  sim: n_eff=48 sig_frac=0.0013 sim_rec=0.9065 frac9=0.854
B noisy=False | true_rec=1.0000 | sim_rec=0.9065 n_eff=48 resid=0.0013 | 72s
  sim: n_eff=47 sig_frac=0.0010 sim_rec=0.8195 frac9=0.723
C noisy=False | true_rec=0.9888 | sim_rec=0.8195 n_eff=47 resid=0.0010 | 68s
  sim: n_eff=48 sig_frac=0.0099 sim_rec=0.8841 frac9=0.812
D noisy=True | true_rec=0.9920 | sim_rec=0.8841 n_eff=48 resid=0.0099 | 128s

{"dictionary_score": 0.9975, "explained_variance": 0.9975, "l0": 11.407, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.6, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}
{"dictionary_score": 0.9978, "explained_variance": 0.9978, "l0": 1.212, "n_atoms": 40, "dead_atoms": 0, "accepted": true, "train_seconds": 2.7, "config": {"n_atoms": 40, "l1": 0.03, "steps": 3000, "seed": 0}}
{"dictionary_score": 0.9968, "explained_variance": 0.9968, "l0": 11.018, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.6, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}
{"dictionary_score": 0.9984, "explained_variance": 0.9984, "l0": 15.491, "n_atoms": 32, "dead_atoms": 0, "accepted": true, "train_seconds": 2.6, "config": {"n_atoms": 32, "l1": 0.03, "steps": 3000, "seed": 0}}
{"dictionary_score": 0.998, "explained_variance": 0.998, "l0": 10.887, "n_atoms": 24, "dead_atoms": 0, "accepted": true, "train_seconds": 2.6, "config": {"n_atoms": 24, "l1": 0.03, "steps": 3000, "seed": 0}}

== A (true F=24, atoms=48, matched=24) ==
  0 mass=  330.5 kurt=  43.71 hit=1 |   2 mass=  320.0 kurt=  46.20 hit=1 |   7 mass=  317.1 kurt=  46.73 hit=1 |   8 mass=  315.9 kurt=  45.91 hit=1 |   4 mass=  314.5 kurt=  47.10 hit=1 |   1 mass=  312.2 kurt=  47.38 hit=1 |   3 mass=  306.0 kurt=  48.13 hit=1 |   5 mass=  299.2 kurt=  49.59 hit=1 |  11 mass=  298.2 kurt=  49.22 hit=1 |  20 mass=  295.4 kurt=  50.03 hit=1 |   9 mass=  295.2 kurt=  50.10 hit=1 |  13 mass=  293.3 kurt=  50.50 hit=1 |  17 mass=  292.3 kurt=  50.00 hit=1 |  14 mass=  291.8 kurt=  50.05 hit=1 |  12 mass=  289.8 kurt=  51.45 hit=1 |  16 mass=  288.5 kurt=  51.53 hit=1
 mass hit-median=292.8 miss-median=14.9 ; kurt hit-med=50.08 miss-med=1341.27
== B (true F=32, atoms=48, matched=32) ==
  1 mass=  605.1 kurt=  25.17 hit=1 |   0 mass=  603.2 kurt=  25.27 hit=1 |   7 mass=  596.4 kurt=  25.07 hit=1 |  10 mass=  596.1 kurt=  25.32 hit=1 |   5 mass=  590.6 kurt=  26.08 hit=1 |   2 mass=  586.0 kurt=  26.35 hit=1 |   4 mass=  584.5 kurt=  25.43 hit=1 |  12 mass=  583.6 kurt=  26.25 hit=1 |   3 mass=  580.7 kurt=  26.10 hit=1 |  16 mass=  571.0 kurt=  26.35 hit=1 |   8 mass=  570.9 kurt=  26.24 hit=1 |  14 mass=  569.9 kurt=  26.13 hit=1 |  11 mass=  567.2 kurt=  26.85 hit=1 |  21 mass=  564.4 kurt=  26.75 hit=1 |  20 mass=  562.0 kurt=  27.77 hit=1 |  23 mass=  558.7 kurt=  27.09 hit=1
 mass hit-median=557.5 miss-median=52.9 ; kurt hit-med=27.32 miss-med=322.09
== C (true F=24, atoms=48, matched=30) ==
  0 mass=  492.0 kurt=  28.36 hit=1 |   2 mass=  450.7 kurt=  32.06 hit=1 |   4 mass=  421.3 kurt=  31.42 hit=1 |   1 mass=  418.5 kurt=  41.71 hit=0 |   7 mass=  405.3 kurt=  33.38 hit=1 |  21 mass=  387.3 kurt=  50.42 hit=0 |  29 mass=  385.9 kurt=  57.64 hit=0 |  10 mass=  367.0 kurt=  36.31 hit=1 |   6 mass=  356.1 kurt=  42.70 hit=1 |  31 mass=  342.4 kurt=  55.94 hit=0 |   3 mass=  337.6 kurt=  48.29 hit=1 |  12 mass=  337.3 kurt=  44.01 hit=1 |  25 mass=  335.4 kurt=  55.92 hit=0 |  22 mass=  327.8 kurt=  43.88 hit=1 |  24 mass=  295.6 kurt=  63.03 hit=0 |  28 mass=  293.1 kurt=  64.64 hit=0
 mass hit-median=182.7 miss-median=266.8 ; kurt hit-med=73.13 miss-med=69.20
== D (true F=22, atoms=48, matched=24) ==
 13 mass= 1102.8 kurt=  18.49 hit=1 |   0 mass= 1062.4 kurt=  20.72 hit=1 |   6 mass= 1027.6 kurt=  21.12 hit=1 |   8 mass= 1010.8 kurt=  21.57 hit=1 |   2 mass=  978.0 kurt=  22.53 hit=1 |   4 mass=  913.3 kurt=  23.03 hit=1 |   7 mass=  857.0 kurt=  22.83 hit=1 |  17 mass=  834.5 kurt=  26.03 hit=1 |   5 mass=  797.7 kurt=  25.37 hit=1 |  11 mass=  794.1 kurt=  27.92 hit=1 |  21 mass=  792.5 kurt=  25.29 hit=1 |  18 mass=  790.6 kurt=  26.48 hit=1 |   1 mass=  753.5 kurt=  28.60 hit=1 |   9 mass=  743.5 kurt=  26.20 hit=1 |  16 mass=  720.4 kurt=  28.25 hit=1 |  26 mass=  532.2 kurt=  36.14 hit=1
 mass hit-median=772.0 miss-median=305.5 ; kurt hit-med=27.20 miss-med=42.85

written

  sim: n_eff=48 sig_frac=0.0031 sim_rec=0.8648 frac9=0.812
REPORT P {"model": "P", "m": 12, "K": 48, "noisy": false, "resid_frac_em": 0.00314, "elbow_gap": 0.85, "elbow_n": 41, "resid_kurt_max": -0.03, "resid_var_frac": 0.0031, "sim_recovery": 0.8648, "sim_n_eff": 48, "sim_frac9": 0.812, "kurt": 2.43, "seconds": 66.9}
  sim: n_eff=48 sig_frac=0.0038 sim_rec=0.8841 frac9=0.833
REPORT R {"model": "R", "m": 12, "K": 48, "noisy": false, "resid_frac_em": 0.00377, "elbow_gap": 0.585, "elbow_n": 47, "resid_kurt_max": -0.3, "resid_var_frac": 0.00376, "sim_recovery": 0.8841, "sim_n_eff": 48, "sim_frac9": 0.833, "kurt": 5.51, "seconds": 71.4}
  sim: n_eff=64 sig_frac=0.0047 sim_rec=0.6170 frac9=0.016
REPORT S {"model": "S", "m": 16, "K": 64, "noisy": false, "resid_frac_em": 0.00469, "elbow_gap": 0.748, "elbow_n": 63, "resid_kurt_max": 0.07, "resid_var_frac": 0.00438, "sim_recovery": 0.617, "sim_n_eff": 64, "sim_frac9": 0.016, "kurt": 0.12, "seconds": 86.3}
  sim: n_eff=48 sig_frac=0.0016 sim_rec=0.9101 frac9=0.833
REPORT T {"model": "T", "m": 12, "K": 48, "noisy": false, "resid_frac_em": 0.00157, "elbow_gap": 0.538, "elbow_n": 40, "resid_kurt_max": 0.07, "resid_var_frac": 0.00155, "sim_recovery": 0.9101, "sim_n_eff": 48, "sim_frac9": 0.833, "kurt": 1.99, "seconds": 59.5}
  sim: n_eff=38 sig_frac=0.0013 sim_rec=0.9679 frac9=0.947
REPORT Q {"model": "Q", "m": 20, "K": 80, "noisy": false, "resid_frac_em": 0.0013, "elbow_gap": 0.162, "elbow_n": 36, "resid_kurt_max": 2.59, "resid_var_frac": 0.00126, "sim_recovery": 0.9679, "sim_n_eff": 38, "sim_frac9": 0.947, "kurt": 20.61, "seconds": 115.8}

dev/A    projkurt max=14.79 med=3.94 | resid_kurt5=3.78 resid_var=0.0012
dev/B    projkurt max=3.01 med=1.61 | resid_kurt5=1.70 resid_var=0.0013
dev/C    projkurt max=7.57 med=3.92 | resid_kurt5=5.45 resid_var=0.0009
dev/D    projkurt max=1.76 med=0.62 | resid_kurt5=-0.22 resid_var=0.0035
audit/P  projkurt max=2.10 med=0.86 | resid_kurt5=-0.56 resid_var=0.0025
audit/Q  projkurt max=9.01 med=3.67 | resid_kurt5=3.69 resid_var=0.0013
audit/R  projkurt max=4.14 med=1.60 | resid_kurt5=-0.30 resid_var=0.0034
audit/S  projkurt max=0.31 med=0.09 | resid_kurt5=-0.37 resid_var=0.0017
audit/T  projkurt max=1.65 med=0.82 | resid_kurt5=-0.13 resid_var=0.0014

S SAE l1=0.10 score=0.9801 l0=23.82 dead=0
S SAE l1=0.30 score=0.8675 l0=16.41 dead=0
S SAE l1=0.60 score=0.6605 l0=9.55 dead=0
S SAE l1=1.00 score=0.3787 l0=5.00 dead=0
chance recovery m=16 K=64 F=32: 0.529
chance recovery m=16 K=64 F=48: 0.516
chance recovery m=16 K=64 F=64: 0.488

dev/A stable_atoms=25 big_stable=24 true_F=24
dev/B stable_atoms=35 big_stable=34 true_F=32
dev/C stable_atoms=16 big_stable=15 true_F=24
dev/D stable_atoms=0 big_stable=0 true_F=22
audit/P stable_atoms=5 big_stable=5 true_F=None
audit/Q stable_atoms=35 big_stable=35 true_F=None
audit/R stable_atoms=16 big_stable=16 true_F=None
audit/S stable_atoms=0 big_stable=0 true_F=None
audit/T stable_atoms=18 big_stable=18 true_F=None

  sim: n_eff=48 sig_frac=0.0030 sim_rec=0.8772 frac9=0.833
P R3 | resid=0.0030 sim=0.8772 frac9=0.833 n_eff=48 stable=20 | 129s
  sim: n_eff=48 sig_frac=0.0015 sim_rec=0.9039 frac9=0.854
T R3 | resid=0.0015 sim=0.9039 frac9=0.854 n_eff=48 stable=34 | 132s

P (48, 12) float32 unit? 1.1920929e-07