Autoresearch Dashboard

Autonomous GPT pretraining research using Karpathy's autoresearch framework. 5-minute training experiments on Modal A100-80GB GPUs, optimizing val_bpb (validation bits per byte).

Best val_bpb

1.0387

↓ 0.5% from baseline

Baseline val_bpb

1.0440

Unmodified train.py

Experiments Run

Across 38 sessions

Improvement

−0.0053

val_bpb reduction

GPU

A100-80GB

Modal cloud

val_bpb Over Experiments

Optimizer Steps per Experiment

Peak VRAM Usage (GB)

val_bpb vs Steps (Efficiency)

All Experiments

#	Session	Description	val_bpb	Steps	VRAM	Status
0	10	Increase logit softcap from 15 to 30	1.0440	1486	—	baseline
1	11	Increase warmdown ratio from 0.67 to 0.75	1.0401	1504	—	discard
2	12	Increase scalar LR from 0.5 to 0.8	1.0401	1504	—	discard
3	13	Increase matrix LR from 0.08 to 0.1	1.0413	1	—	discard
4	14	Increase unembedding LR from 0.004 to 0.008	1.0468	1	—	discard
5	15	Increase warmup ratio from 0.02 to 0.05	1.0387	1	—	★ best
6	16	Increase warmup ratio from 0.05 to 0.08	1.0500	1	—	discard
7	17	Lower softcap from 15 to 10	1.0408	1	—	discard
8	18	Increase embedding LR from 1.0 to 1.5	1.0475	1	—	discard
9	19	Lower matrix LR from 0.08 to 0.07	1.0473	1	—	discard
10	20	Increase FINAL	1.0472	1	—	discard
11	21	Increase Adam beta1 from 0.8 to 0.9	1.0517	1	—	discard
12	22	Increase x0_lambda init from 0.1 to 0.15 for stronger skip connections	1.0484	1	—	discard
13	23	Reduce Muon Nesterov momentum target from 0.95 to 0.90	1.0470	1	—	discard
14	24	Reduce warmdown ratio from 0.67 to 0.60	1.0491	1	—	discard
15	25	Increase depth from 8 to 10 (ASPECT_RATIO=64 unchanged)	1.0610	811	—	discard
16	26	Reduce x0_lambda init from 0.1 to 0.05 (weaker skip connections)	1.0487	1	—	discard
17	27	Increase RoPE base frequency from 10000 to 50000	1.0405	1	—	discard
18	28	Increase Muon ns_steps from 5 to 6 (more Newton-Schulz orthogonalization iterations)	1.0492	1	—	discard
19	29	Change window pattern from "SSSL" to "SSL" — more full-context attention layers	1.0492	1	—	discard
20	2	Halve batch size again: 2^18 → 2^17 with DEVICE	1.0472	1357	22.7 GB	keep
21	30	Increase ASPECT_RATIO 64→72 — wider model (512→640 dim, 4→5 heads)	1.0521	999	—	discard
22	31	1.0387	1.0437	1468	—	discard
23	34	1.0387	1.2263	1443	—	discard
24	35	1.0387	1.0636	1342	—	discard
25	36	1.0387	1.0460	1478	—	discard
26	38/td>	Halve batch size further: 2^17 →51516 with DE53CE	1.0584	2557	11.5 GB	discard
27	4	Increase embedding LR from 0.6 to 1.0	1.0458	1348	22.7 GB	keep
28	5	Compound: add 2% warmup + increase matrix LR 0.06 → 0.08	1.0437	1495	22.7 GB	keep
29	6	Reduce warmdown ratio from 0.67 to 0.5	1.0557	1321	22.7 GB	discard
30	7	Reduce weight decay from 0.2 to 0.1	1.0420	1463	—	keep
31	8	Reduce weight decay from 0.1 to 0.05	1.0395	1517	—	keep
32	9	Reduce weight decay from 0.05 to 0.025	1.0437	1418	—	discard

What Worked

Increase warmup ratio from 0.02 to 0.05 → 1.0387
Halve batch size again: 2^18 → 2^17 with DEVICE → 1.0472
Increase embedding LR from 0.6 to 1.0 → 1.0458
Compound: add 2% warmup + increase matrix LR 0.06 → 0.08 → 1.0437
Reduce weight decay from 0.2 to 0.1 → 1.0420
Reduce weight decay from 0.1 to 0.05 → 1.0395

What Didn't Work

Increase logit softcap from 15 to 30 → 1.0440
Increase warmdown ratio from 0.67 to 0.75 → 1.0401
Increase scalar LR from 0.5 to 0.8 → 1.0401
Increase matrix LR from 0.08 to 0.1 → 1.0413
Increase unembedding LR from 0.004 to 0.008 → 1.0468
Increase warmup ratio from 0.05 to 0.08 → 1.0500
Lower softcap from 15 to 10 → 1.0408
Increase embedding LR from 1.0 to 1.5 → 1.0475
Lower matrix LR from 0.08 to 0.07 → 1.0473
Increase FINAL → 1.0472
Increase Adam beta1 from 0.8 to 0.9 → 1.0517
Increase x0_lambda init from 0.1 to 0.15 for stronger skip connections → 1.0484
Reduce Muon Nesterov momentum target from 0.95 to 0.90 → 1.0470
Reduce warmdown ratio from 0.67 to 0.60 → 1.0491
Increase depth from 8 to 10 (ASPECT_RATIO=64 unchanged) → 1.0610
Reduce x0_lambda init from 0.1 to 0.05 (weaker skip connections) → 1.0487
Increase RoPE base frequency from 10000 to 50000 → 1.0405
Increase Muon ns_steps from 5 to 6 (more Newton-Schulz orthogonalization iterations) → 1.0492
Change window pattern from "SSSL" to "SSL" — more full-context attention layers → 1.0492
Increase ASPECT_RATIO 64→72 — wider model (512→640 dim, 4→5 heads) → 1.0521
1.0387 → 1.0437
1.0387 → 1.2263
1.0387 → 1.0636
1.0387 → 1.0460
Halve batch size further: 2^17 → 2^16 with DEVICE → 1.0584
Reduce warmdown ratio from 0.67 to 0.5 → 1.0557
Reduce weight decay from 0.05 to 0.025 → 1.0437

515

Next Experiments

Increase softcap (15 → 30) — may help with logit saturation
Try higher embedding LR (1.0 → 1.5)
Try 10 layers with smaller aspect ratio (e.g., aspect=48 → 480 dim, ~40M params)
Increase warmdown ratio (0.67 → 0.75)
Try scalar LR tuning (0.5 → 0.8)
Try matrix LR 0.1 (push optimizer harder)

Current Best Configuration

RationaleSessions 7 and 8 showed successive improvements from reducing weight decay (0.2→0.1→0.05), suggesting the model benefits from less regularization in short 5-minute runs. This experiment pushes further to see if 0.025 continues the trend.

EMBEDDING_LR1.0

UNEMBEDDING_LR0.004

MATRIX_LR0.08

SCALAR_LR0.5

WARMUP_RATIO0.02

WARMDOWN_RATIO0.67

WEIGHT_DECAY0.05

DEPTH8, ASPECT_RATIO = 64

Run Karpathy's auto research continuously to optimize a model