mchicao.dev
ProjectsMethodLabGitHub

Tiny Transformer Lab

A blog documenting experiments with local hardware and small language models trainable on consumer-grade AMD GPUs, built directly from research papers and public datasets.

View GitHub Repository↗← Back to Home
Hardware

AMD Radeon RX 6750 GRE (10GB VRAM)

Runtime & Stack

PyTorch 2.x + Native ROCm / HIP

Lab Notes & Experiments

Empirical findings measured through verifiable checkpoints and paired statistical analysis.

Pre-Training · Architecture · ConfirmedOctober 04, 2026
ID: night-qk-norm-holdout

QK-Norm confirmed on holdout: 3M model, 3 of 3 seeds

A technique that previously came out negative worked under a different training regime. With the current recipe and budget, QK-Norm improved the 3M model's predictions on data held out from training. The pending evaluation was completed overnight; training was not repeated.

Empirical findings:
  • Mean holdout NLL dropped from 2.27275 to 2.21031, improving on all three seeds.
  • NLL measures how surprised the model is by the correct tokens: lower is better.
  • Limit: lower NLL does not yet prove good translation or agentic capability.
Holdout NLL (baseline)2.27275
Holdout NLL (QK-Norm)2.21031
Seeds improved3 / 3
Model3M
Technical takeaway:

“A past negative is not final: the same technique can work with another recipe and budget. But NLL is not capability.”

Inference · Test-Time SearchOctober 04, 2026
ID: night-search-vs-greedy

More search at answer time did not guarantee better answers

Single-power beat best-of-N at comparable compute. Still, greedy decoding (directly picking the most likely token) scored higher than the single-power pilot while using far less compute.

Empirical findings:
  • Mean accuracy from best-of-N to single-power: 36.1% → 61.1%.
  • Greedy got 9/12 versus 8/12 for the single-power pilot, at much lower compute.
  • Only 12 already-seen problems and sampling seeds: this does not show a general advantage.
Best-of-N → Single-power36.1% → 61.1%
Greedy9 / 12
Single-power pilot8 / 12
Problems12
Technical takeaway:

“Always compare a complex search against a simple baseline. Beating another expensive search is not enough.”

Post-Training · QLoRA · Local FeasibilityOctober 04, 2026
ID: night-minicpm-qlora-amd

MiniCPM (2.52B) QLoRA works on a 10 GB AMD GPU

Local evidence of technical feasibility, not just an estimate: 4.30M-parameter adapters trained on a frozen, quantized ~2.52B base. All three replicas at 512 context finished correctly, and save, load and resume were validated.

Empirical findings:
  • Observed reserved memory: 512 tokens, 4.05 GiB; 2K, 5.23 GiB; 4K, 9.30 GiB.
  • The 4K run was stopped because it exceeded the 8.5 GiB limit.
  • Limit: few arithmetic examples with 2–3 token answers; this does not show enough memory for long translations or bilingual quality.
Base / Adapters~2.52B / 4.30M
VRAM 512 tok4.05 GiB
VRAM 2K tok5.23 GiB
VRAM 4K tok9.30 GiB (exceeds 8.5)
Technical takeaway:

“QLoRA of a 2.5B model fits in 10 GB up to ~2K context; 4K does not. Quality on long tasks remains unproven.”

RL · Adaptive Curriculum · NegativeOctober 04, 2026
ID: night-adaptive-curriculum-rl

Adaptive curriculum did not improve this RL pilot

First we verified the model produced both correct and incorrect answers, which is needed for a learning signal. Then we compared uniform problem selection against adaptive selection, which prioritized levels where it sometimes succeeded and sometimes failed.

Empirical findings:
  • Baseline 52/80; uniform 51/80; adaptive 51/80.
  • This rule and budget showed no benefit; it does not justify more replicas.
  • A single-seed pilot: it does not show that every adaptive curriculum is useless.
Baseline52 / 80
Uniform51 / 80
Adaptive51 / 80
Seeds1
Technical takeaway:

“Changing which examples are prioritized does not guarantee improvement. A cheap negative avoids longer runs with no favorable signal.”

Architecture · Recurrence · NegativeOctober 04, 2026
ID: night-k2-recurrence-extension

K2: artificially extending recurrence to 32 passes destabilizes it

The existing K2's second pass improved the probe NLL, but artificially extending recurrence to 32 passes made it worse and internal magnitudes grew sharply. This was not a trained K32.

Empirical findings:
  • Probe NLL with two passes: 2.08 → 1.53.
  • With 32 artificially extended passes, NLL worsened to 7.55.
  • The result points to an architectural dependency before trying fixed points or sharing the KV cache.
NLL pass 1 → 22.08 → 1.53
NLL at 32 passes7.55
Internal magnitudesGrow sharply
Trained K32No
Technical takeaway:

“Improving with two passes does not imply improving with many more.”

Pre-Training · Scaling & StabilityOctober 03, 2026
ID: f01-24m-scaling-capped-amp

F01 Data Scaling to 24.5M Tokens & Capped Mixed-Precision (AMP) on AMD ROCm

Overnight pre-training campaign evaluating dense 3M transformer scaling from 1M to 24.576M tokens (108,137 unique documents) using byte-level BPE-4096 on an AMD Radeon RX 6750 GRE 10GB. Validated across 3 independent seeds (42, 43, 44), followed by a 98.3M scaling attempt that uncovered dynamic loss scale overflow and established a capped GradScaler ceiling for rock-solid stability.

Empirical findings:
  • Substantial NLL drop across 3 seeds: Validation NLL dropped from 2.878 nats (E11 control) to a mean of 2.073 nats (s42: 2.080, s43: 2.067, s44: 2.073)—a ~0.805 nats gain and a 55% reduction in perplexity (17.78 → 7.94) with 0 skipped updates.
  • Zero-overlap secondary validation confirmed: Evaluated on 1,024 held-out stories (201.89K targets) completely isolated from training, validation NLL improved from 3.181 nats to 2.387 nats (gain of 0.794 nats), verifying genuine language acquisition rather than memorization.
  • AMP numerical ceiling diagnosed at 98.3M scale: In a 98.304M token run, unconstrained dynamic GradScaler climbed to 65,536, causing an arithmetic overflow and gradient skip at step 15,201 (62.26M targets). Capping the scale factor to 8,192 completely eliminated skips and stabilized FP16 on ROCm/HIP.
  • Parquet SHA-256 data pipeline: Shifted from fragile row-based streaming to local PyArrow Parquet shards with strict SHA-256 validation against official Hugging Face LFS, ensuring bit-for-bit batch reproducibility.
Final Validation NLL2.073 nats (mean 3s)
Perplexity Drop17.78 → 7.94
Held-out NLL (Zero-Overlap)3.181 → 2.387 nats
Capped AMP Scale8,192 (0 skips)
Technical takeaway:

“Scaling clean pre-training data by 4x yields predictable logarithmic NLL gains on small models. Crucially for consumer AMD hardware, long-horizon FP16 training requires bounding the dynamic loss scale to prevent silent gradient overflow.”

Post-Training · SFT vs. GRPOOctober 03, 2026
ID: f04-sft-vs-grpo-matched

F04 Active-Compute Matched Comparison: SFT vs. GRPO on 96.2M Model

Paired active-compute benchmark on the bench-labs/tinctura-v1 96.2M parameter model across two independent seeds (42 and 43), starting from the same foundation integer SFT checkpoint. SFT and GRPO were allocated identical active GPU training time (~911s in seed 42, ~676s in seed 43) under strict answer-boundary enforcement.

Empirical findings:
  • Decisive SFT accuracy victory: SFT outperformed GRPO in both paired seeds (+6.64 pp in s42: 55.08% vs 48.44%; +8.59 pp in s43: 55.47% vs 46.88%).
  • GRPO preserves pre-trained language 5x better: In a 61,440-token held-out language retention audit (2048 context), GRPO exhibited only +0.0009 nats drift vs. base, compared to +0.0055 nats for SFT.
  • Compositional limit on 3-operand math: Both SFT and GRPO scored 0% accuracy on raw expressions like 'a - b + c' due to truncated attention that ignored the third operand without scratchpads.
  • Temperature scaling reveals unguided consensus limits: T=1.2 boosted oracle pass@8 to 81.25%, but unguided majority vote plateaued at 65.6%, showing that stochastic exploration requires an external verifier.
Best Greedy Acc55.47% (SFT s43)
GRPO Greedy Acc46.88% (GRPO s43)
GRPO Language Drift+0.0009 nats (5x less)
Peak VRAM (FP32)4,722 MiB
Technical takeaway:

“When outputs are short and deterministic, direct supervised gradients are more compute-efficient than group exploration. However, GRPO acts as a strong intrinsic regularizer against catastrophic forgetting.”

Post-Training · Supervised SFTOctober 02, 2026
ID: f02-foundation-integer-sft

F02 Arithmetic SFT on 96M Model: Supervised task alignment with zero language decay

Supervised fine-tuning (SFT) with 512 optimizer updates on the bench-labs/tinctura-v1 96.2M base model in FP32 on an AMD Radeon RX 6750 GRE (10GB). Designed as a stable foundation checkpoint for multi-step reasoning tasks after identifying two-step word problem deficits. Processed 293,376 tokens with strict completion-only loss masking, achieving rapid convergence and zero catastrophic forgetting.

Empirical findings:
  • Steady policy loss convergence: dropped cleanly from 1.354 (batch 31) down to 0.214 (batch 511) across 512 effective updates.
  • Strict natural language retention (Canary PASS): initial validation NLL 2.5917 moved to 2.5969 (+0.0052 nats over 4,096 canary tokens), proving zero language collapse.
  • Extreme local VRAM efficiency: peak memory usage was 2,430 MiB (~2.4 GB out of 10GB) and 281.4 active loop seconds (~4.7 min of pure compute).
  • Full cryptographic integrity audit: 5 intermediate checkpoints (batch 32, 64, 128, 256, 512) saved and audited with SHA256 hashes.
Policy Loss1.354 → 0.214
Canary Delta+0.0052 nats
Peak VRAM2,430 MiB
Updates / Tokens512 / 293K
Technical takeaway:

“Completion-only supervised fine-tuning instills accurate arithmetic reasoning without damaging the pre-trained language distribution or exhausting local consumer GPU memory.”

Dev Audit · Scientific GatingOctober 02, 2026
ID: f04-stage2-dev-gate

F04 Stage 2: Negative dev gate and pre-registered scientific discipline before RLVR

Pre-flight evaluation of the 96.2M base model on 128 dev problems prior to launching a 1K-prompt SFT vs. GRPO comparison on two-step word problems. Yielding 0 correct samples in two_step_word (0/256), our pre-registered protocol automatically stopped execution at 0 updates to prevent reward hacking and wasted compute.

Empirical findings:
  • Strong single-op carry addition: 26/32 greedy (81.3%) and 100% Pass@8 with 93.4% format validity.
  • Zero signal on 2-step word problems: 0/256 correct samples and 0.78% valid format, precluding valid RLVR credit assignment without prior SFT.
  • Pre-registered gate discipline PASS: automatic termination at 0 updates rather than modifying parsers or fudging dev gates post hoc.
  • Direct research pivot: proved why F02 Foundation SFT had to precede any multi-step RL comparison.
Carry Addition Pass@8100.0%
Two-Step Word Correct0 / 256
Dev Informative Groups44 / 128
Updates Executed0 (Halted by Gate)
Technical takeaway:

“Real science halts experiments when empirical signal is absent rather than forcing ungrounded updates or manipulating benchmark parsers.”

Post-Training · RLVROctober 02, 2026
ID: f04-grpo-96m

GRPO on 96M Model: Post-training with binary verification on AMD GPU

Post-training with GRPO (Group Relative Policy Optimization) on top of the bench-labs/tinctura-v1 base model (96.2M parameters) from Hugging Face. Full-parameter FP32 tuning on local AMD hardware: 512 rollouts (8 per prompt across 64 arithmetic tasks held out from eval) and 16 optimizer updates guided by a binary reward verifier.

Empirical findings:
  • F00 (Answer Boundary): Strict answer boundary enforcement on newline/EOS; the untrained 96M base achieved 31/32 greedy accuracy (96.9%).
  • Sampled accuracy increased from 78.1% to 89.5% (+11.4%): GRPO effectively shifted probability mass toward correct answers under sampling.
  • Greedy accuracy stayed identical at 96.9% (31/32) and Pass@8 at 100%: it did not expand greedy capabilities but significantly boosted sample yield.
  • Audit and language canary PASS: the language canary metric remained well within safe limits with no observed distribution collapse.
Sampled Acc78.1% → 89.5%
Greedy Acc96.9% (31/32)
Pass@8100%
Rollouts / Updates512 / 16
Technical takeaway:

“GRPO is technically viable on local AMD GPUs and notably boosts sampling consistency, though rigorous benchmarking against additional SFT and multi-seed replications are still needed.”

Data Scaling · 3 SeedsOctober 02, 2026
ID: e11-coverage-campaign

E11: Fresh unique data vs. multi-epoch repetition in 3M model across 3 seeds

Is multi-epoch repetition justified or does fresh token stream strictly dominate? We trained 3M-parameter dense transformers from scratch on 6,144,000 tokens across 3 independent seeds (42, 43, 44), comparing unique non-wrapping stories against the repeated E8 baseline.

Empirical findings:
  • Categorical win across all 3 seeds: Seed 42 (-0.233 nats), Seed 43 (-0.213 nats), Seed 44 (-0.208 nats).
  • Mean validation NLL dropped by -0.2180 nats (from 3.0962 down to 2.8783 nats).
  • 19.58% net perplexity reduction under identical token compute and wall-clock budget (~158s train loop per seed).
  • Architectural rule: prioritize fresh data coverage before pursuing additional model or optimizer complexity.
Repeated NLL (E8)3.0962
Unique Data NLL (E11)2.8783
Perplexity Delta-19.58%
Favorable Seeds3 of 3 (100%)
Technical takeaway:

“Data stream freshness drastically outperforms minor architectural or optimizer tweaks in compute-constrained regimes.”

FFN ArchitectureOctober 2026
ID: ffn-gelu-swiglu

GELU vs. SwiGLU: Empirical testing across 3 seeds

SwiGLU is the de facto standard in models like Llama and Gemma, but does it offer a statistically significant edge in controlled budgets? We evaluated both variants in 3M-parameter models trained from scratch under identical 6,144,000 token budgets.

Empirical findings:
  • In seed42, SwiGLU showed a slight advantage of -0.0072 nats/target (0.72% lower perplexity).
  • When replicated across seed43 and seed44, the advantage vanished: the mean final SwiGLU-GELU delta was +0.0187 nats (GELU finished better in two of the three seeds).
  • Paired t-tests showed no statistical significance at this scale; SwiGLU does not guarantee a universal win without expanded compute.
GELU Throughput37,775 tok/s
SwiGLU Throughput34,323 tok/s
VRAM Peak~262 MiB
Parameters3M
Technical takeaway:

“Scientific rigor requires multi-seed verification before treating industry architectural trends as unquestionable dogma.”

OptimizersOctober 2026
ID: adamw-vs-muon

AdamW vs. Muon: Orthogonalized optimizers at 3M scale

Muon (momentum with orthogonalization via Newton-Schulz iterations) promises to accelerate pre-training. We ran a paired replication against the standard AdamW baseline.

Empirical findings:
  • AdamW consistently achieved lower final NLL in both seed42 and seed43 (3.192 vs 3.302).
  • The difference favoring AdamW at the best checkpoint was ~+0.052 nats.
  • Fixed QK-Norm with Muon exhibited validation generalization degradation past its minimum.
AdamW Best NLL3.1778
Muon Best NLL3.2297
Budget6.14M tokens
Updates1,500 effective
Technical takeaway:

“Optimizers designed for billion-parameter regimes require careful re-tuning before outperforming AdamW on compact model budgets.”

Primary Sources & Reference Architecture

Key Concepts, Papers & Resources

Essential primary sources, core mathematical intuition, and practical hardware realities on consumer AMD GPUs (10GB VRAM).

GRPO (Group Relative Policy Optimization)Post-Training & RL

An online reinforcement learning algorithm that discards PPO's separate value/critic network. For each prompt, it generates a group of completions, scores them with an automated reward verifier, and computes advantages normalized by the group's mean and standard deviation.

Local Hardware Reality (10GB AMD):

On 10GB AMD: Cuts training memory by ~50% versus PPO. Allowed running 8 concurrent rollouts per prompt in native FP32 on our RX 6750 GRE with only 4,722 MiB peak VRAM.

Foundational Papers & High-Trust Resources:
  • DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (Shao et al., 2024) ↗

    Introduces the GRPO formulation and demonstrates mathematical reasoning improvements without a critic network.

  • DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI, 2025) ↗

    Large-scale demonstration of reasoning emergence (self-reflection, long CoT) trained purely via GRPO.

  • Hugging Face TRL GRPO Trainer Documentation ↗

    Practical open-source implementation guidelines, prompt formatting, and advantage computation.

SFT (Supervised Fine-Tuning)Supervised Alignment

Standard maximum-likelihood training on curated prompt-completion pairs using cross-entropy loss applied strictly to completion tokens (loss masking). Directly teaches target syntax, answer conventions, and domain vocabulary via teacher forcing.

Local Hardware Reality (10GB AMD):

On 10GB AMD: Compute and memory champion. Ran 512 updates across 96.2M parameters in 281 seconds with just 2,430 MiB of VRAM.

Foundational Papers & High-Trust Resources:
  • LIMA: Less Is More for Alignment (Zhou et al., 2023) ↗

    Shows that 1,000 meticulously curated examples can align base models effectively without vast datasets.

  • Finetuning with Sampling: SFT Learns Better Than You Think (Karan et al., 2026) ↗

    Proves that supervised fine-tuning on filtered on-policy trajectories matches or exceeds online RL in generalization and retention.

Quantization (INT8, INT4, FP8, AWQ, GGUF)Compression & Serving

Reducing the numeric precision of weights and activations from FP32/FP16 down to 8-bit or 4-bit integers. Reduces memory capacity requirements and memory bandwidth pressure, which is the primary bottleneck in autoregressive token generation.

Local Hardware Reality (10GB AMD):

On 10GB AMD: Enables loading and serving 14B parameter models and fine-tuning 7B models locally that would otherwise trigger Out-Of-Memory errors in unquantized FP16.

Foundational Papers & High-Trust Resources:
  • QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023) ↗

    Introduces NF4 (NormalFloat4) and double quantization to fine-tune models into consumer GPUs without fidelity loss.

  • AWQ: Activation-aware Weight Quantization for LLM Compression (Lin et al., 2023) ↗

    Protects the top 1% salient weights based on activation magnitudes, outperforming round-to-nearest quantization.

  • llama.cpp & GGUF Specification (Gerganov) ↗

    The open standard for fast CPU/GPU quantized inference, supporting k-quants on consumer hardware.

ROCm & HIP (AMD Radeon Open Compute)Hardware & Runtime

AMD's open-source software stack for GPU computation. HIP (Heterogeneous-compute Interface for Portability) is a C++ runtime and kernel language that compiles natively to AMD GPUs via HIP-Clang, while maintaining source-level compatibility with CUDA.

Local Hardware Reality (10GB AMD):

On 10GB AMD: PyTorch with ROCm runs full forward and backward training loops natively on Windows and Linux on consumer RDNA2/RDNA3 silicon without proprietary vendor locks.

Foundational Papers & High-Trust Resources:
  • AMD ROCm Documentation & Architecture Guide ↗

    Official documentation covering GPU architecture, HIP programming model, and ROCm libraries (rocBLAS, MIOpen).

  • PyTorch for AMD ROCm Platform ↗

    Standard binary wheels and setup instructions for running PyTorch on AMD hardware.

  • HIP Programming Guide & hipify Tools ↗

    Guide for converting existing CUDA kernels into portable HIP C++ code.

CUDA (Compute Unified Device Architecture)Hardware & Acceleration

Nvidia's parallel computing platform and API model that established the foundational primitives of deep learning systems. Key architectural concepts include Streaming Multiprocessors (SMs), warps (32 threads executing in SIMT lockstep), shared memory (SRAM), and Tensor Cores.

Local Hardware Reality (10GB AMD):

Understanding memory hierarchy (on-chip SRAM vs off-chip VRAM) is universal across both CUDA and AMD RDNA: memory bandwidth, not compute FLOPs, is almost always the limiting factor during LLM inference.

Foundational Papers & High-Trust Resources:
  • NVIDIA CUDA C++ Programming Guide ↗

    The foundational reference for GPU thread hierarchy, memory coalescing, and hardware execution models.

  • FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (Dao et al., 2022) ↗

    Essential reading on hardware-aware algorithm design, showing how GPU SRAM tiling avoids slow high-bandwidth memory roundtrips.

RLVR (Reinforcement Learning with Verifiable Rewards)Reasoning & Alignment

Reinforcement learning restricted to domains where answer validity can be decided deterministically by automated code: unit test passes, compiler outputs, symbolic solvers (SymPy), or exact match parsers, avoiding vulnerable subjective reward models.

Local Hardware Reality (10GB AMD):

On 10GB AMD: Deterministic verifiers run on the host CPU in milliseconds with zero GPU memory overhead, completely eliminating the need to fit a 7B reward model into VRAM.

Foundational Papers & High-Trust Resources:
  • Training Verifiers to Solve Math Word Problems (Cobbe et al., 2021) ↗

    The GSM8K foundational study analyzing solution verifiers and outcome verification.

  • The Ultimate Guide to Multi-Harness RL (FineEnvs, 2026) ↗

    Practical field guide on reward shaping, token fidelity masks, and preventing zero-contrast rollout groups in agentic RL.

CoT (Chain-of-Thought & Scratchpad Reasoning)Reasoning Dynamics

Generating intermediate reasoning tokens before the final answer. Because a transformer executes a fixed amount of computation per token (depth D), emitting scratchpad tokens allows the network to carry out sequential multi-step state transitions across time.

Local Hardware Reality (10GB AMD):

Direct generation in our 5-layer 96M model collapsed on 3-operand math 'a - b + c' (0% accuracy). CoT is computationally mandatory for compact models to break down chained operations.

Foundational Papers & High-Trust Resources:
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (Wei et al., 2022) ↗

    Seminal paper demonstrating that prompting models to decompose problems into steps unlocks emergent multi-step reasoning.

  • Show Your Work: Scratchpads for Intermediate Computation with Language Models (Nye et al., 2021) ↗

    Proves that small transformers can perform multi-digit addition and polynomial evaluation when trained with intermediate scratchpad traces.

  • Let's Verify Step by Step (Lightman et al., 2023) ↗

    Introduces Process Reward Models (PRMs) to reward individual reasoning steps rather than just the final outcome.

Looping (Looped Transformers & Weight-Tied Recurrence)Neural Architecture

Executing intermediate transformer layers iteratively across multiple computational passes (K > 1) using tied parameters. Decouples the parameter storage footprint of the network from the effective computational depth applied to input tokens.

Local Hardware Reality (10GB AMD):

On 10GB AMD: Cuts weight VRAM by 2x to 4x. In our 3M pre-training experiments, Looped-K2 refined representations across passes, though standard feed-forward layers retained better parameter efficiency per FLOP.

Foundational Papers & High-Trust Resources:
  • Squeezing More from Limited Data with Recursive Transformers (Joshi et al., 2026) ↗

    Comprehensive study on recurrent layer sharing under constrained data budgets.

  • Decoding Looped Transformers Better for (Almost) Free (Karan et al., 2026) ↗

    Presents LoopCD (Loop Contrastive Decoding), using contrast between early and late loop iterations to boost generation quality at inference.

  • SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers (Bhojanapalli et al., 2026) ↗

    Evaluates compute-matched scaling laws comparing recurrent transformers against standard dense and MoE baselines.

Research Roadmap

Upcoming and in-progress experimental milestones in the lab.

F00Completed

Deterministic Arithmetic Evaluator

Deterministic parser with strict answer boundary (newline/EOS). Baseline established at 31/32 greedy accuracy (96.9%) on 96M.

F02Completed

Arithmetic SFT on 96M Model

Supervised fine-tuning across 512 updates completed on local GPU; policy loss reduced to 0.214 with strict language preservation canary PASS (+0.0052 nats).

F04Completed (2 Seeds Paired)

RLVR vs. Additional SFT

Paired active-time matched comparison on 96.2M model across 2 seeds. SFT won on greedy accuracy (55.5% vs 46.9%), while GRPO demonstrated 5x lower language drift (+0.0009 nats vs +0.0055 nats). Gate: FAIL on RL superiority, PASS on retention.

F01Completed (3 Seeds Confirmed)

Unique Token Scaling (24.5M) & Capped AMP

Verified dense 3M scaling to 24.576M unique tokens across 3 seeds (42, 43, 44): validation NLL dropped to 2.073 nats (mean gain of 0.805 nats / 55% perplexity reduction). Capped AMP at 8,192 for numerical stability on ROCm.

F05Planned

LoopCD Inference on 3M Checkpoints

Loop Contrastive Decoding to suppress repetition and stabilize generation during local inference without retraining.

Open and reproducible repository

All training code, audit scripts without retraining, and experiment configurations are version-controlled on GitHub.

github.com/Mchicao/tiny-transformer-lab ↗Back to Home

mchicao.dev · Tiny Transformer Lab · 2026