Neural Networks From First Principles: Raw WASM, JS & WebGPU

1. Perceptrons, Feedforward MLPs & Backpropagation (1958–1986)

In 1958, Frank Rosenblatt introduced the Perceptron, a linear threshold unit computing ŷ = step(w · x + b). Eleven years later, Marvin Minsky and Seymour Papert proved in Perceptrons (1969) that a single linear layer defines only a single flat hyperplane w₀x₀ + w₁x₁ + b = 0 in input space—making it mathematically impossible to represent non-linearly separable predicates such as XOR parity (where opposite corners of the unit square share a class).

The resolution—championed by Rumelhart, Hinton, and Williams (1986) building on Linnainmaa's (1970) reverse-mode automatic differentiation—is to stack hidden layers interleaved with point-wise non-linear activation functions σ(z). Geometrically, each hidden layer folds and stretches the coordinate manifold along d_out hyperplanes so that classes tangled in input space become linearly separable at the final readout layer.

Training a multilayer network requires computing how a nudge to any weight W_i,j^(l) in layer l affects the scalar loss L(ŷ, y) at layer L. Naive finite differences (perturbing each of P parameters one by one) requires O(P) forward passes per step. Reverse-Mode Backpropagation computes the exact gradient for all P parameters in a single backward sweep at the cost of roughly one extra forward pass by caching each layer's pre-activation z^(l) and activation a^(l) and propagating the local error sensitivity δ^(l) = ∂L / ∂z^(l) from output to input:

Forward: z^(l) = a^(l−1)W^(l) + b^(l), a^(l) = σ(z^(l)) | Output Error (Sigmoid + BCE): δ^(L) = ∂L/∂z^(L) = ŷ − y | Backward: δ^(l) = (δ^(l+1)(W^(l+1))^T) ⊙ σ'(z^(l)), ∇_W^(l)L = (a^(l−1))^Tδ^(l), ∇_b^(l)L = δ^(l)

Interactive Trainer & Hyperparameters

2D Manifold Dataset

Hidden Layer Architecture

Activation Function σ(z)

Optimizer

Learning Rate η: 0.050

Epoch

0

BCE Loss

—

Accuracy

—

Per-Layer Gradient Norms ‖∇_WL‖₂ (Backprop Signal Flow)

Live Decision Boundary & Point Probe

Click anywhere on the canvas to run live forward inference & load the sample into the Network Graph below.

Interactive Network Node Graph & Step-by-Step Execution Debugger

Inspect every neuron node (z, a, b, δ) and weighted synapse (W, ∂L/∂W, ΔW) in the active network above. Step forward and backward through each micro-stage of Forward Inference, BCE Loss Evaluation, Reverse-Mode Backpropagation, and Optimizer Weight Updates—and click any node in the graph to inspect its exact arithmetic.

Graph Canvas Scale

Positive weight / signal (W > 0) Negative weight / signal (W < 0) Active Forward Wave (a·W) Active Backprop Wave (δ → ∇W) Weight Update (ΔW)

Neuron & Synapse Inspector (Click any node in graph above)

Neural Network Chapters & Foundations (MPA)

A chapter-by-chapter guide from one neuron to modern LLM frontiers. Each chapter is a true, standalone static multi-page guide with native MathML formulas and live interactive sandboxes.

Three Ideas Carry the Whole Subject

1. A neuron is a weighted sum followed by a squash: Multiply each input by a weight, add them up with a bias, and pass through activation function:

y = f (\sum_{i = 1}^{n} w_{i} x_{i} + b)

Explore the perceptron chapter.

2. Learning means walking downhill on a loss: Score error with loss $L$ and step against its gradient:

w \leftarrow w - η \frac{\partial L}{\partial w}

Explore the gradient descent chapter.

3. Backpropagation finds every slope at once: Chain rule evaluated in reverse mode:

\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a} \cdot \frac{\partial a}{\partial z} \cdot \frac{\partial z}{\partial w}

Explore the backpropagation chapter.

Visual Convolution, Softmax & Self-Attention Playgrounds

Explore the CNN chapter.

Explore the Softmax chapter.

Explore the Transformer chapter.

2. Convolutional Neural Networks: Spatial Priors, Weight Sharing & Pooling (1989–2012)

Why not feed images directly into a dense MLP? Consider a modest 256 × 256 RGB image (196,608 inputs). A single dense layer with 1,000 hidden units requires 196.6 million weights—and worse, if an object shifts by just 2 pixels, every input lands on a completely different weight, forcing the network to re-learn edge detectors at every (x, y) coordinate from scratch.

Convolutional Neural Networks—pioneered by Fukushima's Neocognitron (1980), LeCun's LeNet-5 (1989/1998) for handwritten zip codes, and scaled to GPUs by Krizhevsky, Sutskever, and Hinton's AlexNet (2012)—encode two physical priors of natural signals directly into the architecture:

Locality & Sparse Connectivity: Early visual features (oriented edges, corners, strokes) depend only on a small local neighborhood k × k (here 3 × 3 pixels). Stacking convolutional layers grows the effective receptive field linearly with depth (two 3 × 3 layers see a 5 × 5 window; three see 7 × 7).
Spatial Weight Sharing (Translation Equivariance): The exact same 3 × 3 kernel tensor K^(f) (only 9 weights + 1 bias!) is slid across every row r and column c of the image. Shifting the input shifts the resulting feature map identically: f(T_Δr,Δc I) = T_Δr,Δc f(I).
Local Max-Pooling & Switch Routing: A 2 × 2 stride-2 MaxPool layer outputs the maximum activation in each 2 × 2 patch, granting local translation invariance. During backpropagation, MaxPool acts as a router: it records the argmax index during the forward pass and routes 100% of the upstream gradient ∂L/∂p exclusively to that winning pixel while zeroing the other three.

Forward: Z^(f)_r,c = b_f + ∑_u=0..2 ∑_v=0..2 I_{r+u, c+v} · K^(f)_u,v | Backward Filter Grad: ∂L/∂K^(f)_u,v = ∑_r,c I_{r+u, c+v} · δ^(f)_r,c | Input Grad: ∂L/∂I = δ ∗ rot_180°(K)

Draw or Load an 8×8 Glyph & Train Kernels

Click cells to toggle pixels, or load a preset pattern to inspect how each 3×3 filter responds.

CNN Epoch

0

CE Loss

—

Train Accuracy

—

Live Learned 3×3 Kernels, 6×6 Feature Maps & Class Probabilities

3. Recurrent Neural Networks, BPTT & Gated Memory (LSTM / GRU) (1986–1997)

Feedforward MLPs and CNNs assume fixed-size inputs with no persistent memory across time. To process sequences x₀, x₁, …, x_T−1 of arbitrary length (speech, text, time series, control loops), Jordan (1986) and Elman (1990) introduced Recurrent Neural Networks (RNNs), which maintain a hidden state vector h_t ∈ ℝ^D updated recurrently at every timestep: h_t = tanh(W_xh x_t + W_hh h_t−1 + b_h).

Why Vanilla RNNs Struggle with Long-Term Dependencies (Hochreiter, 1991; Bengio et al., 1994): When we unroll the recurrence across T timesteps and apply the chain rule—an algorithm known as Backpropagation Through Time (BPTT)—the sensitivity of the final loss L_T to an early hidden state h_k is a product of T − k Jacobians:

BPTT Jacobian Chain: ∂h_T / ∂h_k = ∏_j=k+1..T (∂h_j / ∂h_j−1) = ∏_j=k+1..T diag(1 − h_j²) · W_hh^T | GRU Additive Highway: h_t = (1 − z_t) ⊙ h_t−1 + z_t ⊙ h̃_t ⇒ ∂h_t/∂h_t−1 = diag(1 − z_t) + …

Vanishing vs. Exploding Gradients: Because diag(1 − h_j²) is bounded in [0, 1] and W_hh^T is multiplied repeatedly T − k times, if the largest eigenvalue (spectral radius) ρ(W_hh) < 1, the gradient norm ‖∂L_T/∂h₀‖ decays exponentially like ρ^T → 0—wiping out any learning signal from token x₀! Conversely, if ρ(W_hh) > 1, gradients grow exponentially like ρ^T → ∞ (requiring global gradient norm clipping).
The Gated Solution — LSTM (Hochreiter & Schmidhuber, 1997) & GRU (Cho et al., 2014): Instead of overwriting h_t through a multiplicative matrix + saturating tanh at every step, Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU) introduce a learnable sigmoid Update / Forget Gate z_t = σ(W_xzx_t + W_hzh_t−1 + b_z) ∈ (0, 1) that interpolates additively between the old state and the new candidate state: h_t = (1 − z_t) ⊙ h_t−1 + z_t ⊙ h̃_t. When the gate remains closed (z_t → 0) across distractor tokens, ∂h_t/∂h_t−1 ≈ I—forming a Constant Error Carousel that carries both the memory of x₀ forward and the error gradient ∂L_T/∂h₀ backward across dozens or hundreds of timesteps without attenuation!

Interactive RNN vs Gated GRU Temporal Memory Trainer

Recurrent Cell Architecture

Sequence Horizon Length T

Temporal Sequence Task

BPTT Epoch

0

Sequence BCE Loss

—

Recall Accuracy

—

t=0 / t=T−1 Grad Ratio

—

Unrolled Temporal State & Backpropagated Gradient Flow (t = 0 … T−1)

Compare Gated GRU vs Vanilla Elman RNN at T = 12: notice how the GRU's update gate z_t stays small on distractor steps so the backward gradient ‖∂L_T/∂h₀‖ reaches t = 0 intact, whereas Vanilla RNN gradients decay toward zero!

4. Very Deep Neural Networks: Residual Highways (ResNets), Normalization & Init (2015–Present)

By 2014, researchers discovered a counterintuitive barrier when stacking 20, 50, or 100+ layers: even with ReLU activations and careful initialization, plain deep networks had higher training error than shallower networks (the Degradation Problem). Three interlocking mechanisms govern whether signals and gradients survive across extreme depth:

1. Variance-Preserving Weight Initialization (Glorot/Xavier 2010 & Kaiming He 2015): For a linear layer z_j = ∑_i=1..D w_ij a_i with independent zero-mean weights of variance σ_w², the output variance is Var(z_j) = D · σ_w² · Var(a_i). If D · σ_w² ≠ 1, after L layers the signal variance scales as (D · σ_w²)^L—either vanishing to 0 (when weights are too small) or exploding to ∞ (when weights are too large). Xavier init sets σ_w = √(1/D) for linear/tanh layers; Kaiming He init sets σ_w = √(2/D) to compensate for ReLU/GELU zeroing half the distribution.
2. Layer Normalization & Pre-RMSNorm (Ba et al. 2016; Zhang & Sennrich 2019): Even with ideal initialization, parameter updates during training cause activation scales to drift across layers (internal covariate shift). RMSNorm rescales each layer's input vector to unit root-mean-square before the weight projection: RMSNorm(x) = (x / √(mean(x²) + ε)) ⊙ γ, ensuring every block receives well-conditioned O(1) inputs regardless of depth.
3. Residual Skip Highways (He, Zhang, Ren & Sun, 2015): Instead of forcing layer l to learn a full mapping x^(l) = F_l(x^(l−1)), a Residual Block learns an additive perturbation around an identity skip path: x^(l) = x^(l−1) + F_l(RMSNorm(x^(l−1))). Unrolling this recurrence to layer L reveals why 100+ layer ResNets and modern Pre-LN Transformers train effortlessly:

Forward Residual Stream: x^(L) = x^(l) + ∑_k=l..L−1 F_k(x^(k)) | Backward Jacobian Highway: ∂L / ∂x^(l) = (∂L / ∂x^(L)) · (I + ∂/∂x^(l) ∑_k=l..L−1 F_k(x^(k)))

Look at the + I (identity matrix) inside the backward Jacobian above: even if the intermediate weight matrices inside F_k have small singular values, the error gradient ∂L/∂x^(L) from the loss head is added directly to every earlier layer ∂L/∂x^(l) without being multiplied through a chain of L − l matrices!

Interactive 16-Layer Deep Residual & Normalization Lab

Network Depth L (Stacked Blocks)

Residual Skip Connection

Layer Normalization

Weight Initialization Scale σ_w

Step

0

BCE Loss

—

Accuracy

—

Layer 1 / Layer L Grad

—

Per-Layer Activation RMS & Gradient Norm Profile (Layer 1 → Layer L)

Switch Residual Skip Connection to Plain Deep Stack and Normalization to No Normalization at 16 Layers to watch Layer 1's gradient collapse to 0.0000!

5. Transformers & Scaled Dot-Product Self-Attention (2017–Present)

Even with Gated GRUs and LSTMs, recurrent networks suffer from two structural bottlenecks: (1) step t cannot be computed until step t − 1 finishes, preventing parallel GPU/TPU training across the sequence dimension T; and (2) all historical information from tokens 0 … t − 1 must be compressed into a single fixed-width vector h_t−1 ∈ ℝ^D.

In Attention Is All You Need (2017), Vaswani et al. eliminated recurrence entirely in favor of Multi-Head Self-Attention. Every token position i ∈ {0 … T−1} projects its representation x_i into three distinct subspace vectors for each head h:

Query q_i = x_i W_Q^(h): What feature or context is token i looking for?
Key k_j = x_j W_K^(h): What routing descriptor or address does token j advertise?
Value v_j = x_j W_V^(h): What payload content should token j transmit if attended to?

Why the 1 / √d_k Scaling Factor Is Critical: If the components of q_i and k_j have zero mean and unit variance, their dot product q_i · k_j = ∑_{m=1..d_k} q_im k_jm has mean 0 and variance d_k (standard deviation √d_k). For large head dimension d_k, unscaled dot products grow large in magnitude, pushing softmax into extreme one-hot saturation where the Softmax Jacobian ∂P_ij/∂S_ik = P_ij(δ_jk − P_ik) vanishes to zero! Dividing by √d_k restores unit variance prior to softmax.

Attention(Q, K, V) = softmax((Q K^T) / √d_k + M) V | Softmax Row Jacobian: ∂L / ∂S_i,j = P_i,j · (∂L/∂P_i,j − ∑_k P_i,k · ∂L/∂P_i,k)

Train a Micro-Transformer on Algorithmic Sequence Tasks

Sequence Algorithmic Task

Attention Mask M

Probe Input Sequence (4 Tokens from A–H)

Epoch

0

Token CE Loss

—

Token Accuracy

—

Live Multi-Head Attention Heatmaps (Head 1 & Head 2)

Notice how training on Reverse causes the attention matrix to light up along the anti-diagonal, whereas Copy concentrates weight on the main diagonal. Try switching the Mask to Causal (j ≤ i) on Reverse to see why position 0 cannot peek ahead at position 3!

6. Diffusion Models: Continuous Image DDPM & Discrete Masked Text Diffusion (2020–Present)

Directly mapping a single random latent vector z ~ N(0, I) to a complex high-dimensional data manifold (such as natural images or whole paragraphs of text) in a single pass forces a neural network to solve an intractable global coordination problem all at once: GANs suffer from adversarial mode collapse, while single-step VAEs average over modes and produce blurry outputs.

Diffusion Models (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho, Jain & Abbeel, 2020) decompose generation into a trajectory of easy, local denoising refinements. They pair a fixed, parameter-free forward corruption process q(x_1:T | x₀) that gradually destroys all structure in data x₀ over T steps with a learned reverse generative process p_θ(x_t−1 | x_t) that reconstructs the signal step by step.

Part I — Mathematical Anatomy of Continuous Gaussian Diffusion (DDPM & Score Matching)

Forward Markov Chain & the Closed-Form Marginal Shortcut: Given a clean data vector x₀ ∈ ℝ^D and a variance schedule β₁ < β₂ < … < β_T ∈ (0, 1), each forward step attenuates the signal by √(1 − β_t) and injects Gaussian noise of variance β_t: q(x_t | x_t−1) = N(x_t; √(1 − β_t) x_t−1, β_t I). Because the sum of independent Gaussians is still Gaussian, defining α_t = 1 − β_t and the cumulative signal retention product ᾱ_t = ∏_s=1..t α_s yields a remarkable closed-form shortcut—we can sample noisy state x_t at any arbitrary timestep t in O(1) without unrolling steps 1 … t − 1:
Closed-Form Forward Marginal: x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε, where ε ~ N(0, I), SNR(t) = ᾱ_t / (1 − ᾱ_t)
Variational Lower Bound (ELBO), Tweedie's Formula & Stein Score Equivalence: Although the true reverse transition q(x_t−1 | x_t) is intractable without knowing x₀, Bayes' rule shows that conditioned on x₀ it is an exact Gaussian q(x_t−1 | x_t, x₀) = N(x_t−1; μ̃_t(x_t, x₀), β̃_t I). Substituting our closed-form equation x₀ = (x_t − √(1 − ᾱ_t) ε) / √ᾱ_t into the KL divergence D_KL(q(x_t−1|x_t,x₀) ‖ p_θ(x_t−1|x_t)) collapses the entire variational bound into Ho et al.'s noise-prediction MSE objective:
L_simple(θ) = E_{t ~ Uniform(1..T), x₀, ε ~ N(0,I)} [ ‖ ε − ε_θ(√ᾱ_t x₀ + √(1 − ᾱ_t) ε, t) ‖₂² ]
By Tweedie's Formula, predicting the noise vector ε_θ(x_t, t) is mathematically identical (up to a scalar factor) to estimating the Stein Score Function—the gradient of the log probability density of the noisy data distribution: ∇_{x_t} log q(x_t) = − ε_θ(x_t, t) / √(1 − ᾱ_t). Every reverse step literally nudges x_t uphill along the log-density gradient toward high-probability data manifolds!
Reverse Sampling: Stochastic Langevin (DDPM) vs. Deterministic ODE (DDIM) & Classifier-Free Guidance (CFG): Starting from pure Gaussian noise x_T ~ N(0, I), the reverse sampler steps from t = T down to 1:
Reverse Step: x_t−1 = (1 / √α_t) · ( x_t − ((1 − α_t) / √(1 − ᾱ_t)) · ε_θ(x_t, t) ) + σ_t z, z ~ N(0, I)
Setting σ_t = √β_t yields stochastic ancestral DDPM sampling (overdamped Langevin dynamics); setting σ_t = 0 (Song et al.'s DDIM) converts the process into a deterministic Probability Flow ODE that can skip steps (e.g., 1000 → 20 steps). To condition generation on a prompt or class label c, Classifier-Free Guidance (Ho & Salimans, 2022) linearly extrapolates away from the unconditional score: ε̃_θ(x_t, t, c) = (1 + w) · ε_θ(x_t, t, c) − w · ε_θ(x_t, t, ∅).

Part II — Discrete Masked Text Diffusion (D3PM, SEDD & MDLM / Mercury)

Why can't we apply standard Gaussian DDPM directly to text tokens? Because token IDs ("neural" = 412, "gradients" = 891) are discrete categorical indices—adding 0.15 · ε to an integer index has no geometry. Instead, Discrete Diffusion Models (Austin et al., 2021 D3PM; Lou et al., 2024 SEDD; Sahoo et al., 2024 MDLM) define a Continuous-Time Markov Chain (CTMC) directly over the discrete vocabulary V ∪ {[MASK]} using an absorbing state [MASK]:

Forward Masking Schedule: At noise level t ∈ [0, 1] with survival probability α_t = 1 − t, each token x₀⁽ⁱ⁾ independently remains intact with probability α_t or transitions to the absorbing [MASK] state with probability 1 − α_t: q(z_t⁽ⁱ⁾ | x₀⁽ⁱ⁾) = α_t δ_{x₀⁽ⁱ⁾} + (1 − α_t) δ_[MASK]. At t = 1, the entire sequence is [MASK, MASK, …, MASK].
SUBS Parameterization & Bidirectional Denoising Objective: Unlike an autoregressive GPT (which must use a causal triangular mask j ≤ i and can only condition on left context), a Masked Diffusion Language Model uses a full bidirectional Transformer encoder. Under the SUBS (Substitution) parameterization, any already-unmasked token at position i is copied through with probability 1 (carry-over unmasked tokens), and the cross-entropy loss is evaluated exclusively on currently masked positions weighted by the continuous-time NELBO rate −α'_t / (1 − α_t) = 1 / t:
L_MDLM(θ) = E_{t ~ U(0,1), z_t ~ q(z_t|x₀)} [ (1 / t) · ∑_{i : z_t⁽ⁱ⁾ = [MASK]} − log p_θ(x₀⁽ⁱ⁾ | z_t) ]
Parallel Confidence-Guided Unmasking (Why Diffusion LMs Generate in O(K) Steps Instead of O(L)): To generate a sequence of length L, an autoregressive LLM must execute L sequential forward passes (token 1 → token 2 → … → token L). A Masked Diffusion LM starts with L [MASK] tokens, predicts categorical distributions p_θ(x₀⁽ⁱ⁾ | z_t) across all masked positions in a single bidirectional pass, and unmasks the top-k positions with the highest confidence max_v p_θ(v | z_t) in parallel—crystallizing islands of high-certainty words across the whole sentence simultaneously in S ≪ L steps!

Dimension	Continuous Gaussian Diffusion (DDPM / DiT)	Discrete Masked Diffusion (MDLM / SEDD)	Autoregressive LM (GPT Decoder)
State Space	Continuous vectors `x_t ∈ ℝ^D`	Discrete tokens `z_t ∈ (V ∪ {[MASK]})^L`	Discrete prefix `x_1:i−1 ∈ Vⁱ⁻¹`
Forward Corruption	Additive Gaussian noise `√ᾱ_tx₀ + √(1−ᾱ_t)ε`	Independent token masking w.p. `γ(t) = 1 − α_t`	Causal shift (next-token teacher forcing)
Attention Mask	Bidirectional / 2D spatial self-attention	Bidirectional full-sequence attention	Strict lower-triangular causal mask (`j ≤ i`)
Training Loss	Noise/Score MSE `‖ε − ε_θ(x_t, t)‖²`	Time-weighted masked CE `(1/t) ∑ −log p_θ(x₀\|z_t)`	Next-token CE `∑ −log p_θ(x_i \| x_<i)`
Inference Passes	`S` parallel denoising steps (e.g. 16–50)	`S` parallel unmasking steps (`S ≪ L`)	Strictly `L` sequential steps (1 per token)

Part A: Continuous 2D Pixel Sprite DDPM

Target 6×6 Pixel Sprite x₀

Inspect Reverse Step t: 0 (Denoised x₀)

Train Step

0

Noise MSE Loss

—

Part B: Discrete Masked Text Diffusion (Parallel Unmasking)

Target Sequence Seed

Denoiser Step

0

Masked CE Loss

—

7. Decision Models & Single-Pass Pointer Readout Heads (2025–Present)

When software systems use an LLM to route a ticket, select a tool, or score risk, autoregressive token generation is both wasteful and brittle: the model must emit dozens of syntax tokens ({"choice": "..."}), token probabilities fragment across surface synonyms, and raw softmax probabilities are notoriously overconfident on out-of-distribution inputs.

As detailed in our Decision Models report (Jev, Kev, and Laya), a Decision Pointer Head packs the state and candidate options into a structured sequence with reserved option delimiters (‹0› … ‹K−1› or pre-mask slots), runs a single forward pass, and computes the inner product between the projected state representation h_state and each option delimiter's hidden vector h_‹k›. Crucially, it decouples Choice (which option wins under temperature-calibrated softmax) from the Act / Review Gate (which uses margin and entropy features of the raw distribution to abstain when evidence is ambiguous).

z_k = (W_q h_state)^T (W_k h_‹k›) / (√d · T), p = softmax(z), Confidence = 0.65 · p₍₁₎ + 0.35 · (p₍₁₎ − p₍₂₎), Gate ∈ {act, review}

Single-Pass Option Pointer & Calibration Controls

Incident Routing Scenario

Calibration Temperature T: 1.00 Abstention Gate Threshold τ: 0.55

Calibrated Option Distribution & Gate Decision

8. Extensible Plug-and-Play Neural Architecture Workbench

Modern foundation models are modular compositions of reusable primitives. Click any block below to append it to your live neural pipeline, inspect how tensor shapes, parameter counts, FLOPs, and forward/backward norms evolve across layers, or dynamically register a brand-new architecture block (such as Selective State-Space / Mamba SSM or Sparse Mixture-of-Experts Router) via registerBlockType().

Add Building Block to Pipeline

Pipeline Depth

0 blocks

Total Parameters

0

FLOPs / Sequence

0

Input Gradient Norm

—

9. First-Principles Reference: Complete Architecture Matrix & Hardware Execution Guide

Historical Evolution & Inductive Bias Matrix (1958–Present)

Era & Family	Core Primitive	Inductive Bias	Compute & Memory Scaling	What It Solved & Where It Breaks
Perceptron (1958)	`y = step(w·x + b)`	Single linear hyperplane	`O(D)`	Solved linearly separable classification; fails on XOR and non-convex manifolds.
Feedforward MLP + Backprop (1986)	`A = σ(XW + b)`	Universal function approximation over flat vectors	`O(L · D²)`	Learns arbitrary non-linear features via chain rule; lacks spatial or sequential invariance.
Convolutional Nets (1989–2012)	`(I ∗ K)_r,c` + Pooling	2D locality & translation equivariance	`O(H · W · C_in · C_out · k²)`	Dramatically reduces parameters on grids/images; fixed local receptive field per layer.
Recurrent Nets & LSTM/GRU (1986–1997)	`h_t = (1−z_t)⊙h_t−1 + z_t⊙h̃_t`	Markovian temporal state & additive memory gate	`O(T · D²)` sequential steps	Constant Error Carousel prevents temporal vanishing gradients; cannot parallelize across time `T`.
Deep Residual Nets / ResNets (2015)	`x^(l) = x^(l−1) + F_l(Norm(x^(l−1)))`	Identity skip highway + variance standardization	`O(L · D²)` for `L > 100`	Additive `+I` Jacobian prevents gradient shattering at extreme depth.
Transformers (2017)	`softmax(QK^T/√d_k)V`	Dynamic all-to-all token routing	`O(T² · D + T · D²)`	Parallelizes sequence training & eliminates recurrent bottlenecks; quadratic `O(T²)` KV-cache growth.
Continuous Image DDPM (2020)	`ε_θ(√ᾱ_tx₀ + √(1−ᾱ_t)ε, t)`	Iterative score/noise field reversal	`O(S · FLOPs_backbone)` over `S` steps	Stable high-dimensional distribution synthesis without adversarial collapse; requires multiple denoising steps.
Discrete Masked Text Diffusion (2024)	`p_θ(x₀ \| z_t)` over `[MASK]`	Bidirectional parallel token refinement	`O(S · T² · D)` with `S ≪ T`	Generates and edits tokens in parallel by confidence instead of strict left-to-right decoding.
Decision Pointer Heads (2025)	`(W_qh_last)^T(W_kh_‹k›)/T`	Bounded option scoring + calibrated abstention	Single forward pass `O(T · D²)`	Eliminates multi-token JSON generation for classification/routing; cannot invent free-form prose.
Selective SSM / Mamba & Sparse MoE	`h_t = Ā(x_t)h_t−1 + B̄(x_t)x_t`	Input-dependent linear recurrence / conditional capacity	`O(T · D)` state scan; active parameter sparsity	Linear-time long context and massive parameter capacity at fixed FLOPs per token.

Structuring Neural Network Kernels: Raw JS vs Raw WASM vs WebGPU

Runtime Target	Memory Model	Parallelism & Dispatch	Sweet Spot
Raw JS (`Float32Array`)	TypedArray views on V8/JSC heap; zero copy to DOM/Canvas	Single thread (or Web Workers); JIT loop unrolling when loop order is `i → k → j`	Micro-models (`D ≤ 64`), rapid prototyping, zero marshaling overhead
Raw WebAssembly (`wasm32`)	Contiguous linear arena `WebAssembly.Memory` with deterministic 4-byte `f32.load`/`f32.store`	Predictable AOT/JIT machine code, 128-bit `v128` SIMD (`f32x4.mul`), zero GC pauses	CPU inference & training from `D = 32` to `D = 1024` across all browsers
WebGPU (`WGSL @compute`)	Explicit device `storage` buffers + fast on-chip `var<workgroup>` shared SRAM tiles	Thousands of concurrent GPU threads grouped into `@workgroup_size(8, 8)` tiles synchronized via `workgroupBarrier()`	Large batch/matrix sizes (`M, K, N ≥ 256`) where compute FLOPs amortize PCIe/queue submission latency

How the In-Memory Raw WebAssembly (`wasm32`) GEMM Kernel Works

In site/neural-networks/engine.js, buildWasmGemmBytes() constructs a valid 183-byte WebAssembly binary module directly in a Uint8Array—with zero external .wasm files or C/Rust compilers. Both JavaScript and WebAssembly share a contiguous WebAssembly.Memory buffer viewed as a Float32Array:

;; Disassembly of the in-memory compiled gemm_f32(aPtr, bPtr, cPtr, M, K, N) kernel
(func $gemm_f32 (export "gemm_f32")
  (param $aPtr i32) (param $bPtr i32) (param $cPtr i32)
  (param $M i32) (param $K i32) (param $N i32)
  (local $i i32) (local $j i32) (local $k i32) (local $sum f32)
  ;; Triple nested loop over M × N × K reading/writing linear memory in 4-byte f32 strides:
  ;; sum += f32.load(aPtr + ((i * K + k) << 2)) * f32.load(bPtr + ((k * N + j) << 2))
  ;; f32.store(cPtr + ((i * N + j) << 2), sum)
)

Reverse-Mode Autograd & Decoupled AdamW Update

Every neural network training loop follows four exact steps: (1) forward pass caching intermediate pre-activations z^(l) and activations a^(l), (2) loss evaluation at the output head, (3) reverse-mode chain-rule propagation of δ^(l) = ∂L / ∂z^(l), and (4) parameter update:

// Output layer gradient for Sigmoid + Binary Cross-Entropy simplifies to (p - y) / N
for (let i = 0; i < N; i++) dZ[i] = (probs[i] - Y[i]) / N;

// Backpropagate through hidden layers l = L-1 ... 0
for (let l = layers.length - 1; l >= 0; l--) {
  // 1. Parameter gradients: dW = prevA^T · dZ, db = sum(dZ)
  // 2. Propagate error to previous layer: dPrevZ = (dZ · W^T) * activationGrad(prevZ)
  // 3. Decoupled AdamW update:
  m = beta1 * m + (1 - beta1) * g;
  v = beta2 * v + (1 - beta2) * (g * g);
  w = w * (1 - lr * weightDecay) - lr * (mHat / (Math.sqrt(vHat) + 1e-8));
}

Extending the Workbench with `registerBlockType()`

You can register new neural network paradigms into BLOCK_REGISTRY from the browser console or module code at any time:

import { registerBlockType } from "./engine.js";

registerBlockType({
  type: "griffin_rg_lru",
  label: "Real-Gated Linear Recurrent Unit (RG-LRU)",
  category: "recurrent",
  description: "Diagonal gated linear recurrence with hardware-efficient scan.",
  computeStats(shape) {
    return {
      outShape: { ...shape },
      params: 3 * shape.d * shape.d,
      flops: 6 * shape.t * shape.d * shape.d
    };
  }
});

Neural Networks From First Principles