BDH Network – Bio-Distilled Hebbian Network

Engineered a biologically-inspired language model bridging the gap between Transformers and brain-like neural networks. By replacing traditional O(N²) Softmax attention and KV caches with O(N) Linear Attention and Multi-Scale Synaptic Memory, the model achieved a 24x improvement in 500-token memory retention while maintaining a highly efficient, sparse activation profile (~5% active neurons).

1. Linear Attention (O(N)): Replaces traditional O(N²) Softmax attention with linear attention mechanisms, drastically reducing memory usage. 2. Multi-Scale Synaptic Memory: Three parallel matrices running at different decay rates (Fast λ=0.95 for grammar, Medium λ=0.99 for paragraph context, Slow λ=0.995 for overarching theme). 3. Hebbian Learning: Synaptic state matrices updated via Hebbian rule: E_new = E_old × λ + K ⊗ V. Memory stays fixed-size regardless of sequence length. 4. Multiplicative Gating: Sigmoid valves that act as biological "valves" — useless words are multiplied by near zero, critical points open the gate wide. 5. Sparse ReLU-LowRank FFNs: Biologically plausible sparsity where only ~5% of neurons are active, vastly reducing computational overhead. 6. Knowledge Distillation: KL-Divergence pipeline transferring logits from teacher models (Qwen 2.5/3.5, Gemma 3) into efficient 10M–70M parameter BDH models. 7. Continuous Self-Improvement: Autonomous learning loop across web, books, and synthetic data with meta-cognitive confidence scoring and safety-gated rollbacks.