BitByBit – Custom LLM Silicon Architecture

Architected and developed a ground-up, cycle-accurate custom System-on-Chip (SoC) in Verilog designed specifically to run LLMs (NanoGPT, Gemma 3) at the edge. The system bypasses traditional GPU bottlenecks ("Memory Wall") by implementing state-of-the-art AI research—such as hardware FlashAttention and Ternary quantization—directly into silicon logic. Paired with a robust Python-Verilog co-simulation verification pipeline and a 3D-accelerated Next.js telemetry dashboard.

1. Multiplier-Free Ternary Quantization: Custom ternary_mac_unit based on BitNet (b1.58) research, eliminating power-hungry 16-bit multipliers in favor of a 2-bit multiplexer datapath (+1, -1, 0), achieving an estimated ~10x energy reduction per operation. 2. Structured Sparsity Engine: Implemented 2:4 structured sparsity processing elements (sparse_pe.v) inspired by NVIDIA Ampere, dynamically skipping zero-weight multiplications using offline-encoded 4-bit masks to double MAC throughput and halve memory bandwidth. 3. Hardware FlashAttention & Softmax: Designed a tiled FlashAttention hardware sequencer (tiled_attention_ctrl.v) that processes O(N²) attention in localized tiles. Integrated a fused, streaming online softmax unit (online_softmax_unit.v) that normalizes scores without buffering to SRAM, reducing the memory footprint to just 32 bytes. 4. Virtualized KV Cache Memory Management: Developed a server-grade MMU for edge silicon (kv_page_table.v). Built a stack-based hardware page allocator to decouple logical tokens from physical memory, eliminating fragmentation and enabling infinite sliding-window (StreamingLLM) context lengths. 5. Robust Memory Subsystem & Interconnects: Architected an 8-bank scratchpad SRAM with SECDED ECC and bank-level clock gating. Designed AXI4-Lite command processors and AXI4-Master DMA engines for high-bandwidth weight streaming. 6. Python-Verilog Co-Simulation: Orchestrated an end-to-end automated verification pipeline using Icarus Verilog. Dynamically generated testbenches to validate hardware logic against bit-exact Python golden models. 7. Performance Telemetry & Web Dashboard: Built a premium dashboard using Next.js 14 and React Three Fiber with 3D hardware visualization and comparative analytics.