Back to Independent Research

Row-Tiered Mixed-Precision Quantisation for LLMs

Designed & implemented Variable Bit Floating-Point (vBFP) quantisation on Llama-3-8B.

View Code

The Problem

Uniform quantisation for LLMs leaves significant performance and efficiency on the table, failing to account for parameter sensitivity.

The Solution

Engineered a global utility-threshold allocator using a two-pass histogram algorithm to distribute fractional payload bits across 7B parameters. Implemented a custom INT4 CUDA GEMV kernel with on-the-fly unpacking.

Tech Stack

C++CUDAPyTorchPython

ML Performance

Model Architecture
Llama-3-8B
Inference Latency
15.5 tokens/s
Key Metrics
5.91
P P L
4.12
Avg Bits