Blog

LLM inference has become increasingly challenging, with the release of 1+ Trillion Parameter models, occupying 100s of GBs in weights as parameters, and requiring 10s-100s of GPUs/ASICs to serve them efficiently. While traditional inference optimization techniques tend to focus on system-level algorithmic optimizations for decreasing latency (e.g Flash Attention) and increasing throughput (e.g. KV Cache, Paged Attention, Data, Tensor and Pipeline Parallelism). There are other approaches which improve model performance by simplifying the token generation step as part of the model itself. ...