Inference and performance.

Overview

Running a language model involves a series of practical tradeoffs. Responsiveness, resource use, and reliability all matter to the experience of using it.

I am interested in the fundamentals of inference performance and in how to reason about bottlenecks using reproducible, publicly discussable experiments.

Topics I explore

1. Latency and throughput as engineering concepts
2. Measuring responsiveness under changing workloads
3. Resource efficiency and the user experience
4. Clear measurement, repeatability, and communication