Inference engine that cuts LLM inference costs and latency by 20% using KV cache transfer between a small prefill model and a large decode model.
OneTriangle runs prefill on a smaller model and decode on a larger one, transferring per-layer key/value tensors projected into the target model's attention space. This eliminates the cost of large-model prefill on long inputs, which typically dominates inference bills. The system gates each model pair on quality and latency thresholds and falls back to standard prefill when a pair does not meet them. It supports open-weight models including Llama, Qwen, Mistral, Gemma, and DeepSeek, with OneTriangle managing the GPU infrastructure. OneTriangle is a product of OneTriangle.