14× faster embeddings: how we rebuilt the ONNX path in Manticore
Summary
The article explains how Manticore Search achieved ~14x faster embeddings by rebuilding the ONNX path, moving from Candle-based inference to ONNX Runtime, and sharing a single ONNX session across concurrent requests. It details the concurrency model, tuning choices (like disabling intra_op_spinning), and the resulting throughput gains, plus guidance for bulk loading and migration options. Future work includes GPU support and Windows parity considerations.