SecureRouter speeds private AI inference with encrypted model routing

Summary

University of Central Florida researchers say SecureRouter routes encrypted queries to models of different sizes, cutting inference time versus fixed-model approaches while keeping accuracy close to the large-model baseline on most tasks.

Key points

• SecureRouter selects among models ranging from about 4.4 million to 340 million parameters while keeping queries and routing decisions encrypted.
• Against SecFormer, it reduced average inference time by 1.95× across five language-understanding tasks; speedups ranged from 1.83× to 2.19×.
• Across eight benchmark tasks, it was 1.53× faster on average than using a large model for every query.
• Accuracy stayed within a fraction of a percentage point of the large-model baseline on most tasks, but dropped more on a grammatical-analysis task.
• The routing component adds about 39 MB of memory, 4 seconds of latency, and 1.86 GB of network communication in the reported setup.
• The researchers say SecureRouter works with existing MPC frameworks and standard language-model architectures.

Products

Industries