CONCUR: A Framework for Continual Constrained and Unconstrained Routing

Peter Baile Chen1, Weiyue Li2, Dan Roth3, Michael Cafarella1, Samuel Madden1, Jacob Andreas1
1MIT, 2Harvard University, 3Oracle AI and the University of Pennsylvania
Paper Code

AI tasks differ in complexity and are best addressed with different computation strategies (e.g., combinations of models and decoding methods). Hence, an effective routing system that maps tasks to the appropriate strategies is crucial.
However, prior routing frameworks rely on a single model shared across all strategies and leverage a single input representation, which restricts both their generalizability to continual routing and the quality of their routing decisions. To overcome these limitations, we introduce CONCUR, a modular framework that incorporates multiple input representations to enable continual and more accurate routing.

CONCUR

  • CONCUR (1) adopts a modular design, training a separate predictor for each strategy so that extending to new strategies only requires training additional predictors without touching existing models; and (2) uses multiple input representations to better capture the complexity of the routing problem, rather than relying on a single representation.
  • Using the predicted accuracy and cost of applying each strategy to a given task, CONCUR performs both unconstrained and constrained routing by casting them as optimization problems, whose solutions yield the final routing decisions.

Performance of CONCUR

  • We evaluated CONCUR on both in-distribution and out-of-distribution tasks that require substantial knowledge and reasoning. Across continual and non-continual settings, CONCUR consistently outperforms the best single strategy and strong existing routing methods, achieving higher end-to-end accuracy and lower inference cost, while also reducing training cost in the continual setting.

BibTeX

@article{chen2025concur,
    title={CONCUR: A Framework for Continual Constrained and Unconstrained Routing},
    author={Chen, Peter Baile and Li, Weiyue and Roth, Dan and Cafarella, Michael and Madden, Samuel and Andreas, Jacob},
    journal={arXiv preprint arXiv:2512.09386},
    year={2025}
}