Request routing services undermine the reproducibility of open weight LLMs

Large language models (LLMs) are now a standard tool for classifying text in social science. To make these classifications reproducible, researchers are often advised to use open weight models and to set the sampling temperature to zero. We show that this advice is insufficient when models are accessed through a request-routing service, in which queries are sent to different hosting providers through a single API. Using a popular request-routing service, OpenRouter, we classify British manifesto sentences for ideology 100 times per sentence across eight models and 33 hosting providers, alongside five model configurations served on our own hardware, generating approximately 1m classification draws. When using the request-routing service, open weight models consistently fail to return deterministic outputs. Depending on the model, only 26% to 92% of sentences receive the same label in all repeated classifications. This variation is not solely due to different providers fielding queries: for five of the eight models, the majority of classification variance arises within providers. On our own hardware, the same task is perfectly reproducible at temperature=0. We conclude that reproducibility depends on the entire model hosting environment, and recommend that researchers report the fullest available metadata and raw responses when using LLMs.