Measuring Mobile Broadband Performance

The FCC has been running the Measuring Broadband America program since 2011. They have released six well-received yearly reports so far, analyzing the data collected from wired broadband providers. In 2012, the FCC announced an analogous Measuring Mobile Broadband Performance effort and they have been collecting data since 2014. However, the FCC has not released any reports analyzing the mobile data. In this paper we investigate the technical and policy challenges confronting this and other efforts to collect and report on mobile broadband performance. At the core of the debate over mobile performance measurement are questions such as: what constitutes scientifically valid performance measurement for mobile broadband networks? How does one draw valid inferences about user experience, correctly attributing impairments when they exist to different actors or components in the system? Or, how can mobile performance measurement reporting protect privacy while still remaining informative? Answering these questions is more difficult for mobile than fixed broadband for multiple reasons that we document. We explore the current status of mobile broadband performance measurement, building on earlier work that focused on measuring the performance of fixed broadband services.2 Traditionally, drive and walk testing of mobile networks (conducted by companies such as RootMeterics and P3 Communications) have competed with crowd-sourced measurements (like Ookla's Speedtest app and the FCC's app). For each approach, we examine the measurement methodology, analysis procedures, and data availability. We find that while information describing the general structure of the measurement and reporting methodologies is typically available, important details regarding the measurement methodology and how the raw data is modified to produce the final reports is often missing. We dispute published claims3 that driving testing is "scientific," whereas other methodologies are suspect. We conclude that there is no single best approach and that the availability of a multiplicity of measurement platforms and approaches offers significant benefits. Although there is no single best mobile measurement approach for all questions, we explain why it is important to carefully match the test methodology and available data to the questions being asked. Finally, our analysis explores the implications for broadband policy of this evolving ecosystem for mobile broadband measurement.