Aggregating Partial Rankings from Neighbors: Methodology and Empirical Evidence

Many decisions require ordering alternatives: for example, the selection of top candidates for a competitive academic program or the selection of the poorest individuals for a cash transfer program. One common approach consists in aggregating orderings reported by different observers (e.g., committee or community members), but those orderings are typically partial: not all observers rank all applicants. We introduce a novel type of approach, based on pairwise rankings, to (i) aggregate partial orderings reported by multiple observers and (ii) construct confidence intervals for the resulting aggregate ordering. We identify, both theoretically and using simulations, the conditions under which a pairwise approach dominates rank averaging: when reporting error is low, reported orderings are partial, and observers rank alternatives that are close to each other in their true latent ordering. We introduce improvements to rank averaging and pairwise methods and illustrate them using several datasets. We find that, with partial reported orderings, Borda counts (i.e., simple rank averages) are dominated by the averaging of normalized ranks and should never be used in practice.