Extracting the wisdom of a smaller crowd from dependent quantile judgments

The task of this article is to harness the wisdom of a crowd without calibration. We assume each expert to form predictions by linearly combining various information cues inspired by the lens model, and use Gaussian process to account for sampling and judgmental errors in quantile judgments. Without knowing the experts’ observed information cues, we develop a three-step estimation algorithm to factor quantile judgments into “variable profiles” (latent cues underlying each variable of interest) and “expert profiles” (each expert’s weights over these cues). We can inquire about expert similarity using their weights of the latent cues, which preserve the same clustering results as the actual weights of the observed cues up to a full-rank linear transform. We can then depict the diversity and dependency among experts explicitly and retain a subcrowd by picking delegates from each subgroup of experts based on the estimated weights. Simulation and case studies demonstrate that a subcrowd selected this way can represent the entire expert panel well.