How reliable are group informant ratings?

The reliability of food security rating, a variant of the more familiar wealth ranking procedure, was tested in a rural area of Western Honduras. Twenty workshop sessions were conducted in 13 different communities, with members of organized small farmers' groups attended by a large agricultural development project. Participants were all poor farmers with no more than 10 hectares of land. Participants, who generally knew each other well, were split into small sets and each set was asked to rate the food security status of all households in their organized group. Inter-rater agreement was assessed using the Kappa statistic, and related to other characteristics of the informants and workshop sessions, using multiple regression methods. Agreement was very poor (median value 6 = 0.29), especially for the category "Intermittenly Food Insecure," and was associated with time elapsed since training of the session moderators. Women were 49 percent more likely than men to classify a given family as food insecure (P 0.001). The authors put forward seven different hypotheses to explain the poor reliability of the rating method, which should be investigated in future research if the credibility of the method is to be reinforced.