Eliciting Thinking Hierarchy without Prior

A key challenge in crowdsourcing is that majority may make systematic mistakes (e.g. in the classic bat and ball problem: A bat and ball cost \$1.10. The bat costs \$1 more than the ball. How much does the ball cost?, the majority may answer \$.10). Prior work focuses on eliciting the best answer without prior even when the majority is wrong. Here we focus on eliciting the full thinking hierarchy without any prior. By asking a single open response question and eliciting both of each respondent's answer (e.g. \$.05) and guess(es) for other people's answers (e.g. \$.10), we construct an answer-guess matrix that records the number of people who report a specific answer-guess pair (e.g. 10 people answer \$.05 and guess \$.10). By ranking the answers to maximize the sum of the upper triangular area of the matrix, we obtain and visualize the hierarchy of the answers without any prior (e.g. we obtain "\$.05->\$.5->\$.10" hierarchy for bat and ball in our study). Our paradigm is highly practical since unlike prior work, the respondent does not need to perform multiple tasks nor report a distribution. Moreover, the requester does not need prior knowledge to design possible options for respondents. Our empirical studies not only demonstrate the superiority of our approach compared to plurality vote, but also show that more sophisticated people can reason about less sophisticated people's mind and the hierarchy can be approximately described by a directed acyclic graph.