CrowdManager - Combinatorial allocation and pricing of crowdsourcing tasks with time constraints

Abstract

Crowdsourcing markets like Amazon’s Mechanical Turk or Crowdflower are quickly growing in size and popularity. The allocation of workers and compensation approaches in these markets are, however, still very simple. In particular, given a set of tasks that need to be solved within a specific time constraint, no mechanism exists for the requestor to (a) find a suitable set of crowd workers that can solve all of the tasks within the time constraint, and (b) find the “right” price to pay these workers. In this paper, we provide a solution to this problem by introducing CrowdManager – a framework for the combinatorial allocation and pricing of crowdsourcing tasks under budget, completion time, and quality constraints. Our main contribution is a mechanism that allocates tasks to workers such that social welfare is maximized, while obeying the requestor’s time and quality constraints. Workers’ payments are computed using a VCG payment rule. Thus, the resulting mechanism is efficient, truthful, and individually rational. To support our approach we present simulation results that benchmark our mechanism against two baseline approaches employing fixed-priced mechanisms. The simulation results illustrate that our mechanism (i) significantly reduces the requestor’s costs in the majority of settings and (ii) finds solutions in many cases where the baseline approaches either fail or significantly overpay. Furthermore, we show that the allocation as well as VCG payments can be computed in a few seconds, even with hundreds of workers and thousands of tasks.