Recent work on searching the Semantic Web has yielded a wide range of approaches with respect to the style of input, the under- lying search mechanisms and the manner in which results are presented. Each approach has an impact upon the quality of the information re- trieved and the user’s experience of the search process. This highlights the need for formalised and consistent evaluation to benchmark the cov- erage, applicability and usability of existing tools and provide indications of future directions for advancement of the state-of-the-art. In this paper, we describe a comprehensive evaluation methodology which addresses both the underlying performance and the subjective usability of a tool. We present the key outcomes of a recently completed international evalu- ation campaign which adopted this approach and thus identify a number of new requirements for semantic search tools from both the perspective of the underlying technology as well as the user experience.