This paper presents a methodology developed for a study to evaluate the state of the art of automated map generalization in commercial software without applying any customization. The objectives of this study are to learn more about generic and specific requirements for automated map generalization, to show possibilities and limitations of commercial generalization software, and to identify areas for further research. The methodology had to consider all types of heterogeneity to guarantee independent testing and evaluation of available generalization solutions. The paper presents the two main steps of the methodology. The first step is the analysis of map requirements for automated generalization, which consisted of sourcing representative test cases, defining map specifications in generalization constraints, harmonizing constraints across the test cases, and analyzing the types of constraints that were defined. The second step of the methodology is the evaluation of generalized outputs. In this step, three evaluation methods were integrated to balance between human and machine evaluation and to expose possible inconsistencies. In the discussion the applied methodology is evaluated and areas for further research are identified.