Assessment Center Dimensions: Individual differences correlates and meta-analytic incremental validity

International Journal of Selection and Assessment Volume 17 Number 3 September 2009 Assessment Center Dimensions: Individual differences correlates and meta-analytic incremental validity Stephan Dilchert* and Deniz S. Ones** *Department of Management, Box B9-240, Zicklin School of Business, Baruch College, One Bernard Baruch Way, New York, NY 10010, USA. [email protected] **Department of Psychology, University of Minnesota, Minneapolis, MN, USA This study provides an investigation of the nomological net for the seven primary assessment center (AC) dimensions identified by Arthur, Day, McNelly, and Eden (Personnel Psychology, 56, 125–154, 2003). In doing so, the authors provide the first robust estimates of the relationships between all primary AC dimensions with cognitive ability and the Big 5 factors of personality. Additionally, intercorrelations between AC dimensions based on sample sizes much larger than those previously available in the metaanalytic literature are presented. Data were obtained from two large managerial samples (total N ¼ 4985). Primary data on AC dimensions, personality, and cognitive ability interrelationships were subsequently integrated with meta-analytic data to estimate incremental validity for optimally and unit-weighted AC dimension composites as well as overall AC ratings over psychometric tests of personality and cognitive ability. Results show that unit- and optimally weighted composites of construct-based AC dimensions add incremental validity over tests of personality and cognitive ability, while overall AC ratings (including those obtained using subjective methods of data combination) do not. 1. Introduction A ssessment centers (ACs) remain a popular tool for the evaluation of job applicants and employees, especially for managerial jobs. As an assessment This article won the 2009 James C. Johnson Student Paper Award from the International Personnel Assessment Council (IPAC). Portions of this research were contained in an unpublished technical report [Dilchert, S., & Ones, D. S. (2005, October) Using PDI Assessment Factors, Competencies, and Exercises. Minneapolis, MN: Author] and were presented at the annual conference of the Society for Industrial and Organizational Psychology, New York City, April 2007. The authors would like to thank Robert Lewis and Personnel Decisions International for making the two primary data samples available. method, ACs can be designed to measure a multitude of individual differences characteristics (e.g., interpersonal skills, communication skills, personality, cognitive ability). A variety of tools or exercises can be employed to measure these characteristics as part of an AC (e.g., simulations, interviews, in-baskets). Because of the complexities involved in assessing different personal characteristics with a variety of measurement techniques, ACs remain a fruitful field for applied research. The criterion-related and construct validity of AC ratings has been investigated in many primary studies as well as several meta-analyses. By now there is little doubt that ACs are useful tools for performance prediction. Gaugler, Rosenthal, Thornton, and Bentson (1987) reported a meta-analytic operational validity of & 2009 Blackwell Publishing Ltd., 9600 Garsington Road, Oxford, OX4 2DQ, UK and 350 Main St., Malden, MA 02148, USA AC Dimensions: Nomological Net and Incremental Validity r ¼ .36 for overall AC ratings predicting job performance (corrected for criterion unreliability and range restriction). Two meta-analytic updates of the literature published since 1987 (Hardison & Sackett, 2004; Hermelin, Lievens, & Robertson, 2007) report operational validity estimates of r ¼ .28 (corrected for unreliability in the criterion only). These estimates of operational validity for overall AC ratings are commonly referred to when ACs are compared with other methods and predictors to estimate their utility in personnel staffing (e.g., see Schmidt & Hunter, 1998). The issue of AC validity, however, is somewhat more complex. Rather than focusing on validity estimates for overall scores obtained using the AC method, researchers and practitioners may benefit from examining the validity of the constructs underlying such scores. A construct-based approach provides the opportunity to evaluate the true utility of ACs in applied settings by estimating their incremental validity over other commonly used predictors. As this study will show, a more differentiated validation approach based on AC dimensions, rather than overall AC ratings, will yield more promising estimates of operational and especially incremental validity by taking into account different predictor combinations. Recently, Arthur, Day, McNelly, and Edens (2003) took the first step in this direction by conducting a meta-analytic investigation of the primary dimensions underlying AC ratings. Arthur and colleagues argued that estimates of operational validity of overall AC ratings cannot be meaningfully compared with validities of predictors such as personality traits, for example, as this essentially involves comparing an aggregate of different constructs (the overall score across several AC exercises and dimensions) to single constructs (e.g., conscientiousness). Furthermore, comparisons of this sort are comparisons of a method (the AC) with constructs or traits (e.g., cognitive ability, personality), rendering investigations of relative predictive value even less fruitful (Arthur & Villado, 2008). In an attempt to remedy this problem, Arthur and colleagues established a taxonomy of seven primary AC dimensions from a list of 168 lower-order constructs typically measured in ACs (see Appendix Table A1 for definitions of these seven dimensions). The authors provided meta-analytic estimates of the interrelationships among the primary AC dimensions, and showed that they are reasonably related to one another (sample size weighted mean r ¼.52). More importantly, they also investigated the criterion-related validity for six of these AC dimensions in predicting job performance. Their results demonstrate that the higher-order dimensions commonly assessed across various AC exercises are of different value when predicting overall job performance. However, in order to determine the usefulness of AC scores in predicting valued behaviors and & 2009 Blackwell Publishing Ltd. 255 outcomes, we must not only assess their criterionrelated validity but also their potential overlap with more readily available (and cheaper) assessment tools such as psychometric tests of personality and cognitive ability. Such investigations will result in estimates of incremental validity that AC scores can offer over other tools in the prediction of performance and other valued outcomes. Data that speak to this matter are available for overall AC ratings. Collins et al. (2003) meta-analytically examined the nomological net of overall AC ratings that represent aggregates of several constructs. Collins and colleagues’ meta-analysis provides information on the overlap of overall AC ratings with tests of cognitive ability (r ¼ :43) and four of the Big 5 dimensions of personality (agreeableness, extraversion, emotional stability, and openness; r ¼ :12, .36, .26, and .18, respectively). Unfortunately, the quantitative summary by Collins and colleagues does not speak to the overlap of construct-based AC dimensions with these individual differences variables. Similarly, an earlier meta-analytic review published in German (Scholz & Schuler, 1993) only provided estimates for the relationships of cognitive ability and the Big 5 with overall AC ratings. Equivalent investigations for the construct-based primary dimensions identified by Arthur et al. (2003) are lacking. Arthur et al.’s (2003, table 2) definitions of the seven ‘meta-dimensions,’ as well as an examination of the lower-order dimensions that contributed to each of these meta-dimensions, provide an indication of which construct-based AC dimensions can be expected to overlap at least partially with individual differences traits. The most obvious associations can be drawn between AC scores on ‘problem solving’ and cognitive ability. According to Arthur and colleagues, problem solving is the ability to gather information, to effectively analyze data, and to generate viable ideas and solutions to problems. These skills and abilities are at the heart of general mental ability as operationalized by standardized tests (Carroll, 1993), and we would thus expect a moderate to strong relationship between AC problemsolving dimension and cognitive ability test scores. Based on the definition of the six remaining AC dimensions we would not expect much overlap with cognitive ability tests, and thus postulate potential for these AC scores to increment validity of ability measures. However, we stress that cumulative or largescale evidence regarding these relationships is not available in the published literature, and thus their incremental validity is unknown. We would also expect that many of the non-cognitive AC dimensions display moderate to strong associations with personality traits as operationalized by standardized tests. The AC dimensions ‘influencing others’ (conceptually related to leadership skills) and ‘drive’ (which captures aspects of ambition) should both relate International Journal of Selection and Assessment Volume 17 Number 3 September 2009 256 to extraversion, while the latter should also carry some conscientiousness variance, incorporating both achievement and persistence (Arthur et al., 2003). Other AC dimensions that are easily linked to personality traits on a conceptual level are ‘organizing and planning’ (which bears conceptual similarity to facets of conscientiousness, especially order) and ‘tolerance for stress’ (which falls squarely into the domain of emotional stability). Yet again, large-scale evidence for the overlap between these primary AC dimensions and personality traits (whether measured by standardized tests or other methods such as interviews or observer ratings) is lacking. Based on data presented by Schmidt and Hunter (1998), we know that the potential for incremental validity of overall AC scores over tests of general mental ability is small (.02). Similarly, the meta-analytic estimates provided by Collins et al. (2003) can be used to determine that the incremental validity of overall AC ratings over psychometric tests of personality and cognitive ability is negligible. However, the lack of correlational evidence for AC dimensions hinders parallel investigations for construct-based AC scores. If correlations between the primary AC dimensions identified by Arthur et al. (2003) and personality and cognitive ability tests were available, the incremental validity of optimally as well as unit-weighted AC score composites over such tests could be established by multiple regression procedures using the meta-analytic operational validity estimates for all predictors. Additionally, knowledge of these relationships would allow organizations to supplement existing predictor batteries with only a few AC dimensions in a targeted manner, in order to maximize overall validity efficiently. 1.1. The present study The goal of this study is to further AC research by examining the nomological net of the seven primary AC dimensions identified by Arthur et al. (2003). First, we provide the first large-scale investigation of the overlap between AC dimensions with individual differences traits (cognitive ability and personality) in two independent samples (total N ¼ 4985). The sample sizes available in our primary data exceed those of prior investigations of overall AC ratings, even of metaanalytic ones (e.g., Collins et al., 2003; Scholz and Schuler, 1993). Second, we integrate the findings from our primary data with meta-analytic predictive validity estimates for ACs, personality, cognitive ability, as well as meta-analytic data on the interrelationships of these predictors. The resulting matrix of meta-analytic and large sample primary data provides the best estimates of the relationships between AC dimensions, overall AC ratings, the Big 5 personality dimensions, cognitive International Journal of Selection and Assessment Volume 17 Number 3 September 2009 Stephan Dilchert and Deniz S. Ones ability, and job performance. Subsequently, this matrix of intercorrelations is used to estimate the incremental validity of various combinations of AC scores (overall AC ratings, single dimensions, as well as unit- and optimally weighted dimension composites) over tests of personality and cognitive ability (and vice versa). Thus, the present research addresses four previously unanswered questions: (1) What is the relationship between the seven meaningful primary AC dimensions identified by Arthur et al. (2003) and the Big 5 personality factors and cognitive ability? (2) What is the incremental validity that can be expected from AC ratings when these primary dimensions are combined (using both optimal and unit-weights) over tests of cognitive ability and personality? (3) Can useful levels of incremental validity be obtained by adding only one AC dimension to tests of personality and cognitive ability? (4) What is the incremental validity that tests of personality and cognitive ability in turn add over AC dimension composites? 2. Method The analytic approach of this study was twofold: first, we strove to provide the best estimate of incremental validity that ACs offer over tests of personality and cognitive ability, based on meta-analytic estimates of operational validities and predictor relationships where possible. In this investigation, only the previously unknown intercorrelations between AC dimensions and tests of personality and cognitive ability were estimated from primary data. The second goal was to provide a parallel investigation using only primary data on predictor interrelationships, in order to provide a check on the theoretically derived results as they would occur in a real-world assessment setting. 2.1. Meta-analytic data In order to estimate the incremental validity of one predictor over another in predicting a given criterion, the criterion-related validity of each predictor as well as the interrelationship between the predictors must be known. Thus, to inform computations of criterionrelated and incremental validity of ACs, we strove to obtain the best validity estimates for AC ratings, the Big 5 personality factors, and cognitive ability, as well as the best estimates of all predictor interrelationships. For this purpose we consulted the most recent and comprehensive meta-analytic summaries in each research domain. Next, we obtained sample size weighted mean correlations between all predictors (not corrected for unreliability or range restriction) from these meta- & 2009 Blackwell Publishing Ltd. AC Dimensions: Nomological Net and Incremental Validity 257 analyses. Where only corrected estimates were available we obtained the respective artifact distributions and attenuated true validity estimates appropriately.1 We also obtained estimates of each predictor’s criterion-related validity for predicting job performance. For these estimates, operational validities (corrected for unreliability in the criterion and range restriction where appropriate) were obtained where possible. In order to most accurately estimate the operational validity of a predictor combination in applied settings, it is necessary to use uncorrected estimates of predictor intercorrelations, but estimates of criterion-related validity that are corrected for range restriction and unreliability in the criterion measures (Schmidt & Hunter, 1998). The operational validity of predictor combinations with more than two predictors can then be calculated using multiple regression analyses on the full correlation matrix. In many cases multiple meta-analytic estimates were available. In selecting those that would contribute to our analyses, we took care to match predictor and criterion scales to the current investigation. Since ACs are mostly employed for medium to high complexity jobs (e.g., to assess managerial applicants), we chose estimates that most closely matched the predictor and criterion relationships of interest. For example, when meta-analytic estimates were reported by sample type, we chose those values that were obtained in managerial samples. Similarly, when values for different criteria were reported, we chose those estimating the predictive validity for managerial performance or in high complexity jobs (e.g., data from Hough, Ones, & Viswesvaran, 1998, for managerial samples, rather than other meta-analytic values available across jobs; or data on high-complexity jobs from Hunter, 1980, rather than results across job complexity levels). Appendix Table A2 provides a detailed description of all sources, as well as the corrections that were applied and the respective artifact distributions used. lytic values, in order to provide a comparison using data as obtained from an operational AC. Data were gathered from AC evaluations of mid-level managers and top-executives conducted for employment purposes2 over the course of 4 years (2000– 2004). Two different ACs were used to evaluate managers depending on their current managerial level or that of the job they were being considered for. The primary dimensions for which scores were extracted from the two ACs were identical for the two samples and corresponded to those identified by Arthur et al. (2003, see Appendix Table A1). 2.2. Primary data 2.2.2. Sample 2: Top-level managers Managers that underwent the top-level managerial AC (N ¼ 1923) were primarily from mid-level to top-executive management, had an average age of 44.3 years (SD ¼ 6.4), and were mostly male (80.4%). Ethnicity data were available for over 78% of these individuals, 92.6% of whom indicated White, 3.0% Black, 1.9% Hispanic, 1.7% Asian, .03% American Indian, and .5% ‘Other.’ As expected, individuals in this sample on average had an even higher level of education than those in Sample 1, with 42.3% having obtained a Bachelor’s degree, and 48.7% a Master’s or doctorate/ professional degree. The composition of the sample was similar to Sample 1 with respect to industries represented, although individuals in this sample tended In addition to meta-analytic estimates, primary data were employed for two purposes. First, as described earlier, the relationships between Arthur et al.’s (2003) primary AC dimensions and tests of personality and cognitive ability have not been investigated to date. This study constitutes the first such investigation by providing estimates obtained from two large primary samples for AC dimensions/personality/cognitive ability intercorrelations. Additionally, data on the interrelationships between personality variables and cognitive ability test scores as well as among AC dimensions were obtained from our two samples. These were used to conduct parallel investigations to those informed by meta-ana- & 2009 Blackwell Publishing Ltd. 2.2.1. Sample 1: Mid-level managers Managers that underwent the mid-level managerial AC (N ¼ 3062) mostly came from first-line or mid-level management, and were of an average age of 41.3 years (SD ¼ 7.0) and predominantly male (75.6%). Data on ethnic group membership were available for over 75% of these individuals, 90.2% of whom indicated White, 3.8% Black, 3.5% Hispanic, 1.8% Asian, .02% American Indian, and .5% ‘Other’ as their ethnic background. Individuals in this sample were well educated, with 48.1% having obtained a Bachelor’s degree, and 36.5% a Master’s or doctorate/professional degree. Assessees came from over 30 diverse industries (including food processing, electronics manufacturing, wholesale trade, banking and finance, health care, agriculture, construction, government, and transportation), and mostly from large organizations (over 80% from organizations with more than 1000 members, median ¼ 11,000). Managers’ work experience at the time of assessment was 19.6 years on average (SD ¼ 7.6), in which individuals on average worked for over three employers and had managerial responsibility for a duration of 11.8 years (SD ¼ 7.5). The number of employees managed by the individuals in this sample was as high as 7500, with a median of 18 (median number of direct reports ¼ 6). Individuals received an average annual salary of over US$147,000. International Journal of Selection and Assessment Volume 17 Number 3 September 2009 258 to come from larger organizations (median ¼ 20,000 members). The average number of years of work experience at the time of assessment was 22.5 (SD ¼ 7.1), with an average of three employers and managerial responsibility of 16.6 years (SD ¼ 7.0). The number of employees managed by the predominantly top-level managers in this sample was as high as 35,000, with a median of 55. Individuals in this sample received an average annual salary of more than US$309,000, with over 10% of managers earning US$500,000 or more. 2.3. Measures used in primary data collection 2.3.1. ACs Mid-level and top-level managers were evaluated in two different ACs that assessed the same seven higherorder dimensions. The two ACs were designed based on solid scientific principles and current research evidence, and conducted by a consulting company with extensive experience in assessment for selection and development purposes (serving 75 of the Fortune 100 companies). Both ACs used background interviews, an in-basket exercise, and three role-play exercises in the form of a direct report meeting, a strategy presentation, and a task force meeting. Depending on the exercise, individuals were assessed by one or multiple trained and experienced assessors. Assessors evaluated managers’ performance on a number of lower-order competencies within each exercise. Ratings were provided on very detailed behaviorally anchored rating scales. Across exercises in each AC, individual managers were rated on several hundred distinct behaviors that were combined to derive competency scores. Subsequently, AC primary dimension scores were obtained by summing relevant competency scores across AC exercises. The resulting seven primary dimensions were communication, consideration/awareness of others, drive, influencing others, organizing and planning, problem solving, and tolerance for stress/ uncertainty. 2.3.2. Cognitive ability Cognitive ability was assessed using three different psychometric tests: the Watson–Glaser Critical Thinking Appraisal (Watson & Glaser, 1980), the Wesman Personnel Classification Test (Wesman, 1965), and the Raven’s Progressive Matrices (Raven, Raven, & Court, 1998). All three inventories are widely used tests of cognitive ability that see heavy use in personnel selection settings, and are supported by extensive evidence demonstrating their construct validity as well as reliability. Individuals’ scores were transformed to standardized scores based on the normative information available for each measure, and were subsequently International Journal of Selection and Assessment Volume 17 Number 3 September 2009 Stephan Dilchert and Deniz S. Ones combined to obtain an overall score as an indicator of general mental ability. 2.3.3. Personality Personality was assessed using the Global Personality Inventory (GPI; see ePredix, 2001), a personality inventory that was developed to assess the Big 5 personality factors emotional stability, extraversion, openness to experience, agreeableness, and conscientiousness. Facet scales that assess each of the Big 5 factors include, among others, Emotional Control, Optimism, and Stress Tolerance (for emotional stability), Competitiveness, Desire for Achievement, and Sociability (for extraversion), Independence, Creativity, and Vision (for openness), Consideration, Empathy, and Trust (for agreeableness), and Attention to Detail, Dutifulness, and Responsibility (for conscientiousness). The GPI was developed especially for use in employment settings. Extensive evidence exists on its reliability and construct validity across diverse jobs as well as from countries around the world (Schmit, Kihm, & Robie, 2000). 2.4. Analyses Three separate analyses were carried out to arrive at different estimates for operational and incremental validity. First, the meta-analytic data on predictor intercorrelations and criterion-related validity were used in combination with the primary data on AC dimension–predictor relationships to obtain the best estimate of operational validity by means of multiple regression as well as composite correlations. These analyses involved computations of operational validities for AC dimensions as a set, AC dimensions when combined with personality and cognitive ability, and tests of personality and cognitive ability combined. The process was repeated for an optimally weighted composite of AC dimensions as well as for a unit-weighted composite. Unit-weighted estimates were obtained by weighting all AC dimensions equally using the formula for computing composite correlation (see Ghiselli, Campbell, & Zedeck, 1981, chapter 7; also Nunnally, 1978, pp. 166–168), while optimally weighted estimates were obtained using regression weights. In estimating the joint validity of unit-weighted AC dimension composites and other predictors, Big 5 and cognitive ability were optimally weighted based on their meta-analytic regression weights. This procedure in turn required us to first compute composite correlations among AC dimensions with each individual differences predictor, as well as multiple correlations between predictors and the performance criteria, and then subject the resulting correlation matrix to multiple regression analyses. Results from these analyses were then used to compute & 2009 Blackwell Publishing Ltd. AC Dimensions: Nomological Net and Incremental Validity incremental validity of each predictor over all possible predictor combinations. Second, the same analyses were carried out for each single AC dimension in order to evaluate whether useful levels of incremental validity could be obtained using only one, rather than all, AC dimensions. Third, parallel analyses were carried out using primary data on predictor intercorrelations and meta-analytic data for operational validity in the sample of 3062 mid-level managers (criterion information was not available in the primary data). Finally, the same analyses were repeated using the data obtained from the sample of 1923 toplevel managers. Our approach is unique as it yields results that have so far not been available in the literature: even though the operational validity of primary AC dimensions has previously been investigated, their relationships to other predictors were unknown, and incremental validities for AC dimension score composites were thus unavailable. 3. Results 3.1. AC dimension overlap with cognitive ability and personality The observed intercorrelations among AC dimensions, Big 5 personality factors, and cognitive ability are presented in Table 1. Again, the data presented here are the first estimates of this kind, relating the overarching AC dimensions identified by Arthur et al. (2003) to individual differences in personality and cognitive ability. Correlations from the mid-level and top-manager samples are presented separately as well as combined (sample size weighted). For comparative purposes, the correlations between cognitive ability and the Big 5 with overall AC ratings from Collins et al. (2003) and Scholz and Schuler (1993) are also given in Table 1. The correlational pattern was remarkably similar across the two samples of managers, and hence only the results for the combined sample are discussed here. The pattern of relationships between AC dimensions, cognitive ability, and the Big 5 was very compatible with the definitions of the seven AC dimensions offered by Arthur et al. (2003, see Appendix Table A1). As expected, the AC dimensions ‘problem solving’ displayed the strongest relationship with cognitive ability (r ¼.32) and the personality trait openness to experience (r ¼.18). This pattern is intuitively appealing, as problem solving in ACs has been postulated to describe the ability to analyze and reason with information, as well as the ability to generate ideas and imaginative solutions (Arthur et al., 2003). Also as expected, problem solving was the only one of the seven AC & 2009 Blackwell Publishing Ltd. 259 dimensions that displayed a notable relationship to cognitive ability. Other AC dimensions displayed sizable relationships with personality traits, however, and mostly confirmed our expectations. ‘Organizing and planning’ related most strongly to the personality domain of conscientiousness (r ¼.24). This finding is not surprising – orderliness has been postulated as a facet of conscientiousness in many personality taxonomies and has also been empirically established as such (Roberts, Chernyshenko, Stark, & Goldberg, 2005). The AC dimension ‘influencing others,’ which we expected to relate mostly to extraversion, was found to relate to several of the Big 5 personality domains. It indeed displayed a sizable relationship with extraversion (observed r ¼.27), but also related to agreeableness (r ¼.27) and emotional stability (r ¼.24). Influencing others describes an individual’s persuasiveness and surgency. Indeed, most of the lower-order competencies that Arthur et al. (2003) categorized into this primary dimension relate to leadership skills. This was also the case in the present two samples, where many of the lower-order competencies that were included in this dimension related to leading and developing others. Thus, the pattern of relationships with the Big 5 factor of extraversion, but also emotional stability and agreeableness, is in line with the broad leadership literature. These three personality traits relate to leadership effectiveness in general (Judge, Bono, Ilies, & Gerhardt, 2002) as well as transformational leadership in particular (Bono & Judge, 2004). ‘Consideration/awareness of others’ mostly reflects social skills and teamwork-related competencies. Appropriately, this AC dimensions related most strongly to the Big 5 domain of agreeableness (r ¼.27). However, it also reflected individuals’ emotional stability to some extent (r ¼.20). ‘Communication’ was the only AC dimension that displayed virtually no notable relationship with any of the individual differences scales included in this study. The strongest relationship was found to exist between this AC dimension and agreeableness; however, the relationship was quite weak (r ¼.08). Unlike communication, the dimension of ‘drive’ was found to relate to all personality scales. The multitude of relationships as well as their strengths were greater than expected; based on the construct definition this dimension was conceptually only associated with high extraversion and high conscientiousness. While drive related most strongly to extraversion (r ¼.48), it also correlated notably with the remaining Big 5 traits (r ¼.31, .32, .26, and .31 with emotional stability, openness, agreeableness, and conscientiousness, respectively). Post hoc, we can make sense of these relationships with the Big 5 by examining the lowerorder competencies that define drive as a primary AC dimension. The taxonomy established by Arthur et al. International Journal of Selection and Assessment Volume 17 Number 3 September 2009 260 Stephan Dilchert and Deniz S. Ones Table 1. Correlations primary assessment center dimensions with cognitive ability and personality Dimension g ES r N r Sample 1: Mid-level managers Problem solving .34 2995 .09 Organizing and planning .02 2995 .16 Influencing others .02 2995 .22 Consideration of others .06 2994 .20 Communication .03 2995 .06 Drive .09 2995 .34 Tolerance of stress .02 2994 .49 Unit-weighted composite .07 2995 .26 Sample 2: Top-level managers Problem solving .28 1861 .08 Organizing and planning .10 1861 .11 Influencing others .04 1860 .26 Consideration of others .02 1861 .21 Communication .09 1859 .02 Drive .03 1861 .25 Tolerance of stress .03 1861 .52 Unit-weighted composite .13 1878 .20 Both samples combined (sample size weighted) Problem solving .32 4856 .09 Organizing and planning .05 4856 .14 Influencing others .03 4855 .24 Consideration of others .03 4855 .20 Communication .05 4854 .04 Drive .04 4856 .31 Tolerance of stress .02 4855 .50 Unit-weighted composite .09 4873 .24 Overall AC ratingsa .43b 5419 .26b E N O r N r A C N r N r N 2924 2924 2924 2923 2924 2924 2924 2924 .07 .17 .25 .16 .02 .50 .22 .28 2924 2924 2924 2923 2924 2924 2924 2924 .18 .16 .17 .07 .01 .34 .10 .23 2924 2924 2924 2923 2924 2924 2924 2924 .06 .15 .26 .25 .10 .29 .22 .26 2,924 2924 2924 2923 2924 2924 2924 2924 .07 .27 .09 .08 .05 .35 .11 .16 2924 2924 2924 2923 2924 2924 2924 2924 1854 1854 1853 1854 1852 1854 1854 1854 .17 .17 .29 .16 .02 .44 .50 .24 1854 1854 1853 1854 1852 1854 1854 1854 .19 .11 .21 .05 .03 .29 .29 .17 1854 1854 1853 1854 1852 1854 1854 1854 .04 .07 .28 .29 .06 .20 .27 .18 1854 1854 1853 1854 1852 1854 1854 1854 .03 .20 .11 .06 .06 .25 .25 .13 1854 1854 1853 1854 1852 1854 1854 1,854 4778 4778 4777 4777 4776 4778 4778 4778 .11 .17 .27 .16 .00 .48 .33 .26 4,778 4,778 4777 4777 4776 4778 4778 4778 .18 .14 .19 .06 .01 .32 .17 .21 4778 4,778 4777 4777 4776 4778 4778 4778 .05 .12 .27 .27 .08 .26 .24 .23 4778 4778 4777 4777 4776 4778 4778 4778 .05 .24 .10 .07 .05 .31 .16 .15 4778 4778 4777 4777 4776 4778 4778 4778 1023 .36b 1847 .12b 830 .18b 619 .14c 1107 Note: The unit-weighted composite is a sum of the first six dimensions only, in order to provide data consistent with the computations of operational validities, which were only available for these six dimensions from Arthur et al. (2003). aMeta-analytic values for overall AC ratings are presented for comparison purposes. The respective values were obtained from the most recent meta-analytic summaries. For relationships where estimates were not available in the most recent quantitative review, prior meta-analytic work was consulted. bSample size weighted mean r from Collins et al. (2003). cSample size weighted mean r for conscientiousness and achievement motivation (weighted by total N) from Scholz and Schuler (1993). g, general mental ability (overall score on three cognitive ability tests); ES, emotional stability; E, extraversion; O, openness; A, agreeableness; C, conscientiousness. (2003) lists a number of diverse characteristics such as aggressiveness, perseverance, and initiative as belonging to this category. Thus, it is not surprising that drive would also be related to several of the Big 5 personality traits. Lastly, confirming our expectations, ‘tolerance for stress’ exhibited its strongest relationship with emotional stability (r ¼.50) but was also related to extraversion, albeit to a lesser extent (r ¼.33). These results offer a unique contribution to the literature and now provide the basis for a better understanding of the construct underpinnings of AC dimensions. It is important to note that for the purpose of this study, observed (uncorrected) estimates of predictor intercorrelations were of interest. If one were to correct the values presented in Table 1 for attenuation due to unreliability in both types of measures using mean reliabilities from meta-analyses and reliability generalization studies (Connelly, Ones, Ramesh, & Goff, 2008; Hülsheger, Maier, & Stumpp, 2007a, b; Viswesvaran & Ones, 2000), one would obtain International Journal of Selection and Assessment Volume 17 Number 3 September 2009 estimates of construct-level relationships that would be higher than those observed on the measure-level. That is, the relationships between AC dimensions and Big 5 personality traits would on average be 35% larger, while relationships between AC dimensions and cognitive ability would be on average 26% larger than observed values. However, given the applied focus of this study to understand the relative usefulness of AC dimension visà-vis individual differences measures used in personnel selection, our focus on observed predictor interrelationships is appropriate. 3.2. Intercorrelations among AC dimensions Observed intercorrelations among AC dimensions from our two samples are presented in Table 2. Results show that the seven dimensions were moderately related (r ¼ :33 and .34 in the mid-level and top-level manager samples, respectively). These values are no- & 2009 Blackwell Publishing Ltd. AC Dimensions: Nomological Net and Incremental Validity 261 Table 2. Intercorrelations among primary assessment center dimensions in Samples 1 and 2 Dimension 1. Problem solving n 2. Organizing and planning n 3. Influencing others n 4. Consideration/awareness of others n 5. Communication n 6. Drive n 7. Tolerance of stress/uncertainty n 1 .42 3062 .46 3062 .27 3061 .33 3062 .25 3062 .23 3061 2 3 4 5 6 7 .50 1923 .41 1922 .42 1922 .20 1923 .17 1923 .53 1922 .25 1921 .21 1921 .44 1920 .63 1921 .40 1923 .40 1923 .44 1922 .15 1923 .12 1921 .28 1922 .27 1922 .41 1921 .27 1922 .17 1920 .46 1922 .45 3062 .26 3061 .27 3062 .40 3062 .20 3061 .52 3061 .45 3062 .38 3062 .33 3061 .60 3061 .27 3061 .31 3060 .14 3062 .20 3061 .25 3061 Notes: Values below diagonal obtained from Sample 1 (mid-level manager assessment center); values above diagonal from Sample 2 (top-level manager assessment center). ticeably lower than those reported in the quantitative summary of Arthur et al. (2003), where the average sample size weighted mean r among the seven primary AC dimensions was reported as .52. The lower average correlation between the AC dimensions in the primary data may lead to higher estimates of AC validity once dimension composite validities are estimated. One should note that the sample sizes for many of the AC dimension intercorrelations obtained from the primary data in this study are much larger than the cumulated sample sizes for the meta-analytic values reported in Arthur et al. (2003). For nine out of the 15 correlations, the sample sizes in this study exceed those of Arthur and colleagues; for eight of these nine correlations, the sample sizes were more than twice as large as the metaanalytic Ns. Although meta-analytic data have advantages over single sample investigations, well-conducted, large-scale primary data can add significant value, especially in cases like the one at hand, where the meta-analytic investigation does not include much of the unpublished work. 3.3. Validity of AC dimensions 3.3.1. Operational validity As described in the analyses section, meta-analytic validities for the six AC dimensions communication, consideration/awareness of others, drive, influencing others, organizing and planning, and problem solving were obtained from Arthur et al. (2003). These estimates, together with observed intercorrelations among AC dimension scores (both meta-analytic and from primary data), were used to obtain multiple correlations between AC dimensions and overall job performance. The resulting multiple correlations estimate the validity that can be expected from composites of these & 2009 Blackwell Publishing Ltd. six AC dimensions. To this end, we computed validities for both unit-weighted and optimally weighted composites of AC dimensions. These validity estimates are presented in Table 3. The operational validity estimates computed for AC dimension composites based on meta-analytic intercorrelations among the primary dimensions were nearly identical (.44 for unit-weighted and .45 for optimally weighted composites). The same was the case for validity estimates computed based on AC dimension intercorrelations obtained from primary data. However, as expected, these validities were slightly higher due to the lower intercorrelations of AC dimensions in the primary data. For the mid-level manager sample, the values were .49 and .52, and for the top-level manager sample .50 and .51 (unit- and optimally weighted, respectively). In Table 3, the validity of overall AC ratings from a prior meta-analysis is presented for comparative purposes (r ¼ .36; Gaugler et al., 1987). Recall that AC dimension composites that we computed were derived entirely mechanically. Thus, we observe that both unit- and optimally weighted, mechanically combined AC dimension composites yield validity estimates well above those for overall AC ratings, which are often obtained, at least in part, using subjective information combination on the part of assessors. 3.3.2. Incremental validity of AC dimensions over personality and cognitive ability Using the information obtained from this study together with meta-analytically derived predictor intercorrelations, we formed predictor composites of AC dimensions (unit- and optimally weighted), personality test scores, and cognitive ability. The resulting validity estimates were substantial. Adding the primary AC dimensions (unit-weighted) as a set to the Big 5 factors International Journal of Selection and Assessment Volume 17 Number 3 September 2009 262 Stephan Dilchert and Deniz S. Ones Table 3. Incremental validity of assessment center dimensions over psychometric tests for predicting job performance Overall AC ratings (incl. subjective combination) AC dimensions (unit-weighted) AC dimensions (optimally weighted) .36 .44 .45 .12 .02 .00 .17 .12 .09 .27 .15 .12 .49 .52 .26 .16 .11 .31 .19 .13 .50 .51 .27 .14 .10 .30 .15 .12 Meta-analytic estimates Operational validity Incremental validity over . . . Big 5 Cognitive ability Cognitive ability þ Big 5 Mid-level manager sample (N ¼ 3062) Operational validity Incremental validity over . . . Big 5 Cognitive ability Cognitive ability þ Big 5 Top-level manager sample (N ¼ 1923) Operational validity Incremental validity over . . . Big 5 Cognitive ability Cognitive ability þ Big 5 Note: Computations were based on operational validity estimates and observed predictor intercorrelations from sources listed in Appendix Table A2. of personality yields criterion-related validity estimates of .47 for the prediction of job performance (based on meta-analytic predictor intercorrelations), while adding them to measures of cognitive ability increases validity estimates to .68. Combining all three types of predictors available yields an estimate of operational validity of .71. Optimally weighted AC dimension composites performed even slightly better. Similar results were obtained when substituting meta-analytic predictor intercorrelations with those obtained in the primary samples. The detailed computations of operational validities of predictor combinations can be found in Appendix Tables A3–A5 for the meta-analytic and primary samples, respectively. Next, we focus our discussion on incremental validity results. Incremental validity estimates of AC dimensions over personality and cognitive ability were computed based on the multiple correlations reported above; these results are also reported in Table 3. While the incremental validity that can be expected from overall AC ratings over personality measures and tests of cognitive ability has already been reported elsewhere (incremental validity over personality being substantial, see Goffin, Rothstein, & Johnston, 1996; whereas the incremental validity over cognitive ability is negligible, see Hardison, 2005; Schmidt & Hunter, 1998), the estimates of relative incremental value offered by AC dimensions and different ways of combining them are unique to the present study. Results show that mechanically (unit- or optimally) weighted composites of AC dimensions provide incremental validity over tests of cognitive ability and personality. Incremental validity estimates from meta-analytic data were .17 and .27 for unit- and optimally weighted AC dimension composites over International Journal of Selection and Assessment Volume 17 Number 3 September 2009 tests of personality. The incremental validity of AC dimensions over tests of cognitive ability was estimated at .12/.15 (unit-/optimally weighted). Finally, the incremental validity over both types of individual differences predictors was found to be .09/.12. This is in contrast to the findings for overall AC ratings, which provide no incremental validity over tests of personality and cognitive ability. That is, with regard to operational and incremental validity, both unit- and optimally weighted AC dimension composites fare much better than overall AC ratings (which are often derived by subjectively combining information across AC dimensions or making holistic judgments). It is encouraging that the estimates obtained from primary data were very similar. The estimates for operational validities of AC dimension composites were only slightly higher due to the lower average intercorrelations among dimensions, and the overall conclusions regarding incremental validity remain unchanged. The incremental validity estimates from our primary data were .11/.13 and .10/.12 for both types of AC composites in the mid-level and top-manager samples, respectively. These values are perfectly in line with the estimate of .09/.12 obtained from the meta-analytically based data. 3.3.3. Incremental validity of single AC dimensions The incremental validity that can be expected from six of the seven primary AC dimensions is high enough to consider ACs useful tools for evaluating managerial candidates. However, the question remains whether similar incremental validity for performance prediction can be achieved using only one of the seven AC dimensions. This issue is particularly interesting from & 2009 Blackwell Publishing Ltd. AC Dimensions: Nomological Net and Incremental Validity 263 Table 4. Incremental validity of each assessment center dimension over psychometric tests for predicting job performance AC dimensions Problem solving Operational validity .39 Meta-analytic estimates Incremental validity over . . . Big 5 .19 Cognitive ability .04 Cognitive ability þ Big 5 .04 Mid-level manager sample (N ¼ 3062) Incremental validity over . . . Big 5 .24 Cognitive ability .04 Cognitive ability þ Big 5 .04 Top-level manager sample (N ¼ 1923) Incremental validity over . . . Big 5 .22 Cognitive ability .05 Cognitive ability þ Big 5 .05 Organizing and planning Influencing others Consideration/ awareness Communication Drive .37 .38 .25 .33 .31 .12 .10 .07 .14 .11 .09 .07 .06 .04 .16 .08 .07 .05 .09 .07 .15 .11 .06 .19 .11 .08 .09 .07 .05 .17 .08 .08 .09 .11 .05 .15 .08 .05 .19 .10 .09 .10 .05 .04 .17 .07 .07 .10 .07 .03 Note: Computations were based on operational validity estimates and observed predictor intercorrelations from sources listed in Appendix Table A2. an applied perspective. Organizations that are already employing a specific combination of predictor measures may be interested in adding only AC dimensions that supplement their current tools in the most optimal yet efficient manner. The results of our analyses that speak to this issue are presented in Table 4. Naturally, which specific dimension offers the highest incremental value depends on the combination of predictors already being employed for a given criterion. If one were looking to supplement a test of the Big 5 domains of personality in performance prediction by assessing only a single AC dimension, the best choice with regard to incremental validity would be problem solving. This comes as no surprise when considering that this AC dimension is by far the most cognitively loaded of the seven primary dimensions (sample size weighted mean observed r in the present two samples ¼ .32). Because of its high operational validity ( ¼ .39; Arthur et al., 2003) and universally small overlap with the Big 5 domains of personality (mean r in the present two samples ¼ .08), this AC domain increments validity by .19 correlational points when added to the Big 5. Conversely, when looking to increment the validity of tests of cognitive ability, the best choice is the AC dimension influencing others. This leadership related dimension displayed low overlap with cognitive ability scores in our two samples (mean r ¼.03). Because of its high operational validity ( ¼ .38), this dimension would add .11 in incremental validity when added to tests of cognitive ability. This dimension also remains the best choice when tests of cognitive ability and personality are combined with only one AC dimension; the incremental validity estimate is .09 as obtained from meta-analytic data and primary samples. This certainly compares quite favorably with the incremental validity & 2009 Blackwell Publishing Ltd. that would be obtained if all AC dimensions were added to the equation; that meta-analytic estimate was only slightly higher (.09/.12; see Table 3). 3.3.4. Incremental validity of personality and cognitive ability over AC scores While the focus of the present investigation was on the incremental validity of ACs over tests of personality and cognitive ability, we provide a parallel investigation for the value that such tests offer over overall AC ratings and dimension composites. These results are presented in Table 5; again, details on the computations can be found in Appendix Tables A3–A5. The operational validity of the Big 5 as a set, cognitive ability, and the Big 5 and cognitive ability combined were estimated at .29, .56, and .62 based on the available meta-analytic evidence. Incremental validities were computed based on the correlations with AC dimensions obtained from this study and the metaanalytic correlations with overall AC ratings from Collins et al. (2003) and Scholz and Schuler (1993). Results show that cognitive ability and the Big 5 as a set add substantial value over all types of AC ratings. The combined incremental validity of these two predictors over unit- and optimally weighted AC dimension composites was .28, respectively. Similar estimates were obtained from the data based on the two primary samples; the respective estimates were .28 (mid-level manager sample) and .26 (top-level manager sample). 4. Discussion This study provided the first investigation of the nomological net of the seven primary AC dimensions International Journal of Selection and Assessment Volume 17 Number 3 September 2009 264 Stephan Dilchert and Deniz S. Ones Table 5. Incremental validity of psychometric tests over assessment center dimensions for predicting job performance Meta-analytic estimates Operational validity Incremental validity over . . . Overall AC ratings AC dimensions (unit-weighted) AC dimensions (optimally weighted) Mid-level manager sample (N ¼ 3062) Operational validity Incremental validity over . . . AC dimensions (unit-weighted) AC dimensions (optimally weighted) Top-level manager sample (N ¼ 1923) Operational validity Incremental validity over . . . AC dimensions (unit-weighted) AC dimensions (optimally weighted) Big 5 Cognitive ability Big 5 and cognitive ability .29 .56 .62 .05 .03 .11 .22 .25 .25 .26 .27 .29 .26 .56 .67 .03 .06 .23 .23 .28 .28 .26 .56 .66 .03 .05 .20 .20 .26 .26 Note: Computations were based on operational validity estimates and observed predictor intercorrelations from sources listed in Appendix Table A2. established by Arthur et al. (2003) by assessing their overlap with tests of cognitive ability and personality. The data on these relationships were used in conjunction with prior meta-analytic data on the validity of overall AC ratings, AC dimensions, personality, and cognitive ability, in order to evaluate the relative incremental value that AC dimension scores add to tests of personality and cognitive ability. ACs can increase the applied utility of managerial staffing systems by offering information about individuals that supplements data already provided by standardized psychometric tests. Previously, investigations of incremental validity were only available for overall AC ratings, resulting in low to negligible estimates. As this study has shown, a composite of dimension scores offers useful levels of incremental validity when combined with scores of cognitive ability and personality tests (DR ¼ .12 when optimally weighted). However, incrementing utility in such a way requires making use of information on AC dimensions by mechanical combination of primary dimensions rather than using overall AC ratings. The incremental validity of overall AC ratings over tests of personality plus cognitive ability was found to be .00. A major factor that hampers the criterion-related validity of overall AC ratings in operational settings is the fact that most often scores are combined in a nonoptimal fashion across raters, exercises, and dimensions. Such non-optimal approaches include assigning overall scores based on clinical data combination or discussion among raters (see Grove & Meehl, 1996; Grove, Zald, Lebow, Snitz, & Nelson, 2000; Meehl, 1954). Previous research has shown that mechanically combining AC ratings is superior to judgmentally combining assessor ratings (see, e.g., Feltham, 1988). This research shows that this is also the case when it International Journal of Selection and Assessment Volume 17 Number 3 September 2009 comes to incremental validity. The mode of data combination does not seem to matter much in relation to validity, as long as it is mechanical. The operational and incremental validity estimates established in this study were not appreciably different for unit- and optimally weighted AC dimension composites. This was the case for our investigations based on metaanalytic as well as primary data. From an applied perspective, this study yields an interesting conclusion with regard to whether all AC dimensions need to be assessed in order to achieve good incremental validity. Given the fact that ACs are costly tools and also time consuming to administer (cf. Collins et al., 2003), the question of whether useful levels of incremental validity can be achieved with fewer dimensions is worth some consideration. To this end, we conducted parallel analyses estimating the incremental validity of each AC dimension over the Big 5, cognitive ability, and both predictor types combined. The present set of results (from both meta-analytic and primary data) yield a straightforward answer that may be of high utility in applied settings: useful levels of incremental validity can be obtained even when using only one AC dimension. Practitioners looking to design and implement slimmed-down ACs concentrating on only a few dimensions should concentrate on assessing problem solving and influencing others. These dimensions offer the largest incremental value when added to tests of cognitive ability, personality, or both. The challenge in designing such ACs, however, lies in designing exercises targeted at only a few dimensions, as well as in training assessors to capture and rate only assessee behavior relevant to the specific dimension(s) identified to add incremental validity. It is conceivable that a slimmeddown AC designed to measure only a single AC dimension instead results in assessors providing an & 2009 Blackwell Publishing Ltd. AC Dimensions: Nomological Net and Incremental Validity overall evaluation, or at least an AC dimension rating contaminated by an overall impression of the candidate. Such scores would likely function similarly to overall AC ratings that are derived using subjective data combination across several dimensions. However, in well-designed and implemented ACs, significant resources are typically invested in training assessors and providing them with scientifically sound guidelines and tools for evaluating behaviors observed among candidates. Thus, we are optimistic that such slimmed-down, one- or two-dimension ACs can be developed and implemented for organizations looking to supplement their existing assessment tools. However, because these ACs would put a strong emphasis on maximizing overall validity, they may not be suited to provide developmental feedback. It also remains to be seen whether such slimmed-down ACs really differ from other selection tools that include simulation-type assessments, such as work sample tests, for instance. Nonetheless, validity estimates for single AC dimensions as well as composites of AC primary dimensions are encouraging. This research has shown that previous estimates of the operational and incremental validity estimates were hampered by the fact that only overall AC ratings were considered, rather than scores on meaningful AC dimensions. We proposed that overall AC scores do not add incremental validity due to the fact that they are often obtained in a non-optimal fashion (typically using subjective means of data combination). Yet, another major reason is that overall AC scores are often not aligned with other tools that are already being administered. Additional measures increase the overall reliability of a predictor battery and will thus increase overall validity. However, in addition to increasing overall reliability of an assessment battery, construct-based AC dimensions can also add construct coverage. For predictive purposes, the more indicators of a given construct we can administer, the better. Each measure will conceptualize the construct domain slightly differently (after all, this study also showed that AC dimensions are not correlated perfectly even with tests of conceptually related individual differences traits). We now know that the effects of increased construct coverage can go beyond those of simply increasing reliability. Connelly and Ones (2007) showed that adding a test of conscientiousness to another test of conscientiousness can result in large increases in overall validity, simply because the coverage of the predictor construct increases (albeit the overall validity asymptotes to a certain level after a certain number of scales). They found that this was even the case on the facet level, where construct definitions were identical. Similarly, construct-based AC dimensions provide the opportunity to explain variance in job performance with the help of predictor constructs not yet well represented or assessed incompletely in a given test battery. & 2009 Blackwell Publishing Ltd. 265 As discussed above, we can also add AC dimensions that measure constructs not at all represented in certain predictor batteries. For example, if an organization only uses ability and experience-related constructs to identify suitable candidates, an AC that places a strong emphasis on ‘drive’ has the potential to add incremental validity. We know from the I/O psychology literature that this is not the case for overall scores (Schmidt & Hunter, 1998), possibly due to their high correlation with cognitive ability (Collins et al., 2003). Thus, there is a need to employ construct-based dimensions that clearly tease predictor constructs apart and show little overlap with exactly those predictors already in use. Again, an emphasis on constructs assessed in ACs, rather than on scores for the overall method, provides a more fruitful avenue for investigations of incremental validity. In light of the long-lasting controversy on AC construct validity (for a recent exchange, see Lance, 2008; Connelly et al., 2008, and other replies published in Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 1), one might ask the question whether similarly encouraging results of incremental validity would be obtained for individual AC exercise scores than those we established here for AC dimensions. While such question has merit, an investigation of AC exercise score validity over individual differences traits would again constitute an investigation of methods (AC exercises regardless of dimension assessed) vs constructs (e.g., personality or GMA). We are convinced that the potential for incremental validity of AC dimensions as established in our research is based on additional predictor construct coverage, and the opportunity to add individual AC dimensions that assess constructs not yet well assessed by other predictors. It is possible that exercise scores that summarize performance across several predictor constructs function similarly to overall AC ratings. Nonetheless, we see some potential for incremental validity of AC exercise scores when they are combined mechanically such as in unit-weighted composites (compared with previous investigations of overall AC ratings that often include subjective methods of data combination). Unfortunately, investigations of AC exercise validity parallel to the one presented here for AC dimensions are currently impossible – the literature is lacking good validity estimates of AC exercises in predicting job performance. We could only locate a single sample of 359 candidates for first-line supervisor positions (Study 2 in Lance, Newbolt, Gatewood, Foster, French, & Smith, 2000) that contains correlations between AC exercise factors and a job performance criterion. Even investigations of AC exercise validity for distal performance outcomes such as salary are extremely rare (see Lievens, Dilchert, & Ones, in press). In this study, we were fortunate enough to be able to rely on meta- International Journal of Selection and Assessment Volume 17 Number 3 September 2009 266 Stephan Dilchert and Deniz S. Ones analytic evidence on AC dimension criterion-related validity; our field should take strides to accumulate similar data on the usefulness of AC exercises as an additional step toward resolving the AC construct validity question. Notes 1. 2. Predictor intercorrelations should not be corrected for attenuation due to unreliability, because in estimating operational validities we are interested in the predictive efficiency of these measures as they are used in operational settings, recognizing that scores are subject to measurement error. Estimating operational validity in this way is standard procedure in personnel selection research, and the same procedure is often applied in metaanalyses of criterion-related validities of popular predictors (Kuncel, Hezlett, & Ones, 2001; Ones, Dilchert, Viswesvaran, & Judge, 2007; Ones, Viswesvaran, & Schmidt, 1993; Schmidt & Hunter, 1998). Assessment centers were conducted both for developmental as well as selection purposes. However, the predictor intercorrelations contributing to our analyses were very similar for developmental and selection ACs. Thus, overall findings are virtually identical across assessment purposes. For reasons of brevity, we present only results for the combined analysis. References Ackerman, P.L. and Heggestad, E.D. (1997) Intelligence, Personality, and Interests: Evidence for overlapping traits. Psychological Bulletin, 121, 219–245. Arthur, W., Day, E.A., McNelly, T.L. and Edens, P.S. (2003) A Meta-Analysis of the Criterion-Related Validity of Assessment Center Dimensions. Personnel Psychology, 56, 125–154. Arthur, W. and Villado, A.J. (2008) The Importance of Distinguishing Between Constructs and Methods When Comparing Predictors in Personnel Selection Research and Practice. Journal of Applied Psychology, 93, 435–442. Bono, J.E. and Judge, T.A. (2004) Personality and Transformational and Transactional Leadership: A meta-analysis. Journal of Applied Psychology, 89, 901–910. Carroll, J.B. (1993) Human Cognitive Abilities: A survey of factoranalytic studies. New York: Cambridge University Press. Collins, J.M., Schmidt, F.L., Sanchez-Ku, M., Thomas, L., McDaniel, M.A. and Le, H. (2003) Can Basic Individual Differences Shed Light on the Construct Meaning of Assessment Center Evaluations? International Journal of Selection and Assessment, 11, 17–29. Connelly, B.S. and Ones, D.S. (2007). Combining Conscientiousness Scales: Can’t get enough of the trait, baby. Poster session presented at the annual conference of the Society for Industrial and Organizational Psychology, New York City, New York, April. International Journal of Selection and Assessment Volume 17 Number 3 September 2009 Connelly, B.S., Ones, D.S., Ramesh, A. and Goff, M. (2008) A Pragmatic View of Dimensions and Exercises in Assessment Center Ratings. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 127–130. ePredix. (2001) Global Personality Inventory Technical Manual. Minneapolis, MN: ePredix. Feltham, R. (1988) Assessment Centre Decision Making: Judgemental vs. mechanical. Journal of Occupational Psychology, 61, 237–241. Gaugler, B.B., Rosenthal, D.B., Thornton, G.C. and Bentson, C. (1987) Meta-Analysis of Assessment Center Validity. Journal of Applied Psychology, 72, 493–511. Ghiselli, E.E., Campbell, J.P. and Zedeck, S. (1981) Measurement Theory for the Behavioral Sciences. San Francisco: Freeman. Goffin, R.D., Rothstein, M.G. and Johnston, N.G. (1996) Personality Testing and the Assessment Center: Incremental validity for managerial selection. Journal of Applied Psychology, 81, 746–756. Grove, W.M. and Meehl, P.E. (1996) Comparative Efficiency of Informal (Subjective, Impressionistic) and Formal (Mechanical, Algorithmic) Prediction Procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293–323. Grove, W.M., Zald, D.H., Lebow, B.S., Snitz, B.E. and Nelson, C. (2000) Clinical Versus Mechanical Prediction: A metaanalysis. Psychological Assessment, 12, 19–30. Hardison, C.M. (2005). Construct Validity of Assessment Center Overall Ratings: An investigation of relationships with and incremental criterion related validity over Big 5 personality traits and cognitive ability. Unpublished doctoral dissertation, University of Minnesota, Minneapolis, MN. Hardison, C.M. and Sackett, P.R. (2004). Assessment Center Criterion-Related Validity: A meta-analytic update. Annual conference of the Society for Industrial and Organizational Psychology, Chicago, IL. Hermelin, E., Lievens, F. and Robertson, I.T. (2007) The Validity of Assessment Centres for the Prediction of Supervisory Performance Ratings: A meta-analysis. International Journal of Selection and Assessment, 15, 405–411. Hough, L.M., Ones, D.S. and Viswesvaran, C. (1998). Personality Correlates of Managerial Performance Constructs. Poster session presented at the annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX, April. Hülsheger, U.R., Maier, G.W. and Stumpp, T. (2007) Validity of General Mental Ability for the Prediction of Job Performance and Training Success in Germany: A meta-analysis. International Journal of Selection and Assessment, 15, 3–18. Hunter, J.E. (1980) Validity Generalization for 12,000 Jobs: An application of synthetic validity and validity generalization to the general aptitude test battery (GATB). Washington, DC: US Department of Labor, Employment Service. Judge, T.A., Bono, J.E., Ilies, R. and Gerhardt, M.W. (2002) Personality and Leadership: A qualitative and quantitative review. Journal of Applied Psychology, 87, 765–780. Kuncel, N.R., Hezlett, S.A. and Ones, D.S. (2001) A Comprehensive Meta-Analysis of the Predictive Validity of the Graduate Record Examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181. & 2009 Blackwell Publishing Ltd. AC Dimensions: Nomological Net and Incremental Validity Lance, C.E. (2008) Why Assessment Centers Do Not Work the Way They Are Supposed to. Industrial and Organizational Psychology: Perspectives on Science and Practice, 1, 84–97. Lance, C.E., Newbolt, W.H., Gatewood, R.D., Foster, M.R., French, N.R. and Smith, D.E. (2000) Assessment Center Exercise Factors Represent Cross-Situational Specificity, Not Method Bias. Human Performance, 13, 323–353. Lievens, F., Dilchert, S. and Ones, D.S. (in press). The Importance of Exercise and Dimension Factors in Assessment Centers: Simultaneous examinations of constructrelated and criterion-related validity. Human Performance. Meehl, P.E. (1954) Clinical vs. Statistical Prediction: A theoretical analysis and a review of the evidence. Minneapolis, MN: University of Minnesota Press. Nunnally, J.C. (1978) Psychometric Theory (2nd edn). New York: McGraw-Hill. Ones, D.S. (1993). The Construct Validity of Integrity Tests. Unpublished doctoral dissertation, University of Iowa, Iowa City, IA. Ones, D.S., Dilchert, S., Viswesvaran, C. and Judge, T.A. (2007) In Support of Personality Assessment in Organizational Settings. Personnel Psychology, 60, 995–1027. Ones, D.S., Viswesvaran, C. and Reiss, A.D. (1996) Role of Social Desirability in Personality Testing for Personnel Selection: The red herring. Journal of Applied Psychology, 81, 660–679. Ones, D.S., Viswesvaran, C. and Schmidt, F.L. (1993) Comprehensive Meta-Analysis of Integrity Test Validities: Findings and implications for personnel selection and theories of job performance. Journal of Applied Psychology, 78, 679–703. Raven, J., Raven, J.C. and Court, J.C. (1998) Manual for the Raven’s Progressive Matrices and Vocabulary Scales. Oxford, UK: Oxford Psychologists Press. Roberts, B.W., Chernyshenko, O.S., Stark, S.E. and Goldberg, L.R. (2005) The Structure of Conscientiousness: An empirical investigation based on seven major personality questionnaires. Personnel Psychology, 58, 103–139. Schmidt, F.L. and Hunter, J.E. (1998) The Validity and Utility of Selection Methods in Personnel Psychology: Practical and theoretical implications of 85 years of research findings. Psychological Bulletin, 124, 262–274. Schmit, M.J., Kihm, J.A. and Robie, C. (2000) Development of a Global Measure of Personality. Personnel Psychology, 53, 153–193. Scholz, G. and Schuler, H. (1993) Das nomologische Netzwerk des Assessment Centers: eine Metaanalyse [The Nomological Network of the Assessment Center: A metaanalysis]. Zeitschrift für Arbeits- und Organisationspsychologie, 37, 73–85. Viswesvaran, C. and Ones, D.S. (2000) Measurement Error in ‘‘Big Five Factors’’ Personality Assessment: Reliability gen- & 2009 Blackwell Publishing Ltd. 267 eralization across studies and measures. Educational and Psychological Measurement, 60, 224–235. Watson, G.B. and Glaser, E.M. (1980) Watson–Glaser Critical Thinking Appraisal Manual. San Antonio, TX: Psychological Corporation. Wesman, A.G. (1965) Wesman Personnel Classification Test Manual. New York: Psychological Corporation. Appendix Table A1. Primary assessment center dimensions assessed in the current study and corresponding definitions from Arthur et al. (2003) Dimension and definition Communication ‘The extent to which an individual conveys oral and written information and responds to questions and challenges’ Consideration/awareness of others ‘The extent to which an individual’s actions reflect a consideration for the feelings and needs of others as well as an awareness of the impact and implications of decisions relevant to other components both inside and outside the organization’ Drive ‘The extent to which an individual originates and maintains a high activity level, sets high performance standards and persists in their achievement, and expresses the desire to advance to higher job levels’ Influencing others ‘The extent to which an individual persuades others to do something or adopt a point of view in order to produce desired results and takes action in which the dominant influence is one’s own convictions rather than the influence of other’s opinions’ Organizing and planning ‘The extent to which an individual systematically arranges his/ her own work and resources as well as that of others for efficient task accomplishment; and the extent to which an individual anticipates and prepares for the future’ Problem solving ‘The extent to which an individual gathers information; understands relevant technical and professional information; effectively analyzes data and information; generates viable options, ideas, and solutions; selects supportable courses of action for problems and solutions; uses available resources in new ways; and generates and recognizes imaginative solutions’ Tolerance of stress/uncertainty ‘The extent to which an individual maintains effectiveness in diverse situations under varying degrees of pressure, opposition and disappointment’ Note: Adapted from Arthur et al. (2003). International Journal of Selection and Assessment Volume 17 Number 3 September 2009 268 Stephan Dilchert and Deniz S. Ones Table A2. Sources of primary and meta-analytic estimates of predictor interrelationships and operational validities Relationship Predictor interrelationships AC dimensions –intercorrelations AC dimensions–Big 5 AC dimensions –cognitive ability Big 5 –cognitive ability Source Type of estimate used Arthur et al. (2003), as well as primary data from this study Primary data from this study, Samples 1 and 2 combined (N ¼ 4778) Primary data from this study, Samples 1 and 2 combined (N ¼ 4873) Ackerman and Heggestad (1997) Sample size weighted mean r Sample size weighted mean r No meta-analytic estimate available in the literature Sample size weighted mean r No meta-analytic estimate available in the literature Values for 4 of the Big 5 were directly available. The estimate for ‘wellbeing’ was substituted for emotional stability, which was unavailable Sample size weighted mean of estimates for global conscientiousness and achievement motivation Big 5 –intercorrelations Ones (1993); also reported in Ones, Viswesvaran, and Reiss (1996) Overall AC ratings –Big 5 (except conscientiousness) Overall AC ratings –conscientiousness Collins et al. (2003) True score correlations attenuated using meta-analytic reliability estimates (obtained from Viswesvaran & Ones, 2000, for Big 5 and Hülsheger et al., 2007 for cognitive ability) True score correlations attenuated using meta-analytic reliability estimates (obtained from Ones and Viswesvaran, 2000) Sample size weighted mean r Scholz and Schuler (1993) Sample size weighted mean r Overall AC ratings –cognitive ability Operational validities AC dimensions –job performance Overall AC ratings –job performance Cognitive ability –job performance Collins et al. (2003) Sample size weighted mean r Arthur et al. (2003) Operational validity Gaugler et al. (1987) Operational validity Hunter (1980) Operational validity Hough et al. (1998) Operational validity Big 5 –job performance International Journal of Selection and Assessment Volume 17 Number 3 September 2009 Comment Estimate for ‘tolerance for stress’ unavailable Validity for performance criteria Criterion: Overall job performance in high complexity jobs Criterion: Managerial performance & 2009 Blackwell Publishing Ltd. & 2009 Blackwell Publishing Ltd. R w/Big 5 .62c over . . . g and AC .03 R w/Big 5 .47i Big 5 .17 Big 5 .27 g and Big 5 .12 R w/Big 5 R w/g R w/Big 5 and g .57i .71j .74k Dimensions optimally weighted g and Big 5 g .09 .15 R w/g R w/Big 5 and g R .68j .71k .45h Big 5 g and Big 5 g .12 .00 .12 R w/Big 5 and g R .62g .44h Overall rating (incl. subjective com- Dimensions unit-weighted bination) Assessment center ratings Big 5 and AC g .24 .02 R w/Big 5 R w/g .41e .58f Big 5 AC .33 .25 r .36d Cognitive ability R w/Big 5 .67c Big 5 .40 r .56b g and AC .05 AC .23 R .49d Cognitive ability Big 5 and AC .22 R w/Big 5 .52e g .16 R w/g .72f Big 5 .26 R w/Big 5 and g .78g Dimensions unit-weighted Assessment center ratings g and Big 5 .11 R .52d g .19 R w/Big 5 .57e Big 5 .31 R w/g .75f g and Big 5 .13 R w/Big 5 and g .80g Dimensions optimally weighted Notes: All computations based on estimates of predictor interrelationships obtained from the present sample and estimates of operational validity obtained from the meta-analytic literature (see Table A2 for sources). aBased on Big 5 intercorrelations from this sample and Big 5 operational validities from Hough et al. (1998). bFrom Hunter (1980). cBased on a, b, and Big 5–cognitive ability correlation from this sample. d Based on operational validities of AC dimensions from Arthur et al. (2003) and AC dimension intercorrelations from this sample. eBased on d and AC dimension–Big 5 correlations from this sample. fBased on d and AC dimension–cognitive ability correlations from this sample. gBased on sources for d, e, and f. g, general mental ability (overall score on three cognitive ability tests); Big 5, Big 5 dimensions of personality; AC, unit-weighted composite of primary assessment center dimensions. Operational validity R .26a Incremental validity over . . . g AC .11 .03 Personality Big 5 Table A4. Computation of operational and incremental validity: mid-level manager sample (N ¼ 3062) Notes: All computations based on meta-analytic estimates of operational validity and predictor interrelationships, except for the correlations of AC dimensions with cognitive ability and the Big 5, which had to be obtained from the current two samples. aBased on Big 5 intercorrelations from Ones (1993) and Big 5 operational validities from Hough et al. (1998). bFrom Hunter (1980). cBased on a, b, and Big 5–cognitive ability correlation from Ackerman and Heggestad (1997). dFrom Gaugler et al. (1987). eBased on a, d, and overall AC–Big 5 correlations from Collins et al. (2003) and Scholz and Schuler (1993). fBased on b, d, and overall AC–cognitive ability correlation from Collins et al. (2003). gBased on sources for a, b, c, d, e, and f. hBased on operational validities and intercorrelations of AC dimensions from Arthur et al. (2003). iBased on h and AC dimension–Big 5 correlations from the present two samples. jBased on h and AC dimension–cognitive ability correlations from the present two samples. kBased on sources for h, i, and j. g, general mental ability (overall score on three cognitive ability tests); Big 5, Big 5 dimensions of personality; AC, unit-weighted composite of primary assessment center dimensions. Operational validity R r .56b .29a Incremental validity g AC .06 .03 Personality Big 5 Table A3. Computation of operational and incremental validity: meta-analytic data AC Dimensions: Nomological Net and Incremental Validity 269 International Journal of Selection and Assessment Volume 17 Number 3 September 2009 R .50d AC .20 R w/Big 5 .66c Big 5 .39 Cognitive ability Big 5 and AC .23 R w/Big 5 .53e g .14 R w/g .70f Big 5 .27 R w/Big 5 and g .76g Dimensions unit-weighted Assessment center ratings g and Big 5 .10 R .51d g .15 R w/Big 5 .57e Big 5 .30 R w/g .71f g and Big 5 .12 R w/Big 5 and g .77g Dimensions optimally weighted Notes: All computations based on estimates of predictor interrelationships obtained from the present sample and estimates of operational validity obtained from the meta-analytic literature (see Table A2 for sources). aBased on Big 5 intercorrelations from this sample and Big 5 operational validities from Hough et al. (1998). bFrom Hunter (1980). cBased on a, b, and Big 5–cognitive ability correlation from this sample. d Based on operational validities of AC dimensions from Arthur et al. (2003) and AC dimension intercorrelations from this sample. eBased on d and AC dimension–Big 5 correlations from this sample. fBased on d and AC dimension–cognitive ability correlations from this sample. gBased on sources for d, e, and f. g, general mental ability (overall score on three cognitive ability tests); Big 5, Big 5 dimensions of personality; AC, unit-weighted composite of primary assessment center dimensions. Operational validity R r .26a .56b Incremental validity over . . . g AC g and AC .10 .03 .05 Personality Big 5 Table A5. Computation of operational and incremental validity: Top-level manager sample (N ¼ 1921) 270 Stephan Dilchert and Deniz S. Ones International Journal of Selection and Assessment Volume 17 Number 3 September 2009 & 2009 Blackwell Publishing Ltd.