Effects of descriptor specificity and observability on incumbent work analysis ratings

Judgments regarding the requirements of jobs and the requirements of individuals performing those jobs comprise the critical groundwork on which human resource practices are built. Yet, such judgments are potentially limited in a variety of ways. Using a large sample of incumbents (N = 47,137) spanning over 300 different occupations, we present research empirically examining how the specificity and observability of focal descriptors affect subsequent ratings. We use variance component (VC) estimation and meta-analysis to investigate sources of variance and interrater reliability of importance ratings across 5 descriptors (tasks, responsibilities, knowledge, skills, and traits). Results indicate that when ratings are rendered on descriptors of low specificity and low observability (e.g., traits), variance due to rater idiosyncrasies increases and reliability decreases. Implications for work analysis practice and future research are discussed.