Page No 20-27
Shweta R.C. Bajpai H. K. Chaturvedi
National University of Study Army College of National Institute of
and Research in Law, Medical Sciences, Medical Statistics,
Ranchi New Delhi New Delhi
Evaluation of inter-rater agreement (IRA) or inter-rater reliability (IRR), either as a primary
or a secondary component of study is common in various disciplines such as medicine,
psychology, education, anthropology and marketing where the use of raters or observers
as a method of measurement is prevalent. The concept of IRA/IRR is fundamental to the
design and evaluation of research instruments. However, many methods for comparing
variations and statistical tests exist, and as a result, there is often confusion about their
appropriate use. This may lead to incomplete and inconsistent reporting of results.
Consequently, a set of guidelines for reporting reliability and agreement studies has
recently been developed to improve the scientific rigor in which IRA/IRR studies are
conducted and reported (Gisev, Bell & Chen, 2013; Kottner, Audige, & Brorson, 2011).
The objective of this technical note is to present the key concepts in relation to IRA/IRR
and to describe commonly used approaches for its evaluation. The emphasis will be
more on the practical aspects about their use in behavioral and social research rather
than the mathematical derivation of the indices