Techniques for Establishing Inter-Coder Reliability
Establishing acceptable levels of agreement across team members is an important and fundamental process for many projects employing qualitative data and coding systems. There are a variety of ways to engage in this process and a formal evaluation of this agreement is commonly reported as an important step toward increasing the trustworthiness and credibility of a study. Basically, this process provides opportunity for the research team to explore their data and discuss various interpretations, multiple meanings, and coding frameworks as you develop a shared understanding and refined codebook (Ravitch & Carl, 2021).
Dedoose offers features supporting a few approaches to building and maintaining team consistency depending on your needs and stage in the process. Before selecting which approach (or approaches) you want to use, please consider the purpose of each and what value you will gain by engaging with your data from various perspectives.
Reminder: Ultimately, the goal is for the process to result in the development of a robust code system that can be applied with high levels of consistency and trustworthiness between coders/researchers across your project data. Memoing, discussing interpretations among your team, and documenting each stage of this iterative process will result in a transparent process to how your team arrived at and applied the final system. Such documentation is ideal when reporting the methodological approaches you took to produce the final study results and promote greater confidence in the ultimate findings.
The Learning Center outlines 3 common approaches to establishing and evaluating inter-rater reliability:
- Collaborative Code and Compare: Best for early in the process and for those who want to take an approach more grounded in traditional qualitative methodology. You and your collaborators code the same exact file and then compare your code applications and interpretations.
- Document/Media Cloning: A common intermediate step to explore more 'apples-to-apples' comparisons. You and your collaborators each independently code your designated file 'clone' and then review your code applications and interpretations.
- Training Center: Using the Training Center is often a later stage step for those seeking a statistics-based approach to documenting IRR. Training Center tests are also valuable in training others on their codebook and code application criteria.
Caution: Use appropriate caution when employing quantitative approaches to IRR evaluation. Dedoose has included a quantitative assessment of IRR via the Training Center, but we recommend using it in conjunction with other approaches to gauge levels of agreement as it pertains to code applications, interpretations, the purpose of the study, and the actual data available. For more information on establishing a robust code system among multiple coders more aligned with qualitative methods, visit the "Additional Resources" section of this guide