PA Home > Modules Main Page > Part 6: Validity and Reliability > Reliability

6:5 Reliability

Reliability: The likelihood that an assessment is accurate and consistent.

Keys:

  • Setting clear, detailed criteria for the learner to use to create a work and for the specified audience to use to assess the work.
  • Making sure raters understand and apply the criteria in the same way.
Reliability refers to the consistency and accuracy of the assessment tool to measure students' performance. Portfolios are only effective if you establish a fair and consistent way of scoring, recording, and grading them. The key to developing a reliable portfolio is to establish clear and detailed criteria. Criteria are statements of exactly what skill or knowledge the specified learner or group of learners is to master, what constitutes mastery, and how progress toward mastery is measured. Criteria provide the framework for planning instruction and evaluation. In education, when criteria are used to frame instruction, they are called objectives, goals, or learning outcomes. Criteria used to assess the impact of the instruction are referred to as evaluation measures, tests, instruments, and rubrics.

Criteria for both the portfolio and the contents of the portfolio should be established before students undertake their assignments. Doing this ensures that students understand what is expected and that you are actually assessing what you intended to assess. If possible, students should help determine the criteria. This will further ensure clarity, and it will also invest them in the work.

Reliability can also be strengthened through triangulation, a process of demonstrating proficiency in a specific skill by gathering data from three or more sources. This strengthens the accuracy of the scores by providing a more complete picture of students' abilities. Portfolios are successful in this area because they are designed to include evidence of learning progress from a variety of sources, including the student herself, the teacher, peers, and parents.

Portfolios need to be reliable in terms of criteria; however, they also need to have rater reliability. Rater reliability refers to the consistency of scores over time or between raters. You achieve rater reliability with portfolio assessment in two ways. First, you can increase rater reliability by training raters to use the criteria and rating scales in the same way. In portfolio assessment, students participate as raters in peer-assessment and self-assessment; therefore, students should also be trained to use the criteria in the specified way. Similarly, if a teacher who was not involved with setting the criteria is going to evaluate the students' work, she needs to be familiarized with the criteria and how to apply them in evaluation. This is especially important if you plan to rely on other teachers to evaluate your students' portfolios. You can also strengthen rater reliability by involving teacher and student raters in setting the criteria. This allows them to understand and be familiar with the criteria and clear about how to apply the criteria to help evaluate the portfolio and its contents.

Reliability and validity are part of both planning and evaluation. You can set the groundwork for reliability and validity at the inception of the portfolio period and monitor progress along the way as pieces are gathered and added. At the end of the marking period, evaluate the portfolio contents in relation to your original purpose.

Teacher reflection: The first time I used portfolios to determine the grades of my students I relied on my judgment of the students' work. I looked to see if the individual student had made progress in the areas being assessed and on that perception, I made my grading decision. After consulting with my foreign language supervisor, I realized that my grading was not systematic, nor was it fair or accurate. As a result of our conference, I reevaluated my grading system, established benchmark portfolios that represented the various levels of ability, and most importantly, I created a scale for grading that I shared with my students at the beginning of the marking period. These changes increased the reliability of my students' assessment portfolios, and their grades more accurately reflected their abilities. *The teacher reflections shared throughout the manual come from a variety of practicing teachers at secondary and post-secondary level.


Back to the Top
<< Back | Next >>