Risk Score: Student Performance Metric & Risk Factor Definitions
The risk score is a weighted points-based model that represents the student’s risk of dropping out of their academic program and not graduating on time, with 100 representing the maximum amount of risk. Here on the Retention Overview dashboard, you can see students’ risk scores and the actual values of the five performance metrics this sample school is using to calculate a student’s risk.
CourseKey enables customers to take into account up to five student performance metrics currently, each of which are weighted before being added as a risk factor for the calculation of students' risk scores.
Last Activity Recorded | This is the last date a student interacted with any of their courses. When considered as a risk factor, days to last activity is calculated as the number of days since a student’s last activity recorded. |
Attendance | A percentage that represents a student’s attendance versus the total expected attendance time in synchronous attendance sessions in their courses. |
Lateness | A percentage that represents the ratio of a student’s late check-ins compared to all their check-ins (late and on-time) in all synchronous sessions the student has attended in their courses. |
Academics | A percentage that represents a student’s grades performance compared to the expected total grade, considering all their graded activities (assignments, quizzes, discussion topics, etc.). |
Checklists | A percentage that represents the ratio of a student’s actual checklist item submissions count-to-date, compared to their expected checklist item submissions count-to-date, given the time in their program. |
For example, for calculating students’ risk scores, a school might only consider a student's attendance, checklists and lateness performance, and weight them by 50%, 30% and 20% of the score respectively. Before applying weights and thresholds, each of the five student performance metrics in our risk score model, is tallied with a set of assumptions, based on the different use cases we support, and which are described below. (See our article on weight and threshold definitions for more information and examples.)
Last Activity Recorded
A student’s last activity recorded, whether online or in person, is the best way for a school to know how immediate the need for intervention is. If a student is absent online or in person for three or more days (a configurable threshold), this might be a good indicator that the student is in need of outreach.
The last activity recorded metric represents the last date a student interacted with any of their courses. When considered as a risk factor, days to last activity is calculated as the number of whole days since a student’s last activity was recorded, with any fractional amount of days rounded down to the nearest integer that is equal to or below the actual value. For example, a value between 2.01 to 2.99 will be converted to 2, to account for any whole days only.
How does a Student's Days to Last Activity Impact Their Risk Score?
- A student with 0 days to last activity, given a last activity recorded metric’s value of yesterday, would have 0 points added to their risk score.
- If no threshold is set for a school/program, a student would accrue points at a 1:1 percentage basis for their performance in the last activity recorded metric (days to last activity), which are added to their risk score.
- If a threshold is set for a school/program, for every day of inactivity beyond yesterday all the way up to the risk factor threshold, a student would accrue percentage points linearly up to a maximum number of points determined by the risk factor's weight, which are added to their risk score. (See our article on weight and threshold definitions for more information and examples.)
How is the Last Activity Recorded Metric Calculated?
CourseKey supports schools tracking LDA by maintaining data on students' last activity recorded (LAR), to reflect the last date of each student’s engagement in certain in-scope activities carried out within the CourseKey application or an integrated school’s Learning Management System (LMS). See our article on last activity recorded for more information.
Attendance
How much of their courses a student has missed, consecutively and throughout their program, can indicate that a student is at risk and in need for intervention. For example, if a student misses 14-days in a row (a configurable threshold), your school has no choice but to drop them to remain compliant with Title IV. While consecutive absences are a red flag to administrators, frequent, non-consecutive absences can significantly impact a student’s ability to complete their program. Frequent absences can impact grades, hours, and other academic factors.
Calculated as a percentage, the attendance metric represents a student’s attendance versus the total expected attendance time in synchronous attendance sessions in their courses, looking back as far as their program start date or the past year, whichever comes first.
How Does a Student’s Attendance Performance Impact Their Risk Score?
- A student that has perfect performance for the attendance metric (a value of 100%) would have 0 points added to their risk score.
- If no threshold is set for a school/program, a student would accrue points at a 1:1 percentage basis for their performance in the attendance metric, which are added to their risk score.
- If a threshold is set for a school/program, every percentage point decrease in the attendance metric means that a student would accrue risk score points linearly from 99% until the risk factor threshold, and up to a maximum number of points determined by the risk factor's weight, which are added to their risk score. (See our article on weight and threshold definitions for more information and examples.)
How is the Attendance Metric Calculated?
The calculation of the attendance metric takes into account a student’s attendance time in synchronous sessions only, and does not consider attendance in flexible schedule or credit hour courses. Time awarded for asynchronous assignments completed in an integrated school’s LMS are also not taken into account in the calculation.
Different types of courses differ in how a student's actual attendance and expected attendance are calculated, and expectations are outlined below.
Synchronous Courses
- Attendance data is based on the same logic as CourseKey’s timesheet, and the metric calculation takes into account data from any attendance session scheduled within the lookback period, in any course that is active (completed or ongoing) and in which the student is active (not dropped) as of the current date.
- Each scheduled attendance session's start and end date and break duration, various student check-ins and check-outs, and any time adjustments, are used to calculate a student’s actual attendance and expected attendance.
- A student's actual attendance is the difference between a student’s first check-in and the last check-out, minus the break duration (if applicable), plus/minus any time adjustments, for any attendance session scheduled in a course.
- A student's expected attendance is the difference between the start time and the end time, minus the break duration (if applicable), for any attendance session scheduled in a course.
Blended Courses
- The calculation takes into account data from any attendance session scheduled within the lookback period, in any course that is active (completed or ongoing) and in which the student is active (not dropped) as of the current date.
- The expected attendance and actual attendance for synchronous attendance sessions are calculated in a similar way as for synchronous courses.
- The calculation does not take into account time awarded for asynchronous activities (assignments, quizzes, discussion topics, etc.) completed in an integrated school’s LMS.
Wheel Courses (Special Use Case of Synchronous Courses)
- The 'wheel' course approach, commonly used by beauty and wellness schools, applies for calculating the expected time and actual time in synchronous courses that commonly span a few years (2-3 years), and have a rolling enrollment. Students in this type of course are added as course members when they start participating in a course, and then dropped when they complete the course; with each student having their own timeline for participation.
- The calculation takes into account data from any attendance session scheduled within the lookback period, in any course that is active (ongoing) and in which the student is active (not dropped) as of the current date.
- The actual attendance and expected attendance are calculated in the same way as for a synchronous course, but only accounting for attendance sessions scheduled while a student is enrolled (an active member record) in the course.
- The calculation of expected attendance disregards all attendance sessions before a student’s enrollment date, and after the student’s drop date in a course.
Timesheet Makeup
- The calculation takes into account any makeup time recorded (and not voided) within the lookback period.
- Makeup time is stacked to a student’s actual attendance on the day the makeup was completed.
Lateness
How frequently a student is late to class impacts their attendance percentage, but it also provides a surface-level indicator of risk and the need for intervention. Understanding a student’s behavioral history in regards to how frequent they are late for their courses can help tailor your outreach and provide them with appropriate resources and support.
Calculated as a percentage, the lateness metric represents the ratio of a student’s late check-ins compared to all their check-ins (late and on-time) in all synchronous sessions the student has attended in their courses, looking back as far as their program start date or the past year, whichever comes first.
How Does a Student's Lateness Performance Impact Their Risk Score?
- A student that has perfect performance for the lateness metric (a value of 0%) would have 0 points added to their risk score.
- If no threshold is set for a school/program, a student would accrue points at a 1:1 percentage basis for their performance in the lateness metric, which are added to their risk score.
- If a threshold is set for a school/program, every percentage point increase in the lateness metric means that a student would accrue risk score points linearly from 0% until the risk factor threshold, and up to a maximum number of points determined by the risk factor's weight, which are added to their risk score. (See our article on weight and threshold definitions for more information and examples.)
How is the Lateness Metric Calculated?
The calculation of the lateness metric takes into account check-ins (late and on-time) in synchronous attendance sessions only. Late/on-time check-ins in flexible schedule courses or late submissions for asynchronous activities completed in an integrated school’s LMS are not taken into account in the calculation. Different types of courses differ in how the counts of late check-ins and total check-ins are calculated, and these expectations are outlined below.
Synchronous Courses
- Lateness data is based on the same logic as CourseKey’s timesheet, and the metric calculation accounts for data in any attendance session scheduled within the lookback period, in any course that is active (completed or ongoing) and in which the student is active (not dropped) as of the current date.
- The calculation takes into account a student’s late check-ins and total check-ins (late and on-time) at the start of synchronous sessions only (the first check-in, and not any additional check-ins or check-ins after breaks).
- The calculation ignores data in any attendance session in which a student was absent.
- A late check-in means that a student checked-in, using any CourseKey's supported attendance methods, or was checked-in by an instructor manually, after the first minute from the start of a scheduled attendance session.
- Any adjustments to a student’s first check-in in a scheduled attendance session recorded over the last year is accounted for in the determination of lateness.
Blended Courses
- The calculation for blended courses takes into account a student's late check-ins and total check-ins in synchronous attendance sessions only, and is calculated in a similar way as for synchronous courses.
- The calculation does not take into account late submissions for asynchronous activities (assignments, quizzes, discussion topics, etc.) completed in an integrated school’s LMS.
Wheel Courses (Special Use Case of Synchronous Courses)
- The lateness metric for wheel courses is calculated in a similar way as for synchronous courses.
Credit Hour (Point-based Attendance) Courses
- Lateness data is based on the credit hour attendance process that allows instructors to mark and record students’ attendance in a scheduled attendance session as on-time, late, or absent.
- The calculation takes into account the number of a student’s attendance sessions with a late status and the total number of attended sessions (on-time and late), within the lookback period, in any course that is active (completed or ongoing) and in which the student is active (not dropped) as of the current date.
- The calculation ignores data in any attendance session in which a student’s status was marked absent.
- Any adjustments to a student’s mark in an attendance session recorded over the last year is accounted for in the determination of lateness.
Academics
One of the biggest red flags that a student may not complete their program, and an indicator of the need for intervention, is whether they’ve been able to maintain their grades and demonstrate a strong understanding of the subject matter.
Calculated as a percentage, the academics metric represents a student’s grades performance compared to the expected total grade, considering all their graded activities (assignments, quizzes, discussion topics, etc.), looking back as far as the student’s program start date or the past year, whichever comes first.
How Does a Student’s Academics Performance Impact Their Risk Score?
- A student that has perfect performance for the academics metric (a value of 100%) would have 0 points added to their risk score.
- If no threshold is set for a school/program, a student would accrue points at a 1:1 percentage basis for their performance in the academics metric, which are added to their risk score.
- If a threshold is set for a school/program, every percentage point decrease in the academics metric means that a student would accrue risk score points linearly from 99% until the risk factor threshold, and up to a maximum number of points determined by the risk factor's weight, which are added to their risk score. (See our article on weight and threshold definitions for more information and examples.)
How is the Academics Metric Calculated?
In the current implementation, only grades for activities completed by students and/or recorded by instructors in an integrated school’s LMS are taken into account for calculating the academics metric. Grades for assessments completed using the internal CourseKey’s Assessments feature are not taken into account for the calculation. If an activity has not been submitted by the student, or if the student has submitted it but it has not been graded by the instructor, there is no grade for the assignment to be considered in the calculation.
All grades data, independently of the source, are treated in a consistent manner, as follows:
- Each student’s raw score (points) for any activity is transformed to a percentage (a value between 0 and 100), by dividing it by the number of points possible (max grade) for the activity.
- This calculation takes into account any extra credit added to the activity’s raw score, when the value of the raw score is larger than the max score.
- Excluded from this calculation are activity grades with both a raw score and max score of 0 points.
Grades data are synced from an integrated school’s LMS, and each LMS platform allows different activity types and grading methods.
Canvas Grades
- The calculation aggregates scaled grades for a student's activities recorded in Canvas courses within the lookback period.
- For online assignments, discussion topics, external tool assignments, and no-submission/paper assignments, Canvas applies a “unique” score policy when a student has multiple submissions/attempts. If a newer submission is graded, the newer score replaces the previous one, and the calculation takes into account the latest grade of any course activity that is received from Canvas.
- For graded quizzes and surveys, Canvas allows customizing the grading method (or scoring policy), to choose whether to use the highest score, latest score, or average score of all student’s attempts for an individual graded quiz/survey. In the current implementation, the academics metric calculation does not take into account the grading method, and will consider the grade for each individual student attempt, when there is more than one for a student’s graded quiz or survey.
Pivot Point LAB Grades
- The calculation aggregates scaled grades for a student's assessments recorded in Pivot Point Lab groups within the lookback period.
- Pivot Point LAB allows customizing the grading method (or scoring policy) for each assessment, to choose whether to use the student’s first attempt’s grade, last attempt’s grade, highest grade (the highest grade of all attempts), or average grade (the average grade for all attempts). In the current implementation, the academics metric calculation does not take into account the grading method, and will consider the grade for each student’s individual attempt, when there is more than one for an assessment.
Checklists
When enabled in cosmetology, allied health, nursing, trade schools, and other career programs, the Skills Tracker tool automates practical operations tracking and data entry. Students can track practical skills directly on their devices for instructor approval, and school admins can get a clearer view into students’ progress, supporting compliance and retention efforts. Tracking how a student is pacing in completing practical skills (checklist items) in their program can indicate they are falling behind and in need for intervention.
Calculated as a percentage, the checklists metric represents the ratio of a student’s actual checklist item submissions count-to-date, compared to their expected checklist item submissions count-to-date, given the time in their program. This metric is calculated for students with an ‘Active’ status in a Skills Tracker program; and both the start date and end date of the student’s program, which can be updated in the Program Info page, are required for the checklists metric calculation.
How Does a Student’s Checklists Performance Impact Their Risk Score?
- A student that has perfect performance for the checklists metric (a value of 100%) would have 0 points added to their risk score.
- If no threshold is set for a school/program, a student would accrue points at a 1:1 percentage basis for their performance in the checklists metric, which are added to their risk score.
- If a threshold is set for a school/program, every percentage point decrease in the checklists metric means that a student would accrue risk score points linearly from 99% until the risk factor threshold, and up to a maximum number of points determined by the risk factor's weight, which are added to their risk score. (See our article on weight and threshold definitions for more information and examples.)
How is the Checklists Metric Calculated?
The calculation of the checklists risk factor is as follows:
- Data for only one Skills Tracker program is accounted for, and determined by the student’s most recent program enrollment (based on the start date).
- A Skills Tracker program can have one or more checklists, each of them set with a minimum count of submissions (amount or hours) for each required item and flexible items (if applicable). Both required and flexible item counts in each of the checklists in a student’s Skills Tracker program are taken into account in the calculation; and both amount and hours are considered in the item submission counts. Checklists that are archived are ignored for the calculation.
- Checklists can be published in a Skills Tracker program at any time–even after students have already started working on it, and the updated count of expected checklist items (required and flexible) to be completed by students is taken into account for the calculation.
- The calculation of a student’s actual checklist items submissions count-to-date takes into account all submitted and approved (and non-archived) items, required and flexible.
- The calculation starts accounting for any student’s checklist item submissions after they are approved by an instructor/school admin; and ignores any student’s item submission that has not been reviewed/approved, has been rejected (not approved), or has been archived (even after being approved).
- The calculation ignores any item submissions in excess of the minimum count for each required item or the minimum flexible item count.
- A student’s anticipated time for completing a program (the end date minus the start date) is taken into account for calculating a student’s expected checklist item submissions count-to-date.
- The calculation assumes a consistent pace of daily progress, which uses an estimation of the student’s expected daily items count and the number of days the student has been in the program.
- The final checklist metric calculates a percentage for the ratio of a student’s actual checklist item submissions count-to-date compared to the expected checklist item submissions count-to-date.
- A ramp up period of 7 calendar days applies when a student is starting a Skills Tracker program, to avoid calculating a low checklists metric's value during that time.
-
- If the student is progressing at a faster pace than the expected daily items count, the checklist metric's value is calculated as 100%.
-
- A ramp up period of 7 calendar days applies when a student is starting a Skills Tracker program, to avoid calculating a low checklists metric's value during that time.
Example
The following example is to illustrate how a student’s actual checklist item submissions count-to-date and expected checklist item submissions count-to-date, and the checklists metric are calculated.
Let’s assume the following:
- Today’s date is 20-Feb-2023.
- The student’s program start date is 24-Oct-2022.
- The student’s program end date is 16-Apr-2024.
- The student has to submit a total of 2,000 checklist items to complete their program.
- This includes both the minimum counts for each required item in each of the program’s checklists, and the minimum counts for flexible items in each of the program’s checklists.
- The student has completed a total of 200 checklist item submissions to-date (actual checklist item submissions count-to-date).
- This includes approved and not-archived item submissions, for both required and flexible items, in each of the checklists for the student’s program.
Then, the calculation goes as follows:
- The student’s program duration is 540 days.
- This is the number of days between the student program’s start date, 24-Oct-2022; and the student program’s end date, 16-Apr-2024.
- The student has been in the program for 120 days.
- This is the number of days between the student program’s start date, 24-Oct-2022; and the current date, 20-Feb-2023.
- As of the current date, 20-Feb-2023, the student is expected to have completed 444 checklist items (expected checklist item submissions count-to-date).
- This is calculated assuming a consistent pace of daily progress, by dividing the total of 2,000 checklist items in the program by 540 days, which results in an expected daily checklist item count of 3.7. The expected daily checklist item count of 3.7 is multiplied by 120, which is the number of days the student has been in the program.
- The result is a checklist metric's value of 45%, as of the current date.
- This is calculated as a percentage for the ratio of the student’s 200 actual checklist item submissions count-to-date and the 444 expected checklist item submissions count-to-date.
Comments
0 comments
Please sign in to leave a comment.