Weight & Threshold Definitions

Our student risk model allows schools to take into account five performance metrics (last activity recorded, attendance, lateness, academics, and checklists) currently, each of which is weighted before it is accounted for in the calculation of students’ daily risk scores.  The weights and thresholds are knobs that allow schools to fine-tune and adjust their evaluation of student performance metrics in the risk score model.

Here on the Settings page, a permissioned Admin can configure weights and thresholds, to ensure that the students’ risk score calculation is adjusted based on the use cases and expected standards of each school/program.

Weights

A weight represents the percentage contribution of any given performance metric in the model towards a student’s risk score. For example, a school might only consider a student's attendance, checklists and lateness performance, and weight them by 50%, 30% and 20% of the risk score respectively.

Note that the sum of the weights for the metrics accounted for in the calculation of the risk score must add up to 100% at all times.  Any metric that does not have an associated weight is not accounted for in the calculation of the risk score.  Also, if a given metric’s value is null (because there is no student’s data for calculating it), the calculated risk factor and its weight are removed from the weighted average when calculating the student’s final risk score.

Example: How Does Setting Weights Impact the Risk Score Calculation?

As an example, let’s assume the following:

• No thresholds are set (a 0% threshold for the attendance, academics or checklists metric means a student would accrue risk points at a 1:1 percentage points basis for their performance in any of the metrics).
• A program has weights for attendance, academics and checklists equal to 50%, 30%, and 20% respectively.
• Given that the student’s attendance, academics, and checklists metrics are 90%, 85% and 70% respectively:
  • The attendance, academics, and checklists risk factors are calculated as 10%, 15%, and 30% respectively.
  • After accounting for their weights, the computed risk factors for attendance, academics, and checklists are calculated as 5%, 3%, and 9% respectively. (Note that the computed value for any risk factor represents the number of percentage points that are contributed to the student's overall risk score for the day.)

Then, the student’s risk score is 17.

Thresholds

Thresholds can be set to reflect minimum standards for student performance, progress and/or graduation, or to be more aggressive (for instance, reflecting internal stretch-goal metrics) as makes sense for the school's preferred operational strategy.

For any given performance metric in the risk score model, a permissioned school administrator can set a threshold to reflect a minimum standard for students’ performance.  In the example above, no thresholds are set.  But, a student is usually way outside of satisfactory minimums, if for instance, they are only attending 40% of classes, or are late 68% of the time when they actually show up.  So, most schools will prefer to set thresholds in order to represent a higher bar for any risk factor to count fully towards the risk score.

How Does Setting Thresholds Impact Students' Risk Scores?

In the example above (where the attendance metric’s percentage is 90% for a student, with the attendance risk factor weighted at 50%), setting a threshold of 80% would mean the student receives half of weight of the attendance factor counted towards their risk score, or 25 percentage points (as opposed to just 5 percentage points above, when the threshold was not in effect).  This is because 90% is halfway between 100% and the threshold value of 80%. Then, this adds an additional 20 percentage points to the student’s risk score, which increases to 37.

Now consider a different sample scenario, for which the five performance metrics are in use for calculating a students’ risk scores in a program, with weights set at 20% for each metric and thresholds set as noted below.

• Lateness - 10% threshold:
  • The lateness metric is a percentage that represents the ratio of a student’s late check-ins compared to all their check-ins (late and on-time) in all synchronous sessions the student has attended in their courses.
  • A student that has perfect performance for the lateness metric (a value of 0%) would have a lateness factor of 0%, and 0 percentage points added to their risk score. 
  • Every percentage point increase in the lateness factor, all the way up to the 10% threshold, would accrue percentage points linearly up to a maximum of 20 points added to a student’s risk score (given the 20% weight).

• Days to Last Activity (Last Activity Recorded) - 14 days threshold:
  • Days to last activity represents the number of days since a student’s last activity recorded.
  • A student with 0 for days to last activity (a last activity recorded value of yesterday), would have 0 points added to their risk score. 
  • Every day of inactivity beyond yesterday, all the way up to the 14 days threshold, would accrue percentage points linearly up to a maximum of 20 points added to a student’s risk score (given the 20% weight).
    
    
• Attendance - 80% threshold (first example, reiterated for completeness):
  
  • The attendance metric represents a student’s attendance versus the total expected attendance time in synchronous attendance sessions in their courses.
  • A student that has perfect performance for the attendance metric (a value of 100%) would have an attendance factor of 0%, and 0 points added to their risk score.
  • Every percentage point decrease in the attendance factor would mean that a student accrues risk score points linearly from 99% until it falls to the 80% threshold, at which point they would have the maximum 20 points added to their risk score (given the 20% weight).
    
    
• Academics - 80% threshold (requires an LMS integration):
  • The academics metric represents a student’s attendance time performance versus the total expected attendance time in synchronous attendance sessions in their courses.
  • A student that has perfect performance for the academics metric (a value of 100%) would have an academics factor of 0%, and 0 percentage points added to their risk score.
  • Every percentage point decrease in the academics factor would mean that a student accrues risk score points linearly from 99% until it falls to the 80% threshold, at which point they would have the maximum 20 points added to their risk score (given the 20% weight).
    
    
• Checklists - 90% threshold (requires Skills Tracker):
  
  • The checklists metric represents the ratio of a student’s actual checklist item submissions count-to-date, compared to their expected checklist item submissions count-to-date, given the time in their program.
  • A student that has perfect performance for the checklists metric (a value of 100%) would have a checklists factor of 0%, and 0 points added to their risk score.
  • Every percentage point decrease in the checklists factor would mean that a student accrues risk score points linearly from 99% until it falls to the 90% threshold, at which point they would have the maximum 20 points added to their risk score (given the 20% weight).

See our article on student performance metric and risk factor definitions for more detail on calculation considerations and methods for each.