Retention Workspace

The Retention Workspace gives school admins, instructors, and staff visibility into student engagement in their educational experience, so they can identify at-risk students, prioritize interventions, automate relevant outreach activities, and optimize student retention/success efforts.  Note that the Retention Workspace is an upgraded version of the Retention Dashboard with a richer, more compelling experience within the suite of Cortex offerings

The Retention Workspace provides a real-time view of students’ performance across five critical metrics, and combines them into one risk score, with varying levels of severity, to identify students that are at risk of dropping out. The five student performance metrics are last activity recorded (days to last activity), attendance, lateness, academics, and checklists. These metrics serve as indicators of a student's performance in critical areas of their educational experience, and factors to consider when calculating a student’s risk of dropping out.

A risk score, calculated daily for each active student enrolled in every program at a school location/campus is to serve as an initial indicator that a student is trending on or off-track in their academic journey. Using this data, permissioned Admin users can also set up automations to help track and prioritize relevant student outreach activities and interventions. (Note that in the current implementation, students cannot see their risk scores.)

In the Retention Workspace you can:

• Use pulldown filters to refine the selection of students shown in the dashboard.
• View a summary of Students per Risk Level, for the counts of students grouped by risk level.
• View the Students Risk Trend, for a time-based distribution of students grouped by risk level.
• View the Risk Score By Student, for a list of students, or an individual student, per the pulldown filters' selection, including their latest risk score and performance metric values.
• Access Insights, for exploring and drilling down further on student performance and retention rates data.
• Setting up Automations for tracking and prioritizing relevant outreach activities and interventions. (Only accessible to permissioned Admin users.)  
• Configure the Settings for calculating students' risk scores based on a school or program's criteria, and daily alerts. (Only accessible to permissioned Admin users.)
  
  

Accessing the Retention Workspace

The Retention Workspace is available to all users who have Dashboard permissions on the CourseKey portal, and can be accessed by clicking on Retention, under Workspaces, in the left navigation menu; or the Retention tile in the Home page. The Retention Workspace, like any other workspace within the suite of Cortex offerings, is accessible by permissioned Admin users through the Workspaces pulldown menu on the left navigation bar; and the Retention tile in the Home page. The following user's permissions are required to access the Retention Workspace:

• School is Cortex enabled
• User has view/edit Workspace permissions

After clicking Retention in the left navigation, users land on the Overview page by default.  

Contact your Customer Success Manager or submit a support request if access is missing or you would like to learn more.

 

Overview

When first accessing the Retention Workspace, users land in the Overview page.  Within the Retention Workspace, the Overview page can be accessed using the top navigation bar.  

The Overview page includes a set of pulldown filters, a Summary section with the Students per Risk Level visual and the Student Risk Trend visual, and the detailed Risk Score by Student table.

• The list of programs available in the Retention Workspace is populated by either the list of manually created programs uploaded in CourseKey (for non-SIS-integrated institutions using the Skills Tracker tool), or directly through a SIS integration.  Only programs with active student enrollments/members are included.
  • When there are no programs configured for a school, the calculation of students’ risk scores is made for a default school program. Note that the school default program is shown as an option in the Program pulldown filter only when there are no other programs available in a school’s dataset.
  • The student's most recent program enrollment (based on the student program enrollment’s start date, or when unknown, the student program enrollment’s creation date) is the one the risk score calculation is based on.
  • If programs are available for an institution, weights & thresholds for the risk score calculation can be customized individually for each program at the school.  This can be done by permissioned Admin users in Settings.
• Students with an active school enrollment status only (‘Active’, ‘Probation’, or ‘LOA’ for SIS-integrated institutions; and ‘Active’ and ‘LOA’ for non-SIS-integrated institutions using the Skills Tracker tool), are considered for the risk score calculation, and hence, included in tables, graphs, and insights within the Retention Workspace.
  • For non-SIS-integrated institutions using the Skills Tracker tool, students’ program enrollment/member associations can be maintained in the Program Info page, where school admins can update the status, start date and end date of a student’s program enrollment.
  • If an active student does not have an enrollment/member association to a program (created from a SIS integration, or manually using the Skills Tracker tool), the student is associated with a default school program.
  • When the start date of a student’s program enrollment is not known, the current date minus 365 days is set as the start date for the purpose of calculating the risk score.
• Students’ risk scores are calculated daily, taking into account data from the start date of each student’s program enrollment until the previous day.  Should data change for dates prior (for instance, an adjustment to attendance a week ago), prior days’ scores do not change, only the new daily score accounts for new/updated data.
  • The Retention Overview Page shows data on students’ performance metrics and risk scores that are calculated daily, in the early morning hours, and which account for any student activity up to the previous day.  In the case of a delay on the data pipeline processing on any given day, the Retention Overview Page will show the most recent data that is available for students’ performance metrics and risk scores.
  • A student’s risk score is calculated for a given day only if the student is in an active school enrollment status, and has had an activity (per the last activity recorded) within the last 60 days.
  • For the risk score to be calculated, in addition to having had an activity within the last 60 days, a student needs to have an active enrollment and attendance recorded in an active course, grades data synced from an integrated school’s LMS, and/or checklist items submissions in a Skills Tracker program.
  • Students' performance data is tracked from their program enrollment's start date (or if not known, over the last 365 days).
  • If a given performance metric’s value is null (because there is no student’s data for calculating it), it is ignored when calculating the student’s risk score.
• In the Overview page, students are separated into four risk levels – low, medium, high, and severe, for easier viewing and triage. (Numeric ranges for the risk levels can be configured by a permissioned Admin user in Settings.)

Pulldown Filters

A set of pulldown filters can be used to refine the selection of students shown in the dashboard. 

These pulldown filters are cascading, with each selection constraining options for the next set of filters.

• Schools: A list of all school locations/campuses for an institution.  
  • The selection for schools constraints the list of options for the Programs, Courses, and Instructors filters.
  • Note that in the current implementation, students with records in multiple school locations/campuses are only shown for the school location/campus of their most recent program enrollment.
• Programs: A list of programs offered at the school(s) selected in the Schools filter. 
  • Note that a school default program is shown as an option when there are no other programs available in a school’s dataset.
• Courses: A list of courses specific to the school(s) selected in the Schools filter.
• Instructors: A list of instructors for courses specific to the school(s) selected in the Schools filter.
  • The Courses and Instructors pulldown filters are additive–the selection of students will include those that match any of the selected course(s) or instructor(s).
• Statuses: A list of school enrollment statuses for active students, including Active, Probation, and LOA.  (For non-integrated schools, only the Active status is applicable.)
• Risk Levels: A list of the levels (severe, high, medium, low) for grouping students based on their risk score.

You can select multiple parameters and click Apply to narrow the student population you wish to look at.  To start over at any time, you can click the reset icon (circular arrows) on the right. 

Note that identical name values in pulldown filters (such as programs and instructors) are combined into one listing (even if the underlying IDs vary) for readability.

Students Per Risk Level

The Students Per Risk Level donut chart shows the counts and the corresponding percentages for the selection of students, grouped by risk level: low, medium, high, and severe. The population of students represented in the graph include those who match the criteria per the pulldown filters' selection, and for whom a risk score was calculated on the previous day.

Students Risk Trend

The Student Risk Trend stacked area bar chart shows a time-based distribution of students (counts), grouped by risk level: low, medium, high, and severe.  

• The selection of students included in this graph includes those who match the criteria per the pulldown filters' selection, except for Statuses.
• The graph shows data representing students’ risk scores calculated within the date range ending on the previous day.
• The daily counts include students for which a risk score was calculated within the past 365 days, and independently of their current school status.
• You can adjust the graph’s time interval (1 week, 1 month, 3 months, 6 months, 1 year) by clicking on the pulldown on the top right of this graph.

Risk Score By Student

The Risk Score By Student table depicts the actual population of students per the pulldown filters’ selection, including their current risk score and their percentage points for the performance metrics – attendance, lateness, last activity recorded, academics, and checklists.  

• • The population of students in the table includes those who match the criteria for the pulldown filters’ selection, and for whom a risk score was calculated on the previous day.
  • You can use the search bar above the table to look for students based on their first name and/or last name; or sort the list of students in ascending or descending order based on the values (percentage points) of the risk score or any of the performance metrics.
  • You can click any student’s Name within the table to view additional details, including the student’s profile information; performance metrics/risk factors and risk score, and risk score trend. 
  • There are buttons above the table for selecting whether to show the Actual values for performance metrics or the Computed values for risk factors that contribute to the students’ risk scores. Note that when you first access or reload the Retention Overview page, the table shows the actual performance metric values by default.
  • There are controls below the table for selecting the number of students to show per page, and to navigate across pages.
  • The Export button above the table is to download a list of the students, including all rows and columns, in .csv format.

Showing the Actual Values for Students’ Performance Metrics

When the Actual button above the Risk Score By Student table is selected (which is the default option), the data shown in the visualization is for the actual values for students’ performance metrics.  See our article on student performance metric and risk factor definitions to learn more about how performance metrics are calculated.

• If there are no data available for calculating any of the performance metrics for a student, two dashes (‘--’) are shown.  This may occur for a student with no data recorded to date (e.g. for synchronous attendance, course grades, or checklists progress) when the corresponding performance metric has an assigned weight – and is accounted for in the calculation of students’ risk scores.

 

Showing the Computed Values for Students’ Risk Factors

When the Computed button above the Risk Score By Student table is selected, the data shown in the visualization is for the computed values for risk factors that contribute to the students’ overall risk score. See our article on weight and threshold definitions to learn more about how the computed value for a risk score factor is calculated.

• The computed value for any risk factor represents the number of percentage points that are contributed to the student's overall risk score for the day.
• It is possible for the value of any performance metric that is accounted for in the calculation of a student’s risk score (with a weight assigned) to be null – when there is no data recorded to date for calculating it. In such cases, for calculating the student’s risk score, we ignore that particular performance metric and scale up the weights of the remaining metrics so that they add up to a total of 100%. Note that the corresponding risk factor for a performance metric with a null value shows with a value of 0 for the computed score.
• It is also possible for a student's computed scores for the risk factors not to add up to their overall risk score. This is because to simplify the visualization, we round down computed scores that result in a decimal number to the nearest whole number. When this happens for any student in your selection, you will see a message/link: “Why aren't a student's computed scores for the risk factors adding up to the overall risk score?”.

Viewing an Individual Student’s Detailed Data

When you click any student’s Name within the Risk Score By Student table, you are shown a page with a detailed view of the student risk score, along with information from the student’s profile – including the program, email address, student enrollment status, schools in which they are taking courses, start date and graduation date.  

• The Student Risk Score visualization displays the student’s actual and computed risk scores for the previous day. There are buttons for selecting whether to show the Actual values for performance metrics or the Computed values for risk factors that contribute to the student’s risk score. See details on considerations for the calculation of computed values for students’ risk factors (details discussed above).
• You may click the information button for a quick overview of settings for weights and thresholds for the program in which the student is enrolled, and which are used when calculating the student’s computed risk factors.
• The Risk Score Trend stacked area bar chart shows a view on how the student’s risk score has changed over time. This graph shows data representing the student’s risk scores calculated within the date range ending on the previous day and up to 365 days in the past. You can adjust the graph’s time interval (1 week, 1 month, 3 months, 6 months, 1 year) by clicking on the pulldown on the top right of this graph.
• You may click the View Courses button on the top right of this page to go to the student’s Courses page, to further review details on their course engagement and performance.

 

Insights

Beyond the Overview dashboard's top level student performance metrics and risk score, permissioned Admin users are able to review a number of next-level insights that help them drill down further than the Overview, to better understand how student performance and retention rates are tracking over time, and by cohorts and are filter-able (where appropriate to the given Insight) by date range, campus, student enrollment status, program status and course name. 

Retention Insights comes initially with a few templated Insights out of the box, which can be downloaded to CSV or XLSX. Other customizations of these Insights can be discussed with your Client Success Manager. Default-available Retention Insights include:

• LAR visualization and drilldown table: Showing a breakdown of students' Last Activity Recorded by ranges of days. Updated daily.
• Absent Now visualization: Showing counts of students absent now from synchronous courses in session, broken down by course. Note that the data displayed may be up to 60 minutes old since the last refresh.  Admins interested in a more immediate view may wish to review the live Attendance Dashboard. 
• Attendance Percentage in Course to Date table: Student-level breakdown of course attendance.
• Checklist Submission Trend (bar chart and table - requires Skills Tracker): Showing average percentage of optional and required approved items vs. total possible, broken down by program code and name. SIS ID is required for display, and note optional-item completions may cause percentages to exceed 100%.
• Academic Percentage in Course to Date table (Canvas or PivotPoint - requires an LMS integration): Student-level breakdown of average grade performance out of 100% possible points for all grades available to CourseKey, regardless of grading method. Grades evaluations over 100% of possible grade are maxed out to 100%.
  
  

Automations

Permissioned Admin users can set up automations to help track and prioritize relevant outreach activities and interventions. 

Automations include:

• Course Attendance Threshold - Second Absence
• Academic Minimum Threshold Alert for Students (requires an LMS integration)
• Academic Minimum Threshold Alert for Staff (requires an LMS integration)
• Last Activity Recorded - 3-Day Warning
• Last Activity Recorded - 7-Day SMS Alert
• Last Activity Recorded - 7-Day Push Alert
• New Severe-Risk Students Daily Alert
  • Note that this is not in the Automations List – accessible via Retention Overview Settings
    
    

Settings 

To access the Settings page, click on Settings in the top navigation bar. This page is accessible to permissioned Admin users, to configure the Retention Dashboard such that it more closely matches parameters, weights and thresholds for calculating students’ risk score for their schools/programs.

Risk Factors

As part of their custom configuration, each institution has the option of selecting performance metrics/factors to be considered for the calculation of their students’ risk scores. For each individual school, permissioned Admin users can select any of the five performance metrics/risk factors available currently, to calculate daily percentage points for different aspects of a student’s performance. See our article on student performance metric and risk factor definitions to learn more.

• Only selected performance metrics/risk factors show as columns in the Risk Score By Student table (Actual and Computed views). Note that when more than one school is selected in the pulldown filters and you are viewing data for students across schools and programs, you may see columns in the Risk Score By Student table for any performance metric/risk factor for which at least one student has a value for the performance metric that is not null.
• Adjustments to weights and thresholds can only be made to performance metrics/risk factors that are selected in this page. Note that in the current implementation, we are not blocking you from unselecting a performance metric/risk factor when it has an associated weight. Before you proceed with unselecting a performance metric/risk factor, to not account for it in the calculation of students’ risk scores for a school, you want to make sure to set its weight to a value of 0. Not doing so can lead to accounting for that performance metric as a risk factor for calculating a student’s risk score, if there is data available in the system to calculate it.

Weights & Thresholds

The weights and thresholds are available to ensure that the calculation of students’ risk scores are adjusted based on each school or program's use cases and expected standards for student evaluation.  Permissioned Admin users can adjust the performance metrics' weights and thresholds to use for calculating students’ daily risk scores.  See our article on weight and threshold definitions to learn more.

• Adjustments to weights and thresholds can only be made to performance metrics/risk factors that are selected in the Risk Factors page.

Risk Levels

Risk levels are used to flag and group students into four categories – based on their daily risk scores, for the purpose of prioritizing communications and interventions, and optimizing student retention/success activities. Permissioned Admin users can set the minimum score for each risk level, for each school location/campus or program.

• The numeric settings describe the minimum score on a scale of 0-100 that qualifies a student for that risk level.
• Numeric levels should be set in descending order, such that the severe level’s minimum risk score is a greater number than high, high is greater than medium, and medium is greater than low (which by default is set to start at 0).

Daily Alerts

Permissioned Admin users can configure daily alerts, to send an email containing any new students at severe risk to selected staff members each morning.

• Students’ performance metrics and the risk score are calculated daily.  To have staff receive updates by email overnight, with the resulting list of students who are newly elevated to the severe risk level, toggle the switch to ON, and select the staff who should receive this email.
• The available email addresses in the list are limited to CourseKey users who have at least one staff or administrative permission set (non-student users).