Objective Scoring in Immersive Simulation

When two educators supervise the same simulation, they can see completely different situations play out. Inherently, there’s no issue in that – even in practice, similar cases can vary greatly. However, when it comes time for assessment and evaluation of skill, objective scoring is key to ensuring interrater reliability and standardized care.

What is objective scoring in healthcare simulation?

Objective scoring, simply put, is a standardized, measurable way to assess the components of a competency in action.  

There are existing frameworks and tools available to conduct these assessments, like the Creighton Competency Evaluation Instrument (C-CEI)* or the Objective Structured Clinical Examination (OSCE).  

They serve to capture real-time data and bring a more objective approach to non-technical skills evaluation, like communication skills, and can bring a structure to assessing clinical judgment or reasoning.

Why is objective scoring important in healthcare education and training?

There a several ways learning and meeting objectives can be measured; two common methods are formative and summative assessment.  

While formative assessment occurs throughout a course or unit and serves to measure progress against objectives or identify areas for improvement from an educator perspective, summative assessment occurs following a course or at the end of an instructional period. This style of assessment most often evaluates learner knowledge or skills application and can be conducted via practical exams, written exams, and now, via immersive simulation.  

It’s largely agreed that objective scoring is necessary for summative skills assessment, the larger question is what it looks like and how it’s implemented.  

Educators are often equipped with criteria and checklists in traditional simulation which supports a form of objective scoring yet still allows for the educator’s expertise to drive feedback.  

However, small actions or inactions can, at times, be missed. The educator is typically responsible for supervising the whole of the interaction and may also be responsible for equipment management during the simulation. This limits their ability to focus attention on the learner and their actions, hindering the ability to provide a fully comprehensive objective score.  

The issues facing rater reliability are well-known. Even in studies evaluating simulation for high stakes assessment that included a standardized framework, there was disparity amongst evaluators.  

Missed actions can mean impacts to practice that may go unnoticed, which has the potential to lead to issues in patient experience and outcomes. It can also be true that actions that were taken by the learner are missed by the evaluator, showing up as a poorer performance from the learner.  

In practice, competency validation is a necessary part of onboarding or in a nurse residency or onboarding program, for example. Every new hire is measured against the same standard, whichever preceptor or educator is on shift.

In immersive simulation like virtual reality, every action is logged and timestamped, meaning no actions are missed and inactions are flagged automatically.  Each action can be mapped to the competency frameworks and clinical judgment models programs already assess against.

Immersive simulation for high stakes assessment

Virtual reality simulations are already being used to facilitate high stakes assessment with success from large nursing programs around the United States. Authors found that the use of immersive virtual reality simulation mapped to existing nursing competency frameworks were completed “without the bias or subjectivity inherent to clinical assessment.” Authors also reported that the AI-driven nature of the platform provided individualized assessments of the learners’ actions within the scenario, “demonstrating the potential for using (immersive virtual reality) to assess nursing competencies.”  

In the United Kingdom, the University of Lincoln is using VR to conduct OSCEs, with educators reporting that “In many ways, the student experience was enhanced”. Both the objectivity of the platform as well as the repeatability of the simulations provided more opportunities for practice ahead of the exams, with practice time correlating to overall exam performance.  Educators benefited from the implementation, as well, stating that the use of VR is allowing educators to spend “more time in areas to further better what we provide” and saving simulation staff over 200 hours of lab time.  

Virtual simulation is already providing value to both educators and learners as it relates to assessment. This method also allows for continuing to scale up high stakes assessment, particularly relevant for large cohorts or multi-site programs.  

Consistent scoring means an ability to reliably compare scores over time whether at the individual or cohort level. This means that educators can flag areas for remediation early when necessary and can see both a granular and birds’ eye level view of how an individual or cohort is progressing towards competency.  

Additionally, the logs of actions mapped against frameworks equips the facilitator with data and analytics to work from during a debrief or individual feedback session, giving the educator more cognitive space to focus on the clinical performance and clinical reasoning of the learner rather than on maintaining the administrative tasks associated with assessment.  

Both objective scoring and facilitator-led feedback can work in tandem and should be used to complement rather than replace for a comprehensive and meaningful learning experience.

How OMS approaches objective scoring

In the OMS platform, and as noted above, each action is logged and timestamped, then mapped to the components of a competency framework.  

Evidence-based, individualized feedback is then provided to the learner for an opportunity to self-reflect on the clinical performance of the scenario. Each learner receives a clinical performance score from the scenario, and the evidence-based feedback is linked to their specific actions and competencies.

The Insights Dashboard gives faculty and administrators a standardized view across learners. It includes a competency heatmap, cohort trends over time, strengths and areas for improvement in skills, and includes the ability to filter by learner, scenario, framework, and/or date.

This is already in practice – across a multi-site study focusing on multi-patient care in senior-level nursing students, the data and analytics provided by the OMS platform were used in assessing students’ clinical competence alongside the CCEI-C.

Final thoughts

In a perfect world, the same performance would equate to the same score. Objective scoring gives educators a more robust and data-led foundation for consequential decisions around education and training, with a clearer view across the whole program.

Readiness that can be measured consistently is readiness educators can act on.

Want to stay in the know on industry insights like this? Sign up for our monthly newsletter where you’ll get the latest updates on OMS plus industry research and insights straight to your inbox.

---

*The C-CEI is developed by Creighton University College of Nursing. OMS is not affiliated with, or endorsed by, Creighton University.