Improving Online Test Administration and Analysis by Putting Student’s Needs First

The tradeoffs between test security, costs, and impingements on student rights have not reached a happy equilibrium with online test administration. On the contrary, wild swings in all the different directions have left none of the stakeholders in academic integrity satisfied.

Students are crying foul on the overreach into their privacy.

Educational institutions are concerned about costs and the affordability of their programs. Although they have been able to weather the cost swings of remote proctoring technology with the assistance of COVID-associated public funding, it’s unclear how affordable proctoring will be when that funding dries up.

Just how much security those sacrifices in money and privacy are buying remains in question. We’re a long way from the boring old equilibrium of yesteryear.

Getting from where we are now to a new normal where all the stakeholders in academic integrity are happy with the balance between costs, security, and privacy will require a fundamentally different approach to the issue.

The world has changed. Like all of us, students have access to almost limitless resources through the internet. Many of the techniques that instructors and institutions used to ensure the integrity of assessments have been made obsolete by the easy availability of communications technology. The apocryphal files of old tests and term papers maintained by fraternities have given way to third-party websites hosting vast test content databases. Resourceful students have access to millions of term papers and question bank items to tempt them to compromise their integrity.

Some schools’ faculty throw up their hands at the situation and have given up trying to administer timed and proctored exams. They see the attempt as fruitless in the face of the ubiquitous and ever-widening facility of online resources. Or they see remote test security as possible but only at prohibitive cost and an unacceptable level of intrusion into student privacy. Yet, the same technology that has enabled these serious threats to academic integrity has delivered the technological infrastructure to combat them and preserve the rights and privacy of students.

Students have unprecedented power. The runaway inflation in the cost of higher education and the advance of Internet technology have combined to open up the very tight cartel of higher education into a vast market of potential opportunities between in-person ivy-clad campuses, remote learning degrees, and a dizzying array of certifications, and nano degrees. It’s a buyers’ market.

The COVID pandemic has exposed higher education institutions to questions about equal access to education, admission policies, and the ultimate value of their services. Schools can’t rely on the inertia of traditionalism or their lingering prestige to justify academic policies around remote proctoring. When students question assaults on privacy, their questions need to be addressed. Student activism around remote proctoring has shut down proctoring programs at multiple school systems, prompted massive lawsuits, and invited scrutiny from the national legislature. This state of affairs raises the question, why not start talking to students and understand their concerns first? That’s what we have done at Rosalyn.

When Rosalyn reimagined online test administration and analysis, we started by looking at student’s needs first. Our product team empaneled the industry’s first Student Advisory Board (SAB) to advise us on the students’ experience. The SAB consists of nine undergraduate students from universities across the United States and the world. Collating the insights from the SAB with academic research and other student conversations has given Rosalyn unprecedented insight into students’ needs. These students’ needs are a principal factor in solving the equation that balances academic integrity with costs on educational institutions and student privacy.

1. Students’ attitudes towards cheating are somewhat contradictory. On the one hand, the vast majority of students want nothing to do with cheating. On the other, a large number have done it at one time or another. The decision to take unfair advantage during an exam is often situational and opportunistic. Otherwise honest students may justify their ethical lapses believing that their colleagues are cheating too. Only the most conscientious students will unilaterally abstain if they believe that their peers are cheating and not getting caught.
2. Students want test proctoring to work. They want to make sure nobody else is getting away with violating the test rules. They want a level playing field where professors reward hard work with a good score on the assessment and honors and recognition from their program and school.
3. They don’t want to be suspected of violating the test rules when they haven’t done anything wrong. Unfortunately, automated remote test proctoring has exposed innocent students to severe consequences of accusations of academic dishonesty. Students want to know that there are safeguards in place to protect against false accusations. They want to understand how proctors develop evidence of potential violations and how they can defend themselves if they are ever in that situation.
4. They don’t want an unknown stranger peeking into their dorm room or apartment and listening to them through the microphone. They don’t want this stranger judging them for their messy room or their taste in posters on their walls.
5. They want control over their person, their space, and their computer. Students give up a lot of personal agency to attend school. They put themselves under the tutelage of professors and graduate student instructors whose assessment will considerably influence their academic record and career. They subject themselves to the peer pressure of their classmates and conform to the culture of their “alma mater.” Their dorm room or apartment had been the one place where they could exercise the maximum amount of autonomy. The intrusion of video monitoring into the “safe space” of their bedroom is troublesome.
6. Costs are another issue for students. Higher education costs are already so out of whack that the additional burden of proctoring costs is most unwelcome.

One factor left out of the calculation on online test administration is how incredibly convenient it is. The old model of test centers and human proctors had added burdens for students. They had to schedule ahead of time for limited seating. They had to travel to and from a test center, and test fees included the overhead costs of the physical space.  

Online exams can be taken on-demand in the comfort of your home, with no scheduling or travel and test center costs. And digital virtual test delivery affords all kinds of benefits to test sponsors, such as the ease of creating multiple test With continual forms, using massive item banks, and dynamic item creation. These test design features increase content security and help ensure that tests are fair for everyone.

On the security side, online test administration and analysis are getting better and better. At Rosalyn, we continually refine our models to better discriminate between innocuous test-taking behavior and test violations. With continual improvement, we are better able to detect potential test violations and avoid flagging innocent behavior. This improvement in detection ability pays dividends in the deterrence of test violations. If students know they are likely to be caught, they’ll be far less tempted to compromise their integrity.

At the same time that we’re getting better, we’re becoming more efficient so that the costs of invigilating tests to a high level of security are going down over time. Because we use purpose-built AI models at Rosalyn, we can incorporate improvements continually as we accumulate more test session data.

But detecting test violations is just one part of the panoply of initiatives schools pursue to maintain academic integrity. Cultivating a community culture with shared values of mutual respect and integrity is another vital component.  

Rosalyn supports this culture as well by focusing on the student experience in online test administration. You may ask, “What can you do about that? Remote proctoring has to monitor students by video to work.” That’s right. We can’t eliminate monitoring, but we can make it more acceptable to students. We redesigned our onboarding experience to communicate better how our proctoring works and why we use particular security features. We strive to give students as much agency as possible in logging in and configuring their computers to take a test remotely.

We also strive for transparency throughout the test-taking experience. Letting students know what data will be collected, who will see it and how we’ll dispose of it afterward. This kind of open-book disclosure respects students’ privacy and directly addresses their stated concerns.

Rosalyn has made these improvements in the online test experience based on insights from conversations with Rosalyn’s SAB.  As we further develop our online proctoring solution, we will continue to listen closely to this vital constituency. Putting students’ needs first helps improve online test administration and analysis for all the stakeholders.