Satisfaction of the ophthalmology residents with online progress test

ABSTRACT Objective To investigate if there is knowledge improvement over the Residency years, as well as students’ satisfaction about progress test. Finally, to describe the implementation of the online progress test in some Ophthalmological Residency programs in Brazil. Methods It was an observational cross-section study. The participants were all Ophthalmology residents who accepted to join the study. They did an online test and answered demographic and satisfaction questionnaires. Results Two hundred and fifty-nine residents joined the study. 42,86% of the residents answered the demographic questionnaire and 6,95% answered the satisfaction questionnaire. In general, the residents approved the website as a tool for self-learning and to improve residency programs. Conclusion Over the years, the residents acquired knowledge during the residency program. In general, residents were satisfied with the test. The implementation of the online progress test system in the Ophthalmological Residency schools in Brazil was successful.


INTRODUCTION
Online assessment has been widely used to assess learners' knowledge across different fields including medical education.The main advantages of online assessment are the automated marking and instant feedback to students, (1) when compared to paper and pencil assessment.Most online assessment has the design of modular test in which each cohort of residents take a different test.However, this type of assessment has been criticized because they promote short-term memorization and are one-point measurement. (2)herefore, to benefit students' long-term retention, longitudinal testing has been suggested.The progress testing (PT) is the most known and widely used to assess knowledge growth.The PT allows to assess students' knowledge growth during medical training.In the PT, all learners simultaneously took the same test, following the same blueprint, irrespective of their year of training.The test is repeatedly taken during the course and it is based on a specific blueprint. (3)he advantages of the PT over the traditional methods of knowledge evaluation, such as the modular, intermediate or final tests, are that it forces students to study continuously (3) and allow to measure growth of knowledge over time. (4)rogress tests may be organized at a national and international level, which allows us to compare the results of students from different schools (2) and identify students at risk of failing (in qualifying examinations, for example).With this information, possible interventions during the course, like remediation, can be made.Previous studies have already showed students' satisfaction with the PT, considering it a tool that supports knowledge improvement and offers them academic support as future physicians. (5)he PT is already well-stablished in the undergraduate context.However, there is a lack of studies in the graduate context, (6) with few Residency programs using PT showing promising results.An online test in some Ophthalmology Services in Brazil was introduced following the principles of progress test. (3)his study aims to investigate if there is a difference between residents' knowledge in the different years of residency training and students' satisfaction on this new method of assessment.

METHODS
This research was an observational cross-sectional study conducted on an online platform.
Data were collected in 2020 and participants were ophthalmology residents.The participation was voluntary, and all participants signed a consent form.
The blueprint and test application were similar to the previous study, (8) but we used a new set of 125 questions using the Moodle® platform regarding clinical and surgical issues in ophthalmology.
Before the test, residents were asked to fill in a general questionnaire (year of residency, sex, age, previous ophthalmology residency elsewhere and if they were taking preparatory courses for the Ophthalmology Title Test).After the test, they answered a satisfaction questionnaire.Fields for open comments and suggestions were also provided.

Data analysis
Each correct answer received a value of 0.08 value (which corresponds to 10/125), attributing a total score varying from zero to ten.
Descriptive analysis using frequency tables for categorical variables and positions and dispersion measures for numeric variables were used.To compare the difference between years, we conducted the Kruskal-Wallis test, followed by the Dunn test.To compare the participants' knowledge growth, the Friedman test or the Wilcoxon test was conducted.
For the satisfaction questionnaire, we used the mean scores for each question.
Statistical significance level of 0.05 was used.Data were analyzed using the Statistical Analysis System (SAS) for Windows®, version 9.4 (SAS Institute Inc, 2002-2012, Cary, NC, USA).

RESULTS
In total, 259 residents from 13 services around Brazil joined the study: 47 (18.1%) were in the first year of residency, 104 (40.2%) were in the second year, and 108 (41.7%) were in the third year.Of those, only six took the test in 2018, 2019, and 2020.
In total, 111 (42.86%) residents answered the demographic questionnaire.From those, 58 were women and 53 were men, with a mean age of 31.Nine residents had already taken residency in ophthalmology elsewhere, and 43 were taking preparatory courses for the Ophthalmology Title test.
We had also applied the test in 2018, 2019 and 2021, but the number of residents who took the tests for the 3 years was too small (only six).Therefore, unfortunately, it was not possible to make an individual progress analysis.

Knowledge growth
The mean score ranged from 5.0 to 6.4 (Table 1).We found a significant difference between the first and second years and between the first and third years of residency (p-val-ue<0.0001),but no difference between the second and the third years (p-value =0,0619).
Previous studies investigating students' motivation on computer test found that they were stimulated and satisfied with online tests.They showed a positive attitude towards it, as in our study.The most positive aspects of online tests found were the autonomy, the faster feedback and the fact that online tests are more practically feasible. (1)ven though some of the residents considered the test too long and exhausting, we followed the guidelines for progress test and the same pattern of the specialist test.The aim was to introduce a test similar to the real specialist test and which could monitor residents' knowledge growth.Although computerized adaptative testing could decrease the number of items, the implication process is not easy to achieve since it needs support from a psychometrician and a delivery platform. (9)mitations of the study Due to the pandemic, the residents took the test at home; therefore, cheating was not controlled.Since the participation in the progress test was voluntary and not part of the official Residency program, we believe that residents had no reason to cheat.Unfortunately, the number of residents who answered the satisfaction questionnaire was very low, either because it was not compulsory, or because they were tired.Lastly, it was not possible to make an individual analysis of the progress since it was a cross-sectional study.The data collected were just from the year 2020.We had indeed applied the test in 2018, 2019, and 2021, but unfortunately the number of residents who took the tests for the 3 years was too small (only six).Further research should investigate whether this online assessment would differentiate resident's knowledge growth over time, as it is possible in undergraduate training. (10)

CONCLUSION
Progress test was able to demonstrate residents' knowledge growth, and the residents approved the new method of assessment.The implementation of the online tests in the Ophthalmological Residency schools in Brazil was successful.

Satisfaction questionnaire
Only 18 residents (6,95%) answered the satisfaction questionnaire.All mean scores obtained were high.The highest mean score was 10.0 for the validity of the site as a self-learning tool and for the validity of the site as a tool for the improvement of the residency programs, and the lowest mean score was 7.0 for the quality of the discussion of the questions (Table 2).There were only three comments: "The test was too long", "It was a good exercise for the brain" and "The test was too long and exhausting.It could be available longer, in order to allow people to do it at any other time".

DISCUSSION
The study could demonstrate that online tests can be used in Ophthalmology Residency training.This study demonstrated that, by following the principles of progress test, it was possible to differentiate residents' knowledge in different years of residency training, in line with previous studies that demonstrated the improvement of knowledge during residency training. (3,8)In general, residents showed a positive attitude towards this kind of test, although only 6.95% of the residents answered the satisfaction questionnaire.

Table 1 .
Descriptive analysis of the progress test scores for each year

Table 2 .
Scores of the satisfaction questionnaire