New ABMS Studies Show Benefits of MOC and Continuous Assessment

The American Board of Optometry’s (ABO) Maintenance of Certification program was developed in line with the American Board of Medical Specialties (ABMS) model. As our program evolves to enhance the value provided to both Diplomates and to the public, we continue to follow guidance and take cues from ABMS; in particular the American Board of Family Medicine, The American Board of Internal Medicine, The American Board of Pediatrics and the American Board of Ophthalmology. Our new CAP program encompasses many features these ABMS boards included in their pilot programs. While each board’s venture into continuous assessment looks somewhat different, the framework upon which they were built is shared by all.

Several recent studies have looked at the benefits and successes of continuous assessment programs. Known in ABMS terms as longitudinal assessment programs, new indications are that the transition to a more frequent, shorter, topic-focused assessment structure has improved formative knowledge outcomes.

The American Board of Obstetrics and Gynecology recently conducted an internal study to evaluate the effectiveness of journal article-based questions for retention of knowledge. Similar to ABO’s CAP assessments, ABOG Diplomates read assigned articles, respond to questions and must score at a certain threshold to demonstrate competence in the subject area. Question types are divided between understanding questions and application questions.

Using a sample of 344 Diplomates, each completed a series of article-based questions during January – March with access to the article. The series of questions was completed again in the summer without access to the articles. Diplomates’ scores were then compared by question type, using a t-test to determine if information from the original articles was retained.

Results showed that application questions had a high retention rate at 80.58%, while understanding questions yielded a 60.75% retention rate. This is important to note and underscores that a successful article-based, formative assessment needs to contain both question types. While understanding questions ensure comprehension of the article, application questions ensure the knowledge can be applied to practical situations. Overall, results from the study strongly support article-based learning for long-term knowledge retention.

In another study, the American Board of Family Medicine (ABFM) completed a survey analysis for the one-year mark of their longitudinal assessment pilot program. Very similar to ABO’s CAP assessments, ABFM Diplomates answer 25 questions each quarter. Three hundred questions must be answered to determine a pass/fail score over the course of four years. This gives Diplomates the option to take all questions in the first 3 years and have a fourth “free” year or spread the questions out over 4 years.

ABFM Diplomates were given the option to participate in this new pilot program; 71% selected this option and the remaining Diplomates chose to continue with the 1-day recertification exam during the tenth certification year.  Of those electing to participate in the longitudinal assessment pilot, over 95% indicated they were satisfied with their experience after one year. The top concern of pilot participants was the 5-minute time limit per question. However, an analysis of elapsed time per question revealed the average time to answer was 2 minutes and 21 seconds, indicating that time was not a significant factor.

The study also looked at whether the goal of delivering a formative assessment was being met and whether the longitudinal assessment performs comparably to the 1-day recertification exam, allowing a standard of expertise to be set for measured competency of board certification. Upon analysis of the participants’ learning experience, a majority of Diplomates indicated that in addition to learning as they went through the assessment, they were actually changing their practice based on new knowledge. A full 61% indicated they looked up information while answering questions, and 75% reported seeking additional knowledge on a related clinical topic after completing the assessment. While further study is needed, preliminary evidence indicates that providers are identifying knowledge gaps through assessments and then seeking targeted CE to fill those gaps. This is a key objective of formative assessments.

Finally, it seems clear after one year that item hierarchy—the difficulty of individual items compared with each other—is maintained in longitudinal assessment. And over the course of each 300-question cycle, the same breadth and depth of knowledge is measured as in the 1-day recertification exam. These factors allow for a comparable standard to be set between a 1-day exam and longitudinal assessment.

ABO conducted a survey in January 2020, one year after launching the CAP program. The findings from this survey are similar to those of the ABMS Boards’ studies. Changes to CAP were made based on our study including raising the allotted time per question to 10 minutes, making references more user friendly and increasing each CAP cycle from 3 years to 4 years, with year 4 being either a remediation year or a free year. As more research is done by both ABO and ABMS, we will continue to improve our program to ensure we are meeting our objective of providing a high-quality learning experience for Diplomates and a trusted measure of competency for the public we serve.