Future of Testing: Can You Draft a Question Requiring Skills Beyond Recall?

BottomLineopenwebtestOver the past year, I have had the opportunity to hear about advances and issues facing our profession from pediatricians from all across the country. Your thoughtful comments to my last post provide an even greater wealth of insights into Maintenance of Certification in general and testing in particular.

Many agreed that an “open-Internet” approach to testing is important to consider, and many recognized the complexity of such a change. An overarching theme in the responses was the purpose of the test, which is to assess knowledge as an integral component of medical practice. Many of the comments echoed my earlier assertion that technology-aided retrieval of information has become reality – arguably a necessity – in everyday practice. Conversely, several of you pointed out there is not enough time to look up everything in daily practice and worried whether today’s residents were becoming too dependent on the smartphone.

Your comments raised other pertinent questions:

  • If we allowed access to the Internet during tests, could we agree on selected resources and websites that should be accessible (such as Red Book, ePocrates, UpToDate)?
  • Is the use of the Internet appropriate for initial certification as well as Maintenance of Certification?
  • Could we allow an open-Internet test if we could not guarantee that test-takers are prevented from copying and emailing test questions or entering chat rooms?

Right now, the leading test delivery providers are researching the best ways to allow for open-Internet testing while still ensuring the integrity of the examination process, but we’re not there yet.

Assuming that the security challenges can be resolved, an even more important issue is the kinds of questions that would be appropriate for a section of the exam that included Internet access to certain resources. We would want questions that do not simply test recall, but also focus on analytic thinking, data synthesis, evidence assessment, judgment and decision-making.

Our subboards and question writing committees are made up of board-certified pediatricians who see the same kinds of patients as any practicing pediatrician. They work extremely hard at writing fair and valid questions that require higher-level thinking and critical reasoning skills. For now, we will continue to use closed-book, multiple-choice examinations, which offer the most objective methods for scoring. If, at some point in the future, we incorporate Internet-access during examinations, we would need an expanded pool of questions that examine critical reasoning skills.

We invite you to join the effort by drafting sample questions that you think would assess the higher order thinking skills that define a good pediatrician — even if he or she can look up information before making decisions about management or treatment. Our shared goal should be an examination that mirrors to the greatest degree possible the increasingly complex world in which we practice. And as one commenter suggested, physicians don’t practice in a “closed-book world.”

Before you get started, I strongly encourage you to look at some of the resources used by our question writing committees: one is an overview of Bloom’s taxonomy; the other is an excellent tutorial developed by the National Board of Medical Examiners (NBME) on how to write good questions.

You may also want to beta test your question(s) with your colleagues before you submit them. We look forward to seeing the kinds of questions you come up with.

SubmitOpenWebQuestionBloomsTaxWe will share your submissions with our subboards and question writing committees as they wrestle with the same challenges. You shouldn’t expect to recognize questions that you submit on a future examination. All questions undergo multiple rounds of thorough review and careful editing.

In fact, we have always accepted questions from diplomates, and they all go through the same process. You can find our item writing portal through your portfolio on the ABP website. For now, and for ease of use, we created a simpler method for you to share your submissions, accessed through the teal button.

Realistically, developing new questions and designing new exams takes time. We have begun a rigorous discussion about testing, and we are reviewing the best thinking from testing experts as well as medical educators and expert clinical practitioners. In the meantime, we welcome your input as we explore this important issue.SubmitOpenWebQuestion

So give question-writing a try. As you do, I hope you will also continue to share your ideas about what makes a good question. I invite you to use the comment section of this blog to contribute to the discussion. As always, I look forward to reading your insights and thoughtful suggestions.

Certification Exams: Are We Testing Recall or Judgment?


I have a recurring nightmare: I’m a sophomore in college, taking an engineering exam on differential equations that I don’t understand at all. It’s an open book test, and I still have no idea what to do! Somehow, in my dream, I miraculously pass the test, but know with certainty that I should never take another engineering course, just for fun, ever again. So, I became a doctor.

Fast-forward 30 years and I am making rounds with residents and fellows. The computer on wheels (COW) accesses orders, EMR entries, and PubMed. The residents use tablets and smart phones to log in to sites that provide practice guidelines, differential diagnoses, and more. ABP exams have evolved from oral exams in front of a daunting panel of learned practitioners, to unproctored exams on floppy computer disks to today’s proctored, computer-based exams. Real time internet access for clinical care prompts the question: Should my old college open-book exam be reincarnated as an ABP open-internet exam?

Unfortunately, there is precious little research to guide answers to this question. Overall performance on open-book/internet exams does not seem to change much, with those who benefit from the approach offsetting a population of examinees who become more confused by it.  Historically, data on the effects of open-book exams reveal conflicting results on whether the open-book (and presumably the open-internet) exam encourages or hinders learning.

There are also philosophical disagreements on the role of recall in testing. A closed-book exam requires test-takers to recall facts, which are then used toquote212-10 address the diagnosis, treatment, and management of a given clinical scenario. Proponents of the closed-book exam argue that the practice of medicine requires a certain amount of “pocket knowledge” that every practitioner must have to deliver excellent care. Opponents of the closed-book exam chafe at this requirement for fact recall precisely because technology-aided retrieval of information has become reality – arguably a necessity – in everyday practice. It is increasingly unrealistic to expect physicians to recall information the way they did in the past as the body of knowledge in medicine expands exponentially. Regardless of one’s philosophical perspectives, the current closed-book exam design has produced ample psychometric data in support of its fairness, validity, and reliability.

quote12-10If the ABP and other certifying boards were to shift to an open-internet approach, a substantial amount of preparatory work and pilot testing would be needed. Exam questions would presume access to facts and focus on assessing critical thinking, effective search strategies, decision skills, and complex data analysis. Such questions might have more than one correct answer, and the correct answers might be weighted differently such that one of the correct answers might be considered “more correct” than another. Thorough psychometric analysis would be required to document the validity and reliability of such an approach before it could become the basis for certification.

An open-internet exam would still be timed. So, just as in the clinic, there would be insufficient time to look up every imaginable fact. Some examinees might even feel greater pressure with the combination of access to resources and time limits.

Predictably, open access to the internet carries a host of security concerns, which would have to be solved before this approach becomes viable. The June 3, 2012 issue of The Chronicle of Higher Education (“Online Classes See Cheating Go High-Tech” by Jeffrey R. Young, documents the technological arms race between innovators in online education and those intent on using holes in the online systems to cheat. Robust exam security will be necessary to ensure that everyone taking the exam has an equally fair chance at passage.

Finally, economic concerns have been raised. Although many or most pediatricians have already purchased subscriptions to online databases and information resources, this is not true for all. The ABP would want to avoid any notion that purchasing such a subscription was “required” in order to pass the test.

The ABP is thinking hard about these issues and will conduct the necessary research to evaluate open-internet testing. In the meantime, I encourage you to let us know what you think. Please leave a comment below.

Parents, Sick Children, and the Network

BottomLineJustin Vandergrift and his wife’s lives changed abruptly in August of 2011, when they rushed their daughter Kathryn to a North Carolina pediatrician.

Eleven-year-old Kathryn had just been diagnosed with Crohn Disease.

At a follow-up appointment a week later, the Vandergrifts were offered the opportunity to join ImproveCareNow Network.

The ImproveCareNow collaborative, uniting 50 teams, focuses on two specific efforts: to develop a Collaborative Chronic Care Network, and to support a data collection system to help drive improvements in care for children with Crohn disease and ulcerative colitis.

Kathryn Vandergrift

Kathryn Vandergrift

Learn more about the ImproveCareNow Network.

A nurse practitioner explained to Kathryn’s parents that information about how she was doing could be shared anonymously with others participating in the ImproveCareNow Network so everyone in the network could help identify best practices to improve and standardize care.

“I signed that consent form,” Justin said. “I knew they were doing things behind the scenes.”

Justin later traveled with his local site team to an ImproveCareNow learning session, which brought together teams from around the country involved in the network.  Each team included physicians, nursing staff, researchers, and parents – all sitting at the table together, sharing their invaluable perspectives.

“I really came back from the learning session knowing that the physicians actually cared about what our opinion was,” Justin said.

Dr. Ashish Shah, Arizona Pediatric Cardiology Consultants

Dr. Ashish Shah, Arizona Pediatric Cardiology Consultants

Dr. Ashish B. Shah, an ABP-certified pediatric cardiologist at Arizona Pediatric Cardiology Consultants, said the importance of patient involvement is one of the most influential lessons he’s learned since becoming active in his network, the National Pediatric Cardiology Quality Improvement Collaborative.

“Incorporating a diverse group of families enhances the collaborative process,” Dr. Shah said. “These are the people you are caring for. And if you’re not communicating effectively with that group, then you cannot expect to be successful in caring for the patient.”

The National Pediatric Cardiology Quality Improvement Collaborative focuses on improving outcomes for children with cardiovascular disease.

Learn more about the National Pediatric Cardiology Quality Improvement Collaborative, including participating sites.

Having completed a dual medical/business degree while in medical school, Dr. Shah recognizes the value in standardizing care.

“What if the way you’ve been doing it all along isn’t the best way?” Shah said. “In a collaborative, what can you learn from other places, what can you learn from your own internal failures to do better? Maybe we’re doing something really good and we don’t know it, but now we can find out because we can share our data, ideas, and processes.”

This is particularly valuable when the number of specific cases is limited. By uniting different sites throughout the country, networks are able to “experience” more cases, share more data, and learn how to improve the care they deliver. This knowledge is then diffused throughout the participating network sites.

Justin, like other parents and caretakers, also sees how the work of the collaborative impacts Kathryn directly.

“All of these people at all these care centers are inputting data into this database, and it all comes down to a point,” Justin said. “You’re getting this huge funnel of information and knowledge sent directly down to my child when she needs it.”

“That funnel of knowledge, I think, is remarkable,” Justin said. “I get all this knowledge poured through the data down to my child. It’s remarkable to see that standard of care.”

Graphic courtesy of http://www.karrcreative.com.

This is only achieved by each team’s willingness to remain transparent in sharing both their failures and successes.

“You learn that the process you may be accustomed to is not the best,” Dr. Shah said. “However, by taking bits and pieces of successful processes, you can improve upon your existing system.”

Teams must also be willing to resist the urge to hoard knowledge and expertise, recognizing that the ultimate goal is to improving the care and treatment of children in a more collaborative manner. It is a cultural shift taking place in what has always been an extremely competitive field.

The third and possibly most influential necessity is the parent-patient involvement.

“Parent involvement is huge,” Dr. Shah said. “You can say anything as the doctor, but if the family cannot do it or doesn’t buy into it, you achieve nothing.”

Collaborative networks help not only by bringing the family to the table in a conversation with the medical team but also by offering a sense of community for the patients and caretakers.

Since his daughter’s diagnosis, Justin has taken an active, engaged role in Kathryn’s care. And he has encouraged other parents in his situation to seek out and become involved in these networks.

“The sharing of data to standardize care is so important. I need to know as much as possible,” Justin said. “The only way to do that is to get involved in some organizations like the collaborative.”

Dr. Shah shares his passion for understanding the value of these networks.

“Your baby is part of a network of similar babies across the country,” Shah said. “We’d like you to participate because what happens for you and your baby becomes a learning experience for us so we can…improve quality of life.”

The American Board of Pediatrics (ABP) helped support the development and launch of the ImproveCareNow and National Pediatric Cardiology Quality Improvement Collaborative Networks, as well as other similar learning networks.

Through these learning networks, diplomates of the ABP can engage in collaborative quality-improvement efforts while also earning MOC credit.

If you are a diplomate of the ABP interested in earning Part 4 activity points through participation in collaborative networks, please browse the multiple individual and team-based collaborative network activities available by visiting your ABP Portfolio and using the Part 2 and Part 4 Activity Search.

This is what I think. If you would like to let me know what you think, please leave a comment below.

The Patient, the Pediatrician, and the System

bottomlineblog3The Fateful Call

“At 9:30 am last December 14, a fateful phone call interrupted the day of Newtown Connecticut pediatrician Laura Nowacki, MD, FAAP”, reported the July edition of AAP News.1,2  “It was her office nurse calling to say a shooting just occurred at Dr. Nowacki’s daughter’s school. They needed to triage.” While the pediatricians in Newtown mobilized for emergency care on December 14, their subsequent leadership role in violence prevention, media engagement, and mental health care highlights the important determinants of child health that go beyond an office visit—they engaged a “system”. This blog entry explores some key questions surrounding systems-based practice.

What are Systems and Systems-based Practice?

The scientific concept of a “system” is based on the work of Bertalanffy3, 4 and can be thought of as a set of interdependent elements that collectively act as a whole to carry out a specific function.3 Systems thinking focuses on the properties of the whole (holism) and the interactions of the elements rather than the properties of the constituent elements (reductionism). Having been applied to biology, engineering, climatology, and many other disciplines, the systems approach is now making its way firmly into the clinical care of children.

Learning systems-based practice (SBP) begins in medical school and residency. The Accreditation Council for Graduate Medical Education (ACGME) program requirements have defined (SBP) as one of the six core competencies: “Residents must demonstrate an awareness of and responsiveness to the larger context and system of health care, as well as the ability to call effectively on other resources in the system to provide optimal health care”.

Read the ACGME Program Requirements in Pediatrics.

Pediatricians in practice may be the quarterbacks, but they can’t do it alone; they need to work with the teams around them.

Although the competency approach to medical education has progressed steadily, SBP has been the most difficult to operationalize in practice.1 The barriers are certainly real: inadequate support by payers, rapidly changing IT tools, and difficulty in developing the necessary partnerships and teams. However, the medical, social, and economic forces affecting both the patient and the pediatrician will only increase the importance of this competency in the future.

Why is Systems-based Practice Important to the Patient?

The morbidity and mortality for American children in the 21st century arises mainly from complex chronic disease or injury. Both require systems-based practice for prevention and management.  While the scale of the horror in Newtown shocked the country, a dispassionate examination of CDC data in the figure below shows that unintentional injury, homicide, and suicide are among the top three or four killers of American children throughout most of childhood. As pediatricians, all of us either have been or will be touched by this in some way. Last year, AAP’s Council on Injury, Violence, and Poison Prevention outlined the evidence behind a series of systems-based practice recommendations to prevent gun violence.5

10LCID_All_Deaths_By_Age_Group_2010-aComplex chronic disease requires a system of accessible, comprehensive, coordinated care—a fact recognized by the growing number of patient-centered medical homes.

The widespread prevalence of racial and ethnic disparities among children with chronic illnesses only accentuates the need for a systems approach because disparities are multifactorial in origin with a complex interplay of genetic, socio-economic, cultural, and provider contributions.6

Why is Systems-based Practice Important to the Pediatrician and Society?

There is considerable debate about whether other healthcare professionals such as nurse practitioners should deliver primary care to children independent of physician oversight. Although the roles of the nurse practitioner, the pediatrician, and the other team members may vary depending on circumstances and the type of practice, the pediatrician is uniquely trained to integrate systems elements spanning from genome to the whole child to the society and environment in which the child lives. This is vitally important no matter the type of practice: urban, rural, private or academic.

The “ecobiodevelopmental” framework for early identification and intervention among children experiencing extreme adversity (eg, neglect, abuse, malnutrition, violence) and “toxic stress” is an example of a systems-based approach. It offers a scientific framework for how these types of adversity affect the neuro-endocrine and immune systems leading to higher risk not only of developmental and behavioral abnormalities but also cancers, asthma, and cardiovascular disease later in life. The pediatrician has the opportunity to apply this scientific understanding to the screening, anticipatory guidance, and referrals to community resources as well as policy education and advocacy.7,8 If pediatricians limit their scope of involvement to the isolated medical encounter, the competition from other providers offering to provide the same care at lower cost is likely to increase.

From a societal perspective, pediatricians are in a position to impact future adult chronic illness and excess healthcare expenses. Adult cardiovascular disease (CVD) is a case in point. Current models forecast that adolescent obesity will increase future adult obesity and lead to more than 100,000 adult CVD deaths and increase healthcare expenditures by $254 billion by 2035.9 Pediatrician leadership in home and school-based interventions must be an essential component of the response to the childhood obesity epidemic.

What Does Systems-based Practice Mean for the ABP?

Although medical knowledge and patient care have been the traditional competencies assessed in the certification process, they are clearly not sufficient to assure the public of the high quality expected of a diplomate in the 21st century.  In response, several pediatric organizations, foundations, and other groups have supported the incorporation of community pediatrics, population health, and systems-based practice into innovative residency training programs.10

ABP supports these efforts and has been able to incorporate some elements of SBP into its certification programs. As an example of Part 1 of Maintenance of Certification (MOC), ABP is one of many organizations to endorse the ABIM Foundation professionalism charter, which articulates principles of patient primacy and social justice, necessary elements in systems-based practice.11 I am not the only one emphasizing the importance of SBP. Many pediatric groups around the country have formed quality improvement collaboratives (Part 4 of MOC) on SBP-related topics such as the family-centered medical home, obesity prevention and management, access to care, injury prevention, teen mental health, and more.  Finally, ABP led a working group of experts who have defined the SBP milestones that will become part of the indicators of a resident’s readiness for unsupervised practice.

Currently, SBP remains under-represented on certification exams. The task of constructing valid and reliable examination questions reflecting the systems-based practice requirements for violence prevention, obesity management, asthma disparities, toxic stress, and a host of other major child health problems will become easier as the evidence base grows and more and more residency graduates have received explicit training in these domains.  Ultimately, the child, the pediatrician and society will benefit. With sustained engagement, we can hope that events such as those at Newtown will become historical aberrations that spurred significant and lasting change.

This is what I think. Please let me know what you think by leaving a comment below.

  1. Johnson JK, Miller SH, Horowitz SD, et al. Systems-based practice: improving the safety and quality of patient care by recognizing and improving the systems in which we work. In: Henriksen K, Battles JB, Keyes MA, Grady ML, eds. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign). Rockville, MD: Agency for Healthcare Research and Quality (US); 2008;2:321–30.
  2. Wyckoff AS. Sandy Hook pediatricians share grief, advice, hope 6 months after tragedy. AAP News. 2013;34:1.
  3. Bertalanffy LV. An outline of general system theory. Br J Philos Sci. 1950;1(2):134–65.
  4. Bertalanffy LV. General Systems Theory: Foundations, Development, and Application. New York, NY: George Braziller, Inc; 1969.
  5. Dowd, MD, Sege, RD; Council on Injury, Violence, and Poison Prevention Executive Committee. Firearm-Related Injuries Affecting the Pediatric Population. Pediatrics. 2012;130(5):e1416–23. Epub 2012 Oct 18.
  6. Berry JG, Bloom S, Foley S, Palfrey JS. Health inequity in children and youth with chronic health conditions. Pediatrics. 2010;126(Supplement):S111–9.
  7. Garner AS, Shonkoff JP; Committee on Psychosocial Aspects of Child and Family Health, Committee on Early Childhood, Adoption, and Dependent Care, and Section on Developmental and Behavioral Pediatrics. Early childhood adversity, toxic stress, and the role of the pediatrician: translating developmental science into lifelong health. Pediatrics. 2011;129(1):e224–31. Epub 2011 Dec 26.
  8. Johnson SB, Riley AW, Granger DA, Riis J. The Science of early life toxic stress for pediatric practice and advocacy. Pediatrics. 2013;131(2):319–27. Epub 2013 Jan 21.
  9. Heidenreich PA, Trogdon JG, Khavjou OA, et al. Forecasting the future of cardiovascular disease in the United States: a policy statement from the American Heart Association. Circulation. 2011;123(8):933-44. Epub 2011 Jan 24.
  10. Kuo AA, Etzel RA, Chilton LA, et al. Primary care pediatrics and public health: meeting the needs of today’s children. Am J Public Health. 2012;102(12):e17–23. Epub 2012 Oct 18.
  11. ABIM Foundation. American Board of Internal Medicine; ACP-ASIM Foundation, American College of Physicians-American Society of Internal Medicine, European Federation of Internal Medicine. Medical professionalism in the new millennium: a physician charter. Ann Intern Med. 2002;136(3):243–6.

Never a Quiet Breath

Dr D G Nichols

“Never a Quiet Breath and the Power of Transparency”

The Bottom Line: The redefinition of professionalism to include transparent disclosure of outcomes transformed cystic fibrosis into a disease with hope. This “systems view” of professionalism can help all children.


My angle of vision widened as I pushed open the door to the treatment room.  A middle-aged woman, apparently all alone, wept quietly. The object of her grief then came into view.

On-Call in 1979

It was a very busy night, and I was the senior resident on call, when the nurses paged me to report that a three-year-old boy with cystic fibrosis had died. I was consumed with multiple admissions and sick children on all the floors, so it took me an hour to actually get to little Joey. When I finally reached the treatment room, the mother was standing a few feet away from the examination table where her 3 year-old son’s body lay. I reached out to offer some human contact, which seemed to be missing from her very small world.

“Dr. Nichols”, she sobbed, “I’m glad he’s dead. He never took a quiet breath his whole life. Every single breath was a cough or wheeze. His whole life was this struggle just to get air.”

The year was 1979, and cystic fibrosis began and ended in childhood. Today, the median life expectancy of a child born with CF is nearly 40 years.1

What transformed a disease marked by enormous suffering and early death into one where survival into adulthood is the norm?  Surely basic molecular research has made enormous contributions to the understanding and treatment of CF. However, the larger story is about the power of transparency as an act of professionalism.

Transparency as Professionalism

Transparency in this context means the willingness of the care team to disclose the outcomes of their care publicly.  The traditional concept of professionalism directed at the individual physician’s competence in such areas as confidentiality or informed consent will always be important. The good news is that, despite concerns about a crowded residency curriculum and inadequate faculty preparation,2 most residency graduates within the past 5 years feel more confident in their ability to adhere to standard professionalism concepts than did the cohort that graduated more than 5 years ago.3  However, several writers have argued for an enhanced “systems view” of professionalism in the face of enormous external pressures on the doctor-patient (pediatrician-family) relationship. 4,5,6

The systems view argues that the physician’s ability to improve the health of an individual patient depends on the ability to impact the layers surrounding the doctor-patient relationship, namely the rest of the clinical team, the health environment in the patient’s community, and ultimately the entire healthcare system.

Developments in the ensuing years since my night on call have vindicated the suffering of little Joey and the many others like him. Prodded by the Cystic Fibrosis Foundation, each CF center in the US has publicly disclosed the outcomes of its care — life expectancy, pulmonary function, and nutritional status for many years. Best practices have been shared so that all clinical teams can benefit.  As a result, the overall care and outcomes for all CF patients have improved dramatically since my encounter with Joey’s grieving mother.

There are certainly barriers to measurement and disclosure of clinical results ranging from small sample sizes to lagging IT systems to economic competition among providers. But the overall goal of improved health for patients and populations cannot be achieved without physician leadership grounded in a set of values. Transparency signifies accountability and engagement with parents (which was critical to the CF successes). The principle of transparency applies not only to children with serious chronic illnesses but also to well-child care. Click here to view a YouTube video on the Well Visit Planner provided by the Child and Adolescent Health Measurement Initiative, which is based on American Academy of Pediatrics guidelines and for which Maintenance of Certification (MOC) credit from the ABP will soon be available. This resource is supported by a grant from the Department of Health and Human Services.

The learning environment has a profound impact on subsequent practice. If academic medical centers and residency programs become transparent in their clinical outcomes, then graduating residents are more likely to do the same in practice. Click here for a primer on Teaching and Assessing Professionalism developed by the Association of Pediatric Program Directors and the American Board of Pediatrics.

I will never forget the anguish of that mother who had witnessed so much suffering in her little boy. Cystic fibrosis physicians offer an inspiring example for all of us.  Once routine public disclosure of patient outcomes becomes a professional expectation, outcomes, including parent engagement, are likely to improve at a faster pace for all children.

This is what I think. Please let me know what you think by leaving a comment below.

David G. Nichols President, CEO References:

  1. Cystic Fibrosis Foundation. Cystic Fibrosis Foundation Patient Registry 2011 Annual Data Report. 2012:1–32.
  2. Lang CW, Smith PJ, Ross LF. Ethics and professionalism in the pediatric curriculum: a survey of pediatric program directors. Pediatrics. 2009;124(4):1143–1151. doi:10.1542/peds.2009-0658.
  3. Cook AF, Ross LF. Young physicians’ recall about pediatric training in ethics and professionalism and its practical utility. J Pediatr. 2013;Epub(May 21). doi:10.1016/j.jpeds.2013.04.006.
  4. Lesser CS, Lucey CR, Egener B, et al. A behavioral and systems view of professionalism. JAMA. 2010;304(24):2732–2737.
  5. ABIM Foundation, ACP-ASIM Foundation, European Federation of Internal Medicine. Medical professionalism in the new millennium: a physician charter. Ann Intern Med. 2002;136(3):243–246.
  6. Swensen SJ, Meyer GS, Nelson EC, et al. Cottage industry to postindustrial care—the revolution in health care delivery. N Engl J Med. 2010;362(5):e12. doi:10.1056/NEJMp0911199.

Is MOC Just a Game?

ImageMy E-mails

“There’s no MOC Part 4 activity that relates to my area of practice. Is MOC just a game or a make-work activity?”

So began one of the many e-mails I have received over the past few months. One of the wonderful things about assuming a new position at any organization and especially the American Board of Pediatrics (ABP) is the warm welcome offered by so many individuals. An equally important component is the opportunity to listen to frank observations about what an organization is doing right and where it can improve. It’s no secret that Maintenance of Certification (MOC) – consisting of demonstration of professional standing (Part 1), life-long learning (Part 2), and periodic secure examinations (Part 3), and quality improvement (Part 4) – has been a source of intense discussion for all specialty boards. Therefore, I was not surprised that MOC has dominated some of the commentary in my e-mails.

So, is MOC a game or just a make-work activity? Because my e-mail correspondent was referring specifically to the quality improvement (QI) part of MOC, my comments below focus mainly on Part 4 of MOC.

The Evolution of Standards in a Profession

The exploration of my correspondent’s question begins with an appreciation of the mission of a specialty board and the momentous changes in the external environment-particularly in the last 20 years. In exchange for being granted the privilege of practicing in a profession, members of the profession voluntarily set high standards and develop mechanisms to identify those members of the profession who meet the standards.

The late 19th and early part of the 20th centuries witnessed the development of the truly scientific basis for the practice of medicine. A major enhancement in professional standards followed, including the four-year postgraduate medical school (Flexner report), the adoption of residency training programs, and the creation of specialty boards. Throughout much of the 20th century, specialty boards such as the ABP have sought to satisfy their responsibility to the public by certifying that the pediatrician has completed an accredited training program and mastered a body of knowledge, as evidenced by passing a single certifying examination upon graduation from residency training. Until recently, however, there have been only modest efforts to assess the quality of actual practice.

Knowledge Explosion and Quality Gaps as External Drivers in the 21st Century

Figure 1 illustrates the difficulty with the above approach for the 21st century. MOC is an evolution of the recertification effort that started in the 1990s, which in turn developed out of the recognition that the exponential growth in biomedical knowledge meant that pediatricians could not be expected to practice throughout a 30- to 40-year career without demonstrating that they had updated the knowledge base they acquired in pediatric residency. What may have been a defensible position when the ABP was founded in 1933 was not likely to provide adequate assurances to the public considering that scientific knowledge about children’s health had entered an exponential growth phase, which began with President Kennedy’s establishment of the National Institute of Child Health and Human Development (NICHD) of the National Institutes of Health.


Figure 1: Number of publications per decade about children’s health based on PUBMED searches using “children” as the keyword.

In addition to the sheer volume of information, data beginning in the 1960s began to reveal the risk of erosion of physician performance after residency training.1

The tipping point for a complete change in certification standards occurred when the Institute of Medicine (IOM) released its two seminal reports, To Err is Human2 and Crossing the Quality Chasm.3 The press, the public, and the federal government received these bombshell studies, which reported nearly 100,000 deaths in the United States annually because of lapses in patient safety and quality. Thereafter, the medical profession and the American public made systematic QI efforts a priority. In the wake of these reports, all 24 specialty boards agreed to make patient safety and QI requirements of MOC. The ABP assumed a leadership role among specialty boards in promoting systematic improvement in the quality of every pediatrician’s practice as a central ingredient in the designation “Diplomate of the American Board of Pediatrics.”

Once the ABP made the decision to develop a QI program, the question then became how to design a program that would be relevant to more than 65,000 practicing diplomates in general pediatrics and numerous subspecialties in settings ranging from solo practices to large academic medical centers. An additional challenge proved to be the fact that most practicing pediatricians had no formal training in the methods of QI.

Pediatricians own Part 4 of MOC; the ABP just awards the points…

The current Part 4 MOC program has been designed to address these challenges. It includes on-line practice improvement modules (PIMs) in which all participants can learn and apply the basics of a QI program to their practices. Still, some critics have called ABP’s hand-hygiene PIM “just a game.” If that’s so, then it’s a deadly game with more than 2 million nosocomial infections and 90,000 nosocomial infection-related deaths in the US annually. Frankly, there is abundant evidence that specific, directed QI interventions are needed to sustain compliance with hand-hygiene standards and reduce nosocomial infection rates.4 More than 10,000 pediatricians have completed the ABP hand-hygiene PIM with documented improvement in adherence to the recommended guidelines.

Participation in a quality improvement network represents a more rigorous approach to QI that is recognized through Part 4 of MOC. In this pathway, pediatricians join a team or network that develops specific goals, interventions, and registries to improve the health of a population of kids. During a recent visit to the American Academy of Pediatrics (AAP) offices in Chicago, for example, I stumbled upon just such a group of primary care pediatricians from around the country. They had come together to establish a QI Network on Genetics in Primary Care (Figure 2). These pediatricians receive MOC Part 4 credit, but that is not their primary motivation for taking time from their practices to set up this network. They, like thousands of others, have decided to assume ownership of improving care for their patients in a systematic way.


Figure 2. Primary care pediatricians coming together to establish a quality improvement network on genetics in primary care practice. (AAP Quality Improvement Innovation Network, March 2013)

A number of other networks have also been established. The outcomes in central-line associated blood stream infections, inflammatory bowel disease remission, cystic fibrosis pulmonary and nutritional status, and asthma hospital admissions, for example, attest to the power of a network of pediatricians focused on improving care.5,6,7 To date, the ABP has approved more than 300 QI efforts from more than 125 organizations-often with dramatic results and all because the pediatricians involved have collectively decided to assume ownership of a children’s health care problem and make it better. It is true that they are diplomates meeting the requirements of certification, and as such they receive “points” from the ABP for their involvement in QI; however, MOC is an afterthought in these networks and projects. It is the expression of their commitment to improve the health of children (“ownership”) that drives the involvement of these diplomates in QI.

So when some pediatricians argue there are no MOC Part 4 activities that fit their practices, my reply is that ABP is working as fast as it can to promote the development of more QI activities for children, but don’t wait for us. Do what the pediatricians in the more than 125 organizations approved for MOC QI activities (and our colleagues in Figure 2) have done. Go out and take ownership of a quality or safety problem that does relate to your practice; and then be sure and tell us about it. We will help you with the methodologies and proudly offer you MOC Part 4 credit. If you are part of an organization that wishes to tackle a quality improvement project in pediatrics, you can go to http://www.mocactivitymanager.org/overview/ABP/ for more information.

The ABP’s Commitment

Even though my e-mail correspondent did not ask what the ABP does about its own internal QI, it is important to note that we are engaged in similar initiatives. Like any other organization, the ABP finds that gaps in performance develop without constant attention. We believe in applying QI methodology for ourselves first. We have a great staff hard at work trying to help you improve the health care of children by upgrading our technological infrastructure, improving access through our Web site, and simplifying the communications process needed to set up a QI network or check on your MOC status. At the policy level, we are also advocating for streamlining the pediatrician’s demonstration of competency such that MOC credits can, for example, count toward maintenance of licensure (MOL), Medicaid incentive payments, and Joint Commission “Ongoing Professional Practice Evaluation (OPPE).”

Just as pediatricians had to adapt to changing circumstances at the beginning of the last century so that parents would be assured their children were receiving the best possible care, so too must we adapt in this century. Fortunately, I can say with confidence that diplomates of the ABP are leading the change for better care for children.

This is what I think. Please let me know what you think by leaving a comment below.

David G. Nichols
President, CEO


  1. Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142:260-273.
  2. Kohn LT, Corrigan JM, Donaldson MS, eds. To Err Is Human: Building a Safer Health System. Washington, DC: National Academy Press; 2000.
  3. National Research Council. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001.
  4. White CM, Statile AM, Conway PH, et al. Utilizing improvement science methods to improve physician compliance with proper hand hygiene. Pediatrics. 2012;129:e1042-e1050. doi:10.1542/peds.2011-1864.
  5. Crandall WV, Boyle BM, Colletti RB, et al. Development of process and outcome measures for improvement: lessons learned in a quality improvement collaborative for pediatric inflammatory bowel disease. Inflamm Bowel Dis. 2011;17:2184-2191. doi:10.1002/ibd.21702.
  6. Meyer H. Targeted care improvements show promising results for treating children with asthma. Health Aff (Millwood). 2011;30:404-407. doi: 10.1377/hlthaff.2011.0045.
  7. Miller MR, Niedner MF, Huskins WC, et al. Reducing PICU central line-associated bloodstream infections: 3-year results. Pediatrics. 2011;128:e1077-e1083. doi:10.1542/peds.2010-3675.