The Repetitive History of CMS Failing Patients and Donors with Flawed Performance Metrics for Organ Procurement Organizations

Home / News / Latest News / The Repetitive History of CMS Failing Patients and Donors with Flawed Performance Metrics for Organ Procurement Organizations

The Repetitive History of CMS Failing Patients and Donors with Flawed Performance Metrics for Organ Procurement Organizations

Over the fifty-plus-year history of Organ Procurement Organizations (OPOs) in the United States, the Centers for Medicare and Medicaid Services (CMS) has empirically and repeatedly failed to develop and adopt a reliable and comprehensive assessment of their relative performance.

Charged by Congress with this responsibility in 1984, it took CMS until 1998 to adopt its first measure. When that measure was overturned by Congress in 2000, CMS took until 2006 to develop a new measure.  Now, CMS is four years into its latest measure, the November 2020 governing rule, and yet has issued no guidance on its implementation, and a growing body of peer-reviewed research demonstrates dramatic statistical failings and unreliability of the metrics.

This research can be found here:
https://sidandt.org/the-science/peer-reviewed-science

Underlying these CMS failures is a grossly overly simplistic approach that disregards fundamental, long-established methodologies of assessing health system performance across the U.S.; sadly, this is a decades-long trend.

In 1998, CMS’s first metric adopted the then and continuing international measure of donors per million population. Very quickly, it became evident that this measurement was based on the faulty assumption that death rates are uniform across the population of donation service areas/OPOs. In fact, the rates vary from a high of 1227/100,000 to a low of 658/100,000. This recognition enabled the OPO community to gain the support of Congress to require CMS to establish a measurement system that was more reliable, but not before CMS initiated actions to close OPOs under the system. Those decertifications due to questions of the population basis and metric in Los Angeles and Arkansas failed. This is an absolute prologue to the legal setbacks HHS/CMS will face in 2026.

In 2006, CMS subsequently adopted donors per eligible death, which was far more granular because it relied upon death reports from hospitals to OPOs of brain-dead potential donors without contraindicating conditions. This metric was subsequently questioned because it initially contained a subjective element (multi-system organ failure) as a referral rule-out condition that OPOs could too widely interpret. In 2017, UNOS (United Network for Organ Sharing) and CMS updated the definition to include specific laboratory values for each organ, correcting this subjectivity. However, the loss of confidence in the metric was already overwhelming and made worse by allegations that this data was “self-reported” and, therefore, inherently unreliable. It can be argued that the data was not self-reported because it is based upon hospital reports of referrals to OPOs, lab values, and diagnostic tests. In 90 percent of cases, diagnoses that rule referrals out can be traced to physician and laboratory decisions. However, the metric was further challenged by the fact that it did not keep up with the dramatic growth of donations after circulatory death (DCD) and no longer effectively measured OPO performance.

The current CMS OPO metric, adopted in December 2020, has returned the assessment of OPO performance to the earlier days of a highly high-level estimate of donor potential based on hospital deaths. Research documents the metric is replete with errors resulting from its inherent assumption that the causes of death and underlying conditions that allow or prevent donation are consistent and uniform annually and nationwide, as well as assumptions that clinical, demographic, cultural, and religious impacts on donation are uniform across the country; an assumption that appears on no national health policy nor performance measurement of hospitals, MD, and social service providers.

In January 2020, researchers from UNOS and SRTR published a methodological review of the then-proposed Centers for Medicare and Medicaid Services (CMS) Organ Procurement Organization (OPO) performance metric in the American Journal of Transplantation (AJT) [i].

In this article, they noted that the use of a 95th percentile confidence interval intended to reduce volatility in the performance measurement of small OPOs would simultaneously produce a bias in the measurement of larger OPOs. This conclusion was reiterated in public comments submitted to CMS during the rule-making process published with CMS on December 2, 2020, publication of the OPO metric Final Rule[ii]. In response to these public comments, CMS dismissed them and enacted the OPO rule and metric with the 95th percentile confidence interval.

In 2024, the federally contracted SRTR staff re-looked at issue, prepared two abstracts of their research results that demonstrate the bias, and proposed an alternative methodology for presentation at the American Transplant Congress (ATC) in July 2024. The abstracts were approved for submission by the SRTR’s contract manager (The Heath Services Resource Administration (HRSA)) and by the ATC. They were printed in the online application and the AJT supplemental edition for Congress[iii] [iv].

One abstract tested the 2019-2021 rankings of OPOs and found that the methodology was biased against large OPOs. The significance of the bias was substantial, and the abstract identified that the CMS OPO metric could never rank the largest OPO as a Tier One OPO unless all OPOs were ranked Tier One, a near impossibility. The SRTR researchers also submitted an abstract that proposed using an industry-standard observed-to-expected performance measure to offset the volatility of small-volume programs while not disadvantaging large OPOs.

At the opening of the Congress HRSA, at the request of CMS, HRSA ordered the SRTR researchers not to present this research and these abstracts at the ATC meeting.

In September 2024, AJT published an Opinion[v] submitted by researchers in the transplant and donation field condemning the censorship of legitimate and replicable research.

Subsequently, the lead researchers on this opinion piece shared their methodology with peers who replicated the research and combined the two abstract topics into an article accepted for publication and “in the press” in the AJT[vi]. This article identifies the statistical error introduced by the 95th percentile confidence interval that results in a bias against the measurement of performance of large OPOs, an error so egregious that “the largest OPO could never be ranked as a Ter 1 OPO unless 100% of OPOs were in Tier 1, an impossibility. The authors then offered a statistically reliable Observed to Expected performance measure alternative that removes this bias against large OPOs.

The analysis reports that in 2021, 54% of OPOs were wrongly ranked in lower tiers by the faulty CMS 95th percentile confidence interval methodology compared to the O-to-E—methodology, with error rates of 43% in 2020 and 24% in 2019. The analysis also demonstrates that the CMS methodology ranks 42% of OPOs in Tier 3, requiring automatic decertification in 2021, while the O-to-E identifies only 12% of OPOs in Tier 3. The analysis shows a similar reduction in Tier 3 in 2020 (from 40% to 17% in 2020 and 27% to 16% in 2019). Finally, in 2021, 30 (55%) of the 55 OPOs changed tiers due to the methodology; in 2020, 24 OPOs (44%) changed tiers, and in 2019, 14 OPOs (25%) changed tiers.

The immediate impact of these research results is that each of the OPOs that are not routinely in Tier 1 under the existing CMS methodology has highly credible evidence that the CMS OPO metric is statistically unreliable, mis-ranks their performance, and would decertify OPOs that are actually higher performing while maintaining OPOs that were lower performing.

Each OPO can appeal and, if necessary, sue CMS for relying on an unreliable OPO metric. Such appeals and initiation of lawsuits have been used by OPOs under the prior CMS OPO metrics of Donors per Million population and Donors per Eligible Donor. This has resulted in CMS rescinding each decertification effort it has ever undertaken.

An additional area of research that must be pursued is updating the previously published research that shows significant changes in ranking based on other statistical weaknesses of the CMS methodology, which are compounded by the 95th percentile methodology. These include 1)  Schold et al.[vii] that demonstrate mis-ranking due to weaknesses in CDC Cause of Death as the basis of calculating OPO donor potential versus actual hospital Discharge Diagnoses (HCUP) that report up to 41% of OPOs changing Tiers with more granular state inpatient discharge data, 2) SRTR research that identifies 14% of OPOs changing tiers when national variations in donation rate by race are applied to OPO rankings based on the variation in authorization rates by race and varied distribution of races by OPO[viii], and 3) Schold et al. research on the impact of Area Deprivation Index on donation rates [ix]

It is time-critical for all to follow the example of Science in Donation and transplant and educate lawmakers, office holders, opinion makers, the media, and the industry on the data reality beyond the sensational misinformation that has dominated the past few years.

REFERENCES

[i] Snyder JJ, Musgrove D, Zaun D, et al. The Centers for Medicare and Medicaid Services’ proposed metrics for recertification of organ procurement organizations: Evaluation by the Scientific Registry of Transplant Recipients. Am J Transplant. 2020;20:2466–2480. https://doi.org/10.1111/ajt.15842

[ii] DEPARTMENT OF HEALTH AND HUMAN SERVICES Centers for Medicare & Medicaid Services 42 CFR Part 486 [CMS-3380-F] RIN 0938-AU02 Medicare and Medicaid Programs; Organ Procurement Organizations Conditions for Coverage: Revisions to the Outcome Measure Requirements for Organ Procurement Organizations; Final rule

[iii] Abstract# 1168 Are the New CMS Performance Tiers Biased against Larger OPOs? G. Lyden, J. Miller, R. Hirose, J. Snyder, Scientific Registry of Transplant Recipients, Hennepin Healthcare Research Institute, Minneapolis, MN; American Journal of Transplantation Vol. 24Issue 6SupplementS310–S561Published in issue: June 2024.

[iv] Abstract# D294 Reducing Bias against Larger Organ Procurement Organizations in Performance Evaluations; G. Lyden, J. Miller, R. Hirose, J. Snyder, Scientific Registry of Transplant Recipients, Hennepin Healthcare Research Institute, Minneapolis, MN; American Journal of Transplantation Vol. 24 Issue 6 Supplement S562–S1129 Published in issue: June, 2024

[v] Schold JD et al., The dangerous precedent of censoring scientific dissemination, American Journal of Transplantation, 

https://doi.org/10.1016/j.ajt.2024.09.002

[vi] Lopez R, Mohan S, Rodrigue JR, Arrigain S, Brosi D, Lavanchy R, Kaplan B, Pomfret EA, Schold JD, Association of Organ Procurement Organization Volume with CMS Performance Evaluations, American Journal of Transplantation, https://doi.org/10.1016/j.ajt.2024.11.024.

[vii] Donation Data Sources Used for New CMS OPO Regulations; J. Schold, University of Colorado, Aurora, R. Lopez, University of Colorado, Aurora, D. Zingmond, University of California Los Angeles, Los Angeles, CA; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg. S 607, Abstract 566, https://www.amjtransplant.org/article/S1600-6135(23)00474-4/fulltext

[viii] Adjusting for race in metrics of organ procurement organization performance; Jonathan M. Miller 1,2,*, David Zaun 1, Nicholas L. Wood 1,2, Grace R. Lyden 1,2 , Warren T. McKinney 1,2 , Ryutaro Hirose 1,3 , Jon J. Snyder 1,2,4; https://www.amjtransplant.org/article/S1600-6135(24)00122-9/fulltext

[ix] Impact of Area Deprivation Index on Organ Procurement Organization Performance Metrics;J. Schold, University of Colorado Anschutz Medical Campus, Aurora, CO, R. Lopez, University of Colorado, Aurora, CO, S. Mohan, Columbia University Medical Center, New York, NY; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg S471, Abstract 273, https://www.amjtransplant.org/article/S1600-6135(23)00474-4/fulltext