Peer Reviewed Science Continues to Show Major Flaws in CMS Governing RULE

Home / The Science / Peer-Reviewed Science

Peer-Reviewed Science

2024 Published

Are the New CMS Performance Tiers Biased against Larger OPOs?

G. Lyden, J. Miller, R. Hirose, J. Snyder, Scientific Registry of Transplant Recipients, Hennepin Healthcare Research Institute, Minneapolis, MN, American Journal of Transplantation (AJT), Volume 24, Issue 6, Supplement 1; Oral Abstracts Pg. S531, Abstract 1168, https://www.amjtransplant.org/issue/S1600-6135(24)X0006-4

Summary

This study uses simulation to assess for potential biases against larger OPOs in the CMS OPO 2020 metric.
Smaller OPOs in Scenario 2 were much more likely to be placed in Tier 1, resulting in a 95% probability that the largest OPOs would have to recompete compared to only a 26% probability for the smallest OPOs.
Large OPOs have an equal chance of being in Tier 1 only when all OPOs have an underlying rate equal to the previous year’s 75th percentile, but not in other scenarios where all OPOs perform the same.

Reducing Bias against Larger Organ Procurement Organizations in Performance Evaluations

G. Lyden, J. Miller, R. Hirose, J. Snyder, Scientific Registry of Transplant Recipients, Hennepin Healthcare Research Institute, Minneapolis, MN; American Journal of Transplantation (AJT), Volume 24, Issue 6, Supplement 1. Poster Abstracts Pg. S1114, Abstract D294, https://www.amjtransplant.org/issue/S1600-6135(24)X0006-4.

Summary

Under the CMS OPO 2020 metric smaller OPOs have a higher probability of being automatically recertified.
The purpose of this study is to propose an alternative tiering system that is not biased against large OPOs.
In scenario 1, the CMS method incorrectly placed the smallest OPO in Tier 1 72% of the time and virtually never placed the largest OPO in Tier 1.
The O-to-E method has a uniform error rate across OPO volumes when all OPOs perform the same, thus removing the bias against larger OPOs, without sacrificing power to detect underperforming OPOs.

Prediction of Cause, Age, and Location-Consistent Deaths: A Useful Tool to Identify Areas for Improvement for Organ Procurement Organizations

J. Miller 1, D. Zaun 1, N. Wood 1, G. Lyden 1, R. Hirose 2, A. Israni 1, J. Snyder 1, 1 Scientific Registry of Transplant Recipients, Minneapolis, MN, 2 University of Washington, Seattle, WA; American Journal of Transplantation (AJT), Volume 24, Issue 6, Supplement 1. Oral Abstracts Pg. S532, Abstract 1170, https://www.amjtransplant.org/issue/S1600-6135(24)X0006-4

Summary

Data used for CMS’ 2023 OPO evaluations are deaths from 2021. The data lag limits OPOs’ ability to track impacts of changes to their practice in near-real time.
This study presents a method to predict OPOs’ current CALC deaths using Scientific Registry of Transplant Recipients (SRTR) data.
The predicted CALC deaths can be used as a denominator for donation rate or transplant rate in years for which actual CALC deaths are not yet available from CDC or reported by CMS.

Adjusting for race in metrics of organ procurement organization performance

Jonathan M. Miller 1,2,* , David Zaun 1 , Nicholas L. Wood 1,2 , Grace R. Lyden 1,2 , Warren T. McKinney 1,2 , Ryutaro Hirose 1,3 , Jon J. Snyder 1,2,4; https://www.amjtransplant.org/article/S1600-6135(24)00122-9/fulltext

Summary

CMS chose not to adjust for most demographic variables other than age (for the transplant rate), arguing that there is no biological reason that these variables would affect the organ donation/utilization decision.
However, organ donation is a process based on altruism and trust, not a simple biological phenomenon.
When adjusting for race, 8 of the 58 OPOs moved 1 tier: 5 in one direction and 3 the other direction.
Among the OPOs that moved to a lesser tier ranking in our study, 2 of the 3 currently underperform the national rates among White potential donors.
We have shown that failing to adjust for race puts OPOs that are currently performing well among minorities relative to national rates at risk.

Are the Centers for Medicare & Medicaid Services metrics evaluating organ procurement organization performance too fragile?

Jesse D. Schold* Department of Surgery, University of Colorado Anschutz Medical Campus, Aurora, Colorado, USA, Rocio Lopez Department of Surgery, University of Colorado Anschutz Department of Surgery, University of Colorado Anschutz, Sumit Mohan Department of Epidemiology, Columbia University, New York, New York USA; American Journal of Transplantation, https://doi.org/10.1016/j.ajt.2024.03.025

Summary

In statistical terms, Miller et al and Lopez et al research on CMS OPO metrics studies suggest that the CMS models are fragile.
OPO performance). This fragility, in the context of highly consequential ramifications, may be concerning without clear evidence validating poor OPO performance beyond current models.
The policy could lead to a revolving door (or a shrinking pool) of OPOs with additional performance cycles—and the attendant challenges of disruption to the donation/transplant process.

Direct Measurement of DCD Donor Potential

Tom Mone MS, Tom Rosenthal MD, and Tom Seto BS; OneLegacy, Azusa, CA; Transplantation (Accepted for Publication, July 2024)

Summary

Existing methods of comparing organ procurement organization (OPO) performance utilize administrative data as an indirect measurement of donation after circulatory death (DCD).
The purpose of this study was to categorize and quantify reasons that potential DCD donors do not progress to donation to facilitate the direct measurement of OPO donor potential.
OPO donor potential calculated from referral and hospital death record reviews is substantially lower than donor potential determined by administrative data.
Direct measurement of OPO donor potential will be more accurate than any administrative measure, because many factors that rule out donors are not captured in hospital coded data or death certificates.

2023 Published

Population Characteristics and Organ Procurement Organization Performance Metrics

Rocio Lopez, MS, MPH; Sumit Mohan, MD, MPH; Jesse D. Schold, PhD, MStat, Med; JAMA Network Open. 2023;6(10):e2336749. doi:10.1001/jamanetworkopen.2023.36749 October 3, 2023 1/13, https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2809988

Summary

The CMS OPO rule evaluates performance based on an unadjusted donation rate and an age-adjusted transplant rates.
The study evaluates whether adjusting for age and/or area deprivation index yields the same tier assignments as the cause, age, and location consistent (CALC) tier used by CMS.
Age and ADI adjustment resulted in 19.0% to 31.0% reclassification of tier ratings for the OPOs, with 46.6% of OPOs changing tier ranking at least once during the 3-year period. Between 6.9% and 12.1% moved into tier 1 and up to 8.6%(5 of 58) moved into tier 3.

OPO Measured Donation Rate is Highly Volatile Year to Year and Not a Stable Quality Indicator

J. Schold, R. Lopez, University of Colorado, Aurora, CO S. Mohan, Columbia University Medical Center, New York, NY ; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg. S615, Abstract A5, https://www.amjtransplant.org/article/S1600-6135(23)00475-6/fulltext

Summary

With new CMS regulations, Organ Procurement Organizations (OPO) are to be evaluated yearly and certified/decertified every 4 years based on a single year’s data.
The purpose of this study was to assess the volatility of annual evaluations.
OPOs’ donor potential and donor rate are not stable year to year and 24/58 OPOs lie within 5% of a tier edge with many OPOs having shifts in donor potential >5%.
The consequence of yearly measurement may result in well-functioning OPOs inadvertently being decertified

Significant Discrepancies to Evaluate Organ Procurement Organization Performance Based on Exclusion Criteria

J. D. Schold, University of Colorado, Aurora, CO R. Lopez, D. Zingmond, , 2 University of California Los Angeles, Los Angeles, CA. AJT, Volume 23, Issue 6, Supplement 1. Pg. S930, Abstract C55, https://www.amjtransplant.org/article/S1600-6135(23)00475-6/fulltext

Summary

We sought to evaluate whether incorporating data with exclusions (CALC or CALC-adjusted) produce the same tier assignments, which determines OPO certification or decertification.
11/21 OPOs (52%) have greater than 5% difference in actual versus predicted donors. As a result, 29% (6/21) of OPOs change tiers using donor potential measured by CALC compared to that measured by CALC-adjusted.
Conclusions: Contraindicating exclusion factors are not equal across OPO service areas. Current tier assignments using CALC may be unreliable compared to those calculated by CALC-adjusted, using a large sample of OPOs across the country.

Impact of Area Deprivation Index on Organ Procurement Organization Performance Metrics

J. Schold, University of Colorado Anschutz Medical Campus, Aurora, CO, R. Lopez, University of Colorado, Aurora, CO, S. Mohan, Columbia University Medical Center, New York, NY; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg S471, Abstract 273, https://www.amjtransplant.org/article/S1600-6135(23)00474-4/fulltext

Summary

The purpose of this study was to evaluate if applying an ADI-adjustment yields the same tier assignments as the metrics used by CMS.
Compared to unadjusted donation rate and age-adjusted transplantation rate, additional ADI-adjustment resulted in 16% to 41% of OPOs changing tiers (figure) each year. Between 1-2 OPOs moved out of tier 1 and 3-7 moved into tier 1 each year.
ADI-adjustment of performance metrics significantly alters OPO tier assignments.

Cordance by Data Source for Defining Donor Potential Using Cause, Age, and Location Consistent with Donation Used for New CMS OPO Regulations

J. Schold, University of Colorado, Aurora, R. Lopez, University of Colorado, Aurora, D. Zingmond, University of California Los Angeles, Los Angeles, CA; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg. S607, Abstract 566, https://www.amjtransplant.org/article/S1600-6135(23)00474-4/fulltext

Summary

The purpose of this study is to assess whether CALC is a sufficiently reliable and objective determination of donor potential on which to make OPO decertification decisions.
Approximately 20% of OPOs changed tiers with State Independent Databases (SID) as compared to final rule (figure). Similar results are seen with transplant and overall tiers.
Tier assignments significantly change OPOs are reclassified for performance based on the different data sources. Given the structure of the tiering system and the significant ramifications, CMS should revisit their decision to use CALC for certification decisions.

Stability of New CMS Metrics for Organ Procurement Organizations: Comparison of 2 Consecutive Years

A. Israni, J. Snyder, Hennepin Healthcare, Univ of MN, Scientific Registry of Transplant Recipients, Minneapolis, MN; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg. S609 Abstract 571, https://www.amjtransplant.org/article/S1600-6135(23)00475-6/fulltext

Summary

We compared the stability of tiers for the new CMS metrics for donation rate and transplant rate between 2019 and 2020.
For the donation rate metric, between 2019 and 2020, 67% of the OPOs stayed consistent in their tiers and 33% changed tiers. For the overall tiers, 59% stayed consistent and 41% changed tiers.
Illustrating a limitation of using the prior year to set the performance targets.

Adjusting for Race in Metrics of Organ Procurement Organization Performance

J. Miller 1, D. Zaun 1, N. Wood 1, G. Lyden 1, W. T. McKinney 2, J. Snyder 1, 1 Scientific Registry of Transplant Recipients, Minneapolis, MN, 2 Nephrology, Hennepin Healthcare Research Institute, Minneapolis, MN; American Journal of Transplantation (AJT), Volume 23, Issue 6, Supplement 1. Pg S620, Abstract ALB2 https://www.amjtransplant.org/article/S1600-6135(23)00475-6/fulltext,

Summary

The CMS OPO donation rate metric is unadjusted, and the transplant rate metric is adjusted for age only. This study examines the impact of additionally adjusting these metrices for race.
Nationally, donation rates and transplant rates were higher among White potential donors than non-White potential donors.
When adjusting for race, 8 OPOs changed tiers (5 improved their tier, 3 lowered their tier).
Failing to adjust for race risks extreme penalties for OPOs that have high proportions of non-White potential donors

2021 Published

OPO performance improvement and increasing organ transplantation: Metrics are necessary but not sufficient

Kevin O’Connor 1, Alexandra Glazier 2; 1 LifeCenter Northwest, Bellevue, Washington, USA, 2 New England Donor Services, Newton, Massachusetts, USA; Am J Transplant. 2021;00:1–2.  https://www.amjtransplant.org/article/S1600-6135(22)08620-8/fulltext

Summary

The CALC methodology as the data set used to calculate the denominator for the new OPO metrics has its shortcomings.
Its reliance on death certificates widely acknowledged to be fraught with errors and its inability to exclude nonventilated deaths reduces the value of this data set for accurately assessing performance.
OPO performance improvement strategies use multiple measurements to identify opportunities to target and trial operational interventions.

2020 Published

The Centers for Medicare and Medicaid Services’ proposed metrics for recertification of organ procurement organizations: Evaluation by the Scientific Registry of Transplant Recipients

Jon J. Snyder 1,2 Donald Musgrove 1 David Zaun 1 Andrew Wey 1 Nicholas Salkowski 1 John Rosendale 3 Ajay K. Israni 1, 2,4 Ryutaro Hirose 1,5 Bertram L. Kasiske 1,4; Am J Transplant. 2020;00:1–15. https://www.amjtransplant.org/article/S1600-6135(22)22564-7/fulltext

Summary

On December 23, 2019, the US Centers for Medicare and Medicaid Services proposed 2 new standards that organ procurement organizations (OPOs) must meet for recertification.
We examined how OPOs would have fared under the proposed performance standards in 2016-2017.
We conclude that the proposed new standards may result in over half of OPOs facing decertification, and risk adjustment suggests that underlying characteristics of deaths vary regionally such that decertification decisions may be affected.
Therefore, the overall 75th quantile will be too low for small OPOs and too high for large OPOs.

Assessment of National Organ Donation Rates and Organ Procurement Organization Metrics

Luke J. DeRoos, MS; Yuhang Zhou, BS; Wesley J. Marrero, MS; Elliot B. Tapper, MD; Christopher J. Sonnenday, MD; Mariel S. Lavieri, PhD; David W. Hutton, PhD; Neehar D. Parikh, MD, MS JAMA Surg. oi:10.1001/jamasurg.2020.5395

https://jamanetwork.com/journals/jamasurgery/fullarticle/2773525?resultClick=1

Summary

Objective is to evaluate OPO performance metrics using combined mortality and donation data and quantify the associations of population demographics with donation metrics.
We demonstrate significant variability in OPO performance rankings, depending on which donation metric is used.
The performance of OPOs should be evaluated using a range of donation metrics.

Examination of Racial and Ethnic Differences in Deceased Organ Donation Ratio Over Time in the US

Amber B. Kernodle, MD, MPH; Wanying Zhang, MD, MS; Jennifer D. Motter, MHS; Brianna Doby, BA; Luckmini Liyanage, MD; Jacqueline Garonzik-Wang, MD, PhD; Kyle R. Jackson, MD, PhD; Brian J. Boyarsky, MD; Allan B. Massie, PhD, MHS; Tanjala S. Purnell, PhD, MPH; Dorry L. Segev, MD, PhD; JAMA Surg. 2021;156(4):e207083. doi:10.1001/jamasurg.2020.7083, https://jamanetwork.com/journals/jamasurgery/fullarticle/2776217

Summary

Historically, deceased organ donation was lower among Black compared with White populations, motivating efforts to reduce racial disparities.
The study objective is to examine changes in deceased organ donation over time.
The findings of this cohort study suggest that differences in deceased organ donation between White and some racial minority populations have attenuated over time.
Despite improvements, substantial differences remain, suggesting that novel approaches are needed to understand and address relatively lower rates of deceased organ donation among all racial minorities.

2015 Published

Does Social Capital Explain Community-Level Differences in Organ Donor Designation?

KEREN LADIN, RUI WANG, AARON FLEISHMAN, MATTHEW BOGER, and JAMES R. RODRIGUE; The Milbank Quarterly, Vol. 93, No. 3, 2015 (pp. 609-641)

Summary

Groups with higher levels of social capital, racial homogeneity, income, workforce participation, owner-occupied housing, native-born residents, and white residents had higher rates of organ donor designation.
These factors explained more than half the geographic variance in organ donor designation.
A better understanding of social capital may enhance efforts to increase organ donation.
Because community-level factors, including social capital, predict more than half the variation in donor designation, future interventions should tailor strategies to specific communities.

2013 Published

National Death Index Users Guide

CDC; National Center for Health Statistics. National Death Index User’s Guide. Hyattsville, MD. 2013; https://www.cdc.gov/nchs/data/ndi/ndi_users_guide.pdf (accessed 7/31/2024).

Summary

The Public Health Service Act (42 U.S.C. 242m) provides in Section 308(d) that the data collected by the National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC), may be used only for the purpose of health statistical reporting and analysis.
Furthermore, the applicant has assured NCHS that the identifying information:
- Will be used only for statistical purposes in medical and health research.

Home / The Science / Peer-Reviewed Science