This post was co-authored by David Schwartz, CEO and Co-Founder at Ethics Through Analytics, and Michael Shumpert, Data Science Executive at Mosaic Data Science.
As you may know, we have been submitting FOIA requests asking FDA to share data from its various programs. In October, FDA granted[1] our April FOIA request in which we asked the agency to add back demographic data fields that it had previously removed from its public Medical Device Report (“MDRs”) databases. To find potential bias, we encourage manufacturers to use this data to look for any disproportionate impact its devices are having on minorities.
In the dataset we received, we won’t find any definitive evidence because there are many relevant external variables not accounted for and incompleteness in the FDA data set, but the data still can produce clues as to where we should be looking. This month we thought we would do a general survey to see where companies might want to focus attention specifically on devices that might be disproportionately hurting Black Americans.
Results
We start with an overview of where, as a percentage, a significant number of Black Americans are the subject of adverse events in MDRs involving medical devices. We picked the last FDA fiscal year – 2023 – and we looked for product codes that had a high percentage of MDRs for Black Americans.
Methodology
There were about 2,324,000 MDRs filed at FDA in FY 2023. Unfortunately, only a very small number of those – 39,000 – had information on race. That’s a low but still a meaningful number. Unfortunately, there would be a risk of statistical bias if reporters are only making note, for example, when a minority is adversely affected, but the truth is we have no idea why reporters might be reporting race in some cases but not others.
Fortunately, though, the demographic data that is included in the MDRs is close to being in proportion with overall US population numbers. For example, the number of Black Americans in the US is roughly 13.6% and the number of reported MDRs where race is included that designate Black is roughly 11.7%, suggesting that Black adverse events may be slightly less reported than other races. Race data does not appear to be disproportionately reported in cases of minority status. Nonetheless, this is one of the significant limitations of the analysis. We do not know why the reporting of demographics is so low.
Many of the reports indicated that people are a mixture of races, not surprisingly. So, when reporting the number of Black Americans, we included anyone who is at least partly listed as Black.
We have no denominator data to assess how many units of any given product category are in use. Further, we do not know the rate at which Black Americans have access to any particular technology as compared to other Americans. It seems reasonable to assume at this investigatory level that typically Black Americans don’t have disproportionately higher access to medical care in any particular area. Again, this is a back of the envelope calculation, but that assumption allows us to at least investigate.
What we are looking for is any evidence that Black Americans had more adverse events than we would expect. To choose a threshold for reporting in the chart, we noted that according to the most recent census data,[2] the mean percentage of Black Americans across the geographic regions of the United States is 13.6%. To be conservative, we added 3 standard deviations to the mean to get a threshold of 18%, because we would expect some variation in Black percentage rates across regions and frankly across device categories, and this threshold should cover 99% of normally distributed data. In other words, in our analysis we were looking for any medical devices where the MDR rate applicable to Black Americans exceeded 18%.
We also calculated the total number of Black Americans included in the MDRs for all products together and found that rate to be 11.7%, so our threshold of 18% is well above the average for all medical devices. As observed above, that gave us a bit of comfort that there might be no upward systematic error or bias in the reporting. And all of that suggested that 18% was a reasonable cut off to become interested.
To focus on the most significant potential issues, in the visualization we set a threshold for a minimum number of FY 2023 MDRs to be included for reporting. We picked 32 MDRs as the minimum threshold, because that covers 90% of all MDRs. There were plenty of examples below that of where Black Americans had substantially higher than 18% adverse events, but they could easily just be statistical anomalies due to the small sample sizes.
Analysis
Given the methodology we just described, there are lots of obvious weaknesses in the approach and untestable assumptions we needed to make. So, these results really aren’t by any means convincing evidence that there is a problem. This analysis simply attempts to flag any area where perhaps further investigation would be appropriate.
The larger the number of MDRs, the more meaningful the percentage of Black Americans that are included in those MDRs becomes. So, for example, ventricular assist bypass devices produced over 3500 MDRs, and about 40% of those MDRs were for Black Americans. A little bit less significant, dermal cooling packs produced over 2500 MDRs, and about 18% of those were for Black Americans. Blood glucose test systems produced about 800 MDRs, and about 23% of those were for Black Americans.
The research literature on bias in reporting on adverse events provides some helpful context. The reporting of adverse events in healthcare, particularly among Black Americans, reveals some critical disparities and challenges. Black patients have been found more likely to report preventable adverse events attributable to poor care coordination compared to their White counterparts, which is independent of demographic and clinical characteristics.[3] To be clear, these are patient reported adverse events to their caregivers, as opposed to what we are measuring in MDRs which are adverse events reported by manufacturers to the FDA that reflect an event typically reported by caregivers to the manufacturer. Given that this is observational data, we must also note that it’s possible that Black patients simply choose to report more.
Studies indicate that patients of color, including Black and Latino patients, are more likely than White patients to experience adverse events in hospitals, which may have nothing to do with the medical devices but instead relate to the care they received from humans. These disparities are not always accurately reflected in event reporting systems within hospitals, as distinct from the reporting system to the FDA. For instance, a study at a Boston hospital found more adverse events among Black patients (may or may not have been reported to anyone) when using an automated system relying on patient records, compared to a reporting system dependent on clinician identification. This discrepancy suggests that patient safety events for some populations, including Black patients, may be under-reported.[4]
Moreover, Black American patients often experience poor communication with healthcare providers, medical mistrust, and perceived discrimination. This leads to a lack of confidence in the healthcare system and can result in the under-reporting of adverse events. Perceived discrimination, especially among Black American women, arises when they feel their symptoms or problems are discredited, and poor communication is noted when clinicians fail to acknowledge patients' perspectives.[5]
Net, if anything, these studies suggests that some adverse events involving Black Americans failed to bubble up all the way to FDA, causing these numbers to proportionately underrepresent adverse events experienced by Black Americans. As already noted, this is consistent with our calculation that Black Americans represent 13.6% of Americans, but only 11.7% of the MDRs where race is reported. Thus, we think this means our chart is reasonably conservative.
Again, there are too many assumptions and gaps in the data to draw any conclusions, but our hope is that this analysis encourages people to start asking questions.
Conclusion
Our main point is that this data is now publicly accessible, and manufacturers should take a look at it to make sure they are not disproportionately impacting any minority. We selected Black Americans just as an example of the methodology, but the data also includes the five standard categories of race set by OMB, as well as data on ethnicity, sex, and other demographic elements.
The Unpacking Averages™ blog series digs into FDA’s data on the regulation of medical products, going deeper than the published averages. Subscribe to this blog for email notifications.
ENDNOTES
[1] https://content.govdelivery.com/accounts/USFDA/bulletins/376cb73
[2] https://www.census.gov/quickfacts/fact/table/US/PST045222
[3] Pinheiro LC, Reshetnyak E, Safford MM, Kern LM. Racial Disparities in Preventable Adverse Events Attributed to Poor Care Coordination Reported in a National Study of Older US Adults. Med Care. 2021 Oct 1;59(10):901-906. doi: 10.1097/MLR.0000000000001623. PMID: 34387620; PMCID: PMC8446307.
[4] https://betsylehmancenterma.gov/news/racial-bias-can-affect-how-patient-safety-events-are-reported
[5] Cuevas AG, O'Brien K, Saha S. African American experiences in healthcare: "I always feel like I'm getting skipped over". Health Psychol. 2016 Sep;35(9):987-95. doi: 10.1037/hea0000368. Epub 2016 May 12. PMID: 27175576.
Blog Editors
Authors
Member of the Firm
