FDA Oversight of AI Software Developed by Health Care Providers

Introduction

Hardly a day goes by when we don’t see some media report of health care providers experimenting with machine learning, and more recently with generative AI, in the context of patient care. The allure is obvious. But the question is, to what extent do health care providers need to worry about FDA requirements as they use AI?

FDA has been regulating machine learning algorithms used in a clinical context for decades. In the last couple of years, each Fall, FDA has been updating a list of machine learning applications in health care that the agency has cleared or otherwise approved.[1] As of last Fall, the list included 521 products dating back to 1995.

While it’s easy to understand that FDA might regulate AI used, for example, in controlling robotics, by far the predominant use of AI has been in clinical decision support software. In that regard, last September, FDA issued a final guidance on when and how FDA regulates Clinical Decision Support, or CDS, software tools because FDA considers them to be a medical device under FDA’s jurisdiction.[2]

Implementing the 21st Century Cures Act from 2016, regulated CDS software includes that intended to “analyze” “medical information about a patient or other medical information” “for the purpose of supporting or providing recommendations to a healthcare professional about prevention, diagnosis, or treatment of a disease or condition.” That’s broad, although arguably not as broad as FDA’s interpretation of it. The statute then exempts any such software that enables the health care professional “to independently review the basis for such recommendations that such software presents so that it is not the intent that such health care professional rely primarily on any of such recommendations to make a clinical diagnosis or treatment decision regarding an individual patient.”[3] Two citizens petitions have been filed against the FDA CDS final guidance, contending it goes farther than the statute provided (including one by our firm), but that’s another story.

In this blog post, we take on the question of when machine learning software developed by those involved in delivering health care is FDA regulated.

FDA Regulation of Medical Devices

For those who don’t already live in the FDA regulated world, we thought we’d start with a brief overview of medical device regulation. Very brief.

Since 1976, with only a few legislative adjustments, the definition of a medical device has been:

(h)(1) The term "device" … means an instrument, apparatus, implement, machine, contrivance, implant, in vitro reagent, or other similar or related article, including any component, part, or accessory, which is-

…

(B) intended for use in the diagnosis of disease or other conditions, or in the cure, mitigation, treatment, or prevention of disease, in man or other animals, or

(C) intended to affect the structure or any function of the body of man or other animals, and

which does not achieve its primary intended purposes through chemical action within or on the body of man or other animals and which is not dependent upon being metabolized for the achievement of its primary intended purposes.[4]

This definition has been remarkably durable, because instead of focusing on a particular technology, it focuses on how products are intended to be used. The definition was, however, amended in 2016 by the 21st Century Cures Act because the role of software in health care had evolved significantly, creating ambiguities that Congress wanted to address.

It’s important to understand that FDA regulation isn’t punitive in the sense that it’s only intended to apply to bad people. It is part of a social contract which says that technology used to treat or diagnose people is so complicated, and in some cases so inherently dangerous, that we require the involvement of a federal agency in the evaluation and ongoing oversight of its use.

FDA regulation shouldn’t cause resentment, as though it is somehow infringing on a basic human right to make products for medicine unencumbered by federal standards. Americans expect, and frankly assume, that the technology used in their health care that has a direct impact on their safety or in the effectiveness of a treatment has been subjected to federal standards. Medical device companies for decades have innovated some truly remarkable products under FDA oversight. The goal of FDA regulation is not to stop innovation, but instead to make sure that it is done responsibly.

That said, it is equally important that federal regulation not duplicate state regulation of the practice of medicine. That’s the point of the next section.

FDA's Legal Authority and Policy Goal

Constitutionally, FDA’s jurisdiction over products is based on the Interstate Commerce Clause found in Article 1, Section 8, Clause 3 of the U.S. Constitution, which gives Congress the power “to regulate commerce with foreign nations, and among the several states, and with the Indian tribes.” We do not want to make this a treatise on constitutional law explaining all the nuances of the Interstate Commerce Clause, including judicial opinions that extend federal jurisdiction to even those activities that merely impact interstate commerce. Let’s just say federal jurisdiction is broad, not very intuitive, and ever-expanding.[5]

But that Interstate Commerce Clause needs to be viewed also in balance with the 10th Amendment of the Constitution, which reserves to the states any powers not delegated to the federal government. According to the Federation of State Medical Boards (“FSMB”):

Unlike many countries with centralized or national oversight of medical practice, the United States uses a state-based system for medical regulation. This is not by accident. The 10th Amendment of the United States Constitution authorizes the states to establish laws and regulations protecting the health, safety and general welfare of their citizens. Thus, it is the responsibility of the individual states to regulate the practice of medicine. ^[6]

Congress chose to express its understanding of the constitutional limitations directly into the Federal Food Drug and Cosmetic Act as follows:

Nothing in this chapter shall be construed to limit or interfere with the authority of a health care practitioner to prescribe or administer any legally marketed device to a patient for any condition or disease within a legitimate health care practitioner-patient relationship. . . .[7]

FDA implemented that statutory provision through a regulation, specifically the regulation for manufacturer facility registration – which serves as the gating requirement for the impact of many of FDA’s regulations. FDA specifically identified the following as not having to register as device manufacturers:

Licensed practitioners, including physicians, dentists, and optometrists, who manufacture or otherwise alter devices solely for use in their practice.[8]

We could go on, but hopefully you get the gist. Implementing a Constitutional limit, FDA does not regulate a licensed practitioner who makes devices for use in his or her own practice.

But as with almost all regulators, what FDA really hates is a gap. FDA doesn’t like to leave a space where FDA thinks there is risk that falls between two regulatory bodies. Thus, when FDA implements this limitation, they look at what regulatory oversight there is of “licensed practitioners including physicians” regarding the particular activities.

FDA Guidance on Software Developed by Health Care Facilities

FDA's Guidance on Software Including Mobile Medical Apps

FDA does not address the topic of health care facilities developing software in its recent CDS guidance, probably because FDA had already developed a policy on this topic in the agency’s September 2022 final guidance on Policy for Device Software Functions and Mobile Medical Applications. In that guidance, FDA provides that the following uses are not regulated by FDA:

Licensed practitioners, including physicians, dentists, and optometrists, who manufacture a mobile medical app or alter a mobile medical app solely for use in their professional practice and do not label or promote their mobile medical apps to be generally used by other licensed practitioners or other individuals. For example, if Dr. XYZ, a licensed practitioner, creates a mobile medical app called the “XYZ-recorder” that enables attaching an ECG electrode to a smartphone, and provides the “XYZ-recorder” to his/her patient to use it to record the patient’s electrocardiographic readings for 24 hours, Dr. XYZ is not considered a mobile medical app manufacturer. If Dr. XYZ is in a group practice (including a telehealth network) and permits other physicians in the practice to provide the XYZ-recorder to their patients, Dr. XYZ is not considered a mobile medical apps manufacturer. However, if Dr. XYZ, the licensed practitioner, distributes the “XYZ-recorder” and, through labeling or promotion intends to make it generally available to or to be generally used by other physicians (or other specially qualified persons), Dr. XYZ would be considered a mobile medical app manufacturer.[9]

Notice that the language simply repeats the language from the regulation on registration, and then builds that out to explain what it means to use software in the physician’s “own practice”. The word “practice” here is important. The September 2022 Guidance seems to be referencing a practice that qualifies under the doctrine of practice of medicine, i.e., a regulated practice. Thus, we think it is likely that FDA focuses on whether the organization that is making and distributing the software is, itself, engaged in the practice of medicine and therefore otherwise subject to oversight by a state Board of Medicine. Whether these state boards of medicine are in a position to provide such regulatory oversight is another story – more on that later.

Medical Device Data Systems

Prior to FDA’s Mobile Medical App Guidance, this topic was addressed by FDA when it created the category of Medical Device Data Systems (“MDDS”). In that situation, FDA was deregulating software that was used to store and transfer data from medical devices. According to the preamble to the 2011 Final Rule:

A purchaser of an MDDS who has only used, configured, or modified the MDDS in accordance with the original manufacturer’s labeling, instructions for use, intended use, original design, and validation would not be considered a manufacturer for purposes of this regulation. If, however, a user makes any modifications to the MDDS that are outside the parameters of the original manufacturer’s specifications for the device, for purposes of the user’s clinical practice or otherwise for commercial distribution, that user becomes a manufacturer under the MDDS rule, and as a result is subject to applicable device regulations, including registration and listing and the QS [quality system] regulation. Likewise, if a user reconfigures any other product into an MDDS for such purposes, that user would also be a device manufacturer subject to applicable regulations.

A health care facility or other purchaser that buys software or hardware that has not been labeled or otherwise denoted as an MDDS, but is used as an MDDS, is not considered to be a manufacturer. If, however, the purchaser adds to or modifies any hardware or software such that the software is intended to provide the transfer, storage, conversion according to preset specifications, or display of medical device data (or otherwise modifies the product to render it a medical device) and uses it in clinical practice, the purchaser becomes a device manufacturer in accordance with § 807.3(d). If a third-party company or hospital develops its own software protocols or interfaces that have an intended use consistent with an MDDS, or develops, modifies, or creates a system from multiple components of devices and uses it clinically for functions covered by the MDDS classification, then the entity would also be considered a device manufacturer.[10]

To understand that passage, it is important to understand the concept of “intended use”. In the second paragraph, FDA is saying that if a health care facility buys a product that is not intended to be used as MDDS, and uses it as intended, it is not a medical device. But what is significant about that passage is the detail provided regarding a health care facility that modifies software for regulated intended purpose, making it a manufacturer.

Although that regulation was published in 2011, FDA still uses it as the foundation for determining whether an organization qualifies as an MDDS manufacturer. FDA’s website provides the following Q&A:

Who is an MDDS Manufacturer?

An "MDDS manufacturer" may be a health care facility or manufacturer that is engaged in the following activities:

• Modifying a general purpose IT equipment/software or infrastructure for purposes of interfacing with medical devices and performing functionality described in the MDDS rule (transfer, store, display, or convert data).
• Labeling a general purpose IT equipment/software as a MDDS for purposes of interfacing to medical devices and performing functionality described in the MDDS rule (transfer, store, display, or convert data).
• Designing and implementing custom software or hardware for purposes of interfacing with medical devices and performing functionality described in the MDDS rule (transfer, store, display, or convert data).

FDA does not consider the following activities to be "manufacturing" an MDDS:

• Purchasing and using a general purpose IT equipment/software or infrastructure for purposes of interfacing with medical devices and performing functionality described in the MDDS rule (transfer, store, display, or convert data).
• Purchasing and using an MDDS marketed by an MDDS manufacturer for purposes of interfacing with medical devices and performing functionality described in the MDDS rule (transfer, store, display, or convert data).[11]

These statements are all written in the context of MDDS, but they would apply more generally to other forms of software intended to be used for a medical purpose. In these passages, FDA is trying to give guidance to companies on the difference between general IT software, and IT software that qualifies as a medical device manufactured by whomever made or modified it.

FDA Guidance on Other of Health Care Provider Activities That FDA Regulates

There is a long history of FDA asserting jurisdiction over organizations that are traditionally involved in providing health care services. Whatever the motivation a health care provider may have—including undoubtedly improving patient care, the desire to produce innovation, and in some cases the desire to save money—health care organizations, especially large ones, often decide to foray into what might be considered to be the manufacturing of medical devices, which would then bring such activities within the jurisdiction of the FDA. The results can be long, drawn-out battles between FDA and these health care providers. We provide just four examples below.

Lab Developed Tests

The term “lab developed tests” (“LDTs”) refers to diagnostic tests developed by clinical diagnostic laboratories. For years, laboratories were allowed to develop their own versions of tests to meet some unique need of the lab’s patient population, typically in instances where commercial tests were not offered. But the business grew and grew, and started to compete with manufacturers providing in vitro diagnostic test kits.

Initially, FDA simply exercised enforcement discretion, but more recently the agency has announced plans to actively regulate these privately developed and private used lab tests. FDA considers a LDT to be “a type of in vitro diagnostic test that is designed, manufactured and used within a single laboratory”. FDA takes the position that it has authority over LDTs.[12]

Clinical laboratories are separately regulated by the states as well as by other federal agencies such as Centers for Medicare & Medicaid Services (“CMS”) under the Clinical Laboratory Improvement Amendments of 1988 (“CLIA”). However, the two regulatory systems are not the same, where CLIA tends to focus on the laboratory and its operations while FDA tends to focus on the particular product itself. In general, FDA has maintained that it has clear regulatory authority over LDTs, as it does with all in vitro diagnostic tests that meet the definition of “medical device” in the Federal Food Drug and Cosmetic Act.[13]

Hospital Reprocessing of Single Use Medical Devices

Medical devices can be quite expensive. Many single use medical devices are of high quality and, in the opinion of the users, can be cleaned to the point where they can be reused. To save money, hospitals and other users began reprocessing those single use medical devices. FDA took the position that such reprocessing was a manufacturing operation because it extended the life of what was otherwise a single use medical device. FDA came out with guidance explicitly explaining its reach over hospitals engaged in those reprocessing efforts,[14] and in 2002 Congress validated that extension with an amendment to the underlying statute.[15]

Pharmacy Compounding (Drugs)

While involving drugs instead of devices, another example of FDA regulating organizations involved in providing health care services is where pharmacies made or modify drugs to meet the unique needs of a particular patient. For example, a pharmacist might grind up a tablet and put it in a suspension liquid with cherry flavor for a pediatric patient. But as with the other practices listed here, the practice of pharmacies compounding large volumes of drugs grew rapidly. FDA has made it clear that the agency interprets its statutory authority as extending to pharmacies engaged in the compounding of drugs in advance for more than one patient.[16]

Stem Cell Clinics (Biologics)

According to FDA, stem cell therapies may offer the potential to treat diseases or conditions for which few treatments exist.[17] Sometimes called the body’s “master cells,” stem cells are the cells that develop into blood, brain, bones, and all of the body’s organs. Stem cells have the potential to repair, restore, replace, and regenerate cells, and could possibly be used to treat many medical conditions and diseases.

There are literally hundreds of organizations that are generally referred to as stem cell clinics, offering various treatments through stem cells. If you Google the term “stem cell clinic”, you would find a Google map that includes the closest clinics to you. But generally, the products that these clinics are dispensing, FDA would argue, are unapproved biologics regulated by FDA. The agency has been very clear that it believes it has legal authority over such products.

The point of all four of these examples is that health care providers of all types that drift into the production of medical devices should expect at some juncture FDA regulation.

The Scope of the Practice of Medicine

States regulate health care professionals and what they do. FSMB explains how state Boards of Medicine manage errors by physicians such as diagnostic errors or errors in therapeutic judgment:

The ongoing duty of a state medical board goes far beyond the licensing and ongoing registration of physicians. Boards also have the responsibility of determining when a physician’s professional conduct or ability to practice medicine warrants modification, suspension, or revocation of a license to practice medicine. Boards safeguard the public by disciplining physicians who engage in unprofessional, improper, or incompetent medical practice.^[18]

In this post, we want to distinguish between two different activities of health care professionals. The first type is physicians using their clinical knowledge together with perhaps special skills in data science to develop innovative machine learning models that they use in their own practice. We would contrast this with the second type, which are health care provider organizations that may or may not employ physicians but who also hire computer scientists and data scientists to design and create machine learning models with input from physicians.

Physicians Designing Software for Their Own Use

We are excited by the reports in the media that there is a new breed of physician innovators who bring their medical knowledge and data science skills to solving problems. We have heard reports of such physicians designing algorithms that they use in their own practice. We think such physicians ought to be safe from FDA regulation.

Hospitals and Other Health Care Organizations Hiring Software Engineers to Write Code, With the Input of Physicians

As a preliminary matter, we should point out that some hospitals and other health care organizations cannot hire physicians because in some states that would violate the corporate practice of medicine.[19] That’s an important point, because it emphasizes that many corporations cannot by law engage in the practice of medicine and therefore are not regulated by state Boards of Medicine. As such, FDA regulation of their activities would likely not violate any statutory or constitutional prohibition.

Perhaps more to the point, state Boards of Medicine are not set up to regulate software development by non-physicians. There is nothing about their organizational structure nor their statutory mandates that would suggest that they are in the business of reviewing software developed by health care providers.

We would note that virtually every software with a clinical purpose is designed with the aid of licensed physicians giving input throughout the lifecycle. But clearly the fact that a physician offered advice and guidance on the design of software to be made available to other physicians or even to be used in that clinician’s own practice does not necessarily render the resulting software unregulated by FDA.

Conclusion

To create a machine learning model for health care, all you really need is a computer, access to the cloud and access to training data. Many people who work for health care providers have all of that. Many, if not all of them, also are truly committed to helping patients live better healthier lives. It is a natural temptation, and indeed laudable, to try to create machine learning models that will improve patient care.

But those good intentions and the fact that their employer is engaged in health care do not exempt such innovators from FDA oversight. FDA serves an important public health purpose of externally validating algorithms that impact patient care. Health care organizations need to be vigilant to ensure that their well-meaning employees do not drift into FDA regulated territory accidentally, and if they do, to be sure that such innovations are in compliance with FDA regulatory requirements.

For additional information about the issues discussed above, or if you have any questions or concerns related to AI, please contact the Epstein Becker Green attorney who regularly handles your legal matters, or one of the authors of this blog post. Read more about our expansive AI capabilities and offerings here.

---

[1] FDA, Artificial Intelligence and Machine Learning (AI/ML)-Enabled Medical Devices, available at https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices.

[2] FDA, Guidance Document, Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff (Sept. 2022), available at https://www.fda.gov/regulatory-information/search-fda-guidance-documents/clinical-decision-support-software.

[3] 21 U.S.C. § 360j(O).

[4] 21 U.S.C. § 321 (emphasis added).

[5] https://www.thefdalawblog.com/2011/10/fda-continues-efforts-to-expand-power-over-intrastate-commerce/

[6] Federation of State Medical Boards, Understanding Medical Regulation in the United States (last accessed Jan. 25, 2023), at https://www.fsmb.org/siteassets/education/pdf/best-module-text-intro-to-medical-regulation.pdf.

[7] 21 U.S.C. § 396.

[8] 21 C.F.R. § 807.65(d).

[9] FDA, Policy for Device Software Functions and Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff (last issued Sept. 28, 2022), available at https://www.fda.gov/media/80958/download (citations omitted).

[10] 76 Fed. Reg. 8637, 8645 (Feb. 15, 2011)

[11] FDA, Requirements for Health Care Facilities and Manufacturers, available at https://www.fda.gov/medical-devices/medical-device-data-systems/requirements-health-care-facilities-and-manufacturers.

[12] FDA Draft Guidance, “Framework for Regulatory Oversight of Laboratory Developed Tests (LDTs)” (October 2014), available at https://www.fda.gov/media/89841/download.

[13] https://crsreports.congress.gov/product/pdf/IF/IF11389.

[14] FDA, Frequently-Asked-Questions about the Reprocessing and Reuse of Single-Use Devices by Third-Party and Hospital Reprocessors; Three Additional Questions, available at https://www.fda.gov/media/71124/download.

[15] Medical Device User Fee and Modernization Act of 2002, Report 107-728 (Oct. 7, 2002), at 21.

[16] FDA, FD&C Act Provisions that Apply to Human Drug Compounding, available at https://www.fda.gov/drugs/human-drug-compounding/fdc-act-provisions-apply-human-drug-compounding.

[17] FDA, FDA Warns About Stem Cell Therapies, available at https://www.fda.gov/consumers/consumer-updates/fda-warns-about-stem-cell-therapies.

[18] FSMB, About Physician Discipline: How State Medical Boards Regulate Physicians After Licensing, at https://www.fsmb.org/u.s.-medical-regulatory-trends-and-actions/guide-to-medical-regulation-in-the-united-states/about-physician-discipline/ (last accessed Jan. 25, 2023).

[19] https://www.ndlegis.gov/assembly/67-2021/testimony/SHUMSER-2128-20210125-3125-F-TRAYNOR_PAT.pdf