Intertech Engineering Associates, Inc.

The Essential Performance SNAFU part 4 of 4

Actual cases and what manufacturers can do to avoid a SNAFU

Intertech Engineering Associates Inc. has seen a recent trend of problems manufacturers are experiencing that are related to the consequences of decisions made (SNAFU’s) after Essential Performance is defined. Unfortunately, guidance is not clear to manufacturers and some test houses might be making matters worse by not providing clear interpretations to the manufacturers.  Here are two examples of cases Intertech has encountered over the last year.


Observation Case 1: CE Marked Device From Outside of the US, Missing FDA Expected Software Validation

Medical device manufacturers has received a CE mark, are running into trouble when trying to bring their devices into the US, when software is part of the device.  The products are getting caught up in the FDA 510(k) clearance process and are deemed to not have adequate software documentation.


Intertech has been assisting clients, who need to address FDA deficiencies after withdrawing their submissions as a result of receiving multiple questions from the FDA. In some cases, manufacturers are interpreting (per the IEC 60601-1 standard) that if the defined essential performance does not include the product software, that software validation is not required.  Interpretation of clause 14 (PEMS) ofIEC60601-1 is not what drives the FDA 510(k) review process.  The FDA still expects not only a process for software development to be followed but the software outputs from this process to be fully validated, specifically the product software, regardless of whether the software is part of Essential Performance.


The FDA has indicated, as of 2015, 69% of 510(k) submissions result in AI (Additional Information) requests from the FDA. A prior FDA analysis of review times concluded that up to 20% of the submissions have deficiencies with a key finding that the software documentation is inadequate. 1


If medical devices are first marketed outside of the US and have the software but don’t have software validation, they will have problems getting FDA clearance to market in the US.


Observation Case 2: Essential Performance Replacing Risk Analysis

Manufacturers and some test houses are determining that the Essential Performance characteristics are the same as the identification of functions that require risk controls. In this case, a medical device manufacturer can conclude that if software controlled functions are not part of what is deemed Essential Performance or Basic Safety in a medical device, the software itself need not require risk analysis. This approach, in essence, replaces (or would be in conflict with) further risk review of the software as suggested by ISO14971 (Medical devices — Application of risk management to medical devices), which requires an evaluation of hazardous situations and faults that may arise from software. Additionally, hazard analysis is necessary to support safe software as described in the FDA software validation guidance (General Principles of Software Validation).


Intertech is finding that manufacturers are relying solely on the process of defining Essential Performance and not performing functional failure (safety) analysis of their software design or implementation. Failing to analyze fault conditions to identify the hazard or hazardous conditions is a necessary part of risk analysis, and if manufacturers avoid software fault analysis, because a top-down determination of what is essential performance, potential hazards will be missed and be unmitigated. In addition to the safety concerns posed by skipping a software level risk analysis, this poses a compliance issue with the FDA and with the company’s notified body caused due to the missing analysis.


The possibility of cutting short a risk analysis task is appealing to a manufacturer,  even more so if a test house suggests some tasks are not required. Cutting tasks during the certification or clearance step in the process is easily perceived as a budgetary advantage, particularly since the compliance tasks budgets are typically underestimated and the schedule is tight at the end of projects.


Another situation Intertech is encountering is if the medical device software is not considered a part of Basic safety or Essential Performance,  manufacturers and some test houses are deciding that IEC 62304 (Medical Devices – Software Lifecycle Processes) as well as IEC60601-1-8 (General requirements, tests, and guidance for alarm systems in medical electrical equipment) may not apply. Currently, IEC62304 is an FDA recognized standard used to define the activities necessary in software development and IEC60601-1-8 is the standard that defines requirements for alarm systems. Neither of these standards specifically states that product software needs to be part of Basic Safety or Essential Performance as a prerequisite for compliance.


Tactics to avoid the SNAFU

There are some steps manufacturers can take to avoid the pitfalls and improve product quality. Consider the following takeaways to protect you, as it relates to software and essential performance:


Basic Safety and Essential Performance alone is not enough

Basic safety and essential performance are important characteristics for compliance and are key drivers for IEC60601-1 compliance, but they are not the only things that should give manufacturers confidence their product is safe. Software validation, proper risk analysis, software development process and basing decisions via utilization of the risk management process in addition to the compliance activities.


Perform Software Validation

Software validation is not waivable in any medical device. The selection of software validation activities for medical device software should be commensurate with the complexity of the software design and the risks associated with its implementation and the impacts on the intended use.


Software Validation is not just testing

Software validation is not just testing; software validation is about building confidence in the software, which gives the manufacturer, and the regulator confidence in the medical product. The process of validation and building confidence includes addressing activities of a software development lifecycle applied during and after the software is developed. David Vogel highlights in chapter 3 of his textbook, (Medical Device Software Verification, Validation, and Compliance), which software validation is about building confidence that is driven with the philosophy of two prongs of confidence building which include prevention as well as defect elimination.


Perform software safety analysis

Manufacturers developing medical devices with software should put into place, or seek a supplier that has in place, a process to analyze software and evaluate the consequences of software failure on the system during the software design and implementation efforts. This analysis is to be performed on medical device software regardless of the essential performance characteristics, device classification or the software level of concern.


Manufacturers are accountable for product safety

Manufacturers should remember that they are responsible and accountable for their product safety, not the test houses used to provide certifications to standards. If interpretations from a test house do not seem appropriate, question the rationale, as many good test houses are open to the discussion and providing rationale. If the rationale or answers to questions on interpretation to standards from a test house does not result in an agreement, contact your notified body or seek alternative input from a recognized authority.



Pharma Companies Challenges with Combination Devices and Planning for Variation Part 3 of 3

Part 3 of 3: The Sources of Variation in a Combination Device Timeline and Dealing with Software

Combinational products require a teamwork from personnel that requires specialized skills to ensure a fully integrated solution that meets the user needs. Some particular high-level characteristics that are important to design variation to consider when planning and scheduling are listed in the table below:


Drug/Biologic Component Characteristics

  • pharmokinetics
  • container (materials science)
  • pharmodynamics
  • drug stability
  • process controls (particularly for biologics)
  • formulation
  • Clinical data (animal/human)

Disposable Component Characteristics

  • container closure system (CCS)
  • sterility/sterilization
  • manufacturability
  • material variability
  • disposal
  • material compatibility
  • interface
    • drug/reusable
    • user/reusable
  • usability


Reusable Component Characteristics

  • durability
  • reliability
    • for life of product
    • durability of reuse
  • industrial design
  • storage between use
  • software safety systems
  • power and charge
  • electrical safety
  • performance impact of drug  delivery
  • usability
    • human interface
    • disposable interface
  • service
  • EMC
  • Manufacturability of reusable
  • component selection (specifications)

Managing Variation When Software is Part of the Medical device:

As mentioned earlier, the design process can be iterative where inputs/goals into the design process are updated as the actual characteristics of integrated systems are tested and characterized/ measured and compensations are made to optimize.  Iterative development of software can be produced using methodologies such as agile or using scrum methodology or tools, but iterative approach need not be limited to software alone. For example prototype parts and assemblies are developed for technology proofing and feasibility; as these parts change or are molded parts they will have the impact on the overall system. These impacts can be driven by changes to material properties as well as dimensional properties. Testing evaluations are often necessary, even after mathematical analysis to characterize new and changing sources of variation and recognition of the impact of interactions with other parts of the system and the overall impact on drug and biologic delivery.


In a medical device that can be controlled by software, there is an opportunity to mitigate the impact of variability due to the hardware, by software means. The operation of the device can be designed to compensate for detected or known variability, by such means as using feedback control. Adding software control loops in a device design is an example of how additional complexity is added. Complex design requires managing a highly dynamic set of conditions. The determination of if software requirements are met or identifying the shortcomings in executable software are understood when frequent testing integrations are planned for. Testing can be used to confirm development progress as well as providing data to evaluate progress in managing variation in device performance characteristics.


In order to optimize product performance, the key device performance characteristics can be defined early enough for teams to know what problems to solve and what performance to optimize during the product development process. Applying a data-driven process cycle described above, or using quality process approaches such as DMAIC (Define, Measure, Analyze, Improve and Control) support frequent measurement of performance and lay the groundwork to optimize the product design.


Software itself is not a victim to variability; software is repeatable it will perform the instructions the same way each time, but as a part of a system, the software can be a variation solution but also if poorly designed, a significant source of error impacting the ability to meet key system objectives. But that is not to say that software does not contribute to the variability in device performance, it surely does. Errors or omissions in the software design, including algorithm design, and implementation of software code itself does impact the device performance once integrated with the hardware. Testing software upon development iterations helps the developer more easily spot these flaws during development.


Software is used to implement algorithms to manage sensors and handle data which can impact the output performance. Software frequently is used to implement control and or manage variability, either by logic to identify variability and to compensate for identified variability or by indicating to the user to react to unexpected operation occurring. Adding control loops and feedback to product software means that software failure can have unintended impacts on operation and variability. Here are some examples of failures that can impact a combination device:

  • Algorithm Design Failure: algorithms can fail to identify or incorrectly compensate for variability. If the characteristics of the signals being monitored are not correctly recognized the intended system compensations will not be applied when desired.
  • Algorithm interaction: algorithms can have indirect impacts to drug delivery, such as delivery profile does not maximize the particular half-life of the drug and desired length of time and effect the drug should persist in an individual is not optimized.
  • Software Functional Failure: errors in software design and implementation can lead to system failures when the correct combination of events occurs. These are systematic and not random in nature but can be hard to identify and can be long time latent bugs in the software that can lead to system failure at the worst times.


Adherence to a software development process, which allows frequent build-test-optimize, allows teams to integrate software into the device and evaluate the operation and detect failures.



Planning of the lifecycle of a combination product project should be done such that the development of the drug and each of the device components development can be aligned with integration points in mind. There are periods where parts of the device need to be developed to a certain degree, to support the combination product.  Not planning these points may cost additional time to an already long development phase of the product lifecycle.


There are other processes that a pharmaceutical company may not have in place, that need some planning and prep time. Some of these processes may be centered on quality systems necessary for manufacture and during the maintenance phase of the lifecycle, such as device change controls, complaint reporting, unique device identifiers, and service of durables.  But there are important processes that need planning and implementation time that is part of the development phase, and three important ones include considerations for usability and usability studies, adoption of a preproduction configuration management process, and a design control process for medical devices, and if necessary software development which can be complicated.


Pharmaceutical companies who pursue a combination device eventually recognize that variability in the manufacture and design of a reusable and disposable medical device become an additional burden that can stretch out launch times, once major drug milestones and completed. A device development process that is done under controls and is focused on the evolution of a device concept that is clearly defined, designed, documented, tested and optimized is one more likely to succeed. The companies that can target design team activities and project milestones that adopt principles such as DMAIC (Define, Measure, Analyze, Improve and Control) or DMAIC like approaches for measuring performance and proofing design and manufacturing solutions see better development success.

Pharma Companies Challenges with Combination Devices and Planning for Variation Part 2 of 3

As discussed in the first part of this article series there are three important device development activities that are frequently overlooked or underestimated by drug manufacturers in a combination device project. From a project planning perspective, these activities should drive key project milestones, during a combination product development phase of the product lifecycle. This part of the article series covers these keys in more detail.



Key Activity 1: Manufacture of the devices for the development lifecycle:

An important integration point between the drug development timeline and device development timeline to consider is the device build milestones that are needed for each clinical trial. Multiple device builds will be intended for the assessment of safety, effectiveness and include device performance. The Phase 1 clinical requires attention to safety, but later clinical objectives require the device manufacturer to assure the devices used meet important effectiveness criteria to provide assurance in the conclusions of the clinical studies. Devices used in phase 2 and 3 clinical trials require building devices with a known configuration and a configuration management process to be in place.


A pragmatic device design process includes iterations or phases of controlled design, the evaluation and optimization of the design based on specific controlled inputs (or requirements) as well optimization of outputs to address performance as the development commences. The outputs of the design during iterations or phases should be managed and have defined configurations. If the configuration is unmanaged or unknown the design progresses and changes, and perceived improvements and optimization is not controlled or managed. Device development without configuration management would be considered an uncontrolled development. Making changes to a device in this type of an uncontrolled development is no better than ‘trial and error’ development. Confidence in the design and in the future performance of the device is more difficult to assess without some design controls. Device developers find that changes to the design based on results from an uncontrolled system result in incorrect conclusions which lead to development delays due to erroneous data or conclusions.


A configuration management process starts with identifying the items that are to be maintained under configuration control.  Configuration management includes the tracking, control, baselining, storage and providing for the controlled change of defined items.  Typical items that will require configuration management are: records/documents (i.e. requirements, design descriptions or specifications), material composition, software versions/builds, hardware versions, sub-systems, assemblies, and components, manufacturing test systems, calibration methods, as well as test units and units under test themselves.  A procedure or plan should identify how a particular configuration item is to be managed. Within a project configuration, items will be developed and updated.  These configuration items are defined by documents that are to be controlled. It should be determined which documents and items are to be controlled and when, as well as the process to keep items and documents in sync and a process for how changes are made.  Typically an effort to sync configuration documentation should be made part of each iteration or phase planned for.


Key Activity 2: The planning for usability:

Another important integration point between the drug timeline and device timeline to consider is device usability. The FDA frequently states that manufacturers don’t adequately consider use errors, and this leads to clearance issues with the FDA.  The FDA is particularly concerned with medical device developers addressing human factors in their design and testing their design in usability studies to address use errors. Planning the usability studies on the timeline is important to the overall planning of a combination device.

Formative usability testing can be aligned with the timing for some the phase 1 and phase 2 clinical studies done with potential patients and users. Formative testing is used to drive the determination of the design as well as help to capture requirements for the user interface, etc.. Test and evaluations with users and potential patients should be focused around device tasks were anticipated safety is involved and areas where the interaction with the user may be particularly complicated. The evaluation or testing really allows the developer to explore and understand the design options to better suit use conditions and better meet user and or potential patient needs. Drug and biologic design lifecycle is longer, but not planning for formative usability of the device will result in delays much closer to the end of the drug development.


Usability analysis does not just focus on device failure hazards; it also includes identifying and applying measures to eliminate or reduce use-related hazards. These hazards might result from aspects of the user interface design that cause the user to fail to adequately or correctly perceive, read, interpret, understand or act on information from the device.

Screens used for the human interface can be presented and prototyped, in a rapid fashion using high language tools and graphics development kits. When necessary a graphical user interface (GUI) can be generated and quickly evaluated before device function is needed in the development lifecycle for the device. These rapidly developed screens can then be integrated into the device design at a later step of the development process.

The final usability study/test is referred to as the summative usability study. Summative testing are tests with users to validate the final configuration meets the user need and provide for safe and effective use.

Key Activity 3: The coordination and application of design controls to address sources of variability:

Another important set of key activities is applying device development controls and practices to address product variability and performance. The FDA regulatory design controls are defined per the regulation part 21 CFR 820.30. When the device is defined, designed, built, analyzed for risks, verified and validated can all be plotted along the device development timeline. The device design part of the lifecycle can take many different forms, such as more of a serial set of phases or more iterative set of design builds and these can be represented in the project timeline as corresponding milestones as a device design is developed.

Much of the challenge of scheduling of the design of the device is the uncertainty associated with proofing design output solutions, identifying tasks to target project risks, factoring time to solve problems and driving out variability. Optimizing the design comes down to determining the best technical solutions to meet the needs of the users and the requirements. A good design control process provides managers and designers with clear visibility of the design process and the objectives targeting a short cadence of accomplishments. Having a short manageable time frame of tasks and good visibility of the goals for that time frame allows the team to react and effectively manage the design process. Within a timeframe, a team can specify the need/activities to be addressed, identify the problems to solve and the associated project risks than in the timeframe formulate solutions or make corrections to the current design and then evaluate or verify the intended objectives are met and characterize the performance of the solutions.


The project manager for a combinational device should look over the horizon and plan activities and milestones in a schedule to target the effort in a controlled process. To address the most significant risks in a device a controlled process should be put in place to target the sources of variability.

One approach to targeting sources a variability in the combination product

Based on experience the development lifecycle the management of project milestones and its integration points help to ensure the development team stays on target. As the project progresses and objective driven milestones are reached, project risks associated with milestones are being reduced. Similarly, setting device performance milestones can also positively impact variability in the design outputs. What is difficult is planning for what is unknown as the design evolves and iterations are measured.

In a nimble development model applying iterative cadence to define the goals, measure the outputs, analyze the resulting data, and optimize the design has the best results. Successfully driving this controlled cadence is effectively achieving a design control. One approach to driving this cadence in the development lifecycle should be the adoption of scheduling a set of milestones to address the variables that contribute to uncertainty and variability in the important product characteristics and primary mode of action.

At these milestones the evolution of the device/system should include meeting the functionality goals, characterizing and measuring operational outputs as well as optimizing them to meet the performance needed or expected by the end users. The project schedule can be developed indicating the milestones across the list of tasks used for resource management as in the case of the simplified figure below.



(milestones schedule figure)


The simplified Gantt presents a project schedule that is driven by achieving target milestones. These milestones are not going to be the same for each project. They should be determined by considering the overall timeline, the major phases of the drug and device clearance process, and identifying key activities and setting goals and quantifiable objectives to resolve these. Frequently key milestones come from a project risk evaluation that is part of the initial planning process and is consistently evolved through the project, much like the schedule is.

The next section of this article series will dive deeper into sources of variation and how software in a medical device is part of the variation challenge and solution. We would be interested in your own experiences with milestone planning and on tackling key activities in a combination medical project. Please add your comments or questions to keep a dialog going.

Medical Device Validation Series 4 of 4

Part 4, Inadequate requirements usually result in inadequate testing.

This is part four of a four-part series on medical device validation practices.

Here are some telltale signs of inadequate requirements:

  • Software developers need constant access to market and clinical specialists to understand how they want the software to work.
  • Code writing begins before requirements are written or approved.
  • Test authors need frequent access to the software developers to understand how the software should work.
  • Formalized testing finds few if any defects, while “experts” (e.g. developers) with the system find many defects in ad hoc (It could be inadequate testing too, but usually it is a combination of poor requirements and poor testing.)
  • Review of test results is time-consuming and requires “experts” with the software to determine whether tests passed or failed. Note that this does not meet the standard of objective evidence of test results if the results have to be interpreted.
  • A preponderance of tests fail by the testers but are subsequently passed because the “testers didn’t understand how the software works.”

If these red flags fly, you need to go back and rewrite the requirements, making them specific enough so that they can be tested properly. Adequate requirements are the foundation of testing and should be the foundation of software implementation too.  A little extra up-front effort can prevent the whole structure from collapsing later. While testing alone doesn’t amount to validation, proper validation demands testing, and testing depends on testable requirements.

From: Vogel, David A., Ph.D. “Validating Medical Device Software Includes and Goes Beyond Testing.” Medical Product Outsourcing Mar. 2006: n.

Pharma Companies Challenges with Combination Devices and Planning for Variation Part 1 of 3

More and more now Pharma companies are now taking on medical device development as they intend to provide their customers with combination devices. Planning the device development does not always get the attention in project planning as it needs from a project management perspective since the drug development usually takes precedence. Scheduling a drug and a device, in parallel, requires careful management to coordinate the many interactions between drug development and the device development. Planning and scheduling the lifecycle for the product, particularly the development phase, should include integration milestones for the drug and device together. The development long pole is, obviously the drug development and regulatory clearance path, but this does not mean a combination device manufacturer should leave paying attention to the device development for the end. One challenge that is frequently underestimated or overlooked is considering and planning for addressing device variation and its impact on the overall combination devices performance. Combination device developers should plan an iterative device development model as well as integration milestones as a part of the development phase of the lifecycle to also address this. This article series discusses some Pharma company project management approaches as well as some key activities frequently overlooked or underestimated.   


Part 1 of 3: The Combination Device Timeline  

In understanding the development lifecycle for a combination device, the overall milestones to a market release in the US is defined by the regulatory objectives. Developing a pharmaceutical drug requires much in research and testing as it relates to the science we will not cover in this article series, but a high-level explanation of the FDA expected regulatory path is where we will start.  


What is a combinational device? 


The FDA has defined a combination product as:  


“…a combination product is a product composed of two or more different types of medical products (i.e., a combination of a drug, device, and/or biological product with one another). The drugs, devices, and biological products included in combination products are referred to as “constituent parts” of the combination product.”


What are the regulatory pathways for a combination device? 

The development team needs to map out the development process and prepare to develop the product, drug and or the biologic iteratively over a development lifecycle. To do this the developer with regulatory oversite will overlay the device process with the longer drug development timeline.  The timelines for a drug are driven by the regulatory strategy considerations and may be dictated to the manufacturer based on which center has jurisdiction for the product.  The developer can file for an RFD (Request for Designation) and additionally provide a Pre-RFD to get quick feedback on how the FDA will classify the combination product as well as identify the jurisdiction. The jurisdiction is driven by the definition of the primary mode of action (PMOA) and is assigned to one or more of the three possible FDA agency centers by OCP. Additionally, the combination will have one primary jurisdiction center determined based on the PMOA. The FDA centers are: 

  • CDRH – Center for Device and Radiological Health  
  • CDER – Center for Drug Evaluation and Research 
  • CBER – Center for Biologics and Research 
  • OCP – Office of Combination Products 


If the Primary Mode of Action is driven by the drug the primary center of justification will be CDER, if it is the device, it would be CDRH, if it is a biologic then CBER. The FDA can help define the PMOA, as the option to request for a designation RFD is available to manufacturers.  


The next consideration is for the developer to plot out the regulatory submission types once the drug is far enough and ready for controlled laboratory and animal testing. Submission application types can include:  

  • an investigational device exemption application (IDE);  
  • an investigational new drug application (IND);  
  • biologics license application (BLA);  
  • new drug application (NDA); or  
  • premarket approval application (PMA).  
  • Other types of applications that might pertain to a combinational device include (premarket notification 510(k) or abbreviated new drug application ANDA). 


A manufacturer of a combinational device will likely be required to undergo clinical trials under a new drug application (NDA), investigational device exemption (IDE) or an investigational new drug (IND) application. Once the submission pathway is known the clinical trial strategy can be defined identifying the clinical trials as being key milestones for the development process which drives the first stages of the product lifecycle.  


What are important considerations that drive pharmaceutical and device development? 


Drug development requires preclinical evaluation, screening formulation, toxicology testing, pharmacology and biochemistry testing, just to name a few. All of these testing components represent many person-hours of effort that make up the first part of drug development. This is the basic careful science necessary before a manufacturer is even close to clearing a drug or thinking about a combinational product, to deliver the drug.  Once a drug enters the phased clinical step there is a more defined framework to schedule things around. These clinical studies are completed in three premarket clinical phases. 


As mentioned already, that just getting to the clinical phase of drug development is one significant effort, but getting through the three clinical phases, is also a no easy task. There are three key clinical phases and trials for a drug, each having a particular objective necessary to support FDA clearance.  The phase 1 clinical trial is intended to support primary safety. The phase 2 clinical trial will support short-term safety, but it is mainly intended to support effectiveness. Finally, the phase 3 clinical trial is intended to demonstrate safety, dosage, and effectiveness. According to a BIO report looking at clinical phase transition success and likelihood of approval (over a period from 2005 – 2015), drug manufacturers only have a 9.6% chance of successfully getting from phase 1 clinical to drug approval (through all 3 phases).  When a manufacturer decides that they wish to market the drug with a device, most frequently facilitating a delivery system, they really should decide this during the early clinical phases so the device development can occur early enough to support the clinical trials.  


For a combination device, the clinical submission and trials will serve to support the drug approval, but it will often support the device safety and effectiveness as well. Understanding this basic framework of the clearance establishes a timeline where some key device activities and milestones can be mapped. For a combination device, aligning the timelines between the drug and the device becomes critical. 


Developing a drug, from the molecule, through approval takes 10 to 15 years and can cost developers over $1 billion. Device development, from concept to approval can take 2 to 5 years and can cost developers tens of millions of dollars. Because of the cost and risks in both the device and drug development many companies developing combination devices will attempt to choose a drug and or a device that has been cleared for other similar uses and therefore proven to a degree that is possible, for the combined effort. The below example timeline is scaled to an aggressive overall timeframe and is intended to demonstrate relative time periods and some key milestones of the device development over the drug timeline. 



(Timeline  Figure 1) 

The example timeline figure above shows the drug regulatory phases with the device development lifecycle represented on the bottom. The device lifecycle represented reflects the design control elements expected in a common sequential waterfall development lifecycle. This does not suggest that a waterfall lifecycle is explicitly required but intends to represent the primary goals for the first baseline during this time period. It is expected and practical to expect that planning and requirements change, as well as the design, throughout the whole lifecycle. Good design control process should be established to ensure each of the sequential elements of design controls is applied for iterations, as the product evolves and the design output matures. 


At this point, we can discuss three important device development activities that are frequently overlooked or underestimated by drug manufacturers in a combination device project. These activities drive some of the milestones we should establish in our development lifecycle planning.  Three important key device activities are:  

  • Key activity 1: the sourcing or manufacture of devices to support the development lifecycle (particularly for phase 2 and 3 clinical and device verification). What is important about these devices is maintaining an understanding of the configuration of these devices, so interactions and impact on the clinical studies is understood;  
  • Key activity 2: the planning and coordination of usability studies. What is important about usability is that the results drive requirements, design as well evaluation and managing of particular safety risks; and 
  • Key activity 3: the coordination and the application of the appropriate design controls during development to address sources of variability and optimize the device design. What is important about managing the design process is that doing so adds assurance to the maturing of the actual product as well as demonstrating through documentation of the inputs and outputs the developer can establish confidence in the results testing and performance.  


The three keys will be further discussed in the next section of this article series.  We would be interested in your own experiences with these or based on your experiences what other areas are just as or more important.

The Essential Performance SNAFU part 3 of 4

As discussed in a previous part of this series, the family of the general standard IEC60601-1 includes collateral and particular standards that might apply to the manufacturer’s particular device. This part of the article series will highlight what collateral standards to IEC60601-1 have in them as well as what the particular standards address with regards to performance and essential performance of a medical device.


Examples of IEC60601-1 Collateral Standards

Collateral standards cover broad subjects, and for IEC60601-1 there are several. Below is a list of the current popular collaterals to IEC60601-1 and a brief description of what they cover:

These standards may apply to a manufacturer’s device and typically add additional requirements to the clauses of IEC60601-1 and include the testing and testing fixtures that would apply. When applying these standards, testing would require that the determined Essential Performance (EP) be evaluated during the test. After applying testing per the collateral standard there is an evaluation of the essential performance and a determination is made by the tester if EP is impacted to a point where harm can occur. Adding the Essential Performance to the test criteria was all new to certified test houses, used to evaluating only Basic Safety. Evaluation and monitoring of Essential Performance during and after testing requires a more advanced understanding of the device’s clinical impact to correctly conclude on the device’s safety. In reality, the acceptance criteria got a lot harder for testers and test houses to appropriately distinguish a passing device from a failing device.

The Essential Performance characteristics are the aspects we check for safety in standards tests; therefore there is a consequence if the EP items are not chosen correctly. If lower level, technical characteristics are identified as the Essential Performance characteristics, then we may incorrectly fail a device, where safety is not truly impacted.  If true Essential Performance characteristics go unidentified and are not addressed or tested to a point there is a potential for harm, then testing would be ineffective and the device may be released with safety inadequately addressed. This puts a lot of selecting what the Essential Performance characteristic is correct.

What Does System Engineering tell us about determining Performance measures?

Performance measurement from a system engineering perspective may include the consideration of measures of effectiveness and measures of suitability. Measures of effectiveness are measures that correspond to the accomplishment of mission objectives and achievement of the desired results. These performance measures are determined through a capabilities-based assessment of the system and its objectives. Examples of these performance measures include the characteristic of measure of a system’s performance expressed as a speed, payload, range, time-on-station, frequency, or other distinctly quantifiable performance feature. Performance measures considering suitability include a performance measure of an item’s ability to be supported in its intended operational environment. Examples of these performance characteristics typically relate to readiness or operational availability, reliability, maintainability, or the item’s support structure.

The above characteristics of performance are different from technical performance measurements. Technical Performance Measurement (TPM) involves a technique of predicting the future value of a key technical performance parameter of the higher-level end product under development based on current assessments of products lower in the system structure (per EIA-632 “Process for Engineering a System”) 1.

Manufacturers defining there performance characteristics need to determine which performance measures that correspond to mission objectives and suitability when defining Essential Performance and not fall into the trap of cycling through all technical performance measurements made on a device to determine what the essential performance is.

Particular (dash 2 Performance) Standards

As mentioned earlier, Particular standards include a list of Essential Performance characteristics in the standard. Looking at some particular standards for IEC60601-1 will help us understand what these performance characteristics look like. The Essential Performance characteristic may include areas such as the accuracy with which the equipment controls the delivery of energy or therapeutic treatment to the patient, as well as the devices ability to process and display physiological data that will affect patient care decision management.

The IEC60601-1 Particular Standards are available for some specific medical devices. These standards are represented by the IEC 60601-2-xx series, and are often referred to as “dash 2 standards.” Some examples of particular standards for IEC60601-1 are provided below. For those devices that do not have a dash 2 standards, the determination of Essential Performance is left to the manufacturer as defined in IEC60601-.


Examples of Essential Performance Characteristics from the Particular Standards

In addition to providing the Essential Performance characteristics for a medical device, the particular (dash 2) standards also include additional clauses that would replace or delete requirements that are called out in the general standard (IEC60601-1). The below table provides an example of some of the Essential Performance characteristics identified in a sampling of particular standards.

Table 1: Examples of EP items for some medical devices.

The definition of Essential Performance is defined per IEC60601-1 section 3.27 as:

“performance of a clinical function, other than that related to Basic Safety, where loss or degradation beyond the limits specified by the manufacturer results in an unacceptable Risk.

Note Essential Performance is most easily understood by considering whether its absence or degradation would result in an unacceptable Risk.”

In the review of table 1 above, it is clear that the actual acceptable performance levels are not prescribed, but left to the manufacturer. Additionally, the approach for how the characteristics are to be described across a sample of particular standards is not necessarily consistent. In some cases, the clinical function of the essential performance characteristic is not part of the specified item, as in “Alarm signals for high priority alarms” for infusion pumps. But, the consideration for what the actual acceptable performance level is as well as how these essential performance characteristics are to be applied while evaluating or testing the device is left to the discretion of the manufacturer. The note in the definition of 3.27 above does provide for how manufacturers should determine the specifics as well as what the acceptable performance ranges are. Once performance degrades to a degree that the risk is unacceptable (and harm can occur), this is when an unsafe condition or state has occurred (and therefore operating out of the determined range is unacceptable).

FDA Recognized Consensus Standards

The FDA recognizes standards that are deemed appropriate for manufacturers of medical devices to declare conformance with. The table above shows which of the example particular standards are FDA recognized consensus standards. The use of consensus standards to meet premarket submission requirements can help the FDA premarket review process although this is ultimately up to the submitter. Although the FDA reviewer may request the submitter to comply with a particular standard, as well as submit test data, as is the case with ISO 10933-1 for the Biological evaluation of medical devices. Additionally, if the standard is not FDA recognized, it doesn’t mean a reference to it or compliance to will not be considered acceptable by the FDA. The submitter can determine which standards the FDA suggests or recognizes by accessing the FDA website of recognized consensus standards, or determining if an FDA guidance is applicable that may also identify applicable standards.

From clearance standpoint, the FDA may not be as focused on identifying Essential Performance or testing Essential Performance, but they will be looking for the manufacturer demonstrate safety. The FDA has moved over the years to rely on standards, as well as identifying standards they recognize, but they always reserve the right to require more, regardless of what some of the IEC standards may require. In some cases, the FDA will put out FDA guidance documents that provide expectations on manufacturers for how performance is to be indicated and tested. The FDA has also indicated that risk can be used to determine what areas of a medical device require attention, but they have yet to say they are only interested in attending to only Essential Performance characteristics.

The next part of this article series will provide some examples of what has happened to the industry when Essential Performance is not correctly identified, as well as provide some recommendations for manufacturers. We are interested in your experience with Essential Performance and hope you will contact us and share them with us.

1 –“Processes%20for%20Engineering%20a%20System”%207%20Jan%2099.pdf

Understanding Cybersecurity Risk

Date: 1 25 2018    By Geoff Hutchins, Harold Pogue, Mohammad Raza

Understanding Cybersecurity Risk

Like other risk management perspectives, business and product, addressing cybersecurity needs to be considered throughout the development process[1].

A key part of this is understanding the types and classification of the cybersecurity risk[2] as a framework for assessment and development of control measures.

There are several viewpoints that should be considered when setting this up as follows:

  • Classification of the nature of the risk, malware, riskware, spyware etc.
  • Product lifecycle stage, premarket, post-market and legacy device
  • Risk introduced using OTS & SOUP libraries[3]
  • Product intended use, use environment, and hazard profile, life-sustaining, diagnostic, hospital or home
  • Classification by core functions, Identify, Protect, Detect, Respond and Recover
  • Classification by means of access, network connected, wirelessly, USB and medium such as CD

There are several existing frameworks that a manufacturer may consider using:

  • NIST Cybersecurity [4]
  • Common Vulnerability Scoring System (CVSS)4
  • AAMI TIR57 Principles for medical device security – Risk Management
  • UL 2900 Testability of network connected devices.

Cybersecurity Risk Management Process

The cyber risk management should be considered as part of the product lifecycle management:

  • Conduct product requirements cyber risk analysis and assessments.
  • Evaluate code, library and tool vulnerabilities and validations.
  • Plan and execute penetration testing on product configurations.
  • Document testing and re-evaluation plans for new threats
  • Plan for the deployment of updates and associated cybersecurity risks deploy cyber risk.

Some essential ways to mitigate risk include the following:

  • Require the user to update the default password of the device. Default passwords, especially for networked devices, are well known and can be exploited. Use longer passwords with a strong measurement of randomness.
  • When designing the device avoid using WEB/PSK/TKIP. These are no longer considered secure and have been depreciated. They share the same key and if that key is compromised, all devices in the network that communicates with that key are vulnerable.
  • Ensure that all data at rest is encrypted. It is also good practice to create backups prior to any change and to ensure the device user on how to recover the backup. If data is stolen the encryption should maintain privacy.
  • For software as a medical device, it is necessary to designate a patch management facility/process. Patches are common but can sometimes cause problems. Having a process in place that verifies that the patch works on the equipment before wide-scale deployment will minimize risk.
  • Have a periodic device verification plan and perform regular audits.
  • Perform security tests and provide these to the end user. These tests should include encryption, authentication, patches to the software, and version of virus protection.

[1] Content of Premarket Submissions for Management of Cybersecurity in Medical Devices, Guidance for Industry and Food and Drug Administration Staff, October 2, 2014

[2] IEC TR 80001-2-2 Edition 1.0 2012-07 Application of risk management for IT Networks incorporating medical devices.

[3] Guidance for Industry, Cybersecurity for Networked Medical Devices Containing Off-the-Shelf (OTS) Software, January 14, 2005.

[4] Postmarket Management of Cybersecurity in Medical Devices – Final Guidance, FDA presentation January 12, 2017, CDRH Webinar


Medical Device Validation Series 3 of 4: It all Starts with Requirements

Part 3, It All Starts with Requirements

This is part three of a four-part series on medical device validation practices.

Is writing requirements a validation activity? Of course, it is! What has been verified if verification testing is attempted without written requirements? The GPSV points out: “Success in accurately and completely documenting software requirements is a crucial factor in successful validation of the resulting software.”

There is some room for debate on what constitutes a requirement and what constitutes design detail. While much benefit derives from solid, well reviewed requirements that have nothing to do with testing, the testing effort should be based on verifying the correct implementation of requirements.

Tests should test requirements. Too often, tests are written without detailed requirements or not enough detail in requirements (exploratory testing), and the test developer is forced to refer to the software for details regarding how the software works. The result is that the test ends up documenting the way the software works, not necessarily the way it is supposed to work. The requirements for the software are embedded in the test, not in a requirements document. Many problems are associated with this, but perhaps most alarming is that potential problems can get embedded into tests as expected behavior—thus, making it difficult to identify them as problems, and guaranteeing that the behavior will be accepted forever after.

From: Vogel, David A., Ph.D. “Validating Medical Device Software Includes and Goes Beyond Testing.” Medical Product Outsourcing Mar. 2006: .