For your medical device, you have to perform two different kinds of usability evaluations (or tests): Formative and summative usability tests.
The standard which defines the usability testing requirements for medical devices is the IEC 62366, and it’s surprisingly easy to read. Maybe also because it’s really short, so you only fall asleep once or so (in contrast to more lengthy standards which enable more sustained afternoon-napping).
Regarding testing, the standard defines two types of usability tests you have to perform: Summative and formative usability tests. People often ask me about the differences between the two. Instead of continuing this monologue, let’s just do a handy table, shall we?
| Summative Usability Testing | Formative Usability Testing | |
| Method | Has to be a user test! | Whatever you want |
| Participants | Your target users, e.g. doctors or patients | Whoever you want |
| Participant Count | At least 5 for EU MDR At least 15 for US FDA | As many as you want |
| Product Being Tested | Final to-be-released product (or near-final prototype) | Whichever you want |
Okay, you get the idea. In short, the requirements for the formative usability test are, um, quite relaxed. While you might first think “ugh, this sounds unsafe”, I’d say no, it’s not unsafe, and it’s actually a good thing – like one of those 1% things I actually like about our current medical device regulations.
Why? Because it actually, for once, allows you to do useful things by not being to restrictive. Let’s look at some examples of formative and summative usability tests next, and you’ll see what I mean.
Formative Usability Testing Examples
The purpose of your formative usability testing is to shape the product while it’s under development. And that makes a lot of sense! Because you wouldn’t want to go heads-down into your product development cave, only to emerge with a product which nobody wants (and nobody is going to pay you money for). So the goal is to get early feedback, and it’s super useful, even beyond the medical device space! (Maybe that’s this whole “agile” thing which people have since overcomplicated to an infinite degree)
If you think of it as “how can we get some feedback on our device”, this pretty much answers how to do formative usability testing. For example:
- You schedule a video call with a target user (e.g. a physician), share your screen (e.g. via Google Meet), walk them through your current product (software) and write down their feedback.
That’s it. So simple. And then you’d use that feedback to improve / change your product development. - You get your software reviewed by a usability expert.
You could do this by e.g. looking for a freelance UI/UX person and giving them access to your software, with the goal of them giving you feedback on usability aspects you could improve. Not something startups do every day, but might still be useful. - If you have an in-house UI/UX person, simple let them create some review documentation.
This is just a variation of the above – if you have an in-house UI/UX person which can serve as a usability expert (everyone can), then just be sure to document some instances where that person gives feedback on the usability of your software (they probably do that regularly anyway as it’s their job). - You perform a user test.
Same as for the summative usability evaluation, but on an earlier prototype and with less stringent requirements (e.g. no minimum participant count).
The list goes on. You get the idea.
And if I remember correctly, formative usability testing isn’t even required by the IEC 62366. So, you can do it, or you can skip it, up to you. You can guess what most manufacturers do (yeah, they skip it). But it would actually make sense from a business point of view (i.e., not failing) to do it. Anyway!
In contrast, let’s look at how to do summative usability testing.
Summative Usability Testing Examples
Creating a list of examples of summative usability testing is mostly a useless task, because there’s only one method to do it anyway – by actually performing a user test. Still, if you were expecting a list, sure, here you go:
- You perform a user test.
Similar to the option above for the formative usability testing, however this one has more stringent requirements, e.g. regarding minimum participant count. See below.
Yup, so summative usability testing always has to be a user test. Additionally, this user test has quite stringent requirements. Let’s look at those:
- Participants: These should be people who are the target user in your intended use. So if your device is a software for radiologists, they should be radiologists. If your device is for patients with diabetes, they should likely be people with diabetes. Etc.
- Minimum participant count: This is the minimum number of people you’ll need to include in your summative usability test. 5 for EU MDR compliance, 15 for US FDA compliance.
- Product being tested: Best case, this is your final product (hardware / software). In the real world, this is not always possible (especially for hardware), so your next best alternative is a prototype which is pretty much identical with the final product. For software, that would be your software with the final set of features you’re planning to build – maybe you’ll only be adding some bug fixes on top of that later. But it should look and feel like your final product.
How To Do It?
And how do you actually perform this summative usability test in detail? Check out our article on summative usability evaluations to learn more.
Conclusion
Don’t worry about formative usability testing. The summative usability testing is 95% of the work, because it’s a user test on your final product. That’s what counts.
