Let me start off with this: I’m a software engineer myself, and I hate manual testing. It’s terrible. It costs time and it’s not very objective. People forget to run tests or run them differently each time. For our own eQMS software, Formwork, we have hundreds of automated (not manual) tests, and it’s awesome.
But Formwork is not a medical device – in contrast to yours.
You see, the problem with the testing of medical device software is this: For each test, you have to document certain things – metadata, if you will:
- Unique ID of the test
- Test steps: Which steps are performed in this test with which intermediate results, and what are the expected values of those intermediate results?
- Expected end result
- Actual end result
- Traceability to one or many software requirements in your software requirements specification (long story)
- Required configuration of the test computer / device for this test.
It gets a bit more tricky, too. Most of these things actually have to be documented and signed off on before your test is run. Which things? Well, all things except the actual end result (you obviously don’t have that yet, as you haven’t run the test yet, duh).
Okay. So that now puts you in a really crappy place in your code base, because how the hell do you store all that “documentation” alongside your tests?
Automated Testing Of Medical Device Software
Huge tangent: I used to write a lot of Lisp code (Clojure) which is a programming language used mostly by weird people. There, it’s easy to extend your programming language. So you could somehow write a new testing library which enables you to attach the data mentioned above to each test.
But then again, you’re still screwed because it e.g. refers to things which are not in your code at all, specifically the traceability to software requirements which are typically documented in a separate system. Arg!
So, I hope you see by now that we’ve gone down a really crappy rabbit hole, just because we wanted to do automated testing of our medical device. Yes, it’s possible, but along the way, you’ll need to solve:
- Somehow linking your tests with your software requirements documentation which probably lives in a terrible, crappy software (unless you use Formwork)
- Writing your own testing library which can attach the metadata mentioned above to each of your tests
- Keeping your git repository (= tests in code) in sync with your documentation versioning (good luck)
- Exporting all of that in a PDF which your auditors accept.
In my experience, the exporting part is one of the trickiest, because generating PDFs is already hell, and extracting metadata from a testing library is hell, too, so it’s 2x hell, mathematically speaking.
Anyway. All of this is time you could have spent writing your documentation. And unless you plan to ship hundreds new versions of your medical device (in my experience, companies ship between zero and 20 updates), manual test runs and documentation is the way to go, because you’d already be done by now if you had chosen manual documentation. So let’s see how to do that next.
Manual Testing Of Medical Device Software
If you choose to manually document and run your tests, here’s how you do it:
- Create some sort of table, spreadsheet or other crappy structure in which you document the data mentioned in the bullet point list above. Or, wait, use Formwork, because it’s our software, it’s awesome, and already has all the right form fields for you.
- Once you’re done, save it and sign off on it in Formwork to prove to your auditors that you’ve actually defined the tests before running them.
- Run the tests manually. Formwork shows you the steps along the way (with the expected test results) and gives you forms to document the actual test results.
- Save it. Done!
So the documentation part here is really fast. Only running the tests manually sucks.
Conclusion And Time Estimates
You probably still think “this sounds like a terrible solution”. Yes, but only if you’re a software developer, because it’s not elegant on a technical level, but it’s very efficient. Let’s compare the time you’ll spend on each solution:
| Time spent | Automated testing | Manual testing |
| Setting up an automated testing and documentation pipeline | 1 – 3 months, sometimes more | 0 seconds |
| Running the tests | ~5 minutes per run | 4 hours |
So. The point here is that your upfront time cost for automated testing is huuuge. If you’re a startup and your goal is to make it to market in a short time, it’s not feasible to go off and build your own testing library for 1 – 3 months. Every startup I’ve seen so far which thought “hey sure, we do this” disappeared into the over-engineering cave and never came back.
Relevant xkcd (link – I added the red box):
So, in the above chart, it gives you an estimate of how much time you’re allowed to spend on automation, depending on how often you perform a task and which potential time savings an automation would give you. Let’s assume you want to do new releases “monthly” (top column), and you plan to shave off 1 hour (left side). That leaves you with 2 days to spend on this automation.
And 2 days are way too little time to write your own testing library.
(I also made super optimistic assumptions here: Companies tend to ship medical device software updates less regularly than every month. Also, the chart above presumes a total time course of 5 years which is longer than the existence of most startups.)
