Early access registrations are open for Headstart, a predictable, fixed-price program for becoming compliant.

September 20, 2020

QMS Software for Medical Devices (ISO 13485): The Ultimate Comparison

Dr. Oliver Eidel

Software for setting up and maintaining a Quality Management System is something which comes up with any new company I work with. The discussion goes along the lines of:

“So the ISO 13485 has all these requirements regarding documents. Surely, there must be some great software out there which helps us be 13485-compliant?”

Yeah, right. When I got into regulatory work, I also started out being very naive. Here’s the reality: There isn’t. I like to summarize it like this: There’s some software out there, some of it is very expensive, and all of it is crappy.

I’m certainly very opinionated on this topic and I expect other people, especially seasoned regulatory affairs professionals, to have differing opinions. That’s okay. I presume our opinions will differ because we have different expectations.

Here are the expectations which a QMS software technically needs to fulfil:

QMS Software Requirements

  • Creation and formatting of documents: Rich-text formatting, images, tables, attachments.
  • Change history of documents: What has been changed when by whom?
  • Review process of documents: Send a document out for review. Receive comments, change the document, get it re-reviewed.
  • Approval and distribution of documents: Once a document has been approved, inform everyone that it’s in effect now.
  • Signing of documents: Tamper-proof signing which is compliant with national requirements, e.g. 21 CFR Part 11.

I’m sure that all of the software packages which I describe below fulfil these requirements. Your regulatory affairs professional will be satisfied and happy. Great!

Great? Not quite. Technically fulfilling all regulatory requirements may be enough to make your regulatory affairs department happy and survive an audit, but it can still make your life miserable. Why? Here are some requirements which I’d additionally have for QMS software:

QMS Software Requirements in the Real World

  • The software is usable: It has great usability (doesn’t require hour-long training) and everyone in the company uses it regularly without being forced to.
  • The software is accessible: The company has received enough licenses from the vendor so that every employee can access the software any time. (This goes both ways: Some vendors charge outrageous prices for additional user licenses. Some companies are unwilling to pay more money and try to re-use licenses.)
  • The software is opinionated: It comes preconfigured with common-sense defaults which are easy to understand and ensure compliance. It doesn’t require days of configuration efforts, or even worse, a contractor who charges high hourly rates for setting it up for you.

See, no tool on this planet will help your company if only one person is using it while every one else hates to touch it or doesn’t even have access. And that’s exactly where I often see a disconnect between people working in regulatory affairs and software developers, which is unfortunate.

Let’s see what a software developer at a major medical software company says about this:

I basically work in three worlds. The first one is where our code is (like GitHub); it has version control and issues which we engineers use to organize our work. The second world is where project management happens (like Jira); that’s where product managers prioritize on a high level what’s next. And the third world is the regulatory world in which I regularly have to update documentation, like software requirements.

Note that each of these worlds runs in a separate tool. And all three of them are completely disconnected. So, we could continue coding (first world) and nobody would notice if we didn’t update the regulatory documentation (third world).

While the first and seconds worlds make sense, the third world is pure overhead.

Software Developer at a major medical software company

So that’s where we’re at today. QMS software always is a separate “world” in which developers reluctantly have to enter (if they do it at all) to create some documentation. A perfect tool would combine source code version control, project management and regulatory documentation.

Alas, that tool doesn’t exist. Enough complaining for now! Let’s see what the current tools offer.

Huge disclaimer: These are my highly subjective impressions and they may vary strongly from yours or other regulatory affairs people for the reasons above.

Comparison Table

As you’re surely impatient by now, I’ll put the comparison table up top and write about the details further below.

General

Name Type Interface Usability Pricing Price
Confluence Self-hosted So-so So-so Transparent Low
GitLab / GitHub Self-hosted / SaaS Great Great* Transparent Low
Google Docs / Drive SaaS Great Great Transparent Low
Greenlight Guru ? ? ? Opaque Very high?
MasterControl ? ? ? Opaque Very high?
MatrixQMS SaaS Windows 95 So-so Transparent High
Orcanos SaaS Catastrophic Bad Transparent Very high
Polarion Self-hosted Atrocious Difficult Opaque High
Qualio SaaS So-so So-so Opaque Very high

* If all team members are capable of dealing with markdown documents.

Features

Name Change History Linking of Stuff Review / Signing
Confluence Okay Can be done Probably: Plugin needed
GitLab / GitHub Excellent (git) Tricky, unless issues Process needed (e.g. Pull Request)
Google Docs / Drive Good Tricky Probably: Plugin needed
Greenlight Guru      
MasterControl      
MatrixQMS     Included
Polarion Atrocious Can be done Included
Qualio So-so Can be done Included

Tools

I classify tools into two categories: 1) Tools that were designed from the ground up as QMS (“classic” QMS software), and 2) Tools that are generally used for other means but can be re-purposed as QMS (“repurposed” QMS Software).

Let’s start with the repurposed ones.

“Repurposed” QMS Software

Confluence

Confluence is the most common one I encounter. It’s not like people put a lot of thought into it before diving into the Atlassian ecosystem of Confluence + Jira and ending up with a huge, complex, slow-loading mess.

But: It still seems to be a solid option. It’s probably somewhat unique because it allows you to integrate your project management (Jira), source code (Bitbucket) and QMS (Confluence) into one place, more or less.

That is, if you ever figure out how to configure it. Once you’ve achieved that (if at all), you have to convince your team to use it in the right way, and during the audit spend considerable time explaining it to an auditor. I’ve seen auditors who had some prior experience with these tools, so at least it won’t be foreign to them.

But still, it’s a huge pain. For ISO 13485 and medical device compliance you’ll probably need additional plugins for document signing (e.g. Komala) and risk management (e.g. SoftComply) which come with additional cost, complexity and, you guessed it, configuration.

Talking about cost: The pricing of Atlassian tools is publicly available and reasonable, at least for the SaaS version. If you want to self-host, it gets pricey.

If you have prior experience with Atlassian tools, are confident in setting things up and have some basic regulatory knowledge so you know what you actually want to achieve with your setup, then Confluence could be a viable option. Personally, I don’t like it, but that’s also because I’m a developer at heart and most developers have somewhat of an love/hate/abusive relationship with Confluence + Jira.

Pros:

  • Integration with source code repository (Bitbucket) and project management (Jira) possible, in theory
  • Document editing in Confluence is straightforward

Cons:

  • Commonly, additional plugins are needed for signing and risk management
  • Jira and Confluence are slow, e.g. when rendering documents
  • Configuration can be very difficult. Nobody tells you whether your custom configuration is ISO 13485 - compliant until you’re being audited

GitLab / GitHub

This is typically the proposed setup I encounter at tech-driven, developer-heavy organizations. It makes perfect sense: You need some sort of change history for your documents, so why not write them in markdown and put them in git? A technically perfect solution. The developer’s dream. And I agree.

That is, if you can convince your entire organization to write markdown documents and commit files to git (or open pull requests, or whatever your workflow is). It’s certainly doable, and maybe even preferable over training your developers to use another crappy QMS software.

It does come with some additional challenges:

  • “Linking” things like you could in Confluence + Jira can be tricky. If you want to link to an GitHub issue from a markdown document, how do you do it? There’s no autocomplete or linking feature like in Confluence documents.
  • You don’t have any “foreign key constraints” on your links. For example, if you write document A, which refers to documents B and C, and later you delete document C, you won’t realize that it’s still being mentioned in document A.
  • What do you put as markdown document in git, and what do you put into issues? Some companies (mis-)use issues for all sorts of things: Software requirements, feature requests and even signing (?!). Consider that issues don’t underlie the same solid version control like markdown documents in git do.
  • How do you work on documents which are best created as a spreadsheet, like risk management? Similar problem as with Confluence (there, you probably use a plugin). Manually creating tables and calculating data in markdown files is hard. A spreadsheet would be so much easier.

That being said, if you can deal with those drawbacks - and I don’t think any of them is a complete deal-breaker - then GitHub / GitLab is a solid choice. It provides best-in-class version control, has a nice user interface, and pricing is transparent and affordable.

Pros:

  • Perfect version control of documents in git
  • Great user interface
  • Transparent and affordable pricing
  • Choice between self-hosted or SaaS (GitLab)

Cons:

  • Everyone has to learn markdown
  • Linking things will be error-prone
  • A lot of process decisions need to be made: How do we handle signing? Document review?
  • Markdown files aren’t a good fit for spreadsheet-like documents (e.g. risk management)

Google Docs / Drive

Most startups already have GSuite for its email and calendar capabilities. It also includes tools for collaborative document and spreadsheet editing (Docs and Sheets) which are.. best-in-class? At least I can’t think of any commonly used alternative. And the real-time editing and commenting features are very solid.

Given that many established medical device manufacturers run their Quality Management Systems in Microsoft Word and Excel, I think that the GSuite tools are certainly a good choice. Documents can’t get lost in the sense of “I misplaced a USB drive”. The change history and collaborative editing are solid (no more “where’s the copy of the file in which you made your changes?”).

Google Sheets allows you to create spreadsheets for calculation-heavy tasks like for your risk assessment (no need for a plugin like for Confluence; less pain than GitHub / GitLab).

That being said, some similar limitations like for the other “repurposed” tools above apply. You need to have an idea of what ISO 13485 compliance includes, because you need to make many “process decisions” on how to set up your folder structure, how to name your documents, etc.

Document signing is tricky and needs a separate plugin to be solid (you could roll your own, but.. it’s tricky). From my experience, all external document signing plugins suck, because they disrupt your workflow and create copies of your documents as PDFs. Now you have two sources of truth (the original GDoc and the new PDF), great.

The GSuite price is per-user and quite reasonable, especially given that most tech companies already are on GSuite and therefore have no additional cost.

Pros:

  • Most tech companies already have it - no additional cost
  • Great usability, everyone can use it
  • Transparent and affordable pricing
  • You can model almost everything in a Google Doc or Sheet

Cons:

  • A lot of “process decisions” necessary - the burden of keeping things organized in folders and with clear titles essentially lies on you
  • Document review and signing require either a very clear process or a separate plugin, which sucks
  • References between documents are possible, but no consistency is ensured

“Classic” QMS Software

That’s it for the “repurposed” QMS software. Let’s look at more specialized (“classic” QMS) software next. They all have in common that they were specifically developed for modeling a QMS.

As I don’t have first-hand experience with some of them, I’m only going to mention them for the sake of completeness.

Greenlight Guru

Greenlight Guru is always one of the first software mentioned when it comes to QMS software in the US. I haven’t seen it personally yet (happy to review it).

Their pricing model is incredibly opaque (“contact sales”) and it’s said to be incredibly expensive (at least five figures of USD per year).

MasterControl

No first-hand experience, but another QMS software which is commonly mentioned.

MatrixQMS

I tried out a demo account when I was considering using it. While lots of other QMS software vendors don’t publish their pricing, I’m positively surprised that MatrixQMS is transparent here. The QMS starts at 590€ per month (as seen 09/2020) for three users. That’s probably two orders of magnitude more than Confluence / GitHub / GitLab / GSuite, but probably one order of magnitude less than Greenlight Guru and Qualio (yes, it’s crazy out there).

After trying it out for some time, my takeaways were:

  • It more or less covers what you need in a QMS. If you additionally get the ALM, you can also do requirements and risk management.
  • The interface looks like it was created in the nineties.
  • Usability is so-so.

Probably a tool which makes many regulatory professionals happy. And one of those tools which makes software developers want to jump out of the window. I decided against it because I just couldn’t bring myself to shove this down the throats of a team of young, bright software developers. Rather have a tool which everyone uses and risk some audit problems, than having a tool which is only maintained by your regulatory people and isolated from the real world.

Orcanos

I tried it out as they offer a free demo. The interface and usability was so catastrophic that I gave up shortly later. Pricing is also quite incredible at 990$ per month. Seriously?

Polarion

Polarion was purchased by Siemens some time ago. It’s the only QMS software on this page which doesn’t have a cloud-hosted SaaS offering. You’ll shortly learn why.

First off, the fact that its website is a 4-level-deeply-nested subdomain (polarion.plm.automation.siemens.com) already raises eyebrows. But that’s just the prelude of what’s next to come. The publicly available installation manual comprises 80 pages, of which most of them are actual step-by-step instructions of installing Postgres and OpenSSH on your server.

Yeah, that’s right. In a world of cloud-hosted applications and docker containers, Polarion expects you to manually set up a server from scratch, with stateful, configurable dependencies (Postgres) all over the place. For me, this is already a no-go in the context of a startup. You already need one person doing regulatory work full-time, and now you need another engineer to set up and maintain your Polarion server? Remember, this includes regular backups - if you lose that sort of data, you’re lost.

Want to purchase Polarion? It can’t be purchased directly, not even on the official 4-level subdomain and no, the 80-page manual also doesn’t give you a hint. You have to go through resellers who don’t publish their prices. So you might end up with wildly differing offers. Some resellers also give you some sort of pre-configuration and templates on top. It’s a wild world out there. The one good part is that it’s usually a one-off purchase - you pay money once, you get the software, you have to install it, no updates (who’d want to migrate their own bare-metal Postgres DB anyway?).

If we just assume that you magically come across a running Polarion instance which you don’t have to maintain, you’ll be greeted by software which looks like a very complex version of Jira from the nineties. Configuring it yourself? Borderline impossible. Even using it without any training? Hard to imagine.

From what I’ve seen, I think that Polarion, correctly configured, and used by a team which was very, very well trained on it, can ensure good compliance. But it’s certainly not a very usable tool and I can’t imagine any developer using this tool willingly when being presented with modern alternatives like GitHub / GitLab.

Pausing the ranting for a bit, Polarion could work for you if your regulatory person already has experience with it and you have someone to maintain your local installation. In all other cases, it’ll be a (steep) uphill battle.

Qualio

Qualio is the new kid on the block, fueled by VC money and unicorn growth. Its pricing model is just as opaque as Greenlight Guru and the rumors are prices are in a similar range (minimum 5-figure USD per year).

Its simpler than full-blown tools like Polarion. The usability is still not great, and the text editing experience is quite subpar to tools like Confluence and GSuite. There are some neat features like linking documents a document review processes, but.. it just feels like one of those tools which were written by software engineers (or even business people) with no UI/UX engineers (or even customers) in mind.

Think of it like a crappy version of Confluence with some QMS features included. You gain compliance features like change history, review and signing, but your trade-off is that the actual text editing experience is much worse.

I would not choose it. Instead of getting compliance features out of the box while having crappy usability, I’d prioritize usability of a tool and add the compliance features on top via plugins or processes (GitLab / GitHub / GSuite).


But that’s just me. The fact that many of these companies exist shows that I’m certainly not representative for all those medical device companies out there. The enterprise software market is certainly a strange place.

Steve Jobs also noticed that:

What I love about the consumer market, that I always hated about the enterprise market, is that we come up with a product, we try to tell everybody about it, and every person votes for themselves. They go ‘yes’ or ‘no,’ and if enough of them say ‘yes,’ we get to come to work tomorrow. That’s how it works. It’s really simple. With the enterprise market, it’s not so simple. The people that use the products don’t decide for themselves, and the people that make those decisions sometimes are confused.

Congratulations! You read this far.

Get notified when I post something new.

Sign up for my free newsletter.

I work as a regulatory consultant for Healthcare software startups. I try to publish all my knowledge here so that startups can certify their medical devices themselves in the future.

If you're still lost and have further questions, just send me an email and I'll be happy to answer them for free. More about me

No Cookie For You Privacy Policy Imprint
No QMS on this planet will save you from creating crappy software.