Software for setting up and maintaining a Quality Management System is something which comes up with any new company I work with. The discussion goes along the lines of:
“So the ISO 13485 has all these requirements regarding documents. Surely, there must be some great software out there which helps us be 13485-compliant?”
Yeah, right. When I got into regulatory work, I also started out being very naive. Here’s the reality: There isn’t. I like to summarize it like this: There’s some software out there, some of it is very expensive, and all of it is crappy. Update: This has actually changed since we’ve released our own QMS software (chuckle). But I’ll still try to keep this comparison up to date and mostly-unbiased (even though its hard, given the sheer crappiness of software out there).
Regardless, I’m certainly very opinionated on this topic and I expect other people, especially seasoned regulatory affairs professionals, to have differing opinions. That’s okay. I presume our opinions will differ because we have different expectations.
Here are the expectations which a QMS software technically needs to fulfil:
QMS Software Requirements
- Creation and formatting of documents: Rich-text formatting, images, tables, attachments.
- Change history of documents: What has been changed when by whom?
- Review process of documents: Send a document out for review. Receive comments, change the document, get it re-reviewed.
- Approval and distribution of documents: Once a document has been approved, inform everyone that it’s in effect now.
- Signing of documents: Tamper-proof signing which is compliant with national requirements, e.g. 21 CFR Part 11.
I’m sure that all of the software packages which I describe below fulfil these requirements. Your regulatory affairs professional will be satisfied and happy. Great!
Great? Not quite. Technically fulfilling all regulatory requirements may be enough to make your regulatory affairs department happy and survive an audit, but it can still make your life miserable. Why? Here are some requirements which I’d additionally have for QMS software:
QMS Software Requirements in the Real World
- The software is usable: It has great usability (doesn’t require hour-long training) and everyone in the company uses it regularly without being forced to.
- The software is accessible: The company has received enough licenses from the vendor so that every employee can access the software any time. (This goes both ways: Some vendors charge outrageous prices for additional user licenses. Some companies are unwilling to pay more money and try to re-use licenses.)
- The software is opinionated: It comes preconfigured with common-sense defaults which are easy to understand and ensure compliance. It doesn’t require days of configuration efforts, or even worse, a contractor who charges high hourly rates for setting it up for you.
See, no tool on this planet will help your company if only one person is using it while every one else hates to touch it or doesn’t even have access. And that’s exactly where I often see a disconnect between people working in regulatory affairs and software developers, which is unfortunate.
Let’s see what a software developer at a major medical software company says about this:
I basically work in three worlds. The first one is where our code is (like GitHub); it has version control and issues which we engineers use to organize our work. The second world is where project management happens (like Jira); that’s where product managers prioritize on a high level what’s next. And the third world is the regulatory world in which I regularly have to update documentation, like software requirements.
Note that each of these worlds runs in a separate tool. And all three of them are completely disconnected. So, we could continue coding (first world) and nobody would notice if we didn’t update the regulatory documentation (third world).
While the first and seconds worlds make sense, the third world is pure overhead.
Software Developer at a major medical software company
So that’s where we’re at today. QMS software always is a separate “world” in which developers reluctantly have to enter (if they do it at all) to create some documentation. A perfect tool would combine source code version control, project management and regulatory documentation.
Alas, that tool doesn’t exist. Enough complaining for now! Let’s see what the current tools offer.
Huge disclaimer: These are my highly subjective impressions and they may vary strongly from yours or other regulatory affairs people for the reasons above.
As you’re surely impatient by now, I’ll put the comparison table up top and write about the details further below.
We’ve actually written a few articles on how to build a QMS in some of the systems mentioned below. Check out our articles on setting up a QMS in Google Drive or GitHub / GitLab and especially why GitLab turned out to be not a good idea.
|GitLab / GitHub||Self-hosted / SaaS||Great||Great*||Transparent||Low|
|Greenlight Guru||SaaS||?||?||Opaque||Very high|
|codebeamer||SaaS||Windows 98||So-so||Opaque||Very high|
* If all team members are capable of dealing with markdown documents.
|Name||Text Editor||Change History||Linking of Stuff||Review / Signing|
|Confluence||Painful||Okay||Can be done||Third-party plugin needed|
|GitLab / GitHub||Okay for coders (markdown)||Excellent (git)||Tricky||Process required (e.g. Pull Request)|
|Google Docs / Drive||Great||Needs Workarounds||Tricky||Third-party plugin required|
|Greenlight Guru||Nonexistent!||Okay||Can be done||Included, but no text editor|
|MatrixQMS||Like Windows 95||Okay||Can be done||Included|
|Polarion||Like Windows 95||Atrocious||Can be done||Included|
|Qualio||Weird||So-so||Can be done||Included|
|codebeamer||So-so||Can be done|
|Formwork||Great||Good||Can be done easily||Included (also FDA compliant)|
I classify tools into two categories: 1) Tools that were designed from the ground up as QMS (“classic” QMS software), and 2) Tools that are generally used for other means but can be re-purposed as QMS (“repurposed” QMS Software).
Let’s start with the repurposed ones.
“Repurposed” QMS Software
Confluence is the most common one I encounter. It’s not like people put a lot of thought into it before diving into the Atlassian ecosystem of Confluence + Jira and ending up with a huge, complex, slow-loading mess.
But: It still seems to be a solid option. It’s probably somewhat unique because it allows you to integrate your project management (Jira), source code (Bitbucket) and QMS (Confluence) into one place, more or less.
That is, if you ever figure out how to configure it. Once you’ve achieved that (if at all), you have to convince your team to use it in the right way, and during the audit spend considerable time explaining it to an auditor. I’ve seen auditors who had some prior experience with these tools, so at least it won’t be foreign to them.
But still, it’s a huge pain. For ISO 13485 and medical device compliance you’ll probably need additional plugins for document signing (e.g. Komala) and risk management (e.g. SoftComply) which come with additional cost, complexity and, you guessed it, configuration.
Talking about cost: The pricing of Atlassian tools is publicly available and reasonable, at least for the SaaS version. If you want to self-host, it gets pricey.
If you have prior experience with Atlassian tools, are confident in setting things up and have some basic regulatory knowledge so you know what you actually want to achieve with your setup, then Confluence could be a viable option. Personally, I don’t like it, but that’s also because I’m a developer at heart and most developers have somewhat of an love/hate/abusive relationship with Confluence + Jira.
- Integration with source code repository (Bitbucket) and project management (Jira) possible, in theory
- Document editing in Confluence is straightforward
- Commonly, additional plugins are needed for signing and risk management
- Jira and Confluence are slow, e.g. when rendering documents
- Configuration can be very difficult. Nobody tells you whether your custom configuration is ISO 13485 - compliant until you’re being audited
GitLab / GitHub
This is typically the proposed setup I encounter at tech-driven, developer-heavy organizations. It makes perfect sense: You need some sort of change history for your documents, so why not write them in markdown and put them in git? A technically perfect solution. The developer’s dream. And I agree.
That is, if you can convince your entire organization to write markdown documents and commit files to git (or open pull requests, or whatever your workflow is). It’s certainly doable, and maybe even preferable over training your developers to use another crappy QMS software.
It does come with some additional challenges:
- “Linking” things like you could in Confluence + Jira can be tricky. If you want to link to an GitHub issue from a markdown document, how do you do it? There’s no autocomplete or linking feature like in Confluence documents.
- You don’t have any “foreign key constraints” on your links. For example, if you write document A, which refers to documents B and C, and later you delete document C, you won’t realize that it’s still being mentioned in document A.
- What do you put as markdown document in git, and what do you put into issues? Some companies (mis-)use issues for all sorts of things: Software requirements, feature requests and even signing (?!). Consider that issues don’t underlie the same solid version control like markdown documents in git do.
- How do you work on documents which are best created as a spreadsheet, like risk management? Similar problem as with Confluence (there, you probably use a plugin). Manually creating tables and calculating data in markdown files is hard. A spreadsheet would be so much easier.
That being said, if you can deal with those drawbacks - and I don’t think any of them is a complete deal-breaker - then GitHub / GitLab is a solid choice. It provides best-in-class version control, has a nice user interface, and pricing is transparent and affordable.
- Perfect version control of documents in git
- Great user interface
- Transparent and affordable pricing
- Choice between self-hosted or SaaS (GitLab)
- Everyone has to learn markdown
- Linking things will be error-prone
- A lot of process decisions need to be made: How do we handle signing? Document review?
- Markdown files aren’t a good fit for spreadsheet-like documents (e.g. risk management)
Google Docs / Drive
Most startups already have GSuite for its email and calendar capabilities. It also includes tools for collaborative document and spreadsheet editing (Docs and Sheets) which are.. best-in-class? At least I can’t think of any commonly used alternative. And the real-time editing and commenting features are very solid.
Given that many established medical device manufacturers run their Quality Management Systems in Microsoft Word and Excel, I think that the GSuite tools are certainly a good choice. Documents can’t get lost in the sense of “I misplaced a USB drive”. The change history and collaborative editing are solid (no more “where’s the copy of the file in which you made your changes?”).
Google Sheets allows you to create spreadsheets for calculation-heavy tasks like for your risk assessment (no need for a plugin like for Confluence; less pain than GitHub / GitLab).
That being said, some similar limitations like for the other “repurposed” tools above apply. You need to have an idea of what ISO 13485 compliance includes, because you need to make many “process decisions” on how to set up your folder structure, how to name your documents, etc.
Document signing is tricky and needs a separate plugin to be solid (you could roll your own, but.. it’s tricky). From my experience, all external document signing plugins suck, because they disrupt your workflow and create copies of your documents as PDFs. Now you have two sources of truth (the original GDoc and the new PDF), great.
The GSuite price is per-user and quite reasonable, especially given that most tech companies already are on GSuite and therefore have no additional cost.
- Most tech companies already have it - no additional cost
- Great usability, everyone can use it
- Transparent and affordable pricing
- You can model almost everything in a Google Doc or Sheet
- A lot of “process decisions” necessary - the burden of keeping things organized in folders and with clear titles essentially lies on you
- Document review and signing require either a very clear process or a separate plugin, which sucks
- References between documents are possible, but no consistency is ensured
“Classic” QMS Software
That’s it for the “repurposed” QMS software. Let’s look at more specialized (“classic” QMS) software next. They all have in common that they were specifically developed for modeling a QMS.
As I don’t have first-hand experience with some of them, I’m only going to mention them for the sake of completeness.
Greenlight Guru is always one of the first software mentioned when it comes to QMS software in the US. After posting this article, a friendly person from Greenlight Guru actually reached out to me to ask whether we could schedule a call and clarify some of my questions, with the goal of adding more useful information to this article. Sure!
During our call, the person walked me through something of a Greenlight Guru demo - note that this wasn’t a real trial account, it was more like a clickable prototype. So the Greenlight Guru person really only could click on the links which they wanted to click on, because many of the other links wouldn’t work (yep).
The product itself looked quite nice and polished. I think the general QMS features are covered quite well, stuff like reviewing documents, electronic signing, and even requirements management. What struck me as very weird was that the product doesn’t have its own document editor, so it literally relies on you to have Microsoft Word or something similar installed on your computer, writing your documents locally and then subsequently uploading them to the Greenlight Guru software for archiving and signing. A weird process, given that humanity has invented browser-based text editors in the meantime.
I also asked whether they could provide any more details on their pricing model as that would really be useful for this article and it would help a lot of startups make their decision. Unfortunately, they declined and said they only do “individual prices, tailored to the client yadda yadda enterprise sales yak yak”.
Later on, the friendly person followed up via email if they could help me any further with providing more Greenlight Guru information for this article. I said yes, how about your share some of your pricing details. The person never responded. That was the last of my communication with Greenlight Guru.
So my conclusion still stands: Their software is probably not bad, but their business practice is still inherently shady: Their pricing model is really opaque (“contact sales”), it’s said to be incredibly expensive (at least five figures of USD per year), you don’t get access to all features (“tailored to your company”), they don’t provide you with a free trial (they don’t even provide their own sales people with a free trial), you can’t export your data easily, I mean.. damn. Not sure if this is really enticing for startups, or any rational company at all.
No first-hand experience, but another QMS software which is commonly mentioned.
I tried out a demo account when I was considering using it. While lots of other QMS software vendors don’t publish their pricing, I’m positively surprised that MatrixQMS is transparent here. The QMS starts at 590€ per month (as seen 09/2020) for three users. That’s probably two orders of magnitude more than Confluence / GitHub / GitLab / GSuite, but probably one order of magnitude less than Greenlight Guru and Qualio (yes, it’s crazy out there).
After trying it out for some time, my takeaways were:
- It more or less covers what you need in a QMS. If you additionally get the ALM, you can also do requirements and risk management.
- The interface looks like it was created in the nineties.
- Usability is so-so.
Probably a tool which makes many regulatory professionals happy. And one of those tools which makes software developers want to jump out of the window. I decided against it because I just couldn’t bring myself to shove this down the throats of a team of young, bright software developers. Rather have a tool which everyone uses and risk some audit problems, than having a tool which is only maintained by your regulatory people and isolated from the real world.
I tried it out as they offer a free demo. The interface and usability was so catastrophic that I gave up shortly later. Pricing is also quite incredible at 990$ per month. Seriously?
Polarion was purchased by Siemens some time ago. It’s the only QMS software on this page which doesn’t have a cloud-hosted SaaS offering. You’ll shortly learn why.
First off, the fact that its website is a 4-level-deeply-nested subdomain (polarion.plm.automation.siemens.com) already raises eyebrows. But that’s just the prelude of what’s next to come. The publicly available installation manual comprises 80 pages, of which most of them are actual step-by-step instructions of installing Postgres and OpenSSH on your server.
Yeah, that’s right. In a world of cloud-hosted applications and docker containers, Polarion expects you to manually set up a server from scratch, with stateful, configurable dependencies (Postgres) all over the place. For me, this is already a no-go in the context of a startup. You already need one person doing regulatory work full-time, and now you need another engineer to set up and maintain your Polarion server? Remember, this includes regular backups - if you lose that sort of data, you’re lost.
Want to purchase Polarion? It can’t be purchased directly, not even on the official 4-level subdomain and no, the 80-page manual also doesn’t give you a hint. You have to go through resellers who don’t publish their prices. So you might end up with wildly differing offers. Some resellers also give you some sort of pre-configuration and templates on top. It’s a wild world out there. The one good part is that it’s usually a one-off purchase - you pay money once, you get the software, you have to install it, no updates (who’d want to migrate their own bare-metal Postgres DB anyway?).
If we just assume that you magically come across a running Polarion instance which you don’t have to maintain, you’ll be greeted by software which looks like a very complex version of Jira from the nineties. Configuring it yourself? Borderline impossible. Even using it without any training? Hard to imagine.
From what I’ve seen, I think that Polarion, correctly configured, and used by a team which was very, very well trained on it, can ensure good compliance. But it’s certainly not a very usable tool and I can’t imagine any developer using this tool willingly when being presented with modern alternatives like GitHub / GitLab.
Pausing the ranting for a bit, Polarion could work for you if your regulatory person already has experience with it and you have someone to maintain your local installation. In all other cases, it’ll be a (steep) uphill battle.
Qualio is the new kid on the block, fueled by VC money and unicorn growth. Its pricing model is just as opaque as Greenlight Guru and the rumors are prices are in a similar range (minimum 5-figure USD per year).
Its simpler than full-blown tools like Polarion. The usability is still not great, and the text editing experience is quite subpar to tools like Confluence and GSuite. There are some neat features like linking documents a document review processes, but.. it just feels like one of those tools which were written by software engineers (or even business people) with no UI/UX engineers (or even customers) in mind.
Think of it like a crappy version of Confluence with some QMS features included. You gain compliance features like change history, review and signing, but your trade-off is that the actual text editing experience is much worse.
I would not choose it. Instead of getting compliance features out of the box while having crappy usability, I’d prioritize usability of a tool and add the compliance features on top via plugins or processes (GitLab / GitHub / GSuite).
Formwork is a creation of OpenRegulatory. More information can be found here: Formwork
But that’s just me. The fact that many of these companies exist shows that I’m certainly not representative for all those medical device companies out there. The enterprise software market is certainly a strange place.
Steve Jobs also noticed that:
What I love about the consumer market, that I always hated about the enterprise market, is that we come up with a product, we try to tell everybody about it, and every person votes for themselves. They go ‘yes’ or ‘no,’ and if enough of them say ‘yes,’ we get to come to work tomorrow. That’s how it works. It’s really simple. With the enterprise market, it’s not so simple. The people that use the products don’t decide for themselves, and the people that make those decisions sometimes are confused.