How well does an Atlassian Jira / Confluence QMS work?
For whatever reason, Confluence and Jira still seem to be popular choices among software companies. Why? I’m not so sure. Maybe it’s the same reason startups adopted microservices: It doesn’t make sense, but Google does it, so it must be great (spoiler: no). Anyway!
If you’ve noticed that this is turning into a rant, you’re damn right. I hate Jira and Confluence – they’re slow, customizing them is a full-time job, and they create so much complexity in teams that you’re wondering why the hell you introduced the tool in the first place.
But I’m going to attempt to stay objective here, and I’ll start by citing a lead auditor from a medical device Notified Body:
I hate Confluence and Jira.
A lead auditor at a Notified body
Confluence QMS Drawbacks
Some Auditors Don’t Like It
I recently was in touch with a leading auditor of a Notified Body (who shall go unnamed, for obvious reasons). His take on Confluence and Jira is that “it’s a catastrophe” due to the following reasons:
- The self-hosted version of Confluence / Jira is going away as they’re trying to move all their customers to their cloud-hosted, subscription-based offering. This introduces problems because they do something like “rolling releases”, i.e. they deploy new features to your instance whenever they want, with no prior notice. That means that stuff can randomly break (especially eQMS plugins!), with grave consequences (see below). But it also means that you can’t really “validate” the software which is a regulatory requirement. In short, you need to list all its features and how you’ve tested them and document that, and how the hell do you document that if the features are randomly changing all the time?
- The auditor mentioned one particular incident where a customer was impacted by this “automatic upgrade” issue which caused them significant data loss across their documentation. The auditor knew that this was related to Confluence / Jira because they had seen the documentation prior to the upgrade while it was still complete!
These and other incidents have made many auditors more cautious when auditing companies using Jira / Confluence for their QMS. The most important aspect of this is that companies are now often expected to have contingency plans in place in case their Confluence data (or parts of it) randomly disappear. So, in practice, you’ll have to build some sort of backup solution as the cloud-hosted Jira / Confluence instances can no longer be trusted for medical device compliance purposes.
The Data Loss Incident
If you’re wondering why this auditor also hates the Atlassian tools, it’s quite simple: Atlassian had a data loss incident in April 2022 which left a number of companies locked out of their accounts, or with data loss, or both. This is essentially the worst case for running a Jira / Confluence QMS: Imagine you’re getting audited, and you don’t have access to your documents. Ugh.
Exportability
While Confluence and Jira have export features, the output can be described as “well, it exists, but it’s not great”. What I (and auditors) typically see are lengthy Word or PDF files. The tables in those files typically have columns which are so slim that they are unreadable. Having any sort of other data export would involve coding effort on your side, so it’s not feasible for most companies.
That being said, there also are (gasp) benefits to choosing the Atlassian tools for your QMS.
Atlassian Jira / Confluence QMS Benefits
Pricing
The pricing is, like.. not super bad. It’s priced per user, and it’s reasonable.
And yup, that’s it. I don’t have anything else to say in favour of setting up your QMS in Confluence and Jira.
Conclusion
Definitely check out some of my other articles on the topic – startups ask me about setting up their QMS in Confluence and Jira all the time, so I hope this spares my inbox of these painful questions on this painful “software” in the future:
Also, check out this video I recorded in which I attempt to set up a QMS in these Tools Which Shall Not Be Named.
3 comments
A.B.
I appreciate your perspective, but I don’t entirely agree with your stance.
Regarding the comparison of tools like Google Drive or Jira to dedicated QMS software: I understand your point about the “real-world” usage of these tools. However, labeling them as QMS software simply because people force them into that role feels misleading. It’s important to differentiate between tools explicitly designed for compliance-critical tasks and those retrofitted to meet regulatory needs. The analogy of “sleeping in cars” might resonate, but in regulated industries, the consequences of using inappropriate tools can be significant. Shouldn’t we advocate for fit-for-purpose solutions rather than criticizing tools for being used outside their intended scope?
I want to emphasize that my comments aren’t personal attacks. Your insights are valuable, and I don’t disagree with everything you’ve written. However, it’s essential to frame criticisms of tools like Jira or Confluence within the context of their intended use cases while also holding users accountable for choosing the right tools for the job.
As for Formworks, I really appreciate your minimalistic approach and the provision of validation reports and proactive communication about breaking changes. These are strong differentiators. However, the rapid pace of incremental updates raises some questions about validation. Why wouldn’t incremental changes require re-validation? As a customer, how can I differentiate between incremental and breaking changes? Additionally, since Formworks can’t be self-hosted (correct me if I’m wrong), it would be a standout feature if customers could choose the version they’re running and control when to update. While I understand this could depend on infrastructure and architecture, offering such flexibility could set Formworks apart from competitors (not naming Jira here 😉).
BTW, Jira/Confluence does have a free trial – maybe you want to update this in your overview.
A.B.
How does Google Drive or Jira qualify as QMS software? Yes, I know, you can install plugins and invest heavily in configurations, etc., but it would be great to only list software in the overview which is being advertised as QMS software. Otherwise it’s comparing apples and pears. If you’d ask anyone at Google or Atlassian if they offer an eQMS product, the answer would probably be “no”. So, complaining about Jira Cloud updating versions without notifying customers is not really fair. These products simply have different purposes and customers in mind. Blame the customers who are trying to build a medical device on these platforms, but don’t blame the platforms!
I would also like to understand how OpenRegulatory handles updates of its Formworks product. I tried the free trial version for a while and realized that the software was being update without any prior notice (only for October I counted 11 updates). How am I supposed to validate the software and explain this to my auditor? What if I get the auditor you mention in the article – would he hate Formworks because you do something like “rolling updates”?
Dr. Oliver Eidel
Sure, you’re right in the strict technical sense that GDrive and Jira are not marketed as QMS software. But this is typical “ivory-tower thinking” of regulatory people: Saying this like “I told you so! It’s not a real QMS software, and you still chose to use Jira, you’re screwed, and it’s your fault!”. This is neither productive nor helpful, as it ignores the fact that, after all, in reality, people do use Jira / Confluence for their QMS, through whichever contorted way they arrived in this situation.
It’s like blaming people for sleeping in their cars: In your words, from your ivory tower, you could say “If you’d ask any of the car manufacturers if they offer a sleeping product, the answer would probably be “no””. And that’s right. But the real world is less clear-cut, and humans use tools in whichever way they see fit. And that’s fine, and we have to accept that and make it work.
That’s why I do consider GDrive and Jira / Confluence QMS software. Again, not because they’re marketed as such, but because the real world shows that they are being used as QMS software. I’ve talked to a fair share of auditors in this industry, and Jira / Confluence is actually a fairly common choice (sadly so, I might add).
Regarding updates to Formwork: When shipping breaking changes, we message customers in advance. This hasn’t happened yet, as we haven’t had a breaking change. All other changes are incremental and, as you noticed, posted to our changelog. These don’t require re-validation – in fact, we provide a validation report to our customers so that they don’t need to do the entire validation from scratch. Customers always passed their audits with this approach – this is true across auditors, notified bodies and even legislations (MDR / FDA).
And.. the auditor I mentioned in the article likes Formwork 🙂