A startup recently asked me if they can use Formwork to generate their entire technical documentation with our ChatGPT feature. We haven’t built that feature (yet), but what I’d like to talk about is what sort of expectations companies should have when it comes to using ChatGPT for regulatory documentation.
The TLDR is that LLMs will only make some part of the regulatory pain go away, but they won’t be able to completely solve your regulatory problems.
Let’s pause and do a thought experiment for a second. Let’s assume a LLM-based product exists which generates perfect technical documentation for you in one shot – no time effort and customization needed. How useful would this be?
I’d argue that this is only of limited usefulness. The typical activities in regulatory work right now are:
- Drafting regulatory documentation (cool, this is solved).
- Explaining it to auditors (ugh)
- Answering tons of auditor questions about it (ugh)
- Explaining it to your company (ugh)
- Updating it with your company (ugh)
You see, even a perfect LLM would only solve a part of your regulatory pain. That’s because you still have to understand your own documentation, as you’ll have to explain it to auditors and answer their questions.
This, by the way, is the same reason that regulatory compliance is (unfortunately) different from doing your company taxes – with your taxes, you can outsource everything to your tax advisor, because that’s also the person who will handle questions from the financial authorities. With regulatory compliance, it’s unfortunately you who has to handle questions from the regulatory authorities. So you won’t get around actually understanding your own documentation.
ChatGPT For Regulatory Documentation: Useless?
I’m not saying that LLMs are useless – far from it, actually! A lot of regulatory documentation is (let’s be honest) a lot of words with very little content, and drafting that sort of stuff is a job where LLMs really shine. But, again, that’s only a (small) part of the total task.
So, for now, there’s only limited usefulness for ChatGPT for regulatory documentation.
How would a better future look like? I think you’d need some sort of end-to-end process – like, you check in a commit to your code repository, some magical software extracts the regulatory information from that (e.g. which features changed?), that gets submitted to auditors, then they have some magical software which checks whether that’s okay, and reply with their assessment.
So it boils down to being an end-to-end process, where you take out the human element of “a dude has to explain it to auditors, and random dudes / dudesses have to understand the documentation at various points in time”.
See also: We’re In A Rush. Can We Automate Our Technical Documentation Completely?
