Why lawyers (and other professionals) are disappointed with AI software demos
Perhaps the tech isn't to blame.
Why are you disappointed with AI software demos even though LLMs are capable of such magical feats?
Here are some things you should know:
LLMs are not current.
This means any AI tool that is used for RESEARCH needs to be hooked up to an old fashioned real time system that updates an external library that the LLM can access. When you are shown a demo, it might be that this happening, but do you know? Tech folks call this sort of a system - RAG system (retrieval augmented generation). However, to be able to rely on it, you must be Westlaw/Manupatra sure the system is abreast of what is going on.
If it is a specialised service, say tracking one set of regulations or one type of courts, and seriously, you will have better luck being accurate with research using the tool. If it is accessing a global database you are already familiar with or an internal database that you keep updated, that could be another way to mitigate the risk.
Another example.
Our free Document Review Platform.
Here’s how it went.
What it is actually is a rag tag RAG system. Takes any 5 documents, parses them using a PDF/Docx parser, searches for answer to your query (just simple chunking, no hops, no sophistication) using a retrieval LLM (in this case, Voyage Law 2 embedding model). Runs the top 8 results into another LLM context window for reasoning and generation of the answer (in this case, OpenAI's o3 mini or 4o, not sure).
Many were quite happy with the answers, but some flat out found the answers wrong.
This was expected not because we made technologically sub-par choices (which we did) but because its as good as any "general" going to nowhere question answer system can do.
In these two examples, you might already see the problem.
It isn't the technology. Even if we use what we think is the best tech set up for either, without narrowing down on the workflow and the team that is doing the work, and what is sought to be achieved in the LEGAL sense of the term, we will always disappoint.
What does this mean?
Being clear about which regulations need to be updated, and how they need to be handled is what makes the difference in the research example. In a drafting and review, the same thing is at play.
Unless you have a clear idea of what kind of question is being asked, what kind of context is being managed, what kind of risk is being analysed, what kind of document is being drafted, what template is the standard or what the possible outcomes are - drafting, and question/answer engines don't offer as much assurance as they should.
But when you do, you actually see the magic.
In other news
We re-branded our legal AI studio as Axara AI.
What we do is empower legal teams to marry their professional expertise with technical expertise to create AI agents that actually work. Don't forget to try out our free-to-use agents on our website!