How to Unlock the Hidden Knowledge in Your Enterprise

AI and Retrieval-Augmented Generation (RAG) allow enterprises to tap into and exploit knowledge previously locked away in disparate documents, sources, and systems. In the process, they enable faster and more accurate decision-making.

“Already some enterprises have introduced these technologies - not just to reduce the time involved in making important decisions, but also to facilitate better ones,” explains Curtis MacDonald, a Product Manager at Invisible. 

“Those firms that aren’t on the path to doing the same, are likely to find they’re at a distinct disadvantage.” 

How Knowledge is Locked Away

Enterprises of all kinds often know a lot more than they realize, MacDonald says.  

“Since the day they opened their doors, they’ve probably been answering customer or client queries, pitching for business, and managing suppliers. Professional and financial services firms will have also been analyzing laws and regulations, carrying out complex due diligence and advising on sophisticated deals.”

“In the process, they will have generated a lot of knowledge - most of which remains untapped, or at least under-exploited.” 

MacDonald says the key reason for this is that this knowledge is locked away in ‘unstructured’ formats such as memoranda, reports, contracts, and documents.

“Some enterprises have already introduced these technologies - not just to reduce the time involved in making important decisions, but also to facilitate better ones,” explains Curtis MacDonald, a Product Manager at Invisible. “Those firms that aren’t on the path to doing the same are likely to find they’re at a distinct disadvantage.”

MacDonald says that enterprises of all kinds often know a lot more than they realize. “Since the day they opened their doors, they’ve probably been answering customer or client queries, pitching for business, and managing suppliers. Professional and financial services firms will also analyze laws and regulations, carry out complex due diligence, and advise on sophisticated deals.”

“In the process, they have generated a lot of knowledge - most of which remains untapped, or at least under-exploited.”

The systems don’t necessarily integrate or talk to each other, and sometimes, they may not even be digital. So, finding the right information essentially means going through every single system or document bank—a process that can take so much time and effort that it’s often not worth pursuing at all.

“Traditional technologies are great at analyzing information in structured formats, such as spreadsheets,” MacDonald notes. “But they’re much less effective with unstructured formats, which is where most non-numerical knowledge and data is stored. So humans still have to do most of the leg work, searching for information, working out what’s important, and deciphering what it means.” 

“Generative AI is different because it is powered by unstructured data and can recognize patterns and trends in ‘hard’ information such as financial reports, but also in ‘soft’ information like the tone of email responses, and even social media posts or comments.” 

This, MacDonald says, enables it to process and analyze information in ways that traditional technologies cannot, making it uniquely suited to extract insights from the vast stores of unstructured information that businesses accumulate over time.

“Is there anyone left today who can remember exactly what factors led to a company valuation for an acquisition that happened 20 years ago? Or what negotiation strategy was used, and how people felt about it?”

“AI delivers the potential for firms to tap into this information, more or less instantly."

Why RAG is the Key that Unlocks It

Retrieval-Augmented Generation (RAG) is a hybrid approach to AI that integrates retrieval mechanisms into existing generative models. In doing so, it combines the capabilities of a Large Language Model with the ability to retrieve specific, verified information to provide contextually relevant and accurate outputs.

MacDonald notes that Invisible has already helped a major investment bank implement a prototype RAG using a model trained on an initial test set of the firm’s documents. Bankers now interact with the model using a chat interface and get instant answers to questions based on this information. However, he believes this is just the tip of the iceberg of how this technology can be used and argues that it has a broader application. 

One instance, he notes where it is particularly useful is when it comes to carrying out due diligence in M&A.

“You can deploy a RAG model on a combination of proprietary and non-proprietary data to give you fully-formed, insightful answers,” MacDonald observes.

“This might include anything from your own reports, research, emails, and documents through to transcripts of interviews with the CEO, transcripts of industry experts, industry reports that you've downloaded and research analytics from Capital IQ or whatever you’re pulling Bloomberg.”

“You can then combine all of these with the entire data set from due diligence to pinpoint the information you need to know and observe patterns and trends.”

MacDonald says that this will reduce time and effort but also break down silos across an enterprise. 

“At the moment, a lawyer or banker needs to quiz the CFO on what all those documents mean and where they can find specific information. Then the analyst and associates go through, curate it, summarize it, and give it back. They may have missed something or incorrectly identified information.”

“RAG empowers a firm to do this almost instantly. It can do it accurately and provide annotations about where it obtained information, so decision-makers know they can rely on its answers too.”

“Any knowledge learned can also be shared across the whole organization so that other decision-makers take that information into account, too.”

Roadblocks in the Way 

While all of this is already possible, many enterprises still need to overcome several challenges before properly implementing RAG.  

Many lack the capabilities to deploy RAG across their operations, including natural language processing techniques. They will need to hire internal machine learning experts or outsource this to a third-party provider.

Then there is the quality of the data. 

“While AI loves unstructured data, that doesn’t mean it can accept it in any format,” MacDonald observes. “It still needs to be digitized and standardized - at least to some extent - before it can be made usable.”

This is especially true when some of the information to be used is in hard copy format. Here, MacDonald notes that humans still need to organize and upload information into the system.

“It’s likely that, at this stage, enterprises will need to combine automation with human labor to make sure their data set is usable,” he observes. “While this may require some upfront investment, it will likely yield rapid results.”

No replacement for Human Decision Making

Looking ahead, Curtis sees RAG being more deeply integrated into enterprises’ existing IT infrastructures. 

“So, for instance, rather than an analyst having to copy and paste responses into different platforms, like they currently need to, the RAG platform will engage with these systems and update an enterprise's knowledge bank automatically,” he says. 

“This seamless integration will enhance efficiency and ensure that relevant information is available exactly when needed.”

Despite increased intelligence and automation, MacDonald doesn’t believe that RAG will ever become a substitute for human decision-making.

“RAG’s purpose is not to replace human decision making but to enhance it by speeding it up and making it more accurate. That’s where it will be a game-changer.”

“Previously, if it took 10 hours to draft an investment memorandum on a 1,000-page due diligence data room, that could now take an hour or two while the analyst gets coffee,” he concludes. “And it will be executed with greater precision.”

“That alone will save countless hours, improve outcomes, and free knowledge workers up to do what they do best - make decisions.”