The Hidden Cost of Unstructured Data in Your LLM Strategy

An industrial equipment manufacturer spent eight months and €400,000 implementing a retrieval-augmented generation (RAG) system. The goal was straightforward: let engineers query their documentation and procedure manuals through a conversational interface instead of digging through thousands of PDF files.

In testing, the system worked beautifully. Engineers asked questions in natural language, and the LLM retrieved relevant documents and answered correctly. The team launched it to the engineering department. Three weeks later, adoption had stalled. Engineers weren’t using it.

When we investigated, the problem wasn’t technical. It was data. The documentation, spread across twenty years of PDFs, was inconsistent. Some procedures were superseded by newer ones, but nobody had deleted the old versions. Some documents were duplicates with different filenames. Critical procedures were buried in spreadsheets rather than in central documentation. The system was retrieving information, but engineers had learned not to trust it because it would sometimes return outdated guidance.

The LLM wasn’t the problem. The underlying data governance was.

The Unstructured Data Problem

Large language models require data to be useful. For most organisations, useful data is unstructured: documents, emails, PDFs, meeting notes, design files, customer interactions. This data lives across multiple systems, in inconsistent formats, with varying quality and accuracy.

The promise of RAG systems and LLM applications is that you can point them at your existing data and extract value. In practice, that promise collides with reality. Most organisations’ unstructured data is:

Incomplete. Critical information lives in people’s heads or in informal systems outside the main repository. An insurance company’s claims documentation might be in a database, but the informal notes about why a claim was approved or denied live in email or old memos.

Inconsistent. The same concept is named differently across documents. “Client” and “customer” might mean different things. Procedures are documented at different levels of detail. Some documents are a paragraph long; others are 50 pages.

Outdated. Old documentation isn’t deleted; it’s archived. An engineer searching for “procedure for assembly” might find versions from 1995, 2010, 2018, and 2024. Which is current? The system has no way to know.

Duplicated. The same document exists in multiple places, sometimes in different formats. A procedure might exist as a PDF, a Word document, and embedded in an email thread.

Untagged and disorganised. Metadata is missing or inconsistent. Documents lack clear creation dates, ownership, version numbers, or relevance tags.

For humans, this is annoying but manageable. We develop contextual knowledge. We know which documents to trust. We know who wrote what and whether they’re reliable. We understand versioning implicitly.

LLMs have no such context. When you feed them unstructured data, they process it all equally. They have no way to distinguish authoritative documents from outdated ones, current information from historical context, verified facts from speculation.

Why This Costs More Than You Think

Most organisations underestimate the cost of getting unstructured data ready for LLM applications.

Cost of data preparation. Before you can use data in a RAG system, it needs to be reviewed, deduplicated, tagged with metadata, and often reformatted. A manufacturing company might need to review 50,000 documents to identify which are current. That’s not a software problem; that’s a labour-intensive data governance problem.

Cost of quality assurance. Once you’ve implemented a RAG system, you need to test it. Does it return accurate information? Does it confuse outdated and current guidance? Testing at scale requires evaluating hundreds or thousands of query results. Many organisations discover, after launch, that the system is unreliable because the underlying data is.

Cost of ongoing maintenance. Data quality degrades. New documents come in. Old ones become obsolete. Someone needs to curate the data continuously. In organisations we’ve worked with, this is often a half-time role ongoing, once the initial launch is complete.

Cost of false confidence. This is harder to quantify but real. If your LLM-powered system looks authoritative but returns incorrect information, the damage is significant. An engineer following bad guidance can cause defects. A customer service agent relying on inaccurate information damages trust. The cost of unreliable systems is often higher than the cost of no system at all.

In the equipment manufacturer’s case, the €400,000 development cost was dwarfed by the £200,000 cost of data preparation and quality assurance that should have happened first. And the deployment cost—identifying outdated documentation, consolidating duplicates, establishing versioning discipline—would have required another €150,000 in labour.

Had they invested in data governance first, the LLM implementation would have been cheaper, faster, and more effective.

The Governance Question Underlying the Technology Question

The deeper issue is that LLM adoption forces organisations to confront data governance questions they’ve been avoiding.

“Where is our critical information actually stored?” For many organisations, the answer is “everywhere and nowhere.” It’s in databases, file shares, cloud drives, emails, and people’s memories. No single authoritative source exists.

“How do we know what’s current?” Without versioning discipline and metadata standards, you can’t. The best you can do is approximate based on modification dates or trust someone’s memory.

“Who is accountable for data accuracy?” In many organisations, the answer is “nobody explicitly.” Documents are created by various people for various reasons. Nobody is formally responsible for saying “this is accurate” or “this is now obsolete.”

“How do we ensure consistency?” Without data standards and governance, consistency is aspirational. Documentation uses different terminology, different levels of detail, different formats.

LLMs make these questions urgent. You can’t build effective AI systems on top of chaotic data governance. So the real project, before implementing RAG or LLM applications, is data governance.

Three Prerequisites for LLM Success

Based on work with multiple organisations implementing LLM systems, three conditions predict success or failure:

First: Authoritative source designation. You need to identify which documents are authoritative and which are reference material. This might mean designating a particular version of a procedure as “current” and archiving older versions. Or maintaining a master version of a policy document and marking copies as “reference only.” Without this, your LLM will confidently cite outdated information.

Second: Metadata discipline. Every piece of data that will feed an LLM needs consistent metadata: creation date, last modified date, owner/authority, version number, status (current/archived/draft/review). This doesn’t require expensive systems. It requires discipline. A spreadsheet tracking critical documents, with consistent fields, is often enough to start.

Third: Curation responsibility. Someone needs to be formally responsible for data quality in the LLM system. This isn’t the IT team’s job alone. It requires domain expertise. An engineering manager should own procedure documentation. A compliance officer should own policy documents. The IT team supports, but the business owns accuracy.

When these three conditions are present, LLM implementations work. When they’re absent, they fail, regardless of how sophisticated the technology is.

The Data Governance Path

For organisations wanting to pursue LLM capabilities, here’s the sequencing we recommend:

Phase 1 (Weeks 1-4): Audit and locate. Where does critical information live? Make a map. You don’t need perfect completeness, but you need to understand the landscape. What’s in the database? What’s in documents? What’s in emails? What’s in people’s heads?

Phase 2 (Weeks 5-8): Identify authoritative sources. For the highest-priority information (procedures, policies, customer data, financial information), which document or system is the source of truth? In many organisations, the answer is “it’s unclear.” Clarify it. Designate a primary source and version everything else as reference or archive.

Phase 3 (Weeks 9-16): Establish metadata discipline. Retrofit critical documents with consistent metadata. At minimum: current status, owner, last review date. Use a spreadsheet if you don’t have a database. Consistency matters more than sophistication.

Phase 4 (Weeks 17-24): Assign ownership and curation responsibility. Designate who owns data accuracy for different domains. Create a simple process for updates: who can modify documents, what approval is needed, how are changes communicated?

Phase 5 (Ongoing): LLM implementation. Once you have reasonable data governance, you can implement RAG or other LLM systems. They’ll be faster, cheaper, and more reliable because the data is already in reasonably good shape.

This is 4-6 months of focused work, before you write a single line of LLM code. It feels slow. But it’s faster than discovering six months into an expensive LLM project that your data governance is broken.

The Strategic Framing

One mistake organisations make is framing LLM readiness as a “technology problem.” It’s not. It’s a governance and process problem. Data governance isn’t exciting. It doesn’t generate press releases. But it’s the foundation for everything that follows.

In DACH markets specifically, this resonates. German and Swiss organisations typically have stronger data governance traditions than American companies. Compliance frameworks like GDPR are already driving data discipline. Process documentation is valued. The cultural expectation is that data is organised and accountable.

Framing LLM adoption as “a governance opportunity, not just a technology play” often resonates with IT leaders and business stakeholders in these regions. It allows you to invest in data governance, which has benefits beyond LLM applications, while building capabilities for AI systems in parallel.

Bottom Line

LLM adoption requires data governance. Most organisations don’t have it. The cost of data governance is often underestimated and pushed to the end, when it should happen first.

Organisations that succeed invest in data governance as a prerequisite, not an afterthought. They identify authoritative sources. They establish metadata discipline. They assign ownership. Then, when they implement LLM systems, those systems work because the underlying data is reliable.

The equipment manufacturer would have saved hundreds of thousands of euros, months of timeline, and significant frustration if they’d invested in data governance first. Instead, they learned the hard way that LLM success depends on data governance success.

That lesson is cheaper to learn from reading than from building expensive systems on broken data.

This article is based on extensive research. The full evidence base with sources is available upon request.