16 min read · Dec 04, 2025
One of the biggest misunderstandings around archives is this: if a file is sitting there, we assume the information inside it is also accessible.
But that is not always how things work.
You might have a folder full of PDFs, scanned documents, tables, reports, notes, or old records. Technically, you have data. Sometimes you have a lot of it. But when someone comes and wants to find something genuinely useful inside that archive, that data suddenly becomes heavy. Because the file is there, but the information is still scattered.
In archives, the problem is often not missing data. The problem is that the data was not kept in a form that makes it usable later. So the issue is not “do we have the document?” The issue becomes “can we actually use that document later?”
That is exactly where we looked: not just keeping unstructured data, but making it easy to access later.
On the archive side, we saw this very clearly: most of the time, the problem is not that there is too little data. On the contrary, there is too much data. The problem is not scarcity; it is that the data is abundant but scattered, abundant but disconnected, abundant but not ready for use.
People tend to think an archive becomes more valuable as it grows. At first glance, that sounds reasonable. If there are more documents, it feels like there should be more information. But if those documents accumulate without a shared structure, the archive starts to get heavier instead of more valuable as it grows. Because the thing you are looking for is probably there, but the path to it is not clear.
That is why in many archives the problem is not “we do not have data,” but “we have data, but we cannot use it when we need it.”
Because a file and information are not the same thing.
Putting a file into a system does not make the meaning inside it accessible. The file only preserves the container. But what we usually need is not the container itself, but a specific piece of information inside it.
If you have a hundred-page document, then technically you have not lost it. But if you cannot quickly reach the one clause, the one table, or the one date inside that document, then that file is not really serving you in full.
That is why the approach of “we uploaded it, we saved it, it is there” is not enough for archives. It solves storage, but it does not solve access.
Because unstructured data does not behave as a single uniform thing inside itself.
Part of it is plain text, part of it carries tables, part of it gets its meaning from heading structure, and part of it has to be read together with another section sitting next to it. In other words, not only the content of the data matters, but its form too. If you do not take that into account at the moment of storage, then when you try to access it later, the information either gets fragmented or stuck together. [1][2][3]
If you split something too much, you lose context.
If you do not split it at all, the information you need gets buried inside the whole structure.
The same thing happens on the metadata side. If you leave the document as it is, it stays unclear what the content belongs to, what topic it relates to, what date it connects to, or what other records it speaks to.
That is why unstructured data makes later use harder; because it does not carry the order needed for access by itself. That order has to be built separately.
If you want an archive to be accessible, you first have to make this distinction: storing files is one thing, accessing information is another.
You can have thousands of documents, but if you cannot pull out the right information quickly from within them, then what you have is not really an accessible archive. You just have accumulated files.
Because the job does not end once the data is sitting there.
Keeping a PDF or a document uploaded years ago inside the system does not solve much on its own. The real issue starts when a user comes later and wants to find a single date, a single name, a single clause, or a single table inside that document.
We kept hitting the same wall here: what archives often lack is not data, but data that has been kept in a workable state. The document exists inside the system, but it cannot be accessed in pieces. Its content has not been separated. Its source is known, but going down into the information layers inside it is hard.
That is why the approach of “we uploaded it, saved it, let it sit there” is not enough in the long run. Because what you actually need later is not the file itself, but the meaningful pieces inside it.
Because later on, the system largely decides what it can find and how it can find it by looking at how the data was stored.
If you store a document as a single piece, you preserve context but make access harder. If you split everything into very tiny parts, you scatter the context instead. Neither works well on its own. [1][2]
For us, the important thing was not how much we split the data, but where we split it. Does it separate from the heading here? Does the paragraph end here? Does the table establish its own integrity here? Storage done without looking at these things harms access later. [1][2]
A good storage format makes this possible:
In short, what makes an archive accessible is not that it contains documents; it is that those documents are kept in a way that makes them readable, separable, and retrievable later.
We realized this early on: raw data is almost never ready for direct access.
A file does not become usable the moment it is uploaded into the system. There has to be a serious preparation layer in between. If that layer is not set up well, the entire access layer that comes after it weakens.
For us, the important thing in this layer was not just cleaning the data. The main goal was to make the data workable later.
For that, we looked at a few core things:
First, data has to enter the system in a controlled way. Problematic formats should be filtered out at the beginning. If the same file is coming in again and again, that should be recognized.
Second, the structure must not be lost. Especially in unstructured documents, meaning does not live only in the text itself. It lives in the heading, in the table, in the cell order, and sometimes in the position on the page. If that structure falls apart while converting to plain text, the document may look as if it has been read, but the information has still been reduced. [1][2][3]
Another important thing is provenance, meaning the trace of the source. If we are storing a piece, we want to keep not only its text, but also where it came from. Which document, which page, which section. Because if you want reliable access later, you cannot afford to lose the source of the thing you found. [7]
On our side, this preparation moved through several layers.
In the first layer, we ingested the file, validated it, and filtered duplicates. Then we said extracting the text is not enough; we also have to try to preserve the structure. Headings, paragraphs, lists, tables, fields. Because one of the things that makes a document genuinely usable in the future is preserving its internal order as much as possible. [1][2][3]
Then we separated the content into meaningful units. But we did not do that blindly. Splitting into fixed sizes is easy, but it usually produces poor results. We moved closer to a preparation logic that follows natural boundaries. Where does a section end, where does a new topic begin, where does a table carry meaning on its own — that is what we looked at. [1][2]
The next step was the metadata side. In other words, not just producing the text, but also producing the signals around the text. Fields like document type, date, person, institution, and topic come into play here. [1][2]
Finally, the relationships between pieces have to be made visible. Similar documents, repeated records, content clustered around the same topic. These also make later access much easier.
Put together, the picture is simple:
Raw data is ingested.
It is cleaned.
Its structure is preserved.
It is separated at meaningful places.
It is enriched with metadata.
Its source is kept fixed.
Because what makes unstructured data accessible later is not just that it has been stored, but how it has been divided into pieces.
If you make a bad decision here, then no matter how good a system you build later, something will always stay missing. Either the information scatters, the context breaks, or the thing you are looking for gets buried inside a large piece.
What we saw here was this: some things may look physically separate, but semantically they have to stay together. Some other things may sit side by side inside the same document, but actually need to be handled separately.
For example, a heading and the explanation immediately under it should usually stay together. Because the meaning of that paragraph is fed by the heading.
The same kind of thing happens with lists. Sometimes the introduction sentence carries the context for all the items in the list. If you detach it, the items become semantically poorer.
Tables are more sensitive. Pulling a table row by row can sometimes work, but just splitting by row is usually not enough. If the header row, column names, and relevant cells are not considered together, the data remains there but its meaning becomes blurred. [1][2]
In some places the opposite happens. When a very long section stays as one piece, the context is preserved, but access becomes harder.
The general logic became this:
Because later on, the system sees the information through those pieces.
If you store very large pieces, the signal of the thing you are looking for becomes weak.
If you store very small pieces, then this time the context disappears.
In our experience, neither of these extremes worked well on its own.
Fixed-size chunking was easy, but it was usually too crude. Especially in unstructured documents, chunking everything by the same length means dividing the document according to size, not according to meaning.
What worked better was chunking that listened to the natural rhythm of the structure. Does the paragraph end here, does the heading change direction here, does the table complete itself here — when we separated according to that, access became noticeably easier. [1][2]
We did not think in terms of “one correct chunk size” here. Because most of the time, such a thing does not really exist.
The best separation is not a fixed formula that stays the same for every document. It is a structure that changes depending on the type of data, while still protecting a few core principles.
For us, those principles were:
In short, good semantic separation does not crush the information, does not scatter the context, and makes later access easier. [1][2][7]
Separating data at the right semantic places is not enough on its own. Because what makes later access easier is not only the text itself, but also the signals the text carries around it.
What we saw here was this: the same content turns into something very different when it is stored with the right metadata. Because now the system is no longer looking only at “what is written inside this text?” It can also look at “what does this piece belong to, what type does it fall under, in what context does it stand?” [1][2]
The metadata layers that were actually useful for us were mostly these:
Identity metadata
Which document does this piece belong to, which section did it come from, on which page does it sit?
Temporal metadata
Date, period, version, recency information.
Entity metadata
Person name, institution name, place name, project name, product name.
Document type metadata
Is this thing a contract, a report, a form, a table?
Topic metadata
What is this piece about, which concepts does it carry?
Relationship metadata
What other content is this content connected to?
In short, good metadata is not metadata that looks fancy; it is metadata that makes later finding, filtering, separating, and relating easier. [1][2][7]
Because metadata is not a note pasted on afterward. It is part of the layer that determines how the data will live.
In some cases, the text itself is not enough. The user may not know the exact sentence, but they know the date. They may not know the document name, but they know the institution. They may not remember the content, but they know it was a table or a contract.
At that point, metadata stops being a supporting layer. It becomes directly part of the access logic.
There is also the application side of the story. If metadata is set up well, not only search but also filtering, grouping, linking, clustering, and sorting all become easier.
The right storage format does not look very visible at first, but later its effect appears everywhere.
Because well-stored data is not just data that looks more organized. It changes how the archive itself gets used.
When data is stored well, access does not happen through a single path. You do not just search for one word and wait for a result. You can go through the date, through the person, through the institution, through the topic, through similar records. [4][5][6]
In other words, you end up with more than one door into the archive.
There is another comfort too: data that is chunked and marked correctly makes it possible to go directly to the relevant piece inside a large document without getting drowned in it.
Because the value of an archive has less to do with the number of files and more to do with how comfortably you can work with it.
In a badly stored archive, the data exists, but the working comfort is low. Reaching the same information takes a long time. Seeing similar records side by side is hard. Building relationships between documents on the same topic takes extra effort.
In a well-stored archive, the same data feels lighter. Because the content becomes more readable. Similar documents become more visible. Repeated content becomes easier to notice. It becomes clearer what each piece belongs to.
In other words, the archive is the same archive, but the relationship you have with it changes.
This structure also has a direct effect on the application side.
When data is stored well, everything you build on top of it works more comfortably. Filtering becomes more meaningful. Search works better. Structures that show similar content become stronger. Document screens become more informative. Connections between records become more visible. [4][5][6][7]
In short, good storage does not just improve the internal order of the archive. It makes the whole application stronger.
What makes unstructured data valuable is not just storing it, but storing it in the form in which it will be easiest to use later.
The more we looked at this topic, the clearer one thing became: the biggest problem in archives is often not missing data. It is that data was kept from the beginning without being thought through for access.
That is why a good archive is not just an archive that stores.
It first prepares the data.
Then it separates it at meaningful places.
It strengthens it with metadata.
It preserves its source.
And it does all of this to make access easier later.
When unstructured data is stored as-is, it only accumulates.
When it is stored properly, it becomes genuinely usable.
[1] Y. Ding, J. Lee, and S. C. Han, Deep Learning based Visually Rich Document Content Understanding: A Survey, 2024. arXiv:2408.01287 [cs.CL]. [Online]
https://arxiv.org/abs/2408.01287
[2] Q. Zhang, V. S.-J. Huang, B. Wang, J. Zhang, Z. Wang, H. Liang, S. Wang, M. Lin, C. He, and W. Zhang, Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction, 2024. arXiv:2410.21169 [cs.CV]. [Online].
https://arxiv.org/abs/2410.21169
[3] Y. Xu, M. Li, L. Cui, S. Huang, F. Wei, and M. Zhou, LayoutLM: Pre-training of Text and Layout for Document Image Understanding, 2019. arXiv:1912.13318 [cs.CL]. [Online].
https://arxiv.org/abs/1912.13318
[4] S. E. Robertson and H. Zaragoza, “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009. [Online]
https://www.nowpublishers.com/article/Details/INR-019
[5] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, Dense Passage Retrieval for Open-Domain Question Answering, 2020. arXiv:2004.04906 [cs.CL]. [Online].
https://arxiv.org/abs/2004.04906
[6] O. Khattab and M. Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, 2020. arXiv:2004.12832 [cs.IR]. [Online].
https://arxiv.org/abs/2004.12832
[7] T. J. Green, G. Karvounarakis, and V. Tannen, “Provenance Semirings,” in Proceedings of the Twenty-Sixth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS ’07), 2007, pp. 31–40. DOI: 10.1145/1265530.1265535. [Online].
https://doi.org/10.1145/1265530.1265535