Connected islands, not one big database

Every company I have worked with has had the same dream at some point: put everything in one place. One database, one search box, one answer. I understand the appeal, and I no longer try to build it.

Why one big store fails

Data has shapes. A contract is prose with a structure of clauses and amendments. A payroll register is rows and columns with totals that must reconcile. The relationships between companies, people, and policies form a graph, where the interesting facts are the links. A regulation is prose again, but with an authority that a memo does not have. Force all of that into one store and you lose what makes each shape useful. Tables stuffed into a document index cannot be summed. Prose stuffed into tables loses its meaning. A graph flattened into either loses the links.

The attempt I see most often is loading everything, including years of email, into a single retrieval index with an entity graph on top. The graph fills with every name in every message, the store slows down, and the answers get vaguer as the corpus grows. More data made the system worse. I have made this mistake myself, which is how I know what it looks like from the inside.

Islands, by shape

So I build islands. Documents go into a retrieval store that keeps provenance. Structured data goes into a relational database. Relationships go into a registry with one canonical record per company and person, and every alias linked to it. Regulations go into their own corpus with their authority recorded, so that a rule outranks a memo when the two disagree.

Each island is the best tool for its shape, and each can be checked on its own terms: the table reconciles, the document cites, the registry resolves. When one of them is wrong, you can tell which one.

The agent is the integration layer

What connects the islands is not a bigger database. It is an agent, a model with tools, that receives a question, decides which islands it needs, calls each one, and assembles an answer that says where every part came from. Ask about a client, and it resolves the name in the registry first, then pulls the documents filed under that record and the figures booked to it. The person asking never sees the islands. They see one answer with its sources.

What the registry does

The registry deserves its own mention, because it is the piece nobody plans for. In any company with a few years of history, the same customer appears under four names across contracts, spreadsheets, and email. Most “the system doesn’t know about that” problems are really “the system knows it under a different name”. A registry that gives each company one identifier, and treats every other spelling as an alias, is what lets the islands talk about the same thing.

Building it is mostly patient matching: rules for the easy cases, review for the rest, and a record of who resolved what. It is not glamorous work, and it is the work that makes everything else answerable.

Connected islands are less impressive to draw than one big box. They are also what works.