Weighted, Tiered Data: Keeping Fresh Insight Connected to Deep Knowledge

A new interview, workshop reaction, or client brief can change the meaning of years of accumulated research. Yet a large document collection can bury that signal: a familiar theme appears in dozens of old reports, while the new behaviour appears in only a handful of recent contributions. Document volume alone is a poor guide to what matters for the next decision.

The architecture proposed here starts with a different question: what should be closest to the decision today? For a Whale.AI project, that might be the current brief, recent research, live workshop contributions, and the constraints the client has just confirmed. These form a compact working layer. The larger document base remains available beneath it, ready to supply history, definitions, comparisons, and counter-evidence.

This is a design pattern for connecting fresh insight to organisational knowledge. The tiers describe the role of the evidence in an analysis, rather than a claim about a particular deployed storage system.

The top tier is the active evidence set. Each item needs enough context to interpret it: source, observation date, audience, project, and any limits on its use. A recent upload of an old study is still old evidence. Keeping observation and ingestion dates separate helps prevent a refreshed file from looking like a refreshed finding.

The underlying tier contains the broader authorised document collection: previous studies, brand strategy, category reports, historic concepts, and project learning. Documents can be indexed as retrievable passages while retaining links to their parent source, version, and surrounding text.

Between them sits an agent workflow. It formulates a targeted question, searches within the permitted collection, assesses the returned material, and passes useful evidence into synthesis. The diagram shows the intended information flow; a retrieved passage only becomes part of the working interpretation after its relevance and provenance have been checked.

Weighting means making the reasons for prioritising a source explicit. Recency matters, but so do relevance to the brief, fit with the audience, quality of the research, and the authority of the source for the question being asked. A current workshop comment can be a valuable emerging signal without outweighing a well-designed study of the same audience.

Separate eligibility from ranking. Client permissions, approved project scope, and document access determine whether a source can be retrieved at all. Only eligible sources should be ranked. Within that set, an illustrative ranking approach combines task relevance, evidence quality, audience fit, and a recency factor that reflects how quickly the subject changes.

There is no universal set of weights. A question about a fast-moving campaign may need a strong recency preference; a question about established brand commitments may depend on an older authoritative strategy document. Tune the approach against representative questions and researcher judgement. A retrieval score expresses usefulness for a task, not the probability that a claim is true.

The predictive element is anticipatory retrieval. As a theme forms, an agent can propose the questions a researcher is likely to need next: have we seen this before, does it vary by audience, what contradicts it, and which brand constraints matter? It can retrieve candidate context before the team asks each question separately.

Imagine a fresh workshop in which participants describe premium packaging as wasteful. An agent might look for previous sustainability research, evidence about willingness to pay, and current packaging constraints. The purpose is to investigate whether the reaction reflects a wider tension or a specific stimulus, rather than immediately declare that consumers have rejected premium packaging.

Those searches should be bounded by the task, permissions, and a retrieval budget. If a targeted search produces no adequate evidence, the output should say what remains unknown. Predicting a useful question is helpful; filling an evidential gap with a plausible statement is not.

Useful retrieval may complicate the story. Recent participants might favour simplicity while an earlier study links rich packaging to gifting occasions. Keep both findings visible with their dates, audiences, and contexts. The difference could reflect a change over time, a different usage occasion, or a difference in method.

The synthesis should distinguish what the fresh evidence shows, what the archive adds, and what the team infers from the combination. Attach source references to the claims they support. Repeated summaries of the same study should not be counted as independent confirmation.

A human reviewer then decides whether the emerging interpretation is supported, needs qualification, or calls for further research. Generated summaries can help navigation, but should remain labelled as derived material rather than quietly becoming new primary evidence.

The loop needs to run again when meaningful new evidence arrives, a source is superseded, or the brief changes. Refresh the affected retrievals and summaries, preserve the previous interpretation where needed, and make the last review date visible. A live connection to documents does not, by itself, make every insight current.

Evaluate the approach using real project questions. Check whether relevant new signals appear, whether historical context is appropriate, whether citations support the actual claims, and whether contradictions survive synthesis. Include cases where the correct response is that the evidence is insufficient.

For teams working in Whale.AI, the practical ambition is continuity: current client contributions lead the inquiry, accumulated knowledge adds depth, and every resulting direction remains open to challenge. Freshness becomes a property of how evidence is reviewed, rather than a label attached to the newest file.