We have a junior research student who does this on slack.

Before I think he was using AI and other tools to translate his messages since english isn't his first language. His own words, then AI translated it. Sounded kinda clunky, but I could sense the human behind the words and I gave grace since I can only imagine how hard it is to properly communicate your ideas when English isn't your first language.

But now it's gotten to a point where I can tell he's not using it for just translation. I'm gonna need to chat with him.

Redacted example below

```Hey @PERSON_WHO_ASKED_QUESTION Both good, and the retrieval one isn't written down anywhere. In order.

One corpus or four. My lean is one. Same chunks and embeddings tables, source_type column to tell them apart.

Values I'd propose, flat rather than nested: paper, dataset_description, dataset_readme, dataset_contributors, dataset_records, dataset_files. Description and readme split because their units already differ, one row versus one row per paragraph. A discriminator that can't separate those isn't doing much. The existing 440 rows would need backfilling to paper.

Chunk id in the same spirit: dataset doi, source type, ord. So EXAMPLE_DOI.

Worth checking before any of this matters: does chunks.doi carry a foreign key to papers.doi in 0001_init.sql? If it does, a metadata chunk with a dataset doi can't go in that table at all, and separate storage stops being a choice. Ten second read, I haven't done it. Shout if you get there first.

Retrieval is the one that's bigger than it looks. Search once across everything and nothing guarantees a paper chunk and a metadata chunk both land in the top k. The facts we want relate the two, and the generator can only write those if it sees both sides in the same window. So if one type systematically wins the ranking, that class of fact doesn't get worse. It becomes impossible.

Which way it goes I don't know. Two mechanisms pull opposite ways. Metadata chunks are short, tens of tokens against roughly 450 for a paper chunk, so they may just lose. But we embed context header plus text, and on a short chunk the header is most of the vector. The header is the dataset name, which is also most of the query. That points the other way.

Cheaper to measure than argue. Load one dataset's metadata, run a normal dataset level query, record the rank of the first chunk of each source type. Runnable as soon as any one of our four subtasks lands.

After that it's one pool, per source with quotas, or one pool with a floor per source type. I'd rather not pick before there's a measurement. ```

AI;DR (AI; Didn't Read)