With the aim of matching review requests with reviewers, we are exploring several approaches. The goal is to have a system that can create matches between requests for peer review and scientists willing to review preprints.
Matching can occur in both directions:
For each scientist in question, we have a set of DOIs that they have either published or reviewed.
For each DOI (be that a paper looking for review, a paper reviewed by a scientist, or a paper published by a scientist) we have a title and an abstract.
Both directions of matching happen in the same embedding space: every DOI's title and abstract is a point, and a scientist is represented by the cluster of points formed by their published/reviewed papers. The region such a cluster covers can be read as the scientist's area of expertise.
To find reviewers for a paper, the paper is a single point in the space, and we look for the scientists whose clusters lie nearest to that point:
To find papers for a scientist, the scientist's cluster defines the area of expertise, and we look for papers seeking review whose points fall in or near that area:
For all DOIs we create an embedding vector using a suitable embeddings model. These are stored in a vector database. When we receive a request in the form of one or more DOIs, we use vector search to find DOIs or scientists (represented by a collection of DOI vectors associated with the scientist).
Building on the embedding-based search, we use an LLM to identify matches. After retrieving related DOIs or scientists with vector search, the respective titles and abstracts are templated into a prompt to an LLM. The prompt is an instruction to select suitable matches based on the titles and abstracts. The retrieval during the first step augments the context for the LLM. The output of the LLM runs through a deterministic validation, e.g. to ensure the proposed DOIs actually exist.
Instead of passing the titles and abstracts of DOIs/scientists identified via vector search, we only pass the input DOIs with their titles and abstracts in the prompt. The prompt includes instructions on how to make tool calls to use vector search, but also e.g. a full-text search. This approach is closer to agentic loops and RLM.
