Code Review with RAG: Building MergeWise

MergeWise reviews pull requests automatically by combining RAG-based context retrieval with LLM reasoning and GitHub Checks. The interesting engineering problem wasn't "call an LLM on a diff" — it's that a diff alone is nearly meaningless without the surrounding codebase context, and fetching that context has to be fast and cheap enough to run on every PR.

Why retrieval before reasoning

A raw diff shows what changed but not why a change might be wrong — that requires knowing how the changed function is called elsewhere, what conventions the codebase follows, and whether similar code exists nearby. MergeWise indexes a repository's source with FAISS, embedding functions, classes and their surrounding context, then retrieves the most relevant chunks for each changed hunk before the LLM ever sees the diff.

Making review fast enough to run on every PR

• Incremental indexing — only re-embedding files that changed since the last index update, instead of re-indexing the whole repo on every push.
• Inline execution vs. queue — small diffs run synchronously for fast feedback; larger diffs are pushed to a Celery/Redis queue so a big PR doesn't block the check from returning promptly for everyone else.
• Structured logging and fallback mechanisms — if retrieval or the LLM call fails, MergeWise degrades to a lighter diff-only review rather than failing the check entirely, so a transient error doesn't block a merge.

Turning LLM output into a useful GitHub Check

The LLM's output is parsed into structured findings — file, line, severity, rationale — and posted as inline review comments through the GitHub Checks API, not a single wall-of-text comment. That structure is what makes it something a developer actually reads mid-review instead of skimming past, and it's the difference between "an AI commented on my PR" and a review tool that fits into an existing workflow.