Generalist LLMs hit their limits: lessons from building a Liferay-expert AI agent

At Beorn, a large share of our projects revolves around Liferay: new platforms, application maintenance, and above all version upgrades — from old 6.2 instances to the current quarterly DXP releases. Our developers use Claude Code and Codex daily. The conclusion came quickly: on Liferay, generalist LLMs are not merely imprecise. They are dangerous.

This note explains how we built a Liferay-expert agent to fix that, exposed via MCP inside our teams’ working environment. Above all, it tells the journey: the starting assumptions, the ones that didn’t hold, and the trade-offs that led to the final architecture.

Why generalist LLMs fail here

The problem isn’t specific to Liferay, but Liferay is a textbook case. Between 6.2 and 7.4, the platform’s architecture was overhauled several times:

The right answer to “how do I override this behavior?” depends entirely on the client’s version. It is versioned knowledge: the correct answer in 7.4 is a mistake in 6.2, and vice versa.

Yet a generalist LLM is trained on a corpus that flattens precisely this dimension. Fifteen years of Stack Overflow, forums and documentation end up compressed into the same weights, with no temporal axis. The model then produces the statistically average answer — a mix of APIs from several eras, plausible but nonexistent method signatures, JSP hooks proposed for versions that moved to OSGi long ago. Every answer is fluent, confident, and unverifiable without an expert alongside.

For a developer, that’s wasted time. For an architect steering a version upgrade, who needs to know precisely what broke between two versions, why, and how to migrate, it’s disqualifying.

The project’s starting point comes down to one observation: this versioned knowledge exists, and it is public. It’s in the git history of liferay-portal, in the commit messages of core developers, in BREAKING_CHANGES.md. No one had structured it to make it usable by a model. That’s what we set out to do.

Mining twenty years of git history

We built a five-phase Python pipeline over three repositories: liferay-portal, liferay-plugins (6.x history) and liferay-blade-samples.

Phase 1 extracts qualifying commits by two criteria: architectural-decision keywords in the message (deprecat, refactor, migrat, breaking, LPS- ticket references) and a list of target authors — Liferay’s historic core developers, weighted by priority. Each commit is attached to its architectural era via version tags: this is what preserves the versioned dimension that generalist LLMs have lost.

Phase 2 extracts the diffs of these commits, filtered by extension (.java, .md, .bnd, .gradle) and capped at 500 lines. Phase 3 parses BREAKING_CHANGES.md into structured records: what changed, who is affected, how to update, why — exactly the format a migration architect needs.

Phase 4 cross-references all of it into three types of training pairs. The richest type associates a documented breaking change with the actual diff of the corresponding commit, found via the JIRA ticket: you get the decision, its rationale and its concrete implementation all at once.

Raw output: 454,760 pairs, 1.6 GB of JSONL. We thought we had our dataset. What we mostly had was a problem we hadn’t yet seen.

Discovering 87% redundancy

While manually auditing a sample of analyzed pairs — a practice we recommend before any training run — a pattern emerged: the same topics kept coming back, again and again, under barely different wording. Tracing it to the source, the explanation was simple: a single JIRA ticket can generate dozens of commits. Our record, LPS-139131, has 81. Each commit produced a training pair, but the topic and analysis were nearly identical.

Measured on the analyzed sample: 87% redundancy. Fine-tuning on that would have produced a model that over-learns a handful of topics and ignores the rest — the opposite of the goal.

Deduplication became the project’s central undertaking. First question: which similarity metric? The natural approach — textual comparison of contents — proved impractical: at O(n×m) per pair, it doesn’t scale to 454k records and 1.6 GB. We kept it only for the small file of already-analyzed pairs (~700 records). For the large volume, we used Jaccard distance over the sets of files touched by each commit: two commits that modify the same files are considered redundant, two commits touching different files add diversity. It looks like a crude heuristic, but it’s O(1) per pair and captures the reality of the repository well.

The final mechanism works in three layers. Pairs with a ticket are grouped by JIRA ticket; within each group, we keep an anchor (the record with the richest context) plus diversity picks selected greedily — a record is kept only if its maximum similarity with the already-kept records stays below 0.60, up to 3 per ticket. Pairs without a ticket held another surprise: 95% of them were worthless auto-generated commits (SOURCEFORMATTER, REVERT, REGENERATE…). A prefix filter removes them before the same processing, grouped this time by main file.

Result: 134,747 pairs (-70%), 616 MB, ~146M tokens. It’s this dataset, not the raw volume, that made the quality of everything that follows.

Enriching 134k pairs without spending a month on it

A raw diff is not instruction-tuning data. A final phase sends each pair to the Gemini API to extract structured metadata: relevance score, difficulty, architectural pattern (controlled vocabulary: osgi_migration, service_builder_refactor, api_deprecation…), and above all an instruction/answer pair — the question a Liferay developer would actually ask, and a self-sufficient answer with concrete steps.

The first calculation was painful: at 15.7 seconds per pair sequentially, processing 134k pairs would take 24 days. We rewrote the phase asynchronously — a worker semaphore to control API-call concurrency, a lock to serialize JSONL writes — bringing processing down to ~1.3 days with 20 workers.

A multi-day run will be interrupted; that’s not a hypothesis, it’s an operational fact. So we built a checkpoint/resume mechanism keyed by ID: rerunning the same command picks up exactly where execution stopped. In hindsight, it’s one of the most profitable engineering choices of the project.

Fine-tuning: why QLoRA, why a 7B

Three approaches were on the table: RAG alone (near-zero GPU cost, but no internalization of patterns), LoRA/QLoRA fine-tuning (specializing the style, vocabulary and patterns of the domain), and continued pre-training (massive knowledge injection, high GPU cost). We ruled out CPT for a first milestone: the cost/benefit doesn’t justify it until supervised fine-tuning has shown its limits. RAG alone was ruled out too, for a reason confirmed later: consulting a knowledge base does not give the model the language of the domain.

First-run configuration:

The choice of a 7B is not a default compromise. Our analysis of the token distribution showed a median of 505 tokens and a P95 of 4,093 — almost exactly the attention window of this generation’s 7B models. Only ~5% of records need truncation. A 13B or 34B would have multiplied training time by 3 to 8 without the nature of the data justifying it. QLoRA 4-bit, for its part, fits the training on a single GPU, where a full fine-tune would have required a cluster.

Same logic for sequence packing: with a median of 505 tokens for a max_seq_len of 4096, most of each batch would be padding. Packing takes a 7B run on an A100 from 3-4 days down to 1-1.5 days. It’s the single highest-yield parameter in the whole project.

On the serving side, we resisted the temptation to over-provision. A team of 10 developers only generates 2 to 4 concurrent requests in normal use, 5 to 8 at peak. A quantized 7B on a plain L4 (~1,500 tok/s via vLLM) covers the need with margin — a serving A100 would use less than 20% of its VRAM.

The moment of truth: a model that speaks right but invents

The first test of the fine-tuned model alone was instructive, both ways.

The positive first: the model had unmistakably learned. It answered in the language of a Liferay architect — the OSGi vocabulary, the ServiceBuilder patterns, the habit of reasoning by architectural era. Its hallucination rate was markedly lower than the base model’s: learning had worked.

But on such a lightweight model, that rate remained too high for our use. The model spoke like an expert, with an expert’s confidence, while still inventing API details. That’s the worst possible profile for an assistance tool: answers all the more convincing because they’re well phrased. And a structural problem remained, which no fine-tuning solves: a model does not cite its sources. When an architect recommends a migration strategy to a client, “the model said so” is not an argument.

The conclusion imposed itself: fine-tuning and RAG are not two competing options, they are two answers to two different problems. Fine-tuning gives the model the language and reflexes of the domain. RAG anchors it in the reality of the code — exact API signatures, dated breaking changes, JIRA tickets — and makes every answer verifiable. The mined corpus serves both: training data on one side, a vectorized base on the other.

RAG brings a third benefit fine-tuning cannot offer: currency. Recurring synchronization and vectorization scripts reindex the corpus as Liferay releases ship. The model’s weights date from its training; its knowledge base, however, follows the quarterly releases.

Exposing it where developers work: MCP

The question of distribution remained. We considered injecting corpus excerpts directly into the agents’ context (via project instruction files), but the approach doesn’t hold: the corpus is too large, a static selection of excerpts can’t anticipate the question, and every corpus update would force us to touch every project. An external MCP server flips the problem: it’s the agent that queries the knowledge when it needs it, with the exact query, and the server evolves independently of the client projects. Both Claude Code and Codex speak MCP natively; the developer-side integration takes a few lines of .mcp.json.

The server exposes two tools, corresponding to two distinct uses:

• liferay_search returns raw context — official documentation, exact API signatures, structured breaking changes with JIRA tickets — filterable by version. It’s the tool the coding agent calls before writing Liferay, to reason on real APIs itself.
• liferay_ask queries the fine-tuned model, with RAG grounding and cited sources, for expertise questions: implementation choices by version, migration strategy, justification of a pattern.

The version filter is not an ergonomic detail: it’s the query-side materialization of the era segmentation built in phase 1 of the mining. When a developer asks Claude Code to override a behavior on a client project in 7.2, the agent queries the MCP with the matching filter and retrieves the relevant breaking change — dated, with its JIRA ticket and the official migration procedure. For version upgrades, the architect systematically sweeps the breaking changes between two eras to build the migration plan, with verifiable references to put in the architecture document.

What we take away

Three lessons from this project. The data is the real work: the fine-tuning itself takes a day of GPU; the mining and above all the deduplication represented most of the effort, and that’s what makes the quality of the result. Fine-tuning and RAG address different problems: the first gives the model the language of the domain — our tests confirmed it — the second gives answers their anchoring in reality and their traceability; it’s the combination of the two, not either one, that earns the teams’ trust. MCP is the right distribution channel: no new tool to adopt, the expertise arrives in the environment where developers already work.

Today, this work concerns Liferay. But the reasoning goes beyond this case: every large enterprise platform carries, in its git history, its documentation and its changelogs, an architectural expertise that generalist LLMs cannot restore — precisely because it is versioned, contextual and diluted in their training corpora. The value is not in the model: it’s in the ability to turn this technical heritage into expertise directly usable by engineering teams. Liferay is our first experimentation ground. The approach itself is transferable.