Self-hosted LLMs and a RAG over card specifications

How it works

The models run on an NVIDIA DGX Spark that I set up. vLLM serves a 27B Qwen model in NVFP4, a 4-bit number format, alongside an embedding model and a reranker. An embedding model turns text into lists of numbers, so passages with a similar meaning can be found. A reranker then puts the best of those matches first. All three sit behind a LiteLLM gateway, so every service has one address and one API for every model.

I moved the agent services onto the gateway. They no longer carry a fixed model name. Instead they ask the gateway what it serves and pick the chat model themselves. That took one more step than it sounds. The gateway’s plain model list returns bare names, and a bare name can’t tell a chat model from an embedding model or a reranker. So the lookup asks the gateway’s model-info endpoint first, which reports what each model is for. Against a bare vLLM server with no gateway in front, it falls back to the plain list. A name pinned by hand still works. It is checked against everything the server serves, not only the chat models, because a pin is a deliberate choice and a narrower check would reject names the server really does serve. I covered the lookup with tests. The agent services behind the card management assistant run on this stack.

The RAG is a separate system. I built it on a customised RAGFlow over 54 card specifications: 17 from EMV, 4 from GlobalPlatform, 9 for Mastercard M/Chip and 24 from Visa, plus Visa profile XMLs and TakaPay material. EMV is the common standard for chip cards, GlobalPlatform covers how applications are managed on the chip, and the Mastercard and Visa documents describe each network’s own chip applications. Card personalization is the step where a chip card gets its applications and data, and the rules for it are spread across documents like these. Our card personalization engineers use the RAG as their daily reference. They ask a question in plain words and get an answer drawn from the specifications.

I also wrote about running RAGFlow on an Apple M5 Mac.

Key decisions

Host the models ourselves

The agents work with live bank data: products, cards and transactions. I kept the models on our own hardware, so bank data never goes to an outside model API.

One gateway in front of every model

Services reach the models only through the gateway, never through vLLM directly. That way a model can be swapped, renamed or added at the gateway without redeploying a single agent. It matters because the served model does change. A service with a stale model name pinned in its config fails every request until someone edits that config. With the lookup, a service finds the current chat model on its own.

The same gateway also serves the embedding model and the reranker. That is exactly why the lookup was needed: once one address serves three kinds of model, a service has to ask which one is for chat.

Index the real specifications

The engineers’ questions are about exact rules in EMV, GlobalPlatform, Mastercard M/Chip and Visa documents. So the RAG answers from the documents themselves, not from what a model happens to remember. I indexed the specifications the team actually works from, so an answer comes from the same text an engineer would otherwise look up by hand.

Results

I ran the load tests on the DGX Spark myself. A token is roughly a word or part of a word.

• Throughput: about 205 tokens a second across 16 parallel streams, that is, 16 answers being written at the same time.
• Reliability: at a steady 1.5 requests a second, 453 of 453 requests succeeded.
• Prefill: reading the prompt before answering peaked at about 2,410 tokens a second.
• Prefix cache: a hit rate of about 54%. When requests start with the same text, such as a shared system prompt, the server can reuse work it has already done instead of repeating it.

The agent services now run on this stack through the gateway, and the RAG over the card specifications is the card personalization engineers’ daily reference.