Parallel Inference: Running Multiple Models Without the Overhead

Overview

Running a single language model call is straightforward. Running the right model for the right task, routing to a fallback when the primary is unavailable, enforcing per-task timeout budgets, extracting structured verdicts from unstructured text, classifying failures rather than discarding them, and making model uncertainty visible to the analyst — that is the real engineering problem.

Jarvis Nexus solves this with a task-based model routing system. Every task type in the security and trading stack has a primary model and a fallback. The routing is a deliberate human decision encoded in configuration — not a dynamic negotiation at runtime. The system's responsibility is to honour that routing reliably under load, under failure, and under the uncertainty that comes with working with probabilistic systems.

Features

Fifteen Task Routes with Primary and Fallback Models

ChimeraGuard and NetMon share fifteen task types, each mapped to a specific model pair. Fast triage tasks go to a compact, low-latency model. Alert analysis tasks go to the same compact model with a larger reasoning model as fallback. Deep analysis and reasoning tasks go to the largest available reasoning model with a mid-size fallback. Code inspection tasks go to a specialised code model. Embedding tasks go to a dedicated embedding model. Narrative generation, report writing, dossier generation, cross-alert correlation, signal correlation, mathematical analysis, RAG synthesis, and specialised network monitoring tasks each have their own primary and fallback assignment.

The routing principle is consistent: the primary model is chosen for quality or speed appropriate to the task. The fallback is always a smaller or more available model — never a larger one — because the fallback exists to maintain reliability, not to improve output quality.

Per-Task Timeout Budgets

A single global timeout across all task types is the wrong abstraction. Fast triage tasks have a tight timeout because speed matters more than exhaustive analysis. Alert analysis tasks have a much longer timeout because the output drives analyst decisions and quality matters. Each timeout is configured independently and is adjustable per deployment without requiring code changes.

Three Classified Failure Paths

Failures are not all equivalent. Three distinct failure paths are handled and labelled separately. A connection error means the model server is not running or not reachable. A timeout means the model did not respond within the task's budget. An empty response means the model responded but produced no usable text. Each of these produces a fallback response with a specific reason string and a clear label identifying it as a fallback. The analyst dashboard renders fallback responses differently from model-generated responses.

Structured Verdict Extraction

Models return unstructured prose. A structured verdict must be extracted from that prose reliably. The extraction scans the response for five threat levels in descending order — critical, high, medium, low, and benign — looking first for the exact assessment phrase and falling back to keyword presence in the full text. The ordered scan ensures that a response mentioning multiple levels resolves to the highest one consistently, rather than depending on where in the text the model chose to summarise its assessment.

The extracted threat level drives the action-required flag on the alert. Critical and high always require action. Benign and low do not. Medium is resolved contextually based on the rule that triggered the alert and whether the response text contains specific high-confidence threat indicators.

Multi-Source Concurrent Polling

Jarvis Nexus aggregates live data from all active systems simultaneously. iMonitor statistics, Trader369 trade history, ChimeraGuard mesh state, and HTTP health checks for all service endpoints run on independent async polling loops. Each loop has its own interval and its own failure handling. A failure in one polling loop does not stall or degrade the others. The aggregated state is broadcast over a WebSocket server to all connected dashboard clients on every update cycle.

Trading Intelligence Layer

A separate trading brain component reads Trader369's trade history periodically, passes the performance data to the local model for analysis, and writes structured learning observations to a persistent knowledge base. The knowledge base accumulates trade pattern insights — win rates by symbol, timing patterns, signal reliability under different market conditions — that inform future analysis without requiring the full history to be re-read on every cycle.

What It Resolved

The problem Jarvis Nexus was built to address was the gap between having multiple capable local models and having a reliable system that uses the right one for each job. Without task routing, model selection was ad hoc — sometimes the wrong model was used, sometimes a slow model added latency where a fast one was sufficient, and sometimes a model failure produced a silent gap in the alert output because there was no fallback path.

Task routing resolved the selection problem by making the model assignment explicit and reviewable. The configuration shows exactly which model handles which task, which model takes over on failure, and what the timeout budget is for each. Any of those decisions can be changed without touching the code that uses them.

The labelled fallback system resolved the silent failure problem. Previously, a model timeout could produce an alert with no AI content — which an analyst might interpret as a low-confidence finding or simply miss. The labelled fallback ensures the absence of AI analysis is always visible, not invisible.

Usage

Jarvis Nexus runs as a central server that security analysts and the trading system both connect to. The WebSocket dashboard shows the live status of all connected systems, recent alerts with their AI assessments, current model assignments for each task type, and whether any recent alerts used the fallback path. The trading brain runs in the background, producing periodic analysis of trade performance accessible through the same interface.

Benefits

• Right model for each task: fast triage tasks get low-latency models; deep analysis tasks get high-quality reasoning models — without manual intervention per alert.
• Resilience to model failure: every task has a fallback, and every fallback produces a labelled response rather than a silent gap.
• Transparent uncertainty: fallback labels and confidence-based action-required flags make the system's uncertainty visible to analysts.
• Configurable without code changes: timeout budgets, model assignments, and polling intervals are all configuration values, not hardcoded constants.
• Independent failure domains: concurrent polling loops fail independently, so a single unavailable service does not degrade the aggregator's view of all other services.
• Compounding trading knowledge: the persistent knowledge base means each trading analysis cycle builds on prior observations rather than starting from raw data every time.