Skip to content
BSKLab.ai
← All research
Research noteLocal & Private AI

Local & Private AI — Running a Full AI Operation on Private Infrastructure

BSKLab runs nine AI agents, its content pipeline and email triage on a privately networked DGX Spark with a local 27B model — the same public/private architecture we recommend to industrial customers.

Research question

Can a small company run its entire AI operation — agents, content, automation — on private local infrastructure without giving up capability?

The full stack: public web tier, orchestration on the VPS, and private inference on the DGX Spark behind Tailscale.
Traffic crosses controlled boundaries — the public internet never touches the private AI zone directly.
Specialized agents, one orchestrator, and a human gate on everything public.
The content path every agent draft travels — ending with a human merge.

01 / 04

Architecture overview

The system separates three zones with explicit boundaries:

1. **Public zone** — Cloudflare (DNS/edge/security) → bsklab.ai web stack. Internet-facing, nothing else. 2. **Automation zone** — Hostinger VPS running the OpenClaw orchestrator, cron jobs, GitHub integration, n8n, and email workflows. 3. **Private AI zone** — DGX Spark (128GB unified memory, GB10 Grace Blackwell) running vLLM + local models, reachable only over Tailscale. **The public internet is never directly connected to the private AI infrastructure.**

Internet → Cloudflare → bsklab.ai (public web) ↕ GitHub (source of truth: code + Markdown content) ↑ OpenClaw orchestrator (VPS, automation zone) ↕ Tailscale (encrypted overlay) DGX Spark → vLLM → Qwen3.8-27B-FP8 (private inference)

The local inference stack

**MTP speculative decoding (the interesting part).** Qwen3.8 ships a native Multi-Token Prediction head, so vLLM runs self-speculative decoding without a second draft model. With 3 speculative tokens, decode throughput improved by roughly **1.6×** on real agent chat workloads. `num_speculative_tokens=3` is the sweet spot — beyond that, acceptance rates fall and net throughput drops. We deliberately did **not** set `enforce_eager` (CUDA graphs stay on); the combination has been stable since 2026-08-29.

  • **vLLM** (nightly) serving `Qwen/Qwen3.8-27B-FP8` over an OpenAI-compatible API, reachable only over the private overlay
  • FP8 weights + FP8 KV cache, 262K context window, chunked prefill + prefix caching
  • Auto tool choice with `qwen3_coder` tool-call parser and `qwen3` reasoning parser

What "private" buys us in practice

  • **Full agent runtime on local compute.** All 9 agents (research, content, review, email triage) run their inference on the DGX over the private overlay — no prompt leaves the network.
  • **Customer-shaped architecture.** The same zone separation (public / automation / private AI / future industrial edge) is what we recommend to manufacturing customers; BSKLab's own operation is the reference deployment.
  • **Cost + latency profile.** 27B-class reasoning on 128GB unified memory, at electricity cost, with 262K context.

Diagrams

  • **Complete Hybrid AI Architecture** — public cloud, orchestration, and private AI lab in one view
  • **Public Cloud to Private AI** — the four security zones and how traffic crosses them
  • **AI Workforce & Orchestration** — how the 9 agents route through the orchestrator with the human publication gate

Limitations

  • Single-container local model: no hot local fallback if vLLM crashes
  • MTP acceptance rate not yet continuously monitored (measured once, pre/post)
  • ComfyUI and vLLM cannot comfortably coexist at full memory utilization on 128GB unified memory

Next steps

  • Continuous MTP acceptance-rate + tok/s monitoring with alerting
  • A second local inference endpoint as a crash fallback
  • RAG stack (embeddings + reranker) on the DGX for the Industrial Knowledge Engine lab
  • Mini PC / edge node as the first industrial-edge deployment behind Tailscale

Related labs

Appeared in