InvestigationLocal & Private AI
Local Inference for Private Industrial AI
What is actually workable on customer-controlled hardware today: model sizes, embedding and reranking stages, and where the practical limits appear.
Research question
Which parts of a retrieval-grounded industrial AI stack can run entirely on customer-controlled hardware without an unacceptable loss in usefulness?
Background
Confidentiality requirements are a first-order architectural constraint in manufacturing, not a preference. Engineering drawings, process parameters and quality data frequently cannot leave the environment.
Approach
- Ran embedding, reranking and generation stages on local GPU hardware in the lab.
- Compared answer usefulness on retrieval-grounded technical questions rather than open-ended generation.
- Recorded practical operating characteristics: model load, concurrency limits, response latency.
Observations
- Embedding and reranking are the easiest stages to keep local and gave the most benefit per unit of hardware.
- Retrieval-grounded answering over technical documents was usable with locally-hosted models; unconstrained reasoning was noticeably weaker.
- Concurrency, not single-question latency, is the constraint that decides hardware sizing.
Limitations
- Single hardware configuration in a lab environment, not a production plant network.
- Usefulness was judged by internal review against a small question set.
- Model landscape changes quickly; these observations are time-bound.
Next steps
- Test a hybrid split: local retrieval and reranking with controlled-API generation.
- Document a sizing guide once concurrency behaviour is measured properly.
Related labs
Appeared in