February 17, 202610 min read

GDPR AI Chatbot: On-Premise LLM with DGX Spark

Your chatbot is giving wrong answers -- and your customers notice. That was exactly the situation our client faced: HubSpot's built-in chatbot delivered inaccurate, sometimes contradictory responses from their knowledge base. For a company handling daily customer inquiries, this was a serious problem.

The solution? A GDPR-compliant AI chatbot running entirely on-premise -- on an NVIDIA DGX Spark. No cloud. No data transfers. Full control. In this article, we show how we built it and why this architecture is a game changer for European enterprises.


Contents

  • The Problem: Why HubSpot's Chatbot Falls Short

  • Why GDPR Compliance Matters for AI Chatbots

  • The Solution: On-Premise LLM on NVIDIA DGX Spark

  • RAG Architecture: How the Chatbot Accesses Your Knowledge Base

  • Step by Step: How We Built the Chatbot

  • Results and Lessons Learned

  • Is an On-Premise AI Chatbot Right for Your Business?


The Problem: Why HubSpot's Chatbot Falls Short

HubSpot offers a built-in chatbot that can access your knowledge base. In theory, this sounds perfect. In practice, our client encountered three critical issues:

  1. Inaccurate answers: The chatbot often failed to find the right information or combined content from multiple articles into a wrong answer.

  2. No control over the AI model: The underlying language model is operated by HubSpot (or their AI partner). Which model runs, how it was trained, and where data is processed -- the client has no influence.

  3. GDPR concerns: Customer data and internal knowledge base content are transmitted to external servers. For a European company handling sensitive customer data, this is an unacceptable risk.

The client needed a solution that delivers more accurate answers while keeping all data in-house.

Why GDPR Compliance Matters for AI Chatbots

The GDPR (General Data Protection Regulation) imposes strict rules on processing personal data. With AI chatbots, this gets particularly tricky:

  • Customer data in queries: Users often enter personal information -- names, contract numbers, email addresses. This data must not flow uncontrolled to third parties.

  • Knowledge base content: Internal documents, process descriptions, and customer information in the knowledge base can be sensitive.

  • Cloud-based LLMs: With services like ChatGPT or other cloud AI solutions, queries are sent to servers outside the EU -- often to the US. Even with data processing agreements, residual risk remains.

The safest solution: The language model runs locally on your own hardware. No data leaves your corporate network. Period.

That is exactly the approach we chose: An on-premise LLM (Large Language Model) running entirely on the client's infrastructure.

The Solution: On-Premise LLM on NVIDIA DGX Spark

Running a powerful language model locally requires appropriate hardware. Our client chose the NVIDIA DGX Spark -- a compact AI system designed specifically for enterprise use.

[BILD: DGX Spark Unboxing]

Why the DGX Spark?

  • Compact form factor: The DGX Spark is a desktop device -- no server room required. It sits right in our client's office.

  • Sufficient performance: With the integrated NVIDIA Grace Blackwell architecture and 128 GB unified memory, the DGX Spark can run models with up to 200 billion parameters.

  • Enterprise-ready: NVIDIA provides a complete software stack (NVIDIA AI Enterprise) that simplifies LLM operations.

  • Energy-efficient: Compared to a full DGX server rack, the Spark uses a fraction of the energy -- ideal for SMBs.

[BILD: DGX Spark Setup]

On the DGX Spark, we installed an open-source language model optimized for the client's language requirements. The model runs completely locally -- no internet connection needed for inference, no data sent externally.

RAG Architecture: How the Chatbot Accesses Your Knowledge Base

A language model alone is not enough. It needs access to your company's specific content -- in this case, the HubSpot knowledge base. For this, we use a RAG architecture (Retrieval Augmented Generation).

What Is RAG?

RAG combines two components:

  1. Retrieval: The system searches a knowledge base for relevant information matching the user's query.

  2. Generation: The language model formulates a precise answer based on the retrieved information.

The key advantage: The LLM doesn't make things up. It answers exclusively based on your own content. This dramatically reduces so-called hallucinations -- convincing-sounding but incorrect answers.

Our Architecture in Detail

  1. HubSpot API Data Export: We regularly download all articles from the HubSpot knowledge base via the HubSpot API. This runs automatically via cron job.

  2. Chunking & Embedding: Articles are split into meaningful sections (chunks) and converted into numerical vectors (embeddings) that represent the text's meaning.

  3. Vector Database: The embeddings are stored in a local vector database. We use a proven open-source solution that also runs on the DGX Spark.

  4. Chatbot Interface: When a user asks a question, the vector database is searched first for the most relevant content. This content is then passed along with the question to the LLM, which generates an informed answer.

Important: The entire process -- from user query to vector database search to LLM response -- runs on the DGX Spark. No cloud. No external API calls. Full GDPR compliance.

Step by Step: How We Built the Chatbot

For IT leaders and executives planning a similar project, here is our approach:

Phase 1: Analysis & Concept (1 Week)

  • Assessment of the existing HubSpot knowledge base (quality, scope, structure)

  • Definition of requirements: answer quality, response time, languages

  • Selection of the appropriate LLM model and infrastructure

  • GDPR assessment and data protection concept

Phase 2: Hardware Setup (2-3 Days)

  • Installation and configuration of the NVIDIA DGX Spark

  • OS installation and NVIDIA AI Enterprise stack setup

  • Network integration (local network, no internet access for the device)

Phase 3: Data Pipeline & RAG Setup (1-2 Weeks)

  • Connection to HubSpot API for automated data export

  • Implementation of the chunking and embedding pipeline

  • Setting up the vector database

  • Integration of the LLM with the RAG architecture

Phase 4: Testing & Optimization (1 Week)

  • Test runs with real customer queries

  • Fine-tuning retrieval parameters (chunk size, top-K results)

  • Optimization of prompt strategies for better answer quality

  • Performance tests under load

Phase 5: Go-Live & Monitoring

  • Integration into the existing website / customer portal

  • Monitoring dashboard for answer quality and usage statistics

  • Regular knowledge base synchronization

Total duration: approximately 4-6 weeks from concept to go-live.

Results and Lessons Learned

After going live, we observed the following improvements:

  • Significantly higher answer quality: Through the RAG architecture, the chatbot targets the right knowledge articles. The number of wrong or irrelevant answers decreased substantially.

  • GDPR by design: No customer data leaves the corporate network. The data protection officer approved the solution.

  • Fast response times: The DGX Spark delivers answers in seconds -- comparable to cloud-based solutions.

  • Independence: The client is no longer dependent on HubSpot's AI model and can customize the system at any time.

Lessons Learned

  1. Data quality is crucial: Chatbot answer quality depends directly on the quality of knowledge articles. Poorly structured or outdated articles lead to poor answers -- regardless of the AI model.

  2. Chunk size affects precision: Chunks that are too large dilute answers; chunks that are too small lose context. The optimal size must be tested individually for each knowledge base.

  3. Plan for regular updates: Synchronization between HubSpot and the vector database must run automatically -- ideally multiple times per day.

  4. On-premise does not equal complicated: Thanks to modern hardware like the DGX Spark and mature open-source software, running a local LLM is significantly easier than many expect.

Is an On-Premise AI Chatbot Right for Your Business?

An on-premise enterprise AI chatbot is the right solution if:

  • You process sensitive customer data and need to comply with GDPR

  • You want to better leverage an existing knowledge base (HubSpot, Confluence, Zendesk, etc.)

  • You value independence from cloud providers

  • You want to maintain full control over the AI model and data processing

The investment in hardware like the NVIDIA DGX Spark pays for itself quickly -- especially considering that cloud-based LLM APIs become expensive at high usage volumes.


Conclusion

A GDPR-compliant AI chatbot doesn't have to be a compromise. With the right hardware (NVIDIA DGX Spark), a well-designed RAG architecture, and a clean data pipeline, you get a chatbot that answers more accurately than cloud alternatives -- while keeping your data in-house.

The technology is mature. The hardware is available. And the implementation is less complex than many expect.

Want to find out if an on-premise AI chatbot makes sense for your business? We'd be happy to advise you -- no strings attached. Contact us for a free consultation at hypescale.com/contact.