Project Charlemagne: Recovering a Self-Hosted AI Monitoring Environment

Project: Project Saṃsāra — Charlemagne
Category: Network Infrastructure | Linux Administration | AI Operations | Monitoring
Status: Active Development
Case Study: CHR-MAINT-001
October 2026

Engineering Summary –

When I first started building Charlemagne, my goal wasn’t simply to have another AI assistant running on my network. I wanted to create something of my own, something I could understand, maintain, and eventually develop into a much larger part of Project Saṃsāra.

Like any project, though, life happens. Between work, family, and continuing my studies, Charlemagne spent quite some time running in the background without much attention from me.

Rather than jumping straight into updates or rebuilding everything, I decided to find out exactly what I had left running and whether it was still doing what I originally designed it to do.

Using Ubuntu’s diagnostic tools, I began checking system health, network connectivity, routing configurations, and Ollama’s local AI services. Along the way, I confirmed that Charlemagne could still generate responses using GPU acceleration and managed to recover the original scripts I had created for monitoring my home infrastructure.

Of course, finding those scripts doesn’t necessarily mean everything is working. There’s still testing to be done, outdated configurations to review, and a proper logging and maintenance routine to establish.

For me, this is more than just getting an old project running again. It’s another opportunity to put what I’m learning through my CCNA studies into practice while developing better habits in network troubleshooting, Linux administration, and infrastructure documentation.

The goal isn’t just to get Charlemagne working. It’s to understand how it works, why it works, and how I can continue making it better.

Project Overview

Charlemagne is my locally hosted AI assistant and an evolving component of Project Saṃsāra, my personal network and infrastructure engineering lab.

I developed Charlemagne to explore how artificial intelligence can assist with infrastructure monitoring, troubleshooting, documentation, and operational decision-making while remaining within an environment I control.

The project’s long-term vision extends beyond network operations. I want to create an AI assistant that can grow alongside my family, serving as an educational resource, storyteller, and security-conscious household assistant.

Its initial engineering responsibility, however, is much more focused: establishing a reliable foundation for monitoring and understanding my home infrastructure.

The Challenge

After an extended period without hands-on maintenance, I initiated a technical assessment to determine Charlemagne’s operational condition.

Although the server remained accessible, several questions required investigation:

Was the operating system functioning as expected?
Was the local AI service operational?
Could the installed AI model successfully process requests?
Were GPU acceleration and model inference working?
Did the original infrastructure health-check scripts still exist?
Was automated network monitoring still running?

Rather than rebuilding the system or immediately installing updates, I chose to establish an evidence-based baseline.

Phase 1 — System and Network Assessment

The assessment began with standard Linux diagnostic utilities.

Tools used: hostnamectl, uptime, ip addr, ip route, and systemctl.

The system reported:

Component	Observed result
Operating system	Ubuntu 26.04 LTS
Kernel	Linux 7.0.0-29-generic
Architecture	x86-64
System uptime	More than eight weeks
Network interface	Active
IPv4 configuration	DHCP-assigned /24 network
Default route	Present
SSH access	Successful

This confirmed the server’s reported operating environment and basic network configuration.

These results represent an initial assessment, not a complete security or infrastructure health certification.

Phase 2 — Validating Local AI Services

Charlemagne uses Ollama to host locally available language models.

During the investigation, I confirmed that:

The Ollama service was active.
Ollama version 0.24.0 was installed.
Three model tags were available, including Charlemagne, Eshu, and Llama 3.1 8B.
Model loading successfully utilized NVIDIA CUDA acceleration.
All 33 reported model layers were offloaded to the GPU.

Troubleshooting an inference issue

An initial command-line inference test returned no visible response, despite the Ollama service reporting an active state.

I investigated the behavior by examining system journal entries and testing Ollama’s local HTTP API.

The logs showed successful model initialization and an HTTP response, while direct API testing confirmed that Charlemagne could generate meaningful text.

A subsequent measurement reported 37 generated tokens with approximately 0.58 seconds of token-generation time.

This distinction was important: model loading time, total request duration, and actual token-generation time are separate performance measurements.

Outcome: Local AI inference was verified. The earlier command-line behavior and cold-start performance remain subjects for additional investigation.

Phase 3 — Recovering the Original Monitoring Workflow

The next challenge was locating Charlemagne’s original infrastructure monitoring scripts.

Initial filesystem searches returned large numbers of unrelated Python dependencies. I narrowed the investigation through directory inspection and Linux service-discovery commands.

The original Project Saṃsāra directory was successfully located.

It contained three primary scripts:

samsara_healthcheck.sh

Performs network and service availability checks using ICMP and HTTP/HTTPS requests.

check_and_summarize.sh

Coordinates health-check execution, captures the results, and invokes the summarization process.

summarize_healthcheck.sh

Passes recorded health-check output to the locally hosted Charlemagne model for natural-language summarization.

The recovered implementation follows this general workflow:

Network checks → Log capture → Local AI analysis → Human-readable report

The monitoring script contains eight configured checks: two ICMP reachability tests and six HTTP/HTTPS endpoint tests.

Findings from the script review

The initial review identified several areas for improvement:

Some configured monitoring endpoints may be outdated.
HTTP validation does not adequately distinguish all application error responses.
Certificate validation is bypassed for HTTPS checks.
Logging currently emphasizes the latest result instead of structured historical records.
Error handling could be improved.
Automatic execution and alert delivery have not been verified.

Outcome: The original monitoring implementation was recovered and inspected, but its end-to-end operation has not yet been validated.

Phase 4 — Applying CCNA Networking Concepts

I use Project Charlemagne as a practical extension of my CCNA studies.

The maintenance assessment reinforced several networking concepts:

CCNA topic	Practical application
IPv4 addressing	Identifying interface addresses and CIDR prefixes
IP connectivity	Reading Linux routing tables and default routes
Network fundamentals	Understanding local versus routed communication
IP services	Distinguishing DNS and application availability from basic reachability
Network troubleshooting	Comparing ICMP results with HTTP service responses
Network security	Reviewing monitoring access, HTTPS validation, and network exposure
Automation	Investigating scripts, logging, and repeatable operational workflows

This approach connects classroom knowledge with actual infrastructure behavior.

Current Results

Verified:

Ubuntu system accessible through SSH
Network addressing and routing information collected
Ollama service active
Local AI inference functioning
GPU-assisted model loading successful
Original health-check and summarization scripts recovered

Still under investigation:

End-to-end monitoring execution
Accuracy of configured monitoring targets
Reliability of AI-generated health summaries
Historical logging and scheduled execution
Alert notifications
Operating system maintenance and pending updates

No infrastructure configuration changes were required during the initial discovery and validation process.

Next Steps

The next engineering phase will focus on validating monitoring targets, safely executing the recovered health checks, establishing structured logs, and improving error handling.

Longer-term objectives include:

Reliable scheduled infrastructure checks
Historical health reporting
Prometheus and Grafana integration
Local AI-assisted incident summaries
Notifications requiring appropriate human review
Secure integration with additional household systems

Lessons Learned

This project reinforced a principle I consider essential to infrastructure engineering:

A running service is not necessarily a healthy service, and an available system is not necessarily a functioning monitoring platform.

The investigation also demonstrated the value of structured troubleshooting, operational documentation, and validating assumptions before making changes.

Charlemagne is still evolving. Documenting that process is part of the project itself.

Project Saṃsāra — Building, learning, maintaining, and improving infrastructure through practical engineering.