Project Charlemagne: Recovering a Self-Hosted AI Monitoring Environment

Project: Project Saṃsāra — Charlemagne
Category: Network Infrastructure | Linux Administration | AI Operations | Monitoring
Status: Active Development
Case Study: CHR-MAINT-001
October 2026

Project Overview

Charlemagne is my locally hosted AI assistant and an evolving component of Project Saṃsāra, my personal network and infrastructure engineering lab.

I developed Charlemagne to explore how artificial intelligence can assist with infrastructure monitoring, troubleshooting, documentation, and operational decision-making while remaining within an environment I control.

The project’s long-term vision extends beyond network operations. I want to create an AI assistant that can grow alongside my family, serving as an educational resource, storyteller, and security-conscious household assistant.

Its initial engineering responsibility, however, is much more focused: establishing a reliable foundation for monitoring and understanding my home infrastructure.

The Challenge

After an extended period without hands-on maintenance, I initiated a technical assessment to determine Charlemagne’s operational condition.

Although the server remained accessible, several questions required investigation:

• Was the operating system functioning as expected?
• Was the local AI service operational?
• Could the installed AI model successfully process requests?
• Were GPU acceleration and model inference working?
• Did the original infrastructure health-check scripts still exist?
• Was automated network monitoring still running?

Rather than rebuilding the system or immediately installing updates, I chose to establish an evidence-based baseline.

Phase 1 — System and Network Assessment

The assessment began with standard Linux diagnostic utilities.

Tools used: hostnamectl, uptime, ip addr, ip route, and systemctl.

The system reported:

Component	Observed result
Operating system	Ubuntu 26.04 LTS
Kernel	Linux 7.0.0-29-generic
Architecture	x86-64
System uptime	More than eight weeks
Network interface	Active
IPv4 configuration	DHCP-assigned /24 network
Default route	Present
SSH access	Successful

This confirmed the server’s reported operating environment and basic network configuration.

These results represent an initial assessment, not a complete security or infrastructure health certification.

Phase 2 — Validating Local AI Services

Charlemagne uses Ollama to host locally available language models.

During the investigation, I confirmed that:

• The Ollama service was active.
• Ollama version 0.24.0 was installed.
• Three model tags were available, including Charlemagne, Eshu, and Llama 3.1 8B.
• Model loading successfully utilized NVIDIA CUDA acceleration.
• All 33 reported model layers were offloaded to the GPU.

Troubleshooting an inference issue

An initial command-line inference test returned no visible response, despite the Ollama service reporting an active state.

I investigated the behavior by examining system journal entries and testing Ollama’s local HTTP API.

The logs showed successful model initialization and an HTTP response, while direct API testing confirmed that Charlemagne could generate meaningful text.

A subsequent measurement reported 37 generated tokens with approximately 0.58 seconds of token-generation time.

This distinction was important: model loading time, total request duration, and actual token-generation time are separate performance measurements.

Outcome: Local AI inference was verified. The earlier command-line behavior and cold-start performance remain subjects for additional investigation.

Phase 3 — Recovering the Original Monitoring Workflow

The next challenge was locating Charlemagne’s original infrastructure monitoring scripts.

Initial filesystem searches returned large numbers of unrelated Python dependencies. I narrowed the investigation through directory inspection and Linux service-discovery commands.

The original Project Saṃsāra directory was successfully located.

It contained three primary scripts:

samsara_healthcheck.sh

Performs network and service availability checks using ICMP and HTTP/HTTPS requests.

check_and_summarize.sh

Coordinates health-check execution, captures the results, and invokes the summarization process.

summarize_healthcheck.sh

Passes recorded health-check output to the locally hosted Charlemagne model for natural-language summarization.

The recovered implementation follows this general workflow:

Network checks → Log capture → Local AI analysis → Human-readable report

The monitoring script contains eight configured checks: two ICMP reachability tests and six HTTP/HTTPS endpoint tests.

Findings from the script review

The initial review identified several areas for improvement:

• Some configured monitoring endpoints may be outdated.
• HTTP validation does not adequately distinguish all application error responses.
• Certificate validation is bypassed for HTTPS checks.
• Logging currently emphasizes the latest result instead of structured historical records.
• Error handling could be improved.
• Automatic execution and alert delivery have not been verified.

Outcome: The original monitoring implementation was recovered and inspected, but its end-to-end operation has not yet been validated.

Phase 4 — Applying CCNA Networking Concepts

I use Project Charlemagne as a practical extension of my CCNA studies.

The maintenance assessment reinforced several networking concepts:

CCNA topic	Practical application
IPv4 addressing	Identifying interface addresses and CIDR prefixes
IP connectivity	Reading Linux routing tables and default routes
Network fundamentals	Understanding local versus routed communication
IP services	Distinguishing DNS and application availability from basic reachability
Network troubleshooting	Comparing ICMP results with HTTP service responses
Network security	Reviewing monitoring access, HTTPS validation, and network exposure
Automation	Investigating scripts, logging, and repeatable operational workflows

This approach connects classroom knowledge with actual infrastructure behavior.

Current Results

Verified:

• Ubuntu system accessible through SSH
• Network addressing and routing information collected
• Ollama service active
• Local AI inference functioning
• GPU-assisted model loading successful
• Original health-check and summarization scripts recovered

Still under investigation:

• End-to-end monitoring execution
• Accuracy of configured monitoring targets
• Reliability of AI-generated health summaries
• Historical logging and scheduled execution
• Alert notifications
• Operating system maintenance and pending updates

No infrastructure configuration changes were required during the initial discovery and validation process.

Next Steps

The next engineering phase will focus on validating monitoring targets, safely executing the recovered health checks, establishing structured logs, and improving error handling.

Longer-term objectives include:

• Reliable scheduled infrastructure checks
• Historical health reporting
• Prometheus and Grafana integration
• Local AI-assisted incident summaries
• Notifications requiring appropriate human review
• Secure integration with additional household systems

Lessons Learned

This project reinforced a principle I consider essential to infrastructure engineering:

A running service is not necessarily a healthy service, and an available system is not necessarily a functioning monitoring platform.

The investigation also demonstrated the value of structured troubleshooting, operational documentation, and validating assumptions before making changes.

Charlemagne is still evolving. Documenting that process is part of the project itself.

Project Saṃsāra — Building, learning, maintaining, and improving infrastructure through practical engineering.