You built the homelab. Proxmox is humming, Docker containers are stacked, Tailscale connects everything. Then something breaks at 2 AM, and you spend an hour SSH-ing into boxes trying to figure out which service died and why.

That's the gap between running infrastructure and actually observing it. Monitoring turns your homelab from a collection of machines into a system you can reason about. You see failed logins, disk pressure, container restarts, and network anomalies in one place instead of grepping through logs on six different hosts.

Splunk is the industry standard for log aggregation and analysis. The enterprise version costs more than most homelabs are worth, but Splunk Free gives you 500 MB of daily ingestion with no time limit. For a homelab — even a busy one — that's plenty. You get the same search engine, the same dashboarding, the same SPL query language that Fortune 500 SOC teams use. The only things you lose are alerting, authentication, and distributed search. We'll work around those.

This guide walks through the full setup: installing Splunk on Ubuntu, ingesting logs from Proxmox, Docker, and system services, building dashboards that surface what matters, and writing SPL queries that turn raw log data into answers.

Hardware requirements: Splunk Free runs comfortably on 2 CPU cores and 4 GB of RAM. A dedicated VM or container is ideal — don't run it on the same box as your production workloads. Budget 1 GB of disk per day of retention at typical homelab ingestion rates.

01 Installing Splunk Free on Ubuntu

Splunk distributes Linux packages directly. Skip the package manager — you want the official .deb from splunk.com so you control the version.

Download and install

Create a free Splunk account at splunk.com/download if you don't have one. Grab the latest .deb for Linux x86_64. At the time of writing, that's Splunk 9.3.x.

# Download (replace with current version URL from splunk.com)
wget -O splunk.deb 'https://download.splunk.com/products/splunk/releases/9.3.1/linux/splunk-9.3.1-0b8d769cb912-linux-amd64.deb'

# Install
sudo dpkg -i splunk.deb

# Accept license and set admin credentials
sudo /opt/splunk/bin/splunk start --accept-license --answer-yes --seed-passwd 'YourStrongPassword'

# Enable boot start
sudo /opt/splunk/bin/splunk enable boot-start -systemd-managed 1

Splunk is now running on port 8000. Open http://your-server-ip:8000 in a browser and log in with the admin credentials you just set.

Switch to the Free license

By default, Splunk starts with a 60-day Enterprise trial. Switch to the Free license immediately — you don't want features disappearing mid-setup when the trial expires.

# In the web UI:
# Settings → Licensing → Change license group → Free license → Save

# Or via CLI:
sudo /opt/splunk/bin/splunk edit licenser-groups Free -is_active 1
sudo /opt/splunk/bin/splunk restart

The Free license removes authentication (anyone with network access can reach the UI), scheduled searches, and alerting. For a homelab on a private network or behind Tailscale, the auth limitation doesn't matter. We'll address alerting later.

Lock down network access

Since Free license has no built-in auth, firewall it so only your local network or Tailscale subnet can reach the web UI and receiving ports:

# Allow only Tailscale subnet
sudo ufw allow from 100.64.0.0/10 to any port 8000 proto tcp
sudo ufw allow from 100.64.0.0/10 to any port 8089 proto tcp
sudo ufw allow from 100.64.0.0/10 to any port 9997 proto tcp

# Or restrict to your LAN
sudo ufw allow from 192.168.1.0/24 to any port 8000 proto tcp

02 Configuring Data Inputs

Splunk is useless without data. The three main ways to get logs in: monitor local files, receive syslog over the network, and deploy Universal Forwarders on remote hosts. For a homelab, we'll use all three.

Monitor local system logs

If Splunk runs on a host that also generates interesting logs (auth, syslog, kernel), monitor them directly:

# Add via CLI
sudo /opt/splunk/bin/splunk add monitor /var/log/syslog -index main -sourcetype syslog
sudo /opt/splunk/bin/splunk add monitor /var/log/auth.log -index main -sourcetype linux_secure

# Or create /opt/splunk/etc/system/local/inputs.conf:
[monitor:///var/log/syslog]
disabled = false
index = main
sourcetype = syslog

[monitor:///var/log/auth.log]
disabled = false
index = main
sourcetype = linux_secure

Receive syslog from network devices

Proxmox nodes, routers, and switches can forward syslog to Splunk over UDP or TCP. Enable a syslog listener:

# In /opt/splunk/etc/system/local/inputs.conf:
[udp://514]
disabled = false
connection_host = ip
sourcetype = syslog
index = main

[tcp://1514]
disabled = false
connection_host = ip
sourcetype = syslog
index = main

Use TCP port 1514 instead of UDP 514 when possible — TCP guarantees delivery and doesn't silently drop messages under load. UDP 514 is there for devices that only support the traditional syslog protocol.

Forward from Proxmox

On each Proxmox node, add a syslog forwarding rule. This sends all system logs to your Splunk instance:

# On the Proxmox node, create /etc/rsyslog.d/50-splunk.conf:
*.* @@splunk-host:1514

# Restart rsyslog
systemctl restart rsyslog

The @@ prefix means TCP. A single @ would mean UDP. Point splunk-host at the IP or Tailscale hostname of your Splunk box.

Ingest Docker container logs

Docker's default logging driver writes JSON to /var/lib/docker/containers/. Tell Splunk to monitor that path recursively:

# In inputs.conf:
[monitor:///var/lib/docker/containers/*/*-json.log]
disabled = false
index = docker
sourcetype = docker:json

Create the docker index first:

sudo /opt/splunk/bin/splunk add index docker

Each container's logs land with the container ID as the source. We'll parse out the container name in a later step.

Deploy Universal Forwarders

For remote hosts that don't support syslog forwarding (or where you want richer data), install Splunk Universal Forwarder. It's lightweight — around 100 MB of RAM — and sends logs to your Splunk instance over port 9997.

# On the remote host:
wget -O splunkforwarder.deb 'https://download.splunk.com/products/universalforwarder/releases/9.3.1/linux/splunkforwarder-9.3.1-0b8d769cb912-linux-amd64.deb'
sudo dpkg -i splunkforwarder.deb

# Configure and start
sudo /opt/splunkforwarder/bin/splunk start --accept-license --answer-yes --seed-passwd 'ForwarderPass'
sudo /opt/splunkforwarder/bin/splunk add forward-server splunk-host:9997
sudo /opt/splunkforwarder/bin/splunk add monitor /var/log/syslog -index main -sourcetype syslog
sudo /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1

On the Splunk server, enable the receiving port:

sudo /opt/splunk/bin/splunk enable listen 9997

Index strategy: At minimum, create separate indexes for main (system logs), docker (container logs), and network (firewall/router logs). Separate indexes let you set different retention policies and make searches faster by narrowing the scope.

03 Parsing and Field Extraction

Raw logs are searchable out of the box, but structured fields make everything faster. Splunk auto-extracts some fields (timestamp, host, source, sourcetype), but you'll want custom extractions for your specific log formats.

Docker container names

Docker JSON logs include the container ID in the file path but not the human-readable name. Add a transform to extract it:

# In /opt/splunk/etc/system/local/transforms.conf:
[docker_container_id]
REGEX = /var/lib/docker/containers/([a-f0-9]{12})
FORMAT = container_id::$1
DEST_KEY = MetaData:Host

For container names, use a scripted lookup or the Docker API. The simplest approach: run a cron job that dumps docker ps --format '{{.ID}},{{.Names}}' to a CSV that Splunk reads as a lookup table.

# Cron job on Docker host (every 5 minutes):
*/5 * * * * docker ps --format '{{.ID}},{{.Names}}' > /opt/splunk/etc/apps/search/lookups/docker_containers.csv

Proxmox task logs

Proxmox logs VM operations (start, stop, migrate, backup) to /var/log/pve/tasks/. These are structured enough to extract with regex:

# In /opt/splunk/etc/system/local/props.conf:
[pve_tasks]
EXTRACT-vmid = UPID:(?P<pve_node>[^:]+):(?P<pid>[^:]+):(?P<pstart>[^:]+):(?P<starttime>[^:]+):(?P<task_type>[^:]+):(?P<vmid>[^:]+):(?P<user>[^:]+)

SSH authentication events

Splunk's built-in linux_secure sourcetype handles most auth log parsing. Verify the extractions are working:

# Search for failed SSH logins with extracted fields:
index=main sourcetype=linux_secure "Failed password"
| stats count by src_ip, user

If src_ip and user aren't auto-extracted, add manual extractions in props.conf:

[linux_secure]
EXTRACT-ssh_failed = Failed password for (?:invalid user )?(?P<user>\S+) from (?P<src_ip>[\d.]+)

Want the complete infrastructure monitoring stack?

This post covers Splunk setup. The Infrastructure Guides Bundle includes full monitoring configurations for Proxmox, Docker, Tailscale, and backup systems — plus network diagrams, capacity planning templates, and tested configs you can deploy in an afternoon.

Get the Infrastructure Guides — $24 →

04 Building Dashboards

Dashboards turn a pile of logs into a monitoring system. The goal isn't to visualize everything — it's to surface the five or six signals that tell you whether your lab is healthy without reading a single log line.

Homelab health overview

Start with a single dashboard that answers the morning question: "Is everything running?" Here are the panels that matter:

Panel 1: Events over time

index=main OR index=docker
| timechart span=1h count by index

A sudden drop in event volume usually means something stopped logging — which means something stopped running. A spike means something is wrong and screaming about it.

Panel 2: Failed SSH attempts (last 24h)

index=main sourcetype=linux_secure "Failed password"
| stats count by src_ip
| sort -count
| head 20

Panel 3: Docker container errors

index=docker "error" OR "fatal" OR "panic"
| stats count by source
| sort -count

Panel 4: Disk usage by host

If you're collecting system metrics via the Splunk Add-on for Unix (free), you get disk data automatically:

index=os sourcetype=df
| dedup host, Filesystem
| table host, Filesystem, UsePct
| where UsePct > 80

Panel 5: Proxmox VM operations

index=main sourcetype=pve_tasks
| stats count by task_type, vmid
| sort -count

Creating the dashboard

In the Splunk web UI: Search & Reporting → Dashboards → Create New Dashboard. Use the "Dashboard Studio" editor for the new XML-free layout, or Classic Dashboards if you prefer the traditional panel grid.

For each panel: run the search, click "Save As" → "Dashboard Panel", and add it to your overview dashboard. Set the time range to "Last 24 hours" for most panels.

Auto-refresh

Set the dashboard to auto-refresh every 5 minutes. In Classic Dashboard XML, add:

<dashboard refresh="300">

Now you have a single page that tells you the state of your lab at a glance.

05 SPL Queries That Actually Help

SPL (Search Processing Language) is what makes Splunk worth learning. It's a pipe-based language — similar to Unix shell pipelines — where each command transforms the result set for the next one. Once you internalize five or six commands, you can answer almost any question about your infrastructure in real time.

Find which hosts are noisiest

index=* earliest=-24h
| stats count by host
| sort -count

If one host generates 80% of your events, it's either logging too aggressively or something is broken. Both are worth investigating.

Track sudo usage across all hosts

index=main sourcetype=linux_secure "sudo:"
| rex "sudo:\s+(?P<sudo_user>\S+)"
| rex "COMMAND=(?P<sudo_cmd>.+)"
| table _time, host, sudo_user, sudo_cmd
| sort -_time

Detect port scans

If you're ingesting firewall logs (UFW, iptables, or pfSense), find hosts hitting multiple ports in a short window:

index=network sourcetype=ufw_log action=blocked
| bin _time span=5m
| stats dc(DPT) as unique_ports by _time, SRC
| where unique_ports > 10
| sort -unique_ports

Container restart frequency

index=docker "container started" OR "container stopped"
| rex field=source "/var/lib/docker/containers/(?P<container_id>[a-f0-9]{12})"
| stats count by container_id
| where count > 5
| sort -count

A container that restarts repeatedly is crash-looping. This query surfaces it before you notice the service is flapping.

Tailscale connection events

If you forward Tailscale logs to Splunk, track when nodes connect and disconnect:

index=main sourcetype=syslog "tailscaled"
| search "peer connected" OR "peer disconnected"
| table _time, host, _raw

06 Alerting Without Enterprise

Splunk Free doesn't include scheduled searches or built-in alerting. That's the one limitation that stings. But you can work around it with a simple cron job and Splunk's REST API.

The approach

Write a shell script that runs a Splunk search via the CLI, checks the results, and sends a notification (email, webhook, Telegram, Discord) if something is wrong. Schedule it with cron.

#!/bin/bash
# /opt/splunk/scripts/alert-failed-ssh.sh

RESULT=$(/opt/splunk/bin/splunk search 'index=main sourcetype=linux_secure "Failed password" earliest=-15m | stats count' -maxout 1 -output rawdata 2>/dev/null | tail -1)

if [ "$RESULT" -gt 20 ] 2>/dev/null; then
    curl -s -X POST "https://api.telegram.org/bot$TELEGRAM_BOT_TOKEN/sendMessage" \
        -d chat_id="$TELEGRAM_CHAT_ID" \
        -d text="⚠️ SSH Alert: $RESULT failed login attempts in the last 15 minutes"
fi
# Cron: run every 15 minutes
*/15 * * * * /opt/splunk/scripts/alert-failed-ssh.sh

Other alerts worth setting up

It's not as elegant as Enterprise alerting with correlation searches and adaptive thresholds, but it works. For a homelab, a shell script that texts you when something is on fire is all you need.

07 Performance Tuning

Splunk on limited homelab hardware needs a few tweaks to stay responsive.

Control ingestion volume

The 500 MB daily limit sounds generous until you point Splunk at verbose Docker containers. Check your daily ingestion:

index=_internal source=*license_usage.log type=Usage
| stats sum(b) as bytes by idx
| eval MB=round(bytes/1024/1024,2)
| sort -MB

If you're burning through the limit, filter out noisy sources. In inputs.conf, use blacklists to skip health check logs and debug-level chatter:

[monitor:///var/lib/docker/containers/*/*-json.log]
blacklist = healthcheck|GET /health|GET /ready

Set retention policies

Don't store everything forever. Set per-index retention based on how much disk you can spare:

# In /opt/splunk/etc/system/local/indexes.conf:
[main]
frozenTimePeriodInSecs = 2592000
# 30 days

[docker]
frozenTimePeriodInSecs = 604800
# 7 days — container logs are high volume, low archival value

Memory and CPU limits

If Splunk is competing with other services on the same host, cap its resource usage in the systemd unit:

# /etc/systemd/system/Splunkd.service.d/override.conf
[Service]
MemoryMax=4G
CPUQuota=200%
sudo systemctl daemon-reload
sudo systemctl restart Splunkd

What We Didn't Cover

This gets you a working Splunk monitoring stack for your homelab. But a production-grade observability setup goes deeper. The full infrastructure bundle also covers:

Get the Complete Infrastructure Guides Bundle

Full monitoring configs for Proxmox, Docker, Tailscale, and backup systems. Network diagrams, capacity planning templates, and tested deployment scripts. Everything you need to go from bare metal to fully observed infrastructure.

Download the bundle — $24 →

Ongoing Maintenance

A monitoring system that nobody looks at is just a log sink. Build these habits:

Monitoring is the difference between running a homelab and understanding it. Splunk Free gives you the same analytical power that enterprise SOC teams rely on — without the enterprise price tag. Point it at your logs, build a dashboard, and stop guessing what's happening on your network.