If my VPS got hacked, would I notice? How I built a homemade HIDS with AIDE, Telegram alerts, and encrypted backups

A few days ago I wrapped up a security audit of my personal VPS, where this blog and a handful of other projects run. The score was 7.5 out of 10: lots of layers in place (firewall limited to Cloudflare IPs, WAF with custom rules, Authenticated Origin Pull in Traefik, CrowdSec analyzing logs, admin panel behind Cloudflare Access). But there was one gap that bothered me more than the rest.

The gap was easy to describe and fairly hard to close: if an attacker got root access to the server (through the npm supply chain, a CVE in Dokploy, or a compromised OAuth session), I had no way to know. CrowdSec reads HTTP logs, but once an attacker is already inside, they don't need to go through Traefik. The firewall protects the entry point, not what happens once someone is already in. My backups cover content (the blog's SQLite, Docker volumes), not the system state.

I needed something that answered a very specific question. "Has anyone touched a binary, a systemd unit, a script in /etc, an SSH key, without me knowing?" That's exactly the question a HIDS answers.

What a HIDS is and why it made sense for me

HIDS stands for Host-based Intrusion Detection System. The idea is old and not very glamorous: you take an initial snapshot of the filesystem state (permissions, owners, hashes of every file in the paths that matter), and then periodically compare the current state against that snapshot. Any difference you didn't explicitly allow yourself is a sign of tampering.

The nice thing about this approach is that you don't need to detect specific techniques (rootkits, malware, persistence). You detect changes. If the attacker replaces /usr/bin/sshd with a backdoored version, the hash changes. If they add a key to /root/.ssh/authorized_keys, the hash changes. If they drop in a systemd unit for persistence, there it is.

It's also important to know what it doesn't detect: nothing that happens only in RAM (like process injection without touching disk), nothing the attacker removes before the next check, and nothing in paths you're not watching. It's one more layer, not a silver bullet.

There are several tools that do this: AIDE (Advanced Intrusion Detection Environment), Tripwire, Samhain, OSSEC. I went with AIDE for three reasons:

• It's in the official Debian repos. apt install aide aide-common and done. No curl | bash, no compiling anything.

• Flat, readable config. It's a text file where you declare which paths to watch, what to exclude, and which attributes to compare.

• No daemon, nothing to keep running. It's just a binary that reads and compares. One run every 24 h is enough.

Configuring AIDE, the balance between signal and noise

The aide-common package on Debian ships with a huge default configuration, split into snippets under /etc/aide/aide.conf.d/. The idea is that the system auto-generates a consolidated config based on the installed packages. In practice, that config watches too much: directories with caches, paths that change on every apt upgrade, counters in /var. The noise gets so high that you end up ignoring the alerts.

I replaced that config with my own, much shorter one. The idea was to start from scratch and only watch things where an unexpected change is a sign of tampering:

# Database paths
database_in=file:/var/lib/aide/aide.db
database_out=file:/var/lib/aide/aide.db.new
database_new=file:/var/lib/aide/aide.db.new
gzip_dbout=yes

# Reporting
report_summarize_changes=yes
report_grouped=yes
report_ignore_e2fsattrs=VNIE

# Atributos comparados: tipo, perms, owner, group, inode, num links,
# size, mtime, ctime, sha256, ACL, xattrs, selinux
FullCheck = p+u+g+i+n+s+m+c+sha256+acl+xattrs+selinux

# Paths vigilados
/etc                       FullCheck
/usr/bin                   FullCheck
/usr/sbin                  FullCheck
/usr/local/bin             FullCheck
/usr/local/sbin            FullCheck
/usr/local/lib             FullCheck
/usr/lib/systemd/system    FullCheck
/root                      FullCheck

# Exclusiones (regex POSIX extendida)
!/etc/resolv\.conf$
!/etc/mtab$
!/etc/adjtime$
!/root/\.bash_history$
!/root/\.cache
# ...y un puñado más para paths que el orquestador regenera

The real config is about 50 lines long. It covers what matters (system binaries, system and application configs, systemd units, SSH keys, and crontabs under /root) and excludes what doesn't. If the first aide --check after initializing the baseline already shows differences, that's a sign your config still includes volatile paths that should've been excluded. Iterating until you get a clean check is part of the job.

Initializing the baseline:

sudo aide -c /etc/aide/aide.conf --init
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db

In my case it ended up being a database with 3614 entries, about 138 KB after compression. Running aide --check against that baseline takes 3 or 4 seconds (a lot less than the 1-2 minutes I expected). More than reasonable for a daily timer.

Turning the check into an actionable alert

AIDE prints the diff to stdout. Useful if someone reads it, useless if nobody looks. I needed the result to land somewhere I'd see it without going looking for it. Telegram is the channel I already use for all my infra alerts (CF token, certs close to expiring, disk full) because it has three concrete advantages:

• The bot has its own API, no weird libraries needed. A curl is enough.

• It doesn't depend on Cloudflare or on my own infra. If the thing I'm alerting about is my infra, it doesn't alert me through my own infra.

• I control delivery priority (silent vs with sound) from the script.

I have an infra-telegram.sh helper shared by all the host monitoring scripts. It reads credentials from a 0600 root file (never hardcoded) and exposes a simple alert function:

_INFRA_TELEGRAM_ENV="/etc/myinfra/telegram.env"

alert() {
    local title=$1 body=$2 priority=${3:-default}

    [ -r "$_INFRA_TELEGRAM_ENV" ] || return 0
    . "$_INFRA_TELEGRAM_ENV"
    [ -z "${TELEGRAM_BOT_TOKEN:-}" ] && return 0

    local emoji silent="false"
    case "$priority" in
        urgent) emoji="🚨" ;;
        high)   emoji="⚠️" ;;
        *)      emoji="ℹ️"; silent="true" ;;
    esac

    curl -sS -X POST --max-time 10 \
        --data-urlencode "chat_id=${TELEGRAM_CHAT_ID}" \
        --data-urlencode "text=${emoji} ${title}\n\n${body}" \
        --data "disable_notification=${silent}" \
        "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/sendMessage" \
        >/dev/null 2>&1
}

On top of that helper I built an AIDE-specific wrapper. It runs aide --check, parses the Summary: block from the output, and decides what to do based on the exit code:

• rc 0 (no diff): complete silence. I don't want a daily "everything's fine" notification.

• rc 1-13 (added/removed/changed): a high alert with a diff snippet and the command hint to review it.

• rc ≥ 14 (actual AIDE error): an urgent alert. This means the HIDS is broken, not that there's tampering, which is worse.

One parsing detail took me a while to find. AIDE prints Added entries: twice: once in the Summary block with the number, and again as the header of the detail section without a number. A naive awk grabs the number from the first one and then overwrites it with an empty string when it hits the second. The fix was to gate the match with a block flag:

counts=$(echo "$out" | awk '
    /^Summary:/        { in_summary=1; next }
    /^---/             { in_summary=0 }
    in_summary && /Added entries:/   { gsub(/[^0-9]/,"",$NF); a=$NF }
    in_summary && /Removed entries:/ { gsub(/[^0-9]/,"",$NF); r=$NF }
    in_summary && /Changed entries:/ { gsub(/[^0-9]/,"",$NF); c=$NF }
    END { printf "%s %s %s", a+0, r+0, c+0 }
')

The script does not run aide --update automatically. Any diff requires human review. After a legitimate apt upgrade, I manually accept the changes with:

sudo aide -c /etc/aide/aide.conf --update
sudo mv -f /var/lib/aide/aide.db.new /var/lib/aide/aide.db

One thing to keep in mind: aide --update returns a non-zero exit code when there is a diff, just like --check. If you chain it with &&, the mv never runs because update always finds changes (that's the whole point of running it). You need to separate them with ; or run them as separate steps.

Oneshot service + daily timer:

# /etc/systemd/system/check-aide.service
[Unit]
Description=AIDE filesystem integrity check

[Service]
Type=oneshot
ExecStart=/usr/local/bin/check-aide.sh
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

# /etc/systemd/system/check-aide.timer
[Unit]
Description=Daily AIDE filesystem integrity check

[Timer]
OnCalendar=*-*-* HH:MM:00  # ajusta a tu hora preferida
RandomizedDelaySec=10min
Persistent=true

[Install]
WantedBy=timers.target

RandomizedDelaySec keeps all the timers on my VPS from firing at exactly the same second. Persistent=true makes it run at boot if it missed the scheduled window (because of a reboot, for example).

The new problem, AIDE can be the first target

A well-set-up HIDS has one awkward property: its own database is the attacker's first target. If they get root, they can just run aide --update after installing their backdoor so the new "reality" gets absorbed into the baseline. The next comparison won't detect anything unusual.

There are two natural defenses against this:

1. Immutable backups outside the monitored host itself. If the attacker can modify aide.db, they shouldn't be able to modify the copies.

2. Periodic verification that the old backups are still there, intact, and comparable with the current baseline.

The nice part is that both defenses can be built with standard SSH and encryption primitives, no extra services or licenses.

Backups the monitored server can't tamper with

The backup destination is a second VPS I already owned for another project. The important question wasn't "how do I copy the file" (that's just scp), but how do I make sure an attacker with root on the monitored server can't tamper with the copies. The answer ended up being four layers.

Layer 1: SSH key with forced command

I generated a dedicated ed25519 key exclusively for this task (I don't reuse keys for other things). On the receiving side, instead of adding the public key as-is to authorized_keys, I added restrictions. This is plain OpenSSH, no extra software:

from="<ip-del-server-monitorizado>",restrict,command="/home/usuario/bin/aide-backup-receive.sh" ssh-ed25519 AAAA... aide-backup@origen

What that line does is this:

• from="<ip>" makes sshd reject the connection before it even asks for auth if the source IP doesn't match. That's not application-layer defense, it's network-layer defense.

• restrict is a full clamp-down, no PTY, no port forwarding, no agent forwarding, no X11, no user-rc, and it blocks any future option added to the protocol.

• command="..." is the forced command. No matter what the client asks for when it runs ssh, sshd will execute only that command. It doesn't matter if the client tries ssh ... "ls /etc", ssh ... "rm -rf ~", or an interactive shell, the only thing that runs is the specified script. Whatever the client sends as arguments ends up in the SSH_ORIGINAL_COMMAND variable (which my script deliberately ignores).

Testing it is reassuring. From the source server:

ssh -i /root/.ssh/aide_backup_key -p <port> usuario@destino "id; whoami; ls /etc"
# stdout: ERROR: archivo recibido demasiado pequeño (0 bytes)

The client tried three commands. sshd ignored all three and executed the forced command, which read empty stdin and rejected the operation. Not a single line of output from id or ls.

Layer 2: the receiver script decides the file name

The forced command is a script that reads stdin and stores the contents in a file. The subtle point is who gets to decide the file name:

#!/bin/bash
set -euo pipefail

DEST_DIR="$HOME/aide-backups"
DATE=$(date -u +%Y%m%d)                     # <-- nombre = fecha del SERVIDOR
TARGET="$DEST_DIR/aide-${DATE}.db.gz.age"
TMP=$(mktemp -p "$DEST_DIR" "aide-${DATE}.XXXX")

trap 'rm -f "$TMP"' EXIT

# Tope duro de 10 MB
head -c 10485760 > "$TMP"

# Validaciones: tamaño mínimo y magic bytes gzip (1f 8b)
size=$(stat -c%s "$TMP")
[ "$size" -lt 1024 ] && { echo "ERROR: demasiado pequeño" >&2; exit 1; }

magic=$(od -An -N2 -tx1 "$TMP" | tr -d ' \n')
[ "$magic" != "1f8b" ] && { echo "ERROR: no es gzip" >&2; exit 1; }

# (cifrado + audit log van aquí, los explico abajo)

mv -f "$TMP" "$TARGET"
sha256sum "$TARGET" | awk '{print $1}'      # eco del sha para verificación

The client does not control the file name. If the source script tried to upload aide-20251225.db.gz (overwrite an older backup), the destination would still store it as aide-<hoy>.db.gz. That means a compromised source server can at most overwrite today's backup. Copies from previous days stay immutable from its point of view.

The script also validates that what it received looks at least somewhat like a legitimate backup (more than 1 KB, starts with gzip magic bytes), caps the maximum size at 10 MB to prevent disk-fill attacks, and returns the SHA-256 of what it received on stdout. The client compares that SHA with the local file before sending the Telegram confirmation. If they differ, it sends an immediate urgent alert.

Layer 3: at-rest encryption with age

Up to this point, the backups are protected against modification, but not against reading. If the backup server gets compromised, the attacker gets the full history of AIDE databases: filesystem structure, hashes, paths, lots of internal mapping. Useful if they're planning a better-informed next attack.

The solution I used is age, a modern, open source, very simple encryption tool. It generates a keypair in one command:

age-keygen -o ~/.../aide_backup.key
# Output: clave privada en el fichero, pública impresa por stdout
# Public key: age1...

The essential rule is that the private key never touches either server. It lives only on my personal machine, with a copy in my password manager. The public key does go into the receiver script on the backup server, hardcoded as the recipient:

AGE_RECIPIENT="age1..."  # solo la pública

# en la función de recepción:
age -r "$AGE_RECIPIENT" -o "$TMP_AGE" "$TMP_PLAIN"
mv -f "$TMP_AGE" "$TARGET"

The backup server can encrypt (the public key is enough) but it can't decrypt what it encrypted (that needs the private key). If the server gets compromised, the attacker gets pure ciphertext, binary blobs with no value unless they also get the private key that's stored somewhere else.

One detail: encryption preserves the integrity detected by SHA-256 if you compare it correctly. The receiver script still returns the SHA of the plain file (before encryption) so the client can verify it. What changes is what's stored on disk: ciphertext, not plain.

To verify manually from my Mac, I download the ciphertext and decrypt it on the fly without touching disk:

remote_hash=$(ssh backup-server "cat $latest" | age -d -i ~/.../aide_backup.key | shasum -a 256 | awk '{print $1}')
local_hash=$(ssh source-server 'sudo sha256sum /var/lib/aide/aide.db | awk "{print \$1}"')
[ "$local_hash" = "$remote_hash" ] && echo "MATCH" || echo "MISMATCH"

Layer 4: audit log

A cheap little detail that's very useful. The receiver script appends one line for every push attempt, legitimate or not, to ~/aide-backups/.access.log:

2026-04-28T12:00:21Z ip=<origen> outcome=ok           size=142450 sha256=db603b01b30f...
2026-04-28T12:15:19Z ip=<origen> outcome=reject_size  size=0      sha256=none
2026-04-28T12:15:20Z ip=<origen> outcome=reject_size  size=12     sha256=none

This gives you visibility into odd things: pushes outside the schedule, unusual frequency, different IPs. The outcome=reject_* entries are especially interesting. They mean something tried to use the SSH key in a way it shouldn't. In the screenshot above you can see the rejections from my negative tests (empty payloads and 12-byte ones), all correctly blocked before anything was touched.

Threat model

It's worth being explicit about what this setup covers and what it doesn't. The backup is only as safe as the assumptions behind it.

Result

The final system has a few concrete properties that leave me pretty comfortable:

• Maximum detection window for filesystem tampering: 24 h, aligned with the daily check.

• Check runtime: 3-4 seconds.

• Expected false positives: one every time I run apt upgrade and forget to accept the baseline. Acceptable.

• Daily traffic to the backup server: ~140 KB.

• Backup storage: ~12 MB for 90 days of retention.

• Estimated monthly maintenance cost: 5 minutes, mostly to accept legitimate diffs after updates.

• External dependencies: none. It's all standard tools (AIDE from the Debian repos, SSH, age, systemd, curl).

The most interesting part for me wasn't really AIDE itself (it's old and well known), but thinking about the backup channel as a security component. A properly locked-down SSH key with forced command, plus a file name chosen by the server, plus encryption with the private key kept somewhere else gives you immutability and confidentiality guarantees without needing anything exotic. Just primitives that have been around since the 90s.

Operational lessons I wrote down so I don't repeat them:

• The double match in AIDE output with awk (Added entries: appears twice, once with a number and once as a header). Gate it with a block flag.

• aide --update returns a non-zero diff bitfield, so don't chain the baseline mv with &&.

• Creating a file inside a watched path also triggers detection on the parent (its mtime changes). That's not noise, it's an extra valid signal.

• If your Telegram helper silences curl errors with >/dev/null 2>&1, a failed alert is invisible. When something doesn't show up, replay it manually with the response visible so you can see what Telegram said.

Natural next step: add auditd or something based on eBPF to close the gap around RAM-only tampering without disk changes. That's a different kind of beast and I'm not in a rush, but it's on the list.