auditd: linux's built in who did what audit log

the question every incident starts with

something weird happened on your server. a file changed, a login you don't recognize, a process you didn't start. now what. if you don't have logs that answer "who did this and when," you're stuck guessing. that's the entire reason auditd exists. it's the linux kernel's own audit trail, built in, free, and sitting unused on most machines because nobody turned it on.

most people find out about auditd the hard way, after an incident, when they're desperately checking bash history that the attacker already cleared. the fix is boring but it works: turn it on before you need it.

what auditd actually is

auditd is a userspace daemon that talks to the linux kernel's audit subsystem. the kernel can watch specific files, directories, and syscalls, and every time one of those things happens, it writes a structured log entry: what happened, which process did it, which user, and the exact timestamp. it's not a replacement for your normal logs, it's a much more precise layer underneath them.

the tool you use to set the rules is auditctl. the tool you use to search the logs it creates is ausearch. let's break down the actual commands.

watching a sensitive file

auditctl -w /etc/passwd -p wa -k identity

-w /etc/passwd tells auditd to watch that specific file. -p wa sets which actions trigger a log entry, in this case writes and attribute changes, so anyone editing the file or changing its permissions gets logged. -k identity tags the rule with a label so you can search for it later without digging through raw logs. you could swap the path for anything sensitive: /etc/shadow, /etc/sudoers, an ssh authorized_keys file, a config file for your web app. anything that would be bad news if it silently changed is a candidate.

logging every command execution

auditctl -a always,exit -F arch=b64 -S execve -k exec

this one is broader. -a always,exit means append this rule and log it on syscall exit. -F arch=b64 filters to 64 bit processes, since syscall numbers differ between architectures. -S execve is the important part, execve is the syscall used every time a program actually runs something. so this rule logs every command execution on the system, by every user, whether they're on a terminal, a cron job, or a reverse shell someone dropped. -k exec tags it for easy searching, same idea as before.

this is the closest thing linux has to a built in "who ran what, when" ledger, and it works even if someone clears their bash history, because it's not relying on the shell to keep track, it's the kernel itself watching.

pulling the logs back out

ausearch -k identity -i

this searches the audit log for anything tagged with the identity key, and -i makes the output human readable by translating uids into usernames and resolving other numeric fields into plain text. run the same thing with -k exec and you get a timeline of every command executed on the box. this is what you hand to yourself, or a colleague, when you're trying to reconstruct what happened during a suspected incident.

making it stick

rules set with auditctl on the command line disappear on reboot. if you want them permanent, drop them into a file in /etc/audit/rules.d/, for example audit.rules, using the same syntax minus the auditctl command name, then reload with augenrules --load or restart the auditd service. this is the step people skip and then wonder why their watch rules vanished after a routine reboot.

the takeaway

auditd isn't glamorous and it won't stop an attacker on its own. what it does is remove the guesswork after something goes wrong. watch your sensitive files, log execve calls, tag everything with clear keys, and make the rules survive a reboot. do this on your own servers, your homelab, anything you're responsible for, before you need it, because the one thing auditd genuinely cannot do is audit the past. set it up now and the past starts auditing itself.