live response on linux, capture before you reboot

hook

you just noticed something weird on a linux box. maybe cpu is pegged, maybe outbound traffic looks off, maybe a teammate says "hey does this process look right to you." your gut says reboot it and move on. don't. a reboot wipes ram, kills active connections, and clears a bunch of the exact evidence you need to figure out what happened. live response means you capture the volatile stuff first, while the machine is still running, then you shut it down and dig deeper offline. here's the walkthrough, command by command.

ps, spot the odd process

start with a full process listing. you're looking for anything that doesn't belong: a process running from /tmp, something with a random string for a name, a parent process that makes no sense (like a web server spawning a shell).

ps auxf

the f gives you the process tree so you can see parent-child relationships, which is huge for spotting a compromised service that's now running commands it shouldn't. write down pids of anything suspicious, you'll need them for the next steps.

ss, find the beacon or miner calling out

malware has to talk to someone eventually, whether it's a c2 server checking in or a cryptominer phoning home to a pool. ss shows you active network connections tied to processes.

ss -tnp

look for connections to weird ports, unfamiliar ips, or a process you don't recognize holding a socket open. cross-reference the pid here with what you found in ps. if a process has no business making outbound connections and it's doing it anyway, that's your beacon.

lsof +l1, deleted but running binaries

this one's the classic trick attackers use to hide. they run malware, then delete the file from disk. the process keeps running from memory, but there's nothing left on disk for you to find with a normal file search. it's like committing a crime and then burning down the building, except the getaway car is still parked outside running.

lsof +L1

this flag lists open files that have a link count of less than one, meaning the file is deleted but still held open by a running process. if you see a binary in that list, you've found something actively hiding from disk-based detection. grab the pid, and if you can, dump the process memory before you touch anything else, because that memory is the only copy of that binary left.

last, recent logins

next, check who's been logging in and from where.

last -a

look for logins at odd hours, from ips that aren't your normal admin range, or accounts that shouldn't be logging in interactively at all (service accounts, for example). this file lives in /var/log/wtmp and it's one of the first things attackers try to clear, so if it looks suspiciously empty or short, that's a finding on its own.

cron and systemd, hunt persistence

if an attacker got in, they usually don't want to do it twice. they'll plant something that survives a reboot. check both cron and systemd, because modern intrusions use either.

crontab -l
cat /etc/cron.d/*
systemctl list-timers --all
systemctl list-unit-files | grep enabled

you're hunting for jobs or services you didn't create, especially ones pointing to scripts in /tmp, /dev/shm, or hidden directories. a service with an innocuous name that runs a curl command on a timer is a very common pattern worth knowing.

the takeaway

capture first, analyze second. that order matters because volatile evidence, meaning anything living in ram or in an active connection, disappears the moment you reboot. run through ps, ss, lsof +L1, last, and your cron/systemd checks in that order, save the output somewhere off the host (a usb drive, a remote log server, anything that isn't the compromised machine itself), and only then start remediation. for your own systems, build this into a checklist now, before you're panicking during a real incident, and practice it on a lab box so the commands are muscle memory when it counts.