During incident handling, time is precious. I try to make myself take notes and communicate about the incident over logged channels like e-mail and IM - to the point that I think some team members have rules specifically for e-mail from me.

The detailed timelines that can be reconstructed from these notes are crucial when preparing post-mortem documents!

I have a simple philosophy: Fill what’s empty. Empty what’s full. Scratch where it itches Script the process wherever possible.

— Alice Roosevelt Longworth (and Grant Stavely)

Initially, I thought bash history could help.

# Bash settings
export HISTTIMEFORMAT="%s "
export HISTCONTROL=ignoredups
export HISTFILESIZE="9999999"

But what about the command output? There is a better way:

Firstly, and this is something that took getting used to, set up a fancy shell prompt to provide the log with context and time stamping:

--(grant@sensor)-(1/pts/1)-(17:32:33-UTC/28-Apr-08)--
--($:/nsm/)-

Gross! A two-line shell prompt! Trust me, it’s useful.

Then just add to your shell start-up:

# start script to log everything now!
exec /usr/bin/script -f /nsm/var/handlerlog/$USER.shell_log.` date +'%Y-%m-%d:%H:%M:%S' `.$$

Script will then log everything that prints to the console with the prompt providing context.