Found this at the linux-ha archives and loved it.
Perfect description of what Admins see and scheduler guys rarely understand.
Oh, and I should just comment: Big-memory Linux systems seem to get
into trouble with _large_ amounts of buffered writes. Even an
mke2fs on a big partition can show this. It seems that vast
amounts of memory are eaten by the write buffers so the system
starves for memory, and at the same time there is no disk cache
because it has been eaten by the write buffers, so every command
or library read has to go to disk (causing lots more I/O), but of
course the disk queue is full of writes, so reads are slow …
you end up with everything that does disk I/O showing up in state
‘D’ in a process listing.