About Linux IO drain

Found this at the linux-ha archives and loved it.
Perfect description of what Admins see and scheduler guys rarely understand.

Oh, and I should just comment: Big-memory Linux systems seem to get
into trouble with _large_ amounts of buffered writes. Even an
mke2fs on a big partition can show this. It seems that vast
amounts of memory are eaten by the write buffers so the system
starves for memory, and at the same time there is no disk cache
because it has been eaten by the write buffers, so every command
or library read has to go to disk (causing lots more I/O), but of
course the disk queue is full of writes, so reads are slow …
you end up with everything that does disk I/O showing up in state
‘D’ in a process listing.


3 thoughts on “About Linux IO drain

  1. Sad but true. Hate D-stated processes. I even got md_sync hung in D state some weeks ago. Just terrible.
    Any good resources for tuning this?

    • I hold a training on this about once a year πŸ™‚
      Good ressouces… none the way they should be.

      a) IBM has an article in developerworks about setting the buffer cache flush rate.
      Making the write buffers be flushed 1-2 times per second is the key thing on busy high mem servers, since you cannot limit it per se
      b) Memory-wise ensure that you don’t allow 64GB of shared memory on a 64GB box, or you’ll be paging out your kernel. (i.e. oracle shops that can’t think and tune at the same time). In general, calculate ram for the OS
      c) ionice
      d) cgroups, while not up to i.e. AIX standards, it’s still a poor mans ressource scheduler and setting up ressource limits for applications + OS and separating those, will help a lot.

      e) this is two steps:
      marry rich daughter of someone owning a company with huge IT budget.
      migrate to HP-UX. never see state D again.

  2. Thanks for reply, I’ll try to find more information about flush rate and limiting shared mem. However, I can not find this cgroups for lenny debian 2.6.26

    HP-UX – no way, I’m a poor guy πŸ™‚

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s