Buffer I/O Errors

Article ID: 139

Last updated: 13 Feb, 2009

If you receive the following messages listed in the example below, it could be related to a RAID Issue. Most of these errors could be found in dmesg or dmesg.2 or sent to you by email notification

If you receive these errors, Please check your RAID Controller and your RAID array.

Before removing a disk drive please verify that you are removing the correct failed disk drive, as removing the incorrect disk drive may break your RAID.

While the RAID array is running in an unstable or degraded condition, unexpected results can occur.

It can lead to filesystem errors and data loss.

It is best practice to fix this issue as soon as possible.

After you fix your RAID:

Please click on the remove button, then let the system run for a bit and

download and send us your system log files for us to check to see if there are any other issues.

Logs are located here

STATUS -> HARDWARE -> Logs.

The “Logical block” or “sectors” in most instances in the examples below will refer to an issue with one or more of the disk drives.

The following are a few examples of error messages that were caused by either a bad disk drive or a problem with the RAID controller.

Example 1

Buffer I/O error on device dm-6, logical block 235528
lost page write due to I/O error on dm-6
sd 1:0:0:0: rejecting I/O to offline device

Example 2 (failed disk drive)

end_request: I/O error, dev sdf, sector 7007993
Buffer I/O error on device dm-3, logical block 3464313
ata6: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
ata6: status=0x51 { DriveReady SeekComplete Error }
ata6: error=0x40 { UncorrectableError }
ata6: translated ATA stat/err 0x51/40 to SCSI SK/ASC/ASCQ 0x3/11/04
ata6: status=0x51 { DriveReady SeekComplete Error }
sd 5:0:0:0: SCSI error: return code = 0x8000002
sdf: Current: sense key=0x3
ASC=0x11 ASCQ=0x4

Dec 26 14:35:27 nas kernel: end_request: I/O error, dev sdf, sector 7007983

Example 3

Buffer I/O error on device dm-4, logical block 61520

lost page write due to I/O error on dm-4

sd 1:2:0:0: rejecting I/O to offline device

lost page write due to I/O error on dm-4

This example as it turned out was a bad RAID controller.

arcmsr0 abort device command of scsi id=0 lun=0
SCSI error : <8 0 0 0> return code = 0x6000000
end_request: I/O error, dev sda, sector 2427432
Buffer I/O error on device dm-2, logical block 33045
arcmsr0 abort device command of scsi id=0 lun=0
arcmsr0 scsi id=0 lun=0 abort ccb '0xf629fbe0' outstanding command
arcmsr0 abort device command of scsi id=0 lun=0
SCSI error : <8 0 0 0> return code = 0x6000000

In this instance the drive error resulted in filesystem errors

sd 1:0:0:0: rejecting I/O to offline device

printk: 14 messages suppressed.

Buffer I/O error on device dm-7, logical block 98312

lost page write due to I/O error on dm-7

sd 1:0:0:0: rejecting I/O to offline device

Buffer I/O error on device dm-7, logical block 98308

lost page write due to I/O error on dm-7

Critical errors:

2008/11/09 07:47:40|Call Trace:

2008/11/09 07:47:40|[<ffffffff80230de4>] warn_on_slowpath+0x64/0xe0

2008/11/09 07:47:40|[<ffffffff80245e30>] wake_bit_function+0x0/0x30

2008/11/09 07:47:40|[<ffffffff80245e70>] bit_waitqueue+0x10/0xd0

2008/11/09 07:47:40|[<ffffffff80245f88>] wake_up_bit+0x18/0x40

2008/11/09 07:47:40|[<ffffffff80231d6d>] printk+0x8d/0xa0

2008/11/09 07:47:40|[<ffffffff802ecbd8>] ext3_getblk+0xa8/0x1a0

2008/11/09 07:47:40|[<ffffffff802b4c2f>] mark_buffer_dirty+0xaf/0xc0

2008/11/09 07:47:40|[<ffffffff802f2747>] ext3_commit_super+0x57/0xa0

2008/11/09 07:47:40|[<ffffffff802f2bc4>] ext3_handle_error+0x54/0xd0

2008/11/09 07:47:40|[<ffffffff802f2c83>] __ext3_std_error+0x43/0x70

2008/11/09 07:47:40|[<ffffffff802eab3e>] ext3_reserve_inode_write+0x4e/0xb0

2008/11/09 07:47:40|[<ffffffff802c3380>] compat_filldir64+0x0/0xf0

2008/11/09 07:47:40|[<ffffffff802eabc1>] ext3_mark_inode_dirty+0x21/0x70

2008/11/09 07:47:40|[<ffffffff802f9da4>] journal_start+0xa4/0xe0

2008/11/09 07:47:40|[<ffffffff802ecef0>] ext3_dirty_inode+0x90/0xb0