大网数据广告活动
服务器运维问答

Kernel reports writeback errors on a RAID array

· 52CE 我爱测

I have a RAID-6 array md0 for which I have recently started getting bursts of writeback errors in kernel log, like this: [Thu Jun 9 11:56:49 2022] md0: writeback error on inode 1412769769, offset 3035136, sector 285462910760 [Thu…

问题原文

原帖链接 · Anton Tykhyy · CC BY-SA 4.0

I have a RAID-6 array md0 for which I have recently started getting bursts of writeback errors in kernel log, like this:

[Thu Jun  9 11:56:49 2022] md0: writeback error on inode 1412769769, offset 3035136, sector 285462910760
[Thu Jun  9 12:47:17 2022] md0: writeback error on inode 1412769771, offset 0, sector 285491991552
[Thu Jun  9 13:20:45 2022] md0: writeback error on inode 224524727190, offset 0, sector 288770741760
[Thu Jun  9 13:21:32 2022] md0: writeback error on inode 224524727190, offset 2510848, sector 288770744104
[Thu Jun  9 14:03:18 2022] md0: writeback error on inode 224507073535, offset 5963776, sector 285428442496

The sector locations are not quite random: they are concentrated around sector 286000000000. I haven't been able to correlate these bursts with any sort of activity such as high CPU load or high disk utilization. The counts of UDMA CRC errors in drives' SMART logs aren't incrementing when these errors appear. I did a full RAID check (echo check > /sys/block/md0/md/sync_action) with the array unmounted, and ran extended SMART tests on all drives in the array, and these have turned up no errors, reallocated/pending sectors or suspicious SMART values. What should I try next to diagnose this behavior?

$ mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Sat Xxx xx xx:xx:xx xxxx
        Raid Level : raid6
        Array Size : 164062560256 (156462.25 GiB 168000.06 GB)
     Used Dev Size : 11718754304 (11175.88 GiB 12000.00 GB)
      Raid Devices : 16
     Total Devices : 16
       Persistence : Superblock is persistent

     Intent Bitmap : /var/raid/bitmap

       Update Time : Thu Jun  9 15:02:53 2022
             State : clean
    Active Devices : 16
   Working Devices : 16
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : centos7:0
              UUID : xxxxxxxx:xxxxxxxx:xxxxxxxx:xxxxxxxx
            Events : 427699

    Number   Major   Minor   RaidDevice State
      18       8      176        0      active sync   /dev/sdl
... 14 more lines like this ...
      14      65       16       15      active sync   /dev/sdr

Update: apparently there is something wrong either with the array or with the md driver (I have only a 5.6.6 kernel), because I noticed that bits in the array's intent bitmap get stuck after writes to problematic locations.

已采纳解答

原帖链接 · Anton Tykhyy · CC BY-SA 4.0

Looking back over my logs and records, the problem was almost certainly caused by md bad block entries having been erroneously cloned to new disks when I was extending the array. These entries messed up the affected area despite there being nothing wrong with it, and as the filesystem on the array filled up and XFS started using the area, writeback errors appeared seemingly out of nowhere. I listed all bad blocks in the array, cleared them and disabled badblocks logic, and ran raid6check to detect and repair any mismatches:

(for i in {a..p}; do mdadm --examine-badblocks /dev/sd$i; done)>badblocks.txt
# edit the resulting file a bit
mdadm --stop /dev/md0
mdadm --assemble /dev/md0 --uuid xxxxxxxx:xxxxxxxx:xxxxxxxx:xxxxxxxx \
      --update=force-no-bbl --bitmap /var/raid/bitmap
while read s _
do
  p=$(($s/1024-115)) # 115 is the array's data offset in chunks
  echo ============$p===============
  raid6check /dev/md0 $(($p-1)) 3 autorepair
done <badblocks.txt

This fixed the problem.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2022-06-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。