大网数据广告活动
服务器运维问答

AWS ElasticBeanstalk docker-thin-pool getting full and causing re-mount of filesystem as read-only?

· 52CE 我爱测

I can't figure out how AWS sets up their Docker 'thin pool' on ElasticBeanstalk and how it is getting filled. My docker thin pool is filling up somehow and causing my apps to crash when they try to write to disk. This is from ins…

问题原文

原帖链接 · std''OrgnlDave · CC BY-SA 3.0

I can't figure out how AWS sets up their Docker 'thin pool' on ElasticBeanstalk and how it is getting filled. My docker thin pool is filling up somehow and causing my apps to crash when they try to write to disk.

This is from inside the container:

>df -h
>     /dev/xvda1                  25G  1.4G   24G   6%

The EBS does, in fact, have a 25GB disk apportioned to it; 1.6 gb is what du -sh / returns.

Outside in EC2, it starts off innocuously enough... (via lvs)

LV          VG     Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
docker-pool docker twi-aot--- 11.86g             37.50  14.65

However, the filesystem will soon re-mount as read-only. via dmesg:

[2077620.433382] Buffer I/O error on device dm-4, logical block 2501385
[2077620.437372] EXT4-fs warning (device dm-4): ext4_end_bio:329: I/O error -28 writing to inode 4988708 (offset 0 size 8388608 starting block 2501632)
[2077620.444394] EXT4-fs warning (device dm-4): ext4_end_bio:329: I/O error     [2077620.473581] EXT4-fs warning (device dm-4): ext4_end_bio:329: I/O error -28 writing to inode 4988708 (offset 8388608 size 5840896 starting block 2502912)

[2077623.814437] Aborting journal on device dm-4-8.
[2077649.052965] EXT4-fs error (device dm-4): ext4_journal_check_start:56: Detected aborted journal
[2077649.058116] EXT4-fs (dm-4): Remounting filesystem read-only

Back out in EC2 instance-land, Docker reports this: (from docker info)

Pool Name: docker-docker--pool
Pool Blocksize: 524.3 kB
Base Device Size: 107.4 GB
Backing Filesystem: ext4
Data file:
Metadata file:
Data Space Used: 12.73 GB
Data Space Total: 12.73 GB
Data Space Available: 0 B
Metadata Space Used: 3.015 MB
Metadata Space Total: 16.78 MB
Metadata Space Available: 13.76 MB
Thin Pool Minimum Free Space: 1.273 GB

LVS dumps this info:

  --- Logical volume ---
  LV Name                docker-pool
  VG Name                docker
  LV UUID                xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  LV Write Access        read/write
  LV Creation host, time ip-10-0-0-65, 2017-03-25 22:37:38 +0000
  LV Pool metadata       docker-pool_tmeta
  LV Pool data           docker-pool_tdata
  LV Status              available
  # open                 2
  LV Size                11.86 GiB
  Allocated pool data    100.00%
  Allocated metadata     17.77%
  Current LE             3036
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

What is this thin pool, why does it fill up, and how do I stop it from doing so? Also, if I have 20+ GB free from inside the container on my / volume, why does it stop new writes? As far as I can tell it is not connected to files that my programs are writing to.

Thank you!

社区高分解答

原帖链接 · joko · CC BY-SA 3.0

The .ebextensions suggested by David Ellis worked for me. I'm unable to comment on his answer, but I wanted to add that you can create a new EBS volume instead of using a snapshot. To mount a 40GB EBS volume, I used the following:

option_settings:
  - namespace: aws:autoscaling:launchconfiguration
    option_name: BlockDeviceMappings
    value: /dev/xvdcz=:40:true

See also this documentation, which has an example of mapping a new 100GB EBS volume to /dev/sdh.

The true at the end means "delete on terminate".

I created a new .ebextensions directory containing an ebs.config file with the above code, then zipped that directory together with my Dockerrun.aws.json. Note that the Dockerrun file must be at the top level of the zip, not inside a subdirectory.

To find where Elastic Beanstalk is mounting the volume, use lsblk on the failing instance. It was also /dev/xvdcz for me, so maybe that is the standard.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2017-03-28)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。