大网数据广告活动
服务器运维问答

create a CloudWatch Alarm when an ECS service unable to consistently start tasks successfully

· 52CE 我爱测

If I release a new Docker image with a bug to my ECS Service, then the service will attempt to start new Tasks but will keep the old version around if the new tasks fail to start. In that scenario, it will sometimes (not always) …

问题原文

原帖链接 · Rich · CC BY-SA 4.0

If I release a new Docker image with a bug to my ECS Service, then the service will attempt to start new Tasks but will keep the old version around if the new tasks fail to start.

In that scenario, it will sometimes (not always) emit an Event to the bus like:

service xxx is unable to consistently start tasks successfully. For more information, see the Troubleshooting section.

and sometimes it will just emit loads of events like:

service xxx deregistered 1 targets in target-group yyy

I would like a CloudWatch Alarm to fire in this scenario. How can I achieve that?

I cannot see any CloudWatch metrics that track any relevant events that I could use to trigger this Alarm. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cloudwatch-metrics.html

If the Tasks fail to boot then I don't even get any UnHealthyHostCount metrics on the LB Target Group.

I think I will have to create an EventBridge rule to watch for the above named event, but I can't see an obvious way to have that rule trigger an Alarm. I have set a rule to forward "WARN" and "ERROR" events to SNS/email, but I don't always get these events. So I frequently get a restart loop with no alarms firing. :-(

已采纳解答

原帖链接 · Rich · CC BY-SA 4.0

I have the following infrastructure which I think addresses this requirement:

  1. An alarm on the metric AWS/ApplicationELB / UnHealthyHostCount which sometimes fires
  2. An Event rule with the following pattern forwarding to SNS which catches failed tasks:
{
    "source": [
        "aws.ecs"
    ],
    "detail-type": [
        "ECS Task State Change"
    ],
    "detail": {
        "group": [
            "service:${var.ecs_service_name}"
        ],
        "stoppedReason": [
            "Essential container in task exited"
        ]
    }
}
  1. An Event rule with the following pattern forwarding to SNS which catches the "unable to consistently start tasks successfully" event which sometimes fires:
{
    "source": [
        "aws.ecs"
    ],
    "detail-type": [
        "ECS Service Action"
    ],
    "resources": [
        "${var.ecs_service_arn}"
    ],
    "detail": {
        "eventType": ["WARN", "ERROR"]
    }
}
  1. An alarm on the metric AWS/Events / TriggeredRules which fires when 2 or 3 occurs

This is quite a messy approach, but the best I could find. I am disappointed that ECS doesn't publish metrics to track this common case.

(I do not subscribe anything to the SNS topics created above; they exist solely to make the above rules valid. The events are viewable in the ECS console if required.)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2020-04-16)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。