大网数据广告活动
服务器运维问答

Why are pods failing to schedule due to resources when node has plenty available?

· 52CE 我爱测

The pods in my application scale with 1 pod per user (each user gets their own pod). I have the limits for the application container set up like so: resources: limits: cpu: 250m memory: 768Mi requests: cpu: 100m memory: 512Mi The…

问题原文

原帖链接 · Ben Davis · CC BY-SA 4.0

The pods in my application scale with 1 pod per user (each user gets their own pod). I have the limits for the application container set up like so:

  resources:
    limits:
      cpu: 250m
      memory: 768Mi
    requests:
      cpu: 100m
      memory: 512Mi

The nodes in my nodepool have 8GB of memory each. I started up a bunch of user instances to begin testing, and watched my resource metrics go up as I started each one:

CPU:

Memory:

At 15:40, I saw the event logs show this error (note: the first node is excluded using a taint):

0/2 nodes are available: 1 Insufficient memory, 1 node(s) didn't match node selector.

Why did this happen when the memory/cpu requests were still well below the total capacity (~50% for cpu, ~60% mem)?

Here is some relevant info from kubectl describe node:

Non-terminated Pods:          (12 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  ide                         theia-deployment--ac031811--football-6b6d54ddbb-txsd4              110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    13m
  ide                         theia-deployment--ac031811--footballteam-6fb7b68794-cv4c9          110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    12m
  ide                         theia-deployment--ac031811--how-to-play-football-669ddf7c8cjrzl    110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    14m
  ide                         theia-deployment--ac031811--packkide-7bff98d8b6-5twkf              110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    9m54s
  ide                         theia-deployment--ac032611--static-website-8569dd795d-ljsdr        110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    16m
  ide                         theia-deployment--aj090111--spiderboy-6867b46c7d-ntnsb             110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    2m36s
  ide                         theia-deployment--ar041311--tower-defenders-cf8c5dd58-tl4j9        110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    14m
  ide                         theia-deployment--np091707--my-friends-suck-at-coding-fd48ljs7z    110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    4m14s
  ide                         theia-deployment--np091707--topgaming-76b98dbd94-fgdz6             110m (5%)     350m (18%)  528Mi (9%)       832Mi (15%)    5m17s
  kube-system                 csi-azurefile-node-nhbpg                                           30m (1%)      400m (21%)  60Mi (1%)        400Mi (7%)     12d
  kube-system                 kube-proxy-knq65                                                   100m (5%)     0 (0%)      0 (0%)           0 (0%)         12d
  lens-metrics                node-exporter-57zp4                                                10m (0%)      200m (10%)  24Mi (0%)        100Mi (1%)     6d20h

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                       Requests      Limits
  --------                       --------      ------
  cpu                            1130m (59%)   3750m (197%)
  memory                         4836Mi (90%)  7988Mi (148%)
  ephemeral-storage              0 (0%)        0 (0%)
  hugepages-1Gi                  0 (0%)        0 (0%)
  hugepages-2Mi                  0 (0%)        0 (0%)
  attachable-volumes-azure-disk  0             0

已采纳解答

原帖链接 · Ben Davis · CC BY-SA 4.0

I found out that when viewing available capacity, you need to pay attention to Allocatable, and not Capacity. From Azure support:

Please take a look a this document “Resource reservations”, if we follow the example on that document (using round number to 8GB per node):

0.75 + (0.25*4) + (0.20*3) = 0.75GB + 1GB + 0.6GB = 2.35GB / 8GB = 29.37% reserved

For a 8GB server, the amount reserved is around 29.37%, which means:

Amount of memory reserved by node = 29.37% * 8000 = 2349. Allocatable remaining memory = 5651 The first 9 pods will use = 9 * 528 = 4752 Allocatable remaining memory after first pods = 899 (the allocatable memory shown in the kubectl describe node, should be the number available after OS reservation)

In the last number we have to consider the OS reservation that it needs to run, so probably after taking the OS reserved memory, there is not enough space for any more pods on the node, hence the messages.

That will result in an expected behavior, given the calculations.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2020-11-10)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。