大网数据广告活动
服务器运维问答

How to share docker metrics with Prometheus using Ubuntu server Snap Docker Service

· 52CE 我爱测

I am using Ubuntu server with a Docker Service installed from the Server Setup (snap) and Prometheus running in a docker container. Server: Distributor ID: Ubuntu Description: Ubuntu 22.04.3 LTS Release: 22.04 Codename: jammy Doc…

问题原文

原帖链接 · Hagbart Celine · CC BY-SA 4.0

I am using Ubuntu server with a Docker Service installed from the Server Setup (snap) and Prometheus running in a docker container.

Server:

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

Docker

sudo snap services
Service                          Startup  Current   Notes
docker.dockerd                   enabled  active    -
docker.nvidia-container-toolkit  enabled  inactive  -
lxd.activate                     enabled  inactive  -
lxd.daemon                       enabled  inactive  socket-activated
lxd.user-daemon                  enabled  inactive  socket-activated

snap

snap    2.61.1
snapd   2.61.1
series  16
ubuntu  22.04
kernel  5.15.0-91-generic

Prometheus docker-compose.yml

version: '3.8'
volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - /home/xxxx/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

    extra_hosts:
      - "host.docker.internal:host-gateway"

    ports:
      - "9090:9090/tcp"

Docker IP:

ip addr show docker0
    8: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
        link/ether 02:42:18:5c:09:7f brd ff:ff:ff:ff:ff:ff
        inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
           valid_lft forever preferred_lft forever
        inet6 fe80::42:18ff:fe5c:97f/64 scope link
           valid_lft forever preferred_lft forever

Issue

So what I want to do is get metrics for my Prometheus Server running in a docker container. I am using this documentation: https://docs.docker.com/config/daemon/prometheus/

In the beginning it says: Add the following configuration:

Linux: /etc/docker/daemon.json

{
  "metrics-addr": "127.0.0.1:9323"
}

I also tried with:

{
   "metrics-addr": "172.17.0.1:9323"
}

After some trouble I found following question / answer on askubuntu: https://askubuntu.com/questions/1296892/snap-edit-docker-configuration

So I added the daemon.json to

/var/snap/docker/current/etc/docker

and restarted the service with:

sudo snap restart docker

after the restart I still do not get any metrics from the host. I test this using:

curl http://localhost:9323/metrics
curl: (7) Failed to connect to localhost port 9323 after 0 ms: Connection refused
curl http://172.17.0.1:9323/metrics
curl: (7) Failed to connect to 172.17.0.1 port 9323 after 0 ms: Connection refused

Docker system info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.10.4)
  compose: Docker Compose (Docker Inc., v2.17.2)

Server:
 Containers: 20
  Running: 20
  Paused: 0
  Stopped: 0
 Images: 20
 Server Version: 20.10.24
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2806fc1057397dbaeefbea0e4e17bddfbd388f38
 runc version:
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-91-generic
 Operating System: Ubuntu Core 22
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.26GiB
 Name: blackholeport
 ID: AMKQ:XQSW:OPFY:ZLWE:VCVI:BKBB:NVO4:5WYJ:W6AZ:JB7M:7T4S:WXNC
 Docker Root Dir: /var/snap/docker/common/var-lib-docker
 Debug Mode: false
 Username: hagbart
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

docker inspect prometheus

[
    {
        "Id": "ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684",
        "Created": "2024-01-19T10:23:03.520072505Z",
        "Path": "/bin/prometheus",
        "Args": [
            "--config.file=/etc/prometheus/prometheus.yml"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 83237,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-01-19T13:00:24.320370241Z",
            "FinishedAt": "2024-01-19T13:00:17.400204943Z"
        },
        "Image": "sha256:2a72b385beafd2b3685ad337d14d23f6851f0ff335c5cd9a04c692b3eb846196",
        "ResolvConfPath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/resolv.conf",
        "HostnamePath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/hostname",
        "HostsPath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/hosts",
        "LogPath": "/var/snap/docker/common/var-lib-docker/containers/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684/ded87ebe4dafa575a06781beacf7e33541a37d23c1594ce0ac80041929ba5684-json.log",
        "Name": "/prometheus",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/home/lukas/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "prometheus_default",
            "PortBindings": {
                "9090/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9090"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "private",
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": [
                "host.docker.internal:host-gateway"
            ],
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "volume",
                    "Source": "prometheus_prometheus_data",
                    "Target": "/prometheus",
                    "VolumeOptions": {}
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199-init/diff:/var/snap/docker/common/var-lib-docker/overlay2/0aa7d62082adf6dcde04943d937cac5dae5d0609b12ae7319421cbf63131fca3/diff:/var/snap/docker/common/var-lib-docker/overlay2/59878a50f72e66a191110dadd15bf94d352cb1e0a543cf6ab0e96f210d9a16e5/diff:/var/snap/docker/common/var-lib-docker/overlay2/d7dd0384e6e888b48e94d3f32bf1e504c2ed1d11526ba5caba7a6cbaa43f1304/diff:/var/snap/docker/common/var-lib-docker/overlay2/8943641d04172c565f42e75963910fdf6df436bc60d3929284c657b16d831dd1/diff:/var/snap/docker/common/var-lib-docker/overlay2/6c943a9276d504534ec9c66ade625447ad516058dce64053b899283aac500395/diff:/var/snap/docker/common/var-lib-docker/overlay2/416d42bd54e76f8a2c78da2799ad6e6c5d2c3d78b822ed8e17ada9bdc0fc17a7/diff:/var/snap/docker/common/var-lib-docker/overlay2/89ab8fa214a8141179afa2c21071e565dc841d1e0f5a69dc372d5b6514456cb0/diff:/var/snap/docker/common/var-lib-docker/overlay2/d99204b78e6ddf1279d8f1b099318f921a61e9017d457955e5a53c5c2d17be8a/diff:/var/snap/docker/common/var-lib-docker/overlay2/259c1d3c5a275a0c51fe499c1a8a2857427cf592eb094dea65447b591ec66058/diff:/var/snap/docker/common/var-lib-docker/overlay2/721958698d3add899bd2597d3dab22267c3aaa40dc556eb7f0cdb8c564b14a74/diff:/var/snap/docker/common/var-lib-docker/overlay2/c122e772ebb24d66d663577b099800060d10e1d1fda0b38e75ea73bc2fdf6273/diff:/var/snap/docker/common/var-lib-docker/overlay2/b1a9df93abc1fbba6b8f3644eaf5ec9a76b8b2193ad1d934241242fb54b74cd0/diff",
                "MergedDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199/merged",
                "UpperDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199/diff",
                "WorkDir": "/var/snap/docker/common/var-lib-docker/overlay2/5656a88cbb1d0e0fd4ed939b0cdb30de86b7d2e4af2afd06d2ea33d8dd110199/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/home/lukas/data/config/prometheus/prometheus.yml",
                "Destination": "/etc/prometheus/prometheus.yml",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "volume",
                "Name": "prometheus_prometheus_data",
                "Source": "/var/snap/docker/common/var-lib-docker/volumes/prometheus_prometheus_data/_data",
                "Destination": "/prometheus",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "ded87ebe4daf",
            "Domainname": "",
            "User": "nobody",
            "AttachStdin": false,
            "AttachStdout": true,
            "AttachStderr": true,
            "ExposedPorts": {
                "9090/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "--config.file=/etc/prometheus/prometheus.yml"
            ],
            "Image": "prom/prometheus:latest",
            "Volumes": {
                "/prometheus": {}
            },
            "WorkingDir": "/prometheus",
            "Entrypoint": [
                "/bin/prometheus"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "f387601dc597da679d6a2290f4192863d07c16791b9d33e29a4e84eb43b2e5df",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.depends_on": "",
                "com.docker.compose.image": "sha256:2a72b385beafd2b3685ad337d14d23f6851f0ff335c5cd9a04c692b3eb846196",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "prometheus",
                "com.docker.compose.project.config_files": "/home/lukas/BlackholePort/docker/prometheus/docker-compose.yml",
                "com.docker.compose.project.working_dir": "/home/lukas/BlackholePort/docker/prometheus",
                "com.docker.compose.service": "prometheus",
                "com.docker.compose.version": "2.17.2",
                "maintainer": "The Prometheus Authors <[email omitted]>"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "b882dba1196b5ba11a81cedda028552f453ed82e42cec9fc8fc767d4192571f2",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "9090/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "9090"
                    },
                    {
                        "HostIp": "::",
                        "HostPort": "9090"
                    }
                ]
            },
            "SandboxKey": "/run/snap.docker/netns/b882dba1196b",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "prometheus_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "prometheus",
                        "prometheus",
                        "ded87ebe4daf"
                    ],
                    "NetworkID": "971e8873d46285b63d70fb6feeada747176a5427aa7a19127cb7e8dc8a7d8a37",
                    "EndpointID": "b67826c8e2736e5ac55ab67a8f9d67a6822dd3260a214622f65de968b0fa972e",
                    "Gateway": "192.168.32.1",
                    "IPAddress": "192.168.32.2",
                    "IPPrefixLen": 20,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:c0:a8:20:02",
                    "DriverOpts": null
                }
            }
        }
    }
]

docker network ls

docker network ls
NETWORK ID     NAME                    DRIVER    SCOPE
f124d5db3e0b   bridge                  bridge    local
862535ef9371   deluge_default          bridge    local
86bf6736570f   foundryvtt_default      bridge    local
909b8162cdbe   grafana_default         bridge    local
299ce7db89b5   heimdall_default        bridge    local
92a81645a176   host                    host      local
a5e580aa1810   microbin_default        bridge    local
da8d9cb1d3e7   nginx-pm_default        bridge    local
055308848eba   none                    null      local
93afb48bbba2   o-portainer             bridge    local
460a87dcb670   o-security              bridge    local
ea16af5d1414   pihole_default          bridge    local
250440a093d7   portainer_default       bridge    local
971e8873d462   prometheus_default      bridge    local
dcbfb120b9b2   prometheus_monitoring   bridge    local
a482e951879d   prowlarr_default        bridge    local
9ab6f6734815   radarr_default          bridge    local
f6abd6de38d3   sonarr_default          bridge    local
0e062c280f80   spigotmc_default        bridge    local
640505be1145   teamspeak_default       bridge    local

after adding portainer to host network same issue

docker-compose.yml

version: '3.8'
volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - /home/lukas/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

    extra_hosts:
      - "host.docker.internal:host-gateway"
    network_mode: "host"

    ports:
      - "9090:9090/tcp

docker inspect prometheus (network part)

 "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "f114aceaadadca19fe9da85b04dbce4fc9aedae6d7f91a2bed8b3290d3186e82",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/run/snap.docker/netns/default",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "host": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "92a81645a17671cf440926ed83cdb6063790dcf2e18813722d96ccc90614ffb3",
                    "EndpointID": "4a4f2cef2556314c2feb526928b24784005a4617583e10813bae06f56d5db8e2",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }

已采纳解答

原帖链接 · nijave · CC BY-SA 4.0

Docker Compose is creating a network called "prometheus_default" and it's attaching the Prometheus container to that network. You have Docker daemon set to advertise metrics on the docker0 interface which is generally on the bridge default network Docker creates.

On the host where Docker is running, you can verify Docker daemon (-p, to show proccesses) is listening (-l) on the tcp (-t) metrics port number 9323 (-n) via ss or netstat

$ ss -tlpn | grep :9323
LISTEN 0      4096            127.0.0.1:9323       0.0.0.0:*    users:(("dockerd",pid=199176,fd=17))

Option 1

Try adding this config to docker-compose.yaml to add Prometheus to the bridge network

version: '3.8'
volumes:
  prometheus_data: {}

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    volumes:
      - /home/lukas/data/config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'

    extra_hosts:
      - "host.docker.internal:host-gateway"
    network_mode: bridge

    ports:
      - "9090:9090/tcp

while making sure Docker is set to advertise metrics on the default bridge

{
   "metrics-addr": "172.17.0.1:9323"
}

Option 2

Change Docker daemon to listen on prometheus_default network

docker network inspect prometheus_default
[
    {
        "Name": "prometheus_default",
        "Id": "55a40edb42b2f3c1144236a765339e9b1c8dd0542df929430dc07829c5e156d5",
        "Created": "2023-03-31T16:51:48.212544059-04:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "prometheus",
            "com.docker.compose.version": "1.29.2"
        }
    }
]

Set in daemon.json

{
   "metrics-addr": "172.19.0.1:9323"
}

Option 3

Allow any network to access Docker metrics (keep in mind much more services will have access to the Docker daemon metrics interface which may or may not be desirable)

{
  "metrics-addr": "0.0.0.0:9323"
}

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2024-01-19)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。