按照 this 指南,我设置了 cadvisor 来监视本地 docker 容器。 Prometheus 正在抓取 cAdvisor 以获取在 Grafana 中可视化的数据。
我正在尝试获取主机上运行的所有 docker 容器的磁盘使用情况。我想要得到类似于运行后显示的输出
docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 21 10 1.18GB 489.9MB (41%)
Containers 10 8 27B 0B (0%)
Local Volumes 16 5 2.369GB 1.615GB (68%)
Build Cache 0 0 0B 0B
Grafana 用于可视化数据。为了获取磁盘使用情况,我运行以下查询以从 Prometheus 获取数据。
sum(container_fs_usage_bytes)
问题是我得到的值是覆盖文件系统的整个值(82GB)。
有关文件系统使用情况的以下行显示在 cAdvisor (127.0.0.1/metrics) 中
# TYPE container_fs_usage_bytes gauge
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="",container_label_com_docker_compose_container_number="",container_label_com_docker_compose_oneoff="",container_label_com_docker_compose_project="",container_label_com_docker_compose_service="",container_label_com_docker_compose_version="",container_label_maintainer="",container_label_org_label_schema_group="",device="/dev/sda2",id="/",image="",name=""} 4.4317888512e+10
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="",container_label_com_docker_compose_container_number="",container_label_com_docker_compose_oneoff="",container_label_com_docker_compose_project="",container_label_com_docker_compose_service="",container_label_com_docker_compose_version="",container_label_maintainer="",container_label_org_label_schema_group="",device="overlay",id="/",image="",name=""} 4.4317888512e+10
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="",container_label_com_docker_compose_container_number="",container_label_com_docker_compose_oneoff="",container_label_com_docker_compose_project="",container_label_com_docker_compose_service="",container_label_com_docker_compose_version="",container_label_maintainer="",container_label_org_label_schema_group="",device="shm",id="/",image="",name=""} 0
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="",container_label_com_docker_compose_container_number="",container_label_com_docker_compose_oneoff="",container_label_com_docker_compose_project="",container_label_com_docker_compose_service="",container_label_com_docker_compose_version="",container_label_maintainer="",container_label_org_label_schema_group="",device="tmpfs",id="/",image="",name=""} 0
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="0616b7e25768c2764fd083b53a061c53c6a2ffea9f8d5e74f6ec61fbc03aeeba",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="prometheus",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="The Prometheus Authors <[email protected]>",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/6fbef095b2cf0c3f65e906d4ddca4102cd0646590c4003e50186af40f02ef805",image="prom/prometheus:v2.4.2",name="prometheus"} 409600
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="19747419ff433903067d8681ee6149861347fcf65e7db246f4b56301feb53b78",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="caddy",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/611fe3e5812d9c3a3abc9dab27fd8a91f68c2e3acc3257b5926eae0cbf8914cf",image="stefanprodan/caddy",name="caddy"} 53248
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="38ec70818559317528d83c7c8d889ab9ff2dcd78637a3e9cae71083014abc281",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="grafana",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/70cdb5ec332973c7a70cf339f5abc48afa5d8f6e6ff5ba0216031ff1c86aa2c5",image="grafana/grafana:5.2.4",name="grafana"} 204800
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="42eb1e0967f80c097581f64bace779010aac94368f2fd366215b2a2506340a60",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="pushgateway",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="The Prometheus Authors <[email protected]>",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/39049775176f8315f4ee3acf8167a6986ee1ce4fe89cebc20061accec8cdb5cf",image="prom/pushgateway",name="pushgateway"} 69632
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="58ee943e7e48ae304fa7040d44ebeeb440627d311716a70665f4c43f16e6ad54",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="nodeexporter",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/6f4c169111b89ea2bb857fbd0cf52c1c729b17dbdd8e7a8e011a1f9ba7128e90",image="prom/node-exporter:v0.16.0",name="nodeexporter"} 184320
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="db9a938539bf7f1b610cda7420e62088d2bd27c50f52b2510b3bfc9d6c099744",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="cadvisor",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/2d9fee9bd94d9d627259f04e17468840700a78267a894f3fbedda0d375c5e985",image="google/cadvisor:v0.31.0",name="cadvisor"} 86016
container_fs_usage_bytes{container_label_com_docker_compose_config_hash="fdcc8c8d658803bc1254fab7a308bc999f76e777a0f414d2f6c58024530cdfa8",container_label_com_docker_compose_container_number="1",container_label_com_docker_compose_oneoff="False",container_label_com_docker_compose_project="dockprom",container_label_com_docker_compose_service="alertmanager",container_label_com_docker_compose_version="1.17.1",container_label_maintainer="",container_label_org_label_schema_group="monitoring",device="/dev/sda2",id="/docker/c6f72c08c11cfd862fa565bfd190744555ff2d345dfce553b8839a4d13c0d0d2",image="prom/alertmanager:v0.15.2",name="alertmanager"} 73728
这是主机上df -h的输出
overlay 228G 39G 178G 18% /var/lib/docker/overlay2/d3de4ea22cdec9aac513d0799b43b0c7fefdb7b75391b6c7b4bc35f46eec817c/merged
如何从 cAdvisor 在 Grafana 中获得与 docker system df 返回的值相同的值?
您的 Prometheus/cAdvisor 查询是错误的。 cAdvisor 会抓取所有容器以及主机系统的指标。如果您查看 Grafana/Prometheus 图中的 container_fs_usage_bytes 的“原始”数据,您可以看到这一点。您将找到带有
id="/docker/<containerID>"
、name="<containername>"
和 device="dev/sdc"
的所有容器的数据,还可以找到带有 id="/"
、name=""
等主机进程的数据。 device="/rootfs/run"
。
我建议您像这样过滤掉这些条目:
sum(container_fs_usage_bytes{name!=""})
Grafana 不收集和存储任何数据。它只是可视化数据,这些数据已经收集/存储在一些支持的时间序列数据库中。
如果 cAdvisor 能够获取所需的指标,则配置 cAdvisor 存储插件(InfluxDB 是一个好的开始)并在 Grafana 中可视化它们