SparkStreaming或Flink应用的checkpoint路径不释放HDFS存储空间

问题描述 投票:0回答:0

应用实际数据与其使用空间不符

hdfs  dfs -du -h  /landsat_flink_ha   | grep G
42.9 K    89.3 G    /landsat_flink_ha/application_1658448699056_990584

走进去

hdfs dfs -du -h   /landsat_flink_ha/application_1658448699056_990584
0       384 M  /landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d

hdfs dfs -ls /landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d
-rw-r--r--   3 blade supergroup          0 2022-11-18 09:42 /landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d

检查区块

hdfs fsck  /landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d  -files -blocks -racks
Connecting to namenode via http://id-cdh-master-192:50070/fsck?ugi=hdfs&files=1&blocks=1&racks=1&path=%2Flandsat_flink_ha%2Fapplication_1658448699056_990584%2FcompletedCheckpointfbf82284656d
FSCK started by hdfs (auth:SIMPLE) from /172.22.8.89 for path /landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d at Tue Apr 18 06:54:21 UTC 2023
Status: HEALTHY
 Total size:    0 B
 Total dirs:    0
 Total files:   0
 Total symlinks:        0 (Files currently being written: 1)
 Total blocks (validated):  0 (Total open file blocks (not validated): 1)
 Minimally replicated blocks:   0
 Over-replicated blocks:    0
 Under-replicated blocks:   0
 Mis-replicated blocks:     0
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        0
 Missing replicas:      0
 Number of data-nodes:      170
 Number of racks:       1
FSCK ended at Tue Apr 18 06:54:21 UTC 2023 in 8 milliseconds


The filesystem under path '/landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d' is HEALTHY

当我手动删除一些文件时,空间被释放。

***
hdfs dfs -rm -f /landsat_flink_ha/application_1658448699056_990584/completedCheckpointfbf82284656d

hdfs  dfs -du -h  /landsat_flink_ha  
28.6 K    68.3 G    /landsat_flink_ha/application_1658448699056_990584
apache-spark hdfs apache-flink checkpoint
© www.soinside.com 2019 - 2024. All rights reserved.