尽管存在 wal 文件，Postgres 热备仍卡在等待 contrrecord 上

Question

我们正在使用复制槽对带有 postgres16 的 patroni 多数据中心 HA 进行实验。在从头开始构建远程集群的高层，我们通过运行基础备份，而活动正在经历写入。

我们通过 patoni 使用复制槽进行热备。这是活动集群配置的示例

postgresql:
  parameters:
    archive_command: /bin/true
    archive_mode: 'on'
    checkpoint_completion_target: '0.9'
    checkpoint_timeout: 300
    default_statistics_target: 100
    effective_cache_size: 12GB
    effective_io_concurrency: 2
    fsync: 'on'
    full_page_writes: 'on'
    hot_standby: true
    hot_standby_feedback: true
    idle_in_transaction_session_timeout: 10min
    log_min_duration_statement: 10000
    log_min_error_statement: ERROR
    log_min_messages: WARNING
    log_temp_files: 4000
    maintenance_work_mem: 3GB
    max_connections: 512
    max_parallel_maintenance_workers: 4
    max_parallel_workers: 10
    max_parallel_workers_per_gather: 2
    max_prepared_transactions: 1024
    max_replication_slots: 10
    max_slot_wal_keep_size: 20GB
    max_wal_senders: 10
    max_wal_size: 2GB
    max_worker_processes: 10
    min_wal_size: 512MB
    random_page_cost: 2
    shared_buffers: 8GB
    synchronous_commit: remote_write
    track_functions: all
    track_io_timing: 'on'
    wal_buffers: 16MB
    wal_compression: 'on'
    wal_keep_segments: 128
    wal_level: replica
    work_mem: 16MB
  use_pg_rewind: true
  use_slots: true
retry_timeout: 14

我们在实验中看到的是，一旦PG在basebackup后通过patroni启动，它就会转移到patroni领导者/运行状态，然后我们应用备用集群配置

curl -X PATCH http://localhost:8008/config \
     -H "Content-Type: application/json" \
     -d '{
           "standby_cluster": {
             "host": "standby.example.com",
             "port": 5432,
             "primary_slot_name": "standby_slot"
           }
         }' \
     --data-urlencode "force=true"

这会导致 pg rewind ，然后集群在大多数时间移动到 Standbyleader/streaming 状态。但在其他时候，它会在倒带结束时转移到备用领导者/起始状态。我们在 pg 日志中看到的内容类似于下面的内容

2024-08-19 20:02:00 UTC:::@:[762363]:LOG:  contrecord is requested by 93/4C000028
2024-08-19 20:02:00 UTC:::@:[762363]:LOG:  waiting for WAL to become available at 93/4C000040

进一步分析复制槽中的lsn表明该记录是在基础备份后生成的，是在基础备份后由领导者归档的。在备用集群中，与 lsn 关联的 wal 文件存在于 pg_wal dir 中，但数据库在启动时卡住，它永远不会进入流状态，并且复制槽的当前状态不是活动的。 尽管 wal 文件存在于活动集群上，但为什么它没有被发送到热备用，以使其能够将其移动到流状态？

Answer 1

关键问题是引导后的中间状态转换，其中数据库转换为领导者状态，然后应用帕托尼中的备用领导者配置。这会导致 pg 倒带，其中时间线收敛和重做日志似乎存在问题。我们修改了步骤

1. 停止 postgres

2. patronictl 删除集群

3. 清理 DCS 配置

4. 应用备用集群守护者让引导程序通过，无需中间过渡到领导者状态

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    synchronous_mode: false
    slots:
      repslot:
        type: 'physical'
    standby_cluster:
      create_replica_methods:
      - basebackup
      host: 192.168.6.60
      port: 5432
      primary_slot_name: repslot
    postgresql:
      use_pg_rewind: true
      use_slots: true
      parameters:
        wal_level: 'replica'
        hot_standby: 'on'
        synchronous_commit: 'local'
        wal_keep_segments: 960
        wal_compression: 'on'
        archive_mode: 'on'
        archive_command: '/bin/true'
        fsync: 'on'

$PATH=/usr/lib/postgresql/16/bin:$PATH ./patroni.py postgres1.yml:

2024-08-26 09:27:40,341 INFO: Selected new etcd server http://localhost:2379
2024-08-26 09:27:40,344 INFO: No PostgreSQL configuration items changed, nothing to reload.
2024-08-26 09:27:40,347 INFO: Lock owner: None; I am postgresql1
2024-08-26 09:27:40,350 INFO: trying to bootstrap a new standby leader
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/4000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: created temporary replication slot "pg_basebackup_479100"
pg_basebackup: write-ahead log end point: 0/4000100
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
2024-08-26 09:27:45,056 INFO: replica has been created using basebackup
2024-08-26 09:27:45,058 INFO: bootstrapped clone from remote member postgresql://127.0.0.1:5432
2024-08-26 09:27:45,242 INFO: postmaster pid=479106
localhost:5433 - no response
2024-08-26 09:27:45.263 CEST [479106] LOG:  starting PostgreSQL 16.4 (Ubuntu 16.4-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
2024-08-26 09:27:45.263 CEST [479106] LOG:  listening on IPv4 address "127.0.0.1", port 5433
2024-08-26 09:27:45.277 CEST [479106] LOG:  listening on Unix socket "../.s.PGSQL.5433"
2024-08-26 09:27:45.294 CEST [479110] LOG:  database system was interrupted; last known up at 2024-08-26 09:27:44 CEST
2024-08-26 09:27:45.715 CEST [479110] LOG:  entering standby mode
2024-08-26 09:27:45.715 CEST [479110] LOG:  starting backup recovery with redo LSN 0/4000028, checkpoint LSN 0/4000060, on timeline ID 1
2024-08-26 09:27:45.738 CEST [479110] LOG:  redo starts at 0/4000028
2024-08-26 09:27:45.746 CEST [479110] LOG:  completed backup recovery with redo LSN 0/4000028 and end LSN 0/4000100
2024-08-26 09:27:45.746 CEST [479110] LOG:  consistent recovery state reached at 0/4000100
2024-08-26 09:27:45.746 CEST [479106] LOG:  database system is ready to accept read-only connections
2024-08-26 09:27:45.766 CEST [479113] LOG:  started streaming WAL from primary at 0/5000000 on timeline 1
localhost:5433 - accepting connections
localhost:5433 - accepting connections
2024-08-26 09:27:46,271 WARNING: Could not activate Linux watchdog device: Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'
2024-08-26 09:27:46,271 INFO: establishing a new patroni heartbeat connection to postgres
2024-08-26 09:27:46,292 INFO: initialized a new cluster
2024-08-26 09:27:56,288 INFO: no action. I am (postgresql1), the standby leader with the lock

以上似乎解决了问题。