首页虚拟化与容器 VMware vCloud Director 数据库节点磁盘异常问题处理

vCloud Director 数据库节点磁盘异常问题处理

2026年8月22日

0x01 前言

我有多套 vCloud Director 以数据库高可用集群方式运行，即1个 primary 节点及2个 standby 节点。某天收到反馈提及用户通过 WEB GUI 创建虚拟机时页面卡住，创建任务一直停在”进行中”。

排查过程中发现的问题比预想的多：某个 standby 节点存储用完，导致 cell 服务故障，继而发现该节点的 PostgreSQL 流复制已经中断一个多月。

这篇文章记录磁盘异常与数据库重建的完整过程，供参考。

0x02 集群架构

本环境共三台 vCD Appliance（Photon OS 2.0，VCD 10.3.2.19375051），每台同时承载 vCD cell 与嵌入式 PostgreSQL，通过 **repmgr 4.4** 组成数据库 HA：

| 节点 | IP (eth0 / eth1) | repmgr Node ID | repmgr 角色 | 说明 |
| --- | --- | --- | --- | --- |
| vcd1-xx-xx-xx | vcd1-eth0 / vcd1-eth1 | 7893 | standby | 数据库 standby |
| vcd2-xx-xx-xx | vcd2-eth0 / vcd2-eth1 | 32572 | standby | 数据库 standby（本次故障节点）|
| vcd3-xx-xx-xx | vcd3-eth0 / vcd3-eth1 | 14185 | **primary** | 数据库主节点 |

                +---------------------+
                |   vcd3  (primary)   |
                |     vcd3-eth1       |
                +----------+----------+
                           | repmgr 流复制
          +----------------+----------------+
          |                                 |
+---------+---------+            +----------+----------+
| vcd1 (standby)    |            | vcd2 (standby)    |
|    vcd1-eth1      |            |    vcd2-eth1      |
+-------------------+            +-------------------+

VCD cell 访问数据库时，动态指向当前 primary 的 eth1 地址，所以故障切换时 cell 会自动切换数据库连接。

数据库信息如下：

数据库：vcloud
库用户：vcloud
repmgr 元数据库用户：repmgr

0x03 问题1 vcd2 存储满

0x03.1 现象

先正常查看集群数据库健康状态：

$ sudo -u postgres /opt/vmware/vpostgres/10/bin/repmgr \
    -f /opt/vmware/vpostgres/10/etc/repmgr.conf cluster show

 ID    | Name          | Role    | Status    | Upstream        | Location | Priority | Timeline | Connection string
-------+---------------+---------+-----------+-----------------+----------+----------+----------+----------------------------------------------------------------
 7893  | vcd1-xx-xx-xx | standby |   running | vcd3-xx-xx-xx   | default  | 100      | 2        | host=vcd1-eth1 user=repmgr dbname=repmgr connect_timeout=2
 14185 | vcd3-xx-xx-xx | primary | * running |                 | default  | 100      | 2        | host=vcd3-eth1 user=repmgr dbname=repmgr connect_timeout=2
 32572 | vcd2-xx-xx-xx | standby |   running | ! vcd3-xx-xx-xx | default  | 100      | 2        | host=vcd2-eth1 user=repmgr dbname=repmgr connect_timeout=2

WARNING: following issues were detected
  - node "vcd2-xx-xx-xx" (ID: 32572) is not attached to its upstream node "vcd3-xx-xx-xx" (ID: 14185)

vcd2 的 `!` 表示它没有成功 attach 到 primary。进一步查服务，发现 vcd2 的 vmware-vcd 服务为 failed 状态，根分区 100% 满，只剩 17M，Java cell 连日志都写不进去，所以 cell 起不来：

$ systemctl status vmware-vcd
* vmware-vcd.service - LSB: The VMware vCloud Director service
   Loaded: loaded (/etc/rc.d/init.d/vmware-vcd; generated; vendor preset: enabled)
  Drop-In: /etc/systemd/system/vmware-vcd.service.d
           `-10-aftervaos.conf
   Active: failed (Result: exit-code) since Wed 2026-08-19 07:52:17 UTC; 20min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 943 ExecStart=/etc/rc.d/init.d/vmware-vcd start (code=exited, status=154)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/vmware-vcd.service

$ journalctl -u vmware-vcd --no-pager -n 10
Aug 19 07:52:12 vcd2 systemd[1]: Starting LSB: The VMware vCloud Director service...
Aug 19 07:52:17 vcd2 vmware-vcd[943]: Starting vmware-vcd-watchdog: [FAILED]
Aug 19 07:52:17 vcd2 systemd[1]: vmware-vcd.service: Control process exited, code=exited status=154
Aug 19 07:52:17 vcd2 systemd[1]: Failed to start LSB: The VMware vCloud Director service.

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        28G   26G   17M 100%  /

$ tail /opt/vmware/vcloud-director/logs/cell.log | head -20
log4j:ERROR Failed to flush writer,
log4j:ERROR Failed to flush writer,
log4j:ERROR Failed to flush writer,
log4j:ERROR Failed to flush writer

0x03.2 问题定位

根分区大目录排查：

$ du -xh --max-depth=1 / | sort -rh | head -10
26G  /
25G  /opt
1004M  /var
597M  /usr
14M  /etc
36K  /root
12K  /mnt

$ du -sh /opt/vmware/vcloud-director/data/activemq/defaultBroker
22G   /opt/vmware/vcloud-director/data/activemq/defaultBroker

$ find /opt/vmware/vcloud-director/data/activemq/defaultBroker -name 'activemq-data-*.amq' | wc -l
2195

罪魁祸首是 vcd2 上嵌入式 ActiveMQ Broker（KahaDB）的 journal 文件，activemq-data-*.amq 堆积 2195 个，每个 10MB，合计约 21.4GB。对比正常的 vcd1：

| 节点 | `defaultBroker/` 大小 | `.amq` 文件数 |
| --- | --- | --- |
| vcd1（正常） | 23M | 2 |
| vcd2（异常） | **22G** | **2195** |

而且这些文件从 2025年6起，每个月都在新增且从不回收。这说明该节点 KahaDB 的消息确认和 checkpoint 回收机制一直没生效。

说明：.amq 并不是普通日志，而是 ActiveMQ 持久化的 JMS 消息存储（KahaDB journal）。我用 strings 查看文件内容，里面基本都是 com.vmware.vcloud.common.configuration.impl.ConfigUpdatedMessage 这类 VCD 内部 JMS 消息对象。正常情况消息消费后会在 checkpoint 时清理旧 data 文件，但这里一直没清理。与之对应的 .bindings 、.tmp 文件也和这台异常节点一起堆积。

0x03.3 清理

确认当前 cell 已停止服务，defaultBroker 没有被进程占用，也没有 lock 后，为把 2026 年 8 月 1 日之前修改过的 journal 文件全部删除：

$ cd /opt/vmware/vcloud-director/data/activemq/defaultBroker

# 1) 先记录将被删除的文件清单，便于追溯（数量与总大小）
$ find . -maxdepth 1 ! -newermt '2026-08-01' \
    \( -name '*.amq' -o -name '*.bindings' -o -name '*.tmp' \) \
    -printf '%s %f\n' > /root/activemq_deleted_20260819.list
$ wc -l /root/activemq_deleted_20260819.list
2196
$ awk '{s+=$1} END {printf "%.2f GB\n", s/1073741824}' /root/activemq_deleted_20260819.list
20.61 GB

# 2) 执行删除
$ find . -maxdepth 1 ! -newermt '2026-08-01' \
    \( -name '*.amq' -o -name '*.bindings' -o -name '*.tmp' \) -delete

# 3) 确认清理结果与空间
$ find . -maxdepth 1 -name '*.amq' | wc -l
91
$ du -sh .
917M   .
$ df -h /
/dev/sda3  28G  5.1G  21G  20%  /

释放约 21GB，根分区从 100% 回落到 20%。随后启动 cell：

$ systemctl start vmware-vcd

$ systemctl is-active vmware-vcd
active

$ curl -sk --max-time 10 -o /dev/null -w 'HTTP %{http_code} (%{time_total}s)\n' https://vcd2-eth0/api/server_status
HTTP 200 (0.030569s)

cell 恢复正常。

0x04 问题2 vcd2 数据库滞后

0x04.1 standby follow 无法追平

cell 起来后，尝试用 repmgr 让 vcd2 重新跟随 primary：

$ sudo -u postgres /opt/vmware/vpostgres/10/bin/repmgr \
    -f /opt/vmware/vpostgres/10/etc/repmgr.conf standby follow --dry-run

NOTICE: attempting to find and follow current primary
INFO: connected to node 14185, checking for current primary
INFO: follow target is primary node "vcd3-xx-xx-xx" (ID: 14185)
INFO: replication connection to the local node was successful
INFO: replication connection to the follow target node was successful
INFO: local and follow target system identifiers match
DETAIL: system identifier is 7065916388744459236
INFO: timelines are same, this server is not ahead
DETAIL: local node lsn is A85/2A4A83D8, follow target lsn is 218E/6B785D98
INFO: prerequisites for executing STANDBY FOLLOW are met

$ sudo -u postgres /opt/vmware/vpostgres/10/bin/repmgr \
    -f /opt/vmware/vpostgres/10/etc/repmgr.conf standby follow --log-to-file

[2026-08-19 08:31:46] [NOTICE] attempting to find and follow current primary
[2026-08-19 08:31:46] [INFO] timelines are same, this server is not ahead
[2026-08-19 08:31:46] [NOTICE] setting node 32572's upstream to node 14185
[2026-08-19 08:31:46] [NOTICE] restarting server using "sudo /usr/bin/systemctl restart vpostgres"
[2026-08-19 08:32:21] [NOTICE] STANDBY FOLLOW successful
[2026-08-19 08:32:21] [DETAIL] standby attached to upstream node "vcd3-xx-xx-xx" (ID: 14185)

repmgr 元数据更新成功，但 cluster show 依然显示 vcd2 没 attach，仍有感叹号 “!”，看 PostgreSQL 日志：

$ tail /var/vmware/vpostgres/current/pgdata/log/postgresql-*.log
FATAL: could not receive data from WAL stream:
  ERROR:  requested WAL segment 0000000200000A850000002A has already been removed
LOG:  received SIGHUP, reloading configuration files
LOG:  started streaming WAL from primary at A85/2A000000 on timeline 2
FATAL: could not receive data from WAL stream:
  ERROR:  requested WAL segment 0000000200000A850000002A has already been removed

0x04.2 定位问题

vcd2 需要的旧 WAL 段 0000000200000A850000002A 在 primary 上早已被回收，所以无法增量追赶。翻日志发现这个断档不是这次磁盘故障造成的，而是从 7 月 21 日 00:00 就开始每天午夜都重试失败：

2026-07-21 00:00:04.673 UTC [29870] FATAL: requested WAL segment 0000000200000A850000002A has already been removed
2026-07-22 00:00:04.611 UTC [13940] FATAL: requested WAL segment 0000000200000A850000002A has already been removed
2026-07-23 00:00:03.536 UTC [28434] FATAL: requested WAL segment 0000000200000A850000002A has already been removed
2026-07-28 00:00:01.898 UTC [14313] FATAL: requested WAL segment 0000000200000A850000002A has already been removed
2026-07-31 00:00:04.884 UTC [25614] FATAL: requested WAL segment 0000000200000A850000002A has already been removed
2026-08-01 00:00:04.526 UTC [9439]  FATAL: requested WAL segment 0000000200000A850000002A has already been removed
2026-08-07 00:00:04.761 UTC [21975] FATAL: requested WAL segment 0000000200000A850000002A has already been removed

也就是说 vcd2 的数据库副本滞后约一个月，等于一个过期快照。standby follow 只能用来追差距不大且未 diverged 的 standby。当 WAL 已断档，就只能全量重建。

repmgr node status 也印证了滞后量：Replication lag 显示一个异常大的秒数，Last received LSN 长时间停在 A85/2A000000 不动。

0x04.3 全量重建

流程参考 VMware 官方 “Recover from VMware Cloud Director Appliance Standby Cell Failure”，本环境只有 vcd2 故障，不需要处理 primary 和 vcd1：

# 1) 停掉 vcd2 的 cell 与数据库
$ systemctl stop vmware-vcd

$ systemctl stop vpostgres

$ systemctl is-active vmware-vcd vpostgres
inactive
inactive

# 2) 备份旧数据目录
$ mv /var/vmware/vpostgres/current/pgdata \
     /var/vmware/vpostgres/pgdata.bak.$(date +%Y%m%d_%H%M%S)

$ ls -ld /var/vmware/vpostgres/pgdata.bak.*
drwx------ 20 postgres users 4096 Aug 19 08:39 /var/vmware/vpostgres/pgdata.bak.20260819_084020

$ du -sh /var/vmware/vpostgres/pgdata.bak.20260819_084020
17G   /var/vmware/vpostgres/pgdata.bak.20260819_084020

# 3) 从 primary（eth1 vcd3-eth1）全量 clone
$ su - postgres -c "/opt/vmware/vpostgres/current/bin/repmgr \
    -d 'host=vcd3-eth1 user=repmgr dbname=repmgr' \
    -f /opt/vmware/vpostgres/current/etc/repmgr.conf standby clone"

NOTICE: destination directory "/var/vmware/vpostgres/current/pgdata" provided
INFO: connecting to source node
DETAIL: connection string is: user=repmgr dbname=repmgr host=vcd3-eth1
DETAIL: current installation size is 2199 MB
NOTICE: checking for available walsenders on the source node (2 required)
NOTICE: checking replication connections can be made to the source server (2 required)
INFO: creating directory "/var/vmware/vpostgres/current/pgdata"...
NOTICE: starting backup (using pg_basebackup)...
INFO: executing:
  /opt/vmware/vpostgres/current/bin/pg_basebackup -l "repmgr base backup" \
    -D /var/vmware/vpostgres/current/pgdata \
    -d 'user=repmgr dbname=repmgr host=vcd3-eth1' -X stream
NOTICE: standby clone (using pg_basebackup) complete
NOTICE: you can now start your PostgreSQL server
HINT: after starting the server, you need to re-register this standby with "repmgr standby register --force"

# 4) 启动数据库，确认进入 recovery（standby）模式
$ systemctl start vpostgres

$ systemctl is-active vpostgres
active

$ su - postgres -c "/opt/vmware/vpostgres/current/bin/psql -d postgres -tAc 'select pg_is_in_recovery();'"
t

# 检查 clone 生成并指向 primary 的 recovery.conf
$ cat /var/vmware/vpostgres/current/pgdata/recovery.conf
standby_mode = 'on'
primary_conninfo = 'user=repmgr host=vcd3-eth1 application_name=''vcd2-xx-xx-xx'' connect_timeout=2'
recovery_target_timeline = 'latest'

# 5) 再次核验 node status
$ su - postgres -c "/opt/vmware/vpostgres/10/bin/repmgr \
    -f /opt/vmware/vpostgres/10/etc/repmgr.conf node status"

Node "vcd2-xx-xx-xx":
    PostgreSQL version: 10.17
    Total data size: 2198 MB
    Role: standby
    Upstream node: vcd3-xx-xx-xx (ID: 14185)
    Replication lag: 0 seconds
    Last received LSN: 218E/887BA470
    Last replayed LSN: 218E/887BA470

# 6) 启动 cell
$ systemctl start vmware-vcd

$ curl -sk --max-time 10 -o /dev/null -w 'HTTP %{http_code}\n' https://vcd2-eth0/api/server_status
200

最终验证集群状态：

$ sudo -u postgres /opt/vmware/vpostgres/10/bin/repmgr \
    -f /opt/vmware/vpostgres/10/etc/repmgr.conf cluster show

 ID    | Name          | Role    | Status    | Upstream      | Location | Priority | Timeline | Connection string
-------+---------------+---------+-----------+---------------+----------+----------+----------+----------------------------------------------------------------
 7893  | vcd1-xx-xx-xx | standby |   running | vcd3-xx-xx-xx | default  | 100      | 2        | host=vcd1-eth1 user=repmgr dbname=repmgr connect_timeout=2
 14185 | vcd3-xx-xx-xx | primary | * running |               | default  | 100      | 2        | host=vcd3-eth1 user=repmgr dbname=repmgr connect_timeout=2
 32572 | vcd2-xx-xx-xx | standby |   running | vcd3-xx-xx-xx | default  | 100      | 2        | host=vcd2-eth1 user=repmgr dbname=repmgr connect_timeout=2

不再有 “!” 标记，三节点全部健康，vcd2 复制滞后为 0。

0x05 结语

这次排障信息量不小，归纳几个容易踩的点和经验：

嵌入式 ActiveMQ KahaDB 会撑爆根分区。遇到 cell 莫名起不来he 日志刷 log4j:ERROR Failed to flush writer，先检查存储情况，再检查 /opt/vmware/vcloud-director/data/activemq/defaultBroker 。考虑清理旧的 journal 文件，包含：activemq-data-*.amq、过期的 .bindings、.tmp，前提是确认 cell 已停止、目录无进程占用且无 lock。

repmgr standby follow 不是万能的。requested WAL segment 0000000200000A850000002A has already been removed 一出现，就说明 standby 落后太多。primary 的 WAL 已被回收，直接全量 standby clone 重建，别在 follow 上反复耗时间。

要做好健康状态监控。vcd2 的流复制其实断了一个月，是因为没人看 repmgr 的 “!” 标记和每天重试的日志，才拖到这次一起暴露。建议对 repmgr 心跳和cell 服务状态做监控告警。