test: add isolated backup restore rehearsal
This commit is contained in:
@@ -63,6 +63,7 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
||||
3. 已补齐受保护管理环境中的固定凭据操作文件、轮换、撤销、幂等回执和脱敏管理审计；轮换保留授权绑定并要求新令牌重新建立会话，撤销允许在停用范围内执行且不改写历史记录。仍需补齐提交不明故障切点、多节点安全重绑演练、凭据最小权限配置核对和生产密钥托管/审计演练。
|
||||
4. 已补[`orbit-admin retention inventory`](api/retention.md)只读盘点和[阶段报告](reports/phase-6-retention-inventory.md)，能区分过期幂等、确定终态、未知效果、未终结流程、已确认/未确认交付、已消费/未消费Inbox、事件和审计规模；它不执行归档或删除。后续仍需评审各类保留周期、归档格式、受保护记录核对，再执行备份恢复、兼容升级和六类容量负载。未确认交付、有效幂等身份、当前/历史执行证据和未终结流程不能被普通清理破坏。生产HA及RPO/RTO继续按实际拓扑验证，不缩减为本机多进程检查。
|
||||
5. 已补[`orbit-load`](api/load.md)有界受理负载工具：固定请求形状、并发上限、每请求超时、无自动重试、延迟百分位和有限错误分类；已完成一次隔离 MySQL + HTTP 小样本实测，证据见[负载报告](reports/phase-6-load.md)。标准 Go 1.27.1 隔离回归仍需在工具链恢复后复跑；该工具只提供受理层测量，真实 Worker/流程/数据库拓扑容量仍需独立报告。
|
||||
6. 已补[隔离备份恢复演练](api/backup-restore.md)：源库和恢复库均为一次性 MySQL 容器，逻辑备份恢复后核对迁移账本、核心记录数量、固定索引和零外键；证据见[恢复报告](reports/phase-6-backup-restore.md)。该演练不替代生产备份介质、复制切换、跨版本恢复或 RPO/RTO 验收。
|
||||
|
||||
### P3 功能范围核对
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
| `orbit-admin verify` 发布前结构核验 | 已实现只读迁移、零外键和 000026–000028 固定索引检查 | [运维核验](operations.md) |
|
||||
| `orbit-admin retention inventory` 保留清单 | 已实现带 UTC 截止时间的只读盘点；不执行归档、删除或权限变更 | [保留清单](retention.md) |
|
||||
| `sdk/go/cmd/orbit-load` 有界受理负载工具 | 已实现固定并发、无自动重试、延迟百分位和错误分类；只用于隔离环境 | [负载工具](load.md) |
|
||||
| `scripts/test-backup-restore.sh` 隔离备份恢复演练 | 已实现一次性 MySQL 逻辑备份、独立恢复、核心计数及零外键核验 | [备份恢复演练](backup-restore.md) |
|
||||
| `/v1/worker/connect` | P2 已实现并通过阶段验收 | [Worker 契约](worker.md)、[Schema](worker-schema.md) |
|
||||
| 任务相对延迟 `delay_ms` | 已实现数据库计时、原键恢复与实际执行例子 | [任务接口](tasks.md)、[延迟示例](../examples/delayed-task.md) |
|
||||
| 任务冻结策略与 Worker `retryable` | P3 安全重试已实现 | [安全重试](retries.md) |
|
||||
|
||||
20
docs/api/backup-restore.md
Normal file
20
docs/api/backup-restore.md
Normal file
@@ -0,0 +1,20 @@
|
||||
# 隔离备份恢复演练
|
||||
|
||||
`scripts/test-backup-restore.sh` 用两个一次性 MySQL 容器演练逻辑备份恢复：源库按当前迁移建库并写入通用应用、命名空间、凭据和队列，使用 MySQL `mysqldump` 导出，再恢复到独立目标库。脚本随后运行 `orbit-admin verify`，并比较关键核心表数量和目标库外键数量。
|
||||
|
||||
```sh
|
||||
DOCKER=/usr/local/bin/docker \
|
||||
ORBIT_TEST_MYSQL_IMAGE=mysql:8.4 \
|
||||
sh scripts/test-backup-restore.sh
|
||||
```
|
||||
|
||||
运行前需要已经构建当前版本的 `bin/orbit-migrate` 和 `bin/orbit-admin`；也可以通过 `ORBIT_MIGRATE_BIN`、`ORBIT_ADMIN_BIN` 指定绝对路径。脚本只连接自己创建的容器，使用随机 root 密码和随机映射端口；退出时删除容器和临时凭据文件。它不读取、覆盖或清理任何已有数据库。
|
||||
|
||||
演练检查以下结果：
|
||||
|
||||
- 迁移账本完整且 `dirty=false`，当前版本与二进制校验和一致。
|
||||
- 应用、命名空间、凭据、队列和迁移账本的核心记录数量在源库与恢复库相同。
|
||||
- 恢复库 Orbit 表的外键数量为零，固定观测索引完整。
|
||||
- `orbit-admin verify` 只读返回 `complete=true`；它不会修改恢复库或自动修复结构。
|
||||
|
||||
该演练验证逻辑备份文件能在同版本 MySQL 和同一迁移账本下恢复。它不能证明备份介质持久性、跨版本升级、跨故障域复制、生产密钥托管、外部业务副作用核对、RPO/RTO 或在线大库备份窗口；这些必须在目标部署环境单独验收。
|
||||
24
docs/reports/phase-6-backup-restore.md
Normal file
24
docs/reports/phase-6-backup-restore.md
Normal file
@@ -0,0 +1,24 @@
|
||||
# P6：隔离备份恢复演练
|
||||
|
||||
验证日期：2026-09-19。新增 `scripts/test-backup-restore.sh`，在两个一次性 MySQL 容器之间执行逻辑备份和恢复，恢复后用只读结构核验和核心记录计数确认结果。演练不连接已有数据库，也不修改生产数据。
|
||||
|
||||
## 实际命令与环境
|
||||
|
||||
```sh
|
||||
DOCKER=/usr/local/bin/docker ORBIT_TEST_MYSQL_IMAGE=mysql:8.4 \
|
||||
sh scripts/test-backup-restore.sh
|
||||
```
|
||||
|
||||
源库和恢复库分别使用随机 root 密码、随机映射端口和独立数据库名。源库运行当前 `bin/orbit-migrate`，再通过 `bin/orbit-admin bootstrap` 写入一个通用应用、命名空间、凭据和队列；`mysqldump --single-transaction --routines --triggers --events --hex-blob --no-tablespaces` 导出后，通过容器内 MySQL 客户端导入恢复库。退出清理两个容器和临时凭据文件。
|
||||
|
||||
## 实际结果
|
||||
|
||||
```json
|
||||
{"source_core_counts":"1,1,1,1,28","restore_core_counts":"1,1,1,1,28","restore_foreign_key_count":0,"verify":{"schema_version":28,"migration_count":28,"foreign_key_count":0,"required_index_count":6,"complete":true}}
|
||||
```
|
||||
|
||||
源库和恢复库的应用、命名空间、凭据、队列、迁移账本计数一致；恢复库通过迁移校验、6 个固定观测索引和零外键检查。备份文件中没有凭据原文，临时私密操作文件由脚本退出处理。
|
||||
|
||||
## 验收边界
|
||||
|
||||
本次证明同版本 MySQL 的一次逻辑备份可以恢复为可被当前二进制读取的数据库，并保持无外键结构。它没有证明备份介质持久性、备份窗口内的大库锁/IO影响、跨版本升级恢复、复制切换、跨故障域部署、生产密钥托管、外部业务副作用核对或 RPO/RTO；这些仍需目标生产拓扑中的独立演练。恢复库也没有被接入 Worker 或业务流量，避免把恢复演练误当成恢复后的生产接管证明。
|
||||
106
scripts/test-backup-restore.sh
Executable file
106
scripts/test-backup-restore.sh
Executable file
@@ -0,0 +1,106 @@
|
||||
#!/bin/sh
|
||||
# Rehearses a logical MySQL backup into a separate disposable database.
|
||||
# It never points at an existing database and does not modify a production target.
|
||||
set -eu
|
||||
|
||||
TASK_DOCKER=${DOCKER:-docker}
|
||||
TASK_MIGRATE=${ORBIT_MIGRATE_BIN:-bin/orbit-migrate}
|
||||
TASK_ADMIN=${ORBIT_ADMIN_BIN:-bin/orbit-admin}
|
||||
TASK_SOURCE_NAME="orbit-backup-source-$$"
|
||||
TASK_RESTORE_NAME="orbit-backup-restore-$$"
|
||||
TASK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/orbit-backup.XXXXXX")
|
||||
TASK_SOURCE_PID=
|
||||
TASK_RESTORE_PID=
|
||||
|
||||
cleanup() {
|
||||
"$TASK_DOCKER" rm -f "$TASK_SOURCE_NAME" "$TASK_RESTORE_NAME" >/dev/null 2>&1 || true
|
||||
for task_artifact in "$TASK_DIR"/*; do
|
||||
if [ -f "$task_artifact" ]; then unlink "$task_artifact"; fi
|
||||
done
|
||||
rmdir "$TASK_DIR" 2>/dev/null || true
|
||||
}
|
||||
trap cleanup EXIT HUP INT TERM
|
||||
|
||||
if [ ! -x "$TASK_MIGRATE" ] || [ ! -x "$TASK_ADMIN" ]; then
|
||||
printf '%s\n' 'orbit-migrate and orbit-admin binaries are required; build them before this rehearsal.' >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
umask 077
|
||||
TASK_SOURCE_PASSWORD=$(openssl rand -hex 24)
|
||||
TASK_RESTORE_PASSWORD=$(openssl rand -hex 24)
|
||||
|
||||
start_mysql() {
|
||||
container=$1
|
||||
database=$2
|
||||
password=$3
|
||||
"$TASK_DOCKER" run -d --name "$container" \
|
||||
-e MYSQL_ROOT_PASSWORD="$password" \
|
||||
-e MYSQL_DATABASE="$database" \
|
||||
-p 127.0.0.1::3306 "$ORBIT_TEST_MYSQL_IMAGE" >/dev/null
|
||||
ready=0
|
||||
for attempt in $(seq 1 60); do
|
||||
if "$TASK_DOCKER" exec -e MYSQL_PWD="$password" "$container" \
|
||||
mysql -uroot --protocol=tcp -e 'SELECT 1' >/dev/null 2>&1; then
|
||||
ready=1
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
if [ "$ready" -ne 1 ]; then
|
||||
printf 'MySQL container did not become ready: %s\n' "$container" >&2
|
||||
"$TASK_DOCKER" logs --tail 30 "$container" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
query() {
|
||||
container=$1
|
||||
password=$2
|
||||
database=$3
|
||||
sql=$4
|
||||
"$TASK_DOCKER" exec -e MYSQL_PWD="$password" "$container" \
|
||||
mysql -uroot --protocol=tcp --batch --skip-column-names "$database" -e "$sql"
|
||||
}
|
||||
|
||||
ORBIT_TEST_MYSQL_IMAGE=${ORBIT_TEST_MYSQL_IMAGE:-mysql:8.4}
|
||||
start_mysql "$TASK_SOURCE_NAME" orbit_source "$TASK_SOURCE_PASSWORD"
|
||||
start_mysql "$TASK_RESTORE_NAME" orbit_restore "$TASK_RESTORE_PASSWORD"
|
||||
|
||||
TASK_SOURCE_PORT=$("$TASK_DOCKER" port "$TASK_SOURCE_NAME" 3306/tcp | sed 's/.*://')
|
||||
TASK_RESTORE_PORT=$("$TASK_DOCKER" port "$TASK_RESTORE_NAME" 3306/tcp | sed 's/.*://')
|
||||
TASK_SOURCE_DSN="root:$TASK_SOURCE_PASSWORD@tcp(127.0.0.1:$TASK_SOURCE_PORT)/orbit_source"
|
||||
TASK_RESTORE_DSN="root:$TASK_RESTORE_PASSWORD@tcp(127.0.0.1:$TASK_RESTORE_PORT)/orbit_restore"
|
||||
|
||||
ORBIT_MYSQL_DSN="$TASK_SOURCE_DSN" "$TASK_MIGRATE" up >/dev/null
|
||||
ORBIT_MYSQL_DSN="$TASK_SOURCE_DSN" "$TASK_ADMIN" bootstrap \
|
||||
--application backup-rehearsal --namespace default --environment test \
|
||||
--queue restore-check --output "$TASK_DIR/credentials.json" >/dev/null
|
||||
|
||||
TASK_SOURCE_COUNTS=$(query "$TASK_SOURCE_NAME" "$TASK_SOURCE_PASSWORD" orbit_source \
|
||||
"SELECT CONCAT((SELECT COUNT(*) FROM orbit_applications),',',(SELECT COUNT(*) FROM orbit_namespaces),',',(SELECT COUNT(*) FROM orbit_credentials),',',(SELECT COUNT(*) FROM pulse_queues),',',(SELECT COUNT(*) FROM orbit_schema_migrations));")
|
||||
"$TASK_DOCKER" exec -e MYSQL_PWD="$TASK_SOURCE_PASSWORD" "$TASK_SOURCE_NAME" \
|
||||
mysqldump -uroot --protocol=tcp --single-transaction --routines --triggers --events \
|
||||
--hex-blob --no-tablespaces orbit_source > "$TASK_DIR/orbit.sql"
|
||||
|
||||
|
||||
"$TASK_DOCKER" exec -i -e MYSQL_PWD="$TASK_RESTORE_PASSWORD" "$TASK_RESTORE_NAME" \
|
||||
mysql -uroot --protocol=tcp orbit_restore < "$TASK_DIR/orbit.sql"
|
||||
|
||||
TASK_VERIFY=$(ORBIT_MYSQL_DSN="$TASK_RESTORE_DSN" "$TASK_ADMIN" verify)
|
||||
TASK_RESTORE_COUNTS=$(query "$TASK_RESTORE_NAME" "$TASK_RESTORE_PASSWORD" orbit_restore \
|
||||
"SELECT CONCAT((SELECT COUNT(*) FROM orbit_applications),',',(SELECT COUNT(*) FROM orbit_namespaces),',',(SELECT COUNT(*) FROM orbit_credentials),',',(SELECT COUNT(*) FROM pulse_queues),',',(SELECT COUNT(*) FROM orbit_schema_migrations));")
|
||||
TASK_FOREIGN_KEYS=$(query "$TASK_RESTORE_NAME" "$TASK_RESTORE_PASSWORD" information_schema \
|
||||
"SELECT COUNT(*) FROM REFERENTIAL_CONSTRAINTS WHERE CONSTRAINT_SCHEMA='orbit_restore' AND LEFT(TABLE_NAME,6) IN ('orbit_','pulse_','weave_');")
|
||||
|
||||
if [ "$TASK_SOURCE_COUNTS" != "$TASK_RESTORE_COUNTS" ]; then
|
||||
printf 'restored core counts differ: source=%s restore=%s\n' "$TASK_SOURCE_COUNTS" "$TASK_RESTORE_COUNTS" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [ "$TASK_FOREIGN_KEYS" != 0 ]; then
|
||||
printf 'restored schema has %s Orbit foreign keys\n' "$TASK_FOREIGN_KEYS" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf '{"source_core_counts":"%s","restore_core_counts":"%s","restore_foreign_key_count":%s,"verify":%s}\n' \
|
||||
"$TASK_SOURCE_COUNTS" "$TASK_RESTORE_COUNTS" "$TASK_FOREIGN_KEYS" "$TASK_VERIFY"
|
||||
Reference in New Issue
Block a user