feat: measure bounded worker execution with durable verification
This commit is contained in:
@@ -62,7 +62,7 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
||||
2. 已实现[基础运行观测](api/observability.md)及[有界业务状态与事件采样](reports/phase-6-business-observability.md):HTTP/WS、数据库连接池、后台批次、固定任务/交付/流程/Inbox状态计数、最近事件类别、终态耗时样本、固定事件阶段耗时、最早年龄、截断/新鲜度指标和可离线验证的告警规则;独立采集监听默认关闭,固定指标目录拒绝动态标签;发布前只读结构核验由[`orbit-admin verify`](api/operations.md)提供,运维人员也可用 `orbit-admin observe snapshot` 获取同一有界单节点快照。仍需端到端外部效果延迟、全局持久吞吐、Inbox从事件产生到推进完成的精确阶段延迟、全局发送队列、多节点聚合及生产采集/通知验证。
|
||||
3. 已补齐受保护管理环境中的固定凭据操作文件、轮换、撤销、幂等回执和脱敏管理审计;轮换保留授权绑定并要求新令牌重新建立会话,旧 Worker 会话被排空且立即过期,撤销允许在停用范围内执行且不改写历史记录。已用一次性 MySQL 定向验证旧令牌失效、新令牌重绑和零外键;仍需补齐提交不明故障切点、多节点安全重绑演练、凭据最小权限配置核对和生产密钥托管/审计演练。
|
||||
4. 已补[`orbit-admin retention inventory`](api/retention.md)只读盘点和[阶段报告](reports/phase-6-retention-inventory.md),能区分过期幂等、确定终态、未知效果、未终结流程、已确认/未确认交付、已消费/未消费Inbox、事件和审计规模;它不执行归档或删除。后续仍需评审各类保留周期、归档格式、受保护记录核对,再执行备份恢复、兼容升级和六类容量负载。未确认交付、有效幂等身份、当前/历史执行证据和未终结流程不能被普通清理破坏。生产HA及RPO/RTO继续按实际拓扑验证,不缩减为本机多进程检查。
|
||||
5. 已补[`orbit-load`](api/load.md)有界受理负载工具:固定请求形状、并发上限、每请求超时、无自动重试、延迟百分位和有限错误分类;已完成24次冒烟及1000次扩大隔离样本,证据见[负载报告](reports/phase-6-load.md)。标准 Go 1.27.1 隔离回归仍需在工具链恢复后复跑;该工具只提供受理层测量,真实 Worker/流程/数据库拓扑容量仍需独立报告。
|
||||
5. 已补[`orbit-load`](api/load.md)有界受理负载工具及24次/1000次隔离样本,见[受理报告](reports/phase-6-load.md)。新增独立[`orbit-exec-load`](api/execution-load.md),通过真实Worker、任务结果和尝试历史核对,提供短/混合/中长任务的受理、排队、授权执行及完成耗时分布;具体环境、原始JSON和失败分母见[执行测量报告](reports/phase-6-execution-load.md)。标准Go 1.27.1全量仍待复跑,下一步补定时集中触发、Weave流程推进、长期积压和资源曲线;目标数据库拓扑容量仍单独验收。
|
||||
6. 已补[隔离备份恢复演练](api/backup-restore.md):源库和恢复库均为一次性 MySQL 容器,逻辑备份恢复后核对迁移账本、核心记录数量、固定索引和零外键;证据见[恢复报告](reports/phase-6-backup-restore.md)。该演练不替代生产备份介质、复制切换、跨版本恢复或 RPO/RTO 验收。
|
||||
7. 已补[A18 数据库短暂不可用恢复演练](api/db-recovery.md):readiness 在数据库停止期间返回 503,同卷恢复后返回 200,并用原权限范围继续受理任务;证据见[数据库恢复报告](reports/phase-6-db-recovery.md)。该演练不替代主从切换、复制确认、跨故障域或 RPO/RTO 验收。
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@
|
||||
| `orbit-admin observe snapshot` 只读运行快照 | 已实现固定状态、事件窗口和阶段耗时的有界单节点快照 | [运维核验](operations.md) |
|
||||
| `orbit-admin retention inventory` 保留清单 | 已实现带 UTC 截止时间的只读盘点;不执行归档、删除或权限变更 | [保留清单](retention.md) |
|
||||
| `sdk/go/cmd/orbit-load` 有界受理负载工具 | 已实现固定并发、无自动重试、延迟百分位和错误分类;只用于隔离环境 | [负载工具](load.md) |
|
||||
| `sdk/go/cmd/orbit-exec-load` 有界任务执行测量 | 已实现真实Worker、固定混合时长、最终结果/尝试核验及阶段耗时分布 | [执行测量](execution-load.md) |
|
||||
| `scripts/test-backup-restore.sh` 隔离备份恢复演练 | 已实现一次性 MySQL 逻辑备份、独立恢复、核心计数及零外键核验 | [备份恢复演练](backup-restore.md) |
|
||||
| `scripts/test-db-recovery.sh` MySQL 短暂不可用恢复演练 | 已实现 readiness 503、同卷恢复 200 及恢复后任务受理核验 | [数据库恢复演练](db-recovery.md) |
|
||||
| `/v1/worker/connect` | P2 已实现并通过阶段验收 | [Worker 契约](worker.md)、[Schema](worker-schema.md) |
|
||||
|
||||
96
docs/api/execution-load.md
Normal file
96
docs/api/execution-load.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# 有界任务执行测量
|
||||
|
||||
`sdk/go/cmd/orbit-exec-load` 通过公共 Go SDK 运行 HTTP 提交 → WSS Worker 授权/执行/续租/结果确认 → HTTP 读取持久化状态/尝试历史的完整链路。它测量普通 Pulse 任务;`orbit-load` 继续只测受理,两个命令独立。
|
||||
|
||||
## 准备及运行
|
||||
|
||||
使用一次性数据库或专用测试命名空间。凭据需有 `queues:read`、`tasks:submit`、`tasks:read`、`workers:connect`、`workers:consume`,覆盖指定队列。开发测试可通过 `orbit-admin bootstrap ... --worker-access --output /private/execution.credentials.json` 初始化,完整参数见[鉴权手册](authentication.md)。
|
||||
|
||||
启动匹配版本的 Orbit 后,在SDK模块目录执行:
|
||||
|
||||
```sh
|
||||
cd sdk/go
|
||||
export ORBIT_URL='http://127.0.0.1:8080'
|
||||
export ORBIT_CREDENTIALS_FILE='/private/execution.credentials.json'
|
||||
|
||||
# 空处理器的框架路径
|
||||
go run ./cmd/orbit-exec-load \
|
||||
-requests 96 -concurrency 8 -workers 2 -capacity 4 \
|
||||
-durations 0ms -run-timeout 2m
|
||||
|
||||
# 按序号依次使用0ms、100ms、1s的混合任务
|
||||
go run ./cmd/orbit-exec-load \
|
||||
-requests 96 -concurrency 8 -workers 2 -capacity 4 \
|
||||
-durations 0ms,100ms,1s -run-timeout 2m
|
||||
|
||||
# 超过一个租约周期的任务,观察真实续租
|
||||
go run ./cmd/orbit-exec-load \
|
||||
-requests 6 -concurrency 3 -workers 2 -capacity 4 \
|
||||
-durations 35s -run-timeout 2m
|
||||
```
|
||||
|
||||
默认从凭据文件读取token和queue_id,也支持 `ORBIT_TOKEN`、`ORBIT_QUEUE_ID`。环境变量中非空token/queue覆盖文件值;`-queue` 和 `-url` 可覆盖对应环境变量。凭据不作为命令行参数或JSON结果输出。每个Worker均为负载进程中的独立WSS运行器,不是独立主机或故障域;Orbit服务进程保持独立。
|
||||
|
||||
工具读取并输出当前队列配置,不修改队列和应用额度。因此结果包含实际并发/开始速率限制的影响。运行前记录应用额度、服务连接池、数据库配置、CPU/内存限制及数据量,不能用Worker总capacity直接替代队列的有效许可上限。
|
||||
|
||||
## 参数与资源预算
|
||||
|
||||
| 参数 | 默认 | 约束 |
|
||||
| --- | --- | --- |
|
||||
| `-requests` | 100 | 1–10000,独立任务总数 |
|
||||
| `-concurrency` | 8 | 1–128,提交和后续核验两个阶段各自的HTTP并发上限;最多等于任务数 |
|
||||
| `-workers` | 2 | 1–32条独立WSS Worker连接 |
|
||||
| `-capacity` | 4 | 每Worker 1–256,总计最多1024个物理处理器槽 |
|
||||
| `-durations` | `0ms` | 1–16个逗号分隔时长,按序号取模;0–60秒的整毫秒 |
|
||||
| `-timeout` | 5秒 | 单次HTTP请求1毫秒–1分钟,不是任务时限 |
|
||||
| `-run-timeout` | 2分钟 | 整体1秒–1小时;必须比最大模拟时长至少长1秒,仍须为排队留余量 |
|
||||
| `-poll-interval` | 250毫秒 | 每个活跃核验器100毫秒–10秒的查询间隔 |
|
||||
|
||||
整个运行共用截止上下文,结束时所有Worker有界收尾,最多再等5秒。中断或截止停止继续提交/核验,不自动取消、删除已受理任务。未完成任务继续保留原预算和身份;报告失败后不能把它们当作全部未执行。
|
||||
|
||||
每个任务最多一次执行尝试,安全策略为 `RECONCILE_ONLY`。总时限及单次尝试时限等于 `run-timeout`,以数据库首次受理时间计时;客户端整体截止另从本次运行开始计时。提交HTTP不自动重试。Worker沿现有SDK语义重连、续租和恢复控制确认;未知执行不改成明确失败来凑满成功数。
|
||||
|
||||
## 固定身份与核验
|
||||
|
||||
随机 `run_id` 同时进入本次专属处理器名称 `load.exec.{run_id}`、业务键 `orbit-exec-{run_id}`、提交幂等键 `orbit-exec-{run_id}-{sequence六位十进制}` 及输入。序号从0开始,每项输入和预期结果都是 `{run_id, sequence, duration_ms}`。处理器只有可取消等待和回显,不调用文件、数据库或外部业务接口。
|
||||
|
||||
所有提交发起完后,有界核验器读取已确认的任务及尝试。每一项成功必须同时满足:
|
||||
|
||||
- 受理回执身份唯一,队列/处理器/版本/关联键/输入/尝试预算符合本次固定请求。
|
||||
- 任务为 `SUCCEEDED`、`effect_uncertain=false`、`attempt_count=1`,结果匹配预期输入。
|
||||
- 尝试历史恰好一项、没有后续页;属于原任务,代次1、`SUCCEEDED`,授权和结束时间齐全。
|
||||
- 数据库时间满足创建、可执行、领取、授权、结束顺序,任务完成时间与尝试结束时间一致。
|
||||
- 本地处理器对应原任务且该序号恰好调用一次;重复调用及错误输入留在错误计数中。
|
||||
|
||||
成功任务通过HTTP读取持久化事实确认;SDK的 `ReportsConfirmed`、事件通知或处理器返回nil不能替代核验。
|
||||
|
||||
## 报告与错误退出
|
||||
|
||||
标准输出是一行JSON,原始样本见[执行测量报告](../reports/phase-6-execution-load.md)。`complete=true` 才返回退出码0;缺少权限、提交失败/回执未知、读取失败、任务失败/核查、截止、Worker错误或核验不一致均产生不完整报告和非零退出码。
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `attempted` / `not_submitted` | 实际提交调用数 / 截止前未提交数 |
|
||||
| `accepted` / `admission_errors` | 已核验唯一回执数 / 明确拒绝或受理结果无法确认数;后者不等于数据库中没有任务 |
|
||||
| `verified_success` / `unverified` | 已受理项中最终通过完整核验数 / 其余项数 |
|
||||
| `handler_calls` / `duplicate_invocations` | 本地有效输入处理器调用数 / 同一序号多余调用数 |
|
||||
| `unconfirmed_handler_calls` | 处理器执行了但没有收到有效受理回执的调用数;整个运行仍失败 |
|
||||
| `worker_started` / `worker_reconnects` / `worker_renewals_confirmed` | SDK本进程辅助观测,不代表持久成功数 |
|
||||
| `errors` | 有限错误分类及数量,既有每项错误,也有运行级错误,不应求和为失败任务数 |
|
||||
|
||||
每个耗时分布返回 `count`、`p50_ms`、`p95_ms`、`p99_ms`、`max_ms`,采用最近秩。没有样本时百分位为null。分母分别是:
|
||||
|
||||
| 耗时 | 时钟与分母 |
|
||||
| --- | --- |
|
||||
| `admission` | 本机单调时钟;所有实际提交的HTTP交换,包含失败 |
|
||||
| `queue_wait` | 数据库 `available_at` → `authorized_at`;仅完整核验成功项,包含等待领取及开始授权 |
|
||||
| `authorized_execution` | 数据库 `authorized_at` → `ended_at`;仅完整核验成功项,包含开始确认传输、模拟处理、结果回传至结束事务取时点 |
|
||||
| `task_completion` | 数据库 `created_at` → `completed_at`;仅完整核验成功项 |
|
||||
|
||||
后面三个分布使用同一数据库时钟,避免核验轮询顺序污染执行耗时。时间戳记录事务内取时点,不能测量该取时点之后的COMMIT落盘及确认延迟;这些行最终已持久提交由后续HTTP核验确认。它们依赖数据库墙钟;时间逆序会报告 `TIMESTAMP_MISMATCH`。不完整运行的成功子集存在幸存者偏差,不能与完整样本直接比较。整次墙钟包含预检、读取和Worker收尾,不能拿 `requests / 整次耗时` 声称纯执行吞吐。
|
||||
|
||||
## 回归与生产边界
|
||||
|
||||
`make integration` 在隔离MySQL及真实服务上执行6项混合任务的命令冒烟;`make sdk-check` 检查独立模块并执行测量计算、失败/超时/错配及重复调用测试。
|
||||
|
||||
该工具新增普通任务短/混合/中长执行路径测量能力。当前仍需要定时集中触发、Weave流程、长期积压、慢消费者/重连风暴、目标硬件资源曲线、复制切换及跨故障域测试;当前样本不能替代这些生产门槛。
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
`orbit-load` 是独立 Go SDK module 中的受理负载工具,用于在隔离环境测量 HTTP 受理请求的延迟分布。它只提交带独立幂等键的任务,不启动 Worker、不执行业务副作用、不自动重试,也不把测量结果解释为生产吞吐或租户容量保证。
|
||||
|
||||
需要测量真实Worker及任务完成路径时,使用独立的[`orbit-exec-load`](execution-load.md)。
|
||||
|
||||
## 运行
|
||||
|
||||
先为专用测试命名空间准备凭据,再运行:
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
|
||||
| 项目 | 必须提供的证据 | 状态 |
|
||||
| --- | --- | --- |
|
||||
| 受理、派发、流程推进和恢复吞吐 | 固定硬件、数据量、任务/流程混合模型,p50/p95/p99和资源曲线 | 已有24次及1000次隔离受理样本;Worker/流程/积压/资源曲线和生产容量未验收 |
|
||||
| 受理、派发、流程推进和恢复吞吐 | 固定硬件、数据量、任务/流程混合模型,p50/p95/p99和资源曲线 | 已有受理样本及[真实Worker短/混合/中长执行样本](reports/phase-6-execution-load.md);定时/流程/积压/资源曲线和目标生产容量未验收 |
|
||||
| 100万+企业规模推导 | 租户、队列、连接、事件、索引和保存周期的容量模型 | 未验收,不能由租户数直接推导 |
|
||||
| MySQL 复制、切换、RPO/RTO | 目标拓扑上的实际切换、复制确认、恢复点和恢复时间 | 未验收 |
|
||||
| 慢消费者、重连风暴、磁盘和锁等待 | 受控故障负载及积压清空证据 | 未验收 |
|
||||
|
||||
1
docs/reports/data/phase-6-execution-load/deadline.json
Normal file
1
docs/reports/data/phase-6-execution-load/deadline.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"a7b51ef05252222cba0c3a458a2a330c","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:25:48.034952468Z","admission_finished_at":"2026-09-19T18:25:48.127721385Z","finished_at":"2026-09-19T18:25:50.044512177Z","requests":24,"concurrency":4,"workers":1,"capacity_per_worker":1,"durations_ms":[1000],"run_timeout_ms":2000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":24,"not_submitted":0,"accepted":24,"admission_errors":0,"verified_success":0,"unverified":24,"handler_calls":1,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":1,"worker_reconnects":0,"worker_renewals_confirmed":6,"admission":{"count":24,"p50_ms":12.970542,"p95_ms":20.052458,"p99_ms":22.639583,"max_ms":22.639583},"queue_wait":{"count":0,"p50_ms":null,"p95_ms":null,"p99_ms":null,"max_ms":null},"authorized_execution":{"count":0,"p50_ms":null,"p95_ms":null,"p99_ms":null,"max_ms":null},"task_completion":{"count":0,"p50_ms":null,"p95_ms":null,"p99_ms":null,"max_ms":null},"errors":{"INVOCATION_MISMATCH":23,"RUN_TIMEOUT":1,"TASK_NEEDS_REVIEW":1,"VERIFY_TIMEOUT":23},"complete":false}
|
||||
55
docs/reports/data/phase-6-execution-load/environment.json
Normal file
55
docs/reports/data/phase-6-execution-load/environment.json
Normal file
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"date_local": "2026-09-20",
|
||||
"timezone": "Asia/Shanghai",
|
||||
"test_go": "go1.25.14",
|
||||
"declared_go": "1.27.1",
|
||||
"temporary_modfile": true,
|
||||
"mysql": "8.4.10",
|
||||
"innodb_flush_log_at_trx_commit": 1,
|
||||
"sync_binlog": 1,
|
||||
"mysql_global_transaction_isolation": "REPEATABLE-READ",
|
||||
"orbit_transaction_isolation": "READ COMMITTED",
|
||||
"mysql_max_connections": 151,
|
||||
"innodb_buffer_pool_size_bytes": 134217728,
|
||||
"orbit_db_max_open": 20,
|
||||
"orbit_db_max_idle": 5,
|
||||
"docker_vm_cpus": 12,
|
||||
"docker_vm_memory_bytes": 12529274880,
|
||||
"architecture": "arm64",
|
||||
"orbit_processes": 1,
|
||||
"mysql_nodes": 1,
|
||||
"worker_processes": 1,
|
||||
"worker_connections_normal": 2,
|
||||
"worker_capacity_each": 4,
|
||||
"queue_max_concurrency": 4,
|
||||
"queue_rate_per_second": 10,
|
||||
"metrics_enabled": false,
|
||||
"cpu_memory_limits": "No dedicated limits; shared local Docker VM, other workloads possible",
|
||||
"profiles_in_order": [
|
||||
"short",
|
||||
"mixed",
|
||||
"long",
|
||||
"deadline"
|
||||
],
|
||||
"prior_task_counts": [
|
||||
0,
|
||||
96,
|
||||
192,
|
||||
198
|
||||
],
|
||||
"fresh_disposable_database": true,
|
||||
"network": "Go runner shares MySQL container network namespace; loopback HTTP/WSS and MySQL",
|
||||
"schema_verification": {
|
||||
"schema_version": 28,
|
||||
"migration_count": 28,
|
||||
"foreign_key_count": 0,
|
||||
"required_index_count": 6,
|
||||
"complete": true
|
||||
},
|
||||
"source_sha256": {
|
||||
"sdk/go/cmd/orbit-exec-load/main.go": "bcaecce6be9fe8aca7a0d93b4671cbc460b1083232a77d13a04642a96ae45d81",
|
||||
"sdk/go/cmd/orbit-exec-load/measure.go": "ea0a22e6598e12e175f281fd45d511db32e241e532f5bdcf40ca615392fa586e",
|
||||
"sdk/go/cmd/orbit-exec-load/measure_test.go": "bda2ce1e946745e149c7471462d511ad3bcf66352a8d14fab57b2aa1ef280c3d",
|
||||
"sdk/go/cmd/orbit-exec-load/report.go": "3d1da9acc1e63142bd3b4b37041d609e61e172eef56ff27467f1803dee857272"
|
||||
}
|
||||
}
|
||||
1
docs/reports/data/phase-6-execution-load/long.json
Normal file
1
docs/reports/data/phase-6-execution-load/long.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"980e0bccdca1cb9fbb1bdcc78536cc90","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:24:34.92203642Z","admission_finished_at":"2026-09-19T18:24:34.965149878Z","finished_at":"2026-09-19T18:25:48.023298384Z","requests":6,"concurrency":3,"workers":2,"capacity_per_worker":4,"durations_ms":[35000],"run_timeout_ms":120000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":6,"not_submitted":0,"accepted":6,"admission_errors":0,"verified_success":6,"unverified":0,"handler_calls":6,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":6,"worker_reconnects":0,"worker_renewals_confirmed":18,"admission":{"count":6,"p50_ms":13.193791,"p95_ms":28.028083,"p99_ms":28.028083,"max_ms":28.028083},"queue_wait":{"count":6,"p50_ms":1058.772,"p95_ms":38015.493,"p99_ms":38015.493,"max_ms":38015.493},"authorized_execution":{"count":6,"p50_ms":35013.932,"p95_ms":35024.585,"p99_ms":35024.585,"max_ms":35024.585},"task_completion":{"count":6,"p50_ms":36067.492,"p95_ms":73040.078,"p99_ms":73040.078,"max_ms":73040.078},"errors":{},"complete":true}
|
||||
1
docs/reports/data/phase-6-execution-load/mixed.json
Normal file
1
docs/reports/data/phase-6-execution-load/mixed.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"e78c366e6d6acb6cce8159dae900e8ca","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:23:52.454385095Z","admission_finished_at":"2026-09-19T18:23:52.759812553Z","finished_at":"2026-09-19T18:24:34.90756817Z","requests":96,"concurrency":8,"workers":2,"capacity_per_worker":4,"durations_ms":[0,100,1000],"run_timeout_ms":120000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":96,"not_submitted":0,"accepted":96,"admission_errors":0,"verified_success":96,"unverified":0,"handler_calls":96,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":96,"worker_reconnects":0,"worker_renewals_confirmed":0,"admission":{"count":96,"p50_ms":23.680834,"p95_ms":36.472167,"p99_ms":40.274167,"max_ms":40.274167},"queue_wait":{"count":96,"p50_ms":20883.756,"p95_ms":38754.793,"p99_ms":41024.973,"max_ms":41024.973},"authorized_execution":{"count":96,"p50_ms":109.912,"p95_ms":1021.918,"p99_ms":1042.773,"max_ms":1042.773},"task_completion":{"count":96,"p50_ms":20898.154,"p95_ms":39775.854,"p99_ms":42047.09,"max_ms":42047.09},"errors":{},"complete":true}
|
||||
1
docs/reports/data/phase-6-execution-load/short.json
Normal file
1
docs/reports/data/phase-6-execution-load/short.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"e16f23365b9accaef3ce8b41349d77ea","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:23:38.326433797Z","admission_finished_at":"2026-09-19T18:23:38.671806422Z","finished_at":"2026-09-19T18:23:52.44758097Z","requests":96,"concurrency":8,"workers":2,"capacity_per_worker":4,"durations_ms":[0],"run_timeout_ms":120000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":96,"not_submitted":0,"accepted":96,"admission_errors":0,"verified_success":96,"unverified":0,"handler_calls":96,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":96,"worker_reconnects":0,"worker_renewals_confirmed":0,"admission":{"count":96,"p50_ms":26.13175,"p95_ms":38.935083,"p99_ms":54.67775,"max_ms":54.67775},"queue_wait":{"count":96,"p50_ms":5951.689,"p95_ms":12805.411,"p99_ms":13752.092,"max_ms":13752.092},"authorized_execution":{"count":96,"p50_ms":10.405,"p95_ms":23.504,"p99_ms":28.838,"max_ms":28.838},"task_completion":{"count":96,"p50_ms":5959.781,"p95_ms":12813.065,"p99_ms":13768.327,"max_ms":13768.327},"errors":{},"complete":true}
|
||||
71
docs/reports/phase-6-execution-load.md
Normal file
71
docs/reports/phase-6-execution-load.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# P6:真实 Worker 任务执行测量
|
||||
|
||||
日期:2026-09-20(Asia/Shanghai,原始报告时间为UTC)。新增独立命令 `sdk/go/cmd/orbit-exec-load`,保留原 `orbit-load` 的纯受理边界。完整参数、权限、错误退出、分母和时钟定义见[执行测量手册](../api/execution-load.md)。本批交付短/混合/中长任务的可运行测量及正确性核验,不代表生产容量验收。
|
||||
|
||||
## 实现和核验
|
||||
|
||||
负载程序在业务侧独立SDK模块运行,与Orbit服务进程分离,只使用HTTP/WSS公共接口;没有数据库驱动或服务端内部依赖。每次使用独立处理器、固定序号/时长/输入、唯一提交键。HTTP提交不自动重试,任务最多一次尝试,Worker按现有协议心跳、续租、重连和收尾。
|
||||
|
||||
成功数量来自持久化任务及尝试历史的读取,逐项核验状态、输入/结果、队列/版本/业务键、单次尝试、授权/结束时间及单次处理器调用。SDK收到控制确认、处理器返回nil和本地事件都不能单独计为成功。截止或任一错配必须以 `complete=false` 和非零退出码结束;没有完成样本时百分位为null。
|
||||
|
||||
## 最终环境
|
||||
|
||||
原始配置及当前测量源码SHA-256保存在[环境清单](data/phase-6-execution-load/environment.json)。
|
||||
|
||||
| 项目 | 实际值 |
|
||||
| --- | --- |
|
||||
| 主机/运行环境 | macOS arm64、Docker Linux/arm64;共享VM可见12 CPU、12529274880字节内存,未独占硬件 |
|
||||
| 工具链 | 实际 `go1.25.14`,仅临时modfile替换Go指令;仓库仍声明Go 1.27.1 |
|
||||
| 拓扑 | 一次性MySQL 8.4.10,1个Orbit进程,1个负载进程内2条Worker连接,每条capacity=4 |
|
||||
| 网络 | Go运行容器共享MySQL容器网络命名空间;HTTP/WSS和MySQL均走loopback |
|
||||
| MySQL | `innodb_flush_log_at_trx_commit=1`、`sync_binlog=1`、buffer pool=128MiB、max_connections=151 |
|
||||
| 隔离级别 | MySQL全局REPEATABLE-READ;Orbit写事务显式READ COMMITTED |
|
||||
| Orbit | 数据库连接池max_open=20、max_idle=5;鉴权和持久事件/审计启用;metrics未启用 |
|
||||
| 队列 | max_concurrency=4、rate_per_second=10、max_unfinished_tasks=100000;工具未更改配置 |
|
||||
| 核验 | 每个HTTP请求5秒上限、单核验器250ms间隔,核验器最多与提交并发数相同 |
|
||||
| 数据量 | 空库初始化;short/mixed/long/deadline顺序运行,开始时已有任务数分别0/96/192/198;各运行身份隔离 |
|
||||
|
||||
最后 `orbit-admin verify` 返回 schema=28、migration_count=28、foreign_key_count=0、required_index_count=6、complete=true。测试容器退出后销毁一次性数据库和凭据。
|
||||
|
||||
## 原始结果
|
||||
|
||||
前三组均在2分钟整体预算内完整通过,退出0。三个阶段分布的分母分别为96、96、6;都没有重复处理器调用、无回执调用或Worker重连。
|
||||
|
||||
| 样本 | 任务/处理器时长 | 受理p99 ms | 排队p99 ms | 授权执行p99 ms | 任务完成p99 ms | 持久成功/计划数 |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| [short原始JSON](data/phase-6-execution-load/short.json) | 96项,0ms | 54.678 | 13752.092 | 28.838 | 13768.327 | 96/96 |
|
||||
| [mixed原始JSON](data/phase-6-execution-load/mixed.json) | 96项,0/100/1000ms循环,各32项 | 40.274 | 41024.973 | 1042.773 | 42047.090 | 96/96 |
|
||||
| [long原始JSON](data/phase-6-execution-load/long.json) | 6项,每项35000ms | 28.028 | 38015.493 | 35024.585 | 73040.078 | 6/6 |
|
||||
|
||||
长任务总计18次续租确认,最终仍为每项一次尝试、一次调用。该数字验证了此次实际续租路径,不表示节点/网络故障已经发生。
|
||||
|
||||
排队分布包含当前4并发/每秒10次开始的队列约束及调度扫描;Worker声明8个槽不会绕过队列限制。这组数字不能视为去掉配额后的最大吞吐。
|
||||
|
||||
`authorized_execution` 和 `task_completion` 用持久化数据库时间戳计算;终点是结束事务内部取时点,未包含其后COMMIT及确认延迟。已提交事实由后续读取核验。客户端核验顺序不计入这些分布,但查询负载会影响服务资源消耗。没有收集独占CPU/内存/磁盘/锁等待曲线,也没有多轮稳态和生产指标开启后的对照。
|
||||
|
||||
## 失败语义实测
|
||||
|
||||
[deadline原始JSON](data/phase-6-execution-load/deadline.json):24项、4并发提交、1个Worker/1个槽、每项1秒、整体预算2秒。命令按预期退出1,`accepted=24`、`verified_success=0`、`unverified=24`、`complete=false`;本地只有1次处理器调用。读取观察到1项NEEDS_REVIEW,23项未能在截止前验证,三种完成阶段的分布均为 `count=0` / 百分位null。
|
||||
|
||||
错误计数保留 `RUN_TIMEOUT`、`TASK_NEEDS_REVIEW`、`VERIFY_TIMEOUT` 和 `INVOCATION_MISMATCH`。这些计数含运行级及交叠的核验错误,不能求和当成失败任务总数。工具没有删除、强制成功或重新提交这些任务;未观察到终态不等于数据库确认失败,剩余状态留在原身份下供核查。
|
||||
|
||||
## 复现和回归
|
||||
|
||||
按[手册](../api/execution-load.md)准备一次性数据库、当前迁移、带Worker权限的凭据及真实Orbit进程,然后在 `sdk/go` 运行:
|
||||
|
||||
```sh
|
||||
go run ./cmd/orbit-exec-load -requests 96 -concurrency 8 -workers 2 -capacity 4 -durations 0ms -run-timeout 2m
|
||||
go run ./cmd/orbit-exec-load -requests 96 -concurrency 8 -workers 2 -capacity 4 -durations 0ms,100ms,1s -run-timeout 2m
|
||||
go run ./cmd/orbit-exec-load -requests 6 -concurrency 3 -workers 2 -capacity 4 -durations 35s -run-timeout 2m
|
||||
# 此项应失败;必须读取JSON确认是预算不足,不应仅判断任意非零退出。
|
||||
go run ./cmd/orbit-exec-load -requests 24 -concurrency 4 -workers 1 -capacity 1 -durations 1s -run-timeout 2s
|
||||
```
|
||||
|
||||
本轮在上述Docker环境使用临时modfile构建真实Linux二进制,执行了四条等价命令及下列检查:
|
||||
|
||||
- SDK全模块 `go test -race ./...`、`go vet ./...`、`go build ./...` 通过;没有设置原SDK集成凭据的普通单测不冒充全量集成。
|
||||
- 新增单测覆盖:错配/缺失/空字段结果、未知效果、第二次尝试、悬空历史、时间逆序、提交超时/429不重报、整次取消、核查不算成功、读取失败、重复调用、无权限预检及错误脱敏。
|
||||
- SDK完整依赖图218项中,服务端内部/模型/模块和MySQL驱动依赖均为0。
|
||||
- `scripts/test-mysql.sh` 已加入6项真实执行命令冒烟,shell语法检查通过。本轮运行的是四个独立真实样本,未重新执行整个 `make integration`。
|
||||
|
||||
当前不把临时Go 1.25兼容回归升级为声明Go 1.27.1的正式发布结论。下一批继续定时集中触发、流程推进、长期积压、慢消费者/重连风暴及资源曲线,生产HA、复制切换与RPO/RTO仍需目标拓扑证据。
|
||||
@@ -17,6 +17,7 @@
|
||||
| SDK Worker/Event/Workflow 子进程故障 | 同一共享网络命名空间;容器内构建 Orbit 二进制;SDK `TestIntegration(Worker\|Event\|Workflow)Process` | PASS,136.394 秒;派发未送达、租约接管、ACK 提交回执丢失、重试未知结果、计划/优先级/键并发、流程暂停全停接管均通过 |
|
||||
| SDK 应用配额子进程故障 | 独立双命名空间应用配额凭据;共享网络命名空间;`TestIntegrationWorkerProcessApplicationBackpressure` | PASS,1.069 秒;重启后共享额度、原回执和跨命名空间背压保持 |
|
||||
| Orbit HTTP/metrics 子进程 | 同一共享网络命名空间;`TestIntegrationMetricsProcess`;metrics 启用/禁用各一组 | PASS,3.081 秒;监听隔离、隐私、后台采样和优雅退出均通过 |
|
||||
| 真实Worker执行测量 | `orbit-exec-load`;最终短/混合/中长及截止不足四组,环境及原始JSON见[测量报告](phase-6-execution-load.md) | 96+96+6项全部逐项核验,长任务18次续租;24项截止负例按预期非零退出。SDK全模块竞态/vet/build通过,生产容量未验收 |
|
||||
|
||||
## 当前环境仍未覆盖
|
||||
|
||||
|
||||
Reference in New Issue
Block a user