feat: measure bounded worker execution with durable verification
This commit is contained in:
@@ -86,3 +86,4 @@
|
||||
- 进程指标、固定状态采样、格式验证、告警规则和标准回归证据见docs/reports/phase-6-observability.md及docs/reports/phase-6-business-observability.md。仍须补受理/授权/完成与流程推进延迟、持久吞吐/积压的全局聚合、重试/核查/恢复量、发送队列、实际采集告警及负载开销,不能把进程观测通过写成P6已完成。普通清理仍须保护有效幂等、未确认事件、未知结果和未终结流程。
|
||||
- 凭据轮换保留credential_id及订阅/Worker授权绑定,在同一事务替换摘要、固定到期时间、幂等回执和管理审计;新令牌必须重新建立会话,认证路径每次重新核对凭据状态。撤销必须让认证失效并保留原任务、订阅、交付、会话和审计证据,允许在应用或命名空间停用后执行;重复应用同一固定操作幂等返回。管理CLI先把固定操作文件写成0600,再提交;提交不明时保留文件并用原文件重放,不能重新生成秘密。日志、审计摘要、指标和业务表不得保存秘密原文或摘要。不要把轮换同一凭据ID解释成历史令牌仍有效,也不得级联删除订阅、任务或事件。
|
||||
- `sdk/go/cmd/orbit-load` 只用于隔离环境受理测量:每项使用稳定独立幂等键,固定请求形状和并发上限,按请求超时,不自动重试,不启动Worker或执行外部副作用;输出延迟百分位与有限错误分类,失败必须以非零状态返回。其结果不能替代真实Worker、流程推进、数据库拓扑或生产容量验收。
|
||||
- `sdk/go/cmd/orbit-exec-load` 独立测量真实Worker执行:每次运行固定专属处理器/业务键/序号/模拟时长,最多一次尝试且无外部副作用;任务成功须核对持久化结果、唯一尝试、时间顺序及单次本地调用。队列额度只读,提交不重试;整体截止和有界排空后输出实际分母,未完成/未知结果必须非零退出。数据库阶段耗时不得混用客户端轮询观察时间,不将隔离样本解释为生产容量。
|
||||
|
||||
@@ -82,6 +82,8 @@ make protocol-check
|
||||
|
||||
受理层负载采样工具见 [有界受理负载工具](docs/api/load.md),只允许在隔离命名空间或一次性数据库中运行;它不自动重试、不启动 Worker,也不提供生产容量承诺。
|
||||
|
||||
真实Worker与任务完成路径测量使用[`orbit-exec-load`](docs/api/execution-load.md):短/混合/中长模拟任务、持久结果和单次尝试核验、阶段延迟分布;不修改队列额度、不调用外部业务系统。
|
||||
|
||||
SDK 独立发布门禁使用 `make sdk-check`,在 `GOWORK=off` 下检查完整依赖图,并运行 SDK 的竞态测试、vet 和全包构建;规则和接入边界见 [SDK 发布门禁](docs/sdk/release.md)。
|
||||
|
||||
修改已应用迁移时会触发校验和拒绝;新增结构使用新的前向迁移。初始结构尚未部署生产。
|
||||
|
||||
@@ -62,7 +62,7 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
||||
2. 已实现[基础运行观测](api/observability.md)及[有界业务状态与事件采样](reports/phase-6-business-observability.md):HTTP/WS、数据库连接池、后台批次、固定任务/交付/流程/Inbox状态计数、最近事件类别、终态耗时样本、固定事件阶段耗时、最早年龄、截断/新鲜度指标和可离线验证的告警规则;独立采集监听默认关闭,固定指标目录拒绝动态标签;发布前只读结构核验由[`orbit-admin verify`](api/operations.md)提供,运维人员也可用 `orbit-admin observe snapshot` 获取同一有界单节点快照。仍需端到端外部效果延迟、全局持久吞吐、Inbox从事件产生到推进完成的精确阶段延迟、全局发送队列、多节点聚合及生产采集/通知验证。
|
||||
3. 已补齐受保护管理环境中的固定凭据操作文件、轮换、撤销、幂等回执和脱敏管理审计;轮换保留授权绑定并要求新令牌重新建立会话,旧 Worker 会话被排空且立即过期,撤销允许在停用范围内执行且不改写历史记录。已用一次性 MySQL 定向验证旧令牌失效、新令牌重绑和零外键;仍需补齐提交不明故障切点、多节点安全重绑演练、凭据最小权限配置核对和生产密钥托管/审计演练。
|
||||
4. 已补[`orbit-admin retention inventory`](api/retention.md)只读盘点和[阶段报告](reports/phase-6-retention-inventory.md),能区分过期幂等、确定终态、未知效果、未终结流程、已确认/未确认交付、已消费/未消费Inbox、事件和审计规模;它不执行归档或删除。后续仍需评审各类保留周期、归档格式、受保护记录核对,再执行备份恢复、兼容升级和六类容量负载。未确认交付、有效幂等身份、当前/历史执行证据和未终结流程不能被普通清理破坏。生产HA及RPO/RTO继续按实际拓扑验证,不缩减为本机多进程检查。
|
||||
5. 已补[`orbit-load`](api/load.md)有界受理负载工具:固定请求形状、并发上限、每请求超时、无自动重试、延迟百分位和有限错误分类;已完成24次冒烟及1000次扩大隔离样本,证据见[负载报告](reports/phase-6-load.md)。标准 Go 1.27.1 隔离回归仍需在工具链恢复后复跑;该工具只提供受理层测量,真实 Worker/流程/数据库拓扑容量仍需独立报告。
|
||||
5. 已补[`orbit-load`](api/load.md)有界受理负载工具及24次/1000次隔离样本,见[受理报告](reports/phase-6-load.md)。新增独立[`orbit-exec-load`](api/execution-load.md),通过真实Worker、任务结果和尝试历史核对,提供短/混合/中长任务的受理、排队、授权执行及完成耗时分布;具体环境、原始JSON和失败分母见[执行测量报告](reports/phase-6-execution-load.md)。标准Go 1.27.1全量仍待复跑,下一步补定时集中触发、Weave流程推进、长期积压和资源曲线;目标数据库拓扑容量仍单独验收。
|
||||
6. 已补[隔离备份恢复演练](api/backup-restore.md):源库和恢复库均为一次性 MySQL 容器,逻辑备份恢复后核对迁移账本、核心记录数量、固定索引和零外键;证据见[恢复报告](reports/phase-6-backup-restore.md)。该演练不替代生产备份介质、复制切换、跨版本恢复或 RPO/RTO 验收。
|
||||
7. 已补[A18 数据库短暂不可用恢复演练](api/db-recovery.md):readiness 在数据库停止期间返回 503,同卷恢复后返回 200,并用原权限范围继续受理任务;证据见[数据库恢复报告](reports/phase-6-db-recovery.md)。该演练不替代主从切换、复制确认、跨故障域或 RPO/RTO 验收。
|
||||
|
||||
|
||||
@@ -24,6 +24,7 @@
|
||||
| `orbit-admin observe snapshot` 只读运行快照 | 已实现固定状态、事件窗口和阶段耗时的有界单节点快照 | [运维核验](operations.md) |
|
||||
| `orbit-admin retention inventory` 保留清单 | 已实现带 UTC 截止时间的只读盘点;不执行归档、删除或权限变更 | [保留清单](retention.md) |
|
||||
| `sdk/go/cmd/orbit-load` 有界受理负载工具 | 已实现固定并发、无自动重试、延迟百分位和错误分类;只用于隔离环境 | [负载工具](load.md) |
|
||||
| `sdk/go/cmd/orbit-exec-load` 有界任务执行测量 | 已实现真实Worker、固定混合时长、最终结果/尝试核验及阶段耗时分布 | [执行测量](execution-load.md) |
|
||||
| `scripts/test-backup-restore.sh` 隔离备份恢复演练 | 已实现一次性 MySQL 逻辑备份、独立恢复、核心计数及零外键核验 | [备份恢复演练](backup-restore.md) |
|
||||
| `scripts/test-db-recovery.sh` MySQL 短暂不可用恢复演练 | 已实现 readiness 503、同卷恢复 200 及恢复后任务受理核验 | [数据库恢复演练](db-recovery.md) |
|
||||
| `/v1/worker/connect` | P2 已实现并通过阶段验收 | [Worker 契约](worker.md)、[Schema](worker-schema.md) |
|
||||
|
||||
96
docs/api/execution-load.md
Normal file
96
docs/api/execution-load.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# 有界任务执行测量
|
||||
|
||||
`sdk/go/cmd/orbit-exec-load` 通过公共 Go SDK 运行 HTTP 提交 → WSS Worker 授权/执行/续租/结果确认 → HTTP 读取持久化状态/尝试历史的完整链路。它测量普通 Pulse 任务;`orbit-load` 继续只测受理,两个命令独立。
|
||||
|
||||
## 准备及运行
|
||||
|
||||
使用一次性数据库或专用测试命名空间。凭据需有 `queues:read`、`tasks:submit`、`tasks:read`、`workers:connect`、`workers:consume`,覆盖指定队列。开发测试可通过 `orbit-admin bootstrap ... --worker-access --output /private/execution.credentials.json` 初始化,完整参数见[鉴权手册](authentication.md)。
|
||||
|
||||
启动匹配版本的 Orbit 后,在SDK模块目录执行:
|
||||
|
||||
```sh
|
||||
cd sdk/go
|
||||
export ORBIT_URL='http://127.0.0.1:8080'
|
||||
export ORBIT_CREDENTIALS_FILE='/private/execution.credentials.json'
|
||||
|
||||
# 空处理器的框架路径
|
||||
go run ./cmd/orbit-exec-load \
|
||||
-requests 96 -concurrency 8 -workers 2 -capacity 4 \
|
||||
-durations 0ms -run-timeout 2m
|
||||
|
||||
# 按序号依次使用0ms、100ms、1s的混合任务
|
||||
go run ./cmd/orbit-exec-load \
|
||||
-requests 96 -concurrency 8 -workers 2 -capacity 4 \
|
||||
-durations 0ms,100ms,1s -run-timeout 2m
|
||||
|
||||
# 超过一个租约周期的任务,观察真实续租
|
||||
go run ./cmd/orbit-exec-load \
|
||||
-requests 6 -concurrency 3 -workers 2 -capacity 4 \
|
||||
-durations 35s -run-timeout 2m
|
||||
```
|
||||
|
||||
默认从凭据文件读取token和queue_id,也支持 `ORBIT_TOKEN`、`ORBIT_QUEUE_ID`。环境变量中非空token/queue覆盖文件值;`-queue` 和 `-url` 可覆盖对应环境变量。凭据不作为命令行参数或JSON结果输出。每个Worker均为负载进程中的独立WSS运行器,不是独立主机或故障域;Orbit服务进程保持独立。
|
||||
|
||||
工具读取并输出当前队列配置,不修改队列和应用额度。因此结果包含实际并发/开始速率限制的影响。运行前记录应用额度、服务连接池、数据库配置、CPU/内存限制及数据量,不能用Worker总capacity直接替代队列的有效许可上限。
|
||||
|
||||
## 参数与资源预算
|
||||
|
||||
| 参数 | 默认 | 约束 |
|
||||
| --- | --- | --- |
|
||||
| `-requests` | 100 | 1–10000,独立任务总数 |
|
||||
| `-concurrency` | 8 | 1–128,提交和后续核验两个阶段各自的HTTP并发上限;最多等于任务数 |
|
||||
| `-workers` | 2 | 1–32条独立WSS Worker连接 |
|
||||
| `-capacity` | 4 | 每Worker 1–256,总计最多1024个物理处理器槽 |
|
||||
| `-durations` | `0ms` | 1–16个逗号分隔时长,按序号取模;0–60秒的整毫秒 |
|
||||
| `-timeout` | 5秒 | 单次HTTP请求1毫秒–1分钟,不是任务时限 |
|
||||
| `-run-timeout` | 2分钟 | 整体1秒–1小时;必须比最大模拟时长至少长1秒,仍须为排队留余量 |
|
||||
| `-poll-interval` | 250毫秒 | 每个活跃核验器100毫秒–10秒的查询间隔 |
|
||||
|
||||
整个运行共用截止上下文,结束时所有Worker有界收尾,最多再等5秒。中断或截止停止继续提交/核验,不自动取消、删除已受理任务。未完成任务继续保留原预算和身份;报告失败后不能把它们当作全部未执行。
|
||||
|
||||
每个任务最多一次执行尝试,安全策略为 `RECONCILE_ONLY`。总时限及单次尝试时限等于 `run-timeout`,以数据库首次受理时间计时;客户端整体截止另从本次运行开始计时。提交HTTP不自动重试。Worker沿现有SDK语义重连、续租和恢复控制确认;未知执行不改成明确失败来凑满成功数。
|
||||
|
||||
## 固定身份与核验
|
||||
|
||||
随机 `run_id` 同时进入本次专属处理器名称 `load.exec.{run_id}`、业务键 `orbit-exec-{run_id}`、提交幂等键 `orbit-exec-{run_id}-{sequence六位十进制}` 及输入。序号从0开始,每项输入和预期结果都是 `{run_id, sequence, duration_ms}`。处理器只有可取消等待和回显,不调用文件、数据库或外部业务接口。
|
||||
|
||||
所有提交发起完后,有界核验器读取已确认的任务及尝试。每一项成功必须同时满足:
|
||||
|
||||
- 受理回执身份唯一,队列/处理器/版本/关联键/输入/尝试预算符合本次固定请求。
|
||||
- 任务为 `SUCCEEDED`、`effect_uncertain=false`、`attempt_count=1`,结果匹配预期输入。
|
||||
- 尝试历史恰好一项、没有后续页;属于原任务,代次1、`SUCCEEDED`,授权和结束时间齐全。
|
||||
- 数据库时间满足创建、可执行、领取、授权、结束顺序,任务完成时间与尝试结束时间一致。
|
||||
- 本地处理器对应原任务且该序号恰好调用一次;重复调用及错误输入留在错误计数中。
|
||||
|
||||
成功任务通过HTTP读取持久化事实确认;SDK的 `ReportsConfirmed`、事件通知或处理器返回nil不能替代核验。
|
||||
|
||||
## 报告与错误退出
|
||||
|
||||
标准输出是一行JSON,原始样本见[执行测量报告](../reports/phase-6-execution-load.md)。`complete=true` 才返回退出码0;缺少权限、提交失败/回执未知、读取失败、任务失败/核查、截止、Worker错误或核验不一致均产生不完整报告和非零退出码。
|
||||
|
||||
| 字段 | 含义 |
|
||||
| --- | --- |
|
||||
| `attempted` / `not_submitted` | 实际提交调用数 / 截止前未提交数 |
|
||||
| `accepted` / `admission_errors` | 已核验唯一回执数 / 明确拒绝或受理结果无法确认数;后者不等于数据库中没有任务 |
|
||||
| `verified_success` / `unverified` | 已受理项中最终通过完整核验数 / 其余项数 |
|
||||
| `handler_calls` / `duplicate_invocations` | 本地有效输入处理器调用数 / 同一序号多余调用数 |
|
||||
| `unconfirmed_handler_calls` | 处理器执行了但没有收到有效受理回执的调用数;整个运行仍失败 |
|
||||
| `worker_started` / `worker_reconnects` / `worker_renewals_confirmed` | SDK本进程辅助观测,不代表持久成功数 |
|
||||
| `errors` | 有限错误分类及数量,既有每项错误,也有运行级错误,不应求和为失败任务数 |
|
||||
|
||||
每个耗时分布返回 `count`、`p50_ms`、`p95_ms`、`p99_ms`、`max_ms`,采用最近秩。没有样本时百分位为null。分母分别是:
|
||||
|
||||
| 耗时 | 时钟与分母 |
|
||||
| --- | --- |
|
||||
| `admission` | 本机单调时钟;所有实际提交的HTTP交换,包含失败 |
|
||||
| `queue_wait` | 数据库 `available_at` → `authorized_at`;仅完整核验成功项,包含等待领取及开始授权 |
|
||||
| `authorized_execution` | 数据库 `authorized_at` → `ended_at`;仅完整核验成功项,包含开始确认传输、模拟处理、结果回传至结束事务取时点 |
|
||||
| `task_completion` | 数据库 `created_at` → `completed_at`;仅完整核验成功项 |
|
||||
|
||||
后面三个分布使用同一数据库时钟,避免核验轮询顺序污染执行耗时。时间戳记录事务内取时点,不能测量该取时点之后的COMMIT落盘及确认延迟;这些行最终已持久提交由后续HTTP核验确认。它们依赖数据库墙钟;时间逆序会报告 `TIMESTAMP_MISMATCH`。不完整运行的成功子集存在幸存者偏差,不能与完整样本直接比较。整次墙钟包含预检、读取和Worker收尾,不能拿 `requests / 整次耗时` 声称纯执行吞吐。
|
||||
|
||||
## 回归与生产边界
|
||||
|
||||
`make integration` 在隔离MySQL及真实服务上执行6项混合任务的命令冒烟;`make sdk-check` 检查独立模块并执行测量计算、失败/超时/错配及重复调用测试。
|
||||
|
||||
该工具新增普通任务短/混合/中长执行路径测量能力。当前仍需要定时集中触发、Weave流程、长期积压、慢消费者/重连风暴、目标硬件资源曲线、复制切换及跨故障域测试;当前样本不能替代这些生产门槛。
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
`orbit-load` 是独立 Go SDK module 中的受理负载工具,用于在隔离环境测量 HTTP 受理请求的延迟分布。它只提交带独立幂等键的任务,不启动 Worker、不执行业务副作用、不自动重试,也不把测量结果解释为生产吞吐或租户容量保证。
|
||||
|
||||
需要测量真实Worker及任务完成路径时,使用独立的[`orbit-exec-load`](execution-load.md)。
|
||||
|
||||
## 运行
|
||||
|
||||
先为专用测试命名空间准备凭据,再运行:
|
||||
|
||||
@@ -38,7 +38,7 @@
|
||||
|
||||
| 项目 | 必须提供的证据 | 状态 |
|
||||
| --- | --- | --- |
|
||||
| 受理、派发、流程推进和恢复吞吐 | 固定硬件、数据量、任务/流程混合模型,p50/p95/p99和资源曲线 | 已有24次及1000次隔离受理样本;Worker/流程/积压/资源曲线和生产容量未验收 |
|
||||
| 受理、派发、流程推进和恢复吞吐 | 固定硬件、数据量、任务/流程混合模型,p50/p95/p99和资源曲线 | 已有受理样本及[真实Worker短/混合/中长执行样本](reports/phase-6-execution-load.md);定时/流程/积压/资源曲线和目标生产容量未验收 |
|
||||
| 100万+企业规模推导 | 租户、队列、连接、事件、索引和保存周期的容量模型 | 未验收,不能由租户数直接推导 |
|
||||
| MySQL 复制、切换、RPO/RTO | 目标拓扑上的实际切换、复制确认、恢复点和恢复时间 | 未验收 |
|
||||
| 慢消费者、重连风暴、磁盘和锁等待 | 受控故障负载及积压清空证据 | 未验收 |
|
||||
|
||||
1
docs/reports/data/phase-6-execution-load/deadline.json
Normal file
1
docs/reports/data/phase-6-execution-load/deadline.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"a7b51ef05252222cba0c3a458a2a330c","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:25:48.034952468Z","admission_finished_at":"2026-09-19T18:25:48.127721385Z","finished_at":"2026-09-19T18:25:50.044512177Z","requests":24,"concurrency":4,"workers":1,"capacity_per_worker":1,"durations_ms":[1000],"run_timeout_ms":2000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":24,"not_submitted":0,"accepted":24,"admission_errors":0,"verified_success":0,"unverified":24,"handler_calls":1,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":1,"worker_reconnects":0,"worker_renewals_confirmed":6,"admission":{"count":24,"p50_ms":12.970542,"p95_ms":20.052458,"p99_ms":22.639583,"max_ms":22.639583},"queue_wait":{"count":0,"p50_ms":null,"p95_ms":null,"p99_ms":null,"max_ms":null},"authorized_execution":{"count":0,"p50_ms":null,"p95_ms":null,"p99_ms":null,"max_ms":null},"task_completion":{"count":0,"p50_ms":null,"p95_ms":null,"p99_ms":null,"max_ms":null},"errors":{"INVOCATION_MISMATCH":23,"RUN_TIMEOUT":1,"TASK_NEEDS_REVIEW":1,"VERIFY_TIMEOUT":23},"complete":false}
|
||||
55
docs/reports/data/phase-6-execution-load/environment.json
Normal file
55
docs/reports/data/phase-6-execution-load/environment.json
Normal file
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"date_local": "2026-09-20",
|
||||
"timezone": "Asia/Shanghai",
|
||||
"test_go": "go1.25.14",
|
||||
"declared_go": "1.27.1",
|
||||
"temporary_modfile": true,
|
||||
"mysql": "8.4.10",
|
||||
"innodb_flush_log_at_trx_commit": 1,
|
||||
"sync_binlog": 1,
|
||||
"mysql_global_transaction_isolation": "REPEATABLE-READ",
|
||||
"orbit_transaction_isolation": "READ COMMITTED",
|
||||
"mysql_max_connections": 151,
|
||||
"innodb_buffer_pool_size_bytes": 134217728,
|
||||
"orbit_db_max_open": 20,
|
||||
"orbit_db_max_idle": 5,
|
||||
"docker_vm_cpus": 12,
|
||||
"docker_vm_memory_bytes": 12529274880,
|
||||
"architecture": "arm64",
|
||||
"orbit_processes": 1,
|
||||
"mysql_nodes": 1,
|
||||
"worker_processes": 1,
|
||||
"worker_connections_normal": 2,
|
||||
"worker_capacity_each": 4,
|
||||
"queue_max_concurrency": 4,
|
||||
"queue_rate_per_second": 10,
|
||||
"metrics_enabled": false,
|
||||
"cpu_memory_limits": "No dedicated limits; shared local Docker VM, other workloads possible",
|
||||
"profiles_in_order": [
|
||||
"short",
|
||||
"mixed",
|
||||
"long",
|
||||
"deadline"
|
||||
],
|
||||
"prior_task_counts": [
|
||||
0,
|
||||
96,
|
||||
192,
|
||||
198
|
||||
],
|
||||
"fresh_disposable_database": true,
|
||||
"network": "Go runner shares MySQL container network namespace; loopback HTTP/WSS and MySQL",
|
||||
"schema_verification": {
|
||||
"schema_version": 28,
|
||||
"migration_count": 28,
|
||||
"foreign_key_count": 0,
|
||||
"required_index_count": 6,
|
||||
"complete": true
|
||||
},
|
||||
"source_sha256": {
|
||||
"sdk/go/cmd/orbit-exec-load/main.go": "bcaecce6be9fe8aca7a0d93b4671cbc460b1083232a77d13a04642a96ae45d81",
|
||||
"sdk/go/cmd/orbit-exec-load/measure.go": "ea0a22e6598e12e175f281fd45d511db32e241e532f5bdcf40ca615392fa586e",
|
||||
"sdk/go/cmd/orbit-exec-load/measure_test.go": "bda2ce1e946745e149c7471462d511ad3bcf66352a8d14fab57b2aa1ef280c3d",
|
||||
"sdk/go/cmd/orbit-exec-load/report.go": "3d1da9acc1e63142bd3b4b37041d609e61e172eef56ff27467f1803dee857272"
|
||||
}
|
||||
}
|
||||
1
docs/reports/data/phase-6-execution-load/long.json
Normal file
1
docs/reports/data/phase-6-execution-load/long.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"980e0bccdca1cb9fbb1bdcc78536cc90","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:24:34.92203642Z","admission_finished_at":"2026-09-19T18:24:34.965149878Z","finished_at":"2026-09-19T18:25:48.023298384Z","requests":6,"concurrency":3,"workers":2,"capacity_per_worker":4,"durations_ms":[35000],"run_timeout_ms":120000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":6,"not_submitted":0,"accepted":6,"admission_errors":0,"verified_success":6,"unverified":0,"handler_calls":6,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":6,"worker_reconnects":0,"worker_renewals_confirmed":18,"admission":{"count":6,"p50_ms":13.193791,"p95_ms":28.028083,"p99_ms":28.028083,"max_ms":28.028083},"queue_wait":{"count":6,"p50_ms":1058.772,"p95_ms":38015.493,"p99_ms":38015.493,"max_ms":38015.493},"authorized_execution":{"count":6,"p50_ms":35013.932,"p95_ms":35024.585,"p99_ms":35024.585,"max_ms":35024.585},"task_completion":{"count":6,"p50_ms":36067.492,"p95_ms":73040.078,"p99_ms":73040.078,"max_ms":73040.078},"errors":{},"complete":true}
|
||||
1
docs/reports/data/phase-6-execution-load/mixed.json
Normal file
1
docs/reports/data/phase-6-execution-load/mixed.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"e78c366e6d6acb6cce8159dae900e8ca","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:23:52.454385095Z","admission_finished_at":"2026-09-19T18:23:52.759812553Z","finished_at":"2026-09-19T18:24:34.90756817Z","requests":96,"concurrency":8,"workers":2,"capacity_per_worker":4,"durations_ms":[0,100,1000],"run_timeout_ms":120000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":96,"not_submitted":0,"accepted":96,"admission_errors":0,"verified_success":96,"unverified":0,"handler_calls":96,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":96,"worker_reconnects":0,"worker_renewals_confirmed":0,"admission":{"count":96,"p50_ms":23.680834,"p95_ms":36.472167,"p99_ms":40.274167,"max_ms":40.274167},"queue_wait":{"count":96,"p50_ms":20883.756,"p95_ms":38754.793,"p99_ms":41024.973,"max_ms":41024.973},"authorized_execution":{"count":96,"p50_ms":109.912,"p95_ms":1021.918,"p99_ms":1042.773,"max_ms":1042.773},"task_completion":{"count":96,"p50_ms":20898.154,"p95_ms":39775.854,"p99_ms":42047.09,"max_ms":42047.09},"errors":{},"complete":true}
|
||||
1
docs/reports/data/phase-6-execution-load/short.json
Normal file
1
docs/reports/data/phase-6-execution-load/short.json
Normal file
@@ -0,0 +1 @@
|
||||
{"run_id":"e16f23365b9accaef3ce8b41349d77ea","go_version":"go1.25.14","goos":"linux","goarch":"arm64","started_at":"2026-09-19T18:23:38.326433797Z","admission_finished_at":"2026-09-19T18:23:38.671806422Z","finished_at":"2026-09-19T18:23:52.44758097Z","requests":96,"concurrency":8,"workers":2,"capacity_per_worker":4,"durations_ms":[0],"run_timeout_ms":120000,"poll_interval_ms":250,"http_timeout_ms":5000,"queue":{"max_unfinished_tasks":100000,"unfinished_tasks":"0","id":"61e5494b4cbf4c7126454a074a56412d","namespace_id":"68983a8088c43ef337aa69bbd349686e","name":"execution","paused":false,"max_concurrency":4,"rate_per_second":10,"revision":"1"},"attempted":96,"not_submitted":0,"accepted":96,"admission_errors":0,"verified_success":96,"unverified":0,"handler_calls":96,"duplicate_invocations":0,"unconfirmed_handler_calls":0,"worker_started":96,"worker_reconnects":0,"worker_renewals_confirmed":0,"admission":{"count":96,"p50_ms":26.13175,"p95_ms":38.935083,"p99_ms":54.67775,"max_ms":54.67775},"queue_wait":{"count":96,"p50_ms":5951.689,"p95_ms":12805.411,"p99_ms":13752.092,"max_ms":13752.092},"authorized_execution":{"count":96,"p50_ms":10.405,"p95_ms":23.504,"p99_ms":28.838,"max_ms":28.838},"task_completion":{"count":96,"p50_ms":5959.781,"p95_ms":12813.065,"p99_ms":13768.327,"max_ms":13768.327},"errors":{},"complete":true}
|
||||
71
docs/reports/phase-6-execution-load.md
Normal file
71
docs/reports/phase-6-execution-load.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# P6:真实 Worker 任务执行测量
|
||||
|
||||
日期:2026-09-20(Asia/Shanghai,原始报告时间为UTC)。新增独立命令 `sdk/go/cmd/orbit-exec-load`,保留原 `orbit-load` 的纯受理边界。完整参数、权限、错误退出、分母和时钟定义见[执行测量手册](../api/execution-load.md)。本批交付短/混合/中长任务的可运行测量及正确性核验,不代表生产容量验收。
|
||||
|
||||
## 实现和核验
|
||||
|
||||
负载程序在业务侧独立SDK模块运行,与Orbit服务进程分离,只使用HTTP/WSS公共接口;没有数据库驱动或服务端内部依赖。每次使用独立处理器、固定序号/时长/输入、唯一提交键。HTTP提交不自动重试,任务最多一次尝试,Worker按现有协议心跳、续租、重连和收尾。
|
||||
|
||||
成功数量来自持久化任务及尝试历史的读取,逐项核验状态、输入/结果、队列/版本/业务键、单次尝试、授权/结束时间及单次处理器调用。SDK收到控制确认、处理器返回nil和本地事件都不能单独计为成功。截止或任一错配必须以 `complete=false` 和非零退出码结束;没有完成样本时百分位为null。
|
||||
|
||||
## 最终环境
|
||||
|
||||
原始配置及当前测量源码SHA-256保存在[环境清单](data/phase-6-execution-load/environment.json)。
|
||||
|
||||
| 项目 | 实际值 |
|
||||
| --- | --- |
|
||||
| 主机/运行环境 | macOS arm64、Docker Linux/arm64;共享VM可见12 CPU、12529274880字节内存,未独占硬件 |
|
||||
| 工具链 | 实际 `go1.25.14`,仅临时modfile替换Go指令;仓库仍声明Go 1.27.1 |
|
||||
| 拓扑 | 一次性MySQL 8.4.10,1个Orbit进程,1个负载进程内2条Worker连接,每条capacity=4 |
|
||||
| 网络 | Go运行容器共享MySQL容器网络命名空间;HTTP/WSS和MySQL均走loopback |
|
||||
| MySQL | `innodb_flush_log_at_trx_commit=1`、`sync_binlog=1`、buffer pool=128MiB、max_connections=151 |
|
||||
| 隔离级别 | MySQL全局REPEATABLE-READ;Orbit写事务显式READ COMMITTED |
|
||||
| Orbit | 数据库连接池max_open=20、max_idle=5;鉴权和持久事件/审计启用;metrics未启用 |
|
||||
| 队列 | max_concurrency=4、rate_per_second=10、max_unfinished_tasks=100000;工具未更改配置 |
|
||||
| 核验 | 每个HTTP请求5秒上限、单核验器250ms间隔,核验器最多与提交并发数相同 |
|
||||
| 数据量 | 空库初始化;short/mixed/long/deadline顺序运行,开始时已有任务数分别0/96/192/198;各运行身份隔离 |
|
||||
|
||||
最后 `orbit-admin verify` 返回 schema=28、migration_count=28、foreign_key_count=0、required_index_count=6、complete=true。测试容器退出后销毁一次性数据库和凭据。
|
||||
|
||||
## 原始结果
|
||||
|
||||
前三组均在2分钟整体预算内完整通过,退出0。三个阶段分布的分母分别为96、96、6;都没有重复处理器调用、无回执调用或Worker重连。
|
||||
|
||||
| 样本 | 任务/处理器时长 | 受理p99 ms | 排队p99 ms | 授权执行p99 ms | 任务完成p99 ms | 持久成功/计划数 |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| [short原始JSON](data/phase-6-execution-load/short.json) | 96项,0ms | 54.678 | 13752.092 | 28.838 | 13768.327 | 96/96 |
|
||||
| [mixed原始JSON](data/phase-6-execution-load/mixed.json) | 96项,0/100/1000ms循环,各32项 | 40.274 | 41024.973 | 1042.773 | 42047.090 | 96/96 |
|
||||
| [long原始JSON](data/phase-6-execution-load/long.json) | 6项,每项35000ms | 28.028 | 38015.493 | 35024.585 | 73040.078 | 6/6 |
|
||||
|
||||
长任务总计18次续租确认,最终仍为每项一次尝试、一次调用。该数字验证了此次实际续租路径,不表示节点/网络故障已经发生。
|
||||
|
||||
排队分布包含当前4并发/每秒10次开始的队列约束及调度扫描;Worker声明8个槽不会绕过队列限制。这组数字不能视为去掉配额后的最大吞吐。
|
||||
|
||||
`authorized_execution` 和 `task_completion` 用持久化数据库时间戳计算;终点是结束事务内部取时点,未包含其后COMMIT及确认延迟。已提交事实由后续读取核验。客户端核验顺序不计入这些分布,但查询负载会影响服务资源消耗。没有收集独占CPU/内存/磁盘/锁等待曲线,也没有多轮稳态和生产指标开启后的对照。
|
||||
|
||||
## 失败语义实测
|
||||
|
||||
[deadline原始JSON](data/phase-6-execution-load/deadline.json):24项、4并发提交、1个Worker/1个槽、每项1秒、整体预算2秒。命令按预期退出1,`accepted=24`、`verified_success=0`、`unverified=24`、`complete=false`;本地只有1次处理器调用。读取观察到1项NEEDS_REVIEW,23项未能在截止前验证,三种完成阶段的分布均为 `count=0` / 百分位null。
|
||||
|
||||
错误计数保留 `RUN_TIMEOUT`、`TASK_NEEDS_REVIEW`、`VERIFY_TIMEOUT` 和 `INVOCATION_MISMATCH`。这些计数含运行级及交叠的核验错误,不能求和当成失败任务总数。工具没有删除、强制成功或重新提交这些任务;未观察到终态不等于数据库确认失败,剩余状态留在原身份下供核查。
|
||||
|
||||
## 复现和回归
|
||||
|
||||
按[手册](../api/execution-load.md)准备一次性数据库、当前迁移、带Worker权限的凭据及真实Orbit进程,然后在 `sdk/go` 运行:
|
||||
|
||||
```sh
|
||||
go run ./cmd/orbit-exec-load -requests 96 -concurrency 8 -workers 2 -capacity 4 -durations 0ms -run-timeout 2m
|
||||
go run ./cmd/orbit-exec-load -requests 96 -concurrency 8 -workers 2 -capacity 4 -durations 0ms,100ms,1s -run-timeout 2m
|
||||
go run ./cmd/orbit-exec-load -requests 6 -concurrency 3 -workers 2 -capacity 4 -durations 35s -run-timeout 2m
|
||||
# 此项应失败;必须读取JSON确认是预算不足,不应仅判断任意非零退出。
|
||||
go run ./cmd/orbit-exec-load -requests 24 -concurrency 4 -workers 1 -capacity 1 -durations 1s -run-timeout 2s
|
||||
```
|
||||
|
||||
本轮在上述Docker环境使用临时modfile构建真实Linux二进制,执行了四条等价命令及下列检查:
|
||||
|
||||
- SDK全模块 `go test -race ./...`、`go vet ./...`、`go build ./...` 通过;没有设置原SDK集成凭据的普通单测不冒充全量集成。
|
||||
- 新增单测覆盖:错配/缺失/空字段结果、未知效果、第二次尝试、悬空历史、时间逆序、提交超时/429不重报、整次取消、核查不算成功、读取失败、重复调用、无权限预检及错误脱敏。
|
||||
- SDK完整依赖图218项中,服务端内部/模型/模块和MySQL驱动依赖均为0。
|
||||
- `scripts/test-mysql.sh` 已加入6项真实执行命令冒烟,shell语法检查通过。本轮运行的是四个独立真实样本,未重新执行整个 `make integration`。
|
||||
|
||||
当前不把临时Go 1.25兼容回归升级为声明Go 1.27.1的正式发布结论。下一批继续定时集中触发、流程推进、长期积压、慢消费者/重连风暴及资源曲线,生产HA、复制切换与RPO/RTO仍需目标拓扑证据。
|
||||
@@ -17,6 +17,7 @@
|
||||
| SDK Worker/Event/Workflow 子进程故障 | 同一共享网络命名空间;容器内构建 Orbit 二进制;SDK `TestIntegration(Worker\|Event\|Workflow)Process` | PASS,136.394 秒;派发未送达、租约接管、ACK 提交回执丢失、重试未知结果、计划/优先级/键并发、流程暂停全停接管均通过 |
|
||||
| SDK 应用配额子进程故障 | 独立双命名空间应用配额凭据;共享网络命名空间;`TestIntegrationWorkerProcessApplicationBackpressure` | PASS,1.069 秒;重启后共享额度、原回执和跨命名空间背压保持 |
|
||||
| Orbit HTTP/metrics 子进程 | 同一共享网络命名空间;`TestIntegrationMetricsProcess`;metrics 启用/禁用各一组 | PASS,3.081 秒;监听隔离、隐私、后台采样和优雅退出均通过 |
|
||||
| 真实Worker执行测量 | `orbit-exec-load`;最终短/混合/中长及截止不足四组,环境及原始JSON见[测量报告](phase-6-execution-load.md) | 96+96+6项全部逐项核验,长任务18次续租;24项截止负例按预期非零退出。SDK全模块竞态/vet/build通过,生产容量未验收 |
|
||||
|
||||
## 当前环境仍未覆盖
|
||||
|
||||
|
||||
@@ -46,6 +46,7 @@ ORBIT_TEST_SERVER_BINARY="$TASK_TEST_DIR/orbit" "$TASK_GO" test -race -count=1 -
|
||||
"$TASK_GO" build -o "$TASK_TEST_DIR/orbit-migrate" ./cmd/orbit-migrate
|
||||
"$TASK_GO" build -o "$TASK_TEST_DIR/orbit-admin" ./cmd/orbit-admin
|
||||
(cd sdk/go && GOWORK=off "$TASK_GO" build -o "$TASK_TEST_DIR/orbit-load" ./cmd/orbit-load)
|
||||
(cd sdk/go && GOWORK=off "$TASK_GO" build -o "$TASK_TEST_DIR/orbit-exec-load" ./cmd/orbit-exec-load)
|
||||
ORBIT_MYSQL_DSN="$ORBIT_TEST_MYSQL_DSN" "$TASK_TEST_DIR/orbit-migrate" status
|
||||
ORBIT_MYSQL_DSN="$ORBIT_TEST_MYSQL_DSN" "$TASK_TEST_DIR/orbit-admin" verify
|
||||
TASK_SNAPSHOT_JSON=$(ORBIT_MYSQL_DSN="$ORBIT_TEST_MYSQL_DSN" "$TASK_TEST_DIR/orbit-admin" observe snapshot)
|
||||
@@ -80,6 +81,7 @@ if [ -z "$TASK_HTTP_PORT" ]; then printf '%s\n' 'Bootstrap HTTP server did not s
|
||||
(cd sdk/go && ORBIT_URL="http://127.0.0.1:$TASK_HTTP_PORT" ORBIT_CREDENTIALS_FILE="$TASK_TEST_DIR/sdk.credentials.json" "$TASK_GO" run ./examples/tasks)
|
||||
(cd sdk/go && ORBIT_URL="http://127.0.0.1:$TASK_HTTP_PORT" ORBIT_CREDENTIALS_FILE="$TASK_TEST_DIR/sdk.credentials.json" "$TASK_GO" run ./examples/task-query)
|
||||
ORBIT_MYSQL_DSN="$ORBIT_TEST_MYSQL_DSN" "$TASK_TEST_DIR/orbit-admin" bootstrap --application sdk-worker --namespace default --environment test --queue worker-examples --worker-access --schedule-access --event-access --output "$TASK_TEST_DIR/worker.credentials.json"
|
||||
(cd sdk/go && ORBIT_URL="http://127.0.0.1:$TASK_HTTP_PORT" ORBIT_CREDENTIALS_FILE="$TASK_TEST_DIR/worker.credentials.json" "$TASK_TEST_DIR/orbit-exec-load" -requests 6 -concurrency 2 -workers 2 -capacity 2 -durations 0ms,100ms -run-timeout 30s)
|
||||
(cd sdk/go && ORBIT_TEST_WORKER_URL="http://127.0.0.1:$TASK_HTTP_PORT" ORBIT_TEST_WORKER_CREDENTIALS_FILE="$TASK_TEST_DIR/worker.credentials.json" "$TASK_GO" test -race -count=1 -run 'IntegrationWorker(Connection|Runtime)' -v .)
|
||||
(cd sdk/go && ORBIT_TEST_WORKER_URL="http://127.0.0.1:$TASK_HTTP_PORT" ORBIT_TEST_WORKER_CREDENTIALS_FILE="$TASK_TEST_DIR/worker.credentials.json" "$TASK_GO" test -race -count=1 -run '^TestIntegrationSchedule' -v .)
|
||||
(cd sdk/go && ORBIT_TEST_WORKER_URL="http://127.0.0.1:$TASK_HTTP_PORT" ORBIT_TEST_WORKER_CREDENTIALS_FILE="$TASK_TEST_DIR/worker.credentials.json" "$TASK_GO" test -race -count=1 -run '^TestIntegrationQueueResourcesSDK$' -v .)
|
||||
|
||||
139
sdk/go/cmd/orbit-exec-load/main.go
Normal file
139
sdk/go/cmd/orbit-exec-load/main.go
Normal file
@@ -0,0 +1,139 @@
|
||||
// orbit-exec-load 在隔离环境测量真实 HTTP 受理、WSS Worker 和持久化完成链路。
|
||||
// 处理器只有有界等待及回显,不访问外部业务系统。
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
orbit "git.yuntuo.hk/k2safe/orbit/sdk/go"
|
||||
)
|
||||
|
||||
type config struct {
|
||||
URL, Token, QueueID string
|
||||
Requests, Concurrency, Workers, Capacity int
|
||||
Timeout, RunTimeout, PollInterval time.Duration
|
||||
Durations []time.Duration
|
||||
}
|
||||
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func run() error {
|
||||
cfg, err := parseConfig(os.Args[1:])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
client, err := orbit.NewClient(cfg.URL, orbit.Options{Token: cfg.Token, HTTPClient: &http.Client{Timeout: cfg.Timeout}})
|
||||
if err != nil {
|
||||
return errors.New("invalid client configuration")
|
||||
}
|
||||
runID, err := orbit.NewWorkerMessageID()
|
||||
if err != nil {
|
||||
return errors.New("cannot generate run identity")
|
||||
}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
report := measure(ctx, cfg, client, runID)
|
||||
if err := json.NewEncoder(os.Stdout).Encode(report); err != nil {
|
||||
return err
|
||||
}
|
||||
if !report.Complete {
|
||||
return errors.New("execution sample incomplete; inspect report and original run identities")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func parseConfig(args []string) (config, error) {
|
||||
cfg := config{URL: os.Getenv("ORBIT_URL"), Token: os.Getenv("ORBIT_TOKEN"), QueueID: os.Getenv("ORBIT_QUEUE_ID"), Requests: 100, Concurrency: 8, Workers: 2, Capacity: 4, Timeout: 5 * time.Second, RunTimeout: 2 * time.Minute, PollInterval: 250 * time.Millisecond}
|
||||
flags := flag.NewFlagSet("orbit-exec-load", flag.ContinueOnError)
|
||||
var credentials, durations string
|
||||
flags.StringVar(&cfg.URL, "url", cfg.URL, "Orbit HTTP(S) URL")
|
||||
flags.StringVar(&credentials, "credentials", os.Getenv("ORBIT_CREDENTIALS_FILE"), "private credentials file for an isolated namespace")
|
||||
flags.StringVar(&cfg.QueueID, "queue", cfg.QueueID, "existing isolated queue; configuration is not changed")
|
||||
flags.IntVar(&cfg.Requests, "requests", cfg.Requests, "tasks, 1..10000")
|
||||
flags.IntVar(&cfg.Concurrency, "concurrency", cfg.Concurrency, "maximum concurrent submission/verification requests, 1..128")
|
||||
flags.IntVar(&cfg.Workers, "workers", cfg.Workers, "independent WSS Workers in this process, 1..32")
|
||||
flags.IntVar(&cfg.Capacity, "capacity", cfg.Capacity, "concurrent handlers per Worker, 1..256; total at most 1024")
|
||||
flags.DurationVar(&cfg.Timeout, "timeout", cfg.Timeout, "per HTTP request timeout, 1ms..1m")
|
||||
flags.DurationVar(&cfg.RunTimeout, "run-timeout", cfg.RunTimeout, "whole sample deadline, 1s..1h, plus at most 5s Worker shutdown")
|
||||
flags.DurationVar(&cfg.PollInterval, "poll-interval", cfg.PollInterval, "per active verifier polling interval, 100ms..10s")
|
||||
flags.StringVar(&durations, "durations", "0ms", "1..16 round-robin simulated handler durations, e.g. 0ms,100ms,2s; whole milliseconds, 0..60s")
|
||||
if err := flags.Parse(args); err != nil {
|
||||
return config{}, err
|
||||
}
|
||||
if flags.NArg() != 0 {
|
||||
return config{}, errors.New("positional arguments are not supported")
|
||||
}
|
||||
if credentials != "" {
|
||||
f, err := os.Open(credentials)
|
||||
if err != nil {
|
||||
return config{}, errors.New("cannot open private credentials file")
|
||||
}
|
||||
raw, err := io.ReadAll(io.LimitReader(f, (16<<10)+1))
|
||||
f.Close()
|
||||
var saved struct {
|
||||
Token string `json:"token"`
|
||||
QueueID string `json:"queue_id"`
|
||||
}
|
||||
if err != nil || len(raw) > 16<<10 || json.Unmarshal(raw, &saved) != nil {
|
||||
return config{}, errors.New("invalid credentials file (maximum 16 KiB)")
|
||||
}
|
||||
if cfg.Token == "" {
|
||||
cfg.Token = saved.Token
|
||||
}
|
||||
if cfg.QueueID == "" {
|
||||
cfg.QueueID = saved.QueueID
|
||||
}
|
||||
}
|
||||
if cfg.URL == "" || cfg.Token == "" || !validID(cfg.QueueID) {
|
||||
return config{}, errors.New("URL, token and a 32-character lowercase hexadecimal queue ID are required")
|
||||
}
|
||||
if cfg.Requests < 1 || cfg.Requests > 10000 || cfg.Concurrency < 1 || cfg.Concurrency > 128 || cfg.Workers < 1 || cfg.Workers > 32 || cfg.Capacity < 1 || cfg.Capacity > 256 || cfg.Workers*cfg.Capacity > 1024 {
|
||||
return config{}, errors.New("request, concurrency or Worker count is out of bounds")
|
||||
}
|
||||
if cfg.Timeout < time.Millisecond || cfg.Timeout > time.Minute || cfg.RunTimeout < time.Second || cfg.RunTimeout > time.Hour || cfg.PollInterval < 100*time.Millisecond || cfg.PollInterval > 10*time.Second {
|
||||
return config{}, errors.New("timeout or poll interval is out of bounds")
|
||||
}
|
||||
parts := strings.Split(durations, ",")
|
||||
if len(parts) > 16 {
|
||||
return config{}, errors.New("at most 16 handler durations are allowed")
|
||||
}
|
||||
for _, part := range parts {
|
||||
d, err := time.ParseDuration(strings.TrimSpace(part))
|
||||
if err != nil || d < 0 || d > time.Minute || d%time.Millisecond != 0 || d+time.Second > cfg.RunTimeout {
|
||||
return config{}, errors.New("handler duration must be whole milliseconds in 0..60s and allow at least 1s within run-timeout")
|
||||
}
|
||||
cfg.Durations = append(cfg.Durations, d)
|
||||
}
|
||||
if cfg.Concurrency > cfg.Requests {
|
||||
cfg.Concurrency = cfg.Requests
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func validID(id string) bool {
|
||||
if len(id) != 32 {
|
||||
return false
|
||||
}
|
||||
for _, c := range id {
|
||||
if !(c >= '0' && c <= '9' || c >= 'a' && c <= 'f') {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
367
sdk/go/cmd/orbit-exec-load/measure.go
Normal file
367
sdk/go/cmd/orbit-exec-load/measure.go
Normal file
@@ -0,0 +1,367 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
orbit "git.yuntuo.hk/k2safe/orbit/sdk/go"
|
||||
)
|
||||
|
||||
// loadPayload 的固定内容同时是处理器的输入和预期结果;不含凭据或外部地址。
|
||||
type loadPayload struct {
|
||||
RunID string `json:"run_id"` // RunID 隔离本次运行,不作为业务幂等保证。
|
||||
Sequence int `json:"sequence"` // Sequence 从0开始的请求序号,用于发现重复执行及错配。
|
||||
DurationMS int64 `json:"duration_ms"` // DurationMS 模拟等待毫秒数,按本次固定时长数组取模。
|
||||
}
|
||||
|
||||
func decodePayload(raw json.RawMessage) (loadPayload, bool) {
|
||||
var wire struct {
|
||||
RunID *string `json:"run_id"`
|
||||
Sequence *int `json:"sequence"`
|
||||
DurationMS *int64 `json:"duration_ms"`
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(raw))
|
||||
decoder.DisallowUnknownFields()
|
||||
if decoder.Decode(&wire) != nil || wire.RunID == nil || wire.Sequence == nil || wire.DurationMS == nil {
|
||||
return loadPayload{}, false
|
||||
}
|
||||
var trailing any
|
||||
if decoder.Decode(&trailing) != io.EOF {
|
||||
return loadPayload{}, false
|
||||
}
|
||||
return loadPayload{*wire.RunID, *wire.Sequence, *wire.DurationMS}, true
|
||||
}
|
||||
|
||||
type admission struct {
|
||||
sequence int
|
||||
task orbit.Task
|
||||
duration time.Duration
|
||||
code string
|
||||
}
|
||||
|
||||
type taskAPI interface {
|
||||
SubmitTask(context.Context, string, orbit.SubmitTaskRequest) (orbit.SubmitTaskResult, error)
|
||||
GetTask(context.Context, string) (orbit.Task, error)
|
||||
ListTaskAttempts(context.Context, string, orbit.HistoryPageRequest) (orbit.AttemptPage, error)
|
||||
}
|
||||
|
||||
type measurement struct {
|
||||
queue, execution, completed time.Duration
|
||||
}
|
||||
|
||||
func expected(cfg config, runID string, sequence int) loadPayload {
|
||||
return loadPayload{runID, sequence, cfg.Durations[sequence%len(cfg.Durations)].Milliseconds()}
|
||||
}
|
||||
|
||||
func handlerName(runID string) string { return "load.exec." + runID }
|
||||
func businessKey(runID string) string { return "orbit-exec-" + runID }
|
||||
func requestKey(runID string, sequence int) string {
|
||||
return fmt.Sprintf("orbit-exec-%s-%06d", runID, sequence)
|
||||
}
|
||||
|
||||
// invocationLog 有界保存本次每个序号的实际处理器调用,发现重复时不覆盖首条证据。
|
||||
type invocationLog struct {
|
||||
mu sync.Mutex
|
||||
calls []int
|
||||
taskIDs []string
|
||||
invalid int
|
||||
}
|
||||
|
||||
func newInvocationLog(n int) *invocationLog {
|
||||
return &invocationLog{calls: make([]int, n), taskIDs: make([]string, n)}
|
||||
}
|
||||
|
||||
func (log *invocationLog) handler(runCtx context.Context, cfg config, runID string) orbit.WorkerHandler {
|
||||
return orbit.WorkerHandler{Capability: orbit.WorkerCapability{Handler: handlerName(runID), HandlerVersion: "v1", PayloadVersion: "v1"}, Handle: func(ctx context.Context, execution orbit.WorkerExecution) (json.RawMessage, error) {
|
||||
payload, valid := decodePayload(execution.Task.Payload)
|
||||
if !valid || payload.RunID != runID || payload.Sequence < 0 || payload.Sequence >= cfg.Requests || payload != expected(cfg, runID, payload.Sequence) || execution.Task.QueueID != cfg.QueueID || execution.Task.BusinessKey != businessKey(runID) || execution.Task.Handler != handlerName(runID) {
|
||||
log.mu.Lock()
|
||||
log.invalid++
|
||||
log.mu.Unlock()
|
||||
return nil, &orbit.WorkerFailure{Code: "LOAD_INPUT_MISMATCH"}
|
||||
}
|
||||
log.mu.Lock()
|
||||
log.calls[payload.Sequence]++
|
||||
first := log.calls[payload.Sequence] == 1
|
||||
if first {
|
||||
log.taskIDs[payload.Sequence] = execution.Task.ID
|
||||
}
|
||||
log.mu.Unlock()
|
||||
if !first {
|
||||
return nil, &orbit.WorkerFailure{Code: "LOAD_DUPLICATE_EXECUTION"}
|
||||
}
|
||||
timer := time.NewTimer(time.Duration(payload.DurationMS) * time.Millisecond)
|
||||
defer timer.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil, ctx.Err()
|
||||
case <-runCtx.Done():
|
||||
return nil, runCtx.Err()
|
||||
case <-timer.C:
|
||||
return json.Marshal(payload)
|
||||
}
|
||||
}}
|
||||
}
|
||||
|
||||
func measure(parent context.Context, cfg config, client *orbit.Client, runID string) executionReport {
|
||||
r := newReport(cfg, runID)
|
||||
ctx, cancel := context.WithTimeout(parent, cfg.RunTimeout)
|
||||
defer cancel()
|
||||
requestCtx, stop := context.WithTimeout(ctx, cfg.Timeout)
|
||||
queue, err := client.GetQueue(requestCtx, cfg.QueueID)
|
||||
stop()
|
||||
if err != nil {
|
||||
r.Errors["PREFLIGHT_"+classify(err)]++
|
||||
return r.finish()
|
||||
}
|
||||
r.Queue = &queue
|
||||
if queue.ID != cfg.QueueID || queue.Paused || queue.MaxConcurrency == 0 || queue.RatePerSecond == 0 {
|
||||
r.Errors["QUEUE_UNAVAILABLE"]++
|
||||
return r.finish()
|
||||
}
|
||||
log := newInvocationLog(cfg.Requests)
|
||||
var workers []*orbit.Worker
|
||||
for i := 0; i < cfg.Workers; i++ {
|
||||
w, err := orbit.NewWorker(client, orbit.WorkerOptions{QueueID: cfg.QueueID, Capacity: uint32(cfg.Capacity), ShutdownTimeout: 5 * time.Second, Handlers: []orbit.WorkerHandler{log.handler(ctx, cfg, runID)}})
|
||||
if err != nil {
|
||||
r.Errors["WORKER_CONFIG"]++
|
||||
return r.finish()
|
||||
}
|
||||
workers = append(workers, w)
|
||||
}
|
||||
workerCtx, stopWorkers := context.WithCancel(ctx)
|
||||
done := make(chan error, len(workers))
|
||||
for _, w := range workers {
|
||||
go func(w *orbit.Worker) {
|
||||
err := w.Run(workerCtx)
|
||||
if workerCtx.Err() == nil {
|
||||
cancel()
|
||||
} // 不可恢复连接错误立即停止继续提交。
|
||||
done <- err
|
||||
}(w)
|
||||
}
|
||||
admissions := submit(ctx, cfg, client, runID)
|
||||
var successful []admission
|
||||
var admissionDurations []time.Duration
|
||||
seen := map[string]bool{}
|
||||
for _, a := range admissions {
|
||||
admissionDurations = append(admissionDurations, a.duration)
|
||||
if a.code != "" {
|
||||
r.AdmissionErrors++
|
||||
r.Errors["ADMISSION_"+a.code]++
|
||||
continue
|
||||
}
|
||||
if seen[a.task.ID] {
|
||||
r.AdmissionErrors++
|
||||
r.Errors["DUPLICATE_TASK_ID"]++
|
||||
continue
|
||||
}
|
||||
seen[a.task.ID] = true
|
||||
successful = append(successful, a)
|
||||
}
|
||||
r.Attempted = len(admissions)
|
||||
r.Accepted = len(successful)
|
||||
r.NotSubmitted = cfg.Requests - r.Attempted
|
||||
r.Admission = distributionOf(admissionDurations)
|
||||
r.AdmissionFinishedAt = time.Now().UTC()
|
||||
verified := verifyAll(ctx, cfg, client, runID, successful)
|
||||
var queued, executed, completed []time.Duration
|
||||
for _, v := range verified {
|
||||
if v.code != "" {
|
||||
r.Errors[v.code]++
|
||||
continue
|
||||
}
|
||||
r.VerifiedSuccess++
|
||||
queued = append(queued, v.measurement.queue)
|
||||
executed = append(executed, v.measurement.execution)
|
||||
completed = append(completed, v.measurement.completed)
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
r.Errors["RUN_"+classify(ctx.Err())]++
|
||||
}
|
||||
stopWorkers()
|
||||
for range workers {
|
||||
if err := <-done; err != nil {
|
||||
r.Errors["WORKER_STOP"]++
|
||||
}
|
||||
}
|
||||
for _, w := range workers {
|
||||
s := w.Stats()
|
||||
r.WorkerStarted += s.Started
|
||||
r.Reconnects += s.Reconnects
|
||||
r.Renewals += s.RenewalsConfirmed
|
||||
}
|
||||
log.mu.Lock()
|
||||
for i, calls := range log.calls {
|
||||
r.HandlerCalls += calls
|
||||
if calls > 1 {
|
||||
r.DuplicateInvocations += calls - 1
|
||||
}
|
||||
if calls > 0 && !seen[log.taskIDs[i]] {
|
||||
r.UnconfirmedHandlerCalls += calls
|
||||
}
|
||||
}
|
||||
for _, a := range successful {
|
||||
if log.calls[a.sequence] != 1 || log.taskIDs[a.sequence] != a.task.ID {
|
||||
r.Errors["INVOCATION_MISMATCH"]++
|
||||
}
|
||||
}
|
||||
r.Errors["INVALID_HANDLER_INPUT"] += log.invalid
|
||||
log.mu.Unlock()
|
||||
r.QueueWait = distributionOf(queued)
|
||||
r.AuthorizedExecution = distributionOf(executed)
|
||||
r.TaskCompletion = distributionOf(completed)
|
||||
return r.finish()
|
||||
}
|
||||
|
||||
func submit(ctx context.Context, cfg config, api taskAPI, runID string) []admission {
|
||||
jobs := make(chan int)
|
||||
out := make(chan admission, cfg.Requests)
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < cfg.Concurrency; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for sequence := range jobs {
|
||||
if ctx.Err() != nil {
|
||||
continue
|
||||
}
|
||||
payload, _ := json.Marshal(expected(cfg, runID, sequence))
|
||||
start := time.Now()
|
||||
call, stop := context.WithTimeout(ctx, cfg.Timeout)
|
||||
res, err := api.SubmitTask(call, requestKey(runID, sequence), orbit.SubmitTaskRequest{QueueID: cfg.QueueID, Handler: handlerName(runID), HandlerVersion: "v1", PayloadVersion: "v1", BusinessKey: businessKey(runID), Payload: payload, TaskTimeoutMS: uint64(cfg.RunTimeout.Milliseconds()), RecoverySafety: orbit.RecoveryReconcileOnly, Retry: orbit.RetryPolicy{MaxAttempts: 1, AttemptTimeoutMS: uint64(cfg.RunTimeout.Milliseconds())}})
|
||||
stop()
|
||||
a := admission{sequence: sequence, task: res.Task, duration: time.Since(start)}
|
||||
if err != nil {
|
||||
a.code = classify(err)
|
||||
} else if res.Deduplicated || !validID(res.Task.ID) || !matches(res.Task, cfg, runID, sequence, false) {
|
||||
a.code = "INVALID_RECEIPT"
|
||||
}
|
||||
out <- a
|
||||
}
|
||||
}()
|
||||
}
|
||||
feed:
|
||||
for i := 0; i < cfg.Requests; i++ {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
break feed
|
||||
case jobs <- i:
|
||||
}
|
||||
}
|
||||
close(jobs)
|
||||
wg.Wait()
|
||||
close(out)
|
||||
results := make([]admission, 0, cfg.Requests)
|
||||
for a := range out {
|
||||
results = append(results, a)
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
type verification struct {
|
||||
measurement measurement
|
||||
code string
|
||||
}
|
||||
|
||||
func verifyAll(ctx context.Context, cfg config, api taskAPI, runID string, admissions []admission) []verification {
|
||||
jobs := make(chan admission, len(admissions))
|
||||
for _, a := range admissions {
|
||||
jobs <- a
|
||||
}
|
||||
close(jobs)
|
||||
out := make(chan verification, len(admissions))
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < cfg.Concurrency; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for a := range jobs {
|
||||
m, code := verify(ctx, cfg, api, runID, a)
|
||||
out <- verification{m, code}
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(out)
|
||||
results := make([]verification, 0, len(admissions))
|
||||
for v := range out {
|
||||
results = append(results, v)
|
||||
}
|
||||
return results
|
||||
}
|
||||
|
||||
func verify(ctx context.Context, cfg config, api taskAPI, runID string, a admission) (measurement, string) {
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return measurement{}, "VERIFY_" + classify(ctx.Err())
|
||||
}
|
||||
call, stop := context.WithTimeout(ctx, cfg.Timeout)
|
||||
task, err := api.GetTask(call, a.task.ID)
|
||||
stop()
|
||||
if err != nil {
|
||||
return measurement{}, "READ_" + classify(err)
|
||||
}
|
||||
if task.ID != a.task.ID || !matches(task, cfg, runID, a.sequence, false) {
|
||||
return measurement{}, "TASK_MISMATCH"
|
||||
}
|
||||
switch task.State {
|
||||
case orbit.TaskSucceeded:
|
||||
call, stop = context.WithTimeout(ctx, cfg.Timeout)
|
||||
history, err := api.ListTaskAttempts(call, task.ID, orbit.HistoryPageRequest{Limit: 2})
|
||||
stop()
|
||||
if err != nil {
|
||||
return measurement{}, "HISTORY_" + classify(err)
|
||||
}
|
||||
return checkCompletion(cfg, runID, a.sequence, task, history)
|
||||
case orbit.TaskFailed, orbit.TaskCancelled, orbit.TaskNeedsReview:
|
||||
return measurement{}, "TASK_" + string(task.State)
|
||||
case orbit.TaskWaiting, orbit.TaskLeased, orbit.TaskRunning:
|
||||
default:
|
||||
return measurement{}, "UNKNOWN_TASK_STATE"
|
||||
}
|
||||
timer := time.NewTimer(cfg.PollInterval)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
return measurement{}, "VERIFY_" + classify(ctx.Err())
|
||||
case <-timer.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func matches(task orbit.Task, cfg config, runID string, sequence int, result bool) bool {
|
||||
payload, valid := decodePayload(task.Payload)
|
||||
if task.QueueID != cfg.QueueID || task.Handler != handlerName(runID) || task.HandlerVersion != "v1" || task.PayloadVersion != "v1" || task.BusinessKey != businessKey(runID) || task.Retry.MaxAttempts != 1 || task.RecoverySafety != orbit.RecoveryReconcileOnly || !valid || payload != expected(cfg, runID, sequence) {
|
||||
return false
|
||||
}
|
||||
if result {
|
||||
output, valid := decodePayload(task.Result)
|
||||
if !valid || output != expected(cfg, runID, sequence) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func checkCompletion(cfg config, runID string, sequence int, task orbit.Task, history orbit.AttemptPage) (measurement, string) {
|
||||
if task.State != orbit.TaskSucceeded || task.EffectUncertain || task.AttemptCount != 1 || task.CompletedAt == nil || !matches(task, cfg, runID, sequence, true) {
|
||||
return measurement{}, "COMPLETION_MISMATCH"
|
||||
}
|
||||
if len(history.Items) != 1 || history.NextCursor != "" {
|
||||
return measurement{}, "ATTEMPT_COUNT_MISMATCH"
|
||||
}
|
||||
a := history.Items[0]
|
||||
if !validID(a.ID) || a.TaskID != task.ID || a.Epoch != 1 || a.State != orbit.AttemptSucceeded || a.AuthorizedAt == nil || a.EndedAt == nil {
|
||||
return measurement{}, "ATTEMPT_MISMATCH"
|
||||
}
|
||||
if task.CreatedAt.IsZero() || task.AvailableAt.Before(task.CreatedAt) || a.CreatedAt.Before(task.AvailableAt) || a.AuthorizedAt.Before(a.CreatedAt) || a.EndedAt.Before(*a.AuthorizedAt) || !a.EndedAt.Equal(*task.CompletedAt) {
|
||||
return measurement{}, "TIMESTAMP_MISMATCH"
|
||||
}
|
||||
return measurement{queue: a.AuthorizedAt.Sub(task.AvailableAt), execution: a.EndedAt.Sub(*a.AuthorizedAt), completed: task.CompletedAt.Sub(task.CreatedAt)}, ""
|
||||
}
|
||||
268
sdk/go/cmd/orbit-exec-load/measure_test.go
Normal file
268
sdk/go/cmd/orbit-exec-load/measure_test.go
Normal file
@@ -0,0 +1,268 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
orbit "git.yuntuo.hk/k2safe/orbit/sdk/go"
|
||||
)
|
||||
|
||||
const testRunID = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
|
||||
const testQueueID = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
|
||||
|
||||
func testConfig() config {
|
||||
return config{QueueID: testQueueID, Requests: 6, Concurrency: 3, Workers: 2, Capacity: 2, Timeout: time.Second, RunTimeout: time.Minute, PollInterval: 100 * time.Millisecond, Durations: []time.Duration{0, 10 * time.Millisecond}}
|
||||
}
|
||||
|
||||
func completedFixture(cfg config) (orbit.Task, orbit.AttemptPage) {
|
||||
t0 := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
leasing, auth, end := t0.Add(time.Second), t0.Add(2*time.Second), t0.Add(3*time.Second)
|
||||
payload, _ := json.Marshal(expected(cfg, testRunID, 0))
|
||||
task := orbit.Task{ID: "cccccccccccccccccccccccccccccccc", QueueID: cfg.QueueID, Handler: handlerName(testRunID), HandlerVersion: "v1", PayloadVersion: "v1", BusinessKey: businessKey(testRunID), Payload: payload, Result: payload, State: orbit.TaskSucceeded, AttemptCount: 1, CreatedAt: t0, AvailableAt: t0, CompletedAt: &end, RecoverySafety: orbit.RecoveryReconcileOnly, Retry: orbit.RetryPolicy{MaxAttempts: 1}}
|
||||
history := orbit.AttemptPage{Items: []orbit.AttemptSummary{{ID: "dddddddddddddddddddddddddddddddd", TaskID: task.ID, Epoch: 1, State: orbit.AttemptSucceeded, CreatedAt: leasing, AuthorizedAt: &auth, EndedAt: &end}}}
|
||||
return task, history
|
||||
}
|
||||
|
||||
func TestCompletionRequiresResultSingleAttemptAndOrderedServerTimestamps(t *testing.T) {
|
||||
cfg := testConfig()
|
||||
task, history := completedFixture(cfg)
|
||||
m, code := checkCompletion(cfg, testRunID, 0, task, history)
|
||||
if code != "" || m.queue != 2*time.Second || m.execution != time.Second || m.completed != 3*time.Second {
|
||||
t.Fatal(m, code)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
mutate func(*orbit.Task, *orbit.AttemptPage)
|
||||
}{
|
||||
{"wrong-result", func(v *orbit.Task, _ *orbit.AttemptPage) { v.Result = json.RawMessage(`{}`) }},
|
||||
{"missing-zero-valued-fields", func(v *orbit.Task, _ *orbit.AttemptPage) {
|
||||
v.Result = json.RawMessage(`{"run_id":"` + testRunID + `"}`)
|
||||
}},
|
||||
{"null-zero-valued-fields", func(v *orbit.Task, _ *orbit.AttemptPage) {
|
||||
v.Result = json.RawMessage(`{"run_id":"` + testRunID + `","sequence":null,"duration_ms":null}`)
|
||||
}},
|
||||
{"result-other-run", func(v *orbit.Task, _ *orbit.AttemptPage) {
|
||||
v.Result = json.RawMessage(`{"run_id":"other","sequence":0,"duration_ms":0}`)
|
||||
}},
|
||||
{"unknown-effect", func(v *orbit.Task, _ *orbit.AttemptPage) { v.EffectUncertain = true }},
|
||||
{"wrong-task", func(_ *orbit.Task, h *orbit.AttemptPage) { h.Items[0].TaskID = testRunID }},
|
||||
{"second-attempt", func(_ *orbit.Task, h *orbit.AttemptPage) { h.Items = append(h.Items, h.Items[0]) }},
|
||||
{"more-attempt-pages", func(_ *orbit.Task, h *orbit.AttemptPage) { h.NextCursor = "more" }},
|
||||
{"missing-authorization", func(_ *orbit.Task, h *orbit.AttemptPage) { h.Items[0].AuthorizedAt = nil }},
|
||||
{"reversed-time", func(v *orbit.Task, h *orbit.AttemptPage) { h.Items[0].EndedAt = &v.CreatedAt }},
|
||||
{"completion-not-settled", func(v *orbit.Task, _ *orbit.AttemptPage) { v.CompletedAt = nil }},
|
||||
{"wrong-budget", func(v *orbit.Task, _ *orbit.AttemptPage) { v.Retry.MaxAttempts = 2 }},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
v, h := completedFixture(cfg)
|
||||
tc.mutate(&v, &h)
|
||||
if _, code := checkCompletion(cfg, testRunID, 0, v, h); code == "" {
|
||||
t.Fatal("invalid completion accepted")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type stubAPI struct {
|
||||
submit func(context.Context, string, orbit.SubmitTaskRequest) (orbit.SubmitTaskResult, error)
|
||||
get func(context.Context, string) (orbit.Task, error)
|
||||
history func(context.Context, string, orbit.HistoryPageRequest) (orbit.AttemptPage, error)
|
||||
}
|
||||
|
||||
func (s stubAPI) SubmitTask(c context.Context, k string, v orbit.SubmitTaskRequest) (orbit.SubmitTaskResult, error) {
|
||||
return s.submit(c, k, v)
|
||||
}
|
||||
func (s stubAPI) GetTask(c context.Context, k string) (orbit.Task, error) { return s.get(c, k) }
|
||||
func (s stubAPI) ListTaskAttempts(c context.Context, k string, v orbit.HistoryPageRequest) (orbit.AttemptPage, error) {
|
||||
return s.history(c, k, v)
|
||||
}
|
||||
|
||||
func TestSubmissionIsBoundedDoesNotRetryOrCountAmbiguousReceipts(t *testing.T) {
|
||||
cfg := testConfig()
|
||||
var mu sync.Mutex
|
||||
keys := map[string]int{}
|
||||
active, peak := 0, 0
|
||||
api := stubAPI{submit: func(ctx context.Context, key string, in orbit.SubmitTaskRequest) (orbit.SubmitTaskResult, error) {
|
||||
mu.Lock()
|
||||
keys[key]++
|
||||
active++
|
||||
if active > peak {
|
||||
peak = active
|
||||
}
|
||||
mu.Unlock()
|
||||
defer func() { mu.Lock(); active--; mu.Unlock() }()
|
||||
var payload loadPayload
|
||||
if json.Unmarshal(in.Payload, &payload) != nil {
|
||||
t.Error("invalid sample payload")
|
||||
}
|
||||
if key != requestKey(testRunID, payload.Sequence) || in.Retry.MaxAttempts != 1 {
|
||||
t.Error("identity or retry budget changed")
|
||||
}
|
||||
if payload.Sequence == 0 {
|
||||
<-ctx.Done()
|
||||
return orbit.SubmitTaskResult{}, ctx.Err()
|
||||
}
|
||||
if payload.Sequence == 1 {
|
||||
return orbit.SubmitTaskResult{}, &orbit.APIError{StatusCode: 429, Code: "QUEUE_FULL"}
|
||||
}
|
||||
task := orbit.Task{ID: fmt.Sprintf("%032x", payload.Sequence), QueueID: in.QueueID, Handler: in.Handler, HandlerVersion: in.HandlerVersion, PayloadVersion: in.PayloadVersion, BusinessKey: in.BusinessKey, Payload: in.Payload, Retry: in.Retry, RecoverySafety: in.RecoverySafety}
|
||||
return orbit.SubmitTaskResult{Task: task}, nil
|
||||
}}
|
||||
cfg.Timeout = 30 * time.Millisecond
|
||||
results := submit(context.Background(), cfg, api, testRunID)
|
||||
if len(results) != cfg.Requests || len(keys) != cfg.Requests || peak > cfg.Concurrency {
|
||||
t.Fatal("lost requests or exceeded bound", len(results), len(keys), peak)
|
||||
}
|
||||
counts := map[string]int{}
|
||||
for _, a := range results {
|
||||
counts[a.code]++
|
||||
}
|
||||
if !reflect.DeepEqual(counts, map[string]int{"TIMEOUT": 1, "BACKPRESSURE": 1, "": 4}) {
|
||||
t.Fatal(counts)
|
||||
}
|
||||
for _, n := range keys {
|
||||
if n != 1 {
|
||||
t.Fatal("submission retried")
|
||||
}
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if len(submit(ctx, cfg, api, testRunID)) != 0 {
|
||||
t.Fatal("submitted after cancellation")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerificationCannotPromoteReviewTimeoutOrMissingHistory(t *testing.T) {
|
||||
cfg := testConfig()
|
||||
task, history := completedFixture(cfg)
|
||||
for _, state := range []orbit.TaskState{orbit.TaskNeedsReview, orbit.TaskFailed, orbit.TaskCancelled} {
|
||||
v := task
|
||||
v.State = state
|
||||
api := stubAPI{get: func(context.Context, string) (orbit.Task, error) { return v, nil }}
|
||||
if _, code := verify(context.Background(), cfg, api, testRunID, admission{task: task}); code != "TASK_"+string(state) {
|
||||
t.Fatal(state, code)
|
||||
}
|
||||
}
|
||||
waiting := task
|
||||
waiting.State = orbit.TaskRunning
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Millisecond)
|
||||
defer cancel()
|
||||
api := stubAPI{get: func(context.Context, string) (orbit.Task, error) { return waiting, nil }}
|
||||
if _, code := verify(ctx, cfg, api, testRunID, admission{task: task}); code != "VERIFY_TIMEOUT" {
|
||||
t.Fatal(code)
|
||||
}
|
||||
api.get = func(context.Context, string) (orbit.Task, error) { return task, nil }
|
||||
api.history = func(context.Context, string, orbit.HistoryPageRequest) (orbit.AttemptPage, error) {
|
||||
return orbit.AttemptPage{}, errors.New("private history error")
|
||||
}
|
||||
if _, code := verify(context.Background(), cfg, api, testRunID, admission{task: task}); code != "HISTORY_TRANSPORT" {
|
||||
t.Fatal(code)
|
||||
}
|
||||
api.history = func(context.Context, string, orbit.HistoryPageRequest) (orbit.AttemptPage, error) {
|
||||
return history, nil
|
||||
}
|
||||
if _, code := verify(context.Background(), cfg, api, testRunID, admission{task: task}); code != "" {
|
||||
t.Fatal(code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandlerDetectsDuplicatesAndForeignPayloadWithoutExternalEffects(t *testing.T) {
|
||||
cfg := testConfig()
|
||||
task, _ := completedFixture(cfg)
|
||||
log := newInvocationLog(cfg.Requests)
|
||||
h := log.handler(context.Background(), cfg, testRunID)
|
||||
result, err := h.Handle(context.Background(), orbit.WorkerExecution{Task: task})
|
||||
if err != nil || string(result) != string(task.Payload) {
|
||||
t.Fatal(err, string(result))
|
||||
}
|
||||
if _, err = h.Handle(context.Background(), orbit.WorkerExecution{Task: task}); err == nil {
|
||||
t.Fatal("duplicate execution accepted")
|
||||
}
|
||||
task.Payload = json.RawMessage(`{"run_id":"other","sequence":0,"duration_ms":0}`)
|
||||
if _, err = h.Handle(context.Background(), orbit.WorkerExecution{Task: task}); err == nil {
|
||||
t.Fatal("foreign sample accepted")
|
||||
}
|
||||
if log.calls[0] != 2 || log.invalid != 1 {
|
||||
t.Fatal("lost invocation evidence")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportDenominatorsAndRedaction(t *testing.T) {
|
||||
r := newReport(testConfig(), testRunID)
|
||||
r.Accepted = 5
|
||||
r.VerifiedSuccess = 4
|
||||
r.HandlerCalls = 6
|
||||
r = r.finish()
|
||||
if r.Complete || r.Unverified != 1 {
|
||||
t.Fatal("incomplete sample reported complete")
|
||||
}
|
||||
raw, _ := json.Marshal(r)
|
||||
if !strings.Contains(string(raw), `"p99_ms":null`) {
|
||||
t.Fatal("empty distribution not explicit")
|
||||
}
|
||||
d := distributionOf([]time.Duration{3 * time.Millisecond, time.Millisecond, 2 * time.Millisecond})
|
||||
if d.Count != 3 || *d.P50 != 2 || *d.P99 != 3 {
|
||||
t.Fatal(d)
|
||||
}
|
||||
for _, err := range []error{errors.New("secret"), &orbit.APIError{StatusCode: 503, Code: "secret", Message: "token"}} {
|
||||
if strings.Contains(classify(err), "secret") {
|
||||
t.Fatal("error leaked")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigBoundsAndDeterministicMix(t *testing.T) {
|
||||
t.Setenv("ORBIT_URL", "http://127.0.0.1:8080")
|
||||
t.Setenv("ORBIT_TOKEN", "test-token")
|
||||
t.Setenv("ORBIT_QUEUE_ID", testQueueID)
|
||||
t.Setenv("ORBIT_CREDENTIALS_FILE", "")
|
||||
cfg, err := parseConfig([]string{"-durations", "0ms,10ms,2s", "-requests", "3", "-concurrency", "8"})
|
||||
if err != nil || cfg.Concurrency != 3 || expected(cfg, testRunID, 5).DurationMS != 2000 {
|
||||
t.Fatal(cfg, err)
|
||||
}
|
||||
for _, args := range [][]string{{"-requests", "0"}, {"-workers", "32", "-capacity", "256"}, {"-durations", "-1s"}, {"-durations", "1us"}, {"-durations", "60s", "-run-timeout", "60s"}, {"-poll-interval", "1ms"}, {"-timeout", "2h"}, {"-run-timeout", "2h"}} {
|
||||
if _, err := parseConfig(args); err == nil {
|
||||
t.Fatal("invalid bounds accepted", args)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreflightDenialNeverSubmitsOrStartsWorkers(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var paths []string
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
paths = append(paths, r.URL.Path)
|
||||
mu.Unlock()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(403)
|
||||
fmt.Fprint(w, `{"request_id":"`+testRunID+`","error":{"code":"FORBIDDEN","message":"private data"}}`)
|
||||
}))
|
||||
defer server.Close()
|
||||
client, err := orbit.NewClient(server.URL, orbit.Options{Token: "test-token"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := measure(context.Background(), testConfig(), client, testRunID)
|
||||
if r.Complete || r.Attempted != 0 || r.NotSubmitted != 6 || r.Errors["PREFLIGHT_AUTH"] != 1 {
|
||||
t.Fatal(r)
|
||||
}
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if !reflect.DeepEqual(paths, []string{"/v1/queues/" + testQueueID}) {
|
||||
t.Fatal(paths)
|
||||
}
|
||||
raw, _ := json.Marshal(r)
|
||||
if strings.Contains(string(raw), "private data") || strings.Contains(string(raw), "test-token") {
|
||||
t.Fatal("private data leaked")
|
||||
}
|
||||
}
|
||||
121
sdk/go/cmd/orbit-exec-load/report.go
Normal file
121
sdk/go/cmd/orbit-exec-load/report.go
Normal file
@@ -0,0 +1,121 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"math"
|
||||
"runtime"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
orbit "git.yuntuo.hk/k2safe/orbit/sdk/go"
|
||||
)
|
||||
|
||||
// distribution 只计算明确纳入的样本;空集合的百分位返回null,不能冒充零延迟。
|
||||
type distribution struct {
|
||||
Count int `json:"count"` // Count 样本数;完成阶段仅包含通过持久化核验的成功任务。
|
||||
P50 *float64 `json:"p50_ms"` // P50 最近秩中位数,毫秒。
|
||||
P95 *float64 `json:"p95_ms"` // P95 最近秩95百分位,毫秒。
|
||||
P99 *float64 `json:"p99_ms"` // P99 最近秩99百分位,毫秒。
|
||||
Max *float64 `json:"max_ms"` // Max 样本最大耗时,毫秒。
|
||||
}
|
||||
|
||||
// executionReport 是单次有界测量结果,不是容量承诺或持久化审计。
|
||||
type executionReport struct {
|
||||
RunID string `json:"run_id"` // RunID 固定提交键和处理器身份的组成部分。
|
||||
GoVersion string `json:"go_version"` // GoVersion 实际编译工具链版本,不从仓库要求推断。
|
||||
GOOS string `json:"goos"` // GOOS 负载进程操作系统。
|
||||
GOARCH string `json:"goarch"` // GOARCH 负载进程架构。
|
||||
StartedAt time.Time `json:"started_at"` // StartedAt 本机开始时间,包含预检。
|
||||
AdmissionFinishedAt time.Time `json:"admission_finished_at"` // AdmissionFinishedAt 本机受理组结束时间;预检失败时为零值。
|
||||
FinishedAt time.Time `json:"finished_at"` // FinishedAt 核验及Worker收尾结束时间。
|
||||
Requests int `json:"requests"` // Requests 计划独立任务数。
|
||||
Concurrency int `json:"concurrency"` // Concurrency 提交及核验各自最大并发。
|
||||
Workers int `json:"workers"` // Workers 同一负载进程内独立WSS运行器数。
|
||||
Capacity int `json:"capacity_per_worker"` // Capacity 每个运行器物理处理器并发上限。
|
||||
DurationsMS []int64 `json:"durations_ms"` // DurationsMS 按序号循环选取的模拟等待时长。
|
||||
RunTimeoutMS int64 `json:"run_timeout_ms"` // RunTimeoutMS 整体期限,收尾最多另加5秒。
|
||||
PollIntervalMS int64 `json:"poll_interval_ms"` // PollIntervalMS 每个活跃核验器两次读取之间的间隔。
|
||||
HTTPTimeoutMS int64 `json:"http_timeout_ms"` // HTTPTimeoutMS 单次HTTP请求时限,不是任务时限。
|
||||
Queue *orbit.Queue `json:"queue,omitempty"` // Queue 预检时队列额度快照;工具不会修改配置。
|
||||
Attempted int `json:"attempted"` // Attempted 实际发起的提交调用数,不自动重试。
|
||||
NotSubmitted int `json:"not_submitted"` // NotSubmitted 截止/错误前未发起的项数。
|
||||
Accepted int `json:"accepted"` // Accepted 收到并核验唯一回执的项数。
|
||||
AdmissionErrors int `json:"admission_errors"` // AdmissionErrors 明确拒绝或提交结果无法确认的项数,不能当作全部未落库。
|
||||
VerifiedSuccess int `json:"verified_success"` // VerifiedSuccess 最终任务、尝试及结果均通过核验的任务数。
|
||||
Unverified int `json:"unverified"` // Unverified 已受理但未获核验成功的数量,包括失败/核查/仍待执行。
|
||||
HandlerCalls int `json:"handler_calls"` // HandlerCalls 本进程有效输入处理器调用数。
|
||||
DuplicateInvocations int `json:"duplicate_invocations"` // DuplicateInvocations 同一序号超过一次的调用数。
|
||||
UnconfirmedHandlerCalls int `json:"unconfirmed_handler_calls"` // UnconfirmedHandlerCalls 没有已确认受理回执的处理器调用数。
|
||||
WorkerStarted uint64 `json:"worker_started"` // WorkerStarted SDK本进程统计,不替代持久化成功数。
|
||||
Reconnects uint64 `json:"worker_reconnects"` // Reconnects SDK重连计数。
|
||||
Renewals uint64 `json:"worker_renewals_confirmed"` // Renewals SDK续租确认数。
|
||||
Admission distribution `json:"admission"` // Admission 全部提交HTTP交换的本机单调时钟耗时,含失败。
|
||||
QueueWait distribution `json:"queue_wait"` // QueueWait 数据库available_at至authorized_at,含领取及开始授权。
|
||||
AuthorizedExecution distribution `json:"authorized_execution"` // AuthorizedExecution 数据库authorized_at至ended_at,止于结束事务取时点,不包含其后提交确认。
|
||||
TaskCompletion distribution `json:"task_completion"` // TaskCompletion 数据库created_at至completed_at,不受客户端轮询顺序影响。
|
||||
Errors map[string]int `json:"errors"` // Errors 固定分类,不保存内部异常、SQL、凭据或请求原文。
|
||||
Complete bool `json:"complete"` // Complete 仅所有计划任务和本地调用逐项核验且无错误时为true。
|
||||
}
|
||||
|
||||
func newReport(cfg config, runID string) executionReport {
|
||||
r := executionReport{RunID: runID, GoVersion: runtime.Version(), GOOS: runtime.GOOS, GOARCH: runtime.GOARCH, StartedAt: time.Now().UTC(), Requests: cfg.Requests, NotSubmitted: cfg.Requests, Concurrency: cfg.Concurrency, Workers: cfg.Workers, Capacity: cfg.Capacity, RunTimeoutMS: cfg.RunTimeout.Milliseconds(), PollIntervalMS: cfg.PollInterval.Milliseconds(), HTTPTimeoutMS: cfg.Timeout.Milliseconds(), Errors: map[string]int{}}
|
||||
for _, d := range cfg.Durations {
|
||||
r.DurationsMS = append(r.DurationsMS, d.Milliseconds())
|
||||
}
|
||||
return r
|
||||
}
|
||||
func (r executionReport) finish() executionReport {
|
||||
for code, count := range r.Errors {
|
||||
if count == 0 {
|
||||
delete(r.Errors, code)
|
||||
}
|
||||
}
|
||||
r.FinishedAt = time.Now().UTC()
|
||||
r.Unverified = r.Accepted - r.VerifiedSuccess
|
||||
r.Complete = r.Attempted == r.Requests && r.NotSubmitted == 0 && r.AdmissionErrors == 0 && r.Accepted == r.Requests && r.VerifiedSuccess == r.Requests && r.HandlerCalls == r.Requests && r.WorkerStarted == uint64(r.Requests) && r.DuplicateInvocations == 0 && r.UnconfirmedHandlerCalls == 0 && len(r.Errors) == 0
|
||||
return r
|
||||
}
|
||||
func distributionOf(values []time.Duration) distribution {
|
||||
d := distribution{Count: len(values)}
|
||||
if len(values) == 0 {
|
||||
return d
|
||||
}
|
||||
sort.Slice(values, func(i, j int) bool { return values[i] < values[j] })
|
||||
percentile := func(q float64) *float64 {
|
||||
v := float64(values[int(math.Ceil(float64(len(values))*q))-1]) / float64(time.Millisecond)
|
||||
return &v
|
||||
}
|
||||
d.P50 = percentile(.5)
|
||||
d.P95 = percentile(.95)
|
||||
d.P99 = percentile(.99)
|
||||
d.Max = percentile(1)
|
||||
return d
|
||||
}
|
||||
func classify(err error) string {
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
return "TIMEOUT"
|
||||
}
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return "CANCELLED"
|
||||
}
|
||||
var api *orbit.APIError
|
||||
if errors.As(err, &api) {
|
||||
switch api.StatusCode {
|
||||
case 401, 403:
|
||||
return "AUTH"
|
||||
case 409:
|
||||
return "CONFLICT"
|
||||
case 429:
|
||||
return "BACKPRESSURE"
|
||||
case 400, 422:
|
||||
return "INVALID_REQUEST"
|
||||
}
|
||||
return "HTTP"
|
||||
}
|
||||
var h *orbit.HTTPError
|
||||
if errors.As(err, &h) {
|
||||
return "HTTP"
|
||||
}
|
||||
return "TRANSPORT"
|
||||
}
|
||||
Reference in New Issue
Block a user