feat: persist workflow controls and settle stopped branches
This commit is contained in:
@@ -31,19 +31,22 @@
|
||||
|
||||
- P4定义发布使用schema_version=1有限结构，词法作用域禁止跨分支/前向读取，任务策略显式冻结；SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁；原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务；同版本不同内容不得覆盖。保留精确数字，校验存储原文与结构摘要；不得把发布通过或DSL支持误称为实例执行已实现。
|
||||
|
||||
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归（docs/reports/phase-4-definitions.md）。实例启动/固定版本/首批任务同事务的内部存储已补齐；内部事件消费、顺序与组合推进及根接续已接入；TIMER与后续POLL时钟已接入，SIGNAL及最终查询竞争已接入，继续完整控制及实例公共SDK；P4整体和P5–P7仍未完成。
|
||||
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归（docs/reports/phase-4-definitions.md）。实例启动/固定版本/首批任务同事务的内部存储已补齐；内部事件消费、顺序与组合推进及根接续已接入；TIMER与后续POLL时钟已接入，SIGNAL及最终查询竞争已接入，内部控制与停止结清已接入，继续人工恢复及实例公共SDK；P4整体和P5–P7仍未完成。
|
||||
|
||||
- 内部结果交付使用workflow_activation所有者并固定激活ID，TASK经activation.task_id、POLL经轮次校验归属；weave_inbox来源及独立快照与Pulse结果同事务，先于外部结果流行锁，不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史，禁止用当前任务重建旧事件。
|
||||
- weave/engine仅计算有限决策；实例受理、Inbox消费与根接续已接入，完整等待/控制运行器继续开发。Plan最多32个变更，More必须持久化待续；耗尽变更预算仍须遍历其他已活动分支，不能误判在途工作不可达。冻结分支选择，精确数值，不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务，不能把纯快照往返测试当作真实进程恢复。
|
||||
- weave/engine仅计算有限决策；实例受理、Inbox消费与根接续已接入，等待和停止结清运行器已接入，人工恢复继续开发。Plan最多32个变更，More必须持久化待续；耗尽变更预算仍须遍历其他已活动分支，不能误判在途工作不可达。冻结分支选择，精确数值，不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务，不能把纯快照往返测试当作真实进程恢复。
|
||||
|
||||
- 实例内部受理已实现，信号内部运行器已接入，公共实例/信号HTTP及独立SDK尚未开放；结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作，所有失败同事务回滚。首批推进递增revision至2，原键返回冻结回执且仍核验当前权限，不重算截止。
|
||||
- 实例根步骤`$root`/JOIN不属于用户定义，Steps查询排除它；More对应根步骤版本化持久检查点。TASK直接关联激活task_id，POLL通过轮次关联，实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结，旧NULL等待须核实，不能盲目解释；新关联512字节、存储兼容历史764字节。
|
||||
|
||||
- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁，禁止先锁Inbox再锁父资源。来源使用独立冻结快照，旧激活/低版本留痕；数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。
|
||||
- 后继满额用SAVEPOINT回滚整轮新工作，保留已接受结果和待续；恢复必须重读持久图。多次根推进使用独立系统审计操作身份；迟到证据也递增实例历史版本，禁止复用已有workflow.advanced唯一身份。
|
||||
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证，见docs/reports/phase-4-runtime.md。TIMER到期与POLL后续轮次已接入，见docs/13-workflow-clock.md；SIGNAL及最终信号竞争已接入（docs/14-workflow-signals.md）；FAIL_AFTER_SETTLED分支结清、控制API和实例SDK仍须完成，不得按部分自动运行宣称P4/P5完成。
|
||||
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证，见docs/reports/phase-4-runtime.md。TIMER到期与POLL后续轮次已接入，见docs/13-workflow-clock.md；SIGNAL及最终信号竞争已接入（docs/14-workflow-signals.md）；内部FAIL_AFTER_SETTLED分支结清与控制已接入（docs/15-workflow-controls.md）；人工恢复、公共控制API和实例SDK仍须完成，不得按部分自动运行宣称P4/P5完成。
|
||||
|
||||
- TIMER/POLL复用weave_pending_advances及到期索引，时钟固定激活ID；迁移000019只回填原等待/查询时钟，不改截止和预算。后续POLL只由已确认POLL_PENDING开启，首轮精确输入/资源键冻结，每轮Pulse尝试独立计数；满额/暂停不消耗轮数。截止遇到已落库结果先等待Inbox，仍在途则关闭门控但保留ACTIVE和当前任务，许可由Pulse结清；不得由时钟假定外部请求已结束。
|
||||
|
||||
- SIGNAL受理要求显式workflows:signal、固定目标步骤/激活代次/内容契约/精确关联及原因；每实例全部历史信号上限1024，原键先返回冻结回执。accepted_at/expires_at及摘要与不可变审计核验；available_at只用于到期检查退避。
|
||||
- 最终信号与当前查询按原受理/事件时间裁决，PENDING不争夺终结权；保存信号赢家后复用Pulse取消规则，在途任务保留许可直至真实结果。未知/冲突结果转核查，已有核查不得自动清除。迁移000020隔离旧PENDING身份并保留历史原文，不引入外键。
|
||||
|
||||
- 流程控制使用显式workflows:control和独立control_revision；后台结果只改变通用revision。控制标记、门控、原操作回执、根接续、事件与审计同事务，原键恢复仍核对当前权限。RESUME不解除核查或停止意图。
|
||||
- StopKind/StopRequestedAt冻结首次CANCEL或FAILURE目标，后来的用户取消不覆盖原失败目标。结清复用ROOT，每事务最多一个Pulse取消、32项步骤收尾；结果流锁取得后不能再锁另一任务。终态前校验全部历史任务的双向归属、未结束尝试和未释放许可，UNKNOWN不得终结。迁移000021恢复旧停止意图，不添加外键。
|
||||
|
||||
@@ -10,7 +10,7 @@ Orbit 是面向多个业务项目的通用任务调度与流程编排平台，
|
||||
|
||||
已完成 P0–P3 平台功能阶段验收：35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过；[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK，[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例，[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现；[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例；[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例；[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景；[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本；[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI；[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复；完整范围核对见[P3验收](docs/reports/phase-3.md)，流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。
|
||||
|
||||
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现；已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。[定时等待、后续查询轮次及原预算恢复](docs/13-workflow-clock.md)已接入；[外部信号和查询最终信号竞争](docs/14-workflow-signals.md)已接入内部运行器；失败结清及实例公共API/SDK继续开发，发布定义不会启动任务。
|
||||
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现；已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。[定时等待、后续查询轮次及原预算恢复](docs/13-workflow-clock.md)已接入；[外部信号和查询最终信号竞争](docs/14-workflow-signals.md)已接入内部运行器；[暂停/继续/取消与失败结清](docs/15-workflow-controls.md)已接入内部运行器；人工恢复及实例公共API/SDK继续开发，发布定义不会启动任务。
|
||||
|
||||
数据库禁止外键，关联一致性通过服务端事务校验，索引、唯一约束和 CHECK 保留；readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。
|
||||
|
||||
@@ -34,6 +34,7 @@ weave/definition/ 有限结构/类型校验与定义发布契约
|
||||
weave/engine/ 不可变规则的有限步骤决策与精确结果映射
|
||||
weave/instance/ 内部实例受理/查询契约，公共运行API继续开发
|
||||
weave/signal/ 内部可靠信号受理/查询契约及显式激活身份
|
||||
weave/control/ 内部版本化暂停/继续/取消契约
|
||||
internal/config/ 配置校验
|
||||
internal/storage/mysql/ 连接、事务、迁移及集成测试
|
||||
internal/transport/httpserver/ HTTP 接入
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 流程编排功能与设计
|
||||
|
||||
> 版本 0.5 · 2026-09-18。首版功能范围已确认。定义结构、发布校验与不可变版本已实现，见[定义格式](api/workflow-definition-format.md)和[发布API](api/workflow-definitions.md)；[实例受理](11-workflow-admission.md)、[结果消费与根推进](12-workflow-runtime.md)、[TIMER与POLL时钟](13-workflow-clock.md)已接入。[信号等待及最终查询竞争](14-workflow-signals.md)已接入内部运行器；完整管理及实例公共API/SDK仍待P4–P5实现验收。
|
||||
> 版本 0.6 · 2026-09-18。首版功能范围已确认。定义结构、发布校验与不可变版本已实现，见[定义格式](api/workflow-definition-format.md)和[发布API](api/workflow-definitions.md)；[实例受理](11-workflow-admission.md)、[结果消费与根推进](12-workflow-runtime.md)、[TIMER与POLL时钟](13-workflow-clock.md)已接入。[信号等待及最终查询竞争](14-workflow-signals.md)已接入内部运行器；[暂停/继续/取消与停止结清](15-workflow-controls.md)已实现内部链路；人工核查及实例公共API/SDK仍待P4–P5实现验收。
|
||||
|
||||
## 1. 职责与边界
|
||||
|
||||
@@ -152,7 +152,7 @@ Weave 不执行特定业务逻辑，也不把整段业务代码放入平台进
|
||||
|
||||
**功能已确认**：暂停阻止新步骤推进，保留已完成进度，继续接收在途结果与外部回执；继续操作从持久化进度恢复。暂停不表示已经停止外部业务操作。
|
||||
|
||||
以下细则为设计建议：
|
||||
以下细则已按[控制运行约定](15-workflow-controls.md)冻结并实现内部链路，公共HTTP/SDK仍待交付：
|
||||
|
||||
- 暂停与继续带预期控制版本、稳定操作身份、操作人及原因；状态与审计一起提交。新步骤创建必须原子检查控制版本。
|
||||
- 建议暂停还阻止关联任务的新领取、开始授权和执行重试，包含已经创建但尚未开始的任务。由 Pulse 的通用执行门控与 Weave 控制状态同事务更新，领取/开始时读取权威门控，避免仅暂停流程循环却继续派发旧队列任务。
|
||||
|
||||
@@ -203,4 +203,8 @@ Wait新增SignalName和PayloadVersion两个中文注释字段；correlation_key
|
||||
|
||||
## 信号身份与赢家增量（000020）
|
||||
|
||||
Signal新增payload_version、target_activation_sequence、evidence_ref、disposition、available_at；Wait新增winner_signal_id。当前35个实体、含公共Record的301个直接持久化字段声明，全部中文注释。新信号精确关联最多512字节，存储兼容历史764字节；过期检查退避不改变expires_at。赢家唯一约束只防重复，实例/等待/激活/信号归属及不可变受理证明仍由事务验证，外键保持0。历史不完整身份隔离和升级回填见[信号约定](14-workflow-signals.md)。
|
||||
Signal新增payload_version、target_activation_sequence、evidence_ref、disposition、available_at；Wait新增winner_signal_id。截至000020为35个实体、含公共Record的301个直接持久化字段声明，全部中文注释。新信号精确关联最多512字节，存储兼容历史764字节；过期检查退避不改变expires_at。赢家唯一约束只防重复，实例/等待/激活/信号归属及不可变受理证明仍由事务验证，外键保持0。历史不完整身份隔离和升级回填见[信号约定](14-workflow-signals.md)。
|
||||
|
||||
## 停止意图增量（000021）
|
||||
|
||||
Instance新增StopKind和StopRequestedAt，冻结首次用户取消或失败结清目标；CancelRequestedAt仍只表示用户显式取消。成对CHECK防止半条意图，任务所有者索引支持双向归属核验。当前35个实体、303个直接持久化字段声明，均有中文注释。历史停止实例关闭门控并恢复根接续，不改变原截止、预算、控制版本或终态数据；保持零外键，详见[控制约定](15-workflow-controls.md)。
|
||||
|
||||
@@ -28,8 +28,8 @@
|
||||
| P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件；SDK 客户端 | API/SDK 创建查询闭环；重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归（见 P1 报告；真实断路/提交 ACK 丢失注入留后续故障阶段） |
|
||||
| P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复；Worker SDK | 真 Worker 执行；失联/重连/重复确认/旧尝试拒绝；业务效果未知转核查；节点故障接管 | 已完成并通过阶段回归：服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复；测试拓扑和剩余生产故障矩阵见 P2 验收报告 |
|
||||
| P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收：[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景；20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 |
|
||||
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容；端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现，已通过[子项验收](reports/phase-4-definitions.md)；已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md)；[实例受理及首批事务](reports/phase-4-admission.md)已实现；[结果消费和根接续](reports/phase-4-runtime.md)已接入后台，顺序/分支/并行及真实进程恢复已验证；[TIMER与后续POLL轮次](13-workflow-clock.md)已接入；[信号消费和最终查询竞争](14-workflow-signals.md)已接入内部运行器；完整控制和实例公共API/SDK继续开发 |
|
||||
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算；暂停/开始/回执/超时竞争；恢复保留预算；未知结果不盲目重复 | 有界查询轮次/尝试独立预算及原截止恢复已接入内部运行器；最终信号竞争及在途查询结清已接入；完整控制/人工恢复及公共API/SDK继续开发 |
|
||||
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容；端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现，已通过[子项验收](reports/phase-4-definitions.md)；已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md)；[实例受理及首批事务](reports/phase-4-admission.md)已实现；[结果消费和根接续](reports/phase-4-runtime.md)已接入后台，顺序/分支/并行及真实进程恢复已验证；[TIMER与后续POLL轮次](13-workflow-clock.md)已接入；[信号消费和最终查询竞争](14-workflow-signals.md)已接入内部运行器；[内部控制与停止结清](15-workflow-controls.md)已接入；人工恢复和实例公共API/SDK继续开发 |
|
||||
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算；暂停/开始/回执/超时竞争；恢复保留预算；未知结果不盲目重复 | 有界查询轮次/尝试独立预算及原截止恢复已接入内部运行器；最终信号竞争及在途查询结清已接入；内部暂停/继续/取消与失败结清已接入；人工恢复及公共API/SDK继续开发 |
|
||||
| P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行，报告环境/指标/遗留项；容量、RPO/RTO 按真实拓扑验证 | 待开发 |
|
||||
| P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例；所有首版能力可追溯到实现与测试；报告实际生产边界 | 待开发 |
|
||||
|
||||
@@ -39,8 +39,8 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
||||
|
||||
### P4/P5 下一步实施顺序
|
||||
|
||||
1. 完成FAIL_AFTER_SETTLED与显式取消的分支结清：持久取消意图、复用Pulse取消、收集已发出查询/任务结果、等待全部已激活工作结清后才进入失败或取消终态。效果未知仍保留核查。
|
||||
2. 完成实例暂停/继续/取消内部操作，以control_revision、操作幂等身份、明确原因及审计保护控制决定；暂停不延长原截止，继续不覆盖核查或取消状态。
|
||||
1. 已实现FAIL_AFTER_SETTLED与显式取消的内部分支结清，见[控制报告](reports/phase-5-controls.md)：持久取消意图、复用Pulse取消、收集已发出查询/任务结果、等待全部已激活工作结清后才进入失败或取消终态。效果未知仍保留核查。
|
||||
2. 已实现实例暂停/继续/取消内部操作，以control_revision、操作幂等身份、明确原因及审计保护控制决定；暂停不延长原截止，继续不覆盖核查或取消状态。
|
||||
3. 完成人工核查恢复与旧激活隔离，保留原预算、结果和证据，不能通过换激活/新键绕过未知效果或执行上限。
|
||||
4. 同步开放实例/步骤/信号/来源/审计HTTP、独立SDK和端到端场景，覆盖原键恢复、权限、版本冲突、竞争与进程全停；公共能力逐项记录实际验收结果。
|
||||
5. 转入P6/P7的生产故障、容量、运维、发布与全部文档核对。内部存储阶段通过不会提前勾选公共接口或生产验收。
|
||||
|
||||
@@ -74,7 +74,7 @@ v16及以前的未处理记录缺少冻结来源，升级后统一标为QUARANTI
|
||||
|
||||
## 4. 后续存储适配必须满足的条件
|
||||
|
||||
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证；第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md)，第5项的TIMER/POLL已接入[时钟运行器](13-workflow-clock.md)，第5项的信号等待已接入[信号运行器](14-workflow-signals.md)，完整控制、接口和SDK仍继续实现：
|
||||
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证；第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md)，第5项的TIMER/POLL已接入[时钟运行器](13-workflow-clock.md)，第5项的信号等待已接入[信号运行器](14-workflow-signals.md)，内部控制与结清已接入[控制运行器](15-workflow-controls.md)，人工恢复、接口和SDK仍继续实现：
|
||||
|
||||
1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止，并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。
|
||||
2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本；消费确认、步骤变化、后续任务或持久化待续记录同事务提交。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 实例受理与首批工作
|
||||
|
||||
状态：内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口；完整控制仍继续开发；[信号运行器](14-workflow-signals.md)已接入。** Weave相关公开SDK目前仅支持定义发布。[TIMER和后续POLL轮次](13-workflow-clock.md)已接入，TASK及组合节点已可自动推进，不能将此解释成全部等待/控制能力已实现。
|
||||
状态：内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口；[内部控制和结清](15-workflow-controls.md)已接入，人工恢复继续开发；[信号运行器](14-workflow-signals.md)已接入。** Weave相关公开SDK目前仅支持定义发布。[TIMER和后续POLL轮次](13-workflow-clock.md)已接入，TASK及组合节点已可自动推进，不能将此解释成全部等待/控制能力已实现。
|
||||
|
||||
## 1. 受理事务
|
||||
|
||||
@@ -49,7 +49,7 @@ Get返回当前实例视图；Start重报返回受理时冻结的原始视图，
|
||||
| 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例；终态不计入；满额仍可恢复原受理回执 |
|
||||
| 信号关联 | 新受理最多512字节；数据库兼容旧191字符最长764字节，升级不截断历史 |
|
||||
| 截止 | 数据库时间生成；任务、查询与等待均不超过实例总截止 |
|
||||
| 自动运行 | Inbox消费、根待续、TIMER到期及后续POLL轮次已接入；SIGNAL与最终查询竞争已接入，完整控制仍待实现 |
|
||||
| 自动运行 | Inbox消费、根待续、TIMER到期及后续POLL轮次已接入；SIGNAL与最终查询竞争已接入，内部控制与结清已接入，人工恢复仍待实现 |
|
||||
|
||||
迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段；NULL仅表示TIMER或尚未核实的历史元数据，未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储；实例business_key改为utf8mb4_0900_bin。该迁移时为35个实体、含公共Record的295个直接持久化字段声明，全部有中文注释，继续保持零外键。
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 结果消费与自动推进
|
||||
|
||||
状态：内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。[定时等待和后续查询轮次](13-workflow-clock.md)也已接入。[信号等待与最终查询竞争](14-workflow-signals.md)已接入。实例公共HTTP/SDK仍未开放，失败后的分支取消结清及人工恢复继续开发，不能将本阶段等同于完整Weave交付。
|
||||
状态：内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。[定时等待和后续查询轮次](13-workflow-clock.md)也已接入。[信号等待与最终查询竞争](14-workflow-signals.md)已接入。实例公共HTTP/SDK仍未开放，[失败后的分支取消结清与内部控制](15-workflow-controls.md)已接入，人工恢复继续开发，不能将本阶段等同于完整Weave交付。
|
||||
|
||||
## 1. 后台运行和事务边界
|
||||
|
||||
@@ -43,7 +43,7 @@ flowchart LR
|
||||
|
||||
后继创建使用SAVEPOINT隔离本轮结构变更。多后继中任何一个遇到队列或应用满额，全部本轮后继和对应额度预占回滚；已接受的来源结果及步骤完成仍能与ROOT待续记录一起提交。下一次尝试从当前图重新计算，不能留下部分激活、重复后继或多扣用量。持续满额仅移动检查时间，不逐秒制造相同业务审计。
|
||||
|
||||
暂停、关闭门控或停用应用/命名空间时，继续收集已经落库的结果，但不激活新步骤；若还有后继，将其保存为待续。重新允许推进后，从持久记录继续。测试通过直接设置控制状态验证该行为；暂停/继续公共接口尚未交付。
|
||||
暂停、关闭门控或停用应用/命名空间时，继续收集已经落库的结果，但不激活新步骤；若还有后继，将其保存为待续。重新允许推进后，从持久记录继续。测试通过直接设置控制状态验证该行为；内部暂停/继续控制已按[控制运行约定](15-workflow-controls.md)实现，公共HTTP/SDK尚未交付。
|
||||
|
||||
已过实例总截止时不创建后继，未能全部完成则进入核查。最后的有效任务结果在截止前已由Pulse接受，而消费发生较晚时，可以整理完整结果并完成实例。暂停或重启不延长截止。TIMER与POLL的时钟已接入；[SIGNAL及其超期/最终查询裁决](14-workflow-signals.md)已接入。
|
||||
|
||||
@@ -56,7 +56,7 @@ flowchart LR
|
||||
- 轮次耗尽、查询截止、未知/非法值或单轮请求失败：进入核查，不新建轮次绕过Pulse重试预算。
|
||||
- 业务明确失败：按固定实例失败策略处理；REVIEW进入核查，FAIL_AFTER_SETTLED进入CANCELLING并关闭门控。
|
||||
|
||||
下一轮查询派发及到期扫描已实现，详见[时钟、预算与恢复](13-workflow-clock.md)。**尚未实现FAIL_AFTER_SETTLED的剩余分支取消/收集/失败终态判定。** CANCELLING保留未结清事实，不提前宣称FAILED。信号抢先到达时保留当前查询，实际结果结清及竞争见[信号运行约定](14-workflow-signals.md)。
|
||||
下一轮查询派发及到期扫描已实现，详见[时钟、预算与恢复](13-workflow-clock.md)。**FAIL_AFTER_SETTLED的剩余分支取消/收集/失败终态判定已接入[停止运行器](15-workflow-controls.md)。** CANCELLING保留未结清事实，只有全部已知结清后才能进入FAILED。信号抢先到达时保留当前查询，实际结果结清及竞争见[信号运行约定](14-workflow-signals.md)。
|
||||
|
||||
## 5. 回归与后续工作
|
||||
|
||||
@@ -64,4 +64,4 @@ flowchart LR
|
||||
|
||||
进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过；跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。
|
||||
|
||||
持久化信号等待及查询最终信号竞争已按[信号运行约定](14-workflow-signals.md)实现。下一步仍包括完整控制与人工恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
||||
持久化信号等待及查询最终信号竞争已按[信号运行约定](14-workflow-signals.md)实现。内部控制及停止结清也已接入。下一步仍包括人工恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 持久化定时等待与查询轮次
|
||||
|
||||
状态：TIMER到期推进和POLL后续业务轮次已接入Orbit后台运行器。复用MySQL和既有待续索引，不增加外部调度服务或数据库外键。[SIGNAL消费与查询最终信号竞争](14-workflow-signals.md)已进一步接入；实例公共HTTP/SDK以及完整控制/人工恢复仍未交付。
|
||||
状态：TIMER到期推进和POLL后续业务轮次已接入Orbit后台运行器。复用MySQL和既有待续索引，不增加外部调度服务或数据库外键。[SIGNAL消费与查询最终信号竞争](14-workflow-signals.md)已进一步接入；内部控制及停止结清已按[控制约定](15-workflow-controls.md)接入；实例公共HTTP/SDK以及人工恢复仍未交付。
|
||||
|
||||
## 1. 时钟和恢复身份
|
||||
|
||||
@@ -60,4 +60,4 @@ TIMER由待续表直接唤醒，不创建Pulse业务任务，也不占Worker执
|
||||
|
||||
[存储回归](../internal/storage/mysql/workflow_clock_integration_test.go)覆盖到期、竞争、预算、暂停/满额、回滚、归属损坏与未知结果；[v18升级回归](../internal/storage/mysql/workflow_clock_upgrade_integration_test.go)验证回填及原预算保留；[真实进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖全部控制进程退出后的定时和查询接续。执行记录见[阶段报告](reports/phase-4-clock.md)。
|
||||
|
||||
SIGNAL早到/重复/超期处理及最终查询竞争已接入；下一阶段继续失败分支结清、公共控制及实例SDK，完整范围见[持续计划](08-development-plan.md)。
|
||||
SIGNAL早到/重复/超期处理及最终查询竞争已接入；内部失败分支结清和控制已接入；下一阶段继续人工恢复、公共控制及实例SDK，完整范围见[持续计划](08-development-plan.md)。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 外部信号与最终查询结果
|
||||
|
||||
状态：内部 Go 信号受理/查询、SIGNAL 等待、早到保存、截止处理及 POLL 最终信号竞争已接入 MySQL 和后台运行器。**实例与信号的公共 HTTP/独立 SDK 尚未开放**；完整控制、人工恢复和失败分支结清继续按[持续计划](08-development-plan.md)实现。定义发布 SDK 不等于实例运行 SDK。
|
||||
状态：内部 Go 信号受理/查询、SIGNAL 等待、早到保存、截止处理及 POLL 最终信号竞争已接入 MySQL 和后台运行器。**实例与信号的公共 HTTP/独立 SDK 尚未开放**；内部控制和失败分支结清已按[控制约定](15-workflow-controls.md)接入；人工恢复继续按[持续计划](08-development-plan.md)实现。定义发布 SDK 不等于实例运行 SDK。
|
||||
|
||||
## 1. 受理身份与权限
|
||||
|
||||
@@ -55,7 +55,7 @@
|
||||
|
||||
## 4. 存储升级和运行边界
|
||||
|
||||
[迁移000020](../internal/storage/mysql/migrations/000020_workflow_signal_identity.up.sql)增加信号契约/代次/证据/处置/到期检查时间，以及等待的赢家引用。名称扩至128字符，关联保持精确字节，并兼容历史764字节值。35个实体、301个直接持久化字段声明均提供中文说明；数据库无外键、无级联删除。
|
||||
[迁移000020](../internal/storage/mysql/migrations/000020_workflow_signal_identity.up.sql)增加信号契约/代次/证据/处置/到期检查时间，以及等待的赢家引用。名称扩至128字符，关联保持精确字节，并兼容历史764字节值。截至000020为35个实体、301个直接持久化字段声明，均提供中文说明；数据库无外键、无级联删除。
|
||||
|
||||
历史信号没有足够身份和受理证明：旧PENDING标记REJECTED/LEGACY_UNVERIFIED，保留原载荷和时间；已消费历史不被重新解释。旧NULL等待元数据不能推测成有效新契约。迁移为当前SIGNAL等待回填原截止的检查点，保留已有检查点，排除终态实例及旧激活。
|
||||
|
||||
@@ -65,4 +65,4 @@
|
||||
|
||||
[受理回归](../internal/storage/mysql/workflow_signals_integration_test.go)、[消费回归](../internal/storage/mysql/workflow_signal_runtime_integration_test.go)、[真实v19升级](../internal/storage/mysql/workflow_signal_upgrade_integration_test.go)和[实际进程恢复](../internal/storage/mysql/workflow_runtime_process_integration_test.go)对应受理/并发/故障/升级边界，实际运行结果记录在[阶段报告](reports/phase-4-signals.md)。
|
||||
|
||||
下一步完成失败分支取消结清、实例暂停/继续/取消/人工核查恢复，再交付实例与信号HTTP、独立SDK及完整场景。当前内部存储测试不替代尚未开放的公共接口验收，也不替代生产MySQL切换与容量测试。
|
||||
内部失败分支取消结清、实例暂停/继续/取消已接入。下一步完成人工核查恢复，再交付实例与信号HTTP、独立SDK及完整场景。当前内部存储测试不替代尚未开放的公共接口验收，也不替代生产MySQL切换与容量测试。
|
||||
|
||||
63
docs/15-workflow-controls.md
Normal file
63
docs/15-workflow-controls.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# Weave 控制与停止结清
|
||||
|
||||
状态：内部Go暂停、继续、取消服务及后台失败/取消结清运行器已接入。**公共实例/控制HTTP与独立SDK尚未开放，人工核查恢复仍在后续交付范围**。本文说明已实现的持久化规则，不把内部存储回归当作全部Weave或生产验收。
|
||||
|
||||
## 1. 控制操作
|
||||
|
||||
[内部控制契约](../weave/control/service.go)使用`Apply(token, workflowID, idempotencyKey, input)`，显式授权`workflows:control`。每次读取当前命名空间、固定定义版本及其全部队列权限，原键恢复也重新核验权限。actor由认证凭据确定，输入只包含以下字段：
|
||||
|
||||
| 字段 | 语义 |
|
||||
| --- | --- |
|
||||
| operation | PAUSE、RESUME或CANCEL |
|
||||
| expected_control_revision | 调用者观察到的控制版本，正整数，对外JSON为十进制字符串 |
|
||||
| reason | 必填操作原因，最多1024个Unicode字符，写入不可变审计 |
|
||||
|
||||
后台结果会增加实例revision，不能使操作者观察到的control_revision失效。成功控制操作同时增加两个版本；同键相同内容返回首次冻结回执，不再次增加版本或改时间。同键不同内容冲突，其他键携带过期控制版本也冲突。新操作不接受终态实例；已提交原键仍可恢复原回执。
|
||||
|
||||
认证 → 应用额度 → 原操作身份 → 排序队列 → 实例 → 门控的持锁顺序与受理、运行器一致。控制标记、版本、门控、必要的根检查点、事件、审计和回执同事务提交。审计失败时整个操作回滚。
|
||||
|
||||
## 2. 暂停与继续
|
||||
|
||||
PAUSE设置实例paused并关闭Pulse执行门控。已创建但尚未开始的任务不能新领取或取得开始授权；已经获得开始授权的尝试保留许可，按原租约继续上报。暂停与开始竞争由同一个应用/队列/门控事务顺序裁决。
|
||||
|
||||
暂停期间继续保存任务结果、信号与到期事实，不启动新步骤或新查询轮次；待推进工作保存根检查点。RESUME仅解除paused，并保存根接续；实例处于核查或已有停止意图时，门控仍关闭。它不清除未知结果、不撤回取消、不改变固定定义，也不重置截止、轮次或尝试预算。
|
||||
|
||||
已暂停时使用新键再次PAUSE、未暂停时RESUME、已接受用户取消后使用新键再次CANCEL都返回状态冲突；网络未确认时应使用原键核对。原控制回执不会因为稍后的后台结果变化而变成当前视图，最新状态用实例Get读取。
|
||||
|
||||
## 3. 首次停止意图
|
||||
|
||||
停止意图与执行状态独立保存，进入NEEDS_REVIEW后仍能继续结清其他已发出的工作。
|
||||
|
||||
| 字段 | 语义 |
|
||||
| --- | --- |
|
||||
| stop_kind | CANCEL为用户请求，FAILURE为固定FAIL_AFTER_SETTLED策略触发；首次保存后不覆盖 |
|
||||
| stop_requested_at | 首次停止意图的数据库时间，不随重报、核查或后来的控制操作变化 |
|
||||
| cancel_requested_at | 用户显式请求取消的时间；自动失败结清不冒充用户请求 |
|
||||
|
||||
用户取消先提交时，最终按CANCEL目标结清；失败策略先触发时，后来的用户取消仍保留FAILURE目标，最终已知结清后为FAILED。两个时间和操作审计都保留，不用后到命令改写早先失败事实。
|
||||
|
||||
REVIEW策略继续保留核查状态和已发出结果；FAIL_AFTER_SETTLED在失败或核查条件出现时冻结停止意图，阻止新工作并请求其他分支结束。未知效果、非法结果和冲突证据始终保留NEEDS_REVIEW，不能为进入终态自动判定业务成功或失败。
|
||||
|
||||
## 4. 结清过程
|
||||
|
||||
停止使用既有ROOT检查点恢复。每个短事务最多发出一个Pulse任务取消，保持结果流锁取得之后不再锁另一任务；步骤收尾最多32项，超出保存根接续。候选由数据库到期索引读取，多个节点竞争仍只提交一次状态变化。
|
||||
|
||||
1. 核对任务实际所有者、激活、步骤、队列、门控及POLL轮次，不能仅凭gate_id授权取消。未开始的任务复用Pulse取消规则立即结清；执行中的任务只记录取消意图，保留执行许可。
|
||||
2. 收集内部Inbox结果。已成功步骤及输出保留；已知失败或取消保留实际状态；未知结果转核查。查询成功返回PENDING时，停止中的POLL结清为取消，不再发下一轮请求。
|
||||
3. 尚未激活步骤记为SKIPPED，不创建新任务。取消前可靠受理的匹配信号先保留为已收到事实；新的信号保存WORKFLOW_STOPPING拒绝。未结清的纯等待由停止运行器关闭，稍后超时扫描不能覆盖已提交的停止意图。
|
||||
4. 全部活动叶子结清后才关闭尚未完成的组合节点。终态前核验全部历史任务双向归属、结果确认、未结束尝试和未释放许可；不只看当前步骤或当前激活。
|
||||
5. 全部工作已知结清且没有核查条件时，根与实例进入FAILED或CANCELLED，关闭门控、清除待续并保存审计。存在未知或核查条件时保持NEEDS_REVIEW和原停止意图，平台已无有效执行授权时不再空转，后续人工核查操作负责接续；租约失效或UNKNOWN不证明外部执行确实已经结束。
|
||||
|
||||
历史核验按原有限流程上限限制为最多10000条任务元数据，使用一次集合查询和索引关联，避免逐任务数据库往返。门控与所有者双向检查包括旧激活；损坏关联拒绝推进和终结，不依赖外键补救。此上限及每事务一个取消的吞吐仍需按P6实际负载验证，当前不作生产容量承诺。
|
||||
|
||||
CANCELLED仅表示本次流程停止已结清，不撤销先前成功的业务效果。停止回执中的CANCELLING也不表示外部执行已结束；必须通过后续实例、步骤与执行历史判断当前事实。
|
||||
|
||||
## 5. 升级与验证
|
||||
|
||||
[迁移000021](../internal/storage/mysql/migrations/000021_workflow_stop_intent.up.sql)新增两个带中文说明的停止字段及成对CHECK，增加任务所有者查询索引。35个实体、303个直接持久化字段声明，继续禁止数据库外键及级联删除。
|
||||
|
||||
历史显式取消保留cancel_requested_at；历史失败结清用原updated_at建立兼容停止边界。旧停止中的实例关闭门控并回填或提前根检查点；不更改control_revision、定义、输入、原截止或预算，终态及无停止条件的实例保持原样。升级顺序仍为停止旧写入节点 → 执行全部迁移 → 启动新版，不能让旧节点忽略新增停止字段继续写入。
|
||||
|
||||
[控制及结清回归](../internal/storage/mysql/workflow_controls_integration_test.go)覆盖控制版本、权限重报、开始竞争、失败/未知、审计回滚、信号与查询及超过32步骤接续；[真实v20升级](../internal/storage/mysql/workflow_stop_upgrade_integration_test.go)验证兼容回填；[实际Orbit进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖停止意图和结果在全停期间保存后由两个新进程接续。实际结果见[阶段报告](reports/phase-5-controls.md)。
|
||||
|
||||
下一步仍需完成人工核查恢复、公共实例/控制/信号/审计接口、独立SDK及可运行场景，再完成P6–P7生产故障、性能、运维和发布验收。当前测试不替代生产MySQL复制、故障切换、备份恢复或RPO/RTO验证。
|
||||
53
docs/reports/phase-5-controls.md
Normal file
53
docs/reports/phase-5-controls.md
Normal file
@@ -0,0 +1,53 @@
|
||||
# P5 控制与停止结清回归
|
||||
|
||||
日期：2026-09-18。范围为内部Go控制服务、停止模型和后台结清运行器。公共HTTP/独立SDK、人工核查恢复及P6–P7仍未完成，持续目标继续。
|
||||
|
||||
## 已实现范围
|
||||
|
||||
- PAUSE/RESUME/CANCEL显式授权，稳定操作身份与独立control_revision；后台结果不使原控制版本失效。
|
||||
- 门控、控制标记、停止意图、事件、审计、冻结回执及必要根接续同事务。
|
||||
- 首次stop_kind和stop_requested_at保存CANCEL或FAILURE目标；进入核查不丢失，后来的用户取消不改写原失败目标。
|
||||
- FAIL_AFTER_SETTLED与用户取消收集并行分支及已有任务结果；运行中执行保留许可，未知保持核查。
|
||||
- 停止前信号保留、停止后信号拒绝；POLL停止后PENDING不消耗新业务轮次；停止与晚到超时由原意图隔离。
|
||||
- 超过32个步骤继续通过根检查点恢复；终态前检查所有历史任务归属、未结束尝试与许可。
|
||||
- 迁移000021增加两个中文注释字段和一个查询索引，实体仍35个，直接持久化字段303个，无外键。
|
||||
|
||||
详细约定见[控制与结清文档](../15-workflow-controls.md)。每次事务最多一个任务取消，其余步骤收尾最多32项；P6仍须验证真实负载和积压下的吞吐与延迟。
|
||||
|
||||
## 环境与结果
|
||||
|
||||
本机darwin/arm64、Go 1.27.1、Docker MySQL 8.4.10，临时隔离数据库，未接入业务或生产数据库。
|
||||
|
||||
| 检查 | 结果 |
|
||||
| --- | --- |
|
||||
| 首轮真实MySQL专项 | 控制18组及原信号/TIMER/POLL/结果运行器/Pulse取消/新建结构/v7/v9升级通过，`-race`，56.012秒 |
|
||||
| 基础回归 | 最终修订后再次运行`make fmt test vet build`通过，退出0 |
|
||||
| 修订后专项与实际进程 | 23组控制场景、真实v20升级及完整流程进程测试通过，`-race`，46.556秒；流程进程27.09秒、升级4.14秒 |
|
||||
| 全量隔离MySQL、流程进程、SDK与示例 | 修订后完整脚本通过，退出0；存储191.691秒，实际流程进程27.731秒，Worker SDK 94.897秒，最终SDK进程故障组94.033秒；19个SDK示例和2个独立业务示例全部通过 |
|
||||
| 数据结构与文档 | 当前MySQL外键0、无COMMENT字段0；303个直接持久化字段声明均有中文注释；891个本地文档链接、53段JSON检查通过 |
|
||||
|
||||
定向入口：
|
||||
|
||||
```sh
|
||||
go test -race -count=1 -run '^TestIntegration(WorkflowControls|WorkflowStopUpgrade|WorkflowRuntime|WorkflowClock|WorkflowSignalRuntime|WorkerCancellation|SchemaAndConstraints|NoForeignKeysUpgrade|ApplicationResourcesUpgrade)$' -v ./internal/storage/mysql
|
||||
go build -o /tmp/orbit-control-test-server ./cmd/orbit
|
||||
ORBIT_TEST_SERVER_BINARY=/tmp/orbit-control-test-server go test -race -count=1 -run '^TestIntegrationWorkflowRuntimeProcess$' -v ./internal/storage/mysql
|
||||
```
|
||||
|
||||
命令必须配合一次性`ORBIT_TEST_MYSQL_DSN`。跳过不代表通过。[全量脚本](../../scripts/test-mysql.sh)负责隔离数据库、构建实际二进制并显式运行流程进程回归。
|
||||
|
||||
首轮控制专项验证暂停时继续接收结果、使用原control_revision恢复、12个同键控制只形成一条审计、旧控制版本及撤销权限后的原键拒绝、控制/结清审计失败完全回滚，及运行中任务分别成功/失败/取消/未知时的最终事实。并行FAIL_AFTER_SETTLED在兄弟分支真实结清后才失败，UNKNOWN始终NEEDS_REVIEW；纯等待、停止前信号、业务PENDING查询不启动后继；60步骤的剩余59项通过有限批次收尾。
|
||||
|
||||
修订后专项及全量已验证开始与取消的8轮并发竞争、已领取未开始的暂停/继续、无合法反向归属的额外门控任务不能被取消、取消后原截止到达不改写意图，以及真实v20升级。v20用例只证明真实旧结构的数据迁移，实际执行恢复由独立进程测试验证。全量流程进程用例本体26.43秒：并行失败意图和在途兄弟任务经历全停后，仅在收集真实结果后进入FAILED；用户取消后的UNKNOWN由两个新进程恢复为NEEDS_REVIEW，不发出后继，也不产生错误终态。
|
||||
|
||||
完整脚本还通过旧v7外键移除、新建结构、结果订阅、Inbox/Outbox和SDK节点强杀恢复，临时MySQL容器已由脚本清理。移除外键后的000009–000021迁移没有重新引入外键；当前结构实查为0。
|
||||
|
||||
首次全量存储回归中两个新增测试使用内部错误名GATE_CLOSED作为协议码而失败，服务端实际按既有契约返回EXECUTION_PAUSED。已修正测试断言，生产门控规则未放宽；该次失败不计为全量通过，最终结果以修订后重跑为准。
|
||||
|
||||
## 剩余范围
|
||||
|
||||
1. 人工核查恢复及旧激活/原预算保护，不能将解除paused当成解除核查。
|
||||
2. 公共实例/步骤/控制/信号/来源/审计接口、独立SDK和端到端通用示例。
|
||||
3. 完整监控与故障矩阵、性能容量、MySQL高可用/备份恢复、运维与发布核对。
|
||||
|
||||
通过本阶段只能证明相应内部状态与单MySQL进程恢复行为，不能据此宣称整个平台生产验收完成。
|
||||
@@ -64,6 +64,8 @@ func TestIntegrationApplicationResourcesUpgrade(t *testing.T) {
|
||||
retryFinish(t, g, finished, "SUCCEEDED", false)
|
||||
assertApplicationCounts(t, f, 4, 2, 1)
|
||||
// 还原后续新增表和字段，得到真实 v9 结构；现有任务内容保持不变。
|
||||
f.exec(`ALTER TABLE pulse_tasks DROP INDEX ix_task_workflow_owner`)
|
||||
f.exec(`ALTER TABLE weave_instances DROP CHECK ck_workflow_stop_intent, DROP COLUMN stop_kind, DROP COLUMN stop_requested_at`)
|
||||
f.exec(`ALTER TABLE weave_signals DROP CHECK ck_signal_version_sequence, DROP CHECK ck_signal_pending_identity, DROP INDEX ix_signal_expiry, DROP INDEX ix_signal_activation, DROP COLUMN payload_version, DROP COLUMN target_activation_sequence, DROP COLUMN evidence_ref, DROP COLUMN disposition, DROP COLUMN available_at, MODIFY COLUMN type VARCHAR(64) NOT NULL COMMENT '事件或信号的稳定类型标识', MODIFY COLUMN correlation_key VARCHAR(191) NOT NULL COMMENT '外部业务操作关联键，防止旧信号误用于新工作'`)
|
||||
f.exec(`ALTER TABLE weave_waits DROP INDEX uq_wait_winner_signal, DROP COLUMN winner_signal_id`)
|
||||
f.exec(`ALTER TABLE weave_waits DROP COLUMN signal_name, DROP COLUMN payload_version, MODIFY COLUMN correlation_key VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_0900_as_cs NOT NULL COMMENT '外部业务操作关联键，防止旧信号误用于新工作'`)
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
-- 独立保存首次停止意图，避免进入核查后丢失用户取消/失败结清的目标；不增加外键。
|
||||
ALTER TABLE weave_instances
|
||||
ADD COLUMN stop_kind VARCHAR(16) CHARACTER SET ascii COLLATE ascii_bin NULL COMMENT '首次停止原因：CANCEL为用户取消，FAILURE为固定失败策略；为空表示尚未要求结清',
|
||||
ADD COLUMN stop_requested_at DATETIME(6) NULL COMMENT '首次停止意图的数据库时间，重报、进入核查或后来取消均不覆盖';
|
||||
|
||||
-- 历史取消保留原意图时间；旧失败结清及失败策略触发的核查保留原更新时间作为兼容边界。
|
||||
UPDATE weave_instances i SET stop_kind=CASE WHEN cancel_requested_at IS NOT NULL THEN 'CANCEL' ELSE 'FAILURE' END,
|
||||
stop_requested_at=COALESCE(cancel_requested_at,updated_at)
|
||||
WHERE state NOT IN ('SUCCEEDED','FAILED','CANCELLED') AND
|
||||
(cancel_requested_at IS NOT NULL OR state='CANCELLING' OR
|
||||
(JSON_UNQUOTE(JSON_EXTRACT(policy_snapshot,'$.failure_policy'))='FAIL_AFTER_SETTLED' AND EXISTS
|
||||
(SELECT 1 FROM weave_steps s WHERE s.namespace_id=i.namespace_id AND s.workflow_id=i.id AND s.`key`<>'$root' AND s.state IN ('FAILED','CANCELLED','NEEDS_REVIEW'))));
|
||||
|
||||
ALTER TABLE weave_instances
|
||||
ADD CONSTRAINT ck_workflow_stop_intent CHECK ((stop_kind IS NULL AND stop_requested_at IS NULL) OR (stop_kind IS NOT NULL AND stop_kind IN ('CANCEL','FAILURE') AND stop_requested_at IS NOT NULL));
|
||||
|
||||
-- 结清复用根检查点，已有根只提前检查，不更改固定版本、原预算和截止。
|
||||
INSERT INTO weave_pending_advances(namespace_id,id,workflow_id,step_id,expected_step_revision,checkpoint,available_at)
|
||||
SELECT i.namespace_id,RANDOM_BYTES(16),i.id,s.id,s.revision,JSON_OBJECT('schema_version',1,'kind','ROOT'),UTC_TIMESTAMP(6)
|
||||
FROM weave_instances i JOIN weave_steps s ON s.namespace_id=i.namespace_id AND s.workflow_id=i.id AND s.`key`='$root'
|
||||
JOIN weave_activations a ON a.namespace_id=s.namespace_id AND a.workflow_id=s.workflow_id AND a.step_id=s.id AND a.id=s.current_activation_id
|
||||
WHERE i.stop_kind IS NOT NULL AND i.state NOT IN ('SUCCEEDED','FAILED','CANCELLED') AND s.state='ACTIVE' AND a.state='ACTIVE'
|
||||
ON DUPLICATE KEY UPDATE available_at=LEAST(weave_pending_advances.available_at,VALUES(available_at));
|
||||
|
||||
UPDATE pulse_execution_gates g JOIN weave_instances i ON i.namespace_id=g.namespace_id AND i.gate_id=g.id
|
||||
SET g.paused=TRUE,g.revision=g.revision+1,g.updated_at=UTC_TIMESTAMP(6)
|
||||
WHERE i.stop_kind IS NOT NULL AND i.state NOT IN ('SUCCEEDED','FAILED','CANCELLED') AND g.paused=FALSE;
|
||||
|
||||
-- 反向核对流程所有者，防止无外键时遗漏门控被误改的历史任务。
|
||||
ALTER TABLE pulse_tasks ADD KEY ix_task_workflow_owner (namespace_id,owner_kind,owner_id,gate_id,id);
|
||||
@@ -89,7 +89,7 @@ func TestIntegrationNoForeignKeysUpgrade(t *testing.T) {
|
||||
indexes := func() []string {
|
||||
t.Helper()
|
||||
// v15替换retry_of索引、v17增加内部结果唯一索引、v20增加信号索引，下方单独验证定义。
|
||||
rows, err := db.QueryContext(ctx, `SELECT CONCAT_WS(':',TABLE_NAME,INDEX_NAME,NON_UNIQUE,SEQ_IN_INDEX,COLUMN_NAME,COALESCE(SUB_PART,0),INDEX_TYPE,IS_VISIBLE) FROM information_schema.STATISTICS WHERE TABLE_SCHEMA=DATABASE() AND TABLE_NAME NOT IN ('orbit_application_resources','orbit_event_streams','orbit_result_events') AND INDEX_NAME NOT IN ('ix_task_priority_dispatch','ix_delivery_subscription_due','ix_delivery_subscription_lease','ix_delivery_history','ix_delivery_state_history','uq_task_retry_parent','fk_pulse_tasks_3','ix_task_queue_business_created','ix_task_queue_business_state_created','uq_weave_task_result','ix_signal_expiry','ix_signal_activation','uq_wait_winner_signal') ORDER BY TABLE_NAME,INDEX_NAME,SEQ_IN_INDEX`)
|
||||
rows, err := db.QueryContext(ctx, `SELECT CONCAT_WS(':',TABLE_NAME,INDEX_NAME,NON_UNIQUE,SEQ_IN_INDEX,COLUMN_NAME,COALESCE(SUB_PART,0),INDEX_TYPE,IS_VISIBLE) FROM information_schema.STATISTICS WHERE TABLE_SCHEMA=DATABASE() AND TABLE_NAME NOT IN ('orbit_application_resources','orbit_event_streams','orbit_result_events') AND INDEX_NAME NOT IN ('ix_task_priority_dispatch','ix_delivery_subscription_due','ix_delivery_subscription_lease','ix_delivery_history','ix_delivery_state_history','uq_task_retry_parent','fk_pulse_tasks_3','ix_task_queue_business_created','ix_task_queue_business_state_created','uq_weave_task_result','ix_signal_expiry','ix_signal_activation','uq_wait_winner_signal','ix_task_workflow_owner') ORDER BY TABLE_NAME,INDEX_NAME,SEQ_IN_INDEX`)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -179,6 +179,7 @@ func TestIntegrationNoForeignKeysUpgrade(t *testing.T) {
|
||||
}
|
||||
var collation string
|
||||
for _, index := range []struct{ table, name, want string }{
|
||||
{"pulse_tasks", "ix_task_workflow_owner", "namespace_id:1,owner_kind:1,owner_id:1,gate_id:1,id:1"},
|
||||
{"weave_signals", "ix_signal_expiry", "state:1,available_at:1,namespace_id:1,id:1"},
|
||||
{"weave_signals", "ix_signal_activation", "namespace_id:1,workflow_id:1,target_step_key:1,target_activation_sequence:1,state:1,accepted_at:1,id:1"},
|
||||
{"weave_waits", "uq_wait_winner_signal", "namespace_id:0,winner_signal_id:0"},
|
||||
|
||||
@@ -55,6 +55,12 @@ func (r *WorkflowRuntime) advanceClock(ctx context.Context, tx *sql.Tx, g *workf
|
||||
if row.view.State != weave.StepActive && row.view.State != weave.StepWaiting {
|
||||
return false, deleteWorkflowClock(ctx, tx, g, row)
|
||||
}
|
||||
if g.view.StopKind != nil {
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return false, deleteWorkflowClock(ctx, tx, g, row)
|
||||
}
|
||||
var activated time.Time
|
||||
if err := tx.QueryRowContext(ctx, `SELECT created_at FROM weave_activations WHERE namespace_id=? AND id=? AND workflow_id=? AND step_id=?`, c.ns, checkpoint.ActivationID, c.workflow, step).Scan(&activated); err != nil {
|
||||
return false, err
|
||||
|
||||
@@ -187,7 +187,9 @@ func (r *WorkflowRuntime) consume(ctx context.Context, tx *sql.Tx, g *workflowGr
|
||||
}
|
||||
}
|
||||
// 单轮请求失败不能通过新建业务轮次绕过Pulse预算。
|
||||
if state != weave.StepSucceeded {
|
||||
if g.view.StopKind != nil && (state == weave.StepCancelled || state == weave.StepFailed) {
|
||||
reason = "POLL_STOP_REQUEST_SETTLED"
|
||||
} else if state != weave.StepSucceeded {
|
||||
state, reason = weave.StepNeedsReview, "POLL_REQUEST_REVIEW"
|
||||
} else if !source.occurred.Before(deadline) {
|
||||
state, output, reason = weave.StepNeedsReview, nil, "POLL_DEADLINE"
|
||||
@@ -198,7 +200,9 @@ func (r *WorkflowRuntime) consume(ctx context.Context, tx *sql.Tx, g *workflowGr
|
||||
state, output, reason = weave.StepNeedsReview, nil, "POLL_RESULT_INVALID"
|
||||
case outcome == engine.PollPending:
|
||||
state, output, reason = weave.StepWaiting, nil, "POLL_PENDING"
|
||||
if used >= max {
|
||||
if g.view.StopKind != nil {
|
||||
state, reason = weave.StepCancelled, "POLL_STOP_PENDING_SETTLED"
|
||||
} else if used >= max {
|
||||
state, reason = weave.StepNeedsReview, "POLL_ROUNDS_EXHAUSTED"
|
||||
}
|
||||
case outcome == engine.PollFailed:
|
||||
@@ -250,7 +254,7 @@ func (r *WorkflowRuntime) consume(ctx context.Context, tx *sql.Tx, g *workflowGr
|
||||
}
|
||||
if state == weave.StepNeedsReview || state == weave.StepFailed || state == weave.StepCancelled {
|
||||
next := weave.InstanceNeedsReview
|
||||
if (state == weave.StepFailed || state == weave.StepCancelled) && (g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" || g.view.CancelRequestedAt != nil) {
|
||||
if (state == weave.StepFailed || state == weave.StepCancelled) && (g.view.StopKind != nil || g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" || g.view.CancelRequestedAt != nil) {
|
||||
next = weave.InstanceCancelling
|
||||
}
|
||||
if err := holdWorkflow(ctx, tx, g, next, reason, now); err != nil {
|
||||
|
||||
136
internal/storage/mysql/workflow_controls.go
Normal file
136
internal/storage/mysql/workflow_controls.go
Normal file
@@ -0,0 +1,136 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/control"
|
||||
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||
)
|
||||
|
||||
// WorkflowControlStore 控制决定和门控同事务，不在请求中等待外部执行停止。
|
||||
type WorkflowControlStore struct{ *TaskStore }
|
||||
|
||||
func NewWorkflowControlStore(db *sql.DB) *WorkflowControlStore {
|
||||
return &WorkflowControlStore{NewTaskStore(db)}
|
||||
}
|
||||
|
||||
var _ control.Store = (*WorkflowControlStore)(nil)
|
||||
|
||||
func (s *WorkflowControlStore) Apply(ctx context.Context, t auth.Token, id core.ID, key string, in control.Input) (control.Result, error) {
|
||||
var out control.Result
|
||||
err := s.transaction(ctx, t, control.Action, func(tx *sql.Tx, p auth.Permissions) error {
|
||||
if _, err := lockApplicationResources(ctx, tx, t.NamespaceID, false); err != nil {
|
||||
return err
|
||||
}
|
||||
identity, cached, err := reserveIdentity(ctx, tx, t.NamespaceID, "workflow.control", string(id), key, "workflow", id, in)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
g, _, err := lockWorkflowGraph(ctx, tx, t.NamespaceID, id, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, q := range g.version.Analysis.Queues {
|
||||
if !p.AllowsQueue(q) {
|
||||
return fault.NotFound
|
||||
}
|
||||
}
|
||||
if cached != nil {
|
||||
if err := json.Unmarshal(cached, &out); err != nil {
|
||||
return err
|
||||
}
|
||||
if out.Instance.NamespaceID != t.NamespaceID || out.Instance.ID != id || out.Instance.DefinitionVersionID != g.view.DefinitionVersionID {
|
||||
return errors.New("workflow control receipt mismatch")
|
||||
}
|
||||
out.Deduplicated = true
|
||||
return nil
|
||||
}
|
||||
if g.view.State.Terminal() || g.view.ControlRevision != in.ExpectedControlRevision {
|
||||
return fault.Conflict
|
||||
}
|
||||
before := g.view
|
||||
now, err := dbNow(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch in.Operation {
|
||||
case "PAUSE":
|
||||
if g.view.Paused {
|
||||
return fault.Conflict
|
||||
}
|
||||
g.view.Paused = true
|
||||
case "RESUME":
|
||||
if !g.view.Paused {
|
||||
return fault.Conflict
|
||||
}
|
||||
g.view.Paused = false
|
||||
case "CANCEL":
|
||||
if g.view.CancelRequestedAt != nil {
|
||||
return fault.Conflict
|
||||
}
|
||||
g.view.CancelRequestedAt = &now
|
||||
if err := recordWorkflowStop(ctx, tx, g, "CANCEL", now); err != nil {
|
||||
return err
|
||||
}
|
||||
if g.view.State != weave.InstanceNeedsReview {
|
||||
g.view.State = weave.InstanceCancelling
|
||||
}
|
||||
default:
|
||||
return fault.Invalid("unsupported workflow control operation")
|
||||
}
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET paused=?,state=?,cancel_requested_at=?,control_revision=control_revision+1,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND control_revision=? AND revision=?`, g.view.Paused, g.view.State, g.view.CancelRequestedAt, now, t.NamespaceID, id, in.ExpectedControlRevision, g.view.Revision)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return err
|
||||
}
|
||||
g.view.ControlRevision++
|
||||
g.view.Revision++
|
||||
gatePaused := g.view.Paused || g.view.StopKind != nil || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE pulse_execution_gates SET paused=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=?`, gatePaused, now, t.NamespaceID, g.gate); err != nil {
|
||||
return err
|
||||
}
|
||||
if in.Operation == "CANCEL" || in.Operation == "RESUME" {
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
meta := map[string]any{"operation": in.Operation, "paused": g.view.Paused, "state": g.view.State, "control_revision": g.view.ControlRevision, "stop_kind": g.view.StopKind, "stop_requested_at": g.view.StopRequestedAt, "cancel_requested_at": g.view.CancelRequestedAt}
|
||||
if err := emitEvent(ctx, tx, t.NamespaceID, id, g.view.Revision, "workflow", "workflow.controlled", meta); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := appendAudit(ctx, tx, t, identity, id, "workflow", "workflow.control", in.Reason, map[string]any{"paused": before.Paused, "state": before.State, "control_revision": before.ControlRevision, "stop_kind": before.StopKind}, meta); err != nil {
|
||||
return err
|
||||
}
|
||||
out.Instance, _, err = readWorkflowInstance(ctx, tx, t.NamespaceID, id, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return saveIdentity(ctx, tx, t.NamespaceID, identity, out)
|
||||
})
|
||||
return out, err
|
||||
}
|
||||
|
||||
// recordWorkflowStop 冻结首次停止原因；后来的用户取消不能抹去已启动的失败结清。
|
||||
// 调用方同时关闭门控、保存根检查点和审计，不能将此函数单独提交。
|
||||
func recordWorkflowStop(ctx context.Context, tx *sql.Tx, g *workflowGraph, kind string, now time.Time) error {
|
||||
if g.view.StopKind != nil {
|
||||
return nil
|
||||
}
|
||||
if kind != "CANCEL" && kind != "FAILURE" {
|
||||
return errors.New("invalid workflow stop kind")
|
||||
}
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET stop_kind=?,stop_requested_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND stop_kind IS NULL`, kind, now, now, g.view.NamespaceID, g.view.ID, g.view.Revision)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return err
|
||||
}
|
||||
g.view.StopKind = &kind
|
||||
g.view.StopRequestedAt = &now
|
||||
g.view.Revision++
|
||||
return nil
|
||||
}
|
||||
484
internal/storage/mysql/workflow_controls_integration_test.go
Normal file
484
internal/storage/mysql/workflow_controls_integration_test.go
Normal file
@@ -0,0 +1,484 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"os"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/control"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||
)
|
||||
|
||||
func grantWorkflowControl(f *executionFixture) {
|
||||
f.exec(`UPDATE orbit_credentials SET permissions=JSON_ARRAY_APPEND(permissions,'$.actions','workflows:control') WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, f.resources.CredentialID)
|
||||
}
|
||||
func applyControl(t *testing.T, f *executionFixture, v instance.View, operation string) control.Result {
|
||||
t.Helper()
|
||||
out, err := control.New(NewWorkflowControlStore(f.db)).Apply(f.ctx, f.token, v.ID, string(executionID(t)), control.Input{Operation: operation, ExpectedControlRevision: v.ControlRevision, Reason: "controlled integration action"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out
|
||||
}
|
||||
func stopStep(t *testing.T, f *executionFixture, id core.ID) bool {
|
||||
t.Helper()
|
||||
f.exec(`UPDATE weave_pending_advances SET available_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND workflow_id=?`, f.resources.NamespaceID, id)
|
||||
rows, err := f.db.QueryContext(f.ctx, `SELECT 'INBOX',id,revision FROM weave_inbox WHERE namespace_id=? AND workflow_id=? AND state='PENDING' UNION ALL SELECT 'ROOT',id,revision FROM weave_pending_advances WHERE namespace_id=? AND workflow_id=?`, f.resources.NamespaceID, id, f.resources.NamespaceID, id)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var candidates []workflowCandidate
|
||||
for rows.Next() {
|
||||
c := workflowCandidate{ns: f.resources.NamespaceID, workflow: id}
|
||||
if err := rows.Scan(&c.kind, &c.id, &c.revision); err != nil {
|
||||
rows.Close()
|
||||
t.Fatal(err)
|
||||
}
|
||||
candidates = append(candidates, c)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, c := range candidates {
|
||||
if _, err := newRuntime(t, f.db).process(f.ctx, c, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return len(candidates) > 0
|
||||
}
|
||||
func drainStop(t *testing.T, f *executionFixture, id core.ID, state string) {
|
||||
t.Helper()
|
||||
for i := 0; i < 40; i++ {
|
||||
if !stopStep(t, f, id) {
|
||||
runtimeView(t, f, id, state)
|
||||
return
|
||||
}
|
||||
}
|
||||
t.Fatal("stop did not become quiescent")
|
||||
}
|
||||
func controlFixture(t *testing.T, ctx context.Context, db *sql.DB, count int) (*executionFixture, instance.View) {
|
||||
t.Helper()
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, count))
|
||||
grantWorkflowControl(f)
|
||||
return f, v
|
||||
}
|
||||
func TestIntegrationWorkflowControls(t *testing.T) {
|
||||
dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("requires disposable MySQL")
|
||||
}
|
||||
ctx, stop := context.WithTimeout(context.Background(), 150*time.Second)
|
||||
defer stop()
|
||||
db, err := Open(ctx, dsn, 32, 8)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err := Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Run("pause_collect_resume_uses_control_revision", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 2)
|
||||
ref := retryStart(t, f)
|
||||
paused := applyControl(t, f, v, "PAUSE").Instance
|
||||
if !paused.Paused || paused.ControlRevision != 2 || f.used() != 1 {
|
||||
t.Fatal("pause stopped running work or failed version")
|
||||
}
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
after := runtimeView(t, f, v.ID, "RUNNING")
|
||||
if after.Revision <= paused.Revision || after.ControlRevision != paused.ControlRevision || f.count("pulse_tasks", "1=1") != 1 {
|
||||
t.Fatal("background result corrupted control version")
|
||||
}
|
||||
resumed := applyControl(t, f, paused, "RESUME").Instance
|
||||
if resumed.Paused || resumed.ControlRevision != 3 || !resumed.DeadlineAt.Equal(v.DeadlineAt) {
|
||||
t.Fatal("resume reset state or deadline")
|
||||
}
|
||||
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||
if f.count("pulse_tasks", "1=1") != 2 {
|
||||
t.Fatal("resume lost successor")
|
||||
}
|
||||
})
|
||||
t.Run("cancel_waiting_cascades_in_application_and_preserves_receipt", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 3)
|
||||
api := control.New(NewWorkflowControlStore(db))
|
||||
key := string(executionID(t))
|
||||
in := control.Input{Operation: "CANCEL", ExpectedControlRevision: 1, Reason: "stop all work"}
|
||||
out, err := api.Apply(ctx, f.token, v.ID, key, in)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if out.Instance.StopKind == nil || *out.Instance.StopKind != "CANCEL" || out.Instance.StopRequestedAt == nil || out.Instance.State != "CANCELLING" {
|
||||
t.Fatal("cancel intent incomplete")
|
||||
}
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
if f.count("pulse_tasks", "state='CANCELLED'") != 1 || f.count("weave_steps", "state='SKIPPED'") != 2 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.used() != 0 {
|
||||
t.Fatal("cancel did not settle all work")
|
||||
}
|
||||
replay, err := api.Apply(ctx, f.token, v.ID, key, in)
|
||||
if err != nil || !replay.Deduplicated || replay.Instance.State != "CANCELLING" || replay.Instance.ControlRevision != 2 {
|
||||
t.Fatal("control receipt was mutated", err)
|
||||
}
|
||||
_, err = api.Apply(ctx, f.token, v.ID, string(executionID(t)), control.Input{Operation: "PAUSE", ExpectedControlRevision: 2, Reason: "cannot control terminal"})
|
||||
if fault.Code(err) != "STATE_CONFLICT" {
|
||||
t.Fatal("terminal accepted control", err)
|
||||
}
|
||||
})
|
||||
t.Run("concurrent_same_identity_and_stale_control", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
api := control.New(NewWorkflowControlStore(db))
|
||||
key := string(executionID(t))
|
||||
in := control.Input{Operation: "PAUSE", ExpectedControlRevision: 1, Reason: "one decision"}
|
||||
var wg sync.WaitGroup
|
||||
errs := make(chan error, 12)
|
||||
fresh := make(chan bool, 12)
|
||||
for i := 0; i < 12; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
out, err := api.Apply(ctx, f.token, v.ID, key, in)
|
||||
errs <- err
|
||||
fresh <- !out.Deduplicated
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
close(fresh)
|
||||
for err := range errs {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
n := 0
|
||||
for v := range fresh {
|
||||
if v {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n != 1 || f.count("orbit_audit_events", "action='workflow.control'") != 1 || f.count("weave_instances", "control_revision=2 AND paused=TRUE") != 1 {
|
||||
t.Fatal("duplicated control", n)
|
||||
}
|
||||
in.Operation = "CANCEL"
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, key, in); fault.Code(err) != "IDEMPOTENCY_CONFLICT" {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "STATE_CONFLICT" {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
t.Run("permission_and_scope_still_checked_on_replay", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
api := control.New(NewWorkflowControlStore(db))
|
||||
key := string(executionID(t))
|
||||
in := control.Input{Operation: "PAUSE", ExpectedControlRevision: 1, Reason: "authorized pause"}
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, key, in); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.exec(`UPDATE orbit_credentials SET permissions=JSON_SET(permissions,'$.queues',JSON_ARRAY(?)) WHERE namespace_id=? AND id=?`, string(executionID(t)), f.resources.NamespaceID, f.resources.CredentialID)
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, key, in); fault.Code(err) != "NOT_FOUND" {
|
||||
t.Fatal("scope replay bypass", err)
|
||||
}
|
||||
grantWorkflowInstance(t, f)
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, key, in); fault.Code(err) != "PERMISSION_DENIED" {
|
||||
t.Fatal("action replay bypass", err)
|
||||
}
|
||||
})
|
||||
t.Run("control_audit_failure_rolls_back_everything", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
api := control.New(NewWorkflowControlStore(db))
|
||||
key := string(executionID(t))
|
||||
in := control.Input{Operation: "CANCEL", ExpectedControlRevision: 1, Reason: "atomic decision"}
|
||||
f.exec(`CREATE TRIGGER test_control_failure BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='control audit fault'`)
|
||||
defer f.exec(`DROP TRIGGER IF EXISTS test_control_failure`)
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, key, in); err == nil {
|
||||
t.Fatal("audit failure accepted")
|
||||
}
|
||||
if f.count("weave_instances", "control_revision=1 AND stop_kind IS NULL AND cancel_requested_at IS NULL") != 1 || f.count("pulse_execution_gates", "paused=FALSE") != 1 || f.count("weave_pending_advances", "1=1") != 0 {
|
||||
t.Fatal("partial control mutation")
|
||||
}
|
||||
f.exec(`DROP TRIGGER test_control_failure`)
|
||||
if _, err := api.Apply(ctx, f.token, v.ID, key, in); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
})
|
||||
t.Run("settlement_audit_failure_rolls_back_task_and_inbox", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
c := clockCandidate(t, f, v.ID, workflowRootKey)
|
||||
f.exec(`CREATE TRIGGER test_settlement_failure BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='settlement audit fault'`)
|
||||
defer f.exec(`DROP TRIGGER IF EXISTS test_settlement_failure`)
|
||||
if done, err := newRuntime(t, db).process(ctx, c, false); err == nil || done {
|
||||
t.Fatal("partial settlement committed")
|
||||
}
|
||||
if f.count("pulse_tasks", "state='WAITING' AND cancel_requested_at IS NULL") != 1 || f.count("weave_inbox", "1=1") != 0 {
|
||||
t.Fatal("cancel or result escaped transaction")
|
||||
}
|
||||
f.exec(`DROP TRIGGER test_settlement_failure`)
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
})
|
||||
for _, outcome := range []string{"SUCCEEDED", "FAILED", "CANCELLED", "UNKNOWN"} {
|
||||
t.Run("running_cancel_then_"+outcome, func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 2)
|
||||
ref := retryStart(t, f)
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
stopStep(t, f, v.ID)
|
||||
if f.get(ref.TaskID).State != "RUNNING" || f.get(ref.TaskID).CancelRequestedAt == nil || f.used() != 1 {
|
||||
t.Fatal("cancel pretended running work ended")
|
||||
}
|
||||
retryFinish(t, f, ref, outcome, false)
|
||||
want := "CANCELLED"
|
||||
if outcome == "UNKNOWN" {
|
||||
want = "NEEDS_REVIEW"
|
||||
}
|
||||
drainStop(t, f, v.ID, want)
|
||||
after := runtimeView(t, f, v.ID, want)
|
||||
if after.StopKind == nil || *after.StopKind != "CANCEL" || f.used() != 0 || f.count("pulse_tasks", "1=1") != 1 {
|
||||
t.Fatal("stop intent lost or extra task")
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Run("failure_policy_settles_parallel_and_freezes_first_reason", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||
tail := spec.Body.Steps[0]
|
||||
spec.Body.Steps = []definition.Step{runtimeParallel(spec), tail}
|
||||
spec.FailurePolicy = "FAIL_AFTER_SETTLED"
|
||||
v := runtimeStart(t, f, spec)
|
||||
grantWorkflowControl(f)
|
||||
left, right := retryStart(t, f), retryStart(t, f)
|
||||
retryFinish(t, f, left, "FAILED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), left.TaskID)
|
||||
stopping := runtimeView(t, f, v.ID, "CANCELLING")
|
||||
if stopping.StopKind == nil || *stopping.StopKind != "FAILURE" {
|
||||
t.Fatal("failure intent not persisted")
|
||||
}
|
||||
later := applyControl(t, f, stopping, "CANCEL").Instance
|
||||
if *later.StopKind != "FAILURE" || !later.StopRequestedAt.Equal(*stopping.StopRequestedAt) {
|
||||
t.Fatal("user cancel rewrote earlier failure")
|
||||
}
|
||||
stopStep(t, f, v.ID)
|
||||
if f.get(right.TaskID).CancelRequestedAt == nil || f.used() != 1 {
|
||||
t.Fatal("sibling cancellation missing")
|
||||
}
|
||||
retryFinish(t, f, right, "SUCCEEDED", false)
|
||||
drainStop(t, f, v.ID, "FAILED")
|
||||
if f.count("pulse_tasks", "1=1") != 2 || f.count("weave_steps", "`key`='join' AND state='CANCELLED'") != 1 || f.count("weave_steps", "`key`='task0' AND state='SKIPPED'") != 1 {
|
||||
t.Fatal("failed join advanced tail")
|
||||
}
|
||||
})
|
||||
t.Run("unknown_failure_cancels_sibling_but_never_terminal", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||
tail := spec.Body.Steps[0]
|
||||
spec.Body.Steps = []definition.Step{runtimeParallel(spec), tail}
|
||||
spec.FailurePolicy = "FAIL_AFTER_SETTLED"
|
||||
v := runtimeStart(t, f, spec)
|
||||
left, right := retryStart(t, f), retryStart(t, f)
|
||||
retryFinish(t, f, left, "UNKNOWN", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), left.TaskID)
|
||||
stopStep(t, f, v.ID)
|
||||
stopStep(t, f, v.ID)
|
||||
if f.get(right.TaskID).CancelRequestedAt == nil {
|
||||
t.Fatal("review lost failure cancellation")
|
||||
}
|
||||
retryFinish(t, f, right, "CANCELLED", false)
|
||||
drainStop(t, f, v.ID, "NEEDS_REVIEW")
|
||||
if f.used() != 0 || f.count("weave_instances", "completed_at IS NULL AND stop_kind='FAILURE'") != 1 {
|
||||
t.Fatal("unknown result terminalized")
|
||||
}
|
||||
})
|
||||
for _, kind := range []string{"TIMER", "SIGNAL", "EARLY_SIGNAL"} {
|
||||
t.Run("cancel_wait_"+kind, func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := signalSpec(f.resources.QueueID, false)
|
||||
if kind == "TIMER" {
|
||||
spec = timerSpec(f.resources.QueueID, 30000, true)
|
||||
}
|
||||
v := runtimeStart(t, f, spec)
|
||||
grantWorkflowControl(f)
|
||||
grantWorkflowSignal(f)
|
||||
if kind == "EARLY_SIGNAL" {
|
||||
sendSignal(t, f, v, signalInput(v))
|
||||
}
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
if kind != "TIMER" {
|
||||
late := sendSignal(t, f, v, signalInput(v))
|
||||
if late.Signal.Disposition != "WORKFLOW_STOPPING" {
|
||||
t.Fatal("late signal was admitted")
|
||||
}
|
||||
}
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
if f.count("pulse_tasks", "1=1") != 0 {
|
||||
t.Fatal("cancelled wait activated successor")
|
||||
}
|
||||
if kind == "EARLY_SIGNAL" && f.count("weave_signals", "state='CONSUMED'") != 1 {
|
||||
t.Fatal("prior accepted signal lost")
|
||||
}
|
||||
})
|
||||
}
|
||||
for _, running := range []bool{false, true} {
|
||||
name := "poll_between_rounds"
|
||||
if running {
|
||||
name = "poll_running_pending"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
f := newClockFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, pollingSpec(f.resources.QueueID))
|
||||
grantWorkflowControl(f)
|
||||
ref := retryStart(t, f)
|
||||
if !running {
|
||||
finishPoll(t, f, ref, "PENDING")
|
||||
}
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
stopStep(t, f, v.ID)
|
||||
if running {
|
||||
if f.get(ref.TaskID).CancelRequestedAt == nil {
|
||||
t.Fatal("query did not receive cancel")
|
||||
}
|
||||
finishQueryOnly(t, f, ref, "PENDING")
|
||||
}
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_poll_states", "used_rounds=1 AND current_task_id IS NULL AND next_poll_at IS NULL AND resolved_at IS NOT NULL") != 1 {
|
||||
t.Fatal("cancel query added a round or left budget open")
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Run("more_than_32_blocked_steps_resume_root", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 60)
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
if f.count("weave_steps", "state='SKIPPED'") != 59 || f.count("orbit_audit_events", "action='workflow.advance' AND reason='WORKFLOW_STOP_CONTINUATION'") < 1 {
|
||||
t.Fatal("bounded stop lost continuation")
|
||||
}
|
||||
})
|
||||
t.Run("pause_blocks_leased_start_and_resume_preserves_attempt", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
_, ref := f.claim()
|
||||
paused := applyControl(t, f, v, "PAUSE").Instance
|
||||
if _, err := f.a.Start(ctx, f.token, f.binding, executionID(t), ref); fault.Code(err) != "EXECUTION_PAUSED" {
|
||||
t.Fatal("paused lease started", err)
|
||||
}
|
||||
resumed := applyControl(t, f, paused, "RESUME").Instance
|
||||
if _, err := f.a.Start(ctx, f.token, f.binding, executionID(t), ref); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if f.get(ref.TaskID).AttemptCount != 1 || resumed.ControlRevision != 3 {
|
||||
t.Fatal("resume replaced original attempt")
|
||||
}
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||
})
|
||||
t.Run("start_cancel_race_has_one_authority", func(t *testing.T) {
|
||||
for i := 0; i < 8; i++ {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
_, ref := f.claim()
|
||||
api := control.New(NewWorkflowControlStore(db))
|
||||
var startErr, controlErr error
|
||||
var wg sync.WaitGroup
|
||||
ready := make(chan struct{})
|
||||
startID, key := executionID(t), string(executionID(t))
|
||||
wg.Add(2)
|
||||
go func() { defer wg.Done(); <-ready; _, startErr = f.a.Start(ctx, f.token, f.binding, startID, ref) }()
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
<-ready
|
||||
_, controlErr = api.Apply(ctx, f.token, v.ID, key, control.Input{Operation: "CANCEL", ExpectedControlRevision: 1, Reason: "start cancellation race"})
|
||||
}()
|
||||
close(ready)
|
||||
wg.Wait()
|
||||
if controlErr != nil {
|
||||
t.Fatal(controlErr)
|
||||
}
|
||||
if startErr != nil && fault.Code(startErr) != "EXECUTION_PAUSED" {
|
||||
t.Fatal("unexpected start outcome", startErr)
|
||||
}
|
||||
stopStep(t, f, v.ID)
|
||||
if startErr == nil {
|
||||
if f.get(ref.TaskID).State != "RUNNING" || f.used() != 1 {
|
||||
t.Fatal("authorized work released by cancellation")
|
||||
}
|
||||
retryFinish(t, f, ref, "CANCELLED", false)
|
||||
} else if f.get(ref.TaskID).State != "CANCELLED" || f.used() != 0 {
|
||||
t.Fatal("unstarted work not settled")
|
||||
}
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
}
|
||||
})
|
||||
t.Run("extra_gate_reference_cannot_cancel_unrelated_task", func(t *testing.T) {
|
||||
f, v := controlFixture(t, ctx, db, 1)
|
||||
extra := f.submit(1)
|
||||
var gate, activation core.ID
|
||||
if err := db.QueryRowContext(ctx, "SELECT i.gate_id,s.current_activation_id FROM weave_instances i JOIN weave_steps s ON s.namespace_id=i.namespace_id AND s.workflow_id=i.id AND s.`key`='task0' WHERE i.namespace_id=? AND i.id=?", f.resources.NamespaceID, v.ID).Scan(&gate, &activation); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.exec(`UPDATE pulse_tasks SET gate_id=?,owner_kind='workflow_activation',owner_id=? WHERE namespace_id=? AND id=?`, gate, activation, f.resources.NamespaceID, extra.ID)
|
||||
defer f.exec(`UPDATE pulse_tasks SET gate_id=NULL,owner_kind=NULL,owner_id=NULL WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, extra.ID)
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
rejected := false
|
||||
for i := 0; i < 3; i++ {
|
||||
c := clockCandidate(t, f, v.ID, workflowRootKey)
|
||||
if _, err := newRuntime(t, db).process(ctx, c, false); err != nil {
|
||||
rejected = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !rejected || f.get(extra.ID).CancelRequestedAt != nil || f.count("weave_instances", "state='CANCELLED'") != 0 {
|
||||
t.Fatal("gate alone granted cancellation or termination")
|
||||
}
|
||||
f.exec(`UPDATE pulse_tasks SET gate_id=NULL,owner_kind=NULL,owner_id=NULL WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, extra.ID)
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
if f.get(extra.ID).State != "WAITING" {
|
||||
t.Fatal("unrelated task changed")
|
||||
}
|
||||
})
|
||||
for _, withSignal := range []bool{false, true} {
|
||||
name := "cancel_beats_unconsumed_timeout"
|
||||
if withSignal {
|
||||
name = "cancel_collects_signal_after_total_deadline"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := signalSpec(f.resources.QueueID, false)
|
||||
spec.Body.Steps = spec.Body.Steps[:1]
|
||||
spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "signal"}}
|
||||
spec.TimeoutMS = 1000
|
||||
spec.Body.Steps[0].Signal.TimeoutMS = 1000
|
||||
v := runtimeStart(t, f, spec)
|
||||
grantWorkflowControl(f)
|
||||
grantWorkflowSignal(f)
|
||||
var source workflowCandidate
|
||||
if withSignal {
|
||||
accepted := sendSignal(t, f, v, signalInput(v))
|
||||
source = workflowCandidate{kind: "SIGNAL", ns: f.resources.NamespaceID, workflow: v.ID, id: accepted.Signal.ID, revision: accepted.Signal.Revision}
|
||||
} else {
|
||||
source = clockCandidate(t, f, v.ID, "signal")
|
||||
}
|
||||
applyControl(t, f, v, "CANCEL")
|
||||
for time.Now().Before(v.DeadlineAt.Add(20 * time.Millisecond)) {
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
if _, err := newRuntime(t, db).process(ctx, source, false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
drainStop(t, f, v.ID, "CANCELLED")
|
||||
if withSignal && f.count("weave_signals", "state='CONSUMED'") != 1 {
|
||||
t.Fatal("accepted signal expired after cancellation")
|
||||
}
|
||||
if f.count("weave_steps", "state='NEEDS_REVIEW'") != 0 {
|
||||
t.Fatal("later timeout rewrote earlier cancellation")
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Log("versioned control, permission replay, audit rollback, safe cancellation, failure settlement, signal/POLL collection and bounded root continuation verified")
|
||||
}
|
||||
@@ -52,7 +52,7 @@ type workflowPolicy struct {
|
||||
func readWorkflowInstance(ctx context.Context, tx *sql.Tx, ns, id core.ID, lock bool) (instance.View, core.ID, error) {
|
||||
var v instance.View
|
||||
var gate core.ID
|
||||
q := `SELECT i.id,i.namespace_id,i.definition_id,i.definition_version_id,v.version,i.gate_id,i.state,i.revision,i.paused,i.control_revision,i.input,i.output,COALESCE(i.business_key,''),i.deadline_at,i.cancel_requested_at,i.completed_at,i.created_at,i.updated_at
|
||||
q := `SELECT i.id,i.namespace_id,i.definition_id,i.definition_version_id,v.version,i.gate_id,i.state,i.revision,i.paused,i.control_revision,i.input,i.output,COALESCE(i.business_key,''),i.deadline_at,i.cancel_requested_at,i.stop_kind,i.stop_requested_at,i.completed_at,i.created_at,i.updated_at
|
||||
FROM weave_instances i JOIN weave_definition_versions v ON v.namespace_id=i.namespace_id AND v.definition_id=i.definition_id AND v.id=i.definition_version_id
|
||||
JOIN weave_definitions d ON d.namespace_id=i.namespace_id AND d.id=i.definition_id
|
||||
JOIN pulse_execution_gates g ON g.namespace_id=i.namespace_id AND g.id=i.gate_id
|
||||
@@ -61,7 +61,7 @@ func readWorkflowInstance(ctx context.Context, tx *sql.Tx, ns, id core.ID, lock
|
||||
q += ` FOR SHARE`
|
||||
}
|
||||
var input, output []byte
|
||||
err := tx.QueryRowContext(ctx, q, ns, id).Scan(&v.ID, &v.NamespaceID, &v.DefinitionID, &v.DefinitionVersionID, &v.Version, &gate, &v.State, &v.Revision, &v.Paused, &v.ControlRevision, &input, &output, &v.BusinessKey, &v.DeadlineAt, &v.CancelRequestedAt, &v.CompletedAt, &v.CreatedAt, &v.UpdatedAt)
|
||||
err := tx.QueryRowContext(ctx, q, ns, id).Scan(&v.ID, &v.NamespaceID, &v.DefinitionID, &v.DefinitionVersionID, &v.Version, &gate, &v.State, &v.Revision, &v.Paused, &v.ControlRevision, &input, &output, &v.BusinessKey, &v.DeadlineAt, &v.CancelRequestedAt, &v.StopKind, &v.StopRequestedAt, &v.CompletedAt, &v.CreatedAt, &v.UpdatedAt)
|
||||
v.Input, v.Output = input, output
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
err = fault.NotFound
|
||||
|
||||
@@ -152,6 +152,10 @@ func (r *WorkflowRuntime) process(ctx context.Context, c workflowCandidate, skip
|
||||
_, err = tx.ExecContext(ctx, `DELETE FROM weave_pending_advances WHERE namespace_id=? AND id=?`, c.ns, c.id)
|
||||
return err
|
||||
}
|
||||
if g.view.StopKind != nil {
|
||||
done, err = r.settleWorkflow(ctx, tx, g, c, now)
|
||||
return err
|
||||
}
|
||||
if !allow || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling {
|
||||
// 暂停/停用仍按原截止计时；核查和失败结清由控制运行器接续，不清除其状态。
|
||||
if (g.view.State == weave.InstanceRunning || g.view.State == weave.InstanceWaiting) && !now.Before(g.view.DeadlineAt) {
|
||||
@@ -192,7 +196,7 @@ func cloneWorkflowGraph(g *workflowGraph) *workflowGraph {
|
||||
|
||||
// progress 在结果已保存后执行有限决策。额度不足只回滚后继决策，不撤销已确认的结果。
|
||||
func (r *WorkflowRuntime) progress(ctx context.Context, tx *sql.Tx, g *workflowGraph, allow bool, now time.Time) error {
|
||||
if g.view.State.Terminal() || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling {
|
||||
if g.view.State.Terminal() || g.view.StopKind != nil || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling {
|
||||
return nil
|
||||
}
|
||||
allow = allow && now.Before(g.view.DeadlineAt)
|
||||
@@ -232,6 +236,18 @@ func holdWorkflow(ctx context.Context, tx *sql.Tx, g *workflowGraph, state weave
|
||||
if g.view.State.Terminal() {
|
||||
return errors.New("cannot hold completed workflow")
|
||||
}
|
||||
if state == weave.InstanceCancelling || g.view.StopKind != nil || g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" {
|
||||
kind := "FAILURE"
|
||||
if g.view.CancelRequestedAt != nil {
|
||||
kind = "CANCEL"
|
||||
}
|
||||
if err := recordWorkflowStop(ctx, tx, g, kind, now); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if g.view.State == weave.InstanceNeedsReview {
|
||||
state = weave.InstanceNeedsReview
|
||||
}
|
||||
|
||||
@@ -96,7 +96,7 @@ func lockWorkflowGraph(ctx context.Context, tx *sql.Tx, ns, id core.ID, skip boo
|
||||
if len(g.steps) != int(version.Analysis.Steps)+1 || root == nil || root.view.ActivationID == nil {
|
||||
return nil, false, errors.New("workflow step set incomplete")
|
||||
}
|
||||
allow := enabled && !gatePaused && !v.Paused && v.CancelRequestedAt == nil && (v.State == weave.InstanceRunning || v.State == weave.InstanceWaiting)
|
||||
allow := enabled && v.StopKind == nil && !gatePaused && !v.Paused && v.CancelRequestedAt == nil && (v.State == weave.InstanceRunning || v.State == weave.InstanceWaiting)
|
||||
return g, allow, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -262,4 +262,53 @@ func TestIntegrationWorkflowRuntimeProcess(t *testing.T) {
|
||||
t.Fatal("final signal restart lost winner or issued an extra round")
|
||||
}
|
||||
t.Log("early signal persisted before wait activation across all-stop; final signal selected with running query kept permits, then survived another all-stop and accepted PENDING settlement without issuing a second query")
|
||||
for _, p := range processes {
|
||||
stop(p)
|
||||
}
|
||||
failedSpec := runtimeSpec(f.resources.QueueID, 1)
|
||||
tail := failedSpec.Body.Steps[0]
|
||||
failedSpec.Body.Steps = append(failedSpec.Body.Steps[:0], runtimeParallel(failedSpec), tail)
|
||||
failedSpec.FailurePolicy = "FAIL_AFTER_SETTLED"
|
||||
failed := runtimeStart(t, f, failedSpec)
|
||||
left, right := retryStart(t, f), retryStart(t, f)
|
||||
retryFinish(t, f, left, "FAILED", false)
|
||||
start("failure-collection-a")
|
||||
start("failure-collection-b")
|
||||
wait(func() bool { return f.get(right.TaskID).CancelRequestedAt != nil }, "failure cancellation of parallel sibling")
|
||||
if f.used() != 1 || f.get(right.TaskID).State != "RUNNING" {
|
||||
t.Fatal("failure claimed sibling ended")
|
||||
}
|
||||
for _, p := range processes {
|
||||
stop(p)
|
||||
}
|
||||
retryFinish(t, f, right, "SUCCEEDED", false)
|
||||
if f.count("weave_instances", "id=? AND state='CANCELLING' AND stop_kind='FAILURE'", failed.ID) != 1 {
|
||||
t.Fatal("all-stop lost failure intent")
|
||||
}
|
||||
start("failure-settlement-a")
|
||||
start("failure-settlement-b")
|
||||
wait(func() bool {
|
||||
current, err := api.Get(ctx, f.token, failed.ID)
|
||||
return err == nil && current.State == "FAILED"
|
||||
}, "parallel failure settlement across restart")
|
||||
if f.count("pulse_tasks", "1=1") != 10 || f.used() != 0 {
|
||||
t.Fatal("failure restarted a branch or tail")
|
||||
}
|
||||
for _, p := range processes {
|
||||
stop(p)
|
||||
}
|
||||
cancelled := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
grantWorkflowControl(f)
|
||||
ref = retryStart(t, f)
|
||||
applyControl(t, f, cancelled, "CANCEL")
|
||||
retryFinish(t, f, ref, "UNKNOWN", false)
|
||||
start("cancel-review-a")
|
||||
start("cancel-review-b")
|
||||
wait(func() bool {
|
||||
return f.count("weave_instances", "id=? AND state='NEEDS_REVIEW' AND stop_kind='CANCEL' AND completed_at IS NULL", cancelled.ID) == 1 && f.count("weave_pending_advances", "workflow_id=?", cancelled.ID) == 0
|
||||
}, "cancel unknown review without false terminal state")
|
||||
if f.used() != 0 || f.count("pulse_tasks", "1=1") != 11 {
|
||||
t.Fatal("unknown cancel leaked permit or started tail")
|
||||
}
|
||||
t.Log("parallel failure intent and running sibling survived another all-stop, settled FAILED only after result collection; user cancellation plus UNKNOWN recovered NEEDS_REVIEW with no successor or false terminal state")
|
||||
}
|
||||
|
||||
384
internal/storage/mysql/workflow_settlement.go
Normal file
384
internal/storage/mysql/workflow_settlement.go
Normal file
@@ -0,0 +1,384 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
pulse "git.yuntuo.hk/k2safe/orbit/pulse/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||
)
|
||||
|
||||
// settleWorkflow 在根检查点下结清停止意图。每事务最多发送一个任务取消，避免取得结果流锁后再锁另一任务。
|
||||
// 步骤收尾最多32项，超过后保留根接续；停止/核查不依赖节点内存恢复。
|
||||
func (r *WorkflowRuntime) settleWorkflow(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, now time.Time) (bool, error) {
|
||||
if g.view.StopKind == nil || g.view.StopRequestedAt == nil || (*g.view.StopKind != "CANCEL" && *g.view.StopKind != "FAILURE") {
|
||||
return false, errors.New("workflow stop intent invalid")
|
||||
}
|
||||
if *g.view.StopKind == "FAILURE" && g.version.Specification.FailurePolicy != "FAIL_AFTER_SETTLED" {
|
||||
return false, errors.New("workflow failure stop conflicts with fixed policy")
|
||||
}
|
||||
if *g.view.StopKind == "CANCEL" && g.view.CancelRequestedAt == nil {
|
||||
return false, errors.New("workflow cancellation intent missing")
|
||||
}
|
||||
// 已持有应用/全部队列/实例/门控锁；门控漂移不能为停止中的工作重新授权。
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE pulse_execution_gates SET paused=TRUE,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND paused=FALSE`, now, g.view.NamespaceID, g.gate); err != nil {
|
||||
return false, err
|
||||
}
|
||||
var taskID core.ID
|
||||
err := tx.QueryRowContext(ctx, `SELECT id FROM pulse_tasks WHERE namespace_id=? AND gate_id=? AND state NOT IN ('SUCCEEDED','FAILED','CANCELLED') AND cancel_requested_at IS NULL ORDER BY id LIMIT 1`, g.view.NamespaceID, g.gate).Scan(&taskID)
|
||||
if err != nil && !errors.Is(err, sql.ErrNoRows) {
|
||||
return false, err
|
||||
}
|
||||
if err == nil {
|
||||
if err := validateWorkflowStopTask(ctx, tx, g, taskID); err != nil {
|
||||
return false, err
|
||||
}
|
||||
task, err := readExecutionTask(ctx, tx, g.view.NamespaceID, taskID)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if _, err := cancelExecutionTask(ctx, tx, g.view.NamespaceID, task, now, "WORKFLOW_STOP_REQUESTED"); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_TASK_STOP_REQUESTED", now)
|
||||
}
|
||||
var pending int
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_inbox WHERE namespace_id=? AND workflow_id=? AND state='PENDING'`, g.view.NamespaceID, g.view.ID).Scan(&pending); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if pending > 0 {
|
||||
return false, saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second))
|
||||
}
|
||||
keys := make([]string, 0, len(g.steps))
|
||||
for key := range g.steps {
|
||||
if key != workflowRootKey {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
}
|
||||
sort.Strings(keys)
|
||||
changed, active, review := 0, false, false
|
||||
for _, key := range keys {
|
||||
row := g.steps[key]
|
||||
if row.view.State == weave.StepNeedsReview {
|
||||
review = true
|
||||
continue
|
||||
}
|
||||
if row.view.State == weave.StepSucceeded || row.view.State == weave.StepFailed || row.view.State == weave.StepCancelled || row.view.State == weave.StepSkipped {
|
||||
continue
|
||||
}
|
||||
if changed == 32 {
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_STOP_CONTINUATION", now)
|
||||
}
|
||||
if row.view.State == weave.StepBlocked {
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_steps SET state='SKIPPED',revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND current_activation_id IS NULL AND state='BLOCKED'`, now, g.view.NamespaceID, row.view.ID, row.view.Revision)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return false, err
|
||||
}
|
||||
row.view.State = weave.StepSkipped
|
||||
row.view.Revision++
|
||||
changed++
|
||||
if err := emitEvent(ctx, tx, g.view.NamespaceID, row.view.ID, row.view.Revision, "workflow_step", "workflow.step.changed", map[string]any{"workflow_id": g.view.ID, "key": key, "state": row.view.State, "reason": "WORKFLOW_STOP_REQUESTED"}); err != nil {
|
||||
return false, err
|
||||
}
|
||||
continue
|
||||
}
|
||||
switch row.view.Kind {
|
||||
case weave.NodeTask:
|
||||
if row.view.TaskID == nil {
|
||||
return false, errors.New("stopping task step missing task")
|
||||
}
|
||||
task, err := readTask(ctx, tx, g.view.NamespaceID, *row.view.TaskID, false)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if task.State.Terminal() || task.State == pulse.TaskNeedsReview {
|
||||
return false, errors.New("stopping task missing result confirmation")
|
||||
}
|
||||
active = true
|
||||
case weave.NodeSignal, weave.NodeTimer, weave.NodePoll:
|
||||
node, _ := g.program.Step(key)
|
||||
if row.view.Kind == weave.NodePoll {
|
||||
var current *core.ID
|
||||
if err := tx.QueryRowContext(ctx, `SELECT current_task_id FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=?`, g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(¤t); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if current != nil {
|
||||
active = true
|
||||
continue
|
||||
}
|
||||
}
|
||||
// 已在停止前可靠受理的信号保留为事实；停止后的新信号由受理事务拒绝。
|
||||
if signalContract(node) != nil {
|
||||
w, err := readWorkflowSignalWait(ctx, tx, g, row)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
v, err := selectedWorkflowSignal(ctx, tx, g, row, w, now)
|
||||
if err != nil || v != nil {
|
||||
handled, done, reason, err := r.driveWorkflowSignal(ctx, tx, g, row, c, nil, false, now)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !handled {
|
||||
return false, errors.New("stopping signal did not resolve")
|
||||
}
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if done && reason != "" {
|
||||
return true, r.audit(ctx, tx, g, c.id, reason, now)
|
||||
}
|
||||
return done, nil
|
||||
}
|
||||
}
|
||||
if err := cancelWorkflowWait(ctx, tx, g, row, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
changed++
|
||||
case weave.NodeBranch, weave.NodeParallel:
|
||||
// 组合步骤只有在全部活动叶子结清后才能结清，避免掩盖在途工作。
|
||||
default:
|
||||
return false, errors.New("unsupported stopping step")
|
||||
}
|
||||
}
|
||||
if active {
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second)); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if changed > 0 {
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_STOP_COLLECTING", now)
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
settled, uncertain, err := verifyWorkflowStopWork(ctx, tx, g)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !settled {
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second)); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if changed > 0 {
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_STOP_COLLECTING", now)
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
if review || uncertain || g.view.State == weave.InstanceNeedsReview {
|
||||
stateChanged := g.view.State != weave.InstanceNeedsReview
|
||||
if stateChanged {
|
||||
if err := holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, "WORKFLOW_STOP_REVIEW", now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
// 已无平台内有效授权时不逐秒空转；UNKNOWN不证明外部执行已结束，后续核查负责接续。
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE weave_signals SET state='REJECTED',disposition='WORKFLOW_STOPPED_REVIEW',revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND state='PENDING'`, now, g.view.NamespaceID, g.view.ID); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if err := deleteWorkflowClock(ctx, tx, g, g.steps[workflowRootKey]); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if changed > 0 || stateChanged {
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_STOP_REVIEW", now)
|
||||
}
|
||||
return false, nil
|
||||
}
|
||||
for _, key := range keys {
|
||||
row := g.steps[key]
|
||||
if row.view.Kind != weave.NodeBranch && row.view.Kind != weave.NodeParallel {
|
||||
continue
|
||||
}
|
||||
if row.view.State != weave.StepActive && row.view.State != weave.StepWaiting {
|
||||
continue
|
||||
}
|
||||
if changed == 32 {
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_STOP_CONTINUATION", now)
|
||||
}
|
||||
if err := saveWorkflowResultStep(ctx, tx, g, row, weave.StepCancelled, nil, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
changed++
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE weave_signals SET state='REJECTED',disposition='WORKFLOW_STOPPED',revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND state='PENDING'`, now, g.view.NamespaceID, g.view.ID); err != nil {
|
||||
return false, err
|
||||
}
|
||||
next, rootState := weave.InstanceCancelled, weave.StepCancelled
|
||||
if *g.view.StopKind == "FAILURE" {
|
||||
next, rootState = weave.InstanceFailed, weave.StepFailed
|
||||
}
|
||||
if err := saveWorkflowResultStep(ctx, tx, g, g.steps[workflowRootKey], rootState, nil, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET state=?,output=NULL,completed_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, next, now, now, g.view.NamespaceID, g.view.ID, g.view.Revision)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return false, err
|
||||
}
|
||||
g.view.State = next
|
||||
g.view.Output = nil
|
||||
g.view.CompletedAt = &now
|
||||
g.view.Revision++
|
||||
if _, err := tx.ExecContext(ctx, `DELETE FROM weave_pending_advances WHERE namespace_id=? AND workflow_id=?`, g.view.NamespaceID, g.view.ID); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if err := emitEvent(ctx, tx, g.view.NamespaceID, g.view.ID, g.view.Revision, "workflow", "workflow.stopped", map[string]any{"state": next, "stop_kind": g.view.StopKind, "stop_requested_at": g.view.StopRequestedAt}); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, r.audit(ctx, tx, g, c.id, "WORKFLOW_STOPPED", now)
|
||||
}
|
||||
|
||||
func cancelWorkflowWait(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, now time.Time) error {
|
||||
if row.view.Kind == weave.NodeTimer {
|
||||
var count int
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_waits WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND kind='TIMER' AND state='WAITING' AND signal_name IS NULL AND payload_version IS NULL AND winner_signal_id IS NULL`, g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count != 1 {
|
||||
return errors.New("stopping timer wait invalid")
|
||||
}
|
||||
}
|
||||
if row.view.Kind == weave.NodePoll {
|
||||
var activated time.Time
|
||||
if err := tx.QueryRowContext(ctx, `SELECT created_at FROM weave_activations WHERE namespace_id=? AND workflow_id=? AND step_id=? AND id=?`, g.view.NamespaceID, g.view.ID, row.view.ID, row.view.ActivationID).Scan(&activated); err != nil {
|
||||
return err
|
||||
}
|
||||
p, _, err := readWorkflowPollState(ctx, tx, g, row, activated)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if p.CurrentTaskID != nil {
|
||||
return errors.New("cannot cancel in-flight poll wait")
|
||||
}
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_poll_states SET next_poll_at=NULL,resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND resolved_at IS NULL AND current_task_id IS NULL`, now, now, g.view.NamespaceID, p.ID)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE weave_waits SET state='CANCELLED',resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND state='WAITING'`, now, now, g.view.NamespaceID, g.view.ID, row.view.ActivationID); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := saveWorkflowResultStep(ctx, tx, g, row, weave.StepCancelled, nil, now); err != nil {
|
||||
return err
|
||||
}
|
||||
return deleteWorkflowClock(ctx, tx, g, row)
|
||||
}
|
||||
|
||||
// validateWorkflowStopTask 取消前核对实际任务到流程激活/步骤/队列的归属，不能仅凭gate_id关联。
|
||||
func validateWorkflowStopTask(ctx context.Context, tx *sql.Tx, g *workflowGraph, id core.ID) error {
|
||||
var kind string
|
||||
var owner, queue, gate, workflow, step core.ID
|
||||
var key string
|
||||
var direct *core.ID
|
||||
err := tx.QueryRowContext(ctx, "SELECT t.owner_kind,t.owner_id,t.queue_id,t.gate_id,a.workflow_id,a.step_id,s.`key`,a.task_id FROM pulse_tasks t JOIN weave_activations a ON a.namespace_id=t.namespace_id AND a.id=t.owner_id JOIN weave_steps s ON s.namespace_id=a.namespace_id AND s.workflow_id=a.workflow_id AND s.id=a.step_id WHERE t.namespace_id=? AND t.id=?", g.view.NamespaceID, id).Scan(&kind, &owner, &queue, &gate, &workflow, &step, &key, &direct)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
row := g.steps[key]
|
||||
node, ok := g.program.Step(key)
|
||||
if kind != engine.TaskOwnerKind || gate != g.gate || workflow != g.view.ID || row == nil || row.view.ID != step || !ok {
|
||||
return errors.New("stopping task ownership invalid")
|
||||
}
|
||||
if node.Task != nil {
|
||||
if direct == nil || *direct != id || queue != node.Task.QueueID {
|
||||
return errors.New("stopping task direct association invalid")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if node.Poll == nil || direct != nil || queue != node.Poll.Task.QueueID {
|
||||
return errors.New("stopping query association invalid")
|
||||
}
|
||||
var count int
|
||||
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.`round` BETWEEN 1 AND p.used_rounds", g.view.NamespaceID, g.view.ID, owner, id).Scan(&count); err != nil {
|
||||
return err
|
||||
}
|
||||
if count != 1 {
|
||||
return errors.New("stopping query round invalid")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// verifyWorkflowStopWork 终态前复核全部历史任务、双向归属、未结束尝试和许可，不能仅检查当前步骤。
|
||||
func verifyWorkflowStopWork(ctx context.Context, tx *sql.Tx, g *workflowGraph) (settled, uncertain bool, err error) {
|
||||
var invalid int
|
||||
checks := []string{
|
||||
`SELECT COUNT(*) FROM pulse_tasks t JOIN weave_activations a ON a.namespace_id=t.namespace_id AND a.id=t.owner_id WHERE a.namespace_id=? AND a.workflow_id=? AND t.owner_kind='workflow_activation' AND (t.gate_id IS NULL OR t.gate_id<>?)`,
|
||||
`SELECT COUNT(*) FROM weave_activations a LEFT JOIN pulse_tasks t ON t.namespace_id=a.namespace_id AND t.id=a.task_id WHERE a.namespace_id=? AND a.workflow_id=? AND a.task_id IS NOT NULL AND (t.id IS NULL OR t.owner_kind IS NULL OR t.owner_kind<>'workflow_activation' OR t.owner_id<>a.id OR t.gate_id IS NULL OR t.gate_id<>?)`,
|
||||
`SELECT COUNT(*) FROM weave_poll_rounds r LEFT JOIN weave_activations a ON a.namespace_id=r.namespace_id AND a.workflow_id=r.workflow_id AND a.id=r.activation_id LEFT JOIN pulse_tasks t ON t.namespace_id=r.namespace_id AND t.id=r.task_id WHERE r.namespace_id=? AND r.workflow_id=? AND (a.id IS NULL OR t.id IS NULL OR t.owner_kind IS NULL OR t.owner_kind<>'workflow_activation' OR t.owner_id<>r.activation_id OR t.gate_id IS NULL OR t.gate_id<>?)`,
|
||||
}
|
||||
for _, q := range checks {
|
||||
if err := tx.QueryRowContext(ctx, q, g.view.NamespaceID, g.view.ID, g.gate).Scan(&invalid); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
if invalid != 0 {
|
||||
return false, false, errors.New("workflow stop history association invalid")
|
||||
}
|
||||
}
|
||||
// 一次读取关联元数据并通过索引EXISTS核对查询轮次，避免持有应用锁期间逐任务发出SQL。
|
||||
rows, err := tx.QueryContext(ctx, "SELECT t.id,t.state,t.effect_uncertain,t.owner_kind,t.owner_id,t.queue_id,a.workflow_id,a.step_id,a.task_id,s.`key`,EXISTS(SELECT 1 FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=t.namespace_id AND r.workflow_id=a.workflow_id AND r.activation_id=a.id AND r.task_id=t.id AND r.`round` BETWEEN 1 AND p.used_rounds) FROM pulse_tasks t LEFT JOIN weave_activations a ON a.namespace_id=t.namespace_id AND a.id=t.owner_id LEFT JOIN weave_steps s ON s.namespace_id=a.namespace_id AND s.workflow_id=a.workflow_id AND s.id=a.step_id WHERE t.namespace_id=? AND t.gate_id=? ORDER BY t.id LIMIT 10001", g.view.NamespaceID, g.gate)
|
||||
if err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
defer rows.Close()
|
||||
count := 0
|
||||
settled = true
|
||||
for rows.Next() {
|
||||
var id, queue core.ID
|
||||
var owner, workflow, step, direct *core.ID
|
||||
var kind, key sql.NullString
|
||||
var state pulse.TaskState
|
||||
var unknown, round bool
|
||||
if err := rows.Scan(&id, &state, &unknown, &kind, &owner, &queue, &workflow, &step, &direct, &key, &round); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
count++
|
||||
if count > 10000 {
|
||||
return false, false, errors.New("workflow historical task bound exceeded")
|
||||
}
|
||||
row := g.steps[key.String]
|
||||
node, ok := g.program.Step(key.String)
|
||||
if kind.String != engine.TaskOwnerKind || owner == nil || workflow == nil || *workflow != g.view.ID || step == nil || row == nil || row.view.ID != *step || !ok {
|
||||
return false, false, errors.New("workflow stop task association invalid")
|
||||
}
|
||||
if node.Task != nil {
|
||||
if direct == nil || *direct != id || queue != node.Task.QueueID {
|
||||
return false, false, errors.New("workflow stop task parameters invalid")
|
||||
}
|
||||
} else if node.Poll == nil || direct != nil || queue != node.Poll.Task.QueueID || !round {
|
||||
return false, false, errors.New("workflow stop query association invalid")
|
||||
}
|
||||
uncertain = uncertain || unknown || state == pulse.TaskNeedsReview
|
||||
if !state.Terminal() && state != pulse.TaskNeedsReview {
|
||||
settled = false
|
||||
}
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
if !settled {
|
||||
return false, uncertain, nil
|
||||
}
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM pulse_attempts a JOIN pulse_tasks t ON t.namespace_id=a.namespace_id AND t.id=a.task_id WHERE t.namespace_id=? AND t.gate_id=? AND a.state IN ('LEASED','RUNNING')`, g.view.NamespaceID, g.gate).Scan(&invalid); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
if invalid > 0 {
|
||||
return false, uncertain, nil
|
||||
}
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM pulse_permits p JOIN pulse_attempts a ON a.namespace_id=p.namespace_id AND a.id=p.attempt_id JOIN pulse_tasks t ON t.namespace_id=a.namespace_id AND t.id=a.task_id WHERE t.namespace_id=? AND t.gate_id=? AND p.released_at IS NULL`, g.view.NamespaceID, g.gate).Scan(&invalid); err != nil {
|
||||
return false, false, err
|
||||
}
|
||||
return invalid == 0, uncertain, nil
|
||||
}
|
||||
@@ -242,7 +242,7 @@ func (r *WorkflowRuntime) applyWorkflowSignal(ctx context.Context, tx *sql.Tx, g
|
||||
}
|
||||
if state == weave.StepNeedsReview || state == weave.StepFailed {
|
||||
next := weave.InstanceNeedsReview
|
||||
if state == weave.StepFailed && (g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" || g.view.CancelRequestedAt != nil) {
|
||||
if state == weave.StepFailed && (g.view.StopKind != nil || g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" || g.view.CancelRequestedAt != nil) {
|
||||
next = weave.InstanceCancelling
|
||||
}
|
||||
if err := holdWorkflow(ctx, tx, g, next, reason, now); err != nil {
|
||||
@@ -284,6 +284,14 @@ func (r *WorkflowRuntime) expireWorkflowSignal(ctx context.Context, tx *sql.Tx,
|
||||
if v.State != "PENDING" || now.Before(v.ExpiresAt) {
|
||||
return false, nil
|
||||
}
|
||||
if g.view.StopKind != nil && !g.view.State.Terminal() {
|
||||
// 停止事务已关闭新等待；由根收集停止前信号，不能让稍后过期扫描把取消改成超时核查。
|
||||
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
_, err := tx.ExecContext(ctx, `UPDATE weave_signals SET available_at=DATE_ADD(?,INTERVAL 30 SECOND),revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND state='PENDING'`, now, now, c.ns, c.id)
|
||||
return false, err
|
||||
}
|
||||
row := g.steps[v.TargetStepKey]
|
||||
if row != nil && row.view.ActivationID != nil && !g.view.State.Terminal() && (row.view.State == weave.StepWaiting || row.view.State == weave.StepActive) {
|
||||
w, err := readWorkflowSignalWait(ctx, tx, g, row)
|
||||
|
||||
@@ -119,7 +119,7 @@ func selectedWorkflowSignal(ctx context.Context, tx *sql.Tx, g *workflowGraph, r
|
||||
if w.state != "WAITING" {
|
||||
return nil, nil
|
||||
}
|
||||
err := tx.QueryRowContext(ctx, `SELECT id FROM weave_signals WHERE namespace_id=? AND workflow_id=? AND target_step_key=? AND target_activation_sequence=? AND state='PENDING' AND type=? AND payload_version=? AND correlation_key=? AND accepted_at<? ORDER BY accepted_at,id LIMIT 1 FOR UPDATE`, g.view.NamespaceID, g.view.ID, row.view.Key, w.sequence, w.contract.Name, w.contract.PayloadVersion, []byte(w.correlation), w.deadline).Scan(&id)
|
||||
err := tx.QueryRowContext(ctx, `SELECT id FROM weave_signals WHERE namespace_id=? AND workflow_id=? AND target_step_key=? AND target_activation_sequence=? AND state='PENDING' AND type=? AND payload_version=? AND correlation_key=? AND accepted_at<? AND (? IS NULL OR accepted_at<=?) ORDER BY accepted_at,id LIMIT 1 FOR UPDATE`, g.view.NamespaceID, g.view.ID, row.view.Key, w.sequence, w.contract.Name, w.contract.PayloadVersion, []byte(w.correlation), w.deadline, g.view.StopRequestedAt, g.view.StopRequestedAt).Scan(&id)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
@@ -124,6 +124,8 @@ func (s *WorkflowSignalStore) Send(ctx context.Context, t auth.Token, workflow c
|
||||
state, disposition = "REJECTED", "STALE_ACTIVATION"
|
||||
case g.view.State.Terminal():
|
||||
state, disposition = "REJECTED", "WORKFLOW_SETTLED"
|
||||
case g.view.StopKind != nil:
|
||||
state, disposition = "REJECTED", "WORKFLOW_STOPPING"
|
||||
case !now.Before(g.view.DeadlineAt):
|
||||
state, disposition = "EXPIRED", "WORKFLOW_DEADLINE"
|
||||
case row.view.State == weave.StepSkipped || row.view.State == weave.StepSucceeded || row.view.State == weave.StepFailed || row.view.State == weave.StepCancelled || row.view.State == weave.StepNeedsReview:
|
||||
|
||||
160
internal/storage/mysql/workflow_stop_upgrade_integration_test.go
Normal file
160
internal/storage/mysql/workflow_stop_upgrade_integration_test.go
Normal file
@@ -0,0 +1,160 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
driver "github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
// 用真实v20结构验证停止意图与根接续升级，保留原取消时间和业务预算。
|
||||
func TestIntegrationWorkflowStopUpgrade(t *testing.T) {
|
||||
dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("requires disposable MySQL")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||
defer cancel()
|
||||
admin, err := Open(ctx, dsn, 2, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
name := "orbit_stop_upgrade_" + string(executionID(t))
|
||||
if _, err := admin.ExecContext(ctx, "CREATE DATABASE `"+name+"`"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() {
|
||||
cleanup, stop := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer stop()
|
||||
if _, err := admin.ExecContext(cleanup, "DROP DATABASE `"+name+"`"); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
cfg, err := driver.ParseDSN(dsn)
|
||||
if err != nil {
|
||||
t.Fatal("invalid test DSN")
|
||||
}
|
||||
cfg.DBName = name
|
||||
db, err := Open(ctx, cfg.FormatDSN(), 4, 2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
exec := func(q string, args ...any) {
|
||||
t.Helper()
|
||||
if _, err := db.ExecContext(ctx, q, args...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
exec(`CREATE TABLE orbit_schema_migrations (version BIGINT UNSIGNED PRIMARY KEY,name VARCHAR(191) NOT NULL,checksum CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,dirty BOOLEAN NOT NULL,applied_at DATETIME(6) NULL) ENGINE=InnoDB`)
|
||||
migrations, err := Migrations()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, m := range migrations {
|
||||
if m.Version > 20 {
|
||||
break
|
||||
}
|
||||
statements, err := splitSQL(m.SQL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, q := range statements {
|
||||
exec(q)
|
||||
}
|
||||
exec(`INSERT INTO orbit_schema_migrations(version,name,checksum,dirty,applied_at) VALUES (?,?,?,FALSE,UTC_TIMESTAMP(6))`, m.Version, m.Name, m.Checksum)
|
||||
}
|
||||
var now time.Time
|
||||
if err := db.QueryRowContext(ctx, `SELECT UTC_TIMESTAMP(6)`).Scan(&now); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ns := executionID(t)
|
||||
deadline := now.Add(time.Hour)
|
||||
requested := now.Add(-time.Hour)
|
||||
type fixture struct {
|
||||
id, gate, root, activation core.ID
|
||||
kind string
|
||||
}
|
||||
seed := func(state, policy string, cancelled, failed bool) fixture {
|
||||
t.Helper()
|
||||
f := fixture{executionID(t), executionID(t), executionID(t), executionID(t), "FAILURE"}
|
||||
var cancelAt, completed *time.Time
|
||||
if cancelled {
|
||||
cancelAt = &requested
|
||||
f.kind = "CANCEL"
|
||||
}
|
||||
if state == "CANCELLED" {
|
||||
completed = &now
|
||||
}
|
||||
exec(`INSERT INTO pulse_execution_gates(namespace_id,id,paused) VALUES (?,?,FALSE)`, ns, f.gate)
|
||||
exec(`INSERT INTO weave_instances(namespace_id,id,definition_id,definition_version_id,gate_id,state,control_revision,input,policy_snapshot,deadline_at,cancel_requested_at,completed_at,updated_at) VALUES (?,?,?,?,?,?,4,'{"n":9007199254740993}',JSON_OBJECT('failure_policy',?),?,?,?,?)`, ns, f.id, executionID(t), executionID(t), f.gate, state, policy, deadline, cancelAt, completed, now)
|
||||
exec("INSERT INTO weave_steps(namespace_id,id,workflow_id,`key`,kind,state,current_activation_id,revision) VALUES (?,?,?,'$root','JOIN','ACTIVE',?,7)", ns, f.root, f.id, f.activation)
|
||||
exec(`INSERT INTO weave_activations(namespace_id,id,workflow_id,step_id,sequence,state,input) VALUES (?,?,?,?,1,'ACTIVE','null')`, ns, f.activation, f.id, f.root)
|
||||
if failed {
|
||||
exec("INSERT INTO weave_steps(namespace_id,id,workflow_id,`key`,kind,state) VALUES (?,?,?,'failed','TASK','NEEDS_REVIEW')", ns, executionID(t), f.id)
|
||||
}
|
||||
return f
|
||||
}
|
||||
cancellation := seed("NEEDS_REVIEW", "REVIEW", true, false)
|
||||
fail := seed("CANCELLING", "FAIL_AFTER_SETTLED", false, false)
|
||||
uncertain := seed("NEEDS_REVIEW", "FAIL_AFTER_SETTLED", false, true)
|
||||
untouched := seed("NEEDS_REVIEW", "REVIEW", false, true)
|
||||
terminal := seed("CANCELLED", "REVIEW", true, false)
|
||||
originalID := executionID(t)
|
||||
original := `{"schema_version":1,"kind":"ROOT"}`
|
||||
exec(`INSERT INTO weave_pending_advances(namespace_id,id,workflow_id,step_id,expected_step_revision,checkpoint,available_at,revision) VALUES (?,?,?,?,7,?,?,11)`, ns, originalID, fail.id, fail.root, original, deadline)
|
||||
if err := Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := CheckSchema(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, f := range []fixture{cancellation, fail, uncertain} {
|
||||
var kind string
|
||||
var at, due, keptDeadline time.Time
|
||||
var controlRevision, expected uint64
|
||||
var paused bool
|
||||
var raw []byte
|
||||
if err := db.QueryRowContext(ctx, `SELECT i.stop_kind,i.stop_requested_at,i.deadline_at,i.control_revision,g.paused FROM weave_instances i JOIN pulse_execution_gates g ON g.namespace_id=i.namespace_id AND g.id=i.gate_id WHERE i.namespace_id=? AND i.id=?`, ns, f.id).Scan(&kind, &at, &keptDeadline, &controlRevision, &paused); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := now
|
||||
if f.kind == "CANCEL" {
|
||||
want = requested
|
||||
}
|
||||
if kind != f.kind || !at.Equal(want) || !keptDeadline.Equal(deadline) || controlRevision != 4 || !paused {
|
||||
t.Fatal("migration reset stop identity, deadline or control version")
|
||||
}
|
||||
if err := db.QueryRowContext(ctx, `SELECT checkpoint,available_at,expected_step_revision FROM weave_pending_advances WHERE namespace_id=? AND workflow_id=?`, ns, f.id).Scan(&raw, &due, &expected); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var cp workflowCheckpoint
|
||||
if err := json.Unmarshal(raw, &cp); err != nil || cp.Kind != "ROOT" || cp.SchemaVersion != 1 || cp.ActivationID != nil || expected != 7 || !due.Before(deadline) {
|
||||
t.Fatal("stop continuation missing or invalid", err)
|
||||
}
|
||||
}
|
||||
var count int
|
||||
for _, f := range []fixture{untouched, terminal} {
|
||||
if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_instances WHERE namespace_id=? AND id=? AND stop_kind IS NULL AND stop_requested_at IS NULL`, ns, f.id).Scan(&count); err != nil || count != 1 {
|
||||
t.Fatal("unrelated or terminal workflow changed", err)
|
||||
}
|
||||
}
|
||||
if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_pending_advances WHERE namespace_id=?`, ns).Scan(&count); err != nil || count != 3 {
|
||||
t.Fatal("backfill included unrelated workflow", count, err)
|
||||
}
|
||||
if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_pending_advances WHERE namespace_id=? AND id=? AND revision=11 AND checkpoint=?`, ns, originalID, original).Scan(&count); err != nil || count != 1 {
|
||||
t.Fatal("existing root identity changed", err)
|
||||
}
|
||||
if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM information_schema.REFERENTIAL_CONSTRAINTS WHERE CONSTRAINT_SCHEMA=DATABASE()`).Scan(&count); err != nil || count != 0 {
|
||||
t.Fatal("foreign keys introduced", err)
|
||||
}
|
||||
if err := Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Log("real v20 stop intent backfill preserves cancellation time and fixed deadline/control revision, closes gate, preserves existing root identity; unrelated/terminal workflows unchanged; no foreign keys")
|
||||
}
|
||||
@@ -58,7 +58,7 @@ type Permissions struct {
|
||||
Queues []string `json:"queues"`
|
||||
}
|
||||
|
||||
var supported = map[string]bool{"workflows:signal": true, "workflows:start": true, "workflows:read": true, "workflows:definitions:read": true, "workflows:definitions:publish": true, "tasks:review": true, "tasks:retry": true, "tasks:audit:read": true, "events:read": true, "events:manage": true, "events:consume": true, "application:resources:read": true, "application:resources:manage": true, "tasks:submit": true, "tasks:read": true, "tasks:cancel": true, "queues:read": true, "queues:manage": true, "workers:connect": true, "workers:consume": true, "schedules:read": true, "schedules:manage": true}
|
||||
var supported = map[string]bool{"workflows:signal": true, "workflows:control": true, "workflows:start": true, "workflows:read": true, "workflows:definitions:read": true, "workflows:definitions:publish": true, "tasks:review": true, "tasks:retry": true, "tasks:audit:read": true, "events:read": true, "events:manage": true, "events:consume": true, "application:resources:read": true, "application:resources:manage": true, "tasks:submit": true, "tasks:read": true, "tasks:cancel": true, "queues:read": true, "queues:manage": true, "workers:connect": true, "workers:consume": true, "schedules:read": true, "schedules:manage": true}
|
||||
|
||||
func (p Permissions) Validate() error {
|
||||
if len(p.Actions) == 0 || len(p.Actions) > 32 || len(p.Queues) == 0 || len(p.Queues) > 128 {
|
||||
|
||||
48
weave/control/service.go
Normal file
48
weave/control/service.go
Normal file
@@ -0,0 +1,48 @@
|
||||
// Package control 定义流程暂停、继续与取消的版本化操作；解除暂停不会清除核查或停止意图。
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||
)
|
||||
|
||||
const Action = "workflows:control"
|
||||
|
||||
// Input 的稳定操作身份由方法参数传入；并发控制只使用control_revision，不与后台结果版本竞争。
|
||||
type Input struct {
|
||||
Operation string `json:"operation"` // Operation 为PAUSE、RESUME或CANCEL；终结实例不接受新控制操作。
|
||||
ExpectedControlRevision uint64 `json:"expected_control_revision,string"` // ExpectedControlRevision 调用者观察到的控制版本，必须大于0。
|
||||
Reason string `json:"reason"` // Reason 必填操作原因，最多1024个Unicode字符，进入不可变审计。
|
||||
}
|
||||
type Result struct {
|
||||
Instance instance.View `json:"instance"` // Instance 首次操作提交后的冻结回执；当前结果用实例Get读取。
|
||||
Deduplicated bool `json:"deduplicated"` // Deduplicated 原操作恢复，不重复修改控制版本或计时。
|
||||
}
|
||||
type Store interface {
|
||||
Apply(context.Context, auth.Token, core.ID, string, Input) (Result, error)
|
||||
}
|
||||
type Service struct{ store Store }
|
||||
|
||||
func New(store Store) *Service { return &Service{store: store} }
|
||||
func (s *Service) Apply(ctx context.Context, t auth.Token, id core.ID, key string, in Input) (Result, error) {
|
||||
if err := service.ValidateKey(key); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if id.Validate() != nil || in.ExpectedControlRevision == 0 {
|
||||
return Result{}, fault.Invalid("workflow id and positive control revision required")
|
||||
}
|
||||
if in.Operation != "PAUSE" && in.Operation != "RESUME" && in.Operation != "CANCEL" {
|
||||
return Result{}, fault.Invalid("workflow operation must be PAUSE, RESUME or CANCEL")
|
||||
}
|
||||
if !utf8.ValidString(in.Reason) || strings.TrimSpace(in.Reason) == "" || utf8.RuneCountInString(in.Reason) > 1024 || strings.ContainsRune(in.Reason, 0) {
|
||||
return Result{}, fault.Invalid("workflow control reason required, maximum 1024 characters")
|
||||
}
|
||||
return s.store.Apply(ctx, t, id, key, in)
|
||||
}
|
||||
43
weave/control/service_test.go
Normal file
43
weave/control/service_test.go
Normal file
@@ -0,0 +1,43 @@
|
||||
package control
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||
)
|
||||
|
||||
type captureStore struct{ calls int }
|
||||
|
||||
func (s *captureStore) Apply(context.Context, auth.Token, core.ID, string, Input) (Result, error) {
|
||||
s.calls++
|
||||
return Result{}, nil
|
||||
}
|
||||
func TestControlRejectsUnsafeBoundary(t *testing.T) {
|
||||
store := &captureStore{}
|
||||
api := New(store)
|
||||
id := core.ID("11111111111111111111111111111111")
|
||||
base := Input{Operation: "PAUSE", ExpectedControlRevision: 1, Reason: "人工暂停"}
|
||||
for _, operation := range []string{"PAUSE", "RESUME", "CANCEL"} {
|
||||
in := base
|
||||
in.Operation = operation
|
||||
if _, err := api.Apply(context.Background(), auth.Token{}, id, "original", in); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, change := range []func(*Input){func(in *Input) { in.ExpectedControlRevision = 0 }, func(in *Input) { in.Operation = "RETRY" }, func(in *Input) { in.Reason = " " }, func(in *Input) { in.Reason = strings.Repeat("中", 1025) }, func(in *Input) { in.Reason = "reason\x00hidden" }, func(in *Input) { in.Reason = string([]byte{0xff}) }} {
|
||||
in := base
|
||||
change(&in)
|
||||
before := store.calls
|
||||
if _, err := api.Apply(context.Background(), auth.Token{}, id, "original", in); err == nil || store.calls != before {
|
||||
t.Fatal("invalid input reached store")
|
||||
}
|
||||
}
|
||||
for _, key := range []string{"", strings.Repeat("x", 129)} {
|
||||
if _, err := api.Apply(context.Background(), auth.Token{}, id, key, base); err == nil {
|
||||
t.Fatal("invalid operation identity reached store")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -44,6 +44,8 @@ type View struct {
|
||||
BusinessKey string `json:"business_key"` // BusinessKey 原始追踪键。
|
||||
DeadlineAt time.Time `json:"deadline_at"` // DeadlineAt 数据库时间确定的固定总截止。
|
||||
CancelRequestedAt *time.Time `json:"cancel_requested_at,omitempty"` // CancelRequestedAt 已保存的取消意图时间。
|
||||
StopKind *string `json:"stop_kind,omitempty"` // StopKind 首次停止原因CANCEL或FAILURE；核查不会清除。
|
||||
StopRequestedAt *time.Time `json:"stop_requested_at,omitempty"` // StopRequestedAt 首次停止意图时间，后续操作不会延长。
|
||||
CompletedAt *time.Time `json:"completed_at,omitempty"` // CompletedAt 终结时间，非终态为空。
|
||||
CreatedAt time.Time `json:"created_at"` // CreatedAt UTC受理时间。
|
||||
UpdatedAt time.Time `json:"updated_at"` // UpdatedAt 最近状态更新时间。
|
||||
|
||||
@@ -35,6 +35,10 @@ type Instance struct {
|
||||
DeadlineAt time.Time `db:"deadline_at"`
|
||||
// CancelRequestedAt 取消意图的持久化时间；不等于取消已完成；可为空，表示尚未设置该关联或值。
|
||||
CancelRequestedAt *time.Time `db:"cancel_requested_at"`
|
||||
// StopKind 首次停止原因，CANCEL为用户取消、FAILURE为固定失败策略；进入核查后仍保留，为空表示未要求结清。
|
||||
StopKind *string `db:"stop_kind"`
|
||||
// StopRequestedAt 首次停止意图的数据库时间，重报或后续取消不覆盖；与StopKind同时为空。
|
||||
StopRequestedAt *time.Time `db:"stop_requested_at"`
|
||||
// CompletedAt 终结时间；非终态为空，进入终态时必须写入。
|
||||
CompletedAt *time.Time `db:"completed_at"`
|
||||
// RetryOf 来源记录标识，人工重新发起时用于保留历史关联；可为空，表示尚未设置该关联或值。
|
||||
|
||||
@@ -17,7 +17,7 @@ type PendingAdvance struct {
|
||||
StepID core.ID `db:"step_id"`
|
||||
// ExpectedStepRevision 恢复推进时要求的步骤版本，避免使用过期检查点。
|
||||
ExpectedStepRevision uint64 `db:"expected_step_revision"`
|
||||
// Checkpoint 版本化 JSON；ROOT用于分批/暂停续接，TIMER/POLL另固定激活ID以隔离旧唤醒。
|
||||
// Checkpoint 版本化 JSON；ROOT用于分批/暂停/停止结清续接，TIMER/SIGNAL/POLL另固定激活ID以隔离旧唤醒。
|
||||
Checkpoint core.JSON `db:"checkpoint"`
|
||||
// AvailableAt 最早可被领取或处理的时间，统一 UTC。
|
||||
AvailableAt time.Time `db:"available_at"`
|
||||
|
||||
@@ -17,7 +17,7 @@ func (s InstanceState) Terminal() bool {
|
||||
return s == InstanceSucceeded || s == InstanceFailed || s == InstanceCancelled
|
||||
}
|
||||
func (i Instance) AllowsNewWork() bool {
|
||||
return !i.Paused && i.CancelRequestedAt == nil && (i.State == InstanceRunning || i.State == InstanceWaiting)
|
||||
return !i.Paused && i.StopKind == nil && i.CancelRequestedAt == nil && (i.State == InstanceRunning || i.State == InstanceWaiting)
|
||||
}
|
||||
|
||||
type NodeKind string
|
||||
|
||||
Reference in New Issue
Block a user