feat: add audited workflow review and bounded task recovery
This commit is contained in:
@@ -31,17 +31,17 @@
|
|||||||
|
|
||||||
- P4定义发布使用schema_version=1有限结构，词法作用域禁止跨分支/前向读取，任务策略显式冻结；SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁；原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务；同版本不同内容不得覆盖。保留精确数字，校验存储原文与结构摘要；不得把发布通过或DSL支持误称为实例执行已实现。
|
- P4定义发布使用schema_version=1有限结构，词法作用域禁止跨分支/前向读取，任务策略显式冻结；SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁；原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务；同版本不同内容不得覆盖。保留精确数字，校验存储原文与结构摘要；不得把发布通过或DSL支持误称为实例执行已实现。
|
||||||
|
|
||||||
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归（docs/reports/phase-4-definitions.md）。实例启动/固定版本/首批任务同事务的内部存储已补齐；内部事件消费、顺序与组合推进及根接续已接入；TIMER与后续POLL时钟已接入，SIGNAL及最终查询竞争已接入，内部控制与停止结清已接入，继续人工恢复及实例公共SDK；P4整体和P5–P7仍未完成。
|
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归（docs/reports/phase-4-definitions.md）。实例启动/固定版本/首批任务同事务的内部存储已补齐；内部事件消费、顺序与组合推进及根接续已接入；TIMER与后续POLL时钟已接入，SIGNAL及最终查询竞争已接入，内部控制与停止结清已接入，继续POLL/等待恢复及实例公共SDK；P4整体和P5–P7仍未完成。
|
||||||
|
|
||||||
- 内部结果交付使用workflow_activation所有者并固定激活ID，TASK经activation.task_id、POLL经轮次校验归属；weave_inbox来源及独立快照与Pulse结果同事务，先于外部结果流行锁，不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史，禁止用当前任务重建旧事件。
|
- 内部结果交付使用workflow_activation所有者并固定激活ID，TASK经activation.task_id、POLL经轮次校验归属；weave_inbox来源及独立快照与Pulse结果同事务，先于外部结果流行锁，不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史，禁止用当前任务重建旧事件。
|
||||||
- weave/engine仅计算有限决策；实例受理、Inbox消费与根接续已接入，等待和停止结清运行器已接入，人工恢复继续开发。Plan最多32个变更，More必须持久化待续；耗尽变更预算仍须遍历其他已活动分支，不能误判在途工作不可达。冻结分支选择，精确数值，不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务，不能把纯快照往返测试当作真实进程恢复。
|
- weave/engine仅计算有限决策；实例受理、Inbox消费与根接续已接入，等待和停止结清运行器已接入，内部核查/TASK恢复已接入，POLL/等待恢复继续开发。Plan最多32个变更，More必须持久化待续；耗尽变更预算仍须遍历其他已活动分支，不能误判在途工作不可达。冻结分支选择，精确数值，不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务，不能把纯快照往返测试当作真实进程恢复。
|
||||||
|
|
||||||
- 实例内部受理已实现，信号内部运行器已接入，公共实例/信号HTTP及独立SDK尚未开放；结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作，所有失败同事务回滚。首批推进递增revision至2，原键返回冻结回执且仍核验当前权限，不重算截止。
|
- 实例内部受理已实现，信号内部运行器已接入，公共实例/信号HTTP及独立SDK尚未开放；结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作，所有失败同事务回滚。首批推进递增revision至2，原键返回冻结回执且仍核验当前权限，不重算截止。
|
||||||
- 实例根步骤`$root`/JOIN不属于用户定义，Steps查询排除它；More对应根步骤版本化持久检查点。TASK直接关联激活task_id，POLL通过轮次关联，实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结，旧NULL等待须核实，不能盲目解释；新关联512字节、存储兼容历史764字节。
|
- 实例根步骤`$root`/JOIN不属于用户定义，Steps查询排除它；More对应根步骤版本化持久检查点。TASK直接关联激活task_id，POLL通过轮次关联，实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结，旧NULL等待须核实，不能盲目解释；新关联512字节、存储兼容历史764字节。
|
||||||
|
|
||||||
- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁，禁止先锁Inbox再锁父资源。来源使用独立冻结快照，旧激活/低版本留痕；数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。
|
- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁，禁止先锁Inbox再锁父资源。来源使用独立冻结快照，旧激活/低版本留痕；数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。
|
||||||
- 后继满额用SAVEPOINT回滚整轮新工作，保留已接受结果和待续；恢复必须重读持久图。多次根推进使用独立系统审计操作身份；迟到证据也递增实例历史版本，禁止复用已有workflow.advanced唯一身份。
|
- 后继满额用SAVEPOINT回滚整轮新工作，保留已接受结果和待续；恢复必须重读持久图。多次根推进使用独立系统审计操作身份；迟到证据也递增实例历史版本，禁止复用已有workflow.advanced唯一身份。
|
||||||
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证，见docs/reports/phase-4-runtime.md。TIMER到期与POLL后续轮次已接入，见docs/13-workflow-clock.md；SIGNAL及最终信号竞争已接入（docs/14-workflow-signals.md）；内部FAIL_AFTER_SETTLED分支结清与控制已接入（docs/15-workflow-controls.md）；人工恢复、公共控制API和实例SDK仍须完成，不得按部分自动运行宣称P4/P5完成。
|
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证，见docs/reports/phase-4-runtime.md。TIMER到期与POLL后续轮次已接入，见docs/13-workflow-clock.md；SIGNAL及最终信号竞争已接入（docs/14-workflow-signals.md）；内部FAIL_AFTER_SETTLED分支结清与控制已接入（docs/15-workflow-controls.md）；POLL/等待恢复、公共控制API和实例SDK仍须完成，不得按部分自动运行宣称P4/P5完成。
|
||||||
|
|
||||||
- TIMER/POLL复用weave_pending_advances及到期索引，时钟固定激活ID；迁移000019只回填原等待/查询时钟，不改截止和预算。后续POLL只由已确认POLL_PENDING开启，首轮精确输入/资源键冻结，每轮Pulse尝试独立计数；满额/暂停不消耗轮数。截止遇到已落库结果先等待Inbox，仍在途则关闭门控但保留ACTIVE和当前任务，许可由Pulse结清；不得由时钟假定外部请求已结束。
|
- TIMER/POLL复用weave_pending_advances及到期索引，时钟固定激活ID；迁移000019只回填原等待/查询时钟，不改截止和预算。后续POLL只由已确认POLL_PENDING开启，首轮精确输入/资源键冻结，每轮Pulse尝试独立计数；满额/暂停不消耗轮数。截止遇到已落库结果先等待Inbox，仍在途则关闭门控但保留ACTIVE和当前任务，许可由Pulse结清；不得由时钟假定外部请求已结束。
|
||||||
|
|
||||||
@@ -50,3 +50,6 @@
|
|||||||
|
|
||||||
- 流程控制使用显式workflows:control和独立control_revision；后台结果只改变通用revision。控制标记、门控、原操作回执、根接续、事件与审计同事务，原键恢复仍核对当前权限。RESUME不解除核查或停止意图。
|
- 流程控制使用显式workflows:control和独立control_revision；后台结果只改变通用revision。控制标记、门控、原操作回执、根接续、事件与审计同事务，原键恢复仍核对当前权限。RESUME不解除核查或停止意图。
|
||||||
- StopKind/StopRequestedAt冻结首次CANCEL或FAILURE目标，后来的用户取消不覆盖原失败目标。结清复用ROOT，每事务最多一个Pulse取消、32项步骤收尾；结果流锁取得后不能再锁另一任务。终态前校验全部历史任务的双向归属、未结束尝试和未释放许可，UNKNOWN不得终结。迁移000021恢复旧停止意图，不添加外键。
|
- StopKind/StopRequestedAt冻结首次CANCEL或FAILURE目标，后来的用户取消不覆盖原失败目标。结清复用ROOT，每事务最多一个Pulse取消、32项步骤收尾；结果流锁取得后不能再锁另一任务。终态前校验全部历史任务的双向归属、未结束尝试和未释放许可，UNKNOWN不得终结。迁移000021恢复旧停止意图，不添加外键。
|
||||||
|
|
||||||
|
- 人工核查使用显式workflows:recover、当前实例revision、步骤/任务版本及激活身份，必须有原因和证据；执行结论、步骤结论、解除核查分别确认。新决定拒绝PENDING/QUARANTINED Inbox，原键回执仍核验权限。核查回执/额度/双审计/结果流同事务，结果流锁最后取得。
|
||||||
|
- TASK人工重试保留首次绝对截止和精确输入，以新激活/关联任务继续消耗原max_attempts；激活数也受该上限限制，不能无限制造零尝试任务。成功步骤、未知执行、原停止意图或预算耗尽均不得重试。Continue不解除paused、重置预算或撤销停止。POLL跨激活与等待恢复尚未实现，公共HTTP/SDK也未开放，不能据此宣称P5完成。
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ Orbit 是面向多个业务项目的通用任务调度与流程编排平台，
|
|||||||
|
|
||||||
已完成 P0–P3 平台功能阶段验收：35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过；[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK，[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例，[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现；[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例；[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例；[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景；[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本；[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI；[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复；完整范围核对见[P3验收](docs/reports/phase-3.md)，流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。
|
已完成 P0–P3 平台功能阶段验收：35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过；[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK，[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例，[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现；[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例；[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例；[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景；[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本；[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI；[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复；完整范围核对见[P3验收](docs/reports/phase-3.md)，流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。
|
||||||
|
|
||||||
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现；已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。[定时等待、后续查询轮次及原预算恢复](docs/13-workflow-clock.md)已接入；[外部信号和查询最终信号竞争](docs/14-workflow-signals.md)已接入内部运行器；[暂停/继续/取消与失败结清](docs/15-workflow-controls.md)已接入内部运行器；人工恢复及实例公共API/SDK继续开发，发布定义不会启动任务。
|
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现；已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。[定时等待、后续查询轮次及原预算恢复](docs/13-workflow-clock.md)已接入；[外部信号和查询最终信号竞争](docs/14-workflow-signals.md)已接入内部运行器；[暂停/继续/取消与失败结清](docs/15-workflow-controls.md)已接入内部运行器；[人工核查、解除核查与TASK新激活](docs/16-workflow-recovery.md)已接入内部服务；POLL/等待恢复及实例公共API/SDK继续开发，发布定义不会启动任务。
|
||||||
|
|
||||||
数据库禁止外键，关联一致性通过服务端事务校验，索引、唯一约束和 CHECK 保留；readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。
|
数据库禁止外键，关联一致性通过服务端事务校验，索引、唯一约束和 CHECK 保留；readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 流程编排功能与设计
|
# Weave 流程编排功能与设计
|
||||||
|
|
||||||
> 版本 0.6 · 2026-09-18。首版功能范围已确认。定义结构、发布校验与不可变版本已实现，见[定义格式](api/workflow-definition-format.md)和[发布API](api/workflow-definitions.md)；[实例受理](11-workflow-admission.md)、[结果消费与根推进](12-workflow-runtime.md)、[TIMER与POLL时钟](13-workflow-clock.md)已接入。[信号等待及最终查询竞争](14-workflow-signals.md)已接入内部运行器；[暂停/继续/取消与停止结清](15-workflow-controls.md)已实现内部链路；人工核查及实例公共API/SDK仍待P4–P5实现验收。
|
> 版本 0.7 · 2026-09-18。首版功能范围已确认。定义结构、发布校验与不可变版本已实现，见[定义格式](api/workflow-definition-format.md)和[发布API](api/workflow-definitions.md)；[实例受理](11-workflow-admission.md)、[结果消费与根推进](12-workflow-runtime.md)、[TIMER与POLL时钟](13-workflow-clock.md)已接入。[信号等待及最终查询竞争](14-workflow-signals.md)已接入内部运行器；[暂停/继续/取消与停止结清](15-workflow-controls.md)已实现内部链路；[人工核查与TASK恢复](16-workflow-recovery.md)已接入内部服务；POLL/等待恢复及实例公共API/SDK仍待P4–P5实现验收。
|
||||||
|
|
||||||
## 1. 职责与边界
|
## 1. 职责与边界
|
||||||
|
|
||||||
@@ -174,7 +174,7 @@ Weave 不执行特定业务逻辑，也不把整段业务代码放入平台进
|
|||||||
3. 在一个事务内记录管理操作，创建新的步骤激活及关联 Pulse 任务，并固定关联关系。
|
3. 在一个事务内记录管理操作，创建新的步骤激活及关联 Pulse 任务，并固定关联关系。
|
||||||
4. 保留旧激活与旧任务结果，沿用恰当的业务幂等身份。
|
4. 保留旧激活与旧任务结果，沿用恰当的业务幂等身份。
|
||||||
|
|
||||||
这是人工恢复的建议模型，具体 API 在评审后冻结。普通任务人工重试入口应拒绝绕过流程管理的关联任务请求。已经确认成功的步骤不能通过“重试流程”随意再执行一次。
|
执行核查、步骤结论、解除核查及TASK新激活已按[人工恢复约定](16-workflow-recovery.md)实现内部服务，原预算和绝对截止不重置；POLL跨激活及等待恢复仍需补齐，公共API另行交付。普通任务人工重试入口应拒绝绕过流程管理的关联任务请求。已经确认成功的步骤不能通过“重试流程”随意再执行一次。
|
||||||
|
|
||||||
取消和补偿是不同业务操作。若未来增加补偿，补偿本身也必须是有权限、可审计、可重试或可核查的业务步骤；不能宣称撤销流程就能撤回已经生效的外部业务操作。
|
取消和补偿是不同业务操作。若未来增加补偿，补偿本身也必须是有权限、可审计、可重试或可核查的业务步骤；不能宣称撤销流程就能撤回已经生效的外部业务操作。
|
||||||
|
|
||||||
|
|||||||
@@ -208,3 +208,7 @@ Signal新增payload_version、target_activation_sequence、evidence_ref、dispos
|
|||||||
## 停止意图增量（000021）
|
## 停止意图增量（000021）
|
||||||
|
|
||||||
Instance新增StopKind和StopRequestedAt，冻结首次用户取消或失败结清目标；CancelRequestedAt仍只表示用户显式取消。成对CHECK防止半条意图，任务所有者索引支持双向归属核验。当前35个实体、303个直接持久化字段声明，均有中文注释。历史停止实例关闭门控并恢复根接续，不改变原截止、预算、控制版本或终态数据；保持零外键，详见[控制约定](15-workflow-controls.md)。
|
Instance新增StopKind和StopRequestedAt，冻结首次用户取消或失败结清目标；CancelRequestedAt仍只表示用户显式取消。成对CHECK防止半条意图，任务所有者索引支持双向归属核验。当前35个实体、303个直接持久化字段声明，均有中文注释。历史停止实例关闭门控并恢复根接续，不改变原截止、预算、控制版本或终态数据；保持零外键，详见[控制约定](15-workflow-controls.md)。
|
||||||
|
|
||||||
|
## 流程人工核查与TASK新激活
|
||||||
|
|
||||||
|
复用现有激活sequence、任务retry_of、审计和幂等回执，不增加表、字段或外键。TASK人工重试以新激活保留旧执行，原尝试预算跨激活扣除，原绝对截止不改变；执行核查和步骤逻辑结论分别保存。35个实体、303个中文注释字段。当前内部支持范围与POLL/等待恢复剩余项见[核查约定](16-workflow-recovery.md)。
|
||||||
|
|||||||
@@ -28,8 +28,8 @@
|
|||||||
| P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件；SDK 客户端 | API/SDK 创建查询闭环；重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归（见 P1 报告；真实断路/提交 ACK 丢失注入留后续故障阶段） |
|
| P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件；SDK 客户端 | API/SDK 创建查询闭环；重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归（见 P1 报告；真实断路/提交 ACK 丢失注入留后续故障阶段） |
|
||||||
| P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复；Worker SDK | 真 Worker 执行；失联/重连/重复确认/旧尝试拒绝；业务效果未知转核查；节点故障接管 | 已完成并通过阶段回归：服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复；测试拓扑和剩余生产故障矩阵见 P2 验收报告 |
|
| P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复；Worker SDK | 真 Worker 执行；失联/重连/重复确认/旧尝试拒绝；业务效果未知转核查；节点故障接管 | 已完成并通过阶段回归：服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复；测试拓扑和剩余生产故障矩阵见 P2 验收报告 |
|
||||||
| P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收：[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景；20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 |
|
| P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收：[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景；20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 |
|
||||||
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容；端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现，已通过[子项验收](reports/phase-4-definitions.md)；已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md)；[实例受理及首批事务](reports/phase-4-admission.md)已实现；[结果消费和根接续](reports/phase-4-runtime.md)已接入后台，顺序/分支/并行及真实进程恢复已验证；[TIMER与后续POLL轮次](13-workflow-clock.md)已接入；[信号消费和最终查询竞争](14-workflow-signals.md)已接入内部运行器；[内部控制与停止结清](15-workflow-controls.md)已接入；人工恢复和实例公共API/SDK继续开发 |
|
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容；端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现，已通过[子项验收](reports/phase-4-definitions.md)；已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md)；[实例受理及首批事务](reports/phase-4-admission.md)已实现；[结果消费和根接续](reports/phase-4-runtime.md)已接入后台，顺序/分支/并行及真实进程恢复已验证；[TIMER与后续POLL轮次](13-workflow-clock.md)已接入；[信号消费和最终查询竞争](14-workflow-signals.md)已接入内部运行器；[内部控制与停止结清](15-workflow-controls.md)已接入；[内部核查与TASK新激活](16-workflow-recovery.md)已接入；POLL/等待恢复和实例公共API/SDK继续开发 |
|
||||||
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算；暂停/开始/回执/超时竞争；恢复保留预算；未知结果不盲目重复 | 有界查询轮次/尝试独立预算及原截止恢复已接入内部运行器；最终信号竞争及在途查询结清已接入；内部暂停/继续/取消与失败结清已接入；人工恢复及公共API/SDK继续开发 |
|
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算；暂停/开始/回执/超时竞争；恢复保留预算；未知结果不盲目重复 | 有界查询轮次/尝试独立预算及原截止恢复已接入内部运行器；最终信号竞争及在途查询结清已接入；内部暂停/继续/取消与失败结清已接入；执行核查、步骤结论、解除核查及TASK新激活已接入；POLL/等待恢复及公共API/SDK继续开发 |
|
||||||
| P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行，报告环境/指标/遗留项；容量、RPO/RTO 按真实拓扑验证 | 待开发 |
|
| P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行，报告环境/指标/遗留项；容量、RPO/RTO 按真实拓扑验证 | 待开发 |
|
||||||
| P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例；所有首版能力可追溯到实现与测试；报告实际生产边界 | 待开发 |
|
| P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例；所有首版能力可追溯到实现与测试；报告实际生产边界 | 待开发 |
|
||||||
|
|
||||||
@@ -41,7 +41,7 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
|||||||
|
|
||||||
1. 已实现FAIL_AFTER_SETTLED与显式取消的内部分支结清，见[控制报告](reports/phase-5-controls.md)：持久取消意图、复用Pulse取消、收集已发出查询/任务结果、等待全部已激活工作结清后才进入失败或取消终态。效果未知仍保留核查。
|
1. 已实现FAIL_AFTER_SETTLED与显式取消的内部分支结清，见[控制报告](reports/phase-5-controls.md)：持久取消意图、复用Pulse取消、收集已发出查询/任务结果、等待全部已激活工作结清后才进入失败或取消终态。效果未知仍保留核查。
|
||||||
2. 已实现实例暂停/继续/取消内部操作，以control_revision、操作幂等身份、明确原因及审计保护控制决定；暂停不延长原截止，继续不覆盖核查或取消状态。
|
2. 已实现实例暂停/继续/取消内部操作，以control_revision、操作幂等身份、明确原因及审计保护控制决定；暂停不延长原截止，继续不覆盖核查或取消状态。
|
||||||
3. 完成人工核查恢复与旧激活隔离，保留原预算、结果和证据，不能通过换激活/新键绕过未知效果或执行上限。
|
3. 已实现内部执行核查、步骤结论、解除核查及TASK新激活和旧激活隔离，见[核查报告](reports/phase-5-recovery.md)。继续POLL跨激活及等待恢复，保留原预算、轮次、结果和证据，不能通过换激活/新键绕过未知效果或执行上限。
|
||||||
4. 同步开放实例/步骤/信号/来源/审计HTTP、独立SDK和端到端场景，覆盖原键恢复、权限、版本冲突、竞争与进程全停；公共能力逐项记录实际验收结果。
|
4. 同步开放实例/步骤/信号/来源/审计HTTP、独立SDK和端到端场景，覆盖原键恢复、权限、版本冲突、竞争与进程全停；公共能力逐项记录实际验收结果。
|
||||||
5. 转入P6/P7的生产故障、容量、运维、发布与全部文档核对。内部存储阶段通过不会提前勾选公共接口或生产验收。
|
5. 转入P6/P7的生产故障、容量、运维、发布与全部文档核对。内部存储阶段通过不会提前勾选公共接口或生产验收。
|
||||||
|
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ v16及以前的未处理记录缺少冻结来源，升级后统一标为QUARANTI
|
|||||||
|
|
||||||
## 4. 后续存储适配必须满足的条件
|
## 4. 后续存储适配必须满足的条件
|
||||||
|
|
||||||
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证；第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md)，第5项的TIMER/POLL已接入[时钟运行器](13-workflow-clock.md)，第5项的信号等待已接入[信号运行器](14-workflow-signals.md)，内部控制与结清已接入[控制运行器](15-workflow-controls.md)，人工恢复、接口和SDK仍继续实现：
|
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证；第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md)，第5项的TIMER/POLL已接入[时钟运行器](13-workflow-clock.md)，第5项的信号等待已接入[信号运行器](14-workflow-signals.md)，内部控制与结清已接入[控制运行器](15-workflow-controls.md)，[核查与TASK新激活](16-workflow-recovery.md)已接入；POLL/等待恢复、接口和SDK仍继续实现：
|
||||||
|
|
||||||
1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止，并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。
|
1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止，并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。
|
||||||
2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本；消费确认、步骤变化、后续任务或持久化待续记录同事务提交。
|
2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本；消费确认、步骤变化、后续任务或持久化待续记录同事务提交。
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 实例受理与首批工作
|
# Weave 实例受理与首批工作
|
||||||
|
|
||||||
状态：内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口；[内部控制和结清](15-workflow-controls.md)已接入，人工恢复继续开发；[信号运行器](14-workflow-signals.md)已接入。** Weave相关公开SDK目前仅支持定义发布。[TIMER和后续POLL轮次](13-workflow-clock.md)已接入，TASK及组合节点已可自动推进，不能将此解释成全部等待/控制能力已实现。
|
状态：内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口；[内部控制和结清](15-workflow-controls.md)已接入，[内部核查及TASK恢复](16-workflow-recovery.md)已接入，POLL/等待恢复继续开发；[信号运行器](14-workflow-signals.md)已接入。** Weave相关公开SDK目前仅支持定义发布。[TIMER和后续POLL轮次](13-workflow-clock.md)已接入，TASK及组合节点已可自动推进，不能将此解释成全部等待/控制能力已实现。
|
||||||
|
|
||||||
## 1. 受理事务
|
## 1. 受理事务
|
||||||
|
|
||||||
@@ -49,7 +49,7 @@ Get返回当前实例视图；Start重报返回受理时冻结的原始视图，
|
|||||||
| 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例；终态不计入；满额仍可恢复原受理回执 |
|
| 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例；终态不计入；满额仍可恢复原受理回执 |
|
||||||
| 信号关联 | 新受理最多512字节；数据库兼容旧191字符最长764字节，升级不截断历史 |
|
| 信号关联 | 新受理最多512字节；数据库兼容旧191字符最长764字节，升级不截断历史 |
|
||||||
| 截止 | 数据库时间生成；任务、查询与等待均不超过实例总截止 |
|
| 截止 | 数据库时间生成；任务、查询与等待均不超过实例总截止 |
|
||||||
| 自动运行 | Inbox消费、根待续、TIMER到期及后续POLL轮次已接入；SIGNAL与最终查询竞争已接入，内部控制与结清已接入，人工恢复仍待实现 |
|
| 自动运行 | Inbox消费、根待续、TIMER到期及后续POLL轮次已接入；SIGNAL与最终查询竞争已接入，内部控制与结清已接入，内部核查及TASK恢复已接入，POLL/等待恢复仍待实现 |
|
||||||
|
|
||||||
迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段；NULL仅表示TIMER或尚未核实的历史元数据，未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储；实例business_key改为utf8mb4_0900_bin。该迁移时为35个实体、含公共Record的295个直接持久化字段声明，全部有中文注释，继续保持零外键。
|
迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段；NULL仅表示TIMER或尚未核实的历史元数据，未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储；实例business_key改为utf8mb4_0900_bin。该迁移时为35个实体、含公共Record的295个直接持久化字段声明，全部有中文注释，继续保持零外键。
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 结果消费与自动推进
|
# Weave 结果消费与自动推进
|
||||||
|
|
||||||
状态：内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。[定时等待和后续查询轮次](13-workflow-clock.md)也已接入。[信号等待与最终查询竞争](14-workflow-signals.md)已接入。实例公共HTTP/SDK仍未开放，[失败后的分支取消结清与内部控制](15-workflow-controls.md)已接入，人工恢复继续开发，不能将本阶段等同于完整Weave交付。
|
状态：内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。[定时等待和后续查询轮次](13-workflow-clock.md)也已接入。[信号等待与最终查询竞争](14-workflow-signals.md)已接入。实例公共HTTP/SDK仍未开放，[失败后的分支取消结清与内部控制](15-workflow-controls.md)已接入，[内部核查及TASK恢复](16-workflow-recovery.md)已接入，POLL/等待恢复继续开发，不能将本阶段等同于完整Weave交付。
|
||||||
|
|
||||||
## 1. 后台运行和事务边界
|
## 1. 后台运行和事务边界
|
||||||
|
|
||||||
@@ -64,4 +64,4 @@ flowchart LR
|
|||||||
|
|
||||||
进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过；跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。
|
进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过；跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。
|
||||||
|
|
||||||
持久化信号等待及查询最终信号竞争已按[信号运行约定](14-workflow-signals.md)实现。内部控制及停止结清也已接入。下一步仍包括人工恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
持久化信号等待及查询最终信号竞争已按[信号运行约定](14-workflow-signals.md)实现。内部控制及停止结清也已接入。下一步仍包括POLL/等待恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 控制与停止结清
|
# Weave 控制与停止结清
|
||||||
|
|
||||||
状态：内部Go暂停、继续、取消服务及后台失败/取消结清运行器已接入。**公共实例/控制HTTP与独立SDK尚未开放，人工核查恢复仍在后续交付范围**。本文说明已实现的持久化规则，不把内部存储回归当作全部Weave或生产验收。
|
状态：内部Go暂停、继续、取消服务及后台失败/取消结清运行器已接入。**公共实例/控制HTTP与独立SDK尚未开放，[内部核查及TASK新激活](16-workflow-recovery.md)已接入，POLL/等待恢复仍在后续交付范围**。本文说明已实现的持久化规则，不把内部存储回归当作全部Weave或生产验收。
|
||||||
|
|
||||||
## 1. 控制操作
|
## 1. 控制操作
|
||||||
|
|
||||||
@@ -60,4 +60,4 @@ CANCELLED仅表示本次流程停止已结清，不撤销先前成功的业务
|
|||||||
|
|
||||||
[控制及结清回归](../internal/storage/mysql/workflow_controls_integration_test.go)覆盖控制版本、权限重报、开始竞争、失败/未知、审计回滚、信号与查询及超过32步骤接续；[真实v20升级](../internal/storage/mysql/workflow_stop_upgrade_integration_test.go)验证兼容回填；[实际Orbit进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖停止意图和结果在全停期间保存后由两个新进程接续。实际结果见[阶段报告](reports/phase-5-controls.md)。
|
[控制及结清回归](../internal/storage/mysql/workflow_controls_integration_test.go)覆盖控制版本、权限重报、开始竞争、失败/未知、审计回滚、信号与查询及超过32步骤接续；[真实v20升级](../internal/storage/mysql/workflow_stop_upgrade_integration_test.go)验证兼容回填；[实际Orbit进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖停止意图和结果在全停期间保存后由两个新进程接续。实际结果见[阶段报告](reports/phase-5-controls.md)。
|
||||||
|
|
||||||
下一步仍需完成人工核查恢复、公共实例/控制/信号/审计接口、独立SDK及可运行场景，再完成P6–P7生产故障、性能、运维和发布验收。当前测试不替代生产MySQL复制、故障切换、备份恢复或RPO/RTO验证。
|
下一步仍需完成POLL/等待恢复、公共实例/控制/信号/审计接口、独立SDK及可运行场景，再完成P6–P7生产故障、性能、运维和发布验收。当前测试不替代生产MySQL复制、故障切换、备份恢复或RPO/RTO验证。
|
||||||
|
|||||||
76
docs/16-workflow-recovery.md
Normal file
76
docs/16-workflow-recovery.md
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
# Weave 人工核查与接续
|
||||||
|
|
||||||
|
状态：内部Go服务已提供执行核查、步骤结论确认、解除核查，以及TASK新激活重试。公共HTTP、独立SDK和通用接入示例尚未交付；**POLL新激活恢复尚未实现，不能把已有逻辑结果核查当作查询重试已经完成**。完整交付继续按[开发计划](08-development-plan.md)推进。
|
||||||
|
|
||||||
|
## 1. 四类操作
|
||||||
|
|
||||||
|
[内部服务](../weave/recovery/service.go)要求显式`workflows:recover`权限，普通Worker、任务核查和流程暂停权限均不自动包含它。四个方法都接收认证身份、实例ID、稳定幂等键和命令，命名空间与操作者从认证取得。
|
||||||
|
|
||||||
|
| 方法 | 作用 | 不会自动发生的事情 |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| ReviewTask | 核实本实例历史激活拥有的NEEDS_REVIEW任务，写入SUCCEEDED/FAILED、明确执行已结束和证据 | 不把查询请求成功解释成业务成功，不解除步骤或实例核查 |
|
||||||
|
| ResolveStep | 将当前待核查/失败/取消叶子步骤核实为SUCCEEDED或FAILED，保存符合固定契约的逻辑结果 | 不重新执行旧任务，不清除实例核查或原停止意图 |
|
||||||
|
| Continue | 在当前证据和步骤状态允许时解除实例核查，保存根接续 | 不解除显式paused，不延长截止，不撤销取消或失败结清 |
|
||||||
|
| RetryStep | 为已确认失败/取消的TASK建立新的激活和关联任务，继续消耗原尝试预算 | 不重跑成功步骤，不改变输入/业务关联，不为POLL另起新轮次 |
|
||||||
|
|
||||||
|
查询请求、外部业务结果和流程接续是不同事实。例如，查询任务核实为成功但返回PENDING，仍不能将POLL步骤核实为成功；POLL成功结论必须通过固定映射分类为SUCCEEDED。未知执行须先完成ReviewTask，再等待新版本结果进入内部Inbox并消费，才能确认步骤结论。
|
||||||
|
|
||||||
|
## 2. 身份、版本与证据
|
||||||
|
|
||||||
|
所有命令携带：
|
||||||
|
|
||||||
|
| 字段 | 约束 |
|
||||||
|
| --- | --- |
|
||||||
|
| expected_revision | 当前实例revision，JSON十进制字符串；必须大于0，不使用control_revision代替证据版本 |
|
||||||
|
| reason | 1至1024个Unicode字符，非空白，不含NUL |
|
||||||
|
| evidence_ref | 1至512个无空白或控制字符的Unicode字符；受控证据引用，平台不访问该地址 |
|
||||||
|
|
||||||
|
ReviewTask还要求`task_id`、`expected_task_revision`、`execution_settled=true`、SUCCEEDED/FAILED结论和非null JSON结果。声明表示全部历史外部执行已结束且不会继续产生效果；租约失效和UNKNOWN本身不能证明这一点。
|
||||||
|
|
||||||
|
ResolveStep和RetryStep都要求固定`step_key`、当前`activation_id`及`expected_step_revision`。前者要求`execution_settled=true`，后者要求`safe_to_retry=true`。Continue要求`review_resolved=true`，服务端仍独立检查当前图、历史任务与来源事件，不把声明作为绕过检查的开关。
|
||||||
|
|
||||||
|
同键相同命令返回首次冻结回执；同键不同内容冲突。回执恢复仍检查当前凭据和固定定义引用的全部队列权限。后台新增事实使原实例revision失效，新键携带旧版本会冲突；原键可继续核对已提交决定。每个成功人工操作同时递增revision和control_revision，防止旧暂停/继续命令覆盖人工恢复后的控制状态。
|
||||||
|
|
||||||
|
## 3. 事务与历史
|
||||||
|
|
||||||
|
认证 → 应用额度 → 管理操作身份 → 固定版本的排序队列 → 实例 → 门控 → 子项。操作只针对NEEDS_REVIEW实例；新决定要求内部Inbox不存在PENDING或QUARANTINED记录。尚未消费的事实应先消费，损坏来源不能靠人工声明直接放行。
|
||||||
|
|
||||||
|
核查状态、额度、实例/步骤、门控、必要根接续、审计和幂等回执同事务。ReviewTask在所有这些写入完成后发布新的Pulse结果事件，结果流锁最后获取，之后不能再锁另一任务。执行审计与流程审计使用不同操作身份，流程审计明确引用执行审计，二者同事务提交。结果流写入失败时全部回滚。
|
||||||
|
|
||||||
|
步骤核查先校验所有历史任务的双向归属，再检查该步骤全部历史任务均已知结清、无有效尝试和未释放许可。兄弟分支仍可收集自己的在途结果；Continue还会拒绝任何历史未知结果及当前未解决的步骤。关联检查按命名空间和固定队列执行，不使用数据库外键。
|
||||||
|
|
||||||
|
ResolveStep保存当前激活的新管理结论，审计记录前一状态、前一输出和新结果摘要。Pulse原成功/失败结果不因此改写，执行尝试、旧事件、等待和查询轮次记录均保留。真正重新执行时才由RetryStep创建新激活，旧激活及旧任务结果保持原样，旧激活迟到结果只留痕。
|
||||||
|
|
||||||
|
## 4. TASK重试预算
|
||||||
|
|
||||||
|
当前仅TASK支持RetryStep。要求没有取消或失败停止意图，实例和原任务绝对截止尚未到达，相关历史执行均已知失败或取消。核实实际尝试行数、任务attempt_count、连续激活序号、retry_of链、固定处理器/输入/队列/优先级和策略一致性。
|
||||||
|
|
||||||
|
- 新激活sequence递增；新任务retry_of指向前一任务，同一原任务只允许一个直接后继。
|
||||||
|
- 新max_attempts = 固定定义max_attempts − 全部历史激活已用尝试数；首次尝试也计入。
|
||||||
|
- 新任务沿用首次激活的绝对deadline_at、最晚开始时间、精确输入、business_key与concurrency_key，不因人工操作重新计时。
|
||||||
|
- 激活总数也不超过原max_attempts，避免反复取消尚未开始的任务而无限创建零尝试激活。
|
||||||
|
- 原预算耗尽时拒绝新激活；业务关联键不自动提供幂等，接入方须在固定输入中保存正确的外部业务操作身份，并提供安全重试证据。
|
||||||
|
- 新任务受当前应用和队列配额约束；满额或审计失败不能留下新激活、孤立任务或多扣额度。
|
||||||
|
|
||||||
|
创建新任务后实例仍处于核查，门控保持关闭。操作者确认其他事项已处理后调用Continue，才允许新任务取得执行授权；原paused仍为true时还需正常RESUME。
|
||||||
|
|
||||||
|
POLL恢复仍需实现跨激活的原轮次归属、单轮剩余尝试及原查询截止传递。当前RetryStep明确拒绝POLL，禁止通过新建业务轮次规避耗尽的单轮预算。SIGNAL/TIMER逻辑结论可核查，但不通过此操作重新开始等待；恢复等待/查询的完整范围仍需后续实现与验收。
|
||||||
|
|
||||||
|
## 5. 停止与到期
|
||||||
|
|
||||||
|
存在原stop_kind时，核查可消除未知事实，但Continue只返回CANCELLING并接续原停止目标，门控继续关闭；停止运行器核验所有工作结清后才进入CANCELLED或FAILED。后来核实某一步成功也不能撤销先前已提交的停止意图。
|
||||||
|
|
||||||
|
无停止意图且原实例已过期时，Continue拒绝启动新工作。可以继续保存核查证据、确认步骤结论，并通过已有CANCEL和停止结清路径结束实例。TIMER成功核查还要求原等待时长确实已到，成功输出必须为显式null，不能人为提前满足时间条件。
|
||||||
|
|
||||||
|
## 6. 内部调用顺序
|
||||||
|
|
||||||
|
| 场景 | 顺序 |
|
||||||
|
| --- | --- |
|
||||||
|
| TASK效果未知，核实成功 | 读取当前实例/任务 → ReviewTask(SUCCEEDED) → 等待内部结果消费 → 读取当前版本 → ResolveStep(SUCCEEDED) → Continue → 后台接续 |
|
||||||
|
| TASK已确认失败，原预算仍有余量 | 读取当前实例/步骤/激活 → RetryStep → 核对新任务及原截止 → Continue → Worker执行新任务 |
|
||||||
|
| POLL次数耗尽，人工取得最终结果 | 核实历史请求均已结束 → ResolveStep(符合固定成功映射的结果) → Continue；原used_rounds不改变 |
|
||||||
|
| 取消后出现UNKNOWN | ReviewTask → 等待消费 → ResolveStep → Continue → 原取消结清；不创建后继工作 |
|
||||||
|
|
||||||
|
这些是内部服务流程说明，公共接入方目前不能调用未开放的HTTP路由或SDK。后续接口文档和可运行场景须覆盖相同证据与版本要求。
|
||||||
|
|
||||||
|
[存储专项](../internal/storage/mysql/workflow_recovery_integration_test.go)、[实际进程测试](../internal/storage/mysql/workflow_runtime_process_integration_test.go)和[阶段报告](reports/phase-5-recovery.md)记录验证范围。本阶段不替代生产数据库高可用、灾难恢复和容量验收。
|
||||||
40
docs/reports/phase-5-recovery.md
Normal file
40
docs/reports/phase-5-recovery.md
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
# P5 人工核查与TASK恢复回归
|
||||||
|
|
||||||
|
日期：2026-09-18。范围为内部执行核查、叶子步骤结论确认、解除核查和TASK新激活重试。POLL新激活恢复、等待恢复、公共HTTP/独立SDK和完整通用示例仍未完成，持续目标保持完整。
|
||||||
|
|
||||||
|
## 已实现与边界
|
||||||
|
|
||||||
|
实现约定见[核查文档](../16-workflow-recovery.md)。执行事实和业务步骤结论分开确认；显式权限、最新实例/任务/步骤版本、当前激活、原因及证据共同约束操作。新TASK激活保留原输入和绝对截止，历史尝试从原预算扣除，旧激活与旧任务结果不覆盖。
|
||||||
|
|
||||||
|
任务核查、流程审计、内部结果和外部结果流在同一事务提交。未知事实、未结束尝试、未释放许可、待消费或隔离Inbox不能由继续操作跳过。原停止目标、暂停标记和固定定义不改变。无新数据库表或持久化字段，仍为35个实体、303个中文注释字段，禁止外键与级联删除。
|
||||||
|
|
||||||
|
## 环境与证据
|
||||||
|
|
||||||
|
本机darwin/arm64、Go 1.27.1、Docker MySQL 8.4.10，一次性隔离数据库。
|
||||||
|
|
||||||
|
| 检查 | 当前结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| 首轮专项 | 原任务恢复/流程控制通过；新核查两个测试夹具错误：权限ID误以二进制写入JSON、查询助手已消费后再次消费；已修正，该轮不计为通过 |
|
||||||
|
| 修订后专项 | 11组新核查场景、原任务恢复与实际流程进程通过，`-race`，48.899秒；新核查4.47秒、流程进程33.85秒 |
|
||||||
|
| 最终增补 | 增加实际门控保持、跨实例任务/旧版本/非法输出拒绝、原实例过期只能结清等保护与回归 |
|
||||||
|
| 输入与截止来源补强 | 重新验证首次输入契约和截止计算来源后，14组核查、原任务恢复及实际流程进程通过，48.964秒；流程进程32.37秒 |
|
||||||
|
| 最终叶子步骤专项 | 16组核查、原任务恢复及原流程控制通过，29.950秒；核查9.68秒。增加SIGNAL过期核查及TIMER不能提前满足原等待时长的真实时钟验证，生产逻辑未再修改 |
|
||||||
|
| 基础回归 | 补强后再次运行`make fmt test vet build`通过，退出0 |
|
||||||
|
| 完整隔离回归 | 整条脚本通过，退出0；开始时13组核查及全部存储187.796秒，实际流程进程31.601秒，Worker SDK 94.238秒，最终SDK进程故障组90.200秒；19个SDK示例和2个独立业务示例全部通过 |
|
||||||
|
| 数据结构与文档 | 实际MySQL外键0、无COMMENT字段0；303个持久化字段声明均有中文注释；908个本地链接、53段JSON检查通过 |
|
||||||
|
|
||||||
|
进程测试确认全部控制节点退出后，人工执行核查的新结果由两个新节点消费；再次全停后提交解除核查，原取消目标被正确结清。另一个已失败TASK在全停期间建立新激活并保留原任务截止和剩余1次尝试，两个新节点接管后只执行新任务，原失败激活保留。
|
||||||
|
|
||||||
|
完整链路开始后发现首次输入契约与截止来源还可加强，未中断正在执行的完整脚本。该增补由后续真实MySQL专项及重新构建的实际进程验证；不能把此前整套存储187.796秒的记录描述为包含这项增补。最后增加的SIGNAL/TIMER测试也已通过独立专项及vet，生产逻辑未再修改。各临时数据库容器均已由脚本清理。
|
||||||
|
|
||||||
|
存储回归还覆盖同键12路并发、旧激活真实结果重放、普通Pulse入口不得绕过流程归属、撤销队列授权后拒绝原回执、结果流故障全事务回滚、配额满额及审计失败不留新任务、PENDING不能变为业务成功、原任务截止不重置、暂停保留及停止意图恢复。
|
||||||
|
|
||||||
|
完整复现入口为[隔离MySQL脚本](../../scripts/test-mysql.sh)。未提供一次性DSN或实际二进制时跳过的测试不构成通过；专项使用真实MySQL和构建后的Orbit二进制。
|
||||||
|
|
||||||
|
## 未完成范围
|
||||||
|
|
||||||
|
1. POLL跨激活恢复及等待恢复：保留原查询截止、轮次身份、每轮尝试预算和迟到信号隔离，不把剩余范围改成仅支持TASK。
|
||||||
|
2. 实例/步骤/控制/信号/恢复/来源/审计公共HTTP、独立SDK、端到端场景与完整P4–P5验收。
|
||||||
|
3. P6–P7：监控、故障/容量证据、生产MySQL切换与备份恢复、发布和运维交付。
|
||||||
|
|
||||||
|
当前证据只证明对应内部状态与本机单MySQL恢复行为，不宣称整个平台已达到生产验收标准。
|
||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"encoding/hex"
|
"encoding/hex"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"time"
|
||||||
|
|
||||||
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
@@ -61,6 +62,11 @@ func allowTaskRecovery(ctx context.Context, tx *sql.Tx, ns, id core.ID, allowSch
|
|||||||
} else if ownerID != nil {
|
} else if ownerID != nil {
|
||||||
return fault.Conflict
|
return fault.Conflict
|
||||||
}
|
}
|
||||||
|
return taskExecutionSettled(ctx, tx, ns, id)
|
||||||
|
}
|
||||||
|
|
||||||
|
// taskExecutionSettled 仅检查平台内已授权尝试和许可；调用者仍须提供外部执行已结束的证据。
|
||||||
|
func taskExecutionSettled(ctx context.Context, tx *sql.Tx, ns, id core.ID) error {
|
||||||
var active int
|
var active int
|
||||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM pulse_attempts WHERE namespace_id=? AND task_id=? AND state IN ('LEASED','RUNNING')`, ns, id).Scan(&active); err != nil {
|
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM pulse_attempts WHERE namespace_id=? AND task_id=? AND state IN ('LEASED','RUNNING')`, ns, id).Scan(&active); err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -102,23 +108,7 @@ func (s *TaskStore) ReviewTask(ctx context.Context, t auth.Token, id core.ID, ke
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
changed, err := tx.ExecContext(ctx, `UPDATE pulse_tasks SET state=?,result=?,effect_uncertain=FALSE,completed_at=?,updated_at=?,revision=revision+1 WHERE namespace_id=? AND id=? AND revision=? AND state='NEEDS_REVIEW'`, in.Outcome, string(in.Result), now, now, t.NamespaceID, id, in.ExpectedRevision)
|
out, err = applyTaskReview(ctx, tx, t, task, identity, in, now)
|
||||||
if err := exactlyOne(changed, err); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err := releaseQueueTask(ctx, tx, t.NamespaceID, v.QueueID); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
out.Task, err = readTask(ctx, tx, t.NamespaceID, id, false)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
digest, err := requestDigest(in.Result)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
after := map[string]any{"state": in.Outcome, "revision": fmt.Sprint(out.Task.Revision), "effect_uncertain": false, "execution_settled": true, "result_digest": hex.EncodeToString(digest[:])}
|
|
||||||
out.AuditID, err = appendAuditWithEvidence(ctx, tx, t, identity, id, "task", "task.review", in.Reason, in.EvidenceRef, map[string]any{"state": v.State, "revision": fmt.Sprint(v.Revision), "effect_uncertain": v.EffectUncertain}, after)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -183,3 +173,33 @@ func (s *TaskStore) RetryTask(ctx context.Context, t auth.Token, id core.ID, key
|
|||||||
})
|
})
|
||||||
return out, err
|
return out, err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// applyTaskReview 核查状态、额度与任务审计同事务。调用方已校验归属并持锁，须在最后发布结果事件。
|
||||||
|
func applyTaskReview(ctx context.Context, tx *sql.Tx, t auth.Token, task executionTask, identity core.ID, in service.ReviewInput, now time.Time) (service.ReviewResult, error) {
|
||||||
|
var out service.ReviewResult
|
||||||
|
v := task.view
|
||||||
|
if v.State != pulse.TaskNeedsReview || v.Revision != in.ExpectedRevision || task.lease != nil {
|
||||||
|
return out, fault.Conflict
|
||||||
|
}
|
||||||
|
changed, err := tx.ExecContext(ctx, `UPDATE pulse_tasks SET state=?,result=?,effect_uncertain=FALSE,completed_at=?,updated_at=?,revision=revision+1 WHERE namespace_id=? AND id=? AND revision=? AND state='NEEDS_REVIEW'`, in.Outcome, string(in.Result), now, now, t.NamespaceID, v.ID, in.ExpectedRevision)
|
||||||
|
if err := exactlyOne(changed, err); err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
if err := releaseQueueTask(ctx, tx, t.NamespaceID, v.QueueID); err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
out.Task, err = readTask(ctx, tx, t.NamespaceID, v.ID, false)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
digest, err := requestDigest(in.Result)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
after := map[string]any{"state": in.Outcome, "revision": fmt.Sprint(out.Task.Revision), "effect_uncertain": false, "execution_settled": true, "result_digest": hex.EncodeToString(digest[:])}
|
||||||
|
out.AuditID, err = appendAuditWithEvidence(ctx, tx, t, identity, v.ID, "task", "task.review", in.Reason, in.EvidenceRef, map[string]any{"state": v.State, "revision": fmt.Sprint(v.Revision), "effect_uncertain": v.EffectUncertain}, after)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|||||||
292
internal/storage/mysql/workflow_recovery.go
Normal file
292
internal/storage/mysql/workflow_recovery.go
Normal file
@@ -0,0 +1,292 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/jsonvalue"
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||||
|
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// WorkflowRecoveryStore 管理核查事实与接续决定；普通Pulse人工入口仍拒绝流程所有者。
|
||||||
|
type WorkflowRecoveryStore struct{ *TaskStore }
|
||||||
|
|
||||||
|
func NewWorkflowRecoveryStore(db *sql.DB) *WorkflowRecoveryStore {
|
||||||
|
return &WorkflowRecoveryStore{NewTaskStore(db)}
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ recovery.Store = (*WorkflowRecoveryStore)(nil)
|
||||||
|
|
||||||
|
type workflowRecoveryApply func(*sql.Tx, *workflowGraph, core.ID, time.Time, *recovery.Result) (map[string]any, error)
|
||||||
|
|
||||||
|
func (s *WorkflowRecoveryStore) mutate(ctx context.Context, t auth.Token, id core.ID, key, operation string, command any, decision recovery.Decision, apply workflowRecoveryApply) (recovery.Result, error) {
|
||||||
|
var out recovery.Result
|
||||||
|
err := s.transaction(ctx, t, recovery.Action, func(tx *sql.Tx, p auth.Permissions) error {
|
||||||
|
if _, err := lockApplicationResources(ctx, tx, t.NamespaceID, false); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
identity, cached, err := reserveIdentity(ctx, tx, t.NamespaceID, operation, string(id), key, "workflow", id, command)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
g, _, err := lockWorkflowGraph(ctx, tx, t.NamespaceID, id, false)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
for _, q := range g.version.Analysis.Queues {
|
||||||
|
if !p.AllowsQueue(q) {
|
||||||
|
return fault.NotFound
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if cached != nil {
|
||||||
|
if err := json.Unmarshal(cached, &out); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if out.Instance.NamespaceID != t.NamespaceID || out.Instance.ID != id || out.Instance.DefinitionVersionID != g.view.DefinitionVersionID {
|
||||||
|
return errors.New("workflow recovery receipt mismatch")
|
||||||
|
}
|
||||||
|
out.Deduplicated = true
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if g.view.State != weave.InstanceNeedsReview || g.view.Revision != decision.ExpectedRevision {
|
||||||
|
return fault.Conflict
|
||||||
|
}
|
||||||
|
// 新事实须先进入当前图；损坏来源不能用一句人工声明绕过隔离。
|
||||||
|
var pending int
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_inbox WHERE namespace_id=? AND workflow_id=? AND state IN ('PENDING','QUARANTINED')`, t.NamespaceID, id).Scan(&pending); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if pending != 0 {
|
||||||
|
return fault.Conflict
|
||||||
|
}
|
||||||
|
now, err := dbNow(ctx, tx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
before := map[string]any{"state": g.view.State, "revision": g.view.Revision, "control_revision": g.view.ControlRevision, "stop_kind": g.view.StopKind}
|
||||||
|
// 核查期间始终关闭新执行授权；Continue仅在下面完成全部证据校验后按原paused决定是否打开。
|
||||||
|
if _, err := tx.ExecContext(ctx, `UPDATE pulse_execution_gates SET paused=TRUE,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND paused=FALSE`, now, t.NamespaceID, g.gate); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
meta, err := apply(tx, g, identity, now, &out)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET revision=revision+1,control_revision=control_revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, now, t.NamespaceID, id, g.view.Revision)
|
||||||
|
if err := exactlyOne(res, err); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
g.view.Revision++
|
||||||
|
g.view.ControlRevision++
|
||||||
|
meta["state"], meta["revision"], meta["control_revision"], meta["stop_kind"] = g.view.State, g.view.Revision, g.view.ControlRevision, g.view.StopKind
|
||||||
|
out.AuditID, err = appendAuditWithEvidence(ctx, tx, t, identity, id, "workflow", operation, decision.Reason, decision.EvidenceRef, before, meta)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := emitEvent(ctx, tx, t.NamespaceID, id, g.view.Revision, "workflow", "workflow.recovered", map[string]any{"operation": operation, "audit_id": out.AuditID, "state": g.view.State}); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if g.view.StopKind != nil || operation == "workflow.review.continue" {
|
||||||
|
if err := saveWorkflowAdvance(ctx, tx, g, now); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
out.Instance, _, err = readWorkflowInstance(ctx, tx, t.NamespaceID, id, false)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err := saveIdentity(ctx, tx, t.NamespaceID, identity, out); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if operation == "workflow.task.review" {
|
||||||
|
// 只在全部管理状态、回执和审计完成后获取结果流锁；之后不再锁任务。
|
||||||
|
return emitEvent(ctx, tx, t.NamespaceID, out.Task.ID, out.Task.Revision, "task", resultEvent(out.Task.State), map[string]any{"task_id": out.Task.ID, "state": out.Task.State, "effect_uncertain": false, "decision": "MANUAL_REVIEW", "audit_id": out.TaskAuditID, "workflow_audit_id": out.AuditID})
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *WorkflowRecoveryStore) ReviewTask(ctx context.Context, t auth.Token, id core.ID, key string, in recovery.ReviewTaskInput) (recovery.Result, error) {
|
||||||
|
return s.mutate(ctx, t, id, key, "workflow.task.review", in, in.Decision, func(tx *sql.Tx, g *workflowGraph, identity core.ID, now time.Time, out *recovery.Result) (map[string]any, error) {
|
||||||
|
var owned int
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM pulse_tasks t JOIN weave_activations a ON a.namespace_id=t.namespace_id AND a.id=t.owner_id WHERE t.namespace_id=? AND t.id=? AND t.owner_kind='workflow_activation' AND a.workflow_id=?`, t.NamespaceID, in.TaskID, id).Scan(&owned); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if owned != 1 {
|
||||||
|
return nil, fault.NotFound
|
||||||
|
}
|
||||||
|
if err := validateWorkflowStopTask(ctx, tx, g, in.TaskID); err != nil {
|
||||||
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
|
return nil, fault.NotFound
|
||||||
|
}
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
task, err := readExecutionTask(ctx, tx, t.NamespaceID, in.TaskID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := taskExecutionSettled(ctx, tx, t.NamespaceID, in.TaskID); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
taskOperation, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
v, err := applyTaskReview(ctx, tx, t, task, taskOperation, service.ReviewInput{ExpectedRevision: in.ExpectedTaskRevision, Reason: in.Reason, EvidenceRef: in.EvidenceRef, ExecutionSettled: in.ExecutionSettled, Outcome: in.Outcome, Result: in.Result}, now)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out.Task, out.TaskAuditID = &v.Task, &v.AuditID
|
||||||
|
return map[string]any{"task_id": in.TaskID, "task_audit_id": v.AuditID, "task_operation_id": taskOperation, "workflow_operation_id": identity, "execution_settled": true, "outcome": in.Outcome}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func workflowRecoveryStep(g *workflowGraph, in recovery.StepDecision) (*workflowStep, error) {
|
||||||
|
row := g.steps[in.StepKey]
|
||||||
|
if row == nil || in.StepKey == workflowRootKey {
|
||||||
|
return nil, fault.NotFound
|
||||||
|
}
|
||||||
|
if row.view.ActivationID == nil || *row.view.ActivationID != in.ActivationID || row.view.Revision != in.ExpectedStepRevision {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
if row.view.State != weave.StepNeedsReview && row.view.State != weave.StepFailed && row.view.State != weave.StepCancelled {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
return row, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// workflowStepSettled 检查全部历史归属，再核验本步骤的执行与许可；兄弟分支可继续收集在途结果。
|
||||||
|
func workflowStepSettled(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep) error {
|
||||||
|
if _, _, err := verifyWorkflowStopWork(ctx, tx, g); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
queries := []string{
|
||||||
|
`SELECT COUNT(*) FROM weave_activations a JOIN pulse_tasks t ON t.namespace_id=a.namespace_id AND t.owner_id=a.id AND t.owner_kind='workflow_activation' WHERE a.namespace_id=? AND a.workflow_id=? AND a.step_id=? AND (t.state NOT IN ('SUCCEEDED','FAILED','CANCELLED') OR t.effect_uncertain=TRUE)`,
|
||||||
|
`SELECT COUNT(*) FROM weave_activations a JOIN pulse_tasks t ON t.namespace_id=a.namespace_id AND t.owner_id=a.id AND t.owner_kind='workflow_activation' JOIN pulse_attempts x ON x.namespace_id=t.namespace_id AND x.task_id=t.id WHERE a.namespace_id=? AND a.workflow_id=? AND a.step_id=? AND x.state IN ('LEASED','RUNNING')`,
|
||||||
|
`SELECT COUNT(*) FROM weave_activations a JOIN pulse_tasks t ON t.namespace_id=a.namespace_id AND t.owner_id=a.id AND t.owner_kind='workflow_activation' JOIN pulse_attempts x ON x.namespace_id=t.namespace_id AND x.task_id=t.id JOIN pulse_permits p ON p.namespace_id=x.namespace_id AND p.attempt_id=x.id WHERE a.namespace_id=? AND a.workflow_id=? AND a.step_id=? AND p.released_at IS NULL`,
|
||||||
|
}
|
||||||
|
for _, q := range queries {
|
||||||
|
var count int
|
||||||
|
if err := tx.QueryRowContext(ctx, q, g.view.NamespaceID, g.view.ID, row.view.ID).Scan(&count); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if count != 0 {
|
||||||
|
return fault.Conflict
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *WorkflowRecoveryStore) ResolveStep(ctx context.Context, t auth.Token, id core.ID, key string, in recovery.ResolveStepInput) (recovery.Result, error) {
|
||||||
|
return s.mutate(ctx, t, id, key, "workflow.step.resolve", in, in.Decision, func(tx *sql.Tx, g *workflowGraph, _ core.ID, now time.Time, out *recovery.Result) (map[string]any, error) {
|
||||||
|
row, err := workflowRecoveryStep(g, in.StepDecision)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
node, ok := g.program.Step(row.view.Key)
|
||||||
|
if !ok || node.Task == nil && node.Poll == nil && node.Signal == nil && node.Timer == nil {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
if err := workflowStepSettled(ctx, tx, g, row); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if in.Outcome == "SUCCEEDED" {
|
||||||
|
var typ definition.Type
|
||||||
|
switch {
|
||||||
|
case node.Task != nil:
|
||||||
|
typ = node.Task.ResultType
|
||||||
|
case node.Signal != nil:
|
||||||
|
typ = node.Signal.PayloadType
|
||||||
|
case node.Poll != nil:
|
||||||
|
kind, err := g.program.ClassifyPoll(row.view.Key, in.Result)
|
||||||
|
if err != nil || kind != engine.PollSucceeded {
|
||||||
|
return nil, fault.Invalid("poll recovery requires a confirmed business success result")
|
||||||
|
}
|
||||||
|
typ = node.Poll.Task.ResultType
|
||||||
|
case node.Timer != nil:
|
||||||
|
var activated time.Time
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT created_at FROM weave_activations WHERE namespace_id=? AND id=?`, t.NamespaceID, row.view.ActivationID).Scan(&activated); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if now.Before(activated.Add(time.Duration(node.Timer.DelayMS) * time.Millisecond)) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
typ = definition.Type{Kind: "NULL"}
|
||||||
|
}
|
||||||
|
if err := definition.ValidateData(typ, in.Result); err != nil {
|
||||||
|
return nil, fault.Invalid("review result violates fixed step contract")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
beforeState, beforeOutput := row.view.State, row.output
|
||||||
|
if err := saveWorkflowResultStep(ctx, tx, g, row, weave.StepState(in.Outcome), in.Result, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := deleteWorkflowClock(ctx, tx, g, row); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// 等待、轮次和执行来源保持原样；本次逻辑结论由管理审计与激活结果表示。
|
||||||
|
out.Step = &row.view
|
||||||
|
digest, err := requestDigest(in.Result)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return map[string]any{"step_id": row.view.ID, "activation_id": row.view.ActivationID, "previous_state": beforeState, "previous_output": json.RawMessage(beforeOutput), "outcome": in.Outcome, "result_digest": hex.EncodeToString(digest[:]), "execution_settled": true}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *WorkflowRecoveryStore) Continue(ctx context.Context, t auth.Token, id core.ID, key string, in recovery.ContinueInput) (recovery.Result, error) {
|
||||||
|
return s.mutate(ctx, t, id, key, "workflow.review.continue", in, in.Decision, func(tx *sql.Tx, g *workflowGraph, _ core.ID, now time.Time, _ *recovery.Result) (map[string]any, error) {
|
||||||
|
for key, row := range g.steps {
|
||||||
|
if key == workflowRootKey {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if row.view.State == weave.StepNeedsReview || g.view.StopKind == nil && (row.view.State == weave.StepFailed || row.view.State == weave.StepCancelled) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
}
|
||||||
|
_, uncertain, err := verifyWorkflowStopWork(ctx, tx, g)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if uncertain {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
state := weave.InstanceRunning
|
||||||
|
if g.view.StopKind != nil {
|
||||||
|
state = weave.InstanceCancelling
|
||||||
|
} else if !now.Before(g.view.DeadlineAt) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET state=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND state='NEEDS_REVIEW'`, state, now, t.NamespaceID, id, g.view.Revision)
|
||||||
|
if err := exactlyOne(res, err); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
g.view.State, g.view.Revision = state, g.view.Revision+1
|
||||||
|
if _, err := tx.ExecContext(ctx, `UPDATE pulse_execution_gates SET paused=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=?`, g.view.Paused || g.view.StopKind != nil, now, t.NamespaceID, g.gate); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return map[string]any{"review_resolved": true, "paused": g.view.Paused, "deadline_at": g.view.DeadlineAt}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func sameWorkflowJSON(a, b []byte) bool {
|
||||||
|
left, err := jsonvalue.Canonical(a)
|
||||||
|
if err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
right, err := jsonvalue.Canonical(b)
|
||||||
|
return err == nil && bytes.Equal(left, right)
|
||||||
|
}
|
||||||
507
internal/storage/mysql/workflow_recovery_integration_test.go
Normal file
507
internal/storage/mysql/workflow_recovery_integration_test.go
Normal file
@@ -0,0 +1,507 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
pulse "git.yuntuo.hk/k2safe/orbit/pulse/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
|
)
|
||||||
|
|
||||||
|
func grantWorkflowRecovery(f *executionFixture) {
|
||||||
|
f.exec(`UPDATE orbit_credentials SET permissions=JSON_ARRAY_APPEND(permissions,'$.actions','workflows:recover','$.actions','workflows:control','$.actions','tasks:review') WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, f.resources.CredentialID)
|
||||||
|
}
|
||||||
|
func recoveryDecision(t *testing.T, f *executionFixture, id core.ID) recovery.Decision {
|
||||||
|
t.Helper()
|
||||||
|
v := runtimeView(t, f, id, "NEEDS_REVIEW")
|
||||||
|
return recovery.Decision{ExpectedRevision: v.Revision, Reason: "verified external result", EvidenceRef: "evidence://recovery/verified"}
|
||||||
|
}
|
||||||
|
func recoveryStep(t *testing.T, f *executionFixture, id core.ID, key string) recovery.StepDecision {
|
||||||
|
t.Helper()
|
||||||
|
out := recovery.StepDecision{Decision: recoveryDecision(t, f, id), StepKey: key}
|
||||||
|
if err := f.db.QueryRowContext(f.ctx, "SELECT current_activation_id,revision FROM weave_steps WHERE namespace_id=? AND workflow_id=? AND `key`=?", f.resources.NamespaceID, id, key).Scan(&out.ActivationID, &out.ExpectedStepRevision); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
func reviewWorkflowTask(t *testing.T, f *executionFixture, id, task core.ID, outcome string) recovery.Result {
|
||||||
|
t.Helper()
|
||||||
|
v, err := f.tasks.GetTask(f.ctx, f.token, task)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out, err := recovery.New(NewWorkflowRecoveryStore(f.db)).ReviewTask(f.ctx, f.token, id, string(executionID(t)), recovery.ReviewTaskInput{Decision: recoveryDecision(t, f, id), TaskID: task, ExpectedTaskRevision: v.Revision, ExecutionSettled: true, Outcome: pulse.TaskState(outcome), Result: json.RawMessage(`{"n":9007199254740993}`)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
func resolveWorkflowStep(t *testing.T, f *executionFixture, id core.ID, key, outcome, result string) recovery.Result {
|
||||||
|
t.Helper()
|
||||||
|
out, err := recovery.New(NewWorkflowRecoveryStore(f.db)).ResolveStep(f.ctx, f.token, id, string(executionID(t)), recovery.ResolveStepInput{StepDecision: recoveryStep(t, f, id, key), ExecutionSettled: true, Outcome: outcome, Result: json.RawMessage(result)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
func continueWorkflow(t *testing.T, f *executionFixture, id core.ID) recovery.Result {
|
||||||
|
t.Helper()
|
||||||
|
out, err := recovery.New(NewWorkflowRecoveryStore(f.db)).Continue(f.ctx, f.token, id, string(executionID(t)), recovery.ContinueInput{Decision: recoveryDecision(t, f, id), ReviewResolved: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
func TestIntegrationWorkflowRecovery(t *testing.T) {
|
||||||
|
dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN")
|
||||||
|
if dsn == "" {
|
||||||
|
t.Skip("requires disposable MySQL")
|
||||||
|
}
|
||||||
|
ctx, stop := context.WithTimeout(context.Background(), 150*time.Second)
|
||||||
|
defer stop()
|
||||||
|
db, err := Open(ctx, dsn, 32, 8)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
if err := Migrate(ctx, db); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
api := recovery.New(NewWorkflowRecoveryStore(db))
|
||||||
|
fixture := func(t *testing.T, n int) (*executionFixture, instance.View, core.ID) {
|
||||||
|
f, v := controlFixture(t, ctx, db, n)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "UNKNOWN", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
return f, v, ref.TaskID
|
||||||
|
}
|
||||||
|
t.Run("unknown_review_resolve_continue_preserves_history", func(t *testing.T) {
|
||||||
|
f, v, task := fixture(t, 2)
|
||||||
|
old := recoveryStep(t, f, v.ID, "task0")
|
||||||
|
_, err := api.ResolveStep(ctx, f.token, v.ID, string(executionID(t)), recovery.ResolveStepInput{StepDecision: old, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"n":9007199254740993}`)})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("unresolved execution bypassed", err)
|
||||||
|
}
|
||||||
|
tv, _ := f.tasks.GetTask(ctx, f.token, task)
|
||||||
|
_, err = f.tasks.ReviewTask(ctx, f.token, task, string(executionID(t)), service.ReviewInput{ExpectedRevision: tv.Revision, ExecutionSettled: true, Outcome: "SUCCEEDED", Reason: "cannot bypass workflow", EvidenceRef: "evidence://1", Result: json.RawMessage(`{"n":9007199254740993}`)})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("ordinary review bypassed owner", err)
|
||||||
|
}
|
||||||
|
review := reviewWorkflowTask(t, f, v.ID, task, "SUCCEEDED")
|
||||||
|
if review.Task.EffectUncertain || review.Instance.State != "NEEDS_REVIEW" || review.TaskAuditID == nil || f.count("pulse_attempts", "state='UNKNOWN'") != 1 || f.count("orbit_audit_events", "action IN ('workflow.task.review','task.review')") != 2 {
|
||||||
|
t.Fatal("review lost history or released workflow")
|
||||||
|
}
|
||||||
|
// 新执行结果先可靠消费，再由显式步骤核查决定业务输出。
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), task)
|
||||||
|
resolved := resolveWorkflowStep(t, f, v.ID, "task0", "SUCCEEDED", `{"n":9007199254740993}`)
|
||||||
|
if resolved.Instance.State != "NEEDS_REVIEW" || f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("resolution auto continued")
|
||||||
|
}
|
||||||
|
continued := continueWorkflow(t, f, v.ID)
|
||||||
|
if continued.Instance.State != "RUNNING" || !continued.Instance.DeadlineAt.Equal(v.DeadlineAt) {
|
||||||
|
t.Fatal("continue changed deadline")
|
||||||
|
}
|
||||||
|
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||||
|
if f.count("pulse_attempts", "state='UNKNOWN'") != 1 || f.used() != 0 {
|
||||||
|
t.Fatal("old execution rewritten or permit leaked")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("task_retry_new_activation_remaining_budget_and_stale_result", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||||
|
spec.Body.Steps[0].Task.Retry.MaxAttempts = 2
|
||||||
|
spec.Body.Steps[0].Task.Retry.InitialDelayMS = 1000
|
||||||
|
spec.Body.Steps[0].Task.Retry.MaxDelayMS = 1000
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
oldTask, _ := f.tasks.GetTask(ctx, f.token, ref.TaskID)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "task0"), SafeToRetry: true}
|
||||||
|
key := string(executionID(t))
|
||||||
|
out, err := api.RetryStep(ctx, f.token, v.ID, key, in)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Task.ID == ref.TaskID || out.Task.RetryOf == nil || *out.Task.RetryOf != ref.TaskID || out.Task.Retry.MaxAttempts != oldTask.Retry.MaxAttempts-1 || !out.Task.DeadlineAt.Equal(oldTask.DeadlineAt) || !sameWorkflowJSON(out.Task.Payload, oldTask.Payload) || *out.Step.ActivationID == in.ActivationID {
|
||||||
|
t.Fatal("retry replaced identity or reset budget")
|
||||||
|
}
|
||||||
|
if f.count("weave_activations", "id=? AND state='FAILED'", in.ActivationID) != 1 {
|
||||||
|
t.Fatal("old activation rewritten")
|
||||||
|
}
|
||||||
|
// 重放旧激活的真实事件，只留痕而不能完成新任务。
|
||||||
|
f.exec(`UPDATE weave_inbox SET state='PENDING',processed_at=NULL WHERE namespace_id=? AND source_task_id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
if f.count("weave_inbox", "source_task_id=? AND error_code='STALE_ACTIVATION'", ref.TaskID) != 1 {
|
||||||
|
t.Fatal("old activation not isolated")
|
||||||
|
}
|
||||||
|
replayed, err := api.RetryStep(ctx, f.token, v.ID, key, in)
|
||||||
|
if err != nil || !replayed.Deduplicated || replayed.Task.ID != out.Task.ID {
|
||||||
|
t.Fatal("retry replay changed", err)
|
||||||
|
}
|
||||||
|
oldStep := in
|
||||||
|
oldStep.ExpectedRevision = recoveryDecision(t, f, v.ID).ExpectedRevision
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), oldStep); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("old activation controlled new work", err)
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
_, err = api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "task0"), SafeToRetry: true})
|
||||||
|
if oldTask.Retry.MaxAttempts == 2 && fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("exhausted original budget reset", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("same_review_key_concurrency_and_revoked_replay", func(t *testing.T) {
|
||||||
|
f, v, task := fixture(t, 1)
|
||||||
|
tv, _ := f.tasks.GetTask(ctx, f.token, task)
|
||||||
|
in := recovery.ReviewTaskInput{Decision: recoveryDecision(t, f, v.ID), TaskID: task, ExpectedTaskRevision: tv.Revision, ExecutionSettled: true, Outcome: "FAILED", Result: json.RawMessage(`{"known":true}`)}
|
||||||
|
key := string(executionID(t))
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
errs := make(chan error, 12)
|
||||||
|
fresh := make(chan bool, 12)
|
||||||
|
for i := 0; i < 12; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
v, e := api.ReviewTask(ctx, f.token, v.ID, key, in)
|
||||||
|
errs <- e
|
||||||
|
fresh <- !v.Deduplicated
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
close(errs)
|
||||||
|
close(fresh)
|
||||||
|
for e := range errs {
|
||||||
|
if e != nil {
|
||||||
|
t.Fatal(e)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
n := 0
|
||||||
|
for ok := range fresh {
|
||||||
|
if ok {
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if n != 1 || f.count("orbit_audit_events", "action='workflow.task.review'") != 1 || f.count("weave_inbox", "source_task_id=?", task) != 2 {
|
||||||
|
t.Fatal("duplicate review effects")
|
||||||
|
}
|
||||||
|
f.exec(`UPDATE orbit_credentials SET permissions=JSON_SET(permissions,'$.queues',JSON_ARRAY(?)) WHERE namespace_id=? AND id=?`, string(executionID(t)), f.resources.NamespaceID, f.resources.CredentialID)
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, key, in); fault.Code(err) != "NOT_FOUND" {
|
||||||
|
t.Fatal("cached receipt bypassed queue permission", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("review_result_stream_failure_rolls_back_all", func(t *testing.T) {
|
||||||
|
f, v, task := fixture(t, 1)
|
||||||
|
tv, _ := f.tasks.GetTask(ctx, f.token, task)
|
||||||
|
decision := recoveryDecision(t, f, v.ID)
|
||||||
|
trigger := "reject_workflow_review_" + string(executionID(t))
|
||||||
|
f.exec("CREATE TRIGGER " + trigger + " BEFORE INSERT ON orbit_result_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='controlled review stream failure'")
|
||||||
|
in := recovery.ReviewTaskInput{Decision: decision, TaskID: task, ExpectedTaskRevision: tv.Revision, ExecutionSettled: true, Outcome: "FAILED", Result: json.RawMessage(`{"known":true}`)}
|
||||||
|
key := string(executionID(t))
|
||||||
|
_, err := api.ReviewTask(ctx, f.token, v.ID, key, in)
|
||||||
|
f.exec("DROP TRIGGER " + trigger)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("injected failure accepted")
|
||||||
|
}
|
||||||
|
if f.count("pulse_tasks", "id=? AND state='NEEDS_REVIEW' AND effect_uncertain=TRUE", task) != 1 || f.count("orbit_audit_events", "action IN ('workflow.task.review','task.review')") != 0 || f.count("weave_inbox", "source_task_id=?", task) != 1 || recoveryDecision(t, f, v.ID).ExpectedRevision != decision.ExpectedRevision {
|
||||||
|
t.Fatal("partial review committed")
|
||||||
|
}
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, key, in); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
for _, policy := range []string{"REVIEW", "FAIL_AFTER_SETTLED"} {
|
||||||
|
t.Run("resolve_unknown_with_stop_"+policy, func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 2)
|
||||||
|
spec.FailurePolicy = policy
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "UNKNOWN", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
if policy == "REVIEW" {
|
||||||
|
applyControl(t, f, runtimeView(t, f, v.ID, "NEEDS_REVIEW"), "CANCEL")
|
||||||
|
}
|
||||||
|
drainStop(t, f, v.ID, "NEEDS_REVIEW")
|
||||||
|
before := runtimeView(t, f, v.ID, "NEEDS_REVIEW")
|
||||||
|
reviewWorkflowTask(t, f, v.ID, ref.TaskID, "FAILED")
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
resolveWorkflowStep(t, f, v.ID, "task0", "FAILED", `{"known":true}`)
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
want := "CANCELLED"
|
||||||
|
if policy == "FAIL_AFTER_SETTLED" {
|
||||||
|
want = "FAILED"
|
||||||
|
}
|
||||||
|
drainStop(t, f, v.ID, want)
|
||||||
|
after := runtimeView(t, f, v.ID, want)
|
||||||
|
if *after.StopKind != *before.StopKind || !after.StopRequestedAt.Equal(*before.StopRequestedAt) || f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("recovery changed original stop intent")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
t.Run("poll_review_cannot_treat_pending_as_success", func(t *testing.T) {
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
spec := pollingSpec(f.resources.QueueID)
|
||||||
|
spec.Body.Steps[0].Poll.MaxRounds = 1
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
finishPoll(t, f, ref, "PENDING")
|
||||||
|
in := recovery.ResolveStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`"PENDING"`)}
|
||||||
|
if _, err := api.ResolveStep(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "INVALID_ARGUMENT" {
|
||||||
|
t.Fatal("PENDING became business success", err)
|
||||||
|
}
|
||||||
|
resolveWorkflowStep(t, f, v.ID, "query", "SUCCEEDED", `"DONE"`)
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||||
|
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||||
|
if f.count("weave_poll_states", "used_rounds=1 AND max_rounds=1") != 1 || f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("review issued extra query or reset rounds")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("continue_preserves_pause_and_rejects_stale_evidence", func(t *testing.T) {
|
||||||
|
f, v, task := fixture(t, 2)
|
||||||
|
stale := recoveryDecision(t, f, v.ID)
|
||||||
|
applyControl(t, f, runtimeView(t, f, v.ID, "NEEDS_REVIEW"), "PAUSE")
|
||||||
|
tv := f.get(task)
|
||||||
|
in := recovery.ReviewTaskInput{Decision: stale, TaskID: task, ExpectedTaskRevision: tv.Revision, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"n":9007199254740993}`)}
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("stale evidence accepted", err)
|
||||||
|
}
|
||||||
|
reviewWorkflowTask(t, f, v.ID, task, "SUCCEEDED")
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), task)
|
||||||
|
resolveWorkflowStep(t, f, v.ID, "task0", "SUCCEEDED", `{"n":9007199254740993}`)
|
||||||
|
out := continueWorkflow(t, f, v.ID)
|
||||||
|
if !out.Instance.Paused {
|
||||||
|
t.Fatal("recovery cleared explicit pause")
|
||||||
|
}
|
||||||
|
stopStep(t, f, v.ID)
|
||||||
|
if f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("paused recovery created successor")
|
||||||
|
}
|
||||||
|
applyControl(t, f, runtimeView(t, f, v.ID, "RUNNING"), "RESUME")
|
||||||
|
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||||
|
if f.count("pulse_tasks", "1=1") != 2 {
|
||||||
|
t.Fatal("resume did not continue reviewed output")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("retry_backpressure_and_audit_failure_are_atomic", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||||
|
spec.Body.Steps[0].Task.Retry.MaxAttempts = 2
|
||||||
|
spec.Body.Steps[0].Task.Retry.InitialDelayMS = 1000
|
||||||
|
spec.Body.Steps[0].Task.Retry.MaxDelayMS = 1000
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "task0"), SafeToRetry: true}
|
||||||
|
key := string(executionID(t))
|
||||||
|
f.exec(`UPDATE pulse_queues SET max_unfinished_tasks=1 WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, f.resources.QueueID)
|
||||||
|
blocker := f.submit(1)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, key, in); fault.Code(err) != "QUEUE_FULL" {
|
||||||
|
t.Fatal("retry bypassed queue capacity", err)
|
||||||
|
}
|
||||||
|
if f.count("weave_activations", "step_id=(SELECT id FROM weave_steps WHERE namespace_id=? AND workflow_id=? AND `key`='task0')", f.resources.NamespaceID, v.ID) != 1 {
|
||||||
|
t.Fatal("backpressure left new activation")
|
||||||
|
}
|
||||||
|
_, err := f.tasks.CancelTask(ctx, f.token, blocker.ID, string(executionID(t)), service.CancelInput{ExpectedRevision: blocker.Revision, Reason: "release capacity"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
trigger := "retry_audit_reject_" + string(executionID(t))
|
||||||
|
f.exec("CREATE TRIGGER " + trigger + " BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='controlled workflow retry audit failure'")
|
||||||
|
_, err = api.RetryStep(ctx, f.token, v.ID, key, in)
|
||||||
|
f.exec("DROP TRIGGER " + trigger)
|
||||||
|
if err == nil || f.count("pulse_tasks", "retry_of=?", ref.TaskID) != 0 {
|
||||||
|
t.Fatal("retry audit failure left partial task", err)
|
||||||
|
}
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, key, in); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("task_deadline_is_not_extended_by_manual_retry", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||||
|
spec.TimeoutMS = 4000
|
||||||
|
task := spec.Body.Steps[0].Task
|
||||||
|
task.TimeoutMS = 700
|
||||||
|
task.Retry.AttemptTimeoutMS = 200
|
||||||
|
task.Retry.MaxAttempts = 2
|
||||||
|
task.Retry.InitialDelayMS = 100
|
||||||
|
task.Retry.MaxDelayMS = 100
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
time.Sleep(750 * time.Millisecond)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "task0"), SafeToRetry: true}); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("old task deadline extended", err)
|
||||||
|
}
|
||||||
|
if f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("expired task produced retry")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("pending_and_quarantined_evidence_prevent_recovery", func(t *testing.T) {
|
||||||
|
f, v, task := fixture(t, 1)
|
||||||
|
tv := f.get(task)
|
||||||
|
in := recovery.ReviewTaskInput{Decision: recoveryDecision(t, f, v.ID), TaskID: task, ExpectedTaskRevision: tv.Revision, ExecutionSettled: true, Outcome: "FAILED", Result: json.RawMessage(`{"known":true}`)}
|
||||||
|
for _, state := range []string{"PENDING", "QUARANTINED"} {
|
||||||
|
f.exec(`UPDATE weave_inbox SET state=? WHERE namespace_id=? AND source_task_id=?`, state, f.resources.NamespaceID, task)
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("unprocessed evidence bypassed", state, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
f.exec(`UPDATE weave_inbox SET state='PROCESSED' WHERE namespace_id=? AND source_task_id=?`, f.resources.NamespaceID, task)
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, string(executionID(t)), in); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("wrong_task_scope_version_and_output_cannot_resolve", func(t *testing.T) {
|
||||||
|
f, v, task := fixture(t, 1)
|
||||||
|
other, _, foreign := fixture(t, 1)
|
||||||
|
tv := f.get(task)
|
||||||
|
in := recovery.ReviewTaskInput{Decision: recoveryDecision(t, f, v.ID), TaskID: foreign, ExpectedTaskRevision: other.get(foreign).Revision, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"n":9007199254740993}`)}
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "NOT_FOUND" {
|
||||||
|
t.Fatal("foreign task scope accepted", err)
|
||||||
|
}
|
||||||
|
in.TaskID = task
|
||||||
|
in.ExpectedTaskRevision = tv.Revision - 1
|
||||||
|
if _, err := api.ReviewTask(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("old execution version accepted", err)
|
||||||
|
}
|
||||||
|
reviewWorkflowTask(t, f, v.ID, task, "SUCCEEDED")
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), task)
|
||||||
|
step := recoveryStep(t, f, v.ID, "task0")
|
||||||
|
if _, err := api.ResolveStep(ctx, f.token, v.ID, string(executionID(t)), recovery.ResolveStepInput{StepDecision: step, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"unexpected":true}`)}); fault.Code(err) != "INVALID_ARGUMENT" {
|
||||||
|
t.Fatal("invalid fixed output accepted", err)
|
||||||
|
}
|
||||||
|
if _, err := api.Continue(ctx, f.token, v.ID, string(executionID(t)), recovery.ContinueInput{Decision: step.Decision, ReviewResolved: true}); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("unresolved step released", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("expired_instance_can_settle_but_cannot_restart", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 2)
|
||||||
|
spec.TimeoutMS = 800
|
||||||
|
for _, s := range spec.Body.Steps {
|
||||||
|
s.Task.TimeoutMS = 500
|
||||||
|
s.Task.Retry.AttemptTimeoutMS = 200
|
||||||
|
}
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
resolveWorkflowStep(t, f, v.ID, "task0", "SUCCEEDED", `{"n":9007199254740993}`)
|
||||||
|
time.Sleep(850 * time.Millisecond)
|
||||||
|
if _, err := api.Continue(ctx, f.token, v.ID, string(executionID(t)), recovery.ContinueInput{Decision: recoveryDecision(t, f, v.ID), ReviewResolved: true}); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("expired instance restarted", err)
|
||||||
|
}
|
||||||
|
applyControl(t, f, runtimeView(t, f, v.ID, "NEEDS_REVIEW"), "CANCEL")
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
drainStop(t, f, v.ID, "CANCELLED")
|
||||||
|
if f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("expired recovery issued successor")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("retry_rejects_corrupt_frozen_input_and_initial_deadline", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||||
|
spec.Body.Steps[0].Task.Retry.MaxAttempts = 2
|
||||||
|
spec.Body.Steps[0].Task.Retry.InitialDelayMS = 1000
|
||||||
|
spec.Body.Steps[0].Task.Retry.MaxDelayMS = 1000
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "task0"), SafeToRetry: true}
|
||||||
|
old := f.get(ref.TaskID)
|
||||||
|
f.exec(`UPDATE pulse_tasks SET payload='{"wrong":true}' WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||||
|
f.exec(`UPDATE weave_activations SET input='{"wrong":true}' WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, in.ActivationID)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), in); err == nil {
|
||||||
|
t.Fatal("corrupt matching inputs bypassed fixed contract")
|
||||||
|
}
|
||||||
|
f.exec(`UPDATE pulse_tasks SET payload=? WHERE namespace_id=? AND id=?`, string(old.Payload), f.resources.NamespaceID, ref.TaskID)
|
||||||
|
f.exec(`UPDATE weave_activations SET input=? WHERE namespace_id=? AND id=?`, string(old.Payload), f.resources.NamespaceID, in.ActivationID)
|
||||||
|
f.exec(`UPDATE pulse_tasks SET deadline_at=DATE_ADD(deadline_at,INTERVAL 1 SECOND) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), in); err == nil {
|
||||||
|
t.Fatal("mutated initial deadline accepted")
|
||||||
|
}
|
||||||
|
f.exec(`UPDATE pulse_tasks SET deadline_at=? WHERE namespace_id=? AND id=?`, old.DeadlineAt, f.resources.NamespaceID, ref.TaskID)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), in); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("signal_deadline_review_preserves_expired_wait", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := signalSpec(f.resources.QueueID, false)
|
||||||
|
spec.Body.Steps[0].Signal.TimeoutMS = 100
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
time.Sleep(150 * time.Millisecond)
|
||||||
|
runClock(t, f, clockCandidate(t, f, v.ID, "signal"))
|
||||||
|
resolveWorkflowStep(t, f, v.ID, "signal", "SUCCEEDED", `{"n":9007199254740993}`)
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||||
|
if f.count("weave_waits", "workflow_id=? AND state='EXPIRED'", v.ID) != 1 || f.count("weave_signals", "1=1") != 0 {
|
||||||
|
t.Fatal("manual resolution fabricated signal receipt")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("timer_review_cannot_fulfil_original_delay_early", func(t *testing.T) {
|
||||||
|
f := newExecutionFixture(t, ctx, db)
|
||||||
|
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||||
|
spec.TimeoutMS = 1500
|
||||||
|
spec.Body.Steps[0].Task.TimeoutMS = 1000
|
||||||
|
spec.Body.Steps[0].Task.Retry.AttemptTimeoutMS = 800
|
||||||
|
spec.Body.Steps = append(spec.Body.Steps, definition.Step{Key: "timer", Kind: "TIMER", Timer: &definition.Timer{DelayMS: 1500}})
|
||||||
|
spec.Body.OutputType = definition.Type{Kind: "NULL"}
|
||||||
|
spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "timer"}}
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
time.Sleep(400 * time.Millisecond)
|
||||||
|
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
c := clockCandidate(t, f, v.ID, "timer")
|
||||||
|
waitClockDue(t, f, c)
|
||||||
|
runClock(t, f, c)
|
||||||
|
in := recovery.ResolveStepInput{StepDecision: recoveryStep(t, f, v.ID, "timer"), ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`null`)}
|
||||||
|
if _, err := api.ResolveStep(ctx, f.token, v.ID, string(executionID(t)), in); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("manual review shortened original timer", err)
|
||||||
|
}
|
||||||
|
time.Sleep(500 * time.Millisecond)
|
||||||
|
resolveWorkflowStep(t, f, v.ID, "timer", "SUCCEEDED", `null`)
|
||||||
|
if _, err := api.Continue(ctx, f.token, v.ID, string(executionID(t)), recovery.ContinueInput{Decision: recoveryDecision(t, f, v.ID), ReviewResolved: true}); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("elapsed timer extended workflow deadline", err)
|
||||||
|
}
|
||||||
|
applyControl(t, f, runtimeView(t, f, v.ID, "NEEDS_REVIEW"), "CANCEL")
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
drainStop(t, f, v.ID, "CANCELLED")
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -11,6 +11,7 @@ import (
|
|||||||
|
|
||||||
"git.yuntuo.hk/k2safe/orbit/pulse/worker"
|
"git.yuntuo.hk/k2safe/orbit/pulse/worker"
|
||||||
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
mysqldriver "github.com/go-sql-driver/mysql"
|
mysqldriver "github.com/go-sql-driver/mysql"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -311,4 +312,67 @@ func TestIntegrationWorkflowRuntimeProcess(t *testing.T) {
|
|||||||
t.Fatal("unknown cancel leaked permit or started tail")
|
t.Fatal("unknown cancel leaked permit or started tail")
|
||||||
}
|
}
|
||||||
t.Log("parallel failure intent and running sibling survived another all-stop, settled FAILED only after result collection; user cancellation plus UNKNOWN recovered NEEDS_REVIEW with no successor or false terminal state")
|
t.Log("parallel failure intent and running sibling survived another all-stop, settled FAILED only after result collection; user cancellation plus UNKNOWN recovered NEEDS_REVIEW with no successor or false terminal state")
|
||||||
|
for _, p := range processes {
|
||||||
|
stop(p)
|
||||||
|
}
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
reviewWorkflowTask(t, f, cancelled.ID, ref.TaskID, "FAILED")
|
||||||
|
start("review-result-a")
|
||||||
|
start("review-result-b")
|
||||||
|
wait(func() bool { return f.count("weave_inbox", "workflow_id=? AND state='PENDING'", cancelled.ID) == 0 }, "manual review result consumption")
|
||||||
|
for _, p := range processes {
|
||||||
|
stop(p)
|
||||||
|
}
|
||||||
|
resolveWorkflowStep(t, f, cancelled.ID, "task0", "FAILED", `{"verified":true}`)
|
||||||
|
continueWorkflow(t, f, cancelled.ID)
|
||||||
|
start("review-stop-a")
|
||||||
|
start("review-stop-b")
|
||||||
|
wait(func() bool {
|
||||||
|
current, e := api.Get(ctx, f.token, cancelled.ID)
|
||||||
|
return e == nil && current.State == "CANCELLED"
|
||||||
|
}, "reviewed cancellation resumes original stop after all-stop")
|
||||||
|
for _, p := range processes {
|
||||||
|
stop(p)
|
||||||
|
}
|
||||||
|
retrySpec := runtimeSpec(f.resources.QueueID, 1)
|
||||||
|
retrySpec.Body.Steps[0].Task.Retry.MaxAttempts = 2
|
||||||
|
retrySpec.Body.Steps[0].Task.Retry.InitialDelayMS = 1000
|
||||||
|
retrySpec.Body.Steps[0].Task.Retry.MaxDelayMS = 1000
|
||||||
|
retried := runtimeStart(t, f, retrySpec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
start("retry-review-a")
|
||||||
|
start("retry-review-b")
|
||||||
|
wait(func() bool {
|
||||||
|
current, e := api.Get(ctx, f.token, retried.ID)
|
||||||
|
return e == nil && current.State == "NEEDS_REVIEW"
|
||||||
|
}, "failed step enters manual review")
|
||||||
|
for _, p := range processes {
|
||||||
|
stop(p)
|
||||||
|
}
|
||||||
|
old := f.get(ref.TaskID)
|
||||||
|
retriedStep, err := recovery.New(NewWorkflowRecoveryStore(db)).RetryStep(ctx, f.token, retried.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, retried.ID, "task0"), SafeToRetry: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, retried.ID)
|
||||||
|
if retriedStep.Task.Retry.MaxAttempts != 1 || !retriedStep.Task.DeadlineAt.Equal(old.DeadlineAt) {
|
||||||
|
t.Fatal("manual retry reset fixed budget")
|
||||||
|
}
|
||||||
|
start("manual-retry-a")
|
||||||
|
start("manual-retry-b")
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
if ref.TaskID != retriedStep.Task.ID {
|
||||||
|
t.Fatal("restarted worker received wrong activation")
|
||||||
|
}
|
||||||
|
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||||
|
wait(func() bool {
|
||||||
|
current, e := api.Get(ctx, f.token, retried.ID)
|
||||||
|
return e == nil && current.State == "SUCCEEDED"
|
||||||
|
}, "new manual activation completed after restart")
|
||||||
|
if f.count("pulse_tasks", "1=1") != 13 || f.used() != 0 || f.count("weave_activations", "workflow_id=? AND state='FAILED'", retried.ID) != 1 {
|
||||||
|
t.Fatal("manual recovery duplicated work or rewrote old activation")
|
||||||
|
}
|
||||||
|
t.Log("manual execution review, explicit workflow release and new TASK activation survived all-stop and two fresh controllers; original cancellation, task deadline and remaining attempt budget preserved")
|
||||||
}
|
}
|
||||||
|
|||||||
157
internal/storage/mysql/workflow_step_retry.go
Normal file
157
internal/storage/mysql/workflow_step_retry.go
Normal file
@@ -0,0 +1,157 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||||
|
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RetryStep 为TASK建立新的执行激活；保留旧激活、原输入和绝对截止，跨激活扣除已用尝试。
|
||||||
|
// POLL恢复需要同时保留原轮次身份，不在此处降级成新查询轮次。
|
||||||
|
func (s *WorkflowRecoveryStore) RetryStep(ctx context.Context, t auth.Token, id core.ID, key string, in recovery.RetryStepInput) (recovery.Result, error) {
|
||||||
|
return s.mutate(ctx, t, id, key, "workflow.step.retry", in, in.Decision, func(tx *sql.Tx, g *workflowGraph, _ core.ID, now time.Time, out *recovery.Result) (map[string]any, error) {
|
||||||
|
row, err := workflowRecoveryStep(g, in.StepDecision)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if g.view.StopKind != nil || g.view.CancelRequestedAt != nil || !now.Before(g.view.DeadlineAt) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
node, ok := g.program.Step(row.view.Key)
|
||||||
|
if !ok || node.Task == nil {
|
||||||
|
return nil, fault.Invalid("this retry operation currently supports TASK steps only")
|
||||||
|
}
|
||||||
|
if err := workflowStepSettled(ctx, tx, g, row); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
var sequence uint32
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT sequence FROM weave_activations WHERE namespace_id=? AND id=?`, t.NamespaceID, row.view.ActivationID).Scan(&sequence); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// 人工取消尚未开始的任务也不能无限创建激活；每个静态TASK的激活数不超过原max_attempts。
|
||||||
|
if sequence == 0 || sequence >= node.Task.Retry.MaxAttempts {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
history, err := tx.QueryContext(ctx, `SELECT id,sequence,task_id FROM weave_activations WHERE namespace_id=? AND workflow_id=? AND step_id=? ORDER BY sequence LIMIT 101`, t.NamespaceID, id, row.view.ID)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
type prior struct {
|
||||||
|
activation, task core.ID
|
||||||
|
sequence uint32
|
||||||
|
}
|
||||||
|
var previous []prior
|
||||||
|
for history.Next() {
|
||||||
|
var v prior
|
||||||
|
if err := history.Scan(&v.activation, &v.sequence, &v.task); err != nil {
|
||||||
|
history.Close()
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
previous = append(previous, v)
|
||||||
|
}
|
||||||
|
err = history.Err()
|
||||||
|
history.Close()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(previous) != int(sequence) || previous[len(previous)-1].activation != in.ActivationID {
|
||||||
|
return nil, errors.New("workflow retry activation history incomplete")
|
||||||
|
}
|
||||||
|
var first, last service.TaskView
|
||||||
|
var used uint32
|
||||||
|
for i, prior := range previous {
|
||||||
|
if prior.sequence != uint32(i+1) {
|
||||||
|
return nil, errors.New("workflow retry sequence invalid")
|
||||||
|
}
|
||||||
|
v, err := readTask(ctx, tx, t.NamespaceID, prior.task, false)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if (v.State != "FAILED" && v.State != "CANCELLED") || v.EffectUncertain {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
if i == 0 {
|
||||||
|
first = v
|
||||||
|
if definition.ValidateData(node.Task.InputType, first.Payload) != nil || !first.DeadlineAt.Equal(minimumTime(first.CreatedAt.Add(time.Duration(node.Task.TimeoutMS)*time.Millisecond), g.view.DeadlineAt)) {
|
||||||
|
return nil, errors.New("workflow retry initial contract or deadline mismatch")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expectedPolicy := node.Task.Retry
|
||||||
|
if used >= expectedPolicy.MaxAttempts {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
expectedPolicy.MaxAttempts -= used
|
||||||
|
if v.Retry != expectedPolicy || v.AttemptCount > v.Retry.MaxAttempts || v.QueueID != node.Task.QueueID || v.Handler != node.Task.Handler || v.HandlerVersion != node.Task.HandlerVersion || v.PayloadVersion != node.Task.PayloadVersion || v.RecoverySafety != node.Task.RecoverySafety || v.Priority != node.Task.Priority || !sameWorkflowJSON(v.Payload, row.input) || !v.DeadlineAt.Equal(first.DeadlineAt) || v.BusinessKey != first.BusinessKey || v.ConcurrencyKey != first.ConcurrencyKey {
|
||||||
|
return nil, errors.New("workflow retry frozen task history mismatch")
|
||||||
|
}
|
||||||
|
if i == 0 && v.RetryOf != nil || i > 0 && (v.RetryOf == nil || *v.RetryOf != last.ID) {
|
||||||
|
return nil, errors.New("workflow retry task ancestry mismatch")
|
||||||
|
}
|
||||||
|
var attempts uint32
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM pulse_attempts WHERE namespace_id=? AND task_id=?`, t.NamespaceID, v.ID).Scan(&attempts); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if attempts != v.AttemptCount {
|
||||||
|
return nil, errors.New("workflow retry attempt history incomplete")
|
||||||
|
}
|
||||||
|
used += v.AttemptCount
|
||||||
|
last = v
|
||||||
|
}
|
||||||
|
if used >= node.Task.Retry.MaxAttempts || !now.Before(first.DeadlineAt) || first.StartDeadlineAt != nil && !now.Before(*first.StartDeadlineAt) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
policy := node.Task.Retry
|
||||||
|
policy.MaxAttempts -= used
|
||||||
|
input, err := service.NormalizeSubmit(service.SubmitInput{QueueID: first.QueueID, Handler: first.Handler, HandlerVersion: first.HandlerVersion, PayloadVersion: first.PayloadVersion, Payload: first.Payload, BusinessKey: first.BusinessKey, ConcurrencyKey: first.ConcurrencyKey, Priority: first.Priority, RecoverySafety: first.RecoverySafety, Retry: policy, TaskTimeoutMS: node.Task.TimeoutMS, StartDeadlineAt: first.StartDeadlineAt})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
activation, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
taskID, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
task, err := insertAcceptedTaskUntil(ctx, tx, t.NamespaceID, taskID, input, &last.ID, &first.DeadlineAt)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if _, err := tx.ExecContext(ctx, `INSERT INTO weave_activations(namespace_id,id,workflow_id,step_id,sequence,state,task_id,input,created_at,updated_at) VALUES (?,?,?,?,?,'ACTIVE',?,?,?,?)`, t.NamespaceID, activation, id, row.view.ID, sequence+1, taskID, string(row.input), now, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
res, err := tx.ExecContext(ctx, `UPDATE pulse_tasks SET owner_kind=?,owner_id=?,gate_id=? WHERE namespace_id=? AND id=? AND owner_kind IS NULL`, engine.TaskOwnerKind, activation, g.gate, t.NamespaceID, taskID)
|
||||||
|
if err := exactlyOne(res, err); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
res, err = tx.ExecContext(ctx, `UPDATE weave_steps SET state='ACTIVE',current_activation_id=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND current_activation_id=?`, activation, now, t.NamespaceID, row.view.ID, row.view.Revision, row.view.ActivationID)
|
||||||
|
if err := exactlyOne(res, err); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
row.view.State, row.view.ActivationID, row.view.TaskID = weave.StepActive, &activation, &taskID
|
||||||
|
row.view.Revision++
|
||||||
|
row.output = nil
|
||||||
|
if err := deleteWorkflowClock(ctx, tx, g, row); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := emitEvent(ctx, tx, t.NamespaceID, taskID, task.Revision, "task", "task.accepted", map[string]any{"task_id": taskID, "state": task.State, "workflow_id": id, "activation_id": activation, "retry_of": last.ID}); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := emitEvent(ctx, tx, t.NamespaceID, row.view.ID, row.view.Revision, "workflow_step", "workflow.step.changed", map[string]any{"workflow_id": id, "activation_id": activation, "state": row.view.State}); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out.Step, out.Task = &row.view, &task
|
||||||
|
return map[string]any{"step_id": row.view.ID, "previous_activation_id": in.ActivationID, "activation_id": activation, "sequence": sequence + 1, "retry_of": last.ID, "task_id": taskID, "safe_to_retry": true, "used_attempts": used, "remaining_attempts": policy.MaxAttempts, "deadline_at": first.DeadlineAt}, nil
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -58,7 +58,7 @@ type Permissions struct {
|
|||||||
Queues []string `json:"queues"`
|
Queues []string `json:"queues"`
|
||||||
}
|
}
|
||||||
|
|
||||||
var supported = map[string]bool{"workflows:signal": true, "workflows:control": true, "workflows:start": true, "workflows:read": true, "workflows:definitions:read": true, "workflows:definitions:publish": true, "tasks:review": true, "tasks:retry": true, "tasks:audit:read": true, "events:read": true, "events:manage": true, "events:consume": true, "application:resources:read": true, "application:resources:manage": true, "tasks:submit": true, "tasks:read": true, "tasks:cancel": true, "queues:read": true, "queues:manage": true, "workers:connect": true, "workers:consume": true, "schedules:read": true, "schedules:manage": true}
|
var supported = map[string]bool{"workflows:recover": true, "workflows:signal": true, "workflows:control": true, "workflows:start": true, "workflows:read": true, "workflows:definitions:read": true, "workflows:definitions:publish": true, "tasks:review": true, "tasks:retry": true, "tasks:audit:read": true, "events:read": true, "events:manage": true, "events:consume": true, "application:resources:read": true, "application:resources:manage": true, "tasks:submit": true, "tasks:read": true, "tasks:cancel": true, "queues:read": true, "queues:manage": true, "workers:connect": true, "workers:consume": true, "schedules:read": true, "schedules:manage": true}
|
||||||
|
|
||||||
func (p Permissions) Validate() error {
|
func (p Permissions) Validate() error {
|
||||||
if len(p.Actions) == 0 || len(p.Actions) > 32 || len(p.Queues) == 0 || len(p.Queues) > 128 {
|
if len(p.Actions) == 0 || len(p.Actions) > 32 || len(p.Queues) == 0 || len(p.Queues) > 128 {
|
||||||
|
|||||||
@@ -6,7 +6,7 @@ import (
|
|||||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
)
|
)
|
||||||
|
|
||||||
// Activation 表示步骤的一次具体激活，人工恢复通过新激活保留历史。
|
// Activation 表示步骤的一次具体激活；人工重试创建新激活，结果核查通过状态事件和审计保留前后结论。
|
||||||
// 对应表 weave_activations；这是服务端持久化模型，不直接作为 API 响应。
|
// 对应表 weave_activations；这是服务端持久化模型，不直接作为 API 响应。
|
||||||
type Activation struct {
|
type Activation struct {
|
||||||
// Record 包含命名空间、记录标识、并发版本和 UTC 时间。
|
// Record 包含命名空间、记录标识、并发版本和 UTC 时间。
|
||||||
|
|||||||
162
weave/recovery/service.go
Normal file
162
weave/recovery/service.go
Normal file
@@ -0,0 +1,162 @@
|
|||||||
|
// Package recovery 定义有证据的人工核查与接续；不通过暂停开关清除未知执行或恢复预算。
|
||||||
|
package recovery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"unicode"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||||
|
pulse "git.yuntuo.hk/k2safe/orbit/pulse/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Action = "workflows:recover"
|
||||||
|
|
||||||
|
// Decision 核查基于整个实例的最新证据版本；后台新事实会使旧决定失效，原键仍可恢复回执。
|
||||||
|
type Decision struct {
|
||||||
|
ExpectedRevision uint64 `json:"expected_revision,string"` // ExpectedRevision 当前实例版本，不能使用control_revision代替。
|
||||||
|
Reason string `json:"reason"` // Reason 必填核查原因，最多1024个Unicode字符。
|
||||||
|
EvidenceRef string `json:"evidence_ref"` // EvidenceRef 受控证据引用，最多512个无空白或控制字符的Unicode字符；平台不请求地址。
|
||||||
|
}
|
||||||
|
|
||||||
|
type ReviewTaskInput struct {
|
||||||
|
Decision
|
||||||
|
TaskID core.ID `json:"task_id"` // TaskID 本实例任一历史激活拥有的待核查Pulse任务。
|
||||||
|
ExpectedTaskRevision uint64 `json:"expected_task_revision,string"` // ExpectedTaskRevision 任务当前状态版本。
|
||||||
|
ExecutionSettled bool `json:"execution_settled" required:"true"` // ExecutionSettled 显式确认该任务全部历史外部执行已结束且不会继续产生效果。
|
||||||
|
Outcome pulse.TaskState `json:"outcome"` // Outcome SUCCEEDED或FAILED，表示该次请求的核查结论，不代替POLL业务结果。
|
||||||
|
Result json.RawMessage `json:"result"` // Result 非null的核查结果，最多256 KiB，保留精确数字。
|
||||||
|
}
|
||||||
|
|
||||||
|
type StepDecision struct {
|
||||||
|
Decision
|
||||||
|
StepKey string `json:"step_key"` // StepKey 固定定义中的当前步骤，不接受内部根步骤。
|
||||||
|
ActivationID core.ID `json:"activation_id"` // ActivationID 调用者核查的当前激活，禁止旧激活作用于新工作。
|
||||||
|
ExpectedStepRevision uint64 `json:"expected_step_revision,string"` // ExpectedStepRevision 步骤版本，与实例版本共同校验。
|
||||||
|
}
|
||||||
|
|
||||||
|
type ResolveStepInput struct {
|
||||||
|
StepDecision
|
||||||
|
ExecutionSettled bool `json:"execution_settled" required:"true"` // ExecutionSettled 确认本步骤关联外部执行与业务结果已核实。
|
||||||
|
Outcome string `json:"outcome"` // Outcome SUCCEEDED或FAILED；成功必须符合固定节点输出契约。
|
||||||
|
Result json.RawMessage `json:"result" nullable:"true"` // Result 核查后的逻辑输出；TIMER成功输出为显式null，失败结论保留在审计摘要。
|
||||||
|
}
|
||||||
|
|
||||||
|
type ContinueInput struct {
|
||||||
|
Decision
|
||||||
|
ReviewResolved bool `json:"review_resolved" required:"true"` // ReviewResolved 确认全部核查事项已处理；服务端仍检查未决步骤、历史任务与Inbox。
|
||||||
|
}
|
||||||
|
|
||||||
|
type RetryStepInput struct {
|
||||||
|
StepDecision
|
||||||
|
SafeToRetry bool `json:"safe_to_retry" required:"true"` // SafeToRetry 确认旧执行已结束，并按原业务身份再次执行安全；不能只依据未收到响应。
|
||||||
|
}
|
||||||
|
|
||||||
|
type Result struct {
|
||||||
|
Instance instance.View `json:"instance"` // Instance 操作提交时的冻结实例回执；核查步骤不自动解除实例核查。
|
||||||
|
AuditID core.ID `json:"audit_id"` // AuditID 本次流程管理操作的不可变审计标识。
|
||||||
|
Step *instance.StepView `json:"step,omitempty"` // Step 被核查或重新激活步骤的回执。
|
||||||
|
Task *service.TaskView `json:"task,omitempty"` // Task 被核查或新创建的执行任务。
|
||||||
|
TaskAuditID *core.ID `json:"task_audit_id,omitempty"` // TaskAuditID 独立的执行结果核查审计，和流程审计同事务。
|
||||||
|
Deduplicated bool `json:"deduplicated"` // Deduplicated 原操作回执恢复，不再次修改状态或预算。
|
||||||
|
}
|
||||||
|
|
||||||
|
type Store interface {
|
||||||
|
ReviewTask(context.Context, auth.Token, core.ID, string, ReviewTaskInput) (Result, error)
|
||||||
|
ResolveStep(context.Context, auth.Token, core.ID, string, ResolveStepInput) (Result, error)
|
||||||
|
Continue(context.Context, auth.Token, core.ID, string, ContinueInput) (Result, error)
|
||||||
|
RetryStep(context.Context, auth.Token, core.ID, string, RetryStepInput) (Result, error)
|
||||||
|
}
|
||||||
|
type Service struct{ store Store }
|
||||||
|
|
||||||
|
func New(store Store) *Service { return &Service{store: store} }
|
||||||
|
|
||||||
|
func validateDecision(id core.ID, key string, in Decision) error {
|
||||||
|
if id.Validate() != nil || in.ExpectedRevision == 0 {
|
||||||
|
return fault.Invalid("workflow id and positive expected_revision required")
|
||||||
|
}
|
||||||
|
if err := service.ValidateKey(key); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !utf8.ValidString(in.Reason) || strings.TrimSpace(in.Reason) == "" || utf8.RuneCountInString(in.Reason) > 1024 || strings.ContainsRune(in.Reason, 0) {
|
||||||
|
return fault.Invalid("recovery reason required, maximum 1024 characters")
|
||||||
|
}
|
||||||
|
if !utf8.ValidString(in.EvidenceRef) || len(in.EvidenceRef) == 0 || utf8.RuneCountInString(in.EvidenceRef) > 512 {
|
||||||
|
return fault.Invalid("valid evidence_ref required, maximum 512 characters")
|
||||||
|
}
|
||||||
|
for _, r := range in.EvidenceRef {
|
||||||
|
if unicode.IsSpace(r) || unicode.IsControl(r) {
|
||||||
|
return fault.Invalid("evidence_ref cannot contain whitespace or control characters")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func validateStep(in StepDecision) error {
|
||||||
|
if in.ActivationID.Validate() != nil || in.ExpectedStepRevision == 0 || len(in.StepKey) == 0 || len(in.StepKey) > 64 {
|
||||||
|
return fault.Invalid("current step identity and revision required")
|
||||||
|
}
|
||||||
|
for i, r := range in.StepKey {
|
||||||
|
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || i > 0 && (r == '_' || r == '.' || r == '-')) {
|
||||||
|
return fault.Invalid("invalid step key")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
func validateResult(raw json.RawMessage) error {
|
||||||
|
return definition.ValidateData(definition.Type{Kind: "JSON"}, raw)
|
||||||
|
}
|
||||||
|
func (s *Service) ReviewTask(ctx context.Context, t auth.Token, id core.ID, key string, in ReviewTaskInput) (Result, error) {
|
||||||
|
if err := validateDecision(id, key, in.Decision); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if in.TaskID.Validate() != nil || in.ExpectedTaskRevision == 0 || !in.ExecutionSettled || in.Outcome != pulse.TaskSucceeded && in.Outcome != pulse.TaskFailed || strings.TrimSpace(string(in.Result)) == "null" {
|
||||||
|
return Result{}, fault.Invalid("task identity, execution_settled and known outcome required")
|
||||||
|
}
|
||||||
|
if err := validateResult(in.Result); err != nil {
|
||||||
|
return Result{}, fault.Invalid("valid non-null review result required, maximum 256 KiB")
|
||||||
|
}
|
||||||
|
return s.store.ReviewTask(ctx, t, id, key, in)
|
||||||
|
}
|
||||||
|
func (s *Service) ResolveStep(ctx context.Context, t auth.Token, id core.ID, key string, in ResolveStepInput) (Result, error) {
|
||||||
|
if err := validateDecision(id, key, in.Decision); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if err := validateStep(in.StepDecision); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if !in.ExecutionSettled || in.Outcome != "SUCCEEDED" && in.Outcome != "FAILED" {
|
||||||
|
return Result{}, fault.Invalid("execution_settled and known step outcome required")
|
||||||
|
}
|
||||||
|
if err := validateResult(in.Result); err != nil {
|
||||||
|
return Result{}, fault.Invalid("valid review result required, maximum 256 KiB")
|
||||||
|
}
|
||||||
|
return s.store.ResolveStep(ctx, t, id, key, in)
|
||||||
|
}
|
||||||
|
func (s *Service) Continue(ctx context.Context, t auth.Token, id core.ID, key string, in ContinueInput) (Result, error) {
|
||||||
|
if err := validateDecision(id, key, in.Decision); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if !in.ReviewResolved {
|
||||||
|
return Result{}, fault.Invalid("review_resolved=true required")
|
||||||
|
}
|
||||||
|
return s.store.Continue(ctx, t, id, key, in)
|
||||||
|
}
|
||||||
|
func (s *Service) RetryStep(ctx context.Context, t auth.Token, id core.ID, key string, in RetryStepInput) (Result, error) {
|
||||||
|
if err := validateDecision(id, key, in.Decision); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if err := validateStep(in.StepDecision); err != nil {
|
||||||
|
return Result{}, err
|
||||||
|
}
|
||||||
|
if !in.SafeToRetry {
|
||||||
|
return Result{}, fault.Invalid("safe_to_retry=true required")
|
||||||
|
}
|
||||||
|
return s.store.RetryStep(ctx, t, id, key, in)
|
||||||
|
}
|
||||||
77
weave/recovery/service_test.go
Normal file
77
weave/recovery/service_test.go
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
package recovery
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/platform/auth"
|
||||||
|
)
|
||||||
|
|
||||||
|
type captureStore struct{ calls int }
|
||||||
|
|
||||||
|
func (s *captureStore) ReviewTask(context.Context, auth.Token, core.ID, string, ReviewTaskInput) (Result, error) {
|
||||||
|
s.calls++
|
||||||
|
return Result{}, nil
|
||||||
|
}
|
||||||
|
func (s *captureStore) ResolveStep(context.Context, auth.Token, core.ID, string, ResolveStepInput) (Result, error) {
|
||||||
|
s.calls++
|
||||||
|
return Result{}, nil
|
||||||
|
}
|
||||||
|
func (s *captureStore) Continue(context.Context, auth.Token, core.ID, string, ContinueInput) (Result, error) {
|
||||||
|
s.calls++
|
||||||
|
return Result{}, nil
|
||||||
|
}
|
||||||
|
func (s *captureStore) RetryStep(context.Context, auth.Token, core.ID, string, RetryStepInput) (Result, error) {
|
||||||
|
s.calls++
|
||||||
|
return Result{}, nil
|
||||||
|
}
|
||||||
|
func TestRecoveryValidation(t *testing.T) {
|
||||||
|
id := core.ID(strings.Repeat("a", 32))
|
||||||
|
d := Decision{ExpectedRevision: 2, Reason: "核实外部结果", EvidenceRef: "evidence://case/1"}
|
||||||
|
step := StepDecision{Decision: d, StepKey: "stage-1.v2", ActivationID: id, ExpectedStepRevision: 3}
|
||||||
|
store := &captureStore{}
|
||||||
|
s := New(store)
|
||||||
|
ctx := context.Background()
|
||||||
|
token := auth.Token{}
|
||||||
|
if _, err := s.ReviewTask(ctx, token, id, "operation", ReviewTaskInput{Decision: d, TaskID: id, ExpectedTaskRevision: 4, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"n":9007199254740993}`)}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.ResolveStep(ctx, token, id, "operation", ResolveStepInput{StepDecision: step, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`null`)}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.Continue(ctx, token, id, "operation", ContinueInput{Decision: d, ReviewResolved: true}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := s.RetryStep(ctx, token, id, "operation", RetryStepInput{StepDecision: step, SafeToRetry: true}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, change := range []func(*Decision){func(v *Decision) { v.ExpectedRevision = 0 }, func(v *Decision) { v.Reason = " " }, func(v *Decision) { v.Reason = string([]byte{255}) }, func(v *Decision) { v.EvidenceRef = "secret bad" }, func(v *Decision) { v.EvidenceRef = "" }} {
|
||||||
|
bad := d
|
||||||
|
change(&bad)
|
||||||
|
if _, err := s.Continue(ctx, token, id, "operation", ContinueInput{Decision: bad, ReviewResolved: true}); err == nil {
|
||||||
|
t.Fatal("invalid decision accepted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, bad := range []ReviewTaskInput{
|
||||||
|
{Decision: d, TaskID: id, ExpectedTaskRevision: 4, Outcome: "SUCCEEDED", Result: json.RawMessage(`{}`)},
|
||||||
|
{Decision: d, TaskID: id, ExpectedTaskRevision: 4, ExecutionSettled: true, Outcome: "UNKNOWN", Result: json.RawMessage(`{}`)},
|
||||||
|
{Decision: d, TaskID: id, ExpectedTaskRevision: 4, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`null`)},
|
||||||
|
{Decision: d, TaskID: id, ExpectedTaskRevision: 4, ExecutionSettled: true, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"n":1,"n":2}`)},
|
||||||
|
} {
|
||||||
|
if _, err := s.ReviewTask(ctx, token, id, "operation", bad); err == nil {
|
||||||
|
t.Fatal("unsafe execution review accepted")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, err := s.Continue(ctx, token, id, "operation", ContinueInput{Decision: d}); err == nil {
|
||||||
|
t.Fatal("unresolved review continued")
|
||||||
|
}
|
||||||
|
if _, err := s.RetryStep(ctx, token, id, "operation", RetryStepInput{StepDecision: step}); err == nil {
|
||||||
|
t.Fatal("unsafe retry accepted")
|
||||||
|
}
|
||||||
|
if store.calls != 4 {
|
||||||
|
t.Fatal("invalid inputs reached storage", store.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user