feat: recover poll activations within original round budgets
This commit is contained in:
13
AGENTS.md
13
AGENTS.md
@@ -31,17 +31,17 @@
|
|||||||
|
|
||||||
- P4定义发布使用schema_version=1有限结构,词法作用域禁止跨分支/前向读取,任务策略显式冻结;SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁;原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务;同版本不同内容不得覆盖。保留精确数字,校验存储原文与结构摘要;不得把发布通过或DSL支持误称为实例执行已实现。
|
- P4定义发布使用schema_version=1有限结构,词法作用域禁止跨分支/前向读取,任务策略显式冻结;SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁;原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务;同版本不同内容不得覆盖。保留精确数字,校验存储原文与结构摘要;不得把发布通过或DSL支持误称为实例执行已实现。
|
||||||
|
|
||||||
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归(docs/reports/phase-4-definitions.md)。实例启动/固定版本/首批任务同事务的内部存储已补齐;内部事件消费、顺序与组合推进及根接续已接入;TIMER与后续POLL时钟已接入,SIGNAL及最终查询竞争已接入,内部控制与停止结清已接入,继续POLL/等待恢复及实例公共SDK;P4整体和P5–P7仍未完成。
|
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归(docs/reports/phase-4-definitions.md)。实例启动/固定版本/首批任务同事务的内部存储已补齐;内部事件消费、顺序与组合推进及根接续已接入;TIMER与后续POLL时钟已接入,SIGNAL及最终查询竞争已接入,内部控制与停止结清已接入,继续独立SIGNAL/TIMER等待重建及实例公共SDK;P4整体和P5–P7仍未完成。
|
||||||
|
|
||||||
- 内部结果交付使用workflow_activation所有者并固定激活ID,TASK经activation.task_id、POLL经轮次校验归属;weave_inbox来源及独立快照与Pulse结果同事务,先于外部结果流行锁,不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史,禁止用当前任务重建旧事件。
|
- 内部结果交付使用workflow_activation所有者并固定激活ID,TASK经activation.task_id、POLL经轮次校验归属;weave_inbox来源及独立快照与Pulse结果同事务,先于外部结果流行锁,不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史,禁止用当前任务重建旧事件。
|
||||||
- weave/engine仅计算有限决策;实例受理、Inbox消费与根接续已接入,等待和停止结清运行器已接入,内部核查/TASK恢复已接入,POLL/等待恢复继续开发。Plan最多32个变更,More必须持久化待续;耗尽变更预算仍须遍历其他已活动分支,不能误判在途工作不可达。冻结分支选择,精确数值,不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务,不能把纯快照往返测试当作真实进程恢复。
|
- weave/engine仅计算有限决策;实例受理、Inbox消费与根接续已接入,等待和停止结清运行器已接入,内部核查/TASK恢复已接入,独立SIGNAL/TIMER等待重建继续开发。Plan最多32个变更,More必须持久化待续;耗尽变更预算仍须遍历其他已活动分支,不能误判在途工作不可达。冻结分支选择,精确数值,不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务,不能把纯快照往返测试当作真实进程恢复。
|
||||||
|
|
||||||
- 实例内部受理已实现,信号内部运行器已接入,公共实例/信号HTTP及独立SDK尚未开放;结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作,所有失败同事务回滚。首批推进递增revision至2,原键返回冻结回执且仍核验当前权限,不重算截止。
|
- 实例内部受理已实现,信号内部运行器已接入,公共实例/信号HTTP及独立SDK尚未开放;结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作,所有失败同事务回滚。首批推进递增revision至2,原键返回冻结回执且仍核验当前权限,不重算截止。
|
||||||
- 实例根步骤`$root`/JOIN不属于用户定义,Steps查询排除它;More对应根步骤版本化持久检查点。TASK直接关联激活task_id,POLL通过轮次关联,实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结,旧NULL等待须核实,不能盲目解释;新关联512字节、存储兼容历史764字节。
|
- 实例根步骤`$root`/JOIN不属于用户定义,Steps查询排除它;More对应根步骤版本化持久检查点。TASK直接关联激活task_id,POLL通过轮次关联,实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结,旧NULL等待须核实,不能盲目解释;新关联512字节、存储兼容历史764字节。
|
||||||
|
|
||||||
- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁,禁止先锁Inbox再锁父资源。来源使用独立冻结快照,旧激活/低版本留痕;数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。
|
- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁,禁止先锁Inbox再锁父资源。来源使用独立冻结快照,旧激活/低版本留痕;数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。
|
||||||
- 后继满额用SAVEPOINT回滚整轮新工作,保留已接受结果和待续;恢复必须重读持久图。多次根推进使用独立系统审计操作身份;迟到证据也递增实例历史版本,禁止复用已有workflow.advanced唯一身份。
|
- 后继满额用SAVEPOINT回滚整轮新工作,保留已接受结果和待续;恢复必须重读持久图。多次根推进使用独立系统审计操作身份;迟到证据也递增实例历史版本,禁止复用已有workflow.advanced唯一身份。
|
||||||
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证,见docs/reports/phase-4-runtime.md。TIMER到期与POLL后续轮次已接入,见docs/13-workflow-clock.md;SIGNAL及最终信号竞争已接入(docs/14-workflow-signals.md);内部FAIL_AFTER_SETTLED分支结清与控制已接入(docs/15-workflow-controls.md);POLL/等待恢复、公共控制API和实例SDK仍须完成,不得按部分自动运行宣称P4/P5完成。
|
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证,见docs/reports/phase-4-runtime.md。TIMER到期与POLL后续轮次已接入,见docs/13-workflow-clock.md;SIGNAL及最终信号竞争已接入(docs/14-workflow-signals.md);内部FAIL_AFTER_SETTLED分支结清与控制已接入(docs/15-workflow-controls.md);独立SIGNAL/TIMER等待重建、公共控制API和实例SDK仍须完成,不得按部分自动运行宣称P4/P5完成。
|
||||||
|
|
||||||
- TIMER/POLL复用weave_pending_advances及到期索引,时钟固定激活ID;迁移000019只回填原等待/查询时钟,不改截止和预算。后续POLL只由已确认POLL_PENDING开启,首轮精确输入/资源键冻结,每轮Pulse尝试独立计数;满额/暂停不消耗轮数。截止遇到已落库结果先等待Inbox,仍在途则关闭门控但保留ACTIVE和当前任务,许可由Pulse结清;不得由时钟假定外部请求已结束。
|
- TIMER/POLL复用weave_pending_advances及到期索引,时钟固定激活ID;迁移000019只回填原等待/查询时钟,不改截止和预算。后续POLL只由已确认POLL_PENDING开启,首轮精确输入/资源键冻结,每轮Pulse尝试独立计数;满额/暂停不消耗轮数。截止遇到已落库结果先等待Inbox,仍在途则关闭门控但保留ACTIVE和当前任务,许可由Pulse结清;不得由时钟假定外部请求已结束。
|
||||||
|
|
||||||
@@ -52,4 +52,9 @@
|
|||||||
- StopKind/StopRequestedAt冻结首次CANCEL或FAILURE目标,后来的用户取消不覆盖原失败目标。结清复用ROOT,每事务最多一个Pulse取消、32项步骤收尾;结果流锁取得后不能再锁另一任务。终态前校验全部历史任务的双向归属、未结束尝试和未释放许可,UNKNOWN不得终结。迁移000021恢复旧停止意图,不添加外键。
|
- StopKind/StopRequestedAt冻结首次CANCEL或FAILURE目标,后来的用户取消不覆盖原失败目标。结清复用ROOT,每事务最多一个Pulse取消、32项步骤收尾;结果流锁取得后不能再锁另一任务。终态前校验全部历史任务的双向归属、未结束尝试和未释放许可,UNKNOWN不得终结。迁移000021恢复旧停止意图,不添加外键。
|
||||||
|
|
||||||
- 人工核查使用显式workflows:recover、当前实例revision、步骤/任务版本及激活身份,必须有原因和证据;执行结论、步骤结论、解除核查分别确认。新决定拒绝PENDING/QUARANTINED Inbox,原键回执仍核验权限。核查回执/额度/双审计/结果流同事务,结果流锁最后取得。
|
- 人工核查使用显式workflows:recover、当前实例revision、步骤/任务版本及激活身份,必须有原因和证据;执行结论、步骤结论、解除核查分别确认。新决定拒绝PENDING/QUARANTINED Inbox,原键回执仍核验权限。核查回执/额度/双审计/结果流同事务,结果流锁最后取得。
|
||||||
- TASK人工重试保留首次绝对截止和精确输入,以新激活/关联任务继续消耗原max_attempts;激活数也受该上限限制,不能无限制造零尝试任务。成功步骤、未知执行、原停止意图或预算耗尽均不得重试。Continue不解除paused、重置预算或撤销停止。POLL跨激活与等待恢复尚未实现,公共HTTP/SDK也未开放,不能据此宣称P5完成。
|
- TASK人工重试保留首次绝对截止和精确输入,以新激活/关联任务继续消耗原max_attempts;激活数也受该上限限制,不能无限制造零尝试任务。成功步骤、未知执行、原停止意图或预算耗尽均不得重试。Continue不解除paused、重置预算或撤销停止。POLL跨激活恢复已实现(docs/17-workflow-poll-recovery.md),独立SIGNAL/TIMER等待重建尚未实现,公共HTTP/SDK也未开放,不能据此宣称P5完成。
|
||||||
|
|
||||||
|
- POLL人工恢复以新激活重查同一业务轮,FirstRound/FirstRoundAttemptsUsed保留原轮次和已用尝试;实际尝试行、retry_of链和冻结参数必须相符。同轮任务数也受原max_attempts限制,PENDING不得人工重查绕过轮次上限。查询截止按首次激活、请求截止按该轮首任务,不允许重计时;下一业务轮才获得完整单轮策略。
|
||||||
|
- POLL恢复的最终信号等待保留原契约/关联/截止并更换代次,旧winner和已消费信号只留历史;存在当前代次PENDING信号时拒绝重建。迁移000022无外键,新增2个中文注释字段。独立SIGNAL/TIMER等待重建、公共HTTP/SDK与生产验收继续推进。
|
||||||
|
|
||||||
|
- 核查将UNKNOWN请求确认成成功PENDING时,当前RetryStep仍拒绝重查同轮;显式恢复等待/后续业务轮的管理操作尚待实现,必须保留原间隔、轮次与截止并使用新的审计决定,不能由ReviewTask自动解除核查。
|
||||||
|
|||||||
@@ -10,7 +10,7 @@ Orbit 是面向多个业务项目的通用任务调度与流程编排平台,
|
|||||||
|
|
||||||
已完成 P0–P3 平台功能阶段验收:35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过;[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK,[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例,[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现;[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例;[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例;[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景;[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本;[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI;[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复;完整范围核对见[P3验收](docs/reports/phase-3.md),流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。
|
已完成 P0–P3 平台功能阶段验收:35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过;[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK,[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例,[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现;[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例;[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例;[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景;[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本;[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI;[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复;完整范围核对见[P3验收](docs/reports/phase-3.md),流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。
|
||||||
|
|
||||||
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现;已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。[定时等待、后续查询轮次及原预算恢复](docs/13-workflow-clock.md)已接入;[外部信号和查询最终信号竞争](docs/14-workflow-signals.md)已接入内部运行器;[暂停/继续/取消与失败结清](docs/15-workflow-controls.md)已接入内部运行器;[人工核查、解除核查与TASK新激活](docs/16-workflow-recovery.md)已接入内部服务;POLL/等待恢复及实例公共API/SDK继续开发,发布定义不会启动任务。
|
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现;已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。[定时等待、后续查询轮次及原预算恢复](docs/13-workflow-clock.md)已接入;[外部信号和查询最终信号竞争](docs/14-workflow-signals.md)已接入内部运行器;[暂停/继续/取消与失败结清](docs/15-workflow-controls.md)已接入内部运行器;[人工核查、解除核查与TASK新激活](docs/16-workflow-recovery.md)已接入内部服务;[POLL跨激活恢复](docs/17-workflow-poll-recovery.md)已接入;独立SIGNAL/TIMER等待重建及实例公共API/SDK继续开发,发布定义不会启动任务。
|
||||||
|
|
||||||
数据库禁止外键,关联一致性通过服务端事务校验,索引、唯一约束和 CHECK 保留;readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。
|
数据库禁止外键,关联一致性通过服务端事务校验,索引、唯一约束和 CHECK 保留;readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 流程编排功能与设计
|
# Weave 流程编排功能与设计
|
||||||
|
|
||||||
> 版本 0.7 · 2026-09-18。首版功能范围已确认。定义结构、发布校验与不可变版本已实现,见[定义格式](api/workflow-definition-format.md)和[发布API](api/workflow-definitions.md);[实例受理](11-workflow-admission.md)、[结果消费与根推进](12-workflow-runtime.md)、[TIMER与POLL时钟](13-workflow-clock.md)已接入。[信号等待及最终查询竞争](14-workflow-signals.md)已接入内部运行器;[暂停/继续/取消与停止结清](15-workflow-controls.md)已实现内部链路;[人工核查与TASK恢复](16-workflow-recovery.md)已接入内部服务;POLL/等待恢复及实例公共API/SDK仍待P4–P5实现验收。
|
> 版本 0.8 · 2026-09-18。首版功能范围已确认。定义结构、发布校验与不可变版本已实现,见[定义格式](api/workflow-definition-format.md)和[发布API](api/workflow-definitions.md);[实例受理](11-workflow-admission.md)、[结果消费与根推进](12-workflow-runtime.md)、[TIMER与POLL时钟](13-workflow-clock.md)已接入。[信号等待及最终查询竞争](14-workflow-signals.md)已接入内部运行器;[暂停/继续/取消与停止结清](15-workflow-controls.md)已实现内部链路;[人工核查与TASK恢复](16-workflow-recovery.md)已接入内部服务;[POLL跨激活恢复](17-workflow-poll-recovery.md)已接入;独立SIGNAL/TIMER等待重建及实例公共API/SDK仍待P4–P5实现验收。
|
||||||
|
|
||||||
## 1. 职责与边界
|
## 1. 职责与边界
|
||||||
|
|
||||||
@@ -136,7 +136,7 @@ Weave 不执行特定业务逻辑,也不把整段业务代码放入平台进
|
|||||||
|
|
||||||
查询任务自身成功表示取得了一份有效结果。其中“处理中”使 Weave 保存等待与下一次唤醒;最终业务结果则结束查询节点,按定义推进或进入失败处理。单轮请求重试耗尽应转核查,不能通过新建一轮绕过已耗尽的重试预算。
|
查询任务自身成功表示取得了一份有效结果。其中“处理中”使 Weave 保存等待与下一次唤醒;最终业务结果则结束查询节点,按定义推进或进入失败处理。单轮请求重试耗尽应转核查,不能通过新建一轮绕过已耗尽的重试预算。
|
||||||
|
|
||||||
每轮只创建一个查询任务,以命名空间、实例、步骤激活和轮次形成稳定去重身份。轮次之间释放业务 Worker 槽位,等待依赖数据库记录与平台到期索引扫描;SDK 不在处理器中长期 sleep 或维护内存轮询循环。
|
每个步骤激活的每轮只创建一个查询任务,以命名空间、实例、步骤激活和轮次形成稳定去重身份。人工恢复同一业务轮时创建新激活、保留原轮编号并扣除历史尝试,见[查询恢复](17-workflow-poll-recovery.md)。轮次之间释放业务 Worker 槽位,等待依赖数据库记录与平台到期索引扫描;SDK 不在处理器中长期 sleep 或维护内存轮询循环。
|
||||||
|
|
||||||
2026-09-18实现决策:TIMER/POLL直接复用`weave_pending_advances`检查点和后台扫描,替代早期草案的Pulse内部唤醒任务。唤醒只改变流程持久状态,不调用业务处理器,因此无需额外占用任务/尝试/许可记录;业务查询仍由Pulse执行。当前schema使用固定间隔,不提供业务轮次抖动或变长退避,具体冻结和升级行为见[运行约定](13-workflow-clock.md)。
|
2026-09-18实现决策:TIMER/POLL直接复用`weave_pending_advances`检查点和后台扫描,替代早期草案的Pulse内部唤醒任务。唤醒只改变流程持久状态,不调用业务处理器,因此无需额外占用任务/尝试/许可记录;业务查询仍由Pulse执行。当前schema使用固定间隔,不提供业务轮次抖动或变长退避,具体冻结和升级行为见[运行约定](13-workflow-clock.md)。
|
||||||
|
|
||||||
@@ -174,7 +174,7 @@ Weave 不执行特定业务逻辑,也不把整段业务代码放入平台进
|
|||||||
3. 在一个事务内记录管理操作,创建新的步骤激活及关联 Pulse 任务,并固定关联关系。
|
3. 在一个事务内记录管理操作,创建新的步骤激活及关联 Pulse 任务,并固定关联关系。
|
||||||
4. 保留旧激活与旧任务结果,沿用恰当的业务幂等身份。
|
4. 保留旧激活与旧任务结果,沿用恰当的业务幂等身份。
|
||||||
|
|
||||||
执行核查、步骤结论、解除核查及TASK新激活已按[人工恢复约定](16-workflow-recovery.md)实现内部服务,原预算和绝对截止不重置;POLL跨激活及等待恢复仍需补齐,公共API另行交付。普通任务人工重试入口应拒绝绕过流程管理的关联任务请求。已经确认成功的步骤不能通过“重试流程”随意再执行一次。
|
执行核查、步骤结论、解除核查及TASK新激活已按[人工恢复约定](16-workflow-recovery.md)实现内部服务,原预算和绝对截止不重置;[POLL跨激活恢复](17-workflow-poll-recovery.md)已接入,独立SIGNAL/TIMER等待重建仍需补齐,公共API另行交付。普通任务人工重试入口应拒绝绕过流程管理的关联任务请求。已经确认成功的步骤不能通过“重试流程”随意再执行一次。
|
||||||
|
|
||||||
取消和补偿是不同业务操作。若未来增加补偿,补偿本身也必须是有权限、可审计、可重试或可核查的业务步骤;不能宣称撤销流程就能撤回已经生效的外部业务操作。
|
取消和补偿是不同业务操作。若未来增加补偿,补偿本身也必须是有权限、可审计、可重试或可核查的业务步骤;不能宣称撤销流程就能撤回已经生效的外部业务操作。
|
||||||
|
|
||||||
|
|||||||
@@ -207,8 +207,13 @@ Signal新增payload_version、target_activation_sequence、evidence_ref、dispos
|
|||||||
|
|
||||||
## 停止意图增量(000021)
|
## 停止意图增量(000021)
|
||||||
|
|
||||||
Instance新增StopKind和StopRequestedAt,冻结首次用户取消或失败结清目标;CancelRequestedAt仍只表示用户显式取消。成对CHECK防止半条意图,任务所有者索引支持双向归属核验。当前35个实体、303个直接持久化字段声明,均有中文注释。历史停止实例关闭门控并恢复根接续,不改变原截止、预算、控制版本或终态数据;保持零外键,详见[控制约定](15-workflow-controls.md)。
|
Instance新增StopKind和StopRequestedAt,冻结首次用户取消或失败结清目标;CancelRequestedAt仍只表示用户显式取消。成对CHECK防止半条意图,任务所有者索引支持双向归属核验。截至000021为35个实体、303个直接持久化字段声明,均有中文注释。历史停止实例关闭门控并恢复根接续,不改变原截止、预算、控制版本或终态数据;保持零外键,详见[控制约定](15-workflow-controls.md)。
|
||||||
|
|
||||||
## 流程人工核查与TASK新激活
|
## 流程人工核查与TASK新激活
|
||||||
|
|
||||||
复用现有激活sequence、任务retry_of、审计和幂等回执,不增加表、字段或外键。TASK人工重试以新激活保留旧执行,原尝试预算跨激活扣除,原绝对截止不改变;执行核查和步骤逻辑结论分别保存。35个实体、303个中文注释字段。当前内部支持范围与POLL/等待恢复剩余项见[核查约定](16-workflow-recovery.md)。
|
复用现有激活sequence、任务retry_of、审计和幂等回执,不增加表、字段或外键。TASK人工重试以新激活保留旧执行,原尝试预算跨激活扣除,原绝对截止不改变;执行核查和步骤逻辑结论分别保存。35个实体、303个中文注释字段。当前内部支持范围与独立SIGNAL/TIMER等待重建剩余项见[核查约定](16-workflow-recovery.md)。
|
||||||
|
|
||||||
|
|
||||||
|
### 000022:跨激活保留查询轮次与尝试
|
||||||
|
|
||||||
|
PollState新增FirstRound和FirstRoundAttemptsUsed,记录当前激活的首个业务轮及该轮在旧激活已消耗的尝试。UsedRounds继续使用首次激活起的业务轮编号;原单轮尝试和查询/请求绝对截止均不重置。新业务轮恢复原单轮策略,人工恢复同轮只分配剩余尝试。当前35个实体、305个直接持久化字段声明,Go/SQL均有中文注释,无外键。v21升级保留旧数据、时间和预算,详情见[查询恢复](17-workflow-poll-recovery.md)。
|
||||||
|
|||||||
@@ -28,8 +28,8 @@
|
|||||||
| P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件;SDK 客户端 | API/SDK 创建查询闭环;重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归(见 P1 报告;真实断路/提交 ACK 丢失注入留后续故障阶段) |
|
| P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件;SDK 客户端 | API/SDK 创建查询闭环;重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归(见 P1 报告;真实断路/提交 ACK 丢失注入留后续故障阶段) |
|
||||||
| P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复;Worker SDK | 真 Worker 执行;失联/重连/重复确认/旧尝试拒绝;业务效果未知转核查;节点故障接管 | 已完成并通过阶段回归:服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复;测试拓扑和剩余生产故障矩阵见 P2 验收报告 |
|
| P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复;Worker SDK | 真 Worker 执行;失联/重连/重复确认/旧尝试拒绝;业务效果未知转核查;节点故障接管 | 已完成并通过阶段回归:服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复;测试拓扑和剩余生产故障矩阵见 P2 验收报告 |
|
||||||
| P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收:[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景;20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 |
|
| P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收:[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景;20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 |
|
||||||
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容;端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现,已通过[子项验收](reports/phase-4-definitions.md);已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md);[实例受理及首批事务](reports/phase-4-admission.md)已实现;[结果消费和根接续](reports/phase-4-runtime.md)已接入后台,顺序/分支/并行及真实进程恢复已验证;[TIMER与后续POLL轮次](13-workflow-clock.md)已接入;[信号消费和最终查询竞争](14-workflow-signals.md)已接入内部运行器;[内部控制与停止结清](15-workflow-controls.md)已接入;[内部核查与TASK新激活](16-workflow-recovery.md)已接入;POLL/等待恢复和实例公共API/SDK继续开发 |
|
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容;端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现,已通过[子项验收](reports/phase-4-definitions.md);已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md);[实例受理及首批事务](reports/phase-4-admission.md)已实现;[结果消费和根接续](reports/phase-4-runtime.md)已接入后台,顺序/分支/并行及真实进程恢复已验证;[TIMER与后续POLL轮次](13-workflow-clock.md)已接入;[信号消费和最终查询竞争](14-workflow-signals.md)已接入内部运行器;[内部控制与停止结清](15-workflow-controls.md)已接入;[内部核查与TASK新激活](16-workflow-recovery.md)已接入;独立SIGNAL/TIMER等待重建和实例公共API/SDK继续开发 |
|
||||||
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算;暂停/开始/回执/超时竞争;恢复保留预算;未知结果不盲目重复 | 有界查询轮次/尝试独立预算及原截止恢复已接入内部运行器;最终信号竞争及在途查询结清已接入;内部暂停/继续/取消与失败结清已接入;执行核查、步骤结论、解除核查及TASK新激活已接入;POLL/等待恢复及公共API/SDK继续开发 |
|
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算;暂停/开始/回执/超时竞争;恢复保留预算;未知结果不盲目重复 | 有界查询轮次/尝试独立预算及原截止恢复已接入内部运行器;最终信号竞争及在途查询结清已接入;内部暂停/继续/取消与失败结清已接入;执行核查、步骤结论、解除核查及TASK新激活已接入;[POLL跨激活恢复](17-workflow-poll-recovery.md)已接入;独立SIGNAL/TIMER等待重建及公共API/SDK继续开发 |
|
||||||
| P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行,报告环境/指标/遗留项;容量、RPO/RTO 按真实拓扑验证 | 待开发 |
|
| P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行,报告环境/指标/遗留项;容量、RPO/RTO 按真实拓扑验证 | 待开发 |
|
||||||
| P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例;所有首版能力可追溯到实现与测试;报告实际生产边界 | 待开发 |
|
| P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例;所有首版能力可追溯到实现与测试;报告实际生产边界 | 待开发 |
|
||||||
|
|
||||||
@@ -41,8 +41,8 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
|||||||
|
|
||||||
1. 已实现FAIL_AFTER_SETTLED与显式取消的内部分支结清,见[控制报告](reports/phase-5-controls.md):持久取消意图、复用Pulse取消、收集已发出查询/任务结果、等待全部已激活工作结清后才进入失败或取消终态。效果未知仍保留核查。
|
1. 已实现FAIL_AFTER_SETTLED与显式取消的内部分支结清,见[控制报告](reports/phase-5-controls.md):持久取消意图、复用Pulse取消、收集已发出查询/任务结果、等待全部已激活工作结清后才进入失败或取消终态。效果未知仍保留核查。
|
||||||
2. 已实现实例暂停/继续/取消内部操作,以control_revision、操作幂等身份、明确原因及审计保护控制决定;暂停不延长原截止,继续不覆盖核查或取消状态。
|
2. 已实现实例暂停/继续/取消内部操作,以control_revision、操作幂等身份、明确原因及审计保护控制决定;暂停不延长原截止,继续不覆盖核查或取消状态。
|
||||||
3. 已实现内部执行核查、步骤结论、解除核查及TASK新激活和旧激活隔离,见[核查报告](reports/phase-5-recovery.md)。继续POLL跨激活及等待恢复,保留原预算、轮次、结果和证据,不能通过换激活/新键绕过未知效果或执行上限。
|
3. 已实现内部执行核查、步骤结论、解除核查及TASK新激活和旧激活隔离,见[核查报告](reports/phase-5-recovery.md)。POLL跨激活恢复已实现(见[查询恢复](17-workflow-poll-recovery.md));继续核查后已确认PENDING的有界等待接续及独立SIGNAL/TIMER等待重建,保留原预算、轮次、结果和证据,不能通过换激活/新键绕过未知效果或执行上限。
|
||||||
4. 同步开放实例/步骤/信号/来源/审计HTTP、独立SDK和端到端场景,覆盖原键恢复、权限、版本冲突、竞争与进程全停;公共能力逐项记录实际验收结果。
|
4. 同步开放实例/步骤/信号/来源/审计HTTP、独立SDK和端到端场景,覆盖原键恢复、权限、版本冲突、竞争与进程全停;公共步骤/激活查询须返回目标激活代次,调用方据此发送恢复后的信号,不能要求调用方自行猜测sequence。公共能力逐项记录实际验收结果。
|
||||||
5. 转入P6/P7的生产故障、容量、运维、发布与全部文档核对。内部存储阶段通过不会提前勾选公共接口或生产验收。
|
5. 转入P6/P7的生产故障、容量、运维、发布与全部文档核对。内部存储阶段通过不会提前勾选公共接口或生产验收。
|
||||||
|
|
||||||
### P3 功能范围核对
|
### P3 功能范围核对
|
||||||
@@ -119,6 +119,8 @@ P4/P5定时等待与后续查询轮次的实现及回归记录见[时钟报告](
|
|||||||
|
|
||||||
## 7. 文档入口
|
## 7. 文档入口
|
||||||
|
|
||||||
|
- [有界查询恢复](17-workflow-poll-recovery.md)
|
||||||
|
- [POLL恢复验证记录](reports/phase-5-poll-recovery.md)
|
||||||
- [数据模型](07-data-model.md)
|
- [数据模型](07-data-model.md)
|
||||||
- [可靠结果订阅核心](09-result-subscription-core.md)
|
- [可靠结果订阅核心](09-result-subscription-core.md)
|
||||||
- [P3 订阅核心回归](reports/phase-3-result-subscription-core.md)
|
- [P3 订阅核心回归](reports/phase-3-result-subscription-core.md)
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ v16及以前的未处理记录缺少冻结来源,升级后统一标为QUARANTI
|
|||||||
|
|
||||||
## 4. 后续存储适配必须满足的条件
|
## 4. 后续存储适配必须满足的条件
|
||||||
|
|
||||||
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证;第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md),第5项的TIMER/POLL已接入[时钟运行器](13-workflow-clock.md),第5项的信号等待已接入[信号运行器](14-workflow-signals.md),内部控制与结清已接入[控制运行器](15-workflow-controls.md),[核查与TASK新激活](16-workflow-recovery.md)已接入;POLL/等待恢复、接口和SDK仍继续实现:
|
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证;第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md),第5项的TIMER/POLL已接入[时钟运行器](13-workflow-clock.md),第5项的信号等待已接入[信号运行器](14-workflow-signals.md),内部控制与结清已接入[控制运行器](15-workflow-controls.md),[核查与TASK新激活](16-workflow-recovery.md)已接入;独立SIGNAL/TIMER等待重建、接口和SDK仍继续实现:
|
||||||
|
|
||||||
1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止,并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。
|
1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止,并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。
|
||||||
2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本;消费确认、步骤变化、后续任务或持久化待续记录同事务提交。
|
2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本;消费确认、步骤变化、后续任务或持久化待续记录同事务提交。
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 实例受理与首批工作
|
# Weave 实例受理与首批工作
|
||||||
|
|
||||||
状态:内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口;[内部控制和结清](15-workflow-controls.md)已接入,[内部核查及TASK恢复](16-workflow-recovery.md)已接入,POLL/等待恢复继续开发;[信号运行器](14-workflow-signals.md)已接入。** Weave相关公开SDK目前仅支持定义发布。[TIMER和后续POLL轮次](13-workflow-clock.md)已接入,TASK及组合节点已可自动推进,不能将此解释成全部等待/控制能力已实现。
|
状态:内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口;[内部控制和结清](15-workflow-controls.md)已接入,[内部核查及TASK恢复](16-workflow-recovery.md)已接入,独立SIGNAL/TIMER等待重建继续开发;[信号运行器](14-workflow-signals.md)已接入。** Weave相关公开SDK目前仅支持定义发布。[TIMER和后续POLL轮次](13-workflow-clock.md)已接入,TASK及组合节点已可自动推进,不能将此解释成全部等待/控制能力已实现。
|
||||||
|
|
||||||
## 1. 受理事务
|
## 1. 受理事务
|
||||||
|
|
||||||
@@ -49,7 +49,7 @@ Get返回当前实例视图;Start重报返回受理时冻结的原始视图,
|
|||||||
| 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例;终态不计入;满额仍可恢复原受理回执 |
|
| 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例;终态不计入;满额仍可恢复原受理回执 |
|
||||||
| 信号关联 | 新受理最多512字节;数据库兼容旧191字符最长764字节,升级不截断历史 |
|
| 信号关联 | 新受理最多512字节;数据库兼容旧191字符最长764字节,升级不截断历史 |
|
||||||
| 截止 | 数据库时间生成;任务、查询与等待均不超过实例总截止 |
|
| 截止 | 数据库时间生成;任务、查询与等待均不超过实例总截止 |
|
||||||
| 自动运行 | Inbox消费、根待续、TIMER到期及后续POLL轮次已接入;SIGNAL与最终查询竞争已接入,内部控制与结清已接入,内部核查及TASK恢复已接入,POLL/等待恢复仍待实现 |
|
| 自动运行 | Inbox消费、根待续、TIMER到期及后续POLL轮次已接入;SIGNAL与最终查询竞争已接入,内部控制与结清已接入,内部核查及TASK恢复已接入,独立SIGNAL/TIMER等待重建仍待实现 |
|
||||||
|
|
||||||
迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段;NULL仅表示TIMER或尚未核实的历史元数据,未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储;实例business_key改为utf8mb4_0900_bin。该迁移时为35个实体、含公共Record的295个直接持久化字段声明,全部有中文注释,继续保持零外键。
|
迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段;NULL仅表示TIMER或尚未核实的历史元数据,未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储;实例business_key改为utf8mb4_0900_bin。该迁移时为35个实体、含公共Record的295个直接持久化字段声明,全部有中文注释,继续保持零外键。
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 结果消费与自动推进
|
# Weave 结果消费与自动推进
|
||||||
|
|
||||||
状态:内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。[定时等待和后续查询轮次](13-workflow-clock.md)也已接入。[信号等待与最终查询竞争](14-workflow-signals.md)已接入。实例公共HTTP/SDK仍未开放,[失败后的分支取消结清与内部控制](15-workflow-controls.md)已接入,[内部核查及TASK恢复](16-workflow-recovery.md)已接入,POLL/等待恢复继续开发,不能将本阶段等同于完整Weave交付。
|
状态:内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。[定时等待和后续查询轮次](13-workflow-clock.md)也已接入。[信号等待与最终查询竞争](14-workflow-signals.md)已接入。实例公共HTTP/SDK仍未开放,[失败后的分支取消结清与内部控制](15-workflow-controls.md)已接入,[内部核查及TASK恢复](16-workflow-recovery.md)已接入,独立SIGNAL/TIMER等待重建继续开发,不能将本阶段等同于完整Weave交付。
|
||||||
|
|
||||||
## 1. 后台运行和事务边界
|
## 1. 后台运行和事务边界
|
||||||
|
|
||||||
@@ -64,4 +64,4 @@ flowchart LR
|
|||||||
|
|
||||||
进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过;跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。
|
进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过;跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。
|
||||||
|
|
||||||
持久化信号等待及查询最终信号竞争已按[信号运行约定](14-workflow-signals.md)实现。内部控制及停止结清也已接入。下一步仍包括POLL/等待恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
持久化信号等待及查询最终信号竞争已按[信号运行约定](14-workflow-signals.md)实现。内部控制及停止结清也已接入。下一步仍包括独立SIGNAL/TIMER等待重建、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
||||||
|
|||||||
@@ -61,3 +61,5 @@ TIMER由待续表直接唤醒,不创建Pulse业务任务,也不占Worker执
|
|||||||
[存储回归](../internal/storage/mysql/workflow_clock_integration_test.go)覆盖到期、竞争、预算、暂停/满额、回滚、归属损坏与未知结果;[v18升级回归](../internal/storage/mysql/workflow_clock_upgrade_integration_test.go)验证回填及原预算保留;[真实进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖全部控制进程退出后的定时和查询接续。执行记录见[阶段报告](reports/phase-4-clock.md)。
|
[存储回归](../internal/storage/mysql/workflow_clock_integration_test.go)覆盖到期、竞争、预算、暂停/满额、回滚、归属损坏与未知结果;[v18升级回归](../internal/storage/mysql/workflow_clock_upgrade_integration_test.go)验证回填及原预算保留;[真实进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖全部控制进程退出后的定时和查询接续。执行记录见[阶段报告](reports/phase-4-clock.md)。
|
||||||
|
|
||||||
SIGNAL早到/重复/超期处理及最终查询竞争已接入;内部失败分支结清和控制已接入;下一阶段继续人工恢复、公共控制及实例SDK,完整范围见[持续计划](08-development-plan.md)。
|
SIGNAL早到/重复/超期处理及最终查询竞争已接入;内部失败分支结清和控制已接入;下一阶段继续人工恢复、公共控制及实例SDK,完整范围见[持续计划](08-development-plan.md)。
|
||||||
|
|
||||||
|
人工恢复跨激活后,查询原截止、连续业务轮编号和同轮请求预算仍保留;首轮已解析输入始终从首次激活读取。具体规则见[有界查询恢复](17-workflow-poll-recovery.md)。
|
||||||
|
|||||||
@@ -65,4 +65,4 @@
|
|||||||
|
|
||||||
[受理回归](../internal/storage/mysql/workflow_signals_integration_test.go)、[消费回归](../internal/storage/mysql/workflow_signal_runtime_integration_test.go)、[真实v19升级](../internal/storage/mysql/workflow_signal_upgrade_integration_test.go)和[实际进程恢复](../internal/storage/mysql/workflow_runtime_process_integration_test.go)对应受理/并发/故障/升级边界,实际运行结果记录在[阶段报告](reports/phase-4-signals.md)。
|
[受理回归](../internal/storage/mysql/workflow_signals_integration_test.go)、[消费回归](../internal/storage/mysql/workflow_signal_runtime_integration_test.go)、[真实v19升级](../internal/storage/mysql/workflow_signal_upgrade_integration_test.go)和[实际进程恢复](../internal/storage/mysql/workflow_runtime_process_integration_test.go)对应受理/并发/故障/升级边界,实际运行结果记录在[阶段报告](reports/phase-4-signals.md)。
|
||||||
|
|
||||||
内部失败分支取消结清、实例暂停/继续/取消已接入。下一步完成人工核查恢复,再交付实例与信号HTTP、独立SDK及完整场景。当前内部存储测试不替代尚未开放的公共接口验收,也不替代生产MySQL切换与容量测试。
|
内部失败分支取消结清、实例暂停/继续/取消已接入。人工核查及[POLL新激活的最终信号换代](17-workflow-poll-recovery.md)已接入;下一步完成独立等待重建,再交付实例与信号HTTP、独立SDK及完整场景。当前内部存储测试不替代尚未开放的公共接口验收,也不替代生产MySQL切换与容量测试。
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 控制与停止结清
|
# Weave 控制与停止结清
|
||||||
|
|
||||||
状态:内部Go暂停、继续、取消服务及后台失败/取消结清运行器已接入。**公共实例/控制HTTP与独立SDK尚未开放,[内部核查及TASK新激活](16-workflow-recovery.md)已接入,POLL/等待恢复仍在后续交付范围**。本文说明已实现的持久化规则,不把内部存储回归当作全部Weave或生产验收。
|
状态:内部Go暂停、继续、取消服务及后台失败/取消结清运行器已接入。**公共实例/控制HTTP与独立SDK尚未开放,[内部核查及TASK新激活](16-workflow-recovery.md)已接入,独立SIGNAL/TIMER等待重建仍在后续交付范围**。本文说明已实现的持久化规则,不把内部存储回归当作全部Weave或生产验收。
|
||||||
|
|
||||||
## 1. 控制操作
|
## 1. 控制操作
|
||||||
|
|
||||||
@@ -60,4 +60,4 @@ CANCELLED仅表示本次流程停止已结清,不撤销先前成功的业务
|
|||||||
|
|
||||||
[控制及结清回归](../internal/storage/mysql/workflow_controls_integration_test.go)覆盖控制版本、权限重报、开始竞争、失败/未知、审计回滚、信号与查询及超过32步骤接续;[真实v20升级](../internal/storage/mysql/workflow_stop_upgrade_integration_test.go)验证兼容回填;[实际Orbit进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖停止意图和结果在全停期间保存后由两个新进程接续。实际结果见[阶段报告](reports/phase-5-controls.md)。
|
[控制及结清回归](../internal/storage/mysql/workflow_controls_integration_test.go)覆盖控制版本、权限重报、开始竞争、失败/未知、审计回滚、信号与查询及超过32步骤接续;[真实v20升级](../internal/storage/mysql/workflow_stop_upgrade_integration_test.go)验证兼容回填;[实际Orbit进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)覆盖停止意图和结果在全停期间保存后由两个新进程接续。实际结果见[阶段报告](reports/phase-5-controls.md)。
|
||||||
|
|
||||||
下一步仍需完成POLL/等待恢复、公共实例/控制/信号/审计接口、独立SDK及可运行场景,再完成P6–P7生产故障、性能、运维和发布验收。当前测试不替代生产MySQL复制、故障切换、备份恢复或RPO/RTO验证。
|
下一步仍需完成独立SIGNAL/TIMER等待重建、公共实例/控制/信号/审计接口、独立SDK及可运行场景,再完成P6–P7生产故障、性能、运维和发布验收。当前测试不替代生产MySQL复制、故障切换、备份恢复或RPO/RTO验证。
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Weave 人工核查与接续
|
# Weave 人工核查与接续
|
||||||
|
|
||||||
状态:内部Go服务已提供执行核查、步骤结论确认、解除核查,以及TASK新激活重试。公共HTTP、独立SDK和通用接入示例尚未交付;**POLL新激活恢复尚未实现,不能把已有逻辑结果核查当作查询重试已经完成**。完整交付继续按[开发计划](08-development-plan.md)推进。
|
状态:内部Go服务已提供执行核查、步骤结论确认、解除核查,以及TASK/POLL新激活重试。POLL跨激活预算与附属信号换代见[查询恢复](17-workflow-poll-recovery.md)。公共HTTP、独立SDK和通用接入示例尚未交付;独立SIGNAL/TIMER等待重建继续实现。完整交付继续按[开发计划](08-development-plan.md)推进。
|
||||||
|
|
||||||
## 1. 四类操作
|
## 1. 四类操作
|
||||||
|
|
||||||
@@ -11,7 +11,7 @@
|
|||||||
| ReviewTask | 核实本实例历史激活拥有的NEEDS_REVIEW任务,写入SUCCEEDED/FAILED、明确执行已结束和证据 | 不把查询请求成功解释成业务成功,不解除步骤或实例核查 |
|
| ReviewTask | 核实本实例历史激活拥有的NEEDS_REVIEW任务,写入SUCCEEDED/FAILED、明确执行已结束和证据 | 不把查询请求成功解释成业务成功,不解除步骤或实例核查 |
|
||||||
| ResolveStep | 将当前待核查/失败/取消叶子步骤核实为SUCCEEDED或FAILED,保存符合固定契约的逻辑结果 | 不重新执行旧任务,不清除实例核查或原停止意图 |
|
| ResolveStep | 将当前待核查/失败/取消叶子步骤核实为SUCCEEDED或FAILED,保存符合固定契约的逻辑结果 | 不重新执行旧任务,不清除实例核查或原停止意图 |
|
||||||
| Continue | 在当前证据和步骤状态允许时解除实例核查,保存根接续 | 不解除显式paused,不延长截止,不撤销取消或失败结清 |
|
| Continue | 在当前证据和步骤状态允许时解除实例核查,保存根接续 | 不解除显式paused,不延长截止,不撤销取消或失败结清 |
|
||||||
| RetryStep | 为已确认失败/取消的TASK建立新的激活和关联任务,继续消耗原尝试预算 | 不重跑成功步骤,不改变输入/业务关联,不为POLL另起新轮次 |
|
| RetryStep | 为TASK或POLL建立新的激活和关联任务,继续消耗原任务或原业务轮尝试预算 | 不重跑成功步骤,不改变输入/业务关联,POLL恢复不另起业务轮次 |
|
||||||
|
|
||||||
查询请求、外部业务结果和流程接续是不同事实。例如,查询任务核实为成功但返回PENDING,仍不能将POLL步骤核实为成功;POLL成功结论必须通过固定映射分类为SUCCEEDED。未知执行须先完成ReviewTask,再等待新版本结果进入内部Inbox并消费,才能确认步骤结论。
|
查询请求、外部业务结果和流程接续是不同事实。例如,查询任务核实为成功但返回PENDING,仍不能将POLL步骤核实为成功;POLL成功结论必须通过固定映射分类为SUCCEEDED。未知执行须先完成ReviewTask,再等待新版本结果进入内部Inbox并消费,才能确认步骤结论。
|
||||||
|
|
||||||
@@ -43,7 +43,7 @@ ResolveStep保存当前激活的新管理结论,审计记录前一状态、前
|
|||||||
|
|
||||||
## 4. TASK重试预算
|
## 4. TASK重试预算
|
||||||
|
|
||||||
当前仅TASK支持RetryStep。要求没有取消或失败停止意图,实例和原任务绝对截止尚未到达,相关历史执行均已知失败或取消。核实实际尝试行数、任务attempt_count、连续激活序号、retry_of链、固定处理器/输入/队列/优先级和策略一致性。
|
TASK恢复规则如下。要求没有取消或失败停止意图,实例和原任务绝对截止尚未到达,相关历史执行均已知失败或取消。核实实际尝试行数、任务attempt_count、连续激活序号、retry_of链、固定处理器/输入/队列/优先级和策略一致性。
|
||||||
|
|
||||||
- 新激活sequence递增;新任务retry_of指向前一任务,同一原任务只允许一个直接后继。
|
- 新激活sequence递增;新任务retry_of指向前一任务,同一原任务只允许一个直接后继。
|
||||||
- 新max_attempts = 固定定义max_attempts − 全部历史激活已用尝试数;首次尝试也计入。
|
- 新max_attempts = 固定定义max_attempts − 全部历史激活已用尝试数;首次尝试也计入。
|
||||||
@@ -54,7 +54,7 @@ ResolveStep保存当前激活的新管理结论,审计记录前一状态、前
|
|||||||
|
|
||||||
创建新任务后实例仍处于核查,门控保持关闭。操作者确认其他事项已处理后调用Continue,才允许新任务取得执行授权;原paused仍为true时还需正常RESUME。
|
创建新任务后实例仍处于核查,门控保持关闭。操作者确认其他事项已处理后调用Continue,才允许新任务取得执行授权;原paused仍为true时还需正常RESUME。
|
||||||
|
|
||||||
POLL恢复仍需实现跨激活的原轮次归属、单轮剩余尝试及原查询截止传递。当前RetryStep明确拒绝POLL,禁止通过新建业务轮次规避耗尽的单轮预算。SIGNAL/TIMER逻辑结论可核查,但不通过此操作重新开始等待;恢复等待/查询的完整范围仍需后续实现与验收。
|
POLL跨激活恢复已实现,保留原业务轮、同轮剩余尝试、查询总截止和该轮原任务截止;成功返回PENDING的轮次不能人工重查。附属最终信号等待沿用原契约/关联/截止并更换代次,详见[查询恢复](17-workflow-poll-recovery.md)。核查后已确认PENDING的显式等待接续仍需补齐;独立SIGNAL/TIMER逻辑结论可核查,等待重建仍待实现。
|
||||||
|
|
||||||
## 5. 停止与到期
|
## 5. 停止与到期
|
||||||
|
|
||||||
@@ -68,6 +68,7 @@ POLL恢复仍需实现跨激活的原轮次归属、单轮剩余尝试及原查
|
|||||||
| --- | --- |
|
| --- | --- |
|
||||||
| TASK效果未知,核实成功 | 读取当前实例/任务 → ReviewTask(SUCCEEDED) → 等待内部结果消费 → 读取当前版本 → ResolveStep(SUCCEEDED) → Continue → 后台接续 |
|
| TASK效果未知,核实成功 | 读取当前实例/任务 → ReviewTask(SUCCEEDED) → 等待内部结果消费 → 读取当前版本 → ResolveStep(SUCCEEDED) → Continue → 后台接续 |
|
||||||
| TASK已确认失败,原预算仍有余量 | 读取当前实例/步骤/激活 → RetryStep → 核对新任务及原截止 → Continue → Worker执行新任务 |
|
| TASK已确认失败,原预算仍有余量 | 读取当前实例/步骤/激活 → RetryStep → 核对新任务及原截止 → Continue → Worker执行新任务 |
|
||||||
|
| POLL请求失败且仍有预算 | RetryStep → 核对同业务轮和剩余尝试 → Continue;结果PENDING才按原间隔开下一轮 |
|
||||||
| POLL次数耗尽,人工取得最终结果 | 核实历史请求均已结束 → ResolveStep(符合固定成功映射的结果) → Continue;原used_rounds不改变 |
|
| POLL次数耗尽,人工取得最终结果 | 核实历史请求均已结束 → ResolveStep(符合固定成功映射的结果) → Continue;原used_rounds不改变 |
|
||||||
| 取消后出现UNKNOWN | ReviewTask → 等待消费 → ResolveStep → Continue → 原取消结清;不创建后继工作 |
|
| 取消后出现UNKNOWN | ReviewTask → 等待消费 → ResolveStep → Continue → 原取消结清;不创建后继工作 |
|
||||||
|
|
||||||
|
|||||||
65
docs/17-workflow-poll-recovery.md
Normal file
65
docs/17-workflow-poll-recovery.md
Normal file
@@ -0,0 +1,65 @@
|
|||||||
|
# Weave 有界查询的人工恢复
|
||||||
|
|
||||||
|
状态:内部 `RetryStep` 已支持 POLL 新激活,保留跨激活的业务轮次、同轮剩余尝试和绝对截止;附属最终信号等待随新激活换代。核查后已确认 PENDING 的显式等待接续、独立 SIGNAL/TIMER 的等待重建仍待实现,公共实例/恢复 HTTP、独立 SDK 和可运行接入示例继续按[开发计划](08-development-plan.md)交付。验证范围见[阶段报告](reports/phase-5-poll-recovery.md)。
|
||||||
|
|
||||||
|
## 1. 业务轮次与请求尝试
|
||||||
|
|
||||||
|
查询成功且业务结果为 PENDING 时,后台按原间隔开启下一业务轮。请求失败、未知或结果不符合契约时,进入核查;人工恢复重查同一业务轮,不能通过换激活获得新的请求预算。
|
||||||
|
|
||||||
|
例如固定定义最多查询 3 轮,每轮最多尝试 3 次:
|
||||||
|
|
||||||
|
| 工作 | 激活 | 业务轮次 | 本任务尝试上限 | 沿用的截止 |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| 原查询请求失败,已用 1 次 | 1 | 1 | 3 | 第一轮原任务截止、原查询总截止 |
|
||||||
|
| 人工恢复同轮 | 2 | 1 | 2 | 同上,人工操作不重新计时 |
|
||||||
|
| 恢复请求成功返回 PENDING,间隔到达 | 2 | 2 | 3 | 新业务轮的任务截止受原查询总截止截短 |
|
||||||
|
| 第二轮请求失败,已用 1 次,再人工恢复 | 3 | 2 | 2 | 第二轮第一次任务的绝对截止、原查询总截止 |
|
||||||
|
|
||||||
|
恢复前核验真实尝试行数、任务计数、原处理器/版本/策略/输入、业务键、互斥键、所有者、连续激活及 `retry_of` 链。每一业务轮的任务总数也不超过原单轮尝试上限,防止反复取消未开始任务而无限制造激活。一次成功查询已返回 PENDING 的轮次不能人工重试;轮次耗尽须核查最终事实或取消。若 UNKNOWN 经 ReviewTask 被核实为成功 PENDING,现有操作不会自动恢复等待;原预算内显式接续后续业务轮的操作仍需实现,不能借同轮 RetryStep 绕过间隔或次数。
|
||||||
|
|
||||||
|
## 2. 恢复前提和事务
|
||||||
|
|
||||||
|
沿用[人工核查约定](16-workflow-recovery.md):显式 `workflows:recover`、最新实例/步骤版本、当前激活、稳定操作键、原因、证据和 `safe_to_retry=true`。只有 NEEDS_REVIEW 实例中待核查、失败或取消的 POLL 可恢复。已有停止意图、实例/查询/原请求截止到达、次数耗尽或历史执行未知时拒绝。
|
||||||
|
|
||||||
|
UNKNOWN 必须先核实全部外部执行结束,经 ReviewTask 保存确定结果,并消费其新版本 Inbox 后再恢复。声明不能替代服务端的历史归属、执行、许可和未消费/隔离来源检查。
|
||||||
|
|
||||||
|
新激活、新 Pulse 任务、同轮记录、预算、最终信号等待、时钟、额度、审计和幂等回执同事务提交。新任务受当前应用及队列配额约束;满额、唯一性冲突或审计失败全部回滚。相同操作键并发重报只返回一个冻结回执,仍核验当前权限。
|
||||||
|
|
||||||
|
恢复后实例仍是 NEEDS_REVIEW,执行门控关闭;继续核查通过后显式 Continue,原 paused 为 true 时还需 RESUME。恢复不会撤销原取消/失败停止目标。
|
||||||
|
|
||||||
|
## 3. 数据与历史
|
||||||
|
|
||||||
|
[迁移 000022](../internal/storage/mysql/migrations/000022_workflow_poll_recovery.up.sql)为 `PollState` 新增:
|
||||||
|
|
||||||
|
| 字段 | 含义 |
|
||||||
|
| --- | --- |
|
||||||
|
| first_round | 当前激活保存的首个业务轮次。人工恢复沿用旧激活的最后业务轮编号 |
|
||||||
|
| first_round_attempts_used | 该轮在此前激活已用的请求尝试数,从原单轮预算扣除 |
|
||||||
|
|
||||||
|
`used_rounds` 仍是从首次激活起的业务轮次编号,不是当前激活行数。当前激活轮次必须完整覆盖 `[first_round, used_rounds]`;正常进入下一轮时继续 `used_rounds+1`。查询总截止基于首次激活的创建时间,当前激活创建时间不成为新起点。同轮的新任务以 `retry_of` 关联前一任务,沿用该轮第一次任务的绝对截止。
|
||||||
|
|
||||||
|
旧激活状态、输出、任务结果、执行尝试、轮次及信号证据保留;旧查询预算关闭在途/后续检查点并保留原结清时间。来源事件仍固定原激活和任务,旧激活结果只记录 STALE_ACTIVATION,不能完成新任务。停止结清核验所有历史任务,轮次归属按各自的 `first_round` 和 `used_rounds` 检查。
|
||||||
|
|
||||||
|
迁移将历史数据补为 `first_round=1`、已用历史尝试为 0,不更改旧轮次、原截止、预算、状态版本及时间。新增两个 Go 字段和 SQL 列均有中文注释。当前共 35 个持久化实体、305 个直接字段声明;无外键、无级联删除。
|
||||||
|
|
||||||
|
升级需停止旧写入节点,再执行正式迁移并启动新版。旧节点不了解跨激活轮次与截止传递,不能在新节点创建恢复激活后继续混跑。
|
||||||
|
|
||||||
|
## 4. 最终信号的换代
|
||||||
|
|
||||||
|
POLL 带最终信号时,恢复创建同契约、同精确关联键、同原截止的新等待,目标激活代次加一。旧已消费信号及旧 winner 保留,不能自动成为新等待赢家。旧代次再次受理的信号保存为 REJECTED / STALE_ACTIVATION;新代次须重新提交有证据的信号。
|
||||||
|
|
||||||
|
当前代次仍有 PENDING 信号时,先消费这些事实,恢复拒绝直接丢弃。已有赢家时核对其受理证明与关联后保留历史。最终信号原截止已过时不重新开始等待。
|
||||||
|
|
||||||
|
这些规则仅覆盖 POLL 附属等待,不表示独立 SIGNAL/TIMER 重建已经开放。
|
||||||
|
|
||||||
|
## 5. 内部调用场景
|
||||||
|
|
||||||
|
| 场景 | 调用顺序 |
|
||||||
|
| --- | --- |
|
||||||
|
| 已知请求失败且仍有预算 | 读取当前实例/步骤 → RetryStep → 核对新任务及剩余预算 → Continue → Worker 执行 → 原查询分类逻辑接续 |
|
||||||
|
| 请求 UNKNOWN | 核实外部执行结束 → ReviewTask → 等待 Inbox 消费 → 读取当前版本 → RetryStep → Continue |
|
||||||
|
| 恢复后收到旧结果 | 内部 Inbox 消费只保存旧激活处置,不推进当前激活 |
|
||||||
|
| 恢复后收到新最终回执 | 用新激活代次发送同契约信号 → 按原受理时间裁决并结清当前查询 |
|
||||||
|
| 原轮次/请求预算或截止耗尽 | 拒绝新查询;按证据 ResolveStep 或继续取消/停止结清 |
|
||||||
|
|
||||||
|
实现入口是[内部恢复服务](../weave/recovery/service.go)和[POLL 存储事务](../internal/storage/mysql/workflow_poll_retry.go)。这些场景尚不是公共 SDK 用法;公共接口、可运行示例和生产拓扑验收仍须交付。
|
||||||
34
docs/reports/phase-5-poll-recovery.md
Normal file
34
docs/reports/phase-5-poll-recovery.md
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
# P5 有界查询跨激活恢复回归
|
||||||
|
|
||||||
|
日期:2026-09-18。范围为内部 POLL 人工重试及附属最终信号等待换代。核查确认 PENDING 后的显式等待接续、独立 SIGNAL/TIMER 等待重建、实例公共 HTTP/SDK、完整场景与 P6–P7 生产验收仍未完成,持续目标保持完整。
|
||||||
|
|
||||||
|
## 实现与边界
|
||||||
|
|
||||||
|
[查询恢复约定](../17-workflow-poll-recovery.md)将请求尝试与业务轮次分开。人工重试保留原业务轮编号和绝对截止,跨激活扣除真实尝试;成功 PENDING 才允许正常开启下一轮。已结束但未开始过的任务也受每轮任务数上限约束。旧结果只能作用于旧激活,旧信号赢家只保留历史,新等待使用新代次和原契约/关联/截止。
|
||||||
|
|
||||||
|
新激活、任务、轮次、预算、等待、时钟、额度、审计与回执同事务。UNKNOWN、待消费或隔离结果、有效尝试和许可、原停止意图、预算及截止仍由恢复规则约束;恢复后需显式 Continue 才解除核查门控。迁移000022只增加两个带中文注释的预算字段,平台仍为35个持久化实体、305个直接持久化字段声明,无外键、无级联。
|
||||||
|
|
||||||
|
## 环境与证据
|
||||||
|
|
||||||
|
本机 darwin/arm64、Go 1.27.1、Docker MySQL 8.4.10;每轮使用脚本创建的一次性隔离容器及数据库。
|
||||||
|
|
||||||
|
| 检查 | 结果 |
|
||||||
|
| --- | --- |
|
||||||
|
| 首批存储专项 | 新增9组查询恢复、原16组人工核查、时钟/信号、v9应用额度升级、新建Schema及v7无外键升级通过,`-race`,64.729秒 |
|
||||||
|
| 扩展专项 | 12组查询恢复、真实v21预算升级、实际Orbit进程通过,`-race`,54.694秒;恢复13.14秒、升级4.46秒、实际进程35.64秒 |
|
||||||
|
| 基础回归 | `make fmt test vet build`通过,退出0;包含独立SDK和两个业务示例模块 |
|
||||||
|
| 完整隔离回归 | 整条脚本通过,退出0;存储全量208.573秒,实际流程进程包37.148秒,Worker SDK 92.526秒,最终SDK进程故障组97.279秒;19个SDK示例与2个独立业务示例全部通过 |
|
||||||
|
| 模型注释 | 使用Go AST逐字段核对:305个带db标签的直接字段声明,缺中文说明0 |
|
||||||
|
| SQL结构和文档 | 实际v22测试库外键0、无COMMENT业务字段0;932个本地链接与53段JSON检查通过 |
|
||||||
|
|
||||||
|
12组专项覆盖:同轮恢复后进入下一业务轮并再次恢复、同轮剩余尝试耗尽、成功PENDING不能绕过轮次上限、十路同键并发及撤权回执拒绝、审计失败全回滚、UNKNOWN先核查与消费、信号新代次和原截止、原请求截止耗尽、尝试计数与真实行数不符、零尝试激活上限、旧winner证据保留及禁止复用、满额/停止意图拒绝。
|
||||||
|
|
||||||
|
实际进程测试确认全部旧控制进程退出后创建同轮新激活,再由两个全新进程竞争接管;保留原请求截止和剩余2次尝试,收到PENDING后只创建一次下一业务轮,并最终成功。该测试累计16个任务、无额外查询或许可泄漏;此前TASK、停止、信号、TIMER等进程恢复用例也同时运行。
|
||||||
|
|
||||||
|
v21升级夹具用于验证迁移:保留旧轮次1/2、原预算、截止、revision及创建/更新时间,补入first_round=1和历史尝试0;非法恢复预算被CHECK拒绝。v9还原夹具同步删除v22新字段后走正式迁移;不修改历史迁移校验和。所有关联继续由服务端事务检查。
|
||||||
|
|
||||||
|
证据实现见[查询恢复专项](../../internal/storage/mysql/workflow_poll_recovery_integration_test.go)、[v21升级专项](../../internal/storage/mysql/workflow_poll_recovery_upgrade_integration_test.go)和[真实进程专项](../../internal/storage/mysql/workflow_runtime_process_integration_test.go)。
|
||||||
|
|
||||||
|
完整复现入口是[隔离MySQL脚本](../../scripts/test-mysql.sh)。缺少一次性DSN或实际二进制而跳过的用例不计为通过。本地双进程和单MySQL故障验证不替代生产复制、切换、备份恢复、容量或RPO/RTO验收。
|
||||||
|
|
||||||
|
完整回归结束后,一次性测试容器均已由脚本清理。最终检查932个本地链接、53段JSON,错误0;`git diff --check`通过。本轮执行逻辑在完整回归开始后未再修改,后续只补充模型说明与文档中的剩余范围。
|
||||||
@@ -64,6 +64,7 @@ func TestIntegrationApplicationResourcesUpgrade(t *testing.T) {
|
|||||||
retryFinish(t, g, finished, "SUCCEEDED", false)
|
retryFinish(t, g, finished, "SUCCEEDED", false)
|
||||||
assertApplicationCounts(t, f, 4, 2, 1)
|
assertApplicationCounts(t, f, 4, 2, 1)
|
||||||
// 还原后续新增表和字段,得到真实 v9 结构;现有任务内容保持不变。
|
// 还原后续新增表和字段,得到真实 v9 结构;现有任务内容保持不变。
|
||||||
|
f.exec(`ALTER TABLE weave_poll_states DROP CHECK ck_poll_recovery_budget, DROP COLUMN first_round, DROP COLUMN first_round_attempts_used`)
|
||||||
f.exec(`ALTER TABLE pulse_tasks DROP INDEX ix_task_workflow_owner`)
|
f.exec(`ALTER TABLE pulse_tasks DROP INDEX ix_task_workflow_owner`)
|
||||||
f.exec(`ALTER TABLE weave_instances DROP CHECK ck_workflow_stop_intent, DROP COLUMN stop_kind, DROP COLUMN stop_requested_at`)
|
f.exec(`ALTER TABLE weave_instances DROP CHECK ck_workflow_stop_intent, DROP COLUMN stop_kind, DROP COLUMN stop_requested_at`)
|
||||||
f.exec(`ALTER TABLE weave_signals DROP CHECK ck_signal_version_sequence, DROP CHECK ck_signal_pending_identity, DROP INDEX ix_signal_expiry, DROP INDEX ix_signal_activation, DROP COLUMN payload_version, DROP COLUMN target_activation_sequence, DROP COLUMN evidence_ref, DROP COLUMN disposition, DROP COLUMN available_at, MODIFY COLUMN type VARCHAR(64) NOT NULL COMMENT '事件或信号的稳定类型标识', MODIFY COLUMN correlation_key VARCHAR(191) NOT NULL COMMENT '外部业务操作关联键,防止旧信号误用于新工作'`)
|
f.exec(`ALTER TABLE weave_signals DROP CHECK ck_signal_version_sequence, DROP CHECK ck_signal_pending_identity, DROP INDEX ix_signal_expiry, DROP INDEX ix_signal_activation, DROP COLUMN payload_version, DROP COLUMN target_activation_sequence, DROP COLUMN evidence_ref, DROP COLUMN disposition, DROP COLUMN available_at, MODIFY COLUMN type VARCHAR(64) NOT NULL COMMENT '事件或信号的稳定类型标识', MODIFY COLUMN correlation_key VARCHAR(191) NOT NULL COMMENT '外部业务操作关联键,防止旧信号误用于新工作'`)
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
-- 查询恢复保留业务轮次编号及同轮历史尝试,不引入外键或重新计算截止。
|
||||||
|
ALTER TABLE weave_poll_states
|
||||||
|
ADD COLUMN first_round INT UNSIGNED NOT NULL DEFAULT 1 COMMENT '本激活保存的首个业务查询轮次;人工恢复沿用原轮次,不从1重新计数',
|
||||||
|
ADD COLUMN first_round_attempts_used INT UNSIGNED NOT NULL DEFAULT 0 COMMENT '首个业务轮次在旧激活中已消耗的执行尝试数;新任务只获得原单轮预算的剩余部分',
|
||||||
|
ADD CONSTRAINT ck_poll_recovery_budget CHECK (first_round BETWEEN 1 AND max_rounds AND first_round_attempts_used < max_attempts_per_round AND ((used_rounds=0 AND first_round=1 AND first_round_attempts_used=0) OR first_round<=used_rounds));
|
||||||
|
|
||||||
|
ALTER TABLE weave_poll_rounds
|
||||||
|
MODIFY COLUMN `round` INT UNSIGNED NOT NULL COMMENT '业务查询轮次编号,从首次激活起连续计数;人工恢复同一轮时保留编号,由激活身份区分对应任务';
|
||||||
37
internal/storage/mysql/workflow_activation_origin.go
Normal file
37
internal/storage/mysql/workflow_activation_origin.go
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
)
|
||||||
|
|
||||||
|
type workflowActivationBase struct {
|
||||||
|
id core.ID
|
||||||
|
previous *core.ID
|
||||||
|
sequence uint32
|
||||||
|
started time.Time
|
||||||
|
input json.RawMessage
|
||||||
|
}
|
||||||
|
|
||||||
|
// workflowActivationOrigin 按同一逻辑步骤的连续激活序号定位首次预算;新激活创建时间不成为新的计时起点。
|
||||||
|
func workflowActivationOrigin(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep) (workflowActivationBase, error) {
|
||||||
|
var base workflowActivationBase
|
||||||
|
var raw []byte
|
||||||
|
err := tx.QueryRowContext(ctx, `SELECT a.sequence,o.id,o.created_at,o.input,p.id FROM weave_activations a
|
||||||
|
JOIN weave_activations o ON o.namespace_id=a.namespace_id AND o.workflow_id=a.workflow_id AND o.step_id=a.step_id AND o.sequence=1
|
||||||
|
LEFT JOIN weave_activations p ON p.namespace_id=a.namespace_id AND p.workflow_id=a.workflow_id AND p.step_id=a.step_id AND p.sequence=a.sequence-1
|
||||||
|
WHERE a.namespace_id=? AND a.workflow_id=? AND a.step_id=? AND a.id=?`, g.view.NamespaceID, g.view.ID, row.view.ID, row.view.ActivationID).Scan(&base.sequence, &base.id, &base.started, &raw, &base.previous)
|
||||||
|
if err != nil {
|
||||||
|
return base, err
|
||||||
|
}
|
||||||
|
base.input = raw
|
||||||
|
if base.sequence == 0 || base.sequence > 10000 || base.sequence == 1 && (row.view.ActivationID == nil || *row.view.ActivationID != base.id || base.previous != nil) || base.sequence > 1 && base.previous == nil || !sameWorkflowJSON(row.input, base.input) {
|
||||||
|
return base, errors.New("workflow activation origin invalid")
|
||||||
|
}
|
||||||
|
return base, nil
|
||||||
|
}
|
||||||
@@ -79,7 +79,7 @@ func (r *WorkflowRuntime) advanceClock(ctx context.Context, tx *sql.Tx, g *workf
|
|||||||
return changed, nil
|
return changed, nil
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return r.advancePoll(ctx, tx, g, row, c, activated, allow, now)
|
return r.advancePoll(ctx, tx, g, row, c, allow, now)
|
||||||
}
|
}
|
||||||
if checkpoint.Kind == "SIGNAL" {
|
if checkpoint.Kind == "SIGNAL" {
|
||||||
return r.advanceSignalClock(ctx, tx, g, row, c, allow, now)
|
return r.advanceSignalClock(ctx, tx, g, row, c, allow, now)
|
||||||
|
|||||||
@@ -92,7 +92,7 @@ func readWorkflowResult(ctx context.Context, tx *sql.Tx, g *workflowGraph, c wor
|
|||||||
if node.Poll != nil {
|
if node.Poll != nil {
|
||||||
taskDefinition = &node.Poll.Task
|
taskDefinition = &node.Poll.Task
|
||||||
var round uint32
|
var round uint32
|
||||||
err = tx.QueryRowContext(ctx, `SELECT r.round FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.round BETWEEN 1 AND p.used_rounds`, c.ns, c.workflow, source.activation, source.task).Scan(&round)
|
err = tx.QueryRowContext(ctx, `SELECT r.round FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.round BETWEEN p.first_round AND p.used_rounds`, c.ns, c.workflow, source.activation, source.task).Scan(&round)
|
||||||
if errors.Is(err, sql.ErrNoRows) || direct != nil {
|
if errors.Is(err, sql.ErrNoRows) || direct != nil {
|
||||||
return nil, workflowSourceError("RESULT_POLL_OWNER")
|
return nil, workflowSourceError("RESULT_POLL_OWNER")
|
||||||
}
|
}
|
||||||
@@ -167,14 +167,13 @@ func (r *WorkflowRuntime) consume(ctx context.Context, tx *sql.Tx, g *workflowGr
|
|||||||
}
|
}
|
||||||
node, _ := g.program.Step(row.view.Key)
|
node, _ := g.program.Step(row.view.Key)
|
||||||
if node.Poll != nil {
|
if node.Poll != nil {
|
||||||
var used, max uint32
|
budget, lastTask, err := readWorkflowPollBudget(ctx, tx, g, row)
|
||||||
var deadline time.Time
|
if err != nil {
|
||||||
var current *core.ID
|
|
||||||
var resolved *time.Time
|
|
||||||
if err := tx.QueryRowContext(ctx, `SELECT used_rounds,max_rounds,deadline_at,current_task_id,resolved_at FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=? FOR UPDATE`, c.ns, c.workflow, source.activation).Scan(&used, &max, &deadline, ¤t, &resolved); err != nil {
|
|
||||||
return false, err
|
return false, err
|
||||||
}
|
}
|
||||||
if resolved != nil || current == nil || *current != source.task {
|
used, max, deadline := budget.UsedRounds, budget.MaxRounds, budget.DeadlineAt
|
||||||
|
current := budget.CurrentTaskID
|
||||||
|
if budget.ResolvedAt != nil || current == nil || *current != source.task || lastTask != source.task {
|
||||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_POLL_ROUND", now)
|
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_POLL_ROUND", now)
|
||||||
}
|
}
|
||||||
if node.Poll.FinalSignal != nil {
|
if node.Poll.FinalSignal != nil {
|
||||||
|
|||||||
@@ -61,7 +61,7 @@ func appendWorkflowResult(ctx context.Context, tx *sql.Tx, ns, eventID, taskID c
|
|||||||
return errors.New("poll activation cannot own a direct task")
|
return errors.New("poll activation cannot own a direct task")
|
||||||
}
|
}
|
||||||
var round uint32
|
var round uint32
|
||||||
err = tx.QueryRowContext(ctx, `SELECT r.round FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.round BETWEEN 1 AND p.used_rounds`, ns, workflow, ownerID, taskID).Scan(&round)
|
err = tx.QueryRowContext(ctx, `SELECT r.round FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.round BETWEEN p.first_round AND p.used_rounds`, ns, workflow, ownerID, taskID).Scan(&round)
|
||||||
if errors.Is(err, sql.ErrNoRows) {
|
if errors.Is(err, sql.ErrNoRows) {
|
||||||
return errors.New("workflow poll task association invalid")
|
return errors.New("workflow poll task association invalid")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -17,36 +17,66 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// readWorkflowPollState 校验当前激活的完整轮次与原预算;时钟和最终信号共用相同检查。
|
// readWorkflowPollState 校验当前激活的完整轮次与原预算;时钟和最终信号共用相同检查。
|
||||||
func readWorkflowPollState(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, activated time.Time) (weave.PollState, core.ID, error) {
|
func readWorkflowPollState(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep) (weave.PollState, core.ID, error) {
|
||||||
|
p, last, err := readWorkflowPollBudget(ctx, tx, g, row)
|
||||||
|
if err == nil && p.ResolvedAt != nil {
|
||||||
|
err = errors.New("unsettled poll step has resolved budget")
|
||||||
|
}
|
||||||
|
return p, last, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// readWorkflowPollBudget 也允许读取已结清预算,供有证据的人工恢复核对;不重新计时。
|
||||||
|
func readWorkflowPollBudget(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep) (weave.PollState, core.ID, error) {
|
||||||
var p weave.PollState
|
var p weave.PollState
|
||||||
err := tx.QueryRowContext(ctx, `SELECT id,revision,max_rounds,used_rounds,max_attempts_per_round,interval_ms,deadline_at,next_poll_at,current_task_id,resolved_at FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=? FOR UPDATE`, g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(&p.ID, &p.Revision, &p.MaxRounds, &p.UsedRounds, &p.MaxAttemptsPerRound, &p.IntervalMS, &p.DeadlineAt, &p.NextPollAt, &p.CurrentTaskID, &p.ResolvedAt)
|
err := tx.QueryRowContext(ctx, `SELECT id,revision,max_rounds,used_rounds,first_round,first_round_attempts_used,max_attempts_per_round,interval_ms,deadline_at,next_poll_at,current_task_id,resolved_at FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=? FOR UPDATE`, g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(&p.ID, &p.Revision, &p.MaxRounds, &p.UsedRounds, &p.FirstRound, &p.FirstRoundAttemptsUsed, &p.MaxAttemptsPerRound, &p.IntervalMS, &p.DeadlineAt, &p.NextPollAt, &p.CurrentTaskID, &p.ResolvedAt)
|
||||||
|
if err != nil {
|
||||||
|
return p, "", err
|
||||||
|
}
|
||||||
|
base, err := workflowActivationOrigin(ctx, tx, g, row)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return p, "", err
|
return p, "", err
|
||||||
}
|
}
|
||||||
node, _ := g.program.Step(row.view.Key)
|
node, _ := g.program.Step(row.view.Key)
|
||||||
policy := node.Poll
|
policy := node.Poll
|
||||||
if policy == nil || p.MaxRounds != policy.MaxRounds || p.MaxAttemptsPerRound != policy.Task.Retry.MaxAttempts || p.IntervalMS != policy.IntervalMS || p.UsedRounds < 1 || !p.DeadlineAt.Equal(minimumTime(g.view.DeadlineAt, activated.Add(time.Duration(policy.TimeoutMS)*time.Millisecond))) {
|
if policy == nil || p.MaxRounds != policy.MaxRounds || p.MaxAttemptsPerRound != policy.Task.Retry.MaxAttempts || p.IntervalMS != policy.IntervalMS || p.UsedRounds < 1 || p.UsedRounds > p.MaxRounds || p.FirstRound < 1 || p.FirstRound > p.UsedRounds || p.FirstRoundAttemptsUsed >= p.MaxAttemptsPerRound || !p.DeadlineAt.Equal(minimumTime(g.view.DeadlineAt, base.started.Add(time.Duration(policy.TimeoutMS)*time.Millisecond))) {
|
||||||
return p, "", errors.New("workflow poll policy mismatch")
|
return p, "", errors.New("workflow poll policy mismatch")
|
||||||
}
|
}
|
||||||
if p.ResolvedAt != nil {
|
if base.sequence == 1 && (p.FirstRound != 1 || p.FirstRoundAttemptsUsed != 0) {
|
||||||
return p, "", errors.New("unsettled poll step has resolved budget")
|
return p, "", errors.New("initial poll recovery budget invalid")
|
||||||
|
}
|
||||||
|
if base.previous != nil {
|
||||||
|
var used uint32
|
||||||
|
var deadline time.Time
|
||||||
|
var settled *time.Time
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT used_rounds,deadline_at,resolved_at FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=?`, g.view.NamespaceID, g.view.ID, base.previous).Scan(&used, &deadline, &settled); err != nil {
|
||||||
|
return p, "", err
|
||||||
|
}
|
||||||
|
if used != p.FirstRound || !deadline.Equal(p.DeadlineAt) || settled == nil {
|
||||||
|
return p, "", errors.New("poll recovery parent budget invalid")
|
||||||
|
}
|
||||||
|
if _, err := readWorkflowPollRoundHistory(ctx, tx, g, row, p, p.FirstRound); err != nil {
|
||||||
|
return p, "", err
|
||||||
|
}
|
||||||
}
|
}
|
||||||
var count, minRound, maxRound uint32
|
var count, minRound, maxRound uint32
|
||||||
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*),COALESCE(MIN(`round`),0),COALESCE(MAX(`round`),0) FROM weave_poll_rounds WHERE namespace_id=? AND workflow_id=? AND activation_id=?", g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(&count, &minRound, &maxRound); err != nil {
|
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*),COALESCE(MIN(`round`),0),COALESCE(MAX(`round`),0) FROM weave_poll_rounds WHERE namespace_id=? AND workflow_id=? AND activation_id=?", g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(&count, &minRound, &maxRound); err != nil {
|
||||||
return p, "", err
|
return p, "", err
|
||||||
}
|
}
|
||||||
if count != p.UsedRounds || minRound != 1 || maxRound != p.UsedRounds {
|
if count != p.UsedRounds-p.FirstRound+1 || minRound != p.FirstRound || maxRound != p.UsedRounds {
|
||||||
return p, "", errors.New("workflow poll rounds incomplete")
|
return p, "", errors.New("workflow poll rounds incomplete")
|
||||||
}
|
}
|
||||||
var lastTask core.ID
|
var lastTask core.ID
|
||||||
if err := tx.QueryRowContext(ctx, "SELECT task_id FROM weave_poll_rounds WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND `round`=?", g.view.NamespaceID, g.view.ID, row.view.ActivationID, p.UsedRounds).Scan(&lastTask); err != nil {
|
if err := tx.QueryRowContext(ctx, "SELECT task_id FROM weave_poll_rounds WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND `round`=?", g.view.NamespaceID, g.view.ID, row.view.ActivationID, p.UsedRounds).Scan(&lastTask); err != nil {
|
||||||
return p, "", err
|
return p, "", err
|
||||||
}
|
}
|
||||||
|
if p.CurrentTaskID != nil && (*p.CurrentTaskID != lastTask || p.NextPollAt != nil) || p.ResolvedAt != nil && (p.CurrentTaskID != nil || p.NextPollAt != nil) {
|
||||||
|
return p, "", errors.New("poll budget execution pointers invalid")
|
||||||
|
}
|
||||||
return p, lastTask, nil
|
return p, lastTask, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r *WorkflowRuntime) advancePoll(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, c workflowCandidate, activated time.Time, allow bool, now time.Time) (bool, error) {
|
func (r *WorkflowRuntime) advancePoll(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, c workflowCandidate, allow bool, now time.Time) (bool, error) {
|
||||||
p, lastTask, err := readWorkflowPollState(ctx, tx, g, row, activated)
|
p, lastTask, err := readWorkflowPollState(ctx, tx, g, row)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, err
|
return false, err
|
||||||
}
|
}
|
||||||
@@ -196,11 +226,17 @@ func validatePollTaskOwner(ctx context.Context, tx *sql.Tx, g *workflowGraph, ro
|
|||||||
|
|
||||||
// pollTaskInput 复用首轮已经解析并冻结的输入、业务键及互斥键;不重算依赖可能已改变的表达式。
|
// pollTaskInput 复用首轮已经解析并冻结的输入、业务键及互斥键;不重算依赖可能已改变的表达式。
|
||||||
func pollTaskInput(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, spec definition.Task) (service.SubmitInput, error) {
|
func pollTaskInput(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, spec definition.Task) (service.SubmitInput, error) {
|
||||||
var first core.ID
|
origin, err := workflowActivationOrigin(ctx, tx, g, row)
|
||||||
if err := tx.QueryRowContext(ctx, "SELECT task_id FROM weave_poll_rounds WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND `round`=1", g.view.NamespaceID, g.view.ID, row.view.ActivationID).Scan(&first); err != nil {
|
if err != nil {
|
||||||
return service.SubmitInput{}, err
|
return service.SubmitInput{}, err
|
||||||
}
|
}
|
||||||
if err := validatePollTaskOwner(ctx, tx, g, row, first); err != nil {
|
originRow := *row
|
||||||
|
originRow.view.ActivationID = &origin.id
|
||||||
|
var first core.ID
|
||||||
|
if err := tx.QueryRowContext(ctx, "SELECT task_id FROM weave_poll_rounds WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND `round`=1", g.view.NamespaceID, g.view.ID, origin.id).Scan(&first); err != nil {
|
||||||
|
return service.SubmitInput{}, err
|
||||||
|
}
|
||||||
|
if err := validatePollTaskOwner(ctx, tx, g, &originRow, first); err != nil {
|
||||||
return service.SubmitInput{}, err
|
return service.SubmitInput{}, err
|
||||||
}
|
}
|
||||||
base, err := readTask(ctx, tx, g.view.NamespaceID, first, false)
|
base, err := readTask(ctx, tx, g.view.NamespaceID, first, false)
|
||||||
|
|||||||
95
internal/storage/mysql/workflow_poll_history.go
Normal file
95
internal/storage/mysql/workflow_poll_history.go
Normal file
@@ -0,0 +1,95 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||||
|
)
|
||||||
|
|
||||||
|
type workflowPollRoundHistory struct {
|
||||||
|
first, last service.TaskView
|
||||||
|
used, count uint32
|
||||||
|
}
|
||||||
|
|
||||||
|
// readWorkflowPollRoundHistory 跨激活核验同一业务轮的请求链,计数来自真实尝试行;新业务轮不继承上一轮尝试。
|
||||||
|
func readWorkflowPollRoundHistory(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, p weave.PollState, round uint32) (workflowPollRoundHistory, error) {
|
||||||
|
var out workflowPollRoundHistory
|
||||||
|
node, ok := g.program.Step(row.view.Key)
|
||||||
|
if !ok || node.Poll == nil {
|
||||||
|
return out, errors.New("poll history requires fixed poll definition")
|
||||||
|
}
|
||||||
|
frozen, err := pollTaskInput(ctx, tx, g, row, node.Poll.Task)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
rows, err := tx.QueryContext(ctx, "SELECT a.id,a.sequence,a.input,r.task_id,p.first_round,p.first_round_attempts_used,p.used_rounds,p.deadline_at,(SELECT COUNT(*) FROM pulse_attempts x WHERE x.namespace_id=r.namespace_id AND x.task_id=r.task_id) FROM weave_activations a JOIN weave_poll_states p ON p.namespace_id=a.namespace_id AND p.workflow_id=a.workflow_id AND p.activation_id=a.id JOIN weave_poll_rounds r ON r.namespace_id=a.namespace_id AND r.workflow_id=a.workflow_id AND r.activation_id=a.id AND r.`round`=? WHERE a.namespace_id=? AND a.workflow_id=? AND a.step_id=? ORDER BY a.sequence LIMIT 101", round, g.view.NamespaceID, g.view.ID, row.view.ID)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
type prior struct {
|
||||||
|
activation, task core.ID
|
||||||
|
sequence, first, carried, used, attempts uint32
|
||||||
|
deadline time.Time
|
||||||
|
input []byte
|
||||||
|
}
|
||||||
|
var history []prior
|
||||||
|
for rows.Next() {
|
||||||
|
var v prior
|
||||||
|
if err := rows.Scan(&v.activation, &v.sequence, &v.input, &v.task, &v.first, &v.carried, &v.used, &v.deadline, &v.attempts); err != nil {
|
||||||
|
rows.Close()
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
history = append(history, v)
|
||||||
|
}
|
||||||
|
err = rows.Err()
|
||||||
|
rows.Close()
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
if len(history) == 0 || len(history) > int(p.MaxAttemptsPerRound) || history[len(history)-1].activation != *row.view.ActivationID {
|
||||||
|
return out, errors.New("poll round recovery history incomplete")
|
||||||
|
}
|
||||||
|
for i, prior := range history {
|
||||||
|
if prior.first > round || prior.used < round || !prior.deadline.Equal(p.DeadlineAt) || !sameWorkflowJSON(prior.input, row.input) || i > 0 && (prior.sequence != history[i-1].sequence+1 || prior.first != round) || i < len(history)-1 && prior.used != round || prior.first == round && prior.carried != out.used {
|
||||||
|
return out, errors.New("poll round recovery budget mismatch")
|
||||||
|
}
|
||||||
|
task, err := readTask(ctx, tx, g.view.NamespaceID, prior.task, false)
|
||||||
|
if err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
owner := *row
|
||||||
|
owner.view.ActivationID = &prior.activation
|
||||||
|
if err := validatePollTaskOwner(ctx, tx, g, &owner, task.ID); err != nil {
|
||||||
|
return out, err
|
||||||
|
}
|
||||||
|
if i == 0 {
|
||||||
|
out.first = task
|
||||||
|
if !task.DeadlineAt.Equal(minimumTime(task.CreatedAt.Add(time.Duration(frozen.TaskTimeoutMS)*time.Millisecond), p.DeadlineAt)) {
|
||||||
|
return out, errors.New("poll round original deadline invalid")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
expected := frozen.Retry
|
||||||
|
if out.used >= expected.MaxAttempts {
|
||||||
|
return out, errors.New("poll round history exceeds attempts")
|
||||||
|
}
|
||||||
|
expected.MaxAttempts -= out.used
|
||||||
|
if task.Retry != expected || task.AttemptCount > task.Retry.MaxAttempts || prior.attempts != task.AttemptCount || task.QueueID != frozen.QueueID || task.Handler != frozen.Handler || task.HandlerVersion != frozen.HandlerVersion || task.PayloadVersion != frozen.PayloadVersion || task.Priority != frozen.Priority || task.RecoverySafety != frozen.RecoverySafety || !sameWorkflowJSON(task.Payload, frozen.Payload) || task.BusinessKey != frozen.BusinessKey || task.ConcurrencyKey != frozen.ConcurrencyKey || !task.DeadlineAt.Equal(out.first.DeadlineAt) || task.StartDeadlineAt != nil {
|
||||||
|
return out, errors.New("poll round frozen task mismatch")
|
||||||
|
}
|
||||||
|
if i == 0 && task.RetryOf != nil || i > 0 && (task.RetryOf == nil || *task.RetryOf != out.last.ID) {
|
||||||
|
return out, errors.New("poll round task ancestry invalid")
|
||||||
|
}
|
||||||
|
if i < len(history)-1 && (!task.State.Terminal() || task.EffectUncertain) {
|
||||||
|
return out, errors.New("poll round previous execution unresolved")
|
||||||
|
}
|
||||||
|
out.last = task
|
||||||
|
out.used += task.AttemptCount
|
||||||
|
}
|
||||||
|
out.count = uint32(len(history))
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,383 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"os"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/worker"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
|
)
|
||||||
|
|
||||||
|
func recoveryPollSpec(queue core.ID, final bool) definition.Specification {
|
||||||
|
s := pollingSpec(queue)
|
||||||
|
if final {
|
||||||
|
s = finalSignalSpec(queue)
|
||||||
|
}
|
||||||
|
p := s.Body.Steps[0].Poll
|
||||||
|
p.TimeoutMS = 60000
|
||||||
|
p.Task.TimeoutMS = 30000
|
||||||
|
p.Task.Retry.MaxAttempts = 3
|
||||||
|
p.Task.Retry.InitialDelayMS = 1000
|
||||||
|
p.Task.Retry.MaxDelayMS = 1000
|
||||||
|
if final {
|
||||||
|
p.FinalSignal.TimeoutMS = p.TimeoutMS
|
||||||
|
}
|
||||||
|
return s
|
||||||
|
}
|
||||||
|
func retryPollStep(t *testing.T, f *executionFixture, id core.ID) recovery.Result {
|
||||||
|
t.Helper()
|
||||||
|
out, err := recovery.New(NewWorkflowRecoveryStore(f.db)).RetryStep(f.ctx, f.token, id, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, id, "query"), SafeToRetry: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
func failedPoll(t *testing.T, ctx context.Context, db *sql.DB, final bool) (*executionFixture, instance.View, worker.AttemptRef) {
|
||||||
|
t.Helper()
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
v := runtimeStart(t, f, recoveryPollSpec(f.resources.QueueID, final))
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
return f, v, ref
|
||||||
|
}
|
||||||
|
func TestIntegrationWorkflowPollRecovery(t *testing.T) {
|
||||||
|
dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN")
|
||||||
|
if dsn == "" {
|
||||||
|
t.Skip("requires disposable MySQL")
|
||||||
|
}
|
||||||
|
ctx, stop := context.WithTimeout(context.Background(), 150*time.Second)
|
||||||
|
defer stop()
|
||||||
|
db, err := Open(ctx, dsn, 32, 8)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
if err := Migrate(ctx, db); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
api := recovery.New(NewWorkflowRecoveryStore(db))
|
||||||
|
t.Run("retry_same_round_then_next_round_and_retry_again", func(t *testing.T) {
|
||||||
|
f, v, ref := failedPoll(t, ctx, db, false)
|
||||||
|
first := f.get(ref.TaskID)
|
||||||
|
original := recoveryStep(t, f, v.ID, "query")
|
||||||
|
out := retryPollStep(t, f, v.ID)
|
||||||
|
if out.Task.Retry.MaxAttempts != 2 || !out.Task.DeadlineAt.Equal(first.DeadlineAt) || *out.Task.RetryOf != first.ID || *out.Step.ActivationID == original.ActivationID || out.Step.TaskID != nil || !sameWorkflowJSON(out.Task.Payload, first.Payload) {
|
||||||
|
t.Fatal("recovery reset original identity, task deadline or budget")
|
||||||
|
}
|
||||||
|
if f.count("weave_poll_states", "first_round=1 AND first_round_attempts_used=1 AND used_rounds=1 AND resolved_at IS NULL") != 1 || f.count("weave_activations", "id=? AND state='NEEDS_REVIEW'", original.ActivationID) != 1 {
|
||||||
|
t.Fatal("old activation or round changed")
|
||||||
|
}
|
||||||
|
f.exec(`UPDATE weave_inbox SET state='PENDING',processed_at=NULL WHERE namespace_id=? AND source_task_id=?`, f.resources.NamespaceID, first.ID)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), first.ID)
|
||||||
|
if f.count("weave_inbox", "source_task_id=? AND error_code='STALE_ACTIVATION'", first.ID) != 1 {
|
||||||
|
t.Fatal("old result applied to new activation")
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
finishPoll(t, f, ref, "PENDING")
|
||||||
|
c := clockCandidate(t, f, v.ID, "query")
|
||||||
|
waitClockDue(t, f, c)
|
||||||
|
runClock(t, f, c)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
second := f.get(ref.TaskID)
|
||||||
|
if second.Retry.MaxAttempts != 3 || second.RetryOf != nil || second.BusinessKey != first.BusinessKey || second.ConcurrencyKey != first.ConcurrencyKey {
|
||||||
|
t.Fatal("next business round did not receive independent original policy")
|
||||||
|
}
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
out = retryPollStep(t, f, v.ID)
|
||||||
|
if out.Task.Retry.MaxAttempts != 2 || !out.Task.DeadlineAt.Equal(second.DeadlineAt) || *out.Task.RetryOf != second.ID || f.count("weave_poll_states", "first_round=2 AND used_rounds=2 AND first_round_attempts_used=1") != 1 {
|
||||||
|
t.Fatal("second round recovery reset logical round")
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
finishPoll(t, f, ref, "PENDING")
|
||||||
|
c = clockCandidate(t, f, v.ID, "query")
|
||||||
|
waitClockDue(t, f, c)
|
||||||
|
runClock(t, f, c)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
finishPoll(t, f, ref, "DONE")
|
||||||
|
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||||
|
if f.count("pulse_tasks", "1=1") != 5 || f.count("weave_poll_rounds", "`round`=2") != 2 || f.count("weave_poll_states", "first_round=2 AND used_rounds=3 AND resolved_at IS NOT NULL") != 1 || f.used() != 0 {
|
||||||
|
t.Fatal("extra query, lost round or leaked permits")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("repeated_same_round_exhausts_original_attempts", func(t *testing.T) {
|
||||||
|
f, v, _ := failedPoll(t, ctx, db, false)
|
||||||
|
for remaining := uint32(2); remaining > 0; remaining-- {
|
||||||
|
out := retryPollStep(t, f, v.ID)
|
||||||
|
if out.Task.Retry.MaxAttempts != remaining {
|
||||||
|
t.Fatal("attempt budget reset")
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
}
|
||||||
|
_, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" || f.count("pulse_tasks", "1=1") != 3 || f.count("weave_poll_rounds", "`round`=1") != 3 {
|
||||||
|
t.Fatal("original round attempts exceeded", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("successful_pending_cannot_bypass_business_round_limit", func(t *testing.T) {
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
spec := recoveryPollSpec(f.resources.QueueID, false)
|
||||||
|
spec.Body.Steps[0].Poll.MaxRounds = 1
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
finishPoll(t, f, ref, "PENDING")
|
||||||
|
_, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" || f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("PENDING restarted original business round", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("same_key_concurrency_receipt_and_permission", func(t *testing.T) {
|
||||||
|
f, v, _ := failedPoll(t, ctx, db, false)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true}
|
||||||
|
key := string(executionID(t))
|
||||||
|
var wg sync.WaitGroup
|
||||||
|
results := make(chan recovery.Result, 10)
|
||||||
|
errs := make(chan error, 10)
|
||||||
|
for i := 0; i < 10; i++ {
|
||||||
|
wg.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer wg.Done()
|
||||||
|
out, err := api.RetryStep(ctx, f.token, v.ID, key, in)
|
||||||
|
results <- out
|
||||||
|
errs <- err
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
wg.Wait()
|
||||||
|
close(results)
|
||||||
|
close(errs)
|
||||||
|
for err := range errs {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var task core.ID
|
||||||
|
fresh := 0
|
||||||
|
for out := range results {
|
||||||
|
if !out.Deduplicated {
|
||||||
|
fresh++
|
||||||
|
}
|
||||||
|
if task != "" && task != out.Task.ID {
|
||||||
|
t.Fatal("duplicate task")
|
||||||
|
}
|
||||||
|
task = out.Task.ID
|
||||||
|
}
|
||||||
|
if fresh != 1 || f.count("pulse_tasks", "1=1") != 2 || f.count("orbit_audit_events", "action='workflow.step.retry'") != 1 {
|
||||||
|
t.Fatal("duplicate recovery effects")
|
||||||
|
}
|
||||||
|
stale := in
|
||||||
|
stale.ExpectedRevision = recoveryDecision(t, f, v.ID).ExpectedRevision
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), stale); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("old activation accepted", err)
|
||||||
|
}
|
||||||
|
f.exec(`UPDATE orbit_credentials SET permissions=JSON_SET(permissions,'$.queues',JSON_ARRAY(?)) WHERE namespace_id=? AND id=?`, string(executionID(t)), f.resources.NamespaceID, f.resources.CredentialID)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, key, in); fault.Code(err) != "NOT_FOUND" {
|
||||||
|
t.Fatal("receipt bypassed current permissions", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("audit_failure_rolls_back_and_same_key_retries", func(t *testing.T) {
|
||||||
|
f, v, _ := failedPoll(t, ctx, db, false)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true}
|
||||||
|
key := string(executionID(t))
|
||||||
|
trigger := "reject_poll_recovery_" + string(executionID(t))
|
||||||
|
f.exec("CREATE TRIGGER " + trigger + " BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='controlled recovery audit failure'")
|
||||||
|
_, err := api.RetryStep(ctx, f.token, v.ID, key, in)
|
||||||
|
f.exec("DROP TRIGGER " + trigger)
|
||||||
|
if err == nil || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_poll_states", "1=1") != 1 || f.count("weave_pending_advances", "1=1") != 0 || recoveryDecision(t, f, v.ID).ExpectedRevision != in.ExpectedRevision {
|
||||||
|
t.Fatal("partial recovery committed", err)
|
||||||
|
}
|
||||||
|
assertApplicationCounts(t, f, 0, 0, 0)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, key, in); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
assertApplicationCounts(t, f, 1, 0, 0)
|
||||||
|
})
|
||||||
|
t.Run("unknown_requires_execution_review_before_retry", func(t *testing.T) {
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
v := runtimeStart(t, f, recoveryPollSpec(f.resources.QueueID, false))
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "UNKNOWN", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
_, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("unknown execution retried", err)
|
||||||
|
}
|
||||||
|
reviewWorkflowTask(t, f, v.ID, ref.TaskID, "FAILED")
|
||||||
|
_, err = api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("pending review result skipped", err)
|
||||||
|
}
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
out := retryPollStep(t, f, v.ID)
|
||||||
|
if out.Task.Retry.MaxAttempts != 2 || f.count("pulse_attempts", "state='UNKNOWN'") != 1 {
|
||||||
|
t.Fatal("unknown evidence lost or budget reset")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("final_signal_new_generation_keeps_deadline_and_old_signals_stale", func(t *testing.T) {
|
||||||
|
f, v, _ := failedPoll(t, ctx, db, true)
|
||||||
|
grantWorkflowSignal(f)
|
||||||
|
var deadline time.Time
|
||||||
|
if err := db.QueryRowContext(ctx, `SELECT deadline_at FROM weave_waits WHERE namespace_id=?`, f.resources.NamespaceID).Scan(&deadline); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
out := retryPollStep(t, f, v.ID)
|
||||||
|
if f.count("weave_waits", "state='WAITING' AND deadline_at=? AND winner_signal_id IS NULL", deadline) != 1 {
|
||||||
|
t.Fatal("new wait reset deadline or reused winner")
|
||||||
|
}
|
||||||
|
old := sendSignal(t, f, v, pollSignalInput(v))
|
||||||
|
if old.Signal.Disposition != "STALE_ACTIVATION" || old.Signal.State != "REJECTED" {
|
||||||
|
t.Fatal("old generation accepted", old)
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
if ref.TaskID != out.Task.ID {
|
||||||
|
t.Fatal("wrong query activated")
|
||||||
|
}
|
||||||
|
finishPoll(t, f, ref, "PENDING")
|
||||||
|
in := pollSignalInput(v)
|
||||||
|
in.TargetActivationSequence = 2
|
||||||
|
sendSignal(t, f, v, in)
|
||||||
|
runClock(t, f, clockCandidate(t, f, v.ID, "query"))
|
||||||
|
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||||
|
if f.count("pulse_tasks", "1=1") != 2 || f.count("weave_waits", "state='SATISFIED' AND winner_signal_id IS NOT NULL") != 1 {
|
||||||
|
t.Fatal("final signal lost or extra round issued")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("original_task_deadline_not_extended", func(t *testing.T) {
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
spec := recoveryPollSpec(f.resources.QueueID, false)
|
||||||
|
spec.Body.Steps[0].Poll.Task.TimeoutMS = 1200
|
||||||
|
spec.Body.Steps[0].Poll.Task.Retry.AttemptTimeoutMS = 1000
|
||||||
|
v := runtimeStart(t, f, spec)
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
time.Sleep(time.Until(f.get(ref.TaskID).DeadlineAt.Add(20 * time.Millisecond)))
|
||||||
|
_, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true})
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" || f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("original request deadline restarted", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("corrupt_prior_attempt_count_is_rejected", func(t *testing.T) {
|
||||||
|
f, v, ref := failedPoll(t, ctx, db, false)
|
||||||
|
f.exec(`UPDATE pulse_tasks SET attempt_count=0 WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||||
|
_, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true})
|
||||||
|
f.exec(`UPDATE pulse_tasks SET attempt_count=1 WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||||
|
if err == nil || f.count("pulse_tasks", "1=1") != 1 {
|
||||||
|
t.Fatal("attempt row proof bypassed")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("zero_attempt_recoveries_are_also_bounded", func(t *testing.T) {
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
v := runtimeStart(t, f, recoveryPollSpec(f.resources.QueueID, false))
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
for i := 0; i < 3; i++ {
|
||||||
|
var task core.ID
|
||||||
|
if err := db.QueryRowContext(ctx, `SELECT current_task_id FROM weave_poll_states WHERE namespace_id=? AND resolved_at IS NULL`, f.resources.NamespaceID).Scan(&task); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// 在隔离夹具中直接使用所有者内部取消原语,制造未开始即已知取消的完整事件;公共任务入口仍禁止绕过流程。
|
||||||
|
err := WithTx(ctx, db, func(tx *sql.Tx) error {
|
||||||
|
if _, _, err := lockQueue(ctx, tx, f.resources.NamespaceID, f.resources.QueueID); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
current, err := readExecutionTask(ctx, tx, f.resources.NamespaceID, task)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
now, err := dbNow(ctx, tx)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
_, err = cancelExecutionTask(ctx, tx, f.resources.NamespaceID, current, now, "TEST_OWNER_CANCEL_BEFORE_START")
|
||||||
|
return err
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), task)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true}
|
||||||
|
out, err := api.RetryStep(ctx, f.token, v.ID, string(executionID(t)), in)
|
||||||
|
if i == 2 {
|
||||||
|
if fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("unlimited zero-attempt activations", err)
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if out.Task.Retry.MaxAttempts != 3 {
|
||||||
|
t.Fatal("unstarted recovery charged execution attempt")
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if f.count("pulse_tasks", "1=1") != 3 || f.count("pulse_attempts", "1=1") != 0 {
|
||||||
|
t.Fatal("zero-attempt bound failed")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
t.Run("old_final_signal_winner_is_history_not_new_winner", func(t *testing.T) {
|
||||||
|
f := newClockFixture(t, ctx, db)
|
||||||
|
v := runtimeStart(t, f, recoveryPollSpec(f.resources.QueueID, true))
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
grantWorkflowSignal(f)
|
||||||
|
ref := retryStart(t, f)
|
||||||
|
oldSignal := sendSignal(t, f, v, pollSignalInput(v))
|
||||||
|
runClock(t, f, clockCandidate(t, f, v.ID, "query"))
|
||||||
|
finishQueryOnly(t, f, ref, "FAILED")
|
||||||
|
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||||
|
retryPollStep(t, f, v.ID)
|
||||||
|
if f.count("weave_signals", "id=? AND state='CONSUMED'", oldSignal.Signal.ID) != 1 || f.count("weave_waits", "state='WAITING' AND winner_signal_id IS NULL") != 1 || f.count("weave_waits", "winner_signal_id=?", oldSignal.Signal.ID) != 1 {
|
||||||
|
t.Fatal("old winner reused or evidence lost")
|
||||||
|
}
|
||||||
|
continueWorkflow(t, f, v.ID)
|
||||||
|
f.exec(`UPDATE weave_pending_advances SET available_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND workflow_id=?`, f.resources.NamespaceID, v.ID)
|
||||||
|
runClock(t, f, clockCandidate(t, f, v.ID, "query"))
|
||||||
|
if f.count("weave_steps", "workflow_id=? AND kind='POLL' AND state='ACTIVE'", v.ID) != 1 || f.count("pulse_tasks", "state='WAITING'") != 1 {
|
||||||
|
t.Fatal("old signal completed or cancelled new generation")
|
||||||
|
}
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
finishPoll(t, f, ref, "DONE")
|
||||||
|
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||||
|
})
|
||||||
|
t.Run("queue_full_and_cancel_intent_do_not_create_recovery", func(t *testing.T) {
|
||||||
|
f, v, _ := failedPoll(t, ctx, db, false)
|
||||||
|
in := recovery.RetryStepInput{StepDecision: recoveryStep(t, f, v.ID, "query"), SafeToRetry: true}
|
||||||
|
key := string(executionID(t))
|
||||||
|
f.exec(`UPDATE pulse_queues SET max_unfinished_tasks=1 WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, f.resources.QueueID)
|
||||||
|
blocker := f.submit(1)
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, key, in); fault.Code(err) != "QUEUE_FULL" {
|
||||||
|
t.Fatal("recovery bypassed capacity", err)
|
||||||
|
}
|
||||||
|
if f.count("weave_poll_states", "1=1") != 1 || recoveryDecision(t, f, v.ID).ExpectedRevision != in.ExpectedRevision {
|
||||||
|
t.Fatal("full queue left partial recovery")
|
||||||
|
}
|
||||||
|
if _, err := f.tasks.CancelTask(ctx, f.token, blocker.ID, string(executionID(t)), service.CancelInput{ExpectedRevision: blocker.Revision, Reason: "release test capacity"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
applyControl(t, f, runtimeView(t, f, v.ID, "NEEDS_REVIEW"), "CANCEL")
|
||||||
|
in.StepDecision = recoveryStep(t, f, v.ID, "query")
|
||||||
|
if _, err := api.RetryStep(ctx, f.token, v.ID, key, in); fault.Code(err) != "STATE_CONFLICT" {
|
||||||
|
t.Fatal("recovery revoked stop intent", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
}
|
||||||
@@ -0,0 +1,117 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
driver "github.com/go-sql-driver/mysql"
|
||||||
|
)
|
||||||
|
|
||||||
|
// 用真实v21结构验证轮次预算升级;数据夹具用于验证迁移保留,不冒充运行器实例。
|
||||||
|
func TestIntegrationWorkflowPollRecoveryUpgrade(t *testing.T) {
|
||||||
|
dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN")
|
||||||
|
if dsn == "" {
|
||||||
|
t.Skip("requires disposable MySQL")
|
||||||
|
}
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
admin, err := Open(ctx, dsn, 2, 1)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer admin.Close()
|
||||||
|
name := "orbit_poll_recovery_upgrade_" + string(executionID(t))
|
||||||
|
if _, err := admin.ExecContext(ctx, "CREATE DATABASE `"+name+"`"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer func() {
|
||||||
|
cleanup, stop := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
defer stop()
|
||||||
|
if _, err := admin.ExecContext(cleanup, "DROP DATABASE `"+name+"`"); err != nil {
|
||||||
|
t.Error(err)
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
cfg, err := driver.ParseDSN(dsn)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal("invalid test DSN")
|
||||||
|
}
|
||||||
|
cfg.DBName = name
|
||||||
|
db, err := Open(ctx, cfg.FormatDSN(), 4, 2)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer db.Close()
|
||||||
|
exec := func(q string, args ...any) {
|
||||||
|
t.Helper()
|
||||||
|
if _, err := db.ExecContext(ctx, q, args...); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
exec(`CREATE TABLE orbit_schema_migrations (version BIGINT UNSIGNED PRIMARY KEY,name VARCHAR(191) NOT NULL,checksum CHAR(64) CHARACTER SET ascii COLLATE ascii_bin NOT NULL,dirty BOOLEAN NOT NULL,applied_at DATETIME(6) NULL) ENGINE=InnoDB`)
|
||||||
|
migrations, err := Migrations()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, m := range migrations {
|
||||||
|
if m.Version > 21 {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
statements, err := splitSQL(m.SQL)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, q := range statements {
|
||||||
|
exec(q)
|
||||||
|
}
|
||||||
|
exec(`INSERT INTO orbit_schema_migrations(version,name,checksum,dirty,applied_at) VALUES (?,?,?,FALSE,UTC_TIMESTAMP(6))`, m.Version, m.Name, m.Checksum)
|
||||||
|
}
|
||||||
|
|
||||||
|
var now time.Time
|
||||||
|
if err := db.QueryRowContext(ctx, `SELECT UTC_TIMESTAMP(6)`).Scan(&now); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ns, workflow, activation, id, task := executionID(t), executionID(t), executionID(t), executionID(t), executionID(t)
|
||||||
|
deadline := now.Add(time.Hour)
|
||||||
|
exec(`INSERT INTO weave_poll_states(namespace_id,id,workflow_id,activation_id,max_rounds,used_rounds,max_attempts_per_round,interval_ms,deadline_at,current_task_id,revision,created_at,updated_at) VALUES (?,?,?,?,3,2,4,1000,?,?,7,?,?)`, ns, id, workflow, activation, deadline, task, now, now)
|
||||||
|
exec("INSERT INTO weave_poll_rounds(namespace_id,id,workflow_id,activation_id,`round`,task_id) VALUES (?,?,?,?,1,?),(?,?,?,?,2,?)", ns, executionID(t), workflow, activation, executionID(t), ns, executionID(t), workflow, activation, task)
|
||||||
|
if err := Migrate(ctx, db); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := CheckSchema(ctx, db); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var first, carried, used, max, attempts, revision uint32
|
||||||
|
var gotDeadline, created, updated time.Time
|
||||||
|
if err := db.QueryRowContext(ctx, `SELECT first_round,first_round_attempts_used,used_rounds,max_rounds,max_attempts_per_round,revision,deadline_at,created_at,updated_at FROM weave_poll_states WHERE namespace_id=? AND id=?`, ns, id).Scan(&first, &carried, &used, &max, &attempts, &revision, &gotDeadline, &created, &updated); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if first != 1 || carried != 0 || used != 2 || max != 3 || attempts != 4 || revision != 7 || !gotDeadline.Equal(deadline) || !created.Equal(now) || !updated.Equal(now) {
|
||||||
|
t.Fatal("migration changed original policy, revision or timestamps")
|
||||||
|
}
|
||||||
|
var rounds string
|
||||||
|
if err := db.QueryRowContext(ctx, "SELECT GROUP_CONCAT(`round` ORDER BY `round`) FROM weave_poll_rounds WHERE namespace_id=? AND activation_id=?", ns, activation).Scan(&rounds); err != nil || rounds != "1,2" {
|
||||||
|
t.Fatal("original rounds changed", rounds, err)
|
||||||
|
}
|
||||||
|
for _, clause := range []string{"first_round=0", "first_round=3", "first_round_attempts_used=4"} {
|
||||||
|
if _, err := db.ExecContext(ctx, "UPDATE weave_poll_states SET "+clause+" WHERE namespace_id=? AND id=?", ns, id); err == nil {
|
||||||
|
t.Fatal("invalid recovery budget accepted", clause)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
var count int
|
||||||
|
if err := db.QueryRowContext(ctx, `SELECT COUNT(*) FROM information_schema.REFERENTIAL_CONSTRAINTS WHERE CONSTRAINT_SCHEMA=DATABASE()`).Scan(&count); err != nil || count != 0 {
|
||||||
|
t.Fatal("foreign keys introduced", err)
|
||||||
|
}
|
||||||
|
for _, column := range []string{"first_round", "first_round_attempts_used"} {
|
||||||
|
var comment string
|
||||||
|
if err := db.QueryRowContext(ctx, `SELECT COLUMN_COMMENT FROM information_schema.COLUMNS WHERE TABLE_SCHEMA=DATABASE() AND TABLE_NAME='weave_poll_states' AND COLUMN_NAME=?`, column).Scan(&comment); err != nil || !strings.Contains(comment, "业务") {
|
||||||
|
t.Fatal("missing Chinese comment", column, comment, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := Migrate(ctx, db); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Log("real v21 upgrade preserves original rounds, budget, deadline and revisions; default first_round=1/carry=0; invalid recovery budget rejected; zero foreign keys")
|
||||||
|
}
|
||||||
144
internal/storage/mysql/workflow_poll_retry.go
Normal file
144
internal/storage/mysql/workflow_poll_retry.go
Normal file
@@ -0,0 +1,144 @@
|
|||||||
|
package mysql
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"database/sql"
|
||||||
|
"errors"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||||
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/pulse/service"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||||
|
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||||
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
|
)
|
||||||
|
|
||||||
|
// retryWorkflowPoll 重新执行当前业务轮;新激活保留旧轮次、同轮尝试、精确输入及两个绝对截止。
|
||||||
|
func retryWorkflowPoll(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, in recovery.RetryStepInput, now time.Time, out *recovery.Result) (map[string]any, error) {
|
||||||
|
p, last, err := readWorkflowPollBudget(ctx, tx, g, row)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
origin, err := workflowActivationOrigin(ctx, tx, g, row)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !now.Before(p.DeadlineAt) || origin.sequence >= p.MaxRounds*p.MaxAttemptsPerRound {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
h, err := readWorkflowPollRoundHistory(ctx, tx, g, row, p, p.UsedRounds)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if h.last.ID != last || !h.last.State.Terminal() || h.last.EffectUncertain || h.used >= p.MaxAttemptsPerRound || h.count >= p.MaxAttemptsPerRound || !now.Before(h.first.DeadlineAt) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
// 已确认的PENDING应按间隔和业务轮次预算推进,不能借人工恢复再查同一轮。
|
||||||
|
if h.last.State == "SUCCEEDED" {
|
||||||
|
kind, err := g.program.ClassifyPoll(row.view.Key, h.last.Result)
|
||||||
|
if err == nil && kind == engine.PollPending {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
}
|
||||||
|
node, _ := g.program.Step(row.view.Key)
|
||||||
|
var wait *workflowSignalWait
|
||||||
|
if node.Poll.FinalSignal != nil {
|
||||||
|
w, err := readWorkflowSignalWait(ctx, tx, g, row)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !now.Before(w.deadline) {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
var pending int
|
||||||
|
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_signals WHERE namespace_id=? AND workflow_id=? AND target_step_key=? AND target_activation_sequence=? AND state='PENDING'`, g.view.NamespaceID, g.view.ID, row.view.Key, w.sequence).Scan(&pending); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if pending != 0 {
|
||||||
|
return nil, fault.Conflict
|
||||||
|
}
|
||||||
|
if w.winner != nil {
|
||||||
|
if _, err := selectedWorkflowSignal(ctx, tx, g, row, w, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
wait = &w
|
||||||
|
}
|
||||||
|
policy := h.first.Retry
|
||||||
|
policy.MaxAttempts = p.MaxAttemptsPerRound - h.used
|
||||||
|
input, err := service.NormalizeSubmit(service.SubmitInput{QueueID: h.first.QueueID, Handler: h.first.Handler, HandlerVersion: h.first.HandlerVersion, PayloadVersion: h.first.PayloadVersion, Payload: h.first.Payload, BusinessKey: h.first.BusinessKey, ConcurrencyKey: h.first.ConcurrencyKey, Priority: h.first.Priority, RecoverySafety: h.first.RecoverySafety, Retry: policy, TaskTimeoutMS: node.Poll.Task.TimeoutMS})
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
activation, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
taskID, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
task, err := insertAcceptedTaskUntil(ctx, tx, g.view.NamespaceID, taskID, input, &last, &h.first.DeadlineAt)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if _, err := tx.ExecContext(ctx, `INSERT INTO weave_activations(namespace_id,id,workflow_id,step_id,sequence,state,input,created_at,updated_at) VALUES (?,?,?,?,?,'ACTIVE',?,?,?)`, g.view.NamespaceID, activation, g.view.ID, row.view.ID, origin.sequence+1, string(row.input), now, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
result, err := tx.ExecContext(ctx, `UPDATE pulse_tasks SET owner_kind=?,owner_id=?,gate_id=? WHERE namespace_id=? AND id=? AND owner_kind IS NULL`, engine.TaskOwnerKind, activation, g.gate, g.view.NamespaceID, taskID)
|
||||||
|
if err := exactlyOne(result, err); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
// 旧预算仅结清检查点;旧任务、旧激活输出、轮次及已匹配信号不改写。
|
||||||
|
if _, err := tx.ExecContext(ctx, `UPDATE weave_poll_states SET current_task_id=NULL,next_poll_at=NULL,resolved_at=COALESCE(resolved_at,?),revision=revision+1,updated_at=? WHERE namespace_id=? AND id=?`, now, now, g.view.NamespaceID, p.ID); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
pollID, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if _, err := tx.ExecContext(ctx, `INSERT INTO weave_poll_states(namespace_id,id,workflow_id,activation_id,max_rounds,used_rounds,first_round,first_round_attempts_used,max_attempts_per_round,interval_ms,deadline_at,current_task_id,created_at,updated_at) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?)`, g.view.NamespaceID, pollID, g.view.ID, activation, p.MaxRounds, p.UsedRounds, p.UsedRounds, h.used, p.MaxAttemptsPerRound, p.IntervalMS, p.DeadlineAt, taskID, now, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
roundID, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if _, err := tx.ExecContext(ctx, "INSERT INTO weave_poll_rounds(namespace_id,id,workflow_id,activation_id,`round`,task_id) VALUES (?,?,?,?,?,?)", g.view.NamespaceID, roundID, g.view.ID, activation, p.UsedRounds, taskID); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if wait != nil {
|
||||||
|
if _, err := tx.ExecContext(ctx, `UPDATE weave_waits SET state='CANCELLED',resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND state='WAITING'`, now, now, g.view.NamespaceID, wait.id); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
id, err := core.NewID()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if _, err := tx.ExecContext(ctx, `INSERT INTO weave_waits(namespace_id,id,workflow_id,activation_id,kind,correlation_key,deadline_at,signal_name,payload_version,created_at,updated_at) VALUES (?,?,?,?,'POLL',?,?,?,?,?,?)`, g.view.NamespaceID, id, g.view.ID, activation, []byte(wait.correlation), wait.deadline, wait.contract.Name, wait.contract.PayloadVersion, now, now); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
result, err = tx.ExecContext(ctx, `UPDATE weave_steps SET state='ACTIVE',current_activation_id=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND current_activation_id=?`, activation, now, g.view.NamespaceID, row.view.ID, row.view.Revision, row.view.ActivationID)
|
||||||
|
if err := exactlyOne(result, err); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if row.view.TaskID != nil {
|
||||||
|
return nil, errors.New("poll cannot have direct task reference")
|
||||||
|
}
|
||||||
|
row.view.State, row.view.ActivationID = weave.StepActive, &activation
|
||||||
|
row.view.Revision++
|
||||||
|
row.output = nil
|
||||||
|
if err := saveWorkflowClock(ctx, tx, g, row, "POLL", row.view.Revision, p.DeadlineAt); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := emitEvent(ctx, tx, g.view.NamespaceID, taskID, task.Revision, "task", "task.accepted", map[string]any{"task_id": taskID, "state": task.State, "workflow_id": g.view.ID, "activation_id": activation, "retry_of": last, "round": p.UsedRounds}); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if err := emitEvent(ctx, tx, g.view.NamespaceID, row.view.ID, row.view.Revision, "workflow_step", "workflow.step.changed", map[string]any{"workflow_id": g.view.ID, "activation_id": activation, "state": row.view.State}); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
out.Step, out.Task = &row.view, &task
|
||||||
|
return map[string]any{"step_id": row.view.ID, "previous_activation_id": in.ActivationID, "activation_id": activation, "sequence": origin.sequence + 1, "retry_of": last, "task_id": taskID, "safe_to_retry": true, "round": p.UsedRounds, "max_rounds": p.MaxRounds, "used_attempts": h.used, "remaining_attempts": policy.MaxAttempts, "deadline_at": p.DeadlineAt, "task_deadline_at": h.first.DeadlineAt}, nil
|
||||||
|
}
|
||||||
@@ -22,7 +22,7 @@ func TestIntegrationWorkflowRuntimeProcess(t *testing.T) {
|
|||||||
if dsn == "" || binary == "" {
|
if dsn == "" || binary == "" {
|
||||||
t.Skip("requires disposable MySQL and built Orbit binary")
|
t.Skip("requires disposable MySQL and built Orbit binary")
|
||||||
}
|
}
|
||||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
||||||
defer cancel()
|
defer cancel()
|
||||||
admin, err := Open(ctx, dsn, 2, 1)
|
admin, err := Open(ctx, dsn, 2, 1)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -375,4 +375,37 @@ func TestIntegrationWorkflowRuntimeProcess(t *testing.T) {
|
|||||||
t.Fatal("manual recovery duplicated work or rewrote old activation")
|
t.Fatal("manual recovery duplicated work or rewrote old activation")
|
||||||
}
|
}
|
||||||
t.Log("manual execution review, explicit workflow release and new TASK activation survived all-stop and two fresh controllers; original cancellation, task deadline and remaining attempt budget preserved")
|
t.Log("manual execution review, explicit workflow release and new TASK activation survived all-stop and two fresh controllers; original cancellation, task deadline and remaining attempt budget preserved")
|
||||||
|
for _, p := range processes {
|
||||||
|
stop(p)
|
||||||
|
}
|
||||||
|
// 所有控制进程退出后人工重查同轮,再让两台全新节点推进下一业务轮。
|
||||||
|
pollRetry := runtimeStart(t, f, recoveryPollSpec(f.resources.QueueID, false))
|
||||||
|
grantWorkflowRecovery(f)
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
retryFinish(t, f, ref, "FAILED", false)
|
||||||
|
start("poll-review-a")
|
||||||
|
start("poll-review-b")
|
||||||
|
wait(func() bool { return f.count("weave_instances", "id=? AND state='NEEDS_REVIEW'", pollRetry.ID) == 1 }, "poll enters review")
|
||||||
|
for _, p := range processes {
|
||||||
|
stop(p)
|
||||||
|
}
|
||||||
|
old = f.get(ref.TaskID)
|
||||||
|
recovered := retryPollStep(t, f, pollRetry.ID)
|
||||||
|
continueWorkflow(t, f, pollRetry.ID)
|
||||||
|
if recovered.Task.Retry.MaxAttempts != 2 || !recovered.Task.DeadlineAt.Equal(old.DeadlineAt) {
|
||||||
|
t.Fatal("poll recovery restarted budget")
|
||||||
|
}
|
||||||
|
start("poll-retry-a")
|
||||||
|
start("poll-retry-b")
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
finishQueryOnly(t, f, ref, "PENDING")
|
||||||
|
wait(func() bool { return f.count("pulse_tasks", "1=1") == 16 }, "new round after recovered request")
|
||||||
|
ref = retryStart(t, f)
|
||||||
|
finishQueryOnly(t, f, ref, "DONE")
|
||||||
|
wait(func() bool { return f.count("weave_instances", "id=? AND state='SUCCEEDED'", pollRetry.ID) == 1 }, "poll recovery final completion")
|
||||||
|
if f.count("weave_poll_states", "workflow_id=? AND first_round=1 AND used_rounds=2 AND first_round_attempts_used=1 AND resolved_at IS NOT NULL", pollRetry.ID) != 1 || f.count("pulse_tasks", "1=1") != 16 || f.used() != 0 {
|
||||||
|
t.Fatal("recovered query duplicated, reset budget or leaked permit")
|
||||||
|
}
|
||||||
|
t.Log("POLL manual same-round recovery survived all-stop and two fresh controllers; reduced request attempts, original deadline, old activation and subsequent business-round progression preserved")
|
||||||
|
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -250,11 +250,7 @@ func cancelWorkflowWait(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
if row.view.Kind == weave.NodePoll {
|
if row.view.Kind == weave.NodePoll {
|
||||||
var activated time.Time
|
p, _, err := readWorkflowPollState(ctx, tx, g, row)
|
||||||
if err := tx.QueryRowContext(ctx, `SELECT created_at FROM weave_activations WHERE namespace_id=? AND workflow_id=? AND step_id=? AND id=?`, g.view.NamespaceID, g.view.ID, row.view.ID, row.view.ActivationID).Scan(&activated); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
p, _, err := readWorkflowPollState(ctx, tx, g, row, activated)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -300,7 +296,7 @@ func validateWorkflowStopTask(ctx context.Context, tx *sql.Tx, g *workflowGraph,
|
|||||||
return errors.New("stopping query association invalid")
|
return errors.New("stopping query association invalid")
|
||||||
}
|
}
|
||||||
var count int
|
var count int
|
||||||
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.`round` BETWEEN 1 AND p.used_rounds", g.view.NamespaceID, g.view.ID, owner, id).Scan(&count); err != nil {
|
if err := tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.`round` BETWEEN p.first_round AND p.used_rounds", g.view.NamespaceID, g.view.ID, owner, id).Scan(&count); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if count != 1 {
|
if count != 1 {
|
||||||
@@ -326,7 +322,7 @@ func verifyWorkflowStopWork(ctx context.Context, tx *sql.Tx, g *workflowGraph) (
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
// 一次读取关联元数据并通过索引EXISTS核对查询轮次,避免持有应用锁期间逐任务发出SQL。
|
// 一次读取关联元数据并通过索引EXISTS核对查询轮次,避免持有应用锁期间逐任务发出SQL。
|
||||||
rows, err := tx.QueryContext(ctx, "SELECT t.id,t.state,t.effect_uncertain,t.owner_kind,t.owner_id,t.queue_id,a.workflow_id,a.step_id,a.task_id,s.`key`,EXISTS(SELECT 1 FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=t.namespace_id AND r.workflow_id=a.workflow_id AND r.activation_id=a.id AND r.task_id=t.id AND r.`round` BETWEEN 1 AND p.used_rounds) FROM pulse_tasks t LEFT JOIN weave_activations a ON a.namespace_id=t.namespace_id AND a.id=t.owner_id LEFT JOIN weave_steps s ON s.namespace_id=a.namespace_id AND s.workflow_id=a.workflow_id AND s.id=a.step_id WHERE t.namespace_id=? AND t.gate_id=? ORDER BY t.id LIMIT 10001", g.view.NamespaceID, g.gate)
|
rows, err := tx.QueryContext(ctx, "SELECT t.id,t.state,t.effect_uncertain,t.owner_kind,t.owner_id,t.queue_id,a.workflow_id,a.step_id,a.task_id,s.`key`,EXISTS(SELECT 1 FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=t.namespace_id AND r.workflow_id=a.workflow_id AND r.activation_id=a.id AND r.task_id=t.id AND r.`round` BETWEEN p.first_round AND p.used_rounds) FROM pulse_tasks t LEFT JOIN weave_activations a ON a.namespace_id=t.namespace_id AND a.id=t.owner_id LEFT JOIN weave_steps s ON s.namespace_id=a.namespace_id AND s.workflow_id=a.workflow_id AND s.id=a.step_id WHERE t.namespace_id=? AND t.gate_id=? ORDER BY t.id LIMIT 10001", g.view.NamespaceID, g.gate)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, false, err
|
return false, false, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ func (r *WorkflowRuntime) applyWorkflowSignal(ctx context.Context, tx *sql.Tx, g
|
|||||||
var poll weave.PollState
|
var poll weave.PollState
|
||||||
if node.Poll != nil {
|
if node.Poll != nil {
|
||||||
var last core.ID
|
var last core.ID
|
||||||
poll, last, err = readWorkflowPollState(ctx, tx, g, row, w.activated)
|
poll, last, err = readWorkflowPollState(ctx, tx, g, row)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return false, false, "", err
|
return false, false, "", err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -44,6 +44,23 @@ func readWorkflowSignalWait(ctx context.Context, tx *sql.Tx, g *workflowGraph, r
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return w, err
|
return w, err
|
||||||
}
|
}
|
||||||
|
base, err := workflowActivationOrigin(ctx, tx, g, row)
|
||||||
|
if err != nil {
|
||||||
|
return w, err
|
||||||
|
}
|
||||||
|
w.activated = base.started
|
||||||
|
if base.sequence > 1 {
|
||||||
|
original := *row
|
||||||
|
original.view.ActivationID = &base.id
|
||||||
|
original.input = base.input
|
||||||
|
initial, err := readWorkflowSignalWait(ctx, tx, g, &original)
|
||||||
|
if err != nil {
|
||||||
|
return w, err
|
||||||
|
}
|
||||||
|
if initial.correlation != w.correlation {
|
||||||
|
return w, workflowSourceError("SIGNAL_RECOVERY_CORRELATION_INVALID")
|
||||||
|
}
|
||||||
|
}
|
||||||
if name == nil || version == nil {
|
if name == nil || version == nil {
|
||||||
return w, workflowSourceError("SIGNAL_WAIT_UNVERIFIED")
|
return w, workflowSourceError("SIGNAL_WAIT_UNVERIFIED")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,8 +16,7 @@ import (
|
|||||||
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
"git.yuntuo.hk/k2safe/orbit/weave/recovery"
|
||||||
)
|
)
|
||||||
|
|
||||||
// RetryStep 为TASK建立新的执行激活;保留旧激活、原输入和绝对截止,跨激活扣除已用尝试。
|
// RetryStep 为TASK或POLL建立新的执行激活;保留原输入、绝对截止和跨激活尝试预算。
|
||||||
// POLL恢复需要同时保留原轮次身份,不在此处降级成新查询轮次。
|
|
||||||
func (s *WorkflowRecoveryStore) RetryStep(ctx context.Context, t auth.Token, id core.ID, key string, in recovery.RetryStepInput) (recovery.Result, error) {
|
func (s *WorkflowRecoveryStore) RetryStep(ctx context.Context, t auth.Token, id core.ID, key string, in recovery.RetryStepInput) (recovery.Result, error) {
|
||||||
return s.mutate(ctx, t, id, key, "workflow.step.retry", in, in.Decision, func(tx *sql.Tx, g *workflowGraph, _ core.ID, now time.Time, out *recovery.Result) (map[string]any, error) {
|
return s.mutate(ctx, t, id, key, "workflow.step.retry", in, in.Decision, func(tx *sql.Tx, g *workflowGraph, _ core.ID, now time.Time, out *recovery.Result) (map[string]any, error) {
|
||||||
row, err := workflowRecoveryStep(g, in.StepDecision)
|
row, err := workflowRecoveryStep(g, in.StepDecision)
|
||||||
@@ -28,12 +27,15 @@ func (s *WorkflowRecoveryStore) RetryStep(ctx context.Context, t auth.Token, id
|
|||||||
return nil, fault.Conflict
|
return nil, fault.Conflict
|
||||||
}
|
}
|
||||||
node, ok := g.program.Step(row.view.Key)
|
node, ok := g.program.Step(row.view.Key)
|
||||||
if !ok || node.Task == nil {
|
if !ok || node.Task == nil && node.Poll == nil {
|
||||||
return nil, fault.Invalid("this retry operation currently supports TASK steps only")
|
return nil, fault.Invalid("retry currently supports TASK and POLL steps")
|
||||||
}
|
}
|
||||||
if err := workflowStepSettled(ctx, tx, g, row); err != nil {
|
if err := workflowStepSettled(ctx, tx, g, row); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
if node.Poll != nil {
|
||||||
|
return retryWorkflowPoll(ctx, tx, g, row, in, now, out)
|
||||||
|
}
|
||||||
var sequence uint32
|
var sequence uint32
|
||||||
if err := tx.QueryRowContext(ctx, `SELECT sequence FROM weave_activations WHERE namespace_id=? AND id=?`, t.NamespaceID, row.view.ActivationID).Scan(&sequence); err != nil {
|
if err := tx.QueryRowContext(ctx, `SELECT sequence FROM weave_activations WHERE namespace_id=? AND id=?`, t.NamespaceID, row.view.ActivationID).Scan(&sequence); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
|
|||||||
@@ -4,16 +4,16 @@ import (
|
|||||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||||
)
|
)
|
||||||
|
|
||||||
// PollRound 表示一次业务查询轮次,一轮对应一个 Pulse 任务。
|
// PollRound 表示一次激活内的业务查询轮次;人工恢复同轮会在新激活中关联新任务,旧记录保留。
|
||||||
// 对应表 weave_poll_rounds;这是服务端持久化模型,不直接作为 API 响应。
|
// 对应表 weave_poll_rounds;这是服务端持久化模型,不直接作为 API 响应。
|
||||||
type PollRound struct {
|
type PollRound struct {
|
||||||
// Record 包含命名空间、记录标识、并发版本和 UTC 时间。
|
// Record 包含命名空间、记录标识、并发版本和 UTC 时间。
|
||||||
core.Record
|
core.Record
|
||||||
// WorkflowID 所属流程实例标识。
|
// WorkflowID 所属流程实例标识。
|
||||||
WorkflowID core.ID `db:"workflow_id"`
|
WorkflowID core.ID `db:"workflow_id"`
|
||||||
// ActivationID 所属步骤激活,区分旧轮次和人工恢复后的新轮次。
|
// ActivationID 所属步骤激活;人工恢复可在新激活沿用同一业务轮编号。
|
||||||
ActivationID core.ID `db:"activation_id"`
|
ActivationID core.ID `db:"activation_id"`
|
||||||
// Round 本次激活内的查询轮次编号,从 1 开始。
|
// Round 从首次激活起连续的业务轮次;人工恢复同轮保留编号,通过ActivationID区分任务。
|
||||||
Round uint32 `db:"round"`
|
Round uint32 `db:"round"`
|
||||||
// TaskID 关联的 Pulse 任务标识。
|
// TaskID 关联的 Pulse 任务标识。
|
||||||
TaskID core.ID `db:"task_id"`
|
TaskID core.ID `db:"task_id"`
|
||||||
|
|||||||
@@ -13,13 +13,17 @@ type PollState struct {
|
|||||||
core.Record
|
core.Record
|
||||||
// WorkflowID 所属流程实例标识。
|
// WorkflowID 所属流程实例标识。
|
||||||
WorkflowID core.ID `db:"workflow_id"`
|
WorkflowID core.ID `db:"workflow_id"`
|
||||||
// ActivationID 所属步骤激活,区分旧轮次和人工恢复后的新轮次。
|
// ActivationID 所属步骤激活;人工恢复可在新激活沿用同一业务轮编号。
|
||||||
ActivationID core.ID `db:"activation_id"`
|
ActivationID core.ID `db:"activation_id"`
|
||||||
// MaxRounds 业务查询最多轮次,不等于单轮请求重试次数。
|
// MaxRounds 业务查询最多轮次,不等于单轮请求重试次数。
|
||||||
MaxRounds uint32 `db:"max_rounds"`
|
MaxRounds uint32 `db:"max_rounds"`
|
||||||
// UsedRounds 已创建的查询轮次,重启或暂停继续不得重置。
|
// UsedRounds 从首次激活起已创建的业务轮次数;当前激活的本地行数需扣除FirstRound之前的历史。
|
||||||
UsedRounds uint32 `db:"used_rounds"`
|
UsedRounds uint32 `db:"used_rounds"`
|
||||||
// MaxAttemptsPerRound 每轮 Pulse 任务最多尝试数,包含该轮首次请求。
|
// FirstRound 本激活保存的首个业务查询轮次;恢复保留原编号,之前轮次仍留在旧激活。
|
||||||
|
FirstRound uint32 `db:"first_round"`
|
||||||
|
// FirstRoundAttemptsUsed 首个业务轮次在旧激活已用的执行尝试;从原单轮预算扣除,不能通过换激活重置。
|
||||||
|
FirstRoundAttemptsUsed uint32 `db:"first_round_attempts_used"`
|
||||||
|
// MaxAttemptsPerRound 每业务轮跨激活的Pulse尝试总上限,包含该轮首次请求。
|
||||||
MaxAttemptsPerRound uint32 `db:"max_attempts_per_round"`
|
MaxAttemptsPerRound uint32 `db:"max_attempts_per_round"`
|
||||||
// IntervalMS 相邻业务查询轮次的等待间隔,单位毫秒。
|
// IntervalMS 相邻业务查询轮次的等待间隔,单位毫秒。
|
||||||
IntervalMS uint64 `db:"interval_ms"`
|
IntervalMS uint64 `db:"interval_ms"`
|
||||||
|
|||||||
Reference in New Issue
Block a user