feat: consume workflow results and resume durable advances
This commit is contained in:
10
AGENTS.md
10
AGENTS.md
@@ -31,10 +31,14 @@
|
||||
|
||||
- P4定义发布使用schema_version=1有限结构,词法作用域禁止跨分支/前向读取,任务策略显式冻结;SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁;原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务;同版本不同内容不得覆盖。保留精确数字,校验存储原文与结构摘要;不得把发布通过或DSL支持误称为实例执行已实现。
|
||||
|
||||
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归(docs/reports/phase-4-definitions.md)。实例启动/固定版本/首批任务同事务的内部存储已补齐;继续内部事件消费接续、顺序与组合自动推进、持久化等待及实例公共SDK;P4整体和P5–P7仍未完成。
|
||||
- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归(docs/reports/phase-4-definitions.md)。实例启动/固定版本/首批任务同事务的内部存储已补齐;内部事件消费、顺序与组合推进及根接续已接入;继续持久化等待及实例公共SDK;P4整体和P5–P7仍未完成。
|
||||
|
||||
- 内部结果交付使用workflow_activation所有者并固定激活ID,TASK经activation.task_id、POLL经轮次校验归属;weave_inbox来源及独立快照与Pulse结果同事务,先于外部结果流行锁,不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史,禁止用当前任务重建旧事件。
|
||||
- weave/engine仅计算有限决策,完整实例存储运行器尚未接入。Plan最多32个变更,More必须持久化待续;耗尽变更预算仍须遍历其他已活动分支,不能误判在途工作不可达。冻结分支选择,精确数值,不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务,不能把纯快照往返测试当作真实进程恢复。
|
||||
- weave/engine仅计算有限决策;实例受理、Inbox消费与根接续已接入,完整等待/控制运行器继续开发。Plan最多32个变更,More必须持久化待续;耗尽变更预算仍须遍历其他已活动分支,不能误判在途工作不可达。冻结分支选择,精确数值,不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务,不能把纯快照往返测试当作真实进程恢复。
|
||||
|
||||
- 实例内部受理已实现,公共实例HTTP/SDK及消费/等待运行器尚未开放。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作,所有失败同事务回滚。首批推进递增revision至2,原键返回冻结回执且仍核验当前权限,不重算截止。
|
||||
- 实例内部受理已实现,公共实例HTTP/SDK及等待运行器尚未开放;结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作,所有失败同事务回滚。首批推进递增revision至2,原键返回冻结回执且仍核验当前权限,不重算截止。
|
||||
- 实例根步骤`$root`/JOIN不属于用户定义,Steps查询排除它;More对应根步骤版本化持久检查点。TASK直接关联激活task_id,POLL通过轮次关联,实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结,旧NULL等待须核实,不能盲目解释;新关联512字节、存储兼容历史764字节。
|
||||
|
||||
- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁,禁止先锁Inbox再锁父资源。来源使用独立冻结快照,旧激活/低版本留痕;数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。
|
||||
- 后继满额用SAVEPOINT回滚整轮新工作,保留已接受结果和待续;恢复必须重读持久图。多次根推进使用独立系统审计操作身份;迟到证据也递增实例历史版本,禁止复用已有workflow.advanced唯一身份。
|
||||
- 本轮消费运行器和真实Orbit强杀/双节点接管已验证,见docs/reports/phase-4-runtime.md。POLL仅当前轮次分类与next_poll_at保存;下一轮派发、等待到期/信号、FAIL_AFTER_SETTLED分支结清、控制API和实例SDK仍须完成,不得按部分自动运行宣称P4/P5完成。
|
||||
|
||||
@@ -10,7 +10,7 @@ Orbit 是面向多个业务项目的通用任务调度与流程编排平台,
|
||||
|
||||
已完成 P0–P3 平台功能阶段验收:35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过;[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK,[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例,[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现;[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例;[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例;[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景;[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本;[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI;[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复;完整范围核对见[P3验收](docs/reports/phase-3.md),流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。
|
||||
|
||||
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现;已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。Inbox消费、持久化等待运行器和实例公共API/SDK继续开发,发布定义不会启动任务。
|
||||
P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现;已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。持久化等待、后续查询轮次、失败结清及实例公共API/SDK继续开发,发布定义不会启动任务。
|
||||
|
||||
数据库禁止外键,关联一致性通过服务端事务校验,索引、唯一约束和 CHECK 保留;readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。
|
||||
|
||||
|
||||
@@ -81,6 +81,31 @@ func run() error {
|
||||
}
|
||||
httpserver.RegisterSchedules(server, schedule.New(schedules))
|
||||
httpserver.RegisterWorkflowDefinitions(server, definition.New(mysql.NewWorkflowDefinitionStore(db)))
|
||||
workflows, err := mysql.NewWorkflowRuntime(db, node)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
workflowCtx, stopWorkflows := context.WithCancel(root)
|
||||
workflowDone := make(chan struct{})
|
||||
go func() {
|
||||
defer close(workflowDone)
|
||||
ticker := time.NewTicker(time.Second)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-workflowCtx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
call, cancel := context.WithTimeout(workflowCtx, 5*time.Second)
|
||||
_, err := workflows.Tick(call, 100)
|
||||
cancel()
|
||||
if err != nil && workflowCtx.Err() == nil {
|
||||
slog.Error("workflow batch failed", "node_id", node)
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
defer func() { stopWorkflows(); <-workflowDone }()
|
||||
scheduleCtx, stopSchedules := context.WithCancel(root)
|
||||
scheduleDone := make(chan struct{})
|
||||
go func() {
|
||||
|
||||
@@ -28,7 +28,7 @@
|
||||
| P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件;SDK 客户端 | API/SDK 创建查询闭环;重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归(见 P1 报告;真实断路/提交 ACK 丢失注入留后续故障阶段) |
|
||||
| P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复;Worker SDK | 真 Worker 执行;失联/重连/重复确认/旧尝试拒绝;业务效果未知转核查;节点故障接管 | 已完成并通过阶段回归:服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复;测试拓扑和剩余生产故障矩阵见 P2 验收报告 |
|
||||
| P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收:[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景;20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 |
|
||||
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容;端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现,已通过[子项验收](reports/phase-4-definitions.md);已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md);[实例受理及首批事务](reports/phase-4-admission.md)已实现;事件消费推进、持久化等待运行器和实例公共API/SDK仍待实现 |
|
||||
| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容;端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现,已通过[子项验收](reports/phase-4-definitions.md);已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md);[实例受理及首批事务](reports/phase-4-admission.md)已实现;[结果消费和根接续](reports/phase-4-runtime.md)已接入后台,顺序/分支/并行及真实进程恢复已验证;持久化等待、完整控制和实例公共API/SDK继续开发 |
|
||||
| P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算;暂停/开始/回执/超时竞争;恢复保留预算;未知结果不盲目重复 | 待开发 |
|
||||
| P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行,报告环境/指标/遗留项;容量、RPO/RTO 按真实拓扑验证 | 待开发 |
|
||||
| P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例;所有首版能力可追溯到实现与测试;报告实际生产边界 | 待开发 |
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 执行决策与内部结果交付
|
||||
|
||||
状态:P4 内部基础已实现。已接通Pulse结果到Weave持久化Inbox,并提供有限步骤决策器;[实例受理和首批存储适配](11-workflow-admission.md)已补齐。**Inbox消费推进、定时/信号运行器及实例公共API/SDK尚未实现**。不能把本页能力解释成流程已经可以端到端执行。
|
||||
状态:P4 内部基础已实现。已接通Pulse结果到Weave持久化Inbox,并提供有限步骤决策器;[实例受理和首批存储适配](11-workflow-admission.md)已补齐。[Inbox消费推进及根检查点运行器](12-workflow-runtime.md)已接入;定时/信号运行器及实例公共API/SDK仍待实现。不能把本页能力解释成完整公开端到端流程已交付。
|
||||
|
||||
## 1. 为什么结果要独立持久化
|
||||
|
||||
@@ -23,7 +23,7 @@ flowchart LR
|
||||
|
||||
已实现的内部交付触发为`task.succeeded`、`task.failed`、`task.cancelled`、`task.needs_review`。任务暂时失败并等待自动重试,不新增流程激活或内部结果事件。相同完成回执重报不产生第二个事件;同一来源任务状态版本还有独立唯一约束。
|
||||
|
||||
发布后使用旧激活的在途结果仍保留原激活身份;生产者不自行判定是否推进当前激活。未来消费者须识别旧激活/旧轮次并留痕,不能将其用于当前工作。
|
||||
发布后使用旧激活的在途结果仍保留原激活身份;生产者不自行判定是否推进当前激活。当前消费者识别旧激活/旧轮次并留痕,不能将其用于当前工作。
|
||||
|
||||
生产者在现有结果快照查询中一起读取所有者和关联标记;普通结果不增加一次单独网络查询。流程关联核验使用只读关联查询,不在已持有任务锁后再锁实例、步骤或对外订阅。内部Inbox写入发生在外部结果流行锁之前,保持既有锁顺序。
|
||||
|
||||
@@ -74,7 +74,7 @@ v16及以前的未处理记录缺少冻结来源,升级后统一标为QUARANTI
|
||||
|
||||
## 4. 后续存储适配必须满足的条件
|
||||
|
||||
以下是存储运行器的完整要求;第1项初始事务已实现并按[受理报告](reports/phase-4-admission.md)验证,其余消费/运行要求尚未完成,不能按纯单测宣称已完成:
|
||||
以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证;第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md),第5–7项的完整等待、控制、接口和SDK仍继续实现:
|
||||
|
||||
1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止,并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。
|
||||
2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本;消费确认、步骤变化、后续任务或持久化待续记录同事务提交。
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Weave 实例受理与首批工作
|
||||
|
||||
状态:内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。**尚未开放实例HTTP路由、启动SDK或命令行入口;后台Inbox消费、待续扫描和等待运行器尚未接通。** Weave相关公开SDK目前仅支持定义发布。不能将首批任务可执行解释成整个流程已能自动完成。
|
||||
状态:内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口;等待运行器、后续轮询和完整控制仍继续开发。** Weave相关公开SDK目前仅支持定义发布。TASK及组合节点已可自动推进,不能将此解释成全部等待/控制能力已实现。
|
||||
|
||||
## 1. 受理事务
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
| --- | --- |
|
||||
| TASK | 当前激活、精确输入、所属Pulse任务、流程门控及`workflow_activation`所有者 |
|
||||
| BRANCH | 首次选择的case索引;未选中节点分批SKIPPED,不创建业务激活 |
|
||||
| PARALLEL | 父激活、独立分支初始工作;完成汇合仍需要后续消费推进 |
|
||||
| PARALLEL | 父激活、独立分支初始工作;后续结果消费运行器负责完成汇合 |
|
||||
| SIGNAL | WAITING激活、信号名称/内容版本/精确业务关联、固定截止;不分配Worker |
|
||||
| TIMER | WAITING激活和数据库时间计算的等待截止;不生成业务任务 |
|
||||
| POLL | 固定输入、查询状态、首轮记录和唯一当前任务;used_rounds=1;可选最终信号等待与查询截止一致 |
|
||||
@@ -49,14 +49,14 @@ Get返回当前实例视图;Start重报返回受理时冻结的原始视图,
|
||||
| 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例;终态不计入;满额仍可恢复原受理回执 |
|
||||
| 信号关联 | 新受理最多512字节;数据库兼容旧191字符最长764字节,升级不截断历史 |
|
||||
| 截止 | 数据库时间生成;任务、查询与等待均不超过实例总截止 |
|
||||
| 自动运行 | 本阶段尚未实现Inbox消费、待续扫描、信号消费、定时到期处理或后续轮询 |
|
||||
| 自动运行 | Inbox消费和根待续扫描已接入;信号消费、定时到期处理及后续轮询仍待实现 |
|
||||
|
||||
迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段;NULL仅表示TIMER或尚未核实的历史元数据,未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储;实例business_key改为utf8mb4_0900_bin。35个实体、含公共Record的295个直接持久化字段声明,全部有中文注释,继续保持零外键。
|
||||
|
||||
## 5. 下一步接入
|
||||
|
||||
1. 锁定实例、核验来源快照/当前激活,消费确认与后续步骤/任务/待续记录同事务;处理重复、迟到、旧轮次和未知结果。
|
||||
2. 实现有界待续扫描与数据库时间驱动的等待,恢复时保留原截止和次数;开始扫描前补齐暂停/截止/凭据所属应用停用的判定。
|
||||
1. 结果消费与根待续运行器已按[运行器报告](reports/phase-4-runtime.md)验证;继续补齐数据库时间驱动的信号/定时等待。
|
||||
2. 已有扫描核验暂停、截止与所属应用停用;继续后续轮次派发、回执竞争及失败分支结清,恢复保留原截止和次数。
|
||||
3. 接通实例公共HTTP、独立SDK、来源查询和可运行端到端场景,同时实施真实节点退出与接管回归。
|
||||
|
||||
本阶段验证见[实例受理回归记录](reports/phase-4-admission.md)。纯受理测试不是流程运行验收,P4–P7仍按[完整计划](08-development-plan.md)推进。
|
||||
|
||||
67
docs/12-workflow-runtime.md
Normal file
67
docs/12-workflow-runtime.md
Normal file
@@ -0,0 +1,67 @@
|
||||
# Weave 结果消费与自动推进
|
||||
|
||||
状态:内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。实例公共HTTP/SDK仍未开放。信号/定时等待、下一轮查询、失败后的分支取消结清及人工恢复仍继续开发,不能将本阶段等同于完整Weave交付。
|
||||
|
||||
## 1. 后台运行和事务边界
|
||||
|
||||
每个Orbit进程每秒调用一次`WorkflowRuntime.Tick`,一批最多100个候选、调用超时5秒。候选来自`weave_inbox`的PENDING到期索引和`weave_pending_advances`的到期索引,各候选独立事务。选择候选不授予执行权;同一应用繁忙时跳过,后续扫描重读数据库。
|
||||
|
||||
事务锁顺序是身份共享锁 → 应用额度排他锁 → 固定版本涉及的排序队列 → 实例 → 执行门控 → 激活/步骤及消费记录。不能先领取并持有Inbox锁,再倒过来锁应用或任务。运行器不向业务系统发请求,不需要Redis或额外消息队列;业务执行仍由Pulse授权Worker。
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[读取到期Inbox或根检查点] --> B[锁定并核验实例图]
|
||||
B --> C[读取冻结结果并校验来源]
|
||||
C --> D[更新步骤和计算有限决策]
|
||||
D --> E[创建后继工作或保存待续]
|
||||
E --> F[确认消费并写审计]
|
||||
F --> G[同一事务提交]
|
||||
```
|
||||
|
||||
每轮最多32个结构变更。达到预算时保留根步骤版本及ROOT检查点,由后续节点继续;后续计算仍使用固定版本、原输入和已保存结果。旧检查点只能基于当前持锁快照重新计算,不能覆盖更高步骤版本;超前版本或错误根身份拒绝执行。
|
||||
|
||||
## 2. 结果、去重与审计
|
||||
|
||||
消费只使用内部Inbox的独立结果快照。校验结构版本、任务和队列、来源状态版本、原事件身份/类型/版本、激活所属实例和步骤、Pulse所有者及门控、尝试归属;POLL额外校验原轮次。任务当前结果不能替换历史快照。
|
||||
|
||||
| 情况 | 处理 |
|
||||
| --- | --- |
|
||||
| 已确认或已隔离的记录再次被扫描 | 不再次更新步骤、创建任务或写相同消费审计 |
|
||||
| 旧激活、已处理的旧轮次或低于已处理结果版本 | 保留来源,确认并记录稳定分类,不推动当前工作 |
|
||||
| 已结清步骤收到其他历史结果 | 保留处理历史,不重开步骤或改变已完成业务结果 |
|
||||
| 来源身份或结构矛盾 | QUARANTINED;未终结实例转NEEDS_REVIEW并关闭新执行门控 |
|
||||
| SQL/连接或事务异常 | 整笔回滚;仍保留PENDING。扫描器按原记录版本延后30秒,不能覆盖其他节点刚完成的确认 |
|
||||
| 正常TASK成功 | 验证固定输出契约,保存SUCCEEDED后有限推进 |
|
||||
| TASK确认取消 | 步骤保留CANCELLED和来源证据,不能改写为FAILED或启动正常后继 |
|
||||
| TASK失败、未知效果或不合契约的成功结果 | 停止依赖推进,保留原任务结果及稳定核查原因 |
|
||||
|
||||
每个成功提交的消费或实际根推进写系统审计,actor为`system:weave:<node_id>`。审计使用独立操作身份,元信息保存来源Inbox/检查点ID、节点、实例版本及原因;同一个ROOT检查点可分批产生多次真实推进。每次历史变更取得新的实例revision,避免迟到证据与先前事件的唯一身份冲突。
|
||||
|
||||
审计失败会回滚消费确认、步骤状态和新任务。对外任务结果订阅继续使用原链路,内部消费不需要锁订阅或外部结果流。
|
||||
|
||||
## 3. 满额、暂停和截止
|
||||
|
||||
后继创建使用SAVEPOINT隔离本轮结构变更。多后继中任何一个遇到队列或应用满额,全部本轮后继和对应额度预占回滚;已接受的来源结果及步骤完成仍能与ROOT待续记录一起提交。下一次尝试从当前图重新计算,不能留下部分激活、重复后继或多扣用量。持续满额仅移动检查时间,不逐秒制造相同业务审计。
|
||||
|
||||
暂停、关闭门控或停用应用/命名空间时,继续收集已经落库的结果,但不激活新步骤;若还有后继,将其保存为待续。重新允许推进后,从持久记录继续。测试通过直接设置控制状态验证该行为;暂停/继续公共接口尚未交付。
|
||||
|
||||
已过实例总截止时不创建后继,未能全部完成则进入核查。最后的有效任务结果在截止前已由Pulse接受,而消费发生较晚时,可以整理完整结果并完成实例。暂停或重启不延长截止。纯等待实例到期及信号裁决仍需后续等待扫描器。
|
||||
|
||||
## 4. POLL与失败策略的当前边界
|
||||
|
||||
已实现当前轮次的结果消费:
|
||||
|
||||
- 查询请求成功但业务PENDING:保存WAITING、清除当前任务并按原事件时间加间隔保存`next_poll_at`,受原查询截止限制;不将PENDING输出交给后继。
|
||||
- DONE等已声明成功值:按固定契约完成节点;如有最终信号等待,一并结清。
|
||||
- 轮次耗尽、查询截止、未知/非法值或单轮请求失败:进入核查,不新建轮次绕过Pulse重试预算。
|
||||
- 业务明确失败:按固定实例失败策略处理;REVIEW进入核查,FAIL_AFTER_SETTLED进入CANCELLING并关闭门控。
|
||||
|
||||
**尚未实现下一轮查询派发及其到期索引扫描,也未实现FAIL_AFTER_SETTLED的剩余分支取消/收集/失败终态判定。** CANCELLING保留未结清事实,不提前宣称FAILED。信号抢先完成时与当前在途查询的结清竞争,也属于后续必须实现的功能。
|
||||
|
||||
## 5. 回归与后续工作
|
||||
|
||||
[存储回归](../internal/storage/mysql/workflow_runtime_integration_test.go)覆盖重复消费者、精确结果、并行汇合、部分后继满额回滚、审计回滚、暂停与停用接续、分批根恢复、来源隔离、旧激活/旧版本、失败/截止和POLL分类;[进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)强杀唯一Orbit进程,确认死亡后提交任务结果,再启动两个新进程接续。
|
||||
|
||||
进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过;跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。
|
||||
|
||||
下一步仍包括持久化定时/信号等待、轮询后续轮次、完整控制与人工恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。
|
||||
@@ -17,6 +17,10 @@
|
||||
| [06 Weave 流程设计](06-workflow-spec.md) | 已确认的首版功能、研发定义发布、持久化等待、有界查询、暂停恢复与模块协作 |
|
||||
| [07 数据模型](07-data-model.md) | 当前 Go 实体、数据库约束、存储约定与实现边界 |
|
||||
| [08 持续开发计划](08-development-plan.md) | 开发、测试回归、阶段验收与最终交付目标 |
|
||||
| [09 可靠结果订阅](09-result-subscription-core.md) | 提交水位、冻结结果与可靠交付 |
|
||||
| [10 Weave决策与交付](10-workflow-engine-core.md) | 有限步骤决策和内部结果Inbox |
|
||||
| [11 Weave实例受理](11-workflow-admission.md) | 固定版本、初始图和首批任务事务 |
|
||||
| [12 Weave结果消费与推进](12-workflow-runtime.md) | 后台消费、根接续、事务与进程恢复,及尚未完成的等待/控制边界 |
|
||||
| [接口手册](api/README.md) | 实际接口状态、调用说明与后续接口规划 |
|
||||
| [SDK 与场景](sdk/README.md) | 可运行示例、接入方法及分阶段 SDK 交付 |
|
||||
|
||||
|
||||
45
docs/reports/phase-4-runtime.md
Normal file
45
docs/reports/phase-4-runtime.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# P4 结果消费与自动推进回归
|
||||
|
||||
日期:2026-09-18。范围为内部运行器,不是完整Weave或生产高可用验收。
|
||||
|
||||
## 已实现
|
||||
|
||||
- Orbit后台按数据库到期索引消费内部结果和根检查点;同一事务校验固定实例图、接受结果、更新步骤、创建后继或保存待续,并写确认、事件和审计。
|
||||
- 顺序、分支、有限并行汇合及32步预算接续;重复消费者不重复创建后继。
|
||||
- 满额SAVEPOINT回滚本轮后继,保留已确认结果;暂停/停用收集在途结果并保留检查点;失败或超期关闭新执行门控。
|
||||
- 来源/激活/任务版本校验,非法来源隔离,旧激活或旧结果留痕;数据库故障不被误认成业务失败。
|
||||
- POLL当前轮次的成功/等待/失败/核查分类,独立保留轮次与尝试预算、绝对截止和下一查询时间。
|
||||
- 系统审计保存节点、来源记录与实例版本;真实进程强杀、全停时保存结果和双节点接管。
|
||||
|
||||
实现说明及明确边界见[运行器文档](../12-workflow-runtime.md)。本轮未增加实体、持久化字段、迁移或外键。
|
||||
|
||||
## 环境与执行记录
|
||||
|
||||
本机darwin/arm64,Go 1.27.1,独立Docker MySQL 8.4.10;隔离一次性数据库。未使用业务数据库或远端生产环境。
|
||||
|
||||
| 检查 | 结果 |
|
||||
| --- | --- |
|
||||
| `make fmt test vet build` | 已通过;服务端、独立SDK及两个业务示例module |
|
||||
| 运行器真实MySQL定向回归 | 14个一级场景,含任务失败/未知/取消/非法输出4种分类及5个POLL分类;最终修订连续3轮通过,20.018秒 |
|
||||
| 真实Orbit进程故障 | 已通过:确认唯一控制进程SIGKILL退出后保存结果,两个新进程只生成一个后继,最终3个任务、3条消费确认及3条推进审计 |
|
||||
| 全量`make integration`等价脚本 | 已通过,退出码0;全存储132.893秒、流程进程8.506秒、Worker SDK 91.172秒、SDK进程故障93.679秒;19个SDK示例及2个独立业务示例全部通过 |
|
||||
|
||||
定向命令为`go test -race -count=3 -run '^TestIntegrationWorkflowRuntime$' -v ./internal/storage/mysql`,依赖`ORBIT_TEST_MYSQL_DSN`指向隔离测试库。进程用例额外需要构建后的`ORBIT_TEST_SERVER_BINARY`;全量入口[脚本](../../scripts/test-mysql.sh)负责创建隔离库、构建并执行该用例。
|
||||
|
||||
文档复核834个本地链接与53个JSON代码块,错误0;差异空白检查通过。
|
||||
|
||||
数据库复核MySQL 8.4.10、外键0、缺少COMMENT的业务列0。完整迁移升级回归与现有索引/数据保留检查通过。
|
||||
|
||||
开发中真实回归发现并修复了:未完成步骤NULL输出的SQL读取不兼容、分批复用检查点引发审计操作身份冲突;随后增加迟到结果历史版本回归,确保新确认不与先前实例事件冲突。后继额度中途失败和审计失败分别注入验证了事务回滚。
|
||||
|
||||
全量回归启动后,代码审阅补充了TASK确认取消保留CANCELLED状态,并新增对应真实Worker回归。最终修订另跑全部运行器场景三轮(20.018秒)、真实进程故障(9.183秒)及基础检查;全量记录与最后定向记录分别保留,不能把较早的全量结果声称为包含后续修改。
|
||||
|
||||
## 尚未完成的原定范围
|
||||
|
||||
1. SIGNAL/TIMER等待消费及到期扫描;早到、重复、超时和回执竞态。
|
||||
2. POLL下一轮派发、完整定时扫描、信号与在途查询结清;已有next_poll_at不代表下一轮已自动运行。
|
||||
3. FAIL_AFTER_SETTLED分支取消和失败终态收敛、暂停/继续/取消公共管理API、带证据人工恢复。
|
||||
4. 实例/步骤/来源公共HTTP和独立SDK、全节点可运行接入例子。
|
||||
5. 全量监控、负载与故障矩阵、MySQL高可用拓扑和备份恢复、P6–P7交付审计。
|
||||
|
||||
本地进程回归使用内部Start和Pulse执行存储驱动业务结果;这验证真实控制进程接管,不能替代公共HTTP/SDK端到端验收,也不证明生产数据库RPO/RTO或容量。
|
||||
274
internal/storage/mysql/workflow_consume.go
Normal file
274
internal/storage/mysql/workflow_consume.go
Normal file
@@ -0,0 +1,274 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/orbit/internal/jsonvalue"
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/platform/events"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||
)
|
||||
|
||||
// workflowSourceError 只用于已核实的来源/契约矛盾;SQL或连接错误不能据此隔离业务结果。
|
||||
type workflowSourceError string
|
||||
|
||||
func (e workflowSourceError) Error() string { return string(e) }
|
||||
|
||||
type workflowResult struct {
|
||||
event, task, activation core.ID
|
||||
revision uint64
|
||||
snapshot events.TaskResult
|
||||
occurred time.Time
|
||||
step *workflowStep
|
||||
}
|
||||
|
||||
func readWorkflowResult(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, now time.Time) (*workflowResult, error) {
|
||||
var state string
|
||||
var due time.Time
|
||||
var raw []byte
|
||||
source := &workflowResult{}
|
||||
err := tx.QueryRowContext(ctx, `SELECT event_id,source_task_id,source_activation_id,source_revision,snapshot,state,available_at FROM weave_inbox WHERE namespace_id=? AND id=? AND workflow_id=? FOR UPDATE`, c.ns, c.id, c.workflow).Scan(&source.event, &source.task, &source.activation, &source.revision, &raw, &state, &due)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if state != "PENDING" || due.After(now) {
|
||||
return nil, nil
|
||||
}
|
||||
if _, err := jsonvalue.Canonical(raw); err != nil {
|
||||
return nil, workflowSourceError("RESULT_SNAPSHOT_INVALID")
|
||||
}
|
||||
decoder := json.NewDecoder(bytes.NewReader(raw))
|
||||
decoder.DisallowUnknownFields()
|
||||
if decoder.Decode(&source.snapshot) != nil || source.snapshot.SchemaVersion != 1 || source.snapshot.TaskID != source.task || source.snapshot.Revision != source.revision || source.revision == 0 || !events.ResultType("task."+strings.ToLower(source.snapshot.State)) {
|
||||
return nil, workflowSourceError("RESULT_SNAPSHOT_INVALID")
|
||||
}
|
||||
var aggregate core.ID
|
||||
var kind, eventType string
|
||||
var rev uint64
|
||||
err = tx.QueryRowContext(ctx, `SELECT aggregate_kind,aggregate_id,aggregate_revision,type,occurred_at FROM orbit_events WHERE namespace_id=? AND id=?`, c.ns, source.event).Scan(&kind, &aggregate, &rev, &eventType, &source.occurred)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, workflowSourceError("RESULT_EVENT_MISSING")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if kind != "task" || aggregate != source.task || rev != source.revision || eventType != "task."+strings.ToLower(source.snapshot.State) || source.occurred.After(now) {
|
||||
return nil, workflowSourceError("RESULT_EVENT_MISMATCH")
|
||||
}
|
||||
var stepID core.ID
|
||||
var direct *core.ID
|
||||
err = tx.QueryRowContext(ctx, `SELECT step_id,task_id FROM weave_activations WHERE namespace_id=? AND id=? AND workflow_id=?`, c.ns, source.activation, c.workflow).Scan(&stepID, &direct)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, workflowSourceError("RESULT_ACTIVATION_MISSING")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, step := range g.steps {
|
||||
if step.view.ID == stepID {
|
||||
source.step = step
|
||||
break
|
||||
}
|
||||
}
|
||||
if source.step == nil {
|
||||
return nil, workflowSourceError("RESULT_STEP_MISSING")
|
||||
}
|
||||
node, ok := g.program.Step(source.step.view.Key)
|
||||
if !ok || node.Task == nil && node.Poll == nil {
|
||||
return nil, workflowSourceError("RESULT_STEP_KIND")
|
||||
}
|
||||
taskDefinition := node.Task
|
||||
if node.Poll != nil {
|
||||
taskDefinition = &node.Poll.Task
|
||||
var round uint32
|
||||
err = tx.QueryRowContext(ctx, `SELECT r.round FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.round BETWEEN 1 AND p.used_rounds`, c.ns, c.workflow, source.activation, source.task).Scan(&round)
|
||||
if errors.Is(err, sql.ErrNoRows) || direct != nil {
|
||||
return nil, workflowSourceError("RESULT_POLL_OWNER")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
} else if direct == nil || *direct != source.task {
|
||||
return nil, workflowSourceError("RESULT_TASK_OWNER")
|
||||
}
|
||||
var queue core.ID
|
||||
var owner, gate *core.ID
|
||||
var ownerKind sql.NullString
|
||||
var currentRevision uint64
|
||||
err = tx.QueryRowContext(ctx, `SELECT queue_id,owner_kind,owner_id,gate_id,revision FROM pulse_tasks WHERE namespace_id=? AND id=?`, c.ns, source.task).Scan(&queue, &ownerKind, &owner, &gate, ¤tRevision)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, workflowSourceError("RESULT_TASK_MISSING")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if queue != source.snapshot.QueueID || queue != taskDefinition.QueueID || ownerKind.String != engine.TaskOwnerKind || owner == nil || *owner != source.activation || gate == nil || *gate != g.gate || currentRevision < source.revision {
|
||||
return nil, workflowSourceError("RESULT_TASK_ASSOCIATION")
|
||||
}
|
||||
if source.snapshot.CurrentAttemptID != nil {
|
||||
var attempt core.ID
|
||||
err = tx.QueryRowContext(ctx, `SELECT id FROM pulse_attempts WHERE namespace_id=? AND id=? AND task_id=?`, c.ns, source.snapshot.CurrentAttemptID, source.task).Scan(&attempt)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil, workflowSourceError("RESULT_ATTEMPT_ASSOCIATION")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return source, nil
|
||||
}
|
||||
|
||||
func (r *WorkflowRuntime) consume(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, allow bool, now time.Time) (bool, error) {
|
||||
source, err := readWorkflowResult(ctx, tx, g, c, now)
|
||||
var invalid workflowSourceError
|
||||
if errors.As(err, &invalid) {
|
||||
if !g.view.State.Terminal() {
|
||||
if err := holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, string(invalid), now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "QUARANTINED", string(invalid), now)
|
||||
}
|
||||
if err != nil || source == nil {
|
||||
return false, err
|
||||
}
|
||||
row := source.step
|
||||
if row.view.ActivationID == nil || *row.view.ActivationID != source.activation {
|
||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_ACTIVATION", now)
|
||||
}
|
||||
var newer int
|
||||
if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_inbox WHERE namespace_id=? AND source_task_id=? AND source_revision>? AND state='PROCESSED'`, c.ns, source.task, source.revision).Scan(&newer); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if newer > 0 {
|
||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_RESULT", now)
|
||||
}
|
||||
if g.view.State.Terminal() || row.view.State != weave.StepActive && row.view.State != weave.StepWaiting {
|
||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STEP_ALREADY_SETTLED", now)
|
||||
}
|
||||
state, output, reason := weave.StepSucceeded, source.snapshot.Result, "TASK_RESULT"
|
||||
if source.snapshot.EffectUncertain || source.snapshot.State == "NEEDS_REVIEW" {
|
||||
state, output, reason = weave.StepNeedsReview, nil, "TASK_RESULT_UNCERTAIN"
|
||||
} else if source.snapshot.State == "CANCELLED" {
|
||||
state, output, reason = weave.StepCancelled, nil, "TASK_RESULT_CANCELLED"
|
||||
} else if source.snapshot.State != "SUCCEEDED" {
|
||||
state, output, reason = weave.StepFailed, nil, "TASK_RESULT_FAILED"
|
||||
}
|
||||
node, _ := g.program.Step(row.view.Key)
|
||||
if node.Poll != nil {
|
||||
var used, max uint32
|
||||
var deadline time.Time
|
||||
var current *core.ID
|
||||
var resolved *time.Time
|
||||
if err := tx.QueryRowContext(ctx, `SELECT used_rounds,max_rounds,deadline_at,current_task_id,resolved_at FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=? FOR UPDATE`, c.ns, c.workflow, source.activation).Scan(&used, &max, &deadline, ¤t, &resolved); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if resolved != nil || current == nil || *current != source.task {
|
||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_POLL_ROUND", now)
|
||||
}
|
||||
// 单轮请求失败不能通过新建业务轮次绕过Pulse预算。
|
||||
if state != weave.StepSucceeded {
|
||||
state, reason = weave.StepNeedsReview, "POLL_REQUEST_REVIEW"
|
||||
} else if !source.occurred.Before(deadline) {
|
||||
state, output, reason = weave.StepNeedsReview, nil, "POLL_DEADLINE"
|
||||
} else {
|
||||
outcome, classifyErr := g.program.ClassifyPoll(row.view.Key, output)
|
||||
switch {
|
||||
case classifyErr != nil:
|
||||
state, output, reason = weave.StepNeedsReview, nil, "POLL_RESULT_INVALID"
|
||||
case outcome == engine.PollPending:
|
||||
state, output, reason = weave.StepWaiting, nil, "POLL_PENDING"
|
||||
if used >= max {
|
||||
state, reason = weave.StepNeedsReview, "POLL_ROUNDS_EXHAUSTED"
|
||||
}
|
||||
case outcome == engine.PollFailed:
|
||||
state, output, reason = weave.StepFailed, nil, "POLL_BUSINESS_FAILED"
|
||||
case outcome == engine.PollNeedsReview:
|
||||
state, output, reason = weave.StepNeedsReview, nil, "POLL_BUSINESS_REVIEW"
|
||||
}
|
||||
}
|
||||
var next, settled *time.Time
|
||||
if state == weave.StepWaiting {
|
||||
v := minimumTime(source.occurred.Add(time.Duration(node.Poll.IntervalMS)*time.Millisecond), deadline)
|
||||
next = &v
|
||||
} else {
|
||||
settled = &now
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE weave_poll_states SET current_task_id=NULL,next_poll_at=?,resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND activation_id=?`, next, settled, now, c.ns, c.workflow, source.activation); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if settled != nil {
|
||||
waitState := "CANCELLED"
|
||||
if state == weave.StepSucceeded {
|
||||
waitState = "SATISFIED"
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE weave_waits SET state=?,resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND state='WAITING'`, waitState, now, now, c.ns, c.workflow, source.activation); err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
} else if state == weave.StepSucceeded && (bytes.Equal(bytes.TrimSpace(output), []byte("null")) || definition.ValidateData(node.Task.ResultType, output) != nil) {
|
||||
state, output, reason = weave.StepNeedsReview, nil, "TASK_RESULT_INVALID"
|
||||
}
|
||||
if err := saveWorkflowResultStep(ctx, tx, g, row, state, output, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if state == weave.StepNeedsReview || state == weave.StepFailed || state == weave.StepCancelled {
|
||||
next := weave.InstanceNeedsReview
|
||||
if (state == weave.StepFailed || state == weave.StepCancelled) && (g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" || g.view.CancelRequestedAt != nil) {
|
||||
next = weave.InstanceCancelling
|
||||
}
|
||||
if err := holdWorkflow(ctx, tx, g, next, reason, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
} else if err := r.progress(ctx, tx, g, allow, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", reason, now)
|
||||
}
|
||||
|
||||
func saveWorkflowResultStep(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, state weave.StepState, output json.RawMessage, now time.Time) error {
|
||||
var raw any
|
||||
if output != nil {
|
||||
raw = string(output)
|
||||
}
|
||||
var completed *time.Time
|
||||
if state == weave.StepSucceeded || state == weave.StepFailed || state == weave.StepCancelled {
|
||||
completed = &now
|
||||
}
|
||||
r, err := tx.ExecContext(ctx, `UPDATE weave_activations SET state=?,output=?,completed_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND workflow_id=? AND step_id=? AND state=?`, state, raw, completed, now, g.view.NamespaceID, row.view.ActivationID, g.view.ID, row.view.ID, row.view.State)
|
||||
if err := exactlyOne(r, err); err != nil {
|
||||
return err
|
||||
}
|
||||
r, err = tx.ExecContext(ctx, `UPDATE weave_steps SET state=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND current_activation_id=?`, state, now, g.view.NamespaceID, row.view.ID, row.view.Revision, row.view.ActivationID)
|
||||
if err := exactlyOne(r, err); err != nil {
|
||||
return err
|
||||
}
|
||||
row.view.State = state
|
||||
row.view.Revision++
|
||||
row.output = output
|
||||
r, err = tx.ExecContext(ctx, `UPDATE weave_instances SET revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, now, g.view.NamespaceID, g.view.ID, g.view.Revision)
|
||||
if err := exactlyOne(r, err); err != nil {
|
||||
return err
|
||||
}
|
||||
g.view.Revision++
|
||||
return emitEvent(ctx, tx, g.view.NamespaceID, row.view.ID, row.view.Revision, "workflow_step", "workflow.step.changed", map[string]any{"workflow_id": g.view.ID, "key": row.view.Key, "state": state, "activation_id": row.view.ActivationID})
|
||||
}
|
||||
|
||||
func (r *WorkflowRuntime) confirmWorkflowResult(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, state, reason string, now time.Time) error {
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_inbox SET state=?,processed_at=?,error_code=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND workflow_id=? AND state='PENDING'`, state, now, reason, now, c.ns, c.id, c.workflow)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return err
|
||||
}
|
||||
return r.audit(ctx, tx, g, c.id, reason, now)
|
||||
}
|
||||
259
internal/storage/mysql/workflow_runtime.go
Normal file
259
internal/storage/mysql/workflow_runtime.go
Normal file
@@ -0,0 +1,259 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/orbit/internal/fault"
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||
)
|
||||
|
||||
// WorkflowRuntime 消费持久化结果与根推进检查点;不持有业务执行租约或调用业务系统。
|
||||
// 每个候选独立短事务,数据库行锁决定唯一推进权,节点内存不保存恢复游标。
|
||||
type WorkflowRuntime struct {
|
||||
db *sql.DB
|
||||
node core.ID
|
||||
}
|
||||
|
||||
func NewWorkflowRuntime(db *sql.DB, node core.ID) (*WorkflowRuntime, error) {
|
||||
if db == nil || node.Validate() != nil {
|
||||
return nil, errors.New("invalid workflow runtime configuration")
|
||||
}
|
||||
return &WorkflowRuntime{db: db, node: node}, nil
|
||||
}
|
||||
|
||||
type WorkflowRuntimeStats struct {
|
||||
Examined int // Examined 本批取出的持久化候选数,包含被其他节点先处理或跳过的记录。
|
||||
Consumed int // Consumed 已提交确认或隔离的内部结果数,事务回滚不计入。
|
||||
Advanced int // Advanced 已提交推进的根检查点数,不等于业务任务执行次数。
|
||||
}
|
||||
|
||||
type workflowCandidate struct {
|
||||
kind string
|
||||
ns, id, workflow core.ID
|
||||
revision uint64
|
||||
}
|
||||
|
||||
// Tick 同时读取两种到期索引,各取最多limit项再全局选前limit项;不先锁Inbox再反锁父资源。
|
||||
func (r *WorkflowRuntime) Tick(ctx context.Context, limit int) (WorkflowRuntimeStats, error) {
|
||||
var stats WorkflowRuntimeStats
|
||||
if limit < 1 || limit > 100 {
|
||||
return stats, fault.Invalid("workflow batch limit must be 1 to 100")
|
||||
}
|
||||
rows, err := r.db.QueryContext(ctx, `(SELECT 'INBOX' kind,namespace_id,id,workflow_id,revision,available_at FROM weave_inbox WHERE state='PENDING' AND available_at<=UTC_TIMESTAMP(6) ORDER BY available_at,namespace_id,id LIMIT ?)
|
||||
UNION ALL (SELECT 'ROOT' kind,namespace_id,id,workflow_id,revision,available_at FROM weave_pending_advances WHERE available_at<=UTC_TIMESTAMP(6) ORDER BY available_at,namespace_id,id LIMIT ?)
|
||||
ORDER BY available_at,namespace_id,id LIMIT ?`, limit, limit, limit)
|
||||
if err != nil {
|
||||
return stats, err
|
||||
}
|
||||
var candidates []workflowCandidate
|
||||
for rows.Next() {
|
||||
var c workflowCandidate
|
||||
var due time.Time
|
||||
if err := rows.Scan(&c.kind, &c.ns, &c.id, &c.workflow, &c.revision, &due); err != nil {
|
||||
rows.Close()
|
||||
return stats, err
|
||||
}
|
||||
candidates = append(candidates, c)
|
||||
}
|
||||
err = rows.Err()
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return stats, err
|
||||
}
|
||||
var failures []error
|
||||
for _, c := range candidates {
|
||||
stats.Examined++
|
||||
done, err := r.process(ctx, c, true)
|
||||
if err != nil {
|
||||
// 仅移动同一版本的未消费记录,不能覆盖其他节点刚提交的确认或待续状态。
|
||||
// 数据库异常不被归类为业务失败;来源保留供下一批恢复。
|
||||
if ctx.Err() == nil {
|
||||
err = errors.Join(err, r.deferCandidate(ctx, c))
|
||||
}
|
||||
failures = append(failures, err)
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
} else if done {
|
||||
if c.kind == "INBOX" {
|
||||
stats.Consumed++
|
||||
} else {
|
||||
stats.Advanced++
|
||||
}
|
||||
}
|
||||
}
|
||||
return stats, errors.Join(failures...)
|
||||
}
|
||||
|
||||
func (r *WorkflowRuntime) deferCandidate(ctx context.Context, c workflowCandidate) error {
|
||||
q := `UPDATE weave_pending_advances SET available_at=DATE_ADD(UTC_TIMESTAMP(6),INTERVAL 30 SECOND),revision=revision+1,updated_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=? AND revision=?`
|
||||
if c.kind == "INBOX" {
|
||||
q = `UPDATE weave_inbox SET available_at=DATE_ADD(UTC_TIMESTAMP(6),INTERVAL 30 SECOND),error_code='RUNTIME_UNAVAILABLE',revision=revision+1,updated_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=? AND revision=? AND state='PENDING'`
|
||||
}
|
||||
_, err := r.db.ExecContext(ctx, q, c.ns, c.id, c.revision)
|
||||
return err
|
||||
}
|
||||
|
||||
func (r *WorkflowRuntime) process(ctx context.Context, c workflowCandidate, skip bool) (bool, error) {
|
||||
done := false
|
||||
err := WithTx(ctx, r.db, func(tx *sql.Tx) error {
|
||||
g, allow, err := lockWorkflowGraph(ctx, tx, c.ns, c.workflow, skip)
|
||||
if err != nil || g == nil {
|
||||
return err
|
||||
}
|
||||
now, err := dbNow(ctx, tx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if c.kind == "INBOX" {
|
||||
done, err = r.consume(ctx, tx, g, c, allow, now)
|
||||
return err
|
||||
}
|
||||
var step core.ID
|
||||
var expected uint64
|
||||
var raw []byte
|
||||
var due time.Time
|
||||
err = tx.QueryRowContext(ctx, `SELECT step_id,expected_step_revision,checkpoint,available_at FROM weave_pending_advances WHERE namespace_id=? AND id=? AND workflow_id=? FOR UPDATE`, c.ns, c.id, c.workflow).Scan(&step, &expected, &raw, &due)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if due.After(now) {
|
||||
return nil
|
||||
}
|
||||
var checkpoint struct {
|
||||
SchemaVersion uint32 `json:"schema_version"`
|
||||
Kind string `json:"kind"`
|
||||
}
|
||||
root := g.steps[workflowRootKey]
|
||||
if json.Unmarshal(raw, &checkpoint) != nil || checkpoint.SchemaVersion != 1 || checkpoint.Kind != "ROOT" || step != root.view.ID || expected > root.view.Revision {
|
||||
return errors.New("workflow checkpoint invalid")
|
||||
}
|
||||
if g.view.State.Terminal() {
|
||||
_, err = tx.ExecContext(ctx, `DELETE FROM weave_pending_advances WHERE namespace_id=? AND id=?`, c.ns, c.id)
|
||||
return err
|
||||
}
|
||||
if !allow || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling {
|
||||
// 暂停/停用仍按原截止计时;核查和失败结清由控制运行器接续,不清除其状态。
|
||||
if (g.view.State == weave.InstanceRunning || g.view.State == weave.InstanceWaiting) && !now.Before(g.view.DeadlineAt) {
|
||||
if err := holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, "WORKFLOW_DEADLINE", now); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := r.audit(ctx, tx, g, c.id, "WORKFLOW_DEADLINE", now); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second))
|
||||
}
|
||||
beforeRevision := g.view.Revision
|
||||
if err := r.progress(ctx, tx, g, allow, now); err != nil {
|
||||
return err
|
||||
}
|
||||
if g.view.Revision == beforeRevision { // 额度不足只延后检查点,不逐秒制造业务审计。
|
||||
return nil
|
||||
}
|
||||
if err := r.audit(ctx, tx, g, c.id, "ROOT_ADVANCE", now); err != nil {
|
||||
return err
|
||||
}
|
||||
done = true
|
||||
return nil
|
||||
})
|
||||
return done && err == nil, err
|
||||
}
|
||||
|
||||
func cloneWorkflowGraph(g *workflowGraph) *workflowGraph {
|
||||
out := *g
|
||||
out.steps = make(map[string]*workflowStep, len(g.steps))
|
||||
for key, row := range g.steps {
|
||||
copy := *row
|
||||
out.steps[key] = ©
|
||||
}
|
||||
return &out
|
||||
}
|
||||
|
||||
// progress 在结果已保存后执行有限决策。额度不足只回滚后继决策,不撤销已确认的结果。
|
||||
func (r *WorkflowRuntime) progress(ctx context.Context, tx *sql.Tx, g *workflowGraph, allow bool, now time.Time) error {
|
||||
if g.view.State.Terminal() || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling {
|
||||
return nil
|
||||
}
|
||||
allow = allow && now.Before(g.view.DeadlineAt)
|
||||
d, err := g.program.Plan(workflowSnapshot(g, allow), 32)
|
||||
if err != nil {
|
||||
var problem *engine.Problem
|
||||
if !errors.As(err, &problem) {
|
||||
return err
|
||||
}
|
||||
return holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, problem.Code, now)
|
||||
}
|
||||
if !d.Complete && !now.Before(g.view.DeadlineAt) {
|
||||
return holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, "WORKFLOW_DEADLINE", now)
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `SAVEPOINT workflow_successors`); err != nil {
|
||||
return err
|
||||
}
|
||||
before := cloneWorkflowGraph(g)
|
||||
err = applyWorkflowDecision(ctx, tx, g, d, now)
|
||||
if errors.Is(err, fault.QueueFull) || errors.Is(err, fault.ApplicationFull) {
|
||||
if _, rollback := tx.ExecContext(ctx, `ROLLBACK TO SAVEPOINT workflow_successors`); rollback != nil {
|
||||
return rollback
|
||||
}
|
||||
*g = *before
|
||||
return saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second))
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !allow && !d.Complete {
|
||||
return saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func holdWorkflow(ctx context.Context, tx *sql.Tx, g *workflowGraph, state weave.InstanceState, reason string, now time.Time) error {
|
||||
if g.view.State.Terminal() {
|
||||
return errors.New("cannot hold completed workflow")
|
||||
}
|
||||
if g.view.State == weave.InstanceNeedsReview {
|
||||
state = weave.InstanceNeedsReview
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, `UPDATE pulse_execution_gates SET paused=TRUE,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=?`, now, g.view.NamespaceID, g.gate); err != nil {
|
||||
return err
|
||||
}
|
||||
r, err := tx.ExecContext(ctx, `UPDATE weave_instances SET state=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, state, now, g.view.NamespaceID, g.view.ID, g.view.Revision)
|
||||
if err := exactlyOne(r, err); err != nil {
|
||||
return err
|
||||
}
|
||||
g.view.State = state
|
||||
g.view.Revision++
|
||||
return emitEvent(ctx, tx, g.view.NamespaceID, g.view.ID, g.view.Revision, "workflow", "workflow.blocked", map[string]any{"state": state, "reason": reason})
|
||||
}
|
||||
|
||||
func (r *WorkflowRuntime) audit(ctx context.Context, tx *sql.Tx, g *workflowGraph, operation core.ID, reason string, now time.Time) error {
|
||||
id, err := core.NewID()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// 迟到证据也占用新的实例历史版本,避免与同一实例版本先前的推进事件冲突。
|
||||
res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, now, g.view.NamespaceID, g.view.ID, g.view.Revision)
|
||||
if err := exactlyOne(res, err); err != nil {
|
||||
return err
|
||||
}
|
||||
g.view.Revision++
|
||||
meta := map[string]any{"state": g.view.State, "revision": g.view.Revision, "node_id": r.node, "operation_id": operation, "reason": reason}
|
||||
raw, err := json.Marshal(meta)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.ExecContext(ctx, "INSERT INTO orbit_audit_events(namespace_id,id,operation_id,actor_id,action,resource_kind,resource_id,reason,`before`,`after`) VALUES (?,?,?,?,'workflow.advance','workflow',?,?,'null',?)", g.view.NamespaceID, id, id, "system:weave:"+string(r.node), g.view.ID, reason, string(raw)); err != nil {
|
||||
return err
|
||||
}
|
||||
return emitEvent(ctx, tx, g.view.NamespaceID, g.view.ID, g.view.Revision, "workflow", "workflow.advanced", meta)
|
||||
}
|
||||
122
internal/storage/mysql/workflow_runtime_graph.go
Normal file
122
internal/storage/mysql/workflow_runtime_graph.go
Normal file
@@ -0,0 +1,122 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||
weave "git.yuntuo.hk/k2safe/orbit/weave/model"
|
||||
)
|
||||
|
||||
// lockWorkflowGraph 只由内部运行器调用。身份→应用额度→排序队列→实例→门控,
|
||||
// 不用启动凭据继续授权;停用身份仍收集在途结果,但禁止创建新工作。
|
||||
func lockWorkflowGraph(ctx context.Context, tx *sql.Tx, ns, id core.ID, skip bool) (*workflowGraph, bool, error) {
|
||||
var enabled bool
|
||||
if err := tx.QueryRowContext(ctx, `SELECT n.enabled AND a.enabled FROM orbit_namespaces n JOIN orbit_applications a ON a.id=n.application_id WHERE n.id=? FOR SHARE`, ns).Scan(&enabled); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if ok, err := lockApplicationResources(ctx, tx, ns, skip); err != nil || !ok {
|
||||
return nil, false, err
|
||||
}
|
||||
v, gate, err := readWorkflowInstance(ctx, tx, ns, id, false)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
version, err := instanceVersion(ctx, tx, v)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
for _, queue := range version.Analysis.Queues {
|
||||
if _, _, err := lockQueueAfterApplication(ctx, tx, ns, queue); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
}
|
||||
var locked core.ID
|
||||
if err := tx.QueryRowContext(ctx, `SELECT id FROM weave_instances WHERE namespace_id=? AND id=? FOR UPDATE`, ns, id).Scan(&locked); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
v, currentGate, err := readWorkflowInstance(ctx, tx, ns, id, false)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if currentGate != gate || v.DefinitionVersionID != version.ID {
|
||||
return nil, false, errors.New("workflow identity changed")
|
||||
}
|
||||
var gatePaused bool
|
||||
if err := tx.QueryRowContext(ctx, `SELECT paused FROM pulse_execution_gates WHERE namespace_id=? AND id=? FOR UPDATE`, ns, gate).Scan(&gatePaused); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
program, err := engine.Compile(version.Specification)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
g := &workflowGraph{view: v, gate: gate, version: version, program: program, steps: map[string]*workflowStep{}}
|
||||
rows, err := tx.QueryContext(ctx, "SELECT s.id,s.`key`,s.kind,s.state,s.revision,s.current_activation_id,a.task_id,a.id,a.state,a.input,a.output,t.id,t.queue_id FROM weave_steps s LEFT JOIN weave_activations a ON a.namespace_id=s.namespace_id AND a.id=s.current_activation_id AND a.workflow_id=s.workflow_id AND a.step_id=s.id LEFT JOIN pulse_tasks t ON t.namespace_id=s.namespace_id AND t.id=a.task_id AND t.owner_kind='workflow_activation' AND t.owner_id=a.id AND t.gate_id=? WHERE s.namespace_id=? AND s.workflow_id=? ORDER BY s.`key` LIMIT 130", gate, ns, id)
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
row := &workflowStep{}
|
||||
var activation, task, queue *core.ID
|
||||
var state sql.NullString
|
||||
var input, output []byte
|
||||
if err := rows.Scan(&row.view.ID, &row.view.Key, &row.view.Kind, &row.view.State, &row.view.Revision, &row.view.ActivationID, &row.view.TaskID, &activation, &state, &input, &output, &task, &queue); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
row.input, row.output = input, output
|
||||
key := row.view.Key
|
||||
node, ok := program.Step(key)
|
||||
if key == workflowRootKey {
|
||||
if row.view.Kind != weave.NodeJoin {
|
||||
return nil, false, errors.New("workflow root kind invalid")
|
||||
}
|
||||
} else if !ok || node.Kind != string(row.view.Kind) {
|
||||
return nil, false, errors.New("workflow step definition mismatch")
|
||||
}
|
||||
inactive := row.view.State == weave.StepBlocked || row.view.State == weave.StepSkipped
|
||||
if inactive && row.view.ActivationID != nil || !inactive && (row.view.ActivationID == nil || activation == nil || *activation != *row.view.ActivationID || state.String != string(row.view.State)) {
|
||||
return nil, false, errors.New("workflow current activation mismatch")
|
||||
}
|
||||
if !inactive && (row.view.Kind == weave.NodeTask) != (row.view.TaskID != nil) {
|
||||
return nil, false, errors.New("workflow task ownership mismatch")
|
||||
}
|
||||
if row.view.TaskID != nil && (task == nil || queue == nil || *task != *row.view.TaskID || node.Task == nil || *queue != node.Task.QueueID) {
|
||||
return nil, false, errors.New("workflow task association invalid")
|
||||
}
|
||||
g.steps[key] = row
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
root := g.steps[workflowRootKey]
|
||||
if len(g.steps) != int(version.Analysis.Steps)+1 || root == nil || root.view.ActivationID == nil {
|
||||
return nil, false, errors.New("workflow step set incomplete")
|
||||
}
|
||||
allow := enabled && !gatePaused && !v.Paused && v.CancelRequestedAt == nil && (v.State == weave.InstanceRunning || v.State == weave.InstanceWaiting)
|
||||
return g, allow, nil
|
||||
}
|
||||
|
||||
func workflowSnapshot(g *workflowGraph, allow bool) engine.Snapshot {
|
||||
s := engine.Snapshot{WorkflowID: g.view.ID, BusinessKey: g.view.BusinessKey, Input: g.view.Input, Steps: map[string]engine.State{}, AllowNew: allow}
|
||||
for key, row := range g.steps {
|
||||
if key != workflowRootKey {
|
||||
s.Steps[key] = engine.State{Status: row.view.State, Input: row.input, Output: row.output}
|
||||
}
|
||||
}
|
||||
return s
|
||||
}
|
||||
|
||||
// saveWorkflowAdvance 保留根步骤的最新版本;重新调度不重置业务截止或任务预算。
|
||||
func saveWorkflowAdvance(ctx context.Context, tx *sql.Tx, g *workflowGraph, due time.Time) error {
|
||||
id, err := core.NewID()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
root := g.steps[workflowRootKey]
|
||||
_, err = tx.ExecContext(ctx, `INSERT INTO weave_pending_advances(namespace_id,id,workflow_id,step_id,expected_step_revision,checkpoint,available_at) VALUES (?,?,?,?,?,'{"schema_version":1,"kind":"ROOT"}',?) ON DUPLICATE KEY UPDATE expected_step_revision=VALUES(expected_step_revision),checkpoint=VALUES(checkpoint),available_at=VALUES(available_at),revision=revision+1,updated_at=UTC_TIMESTAMP(6)`, g.view.NamespaceID, id, g.view.ID, root.view.ID, root.view.Revision, due)
|
||||
return err
|
||||
}
|
||||
489
internal/storage/mysql/workflow_runtime_integration_test.go
Normal file
489
internal/storage/mysql/workflow_runtime_integration_test.go
Normal file
@@ -0,0 +1,489 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
core "git.yuntuo.hk/k2safe/orbit/model"
|
||||
"git.yuntuo.hk/k2safe/orbit/pulse/worker"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/definition"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/engine"
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||
)
|
||||
|
||||
func runtimeSpec(queue core.ID, count int) definition.Specification {
|
||||
spec := workflowTestSpec(queue)
|
||||
spec.TimeoutMS = 120000
|
||||
base := *spec.Body.Steps[0].Task
|
||||
base.TimeoutMS = 30000
|
||||
base.Retry.AttemptTimeoutMS = 20000
|
||||
spec.Body.Steps = nil
|
||||
for i := 0; i < count; i++ {
|
||||
task := base
|
||||
if i > 0 {
|
||||
task.Input = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: fmt.Sprintf("task%d", i-1)}}
|
||||
}
|
||||
spec.Body.Steps = append(spec.Body.Steps, definition.Step{Key: fmt.Sprintf("task%d", i), Kind: "TASK", Task: &task})
|
||||
}
|
||||
spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: fmt.Sprintf("task%d", count-1)}}
|
||||
return spec
|
||||
}
|
||||
func runtimeParallel(spec definition.Specification) definition.Step {
|
||||
parallel := definition.Parallel{}
|
||||
for _, key := range []string{"left", "right"} {
|
||||
task := *spec.Body.Steps[0].Task
|
||||
parallel.Branches = append(parallel.Branches, definition.ParallelBranch{Name: key, Body: definition.Block{Steps: []definition.Step{{Key: key, Kind: "TASK", Task: &task}}, OutputType: spec.InputType, Output: definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: key}}}})
|
||||
}
|
||||
return definition.Step{Key: "join", Kind: "PARALLEL", Parallel: ¶llel}
|
||||
}
|
||||
func newRuntime(t *testing.T, db *sql.DB) *WorkflowRuntime {
|
||||
t.Helper()
|
||||
r, err := NewWorkflowRuntime(db, executionID(t))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return r
|
||||
}
|
||||
func runtimeStart(t *testing.T, f *executionFixture, spec definition.Specification) instance.View {
|
||||
t.Helper()
|
||||
grantWorkflowInstance(t, f)
|
||||
in := publishInstanceSpec(t, f, spec)
|
||||
out, err := instance.New(NewWorkflowInstanceStore(f.db)).Start(f.ctx, f.token, string(executionID(t)), in)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return out.Instance
|
||||
}
|
||||
func runtimeInbox(t *testing.T, f *executionFixture, task core.ID) workflowCandidate {
|
||||
t.Helper()
|
||||
c := workflowCandidate{kind: "INBOX", ns: f.resources.NamespaceID}
|
||||
if err := f.db.QueryRowContext(f.ctx, `SELECT id,workflow_id,revision FROM weave_inbox WHERE namespace_id=? AND source_task_id=? ORDER BY source_revision DESC LIMIT 1`, c.ns, task).Scan(&c.id, &c.workflow, &c.revision); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
func runtimeConsume(t *testing.T, f *executionFixture, r *WorkflowRuntime, task core.ID) {
|
||||
t.Helper()
|
||||
if done, err := r.process(f.ctx, runtimeInbox(t, f, task), false); err != nil || !done {
|
||||
t.Fatalf("consume done=%v err=%v", done, err)
|
||||
}
|
||||
}
|
||||
func runtimeView(t *testing.T, f *executionFixture, id core.ID, state string) instance.View {
|
||||
t.Helper()
|
||||
v, err := instance.New(NewWorkflowInstanceStore(f.db)).Get(f.ctx, f.token, id)
|
||||
if err != nil || string(v.State) != state {
|
||||
t.Fatalf("state=%s want=%s err=%v", v.State, state, err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
func runtimeAdvance(t *testing.T, f *executionFixture, r *WorkflowRuntime, id core.ID) {
|
||||
t.Helper()
|
||||
f.exec(`UPDATE weave_pending_advances SET available_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND workflow_id=?`, f.resources.NamespaceID, id)
|
||||
c := workflowCandidate{kind: "ROOT", ns: f.resources.NamespaceID, workflow: id}
|
||||
if err := f.db.QueryRowContext(f.ctx, `SELECT id,revision FROM weave_pending_advances WHERE namespace_id=? AND workflow_id=?`, c.ns, id).Scan(&c.id, &c.revision); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if done, err := r.process(f.ctx, c, false); err != nil || !done {
|
||||
t.Fatalf("advance done=%v err=%v", done, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIntegrationWorkflowRuntime(t *testing.T) {
|
||||
dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("requires disposable MySQL")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second)
|
||||
defer cancel()
|
||||
db, err := Open(ctx, dsn, 32, 8)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err := Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Run("sequence_duplicate_consumers_and_frozen_result", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
// 当前任务不是事件快照;后续读取不得把改过的当前值误用为先前结果。
|
||||
f.exec(`UPDATE pulse_tasks SET result='{"n":1}' WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||
c := runtimeInbox(t, f, ref.TaskID)
|
||||
var wg sync.WaitGroup
|
||||
failures := make(chan error, 12)
|
||||
committed := make(chan bool, 12)
|
||||
for i := 0; i < 12; i++ {
|
||||
r := newRuntime(t, db)
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); done, e := r.process(ctx, c, false); failures <- e; committed <- done }()
|
||||
}
|
||||
wg.Wait()
|
||||
close(failures)
|
||||
close(committed)
|
||||
for err := range failures {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
n := 0
|
||||
for done := range committed {
|
||||
if done {
|
||||
n++
|
||||
}
|
||||
}
|
||||
if n != 1 || f.count("pulse_tasks", "1=1") != 2 || f.count("orbit_audit_events", "action='workflow.advance'") != 1 {
|
||||
t.Fatal("duplicate successor/audit", n)
|
||||
}
|
||||
second := retryStart(t, f)
|
||||
if !strings.Contains(string(f.get(second.TaskID).Payload), "9007199254740993") {
|
||||
t.Fatal("mutable task value replaced frozen result")
|
||||
}
|
||||
retryFinish(t, f, second, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), second.TaskID)
|
||||
after := runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||
if !strings.Contains(string(after.Output), "9007199254740993") || f.count("pulse_execution_gates", "paused=TRUE") != 1 || f.count("weave_pending_advances", "1=1") != 0 {
|
||||
t.Fatal("final output/gate/checkpoint")
|
||||
}
|
||||
})
|
||||
t.Run("parallel_results_join_once", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||
tail := spec.Body.Steps[0]
|
||||
tail.Key = "tail"
|
||||
spec.Body.Steps = []definition.Step{runtimeParallel(spec), tail}
|
||||
spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "tail"}}
|
||||
v := runtimeStart(t, f, spec)
|
||||
refs := []worker.AttemptRef{retryStart(t, f), retryStart(t, f)}
|
||||
for _, ref := range refs {
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
}
|
||||
cs := []workflowCandidate{runtimeInbox(t, f, refs[0].TaskID), runtimeInbox(t, f, refs[1].TaskID)}
|
||||
var wg sync.WaitGroup
|
||||
errs := make(chan error, 2)
|
||||
for _, c := range cs {
|
||||
r := newRuntime(t, db)
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); _, err := r.process(ctx, c, false); errs <- err }()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if f.count("pulse_tasks", "1=1") != 3 || f.count("weave_steps", "`key`='join' AND state='SUCCEEDED'") != 1 {
|
||||
t.Fatal("parallel duplicate or incomplete join")
|
||||
}
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||
})
|
||||
t.Run("backpressure_preserves_result_and_rolls_back_partial_successors", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||
spec.Body.Steps = append(spec.Body.Steps, runtimeParallel(spec))
|
||||
v := runtimeStart(t, f, spec)
|
||||
f.exec(`UPDATE pulse_queues SET max_unfinished_tasks=1 WHERE namespace_id=?`, f.resources.NamespaceID)
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_steps", "`key`='task0' AND state='SUCCEEDED'") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.count("weave_pending_advances", "1=1") != 1 {
|
||||
t.Fatal("backpressure lost result or partial tasks leaked")
|
||||
}
|
||||
if instanceCount(f, `SELECT unfinished_tasks FROM pulse_queues WHERE namespace_id=?`, f.resources.NamespaceID) != 0 {
|
||||
t.Fatal("savepoint leaked queue quota")
|
||||
}
|
||||
f.exec(`UPDATE pulse_queues SET max_unfinished_tasks=2 WHERE namespace_id=?`, f.resources.NamespaceID)
|
||||
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||
if f.count("pulse_tasks", "1=1") != 3 || f.count("weave_pending_advances", "1=1") != 0 {
|
||||
t.Fatal("durable continuation did not resume")
|
||||
}
|
||||
})
|
||||
t.Run("audit_failure_rolls_back_consume_and_successor", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
f.exec(`CREATE TRIGGER test_reject_workflow_advance BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='injected workflow advance failure'`)
|
||||
_, err := newRuntime(t, db).process(ctx, runtimeInbox(t, f, ref.TaskID), false)
|
||||
f.exec(`DROP TRIGGER test_reject_workflow_advance`)
|
||||
if err == nil || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PENDING'") != 1 || f.count("weave_steps", "`key`='task0' AND state='ACTIVE'") != 1 {
|
||||
t.Fatal("atomic result/advance rollback failed", err)
|
||||
}
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
})
|
||||
t.Run("paused_collects_then_resumes_from_checkpoint", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
// 控制API尚未实现;此处直接建立其需要维护的持久状态,验证运行器不会越过门控。
|
||||
f.exec(`UPDATE weave_instances SET paused=TRUE WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID)
|
||||
f.exec(`UPDATE pulse_execution_gates SET paused=TRUE WHERE namespace_id=?`, f.resources.NamespaceID)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.count("weave_pending_advances", "1=1") != 1 {
|
||||
t.Fatal("paused continuation lost")
|
||||
}
|
||||
f.exec(`UPDATE weave_instances SET paused=FALSE WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID)
|
||||
f.exec(`UPDATE pulse_execution_gates SET paused=FALSE WHERE namespace_id=?`, f.resources.NamespaceID)
|
||||
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||
if f.count("pulse_tasks", "1=1") != 2 {
|
||||
t.Fatal("resume failed")
|
||||
}
|
||||
})
|
||||
t.Run("root_budget_restarts_without_memory", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||
body := spec.Body
|
||||
unselected := body
|
||||
unselected.Steps = nil
|
||||
for i := 0; i < 75; i++ {
|
||||
task := *body.Steps[0].Task
|
||||
unselected.Steps = append(unselected.Steps, definition.Step{Key: fmt.Sprintf("skip%d", i), Kind: "TASK", Task: &task})
|
||||
}
|
||||
unselected.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "skip74"}}
|
||||
zero, one := definition.Value{Kind: "LITERAL", Literal: json.RawMessage(`0`)}, definition.Value{Kind: "LITERAL", Literal: json.RawMessage(`1`)}
|
||||
spec.Body.Steps = []definition.Step{{Key: "choice", Kind: "BRANCH", Branch: &definition.Branch{Cases: []definition.BranchCase{{When: definition.Predicate{Op: "EQ", Left: &zero, Right: &one}, Body: unselected}}, Default: &body}}}
|
||||
spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "choice"}}
|
||||
v := runtimeStart(t, f, spec)
|
||||
if f.count("weave_steps", "state='SKIPPED'") != 31 || f.count("pulse_tasks", "1=1") != 0 {
|
||||
t.Fatal("initial budget")
|
||||
}
|
||||
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||
if f.count("weave_steps", "state='SKIPPED'") != 75 || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_pending_advances", "1=1") != 0 {
|
||||
t.Fatal("bounded restart failed")
|
||||
}
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
runtimeView(t, f, v.ID, "SUCCEEDED")
|
||||
})
|
||||
t.Run("invalid_source_quarantined_without_advancing", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
f.exec(`UPDATE weave_inbox SET snapshot=JSON_SET(snapshot,'$.revision','9999') WHERE namespace_id=?`, f.resources.NamespaceID)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
runtimeView(t, f, v.ID, "NEEDS_REVIEW")
|
||||
if f.count("weave_inbox", "state='QUARANTINED' AND error_code='RESULT_SNAPSHOT_INVALID'") != 1 || f.count("pulse_tasks", "1=1") != 1 || f.count("pulse_execution_gates", "paused=TRUE") != 1 {
|
||||
t.Fatal("corrupt source escaped quarantine")
|
||||
}
|
||||
})
|
||||
t.Run("failed_and_invalid_results_close_gate", func(t *testing.T) {
|
||||
for _, outcome := range []string{"FAILED", "UNKNOWN", "CANCELLED", "INVALID"} {
|
||||
t.Run(outcome, func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
if outcome == "CANCELLED" {
|
||||
// 流程控制API尚未实现;建立其应保存的取消意图,再由真实Worker确认。
|
||||
f.exec(`UPDATE weave_instances SET state='CANCELLING',cancel_requested_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID)
|
||||
f.exec(`UPDATE pulse_execution_gates SET paused=TRUE WHERE namespace_id=?`, f.resources.NamespaceID)
|
||||
f.exec(`UPDATE pulse_tasks SET cancel_requested_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||
}
|
||||
if outcome == "INVALID" {
|
||||
if _, err := f.a.Finish(ctx, f.token, f.binding, executionID(t), worker.FinishInput{Ref: ref, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"wrong":true}`)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
} else {
|
||||
retryFinish(t, f, ref, outcome, false)
|
||||
}
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
expected := "NEEDS_REVIEW"
|
||||
if outcome == "CANCELLED" {
|
||||
expected = "CANCELLING"
|
||||
}
|
||||
runtimeView(t, f, v.ID, expected)
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("pulse_execution_gates", "paused=TRUE") != 1 {
|
||||
t.Fatal("failure did not gate successors")
|
||||
}
|
||||
if outcome == "CANCELLED" && f.count("weave_steps", "`key`='task0' AND state='CANCELLED'") != 1 {
|
||||
t.Fatal("confirmed cancellation was rewritten as failure")
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
t.Run("deadline_does_not_create_successor", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
f.exec(`UPDATE weave_instances SET deadline_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
runtimeView(t, f, v.ID, "NEEDS_REVIEW")
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 {
|
||||
t.Fatal("deadline lost accepted result or created new work")
|
||||
}
|
||||
})
|
||||
t.Run("old_activation_and_cross_namespace_cannot_advance", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 1))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
c := runtimeInbox(t, f, ref.TaskID)
|
||||
other := newExecutionFixture(t, ctx, db)
|
||||
foreign := c
|
||||
foreign.ns = other.resources.NamespaceID
|
||||
if done, err := newRuntime(t, db).process(ctx, foreign, false); err == nil || done {
|
||||
t.Fatal("cross namespace consumer accepted")
|
||||
}
|
||||
// 用独立受理任务构造下一激活;旧激活和旧结果完整保留。
|
||||
next := f.submit(1)
|
||||
activation := executionID(t)
|
||||
var step, gate core.ID
|
||||
if err := db.QueryRowContext(ctx, "SELECT s.id,i.gate_id FROM weave_steps s JOIN weave_instances i ON i.namespace_id=s.namespace_id AND i.id=s.workflow_id WHERE s.namespace_id=? AND s.workflow_id=? AND s.`key`='task0'", c.ns, v.ID).Scan(&step, &gate); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f.exec(`INSERT INTO weave_activations(namespace_id,id,workflow_id,step_id,sequence,state,task_id,input) VALUES (?,?,?,?,2,'ACTIVE',?,'{"n":9007199254740993}')`, c.ns, activation, v.ID, step, next.ID)
|
||||
f.exec(`UPDATE pulse_tasks SET owner_kind=?,owner_id=?,gate_id=? WHERE namespace_id=? AND id=?`, engine.TaskOwnerKind, activation, gate, c.ns, next.ID)
|
||||
f.exec(`UPDATE weave_steps SET current_activation_id=? WHERE namespace_id=? AND id=?`, activation, c.ns, step)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
if f.count("weave_inbox", "state='PROCESSED' AND error_code='STALE_ACTIVATION'") != 1 || f.count("weave_steps", "`key`='task0' AND state='ACTIVE'") != 1 {
|
||||
t.Fatal("old activation advanced current work")
|
||||
}
|
||||
})
|
||||
t.Run("newer_review_result_then_older_success_does_not_clear_review", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
older := runtimeInbox(t, f, ref.TaskID)
|
||||
f.exec(`UPDATE pulse_tasks SET state='NEEDS_REVIEW',effect_uncertain=TRUE,completed_at=NULL,revision=revision+1 WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID)
|
||||
task := f.get(ref.TaskID)
|
||||
if err := WithTx(ctx, db, func(tx *sql.Tx) error {
|
||||
return emitEvent(ctx, tx, f.resources.NamespaceID, ref.TaskID, task.Revision, "task", "task.needs_review", map[string]any{"reason": "test later uncertainty"})
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := newRuntime(t, db)
|
||||
runtimeConsume(t, f, r, ref.TaskID)
|
||||
if done, err := r.process(ctx, older, false); err != nil || !done {
|
||||
t.Fatal(done, err)
|
||||
}
|
||||
runtimeView(t, f, v.ID, "NEEDS_REVIEW")
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "error_code='STALE_RESULT'") != 1 || f.count("orbit_audit_events", "action='workflow.advance'") != 2 {
|
||||
t.Fatal("late result regressed workflow or lost audit")
|
||||
}
|
||||
})
|
||||
t.Run("disabled_application_collects_without_new_work", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
f.exec(`UPDATE orbit_applications SET enabled=FALSE WHERE id=?`, f.resources.ApplicationID)
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.count("weave_pending_advances", "1=1") != 1 {
|
||||
t.Fatal("disabled app dropped result or admitted successor")
|
||||
}
|
||||
f.exec(`UPDATE orbit_applications SET enabled=TRUE WHERE id=?`, f.resources.ApplicationID)
|
||||
runtimeAdvance(t, f, newRuntime(t, db), v.ID)
|
||||
if f.count("pulse_tasks", "1=1") != 2 {
|
||||
t.Fatal("reenabled application did not resume")
|
||||
}
|
||||
})
|
||||
t.Run("poll_pending_success_exhaustion_and_request_failure", func(t *testing.T) {
|
||||
for _, mode := range []string{"PENDING", "DONE", "EXHAUSTED", "INVALID", "REQUEST_FAILED"} {
|
||||
t.Run(mode, func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
spec := runtimeSpec(f.resources.QueueID, 1)
|
||||
task := *spec.Body.Steps[0].Task
|
||||
task.TimeoutMS = 10000
|
||||
task.Retry.AttemptTimeoutMS = 5000
|
||||
task.ResultType = definition.Type{Kind: "STRING", Enum: []string{"PENDING", "DONE"}}
|
||||
poll := definition.Poll{Task: task, MaxRounds: 3, TimeoutMS: 30000, IntervalMS: 1000, Pending: []string{"PENDING"}, Succeeded: []string{"DONE"}}
|
||||
if mode == "EXHAUSTED" {
|
||||
poll.MaxRounds = 1
|
||||
}
|
||||
spec.Body = definition.Block{Steps: []definition.Step{{Key: "query", Kind: "POLL", Poll: &poll}}, OutputType: task.ResultType, Output: definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "query"}}}
|
||||
v := runtimeStart(t, f, spec)
|
||||
ref := retryStart(t, f)
|
||||
result, outcome, errorCode := mode, "SUCCEEDED", ""
|
||||
if mode == "EXHAUSTED" {
|
||||
result = "PENDING"
|
||||
}
|
||||
if mode == "REQUEST_FAILED" {
|
||||
outcome, errorCode = "FAILED", "REMOTE_FAILURE"
|
||||
}
|
||||
raw, _ := json.Marshal(result)
|
||||
if _, err := f.a.Finish(ctx, f.token, f.binding, executionID(t), worker.FinishInput{Ref: ref, Outcome: outcome, Result: raw, ErrorCode: errorCode}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
runtimeConsume(t, f, newRuntime(t, db), ref.TaskID)
|
||||
expected := "NEEDS_REVIEW"
|
||||
if mode == "DONE" {
|
||||
expected = "SUCCEEDED"
|
||||
}
|
||||
if mode == "PENDING" {
|
||||
expected = "WAITING"
|
||||
}
|
||||
runtimeView(t, f, v.ID, expected)
|
||||
if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_poll_states", "used_rounds=1 AND current_task_id IS NULL") != 1 {
|
||||
t.Fatal("poll reset budget or started overlapping work")
|
||||
}
|
||||
if mode == "PENDING" {
|
||||
if f.count("weave_poll_states", "next_poll_at IS NOT NULL AND resolved_at IS NULL") != 1 || f.count("weave_steps", "`key`='query' AND state='WAITING'") != 1 {
|
||||
t.Fatal("pending poll mistaken for completion")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
t.Run("scanner_backoff_and_busy_application", func(t *testing.T) {
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2))
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
c := runtimeInbox(t, f, ref.TaskID)
|
||||
r := newRuntime(t, db)
|
||||
lock, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := lockApplicationResources(ctx, lock, c.ns, false); err != nil {
|
||||
lock.Rollback()
|
||||
t.Fatal(err)
|
||||
}
|
||||
call, cancel := context.WithTimeout(ctx, time.Second)
|
||||
done, err := r.process(call, c, true)
|
||||
cancel()
|
||||
lock.Rollback()
|
||||
if err != nil || done {
|
||||
t.Fatal("busy application not skipped", done, err)
|
||||
}
|
||||
f.exec(`UPDATE weave_inbox SET available_at='2000-01-01' WHERE namespace_id=?`, c.ns)
|
||||
f.exec(`CREATE TRIGGER test_reject_runtime_tick BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='injected scanner transaction failure'`)
|
||||
stats, err := r.Tick(ctx, 1)
|
||||
f.exec(`DROP TRIGGER test_reject_runtime_tick`)
|
||||
if err == nil || stats.Consumed != 0 || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PENDING' AND available_at>UTC_TIMESTAMP(6) AND error_code='RUNTIME_UNAVAILABLE'") != 1 {
|
||||
t.Fatal("scanner lost rollback or failure backoff", stats, err)
|
||||
}
|
||||
if done, err := r.process(ctx, runtimeInbox(t, f, ref.TaskID), false); err != nil || done {
|
||||
t.Fatal("consumer ignored durable backoff", done, err)
|
||||
}
|
||||
f.exec(`UPDATE weave_inbox SET available_at='2000-01-01' WHERE namespace_id=?`, c.ns)
|
||||
stats, err = r.Tick(ctx, 1)
|
||||
if err != nil || stats.Consumed != 1 || f.count("pulse_tasks", "1=1") != 2 {
|
||||
t.Fatal("scanner did not recover", stats, err)
|
||||
}
|
||||
// 旧失败回调不能延后或改写已被新事务确认的记录。
|
||||
if err := r.deferCandidate(ctx, c); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if f.count("weave_inbox", "state='PROCESSED' AND error_code='TASK_RESULT'") != 1 {
|
||||
t.Fatal("old deferral overwrote acknowledgement")
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
package mysql
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/orbit/weave/instance"
|
||||
mysqldriver "github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
// 真实运行Orbit控制进程;强杀唯一控制节点后写入结果,再由两个全新进程竞争接续。
|
||||
// 这里只验证内部实例受理+Pulse执行存储+真实控制进程,公共实例HTTP/SDK另行验收。
|
||||
func TestIntegrationWorkflowRuntimeProcess(t *testing.T) {
|
||||
dsn, binary := os.Getenv("ORBIT_TEST_MYSQL_DSN"), os.Getenv("ORBIT_TEST_SERVER_BINARY")
|
||||
if dsn == "" || binary == "" {
|
||||
t.Skip("requires disposable MySQL and built Orbit binary")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
admin, err := Open(ctx, dsn, 2, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer admin.Close()
|
||||
name := "orbit_workflow_process_" + string(executionID(t))
|
||||
if _, err := admin.ExecContext(ctx, "CREATE DATABASE `"+name+"`"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer func() {
|
||||
cleanup, stop := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer stop()
|
||||
if _, err := admin.ExecContext(cleanup, "DROP DATABASE `"+name+"`"); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
cfg, err := mysqldriver.ParseDSN(dsn)
|
||||
if err != nil {
|
||||
t.Fatal("invalid disposable DSN")
|
||||
}
|
||||
cfg.DBName = name
|
||||
db, err := Open(ctx, cfg.FormatDSN(), 16, 4)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
if err := Migrate(ctx, db); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := newExecutionFixture(t, ctx, db)
|
||||
v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 3))
|
||||
dir := t.TempDir()
|
||||
type controller struct {
|
||||
cmd *exec.Cmd
|
||||
done chan error
|
||||
log *os.File
|
||||
stopped bool
|
||||
}
|
||||
var processes []*controller
|
||||
stop := func(p *controller) {
|
||||
t.Helper()
|
||||
if p.stopped {
|
||||
return
|
||||
}
|
||||
p.stopped = true
|
||||
if err := p.cmd.Process.Kill(); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
select {
|
||||
case <-p.done:
|
||||
case <-time.After(10 * time.Second):
|
||||
t.Error("controller did not exit after SIGKILL")
|
||||
}
|
||||
p.log.Close()
|
||||
}
|
||||
defer func() {
|
||||
for _, p := range processes {
|
||||
stop(p)
|
||||
}
|
||||
}()
|
||||
start := func(label string) *controller {
|
||||
t.Helper()
|
||||
log, err := os.Create(filepath.Join(dir, label+".log"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, binary)
|
||||
cmd.Env = append(os.Environ(), "ORBIT_MYSQL_DSN="+cfg.FormatDSN(), "ORBIT_HTTP_ADDR=127.0.0.1:0")
|
||||
cmd.Stdout, cmd.Stderr = log, log
|
||||
if err := cmd.Start(); err != nil {
|
||||
log.Close()
|
||||
t.Fatal(err)
|
||||
}
|
||||
p := &controller{cmd: cmd, done: make(chan error, 1), log: log}
|
||||
processes = append(processes, p)
|
||||
go func() { p.done <- cmd.Wait() }()
|
||||
return p
|
||||
}
|
||||
wait := func(check func() bool, label string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(20 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if check() {
|
||||
return
|
||||
}
|
||||
for _, p := range processes {
|
||||
if p.stopped {
|
||||
continue
|
||||
}
|
||||
select {
|
||||
case err := <-p.done:
|
||||
p.stopped = true
|
||||
t.Fatalf("controller exited unexpectedly (%s): %v", label, err)
|
||||
default:
|
||||
}
|
||||
}
|
||||
time.Sleep(25 * time.Millisecond)
|
||||
}
|
||||
t.Fatalf("timed out waiting for %s", label)
|
||||
}
|
||||
first := start("only-controller")
|
||||
ref := retryStart(t, f)
|
||||
retryFinish(t, f, ref, "SUCCEEDED", false)
|
||||
wait(func() bool { return f.count("pulse_tasks", "1=1") == 2 }, "first successor")
|
||||
second := retryStart(t, f)
|
||||
stop(first) // 必须等Wait确认死亡,才把下一结果写入;排除活节点代为处理。
|
||||
retryFinish(t, f, second, "SUCCEEDED", false)
|
||||
if f.count("weave_inbox", "source_task_id=? AND state='PENDING'", second.TaskID) != 1 || f.count("pulse_tasks", "1=1") != 2 {
|
||||
t.Fatal("result was not left durable while every controller was stopped")
|
||||
}
|
||||
start("replacement-a")
|
||||
start("replacement-b")
|
||||
wait(func() bool { return f.count("pulse_tasks", "1=1") == 3 }, "two-node continuation")
|
||||
third := retryStart(t, f)
|
||||
retryFinish(t, f, third, "SUCCEEDED", false)
|
||||
api := instance.New(NewWorkflowInstanceStore(db))
|
||||
wait(func() bool {
|
||||
current, err := api.Get(ctx, f.token, v.ID)
|
||||
return err == nil && current.State == "SUCCEEDED"
|
||||
}, "completed workflow")
|
||||
if f.count("pulse_tasks", "1=1") != 3 || f.count("weave_inbox", "state='PROCESSED'") != 3 || f.count("orbit_audit_events", "action='workflow.advance'") != 3 || f.count("weave_pending_advances", "1=1") != 0 {
|
||||
t.Fatal("process restart duplicated work or lost confirmation")
|
||||
}
|
||||
t.Log("sole Orbit process SIGKILL confirmed; result committed with all controllers stopped; two new Orbit processes completed exactly three tasks and three result confirmations")
|
||||
}
|
||||
@@ -9,8 +9,13 @@ TASK_TEST_DIR=$(mktemp -d "${TMPDIR:-/tmp}/orbit-model-test.XXXXXX")
|
||||
TASK_SERVER_PID=
|
||||
cleanup() {
|
||||
if [ -n "$TASK_SERVER_PID" ]; then kill "$TASK_SERVER_PID" >/dev/null 2>&1 || true; wait "$TASK_SERVER_PID" 2>/dev/null || true; fi
|
||||
"$TASK_DOCKER" rm -f "$TASK_TEST_NAME" >/dev/null 2>&1 || true
|
||||
rm -rf "$TASK_TEST_DIR"
|
||||
"$TASK_DOCKER" stop "$TASK_TEST_NAME" >/dev/null 2>&1 || true
|
||||
"$TASK_DOCKER" container rm "$TASK_TEST_NAME" >/dev/null 2>&1 || true
|
||||
# 仅清理由本脚本创建的普通文件;出现意外子目录时保留目录供核查。
|
||||
for task_artifact in "$TASK_TEST_DIR"/*; do
|
||||
if [ -f "$task_artifact" ]; then unlink "$task_artifact"; fi
|
||||
done
|
||||
rmdir "$TASK_TEST_DIR"
|
||||
}
|
||||
trap cleanup EXIT HUP INT TERM
|
||||
umask 077
|
||||
@@ -32,6 +37,7 @@ export ORBIT_TEST_MYSQL_DSN="root:$TASK_DB_PASSWORD@tcp(127.0.0.1:$TASK_DB_PORT)
|
||||
"$TASK_GO" test -race -count=1 -run Integration -v ./internal/storage/mysql
|
||||
|
||||
"$TASK_GO" build -o "$TASK_TEST_DIR/orbit" ./cmd/orbit
|
||||
ORBIT_TEST_SERVER_BINARY="$TASK_TEST_DIR/orbit" "$TASK_GO" test -race -count=1 -run '^TestIntegrationWorkflowRuntimeProcess$' -v ./internal/storage/mysql
|
||||
"$TASK_GO" build -o "$TASK_TEST_DIR/orbit-migrate" ./cmd/orbit-migrate
|
||||
"$TASK_GO" build -o "$TASK_TEST_DIR/orbit-admin" ./cmd/orbit-admin
|
||||
ORBIT_MYSQL_DSN="$ORBIT_TEST_MYSQL_DSN" "$TASK_TEST_DIR/orbit-migrate" status
|
||||
|
||||
Reference in New Issue
Block a user