diff --git a/AGENTS.md b/AGENTS.md index 37c3f29..a076191 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -31,10 +31,14 @@ - P4定义发布使用schema_version=1有限结构,词法作用域禁止跨分支/前向读取,任务策略显式冻结;SDK类型独立。发布按认证→应用额度→操作身份→定义→排序队列持锁;原键与同版本去重仍校验当前权限。定义/版本/revision/审计/回执同事务;同版本不同内容不得覆盖。保留精确数字,校验存储原文与结构摘要;不得把发布通过或DSL支持误称为实例执行已实现。 -- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归(docs/reports/phase-4-definitions.md)。实例启动/固定版本/首批任务同事务的内部存储已补齐;继续内部事件消费接续、顺序与组合自动推进、持久化等待及实例公共SDK;P4整体和P5–P7仍未完成。 +- P4定义子项已通过真实MySQL、HTTP丢回执、独立SDK和全量回归(docs/reports/phase-4-definitions.md)。实例启动/固定版本/首批任务同事务的内部存储已补齐;内部事件消费、顺序与组合推进及根接续已接入;继续持久化等待及实例公共SDK;P4整体和P5–P7仍未完成。 - 内部结果交付使用workflow_activation所有者并固定激活ID,TASK经activation.task_id、POLL经轮次校验归属;weave_inbox来源及独立快照与Pulse结果同事务,先于外部结果流行锁,不在结果路径反锁实例/步骤/订阅。迁移000017隔离缺快照历史,禁止用当前任务重建旧事件。 -- weave/engine仅计算有限决策,完整实例存储运行器尚未接入。Plan最多32个变更,More必须持久化待续;耗尽变更预算仍须遍历其他已活动分支,不能误判在途工作不可达。冻结分支选择,精确数值,不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务,不能把纯快照往返测试当作真实进程恢复。 +- weave/engine仅计算有限决策;实例受理、Inbox消费与根接续已接入,完整等待/控制运行器继续开发。Plan最多32个变更,More必须持久化待续;耗尽变更预算仍须遍历其他已活动分支,不能误判在途工作不可达。冻结分支选择,精确数值,不以PENDING查询结果完成POLL。后续消费确认/当前激活校验/新任务或检查点必须同事务,不能把纯快照往返测试当作真实进程恢复。 -- 实例内部受理已实现,公共实例HTTP/SDK及消费/等待运行器尚未开放。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作,所有失败同事务回滚。首批推进递增revision至2,原键返回冻结回执且仍核验当前权限,不重算截止。 +- 实例内部受理已实现,公共实例HTTP/SDK及等待运行器尚未开放;结果消费和根待续扫描已接入后台。Start按认证→应用额度→原身份→排序队列建立门控/固定版本/全部步骤/首批工作,所有失败同事务回滚。首批推进递增revision至2,原键返回冻结回执且仍核验当前权限,不重算截止。 - 实例根步骤`$root`/JOIN不属于用户定义,Steps查询排除它;More对应根步骤版本化持久检查点。TASK直接关联激活task_id,POLL通过轮次关联,实际任务截止必须受实例/查询绝对截止限制。Wait名称及契约版本冻结,旧NULL等待须核实,不能盲目解释;新关联512字节、存储兼容历史764字节。 + +- 结果消费按身份→应用额度→排序队列→实例→门控→步骤/Inbox持锁,禁止先锁Inbox再锁父资源。来源使用独立冻结快照,旧激活/低版本留痕;数据库故障不归类为业务失败。消费确认、步骤、后继或根待续、事件及审计同事务。 +- 后继满额用SAVEPOINT回滚整轮新工作,保留已接受结果和待续;恢复必须重读持久图。多次根推进使用独立系统审计操作身份;迟到证据也递增实例历史版本,禁止复用已有workflow.advanced唯一身份。 +- 本轮消费运行器和真实Orbit强杀/双节点接管已验证,见docs/reports/phase-4-runtime.md。POLL仅当前轮次分类与next_poll_at保存;下一轮派发、等待到期/信号、FAIL_AFTER_SETTLED分支结清、控制API和实例SDK仍须完成,不得按部分自动运行宣称P4/P5完成。 diff --git a/README.md b/README.md index f1c9659..f6ae344 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Orbit 是面向多个业务项目的通用任务调度与流程编排平台, 已完成 P0–P3 平台功能阶段验收:35 个数据模型、版本化迁移、凭据及范围校验、队列与任务 API、WSS 执行、独立 SDK、自动 Worker 和 [执行尝试/证据查询](docs/api/execution-history.md)。[Worker Schema](docs/api/worker-schema.md) 与真实进程强杀恢复回归已通过;[安全重试](docs/api/retries.md)、[定时计划](docs/api/schedules.md)、[运行中取消](docs/api/cancellation.md) 、[队列资源控制](docs/api/queues.md) 、[应用共享配额](docs/api/application-resources.md) 、[资源键互斥](docs/api/keyed-concurrency.md) 和 [优先级/节点公平派发](docs/api/priority-and-fairness.md) 已接入服务端和 SDK,[可靠结果订阅](docs/api/subscriptions.md) 已实现持久化核心、HTTP/WSS、管理 SDK 与示例,[消费 SDK](docs/sdk/event-consumer.md) 和 [独立 MySQL Inbox 示例](docs/examples/result-inbox.md) 已实现;[批量任务受理](docs/api/batch-tasks.md) 已提供逐项幂等、失败分类与 SDK 示例;[人工核查与重试](docs/api/task-recovery.md) 已提供带证据的管理接口、关联新任务、审计查询及运行示例;[业务键与时间范围查询](docs/api/task-queries.md) 已提供过滤、分页与SDK场景;[任务来源查询](docs/api/task-source.md) 已提供人工重试父任务、计划触发记录与固定版本;[业务事务Outbox](docs/examples/task-outbox.md) 已提供独立业务库、稳定提交身份、发送恢复与CLI;[相对延迟执行](docs/examples/delayed-task.md)已提供数据库计时及原请求恢复;完整范围核对见[P3验收](docs/reports/phase-3.md),流程引擎与生产验证按 [持续开发计划](docs/08-development-plan.md) 继续实现。 -P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现;已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。Inbox消费、持久化等待运行器和实例公共API/SDK继续开发,发布定义不会启动任务。 +P4已实现[流程定义与不可变版本发布](docs/api/workflow-definitions.md)、[结构版本1](docs/api/workflow-definition-format.md)及[SDK发布例子](docs/examples/workflow-definition.md)。[有限步骤决策器与内部结果交付](docs/10-workflow-engine-core.md)已实现;已补[实例受理及首批工作存储](docs/11-workflow-admission.md)。[结果消费、顺序/分支/并行推进及根检查点接续](docs/12-workflow-runtime.md)已接入后台运行器。持久化等待、后续查询轮次、失败结清及实例公共API/SDK继续开发,发布定义不会启动任务。 数据库禁止外键,关联一致性通过服务端事务校验,索引、唯一约束和 CHECK 保留;readiness 同时检查迁移及零外键要求。这不表示完整平台已具备生产服务能力。验证证据见 [P0](docs/reports/phase-0.md)、[P1](docs/reports/phase-1.md)、[P2 阶段验收](docs/reports/phase-2.md)、[安全重试](docs/reports/phase-3-safe-retry.md)、[计划调度](docs/reports/phase-3-schedules.md)、[取消与无外键回归](docs/reports/phase-3-cancellation.md) 及 [队列资源回归](docs/reports/phase-3-queue-resources.md)。 diff --git a/cmd/orbit/main.go b/cmd/orbit/main.go index 913e62a..d502da6 100644 --- a/cmd/orbit/main.go +++ b/cmd/orbit/main.go @@ -81,6 +81,31 @@ func run() error { } httpserver.RegisterSchedules(server, schedule.New(schedules)) httpserver.RegisterWorkflowDefinitions(server, definition.New(mysql.NewWorkflowDefinitionStore(db))) + workflows, err := mysql.NewWorkflowRuntime(db, node) + if err != nil { + return err + } + workflowCtx, stopWorkflows := context.WithCancel(root) + workflowDone := make(chan struct{}) + go func() { + defer close(workflowDone) + ticker := time.NewTicker(time.Second) + defer ticker.Stop() + for { + select { + case <-workflowCtx.Done(): + return + case <-ticker.C: + call, cancel := context.WithTimeout(workflowCtx, 5*time.Second) + _, err := workflows.Tick(call, 100) + cancel() + if err != nil && workflowCtx.Err() == nil { + slog.Error("workflow batch failed", "node_id", node) + } + } + } + }() + defer func() { stopWorkflows(); <-workflowDone }() scheduleCtx, stopSchedules := context.WithCancel(root) scheduleDone := make(chan struct{}) go func() { diff --git a/docs/08-development-plan.md b/docs/08-development-plan.md index 5b1aa02..cfe2ad7 100644 --- a/docs/08-development-plan.md +++ b/docs/08-development-plan.md @@ -28,7 +28,7 @@ | P1 任务受理闭环 | 应用凭据与作用域权限、队列、幂等提交、查询、待执行取消、受理事件;SDK 客户端 | API/SDK 创建查询闭环;重复键与内容冲突、响应未确认重报、跨命名空间拒绝、参数边界 | 已完成并通过阶段回归(见 P1 报告;真实断路/提交 ACK 丢失注入留后续故障阶段) | | P2 Worker 执行闭环 | WSS 接入、能力/容量、领取与开始、租约、续租、结果确认、排空与恢复;Worker SDK | 真 Worker 执行;失联/重连/重复确认/旧尝试拒绝;业务效果未知转核查;节点故障接管 | 已完成并通过阶段回归:服务端、自动 SDK、历史查询、完整 Schema 与真实进程强杀恢复;测试拓扑和剩余生产故障矩阵见 P2 验收报告 | | P3 Pulse 完整能力 | 安全重试、延时、Cron/固定间隔、漏跑策略、计划版本、运行中取消、并发许可、配额/背压、可靠结果订阅、批量受理、独立任务核查/人工重试、查询筛选与事务Outbox接入 | 多实例领取/触发竞争、重试上限、取消竞争、队列隔离、积压恢复、事件重复交付 | 已完成平台功能阶段验收:[原范围逐项核对](reports/phase-3.md)、[本轮完整回归](reports/phase-3-delay-recovery.md)。补齐相对延迟、改参数关联示例和四种重试全停恢复场景;20个可运行示例全部通过。全平台生产门槛及SDK首版交付剩余项仍按P6–P7推进 | -| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容;端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现,已通过[子项验收](reports/phase-4-definitions.md);已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md);[实例受理及首批事务](reports/phase-4-admission.md)已实现;事件消费推进、持久化等待运行器和实例公共API/SDK仍待实现 | +| P4 Weave 主流程 | SDK 定义构建与发布校验、实例、顺序/分支/结构化并行汇合、信号/定时等待、版本绑定 | 重启接续、早到/重复信号、并行竞争、内部事件重放、发布版本兼容;端到端通用流程 | 定义格式/词法类型校验、不可变发布、独立SDK及六类节点定义例子已实现,已通过[子项验收](reports/phase-4-definitions.md);已补[有限决策器及内部结果交付](reports/phase-4-engine-core.md);[实例受理及首批事务](reports/phase-4-admission.md)已实现;[结果消费和根接续](reports/phase-4-runtime.md)已接入后台,顺序/分支/并行及真实进程恢复已验证;持久化等待、完整控制和实例公共API/SDK继续开发 | | P5 Weave 恢复与查询 | 有界查询、暂停继续、取消、人工核查恢复、控制门控与审计 | 轮次与尝试独立预算;暂停/开始/回执/超时竞争;恢复保留预算;未知结果不盲目重复 | 待开发 | | P6 生产验证与运维 | 完整监控、凭据轮换、备份恢复、滚动升级、运行手册、可复现负载和故障工具 | 对照生产验收清单执行,报告环境/指标/遗留项;容量、RPO/RTO 按真实拓扑验证 | 待开发 | | P7 首版交付核对 | SDK 兼容与发布准备、完整 API 手册、场景示例、全量回归及交付清单 | 新接入者按文档跑通全部示例;所有首版能力可追溯到实现与测试;报告实际生产边界 | 待开发 | diff --git a/docs/10-workflow-engine-core.md b/docs/10-workflow-engine-core.md index 1460f1c..e017786 100644 --- a/docs/10-workflow-engine-core.md +++ b/docs/10-workflow-engine-core.md @@ -1,6 +1,6 @@ # Weave 执行决策与内部结果交付 -状态:P4 内部基础已实现。已接通Pulse结果到Weave持久化Inbox,并提供有限步骤决策器;[实例受理和首批存储适配](11-workflow-admission.md)已补齐。**Inbox消费推进、定时/信号运行器及实例公共API/SDK尚未实现**。不能把本页能力解释成流程已经可以端到端执行。 +状态:P4 内部基础已实现。已接通Pulse结果到Weave持久化Inbox,并提供有限步骤决策器;[实例受理和首批存储适配](11-workflow-admission.md)已补齐。[Inbox消费推进及根检查点运行器](12-workflow-runtime.md)已接入;定时/信号运行器及实例公共API/SDK仍待实现。不能把本页能力解释成完整公开端到端流程已交付。 ## 1. 为什么结果要独立持久化 @@ -23,7 +23,7 @@ flowchart LR 已实现的内部交付触发为`task.succeeded`、`task.failed`、`task.cancelled`、`task.needs_review`。任务暂时失败并等待自动重试,不新增流程激活或内部结果事件。相同完成回执重报不产生第二个事件;同一来源任务状态版本还有独立唯一约束。 -发布后使用旧激活的在途结果仍保留原激活身份;生产者不自行判定是否推进当前激活。未来消费者须识别旧激活/旧轮次并留痕,不能将其用于当前工作。 +发布后使用旧激活的在途结果仍保留原激活身份;生产者不自行判定是否推进当前激活。当前消费者识别旧激活/旧轮次并留痕,不能将其用于当前工作。 生产者在现有结果快照查询中一起读取所有者和关联标记;普通结果不增加一次单独网络查询。流程关联核验使用只读关联查询,不在已持有任务锁后再锁实例、步骤或对外订阅。内部Inbox写入发生在外部结果流行锁之前,保持既有锁顺序。 @@ -74,7 +74,7 @@ v16及以前的未处理记录缺少冻结来源,升级后统一标为QUARANTI ## 4. 后续存储适配必须满足的条件 -以下是存储运行器的完整要求;第1项初始事务已实现并按[受理报告](reports/phase-4-admission.md)验证,其余消费/运行要求尚未完成,不能按纯单测宣称已完成: +以下保留存储运行器的完整要求。第1项已按[受理报告](reports/phase-4-admission.md)验证;第2–4项结果消费和根接续已接入[实际运行器](12-workflow-runtime.md),第5–7项的完整等待、控制、接口和SDK仍继续实现: 1. 启动时认证、固定不可变版本/输入和策略、校验全部队列、计算实例截止,并在同事务创建实例、门控、首批激活/任务及审计回执。相同身份返回原实例。 2. 消费者领取内部事件后核对命名空间、来源任务版本、激活/轮次及当前步骤版本;消费确认、步骤变化、后续任务或持久化待续记录同事务提交。 diff --git a/docs/11-workflow-admission.md b/docs/11-workflow-admission.md index 6ae3462..dc88d3a 100644 --- a/docs/11-workflow-admission.md +++ b/docs/11-workflow-admission.md @@ -1,6 +1,6 @@ # Weave 实例受理与首批工作 -状态:内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。**尚未开放实例HTTP路由、启动SDK或命令行入口;后台Inbox消费、待续扫描和等待运行器尚未接通。** Weave相关公开SDK目前仅支持定义发布。不能将首批任务可执行解释成整个流程已能自动完成。 +状态:内部Go服务与MySQL存储适配已实现实例Start/Get/Steps、固定版本、首批决策落库和可靠结果交付。[后台Inbox消费和根待续扫描](12-workflow-runtime.md)已接通。**尚未开放实例HTTP路由、启动SDK或命令行入口;等待运行器、后续轮询和完整控制仍继续开发。** Weave相关公开SDK目前仅支持定义发布。TASK及组合节点已可自动推进,不能将此解释成全部等待/控制能力已实现。 ## 1. 受理事务 @@ -23,7 +23,7 @@ | --- | --- | | TASK | 当前激活、精确输入、所属Pulse任务、流程门控及`workflow_activation`所有者 | | BRANCH | 首次选择的case索引;未选中节点分批SKIPPED,不创建业务激活 | -| PARALLEL | 父激活、独立分支初始工作;完成汇合仍需要后续消费推进 | +| PARALLEL | 父激活、独立分支初始工作;后续结果消费运行器负责完成汇合 | | SIGNAL | WAITING激活、信号名称/内容版本/精确业务关联、固定截止;不分配Worker | | TIMER | WAITING激活和数据库时间计算的等待截止;不生成业务任务 | | POLL | 固定输入、查询状态、首轮记录和唯一当前任务;used_rounds=1;可选最终信号等待与查询截止一致 | @@ -49,14 +49,14 @@ Get返回当前实例视图;Start重报返回受理时冻结的原始视图, | 活跃实例 | 每个namespace最多10000个RUNNING/WAITING/CANCELLING/NEEDS_REVIEW实例;终态不计入;满额仍可恢复原受理回执 | | 信号关联 | 新受理最多512字节;数据库兼容旧191字符最长764字节,升级不截断历史 | | 截止 | 数据库时间生成;任务、查询与等待均不超过实例总截止 | -| 自动运行 | 本阶段尚未实现Inbox消费、待续扫描、信号消费、定时到期处理或后续轮询 | +| 自动运行 | Inbox消费和根待续扫描已接入;信号消费、定时到期处理及后续轮询仍待实现 | 迁移000018为Wait增加signal_name和payload_version两个带中文注释的可空字段;NULL仅表示TIMER或尚未核实的历史元数据,未来消费者不得把历史NULL自动解释为有效的新信号等待。correlation_key转为精确字节存储;实例business_key改为utf8mb4_0900_bin。35个实体、含公共Record的295个直接持久化字段声明,全部有中文注释,继续保持零外键。 ## 5. 下一步接入 -1. 锁定实例、核验来源快照/当前激活,消费确认与后续步骤/任务/待续记录同事务;处理重复、迟到、旧轮次和未知结果。 -2. 实现有界待续扫描与数据库时间驱动的等待,恢复时保留原截止和次数;开始扫描前补齐暂停/截止/凭据所属应用停用的判定。 +1. 结果消费与根待续运行器已按[运行器报告](reports/phase-4-runtime.md)验证;继续补齐数据库时间驱动的信号/定时等待。 +2. 已有扫描核验暂停、截止与所属应用停用;继续后续轮次派发、回执竞争及失败分支结清,恢复保留原截止和次数。 3. 接通实例公共HTTP、独立SDK、来源查询和可运行端到端场景,同时实施真实节点退出与接管回归。 本阶段验证见[实例受理回归记录](reports/phase-4-admission.md)。纯受理测试不是流程运行验收,P4–P7仍按[完整计划](08-development-plan.md)推进。 diff --git a/docs/12-workflow-runtime.md b/docs/12-workflow-runtime.md new file mode 100644 index 0000000..205293b --- /dev/null +++ b/docs/12-workflow-runtime.md @@ -0,0 +1,67 @@ +# Weave 结果消费与自动推进 + +状态:内部实例的TASK结果消费、顺序、分支、有限并行汇合和根检查点接续已接入Orbit后台运行器。实例公共HTTP/SDK仍未开放。信号/定时等待、下一轮查询、失败后的分支取消结清及人工恢复仍继续开发,不能将本阶段等同于完整Weave交付。 + +## 1. 后台运行和事务边界 + +每个Orbit进程每秒调用一次`WorkflowRuntime.Tick`,一批最多100个候选、调用超时5秒。候选来自`weave_inbox`的PENDING到期索引和`weave_pending_advances`的到期索引,各候选独立事务。选择候选不授予执行权;同一应用繁忙时跳过,后续扫描重读数据库。 + +事务锁顺序是身份共享锁 → 应用额度排他锁 → 固定版本涉及的排序队列 → 实例 → 执行门控 → 激活/步骤及消费记录。不能先领取并持有Inbox锁,再倒过来锁应用或任务。运行器不向业务系统发请求,不需要Redis或额外消息队列;业务执行仍由Pulse授权Worker。 + +```mermaid +flowchart LR + A[读取到期Inbox或根检查点] --> B[锁定并核验实例图] + B --> C[读取冻结结果并校验来源] + C --> D[更新步骤和计算有限决策] + D --> E[创建后继工作或保存待续] + E --> F[确认消费并写审计] + F --> G[同一事务提交] +``` + +每轮最多32个结构变更。达到预算时保留根步骤版本及ROOT检查点,由后续节点继续;后续计算仍使用固定版本、原输入和已保存结果。旧检查点只能基于当前持锁快照重新计算,不能覆盖更高步骤版本;超前版本或错误根身份拒绝执行。 + +## 2. 结果、去重与审计 + +消费只使用内部Inbox的独立结果快照。校验结构版本、任务和队列、来源状态版本、原事件身份/类型/版本、激活所属实例和步骤、Pulse所有者及门控、尝试归属;POLL额外校验原轮次。任务当前结果不能替换历史快照。 + +| 情况 | 处理 | +| --- | --- | +| 已确认或已隔离的记录再次被扫描 | 不再次更新步骤、创建任务或写相同消费审计 | +| 旧激活、已处理的旧轮次或低于已处理结果版本 | 保留来源,确认并记录稳定分类,不推动当前工作 | +| 已结清步骤收到其他历史结果 | 保留处理历史,不重开步骤或改变已完成业务结果 | +| 来源身份或结构矛盾 | QUARANTINED;未终结实例转NEEDS_REVIEW并关闭新执行门控 | +| SQL/连接或事务异常 | 整笔回滚;仍保留PENDING。扫描器按原记录版本延后30秒,不能覆盖其他节点刚完成的确认 | +| 正常TASK成功 | 验证固定输出契约,保存SUCCEEDED后有限推进 | +| TASK确认取消 | 步骤保留CANCELLED和来源证据,不能改写为FAILED或启动正常后继 | +| TASK失败、未知效果或不合契约的成功结果 | 停止依赖推进,保留原任务结果及稳定核查原因 | + +每个成功提交的消费或实际根推进写系统审计,actor为`system:weave:`。审计使用独立操作身份,元信息保存来源Inbox/检查点ID、节点、实例版本及原因;同一个ROOT检查点可分批产生多次真实推进。每次历史变更取得新的实例revision,避免迟到证据与先前事件的唯一身份冲突。 + +审计失败会回滚消费确认、步骤状态和新任务。对外任务结果订阅继续使用原链路,内部消费不需要锁订阅或外部结果流。 + +## 3. 满额、暂停和截止 + +后继创建使用SAVEPOINT隔离本轮结构变更。多后继中任何一个遇到队列或应用满额,全部本轮后继和对应额度预占回滚;已接受的来源结果及步骤完成仍能与ROOT待续记录一起提交。下一次尝试从当前图重新计算,不能留下部分激活、重复后继或多扣用量。持续满额仅移动检查时间,不逐秒制造相同业务审计。 + +暂停、关闭门控或停用应用/命名空间时,继续收集已经落库的结果,但不激活新步骤;若还有后继,将其保存为待续。重新允许推进后,从持久记录继续。测试通过直接设置控制状态验证该行为;暂停/继续公共接口尚未交付。 + +已过实例总截止时不创建后继,未能全部完成则进入核查。最后的有效任务结果在截止前已由Pulse接受,而消费发生较晚时,可以整理完整结果并完成实例。暂停或重启不延长截止。纯等待实例到期及信号裁决仍需后续等待扫描器。 + +## 4. POLL与失败策略的当前边界 + +已实现当前轮次的结果消费: + +- 查询请求成功但业务PENDING:保存WAITING、清除当前任务并按原事件时间加间隔保存`next_poll_at`,受原查询截止限制;不将PENDING输出交给后继。 +- DONE等已声明成功值:按固定契约完成节点;如有最终信号等待,一并结清。 +- 轮次耗尽、查询截止、未知/非法值或单轮请求失败:进入核查,不新建轮次绕过Pulse重试预算。 +- 业务明确失败:按固定实例失败策略处理;REVIEW进入核查,FAIL_AFTER_SETTLED进入CANCELLING并关闭门控。 + +**尚未实现下一轮查询派发及其到期索引扫描,也未实现FAIL_AFTER_SETTLED的剩余分支取消/收集/失败终态判定。** CANCELLING保留未结清事实,不提前宣称FAILED。信号抢先完成时与当前在途查询的结清竞争,也属于后续必须实现的功能。 + +## 5. 回归与后续工作 + +[存储回归](../internal/storage/mysql/workflow_runtime_integration_test.go)覆盖重复消费者、精确结果、并行汇合、部分后继满额回滚、审计回滚、暂停与停用接续、分批根恢复、来源隔离、旧激活/旧版本、失败/截止和POLL分类;[进程回归](../internal/storage/mysql/workflow_runtime_process_integration_test.go)强杀唯一Orbit进程,确认死亡后提交任务结果,再启动两个新进程接续。 + +进程回归由`make integration`构建二进制后显式执行。缺少一次性数据库或`ORBIT_TEST_SERVER_BINARY`时测试会跳过;跳过不构成进程恢复证据。验证记录见[本阶段报告](reports/phase-4-runtime.md)。 + +下一步仍包括持久化定时/信号等待、轮询后续轮次、完整控制与人工恢复、实例/来源公共API、独立SDK和通用端到端场景。P4整体及P5–P7继续按[完整计划](08-development-plan.md)执行。 diff --git a/docs/README.md b/docs/README.md index 706a0ca..63182a2 100644 --- a/docs/README.md +++ b/docs/README.md @@ -17,6 +17,10 @@ | [06 Weave 流程设计](06-workflow-spec.md) | 已确认的首版功能、研发定义发布、持久化等待、有界查询、暂停恢复与模块协作 | | [07 数据模型](07-data-model.md) | 当前 Go 实体、数据库约束、存储约定与实现边界 | | [08 持续开发计划](08-development-plan.md) | 开发、测试回归、阶段验收与最终交付目标 | +| [09 可靠结果订阅](09-result-subscription-core.md) | 提交水位、冻结结果与可靠交付 | +| [10 Weave决策与交付](10-workflow-engine-core.md) | 有限步骤决策和内部结果Inbox | +| [11 Weave实例受理](11-workflow-admission.md) | 固定版本、初始图和首批任务事务 | +| [12 Weave结果消费与推进](12-workflow-runtime.md) | 后台消费、根接续、事务与进程恢复,及尚未完成的等待/控制边界 | | [接口手册](api/README.md) | 实际接口状态、调用说明与后续接口规划 | | [SDK 与场景](sdk/README.md) | 可运行示例、接入方法及分阶段 SDK 交付 | diff --git a/docs/reports/phase-4-runtime.md b/docs/reports/phase-4-runtime.md new file mode 100644 index 0000000..3f4df21 --- /dev/null +++ b/docs/reports/phase-4-runtime.md @@ -0,0 +1,45 @@ +# P4 结果消费与自动推进回归 + +日期:2026-09-18。范围为内部运行器,不是完整Weave或生产高可用验收。 + +## 已实现 + +- Orbit后台按数据库到期索引消费内部结果和根检查点;同一事务校验固定实例图、接受结果、更新步骤、创建后继或保存待续,并写确认、事件和审计。 +- 顺序、分支、有限并行汇合及32步预算接续;重复消费者不重复创建后继。 +- 满额SAVEPOINT回滚本轮后继,保留已确认结果;暂停/停用收集在途结果并保留检查点;失败或超期关闭新执行门控。 +- 来源/激活/任务版本校验,非法来源隔离,旧激活或旧结果留痕;数据库故障不被误认成业务失败。 +- POLL当前轮次的成功/等待/失败/核查分类,独立保留轮次与尝试预算、绝对截止和下一查询时间。 +- 系统审计保存节点、来源记录与实例版本;真实进程强杀、全停时保存结果和双节点接管。 + +实现说明及明确边界见[运行器文档](../12-workflow-runtime.md)。本轮未增加实体、持久化字段、迁移或外键。 + +## 环境与执行记录 + +本机darwin/arm64,Go 1.27.1,独立Docker MySQL 8.4.10;隔离一次性数据库。未使用业务数据库或远端生产环境。 + +| 检查 | 结果 | +| --- | --- | +| `make fmt test vet build` | 已通过;服务端、独立SDK及两个业务示例module | +| 运行器真实MySQL定向回归 | 14个一级场景,含任务失败/未知/取消/非法输出4种分类及5个POLL分类;最终修订连续3轮通过,20.018秒 | +| 真实Orbit进程故障 | 已通过:确认唯一控制进程SIGKILL退出后保存结果,两个新进程只生成一个后继,最终3个任务、3条消费确认及3条推进审计 | +| 全量`make integration`等价脚本 | 已通过,退出码0;全存储132.893秒、流程进程8.506秒、Worker SDK 91.172秒、SDK进程故障93.679秒;19个SDK示例及2个独立业务示例全部通过 | + +定向命令为`go test -race -count=3 -run '^TestIntegrationWorkflowRuntime$' -v ./internal/storage/mysql`,依赖`ORBIT_TEST_MYSQL_DSN`指向隔离测试库。进程用例额外需要构建后的`ORBIT_TEST_SERVER_BINARY`;全量入口[脚本](../../scripts/test-mysql.sh)负责创建隔离库、构建并执行该用例。 + +文档复核834个本地链接与53个JSON代码块,错误0;差异空白检查通过。 + +数据库复核MySQL 8.4.10、外键0、缺少COMMENT的业务列0。完整迁移升级回归与现有索引/数据保留检查通过。 + +开发中真实回归发现并修复了:未完成步骤NULL输出的SQL读取不兼容、分批复用检查点引发审计操作身份冲突;随后增加迟到结果历史版本回归,确保新确认不与先前实例事件冲突。后继额度中途失败和审计失败分别注入验证了事务回滚。 + +全量回归启动后,代码审阅补充了TASK确认取消保留CANCELLED状态,并新增对应真实Worker回归。最终修订另跑全部运行器场景三轮(20.018秒)、真实进程故障(9.183秒)及基础检查;全量记录与最后定向记录分别保留,不能把较早的全量结果声称为包含后续修改。 + +## 尚未完成的原定范围 + +1. SIGNAL/TIMER等待消费及到期扫描;早到、重复、超时和回执竞态。 +2. POLL下一轮派发、完整定时扫描、信号与在途查询结清;已有next_poll_at不代表下一轮已自动运行。 +3. FAIL_AFTER_SETTLED分支取消和失败终态收敛、暂停/继续/取消公共管理API、带证据人工恢复。 +4. 实例/步骤/来源公共HTTP和独立SDK、全节点可运行接入例子。 +5. 全量监控、负载与故障矩阵、MySQL高可用拓扑和备份恢复、P6–P7交付审计。 + +本地进程回归使用内部Start和Pulse执行存储驱动业务结果;这验证真实控制进程接管,不能替代公共HTTP/SDK端到端验收,也不证明生产数据库RPO/RTO或容量。 diff --git a/internal/storage/mysql/workflow_consume.go b/internal/storage/mysql/workflow_consume.go new file mode 100644 index 0000000..a08b528 --- /dev/null +++ b/internal/storage/mysql/workflow_consume.go @@ -0,0 +1,274 @@ +package mysql + +import ( + "bytes" + "context" + "database/sql" + "encoding/json" + "errors" + "strings" + "time" + + "git.yuntuo.hk/k2safe/orbit/internal/jsonvalue" + core "git.yuntuo.hk/k2safe/orbit/model" + "git.yuntuo.hk/k2safe/orbit/platform/events" + "git.yuntuo.hk/k2safe/orbit/weave/definition" + "git.yuntuo.hk/k2safe/orbit/weave/engine" + weave "git.yuntuo.hk/k2safe/orbit/weave/model" +) + +// workflowSourceError 只用于已核实的来源/契约矛盾;SQL或连接错误不能据此隔离业务结果。 +type workflowSourceError string + +func (e workflowSourceError) Error() string { return string(e) } + +type workflowResult struct { + event, task, activation core.ID + revision uint64 + snapshot events.TaskResult + occurred time.Time + step *workflowStep +} + +func readWorkflowResult(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, now time.Time) (*workflowResult, error) { + var state string + var due time.Time + var raw []byte + source := &workflowResult{} + err := tx.QueryRowContext(ctx, `SELECT event_id,source_task_id,source_activation_id,source_revision,snapshot,state,available_at FROM weave_inbox WHERE namespace_id=? AND id=? AND workflow_id=? FOR UPDATE`, c.ns, c.id, c.workflow).Scan(&source.event, &source.task, &source.activation, &source.revision, &raw, &state, &due) + if errors.Is(err, sql.ErrNoRows) { + return nil, nil + } + if err != nil { + return nil, err + } + if state != "PENDING" || due.After(now) { + return nil, nil + } + if _, err := jsonvalue.Canonical(raw); err != nil { + return nil, workflowSourceError("RESULT_SNAPSHOT_INVALID") + } + decoder := json.NewDecoder(bytes.NewReader(raw)) + decoder.DisallowUnknownFields() + if decoder.Decode(&source.snapshot) != nil || source.snapshot.SchemaVersion != 1 || source.snapshot.TaskID != source.task || source.snapshot.Revision != source.revision || source.revision == 0 || !events.ResultType("task."+strings.ToLower(source.snapshot.State)) { + return nil, workflowSourceError("RESULT_SNAPSHOT_INVALID") + } + var aggregate core.ID + var kind, eventType string + var rev uint64 + err = tx.QueryRowContext(ctx, `SELECT aggregate_kind,aggregate_id,aggregate_revision,type,occurred_at FROM orbit_events WHERE namespace_id=? AND id=?`, c.ns, source.event).Scan(&kind, &aggregate, &rev, &eventType, &source.occurred) + if errors.Is(err, sql.ErrNoRows) { + return nil, workflowSourceError("RESULT_EVENT_MISSING") + } + if err != nil { + return nil, err + } + if kind != "task" || aggregate != source.task || rev != source.revision || eventType != "task."+strings.ToLower(source.snapshot.State) || source.occurred.After(now) { + return nil, workflowSourceError("RESULT_EVENT_MISMATCH") + } + var stepID core.ID + var direct *core.ID + err = tx.QueryRowContext(ctx, `SELECT step_id,task_id FROM weave_activations WHERE namespace_id=? AND id=? AND workflow_id=?`, c.ns, source.activation, c.workflow).Scan(&stepID, &direct) + if errors.Is(err, sql.ErrNoRows) { + return nil, workflowSourceError("RESULT_ACTIVATION_MISSING") + } + if err != nil { + return nil, err + } + for _, step := range g.steps { + if step.view.ID == stepID { + source.step = step + break + } + } + if source.step == nil { + return nil, workflowSourceError("RESULT_STEP_MISSING") + } + node, ok := g.program.Step(source.step.view.Key) + if !ok || node.Task == nil && node.Poll == nil { + return nil, workflowSourceError("RESULT_STEP_KIND") + } + taskDefinition := node.Task + if node.Poll != nil { + taskDefinition = &node.Poll.Task + var round uint32 + err = tx.QueryRowContext(ctx, `SELECT r.round FROM weave_poll_rounds r JOIN weave_poll_states p ON p.namespace_id=r.namespace_id AND p.workflow_id=r.workflow_id AND p.activation_id=r.activation_id WHERE r.namespace_id=? AND r.workflow_id=? AND r.activation_id=? AND r.task_id=? AND r.round BETWEEN 1 AND p.used_rounds`, c.ns, c.workflow, source.activation, source.task).Scan(&round) + if errors.Is(err, sql.ErrNoRows) || direct != nil { + return nil, workflowSourceError("RESULT_POLL_OWNER") + } + if err != nil { + return nil, err + } + } else if direct == nil || *direct != source.task { + return nil, workflowSourceError("RESULT_TASK_OWNER") + } + var queue core.ID + var owner, gate *core.ID + var ownerKind sql.NullString + var currentRevision uint64 + err = tx.QueryRowContext(ctx, `SELECT queue_id,owner_kind,owner_id,gate_id,revision FROM pulse_tasks WHERE namespace_id=? AND id=?`, c.ns, source.task).Scan(&queue, &ownerKind, &owner, &gate, ¤tRevision) + if errors.Is(err, sql.ErrNoRows) { + return nil, workflowSourceError("RESULT_TASK_MISSING") + } + if err != nil { + return nil, err + } + if queue != source.snapshot.QueueID || queue != taskDefinition.QueueID || ownerKind.String != engine.TaskOwnerKind || owner == nil || *owner != source.activation || gate == nil || *gate != g.gate || currentRevision < source.revision { + return nil, workflowSourceError("RESULT_TASK_ASSOCIATION") + } + if source.snapshot.CurrentAttemptID != nil { + var attempt core.ID + err = tx.QueryRowContext(ctx, `SELECT id FROM pulse_attempts WHERE namespace_id=? AND id=? AND task_id=?`, c.ns, source.snapshot.CurrentAttemptID, source.task).Scan(&attempt) + if errors.Is(err, sql.ErrNoRows) { + return nil, workflowSourceError("RESULT_ATTEMPT_ASSOCIATION") + } + if err != nil { + return nil, err + } + } + return source, nil +} + +func (r *WorkflowRuntime) consume(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, allow bool, now time.Time) (bool, error) { + source, err := readWorkflowResult(ctx, tx, g, c, now) + var invalid workflowSourceError + if errors.As(err, &invalid) { + if !g.view.State.Terminal() { + if err := holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, string(invalid), now); err != nil { + return false, err + } + } + return true, r.confirmWorkflowResult(ctx, tx, g, c, "QUARANTINED", string(invalid), now) + } + if err != nil || source == nil { + return false, err + } + row := source.step + if row.view.ActivationID == nil || *row.view.ActivationID != source.activation { + return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_ACTIVATION", now) + } + var newer int + if err := tx.QueryRowContext(ctx, `SELECT COUNT(*) FROM weave_inbox WHERE namespace_id=? AND source_task_id=? AND source_revision>? AND state='PROCESSED'`, c.ns, source.task, source.revision).Scan(&newer); err != nil { + return false, err + } + if newer > 0 { + return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_RESULT", now) + } + if g.view.State.Terminal() || row.view.State != weave.StepActive && row.view.State != weave.StepWaiting { + return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STEP_ALREADY_SETTLED", now) + } + state, output, reason := weave.StepSucceeded, source.snapshot.Result, "TASK_RESULT" + if source.snapshot.EffectUncertain || source.snapshot.State == "NEEDS_REVIEW" { + state, output, reason = weave.StepNeedsReview, nil, "TASK_RESULT_UNCERTAIN" + } else if source.snapshot.State == "CANCELLED" { + state, output, reason = weave.StepCancelled, nil, "TASK_RESULT_CANCELLED" + } else if source.snapshot.State != "SUCCEEDED" { + state, output, reason = weave.StepFailed, nil, "TASK_RESULT_FAILED" + } + node, _ := g.program.Step(row.view.Key) + if node.Poll != nil { + var used, max uint32 + var deadline time.Time + var current *core.ID + var resolved *time.Time + if err := tx.QueryRowContext(ctx, `SELECT used_rounds,max_rounds,deadline_at,current_task_id,resolved_at FROM weave_poll_states WHERE namespace_id=? AND workflow_id=? AND activation_id=? FOR UPDATE`, c.ns, c.workflow, source.activation).Scan(&used, &max, &deadline, ¤t, &resolved); err != nil { + return false, err + } + if resolved != nil || current == nil || *current != source.task { + return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", "STALE_POLL_ROUND", now) + } + // 单轮请求失败不能通过新建业务轮次绕过Pulse预算。 + if state != weave.StepSucceeded { + state, reason = weave.StepNeedsReview, "POLL_REQUEST_REVIEW" + } else if !source.occurred.Before(deadline) { + state, output, reason = weave.StepNeedsReview, nil, "POLL_DEADLINE" + } else { + outcome, classifyErr := g.program.ClassifyPoll(row.view.Key, output) + switch { + case classifyErr != nil: + state, output, reason = weave.StepNeedsReview, nil, "POLL_RESULT_INVALID" + case outcome == engine.PollPending: + state, output, reason = weave.StepWaiting, nil, "POLL_PENDING" + if used >= max { + state, reason = weave.StepNeedsReview, "POLL_ROUNDS_EXHAUSTED" + } + case outcome == engine.PollFailed: + state, output, reason = weave.StepFailed, nil, "POLL_BUSINESS_FAILED" + case outcome == engine.PollNeedsReview: + state, output, reason = weave.StepNeedsReview, nil, "POLL_BUSINESS_REVIEW" + } + } + var next, settled *time.Time + if state == weave.StepWaiting { + v := minimumTime(source.occurred.Add(time.Duration(node.Poll.IntervalMS)*time.Millisecond), deadline) + next = &v + } else { + settled = &now + } + if _, err := tx.ExecContext(ctx, `UPDATE weave_poll_states SET current_task_id=NULL,next_poll_at=?,resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND activation_id=?`, next, settled, now, c.ns, c.workflow, source.activation); err != nil { + return false, err + } + if settled != nil { + waitState := "CANCELLED" + if state == weave.StepSucceeded { + waitState = "SATISFIED" + } + if _, err := tx.ExecContext(ctx, `UPDATE weave_waits SET state=?,resolved_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND workflow_id=? AND activation_id=? AND state='WAITING'`, waitState, now, now, c.ns, c.workflow, source.activation); err != nil { + return false, err + } + } + } else if state == weave.StepSucceeded && (bytes.Equal(bytes.TrimSpace(output), []byte("null")) || definition.ValidateData(node.Task.ResultType, output) != nil) { + state, output, reason = weave.StepNeedsReview, nil, "TASK_RESULT_INVALID" + } + if err := saveWorkflowResultStep(ctx, tx, g, row, state, output, now); err != nil { + return false, err + } + if state == weave.StepNeedsReview || state == weave.StepFailed || state == weave.StepCancelled { + next := weave.InstanceNeedsReview + if (state == weave.StepFailed || state == weave.StepCancelled) && (g.version.Specification.FailurePolicy == "FAIL_AFTER_SETTLED" || g.view.CancelRequestedAt != nil) { + next = weave.InstanceCancelling + } + if err := holdWorkflow(ctx, tx, g, next, reason, now); err != nil { + return false, err + } + } else if err := r.progress(ctx, tx, g, allow, now); err != nil { + return false, err + } + return true, r.confirmWorkflowResult(ctx, tx, g, c, "PROCESSED", reason, now) +} + +func saveWorkflowResultStep(ctx context.Context, tx *sql.Tx, g *workflowGraph, row *workflowStep, state weave.StepState, output json.RawMessage, now time.Time) error { + var raw any + if output != nil { + raw = string(output) + } + var completed *time.Time + if state == weave.StepSucceeded || state == weave.StepFailed || state == weave.StepCancelled { + completed = &now + } + r, err := tx.ExecContext(ctx, `UPDATE weave_activations SET state=?,output=?,completed_at=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND workflow_id=? AND step_id=? AND state=?`, state, raw, completed, now, g.view.NamespaceID, row.view.ActivationID, g.view.ID, row.view.ID, row.view.State) + if err := exactlyOne(r, err); err != nil { + return err + } + r, err = tx.ExecContext(ctx, `UPDATE weave_steps SET state=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=? AND current_activation_id=?`, state, now, g.view.NamespaceID, row.view.ID, row.view.Revision, row.view.ActivationID) + if err := exactlyOne(r, err); err != nil { + return err + } + row.view.State = state + row.view.Revision++ + row.output = output + r, err = tx.ExecContext(ctx, `UPDATE weave_instances SET revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, now, g.view.NamespaceID, g.view.ID, g.view.Revision) + if err := exactlyOne(r, err); err != nil { + return err + } + g.view.Revision++ + return emitEvent(ctx, tx, g.view.NamespaceID, row.view.ID, row.view.Revision, "workflow_step", "workflow.step.changed", map[string]any{"workflow_id": g.view.ID, "key": row.view.Key, "state": state, "activation_id": row.view.ActivationID}) +} + +func (r *WorkflowRuntime) confirmWorkflowResult(ctx context.Context, tx *sql.Tx, g *workflowGraph, c workflowCandidate, state, reason string, now time.Time) error { + res, err := tx.ExecContext(ctx, `UPDATE weave_inbox SET state=?,processed_at=?,error_code=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND workflow_id=? AND state='PENDING'`, state, now, reason, now, c.ns, c.id, c.workflow) + if err := exactlyOne(res, err); err != nil { + return err + } + return r.audit(ctx, tx, g, c.id, reason, now) +} diff --git a/internal/storage/mysql/workflow_runtime.go b/internal/storage/mysql/workflow_runtime.go new file mode 100644 index 0000000..f333b48 --- /dev/null +++ b/internal/storage/mysql/workflow_runtime.go @@ -0,0 +1,259 @@ +package mysql + +import ( + "context" + "database/sql" + "encoding/json" + "errors" + "time" + + "git.yuntuo.hk/k2safe/orbit/internal/fault" + core "git.yuntuo.hk/k2safe/orbit/model" + "git.yuntuo.hk/k2safe/orbit/weave/engine" + weave "git.yuntuo.hk/k2safe/orbit/weave/model" +) + +// WorkflowRuntime 消费持久化结果与根推进检查点;不持有业务执行租约或调用业务系统。 +// 每个候选独立短事务,数据库行锁决定唯一推进权,节点内存不保存恢复游标。 +type WorkflowRuntime struct { + db *sql.DB + node core.ID +} + +func NewWorkflowRuntime(db *sql.DB, node core.ID) (*WorkflowRuntime, error) { + if db == nil || node.Validate() != nil { + return nil, errors.New("invalid workflow runtime configuration") + } + return &WorkflowRuntime{db: db, node: node}, nil +} + +type WorkflowRuntimeStats struct { + Examined int // Examined 本批取出的持久化候选数,包含被其他节点先处理或跳过的记录。 + Consumed int // Consumed 已提交确认或隔离的内部结果数,事务回滚不计入。 + Advanced int // Advanced 已提交推进的根检查点数,不等于业务任务执行次数。 +} + +type workflowCandidate struct { + kind string + ns, id, workflow core.ID + revision uint64 +} + +// Tick 同时读取两种到期索引,各取最多limit项再全局选前limit项;不先锁Inbox再反锁父资源。 +func (r *WorkflowRuntime) Tick(ctx context.Context, limit int) (WorkflowRuntimeStats, error) { + var stats WorkflowRuntimeStats + if limit < 1 || limit > 100 { + return stats, fault.Invalid("workflow batch limit must be 1 to 100") + } + rows, err := r.db.QueryContext(ctx, `(SELECT 'INBOX' kind,namespace_id,id,workflow_id,revision,available_at FROM weave_inbox WHERE state='PENDING' AND available_at<=UTC_TIMESTAMP(6) ORDER BY available_at,namespace_id,id LIMIT ?) + UNION ALL (SELECT 'ROOT' kind,namespace_id,id,workflow_id,revision,available_at FROM weave_pending_advances WHERE available_at<=UTC_TIMESTAMP(6) ORDER BY available_at,namespace_id,id LIMIT ?) + ORDER BY available_at,namespace_id,id LIMIT ?`, limit, limit, limit) + if err != nil { + return stats, err + } + var candidates []workflowCandidate + for rows.Next() { + var c workflowCandidate + var due time.Time + if err := rows.Scan(&c.kind, &c.ns, &c.id, &c.workflow, &c.revision, &due); err != nil { + rows.Close() + return stats, err + } + candidates = append(candidates, c) + } + err = rows.Err() + rows.Close() + if err != nil { + return stats, err + } + var failures []error + for _, c := range candidates { + stats.Examined++ + done, err := r.process(ctx, c, true) + if err != nil { + // 仅移动同一版本的未消费记录,不能覆盖其他节点刚提交的确认或待续状态。 + // 数据库异常不被归类为业务失败;来源保留供下一批恢复。 + if ctx.Err() == nil { + err = errors.Join(err, r.deferCandidate(ctx, c)) + } + failures = append(failures, err) + if ctx.Err() != nil { + break + } + } else if done { + if c.kind == "INBOX" { + stats.Consumed++ + } else { + stats.Advanced++ + } + } + } + return stats, errors.Join(failures...) +} + +func (r *WorkflowRuntime) deferCandidate(ctx context.Context, c workflowCandidate) error { + q := `UPDATE weave_pending_advances SET available_at=DATE_ADD(UTC_TIMESTAMP(6),INTERVAL 30 SECOND),revision=revision+1,updated_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=? AND revision=?` + if c.kind == "INBOX" { + q = `UPDATE weave_inbox SET available_at=DATE_ADD(UTC_TIMESTAMP(6),INTERVAL 30 SECOND),error_code='RUNTIME_UNAVAILABLE',revision=revision+1,updated_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=? AND revision=? AND state='PENDING'` + } + _, err := r.db.ExecContext(ctx, q, c.ns, c.id, c.revision) + return err +} + +func (r *WorkflowRuntime) process(ctx context.Context, c workflowCandidate, skip bool) (bool, error) { + done := false + err := WithTx(ctx, r.db, func(tx *sql.Tx) error { + g, allow, err := lockWorkflowGraph(ctx, tx, c.ns, c.workflow, skip) + if err != nil || g == nil { + return err + } + now, err := dbNow(ctx, tx) + if err != nil { + return err + } + if c.kind == "INBOX" { + done, err = r.consume(ctx, tx, g, c, allow, now) + return err + } + var step core.ID + var expected uint64 + var raw []byte + var due time.Time + err = tx.QueryRowContext(ctx, `SELECT step_id,expected_step_revision,checkpoint,available_at FROM weave_pending_advances WHERE namespace_id=? AND id=? AND workflow_id=? FOR UPDATE`, c.ns, c.id, c.workflow).Scan(&step, &expected, &raw, &due) + if errors.Is(err, sql.ErrNoRows) { + return nil + } + if err != nil { + return err + } + if due.After(now) { + return nil + } + var checkpoint struct { + SchemaVersion uint32 `json:"schema_version"` + Kind string `json:"kind"` + } + root := g.steps[workflowRootKey] + if json.Unmarshal(raw, &checkpoint) != nil || checkpoint.SchemaVersion != 1 || checkpoint.Kind != "ROOT" || step != root.view.ID || expected > root.view.Revision { + return errors.New("workflow checkpoint invalid") + } + if g.view.State.Terminal() { + _, err = tx.ExecContext(ctx, `DELETE FROM weave_pending_advances WHERE namespace_id=? AND id=?`, c.ns, c.id) + return err + } + if !allow || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling { + // 暂停/停用仍按原截止计时;核查和失败结清由控制运行器接续,不清除其状态。 + if (g.view.State == weave.InstanceRunning || g.view.State == weave.InstanceWaiting) && !now.Before(g.view.DeadlineAt) { + if err := holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, "WORKFLOW_DEADLINE", now); err != nil { + return err + } + if err := r.audit(ctx, tx, g, c.id, "WORKFLOW_DEADLINE", now); err != nil { + return err + } + } + return saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second)) + } + beforeRevision := g.view.Revision + if err := r.progress(ctx, tx, g, allow, now); err != nil { + return err + } + if g.view.Revision == beforeRevision { // 额度不足只延后检查点,不逐秒制造业务审计。 + return nil + } + if err := r.audit(ctx, tx, g, c.id, "ROOT_ADVANCE", now); err != nil { + return err + } + done = true + return nil + }) + return done && err == nil, err +} + +func cloneWorkflowGraph(g *workflowGraph) *workflowGraph { + out := *g + out.steps = make(map[string]*workflowStep, len(g.steps)) + for key, row := range g.steps { + copy := *row + out.steps[key] = © + } + return &out +} + +// progress 在结果已保存后执行有限决策。额度不足只回滚后继决策,不撤销已确认的结果。 +func (r *WorkflowRuntime) progress(ctx context.Context, tx *sql.Tx, g *workflowGraph, allow bool, now time.Time) error { + if g.view.State.Terminal() || g.view.State == weave.InstanceNeedsReview || g.view.State == weave.InstanceCancelling { + return nil + } + allow = allow && now.Before(g.view.DeadlineAt) + d, err := g.program.Plan(workflowSnapshot(g, allow), 32) + if err != nil { + var problem *engine.Problem + if !errors.As(err, &problem) { + return err + } + return holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, problem.Code, now) + } + if !d.Complete && !now.Before(g.view.DeadlineAt) { + return holdWorkflow(ctx, tx, g, weave.InstanceNeedsReview, "WORKFLOW_DEADLINE", now) + } + if _, err := tx.ExecContext(ctx, `SAVEPOINT workflow_successors`); err != nil { + return err + } + before := cloneWorkflowGraph(g) + err = applyWorkflowDecision(ctx, tx, g, d, now) + if errors.Is(err, fault.QueueFull) || errors.Is(err, fault.ApplicationFull) { + if _, rollback := tx.ExecContext(ctx, `ROLLBACK TO SAVEPOINT workflow_successors`); rollback != nil { + return rollback + } + *g = *before + return saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second)) + } + if err != nil { + return err + } + if !allow && !d.Complete { + return saveWorkflowAdvance(ctx, tx, g, now.Add(time.Second)) + } + return nil +} + +func holdWorkflow(ctx context.Context, tx *sql.Tx, g *workflowGraph, state weave.InstanceState, reason string, now time.Time) error { + if g.view.State.Terminal() { + return errors.New("cannot hold completed workflow") + } + if g.view.State == weave.InstanceNeedsReview { + state = weave.InstanceNeedsReview + } + if _, err := tx.ExecContext(ctx, `UPDATE pulse_execution_gates SET paused=TRUE,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=?`, now, g.view.NamespaceID, g.gate); err != nil { + return err + } + r, err := tx.ExecContext(ctx, `UPDATE weave_instances SET state=?,revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, state, now, g.view.NamespaceID, g.view.ID, g.view.Revision) + if err := exactlyOne(r, err); err != nil { + return err + } + g.view.State = state + g.view.Revision++ + return emitEvent(ctx, tx, g.view.NamespaceID, g.view.ID, g.view.Revision, "workflow", "workflow.blocked", map[string]any{"state": state, "reason": reason}) +} + +func (r *WorkflowRuntime) audit(ctx context.Context, tx *sql.Tx, g *workflowGraph, operation core.ID, reason string, now time.Time) error { + id, err := core.NewID() + if err != nil { + return err + } + // 迟到证据也占用新的实例历史版本,避免与同一实例版本先前的推进事件冲突。 + res, err := tx.ExecContext(ctx, `UPDATE weave_instances SET revision=revision+1,updated_at=? WHERE namespace_id=? AND id=? AND revision=?`, now, g.view.NamespaceID, g.view.ID, g.view.Revision) + if err := exactlyOne(res, err); err != nil { + return err + } + g.view.Revision++ + meta := map[string]any{"state": g.view.State, "revision": g.view.Revision, "node_id": r.node, "operation_id": operation, "reason": reason} + raw, err := json.Marshal(meta) + if err != nil { + return err + } + if _, err = tx.ExecContext(ctx, "INSERT INTO orbit_audit_events(namespace_id,id,operation_id,actor_id,action,resource_kind,resource_id,reason,`before`,`after`) VALUES (?,?,?,?,'workflow.advance','workflow',?,?,'null',?)", g.view.NamespaceID, id, id, "system:weave:"+string(r.node), g.view.ID, reason, string(raw)); err != nil { + return err + } + return emitEvent(ctx, tx, g.view.NamespaceID, g.view.ID, g.view.Revision, "workflow", "workflow.advanced", meta) +} diff --git a/internal/storage/mysql/workflow_runtime_graph.go b/internal/storage/mysql/workflow_runtime_graph.go new file mode 100644 index 0000000..69574e8 --- /dev/null +++ b/internal/storage/mysql/workflow_runtime_graph.go @@ -0,0 +1,122 @@ +package mysql + +import ( + "context" + "database/sql" + "errors" + "time" + + core "git.yuntuo.hk/k2safe/orbit/model" + "git.yuntuo.hk/k2safe/orbit/weave/engine" + weave "git.yuntuo.hk/k2safe/orbit/weave/model" +) + +// lockWorkflowGraph 只由内部运行器调用。身份→应用额度→排序队列→实例→门控, +// 不用启动凭据继续授权;停用身份仍收集在途结果,但禁止创建新工作。 +func lockWorkflowGraph(ctx context.Context, tx *sql.Tx, ns, id core.ID, skip bool) (*workflowGraph, bool, error) { + var enabled bool + if err := tx.QueryRowContext(ctx, `SELECT n.enabled AND a.enabled FROM orbit_namespaces n JOIN orbit_applications a ON a.id=n.application_id WHERE n.id=? FOR SHARE`, ns).Scan(&enabled); err != nil { + return nil, false, err + } + if ok, err := lockApplicationResources(ctx, tx, ns, skip); err != nil || !ok { + return nil, false, err + } + v, gate, err := readWorkflowInstance(ctx, tx, ns, id, false) + if err != nil { + return nil, false, err + } + version, err := instanceVersion(ctx, tx, v) + if err != nil { + return nil, false, err + } + for _, queue := range version.Analysis.Queues { + if _, _, err := lockQueueAfterApplication(ctx, tx, ns, queue); err != nil { + return nil, false, err + } + } + var locked core.ID + if err := tx.QueryRowContext(ctx, `SELECT id FROM weave_instances WHERE namespace_id=? AND id=? FOR UPDATE`, ns, id).Scan(&locked); err != nil { + return nil, false, err + } + v, currentGate, err := readWorkflowInstance(ctx, tx, ns, id, false) + if err != nil { + return nil, false, err + } + if currentGate != gate || v.DefinitionVersionID != version.ID { + return nil, false, errors.New("workflow identity changed") + } + var gatePaused bool + if err := tx.QueryRowContext(ctx, `SELECT paused FROM pulse_execution_gates WHERE namespace_id=? AND id=? FOR UPDATE`, ns, gate).Scan(&gatePaused); err != nil { + return nil, false, err + } + program, err := engine.Compile(version.Specification) + if err != nil { + return nil, false, err + } + g := &workflowGraph{view: v, gate: gate, version: version, program: program, steps: map[string]*workflowStep{}} + rows, err := tx.QueryContext(ctx, "SELECT s.id,s.`key`,s.kind,s.state,s.revision,s.current_activation_id,a.task_id,a.id,a.state,a.input,a.output,t.id,t.queue_id FROM weave_steps s LEFT JOIN weave_activations a ON a.namespace_id=s.namespace_id AND a.id=s.current_activation_id AND a.workflow_id=s.workflow_id AND a.step_id=s.id LEFT JOIN pulse_tasks t ON t.namespace_id=s.namespace_id AND t.id=a.task_id AND t.owner_kind='workflow_activation' AND t.owner_id=a.id AND t.gate_id=? WHERE s.namespace_id=? AND s.workflow_id=? ORDER BY s.`key` LIMIT 130", gate, ns, id) + if err != nil { + return nil, false, err + } + defer rows.Close() + for rows.Next() { + row := &workflowStep{} + var activation, task, queue *core.ID + var state sql.NullString + var input, output []byte + if err := rows.Scan(&row.view.ID, &row.view.Key, &row.view.Kind, &row.view.State, &row.view.Revision, &row.view.ActivationID, &row.view.TaskID, &activation, &state, &input, &output, &task, &queue); err != nil { + return nil, false, err + } + row.input, row.output = input, output + key := row.view.Key + node, ok := program.Step(key) + if key == workflowRootKey { + if row.view.Kind != weave.NodeJoin { + return nil, false, errors.New("workflow root kind invalid") + } + } else if !ok || node.Kind != string(row.view.Kind) { + return nil, false, errors.New("workflow step definition mismatch") + } + inactive := row.view.State == weave.StepBlocked || row.view.State == weave.StepSkipped + if inactive && row.view.ActivationID != nil || !inactive && (row.view.ActivationID == nil || activation == nil || *activation != *row.view.ActivationID || state.String != string(row.view.State)) { + return nil, false, errors.New("workflow current activation mismatch") + } + if !inactive && (row.view.Kind == weave.NodeTask) != (row.view.TaskID != nil) { + return nil, false, errors.New("workflow task ownership mismatch") + } + if row.view.TaskID != nil && (task == nil || queue == nil || *task != *row.view.TaskID || node.Task == nil || *queue != node.Task.QueueID) { + return nil, false, errors.New("workflow task association invalid") + } + g.steps[key] = row + } + if err := rows.Err(); err != nil { + return nil, false, err + } + root := g.steps[workflowRootKey] + if len(g.steps) != int(version.Analysis.Steps)+1 || root == nil || root.view.ActivationID == nil { + return nil, false, errors.New("workflow step set incomplete") + } + allow := enabled && !gatePaused && !v.Paused && v.CancelRequestedAt == nil && (v.State == weave.InstanceRunning || v.State == weave.InstanceWaiting) + return g, allow, nil +} + +func workflowSnapshot(g *workflowGraph, allow bool) engine.Snapshot { + s := engine.Snapshot{WorkflowID: g.view.ID, BusinessKey: g.view.BusinessKey, Input: g.view.Input, Steps: map[string]engine.State{}, AllowNew: allow} + for key, row := range g.steps { + if key != workflowRootKey { + s.Steps[key] = engine.State{Status: row.view.State, Input: row.input, Output: row.output} + } + } + return s +} + +// saveWorkflowAdvance 保留根步骤的最新版本;重新调度不重置业务截止或任务预算。 +func saveWorkflowAdvance(ctx context.Context, tx *sql.Tx, g *workflowGraph, due time.Time) error { + id, err := core.NewID() + if err != nil { + return err + } + root := g.steps[workflowRootKey] + _, err = tx.ExecContext(ctx, `INSERT INTO weave_pending_advances(namespace_id,id,workflow_id,step_id,expected_step_revision,checkpoint,available_at) VALUES (?,?,?,?,?,'{"schema_version":1,"kind":"ROOT"}',?) ON DUPLICATE KEY UPDATE expected_step_revision=VALUES(expected_step_revision),checkpoint=VALUES(checkpoint),available_at=VALUES(available_at),revision=revision+1,updated_at=UTC_TIMESTAMP(6)`, g.view.NamespaceID, id, g.view.ID, root.view.ID, root.view.Revision, due) + return err +} diff --git a/internal/storage/mysql/workflow_runtime_integration_test.go b/internal/storage/mysql/workflow_runtime_integration_test.go new file mode 100644 index 0000000..8b8fb9e --- /dev/null +++ b/internal/storage/mysql/workflow_runtime_integration_test.go @@ -0,0 +1,489 @@ +package mysql + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "os" + "strings" + "sync" + "testing" + "time" + + core "git.yuntuo.hk/k2safe/orbit/model" + "git.yuntuo.hk/k2safe/orbit/pulse/worker" + "git.yuntuo.hk/k2safe/orbit/weave/definition" + "git.yuntuo.hk/k2safe/orbit/weave/engine" + "git.yuntuo.hk/k2safe/orbit/weave/instance" +) + +func runtimeSpec(queue core.ID, count int) definition.Specification { + spec := workflowTestSpec(queue) + spec.TimeoutMS = 120000 + base := *spec.Body.Steps[0].Task + base.TimeoutMS = 30000 + base.Retry.AttemptTimeoutMS = 20000 + spec.Body.Steps = nil + for i := 0; i < count; i++ { + task := base + if i > 0 { + task.Input = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: fmt.Sprintf("task%d", i-1)}} + } + spec.Body.Steps = append(spec.Body.Steps, definition.Step{Key: fmt.Sprintf("task%d", i), Kind: "TASK", Task: &task}) + } + spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: fmt.Sprintf("task%d", count-1)}} + return spec +} +func runtimeParallel(spec definition.Specification) definition.Step { + parallel := definition.Parallel{} + for _, key := range []string{"left", "right"} { + task := *spec.Body.Steps[0].Task + parallel.Branches = append(parallel.Branches, definition.ParallelBranch{Name: key, Body: definition.Block{Steps: []definition.Step{{Key: key, Kind: "TASK", Task: &task}}, OutputType: spec.InputType, Output: definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: key}}}}) + } + return definition.Step{Key: "join", Kind: "PARALLEL", Parallel: ¶llel} +} +func newRuntime(t *testing.T, db *sql.DB) *WorkflowRuntime { + t.Helper() + r, err := NewWorkflowRuntime(db, executionID(t)) + if err != nil { + t.Fatal(err) + } + return r +} +func runtimeStart(t *testing.T, f *executionFixture, spec definition.Specification) instance.View { + t.Helper() + grantWorkflowInstance(t, f) + in := publishInstanceSpec(t, f, spec) + out, err := instance.New(NewWorkflowInstanceStore(f.db)).Start(f.ctx, f.token, string(executionID(t)), in) + if err != nil { + t.Fatal(err) + } + return out.Instance +} +func runtimeInbox(t *testing.T, f *executionFixture, task core.ID) workflowCandidate { + t.Helper() + c := workflowCandidate{kind: "INBOX", ns: f.resources.NamespaceID} + if err := f.db.QueryRowContext(f.ctx, `SELECT id,workflow_id,revision FROM weave_inbox WHERE namespace_id=? AND source_task_id=? ORDER BY source_revision DESC LIMIT 1`, c.ns, task).Scan(&c.id, &c.workflow, &c.revision); err != nil { + t.Fatal(err) + } + return c +} +func runtimeConsume(t *testing.T, f *executionFixture, r *WorkflowRuntime, task core.ID) { + t.Helper() + if done, err := r.process(f.ctx, runtimeInbox(t, f, task), false); err != nil || !done { + t.Fatalf("consume done=%v err=%v", done, err) + } +} +func runtimeView(t *testing.T, f *executionFixture, id core.ID, state string) instance.View { + t.Helper() + v, err := instance.New(NewWorkflowInstanceStore(f.db)).Get(f.ctx, f.token, id) + if err != nil || string(v.State) != state { + t.Fatalf("state=%s want=%s err=%v", v.State, state, err) + } + return v +} +func runtimeAdvance(t *testing.T, f *executionFixture, r *WorkflowRuntime, id core.ID) { + t.Helper() + f.exec(`UPDATE weave_pending_advances SET available_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND workflow_id=?`, f.resources.NamespaceID, id) + c := workflowCandidate{kind: "ROOT", ns: f.resources.NamespaceID, workflow: id} + if err := f.db.QueryRowContext(f.ctx, `SELECT id,revision FROM weave_pending_advances WHERE namespace_id=? AND workflow_id=?`, c.ns, id).Scan(&c.id, &c.revision); err != nil { + t.Fatal(err) + } + if done, err := r.process(f.ctx, c, false); err != nil || !done { + t.Fatalf("advance done=%v err=%v", done, err) + } +} + +func TestIntegrationWorkflowRuntime(t *testing.T) { + dsn := os.Getenv("ORBIT_TEST_MYSQL_DSN") + if dsn == "" { + t.Skip("requires disposable MySQL") + } + ctx, cancel := context.WithTimeout(context.Background(), 120*time.Second) + defer cancel() + db, err := Open(ctx, dsn, 32, 8) + if err != nil { + t.Fatal(err) + } + defer db.Close() + if err := Migrate(ctx, db); err != nil { + t.Fatal(err) + } + t.Run("sequence_duplicate_consumers_and_frozen_result", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + // 当前任务不是事件快照;后续读取不得把改过的当前值误用为先前结果。 + f.exec(`UPDATE pulse_tasks SET result='{"n":1}' WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID) + c := runtimeInbox(t, f, ref.TaskID) + var wg sync.WaitGroup + failures := make(chan error, 12) + committed := make(chan bool, 12) + for i := 0; i < 12; i++ { + r := newRuntime(t, db) + wg.Add(1) + go func() { defer wg.Done(); done, e := r.process(ctx, c, false); failures <- e; committed <- done }() + } + wg.Wait() + close(failures) + close(committed) + for err := range failures { + if err != nil { + t.Fatal(err) + } + } + n := 0 + for done := range committed { + if done { + n++ + } + } + if n != 1 || f.count("pulse_tasks", "1=1") != 2 || f.count("orbit_audit_events", "action='workflow.advance'") != 1 { + t.Fatal("duplicate successor/audit", n) + } + second := retryStart(t, f) + if !strings.Contains(string(f.get(second.TaskID).Payload), "9007199254740993") { + t.Fatal("mutable task value replaced frozen result") + } + retryFinish(t, f, second, "SUCCEEDED", false) + runtimeConsume(t, f, newRuntime(t, db), second.TaskID) + after := runtimeView(t, f, v.ID, "SUCCEEDED") + if !strings.Contains(string(after.Output), "9007199254740993") || f.count("pulse_execution_gates", "paused=TRUE") != 1 || f.count("weave_pending_advances", "1=1") != 0 { + t.Fatal("final output/gate/checkpoint") + } + }) + t.Run("parallel_results_join_once", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + spec := runtimeSpec(f.resources.QueueID, 1) + tail := spec.Body.Steps[0] + tail.Key = "tail" + spec.Body.Steps = []definition.Step{runtimeParallel(spec), tail} + spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "tail"}} + v := runtimeStart(t, f, spec) + refs := []worker.AttemptRef{retryStart(t, f), retryStart(t, f)} + for _, ref := range refs { + retryFinish(t, f, ref, "SUCCEEDED", false) + } + cs := []workflowCandidate{runtimeInbox(t, f, refs[0].TaskID), runtimeInbox(t, f, refs[1].TaskID)} + var wg sync.WaitGroup + errs := make(chan error, 2) + for _, c := range cs { + r := newRuntime(t, db) + wg.Add(1) + go func() { defer wg.Done(); _, err := r.process(ctx, c, false); errs <- err }() + } + wg.Wait() + close(errs) + for err := range errs { + if err != nil { + t.Fatal(err) + } + } + if f.count("pulse_tasks", "1=1") != 3 || f.count("weave_steps", "`key`='join' AND state='SUCCEEDED'") != 1 { + t.Fatal("parallel duplicate or incomplete join") + } + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + runtimeView(t, f, v.ID, "SUCCEEDED") + }) + t.Run("backpressure_preserves_result_and_rolls_back_partial_successors", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + spec := runtimeSpec(f.resources.QueueID, 1) + spec.Body.Steps = append(spec.Body.Steps, runtimeParallel(spec)) + v := runtimeStart(t, f, spec) + f.exec(`UPDATE pulse_queues SET max_unfinished_tasks=1 WHERE namespace_id=?`, f.resources.NamespaceID) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_steps", "`key`='task0' AND state='SUCCEEDED'") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.count("weave_pending_advances", "1=1") != 1 { + t.Fatal("backpressure lost result or partial tasks leaked") + } + if instanceCount(f, `SELECT unfinished_tasks FROM pulse_queues WHERE namespace_id=?`, f.resources.NamespaceID) != 0 { + t.Fatal("savepoint leaked queue quota") + } + f.exec(`UPDATE pulse_queues SET max_unfinished_tasks=2 WHERE namespace_id=?`, f.resources.NamespaceID) + runtimeAdvance(t, f, newRuntime(t, db), v.ID) + if f.count("pulse_tasks", "1=1") != 3 || f.count("weave_pending_advances", "1=1") != 0 { + t.Fatal("durable continuation did not resume") + } + }) + t.Run("audit_failure_rolls_back_consume_and_successor", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + f.exec(`CREATE TRIGGER test_reject_workflow_advance BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='injected workflow advance failure'`) + _, err := newRuntime(t, db).process(ctx, runtimeInbox(t, f, ref.TaskID), false) + f.exec(`DROP TRIGGER test_reject_workflow_advance`) + if err == nil || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PENDING'") != 1 || f.count("weave_steps", "`key`='task0' AND state='ACTIVE'") != 1 { + t.Fatal("atomic result/advance rollback failed", err) + } + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + }) + t.Run("paused_collects_then_resumes_from_checkpoint", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + // 控制API尚未实现;此处直接建立其需要维护的持久状态,验证运行器不会越过门控。 + f.exec(`UPDATE weave_instances SET paused=TRUE WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID) + f.exec(`UPDATE pulse_execution_gates SET paused=TRUE WHERE namespace_id=?`, f.resources.NamespaceID) + retryFinish(t, f, ref, "SUCCEEDED", false) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.count("weave_pending_advances", "1=1") != 1 { + t.Fatal("paused continuation lost") + } + f.exec(`UPDATE weave_instances SET paused=FALSE WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID) + f.exec(`UPDATE pulse_execution_gates SET paused=FALSE WHERE namespace_id=?`, f.resources.NamespaceID) + runtimeAdvance(t, f, newRuntime(t, db), v.ID) + if f.count("pulse_tasks", "1=1") != 2 { + t.Fatal("resume failed") + } + }) + t.Run("root_budget_restarts_without_memory", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + spec := runtimeSpec(f.resources.QueueID, 1) + body := spec.Body + unselected := body + unselected.Steps = nil + for i := 0; i < 75; i++ { + task := *body.Steps[0].Task + unselected.Steps = append(unselected.Steps, definition.Step{Key: fmt.Sprintf("skip%d", i), Kind: "TASK", Task: &task}) + } + unselected.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "skip74"}} + zero, one := definition.Value{Kind: "LITERAL", Literal: json.RawMessage(`0`)}, definition.Value{Kind: "LITERAL", Literal: json.RawMessage(`1`)} + spec.Body.Steps = []definition.Step{{Key: "choice", Kind: "BRANCH", Branch: &definition.Branch{Cases: []definition.BranchCase{{When: definition.Predicate{Op: "EQ", Left: &zero, Right: &one}, Body: unselected}}, Default: &body}}} + spec.Body.Output = definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "choice"}} + v := runtimeStart(t, f, spec) + if f.count("weave_steps", "state='SKIPPED'") != 31 || f.count("pulse_tasks", "1=1") != 0 { + t.Fatal("initial budget") + } + runtimeAdvance(t, f, newRuntime(t, db), v.ID) + runtimeAdvance(t, f, newRuntime(t, db), v.ID) + if f.count("weave_steps", "state='SKIPPED'") != 75 || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_pending_advances", "1=1") != 0 { + t.Fatal("bounded restart failed") + } + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + runtimeView(t, f, v.ID, "SUCCEEDED") + }) + t.Run("invalid_source_quarantined_without_advancing", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + f.exec(`UPDATE weave_inbox SET snapshot=JSON_SET(snapshot,'$.revision','9999') WHERE namespace_id=?`, f.resources.NamespaceID) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + runtimeView(t, f, v.ID, "NEEDS_REVIEW") + if f.count("weave_inbox", "state='QUARANTINED' AND error_code='RESULT_SNAPSHOT_INVALID'") != 1 || f.count("pulse_tasks", "1=1") != 1 || f.count("pulse_execution_gates", "paused=TRUE") != 1 { + t.Fatal("corrupt source escaped quarantine") + } + }) + t.Run("failed_and_invalid_results_close_gate", func(t *testing.T) { + for _, outcome := range []string{"FAILED", "UNKNOWN", "CANCELLED", "INVALID"} { + t.Run(outcome, func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + if outcome == "CANCELLED" { + // 流程控制API尚未实现;建立其应保存的取消意图,再由真实Worker确认。 + f.exec(`UPDATE weave_instances SET state='CANCELLING',cancel_requested_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID) + f.exec(`UPDATE pulse_execution_gates SET paused=TRUE WHERE namespace_id=?`, f.resources.NamespaceID) + f.exec(`UPDATE pulse_tasks SET cancel_requested_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID) + } + if outcome == "INVALID" { + if _, err := f.a.Finish(ctx, f.token, f.binding, executionID(t), worker.FinishInput{Ref: ref, Outcome: "SUCCEEDED", Result: json.RawMessage(`{"wrong":true}`)}); err != nil { + t.Fatal(err) + } + } else { + retryFinish(t, f, ref, outcome, false) + } + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + expected := "NEEDS_REVIEW" + if outcome == "CANCELLED" { + expected = "CANCELLING" + } + runtimeView(t, f, v.ID, expected) + if f.count("pulse_tasks", "1=1") != 1 || f.count("pulse_execution_gates", "paused=TRUE") != 1 { + t.Fatal("failure did not gate successors") + } + if outcome == "CANCELLED" && f.count("weave_steps", "`key`='task0' AND state='CANCELLED'") != 1 { + t.Fatal("confirmed cancellation was rewritten as failure") + } + }) + } + }) + t.Run("deadline_does_not_create_successor", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + f.exec(`UPDATE weave_instances SET deadline_at=UTC_TIMESTAMP(6) WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, v.ID) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + runtimeView(t, f, v.ID, "NEEDS_REVIEW") + if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 { + t.Fatal("deadline lost accepted result or created new work") + } + }) + t.Run("old_activation_and_cross_namespace_cannot_advance", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 1)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + c := runtimeInbox(t, f, ref.TaskID) + other := newExecutionFixture(t, ctx, db) + foreign := c + foreign.ns = other.resources.NamespaceID + if done, err := newRuntime(t, db).process(ctx, foreign, false); err == nil || done { + t.Fatal("cross namespace consumer accepted") + } + // 用独立受理任务构造下一激活;旧激活和旧结果完整保留。 + next := f.submit(1) + activation := executionID(t) + var step, gate core.ID + if err := db.QueryRowContext(ctx, "SELECT s.id,i.gate_id FROM weave_steps s JOIN weave_instances i ON i.namespace_id=s.namespace_id AND i.id=s.workflow_id WHERE s.namespace_id=? AND s.workflow_id=? AND s.`key`='task0'", c.ns, v.ID).Scan(&step, &gate); err != nil { + t.Fatal(err) + } + f.exec(`INSERT INTO weave_activations(namespace_id,id,workflow_id,step_id,sequence,state,task_id,input) VALUES (?,?,?,?,2,'ACTIVE',?,'{"n":9007199254740993}')`, c.ns, activation, v.ID, step, next.ID) + f.exec(`UPDATE pulse_tasks SET owner_kind=?,owner_id=?,gate_id=? WHERE namespace_id=? AND id=?`, engine.TaskOwnerKind, activation, gate, c.ns, next.ID) + f.exec(`UPDATE weave_steps SET current_activation_id=? WHERE namespace_id=? AND id=?`, activation, c.ns, step) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + if f.count("weave_inbox", "state='PROCESSED' AND error_code='STALE_ACTIVATION'") != 1 || f.count("weave_steps", "`key`='task0' AND state='ACTIVE'") != 1 { + t.Fatal("old activation advanced current work") + } + }) + t.Run("newer_review_result_then_older_success_does_not_clear_review", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + older := runtimeInbox(t, f, ref.TaskID) + f.exec(`UPDATE pulse_tasks SET state='NEEDS_REVIEW',effect_uncertain=TRUE,completed_at=NULL,revision=revision+1 WHERE namespace_id=? AND id=?`, f.resources.NamespaceID, ref.TaskID) + task := f.get(ref.TaskID) + if err := WithTx(ctx, db, func(tx *sql.Tx) error { + return emitEvent(ctx, tx, f.resources.NamespaceID, ref.TaskID, task.Revision, "task", "task.needs_review", map[string]any{"reason": "test later uncertainty"}) + }); err != nil { + t.Fatal(err) + } + r := newRuntime(t, db) + runtimeConsume(t, f, r, ref.TaskID) + if done, err := r.process(ctx, older, false); err != nil || !done { + t.Fatal(done, err) + } + runtimeView(t, f, v.ID, "NEEDS_REVIEW") + if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "error_code='STALE_RESULT'") != 1 || f.count("orbit_audit_events", "action='workflow.advance'") != 2 { + t.Fatal("late result regressed workflow or lost audit") + } + }) + t.Run("disabled_application_collects_without_new_work", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + f.exec(`UPDATE orbit_applications SET enabled=FALSE WHERE id=?`, f.resources.ApplicationID) + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PROCESSED'") != 1 || f.count("weave_pending_advances", "1=1") != 1 { + t.Fatal("disabled app dropped result or admitted successor") + } + f.exec(`UPDATE orbit_applications SET enabled=TRUE WHERE id=?`, f.resources.ApplicationID) + runtimeAdvance(t, f, newRuntime(t, db), v.ID) + if f.count("pulse_tasks", "1=1") != 2 { + t.Fatal("reenabled application did not resume") + } + }) + t.Run("poll_pending_success_exhaustion_and_request_failure", func(t *testing.T) { + for _, mode := range []string{"PENDING", "DONE", "EXHAUSTED", "INVALID", "REQUEST_FAILED"} { + t.Run(mode, func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + spec := runtimeSpec(f.resources.QueueID, 1) + task := *spec.Body.Steps[0].Task + task.TimeoutMS = 10000 + task.Retry.AttemptTimeoutMS = 5000 + task.ResultType = definition.Type{Kind: "STRING", Enum: []string{"PENDING", "DONE"}} + poll := definition.Poll{Task: task, MaxRounds: 3, TimeoutMS: 30000, IntervalMS: 1000, Pending: []string{"PENDING"}, Succeeded: []string{"DONE"}} + if mode == "EXHAUSTED" { + poll.MaxRounds = 1 + } + spec.Body = definition.Block{Steps: []definition.Step{{Key: "query", Kind: "POLL", Poll: &poll}}, OutputType: task.ResultType, Output: definition.Value{Kind: "REFERENCE", Reference: &definition.Reference{Source: "query"}}} + v := runtimeStart(t, f, spec) + ref := retryStart(t, f) + result, outcome, errorCode := mode, "SUCCEEDED", "" + if mode == "EXHAUSTED" { + result = "PENDING" + } + if mode == "REQUEST_FAILED" { + outcome, errorCode = "FAILED", "REMOTE_FAILURE" + } + raw, _ := json.Marshal(result) + if _, err := f.a.Finish(ctx, f.token, f.binding, executionID(t), worker.FinishInput{Ref: ref, Outcome: outcome, Result: raw, ErrorCode: errorCode}); err != nil { + t.Fatal(err) + } + runtimeConsume(t, f, newRuntime(t, db), ref.TaskID) + expected := "NEEDS_REVIEW" + if mode == "DONE" { + expected = "SUCCEEDED" + } + if mode == "PENDING" { + expected = "WAITING" + } + runtimeView(t, f, v.ID, expected) + if f.count("pulse_tasks", "1=1") != 1 || f.count("weave_poll_states", "used_rounds=1 AND current_task_id IS NULL") != 1 { + t.Fatal("poll reset budget or started overlapping work") + } + if mode == "PENDING" { + if f.count("weave_poll_states", "next_poll_at IS NOT NULL AND resolved_at IS NULL") != 1 || f.count("weave_steps", "`key`='query' AND state='WAITING'") != 1 { + t.Fatal("pending poll mistaken for completion") + } + } + }) + } + }) + t.Run("scanner_backoff_and_busy_application", func(t *testing.T) { + f := newExecutionFixture(t, ctx, db) + runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 2)) + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + c := runtimeInbox(t, f, ref.TaskID) + r := newRuntime(t, db) + lock, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted}) + if err != nil { + t.Fatal(err) + } + if _, err := lockApplicationResources(ctx, lock, c.ns, false); err != nil { + lock.Rollback() + t.Fatal(err) + } + call, cancel := context.WithTimeout(ctx, time.Second) + done, err := r.process(call, c, true) + cancel() + lock.Rollback() + if err != nil || done { + t.Fatal("busy application not skipped", done, err) + } + f.exec(`UPDATE weave_inbox SET available_at='2000-01-01' WHERE namespace_id=?`, c.ns) + f.exec(`CREATE TRIGGER test_reject_runtime_tick BEFORE INSERT ON orbit_audit_events FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='injected scanner transaction failure'`) + stats, err := r.Tick(ctx, 1) + f.exec(`DROP TRIGGER test_reject_runtime_tick`) + if err == nil || stats.Consumed != 0 || f.count("pulse_tasks", "1=1") != 1 || f.count("weave_inbox", "state='PENDING' AND available_at>UTC_TIMESTAMP(6) AND error_code='RUNTIME_UNAVAILABLE'") != 1 { + t.Fatal("scanner lost rollback or failure backoff", stats, err) + } + if done, err := r.process(ctx, runtimeInbox(t, f, ref.TaskID), false); err != nil || done { + t.Fatal("consumer ignored durable backoff", done, err) + } + f.exec(`UPDATE weave_inbox SET available_at='2000-01-01' WHERE namespace_id=?`, c.ns) + stats, err = r.Tick(ctx, 1) + if err != nil || stats.Consumed != 1 || f.count("pulse_tasks", "1=1") != 2 { + t.Fatal("scanner did not recover", stats, err) + } + // 旧失败回调不能延后或改写已被新事务确认的记录。 + if err := r.deferCandidate(ctx, c); err != nil { + t.Fatal(err) + } + if f.count("weave_inbox", "state='PROCESSED' AND error_code='TASK_RESULT'") != 1 { + t.Fatal("old deferral overwrote acknowledgement") + } + }) +} diff --git a/internal/storage/mysql/workflow_runtime_process_integration_test.go b/internal/storage/mysql/workflow_runtime_process_integration_test.go new file mode 100644 index 0000000..453d814 --- /dev/null +++ b/internal/storage/mysql/workflow_runtime_process_integration_test.go @@ -0,0 +1,148 @@ +package mysql + +import ( + "context" + "os" + "os/exec" + "path/filepath" + "testing" + "time" + + "git.yuntuo.hk/k2safe/orbit/weave/instance" + mysqldriver "github.com/go-sql-driver/mysql" +) + +// 真实运行Orbit控制进程;强杀唯一控制节点后写入结果,再由两个全新进程竞争接续。 +// 这里只验证内部实例受理+Pulse执行存储+真实控制进程,公共实例HTTP/SDK另行验收。 +func TestIntegrationWorkflowRuntimeProcess(t *testing.T) { + dsn, binary := os.Getenv("ORBIT_TEST_MYSQL_DSN"), os.Getenv("ORBIT_TEST_SERVER_BINARY") + if dsn == "" || binary == "" { + t.Skip("requires disposable MySQL and built Orbit binary") + } + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + admin, err := Open(ctx, dsn, 2, 1) + if err != nil { + t.Fatal(err) + } + defer admin.Close() + name := "orbit_workflow_process_" + string(executionID(t)) + if _, err := admin.ExecContext(ctx, "CREATE DATABASE `"+name+"`"); err != nil { + t.Fatal(err) + } + defer func() { + cleanup, stop := context.WithTimeout(context.Background(), 10*time.Second) + defer stop() + if _, err := admin.ExecContext(cleanup, "DROP DATABASE `"+name+"`"); err != nil { + t.Error(err) + } + }() + cfg, err := mysqldriver.ParseDSN(dsn) + if err != nil { + t.Fatal("invalid disposable DSN") + } + cfg.DBName = name + db, err := Open(ctx, cfg.FormatDSN(), 16, 4) + if err != nil { + t.Fatal(err) + } + defer db.Close() + if err := Migrate(ctx, db); err != nil { + t.Fatal(err) + } + f := newExecutionFixture(t, ctx, db) + v := runtimeStart(t, f, runtimeSpec(f.resources.QueueID, 3)) + dir := t.TempDir() + type controller struct { + cmd *exec.Cmd + done chan error + log *os.File + stopped bool + } + var processes []*controller + stop := func(p *controller) { + t.Helper() + if p.stopped { + return + } + p.stopped = true + if err := p.cmd.Process.Kill(); err != nil { + t.Error(err) + } + select { + case <-p.done: + case <-time.After(10 * time.Second): + t.Error("controller did not exit after SIGKILL") + } + p.log.Close() + } + defer func() { + for _, p := range processes { + stop(p) + } + }() + start := func(label string) *controller { + t.Helper() + log, err := os.Create(filepath.Join(dir, label+".log")) + if err != nil { + t.Fatal(err) + } + cmd := exec.CommandContext(ctx, binary) + cmd.Env = append(os.Environ(), "ORBIT_MYSQL_DSN="+cfg.FormatDSN(), "ORBIT_HTTP_ADDR=127.0.0.1:0") + cmd.Stdout, cmd.Stderr = log, log + if err := cmd.Start(); err != nil { + log.Close() + t.Fatal(err) + } + p := &controller{cmd: cmd, done: make(chan error, 1), log: log} + processes = append(processes, p) + go func() { p.done <- cmd.Wait() }() + return p + } + wait := func(check func() bool, label string) { + t.Helper() + deadline := time.Now().Add(20 * time.Second) + for time.Now().Before(deadline) { + if check() { + return + } + for _, p := range processes { + if p.stopped { + continue + } + select { + case err := <-p.done: + p.stopped = true + t.Fatalf("controller exited unexpectedly (%s): %v", label, err) + default: + } + } + time.Sleep(25 * time.Millisecond) + } + t.Fatalf("timed out waiting for %s", label) + } + first := start("only-controller") + ref := retryStart(t, f) + retryFinish(t, f, ref, "SUCCEEDED", false) + wait(func() bool { return f.count("pulse_tasks", "1=1") == 2 }, "first successor") + second := retryStart(t, f) + stop(first) // 必须等Wait确认死亡,才把下一结果写入;排除活节点代为处理。 + retryFinish(t, f, second, "SUCCEEDED", false) + if f.count("weave_inbox", "source_task_id=? AND state='PENDING'", second.TaskID) != 1 || f.count("pulse_tasks", "1=1") != 2 { + t.Fatal("result was not left durable while every controller was stopped") + } + start("replacement-a") + start("replacement-b") + wait(func() bool { return f.count("pulse_tasks", "1=1") == 3 }, "two-node continuation") + third := retryStart(t, f) + retryFinish(t, f, third, "SUCCEEDED", false) + api := instance.New(NewWorkflowInstanceStore(db)) + wait(func() bool { + current, err := api.Get(ctx, f.token, v.ID) + return err == nil && current.State == "SUCCEEDED" + }, "completed workflow") + if f.count("pulse_tasks", "1=1") != 3 || f.count("weave_inbox", "state='PROCESSED'") != 3 || f.count("orbit_audit_events", "action='workflow.advance'") != 3 || f.count("weave_pending_advances", "1=1") != 0 { + t.Fatal("process restart duplicated work or lost confirmation") + } + t.Log("sole Orbit process SIGKILL confirmed; result committed with all controllers stopped; two new Orbit processes completed exactly three tasks and three result confirmations") +} diff --git a/scripts/test-mysql.sh b/scripts/test-mysql.sh index 926c34e..f7a4f33 100755 --- a/scripts/test-mysql.sh +++ b/scripts/test-mysql.sh @@ -9,8 +9,13 @@ TASK_TEST_DIR=$(mktemp -d "${TMPDIR:-/tmp}/orbit-model-test.XXXXXX") TASK_SERVER_PID= cleanup() { if [ -n "$TASK_SERVER_PID" ]; then kill "$TASK_SERVER_PID" >/dev/null 2>&1 || true; wait "$TASK_SERVER_PID" 2>/dev/null || true; fi - "$TASK_DOCKER" rm -f "$TASK_TEST_NAME" >/dev/null 2>&1 || true - rm -rf "$TASK_TEST_DIR" + "$TASK_DOCKER" stop "$TASK_TEST_NAME" >/dev/null 2>&1 || true + "$TASK_DOCKER" container rm "$TASK_TEST_NAME" >/dev/null 2>&1 || true + # 仅清理由本脚本创建的普通文件;出现意外子目录时保留目录供核查。 + for task_artifact in "$TASK_TEST_DIR"/*; do + if [ -f "$task_artifact" ]; then unlink "$task_artifact"; fi + done + rmdir "$TASK_TEST_DIR" } trap cleanup EXIT HUP INT TERM umask 077 @@ -32,6 +37,7 @@ export ORBIT_TEST_MYSQL_DSN="root:$TASK_DB_PASSWORD@tcp(127.0.0.1:$TASK_DB_PORT) "$TASK_GO" test -race -count=1 -run Integration -v ./internal/storage/mysql "$TASK_GO" build -o "$TASK_TEST_DIR/orbit" ./cmd/orbit +ORBIT_TEST_SERVER_BINARY="$TASK_TEST_DIR/orbit" "$TASK_GO" test -race -count=1 -run '^TestIntegrationWorkflowRuntimeProcess$' -v ./internal/storage/mysql "$TASK_GO" build -o "$TASK_TEST_DIR/orbit-migrate" ./cmd/orbit-migrate "$TASK_GO" build -o "$TASK_TEST_DIR/orbit-admin" ./cmd/orbit-admin ORBIT_MYSQL_DSN="$ORBIT_TEST_MYSQL_DSN" "$TASK_TEST_DIR/orbit-migrate" status