diff --git a/ADVANCED.md b/ADVANCED.md new file mode 100644 index 0000000..83a8186 --- /dev/null +++ b/ADVANCED.md @@ -0,0 +1,147 @@ +# 扩展业务模块 + +本批提供八类扩展,以及分页、审计和显式数据库迁移。模块按需导入;`omni.New` 与业务 Handler 不绑定存储、队列或网络实现。JSON、配置和新示例使用 lowerCamelCase;Go 导出标识符和 `_test.go` 保持语言约定。 + +按本次范围决定,**不集成 Orbit,不引入 Orbit SDK,也不实现工作流编排**。通用消息队列与 Orbit 的持久流程、执行授权、结果订阅和人工恢复不是等价能力。业务项目以后需要 Orbit,可独立依赖它的 SDK;当前没有承诺只换队列适配器就能迁移其全部语义。 + +## 能力与边界 + +| 模块 | 本批实现 | 明确边界 | +| --- | --- | --- | +| `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI | +| `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 | +| `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider | +| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 | +| `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 | +| `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 | +| `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 | +| `storage` | Local、S3 兼容(含 R2 配置)、OSS 官方 SDK 适配、临时下载链接 | 云平台专属功能单独扩展;本批没有云账号实网验收 | +| `database/migrate` | MySQL 显式部署迁移、摘要预览、失败 dirty 阻断 | 不随应用启动自动改表;无自动 down/force;已应用文件摘要需部署方留存校对 | +| `audit` | 可注入 Sink、字段白名单、JSON 行串行输出、错误返回 | 不是防篡改存储;与业务同事务的审计要实现事务型 Sink | +| `pagination` | page/pageSize、最大偏移与页长、统一结果 | offset 分页,不承诺跨页快照,不实现游标分页 | + +## 对象存储 + +业务只持有 `storage.Store`,调用 `Put`、`Get`、`Delete`。构造函数属于启动组合处: + +```go +type FileService struct { Files storage.Store } + +// Local:私有目录,流式临时文件 + 原子替换。 +files, err := local.New("/srv/app/objects", 32<<20) + +// S3:AWS/MinIO 或其他经过契约测试的 S3 兼容服务。 +files, err := s3.Open(s3.Config{ + Endpoint: "s3.us-east-1.amazonaws.com", Region: "us-east-1", + Bucket: "example-bucket", Secure: true, MaxBytes: 32<<20, + Credentials: credentialProvider, Transport: transport, +}) + +// Cloudflare R2:复用 S3 适配器,签名区域为 auto。 +cfg, err := s3.R2Config(accountID, bucket, credentialProvider) +cfg.Transport = transport +files, err := s3.Open(cfg) + +// OSS:单独适配官方 Go SDK v2,保留平台签名与凭据提供器。 +client := ossSdk.NewClient(ossSdk.LoadDefaultConfig(). + WithRegion("cn-hangzhou"). + WithCredentialsProvider(ossCredentials.NewEnvironmentVariableCredentialsProvider())) +files, err := oss.New(client, bucket, 32<<20) +``` + +上方是互斥初始化片段,实际代码需逐次处理错误。import 路径分别为 `go-omni/storage/local`、`go-omni/storage/s3`、`go-omni/storage/oss`(完整 module 前缀 `git.yuntuo.hk/k2safe/`);`ossSdk` 与 `ossCredentials` 来自 `github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss` 及其 `credentials` 子包。S3 凭据来自 `github.com/minio/minio-go/v7/pkg/credentials`。 + +统一约定:key 是对象标识,不是磁盘路径;拒绝绝对路径、`..`、空路径段和反斜杠。Put 必须传已知长度,调用方保证与内容一致。默认单对象 32 MiB,流式读写;OSS 本适配器只做 simple upload,最大配置 5 GiB。Get 返回的 `Body` 由调用方关闭,读取过程也应有 context 截止。Delete 缺失对象成功;Get 缺失对象返回 `storage.ErrNotFound`。底层客户端/传输连接池由应用管理,不会在单次操作后关闭。 + +业务需要临时链接时显式依赖额外接口 `storage.Presigner`。S3/OSS 实现 `PresignGet`;Local 用 `storage.NewSigner(files, downloadBaseURL, secret)` 提供 HTTP 下载 Handler。链接 TTL 为 1 秒到 1 小时,私钥/临时凭据可能让云链接提前失效。签发链接前必须完成业务授权;对象 key 及 bucket 本身不是租户权限。签名链接不要写日志。首版未封装前端直传、分片续传、生命周期、ACL、CDN 自定义域名和版本管理。 + +R2 实现 S3 的子集,配置入口使用标准账户 endpoint;法域专属 endpoint 可显式配置。兼容性依据:[R2 S3 API](https://developers.cloudflare.com/r2/api/s3/api/)、[R2 签名链接](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)。OSS 使用其[官方 Go SDK v2](https://www.alibabacloud.com/help/en/oss/developer-reference/manual-for-go-sdk-v2/),没有假定全部 S3 行为与 OSS 等价。 + +验证区分:Local 契约、OSS SDK HTTP fixture、S3 SDK fixture 和 R2 签名参数属于本地测试;Linux runner 再跑真实 MinIO 上传/读取/删除与签名拒绝。**MinIO 通过不能代替 OSS/R2 实网验证。**迁移存储平台还需要复制已有对象和调整部署权限,不只是切配置。 + +## 可靠队列 + +```go +backend, err := queueRedis.New(redisClient, queueRedis.Config{ + Namespace: "serviceA:jobs:v1", MaxRecords: 1000, + MaxPayloadBytes: 64 << 10, DedupTTL: 24*time.Hour, +}) +consumer, err := queueRedis.NewConsumer(backend, queueRedis.ConsumerConfig{ + Workers: 4, MaxAttempts: 3, + TaskTimeout: 30*time.Second, VisibilityTimeout: time.Minute, +}) + +// ID 与内容在业务端持久化;超时后复用原 ID 和原内容。 +receipt, err := backend.Publish(ctx, queue.Message{ + ID: operationID, Name: "generateReport", Payload: payload, +}) + +// 在服务生命周期中运行;返回错误须让监控/进程管理感知。 +err = consumer.Run(serviceContext, func(ctx context.Context, item queue.Delivery) error { + return generateReportIdempotently(ctx, item.ID, item.Payload) +}) +``` + +业务依赖 `queue.Publisher` / `queue.Consumer`,Redis 类型留在组合处。receipt 表示受理,不代表业务成功。普通 error/panic 按配置重试,`queue.Reject(code)` 保留为失败消息;超过次数同样保留。`DeadLetters(after, limit)` 是有界分页,`DiscardDead` 仅在人工核对后删除匹配的失败消息,并继续保留一段提交去重窗口。错误正文不写入失败记录。需要再次处理时由业务决定新操作身份及去重策略,框架不自动无限重投。 + +一个 namespace 使用一个受管 consumer group;多个 consumer 可竞争消费同一队列。配置一致性检查拒绝不同的容量/去重策略和执行超时/重试策略混用。确认检查当前 pending owner,旧执行不能确认新执行;但不能阻止旧处理器已发出的外部副作用,因此仍必须业务幂等。处理器需响应 context;Go 无法强杀不配合的函数,超过 shutdownTimeout 返回错误时,最多 Workers 个处理器可能尚未退出。 + +容量包含待处理、失败及未过期的已完成身份,满额拒绝新 ID,保留原 ID 重报回执。完成/人工丢弃后的身份按 DedupTTL 过期;未完成/失败消息不自动清除,不用 MAXLEN 静默裁掉未消费消息。Redis 需启用合适的 AOF/复制策略与 noeviction;本批验证的是投递/接管协议,未证明 Redis 主从故障切换后零丢失。异常进程留下的无 pending consumer 元数据可在运维核实后清理,不能用 DELCONSUMER 丢掉尚未接管的 pending。 + +`task.Runner` 仍适用于本进程短任务和定时轮询,重启会丢失;需要持久交付时使用 `queue`。这里没有分布式 Cron、业务工作流、事务 Outbox 或 exactly-once 承诺。 + +## 认证与授权 + +`auth.NewJWT` 要求固定 issuer/audience 和可信 kid→公钥列表;只开放 RS256 / EdDSA,必须有 exp/sub,检查有效期、nbf 与 iat。`ReplaceKeys` 原子替换列表,轮换期可同时保留新旧 kid。JWT 签发、私钥托管、登录口令验证留给身份服务。 + +`auth.NewSessions(cacheStore, "sessions:", time.Hour)` 签发随机 opaque bearer token;存储仅使用 token 的 SHA-256 键,可 `Revoke`。Redis 缓存可共享 Session,内存缓存只在单实例生效且可能因容量淘汰提前登出。不要把 Session token 当普通缓存值写日志。 + +`RequireAuth(authenticator)` 在前,`RequirePermission(policy, action, resource)` 在后。Policy 接收已验证 Principal,业务自行校验租户、角色、资源归属。示例 `examples/platform` 验证了无身份 401、角色不足 403 和授权写入。 + +## 配置与日志热更新 + +`config.LoadFile(path, prefix)` 按默认值→文件→非空环境变量加载并验证。两种文件格式都使用 JSON tag 的 lowerCamelCase,duration 可以写 `"5s"`;文件上限 1 MiB,拒绝未知字段、多文档和非法值。自定义 prefix 要包含期望的尾部下划线,例如 `BILLING_`。空环境变量沿用文件值,不清空已配置字段。 + +```yaml +address: '127.0.0.1:8080' +transport: native +shutdownTimeout: 10s +log: + level: info + format: json +``` + +先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, prefix, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。 + +## 熔断与观测 + +为每个上游创建 `resilience.New(Config{MaxConcurrent: 16, FailureThreshold: 5})`;通用调用用 `Do`,HTTP 用 `resilience.Transport{Base: transport, Breaker: breaker}`。HTTP 槽位覆盖响应体读取,必须关闭 body;网络错误/5xx 计失败,冷却后只放行一个探针。不要对效果未知的写操作自行换身份重试。 + +`telemetry.New` 接受 OTel `sdktrace.SpanExporter` 和 `sdkmetric.Reader`;可注入官方 OTLP exporter、PeriodicReader 或测试用 exporter,基础包不强制选择 collector 协议。`app.Use(runtime.Middleware())` 采集入站,`telemetry.Transport` 采集出站并传播 W3C trace context;Redis 使用 `client.AddHook(telemetry.RedisHook{Runtime: runtime})`,SQL 使用 `telemetry.SQL{DB: db, Runtime: runtime}`。业务操作/事务可用 `runtime.Do(ctx, "fixedOperationName", fn)` 包裹。 + +创建者负责在停止请求/Worker 后调用 `runtime.Shutdown(ctx)` 刷新并关闭 exporter/reader。指标按注册路由模板和固定方法分类;自动采集不记录 URL query、SQL 正文、Redis key、token 或业务载荷。业务自定义 span 名称同样应固定,避免用用户 ID 或对象 key。 + +## 分布式限流与幂等 + +`middleware/redis.NewRateLimiter(client, namespace, cfg)` 跨副本使用 Redis 时间与原子令牌桶;满额/存储失败返回 503,额度不足返回 429 和 Retry-After。`NewIdempotencyStore` 注入 `middleware.IdempotencyConfig.Store`;Scope 必须来自认证后的租户/用户,先认证再幂等。不同请求内容复用同一身份返回 409,已完成请求回放原响应和 requestId。 + +claim 在开始业务前持久保存,完成按 owner 原子检查。进程崩溃留下的 running claim 不自动按租约过期;返回 409 等待核查,`Resolve` 仅由运维核对后显式调用。已保存的完成/失败记录按配置 TTL 清理,TTL 过后不再提供业务去重保证。失败结果、含 Set-Cookie/no-store 的响应不用于成功回放。共享 Redis 需要 noeviction 和独立 namespace,数据丢失时不存在框架层面的重复执行保证。 + +## 迁移、审计与分页 + +迁移命名:`0001_createUsers.up.sql`,可附 `.down.sql`,但第一版 CLI 只执行 up。`go-omni migrate --dir migrations` 默认打印版本/方向/SHA-256 清单,不连接数据库;部署步骤显式传 `--apply` 才读取 `GO_OMNI_DB_DSN` 执行。迁移使用独立连接与数据库锁;MySQL DDL 可能部分提交,失败保留 dirty,禁止自动 force 或回滚。当前基于 golang-migrate,数据库版本表不保存每个历史文件校验和,部署系统应保留并校对预览清单。取消在当前有界 SQL 操作结束后停止,重试需新建 Runner。 + +审计:`audit.New(sink, "approvedField")` 建白名单,再 `Record(ctx, Event{ActorID, Action, Resource, Outcome, Fields})`;补入服务端时间、requestId/traceId,只保留明确允许的属性,Sink 失败原样返回。JSONSink 不保证与业务提交原子一致,需要这种语义时由业务在同一数据库事务内提供 Sink/Outbox。 + +分页:`pagination.Parse(request.URL.Query(), pagination.Config{})` 默认 page=1/pageSize=20,上限 100,最大 offset=1,000,000;拒绝重复参数、溢出和越界。`pagination.NewResult(request, items, total)` 输出 items/page/pageSize/total/totalPages,空列表为 `[]`。查询与租户过滤仍由 Repository 负责。 + +## 验证入口 + +```sh +go test -race ./... +go vet ./... +go build ./... +python3 scripts/verifyModules.py --advanced --goproxy https://goproxy.cn,direct --output performance/advancedLinux.json +``` + +Linux runner 顺序运行编译、Redis、MinIO、MySQL 四个容器,各限 2 CPU、2 GiB、256 pids;数据阶段无外网、无发布端口,不连接现有数据库。原始结果和源码摘要以生成的 JSON/TXT 为准,未提供云凭据的 OSS/R2 不标为实网通过。 diff --git a/MODULES.md b/MODULES.md index f18bed0..86e010e 100644 --- a/MODULES.md +++ b/MODULES.md @@ -1,5 +1,7 @@ # 业务基础模块 +第二批的共享 Redis 中间件、可靠队列、多平台存储及其他模块见 [扩展模块](ADVANCED.md)。下文的单进程约束指本页原有默认实现。 + 这批能力位于 main,尚未包含在 `v0.6.0-preview` 标签中。统一示例见 [examples/modules](examples/modules/README.md)。业务入口继续使用 `omni.New`,中间件依赖 `httpx`,不依赖具体服务端网络实现。各模块按需创建、注入 Service,再注册到应用关闭流程。 diff --git a/README.md b/README.md index 474d6b1..0114409 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,13 @@ go-omni/ ├── cache/ # 有界内存缓存、Redis 适配与合并回源 ├── httpc/ # 出站 HTTP 连接池、超时和重试 ├── task/ # 有界后台任务与间隔调度 +├── queue/ # 持久队列接口与 Redis Streams 适配 +├── storage/ # Local、S3/R2、OSS 与临时下载链接 +├── openapi/ # 路由和 OpenAPI 3.0.3 契约 +├── telemetry/ # 可注入 OpenTelemetry 观测 +├── resilience/ # 熔断与上游并发隔离 +├── audit/ # 显式业务审计与可注入 Sink +├── pagination/ # 有界分页和统一结果 ├── transport/nethttp/ # 标准库网络适配器 ├── transport/native/ # 自研 HTTP/1.1 服务端预览 ├── transport/http2/ # opt-in HTTP/2 实验适配器 @@ -57,6 +64,9 @@ main 分支新增缓存、出站 HTTP、限流/幂等/超时中间件、后台 使用方式、默认容量和进程内能力的适用范围见 [业务基础模块](MODULES.md), 可以直接运行 [组合示例](examples/modules/README.md)。这些新增能力尚未包含在上面的历史版本标签中。 +扩展模块还包括 JWT/Session、Policy、文件配置及安全热更新、Redis 共享中间件、持久队列、多平台对象存储、OpenAPI、OpenTelemetry、熔断、迁移、审计和分页。 +使用与边界见 [扩展模块](ADVANCED.md),可运行 [文件服务组合示例](examples/platform/README.md)。go-omni 不集成 Orbit;需要流程编排的业务项目独立接入。 + ## 快速开始 ```go diff --git a/ROADMAP.md b/ROADMAP.md index 27b2fb8..a14cadd 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -81,4 +81,11 @@ - 最终代码已通过 Linux amd64 全仓库 race/vet/build、独立生成项目 CRUD、真实 Redis 7.4.11 与三种网络层 TLS 组合测试;测试容器清理完成、原有 7 个业务容器仍运行。提交、源码摘要与完整输出见 [验收记录](performance/modulesAcceptance.md)。 - 当前幂等、限流、任务调度和合并回源均为单进程能力;Redis 仅提供缓存存储,不代表已实现分布式去重、任务持久化或跨副本选主。eTax 指定业务模块接入依用户要求暂缓。 +## 7. 扩展业务模块(本批) + +- 新增 OpenAPI、JWT/Session/Policy、OpenTelemetry、文件配置与日志级别热更新、熔断/并发隔离、Redis 共享限流/幂等、Redis Streams 持久队列、Local/S3/R2/OSS 存储;补充迁移、审计和分页。 +- 业务依赖独立接口,云平台 SDK 和 Redis 留在适配器中;按本次范围决定不集成 Orbit,不实现工作流编排。 +- 已完成本地模块测试及 race 检查,隔离 Redis 的共享状态、重复提交、失败消息、超时接管和旧确认拒绝已通过第一轮验证。完整 Linux、真实 MinIO/MySQL 验收记录待本批验证完成后更新。 +- 契约、使用示例与限制见 [扩展模块](ADVANCED.md);验证入口为 `scripts/verifyModules.py --advanced`,继续限制单个测试容器 2 CPU / 2 GiB / 256 pids。 + 以上阶段以验收结果判断完成。当前仍是基础框架试用阶段;MySQL、适配层契约和一次业务模块试用已完成,自研网络层已有可运行预览,稳定性与生产规模验证尚未完成。 diff --git a/audit/audit.go b/audit/audit.go new file mode 100644 index 0000000..fd3479a --- /dev/null +++ b/audit/audit.go @@ -0,0 +1,106 @@ +// Package audit records explicitly selected business events without request bodies. +package audit + +import ( + "context" + "encoding/json" + "errors" + "git.yuntuo.hk/k2safe/go-omni/logx" + "io" + "sync" + "time" +) + +type Event struct { + Time time.Time `json:"time"` + ActorID string `json:"actorId"` + TenantID string `json:"tenantId,omitempty"` + Action string `json:"action"` + Resource string `json:"resource"` + Outcome string `json:"outcome"` + RequestID string `json:"requestId,omitempty"` + TraceID string `json:"traceId,omitempty"` + Fields map[string]string `json:"fields,omitempty"` +} +type Sink interface { + Write(context.Context, Event) error +} +type Recorder struct { + sink Sink + allowed map[string]bool +} + +func New(sink Sink, allowedFields ...string) (*Recorder, error) { + if sink == nil || len(allowedFields) > 32 { + return nil, errors.New("audit: sink and at most 32 fields required") + } + r := &Recorder{sink: sink, allowed: map[string]bool{}} + for _, name := range allowedFields { + if name == "" || len(name) > 64 { + return nil, errors.New("audit: invalid field") + } + r.allowed[name] = true + } + return r, nil +} +func (r *Recorder) Record(ctx context.Context, event Event) error { + if ctx == nil { + return errors.New("audit: nil context") + } + if err := ctx.Err(); err != nil { + return err + } + if event.ActorID == "" || event.Action == "" || event.Resource == "" || (event.Outcome != "success" && event.Outcome != "failure" && event.Outcome != "denied") { + return errors.New("audit: actor/action/resource and valid outcome required") + } + for _, value := range []string{event.ActorID, event.TenantID, event.Action, event.Resource} { + if len(value) > 512 { + return errors.New("audit: event field too large") + } + } + event.Time = time.Now().UTC() + event.RequestID = logx.RequestID(ctx) + event.TraceID = logx.TraceID(ctx) + filtered := map[string]string{} + for name, value := range event.Fields { + if r.allowed[name] { + if len(value) > 1024 { + return errors.New("audit: attribute too large") + } + filtered[name] = value + } + } + event.Fields = filtered + return r.sink.Write(ctx, event) +} + +// JSONSink borrows its writer and serializes complete JSON lines. Write errors +// are returned; callers choose whether a failed audit must reject an operation. +type JSONSink struct { + mu sync.Mutex + writer io.Writer +} + +func NewJSONSink(writer io.Writer) (*JSONSink, error) { + if writer == nil { + return nil, errors.New("audit: writer required") + } + return &JSONSink{writer: writer}, nil +} +func (s *JSONSink) Write(ctx context.Context, event Event) error { + data, err := json.Marshal(event) + if err != nil { + return err + } + data = append(data, '\n') + s.mu.Lock() + defer s.mu.Unlock() + if err := ctx.Err(); err != nil { + return err + } + n, err := s.writer.Write(data) + if err == nil && n != len(data) { + return io.ErrShortWrite + } + return err +} diff --git a/audit/audit_test.go b/audit/audit_test.go new file mode 100644 index 0000000..ba9ab72 --- /dev/null +++ b/audit/audit_test.go @@ -0,0 +1,50 @@ +package audit + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "git.yuntuo.hk/k2safe/go-omni/logx" + "io" + "sync" + "testing" +) + +type failedWriter struct{} + +func (failedWriter) Write([]byte) (int, error) { return 0, io.ErrClosedPipe } +func TestAllowlistCorrelationAndWriteFailure(t *testing.T) { + var buf bytes.Buffer + sink, _ := NewJSONSink(&buf) + recorder, _ := New(sink, "status") + ctx := logx.WithRequestID(context.Background(), "r1") + ctx = logx.WithTraceID(ctx, "t1") + event := Event{ActorID: "u1", Action: "update", Resource: "object:1", Outcome: "success", Fields: map[string]string{"status": "ok", "password": "secret"}} + var wg sync.WaitGroup + for i := 0; i < 20; i++ { + wg.Add(1) + go func() { + defer wg.Done() + if err := recorder.Record(ctx, event); err != nil { + t.Error(err) + } + }() + } + wg.Wait() + decoder := json.NewDecoder(&buf) + for i := 0; i < 20; i++ { + var got Event + if err := decoder.Decode(&got); err != nil { + t.Fatal(err) + } + if got.Fields["password"] != "" || got.RequestID != "r1" || got.TraceID != "t1" || got.Time.IsZero() { + t.Fatal(got) + } + } + bad, _ := NewJSONSink(failedWriter{}) + recorder, _ = New(bad) + if err := recorder.Record(ctx, event); !errors.Is(err, io.ErrClosedPipe) { + t.Fatal(err) + } +} diff --git a/auth/adapters_test.go b/auth/adapters_test.go new file mode 100644 index 0000000..932bd40 --- /dev/null +++ b/auth/adapters_test.go @@ -0,0 +1,105 @@ +package auth + +import ( + "context" + "crypto/ed25519" + "crypto/rand" + "errors" + "git.yuntuo.hk/k2safe/go-omni/cache" + "github.com/golang-jwt/jwt/v5" + "strings" + "testing" + "time" +) + +func TestJWTValidationAndRotation(t *testing.T) { + pub, private, err := ed25519.GenerateKey(rand.Reader) + if err != nil { + t.Fatal(err) + } + j, err := NewJWT(JWTConfig{Issuer: "issuer", Audience: "api", Keys: map[string]JWTKey{"first": {Algorithm: "EdDSA", PublicKey: pub}}}) + if err != nil { + t.Fatal(err) + } + claims := JWTClaims{RegisteredClaims: jwt.RegisteredClaims{Issuer: "issuer", Subject: "user1", Audience: jwt.ClaimStrings{"api"}, ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour))}, TenantID: "tenant1", Roles: []string{"reader"}} + sign := func(c JWTClaims, kid string, method jwt.SigningMethod, key any) string { + t.Helper() + token := jwt.NewWithClaims(method, c) + token.Header["kid"] = kid + raw, err := token.SignedString(key) + if err != nil { + t.Fatal(err) + } + return raw + } + raw := sign(claims, "first", jwt.SigningMethodEdDSA, private) + // Constructor clones caller-owned key bytes. + pub[0] ^= 1 + p, err := j.Authenticate(context.Background(), raw) + if err != nil || p.TenantID != "tenant1" || !p.HasRole("reader") { + t.Fatalf("%+v %v", p, err) + } + for name, change := range map[string]func(*JWTClaims){"issuer": func(c *JWTClaims) { c.Issuer = "bad" }, "audience": func(c *JWTClaims) { c.Audience = jwt.ClaimStrings{"other"} }, "expired": func(c *JWTClaims) { c.ExpiresAt = jwt.NewNumericDate(time.Now().Add(-time.Hour)) }, "noExpiry": func(c *JWTClaims) { c.ExpiresAt = nil }, "future": func(c *JWTClaims) { c.NotBefore = jwt.NewNumericDate(time.Now().Add(time.Hour)) }, "subject": func(c *JWTClaims) { c.Subject = "" }} { + t.Run(name, func(t *testing.T) { + bad := claims + change(&bad) + if _, err = j.Authenticate(context.Background(), sign(bad, "first", jwt.SigningMethodEdDSA, private)); !errors.Is(err, ErrInvalidToken) { + t.Fatal(err) + } + }) + } + for _, invalid := range []string{sign(claims, "missing", jwt.SigningMethodEdDSA, private), sign(claims, "first", jwt.SigningMethodHS256, []byte("invalid secret")), strings.Repeat("x", 8193)} { + if _, err = j.Authenticate(context.Background(), invalid); !errors.Is(err, ErrInvalidToken) { + t.Fatal(err) + } + } + if err = j.ReplaceKeys(map[string]JWTKey{"second": {Algorithm: "EdDSA", PublicKey: private.Public()}}); err != nil { + t.Fatal(err) + } + if _, err = j.Authenticate(context.Background(), raw); !errors.Is(err, ErrInvalidToken) { + t.Fatal(err) + } + if _, err = j.Authenticate(context.Background(), sign(claims, "second", jwt.SigningMethodEdDSA, private)); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if _, err = j.Authenticate(ctx, raw); !errors.Is(err, context.Canceled) { + t.Fatal(err) + } +} +func TestSessionsRevokeAndIsolation(t *testing.T) { + mem, _ := cache.NewMemory(cache.MemoryConfig{}) + defer mem.Close() + s, err := NewSessions(mem, "session:", time.Minute) + if err != nil { + t.Fatal(err) + } + ctx := context.Background() + token, err := s.Issue(ctx, Principal{UserID: "u1", TenantID: "t1", Roles: []string{"reader"}}) + if err != nil { + t.Fatal(err) + } + key, _ := s.key(token) + if strings.Contains(key, token) { + t.Fatal("plaintext session secret stored") + } + p, err := s.Authenticate(ctx, token) + if err != nil || p.UserID != "u1" { + t.Fatal(p, err) + } + p.Roles[0] = "admin" + p, _ = s.Authenticate(ctx, token) + if p.HasRole("admin") { + t.Fatal("principal alias") + } + if err = s.Revoke(ctx, token); err != nil { + t.Fatal(err) + } + if _, err = s.Authenticate(ctx, token); !errors.Is(err, ErrInvalidToken) { + t.Fatal(err) + } + if _, err = s.Authenticate(ctx, "bad"); !errors.Is(err, ErrInvalidToken) { + t.Fatal(err) + } +} diff --git a/auth/auth.go b/auth/auth.go index 20d96ef..f56962a 100644 --- a/auth/auth.go +++ b/auth/auth.go @@ -21,9 +21,10 @@ var ( // Principal is the authenticated identity made available to business handlers. // Claims are optional provider-specific values and should not be required by core business code. type Principal struct { - UserID string - Roles []string - Claims map[string]any + UserID string `json:"userId"` + TenantID string `json:"tenantId,omitempty"` + Roles []string `json:"roles,omitempty"` + Claims map[string]any `json:"claims,omitempty"` } func (p Principal) HasRole(role string) bool { diff --git a/auth/jwt.go b/auth/jwt.go new file mode 100644 index 0000000..21b8057 --- /dev/null +++ b/auth/jwt.go @@ -0,0 +1,127 @@ +package auth + +import ( + "context" + "crypto/ed25519" + "crypto/rsa" + "crypto/x509" + "errors" + "sync" + "time" + + "github.com/golang-jwt/jwt/v5" +) + +// JWTKey binds a trusted key ID to exactly one asymmetric signing algorithm. +// Keys come from server configuration, never a token's jku/x5u URL. +type JWTKey struct { + Algorithm string + PublicKey any +} +type JWTConfig struct { + Issuer string `json:"issuer"` + Audience string `json:"audience"` + Leeway time.Duration `json:"leeway"` + Keys map[string]JWTKey `json:"-"` +} +type JWT struct { + config JWTConfig + mu sync.RWMutex + keys map[string]JWTKey +} +type JWTClaims struct { + jwt.RegisteredClaims + TenantID string `json:"tenantId,omitempty"` + Roles []string `json:"roles,omitempty"` +} + +func NewJWT(cfg JWTConfig) (*JWT, error) { + if cfg.Issuer == "" || cfg.Audience == "" || cfg.Leeway < 0 || cfg.Leeway > 2*time.Minute { + return nil, errors.New("auth: issuer/audience required, leeway must be 0..2m") + } + j := &JWT{config: cfg} + j.config.Keys = nil + if err := j.ReplaceKeys(cfg.Keys); err != nil { + return nil, err + } + return j, nil +} + +// ReplaceKeys atomically rotates the trusted key set. Retain old keys until +// their tokens expire if a rotation should preserve existing sessions. +func (j *JWT) ReplaceKeys(keys map[string]JWTKey) error { + if len(keys) == 0 || len(keys) > 64 { + return errors.New("auth: require 1..64 JWT keys") + } + copyKeys := make(map[string]JWTKey, len(keys)) + for id, key := range keys { + if id == "" || len(id) > 128 { + return errors.New("auth: invalid key ID") + } + switch key.Algorithm { + case "RS256": + k, ok := key.PublicKey.(*rsa.PublicKey) + if !ok || k == nil || k.N == nil || k.N.BitLen() < 2048 || k.E < 3 { + return errors.New("auth: RS256 requires a >=2048-bit RSA public key") + } + case "EdDSA": + k, ok := key.PublicKey.(ed25519.PublicKey) + if !ok || len(k) != ed25519.PublicKeySize { + return errors.New("auth: EdDSA requires an Ed25519 public key") + } + default: + return errors.New("auth: supported algorithms are RS256 and EdDSA") + } + encoded, err := x509.MarshalPKIXPublicKey(key.PublicKey) + if err != nil { + return err + } + key.PublicKey, err = x509.ParsePKIXPublicKey(encoded) + if err != nil { + return err + } + copyKeys[id] = key + } + j.mu.Lock() + j.keys = copyKeys + j.mu.Unlock() + return nil +} + +func (j *JWT) Authenticate(ctx context.Context, raw string) (Principal, error) { + if ctx == nil { + return Principal{}, ErrInvalidToken + } + if err := ctx.Err(); err != nil { + return Principal{}, err + } + if len(raw) == 0 || len(raw) > 8192 { + return Principal{}, ErrInvalidToken + } + claims := new(JWTClaims) + token, err := jwt.ParseWithClaims(raw, claims, func(t *jwt.Token) (any, error) { + id, ok := t.Header["kid"].(string) + if !ok { + return nil, ErrInvalidToken + } + j.mu.RLock() + key, exists := j.keys[id] + j.mu.RUnlock() + if !exists || t.Method.Alg() != key.Algorithm { + return nil, ErrInvalidToken + } + return key.PublicKey, nil + }, jwt.WithValidMethods([]string{"RS256", "EdDSA"}), jwt.WithIssuer(j.config.Issuer), jwt.WithAudience(j.config.Audience), jwt.WithExpirationRequired(), jwt.WithIssuedAt(), jwt.WithLeeway(j.config.Leeway)) + if err != nil || !token.Valid || claims.Subject == "" || len(claims.Subject) > 256 || len(claims.TenantID) > 256 || len(claims.Roles) > 64 { + return Principal{}, ErrInvalidToken + } + for _, role := range claims.Roles { + if role == "" || len(role) > 128 { + return Principal{}, ErrInvalidToken + } + } + if err := ctx.Err(); err != nil { + return Principal{}, err + } + return Principal{UserID: claims.Subject, TenantID: claims.TenantID, Roles: append([]string(nil), claims.Roles...)}, nil +} diff --git a/auth/policy.go b/auth/policy.go new file mode 100644 index 0000000..fe53f2f --- /dev/null +++ b/auth/policy.go @@ -0,0 +1,34 @@ +package auth + +import ( + "context" + "errors" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/httpx" +) + +type Policy interface { + Authorize(context.Context, Principal, string, string) error +} +type PolicyFunc func(context.Context, Principal, string, string) error + +func (f PolicyFunc) Authorize(ctx context.Context, p Principal, action, resource string) error { + return f(ctx, p, action, resource) +} +func RequirePermission(policy Policy, action string, resource func(*httpx.Context) string) httpx.Middleware { + return func(next httpx.Handler) httpx.Handler { + return func(c *httpx.Context) error { + p, ok := FromContext(c) + if !ok { + return apierrors.Unauthorized("UNAUTHENTICATED", "authentication required") + } + if policy == nil || action == "" || resource == nil { + return apierrors.Internal(errors.New("auth: incomplete permission policy")) + } + if err := policy.Authorize(c.Context(), p, action, resource(c)); err != nil { + return apierrors.Forbidden("FORBIDDEN", "permission denied").WithCause(err) + } + return next(c) + } + } +} diff --git a/auth/session.go b/auth/session.go new file mode 100644 index 0000000..bb98411 --- /dev/null +++ b/auth/session.go @@ -0,0 +1,87 @@ +package auth + +import ( + "context" + "crypto/rand" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "encoding/json" + "errors" + "time" + + "git.yuntuo.hk/k2safe/go-omni/cache" +) + +// Sessions stores server-issued opaque bearer credentials as SHA-256 keys. +// The caller verifies login credentials before Issue; Store owns persistence. +type Sessions struct { + store cache.Store + prefix string + ttl time.Duration +} + +func NewSessions(store cache.Store, prefix string, ttl time.Duration) (*Sessions, error) { + if store == nil || len(prefix) > 512 || ttl < time.Second || ttl > 30*24*time.Hour { + return nil, errors.New("auth: invalid session configuration") + } + return &Sessions{store: store, prefix: prefix, ttl: ttl}, nil +} +func (s *Sessions) Issue(ctx context.Context, p Principal) (string, error) { + if p.UserID == "" || len(p.UserID) > 256 || len(p.TenantID) > 256 { + return "", ErrInvalidToken + } + data, err := json.Marshal(p) + if err != nil { + return "", err + } + if len(data) > 16384 { + return "", errors.New("auth: session principal too large") + } + secret := make([]byte, 32) + if _, err := rand.Read(secret); err != nil { + return "", err + } + token := base64.RawURLEncoding.EncodeToString(secret) + key, _ := s.key(token) + if err := s.store.Set(ctx, key, data, s.ttl); err != nil { + return "", err + } + return token, nil +} +func (s *Sessions) key(token string) (string, error) { + if len(token) != 43 { + return "", ErrInvalidToken + } + b, err := base64.RawURLEncoding.DecodeString(token) + if err != nil || len(b) != 32 { + return "", ErrInvalidToken + } + sum := sha256.Sum256(b) + return s.prefix + hex.EncodeToString(sum[:]), nil +} +func (s *Sessions) Authenticate(ctx context.Context, token string) (Principal, error) { + key, err := s.key(token) + if err != nil { + return Principal{}, err + } + data, err := s.store.Get(ctx, key) + if errors.Is(err, cache.ErrMiss) { + return Principal{}, ErrInvalidToken + } + if err != nil { + return Principal{}, err + } + var p Principal + if len(data) > 16384 || json.Unmarshal(data, &p) != nil || p.UserID == "" { + return Principal{}, ErrInvalidToken + } + return p, nil +} +func (s *Sessions) Revoke(ctx context.Context, token string) error { + key, err := s.key(token) + if err != nil { + return err + } + return s.store.Delete(ctx, key) +} diff --git a/cmd/go-omni/main.go b/cmd/go-omni/main.go index e46f111..6d866b5 100644 --- a/cmd/go-omni/main.go +++ b/cmd/go-omni/main.go @@ -17,6 +17,11 @@ func main() { } switch os.Args[1] { + case "migrate": + if err := runMigrate(os.Args[2:]); err != nil { + fmt.Fprintln(os.Stderr, "go-omni migrate:", err) + os.Exit(1) + } case "add": if err := runAdd(os.Args[2:]); err != nil { fmt.Fprintln(os.Stderr, "go-omni add:", err) @@ -108,6 +113,7 @@ func usage() { fmt.Fprintln(os.Stderr, "GoOmni project tool") fmt.Fprintln(os.Stderr, "") fmt.Fprintln(os.Stderr, "Usage:") + fmt.Fprintln(os.Stderr, " go-omni migrate [--dir migrations] [--apply] [--dsn-env GO_OMNI_DB_DSN]") fmt.Fprintln(os.Stderr, " go-omni new [--module ] [--force] ") fmt.Fprintln(os.Stderr, " go-omni add [--dir ] Generate an isolated business module") fmt.Fprintln(os.Stderr, " --framework-version Pin a remote framework version (defaults to CLI version)") diff --git a/cmd/go-omni/migrate.go b/cmd/go-omni/migrate.go new file mode 100644 index 0000000..63c9362 --- /dev/null +++ b/cmd/go-omni/migrate.go @@ -0,0 +1,60 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + migration "git.yuntuo.hk/k2safe/go-omni/database/migrate" + "io" + "os" + "os/signal" + "syscall" +) + +func runMigrate(args []string) error { + flags := flag.NewFlagSet("migrate", flag.ContinueOnError) + flags.SetOutput(io.Discard) + directory := flags.String("dir", "migrations", "SQL directory") + dsnEnv := flags.String("dsn-env", "GO_OMNI_DB_DSN", "DSN environment variable") + apply := flags.Bool("apply", false, "apply pending up migrations") + if err := flags.Parse(args); err != nil { + return err + } + if flags.NArg() != 0 { + return errors.New("migrate: unexpected argument") + } + _, manifest, err := migration.Snapshot(os.DirFS(*directory), ".") + if err != nil { + return err + } + if !*apply { + data, err := json.MarshalIndent(manifest, "", " ") + if err != nil { + return err + } + fmt.Println(string(data)) + return nil + } + dsn := os.Getenv(*dsnEnv) + if dsn == "" { + return errors.New("migrate: DSN environment variable is empty") + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + runner, err := migration.NewMySQL(ctx, dsn, os.DirFS(*directory), ".") + if err != nil { + return err + } + defer runner.Close() + if err = runner.Up(ctx); err != nil { + return err + } + version, dirty, err := runner.Version() + if err != nil { + return err + } + fmt.Printf("migration version=%d dirty=%t\n", version, dirty) + return nil +} diff --git a/cmd/go-omni/migrate_test.go b/cmd/go-omni/migrate_test.go new file mode 100644 index 0000000..6a812b1 --- /dev/null +++ b/cmd/go-omni/migrate_test.go @@ -0,0 +1,25 @@ +package main + +import ( + "os" + "path/filepath" + "testing" +) + +func TestMigrationPreviewDoesNotConnect(t *testing.T) { + dir := t.TempDir() + if err := os.WriteFile(filepath.Join(dir, "0001_createItems.up.sql"), []byte("CREATE TABLE testItems(id BIGINT);"), 0600); err != nil { + t.Fatal(err) + } + t.Setenv("GO_OMNI_DB_DSN", "not a DSN") + if err := runMigrate([]string{"--dir", dir}); err != nil { + t.Fatal("preview tried to connect", err) + } + t.Setenv("GO_OMNI_DB_DSN", "") + if err := runMigrate([]string{"--dir", dir, "--apply"}); err == nil { + t.Fatal("missing DSN accepted") + } + if err := runMigrate([]string{"--dir", dir, "--force"}); err == nil { + t.Fatal("unsafe unknown flag accepted") + } +} diff --git a/config/env.go b/config/env.go index fa200aa..7267680 100644 --- a/config/env.go +++ b/config/env.go @@ -13,15 +13,18 @@ const defaultEnvPrefix = "GO_OMNI_" // LoadFromEnv loads application settings from GO_OMNI_* variables. The // database settings remain owned by database.ConfigFromEnv. func LoadFromEnv(prefix string) (Config, error) { + return applyEnv(Default(), prefix) +} + +func applyEnv(cfg Config, prefix string) (Config, error) { prefix = strings.TrimSpace(prefix) if prefix == "" { prefix = defaultEnvPrefix } - cfg := Default() cfg.Address = envString(prefix+"HTTP_ADDR", cfg.Address) cfg.Transport = envString(prefix+"TRANSPORT", cfg.Transport) - cfg.TLS.CertFile = envString(prefix+"TLS_CERT_FILE", "") - cfg.TLS.KeyFile = envString(prefix+"TLS_KEY_FILE", "") + cfg.TLS.CertFile = envString(prefix+"TLS_CERT_FILE", cfg.TLS.CertFile) + cfg.TLS.KeyFile = envString(prefix+"TLS_KEY_FILE", cfg.TLS.KeyFile) cfg.RequestIDHeader = envString(prefix+"REQUEST_ID_HEADER", cfg.RequestIDHeader) if value := strings.TrimSpace(os.Getenv(prefix + "TRUSTED_PROXIES")); value != "" { cfg.TrustedProxies = strings.Split(value, ",") diff --git a/config/file.go b/config/file.go new file mode 100644 index 0000000..e29af26 --- /dev/null +++ b/config/file.go @@ -0,0 +1,203 @@ +package config + +import ( + "bytes" + "context" + "encoding/json" + "errors" + "git.yuntuo.hk/k2safe/go-omni/logx" + "gopkg.in/yaml.v3" + "io" + "os" + "path/filepath" + "reflect" + "strings" + "sync" + "time" +) + +// DecodeFile merges a bounded JSON/YAML document into defaults. It uses JSON +// field names for both formats, rejects unknown fields and accepts durations +// such as "3s". Application-specific validation remains with the caller. +func DecodeFile[T any](path string, defaults T) (T, error) { + var zero T + f, err := os.Open(path) + if err != nil { + return zero, err + } + defer f.Close() + data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1)) + if err != nil { + return zero, err + } + if len(data) > 1<<20 { + return zero, errors.New("config: file exceeds 1 MiB") + } + var value any + switch strings.ToLower(filepath.Ext(path)) { + case ".yaml", ".yml": + dec := yaml.NewDecoder(bytes.NewReader(data)) + if err := dec.Decode(&value); err != nil { + return zero, err + } + var extra any + if err := dec.Decode(&extra); err != io.EOF { + return zero, errors.New("config: one YAML document required") + } + case ".json": + dec := json.NewDecoder(bytes.NewReader(data)) + dec.UseNumber() + if err := dec.Decode(&value); err != nil { + return zero, err + } + var extra any + if err := dec.Decode(&extra); err != io.EOF { + return zero, errors.New("config: one JSON document required") + } + default: + return zero, errors.New("config: expected .json, .yaml or .yml") + } + if _, ok := value.(map[string]any); !ok { + return zero, errors.New("config: root must be an object") + } + value, err = durations(value, reflect.TypeOf(defaults)) + if err != nil { + return zero, err + } + encoded, err := json.Marshal(value) + if err != nil { + return zero, err + } + // Deep copy defaults so errors cannot partially mutate caller-owned maps. + original, err := json.Marshal(defaults) + if err != nil { + return zero, err + } + var result T + if err = json.Unmarshal(original, &result); err != nil { + return zero, err + } + dec := json.NewDecoder(bytes.NewReader(encoded)) + dec.DisallowUnknownFields() + if err = dec.Decode(&result); err != nil { + return zero, err + } + return result, nil +} +func durations(v any, t reflect.Type) (any, error) { + if t == nil { + return v, nil + } + for t.Kind() == reflect.Pointer { + t = t.Elem() + } + if t == reflect.TypeOf(time.Duration(0)) { + if s, ok := v.(string); ok { + d, err := time.ParseDuration(s) + return int64(d), err + } + return v, nil + } + if t.Kind() == reflect.Struct { + if m, ok := v.(map[string]any); ok { + for n := 0; n < t.NumField(); n++ { + field := t.Field(n) + name := strings.Split(field.Tag.Get("json"), ",")[0] + if name == "-" { + continue + } + if name == "" { + name = field.Name + } + if item, exists := m[name]; exists { + converted, err := durations(item, field.Type) + if err != nil { + return nil, err + } + m[name] = converted + } + } + } + } + if t.Kind() == reflect.Slice { + if items, ok := v.([]any); ok { + for n, item := range items { + converted, err := durations(item, t.Elem()) + if err != nil { + return nil, err + } + items[n] = converted + } + } + } + return v, nil +} +func LoadFile(path, prefix string) (Config, error) { + cfg, err := DecodeFile(path, Default()) + if err != nil { + return Config{}, err + } + return applyEnv(cfg, prefix) +} + +// Reloader only accepts log.level changes. Static changes and invalid files +// leave the current snapshot and logger untouched. Watch does not own a goroutine. +type Reloader struct { + mu sync.Mutex + path, prefix string + current Config + logger *logx.Logger +} + +func NewReloader(path, prefix string, logger *logx.Logger) (*Reloader, error) { + cfg, err := LoadFile(path, prefix) + if err != nil { + return nil, err + } + if err = logger.SetLevel(cfg.Log.Level); err != nil { + return nil, err + } + return &Reloader{path: path, prefix: prefix, current: cfg, logger: logger}, nil +} +func (r *Reloader) Reload() error { + r.mu.Lock() + defer r.mu.Unlock() + next, err := LoadFile(r.path, r.prefix) + if err != nil { + return err + } + comparable := next + comparable.Log.Level = r.current.Log.Level + if !reflect.DeepEqual(comparable, r.current) { + return errors.New("config: static fields changed; restart required") + } + if err = r.logger.SetLevel(next.Log.Level); err != nil { + return err + } + r.current = next + return nil +} +func (r *Reloader) Snapshot() Config { + r.mu.Lock() + defer r.mu.Unlock() + cfg := r.current + cfg.TrustedProxies = append([]string(nil), cfg.TrustedProxies...) + return cfg +} +func (r *Reloader) Watch(ctx context.Context, interval time.Duration, onError func(error)) error { + if ctx == nil || interval < 100*time.Millisecond { + return errors.New("config: context and interval >=100ms required") + } + ticker := time.NewTicker(interval) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return ctx.Err() + case <-ticker.C: + if err := r.Reload(); err != nil && onError != nil { + onError(err) + } + } + } +} diff --git a/config/file_test.go b/config/file_test.go new file mode 100644 index 0000000..2f43c98 --- /dev/null +++ b/config/file_test.go @@ -0,0 +1,59 @@ +package config + +import ( + "bytes" + "git.yuntuo.hk/k2safe/go-omni/logx" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestFileMergeAndSafeReload(t *testing.T) { + path := filepath.Join(t.TempDir(), "service.yaml") + write := func(raw string) { + t.Helper() + if err := os.WriteFile(path, []byte(raw), 0600); err != nil { + t.Fatal(err) + } + } + write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n") + t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123") + cfg, err := LoadFile(path, "TEST_MODULE_") + if err != nil || cfg.Address != ":9123" || cfg.ShutdownTimeout != 7*time.Second { + t.Fatal(cfg, err) + } + var output bytes.Buffer + lc := cfg.Log + lc.Writer = &output + l, err := logx.New(lc) + if err != nil { + t.Fatal(err) + } + defer l.Close() + child := l.With("scope", "test") + r, err := NewReloader(path, "TEST_MODULE_", l) + if err != nil { + t.Fatal(err) + } + child.Debug("hidden") + write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: debug\n") + if err = r.Reload(); err != nil { + t.Fatal(err) + } + child.Debug("visible") + if strings.Contains(output.String(), "hidden") || !strings.Contains(output.String(), "visible") { + t.Fatal(output.String()) + } + write("shutdownTimeout: 8s\nlog:\n level: error\n") + if err = r.Reload(); err == nil || r.Snapshot().Log.Level != "debug" { + t.Fatal("static reload changed live config", err) + } + for _, raw := range []string{"unknownOption: true", "log:\n level: bad", "shutdownTimeout: invalid", "address: x\naddress: y", "---\n{}\n---\n{}"} { + write(raw) + if err = r.Reload(); err == nil { + t.Fatal(raw) + } + } +} diff --git a/database/migrate/migrate.go b/database/migrate/migrate.go new file mode 100644 index 0000000..51e3cda --- /dev/null +++ b/database/migrate/migrate.go @@ -0,0 +1,245 @@ +// Package migrate provides explicit MySQL schema migration jobs. Application +// startup never runs migrations automatically; use a separate deployment step. +package migrate + +import ( + "archive/zip" + "bytes" + "context" + "crypto/sha256" + "database/sql" + "encoding/hex" + "errors" + "fmt" + mysqlDriver "github.com/go-sql-driver/mysql" + migration "github.com/golang-migrate/migrate/v4" + migrateMySQL "github.com/golang-migrate/migrate/v4/database/mysql" + "github.com/golang-migrate/migrate/v4/source/iofs" + "io" + "io/fs" + "path" + "regexp" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +type File struct { + Version uint64 `json:"version"` + Name string `json:"name"` + Direction string `json:"direction"` + SHA256 string `json:"sha256"` +} + +var filename = regexp.MustCompile(`^([0-9]+)_([A-Za-z][A-Za-z0-9]*)\.(up|down)\.sql$`) + +// Snapshot returns immutable migration bytes and a reviewable digest manifest. +// Follow lowerCamelCase names, e.g. 0001_createUsers.up.sql. +func Snapshot(source fs.FS, directory string) (fs.FS, []File, error) { + if source == nil || !fs.ValidPath(directory) { + return nil, nil, errors.New("migrate: invalid source directory") + } + entries, err := fs.ReadDir(source, directory) + if err != nil { + return nil, nil, err + } + var archive bytes.Buffer + writer := zip.NewWriter(&archive) + manifest := []File{} + seen := map[string]bool{} + total := 0 + for _, entry := range entries { + if entry.IsDir() { + continue + } + parts := filename.FindStringSubmatch(entry.Name()) + if parts == nil { + if strings.HasSuffix(entry.Name(), ".sql") { + return nil, nil, fmt.Errorf("migrate: invalid migration filename %s", entry.Name()) + } + continue + } + if len(manifest) >= 1000 { + return nil, nil, errors.New("migrate: too many files") + } + version, err := strconv.ParseUint(parts[1], 10, 32) + if err != nil || version == 0 { + return nil, nil, errors.New("migrate: version must be 1..4294967295") + } + identity := fmt.Sprint(version) + parts[3] + if seen[identity] { + return nil, nil, errors.New("migrate: duplicate migration version/direction") + } + seen[identity] = true + file, err := source.Open(path.Join(directory, entry.Name())) + if err != nil { + return nil, nil, err + } + data, err := io.ReadAll(io.LimitReader(file, (1<<20)+1)) + closeErr := file.Close() + if err == nil { + err = closeErr + } + if err != nil { + return nil, nil, err + } + total += len(data) + if len(data) == 0 || len(data) > 1<<20 || total > 16<<20 { + return nil, nil, errors.New("migrate: files must be 1..1MiB and <=16MiB total") + } + sum := sha256.Sum256(data) + manifest = append(manifest, File{Version: version, Name: parts[2], Direction: parts[3], SHA256: hex.EncodeToString(sum[:])}) + fileWriter, err := writer.CreateHeader(&zip.FileHeader{Name: entry.Name(), Method: zip.Store}) + if err != nil { + return nil, nil, err + } + if _, err = fileWriter.Write(data); err != nil { + return nil, nil, err + } + } + if len(manifest) == 0 { + return nil, nil, errors.New("migrate: no SQL migrations") + } + for _, file := range manifest { + if file.Direction == "down" && !seen[fmt.Sprint(file.Version)+"up"] { + return nil, nil, errors.New("migrate: down file lacks up migration") + } + } + sort.Slice(manifest, func(i, j int) bool { + if manifest[i].Version == manifest[j].Version { + return manifest[i].Direction < manifest[j].Direction + } + return manifest[i].Version < manifest[j].Version + }) + if err = writer.Close(); err != nil { + return nil, nil, err + } + files, err := zip.NewReader(bytes.NewReader(archive.Bytes()), int64(archive.Len())) + return files, manifest, err +} + +type Runner struct { + mu sync.Mutex + engine *migration.Migrate + db *sql.DB + closed bool + stopped bool +} + +func NewMySQL(ctx context.Context, dsn string, source fs.FS, directory string) (runner *Runner, err error) { + if ctx == nil { + return nil, errors.New("migrate: context required") + } + snapshot, _, err := Snapshot(source, directory) + if err != nil { + return nil, err + } + cfg, err := mysqlDriver.ParseDSN(dsn) + if err != nil || cfg.DBName == "" { + return nil, errors.New("migrate: valid MySQL DSN with database required") + } + cfg.MultiStatements = true + if cfg.Timeout == 0 || cfg.Timeout > 5*time.Second { + cfg.Timeout = 5 * time.Second + } + if cfg.ReadTimeout == 0 || cfg.ReadTimeout > 30*time.Second { + cfg.ReadTimeout = 30 * time.Second + } + if cfg.WriteTimeout == 0 || cfg.WriteTimeout > 30*time.Second { + cfg.WriteTimeout = 30 * time.Second + } + connector, err := mysqlDriver.NewConnector(cfg) + if err != nil { + return nil, err + } + db := sql.OpenDB(connector) + db.SetMaxOpenConns(1) + db.SetMaxIdleConns(1) + defer func() { + if err != nil { + db.Close() + } + }() + conn, err := db.Conn(ctx) + if err != nil { + return nil, err + } + driver, err := migrateMySQL.WithConnection(ctx, conn, &migrateMySQL.Config{DatabaseName: cfg.DBName, StatementTimeout: 30 * time.Second}) + if err != nil { + conn.Close() + return nil, err + } + src, err := iofs.New(snapshot, ".") + if err != nil { + driver.Close() + return nil, err + } + engine, err := migration.NewWithInstance("iofs", src, cfg.DBName, driver) + if err != nil { + src.Close() + driver.Close() + return nil, err + } + engine.LockTimeout = 10 * time.Second + return &Runner{engine: engine, db: db}, nil +} + +// Up applies pending up scripts. MySQL DDL can partially commit; a failed +// migration remains dirty and requires operator investigation. No automatic +// down/force/dirty reset is provided. Cancellation stops after the current SQL +// operation; statements/read I/O are bounded to 30 seconds. +func (r *Runner) Up(ctx context.Context) error { + if ctx == nil { + return errors.New("migrate: context required") + } + if err := ctx.Err(); err != nil { + return err + } + r.mu.Lock() + defer r.mu.Unlock() + if r.closed || r.stopped { + return errors.New("migrate: closed") + } + done := make(chan error, 1) + go func() { done <- r.engine.Up() }() + var err error + select { + case err = <-done: + case <-ctx.Done(): + r.stopped = true // golang-migrate retains its graceful-stop state; create a fresh runner to retry. + select { + case r.engine.GracefulStop <- true: + default: + } + err = <-done + return errors.Join(ctx.Err(), err) + } + if errors.Is(err, migration.ErrNoChange) { + return nil + } + return err +} +func (r *Runner) Version() (uint, bool, error) { + r.mu.Lock() + defer r.mu.Unlock() + if r.closed { + return 0, false, errors.New("migrate: closed") + } + version, dirty, err := r.engine.Version() + if errors.Is(err, migration.ErrNilVersion) { + return 0, false, nil + } + return version, dirty, err +} +func (r *Runner) Close() error { + r.mu.Lock() + defer r.mu.Unlock() + if r.closed { + return nil + } + r.closed = true + a, b := r.engine.Close() + return errors.Join(a, b, r.db.Close()) +} diff --git a/database/migrate/migrate_test.go b/database/migrate/migrate_test.go new file mode 100644 index 0000000..972dede --- /dev/null +++ b/database/migrate/migrate_test.go @@ -0,0 +1,99 @@ +package migrate + +import ( + "context" + "database/sql" + "git.yuntuo.hk/k2safe/go-omni/telemetry" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + "io/fs" + "os" + "testing" + "testing/fstest" + "time" +) + +func TestSnapshotBoundsAndIdentity(t *testing.T) { + source := fstest.MapFS{"0001_createItems.up.sql": &fstest.MapFile{Data: []byte("SELECT 1;")}} + f, manifest, err := Snapshot(source, ".") + if err != nil || len(manifest) != 1 { + t.Fatal(manifest, err) + } + source["0001_createItems.up.sql"].Data[0] = 'x' + raw, err := fs.ReadFile(f, "0001_createItems.up.sql") + if err != nil || string(raw) != "SELECT 1;" { + t.Fatal(string(raw), err) + } + for _, files := range []fstest.MapFS{{"0001_bad_name.up.sql": &fstest.MapFile{Data: []byte("x")}}, {"0001_a.up.sql": &fstest.MapFile{Data: []byte("x")}, "1_b.up.sql": &fstest.MapFile{Data: []byte("x")}}, {"0001_a.down.sql": &fstest.MapFile{Data: []byte("x")}}, {"0001_a.up.sql": &fstest.MapFile{Data: make([]byte, (1<<20)+1)}}} { + if _, _, err = Snapshot(files, "."); err == nil { + t.Fatal("invalid migration accepted") + } + } +} +func TestMySQLIntegration(t *testing.T) { + dsn := os.Getenv("GO_OMNI_MIGRATE_TEST_DSN") + if dsn == "" { + t.Skip("requires disposable MySQL database") + } + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + files := fstest.MapFS{"0001_createItems.up.sql": &fstest.MapFile{Data: []byte("CREATE TABLE testItems (id BIGINT NOT NULL PRIMARY KEY, value VARCHAR(64) NOT NULL);")}, "0002_seedItems.up.sql": &fstest.MapFile{Data: []byte("INSERT INTO testItems (id,value) VALUES (1,'original');")}} + runner, err := NewMySQL(ctx, dsn, files, ".") + if err != nil { + t.Fatal(err) + } + if err = runner.Up(ctx); err != nil { + t.Fatal(err) + } + if err = runner.Up(ctx); err != nil { + t.Fatal("repeat", err) + } + v, dirty, err := runner.Version() + if err != nil || v != 2 || dirty { + t.Fatal(v, dirty, err) + } + runner.Close() + db, err := sql.Open("mysql", dsn) + if err != nil { + t.Fatal(err) + } + defer db.Close() + exporter := tracetest.NewInMemoryExporter() + tr, _ := telemetry.New(telemetry.Config{Service: "dbTest", SampleRatio: 1, Exporter: exporter}) + defer tr.Close() + instrumented := telemetry.SQL{DB: db, Runtime: tr} + if _, err = instrumented.ExecContext(ctx, "UPDATE testItems SET value=? WHERE id=1", "secret parameter"); err != nil { + t.Fatal(err) + } + rows, err := instrumented.QueryContext(ctx, "SELECT value FROM testItems WHERE id=1") + if err != nil { + t.Fatal(err) + } + if !rows.Next() { + t.Fatal("missing row") + } + var value string + if err = rows.Scan(&value); err != nil || value != "secret parameter" { + t.Fatal(value, err) + } + rows.Close() + tr.Flush(ctx) + if len(exporter.GetSpans()) != 2 { + t.Fatal("SQL spans missing") + } + files["0003_failMigration.up.sql"] = &fstest.MapFile{Data: []byte("INSERT INTO missingTable VALUES (1);")} + runner, err = NewMySQL(ctx, dsn, files, ".") + if err != nil { + t.Fatal(err) + } + defer runner.Close() + if err = runner.Up(ctx); err == nil { + t.Fatal("bad migration passed") + } + v, dirty, err = runner.Version() + if err != nil || v != 3 || !dirty { + t.Fatal(v, dirty, err) + } + if err = runner.Up(ctx); err == nil { + t.Fatal("dirty migration auto-cleared") + } +} diff --git a/examples/platform/README.md b/examples/platform/README.md new file mode 100644 index 0000000..26a7558 --- /dev/null +++ b/examples/platform/README.md @@ -0,0 +1,14 @@ +# 扩展模块组合示例 + +业务通过 `storage.Store` 写文件,身份接口和 Policy 控制权限,OpenAPI 与路由一起注册,链路追踪关联 HTTP 与文件操作。`newApp` 不依赖 Local/S3/OSS 类型;启动处本例选择 Local。可替换成 S3/R2/OSS 构造结果,业务路由不变。 + +```sh +export GO_OMNI_DEMO_TOKEN='use-a-private-development-token' +go run ./examples/platform --root /tmp/goOmniDemoFiles +``` + +接口:`POST /api/files`,JSON 为 `{"name":"hello.txt","content":"hello"}`,请求加 `Authorization: Bearer `;返回 201 和存储信息。`GET /openapi.json` 同样要求身份。 + +也可传 `--config service.yaml`,配置规则见 [扩展模块](../../ADVANCED.md)。本例的静态身份只用于演示,实际应用注入 `auth.JWT` 或 `auth.Sessions`;trace provider 默认不外发,接入 collector 时注入 exporter/reader。 + +`TestPlatformTLSAdapters` 通过三个 Runner 的真实 TLS 请求验证 401/403/201、文件内容、非法 key、OpenAPI 和关闭。文件根目录必须是应用私有目录。示例不执行自动数据迁移,也不启动 Orbit。 diff --git a/examples/platform/main.go b/examples/platform/main.go new file mode 100644 index 0000000..f86fb51 --- /dev/null +++ b/examples/platform/main.go @@ -0,0 +1,132 @@ +// platform composes storage, authentication, permission policy, tracing and +// OpenAPI using provider-neutral business dependencies. +package main + +import ( + "context" + "errors" + "flag" + "log" + "os" + "os/signal" + "strings" + "syscall" + + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/auth" + "git.yuntuo.hk/k2safe/go-omni/config" + "git.yuntuo.hk/k2safe/go-omni/openapi" + "git.yuntuo.hk/k2safe/go-omni/storage" + "git.yuntuo.hk/k2safe/go-omni/storage/local" + "git.yuntuo.hk/k2safe/go-omni/telemetry" +) + +type dependencies struct { + Files storage.Store + Auth auth.Authenticator + Tracing *telemetry.Runtime +} +type upload struct { + Name string `json:"name" binding:"required,max=128"` + Content string `json:"content" binding:"max=4096"` +} + +// Dependencies are borrowed; the composition root owns their shutdown. +func newApp(cfg omni.Config, d dependencies) (*omni.App, error) { + if d.Files == nil || d.Auth == nil || d.Tracing == nil { + return nil, errors.New("platform: dependencies required") + } + app := omni.New(omni.WithConfig(cfg)) + app.Use(d.Tracing.Middleware()) + api, err := openapi.New("File service", "1") + if err != nil { + return nil, err + } + input, err := openapi.SchemaOf[upload]() + if err != nil { + return nil, err + } + output, err := openapi.SchemaOf[storage.Info]() + if err != nil { + return nil, err + } + policy := auth.PolicyFunc(func(_ context.Context, p auth.Principal, action, resource string) error { + if p.TenantID != "" && p.HasRole("writer") && action == "create" && resource == "files" { + return nil + } + return errors.New("denied") + }) + err = api.Handle(app, openapi.Operation{Method: "POST", Path: "/api/files", OperationID: "createFile", Summary: "Store a small text file", Request: input, BearerAuth: true, Responses: map[int]openapi.Response{201: {Description: "Created", Schema: openapi.Envelope(output)}}}, func(c *omni.Context) error { + var in upload + if err := c.BindJSON(&in); err != nil { + return err + } + p, _ := auth.FromContext(c) + if strings.ContainsAny(in.Name, "/\\") { + return omni.BadRequest("INVALID_NAME", "name must be one path segment") + } + key := p.TenantID + "/" + in.Name + if err := storage.Check(c.Context(), key); err != nil { + return omni.BadRequest("INVALID_KEY", "invalid storage key") + } + var info storage.Info + err := d.Tracing.Do(c.Context(), "files.put", func(ctx context.Context) error { + var err error + info, err = d.Files.Put(ctx, key, strings.NewReader(in.Content), int64(len(in.Content)), "text/plain; charset=utf-8") + return err + }) + if err != nil { + return err + } + return c.Created(info) + }, auth.RequireAuth(d.Auth), auth.RequirePermission(policy, "create", func(*omni.Context) string { return "files" })) + if err != nil { + app.Close() + return nil, err + } + app.GET("/openapi.json", api.Handler(), auth.RequireAuth(d.Auth)) + return app, nil +} +func run(ctx context.Context, configFile, root string) error { + var cfg config.Config + var err error + if configFile != "" { + cfg, err = config.LoadFile(configFile, "") + } else { + cfg, err = config.LoadFromEnv("") + if os.Getenv("GO_OMNI_HTTP_ADDR") == "" { + cfg.Address = "127.0.0.1:8080" + } + } + if err != nil { + return err + } + token := os.Getenv("GO_OMNI_DEMO_TOKEN") + if token == "" { + return errors.New("set GO_OMNI_DEMO_TOKEN for the demonstration credential") + } + files, err := local.New(root, 64<<10) + if err != nil { + return err + } + tracing, err := telemetry.New(telemetry.Config{Service: "fileService", SampleRatio: 1}) + if err != nil { + return err + } + defer tracing.Close() + app, err := newApp(cfg, dependencies{Files: files, Auth: auth.StaticAuthenticator{token: {UserID: "demoUser", TenantID: "demoTenant", Roles: []string{"writer"}}}, Tracing: tracing}) + if err != nil { + return err + } + return app.RunContext(ctx) +} +func main() { + file := flag.String("config", "", "optional JSON/YAML config") + root := flag.String("root", "./fileData", "private local object directory") + flag.Parse() + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + if err := run(ctx, *file, *root); err != nil { + log.Fatal(err) + } +} diff --git a/examples/platform/main_test.go b/examples/platform/main_test.go new file mode 100644 index 0000000..25bdfe5 --- /dev/null +++ b/examples/platform/main_test.go @@ -0,0 +1,102 @@ +package main + +import ( + "context" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/auth" + "git.yuntuo.hk/k2safe/go-omni/config" + "git.yuntuo.hk/k2safe/go-omni/internal/testutil" + "git.yuntuo.hk/k2safe/go-omni/storage/local" + "git.yuntuo.hk/k2safe/go-omni/telemetry" + "io" + "net" + "net/http" + "strings" + "testing" + "time" +) + +func TestPlatformTLSAdapters(t *testing.T) { + for _, name := range []string{omni.TransportNetHTTP, omni.TransportNative, omni.TransportHTTP2} { + t.Run(name, func(t *testing.T) { + cfg := config.Default() + cfg.Transport = name + cert, key, tlsClient := testutil.CertificateFiles(t) + cfg.TLS = config.TLSConfig{CertFile: cert, KeyFile: key} + store, _ := local.New(t.TempDir(), 64<<10) + tr, _ := telemetry.New(telemetry.Config{Service: "platformTest", SampleRatio: 1}) + defer tr.Close() + app, err := newApp(cfg, dependencies{Files: store, Auth: auth.StaticAuthenticator{"writer": {UserID: "u", TenantID: "t", Roles: []string{"writer"}}, "reader": {UserID: "r", TenantID: "t"}}, Tracing: tr}) + if err != nil { + t.Fatal(err) + } + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- app.ServeContext(ctx, listener) }() + defer func() { + cancel() + select { + case err := <-done: + if err != nil { + t.Error(err) + } + case <-time.After(5 * time.Second): + listener.Close() + t.Error("shutdown timeout") + } + }() + client := &http.Client{Timeout: 3 * time.Second, Transport: &http.Transport{TLSClientConfig: tlsClient, ForceAttemptHTTP2: true}} + defer client.CloseIdleConnections() + request := func(method, path, body, token string) (int, string) { + t.Helper() + r, _ := http.NewRequest(method, "https://"+listener.Addr().String()+path, strings.NewReader(body)) + r.Header.Set("Authorization", "Bearer "+token) + r.Header.Set("Content-Type", "application/json") + response, err := client.Do(r) + if err != nil { + t.Fatal(err) + } + raw, err := io.ReadAll(response.Body) + response.Body.Close() + if err != nil { + t.Fatal(err) + } + if response.TLS == nil || len(response.TLS.VerifiedChains) == 0 { + t.Fatal("TLS unverified") + } + if name == omni.TransportHTTP2 && response.ProtoMajor != 2 { + t.Fatal("protocol") + } + return response.StatusCode, string(raw) + } + for _, test := range []struct { + token string + status int + }{{"", 401}, {"reader", 403}, {"writer", 201}} { + status, body := request("POST", "/api/files", `{"name":"hello.txt","content":"hello"}`, test.token) + if status != test.status { + t.Fatal(status, body) + } + } + object, err := store.Get(context.Background(), "t/hello.txt") + if err != nil { + t.Fatal(err) + } + raw, _ := io.ReadAll(object.Body) + object.Body.Close() + if string(raw) != "hello" { + t.Fatal(string(raw)) + } + if status, _ := request("POST", "/api/files", `{"name":"../escape","content":"x"}`, "writer"); status != 400 { + t.Fatal(status) + } + if status, body := request("GET", "/openapi.json", "", "writer"); status != 200 || !strings.Contains(body, `"/api/files"`) { + t.Fatal(status, body) + } + }) + } +} diff --git a/go.mod b/go.mod index ba19c3d..4ededc7 100644 --- a/go.mod +++ b/go.mod @@ -1,25 +1,48 @@ module git.yuntuo.hk/k2safe/go-omni -go 1.22 +go 1.22.0 require ( + github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0 github.com/go-playground/validator/v10 v10.20.0 github.com/go-sql-driver/mysql v1.9.3 + github.com/golang-jwt/jwt/v5 v5.3.1 + github.com/golang-migrate/migrate/v4 v4.18.1 + github.com/minio/minio-go/v7 v7.0.83 github.com/redis/go-redis/v9 v9.17.3 - golang.org/x/mod v0.17.0 - golang.org/x/net v0.25.0 + go.opentelemetry.io/otel v1.31.0 + go.opentelemetry.io/otel/metric v1.31.0 + go.opentelemetry.io/otel/sdk v1.31.0 + go.opentelemetry.io/otel/sdk/metric v1.31.0 + go.opentelemetry.io/otel/trace v1.31.0 + golang.org/x/mod v0.21.0 + golang.org/x/net v0.33.0 + gopkg.in/yaml.v3 v3.0.1 ) require ( filippo.io/edwards25519 v1.1.0 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect + github.com/dustin/go-humanize v1.0.1 // indirect github.com/gabriel-vasile/mimetype v1.4.3 // indirect + github.com/go-ini/ini v1.67.0 // indirect + github.com/go-logr/logr v1.4.2 // indirect + github.com/go-logr/stdr v1.2.2 // indirect github.com/go-playground/locales v0.14.1 // indirect github.com/go-playground/universal-translator v0.18.1 // indirect + github.com/goccy/go-json v0.10.4 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/hashicorp/errwrap v1.1.0 // indirect + github.com/hashicorp/go-multierror v1.1.1 // indirect + github.com/klauspost/compress v1.17.11 // indirect + github.com/klauspost/cpuid/v2 v2.2.9 // indirect github.com/leodido/go-urn v1.4.0 // indirect - github.com/stretchr/testify v1.9.0 // indirect - golang.org/x/crypto v0.23.0 // indirect - golang.org/x/sys v0.20.0 // indirect - golang.org/x/text v0.15.0 // indirect + github.com/minio/md5-simd v1.1.2 // indirect + github.com/rs/xid v1.6.0 // indirect + go.uber.org/atomic v1.7.0 // indirect + golang.org/x/crypto v0.31.0 // indirect + golang.org/x/sys v0.28.0 // indirect + golang.org/x/text v0.21.0 // indirect + golang.org/x/time v0.5.0 // indirect ) diff --git a/go.sum b/go.sum index 2179257..babf363 100644 --- a/go.sum +++ b/go.sum @@ -1,17 +1,45 @@ filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA= filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= +github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0= +github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= +github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY= +github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU= +github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0 h1:uWzn3io54f9L9mvwsQQSv1KpkkFA06hBxI++RvIyvpI= +github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0/go.mod h1:FTzydeQVmR24FI0D6XWUOMKckjXehM/jgMn1xC+DA9M= github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs= github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c= github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA= github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78= github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc= +github.com/dhui/dktest v0.4.3 h1:wquqUxAFdcUgabAVLvSCOKOlag5cIZuaOjYIBOWdsR0= +github.com/dhui/dktest v0.4.3/go.mod h1:zNK8IwktWzQRm6I/l2Wjp7MakiyaFWv4G1hjmodmMTs= +github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk= +github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E= +github.com/docker/docker v27.2.0+incompatible h1:Rk9nIVdfH3+Vz4cyI/uhbINhEZ/oLmc+CBXmH6fbNk4= +github.com/docker/docker v27.2.0+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk= +github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c= +github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc= +github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4= +github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk= +github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY= +github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= +github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= +github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0= github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk= +github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A= +github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= +github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s= github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4= github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA= @@ -22,23 +50,85 @@ github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBEx github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM= github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo= github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU= +github.com/goccy/go-json v0.10.4 h1:JSwxQzIqKfmFX1swYPpUThQZp/Ka4wzJdK0LWVytLPM= +github.com/goccy/go-json v0.10.4/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M= +github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= +github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= +github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY= +github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= +github.com/golang-migrate/migrate/v4 v4.18.1 h1:JML/k+t4tpHCpQTCAD62Nu43NUFzHY4CV3uAuvHGC+Y= +github.com/golang-migrate/migrate/v4 v4.18.1/go.mod h1:HAX6m3sQgcdO81tdjn5exv20+3Kb13cmGli1hrD6hks= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I= +github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= +github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo= +github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM= +github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc= +github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0= +github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY= +github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8= github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ= github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI= +github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw= +github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o= +github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34= +github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM= +github.com/minio/minio-go/v7 v7.0.83 h1:W4Kokksvlz3OKf3OqIlzDNKd4MERlC2oN8YptwJ0+GA= +github.com/minio/minio-go/v7 v7.0.83/go.mod h1:57YXpvc5l3rjPdhqNrDsvVlY0qPI6UTk1bflAe+9doY= +github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0= +github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo= +github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0= +github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y= +github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A= +github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc= +github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= +github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= +github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug= +github.com/opencontainers/image-spec v1.1.0/go.mod h1:W4s4sFTMaBeK1BQLXbG4AdM2szdn85PY75RI83NrTrM= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= github.com/redis/go-redis/v9 v9.17.3 h1:fN29NdNrE17KttK5Ndf20buqfDZwGNgoUr9qjl1DQx4= github.com/redis/go-redis/v9 v9.17.3/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370= +github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU= +github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg= github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY= -golang.org/x/crypto v0.23.0 h1:dIJU/v2J8Mdglj/8rJ6UUOM3Zc9zLZxVZwwxMooUSAI= -golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8= -golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA= -golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c= -golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac= -golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= -golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y= -golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= -golang.org/x/text v0.15.0 h1:h1V/4gjBv8v9cjcR6+AR5+/cIYK5N/WAgiv4xlsEtAk= -golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 h1:TT4fX+nBOA/+LUkobKGW1ydGcn+G3vRw9+g5HwCphpk= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0/go.mod h1:L7UH0GbB0p47T4Rri3uHjbpCFYrVrwc1I25QhNPiGK8= +go.opentelemetry.io/otel v1.31.0 h1:NsJcKPIW0D0H3NgzPDHmo0WW6SptzPdqg/L1zsIm2hY= +go.opentelemetry.io/otel v1.31.0/go.mod h1:O0C14Yl9FgkjqcCZAsE053C13OaddMYr/hz6clDkEJE= +go.opentelemetry.io/otel/metric v1.31.0 h1:FSErL0ATQAmYHUIzSezZibnyVlft1ybhy4ozRPcF2fE= +go.opentelemetry.io/otel/metric v1.31.0/go.mod h1:C3dEloVbLuYoX41KpmAhOqNriGbA+qqH6PQ5E5mUfnY= +go.opentelemetry.io/otel/sdk v1.31.0 h1:xLY3abVHYZ5HSfOg3l2E5LUj2Cwva5Y7yGxnSW9H5Gk= +go.opentelemetry.io/otel/sdk v1.31.0/go.mod h1:TfRbMdhvxIIr/B2N2LQW2S5v9m3gOQ/08KsbbO5BPT0= +go.opentelemetry.io/otel/sdk/metric v1.31.0 h1:i9hxxLJF/9kkvfHppyLL55aW7iIJz4JjxTeYusH7zMc= +go.opentelemetry.io/otel/sdk/metric v1.31.0/go.mod h1:CRInTMVvNhUKgSAMbKyTMxqOBC0zgyxzW55lZzX43Y8= +go.opentelemetry.io/otel/trace v1.31.0 h1:ffjsj1aRouKewfr85U2aGagJ46+MvodynlQ1HYdmJys= +go.opentelemetry.io/otel/trace v1.31.0/go.mod h1:TXZkRk7SM2ZQLtR6eoAWQFIHPvzQ06FJAsO1tJg480A= +go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw= +go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc= +golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U= +golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= +golang.org/x/mod v0.21.0 h1:vvrHzRwRfVKSiLrG+d4FMl/Qi4ukBCE6kZlTUkDYRT0= +golang.org/x/mod v0.21.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY= +golang.org/x/net v0.33.0 h1:74SYHlV8BIgHIFC/LrYkOGIwL19eTYXQ5wc6TBuO36I= +golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4= +golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA= +golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo= +golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ= +golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= +golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/internal/storagefixture/server.go b/internal/storagefixture/server.go new file mode 100644 index 0000000..5f561ee --- /dev/null +++ b/internal/storagefixture/server.go @@ -0,0 +1,63 @@ +// Package storagefixture supplies a bounded protocol fixture, not a cloud service. +package storagefixture + +import ( + "crypto/md5" + "fmt" + "io" + "net/http" + "sync" + "time" +) + +type value struct { + data []byte + contentType string + etag string +} + +func Handler() http.Handler { + var mu sync.Mutex + values := map[string]value{} + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + defer mu.Unlock() + if r.Header.Get("Authorization") == "" && r.URL.Query().Get("X-Amz-Signature") == "" && r.URL.Query().Get("x-oss-signature") == "" { + http.Error(w, "signature required", 403) + return + } + key := r.URL.Path + switch r.Method { + case "PUT": + data, err := io.ReadAll(io.LimitReader(r.Body, 1025)) + if err != nil || len(data) > 1024 { + http.Error(w, "too large", 413) + return + } + v := value{data: data, contentType: r.Header.Get("Content-Type"), etag: fmt.Sprintf("%x", md5.Sum(data))} + values[key] = v + w.Header().Set("ETag", `"`+v.etag+`"`) + w.WriteHeader(200) + case "DELETE": + delete(values, key) + w.WriteHeader(204) + case "GET", "HEAD": + v, ok := values[key] + if !ok { + w.Header().Set("Content-Type", "application/xml") + w.WriteHeader(404) + fmt.Fprint(w, `NoSuchKeynot found`) + return + } + w.Header().Set("Content-Length", fmt.Sprint(len(v.data))) + w.Header().Set("Content-Type", v.contentType) + w.Header().Set("ETag", `"`+v.etag+`"`) + w.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat)) + if r.Method == "GET" { + w.Write(v.data) + } + default: + http.Error(w, "unsupported", 405) + } + }) +} diff --git a/logx/logx.go b/logx/logx.go index 1ed93fe..b2870f0 100644 --- a/logx/logx.go +++ b/logx/logx.go @@ -21,14 +21,14 @@ import ( // standard stream. File takes precedence over Output and appends to the named // file. type Config struct { - Level string - Format string - Output string - File string - Service string - Environment string - AddSource bool - Writer io.Writer + Level string `json:"level"` + Format string `json:"format"` + Output string `json:"output"` + File string `json:"file"` + Service string `json:"service"` + Environment string `json:"environment"` + AddSource bool `json:"addSource"` + Writer io.Writer `json:"-"` } // Default returns the default production-friendly logger configuration. @@ -45,6 +45,7 @@ func Default() Config { type Logger struct { logger *slog.Logger closer *closeState + level *slog.LevelVar } type closeState struct { @@ -78,7 +79,9 @@ func New(cfg Config) (*Logger, error) { return nil, err } - opts := &slog.HandlerOptions{Level: level, AddSource: cfg.AddSource} + dynamicLevel := new(slog.LevelVar) + dynamicLevel.Set(level) + opts := &slog.HandlerOptions{Level: dynamicLevel, AddSource: cfg.AddSource} var handler slog.Handler if cfg.Format == "text" { handler = slog.NewTextHandler(writer, opts) @@ -97,7 +100,7 @@ func New(cfg Config) (*Logger, error) { if closeWriter != nil { closer = &closeState{fn: closeWriter} } - return &Logger{logger: logger, closer: closer}, nil + return &Logger{logger: logger, closer: closer, level: dynamicLevel}, nil } // Setup replaces the process-wide logger used by the package-level helpers. @@ -152,14 +155,27 @@ func (l *Logger) With(args ...any) *Logger { if l == nil || l.logger == nil { return defaultLogger.Load().With(args...) } - return &Logger{logger: l.logger.With(args...), closer: l.closer} + return &Logger{logger: l.logger.With(args...), closer: l.closer, level: l.level} } func (l *Logger) WithGroup(name string) *Logger { if l == nil || l.logger == nil { return defaultLogger.Load().WithGroup(name) } - return &Logger{logger: l.logger.WithGroup(name), closer: l.closer} + return &Logger{logger: l.logger.WithGroup(name), closer: l.closer, level: l.level} +} + +// SetLevel atomically changes the threshold shared by this logger and children. +func (l *Logger) SetLevel(value string) error { + if l == nil || l.level == nil { + return errors.New("logx: logger does not support dynamic levels") + } + level, err := parseLevel(value) + if err != nil { + return err + } + l.level.Set(level) + return nil } // Close releases resources owned by the root logger. It is safe to call more diff --git a/middleware/idempotency.go b/middleware/idempotency.go index 95965cc..d499a09 100644 --- a/middleware/idempotency.go +++ b/middleware/idempotency.go @@ -17,12 +17,13 @@ import ( ) type IdempotencyConfig struct { - TTL time.Duration `json:"ttl"` - MaxEntries int `json:"maxEntries"` - MaxBytes int `json:"maxBytes"` - MaxRequestBytes int64 `json:"maxRequestBytes"` - MaxResponseBytes int `json:"maxResponseBytes"` - MaxInFlight int `json:"maxInFlight"` + Store IdempotencyStore `json:"-"` // Optional shared store; nil keeps local behavior. + TTL time.Duration `json:"ttl"` + MaxEntries int `json:"maxEntries"` + MaxBytes int `json:"maxBytes"` + MaxRequestBytes int64 `json:"maxRequestBytes"` + MaxResponseBytes int `json:"maxResponseBytes"` + MaxInFlight int `json:"maxInFlight"` // Scope must return a server-verified tenant/user identity. Register auth // before this middleware. Client-provided identity is not authorization. Scope func(*httpx.Context) (string, error) `json:"-"` @@ -53,6 +54,9 @@ func NewIdempotency(cfg IdempotencyConfig) (*Idempotency, error) { if cfg.TTL == 0 { cfg.TTL = 10 * time.Minute } + if cfg.Store != nil && cfg.TTL > 30*24*time.Hour { + return nil, errors.New("middleware: shared idempotency TTL exceeds 30d") + } if cfg.MaxEntries == 0 { cfg.MaxEntries = 256 } @@ -125,6 +129,9 @@ func (i *Idempotency) Middleware() httpx.Middleware { copy(fingerprint[:], hash.Sum(nil)) namespace, _ := json.Marshal([]string{scope, key}) id := sha256.Sum256(namespace) + if i.config.Store != nil { + return i.shared(c, next, id, fingerprint) + } i.mu.Lock() now := i.now() for k, v := range i.items { diff --git a/middleware/rateLimit.go b/middleware/rateLimit.go index 1ebe1c1..7bfd5e2 100644 --- a/middleware/rateLimit.go +++ b/middleware/rateLimit.go @@ -13,6 +13,7 @@ import ( ) type RateLimitConfig struct { + Backend RateLimitBackend `json:"-"` Rate float64 `json:"rate"` // Tokens per second, per key. Burst int `json:"burst"` MaxKeys int `json:"maxKeys"` @@ -64,7 +65,17 @@ func (l *RateLimiter) Middleware() httpx.Middleware { if key == "" || len(key) > 1024 { return apierrors.BadRequest("INVALID_RATE_KEY", "invalid rate limit key") } - delay, full := l.allow(key) + var delay time.Duration + var full bool + if l.config.Backend != nil { + var err error + delay, err = l.config.Backend.Take(c.Context(), key) + if err != nil { + return apierrors.New(503, "RATE_LIMIT_UNAVAILABLE", "rate limiter unavailable").WithCause(err) + } + } else { + delay, full = l.allow(key) + } if full { return apierrors.New(503, "RATE_LIMIT_CAPACITY", "rate limiter capacity reached") } diff --git a/middleware/redis/store.go b/middleware/redis/store.go new file mode 100644 index 0000000..c67751b --- /dev/null +++ b/middleware/redis/store.go @@ -0,0 +1,178 @@ +// Package redis supplies atomic shared middleware state in one Redis hash slot. +// Use a dedicated namespace and a noeviction Redis deployment for correctness. +package redis + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/middleware" + goredis "github.com/redis/go-redis/v9" + "strings" + "time" +) + +type IdempotencyConfig struct { + MaxEntries int `json:"maxEntries"` + MaxRecordBytes int `json:"maxRecordBytes"` +} +type IdempotencyStore struct { + client goredis.Scripter + keys []string + cfg IdempotencyConfig + signature string +} + +func namespace(prefix, kind string) (string, error) { + if prefix == "" || len(prefix) > 256 || strings.ContainsAny(prefix, "{}\r\n") { + return "", errors.New("middleware/redis: nonempty bounded namespace without braces required") + } + sum := sha256.Sum256([]byte(prefix)) + return "goOmni:{" + hex.EncodeToString(sum[:]) + "}:" + kind, nil +} +func NewIdempotencyStore(client goredis.Scripter, prefix string, cfg IdempotencyConfig) (*IdempotencyStore, error) { + if client == nil || cfg.MaxEntries < 0 || cfg.MaxRecordBytes < 0 { + return nil, errors.New("middleware/redis: invalid configuration") + } + if cfg.MaxEntries == 0 { + cfg.MaxEntries = 128 + } + if cfg.MaxRecordBytes == 0 { + cfg.MaxRecordBytes = 128 << 10 + } + if cfg.MaxEntries > 100000 || cfg.MaxRecordBytes < 512 || cfg.MaxRecordBytes > 1<<20 { + return nil, errors.New("middleware/redis: capacity outside supported bounds") + } + base, err := namespace(prefix, "idempotency") + if err != nil { + return nil, err + } + return &IdempotencyStore{client: client, keys: []string{base + ":records", base + ":expiry", base + ":config"}, cfg: cfg, signature: fmt.Sprintf("%d:%d", cfg.MaxEntries, cfg.MaxRecordBytes)}, nil +} + +var claim = goredis.NewScript(` +local config=redis.call('GET',KEYS[3]) +if config and config~=ARGV[5] then return redis.error_reply('configuration mismatch') end +if not config then redis.call('SET',KEYS[3],ARGV[5]) end +local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000) +local expired=redis.call('ZRANGEBYSCORE',KEYS[2],'-inf',now,'LIMIT',0,128) +for _,key in ipairs(expired) do redis.call('HDEL',KEYS[1],key);redis.call('ZREM',KEYS[2],key) end +local expiry=redis.call('ZSCORE',KEYS[2],ARGV[1]);if expiry and tonumber(expiry)<=now then redis.call('HDEL',KEYS[1],ARGV[1]);redis.call('ZREM',KEYS[2],ARGV[1]) end +local value=redis.call('HGET',KEYS[1],ARGV[1]);if value then return {0,value} end +if redis.call('HLEN',KEYS[1])>=tonumber(ARGV[4]) then return redis.error_reply('idempotency capacity reached') end +value=cjson.encode({fingerprint=ARGV[2],owner=ARGV[3],state='running'}) +redis.call('HSET',KEYS[1],ARGV[1],value);return {1,value} +`) + +func (s *IdempotencyStore) Claim(ctx context.Context, key, fp, owner string) (middleware.IdempotencyRecord, bool, error) { + if len(key) != 64 || len(fp) != 64 || len(owner) != 32 { + return middleware.IdempotencyRecord{}, false, errors.New("middleware/redis: invalid claim") + } + result, err := claim.Run(ctx, s.client, s.keys, key, fp, owner, s.cfg.MaxEntries, s.signature).Slice() + if err != nil { + return middleware.IdempotencyRecord{}, false, err + } + if len(result) != 2 { + return middleware.IdempotencyRecord{}, false, errors.New("middleware/redis: invalid claim result") + } + value, ok := result[1].(string) + if !ok || len(value) > s.cfg.MaxRecordBytes { + return middleware.IdempotencyRecord{}, false, errors.New("middleware/redis: invalid record") + } + var record middleware.IdempotencyRecord + err = json.Unmarshal([]byte(value), &record) + return record, result[0] == int64(1), err +} + +var complete = goredis.NewScript(` +local current=redis.call('HGET',KEYS[1],ARGV[1]);if not current then return redis.error_reply('claim missing') end +local value=cjson.decode(current) +if value.owner~=ARGV[2] or value.state~='running' then return redis.error_reply('claim owner/state changed') end +local response=ARGV[3] +if response~='' then value.state='completed';value.response=cjson.decode(response) else value.state='failed' end +local encoded=cjson.encode(value) +if string.len(encoded)>tonumber(ARGV[5]) then value.state='failed';value.response=nil;encoded=cjson.encode(value) end +local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000) +redis.call('HSET',KEYS[1],ARGV[1],encoded);redis.call('ZADD',KEYS[2],now+tonumber(ARGV[4]),ARGV[1]);return 1 +`) + +func (s *IdempotencyStore) Complete(ctx context.Context, key, owner string, response *middleware.StoredResponse, ttl time.Duration) error { + if ttl <= 0 || ttl > 30*24*time.Hour { + return errors.New("middleware/redis: TTL must be positive and <=30d") + } + var encoded []byte + var err error + if response != nil { + encoded, err = json.Marshal(response) + if err != nil { + return err + } + if len(encoded) > s.cfg.MaxRecordBytes { + encoded = nil + } + } + return complete.Run(ctx, s.client, s.keys, key, owner, string(encoded), max(1, ttl.Milliseconds()), s.cfg.MaxRecordBytes).Err() +} + +// Resolve removes only a matching non-replayable claim after the operator has +// reconciled the business outcome. Never invoke automatically on a timeout. +func (s *IdempotencyStore) Resolve(ctx context.Context, key, owner string) error { + return goredis.NewScript(`local v=redis.call('HGET',KEYS[1],ARGV[1]);if not v then return 0 end;local r=cjson.decode(v);if r.owner~=ARGV[2] or r.state=='completed' then return redis.error_reply('claim cannot be resolved') end;redis.call('HDEL',KEYS[1],ARGV[1]);redis.call('ZREM',KEYS[2],ARGV[1]);return 1`).Run(ctx, s.client, s.keys, key, owner).Err() +} + +type limiter struct { + client goredis.Scripter + keys []string + cfg middleware.RateLimitConfig + signature string +} + +func NewRateLimiter(client goredis.Scripter, prefix string, cfg middleware.RateLimitConfig) (*middleware.RateLimiter, error) { + if client == nil { + return nil, errors.New("middleware/redis: client required") + } + if cfg.MaxKeys == 0 { + cfg.MaxKeys = 10000 + } + if cfg.IdleTTL == 0 { + cfg.IdleTTL = 24 * time.Hour + } + if _, err := middleware.NewRateLimiter(cfg); err != nil { + return nil, err + } + base, err := namespace(prefix, "rate") + if err != nil { + return nil, err + } + backend := &limiter{client: client, keys: []string{base + ":buckets", base + ":seen", base + ":config"}, cfg: cfg, signature: fmt.Sprintf("%g:%d:%d:%d", cfg.Rate, cfg.Burst, cfg.MaxKeys, cfg.IdleTTL.Milliseconds())} + cfg.Backend = backend + return middleware.NewRateLimiter(cfg) +} + +var take = goredis.NewScript(` +local config=redis.call('GET',KEYS[3]);if config and config~=ARGV[6] then return redis.error_reply('configuration mismatch') end +if not config then redis.call('SET',KEYS[3],ARGV[6]) end +local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000) +local expired=redis.call('ZRANGEBYSCORE',KEYS[2],'-inf',now-tonumber(ARGV[5]),'LIMIT',0,128) +for _,key in ipairs(expired) do redis.call('HDEL',KEYS[1],key);redis.call('ZREM',KEYS[2],key) end +local value=redis.call('HGET',KEYS[1],ARGV[1]);local tokens=tonumber(ARGV[3]) +if value then local b=cjson.decode(value);tokens=math.min(tokens,b.tokens+math.max(0,now-b.time)*tonumber(ARGV[2])/1000) +elseif redis.call('HLEN',KEYS[1])>=tonumber(ARGV[4]) then return -1 end +local delay=0;if tokens>=1 then tokens=tokens-1 else delay=math.ceil((1-tokens)*1000/tonumber(ARGV[2])) end +redis.call('HSET',KEYS[1],ARGV[1],cjson.encode({tokens=tokens,time=now}));redis.call('ZADD',KEYS[2],now,ARGV[1]);return delay +`) + +func (l *limiter) Take(ctx context.Context, key string) (time.Duration, error) { + sum := sha256.Sum256([]byte(key)) + delay, err := take.Run(ctx, l.client, l.keys, hex.EncodeToString(sum[:]), l.cfg.Rate, l.cfg.Burst, l.cfg.MaxKeys, l.cfg.IdleTTL.Milliseconds(), l.signature).Int64() + if err != nil { + return 0, err + } + if delay < 0 { + return 0, middleware.ErrLimitCapacity + } + return time.Duration(delay) * time.Millisecond, nil +} diff --git a/middleware/redis/store_test.go b/middleware/redis/store_test.go new file mode 100644 index 0000000..5e9fb12 --- /dev/null +++ b/middleware/redis/store_test.go @@ -0,0 +1,145 @@ +package redis + +import ( + "context" + "fmt" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/middleware" + goredis "github.com/redis/go-redis/v9" + "net/http" + "net/http/httptest" + "os" + "strings" + "sync" + "sync/atomic" + "testing" + "time" +) + +func TestSharedRedisIntegration(t *testing.T) { + address := os.Getenv("GO_OMNI_REDIS_TEST_ADDR") + if address == "" { + t.Skip("requires isolated Redis") + } + client := goredis.NewClient(&goredis.Options{Addr: address, PoolSize: 8, MaxRetries: -1, ContextTimeoutEnabled: true, ReadTimeout: time.Second, WriteTimeout: time.Second}) + defer client.Close() + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second) + defer cancel() + prefix := fmt.Sprint("shared:", time.Now().UnixNano()) + s, err := NewIdempotencyStore(client, prefix, IdempotencyConfig{MaxEntries: 256}) + if err != nil { + t.Fatal(err) + } + defer client.Del(context.Background(), s.keys...) + key, fp := strings.Repeat("a", 64), strings.Repeat("b", 64) + var claims atomic.Int32 + var wg sync.WaitGroup + owners := make(chan string, 32) + for i := 0; i < 32; i++ { + wg.Add(1) + go func(i int) { + defer wg.Done() + owner := fmt.Sprintf("%032x", i) + _, ok, err := s.Claim(ctx, key, fp, owner) + if err != nil { + t.Error(err) + } + if ok { + claims.Add(1) + owners <- owner + } + }(i) + } + wg.Wait() + if claims.Load() != 1 { + t.Fatal(claims.Load()) + } + owner := <-owners + if err = s.Complete(ctx, key, "wrong", nil, time.Minute); err == nil { + t.Fatal("foreign completion accepted") + } + if err = s.Complete(ctx, key, owner, &middleware.StoredResponse{Status: 201, Headers: http.Header{}, Body: []byte{0, 1, 255}}, time.Minute); err != nil { + t.Fatal(err) + } + record, ok, err := s.Claim(ctx, key, fp, strings.Repeat("c", 32)) + if err != nil || ok || record.State != "completed" || len(record.Response.Body) != 3 { + t.Fatal(record, ok, err) + } + // Two independently constructed apps use the same backend, so only one runs. + var executions atomic.Int32 + makeApp := func() *omni.App { + backend, _ := NewIdempotencyStore(client, prefix, IdempotencyConfig{MaxEntries: 256}) + idem, err := middleware.NewIdempotency(middleware.IdempotencyConfig{Store: backend, Scope: func(*httpx.Context) (string, error) { return "verifiedTenant", nil }}) + if err != nil { + t.Fatal(err) + } + a := omni.New(omni.WithLogger(false)) + a.POST("/submit", func(c *httpx.Context) error { executions.Add(1); return c.OK("saved") }, idem.Middleware()) + return a + } + a, b := makeApp(), makeApp() + request := func(a *omni.App, body string) *httptest.ResponseRecorder { + r := httptest.NewRequest("POST", "/submit", strings.NewReader(body)) + r.Header.Set("Idempotency-Key", "stableKey") + w := httptest.NewRecorder() + a.ServeHTTP(w, r) + return w + } + first, second := request(a, `{}`), request(b, `{}`) + if first.Code != 200 || second.Code != 200 || first.Body.String() != second.Body.String() || second.Header().Get("Idempotency-Replayed") != "true" || executions.Load() != 1 { + t.Fatal(first.Code, second.Code, second.Body.String(), executions.Load()) + } + if request(b, `{"changed":true}`).Code != 409 { + t.Fatal("conflict accepted") + } + // Shared limits do not multiply when a second replica is added. + limitPrefix := prefix + "limit" + cfg := middleware.RateLimitConfig{Rate: .01, Burst: 2, Key: func(*httpx.Context) string { return "tenant" }} + r1, _ := NewRateLimiter(client, limitPrefix, cfg) + r2, _ := NewRateLimiter(client, limitPrefix, cfg) + base, _ := namespace(limitPrefix, "rate") + defer client.Del(context.Background(), base+":buckets", base+":seen", base+":config") + limited := func(l *middleware.RateLimiter) int { + a := omni.New(omni.WithLogger(false)) + a.GET("/", func(c *httpx.Context) error { return c.OK("ok") }, l.Middleware()) + w := httptest.NewRecorder() + a.ServeHTTP(w, httptest.NewRequest("GET", "/", nil)) + return w.Code + } + if limited(r1) != 200 || limited(r2) != 200 || limited(r1) != 429 { + t.Fatal("limits multiplied across instances") + } + // Redis failure must fail closed before executing a new request. + client.Close() + if status := limited(r2); status != 503 { + t.Fatal(status) + } +} + +func TestExpiredTargetBeyondCleanupBatch(t *testing.T) { + address := os.Getenv("GO_OMNI_REDIS_TEST_ADDR") + if address == "" { + t.Skip("requires isolated Redis") + } + client := goredis.NewClient(&goredis.Options{Addr: address}) + defer client.Close() + s, _ := NewIdempotencyStore(client, fmt.Sprint("expiry:", time.Now().UnixNano()), IdempotencyConfig{MaxEntries: 256}) + defer client.Del(context.Background(), s.keys...) + ctx := context.Background() + owner := strings.Repeat("1", 32) + fp := strings.Repeat("2", 64) + for i := 0; i < 140; i++ { + key := fmt.Sprintf("%064x", i) + if _, ok, err := s.Claim(ctx, key, fp, owner); err != nil || !ok { + t.Fatal(err) + } + if err := s.Complete(ctx, key, owner, nil, time.Minute); err != nil { + t.Fatal(err) + } + client.ZAdd(ctx, s.keys[1], goredis.Z{Score: 0, Member: key}) + } + if _, ok, err := s.Claim(ctx, fmt.Sprintf("%064x", 139), fp, owner); err != nil || !ok { + t.Fatal("expired target retained beyond TTL", ok, err) + } +} diff --git a/middleware/sharedIdempotency.go b/middleware/sharedIdempotency.go new file mode 100644 index 0000000..77483e5 --- /dev/null +++ b/middleware/sharedIdempotency.go @@ -0,0 +1,102 @@ +package middleware + +import ( + "context" + "crypto/rand" + "encoding/hex" + "errors" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" + "net/http" + "strings" + "time" +) + +type StoredResponse struct { + Status int `json:"status"` + Headers http.Header `json:"headers"` + Body []byte `json:"body"` +} +type IdempotencyRecord struct { + Fingerprint string `json:"fingerprint"` + Owner string `json:"owner"` + State string `json:"state"` + Response *StoredResponse `json:"response,omitempty"` +} + +// IdempotencyStore must atomically claim absent keys, compare owners when +// finishing and retain in-flight claims until explicit resolution. Only completed +// or failed records expire; an uncertain outcome must never become a new claim. +type IdempotencyStore interface { + Claim(context.Context, string, string, string) (IdempotencyRecord, bool, error) + Complete(context.Context, string, string, *StoredResponse, time.Duration) error +} + +func (i *Idempotency) shared(c *httpx.Context, next httpx.Handler, id, fingerprint [32]byte) error { + key, fp := hex.EncodeToString(id[:]), hex.EncodeToString(fingerprint[:]) + token := make([]byte, 16) + if _, err := rand.Read(token); err != nil { + return err + } + owner := hex.EncodeToString(token) + record, claimed, err := i.config.Store.Claim(c.Context(), key, fp, owner) + if err != nil { + return apierrors.New(503, "IDEMPOTENCY_UNAVAILABLE", "idempotency store unavailable").WithCause(err) + } + if !claimed { + if record.Fingerprint != fp { + return apierrors.New(409, "IDEMPOTENCY_CONFLICT", "key was used with a different request") + } + if record.State != "completed" || record.Response == nil { + return apierrors.New(409, "IDEMPOTENCY_UNAVAILABLE", "request is in progress or outcome requires reconciliation") + } + response := record.Response + headers := response.Headers.Clone() + headers.Set("Idempotency-Replayed", "true") + return c.WriteResponse(response.Status, headers, response.Body) + } + completed := false + finish := func(response *StoredResponse) error { + ctx, cancel := context.WithTimeout(context.WithoutCancel(c.Context()), time.Second) + defer cancel() + return i.config.Store.Complete(ctx, key, owner, response, i.config.TTL) + } + defer func() { + if !completed { + if err := finish(nil); err != nil { + logx.ErrorContext(c.Context(), "idempotency.persistFailed", "error", err) + } + } + }() + buffer := &boundedResponse{headers: c.Raw().Header().Clone(), limit: i.config.MaxResponseBytes} + child := c.Clone(c.Context(), buffer) + err = next(child) + if ctxErr := child.Context().Err(); ctxErr != nil { + return ctxErr + } + if err != nil { + child.WriteError(err) + } + if buffer.err != nil { + return apierrors.Internal(buffer.err) + } + if buffer.status == 0 { + buffer.status = 200 + } + var response *StoredResponse + if buffer.status < 500 && buffer.headers.Get("Set-Cookie") == "" && !strings.Contains(strings.ToLower(buffer.headers.Get("Cache-Control")), "no-store") { + response = &StoredResponse{Status: buffer.status, Headers: buffer.headers.Clone(), Body: append([]byte(nil), buffer.body.Bytes()...)} + } + if err = finish(response); err != nil { + return apierrors.New(503, "IDEMPOTENCY_UNAVAILABLE", "outcome requires reconciliation").WithCause(err) + } + completed = true + return c.WriteResponse(buffer.status, buffer.headers, buffer.body.Bytes()) +} + +var ErrLimitCapacity = errors.New("middleware: shared rate limiter capacity") + +type RateLimitBackend interface { + Take(context.Context, string) (time.Duration, error) +} diff --git a/openapi/openapi.go b/openapi/openapi.go new file mode 100644 index 0000000..cf84f45 --- /dev/null +++ b/openapi/openapi.go @@ -0,0 +1,267 @@ +// Package openapi registers routes and their OpenAPI 3.0.3 contracts together. +package openapi + +import ( + "encoding/json" + "errors" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "net/http" + "reflect" + "regexp" + "strings" + "sync" + "time" +) + +type Schema struct { + Type string `json:"type,omitempty"` + Format string `json:"format,omitempty"` + Description string `json:"description,omitempty"` + Properties map[string]*Schema `json:"properties,omitempty"` + Required []string `json:"required,omitempty"` + Items *Schema `json:"items,omitempty"` + AdditionalProperties *Schema `json:"additionalProperties,omitempty"` + Nullable bool `json:"nullable,omitempty"` +} +type Parameter struct { + Name string `json:"name"` + In string `json:"in"` + Required bool `json:"required"` + Description string `json:"description,omitempty"` + Schema *Schema `json:"schema"` +} +type Response struct { + Description string + Schema *Schema +} +type Operation struct { + Method, Path, OperationID, Summary string + Parameters []Parameter + Request *Schema + Responses map[int]Response + BearerAuth bool +} +type Registrar interface { + Handle(string, string, httpx.Handler, ...httpx.Middleware) +} +type API struct { + mu sync.RWMutex + document map[string]any + paths map[string]map[string]any + ids map[string]bool +} + +var identifier = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_]*$`) + +func New(title, version string) (*API, error) { + if title == "" || version == "" { + return nil, errors.New("openapi: title and version required") + } + paths := make(map[string]map[string]any) + return &API{paths: paths, ids: make(map[string]bool), document: map[string]any{"openapi": "3.0.3", "info": map[string]string{"title": title, "version": version}, "paths": paths, "components": map[string]any{"securitySchemes": map[string]any{"bearerAuth": map[string]string{"type": "http", "scheme": "bearer"}}}}}, nil +} + +// Handle registers an absolute application route. To document a group, include +// its full prefix here and pass middleware explicitly. Schemas describe wire +// responses; Envelope wraps a data schema in GoOmni's standard JSON envelope. +func (a *API) Handle(router Registrar, op Operation, handler httpx.Handler, middleware ...httpx.Middleware) error { + if router == nil || handler == nil || !identifier.MatchString(op.OperationID) || !strings.HasPrefix(op.Path, "/") || strings.ContainsAny(op.Path, "*{}?# ") { + return errors.New("openapi: invalid route") + } + method := strings.ToLower(op.Method) + switch method { + case "get", "post", "put", "patch", "delete", "head", "options": + default: + return errors.New("openapi: unsupported HTTP method") + } + parts := strings.Split(op.Path, "/") + params := make(map[string]bool) + for n, part := range parts { + if strings.HasPrefix(part, ":") { + name := part[1:] + if !identifier.MatchString(name) || params[name] { + return errors.New("openapi: invalid path parameter") + } + parts[n] = "{" + name + "}" + params[name] = true + } + } + path := strings.Join(parts, "/") + seen := make(map[string]bool) + for _, p := range op.Parameters { + if p.Name == "" || p.Schema == nil || seen[p.In+":"+p.Name] { + return errors.New("openapi: invalid or duplicate parameter") + } + seen[p.In+":"+p.Name] = true + switch p.In { + case "path": + if !params[p.Name] || !p.Required { + return errors.New("openapi: path parameter must match and be required") + } + delete(params, p.Name) + case "query", "header", "cookie": + default: + return errors.New("openapi: invalid parameter location") + } + } + if len(params) != 0 || len(op.Responses) == 0 { + return errors.New("openapi: declare all path parameters and at least one response") + } + responses := map[string]any{} + for code, response := range op.Responses { + if code < 100 || code > 599 || response.Description == "" { + return errors.New("openapi: invalid response") + } + item := map[string]any{"description": response.Description} + if response.Schema != nil { + item["content"] = map[string]any{"application/json": map[string]any{"schema": response.Schema}} + } + responses[fmt.Sprint(code)] = item + } + operation := map[string]any{"operationId": op.OperationID, "summary": op.Summary, "responses": responses} + if len(op.Parameters) > 0 { + operation["parameters"] = op.Parameters + } + if op.Request != nil { + operation["requestBody"] = map[string]any{"required": true, "content": map[string]any{"application/json": map[string]any{"schema": op.Request}}} + } + if op.BearerAuth { + operation["security"] = []map[string][]string{{"bearerAuth": {}}} + } + // Detach caller-owned maps/slices before concurrent document reads. + encoded, err := json.Marshal(operation) + if err != nil { + return err + } + var frozen any + if err = json.Unmarshal(encoded, &frozen); err != nil { + return err + } + a.mu.Lock() + defer a.mu.Unlock() + if a.ids[op.OperationID] || a.paths[path][method] != nil { + return errors.New("openapi: duplicate operation ID or route") + } + router.Handle(strings.ToUpper(method), op.Path, handler, middleware...) + if a.paths[path] == nil { + a.paths[path] = make(map[string]any) + } + a.paths[path][method] = frozen + a.ids[op.OperationID] = true + return nil +} +func (a *API) JSON() ([]byte, error) { + a.mu.RLock() + defer a.mu.RUnlock() + return json.MarshalIndent(a.document, "", " ") +} +func (a *API) Handler() httpx.Handler { + return func(c *httpx.Context) error { + data, err := a.JSON() + if err != nil { + return err + } + return c.WriteResponse(200, http.Header{"Content-Type": {"application/json; charset=utf-8"}}, data) + } +} +func Envelope(data *Schema) *Schema { + return &Schema{Type: "object", Required: []string{"code", "message"}, Properties: map[string]*Schema{"code": {Type: "string"}, "message": {Type: "string"}, "requestId": {Type: "string"}, "data": data}} +} +func SchemaOf[T any]() (*Schema, error) { + return schema(reflect.TypeOf((*T)(nil)).Elem(), map[reflect.Type]bool{}, 0) +} +func schema(t reflect.Type, stack map[reflect.Type]bool, depth int) (*Schema, error) { + if depth > 32 || stack[t] { + return nil, errors.New("openapi: recursive DTO requires an explicit schema") + } + if t.Kind() == reflect.Pointer { + s, err := schema(t.Elem(), stack, depth+1) + if err == nil { + s.Nullable = true + } + return s, err + } + if t == reflect.TypeOf(time.Time{}) { + return &Schema{Type: "string", Format: "date-time"}, nil + } + if t == reflect.TypeOf(json.RawMessage{}) { + return &Schema{}, nil + } + if t == reflect.TypeOf(json.Number("")) { + return &Schema{Type: "number"}, nil + } + marshaler := reflect.TypeOf((*json.Marshaler)(nil)).Elem() + if t.Implements(marshaler) || reflect.PointerTo(t).Implements(marshaler) { + return nil, errors.New("openapi: custom JSON marshaler requires an explicit schema") + } + switch t.Kind() { + case reflect.String: + return &Schema{Type: "string"}, nil + case reflect.Bool: + return &Schema{Type: "boolean"}, nil + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + return &Schema{Type: "integer"}, nil + case reflect.Float32, reflect.Float64: + return &Schema{Type: "number"}, nil + case reflect.Slice, reflect.Array: + if t.Elem().Kind() == reflect.Uint8 && t.Kind() == reflect.Slice { + return &Schema{Type: "string", Format: "byte"}, nil + } + s, err := schema(t.Elem(), stack, depth+1) + return &Schema{Type: "array", Items: s}, err + case reflect.Map: + if t.Key().Kind() != reflect.String { + return nil, errors.New("openapi: map keys must be strings") + } + s, err := schema(t.Elem(), stack, depth+1) + return &Schema{Type: "object", AdditionalProperties: s}, err + case reflect.Interface: + return &Schema{}, nil + case reflect.Struct: + stack[t] = true + defer delete(stack, t) + s := &Schema{Type: "object", Properties: map[string]*Schema{}} + for n := 0; n < t.NumField(); n++ { + f := t.Field(n) + if f.PkgPath != "" { + continue + } + tag := strings.Split(f.Tag.Get("json"), ",") + if tag[0] == "-" { + continue + } + if f.Anonymous && tag[0] == "" { + return nil, errors.New("openapi: embedded fields require an explicit JSON name") + } + name := tag[0] + if name == "" { + name = f.Name + } + child, err := schema(f.Type, stack, depth+1) + if err != nil { + return nil, err + } + if s.Properties[name] != nil { + return nil, errors.New("openapi: duplicate JSON field") + } + s.Properties[name] = child + optional := false + for _, option := range tag[1:] { + if option == "string" { + child.Type = "string" + child.Format = "" + } + if option == "omitempty" || option == "omitzero" { + optional = true + } + } + if !optional { + s.Required = append(s.Required, name) + } + } + return s, nil + default: + return nil, fmt.Errorf("openapi: unsupported DTO type %s", t) + } +} diff --git a/openapi/openapi_test.go b/openapi/openapi_test.go new file mode 100644 index 0000000..f9d706a --- /dev/null +++ b/openapi/openapi_test.go @@ -0,0 +1,54 @@ +package openapi_test + +import ( + "encoding/json" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/openapi" + "net/http/httptest" + "testing" +) + +func TestRouteAndContract(t *testing.T) { + type item struct { + ID int64 `json:"id,string"` + RequestID string `json:"requestId,omitempty"` + Payload json.RawMessage `json:"payload"` + } + schema, err := openapi.SchemaOf[item]() + if err != nil || schema.Properties["id"].Type != "string" || schema.Properties["payload"].Type != "" { + t.Fatal(schema, err) + } + api, _ := openapi.New("Test API", "1") + app := omni.New(omni.WithLogger(false)) + op := openapi.Operation{Method: "GET", Path: "/items/:id", OperationID: "getItem", Parameters: []openapi.Parameter{{Name: "id", In: "path", Required: true, Schema: &openapi.Schema{Type: "string"}}}, Responses: map[int]openapi.Response{200: {Description: "OK", Schema: openapi.Envelope(schema)}}} + if err = api.Handle(app, op, func(c *httpx.Context) error { return c.OK(item{ID: 1, Payload: json.RawMessage(`{}`)}) }); err != nil { + t.Fatal(err) + } + // Caller edits must not mutate the published contract. + schema.Properties["id"].Type = "boolean" + if err = api.Handle(app, op, func(*httpx.Context) error { return nil }); err == nil { + t.Fatal("duplicate accepted") + } + app.GET("/openapi.json", api.Handler()) + r := httptest.NewRecorder() + app.ServeHTTP(r, httptest.NewRequest("GET", "/openapi.json", nil)) + var doc struct { + OpenAPI string `json:"openapi"` + Paths map[string]json.RawMessage `json:"paths"` + } + if err = json.Unmarshal(r.Body.Bytes(), &doc); err != nil || doc.OpenAPI != "3.0.3" || doc.Paths["/items/{id}"] == nil { + t.Fatal(r.Body.String(), err) + } + r = httptest.NewRecorder() + app.ServeHTTP(r, httptest.NewRequest("GET", "/items/1", nil)) + if r.Code != 200 { + t.Fatal(r.Code) + } + type recursive struct { + Next *recursive `json:"next"` + } + if _, err = openapi.SchemaOf[recursive](); err == nil { + t.Fatal("recursive DTO silently truncated") + } +} diff --git a/pagination/pagination.go b/pagination/pagination.go new file mode 100644 index 0000000..598fb4b --- /dev/null +++ b/pagination/pagination.go @@ -0,0 +1,73 @@ +// Package pagination defines bounded offset pagination with camelCase fields. +package pagination + +import ( + "errors" + "net/url" + "strconv" +) + +type Config struct { + DefaultSize int `json:"defaultSize"` + MaxSize int `json:"maxSize"` + MaxOffset int64 `json:"maxOffset"` +} +type Request struct { + Page int64 `json:"page"` + PageSize int `json:"pageSize"` +} + +func (r Request) Offset() int64 { return (r.Page - 1) * int64(r.PageSize) } +func Parse(query url.Values, cfg Config) (Request, error) { + if cfg.DefaultSize == 0 { + cfg.DefaultSize = 20 + } + if cfg.MaxSize == 0 { + cfg.MaxSize = 100 + } + if cfg.MaxOffset == 0 { + cfg.MaxOffset = 1000000 + } + if cfg.DefaultSize < 1 || cfg.MaxSize < cfg.DefaultSize || cfg.MaxSize > 10000 || cfg.MaxOffset < 0 { + return Request{}, errors.New("pagination: invalid configuration") + } + page, size := int64(1), int64(cfg.DefaultSize) + for key, target := range map[string]*int64{"page": &page, "pageSize": &size} { + if v, ok := query[key]; ok { + if len(v) != 1 { + return Request{}, errors.New("pagination: duplicate parameter") + } + n, err := strconv.ParseInt(v[0], 10, 64) + if err != nil || n < 1 { + return Request{}, errors.New("pagination: positive integer required") + } + *target = n + } + } + if size > int64(cfg.MaxSize) || (page-1) > cfg.MaxOffset/size { + return Request{}, errors.New("pagination: size or offset exceeds limit") + } + return Request{Page: page, PageSize: int(size)}, nil +} + +type Result[T any] struct { + Items []T `json:"items"` + Page int64 `json:"page"` + PageSize int `json:"pageSize"` + Total int64 `json:"total"` + TotalPages int64 `json:"totalPages"` +} + +func NewResult[T any](r Request, items []T, total int64) (Result[T], error) { + if r.Page < 1 || r.PageSize < 1 || total < 0 || len(items) > r.PageSize || int64(len(items)) > total { + return Result[T]{}, errors.New("pagination: invalid result") + } + if items == nil { + items = []T{} + } + pages := total / int64(r.PageSize) + if total%int64(r.PageSize) != 0 { + pages++ + } + return Result[T]{Items: items, Page: r.Page, PageSize: r.PageSize, Total: total, TotalPages: pages}, nil +} diff --git a/pagination/pagination_test.go b/pagination/pagination_test.go new file mode 100644 index 0000000..65b5ab8 --- /dev/null +++ b/pagination/pagination_test.go @@ -0,0 +1,30 @@ +package pagination + +import ( + "encoding/json" + "math" + "net/url" + "strings" + "testing" +) + +func TestBoundsAndEnvelope(t *testing.T) { + for _, raw := range []string{"page=0", "page=1&page=2", "pageSize=101", "page=9223372036854775807&pageSize=100", "pageSize=-1"} { + q, _ := url.ParseQuery(raw) + if _, err := Parse(q, Config{}); err == nil { + t.Fatal(raw) + } + } + r, err := Parse(url.Values{"page": {"2"}, "pageSize": {"30"}}, Config{}) + if err != nil || r.Offset() != 30 { + t.Fatal(r, err) + } + result, err := NewResult(r, []string(nil), math.MaxInt64) + if err != nil || result.TotalPages != math.MaxInt64/30+1 { + t.Fatal(result, err) + } + raw, _ := json.Marshal(result) + if !strings.Contains(string(raw), `"items":[]`) || !strings.Contains(string(raw), `"pageSize":30`) { + t.Fatal(string(raw)) + } +} diff --git a/queue/queue.go b/queue/queue.go new file mode 100644 index 0000000..ec35ff2 --- /dev/null +++ b/queue/queue.go @@ -0,0 +1,62 @@ +// Package queue defines durable, at-least-once work delivery. Handlers must be +// idempotent: a crash after a business commit but before ACK can cause redelivery. +// This is a message queue, not a workflow engine or a distributed transaction. +package queue + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "strings" +) + +var ErrFull = errors.New("queue: capacity reached") +var ErrConflict = errors.New("queue: message ID already has different content") +var ErrLeaseLost = errors.New("queue: delivery ownership changed") + +type Message struct { + ID string `json:"id"` // Stable producer identity, saved before submitting. + Name string `json:"name"` + Payload json.RawMessage `json:"payload"` +} +type Receipt struct { + ID string `json:"id"` + BackendID string `json:"backendId"` +} +type Delivery struct { + Message + Attempt int64 `json:"attempt"` +} +type Handler func(context.Context, Delivery) error +type Publisher interface { + Publish(context.Context, Message) (Receipt, error) +} +type Consumer interface { + Run(context.Context, Handler) error +} + +// Reject marks a known permanent failure; it is retained for operator review. +// Ordinary errors are retried up to the consumer's configured attempt limit. +type Rejection struct{ Code string } + +func (e *Rejection) Error() string { return "queue: rejected: " + e.Code } +func Reject(code string) error { return &Rejection{Code: code} } + +func Validate(ctx context.Context, m Message, maxBytes int) error { + if ctx == nil { + return errors.New("queue: context required") + } + if err := ctx.Err(); err != nil { + return err + } + for _, v := range []string{m.ID, m.Name} { + if v == "" || len(v) > 128 || strings.IndexFunc(v, func(r rune) bool { return r < 33 || r > 126 }) >= 0 { + return errors.New("queue: ID and name must be 1..128 printable ASCII characters") + } + } + if len(m.Payload) > maxBytes || !json.Valid(m.Payload) { + return fmt.Errorf("queue: payload must be valid JSON within %d bytes", maxBytes) + } + return nil +} diff --git a/queue/redis/consumer.go b/queue/redis/consumer.go new file mode 100644 index 0000000..9dfa929 --- /dev/null +++ b/queue/redis/consumer.go @@ -0,0 +1,213 @@ +package redis + +import ( + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "strings" + "sync" + "sync/atomic" + "time" + + "git.yuntuo.hk/k2safe/go-omni/queue" + goredis "github.com/redis/go-redis/v9" +) + +type ConsumerConfig struct { + Workers int `json:"workers"` + MaxAttempts int64 `json:"maxAttempts"` + TaskTimeout time.Duration `json:"taskTimeout"` + VisibilityTimeout time.Duration `json:"visibilityTimeout"` + ShutdownTimeout time.Duration `json:"shutdownTimeout"` +} +type Consumer struct { + store *Store + cfg ConsumerConfig + used atomic.Bool +} + +func NewConsumer(store *Store, cfg ConsumerConfig) (*Consumer, error) { + if store == nil || cfg.Workers < 0 || cfg.MaxAttempts < 0 || cfg.TaskTimeout < 0 || cfg.VisibilityTimeout < 0 || cfg.ShutdownTimeout < 0 { + return nil, errors.New("queue/redis: invalid consumer configuration") + } + if cfg.Workers == 0 { + cfg.Workers = 4 + } + if cfg.MaxAttempts == 0 { + cfg.MaxAttempts = 3 + } + if cfg.TaskTimeout == 0 { + cfg.TaskTimeout = 30 * time.Second + } + if cfg.VisibilityTimeout == 0 { + cfg.VisibilityTimeout = 2 * cfg.TaskTimeout + } + if cfg.ShutdownTimeout == 0 { + cfg.ShutdownTimeout = 5 * time.Second + } + if cfg.Workers > 128 || cfg.MaxAttempts > 100 || cfg.TaskTimeout < 10*time.Millisecond || cfg.TaskTimeout > time.Hour || cfg.VisibilityTimeout < 2*cfg.TaskTimeout || cfg.VisibilityTimeout > 24*time.Hour || cfg.ShutdownTimeout > time.Minute { + return nil, errors.New("queue/redis: consumer limits exceeded") + } + return &Consumer{store: store, cfg: cfg}, nil +} + +// Run is called once. Cancellation stops fetching and cancels handlers; ACK is +// deferred to a later delivery if cancellation makes the outcome uncertain. +// A handler must honor context cancellation. Go cannot terminate an arbitrary +// function; an ignoring handler can outlive ShutdownTimeout (at most Workers). +func (c *Consumer) Run(ctx context.Context, handler queue.Handler) error { + if ctx == nil || handler == nil { + return errors.New("queue/redis: context and handler required") + } + if !c.used.CompareAndSwap(false, true) { + return errors.New("queue/redis: consumer already run") + } + signature := fmt.Sprintf("%d:%d:%d", c.cfg.MaxAttempts, c.cfg.TaskTimeout.Milliseconds(), c.cfg.VisibilityTimeout.Milliseconds()) + if err := goredis.NewScript(`local v=redis.call('GET',KEYS[1]);if v and v~=ARGV[1] then return redis.error_reply('consumer configuration mismatch') end;redis.call('SET',KEYS[1],ARGV[1]);return 1`).Run(ctx, c.store.client, []string{c.store.keys[5]}, signature).Err(); err != nil { + return err + } + if err := c.store.client.XGroupCreateMkStream(ctx, c.store.keys[0], "workers", "0").Err(); err != nil && !strings.HasPrefix(err.Error(), "BUSYGROUP ") { + return err + } + var token [16]byte + if _, err := rand.Read(token[:]); err != nil { + return err + } + runCtx, cancel := context.WithCancel(ctx) + defer cancel() + failures := make(chan error, c.cfg.Workers) + var wg sync.WaitGroup + for n := 0; n < c.cfg.Workers; n++ { + wg.Add(1) + go func(n int) { + defer wg.Done() + if err := c.loop(runCtx, hex.EncodeToString(token[:])+fmt.Sprint(n), handler); err != nil { + failures <- err + cancel() + } + }(n) + } + done := make(chan struct{}) + go func() { wg.Wait(); close(done) }() + select { + case <-done: + case <-runCtx.Done(): + select { + case <-done: + case <-time.After(c.cfg.ShutdownTimeout): + return errors.New("queue/redis: handler shutdown timeout") + } + } + select { + case err := <-failures: + return err + default: + return ctx.Err() + } +} +func (c *Consumer) loop(ctx context.Context, owner string, handler queue.Handler) error { + s := c.store + defer func() { + cleanup, cancel := context.WithTimeout(context.Background(), time.Second) + defer cancel() + _ = goredis.NewScript(`local p=redis.call('XPENDING',KEYS[1],'workers','-','+',1,ARGV[1]);if #p==0 then return redis.call('XGROUP','DELCONSUMER',KEYS[1],'workers',ARGV[1]) end;return 0`).Run(cleanup, s.client, []string{s.keys[0]}, owner).Err() + }() + cursor := "0-0" + for ctx.Err() == nil { + messages, next, err := s.client.XAutoClaim(ctx, &goredis.XAutoClaimArgs{Stream: s.keys[0], Group: "workers", Consumer: owner, MinIdle: c.cfg.VisibilityTimeout, Start: cursor, Count: 1}).Result() + if err != nil && err != goredis.Nil { + if ctx.Err() != nil { + return nil + } + return err + } + cursor = next + if cursor == "" { + cursor = "0-0" + } + if len(messages) == 0 { + streams, err := s.client.XReadGroup(ctx, &goredis.XReadGroupArgs{Group: "workers", Consumer: owner, Streams: []string{s.keys[0], ">"}, Count: 1, Block: -1}).Result() + if err != nil && err != goredis.Nil { + if ctx.Err() != nil { + return nil + } + return err + } + if len(streams) > 0 { + messages = streams[0].Messages + } + } + if len(messages) == 0 { + select { + case <-ctx.Done(): + return nil + case <-time.After(25 * time.Millisecond): + } + continue + } + message := messages[0] + raw, ok := message.Values["message"].(string) + var m queue.Message + if !ok || len(raw) > s.cfg.MaxPayloadBytes+1024 || json.Unmarshal([]byte(raw), &m) != nil { + return errors.New("queue/redis: corrupt message; operator review required") + } + pending, err := s.client.XPendingExt(ctx, &goredis.XPendingExtArgs{Stream: s.keys[0], Group: "workers", Start: message.ID, End: message.ID, Count: 1}).Result() + if err != nil { + if ctx.Err() != nil { + return nil + } + return err + } + if len(pending) != 1 || pending[0].Consumer != owner { + continue + } + delivery := queue.Delivery{Message: m, Attempt: pending[0].RetryCount} + reason := "" + if delivery.Attempt > c.cfg.MaxAttempts { + reason = "attemptsExhausted" + } else { + workCtx, cancel := context.WithTimeout(ctx, c.cfg.TaskTimeout) + err = invoke(workCtx, handler, delivery) + deadlineErr := workCtx.Err() + cancel() + if ctx.Err() != nil { + return nil + } + var rejection *queue.Rejection + if errors.As(err, &rejection) { + reason = "rejected" + } else if err != nil || deadlineErr != nil { + if delivery.Attempt < c.cfg.MaxAttempts { + continue + } + reason = "attemptsExhausted" + } + } + ackCtx, ackCancel := context.WithTimeout(ctx, 3*time.Second) + err = s.finish(ackCtx, message.ID, owner, delivery, reason) + ackCancel() + if err != nil && !errors.Is(err, queue.ErrLeaseLost) { + if ctx.Err() != nil { + return nil + } + return err + } + } + return nil +} +func invoke(ctx context.Context, handler queue.Handler, delivery queue.Delivery) (err error) { + defer func() { + if recover() != nil { + err = errors.New("queue: handler panic") + } + }() + if err = ctx.Err(); err != nil { + return err + } + return handler(ctx, delivery) +} + +var _ queue.Consumer = (*Consumer)(nil) diff --git a/queue/redis/consumer_test.go b/queue/redis/consumer_test.go new file mode 100644 index 0000000..1b761d0 --- /dev/null +++ b/queue/redis/consumer_test.go @@ -0,0 +1,154 @@ +package redis + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/queue" + goredis "github.com/redis/go-redis/v9" + "os" + "sync/atomic" + "testing" + "time" +) + +func testStore(t *testing.T, max int) *Store { + t.Helper() + address := os.Getenv("GO_OMNI_REDIS_TEST_ADDR") + if address == "" { + t.Skip("requires isolated Redis") + } + client := goredis.NewClient(&goredis.Options{Addr: address, PoolSize: 4, MaxRetries: -1, ContextTimeoutEnabled: true, ReadTimeout: time.Second, WriteTimeout: time.Second}) + s, err := New(client, Config{Namespace: fmt.Sprint(t.Name(), time.Now().UnixNano()), MaxRecords: max, DedupTTL: time.Second}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { client.Del(context.Background(), s.keys...); client.Close() }) + return s +} +func message(id string) queue.Message { + return queue.Message{ID: id, Name: "test", Payload: json.RawMessage(`{"value":1}`)} +} +func eventually(t *testing.T, f func() bool) { + t.Helper() + deadline := time.Now().Add(3 * time.Second) + for !f() { + if time.Now().After(deadline) { + t.Fatal("condition did not recover") + } + time.Sleep(10 * time.Millisecond) + } +} +func TestQueueRedisIntegration(t *testing.T) { + t.Run("publicationAndCapacity", func(t *testing.T) { + s := testStore(t, 1) + ctx := context.Background() + receipt, err := s.Publish(ctx, message("first")) + if err != nil { + t.Fatal(err) + } + again, err := s.Publish(ctx, message("first")) + if err != nil || again != receipt { + t.Fatal(again, err) + } + changed := message("first") + changed.Payload = json.RawMessage(`2`) + if _, err = s.Publish(ctx, changed); !errors.Is(err, queue.ErrConflict) { + t.Fatal(err) + } + if _, err = s.Publish(ctx, message("second")); !errors.Is(err, queue.ErrFull) { + t.Fatal(err) + } + }) + t.Run("staleAcknowledgement", func(t *testing.T) { + s := testStore(t, 10) + ctx := context.Background() + receipt, _ := s.Publish(ctx, message("first")) + s.client.XGroupCreateMkStream(ctx, s.keys[0], "workers", "0") + s.client.XReadGroup(ctx, &goredis.XReadGroupArgs{Group: "workers", Consumer: "old", Streams: []string{s.keys[0], ">"}, Count: 1, Block: -1}) + if err := s.client.XClaim(ctx, &goredis.XClaimArgs{Stream: s.keys[0], Group: "workers", Consumer: "new", MinIdle: 0, Messages: []string{receipt.BackendID}}).Err(); err != nil { + t.Fatal(err) + } + delivery := queue.Delivery{Message: message("first"), Attempt: 2} + if err := s.finish(ctx, receipt.BackendID, "old", delivery, ""); !errors.Is(err, queue.ErrLeaseLost) { + t.Fatal(err) + } + if err := s.finish(ctx, receipt.BackendID, "new", delivery, ""); err != nil { + t.Fatal(err) + } + if n := s.client.XLen(ctx, s.keys[0]).Val(); n != 0 { + t.Fatal(n) + } + }) + t.Run("crashReclaimRetryAndDeadLetters", func(t *testing.T) { + s := testStore(t, 10) + ctx := context.Background() + s.Publish(ctx, message("crashed")) + s.Publish(ctx, message("retry")) + s.Publish(ctx, message("reject")) + s.client.XGroupCreateMkStream(ctx, s.keys[0], "workers", "0") + s.client.XReadGroup(ctx, &goredis.XReadGroupArgs{Group: "workers", Consumer: "crashedProcess", Streams: []string{s.keys[0], ">"}, Count: 1, Block: -1}) + cfg := ConsumerConfig{Workers: 1, MaxAttempts: 3, TaskTimeout: 50 * time.Millisecond, VisibilityTimeout: 100 * time.Millisecond, ShutdownTimeout: time.Second} + consumer, _ := NewConsumer(s, cfg) + second, _ := NewConsumer(s, cfg) + runCtx, cancel := context.WithCancel(ctx) + var retries, reclaimed, rejected atomic.Int32 + handler := func(ctx context.Context, d queue.Delivery) error { + switch d.ID { + case "crashed": + if d.Attempt != 2 { + t.Error("reclaim count", d.Attempt) + } + reclaimed.Add(1) + return nil + case "retry": + retries.Add(1) + return errors.New("known retryable failure") + default: + rejected.Add(1) + return queue.Reject("invalidInput") + } + } + done := make(chan error, 2) + go func() { done <- consumer.Run(runCtx, handler) }() + go func() { done <- second.Run(runCtx, handler) }() + defer cancel() + eventually(t, func() bool { + dead, err := s.DeadLetters(ctx, "", 100) + return err == nil && len(dead) == 2 && reclaimed.Load() == 1 + }) + cancel() + for i := 0; i < 2; i++ { + if err := <-done; !errors.Is(err, context.Canceled) { + t.Fatal(err) + } + } + if retries.Load() != 3 || rejected.Load() != 1 { + t.Fatal(retries.Load(), rejected.Load()) + } + dead, err := s.DeadLetters(ctx, "", 1) + if err != nil || len(dead) != 1 { + t.Fatal(dead, err) + } + rest, err := s.DeadLetters(ctx, dead[0].StreamID, 1) + if err != nil || len(rest) != 1 || rest[0].StreamID == dead[0].StreamID { + t.Fatal(rest, err) + } + if err = s.DiscardDead(ctx, dead[0].Message.ID, dead[0].StreamID); err != nil { + t.Fatal(err) + } + remaining, _ := s.DeadLetters(ctx, "", 100) + if len(remaining) != 1 { + t.Fatal(remaining) + } + // Retained completed IDs expire and release publication capacity. + eventually(t, func() bool { + _, err := s.Publish(ctx, message("crashed")) + if err != nil { + return false + } + return s.client.XLen(ctx, s.keys[0]).Val() == 2 + }) + }) +} diff --git a/queue/redis/store.go b/queue/redis/store.go new file mode 100644 index 0000000..3f6535b --- /dev/null +++ b/queue/redis/store.go @@ -0,0 +1,187 @@ +// Package redis implements one managed Redis Streams consumer group per queue. +// Use Redis 6.2+, persistence, noeviction and a private namespace. The application +// owns the client. Delivery durability depends on the Redis deployment. +package redis + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "time" + + "git.yuntuo.hk/k2safe/go-omni/queue" + goredis "github.com/redis/go-redis/v9" +) + +type Config struct { + Namespace string `json:"namespace"` + MaxRecords int `json:"maxRecords"` // Includes pending, dead and retained completed identities. + MaxPayloadBytes int `json:"maxPayloadBytes"` + DedupTTL time.Duration `json:"dedupTtl"` // Starts at ACK; pending/dead records never expire automatically. +} +type Store struct { + client goredis.UniversalClient + cfg Config + keys []string + signature string +} + +func New(client goredis.UniversalClient, cfg Config) (*Store, error) { + if client == nil || cfg.Namespace == "" || len(cfg.Namespace) > 256 || cfg.MaxRecords < 0 || cfg.MaxPayloadBytes < 0 || cfg.DedupTTL < 0 { + return nil, errors.New("queue/redis: invalid configuration") + } + if cfg.MaxRecords == 0 { + cfg.MaxRecords = 1000 + } + if cfg.MaxPayloadBytes == 0 { + cfg.MaxPayloadBytes = 64 << 10 + } + if cfg.DedupTTL == 0 { + cfg.DedupTTL = 24 * time.Hour + } + if cfg.MaxRecords > 100000 || cfg.MaxPayloadBytes > 1<<20 || cfg.DedupTTL < time.Second || cfg.DedupTTL > 30*24*time.Hour { + return nil, errors.New("queue/redis: configuration exceeds supported bounds") + } + sum := sha256.Sum256([]byte(cfg.Namespace)) + base := "goOmni:{" + hex.EncodeToString(sum[:]) + "}:queue" + return &Store{client: client, cfg: cfg, keys: []string{base + ":stream", base + ":records", base + ":expiry", base + ":dead", base + ":config", base + ":consumerConfig"}, signature: fmt.Sprintf("%d:%d:%d", cfg.MaxRecords, cfg.MaxPayloadBytes, cfg.DedupTTL.Milliseconds())}, nil +} + +var publish = goredis.NewScript(` +local config=redis.call('GET',KEYS[5]);if config and config~=ARGV[5] then return redis.error_reply('configuration mismatch') end +if not config then redis.call('SET',KEYS[5],ARGV[5]) end +local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000) +local expired=redis.call('ZRANGEBYSCORE',KEYS[3],'-inf',now,'LIMIT',0,128) +for _,id in ipairs(expired) do redis.call('HDEL',KEYS[2],id);redis.call('ZREM',KEYS[3],id) end +local expiry=redis.call('ZSCORE',KEYS[3],ARGV[1]);if expiry and tonumber(expiry)<=now then redis.call('HDEL',KEYS[2],ARGV[1]);redis.call('ZREM',KEYS[3],ARGV[1]) end +local old=redis.call('HGET',KEYS[2],ARGV[1]);if old then local r=cjson.decode(old);if r.fingerprint~=ARGV[2] then return {-1,''} end;return {0,r.streamId} end +if redis.call('HLEN',KEYS[2])>=tonumber(ARGV[4]) then return {-2,''} end +local id=redis.call('XADD',KEYS[1],'*','message',ARGV[3]) +redis.call('HSET',KEYS[2],ARGV[1],cjson.encode({fingerprint=ARGV[2],streamId=id,state='pending'}));return {1,id} +`) + +func (s *Store) Publish(ctx context.Context, m queue.Message) (queue.Receipt, error) { + if err := queue.Validate(ctx, m, s.cfg.MaxPayloadBytes); err != nil { + return queue.Receipt{}, err + } + raw, _ := json.Marshal(m) + if len(raw) > s.cfg.MaxPayloadBytes+1024 { + return queue.Receipt{}, errors.New("queue/redis: encoded message exceeds limit") + } + sum := sha256.Sum256(raw) + r, err := publish.Run(ctx, s.client, s.keys, m.ID, hex.EncodeToString(sum[:]), string(raw), s.cfg.MaxRecords, s.signature).Slice() + if err != nil { + return queue.Receipt{}, err + } + if len(r) != 2 { + return queue.Receipt{}, errors.New("queue/redis: invalid publish result") + } + switch r[0] { + case int64(-1): + return queue.Receipt{}, queue.ErrConflict + case int64(-2): + return queue.Receipt{}, queue.ErrFull + } + id, ok := r[1].(string) + if !ok { + return queue.Receipt{}, errors.New("queue/redis: invalid receipt") + } + return queue.Receipt{ID: m.ID, BackendID: id}, nil +} + +var finish = goredis.NewScript(` +local p=redis.call('XPENDING',KEYS[1],'workers',ARGV[1],ARGV[1],1) +if #p~=1 or p[1][2]~=ARGV[2] then return 0 end +local raw=redis.call('HGET',KEYS[2],ARGV[3]);if not raw then return redis.error_reply('message record missing') end +local record=cjson.decode(raw);if record.streamId~=ARGV[1] or record.state~='pending' then return redis.error_reply('message identity mismatch') end +if ARGV[4]~='' then + record.state='dead';redis.call('ZADD',KEYS[4],0,ARGV[1]);record.dead=cjson.decode(ARGV[4]) +else + record.state='completed';local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000);redis.call('ZADD',KEYS[3],now+tonumber(ARGV[5]),ARGV[3]) +end +redis.call('HSET',KEYS[2],ARGV[3],cjson.encode(record));redis.call('XACK',KEYS[1],'workers',ARGV[1]) +if ARGV[4]=='' then redis.call('XDEL',KEYS[1],ARGV[1]) end +return 1 +`) + +func (s *Store) finish(ctx context.Context, id, owner string, m queue.Delivery, reason string) error { + var dead []byte + if reason != "" { + dead, _ = json.Marshal(map[string]any{"attempt": m.Attempt, "reason": reason}) + } + n, err := finish.Run(ctx, s.client, s.keys, id, owner, m.ID, string(dead), s.cfg.DedupTTL.Milliseconds()).Int() + if err != nil { + return err + } + if n != 1 { + return queue.ErrLeaseLost + } + return nil +} + +type DeadLetter struct { + StreamID string `json:"streamId"` + Message queue.Message `json:"message"` + Attempt int64 `json:"attempt"` + Reason string `json:"reason"` +} + +// DeadLetters returns a bounded page, ordered by opaque Redis stream ID. Pass +// the last StreamID as after. Payloads remain retained until explicit removal. +func (s *Store) DeadLetters(ctx context.Context, after string, limit int64) ([]DeadLetter, error) { + if ctx == nil || limit < 1 || limit > 100 { + return nil, errors.New("queue/redis: context and limit 1..100 required") + } + min := "-" + if after != "" { + min = "(" + after + } + ids, err := s.client.ZRangeByLex(ctx, s.keys[3], &goredis.ZRangeBy{Min: min, Max: "+", Count: limit}).Result() + if err != nil { + return nil, err + } + out := make([]DeadLetter, 0, len(ids)) + for _, id := range ids { + messages, err := s.client.XRangeN(ctx, s.keys[0], id, id, 1).Result() + if err != nil { + return nil, err + } + if len(messages) == 0 { + continue + } + var m queue.Message + raw, _ := messages[0].Values["message"].(string) + if err = json.Unmarshal([]byte(raw), &m); err != nil { + return nil, err + } + record, err := s.client.HGet(ctx, s.keys[1], m.ID).Result() + if err == goredis.Nil { + continue + } + if err != nil { + return nil, err + } + var saved struct { + Dead struct { + Attempt int64 `json:"attempt"` + Reason string `json:"reason"` + } `json:"dead"` + } + if err = json.Unmarshal([]byte(record), &saved); err != nil { + return nil, err + } + out = append(out, DeadLetter{StreamID: id, Message: m, Attempt: saved.Dead.Attempt, Reason: saved.Dead.Reason}) + } + return out, nil +} + +// DiscardDead deletes only a matching dead delivery after operator review. +// It retains the producer identity for DedupTTL, preventing immediate resubmit. +func (s *Store) DiscardDead(ctx context.Context, messageID, streamID string) error { + return goredis.NewScript(`local raw=redis.call('HGET',KEYS[2],ARGV[1]);if not raw then return 0 end;local r=cjson.decode(raw);if r.state~='dead' or r.streamId~=ARGV[2] then return redis.error_reply('dead identity mismatch') end;r.state='discarded';r.dead=nil;redis.call('HSET',KEYS[2],ARGV[1],cjson.encode(r));redis.call('ZREM',KEYS[4],ARGV[2]);redis.call('XDEL',KEYS[1],ARGV[2]);local t=redis.call('TIME');redis.call('ZADD',KEYS[3],t[1]*1000+math.floor(t[2]/1000)+tonumber(ARGV[3]),ARGV[1]);return 1`).Run(ctx, s.client, s.keys, messageID, streamID, s.cfg.DedupTTL.Milliseconds()).Err() +} + +var _ queue.Publisher = (*Store)(nil) diff --git a/resilience/breaker.go b/resilience/breaker.go new file mode 100644 index 0000000..78eb112 --- /dev/null +++ b/resilience/breaker.go @@ -0,0 +1,129 @@ +// Package resilience provides per-dependency circuit breaking and concurrency isolation. +package resilience + +import ( + "context" + "errors" + "sync" + "time" +) + +var ErrOpen = errors.New("resilience: circuit open") +var ErrBusy = errors.New("resilience: concurrency limit reached") + +type Config struct { + FailureThreshold int `json:"failureThreshold"` + MaxConcurrent int `json:"maxConcurrent"` + OpenTimeout time.Duration `json:"openTimeout"` + IsFailure func(error) bool `json:"-"` +} +type Breaker struct { + mu sync.Mutex + config Config + active, failures int + state string + until time.Time + generation uint64 + now func() time.Time +} +type Snapshot struct { + State string `json:"state"` + Active int `json:"active"` + Failures int `json:"failures"` +} + +func New(cfg Config) (*Breaker, error) { + if cfg.FailureThreshold < 0 || cfg.MaxConcurrent < 0 || cfg.OpenTimeout < 0 { + return nil, errors.New("resilience: invalid configuration") + } + if cfg.FailureThreshold == 0 { + cfg.FailureThreshold = 5 + } + if cfg.MaxConcurrent == 0 { + cfg.MaxConcurrent = 32 + } + if cfg.OpenTimeout == 0 { + cfg.OpenTimeout = 5 * time.Second + } + if cfg.IsFailure == nil { + cfg.IsFailure = func(err error) bool { return err != nil && !errors.Is(err, context.Canceled) } + } + return &Breaker{config: cfg, state: "closed", now: time.Now}, nil +} + +// Acquire returns a completion callback; it must be called exactly once. +// A half-open circuit admits one probe. Old-generation results cannot close it. +func (b *Breaker) Acquire(ctx context.Context) (func(bool), error) { + if ctx == nil { + return nil, errors.New("resilience: nil context") + } + if err := ctx.Err(); err != nil { + return nil, err + } + b.mu.Lock() + defer b.mu.Unlock() + if b.state == "open" { + if b.now().Before(b.until) { + return nil, ErrOpen + } + b.state = "halfOpen" + b.generation++ + } + if b.state == "halfOpen" && b.active > 0 { + return nil, ErrOpen + } + if b.active >= b.config.MaxConcurrent { + return nil, ErrBusy + } + b.active++ + generation := b.generation + var once sync.Once + return func(success bool) { + once.Do(func() { + b.mu.Lock() + defer b.mu.Unlock() + b.active-- + if generation != b.generation { + return + } + if success { + b.failures = 0 + if b.state == "halfOpen" { + b.state = "closed" + b.generation++ + } + return + } + b.failures++ + if b.state == "halfOpen" || b.failures >= b.config.FailureThreshold { + b.state = "open" + b.until = b.now().Add(b.config.OpenTimeout) + b.generation++ + } + }) + }, nil +} +func (b *Breaker) Do(ctx context.Context, work func(context.Context) error) (err error) { + if work == nil { + return errors.New("resilience: nil work") + } + done, err := b.Acquire(ctx) + if err != nil { + return err + } + complete := false + defer func() { + if !complete { + done(false) + } + }() + err = work(ctx) + done(!b.config.IsFailure(err)) + complete = true + return err +} +func (b *Breaker) Snapshot() Snapshot { + b.mu.Lock() + defer b.mu.Unlock() + return Snapshot{State: b.state, Active: b.active, Failures: b.failures} +} diff --git a/resilience/breaker_test.go b/resilience/breaker_test.go new file mode 100644 index 0000000..70f509a --- /dev/null +++ b/resilience/breaker_test.go @@ -0,0 +1,75 @@ +package resilience + +import ( + "context" + "errors" + "io" + "net/http" + "strings" + "testing" + "time" +) + +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } +func TestBreakerProbeAndLateResult(t *testing.T) { + b, _ := New(Config{FailureThreshold: 1, MaxConcurrent: 2, OpenTimeout: time.Second}) + now := time.Now() + b.now = func() time.Time { return now } + ctx := context.Background() + first, _ := b.Acquire(ctx) + late, _ := b.Acquire(ctx) + first(false) + late(true) + if b.Snapshot().State != "open" { + t.Fatal(b.Snapshot()) + } + if _, err := b.Acquire(ctx); !errors.Is(err, ErrOpen) { + t.Fatal(err) + } + now = now.Add(time.Second) + probe, err := b.Acquire(ctx) + if err != nil { + t.Fatal(err) + } + if _, err = b.Acquire(ctx); !errors.Is(err, ErrOpen) { + t.Fatal(err) + } + probe(true) + probe(false) + if s := b.Snapshot(); s.State != "closed" || s.Active != 0 { + t.Fatal(s) + } +} +func TestTransportHoldsSlotAndReleasesPanic(t *testing.T) { + b, _ := New(Config{MaxConcurrent: 1}) + tr := Transport{Breaker: b, Base: roundTripFunc(func(*http.Request) (*http.Response, error) { + return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader("hello"))}, nil + })} + req, _ := http.NewRequest("GET", "http://example.test", nil) + r, err := tr.RoundTrip(req) + if err != nil { + t.Fatal(err) + } + if _, err = tr.RoundTrip(req); !errors.Is(err, ErrBusy) { + t.Fatal(err) + } + io.ReadAll(r.Body) + r.Body.Close() + if b.Snapshot().Active != 0 { + t.Fatal(b.Snapshot()) + } + tr.Base = roundTripFunc(func(*http.Request) (*http.Response, error) { panic("fixture") }) + func() { + defer func() { + if recover() == nil { + t.Error("panic lost") + } + }() + tr.RoundTrip(req) + }() + if b.Snapshot().Active != 0 { + t.Fatal(b.Snapshot()) + } +} diff --git a/resilience/http.go b/resilience/http.go new file mode 100644 index 0000000..3606fef --- /dev/null +++ b/resilience/http.go @@ -0,0 +1,67 @@ +package resilience + +import ( + "errors" + "io" + "net/http" +) + +// Transport isolates one logical upstream. Create separate Breakers for +// different dependencies. Slots remain occupied until body Close/EOF. +type Transport struct { + Base http.RoundTripper + Breaker *Breaker +} + +func (t Transport) RoundTrip(r *http.Request) (*http.Response, error) { + if t.Breaker == nil { + return nil, errors.New("resilience: breaker required") + } + done, err := t.Breaker.Acquire(r.Context()) + if err != nil { + return nil, err + } + defer func() { + if v := recover(); v != nil { + done(false) + panic(v) + } + }() + base := t.Base + if base == nil { + base = http.DefaultTransport + } + response, err := base.RoundTrip(r) + if err != nil { + done(!t.Breaker.config.IsFailure(err)) + if response != nil && response.Body != nil { + response.Body.Close() + } + return nil, err + } + if response == nil || response.Body == nil { + done(false) + return nil, errors.New("resilience: invalid transport response") + } + response.Body = &breakerBody{ReadCloser: response.Body, done: done, success: response.StatusCode < 500} + return response, nil +} + +type breakerBody struct { + io.ReadCloser + done func(bool) + success bool +} + +func (b *breakerBody) Read(p []byte) (int, error) { + n, err := b.ReadCloser.Read(p) + if err != nil { + b.done(b.success && err == io.EOF) + } + return n, err +} +func (b *breakerBody) Close() error { + err := b.ReadCloser.Close() + b.done(b.success && err == nil) + return err +} diff --git a/scripts/verifyModules.py b/scripts/verifyModules.py index a910926..e73c0c2 100644 --- a/scripts/verifyModules.py +++ b/scripts/verifyModules.py @@ -22,6 +22,9 @@ def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--image", default="golang:1.25") parser.add_argument("--redis-image", default="redis:7.4-alpine") + parser.add_argument("--advanced", action="store_true", help="also verify shared state, durable queue, S3 and MySQL migrations") + parser.add_argument("--s3-image", default="quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z") + parser.add_argument("--mysql-image", default="m.daocloud.io/docker.io/library/mysql:8.4") parser.add_argument("--goproxy", default="") parser.add_argument("--wall-timeout", type=int, default=900) parser.add_argument("--output", type=Path, default=Path("performance/modulesLinux.json")) @@ -40,9 +43,13 @@ def main(): output_path.parent.mkdir(parents=True, exist_ok=True) log_path = output_path.with_suffix(".txt") prefix = "go-omni-modules-" + uuid.uuid4().hex[:12] - names = [prefix + "-" + stage for stage in ("build", "redis")] + stages = ("build", "redis", "s3", "mysql") if args.advanced else ("build", "redis") + names = [prefix + "-" + stage for stage in stages] paths = [root / "go.mod", root / "go.sum", root / "httpx/context.go", Path(__file__).resolve()] - for directory in ("cache", "httpc", "task", "middleware", "cmd/go-omni", "examples/modules"): + directories = ["cache", "httpc", "task", "middleware", "cmd/go-omni", "examples/modules"] + if args.advanced: + directories += ["auth", "config", "logx", "openapi", "telemetry", "resilience", "storage", "queue", "audit", "pagination", "database/migrate", "internal/storagefixture", "examples/platform"] + for directory in directories: paths.extend(p for p in (root / directory).rglob("*") if p.is_file() and p.suffix in (".go", ".tmpl")) report = { "status": "running", "startedAt": datetime.now(timezone.utc).isoformat(), @@ -86,7 +93,10 @@ def main(): report["cleanedContainers"].append(name) try: - for image in (args.image, args.redis_image): + images = [args.image, args.redis_image] + if args.advanced: + images += [args.s3_image, args.mysql_image] + for image in images: result = subprocess.run([docker, "image", "inspect", image], capture_output=True, timeout=15) if result.returncode: print("Pulling " + image, flush=True) @@ -101,9 +111,11 @@ def main(): "-e", "GOWORK=off", "-e", "GOMAXPROCS=2", "-e", "GO_OMNI_SCAFFOLD_INTEGRATION=1"] if args.goproxy: arguments += ["-e", "GOPROXY=" + args.goproxy] - arguments += [args.image, "sh", "-c", - "go version && go test -race -count=1 -timeout=240s ./... && go vet ./... && go build ./... && " - "CGO_ENABLED=0 go test -c -o /output/redis.test ./cache/redis"] + checks = "go version && go test -race -count=1 -timeout=240s ./... && go vet ./... && go build ./... && CGO_ENABLED=0 go test -c -o /output/redis.test ./cache/redis" + if args.advanced: + for binary, package in (("shared", "middleware/redis"), ("queue", "queue/redis"), ("s3", "storage/s3"), ("mysql", "database/migrate")): + checks += f" && CGO_ENABLED=0 go test -c -o /output/{binary}.test ./{package}" + arguments += [args.image, "sh", "-c", checks] container(names[0], arguments) print("Real Redis TTL/read/write/delete/failure checks; external network disabled", flush=True) redis_check = '''set -eu @@ -118,15 +130,54 @@ until redis-cli -h 127.0.0.1 ping >/dev/null 2>&1; do done /test/redis.test -test.v -test.run '^TestRedisIntegration$' -test.count=1 -test.timeout=30s ''' + if args.advanced: + redis_check += "/test/shared.test -test.v -test.count=1 -test.timeout=45s\n/test/queue.test -test.v -test.count=1 -test.timeout=45s\n" container(names[1], ["--network", "none", "--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", "--user", "999:999", "--tmpfs", "/tmp:rw,nosuid,noexec,size=64m,mode=1777", "-v", f"{temporary}:/test:ro", "--entrypoint", "sh", "-e", "GOMAXPROCS=2", "-e", "GO_OMNI_REDIS_TEST_ADDR=127.0.0.1:6379", args.redis_image, "-c", redis_check]) + if args.advanced: + isolated = ["--network", "none", "--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges", + "-v", f"{temporary}:/test:ro", "--entrypoint", "sh", "-e", "GOMAXPROCS=2"] + print("Real S3 object operations and signature verification in isolated MinIO", flush=True) + s3_check = '''set -eu +minio --version +minio server /data --address 127.0.0.1:9000 --console-address 127.0.0.1:9001 >/tmp/minio.log 2>&1 & +server=$! +trap 'kill "$server" 2>/dev/null || true; wait "$server" 2>/dev/null || true' EXIT +/test/s3.test -test.v -test.run '^TestS3Integration$' -test.count=1 -test.timeout=45s || { cat /tmp/minio.log; exit 1; } +''' + container(names[2], isolated + ["--user", "1000:1000", "--tmpfs", "/tmp:rw,nosuid,noexec,size=64m,mode=1777", + "--tmpfs", "/data:rw,nosuid,noexec,size=128m,mode=1777", "-e", "MINIO_ROOT_USER=goOmniTest", "-e", "MINIO_ROOT_PASSWORD=goOmniTestSecret", + "-e", "GO_OMNI_S3_TEST_ADDR=127.0.0.1:9000", args.s3_image, "-c", s3_check]) + print("Real MySQL migrations, repeat apply, dirty-state blocking and SQL spans", flush=True) + mysql_check = '''set -eu +mysqld --version +mkdir /tmp/mysql +mysqld --no-defaults --initialize-insecure --datadir=/tmp/mysql --innodb-redo-log-capacity=67108864 >/tmp/init.log 2>&1 || { cat /tmp/init.log; exit 1; } +mysqld --no-defaults --datadir=/tmp/mysql --socket=/tmp/mysql.sock --pid-file=/tmp/mysql.pid --bind-address=127.0.0.1 --port=3306 --mysqlx=OFF --skip-log-bin --skip-name-resolve --innodb-buffer-pool-size=134217728 --innodb-redo-log-capacity=67108864 --max-connections=8 --log-error=/tmp/mysql.log & +server=$! +trap 'kill "$server" 2>/dev/null || true; wait "$server" 2>/dev/null || true' EXIT +attempt=0 +until mysql --no-defaults --socket=/tmp/mysql.sock -uroot -e 'SELECT 1' >/dev/null 2>&1; do + attempt=$((attempt + 1)); if [ "$attempt" -gt 200 ]; then cat /tmp/mysql.log; exit 1; fi; sleep 0.1 +done +mysql --no-defaults --socket=/tmp/mysql.sock -uroot -e 'CREATE DATABASE omni_test' +/test/mysql.test -test.v -test.run '^TestMySQLIntegration$' -test.count=1 -test.timeout=60s +''' + container(names[3], isolated + ["--user", "999:999", "--tmpfs", "/tmp:rw,nosuid,noexec,size=512m,mode=1777", + "--tmpfs", "/var/lib/mysql:rw,nosuid,noexec,size=1m,mode=1777", + "-e", "GO_OMNI_MIGRATE_TEST_DSN=root@unix(/tmp/mysql.sock)/omni_test?parseTime=true&timeout=2s&readTimeout=5s&writeTimeout=5s", + args.mysql_image, "-c", mysql_check]) log.flush() content = log_path.read_text() if "--- PASS: TestRedisIntegration" not in content: raise RuntimeError("missing real Redis pass evidence") + if args.advanced: + for test in ("TestSharedRedisIntegration", "TestExpiredTargetBeyondCleanupBatch", "TestQueueRedisIntegration", "TestS3Integration", "TestMySQLIntegration"): + if "--- PASS: " + test not in content: + raise RuntimeError("missing integration pass evidence: " + test) report["status"] = "passed" except (Exception, KeyboardInterrupt) as error: failure = str(error) or type(error).__name__ diff --git a/storage/local/local.go b/storage/local/local.go new file mode 100644 index 0000000..20c8a2b --- /dev/null +++ b/storage/local/local.go @@ -0,0 +1,171 @@ +// Package local stores objects atomically in a private, flat managed directory. +// This directory must not be writable by untrusted local users/processes. +package local + +import ( + "context" + "crypto/sha256" + "encoding/binary" + "encoding/hex" + "encoding/json" + "errors" + "git.yuntuo.hk/k2safe/go-omni/storage" + "io" + "os" + "path/filepath" + "strings" +) + +type Store struct { + root string + maxBytes int64 +} + +func New(root string, maxBytes int64) (*Store, error) { + if root == "" || maxBytes < 0 || maxBytes > 1<<40 { + return nil, errors.New("storage/local: invalid configuration") + } + if maxBytes == 0 { + maxBytes = 32 << 20 + } + if err := os.MkdirAll(root, 0700); err != nil { + return nil, err + } + root, err := filepath.EvalSymlinks(root) + if err != nil { + return nil, err + } + root, err = filepath.Abs(root) + if err != nil { + return nil, err + } + info, err := os.Stat(root) + if err != nil { + return nil, err + } + if !info.IsDir() || info.Mode().Perm()&0022 != 0 { + return nil, errors.New("storage/local: root must be a private writable directory") + } + return &Store{root: root, maxBytes: maxBytes}, nil +} +func (s *Store) path(key string) string { + sum := sha256.Sum256([]byte(key)) + return filepath.Join(s.root, hex.EncodeToString(sum[:])+".object") +} +func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int64, contentType string) (storage.Info, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.Info{}, err + } + if reader == nil || size < 0 || len(contentType) > 256 || strings.ContainsAny(contentType, "\r\n") { + return storage.Info{}, errors.New("storage/local: invalid upload") + } + if size > s.maxBytes { + return storage.Info{}, storage.ErrTooLarge + } + info := storage.Info{Key: key, Size: size, ContentType: contentType} + metadata, _ := json.Marshal(info) + f, err := os.CreateTemp(s.root, ".upload-") + if err != nil { + return storage.Info{}, err + } + defer os.Remove(f.Name()) + defer f.Close() + if err = binary.Write(f, binary.BigEndian, uint32(len(metadata))); err != nil { + return storage.Info{}, err + } + if _, err = f.Write(metadata); err != nil { + return storage.Info{}, err + } + n, err := io.Copy(f, io.LimitReader(storage.ContextReader{Context: ctx, Reader: reader}, size+1)) + if err != nil { + return storage.Info{}, err + } + if n != size { + return storage.Info{}, errors.New("storage/local: declared size mismatch") + } + if err = ctx.Err(); err != nil { + return storage.Info{}, err + } + if err = f.Sync(); err != nil { + return storage.Info{}, err + } + stat, err := f.Stat() + if err != nil { + return storage.Info{}, err + } + if err = f.Close(); err != nil { + return storage.Info{}, err + } + if err = os.Rename(f.Name(), s.path(key)); err != nil { + return storage.Info{}, err + } + info.ModifiedAt = stat.ModTime() + return info, nil +} +func (s *Store) Get(ctx context.Context, key string) (storage.Object, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.Object{}, err + } + path := s.path(key) + stat, err := os.Lstat(path) + if os.IsNotExist(err) { + return storage.Object{}, storage.ErrNotFound + } + if err != nil { + return storage.Object{}, err + } + if !stat.Mode().IsRegular() { + return storage.Object{}, errors.New("storage/local: not a regular managed object") + } + f, err := os.Open(path) + if err != nil { + return storage.Object{}, err + } + // A concurrent Put can atomically replace the path between Lstat/Open. + // Validate size against the descriptor whose bytes we actually read. + stat, err = f.Stat() + if err != nil { + f.Close() + return storage.Object{}, err + } + ok := false + defer func() { + if !ok { + f.Close() + } + }() + var length uint32 + if err = binary.Read(f, binary.BigEndian, &length); err != nil { + return storage.Object{}, err + } + if length > 4096 { + return storage.Object{}, errors.New("storage/local: invalid object header") + } + data := make([]byte, length) + if _, err = io.ReadFull(f, data); err != nil { + return storage.Object{}, err + } + var info storage.Info + if json.Unmarshal(data, &info) != nil || info.Key != key || info.Size < 0 || info.Size > s.maxBytes || int64(length)+4+info.Size != stat.Size() { + return storage.Object{}, errors.New("storage/local: corrupt object") + } + info.ModifiedAt = stat.ModTime() + ok = true + return storage.Object{Info: info, Body: &body{Reader: storage.ContextReader{Context: ctx, Reader: io.LimitReader(f, info.Size)}, Closer: f}}, nil +} + +type body struct { + io.Reader + io.Closer +} + +func (s *Store) Delete(ctx context.Context, key string) error { + if err := storage.Check(ctx, key); err != nil { + return err + } + err := os.Remove(s.path(key)) + if os.IsNotExist(err) { + return nil + } + return err +} diff --git a/storage/local/local_test.go b/storage/local/local_test.go new file mode 100644 index 0000000..fcf4ee7 --- /dev/null +++ b/storage/local/local_test.go @@ -0,0 +1,42 @@ +package local + +import ( + "context" + "git.yuntuo.hk/k2safe/go-omni/storage/storagetest" + "io" + "os" + "strings" + "testing" +) + +func TestContractAndAtomicFailure(t *testing.T) { + root := t.TempDir() + s, err := New(root, 64) + if err != nil { + t.Fatal(err) + } + storagetest.Run(t, s) + ctx := context.Background() + if _, err = s.Put(ctx, "saved", strings.NewReader("old"), 3, "text/plain"); err != nil { + t.Fatal(err) + } + if _, err = s.Put(ctx, "saved", strings.NewReader("short"), 20, "text/plain"); err == nil { + t.Fatal("short upload committed") + } + again, _ := New(root, 64) + object, err := again.Get(ctx, "saved") + if err != nil { + t.Fatal(err) + } + raw, _ := io.ReadAll(object.Body) + object.Body.Close() + if string(raw) != "old" { + t.Fatal(string(raw)) + } + if err = os.Symlink(s.path("saved"), s.path("linked")); err != nil { + t.Fatal(err) + } + if _, err = s.Get(ctx, "linked"); err == nil { + t.Fatal("symlink followed") + } +} diff --git a/storage/oss/oss.go b/storage/oss/oss.go new file mode 100644 index 0000000..57904dd --- /dev/null +++ b/storage/oss/oss.go @@ -0,0 +1,101 @@ +// Package oss adapts the caller-owned official Alibaba Cloud OSS v2 client. +package oss + +import ( + "context" + "errors" + "io" + "net/url" + "strings" + "time" + + "git.yuntuo.hk/k2safe/go-omni/storage" + sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss" +) + +type Store struct { + client *sdk.Client + bucket string + maxBytes int64 +} + +// New borrows a configured SDK client. This adapter uses simple uploads; large +// multipart uploads are a separate provider capability, not buffered in memory. +func New(client *sdk.Client, bucket string, maxBytes int64) (*Store, error) { + if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<30 { + return nil, errors.New("storage/oss: invalid client, bucket or size limit") + } + if maxBytes == 0 { + maxBytes = 32 << 20 + } + return &Store{client: client, bucket: bucket, maxBytes: maxBytes}, nil +} + +func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int64, contentType string) (storage.Info, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.Info{}, err + } + if reader == nil || size < 0 || len(contentType) > 256 || strings.ContainsAny(contentType, "\r\n") { + return storage.Info{}, errors.New("storage/oss: invalid upload") + } + if size > s.maxBytes { + return storage.Info{}, storage.ErrTooLarge + } + _, err := s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: storage.ContextReader{Context: ctx, Reader: reader}}) + if err != nil { + return storage.Info{}, mapError(err) + } + return storage.Info{Key: key, Size: size, ContentType: contentType, ModifiedAt: time.Now().UTC()}, nil +} +func (s *Store) Get(ctx context.Context, key string) (storage.Object, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.Object{}, err + } + r, err := s.client.GetObject(ctx, &sdk.GetObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key)}) + if err != nil { + return storage.Object{}, mapError(err) + } + if r.ContentLength < 0 || r.ContentLength > s.maxBytes { + r.Body.Close() + return storage.Object{}, storage.ErrTooLarge + } + info := storage.Info{Key: key, Size: r.ContentLength, ContentType: sdk.ToString(r.ContentType)} + if r.LastModified != nil { + info.ModifiedAt = *r.LastModified + } + return storage.Object{Info: info, Body: r.Body}, nil +} +func (s *Store) Delete(ctx context.Context, key string) error { + if err := storage.Check(ctx, key); err != nil { + return err + } + _, err := s.client.DeleteObject(ctx, &sdk.DeleteObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key)}) + err = mapError(err) + if errors.Is(err, storage.ErrNotFound) { + return nil + } + return err +} +func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (*url.URL, error) { + if err := storage.Check(ctx, key); err != nil { + return nil, err + } + if ttl < time.Second || ttl > time.Hour { + return nil, errors.New("storage/oss: TTL must be 1s..1h") + } + r, err := s.client.Presign(ctx, &sdk.GetObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ResponseContentDisposition: sdk.Ptr("attachment")}, sdk.PresignExpires(ttl)) + if err != nil { + return nil, err + } + return url.Parse(r.URL) +} +func mapError(err error) error { + var serviceError *sdk.ServiceError + if errors.As(err, &serviceError) && serviceError.Code == "NoSuchKey" { + return storage.ErrNotFound + } + return err +} + +var _ storage.Store = (*Store)(nil) +var _ storage.Presigner = (*Store)(nil) diff --git a/storage/oss/oss_test.go b/storage/oss/oss_test.go new file mode 100644 index 0000000..1217528 --- /dev/null +++ b/storage/oss/oss_test.go @@ -0,0 +1,31 @@ +package oss + +import ( + "context" + "git.yuntuo.hk/k2safe/go-omni/internal/storagefixture" + "git.yuntuo.hk/k2safe/go-omni/storage/storagetest" + sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss" + "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestSDKProtocolContract(t *testing.T) { + server := httptest.NewTLSServer(storagefixture.Handler()) + defer server.Close() + cfg := sdk.LoadDefaultConfig().WithRegion("cn-hangzhou").WithEndpoint(server.URL).WithUsePathStyle(true).WithCredentialsProvider(credentials.NewStaticCredentialsProvider("fixtureAccess", "fixtureSecret")).WithHttpClient(server.Client()) + store, err := New(sdk.NewClient(cfg), "testbucket", 64) + if err != nil { + t.Fatal(err) + } + storagetest.Run(t, store) + u, err := store.PresignGet(context.Background(), "a/file.txt", time.Minute) + if err != nil || u.Query().Get("x-oss-signature") == "" || !strings.Contains(u.Path, "a/file.txt") { + t.Fatal("invalid signed URL", err) + } + if _, err = store.PresignGet(context.Background(), "a", 2*time.Hour); err == nil { + t.Fatal("TTL accepted") + } +} diff --git a/storage/s3/config.go b/storage/s3/config.go new file mode 100644 index 0000000..5e407cb --- /dev/null +++ b/storage/s3/config.go @@ -0,0 +1,42 @@ +package s3 + +import ( + "errors" + "net/http" + "regexp" + + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" +) + +// Open creates a store using an injected credential provider and transport. +// The application owns transport shutdown; no global HTTP defaults are changed. +type Config struct { + Endpoint string `json:"endpoint"` // Host[:port], without scheme or path. + Region string `json:"region"` + Bucket string `json:"bucket"` + Secure bool `json:"secure"` + MaxBytes int64 `json:"maxBytes"` + Credentials *credentials.Credentials `json:"-"` + Transport http.RoundTripper `json:"-"` +} + +func Open(cfg Config) (*Store, error) { + if cfg.Credentials == nil { + return nil, errors.New("storage/s3: credentials provider required") + } + client, err := minio.New(cfg.Endpoint, &minio.Options{Creds: cfg.Credentials, Secure: cfg.Secure, Region: cfg.Region, Transport: cfg.Transport, BucketLookup: minio.BucketLookupPath}) + if err != nil { + return nil, err + } + return New(client, cfg.Bucket, cfg.MaxBytes) +} + +// R2Config selects Cloudflare's standard S3 API endpoint. Jurisdiction-specific +// endpoints can be set explicitly with Config. R2 implements an S3 subset. +func R2Config(accountID, bucket string, creds *credentials.Credentials) (Config, error) { + if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(accountID) { + return Config{}, errors.New("storage/s3: invalid R2 account ID") + } + return Config{Endpoint: accountID + ".r2.cloudflarestorage.com", Region: "auto", Bucket: bucket, Secure: true, Credentials: creds}, nil +} diff --git a/storage/s3/s3.go b/storage/s3/s3.go new file mode 100644 index 0000000..11a58a6 --- /dev/null +++ b/storage/s3/s3.go @@ -0,0 +1,93 @@ +// Package s3 adapts a caller-owned MinIO client to S3-compatible object storage. +package s3 + +import ( + "context" + "errors" + "git.yuntuo.hk/k2safe/go-omni/storage" + "github.com/minio/minio-go/v7" + "io" + "net/url" + "strings" + "time" +) + +type Store struct { + client *minio.Client + bucket string + maxBytes int64 +} + +func New(client *minio.Client, bucket string, maxBytes int64) (*Store, error) { + if client == nil || bucket == "" || maxBytes < 0 { + return nil, errors.New("storage/s3: client, bucket and nonnegative limit required") + } + if maxBytes == 0 { + maxBytes = 32 << 20 + } + return &Store{client: client, bucket: bucket, maxBytes: maxBytes}, nil +} +func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int64, contentType string) (storage.Info, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.Info{}, err + } + if reader == nil || size < 0 || len(contentType) > 256 || strings.ContainsAny(contentType, "\r\n") { + return storage.Info{}, errors.New("storage/s3: invalid upload") + } + if size > s.maxBytes { + return storage.Info{}, storage.ErrTooLarge + } + result, err := s.client.PutObject(ctx, s.bucket, key, storage.ContextReader{Context: ctx, Reader: reader}, size, minio.PutObjectOptions{ContentType: contentType}) + if err != nil { + return storage.Info{}, err + } + return storage.Info{Key: key, Size: result.Size, ContentType: contentType, ModifiedAt: time.Now().UTC()}, nil +} +func (s *Store) Get(ctx context.Context, key string) (storage.Object, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.Object{}, err + } + object, err := s.client.GetObject(ctx, s.bucket, key, minio.GetObjectOptions{}) + if err != nil { + return storage.Object{}, mapError(err) + } + stat, err := object.Stat() + if err != nil { + object.Close() + return storage.Object{}, mapError(err) + } + if stat.Size > s.maxBytes { + object.Close() + return storage.Object{}, storage.ErrTooLarge + } + return storage.Object{Info: storage.Info{Key: key, Size: stat.Size, ContentType: stat.ContentType, ModifiedAt: stat.LastModified}, Body: object}, nil +} +func (s *Store) Delete(ctx context.Context, key string) error { + if err := storage.Check(ctx, key); err != nil { + return err + } + err := s.client.RemoveObject(ctx, s.bucket, key, minio.RemoveObjectOptions{}) + if errors.Is(mapError(err), storage.ErrNotFound) { + return nil + } + return err +} +func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (*url.URL, error) { + if err := storage.Check(ctx, key); err != nil { + return nil, err + } + if ttl < time.Second || ttl > time.Hour { + return nil, errors.New("storage/s3: TTL must be 1s..1h") + } + return s.client.PresignedGetObject(ctx, s.bucket, key, ttl, url.Values{"response-content-disposition": {"attachment"}}) +} +func mapError(err error) error { + if err == nil { + return nil + } + switch minio.ToErrorResponse(err).Code { + case "NoSuchKey", "NoSuchObject": + return storage.ErrNotFound + } + return err +} diff --git a/storage/s3/s3_test.go b/storage/s3/s3_test.go new file mode 100644 index 0000000..05fb010 --- /dev/null +++ b/storage/s3/s3_test.go @@ -0,0 +1,104 @@ +package s3 + +import ( + "context" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/internal/storagefixture" + "git.yuntuo.hk/k2safe/go-omni/storage/storagetest" + "github.com/minio/minio-go/v7" + "github.com/minio/minio-go/v7/pkg/credentials" + "io" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" + "time" +) + +func TestS3ProtocolAndR2Config(t *testing.T) { + server := httptest.NewTLSServer(storagefixture.Handler()) + defer server.Close() + creds := credentials.NewStaticV4("fixtureAccess", "fixtureSecret", "") + s, err := Open(Config{Endpoint: strings.TrimPrefix(server.URL, "https://"), Region: "us-east-1", Bucket: "testbucket", Secure: true, MaxBytes: 64, Credentials: creds, Transport: server.Client().Transport}) + if err != nil { + t.Fatal(err) + } + storagetest.Run(t, s) + cfg, err := R2Config(strings.Repeat("a", 32), "bucket", creds) + if err != nil { + t.Fatal(err) + } + r2, err := Open(cfg) + if err != nil { + t.Fatal(err) + } + u, err := r2.PresignGet(context.Background(), "path/file", time.Minute) + if err != nil || u.Host != strings.Repeat("a", 32)+".r2.cloudflarestorage.com" || u.Query().Get("X-Amz-Signature") == "" || !strings.Contains(u.Query().Get("X-Amz-Credential"), "/auto/s3/") { + t.Fatal("R2 presign settings", err) + } +} +func TestS3Integration(t *testing.T) { + address := os.Getenv("GO_OMNI_S3_TEST_ADDR") + if address == "" { + t.Skip("requires isolated MinIO") + } + ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second) + defer cancel() + ready := false + for n := 0; n < 100; n++ { + req, _ := http.NewRequestWithContext(ctx, "GET", "http://"+address+"/minio/health/ready", nil) + r, err := http.DefaultClient.Do(req) + if err == nil { + r.Body.Close() + if r.StatusCode == 200 { + ready = true + break + } + } + if ctx.Err() != nil { + break + } + time.Sleep(100 * time.Millisecond) + } + if !ready { + t.Fatal("isolated S3 server did not start") + } + client, err := minio.New(address, &minio.Options{Creds: credentials.NewStaticV4("goOmniTest", "goOmniTestSecret", ""), Region: "us-east-1"}) + if err != nil { + t.Fatal(err) + } + bucket := fmt.Sprintf("test-%d", time.Now().UnixNano()) + if err = client.MakeBucket(ctx, bucket, minio.MakeBucketOptions{}); err != nil { + t.Fatal(err) + } + defer client.RemoveBucket(context.Background(), bucket) + s, _ := New(client, bucket, 64) + storagetest.Run(t, s) + s.Put(ctx, "signed", strings.NewReader("hello"), 5, "text/plain") + defer s.Delete(context.Background(), "signed") + u, err := s.PresignGet(ctx, "signed", time.Minute) + if err != nil { + t.Fatal(err) + } + response, err := http.Get(u.String()) + if err != nil { + t.Fatal(err) + } + raw, _ := io.ReadAll(response.Body) + response.Body.Close() + if response.StatusCode != 200 || string(raw) != "hello" { + t.Fatal(response.StatusCode, string(raw)) + } + q := u.Query() + q.Set("X-Amz-Signature", strings.Repeat("0", 64)) + u.RawQuery = q.Encode() + response, err = http.Get(u.String()) + if err != nil { + t.Fatal(err) + } + response.Body.Close() + if response.StatusCode != 403 { + t.Fatal("tampered signature", response.StatusCode) + } +} diff --git a/storage/signed.go b/storage/signed.go new file mode 100644 index 0000000..d26d977 --- /dev/null +++ b/storage/signed.go @@ -0,0 +1,93 @@ +package storage + +import ( + "context" + "crypto/hmac" + "crypto/sha256" + "encoding/base64" + "errors" + "io" + "net/http" + "net/url" + "strconv" + "time" +) + +type Signer struct { + store Store + base *url.URL + secret []byte + now func() time.Time +} + +func NewSigner(store Store, baseURL string, secret []byte) (*Signer, error) { + base, err := url.Parse(baseURL) + if err != nil { + return nil, err + } + if store == nil || len(secret) < 32 || base.Host == "" || (base.Scheme != "https" && base.Scheme != "http") || base.RawQuery != "" || base.Fragment != "" || base.User != nil { + return nil, errors.New("storage: store, HTTP base URL and >=32-byte secret required") + } + return &Signer{store: store, base: base, secret: append([]byte(nil), secret...), now: time.Now}, nil +} +func (s *Signer) signature(key, expiry string) string { + mac := hmac.New(sha256.New, s.secret) + io.WriteString(mac, "GET\n"+key+"\n"+expiry) + return base64.RawURLEncoding.EncodeToString(mac.Sum(nil)) +} +func (s *Signer) PresignGet(ctx context.Context, key string, ttl time.Duration) (*url.URL, error) { + if err := Check(ctx, key); err != nil { + return nil, err + } + if ttl < time.Second || ttl > time.Hour { + return nil, errors.New("storage: TTL must be 1s..1h") + } + expiry := strconv.FormatInt(s.now().Add(ttl).Unix(), 10) + u := *s.base + u.RawQuery = url.Values{"key": {key}, "expires": {expiry}, "signature": {s.signature(key, expiry)}}.Encode() + return &u, nil +} + +// ServeHTTP serves signed downloads as attachments. Issue links only after +// authorization; possession of an unexpired URL grants access to its object. +func (s *Signer) ServeHTTP(w http.ResponseWriter, r *http.Request) { + if r.Method != "GET" && r.Method != "HEAD" { + w.Header().Set("Allow", "GET, HEAD") + http.Error(w, "method not allowed", 405) + return + } + q, err := url.ParseQuery(r.URL.RawQuery) + if err != nil || len(q) != 3 || len(q["key"]) != 1 || len(q["expires"]) != 1 || len(q["signature"]) != 1 { + http.Error(w, "invalid link", 403) + return + } + key, expiry := q.Get("key"), q.Get("expires") + expires, err := strconv.ParseInt(expiry, 10, 64) + now := s.now().Unix() + if err != nil || expires <= now || expires > now+3600 || Check(r.Context(), key) != nil || !hmac.Equal([]byte(q.Get("signature")), []byte(s.signature(key, expiry))) { + http.Error(w, "invalid or expired link", 403) + return + } + object, err := s.store.Get(r.Context(), key) + if err != nil { + status := 500 + if errors.Is(err, ErrNotFound) { + status = 404 + } + http.Error(w, "object unavailable", status) + return + } + defer object.Body.Close() + contentType := object.ContentType + if contentType == "" { + contentType = "application/octet-stream" + } + w.Header().Set("Content-Type", contentType) + w.Header().Set("Content-Disposition", "attachment") + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set("Cache-Control", "private, no-store") + w.Header().Set("Content-Length", strconv.FormatInt(object.Size, 10)) + if r.Method == "GET" { + _, _ = io.Copy(w, object.Body) + } +} diff --git a/storage/signed_test.go b/storage/signed_test.go new file mode 100644 index 0000000..4d1570e --- /dev/null +++ b/storage/signed_test.go @@ -0,0 +1,35 @@ +package storage_test + +import ( + "context" + "git.yuntuo.hk/k2safe/go-omni/storage" + "git.yuntuo.hk/k2safe/go-omni/storage/local" + "net/http/httptest" + "strings" + "testing" + "time" +) + +func TestLocalSignedDownload(t *testing.T) { + s, _ := local.New(t.TempDir(), 64) + s.Put(context.Background(), "file", strings.NewReader("hello"), 5, "text/plain") + signer, _ := storage.NewSigner(s, "https://example.test/download", []byte(strings.Repeat("x", 32))) + u, err := signer.PresignGet(context.Background(), "file", time.Minute) + if err != nil { + t.Fatal(err) + } + request := httptest.NewRequest("GET", u.String(), nil) + response := httptest.NewRecorder() + signer.ServeHTTP(response, request) + if response.Code != 200 || response.Body.String() != "hello" || response.Header().Get("Content-Disposition") != "attachment" { + t.Fatal(response) + } + q := u.Query() + q.Set("key", "another") + u.RawQuery = q.Encode() + response = httptest.NewRecorder() + signer.ServeHTTP(response, httptest.NewRequest("GET", u.String(), nil)) + if response.Code != 403 { + t.Fatal(response.Code) + } +} diff --git a/storage/storage.go b/storage/storage.go new file mode 100644 index 0000000..c7897a5 --- /dev/null +++ b/storage/storage.go @@ -0,0 +1,68 @@ +// Package storage defines streaming object storage; callers own returned bodies. +package storage + +import ( + "context" + "errors" + "io" + "net/url" + "strings" + "time" + "unicode" + "unicode/utf8" +) + +var ErrNotFound = errors.New("storage: object not found") +var ErrTooLarge = errors.New("storage: object exceeds limit") + +type Info struct { + Key string `json:"key"` + Size int64 `json:"size"` + ContentType string `json:"contentType"` + ModifiedAt time.Time `json:"modifiedAt"` +} +type Object struct { + Info + Body io.ReadCloser +} +type Store interface { + Put(context.Context, string, io.Reader, int64, string) (Info, error) + Get(context.Context, string) (Object, error) + Delete(context.Context, string) error +} + +// Presigner is an optional capability. URLs grant temporary read access and +// must not be logged. Providers may expire them earlier when credentials expire. +type Presigner interface { + PresignGet(context.Context, string, time.Duration) (*url.URL, error) +} + +func Check(ctx context.Context, key string) error { + if ctx == nil { + return errors.New("storage: nil context") + } + if err := ctx.Err(); err != nil { + return err + } + if key == "" || len(key) > 1024 || !utf8.ValidString(key) || strings.Contains(key, "\\") || strings.IndexFunc(key, unicode.IsControl) >= 0 { + return errors.New("storage: invalid key") + } + for _, part := range strings.Split(key, "/") { + if part == "" || part == "." || part == ".." { + return errors.New("storage: invalid key path") + } + } + return nil +} + +type ContextReader struct { + Context context.Context + Reader io.Reader +} + +func (r ContextReader) Read(p []byte) (int, error) { + if err := r.Context.Err(); err != nil { + return 0, err + } + return r.Reader.Read(p) +} diff --git a/storage/storagetest/contract.go b/storage/storagetest/contract.go new file mode 100644 index 0000000..63d16c7 --- /dev/null +++ b/storage/storagetest/contract.go @@ -0,0 +1,59 @@ +// Package storagetest provides shared adapter checks. Use a dedicated test +// bucket/root and a Store configured with a 64-byte object limit. +package storagetest + +import ( + "context" + "errors" + "git.yuntuo.hk/k2safe/go-omni/storage" + "io" + "strings" + "testing" +) + +func Run(t *testing.T, store storage.Store) { + t.Helper() + ctx := context.Background() + key := "contract/文件 +%.txt" + defer store.Delete(ctx, key) + if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) { + t.Fatal("missing object", err) + } + for _, body := range []string{"original", "replacement"} { + info, err := store.Put(ctx, key, strings.NewReader(body), int64(len(body)), "text/plain") + if err != nil || info.Size != int64(len(body)) { + t.Fatal(info, err) + } + object, err := store.Get(ctx, key) + if err != nil { + t.Fatal(err) + } + data, err := io.ReadAll(object.Body) + object.Body.Close() + if err != nil || string(data) != body || object.ContentType != "text/plain" { + t.Fatal(string(data), object.Info, err) + } + } + if _, err := store.Put(ctx, "tooBig", strings.NewReader(strings.Repeat("a", 65)), 65, "text/plain"); !errors.Is(err, storage.ErrTooLarge) { + t.Fatal(err) + } + for _, key := range []string{"../bad", "a/../b", "/absolute", "a//b", "a\\b"} { + if _, err := store.Put(ctx, key, strings.NewReader("a"), 1, "text/plain"); err == nil { + t.Fatal(key) + } + } + canceled, cancel := context.WithCancel(ctx) + cancel() + if _, err := store.Get(canceled, key); !errors.Is(err, context.Canceled) { + t.Fatal(err) + } + if err := store.Delete(ctx, key); err != nil { + t.Fatal(err) + } + if err := store.Delete(ctx, key); err != nil { + t.Fatal(err) + } + if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) { + t.Fatal(err) + } +} diff --git a/telemetry/adapters.go b/telemetry/adapters.go new file mode 100644 index 0000000..2ae8e35 --- /dev/null +++ b/telemetry/adapters.go @@ -0,0 +1,78 @@ +package telemetry + +import ( + "context" + "database/sql" + "github.com/redis/go-redis/v9" + "go.opentelemetry.io/otel/codes" + "go.opentelemetry.io/otel/trace" + "io" + "net" + "sync" +) + +type spanBody struct { + io.ReadCloser + span trace.Span + once sync.Once +} + +func (b *spanBody) finish(err error) { + b.once.Do(func() { + if err != nil && err != io.EOF { + b.span.SetStatus(codes.Error, "body read failed") + } + b.span.End() + }) +} +func (b *spanBody) Read(p []byte) (int, error) { + n, err := b.ReadCloser.Read(p) + if err != nil { + b.finish(err) + } + return n, err +} +func (b *spanBody) Close() error { err := b.ReadCloser.Close(); b.finish(err); return err } + +type RedisHook struct{ Runtime *Runtime } + +func (h RedisHook) DialHook(next redis.DialHook) redis.DialHook { + return func(ctx context.Context, network, addr string) (conn net.Conn, err error) { + err = h.Runtime.Do(ctx, "redis.connect", func(ctx context.Context) error { conn, err = next(ctx, network, addr); return err }) + return + } +} +func (h RedisHook) ProcessHook(next redis.ProcessHook) redis.ProcessHook { + return func(ctx context.Context, cmd redis.Cmder) error { + var result error + _ = h.Runtime.Do(ctx, "redis.command", func(ctx context.Context) error { + result = next(ctx, cmd) + if result == redis.Nil { + return nil + } + return result + }) + return result + } +} +func (h RedisHook) ProcessPipelineHook(next redis.ProcessPipelineHook) redis.ProcessPipelineHook { + return func(ctx context.Context, cmds []redis.Cmder) error { + return h.Runtime.Do(ctx, "redis.pipeline", func(ctx context.Context) error { return next(ctx, cmds) }) + } +} + +// SQL records execution spans without SQL text or arguments. Query spans cover +// obtaining rows; iteration/Scan remain caller-owned. Use Do around transactions. +type SQL struct { + DB *sql.DB + Runtime *Runtime +} + +func (d SQL) ExecContext(ctx context.Context, query string, args ...any) (result sql.Result, err error) { + err = d.Runtime.Do(ctx, "db.exec", func(ctx context.Context) error { result, err = d.DB.ExecContext(ctx, query, args...); return err }) + return +} +func (d SQL) QueryContext(ctx context.Context, query string, args ...any) (rows *sql.Rows, err error) { + err = d.Runtime.Do(ctx, "db.query", func(ctx context.Context) error { rows, err = d.DB.QueryContext(ctx, query, args...); return err }) + return +} diff --git a/telemetry/telemetry.go b/telemetry/telemetry.go new file mode 100644 index 0000000..b6ca4f3 --- /dev/null +++ b/telemetry/telemetry.go @@ -0,0 +1,187 @@ +// Package telemetry adds opt-in OpenTelemetry without process-global providers. +package telemetry + +import ( + "context" + "errors" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/codes" + "go.opentelemetry.io/otel/metric" + "go.opentelemetry.io/otel/propagation" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" + sdktrace "go.opentelemetry.io/otel/sdk/trace" + "go.opentelemetry.io/otel/trace" + "math" + "net/http" + "time" +) + +type Config struct { + Service string `json:"service"` + SampleRatio float64 `json:"sampleRatio"` + Exporter sdktrace.SpanExporter `json:"-"` + Reader sdkmetric.Reader `json:"-"` +} +type Runtime struct { + traces *sdktrace.TracerProvider + meters *sdkmetric.MeterProvider + tracer trace.Tracer + propagator propagation.TextMapPropagator + duration metric.Float64Histogram + requests metric.Int64Counter +} + +func New(cfg Config) (*Runtime, error) { + if cfg.Service == "" || math.IsNaN(cfg.SampleRatio) || cfg.SampleRatio < 0 || cfg.SampleRatio > 1 { + return nil, errors.New("telemetry: service and sampleRatio in 0..1 required") + } + res := resource.NewSchemaless(attribute.String("service.name", cfg.Service)) + traceOptions := []sdktrace.TracerProviderOption{sdktrace.WithResource(res), sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(cfg.SampleRatio)))} + if cfg.Exporter != nil { + traceOptions = append(traceOptions, sdktrace.WithBatcher(cfg.Exporter, sdktrace.WithMaxQueueSize(512), sdktrace.WithMaxExportBatchSize(128), sdktrace.WithExportTimeout(3*time.Second))) + } + metricOptions := []sdkmetric.Option{sdkmetric.WithResource(res)} + if cfg.Reader != nil { + metricOptions = append(metricOptions, sdkmetric.WithReader(cfg.Reader)) + } + r := &Runtime{traces: sdktrace.NewTracerProvider(traceOptions...), meters: sdkmetric.NewMeterProvider(metricOptions...), propagator: propagation.TraceContext{}} + r.tracer = r.traces.Tracer("git.yuntuo.hk/k2safe/go-omni") + meter := r.meters.Meter("git.yuntuo.hk/k2safe/go-omni") + r.duration, _ = meter.Float64Histogram("http.server.request.duration", metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10)) + r.requests, _ = meter.Int64Counter("http.server.requests") + return r, nil +} +func (r *Runtime) Start(ctx context.Context, name string, options ...trace.SpanStartOption) (context.Context, trace.Span) { + ctx, span := r.tracer.Start(ctx, name, options...) + ctx = logx.WithTraceID(ctx, span.SpanContext().TraceID().String()) + return ctx, span +} +func (r *Runtime) Do(ctx context.Context, name string, work func(context.Context) error) (err error) { + if ctx == nil || work == nil || name == "" { + return errors.New("telemetry: context, operation name and work required") + } + ctx, span := r.Start(ctx, name) + defer span.End() + defer func() { + if p := recover(); p != nil { + span.SetStatus(codes.Error, "panic") + panic(p) + } + }() + err = work(ctx) + if err != nil { + span.SetStatus(codes.Error, "operation failed") + } + return err +} +func (r *Runtime) Middleware() httpx.Middleware { + return func(next httpx.Handler) httpx.Handler { + return func(c *httpx.Context) (err error) { + parent := c.Context() + ctx := r.propagator.Extract(parent, propagation.HeaderCarrier(c.Request().Header)) + route := httpx.RoutePattern(parent) + if route == "" { + route = "unmatched" + } + method := c.Request().Method + switch method { + case "GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS", "CONNECT", "TRACE": + default: + method = "OTHER" + } + ctx, span := r.Start(ctx, method+" "+route, trace.WithSpanKind(trace.SpanKindServer)) + c.WithContext(ctx) + started := time.Now() + defer func() { + status := c.Status() + if err != nil && !c.Responded() { + status = apierrors.From(err).Status + } + p := recover() + if p != nil { + status = 500 + } + attrs := []attribute.KeyValue{attribute.String("http.request.method", method), attribute.String("http.route", route), attribute.Int("http.response.status_code", status)} + span.SetAttributes(attrs...) + if status >= 500 { + span.SetStatus(codes.Error, "request failed") + } + r.requests.Add(ctx, 1, metric.WithAttributes(attrs...)) + r.duration.Record(ctx, time.Since(started).Seconds(), metric.WithAttributes(attrs...)) + span.End() + // Preserve correlation for outer access-log middleware without + // exposing an ended span as its active span. + c.WithContext(logx.WithTraceID(parent, span.SpanContext().TraceID().String())) + if p != nil { + panic(p) + } + }() + return next(c) + } + } +} +func (r *Runtime) Shutdown(ctx context.Context) error { + return errors.Join(r.traces.Shutdown(ctx), r.meters.Shutdown(ctx)) +} +func (r *Runtime) Close() error { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + return r.Shutdown(ctx) +} +func (r *Runtime) Flush(ctx context.Context) error { + return errors.Join(r.traces.ForceFlush(ctx), r.meters.ForceFlush(ctx)) +} + +// Transport borrows Base. Headers are injected on a clone, and the span lives +// until EOF/Close. URL paths, query strings and credentials are never recorded. +type Transport struct { + Runtime *Runtime + Base http.RoundTripper +} + +func (t Transport) RoundTrip(request *http.Request) (*http.Response, error) { + if t.Runtime == nil { + return nil, errors.New("telemetry: runtime required") + } + ctx, span := t.Runtime.Start(request.Context(), "http.client", trace.WithSpanKind(trace.SpanKindClient)) + defer func() { + if p := recover(); p != nil { + span.SetStatus(codes.Error, "panic") + span.End() + panic(p) + } + }() + req := request.Clone(ctx) + if req.Header == nil { + req.Header = make(http.Header) + } + t.Runtime.propagator.Inject(ctx, propagation.HeaderCarrier(req.Header)) + base := t.Base + if base == nil { + base = http.DefaultTransport + } + response, err := base.RoundTrip(req) + if err != nil { + span.SetStatus(codes.Error, "transport failed") + span.End() + if response != nil && response.Body != nil { + response.Body.Close() + } + return nil, err + } + if response == nil || response.Body == nil { + span.SetStatus(codes.Error, "invalid response") + span.End() + return nil, errors.New("telemetry: invalid transport response") + } + span.SetAttributes(attribute.Int("http.response.status_code", response.StatusCode)) + if response.StatusCode >= 500 { + span.SetStatus(codes.Error, "upstream failed") + } + response.Body = &spanBody{ReadCloser: response.Body, span: span} + return response, nil +} diff --git a/telemetry/telemetry_test.go b/telemetry/telemetry_test.go new file mode 100644 index 0000000..8efdafc --- /dev/null +++ b/telemetry/telemetry_test.go @@ -0,0 +1,117 @@ +package telemetry_test + +import ( + "context" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" + "git.yuntuo.hk/k2safe/go-omni/telemetry" + "github.com/redis/go-redis/v9" + sdkmetric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/metric/metricdata" + "go.opentelemetry.io/otel/sdk/trace/tracetest" + "go.opentelemetry.io/otel/trace" + "io" + "net/http" + "net/http/httptest" + "strings" + "testing" +) + +func TestHTTPTracePropagationMetricsAndRedisNil(t *testing.T) { + exporter := tracetest.NewInMemoryExporter() + reader := sdkmetric.NewManualReader() + runtime, err := telemetry.New(telemetry.Config{Service: "test", SampleRatio: 1, Exporter: exporter, Reader: reader}) + if err != nil { + t.Fatal(err) + } + defer runtime.Close() + ctx := context.Background() + upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if !strings.HasPrefix(r.Header.Get("Traceparent"), "00-11111111111111111111111111111111-") { + t.Error("missing propagated trace") + } + io.WriteString(w, "ok") + })) + defer upstream.Close() + app := omni.New(omni.WithLogger(false)) + app.Use(func(next httpx.Handler) httpx.Handler { + return func(c *httpx.Context) error { + err := next(c) + if logx.TraceID(c.Context()) != "11111111111111111111111111111111" { + t.Error("outer access log lost trace correlation") + } + return err + } + }) + app.Use(runtime.Middleware()) + app.GET("/users/:id", func(c *httpx.Context) error { + if logx.TraceID(c.Context()) != "11111111111111111111111111111111" { + t.Error("log correlation") + } + req, _ := http.NewRequestWithContext(c.Context(), "GET", upstream.URL+"?secret=hidden", nil) + resp, err := (telemetry.Transport{Runtime: runtime}).RoundTrip(req) + if err != nil { + return err + } + io.ReadAll(resp.Body) + resp.Body.Close() + if req.Header.Get("Traceparent") != "" { + t.Error("caller headers mutated") + } + return c.OK("ok") + }) + req := httptest.NewRequest("GET", "/users/privateUser?secret=hidden", nil) + req.Header.Set("Traceparent", "00-11111111111111111111111111111111-2222222222222222-01") + rec := httptest.NewRecorder() + app.ServeHTTP(rec, req) + if rec.Code != 200 { + t.Fatal(rec.Code) + } + hook := telemetry.RedisHook{Runtime: runtime} + if err = hook.ProcessHook(func(context.Context, redis.Cmder) error { return redis.Nil })(ctx, redis.NewStringCmd(ctx, "GET", "secret")); err != redis.Nil { + t.Fatal("cache miss was swallowed", err) + } + if err = runtime.Flush(ctx); err != nil { + t.Fatal(err) + } + spans := exporter.GetSpans() + if len(spans) != 3 { + t.Fatalf("got %d spans", len(spans)) + } + var client, server bool + for _, s := range spans { + if strings.Contains(s.Name, "privateUser") || strings.Contains(s.Name, "secret") { + t.Fatal(s.Name) + } + if s.SpanKind == trace.SpanKindServer { + server = true + if s.Name != "GET /users/:id" || s.Parent.SpanID().String() != "2222222222222222" { + t.Fatal(s.Name, s.Parent) + } + } + if s.SpanKind == trace.SpanKindClient { + client = true + } + } + if !client || !server { + t.Fatal("missing client/server span") + } + var metrics metricdata.ResourceMetrics + if err = reader.Collect(ctx, &metrics); err != nil { + t.Fatal(err) + } + var count int64 + for _, scope := range metrics.ScopeMetrics { + for _, m := range scope.Metrics { + if sum, ok := m.Data.(metricdata.Sum[int64]); ok { + for _, p := range sum.DataPoints { + count += p.Value + } + } + } + } + if count != 1 { + t.Fatal("request metric", count) + } +}