Compare commits
2 Commits
f91034477b
...
04a1f7999d
| Author | SHA1 | Date | |
|---|---|---|---|
| 04a1f7999d | |||
| 208a9c3fbd |
149
ADVANCED.md
Normal file
149
ADVANCED.md
Normal file
@@ -0,0 +1,149 @@
|
||||
# 扩展业务模块
|
||||
|
||||
本批提供八类扩展,以及分页、审计和显式数据库迁移。模块按需导入;`omni.New` 与业务 Handler 不绑定存储、队列或网络实现。JSON、配置和新示例使用 lowerCamelCase;Go 导出标识符和 `_test.go` 保持语言约定。
|
||||
|
||||
按本次范围决定,**不集成 Orbit,不引入 Orbit SDK,也不实现工作流编排**。通用消息队列与 Orbit 的持久流程、执行授权、结果订阅和人工恢复不是等价能力。业务项目以后需要 Orbit,可独立依赖它的 SDK;当前没有承诺只换队列适配器就能迁移其全部语义。
|
||||
|
||||
## 能力与边界
|
||||
|
||||
| 模块 | 本批实现 | 明确边界 |
|
||||
| --- | --- | --- |
|
||||
| `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI |
|
||||
| `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 |
|
||||
| `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider |
|
||||
| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 |
|
||||
| `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 |
|
||||
| `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 |
|
||||
| `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 |
|
||||
| `storage` | Local、S3 兼容(含 R2 配置)、OSS 官方 SDK 适配、临时下载链接 | 云平台专属功能单独扩展;本批没有云账号实网验收 |
|
||||
| `database/migrate` | MySQL 显式部署迁移、摘要预览、失败 dirty 阻断 | 不随应用启动自动改表;无自动 down/force;已应用文件摘要需部署方留存校对 |
|
||||
| `audit` | 可注入 Sink、字段白名单、JSON 行串行输出、错误返回 | 不是防篡改存储;与业务同事务的审计要实现事务型 Sink |
|
||||
| `pagination` | page/pageSize、最大偏移与页长、统一结果 | offset 分页,不承诺跨页快照,不实现游标分页 |
|
||||
|
||||
## 对象存储
|
||||
|
||||
业务只持有 `storage.Store`,调用 `Put`、`Get`、`Delete`。构造函数属于启动组合处:
|
||||
|
||||
```go
|
||||
type FileService struct { Files storage.Store }
|
||||
|
||||
// Local:私有目录,流式临时文件 + 原子替换。
|
||||
files, err := local.New("/srv/app/objects", 32<<20)
|
||||
|
||||
// S3:AWS/MinIO 或其他经过契约测试的 S3 兼容服务。
|
||||
files, err := s3.Open(s3.Config{
|
||||
Endpoint: "s3.us-east-1.amazonaws.com", Region: "us-east-1",
|
||||
Bucket: "example-bucket", Secure: true, MaxBytes: 32<<20,
|
||||
Credentials: credentialProvider, Transport: transport,
|
||||
})
|
||||
|
||||
// Cloudflare R2:复用 S3 适配器,签名区域为 auto。
|
||||
cfg, err := s3.R2Config(accountID, bucket, credentialProvider)
|
||||
cfg.Transport = transport
|
||||
files, err := s3.Open(cfg)
|
||||
|
||||
// OSS:单独适配官方 Go SDK v2,保留平台签名与凭据提供器。
|
||||
client := ossSdk.NewClient(ossSdk.LoadDefaultConfig().
|
||||
WithRegion("cn-hangzhou").
|
||||
WithCredentialsProvider(ossCredentials.NewEnvironmentVariableCredentialsProvider()))
|
||||
files, err := oss.New(client, bucket, 32<<20)
|
||||
```
|
||||
|
||||
上方是互斥初始化片段,实际代码需逐次处理错误。import 路径分别为 `go-omni/storage/local`、`go-omni/storage/s3`、`go-omni/storage/oss`(完整 module 前缀 `git.yuntuo.hk/k2safe/`);`ossSdk` 与 `ossCredentials` 来自 `github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss` 及其 `credentials` 子包。S3 凭据来自 `github.com/minio/minio-go/v7/pkg/credentials`。
|
||||
|
||||
统一约定:key 是对象标识,不是磁盘路径;拒绝绝对路径、`..`、空路径段和反斜杠。Put 必须传已知长度,调用方保证与内容一致。默认单对象 32 MiB,流式读写;OSS 本适配器只做 simple upload,最大配置 5 GiB。Get 返回的 `Body` 由调用方关闭,读取过程也应有 context 截止。Delete 缺失对象成功;Get 缺失对象返回 `storage.ErrNotFound`。底层客户端/传输连接池由应用管理,不会在单次操作后关闭。
|
||||
|
||||
业务需要临时链接时显式依赖额外接口 `storage.Presigner`。S3/OSS 实现 `PresignGet`;Local 用 `storage.NewSigner(files, downloadBaseURL, secret)` 提供 HTTP 下载 Handler。链接 TTL 为 1 秒到 1 小时,私钥/临时凭据可能让云链接提前失效。签发链接前必须完成业务授权;对象 key 及 bucket 本身不是租户权限。签名链接不要写日志。首版未封装前端直传、分片续传、生命周期、ACL、CDN 自定义域名和版本管理。
|
||||
|
||||
R2 实现 S3 的子集,配置入口使用标准账户 endpoint;法域专属 endpoint 可显式配置。兼容性依据:[R2 S3 API](https://developers.cloudflare.com/r2/api/s3/api/)、[R2 签名链接](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)。OSS 使用其[官方 Go SDK v2](https://www.alibabacloud.com/help/en/oss/developer-reference/manual-for-go-sdk-v2/),没有假定全部 S3 行为与 OSS 等价。
|
||||
|
||||
验证区分:Local 契约、OSS SDK HTTP fixture、S3 SDK fixture 和 R2 签名参数属于本地测试;Linux runner 再跑真实 MinIO 上传/读取/删除与签名拒绝。**MinIO 通过不能代替 OSS/R2 实网验证。**迁移存储平台还需要复制已有对象和调整部署权限,不只是切配置。
|
||||
|
||||
## 可靠队列
|
||||
|
||||
```go
|
||||
backend, err := queueRedis.New(redisClient, queueRedis.Config{
|
||||
Namespace: "serviceA:jobs:v1", MaxRecords: 1000,
|
||||
MaxPayloadBytes: 64 << 10, DedupTTL: 24*time.Hour,
|
||||
})
|
||||
consumer, err := queueRedis.NewConsumer(backend, queueRedis.ConsumerConfig{
|
||||
Workers: 4, MaxAttempts: 3,
|
||||
TaskTimeout: 30*time.Second, VisibilityTimeout: time.Minute,
|
||||
})
|
||||
|
||||
// ID 与内容在业务端持久化;超时后复用原 ID 和原内容。
|
||||
receipt, err := backend.Publish(ctx, queue.Message{
|
||||
ID: operationID, Name: "generateReport", Payload: payload,
|
||||
})
|
||||
|
||||
// 在服务生命周期中运行;返回错误须让监控/进程管理感知。
|
||||
err = consumer.Run(serviceContext, func(ctx context.Context, item queue.Delivery) error {
|
||||
return generateReportIdempotently(ctx, item.ID, item.Payload)
|
||||
})
|
||||
```
|
||||
|
||||
业务依赖 `queue.Publisher` / `queue.Consumer`,Redis 类型留在组合处。receipt 表示受理,不代表业务成功。普通 error/panic 按配置重试,`queue.Reject(code)` 保留为失败消息;超过次数同样保留。`DeadLetters(after, limit)` 是有界分页,`DiscardDead` 仅在人工核对后删除匹配的失败消息,并继续保留一段提交去重窗口。错误正文不写入失败记录。需要再次处理时由业务决定新操作身份及去重策略,框架不自动无限重投。
|
||||
|
||||
一个 namespace 使用一个受管 consumer group;多个 consumer 可竞争消费同一队列。配置一致性检查拒绝不同的容量/去重策略和执行超时/重试策略混用。确认检查当前 pending owner,旧执行不能确认新执行;但不能阻止旧处理器已发出的外部副作用,因此仍必须业务幂等。处理器需响应 context;Go 无法强杀不配合的函数,超过 shutdownTimeout 返回错误时,最多 Workers 个处理器可能尚未退出。
|
||||
|
||||
容量包含待处理、失败及未过期的已完成身份,满额拒绝新 ID,保留原 ID 重报回执。完成/人工丢弃后的身份按 DedupTTL 过期;未完成/失败消息不自动清除,不用 MAXLEN 静默裁掉未消费消息。Redis 需启用合适的 AOF/复制策略与 noeviction;本批验证的是投递/接管协议,未证明 Redis 主从故障切换后零丢失。异常进程留下的无 pending consumer 元数据可在运维核实后清理,不能用 DELCONSUMER 丢掉尚未接管的 pending。
|
||||
|
||||
`task.Runner` 仍适用于本进程短任务和定时轮询,重启会丢失;需要持久交付时使用 `queue`。这里没有分布式 Cron、业务工作流、事务 Outbox 或 exactly-once 承诺。
|
||||
|
||||
## 认证与授权
|
||||
|
||||
`auth.NewJWT` 要求固定 issuer/audience 和可信 kid→公钥列表;只开放 RS256 / EdDSA,必须有 exp/sub,检查有效期、nbf 与 iat。`ReplaceKeys` 原子替换列表,轮换期可同时保留新旧 kid。JWT 签发、私钥托管、登录口令验证留给身份服务。
|
||||
|
||||
`auth.NewSessions(cacheStore, "sessions:", time.Hour)` 签发随机 opaque bearer token;存储仅使用 token 的 SHA-256 键,可 `Revoke`。Redis 缓存可共享 Session,内存缓存只在单实例生效且可能因容量淘汰提前登出。不要把 Session token 当普通缓存值写日志。
|
||||
|
||||
`RequireAuth(authenticator)` 在前,`RequirePermission(policy, action, resource)` 在后。Policy 接收已验证 Principal,业务自行校验租户、角色、资源归属。示例 `examples/platform` 验证了无身份 401、角色不足 403 和授权写入。
|
||||
|
||||
## 配置与日志热更新
|
||||
|
||||
`config.LoadFile(path, prefix)` 按默认值→文件→非空环境变量加载并验证。两种文件格式都使用 JSON tag 的 lowerCamelCase,duration 可以写 `"5s"`;文件上限 1 MiB,拒绝未知字段、多文档和非法值。自定义 prefix 要包含期望的尾部下划线,例如 `BILLING_`。空环境变量沿用文件值,不清空已配置字段。
|
||||
|
||||
```yaml
|
||||
address: '127.0.0.1:8080'
|
||||
transport: native
|
||||
shutdownTimeout: 10s
|
||||
log:
|
||||
level: info
|
||||
format: json
|
||||
```
|
||||
|
||||
先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, prefix, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。
|
||||
|
||||
## 熔断与观测
|
||||
|
||||
为每个上游创建 `resilience.New(Config{MaxConcurrent: 16, FailureThreshold: 5})`;通用调用用 `Do`,HTTP 用 `resilience.Transport{Base: transport, Breaker: breaker}`。HTTP 槽位覆盖响应体读取,必须关闭 body;网络错误/5xx 计失败,冷却后只放行一个探针。不要对效果未知的写操作自行换身份重试。
|
||||
|
||||
`telemetry.New` 接受 OTel `sdktrace.SpanExporter` 和 `sdkmetric.Reader`;可注入官方 OTLP exporter、PeriodicReader 或测试用 exporter,基础包不强制选择 collector 协议。`app.Use(runtime.Middleware())` 采集入站,`telemetry.Transport` 采集出站并传播 W3C trace context;Redis 使用 `client.AddHook(telemetry.RedisHook{Runtime: runtime})`,SQL 使用 `telemetry.SQL{DB: db, Runtime: runtime}`。业务操作/事务可用 `runtime.Do(ctx, "fixedOperationName", fn)` 包裹。
|
||||
|
||||
创建者负责在停止请求/Worker 后调用 `runtime.Shutdown(ctx)` 刷新并关闭 exporter/reader。指标按注册路由模板和固定方法分类;自动采集不记录 URL query、SQL 正文、Redis key、token 或业务载荷。业务自定义 span 名称同样应固定,避免用用户 ID 或对象 key。
|
||||
|
||||
## 分布式限流与幂等
|
||||
|
||||
`middleware/redis.NewRateLimiter(client, namespace, cfg)` 跨副本使用 Redis 时间与原子令牌桶;满额/存储失败返回 503,额度不足返回 429 和 Retry-After。`NewIdempotencyStore` 注入 `middleware.IdempotencyConfig.Store`;Scope 必须来自认证后的租户/用户,先认证再幂等。不同请求内容复用同一身份返回 409,已完成请求回放原响应和 requestId。
|
||||
|
||||
claim 在开始业务前持久保存,完成按 owner 原子检查。进程崩溃留下的 running claim 不自动按租约过期;返回 409 等待核查,`Resolve` 仅由运维核对后显式调用。已保存的完成/失败记录按配置 TTL 清理,TTL 过后不再提供业务去重保证。失败结果、含 Set-Cookie/no-store 的响应不用于成功回放。共享 Redis 需要 noeviction 和独立 namespace,数据丢失时不存在框架层面的重复执行保证。
|
||||
|
||||
## 迁移、审计与分页
|
||||
|
||||
迁移命名:`0001_createUsers.up.sql`,可附 `.down.sql`,但第一版 CLI 只执行 up。`go-omni migrate --dir migrations` 默认打印版本/方向/SHA-256 清单,不连接数据库;部署步骤显式传 `--apply` 才读取 `GO_OMNI_DB_DSN` 执行。迁移使用独立连接与数据库锁;MySQL DDL 可能部分提交,失败保留 dirty,禁止自动 force 或回滚。当前基于 golang-migrate,数据库版本表不保存每个历史文件校验和,部署系统应保留并校对预览清单。取消在当前有界 SQL 操作结束后停止,重试需新建 Runner。
|
||||
|
||||
审计:`audit.New(sink, "approvedField")` 建白名单,再 `Record(ctx, Event{ActorID, Action, Resource, Outcome, Fields})`;补入服务端时间、requestId/traceId,只保留明确允许的属性,Sink 失败原样返回。JSONSink 不保证与业务提交原子一致,需要这种语义时由业务在同一数据库事务内提供 Sink/Outbox。
|
||||
|
||||
分页:`pagination.Parse(request.URL.Query(), pagination.Config{})` 默认 page=1/pageSize=20,上限 100,最大 offset=1,000,000;拒绝重复参数、溢出和越界。`pagination.NewResult(request, items, total)` 输出 items/page/pageSize/total/totalPages,空列表为 `[]`。查询与租户过滤仍由 Repository 负责。
|
||||
|
||||
## 验证入口
|
||||
|
||||
```sh
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
python3 scripts/verifyModules.py --advanced --goproxy https://goproxy.cn,direct --output performance/advancedLinux.json
|
||||
```
|
||||
|
||||
Linux runner 顺序运行编译、Redis、MinIO、MySQL 四个容器,各限 2 CPU、2 GiB、256 pids;数据阶段无外网、无发布端口,不连接现有数据库。原始结果和源码摘要以生成的 JSON/TXT 为准,未提供云凭据的 OSS/R2 不标为实网通过。
|
||||
|
||||
本批已通过上述 Linux 验证,源代码提交、98 个文件摘要、服务版本和边界见 [验收记录](performance/advancedAcceptance.md)。
|
||||
@@ -1,5 +1,7 @@
|
||||
# 业务基础模块
|
||||
|
||||
第二批的共享 Redis 中间件、可靠队列、多平台存储及其他模块见 [扩展模块](ADVANCED.md)。下文的单进程约束指本页原有默认实现。
|
||||
|
||||
这批能力位于 main,尚未包含在 `v0.6.0-preview` 标签中。统一示例见
|
||||
[examples/modules](examples/modules/README.md)。业务入口继续使用 `omni.New`,中间件依赖
|
||||
`httpx`,不依赖具体服务端网络实现。各模块按需创建、注入 Service,再注册到应用关闭流程。
|
||||
|
||||
10
README.md
10
README.md
@@ -46,6 +46,13 @@ go-omni/
|
||||
├── cache/ # 有界内存缓存、Redis 适配与合并回源
|
||||
├── httpc/ # 出站 HTTP 连接池、超时和重试
|
||||
├── task/ # 有界后台任务与间隔调度
|
||||
├── queue/ # 持久队列接口与 Redis Streams 适配
|
||||
├── storage/ # Local、S3/R2、OSS 与临时下载链接
|
||||
├── openapi/ # 路由和 OpenAPI 3.0.3 契约
|
||||
├── telemetry/ # 可注入 OpenTelemetry 观测
|
||||
├── resilience/ # 熔断与上游并发隔离
|
||||
├── audit/ # 显式业务审计与可注入 Sink
|
||||
├── pagination/ # 有界分页和统一结果
|
||||
├── transport/nethttp/ # 标准库网络适配器
|
||||
├── transport/native/ # 自研 HTTP/1.1 服务端预览
|
||||
├── transport/http2/ # opt-in HTTP/2 实验适配器
|
||||
@@ -57,6 +64,9 @@ main 分支新增缓存、出站 HTTP、限流/幂等/超时中间件、后台
|
||||
使用方式、默认容量和进程内能力的适用范围见 [业务基础模块](MODULES.md),
|
||||
可以直接运行 [组合示例](examples/modules/README.md)。这些新增能力尚未包含在上面的历史版本标签中。
|
||||
|
||||
扩展模块还包括 JWT/Session、Policy、文件配置及安全热更新、Redis 共享中间件、持久队列、多平台对象存储、OpenAPI、OpenTelemetry、熔断、迁移、审计和分页。
|
||||
使用与边界见 [扩展模块](ADVANCED.md),可运行 [文件服务组合示例](examples/platform/README.md)。go-omni 不集成 Orbit;需要流程编排的业务项目独立接入。
|
||||
|
||||
## 快速开始
|
||||
|
||||
```go
|
||||
|
||||
@@ -81,4 +81,11 @@
|
||||
- 最终代码已通过 Linux amd64 全仓库 race/vet/build、独立生成项目 CRUD、真实 Redis 7.4.11 与三种网络层 TLS 组合测试;测试容器清理完成、原有 7 个业务容器仍运行。提交、源码摘要与完整输出见 [验收记录](performance/modulesAcceptance.md)。
|
||||
- 当前幂等、限流、任务调度和合并回源均为单进程能力;Redis 仅提供缓存存储,不代表已实现分布式去重、任务持久化或跨副本选主。eTax 指定业务模块接入依用户要求暂缓。
|
||||
|
||||
## 7. 扩展业务模块(本批)
|
||||
|
||||
- 新增 OpenAPI、JWT/Session/Policy、OpenTelemetry、文件配置与日志级别热更新、熔断/并发隔离、Redis 共享限流/幂等、Redis Streams 持久队列、Local/S3/R2/OSS 存储;补充迁移、审计和分页。
|
||||
- 业务依赖独立接口,云平台 SDK 和 Redis 留在适配器中;按本次范围决定不集成 Orbit,不实现工作流编排。
|
||||
- 已完成本地模块测试及 race 检查,Linux 全仓库 race/vet/build、生成项目 CRUD、三个 Runner 的 TLS 示例和独立 Redis/MinIO/MySQL 均通过。共享状态、重复提交、失败消息、未确认消息接管、旧确认拒绝、对象签名和迁移 dirty 阻断有对应证据,见 [扩展模块验收](performance/advancedAcceptance.md)。OSS/R2 尚无云账号实网验收。
|
||||
- 契约、使用示例与限制见 [扩展模块](ADVANCED.md);验证入口为 `scripts/verifyModules.py --advanced`,继续限制单个测试容器 2 CPU / 2 GiB / 256 pids。
|
||||
|
||||
以上阶段以验收结果判断完成。当前仍是基础框架试用阶段;MySQL、适配层契约和一次业务模块试用已完成,自研网络层已有可运行预览,稳定性与生产规模验证尚未完成。
|
||||
|
||||
106
audit/audit.go
Normal file
106
audit/audit.go
Normal file
@@ -0,0 +1,106 @@
|
||||
// Package audit records explicitly selected business events without request bodies.
|
||||
package audit
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"io"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Event struct {
|
||||
Time time.Time `json:"time"`
|
||||
ActorID string `json:"actorId"`
|
||||
TenantID string `json:"tenantId,omitempty"`
|
||||
Action string `json:"action"`
|
||||
Resource string `json:"resource"`
|
||||
Outcome string `json:"outcome"`
|
||||
RequestID string `json:"requestId,omitempty"`
|
||||
TraceID string `json:"traceId,omitempty"`
|
||||
Fields map[string]string `json:"fields,omitempty"`
|
||||
}
|
||||
type Sink interface {
|
||||
Write(context.Context, Event) error
|
||||
}
|
||||
type Recorder struct {
|
||||
sink Sink
|
||||
allowed map[string]bool
|
||||
}
|
||||
|
||||
func New(sink Sink, allowedFields ...string) (*Recorder, error) {
|
||||
if sink == nil || len(allowedFields) > 32 {
|
||||
return nil, errors.New("audit: sink and at most 32 fields required")
|
||||
}
|
||||
r := &Recorder{sink: sink, allowed: map[string]bool{}}
|
||||
for _, name := range allowedFields {
|
||||
if name == "" || len(name) > 64 {
|
||||
return nil, errors.New("audit: invalid field")
|
||||
}
|
||||
r.allowed[name] = true
|
||||
}
|
||||
return r, nil
|
||||
}
|
||||
func (r *Recorder) Record(ctx context.Context, event Event) error {
|
||||
if ctx == nil {
|
||||
return errors.New("audit: nil context")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if event.ActorID == "" || event.Action == "" || event.Resource == "" || (event.Outcome != "success" && event.Outcome != "failure" && event.Outcome != "denied") {
|
||||
return errors.New("audit: actor/action/resource and valid outcome required")
|
||||
}
|
||||
for _, value := range []string{event.ActorID, event.TenantID, event.Action, event.Resource} {
|
||||
if len(value) > 512 {
|
||||
return errors.New("audit: event field too large")
|
||||
}
|
||||
}
|
||||
event.Time = time.Now().UTC()
|
||||
event.RequestID = logx.RequestID(ctx)
|
||||
event.TraceID = logx.TraceID(ctx)
|
||||
filtered := map[string]string{}
|
||||
for name, value := range event.Fields {
|
||||
if r.allowed[name] {
|
||||
if len(value) > 1024 {
|
||||
return errors.New("audit: attribute too large")
|
||||
}
|
||||
filtered[name] = value
|
||||
}
|
||||
}
|
||||
event.Fields = filtered
|
||||
return r.sink.Write(ctx, event)
|
||||
}
|
||||
|
||||
// JSONSink borrows its writer and serializes complete JSON lines. Write errors
|
||||
// are returned; callers choose whether a failed audit must reject an operation.
|
||||
type JSONSink struct {
|
||||
mu sync.Mutex
|
||||
writer io.Writer
|
||||
}
|
||||
|
||||
func NewJSONSink(writer io.Writer) (*JSONSink, error) {
|
||||
if writer == nil {
|
||||
return nil, errors.New("audit: writer required")
|
||||
}
|
||||
return &JSONSink{writer: writer}, nil
|
||||
}
|
||||
func (s *JSONSink) Write(ctx context.Context, event Event) error {
|
||||
data, err := json.Marshal(event)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data = append(data, '\n')
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
n, err := s.writer.Write(data)
|
||||
if err == nil && n != len(data) {
|
||||
return io.ErrShortWrite
|
||||
}
|
||||
return err
|
||||
}
|
||||
50
audit/audit_test.go
Normal file
50
audit/audit_test.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package audit
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"io"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type failedWriter struct{}
|
||||
|
||||
func (failedWriter) Write([]byte) (int, error) { return 0, io.ErrClosedPipe }
|
||||
func TestAllowlistCorrelationAndWriteFailure(t *testing.T) {
|
||||
var buf bytes.Buffer
|
||||
sink, _ := NewJSONSink(&buf)
|
||||
recorder, _ := New(sink, "status")
|
||||
ctx := logx.WithRequestID(context.Background(), "r1")
|
||||
ctx = logx.WithTraceID(ctx, "t1")
|
||||
event := Event{ActorID: "u1", Action: "update", Resource: "object:1", Outcome: "success", Fields: map[string]string{"status": "ok", "password": "secret"}}
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 20; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if err := recorder.Record(ctx, event); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
decoder := json.NewDecoder(&buf)
|
||||
for i := 0; i < 20; i++ {
|
||||
var got Event
|
||||
if err := decoder.Decode(&got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Fields["password"] != "" || got.RequestID != "r1" || got.TraceID != "t1" || got.Time.IsZero() {
|
||||
t.Fatal(got)
|
||||
}
|
||||
}
|
||||
bad, _ := NewJSONSink(failedWriter{})
|
||||
recorder, _ = New(bad)
|
||||
if err := recorder.Record(ctx, event); !errors.Is(err, io.ErrClosedPipe) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
105
auth/adapters_test.go
Normal file
105
auth/adapters_test.go
Normal file
@@ -0,0 +1,105 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rand"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/cache"
|
||||
"github.com/golang-jwt/jwt/v5"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestJWTValidationAndRotation(t *testing.T) {
|
||||
pub, private, err := ed25519.GenerateKey(rand.Reader)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
j, err := NewJWT(JWTConfig{Issuer: "issuer", Audience: "api", Keys: map[string]JWTKey{"first": {Algorithm: "EdDSA", PublicKey: pub}}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
claims := JWTClaims{RegisteredClaims: jwt.RegisteredClaims{Issuer: "issuer", Subject: "user1", Audience: jwt.ClaimStrings{"api"}, ExpiresAt: jwt.NewNumericDate(time.Now().Add(time.Hour))}, TenantID: "tenant1", Roles: []string{"reader"}}
|
||||
sign := func(c JWTClaims, kid string, method jwt.SigningMethod, key any) string {
|
||||
t.Helper()
|
||||
token := jwt.NewWithClaims(method, c)
|
||||
token.Header["kid"] = kid
|
||||
raw, err := token.SignedString(key)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return raw
|
||||
}
|
||||
raw := sign(claims, "first", jwt.SigningMethodEdDSA, private)
|
||||
// Constructor clones caller-owned key bytes.
|
||||
pub[0] ^= 1
|
||||
p, err := j.Authenticate(context.Background(), raw)
|
||||
if err != nil || p.TenantID != "tenant1" || !p.HasRole("reader") {
|
||||
t.Fatalf("%+v %v", p, err)
|
||||
}
|
||||
for name, change := range map[string]func(*JWTClaims){"issuer": func(c *JWTClaims) { c.Issuer = "bad" }, "audience": func(c *JWTClaims) { c.Audience = jwt.ClaimStrings{"other"} }, "expired": func(c *JWTClaims) { c.ExpiresAt = jwt.NewNumericDate(time.Now().Add(-time.Hour)) }, "noExpiry": func(c *JWTClaims) { c.ExpiresAt = nil }, "future": func(c *JWTClaims) { c.NotBefore = jwt.NewNumericDate(time.Now().Add(time.Hour)) }, "subject": func(c *JWTClaims) { c.Subject = "" }} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
bad := claims
|
||||
change(&bad)
|
||||
if _, err = j.Authenticate(context.Background(), sign(bad, "first", jwt.SigningMethodEdDSA, private)); !errors.Is(err, ErrInvalidToken) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
}
|
||||
for _, invalid := range []string{sign(claims, "missing", jwt.SigningMethodEdDSA, private), sign(claims, "first", jwt.SigningMethodHS256, []byte("invalid secret")), strings.Repeat("x", 8193)} {
|
||||
if _, err = j.Authenticate(context.Background(), invalid); !errors.Is(err, ErrInvalidToken) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err = j.ReplaceKeys(map[string]JWTKey{"second": {Algorithm: "EdDSA", PublicKey: private.Public()}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = j.Authenticate(context.Background(), raw); !errors.Is(err, ErrInvalidToken) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = j.Authenticate(context.Background(), sign(claims, "second", jwt.SigningMethodEdDSA, private)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if _, err = j.Authenticate(ctx, raw); !errors.Is(err, context.Canceled) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
func TestSessionsRevokeAndIsolation(t *testing.T) {
|
||||
mem, _ := cache.NewMemory(cache.MemoryConfig{})
|
||||
defer mem.Close()
|
||||
s, err := NewSessions(mem, "session:", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
token, err := s.Issue(ctx, Principal{UserID: "u1", TenantID: "t1", Roles: []string{"reader"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
key, _ := s.key(token)
|
||||
if strings.Contains(key, token) {
|
||||
t.Fatal("plaintext session secret stored")
|
||||
}
|
||||
p, err := s.Authenticate(ctx, token)
|
||||
if err != nil || p.UserID != "u1" {
|
||||
t.Fatal(p, err)
|
||||
}
|
||||
p.Roles[0] = "admin"
|
||||
p, _ = s.Authenticate(ctx, token)
|
||||
if p.HasRole("admin") {
|
||||
t.Fatal("principal alias")
|
||||
}
|
||||
if err = s.Revoke(ctx, token); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = s.Authenticate(ctx, token); !errors.Is(err, ErrInvalidToken) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = s.Authenticate(ctx, "bad"); !errors.Is(err, ErrInvalidToken) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -21,9 +21,10 @@ var (
|
||||
// Principal is the authenticated identity made available to business handlers.
|
||||
// Claims are optional provider-specific values and should not be required by core business code.
|
||||
type Principal struct {
|
||||
UserID string
|
||||
Roles []string
|
||||
Claims map[string]any
|
||||
UserID string `json:"userId"`
|
||||
TenantID string `json:"tenantId,omitempty"`
|
||||
Roles []string `json:"roles,omitempty"`
|
||||
Claims map[string]any `json:"claims,omitempty"`
|
||||
}
|
||||
|
||||
func (p Principal) HasRole(role string) bool {
|
||||
|
||||
127
auth/jwt.go
Normal file
127
auth/jwt.go
Normal file
@@ -0,0 +1,127 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/ed25519"
|
||||
"crypto/rsa"
|
||||
"crypto/x509"
|
||||
"errors"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/golang-jwt/jwt/v5"
|
||||
)
|
||||
|
||||
// JWTKey binds a trusted key ID to exactly one asymmetric signing algorithm.
|
||||
// Keys come from server configuration, never a token's jku/x5u URL.
|
||||
type JWTKey struct {
|
||||
Algorithm string
|
||||
PublicKey any
|
||||
}
|
||||
type JWTConfig struct {
|
||||
Issuer string `json:"issuer"`
|
||||
Audience string `json:"audience"`
|
||||
Leeway time.Duration `json:"leeway"`
|
||||
Keys map[string]JWTKey `json:"-"`
|
||||
}
|
||||
type JWT struct {
|
||||
config JWTConfig
|
||||
mu sync.RWMutex
|
||||
keys map[string]JWTKey
|
||||
}
|
||||
type JWTClaims struct {
|
||||
jwt.RegisteredClaims
|
||||
TenantID string `json:"tenantId,omitempty"`
|
||||
Roles []string `json:"roles,omitempty"`
|
||||
}
|
||||
|
||||
func NewJWT(cfg JWTConfig) (*JWT, error) {
|
||||
if cfg.Issuer == "" || cfg.Audience == "" || cfg.Leeway < 0 || cfg.Leeway > 2*time.Minute {
|
||||
return nil, errors.New("auth: issuer/audience required, leeway must be 0..2m")
|
||||
}
|
||||
j := &JWT{config: cfg}
|
||||
j.config.Keys = nil
|
||||
if err := j.ReplaceKeys(cfg.Keys); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return j, nil
|
||||
}
|
||||
|
||||
// ReplaceKeys atomically rotates the trusted key set. Retain old keys until
|
||||
// their tokens expire if a rotation should preserve existing sessions.
|
||||
func (j *JWT) ReplaceKeys(keys map[string]JWTKey) error {
|
||||
if len(keys) == 0 || len(keys) > 64 {
|
||||
return errors.New("auth: require 1..64 JWT keys")
|
||||
}
|
||||
copyKeys := make(map[string]JWTKey, len(keys))
|
||||
for id, key := range keys {
|
||||
if id == "" || len(id) > 128 {
|
||||
return errors.New("auth: invalid key ID")
|
||||
}
|
||||
switch key.Algorithm {
|
||||
case "RS256":
|
||||
k, ok := key.PublicKey.(*rsa.PublicKey)
|
||||
if !ok || k == nil || k.N == nil || k.N.BitLen() < 2048 || k.E < 3 {
|
||||
return errors.New("auth: RS256 requires a >=2048-bit RSA public key")
|
||||
}
|
||||
case "EdDSA":
|
||||
k, ok := key.PublicKey.(ed25519.PublicKey)
|
||||
if !ok || len(k) != ed25519.PublicKeySize {
|
||||
return errors.New("auth: EdDSA requires an Ed25519 public key")
|
||||
}
|
||||
default:
|
||||
return errors.New("auth: supported algorithms are RS256 and EdDSA")
|
||||
}
|
||||
encoded, err := x509.MarshalPKIXPublicKey(key.PublicKey)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
key.PublicKey, err = x509.ParsePKIXPublicKey(encoded)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
copyKeys[id] = key
|
||||
}
|
||||
j.mu.Lock()
|
||||
j.keys = copyKeys
|
||||
j.mu.Unlock()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (j *JWT) Authenticate(ctx context.Context, raw string) (Principal, error) {
|
||||
if ctx == nil {
|
||||
return Principal{}, ErrInvalidToken
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return Principal{}, err
|
||||
}
|
||||
if len(raw) == 0 || len(raw) > 8192 {
|
||||
return Principal{}, ErrInvalidToken
|
||||
}
|
||||
claims := new(JWTClaims)
|
||||
token, err := jwt.ParseWithClaims(raw, claims, func(t *jwt.Token) (any, error) {
|
||||
id, ok := t.Header["kid"].(string)
|
||||
if !ok {
|
||||
return nil, ErrInvalidToken
|
||||
}
|
||||
j.mu.RLock()
|
||||
key, exists := j.keys[id]
|
||||
j.mu.RUnlock()
|
||||
if !exists || t.Method.Alg() != key.Algorithm {
|
||||
return nil, ErrInvalidToken
|
||||
}
|
||||
return key.PublicKey, nil
|
||||
}, jwt.WithValidMethods([]string{"RS256", "EdDSA"}), jwt.WithIssuer(j.config.Issuer), jwt.WithAudience(j.config.Audience), jwt.WithExpirationRequired(), jwt.WithIssuedAt(), jwt.WithLeeway(j.config.Leeway))
|
||||
if err != nil || !token.Valid || claims.Subject == "" || len(claims.Subject) > 256 || len(claims.TenantID) > 256 || len(claims.Roles) > 64 {
|
||||
return Principal{}, ErrInvalidToken
|
||||
}
|
||||
for _, role := range claims.Roles {
|
||||
if role == "" || len(role) > 128 {
|
||||
return Principal{}, ErrInvalidToken
|
||||
}
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return Principal{}, err
|
||||
}
|
||||
return Principal{UserID: claims.Subject, TenantID: claims.TenantID, Roles: append([]string(nil), claims.Roles...)}, nil
|
||||
}
|
||||
34
auth/policy.go
Normal file
34
auth/policy.go
Normal file
@@ -0,0 +1,34 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
)
|
||||
|
||||
type Policy interface {
|
||||
Authorize(context.Context, Principal, string, string) error
|
||||
}
|
||||
type PolicyFunc func(context.Context, Principal, string, string) error
|
||||
|
||||
func (f PolicyFunc) Authorize(ctx context.Context, p Principal, action, resource string) error {
|
||||
return f(ctx, p, action, resource)
|
||||
}
|
||||
func RequirePermission(policy Policy, action string, resource func(*httpx.Context) string) httpx.Middleware {
|
||||
return func(next httpx.Handler) httpx.Handler {
|
||||
return func(c *httpx.Context) error {
|
||||
p, ok := FromContext(c)
|
||||
if !ok {
|
||||
return apierrors.Unauthorized("UNAUTHENTICATED", "authentication required")
|
||||
}
|
||||
if policy == nil || action == "" || resource == nil {
|
||||
return apierrors.Internal(errors.New("auth: incomplete permission policy"))
|
||||
}
|
||||
if err := policy.Authorize(c.Context(), p, action, resource(c)); err != nil {
|
||||
return apierrors.Forbidden("FORBIDDEN", "permission denied").WithCause(err)
|
||||
}
|
||||
return next(c)
|
||||
}
|
||||
}
|
||||
}
|
||||
87
auth/session.go
Normal file
87
auth/session.go
Normal file
@@ -0,0 +1,87 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/cache"
|
||||
)
|
||||
|
||||
// Sessions stores server-issued opaque bearer credentials as SHA-256 keys.
|
||||
// The caller verifies login credentials before Issue; Store owns persistence.
|
||||
type Sessions struct {
|
||||
store cache.Store
|
||||
prefix string
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
func NewSessions(store cache.Store, prefix string, ttl time.Duration) (*Sessions, error) {
|
||||
if store == nil || len(prefix) > 512 || ttl < time.Second || ttl > 30*24*time.Hour {
|
||||
return nil, errors.New("auth: invalid session configuration")
|
||||
}
|
||||
return &Sessions{store: store, prefix: prefix, ttl: ttl}, nil
|
||||
}
|
||||
func (s *Sessions) Issue(ctx context.Context, p Principal) (string, error) {
|
||||
if p.UserID == "" || len(p.UserID) > 256 || len(p.TenantID) > 256 {
|
||||
return "", ErrInvalidToken
|
||||
}
|
||||
data, err := json.Marshal(p)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(data) > 16384 {
|
||||
return "", errors.New("auth: session principal too large")
|
||||
}
|
||||
secret := make([]byte, 32)
|
||||
if _, err := rand.Read(secret); err != nil {
|
||||
return "", err
|
||||
}
|
||||
token := base64.RawURLEncoding.EncodeToString(secret)
|
||||
key, _ := s.key(token)
|
||||
if err := s.store.Set(ctx, key, data, s.ttl); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return token, nil
|
||||
}
|
||||
func (s *Sessions) key(token string) (string, error) {
|
||||
if len(token) != 43 {
|
||||
return "", ErrInvalidToken
|
||||
}
|
||||
b, err := base64.RawURLEncoding.DecodeString(token)
|
||||
if err != nil || len(b) != 32 {
|
||||
return "", ErrInvalidToken
|
||||
}
|
||||
sum := sha256.Sum256(b)
|
||||
return s.prefix + hex.EncodeToString(sum[:]), nil
|
||||
}
|
||||
func (s *Sessions) Authenticate(ctx context.Context, token string) (Principal, error) {
|
||||
key, err := s.key(token)
|
||||
if err != nil {
|
||||
return Principal{}, err
|
||||
}
|
||||
data, err := s.store.Get(ctx, key)
|
||||
if errors.Is(err, cache.ErrMiss) {
|
||||
return Principal{}, ErrInvalidToken
|
||||
}
|
||||
if err != nil {
|
||||
return Principal{}, err
|
||||
}
|
||||
var p Principal
|
||||
if len(data) > 16384 || json.Unmarshal(data, &p) != nil || p.UserID == "" {
|
||||
return Principal{}, ErrInvalidToken
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
func (s *Sessions) Revoke(ctx context.Context, token string) error {
|
||||
key, err := s.key(token)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return s.store.Delete(ctx, key)
|
||||
}
|
||||
@@ -72,3 +72,20 @@ python3 scripts/verifyDatabase.py --cli /tmp/go-omni --framework-dir .
|
||||
```
|
||||
|
||||
数据库验收需要本地 Docker,使用独立 MySQL 8.4 容器;不会连接已有业务数据库,结束后清理测试容器与数据卷。
|
||||
|
||||
## 显式迁移(main)
|
||||
|
||||
SQL 命名为 `0001_createUsers.up.sql`。先预览版本和 SHA-256 清单:
|
||||
|
||||
```sh
|
||||
go-omni migrate --dir migrations
|
||||
```
|
||||
|
||||
部署步骤配置 `GO_OMNI_DB_DSN`,显式执行:
|
||||
|
||||
```sh
|
||||
go-omni migrate --dir migrations --apply
|
||||
```
|
||||
|
||||
预览不连接数据库;应用启动不自动迁移。失败保留 dirty 状态,不提供自动 down/force。
|
||||
脚手架原有 `schema.sql` 不会自动变成有版本的迁移,已有库需要先确定基线。详见 [扩展模块](../../ADVANCED.md)。
|
||||
|
||||
@@ -17,6 +17,11 @@ func main() {
|
||||
}
|
||||
|
||||
switch os.Args[1] {
|
||||
case "migrate":
|
||||
if err := runMigrate(os.Args[2:]); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "go-omni migrate:", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
case "add":
|
||||
if err := runAdd(os.Args[2:]); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "go-omni add:", err)
|
||||
@@ -108,6 +113,7 @@ func usage() {
|
||||
fmt.Fprintln(os.Stderr, "GoOmni project tool")
|
||||
fmt.Fprintln(os.Stderr, "")
|
||||
fmt.Fprintln(os.Stderr, "Usage:")
|
||||
fmt.Fprintln(os.Stderr, " go-omni migrate [--dir migrations] [--apply] [--dsn-env GO_OMNI_DB_DSN]")
|
||||
fmt.Fprintln(os.Stderr, " go-omni new [--module <module>] [--force] <directory>")
|
||||
fmt.Fprintln(os.Stderr, " go-omni add <name> [--dir <project>] Generate an isolated business module")
|
||||
fmt.Fprintln(os.Stderr, " --framework-version <version> Pin a remote framework version (defaults to CLI version)")
|
||||
|
||||
60
cmd/go-omni/migrate.go
Normal file
60
cmd/go-omni/migrate.go
Normal file
@@ -0,0 +1,60 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
migration "git.yuntuo.hk/k2safe/go-omni/database/migrate"
|
||||
"io"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
func runMigrate(args []string) error {
|
||||
flags := flag.NewFlagSet("migrate", flag.ContinueOnError)
|
||||
flags.SetOutput(io.Discard)
|
||||
directory := flags.String("dir", "migrations", "SQL directory")
|
||||
dsnEnv := flags.String("dsn-env", "GO_OMNI_DB_DSN", "DSN environment variable")
|
||||
apply := flags.Bool("apply", false, "apply pending up migrations")
|
||||
if err := flags.Parse(args); err != nil {
|
||||
return err
|
||||
}
|
||||
if flags.NArg() != 0 {
|
||||
return errors.New("migrate: unexpected argument")
|
||||
}
|
||||
_, manifest, err := migration.Snapshot(os.DirFS(*directory), ".")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !*apply {
|
||||
data, err := json.MarshalIndent(manifest, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Println(string(data))
|
||||
return nil
|
||||
}
|
||||
dsn := os.Getenv(*dsnEnv)
|
||||
if dsn == "" {
|
||||
return errors.New("migrate: DSN environment variable is empty")
|
||||
}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
runner, err := migration.NewMySQL(ctx, dsn, os.DirFS(*directory), ".")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer runner.Close()
|
||||
if err = runner.Up(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
version, dirty, err := runner.Version()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("migration version=%d dirty=%t\n", version, dirty)
|
||||
return nil
|
||||
}
|
||||
25
cmd/go-omni/migrate_test.go
Normal file
25
cmd/go-omni/migrate_test.go
Normal file
@@ -0,0 +1,25 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMigrationPreviewDoesNotConnect(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(dir, "0001_createItems.up.sql"), []byte("CREATE TABLE testItems(id BIGINT);"), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("GO_OMNI_DB_DSN", "not a DSN")
|
||||
if err := runMigrate([]string{"--dir", dir}); err != nil {
|
||||
t.Fatal("preview tried to connect", err)
|
||||
}
|
||||
t.Setenv("GO_OMNI_DB_DSN", "")
|
||||
if err := runMigrate([]string{"--dir", dir, "--apply"}); err == nil {
|
||||
t.Fatal("missing DSN accepted")
|
||||
}
|
||||
if err := runMigrate([]string{"--dir", dir, "--force"}); err == nil {
|
||||
t.Fatal("unsafe unknown flag accepted")
|
||||
}
|
||||
}
|
||||
@@ -13,15 +13,18 @@ const defaultEnvPrefix = "GO_OMNI_"
|
||||
// LoadFromEnv loads application settings from GO_OMNI_* variables. The
|
||||
// database settings remain owned by database.ConfigFromEnv.
|
||||
func LoadFromEnv(prefix string) (Config, error) {
|
||||
return applyEnv(Default(), prefix)
|
||||
}
|
||||
|
||||
func applyEnv(cfg Config, prefix string) (Config, error) {
|
||||
prefix = strings.TrimSpace(prefix)
|
||||
if prefix == "" {
|
||||
prefix = defaultEnvPrefix
|
||||
}
|
||||
cfg := Default()
|
||||
cfg.Address = envString(prefix+"HTTP_ADDR", cfg.Address)
|
||||
cfg.Transport = envString(prefix+"TRANSPORT", cfg.Transport)
|
||||
cfg.TLS.CertFile = envString(prefix+"TLS_CERT_FILE", "")
|
||||
cfg.TLS.KeyFile = envString(prefix+"TLS_KEY_FILE", "")
|
||||
cfg.TLS.CertFile = envString(prefix+"TLS_CERT_FILE", cfg.TLS.CertFile)
|
||||
cfg.TLS.KeyFile = envString(prefix+"TLS_KEY_FILE", cfg.TLS.KeyFile)
|
||||
cfg.RequestIDHeader = envString(prefix+"REQUEST_ID_HEADER", cfg.RequestIDHeader)
|
||||
if value := strings.TrimSpace(os.Getenv(prefix + "TRUSTED_PROXIES")); value != "" {
|
||||
cfg.TrustedProxies = strings.Split(value, ",")
|
||||
|
||||
203
config/file.go
Normal file
203
config/file.go
Normal file
@@ -0,0 +1,203 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"gopkg.in/yaml.v3"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// DecodeFile merges a bounded JSON/YAML document into defaults. It uses JSON
|
||||
// field names for both formats, rejects unknown fields and accepts durations
|
||||
// such as "3s". Application-specific validation remains with the caller.
|
||||
func DecodeFile[T any](path string, defaults T) (T, error) {
|
||||
var zero T
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
defer f.Close()
|
||||
data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
if len(data) > 1<<20 {
|
||||
return zero, errors.New("config: file exceeds 1 MiB")
|
||||
}
|
||||
var value any
|
||||
switch strings.ToLower(filepath.Ext(path)) {
|
||||
case ".yaml", ".yml":
|
||||
dec := yaml.NewDecoder(bytes.NewReader(data))
|
||||
if err := dec.Decode(&value); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
var extra any
|
||||
if err := dec.Decode(&extra); err != io.EOF {
|
||||
return zero, errors.New("config: one YAML document required")
|
||||
}
|
||||
case ".json":
|
||||
dec := json.NewDecoder(bytes.NewReader(data))
|
||||
dec.UseNumber()
|
||||
if err := dec.Decode(&value); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
var extra any
|
||||
if err := dec.Decode(&extra); err != io.EOF {
|
||||
return zero, errors.New("config: one JSON document required")
|
||||
}
|
||||
default:
|
||||
return zero, errors.New("config: expected .json, .yaml or .yml")
|
||||
}
|
||||
if _, ok := value.(map[string]any); !ok {
|
||||
return zero, errors.New("config: root must be an object")
|
||||
}
|
||||
value, err = durations(value, reflect.TypeOf(defaults))
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
// Deep copy defaults so errors cannot partially mutate caller-owned maps.
|
||||
original, err := json.Marshal(defaults)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
var result T
|
||||
if err = json.Unmarshal(original, &result); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
dec := json.NewDecoder(bytes.NewReader(encoded))
|
||||
dec.DisallowUnknownFields()
|
||||
if err = dec.Decode(&result); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
func durations(v any, t reflect.Type) (any, error) {
|
||||
if t == nil {
|
||||
return v, nil
|
||||
}
|
||||
for t.Kind() == reflect.Pointer {
|
||||
t = t.Elem()
|
||||
}
|
||||
if t == reflect.TypeOf(time.Duration(0)) {
|
||||
if s, ok := v.(string); ok {
|
||||
d, err := time.ParseDuration(s)
|
||||
return int64(d), err
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
if t.Kind() == reflect.Struct {
|
||||
if m, ok := v.(map[string]any); ok {
|
||||
for n := 0; n < t.NumField(); n++ {
|
||||
field := t.Field(n)
|
||||
name := strings.Split(field.Tag.Get("json"), ",")[0]
|
||||
if name == "-" {
|
||||
continue
|
||||
}
|
||||
if name == "" {
|
||||
name = field.Name
|
||||
}
|
||||
if item, exists := m[name]; exists {
|
||||
converted, err := durations(item, field.Type)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
m[name] = converted
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if t.Kind() == reflect.Slice {
|
||||
if items, ok := v.([]any); ok {
|
||||
for n, item := range items {
|
||||
converted, err := durations(item, t.Elem())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
items[n] = converted
|
||||
}
|
||||
}
|
||||
}
|
||||
return v, nil
|
||||
}
|
||||
func LoadFile(path, prefix string) (Config, error) {
|
||||
cfg, err := DecodeFile(path, Default())
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
return applyEnv(cfg, prefix)
|
||||
}
|
||||
|
||||
// Reloader only accepts log.level changes. Static changes and invalid files
|
||||
// leave the current snapshot and logger untouched. Watch does not own a goroutine.
|
||||
type Reloader struct {
|
||||
mu sync.Mutex
|
||||
path, prefix string
|
||||
current Config
|
||||
logger *logx.Logger
|
||||
}
|
||||
|
||||
func NewReloader(path, prefix string, logger *logx.Logger) (*Reloader, error) {
|
||||
cfg, err := LoadFile(path, prefix)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err = logger.SetLevel(cfg.Log.Level); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Reloader{path: path, prefix: prefix, current: cfg, logger: logger}, nil
|
||||
}
|
||||
func (r *Reloader) Reload() error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
next, err := LoadFile(r.path, r.prefix)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
comparable := next
|
||||
comparable.Log.Level = r.current.Log.Level
|
||||
if !reflect.DeepEqual(comparable, r.current) {
|
||||
return errors.New("config: static fields changed; restart required")
|
||||
}
|
||||
if err = r.logger.SetLevel(next.Log.Level); err != nil {
|
||||
return err
|
||||
}
|
||||
r.current = next
|
||||
return nil
|
||||
}
|
||||
func (r *Reloader) Snapshot() Config {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
cfg := r.current
|
||||
cfg.TrustedProxies = append([]string(nil), cfg.TrustedProxies...)
|
||||
return cfg
|
||||
}
|
||||
func (r *Reloader) Watch(ctx context.Context, interval time.Duration, onError func(error)) error {
|
||||
if ctx == nil || interval < 100*time.Millisecond {
|
||||
return errors.New("config: context and interval >=100ms required")
|
||||
}
|
||||
ticker := time.NewTicker(interval)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-ticker.C:
|
||||
if err := r.Reload(); err != nil && onError != nil {
|
||||
onError(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
59
config/file_test.go
Normal file
59
config/file_test.go
Normal file
@@ -0,0 +1,59 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestFileMergeAndSafeReload(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "service.yaml")
|
||||
write := func(raw string) {
|
||||
t.Helper()
|
||||
if err := os.WriteFile(path, []byte(raw), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n")
|
||||
t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123")
|
||||
cfg, err := LoadFile(path, "TEST_MODULE_")
|
||||
if err != nil || cfg.Address != ":9123" || cfg.ShutdownTimeout != 7*time.Second {
|
||||
t.Fatal(cfg, err)
|
||||
}
|
||||
var output bytes.Buffer
|
||||
lc := cfg.Log
|
||||
lc.Writer = &output
|
||||
l, err := logx.New(lc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer l.Close()
|
||||
child := l.With("scope", "test")
|
||||
r, err := NewReloader(path, "TEST_MODULE_", l)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
child.Debug("hidden")
|
||||
write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: debug\n")
|
||||
if err = r.Reload(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
child.Debug("visible")
|
||||
if strings.Contains(output.String(), "hidden") || !strings.Contains(output.String(), "visible") {
|
||||
t.Fatal(output.String())
|
||||
}
|
||||
write("shutdownTimeout: 8s\nlog:\n level: error\n")
|
||||
if err = r.Reload(); err == nil || r.Snapshot().Log.Level != "debug" {
|
||||
t.Fatal("static reload changed live config", err)
|
||||
}
|
||||
for _, raw := range []string{"unknownOption: true", "log:\n level: bad", "shutdownTimeout: invalid", "address: x\naddress: y", "---\n{}\n---\n{}"} {
|
||||
write(raw)
|
||||
if err = r.Reload(); err == nil {
|
||||
t.Fatal(raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
245
database/migrate/migrate.go
Normal file
245
database/migrate/migrate.go
Normal file
@@ -0,0 +1,245 @@
|
||||
// Package migrate provides explicit MySQL schema migration jobs. Application
|
||||
// startup never runs migrations automatically; use a separate deployment step.
|
||||
package migrate
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
mysqlDriver "github.com/go-sql-driver/mysql"
|
||||
migration "github.com/golang-migrate/migrate/v4"
|
||||
migrateMySQL "github.com/golang-migrate/migrate/v4/database/mysql"
|
||||
"github.com/golang-migrate/migrate/v4/source/iofs"
|
||||
"io"
|
||||
"io/fs"
|
||||
"path"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type File struct {
|
||||
Version uint64 `json:"version"`
|
||||
Name string `json:"name"`
|
||||
Direction string `json:"direction"`
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
var filename = regexp.MustCompile(`^([0-9]+)_([A-Za-z][A-Za-z0-9]*)\.(up|down)\.sql$`)
|
||||
|
||||
// Snapshot returns immutable migration bytes and a reviewable digest manifest.
|
||||
// Follow lowerCamelCase names, e.g. 0001_createUsers.up.sql.
|
||||
func Snapshot(source fs.FS, directory string) (fs.FS, []File, error) {
|
||||
if source == nil || !fs.ValidPath(directory) {
|
||||
return nil, nil, errors.New("migrate: invalid source directory")
|
||||
}
|
||||
entries, err := fs.ReadDir(source, directory)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
var archive bytes.Buffer
|
||||
writer := zip.NewWriter(&archive)
|
||||
manifest := []File{}
|
||||
seen := map[string]bool{}
|
||||
total := 0
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
parts := filename.FindStringSubmatch(entry.Name())
|
||||
if parts == nil {
|
||||
if strings.HasSuffix(entry.Name(), ".sql") {
|
||||
return nil, nil, fmt.Errorf("migrate: invalid migration filename %s", entry.Name())
|
||||
}
|
||||
continue
|
||||
}
|
||||
if len(manifest) >= 1000 {
|
||||
return nil, nil, errors.New("migrate: too many files")
|
||||
}
|
||||
version, err := strconv.ParseUint(parts[1], 10, 32)
|
||||
if err != nil || version == 0 {
|
||||
return nil, nil, errors.New("migrate: version must be 1..4294967295")
|
||||
}
|
||||
identity := fmt.Sprint(version) + parts[3]
|
||||
if seen[identity] {
|
||||
return nil, nil, errors.New("migrate: duplicate migration version/direction")
|
||||
}
|
||||
seen[identity] = true
|
||||
file, err := source.Open(path.Join(directory, entry.Name()))
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(file, (1<<20)+1))
|
||||
closeErr := file.Close()
|
||||
if err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
total += len(data)
|
||||
if len(data) == 0 || len(data) > 1<<20 || total > 16<<20 {
|
||||
return nil, nil, errors.New("migrate: files must be 1..1MiB and <=16MiB total")
|
||||
}
|
||||
sum := sha256.Sum256(data)
|
||||
manifest = append(manifest, File{Version: version, Name: parts[2], Direction: parts[3], SHA256: hex.EncodeToString(sum[:])})
|
||||
fileWriter, err := writer.CreateHeader(&zip.FileHeader{Name: entry.Name(), Method: zip.Store})
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if _, err = fileWriter.Write(data); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
}
|
||||
if len(manifest) == 0 {
|
||||
return nil, nil, errors.New("migrate: no SQL migrations")
|
||||
}
|
||||
for _, file := range manifest {
|
||||
if file.Direction == "down" && !seen[fmt.Sprint(file.Version)+"up"] {
|
||||
return nil, nil, errors.New("migrate: down file lacks up migration")
|
||||
}
|
||||
}
|
||||
sort.Slice(manifest, func(i, j int) bool {
|
||||
if manifest[i].Version == manifest[j].Version {
|
||||
return manifest[i].Direction < manifest[j].Direction
|
||||
}
|
||||
return manifest[i].Version < manifest[j].Version
|
||||
})
|
||||
if err = writer.Close(); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
files, err := zip.NewReader(bytes.NewReader(archive.Bytes()), int64(archive.Len()))
|
||||
return files, manifest, err
|
||||
}
|
||||
|
||||
type Runner struct {
|
||||
mu sync.Mutex
|
||||
engine *migration.Migrate
|
||||
db *sql.DB
|
||||
closed bool
|
||||
stopped bool
|
||||
}
|
||||
|
||||
func NewMySQL(ctx context.Context, dsn string, source fs.FS, directory string) (runner *Runner, err error) {
|
||||
if ctx == nil {
|
||||
return nil, errors.New("migrate: context required")
|
||||
}
|
||||
snapshot, _, err := Snapshot(source, directory)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg, err := mysqlDriver.ParseDSN(dsn)
|
||||
if err != nil || cfg.DBName == "" {
|
||||
return nil, errors.New("migrate: valid MySQL DSN with database required")
|
||||
}
|
||||
cfg.MultiStatements = true
|
||||
if cfg.Timeout == 0 || cfg.Timeout > 5*time.Second {
|
||||
cfg.Timeout = 5 * time.Second
|
||||
}
|
||||
if cfg.ReadTimeout == 0 || cfg.ReadTimeout > 30*time.Second {
|
||||
cfg.ReadTimeout = 30 * time.Second
|
||||
}
|
||||
if cfg.WriteTimeout == 0 || cfg.WriteTimeout > 30*time.Second {
|
||||
cfg.WriteTimeout = 30 * time.Second
|
||||
}
|
||||
connector, err := mysqlDriver.NewConnector(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
db := sql.OpenDB(connector)
|
||||
db.SetMaxOpenConns(1)
|
||||
db.SetMaxIdleConns(1)
|
||||
defer func() {
|
||||
if err != nil {
|
||||
db.Close()
|
||||
}
|
||||
}()
|
||||
conn, err := db.Conn(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
driver, err := migrateMySQL.WithConnection(ctx, conn, &migrateMySQL.Config{DatabaseName: cfg.DBName, StatementTimeout: 30 * time.Second})
|
||||
if err != nil {
|
||||
conn.Close()
|
||||
return nil, err
|
||||
}
|
||||
src, err := iofs.New(snapshot, ".")
|
||||
if err != nil {
|
||||
driver.Close()
|
||||
return nil, err
|
||||
}
|
||||
engine, err := migration.NewWithInstance("iofs", src, cfg.DBName, driver)
|
||||
if err != nil {
|
||||
src.Close()
|
||||
driver.Close()
|
||||
return nil, err
|
||||
}
|
||||
engine.LockTimeout = 10 * time.Second
|
||||
return &Runner{engine: engine, db: db}, nil
|
||||
}
|
||||
|
||||
// Up applies pending up scripts. MySQL DDL can partially commit; a failed
|
||||
// migration remains dirty and requires operator investigation. No automatic
|
||||
// down/force/dirty reset is provided. Cancellation stops after the current SQL
|
||||
// operation; statements/read I/O are bounded to 30 seconds.
|
||||
func (r *Runner) Up(ctx context.Context) error {
|
||||
if ctx == nil {
|
||||
return errors.New("migrate: context required")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.closed || r.stopped {
|
||||
return errors.New("migrate: closed")
|
||||
}
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- r.engine.Up() }()
|
||||
var err error
|
||||
select {
|
||||
case err = <-done:
|
||||
case <-ctx.Done():
|
||||
r.stopped = true // golang-migrate retains its graceful-stop state; create a fresh runner to retry.
|
||||
select {
|
||||
case r.engine.GracefulStop <- true:
|
||||
default:
|
||||
}
|
||||
err = <-done
|
||||
return errors.Join(ctx.Err(), err)
|
||||
}
|
||||
if errors.Is(err, migration.ErrNoChange) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (r *Runner) Version() (uint, bool, error) {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.closed {
|
||||
return 0, false, errors.New("migrate: closed")
|
||||
}
|
||||
version, dirty, err := r.engine.Version()
|
||||
if errors.Is(err, migration.ErrNilVersion) {
|
||||
return 0, false, nil
|
||||
}
|
||||
return version, dirty, err
|
||||
}
|
||||
func (r *Runner) Close() error {
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if r.closed {
|
||||
return nil
|
||||
}
|
||||
r.closed = true
|
||||
a, b := r.engine.Close()
|
||||
return errors.Join(a, b, r.db.Close())
|
||||
}
|
||||
99
database/migrate/migrate_test.go
Normal file
99
database/migrate/migrate_test.go
Normal file
@@ -0,0 +1,99 @@
|
||||
package migrate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"git.yuntuo.hk/k2safe/go-omni/telemetry"
|
||||
"go.opentelemetry.io/otel/sdk/trace/tracetest"
|
||||
"io/fs"
|
||||
"os"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSnapshotBoundsAndIdentity(t *testing.T) {
|
||||
source := fstest.MapFS{"0001_createItems.up.sql": &fstest.MapFile{Data: []byte("SELECT 1;")}}
|
||||
f, manifest, err := Snapshot(source, ".")
|
||||
if err != nil || len(manifest) != 1 {
|
||||
t.Fatal(manifest, err)
|
||||
}
|
||||
source["0001_createItems.up.sql"].Data[0] = 'x'
|
||||
raw, err := fs.ReadFile(f, "0001_createItems.up.sql")
|
||||
if err != nil || string(raw) != "SELECT 1;" {
|
||||
t.Fatal(string(raw), err)
|
||||
}
|
||||
for _, files := range []fstest.MapFS{{"0001_bad_name.up.sql": &fstest.MapFile{Data: []byte("x")}}, {"0001_a.up.sql": &fstest.MapFile{Data: []byte("x")}, "1_b.up.sql": &fstest.MapFile{Data: []byte("x")}}, {"0001_a.down.sql": &fstest.MapFile{Data: []byte("x")}}, {"0001_a.up.sql": &fstest.MapFile{Data: make([]byte, (1<<20)+1)}}} {
|
||||
if _, _, err = Snapshot(files, "."); err == nil {
|
||||
t.Fatal("invalid migration accepted")
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestMySQLIntegration(t *testing.T) {
|
||||
dsn := os.Getenv("GO_OMNI_MIGRATE_TEST_DSN")
|
||||
if dsn == "" {
|
||||
t.Skip("requires disposable MySQL database")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
files := fstest.MapFS{"0001_createItems.up.sql": &fstest.MapFile{Data: []byte("CREATE TABLE testItems (id BIGINT NOT NULL PRIMARY KEY, value VARCHAR(64) NOT NULL);")}, "0002_seedItems.up.sql": &fstest.MapFile{Data: []byte("INSERT INTO testItems (id,value) VALUES (1,'original');")}}
|
||||
runner, err := NewMySQL(ctx, dsn, files, ".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = runner.Up(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = runner.Up(ctx); err != nil {
|
||||
t.Fatal("repeat", err)
|
||||
}
|
||||
v, dirty, err := runner.Version()
|
||||
if err != nil || v != 2 || dirty {
|
||||
t.Fatal(v, dirty, err)
|
||||
}
|
||||
runner.Close()
|
||||
db, err := sql.Open("mysql", dsn)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer db.Close()
|
||||
exporter := tracetest.NewInMemoryExporter()
|
||||
tr, _ := telemetry.New(telemetry.Config{Service: "dbTest", SampleRatio: 1, Exporter: exporter})
|
||||
defer tr.Close()
|
||||
instrumented := telemetry.SQL{DB: db, Runtime: tr}
|
||||
if _, err = instrumented.ExecContext(ctx, "UPDATE testItems SET value=? WHERE id=1", "secret parameter"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rows, err := instrumented.QueryContext(ctx, "SELECT value FROM testItems WHERE id=1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !rows.Next() {
|
||||
t.Fatal("missing row")
|
||||
}
|
||||
var value string
|
||||
if err = rows.Scan(&value); err != nil || value != "secret parameter" {
|
||||
t.Fatal(value, err)
|
||||
}
|
||||
rows.Close()
|
||||
tr.Flush(ctx)
|
||||
if len(exporter.GetSpans()) != 2 {
|
||||
t.Fatal("SQL spans missing")
|
||||
}
|
||||
files["0003_failMigration.up.sql"] = &fstest.MapFile{Data: []byte("INSERT INTO missingTable VALUES (1);")}
|
||||
runner, err = NewMySQL(ctx, dsn, files, ".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer runner.Close()
|
||||
if err = runner.Up(ctx); err == nil {
|
||||
t.Fatal("bad migration passed")
|
||||
}
|
||||
v, dirty, err = runner.Version()
|
||||
if err != nil || v != 3 || !dirty {
|
||||
t.Fatal(v, dirty, err)
|
||||
}
|
||||
if err = runner.Up(ctx); err == nil {
|
||||
t.Fatal("dirty migration auto-cleared")
|
||||
}
|
||||
}
|
||||
14
examples/platform/README.md
Normal file
14
examples/platform/README.md
Normal file
@@ -0,0 +1,14 @@
|
||||
# 扩展模块组合示例
|
||||
|
||||
业务通过 `storage.Store` 写文件,身份接口和 Policy 控制权限,OpenAPI 与路由一起注册,链路追踪关联 HTTP 与文件操作。`newApp` 不依赖 Local/S3/OSS 类型;启动处本例选择 Local。可替换成 S3/R2/OSS 构造结果,业务路由不变。
|
||||
|
||||
```sh
|
||||
export GO_OMNI_DEMO_TOKEN='use-a-private-development-token'
|
||||
go run ./examples/platform --root /tmp/goOmniDemoFiles
|
||||
```
|
||||
|
||||
接口:`POST /api/files`,JSON 为 `{"name":"hello.txt","content":"hello"}`,请求加 `Authorization: Bearer <GO_OMNI_DEMO_TOKEN>`;返回 201 和存储信息。`GET /openapi.json` 同样要求身份。
|
||||
|
||||
也可传 `--config service.yaml`,配置规则见 [扩展模块](../../ADVANCED.md)。本例的静态身份只用于演示,实际应用注入 `auth.JWT` 或 `auth.Sessions`;trace provider 默认不外发,接入 collector 时注入 exporter/reader。
|
||||
|
||||
`TestPlatformTLSAdapters` 通过三个 Runner 的真实 TLS 请求验证 401/403/201、文件内容、非法 key、OpenAPI 和关闭。文件根目录必须是应用私有目录。示例不执行自动数据迁移,也不启动 Orbit。
|
||||
132
examples/platform/main.go
Normal file
132
examples/platform/main.go
Normal file
@@ -0,0 +1,132 @@
|
||||
// platform composes storage, authentication, permission policy, tracing and
|
||||
// OpenAPI using provider-neutral business dependencies.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"flag"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"syscall"
|
||||
|
||||
omni "git.yuntuo.hk/k2safe/go-omni"
|
||||
"git.yuntuo.hk/k2safe/go-omni/auth"
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
"git.yuntuo.hk/k2safe/go-omni/openapi"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/local"
|
||||
"git.yuntuo.hk/k2safe/go-omni/telemetry"
|
||||
)
|
||||
|
||||
type dependencies struct {
|
||||
Files storage.Store
|
||||
Auth auth.Authenticator
|
||||
Tracing *telemetry.Runtime
|
||||
}
|
||||
type upload struct {
|
||||
Name string `json:"name" binding:"required,max=128"`
|
||||
Content string `json:"content" binding:"max=4096"`
|
||||
}
|
||||
|
||||
// Dependencies are borrowed; the composition root owns their shutdown.
|
||||
func newApp(cfg omni.Config, d dependencies) (*omni.App, error) {
|
||||
if d.Files == nil || d.Auth == nil || d.Tracing == nil {
|
||||
return nil, errors.New("platform: dependencies required")
|
||||
}
|
||||
app := omni.New(omni.WithConfig(cfg))
|
||||
app.Use(d.Tracing.Middleware())
|
||||
api, err := openapi.New("File service", "1")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
input, err := openapi.SchemaOf[upload]()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
output, err := openapi.SchemaOf[storage.Info]()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
policy := auth.PolicyFunc(func(_ context.Context, p auth.Principal, action, resource string) error {
|
||||
if p.TenantID != "" && p.HasRole("writer") && action == "create" && resource == "files" {
|
||||
return nil
|
||||
}
|
||||
return errors.New("denied")
|
||||
})
|
||||
err = api.Handle(app, openapi.Operation{Method: "POST", Path: "/api/files", OperationID: "createFile", Summary: "Store a small text file", Request: input, BearerAuth: true, Responses: map[int]openapi.Response{201: {Description: "Created", Schema: openapi.Envelope(output)}}}, func(c *omni.Context) error {
|
||||
var in upload
|
||||
if err := c.BindJSON(&in); err != nil {
|
||||
return err
|
||||
}
|
||||
p, _ := auth.FromContext(c)
|
||||
if strings.ContainsAny(in.Name, "/\\") {
|
||||
return omni.BadRequest("INVALID_NAME", "name must be one path segment")
|
||||
}
|
||||
key := p.TenantID + "/" + in.Name
|
||||
if err := storage.Check(c.Context(), key); err != nil {
|
||||
return omni.BadRequest("INVALID_KEY", "invalid storage key")
|
||||
}
|
||||
var info storage.Info
|
||||
err := d.Tracing.Do(c.Context(), "files.put", func(ctx context.Context) error {
|
||||
var err error
|
||||
info, err = d.Files.Put(ctx, key, strings.NewReader(in.Content), int64(len(in.Content)), "text/plain; charset=utf-8")
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return c.Created(info)
|
||||
}, auth.RequireAuth(d.Auth), auth.RequirePermission(policy, "create", func(*omni.Context) string { return "files" }))
|
||||
if err != nil {
|
||||
app.Close()
|
||||
return nil, err
|
||||
}
|
||||
app.GET("/openapi.json", api.Handler(), auth.RequireAuth(d.Auth))
|
||||
return app, nil
|
||||
}
|
||||
func run(ctx context.Context, configFile, root string) error {
|
||||
var cfg config.Config
|
||||
var err error
|
||||
if configFile != "" {
|
||||
cfg, err = config.LoadFile(configFile, "")
|
||||
} else {
|
||||
cfg, err = config.LoadFromEnv("")
|
||||
if os.Getenv("GO_OMNI_HTTP_ADDR") == "" {
|
||||
cfg.Address = "127.0.0.1:8080"
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
token := os.Getenv("GO_OMNI_DEMO_TOKEN")
|
||||
if token == "" {
|
||||
return errors.New("set GO_OMNI_DEMO_TOKEN for the demonstration credential")
|
||||
}
|
||||
files, err := local.New(root, 64<<10)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tracing, err := telemetry.New(telemetry.Config{Service: "fileService", SampleRatio: 1})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tracing.Close()
|
||||
app, err := newApp(cfg, dependencies{Files: files, Auth: auth.StaticAuthenticator{token: {UserID: "demoUser", TenantID: "demoTenant", Roles: []string{"writer"}}}, Tracing: tracing})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return app.RunContext(ctx)
|
||||
}
|
||||
func main() {
|
||||
file := flag.String("config", "", "optional JSON/YAML config")
|
||||
root := flag.String("root", "./fileData", "private local object directory")
|
||||
flag.Parse()
|
||||
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
if err := run(ctx, *file, *root); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
102
examples/platform/main_test.go
Normal file
102
examples/platform/main_test.go
Normal file
@@ -0,0 +1,102 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
omni "git.yuntuo.hk/k2safe/go-omni"
|
||||
"git.yuntuo.hk/k2safe/go-omni/auth"
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
"git.yuntuo.hk/k2safe/go-omni/internal/testutil"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/local"
|
||||
"git.yuntuo.hk/k2safe/go-omni/telemetry"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestPlatformTLSAdapters(t *testing.T) {
|
||||
for _, name := range []string{omni.TransportNetHTTP, omni.TransportNative, omni.TransportHTTP2} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
cfg := config.Default()
|
||||
cfg.Transport = name
|
||||
cert, key, tlsClient := testutil.CertificateFiles(t)
|
||||
cfg.TLS = config.TLSConfig{CertFile: cert, KeyFile: key}
|
||||
store, _ := local.New(t.TempDir(), 64<<10)
|
||||
tr, _ := telemetry.New(telemetry.Config{Service: "platformTest", SampleRatio: 1})
|
||||
defer tr.Close()
|
||||
app, err := newApp(cfg, dependencies{Files: store, Auth: auth.StaticAuthenticator{"writer": {UserID: "u", TenantID: "t", Roles: []string{"writer"}}, "reader": {UserID: "r", TenantID: "t"}}, Tracing: tr})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- app.ServeContext(ctx, listener) }()
|
||||
defer func() {
|
||||
cancel()
|
||||
select {
|
||||
case err := <-done:
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
listener.Close()
|
||||
t.Error("shutdown timeout")
|
||||
}
|
||||
}()
|
||||
client := &http.Client{Timeout: 3 * time.Second, Transport: &http.Transport{TLSClientConfig: tlsClient, ForceAttemptHTTP2: true}}
|
||||
defer client.CloseIdleConnections()
|
||||
request := func(method, path, body, token string) (int, string) {
|
||||
t.Helper()
|
||||
r, _ := http.NewRequest(method, "https://"+listener.Addr().String()+path, strings.NewReader(body))
|
||||
r.Header.Set("Authorization", "Bearer "+token)
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
response, err := client.Do(r)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, err := io.ReadAll(response.Body)
|
||||
response.Body.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if response.TLS == nil || len(response.TLS.VerifiedChains) == 0 {
|
||||
t.Fatal("TLS unverified")
|
||||
}
|
||||
if name == omni.TransportHTTP2 && response.ProtoMajor != 2 {
|
||||
t.Fatal("protocol")
|
||||
}
|
||||
return response.StatusCode, string(raw)
|
||||
}
|
||||
for _, test := range []struct {
|
||||
token string
|
||||
status int
|
||||
}{{"", 401}, {"reader", 403}, {"writer", 201}} {
|
||||
status, body := request("POST", "/api/files", `{"name":"hello.txt","content":"hello"}`, test.token)
|
||||
if status != test.status {
|
||||
t.Fatal(status, body)
|
||||
}
|
||||
}
|
||||
object, err := store.Get(context.Background(), "t/hello.txt")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := io.ReadAll(object.Body)
|
||||
object.Body.Close()
|
||||
if string(raw) != "hello" {
|
||||
t.Fatal(string(raw))
|
||||
}
|
||||
if status, _ := request("POST", "/api/files", `{"name":"../escape","content":"x"}`, "writer"); status != 400 {
|
||||
t.Fatal(status)
|
||||
}
|
||||
if status, body := request("GET", "/openapi.json", "", "writer"); status != 200 || !strings.Contains(body, `"/api/files"`) {
|
||||
t.Fatal(status, body)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
37
go.mod
37
go.mod
@@ -1,25 +1,48 @@
|
||||
module git.yuntuo.hk/k2safe/go-omni
|
||||
|
||||
go 1.22
|
||||
go 1.22.0
|
||||
|
||||
require (
|
||||
github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0
|
||||
github.com/go-playground/validator/v10 v10.20.0
|
||||
github.com/go-sql-driver/mysql v1.9.3
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1
|
||||
github.com/golang-migrate/migrate/v4 v4.18.1
|
||||
github.com/minio/minio-go/v7 v7.0.83
|
||||
github.com/redis/go-redis/v9 v9.17.3
|
||||
golang.org/x/mod v0.17.0
|
||||
golang.org/x/net v0.25.0
|
||||
go.opentelemetry.io/otel v1.31.0
|
||||
go.opentelemetry.io/otel/metric v1.31.0
|
||||
go.opentelemetry.io/otel/sdk v1.31.0
|
||||
go.opentelemetry.io/otel/sdk/metric v1.31.0
|
||||
go.opentelemetry.io/otel/trace v1.31.0
|
||||
golang.org/x/mod v0.21.0
|
||||
golang.org/x/net v0.33.0
|
||||
gopkg.in/yaml.v3 v3.0.1
|
||||
)
|
||||
|
||||
require (
|
||||
filippo.io/edwards25519 v1.1.0 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/go-logr/logr v1.4.2 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/go-playground/locales v0.14.1 // indirect
|
||||
github.com/go-playground/universal-translator v0.18.1 // indirect
|
||||
github.com/goccy/go-json v0.10.4 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/hashicorp/errwrap v1.1.0 // indirect
|
||||
github.com/hashicorp/go-multierror v1.1.1 // indirect
|
||||
github.com/klauspost/compress v1.17.11 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
|
||||
github.com/leodido/go-urn v1.4.0 // indirect
|
||||
github.com/stretchr/testify v1.9.0 // indirect
|
||||
golang.org/x/crypto v0.23.0 // indirect
|
||||
golang.org/x/sys v0.20.0 // indirect
|
||||
golang.org/x/text v0.15.0 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
go.uber.org/atomic v1.7.0 // indirect
|
||||
golang.org/x/crypto v0.31.0 // indirect
|
||||
golang.org/x/sys v0.28.0 // indirect
|
||||
golang.org/x/text v0.21.0 // indirect
|
||||
golang.org/x/time v0.5.0 // indirect
|
||||
)
|
||||
|
||||
110
go.sum
110
go.sum
@@ -1,17 +1,45 @@
|
||||
filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA=
|
||||
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
|
||||
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 h1:L/gRVlceqvL25UVaW/CKtUDjefjrs0SPonmDGUVOYP0=
|
||||
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
|
||||
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
|
||||
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
|
||||
github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0 h1:uWzn3io54f9L9mvwsQQSv1KpkkFA06hBxI++RvIyvpI=
|
||||
github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0/go.mod h1:FTzydeQVmR24FI0D6XWUOMKckjXehM/jgMn1xC+DA9M=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/dhui/dktest v0.4.3 h1:wquqUxAFdcUgabAVLvSCOKOlag5cIZuaOjYIBOWdsR0=
|
||||
github.com/dhui/dktest v0.4.3/go.mod h1:zNK8IwktWzQRm6I/l2Wjp7MakiyaFWv4G1hjmodmMTs=
|
||||
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
|
||||
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
|
||||
github.com/docker/docker v27.2.0+incompatible h1:Rk9nIVdfH3+Vz4cyI/uhbINhEZ/oLmc+CBXmH6fbNk4=
|
||||
github.com/docker/docker v27.2.0+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
|
||||
github.com/docker/go-connections v0.5.0 h1:USnMq7hx7gwdVZq1L49hLXaFtUdTADjXGp+uj1Br63c=
|
||||
github.com/docker/go-connections v0.5.0/go.mod h1:ov60Kzw0kKElRwhNs9UlUHAE/F9Fe6GLaXnqyDdmEXc=
|
||||
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
|
||||
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
|
||||
github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
|
||||
github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0=
|
||||
github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk=
|
||||
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
|
||||
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
|
||||
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
|
||||
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
|
||||
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
|
||||
@@ -22,23 +50,85 @@ github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBEx
|
||||
github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM=
|
||||
github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo=
|
||||
github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU=
|
||||
github.com/goccy/go-json v0.10.4 h1:JSwxQzIqKfmFX1swYPpUThQZp/Ka4wzJdK0LWVytLPM=
|
||||
github.com/goccy/go-json v0.10.4/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
|
||||
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
|
||||
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
|
||||
github.com/golang-migrate/migrate/v4 v4.18.1 h1:JML/k+t4tpHCpQTCAD62Nu43NUFzHY4CV3uAuvHGC+Y=
|
||||
github.com/golang-migrate/migrate/v4 v4.18.1/go.mod h1:HAX6m3sQgcdO81tdjn5exv20+3Kb13cmGli1hrD6hks=
|
||||
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
|
||||
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
|
||||
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
|
||||
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
|
||||
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
|
||||
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
|
||||
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY=
|
||||
github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
|
||||
github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ=
|
||||
github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI=
|
||||
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
|
||||
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.0.83 h1:W4Kokksvlz3OKf3OqIlzDNKd4MERlC2oN8YptwJ0+GA=
|
||||
github.com/minio/minio-go/v7 v7.0.83/go.mod h1:57YXpvc5l3rjPdhqNrDsvVlY0qPI6UTk1bflAe+9doY=
|
||||
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
|
||||
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
|
||||
github.com/moby/term v0.5.0 h1:xt8Q1nalod/v7BqbG21f8mQPqH+xAaC9C3N3wfWbVP0=
|
||||
github.com/moby/term v0.5.0/go.mod h1:8FzsFHVUBGZdbDsJw/ot+X+d5HLUbvklYLJ9uGfcI3Y=
|
||||
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
|
||||
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
|
||||
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
|
||||
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
|
||||
github.com/opencontainers/image-spec v1.1.0 h1:8SG7/vwALn54lVB/0yZ/MMwhFrPYtpEHQb2IpWsCzug=
|
||||
github.com/opencontainers/image-spec v1.1.0/go.mod h1:W4s4sFTMaBeK1BQLXbG4AdM2szdn85PY75RI83NrTrM=
|
||||
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.17.3 h1:fN29NdNrE17KttK5Ndf20buqfDZwGNgoUr9qjl1DQx4=
|
||||
github.com/redis/go-redis/v9 v9.17.3/go.mod h1:u410H11HMLoB+TP67dz8rL9s6QW2j76l0//kSOd3370=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
|
||||
github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
golang.org/x/crypto v0.23.0 h1:dIJU/v2J8Mdglj/8rJ6UUOM3Zc9zLZxVZwwxMooUSAI=
|
||||
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
|
||||
golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=
|
||||
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
|
||||
golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
|
||||
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
|
||||
golang.org/x/sys v0.20.0 h1:Od9JTbYCk261bKm4M/mw7AklTlFYIa0bIp9BgSm1S8Y=
|
||||
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/text v0.15.0 h1:h1V/4gjBv8v9cjcR6+AR5+/cIYK5N/WAgiv4xlsEtAk=
|
||||
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 h1:TT4fX+nBOA/+LUkobKGW1ydGcn+G3vRw9+g5HwCphpk=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0/go.mod h1:L7UH0GbB0p47T4Rri3uHjbpCFYrVrwc1I25QhNPiGK8=
|
||||
go.opentelemetry.io/otel v1.31.0 h1:NsJcKPIW0D0H3NgzPDHmo0WW6SptzPdqg/L1zsIm2hY=
|
||||
go.opentelemetry.io/otel v1.31.0/go.mod h1:O0C14Yl9FgkjqcCZAsE053C13OaddMYr/hz6clDkEJE=
|
||||
go.opentelemetry.io/otel/metric v1.31.0 h1:FSErL0ATQAmYHUIzSezZibnyVlft1ybhy4ozRPcF2fE=
|
||||
go.opentelemetry.io/otel/metric v1.31.0/go.mod h1:C3dEloVbLuYoX41KpmAhOqNriGbA+qqH6PQ5E5mUfnY=
|
||||
go.opentelemetry.io/otel/sdk v1.31.0 h1:xLY3abVHYZ5HSfOg3l2E5LUj2Cwva5Y7yGxnSW9H5Gk=
|
||||
go.opentelemetry.io/otel/sdk v1.31.0/go.mod h1:TfRbMdhvxIIr/B2N2LQW2S5v9m3gOQ/08KsbbO5BPT0=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.31.0 h1:i9hxxLJF/9kkvfHppyLL55aW7iIJz4JjxTeYusH7zMc=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.31.0/go.mod h1:CRInTMVvNhUKgSAMbKyTMxqOBC0zgyxzW55lZzX43Y8=
|
||||
go.opentelemetry.io/otel/trace v1.31.0 h1:ffjsj1aRouKewfr85U2aGagJ46+MvodynlQ1HYdmJys=
|
||||
go.opentelemetry.io/otel/trace v1.31.0/go.mod h1:TXZkRk7SM2ZQLtR6eoAWQFIHPvzQ06FJAsO1tJg480A=
|
||||
go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw=
|
||||
go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
|
||||
golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
|
||||
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
|
||||
golang.org/x/mod v0.21.0 h1:vvrHzRwRfVKSiLrG+d4FMl/Qi4ukBCE6kZlTUkDYRT0=
|
||||
golang.org/x/mod v0.21.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
|
||||
golang.org/x/net v0.33.0 h1:74SYHlV8BIgHIFC/LrYkOGIwL19eTYXQ5wc6TBuO36I=
|
||||
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
|
||||
golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
|
||||
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo=
|
||||
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
|
||||
golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=
|
||||
golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
|
||||
63
internal/storagefixture/server.go
Normal file
63
internal/storagefixture/server.go
Normal file
@@ -0,0 +1,63 @@
|
||||
// Package storagefixture supplies a bounded protocol fixture, not a cloud service.
|
||||
package storagefixture
|
||||
|
||||
import (
|
||||
"crypto/md5"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type value struct {
|
||||
data []byte
|
||||
contentType string
|
||||
etag string
|
||||
}
|
||||
|
||||
func Handler() http.Handler {
|
||||
var mu sync.Mutex
|
||||
values := map[string]value{}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if r.Header.Get("Authorization") == "" && r.URL.Query().Get("X-Amz-Signature") == "" && r.URL.Query().Get("x-oss-signature") == "" {
|
||||
http.Error(w, "signature required", 403)
|
||||
return
|
||||
}
|
||||
key := r.URL.Path
|
||||
switch r.Method {
|
||||
case "PUT":
|
||||
data, err := io.ReadAll(io.LimitReader(r.Body, 1025))
|
||||
if err != nil || len(data) > 1024 {
|
||||
http.Error(w, "too large", 413)
|
||||
return
|
||||
}
|
||||
v := value{data: data, contentType: r.Header.Get("Content-Type"), etag: fmt.Sprintf("%x", md5.Sum(data))}
|
||||
values[key] = v
|
||||
w.Header().Set("ETag", `"`+v.etag+`"`)
|
||||
w.WriteHeader(200)
|
||||
case "DELETE":
|
||||
delete(values, key)
|
||||
w.WriteHeader(204)
|
||||
case "GET", "HEAD":
|
||||
v, ok := values[key]
|
||||
if !ok {
|
||||
w.Header().Set("Content-Type", "application/xml")
|
||||
w.WriteHeader(404)
|
||||
fmt.Fprint(w, `<Error><Code>NoSuchKey</Code><Message>not found</Message></Error>`)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Length", fmt.Sprint(len(v.data)))
|
||||
w.Header().Set("Content-Type", v.contentType)
|
||||
w.Header().Set("ETag", `"`+v.etag+`"`)
|
||||
w.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat))
|
||||
if r.Method == "GET" {
|
||||
w.Write(v.data)
|
||||
}
|
||||
default:
|
||||
http.Error(w, "unsupported", 405)
|
||||
}
|
||||
})
|
||||
}
|
||||
40
logx/logx.go
40
logx/logx.go
@@ -21,14 +21,14 @@ import (
|
||||
// standard stream. File takes precedence over Output and appends to the named
|
||||
// file.
|
||||
type Config struct {
|
||||
Level string
|
||||
Format string
|
||||
Output string
|
||||
File string
|
||||
Service string
|
||||
Environment string
|
||||
AddSource bool
|
||||
Writer io.Writer
|
||||
Level string `json:"level"`
|
||||
Format string `json:"format"`
|
||||
Output string `json:"output"`
|
||||
File string `json:"file"`
|
||||
Service string `json:"service"`
|
||||
Environment string `json:"environment"`
|
||||
AddSource bool `json:"addSource"`
|
||||
Writer io.Writer `json:"-"`
|
||||
}
|
||||
|
||||
// Default returns the default production-friendly logger configuration.
|
||||
@@ -45,6 +45,7 @@ func Default() Config {
|
||||
type Logger struct {
|
||||
logger *slog.Logger
|
||||
closer *closeState
|
||||
level *slog.LevelVar
|
||||
}
|
||||
|
||||
type closeState struct {
|
||||
@@ -78,7 +79,9 @@ func New(cfg Config) (*Logger, error) {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
opts := &slog.HandlerOptions{Level: level, AddSource: cfg.AddSource}
|
||||
dynamicLevel := new(slog.LevelVar)
|
||||
dynamicLevel.Set(level)
|
||||
opts := &slog.HandlerOptions{Level: dynamicLevel, AddSource: cfg.AddSource}
|
||||
var handler slog.Handler
|
||||
if cfg.Format == "text" {
|
||||
handler = slog.NewTextHandler(writer, opts)
|
||||
@@ -97,7 +100,7 @@ func New(cfg Config) (*Logger, error) {
|
||||
if closeWriter != nil {
|
||||
closer = &closeState{fn: closeWriter}
|
||||
}
|
||||
return &Logger{logger: logger, closer: closer}, nil
|
||||
return &Logger{logger: logger, closer: closer, level: dynamicLevel}, nil
|
||||
}
|
||||
|
||||
// Setup replaces the process-wide logger used by the package-level helpers.
|
||||
@@ -152,14 +155,27 @@ func (l *Logger) With(args ...any) *Logger {
|
||||
if l == nil || l.logger == nil {
|
||||
return defaultLogger.Load().With(args...)
|
||||
}
|
||||
return &Logger{logger: l.logger.With(args...), closer: l.closer}
|
||||
return &Logger{logger: l.logger.With(args...), closer: l.closer, level: l.level}
|
||||
}
|
||||
|
||||
func (l *Logger) WithGroup(name string) *Logger {
|
||||
if l == nil || l.logger == nil {
|
||||
return defaultLogger.Load().WithGroup(name)
|
||||
}
|
||||
return &Logger{logger: l.logger.WithGroup(name), closer: l.closer}
|
||||
return &Logger{logger: l.logger.WithGroup(name), closer: l.closer, level: l.level}
|
||||
}
|
||||
|
||||
// SetLevel atomically changes the threshold shared by this logger and children.
|
||||
func (l *Logger) SetLevel(value string) error {
|
||||
if l == nil || l.level == nil {
|
||||
return errors.New("logx: logger does not support dynamic levels")
|
||||
}
|
||||
level, err := parseLevel(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
l.level.Set(level)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Close releases resources owned by the root logger. It is safe to call more
|
||||
|
||||
@@ -17,12 +17,13 @@ import (
|
||||
)
|
||||
|
||||
type IdempotencyConfig struct {
|
||||
TTL time.Duration `json:"ttl"`
|
||||
MaxEntries int `json:"maxEntries"`
|
||||
MaxBytes int `json:"maxBytes"`
|
||||
MaxRequestBytes int64 `json:"maxRequestBytes"`
|
||||
MaxResponseBytes int `json:"maxResponseBytes"`
|
||||
MaxInFlight int `json:"maxInFlight"`
|
||||
Store IdempotencyStore `json:"-"` // Optional shared store; nil keeps local behavior.
|
||||
TTL time.Duration `json:"ttl"`
|
||||
MaxEntries int `json:"maxEntries"`
|
||||
MaxBytes int `json:"maxBytes"`
|
||||
MaxRequestBytes int64 `json:"maxRequestBytes"`
|
||||
MaxResponseBytes int `json:"maxResponseBytes"`
|
||||
MaxInFlight int `json:"maxInFlight"`
|
||||
// Scope must return a server-verified tenant/user identity. Register auth
|
||||
// before this middleware. Client-provided identity is not authorization.
|
||||
Scope func(*httpx.Context) (string, error) `json:"-"`
|
||||
@@ -53,6 +54,9 @@ func NewIdempotency(cfg IdempotencyConfig) (*Idempotency, error) {
|
||||
if cfg.TTL == 0 {
|
||||
cfg.TTL = 10 * time.Minute
|
||||
}
|
||||
if cfg.Store != nil && cfg.TTL > 30*24*time.Hour {
|
||||
return nil, errors.New("middleware: shared idempotency TTL exceeds 30d")
|
||||
}
|
||||
if cfg.MaxEntries == 0 {
|
||||
cfg.MaxEntries = 256
|
||||
}
|
||||
@@ -125,6 +129,9 @@ func (i *Idempotency) Middleware() httpx.Middleware {
|
||||
copy(fingerprint[:], hash.Sum(nil))
|
||||
namespace, _ := json.Marshal([]string{scope, key})
|
||||
id := sha256.Sum256(namespace)
|
||||
if i.config.Store != nil {
|
||||
return i.shared(c, next, id, fingerprint)
|
||||
}
|
||||
i.mu.Lock()
|
||||
now := i.now()
|
||||
for k, v := range i.items {
|
||||
|
||||
@@ -13,6 +13,7 @@ import (
|
||||
)
|
||||
|
||||
type RateLimitConfig struct {
|
||||
Backend RateLimitBackend `json:"-"`
|
||||
Rate float64 `json:"rate"` // Tokens per second, per key.
|
||||
Burst int `json:"burst"`
|
||||
MaxKeys int `json:"maxKeys"`
|
||||
@@ -64,7 +65,17 @@ func (l *RateLimiter) Middleware() httpx.Middleware {
|
||||
if key == "" || len(key) > 1024 {
|
||||
return apierrors.BadRequest("INVALID_RATE_KEY", "invalid rate limit key")
|
||||
}
|
||||
delay, full := l.allow(key)
|
||||
var delay time.Duration
|
||||
var full bool
|
||||
if l.config.Backend != nil {
|
||||
var err error
|
||||
delay, err = l.config.Backend.Take(c.Context(), key)
|
||||
if err != nil {
|
||||
return apierrors.New(503, "RATE_LIMIT_UNAVAILABLE", "rate limiter unavailable").WithCause(err)
|
||||
}
|
||||
} else {
|
||||
delay, full = l.allow(key)
|
||||
}
|
||||
if full {
|
||||
return apierrors.New(503, "RATE_LIMIT_CAPACITY", "rate limiter capacity reached")
|
||||
}
|
||||
|
||||
178
middleware/redis/store.go
Normal file
178
middleware/redis/store.go
Normal file
@@ -0,0 +1,178 @@
|
||||
// Package redis supplies atomic shared middleware state in one Redis hash slot.
|
||||
// Use a dedicated namespace and a noeviction Redis deployment for correctness.
|
||||
package redis
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.yuntuo.hk/k2safe/go-omni/middleware"
|
||||
goredis "github.com/redis/go-redis/v9"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type IdempotencyConfig struct {
|
||||
MaxEntries int `json:"maxEntries"`
|
||||
MaxRecordBytes int `json:"maxRecordBytes"`
|
||||
}
|
||||
type IdempotencyStore struct {
|
||||
client goredis.Scripter
|
||||
keys []string
|
||||
cfg IdempotencyConfig
|
||||
signature string
|
||||
}
|
||||
|
||||
func namespace(prefix, kind string) (string, error) {
|
||||
if prefix == "" || len(prefix) > 256 || strings.ContainsAny(prefix, "{}\r\n") {
|
||||
return "", errors.New("middleware/redis: nonempty bounded namespace without braces required")
|
||||
}
|
||||
sum := sha256.Sum256([]byte(prefix))
|
||||
return "goOmni:{" + hex.EncodeToString(sum[:]) + "}:" + kind, nil
|
||||
}
|
||||
func NewIdempotencyStore(client goredis.Scripter, prefix string, cfg IdempotencyConfig) (*IdempotencyStore, error) {
|
||||
if client == nil || cfg.MaxEntries < 0 || cfg.MaxRecordBytes < 0 {
|
||||
return nil, errors.New("middleware/redis: invalid configuration")
|
||||
}
|
||||
if cfg.MaxEntries == 0 {
|
||||
cfg.MaxEntries = 128
|
||||
}
|
||||
if cfg.MaxRecordBytes == 0 {
|
||||
cfg.MaxRecordBytes = 128 << 10
|
||||
}
|
||||
if cfg.MaxEntries > 100000 || cfg.MaxRecordBytes < 512 || cfg.MaxRecordBytes > 1<<20 {
|
||||
return nil, errors.New("middleware/redis: capacity outside supported bounds")
|
||||
}
|
||||
base, err := namespace(prefix, "idempotency")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &IdempotencyStore{client: client, keys: []string{base + ":records", base + ":expiry", base + ":config"}, cfg: cfg, signature: fmt.Sprintf("%d:%d", cfg.MaxEntries, cfg.MaxRecordBytes)}, nil
|
||||
}
|
||||
|
||||
var claim = goredis.NewScript(`
|
||||
local config=redis.call('GET',KEYS[3])
|
||||
if config and config~=ARGV[5] then return redis.error_reply('configuration mismatch') end
|
||||
if not config then redis.call('SET',KEYS[3],ARGV[5]) end
|
||||
local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000)
|
||||
local expired=redis.call('ZRANGEBYSCORE',KEYS[2],'-inf',now,'LIMIT',0,128)
|
||||
for _,key in ipairs(expired) do redis.call('HDEL',KEYS[1],key);redis.call('ZREM',KEYS[2],key) end
|
||||
local expiry=redis.call('ZSCORE',KEYS[2],ARGV[1]);if expiry and tonumber(expiry)<=now then redis.call('HDEL',KEYS[1],ARGV[1]);redis.call('ZREM',KEYS[2],ARGV[1]) end
|
||||
local value=redis.call('HGET',KEYS[1],ARGV[1]);if value then return {0,value} end
|
||||
if redis.call('HLEN',KEYS[1])>=tonumber(ARGV[4]) then return redis.error_reply('idempotency capacity reached') end
|
||||
value=cjson.encode({fingerprint=ARGV[2],owner=ARGV[3],state='running'})
|
||||
redis.call('HSET',KEYS[1],ARGV[1],value);return {1,value}
|
||||
`)
|
||||
|
||||
func (s *IdempotencyStore) Claim(ctx context.Context, key, fp, owner string) (middleware.IdempotencyRecord, bool, error) {
|
||||
if len(key) != 64 || len(fp) != 64 || len(owner) != 32 {
|
||||
return middleware.IdempotencyRecord{}, false, errors.New("middleware/redis: invalid claim")
|
||||
}
|
||||
result, err := claim.Run(ctx, s.client, s.keys, key, fp, owner, s.cfg.MaxEntries, s.signature).Slice()
|
||||
if err != nil {
|
||||
return middleware.IdempotencyRecord{}, false, err
|
||||
}
|
||||
if len(result) != 2 {
|
||||
return middleware.IdempotencyRecord{}, false, errors.New("middleware/redis: invalid claim result")
|
||||
}
|
||||
value, ok := result[1].(string)
|
||||
if !ok || len(value) > s.cfg.MaxRecordBytes {
|
||||
return middleware.IdempotencyRecord{}, false, errors.New("middleware/redis: invalid record")
|
||||
}
|
||||
var record middleware.IdempotencyRecord
|
||||
err = json.Unmarshal([]byte(value), &record)
|
||||
return record, result[0] == int64(1), err
|
||||
}
|
||||
|
||||
var complete = goredis.NewScript(`
|
||||
local current=redis.call('HGET',KEYS[1],ARGV[1]);if not current then return redis.error_reply('claim missing') end
|
||||
local value=cjson.decode(current)
|
||||
if value.owner~=ARGV[2] or value.state~='running' then return redis.error_reply('claim owner/state changed') end
|
||||
local response=ARGV[3]
|
||||
if response~='' then value.state='completed';value.response=cjson.decode(response) else value.state='failed' end
|
||||
local encoded=cjson.encode(value)
|
||||
if string.len(encoded)>tonumber(ARGV[5]) then value.state='failed';value.response=nil;encoded=cjson.encode(value) end
|
||||
local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000)
|
||||
redis.call('HSET',KEYS[1],ARGV[1],encoded);redis.call('ZADD',KEYS[2],now+tonumber(ARGV[4]),ARGV[1]);return 1
|
||||
`)
|
||||
|
||||
func (s *IdempotencyStore) Complete(ctx context.Context, key, owner string, response *middleware.StoredResponse, ttl time.Duration) error {
|
||||
if ttl <= 0 || ttl > 30*24*time.Hour {
|
||||
return errors.New("middleware/redis: TTL must be positive and <=30d")
|
||||
}
|
||||
var encoded []byte
|
||||
var err error
|
||||
if response != nil {
|
||||
encoded, err = json.Marshal(response)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(encoded) > s.cfg.MaxRecordBytes {
|
||||
encoded = nil
|
||||
}
|
||||
}
|
||||
return complete.Run(ctx, s.client, s.keys, key, owner, string(encoded), max(1, ttl.Milliseconds()), s.cfg.MaxRecordBytes).Err()
|
||||
}
|
||||
|
||||
// Resolve removes only a matching non-replayable claim after the operator has
|
||||
// reconciled the business outcome. Never invoke automatically on a timeout.
|
||||
func (s *IdempotencyStore) Resolve(ctx context.Context, key, owner string) error {
|
||||
return goredis.NewScript(`local v=redis.call('HGET',KEYS[1],ARGV[1]);if not v then return 0 end;local r=cjson.decode(v);if r.owner~=ARGV[2] or r.state=='completed' then return redis.error_reply('claim cannot be resolved') end;redis.call('HDEL',KEYS[1],ARGV[1]);redis.call('ZREM',KEYS[2],ARGV[1]);return 1`).Run(ctx, s.client, s.keys, key, owner).Err()
|
||||
}
|
||||
|
||||
type limiter struct {
|
||||
client goredis.Scripter
|
||||
keys []string
|
||||
cfg middleware.RateLimitConfig
|
||||
signature string
|
||||
}
|
||||
|
||||
func NewRateLimiter(client goredis.Scripter, prefix string, cfg middleware.RateLimitConfig) (*middleware.RateLimiter, error) {
|
||||
if client == nil {
|
||||
return nil, errors.New("middleware/redis: client required")
|
||||
}
|
||||
if cfg.MaxKeys == 0 {
|
||||
cfg.MaxKeys = 10000
|
||||
}
|
||||
if cfg.IdleTTL == 0 {
|
||||
cfg.IdleTTL = 24 * time.Hour
|
||||
}
|
||||
if _, err := middleware.NewRateLimiter(cfg); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
base, err := namespace(prefix, "rate")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
backend := &limiter{client: client, keys: []string{base + ":buckets", base + ":seen", base + ":config"}, cfg: cfg, signature: fmt.Sprintf("%g:%d:%d:%d", cfg.Rate, cfg.Burst, cfg.MaxKeys, cfg.IdleTTL.Milliseconds())}
|
||||
cfg.Backend = backend
|
||||
return middleware.NewRateLimiter(cfg)
|
||||
}
|
||||
|
||||
var take = goredis.NewScript(`
|
||||
local config=redis.call('GET',KEYS[3]);if config and config~=ARGV[6] then return redis.error_reply('configuration mismatch') end
|
||||
if not config then redis.call('SET',KEYS[3],ARGV[6]) end
|
||||
local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000)
|
||||
local expired=redis.call('ZRANGEBYSCORE',KEYS[2],'-inf',now-tonumber(ARGV[5]),'LIMIT',0,128)
|
||||
for _,key in ipairs(expired) do redis.call('HDEL',KEYS[1],key);redis.call('ZREM',KEYS[2],key) end
|
||||
local value=redis.call('HGET',KEYS[1],ARGV[1]);local tokens=tonumber(ARGV[3])
|
||||
if value then local b=cjson.decode(value);tokens=math.min(tokens,b.tokens+math.max(0,now-b.time)*tonumber(ARGV[2])/1000)
|
||||
elseif redis.call('HLEN',KEYS[1])>=tonumber(ARGV[4]) then return -1 end
|
||||
local delay=0;if tokens>=1 then tokens=tokens-1 else delay=math.ceil((1-tokens)*1000/tonumber(ARGV[2])) end
|
||||
redis.call('HSET',KEYS[1],ARGV[1],cjson.encode({tokens=tokens,time=now}));redis.call('ZADD',KEYS[2],now,ARGV[1]);return delay
|
||||
`)
|
||||
|
||||
func (l *limiter) Take(ctx context.Context, key string) (time.Duration, error) {
|
||||
sum := sha256.Sum256([]byte(key))
|
||||
delay, err := take.Run(ctx, l.client, l.keys, hex.EncodeToString(sum[:]), l.cfg.Rate, l.cfg.Burst, l.cfg.MaxKeys, l.cfg.IdleTTL.Milliseconds(), l.signature).Int64()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if delay < 0 {
|
||||
return 0, middleware.ErrLimitCapacity
|
||||
}
|
||||
return time.Duration(delay) * time.Millisecond, nil
|
||||
}
|
||||
145
middleware/redis/store_test.go
Normal file
145
middleware/redis/store_test.go
Normal file
@@ -0,0 +1,145 @@
|
||||
package redis
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
omni "git.yuntuo.hk/k2safe/go-omni"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
"git.yuntuo.hk/k2safe/go-omni/middleware"
|
||||
goredis "github.com/redis/go-redis/v9"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSharedRedisIntegration(t *testing.T) {
|
||||
address := os.Getenv("GO_OMNI_REDIS_TEST_ADDR")
|
||||
if address == "" {
|
||||
t.Skip("requires isolated Redis")
|
||||
}
|
||||
client := goredis.NewClient(&goredis.Options{Addr: address, PoolSize: 8, MaxRetries: -1, ContextTimeoutEnabled: true, ReadTimeout: time.Second, WriteTimeout: time.Second})
|
||||
defer client.Close()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
|
||||
defer cancel()
|
||||
prefix := fmt.Sprint("shared:", time.Now().UnixNano())
|
||||
s, err := NewIdempotencyStore(client, prefix, IdempotencyConfig{MaxEntries: 256})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer client.Del(context.Background(), s.keys...)
|
||||
key, fp := strings.Repeat("a", 64), strings.Repeat("b", 64)
|
||||
var claims atomic.Int32
|
||||
var wg sync.WaitGroup
|
||||
owners := make(chan string, 32)
|
||||
for i := 0; i < 32; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
owner := fmt.Sprintf("%032x", i)
|
||||
_, ok, err := s.Claim(ctx, key, fp, owner)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
if ok {
|
||||
claims.Add(1)
|
||||
owners <- owner
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
if claims.Load() != 1 {
|
||||
t.Fatal(claims.Load())
|
||||
}
|
||||
owner := <-owners
|
||||
if err = s.Complete(ctx, key, "wrong", nil, time.Minute); err == nil {
|
||||
t.Fatal("foreign completion accepted")
|
||||
}
|
||||
if err = s.Complete(ctx, key, owner, &middleware.StoredResponse{Status: 201, Headers: http.Header{}, Body: []byte{0, 1, 255}}, time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
record, ok, err := s.Claim(ctx, key, fp, strings.Repeat("c", 32))
|
||||
if err != nil || ok || record.State != "completed" || len(record.Response.Body) != 3 {
|
||||
t.Fatal(record, ok, err)
|
||||
}
|
||||
// Two independently constructed apps use the same backend, so only one runs.
|
||||
var executions atomic.Int32
|
||||
makeApp := func() *omni.App {
|
||||
backend, _ := NewIdempotencyStore(client, prefix, IdempotencyConfig{MaxEntries: 256})
|
||||
idem, err := middleware.NewIdempotency(middleware.IdempotencyConfig{Store: backend, Scope: func(*httpx.Context) (string, error) { return "verifiedTenant", nil }})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
a := omni.New(omni.WithLogger(false))
|
||||
a.POST("/submit", func(c *httpx.Context) error { executions.Add(1); return c.OK("saved") }, idem.Middleware())
|
||||
return a
|
||||
}
|
||||
a, b := makeApp(), makeApp()
|
||||
request := func(a *omni.App, body string) *httptest.ResponseRecorder {
|
||||
r := httptest.NewRequest("POST", "/submit", strings.NewReader(body))
|
||||
r.Header.Set("Idempotency-Key", "stableKey")
|
||||
w := httptest.NewRecorder()
|
||||
a.ServeHTTP(w, r)
|
||||
return w
|
||||
}
|
||||
first, second := request(a, `{}`), request(b, `{}`)
|
||||
if first.Code != 200 || second.Code != 200 || first.Body.String() != second.Body.String() || second.Header().Get("Idempotency-Replayed") != "true" || executions.Load() != 1 {
|
||||
t.Fatal(first.Code, second.Code, second.Body.String(), executions.Load())
|
||||
}
|
||||
if request(b, `{"changed":true}`).Code != 409 {
|
||||
t.Fatal("conflict accepted")
|
||||
}
|
||||
// Shared limits do not multiply when a second replica is added.
|
||||
limitPrefix := prefix + "limit"
|
||||
cfg := middleware.RateLimitConfig{Rate: .01, Burst: 2, Key: func(*httpx.Context) string { return "tenant" }}
|
||||
r1, _ := NewRateLimiter(client, limitPrefix, cfg)
|
||||
r2, _ := NewRateLimiter(client, limitPrefix, cfg)
|
||||
base, _ := namespace(limitPrefix, "rate")
|
||||
defer client.Del(context.Background(), base+":buckets", base+":seen", base+":config")
|
||||
limited := func(l *middleware.RateLimiter) int {
|
||||
a := omni.New(omni.WithLogger(false))
|
||||
a.GET("/", func(c *httpx.Context) error { return c.OK("ok") }, l.Middleware())
|
||||
w := httptest.NewRecorder()
|
||||
a.ServeHTTP(w, httptest.NewRequest("GET", "/", nil))
|
||||
return w.Code
|
||||
}
|
||||
if limited(r1) != 200 || limited(r2) != 200 || limited(r1) != 429 {
|
||||
t.Fatal("limits multiplied across instances")
|
||||
}
|
||||
// Redis failure must fail closed before executing a new request.
|
||||
client.Close()
|
||||
if status := limited(r2); status != 503 {
|
||||
t.Fatal(status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExpiredTargetBeyondCleanupBatch(t *testing.T) {
|
||||
address := os.Getenv("GO_OMNI_REDIS_TEST_ADDR")
|
||||
if address == "" {
|
||||
t.Skip("requires isolated Redis")
|
||||
}
|
||||
client := goredis.NewClient(&goredis.Options{Addr: address})
|
||||
defer client.Close()
|
||||
s, _ := NewIdempotencyStore(client, fmt.Sprint("expiry:", time.Now().UnixNano()), IdempotencyConfig{MaxEntries: 256})
|
||||
defer client.Del(context.Background(), s.keys...)
|
||||
ctx := context.Background()
|
||||
owner := strings.Repeat("1", 32)
|
||||
fp := strings.Repeat("2", 64)
|
||||
for i := 0; i < 140; i++ {
|
||||
key := fmt.Sprintf("%064x", i)
|
||||
if _, ok, err := s.Claim(ctx, key, fp, owner); err != nil || !ok {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.Complete(ctx, key, owner, nil, time.Minute); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
client.ZAdd(ctx, s.keys[1], goredis.Z{Score: 0, Member: key})
|
||||
}
|
||||
if _, ok, err := s.Claim(ctx, fmt.Sprintf("%064x", 139), fp, owner); err != nil || !ok {
|
||||
t.Fatal("expired target retained beyond TTL", ok, err)
|
||||
}
|
||||
}
|
||||
102
middleware/sharedIdempotency.go
Normal file
102
middleware/sharedIdempotency.go
Normal file
@@ -0,0 +1,102 @@
|
||||
package middleware
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type StoredResponse struct {
|
||||
Status int `json:"status"`
|
||||
Headers http.Header `json:"headers"`
|
||||
Body []byte `json:"body"`
|
||||
}
|
||||
type IdempotencyRecord struct {
|
||||
Fingerprint string `json:"fingerprint"`
|
||||
Owner string `json:"owner"`
|
||||
State string `json:"state"`
|
||||
Response *StoredResponse `json:"response,omitempty"`
|
||||
}
|
||||
|
||||
// IdempotencyStore must atomically claim absent keys, compare owners when
|
||||
// finishing and retain in-flight claims until explicit resolution. Only completed
|
||||
// or failed records expire; an uncertain outcome must never become a new claim.
|
||||
type IdempotencyStore interface {
|
||||
Claim(context.Context, string, string, string) (IdempotencyRecord, bool, error)
|
||||
Complete(context.Context, string, string, *StoredResponse, time.Duration) error
|
||||
}
|
||||
|
||||
func (i *Idempotency) shared(c *httpx.Context, next httpx.Handler, id, fingerprint [32]byte) error {
|
||||
key, fp := hex.EncodeToString(id[:]), hex.EncodeToString(fingerprint[:])
|
||||
token := make([]byte, 16)
|
||||
if _, err := rand.Read(token); err != nil {
|
||||
return err
|
||||
}
|
||||
owner := hex.EncodeToString(token)
|
||||
record, claimed, err := i.config.Store.Claim(c.Context(), key, fp, owner)
|
||||
if err != nil {
|
||||
return apierrors.New(503, "IDEMPOTENCY_UNAVAILABLE", "idempotency store unavailable").WithCause(err)
|
||||
}
|
||||
if !claimed {
|
||||
if record.Fingerprint != fp {
|
||||
return apierrors.New(409, "IDEMPOTENCY_CONFLICT", "key was used with a different request")
|
||||
}
|
||||
if record.State != "completed" || record.Response == nil {
|
||||
return apierrors.New(409, "IDEMPOTENCY_UNAVAILABLE", "request is in progress or outcome requires reconciliation")
|
||||
}
|
||||
response := record.Response
|
||||
headers := response.Headers.Clone()
|
||||
headers.Set("Idempotency-Replayed", "true")
|
||||
return c.WriteResponse(response.Status, headers, response.Body)
|
||||
}
|
||||
completed := false
|
||||
finish := func(response *StoredResponse) error {
|
||||
ctx, cancel := context.WithTimeout(context.WithoutCancel(c.Context()), time.Second)
|
||||
defer cancel()
|
||||
return i.config.Store.Complete(ctx, key, owner, response, i.config.TTL)
|
||||
}
|
||||
defer func() {
|
||||
if !completed {
|
||||
if err := finish(nil); err != nil {
|
||||
logx.ErrorContext(c.Context(), "idempotency.persistFailed", "error", err)
|
||||
}
|
||||
}
|
||||
}()
|
||||
buffer := &boundedResponse{headers: c.Raw().Header().Clone(), limit: i.config.MaxResponseBytes}
|
||||
child := c.Clone(c.Context(), buffer)
|
||||
err = next(child)
|
||||
if ctxErr := child.Context().Err(); ctxErr != nil {
|
||||
return ctxErr
|
||||
}
|
||||
if err != nil {
|
||||
child.WriteError(err)
|
||||
}
|
||||
if buffer.err != nil {
|
||||
return apierrors.Internal(buffer.err)
|
||||
}
|
||||
if buffer.status == 0 {
|
||||
buffer.status = 200
|
||||
}
|
||||
var response *StoredResponse
|
||||
if buffer.status < 500 && buffer.headers.Get("Set-Cookie") == "" && !strings.Contains(strings.ToLower(buffer.headers.Get("Cache-Control")), "no-store") {
|
||||
response = &StoredResponse{Status: buffer.status, Headers: buffer.headers.Clone(), Body: append([]byte(nil), buffer.body.Bytes()...)}
|
||||
}
|
||||
if err = finish(response); err != nil {
|
||||
return apierrors.New(503, "IDEMPOTENCY_UNAVAILABLE", "outcome requires reconciliation").WithCause(err)
|
||||
}
|
||||
completed = true
|
||||
return c.WriteResponse(buffer.status, buffer.headers, buffer.body.Bytes())
|
||||
}
|
||||
|
||||
var ErrLimitCapacity = errors.New("middleware: shared rate limiter capacity")
|
||||
|
||||
type RateLimitBackend interface {
|
||||
Take(context.Context, string) (time.Duration, error)
|
||||
}
|
||||
267
openapi/openapi.go
Normal file
267
openapi/openapi.go
Normal file
@@ -0,0 +1,267 @@
|
||||
// Package openapi registers routes and their OpenAPI 3.0.3 contracts together.
|
||||
package openapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
"net/http"
|
||||
"reflect"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Schema struct {
|
||||
Type string `json:"type,omitempty"`
|
||||
Format string `json:"format,omitempty"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Properties map[string]*Schema `json:"properties,omitempty"`
|
||||
Required []string `json:"required,omitempty"`
|
||||
Items *Schema `json:"items,omitempty"`
|
||||
AdditionalProperties *Schema `json:"additionalProperties,omitempty"`
|
||||
Nullable bool `json:"nullable,omitempty"`
|
||||
}
|
||||
type Parameter struct {
|
||||
Name string `json:"name"`
|
||||
In string `json:"in"`
|
||||
Required bool `json:"required"`
|
||||
Description string `json:"description,omitempty"`
|
||||
Schema *Schema `json:"schema"`
|
||||
}
|
||||
type Response struct {
|
||||
Description string
|
||||
Schema *Schema
|
||||
}
|
||||
type Operation struct {
|
||||
Method, Path, OperationID, Summary string
|
||||
Parameters []Parameter
|
||||
Request *Schema
|
||||
Responses map[int]Response
|
||||
BearerAuth bool
|
||||
}
|
||||
type Registrar interface {
|
||||
Handle(string, string, httpx.Handler, ...httpx.Middleware)
|
||||
}
|
||||
type API struct {
|
||||
mu sync.RWMutex
|
||||
document map[string]any
|
||||
paths map[string]map[string]any
|
||||
ids map[string]bool
|
||||
}
|
||||
|
||||
var identifier = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9_]*$`)
|
||||
|
||||
func New(title, version string) (*API, error) {
|
||||
if title == "" || version == "" {
|
||||
return nil, errors.New("openapi: title and version required")
|
||||
}
|
||||
paths := make(map[string]map[string]any)
|
||||
return &API{paths: paths, ids: make(map[string]bool), document: map[string]any{"openapi": "3.0.3", "info": map[string]string{"title": title, "version": version}, "paths": paths, "components": map[string]any{"securitySchemes": map[string]any{"bearerAuth": map[string]string{"type": "http", "scheme": "bearer"}}}}}, nil
|
||||
}
|
||||
|
||||
// Handle registers an absolute application route. To document a group, include
|
||||
// its full prefix here and pass middleware explicitly. Schemas describe wire
|
||||
// responses; Envelope wraps a data schema in GoOmni's standard JSON envelope.
|
||||
func (a *API) Handle(router Registrar, op Operation, handler httpx.Handler, middleware ...httpx.Middleware) error {
|
||||
if router == nil || handler == nil || !identifier.MatchString(op.OperationID) || !strings.HasPrefix(op.Path, "/") || strings.ContainsAny(op.Path, "*{}?# ") {
|
||||
return errors.New("openapi: invalid route")
|
||||
}
|
||||
method := strings.ToLower(op.Method)
|
||||
switch method {
|
||||
case "get", "post", "put", "patch", "delete", "head", "options":
|
||||
default:
|
||||
return errors.New("openapi: unsupported HTTP method")
|
||||
}
|
||||
parts := strings.Split(op.Path, "/")
|
||||
params := make(map[string]bool)
|
||||
for n, part := range parts {
|
||||
if strings.HasPrefix(part, ":") {
|
||||
name := part[1:]
|
||||
if !identifier.MatchString(name) || params[name] {
|
||||
return errors.New("openapi: invalid path parameter")
|
||||
}
|
||||
parts[n] = "{" + name + "}"
|
||||
params[name] = true
|
||||
}
|
||||
}
|
||||
path := strings.Join(parts, "/")
|
||||
seen := make(map[string]bool)
|
||||
for _, p := range op.Parameters {
|
||||
if p.Name == "" || p.Schema == nil || seen[p.In+":"+p.Name] {
|
||||
return errors.New("openapi: invalid or duplicate parameter")
|
||||
}
|
||||
seen[p.In+":"+p.Name] = true
|
||||
switch p.In {
|
||||
case "path":
|
||||
if !params[p.Name] || !p.Required {
|
||||
return errors.New("openapi: path parameter must match and be required")
|
||||
}
|
||||
delete(params, p.Name)
|
||||
case "query", "header", "cookie":
|
||||
default:
|
||||
return errors.New("openapi: invalid parameter location")
|
||||
}
|
||||
}
|
||||
if len(params) != 0 || len(op.Responses) == 0 {
|
||||
return errors.New("openapi: declare all path parameters and at least one response")
|
||||
}
|
||||
responses := map[string]any{}
|
||||
for code, response := range op.Responses {
|
||||
if code < 100 || code > 599 || response.Description == "" {
|
||||
return errors.New("openapi: invalid response")
|
||||
}
|
||||
item := map[string]any{"description": response.Description}
|
||||
if response.Schema != nil {
|
||||
item["content"] = map[string]any{"application/json": map[string]any{"schema": response.Schema}}
|
||||
}
|
||||
responses[fmt.Sprint(code)] = item
|
||||
}
|
||||
operation := map[string]any{"operationId": op.OperationID, "summary": op.Summary, "responses": responses}
|
||||
if len(op.Parameters) > 0 {
|
||||
operation["parameters"] = op.Parameters
|
||||
}
|
||||
if op.Request != nil {
|
||||
operation["requestBody"] = map[string]any{"required": true, "content": map[string]any{"application/json": map[string]any{"schema": op.Request}}}
|
||||
}
|
||||
if op.BearerAuth {
|
||||
operation["security"] = []map[string][]string{{"bearerAuth": {}}}
|
||||
}
|
||||
// Detach caller-owned maps/slices before concurrent document reads.
|
||||
encoded, err := json.Marshal(operation)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var frozen any
|
||||
if err = json.Unmarshal(encoded, &frozen); err != nil {
|
||||
return err
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
if a.ids[op.OperationID] || a.paths[path][method] != nil {
|
||||
return errors.New("openapi: duplicate operation ID or route")
|
||||
}
|
||||
router.Handle(strings.ToUpper(method), op.Path, handler, middleware...)
|
||||
if a.paths[path] == nil {
|
||||
a.paths[path] = make(map[string]any)
|
||||
}
|
||||
a.paths[path][method] = frozen
|
||||
a.ids[op.OperationID] = true
|
||||
return nil
|
||||
}
|
||||
func (a *API) JSON() ([]byte, error) {
|
||||
a.mu.RLock()
|
||||
defer a.mu.RUnlock()
|
||||
return json.MarshalIndent(a.document, "", " ")
|
||||
}
|
||||
func (a *API) Handler() httpx.Handler {
|
||||
return func(c *httpx.Context) error {
|
||||
data, err := a.JSON()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return c.WriteResponse(200, http.Header{"Content-Type": {"application/json; charset=utf-8"}}, data)
|
||||
}
|
||||
}
|
||||
func Envelope(data *Schema) *Schema {
|
||||
return &Schema{Type: "object", Required: []string{"code", "message"}, Properties: map[string]*Schema{"code": {Type: "string"}, "message": {Type: "string"}, "requestId": {Type: "string"}, "data": data}}
|
||||
}
|
||||
func SchemaOf[T any]() (*Schema, error) {
|
||||
return schema(reflect.TypeOf((*T)(nil)).Elem(), map[reflect.Type]bool{}, 0)
|
||||
}
|
||||
func schema(t reflect.Type, stack map[reflect.Type]bool, depth int) (*Schema, error) {
|
||||
if depth > 32 || stack[t] {
|
||||
return nil, errors.New("openapi: recursive DTO requires an explicit schema")
|
||||
}
|
||||
if t.Kind() == reflect.Pointer {
|
||||
s, err := schema(t.Elem(), stack, depth+1)
|
||||
if err == nil {
|
||||
s.Nullable = true
|
||||
}
|
||||
return s, err
|
||||
}
|
||||
if t == reflect.TypeOf(time.Time{}) {
|
||||
return &Schema{Type: "string", Format: "date-time"}, nil
|
||||
}
|
||||
if t == reflect.TypeOf(json.RawMessage{}) {
|
||||
return &Schema{}, nil
|
||||
}
|
||||
if t == reflect.TypeOf(json.Number("")) {
|
||||
return &Schema{Type: "number"}, nil
|
||||
}
|
||||
marshaler := reflect.TypeOf((*json.Marshaler)(nil)).Elem()
|
||||
if t.Implements(marshaler) || reflect.PointerTo(t).Implements(marshaler) {
|
||||
return nil, errors.New("openapi: custom JSON marshaler requires an explicit schema")
|
||||
}
|
||||
switch t.Kind() {
|
||||
case reflect.String:
|
||||
return &Schema{Type: "string"}, nil
|
||||
case reflect.Bool:
|
||||
return &Schema{Type: "boolean"}, nil
|
||||
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64:
|
||||
return &Schema{Type: "integer"}, nil
|
||||
case reflect.Float32, reflect.Float64:
|
||||
return &Schema{Type: "number"}, nil
|
||||
case reflect.Slice, reflect.Array:
|
||||
if t.Elem().Kind() == reflect.Uint8 && t.Kind() == reflect.Slice {
|
||||
return &Schema{Type: "string", Format: "byte"}, nil
|
||||
}
|
||||
s, err := schema(t.Elem(), stack, depth+1)
|
||||
return &Schema{Type: "array", Items: s}, err
|
||||
case reflect.Map:
|
||||
if t.Key().Kind() != reflect.String {
|
||||
return nil, errors.New("openapi: map keys must be strings")
|
||||
}
|
||||
s, err := schema(t.Elem(), stack, depth+1)
|
||||
return &Schema{Type: "object", AdditionalProperties: s}, err
|
||||
case reflect.Interface:
|
||||
return &Schema{}, nil
|
||||
case reflect.Struct:
|
||||
stack[t] = true
|
||||
defer delete(stack, t)
|
||||
s := &Schema{Type: "object", Properties: map[string]*Schema{}}
|
||||
for n := 0; n < t.NumField(); n++ {
|
||||
f := t.Field(n)
|
||||
if f.PkgPath != "" {
|
||||
continue
|
||||
}
|
||||
tag := strings.Split(f.Tag.Get("json"), ",")
|
||||
if tag[0] == "-" {
|
||||
continue
|
||||
}
|
||||
if f.Anonymous && tag[0] == "" {
|
||||
return nil, errors.New("openapi: embedded fields require an explicit JSON name")
|
||||
}
|
||||
name := tag[0]
|
||||
if name == "" {
|
||||
name = f.Name
|
||||
}
|
||||
child, err := schema(f.Type, stack, depth+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if s.Properties[name] != nil {
|
||||
return nil, errors.New("openapi: duplicate JSON field")
|
||||
}
|
||||
s.Properties[name] = child
|
||||
optional := false
|
||||
for _, option := range tag[1:] {
|
||||
if option == "string" {
|
||||
child.Type = "string"
|
||||
child.Format = ""
|
||||
}
|
||||
if option == "omitempty" || option == "omitzero" {
|
||||
optional = true
|
||||
}
|
||||
}
|
||||
if !optional {
|
||||
s.Required = append(s.Required, name)
|
||||
}
|
||||
}
|
||||
return s, nil
|
||||
default:
|
||||
return nil, fmt.Errorf("openapi: unsupported DTO type %s", t)
|
||||
}
|
||||
}
|
||||
54
openapi/openapi_test.go
Normal file
54
openapi/openapi_test.go
Normal file
@@ -0,0 +1,54 @@
|
||||
package openapi_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
omni "git.yuntuo.hk/k2safe/go-omni"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
"git.yuntuo.hk/k2safe/go-omni/openapi"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRouteAndContract(t *testing.T) {
|
||||
type item struct {
|
||||
ID int64 `json:"id,string"`
|
||||
RequestID string `json:"requestId,omitempty"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
}
|
||||
schema, err := openapi.SchemaOf[item]()
|
||||
if err != nil || schema.Properties["id"].Type != "string" || schema.Properties["payload"].Type != "" {
|
||||
t.Fatal(schema, err)
|
||||
}
|
||||
api, _ := openapi.New("Test API", "1")
|
||||
app := omni.New(omni.WithLogger(false))
|
||||
op := openapi.Operation{Method: "GET", Path: "/items/:id", OperationID: "getItem", Parameters: []openapi.Parameter{{Name: "id", In: "path", Required: true, Schema: &openapi.Schema{Type: "string"}}}, Responses: map[int]openapi.Response{200: {Description: "OK", Schema: openapi.Envelope(schema)}}}
|
||||
if err = api.Handle(app, op, func(c *httpx.Context) error { return c.OK(item{ID: 1, Payload: json.RawMessage(`{}`)}) }); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Caller edits must not mutate the published contract.
|
||||
schema.Properties["id"].Type = "boolean"
|
||||
if err = api.Handle(app, op, func(*httpx.Context) error { return nil }); err == nil {
|
||||
t.Fatal("duplicate accepted")
|
||||
}
|
||||
app.GET("/openapi.json", api.Handler())
|
||||
r := httptest.NewRecorder()
|
||||
app.ServeHTTP(r, httptest.NewRequest("GET", "/openapi.json", nil))
|
||||
var doc struct {
|
||||
OpenAPI string `json:"openapi"`
|
||||
Paths map[string]json.RawMessage `json:"paths"`
|
||||
}
|
||||
if err = json.Unmarshal(r.Body.Bytes(), &doc); err != nil || doc.OpenAPI != "3.0.3" || doc.Paths["/items/{id}"] == nil {
|
||||
t.Fatal(r.Body.String(), err)
|
||||
}
|
||||
r = httptest.NewRecorder()
|
||||
app.ServeHTTP(r, httptest.NewRequest("GET", "/items/1", nil))
|
||||
if r.Code != 200 {
|
||||
t.Fatal(r.Code)
|
||||
}
|
||||
type recursive struct {
|
||||
Next *recursive `json:"next"`
|
||||
}
|
||||
if _, err = openapi.SchemaOf[recursive](); err == nil {
|
||||
t.Fatal("recursive DTO silently truncated")
|
||||
}
|
||||
}
|
||||
73
pagination/pagination.go
Normal file
73
pagination/pagination.go
Normal file
@@ -0,0 +1,73 @@
|
||||
// Package pagination defines bounded offset pagination with camelCase fields.
|
||||
package pagination
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/url"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
DefaultSize int `json:"defaultSize"`
|
||||
MaxSize int `json:"maxSize"`
|
||||
MaxOffset int64 `json:"maxOffset"`
|
||||
}
|
||||
type Request struct {
|
||||
Page int64 `json:"page"`
|
||||
PageSize int `json:"pageSize"`
|
||||
}
|
||||
|
||||
func (r Request) Offset() int64 { return (r.Page - 1) * int64(r.PageSize) }
|
||||
func Parse(query url.Values, cfg Config) (Request, error) {
|
||||
if cfg.DefaultSize == 0 {
|
||||
cfg.DefaultSize = 20
|
||||
}
|
||||
if cfg.MaxSize == 0 {
|
||||
cfg.MaxSize = 100
|
||||
}
|
||||
if cfg.MaxOffset == 0 {
|
||||
cfg.MaxOffset = 1000000
|
||||
}
|
||||
if cfg.DefaultSize < 1 || cfg.MaxSize < cfg.DefaultSize || cfg.MaxSize > 10000 || cfg.MaxOffset < 0 {
|
||||
return Request{}, errors.New("pagination: invalid configuration")
|
||||
}
|
||||
page, size := int64(1), int64(cfg.DefaultSize)
|
||||
for key, target := range map[string]*int64{"page": &page, "pageSize": &size} {
|
||||
if v, ok := query[key]; ok {
|
||||
if len(v) != 1 {
|
||||
return Request{}, errors.New("pagination: duplicate parameter")
|
||||
}
|
||||
n, err := strconv.ParseInt(v[0], 10, 64)
|
||||
if err != nil || n < 1 {
|
||||
return Request{}, errors.New("pagination: positive integer required")
|
||||
}
|
||||
*target = n
|
||||
}
|
||||
}
|
||||
if size > int64(cfg.MaxSize) || (page-1) > cfg.MaxOffset/size {
|
||||
return Request{}, errors.New("pagination: size or offset exceeds limit")
|
||||
}
|
||||
return Request{Page: page, PageSize: int(size)}, nil
|
||||
}
|
||||
|
||||
type Result[T any] struct {
|
||||
Items []T `json:"items"`
|
||||
Page int64 `json:"page"`
|
||||
PageSize int `json:"pageSize"`
|
||||
Total int64 `json:"total"`
|
||||
TotalPages int64 `json:"totalPages"`
|
||||
}
|
||||
|
||||
func NewResult[T any](r Request, items []T, total int64) (Result[T], error) {
|
||||
if r.Page < 1 || r.PageSize < 1 || total < 0 || len(items) > r.PageSize || int64(len(items)) > total {
|
||||
return Result[T]{}, errors.New("pagination: invalid result")
|
||||
}
|
||||
if items == nil {
|
||||
items = []T{}
|
||||
}
|
||||
pages := total / int64(r.PageSize)
|
||||
if total%int64(r.PageSize) != 0 {
|
||||
pages++
|
||||
}
|
||||
return Result[T]{Items: items, Page: r.Page, PageSize: r.PageSize, Total: total, TotalPages: pages}, nil
|
||||
}
|
||||
30
pagination/pagination_test.go
Normal file
30
pagination/pagination_test.go
Normal file
@@ -0,0 +1,30 @@
|
||||
package pagination
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"math"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestBoundsAndEnvelope(t *testing.T) {
|
||||
for _, raw := range []string{"page=0", "page=1&page=2", "pageSize=101", "page=9223372036854775807&pageSize=100", "pageSize=-1"} {
|
||||
q, _ := url.ParseQuery(raw)
|
||||
if _, err := Parse(q, Config{}); err == nil {
|
||||
t.Fatal(raw)
|
||||
}
|
||||
}
|
||||
r, err := Parse(url.Values{"page": {"2"}, "pageSize": {"30"}}, Config{})
|
||||
if err != nil || r.Offset() != 30 {
|
||||
t.Fatal(r, err)
|
||||
}
|
||||
result, err := NewResult(r, []string(nil), math.MaxInt64)
|
||||
if err != nil || result.TotalPages != math.MaxInt64/30+1 {
|
||||
t.Fatal(result, err)
|
||||
}
|
||||
raw, _ := json.Marshal(result)
|
||||
if !strings.Contains(string(raw), `"items":[]`) || !strings.Contains(string(raw), `"pageSize":30`) {
|
||||
t.Fatal(string(raw))
|
||||
}
|
||||
}
|
||||
44
performance/advancedAcceptance.md
Normal file
44
performance/advancedAcceptance.md
Normal file
@@ -0,0 +1,44 @@
|
||||
# 扩展模块验收
|
||||
|
||||
代码提交:`208a9c3fbdb7ccf2ed409ba380050842e89829d8`。本报告与随后文档提交没有改变被测运行代码。
|
||||
|
||||
2026-09-23 在独立 Linux amd64 验证目录运行,开始 06:51:31 UTC,完成 06:55:56 UTC。工作树在启动时干净。原始 [JSON](advancedLinux.json) 记录 98 个源码/模板/脚本摘要、镜像 ID、命令和容器清理;[TXT](advancedLinux.txt) 保存实际输出。回传本机后已逐个核对源码摘要及日志 SHA-256。
|
||||
|
||||
```sh
|
||||
python3 scripts/verifyModules.py --advanced \
|
||||
--goproxy https://goproxy.cn,direct \
|
||||
--output performance/advancedLinux.json
|
||||
```
|
||||
|
||||
## 环境与隔离
|
||||
|
||||
- Go 1.25.14,Linux amd64。实际依赖版本以提交内 go.mod/go.sum 为准;新增依赖后已覆盖原有网络适配器回归。
|
||||
- Redis 7.4.11,MinIO RELEASE.2025-09-07T16-13-09Z,MySQL 8.4.11。
|
||||
- 编译、Redis、MinIO、MySQL 顺序运行,最多一个测试容器;每个硬限制 2 CPU / 2 GiB 内存与内存加 swap / 256 pids。
|
||||
- 数据服务阶段 `--network none`,仅容器回环/Unix socket,无发布端口,数据位于本次临时目录/tmpfs。没有连接现有业务数据库。
|
||||
- 四个测试容器已清理,原有 7 个业务容器均仍运行。
|
||||
|
||||
## 通过范围
|
||||
|
||||
| 项目 | 实际证据 |
|
||||
| --- | --- |
|
||||
| 全仓库检查 | `go test -race -count=1 -timeout=240s ./...`、`go vet ./...`、`go build ./...`;脚手架集成开关已开启,包含独立生成项目 |
|
||||
| 认证与配置 | JWT 过期/缺 exp/错误发行者或受众/算法混用/轮换、Session 撤销与状态隔离、文件环境覆盖、非法/静态配置拒绝、子日志级别更新 |
|
||||
| 契约与观测 | OpenAPI 路径参数与 DTO、重复注册拒绝、W3C 入站/出站关联、外层访问日志 traceId、HTTP 指标、Redis miss 错误保持、SQL spans |
|
||||
| 熔断与通用模块 | 冷却探针、迟到结果、响应体期间槽位、panic 释放、分页溢出与重复参数、审计白名单与写入失败 |
|
||||
| 组合示例 | nethttp/native/实验 HTTP2 的真实 TLS 401/403/201、对象内容、非法路径、OpenAPI 和关闭;原有 modules/reference 回归同批通过 |
|
||||
| 共享 Redis | 并发 claim 唯一赢家、错误 owner 拒绝、二进制响应回放、两个应用实例仅执行一次、冲突 409、共享限流 429、存储失败 503、过期目标不受批次清理顺序影响 |
|
||||
| Redis 队列 | 重报回执、内容冲突、容量拒绝、旧 owner 无法 ACK、保留未确认消息后自动接管、两个 Consumer 竞争、有限重试/永久拒绝、失败消息分页/受控删除、完成身份 TTL |
|
||||
| 对象存储 | Local 契约及失败上传保留原对象、拒绝符号链接、签名篡改;OSS SDK 协议 fixture;S3 SDK fixture 和 R2 endpoint/region/签名参数;真实 MinIO 上传/覆盖/读取/删除与错误签名 403 |
|
||||
| MySQL 迁移 | 全新测试库 up、重复 up 不重执行、版本读取、SQL 查询/更新观测、失败迁移保留 dirty、后续执行被阻断 |
|
||||
|
||||
真实 Redis/MinIO/MySQL 阶段使用 Go 编译的静态测试程序;该阶段没有启用 race。全仓库 race 阶段不连接这些外部数据服务。两类验证各自的范围不混为一项。
|
||||
|
||||
## 保留边界
|
||||
|
||||
- 本批按用户决定不集成 Orbit,未修改 Orbit 或业务项目,也没有用通用队列模拟其流程编排协议。
|
||||
- OSS/R2 没有真实云账号、bucket 权限、临时凭据轮换、地域/公网故障验收;MinIO 通过不能替代这些验证。
|
||||
- 队列接管通过真实 Redis 中的未确认 delivery 验证,没有本批 Worker 进程强杀、Redis AOF 重启/复制切换、长期负载或 exactly-once 证明。测试 Redis 关闭持久化以隔离协议检查,生产需另行配置持久化与故障验收。
|
||||
- 没有在生产业务上自动执行迁移;没有证明任意 DDL 可事务回滚。MySQL 失败保留 dirty,由部署流程核查。
|
||||
- OpenTelemetry 的采集/传播/exporter 接口通过本批测试,未连接实际 OTLP collector 或告警系统。
|
||||
- 这是扩展模块的首版功能验收,不是生产规模容量或完整运维验收。
|
||||
303
performance/advancedLinux.json
Normal file
303
performance/advancedLinux.json
Normal file
@@ -0,0 +1,303 @@
|
||||
{
|
||||
"status": "passed",
|
||||
"startedAt": "2026-09-23T06:51:31.468977+00:00",
|
||||
"testedCommit": "208a9c3fbdb7ccf2ed409ba380050842e89829d8",
|
||||
"workingTreeStatus": "",
|
||||
"sourceSha256": {
|
||||
"audit/audit.go": "48da5d6ecd498230d22db04413d379650b9810b3faee5fb5d52415ab6f681eea",
|
||||
"audit/audit_test.go": "7ecb96dcc8294c905465bb026030a2b0d2d5064359a9cb6b95199d5d15491c49",
|
||||
"auth/adapters_test.go": "a45dd935fd91b91d16435c97c3b6b6b82615a55bc16463eb94448704a2433fe1",
|
||||
"auth/auth.go": "067b75edf2d73338ceeff3b70698e6f5426606c5d510a7d0249bcf6fd7796eb8",
|
||||
"auth/auth_test.go": "158626ac86c504a337e37faad65b2bad77d68f54ce6b6ac2cda9d287aeefa963",
|
||||
"auth/jwt.go": "cfd4ead3712e921cb08d65b9ea81a06d1b6de420bc94cf240540a95959a2f241",
|
||||
"auth/policy.go": "f02cf32593033908921172190a65debaf9f8c0af616ceb95982ea94649170b62",
|
||||
"auth/session.go": "544c71e00c6af913722c1cca510e1fb2938925234c246e0ec52de22284a81b18",
|
||||
"cache/cache.go": "4aad0a644c0c9ce6353e61e70059715bea5402f0d7132b50003ee126c3836cad",
|
||||
"cache/cache_test.go": "c850cd8a90c1e6a8d701a415e283d4147fbd66d12d054002014cbf348863aea2",
|
||||
"cache/memory.go": "3ac2a710a0d4677ad58dd6817005033ef7cc939be7eb3e40476212ce7f18ad42",
|
||||
"cache/redis/redis.go": "3f522258da8a0fc6627645573221e9f5ccbdfea0fe8a1450985cfd084e2c18ce",
|
||||
"cache/redis/redis_test.go": "4948647b2d31803252af1ee4e4b96372b53f9fac46f03ab064b40abe2d5a0aa1",
|
||||
"cmd/go-omni/main.go": "811cd23a5288cd0a7efc132782d9509307f736afbcbbb2d001c59aad9d8633a9",
|
||||
"cmd/go-omni/main_test.go": "312e0dea1d46cbbd436acdb4fcac0704d4781746d9e8433f1203f67ef9601065",
|
||||
"cmd/go-omni/migrate.go": "f1ce89fbe419205eb82172da2b90e8888905d2b85d1a9db2a72b568bcf849a70",
|
||||
"cmd/go-omni/migrate_test.go": "15d9a4aaebf9f899e523e908f1fa3c2282eb1e6c8cadbefae6b68f1ae4e00e6e",
|
||||
"cmd/go-omni/scaffold/addModule.go": "8f43bd68c2b3820cf1d036c4f2c5f9d06d1c7c8e5114f5081597d905f8d8cdcc",
|
||||
"cmd/go-omni/scaffold/addModule_test.go": "667c45312f628f0fb5b8206ddef7ccc02a75f33b72f73fd6a17b03da0cb62e91",
|
||||
"cmd/go-omni/scaffold/moduleTemplates/README.md.tmpl": "9d3bfc46cac5618bcff7d8a6b2cc9f123e818aeb3dee9bcdfe0f91db584994bd",
|
||||
"cmd/go-omni/scaffold/moduleTemplates/handler-handler.go.tmpl": "729cd0f6faba53d5d2c1492935e12ac2adcefde4490748e25c18be7ccb3f7139",
|
||||
"cmd/go-omni/scaffold/moduleTemplates/model-model.go.tmpl": "a005611f6b455d4d8469e316bc26fa571043b2da1834db2b3dd14a96d66221d7",
|
||||
"cmd/go-omni/scaffold/moduleTemplates/module.go.tmpl": "fb0be7fd571ba2c0edb01d70d68cdbbfe53561caab98346bf1ac7cd8df07205a",
|
||||
"cmd/go-omni/scaffold/moduleTemplates/repository-repository.go.tmpl": "902da3d9f12395fb226178c4428ff547e1c2fce3261882dba98d1400d8c90747",
|
||||
"cmd/go-omni/scaffold/moduleTemplates/service-service.go.tmpl": "49d26044a69a43e983aa5ba4b5b2399b26b77898b23e230751ad1372c2af3178",
|
||||
"cmd/go-omni/scaffold/scaffold.go": "15baca1b5c4607c8f96fb6f6d4b96a661a86899012f93a6c380bc6d894b18803",
|
||||
"cmd/go-omni/scaffold/scaffold_test.go": "58aab9251b01e394bbc90aeb538ff035dd32c73c0d4815132ac6cbc086527cda",
|
||||
"cmd/go-omni/scaffold/templates/README.md.tmpl": "dc5ce58c49b4c0b50a87fdca794513cad38f428975bdfe772e89944e39614bd4",
|
||||
"cmd/go-omni/scaffold/templates/cmd-server-main.go.tmpl": "07c4f81109ece43fcf15bb807ada5ae6dc7e732babc7293269514a21cd389955",
|
||||
"cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl": "81c4f7fbc919d74b738d3a7338248cbd83cfa0803b2058e8ee184e3ce51e7e9b",
|
||||
"cmd/go-omni/scaffold/templates/go.mod.tmpl": "7d3bb15d2292e36939cc1905258d2800aeaf0e37d7d041b2ff8cfdb6fc84c57a",
|
||||
"cmd/go-omni/scaffold/templates/internal-handler-user.go.tmpl": "7f8dbcd2f9b0129a90160d6059e47d19d4758274060a1a3e5160055d0592d34f",
|
||||
"cmd/go-omni/scaffold/templates/internal-model-errors.go.tmpl": "f123eace52952321b0665a92a2923514c52c8c3bf13df1065710e9675750f837",
|
||||
"cmd/go-omni/scaffold/templates/internal-model-user.go.tmpl": "b391143e7386e169b53e7fed25de1146e8c52f3fa287b12728e8d22b33f386de",
|
||||
"cmd/go-omni/scaffold/templates/internal-repository-memory.go.tmpl": "148abc273c1fab3f28536706792762ae41e8687dfe9986c374372f162600feac",
|
||||
"cmd/go-omni/scaffold/templates/internal-repository-mysql.go.tmpl": "34bd333e29211a0025ff6ebaea18c7f142f18f1cc451d692f9d360e8c8f6f635",
|
||||
"cmd/go-omni/scaffold/templates/internal-repository-users.go.tmpl": "6d7dd80f34a9c215daeb4a86bfbe11e845e7ff7e9864c8af43e1c2ec3a8b28ec",
|
||||
"cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl": "3fde559ac634a6c85d322dbc3487691a192644268e85aa3911dd2561a4116196",
|
||||
"cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl": "2504d9750194a543121c9e93328f5f2a535f1db9f8dbd361edf3681cdfa21c39",
|
||||
"cmd/go-omni/scaffold/templates/schema.sql.tmpl": "7c5108f18fc7d8ff75c97afd96fcaa9f8ed026c79c71c5612f55572d73354abb",
|
||||
"cmd/go-omni/scaffold/version.go": "4d2d1b0d6d0ea716ff4cb7ff08cc25e6f91e06b1a4585176e3128a22e2f2f4ad",
|
||||
"config/config.go": "5cbdf774afa62001910bac3d6523399dc5c9dfdafc6111ab86bd5ce1502d7811",
|
||||
"config/config_test.go": "53b72d5bbe2b5a38c0e3c17bd426f1bb5b43beb2014e7391aef9c438d9e521ac",
|
||||
"config/env.go": "2c05cc2d97ae778158a52d390bcec4226d4196833e1e051a76a2eeb425997c0b",
|
||||
"config/file.go": "2e9f0f9ae30f995994ba0841c27bd27ede7f0652b2c822d71140f439e023d95e",
|
||||
"config/file_test.go": "1479d625aad26a1f773bc4a08c146347f163bbf7b4f11385a32f687119df5817",
|
||||
"database/migrate/migrate.go": "fd6cd27d2dce5b35ffe80eb5442c47c657b93ffc6545bbcc1ee4651a70a4f80b",
|
||||
"database/migrate/migrate_test.go": "d7ba91eed8cfd97e4ef0741d582f493dd411b65ad3192f05b167f75d9b9dcf90",
|
||||
"examples/modules/main.go": "6b44b981c0f654e241664c94612cd95ba8a394155b589add0814d81397156a26",
|
||||
"examples/modules/main_test.go": "4ceb869a7e1ab3fb037af5411ec3dec47af66738adb66c4db01498d446aab7fb",
|
||||
"examples/platform/main.go": "f4fea7a6cdb37413ccab847de68fcbcf8608142858005e1c75754692e2c676d9",
|
||||
"examples/platform/main_test.go": "fd120cfcda2dcaeef5038bdee0ffaaff88bec4450e285ecb4a4de458230d7a12",
|
||||
"go.mod": "f025c81671bfe1399028ce52b934c3b9c264a788c563dfe63a8eb3bc1c458093",
|
||||
"go.sum": "8db5712ded87123d25b155ae67c5c5b168e8dc39d84e00a256766c5ef199cb76",
|
||||
"httpc/client.go": "bbec35b6234a554a4aefd0fef97ace3850288642faab2e382dcbe19a6cb95cd8",
|
||||
"httpc/client_test.go": "972ebd2efab618629e394484c288afe4e675fcfe0c50121d9fa9071680d6e763",
|
||||
"httpx/context.go": "104434499f074eedf2c54e4d0e90db587feb230ca8bb76803b87f2e42d9a8118",
|
||||
"internal/storagefixture/server.go": "defacc905246a643faf5371e437f9ce458f05461e0a922766034d1f9ffdaf003",
|
||||
"logx/logx.go": "1307adf3758967651dd802b814fc0f5804280163a4dc463478bf3667af641740",
|
||||
"logx/logx_test.go": "0f99eff445decdd81a55b3d5374c22d2e038d2a4748adb2c9367d6d03e481671",
|
||||
"middleware/idempotency.go": "a67e0040cbb15cd0140379154017e8f1549b06fc1f307c2549dad515d977e05f",
|
||||
"middleware/middleware.go": "a1e6cfae40d3e1701238a680009d66121f1bbaed786e81b3f6133eb1198a1cf6",
|
||||
"middleware/middleware_test.go": "9249aac16f251858bfe7937f768dc628bf4cc392af75f90223ccacd0f13d2cb9",
|
||||
"middleware/protection_test.go": "4347bb6074078c4c7bd9f65f5fa13a0ac2bc9a8615b297588cc3358aede69a7e",
|
||||
"middleware/rateLimit.go": "fa78a39495a432b3c3349915421a60bed8a76b0921d1f11d7962171fcb1e25ae",
|
||||
"middleware/redis/store.go": "168cac5588ed56e6ef33f2a5de6f8e91599ddf0583f509dd46f5210940fb148a",
|
||||
"middleware/redis/store_test.go": "6659dd4f486da08f4bffb27c698f18c67c3f8c3b59d9f3ebc68e7cd79b9dae15",
|
||||
"middleware/request_id.go": "d52bf17bf80245bafe878026b044857442f0af8df689bcaed4d967d006cbd947",
|
||||
"middleware/sharedIdempotency.go": "2fcc5dec3564680ee8039a8d43451948165553524c9c85248ec7218a5fa68ced",
|
||||
"middleware/timeout.go": "e2f446d992c9f0c986deb75fa71297f19c78f8e43593f174cd19fe93e385e440",
|
||||
"openapi/openapi.go": "13cc2a4fcaa4a50e6f38a38a3d168b6240dcdb49b2ebb5a2782f744f2635857d",
|
||||
"openapi/openapi_test.go": "c92f4a2f548f0304708ac157402fb8641273473e04c524226f2429c2b64b99ab",
|
||||
"pagination/pagination.go": "5c3ca00d27769dcef18d234ac14d5ccc635759ab5be77ef0c765aeba66d39532",
|
||||
"pagination/pagination_test.go": "f78fa86928d4d34a763e96423b29148bc74403aaf58c04d58fe29e225f6598eb",
|
||||
"queue/queue.go": "99a0a8d577dc401d7e69d01602c1889d93d370514bc333111706602044de158a",
|
||||
"queue/redis/consumer.go": "fdb9163db4917f27e4d716e5b42c64dc9694809c460e53b0bff034ebe2859be6",
|
||||
"queue/redis/consumer_test.go": "ea10f5e4804a3350bcee8bdc4bd0debe42820bd3451e28e36f6afc6aeadc0a46",
|
||||
"queue/redis/store.go": "ab5320bcdc7e80c7f3fdc77d583d895055d8970e0a22dbd78bac9b2801cfc634",
|
||||
"resilience/breaker.go": "ca1052185b7dd92e0c289c50aab7f37fb0e6aec6c4c214ae02dff7b8eb91b137",
|
||||
"resilience/breaker_test.go": "6f5abc9534d2bf2506afa3f79fcbb2efb18fa9d488cb010e870c42aa1c8c20d1",
|
||||
"resilience/http.go": "b84a64cef10943af51035c4fc19569b06cde0872046adf1bd1569b6b38170df4",
|
||||
"scripts/verifyModules.py": "ce68eb1f095bbaf721f924ec67c6fa48e0669303f3cdd83be56df06f97e18277",
|
||||
"storage/local/local.go": "ce5fac2e54680b96db3e7dfd9a8d065888f246c5607882607d79c302ee3d231b",
|
||||
"storage/local/local_test.go": "3a2241840968df8e37fd67169c3ef708228449e81f65aff908c8b60d49434dc0",
|
||||
"storage/oss/oss.go": "4b189138172ff42fb8c7549750eea59286c52bcc336f2e4da9d3831cdf3b41b9",
|
||||
"storage/oss/oss_test.go": "555a954010b1218804dc7951fc75e42eae24eba9a99c0871f5ba787f3fc87d45",
|
||||
"storage/s3/config.go": "626ec6a2c0b06c95ee7eb7aa0832b190cfcdeb8a127b6b89e99204040bc3b041",
|
||||
"storage/s3/s3.go": "a9f92c4c59930dd6bf9ec630bcb128ec46bef335f97704ca47647a9b73e15a58",
|
||||
"storage/s3/s3_test.go": "da670fb98a31ff1498d02682cdcd41e3adbab22a44b37f88907b1ea93ab2977f",
|
||||
"storage/signed.go": "72f0255c6ddd275298ba1b9fd50eb48c1858cc8102c43f6764e46f209fffb634",
|
||||
"storage/signed_test.go": "eaffd40894018814e976477f96cf0fbaecfa1987caad78830d9fc8e96995882f",
|
||||
"storage/storage.go": "cb670872f01cf1d7b4ebf21cec9fc6f52ad93bfdf523b2f28bbb8d9c7fe39a6c",
|
||||
"storage/storagetest/contract.go": "e024c5cd3c8a6c2ed656bad370a707f6960f4e717bc5d7dfd5624f6076c00605",
|
||||
"task/task.go": "ef671a9b7d84e7f53cdb2dcb787881e47b00ef0afb29fb6e4698fa8765a7bfe7",
|
||||
"task/task_test.go": "6e33ecd81345ece6102f48e60050a0d990e1bf75c3840e262d5da2962c743681",
|
||||
"telemetry/adapters.go": "a892ef0c2b268caf266a3ea0f833ecf7e9228ff59aeaeecae6fe4a2170ba51e2",
|
||||
"telemetry/telemetry.go": "5567b9d68f52ca54e34568c5f0c9a5b985d90c0c4b5224e57622d6f60adebe69",
|
||||
"telemetry/telemetry_test.go": "7040df811a5cbf9108af3d5e0a98639d1e31efdc8ce4eee6ac0332f3a58691de"
|
||||
},
|
||||
"containerLimits": {
|
||||
"cpus": 2,
|
||||
"memoryBytes": 2147483648,
|
||||
"memoryAndSwapBytes": 2147483648,
|
||||
"pidsLimit": 256
|
||||
},
|
||||
"wallTimeoutSeconds": 900,
|
||||
"imageIds": {
|
||||
"golang:1.25": "sha256:699337d620559a59b4a2bb298ad59611e535d2ee755a34cf2d2a98f37578dc80",
|
||||
"redis:7.4-alpine": "sha256:858f009f9709ce576febc734aa78b8f6d624b82571f9ddb6bda4377c833b3499",
|
||||
"quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"m.daocloud.io/docker.io/library/mysql:8.4": "sha256:b3b90af2a6552ae30c266fdb7d5dd55f3afb72404bb78d37fe8a23eb857fd3fb"
|
||||
},
|
||||
"commands": [
|
||||
[
|
||||
"/usr/bin/docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"--name",
|
||||
"go-omni-modules-7387653de963-build",
|
||||
"--cpus",
|
||||
"2",
|
||||
"--memory",
|
||||
"2048m",
|
||||
"--memory-swap",
|
||||
"2048m",
|
||||
"--pids-limit",
|
||||
"256",
|
||||
"--log-opt",
|
||||
"max-size=10m",
|
||||
"--log-opt",
|
||||
"max-file=1",
|
||||
"-v",
|
||||
"/root/go-omni-validation:/workspace:ro",
|
||||
"-v",
|
||||
"/tmp/goOmniModulesc4suqkr6:/output",
|
||||
"-w",
|
||||
"/workspace",
|
||||
"-e",
|
||||
"GOWORK=off",
|
||||
"-e",
|
||||
"GOMAXPROCS=2",
|
||||
"-e",
|
||||
"GO_OMNI_SCAFFOLD_INTEGRATION=1",
|
||||
"-e",
|
||||
"GOPROXY=https://goproxy.cn,direct",
|
||||
"golang:1.25",
|
||||
"sh",
|
||||
"-c",
|
||||
"go version && go test -race -count=1 -timeout=240s ./... && go vet ./... && go build ./... && CGO_ENABLED=0 go test -c -o /output/redis.test ./cache/redis && CGO_ENABLED=0 go test -c -o /output/shared.test ./middleware/redis && CGO_ENABLED=0 go test -c -o /output/queue.test ./queue/redis && CGO_ENABLED=0 go test -c -o /output/s3.test ./storage/s3 && CGO_ENABLED=0 go test -c -o /output/mysql.test ./database/migrate"
|
||||
],
|
||||
[
|
||||
"/usr/bin/docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"--name",
|
||||
"go-omni-modules-7387653de963-redis",
|
||||
"--cpus",
|
||||
"2",
|
||||
"--memory",
|
||||
"2048m",
|
||||
"--memory-swap",
|
||||
"2048m",
|
||||
"--pids-limit",
|
||||
"256",
|
||||
"--log-opt",
|
||||
"max-size=10m",
|
||||
"--log-opt",
|
||||
"max-file=1",
|
||||
"--network",
|
||||
"none",
|
||||
"--read-only",
|
||||
"--cap-drop",
|
||||
"ALL",
|
||||
"--security-opt",
|
||||
"no-new-privileges",
|
||||
"--user",
|
||||
"999:999",
|
||||
"--tmpfs",
|
||||
"/tmp:rw,nosuid,noexec,size=64m,mode=1777",
|
||||
"-v",
|
||||
"/tmp/goOmniModulesc4suqkr6:/test:ro",
|
||||
"--entrypoint",
|
||||
"sh",
|
||||
"-e",
|
||||
"GOMAXPROCS=2",
|
||||
"-e",
|
||||
"GO_OMNI_REDIS_TEST_ADDR=127.0.0.1:6379",
|
||||
"redis:7.4-alpine",
|
||||
"-c",
|
||||
"set -eu\nredis-server --version\nredis-server --bind 127.0.0.1 --port 6379 --save '' --appendonly no --maxmemory 32mb --daemonize yes --pidfile /tmp/redis.pid --logfile /tmp/redis.log --dir /tmp\ntrap 'redis-cli -h 127.0.0.1 shutdown nosave >/dev/null 2>&1 || true' EXIT\nattempt=0\nuntil redis-cli -h 127.0.0.1 ping >/dev/null 2>&1; do\n attempt=$((attempt + 1))\n if [ \"$attempt\" -gt 50 ]; then cat /tmp/redis.log; exit 1; fi\n sleep 0.1\ndone\n/test/redis.test -test.v -test.run '^TestRedisIntegration$' -test.count=1 -test.timeout=30s\n/test/shared.test -test.v -test.count=1 -test.timeout=45s\n/test/queue.test -test.v -test.count=1 -test.timeout=45s\n"
|
||||
],
|
||||
[
|
||||
"/usr/bin/docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"--name",
|
||||
"go-omni-modules-7387653de963-s3",
|
||||
"--cpus",
|
||||
"2",
|
||||
"--memory",
|
||||
"2048m",
|
||||
"--memory-swap",
|
||||
"2048m",
|
||||
"--pids-limit",
|
||||
"256",
|
||||
"--log-opt",
|
||||
"max-size=10m",
|
||||
"--log-opt",
|
||||
"max-file=1",
|
||||
"--network",
|
||||
"none",
|
||||
"--read-only",
|
||||
"--cap-drop",
|
||||
"ALL",
|
||||
"--security-opt",
|
||||
"no-new-privileges",
|
||||
"-v",
|
||||
"/tmp/goOmniModulesc4suqkr6:/test:ro",
|
||||
"--entrypoint",
|
||||
"sh",
|
||||
"-e",
|
||||
"GOMAXPROCS=2",
|
||||
"--user",
|
||||
"1000:1000",
|
||||
"--tmpfs",
|
||||
"/tmp:rw,nosuid,noexec,size=64m,mode=1777",
|
||||
"--tmpfs",
|
||||
"/data:rw,nosuid,noexec,size=128m,mode=1777",
|
||||
"-e",
|
||||
"MINIO_ROOT_USER=goOmniTest",
|
||||
"-e",
|
||||
"MINIO_ROOT_PASSWORD=goOmniTestSecret",
|
||||
"-e",
|
||||
"GO_OMNI_S3_TEST_ADDR=127.0.0.1:9000",
|
||||
"quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z",
|
||||
"-c",
|
||||
"set -eu\nminio --version\nminio server /data --address 127.0.0.1:9000 --console-address 127.0.0.1:9001 >/tmp/minio.log 2>&1 &\nserver=$!\ntrap 'kill \"$server\" 2>/dev/null || true; wait \"$server\" 2>/dev/null || true' EXIT\n/test/s3.test -test.v -test.run '^TestS3Integration$' -test.count=1 -test.timeout=45s || { cat /tmp/minio.log; exit 1; }\n"
|
||||
],
|
||||
[
|
||||
"/usr/bin/docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"--name",
|
||||
"go-omni-modules-7387653de963-mysql",
|
||||
"--cpus",
|
||||
"2",
|
||||
"--memory",
|
||||
"2048m",
|
||||
"--memory-swap",
|
||||
"2048m",
|
||||
"--pids-limit",
|
||||
"256",
|
||||
"--log-opt",
|
||||
"max-size=10m",
|
||||
"--log-opt",
|
||||
"max-file=1",
|
||||
"--network",
|
||||
"none",
|
||||
"--read-only",
|
||||
"--cap-drop",
|
||||
"ALL",
|
||||
"--security-opt",
|
||||
"no-new-privileges",
|
||||
"-v",
|
||||
"/tmp/goOmniModulesc4suqkr6:/test:ro",
|
||||
"--entrypoint",
|
||||
"sh",
|
||||
"-e",
|
||||
"GOMAXPROCS=2",
|
||||
"--user",
|
||||
"999:999",
|
||||
"--tmpfs",
|
||||
"/tmp:rw,nosuid,noexec,size=512m,mode=1777",
|
||||
"--tmpfs",
|
||||
"/var/lib/mysql:rw,nosuid,noexec,size=1m,mode=1777",
|
||||
"-e",
|
||||
"GO_OMNI_MIGRATE_TEST_DSN=root@unix(/tmp/mysql.sock)/omni_test?parseTime=true&timeout=2s&readTimeout=5s&writeTimeout=5s",
|
||||
"m.daocloud.io/docker.io/library/mysql:8.4",
|
||||
"-c",
|
||||
"set -eu\nmysqld --version\nmkdir /tmp/mysql\nmysqld --no-defaults --initialize-insecure --datadir=/tmp/mysql --innodb-redo-log-capacity=67108864 >/tmp/init.log 2>&1 || { cat /tmp/init.log; exit 1; }\nmysqld --no-defaults --datadir=/tmp/mysql --socket=/tmp/mysql.sock --pid-file=/tmp/mysql.pid --bind-address=127.0.0.1 --port=3306 --mysqlx=OFF --skip-log-bin --skip-name-resolve --innodb-buffer-pool-size=134217728 --innodb-redo-log-capacity=67108864 --max-connections=8 --log-error=/tmp/mysql.log &\nserver=$!\ntrap 'kill \"$server\" 2>/dev/null || true; wait \"$server\" 2>/dev/null || true' EXIT\nattempt=0\nuntil mysql --no-defaults --socket=/tmp/mysql.sock -uroot -e 'SELECT 1' >/dev/null 2>&1; do\n attempt=$((attempt + 1)); if [ \"$attempt\" -gt 200 ]; then cat /tmp/mysql.log; exit 1; fi; sleep 0.1\ndone\nmysql --no-defaults --socket=/tmp/mysql.sock -uroot -e 'CREATE DATABASE omni_test'\n/test/mysql.test -test.v -test.run '^TestMySQLIntegration$' -test.count=1 -test.timeout=60s\n"
|
||||
]
|
||||
],
|
||||
"cleanedContainers": [
|
||||
"go-omni-modules-7387653de963-build",
|
||||
"go-omni-modules-7387653de963-redis",
|
||||
"go-omni-modules-7387653de963-s3",
|
||||
"go-omni-modules-7387653de963-mysql"
|
||||
],
|
||||
"notes": [
|
||||
"Full repository race/vet/build plus independent generated-module CRUD in the compiler container.",
|
||||
"Real Redis uses a statically linked non-race test binary, loopback only, network=none and no published ports.",
|
||||
"At most one validation container; no pre-existing service or database is accessed."
|
||||
],
|
||||
"logSha256": "e76d05ff859f96330f19e4cb2a96241cf5d63cf118abd80db16fe67ff3a1c51f",
|
||||
"finishedAt": "2026-09-23T06:55:56.890295+00:00",
|
||||
"preExistingContainersStillRunning": 7
|
||||
}
|
||||
119
performance/advancedLinux.txt
Normal file
119
performance/advancedLinux.txt
Normal file
@@ -0,0 +1,119 @@
|
||||
go version go1.25.14 linux/amd64
|
||||
go: downloading github.com/golang-jwt/jwt/v5 v5.3.1
|
||||
go: downloading github.com/redis/go-redis/v9 v9.17.3
|
||||
go: downloading golang.org/x/mod v0.21.0
|
||||
go: downloading github.com/go-sql-driver/mysql v1.9.3
|
||||
go: downloading github.com/golang-migrate/migrate/v4 v4.18.1
|
||||
go: downloading gopkg.in/yaml.v3 v3.0.1
|
||||
go: downloading go.opentelemetry.io/otel/sdk v1.31.0
|
||||
go: downloading go.opentelemetry.io/otel v1.31.0
|
||||
go: downloading github.com/aliyun/alibabacloud-oss-go-sdk-v2 v1.6.0
|
||||
go: downloading github.com/minio/minio-go/v7 v7.0.83
|
||||
go: downloading golang.org/x/net v0.33.0
|
||||
go: downloading github.com/go-playground/validator/v10 v10.20.0
|
||||
go: downloading go.opentelemetry.io/otel/metric v1.31.0
|
||||
go: downloading go.opentelemetry.io/otel/sdk/metric v1.31.0
|
||||
go: downloading go.opentelemetry.io/otel/trace v1.31.0
|
||||
go: downloading filippo.io/edwards25519 v1.1.0
|
||||
go: downloading github.com/cespare/xxhash/v2 v2.3.0
|
||||
go: downloading github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f
|
||||
go: downloading github.com/hashicorp/go-multierror v1.1.1
|
||||
go: downloading go.uber.org/atomic v1.7.0
|
||||
go: downloading golang.org/x/time v0.5.0
|
||||
go: downloading github.com/goccy/go-json v0.10.4
|
||||
go: downloading github.com/google/uuid v1.6.0
|
||||
go: downloading github.com/klauspost/compress v1.17.11
|
||||
go: downloading github.com/minio/md5-simd v1.1.2
|
||||
go: downloading github.com/go-ini/ini v1.67.0
|
||||
go: downloading github.com/gabriel-vasile/mimetype v1.4.3
|
||||
go: downloading github.com/go-playground/universal-translator v0.18.1
|
||||
go: downloading github.com/leodido/go-urn v1.4.0
|
||||
go: downloading golang.org/x/crypto v0.31.0
|
||||
go: downloading golang.org/x/text v0.21.0
|
||||
go: downloading golang.org/x/sys v0.28.0
|
||||
go: downloading github.com/hashicorp/errwrap v1.1.0
|
||||
go: downloading github.com/klauspost/cpuid/v2 v2.2.9
|
||||
go: downloading github.com/dustin/go-humanize v1.0.1
|
||||
go: downloading github.com/rs/xid v1.6.0
|
||||
go: downloading github.com/go-playground/locales v0.14.1
|
||||
go: downloading github.com/go-logr/logr v1.4.2
|
||||
go: downloading github.com/go-logr/stdr v1.2.2
|
||||
ok git.yuntuo.hk/k2safe/go-omni 1.098s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/app 1.652s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/audit 1.016s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/auth 1.048s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/cache 1.034s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/cache/redis 1.014s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni 1.194s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/scaffold 4.092s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/config 1.042s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/database 1.013s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/database/migrate 1.053s
|
||||
? git.yuntuo.hk/k2safe/go-omni/errors [no test files]
|
||||
? git.yuntuo.hk/k2safe/go-omni/examples/basic [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/examples/modules 1.072s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/examples/platform 1.105s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/examples/reference 1.156s
|
||||
? git.yuntuo.hk/k2safe/go-omni/examples/reference/handler [no test files]
|
||||
? git.yuntuo.hk/k2safe/go-omni/examples/reference/model [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/examples/reference/repository 1.016s
|
||||
? git.yuntuo.hk/k2safe/go-omni/examples/reference/service [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/health 1.040s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/httpc 1.102s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/httpx 1.033s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/internal/perfcheck 1.050s
|
||||
? git.yuntuo.hk/k2safe/go-omni/internal/storagefixture [no test files]
|
||||
? git.yuntuo.hk/k2safe/go-omni/internal/testutil [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/logx 1.013s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/metrics 1.054s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/middleware 1.059s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/middleware/redis 1.036s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/openapi 1.036s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/pagination 1.013s
|
||||
? git.yuntuo.hk/k2safe/go-omni/queue [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/queue/redis 1.014s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/resilience 1.014s
|
||||
? git.yuntuo.hk/k2safe/go-omni/response [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/storage 1.020s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/storage/local 1.021s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/storage/oss 1.029s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/storage/s3 1.035s
|
||||
? git.yuntuo.hk/k2safe/go-omni/storage/storagetest [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/task 1.092s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/telemetry 1.038s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/transport 1.053s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/transport/http2 10.647s
|
||||
? git.yuntuo.hk/k2safe/go-omni/transport/internal/routing [no test files]
|
||||
ok git.yuntuo.hk/k2safe/go-omni/transport/native 2.513s
|
||||
ok git.yuntuo.hk/k2safe/go-omni/transport/nethttp 1.071s
|
||||
? git.yuntuo.hk/k2safe/go-omni/transport/transporttest [no test files]
|
||||
? git.yuntuo.hk/k2safe/go-omni/validate [no test files]
|
||||
Redis server v=7.4.11 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64 build=40ff01a501d8e4b6
|
||||
=== RUN TestRedisIntegration
|
||||
--- PASS: TestRedisIntegration (0.09s)
|
||||
PASS
|
||||
=== RUN TestSharedRedisIntegration
|
||||
--- PASS: TestSharedRedisIntegration (0.01s)
|
||||
=== RUN TestExpiredTargetBeyondCleanupBatch
|
||||
--- PASS: TestExpiredTargetBeyondCleanupBatch (0.03s)
|
||||
PASS
|
||||
=== RUN TestQueueRedisIntegration
|
||||
=== RUN TestQueueRedisIntegration/publicationAndCapacity
|
||||
=== RUN TestQueueRedisIntegration/staleAcknowledgement
|
||||
=== RUN TestQueueRedisIntegration/crashReclaimRetryAndDeadLetters
|
||||
--- PASS: TestQueueRedisIntegration (1.11s)
|
||||
--- PASS: TestQueueRedisIntegration/publicationAndCapacity (0.00s)
|
||||
--- PASS: TestQueueRedisIntegration/staleAcknowledgement (0.00s)
|
||||
--- PASS: TestQueueRedisIntegration/crashReclaimRetryAndDeadLetters (1.11s)
|
||||
PASS
|
||||
minio version RELEASE.2025-09-07T16-13-09Z (commit-id=07c3a429bfed433e49018cb0f78a52145d4bedeb)
|
||||
Runtime: go1.24.6 linux/amd64
|
||||
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
|
||||
Copyright: 2015-2025 MinIO, Inc.
|
||||
=== RUN TestS3Integration
|
||||
--- PASS: TestS3Integration (0.33s)
|
||||
PASS
|
||||
/usr/sbin/mysqld Ver 8.4.11 for Linux on x86_64 (MySQL Community Server - GPL)
|
||||
=== RUN TestMySQLIntegration
|
||||
--- PASS: TestMySQLIntegration (0.03s)
|
||||
PASS
|
||||
62
queue/queue.go
Normal file
62
queue/queue.go
Normal file
@@ -0,0 +1,62 @@
|
||||
// Package queue defines durable, at-least-once work delivery. Handlers must be
|
||||
// idempotent: a crash after a business commit but before ACK can cause redelivery.
|
||||
// This is a message queue, not a workflow engine or a distributed transaction.
|
||||
package queue
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var ErrFull = errors.New("queue: capacity reached")
|
||||
var ErrConflict = errors.New("queue: message ID already has different content")
|
||||
var ErrLeaseLost = errors.New("queue: delivery ownership changed")
|
||||
|
||||
type Message struct {
|
||||
ID string `json:"id"` // Stable producer identity, saved before submitting.
|
||||
Name string `json:"name"`
|
||||
Payload json.RawMessage `json:"payload"`
|
||||
}
|
||||
type Receipt struct {
|
||||
ID string `json:"id"`
|
||||
BackendID string `json:"backendId"`
|
||||
}
|
||||
type Delivery struct {
|
||||
Message
|
||||
Attempt int64 `json:"attempt"`
|
||||
}
|
||||
type Handler func(context.Context, Delivery) error
|
||||
type Publisher interface {
|
||||
Publish(context.Context, Message) (Receipt, error)
|
||||
}
|
||||
type Consumer interface {
|
||||
Run(context.Context, Handler) error
|
||||
}
|
||||
|
||||
// Reject marks a known permanent failure; it is retained for operator review.
|
||||
// Ordinary errors are retried up to the consumer's configured attempt limit.
|
||||
type Rejection struct{ Code string }
|
||||
|
||||
func (e *Rejection) Error() string { return "queue: rejected: " + e.Code }
|
||||
func Reject(code string) error { return &Rejection{Code: code} }
|
||||
|
||||
func Validate(ctx context.Context, m Message, maxBytes int) error {
|
||||
if ctx == nil {
|
||||
return errors.New("queue: context required")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, v := range []string{m.ID, m.Name} {
|
||||
if v == "" || len(v) > 128 || strings.IndexFunc(v, func(r rune) bool { return r < 33 || r > 126 }) >= 0 {
|
||||
return errors.New("queue: ID and name must be 1..128 printable ASCII characters")
|
||||
}
|
||||
}
|
||||
if len(m.Payload) > maxBytes || !json.Valid(m.Payload) {
|
||||
return fmt.Errorf("queue: payload must be valid JSON within %d bytes", maxBytes)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
213
queue/redis/consumer.go
Normal file
213
queue/redis/consumer.go
Normal file
@@ -0,0 +1,213 @@
|
||||
package redis
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/queue"
|
||||
goredis "github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
type ConsumerConfig struct {
|
||||
Workers int `json:"workers"`
|
||||
MaxAttempts int64 `json:"maxAttempts"`
|
||||
TaskTimeout time.Duration `json:"taskTimeout"`
|
||||
VisibilityTimeout time.Duration `json:"visibilityTimeout"`
|
||||
ShutdownTimeout time.Duration `json:"shutdownTimeout"`
|
||||
}
|
||||
type Consumer struct {
|
||||
store *Store
|
||||
cfg ConsumerConfig
|
||||
used atomic.Bool
|
||||
}
|
||||
|
||||
func NewConsumer(store *Store, cfg ConsumerConfig) (*Consumer, error) {
|
||||
if store == nil || cfg.Workers < 0 || cfg.MaxAttempts < 0 || cfg.TaskTimeout < 0 || cfg.VisibilityTimeout < 0 || cfg.ShutdownTimeout < 0 {
|
||||
return nil, errors.New("queue/redis: invalid consumer configuration")
|
||||
}
|
||||
if cfg.Workers == 0 {
|
||||
cfg.Workers = 4
|
||||
}
|
||||
if cfg.MaxAttempts == 0 {
|
||||
cfg.MaxAttempts = 3
|
||||
}
|
||||
if cfg.TaskTimeout == 0 {
|
||||
cfg.TaskTimeout = 30 * time.Second
|
||||
}
|
||||
if cfg.VisibilityTimeout == 0 {
|
||||
cfg.VisibilityTimeout = 2 * cfg.TaskTimeout
|
||||
}
|
||||
if cfg.ShutdownTimeout == 0 {
|
||||
cfg.ShutdownTimeout = 5 * time.Second
|
||||
}
|
||||
if cfg.Workers > 128 || cfg.MaxAttempts > 100 || cfg.TaskTimeout < 10*time.Millisecond || cfg.TaskTimeout > time.Hour || cfg.VisibilityTimeout < 2*cfg.TaskTimeout || cfg.VisibilityTimeout > 24*time.Hour || cfg.ShutdownTimeout > time.Minute {
|
||||
return nil, errors.New("queue/redis: consumer limits exceeded")
|
||||
}
|
||||
return &Consumer{store: store, cfg: cfg}, nil
|
||||
}
|
||||
|
||||
// Run is called once. Cancellation stops fetching and cancels handlers; ACK is
|
||||
// deferred to a later delivery if cancellation makes the outcome uncertain.
|
||||
// A handler must honor context cancellation. Go cannot terminate an arbitrary
|
||||
// function; an ignoring handler can outlive ShutdownTimeout (at most Workers).
|
||||
func (c *Consumer) Run(ctx context.Context, handler queue.Handler) error {
|
||||
if ctx == nil || handler == nil {
|
||||
return errors.New("queue/redis: context and handler required")
|
||||
}
|
||||
if !c.used.CompareAndSwap(false, true) {
|
||||
return errors.New("queue/redis: consumer already run")
|
||||
}
|
||||
signature := fmt.Sprintf("%d:%d:%d", c.cfg.MaxAttempts, c.cfg.TaskTimeout.Milliseconds(), c.cfg.VisibilityTimeout.Milliseconds())
|
||||
if err := goredis.NewScript(`local v=redis.call('GET',KEYS[1]);if v and v~=ARGV[1] then return redis.error_reply('consumer configuration mismatch') end;redis.call('SET',KEYS[1],ARGV[1]);return 1`).Run(ctx, c.store.client, []string{c.store.keys[5]}, signature).Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.store.client.XGroupCreateMkStream(ctx, c.store.keys[0], "workers", "0").Err(); err != nil && !strings.HasPrefix(err.Error(), "BUSYGROUP ") {
|
||||
return err
|
||||
}
|
||||
var token [16]byte
|
||||
if _, err := rand.Read(token[:]); err != nil {
|
||||
return err
|
||||
}
|
||||
runCtx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
failures := make(chan error, c.cfg.Workers)
|
||||
var wg sync.WaitGroup
|
||||
for n := 0; n < c.cfg.Workers; n++ {
|
||||
wg.Add(1)
|
||||
go func(n int) {
|
||||
defer wg.Done()
|
||||
if err := c.loop(runCtx, hex.EncodeToString(token[:])+fmt.Sprint(n), handler); err != nil {
|
||||
failures <- err
|
||||
cancel()
|
||||
}
|
||||
}(n)
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { wg.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-runCtx.Done():
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(c.cfg.ShutdownTimeout):
|
||||
return errors.New("queue/redis: handler shutdown timeout")
|
||||
}
|
||||
}
|
||||
select {
|
||||
case err := <-failures:
|
||||
return err
|
||||
default:
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
func (c *Consumer) loop(ctx context.Context, owner string, handler queue.Handler) error {
|
||||
s := c.store
|
||||
defer func() {
|
||||
cleanup, cancel := context.WithTimeout(context.Background(), time.Second)
|
||||
defer cancel()
|
||||
_ = goredis.NewScript(`local p=redis.call('XPENDING',KEYS[1],'workers','-','+',1,ARGV[1]);if #p==0 then return redis.call('XGROUP','DELCONSUMER',KEYS[1],'workers',ARGV[1]) end;return 0`).Run(cleanup, s.client, []string{s.keys[0]}, owner).Err()
|
||||
}()
|
||||
cursor := "0-0"
|
||||
for ctx.Err() == nil {
|
||||
messages, next, err := s.client.XAutoClaim(ctx, &goredis.XAutoClaimArgs{Stream: s.keys[0], Group: "workers", Consumer: owner, MinIdle: c.cfg.VisibilityTimeout, Start: cursor, Count: 1}).Result()
|
||||
if err != nil && err != goredis.Nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
cursor = next
|
||||
if cursor == "" {
|
||||
cursor = "0-0"
|
||||
}
|
||||
if len(messages) == 0 {
|
||||
streams, err := s.client.XReadGroup(ctx, &goredis.XReadGroupArgs{Group: "workers", Consumer: owner, Streams: []string{s.keys[0], ">"}, Count: 1, Block: -1}).Result()
|
||||
if err != nil && err != goredis.Nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
if len(streams) > 0 {
|
||||
messages = streams[0].Messages
|
||||
}
|
||||
}
|
||||
if len(messages) == 0 {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil
|
||||
case <-time.After(25 * time.Millisecond):
|
||||
}
|
||||
continue
|
||||
}
|
||||
message := messages[0]
|
||||
raw, ok := message.Values["message"].(string)
|
||||
var m queue.Message
|
||||
if !ok || len(raw) > s.cfg.MaxPayloadBytes+1024 || json.Unmarshal([]byte(raw), &m) != nil {
|
||||
return errors.New("queue/redis: corrupt message; operator review required")
|
||||
}
|
||||
pending, err := s.client.XPendingExt(ctx, &goredis.XPendingExtArgs{Stream: s.keys[0], Group: "workers", Start: message.ID, End: message.ID, Count: 1}).Result()
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
if len(pending) != 1 || pending[0].Consumer != owner {
|
||||
continue
|
||||
}
|
||||
delivery := queue.Delivery{Message: m, Attempt: pending[0].RetryCount}
|
||||
reason := ""
|
||||
if delivery.Attempt > c.cfg.MaxAttempts {
|
||||
reason = "attemptsExhausted"
|
||||
} else {
|
||||
workCtx, cancel := context.WithTimeout(ctx, c.cfg.TaskTimeout)
|
||||
err = invoke(workCtx, handler, delivery)
|
||||
deadlineErr := workCtx.Err()
|
||||
cancel()
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
var rejection *queue.Rejection
|
||||
if errors.As(err, &rejection) {
|
||||
reason = "rejected"
|
||||
} else if err != nil || deadlineErr != nil {
|
||||
if delivery.Attempt < c.cfg.MaxAttempts {
|
||||
continue
|
||||
}
|
||||
reason = "attemptsExhausted"
|
||||
}
|
||||
}
|
||||
ackCtx, ackCancel := context.WithTimeout(ctx, 3*time.Second)
|
||||
err = s.finish(ackCtx, message.ID, owner, delivery, reason)
|
||||
ackCancel()
|
||||
if err != nil && !errors.Is(err, queue.ErrLeaseLost) {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func invoke(ctx context.Context, handler queue.Handler, delivery queue.Delivery) (err error) {
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
err = errors.New("queue: handler panic")
|
||||
}
|
||||
}()
|
||||
if err = ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
return handler(ctx, delivery)
|
||||
}
|
||||
|
||||
var _ queue.Consumer = (*Consumer)(nil)
|
||||
154
queue/redis/consumer_test.go
Normal file
154
queue/redis/consumer_test.go
Normal file
@@ -0,0 +1,154 @@
|
||||
package redis
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.yuntuo.hk/k2safe/go-omni/queue"
|
||||
goredis "github.com/redis/go-redis/v9"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func testStore(t *testing.T, max int) *Store {
|
||||
t.Helper()
|
||||
address := os.Getenv("GO_OMNI_REDIS_TEST_ADDR")
|
||||
if address == "" {
|
||||
t.Skip("requires isolated Redis")
|
||||
}
|
||||
client := goredis.NewClient(&goredis.Options{Addr: address, PoolSize: 4, MaxRetries: -1, ContextTimeoutEnabled: true, ReadTimeout: time.Second, WriteTimeout: time.Second})
|
||||
s, err := New(client, Config{Namespace: fmt.Sprint(t.Name(), time.Now().UnixNano()), MaxRecords: max, DedupTTL: time.Second})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { client.Del(context.Background(), s.keys...); client.Close() })
|
||||
return s
|
||||
}
|
||||
func message(id string) queue.Message {
|
||||
return queue.Message{ID: id, Name: "test", Payload: json.RawMessage(`{"value":1}`)}
|
||||
}
|
||||
func eventually(t *testing.T, f func() bool) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(3 * time.Second)
|
||||
for !f() {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("condition did not recover")
|
||||
}
|
||||
time.Sleep(10 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
func TestQueueRedisIntegration(t *testing.T) {
|
||||
t.Run("publicationAndCapacity", func(t *testing.T) {
|
||||
s := testStore(t, 1)
|
||||
ctx := context.Background()
|
||||
receipt, err := s.Publish(ctx, message("first"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, err := s.Publish(ctx, message("first"))
|
||||
if err != nil || again != receipt {
|
||||
t.Fatal(again, err)
|
||||
}
|
||||
changed := message("first")
|
||||
changed.Payload = json.RawMessage(`2`)
|
||||
if _, err = s.Publish(ctx, changed); !errors.Is(err, queue.ErrConflict) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = s.Publish(ctx, message("second")); !errors.Is(err, queue.ErrFull) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
t.Run("staleAcknowledgement", func(t *testing.T) {
|
||||
s := testStore(t, 10)
|
||||
ctx := context.Background()
|
||||
receipt, _ := s.Publish(ctx, message("first"))
|
||||
s.client.XGroupCreateMkStream(ctx, s.keys[0], "workers", "0")
|
||||
s.client.XReadGroup(ctx, &goredis.XReadGroupArgs{Group: "workers", Consumer: "old", Streams: []string{s.keys[0], ">"}, Count: 1, Block: -1})
|
||||
if err := s.client.XClaim(ctx, &goredis.XClaimArgs{Stream: s.keys[0], Group: "workers", Consumer: "new", MinIdle: 0, Messages: []string{receipt.BackendID}}).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
delivery := queue.Delivery{Message: message("first"), Attempt: 2}
|
||||
if err := s.finish(ctx, receipt.BackendID, "old", delivery, ""); !errors.Is(err, queue.ErrLeaseLost) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.finish(ctx, receipt.BackendID, "new", delivery, ""); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if n := s.client.XLen(ctx, s.keys[0]).Val(); n != 0 {
|
||||
t.Fatal(n)
|
||||
}
|
||||
})
|
||||
t.Run("crashReclaimRetryAndDeadLetters", func(t *testing.T) {
|
||||
s := testStore(t, 10)
|
||||
ctx := context.Background()
|
||||
s.Publish(ctx, message("crashed"))
|
||||
s.Publish(ctx, message("retry"))
|
||||
s.Publish(ctx, message("reject"))
|
||||
s.client.XGroupCreateMkStream(ctx, s.keys[0], "workers", "0")
|
||||
s.client.XReadGroup(ctx, &goredis.XReadGroupArgs{Group: "workers", Consumer: "crashedProcess", Streams: []string{s.keys[0], ">"}, Count: 1, Block: -1})
|
||||
cfg := ConsumerConfig{Workers: 1, MaxAttempts: 3, TaskTimeout: 50 * time.Millisecond, VisibilityTimeout: 100 * time.Millisecond, ShutdownTimeout: time.Second}
|
||||
consumer, _ := NewConsumer(s, cfg)
|
||||
second, _ := NewConsumer(s, cfg)
|
||||
runCtx, cancel := context.WithCancel(ctx)
|
||||
var retries, reclaimed, rejected atomic.Int32
|
||||
handler := func(ctx context.Context, d queue.Delivery) error {
|
||||
switch d.ID {
|
||||
case "crashed":
|
||||
if d.Attempt != 2 {
|
||||
t.Error("reclaim count", d.Attempt)
|
||||
}
|
||||
reclaimed.Add(1)
|
||||
return nil
|
||||
case "retry":
|
||||
retries.Add(1)
|
||||
return errors.New("known retryable failure")
|
||||
default:
|
||||
rejected.Add(1)
|
||||
return queue.Reject("invalidInput")
|
||||
}
|
||||
}
|
||||
done := make(chan error, 2)
|
||||
go func() { done <- consumer.Run(runCtx, handler) }()
|
||||
go func() { done <- second.Run(runCtx, handler) }()
|
||||
defer cancel()
|
||||
eventually(t, func() bool {
|
||||
dead, err := s.DeadLetters(ctx, "", 100)
|
||||
return err == nil && len(dead) == 2 && reclaimed.Load() == 1
|
||||
})
|
||||
cancel()
|
||||
for i := 0; i < 2; i++ {
|
||||
if err := <-done; !errors.Is(err, context.Canceled) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if retries.Load() != 3 || rejected.Load() != 1 {
|
||||
t.Fatal(retries.Load(), rejected.Load())
|
||||
}
|
||||
dead, err := s.DeadLetters(ctx, "", 1)
|
||||
if err != nil || len(dead) != 1 {
|
||||
t.Fatal(dead, err)
|
||||
}
|
||||
rest, err := s.DeadLetters(ctx, dead[0].StreamID, 1)
|
||||
if err != nil || len(rest) != 1 || rest[0].StreamID == dead[0].StreamID {
|
||||
t.Fatal(rest, err)
|
||||
}
|
||||
if err = s.DiscardDead(ctx, dead[0].Message.ID, dead[0].StreamID); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
remaining, _ := s.DeadLetters(ctx, "", 100)
|
||||
if len(remaining) != 1 {
|
||||
t.Fatal(remaining)
|
||||
}
|
||||
// Retained completed IDs expire and release publication capacity.
|
||||
eventually(t, func() bool {
|
||||
_, err := s.Publish(ctx, message("crashed"))
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
return s.client.XLen(ctx, s.keys[0]).Val() == 2
|
||||
})
|
||||
})
|
||||
}
|
||||
187
queue/redis/store.go
Normal file
187
queue/redis/store.go
Normal file
@@ -0,0 +1,187 @@
|
||||
// Package redis implements one managed Redis Streams consumer group per queue.
|
||||
// Use Redis 6.2+, persistence, noeviction and a private namespace. The application
|
||||
// owns the client. Delivery durability depends on the Redis deployment.
|
||||
package redis
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/queue"
|
||||
goredis "github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Namespace string `json:"namespace"`
|
||||
MaxRecords int `json:"maxRecords"` // Includes pending, dead and retained completed identities.
|
||||
MaxPayloadBytes int `json:"maxPayloadBytes"`
|
||||
DedupTTL time.Duration `json:"dedupTtl"` // Starts at ACK; pending/dead records never expire automatically.
|
||||
}
|
||||
type Store struct {
|
||||
client goredis.UniversalClient
|
||||
cfg Config
|
||||
keys []string
|
||||
signature string
|
||||
}
|
||||
|
||||
func New(client goredis.UniversalClient, cfg Config) (*Store, error) {
|
||||
if client == nil || cfg.Namespace == "" || len(cfg.Namespace) > 256 || cfg.MaxRecords < 0 || cfg.MaxPayloadBytes < 0 || cfg.DedupTTL < 0 {
|
||||
return nil, errors.New("queue/redis: invalid configuration")
|
||||
}
|
||||
if cfg.MaxRecords == 0 {
|
||||
cfg.MaxRecords = 1000
|
||||
}
|
||||
if cfg.MaxPayloadBytes == 0 {
|
||||
cfg.MaxPayloadBytes = 64 << 10
|
||||
}
|
||||
if cfg.DedupTTL == 0 {
|
||||
cfg.DedupTTL = 24 * time.Hour
|
||||
}
|
||||
if cfg.MaxRecords > 100000 || cfg.MaxPayloadBytes > 1<<20 || cfg.DedupTTL < time.Second || cfg.DedupTTL > 30*24*time.Hour {
|
||||
return nil, errors.New("queue/redis: configuration exceeds supported bounds")
|
||||
}
|
||||
sum := sha256.Sum256([]byte(cfg.Namespace))
|
||||
base := "goOmni:{" + hex.EncodeToString(sum[:]) + "}:queue"
|
||||
return &Store{client: client, cfg: cfg, keys: []string{base + ":stream", base + ":records", base + ":expiry", base + ":dead", base + ":config", base + ":consumerConfig"}, signature: fmt.Sprintf("%d:%d:%d", cfg.MaxRecords, cfg.MaxPayloadBytes, cfg.DedupTTL.Milliseconds())}, nil
|
||||
}
|
||||
|
||||
var publish = goredis.NewScript(`
|
||||
local config=redis.call('GET',KEYS[5]);if config and config~=ARGV[5] then return redis.error_reply('configuration mismatch') end
|
||||
if not config then redis.call('SET',KEYS[5],ARGV[5]) end
|
||||
local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000)
|
||||
local expired=redis.call('ZRANGEBYSCORE',KEYS[3],'-inf',now,'LIMIT',0,128)
|
||||
for _,id in ipairs(expired) do redis.call('HDEL',KEYS[2],id);redis.call('ZREM',KEYS[3],id) end
|
||||
local expiry=redis.call('ZSCORE',KEYS[3],ARGV[1]);if expiry and tonumber(expiry)<=now then redis.call('HDEL',KEYS[2],ARGV[1]);redis.call('ZREM',KEYS[3],ARGV[1]) end
|
||||
local old=redis.call('HGET',KEYS[2],ARGV[1]);if old then local r=cjson.decode(old);if r.fingerprint~=ARGV[2] then return {-1,''} end;return {0,r.streamId} end
|
||||
if redis.call('HLEN',KEYS[2])>=tonumber(ARGV[4]) then return {-2,''} end
|
||||
local id=redis.call('XADD',KEYS[1],'*','message',ARGV[3])
|
||||
redis.call('HSET',KEYS[2],ARGV[1],cjson.encode({fingerprint=ARGV[2],streamId=id,state='pending'}));return {1,id}
|
||||
`)
|
||||
|
||||
func (s *Store) Publish(ctx context.Context, m queue.Message) (queue.Receipt, error) {
|
||||
if err := queue.Validate(ctx, m, s.cfg.MaxPayloadBytes); err != nil {
|
||||
return queue.Receipt{}, err
|
||||
}
|
||||
raw, _ := json.Marshal(m)
|
||||
if len(raw) > s.cfg.MaxPayloadBytes+1024 {
|
||||
return queue.Receipt{}, errors.New("queue/redis: encoded message exceeds limit")
|
||||
}
|
||||
sum := sha256.Sum256(raw)
|
||||
r, err := publish.Run(ctx, s.client, s.keys, m.ID, hex.EncodeToString(sum[:]), string(raw), s.cfg.MaxRecords, s.signature).Slice()
|
||||
if err != nil {
|
||||
return queue.Receipt{}, err
|
||||
}
|
||||
if len(r) != 2 {
|
||||
return queue.Receipt{}, errors.New("queue/redis: invalid publish result")
|
||||
}
|
||||
switch r[0] {
|
||||
case int64(-1):
|
||||
return queue.Receipt{}, queue.ErrConflict
|
||||
case int64(-2):
|
||||
return queue.Receipt{}, queue.ErrFull
|
||||
}
|
||||
id, ok := r[1].(string)
|
||||
if !ok {
|
||||
return queue.Receipt{}, errors.New("queue/redis: invalid receipt")
|
||||
}
|
||||
return queue.Receipt{ID: m.ID, BackendID: id}, nil
|
||||
}
|
||||
|
||||
var finish = goredis.NewScript(`
|
||||
local p=redis.call('XPENDING',KEYS[1],'workers',ARGV[1],ARGV[1],1)
|
||||
if #p~=1 or p[1][2]~=ARGV[2] then return 0 end
|
||||
local raw=redis.call('HGET',KEYS[2],ARGV[3]);if not raw then return redis.error_reply('message record missing') end
|
||||
local record=cjson.decode(raw);if record.streamId~=ARGV[1] or record.state~='pending' then return redis.error_reply('message identity mismatch') end
|
||||
if ARGV[4]~='' then
|
||||
record.state='dead';redis.call('ZADD',KEYS[4],0,ARGV[1]);record.dead=cjson.decode(ARGV[4])
|
||||
else
|
||||
record.state='completed';local clock=redis.call('TIME');local now=clock[1]*1000+math.floor(clock[2]/1000);redis.call('ZADD',KEYS[3],now+tonumber(ARGV[5]),ARGV[3])
|
||||
end
|
||||
redis.call('HSET',KEYS[2],ARGV[3],cjson.encode(record));redis.call('XACK',KEYS[1],'workers',ARGV[1])
|
||||
if ARGV[4]=='' then redis.call('XDEL',KEYS[1],ARGV[1]) end
|
||||
return 1
|
||||
`)
|
||||
|
||||
func (s *Store) finish(ctx context.Context, id, owner string, m queue.Delivery, reason string) error {
|
||||
var dead []byte
|
||||
if reason != "" {
|
||||
dead, _ = json.Marshal(map[string]any{"attempt": m.Attempt, "reason": reason})
|
||||
}
|
||||
n, err := finish.Run(ctx, s.client, s.keys, id, owner, m.ID, string(dead), s.cfg.DedupTTL.Milliseconds()).Int()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if n != 1 {
|
||||
return queue.ErrLeaseLost
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type DeadLetter struct {
|
||||
StreamID string `json:"streamId"`
|
||||
Message queue.Message `json:"message"`
|
||||
Attempt int64 `json:"attempt"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// DeadLetters returns a bounded page, ordered by opaque Redis stream ID. Pass
|
||||
// the last StreamID as after. Payloads remain retained until explicit removal.
|
||||
func (s *Store) DeadLetters(ctx context.Context, after string, limit int64) ([]DeadLetter, error) {
|
||||
if ctx == nil || limit < 1 || limit > 100 {
|
||||
return nil, errors.New("queue/redis: context and limit 1..100 required")
|
||||
}
|
||||
min := "-"
|
||||
if after != "" {
|
||||
min = "(" + after
|
||||
}
|
||||
ids, err := s.client.ZRangeByLex(ctx, s.keys[3], &goredis.ZRangeBy{Min: min, Max: "+", Count: limit}).Result()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]DeadLetter, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
messages, err := s.client.XRangeN(ctx, s.keys[0], id, id, 1).Result()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(messages) == 0 {
|
||||
continue
|
||||
}
|
||||
var m queue.Message
|
||||
raw, _ := messages[0].Values["message"].(string)
|
||||
if err = json.Unmarshal([]byte(raw), &m); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
record, err := s.client.HGet(ctx, s.keys[1], m.ID).Result()
|
||||
if err == goredis.Nil {
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var saved struct {
|
||||
Dead struct {
|
||||
Attempt int64 `json:"attempt"`
|
||||
Reason string `json:"reason"`
|
||||
} `json:"dead"`
|
||||
}
|
||||
if err = json.Unmarshal([]byte(record), &saved); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, DeadLetter{StreamID: id, Message: m, Attempt: saved.Dead.Attempt, Reason: saved.Dead.Reason})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// DiscardDead deletes only a matching dead delivery after operator review.
|
||||
// It retains the producer identity for DedupTTL, preventing immediate resubmit.
|
||||
func (s *Store) DiscardDead(ctx context.Context, messageID, streamID string) error {
|
||||
return goredis.NewScript(`local raw=redis.call('HGET',KEYS[2],ARGV[1]);if not raw then return 0 end;local r=cjson.decode(raw);if r.state~='dead' or r.streamId~=ARGV[2] then return redis.error_reply('dead identity mismatch') end;r.state='discarded';r.dead=nil;redis.call('HSET',KEYS[2],ARGV[1],cjson.encode(r));redis.call('ZREM',KEYS[4],ARGV[2]);redis.call('XDEL',KEYS[1],ARGV[2]);local t=redis.call('TIME');redis.call('ZADD',KEYS[3],t[1]*1000+math.floor(t[2]/1000)+tonumber(ARGV[3]),ARGV[1]);return 1`).Run(ctx, s.client, s.keys, messageID, streamID, s.cfg.DedupTTL.Milliseconds()).Err()
|
||||
}
|
||||
|
||||
var _ queue.Publisher = (*Store)(nil)
|
||||
129
resilience/breaker.go
Normal file
129
resilience/breaker.go
Normal file
@@ -0,0 +1,129 @@
|
||||
// Package resilience provides per-dependency circuit breaking and concurrency isolation.
|
||||
package resilience
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
var ErrOpen = errors.New("resilience: circuit open")
|
||||
var ErrBusy = errors.New("resilience: concurrency limit reached")
|
||||
|
||||
type Config struct {
|
||||
FailureThreshold int `json:"failureThreshold"`
|
||||
MaxConcurrent int `json:"maxConcurrent"`
|
||||
OpenTimeout time.Duration `json:"openTimeout"`
|
||||
IsFailure func(error) bool `json:"-"`
|
||||
}
|
||||
type Breaker struct {
|
||||
mu sync.Mutex
|
||||
config Config
|
||||
active, failures int
|
||||
state string
|
||||
until time.Time
|
||||
generation uint64
|
||||
now func() time.Time
|
||||
}
|
||||
type Snapshot struct {
|
||||
State string `json:"state"`
|
||||
Active int `json:"active"`
|
||||
Failures int `json:"failures"`
|
||||
}
|
||||
|
||||
func New(cfg Config) (*Breaker, error) {
|
||||
if cfg.FailureThreshold < 0 || cfg.MaxConcurrent < 0 || cfg.OpenTimeout < 0 {
|
||||
return nil, errors.New("resilience: invalid configuration")
|
||||
}
|
||||
if cfg.FailureThreshold == 0 {
|
||||
cfg.FailureThreshold = 5
|
||||
}
|
||||
if cfg.MaxConcurrent == 0 {
|
||||
cfg.MaxConcurrent = 32
|
||||
}
|
||||
if cfg.OpenTimeout == 0 {
|
||||
cfg.OpenTimeout = 5 * time.Second
|
||||
}
|
||||
if cfg.IsFailure == nil {
|
||||
cfg.IsFailure = func(err error) bool { return err != nil && !errors.Is(err, context.Canceled) }
|
||||
}
|
||||
return &Breaker{config: cfg, state: "closed", now: time.Now}, nil
|
||||
}
|
||||
|
||||
// Acquire returns a completion callback; it must be called exactly once.
|
||||
// A half-open circuit admits one probe. Old-generation results cannot close it.
|
||||
func (b *Breaker) Acquire(ctx context.Context) (func(bool), error) {
|
||||
if ctx == nil {
|
||||
return nil, errors.New("resilience: nil context")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
if b.state == "open" {
|
||||
if b.now().Before(b.until) {
|
||||
return nil, ErrOpen
|
||||
}
|
||||
b.state = "halfOpen"
|
||||
b.generation++
|
||||
}
|
||||
if b.state == "halfOpen" && b.active > 0 {
|
||||
return nil, ErrOpen
|
||||
}
|
||||
if b.active >= b.config.MaxConcurrent {
|
||||
return nil, ErrBusy
|
||||
}
|
||||
b.active++
|
||||
generation := b.generation
|
||||
var once sync.Once
|
||||
return func(success bool) {
|
||||
once.Do(func() {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
b.active--
|
||||
if generation != b.generation {
|
||||
return
|
||||
}
|
||||
if success {
|
||||
b.failures = 0
|
||||
if b.state == "halfOpen" {
|
||||
b.state = "closed"
|
||||
b.generation++
|
||||
}
|
||||
return
|
||||
}
|
||||
b.failures++
|
||||
if b.state == "halfOpen" || b.failures >= b.config.FailureThreshold {
|
||||
b.state = "open"
|
||||
b.until = b.now().Add(b.config.OpenTimeout)
|
||||
b.generation++
|
||||
}
|
||||
})
|
||||
}, nil
|
||||
}
|
||||
func (b *Breaker) Do(ctx context.Context, work func(context.Context) error) (err error) {
|
||||
if work == nil {
|
||||
return errors.New("resilience: nil work")
|
||||
}
|
||||
done, err := b.Acquire(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
complete := false
|
||||
defer func() {
|
||||
if !complete {
|
||||
done(false)
|
||||
}
|
||||
}()
|
||||
err = work(ctx)
|
||||
done(!b.config.IsFailure(err))
|
||||
complete = true
|
||||
return err
|
||||
}
|
||||
func (b *Breaker) Snapshot() Snapshot {
|
||||
b.mu.Lock()
|
||||
defer b.mu.Unlock()
|
||||
return Snapshot{State: b.state, Active: b.active, Failures: b.failures}
|
||||
}
|
||||
75
resilience/breaker_test.go
Normal file
75
resilience/breaker_test.go
Normal file
@@ -0,0 +1,75 @@
|
||||
package resilience
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type roundTripFunc func(*http.Request) (*http.Response, error)
|
||||
|
||||
func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
|
||||
func TestBreakerProbeAndLateResult(t *testing.T) {
|
||||
b, _ := New(Config{FailureThreshold: 1, MaxConcurrent: 2, OpenTimeout: time.Second})
|
||||
now := time.Now()
|
||||
b.now = func() time.Time { return now }
|
||||
ctx := context.Background()
|
||||
first, _ := b.Acquire(ctx)
|
||||
late, _ := b.Acquire(ctx)
|
||||
first(false)
|
||||
late(true)
|
||||
if b.Snapshot().State != "open" {
|
||||
t.Fatal(b.Snapshot())
|
||||
}
|
||||
if _, err := b.Acquire(ctx); !errors.Is(err, ErrOpen) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
now = now.Add(time.Second)
|
||||
probe, err := b.Acquire(ctx)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = b.Acquire(ctx); !errors.Is(err, ErrOpen) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
probe(true)
|
||||
probe(false)
|
||||
if s := b.Snapshot(); s.State != "closed" || s.Active != 0 {
|
||||
t.Fatal(s)
|
||||
}
|
||||
}
|
||||
func TestTransportHoldsSlotAndReleasesPanic(t *testing.T) {
|
||||
b, _ := New(Config{MaxConcurrent: 1})
|
||||
tr := Transport{Breaker: b, Base: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
||||
return &http.Response{StatusCode: 200, Body: io.NopCloser(strings.NewReader("hello"))}, nil
|
||||
})}
|
||||
req, _ := http.NewRequest("GET", "http://example.test", nil)
|
||||
r, err := tr.RoundTrip(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = tr.RoundTrip(req); !errors.Is(err, ErrBusy) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
io.ReadAll(r.Body)
|
||||
r.Body.Close()
|
||||
if b.Snapshot().Active != 0 {
|
||||
t.Fatal(b.Snapshot())
|
||||
}
|
||||
tr.Base = roundTripFunc(func(*http.Request) (*http.Response, error) { panic("fixture") })
|
||||
func() {
|
||||
defer func() {
|
||||
if recover() == nil {
|
||||
t.Error("panic lost")
|
||||
}
|
||||
}()
|
||||
tr.RoundTrip(req)
|
||||
}()
|
||||
if b.Snapshot().Active != 0 {
|
||||
t.Fatal(b.Snapshot())
|
||||
}
|
||||
}
|
||||
67
resilience/http.go
Normal file
67
resilience/http.go
Normal file
@@ -0,0 +1,67 @@
|
||||
package resilience
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Transport isolates one logical upstream. Create separate Breakers for
|
||||
// different dependencies. Slots remain occupied until body Close/EOF.
|
||||
type Transport struct {
|
||||
Base http.RoundTripper
|
||||
Breaker *Breaker
|
||||
}
|
||||
|
||||
func (t Transport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
if t.Breaker == nil {
|
||||
return nil, errors.New("resilience: breaker required")
|
||||
}
|
||||
done, err := t.Breaker.Acquire(r.Context())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() {
|
||||
if v := recover(); v != nil {
|
||||
done(false)
|
||||
panic(v)
|
||||
}
|
||||
}()
|
||||
base := t.Base
|
||||
if base == nil {
|
||||
base = http.DefaultTransport
|
||||
}
|
||||
response, err := base.RoundTrip(r)
|
||||
if err != nil {
|
||||
done(!t.Breaker.config.IsFailure(err))
|
||||
if response != nil && response.Body != nil {
|
||||
response.Body.Close()
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if response == nil || response.Body == nil {
|
||||
done(false)
|
||||
return nil, errors.New("resilience: invalid transport response")
|
||||
}
|
||||
response.Body = &breakerBody{ReadCloser: response.Body, done: done, success: response.StatusCode < 500}
|
||||
return response, nil
|
||||
}
|
||||
|
||||
type breakerBody struct {
|
||||
io.ReadCloser
|
||||
done func(bool)
|
||||
success bool
|
||||
}
|
||||
|
||||
func (b *breakerBody) Read(p []byte) (int, error) {
|
||||
n, err := b.ReadCloser.Read(p)
|
||||
if err != nil {
|
||||
b.done(b.success && err == io.EOF)
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
func (b *breakerBody) Close() error {
|
||||
err := b.ReadCloser.Close()
|
||||
b.done(b.success && err == nil)
|
||||
return err
|
||||
}
|
||||
@@ -22,6 +22,9 @@ def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--image", default="golang:1.25")
|
||||
parser.add_argument("--redis-image", default="redis:7.4-alpine")
|
||||
parser.add_argument("--advanced", action="store_true", help="also verify shared state, durable queue, S3 and MySQL migrations")
|
||||
parser.add_argument("--s3-image", default="quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z")
|
||||
parser.add_argument("--mysql-image", default="m.daocloud.io/docker.io/library/mysql:8.4")
|
||||
parser.add_argument("--goproxy", default="")
|
||||
parser.add_argument("--wall-timeout", type=int, default=900)
|
||||
parser.add_argument("--output", type=Path, default=Path("performance/modulesLinux.json"))
|
||||
@@ -40,9 +43,13 @@ def main():
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
log_path = output_path.with_suffix(".txt")
|
||||
prefix = "go-omni-modules-" + uuid.uuid4().hex[:12]
|
||||
names = [prefix + "-" + stage for stage in ("build", "redis")]
|
||||
stages = ("build", "redis", "s3", "mysql") if args.advanced else ("build", "redis")
|
||||
names = [prefix + "-" + stage for stage in stages]
|
||||
paths = [root / "go.mod", root / "go.sum", root / "httpx/context.go", Path(__file__).resolve()]
|
||||
for directory in ("cache", "httpc", "task", "middleware", "cmd/go-omni", "examples/modules"):
|
||||
directories = ["cache", "httpc", "task", "middleware", "cmd/go-omni", "examples/modules"]
|
||||
if args.advanced:
|
||||
directories += ["auth", "config", "logx", "openapi", "telemetry", "resilience", "storage", "queue", "audit", "pagination", "database/migrate", "internal/storagefixture", "examples/platform"]
|
||||
for directory in directories:
|
||||
paths.extend(p for p in (root / directory).rglob("*") if p.is_file() and p.suffix in (".go", ".tmpl"))
|
||||
report = {
|
||||
"status": "running", "startedAt": datetime.now(timezone.utc).isoformat(),
|
||||
@@ -86,7 +93,10 @@ def main():
|
||||
report["cleanedContainers"].append(name)
|
||||
|
||||
try:
|
||||
for image in (args.image, args.redis_image):
|
||||
images = [args.image, args.redis_image]
|
||||
if args.advanced:
|
||||
images += [args.s3_image, args.mysql_image]
|
||||
for image in images:
|
||||
result = subprocess.run([docker, "image", "inspect", image], capture_output=True, timeout=15)
|
||||
if result.returncode:
|
||||
print("Pulling " + image, flush=True)
|
||||
@@ -101,9 +111,11 @@ def main():
|
||||
"-e", "GOWORK=off", "-e", "GOMAXPROCS=2", "-e", "GO_OMNI_SCAFFOLD_INTEGRATION=1"]
|
||||
if args.goproxy:
|
||||
arguments += ["-e", "GOPROXY=" + args.goproxy]
|
||||
arguments += [args.image, "sh", "-c",
|
||||
"go version && go test -race -count=1 -timeout=240s ./... && go vet ./... && go build ./... && "
|
||||
"CGO_ENABLED=0 go test -c -o /output/redis.test ./cache/redis"]
|
||||
checks = "go version && go test -race -count=1 -timeout=240s ./... && go vet ./... && go build ./... && CGO_ENABLED=0 go test -c -o /output/redis.test ./cache/redis"
|
||||
if args.advanced:
|
||||
for binary, package in (("shared", "middleware/redis"), ("queue", "queue/redis"), ("s3", "storage/s3"), ("mysql", "database/migrate")):
|
||||
checks += f" && CGO_ENABLED=0 go test -c -o /output/{binary}.test ./{package}"
|
||||
arguments += [args.image, "sh", "-c", checks]
|
||||
container(names[0], arguments)
|
||||
print("Real Redis TTL/read/write/delete/failure checks; external network disabled", flush=True)
|
||||
redis_check = '''set -eu
|
||||
@@ -118,15 +130,54 @@ until redis-cli -h 127.0.0.1 ping >/dev/null 2>&1; do
|
||||
done
|
||||
/test/redis.test -test.v -test.run '^TestRedisIntegration$' -test.count=1 -test.timeout=30s
|
||||
'''
|
||||
if args.advanced:
|
||||
redis_check += "/test/shared.test -test.v -test.count=1 -test.timeout=45s\n/test/queue.test -test.v -test.count=1 -test.timeout=45s\n"
|
||||
container(names[1], ["--network", "none", "--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges",
|
||||
"--user", "999:999", "--tmpfs", "/tmp:rw,nosuid,noexec,size=64m,mode=1777",
|
||||
"-v", f"{temporary}:/test:ro", "--entrypoint", "sh",
|
||||
"-e", "GOMAXPROCS=2", "-e", "GO_OMNI_REDIS_TEST_ADDR=127.0.0.1:6379",
|
||||
args.redis_image, "-c", redis_check])
|
||||
if args.advanced:
|
||||
isolated = ["--network", "none", "--read-only", "--cap-drop", "ALL", "--security-opt", "no-new-privileges",
|
||||
"-v", f"{temporary}:/test:ro", "--entrypoint", "sh", "-e", "GOMAXPROCS=2"]
|
||||
print("Real S3 object operations and signature verification in isolated MinIO", flush=True)
|
||||
s3_check = '''set -eu
|
||||
minio --version
|
||||
minio server /data --address 127.0.0.1:9000 --console-address 127.0.0.1:9001 >/tmp/minio.log 2>&1 &
|
||||
server=$!
|
||||
trap 'kill "$server" 2>/dev/null || true; wait "$server" 2>/dev/null || true' EXIT
|
||||
/test/s3.test -test.v -test.run '^TestS3Integration$' -test.count=1 -test.timeout=45s || { cat /tmp/minio.log; exit 1; }
|
||||
'''
|
||||
container(names[2], isolated + ["--user", "1000:1000", "--tmpfs", "/tmp:rw,nosuid,noexec,size=64m,mode=1777",
|
||||
"--tmpfs", "/data:rw,nosuid,noexec,size=128m,mode=1777", "-e", "MINIO_ROOT_USER=goOmniTest", "-e", "MINIO_ROOT_PASSWORD=goOmniTestSecret",
|
||||
"-e", "GO_OMNI_S3_TEST_ADDR=127.0.0.1:9000", args.s3_image, "-c", s3_check])
|
||||
print("Real MySQL migrations, repeat apply, dirty-state blocking and SQL spans", flush=True)
|
||||
mysql_check = '''set -eu
|
||||
mysqld --version
|
||||
mkdir /tmp/mysql
|
||||
mysqld --no-defaults --initialize-insecure --datadir=/tmp/mysql --innodb-redo-log-capacity=67108864 >/tmp/init.log 2>&1 || { cat /tmp/init.log; exit 1; }
|
||||
mysqld --no-defaults --datadir=/tmp/mysql --socket=/tmp/mysql.sock --pid-file=/tmp/mysql.pid --bind-address=127.0.0.1 --port=3306 --mysqlx=OFF --skip-log-bin --skip-name-resolve --innodb-buffer-pool-size=134217728 --innodb-redo-log-capacity=67108864 --max-connections=8 --log-error=/tmp/mysql.log &
|
||||
server=$!
|
||||
trap 'kill "$server" 2>/dev/null || true; wait "$server" 2>/dev/null || true' EXIT
|
||||
attempt=0
|
||||
until mysql --no-defaults --socket=/tmp/mysql.sock -uroot -e 'SELECT 1' >/dev/null 2>&1; do
|
||||
attempt=$((attempt + 1)); if [ "$attempt" -gt 200 ]; then cat /tmp/mysql.log; exit 1; fi; sleep 0.1
|
||||
done
|
||||
mysql --no-defaults --socket=/tmp/mysql.sock -uroot -e 'CREATE DATABASE omni_test'
|
||||
/test/mysql.test -test.v -test.run '^TestMySQLIntegration$' -test.count=1 -test.timeout=60s
|
||||
'''
|
||||
container(names[3], isolated + ["--user", "999:999", "--tmpfs", "/tmp:rw,nosuid,noexec,size=512m,mode=1777",
|
||||
"--tmpfs", "/var/lib/mysql:rw,nosuid,noexec,size=1m,mode=1777",
|
||||
"-e", "GO_OMNI_MIGRATE_TEST_DSN=root@unix(/tmp/mysql.sock)/omni_test?parseTime=true&timeout=2s&readTimeout=5s&writeTimeout=5s",
|
||||
args.mysql_image, "-c", mysql_check])
|
||||
log.flush()
|
||||
content = log_path.read_text()
|
||||
if "--- PASS: TestRedisIntegration" not in content:
|
||||
raise RuntimeError("missing real Redis pass evidence")
|
||||
if args.advanced:
|
||||
for test in ("TestSharedRedisIntegration", "TestExpiredTargetBeyondCleanupBatch", "TestQueueRedisIntegration", "TestS3Integration", "TestMySQLIntegration"):
|
||||
if "--- PASS: " + test not in content:
|
||||
raise RuntimeError("missing integration pass evidence: " + test)
|
||||
report["status"] = "passed"
|
||||
except (Exception, KeyboardInterrupt) as error:
|
||||
failure = str(error) or type(error).__name__
|
||||
|
||||
171
storage/local/local.go
Normal file
171
storage/local/local.go
Normal file
@@ -0,0 +1,171 @@
|
||||
// Package local stores objects atomically in a private, flat managed directory.
|
||||
// This directory must not be writable by untrusted local users/processes.
|
||||
package local
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/binary"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
root string
|
||||
maxBytes int64
|
||||
}
|
||||
|
||||
func New(root string, maxBytes int64) (*Store, error) {
|
||||
if root == "" || maxBytes < 0 || maxBytes > 1<<40 {
|
||||
return nil, errors.New("storage/local: invalid configuration")
|
||||
}
|
||||
if maxBytes == 0 {
|
||||
maxBytes = 32 << 20
|
||||
}
|
||||
if err := os.MkdirAll(root, 0700); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
root, err := filepath.EvalSymlinks(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
root, err = filepath.Abs(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
info, err := os.Stat(root)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !info.IsDir() || info.Mode().Perm()&0022 != 0 {
|
||||
return nil, errors.New("storage/local: root must be a private writable directory")
|
||||
}
|
||||
return &Store{root: root, maxBytes: maxBytes}, nil
|
||||
}
|
||||
func (s *Store) path(key string) string {
|
||||
sum := sha256.Sum256([]byte(key))
|
||||
return filepath.Join(s.root, hex.EncodeToString(sum[:])+".object")
|
||||
}
|
||||
func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int64, contentType string) (storage.Info, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if reader == nil || size < 0 || len(contentType) > 256 || strings.ContainsAny(contentType, "\r\n") {
|
||||
return storage.Info{}, errors.New("storage/local: invalid upload")
|
||||
}
|
||||
if size > s.maxBytes {
|
||||
return storage.Info{}, storage.ErrTooLarge
|
||||
}
|
||||
info := storage.Info{Key: key, Size: size, ContentType: contentType}
|
||||
metadata, _ := json.Marshal(info)
|
||||
f, err := os.CreateTemp(s.root, ".upload-")
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
defer os.Remove(f.Name())
|
||||
defer f.Close()
|
||||
if err = binary.Write(f, binary.BigEndian, uint32(len(metadata))); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if _, err = f.Write(metadata); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
n, err := io.Copy(f, io.LimitReader(storage.ContextReader{Context: ctx, Reader: reader}, size+1))
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if n != size {
|
||||
return storage.Info{}, errors.New("storage/local: declared size mismatch")
|
||||
}
|
||||
if err = ctx.Err(); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if err = f.Sync(); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if err = f.Close(); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if err = os.Rename(f.Name(), s.path(key)); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
info.ModifiedAt = stat.ModTime()
|
||||
return info, nil
|
||||
}
|
||||
func (s *Store) Get(ctx context.Context, key string) (storage.Object, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
path := s.path(key)
|
||||
stat, err := os.Lstat(path)
|
||||
if os.IsNotExist(err) {
|
||||
return storage.Object{}, storage.ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
if !stat.Mode().IsRegular() {
|
||||
return storage.Object{}, errors.New("storage/local: not a regular managed object")
|
||||
}
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
// A concurrent Put can atomically replace the path between Lstat/Open.
|
||||
// Validate size against the descriptor whose bytes we actually read.
|
||||
stat, err = f.Stat()
|
||||
if err != nil {
|
||||
f.Close()
|
||||
return storage.Object{}, err
|
||||
}
|
||||
ok := false
|
||||
defer func() {
|
||||
if !ok {
|
||||
f.Close()
|
||||
}
|
||||
}()
|
||||
var length uint32
|
||||
if err = binary.Read(f, binary.BigEndian, &length); err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
if length > 4096 {
|
||||
return storage.Object{}, errors.New("storage/local: invalid object header")
|
||||
}
|
||||
data := make([]byte, length)
|
||||
if _, err = io.ReadFull(f, data); err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
var info storage.Info
|
||||
if json.Unmarshal(data, &info) != nil || info.Key != key || info.Size < 0 || info.Size > s.maxBytes || int64(length)+4+info.Size != stat.Size() {
|
||||
return storage.Object{}, errors.New("storage/local: corrupt object")
|
||||
}
|
||||
info.ModifiedAt = stat.ModTime()
|
||||
ok = true
|
||||
return storage.Object{Info: info, Body: &body{Reader: storage.ContextReader{Context: ctx, Reader: io.LimitReader(f, info.Size)}, Closer: f}}, nil
|
||||
}
|
||||
|
||||
type body struct {
|
||||
io.Reader
|
||||
io.Closer
|
||||
}
|
||||
|
||||
func (s *Store) Delete(ctx context.Context, key string) error {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return err
|
||||
}
|
||||
err := os.Remove(s.path(key))
|
||||
if os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
42
storage/local/local_test.go
Normal file
42
storage/local/local_test.go
Normal file
@@ -0,0 +1,42 @@
|
||||
package local
|
||||
|
||||
import (
|
||||
"context"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
|
||||
"io"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestContractAndAtomicFailure(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
s, err := New(root, 64)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
storagetest.Run(t, s)
|
||||
ctx := context.Background()
|
||||
if _, err = s.Put(ctx, "saved", strings.NewReader("old"), 3, "text/plain"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = s.Put(ctx, "saved", strings.NewReader("short"), 20, "text/plain"); err == nil {
|
||||
t.Fatal("short upload committed")
|
||||
}
|
||||
again, _ := New(root, 64)
|
||||
object, err := again.Get(ctx, "saved")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := io.ReadAll(object.Body)
|
||||
object.Body.Close()
|
||||
if string(raw) != "old" {
|
||||
t.Fatal(string(raw))
|
||||
}
|
||||
if err = os.Symlink(s.path("saved"), s.path("linked")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err = s.Get(ctx, "linked"); err == nil {
|
||||
t.Fatal("symlink followed")
|
||||
}
|
||||
}
|
||||
101
storage/oss/oss.go
Normal file
101
storage/oss/oss.go
Normal file
@@ -0,0 +1,101 @@
|
||||
// Package oss adapts the caller-owned official Alibaba Cloud OSS v2 client.
|
||||
package oss
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
client *sdk.Client
|
||||
bucket string
|
||||
maxBytes int64
|
||||
}
|
||||
|
||||
// New borrows a configured SDK client. This adapter uses simple uploads; large
|
||||
// multipart uploads are a separate provider capability, not buffered in memory.
|
||||
func New(client *sdk.Client, bucket string, maxBytes int64) (*Store, error) {
|
||||
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<30 {
|
||||
return nil, errors.New("storage/oss: invalid client, bucket or size limit")
|
||||
}
|
||||
if maxBytes == 0 {
|
||||
maxBytes = 32 << 20
|
||||
}
|
||||
return &Store{client: client, bucket: bucket, maxBytes: maxBytes}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int64, contentType string) (storage.Info, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if reader == nil || size < 0 || len(contentType) > 256 || strings.ContainsAny(contentType, "\r\n") {
|
||||
return storage.Info{}, errors.New("storage/oss: invalid upload")
|
||||
}
|
||||
if size > s.maxBytes {
|
||||
return storage.Info{}, storage.ErrTooLarge
|
||||
}
|
||||
_, err := s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: storage.ContextReader{Context: ctx, Reader: reader}})
|
||||
if err != nil {
|
||||
return storage.Info{}, mapError(err)
|
||||
}
|
||||
return storage.Info{Key: key, Size: size, ContentType: contentType, ModifiedAt: time.Now().UTC()}, nil
|
||||
}
|
||||
func (s *Store) Get(ctx context.Context, key string) (storage.Object, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
r, err := s.client.GetObject(ctx, &sdk.GetObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key)})
|
||||
if err != nil {
|
||||
return storage.Object{}, mapError(err)
|
||||
}
|
||||
if r.ContentLength < 0 || r.ContentLength > s.maxBytes {
|
||||
r.Body.Close()
|
||||
return storage.Object{}, storage.ErrTooLarge
|
||||
}
|
||||
info := storage.Info{Key: key, Size: r.ContentLength, ContentType: sdk.ToString(r.ContentType)}
|
||||
if r.LastModified != nil {
|
||||
info.ModifiedAt = *r.LastModified
|
||||
}
|
||||
return storage.Object{Info: info, Body: r.Body}, nil
|
||||
}
|
||||
func (s *Store) Delete(ctx context.Context, key string) error {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := s.client.DeleteObject(ctx, &sdk.DeleteObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key)})
|
||||
err = mapError(err)
|
||||
if errors.Is(err, storage.ErrNotFound) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (*url.URL, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ttl < time.Second || ttl > time.Hour {
|
||||
return nil, errors.New("storage/oss: TTL must be 1s..1h")
|
||||
}
|
||||
r, err := s.client.Presign(ctx, &sdk.GetObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ResponseContentDisposition: sdk.Ptr("attachment")}, sdk.PresignExpires(ttl))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return url.Parse(r.URL)
|
||||
}
|
||||
func mapError(err error) error {
|
||||
var serviceError *sdk.ServiceError
|
||||
if errors.As(err, &serviceError) && serviceError.Code == "NoSuchKey" {
|
||||
return storage.ErrNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
var _ storage.Store = (*Store)(nil)
|
||||
var _ storage.Presigner = (*Store)(nil)
|
||||
31
storage/oss/oss_test.go
Normal file
31
storage/oss/oss_test.go
Normal file
@@ -0,0 +1,31 @@
|
||||
package oss
|
||||
|
||||
import (
|
||||
"context"
|
||||
"git.yuntuo.hk/k2safe/go-omni/internal/storagefixture"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
|
||||
sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
|
||||
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestSDKProtocolContract(t *testing.T) {
|
||||
server := httptest.NewTLSServer(storagefixture.Handler())
|
||||
defer server.Close()
|
||||
cfg := sdk.LoadDefaultConfig().WithRegion("cn-hangzhou").WithEndpoint(server.URL).WithUsePathStyle(true).WithCredentialsProvider(credentials.NewStaticCredentialsProvider("fixtureAccess", "fixtureSecret")).WithHttpClient(server.Client())
|
||||
store, err := New(sdk.NewClient(cfg), "testbucket", 64)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
storagetest.Run(t, store)
|
||||
u, err := store.PresignGet(context.Background(), "a/file.txt", time.Minute)
|
||||
if err != nil || u.Query().Get("x-oss-signature") == "" || !strings.Contains(u.Path, "a/file.txt") {
|
||||
t.Fatal("invalid signed URL", err)
|
||||
}
|
||||
if _, err = store.PresignGet(context.Background(), "a", 2*time.Hour); err == nil {
|
||||
t.Fatal("TTL accepted")
|
||||
}
|
||||
}
|
||||
42
storage/s3/config.go
Normal file
42
storage/s3/config.go
Normal file
@@ -0,0 +1,42 @@
|
||||
package s3
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"regexp"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
// Open creates a store using an injected credential provider and transport.
|
||||
// The application owns transport shutdown; no global HTTP defaults are changed.
|
||||
type Config struct {
|
||||
Endpoint string `json:"endpoint"` // Host[:port], without scheme or path.
|
||||
Region string `json:"region"`
|
||||
Bucket string `json:"bucket"`
|
||||
Secure bool `json:"secure"`
|
||||
MaxBytes int64 `json:"maxBytes"`
|
||||
Credentials *credentials.Credentials `json:"-"`
|
||||
Transport http.RoundTripper `json:"-"`
|
||||
}
|
||||
|
||||
func Open(cfg Config) (*Store, error) {
|
||||
if cfg.Credentials == nil {
|
||||
return nil, errors.New("storage/s3: credentials provider required")
|
||||
}
|
||||
client, err := minio.New(cfg.Endpoint, &minio.Options{Creds: cfg.Credentials, Secure: cfg.Secure, Region: cfg.Region, Transport: cfg.Transport, BucketLookup: minio.BucketLookupPath})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(client, cfg.Bucket, cfg.MaxBytes)
|
||||
}
|
||||
|
||||
// R2Config selects Cloudflare's standard S3 API endpoint. Jurisdiction-specific
|
||||
// endpoints can be set explicitly with Config. R2 implements an S3 subset.
|
||||
func R2Config(accountID, bucket string, creds *credentials.Credentials) (Config, error) {
|
||||
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(accountID) {
|
||||
return Config{}, errors.New("storage/s3: invalid R2 account ID")
|
||||
}
|
||||
return Config{Endpoint: accountID + ".r2.cloudflarestorage.com", Region: "auto", Bucket: bucket, Secure: true, Credentials: creds}, nil
|
||||
}
|
||||
93
storage/s3/s3.go
Normal file
93
storage/s3/s3.go
Normal file
@@ -0,0 +1,93 @@
|
||||
// Package s3 adapts a caller-owned MinIO client to S3-compatible object storage.
|
||||
package s3
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"io"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
client *minio.Client
|
||||
bucket string
|
||||
maxBytes int64
|
||||
}
|
||||
|
||||
func New(client *minio.Client, bucket string, maxBytes int64) (*Store, error) {
|
||||
if client == nil || bucket == "" || maxBytes < 0 {
|
||||
return nil, errors.New("storage/s3: client, bucket and nonnegative limit required")
|
||||
}
|
||||
if maxBytes == 0 {
|
||||
maxBytes = 32 << 20
|
||||
}
|
||||
return &Store{client: client, bucket: bucket, maxBytes: maxBytes}, nil
|
||||
}
|
||||
func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int64, contentType string) (storage.Info, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
if reader == nil || size < 0 || len(contentType) > 256 || strings.ContainsAny(contentType, "\r\n") {
|
||||
return storage.Info{}, errors.New("storage/s3: invalid upload")
|
||||
}
|
||||
if size > s.maxBytes {
|
||||
return storage.Info{}, storage.ErrTooLarge
|
||||
}
|
||||
result, err := s.client.PutObject(ctx, s.bucket, key, storage.ContextReader{Context: ctx, Reader: reader}, size, minio.PutObjectOptions{ContentType: contentType})
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
return storage.Info{Key: key, Size: result.Size, ContentType: contentType, ModifiedAt: time.Now().UTC()}, nil
|
||||
}
|
||||
func (s *Store) Get(ctx context.Context, key string) (storage.Object, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.Object{}, err
|
||||
}
|
||||
object, err := s.client.GetObject(ctx, s.bucket, key, minio.GetObjectOptions{})
|
||||
if err != nil {
|
||||
return storage.Object{}, mapError(err)
|
||||
}
|
||||
stat, err := object.Stat()
|
||||
if err != nil {
|
||||
object.Close()
|
||||
return storage.Object{}, mapError(err)
|
||||
}
|
||||
if stat.Size > s.maxBytes {
|
||||
object.Close()
|
||||
return storage.Object{}, storage.ErrTooLarge
|
||||
}
|
||||
return storage.Object{Info: storage.Info{Key: key, Size: stat.Size, ContentType: stat.ContentType, ModifiedAt: stat.LastModified}, Body: object}, nil
|
||||
}
|
||||
func (s *Store) Delete(ctx context.Context, key string) error {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return err
|
||||
}
|
||||
err := s.client.RemoveObject(ctx, s.bucket, key, minio.RemoveObjectOptions{})
|
||||
if errors.Is(mapError(err), storage.ErrNotFound) {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (s *Store) PresignGet(ctx context.Context, key string, ttl time.Duration) (*url.URL, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ttl < time.Second || ttl > time.Hour {
|
||||
return nil, errors.New("storage/s3: TTL must be 1s..1h")
|
||||
}
|
||||
return s.client.PresignedGetObject(ctx, s.bucket, key, ttl, url.Values{"response-content-disposition": {"attachment"}})
|
||||
}
|
||||
func mapError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
switch minio.ToErrorResponse(err).Code {
|
||||
case "NoSuchKey", "NoSuchObject":
|
||||
return storage.ErrNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
104
storage/s3/s3_test.go
Normal file
104
storage/s3/s3_test.go
Normal file
@@ -0,0 +1,104 @@
|
||||
package s3
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"git.yuntuo.hk/k2safe/go-omni/internal/storagefixture"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestS3ProtocolAndR2Config(t *testing.T) {
|
||||
server := httptest.NewTLSServer(storagefixture.Handler())
|
||||
defer server.Close()
|
||||
creds := credentials.NewStaticV4("fixtureAccess", "fixtureSecret", "")
|
||||
s, err := Open(Config{Endpoint: strings.TrimPrefix(server.URL, "https://"), Region: "us-east-1", Bucket: "testbucket", Secure: true, MaxBytes: 64, Credentials: creds, Transport: server.Client().Transport})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
storagetest.Run(t, s)
|
||||
cfg, err := R2Config(strings.Repeat("a", 32), "bucket", creds)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r2, err := Open(cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
u, err := r2.PresignGet(context.Background(), "path/file", time.Minute)
|
||||
if err != nil || u.Host != strings.Repeat("a", 32)+".r2.cloudflarestorage.com" || u.Query().Get("X-Amz-Signature") == "" || !strings.Contains(u.Query().Get("X-Amz-Credential"), "/auto/s3/") {
|
||||
t.Fatal("R2 presign settings", err)
|
||||
}
|
||||
}
|
||||
func TestS3Integration(t *testing.T) {
|
||||
address := os.Getenv("GO_OMNI_S3_TEST_ADDR")
|
||||
if address == "" {
|
||||
t.Skip("requires isolated MinIO")
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
|
||||
defer cancel()
|
||||
ready := false
|
||||
for n := 0; n < 100; n++ {
|
||||
req, _ := http.NewRequestWithContext(ctx, "GET", "http://"+address+"/minio/health/ready", nil)
|
||||
r, err := http.DefaultClient.Do(req)
|
||||
if err == nil {
|
||||
r.Body.Close()
|
||||
if r.StatusCode == 200 {
|
||||
ready = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
if !ready {
|
||||
t.Fatal("isolated S3 server did not start")
|
||||
}
|
||||
client, err := minio.New(address, &minio.Options{Creds: credentials.NewStaticV4("goOmniTest", "goOmniTestSecret", ""), Region: "us-east-1"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
bucket := fmt.Sprintf("test-%d", time.Now().UnixNano())
|
||||
if err = client.MakeBucket(ctx, bucket, minio.MakeBucketOptions{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer client.RemoveBucket(context.Background(), bucket)
|
||||
s, _ := New(client, bucket, 64)
|
||||
storagetest.Run(t, s)
|
||||
s.Put(ctx, "signed", strings.NewReader("hello"), 5, "text/plain")
|
||||
defer s.Delete(context.Background(), "signed")
|
||||
u, err := s.PresignGet(ctx, "signed", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
response, err := http.Get(u.String())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := io.ReadAll(response.Body)
|
||||
response.Body.Close()
|
||||
if response.StatusCode != 200 || string(raw) != "hello" {
|
||||
t.Fatal(response.StatusCode, string(raw))
|
||||
}
|
||||
q := u.Query()
|
||||
q.Set("X-Amz-Signature", strings.Repeat("0", 64))
|
||||
u.RawQuery = q.Encode()
|
||||
response, err = http.Get(u.String())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != 403 {
|
||||
t.Fatal("tampered signature", response.StatusCode)
|
||||
}
|
||||
}
|
||||
93
storage/signed.go
Normal file
93
storage/signed.go
Normal file
@@ -0,0 +1,93 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Signer struct {
|
||||
store Store
|
||||
base *url.URL
|
||||
secret []byte
|
||||
now func() time.Time
|
||||
}
|
||||
|
||||
func NewSigner(store Store, baseURL string, secret []byte) (*Signer, error) {
|
||||
base, err := url.Parse(baseURL)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if store == nil || len(secret) < 32 || base.Host == "" || (base.Scheme != "https" && base.Scheme != "http") || base.RawQuery != "" || base.Fragment != "" || base.User != nil {
|
||||
return nil, errors.New("storage: store, HTTP base URL and >=32-byte secret required")
|
||||
}
|
||||
return &Signer{store: store, base: base, secret: append([]byte(nil), secret...), now: time.Now}, nil
|
||||
}
|
||||
func (s *Signer) signature(key, expiry string) string {
|
||||
mac := hmac.New(sha256.New, s.secret)
|
||||
io.WriteString(mac, "GET\n"+key+"\n"+expiry)
|
||||
return base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
|
||||
}
|
||||
func (s *Signer) PresignGet(ctx context.Context, key string, ttl time.Duration) (*url.URL, error) {
|
||||
if err := Check(ctx, key); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if ttl < time.Second || ttl > time.Hour {
|
||||
return nil, errors.New("storage: TTL must be 1s..1h")
|
||||
}
|
||||
expiry := strconv.FormatInt(s.now().Add(ttl).Unix(), 10)
|
||||
u := *s.base
|
||||
u.RawQuery = url.Values{"key": {key}, "expires": {expiry}, "signature": {s.signature(key, expiry)}}.Encode()
|
||||
return &u, nil
|
||||
}
|
||||
|
||||
// ServeHTTP serves signed downloads as attachments. Issue links only after
|
||||
// authorization; possession of an unexpired URL grants access to its object.
|
||||
func (s *Signer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != "GET" && r.Method != "HEAD" {
|
||||
w.Header().Set("Allow", "GET, HEAD")
|
||||
http.Error(w, "method not allowed", 405)
|
||||
return
|
||||
}
|
||||
q, err := url.ParseQuery(r.URL.RawQuery)
|
||||
if err != nil || len(q) != 3 || len(q["key"]) != 1 || len(q["expires"]) != 1 || len(q["signature"]) != 1 {
|
||||
http.Error(w, "invalid link", 403)
|
||||
return
|
||||
}
|
||||
key, expiry := q.Get("key"), q.Get("expires")
|
||||
expires, err := strconv.ParseInt(expiry, 10, 64)
|
||||
now := s.now().Unix()
|
||||
if err != nil || expires <= now || expires > now+3600 || Check(r.Context(), key) != nil || !hmac.Equal([]byte(q.Get("signature")), []byte(s.signature(key, expiry))) {
|
||||
http.Error(w, "invalid or expired link", 403)
|
||||
return
|
||||
}
|
||||
object, err := s.store.Get(r.Context(), key)
|
||||
if err != nil {
|
||||
status := 500
|
||||
if errors.Is(err, ErrNotFound) {
|
||||
status = 404
|
||||
}
|
||||
http.Error(w, "object unavailable", status)
|
||||
return
|
||||
}
|
||||
defer object.Body.Close()
|
||||
contentType := object.ContentType
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Disposition", "attachment")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Cache-Control", "private, no-store")
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(object.Size, 10))
|
||||
if r.Method == "GET" {
|
||||
_, _ = io.Copy(w, object.Body)
|
||||
}
|
||||
}
|
||||
35
storage/signed_test.go
Normal file
35
storage/signed_test.go
Normal file
@@ -0,0 +1,35 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/local"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestLocalSignedDownload(t *testing.T) {
|
||||
s, _ := local.New(t.TempDir(), 64)
|
||||
s.Put(context.Background(), "file", strings.NewReader("hello"), 5, "text/plain")
|
||||
signer, _ := storage.NewSigner(s, "https://example.test/download", []byte(strings.Repeat("x", 32)))
|
||||
u, err := signer.PresignGet(context.Background(), "file", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
request := httptest.NewRequest("GET", u.String(), nil)
|
||||
response := httptest.NewRecorder()
|
||||
signer.ServeHTTP(response, request)
|
||||
if response.Code != 200 || response.Body.String() != "hello" || response.Header().Get("Content-Disposition") != "attachment" {
|
||||
t.Fatal(response)
|
||||
}
|
||||
q := u.Query()
|
||||
q.Set("key", "another")
|
||||
u.RawQuery = q.Encode()
|
||||
response = httptest.NewRecorder()
|
||||
signer.ServeHTTP(response, httptest.NewRequest("GET", u.String(), nil))
|
||||
if response.Code != 403 {
|
||||
t.Fatal(response.Code)
|
||||
}
|
||||
}
|
||||
68
storage/storage.go
Normal file
68
storage/storage.go
Normal file
@@ -0,0 +1,68 @@
|
||||
// Package storage defines streaming object storage; callers own returned bodies.
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
var ErrNotFound = errors.New("storage: object not found")
|
||||
var ErrTooLarge = errors.New("storage: object exceeds limit")
|
||||
|
||||
type Info struct {
|
||||
Key string `json:"key"`
|
||||
Size int64 `json:"size"`
|
||||
ContentType string `json:"contentType"`
|
||||
ModifiedAt time.Time `json:"modifiedAt"`
|
||||
}
|
||||
type Object struct {
|
||||
Info
|
||||
Body io.ReadCloser
|
||||
}
|
||||
type Store interface {
|
||||
Put(context.Context, string, io.Reader, int64, string) (Info, error)
|
||||
Get(context.Context, string) (Object, error)
|
||||
Delete(context.Context, string) error
|
||||
}
|
||||
|
||||
// Presigner is an optional capability. URLs grant temporary read access and
|
||||
// must not be logged. Providers may expire them earlier when credentials expire.
|
||||
type Presigner interface {
|
||||
PresignGet(context.Context, string, time.Duration) (*url.URL, error)
|
||||
}
|
||||
|
||||
func Check(ctx context.Context, key string) error {
|
||||
if ctx == nil {
|
||||
return errors.New("storage: nil context")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if key == "" || len(key) > 1024 || !utf8.ValidString(key) || strings.Contains(key, "\\") || strings.IndexFunc(key, unicode.IsControl) >= 0 {
|
||||
return errors.New("storage: invalid key")
|
||||
}
|
||||
for _, part := range strings.Split(key, "/") {
|
||||
if part == "" || part == "." || part == ".." {
|
||||
return errors.New("storage: invalid key path")
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type ContextReader struct {
|
||||
Context context.Context
|
||||
Reader io.Reader
|
||||
}
|
||||
|
||||
func (r ContextReader) Read(p []byte) (int, error) {
|
||||
if err := r.Context.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return r.Reader.Read(p)
|
||||
}
|
||||
59
storage/storagetest/contract.go
Normal file
59
storage/storagetest/contract.go
Normal file
@@ -0,0 +1,59 @@
|
||||
// Package storagetest provides shared adapter checks. Use a dedicated test
|
||||
// bucket/root and a Store configured with a 64-byte object limit.
|
||||
package storagetest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func Run(t *testing.T, store storage.Store) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
key := "contract/文件 +%.txt"
|
||||
defer store.Delete(ctx, key)
|
||||
if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Fatal("missing object", err)
|
||||
}
|
||||
for _, body := range []string{"original", "replacement"} {
|
||||
info, err := store.Put(ctx, key, strings.NewReader(body), int64(len(body)), "text/plain")
|
||||
if err != nil || info.Size != int64(len(body)) {
|
||||
t.Fatal(info, err)
|
||||
}
|
||||
object, err := store.Get(ctx, key)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, err := io.ReadAll(object.Body)
|
||||
object.Body.Close()
|
||||
if err != nil || string(data) != body || object.ContentType != "text/plain" {
|
||||
t.Fatal(string(data), object.Info, err)
|
||||
}
|
||||
}
|
||||
if _, err := store.Put(ctx, "tooBig", strings.NewReader(strings.Repeat("a", 65)), 65, "text/plain"); !errors.Is(err, storage.ErrTooLarge) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, key := range []string{"../bad", "a/../b", "/absolute", "a//b", "a\\b"} {
|
||||
if _, err := store.Put(ctx, key, strings.NewReader("a"), 1, "text/plain"); err == nil {
|
||||
t.Fatal(key)
|
||||
}
|
||||
}
|
||||
canceled, cancel := context.WithCancel(ctx)
|
||||
cancel()
|
||||
if _, err := store.Get(canceled, key); !errors.Is(err, context.Canceled) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := store.Delete(ctx, key); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := store.Delete(ctx, key); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
78
telemetry/adapters.go
Normal file
78
telemetry/adapters.go
Normal file
@@ -0,0 +1,78 @@
|
||||
package telemetry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"io"
|
||||
"net"
|
||||
"sync"
|
||||
)
|
||||
|
||||
type spanBody struct {
|
||||
io.ReadCloser
|
||||
span trace.Span
|
||||
once sync.Once
|
||||
}
|
||||
|
||||
func (b *spanBody) finish(err error) {
|
||||
b.once.Do(func() {
|
||||
if err != nil && err != io.EOF {
|
||||
b.span.SetStatus(codes.Error, "body read failed")
|
||||
}
|
||||
b.span.End()
|
||||
})
|
||||
}
|
||||
func (b *spanBody) Read(p []byte) (int, error) {
|
||||
n, err := b.ReadCloser.Read(p)
|
||||
if err != nil {
|
||||
b.finish(err)
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
func (b *spanBody) Close() error { err := b.ReadCloser.Close(); b.finish(err); return err }
|
||||
|
||||
type RedisHook struct{ Runtime *Runtime }
|
||||
|
||||
func (h RedisHook) DialHook(next redis.DialHook) redis.DialHook {
|
||||
return func(ctx context.Context, network, addr string) (conn net.Conn, err error) {
|
||||
err = h.Runtime.Do(ctx, "redis.connect", func(ctx context.Context) error { conn, err = next(ctx, network, addr); return err })
|
||||
return
|
||||
}
|
||||
}
|
||||
func (h RedisHook) ProcessHook(next redis.ProcessHook) redis.ProcessHook {
|
||||
return func(ctx context.Context, cmd redis.Cmder) error {
|
||||
var result error
|
||||
_ = h.Runtime.Do(ctx, "redis.command", func(ctx context.Context) error {
|
||||
result = next(ctx, cmd)
|
||||
if result == redis.Nil {
|
||||
return nil
|
||||
}
|
||||
return result
|
||||
})
|
||||
return result
|
||||
}
|
||||
}
|
||||
func (h RedisHook) ProcessPipelineHook(next redis.ProcessPipelineHook) redis.ProcessPipelineHook {
|
||||
return func(ctx context.Context, cmds []redis.Cmder) error {
|
||||
return h.Runtime.Do(ctx, "redis.pipeline", func(ctx context.Context) error { return next(ctx, cmds) })
|
||||
}
|
||||
}
|
||||
|
||||
// SQL records execution spans without SQL text or arguments. Query spans cover
|
||||
// obtaining rows; iteration/Scan remain caller-owned. Use Do around transactions.
|
||||
type SQL struct {
|
||||
DB *sql.DB
|
||||
Runtime *Runtime
|
||||
}
|
||||
|
||||
func (d SQL) ExecContext(ctx context.Context, query string, args ...any) (result sql.Result, err error) {
|
||||
err = d.Runtime.Do(ctx, "db.exec", func(ctx context.Context) error { result, err = d.DB.ExecContext(ctx, query, args...); return err })
|
||||
return
|
||||
}
|
||||
func (d SQL) QueryContext(ctx context.Context, query string, args ...any) (rows *sql.Rows, err error) {
|
||||
err = d.Runtime.Do(ctx, "db.query", func(ctx context.Context) error { rows, err = d.DB.QueryContext(ctx, query, args...); return err })
|
||||
return
|
||||
}
|
||||
187
telemetry/telemetry.go
Normal file
187
telemetry/telemetry.go
Normal file
@@ -0,0 +1,187 @@
|
||||
// Package telemetry adds opt-in OpenTelemetry without process-global providers.
|
||||
package telemetry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/metric"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
|
||||
"go.opentelemetry.io/otel/sdk/resource"
|
||||
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"math"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Service string `json:"service"`
|
||||
SampleRatio float64 `json:"sampleRatio"`
|
||||
Exporter sdktrace.SpanExporter `json:"-"`
|
||||
Reader sdkmetric.Reader `json:"-"`
|
||||
}
|
||||
type Runtime struct {
|
||||
traces *sdktrace.TracerProvider
|
||||
meters *sdkmetric.MeterProvider
|
||||
tracer trace.Tracer
|
||||
propagator propagation.TextMapPropagator
|
||||
duration metric.Float64Histogram
|
||||
requests metric.Int64Counter
|
||||
}
|
||||
|
||||
func New(cfg Config) (*Runtime, error) {
|
||||
if cfg.Service == "" || math.IsNaN(cfg.SampleRatio) || cfg.SampleRatio < 0 || cfg.SampleRatio > 1 {
|
||||
return nil, errors.New("telemetry: service and sampleRatio in 0..1 required")
|
||||
}
|
||||
res := resource.NewSchemaless(attribute.String("service.name", cfg.Service))
|
||||
traceOptions := []sdktrace.TracerProviderOption{sdktrace.WithResource(res), sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(cfg.SampleRatio)))}
|
||||
if cfg.Exporter != nil {
|
||||
traceOptions = append(traceOptions, sdktrace.WithBatcher(cfg.Exporter, sdktrace.WithMaxQueueSize(512), sdktrace.WithMaxExportBatchSize(128), sdktrace.WithExportTimeout(3*time.Second)))
|
||||
}
|
||||
metricOptions := []sdkmetric.Option{sdkmetric.WithResource(res)}
|
||||
if cfg.Reader != nil {
|
||||
metricOptions = append(metricOptions, sdkmetric.WithReader(cfg.Reader))
|
||||
}
|
||||
r := &Runtime{traces: sdktrace.NewTracerProvider(traceOptions...), meters: sdkmetric.NewMeterProvider(metricOptions...), propagator: propagation.TraceContext{}}
|
||||
r.tracer = r.traces.Tracer("git.yuntuo.hk/k2safe/go-omni")
|
||||
meter := r.meters.Meter("git.yuntuo.hk/k2safe/go-omni")
|
||||
r.duration, _ = meter.Float64Histogram("http.server.request.duration", metric.WithUnit("s"), metric.WithExplicitBucketBoundaries(.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10))
|
||||
r.requests, _ = meter.Int64Counter("http.server.requests")
|
||||
return r, nil
|
||||
}
|
||||
func (r *Runtime) Start(ctx context.Context, name string, options ...trace.SpanStartOption) (context.Context, trace.Span) {
|
||||
ctx, span := r.tracer.Start(ctx, name, options...)
|
||||
ctx = logx.WithTraceID(ctx, span.SpanContext().TraceID().String())
|
||||
return ctx, span
|
||||
}
|
||||
func (r *Runtime) Do(ctx context.Context, name string, work func(context.Context) error) (err error) {
|
||||
if ctx == nil || work == nil || name == "" {
|
||||
return errors.New("telemetry: context, operation name and work required")
|
||||
}
|
||||
ctx, span := r.Start(ctx, name)
|
||||
defer span.End()
|
||||
defer func() {
|
||||
if p := recover(); p != nil {
|
||||
span.SetStatus(codes.Error, "panic")
|
||||
panic(p)
|
||||
}
|
||||
}()
|
||||
err = work(ctx)
|
||||
if err != nil {
|
||||
span.SetStatus(codes.Error, "operation failed")
|
||||
}
|
||||
return err
|
||||
}
|
||||
func (r *Runtime) Middleware() httpx.Middleware {
|
||||
return func(next httpx.Handler) httpx.Handler {
|
||||
return func(c *httpx.Context) (err error) {
|
||||
parent := c.Context()
|
||||
ctx := r.propagator.Extract(parent, propagation.HeaderCarrier(c.Request().Header))
|
||||
route := httpx.RoutePattern(parent)
|
||||
if route == "" {
|
||||
route = "unmatched"
|
||||
}
|
||||
method := c.Request().Method
|
||||
switch method {
|
||||
case "GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS", "CONNECT", "TRACE":
|
||||
default:
|
||||
method = "OTHER"
|
||||
}
|
||||
ctx, span := r.Start(ctx, method+" "+route, trace.WithSpanKind(trace.SpanKindServer))
|
||||
c.WithContext(ctx)
|
||||
started := time.Now()
|
||||
defer func() {
|
||||
status := c.Status()
|
||||
if err != nil && !c.Responded() {
|
||||
status = apierrors.From(err).Status
|
||||
}
|
||||
p := recover()
|
||||
if p != nil {
|
||||
status = 500
|
||||
}
|
||||
attrs := []attribute.KeyValue{attribute.String("http.request.method", method), attribute.String("http.route", route), attribute.Int("http.response.status_code", status)}
|
||||
span.SetAttributes(attrs...)
|
||||
if status >= 500 {
|
||||
span.SetStatus(codes.Error, "request failed")
|
||||
}
|
||||
r.requests.Add(ctx, 1, metric.WithAttributes(attrs...))
|
||||
r.duration.Record(ctx, time.Since(started).Seconds(), metric.WithAttributes(attrs...))
|
||||
span.End()
|
||||
// Preserve correlation for outer access-log middleware without
|
||||
// exposing an ended span as its active span.
|
||||
c.WithContext(logx.WithTraceID(parent, span.SpanContext().TraceID().String()))
|
||||
if p != nil {
|
||||
panic(p)
|
||||
}
|
||||
}()
|
||||
return next(c)
|
||||
}
|
||||
}
|
||||
}
|
||||
func (r *Runtime) Shutdown(ctx context.Context) error {
|
||||
return errors.Join(r.traces.Shutdown(ctx), r.meters.Shutdown(ctx))
|
||||
}
|
||||
func (r *Runtime) Close() error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
return r.Shutdown(ctx)
|
||||
}
|
||||
func (r *Runtime) Flush(ctx context.Context) error {
|
||||
return errors.Join(r.traces.ForceFlush(ctx), r.meters.ForceFlush(ctx))
|
||||
}
|
||||
|
||||
// Transport borrows Base. Headers are injected on a clone, and the span lives
|
||||
// until EOF/Close. URL paths, query strings and credentials are never recorded.
|
||||
type Transport struct {
|
||||
Runtime *Runtime
|
||||
Base http.RoundTripper
|
||||
}
|
||||
|
||||
func (t Transport) RoundTrip(request *http.Request) (*http.Response, error) {
|
||||
if t.Runtime == nil {
|
||||
return nil, errors.New("telemetry: runtime required")
|
||||
}
|
||||
ctx, span := t.Runtime.Start(request.Context(), "http.client", trace.WithSpanKind(trace.SpanKindClient))
|
||||
defer func() {
|
||||
if p := recover(); p != nil {
|
||||
span.SetStatus(codes.Error, "panic")
|
||||
span.End()
|
||||
panic(p)
|
||||
}
|
||||
}()
|
||||
req := request.Clone(ctx)
|
||||
if req.Header == nil {
|
||||
req.Header = make(http.Header)
|
||||
}
|
||||
t.Runtime.propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))
|
||||
base := t.Base
|
||||
if base == nil {
|
||||
base = http.DefaultTransport
|
||||
}
|
||||
response, err := base.RoundTrip(req)
|
||||
if err != nil {
|
||||
span.SetStatus(codes.Error, "transport failed")
|
||||
span.End()
|
||||
if response != nil && response.Body != nil {
|
||||
response.Body.Close()
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
if response == nil || response.Body == nil {
|
||||
span.SetStatus(codes.Error, "invalid response")
|
||||
span.End()
|
||||
return nil, errors.New("telemetry: invalid transport response")
|
||||
}
|
||||
span.SetAttributes(attribute.Int("http.response.status_code", response.StatusCode))
|
||||
if response.StatusCode >= 500 {
|
||||
span.SetStatus(codes.Error, "upstream failed")
|
||||
}
|
||||
response.Body = &spanBody{ReadCloser: response.Body, span: span}
|
||||
return response, nil
|
||||
}
|
||||
117
telemetry/telemetry_test.go
Normal file
117
telemetry/telemetry_test.go
Normal file
@@ -0,0 +1,117 @@
|
||||
package telemetry_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
omni "git.yuntuo.hk/k2safe/go-omni"
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpx"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"git.yuntuo.hk/k2safe/go-omni/telemetry"
|
||||
"github.com/redis/go-redis/v9"
|
||||
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
|
||||
"go.opentelemetry.io/otel/sdk/metric/metricdata"
|
||||
"go.opentelemetry.io/otel/sdk/trace/tracetest"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestHTTPTracePropagationMetricsAndRedisNil(t *testing.T) {
|
||||
exporter := tracetest.NewInMemoryExporter()
|
||||
reader := sdkmetric.NewManualReader()
|
||||
runtime, err := telemetry.New(telemetry.Config{Service: "test", SampleRatio: 1, Exporter: exporter, Reader: reader})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer runtime.Close()
|
||||
ctx := context.Background()
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if !strings.HasPrefix(r.Header.Get("Traceparent"), "00-11111111111111111111111111111111-") {
|
||||
t.Error("missing propagated trace")
|
||||
}
|
||||
io.WriteString(w, "ok")
|
||||
}))
|
||||
defer upstream.Close()
|
||||
app := omni.New(omni.WithLogger(false))
|
||||
app.Use(func(next httpx.Handler) httpx.Handler {
|
||||
return func(c *httpx.Context) error {
|
||||
err := next(c)
|
||||
if logx.TraceID(c.Context()) != "11111111111111111111111111111111" {
|
||||
t.Error("outer access log lost trace correlation")
|
||||
}
|
||||
return err
|
||||
}
|
||||
})
|
||||
app.Use(runtime.Middleware())
|
||||
app.GET("/users/:id", func(c *httpx.Context) error {
|
||||
if logx.TraceID(c.Context()) != "11111111111111111111111111111111" {
|
||||
t.Error("log correlation")
|
||||
}
|
||||
req, _ := http.NewRequestWithContext(c.Context(), "GET", upstream.URL+"?secret=hidden", nil)
|
||||
resp, err := (telemetry.Transport{Runtime: runtime}).RoundTrip(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
io.ReadAll(resp.Body)
|
||||
resp.Body.Close()
|
||||
if req.Header.Get("Traceparent") != "" {
|
||||
t.Error("caller headers mutated")
|
||||
}
|
||||
return c.OK("ok")
|
||||
})
|
||||
req := httptest.NewRequest("GET", "/users/privateUser?secret=hidden", nil)
|
||||
req.Header.Set("Traceparent", "00-11111111111111111111111111111111-2222222222222222-01")
|
||||
rec := httptest.NewRecorder()
|
||||
app.ServeHTTP(rec, req)
|
||||
if rec.Code != 200 {
|
||||
t.Fatal(rec.Code)
|
||||
}
|
||||
hook := telemetry.RedisHook{Runtime: runtime}
|
||||
if err = hook.ProcessHook(func(context.Context, redis.Cmder) error { return redis.Nil })(ctx, redis.NewStringCmd(ctx, "GET", "secret")); err != redis.Nil {
|
||||
t.Fatal("cache miss was swallowed", err)
|
||||
}
|
||||
if err = runtime.Flush(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
spans := exporter.GetSpans()
|
||||
if len(spans) != 3 {
|
||||
t.Fatalf("got %d spans", len(spans))
|
||||
}
|
||||
var client, server bool
|
||||
for _, s := range spans {
|
||||
if strings.Contains(s.Name, "privateUser") || strings.Contains(s.Name, "secret") {
|
||||
t.Fatal(s.Name)
|
||||
}
|
||||
if s.SpanKind == trace.SpanKindServer {
|
||||
server = true
|
||||
if s.Name != "GET /users/:id" || s.Parent.SpanID().String() != "2222222222222222" {
|
||||
t.Fatal(s.Name, s.Parent)
|
||||
}
|
||||
}
|
||||
if s.SpanKind == trace.SpanKindClient {
|
||||
client = true
|
||||
}
|
||||
}
|
||||
if !client || !server {
|
||||
t.Fatal("missing client/server span")
|
||||
}
|
||||
var metrics metricdata.ResourceMetrics
|
||||
if err = reader.Collect(ctx, &metrics); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var count int64
|
||||
for _, scope := range metrics.ScopeMetrics {
|
||||
for _, m := range scope.Metrics {
|
||||
if sum, ok := m.Data.(metricdata.Sum[int64]); ok {
|
||||
for _, p := range sum.DataPoints {
|
||||
count += p.Value
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if count != 1 {
|
||||
t.Fatal("request metric", count)
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user