feat: complete API-driven projects with infrastructure and business acceptance

This commit is contained in:
2026-09-23 17:33:50 +08:00
parent 04a1f7999d
commit 2c74dc0859
137 changed files with 11926 additions and 105 deletions

115
storage/README.md Normal file
View File

@@ -0,0 +1,115 @@
# 对象存储与上传
核心 `storage.Store` 保持 `Put/Get/Delete`,不会强迫每个适配器实现云厂商的所有能力。业务可按需依赖可选接口:
| 能力 | 接口 | Local | S3 / R2 | OSS |
| --- | --- | --- | --- | --- |
| 读写删除 | Store | 支持 | 支持 | 支持 |
| 临时下载 | Presigner | NewSigner | 支持 | 支持 |
| 限定大小/类型的直传 | UploadPresigner | NewUploadSigner | 支持 | 支持,需要配置签名 Header |
| 创建、上传、查询、完成、取消分片 | MultipartStore | 当前不提供 | 支持 | 支持 |
| 分片直传签名 | PartPresigner | 当前不提供 | 支持 | 支持,需要配置签名 Header |
没有固定业务字段、目录规则或身份模型。鉴权、对象归属、业务配额由应用定义。SDK 客户端和连接池由应用持有;适配器不会在一次调用后关闭客户端。
## 服务端上传限制
```go
uploader, err := storage.NewUploader(files, storage.UploadPolicy{
MaxBytes: 32 << 20,
ContentTypes: []string{"image/png", "image/jpeg", "application/pdf"},
})
if err != nil { return err }
info, err := uploader.Put(ctx, key, reader, size, contentType)
```
`files` 是任意 Store。MaxBytes 必须为正数;ContentTypes 是精确 MIME 白名单,忽略参数,不接受通配符。空白名单允许所有合法声明类型。单次上传检查声明大小、流的实际长度;设置类型白名单时,再以头部最多 512 字节嗅探结果核对声明类型。适配器本身的大小限制仍然有效,建议与策略统一。这里只读取有限前缀,不把完整文件放进内存。
MIME 嗅探用于粗粒度识别,不验证完整文件结构或病毒;CSV/JSON 等类型可能被嗅探成 text/plain。需要复杂格式判断时,在业务校验流程中处理。必须传入可终止、正确返回 EOF 的 reader;ContextReader 不能中断任意用户 Reader 中已经阻塞的 Read。
使用 Local 时先写临时文件,校验失败不会替换旧对象。云端网络中断可能产生结果未知或遗留未完成上传;使用唯一暂存 key,并通过查询/对账确认后再更新业务记录,不要把上传 error 当成绝对未写入。
## 云端直传
```go
// files 为 S3/R2 或 OSS 的 Store;Uploader 统一执行策略检查。
uploader, err := storage.NewUploader(files, storage.UploadPolicy{
MaxBytes: 32 << 20, ContentTypes: []string{"image/png", "image/jpeg"},
})
if err != nil { return err }
// 业务先授权并生成暂存 key,随后签发。
request, err := uploader.PresignPut(ctx, key, size, "image/png", 5*time.Minute)
```
返回 `method/url/headers/expiresAt`,有效期 1 秒到 1 小时。签名绑定 key、方法、确切 Content-Length 和声明 Content-Type,缺少这些签名约束时适配器报错。SDK 凭据提前失效时,链接也可能提前失效。
浏览器请求使用原始 File/Blob 作为 body,不能额外包 FormData;Content-Length 由浏览器按 body 长度生成,禁止手动设置。其余 Headers 按返回值原样发送:
```js
const headers = new Headers();
for (const [name, values] of Object.entries(request.headers)) {
if (name.toLowerCase() !== 'content-length') headers.set(name, values.join(', '));
}
const response = await fetch(request.url, {
method: request.method, headers, body: file,
});
if (!response.ok) throw new Error(`upload failed: ${response.status}`);
```
桶的 CORS 由部署方配置,允许业务域名、PUT 和返回的请求头;分片直传还需暴露 ETag 响应头。前端不能得到永久凭据。链接在到期前可能重复使用并覆盖同一个 key,不是一次性凭证。用随机暂存 key,业务确认前不要当作已发布文件。
**云端直传的类型限制针对声明 Header,无法由 API 服务实时检查文件字节。** 内容校验/扫描需要后处理;验证通过前不要公开文件。前端报告“上传完成”不等于文件可信。具体平台仍需实网验收,不能仅凭 S3 兼容声明判断一致。
OSS 使用官方 Go SDK v2:
```go
client := ossSdk.NewClient(ossSdk.LoadDefaultConfig().
WithRegion("cn-hangzhou").
WithSignatureVersion(ossSdk.SignatureVersionV4).
WithAdditionalHeaders([]string{"content-length"}).
WithCredentialsProvider(credentialProvider))
files, err := oss.New(client, bucket, 32<<20)
```
适配器不修改传入的 client;缺少签名大小约束会返回明确错误。SDK 的 presign 实现与返回 SignedHeaders 都经过检查。
## 本地签名上传
```go
files, err := local.New("./privateFiles", 32<<20)
if err != nil { return err }
signer, err := storage.NewUploadSigner(files, "https://files.example/upload", secret,
storage.UploadPolicy{MaxBytes: 32<<20, ContentTypes: []string{"image/png"}})
if err != nil { return err }
// signer 实现 http.Handler,可挂到现有的 HTTP 服务;secret 至少 32 字节。
request, err := signer.PresignPut(ctx, key, size, "image/png", time.Minute)
```
Handler 检查签名、有效期、单值参数、Content-Length/Content-Type,以及实际内容大小和嗅探类型,成功返回 201。本地存储只开放一个受控入口,不把磁盘目录映射成公共目录。原 `NewSigner` 继续只服务下载。请求超时、CORS、并发限制和速率限制由承载该 Handler 的服务配置。
## 分片与续传
S3/R2/OSS 均实现 `MultipartStore` 和 `PartPresigner`。Uploader 也暴露相同操作;不支持的底层返回 `ErrUnsupported`。
```go
session, err := uploader.CreateMultipart(ctx, key, "application/pdf")
if err != nil { return err }
// 业务持久保存 session,与已授权调用者关联,不能信任客户端任意传入的 key/uploadId。
part, err := uploader.UploadPart(ctx, session, 1, reader, size)
// 或签发单个分片的 PUT 请求:
request, err := uploader.PresignPart(ctx, session, 1, size, 5*time.Minute)
// 重连后分页读取实际已上传的分片,NextMarker == 0 表示结束。
page, err := uploader.ListParts(ctx, session, 0, 1000)
// 按 Number 升序从 1 连续提交,ETag 使用存储平台的返回值。
info, err := uploader.CompleteMultipart(ctx, session, parts)
// 用户取消/任务过期时,用仍有效的 context 显式清理。
err = uploader.AbortMultipart(ctx, session)
```
以上为互斥分支的 API 索引,实际调用需逐步处理 error。续传由业务保存 session,查询已有分片后重传缺失部分;框架不维护有身份语义的上传任务表,也不代替业务调度重试。
统一边界:每个对象最多 10,000 片、每片最多 5 GiB,除末片外至少 5 MiB;这是适配层为跨平台复用选取的共同约束。空文件走普通 Put。ListParts 每页最多 1,000 条。平台/桶限制可能更低。默认单对象仍为 32 MiB,需要大文件时显式提高适配器与 UploadPolicy 的限制;OSS 简单 Put 和云端单次签名 PUT 最大 5 GiB,更大使用分片。
Complete 会向平台分页核对全部分片的序号、ETag、真实大小和累计上限,前端填写的 `Part.Size` 不参与信任判断。完成时停止并发替换分片;Provider 仍会再次校验 ETag。返回 Info 的 Size 为已核对大小,ContentType 留空,完整对象元数据可通过 Get 获取。Multipart 的类型策略检查创建时的声明,实际内容应在暂存对象合并后校验。未完成的分片占用存储空间:应用需要超时取消或配置桶生命周期;URL 过期不会自动清理分片。
验收:Local 实际字节与原子性、S3/OSS SDK HTTP 协议测试、R2 签名配置测试、真实隔离 MinIO 验证。OSS/R2 尚未使用云账号实测。复现 MinIO 验证:先准备镜像,再运行 `python3 scripts/verifyUploads.py`;报告见 [uploadsLocal.json](../performance/uploadsLocal.json)。

173
storage/multipart.go Normal file
View File

@@ -0,0 +1,173 @@
package storage
import (
"context"
"errors"
"io"
"strings"
"time"
"unicode"
)
// MultipartUpload is provider state, not authorization. Applications must bind
// it to the authenticated caller in their own durable session store.
type MultipartUpload struct {
Key string `json:"key"`
UploadID string `json:"uploadId"`
}
type Part struct {
Number int `json:"number"`
ETag string `json:"etag"`
Size int64 `json:"size"`
}
type PartsPage struct {
Parts []Part `json:"parts"`
NextMarker int `json:"nextMarker"`
}
type MultipartStore interface {
CreateMultipart(context.Context, string, string) (MultipartUpload, error)
UploadPart(context.Context, MultipartUpload, int, io.Reader, int64) (Part, error)
ListParts(context.Context, MultipartUpload, int, int) (PartsPage, error)
CompleteMultipart(context.Context, MultipartUpload, []Part) (Info, error)
AbortMultipart(context.Context, MultipartUpload) error
}
type PartPresigner interface {
PresignPart(context.Context, MultipartUpload, int, int64, time.Duration) (UploadRequest, error)
}
func CheckMultipart(ctx context.Context, upload MultipartUpload) error {
if err := Check(ctx, upload.Key); err != nil {
return err
}
if upload.UploadID == "" || len(upload.UploadID) > 2048 || strings.IndexFunc(upload.UploadID, unicode.IsControl) >= 0 {
return errors.New("storage: invalid upload ID")
}
return nil
}
func CheckPart(ctx context.Context, upload MultipartUpload, number int, size, maxBytes int64) error {
if err := CheckMultipart(ctx, upload); err != nil {
return err
}
if number < 1 || number > 10000 || size <= 0 {
return errors.New("storage: part number must be 1..10000 and size positive")
}
if size > maxBytes || size > 5<<30 {
return ErrTooLarge
}
return nil
}
func CheckPartsPage(ctx context.Context, upload MultipartUpload, after, limit int) error {
if err := CheckMultipart(ctx, upload); err != nil {
return err
}
if after < 0 || after > 10000 || limit < 1 || limit > 1000 {
return errors.New("storage: invalid part pagination")
}
return nil
}
// VerifyParts checks provider-reported sizes/ETags before completing. Caller
// sizes are not trusted. Do not upload/replace parts concurrently with completion.
// A common 5 MiB minimum (except last) keeps S3/OSS interchangeable.
func VerifyParts(ctx context.Context, store MultipartStore, upload MultipartUpload, parts []Part, maxBytes int64) (int64, error) {
if err := CheckMultipart(ctx, upload); err != nil {
return 0, err
}
if len(parts) == 0 || len(parts) > 10000 || maxBytes <= 0 {
return 0, errors.New("storage: invalid completion")
}
for i, p := range parts {
if p.Number != i+1 || p.ETag == "" || len(p.ETag) > 256 || strings.ContainsAny(p.ETag, "\r\n") {
return 0, errors.New("storage: ordered contiguous parts and ETags required")
}
}
var total int64
marker, seen := 0, 0
for {
page, err := store.ListParts(ctx, upload, marker, 1000)
if err != nil {
return 0, err
}
if len(page.Parts) > 1000 {
return 0, errors.New("storage: invalid provider page")
}
for _, actual := range page.Parts {
if seen >= len(parts) || actual.Number != parts[seen].Number || strings.Trim(actual.ETag, "\"") != strings.Trim(parts[seen].ETag, "\"") || actual.Size <= 0 {
return 0, errors.New("storage: parts changed or incomplete")
}
if actual.Size > 5<<30 || actual.Size > maxBytes-total {
return 0, ErrTooLarge
}
if seen < len(parts)-1 && actual.Size < 5<<20 {
return 0, errors.New("storage: non-final part smaller than 5 MiB")
}
total += actual.Size
seen++
}
if page.NextMarker == 0 {
break
}
if len(page.Parts) == 0 || page.NextMarker <= marker || page.NextMarker > 10000 || page.NextMarker != page.Parts[len(page.Parts)-1].Number {
return 0, errors.New("storage: invalid provider pagination")
}
marker = page.NextMarker
}
if seen != len(parts) {
return 0, errors.New("storage: missing parts")
}
return total, nil
}
func (u *Uploader) CreateMultipart(ctx context.Context, key, contentType string) (MultipartUpload, error) {
if err := u.policy.Check(0, contentType); err != nil {
return MultipartUpload{}, err
}
s, ok := u.store.(MultipartStore)
if !ok {
return MultipartUpload{}, ErrUnsupported
}
return s.CreateMultipart(ctx, key, contentType)
}
func (u *Uploader) UploadPart(ctx context.Context, upload MultipartUpload, number int, body io.Reader, size int64) (Part, error) {
if err := CheckPart(ctx, upload, number, size, u.policy.MaxBytes); err != nil {
return Part{}, err
}
s, ok := u.store.(MultipartStore)
if !ok {
return Part{}, ErrUnsupported
}
return s.UploadPart(ctx, upload, number, body, size)
}
func (u *Uploader) PresignPart(ctx context.Context, upload MultipartUpload, number int, size int64, ttl time.Duration) (UploadRequest, error) {
if err := CheckPart(ctx, upload, number, size, u.policy.MaxBytes); err != nil {
return UploadRequest{}, err
}
s, ok := u.store.(PartPresigner)
if !ok {
return UploadRequest{}, ErrUnsupported
}
return s.PresignPart(ctx, upload, number, size, ttl)
}
func (u *Uploader) ListParts(ctx context.Context, upload MultipartUpload, after, limit int) (PartsPage, error) {
s, ok := u.store.(MultipartStore)
if !ok {
return PartsPage{}, ErrUnsupported
}
return s.ListParts(ctx, upload, after, limit)
}
func (u *Uploader) CompleteMultipart(ctx context.Context, upload MultipartUpload, parts []Part) (Info, error) {
s, ok := u.store.(MultipartStore)
if !ok {
return Info{}, ErrUnsupported
}
if _, err := VerifyParts(ctx, s, upload, parts, u.policy.MaxBytes); err != nil {
return Info{}, err
}
return s.CompleteMultipart(ctx, upload, parts)
}
func (u *Uploader) AbortMultipart(ctx context.Context, upload MultipartUpload) error {
s, ok := u.store.(MultipartStore)
if !ok {
return ErrUnsupported
}
return s.AbortMultipart(ctx, upload)
}

60
storage/multipart_test.go Normal file
View File

@@ -0,0 +1,60 @@
package storage
import (
"context"
"errors"
"testing"
)
type listedParts struct {
MultipartStore
parts []Part
calls int
loop bool
}
func (s *listedParts) ListParts(ctx context.Context, u MultipartUpload, marker, limit int) (PartsPage, error) {
s.calls++
if s.loop {
return PartsPage{Parts: []Part{{Number: 1, ETag: "a", Size: 5 << 20}}, NextMarker: marker}, nil
}
end := min(marker+limit, len(s.parts))
next := 0
if end < len(s.parts) {
next = end
}
return PartsPage{Parts: s.parts[marker:end], NextMarker: next}, nil
}
func TestVerifyPartsUsesProviderSizes(t *testing.T) {
upload := MultipartUpload{Key: "file", UploadID: "id"}
first := Part{Number: 1, ETag: "a", Size: 5 << 20}
last := Part{Number: 2, ETag: "b", Size: 3}
supplied := []Part{{Number: 1, ETag: "a", Size: 1}, {Number: 2, ETag: "b", Size: 1}}
s := &listedParts{parts: []Part{first, last}}
n, err := VerifyParts(context.Background(), s, upload, supplied, 6<<20)
if err != nil || n != (5<<20)+3 {
t.Fatal(n, err)
}
if _, err = VerifyParts(context.Background(), s, upload, supplied, 5<<20); !errors.Is(err, ErrTooLarge) {
t.Fatal(err)
}
for _, parts := range [][]Part{nil, {last, first}, {first, first}, {{Number: 1, ETag: "changed"}, last}, {first}} {
if _, err := VerifyParts(context.Background(), s, upload, parts, 6<<20); err == nil {
t.Fatal("bad completion accepted", parts)
}
}
s.parts[0].Size = 1
if _, err = VerifyParts(context.Background(), s, upload, supplied, 6<<20); err == nil {
t.Fatal("small non-final part accepted")
}
}
func TestVerifyPartsPagination(t *testing.T) {
s := &listedParts{}
for i := 1; i <= 1001; i++ {
s.parts = append(s.parts, Part{Number: i, ETag: "etag", Size: 5 << 20})
}
n, err := VerifyParts(context.Background(), s, MultipartUpload{Key: "file", UploadID: "id"}, s.parts, 10<<30)
if err != nil || s.calls != 2 || n != 1001*(5<<20) {
t.Fatal(n, s.calls, err)
}
}

View File

@@ -19,10 +19,10 @@ type Store struct {
maxBytes int64
}
// New borrows a configured SDK client. This adapter uses simple uploads; large
// multipart uploads are a separate provider capability, not buffered in memory.
// New borrows a configured SDK client. Put uses simple uploads (up to 5 GiB).
// Larger objects use the optional MultipartStore capability.
func New(client *sdk.Client, bucket string, maxBytes int64) (*Store, error) {
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<30 {
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<40 {
return nil, errors.New("storage/oss: invalid client, bucket or size limit")
}
if maxBytes == 0 {
@@ -41,7 +41,14 @@ func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int6
if size > s.maxBytes {
return storage.Info{}, storage.ErrTooLarge
}
_, err := s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: storage.ContextReader{Context: ctx, Reader: reader}})
if size > 5<<30 {
return storage.Info{}, storage.ErrTooLarge
}
checked, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, reader, size, contentType)
if err != nil {
return storage.Info{}, err
}
_, err = s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: checked})
if err != nil {
return storage.Info{}, mapError(err)
}

133
storage/oss/upload.go Normal file
View File

@@ -0,0 +1,133 @@
package oss
import (
"context"
"errors"
"io"
"net/http"
"strconv"
"time"
"git.yuntuo.hk/k2safe/go-omni/storage"
sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
)
// PresignPut requires an OSS V4 client configured with
// WithAdditionalHeaders([]string{"content-length"}). Missing signed constraints
// fail closed; the borrowed client is never mutated by this adapter.
func (s *Store) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (storage.UploadRequest, error) {
if err := storage.CheckUpload(ctx, key, size, s.maxBytes, contentType, ttl); err != nil {
return storage.UploadRequest{}, err
}
if size > 5<<30 {
return storage.UploadRequest{}, storage.ErrTooLarge
}
if contentType == "" {
contentType = "application/octet-stream"
}
r, err := s.client.Presign(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType)}, sdk.PresignExpires(ttl))
if err != nil {
return storage.UploadRequest{}, err
}
return uploadRequest(r, size, contentType)
}
func uploadRequest(r *sdk.PresignResult, size int64, contentType string) (storage.UploadRequest, error) {
headers := make(http.Header)
for k, v := range r.SignedHeaders {
headers.Set(k, v)
}
if headers.Get("Content-Length") != strconv.FormatInt(size, 10) || (contentType != "" && headers.Get("Content-Type") != contentType) {
return storage.UploadRequest{}, errors.New("storage/oss: upload requires signed Content-Length and Content-Type; configure V4 and WithAdditionalHeaders(content-length)")
}
return storage.UploadRequest{Method: r.Method, URL: r.URL, Headers: headers, ExpiresAt: r.Expiration}, nil
}
func (s *Store) CreateMultipart(ctx context.Context, key, contentType string) (storage.MultipartUpload, error) {
if err := storage.Check(ctx, key); err != nil {
return storage.MultipartUpload{}, err
}
if _, err := storage.MediaType(contentType); err != nil {
return storage.MultipartUpload{}, err
}
if contentType == "" {
contentType = "application/octet-stream"
}
r, err := s.client.InitiateMultipartUpload(ctx, &sdk.InitiateMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentType: sdk.Ptr(contentType)})
if err != nil {
return storage.MultipartUpload{}, err
}
return storage.MultipartUpload{Key: key, UploadID: sdk.ToString(r.UploadId)}, nil
}
func (s *Store) UploadPart(ctx context.Context, upload storage.MultipartUpload, number int, body io.Reader, size int64) (storage.Part, error) {
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
return storage.Part{}, err
}
reader, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, body, size, "")
if err != nil {
return storage.Part{}, err
}
r, err := s.client.UploadPart(ctx, &sdk.UploadPartRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumber: int32(number), ContentLength: sdk.Ptr(size), Body: reader})
if err != nil {
return storage.Part{}, mapError(err)
}
return storage.Part{Number: number, ETag: sdk.ToString(r.ETag), Size: size}, nil
}
func (s *Store) PresignPart(ctx context.Context, upload storage.MultipartUpload, number int, size int64, ttl time.Duration) (storage.UploadRequest, error) {
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
return storage.UploadRequest{}, err
}
if err := storage.CheckUpload(ctx, upload.Key, size, s.maxBytes, "", ttl); err != nil {
return storage.UploadRequest{}, err
}
r, err := s.client.Presign(ctx, &sdk.UploadPartRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumber: int32(number), ContentLength: sdk.Ptr(size), RequestCommon: sdk.RequestCommon{Headers: map[string]string{"Content-Type": "application/octet-stream"}}}, sdk.PresignExpires(ttl))
if err != nil {
return storage.UploadRequest{}, err
}
return uploadRequest(r, size, "application/octet-stream")
}
func (s *Store) ListParts(ctx context.Context, upload storage.MultipartUpload, after, limit int) (storage.PartsPage, error) {
if err := storage.CheckPartsPage(ctx, upload, after, limit); err != nil {
return storage.PartsPage{}, err
}
r, err := s.client.ListParts(ctx, &sdk.ListPartsRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumberMarker: int32(after), MaxParts: int32(limit)})
if err != nil {
return storage.PartsPage{}, err
}
page := storage.PartsPage{Parts: make([]storage.Part, 0, len(r.Parts))}
for _, p := range r.Parts {
page.Parts = append(page.Parts, storage.Part{Number: int(p.PartNumber), ETag: sdk.ToString(p.ETag), Size: p.Size})
}
if r.IsTruncated {
page.NextMarker = int(r.NextPartNumberMarker)
}
return page, nil
}
func (s *Store) CompleteMultipart(ctx context.Context, upload storage.MultipartUpload, parts []storage.Part) (storage.Info, error) {
size, err := storage.VerifyParts(ctx, s, upload, parts, s.maxBytes)
if err != nil {
return storage.Info{}, err
}
completed := make([]sdk.UploadPart, len(parts))
for i, p := range parts {
completed[i] = sdk.UploadPart{PartNumber: int32(p.Number), ETag: sdk.Ptr(p.ETag)}
}
_, err = s.client.CompleteMultipartUpload(ctx, &sdk.CompleteMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), CompleteMultipartUpload: &sdk.CompleteMultipartUpload{Parts: completed}})
if err != nil {
return storage.Info{}, err
}
return storage.Info{Key: upload.Key, Size: size, ModifiedAt: time.Now().UTC()}, nil
}
func (s *Store) AbortMultipart(ctx context.Context, upload storage.MultipartUpload) error {
if err := storage.CheckMultipart(ctx, upload); err != nil {
return err
}
_, err := s.client.AbortMultipartUpload(ctx, &sdk.AbortMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID)})
var serviceError *sdk.ServiceError
if errors.As(err, &serviceError) && serviceError.Code == "NoSuchUpload" {
return nil
}
return err
}
var _ storage.UploadPresigner = (*Store)(nil)
var _ storage.MultipartStore = (*Store)(nil)
var _ storage.PartPresigner = (*Store)(nil)

View File

@@ -0,0 +1,29 @@
package oss
import (
"context"
"net/http/httptest"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/internal/storagefixture"
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
)
func TestUploadProtocol(t *testing.T) {
server := httptest.NewTLSServer(storagefixture.MultipartHandler())
defer server.Close()
cfg := sdk.LoadDefaultConfig().WithRegion("cn-hangzhou").WithEndpoint(server.URL).WithUsePathStyle(true).WithCredentialsProvider(credentials.NewStaticCredentialsProvider("fixtureAccess", "fixtureSecret")).WithHttpClient(server.Client()).WithSignatureVersion(sdk.SignatureVersionV4)
missing, _ := New(sdk.NewClient(cfg), "testbucket", 6<<20)
if _, err := missing.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute); err == nil {
t.Fatal("unsigned length accepted")
}
cfg.WithAdditionalHeaders([]string{"content-length"})
s, err := New(sdk.NewClient(cfg), "testbucket", 6<<20)
if err != nil {
t.Fatal(err)
}
storagetest.RunUploads(t, s, server.Client())
}

View File

@@ -19,7 +19,7 @@ type Store struct {
}
func New(client *minio.Client, bucket string, maxBytes int64) (*Store, error) {
if client == nil || bucket == "" || maxBytes < 0 {
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<40 {
return nil, errors.New("storage/s3: client, bucket and nonnegative limit required")
}
if maxBytes == 0 {
@@ -37,7 +37,11 @@ func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int6
if size > s.maxBytes {
return storage.Info{}, storage.ErrTooLarge
}
result, err := s.client.PutObject(ctx, s.bucket, key, storage.ContextReader{Context: ctx, Reader: reader}, size, minio.PutObjectOptions{ContentType: contentType})
checked, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, reader, size, contentType)
if err != nil {
return storage.Info{}, err
}
result, err := s.client.PutObject(ctx, s.bucket, key, checked, size, minio.PutObjectOptions{ContentType: contentType})
if err != nil {
return storage.Info{}, err
}

View File

@@ -73,6 +73,26 @@ func TestS3Integration(t *testing.T) {
t.Fatal(err)
}
defer client.RemoveBucket(context.Background(), bucket)
uploads, _ := New(client, bucket, 6<<20)
httpClient := &http.Client{Timeout: 10 * time.Second}
storagetest.RunUploads(t, uploads, httpClient)
link, err := uploads.PresignPut(ctx, "constrained", 5, "text/plain", time.Minute)
if err != nil {
t.Fatal(err)
}
for _, tc := range []struct{ body, contentType string }{{"longer", "text/plain"}, {"hello", "image/png"}} {
req, _ := http.NewRequestWithContext(ctx, link.Method, link.URL, strings.NewReader(tc.body))
req.Header = link.Headers.Clone()
req.Header.Set("Content-Type", tc.contentType)
response, requestErr := httpClient.Do(req)
if requestErr != nil {
t.Fatal(requestErr)
}
response.Body.Close()
if response.StatusCode != 403 {
t.Fatalf("tampered upload constraints accepted: %d", response.StatusCode)
}
}
s, _ := New(client, bucket, 64)
storagetest.Run(t, s)
s.Put(ctx, "signed", strings.NewReader("hello"), 5, "text/plain")

128
storage/s3/upload.go Normal file
View File

@@ -0,0 +1,128 @@
package s3
import (
"context"
"errors"
"io"
"net/http"
"net/url"
"strconv"
"strings"
"time"
"git.yuntuo.hk/k2safe/go-omni/storage"
"github.com/minio/minio-go/v7"
)
func (s *Store) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (storage.UploadRequest, error) {
if err := storage.CheckUpload(ctx, key, size, s.maxBytes, contentType, ttl); err != nil {
return storage.UploadRequest{}, err
}
if size > 5<<30 {
return storage.UploadRequest{}, storage.ErrTooLarge
}
if contentType == "" {
contentType = "application/octet-stream"
}
headers := http.Header{"Content-Length": {strconv.FormatInt(size, 10)}, "Content-Type": {contentType}}
return s.signUpload(ctx, key, nil, headers, ttl)
}
func (s *Store) signUpload(ctx context.Context, key string, query url.Values, headers http.Header, ttl time.Duration) (storage.UploadRequest, error) {
expires := time.Now().UTC().Add(ttl)
u, err := s.client.PresignHeader(ctx, http.MethodPut, s.bucket, key, ttl, query, headers)
if err != nil {
return storage.UploadRequest{}, err
}
signed := ";" + u.Query().Get("X-Amz-SignedHeaders") + ";"
for name := range headers {
if !strings.Contains(signed, ";"+strings.ToLower(name)+";") {
return storage.UploadRequest{}, errors.New("storage/s3: upload constraints were not signed (SigV4 required)")
}
}
return storage.UploadRequest{Method: http.MethodPut, URL: u.String(), Headers: headers, ExpiresAt: expires}, nil
}
func (s *Store) CreateMultipart(ctx context.Context, key, contentType string) (storage.MultipartUpload, error) {
if err := storage.Check(ctx, key); err != nil {
return storage.MultipartUpload{}, err
}
if _, err := storage.MediaType(contentType); err != nil {
return storage.MultipartUpload{}, err
}
if contentType == "" {
contentType = "application/octet-stream"
}
id, err := (minio.Core{Client: s.client}).NewMultipartUpload(ctx, s.bucket, key, minio.PutObjectOptions{ContentType: contentType})
if err != nil {
return storage.MultipartUpload{}, err
}
return storage.MultipartUpload{Key: key, UploadID: id}, nil
}
func (s *Store) UploadPart(ctx context.Context, upload storage.MultipartUpload, number int, body io.Reader, size int64) (storage.Part, error) {
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
return storage.Part{}, err
}
reader, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, body, size, "")
if err != nil {
return storage.Part{}, err
}
part, err := (minio.Core{Client: s.client}).PutObjectPart(ctx, s.bucket, upload.Key, upload.UploadID, number, reader, size, minio.PutObjectPartOptions{})
if err != nil {
return storage.Part{}, err
}
return storage.Part{Number: number, ETag: part.ETag, Size: size}, nil
}
func (s *Store) PresignPart(ctx context.Context, upload storage.MultipartUpload, number int, size int64, ttl time.Duration) (storage.UploadRequest, error) {
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
return storage.UploadRequest{}, err
}
if err := storage.CheckUpload(ctx, upload.Key, size, s.maxBytes, "", ttl); err != nil {
return storage.UploadRequest{}, err
}
return s.signUpload(ctx, upload.Key, url.Values{"uploadId": {upload.UploadID}, "partNumber": {strconv.Itoa(number)}}, http.Header{"Content-Length": {strconv.FormatInt(size, 10)}, "Content-Type": {"application/octet-stream"}}, ttl)
}
func (s *Store) ListParts(ctx context.Context, upload storage.MultipartUpload, after, limit int) (storage.PartsPage, error) {
if err := storage.CheckPartsPage(ctx, upload, after, limit); err != nil {
return storage.PartsPage{}, err
}
r, err := (minio.Core{Client: s.client}).ListObjectParts(ctx, s.bucket, upload.Key, upload.UploadID, after, limit)
if err != nil {
return storage.PartsPage{}, err
}
page := storage.PartsPage{Parts: make([]storage.Part, 0, len(r.ObjectParts))}
for _, p := range r.ObjectParts {
page.Parts = append(page.Parts, storage.Part{Number: p.PartNumber, ETag: p.ETag, Size: p.Size})
}
if r.IsTruncated {
page.NextMarker = r.NextPartNumberMarker
}
return page, nil
}
func (s *Store) CompleteMultipart(ctx context.Context, upload storage.MultipartUpload, parts []storage.Part) (storage.Info, error) {
size, err := storage.VerifyParts(ctx, s, upload, parts, s.maxBytes)
if err != nil {
return storage.Info{}, err
}
completed := make([]minio.CompletePart, len(parts))
for i, p := range parts {
completed[i] = minio.CompletePart{PartNumber: p.Number, ETag: strings.Trim(p.ETag, "\"")}
}
_, err = (minio.Core{Client: s.client}).CompleteMultipartUpload(ctx, s.bucket, upload.Key, upload.UploadID, completed, minio.PutObjectOptions{})
if err != nil {
return storage.Info{}, err
}
return storage.Info{Key: upload.Key, Size: size, ModifiedAt: time.Now().UTC()}, nil
}
func (s *Store) AbortMultipart(ctx context.Context, upload storage.MultipartUpload) error {
if err := storage.CheckMultipart(ctx, upload); err != nil {
return err
}
err := (minio.Core{Client: s.client}).AbortMultipartUpload(ctx, s.bucket, upload.Key, upload.UploadID)
if minio.ToErrorResponse(err).Code == "NoSuchUpload" {
return nil
}
return err
}
var _ storage.UploadPresigner = (*Store)(nil)
var _ storage.MultipartStore = (*Store)(nil)
var _ storage.PartPresigner = (*Store)(nil)

34
storage/s3/upload_test.go Normal file
View File

@@ -0,0 +1,34 @@
package s3
import (
"context"
"net/http/httptest"
"strings"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/internal/storagefixture"
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
"github.com/minio/minio-go/v7/pkg/credentials"
)
func TestUploadProtocol(t *testing.T) {
server := httptest.NewTLSServer(storagefixture.MultipartHandler())
defer server.Close()
s, err := Open(Config{Endpoint: strings.TrimPrefix(server.URL, "https://"), Region: "us-east-1", Bucket: "testbucket", Secure: true, MaxBytes: 6 << 20, Credentials: credentials.NewStaticV4("fixtureAccess", "fixtureSecret", ""), Transport: server.Client().Transport})
if err != nil {
t.Fatal(err)
}
storagetest.RunUploads(t, s, server.Client())
}
func TestR2UploadSigning(t *testing.T) {
cfg, _ := R2Config(strings.Repeat("a", 32), "testbucket", credentials.NewStaticV4("fixtureAccess", "fixtureSecret", ""))
s, err := Open(cfg)
if err != nil {
t.Fatal(err)
}
r, err := s.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute)
if err != nil || !strings.Contains(r.URL, "/auto/s3/") && !strings.Contains(r.URL, "%2Fauto%2Fs3%2F") {
t.Fatal("R2 upload signing", err)
}
}

View File

@@ -14,10 +14,11 @@ import (
)
type Signer struct {
store Store
base *url.URL
secret []byte
now func() time.Time
store Store
base *url.URL
secret []byte
now func() time.Time
uploadPolicy *UploadPolicy
}
func NewSigner(store Store, baseURL string, secret []byte) (*Signer, error) {
@@ -51,6 +52,10 @@ func (s *Signer) PresignGet(ctx context.Context, key string, ttl time.Duration)
// ServeHTTP serves signed downloads as attachments. Issue links only after
// authorization; possession of an unexpired URL grants access to its object.
func (s *Signer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodPut && s.uploadPolicy != nil {
s.serveUpload(w, r)
return
}
if r.Method != "GET" && r.Method != "HEAD" {
w.Header().Set("Allow", "GET, HEAD")
http.Error(w, "method not allowed", 405)

98
storage/signedUpload.go Normal file
View File

@@ -0,0 +1,98 @@
package storage
import (
"context"
"crypto/hmac"
"crypto/sha256"
"encoding/base64"
"errors"
"io"
"net/http"
"net/url"
"strconv"
"time"
)
// NewUploadSigner extends the local download handler with signed PUT requests.
// No global CORS policy is installed; applications choose their allowed origins.
// URLs are reusable until expiry. Use random staging keys and authorize issuance.
func NewUploadSigner(store Store, baseURL string, secret []byte, policy UploadPolicy) (*Signer, error) {
if err := policy.Validate(); err != nil {
return nil, err
}
s, err := NewSigner(store, baseURL, secret)
if err != nil {
return nil, err
}
policy.ContentTypes = append([]string(nil), policy.ContentTypes...)
s.uploadPolicy = &policy
return s, nil
}
func (s *Signer) uploadSignature(key, expiry, size, contentType string) string {
mac := hmac.New(sha256.New, s.secret)
io.WriteString(mac, "PUT\n"+key+"\n"+expiry+"\n"+size+"\n"+contentType)
return base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
}
func (s *Signer) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (UploadRequest, error) {
if s.uploadPolicy == nil {
return UploadRequest{}, ErrUnsupported
}
if contentType == "" {
contentType = "application/octet-stream"
}
if err := CheckUpload(ctx, key, size, s.uploadPolicy.MaxBytes, contentType, ttl); err != nil {
return UploadRequest{}, err
}
if err := s.uploadPolicy.Check(size, contentType); err != nil {
return UploadRequest{}, err
}
expires := s.now().Add(ttl).Truncate(time.Second)
expiry, n := strconv.FormatInt(expires.Unix(), 10), strconv.FormatInt(size, 10)
u := *s.base
u.RawQuery = url.Values{"key": {key}, "expires": {expiry}, "size": {n}, "contentType": {contentType}, "signature": {s.uploadSignature(key, expiry, n, contentType)}}.Encode()
return UploadRequest{Method: http.MethodPut, URL: u.String(), Headers: http.Header{"Content-Length": {n}, "Content-Type": {contentType}}, ExpiresAt: expires}, nil
}
func (s *Signer) serveUpload(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-store")
q, err := url.ParseQuery(r.URL.RawQuery)
valid := err == nil && len(q) == 5
for _, key := range []string{"key", "expires", "size", "contentType", "signature"} {
valid = valid && len(q[key]) == 1
}
if !valid {
http.Error(w, "invalid link", 403)
return
}
key, expiry, n, contentType := q.Get("key"), q.Get("expires"), q.Get("size"), q.Get("contentType")
expires, err := strconv.ParseInt(expiry, 10, 64)
if err != nil || expires <= s.now().Unix() || expires > s.now().Unix()+3600 || Check(r.Context(), key) != nil || !hmac.Equal([]byte(q.Get("signature")), []byte(s.uploadSignature(key, expiry, n, contentType))) {
http.Error(w, "invalid or expired link", 403)
return
}
size, err := strconv.ParseInt(n, 10, 64)
if err != nil || s.uploadPolicy.Check(size, contentType) != nil || r.ContentLength != size || len(r.Header.Values("Content-Type")) != 1 || r.Header.Get("Content-Type") != contentType {
http.Error(w, "upload constraints mismatch", 400)
return
}
body := http.MaxBytesReader(w, r.Body, size)
defer body.Close()
uploader, _ := NewUploader(s.store, *s.uploadPolicy)
_, err = uploader.Put(r.Context(), key, body, size, contentType)
if err != nil {
status := 500
var limit *http.MaxBytesError
switch {
case errors.Is(err, ErrTooLarge), errors.As(err, &limit):
status = 413
case errors.Is(err, ErrContentType):
status = 415
case errors.Is(err, ErrSizeMismatch):
status = 400
}
http.Error(w, "upload rejected", status)
return
}
w.WriteHeader(http.StatusCreated)
}
var _ UploadPresigner = (*Signer)(nil)

View File

@@ -9,11 +9,13 @@ import (
"io"
"strings"
"testing"
"time"
)
func Run(t *testing.T, store storage.Store) {
t.Helper()
ctx := context.Background()
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
key := "contract/文件 +%.txt"
defer store.Delete(ctx, key)
if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) {
@@ -37,13 +39,19 @@ func Run(t *testing.T, store storage.Store) {
if _, err := store.Put(ctx, "tooBig", strings.NewReader(strings.Repeat("a", 65)), 65, "text/plain"); !errors.Is(err, storage.ErrTooLarge) {
t.Fatal(err)
}
for _, size := range []int64{0, 2, 4} {
if _, err := store.Put(ctx, "sizeMismatch", strings.NewReader("abc"), size, "text/plain"); err == nil {
t.Fatalf("declared size %d accepted for 3 bytes", size)
}
}
defer store.Delete(context.Background(), "sizeMismatch")
for _, key := range []string{"../bad", "a/../b", "/absolute", "a//b", "a\\b"} {
if _, err := store.Put(ctx, key, strings.NewReader("a"), 1, "text/plain"); err == nil {
t.Fatal(key)
}
}
canceled, cancel := context.WithCancel(ctx)
cancel()
canceled, cancelRequest := context.WithCancel(ctx)
cancelRequest()
if _, err := store.Get(canceled, key); !errors.Is(err, context.Canceled) {
t.Fatal(err)
}

View File

@@ -0,0 +1,127 @@
package storagetest
import (
"context"
"io"
"net/http"
"strconv"
"strings"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/storage"
)
type UploadStore interface {
storage.Store
storage.UploadPresigner
storage.MultipartStore
storage.PartPresigner
}
func sendUpload(t *testing.T, client *http.Client, link storage.UploadRequest, body string) *http.Response {
t.Helper()
req, err := http.NewRequest(link.Method, link.URL, strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
req.Header = link.Headers.Clone()
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
t.Fatalf("upload rejected: %d %s", resp.StatusCode, raw)
}
return resp
}
// RunUploads exercises both SDK upload and direct HTTP requests against a
// caller-owned server with maxBytes >=6 MiB. The caller cleans its bucket.
func RunUploads(t *testing.T, s UploadStore, client *http.Client) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
link, err := s.PresignPut(ctx, "direct", 5, "text/plain", time.Minute)
if err != nil {
t.Fatal(err)
}
if link.Method != "PUT" || link.Headers.Get("Content-Length") != "5" || link.Headers.Get("Content-Type") != "text/plain" {
t.Fatal("missing constraints", link.Headers)
}
resp := sendUpload(t, client, link, "hello")
resp.Body.Close()
defer s.Delete(context.Background(), "direct")
object, err := s.Get(ctx, "direct")
if err != nil {
t.Fatal(err)
}
raw, _ := io.ReadAll(object.Body)
object.Body.Close()
if string(raw) != "hello" {
t.Fatal("bad direct body")
}
if _, err = s.PresignPut(ctx, "direct", -1, "text/plain", time.Minute); err == nil {
t.Fatal("negative size accepted")
}
if _, err = s.PresignPut(ctx, "direct", 1, "text/plain", 2*time.Hour); err == nil {
t.Fatal("long TTL accepted")
}
upload, err := s.CreateMultipart(ctx, "multipart", "text/plain")
if err != nil {
t.Fatal(err)
}
defer s.AbortMultipart(context.Background(), upload)
data := strings.Repeat("a", 5<<20)
first, err := s.UploadPart(ctx, upload, 1, strings.NewReader(data), int64(len(data)))
if err != nil {
t.Fatal(err)
}
link, err = s.PresignPart(ctx, upload, 2, 5, time.Minute)
if err != nil {
t.Fatal(err)
}
resp = sendUpload(t, client, link, "world")
etag := resp.Header.Get("ETag")
resp.Body.Close()
if etag == "" {
t.Fatal("no part ETag")
}
page, err := s.ListParts(ctx, upload, 0, 1)
if err != nil || len(page.Parts) != 1 || page.NextMarker != 1 {
t.Fatal(page, err)
}
next, err := s.ListParts(ctx, upload, page.NextMarker, 1)
if err != nil || len(next.Parts) != 1 || next.Parts[0].Number != 2 || next.Parts[0].Size != 5 || next.NextMarker != 0 {
t.Fatal(next, err)
}
first.Size = 1 // The provider's size must prevail over caller metadata.
info, err := s.CompleteMultipart(ctx, upload, []storage.Part{first, {Number: 2, ETag: etag, Size: 1}})
if err != nil || info.Size != int64(len(data)+5) {
t.Fatal(info, err)
}
defer s.Delete(context.Background(), "multipart")
object, err = s.Get(ctx, "multipart")
if err != nil {
t.Fatal(err)
}
raw, err = io.ReadAll(object.Body)
object.Body.Close()
if err != nil || string(raw) != data+"world" {
t.Fatal("multipart bytes mismatch", len(raw), err)
}
canceled, err := s.CreateMultipart(ctx, "canceled", "text/plain")
if err != nil {
t.Fatal(err)
}
for i := 0; i < 2; i++ {
if err = s.AbortMultipart(ctx, canceled); err != nil {
t.Fatal("abort "+strconv.Itoa(i), err)
}
}
if _, err = s.ListParts(ctx, canceled, 0, 1); err == nil {
t.Fatal("aborted upload remains")
}
}

219
storage/upload.go Normal file
View File

@@ -0,0 +1,219 @@
package storage
import (
"bytes"
"context"
"errors"
"io"
"mime"
"net/http"
"strings"
"time"
)
var ErrUnsupported = errors.New("storage: capability not supported")
var ErrContentType = errors.New("storage: content type not allowed")
var ErrSizeMismatch = errors.New("storage: declared size mismatch")
// UploadRequest is a short-lived bearer capability; never log its URL. Headers
// must accompany the request. Browsers derive Content-Length from the Blob body.
type UploadRequest struct {
Method string `json:"method"`
URL string `json:"url"`
Headers http.Header `json:"headers"`
ExpiresAt time.Time `json:"expiresAt"`
}
type UploadPresigner interface {
PresignPut(context.Context, string, int64, string, time.Duration) (UploadRequest, error)
}
// UploadPolicy contains only transport-neutral constraints. ContentTypes is an
// exact MIME allowlist (parameters are ignored). Empty permits any valid type.
type UploadPolicy struct {
MaxBytes int64 `json:"maxBytes"`
ContentTypes []string `json:"contentTypes"`
}
func (p UploadPolicy) Validate() error {
if p.MaxBytes <= 0 || p.MaxBytes > 5<<40 {
return errors.New("storage: maxBytes must be 1..5 TiB")
}
for _, value := range p.ContentTypes {
t, err := MediaType(value)
if err != nil || strings.Contains(t, "*") {
return ErrContentType
}
}
return nil
}
func MediaType(value string) (string, error) {
if value == "" {
return "application/octet-stream", nil
}
if len(value) > 256 || strings.ContainsAny(value, "\r\n") {
return "", ErrContentType
}
t, _, err := mime.ParseMediaType(value)
if err != nil || !strings.Contains(t, "/") || strings.Contains(t, "*") {
return "", ErrContentType
}
return strings.ToLower(t), nil
}
func (p UploadPolicy) Check(size int64, contentType string) error {
if err := p.Validate(); err != nil {
return err
}
if size < 0 {
return ErrSizeMismatch
}
if size > p.MaxBytes {
return ErrTooLarge
}
t, err := MediaType(contentType)
if err != nil {
return err
}
if len(p.ContentTypes) == 0 {
return nil
}
for _, allowed := range p.ContentTypes {
a, _ := MediaType(allowed)
if a == t {
return nil
}
}
return ErrContentType
}
func CheckUpload(ctx context.Context, key string, size, maxBytes int64, contentType string, ttl time.Duration) error {
if err := Check(ctx, key); err != nil {
return err
}
if err := (UploadPolicy{MaxBytes: maxBytes}).Check(size, contentType); err != nil {
return err
}
if ttl < time.Second || ttl > time.Hour {
return errors.New("storage: TTL must be 1s..1h")
}
return nil
}
// PrepareUpload checks the first 512 bytes against the declared type when an
// allowlist is set, then streams the body with exact-length checking. It does
// not close the caller's reader. MIME sniffing is not antivirus/file validation.
func (p UploadPolicy) PrepareUpload(ctx context.Context, reader io.Reader, size int64, contentType string) (io.Reader, error) {
if ctx == nil || reader == nil {
return nil, errors.New("storage: context and reader required")
}
if err := ctx.Err(); err != nil {
return nil, err
}
if err := p.Check(size, contentType); err != nil {
return nil, err
}
var r io.Reader = ContextReader{Context: ctx, Reader: reader}
if len(p.ContentTypes) > 0 && size > 0 {
prefix := make([]byte, min(size, 512))
if _, err := io.ReadFull(r, prefix); err != nil {
return nil, errors.Join(ErrSizeMismatch, err)
}
detected, _ := MediaType(http.DetectContentType(prefix))
declared, _ := MediaType(contentType)
if detected != declared {
return nil, ErrContentType
}
r = io.MultiReader(bytes.NewReader(prefix), r)
}
if size == 0 {
var b [1]byte
n, err := io.ReadFull(r, b[:])
if n > 0 {
return nil, ErrSizeMismatch
}
if err != io.EOF {
return nil, err
}
return bytes.NewReader(nil), nil
}
return &exactReader{reader: r, remaining: size}, nil
}
type exactReader struct {
reader io.Reader
remaining int64
done bool
}
func (r *exactReader) Read(p []byte) (int, error) {
if len(p) == 0 {
return 0, nil
}
if r.done {
return 0, io.EOF
}
if int64(len(p)) > r.remaining {
p = p[:r.remaining]
}
n, err := r.reader.Read(p)
r.remaining -= int64(n)
if r.remaining == 0 {
r.done = true
// Probe before handing the last byte to a known-length SDK, which might
// otherwise stop reading without observing a trailing error.
var b [1]byte
extra, probeErr := io.ReadFull(r.reader, b[:])
if extra > 0 {
return max(0, n-1), ErrSizeMismatch
}
if probeErr != io.EOF {
return max(0, n-1), probeErr
}
if err != nil && err != io.EOF {
return max(0, n-1), err
}
return n, io.EOF
}
if err == io.EOF {
r.done = true
return n, ErrSizeMismatch
}
return n, err
}
// Uploader applies a policy without coupling the framework to business fields.
// Store remains unchanged; optional operations return ErrUnsupported when absent.
type Uploader struct {
store Store
policy UploadPolicy
}
func NewUploader(store Store, policy UploadPolicy) (*Uploader, error) {
if store == nil {
return nil, errors.New("storage: store required")
}
if err := policy.Validate(); err != nil {
return nil, err
}
policy.ContentTypes = append([]string(nil), policy.ContentTypes...)
return &Uploader{store: store, policy: policy}, nil
}
func (u *Uploader) Put(ctx context.Context, key string, body io.Reader, size int64, contentType string) (Info, error) {
if err := Check(ctx, key); err != nil {
return Info{}, err
}
reader, err := u.policy.PrepareUpload(ctx, body, size, contentType)
if err != nil {
return Info{}, err
}
return u.store.Put(ctx, key, reader, size, contentType)
}
func (u *Uploader) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (UploadRequest, error) {
if err := u.policy.Check(size, contentType); err != nil {
return UploadRequest{}, err
}
signer, ok := u.store.(UploadPresigner)
if !ok {
return UploadRequest{}, ErrUnsupported
}
return signer.PresignPut(ctx, key, size, contentType, ttl)
}

128
storage/upload_test.go Normal file
View File

@@ -0,0 +1,128 @@
package storage_test
import (
"bytes"
"context"
"errors"
"io"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/storage"
"git.yuntuo.hk/k2safe/go-omni/storage/local"
)
func TestUploadPolicyActualBytes(t *testing.T) {
ctx := context.Background()
files, _ := local.New(t.TempDir(), 1024)
uploader, err := storage.NewUploader(files, storage.UploadPolicy{MaxBytes: 64, ContentTypes: []string{"text/plain"}})
if err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
name, body, contentType string
size int64
want error
}{
{"valid", "hello", "text/plain; charset=utf-8", 5, nil},
{"tooLong", "hello!", "text/plain", 5, storage.ErrSizeMismatch},
{"tooShort", "hello", "text/plain", 6, storage.ErrSizeMismatch},
{"emptyMismatch", "x", "text/plain", 0, storage.ErrSizeMismatch},
{"overLimit", "hello", "text/plain", 65, storage.ErrTooLarge},
{"declaredType", "hello", "image/png", 5, storage.ErrContentType},
{"forgedType", "<html>hello</html>", "text/plain", 18, storage.ErrContentType},
} {
t.Run(tc.name, func(t *testing.T) {
_, err := uploader.Put(ctx, tc.name, strings.NewReader(tc.body), tc.size, tc.contentType)
if !errors.Is(err, tc.want) {
t.Fatalf("%v want %v", err, tc.want)
}
object, getErr := files.Get(ctx, tc.name)
if getErr == nil {
object.Body.Close()
}
if tc.want != nil && !errors.Is(getErr, storage.ErrNotFound) {
t.Fatal("rejected object committed", getErr)
}
})
}
canceled, cancel := context.WithCancel(ctx)
cancel()
if _, err := uploader.Put(canceled, "cancel", strings.NewReader("x"), 1, "text/plain"); !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
if _, err := uploader.CreateMultipart(ctx, "local", "text/plain"); !errors.Is(err, storage.ErrUnsupported) {
t.Fatal(err)
}
if _, err := uploader.PresignPut(ctx, "local", 1, "text/plain", time.Minute); !errors.Is(err, storage.ErrUnsupported) {
t.Fatal(err)
}
}
func TestSignedUploadConstraintsAndAtomicity(t *testing.T) {
files, _ := local.New(t.TempDir(), 1024)
signer, err := storage.NewUploadSigner(files, "https://files.example/upload", bytes.Repeat([]byte{1}, 32), storage.UploadPolicy{MaxBytes: 64, ContentTypes: []string{"text/plain"}})
if err != nil {
t.Fatal(err)
}
link, err := signer.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute)
if err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
name, body string
mutate func(*http.Request)
status int
}{
{"success", "hello", nil, 201},
{"key", "hello", func(r *http.Request) { q := r.URL.Query(); q.Set("key", "other"); r.URL.RawQuery = q.Encode() }, 403},
{"expiry", "hello", func(r *http.Request) { q := r.URL.Query(); q.Set("expires", "1"); r.URL.RawQuery = q.Encode() }, 403},
{"type", "hello", func(r *http.Request) { r.Header.Set("Content-Type", "image/png") }, 400},
{"length", "hello", func(r *http.Request) { r.ContentLength = 6 }, 400},
{"extraBytes", "longer", func(r *http.Request) { r.ContentLength = 5 }, 413},
{"shortBody", "no", func(r *http.Request) { r.ContentLength = 5 }, 400},
{"duplicateKey", "hello", func(r *http.Request) { q := r.URL.Query(); q.Add("key", "file"); r.URL.RawQuery = q.Encode() }, 403},
} {
t.Run(tc.name, func(t *testing.T) {
req := httptest.NewRequest("PUT", link.URL, strings.NewReader(tc.body))
req.Header = link.Headers.Clone()
if tc.mutate != nil {
tc.mutate(req)
}
w := httptest.NewRecorder()
signer.ServeHTTP(w, req)
if w.Code != tc.status {
t.Fatalf("status=%d want %d, %s", w.Code, tc.status, w.Body.String())
}
})
}
object, err := files.Get(context.Background(), "file")
if err != nil {
t.Fatal(err)
}
data, _ := io.ReadAll(object.Body)
object.Body.Close()
if string(data) != "hello" {
t.Fatal("failed upload overwrote good object")
}
download, _ := signer.PresignGet(context.Background(), "file", time.Minute)
req := httptest.NewRequest("PUT", download.String(), strings.NewReader("hello"))
req.Header = link.Headers.Clone()
w := httptest.NewRecorder()
signer.ServeHTTP(w, req)
if w.Code != 403 {
t.Fatal("GET signature reused for PUT")
}
// Original read-only constructor must never accept signed uploads.
readOnly, _ := storage.NewSigner(files, "https://files.example/upload", bytes.Repeat([]byte{1}, 32))
if _, err = readOnly.PresignPut(context.Background(), "file", 1, "text/plain", time.Minute); !errors.Is(err, storage.ErrUnsupported) {
t.Fatal(err)
}
u, _ := url.Parse(link.URL)
if u.Query().Get("signature") == "" {
t.Fatal("unsigned URL")
}
}