feat: complete API-driven projects with infrastructure and business acceptance
This commit is contained in:
115
storage/README.md
Normal file
115
storage/README.md
Normal file
@@ -0,0 +1,115 @@
|
||||
# 对象存储与上传
|
||||
|
||||
核心 `storage.Store` 保持 `Put/Get/Delete`,不会强迫每个适配器实现云厂商的所有能力。业务可按需依赖可选接口:
|
||||
|
||||
| 能力 | 接口 | Local | S3 / R2 | OSS |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 读写删除 | Store | 支持 | 支持 | 支持 |
|
||||
| 临时下载 | Presigner | NewSigner | 支持 | 支持 |
|
||||
| 限定大小/类型的直传 | UploadPresigner | NewUploadSigner | 支持 | 支持,需要配置签名 Header |
|
||||
| 创建、上传、查询、完成、取消分片 | MultipartStore | 当前不提供 | 支持 | 支持 |
|
||||
| 分片直传签名 | PartPresigner | 当前不提供 | 支持 | 支持,需要配置签名 Header |
|
||||
|
||||
没有固定业务字段、目录规则或身份模型。鉴权、对象归属、业务配额由应用定义。SDK 客户端和连接池由应用持有;适配器不会在一次调用后关闭客户端。
|
||||
|
||||
## 服务端上传限制
|
||||
|
||||
```go
|
||||
uploader, err := storage.NewUploader(files, storage.UploadPolicy{
|
||||
MaxBytes: 32 << 20,
|
||||
ContentTypes: []string{"image/png", "image/jpeg", "application/pdf"},
|
||||
})
|
||||
if err != nil { return err }
|
||||
info, err := uploader.Put(ctx, key, reader, size, contentType)
|
||||
```
|
||||
|
||||
`files` 是任意 Store。MaxBytes 必须为正数;ContentTypes 是精确 MIME 白名单,忽略参数,不接受通配符。空白名单允许所有合法声明类型。单次上传检查声明大小、流的实际长度;设置类型白名单时,再以头部最多 512 字节嗅探结果核对声明类型。适配器本身的大小限制仍然有效,建议与策略统一。这里只读取有限前缀,不把完整文件放进内存。
|
||||
|
||||
MIME 嗅探用于粗粒度识别,不验证完整文件结构或病毒;CSV/JSON 等类型可能被嗅探成 text/plain。需要复杂格式判断时,在业务校验流程中处理。必须传入可终止、正确返回 EOF 的 reader;ContextReader 不能中断任意用户 Reader 中已经阻塞的 Read。
|
||||
|
||||
使用 Local 时先写临时文件,校验失败不会替换旧对象。云端网络中断可能产生结果未知或遗留未完成上传;使用唯一暂存 key,并通过查询/对账确认后再更新业务记录,不要把上传 error 当成绝对未写入。
|
||||
|
||||
## 云端直传
|
||||
|
||||
```go
|
||||
// files 为 S3/R2 或 OSS 的 Store;Uploader 统一执行策略检查。
|
||||
uploader, err := storage.NewUploader(files, storage.UploadPolicy{
|
||||
MaxBytes: 32 << 20, ContentTypes: []string{"image/png", "image/jpeg"},
|
||||
})
|
||||
if err != nil { return err }
|
||||
// 业务先授权并生成暂存 key,随后签发。
|
||||
request, err := uploader.PresignPut(ctx, key, size, "image/png", 5*time.Minute)
|
||||
```
|
||||
|
||||
返回 `method/url/headers/expiresAt`,有效期 1 秒到 1 小时。签名绑定 key、方法、确切 Content-Length 和声明 Content-Type,缺少这些签名约束时适配器报错。SDK 凭据提前失效时,链接也可能提前失效。
|
||||
|
||||
浏览器请求使用原始 File/Blob 作为 body,不能额外包 FormData;Content-Length 由浏览器按 body 长度生成,禁止手动设置。其余 Headers 按返回值原样发送:
|
||||
|
||||
```js
|
||||
const headers = new Headers();
|
||||
for (const [name, values] of Object.entries(request.headers)) {
|
||||
if (name.toLowerCase() !== 'content-length') headers.set(name, values.join(', '));
|
||||
}
|
||||
const response = await fetch(request.url, {
|
||||
method: request.method, headers, body: file,
|
||||
});
|
||||
if (!response.ok) throw new Error(`upload failed: ${response.status}`);
|
||||
```
|
||||
|
||||
桶的 CORS 由部署方配置,允许业务域名、PUT 和返回的请求头;分片直传还需暴露 ETag 响应头。前端不能得到永久凭据。链接在到期前可能重复使用并覆盖同一个 key,不是一次性凭证。用随机暂存 key,业务确认前不要当作已发布文件。
|
||||
|
||||
**云端直传的类型限制针对声明 Header,无法由 API 服务实时检查文件字节。** 内容校验/扫描需要后处理;验证通过前不要公开文件。前端报告“上传完成”不等于文件可信。具体平台仍需实网验收,不能仅凭 S3 兼容声明判断一致。
|
||||
|
||||
OSS 使用官方 Go SDK v2:
|
||||
|
||||
```go
|
||||
client := ossSdk.NewClient(ossSdk.LoadDefaultConfig().
|
||||
WithRegion("cn-hangzhou").
|
||||
WithSignatureVersion(ossSdk.SignatureVersionV4).
|
||||
WithAdditionalHeaders([]string{"content-length"}).
|
||||
WithCredentialsProvider(credentialProvider))
|
||||
files, err := oss.New(client, bucket, 32<<20)
|
||||
```
|
||||
|
||||
适配器不修改传入的 client;缺少签名大小约束会返回明确错误。SDK 的 presign 实现与返回 SignedHeaders 都经过检查。
|
||||
|
||||
## 本地签名上传
|
||||
|
||||
```go
|
||||
files, err := local.New("./privateFiles", 32<<20)
|
||||
if err != nil { return err }
|
||||
signer, err := storage.NewUploadSigner(files, "https://files.example/upload", secret,
|
||||
storage.UploadPolicy{MaxBytes: 32<<20, ContentTypes: []string{"image/png"}})
|
||||
if err != nil { return err }
|
||||
// signer 实现 http.Handler,可挂到现有的 HTTP 服务;secret 至少 32 字节。
|
||||
request, err := signer.PresignPut(ctx, key, size, "image/png", time.Minute)
|
||||
```
|
||||
|
||||
Handler 检查签名、有效期、单值参数、Content-Length/Content-Type,以及实际内容大小和嗅探类型,成功返回 201。本地存储只开放一个受控入口,不把磁盘目录映射成公共目录。原 `NewSigner` 继续只服务下载。请求超时、CORS、并发限制和速率限制由承载该 Handler 的服务配置。
|
||||
|
||||
## 分片与续传
|
||||
|
||||
S3/R2/OSS 均实现 `MultipartStore` 和 `PartPresigner`。Uploader 也暴露相同操作;不支持的底层返回 `ErrUnsupported`。
|
||||
|
||||
```go
|
||||
session, err := uploader.CreateMultipart(ctx, key, "application/pdf")
|
||||
if err != nil { return err }
|
||||
// 业务持久保存 session,与已授权调用者关联,不能信任客户端任意传入的 key/uploadId。
|
||||
part, err := uploader.UploadPart(ctx, session, 1, reader, size)
|
||||
// 或签发单个分片的 PUT 请求:
|
||||
request, err := uploader.PresignPart(ctx, session, 1, size, 5*time.Minute)
|
||||
// 重连后分页读取实际已上传的分片,NextMarker == 0 表示结束。
|
||||
page, err := uploader.ListParts(ctx, session, 0, 1000)
|
||||
// 按 Number 升序从 1 连续提交,ETag 使用存储平台的返回值。
|
||||
info, err := uploader.CompleteMultipart(ctx, session, parts)
|
||||
// 用户取消/任务过期时,用仍有效的 context 显式清理。
|
||||
err = uploader.AbortMultipart(ctx, session)
|
||||
```
|
||||
|
||||
以上为互斥分支的 API 索引,实际调用需逐步处理 error。续传由业务保存 session,查询已有分片后重传缺失部分;框架不维护有身份语义的上传任务表,也不代替业务调度重试。
|
||||
|
||||
统一边界:每个对象最多 10,000 片、每片最多 5 GiB,除末片外至少 5 MiB;这是适配层为跨平台复用选取的共同约束。空文件走普通 Put。ListParts 每页最多 1,000 条。平台/桶限制可能更低。默认单对象仍为 32 MiB,需要大文件时显式提高适配器与 UploadPolicy 的限制;OSS 简单 Put 和云端单次签名 PUT 最大 5 GiB,更大使用分片。
|
||||
|
||||
Complete 会向平台分页核对全部分片的序号、ETag、真实大小和累计上限,前端填写的 `Part.Size` 不参与信任判断。完成时停止并发替换分片;Provider 仍会再次校验 ETag。返回 Info 的 Size 为已核对大小,ContentType 留空,完整对象元数据可通过 Get 获取。Multipart 的类型策略检查创建时的声明,实际内容应在暂存对象合并后校验。未完成的分片占用存储空间:应用需要超时取消或配置桶生命周期;URL 过期不会自动清理分片。
|
||||
|
||||
验收:Local 实际字节与原子性、S3/OSS SDK HTTP 协议测试、R2 签名配置测试、真实隔离 MinIO 验证。OSS/R2 尚未使用云账号实测。复现 MinIO 验证:先准备镜像,再运行 `python3 scripts/verifyUploads.py`;报告见 [uploadsLocal.json](../performance/uploadsLocal.json)。
|
||||
173
storage/multipart.go
Normal file
173
storage/multipart.go
Normal file
@@ -0,0 +1,173 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// MultipartUpload is provider state, not authorization. Applications must bind
|
||||
// it to the authenticated caller in their own durable session store.
|
||||
type MultipartUpload struct {
|
||||
Key string `json:"key"`
|
||||
UploadID string `json:"uploadId"`
|
||||
}
|
||||
type Part struct {
|
||||
Number int `json:"number"`
|
||||
ETag string `json:"etag"`
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
type PartsPage struct {
|
||||
Parts []Part `json:"parts"`
|
||||
NextMarker int `json:"nextMarker"`
|
||||
}
|
||||
type MultipartStore interface {
|
||||
CreateMultipart(context.Context, string, string) (MultipartUpload, error)
|
||||
UploadPart(context.Context, MultipartUpload, int, io.Reader, int64) (Part, error)
|
||||
ListParts(context.Context, MultipartUpload, int, int) (PartsPage, error)
|
||||
CompleteMultipart(context.Context, MultipartUpload, []Part) (Info, error)
|
||||
AbortMultipart(context.Context, MultipartUpload) error
|
||||
}
|
||||
type PartPresigner interface {
|
||||
PresignPart(context.Context, MultipartUpload, int, int64, time.Duration) (UploadRequest, error)
|
||||
}
|
||||
|
||||
func CheckMultipart(ctx context.Context, upload MultipartUpload) error {
|
||||
if err := Check(ctx, upload.Key); err != nil {
|
||||
return err
|
||||
}
|
||||
if upload.UploadID == "" || len(upload.UploadID) > 2048 || strings.IndexFunc(upload.UploadID, unicode.IsControl) >= 0 {
|
||||
return errors.New("storage: invalid upload ID")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func CheckPart(ctx context.Context, upload MultipartUpload, number int, size, maxBytes int64) error {
|
||||
if err := CheckMultipart(ctx, upload); err != nil {
|
||||
return err
|
||||
}
|
||||
if number < 1 || number > 10000 || size <= 0 {
|
||||
return errors.New("storage: part number must be 1..10000 and size positive")
|
||||
}
|
||||
if size > maxBytes || size > 5<<30 {
|
||||
return ErrTooLarge
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func CheckPartsPage(ctx context.Context, upload MultipartUpload, after, limit int) error {
|
||||
if err := CheckMultipart(ctx, upload); err != nil {
|
||||
return err
|
||||
}
|
||||
if after < 0 || after > 10000 || limit < 1 || limit > 1000 {
|
||||
return errors.New("storage: invalid part pagination")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// VerifyParts checks provider-reported sizes/ETags before completing. Caller
|
||||
// sizes are not trusted. Do not upload/replace parts concurrently with completion.
|
||||
// A common 5 MiB minimum (except last) keeps S3/OSS interchangeable.
|
||||
func VerifyParts(ctx context.Context, store MultipartStore, upload MultipartUpload, parts []Part, maxBytes int64) (int64, error) {
|
||||
if err := CheckMultipart(ctx, upload); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if len(parts) == 0 || len(parts) > 10000 || maxBytes <= 0 {
|
||||
return 0, errors.New("storage: invalid completion")
|
||||
}
|
||||
for i, p := range parts {
|
||||
if p.Number != i+1 || p.ETag == "" || len(p.ETag) > 256 || strings.ContainsAny(p.ETag, "\r\n") {
|
||||
return 0, errors.New("storage: ordered contiguous parts and ETags required")
|
||||
}
|
||||
}
|
||||
var total int64
|
||||
marker, seen := 0, 0
|
||||
for {
|
||||
page, err := store.ListParts(ctx, upload, marker, 1000)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if len(page.Parts) > 1000 {
|
||||
return 0, errors.New("storage: invalid provider page")
|
||||
}
|
||||
for _, actual := range page.Parts {
|
||||
if seen >= len(parts) || actual.Number != parts[seen].Number || strings.Trim(actual.ETag, "\"") != strings.Trim(parts[seen].ETag, "\"") || actual.Size <= 0 {
|
||||
return 0, errors.New("storage: parts changed or incomplete")
|
||||
}
|
||||
if actual.Size > 5<<30 || actual.Size > maxBytes-total {
|
||||
return 0, ErrTooLarge
|
||||
}
|
||||
if seen < len(parts)-1 && actual.Size < 5<<20 {
|
||||
return 0, errors.New("storage: non-final part smaller than 5 MiB")
|
||||
}
|
||||
total += actual.Size
|
||||
seen++
|
||||
}
|
||||
if page.NextMarker == 0 {
|
||||
break
|
||||
}
|
||||
if len(page.Parts) == 0 || page.NextMarker <= marker || page.NextMarker > 10000 || page.NextMarker != page.Parts[len(page.Parts)-1].Number {
|
||||
return 0, errors.New("storage: invalid provider pagination")
|
||||
}
|
||||
marker = page.NextMarker
|
||||
}
|
||||
if seen != len(parts) {
|
||||
return 0, errors.New("storage: missing parts")
|
||||
}
|
||||
return total, nil
|
||||
}
|
||||
func (u *Uploader) CreateMultipart(ctx context.Context, key, contentType string) (MultipartUpload, error) {
|
||||
if err := u.policy.Check(0, contentType); err != nil {
|
||||
return MultipartUpload{}, err
|
||||
}
|
||||
s, ok := u.store.(MultipartStore)
|
||||
if !ok {
|
||||
return MultipartUpload{}, ErrUnsupported
|
||||
}
|
||||
return s.CreateMultipart(ctx, key, contentType)
|
||||
}
|
||||
func (u *Uploader) UploadPart(ctx context.Context, upload MultipartUpload, number int, body io.Reader, size int64) (Part, error) {
|
||||
if err := CheckPart(ctx, upload, number, size, u.policy.MaxBytes); err != nil {
|
||||
return Part{}, err
|
||||
}
|
||||
s, ok := u.store.(MultipartStore)
|
||||
if !ok {
|
||||
return Part{}, ErrUnsupported
|
||||
}
|
||||
return s.UploadPart(ctx, upload, number, body, size)
|
||||
}
|
||||
func (u *Uploader) PresignPart(ctx context.Context, upload MultipartUpload, number int, size int64, ttl time.Duration) (UploadRequest, error) {
|
||||
if err := CheckPart(ctx, upload, number, size, u.policy.MaxBytes); err != nil {
|
||||
return UploadRequest{}, err
|
||||
}
|
||||
s, ok := u.store.(PartPresigner)
|
||||
if !ok {
|
||||
return UploadRequest{}, ErrUnsupported
|
||||
}
|
||||
return s.PresignPart(ctx, upload, number, size, ttl)
|
||||
}
|
||||
func (u *Uploader) ListParts(ctx context.Context, upload MultipartUpload, after, limit int) (PartsPage, error) {
|
||||
s, ok := u.store.(MultipartStore)
|
||||
if !ok {
|
||||
return PartsPage{}, ErrUnsupported
|
||||
}
|
||||
return s.ListParts(ctx, upload, after, limit)
|
||||
}
|
||||
func (u *Uploader) CompleteMultipart(ctx context.Context, upload MultipartUpload, parts []Part) (Info, error) {
|
||||
s, ok := u.store.(MultipartStore)
|
||||
if !ok {
|
||||
return Info{}, ErrUnsupported
|
||||
}
|
||||
if _, err := VerifyParts(ctx, s, upload, parts, u.policy.MaxBytes); err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
return s.CompleteMultipart(ctx, upload, parts)
|
||||
}
|
||||
func (u *Uploader) AbortMultipart(ctx context.Context, upload MultipartUpload) error {
|
||||
s, ok := u.store.(MultipartStore)
|
||||
if !ok {
|
||||
return ErrUnsupported
|
||||
}
|
||||
return s.AbortMultipart(ctx, upload)
|
||||
}
|
||||
60
storage/multipart_test.go
Normal file
60
storage/multipart_test.go
Normal file
@@ -0,0 +1,60 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type listedParts struct {
|
||||
MultipartStore
|
||||
parts []Part
|
||||
calls int
|
||||
loop bool
|
||||
}
|
||||
|
||||
func (s *listedParts) ListParts(ctx context.Context, u MultipartUpload, marker, limit int) (PartsPage, error) {
|
||||
s.calls++
|
||||
if s.loop {
|
||||
return PartsPage{Parts: []Part{{Number: 1, ETag: "a", Size: 5 << 20}}, NextMarker: marker}, nil
|
||||
}
|
||||
end := min(marker+limit, len(s.parts))
|
||||
next := 0
|
||||
if end < len(s.parts) {
|
||||
next = end
|
||||
}
|
||||
return PartsPage{Parts: s.parts[marker:end], NextMarker: next}, nil
|
||||
}
|
||||
func TestVerifyPartsUsesProviderSizes(t *testing.T) {
|
||||
upload := MultipartUpload{Key: "file", UploadID: "id"}
|
||||
first := Part{Number: 1, ETag: "a", Size: 5 << 20}
|
||||
last := Part{Number: 2, ETag: "b", Size: 3}
|
||||
supplied := []Part{{Number: 1, ETag: "a", Size: 1}, {Number: 2, ETag: "b", Size: 1}}
|
||||
s := &listedParts{parts: []Part{first, last}}
|
||||
n, err := VerifyParts(context.Background(), s, upload, supplied, 6<<20)
|
||||
if err != nil || n != (5<<20)+3 {
|
||||
t.Fatal(n, err)
|
||||
}
|
||||
if _, err = VerifyParts(context.Background(), s, upload, supplied, 5<<20); !errors.Is(err, ErrTooLarge) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, parts := range [][]Part{nil, {last, first}, {first, first}, {{Number: 1, ETag: "changed"}, last}, {first}} {
|
||||
if _, err := VerifyParts(context.Background(), s, upload, parts, 6<<20); err == nil {
|
||||
t.Fatal("bad completion accepted", parts)
|
||||
}
|
||||
}
|
||||
s.parts[0].Size = 1
|
||||
if _, err = VerifyParts(context.Background(), s, upload, supplied, 6<<20); err == nil {
|
||||
t.Fatal("small non-final part accepted")
|
||||
}
|
||||
}
|
||||
func TestVerifyPartsPagination(t *testing.T) {
|
||||
s := &listedParts{}
|
||||
for i := 1; i <= 1001; i++ {
|
||||
s.parts = append(s.parts, Part{Number: i, ETag: "etag", Size: 5 << 20})
|
||||
}
|
||||
n, err := VerifyParts(context.Background(), s, MultipartUpload{Key: "file", UploadID: "id"}, s.parts, 10<<30)
|
||||
if err != nil || s.calls != 2 || n != 1001*(5<<20) {
|
||||
t.Fatal(n, s.calls, err)
|
||||
}
|
||||
}
|
||||
@@ -19,10 +19,10 @@ type Store struct {
|
||||
maxBytes int64
|
||||
}
|
||||
|
||||
// New borrows a configured SDK client. This adapter uses simple uploads; large
|
||||
// multipart uploads are a separate provider capability, not buffered in memory.
|
||||
// New borrows a configured SDK client. Put uses simple uploads (up to 5 GiB).
|
||||
// Larger objects use the optional MultipartStore capability.
|
||||
func New(client *sdk.Client, bucket string, maxBytes int64) (*Store, error) {
|
||||
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<30 {
|
||||
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<40 {
|
||||
return nil, errors.New("storage/oss: invalid client, bucket or size limit")
|
||||
}
|
||||
if maxBytes == 0 {
|
||||
@@ -41,7 +41,14 @@ func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int6
|
||||
if size > s.maxBytes {
|
||||
return storage.Info{}, storage.ErrTooLarge
|
||||
}
|
||||
_, err := s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: storage.ContextReader{Context: ctx, Reader: reader}})
|
||||
if size > 5<<30 {
|
||||
return storage.Info{}, storage.ErrTooLarge
|
||||
}
|
||||
checked, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, reader, size, contentType)
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
_, err = s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: checked})
|
||||
if err != nil {
|
||||
return storage.Info{}, mapError(err)
|
||||
}
|
||||
|
||||
133
storage/oss/upload.go
Normal file
133
storage/oss/upload.go
Normal file
@@ -0,0 +1,133 @@
|
||||
package oss
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
|
||||
)
|
||||
|
||||
// PresignPut requires an OSS V4 client configured with
|
||||
// WithAdditionalHeaders([]string{"content-length"}). Missing signed constraints
|
||||
// fail closed; the borrowed client is never mutated by this adapter.
|
||||
func (s *Store) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (storage.UploadRequest, error) {
|
||||
if err := storage.CheckUpload(ctx, key, size, s.maxBytes, contentType, ttl); err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
if size > 5<<30 {
|
||||
return storage.UploadRequest{}, storage.ErrTooLarge
|
||||
}
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
r, err := s.client.Presign(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType)}, sdk.PresignExpires(ttl))
|
||||
if err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
return uploadRequest(r, size, contentType)
|
||||
}
|
||||
func uploadRequest(r *sdk.PresignResult, size int64, contentType string) (storage.UploadRequest, error) {
|
||||
headers := make(http.Header)
|
||||
for k, v := range r.SignedHeaders {
|
||||
headers.Set(k, v)
|
||||
}
|
||||
if headers.Get("Content-Length") != strconv.FormatInt(size, 10) || (contentType != "" && headers.Get("Content-Type") != contentType) {
|
||||
return storage.UploadRequest{}, errors.New("storage/oss: upload requires signed Content-Length and Content-Type; configure V4 and WithAdditionalHeaders(content-length)")
|
||||
}
|
||||
return storage.UploadRequest{Method: r.Method, URL: r.URL, Headers: headers, ExpiresAt: r.Expiration}, nil
|
||||
}
|
||||
func (s *Store) CreateMultipart(ctx context.Context, key, contentType string) (storage.MultipartUpload, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.MultipartUpload{}, err
|
||||
}
|
||||
if _, err := storage.MediaType(contentType); err != nil {
|
||||
return storage.MultipartUpload{}, err
|
||||
}
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
r, err := s.client.InitiateMultipartUpload(ctx, &sdk.InitiateMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentType: sdk.Ptr(contentType)})
|
||||
if err != nil {
|
||||
return storage.MultipartUpload{}, err
|
||||
}
|
||||
return storage.MultipartUpload{Key: key, UploadID: sdk.ToString(r.UploadId)}, nil
|
||||
}
|
||||
func (s *Store) UploadPart(ctx context.Context, upload storage.MultipartUpload, number int, body io.Reader, size int64) (storage.Part, error) {
|
||||
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
|
||||
return storage.Part{}, err
|
||||
}
|
||||
reader, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, body, size, "")
|
||||
if err != nil {
|
||||
return storage.Part{}, err
|
||||
}
|
||||
r, err := s.client.UploadPart(ctx, &sdk.UploadPartRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumber: int32(number), ContentLength: sdk.Ptr(size), Body: reader})
|
||||
if err != nil {
|
||||
return storage.Part{}, mapError(err)
|
||||
}
|
||||
return storage.Part{Number: number, ETag: sdk.ToString(r.ETag), Size: size}, nil
|
||||
}
|
||||
func (s *Store) PresignPart(ctx context.Context, upload storage.MultipartUpload, number int, size int64, ttl time.Duration) (storage.UploadRequest, error) {
|
||||
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
if err := storage.CheckUpload(ctx, upload.Key, size, s.maxBytes, "", ttl); err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
r, err := s.client.Presign(ctx, &sdk.UploadPartRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumber: int32(number), ContentLength: sdk.Ptr(size), RequestCommon: sdk.RequestCommon{Headers: map[string]string{"Content-Type": "application/octet-stream"}}}, sdk.PresignExpires(ttl))
|
||||
if err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
return uploadRequest(r, size, "application/octet-stream")
|
||||
}
|
||||
func (s *Store) ListParts(ctx context.Context, upload storage.MultipartUpload, after, limit int) (storage.PartsPage, error) {
|
||||
if err := storage.CheckPartsPage(ctx, upload, after, limit); err != nil {
|
||||
return storage.PartsPage{}, err
|
||||
}
|
||||
r, err := s.client.ListParts(ctx, &sdk.ListPartsRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumberMarker: int32(after), MaxParts: int32(limit)})
|
||||
if err != nil {
|
||||
return storage.PartsPage{}, err
|
||||
}
|
||||
page := storage.PartsPage{Parts: make([]storage.Part, 0, len(r.Parts))}
|
||||
for _, p := range r.Parts {
|
||||
page.Parts = append(page.Parts, storage.Part{Number: int(p.PartNumber), ETag: sdk.ToString(p.ETag), Size: p.Size})
|
||||
}
|
||||
if r.IsTruncated {
|
||||
page.NextMarker = int(r.NextPartNumberMarker)
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
func (s *Store) CompleteMultipart(ctx context.Context, upload storage.MultipartUpload, parts []storage.Part) (storage.Info, error) {
|
||||
size, err := storage.VerifyParts(ctx, s, upload, parts, s.maxBytes)
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
completed := make([]sdk.UploadPart, len(parts))
|
||||
for i, p := range parts {
|
||||
completed[i] = sdk.UploadPart{PartNumber: int32(p.Number), ETag: sdk.Ptr(p.ETag)}
|
||||
}
|
||||
_, err = s.client.CompleteMultipartUpload(ctx, &sdk.CompleteMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), CompleteMultipartUpload: &sdk.CompleteMultipartUpload{Parts: completed}})
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
return storage.Info{Key: upload.Key, Size: size, ModifiedAt: time.Now().UTC()}, nil
|
||||
}
|
||||
func (s *Store) AbortMultipart(ctx context.Context, upload storage.MultipartUpload) error {
|
||||
if err := storage.CheckMultipart(ctx, upload); err != nil {
|
||||
return err
|
||||
}
|
||||
_, err := s.client.AbortMultipartUpload(ctx, &sdk.AbortMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID)})
|
||||
var serviceError *sdk.ServiceError
|
||||
if errors.As(err, &serviceError) && serviceError.Code == "NoSuchUpload" {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
var _ storage.UploadPresigner = (*Store)(nil)
|
||||
var _ storage.MultipartStore = (*Store)(nil)
|
||||
var _ storage.PartPresigner = (*Store)(nil)
|
||||
29
storage/oss/upload_test.go
Normal file
29
storage/oss/upload_test.go
Normal file
@@ -0,0 +1,29 @@
|
||||
package oss
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/internal/storagefixture"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
|
||||
sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss"
|
||||
"github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials"
|
||||
)
|
||||
|
||||
func TestUploadProtocol(t *testing.T) {
|
||||
server := httptest.NewTLSServer(storagefixture.MultipartHandler())
|
||||
defer server.Close()
|
||||
cfg := sdk.LoadDefaultConfig().WithRegion("cn-hangzhou").WithEndpoint(server.URL).WithUsePathStyle(true).WithCredentialsProvider(credentials.NewStaticCredentialsProvider("fixtureAccess", "fixtureSecret")).WithHttpClient(server.Client()).WithSignatureVersion(sdk.SignatureVersionV4)
|
||||
missing, _ := New(sdk.NewClient(cfg), "testbucket", 6<<20)
|
||||
if _, err := missing.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute); err == nil {
|
||||
t.Fatal("unsigned length accepted")
|
||||
}
|
||||
cfg.WithAdditionalHeaders([]string{"content-length"})
|
||||
s, err := New(sdk.NewClient(cfg), "testbucket", 6<<20)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
storagetest.RunUploads(t, s, server.Client())
|
||||
}
|
||||
@@ -19,7 +19,7 @@ type Store struct {
|
||||
}
|
||||
|
||||
func New(client *minio.Client, bucket string, maxBytes int64) (*Store, error) {
|
||||
if client == nil || bucket == "" || maxBytes < 0 {
|
||||
if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<40 {
|
||||
return nil, errors.New("storage/s3: client, bucket and nonnegative limit required")
|
||||
}
|
||||
if maxBytes == 0 {
|
||||
@@ -37,7 +37,11 @@ func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int6
|
||||
if size > s.maxBytes {
|
||||
return storage.Info{}, storage.ErrTooLarge
|
||||
}
|
||||
result, err := s.client.PutObject(ctx, s.bucket, key, storage.ContextReader{Context: ctx, Reader: reader}, size, minio.PutObjectOptions{ContentType: contentType})
|
||||
checked, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, reader, size, contentType)
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
result, err := s.client.PutObject(ctx, s.bucket, key, checked, size, minio.PutObjectOptions{ContentType: contentType})
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
|
||||
@@ -73,6 +73,26 @@ func TestS3Integration(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer client.RemoveBucket(context.Background(), bucket)
|
||||
uploads, _ := New(client, bucket, 6<<20)
|
||||
httpClient := &http.Client{Timeout: 10 * time.Second}
|
||||
storagetest.RunUploads(t, uploads, httpClient)
|
||||
link, err := uploads.PresignPut(ctx, "constrained", 5, "text/plain", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct{ body, contentType string }{{"longer", "text/plain"}, {"hello", "image/png"}} {
|
||||
req, _ := http.NewRequestWithContext(ctx, link.Method, link.URL, strings.NewReader(tc.body))
|
||||
req.Header = link.Headers.Clone()
|
||||
req.Header.Set("Content-Type", tc.contentType)
|
||||
response, requestErr := httpClient.Do(req)
|
||||
if requestErr != nil {
|
||||
t.Fatal(requestErr)
|
||||
}
|
||||
response.Body.Close()
|
||||
if response.StatusCode != 403 {
|
||||
t.Fatalf("tampered upload constraints accepted: %d", response.StatusCode)
|
||||
}
|
||||
}
|
||||
s, _ := New(client, bucket, 64)
|
||||
storagetest.Run(t, s)
|
||||
s.Put(ctx, "signed", strings.NewReader("hello"), 5, "text/plain")
|
||||
|
||||
128
storage/s3/upload.go
Normal file
128
storage/s3/upload.go
Normal file
@@ -0,0 +1,128 @@
|
||||
package s3
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"github.com/minio/minio-go/v7"
|
||||
)
|
||||
|
||||
func (s *Store) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (storage.UploadRequest, error) {
|
||||
if err := storage.CheckUpload(ctx, key, size, s.maxBytes, contentType, ttl); err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
if size > 5<<30 {
|
||||
return storage.UploadRequest{}, storage.ErrTooLarge
|
||||
}
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
headers := http.Header{"Content-Length": {strconv.FormatInt(size, 10)}, "Content-Type": {contentType}}
|
||||
return s.signUpload(ctx, key, nil, headers, ttl)
|
||||
}
|
||||
func (s *Store) signUpload(ctx context.Context, key string, query url.Values, headers http.Header, ttl time.Duration) (storage.UploadRequest, error) {
|
||||
expires := time.Now().UTC().Add(ttl)
|
||||
u, err := s.client.PresignHeader(ctx, http.MethodPut, s.bucket, key, ttl, query, headers)
|
||||
if err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
signed := ";" + u.Query().Get("X-Amz-SignedHeaders") + ";"
|
||||
for name := range headers {
|
||||
if !strings.Contains(signed, ";"+strings.ToLower(name)+";") {
|
||||
return storage.UploadRequest{}, errors.New("storage/s3: upload constraints were not signed (SigV4 required)")
|
||||
}
|
||||
}
|
||||
return storage.UploadRequest{Method: http.MethodPut, URL: u.String(), Headers: headers, ExpiresAt: expires}, nil
|
||||
}
|
||||
func (s *Store) CreateMultipart(ctx context.Context, key, contentType string) (storage.MultipartUpload, error) {
|
||||
if err := storage.Check(ctx, key); err != nil {
|
||||
return storage.MultipartUpload{}, err
|
||||
}
|
||||
if _, err := storage.MediaType(contentType); err != nil {
|
||||
return storage.MultipartUpload{}, err
|
||||
}
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
id, err := (minio.Core{Client: s.client}).NewMultipartUpload(ctx, s.bucket, key, minio.PutObjectOptions{ContentType: contentType})
|
||||
if err != nil {
|
||||
return storage.MultipartUpload{}, err
|
||||
}
|
||||
return storage.MultipartUpload{Key: key, UploadID: id}, nil
|
||||
}
|
||||
func (s *Store) UploadPart(ctx context.Context, upload storage.MultipartUpload, number int, body io.Reader, size int64) (storage.Part, error) {
|
||||
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
|
||||
return storage.Part{}, err
|
||||
}
|
||||
reader, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, body, size, "")
|
||||
if err != nil {
|
||||
return storage.Part{}, err
|
||||
}
|
||||
part, err := (minio.Core{Client: s.client}).PutObjectPart(ctx, s.bucket, upload.Key, upload.UploadID, number, reader, size, minio.PutObjectPartOptions{})
|
||||
if err != nil {
|
||||
return storage.Part{}, err
|
||||
}
|
||||
return storage.Part{Number: number, ETag: part.ETag, Size: size}, nil
|
||||
}
|
||||
func (s *Store) PresignPart(ctx context.Context, upload storage.MultipartUpload, number int, size int64, ttl time.Duration) (storage.UploadRequest, error) {
|
||||
if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
if err := storage.CheckUpload(ctx, upload.Key, size, s.maxBytes, "", ttl); err != nil {
|
||||
return storage.UploadRequest{}, err
|
||||
}
|
||||
return s.signUpload(ctx, upload.Key, url.Values{"uploadId": {upload.UploadID}, "partNumber": {strconv.Itoa(number)}}, http.Header{"Content-Length": {strconv.FormatInt(size, 10)}, "Content-Type": {"application/octet-stream"}}, ttl)
|
||||
}
|
||||
func (s *Store) ListParts(ctx context.Context, upload storage.MultipartUpload, after, limit int) (storage.PartsPage, error) {
|
||||
if err := storage.CheckPartsPage(ctx, upload, after, limit); err != nil {
|
||||
return storage.PartsPage{}, err
|
||||
}
|
||||
r, err := (minio.Core{Client: s.client}).ListObjectParts(ctx, s.bucket, upload.Key, upload.UploadID, after, limit)
|
||||
if err != nil {
|
||||
return storage.PartsPage{}, err
|
||||
}
|
||||
page := storage.PartsPage{Parts: make([]storage.Part, 0, len(r.ObjectParts))}
|
||||
for _, p := range r.ObjectParts {
|
||||
page.Parts = append(page.Parts, storage.Part{Number: p.PartNumber, ETag: p.ETag, Size: p.Size})
|
||||
}
|
||||
if r.IsTruncated {
|
||||
page.NextMarker = r.NextPartNumberMarker
|
||||
}
|
||||
return page, nil
|
||||
}
|
||||
func (s *Store) CompleteMultipart(ctx context.Context, upload storage.MultipartUpload, parts []storage.Part) (storage.Info, error) {
|
||||
size, err := storage.VerifyParts(ctx, s, upload, parts, s.maxBytes)
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
completed := make([]minio.CompletePart, len(parts))
|
||||
for i, p := range parts {
|
||||
completed[i] = minio.CompletePart{PartNumber: p.Number, ETag: strings.Trim(p.ETag, "\"")}
|
||||
}
|
||||
_, err = (minio.Core{Client: s.client}).CompleteMultipartUpload(ctx, s.bucket, upload.Key, upload.UploadID, completed, minio.PutObjectOptions{})
|
||||
if err != nil {
|
||||
return storage.Info{}, err
|
||||
}
|
||||
return storage.Info{Key: upload.Key, Size: size, ModifiedAt: time.Now().UTC()}, nil
|
||||
}
|
||||
func (s *Store) AbortMultipart(ctx context.Context, upload storage.MultipartUpload) error {
|
||||
if err := storage.CheckMultipart(ctx, upload); err != nil {
|
||||
return err
|
||||
}
|
||||
err := (minio.Core{Client: s.client}).AbortMultipartUpload(ctx, s.bucket, upload.Key, upload.UploadID)
|
||||
if minio.ToErrorResponse(err).Code == "NoSuchUpload" {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
var _ storage.UploadPresigner = (*Store)(nil)
|
||||
var _ storage.MultipartStore = (*Store)(nil)
|
||||
var _ storage.PartPresigner = (*Store)(nil)
|
||||
34
storage/s3/upload_test.go
Normal file
34
storage/s3/upload_test.go
Normal file
@@ -0,0 +1,34 @@
|
||||
package s3
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/internal/storagefixture"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/storagetest"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
func TestUploadProtocol(t *testing.T) {
|
||||
server := httptest.NewTLSServer(storagefixture.MultipartHandler())
|
||||
defer server.Close()
|
||||
s, err := Open(Config{Endpoint: strings.TrimPrefix(server.URL, "https://"), Region: "us-east-1", Bucket: "testbucket", Secure: true, MaxBytes: 6 << 20, Credentials: credentials.NewStaticV4("fixtureAccess", "fixtureSecret", ""), Transport: server.Client().Transport})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
storagetest.RunUploads(t, s, server.Client())
|
||||
}
|
||||
func TestR2UploadSigning(t *testing.T) {
|
||||
cfg, _ := R2Config(strings.Repeat("a", 32), "testbucket", credentials.NewStaticV4("fixtureAccess", "fixtureSecret", ""))
|
||||
s, err := Open(cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r, err := s.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute)
|
||||
if err != nil || !strings.Contains(r.URL, "/auto/s3/") && !strings.Contains(r.URL, "%2Fauto%2Fs3%2F") {
|
||||
t.Fatal("R2 upload signing", err)
|
||||
}
|
||||
}
|
||||
@@ -14,10 +14,11 @@ import (
|
||||
)
|
||||
|
||||
type Signer struct {
|
||||
store Store
|
||||
base *url.URL
|
||||
secret []byte
|
||||
now func() time.Time
|
||||
store Store
|
||||
base *url.URL
|
||||
secret []byte
|
||||
now func() time.Time
|
||||
uploadPolicy *UploadPolicy
|
||||
}
|
||||
|
||||
func NewSigner(store Store, baseURL string, secret []byte) (*Signer, error) {
|
||||
@@ -51,6 +52,10 @@ func (s *Signer) PresignGet(ctx context.Context, key string, ttl time.Duration)
|
||||
// ServeHTTP serves signed downloads as attachments. Issue links only after
|
||||
// authorization; possession of an unexpired URL grants access to its object.
|
||||
func (s *Signer) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == http.MethodPut && s.uploadPolicy != nil {
|
||||
s.serveUpload(w, r)
|
||||
return
|
||||
}
|
||||
if r.Method != "GET" && r.Method != "HEAD" {
|
||||
w.Header().Set("Allow", "GET, HEAD")
|
||||
http.Error(w, "method not allowed", 405)
|
||||
|
||||
98
storage/signedUpload.go
Normal file
98
storage/signedUpload.go
Normal file
@@ -0,0 +1,98 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/hmac"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
// NewUploadSigner extends the local download handler with signed PUT requests.
|
||||
// No global CORS policy is installed; applications choose their allowed origins.
|
||||
// URLs are reusable until expiry. Use random staging keys and authorize issuance.
|
||||
func NewUploadSigner(store Store, baseURL string, secret []byte, policy UploadPolicy) (*Signer, error) {
|
||||
if err := policy.Validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
s, err := NewSigner(store, baseURL, secret)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
policy.ContentTypes = append([]string(nil), policy.ContentTypes...)
|
||||
s.uploadPolicy = &policy
|
||||
return s, nil
|
||||
}
|
||||
func (s *Signer) uploadSignature(key, expiry, size, contentType string) string {
|
||||
mac := hmac.New(sha256.New, s.secret)
|
||||
io.WriteString(mac, "PUT\n"+key+"\n"+expiry+"\n"+size+"\n"+contentType)
|
||||
return base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
|
||||
}
|
||||
func (s *Signer) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (UploadRequest, error) {
|
||||
if s.uploadPolicy == nil {
|
||||
return UploadRequest{}, ErrUnsupported
|
||||
}
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
if err := CheckUpload(ctx, key, size, s.uploadPolicy.MaxBytes, contentType, ttl); err != nil {
|
||||
return UploadRequest{}, err
|
||||
}
|
||||
if err := s.uploadPolicy.Check(size, contentType); err != nil {
|
||||
return UploadRequest{}, err
|
||||
}
|
||||
expires := s.now().Add(ttl).Truncate(time.Second)
|
||||
expiry, n := strconv.FormatInt(expires.Unix(), 10), strconv.FormatInt(size, 10)
|
||||
u := *s.base
|
||||
u.RawQuery = url.Values{"key": {key}, "expires": {expiry}, "size": {n}, "contentType": {contentType}, "signature": {s.uploadSignature(key, expiry, n, contentType)}}.Encode()
|
||||
return UploadRequest{Method: http.MethodPut, URL: u.String(), Headers: http.Header{"Content-Length": {n}, "Content-Type": {contentType}}, ExpiresAt: expires}, nil
|
||||
}
|
||||
func (s *Signer) serveUpload(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
q, err := url.ParseQuery(r.URL.RawQuery)
|
||||
valid := err == nil && len(q) == 5
|
||||
for _, key := range []string{"key", "expires", "size", "contentType", "signature"} {
|
||||
valid = valid && len(q[key]) == 1
|
||||
}
|
||||
if !valid {
|
||||
http.Error(w, "invalid link", 403)
|
||||
return
|
||||
}
|
||||
key, expiry, n, contentType := q.Get("key"), q.Get("expires"), q.Get("size"), q.Get("contentType")
|
||||
expires, err := strconv.ParseInt(expiry, 10, 64)
|
||||
if err != nil || expires <= s.now().Unix() || expires > s.now().Unix()+3600 || Check(r.Context(), key) != nil || !hmac.Equal([]byte(q.Get("signature")), []byte(s.uploadSignature(key, expiry, n, contentType))) {
|
||||
http.Error(w, "invalid or expired link", 403)
|
||||
return
|
||||
}
|
||||
size, err := strconv.ParseInt(n, 10, 64)
|
||||
if err != nil || s.uploadPolicy.Check(size, contentType) != nil || r.ContentLength != size || len(r.Header.Values("Content-Type")) != 1 || r.Header.Get("Content-Type") != contentType {
|
||||
http.Error(w, "upload constraints mismatch", 400)
|
||||
return
|
||||
}
|
||||
body := http.MaxBytesReader(w, r.Body, size)
|
||||
defer body.Close()
|
||||
uploader, _ := NewUploader(s.store, *s.uploadPolicy)
|
||||
_, err = uploader.Put(r.Context(), key, body, size, contentType)
|
||||
if err != nil {
|
||||
status := 500
|
||||
var limit *http.MaxBytesError
|
||||
switch {
|
||||
case errors.Is(err, ErrTooLarge), errors.As(err, &limit):
|
||||
status = 413
|
||||
case errors.Is(err, ErrContentType):
|
||||
status = 415
|
||||
case errors.Is(err, ErrSizeMismatch):
|
||||
status = 400
|
||||
}
|
||||
http.Error(w, "upload rejected", status)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusCreated)
|
||||
}
|
||||
|
||||
var _ UploadPresigner = (*Signer)(nil)
|
||||
@@ -9,11 +9,13 @@ import (
|
||||
"io"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func Run(t *testing.T, store storage.Store) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
key := "contract/文件 +%.txt"
|
||||
defer store.Delete(ctx, key)
|
||||
if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) {
|
||||
@@ -37,13 +39,19 @@ func Run(t *testing.T, store storage.Store) {
|
||||
if _, err := store.Put(ctx, "tooBig", strings.NewReader(strings.Repeat("a", 65)), 65, "text/plain"); !errors.Is(err, storage.ErrTooLarge) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, size := range []int64{0, 2, 4} {
|
||||
if _, err := store.Put(ctx, "sizeMismatch", strings.NewReader("abc"), size, "text/plain"); err == nil {
|
||||
t.Fatalf("declared size %d accepted for 3 bytes", size)
|
||||
}
|
||||
}
|
||||
defer store.Delete(context.Background(), "sizeMismatch")
|
||||
for _, key := range []string{"../bad", "a/../b", "/absolute", "a//b", "a\\b"} {
|
||||
if _, err := store.Put(ctx, key, strings.NewReader("a"), 1, "text/plain"); err == nil {
|
||||
t.Fatal(key)
|
||||
}
|
||||
}
|
||||
canceled, cancel := context.WithCancel(ctx)
|
||||
cancel()
|
||||
canceled, cancelRequest := context.WithCancel(ctx)
|
||||
cancelRequest()
|
||||
if _, err := store.Get(canceled, key); !errors.Is(err, context.Canceled) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
127
storage/storagetest/uploads.go
Normal file
127
storage/storagetest/uploads.go
Normal file
@@ -0,0 +1,127 @@
|
||||
package storagetest
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
)
|
||||
|
||||
type UploadStore interface {
|
||||
storage.Store
|
||||
storage.UploadPresigner
|
||||
storage.MultipartStore
|
||||
storage.PartPresigner
|
||||
}
|
||||
|
||||
func sendUpload(t *testing.T, client *http.Client, link storage.UploadRequest, body string) *http.Response {
|
||||
t.Helper()
|
||||
req, err := http.NewRequest(link.Method, link.URL, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req.Header = link.Headers.Clone()
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
|
||||
resp.Body.Close()
|
||||
t.Fatalf("upload rejected: %d %s", resp.StatusCode, raw)
|
||||
}
|
||||
return resp
|
||||
}
|
||||
|
||||
// RunUploads exercises both SDK upload and direct HTTP requests against a
|
||||
// caller-owned server with maxBytes >=6 MiB. The caller cleans its bucket.
|
||||
func RunUploads(t *testing.T, s UploadStore, client *http.Client) {
|
||||
t.Helper()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
link, err := s.PresignPut(ctx, "direct", 5, "text/plain", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if link.Method != "PUT" || link.Headers.Get("Content-Length") != "5" || link.Headers.Get("Content-Type") != "text/plain" {
|
||||
t.Fatal("missing constraints", link.Headers)
|
||||
}
|
||||
resp := sendUpload(t, client, link, "hello")
|
||||
resp.Body.Close()
|
||||
defer s.Delete(context.Background(), "direct")
|
||||
object, err := s.Get(ctx, "direct")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, _ := io.ReadAll(object.Body)
|
||||
object.Body.Close()
|
||||
if string(raw) != "hello" {
|
||||
t.Fatal("bad direct body")
|
||||
}
|
||||
if _, err = s.PresignPut(ctx, "direct", -1, "text/plain", time.Minute); err == nil {
|
||||
t.Fatal("negative size accepted")
|
||||
}
|
||||
if _, err = s.PresignPut(ctx, "direct", 1, "text/plain", 2*time.Hour); err == nil {
|
||||
t.Fatal("long TTL accepted")
|
||||
}
|
||||
upload, err := s.CreateMultipart(ctx, "multipart", "text/plain")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer s.AbortMultipart(context.Background(), upload)
|
||||
data := strings.Repeat("a", 5<<20)
|
||||
first, err := s.UploadPart(ctx, upload, 1, strings.NewReader(data), int64(len(data)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
link, err = s.PresignPart(ctx, upload, 2, 5, time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resp = sendUpload(t, client, link, "world")
|
||||
etag := resp.Header.Get("ETag")
|
||||
resp.Body.Close()
|
||||
if etag == "" {
|
||||
t.Fatal("no part ETag")
|
||||
}
|
||||
page, err := s.ListParts(ctx, upload, 0, 1)
|
||||
if err != nil || len(page.Parts) != 1 || page.NextMarker != 1 {
|
||||
t.Fatal(page, err)
|
||||
}
|
||||
next, err := s.ListParts(ctx, upload, page.NextMarker, 1)
|
||||
if err != nil || len(next.Parts) != 1 || next.Parts[0].Number != 2 || next.Parts[0].Size != 5 || next.NextMarker != 0 {
|
||||
t.Fatal(next, err)
|
||||
}
|
||||
first.Size = 1 // The provider's size must prevail over caller metadata.
|
||||
info, err := s.CompleteMultipart(ctx, upload, []storage.Part{first, {Number: 2, ETag: etag, Size: 1}})
|
||||
if err != nil || info.Size != int64(len(data)+5) {
|
||||
t.Fatal(info, err)
|
||||
}
|
||||
defer s.Delete(context.Background(), "multipart")
|
||||
object, err = s.Get(ctx, "multipart")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
raw, err = io.ReadAll(object.Body)
|
||||
object.Body.Close()
|
||||
if err != nil || string(raw) != data+"world" {
|
||||
t.Fatal("multipart bytes mismatch", len(raw), err)
|
||||
}
|
||||
canceled, err := s.CreateMultipart(ctx, "canceled", "text/plain")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
if err = s.AbortMultipart(ctx, canceled); err != nil {
|
||||
t.Fatal("abort "+strconv.Itoa(i), err)
|
||||
}
|
||||
}
|
||||
if _, err = s.ListParts(ctx, canceled, 0, 1); err == nil {
|
||||
t.Fatal("aborted upload remains")
|
||||
}
|
||||
}
|
||||
219
storage/upload.go
Normal file
219
storage/upload.go
Normal file
@@ -0,0 +1,219 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"mime"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
var ErrUnsupported = errors.New("storage: capability not supported")
|
||||
var ErrContentType = errors.New("storage: content type not allowed")
|
||||
var ErrSizeMismatch = errors.New("storage: declared size mismatch")
|
||||
|
||||
// UploadRequest is a short-lived bearer capability; never log its URL. Headers
|
||||
// must accompany the request. Browsers derive Content-Length from the Blob body.
|
||||
type UploadRequest struct {
|
||||
Method string `json:"method"`
|
||||
URL string `json:"url"`
|
||||
Headers http.Header `json:"headers"`
|
||||
ExpiresAt time.Time `json:"expiresAt"`
|
||||
}
|
||||
|
||||
type UploadPresigner interface {
|
||||
PresignPut(context.Context, string, int64, string, time.Duration) (UploadRequest, error)
|
||||
}
|
||||
|
||||
// UploadPolicy contains only transport-neutral constraints. ContentTypes is an
|
||||
// exact MIME allowlist (parameters are ignored). Empty permits any valid type.
|
||||
type UploadPolicy struct {
|
||||
MaxBytes int64 `json:"maxBytes"`
|
||||
ContentTypes []string `json:"contentTypes"`
|
||||
}
|
||||
|
||||
func (p UploadPolicy) Validate() error {
|
||||
if p.MaxBytes <= 0 || p.MaxBytes > 5<<40 {
|
||||
return errors.New("storage: maxBytes must be 1..5 TiB")
|
||||
}
|
||||
for _, value := range p.ContentTypes {
|
||||
t, err := MediaType(value)
|
||||
if err != nil || strings.Contains(t, "*") {
|
||||
return ErrContentType
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func MediaType(value string) (string, error) {
|
||||
if value == "" {
|
||||
return "application/octet-stream", nil
|
||||
}
|
||||
if len(value) > 256 || strings.ContainsAny(value, "\r\n") {
|
||||
return "", ErrContentType
|
||||
}
|
||||
t, _, err := mime.ParseMediaType(value)
|
||||
if err != nil || !strings.Contains(t, "/") || strings.Contains(t, "*") {
|
||||
return "", ErrContentType
|
||||
}
|
||||
return strings.ToLower(t), nil
|
||||
}
|
||||
func (p UploadPolicy) Check(size int64, contentType string) error {
|
||||
if err := p.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if size < 0 {
|
||||
return ErrSizeMismatch
|
||||
}
|
||||
if size > p.MaxBytes {
|
||||
return ErrTooLarge
|
||||
}
|
||||
t, err := MediaType(contentType)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(p.ContentTypes) == 0 {
|
||||
return nil
|
||||
}
|
||||
for _, allowed := range p.ContentTypes {
|
||||
a, _ := MediaType(allowed)
|
||||
if a == t {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
return ErrContentType
|
||||
}
|
||||
func CheckUpload(ctx context.Context, key string, size, maxBytes int64, contentType string, ttl time.Duration) error {
|
||||
if err := Check(ctx, key); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := (UploadPolicy{MaxBytes: maxBytes}).Check(size, contentType); err != nil {
|
||||
return err
|
||||
}
|
||||
if ttl < time.Second || ttl > time.Hour {
|
||||
return errors.New("storage: TTL must be 1s..1h")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PrepareUpload checks the first 512 bytes against the declared type when an
|
||||
// allowlist is set, then streams the body with exact-length checking. It does
|
||||
// not close the caller's reader. MIME sniffing is not antivirus/file validation.
|
||||
func (p UploadPolicy) PrepareUpload(ctx context.Context, reader io.Reader, size int64, contentType string) (io.Reader, error) {
|
||||
if ctx == nil || reader == nil {
|
||||
return nil, errors.New("storage: context and reader required")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := p.Check(size, contentType); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var r io.Reader = ContextReader{Context: ctx, Reader: reader}
|
||||
if len(p.ContentTypes) > 0 && size > 0 {
|
||||
prefix := make([]byte, min(size, 512))
|
||||
if _, err := io.ReadFull(r, prefix); err != nil {
|
||||
return nil, errors.Join(ErrSizeMismatch, err)
|
||||
}
|
||||
detected, _ := MediaType(http.DetectContentType(prefix))
|
||||
declared, _ := MediaType(contentType)
|
||||
if detected != declared {
|
||||
return nil, ErrContentType
|
||||
}
|
||||
r = io.MultiReader(bytes.NewReader(prefix), r)
|
||||
}
|
||||
if size == 0 {
|
||||
var b [1]byte
|
||||
n, err := io.ReadFull(r, b[:])
|
||||
if n > 0 {
|
||||
return nil, ErrSizeMismatch
|
||||
}
|
||||
if err != io.EOF {
|
||||
return nil, err
|
||||
}
|
||||
return bytes.NewReader(nil), nil
|
||||
}
|
||||
return &exactReader{reader: r, remaining: size}, nil
|
||||
}
|
||||
|
||||
type exactReader struct {
|
||||
reader io.Reader
|
||||
remaining int64
|
||||
done bool
|
||||
}
|
||||
|
||||
func (r *exactReader) Read(p []byte) (int, error) {
|
||||
if len(p) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if r.done {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if int64(len(p)) > r.remaining {
|
||||
p = p[:r.remaining]
|
||||
}
|
||||
n, err := r.reader.Read(p)
|
||||
r.remaining -= int64(n)
|
||||
if r.remaining == 0 {
|
||||
r.done = true
|
||||
// Probe before handing the last byte to a known-length SDK, which might
|
||||
// otherwise stop reading without observing a trailing error.
|
||||
var b [1]byte
|
||||
extra, probeErr := io.ReadFull(r.reader, b[:])
|
||||
if extra > 0 {
|
||||
return max(0, n-1), ErrSizeMismatch
|
||||
}
|
||||
if probeErr != io.EOF {
|
||||
return max(0, n-1), probeErr
|
||||
}
|
||||
if err != nil && err != io.EOF {
|
||||
return max(0, n-1), err
|
||||
}
|
||||
return n, io.EOF
|
||||
}
|
||||
if err == io.EOF {
|
||||
r.done = true
|
||||
return n, ErrSizeMismatch
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
// Uploader applies a policy without coupling the framework to business fields.
|
||||
// Store remains unchanged; optional operations return ErrUnsupported when absent.
|
||||
type Uploader struct {
|
||||
store Store
|
||||
policy UploadPolicy
|
||||
}
|
||||
|
||||
func NewUploader(store Store, policy UploadPolicy) (*Uploader, error) {
|
||||
if store == nil {
|
||||
return nil, errors.New("storage: store required")
|
||||
}
|
||||
if err := policy.Validate(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
policy.ContentTypes = append([]string(nil), policy.ContentTypes...)
|
||||
return &Uploader{store: store, policy: policy}, nil
|
||||
}
|
||||
func (u *Uploader) Put(ctx context.Context, key string, body io.Reader, size int64, contentType string) (Info, error) {
|
||||
if err := Check(ctx, key); err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
reader, err := u.policy.PrepareUpload(ctx, body, size, contentType)
|
||||
if err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
return u.store.Put(ctx, key, reader, size, contentType)
|
||||
}
|
||||
func (u *Uploader) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (UploadRequest, error) {
|
||||
if err := u.policy.Check(size, contentType); err != nil {
|
||||
return UploadRequest{}, err
|
||||
}
|
||||
signer, ok := u.store.(UploadPresigner)
|
||||
if !ok {
|
||||
return UploadRequest{}, ErrUnsupported
|
||||
}
|
||||
return signer.PresignPut(ctx, key, size, contentType, ttl)
|
||||
}
|
||||
128
storage/upload_test.go
Normal file
128
storage/upload_test.go
Normal file
@@ -0,0 +1,128 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage"
|
||||
"git.yuntuo.hk/k2safe/go-omni/storage/local"
|
||||
)
|
||||
|
||||
func TestUploadPolicyActualBytes(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
files, _ := local.New(t.TempDir(), 1024)
|
||||
uploader, err := storage.NewUploader(files, storage.UploadPolicy{MaxBytes: 64, ContentTypes: []string{"text/plain"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name, body, contentType string
|
||||
size int64
|
||||
want error
|
||||
}{
|
||||
{"valid", "hello", "text/plain; charset=utf-8", 5, nil},
|
||||
{"tooLong", "hello!", "text/plain", 5, storage.ErrSizeMismatch},
|
||||
{"tooShort", "hello", "text/plain", 6, storage.ErrSizeMismatch},
|
||||
{"emptyMismatch", "x", "text/plain", 0, storage.ErrSizeMismatch},
|
||||
{"overLimit", "hello", "text/plain", 65, storage.ErrTooLarge},
|
||||
{"declaredType", "hello", "image/png", 5, storage.ErrContentType},
|
||||
{"forgedType", "<html>hello</html>", "text/plain", 18, storage.ErrContentType},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
_, err := uploader.Put(ctx, tc.name, strings.NewReader(tc.body), tc.size, tc.contentType)
|
||||
if !errors.Is(err, tc.want) {
|
||||
t.Fatalf("%v want %v", err, tc.want)
|
||||
}
|
||||
object, getErr := files.Get(ctx, tc.name)
|
||||
if getErr == nil {
|
||||
object.Body.Close()
|
||||
}
|
||||
if tc.want != nil && !errors.Is(getErr, storage.ErrNotFound) {
|
||||
t.Fatal("rejected object committed", getErr)
|
||||
}
|
||||
})
|
||||
}
|
||||
canceled, cancel := context.WithCancel(ctx)
|
||||
cancel()
|
||||
if _, err := uploader.Put(canceled, "cancel", strings.NewReader("x"), 1, "text/plain"); !errors.Is(err, context.Canceled) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := uploader.CreateMultipart(ctx, "local", "text/plain"); !errors.Is(err, storage.ErrUnsupported) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := uploader.PresignPut(ctx, "local", 1, "text/plain", time.Minute); !errors.Is(err, storage.ErrUnsupported) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
func TestSignedUploadConstraintsAndAtomicity(t *testing.T) {
|
||||
files, _ := local.New(t.TempDir(), 1024)
|
||||
signer, err := storage.NewUploadSigner(files, "https://files.example/upload", bytes.Repeat([]byte{1}, 32), storage.UploadPolicy{MaxBytes: 64, ContentTypes: []string{"text/plain"}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
link, err := signer.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, tc := range []struct {
|
||||
name, body string
|
||||
mutate func(*http.Request)
|
||||
status int
|
||||
}{
|
||||
{"success", "hello", nil, 201},
|
||||
{"key", "hello", func(r *http.Request) { q := r.URL.Query(); q.Set("key", "other"); r.URL.RawQuery = q.Encode() }, 403},
|
||||
{"expiry", "hello", func(r *http.Request) { q := r.URL.Query(); q.Set("expires", "1"); r.URL.RawQuery = q.Encode() }, 403},
|
||||
{"type", "hello", func(r *http.Request) { r.Header.Set("Content-Type", "image/png") }, 400},
|
||||
{"length", "hello", func(r *http.Request) { r.ContentLength = 6 }, 400},
|
||||
{"extraBytes", "longer", func(r *http.Request) { r.ContentLength = 5 }, 413},
|
||||
{"shortBody", "no", func(r *http.Request) { r.ContentLength = 5 }, 400},
|
||||
{"duplicateKey", "hello", func(r *http.Request) { q := r.URL.Query(); q.Add("key", "file"); r.URL.RawQuery = q.Encode() }, 403},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
req := httptest.NewRequest("PUT", link.URL, strings.NewReader(tc.body))
|
||||
req.Header = link.Headers.Clone()
|
||||
if tc.mutate != nil {
|
||||
tc.mutate(req)
|
||||
}
|
||||
w := httptest.NewRecorder()
|
||||
signer.ServeHTTP(w, req)
|
||||
if w.Code != tc.status {
|
||||
t.Fatalf("status=%d want %d, %s", w.Code, tc.status, w.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
object, err := files.Get(context.Background(), "file")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
data, _ := io.ReadAll(object.Body)
|
||||
object.Body.Close()
|
||||
if string(data) != "hello" {
|
||||
t.Fatal("failed upload overwrote good object")
|
||||
}
|
||||
download, _ := signer.PresignGet(context.Background(), "file", time.Minute)
|
||||
req := httptest.NewRequest("PUT", download.String(), strings.NewReader("hello"))
|
||||
req.Header = link.Headers.Clone()
|
||||
w := httptest.NewRecorder()
|
||||
signer.ServeHTTP(w, req)
|
||||
if w.Code != 403 {
|
||||
t.Fatal("GET signature reused for PUT")
|
||||
}
|
||||
// Original read-only constructor must never accept signed uploads.
|
||||
readOnly, _ := storage.NewSigner(files, "https://files.example/upload", bytes.Repeat([]byte{1}, 32))
|
||||
if _, err = readOnly.PresignPut(context.Background(), "file", 1, "text/plain", time.Minute); !errors.Is(err, storage.ErrUnsupported) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
u, _ := url.Parse(link.URL)
|
||||
if u.Query().Get("signature") == "" {
|
||||
t.Fatal("unsigned URL")
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user