Simple no frills AWS S3 Golang Library using REST with V4 Signing (without AWS Go SDK)

simples3 : Simple no frills AWS S3 Library using REST with V4 Signing

Overview GoDoc Go Report Card GoCover Zerodha Tech

SimpleS3 is a golang library for uploading and deleting objects on S3 buckets using REST API calls or Presigned URLs signed using AWS Signature Version 4.

Install

go get github.com/rhnvrm/simples3

Example

testTxt, _ := os.Open("testdata/test.txt")
defer testTxt.Close()

// Create an instance of the package
// You can either create by manually supplying credentials
// (preferably using Environment vars)
s3 := simples3.New(Region, AWSAccessKey, AWSSecretKey)
// or you can use this on an EC2 instance to 
// obtain credentials from IAM attached to the instance.
s3, _ := simples3.NewUsingIAM(Region)

// You can also set a custom endpoint to a compatible s3 instance. 
s3.SetEndpoint(CustomEndpoint)

// Note: Consider adding a testTxt.Seek(0, 0)
// in case you have read 
// the body, as the pointer is shared by the library.

// File Upload is as simple as providing the following
// details.
resp, err := s3.FileUpload(simples3.UploadInput{
    Bucket:      AWSBucket,
    ObjectKey:   "test.txt",
    ContentType: "text/plain",
    FileName:    "test.txt",
    Body:        testTxt,
})

// Similarly, Files can be deleted.
err := s3.FileDelete(simples3.DeleteInput{
    Bucket:    os.Getenv("AWS_S3_BUCKET"),
    ObjectKey: "test.txt",
})

// You can also download the file.
file, _ := s3.FileDownload(simples3.DownloadInput{
    Bucket:    AWSBucket,
    ObjectKey: "test.txt",
})
data, _ := ioutil.ReadAll(file)
file.Close()

// You can also use this library to generate
// Presigned URLs that can for eg. be used to
// GET/PUT files on S3 through the browser.
var time, _ = time.Parse(time.RFC1123, "Fri, 24 May 2013 00:00:00 GMT")

url := s.GeneratePresignedURL(PresignedInput{
    Bucket:        AWSBucket,
    ObjectKey:     "test.txt",
    Method:        "GET",
    Timestamp:     time,
    ExpirySeconds: 86400,
})

Contributing

You are more than welcome to contribute to this project. Fork and make a Pull Request, or create an Issue if you see any problem or want to propose a feature.

Author

Rohan Verma [email protected]

License

BSD-2-Clause-FreeBSD

Owner
Rohan Verma
Software Dev @zerodhatech | Intern @dell | Google Summer of Code'16 @apache | @ACM-SNU
Rohan Verma
Comments
  • Encoding issues for object keys with special characters

    Encoding issues for object keys with special characters

    Hey,

    so I stumbled across something odd. Uploading files with filenames containing special characters works flawlessly, but downloading or deleting them fails. In my particular case, my local Minio instance complains that the request signature does not match, but I think that is just a side effect.

    Consider this test example:

    package storage
    
    import (
    	"bytes"
    	"path"
    	"testing"
    
    	"github.com/rhnvrm/simples3"
    )
    
    func TestFileName(t *testing.T) {
    	inData := []byte("HelloWorld\nFooBar\nOneTwoThree\n")
    	testKey := "example?file%with$special&chars(1).txt"
    
    	s3 = simples3.New("S3Region", "S3AccessKey", "S3SecretKey")
    	s3.SetEndpoint("http://localhost:9000")
    
    	_, err := s3.FileUpload(simples3.UploadInput{
    		Bucket:      "simples3",
    		ObjectKey:   testKey,
    		ContentType: "text/plain",
    		FileName:    path.Base(testKey),
    		Body:        bytes.NewReader(inData),
    	})
    	if err != nil {
    		t.Fatalf("Failed to store file %s: %s", testKey, err)
    		return
    	}
    	_, err = s3.FileDownload(simples3.DownloadInput{
    		Bucket:    "simples3",
    		ObjectKey: testKey,
    	})
    	if err != nil {
    		t.Fatalf("Failed to get file %s: %s", testKey, err)
    	}
    
    	err = s3.FileDelete(simples3.DeleteInput{
    		Bucket:    "simples3",
    		ObjectKey: testKey,
    	})
    	if err != nil {
    		t.Fatalf("Failed to delete file %s: %s\n", testKey, err)
    	}
    }
    

    It fails with the following error (go test -run.test TestFileName simples3_test.go):

    --- FAIL: TestFileName (0.11s)
        simples3_test.go:35: Failed to get file example?file%with$special&chars(1).txt: status code: 403 Forbidden
    

    The Server logs the following:

    localhost [REQUEST s3.PostPolicyBucket] 12:11:08.211
    localhost POST /simples3
    localhost Host: localhost:9000
    localhost Accept-Encoding: gzip
    localhost Content-Length: 1938
    localhost Content-Type: multipart/form-data; boundary=8f92e8553bf46e645c1f13ef35cb60bc203127facf66cdbcf215f5b2f690
    localhost User-Agent: Go-http-client/1.1
    localhost <BODY>
    localhost [RESPONSE] [12:11:08.211] [ Duration 8.449ms  ↑ 2.0 KiB  ↓ 691 B ]
    localhost 201 Created
    localhost X-Amz-Request-Id: 16409DE3997F5B48
    localhost X-Xss-Protection: 1; mode=block
    localhost Accept-Ranges: bytes
    localhost ETag: "ea3b2e64587b58a724e2f5a15a45a1f7-1"
    localhost Content-Type: application/xml
    localhost Location: http://localhost:9000/simples3/example%3Ffile%25with$special&chars%281%29.txt
    localhost Server: MinIO/RELEASE.2020-02-20T22-51-23Z
    localhost Vary: Origin
    localhost Content-Length: 305
    localhost Content-Security-Policy: block-all-mixed-content
    localhost <BODY>
    localhost 
    localhost [REQUEST s3.GetObject] 12:11:08.212
    localhost GET /simples3/example?file%with$special&chars(1).txt
    localhost Host: localhost:9000
    localhost Authorization: REDACTED
    localhost Content-Length: 0
    localhost Date: 20201023T121108Z
    localhost User-Agent: Go-http-client/1.1
    localhost X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    localhost Accept-Encoding: gzip
    localhost <BODY>
    localhost [RESPONSE] [12:11:08.212] [ Duration 467µs  ↑ 87 B  ↓ 658 B ]
    localhost 403 Forbidden
    localhost X-Xss-Protection: 1; mode=block
    localhost Accept-Ranges: bytes
    localhost Content-Length: 401
    localhost Content-Security-Policy: block-all-mixed-content
    localhost Content-Type: application/xml
    localhost Server: MinIO/RELEASE.2020-02-20T22-51-23Z
    localhost Vary: Origin
    localhost X-Amz-Request-Id: 16409DE39A0FE29B
    localhost <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Key>example</Key><BucketName>simples3</BucketName><Resource>/simples3/example</Resource><RequestId>16409DE39A0FE29B</RequestId><HostId>4a57eb7d-dd77-496f-aee1-f604ca81f4f8</HostId></Error>
    localhost 
    

    On the server side, the file is correctly stored as example?file%with$special&chars(1).txt.


    So, my suspicion here is that URL-encoding is not necessary when POSTing the object (because the object key is in the body), but it is necessary for GET and DELETE requests, because the object key is part of the URL.

    What do you think about this? Should the Upload and Delete functions maybe by default URL-encode the object key?

  • Fix URL encoding issue for filenames with special characters

    Fix URL encoding issue for filenames with special characters

    Hey,

    as discussed in #7 , I have ported over the path encoding function from minio-go. Before that, I also added some tests to check the new & correct behavior. I have run these tests against AWS S3 buckets (and they were successful), I still want to try it out with a Minio instance (but I didn't get around to it yet). Once I'm done with that, I will squash the commits.

    Feedback welcome. :-)

  • Temporary session tokens don't seem to work

    Temporary session tokens don't seem to work

    I've the following tokens in my env:

    AWS_ACCESS_KEY_ID=REDACTED
    AWS_SECRET_ACCESS_KEY=REDACTED
    AWS_SESSION_TOKEN=REDACTED
    

    With this simple initializing code:

    	var (
    		accessKey, _ = os.LookupEnv("AWS_ACCESS_KEY_ID")
    		secretKey, _ = os.LookupEnv("AWS_SECRET_ACCESS_KEY")
    		region, _    = os.LookupEnv("AWS_DEFAULT_REGION")
    		token, _     = os.LookupEnv("AWS_SESSION_TOKEN")
    	)
    
    	// Set a default region.
    	if region == "" {
    		region = "ap-south-1"
    	}
    
    	// Lookup for env keys if they are present and initalise S3 based on those keys.
    	if accessKey != "" && secretKey != "" {
    		s3 := simples3.New(region, accessKey, secretKey)
    		s3.SetToken(token)
    		return s3
    	} else {
    		// Else check if IAM is accessible in this env.
    		s3, err := simples3.NewUsingIAM(region)
    		if err != nil {
    			fmt.Println("error initialising s3 client using IAM: %v", err)
    			os.Exit(1)
    		}
    		return s3
    	}
    

    I am trying to access a bucket and these credentials have access to them. With simpleS3 I am getting status code: 403 Forbidden on .FileDownload() method.

    However, same works when I do:

    aws s3 cp s3://my-bucket/my-key/data.json /tmp/data.json
    download: s3://my-bucket/my-key/data.json to ../../../../../../tmp/data.json
    

    I tried creating a temporary user with just access key/secret key and with same IAM policies attached and I was able to access it via simpleS3... which makes me wonder if this .Token is not being used properly.

  • Allow user to specify protocol for custom endpoint

    Allow user to specify protocol for custom endpoint

    First of all, thanks for this simple-to-use and dependency-free library!

    In local setups, such as a Minio [1] instance in a Docker container, the protocol may not always be HTTPS (like for public production endpoints), but instead just plain HTTP. To support this use-case, the user can specify a protocol at the beginning of the custom endpoint URI (either http:// or https://).

    To maintain backwards compatibility, the protocol defaults to HTTPS.

    The patch also includes a test case to assert this behavior.

    [1] https://hub.docker.com/r/minio/minio

  • detectFileSize does not measure size of the entire stream

    detectFileSize does not measure size of the entire stream

    Hello,

    When uploading small objects to my local Minio instance, I got the following error from s3.FileUpload()

    400 Bad Request: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>EntityTooSmall</Code><Message>Your proposed upload is smaller than the minimum allowed object size.</Message><BucketName>u9k-dev</BucketName><Resource>/u9k-dev</Resource><RequestId>16354459047EE460</RequestId><HostId>4a57eb7d-dd77-496f-aee1-f604ca81f4f8</HostId></Error>
    

    This was strange because I should also be able to upload small files. I figured out that it was happening after I added a content type detection function in my code, which does the following:

    // adapted from https://golangcode.com/get-the-content-type-of-file/
    func getFileContentType(r io.Reader) string {
    	// Only the first 512 bytes are used to sniff the content type.
    	buffer := make([]byte, 512)
    
    	_, err := r.Read(buffer)
    	if err != nil {
    		log.Printf("Failed to detect Content-Type: %s\n", err)
    		return "application/octet-stream"
    	}
    
    	// Use the net/http package's handy DectectContentType function. Always returns a valid
    	// content-type by returning "application/octet-stream" if no others seemed to match.
    	contentType := http.DetectContentType(buffer)
    	return contentType
    }
    

    Nothing complicated, just reads the first 512 bytes of the filestream and analyzes it.

    But this breaks s3.FileUpload, in particular the detectFileSize routine, because it does not count from the beginning. I believe this is a bug.

    	pos, err := body.Seek(0, 1) // this does not do anything, because it is seeking 0 bytes relative to the current position (1 = io.SeekCurrent)
    	if err != nil {
    		return -1, err
    	}
    	defer body.Seek(pos, 0)
    
    	n, err := body.Seek(0, 2)
    	if err != nil {
    		return -1, err
    	}
    	return n, nil
    

    See https://godoc.org/io#Seeker for reference.

    A version of the function that calculates the size of the entire stream would look like this:

    	pos, err := body.Seek(0, io.SeekStart)
    	if err != nil {
    		return -1, err
    	}
    	defer body.Seek(pos, 0)
    
    	n, err := body.Seek(0, io.SeekEnd)
    	if err != nil {
    		return -1, err
    	}
    	return n, nil
    

    Please let me know (and document) if this is intended behavior.

    https://github.com/rhnvrm/simples3/blob/59b9f0b2d2b30130019b42b49ee3108b1f9f4493/simples3.go#L191

  • feat: Adds support for custom metadata

    feat: Adds support for custom metadata

    assigning metadata for files will be beneficial to find files. ref: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/add-origin-custom-headers.html

  • Add Endpoint for some compatible instance and fix time format

    Add Endpoint for some compatible instance and fix time format

    • Fix: InvalidExpiration
    • Fix: RequestTimeTooSkewed
    • Add .env for test

    Compatible s3: https://developer.qiniu.com/kodo/manual/4088/s3-access-domainname

  • feat: add support for DigitalOcean

    feat: add support for DigitalOcean

    This commit adds a x-amz-date header which is an undocumented requirement of DO API requests.

    Ref https://github.com/knadh/listmonk/issues/715#issuecomment-1045119499

  • NewUsingIAM() blocks forever outside of AWS

    NewUsingIAM() blocks forever outside of AWS

    Accidentally initialized NewUsingIAM() outside of the AWS/IAM environment and the request blocked forever. Given that the instance metadata is generated from AWS "magic" URLs that are instant, I guess hardcoding a short $N second timeout here should be fine, given that:

    • NewUsingIAM() is currently unable to take any additional config.
    • Calling this outside of the AWS environment should return an error.

    https://github.com/rhnvrm/simples3/blob/ad0419ef77c905b3909459f5eaaa4cefe2232981/simples3.go#L151

  • Fix: Add X-Amz-Security-Token to presigned url

    Fix: Add X-Amz-Security-Token to presigned url

    Include the x-amz-security-token if signed with temporary credentials.

    Ref:https://stackoverflow.com/questions/49306621/how-secret-is-the-session-token-in-aws-temporary-security-credentials/49344291#49344291

  • B2 - invalid timestamp using S3 API

    B2 - invalid timestamp using S3 API

    Related to: https://github.com/rhnvrm/simples3/issues/17#issuecomment-1076838291

    I'm using listmonk with an S3-compatible B2 bucket. But I get the following error.

    2022/03/23 21:23:08error uploading file: status code: 400 : "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\n<Error>\n <Code>InvalidRequest</Code>\n <Message>Timestamp '20220323T212308Z' is invalid</Message>\n</Error>\n"
    

    I tested using an AWS S3 bucket and was able to successfully upload an image. I tried to see if maybe there was an issue with the datetime string formatting. But I'm starting to suspect there is something slightly different in Backblaze's API versus AWS S3.

    package main
    
    import (
      "fmt"
      "time"
    )
    
    func main() {
      amzDateISO8601TimeFormat := "20060102T150405Z"
      time := time.Now().UTC()
      amzdate := time.Format(amzDateISO8601TimeFormat)
      fmt.Println(time)
      fmt.Println(amzdate)
    }
    
A package for access aws service using AWS SDK for Golang

goaws ?? A package for access aws service using AWS SDK for Golang Advantage with goaws package Example for get user list IAM with AWS SDK for Golang

Nov 25, 2021
Simple CRUD API written in Go, built using AWS SAM tool and using the AWS' infrastructure.
Simple CRUD API written in Go, built using AWS SAM tool and using the AWS' infrastructure.

tutor-pet API Simple CRUD API written in Go, built using AWS SAM tool and using the AWS' infrastructure. Macro architecture: Code architecture: Pre-Re

Aug 17, 2022
Una prueba técnica: Servicio Golang REST API local, sobre Docker, gRPC, AWS Serverless y sobre Kubernetes en AWS EC2

Una prueba técnica: Servicio Golang REST API local, sobre Docker, gRPC, AWS Serverless y sobre Kubernetes en AWS EC2

May 7, 2022
AWS Mock with using GO SDK V2 middleware

AWS Mock with using GO SDK V2 middleware

Dec 17, 2021
Aws-cdk-go-examples - Example projects using the AWS CDK by Golang

aws-cdk-go-examples Example projects using the AWS CDK by Golang Useful commands

Nov 24, 2022
A go sdk for baidu netdisk open platform 百度网盘开放平台 Go SDK

Pan Go Sdk 该代码库为百度网盘开放平台Go语言的SDK

Nov 22, 2022
Nextengine-sdk-go: the NextEngine SDK for the Go programming language

NextEngine SDK for Go nextengine-sdk-go is the NextEngine SDK for the Go programming language. Getting Started Install go get github.com/takaaki-s/nex

Dec 7, 2021
Commercetools-go-sdk is fork of original commercetools-go-sdk

commercetools-go-sdk The Commercetools Go SDK is automatically generated based on the official API specifications of Commercetools. It should therefor

Dec 13, 2021
Sdk-go - Go version of the Synapse SDK

synapsesdk-go Synapse Protocol's Go SDK. Currently in super duper alpha, do not

Jan 7, 2022
Redash-go-sdk - An SDK for the programmatic management of Redash, in Go
Redash-go-sdk - An SDK for the programmatic management of Redash, in Go

Redash Go SDK An SDK for the programmatic management of Redash. The main compone

Dec 13, 2022
efsu is for accessing AWS EFS from your machine without a VPN

efsu: VPN-less access to AWS EFS efsu is for accessing AWS EFS from your machine without a VPN. It achieves this by deploying a Lambda function and sh

Mar 11, 2022
Feb 7, 2022
Unofficial Go SDK for GoPay Payments REST API

Unofficial Go SDK for GoPay Payments REST API Installation go get https://github.com/apparently-studio/gopay-go-api Basic usage client := gopay.NewCl

Nov 18, 2022
AWS SDK for the Go programming language.

AWS SDK for Go aws-sdk-go is the official AWS SDK for the Go programming language. Checkout our release notes for information about the latest bug fix

Dec 31, 2022
aws-sdk-go-v2 multipart upload poc

multipart-upload-example aws-sdk-go-v2 multipart upload poc There are 2 examples of uploading multipart with aws-sdk-go-v2: Manually, starting by call

Dec 3, 2021
Getting presigned urls for S3 with AWS SDK Go V2. Easy deploy with Velcel CLI.

S3-Presigned-Urls-Vercel-Serverless Setup yarn install && yarn setup Run(Local) yarn start You need to set environment variables with os.setenv for lo

Jan 14, 2022
Go package providing opinionated tools and methods for working with the `aws-sdk-go/service/cloudfront` package.

go-aws-cloudfront Go package providing opinionated tools and methods for working with the aws-sdk-go/service/cloudfront package. Documentation Tools $

Feb 2, 2022
"there" also called "GoThere" aims to be a simple Go Library to reduce redundant code for REST APIs.

there "there" also called "GoThere" aims to be a simple Go Library to reduce redundant code for REST APIs. Despite the existence of the other librarie

Dec 25, 2022