Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)

Swift

This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloud files from the Go Language

See here for package docs

http://godoc.org/github.com/ncw/swift/v2

Build Status GoDoc

Install

Use go to install the library

go get github.com/ncw/swift/v2

Usage

See here for full package docs

Here is a short example from the docs

import "github.com/ncw/swift/v2"

// Create a connection
c := swift.Connection{
UserName: "user",
ApiKey:   "key",
AuthUrl:  "auth_url",
Domain:   "domain", // Name of the domain (v3 auth only)
Tenant:   "tenant", // Name of the tenant (v2 auth only)
}
// Authenticate
err := c.Authenticate()
if err != nil {
panic(err)
}
// List all the containers
containers, err := c.ContainerNames(nil)
fmt.Println(containers)
// etc...

Migrating from v1

The library has current major version v2. If you want to migrate from the first version of library github.com/ncw/swift you have to explicitly add the /v2 suffix to the imports.

Most of the exported functions were added a new context.Context parameter in the v2, which you will have to provide when migrating.

Additions

The rs sub project contains a wrapper for the Rackspace specific CDN Management interface.

Testing

To run the tests you can either use an embedded fake Swift server either use a real Openstack Swift server or a Rackspace Cloud files account.

When using a real Swift server, you need to set these environment variables before running the tests

export SWIFT_API_USER='user'
export SWIFT_API_KEY='key'
export SWIFT_AUTH_URL='https://url.of.auth.server/v1.0'

And optionally these if using v2 authentication

export SWIFT_TENANT='TenantName'
export SWIFT_TENANT_ID='TenantId'

And optionally these if using v3 authentication

export SWIFT_TENANT='TenantName'
export SWIFT_TENANT_ID='TenantId'
export SWIFT_API_DOMAIN_ID='domain id'
export SWIFT_API_DOMAIN='domain name'

And optionally these if using v3 trust

export SWIFT_TRUST_ID='TrustId'

And optionally this if you want to skip server certificate validation

export SWIFT_AUTH_INSECURE=1

And optionally this to configure the connect channel timeout, in seconds

export SWIFT_CONNECTION_CHANNEL_TIMEOUT=60

And optionally this to configure the data channel timeout, in seconds

export SWIFT_DATA_CHANNEL_TIMEOUT=60

Then run the tests with go test

License

This is free software under the terms of MIT license (check COPYING file included in this package).

Contact and support

The project website is at:

There you can file bug reports, ask for help or contribute patches.

Authors

Contributors

Owner
Nick Craig-Wood
Open source. Rclone. Family. Sleep. Pick any two. Having fun with Go.
Nick Craig-Wood
Comments
  • Support for Bulk Delete

    Support for Bulk Delete

    • Connection.BulkDelete()

      deletes named objects at once using bulk delete API.

    • BulkDeleteResult

      stores results of BulkDelete operation.

    The referenced specification is Rackspace Cloud Files API: http://docs.rackspace.com/files/api/v1/cf-devguide/content/Bulk_Delete-d1e2338.html

    Tested on Rackspace.

    See #8 for discussions.

  • Go tests fail on v2.0 endpoint

    Go tests fail on v2.0 endpoint

    As requested I opened an issue because the Go tests fail on my OpenStack Mitaka backend with v2.0 authentication.

    https://github.com/restic/restic/issues/1039#issuecomment-309209227

    ~/src/github.com/ncw/swift$ source openrc.sh
    ~/src/github.com/ncw/swift$ export SWIFT_TENANT_ID=$OS_TENANT_ID
    ~/src/github.com/ncw/swift$ export SWIFT_API_KEY=$OS_PASSWORD
    ~/src/github.com/ncw/swift$ export SWIFT_AUTH_URL=$OS_AUTH_URL
    ~/src/github.com/ncw/swift$ export SWIFT_TENANT=$OS_TENANT
    ~/src/github.com/ncw/swift$ export SWIFT_API_USER=$OS_USERNAME
    ~/src/github.com/ncw/swift$ go version
    go version go1.8.3 linux/amd64
    ~/src/github.com/ncw/swift$ go test
    Waiting for server to start . Started
    --- FAIL: TestAuthenticateRace (1.10s)
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    	swift_test.go:456: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestSerializeConnectionJson (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestSerializeConnectionXml (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestOnReAuth (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestAccount (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestAccountUpdate (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerCreate (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainer (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainersAll (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainersAllWithLimit (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerUpdate (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerNames (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerNamesAll (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestContainerNamesAllWithLimit (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPutString (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPut (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectEmpty (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPutBytes (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectPutMimeType (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectCreate (0.13s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectGetString (0.09s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectGetBytes (0.10s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    --- FAIL: TestObjectOpen (0.11s)
    	swift_test.go:134: Auth failed Response didn't have storage url and auth token
    ^Csignal: interrupt
    FAIL	github.com/ncw/swift	4.409s
    
  • Rackspace v2 Auth giving 400 Bad Request on Authenticate

    Rackspace v2 Auth giving 400 Bad Request on Authenticate

    Having both authentication methods in one request causes a bad request response from Rackspace. After debugging with support, the tech instructed that the RAX-KSKEY method is the correct one and only one to use.

  • Retry when getting an EOF on HEAD/GET requests because of a Golang bug

    Retry when getting an EOF on HEAD/GET requests because of a Golang bug

    When running a 30 minutes testsuite (https://github.com/lebauce/distribution), I faced a weird issue : I sometimes got a url.Error : "Head https://my.swift.server/v1/AUTH_abcdb3abcda34a1c8d618164d99eb53e/test/myobject : EOF". I'm using the 1.4.2 version of Go.

    The issue seems to be in the net/http library and is described in both tickets : https://github.com/golang/go/issues/4677 https://github.com/golang/go/issues/8946

    A fix on net/http is under review : https://golang.org/cl/3210

    A fix for this problem is simply to send the request again when the request was a HEAD or a GET

  • Time difference between server and client

    Time difference between server and client

    Hi, I've noticed that when I have a wrong timezone on the client (e.g. when booted with live CD in a VM) I get this error

    "Response didn't have storage url and auth token"

    from https://github.com/ncw/swift/blob/v1/swift.go#L535, but that's not what actually happens there, I checked and compared the responses and if I correct the time everything works just fine, so at the very least the error message isn't accurate if that's intended behavior. Is it though? Is there a requirement that time is synced between client and server? Seems strange. Seems to me that the token or something gets expired right after it gets assigned an expiration date in the same function call.

  • add workarounds for servers that don't paginate correctly

    add workarounds for servers that don't paginate correctly

    Some Swift API implementations (I've observed this with Ceph RADOSGW) can return fewer results than specified by the "limit" parameter, even when we have not reached the end of the listing.

    It's unclear to me from reading the API docs whether this is a violation of the API specification, but since it happens in the wild, it's best to be able to handle it.

    One way of doing this is to simply keep fetching pages until we receive an empty page. Another is to assume that pages within a certain percentage of limit are not the last page. Given the tradeoffs involved, let's support both.

  • Please tag a release

    Please tag a release

    For restic, we'll switch to dep shortly, and it would be very helpful if you could tag a release. Thanks!

    https://dave.cheney.net/2016/06/24/gophers-please-tag-your-releases

  • Doing a reauth doesn't eat the first bytes in the body

    Doing a reauth doesn't eat the first bytes in the body

    This should hopefully fix #71. It uses Expect: 100-continue. The trick to get it working properly was to set TransferEncoding: []string{"chunked"} on the request object.

  • CDN Management

    CDN Management

    I've hacked your source to do CDN enabling/disabling of containers.

    Two relatively big changes were:

    • Changed storageOpts to the more generic requestOpts name.
    • Added a "manage" method similar to "storage", but they now wrap a new "call" method so the correct storage/management URL gets called.

    There are also tests in place which pass.

    Please get back to me if there is anything I need to change.

  • Add inspectable context parameter to API calls

    Add inspectable context parameter to API calls

    I am currently working on a product in which multiple goroutines are sharing the same swift client, and we need to be able to correlate the requests the client is making with the specific goroutine which initiated the action. To do this we pass down the request ID and make it accessible to a custom http.Transport for logging. We do this successfully with other storage clients. However, with this Swift client we've had to do some hacking. We append request ID to the Connection.UserAgent field, then extracting in our overriden http.Transport.RoundTrip method before dispatching the request.

    This seemed to us was a generic enough use-case to open an issue. We suggest Google's approach of adding a context parameter to each API function call, https://godoc.org/cloud.google.com/go/storage I understand this would mean updating the API functions to accept a context AND attach the context to the requests the client makes to allow a custom Transport to inspect the request's context. Not only would it support our use case, but make it far easier for your users to process custom data in their net/http interface implementations.

  • Allow using adminURL as storage endpoint URL

    Allow using adminURL as storage endpoint URL

    According to http://developer.openstack.org/api-ref-identity-v2.html#authenticate-v2.0 the endpoints response parameter is defined as:

    One or more endpoints objects. Each object shows the adminURL, region, internalURL, id, and publicURL for the endpoint.

    Reading the adminURL field and the ability to use it as an endpoint, was missing from this library.

    This PR changes the signature of Authenticator.StorageUrl to receive an EndpointType instead of a boolean. Also, I've added an EndpointType field to the Connection struct to allow users to specify a custom endpoint type.

    I tried to keep everything compatible so the Internal boolean field remains in the Connection struct, if the EndpointType is not set and the Internal flag is set, the chosen endpoint type will be the internal url otherwise the public url is used.

  • Allow extra headers of HEAD request for container/object requests.

    Allow extra headers of HEAD request for container/object requests.

    Hi team,

    I'd like to add headers support for HEAD requests that might help in some case like X-Newest: true (https://github.com/ncw/swift/issues/100) or some custom headers.

    Thanks!

  • Support non-standard time format

    Support non-standard time format

    We are working with a Swift backend server that returns last_modified formatted as 2022-08-18T04:02:34Z instead of 2022-08-18T04:02:34 . Because of that, the Objects() call returns the following error:

    parsing time "2022-08-18T04:02:34Z": extra text: "Z"
    

    As per the OpenStack Storage API reference,

    The date and time stamp format is ISO 8601:

    2022-08-18T04:02:34Z appears to be a valid ISO 8601 format according to wikipedia.

    ~~The easiest way to support it is probably to make time format a setting that can be changed by the user. This PR does that.~~

    This removes the Z suffix from object.ServerLastModified before using it.

    Please review.

    Thanks!

  • Implement federated (v3oidcaccesstoken) auth

    Implement federated (v3oidcaccesstoken) auth

    v3oidcaccesstoken requires two authentication requests:

    1. Retrieving an unscoped token using an oidc access token
    2. Retrieving a scoped token using the unscoped token (already implemented in v3auth)

    Change summary:

    • Extracted method for auth request handling
    • Added missing fields to Connection, e.g. IdentityProvider
      • Added mappings to the relevant openstack environment variables
    • Added interface TwoStageAuthenticator
    • Implemented PrelimRequest and PrelimResponse for v3auth

    Sadly I saw no reasonable way to test these changes, but I'm completely open to suggestions. This includes the way I extended the authentication handling, which may or may not be optimal.

    These changes are a follow up to this forum thread: https://forum.rclone.org/t/swift-env-auth-true-env-os-access-token-http-error-404-404-not-found/22520

  • No support for rate-limit errors with retry-after headers.

    No support for rate-limit errors with retry-after headers.

    When receiving errors for creating/updating objects/containers the headers are thrown away, this makes it impossible to respect retry-after headers which can be present in e.g. http 503 Service Unavailable or 429 Too many requests. (Unless Ḯm missing something obvious?) A possible non breaking solution would be to add an optional field "Headers" to swift.Error and append headers to this error. Another solution which would be breaking could be to simply return both error and headers from ObjectUpdate etc.

  • Document about authentication flow may be misleading?

    Document about authentication flow may be misleading?

    The Authentication method says:

    If you don't call it before calling one of the connection methods then it will be called for you on the first access.

    And the Call method says

    This will Authenticate if necessary, and re-authenticate if it receives a 401 error which means the token has expired

    But if user create the connection object and use Call method directly, program will receive SIGSEGV, because the conn.authLock member is not initialized yet. Other method like ObjectPut ObjectDelete also suffer from this problem.

    Although the sample code do call Authenticate after connection object construction. I thought the document may be a little misleading. :D

  • slo: no headers for segments

    slo: no headers for segments

    When creating a slo, no headers are written to the segment objects: headers, err := file.conn.ObjectPut(file.segmentContainer, segmentName, segmentReader, true, "", file.contentType, nil) https://github.com/ncw/swift/blob/master/largeobjects.go#L403

    In my case I want to pass a X-Delete-At parameter also to the segments that are created. Otherwise they will be orphaned once the actual file is expired and deleted.

    Can we pass in the headers from the file itself? headers, err := file.conn.ObjectPut(file.segmentContainer, segmentName, segmentReader, true, "", file.contentType, file.headers)

    thanks!

A simple image hosting script in Golang (Smart Storage with Telegram Cloud Storage)

image-upload-tg It's a simple image hosting script in Golang. It's have http server and image can be uploaded from here, also images store temporary i

Jan 19, 2022
Contentrouter - Protect static content via Firebase Hosting with Cloud Run and Google Cloud Storage

contentrouter A Cloud Run service to gate static content stored in Google Cloud

Jan 2, 2022
Container Storage Interface driver for Synology NAS

Synology CSI Driver for Kubernetes The official Container Storage Interface driver for Synology NAS. Container Images & Kubernetes Compatibility Drive

Jan 5, 2023
cloud-native local storage management system
cloud-native local storage management system

Open-Local是由多个组件构成的本地磁盘管理系统,目标是解决当前 Kubernetes 本地存储能力缺失问题。通过Open-Local,使用本地存储会像集中式存储一样简单。

Dec 30, 2022
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient
GoDrive: A cloud storage system similar to Dropbox or Google Drive, with resilient

Cloud Storage Service Author: Marisa Tania, Ryan Tjakrakartadinata Professor: Matthew Malensek See project spec here: https://www.cs.usfca.edu/~mmalen

Dec 7, 2021
Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang
Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang

Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang, i.e. Cloudpods is a cloud on clouds. Cloudpods is able to manage not only on-premise KVM/baremetals, but also resources from many cloud accounts across many cloud providers. It hides the differences of underlying cloud providers and exposes one set of APIs that allow programatically interacting with these many clouds.

Jan 11, 2022
The extensible SQL interface to your favorite cloud APIs.
The extensible SQL interface to your favorite cloud APIs.

The extensible SQL interface to your favorite cloud APIs.

Jan 4, 2023
Sample apps and code written for Google Cloud in the Go programming language.
Sample apps and code written for Google Cloud in the Go programming language.

Google Cloud Platform Go Samples This repository holds sample code written in Go that demonstrates the Google Cloud Platform. Some samples have accomp

Jan 9, 2023
Lightweight Cloud Instance Contextualizer
Lightweight Cloud Instance Contextualizer

Flamingo Flamingo is a lightweight contextualization tool that aims to handle initialization of cloud instances. It is meant to be a replacement for c

Jun 18, 2022
Terraform provider for HashiCorp Cloud Platform.

HashiCorp Cloud Platform (HCP) Terraform Provider Requirements Terraform >= 0.12.x Go >= 1.14 Building The Provider Clone the repository Enter the rep

Dec 25, 2022
The Cloud Posse Terraform Provider for various utilities (E.g. deep merging)
The Cloud Posse Terraform Provider for various utilities (E.g. deep merging)

terraform-provider-utils Terraform provider to add additional missing functionality to Terraform This project is part of our comprehensive "SweetOps"

Jan 7, 2023
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉

Infracost shows cloud cost estimates for Terraform projects. It helps developers, devops and others to quickly see the cost breakdown and compare different options upfront.

Jan 2, 2023
Cloud-native way to provide elastic Jupyter Notebook services on Kubernetes
Cloud-native way to provide elastic Jupyter Notebook services on Kubernetes

elastic-jupyter-operator: Elastic Jupyter on Kubernetes Kubernetes 原生的弹性 Jupyter 即服务 介绍 为用户按需提供弹性的 Jupyter Notebook 服务。elastic-jupyter-operator 提供以下特性

Dec 29, 2022
Google Cloud Client Libraries for Go.
Google Cloud Client Libraries for Go.

Google Cloud Client Libraries for Go.

Jan 8, 2023
A Cloud Native Buildpack for Go

The Go Paketo Buildpack provides a set of collaborating buildpacks that enable the building of a Go-based application.

Dec 14, 2022
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.

Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload. Run tools like masscan, puredns, ffuf, httpx or anything you need and get results quickly!

Jan 6, 2023
☁️🏃 Get up and running with Go on Google Cloud.

Get up and running with Go and gRPC on Google Cloud Platform, with this lightweight, opinionated, batteries-included service SDK.

Dec 20, 2022
Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.
Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Dec 30, 2022
Use Google Cloud KMS as an io.Reader and rand.Source.

Google Cloud KMS Go io.Reader and rand.Source This package provides a struct that implements Go's io.Reader and math/rand.Source interfaces, using Goo

Dec 1, 2022