Go client library SDK for Ably realtime messaging service

Ably Go

A Go client library for www.ably.io, the realtime messaging service.

Installation

~ $ go get -u github.com/ably/ably-go/ably

Feature support

This library implements the Ably REST and Realtime client APIs.

REST API

In respect of the Ably REST API, this library targets the Ably 1.1 client library specification, with some omissions as follows (see the client library specification for specification references):

Feature
Push notifications admin API
JWT authentication

It is intended that this library is upgraded incrementally, with 1.1 feature support expanded in successive minor releases. If there are features that are currently missing that are a high priority for your use-case then please contact Ably customer support. Pull Requests are also welcomed.

Realtime API

In respect of the Realtime API, this is an early experimental implementation that targets the (now superseded) 0.8 library specification. This means that there are significant shortfalls in functionality; the principal issues are:

  • there is no channel suspended state; this means that the client will not automatically reattach to channels if a connection becomes suspended and then resumes, and presence members associated with the client will not be automatically re-entered;

  • transient realtime publishing is not supported, so a call to publish() on a realtime channel will trigger attachment of the channel;

  • inband reauthentication is not supported; expiring tokens will trigger a disconnection and resume of a realtime connection.

As with the REST API, it is intended that this library is upgraded incrementally and brought into line with the 1.1 specification. If there are features that are currently missing that are a high priority for your use-case then please contact Ably customer support. Pull Requests are also welcomed.

Using the Realtime API

Creating a client

client, err := ably.NewRealtimeClient(ably.NewClientOptions("xxx:xxx"))
if err != nil {
	panic(err)
}

channel := client.Channels.Get("test")

Subscribing to a channel for all events

sub, err := channel.Subscribe()
if err != nil {
	panic(err)
}

for msg := range sub.MessageChannel() {
	fmt.Println("Received message:", msg)
}

Subscribing to a channel for EventName1 and EventName2 events

sub, err := channel.Subscribe("EventName1", "EventName2")
if err != nil {
	panic(err)
}

for msg := range sub.MessageChannel() {
	fmt.Println("Received message:", msg)
}

Publishing to a channel

// send request to a server
res, err := channel.Publish("EventName1", "EventData1")
if err != nil {
	panic(err)
}

// await confirmation
if err = res.Wait(); err != nil {
	panic(err)
}

Announcing presence on a channel

// send request to a server
res, err := channel.Presence.Enter("presence data")
if err != nil {
	panic(err)
}

// await confirmation
if err = res.Wait(); err != nil {
	panic(err)
}

Announcing presence on a channel on behalf of other client

// send request to a server
res, err := channel.Presence.EnterClient("clientID", "presence data")
if err != nil {
	panic(err)
}

// await confirmation
if err = res.Wait(); err != nil {
	panic(err)
}

Getting all clients present on a channel

clients, err := channel.Presence.Get(true)
if err != nil {
	panic(err)
}

for _, client := range clients {
	fmt.Println("Present client:", client)
}

Subscribing to all presence messages

sub, err := channel.Presence.Subscribe()
if err != nil {
	panic(err)
}

for msg := range sub.PresenceChannel() {
	fmt.Println("Presence event:", msg)
}

Subscribing to 'Enter' presence messages only

sub, err := channel.Presence.Subscribe(proto.PresenceEnter)
if err != nil {
	panic(err)
}

for msg := range sub.PresenceChannel() {
	fmt.Println("Presence event:", msg)
}

Using the REST API

Introduction

All examples assume a client and/or channel has been created as follows:

client, err := ably.NewRestClient(ably.NewClientOptions("xxx:xxx"))
if err != nil {
	panic(err)
}

channel := client.Channels.Get("test", nil)

Publishing a message to a channel

err = channel.Publish("HelloEvent", "Hello!")
if err != nil {
	panic(err)
}

Querying the History

page, err := channel.History(nil)
for ; err == nil; page, err = page.Next() {
	for _, message := range page.Messages() {
		fmt.Println(message)
	}
}
if err != nil {
	panic(err)
}

Presence on a channel

page, err := channel.Presence.Get(nil)
for ; err == nil; page, err = page.Next() {
	for _, presence := range page.PresenceMessages() {
		fmt.Println(presence)
	}
}
if err != nil {
	panic(err)
}

Querying the Presence History

page, err := channel.Presence.History(nil)
for ; err == nil; page, err = page.Next() {
	for _, presence := range page.PresenceMessages() {
		fmt.Println(presence)
	}
}
if err != nil {
	panic(err)
}

Generate Token and Token Request

client.Auth.RequestToken()
client.Auth.CreateTokenRequest()

Fetching your application's stats

page, err := client.Stats(&ably.PaginateParams{})
for ; err == nil; page, err = page.Next() {
	for _, stat := range page.Stats() {
		fmt.Println(stat)
	}
}
if err != nil {
	panic(err)
}

Known limitations (work in progress)

As the library is actively developed couple of features are not there yet:

  • Realtime connection recovery is not implemented
  • Realtime connection failure handling is not implemented
  • ChannelsOptions and CipherParams are not supported when creating a Channel
  • Realtime Ping function is not implemented

Release process

This library uses semantic versioning. For each release, the following needs to be done:

  • Create a branch for the release, named like release/1.1.6
  • Replace all references of the current version number with the new version number and commit the changes
  • Run github_changelog_generator to automate the update of the CHANGELOG. This may require some manual intervention, both in terms of how the command is run and how the change log file is modified. Your mileage may vary:
    • The command you will need to run will look something like this: github_changelog_generator -u ably -p ably-go --since-tag v1.1.4 --output delta.md
    • Using the command above, --output delta.md writes changes made after --since-tag to a new file
    • The contents of that new file (delta.md) then need to be manually inserted at the top of the CHANGELOG.md, changing the "Unreleased" heading and linking with the current version numbers
    • Also ensure that the "Full Changelog" link points to the new version tag instead of the HEAD
    • Commit this change: git add CHANGELOG.md && git commit -m "Update change log."
  • Commit CHANGELOG
  • Make a PR against main
  • Once the PR is approved, merge it into main
  • Add a tag to the new main head commit and push to origin such as git tag v1.1.6 && git push origin v1.1.6

Support and feedback

Please visit http://support.ably.io/ for access to our knowledgebase and to ask for any assistance.

You can also view the community reported Github issues.

Contributing

Because this package uses internal packages, all fork development has to happen under $GOPATH/src/github.com/ably/ably-go to prevent use of internal package not allowed errors.

  1. Fork github.com/ably/ably-go
  2. go to the ably-go directory: cd $GOPATH/src/github.com/ably/ably-go
  3. add your fork as a remote: git remote add fork [email protected]:your-username/ably-go
  4. create your feature branch: git checkout -b my-new-feature
  5. commit your changes (git commit -am 'Add some feature')
  6. ensure you have added suitable tests and the test suite is passing for both JSON and MessagePack protocols by running scripts/test.sh.
  7. push to the branch: git push fork my-new-feature
  8. create a new Pull Request
Owner
Ably Realtime - our client library SDKs and libraries
Ably is a serverless infrastructure provider for the realtime Internet. We solve the complex problems to do with maintaining a global data stream network.
Ably Realtime - our client library SDKs and libraries
Comments
  • Replace obsolete websocket library -- allows WebAssembly

    Replace obsolete websocket library -- allows WebAssembly

    This replaces the unmaintained golang.org/x/net/websocket websocket library with nhooyr.io/websocket. This allows compiling go-ably to WebAssembly and using Ably from a browser.

    resolves #368

  • Add connection key to messages.

    Add connection key to messages.

    This fixes #442 a customer previously raised a PR to add ConnectionKey to the message. However the value of this new connection key field was never populated, so it still was not possible for a rest client to publish on behalf of a realtime client. This PR populates the connection key when rest messages are published.

  • Bugfix: REST publish encoding was

    Bugfix: REST publish encoding was "utf8" should be "utf-8"

    There was an error which prevented me from running tests and vetting the application. The package name "internal" is not valid and Go v1.8 complains. I've moved the package up 1 directory so instead of /ably/internal/ablyutil it's /ably/ablyutil.

    I've also fixed the encoding which is being set when publishing a REST message. The clients and everywhere else in the library use "utf-8" as the encoding, but this was using "utf8" which was causing an encoding error on the client when trying to decode the message.

  • Structure change

    Structure change

    @lmars Would you mind taking a look into this?

    To keep things more or less consistent with other libraries, I had to find a way to have an equivalent of Ably::Rest::Client.new() or new Ably.Rest.Client(). So I created a struct in the ably namespace for Rest which doesn't do anything special apart from being a fake sub-package and allow me to call ably.Rest.NewClient().

    We could accept that people will call rest.Client like we did in most tests but I don't think that it conveys the right meaning (it's an ably client, not just an HTTP client). Unless you think we should advise to rename package on import (import ably "github.com/ably/ably-go"), but again I don't think it's conventional.

    Thoughts?

  • v1.2 API design

    v1.2 API design

    This issue tracks and discuss the changes to be made to the public API in order to make it 1.2 spec-compliant.

    Subscriptions: from channels to callbacks

    For some features, we make a long-lived subscription to a series of events that the caller receives. The spec says those events are delivered by calling a callback. In Go, instead, we were sending them to channels, from which user code receive events at their own. For 1.2 we're taking callbacks. See the relevant discussion.

    So we go from something like:

    ch := make(chan ably.ChannelStateChange)
    channel.On(ably.ChannelStateChange, ably.ConnectionEventAttached)
    for change := range ch {
        // ...
    }
    

    To:

    channel.On(ably.ConnectionEventAttached, func(change ably.ChannelStateChange) {
        // ...
    })
    

    The list of such features is:

    • [x] Connection events (#144)
    • [x] Channel events (#190)
    • [x] Messages (#202)
    • [x] Pressence messages (#202)

    Options

    We've decided to implement the classes from the spec that define options as functional options instead.

    Each option is defined as a package-level function, with a With prefix. Additionally, except ClientOptions, each function has first the name of what they're options for as prefix (e. g. RestChannel.publish(params?)PublishBatchWithParams). (Discussion below.)

    • [x] ClientOptions and AuthOptions (#145)
      • [x] Converted to package-level functions (#205).
    • [x] ChannelOptions (#146)
      • [x] Converted to package-level functions (#205).

    Optional arguments

    Optional arguments are translated as functional options, as described above. (Discussion below.)

    If an optional argument is itself an options aggregate per the spec, the options "collapse" into a single level.

    io actions: from Result to blocking calls

    The spec marks with a io annotation values that result from making an IO operation (to the network, typically). In some environments (JavaScript, Cocoa) those are implemented by passing them to a callback or an equivalent Promise-style mechanism. In others, it's just a normal function call that blocks until the operation is complete and returns the value. If the user wants to do more than one thing at once, they use threads or some equivalent.

    In Go, it's common-place to do the latter: calls that hit the network block for the whole duration, and if you want asynchrony, you spawn goroutines. But we're using a two-stage Promise-like thing in which a method first initiates the operation on the background and returns (ably.Result, error) immediately; then, you can call ably.Result.Wait() to wait for the operation to complete, which may produce.

    This is weird for Go (see ably/docs#155) and we're switching to fully blocking calls for the duration of the operation.

    So this:

    result, err := channel.Attach()
    if err != nil {
        // ...
    }
    err = result.Wait()
    if err != nil {
        // ...
    }
    

    Becomes:

    err := channel.Attach()
    if err != nil {
        // ...
    }
    

    Context

    In Go, it has become common-place for IO operations to take a context.Context. For 1.2, we will implement the io annotation by adding a first context.Context argument in such methods.

    For HTTP requests, the context is passed in the underlying *http.Request.

    For operations that result in WebSocket sends and receives, the situation is a bit more complex. A WebSocket is ultimately a net.Conn, which is just a io.Reader and io.Writer; those don't take contexts. So the context would only be used to unblock the call at the point it's waiting for a WebSocket receive, e. g. an ATTACHED message.

    This raises the question: do we then act as if the operation has failed entirely? E. g. do we go back to the INITIALIZED/DETACHED state from ATTACHING?

    1. If we do, then the client and the server may disagree on the channel state. Normally, this possible discrepancy is resolved because the connection is broken and reestablished after a receive timeout or some other network condition that causes the error.
    2. If we don't, then we cause the surprising effect that the operation returns an error, yet we see the consequences of success anyway, e. g. start receiving messages on the channel.
    3. Another possibility is not taking contexts at all for WebSocket operations, but that seems inconsistent.

    I'd say we can go with 2 and just implement a transparent way of dealing with the inconsistency, e. g. automatically detach once the ATTACHED arrives. This should be in the spec too.

    Misc: names, argument order, etc.

    The rest would just be straightforward: make sure we expose only the API from the spec, in the form and shape in which the spec defines it. This includes renaming, removing things or moving them to separate packages, reordering arguments, etc.

    ┆Issue is synchronized with this Jira Story by Unito

  • Client appears to be leaking TCP connections/file descriptors

    Client appears to be leaking TCP connections/file descriptors

    We have some code in our service that is doing this:

    import (
    	"github.com/ably/ably-go/ably"
    )
    
    msg := "..."
    client := ably.NewRestClient(ably.NewClientOptions(apiKey))
    
    if err := client.Channel("private:some-channel").Publish("data_ready", string(msg)); err != nil {
    	return err
    }
    

    We are using revision 0cadbda3606928bcf94acc34cb9cc9ca0b9511ac of the ably-go client.

    When this runs, the service starts to leak a large number of TCP connections to Ably's rest.ably.io load balancer. Here's the output of lsof -p {service pid} after a few seconds of running:

    COMMAND   PID   USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
    {service} 27085 {user}   14u     IPv4 42056111      0t0      TCP ip-10-0-2-94.ec2.internal:43861->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   15u     IPv4 42057076      0t0      TCP ip-10-0-2-94.ec2.internal:46022->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   16u     IPv4 42057083      0t0      TCP ip-10-0-2-94.ec2.internal:43913->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   17u     IPv4 42056169      0t0      TCP ip-10-0-2-94.ec2.internal:46034->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   18u     IPv4 42057114      0t0      TCP ip-10-0-2-94.ec2.internal:43933->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   19u     IPv4 42057151      0t0      TCP ip-10-0-2-94.ec2.internal:43959->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   20u     IPv4 42057127      0t0      TCP ip-10-0-2-94.ec2.internal:20860->ec2-52-20-107-126.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   21u     IPv4 42056184      0t0      TCP ip-10-0-2-94.ec2.internal:46050->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   22u     IPv4 42056206      0t0      TCP ip-10-0-2-94.ec2.internal:46078->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   23u     IPv4 42056216      0t0      TCP ip-10-0-2-94.ec2.internal:43975->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   24u     IPv4 42056233      0t0      TCP ip-10-0-2-94.ec2.internal:46102->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   25u     IPv4 42056248      0t0      TCP ip-10-0-2-94.ec2.internal:44005->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   26u     IPv4 42057212      0t0      TCP ip-10-0-2-94.ec2.internal:46128->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   28u     IPv4 42056281      0t0      TCP ip-10-0-2-94.ec2.internal:44039->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   29u     IPv4 42056288      0t0      TCP ip-10-0-2-94.ec2.internal:46156->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    

    Eventually the connections enter a CLOSE_WAIT state, but still persist:

    COMMAND   PID   USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
    {service} 27085 {user}   12u     IPv4 42056997      0t0      TCP ip-10-0-2-94.ec2.internal:45966->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   13u     IPv4 42057706      0t0      TCP ip-10-0-2-94.ec2.internal:44331->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   14u     IPv4 42056111      0t0      TCP ip-10-0-2-94.ec2.internal:43861->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   15u     IPv4 42057076      0t0      TCP ip-10-0-2-94.ec2.internal:46022->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   16u     IPv4 42057083      0t0      TCP ip-10-0-2-94.ec2.internal:43913->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   17u     IPv4 42056169      0t0      TCP ip-10-0-2-94.ec2.internal:46034->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   18u     IPv4 42057114      0t0      TCP ip-10-0-2-94.ec2.internal:43933->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   19u     IPv4 42057151      0t0      TCP ip-10-0-2-94.ec2.internal:43959->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   20u     IPv4 42057127      0t0      TCP ip-10-0-2-94.ec2.internal:20860->ec2-52-20-107-126.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   21u     IPv4 42056184      0t0      TCP ip-10-0-2-94.ec2.internal:46050->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   22u     IPv4 42056206      0t0      TCP ip-10-0-2-94.ec2.internal:46078->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   23u     IPv4 42056216      0t0      TCP ip-10-0-2-94.ec2.internal:43975->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   24u     IPv4 42056233      0t0      TCP ip-10-0-2-94.ec2.internal:46102->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   25u     IPv4 42056248      0t0      TCP ip-10-0-2-94.ec2.internal:44005->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   26u     IPv4 42057212      0t0      TCP ip-10-0-2-94.ec2.internal:46128->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   27u     IPv4 42056339      0t0      TCP ip-10-0-2-94.ec2.internal:44057->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   28u     IPv4 42056281      0t0      TCP ip-10-0-2-94.ec2.internal:44039->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   29u     IPv4 42056288      0t0      TCP ip-10-0-2-94.ec2.internal:46156->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   30u     IPv4 42057383      0t0      TCP ip-10-0-2-94.ec2.internal:46186->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   31u     IPv4 42056381      0t0      TCP ip-10-0-2-94.ec2.internal:44093->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   32u     IPv4 42057433      0t0      TCP ip-10-0-2-94.ec2.internal:46226->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   33u     IPv4 42057489      0t0      TCP ip-10-0-2-94.ec2.internal:46276->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   34u     IPv4 42056411      0t0      TCP ip-10-0-2-94.ec2.internal:44117->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   35u     IPv4 42057454      0t0      TCP ip-10-0-2-94.ec2.internal:46246->ec2-52-207-88-5.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   36u     IPv4 42056469      0t0      TCP ip-10-0-2-94.ec2.internal:44169->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   37u     IPv4 42056431      0t0      TCP ip-10-0-2-94.ec2.internal:44137->ec2-52-0-70-16.compute-1.amazonaws.com:https (CLOSE_WAIT)
    {service} 27085 {user}   38u     IPv4 42057523      0t0      TCP ip-10-0-2-94.ec2.internal:46306->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   39u     IPv4 42056511      0t0      TCP ip-10-0-2-94.ec2.internal:44201->ec2-52-0-70-16.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   40u     IPv4 42056539      0t0      TCP ip-10-0-2-94.ec2.internal:46338->ec2-52-207-88-5.compute-1.amazonaws.com:https (ESTABLISHED)
    

    Eventually (within 5-10 minutes) lsof stops reporting the state of the connection and merely states it's a TCP connection:

    COMMAND   PID   USER   FD      TYPE   DEVICE SIZE/OFF     NODE NAME
    {service} 27085 {user}   12u     sock      0,8      0t0 42056997 protocol: TCP
    {service} 27085 {user}   13u     sock      0,8      0t0 42057706 protocol: TCP
    {service} 27085 {user}   14u     sock      0,8      0t0 42056111 protocol: TCP
    {service} 27085 {user}   15u     sock      0,8      0t0 42057076 protocol: TCP
    {service} 27085 {user}   16u     sock      0,8      0t0 42057083 protocol: TCP
    {service} 27085 {user}   17u     sock      0,8      0t0 42056169 protocol: TCP
    {service} 27085 {user}   18u     sock      0,8      0t0 42057114 protocol: TCP
    {service} 27085 {user}   19u     sock      0,8      0t0 42057151 protocol: TCP
    {service} 27085 {user}   20u     IPv4 42057127      0t0      TCP ip-10-0-2-94.ec2.internal:20860->ec2-52-20-107-126.compute-1.amazonaws.com:https (ESTABLISHED)
    {service} 27085 {user}   21u     sock      0,8      0t0 42056184 protocol: TCP
    {service} 27085 {user}   22u     sock      0,8      0t0 42056206 protocol: TCP
    {service} 27085 {user}   23u     sock      0,8      0t0 42056216 protocol: TCP
    {service} 27085 {user}   24u     sock      0,8      0t0 42056233 protocol: TCP
    {service} 27085 {user}   25u     sock      0,8      0t0 42056248 protocol: TCP
    {service} 27085 {user}   26u     sock      0,8      0t0 42057212 protocol: TCP
    {service} 27085 {user}   27u     sock      0,8      0t0 42056339 protocol: TCP
    {service} 27085 {user}   28u     sock      0,8      0t0 42056281 protocol: TCP
    {service} 27085 {user}   29u     sock      0,8      0t0 42056288 protocol: TCP
    {service} 27085 {user}   30u     sock      0,8      0t0 42057383 protocol: TCP
    {service} 27085 {user}   31u     sock      0,8      0t0 42056381 protocol: TCP
    {service} 27085 {user}   32u     sock      0,8      0t0 42057433 protocol: TCP
    {service} 27085 {user}   33u     sock      0,8      0t0 42057489 protocol: TCP
    {service} 27085 {user}   34u     sock      0,8      0t0 42056411 protocol: TCP
    {service} 27085 {user}   35u     sock      0,8      0t0 42057454 protocol: TCP
    {service} 27085 {user}   36u     sock      0,8      0t0 42056469 protocol: TCP
    {service} 27085 {user}   37u     sock      0,8      0t0 42056431 protocol: TCP
    {service} 27085 {user}   38u     sock      0,8      0t0 42057523 protocol: TCP
    {service} 27085 {user}   39u     sock      0,8      0t0 42056511 protocol: TCP
    {service} 27085 {user}   40u     sock      0,8      0t0 42056539 protocol: TCP
    {service} 27085 {user}   41u     sock      0,8      0t0 42056547 protocol: TCP
    

    It appears rest.ably.io is resolving to the load balancer mentioned in the lsof output:

    dig rest.ably.io
    
    ; <<>> DiG 9.10.3-P4-Ubuntu <<>> rest.ably.io
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52768
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 1280
    ;; QUESTION SECTION:
    ;rest.ably.io.			IN	A
    
    ;; ANSWER SECTION:
    rest.ably.io.		17	IN	A	52.207.88.5
    rest.ably.io.		17	IN	A	52.0.70.16
    
    ;; Query time: 0 msec
    ;; SERVER: 127.0.0.1#53(127.0.0.1)
    ;; WHEN: Mon Sep 24 12:07:02 UTC 2018
    ;; MSG SIZE  rcvd: 73
    

    These leaking connections eventually overwhelm our service, causing our systems to fail. Looking at the client interface it appears there's no obligation on the consumer of the client lib to call a Close() method, so it seems like the client may be failing to do some cleanup internally?

  • Go JSON / Binary / String support

    Go JSON / Binary / String support

    As discussed with @kouno, we need to be aware that ALL of our client libraries natively support three payload data types namely:

    • String UTF-8
    • Byte arrays (binary)
    • JSON

    Typically in dynamically typed languages, when you subscribe to messages, you will be delivered the data type automatically because when the message is received, it is decoded automatically, see https://github.com/ably/ably-ruby/blob/master/SPEC.md#ablymodelsmessageencoders.

    In more strongly typed languages such as Java, we still encode & decode automatically and return the correct data type, see https://github.com/ably/ably-java/blob/db50628fceb0af37c63c33693ab77dbb35d3c245/src/io/ably/types/BaseMessage.java#L97-L135.

    However, in Go, JSON is not a common data type and instead the norm is to Marshal JSON strings into typed Structs. The problem for us is that whilst we return a JSON data type in Ruby / Javascript / Java, for us to do this in Go we'd need to return a generic {{Interface{}}} object, see http://golang.org/src/encoding/json/decode.go?s=2340:2388#L57. We could do this, but my understanding is that this is largely frowned upon the Go community and instead JSON should be marshalled to a Struct. If we did return a generic interface, it's quite probable the developer would only then need to marshal it into another object so we're just adding unnecessary processing.

    I don't have a solution, but am raising this issue so that we can discuss.

    @paddybyers any thoughts?

  • Update imports to reference vendored packages

    Update imports to reference vendored packages

    The dependencies have already been vendored using godep (see 5d8ca38) but they were not being referenced.

    This is the result of running godep save -r ./....

  • Sometimes integration tests fail with a websocket dial error.

    Sometimes integration tests fail with a websocket dial error.

    One example of this was seen on this run https://github.com/ably/ably-go/runs/6405356210?check_suite_focus=true

    === RUN   TestRealtimePresence_Sync250
        realtime_presence_integration_test.go:71: 
            	Error Trace:	realtime_presence_integration_test.go:71
            	Error:      	Received unexpected error:
            	            	[ErrorInfo :websocket.Dial wss://sandbox-realtime.ably.io:443?echo=true&format=msgpack&heartbeats=true&key=_tmp_DELqOQ.4bzRew%3AZnRMoEAyCUwMR0BHKwYz3y7HUeuzkGl4FgEYMuHbmno&timestamp=1652358451326&v=1.2: read tcp 10.1.1.102:52358->52.85.35.170:443: read: connection reset by peer code=80003 disconnected statusCode=0] See https://help.ably.io/error/80003
            	Test:       	TestRealtimePresence_Sync250
    

    Note the error message "read: connection reset by peer". This feels like the sandbox server is closing the connection and the test then has an error because it has been disconnected.

    ┆Issue is synchronized with this Jira Task by Unito

  • Ablytest

    Ablytest

    Hello :wave:

    Thanks so much for the hard work in getting 1.2 released - I'm really excited to try it.

    To that end, we're trying to update from 1.1.5 to 1.2 but running into some difficulty. Some of the APIs have changed a good deal. But specifically preventing us right now is that ablytest has become internal: https://github.com/ably/ably-go/tree/v1.2.0/ably/internal/ablytest

    We've been using ablytest for integration tests for a long time (since this discussion from last year: https://github.com/ably/ably-go/issues/140). We're setting up a sandbox client and then publishing messages to it to make sure that all of our routing and message handling is working correctly. It has been very useful.

    We use two pieces of the ablytest package:

    ablytest.NewRestClient(nil) ablytest.Wait() (for publishing messages)

    Does the Ably team have a recommended replacement for the sandbox client? Or am I missing something?

    Thanks!

    ┆Issue is synchronized with this Jira Uncategorised by Unito

  • Testing challenges around use of embedded sync.Mutex in types

    Testing challenges around use of embedded sync.Mutex in types

    Some of the the types used in this project have sync.Mutex embedded directly into them e.g RealtimeChannel https://github.com/ably/ably-go/blob/main/ably/realtime_channel.go#L169 and RealtimePresence https://github.com/ably/ably-go/blob/main/ably/realtime_presence.go#L22

    the sync package states Values containing the types defined in this package should not be copied. This creates a challenge when we want to copy a mocked type that contains a sync.Mutex into a unit tests. We simply can't do this as go vet (correctly) objects to copying the mutex. We also have no way to test whether any particular sync.Mutex is locked or unlocked as a result of code execution because we can't use dependency injection to inject a mock mutex into the code.

    It might be worth investigating if representing sync.Mutex with the sync.Locker interface would make some of the code in this project testable at the unit test level. Interfaces are essentially mocking points and using sync.Locker could allow us to use a mock lock to make assertions in unit tests.

    ┆Issue is synchronized with this Jira Uncategorised by Unito

  • ably-go should not convert `[]byte` messages to strings

    ably-go should not convert `[]byte` messages to strings

    The code below send a message of type []byte. But the subscriber receives the message of type string.

    https://go.dev/play/p/VUqZbF7B6Wv

    2022/12/22 14:03:06 sending message [1 2 3 4] of type []uint8
    2022/12/22 14:03:06 Received message  of type: string
    

    According to the Spec (RSL6a) All messages received will be decoded automatically based on the encoding field and the payloads will be converted into the format they were originally sent using i.e. binary, string, or JSON

    In this case the message was sent as []byte so should have been received as as []byte.

    package main
    
    import (
    	"context"
    	"github.com/ably/ably-go/ably"
    	"log"
    	"os"
    )
    
    func sub(client *ably.Realtime, done chan struct{}) {
    	channel := client.Channels.Get("quickstart")
    	_, err := channel.Subscribe(context.Background(), "greeting", func(msg *ably.Message) {
    		log.Printf("Received message %v of type: %T", msg.Data, msg.Data)
    		close(done)
    	})
    	if err != nil {
    		panic(err)
    	}
    
    }
    
    func main() {
    
    	key := os.Getenv("ABLY_KEY")
    	client, err := ably.NewRealtime(ably.WithKey(key))
    	if err != nil {
    		panic(err)
    	}
    	done := make(chan struct{})
    	go sub(client, done)
    
    	channel := client.Channels.Get("quickstart")
    	data := []byte{1, 2, 3, 4}
    	log.Printf("sending message %v of type %T", data, data)
    	err = channel.Publish(context.Background(), "greeting", data)
    	if err != nil {
    		panic(err)
    	}
    	<-done
    }
    
  • Message Delta Compression support: subscription to VCDiff-encoded delta stream

    Message Delta Compression support: subscription to VCDiff-encoded delta stream

    This SDK does not need to implement this via a plugin, however we'll need to find an appropriate VCDiff codec implementation to utilise.

    It's not obvious to me why, but previously these two issues were individually created for this, despite them being deeply related in nature (i.e. both required parts of the whole feature). I'm going to close them as the implementation should happen under this issue, but listing them here for reference:

    • https://github.com/ably/ably-go/issues/241
    • https://github.com/ably/ably-go/issues/242
  • JWT (JSON Web Token) authentication

    JWT (JSON Web Token) authentication

    Lack of support for JWTs is documented under Known Limitations. It's also highlighted in the tests.

    Also, the concept of "unimplemented test" was introduced at some point to the backlog. I'm going to close all of these issues as they add no value to the backlog (the work to be done is under this issue) however I am recording them here for reference:

    • https://github.com/ably/ably-go/issues/408
    • https://github.com/ably/ably-go/issues/409
  • Inconsistent codebase state around whether `SUSPENDED` channel state is implemented

    Inconsistent codebase state around whether `SUSPENDED` channel state is implemented

    Our readme states:

    There is no channel suspended state; this means that the client will not automatically reattach to channels if a connection becomes suspended and then resumes, and presence members associated with the client will not be automatically re-entered.

    A code search for "SUSPENDED" reveals a few TODO comments against channel state code.

    Also, concept of "unimplemented test" was introduced at some point to the backlog. I'm going to close all of these issues as they add no value to the backlog (the work to be done is under this issue) however I am recording them here for reference:

    • https://github.com/ably/ably-go/issues/394
    • https://github.com/ably/ably-go/issues/395
    • https://github.com/ably/ably-go/issues/396
    • https://github.com/ably/ably-go/issues/397
    • https://github.com/ably/ably-go/issues/399
    • https://github.com/ably/ably-go/issues/401

    In the absence of deeper analysis I'm going to mark this issue as a bug for now.

  • Experimental/proto2.0

    Experimental/proto2.0

    This PR is a quick hack to allow me to load-test the new time series server-side code. It does not handle the protocol changes, especially everything around reconnections reconnections.

    Do not merge!

  • ably.WithEnvironment(

    ably.WithEnvironment("dev:name") gives a confusing error message

    ably.WithEnvironment("dev:name") 
    

    or any environment name containing a colon will give a confusing error message.

    [ERROR] Received recoverable error parse "wss://dev:name-realtime.ably.io": invalid port ":name-realtime.ably.io" after host`
    

    This causes additional pain as the colon form is used internally by Ably's Devops tooling.

    A clearer, more helpful error message would make these failures easier to understand.

Simple-messaging - Brokerless messaging. Pub/Sub. Producer/Consumer. Pure Go. No C.

Simple Messaging Simple messaging for pub/sub and producer/consumer. Pure Go! Usage Request-Response Producer: consumerAddr, err := net.ResolveTCPAddr

Jan 20, 2022
socket.io library for golang, a realtime application framework.

go-socket.io go-socket.io is library an implementation of Socket.IO in Golang, which is a realtime application framework. Current this library support

Jan 8, 2023
Golang client for NATS, the cloud native messaging system.

NATS - Go Client A Go client for the NATS messaging system. Installation # Go client go get github.com/nats-io/nats.go/ # Server go get github.com/na

Jan 5, 2023
Go-notification - Realtime notification system with golang

Realtime notification system Used Apache kafka gRPC & PROTOBUF MongoDB restapi w

Aug 19, 2022
A dead simple Go library for sending notifications to various messaging services.
A dead simple Go library for sending notifications to various messaging services.

A dead simple Go library for sending notifications to various messaging services. About Notify arose from my own need for one of my api server running

Jan 7, 2023
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.

Beaver A Real Time Messaging Server. Beaver is a real-time messaging server. With beaver you can easily build scalable in-app notifications, realtime

Jan 1, 2023
Scalable real-time messaging server in language-agnostic way
Scalable real-time messaging server in language-agnostic way

Centrifugo is a scalable real-time messaging server in language-agnostic way. Centrifugo works in conjunction with application backend written in any

Jan 2, 2023
websocket based messaging server written in golang

Guble Messaging Server Guble is a simple user-facing messaging and data replication server written in Go. Overview Guble is in an early state (release

Oct 19, 2022
Abstraction layer for simple rabbitMQ connection, messaging and administration
Abstraction layer for simple rabbitMQ connection, messaging and administration

Jazz Abstraction layer for quick and simple rabbitMQ connection, messaging and administration. Inspired by Jazz Jackrabbit and his eternal hatred towa

Dec 12, 2022
High-Performance server for NATS, the cloud native messaging system.
High-Performance server for NATS, the cloud native messaging system.

NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo

Jan 2, 2023
A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.
A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.

Apache Kafka in 6 minutes A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application. In thi

Oct 27, 2021
Golang Restful API Messaging using GORM ORM (MySQL) Gorilla Mux

Golang Restful API Messaging using GORM ORM (MySQL) Gorilla Mux Getting Started Folder Structure This is my folder structure under my $GOPATH or $HOME

Dec 14, 2021
Apr 12, 2022
Go-threema - Threema messaging from Go

Threema messaging from Go This is a Threema bot library written in Go. It uses p

Nov 6, 2022
golang client library to Viessmann Vitotrol web service

Package go-vitotrol provides access to the Viessmann™ Vitotrol™ cloud API for controlling/monitoring boilers. See https://www.viessmann.com/app_vitoda

Nov 16, 2022
RES Service protocol library for Go

RES Service for Go Synchronize Your Clients Go package used to create REST, real time, and RPC APIs, where all your reactive web clients are synchroni

Nov 23, 2022
Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 (and later).

Dec 28, 2022
Apache Pulsar Go Client Library

Apache Pulsar Go Client Library A Go client library for the Apache Pulsar project. Goal This projects is developing a pure-Go client library for Pulsa

Jan 4, 2023
It's client library written in Golang for interacting with Linkedin Cruise Control using its HTTP API.

go-cruise-control It's client library (written in Golang) for interacting with Linkedin Cruise Control using its HTTP API. Supported Cruise Control ve

Jan 10, 2022