expressive DynamoDB library for Go

@guregu So I'm still trying to track down the cause of this, but I have the following code:

t := db.db.Table("Location")
	k := make([]dynamo.Keyed, len(keys))
	for i, key := range keys {
		k[i] = dynamo.Keys{key, nil}
	}
	var ls []Location
	err := t.Batch("Id").Get(k...).All(&ls)
	retVal := make([]LocationDataResult, len(keys))
	for i := range keys {
		if i < len(ls) {
			retVal[i] = LocationDataResult{&ls[i], err}
		} else {
			retVal[i] = LocationDataResult{Error:err}
			//TODO--If error is nil, we say the object wasn't found
		}
	}
	return retVal

And when I pass in 2 keys, where one can be "found" and the other can't (as in, there's a single object in my DB table, and only one of those keys corresponds to it), I receive back 2 copies of the one "valid" object. I would expect to get back only one result?

I do see that the Keys are created correctly, and are getting appended onto the BatchGet correctly. That's where I'm at right now, and will continue to investigate further.

I have a server that makes frequent use of your excellent dynamo package to save various data. When an error is encountered, I print it out with a statement such as log.Println(err). Recently, I have been getting the error

SerializationException: status code: 400, request id: O6MMFM7MPC2NI4DV407EI50J0JVV4KQNSO5AEMVJF66Q9ASUAAJG

The last time I got a SerializationError, it was due to #137 which was a bug with this package.

This time, the code throwing the error was performing an Update operation. I suspect that the problem was update.Seting a string to an empty value. I will continue to look into the cause of the issue, but I am of the opinion that your library should catch this issue before it hits dynamo and either throw an preemptive and descriptive error, panic, or change the Update to a Delete (in my order of preference).

    update := db.userTable.Update(userIndexID, userID)
    update.Set("color", user.Color) // TextMarshaler
    update.Set("experience", user.Experience) // int
    update.Set("money", user.Money) // int
    if len(user.Achievements) > 0 {
        update.SetSet("achievements", user.Achievements) // map[TextMarshaler]Empty (Empty is struct{})
    }
    update.Set("statistics", user.Statistics) // map[TextMarshaler]float64
    if len(user.CanonicalName) > 0 { // added this check today, I'm now seeing if the error still appears
        update.Set("canonicalName", string(user.CanonicalName)) // UserName (string)
    }
    if len(user.Name) > 0 { // added this check today, I'm now seeing if the error still appears
        update.Set("name", user.Name) // UserName (string)
    }
    if user.Updated > 0 {
        update.Set("updated", user.Updated) // int64
    }
    return update.Run()

Further research reveals that empty strings are now supported by dynamodb for non-key attributes (see https://aws.amazon.com/about-aws/whats-new/2020/05/amazon-dynamodb-now-supports-empty-values-for-non-key-string-and-binary-attributes-in-dynamodb-tables/). I am continuing to investigate the true cause of the error (it might be that the version of the AWS SDK used is not high enough to support empty strings).

Do you know of a better way to diagnose SerializationExceptions than blindly guessing what caused it? The problem I just described has gone unnoticed in my server's logs for the last 3+ days, possibly causing data loss for a subset of users, and I would like to avoid this type of problem in the future.

I'm getting paginated query result 10 items each by callilng Query.Limit(10) and pass LastEvaluatedKey returned to next query. It seems to work correctly in usual cases, of course. But in case DynamoDB divides result due to 1MB limitation, LastEvaluatedKey returned by dynamo points an item ahead of the last item of result.

To my understanding, in such a case dynamo iterates to query internally to fulfill the result with 10 items then return it to client. But LastEvaluatedKey returned by DynamoDB in last query will not always match to the last item of returned to client. It seems be because dynamo passes Limit = 10, which is the value I passed to dynamo, to DynamoDB in every iterating query and LastEvaluatedKey points the last item of list DynamoDB returned, but not the last item of list returned to client.

Is it an expected behavior? Is there any way to get correct LastEvaluatedKey even in above case?

to reproduce:

err = table.Put(struct { ID string Test []string }{"test", []string{"hi", ""}}).Run() if err != nil { panic(err) }

results in:

panic: SerializationException: status code: 400, request id:

By dumping the marshalled output, it looks like it's being turned into:

{ M: { Test: { L: [{ S: "hi" },<invalid value>] }, ID: { S: "test" } } }

func (h Historical) BatchUpdateCustomerData(customerID string) (int, error) {
	fmt.Printf("Updating customer: %s", customerID)
	items := []map[string]*dynamodb.AttributeValue{}
	tbl := h.gDyn.Table(h.table)
	qry := tbl.Get("customer_id", customerID).Range("date", gdynamo.Greater, 0).Index(h.index)
	ct, _ := qry.Count()
	fmt.Sprintf("updating %d itmes", ct)
	qry.All(&items)
	btch := tbl.Batch().Write()
	for _, item := range items {
		btch.Put(item)
	}

	res, err := btch.Run()

	fmt.Sprintf("updated %d itmes", ct)
	if err != nil {
		return res, err
	}

	return res, nil
}

The above code executes fine for small data sets. For 4000+ items, I'm getting an error that looks like this:

Error: RequestCanceled: request context canceled
caused by: context deadline exceeded

I am trying to do a batch put, and no matter what I try, I get the following error: Error: ValidationException: The provided key element does not match the schema

The hash key in my table is id The range key in my table is date

This is my code:

func (h Historical) BatchUpdateCustomerData(customerID string) (int, error) {
	fmt.Printf("Updating customer: %s", customerID)
	items := []map[string]*dynamodb.AttributeValue{}
	tbl := h.gDyn.Table(h.table)
	qry := tbl.Get("customer_id", customerID).Range("date", gdynamo.Greater, 0).Index(h.index)
	x, _ := qry.Count()
	print(x)
	qry.All(&items)
	btch := tbl.Batch().Write().Put(gdynamo.AWSEncoding(items))

	res, err := btch.Run()

	if err != nil {
		return res, err
	}

	return res, nil
}

I am trying to retrieve data from a table, and put it back, in an unchanged form.

I tried something more basic, rather than items := []map[string]*dynamodb.AttributeValue{} I tried items := []map[string]interface{}{} and that didn't work either.

I also tried a simple struct:

type Updatable struct {
	ID         string  `dynamo:"id" json:"id"`
	Date       float64 `dynamo:"date" json:"date"`
	CustomerID string  `dynamo:"customer_id" json:"customer_id"`
}

And that also didn't work.

I keep getting the same error.

Even if I specify in the Batch() method my hash and range key specifically, such that Batch("id", "date"), I keep getting the same error.

If I have a table in DynamoDB, say Animals, where I insert both cats and dogs, is there a best practice way of fetching both types, at once, into an Animal list?

For Example in a batched query, I just want to hydrate all of the animals, but ensure all their properties are initialized. I know the following code won't work (how could it, without knowing what animal type you're trying to populate into), but am unsure of the best-practice way of doing this.

var ls []Animal
err := t.Batch("Id").Get(k...).All(&ls)

For reference, there is a property on every animal, called AnimalType that was meant to be used as a differentiator. Do I first have to batch query to get the types, and then somehow populate my animals from there?

The ConsumedCapacity type in guregu is basically the same as the ConsumedCapacity type in the AWS SDK (but "with less pointers"), except that where the Table, GlobalSecondaryIndexes, and LocalSecondaryIndexes fields in the AWS SDK further break down reads and writes...

// some fields omitted
type ConsumedCapacity struct {
	// The amount of throughput consumed on each global index affected by the operation.
	GlobalSecondaryIndexes map[string]*Capacity `type:"map"`

	// The amount of throughput consumed on each local index affected by the operation.
	LocalSecondaryIndexes map[string]*Capacity `type:"map"`

	// The amount of throughput consumed on the table affected by the operation.
	Table *Capacity `type:"structure"`
}

type Capacity struct {
	// The total number of capacity units consumed on a table or an index.
	CapacityUnits *float64 `type:"double"`

	// The total number of read capacity units consumed on a table or an index.
	ReadCapacityUnits *float64 `type:"double"`

	// The total number of write capacity units consumed on a table or an index.
	WriteCapacityUnits *float64 `type:"double"`
}

...guregu has opted to throw away the read/write breakdown and just report the total (corresponding to CapacityUnits above):

// some fields omitted
type ConsumedCapacity struct {
	// GSI is a map of Global Secondary Index names to consumed capacity units.
	GSI map[string]float64
	// GSI is a map of Local Secondary Index names to consumed capacity units.
	LSI map[string]float64
	// Table is the amount of throughput consumed by the table.
	Table float64
}

This makes it impossible to accurately break down consumed capacity for complex queries, and for no reason I can discern. If the goal is to have fewer pointers, you can still have a Capacity struct that breaks down read and write according to the information returned by Dynamo.

I can see a couple ways forward to fix this:

1. Change the custom ConsumedCapacity type to include all three pieces of information present in the AWS response. This would unfortunately have to be a breaking change, or would require new fields to be introduced.
2. Expose the raw AWS SDK ConsumedCapacity type, pointers and all. This would not be breaking.

Fixes #186. Previously, PagingIter.LastEvaluatedKey always returned the LastEvaluatedKey given to us by AWS, regardless of whether we were at the end of the iterator. Limit (not SearchLimit) also triggered this behavior.

This patch improves PagingIter, having it automatically determine LastEvaluatedKey based on the actual last value that we have locally evaluated.

Usually we can determine this without any extra effort, but rarely we will need to call DescribeTable to fetch the necessary key schemas. In this case, the table's description is cached inside of dynamo.Table via atomics.

This won't affect you the following use cases, which already had correct behavior:

• you always iterated to the end of an iterator before calling LastEvaluatedKey()
• you didn't use Limit (SearchLimit is OK)

I was getting a cryptic error messages when trying to create a table, after looking at the source code I realized it was because I was missing the ID, which by the way I thought was determined by convention. The error message was index out of range [0] with length 0.

After looking at the issues I found an issue that had this struct tag dynamo:"id,hash" in the code that showed how to specify the ID. Then I looked into the documentaiton to see if this was there but couldnt find anything. Is there any place where all of the possible struct tags are doucmented? I'm still trying to specify the sort key in the code.

Currently, you cannot specify a client token for Dynamo transactions. This PR adds that functionality through a new method to the WriteTx struct

IdempotentWithToken(enabled bool, token string) *WriteTx

This method will use the token provided, only if it is not blank. If it is not blank and enabled, it will use that token for requests.

Migration to AWS SDK v2.

This PR is based on @niltonkummer's work, and I made modifications to pass tests and finish as a PR to the main stream. This version is working fine with my own application (a DB performance test tool).

some notes:

• TestTx has a high chance of failing because of https://github.com/aws/aws-sdk/issues/413. It only passes once on 70 tries in my experiments.
• I removed most of retry logic since the SDK already has its own logic.

Problem encode.go

tags := strings.Split(field.Tag.Get("dynamo"), ",")

this is not flexible

Advice user should be able to customize the tag name they want, like json

Can we provide a public function MarshalStruct for external calls, in case the caller just needs to add some extra fields to the marshal process but doesn't want to actually handle the marshal process himself?

For example:

// dynamodb table
type Item struct {
    PK string `dynamo:",hash"`
}

type User struct {
    Username string
    Email string
}

type UserItem struct {
    *Item
    *User
}

// We want to add PK automatically in the marshal process
func (u *User) MarshalDynamoItem() (map[string]*dynamodb.AttributeValue, error) {
    return dynamo.MarshalStruct(&UserItem{&Item{PK:"U#"+u.Username}, u})
}

I have noticed that when storing time.Time values with the unixtime option, that zero value are always omitted. With the following struct, the MyFIeld is never stored if its value is 0.

type MyStruct struct {
	MyField     time.Time `dynamo:",unixtime,allowempty"`
}

The top level README is also slightly misleading. It talks about the TextMarshaller defaulting to omitempty and time.Time triggering this behaviour, but it doesn't mention that when using the unixtime enconding, the TextMarshaller is ignored and there is special code in the library to always omitempty.

Without this field present my objects don't appear in my global secondary index which does make it a touch difficult for me to query them. There may be smarter things I can do with AWS, but I'd prefer to just store 0 in that field.

Hello,

How would you recommend I do something like this with this library:

dynamoDB.updateItem({
  TableName: "Users",
  Key: { "UserId": { S: "c6af9ac6-7b61" } },
  ExpressionAttributeValues: { ":inc": {N: "1"} },
  UpdateExpression: "ADD loginCount :inc"
})

I am currently fetching the row getting the value then adding 1 to it but I am worried about race conditions.