The Go library is the reference implementation, though not everyone uses Go. That's why it's useful to link to other implementations so people can use KSUID in the language of their choice.

Has any analysis of common hash distribution of ksuid values been done? There exists use cases where you want the time ordering for client side uses of ids but on the server side use the ids for distributed storage in a cluster data store. In this case a typical strategy is to hash the ids and use that to determine a shard within the datastore for that entity.

Does the timestamp prefixing cause hashes of ksuids to cluster under common hash functions, e.g. MD5, various FNV or SHA algorithms etc?

The changes in the PR optimize the implementation of NewRandom, here's a highlight of the changes:

• Use a non-cryptographic random source to avoid having to make a syscall to generate KSUIDs, use Go's math/rand.Rand seeded with a crypto-random source
• Get rid of of the payload memory pool, use a global buffer instead. Since we have to pay the cost of using a mutex to access the random generator, we load the random bytes in a global buffer synchronized on the same mutex and then copy the results to a local variable.
• Load the payload directly into the generated KSUID variable and inline the time serialization from FromParts.
• Change ReadFull to ReadAtLeast, for some reason the compiled failed to inline this function call and resulted in ~3% of the remaining cost of NewRandom.

Here are the numbers before and after:

$ go test -run _ -v -bench New -cpuprofile /tmp/cpu -benchmem -memprofile /tmp/mem
BenchmarkNew-4   	 1000000	      1234 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/segmentio/ksuid	1.258s

$ go test -run _ -v -bench New -cpuprofile /tmp/cpu -benchmem -memprofile /tmp/mem
BenchmarkNew-4   	20000000	       105 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/segmentio/ksuid	2.228s

Please take a look and let me know if something should be changed.

I'm very exciting about ksuid. But when I want create a new table on postgres with primary key is ksuid. So what kind of data type should I set for ksuid ? How about performance of it in big query ? Beause with UUID, it's supported by data type is uuid. Thank you so much

This PR adds data structures to work with KSUIDs.

The first one is the Sequence type, which can be used to generate an ordered sequence of KSUIDs.

The second is the CompressedSet type, which is an ordered set of KSUIDs that uses techniques inspired from bitmap compression to efficiently store KSUIDs, especially when IDs are packed in sequences or close to each other.

Here are the test and benchmarks:

$ go test -v -run CompressedSet -bench CompressedSet -benchmem
=== RUN   TestCompressedSet
=== RUN   TestCompressedSet/sparse
=== RUN   TestCompressedSet/packed
=== RUN   TestCompressedSet/mixed
--- PASS: TestCompressedSet (0.00s)
    --- PASS: TestCompressedSet/sparse (0.00s)
    	set_test.go:126: original 20000 B, compressed 17102 B (14.49%)
    --- PASS: TestCompressedSet/packed (0.00s)
    	set_test.go:126: original 20000 B, compressed 2153 B (89.23%)
    --- PASS: TestCompressedSet/mixed (0.00s)
    	set_test.go:126: original 20000 B, compressed 5022 B (74.89%)
goos: darwin
goarch: amd64
pkg: github.com/segmentio/ksuid
BenchmarkCompressedSet/write-4         	    3000	    465194 ns/op	      32 B/op	       1 allocs/op
BenchmarkCompressedSet/read-4          	   20000	     62202 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/segmentio/ksuid	3.380s

• 465us to compress 1000 KSUIDs the profile shows that this is largely dominated by the sorting operation, I'll see what I can do about this
• 62us to decompress 1000 KSUIDs I think this is pretty good perf, especially considering the kind of compression ratios that this data structure can achieve when storing lots of KSUIDs that were produced around the same time or using sequences.

Please take a look and let me know if anything should be changed.

Readme says that the timestamp provides over 100 years of life. What happens after 100 years? Does it wrap and use the same timestamp around 100 years ago? Will it break index logicality or other things? Moving 4 bits from the random part to the timestamp would make it 2000 years and wouldn't change much the library guarantees so I wonder why not do it?

Maybe I'm misunderstanding something, but, isn't the date in README wrong?

I mean, where it says

The timestamp epoch is adjusted to March 5th, 2014 [...]

, shouldn't it be May 13th, 2014 ? Given that

go run ./ksuid -f inspect 000000000000000000000000000

outputs

REPRESENTATION:

  String: 000000000000000000000000000
     Raw: 0000000000000000000000000000000000000000

COMPONENTS:

       Time: 2014-05-13 18:53:20 +0200 CEST
  Timestamp: 0
    Payload: 00000000000000000000000000000000

Now that the Go ecosystem started to lean towards using semantic versioning, using git tags for this package might help people utilizing the new way in their applications. Also, gives the impression that this package is production ready.

What do you think?

When I created #49 I used the rust implementation that I thought was best based on looking at github/crates.io

Though having actually used it now, it's very much lacking, so I ended up implementing a new one that has a saner internal representation, integrates with the rest of the Rust ecosystem, and tests a good chunk of the KSUID value space for correctness, as can be seen in: https://github.com/svix/rust-ksuid/blob/main/tests/test_kuids.txt

Hi, To set DEFAULT value for an PK of table, I wonder if has anyone tried to generate a valid ksuid value by using SQL.

For example for UUIDv4, we can generate default value if uuid-ossp extension is not installed for PostrgreSQL as:

`CREATE TABLE IF NOT EXISTS table (
   id VARCHAR(36) DEFAULT md5(random()::text || clock_timestamp()::text)::uuid  NOT NULL,
....

Of course it'll be better to generate from application but, I it makes simpler to add default value for manual interactions with tables.

Has anyone tried to develop an SQL function to geneate ksuid?

Hi, this is an observation, not a bug.

ksuid.go says

Go's default random number generators are not safe for concurrent use by multiple goroutines, the use of the rander and randBuffer are explicitly synchronized here.

Thus it proceeds to lock its own randMutex.

I have the intuition that crypto/rand.Reader would be broken if it wasn't safe for concurrent use, and that actually it is safe. This seems confirmed by the implementations in rand_unix.go and rand_windows.go which already use their own mutex.

I was curious if removing randMutex would improve the benchmark numbers, but it doesn't.

Still, randMutex might be unnecessary.

Greetings,

I am compiling a GUID summary sheet which tries to aggregate different dimensions with the intent to help engineers pick the right generator.

Would you be kind enough to help me validate the KSUID column and point to any issues?

Thank you

If there's no technical problem could make it configurable? Like:

// It will return a configurable instance.
instance := ksuid.NewInstance(ksuid.Config{
   MaxPayloadLength: 32,
})

And if possible a method to replace the global instance like golang zap:

instance := ksuid.NewInstance(ksuid.Config{
   MaxPayloadLength: 32,
})

// instance will create a 32 bit payload while ksuid.New() will create a 16 bit payload.
// to avoid that we could do something like:

undo := ksuid.ReplaceGlobals(instance)
defer undo()

// here the ksuid and instance are both configurable in the same way so I could call 
// ksuid.New() or instance.New() and both will create a 32 bit payload.

Hey, I created a postgres extension that was written in rust, and can be used in postgres to generate ksuid values for various stuff.

check it here - https://github.com/spa5k/uids-postgres

mainly created it for experiment and lack of a good postgres extension written in ksuid, ulid and nanoid.

One extension for all 3 of em.

Creation You can create a new instance synchronously:

const ksuidFromSync = KSUID.randomSync() Or asynchronously:

const ksuidFromAsync = await KSUID.random()

It seems that it wraps around rather than returning an error.

https://go.dev/play/p/s2aA3-ERd-E

Uses timestamp 2000-01-01T00:00:00.00Z and zeros for the payload. Returns: WfeXVK2UN8Qh86bSI6R4zNvC7ge

$ ksuid -f inspect WfeXVK2UN8Qh86bSI6R4zNvC7ge

REPRESENTATION:

  String: WfeXVK2UN8Qh86bSI6R4zNvC7ge
     Raw: E4FAF58000000000000000000000000000000000

COMPONENTS:

       Time: 2136-02-07 01:28:16 -0500 EST
  Timestamp: 3841652096
    Payload: 00000000000000000000000000000000