go-pmem is a project that adds native persistent memory support to Go.

Hello,

I am interested in investigating persistent memory for a db project and am wondering if this library can be build on a Windows 11 (64-bit) platform as well for testing?

What version of Go are you using (go version)?

$ go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

$ go env

What did you do?

I wanted to import go-pmem/transaction/redis, to my project to use pnew(), pmake(), t_string, dictionary, etc.

What did you expect to see?

go get should able to import.

What did you see instead?

The errors: go get github.com/vmware/go-pmem-transaction/pmem "#github.com/vmware/go-pmem-transaction/transaction" ../work/pkg/mod/github.com/vmware/[email protected]/transaction/movnt.go:26:2: undefined: runtime.FlushRange ../work/pkg/mod/github.com/vmware/[email protected]/transaction/optEnd.go:73:3: undefined: runtime.Fence ../work/pkg/mod/github.com/vmware/[email protected]/transaction/optEnd.go:85:3: undefined: runtime.FlushRange ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:97:15: undefined: pnew ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:97:19: type redoTxHeader is not an expression ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:104:3: undefined: runtime.PersistRange ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:106:3: undefined: runtime.PersistRange ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:135:8: undefined: pnew ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:135:12: type redoTx is not an expression ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:137:11: undefined: pmake ../work/pkg/mod/github.com/vmware/[email protected]/transaction/redoTx.go:137:11: too many errors

This change adds a new Go keyword called "txn", which can be used as:

txn() {
 // any code
}

The parser/AST creator will identify this keyword & inject some statements to initialize transaction variables & begin a transaction. There is no other functionality yet. All the code within txn() {} will execute as they should.

To build code using txn keyword: go build -txn

If you use only go build & have txn keyword in your code, you get a syntax error "syntax error: txn() use not supported. Add -txn flag while building"

Testing: all.bash with & without -txn flag

And a simple code like this works as intended:

import (
	"github.com/vmware/go-pmem-transaction/pmem"
	"github.com/vmware/go-pmem-transaction/transaction"
	"fmt"
	"os"
)

func main() {
	os.Remove("testFile")
	pmem.Init("testFile")
	var a int
	txn() {
		a = 2
		fmt.Println(a)
	}

	txn() {
		b := 3
		fmt.Println(b)
	}
}

Includes all the changes to-date (including the ability to use sync.(*RWMutex).Lock & Unlock & carry transactions across function calls). All changes are fully functional only for undo transactions. The changes have some TODOs marked with TODO: (mohitv) which I plan to improve as we progress.

This commit changes GC to be run in a new goroutine rather than as a stop the world (STW) event. This significantly speeds up the re-initialization function.

Earlier, GC was run as a stop-the-world event and initialization was declared to be complete only after the complete execution of GC. Now, initialization completes as soon as GC is started in a separate goroutine.

To compare impact of running garbage collection in a new goroutine vs as a STW event, I measured the memtier benchmark throughput in the two cases. The below graph shows the throughput while running memtier benchmark against Go Redis in two cases - (i) when GC is run in a new goroutine (ii) when GC is run as a STW event. The throughput of memtier benchmark is measured in kilo-operations per second. It is seen that, up to 700 milliseconds, the throughput in case (i) is pretty low. From 750 milliseconds onwards, the throughput picks up and is seen to be similar in the two cases.

Memtier benchmark configuration

Num. clients = 10 Num threads = 10 Num. pipeline requests = 100 Num. requests = 1000000

Initialization Time Comparison

Initialization time was compared in the two cases using Go Redis and memtier benchmark. About 2.6M Key value pairs were inserted into Go redis and the heap load time measured. The total database size was 1.28GB. More than 10x speedup was observed. No pointer swizzling was done in the two cases.

GC run as STW - 3.53 s GC run in a new goroutine - 270.62 ms

I ran memtier benchmark on go-redis-pmem with the master branch and go1.12b branch. With Go 1.12 branch, performance is seen to drop by about 20%.

$ memtier_benchmark --ratio=1:0 --data-size-range=32-1000 --random-data --clients=5 —threads=5 --distinct-client-seed --requests=830000 --hide-histogram --pipeline=128 —randomize

Go 1.12 branch

| Ops/sec | Hits/sec | Misses/sec | Avg. Latency | p50 Latency | p99 Latency | p99.9 Latency | KB/sec | |---------|----------|------------|--------------|-------------|-------------|---------------|--------| | 337695.26 | 0.00 | 0.00 | 7.53648 | 5.67900 | 42.01500 | 183.80700 | 185881.66 |

Master branch

| Ops/sec | Hits/sec | Misses/sec | Avg. Latency | p50 Latency | p99 Latency | p99.9 Latency | KB/sec | |---------|----------|------------|--------------|-------------|-------------|---------------|--------| | 424392.20 | 0.00 | 0.00 | 5.99227 | 4.15100 | 61.37500 | 154.49500 | 233716.19 |

I ran a few other benchmarks as well and the following are the performance results:

1. binaryTree

depth = 21

Running time a. Go 1.12 branch - 45.91 seconds b. Master branch - 46.51 seconds

2. fannkuch

length = 11

Running time a. Go 1.12 branch - 38.30 seconds b. Master branch - 39.21 seconds

3. n-body

steps = 3000000

Running time a. Go 1.12 branch - 30.19 seconds b. Master branch - 30.01 seconds

This change adds a new go build flag called "txn". This flag is read by compiler and will be used to protect normal compiler path from future transaction related changes. The changes are similar to how compiler reads msan/race build flags.

Example use: go build -txn Testing: I ran go build with an unrecognized flag & with "-txn" & see expected outputs. I also ran "all.bash" with -txn flag passed to all tests. Everything passes.

What version of Go are you using (go version)?

$ go version

go version devel +51612660da Thu Jan 21 18:58:07 2021 -0800 linux/amd64

Does this issue reproduce with the latest release?

This problem is specific to the example.go program under go-pmem. I can run other go example programs well.

What operating system and processor architecture are you using (go env)?

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/wkyu/.cache/go-build"
GOENV="/home/wkyu/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/wkyu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/wkyu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/wkyu/research/gc/go-pmem"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/wkyu/research/gc/go-pmem/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="0"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build184554646=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Followed the README.md until the following steps.

wkyu@inv10:~/research/gc/go-pmem$ go get -u github.com/vmware/go-pmem-transaction/...
go: found github.com/vmware/go-pmem-transaction/... in github.com/vmware/go-pmem-transaction v0.0.0-20210401173707-6e2b76e1a20d wkyu@inv10:~/research/gc/go-pmem$ cd design/ wkyu@inv10:~/research/gc/go-pmem/design$ GO111MODULE=off ../bin/go build -txn example.go example.go:7:2: cannot find package "github.com/vmware/go-pmem-transaction/pmem" in any of: /home/wkyu/research/gc/go-pmem/src/github.com/vmware/go-pmem-transaction/pmem (from $GOROOT) /home/wkyu/go/src/github.com/vmware/go-pmem-transaction/pmem (from $GOPATH) example.go:8:2: cannot find package "github.com/vmware/go-pmem-transaction/transaction" in any of: /home/wkyu/research/gc/go-pmem/src/github.com/vmware/go-pmem-transaction/transaction (from $GOROOT) /home/wkyu/go/src/github.com/vmware/go-pmem-transaction/transaction (from $GOPATH)

I only used the examle.go under the latest checkout from https://github.com/jerrinsg/go-pmem.git

What did you expect to see?

I expect the example.go to run smoothly.

What did you see instead?

The code breaks with an error on not able to find pmem and transaction modules. They are both available under my default GOPATH as shown below.

wkyu@inv10:~/research/gc/go-pmem/design$ ls /home/wkyu/go/pkg/mod/github.com/vmware/[email protected]/ CONTRIBUTING.md examples LICENSE NOTICE pmem pmemtest README.md transaction txtest

ONLY change all.bash to enable testing with -txn flag. Includes fix to do clean.bash, so Travis will not reuse cached test data. Also adds to the GO_GCFLAGS instead of just writing to it.

Repro instructions: cd src/ ./all.bash

Failing signature: --- FAIL: TestAnalyzeAnnotations (0.00s) annotations_test.go:101: failed to trace the program: gWaiting in makerunnable 36:[g:8 p:0 GoSched/17 [0 0 0] [] 1eb2a ->0] [waiting GoBlockSync:11f6f,,,] --- FAIL: TestAnalyzeAnnotationGC (0.00s) annotations_test.go:271: failed to trace the program: p already running 24, 59:[g:66 p:2 GoStart/14 [66 0 0] [] 15984 ->0]

Hello,

I am trying to build go-pmem on a Windows 11 using MSYS2 (Mingw64) for which I am running Go 1.18.

lonni@DESKTOP10 MINGW64 ~/go-pmem/test/src
$ ./make.bat
ERROR: Cannot find C:/msys64/mingw64/bin/go\bin\go.exe
Set GOROOT_BOOTSTRAP to a working Go tree >= Go 1.4.

I have tried using "set" and "export" to set the GOROOT_BOOTSTRAP variable, but I still get the same result.

Can you please advise?

What version of Go are you using (go version)?

Latest

Does this issue reproduce with the latest release?

N/A

What operating system and processor architecture are you using (go env)?

Linux x86

It would be great to add support for pmem over rdma like is included in the pmemdk, with rpmem. Similar to https://github.com/pmem/rpma and https://pmem.io/pmdk/librpmem/

Running go-redis-pmem with memtier benchmark is resulting in the following runtime error:

./app
First time initialization
Go-redis is ready to accept connections
Dictionary used / size: 4079 / 1024  ,Resize table to:  4096
panic: runtime error: index out of range [115] with length 0

memtier_benchmark --ratio=1:0 --data-size-range=32-1000 --random-data --clients=5 —threads=5 --distinct-client-seed --requests=50000 --hide-histogram --pipeline=128 —randomize

goroutine 60 [running]:
github.com/vmware-samples/go-redis-pmem/redis.(*dict).lockShard(0xc006308000, 0x1, 0x73)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/dictionary.go:368 +0x15a
github.com/vmware-samples/go-redis-pmem/redis.(*dict).lockKey(0xc006308000, 0xc001979b10, 0xf, 0xf)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/dictionary.go:309 +0xba
github.com/vmware-samples/go-redis-pmem/redis.(*redisDb).lockKeyWrite(0xc006304000, 0xc001979b10, 0xf, 0xf)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/db.go:129 +0x85
github.com/vmware-samples/go-redis-pmem/redis.setGeneric(0xc000420240, 0x0, 0xc001979b10, 0xf, 0xf, 0xc0019209c0, 0xb1, 0xb1, 0x0, 0x0, ...)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/t_string.go:94 +0x7a
github.com/vmware-samples/go-redis-pmem/redis.setCommand(0xc000420240)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/t_string.go:61 +0x316
github.com/vmware-samples/go-redis-pmem/redis.(*client).processCommand(0xc000420240)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/server.go:425 +0x10d
github.com/vmware-samples/go-redis-pmem/redis.(*client).processInput(0xc000420240)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/server.go:317 +0x9c
github.com/vmware-samples/go-redis-pmem/redis.(*server).handleClient(0xc0001a0590, 0xc000296008)
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/server.go:272 +0x5f
created by github.com/vmware-samples/go-redis-pmem/redis.(*server).Start
	/home/jerrin/gocode/src/github.com/vmware-samples/go-redis-pmem/redis/server.go:199 +0x199
panic: runtime error: index out of range [1109] with length 0

Error seems to be during lock acquisition inside a txn{} block:

364 func (d *dict) lockShard(t, s int) {
365     // ReadOnly commands will aquire readOnly tx and read locks, otherwise
366     // WLock is aquired.
367     txn("undo") {
368         d.tab[t].bucketlock[s].Lock()                         // <<<<<<<<<< ERROR SEEN HERE
369     }
370 }

This error is seen while running with the txn{} version of go-redis-pmem. It is not seen with the Log3 version of go-redis-pmem. Assigning to Mohit to take a look.

If we use copy() within txn(), the store will happen, but the old values will not be logged before the update. Go compiler replaces copy with a runtime call very early during compilation. So during ssa creation, there is no easy way to identify a node which was originally "OCOPY" node.

Ideally go-pmem needs to support transactional copy() semantics within a txn() block OR throw a syntax error if user uses copy() within txn() block.

When the pmem file name is long, or filename has the full path and you initialize the pmem file with the call pmem.Init(/mnt/ext4-pmem0/mohitv/myPmemFiles/pmemFile0.txt) the program fails with the error: fatal error: malloc deadlock

Most likely, the long file name triggers a recursive memory allocation call within runtime.

Type profiling is used to keep track of how many times a datatype has been allocated. This is used to promote heavily allocated types to be specially cached in mcache so that future allocations would be fast. Currently a large static array of size 50,000 is used to track these stats. This is because the index of a type within a Go binary cannot be known at runtime currently, so an offset from a static location typeBase is used to determine type index. But using a large static array is not a good idea. It increases the binary size and also could cause runtime issues. For example, when etcd was compiled with go-pmem certain type fields were seen to overshoot the index value of 50,000.

In Go, a string is internally a slice of bytes. Right now we don't support creating a slice with pnew() builtin. Currently, code like s := pnew(slice) *s = "hello"

works. But s is actually pointing to volatile memory. This will create confusion. Instead, we should panic for now.