Reinforcement Learning in Go

Using go1.14.1 with https://github.com/aunum/gold/commit/8dedf68d0993b8fb4355edfc560c1567ea9fb80c I get a panic in taxi experiment:

go run -trimpath ./pkg/v1/agent/q/experiments/taxi/main.go
ℹ  tracking data in /var/folders/zy/lzyp__pd7vx6762s5jcgkkl80000gn/T/stats.104998610.json
ℹ  connecting to existing container "stupefied_taussig"
✔  connected to server "gym"
✔  created env: f07159db-890d-45ca-a295-aa9e383507de
ℹ  serving agent api/ui on http://localhost:39421
ℹ  running for 5000 episodes
✖  rpc error: code = Unknown desc = Exception calling application: 'int' object has no attribute 'ravel'
goroutine 1 [running]:
runtime/debug.Stack(0x75, 0x0, 0x0)
        runtime/debug/stack.go:24 +0x9d
runtime/debug.PrintStack()
        runtime/debug/stack.go:16 +0x22
github.com/aunum/log.fatalf(0xc0007e0100, 0x1c76e46, 0x3, 0xc0007e1f28, 0x1, 0x1)
        github.com/aunum/[email protected]/log.go:77 +0x11d
github.com/aunum/log.Fatalf(...)
        github.com/aunum/[email protected]/log.go:64
github.com/aunum/log.Fatal(0xc0007e1f28, 0x1, 0x1)
        github.com/aunum/[email protected]/log.go:109 +0xa3
github.com/aunum/gold/pkg/v1/common/require.NoError(...)
        github.com/aunum/gold/pkg/v1/common/require/require.go:18
main.main()
        command-line-arguments/main.go:36 +0x303
exit status 1

Cloned repository and using the quick start instructions.

$ go run ./pkg/v1/agent/deepq/experiments/cartpole/main.go # github.com/aunum/gold/pkg/v1/dense pkg/v1/dense/value.go:57:19: cannot convert FauxZero (type float64) to type complex64 pkg/v1/dense/value.go:59:20: cannot convert FauxZero (type float64) to type complex128

Using go v1.14 on linux/amd64

It seems that the only working environment is CartPole-v1. If I try to use LunarLander-v2 or something else from Box2D, I get an error, for example:

	server, err := env.FindOrCreate(env.GymServerConfig)
	if err != nil {
		log.Fatal(err)
	}
	defer server.Close()

	environment, err = server.Make("LunarLander-v2", env.WithRecorder())
	if err != nil {
		log.Fatal(err)
	}

Error:

2020/04/10 18:15:13 rpc error: code = Unknown desc = Exception calling application: module 'gym.envs.box2d' has no attribute 'LunarLander'

I wonder whether it's expected behavior or not? Should I perform some extra steps to enable additional environments?

@JetSetIlly reported https://github.com/aunum/gold/issues/2 and discovered it was caused by https://go-review.googlesource.com/c/go/+/226197/ and recommended the following fix

A breaking change occurred in Go between 13.6 and 14.1 with float64 to complex64 casting, still need to investigate more closes https://github.com/aunum/gold/issues/2

Hi,

I'm trying to create my own agent, following the example here: https://github.com/aunum/gold/tree/master/pkg/v1/agent/reinforce It's an example for CartPole-v0, while I'm playing around with LunarLander-v2.

I've come now to implementing the "step" function. And I realized that Env.Step() function takes int as an argument. Maybe that's what is needed for the CartPole-v0, but LunarLander-v2's step is float: https://github.com/openai/gym/blob/master/gym/envs/box2d/lunar_lander.py#L111

# Action is two floats [main engine, left-right engines].
# Main engine: -1..0 off, 0..+1 throttle from 50% to 100% power. Engine can't work with less than 50% power.
# Left-right:  -1.0..-0.5 fire left engine, +0.5..+1.0 fire right engine, -0.5..0.5 off

Pretty much everywhere in the gold library where I meet anything action-related - it's int.

How should it be mapped to the continuous environments? Or is it not possible at the moment? If so, would it be difficult to implement it?

C:\Users\<...>\go\pkg\mod\github.com\aunum\[email protected]\pkg\v1\agent\q\experiments\taxi>go run main.go
# github.com/ory/dockertest/docker/pkg/system
..\..\..\..\..\..\..\..\ory\[email protected]+incompatible\docker\pkg\system\filesys_windows.go:113:24: cannot use uintptr(unsafe.Pointer(&sd[0])) (type uintptr) as type *"golang.org/x/sys/windows".SECURITY_DESCRIPTOR in assignment

go version go1.16.6 windows/amd64

Hi,

I'm running a lot of episodes of LunarLander in a loop (maybe hundreds). When I call environment.End() in the end, I get all the episodes information printed, like this:

...

  "15":
    episode_id: 15
    episode_length: 80
    reward: 20.416142
    timestamp:
      seconds: 1588666897

...

But, for some reason, it records only a few videos:

Videos:
  "0":
    content_type: video/mp4
  "1":
    content_type: video/mp4
    episode_id: 1
  "8":
    content_type: video/mp4
    episode_id: 8
  "27":
    content_type: video/mp4
    episode_id: 27

I'm having a hard time trying to find the parameter that can turn on saving video for each and every episode. Is it even possible?

Currently toy text problems don't have visualization because the only way they are rendered is with ansi color codes. Should provide a means of rendering that locally in the terminal.

https://github.com/aunum/gold/issues/5 was caused by a local build of the sphere container that hadn't been pushed and highlighted the need for build pipelines