A recommender system service based on collaborative filtering written in Go

Language: English | 中文

gorse: Go Recommender System Engine

Build Coverage Report GoDoc RTD Demo
build codecov Go Report Card GoDoc Documentation Status Website

gorse is an offline recommender system backend based on collaborative filtering written in Go.

This project is aim to provide a high performance, easy-to-use, programming language irrelevant recommender micro-service based on collaborative filtering. We could build a simple recommender system on it, or set up a more sophisticated recommender system using candidates generated by it. It features:

  • Implements 7 rating based recommenders and 4 ranking based recommenders.
  • Supports data loading, data splitting, model training, model evaluation and model selection.
  • Provides the data import/export tool, model evaluation tool and RESTful recomender server.
  • Accelerates computations by SIMD instructions and multi-threading.

For more information:

  • Visit GoDoc for detailed documentation of codes.
  • Visit ReadTheDocs for tutorials, examples and usages.
  • Visit SteamLens for a Steam games recommender system based on gorse.

Install

  • Download from release.
  • Build from source:

Install Golang and run go get:

$ go get github.com/zhenghaoz/gorse/...

It will download all packages and build the gorse command line into your $GOBIN path.

If your CPU supports AVX2 and FMA3 instructions, use the avx2 build tag to enable AVX2 and FMA3 instructions.

$ go get -tags='avx2' github.com/zhenghaoz/gorse/...

Usage

gorse is an offline recommender system backend based on collaborative filtering written in Go.

Usage:
  gorse [flags]
  gorse [command]

Available Commands:
  export-feedback Export feedback to CSV
  export-items    Export items to CSV
  help            Help about any command
  import-feedback Import feedback from CSV
  import-items    Import items from CSV
  serve           Start a recommender sever
  test            Test a model by cross validation
  version         Check the version

Flags:
  -h, --help   help for gorse

Use "gorse [command] --help" for more information about a command.

Evaluate a Recommendation Model

gorse provides the tool to evaluate models. We can run gorse test -h or check online documents to learn its usage. For example:

$ gorse test bpr --load-csv u.data --csv-sep $'\t' --eval-precision --eval-recall --eval-ndcg --eval-map --eval-mrr
...
+--------------+----------+----------+----------+----------+----------+----------------------+
|              |  FOLD 1  |  FOLD 2  |  FOLD 3  |  FOLD 4  |  FOLD 5  |         MEAN         |
+--------------+----------+----------+----------+----------+----------+----------------------+
| Precision@10 | 0.321041 | 0.327128 | 0.321951 | 0.318664 | 0.317197 | 0.321196(±0.005931)  |
| Recall@10    | 0.212509 | 0.213825 | 0.213336 | 0.206255 | 0.210764 | 0.211338(±0.005083)  |
| NDCG@10      | 0.380665 | 0.385125 | 0.380003 | 0.369115 | 0.375538 | 0.378089(±0.008974)  |
| MAP@10       | 0.122098 | 0.123345 | 0.119723 | 0.116305 | 0.119468 | 0.120188(±0.003883)  |
| MRR@10       | 0.605354 | 0.601110 | 0.600359 | 0.577333 | 0.599930 | 0.596817(±0.019484)  |
+--------------+----------+----------+----------+----------+----------+----------------------+

u.data is the CSV file of ratings in MovieLens 100K dataset and u.item is the CSV file of items in MovieLens 100K dataset. All CLI tools are listed in the CLI-Tools section of Wiki.

Setup a Recommender Server

It's easy to setup a recomendation service with gorse.

  • Step 1: Import feedback and items.
$ gorse import-feedback ~/.gorse/gorse.db u.data --sep $'\t' --timestamp 2
$ gorse import-items ~/.gorse/gorse.db u.item --sep '|'

It imports feedback and items from CSV files into the database file ~/.gorse/gorse.db. The low level storage engine is implemented by BoltDB.

  • Step 2: Start a server.
$ gorse serve -c config.toml

It loads configurations from config.toml and start a recommendation server. It may take a while to generate all recommendations. Detailed information about configuration is in the Configuration section of Wiki. Before set hyper-parameters for the model, it is useful to test the performance of chosen hyper-parameters by the model evaluation tool.

  • Step 3: Get recommendations.
$ curl 127.0.0.1:8080/recommends/1?number=5

It requests 5 recommended items for the 1-th user. The response might be:

[
    {
        "ItemId": "919",
        "Popularity": 96,
        "Timestamp": "1995-01-01T00:00:00Z",
        "Score": 1
    },
    {
        "ItemId": "474",
        "Popularity": 194,
        "Timestamp": "1963-01-01T00:00:00Z",
        "Score": 0.9486470268850127
    },
    ...
]

"ItemId" is the ID of the item and "Score" is the score generated by the recommendation model used to rank. See RESTful APIs in Wiki for more information about RESTful APIs.

Use gorse in Go

Also, gorse could be imported and used in Go application. There is an example that fits a recommender and generate recommended items:

package main

import (
	"fmt"
	"github.com/zhenghaoz/gorse/base"
	"github.com/zhenghaoz/gorse/core"
	"github.com/zhenghaoz/gorse/model"
)

func main() {
	// Load dataset
	data := core.LoadDataFromBuiltIn("ml-100k")
	// Split dataset
	train, test := core.Split(data, 0.2)
	// Create model
	bpr := model.NewBPR(base.Params{
		base.NFactors:   10,
		base.Reg:        0.01,
		base.Lr:         0.05,
		base.NEpochs:    100,
		base.InitMean:   0,
		base.InitStdDev: 0.001,
	})
	// Fit model
	bpr.Fit(train, nil)
	// Evaluate model
	scores := core.EvaluateRank(bpr, test, train, 10, core.Precision, core.Recall, core.NDCG)
	fmt.Printf("Precision@10 = %.5f\n", scores[0])
	fmt.Printf("Recall@10 = %.5f\n", scores[1])
	fmt.Printf("NDCG@10 = %.5f\n", scores[1])
	// Generate recommendations for user(4):
	// Get all items in the full dataset
	items := core.Items(data)
	// Get user(4)'s ratings in the training dataset
	excludeItems := train.User("4")
	// Get top 10 recommended items (excluding rated items) for user(4) using BPR
	recommendItems, _ := core.Top(items, "4", 10, excludeItems, bpr)
	fmt.Printf("Recommend for user(4) = %v\n", recommendItems)
}

The output should be:

2019/11/14 08:07:45 Fit BPR with hyper-parameters: n_factors = 10, n_epochs = 100, lr = 0.05, reg = 0.01, init_mean = 0, init_stddev = 0.001
2019/11/14 08:07:45 epoch = 1/100, loss = 55451.70899118173
...
2019/11/14 08:07:49 epoch = 100/100, loss = 10093.29427682404
Precision@10 = 0.31699
Recall@10 = 0.20516
NDCG@10 = 0.20516
Recommend for 4-th user = [288 313 245 307 328 332 327 682 346 879]

Recommenders

There are 11 recommendation models implemented by gorse.

Model Data Task Multi-threading Fit
explicit implicit weight rating ranking
BaseLine ✔️ ✔️ ✔️
NMF ✔️ ✔️ ✔️
SVD ✔️ ✔️ ✔️
SVD++ ✔️ ✔️ ✔️ ✔️
KNN ✔️ ✔️ ✔️ ✔️
CoClustering ✔️ ✔️ ✔️ ✔️
SlopeOne ✔️ ✔️ ✔️ ✔️
ItemPop ✔️ ✔️ ✔️
KNN (Implicit) ✔️ ✔️ ✔️ ✔️ ✔️
WRMF ✔️ ✔️ ✔️ ✔️
BPR ✔️ ✔️ ✔️
  • Cross-validation of rating models on MovieLens 1M [Source].
Model RMSE MAE Time (AVX2)
SlopeOne 0.90683 0.71541 0:00:26
CoClustering 0.90701 0.71212 0:00:08
KNN 0.86462 0.67663 0:02:07
SVD 0.84252 0.66189 0:02:21 0:01:48
SVD++ 0.84194 0.66156 0:03:39 0:02:47
  • Cross-validation of ranking models on MovieLens 100K [Source].
Model Precision@10 Recall@10 MAP@10 NDCG@10 MRR@10 Time
ItemPop 0.19081 0.11584 0.05364 0.21785 0.40991 0:00:03
KNN 0.28584 0.19328 0.11358 0.34746 0.57766 0:00:41
BPR 0.32083 0.20906 0.11848 0.37643 0.59818 0:00:13
WRMF 0.34727 0.23665 0.14550 0.41614 0.65439 0:00:14

Performance

gorse is much faster than Surprise, and comparable to librec while using less memory space than both of them. The memory efficiency is achieved by sophisticated data structures.

  • Cross-validation of SVD on MovieLens 100K [Source]:

  • Cross-validation of SVD on MovieLens 1M [Source]:

Contributors

Any kind of contribution is expected: report a bug, give a advice or even create a pull request.

Acknowledgments

gorse is inspired by following projects:

Limitations

gorse has limitations and might not be applicable to some scenarios:

  • No Scalability: gorse is a recommendation service on a single host, so it's unable to handle large data.
  • No Features: gorse exploits interactions between items and users while features of items and users are ignored.
Owner
Zhenghao Zhang
Knowledge is power, France is bacon.
Zhenghao Zhang
Comments
  • Prevent recommending an item for an user who provided feedback

    Prevent recommending an item for an user who provided feedback

    I have been trying the gorse with bpr model to recommend content to users, whom have option to give feedback which are fed into gorse.

    I'm using gorse as standalone web server and accessing with it's own apis.

    What I'm trying to accomplish is preventing the engine from recommending content the user already provided feedback. Bu I have looked both into the docs and the code but nothing really looked like what I wanted. There is somewhat related #33 but it recommends an entry only once and never again.

    Is there any way to accomplish what I want.

  • cdn好像不太稳定

    cdn好像不太稳定

    https://cdn.gorse.io/example/github.sql 经常访问不了,但有时候又可以

    会造成docker安装的时候错误

    [root@loaclhost ~]# docker run -p 8088:8088 zhenghaoz/gorse-in-one --playground Welcome to Gorse v0.4.8 Playground {"level":"fatal","ts":1666740804.338797,"caller":"gorse-in-one/main.go:89","msg":"failed to initialize database","error":"Get \"https://cdn.gorse.io/example/github.sql\": dial tcp 185.183.84.75:443: i/o timeout","errorVerbose":"Get \"https://cdn.gorse.io/example/github.sql\": dial tcp 185.183.84.75:443: i/o timeout\nmain.initializeDatabase:177: ","stacktrace":"main.glob..func1\n\t/go/gorse/cmd/gorse-in-one/main.go:89\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:918\nmain.main\n\t/go/gorse/cmd/gorse-in-one/main.go:144\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}

  • gorse-in-one启动时候还有个panic

    gorse-in-one启动时候还有个panic

    只在启动时候出现, 不过捕获到了... 就是看着难受 -_-! { "level":"error", "ts":1656313095.142161, "caller":"base/util.go:61", "msg":"panic recovered", "panic":"runtime error: index out of range [-1]", "stacktrace":"github.com/zhenghaoz/gorse/base.CheckPanic /Users/ak47/dec/gorse-master/base/util.go:61 runtime.gopanic /usr/local/go/src/runtime/panic.go:838 runtime.goPanicIndex /usr/local/go/src/runtime/panic.go:89 github.com/zhenghaoz/gorse/base/search.(*IVF).Build.func1 /Users/ak47/dec/gorse-master/base/search/ivf.go:205 github.com/zhenghaoz/gorse/base/parallel.Parallel /Users/ak47/dec/gorse-master/base/parallel/parallel.go:31 github.com/zhenghaoz/gorse/base/search.(*IVF).Build /Users/ak47/dec/gorse-master/base/search/ivf.go:192 github.com/zhenghaoz/gorse/base/search.(*IVFBuilder).Build /Users/ak47/dec/gorse-master/base/search/ivf.go:274 github.com/zhenghaoz/gorse/master.(*Master).findItemNeighborsIVF /Users/ak47/dec/gorse-master/master/tasks.go:386 github.com/zhenghaoz/gorse/master.(*Master).runFindItemNeighborsTask /Users/ak47/dec/gorse-master/master/tasks.go:254 github.com/zhenghaoz/gorse/master.(*Master).runRankingRelatedTasks /Users/ak47/dec/gorse-master/master/tasks.go:857 github.com/zhenghaoz/gorse/master.(*Master).RunPrivilegedTasksLoop /Users/ak47/dec/gorse-master/master/master.go:277" }

  • Migration error with PostgresSQL database

    Migration error with PostgresSQL database

    I want to use Gorse with PostgreSQL (latest alpine image). The second and re-launch of containers are performed with the following error in the log.

    master_1    | 2022/05/23 14:58:11 /go/pkg/mod/gorm.io/driver/[email protected]/migrator.go:270 pq: syntax error at or near "not"
    master_1    | [0.374ms] [rows:0] ALTER TABLE "users" ALTER COLUMN "labels" TYPE json not null default '[]'
    postgres_1  | 2022-05-23 14:58:11.740 UTC [33] ERROR:  syntax error at or near "not" at character 53
    postgres_1  | 2022-05-23 14:58:11.740 UTC [33] STATEMENT:  ALTER TABLE "users" ALTER COLUMN "labels" TYPE json not null default '[]'
    master_1    | {"level":"fatal","ts":1653317891.7413962,"caller":"master/master.go:190","msg":"failed to init database","error":"pq: syntax error at or near \"not\"","errorVerbose":"pq: syntax error at or near \"not\"\n/go/gorse/storage/data/sql.go:134: ","stacktrace":"github.com/zhenghaoz/gorse/master.(*Master).Serve\n\t/go/gorse/master/master.go:190\nmain.glob..func1\n\t/go/gorse/cmd/gorse-master/main.go:57\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:883\nmain.main\n\t/go/gorse/cmd/gorse-master/main.go:70\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:250"}
    

    Gorse version I'm tried both nightly and latest tag of docker images.

    My docker-compose.yml file:
    version: "3"
    services:
      redis:
        image: redis
        restart: unless-stopped
        ports:
          - 6379:6379
    
      postgres:
        image: postgres:alpine
        restart: unless-stopped
        ports:
          - 5432:5432
        environment:
          POSTGRES_DB: "gorse"
          POSTGRES_USER: "gorse"
          POSTGRES_PASSWORD: "gorse_pass"
          PGDATA: /data/postgres
        volumes:
          - pg_data:/data/postgres
    
      worker:
        image: zhenghaoz/gorse-worker:nightly
        restart: unless-stopped
        ports:
          - 8089:8089
        command: >
          --master-host master --master-port 8086 
          --http-host 0.0.0.0 --http-port 8089
          --log-path /var/log/gorse/worker.log 
          --cache-path /var/lib/gorse/worker_cache.data
        volumes:
          - gorse_log:/var/log/gorse
          - worker_data:/var/lib/gorse
        depends_on:
          - master
    
      server:
        image: zhenghaoz/gorse-server:nightly
        restart: unless-stopped
        ports:
          - 8087:8087
        environment:
          GORSE_CACHE_STORE: redis://redis:6379
          GORSE_DATA_STORE: postgres://gorse:gorse_pass@postgres:5432/gorse?sslmode=disable
        command: >
          --master-host master --master-port 8086 
          --http-host 0.0.0.0 --http-port 8087
          --log-path /var/log/gorse/server.log 
          --cache-path /var/lib/gorse/server_cache.data
        volumes:
          - gorse_log:/var/log/gorse
          - server_data:/var/lib/gorse
        depends_on:
          - master
    
      master:
        image: zhenghaoz/gorse-master:nightly
        restart: unless-stopped
        ports:
          - 8086:8086
          - 8088:8088
        environment:
          GORSE_CACHE_STORE: redis://redis:6379
          GORSE_DATA_STORE: postgres://gorse:gorse_pass@postgres:5432/gorse?sslmode=disable
        command: >
          -c /etc/gorse/config.toml 
          --log-path /var/log/gorse/master.log 
          --cache-path /var/lib/gorse/master_cache.data
        volumes:
          - ./config.toml:/etc/gorse/config.toml
          - gorse_log:/var/log/gorse
          - master_data:/var/lib/gorse
        depends_on:
          - redis
          - postgres
          
     
    volumes:
      pg_data:
      worker_data:
      server_data:
      master_data:
      gorse_log:
    

    What's wrong?

  • Inserting feedback fails on DB constraint

    Inserting feedback fails on DB constraint

    Gorse version latest

    Describe the bug Whenever I try to insert feedback through API or dashboard, it fails with the message

    Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`","errorVerbose":"Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`
    

    To Reproduce Startup Gorse on clean DB -> Import feedback through dashboard -> On confirm import it fails

    Expected behavior The insert should pass for feedback.

    Additional context I have no problem with inserting users and items. I had the same problem even with items and users imported.

    Stack trace:

    {"level":"error","ts":1634292942.5676055,"caller":"server/rest.go:1323","msg":"internal server error","error":"Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`","errorVerbose":"Error 4025: CONSTRAINT `users.labels` failed for `gorse`.`users`\n/go/gorse/storage/data/sql.go:805: ","stacktrace":"github.com/zhenghaoz/gorse/server.InternalServerError\n\t/go/gorse/server/rest.go:1323\ngithub.com/zhenghaoz/gorse/server.(*RestServer).insertFeedback.func1\n\t/go/gorse/server/rest.go:1153\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/filter.go:21\ngithub.com/zhenghaoz/gorse/server.LogFilter\n\t/go/gorse/server/rest.go:69\ngithub.com/emicklei/go-restful/v3.(*FilterChain).ProcessFilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/filter.go:19\ngithub.com/emicklei/go-restful/v3.(*Container).dispatch\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/container.go:291\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2046\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2424\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2878\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1929"}
    

    Screenshot from 2021-10-15 12-53-25 Screenshot from 2021-10-15 12-53-33

    Gorse config:

    # This section declares settings for the database.
    [database]
    
    cache_store = "redis://localhost:6379"
    
    data_store = "<connection string>"
    
    cache_size = 50
    
    auto_insert_user = true
    
    auto_insert_item = false
    
    positive_feedback_types = ["watch_full"]
    
    click_feedback_types = []
    
    read_feedback_type = "view"
    
    positive_feedback_ttl = 0
    
    item_ttl = 0
    
    [master]
    port = 8086                     # master port
    host = "0.0.0.0"                # master host
    http_port = 8088                # HTTP API port
    http_host = "0.0.0.0"           # HTTP API host
    n_jobs = 4                      # number of working jobs
    meta_timeout = 10               # cluster meta timeout (second)
    
    [server]
    default_n = 20                  # default number of returned items
    api_key = "<API-key>"  # secret key for RESTful APIs (SSL required)
    
    [recommend]
    
    popular_window = 7
    
    fit_period = 10
    
    search_period = 60
    
    search_epoch = 100
    
    search_trials = 10
    
    refresh_recommend_period = 1
    
    fallback_recommend = ["item_based", "popular"]
    explore_latest_num = 20
    
    item_neighbor_type = "auto"
    
    user_neighbor_type = "auto"
    
    enable_latest_recommend = true
    
    enable_popular_recommend = true
    
    enable_user_based_recommend = true
    
    enable_item_based_recommend = true
    
    enable_collaborative_recommend = true
    
  • Feedbacks import problem

    Feedbacks import problem

    Gorse version last version

    Describe the bug I imported my feedbacks in the Import feedbacks window on Gorse. However, in the Gorse Dashboard Overview I do not have any feedback displayed. Moreover, then, I exported the feedbacks from Gorse and there is literally nothing in there. It is as if Gorse fakes to import the feedbacks, but in reality it does not. I don't think it's a TimeStamp Type issue (but maybe it is), as I tried with many different types and it still doesn't work.

    Even when I try with a csv file of 5 lines like this one it does not work : favori,UserId,ItemId,Timestamp favori,3423,186758,2022-06-24 21:20:11 favori,3423,186758,2022-06-24 21:20:13 favori,2823,186085,2022-06-24 21:20:19 favori,2140,185965,2022-06-24 21:20:20

    feedback_jeudi.csv

    Can you help me please ?

  • [solved] Extend the api to get last viewed items

    [solved] Extend the api to get last viewed items

    I know the recommender is responsible for making recommends, but I think it would be useful to get access to the last viewed items over the api. This should be easily made.

    All the needed data is already collected for gorse. So it would be nice to get access to the last 10 viewed items, instead of collecting the same data again.

    I know this is not the primary goal of you recommender, but I guess this is also a typical use case for others.

  • Memory build-up on Redis

    Memory build-up on Redis

    Gorse version 0.3.4

    Describe the bug I've been noticing a memory build-up (probably a leakage) on the Gorse Database used on Redis. Not sure why but the memory has been slowly building until it reaches 100%.

    To Reproduce Let gorse running for a few days and memory should hit 100%.

  • valid positive 0

    valid positive 0

    0 valid positive

    Gorse version

    zhenghaoz/gorse-master:latest

    Describe the bug

    Status from the overview:

    USERS 201,393 ITEMS 402 TOTAL POSITIVE 58,548 VALID POSITIVE 0 VALID NEGATIVE 0

    Search click-through rate prediction model | Failed | 0000/12/31 20:06 |   | No feedback found. -- | -- | -- | -- | -- Fit collaborative filtering model | Failed | 0000/12/31 20:06 |   | No feedback found. Fit click-through rate prediction model | Failed | 0000/12/31 20:06 |   | No feedback found. Search collaborative filtering model | Failed | 0000/12/31 20:06 |   | No feedback found.

    positive feedback in config.toml: positive_feedback_types = ["contact"] I can see just item-related recommendations

    To Reproduce

    config.yml:

    # This section declares settings for the database.
    [database]
    # The database for caching, support Redis only:
    #   redis://<user>:<password>@<host>:<port>/<db_number>
    cache_store = "redis://redis:6380"
    # The database for persist data, support MySQL, Postgres, ClickHouse and MongoDB:
    #   mysql://[username[:password]@][protocol[(address)]]/dbname[?param1=value1&...&paramN=valueN]
    #   postgres://bob:[email protected]:5432/mydb?sslmode=verify-full
    #   clickhouse://user:password@host[:port]/database?param1=value1&...&paramN=valueN
    #   mongodb://[username:password@]host1[:port1][,...hostN[:portN]][/[defaultauthdb][?options]]
    data_store = "mysql://gorse:gorse_pass@tcp(mysql:3306)/gorse?parseTime=true"
    # The cache size for recommended/popular/latest items. The default value is 100.
    cache_size = 200
    # Insert new users while inserting feedback. The default value is true.
    auto_insert_user = true
    # Insert new items while inserting feedback. The default value is true.
    auto_insert_item = true
    # The feedback types for positive events.
    positive_feedback_types = ["contact"]
    # The feedback types for read events.
    read_feedback_types = ["read"]
    # The time-to-live (days) of positive feedback, 0 means disabled. The default value is 0.
    positive_feedback_ttl = 0
    # The time-to-live (days) of items, 0 means disabled. The default value is 0.
    item_ttl = 0
    # This section declares settings for the master node.
    [master]
    port = 8086                     # master port
    host = "0.0.0.0"                # master host
    http_port = 8088                # HTTP API port
    http_host = "0.0.0.0"           # HTTP API host
    n_jobs = 4                      # number of working jobs
    meta_timeout = 10               # cluster meta timeout (second)
    # This section declares settings for the server node.
    [server]
    default_n = 20                  # default number of returned items
    api_key = ""                    # secret key for RESTful APIs (SSL required)
    # This section declares settings for recommendation.
    [recommend]
    # The time window of popular items (days). The default values is 180.
    popular_window = 30
    # The time period for model fitting (minutes). The default values is 60.
    fit_period = 10
    # The time period for model searching (minutes). The default values is 100.
    search_period = 60
    # The number of epochs for model searching. The default values is 100.
    search_epoch = 100
    # The number of trials for model searching. The default values is 10.
    search_trials = 10
    # The time period to refresh recommendation for inactive users (days). The default values is 5.
    refresh_recommend_period = 1
    # The fallback recommendation method is used when cached recommendation drained out:
    #   item_based: Recommend similar items to cold-start users.
    #   popular: Recommend popular items to cold-start users.
    #   latest: Recommend latest items to cold-start users.
    # Recommenders are used in order. The default values is ["latest"].
    fallback_recommend = ["item_based", "latest"]
    # The type of neighbors for items. There are three types:
    #   similar: Neighbors are found by number of common labels.
    #   related: Neighbors are found by number of common users.
    #   auto: If a item have labels, neighbors are found by number of common labels.
    #         If this item have no labels, neighbors are found by number of common users.
    # The default values is "auto".
    item_neighbor_type = "similar"
    # The type of neighbors for users. There are three types:
    #   similar: Neighbors are found by number of common labels.
    #   related: Neighbors are found by number of common liked items.
    #   auto: If a user have labels, neighbors are found by number of common labels.
    #         If this user have no labels, neighbors are found by number of common liked items.
    # The default values is "auto".
    user_neighbor_type = "similar"
    # Enable latest recommendation during offline recommendation. The default values is false.
    enable_latest_recommend = true
    # Enable popular recommendation during offline recommendation. The default values is false.
    enable_popular_recommend = true
    # Enable user-based similarity recommendation during offline recommendation. The default values is false.
    enable_user_based_recommend = true
    # Enable item-based similarity recommendation during offline recommendation. The default values is false.
    enable_item_based_recommend = true
    # Enable collaborative filtering recommendation during offline recommendation. The default values is true.
    enable_collaborative_recommend = true
    # Enable click-though rate prediction during offline recommendation. Otherwise, results from multi-way recommendation
    # would be merged randomly. The default values is true.
    enable_click_through_prediction = true
    # The explore recommendation method is used to inject popular items or latest items into recommended result:
    #   popular: Recommend popular items to cold-start users.
    #   latest: Recommend latest items to cold-start users.
    # Recommenders are used in order. The default values is { popular = 0.0, latest = 0.0 }.
    explore_recommend = { popular = 0.1, latest = 0.2 }
    

    docker-compose.yaml

    version: "3"
    services:
      redis:
        image: redis
        restart: unless-stopped
        ports:
          - 6380:6380
      mysql:
        image: mysql/mysql-server
        restart: unless-stopped
        ports:
          - 3306:3306
        environment:
          MYSQL_ROOT_PASSWORD: root_pass
          MYSQL_DATABASE: gorse
          MYSQL_USER: gorse
          MYSQL_PASSWORD: gorse_pass
        volumes:
          - ./var/lib/mysql:/var/lib/mysql
      worker:
        image: zhenghaoz/gorse-worker
        restart: unless-stopped
        ports:
          - 8089:8089
        command: >
          --master-host master --master-port 8086 --http-host 0.0.0.0 --http-port 8089
          --log-path /var/log/gorse/worker.log --cache-path /var/lib/gorse/worker_cache.data
        volumes:
          - ./var/log/gorse:/var/log/gorse
          - ./var/lib/gorse:/var/lib/gorse
      server:
        image: zhenghaoz/gorse-server
        restart: unless-stopped
        ports:
          - 8087:8087
        command: >
          --master-host master --master-port 8086 --http-host 0.0.0.0 --http-port 8087
          --log-path /var/log/gorse/server.log --cache-path /var/lib/gorse/server_cache.data
        volumes:
          - ./var/log/gorse:/var/log/gorse
          - ./var/lib/gorse:/var/lib/gorse
      master:
        image: zhenghaoz/gorse-master
        restart: unless-stopped
        ports:
          - 8086:8086
          - 8088:8088
        command: -c /etc/gorse/config.toml --log-path /var/log/gorse/master.log --cache-path /var/lib/gorse/master_cache.data
        volumes:
          - ./config.toml:/etc/gorse/config.toml
          - ./var/log/gorse:/var/log/gorse
          - ./var/lib/gorse:/var/lib/gorse
    

    (simplified) data

    feedback.sql

    DROP TABLE IF EXISTS `feedback`;
    CREATE TABLE `feedback` (
      `feedback_type` varchar(256) NOT NULL,
      `user_id` integer NOT NULL,
      `item_id` integer NOT NULL,
      `time_stamp` timestamp NOT NULL,
      `comment` text NOT NULL,
      PRIMARY KEY (`feedback_type`,`user_id`,`item_id`),
      KEY `user_id` (`user_id`)
    ); 
    LOCK TABLES `feedback` WRITE;
    INSERT INTO `feedback` VALUES
    ('contact', 9285, 2, '2013-01-21 18:47:08', ''),('contact', 9521, 2, '2013-04-22 14:21:10', ''),('contac
    t', 12614, 2, '2013-04-22 14:21:10', ''),('contact', 39577, 2, '2013-04-01 18:47:47', ''),('contact', 42
    083, 2, '2013-04-08 17:28:57', '');UNLOCK TABLES;
    

    users.sql

    DROP TABLE IF EXISTS `users`;
    
    CREATE TABLE `users` (
      `user_id` integer NOT NULL,
      `labels` json NOT NULL,
      `comment` text NOT NULL,
      `subscribe` json NOT NULL,
      PRIMARY KEY (`user_id`)
    ) ;
    LOCK TABLES `users` WRITE;
    INSERT INTO `users` VALUES
    (111830, '["Uruguay", "Montevideo", "Pocitos"]', '', '[]'),(113915, '["Uruguay", "Montevideo", "Pocitos"
    ]', '', '[]'),(119975, '["Uruguay", "Montevideo", "Pocitos"]', '', '[]'),(121938, '["Uruguay", "Montevid
    eo", "Pocitos"]', '', '[]');UNLOCK TABLES;
    

    items.sql

    DROP TABLE IF EXISTS `items`;
    CREATE TABLE `items` (
      `item_id` integer NOT NULL,
      `time_stamp` timestamp NOT NULL,
      `labels` json NOT NULL,
      `comment` text NOT NULL,
      `is_hidden` tinyint(1) NOT NULL DEFAULT '0',
      `categories` json NOT NULL,
      PRIMARY KEY (`item_id`)
    ) ;
    
    LOCK TABLES `items` WRITE;
    INSERT INTO `items` VALUES 
    (1257, '2013-09-27 02:10:06', '["Uruguay", "Montevideo", "Pocitos"]', '', 0, '[]'),(2814, '2014-10-09 13
    :26:37', '["amoblado", "amenities", "encargado", "portero", "vigilancia", "seguridad", "Argentina", "Bs.
    As. G.B.A. Zona Norte", "Pilar", "La Lonja"]', '', 0, '[]');UNLOCK TABLES;
    

    Expected behavior

    recommendations

    Additional context

    logs

    master.log

    {"level":"info","ts":1638877759.9023194,"caller":"master/tasks.go:60","msg":"load dataset","positive_feedback_types":["contact"],"read_feedback_types":["read"],"item_ttl":0,"feedback_ttl":0,
    "positive_feedback_types":["contact"]}
    
    
  • pq: sorry, too many clients already

    pq: sorry, too many clients already

    Hi, I have updated my gorse version to latest v0.2.5, along with the latest config.toml . Now we are getting below error after the application runs for few time:-

    {"level":"error","ts":1631013939.8973596,"caller":"server/rest.go:1252","msg":"internal server error","error":"pq: sorry, too many clients already","errorVerbose":"pq: sorry, too many clients already\n/go/gorse/storage/data/sql.go:226: ","stacktrace":"github.com/zhenghaoz/gorse/server.internalservererror\n\t/go/gorse/server/rest.go:1252\ngithub.com/zhenghaoz/gorse/server.(*restserver).getmeasurements\n\t/go/gorse/server/rest.go:1234\ngithub.com/emicklei/go-restful/v3.(*filterchain).processfilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/[email protected]/filter.go:21\ngithub.com/zhenghaoz/gorse/server.logfilter\n\t/go/gorse/server/rest.go:71\ngithub.com/emicklei/go-restful/v3.(*filterchain).processfilter\n\t/go/pkg/mod/github.com/emicklei/go-restful/v3:1932"}
    {"level":"info","ts":1631013939.897418,"caller":"server/rest.go:74","msg":"GET /api/measurements/ClickThroughRate?n=30","status_code":500}
    

    I have used the docker implementation with Postgres as backend database. Thanks.

  • Information regarding valid time formats.

    Information regarding valid time formats.

    Hi, I recieve errors in the form of 'failed to parse datetime `` at line x' in the console while loading csv's. Could you point me to the valid datetime formats acceptable for timestamps in the csv file? Any help would be appreciated.

    Thanks, Avinash.

  • Data race: worker.Sync() and worker.Recommend()

    Data race: worker.Sync() and worker.Recommend()

    Gorse version Version: unknown-version API version: v0.2.7 Go version: go1.19.3 Git commit: 762f21f3154f35877fe376ba435c11e7eaaf5edc

    Describe the bug A data race occurs when the worker configuration is updated. Sync() write w.Config, and Recommend() read.

    Perhaps it is better to sync the configuration only between reranking cycles? Because the sync operation is much faster than the calculation of recommendations, as well as changing the configuration parameters during the ranking process can lead to incorrect result.

    To Reproduce Test go test -v -race github.com/zhenghaoz/gorse/worker -run TestWorker_SyncRecommend

    Expected behavior no data race

  • More label types

    More label types

    Currently, only string labels are supported. This makes it impossible to have numeric relations like a price. ("User 1 likes Items that cost below 300 $")

    It would be helpful to have more types of labels like numeric (i.e. prices) and dates (i.e. event dates).

    I helped myself by having price ranges as string labels (i.e. "200-300$). This is a good workaround but does not fully solve the problem.

  • Filter possible items for recommendation

    Filter possible items for recommendation

    Currently, the only possibility to filter items for a recommendation is the use of categories. Only one category can be used as a filter. I.e. this scenario in an e-com shoe store is not possible: "Recommend shoes from category sneaker with colors black, white or gray."

    I think it would be helpful to have the opportunity to filter by multiple categories, as well as by certain labels.

  • implement queue in data storage

    implement queue in data storage

    Background

    In Gorse v0.4.x, neighbors, and models are not updated until the next round of full dataset loading. In practice, it would be better to update neighbors or models once a new user, item, or feedback is inserted. A queue is required to achieve this goal.

    Design

    type Database interface {
    
           // Enqueue put the value to the back of a named queue.
           Enqueue(name, value string) error
    
           // Dequeue remove the front value of a named queue and return this value. 
           // If no value in the queue, return io.EOF.
           Dequeue(name string) (value string, err error) 
    }
    
  • support more APIs in Go client

    support more APIs in Go client

A Kubernetes Native Batch System (Project under CNCF)
A Kubernetes Native Batch System (Project under CNCF)

Volcano is a batch system built on Kubernetes. It provides a suite of mechanisms that are commonly required by many classes of batch & elastic workloa

Jan 9, 2023
Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.
Learn how to design large-scale systems. Prep for the system design interview.  Includes Anki flashcards.

English ∙ 日本語 ∙ 简体中文 ∙ 繁體中文 | العَرَبِيَّة‎ ∙ বাংলা ∙ Português do Brasil ∙ Deutsch ∙ ελληνικά ∙ עברית ∙ Italiano ∙ 한국어 ∙ فارسی ∙ Polski ∙ русский язы

Jan 9, 2023
Go implementation of the yolo v3 object detection system
Go implementation of the yolo v3 object detection system

Go YOLO V3 This repository provides a plug and play implementation of the Yolo V3 object detection system in Go, leveraging gocv. Prerequisites Since

Dec 14, 2022
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.
PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage.

中文 | English PaddleDTX PaddleDTX is a solution that focused on distributed machine learning technology based on decentralized storage. It solves the d

Dec 14, 2022
Derivative microservices based in Go.

Derivative Microservices Essentially this repository contains a re-write of the Islandora microserives: houdini, homarus, hypercube, and FITS (a TODO)

Dec 17, 2021
small go-based app for note taking

Description A small note taking web-app which I am building for myself just for fun and learning purposes. My goal is too use as much of the standard

Oct 8, 2022
Genetic Algorithms library written in Go / golang

Description Genetic Algorithms for Go/Golang Install $ go install git://github.com/thoj/go-galib.git Compiling examples: $ git clone git://github.com

Sep 27, 2022
Neural Networks written in go

gobrain Neural Networks written in go Getting Started The version 1.0.0 includes just basic Neural Network functions such as Feed Forward and Elman Re

Dec 20, 2022
A Naive Bayes SMS spam classifier written in Go.
A Naive Bayes SMS spam classifier written in Go.

Ham (SMS spam classifier) Summary The purpose of this project is to demonstrate a simple probabilistic SMS spam classifier in Go. This supervised lear

Sep 9, 2022
k-means clustering algorithm implementation written in Go
k-means clustering algorithm implementation written in Go

kmeans k-means clustering algorithm implementation written in Go What It Does k-means clustering partitions a multi-dimensional data set into k cluste

Dec 6, 2022
a* pathfinding algorithm written in go

astar a* (a-star) pathfinding algorithm written in go Wikipedia: EN: A* search algorithm DE: A*-Algorithmus Install go get github.com/jpierer/astar@ma

Mar 21, 2022
A simple utility, written in Go, for interacting with Salesforce.

Salesforce CLI A simple utility, written in Go, for interacting with Salesforce. Currently only specific functionality is implemented, and the output

Dec 14, 2021
A simple yet customisable program written in go to make hackerman-like terminal effects.
A simple yet customisable program written in go to make hackerman-like terminal effects.

stuntman a simple program written in go to make you look like a hackerman Demo stuntman -binar -width 90 -color cyan stuntman -text -width 90 -vertgap

Aug 4, 2022
Suricate-bank - API to transfer money between accounts at Suricate Bank,written in Go

⚠️ WORK IN PROGRESS ⚠️ Suricate Bank is an api that creates accounts and transfe

Oct 8, 2022
ncurses matrix/log app written in go to visualize chess problems.

dorrella/matrix-curses Matrix using ncurses and gbin/goncurses. Visual matrix based puzzles Install need libncurses-dev. Probably hard to run on windo

Jan 12, 2022
An open source recommender system service written in Go
An open source recommender system service written in Go

gorse: Go Recommender System Engine Gorse is an open-source recommendation system written in Go. Gorse aims to be a universal open-source recommender

Jan 1, 2023
Collaborative Filtering (CF) Algorithms in Go!

Go Recommend Recommendation algorithms (Collaborative Filtering) in Go! Background Collaborative Filtering (CF) is oftentimes used for item recommenda

Dec 28, 2022
Stackoverflow-Tag-Recommender - Calculate similarity of different tags based on the data of stackoverflow

Recommender System with Stackoverflow's Data This project is written to recommen

Apr 4, 2022
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Jan 8, 2023
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Jan 2, 2023