A possible solution for issue #24.

• Adds an Attribute interface and the CategoricalAttribute and FloatAttribute implementations ** CategoricalAttribute is for discrete string values ** FloatAttribute is for numeric values
• Instances Combines an Attributes slice and storage for a self-contained dataset representation.
• KNNClassifier has been modified to use Attribute and Instance types ** See the test file for use of the API
• CSV handling needed to be moved back into base due to a circular dependency ** Also adds the datasets used to test CSV handling
• Doesn't get rid of skelterjohn/go.matrix dependency yet because the SwapRows() method has no equivalent in the new library.

When training a random forest, I see vastly different, and poorer results when using golearn, than when using Python's scikit-learn. Unfortunately the dataset is confidential, so I can't share it here online. However, I'm using the same train/test split, and have ensured that data is represented the same way (they're all floats in both).

When using scikit learn

Auc: 0.943867958106
Confusion Matrix
[[35878  1876]
 [ 5402 16388]]
             precision    recall  f1-score   support

          0       0.87      0.95      0.91     37754
          1       0.90      0.75      0.82     21790

avg / total       0.88      0.88      0.88     59544

When using golearn, with the same number of estimators..

Reference Class True Positives  False Positives True Negatives  Precision   Recall  F1 Score
--------------- --------------  --------------- --------------  ---------   ------  --------
1.00        4199        1366        15858       0.7545      0.4263  0.5448
0.00        15858       5651        4199        0.7373      0.9207  0.8188
Overall accuracy: 0.7408

As you can see, there's a big drop in precision and recall, on both outcomes. Any ideas as to what could be the problem @Sentimentron ?

Trying to benchmark the k-NN Classifier with a large dataset with a large number of features. I'm using this in particular:

http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass.html#aloi

Had to do a fair bit of yak shaving to get edf to work:

• The math here might be off? When the numbers are small there aren't likely to be problems, but when rowsPerPage is about 3.96 and you have 108,000 rows, the lack-of-rounding error compounds and you end up not asking for enough pages. I think rowsPerPage needs to be math.Floored.
• extend in alloc.go doesn't appear to consider edfAnonMode. I get a panic on line 21 because e.f is nil. I extracted a fileSize variable which I set to os.Getpagesize() in case we're in edfAnonMode. Few issues here: (a) f isn't a great name for a struct field, (b) my solution was hacky, not sure where else various modes aren't being considered, but also don't want to riddle the package with switches on mode, (c) not sure why os.Getpagesize() is the correct value, I copied it from map.go, but this value should probably be extracted as a constant somewhere.
• The purpose of the startBlock == 0 guard in AllocPages isn't clear, but I found I needed to move e.extend(pagesRequested) up before the if block.
• I tried to the String function in fixed.go blowing up when I tried to use it for some debugging output. It assumes f.alloc has non-zero length, and that f.alloc[0] has length at least 61. Got index out of range panics. Not sure whether these assumptions are supposed to hold and weren't for some other reason that's broken, so I just worked around it by removing anything related to alloc from the Sprintf interpolation arguments, and so I got rid of the if altogether. The source of the 61 number wasn't clear.

I didn't want to submit a PR with these fixes because (a) the tests don't give me a lot of confidence that the changes haven't broken something else, (b) lot of my changes felt hacky, and feels like the right solution would involve heavier refactoring.

Couple big-picture questions about EDF:

• What's its origin story? What problem is it trying to solve? My vague guess is to give some sort of common interface for the internal data structures so that one can have an in-memory backing or a file-system backing, presumably to facilitate something like an HDFS-backed distributed computing scenario? Why else would file-system backing be desirable?
• Why is it in the golearn repo? Seems like it's its own project.
• It's incredibly sloooow. My dataset is 108,000 rows, 128 float attributes, and a single label class which can take on one of about 1000 labels. I randomly split the dataset roughly 60/40 into a training and validation set, and saved off those CSVs ahead of time (so I'm not using the testTrainSplit function). For each of those files, the ParseCSVToInstances takes about 30s, never mind actually doing the kNN classification (literally on the order of a few years, made some tweaks but still on the order of an hour). For comparison I have code that parses both CSVs and does the kNN-classification in about 30s total. Hard to pinpoint where all the slowness is coming from, but seems scary.

If I wrap the code inside TestRandomForest1 inside a 10-iteration for loop, I get the following panic:

panic: cannot allocate memory

I'm running this on an m3.2xlarge EC2 instance (Linux 3.13.0-32-generic x86_64). The first 8 or so iterations are likely to succeed, the problem almost always occurs when iterating 10 or more times.

Other things to note: If I increase the forest size to something like 50, then 2-3 iterations suffice to cause the panic. Also, if I remove the call to rf.Predict(testData) (and all subsequent code depending on the result of that Predict), then the panics do not occur.

Here's the full output, starting from the panic (not including the output of the first few iterations that succeed):

panic: cannot allocate memory

goroutine 184 [running]:
runtime.panic(0x5851e0, 0xc)
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/runtime/panic.c:279 +0xf5
github.com/sjwhitworth/golearn/base.NewDenseInstances(0xc2083f6b00)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/base/dense.go:29 +0x67
github.com/sjwhitworth/golearn/base.GeneratePredictionVector(0x7fd6cf13a9f8, 0xc2083f6b00, 0x0, 0x0)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/base/util_instances.go:16 +0xa3
github.com/sjwhitworth/golearn/trees.(*DecisionTreeNode).Predict(0xc2082189b0, 0x7fd6cf13a9f8, 0xc2083f6b00, 0x0, 0x0)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/trees/id3.go:199 +0xad
github.com/sjwhitworth/golearn/trees.(*ID3DecisionTree).Predict(0xc20807bc40, 0x7fd6cf13a9f8, 0xc2083f6b00, 0x0, 0x0)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/trees/id3.go:278 +0x51
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:143 +0x140
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 16 [chan receive]:
testing.RunTests(0x601658, 0x69c010, 0x1, 0x1, 0x575401)
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/testing/testing.go:505 +0x923
testing.Main(0x601658, 0x69c010, 0x1, 0x1, 0x6a8700, 0x0, 0x0, 0x6a8700, 0x0, 0x0)
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/testing/testing.go:435 +0x84
main.main()
    github.com/sjwhitworth/golearn/ensemble/_test/_testmain.go:47 +0x9c

goroutine 19 [finalizer wait]:
runtime.park(0x413090, 0x6a4ef8, 0x6a3a09)
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x6a4ef8, 0x6a3a09)
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/runtime/proc.c:1445

goroutine 20 [runnable]:
github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict(0xc208818380, 0x7fd6cf13a9f8, 0xc208818340, 0x0, 0x0)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:154 +0x2cf
github.com/sjwhitworth/golearn/ensemble.(*RandomForest).Predict(0xc20807bc20, 0x7fd6cf13a9f8, 0xc208818340, 0x0, 0x0)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/ensemble/randomforest.go:45 +0x51
github.com/sjwhitworth/golearn/ensemble.TestRandomForest1(0xc208050090)
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/ensemble/randomforest_test.go:29 +0x1a3
testing.tRunner(0xc208050090, 0x69c010)
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/testing/testing.go:422 +0x8b
created by testing.RunTests
    /home/ubuntu/.gvm/gos/go1.3/src/pkg/testing/testing.go:504 +0x8db

goroutine 191 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 190 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 189 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 188 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 187 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 186 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 185 [runnable]:
github.com/sjwhitworth/golearn/meta.func·004()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:138
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:149 +0x25b

goroutine 183 [chan receive]:
github.com/sjwhitworth/golearn/meta.func·003()
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:113 +0x8c
created by github.com/sjwhitworth/golearn/meta.(*BaggedModel).Predict
    /home/ubuntu/.gvm/pkgsets/go1.3/global/src/github.com/sjwhitworth/golearn/meta/bagging.go:131 +0x175
exit status 2

This of course makes it hard to write benchmarking tests which expect to be able to execute the code multiple times in order to time it.

When installing everything from scratch on my Raspberry Pi, I get this error when trying to run the tests.

# github.com/riobard/go-mmap
../riobard/go-mmap/mmap_linux.go:8: undefined: syscall.MAP_32BIT
../riobard/go-mmap/mmap_linux.go:8: const initializer (<T>)(syscall.MAP_32BIT) is not a constant
../riobard/go-mmap/mmap_linux.go:17: undefined: syscall.MAP_STACK
../riobard/go-mmap/mmap_linux.go:17: const initializer (<T>)(syscall.MAP_STACK) is not a constant
../riobard/go-mmap/mmap_linux.go:18: undefined: syscall.MAP_HUGETLB
../riobard/go-mmap/mmap_linux.go:18: const initializer (<T>)(syscall.MAP_HUGETLB) is not a constant

Any ideas @Sentimentron?

This patch implements equal-width histogram binning and Chi-merge discretisation for converting FloatAttributes attributes into CategoricalAttributes for use with ID3 decision trees, naive Bayes etc.

This patch

• Adds the meta sub-package which implements bagging
• Adds RandomForest in the ensemble sub-package
• Adds ID3DecisionTree and RandomTree to the trees sub-package

ID3DecisionTree performs similarly to the reference, RandomForest performance is competitive with (but a little lower than) the equivalent WEKA classifier.

Haven't finished yet, just open for viewing difference between branches.

• [x] Implement polynomial kernel
• [x] Implement euclidean distance (2 norm)
• [x] Implement L1 norm
• [x] Implement RBF kernel

Update

Some more TODOs:

• [ ] Extend pairwise metrics to accept matrix input
• [x] Maybe integrate all kernels into one file if the code isn't too big
• [x] Doc the kernel formulas, so others can know which kernel we are using.

I reckon it's time to give this package some love again, and lots of people have been asking for this capability (see #187, #159, should help get #188 started). It's taken a long time because I wasn't sure of the right way to do it, but this change extends base.Classifier with a few new methods:

• Save(filePath string) error - saves a Classifier to the filesystem.
• Load(filePath string) error - reloads a Classifier from the filesystem (see caveats).
• LoadWithPrefix/SaveWithPrefix
• GetMetadata - returns basic information about a classifier.

The caveats are that if you're using an ensemble classifier (e.g. one-vs-all), you have to manually add the same number of unfitted classifiers into it before you call Load, otherwise odd stuff will happen. This is because otherwise the ensemble classifiers would have to know about every other Classifier, including any custom ones you've written. The LoadWithPrefix/SaveWithPrefix methods are used by ensemble classifiers so everything can share the same file.

Another caveat: validation could be better, there will be bugs, don't use this in a security-sensitive context (e.g. allowing users to upload pre-trained models), plus the format is unstable and may change at any time.

You can Save and Load:

• MultiLinearSVC
• RandomForest
• KNN
• LinearSVC
• BaggedModel
• OneVsAllModel
• BernoulliNBClassifier
• `ID3DecisionTree
• RandomTree

The Go blas package requires libcblas to be present on the system. Travis environments run Ubuntu Linux so we need to install libatlas-base-dev.

Signed-off-by: Pekka Enberg [email protected]

Call me crazy, but BatchGradientDescent doesn't find the min, nor the argmin, as the GradientDescent part of the function name and the optimisation package name would suggest. It also doesn't do parameter estimation as the tests suggest.

I actually don't know what "parameter estimation" even means in this context, but I'm guessing that, assuming y = a1*x1 + a2*x2 + ... + an*xn (dot product), it attempts to figure out that the linear coefficient parameters a1, ..., an are, given a bunch of observed values y and observed tuples (x1, ..., xn).

Can someone enlighten me on what this code does and/or is supposed to do?

Hi, I am trying to run a simple example and I'm facing the error below.

❯ go run cmd/main.go

github.com/sjwhitworth/golearn/knn

../../../go/pkg/mod/github.com/sjwhitworth/[email protected]/knn/knn.go:132:16: KNN.optimisedEuclideanPredict undefined (type *KNNClassifier has no field or method optimisedEuclideanPredict)

❯ go version go version go1.19 linux/amd64

❯ cat go.mod module localhost/mlpipeline

go 1.19

require github.com/sjwhitworth/golearn v0.0.0-20211014193759-a8b69c276cd8

require ( github.com/gonum/blas v0.0.0-20181208220705-f22b278b28ac // indirect github.com/gonum/lapack v0.0.0-20181123203213-e4cdc5a0bff9 // indirect github.com/gonum/matrix v0.0.0-20181209220409-c518dec07be9 // indirect github.com/google/go-cmp v0.4.0 // indirect github.com/guptarohit/asciigraph v0.5.1 // indirect github.com/mattn/go-runewidth v0.0.7 // indirect github.com/olekukonko/tablewriter v0.0.4 // indirect github.com/rocketlaunchr/dataframe-go v0.0.0-20201007021539-67b046771f0b // indirect golang.org/x/exp v0.0.0-20200331195152-e8c3332aa8e5 // indirect golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a // indirect gonum.org/v1/gonum v0.8.1 // indirect )

... code ...

// KNN classifier
cls := knn.NewKnnClassifier("euclidean", "linear", 2)

Error in download golearn with 'go get ...' (go1.8.4)

$ go get github.com/sjwhitworth/golearn@latest                       
go: github.com/sjwhitworth/[email protected] requires
        github.com/rocketlaunchr/[email protected] requires
        github.com/blend/[email protected]: reading github.com/blend/go-sdk/go.mod at revision v1.1.1: unknown revision v1.1.1

Maybe the vision1.1.1 of go-sdk is too old to use. I don't know how to download now!

Hello, I have an experimental high performance XGBoost (tree_method=exact only) implementation here:

https://github.com/Statfactory/cortado (python + llvm) https://github.com/Statfactory/cortado-fs (F#) https://github.com/Statfactory/JuML.jl (Julia)

I could port it to golang with some help if there is interest:)

Adam

I'm trying to train/predict iris.csv and got problem that can not pass string attributes into Fit since CategoryAttribute is not FloatAttribute. I want to know how to convert "Iris-setosa", "Iris-versicolor" and "Iris-virsinica" into 0, 1, 2 or (0.0, 0.3333, 0.6666). Could you please let me know the way?

package main

import (
	"fmt"
	"log"

	"github.com/sjwhitworth/golearn/base"
	"github.com/sjwhitworth/golearn/evaluation"
	"github.com/sjwhitworth/golearn/linear_models"
)

func main() {
	inst, err := base.ParseCSVToInstances("datasets/iris.csv", false)
	if err != nil {
		log.Fatal(err)
	}

	lr, err := linear_models.NewLogisticRegression("l2", 1.0, 1e-6)
	if err != nil {
		log.Fatal(err)
	}

	trainData, testData := base.InstancesTrainTestSplit(inst, 0.50)

	lr.Fit(trainData)

	pred, err := lr.Predict(testData)
	if err != nil {
		log.Fatal(err)
	}
	confusionMat, err := evaluation.GetConfusionMatrix(testData, pred)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(evaluation.GetSummary(confusionMat))
}