It may be possible that the amount of segments is lower than 3 which caused a runtime panic in the license loader. We now skip those paths to fix that issue.

Found: https://github.com/cri-o/cri-o/runs/7037925519

I've occasionally been seeing the following panic when running identity_license:

panic: send on closed channel

goroutine 51 [running]:
github.com/google/licenseclassifier/tools/identify_license/backend.(*ClassifierBackend).ClassifyLicenses.func1.1(0xc0043a1910, 0xc003466900)
        licenseclassifier/tools/identify_license/backend/backend.go:82 +0x4e
github.com/google/licenseclassifier/tools/identify_license/backend.(*ClassifierBackend).ClassifyLicenses.func1(0x7ffeefbff9b9, 0x13)
        licenseclassifier/tools/identify_license/backend/backend.go:87 +0xb8
created by github.com/google/licenseclassifier/tools/identify_license/backend.(*ClassifierBackend).ClassifyLicenses
        licenseclassifier/tools/identify_license/backend/backend.go:92 +0x1ea

The commit message explains this issue and the fix:

The analyze function calls Done on the waitgroup and then opens up
a new slot by writing into the `task` channel. If this is the last
goroutine running, this can cause an issue: there is a goroutine
running that is waiting for the waitgroup to yield. After this it
closes the `task` channel. If it gets closed before the
`analyze` goroutine manages to send true into the channel, a panic
can occur.

The channel send and `wg.Done()` call could be re-ordeded, but also
there is no strict need to close the channel in this instance, as
there is no receiver for it after this point.

Happy to adjust to something else if there are different opinions on how to deal with this.

Fixed comment to work when using go get command

Currently when using go get -u github.com/google/licenseclassifier/... I get the following error:

/opt/gocode/pkg/mod/github.com/google/[email protected]/licenses/embed.go:8:12: pattern *.db: no matching files found /opt/gocode/pkg/mod/github.com/google/[email protected]/licenses/embed.go:8:12: pattern *.db: no matching files found

But when I changed ( locally ) the attached change it seems to work.

Env: Ubuntu 16 with Go 16.15 installed

These two .db files are basically looked up directly at runtime. Does anyone have any objection to switching this to go:embed? And is there a maintainer around who would want to merge such a fix? This would need golang 1.16 as a minimum, but that's pretty reasonable.

https://github.com/google/licenseclassifier/blob/main/licenses/forbidden_licenses.db

Cheers!

This allows users of https://github.com/google/go-licenses to run the tool in a go module compliant way by depending on this package, and running the tool. Without this that invariably fails with 'licenses.db cannot be found'.

With this, users can add

github.com/google/go-licenses and github.com/google/licenseclassifier/licenses

to their hack/tools.go file, and then the run of the license classifier is hermetic (uses only the versions known by go modules, which prevents weird inconsistencies due to mismatched tool versions).

This is the BSD-3-Clause license used by the Go language project.

The motivation for adding this is to add the header version, which is otherwise not detected.

The license text is copied from: https://go.googlesource.com/go/+/refs/heads/master/LICENSE

This permits using this package in environments without a GOPATH or without requiring filesystem operations (e.g. by embedding licenses.db).

Also remove unnecessary mutex and loop counter from registerLicenses.

Thank you for this amazing library! I'm amazed by its speed and accuracy, this is truly great work!

One problem I notice is that:

When I use MultipleMatch and parse the offset & extent, I found them to be inaccurate. This is understandable if the algorithm is not perfect. However, I found cases when offset and extent is invalid for the input full text -- offset + extent is greater than full text length.

My guess is that offset and extent did not take the text normalization step into consideration: https://github.com/google/licenseclassifier/blob/bb04aff29e72e636ba260ec61150c6e15f111d7e/classifier.go#L162. So they are in fact offset and extent of the normalized text, which doesn't match the original full text.

Without an actual code-level dependency, go mod vendor will elide the licenses directory, which is where the actual DB files are stored. Without those, it's not very useful to vendor this lib.

@wcn3

In the case where I distribute the binary and the db file separately, I want to be able to specify where licenseclassifier should look for the db. This is related to go-licences (https://github.com/google/go-licenses/blob/master/licenses/classifier.go)

As of today, you need the source of the go-license binary in the right path (GOPATH, …) to be able to use it.

Another alternative would be to embedded the .db in the binary :angel:

This allows library users the flexibility to load their DB from an alternate location, including the network or an embedded array, if necessary.

Also fixed a bug whereby archive serialization would store the whole path. Now, the archive properly stores only the ID instead of the entire path to the ID. This happened when an absolute path was used for the directory, which is useful if you're using a DB in an alternate location.

We're considering to add the identify_license tool as what we call a "scanner" to ORT, see here for some context. As such it would be nice if we could easily bootstrap identify_license by simply downloading a binary for the respective platform.

So, would it be possible to cut a new release (also see https://github.com/google/licenseclassifier/issues/37) and attach binary assets to it?

DefaultClassifier is a bit expensive as it tokenizes and normalizes licenses. We don't want to perform it every run. Is there any way to pass a parsed docs that is a private field? What if we will add NewClassifierWithDocs or something like that? https://github.com/aquasecurity/licenseclassifier/blob/c913e304a1534c4580fa70c2c3af5cd85d99fc9c/v2/classifier.go#L194

Describe the issue In the computeQ when the threshold is set to 1.0 the granularity is being calculated as 10, but if we set the threshold to 0.95, 0.99, or 0.999 the granularity is being calculated as 19, 99, 999, respectively where there is exponential growth and also the granularity is greater than the granularity set at maxThresold(1.0) which is 10.

Is this intentional?

A problem occurring due to this issue is that when we set the threshold to 0.95 or greater a lot of licenses are not being detected which in the case we set to 0.9 are easily being detected.

I ran the program for around 17,300 license files out of which around 2950 BSD-3-Clause, 850 BSD-2-Clause and some other licenses were not at all detected which were otherwise detected at a granularity of 10 because at that threshold the granularity is greater than 20 and nearly reaches 100.

A possible solution would be to set the granularity to 10 for a threshold greater than 0.9 and it will also handle the divide by zero cases.

Perhaps it would be good to have a main package file in v2 of licenseclassifier as in v1. Alongside licensedetection capabilites, it can also support copyright detection, JSON results and scanning entire directory for faster and more efficient results.

Thoughts?

I'm using sqllite3 with blessing license, I'd like to add it to unencumbered licenses in this library. Can I do that?

https://spdx.org/licenses/blessing.html