Fixes #4

On Win make code-generate produces unreasonable Bayesian classifier weights from Linguist samples silently, failing only the final classification tests.

TestPlan:

• passing tests on Win CI

  go test ./internal/code-generator/... \
   -run Test_GeneratorTestSuite -testify.m TestGenerationFiles
  

Purpsoe

[Fixes]:

1. Correct failing test cases for Java Bindings so that make test command under Java does not fail.
2. Expose isTest method in java bindings.

[Features] :

1. Export new function GetLanguageType at enry package in go & expose the same at Java Bindings.

I couldn't find any way in this repo on how to get the shared library so it can be used from other languages.

So is there a prebuilt one somewhere or some instructions on how to build your own?

Several cosmetic changes

• API function declarations order follows tests order
• Linguist lazy loading logic unified & re-used, as much as possible between tests&benchmark
• Separate test suite extracted for running over Linguist samples/fixtures

The Linguist update automation runs once a day, so if the generated PR isn't merged by then, it will open another one!

https://github.com/go-enry/go-enry/pull/68 #70

As discussed in https://github.com/go-enry/go-enry/issues/54, this provides an API for accessing a LanguageInfo struct which is populated with all the data from the Linguist YAML source file. Functions are provided to access the LanguageInfo by name or ID.

The other top-level functions like GetLanguageExtensions, GetLanguageGroup, etc. could in principle be implemented using this structure, which would simplify the code generation. But that would be a big change so I didn't do any of that. Perhaps in the next major version something like that would make sense.

cc @tclem

Closes https://github.com/go-enry/go-enry/issues/54

This is a blueprint for all other methods, dealing with go slice conversion.

It still lacks on build automation (and there is no release automation whatsoever), but this is already useful.

This PR remote the old substring package from @toqueteos (sorry dude) and use the internal regex package to use oniguruma regexp with all the regular expressions.

I discovered this through our Gitea server flagging files as vendored through its use of enry.IsVendor(). Paths like oslo_cache/_bmemcache_pool.py and playbooks/roles/create-venv/tasks/main.yaml are inappropriately marked vendored.

My hunch is that the first path is matching https://github.com/go-enry/go-enry/blob/7168084e5e5de38b915b1874528ff73f20a86b69/data/vendor.go#L9 and the second is matching https://github.com/go-enry/go-enry/blob/7168084e5e5de38b915b1874528ff73f20a86b69/data/vendor.go#L110

I've written a little reproducer that removes Gitea from the equation:

package main

import "fmt"
import "regexp"
import "github.com/go-enry/go-enry/v2"

func main() {
	input_str1 := "oslo_cache/_bmemcache_pool.py"

	rawregex1, _ := regexp.MatchString(`(^|/)cache/`, input_str1)
	fmt.Println("Raw regex:", rawregex1)

	vendor1 := enry.IsVendor(input_str1)
	fmt.Println("IsVendor:", vendor1)

	input_str2 := "playbooks/roles/create-venv/tasks/main.yaml"

	rawregex2, _ := regexp.MatchString(`(^|/)env/`, input_str2)
	fmt.Println("Raw regex:", rawregex2)

	vendor2 := enry.IsVendor(input_str2)
	fmt.Println("IsVendor:", vendor2)
}

When you run this the results are:

Raw regex: false
IsVendor: true
Raw regex: false
IsVendor: true

What this shows us is that the raw input regexes appear to behave as expected. Neither of our example input strings matches which is what we expect. But when we call IsVendor() the result becomes true. I suspect that the init function https://github.com/go-enry/go-enry/blob/7168084e5e5de38b915b1874528ff73f20a86b69/utils.go#L139-L246 is either adding rules that collide or introducing some bug to the expanded regex that causes this to happen.

Most diffs to go checksum files are pure noise, I'm wondering if anyone else agrees it should be marked as generated so tools like gitea can hide diffs on it? Linguist doesn't do it but I think diverging here is fine.

Rather than creating the branch for the update PR ahead of time using the date, this changes it to use the short hash of the Linguist commit that was found, and updates the code so that if the branch already exists, it will exit without creating a PR.

This branch name should be the same between runs of the workflow (unless the Linguist release tag is changed, which warrants another update anyway) and should address the problem of creating one PR a day until the update is merged.

You can see an example of a PR created by this code here: https://github.com/look/go-enry/pull/8 (note the branch name)

closes https://github.com/go-enry/go-enry/issues/69

cc @bzz @lafriks

This is alternative to #138 where on build-time we always generate unaltered regexp syntax for all the rules and make runtime checks, similar to #65.

This, by itself, does not solve the problem of dealing with more non-RE2 syntax coming from linguist, only renders it more visible. The only solution that I can see now that would not require everyone using native library (oniguruma) or compromising on predictive accuracy (due to missing rules for unsupported syntax) is to try shipping another regexp engine in go that would support the necessary syntax.

TODOs

• [x] update content heuristics generation
• [ ] update vendor generation
• [ ] add Oniguruma-only tests, as in #65
• [ ] add https://github.com/dlclark/regexp2 backend option

This change does several things:

• ~refactors optimization of vendor regexp collation for IsVendor()~
• ~moves it to build-time "code generation" phase (instead of runtime at package initialisation)~
• introduces RE2 syntax check for vendor.go (case that fails #137), the same as we use for heuristics from content.go (and skip the rules with unsupported syntax)
• adds new CI profiles to test code generation

The attempt is made for the checks to be RE-lib-specific, thus make code-generate now also respects the same --tags and should be passed when using of oniguruma is desired.

I've found out that this project uses JNAerator (https://github.com/nativelibs4java/JNAerator) and JNAerator does not seem to be maintained anymore, being Sep 30, 2015 its last commit date.

The JNAerator in this project uses JNA version 4.1.0 and this version does not provide for 'linux-aarch64'. (4.2.0 and above does.)

I am using go-enry in my project and it gives out the following error in Docker for Apple Silicon. java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-aarch64/libjnidispatch.so) not found in resource path

I hope there's a way to fix this!

This PR enables the support of all the non-support heuristics due to the go regexp engine.

• Exposes the original regular expressions to the regex package.
• The regex package now handles the conversions of the Ruby regular expression to the go-ish version.
• The heuristics rules now support nil regular expressions since some of the heuristics can't compile using the standard library.

I added some tests using the linguist fixtures and passes all the fixtures (using onigurama).

All python wrappers are memory leaking.

This may happen in several places:

1. In CFFI - layer (when structures are converted between Python and Go runtimes)
2. In CGo shared library (when C structures are converted in Go instances)
3. Somewhere in enry library (nearly impossible, because all wrappers are leaking)

You can reproduce it using this test script:

import os

import enry
import psutil

process = psutil.Process(os.getpid())

content = "import os\nprint('Hello, world!')".encode()
path = "test.py"
initial_usage = process.memory_info().rss

for _ in range(10000):
    enry.get_language(path, content)

print(round(process.memory_info().rss / initial_usage * 100, 2))

Here is some info about memory usage of get_language function (in % from initial ram usage), you can see that it's leaking: 1000 iterations: ~100.5 10000 iterations: ~102.4 100000 iterations: ~111.1