I am referring to this:

package main
import "gopkg.in/jdkato/prose.v2"
func main() { prose.NewDocument("Go is ...") }

The NewDocument is actually in gopkg.in/jdkato/prose.v2/summarize now.

However, go get gopkg.in/jdkato/prose.v2/summarize does not work either. The package does not compile due to usage of internal package. This is a typical error when using the gopkg.in service. gopkg.in only redirects the git URI, but does not rewrite package import paths. As a result, the referred imports will actually point back to the master branch, nullifying the essential purpose of versioning. To use gopkg.in properly, you need to manually rewrite the import paths in the entire repo in the release tags/branches (or just stop using gopkg.in for multi package repos..).

I saw your repo from Hacker News, but your repo fails to build on smallrepo. Detailed build log here:

https://smallrepo.com/builds/20180717-175536-bc73d63d

Thanks.

This moves the logic that was in chunk_test.go into a new exported function named Chunk. The primary difference is that, instead of returning a slice of locations, we're now returning a slice of strings (i.e., the actual chunks).

This changes the usage from

words := tokenize.TextToWords(text)
tagger := tag.NewPerceptronTagger()
tagged := tagger.Tag(words)
rs := Locate(tagged, TreebankNamedEntities)

for r, loc := range rs {
    res := ""
    for t, tt := range tagged[loc[0]:loc[1]] {
        if t != 0 {
            res += " "
        }
        res += tt.Text
    }

    if r >= len(expected) {
        t.Error("ERROR unexpected result: " + res)
    } else {
        if res != expected[r] {
            t.Error("ERROR", res, "!=", expected[r])
        }
    }
}

to

words := tokenize.TextToWords(text)
tagger := tag.NewPerceptronTagger()
tagged := tagger.Tag(words)

for i, chunk := range Chunk(tagged, TreebankNamedEntities) {
    if i >= len(expected) {
        t.Error("ERROR unexpected result: " + chunk)
    } else {
        if chunk != expected[i] {
            t.Error("ERROR", chunk, "!=", expected[i])
        }
    }
}

/cc @elliott5

Have you considered adding the Coleman–Liau index for completeness? Even though "opinion varies on its accuracy": https://en.wikipedia.org/wiki/Coleman%E2%80%93Liau_index

On the subject of suspect measures, a composite "years-of-education" metric, taking the average of scores together (and their standard deviation) may be of use: https://github.com/elliott5/readability/blob/master/assess.go

Finally, for giving feedback to users on how to change their prose to be easier to read, it would be great if your analysis could store:

• sentences with their word length; and
• words with their syllable length and frequency (the product of the two ranking non-readability).

Keep up the good work!

I tried to go get the hugo repo. I am using Go 1.14 (with go modules)

go get github.com/gohugoio/hugo

package github.com/jdkato/prose/transform: cannot find package "github.com/jdkato/prose/transform" in any of:
	/usr/local/go/src/github.com/jdkato/prose/transform (from $GOROOT)
	/Users/x/GOPATH/src/github.com/jdkato/prose/transform (from $GOPATH)

I'm wondering why it is saying transform can't be found?

Very useful project.

I wanted to test the title case functionality for use in Hugo, but we vendor our libraries, and I get a ../vendor/github.com/jdkato/prose/transform/title.go:9:2: use of internal package not allowed when importing github.com/jdkato/prose/transform.

See https://github.com/gohugoio/hugo/pull/3753

I have been Googling this, and it seems there is no (simple) workaround other than avoiding the use of internal packages in libraries.

This is rough outline of some improvements I'd like to make.

Documentation

• [x] Improve and update README.
• [ ] Add .github files.

tokenize

• [x] Port the pragmatic_segmenter.
• [x] Get PunktSentenceTokenizer passing the The Golden Rules and possibly submit a PR upstream.

tag

• [x] Finish porting the PerceptronTagger (just training functionality left).
• [x] ~~Port the TnT Tagger~~ (not going to make v1.0).
• [ ] Improve testing strategy (we currently rely on NLTK).

transform

• [x] Improve Title and add support for variations (e.g., AP style).
• [ ] Port Change Case.

summarize

• [x] Finish working on Syllables.
• [x] Add a composite "years-of-education" metric.
• [ ] Add the ability to update a Document's content without having to recalculate all of its statistics.
• [x] Expand test suite.

This PR changes iterTokenizer struct to contain the parameters that can be overriden. This will allow future changes to NewDocument that use a custom tokenizer.

Will help with issue #41 and #32

Test: existing unit test

I tried installing prose using the command provided go get gopkg.in/jdkato/prose.v2 and this is the error log printed on my terminal.

# gopkg.in/jdkato/prose.v2
../gopkg.in/jdkato/prose.v2/extract.go:323:23: est.AtVec undefined (type *mat.VecDense has no field or method AtVec)
../gopkg.in/jdkato/prose.v2/extract.go:335:25: est.AtVec undefined (type *mat.VecDense has no field or method AtVec)
../gopkg.in/jdkato/prose.v2/extract.go:355:17: count.AtVec undefined (type *mat.VecDense has no field or method AtVec)
../gopkg.in/jdkato/prose.v2/extract.go:525:27: count.AtVec undefined (type *mat.VecDense has no field or method AtVec)

go-lang version : go1.9.7 darwin/amd64

I'm trying to use TItleConverter.Title(), but it panics on me, when the string contains certain multibyte characters.

Crashing testcases:

        type tc struct {
		name  string
		input string
		want  string
	}
	tests := []tc{
                tc{"panic", "This Agreement, dated [DATE] (the “Effective Date”) for Design Services (the “Agreement”) is between [DESIGNER NAME], of [DESIGNER COMPANY](“Designer”), and [CLIENT NAME], of [CLIENT COMPANY] (“Client”) (together known as the “Parties”), for the performance of said Design Services and the production of Deliverables, as described in Schedule A, attached hereto and incorporated herein by reference.", "panic"},
		tc{"panic", "Crash,”“us,” “our” or “we” means Crash Network, Inc. (d/b/a Crash) and its subsidiaries and affiliates.", "panic"},
		tc{"panic", "a “[“New Entity”],” an [Institution] and [Institution].", "panic"},
        }

Hi, I can see that we have a comprehensive list of tags but can't find anything for label (for example, PERSON, GPE, etc). Would be nice if someone can redirect me to the list, even if it is somewhere in the source code.

This PR allows the user to provide a different tokenizer.

Users can specify their own Tokenizer in the DocOpts. This replaces the boolean Tokenize option (set Tokenizer to nil to disable.)

Currently only IterTokenizer is provided, which can be customized with its own Using options. func Tokenize becomes public to allow users to provide their own implementation and completely replace IterTokenizer.

Model and Extractor need to use the same Tokenizer as Document, so this PR modifies those APIs to be consistent.

(Also separating makeCorpus from extracterFromData to simplify parameter passing.)

This solves issue #41 and #32

I'm a bit surprised to see this:

type Document struct {
        Model *Model
        Text  string

        // TODO: Store offsets (begin, end) instead of `text` field.
        entities  []Entity
        sentences []Sentence
        tokens    []*Token
}

// A Sentence represents a segmented portion of text.
type Sentence struct {
        Text string // The sentence's text.
}

If I care about finding sentences first, then the words within them, I need to take two passes, right?

 	// First we will do only segmentation, to break up sentences
    doc, err := prose.NewDocument(string(content), prose.WithTagging(false), prose.WithTokenization(false), prose.WithExtraction(false))
	if err != nil {
        log.Fatal(err)
    }

    // Iterate over the doc's sentences, and words within them
    sents := doc.Sentences()
    fmt.Println(len(sents))
    for _, sent := range sents {
        fmt.Println(sent.Text)

		sdoc, err := prose.NewDocument(sent.Text, prose.WithTagging(false), prose.WithExtraction(false), prose.WithSegmentation(false))
		if err != nil {
			log.Fatal(err)
		}

		// Iterate over the doc's tokens:
		for _, tok := range sdoc.Tokens() {
			fmt.Println(tok.Text, tok.Tag, tok.Label)
		}

		// Iterate over the doc's named-entities:
		for _, ent := range sdoc.Entities() {
			fmt.Println(ent.Text, ent.Label)
		}
    }

I suspect it's less efficient that way.

Likewise, tokens include named entities, but it might make more sense to be able to iterate tokens in a sentence in such a way that each token is either a named entity or a regular token?

If you intend to store offsets, like the TODO comment says, maybe this can be worked around, by finding overlapping offset ranges (e.g. sentence 2 goes from character position 240 to 267; word 10 goes from 240 to 247; named entity 2 goes from 248 to 257; etc... then I can see that word 10 and named entity 2 are both part of sentence 2, even if you don't offer a hierarchical model).

I am seeing a ~100ms overhead to process any document (text), which seems like it can't be correct given the performance data listed for large corpuses. I've been porting a large Python/NTLK app to Go, but my legacy Python/NLTK text parsing is running ~250x faster than my new go/prose implementation, which seems like I must be doing something wrong.

Did some external dependency break the performance of prose, or am I using the API incorrectly?

Simple performance test (Go 1.18)

Here's a simple test that processes the same short sentence twice. Both executions take ~100ms.

Code:

package main

import (
	"fmt"
	"time"

	"github.com/jdkato/prose/v2"
)

var text = "This is a simple test."

func main() {
	for i := 0; i < 2; i++ {
		start := time.Now()
		doc, err := prose.NewDocument(
			text,
			prose.WithExtraction(false),
			prose.WithSegmentation(false))
		duration := time.Since(start)
		fmt.Println(duration)
		if err != nil {
			panic(err)
		}

		// Iterate over the doc's tokens:
		fmt.Print("   ")
		for _, tok := range doc.Tokens() {
			fmt.Printf("(%v, %v)  ", tok.Text, tok.Tag)
		}
		fmt.Println()
	}
}

Output:

$ go run .
118.549243ms
   (This, DT)  (is, VBZ)  (a, DT)  (simple, JJ)  (test, NN)  (., .)  
117.214746ms
   (This, DT)  (is, VBZ)  (a, DT)  (simple, JJ)  (test, NN)  (., .)  
$

Comparison test using NLTK in Python (3.8)

When I run the same test using NLTK in Python, the first document processed also has ~100ms of overhead, but all subsequent documents are processed very quickly (~400usec in the example below):

Sample code:

#!/usr/bin/env python
import nltk
from datetime import datetime

text = "This is a simple test."

for _ in range(2):
    start = datetime.now()
    raw_tokens = nltk.word_tokenize(text)
    pos_tokens = nltk.pos_tag(raw_tokens)
    duration = datetime.now() - start
    print(duration)
    print(f'   {pos_tokens}')

Output:

$ ./test-nltk.py 
0:00:00.092738
   [('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('simple', 'JJ'), ('test', 'NN'), ('.', '.')]
0:00:00.000415
   [('This', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('simple', 'JJ'), ('test', 'NN'), ('.', '.')]
$

I'd recommend updating the dependency neurosnap/sentences to use its new location (github.com/neurosnap/sentences) and the latest version (v1.0.9). This version fixes a very minor security problem where the readme linked to some amazon s3 buckets for downloading binaries. When repositories vendor jdkato/prose they will also get neurosnap/sentences, and it is better if they get this updated version.

Installed prose with go get github.com/jdkato/prose/v2 and copied the code from the tokenization example in the readme.

The output I get is

@jdkato NN
, ,
go VB
to TO
http://example.com NN
thanks NNS
: :
) )
. .

Instead of the expected one reported in the example:

        // @jdkato NN
        // , ,
        // go VB
        // to TO
        // http://example.com NN
        // thanks NNS
        // :) SYM
        // . .

The difference is that the smile ":)" symbol is not recognized.

$ go version go version go1.16.5 linux/amd64

Hi i have a function that utilizes prose/v2.

it works fine as it passes my tests (just the logic though).

When i deploy it on AWS lambda, it doesn't run. I have logs right to the time I do

prose.NewDocument(stringVal)

And nothing logs out after that, no error nothing. albeit a noob with aws lambda, so maybe there are logs i'm not looking at.

What is it doing internally? Wonder why it would just end