io.Reader is a more generic abstraction for input: it could be a file or it could be a reader of stdin.

The changes will enable ccat to use the NewScannaer API to create a Scanner. See https://github.com/jingweno/ccat/commit/e7385e67afd8690c1ea677abfc13f2807aadfe05.

Would you be interested in accepting https://github.com/iafan/syntaxhighlight/commit/0da6819d25a1bdbf3c6206cc9ffece21d771bb57 as a PR?

This commit introduces AsOrderedListHTML function which, in addition to highlighting the source, wraps each line in <li>..</li>, and then adds <ol>..</ol> global wrapper. The resulting highlighted source code now has line numbers.

Commit comes with full test coverage.

This issue is largely documented downstream at https://github.com/shurcooL/go/issues/19#issuecomment-97699653, but to quickly summarize:

After #11 was merged, syntaxhighlight package now indirectly imports an extremely large number of imports that are largely unrelated and unneeded to the core functionality of this package.

https://godoc.org/github.com/sourcegraph/syntaxhighlight?import-graph&hide=1

The vast majority of that is coming from the two imports sourcegraph.com/sourcegraph/go-sourcegraph/sourcegraph and sourcegraph.com/sourcegraph/vcsstore/vcsclient which are only needed to pull in the following types:

• vcsstore.FileWithRange
• sourcegraph.SourceCode
• sourcegraph.SourceCodeLine
• sourcegraph.SourceCodeToken

Despite those types being small, the packages that contain them are higher level and import many other dependencies.

We should fix this so that users of syntaxhighlight that only need the core functionality and do not use the NilAnnotator should not need to pull in that many dependencies.

Hi,

I'm trying to use the public interface/API of this package externally, but this API could be better:

const (
    WHITESPACE = iota
    STRING
    KEYWORD
    COMMENT
    TYPE
    LITERAL
    PUNCTUATION
    PLAINTEXT
    TAG
    HTMLTAG
    HTMLATTRNAME
    HTMLATTRVALUE
    DECIMAL
)

First, it's not idiomatic Go style to use ALLCAPS; these consts should be Whitespace, etc. See style guide section on Mixed Caps. As far as I can tell, it will not cause naming conflicts inside the package.

Second, these kind values are untyped constants.

I'd prefer if they were typed, for example syntaxhighlight.Kind, so that I can use that type in a struct in my package.

According to http://godoc.org/github.com/sourcegraph/syntaxhighlight?importers, the only public importers of this package are all my packages. Do you use the current WHITESPACE, etc. outside this package?

Do you agree it would be an improvement and would you take a PR that changes the above code to something like this?

// Kind represents a syntax highlighting class which will be assigned to elements. A style will assign visual properties to each kind.
type Kind uint8

const (
    Whitespace Kind = iota
    String
    Keyword
    Comment
    Type
    Literal
    Punctuation
    Plaintext
    Tag
    HTMLTag
    HTMLAttrName
    HTMLAttrValue
    Decimal
)

Also, Keywords probably does not need to be exported, does it? Unless users of this package are meant to be able to make changes to it (if it's meant to be exported, it should be documented, it's currently not).

• I'm guessing that trying to write to benchmark.txt inside travis is likely creating the error due to permissions. Any file changes inside Travis will be lost anyway, so there's no point in doing that.
• Instead, I just opted to execute the test and benchmarks and show their output in Travis.
• Also make sure the Go code is gofmted.
• Use Go 1.3.

This is more of me not understanding how to use this package, but I'm not finding any css in here. Are we supposed to bring our own?

For example, compare the way github displays simple.go:

https://github.com/sourcegraph/syntaxhighlight/blob/master/testdata/simple.go

Versus the output html of this package for that simple.go file:

https://rawgithub.com/sourcegraph/syntaxhighlight/master/testdata/simple.go.html

Are we supposed to create the css ourselves from scratch, or are there any existing ones that can be used?

Also, all the newlines are missing. Is it because the css is missing or another reason?

Thanks!

I ran go test and everything passes successfully. But verify that nothing is wrong.

Note: gofmt -s outputs code which might not be compatible with earlier versions of Go.

• Update testdata/underscore.go.html to match expected output after input change in b4219f1fb43b89dede6b8a2bc0c03f909cd599f7.
• Fix example verification to occur (see http://godoc.org/testing#hdr-Examples), and update expected output to match actual output.
• Copy latest version of example_test.go into README.md so they are in sync.

When I asked why this branch wasn't merged into master yet, you said the most likely reason was that you simply forgot about it.

So I'm making this PR for your convenience as a reminder. As far as I can tell, this approach of using text/template should be better overall (more general and simple, handles Unicode, 42% slower).

This implements #20 using functional parameters:

// default backward-compatible call
got, err := AsHTML(input)

// with optional parameter
got, err := AsHTML(input, OrderedList())

If the user has an io.Reader, they can now use NewScannerReader to more directly create a Scanner.

Previously, they would be forced to either read all bytes from the reader into a byte slice and use that (which then indirectly gets wrapped by a bytes.NewReader), or copy this func in their code.

Resolves #14. /cc @jingweno

This issue contains a list of Renovate updates and their statuses.

This repository currently has no open or pending branches.

• [ ] Check this box to trigger a request for Renovate to run again on this repository

Welcome to Renovate! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.

Configuration Summary

Based on the default config's presets, Renovate will:

• Start dependency updates only once this onboarding PR is merged
• Enable Renovate Dependency Dashboard creation.
• If Renovate detects semantic commits, it will use semantic commit type fix for dependencies and chore for all others.
• Ignore node_modules, bower_components, vendor and various test/tests directories.
• Autodetect whether to pin dependencies or maintain ranges.
• Rate limit PR creation to a maximum of two per hour.
• Limit to maximum 10 open PRs at any time.
• Group known monorepo packages together.
• Use curated list of recommended non-monorepo package groupings.
• A collection of workarounds for known problems with packages.
• Run Renovate on following schedule: on the 1st through 7th day of the month

🔡 Would you like to change the way Renovate is upgrading your dependencies? Simply edit the renovate.json in this branch with your custom config and the list of Pull Requests in the "What to Expect" section below will be updated the next time Renovate runs.

What to Expect

It looks like your repository dependencies are already up-to-date and no Pull Requests will be necessary right away.

❓ Got questions? Check out Renovate's Docs, particularly the Getting Started section. If you need any further assistance then you can also request help here.

This PR has been generated by Mend Renovate. View repository job log here.

The way underlying text/scanner package works is it produces two different tokens for combined punctuation like <=, >=, !=, :=, ||, :: and so on. For example, != would be represented in the highlighted output as follows:

<span class="pun">!</span><span class="pun">=</span>

This is suboptimal in terms of final HTML size, and also prevents font ligatures from being displayed properly (see e.g. Fira Code font).

This PR implements token normalization so that adjacent tokens of the same kind are merged together. The example above now looks like this:

<span class="pun">!=</span>

...and allows for proper ligature rendering.

I am writing a debugger for Go and came across this project via Dmitri Shuralyov and his use of this which uses go/scanner.

One improvement I'd like to see is more fined-grained control over what gets tagged. Rather than reinvent stuff when not necessary, I consulted Pygments to see what it has.

This is from pygments/formatters/terminal.py;

    Whitespace:         ('lightgray',   'darkgray'),
    Comment:            ('lightgray',   'darkgray'),
    Comment.Preproc:    ('teal',        'turquoise'),
    Keyword:            ('darkblue',    'blue'),
    Keyword.Type:       ('teal',        'turquoise'),
    Operator.Word:      ('purple',      'fuchsia'),
    Name.Builtin:       ('teal',        'turquoise'),
    Name.Function:      ('darkgreen',   'green'),
    Name.Namespace:     ('_teal_',      '_turquoise_'),
    Name.Class:         ('_darkgreen_', '_green_'),
    Name.Exception:     ('teal',        'turquoise'),
    Name.Decorator:     ('darkgray',    'lightgray'),
    Name.Variable:      ('darkred',     'red'),
    Name.Constant:      ('darkred',     'red'),
    Name.Attribute:     ('teal',        'turquoise'),
    Name.Tag:           ('blue',        'blue'),
    String:             ('brown',       'brown'),
    Number:             ('darkblue',    'blue'),

There are some things above that aren't relevant to Go (but you might want to add them for other languages), and there are some things that should be added above for go like Rune, possibly Case label and so on. So here is a list below:

• Whitespace
• Comment
• Keyword
• Type name
• Built in
• Function
• Variable
• Exported name
• Number
• Rune

Kind is a uint8 so there is plenty of space to add constants to this.

If you would like me to create a pull request for this, let me know.

Thanks.