Example:

[GIN-debug] Listening and serving HTTP on :8080
Creating invoice for a new API request
2018/09/02 22:45:30 Sending invoice in response: [...]
[GIN] 2018/09/02 - 22:45:30 | 402 |    279.2542ms |       127.0.0.1 | GET      /qr?data=testtext

So it's Creating invoice for a new API request, but also Checking invoice for hash [...] and potentially others.

Reproduction

1. Create backend with 2 endpoints, "a" and "b"
   • "a" is 1 Satoshi, "b" is 10 Satoshis
2. Client sends initial request to each
3. Client pays invoice for "a"
4. Client sends final request to "b", but with the preimage of the payment for "a"
5. => Request works although the client didn't pay the related invoice

Problem

The middleware doesn't track which invoice was created for which endpoint, it doesn't know about the endpoints at all. It just checks if the preimage was already used as payment proof and then checks the LN node for the invoice's existence and settlement.

Possible solution

1. When the initial request arrives we're in the correct middleware instance, so we cache the URL path along with the preimage or its hash
2. When the final request arrives we don't just check if it's valid (not used before, invoice settled), but also if the current request URL path is the same we previously cached. (Lookup via the preimage or its hash)

Note 1: It's not enough to have an in-memory cache of just the preimage or its hash per middleware instance, because in case of a horizontally scaled web service the caches wouldn't work properly anymore. It's probably best to use the existing storage client implementations, so the web service developer can choose for example Redis when he wants to scale horizontally.

Currently when a developer uses an API that's paywalled with ln-paywall (or ElementProject's paypercall and potentially other projects in the future), he has to take care of:

• Sending two physical requests for one logical one
• Handling the 402 response
• Establishing a connection to his Lightning Network node by himself and calling its RPC / HTTP API

All by himself.

This can be easily wrapped in a way so that the developer only has to create one instance of a client, similar to the net/http.Client, create a single net/http.Request and then call client.Do(...). Just as with the net/http.Client. All the payment handling can happen automatically in the background.

The first goal should be a minimal working version. More advaned features should be added in the future with separate issues created for each feature.

Minimal working version:

• Connects to and works with lnd
• No convenience methods (Get(...), Post(...)), just Do(...)
  • Are they exposed by the embedded HTTP client? Can they be hidden in that case?
• No configuration like "only pay if the amount in the invoice is <= x BTC", just always pay the invoice

Future features:

• Support other LN node implementations
• Make payments configurable, e.g. "only pay if the amount in the invoice is <= x BTC"
• ...

1. Add code coverage analysis, for example: go test -v -race -coverprofile="coverage.txt" -covermode=atomic ./...
2. Add reporting to service like Codecov or Coveralls
3. Add badge to README

See:

• https://blog.golang.org/cover
• https://github.com/codecov/example-go

Groupcache is a distributed cache, but unlike Redis, it doesn't need an extra server - it connects to its peers directly. While skimming info about it I read that you can't update or delete entries, because it's just meant as the most simple cache, but that's currently sufficient for ln-paywall's needs.

https://github.com/golang/groupcache

Currently the value in the "x-preimage" header is required to be Base64, because that's the format that's used in listed invoices when executing lncli listinvoices. Not only for preimages, but also for their hashes.

But on the other hand the LightningClient's method LookupInvoice(...) expects a hex-encoded payment hash.

That's why in our package, when receiving a request with a correct x-preimage header, we need to:

1. Base64 Decode the preimage
2. Sha256 the previous value
3. Hex the previous value
4. Send it to lnd

So:

1. Figure out which format is best for using in the header (Plain preimage, Base64 or hex)
2. Change our implementation in case it's not Base64
3. Document which format is expected

Currently a new gRPC connection to lnd is created for every request.

It should be possible to create the gRPC connection only once when one of the middlewares is created, for example when NewHandlerFuncMiddleware(...) is called, but before the handler func is returned.

func createHandlerFunc(invoiceOptions InvoiceOptions, lndOptions LNDoptions, storageClient StorageClient, next http.HandlerFunc) func(w http.ResponseWriter, r *http.Request) {
	// <---- create the gRPC connection here and reuse it within the below func
	return func(w http.ResponseWriter, r *http.Request) {

Care must be taken regarding connections that are aborted/lost/..., for example due to a network error. Is it possible to reconnect with the existing connection object/ref?

Currently the middleware factory functions contain a lot of duplicated code. There should be only one web framework-independent function that contains the main middleware logic, and then web framework-specific implementations that handle the details of the used web frameworks.

This PR contains that, including a fix for the middleware for Echo.

• The invoice response contains a 200 OK status code instead of 402 Payment Required
• All error responses wrap the error message in JSON instead of just using the error message as text, as the other middlewares do. It should be the same for all middlewares, and it should be the text directly.

As mentioned in the v0.5.0 release notes, the ln.charge implementation of the wall.LNclient interface currently becomes slower when the amount of invoices in the Lightning Charge server increases.

The reason is that ln-paywall currently uses the payment hash (a.k.a. preimage hash) of an invoice as ID, which works fine with lnd (and would also work with eclair). But Lightning Charge uses its own randomly generated IDs, so instead of fetching the specific ID we need, we fetch all IDs and filter out the one we're looking for by comparing the preimage hashes.

The solution is to add the implementation specific ID as field to the wall.invoiceMetaData so it can be stored, and when a client sends the final request we can still use the preimage hash to fetch the metadata from the storage, but then use the ID within that data to make the lookup on Lightning Charge.

This leads to the ID and the preimage hash fields being the same when using lnd. We could omit one of the fields and just use one of them, but this could also lead to confusion later.

• Previously a client just had to send any preimage that corresponds to a settled invoice in the LN node, without the invoice having to originate from the middleware, or from the same endpoint as used in the current request. So he could pay cheap invoices and use expensive endpoints for example.
• The fix required storing metadata in the DB, e.g. the HTTP method and URL path, while being backward compatible and not send those data encrypted as token in the first response, requiring the same token and decrypting and using the values during the second request
• Both modes are interesting and have their pros and cons, so evaluate supporting both in the future
• Storing the metadata required changing the wall.StorageClient interface and its implementations in the storage package

Closes #16

Your FOSSA integration was successful! Attached in this PR is a badge and license report to track scan status in your README.

Below are docs for integrating FOSSA license checks into your CI:

• CircleCI
• TravisCI
• Jenkins
• Other

Currently pay.Client only implements a subset of the http.Client's methods. I think it should be fully compatible though.

Check out gentlemen for creating HTTP clients with middleware.

Currently the storage implementations all marshal and unmarshal to/from JSON. For example, a wall.invoiceMetaData object is first marshalled to JSON and then the resulting string is stored as value in Redis.

JSON is nice because when problems occur (e.g. a customer says his HTTP client sent the correct X-Preimage header but the web service responded with an error "Corresponding invoice not found" or something similar), you can just check the storage and have a look at the stored data, easily seeing what's in there without having to decode anything. This might be harder for the Go Map (it's only visible / only exists within the running web service) and the bbolt DB (file is locked while in use), but easier for Redis and future storages like etcd and Consul, where even some web dashboards for exploring the storage contents exist.

The downside of JSON though is that it's verbose. A binary format can be much more compact and thus require less storage.

In Go, there are "gobs". See:

• https://godoc.org/encoding/gob
• https://blog.golang.org/gobs-of-data

Before implementing this, we should first benchmark if it makes sense at all for the limited types of data we store (currently only wall.invoiceMetaData). A 50% reduction in storage would be nice, or maybe 30% is enough?

Also be aware that when storing gobs, other programming languages can't deal with the data! For example when a customer uses one lnd instance, but two web services with one using ln-paywall and the other using a compatible Java middleware, and he wants to prevent web service clients from cheating by reusing preimages. Cheating example: Send first request to the Go web service, pay the invoice, send final request with preimage to Go web service, then send another request immediately with the same preimage to the Java web service. If both web services would have separate storage mechanisms the client could cheat that way. So a common storage must be used, like Redis for example. But when the stored value in Redis is a gob, the Java middleware can't read it. So:

1. Using gobs instead of JSON must be optional, off by default
2. This downside must be mentioned in the comments for the respective option (probably in the storage implementation's specific options object)

As mentioned in https://github.com/philippgille/ln-paywall/issues/16#issuecomment-423765859, ln-paywall stores invoice metadata in the storage, so it doesn't require the X-Token that paypercall requires. The token in paypercall is used to not only include the invoice ID, but also store the metadata (HTTP method, URL path) in the token, send it to the client, then the client has to include it in the final request so the middleware can read the metadata from the token again and verify it matches the method + path of the current request, so the client can't cheat.

So while ln-paywall doesn't require it, it would be nice to be compatible with paypercall clients. Although I'm not aware of any client-side libraries (other than our own ln-paywall client, which will gain paypercall compatibility with #23), this doesn't mean no client-side code exists (could be just plain HTTP requests), and also it would be nice to offer this compatibility anyway.

So: After #25 is implemented, after which the middleware will accept not only the preimage of a payment, but also the invoice ID, this invoice ID can just be sent as X-Token in the response to the initial request.

I don't think this should be the default though. So make it configurable with some new wall.MiddlewareOptions.

Currently the invoice metadata is stored in the DB forever.

This isn't necessary from a company's point of view, because 1) the invoices are already stored in the LN node and 2) companies that need to keep invoices around for a couple of years need additional data anyway, so the metadata in the ln-paywall storage aren't of use here.

And it's also not necessary from an implementation point of view (since #24), because 1) LN invoices expire after some time (can be configured when creating the invoice), so when an invoice isn't paid until its expiry, the metadata is of no use anymore, and 2) due to the final request now only working when the DB has a metadata entry about the corresponding initial invoice request, cheating with reusing preimages that aren't in the storage doesn't work.

The two cases in detail:

• Invoice request is sent, metadata is created. The invoice expires after 1 hour (default in the LN protocol standard, see BOLT 11.
  • If the client doesn't pay within 1 hour, we can throw away the metadata anyway, because the client is technically not able to pay the invoice later (due to invoice expiry).
  • If the client does pay within 1 hour, we should give him some time to send the final request, but that shouldn't be unlimited. Considering the programmatic nature of HTTP requests, a shorter expiry might make sense (invoice payment is often done by humans, with wallet software which might require some channel syncing first etc.), but 1 hour might be fine as well. This needs to be configurable.
• Invoice request is sent, metadata created, final request sent, metadata marked as "used", so everything is done. Until #24 the preimage needed to be kept around in storage, because otherwise a client could just use any preimage that corresponds to an invoice in the LN node and the middleware would have accepted it, but now, if a request contains a preimage, which is not in the storage, it's rejected immediately! BUT the error message might confuse clients that accidentally reuse a preimage in a second request. So for usability reasons the metadata should be kept around for a bit as well. This expiry should not start from the invoice creation, but the use (the final request).

So:

• Unused metadata should be kept for 1 hour (configurable) after invoice creation, to give the client time to send the final request after payment
• Used metadata should be kept for 1 hour (configurable) after usage, to give the client proper error messages when reusing a preimage

So far the middleware required the preimage as payment proof in the second request.

But since PR #24 invoice metadata is stored in the DB, with the invoice ID being used as the key, so now we could also just accept an invoice ID and look up the status in the DB or check the settlement on the LN node.

This would make it much easier to implement clients that don't have access to the preimage. For example: Let's say the API is used on a website, but without automated payments via an LN node operated by the people running the website, and instead the website shows the user a QR code so he can pay for the API usage. Up until now the website had to ask for the preimage. This is an extra step in itself, but even worse, if the user paid via mobile wallet, but browsed the website via desktop PC, he now had to get the preimage from his smartphone to the PC just to proof that he made the payment. When the invoice ID is enough, the user doesn't have to input anything. The website can decode the LN invoice and take the invoice ID, show the user the QR code and a button "next"/"continue" or "paid", and then the website can just send the request with the invoice ID.

One example of such a website is https://lightning.ws, where the deployed APIs have a "Try out" section where the user has to do exactly the steps mentioned above.