TL;DR I can't get tiles out of go-pmtiles v0.5 for some files which work well through pmtiles-js.

I suspect it is linked with the file having leaf directories.

| File | Size | JS - test URL | GO - test URL | | --- | -- | -- | -- | | Bugianen_Cervino | 0.2 GB | :ok: | :ok: wtracks C-go | Bugianen | 3.5 GB (sorry!) | :ok: wtracks B-js | :no_entry: wtracks B-go | eslo14_walps | 1.5 G | :ok: | :no_entry:

Context

I am integrating some pmtiles maps in https://opoto.github.io/wtracks/

After merging pmtiles support (opoto/wtracks#21), I have working, but excruciatingly slow pmtiles-js-on-HTTP1 maps (that we discussed in protomaps/PMTiles#16). For example IT Bugianen (find it in More... menu or though the custom link above)

Now I'm trying to evaluate performance on go-pmtiles on the same map, but it does not work (metadata works 404 on tiles I tried). See the links to download/test the files above. I including a valid file to show that the issue is not how my server setup on montagne.top.

To test on wtracks, search for a covered area for both maps eg "Aosta". Or test with curl:

curl -i https://www.montagne.top/maki/Bugianen_Cervino/16/34100/23382.jpg
HTTP/1.1 200 OK
curl -i https://www.montagne.top/maki/Bugianen/16/34100/23382.jpg
HTTP/1.1 404 Not Found
curl -I https://www.montagne.top/tile/eslo14_walps/16/34100/23382.png
HTTP/1.1 404 Not Found

I can try to debug it myself at some point, but figured maybe it's a known issue with big files?

could fix #3 (sorry, first pull request was based on wrong branch)

some points to extend:

• [ ] write proper documentation on how to use and modify
• [ ] add testing pipeline to pass before creating a new image
• [ ] ..

Used go-pmtiles to convert a MBTiles with compressed vector tiles. It did not work in maplibre. Then I made the MBTiles with uncompressed vector tiles and converted again. Then it worked just fine. Am I doing something wrong?

It seems that tiles are not cached (trying on latest chrome, linux)

Looking at others they all set cache-control:

• OpenStreetMap: cache-control: max-age=604800, stale-while-revalidate=604800, stale-if-error=604800
• OpenTopoMap: Cache-Control: max-age=439803
• Swisstopo: cache-control: public, must-revalidate, max-age=3600
• IGN: access-control-max-age: 43200 etc

I'll admit up front that I am least familiar with Go apps generally, so please excuse any very basic misses here. I love the concept of PMTiles and hope to test, initially focusing on python, but it seems this Go server is the more robust option.

I had hoped to take a shortcut to free deployment via Heroku for testing (following this workaround for compiled app https://medium.com/ki-labs-engineering/deploying-a-native-go-binary-on-heroku-6d4c955819d8), but the lack of configuration of port seems to be the main blocker. There appears to be no way to utilize a dynamic port currently.

Please disregard if this is an edge case, or if there is generally a better way to approach deployment of a precompiled Go app such as this. Excited to figure this out...

./pmtiles 
2022/10/27 14:16:01 main.go:22: Command is required.

There doesn't seem to be any way to get the available commands listed. ./pmtiles --help doesn't work.

It'd be great if I can configure options using environment variables to have deployments be easily managed via docker, etc.

PMTILES_PATH=https://storage.googleapis.com/my-tiles
PMTILES_CORS=*
PMTILES_CACHE=2000

/pmtiles show file:// public/catch.pmtiles 
total size: 13 MB
tile type: Vector Protobuf (MVT)
bounds: 140.962409,-39.136654 149.974995,-33.981390
min zoom: 0
max zoom: 15
center: 142.476196,-38.371809
center zoom: 15
addressed tiles count: 322258
tile entries count: 55994
tile contents count: 42135
clustered: true

Surprised it doesn't list what spec version the .pmtiles file is - that would be helpful.

This PR replaces all the calls to log.Fatal or panic in the pmtiles package libraries with error responses. The main.go tool has been updated accordingly. This allows the pmtiles package to be used by other code without having to trap fatal errors.

... since they are allowed in object names for some providers, and otherwise we get 404 error.

Also it is quite hard to debug as there is no log at all in this case. Suggestions:

• Return 400 Bad Request instead of 404
• Log Malformed URL "foo"

Note: PR made in the browser, untested, and my first line of Go, if we can call it that :laughing:

• Disable the compressor
• Add an option to disable deduplication (requires prior knowledge)
• num tile contents data needs to be deduplication-aware
• Fix header values and reading MBTiles/PMTilesv2 metadata

It would be useful to be able to extract tiles from a pmtiles file for a certain area and put the extracted tiles into a new pmtiles file.

It could look something like this:

pmtiles extract input.pmtiles output.pmtiles --bounds "47.5,7.5,48.0,8.5" 

A usecase could be to have a global tileset and allow users to download an extract for a city or so.

I was using the writer interface of v2 combined with github.com/paulmach/orb to create my pmtiles file.

	writer := pmtiles.NewWriter("./base.pmtiles")
	for z := minZoom; z <= maxZoom; z++ {
		covering := tilecover.Bound(bound, z)
		for tile := range covering {
		    layers := mvt.Layers{.....layers.....}
		    tileData, _ := mvt.Marshal(layers)
    
		    writer.WriteTile(
			    pmtiles.Zxy{
				    Z: uint8(tile.Z),
				    X: tile.X,
				    Y: tile.Y,
			    },
			    tileData,
		    )
		}
	}

	metadata, err := json.Marshal(pmtiles.Metadata{
           ...metadata...
         })
	if err != nil {
		return err
	}
	writer.Finalize(metadata)

Is something similar still possible using v3?

To create a better separation of concerns, it might be a good idea to split the pmtiles folder. The pmtiles folder can hold all the library code for reading and writing pmtiles. Next to a cmd folder that can hold the cli related commands and logic for handling flags, etc.

If the directory does not fit in only the root, right now the code defaults to creating a root with leaves of a fixed size 4, 8, 16... kB.

It should be smart enough for two cases:

• If the total directory is slightly larger than 16,384, the root should have a mix of tile entries and leaf pointers
• If the total directory is much larger, it should try to fill as much of the 16,384 as possible.