General purpose library for reading, writing and working with OpenStreetMap data

osm CI Go Report Card Go Reference

This package is a general purpose library for reading, writing and working with OpenStreetMap data in Go (golang). It has the ability to read OSM XML and PBF data formats available at planet.osm.org or via the v0.6 API.

Made available by the package are the following types:

  • Node
  • Way
  • Relation
  • Changeset
  • Note
  • User

And the following “container” types:

List of sub-package utilities

  • annotate - adds lon/lat, version, changeset and orientation data to way and relation members
  • osmapi - supports all the v0.6 read/data endpoints
  • osmgeojson - OSM to GeoJSON conversion compatible with osmtogeojson
  • osmpbf - stream processing of *.osm.pbf files
  • osmxml - stream processing of *.osm xml files
  • replication - fetch replication state and change files

Concepts

This package refers to the core OSM data types as Objects. The Node, Way, Relation, Changeset, Note and User types implement the osm.Object interface and can be referenced using the osm.ObjectID type. As a result it is possible to have a slice of []osm.Object that contains nodes, changesets and users.

Individual versions of the core OSM Map Data types are referred to as Elements and the set of versions for a give Node, Way or Relation is referred to as a Feature. For example, an osm.ElementID could refer to "Node with id 10 and version 3" and the osm.FeatureID would refer to "all versions of node with id 10." Put another way, features represent a road and how it's changed over time and an element is a specific version of that feature.

A number of helper methods are provided for dealing with features and elements. The idea is to make it easy to work with a Way and its member nodes, for example.

Scanning large data files

For small data it is possible to use the encoding/xml package in the Go standard library to marshal/unmarshal the data. This is typically done using the osm.OSM or osm.Change "container" structs.

For large data the package defines the Scanner interface implemented in both the osmxml and osmpbf sub-packages.

type osm.Scanner interface {
	Scan() bool
	Object() osm.Object
	Err() error
	Close() error
}

This interface is designed to mimic the bufio.Scanner interface found in the Go standard library.

Example usage:

f, err := os.Open("./delaware-latest.osm.pbf")
if err != nil {
	panic(err)
}
defer f.Close()

scanner := osmpbf.New(context.Background(), f, 3)
defer scanner.Close()

for scanner.Scan() {
	o := scanner.Object()
	// do something
}

scanErr := scanner.Err()
if scanErr != nil {
	panic(scanErr)
}

Note: Scanners are not safe for parallel use. One should feed the objects into a channel and have workers read from that.

Comments
  • Add support for node locations on ways

    Add support for node locations on ways

    Hi,

    I would like to add support for ways with node locations on them, an extension of the OSM PBF format put forward by @joto.

    This is the approach I have taken. I started by taking over the latest version of osmformat.proto, which has delta encoded packed sint64 fields numbered 9 (lat) and 10 (lon) in addition to the field 8 (refs) containing the IDs of all the nodes in the way. Then, after running protoc and taking a look at the current code for way nodes and node coordinates, I added similar logic for inserting the way node coordinates (if present) into the already existing WayNode.Lat and WayNode.Lon fields. Testing with a file generated by osmium add-locations-to-ways, this seems to give the desired result and processing regular OSM PBF files should be unaffected.

    Although it seems to me this pull request is OK, I would appreciate if you could look into it, especially since I don't yet feel very confident working with protocol buffers. Here are a few additional remarks:

    • When taking over the latest osmformat.proto, I have taken care not to overwrite the existing changes related to gogoprotobuf and to eliminate conversions from []byte to string.
    • Before all of the steps I mentioned so far, I have just run the protoc command from generate.go with no other changes. This already modified the generated Go files, probably because I am using the most recent version. Since I don't know what version was used previously, I stuck with these changes and kept them as the first commit.
    • The most important consequence of running protoc was the change of the package name from osmpbf to OSMPBF in the generated Go files. It did not seem a good idea to manually change the generated files, so I've chosen to replace the import paths to this internal package with the uppercase verision, as can be seen in the second commit.

    I'd love to hear your feedback on this!

  • Huge Memory Usage

    Huge Memory Usage

    Hi there. First off, thanks for this great library.

    When trying to scan a 700Mb *osm.pbf file, the mem usage spikes to over 145Gb with just a single go routine. I know this because I am trying this task in an AWS CodeBuild project with max memory allowed (145 Gb) and it still gets killed.

    This is surprising considering that the Osmosis CLI tool can accomplish the same task (scanning the 700Mb pbf file and filtering by a bounding box) without breaching 7Gb of memory.

    Why does this tool take over 20 times more memory than Osmosis, and is there anything we can do to improve this? I'm willing to help if it's a big task.

    Thanks.

  • Unmarshaling of Overpass JSON fails on version field

    Unmarshaling of Overpass JSON fails on version field

    The version information is encoded as JSON number instead of a string by Overpass API. This causes unmarshaling it to fail with error json: cannot unmarshal number into Go struct field .version of type string

    Take, for example, the minimal JSON output of overpass (http://overpass-api.de/api/interpreter?data=[out:json];out;), e.g.

    {
      "version": 0.6,
      "generator": "Overpass API 0.7.59 e21c39fe",
      "osm3s": {
        "timestamp_osm_base": "2022-11-17T17:14:39Z",
        "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
      },
      "elements": []
    }
    

    Minimum viable example failure: https://go.dev/play/p/n7kBY3CCbHC

    package main
    
    import (
    	"encoding/json"
    	"fmt"
    
    	"github.com/paulmach/osm"
    )
    
    func main() {
    	var overpass osm.OSM
    	buf := []byte(`{
      "version": 0.6,
      "generator": "Overpass API 0.7.59 e21c39fe",
      "osm3s": {
        "timestamp_osm_base": "2022-11-17T17:14:39Z",
        "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
      },
      "elements": []
    }`)
    	err := json.Unmarshal(buf, &overpass)
    	if err != nil {
    
    		fmt.Println(err)
    	}
    }
    
  • [Feature Request] Provide timestamp to replication sequence number function

    [Feature Request] Provide timestamp to replication sequence number function

    This enables us to reimplement this script in go. Your library is a great candidate to use when working with osm data.

    https://github.com/openstreetmap/osm2pgsql/blob/master/scripts/osm2pgsql-replication

  • Feature request: Specify which OSM entities should be read from the file

    Feature request: Specify which OSM entities should be read from the file

    Hello and thank you for this library.

    In libosmium, osmium::io::Reader can be initialized with osmium::osm_entities::bits to specify which OSM entities should be parsed and passed to the handler (docs, example). Specifying only the entities you want to handle does significantly improve read performance in my experience with libosmium.

    I believe it can be done without changing the current behavior by adding new initialization function, perhaps osmpbf.NewWithEntities.

    I appreciate your consideration.

  • Tags.Find() should return nil not

    Tags.Find() should return nil not ""

    OSM nodes can contain tags which specify only a key but no value. For example "building=" is common. Tags.Find() should return [nil] if no key is found, since returning "" could mean that either the tag is missing or that it is present but empty. It is common to want to catch even empty tags.

  • gogo/protobuf security issue

    gogo/protobuf security issue

    Thanks for a great project.

    There is a security issue detected by Dependabot https://github.com/gogo/protobuf/issues/752.

    All that is needed is to upgrade the gogo version to the latest one (v1.3.2 I believe) where this is fixed.

    As a side note, the gogo project is not really maintained anymore. Has anyone tested the performance of using the standard proto package?

  • osmpbf: support

    osmpbf: support "sparse" dense nodes

    Not all pbf files will have all the timestamp, username, changeset, etc. data so we should be okay with skipping that. No need to stop and return an error.

    https://github.com/paulmach/osm/issues/31

    @dmholtz does this work on your file?

  • How to handle dense nodes?

    How to handle dense nodes?

    Decoding OSM PBF files, which contain 'dense nodes' fails:

    panic: dense node does not include all required fields
    
    goroutine 1 [running]:
    main.main()
            /Users/[user]/repos/[repo]/cmd/importer/main.go:76 +0x468
    exit status 2
    

    The error occurs within the example code provided in the README.md of the osmpbf package.

    The error does not occur with OSM PBF file, which do not contain 'dense nodes' like Antarctica from Geofabrik

    Is this a bug or have I missed anything?

    After some research, it seems to me that another library just ignores the missing fields and the decoding suceeds. Thus, the issue seems to be closely related to my OSM PBF file (which I can unfortunately not share due to its size). It would be interesting to hear if your implementation supports a less strict mode, which also just ignores these fields.

  • Question: Why output and serializer chanel

    Question: Why output and serializer chanel

    Is there a specific reason to have in osmpbf part of the project n output chanels and feeding them in order to a serializer chanel ? (decode.go function line number

    The upside is it is making testing much easier since we get predictable outputs, however in my case I do not see not a reason that my application needs a special ordering of the elements : Do I miss something important why one should process objects in order ? (In other words: why doesn't the deocoder directly push to serializer)

  • protobuf: port to google protobuf

    protobuf: port to google protobuf

    I replaced gogo protobuf by the plain google protobuf implementation.

    I had to rewrite some portions regarding nullability. Hope I got it right, there might be better way to accomplish.

    Overall I was surprised that the CPU performance drop is below 3% , I was expecting much more. The number of allocations is much higher for some benchmarks, since I had to alloc temporary objects because of nullability changes above. The only one I am concerned is the change in memory for BenchmarkLondon_relations-16

    BenchmarkDiff_Marshal-16                     33708352       33497339       -0.63%
    BenchmarkDiff_Unmarshal-16                   66501681       65426347       -1.62%
    BenchmarkElementID_Sort-16                   648905         653604         +0.72%
    BenchmarkFeatureIDs_Sort-16                  636472         631671         -0.75%
    BenchmarkChange_MarshalXML-16                1790183        1744972        -2.53%
    BenchmarkChangeSet_Marshal-16                79831          77896          -2.42%
    BenchmarkUnmarshalWays-16                    5077           6401           +26.08%
    BenchmarkUnmarshalRelations-16               5508           6938           +25.96%
    BenchmarkChangeset_Marshal_gzip-16           450378         456373         +1.33%
    BenchmarkChangeset_UnmarshalXML-16           3468850        3504131        +1.02%
    BenchmarkUnmarshalChangeset-16               73306          81317          +10.93%
    BenchmarkUnmarshalChangeset_gzip-16          137123         144377         +5.29%
    BenchmarkRelations-16                        638028         633393         -0.73%
    BenchmarkWay-16                              14600          14479          -0.83%
    BenchmarkWays-16                             23030          22524          -2.20%
    BenchmarkConvert-16                          1211337        1202652        -0.72%
    BenchmarkConvertAnnotated-16                 947576         940853         -0.71%
    BenchmarkConvert_NoID-16                     1084547        1063003        -1.99%
    BenchmarkConvert_NoMeta-16                   996524         991362         -0.52%
    BenchmarkConvert_NoRelationMembership-16     1157384        1148647        -0.75%
    BenchmarkConvert_NoIDsMetaMembership-16      747993         748889         +0.12%
    BenchmarkLondon-16                           257786472      262947900      +2.00%
    BenchmarkLondon_withFiltersTrue-16           266369759      269555246      +1.20%
    BenchmarkLondon_withFiltersFalse-16          208942287      206641137      -1.10%
    BenchmarkLondon_nodes-16                     189334304      192258210      +1.54%
    BenchmarkLondon_ways-16                      147202951      150169294      +2.02%
    BenchmarkLondon_relations-16                 90644410       93034313       +2.64%
    BenchmarkAndorra-16                          3074774010     3111769317     +1.20%
    
    benchmark                                    old allocs     new allocs     delta
    BenchmarkDiff_Marshal-16                     129484         129484         +0.00%
    BenchmarkDiff_Unmarshal-16                   422916         422916         +0.00%
    BenchmarkElementID_Sort-16                   1              1              +0.00%
    BenchmarkFeatureIDs_Sort-16                  1              1              +0.00%
    BenchmarkChange_MarshalXML-16                4385           4385           +0.00%
    BenchmarkChangeSet_Marshal-16                291            242            -16.84%
    BenchmarkUnmarshalWays-16                    74             97             +31.08%
    BenchmarkUnmarshalRelations-16               81             105            +29.63%
    BenchmarkChangeset_Marshal_gzip-16           315            265            -15.87%
    BenchmarkChangeset_UnmarshalXML-16           23326          23326          +0.00%
    BenchmarkUnmarshalChangeset-16               346            617            +78.32%
    BenchmarkUnmarshalChangeset_gzip-16          390            660            +69.23%
    BenchmarkRelations-16                        1417           1417           +0.00%
    BenchmarkWay-16                              126            126            +0.00%
    BenchmarkWays-16                             116            116            +0.00%
    BenchmarkConvert-16                          10114          10114          +0.00%
    BenchmarkConvertAnnotated-16                 10107          10107          +0.00%
    BenchmarkConvert_NoID-16                     8501           8501           +0.00%
    BenchmarkConvert_NoMeta-16                   6898           6898           +0.00%
    BenchmarkConvert_NoRelationMembership-16     9531           9531           +0.00%
    BenchmarkConvert_NoIDsMetaMembership-16      4702           4703           +0.02%
    BenchmarkLondon-16                           4805957        4811479        +0.11%
    BenchmarkLondon_withFiltersTrue-16           4805954        4811469        +0.11%
    BenchmarkLondon_withFiltersFalse-16          1362152        1367677        +0.41%
    BenchmarkLondon_nodes-16                     3392981        3398512        +0.16%
    BenchmarkLondon_ways-16                      1793525        1799043        +0.31%
    BenchmarkLondon_relations-16                 457585         463104         +1.21%
    BenchmarkAndorra-16                          13038438       13038444       +0.00%
    
    benchmark                                    old bytes     new bytes     delta
    BenchmarkDiff_Marshal-16                     11232715      11232715      +0.00%
    BenchmarkDiff_Unmarshal-16                   15975454      15975430      -0.00%
    BenchmarkElementID_Sort-16                   24            24            +0.00%
    BenchmarkFeatureIDs_Sort-16                  24            24            +0.00%
    BenchmarkChange_MarshalXML-16                593113        593113        +0.00%
    BenchmarkChangeSet_Marshal-16                68687         37545         -45.34%
    BenchmarkUnmarshalWays-16                    6888          6984          +1.39%
    BenchmarkUnmarshalRelations-16               7248          7536          +3.97%
    BenchmarkChangeset_Marshal_gzip-16           891254        860106        -3.49%
    BenchmarkChangeset_UnmarshalXML-16           1131472       1131467       -0.00%
    BenchmarkUnmarshalChangeset-16               148593        173889        +17.02%
    BenchmarkUnmarshalChangeset_gzip-16          225217        250481        +11.22%
    BenchmarkRelations-16                        245465        245441        -0.01%
    BenchmarkWay-16                              9622          9622          +0.00%
    BenchmarkWays-16                             11640         11639         -0.01%
    BenchmarkConvert-16                          913288        913285        -0.00%
    BenchmarkConvertAnnotated-16                 830581        830577        -0.00%
    BenchmarkConvert_NoID-16                     890772        890778        +0.00%
    BenchmarkConvert_NoMeta-16                   702989        703007        +0.00%
    BenchmarkConvert_NoRelationMembership-16     895883        895846        -0.00%
    BenchmarkConvert_NoIDsMetaMembership-16      663290        663296        +0.00%
    BenchmarkLondon-16                           918437913     938188674     +2.15%
    BenchmarkLondon_withFiltersTrue-16           918437519     938187697     +2.15%
    BenchmarkLondon_withFiltersFalse-16          360101121     379852203     +5.48%
    BenchmarkLondon_nodes-16                     613038341     632790024     +3.22%
    BenchmarkLondon_ways-16                      432007871     451758126     +4.57%
    BenchmarkLondon_relations-16                 178275503     198026162     +11.08%
    BenchmarkAndorra-16                          471254696     471255184     +0.00%```
A go library for reading and creating ISO9660 images

iso9660 A package for reading and creating ISO9660, forked from https://github.com/kdomanski/iso9660. Requires Go 1.13 or newer. Joliet and Rock Ridge

Mar 4, 2021
The android-go project provides a platform for writing native Android apps in Go programming language.
The android-go project provides a platform for writing native Android apps in Go programming language.

android-go The android-go project aims to provide a platform (namely an SDK) for writing native Android apps in Go programming language. All things he

Jan 5, 2023
A toy ray tracer to practice writing go.

pbrt-go A toy ray tracer written to practice writing go. Slowly working towards an implementation based on the excellent 3rd edition of PBRT, because

Oct 19, 2021
Publish Your GIS Data(Vector Data) to PostGIS and Geoserver
Publish Your GIS Data(Vector Data) to PostGIS and Geoserver

GISManager Publish Your GIS Data(Vector Data) to PostGIS and Geoserver How to install: go get -v github.com/hishamkaram/gismanager Usage: testdata fol

Sep 26, 2022
A repository for plotting and visualizing data

Gonum Plot gonum/plot is the new, official fork of code.google.com/p/plotinum. It provides an API for building and drawing plots in Go. Note that this

Dec 27, 2022
An API which allows you to upload an image and responds with the same image, stripped of EXIF data

strip-metadata This is an API which allows you to upload an image and responds with the same image, stripped of EXIF data. How to run You need to have

Nov 25, 2021
Decode embedded EXIF meta data from image files.

goexif Provides decoding of basic exif and tiff encoded data. Still in alpha - no guarantees. Suggestions and pull requests are welcome. Functionality

Dec 17, 2022
Efficient moving window for high-speed data processing.

Moving Window Data Structure Copyright (c) 2012. Jake Brukhman. ([email protected]). All rights reserved. See the LICENSE file for BSD-style license. I

Sep 4, 2022
This is old and unmaintained code, ignore it. starfish is a simple, SDL based, 2D graphics and user input library for Go. If you intend to work on it, please fork from the 'devel' branch, not 'master'. Current release: 0.12.0

What is starfish? What starfish is: starfish is a simple 2D graphics and user input library for Go built on SDL. What starfish is not: While it is bui

Jun 4, 2019
Image processing library and rendering toolkit for Go.

blend Image processing library and rendering toolkit for Go. (WIP) Installation: This library is compatible with Go1. go get github.com/phrozen/blend

Nov 11, 2022
A lightning fast image processing and resizing library for Go

govips A lightning fast image processing and resizing library for Go This package wraps the core functionality of libvips image processing library by

Jan 8, 2023
A library to read, write, and transform Stereolithography (.stl) files in Go.

stl A library to read, write, and transform Stereolithography (.stl) files in Go. It is used in the command line STL manipulation tool stltool. Featur

Sep 26, 2022
Go cross-platform glfw library for creating an OpenGL context and receiving events.

glfw Package glfw experimentally provides a glfw-like API with desktop (via glfw) and browser (via HTML5 canvas) backends. It is used for creating a G

Sep 27, 2022
go-pix is a Go library for generating Pix transactions using Copy and Paste or QR codes. 💳 💰
go-pix is a Go library for generating Pix transactions using Copy and Paste or QR codes. 💳 💰

go-pix go-pix is a Go library for generating Pix transactions using Copy and Paste or QR codes.

Sep 12, 2022
JPEG-MPO Decoder / Converter Library and CLI Tool

MPO Decoder Library Simple Go JPEG MPO (Multi Picture Object) Decoder - Library and CLI Tool The library and CLI tool contain the ability to convert M

Sep 26, 2022
geoserver is a Go library for manipulating a GeoServer instance via the GeoServer REST API.
geoserver is a Go library for manipulating a GeoServer instance via the GeoServer REST API.

Geoserver geoserver Is a Go Package For Manipulating a GeoServer Instance via the GeoServer REST API. How to install: go get -v gopkg.in/hishamkaram/g

Dec 22, 2022
S2 geometry library in Go

Overview S2 is a library for spherical geometry that aims to have the same robustness, flexibility, and performance as the best planar geometry librar

Jan 8, 2023
Go package for fast high-level image processing powered by libvips C library

bimg Small Go package for fast high-level image processing using libvips via C bindings, providing a simple programmatic API. bimg was designed to be

Jan 2, 2023
Go bindings to OpenGL Utility Library

GLU This package offers minimal bindings for GLU functions. Usage go get github.com/go-gl-legacy/glu License Copyright 2012 The go-gl Authors. All ri

Aug 18, 2018