Ruuvi-go-gateway - Software replica of the Ruuvi Gateway

Last update: Dec 21, 2022

Comments: 5

ruuvi-go-gateway

ruuvi-go-gateway is a software that tries to replicate Ruuvi Gateway MQTT and HTTP POST (for custom servers) features so that users without a physical Ruuvi Gateway can still use software and tools created for the gateway, such as RuuviBridge.

Features

Supports publishing BLE data to MQTT in real time in same format as Ruuvi Gateway
Supports sending latest BLE data via HTTP POST in same format as Ruuvi Gateway
Can send either just Ruuvi data or all scanned BLE data (configurable, like with the Gateway)

Requirements

Linux-based OS (the bluetooth stack varies too greatly between operating systems and it would be simply too much work to support all of them separately)
Bluetooth adapter supporting Bluetooth Low Energy
10MB of disk space
20MB of RAM (typical usage around 10MB)

Configuration

Check config.sample.yml for a sample config. By default the gateway assumes to find a file called config.yml in the current working directory, but that can be overridden with -config /path/to/config.yml command line flag.

Installation

Recommended method is using Docker with the prebuilt dockerimage: ghcr.io/scrin/ruuvi-go-gateway for which you can use the provided composefile

Without docker you can download prebuilt binaries from the releases page. For production use it's recommended to set up as a service.

Note that running the standalone binaries without root requires some extra capabilities be set to the binary to grant it permissions to scan for ble, this can be done with:

sudo setcap 'cap_net_raw,cap_net_admin+eip' ./ruuvi-go-gateway

Owner

Scrin

I code stuff.

https://github.com/Scrin/ruuvi-go-gateway

Comments

No Ruuvitags detected

I have this running in docker on a Raspberry PI 4 using the sample docker compose file. I have 6 Ruuvitags nearby (all of which are detectable using the native Ruuvi iPhone app).

When I turn on trace level debug I can see lots of BLE advertisements but none are detected as being for the Ruuvitags. For example:

[2022-7-12 08:46:55] [trace] [/go/src/github.com/Scrin/ruuvi-go-gateway/gateway/gateway.go] Received data from BLE adapter, is_ruuvi: false, data: 4C0003161100000277C0A856200000000000000000000000000B, mac: 0D:72:48:9F:69:10, rssi: -59

[2022-7-12 08:46:55] [trace] [/go/src/github.com/Scrin/ruuvi-go-gateway/gateway/gateway.go] Received data from BLE adapter, is_ruuvi: false, data: 4C0002152686F39CBADA4658854AA62E7E5E8B8D000100000B, mac: 05:3E:AB:3B:0D:ED, rssi: -61

[2022-7-12 08:46:55] [trace] [/go/src/github.com/Scrin/ruuvi-go-gateway/gateway/gateway.go] Received data from BLE adapter, is_ruuvi: false, data: A705051001000000000000022200CA, mac: F3:33:A1:94:48:35, rssi: -66

So, I assume the BLE stack is working ok. If I change all_advertisements to true, I can see MQTT posts for the detected devices. So MQTT is working too. My tags are using data format 5 and running recent firmware.

Have you any idea why the gateway would see everything EXCEPT my Ruuvitags?

Possible Race Condition? "concurrent map iteration and map write"

Greetings, thanks for making this! I'm trying to use it to collect data from my Ruuvi sensors. I've attempted to run it overnight twice now and woken up to this panic.

./ruuvi-go-gateway 
fatal error: concurrent map iteration and map write

goroutine 6 [running]:
runtime.throw({0x7531e3, 0x45ebd9})
	/usr/local/go/src/runtime/panic.go:1198 +0x71 fp=0xc0000bb870 sp=0xc0000bb840 pc=0x434951
runtime.mapiternext(0x19)
	/usr/local/go/src/runtime/map.go:858 +0x4eb fp=0xc0000bb8e0 sp=0xc0000bb870 pc=0x410ccb
reflect.mapiternext(0x95)
	/usr/local/go/src/runtime/map.go:1346 +0x19 fp=0xc0000bb8f8 sp=0xc0000bb8e0 pc=0x45eb19
reflect.(*MapIter).Next(0xc0002afe40)
	/usr/local/go/src/reflect/value.go:1618 +0x99 fp=0xc0000bb918 sp=0xc0000bb8f8 pc=0x485c79
encoding/json.mapEncoder.encode({0x6ddee0}, 0xc000036080, {0x6ef100, 0xc0005383b0, 0xc0002aef80}, {0x7, 0x0})
	/usr/local/go/src/encoding/json/encode.go:799 +0x305 fp=0xc0000bbad8 sp=0xc0000bb918 pc=0x514e05
encoding/json.mapEncoder.encode-fm(0x724700, {0x6ef100, 0xc0005383b0, 0x30}, {0x3c, 0x77})
	/usr/local/go/src/encoding/json/encode.go:779 +0x45 fp=0xc0000bbb18 sp=0xc0000bbad8 pc=0x51df45
encoding/json.structEncoder.encode({{{0xc0000cad80, 0xc0000bbc50, 0x4b1af9}, 0xc0001cf8c0}}, 0xc000036080, {0x724700, 0xc000538370, 0x7}, {0x0, 0x1})
	/usr/local/go/src/encoding/json/encode.go:761 +0x1f4 fp=0xc0000bbbc8 sp=0xc0000bbb18 pc=0x514854
encoding/json.structEncoder.encode-fm(0x6fc180, {0x724700, 0xc000538370, 0x475069}, {0xc0, 0xe2})
	/usr/local/go/src/encoding/json/encode.go:732 +0x69 fp=0xc0000bbc20 sp=0xc0000bbbc8 pc=0x51de89
encoding/json.structEncoder.encode({{{0xc0001962d0, 0xc0000bbd80, 0x4b7496}, 0xc0001cf920}}, 0xc000036080, {0x6fc180, 0xc000538370, 0x0}, {0x0, 0x1})
	/usr/local/go/src/encoding/json/encode.go:761 +0x1f4 fp=0xc0000bbcd0 sp=0xc0000bbc20 pc=0x514854
encoding/json.structEncoder.encode-fm(0x6fc180, {0x6fc180, 0xc000538370, 0x700d80}, {0x0, 0x30})
	/usr/local/go/src/encoding/json/encode.go:732 +0x69 fp=0xc0000bbd28 sp=0xc0000bbcd0 pc=0x51de89
encoding/json.(*encodeState).reflectValue(0xc0000bbdc0, {0x6fc180, 0xc000538370, 0x40e2d4}, {0xc0, 0x8d})
	/usr/local/go/src/encoding/json/encode.go:360 +0x78 fp=0xc0000bbd88 sp=0xc0000bbd28 pc=0x5123f8
encoding/json.(*encodeState).marshal(0xc000538370, {0x6fc180, 0xc000538370}, {0x50, 0xbe})
	/usr/local/go/src/encoding/json/encode.go:332 +0xfa fp=0xc0000bbe00 sp=0xc0000bbd88 pc=0x51201a
encoding/json.Marshal({0x6fc180, 0xc000538370})
	/usr/local/go/src/encoding/json/encode.go:161 +0x45 fp=0xc0000bbe60 sp=0xc0000bbe00 pc=0x511805
github.com/Scrin/ruuvi-go-gateway/sender.SetupHTTP.func1()
	/github/workspace/sender/http.go:57 +0x1cf fp=0xc0000bbfe0 sp=0xc0000bbe60 pc=0x69b08f
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1581 +0x1 fp=0xc0000bbfe8 sp=0xc0000bbfe0 pc=0x4646a1
created by github.com/Scrin/ruuvi-go-gateway/sender.SetupHTTP
	/github/workspace/sender/http.go:46 +0x2e8

goroutine 1 [chan receive (nil chan), 870 minutes]:
github.com/go-ble/ble/linux.(*Device).Scan(0xc000021ae0, {0x7bcf30, 0xc00002c638}, 0x40, 0x0)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/device.go:178 +0x8b
github.com/go-ble/ble.Scan({0x7bcf30, 0xc00002c638}, 0x70, 0xc000196240, 0x0)
	/go/pkg/mod/github.com/go-ble/[email protected]/gatt.go:92 +0x187
github.com/Scrin/ruuvi-go-gateway/gateway.Run({{0xc00002a330, 0x11}, 0x0, 0x0, 0xc0000c3730, 0xc00008c680, {{0xc00002ce60, 0x6}, {0xc00002ce78, 0x4}, ...}, ...})
	/github/workspace/gateway/gateway.go:67 +0x505
main.main()
	/github/workspace/cmd/ruuvi-go-gateway/main.go:33 +0x418

goroutine 7 [syscall]:
syscall.Syscall(0x0, 0x3, 0xc0001ac000, 0x1000)
	/usr/local/go/src/syscall/asm_linux_amd64.s:20 +0x5
golang.org/x/sys/unix.read(0x44b0c5, {0xc0001ac000, 0x0, 0xc0000f4300})
	/go/pkg/mod/golang.org/x/[email protected]/unix/zsyscall_linux.go:1215 +0x4d
golang.org/x/sys/unix.Read(...)
	/go/pkg/mod/golang.org/x/[email protected]/unix/syscall_unix.go:157
github.com/go-ble/ble/linux/hci/socket.(*Socket).Read(0xc000088680, {0xc0001ac000, 0x1000, 0xc0001ac000})
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/socket/socket.go:123 +0x75
github.com/go-ble/ble/linux/hci.(*HCI).sktLoop(0xc0000b1040)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/hci.go:300 +0xc4
created by github.com/go-ble/ble/linux/hci.(*HCI).Init
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/hci.go:153 +0x5f2

goroutine 8 [select, 870 minutes]:
github.com/go-ble/ble/linux/hci.(*HCI).Accept(0xc0000b1040)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/hci/gap.go:185 +0xad
github.com/go-ble/ble/linux.loop(0x0, 0xc00008c8c0, 0x0)
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/device.go:55 +0x4a
created by github.com/go-ble/ble/linux.NewDeviceWithNameAndHandler
	/go/pkg/mod/github.com/go-ble/[email protected]/linux/device.go:48 +0x1fd

goroutine 1754812 [select]:
net/http.(*persistConn).writeLoop(0xc00008b7a0)
	/usr/local/go/src/net/http/transport.go:2386 +0xfb
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1748 +0x1e65

goroutine 1754811 [IO wait]:
internal/poll.runtime_pollWait(0x7f4a91a85798, 0x72)
	/usr/local/go/src/runtime/netpoll.go:229 +0x89
internal/poll.(*pollDesc).wait(0xc0008cc200, 0xc0005bc000, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0008cc200, {0xc0005bc000, 0x1000, 0x1000})
	/usr/local/go/src/internal/poll/fd_unix.go:167 +0x25a
net.(*netFD).Read(0xc0008cc200, {0xc0005bc000, 0x0, 0x0})
	/usr/local/go/src/net/fd_posix.go:56 +0x29
net.(*conn).Read(0xc00020c040, {0xc0005bc000, 0xe, 0x0})
	/usr/local/go/src/net/net.go:183 +0x45
net/http.(*persistConn).Read(0xc00008b7a0, {0xc0005bc000, 0x447380, 0xc000217ec8})
	/usr/local/go/src/net/http/transport.go:1926 +0x4e
bufio.(*Reader).fill(0xc00020e5a0)
	/usr/local/go/src/bufio/bufio.go:101 +0x103
bufio.(*Reader).Peek(0xc00020e5a0, 0x1)
	/usr/local/go/src/bufio/bufio.go:139 +0x5d
net/http.(*persistConn).readLoop(0xc00008b7a0)
	/usr/local/go/src/net/http/transport.go:2087 +0x1ac
created by net/http.(*Transport).dialConn
	/usr/local/go/src/net/http/transport.go:1747 +0x1e05

Configuration option for Bluetooth device
The BLE scanner uses the system default device (https://github.com/Scrin/ruuvi-go-gateway/blob/master/gateway/gateway.go#L35).

In my case I have multiple bluetooth adapters and would like to configure which adapter to use for the gateway:

# hcitool dev Devices: hci0 XX:XX:XX:XX:XX:XX hci1 YY:YY:YY:YY:YY:YY
Feature Request - MQTT Will Message

Would it be possible to enhance the MQTT connection to send a Will Message on connection?

Ideally the message payload and topic could be set in the config file.

Thanks for considering.
Rate Limit

Would it be possible to add a configuration to set a rate limit for tag readings?

Maybe something like "only send readings if they are x seconds older than the last reading" (for any sensor/metric).

Related tags

Microservices ruuvi-go-gateway

HTTP API Gateway

Manba/简体中文 Manba is a restful API gateway based on HTTP, which can be used as a unified API access layer. Tutorial A very detailed tutorial for beginn

Dec 29, 2022

This is a shopping basket workshop that shows how to use KrakenD API Gateway.

Go Restful Microservices and KrakenD API Gateway Workshop This is a shopping basket workshop that shows how to use KrakenD API Gateway. Consist of 5 m

Jan 3, 2023

A microservice gateway developed based on golang.With a variety of plug-ins which can be expanded by itself, plug and play. what's more,it can quickly help enterprises manage API services and improve the stability and security of API services.

Goku API gateway is a microservice gateway developed based on golang. It can achieve the purposes of high-performance HTTP API forwarding, multi tenant management, API access control, etc. it has a powerful custom plug-in system, which can be expanded by itself, and can quickly help enterprises manage API services and improve the stability and security of API services.

Dec 29, 2022

TiDB Gateway

TiDB Gateway Building go build Running ./tidbgw Optionally set the address of PD and address to listen on via flags. Using Connect your MySQL client

Jul 15, 2022

Blue is a lightweight cloud-native gateway solution to handle millions of routing endpoints with a large number of connections.

Jan 19, 2022

Test your API gateway routed lambdas locally and in CI

Lambo Test API Gateway wrapped lambda functions locally. Lambo can also be used to test API GW lambdas in CI without needing docker-in-docker. It will

Jun 22, 2022

a simple api gateway

鉴权网关根据登录接口返回UIN判断是否登录/退出成功自动添加跨域处理会话重启即失效 TLS鉴权负载均衡随机 Config # 开启验证 enable_verify: false # 验证证书 ca_path: "./conf/ca.pem" module_cert_pem: "./conf

Feb 10, 2022

The Consul API Gateway is a dedicated ingress solution for intelligently routing traffic to applications running on a Consul Service Mesh.

Dec 14, 2022

A gateway to expose s3 standard API for any storage type.

s3-gateway this project is used to be compatible with any other storage type. there are tons of object storage services in the internat. However, many

Nov 28, 2021

Micro-serviço em Golang para processar pagamentos de um Gateway.

Go Payment Processor Projeto Este repositório contém um micro-serviço em Golang que faz parte de um conjunto de serviços necessário para o projeto do

Dec 13, 2021

A simple HTTP/HTTPS API Gateway in Golang.

mice A simple API Gateway in Golang. Installation: Using go install: First install Go, if not already there Set GOPATH and GOBIN if not already set as

May 19, 2022

Cortex Gateway: a microservice which strives to help you administrating and operating your Cortex Cluster in multi tenant environments

Cortex Gateway Cortex Gateway is a microservice which strives to help you administrating and operating your Cortex Cluster in multi tenant environment

Jan 14, 2022

Search-gateway - Golang restfull Service

Search Gateway Specifications Gin framework Development PORT=3000 go run main.go

Jan 25, 2022

A lightweight job scheduler based on priority queue with timeout, retry, replica, context cancellation and easy semantics for job chaining. Build for golang web apps.

Table of Contents Introduction What is RIO? Concern An asynchronous job processor Easy management of these goroutines and chaining them Introduction W

Dec 9, 2022

Litestream-read-replica-demo - A demo application for running live read replication on fly.io with Litestream

Litestream Read Replica Demo A demo application for running live read replicatio

Oct 18, 2022

protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

Dec 19, 2022