Rdebug — Real Debugger

RDebug - Real Debugger

Gitter License

Translation

1. Introduction

Rdebug is an open source tool chain that focusing on efficiency of daily developing, testing and debugging for developers. The code will be robust.

1.1 Background

As micro-service architecture makes services isomerous, easy-to-deploy and superbly extensible, large systems are divided into several micro services.

But, micro-service architecture is not a silver bullet. It has several drawbacks. One service is divided into several small services, test environment becomes hard to maintain. Unit test needs to mock downstream services. This will downgrade the efficiency of developing, and increase production bugs.

We do believe this industry needs an evolution.

1.2 Principle

Efficiency improvement, release cycle reduction, code quality improvement, production bug reduction

1.3 When to use

Suitable for old api refactoring that traffic have already been recorded.

Not suitable for new api or api that has no traffic recorded.

Support for new api is under investigation.

2. Quick Started

Traffic Recording

# Modify php-fpm config, set `clear_env = no`

# Compile koala-libc.so & koala-recorder.so

# Start php-fpm with koala-libc.so & koala-recorder.so
# Set environment first
$ export KOALA_SO=/path/to/koala-recorder.so 
$ export KOALA_RECORD_TO_DIR=/path/to/your-save-recorded-session-dir
$ export LC_CTYPE="C"

# Recording
# macOS
$ DYLD_INSERT_LIBRARIES="/path/to/koala-libc.so" DYLD_FORCE_FLAT_NAMESPACE="y" /path/to/sbin/php-fpm

# or, Linux
$ LD_PRELOAD="/path/to/koala-libc.so" /path/to/sbin/php-fpm

Traffic Replaying

Three ways to replay: replay from source, replay by midi.phar, replay by composer vendor/bin.

# Source
$ git clone https://github.com/didi/rdebug.git
$ cd rdebug/php/midi
$ sh install.sh
$ cd /path/to/your/project
$ /path/to/rdebug/php/midi/bin/midi run -f RECORD-SESSION-FILE

# Or, Phar
$ wget -O midi.phar -q https://github.com/didi/rdebug/raw/master/output/bin/midi.phar
$ midi.phar run -f RECORD-SESSION-FILE

# Or, Composer global
$ composer global require rdebug/midi
$ cd /path/to/your/project
$ ~/.composer/vendor/bin/midi run -f RECORD-SESSION-FILE

# Or, Composer
$ cd /path/to/your/project
$ composer require rdebug/midi --dev
$ ./vendor/bin/midi run -f RECORD-SESSION-FILE

PHP Example

3. Technical Details

We are using real production traffic to test on non-production environment. So, the core features of Rdebug are traffic recording and traffic replaying.

  • Traffic Recording

    Recording production traffic, including RPC between micro-services. The most difficult part is to trace requests that will span multiple services.

  • Traffic Replaying

    Replaying recorded production traffic against non-production environment, mock downstream traffic based on traffic matching. The most difficult part is interception and traffic matching.

koala-midi

3.1 Koala & Koala-libc

Koala and Koala-libc is the engine of Rdebug.

Because the problems such as RPC tracing, libc interception that a traffic replaying system may concern is what a traffic recording system concerns. The traffic recording and replaying system shares the same engine.

Koala

Koala is written in Go, libc contains some c++ code for interception part which will complied to koala-recorder.so and koala-replayer.so

Koala support replaying and recording separately as well as simultaneously.

For more details: koala

Koala-libc

koala-libc is written in C, and compiled to koala-libc.so

Go does not support fork. To avoid problems bring by PHP-FPM fork worker, you need to inject koala-libc.so into parent process of PHP. So that koala-libc.so will forward requests libc intercepted to *koala-recorder.so when accept in child process

For more details: koala-libc

3.2 Midi

Midi is a traffic replay client for PHP using command line written in PHP.

koala-replayer.so is embedded in Midi, and it can replay production traffic against new code, analyze test result, generate diff report, trace report, coverage report.

Midi support Xdebug.

For more details: midi

4. Compile

4.1 Requirement

Koala & Koala-libc

  • GCC >= 4.8
  • Go >= 1.8
  • Glide

Midi

  • macOS/Linux
  • PHP >= 7.0
  • Xdebug (Optional)
  • Composer (Optional)

4.2 Compile Koala-libc

$ cd koala-libc
$ sh build.sh

will generate ../output/libs/koala-libc.so

4.3 Compile Koala

$ cd koala

# install depends
$ sh build.sh vendor

# koala-recorder.so
$ sh build.sh recorder

# koala-replayer.so
$ sh build.sh

will generate ../output/libs/koala-recorder.so and ../output/libs/koala-replayer.so

4.4 Compile midi.phar

Before compile midi.pahr, recommend compile koala-replayer.so at your system and store in php/midi/res/replayer directory.

php/midi/res/replayer/koala-replayer.so in repository is for macOS.

$ cd php/midi
$ sh build.sh

will generate ../output/bin/midi.phar

Default phar will not include DiPlugin plugin, which is an internal plugin of didi.

If your want to compile with DiPlugin plugin:

$ cd php/midi
$ sh build.sh midi-diplugin

5. Usage

5.1 Traffic Recording

Record traffic in file or Elastic.

Set environment variable KOALA_RECORD_TO_DIR to save record to specified dir

Inject so. Use DYLD_INSERT_LIBRARIES in macOs or LD_PRELOAD in Linux

For more details: Recorder

5.1.1 macOS Recording

$ DYLD_INSERT_LIBRARIES="/path/to/koala-libc.so:/usr/lib/libcurl.dylib" DYLD_FORCE_FLAT_NAMESPACE="y" LC_CTYPE="C" KOALA_SO=/path/to/koala-recorder.so KOALA_RECORD_TO_DIR=/tmp /usr/local/sbin/php-fpm

5.1.2 Linux Recording

LD_PRELOAD="/path/to/koala-libc.so /usr/lib64/libcurl.so.4" LC_CTYPE="C" KOALA_SO="/path/to/koala-recorder.so" KOALA_RECORD_TO_DIR=/tmp /usr/local/sbin/php-fpm

Koala support writing to Elastic by replacing KOALA_RECORD_TO_DIR with KOALA_RECORD_TO_ES which value is elastic write url

5.2 Traffic Replay

Traffic replaying is to replay recorded traffic in production against non-production environment. You need only to deploy your code w/o deploying downstream services.

Below is a simple example of replaying with '-f', which specified the file to replay.

5.2.1 Replay

# Source
$ /path/to/rdebug/php/midi/bin/midi run -f RECORD-SESSION-FILE

# Or, phar
$ midi.phar run -f RECORD-SESSION-FILE

# Or, composer vendor bin
$ ./vendor/bin/midi -f RECORD-SESSION-FILE

For more details: Replay

Use -v, -vv or -vvv for more verbose log.

5.2.2 Report

Use -R-T-C to generate:Replay report, Trace report, Coverage report

report

report-upstream

report-coverage

For more details: Midi

6. More

6.1 Translation

6.2 Documentation

6.3 Contribute

Any contributions are welcome. More Details

6.4 Contract

  • Add Weixin

  • QQ Group

QQ

6.5 License

Rdebug is licensed under the Apache 2.0 License. LICENSE

6.6 Contributors

Special thanks to TaoWen Developer of Koala & Koala-libc.

Thanks to all the people who contribute.

Owner
DiDi
滴滴出行
DiDi
Comments
  • Can you make a Dockerfile?

    Can you make a Dockerfile?

    Can you make a Dockerfile and push the docker image to hub.docker.com?

    So I can quickly start testing , no need spend the time for Rdebug installation.

  • alpine 3.10编译报错

    alpine 3.10编译报错

    /go/src/github.com/didi/rdebug/koala # ./build.sh recorder
    +++ dirname ./build.sh
    ++ cd ../.
    ++ pwd -P
    + RDEBUG=/go/src/github.com/didi/rdebug
    + case $1 in
    + export GOPATH=/tmp/build-golang
    + GOPATH=/tmp/build-golang
    + export 'CGO_CFLAGS=-DKOALA_LIBC_NETWORK_HOOK -DKOALA_LIBC_FILE_HOOK'
    + CGO_CFLAGS='-DKOALA_LIBC_NETWORK_HOOK -DKOALA_LIBC_FILE_HOOK'
    + export 'CGO_CPPFLAGS=-DKOALA_LIBC_NETWORK_HOOK -DKOALA_LIBC_FILE_HOOK'
    + CGO_CPPFLAGS='-DKOALA_LIBC_NETWORK_HOOK -DKOALA_LIBC_FILE_HOOK'
    + export 'CGO_CXXFLAGS=-std=c++11 -Wno-ignored-attributes'
    + CGO_CXXFLAGS='-std=c++11 -Wno-ignored-attributes'
    + exec go build -tags=koala_recorder -buildmode=c-shared -o /go/src/github.com/didi/rdebug/output/libs/koala-recorder.so github.com/didi/rdebug/koala/cmd/recorder
    # github.com/didi/rdebug/koala/gateway/gw4libc
    In file included from file_hook.cpp:11:
    file_hook.cpp:73:11: error: redefinition of 'template<class ... Args> decltype (fopen(real::fopen::args ...)) real::fopen(Args ...)'
     INTERPOSE(fopen)(const char *filename, const char *opentype) {
               ^~~~~
    interpose.h:38:10: note: in definition of macro 'INTERPOSE'
         auto NAME(Args... args) -> decltype(::NAME(args...)) { \
              ^~~~
    file_hook.cpp:19:11: note: 'template<class ... Args> decltype (fopen(real::fopen::args ...)) real::fopen(Args ...)' previously declared here
     INTERPOSE(fopen64)(const char *filename, const char *opentype) {
               ^~~~~~~
    interpose.h:38:10: note: in definition of macro 'INTERPOSE'
         auto NAME(Args... args) -> decltype(::NAME(args...)) { \
              ^~~~
    file_hook.cpp:73:11: error: redefinition of 'FILE* fopen(const char*, const char*)'
     INTERPOSE(fopen)(const char *filename, const char *opentype) {
               ^~~~~
    interpose.h:49:31: note: in definition of macro 'INTERPOSE'
       extern "C" decltype(::NAME) NAME __attribute__((weak, alias("__interpose_" #NAME))); \
                                   ^~~~
    file_hook.cpp:19:11: note: 'FILE* fopen(const char*, const char*)' previously defined here
     INTERPOSE(fopen64)(const char *filename, const char *opentype) {
               ^~~~~~~
    interpose.h:49:31: note: in definition of macro 'INTERPOSE'
       extern "C" decltype(::NAME) NAME __attribute__((weak, alias("__interpose_" #NAME))); \
                                   ^~~~
    file_hook.cpp:101:11: error: redefinition of 'template<class ... Args> decltype (open(real::open::args ...)) real::open(Args ...)'
     INTERPOSE(open)(const char *filename, int flags, ...) {
               ^~~~
    interpose.h:38:10: note: in definition of macro 'INTERPOSE'
         auto NAME(Args... args) -> decltype(::NAME(args...)) { \
              ^~~~
    file_hook.cpp:47:11: note: 'template<class ... Args> decltype (open(real::open::args ...)) real::open(Args ...)' previously declared here
     INTERPOSE(open64)(const char *filename, int flags, mode_t mode) {
               ^~~~~~
    interpose.h:38:10: note: in definition of macro 'INTERPOSE'
         auto NAME(Args... args) -> decltype(::NAME(args...)) { \
              ^~~~
    file_hook.cpp:101:11: error: redefinition of 'int open(const char*, int, ...)'
     INTERPOSE(open)(const char *filename, int flags, ...) {
               ^~~~
    interpose.h:49:31: note: in definition of macro 'INTERPOSE'
       extern "C" decltype(::NAME) NAME __attribute__((weak, alias("__interpose_" #NAME))); \
                                   ^~~~
    file_hook.cpp:47:11: note: 'int open(const char*, int, ...)' previously defined here
     INTERPOSE(open64)(const char *filename, int flags, mode_t mode) {
               ^~~~~~
    interpose.h:49:31: note: in definition of macro 'INTERPOSE'
       extern "C" decltype(::NAME) NAME __attribute__((weak, alias("__interpose_" #NAME))); \
                                   ^~~~
    
  • How to play back the frequently changing API interface

    How to play back the frequently changing API interface

    怎样去回放经常发生变动的API接口? 长久以来,我对这类的组件也和你们didi一样有同样的想法,但苦于下面两个问题无法实现,从而一直搁置。我看你们的文档上,并没有阐述对这个问题的解决,不知道是不是?

    1、针对于 /books POST (new Book) 和 接踵而来的 /books/:bookId PATCH 这样的接口,对前端是有感知的,但是记录下来,对于后端确实无感知的。 2、API接口发生微调的接口,即只要改变一点点数据结构,即之前记录全部无效。

    How to play back the API interface that changes frequently? For a long time, I have the same idea for this type of component as you did with didi, but I have been stuck with the following two problems. I see your document, and did not explain the solution to this problem, I don't know if it is?

    1. The interface for /books POST (new Book) and the incoming /books/:bookId PATCH is perceptual to the front end, but it is recorded and is not perceived by the back end.
    2. The interface of the API interface is fine-tuned, that is, as long as the data structure is changed a little, that is, all the previous records are invalid.
  • ERROR: ld.so

    ERROR: ld.so

    LD_PRELOAD="/usr/lib64/koala-libc.so /usr/lib64/libcurl.so.4" LC_CTYPE="C" KOALA_SO=/usr/lib64/koala-recorder.so KOALA_RECORD_TO_DIR=/tmp/rdebug/record/ /opt/remi/php72/root/usr/sbin/php-fpm

    ERROR: ld.so: object '/usr/lib64/koala-libc.so' from LD_PRELOAD cannot be preloaded: ignored.

  • Linux 使用 midi 回放多个本地文件时进程卡住

    Linux 使用 midi 回放多个本地文件时进程卡住

    概述

    Linux 下使用 midi run 命令加多个 -f 参数回放多个文件时会卡住,而逐个回放则没有问题

    环境

    CentOS 7.2 x64

    最小复现场景

    在同一环境下进行录制、回放

    1. 本地服务及 rdebug 正常部署,so 均在本地编译得到,php-fpm 子进程 lsof 包含 output/libs/koala-recorder.so, koala-libc.so
    2. 请求本地 http 服务,录制文件输出到 output/session/xxx (文件大小 100k)
    3. 拷贝上述文件到本地应用 public 目录下:cp output/session/xxx $my_project/public/1
    4. 使用当前环境编译的 rdebug/output/bin/midi.phar 进行回放(使用同一文件回放多次模拟,其实与回放不同文件效果相同)
    $ cd $my_project/public/
    $ ~/rdebug/output/bin/midi.phar run -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -f 1 -D
    Finally will replay 13 sessions.
    Progressing:  1/13 [>>>-------------------------]   7% 8.0 MiB
    Diff Result: Response Body No Diff!
    ...
    Progressing: 12/13 [>>>>>>>>>>>>>>>>>>>>>>>>>>--]  92% 10.0 MiB  # 每次都卡在第 12 个文件
    

    info

    • midi 进程,php -S 进程内存占用均在 100K 以内,cpu 占用正常
    • midi 进程 socket 信息
    TCP localhost:54747->localhost:5514 (ESTABLISHED)
    
    • php -S 进程 socket 信息
    TCP *:5514 (LISTEN)
    TCP localhost:5516 (LISTEN)
    TCP localhost:5515 (LISTEN)
    TCP localhost:5514->localhost:54747 (ESTABLISHED)
    TCP localhost:36718->localhost:5515 (ESTABLISHED)
    TCP localhost:5515->localhost:36718 (ESTABLISHED)
    

    似乎 localhost:54747 <----> localhost:5514 之间的交互出现了问题,阻塞后续请求

    调试过程

    确认是否与接口响应数据类型有关

    源码的默认处理是按照 json utf-8 实现的,但实际验证 xml 也可以进行回放

    • 回放接口响应为 json utf-8 的请求,文件大小 200k,稳定卡在第 34 个文件

    确认是否与接口响应大小有关

    发送响应结果较小的请求,生成录制文件大小 20k,重复上述步骤,每次回放 3000 个文件,尝试多次偶现卡在第 300, 500 个文件的情况

    确认是否与缓存有关

    修改 Midi/Config.yml 中 memory_limit 为 100G,重新编译后重复上述步骤,未能解决问题

    确认是否与垃圾回收有关

    修改 Midi/Config.yml 中 gc-global-status-timeout = 10s,重新编译后重复上述步骤,未能解决问题

    确认是否与 mock 请求超时有关

    修改 Midi/Config.yml 中 inbound-read-timeout = 5s,进程不再卡住,超时后提示 Seem Koala response empty, retry...

    ...
    Progressing: 12/13 [>>>>>>>>>>>>>>>>>>>>>>>>>>--]  92% 10.0 MiB
    Seems Koala response empty, retry ...
    Seems Koala response empty, retry ...
    Koala response empty: OK
    Progressing: 13/13 [>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 100% 10.0 MiB
    Seems Koala response empty, retry ...
    Seems Koala response empty, retry ...
    Koala response empty: OK
    Summary: Replayed 13 Sessions No DIFF. Spent 186 s.
    

    但未能解决问题,第 12, 13 个请求返回为空

    环境

    更换另一台 CentOS 7.2 机器,存在同样问题

  • Fix koala-replayer.so compile error

    Fix koala-replayer.so compile error

    When trying to compile koala base on "4.3 Compile Koala" in wiki, "cd koala; sh build.sh vendor; sh build.sh recorder" will run correctly with default "export GOPATH=/tmp/build-golang". But "sh build.sh" will fail becase lacking of this default GOPATH setting. Result in compile error like: https://github.com/didi/rdebug/issues/21

  • Koala matcher optimize

    Koala matcher optimize

    1、每一个重定向到 outbound 的请求,新增一个上下文,记录历史匹配情况,方便后面一些决策:譬如当有多个最高值时,优先取之前匹配上次数多的连接,尽量把匹配的请求落到一个录制的请求里 2、针对 http 协议的分词优化,增加 uri 和 path 的权重,解决出现匹配上不同 uri 的情况 3、调整了下代码位置,拆分出 chunk 和 cosin(代码内容未变) 4、拆分出 lexer,用于相识度匹配时的分词实现 5、去除 simulateHttp、Mysql 等,考虑到如果录制里没有相关请求,即使 simulate 也意义不大;去除后直接进行流量匹配

  • Fix buildActions bug

    Fix buildActions bug

    Intro

    As described in the comments, buildActions() in php/midi/src/Midi/Koala/ParseRecorded.php is using for fix http calloutbound when one connection come with multi actions. But can not adapt to condition like:

    • action1
      • request: GET /path/abc?...
      • response: (empty response)
    • action2
      • request: (empty request)
      • response: abc (response part1)
    • action3
      • request: (empty request)
      • response: def (response part2) ...

    abc, def are both part of whole HTTP response in only one session, it is split to multiple parts because syscall recv() need to receive twice at least to hold the entire response.

    When will it appear

    1. A large reponse(over 64kb) in Upstream Calls
    2. A chunked response in Upstream Calls with Transfer-Encoding:chunked in header

    Result

    Can not replay correctly. In the report, response part1 in action2 was matched with 100% similarity, but reponse in action3 was tagged with "Unknown" Protocol.

    Fix

    original: Reqeust = action1.request, Response = action2.response; now: Request = action1.request, Response = action2.response + action3.response + ... + actionN.response

Visualise Go program GC trace data in real time

This project is no longer maintained I'm sorry but I do not have the bandwidth to maintain this tool. Please do not send issues or PRs. Thank you. gcv

Dec 14, 2022
gosivy - Real-time visualization tool for Go process metrics
 gosivy - Real-time visualization tool for Go process metrics

Gosivy tracks Go process's metrics and plot their evolution over time right into your terminal, no matter where it's running on. It helps you understand how your application consumes the resources.

Nov 27, 2022
Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds
Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds

Hard Drive S.M.A.R.T Monitoring, Historical Trends & Real World Failure Thresholds

Jan 1, 2023
Log-generator - A simple CLI tool that generates near real logs for testing

Log-generator - A simple CLI tool that generates near real logs for testing

Jan 22, 2022
GAPID: Graphics API Debugger
GAPID: Graphics API Debugger

GAPID: Graphics API Debugger

Dec 24, 2022
Delve is a debugger for the Go programming language.
Delve is a debugger for the Go programming language.

The GitHub issue tracker is for bugs only. Please use the developer mailing list for any feature proposals and discussions. About Delve Installation L

Dec 29, 2022
Interactive Go interpreter and debugger with REPL, Eval, generics and Lisp-like macros

gomacro - interactive Go interpreter and debugger with generics and macros gomacro is an almost complete Go interpreter, implemented in pure Go. It of

Dec 30, 2022
ROFL eXtension Debugger - for League of Legends replay files

ROFLXD - ROFL eXtension Debugger This is an incomplete project. The original goal of this project was to convert GLR replays to a form that was playab

Nov 27, 2022
Delve is a debugger for the Go programming language.
Delve is a debugger for the Go programming language.

The GitHub issue tracker is for bugs only. Please use the developer mailing list for any feature proposals and discussions. About Delve Installation L

Dec 29, 2022
Edb - An eBPF program debugger

EDB (eBPF debugger) edb is a debugger(like gdb and dlv) for eBPF programs. Norma

Dec 31, 2022
A small tool that allows a process to ask a debugger to attach to it.

Client and server for a process to request attach by gdlv. These two packages allow a program to request that a debugger attach to it. The motivating

Feb 1, 2022
A LoRaWAN nodes' and network simulator that works with a real LoRaWAN environment (such as Chirpstack) and equipped with a web interface for real-time interaction.
A LoRaWAN nodes' and network simulator that works with a real LoRaWAN environment (such as Chirpstack) and equipped with a web interface for real-time interaction.

LWN Simulator A LoRaWAN nodes' simulator to simulate a LoRaWAN Network. Table of Contents General Info Requirements Installation General Info LWN Simu

Nov 20, 2022
Real-time Map displays real-time positions of public transport vehicles in Helsinki
Real-time Map displays real-time positions of public transport vehicles in Helsinki

Real-time Map Real-time Map displays real-time positions of public transport vehicles in Helsinki. It's a showcase for Proto.Actor - an ultra-fast dis

Nov 30, 2022
Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

Jan 5, 2023
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.

unitdb Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. As Unitdb satisfy the

Jan 1, 2023
Simple and easy to use client for stock market, forex and crypto data from finnhub.io written in Go. Access real-time financial market data from 60+ stock exchanges, 10 forex brokers, and 15+ crypto exchanges

go-finnhub Simple and easy to use client for stock, forex and crpyto data from finnhub.io written in Go. Access real-time market data from 60+ stock e

Dec 28, 2022
Real-time Geospatial and Geofencing
Real-time Geospatial and Geofencing

Tile38 is an open source (MIT licensed), in-memory geolocation data store, spatial index, and realtime geofence. It supports a variety of object types

Jan 4, 2023
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Realtime API Gateway Synchronize Your Clients Visit Resgate.io for guides, live demos, and resources. Resgate is a Go project implementing a realtime

Dec 31, 2022
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.

Beaver A Real Time Messaging Server. Beaver is a real-time messaging server. With beaver you can easily build scalable in-app notifications, realtime

Jan 1, 2023
Scalable real-time messaging server in language-agnostic way
Scalable real-time messaging server in language-agnostic way

Centrifugo is a scalable real-time messaging server in language-agnostic way. Centrifugo works in conjunction with application backend written in any

Jan 2, 2023