🔥 Continuous profiling platform — debug performance issues in your code!

We upgrade our argocd app and try to see the profile after adding a patch. We used pyroscope pull model to collect profiles but it looks like the displayed profiles are really outdated.

db.(*db).ListClusters function shouldn't exist and the stacktrace is the old version (before upgrade).

Compared with the profile I got from parca, the function name should be db.(*db).GetClusterServersByName.

I just don't understand why this case would happen in pyroscope?

Brief

• https://github.com/pyroscope-io/pyroscope/issues/1586

Changes

• add additional option sandwich to Flamegraph component
• add sandwich view to Flamegraph component

demo link : https://pr-1613.pyroscope.io/?query=rideshare-app-golang.alloc_objects%7B%7D

https://user-images.githubusercontent.com/47758224/198092939-83a6701f-a514-4474-a7f3-477cc06bb6a9.mov

Concerns

• not sure yet how to combine bottom nodes in callers flamegraph since flamegraph only knows how to grow from top to bottom(maybe you have any ideas?)
• we should have empty top lvl node to accumulate values and to "grow" flamegraph from top to bottom (or it's okay ?)

Brief

• add annotations plugin for timelines

Changes

• demo screencast:

https://user-images.githubusercontent.com/7372044/195332521-958413e6-03ca-4e25-bf0a-d060dd58b519.mp4

• demo: https://pr-1605.pyroscope.io/?query=rideshare-app-golang.alloc_objects%7B%7D

Concerns

Brief

• add possibility to sync timeline selection https://github.com/pyroscope-io/pyroscope/issues/1614

Changes

• UI which allows user to sync timeline selection when it's out of range
• DEMO https://pr-1615.pyroscope.io/

Brief

• https://github.com/pyroscope-io/pyroscope/issues/1481

Changes

• change diff table values from absolute to percent

demo link: https://pr-1697.pyroscope.io/comparison-diff?query=rideshare-app-golang.alloc_objects%7B%7D&rightQuery=rideshare-app-golang.alloc_objects%7B%7D&leftQuery=rideshare-app-golang.alloc_objects%7B%7D

https://user-images.githubusercontent.com/47758224/201096384-0e17789a-f533-4d0b-9598-f81ad5fc8b50.mov

Concerns

• im using .toFixed(2) and for some values that looks strange

i think we can either show smth like < 0.01 % like we do with absolute values or increase number of digits after comma

Brief

• docs.google

Changes

• add /exemplars/single page
• rename /tracing -> /exemplars/merge
• add heatmap component
• add selection to heatmap compoent
• fetch flamegraph with heatmap selected area values

Concerns

• resize bug

The pyroscope will take up a lot of CPU time when I start up the pyroscope-server. By CPU ppof I found that periodicTask take up a lot of CPU time. How can it be optimized?

Brief

• https://github.com/pyroscope-io/pyroscope/issues/1596

Changes

• add gif and heatmap action description
• show gif when heatmap has no selected area

demo link: https://pr-1658.pyroscope.io/exemplars/single?query=rideshare-app-golang.cpu%7B%7D

https://user-images.githubusercontent.com/47758224/199260086-4c51eb68-9ae9-4809-852b-681772dd7c6a.mov

Concerns

Short summary on this PR We don't edit any code in Flot anymore. But there are cases when new functionality based on some built-in plugins. In case of upgrading time selection functionality I extracted react-flot/flot/jquery.flot.selection.min into a file ./TimelineChartSelection, where I customised existing logic in accordance with task requirements. It's important to note, that there are 2 types of timeline selection: double and single. All customisation was applied only for single type of selection. Double works in old fashion way (using built-in Flot selection, not plugin-built one).

I also extended in a bit TimelineChartWrapper API. Now you can set colors for selection boundaries and overlay. Adding colors depending on color mode is in progress

Fix on issue #257.

The duplicate request was caused by fetching the timeline inside FlameGraphRenderer component and It's parent. The issue was fixed by removing the FlameGraphRenderer fetching and pulling the data from the redux store, this change required the addition of extra keys in the redux store to scope the flamebearer data as discussed in the thread below to keep the the comparison view and comparison diff functionality.

Any suggestions or tips are welcome. Thanks.

Brief

• https://github.com/pyroscope-io/pyroscope/issues/1726

Changes

• add ability to show 2nd toolbar row
• add "more" button to open collapsed toolbar items

Hi,

We have pyroscope server version 0.34 that we only use to ingest jfr profiling data from 18 java apps through java-agent 0.10.2. In all apps the java agent configuration is the same:

export PYROSCOPE_SERVER_ADDRESS=http://mypyroscopeserver:4040
export PYROSCOPE_APPLICATION_NAME=myserver
export PYROSCOPE_FORMAT=jfr
export PYROSCOPE_PROFILER_ALLOC=0
export PYROSCOPE_PROFILER_LOCK=0
export PYROSCOPE_PROFILER_EVENT=wall

So that's a good amount of data.

The problem is that cpu consumption of JFR Parser is very very high.

Next screenshot from vmstat, top and lscpu

Next screenshot from pyroscope:

Is there any room for improvement in cpu consumption of jfr parser?

Thank you so much for your great work and product.

Best regards.

We need to connect our own agent to the pyroscope server. Our own agent produces flames in svg format. I hope someone can guide us on how to do it, thank you very much!

Environment:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G14042

$ python --version
Python 3.9.9

$ sysctl kern.version
kern.version: Darwin Kernel Version 17.7.0: Fri Oct 30 13:34:27 PDT 2020; root:xnu-4570.71.82.8~1/RELEASE_X86_64

Attempting to install pyroscope gives:

$ pip install pyroscope-io
Collecting pyroscope-io
  Using cached pyroscope-io-0.8.0.tar.gz (11 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting milksnakex>=0.1.6
  Using cached milksnakex-0.1.6-py2.py3-none-any.whl (10 kB)
Collecting cffi>=1.6.0
  Using cached cffi-1.15.1-cp310-cp310-macosx_10_9_x86_64.whl (179 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: pyroscope-io
  Building wheel for pyroscope-io (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pyroscope-io (pyproject.toml) did not run successfully.
  │ exit code: 101
  ╰─> [25 lines of output]
      /private/var/folders/tz/lx2rh_nj1853_wlh5ypgn9l80000gn/T/pip-build-env-z5xila5r/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:508: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
        warnings.warn(msg, warning_class)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-10.9-universal2-cpython-310
      creating build/lib.macosx-10.9-universal2-cpython-310/pyroscope_io
      copying pyroscope_io/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/pyroscope_io
      creating build/lib.macosx-10.9-universal2-cpython-310/pyroscope
      copying pyroscope/__init__.py -> build/lib.macosx-10.9-universal2-cpython-310/pyroscope
          Updating git repository `https://github.com/omarabid/read-process-memory.git`
      error: failed to get `ffikit` as a dependency of package `pyroscope_ffi v0.1.0 (/private/var/folders/tz/lx2rh_nj1853_wlh5ypgn9l80000gn/T/pip-install-_tw2u9re/pyroscope-io_248ab0dc47d4422eaf9cf9f5a90c3285/lib)`
      
      Caused by:
        failed to load source for dependency `ffikit`
      
      Caused by:
        Unable to update /private/var/folders/tz/lx2rh_nj1853_wlh5ypgn9l80000gn/T/pip-install-_tw2u9re/ffikit
      
      Caused by:
        failed to read `/private/var/folders/tz/lx2rh_nj1853_wlh5ypgn9l80000gn/T/pip-install-_tw2u9re/ffikit/Cargo.toml`
      
      Caused by:
        No such file or directory (os error 2)
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyroscope-io
Failed to build pyroscope-io
ERROR: Could not build wheels for pyroscope-io, which is required to install pyproject.toml-based projects

Feature Request

Support for GPU profiling similar to what's currently being offered for CPU.

Alternative solutions

1. Sentry's (OSS) OpenTelemetry Collector collects GPU metrics
2. Framework-specific like PyTorch Profiler or TensorFlow Profiler that could be incorporated into this product to provide it as a service out of the box.

Is there a use case or business reason for this request?

The CPU market is growing at a compound annual growth rate (CAGR) of 4.36% while the GPU market grows at a CAGR of 33.4%. Also NVIDIA has the biggest market share at 80%.

Hi,

In our production environment we have started to see next error after upgrade pyroscope server to 0.34.1-1:

[root@myserver ~]# systemctl status pyroscope-server.service
? pyroscope-server.service - Open source continuous profiling software
   Loaded: loaded (/usr/lib/systemd/system/pyroscope-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-12-05 20:56:27 CET; 1 day 14h ago
     Docs: https://pyroscope.io/
 Main PID: 25562 (pyroscope)
    Tasks: 14 (limit: 294571)
   Memory: 38.4G
   CGroup: /system.slice/pyroscope-server.service
           +-25562 /usr/bin/pyroscope server

Dec 07 11:23:01 myserver pyroscope[25562]: time="2022-12-07T11:23:01.694475" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="d2_myserver_3.lock_count{}"
Dec 07 11:23:09 myserver pyroscope[25562]: time="2022-12-07T11:23:09.900733" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="myserver.BPEL.alloc_in_new_tlab_bytes{}"
Dec 07 11:23:09 myserver pyroscope[25562]: time="2022-12-07T11:23:09.900782" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="myserver.BPEL.alloc_outside_tlab_bytes{}"
Dec 07 11:23:09 myserver pyroscope[25562]: time="2022-12-07T11:23:09.900796" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="myserver.BPEL.cpu{}"
Dec 07 11:23:09 myserver pyroscope[25562]: time="2022-12-07T11:23:09.900808" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="myserver.BPEL.lock_duration{}"
Dec 07 11:23:16 myserver pyroscope[25562]: time="2022-12-07T11:23:16.084630" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="AdminServer.alloc_in_new_tlab_bytes{}"
Dec 07 11:23:16 myserver pyroscope[25562]: time="2022-12-07T11:23:16.084675" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="AdminServer.alloc_outside_tlab_bytes{}"
Dec 07 11:23:16 myserver pyroscope[25562]: time="2022-12-07T11:23:16.084697" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="AdminServer.itimer{}"
Dec 07 11:23:16 myserver pyroscope[25562]: time="2022-12-07T11:23:16.084711" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="d2_myserver_2.lock_duration{}"
Dec 07 11:23:16 myserver pyroscope[25562]: time="2022-12-07T11:23:16.084728" level=error msg="error happened while ingesting data" file=" storage/queue.go:86" error="could not write because of retention settings" key="d2_myserver_2.wall{}"

Our retention policy (while storage in S3 is being developed ;)) is:

cat /etc/pyroscope/server.yml
---
retention-levels:
 0: 72h  # 3 days
 1: 144h # 6 days
 2: 216h # 9 days

PD: Keep going. You are doing incredibly well. Pyroscope is a great software.

Best regards.