DataKit is collection agent for DataFlux.

Last update: Dec 29, 2022

Comments: 3

DataKit

DataKit is collection agent for DataFlux

Build

Dependencies

apt-get install gcc-multilib: for building oracle input
apt-get install tree: for Makefile usage
packr2: for packaging manuals
goyacc: for pipeline grammar generation

DataKit was designed to be build across Darwin, Linux and Windows. For example, you can build windows/amd64 release under Darwin:

# build windows exe on Mac
LOCAL=windows/amd64 make

# build linux binary on Mac
LOCAL=linux/amd64 make

# build on different OS/Arch, DataKit build tool will
# detect current OS/Arch and build the default one
make

and is's possible to huild all release under Darwin:

# build all os-arch platform under testing rule
make testing

All the output binary under dist:

dist
├  [4.0K]  datakit-linux-386
│   ├ [ 51M]  datakit
│   └ [4.0K]  externals
│       └ [8.5M]  oracle
├  [4.0K]  datakit-linux-amd64
│   ├ [ 60M]  datakit
│   └ [4.0K]  externals
│       └ [9.9M]  oracle
├  [4.0K]  datakit-linux-arm
│   └ [ 51M]  datakit
├  [4.0K]  datakit-linux-arm64
│   └ [ 56M]  datakit
├  [4.0K]  datakit-windows-386
│   └ [ 52M]  datakit.exe
├  [4.0K]  datakit-windows-amd64
│   └ [ 60M]  datakit.exe
└  [4.0K]  test
    ├  [ 12M]  installer-linux-386
    ├  [ 15M]  installer-linux-amd64
    ├  [ 12M]  installer-linux-arm
    ├  [ 14M]  installer-linux-arm64
    ├  [ 13M]  installer-windows-386.exe
    ├  [ 15M]  installer-windows-amd64.exe
    └  [ 204]  version

9 directories, 15 files

Note: The Darwin release can not be build under Linux/Windows, because we applied CGO for Darwin release. BTW, Windows lack(default) of many build tools(such as make), we still recommand to build under Linux and Darwin.

More references

Datakit-How-TO
Datakit Install
- On host
- DaemonSet

Similar Resources

Simple and extensible monitoring agent / library for Kubernetes: https://gravitational.com/blog/monitoring_kubernetes_satellite/

Satellite Satellite is an agent written in Go for collecting health information in a kubernetes cluster. It is both a library and an application. As a

Nov 10, 2022

The mep-agent module provides proxy services for 3rd applications to MEP.

Mep-Agent Introduction Mep-Agent is a middleware that provides proxy services for third-party apps. It can help apps, which do not implement the ETSI

Mar 9, 2022

Very powerful server agent for collecting & sending logs & metrics with an easy-to-use web console.

logkit-community 中文版 Introduce Very powerful server agent for collecting & sending logs & metrics with an easy-to-use web console. logkit-community De

Dec 29, 2022

Agent-less vulnerability scanner for Linux, FreeBSD, Container, WordPress, Programming language libraries, Network devices

Vuls: VULnerability Scanner Vulnerability scanner for Linux/FreeBSD, agent-less, written in Go. We have a slack team. Join slack team Twitter: @vuls_e

Jan 9, 2023

Extended ssh-agent which supports git commit signing over ssh

ssh-agentx ssh-agentx Rationale Requirements Configuration ssh-agentx Configuration ssh-gpg-signer Linux Windows Signing commits after configuration T

Jun 29, 2022

Sign Container Images with cosign and Verify signature by using Open Policy Agent (OPA)

Sign Container Images with cosign and Verify signature by using Open Policy Agent (OPA) In the beginning, I believe it is worth saying that this proje

Nov 30, 2022

nano-gpu-agent is a Kubernetes device plugin for GPU resources allocation on node.

Nano GPU Agent About this Project Nano GPU Agent is a Kubernetes device plugin implement for gpu allocation and use in container. It runs as a Daemons

Dec 29, 2022

Go driver for PostgreSQL over SSH. This driver can connect to postgres on a server via SSH using the local ssh-agent, password, or private-key.

pqssh Go driver for PostgreSQL over SSH. This driver can connect to postgres on a server via SSH using the local ssh-agent, password, or private-key.

Nov 6, 2022

runtime - an abstraction library on top of the Open Policy Agent (OPA)

runtime - an abstraction library on top of the Open Policy Agent (OPA) Introduction The "runtime" project is a library that sits on top of OPA. The go

Nov 7, 2022

go-opa-validate is an open-source lib that evaluates OPA (open policy agent) policy against JSON or YAML data.

go-opa-validate go-opa-validate is an open-source lib that evaluates OPA (open policy agent) policy against JSON or YAML data. Installation Usage Cont

Nov 17, 2022

A plugin for running Open Policy Agent (OPA) in AWS Lambda as a Lambda Extension.

opa-lambda-extension-plugin A custom plugin for running Open Policy Agent (OPA) in AWS Lambda as a Lambda Extension. To learn more about how Lambda Ex

Jan 2, 2023

MySQL replication topology manager - agent (daemon)

orchestrator-agent MySQL topology agent (daemon) orchestrator-agent is a sub-project of orchestrator. It is a service that runs on MySQL hosts and com

Mar 8, 2022

A very simple ssh-agent that signs requests in parallel

ssh-agent A very simple ssh-agent that signs requests in parallel. Usage To install and run the agent simply run: $ go install github.com/Woellchen/ss

Dec 14, 2022

mesh-kridik is an open-source security scanner that performs various security checks on a Kubernetes cluster with istio service mesh and is leveraged by OPA (Open Policy Agent) to enforce security rules.

mesh-kridik Enhance your Kubernetes service mesh security !! mesh-kridik is an open-source security scanner that performs various security checks on a

Dec 14, 2022

Comments

1. 修复 Ubuntu cgroup 没有 memory.memsw.limit_in_bytes 文件时导致 cgroup 限制设置失败

安装环境: 腾讯云->宝塔Linux->Ubuntu Server 20.04 LTS 64bit

内核信息: Linux VM-16-15-ubuntu 5.4.0-126-generic #142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

安装 datakit 报错信息:

问题原因: https://github.com/containerd/cgroups 库去 open memory.memsw.limit_in_bytes 文件，由于 Ubuntu 没有开启内存 swap 所以 cgroup 没有该文件，open 导致报错 permission denied

修复参考: https://github.com/concourse/concourse/issues/6649

https://github.com/moby/moby/blob/4bbc52c04b83c4ab9c47f2404720fa0c6a4cff40/pkg/sysinfo/sysinfo_linux.go#L110-L113
Add license scan report and status
Your FOSSA integration was successful! Attached in this PR is a badge and license report to track scan status in your README.

Below are docs for integrating FOSSA license checks into your CI:

CircleCI

TravisCI

Jenkins

Other

installer-linux-arm64-1.1.9-rc7 failed at first try but succeeded at the second try

Host Environment: MacBook 12.0.1 (21A559) with Apple M1 Dockerfile: Attached in the thread installation logs: Attached below

How to reproduce:

replace the token with a valid one in docker file.
docker build . -t datakit
docker run -it --entrypoint bash datakit
./install.sh
echo $? # expect 1
./install.sh
echo $? #expect 0

root@dd2e01a6b885:/# ./install.sh
 Detect installer installer-linux-arm64-1.1.9-rc7

* Downloading installer /tmp/dk-installer
##################################################################################################################################################################################################### 100.0%

* Installing DataKit...
tee: /var/log/datakit/install.log: No such file or directory
2021-12-19T04:26:11.306Z	INFO	installer	installer/main.go:245	stoping datakit...
2021-12-19T04:26:11.310Z	WARN	installer	installer/main.go:247	stop service: Failed to stop datakit: exit status 1, ignored
2021-12-19T04:26:11.310Z	DEBUG	installer	installer/main.go:659	deprecated install path /usr/local/cloudcare/dataflux/datakit not exists, ignored
Downloading(datakit)... 17 MB/17 MB
Downloading(   data)... 34 MB/34 MB
2021-12-19T04:26:53.322Z	INFO	installer	installer/main.go:269	Installing version 1.1.9-rc7...
2021-12-19T04:26:53.322Z	WARN	installer	installer/main.go:348	uninstall service: Failed to uninstall datakit: remove /etc/init.d/datakit: no such file or directory, ignored
2021-12-19T04:26:53.323Z	DEBUG	dataway	dataway/dataway.go:208	httpCli: 0x4000244a00
2021-12-19T04:26:53.323Z	INFO	installer	installer/main.go:498	set cloud provider to  ok
2021-12-19T04:26:53.324Z	DEBUG	installer	installer/main.go:501	main config:
default_enabled_inputs = ["cpu", "disk", "diskio", "mem", "swap", "system", "hostobject", "net", "host_processes", "container"]
install_date = 2021-12-19T04:26:53Z
upgrade_date = 0001-01-01T00:00:00Z
namespace = ""
install_version = "1.1.9-rc7"
protect_mode = true
enable_election = false

[[black_lists]]
  hosts = []
  inputs = []

[[white_lists]]
  hosts = []
  inputs = []

[dca]
  enable = false
  listen = "0.0.0.0:9531"
  white_list = []

[http_api]
  rum_origin_ip_header = "X-Forwarded-For"
  listen = "localhost:9529"
  disable_404page = false
  rum_app_id_white_list = []

[io]
  feed_chan_size = 1024
  high_frequency_feed_chan_size = 2048
  max_cache_count = 1024
  cache_dump_threshold = 512
  max_dynamic_cache_count = 1024
  dynamic_cache_dump_threshold = 512
  flush_interval = "10s"
  output_file = ""
  output_file_inputs = []
  enable_cache = false

[dataway]
  urls = ["https://aws-openway.guance.com?token=tkn_I_hope_there_is_no_revision_history"]
  timeout = "5s"
  http_proxy = ""
  max_fail = 20

[logging]
  log = "/var/log/datakit/log"
  gin_log = "/var/log/datakit/gin.log"
  level = "info"
  disable_color = false
  rotate = 32

[global_tags]
  cluster = ""
  project = ""
  site = ""

[environments]
  ENV_HOSTNAME = ""

[cgroup]
  enable = true
  cpu_max = 30.0
  cpu_min = 5.0

[tracer]
  trace_enabled = false
  host = ""
  port = ""
  service = ""
  version = ""
  logs_startup = false
  debug = false
  env = ""

[git_repos]
  pull_interval = "1m"

  [[git_repos.repo]]
    enable = false
    url = ""
    ssh_private_key_path = ""
    ssh_private_key_password = ""
    branch = "master"

2021-12-19T04:26:53.324Z	INFO	config	config/cfg.go:520	here is hostname:
2021-12-19T04:26:53.324Z	INFO	config	config/cfg.go:523	set hostname to dd2e01a6b885
2021-12-19T04:26:53.324Z	INFO	installer	installer/main.go:449	installing service datakit...
2021-12-19T04:26:53.328Z	INFO	installer	installer/main.go:274	starting service datakit...
2021-12-19T04:26:53.355Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/bin/datakit...
2021-12-19T04:26:53.355Z	WARN	config	config/cfg.go:841	remove /usr/local/bin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.355Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/sbin/datakit...
2021-12-19T04:26:53.355Z	WARN	config	config/cfg.go:841	remove /usr/local/sbin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.355Z	DEBUG	config	config/cfg.go:839	remove link /sbin/datakit...
2021-12-19T04:26:53.355Z	WARN	config	config/cfg.go:841	remove /sbin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.356Z	DEBUG	config	config/cfg.go:839	remove link /usr/sbin/datakit...
2021-12-19T04:26:53.356Z	DEBUG	config	config/cfg.go:839	remove link /usr/bin/datakit...
2021-12-19T04:26:53.356Z	WARN	config	config/cfg.go:841	remove /usr/bin/datakit: no such file or directory, ignored
2021-12-19T04:26:53.356Z	INFO	installer	installer/main.go:289	:) Install Success!

	Visit http://localhost:9529/man/changelog to see DataKit change logs.
	Visit http://localhost:9529/monitor to see DataKit running status.
	Visit http://localhost:9529/man to see DataKit manuals.

root@dd2e01a6b885:/# echo $?
1
root@dd2e01a6b885:/# ./install.sh
 Detect installer installer-linux-arm64-1.1.9-rc7

* Downloading installer /tmp/dk-installer
##################################################################################################################################################################################################### 100.0%

* Installing DataKit...
2021-12-19T04:27:35.719Z	INFO	installer	installer/main.go:245	stoping datakit...
2021-12-19T04:27:35.728Z	DEBUG	installer	installer/main.go:659	deprecated install path /usr/local/cloudcare/dataflux/datakit not exists, ignored
Downloading(datakit)... 17 MB/17 MB
Downloading(   data)... 34 MB/34 MB
2021-12-19T04:28:45.151Z	INFO	installer	installer/main.go:269	Installing version 1.1.9-rc7...
2021-12-19T04:28:45.157Z	DEBUG	dataway	dataway/dataway.go:208	httpCli: 0x4000132140
2021-12-19T04:28:45.158Z	INFO	installer	installer/main.go:498	set cloud provider to  ok
2021-12-19T04:28:45.160Z	DEBUG	installer	installer/main.go:501	main config:
default_enabled_inputs = ["cpu", "disk", "diskio", "mem", "swap", "system", "hostobject", "net", "host_processes", "container"]
install_date = 2021-12-19T04:28:45Z
upgrade_date = 0001-01-01T00:00:00Z
namespace = ""
install_version = "1.1.9-rc7"
protect_mode = true
enable_election = false

[[black_lists]]
  hosts = []
  inputs = []

[[white_lists]]
  hosts = []
  inputs = []

[dca]
  enable = false
  listen = "0.0.0.0:9531"
  white_list = []

[http_api]
  rum_origin_ip_header = "X-Forwarded-For"
  listen = "localhost:9529"
  disable_404page = false
  rum_app_id_white_list = []

[io]
  feed_chan_size = 1024
  high_frequency_feed_chan_size = 2048
  max_cache_count = 1024
  cache_dump_threshold = 512
  max_dynamic_cache_count = 1024
  dynamic_cache_dump_threshold = 512
  flush_interval = "10s"
  output_file = ""
  output_file_inputs = []
  enable_cache = false

[dataway]
  urls = ["https://aws-openway.guance.com?token=tkn_I_hope_there_is_no_revision_history"]
  timeout = "5s"
  http_proxy = ""
  max_fail = 20

[logging]
  log = "/var/log/datakit/log"
  gin_log = "/var/log/datakit/gin.log"
  level = "info"
  disable_color = false
  rotate = 32

[global_tags]
  cluster = ""
  project = ""
  site = ""

[environments]
  ENV_HOSTNAME = ""

[cgroup]
  enable = true
  cpu_max = 30.0
  cpu_min = 5.0

[tracer]
  trace_enabled = false
  host = ""
  port = ""
  service = ""
  version = ""
  logs_startup = false
  debug = false
  env = ""

[git_repos]
  pull_interval = "1m"

  [[git_repos.repo]]
    enable = false
    url = ""
    ssh_private_key_path = ""
    ssh_private_key_password = ""
    branch = "master"

2021-12-19T04:28:45.161Z	INFO	config	config/cfg.go:520	here is hostname:
2021-12-19T04:28:45.161Z	INFO	config	config/cfg.go:523	set hostname to dd2e01a6b885
2021-12-19T04:28:45.163Z	INFO	installer	installer/main.go:449	installing service datakit...
2021-12-19T04:28:45.167Z	INFO	installer	installer/main.go:274	starting service datakit...
2021-12-19T04:28:45.198Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/bin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /usr/local/sbin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /sbin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /usr/sbin/datakit...
2021-12-19T04:28:45.199Z	DEBUG	config	config/cfg.go:839	remove link /usr/bin/datakit...
2021-12-19T04:28:45.199Z	INFO	installer	installer/main.go:289	:) Install Success!

	Visit http://localhost:9529/man/changelog to see DataKit change logs.
	Visit http://localhost:9529/monitor to see DataKit running status.
	Visit http://localhost:9529/man to see DataKit manuals.

root@dd2e01a6b885:/# echo $?
0
root@dd2e01a6b885:/#

DataKit is collection agent for DataFlux.

DataKit

Build

Dependencies

More references

Owner

Similar Resources

Comments

1. 修复 Ubuntu cgroup 没有 memory.memsw.limit_in_bytes 文件时导致 cgroup 限制设置失败

Add license scan report and status

installer-linux-arm64-1.1.9-rc7 failed at first try but succeeded at the second try

Related tags

Go Collection Stream API, inspired in Java 8 Stream.

mackerel-agent is an agent program to post your hosts' metrics to mackerel.io.

Igo Agent is the agent of Igo, a command-line tool, through which you can quickly start Igo

Shoes-agent - Framework for myshoes provider using agent

Coriolis-snapshot-agent - Coriolis snapshot agent leverages the blk-snap kernel module to create copy-on-write snapshots of block devices

Integrated ssh-agent for windows. (pageant compatible. openSSH ssh-agent etc ..)

User agent string parser in golang

HTTP User Agent parser for the Go programming language.

Web user interface and service agent for the monitoring and remote management of WinAFL.

Cloudinsight Agent is a system tool that monitors system processes and services, and sends information back to your Cloudinsight account.