Isaac Gazimbe - DevOps Golang Tools
DevOps Golang Tools
All programs have --help
to list the available options.
See Also:
repos which contain hundreds of Cloud, Big Data, NoSQL, Web, Linux and other DevOps infrastructure tools. Isaac Gazimbe
Cloud & Big Data Contractor, Johanessburg South Africa
(you're welcome to connect with me on LinkedIn)
make update
if updating and not just git pull
as you will often need the latest library submodule and possibly new upstream libraries
Make sure you run Quick Start
Ready to run Docker image
All programs and their pre-compiled dependencies can be found ready to run on DockerHub.
List all programs:
docker run harisekhon/go-tools
Run any given program:
docker run harisekhon/go-tools
Automated Build from source
installs git, make, pulls the repo and build the binaries:
curl -L https://git.io/go-bootstrap | sh
or manually:
git clone https://github.com/harisekhon/devops-golang-tools go-tools
cd go-tools
make
The make
command automates building the go binaries which can then be copied around to other systems of the same family, eg. Linux or Mac.
Alternatively there is shebang magic which means each .go
program can be called directly like a script and it'll runtime compile and execute instantly like a scripted language. This is a neat trick for quick usage and testing built on go run
, but for frequent use the compiled binaries are usually the way to go.
Detailed Build Instructions are available further down.
Usage
All programs come with a --help
switch which includes a program description and the list of command line options.
Environment variables are supported for convenience and also to hide credentials from being exposed in the process list eg. $PASSWORD
, $TRAVIS_TOKEN
. These are indicated in the --help
descriptions in brackets next to each option and often have more specific overrides with higher precedence eg. $AMBARI_HOST
, $HBASE_HOST
take priority over $HOST
.
DevOps Golang Tools - Inventory
- Linux:
uniq2.go
- likeuniq
but you don't have to sort first and it preserves the orderinghttpfirst.go
- returns the first http/https url address argument to respond (fastest multi-threaded reply using go channels). More sophisticated version isfind_active_server.py
in the DevOps Python tools repo which can handle multi-master clusters, tcp sockets, regex etc.colors.go
- prints a table of terminal colors and their escape codes for doing fancy shell stuffwelcome.go
- cool spinning welcome message greeting your username and showing last login time and user to put in your shell's.profile
(there are also Python and Perl versions in my DevOps Python Tools and DevOps Perl Tools repos)
Detailed Build Instructions
Manual Setup
Enter the go-tools directory and run git submodule init and git submodule update to fetch my library repo:
git clone https://github.com/harisekhon/devops-golang-tools go-tools
cd go-tools
git submodule init
git submodule update
go build
Mac OS X
The automated build also works on Mac OS X but you'll need to install Apple XCode (on recent Macs just typing git
is enough to trigger Xcode install).
I also recommend you get HomeBrew to install other useful tools and libraries you may need like OpenSSL for development headers and tools such as wget (these are installed automatically if Homebrew is detected on Mac OS X):
bash-tools/setup/install_homebrew.sh
Configuration for Strict Domain / FQDN validation
Strict validations include host/domain/FQDNs using TLDs which are populated from the official IANA list is done via my PyLib library submodule - see there for details on configuring this to permit custom TLDs like .local
, .intranet
, .vm
, .cloud
etc. (all already included in there because they're common across companies internal environments).
Updating
Run make update
. This will git pull and then git submodule update which is necessary to pick up corresponding library updates.
Testing
Continuous Integration is run on this repo with tests for success and failure scenarios:
- integration tests of the top level programs using the libraries for things like option parsing
- functional tests for the top level programs using local test data and Docker containers
To trigger all tests run:
make test
which will start with the underlying libraries, then move on to top level integration tests and functional tests using docker containers if docker is available.
Contributions
Patches, improvements and even general feedback are welcome in the form of GitHub pull requests and issue tickets.
See Also
-
DevOps Bash Tools - 700+ DevOps Bash Scripts, Advanced
.bashrc
,.vimrc
,.screenrc
,.tmux.conf
,.gitconfig
, CI configs & Utility Code Library - AWS, GCP, Kubernetes, Docker, Kafka, Hadoop, SQL, BigQuery, Hive, Impala, PostgreSQL, MySQL, LDAP, DockerHub, Jenkins, Spotify API & MP3 tools, Git tricks, GitHub API, GitLab API, BitBucket API, Code & build linting, package management for Linux / Mac / Python / Perl / Ruby / NodeJS / Golang, and lots more random goodies -
SQL Scripts - 100+ SQL Scripts - PostgreSQL, MySQL, AWS Athena, Google BigQuery
-
DevOps Python Tools - 80+ DevOps CLI tools for AWS, GCP, Hadoop, HBase, Spark, Log Anonymizer, Ambari Blueprints, AWS CloudFormation, Linux, Docker, Spark Data Converters & Validators (Avro / Parquet / JSON / CSV / INI / XML / YAML), Elasticsearch, Solr, Travis CI, Pig, IPython
-
DevOps Perl Tools - 25+ DevOps CLI tools for Hadoop, HDFS, Hive, Solr/SolrCloud CLI, Log Anonymizer, Nginx stats & HTTP(S) URL watchers for load balanced web farms, Dockerfiles & SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles, Pig Latin, Neo4j, InfluxDB), Ambari FreeIPA Kerberos, Datameer, Linux...
-
The Advanced Nagios Plugins Collection - 450+ programs for Nagios monitoring your Hadoop & NoSQL clusters. Covers every Hadoop vendor's management API and every major NoSQL technology (HBase, Cassandra, MongoDB, Elasticsearch, Solr, Riak, Redis etc.) as well as message queues (Kafka, RabbitMQ), continuous integration (Jenkins, Travis CI) and traditional infrastructure (SSL, Whois, DNS, Linux)
-
HAProxy Configs - 80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Elasticsearch, SolrCloud, HBase, Cloudera, Hortonworks, MapR, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, ZooKeeper, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, SSH, RabbitMQ, Redis, Riak, Rancher etc.
-
Templates - dozens of Code & Config templates - AWS, GCP, Docker, Jenkins, Terraform, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions Workflows, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, M4 etc.
-
Kubernetes configs - Kubernetes YAML configs - Best Practices, Tips & Tricks are baked right into the templates for future deployments
-
Dockerfiles - 50+ DockerHub public images for Docker & Kubernetes - Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Mesos, Consul, Riak, OpenTSDB, Jython, Advanced Nagios Plugins & DevOps Tools repos on Alpine, CentOS, Debian, Fedora, Ubuntu, Superset, H2O, Serf, Alluxio / Tachyon, FakeS3
-
PyLib - Python library
-
Perl Lib - Perl library