cloudquery powered by Osquery
cloudquery is Osquery extension to fetch cloud telemetry from AWS, GCP, and Azure. It is extensible so that one can add support for new tables easily, and configurable so that one can change the table schema as well.
Contents
Build and test extension
Build
- Install prerequisites
- make
- go
- Set environment variable for extension home (it shoud be path-to-repo/cloudquery/extension)
export CLOUDQUERY_EXT_HOME=/home/user/work/cloudquery/extension - Build the extension:
make
Test
With osqueryi
- Copy extension configuration sample file:
cp ${CLOUDQUERY_EXT_HOME}/extension_config.json.sample ${CLOUDQUERY_EXT_HOME}/config/extension_config.json
- Edit
${CLOUDQUERY_EXT_HOME}/config/extension_config.jsonwith your cloud accounts. You can add multiple accounts for each cloud provider. Change logging path and other parameters to suit your needs. Make sure log path is writable. - Start osqueryi
osqueryi --nodisable_extensions --extension ${CLOUDQUERY_EXT_HOME}/../cloudquery - Query data:
SELECT account_id, region_code, image_id, image_type FROM aws_ec2_image;
With osqueryd
- Build and install cloudquery:
make build sudo make install
- Edit (or create)
/etc/osquery/extensions.loadfile and append the following line:/usr/local/bin/cloudquery.ext - Edit
/opt/cloudquery/config/extension_config.jsonwith your cloud accounts. You can add multiple accounts for each cloud provider. Change logging path and other parameters to suit your needs. - Add following flags to
/etc/osquery/osquery.flags(your flag file path could be different)
--extensions_autoload=/etc/osquery/extensions.load
--disable_extensions=false
- Restart osquery service:
sudo service osqueryd restart
Test with docker
Setup credentials
Setup credentials before proceeding to testing with
osqueryiorosqueryd
-
Create a config directory on the host to hold the credentials for your cloud accounts (~/config is an example, but this could be any directory).
-
Make a copy of extension_config.json.sample as
extension_config.jsonin a directory calledconfig(can be anywhere on your machine) -
Copy cloud credentials to the
configdirectory- For AWS:
$HOME/.aws/credentials - For GCP:
your-serviceAccount.jsonor any JSON file that contains GCP credentials - For Azure:
my.author any file that holds Azure credentials
- For AWS:
-
If using AWS cloud, update the following fields in
awssection inconfig/extension_config.jsonfile:credentialFileshould be set to/opt/cloudquery/etc/config/credentialsidshould match AWS account IDprofileNameshould be same as the profile in your.aws/credentialsfile- Guide to create AWS credentials: https://docs.aws.amazon.com/general/latest/gr/aws-security-credentials.html
-
If using Google cloud, update
keyFileingcpsection inextension_config.jsonfile. It should be changed to/opt/cloudquery/etc/config/your-serviceAccount.jsonwhereyour-serviceAccount.jsonis the JSON key file that contains GCP credentials- Guide to create GCP credentials: https://cloud.google.com/iam/docs/creating-managing-service-account-keys
-
If using Azure, update the following fields in
azuresection inextension_config.jsonfile:authFileshould be set to/opt/cloudquery/etc/config/my.auth.my.authshould be the name of the file that contains your Azure credentials.subscriptionIdandtenantIdfields should be changed to values from your Azure account- Guide to create Azure credentials: https://docs.microsoft.com/en-us/cli/azure/create-an-azure-service-principal-azure-cli?view=azure-cli-latest
Run osqueryi inside cloudquery container
docker run --rm -it --name cloudquery \
-v <absolute path to host config directory>:/opt/cloudquery/etc/config \
uptycs/cloudquery:latest \
osqueryi --extension /usr/local/bin/cloudquery.ext
Run osqueryd from cloudquery container
Following files and directories can be mounted from the host:
/opt/cloudquery/logs- Directory that contains the logs/opt/cloudquery/etc/osquery.flags- Osquery flags file/opt/cloudquery/etc/osquery.conf- Osquery configuration JSON file/opt/cloudquery/etc/config- Directory that contains Cloud provider credentials and cloudquery configuration JSON
Sample Osquery configuration with scheduled queries that can be overwritten via osquery.conf:
{
"schedule": {
"gcp_compute_network": {
"query": "SELECT * FROM gcp_compute_network;",
"interval": 120
},
"aws_s3_bucket": {
"query": "SELECT * FROM aws_s3_bucket;",
"interval": 120
},
"azure_compute_vm": {
"query": "SELECT * FROM azure_compute_vm;",
"interval": 120
}
}
}
docker run --rm -d --name cloudquery \
-v <absolute path to host config directory>:/opt/cloudquery/etc/config \
uptycs/cloudquery:latest
