d18n is a data desensitization tool for RDBMS.

Introduction

Language

logo

d18n is a numeronym short for "data-desensitization", sounds like "d-eighteen-n".

As its name says, d18n can mask data to make it desensitized. In addition, d18n can do many other things.

  • d18n is a portable RDBMS cmd client. e.g., MySQL, PostgreSQL, Oracle, SQL Server ...
  • save query result into a file, e.g., xlsx, csv, txt, sql, html, json ...
  • detect sensitive info (like PII) from a file or a SQL query.
  • import data from files into different types of databases.
  • lint data file, to check if its format is compatible before import it into some database.

It can be used as a portable cmd client or imported as a package by other tools.

For more details and latest updates, see doc and release notes.

Build

d18n develop with Golang 1.16+, please install first.

git clone github.com/LianjiaTech/d18n
cd d18n

# Mac or Linux
make build

# Windows
go build -o d18n cmd\d18n\d18n.go

Cross-platform compile

Golang support many

~ $ make release
...

~ $ ls release
d18n.darwin-amd64  d18n.darwin-arm64  d18n.linux-amd64  d18n.windows-amd64

Quick Start

Simple Example

~ $ d18n --defaults-extra-file test/my.cnf --query "show databases"
+--------------------+
|      DATABASE      |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sakila             |
| sys                |
| test               |
| world_x            |
+--------------------+

License

d18n is under the Apache 2.0 license. See the LICENSE file for details.

Owner
Similar Resources

Library for scanning data from a database into Go structs and more

scany Overview Go favors simplicity, and it's pretty common to work with a database via driver directly without any ORM. It provides great control and

Jan 9, 2023

Workshop: Building data-intensive apps with SingleStore, Redpanda, and Golang

 Workshop: Building data-intensive apps with SingleStore, Redpanda, and Golang

This repo provides a starting point for building applications using SingleStore, Redpanda (by Vectorized), and the Go language. SingleStore is a scale-out relational database built for data-intensive workloads. Redpanda is a Kafka API compatible streaming platform for mission-critical workloads created by the team at Vectorized.

Nov 4, 2022

An observability database aims to ingest, analyze and store Metrics, Tracing and Logging data.

An observability database aims to ingest, analyze and store Metrics, Tracing and Logging data.

BanyanDB BanyanDB, as an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data. It's designed to handle observab

Dec 31, 2022

Create key value sqlite3 database from tabular data, fast.

Create key value sqlite3 database from tabular data, fast.

Turn tabular data into a lookup table using sqlite3. This is a working PROTOTYPE with limitations, e.g. no customizations, the table definition is fixed, etc.

Apr 2, 2022

Use SQL to data from CSV files. Open source CLI. No DB required.

Use SQL to data from CSV files. Open source CLI. No DB required.

CSV Plugin for Steampipe Use SQL to query data from CSV files. Get started → Documentation: Table definitions & examples Community: Slack Channel Get

Nov 10, 2022

MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads.

MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads.

What is MatrixOne? MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads. It provides an end-to-end data

Dec 26, 2022

Go library that stores data in Redis with SQL-like schema

Go library that stores data in Redis with SQL-like schema. The goal of this library is we can store data in Redis with table form.

Mar 14, 2022

Make a sqlite3 database from tabular data, fast.

Make a sqlite3 database from tabular data, fast.

MAKTA make a database from tabular data Turn tabular data into a lookup table using sqlite3. This is a working PROTOTYPE with limitations, e.g. no cus

Apr 2, 2022

Get data from .csv files use SQL-like queries.

csvql Get data from .csv files use SQL-like queries. Задание Необходимо написать консольную программу, которая по заданному клиентом запросу осуществл

Dec 7, 2021
Comments
  • 项目引用了github.com/dgrijalva/jwt-go等457个开源组件,存在14个漏洞,建议升级

    项目引用了github.com/dgrijalva/jwt-go等457个开源组件,存在14个漏洞,建议升级

    大佬,你好,我是@abbykimi,我IDE运行您这个项目的时候,提示有几个漏洞,项目调用了github.com/dgrijalva/jwt-go等457个开源组件,存在14个安全漏洞,建议你升级下。

    漏洞标题:jwt-go 安全漏洞
    漏洞编号:CVE-2020-26160
    漏洞描述:
    jwt-go是个人开发者的一个Go语言的JWT实现。
    jwt-go 4.0.0-preview1之前版本存在安全漏洞。攻击者可利用该漏洞在使用[]string{} for m[\"aud\"](规范允许)的情况下绕过预期的访问限制。
    影响范围:(∞, 4.0.0-preview1)
    最小修复版本:4.0.0-preview1
    引入路径:
    github.com/LianjiaTech/d18n@->github.com/pingcap/[email protected]>go.etcd.io/[email protected]>github.com/dgrijalva/[email protected]+incompatible
    

    另外13个漏洞,如需查看详细报告、复测或持续监测您的项目,戳这里https://www.mfsec.cn/jr?p=m5a8ab

    如果你对这个issues有任何疑问可以回复我哈( @abbykimi ),我会及时回复你的。

  • Can you support random dates?

    Can you support random dates?

    能支持日期随机生成吗?比如生日,给个100年以内的date,Birthday生成的‘NNNN-NN-NN’对于日期类型无法识别

    Google translate Can you support random dates? For example, birthdays, given a date within 100 years, the "NNNN-NN-NN" generated by the birth cannot be recognized for a certain date type

  • d18n能否支持以mysql客户端的形式连接mongo-bi-connector进行敏感数据扫描?

    d18n能否支持以mysql客户端的形式连接mongo-bi-connector进行敏感数据扫描?

    d18n很好的解决了rdbms敏感数据识别问题,d18n能否支持以mysql客户端的形式连接mongo-bi-connector进行敏感数据扫描?连接mongo-bi-connector的mysql客户端方式只是增加了一个--default-auth=mongosql_auth参数,用d18n尝试不支持,期待支持! image

Related tags
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

Nov 9, 2022
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.
OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases and file formats using SQL.

OctoSQL OctoSQL is a query tool that allows you to join, analyse and transform data from multiple databases, streaming sources and file formats using

Dec 29, 2022
Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具,满足用户非常灵活的同步、迁移需求。
Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具,满足用户非常灵活的同步、迁移需求。

RedisShake is mainly used to synchronize data from one redis to another. Thanks to the Douyu's WSD team for the support. 中文文档 English tutorial 中文使用文档

Dec 29, 2022
Small tool that analyzes the data of my crappy file format for catalogging things

Things Catalog Analyzer I recently started catalogging all the things I own. I simply wanted to have an overview over my things, think about what I ca

Nov 7, 2021
A tool for finding corrupted data rows in TiDB

tidb-bad-rows A tool for finding corrupted data rows in TiDB. It scans the target table and using a divide and conquer paradigm to locate all corrupte

Nov 17, 2021
A database connection tool for sensitive data
A database connection tool for sensitive data

go-sql 用于快速统计数据库行数、敏感字段匹配、数据库连接情况。 usage ./go-sql_darwin_amd64 -h ./go-sql_darwin_amd64 -f db.yaml -k name,user ./go-sql_darwin_amd64 -f db.yaml --min

Apr 4, 2022
Mergestat - a command-line tool for running SQL queries on git repositories and related data sources
Mergestat - a command-line tool for running SQL queries on git repositories and related data sources

Query git repositories with SQL. Generate reports, perform status checks, analyze codebases. ?? ??

Dec 30, 2022
A fast data generator that's multi-table aware and supports multi-row DML.
A fast data generator that's multi-table aware and supports multi-row DML.

If you need to generate a lot of random data for your database tables but don't want to spend hours configuring a custom tool for the job, then datage

Dec 26, 2022
Sync MySQL data into elasticsearch
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

Dec 30, 2022
Convert data exports from various services to a single SQLite database
Convert data exports from various services to a single SQLite database

Bionic Bionic is a tool to convert data exports from web apps to a single SQLite database. Bionic currently supports data exports from Google, Apple H

Dec 9, 2022