MySQL replication topology manager - agent (daemon)

orchestrator-agent

MySQL topology agent (daemon)

orchestrator-agent is a sub-project of orchestrator. It is a service that runs on MySQL hosts and communicates with orchestrator.

orchestrator-agent is capable of proving operating system, file system and LVM information to orchestrator, as well as invoke certain commands and scripts.

The primary drive for developing orchestrator-agent was Outbrain's need for a controlled seeding mechanism, where MySQL instances would duplicate onto new/corrupt installations. As such, orchestrator-agent's functionality is tightly coupled with the backup/seed mechanisms used by Outbrain, as described following. Though easily extendible, orchestrator-agent was not developed with a general purpose backup/restore/orchestration capabilities in mind.

Whether or not orchestrator-agent is useful to you depends on your needs.

Generic functionality offered by orchestrator-agent:
  • Detection of the MySQL service, starting and stopping (start/stop/status commands provided via configuration)
  • Detection of MySQL port, data directory (assumes configuration is /etc/my.cnf)
  • Calculation of disk usage on data directory mount point
  • Tailing the error log file
  • Discovery (the mere existence of the orchestrator-agent service on a host may suggest the existence or need of existence of a MySQL service)
Specialized functionality offered by orchestrator-agent:
  • Detection of LVM snapshots on MySQL host (snapshots that are MySQL specific)
  • Creation of new snapshots
  • Mounting/umounting of LVM snapshots
  • Detection of DC-local and DC-agnostic snapshots available for a given cluster
  • Transmitting/receiving seed data

The Outbrain seed method

The following does not discuss hard-backups where binary/logical data is written to an external/remote disk or a tape.

At Outbrain we use LVM snapshots. Each MySQL replication topology has designated slaves which serve as snapshot servers. These servers do LVM snapshots daily, and keep such snapshots open for a few days. Thus it is possible that a server has, say, 5 open (and unmounted) LVM snapshots at a given time.

Upon need, we are able to mount any such snapshot in near zero time and restart MySQL on the host using mounted data directory. We are thus able to recover any one of few days back at speed of InnoDB crash-recovery.

This serves two purposes: an immediate recovery/sanity check for destructive operations (unintentional or malicious UPDATE or DROP) as well as a seed source for new/corrupt servers.

The choice of snapshot slaves is not random; we have multiple data centers and we keep at least one snapshot server per topology per data center, such that upon need we can make DC-local copies. For this purpose, the daily snapshot process reports, upon success, the availability of the snapshot along with any metadata required on cluster/DC.

orchestrator-agent thus depends on external (configurable) commands to:

  • Detect where in local and remote DCs it can find an appropriate snapshot
  • Find said snapshot on server, mount it
  • Stop MySQL on target host, clear data on MySQL data directory
  • Initiate send/receive process
  • Cleanup data after copy (e.g. remove .pid files if any)
  • Unmount snapshot
  • etc.

The orchestrator & orchestrator-agent architecture

orchestrator is a standalone, centralized service/command line tool. When acting as a service, it provides with web API and web interface to allow replication topology refactoring, long query control, and more.

Coupled with orchestrator-agent, orchestrator is further able to assist in seeding new/corrupt servers. orchestrator-agent does not initiate anything by itself, but is in fact controlled by orchestrator.

When started, orchestrator-agent chooses a random, secret token and attempts to connect to the centralized orchestrator service API (location configurable). It then registers at the orchestrator service with its secret token.

orchestrator-agent then serves via HTTP API, and for all but the simplest commands requires the secret token.

At this point orchestrator becomes the major player; having multiple orchestrator-agent registered it is able to coordinate operations such as snapshot mounting, space cleanup, send and receive so as to establish a successful seed (a binary copy of a MySQL data directory).

orchestrator-agent only provides the minimal and required operating system functionality and does not interact with the MySQL service directly (i.e. no credentials required and no SQL queries invoked). Any and all queries are invoked by the centralized orchestrator service.

Configuration

Orchestrator-agent uses a configuration file, located in either /etc/orchestrator-agent.conf.json or relative path to binary conf/orchestrator-agent.conf.json or orchestrator-agent.conf.json.

Note that the agent will use the config file in its relative conf path first.

The following is a complete list of configuration parameters:

  • SnapshotMountPoint (string), a known mountpoint onto which a mount command will mount snapshot volumes
  • ContinuousPollSeconds (uint), internal clocking interval (default 60 seconds)
  • ResubmitAgentIntervalMinutes (uint), interval at which the agent re-submits itself to orchestrator daemon
  • CreateSnapshotCommand (string), command which creates new LVM snapshot of MySQL data
  • AvailableLocalSnapshotHostsCommand (string), command which returns list of hosts in local DC on which recent snapshots are available
  • AvailableSnapshotHostsCommand (string), command which returns list of hosts in all DCs on which recent snapshots are available
  • SnapshotVolumesFilter (string), free text which identifies MySQL data snapshots (as opposed to other, unrelated snapshots)
  • MySQLDatadirCommand (string), command which returns the data directory (e.g. grep datadir /etc/my.cnf | head -n 1 | awk -F= '{print $2}')
  • MySQLPortCommand (string), command which returns the MySQL port
  • MySQLDeleteDatadirContentCommand (string), command which purges the MySQL data directory
  • MySQLServiceStopCommand (string), command which stops the MySQL service (e.g. service mysql stop)
  • MySQLServiceStartCommand (string), command which starts the MySQL service
  • MySQLServiceStatusCommand (string), command that checks status of service (expecting exit code 1 when service is down)
  • ReceiveSeedDataCommand (string), command which listen on data, must accept arguments: target directory, listen port
  • SendSeedDataCommand (string), command which sends data, must accept arguments: source directory, target host, target port
  • PostCopyCommand (string), command to be executed after the seed is complete (cleanup)
  • AgentsServer (string), Required URL of your orchestrator daemon, You must add the port the orchestrator server expects to talk to agents to (see below, e.g. https://my.orchestrator.daemon:3001)
  • HTTPPort (uint), Port to listen on
  • HTTPAuthUser (string), Basic auth user (default empty, meaning no auth)
  • HTTPAuthPassword (string), Basic auth password
  • UseSSL (bool), If true then serving via https protocol
  • SSLSkipVerify (bool), When connecting to orchestrator via SSL, whether to ignore certification error
  • SSLPrivateKeyFile (string), When serving via https, location of SSL private key file
  • SSLCertFile (string), When serving via https, location of SSL certification file
  • HttpTimeoutSeconds (int), HTTP GET request timeout (when connecting to orchestrator)

An example configuration file may be:

{
    "SnapshotMountPoint": "/var/tmp/mysql-mount",
    "AgentsServer": "https://my.orchestrator.daemon:3001",
    "ContinuousPollSeconds" : 60,
    "ResubmitAgentIntervalMinutes": 60,
    "CreateSnapshotCommand":                "/path/to/snapshot-command.bash",
    "AvailableLocalSnapshotHostsCommand":   "/path/to/snapshot-local-availability-command.bash",
    "AvailableSnapshotHostsCommand":        "/path/to/snapshot-availability-command.bash",
    "SnapshotVolumesFilter":                "mysql-snap",
    "MySQLDatadirCommand":                  "set $(grep datadir /etc/my.cnf | head -n 1 | awk -F= '{print $2}') ; echo $1",
    "MySQLPortCommand":                     "set $(grep ^port /etc/my.cnf | head -n 1 | awk -F= '{print $2}') ; echo $1",
    "MySQLDeleteDatadirContentCommand":     "set $(grep datadir /etc/my.cnf | head -n 1 | awk -F= '{print $2}') ; rm --preserve-root -rf $1/*",
    "MySQLServiceStopCommand":      "/etc/init.d/mysql stop",
    "MySQLServiceStartCommand":     "/etc/init.d/mysql start",
    "MySQLServiceStatusCommand":    "/etc/init.d/mysql status",
    "ReceiveSeedDataCommand":       "/path/to/data-receive.bash",
    "SendSeedDataCommand":          "/path/to/data-send.bash",
    "PostCopyCommand":              "set $(grep datadir /etc/my.cnf | head -n 1 | awk -F= '{print $2}') ; rm -f $1/*.pid",
    "HTTPPort": 3002,
    "HTTPAuthUser": "",
    "HTTPAuthPassword": "",
    "UseSSL": false,
    "SSLSkipVerify": false,
    "SSLCertFile": "",
    "SSLPrivateKeyFile": "",
    "HttpTimeoutSeconds": 10
}

Necessary matching configuration on the Orchestrator Server side

If you initially deployed orchestrator with a minimally working configuration, you will need to make some changes on the server side to prepare it for newly deployed agents. The configuration lines needed on the server side to support agents are

  • ServeAgentsHttp (bool), Must be set to true to get the orchestrator server listening for agents
  • AgentsServerPort (String), The port on which the server should listen to agents. Shoult match the port you define for agents in AgentsServer.

Requirements:

  • Linux, 64bit. Tested on CentOS 5 and Ubuntu Server 12.04+
  • MySQL 5.1+
  • LVM, free space in volume group, if snapshot functionality is required
  • orchestrator-agent assumes a single MySQL running on the machine

Extending orchestrator-agent

Yes please. orchestrator-agent is open to pull-requests. Desired functionality is for example the initiation and immediate transfer of backup data via xtrabackup. The same can be done via mysqldump or mydumper etc.

Authored by Shlomi Noach at GitHub. Previously at Booking.com and Outbrain

Owner
GitHub
How people build software.
GitHub
Comments
  • add Vagrantfile and Dockerfile, prepare to publish packagecloud.io

    add Vagrantfile and Dockerfile, prepare to publish packagecloud.io

    replace outbrain to github in golang url add packageloud-ruby into Vagrant little bit code clear tested packagecloud repo https://packagecloud.io/Slach/orchestrator-agent

    Signed-off-by: Slach [email protected]

  • Build Pipelines Migration Needed

    Build Pipelines Migration Needed

    :wave: Greetings from @github/devcon !

    As you may not know from our many Team posts, we are amidst a migration from Jenkins-based infrastructure to using Build Pipelines for internal CI.

    This repository was marked for exemption in the inital Q5 migration. In Q1, however, we intend to fully decommission all of Jenkins. That means the outstanding jobs need to either be deleted or migrated by September 1st.

    Our records indicate the following CI jobs are still running on Jenkins for this repository:

    • [ ] orchestrator-agent-build-deploy-tarball

    Please use this as a tracking issue for remediation efforts. We will comment on this issue with reminders as we get closer to decomission date.

    Team Assigned This Migration:

    @github/devcon

    The assignee of this issue is inferred based on a combination of CI Ownership Spreadsheet and Exemption Request. If this is assigned incorrectly, please find the proper owner.

    If you have any questions, find us in #build on Slack or File an Issue

  • support for /api/mysql-relay-log-index-file

    support for /api/mysql-relay-log-index-file

    /api/mysql-relay-log-index-file heuristically returns the name of the relay log index file.

    There is a hidden assumption that this file lies under the MySQL datadir. This assumption may be lifted in the future.

  • API for binlog binary contents

    API for binlog binary contents

    osagent and API to provide /api/mysql-relaylog-contents-tail/:relaylog/:start as encoded binary contents of the relay logs. That is, the relay logs are not extracted via mysqlbinlog. Instead, the very binary contents of the relay logs are sliced (prepended with magic header and with initial format description)

  • Fix CI build path

    Fix CI build path

    Although the project is on https://github.com/github/orchestrator-agent, the import path should be https://github.com/outbrain/orchestrator-agent

    yay golang

  • CI scripts

    CI scripts

    Adding CI scripts.

    This will enable orchestrator-agent to have CI tests per push, with master being a protected branch.

    This initial PR is initially simplified and we will iterate on the CI scripts later on. In particular, we will edit the build TAR archive to include the entire resources tree.

    cc @github/database-infrastructure

  • api: binlogs and relay logs

    api: binlogs and relay logs

    support for:

    • /api/mysql-relay-log-index-file - file name of relay log index
    • /api/mysql-relay-log-files - listing of relay log files
    • /api/mysql-relay-log-end-coordinates - current last coordinates of relay logs (== file size of last relay log)
    • /api/mysql-binlog-contents - returns contents of requested binlogs (query params: multi binlog file names, optional start (of first file), stop (of last file) positions. Returns binlog content, compressed, base64 encoded.
    • /api/mysql-relaylog-contents-tail/:relaylog/:start - return relay log contents, starting given file:pos, ending at last coordinates. Return value compressed, then base64 encoded.
  • api for getting binary log contents

    api for getting binary log contents

    adding:

    • /api/mysql-binlog-contents
    • /api/mysql-relaylog-contents-tail/:relaylog/:start

    both of which return the binary log contents of given files.

    • /api/mysql-binlog-contents expect absolute file names, multiple files supported via binlog=. start and stop positions optional.
    • /api/mysql-relaylog-contents-tail/:relaylog/:start expects one relay log file, either absolute path or not
  • Add support for xtrabackup, xtrabackup-stream, mydumper, mysqldump streaming methods for slave provisioning

    Add support for xtrabackup, xtrabackup-stream, mydumper, mysqldump streaming methods for slave provisioning

    • [x] On agent get mysql datadir, log_error, innodb_log_group_home_dir from my.cnf. We can use some type of notifications mechanism (like fsnotify) to subscribe for changes in config files and reload configuration dynamically on change. This will help us to keep in touch with changes in MySQL configuration without a need to restart orchestrator-agent
    • [x] [AGENT] New agent function DeleteDirContents, which we will use later in order to remove mysql datadir contents and for /api/delete-mysql-backupdir and /api/delete-mysql-datadir
    • [x] [AGENT] New agent API /api/delete-mysql-backupdir, which we will use later in order to delete backups after seed will be completed.
    • [x] [AGENT] Change /api/delete-mysql-datadir from using command from config to DeleteDirContents
    • [x] [AGENT] New agent API – /api/mysql-backupdir-available-space Will be used in pre-seed checks. Create new API on agent, add backupDirAvailiable space to type Agent struct
    • [x] [AGENT] New agent API /api/available-seed-methods Agent need to provide information about available seed methods – this depends on binaries we have installed on it. Right now we will support LVM, xtrabackup, xtrabackup-stream, mydumper and mysqldump. Create new API on agent, add seedMethods to type Agent struct
    • [x] [AGENT] New agent API /api/mysql-info Agent need to provide information about MySQL version, MySQL datadir location, if this server is already slave (has output from SHOW SLAVE STATUS command, will fail if our targetHost is already a slave), if this server is already master (SHOW SLAVE HOSTS command, will fail is our targetHost is already a master), if it has currently active connections to MySQL(SELECT COUNT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE USER NOT IN (‘MySQLUser’, ‘other_system_users’), will fail if number of active connections to targetHost is larger than some thresold) We will use it in pre-seed check on targetHost and SourceHost. Create new API on agent, new type MySQLInfo struct and add type MySQLInfo struct to type Agent struct
    • [x] [AGENT] New agent API /api/mysql-databases Agent need to provide information about databases, engines and db sizes, which present on server. We need this because:
    • Not all engines supports xtrabackup (for example TokuDB)
    • Sometimes we want to restore only some user databases and save old mysql db with logins\pass information (partial backup)
    • Physical size is the size of database on disk (du database_folder) - we will need it for partial xtrabackup in order to check if there is enough space in backup directory on sourceHost and in backup directory\data directory on targetHost
    • Logical size in the size of data in the database (estimations) - we will need it for partial mydumper\mysqlbackup in order to check if there is enough space in backup directory on sourceHost and in backup directory\data directory on targetHost. We can calculate this use some estimations: -- if we compress backup, multiply physical size * 0.5 -- if we don't compress, multiply physical size * 0.8 Create Api on agent, new type MySQLDatabases struct, new type MySQLDatabaseInfo struct and add MySQLDatabases to type Agent struct. As a result we will get JSON like this: {"MySQLDatabases":{"employees":{"Engines":["InnoDB"],"PhysicalSize":186929946,"LogicalSize":112157967},"log":{"Engines":["InnoDB"],"PhysicalSize":32307770,"LogicalSize":19384662},"test":{"Engines":["InnoDB","MyISAM"],"PhysicalSize":116501,"LogicalSize":69900},"test2":{"Engines":["InnoDB"],"PhysicalSize":106921,"LogicalSize":64152}},"InnoDBLogSize":100663296}
    • [x] In order to do this agent will be needed to have an access to MySQL installed on server, so we need to add new configuration variables MySQLTopologyUser and MySQLTopologyPassword for agent config
    • [x] [AGENT] Add new agent configuration variable MySQLBackupDir - path where we will store backups
    • [x] [AGENT] Add new agent configuration variable SeedPort- port we will use in order to stream xtrabackup and copy data
    • [x] [AGENT] Add new agent configuration variable MySQLReplicationUser- user that will be used for replication
    • [x] [AGENT] Add new agent configuration variable MySQLReplicationPassword- password for MySQLReplicationUser
    • [x] [AGENT] Add new agent configuration variable MySQLBackupUsersOnTargetHost. When we perform partial backup, we also need to backup system databases. If MySQLBackupUsersOnTargetHost is empty, before restoring backup on targetHost we will first backup mysql database on it and restore after seed operation completes. If MySQLBackupUsersOnTargetHost is set, we will backup only these users and restore them after seed operation completes
    • [x] [AGENT] Add some agent configuration variables for MyDumper\xtrabackup (like number of threads etc…)

    • [ ] [ORCHESTRATOR] Change GetAgent function so we get all new data and put it into type Agent struct
    • [ ] [ORCHESTRATOR] Let’s keep func Seed as main entrance point for all seeding operations, but add additional param seedMethod and make a switch in this function to start different operations for different seed methods. We should also add optional parameter with a list of databases to copy. If this parameter is missing - we copy all databases. Also in case of xtrabackup stream we need to add another bool parameter streamToDatadir – if it is true we will stream backup directly to MySQL datadir and will add some prerequisite actions. If it false – we stream backup to MySQLBackupDir. So API call for seed would look like agent-seed/:targetHost/:sourceHost/:seedMethod/:streamToDatadir/:optionalListOfDatabases
    • [ ] [ORCHESTRATOR] Change logic of Submit agent function. Right now when we first time submit the agent we get only basic data from it (hostname, port, token, last_submited) and thus we need to wait for AgentPollIMinutes interval so this agent became outdated and we get additional data using UpdateAgentInfo. This interval by default is 60 min, so we get all data and can start seeding only after 60 min. We can add logic, so if this agent is submitted for the first time and we don’t have information about It in orchestrator db - run UpdateAgentInfo immediately and get all other data
    • [ ] [ORCHESTRATOR] Change func SubmitSeedEntry function. Add seedMethod as param to it and to agent_seed table
    • [ ] [ORCHESTRATOR] We will keep updateSeedStateEntry and updateSeedComplete the same as they are and will use them to track all other seed methods

    • [x] [AGENT] New agent API /api/start-local-backup/:seedId/:seedMethod/:optionalListOfDatabases Will be used on sourceHost to:
    • create a folder to store backup(possible naming convention using backup date)
    • start mysqldump/mydumper/xtrabackup(without streaming) with necessary params (threads, optionalListOfDatabases if they are present)
    • Check that replication user exists on sourceHost. If it not exists - create it and GRANT REPLICATION SLAVE ON . TO it This API call should return path to created folder with backup
    • [x] [AGENT] New agent API /api/receive-backup/:seedId/:seedMethod/:backupFolder (backupFolder must be url encoded)
    • if backup folder == config.Config.MySQLDatadir: -- backup users to config.Config.MySQLBackupDir (either all, or only those specified in config.Config.MySQLBackupUsersOnTargetHost) -- stop MySQL -- remove everything from MySQL datadir -- remove ib_logfiles from innodb_log_group_home_dir
    • else create backup folder
    • start netcat to listen on SeedPort port with –-wait parameter = 30 sec in order to terminate netcat after seeding operation complete
    • if seedMethod = xtrabackup-stream pipe netcat to xbstream. For others pipe it to tar
    • [x] [AGENT] New agent API /api/send-local-backup/:seedId/:targetHost/:backupFolder will be used on sourceHost to tar.gz contents of backup folder and send archive to targetHost on SeedPort port using netcat
    • [x] [AGENT] New agent API /api/start-streaming-backup/:seedId/:targetHost/:optionalListOfDatabases Will be used to start xtrabackup stream and send it using netcat to targetHost on SeedPort. -- Check that replication user exists on sourceHost. If it not exists - create it and GRANT REPLICATION SLAVE ON . TO it
    • [x] [AGENT] New agent API /api/start-restore/:seedId/:seedMethod/:sourceHost/:sourcePort/:backupFolder/:optionalListOfDatabases will be used on targetHost to:
    • if optionalListOfDatabases is not empty - add replicate-do-db to targetHost my.cnf and restart MySQL
    • backup users to config.Config.MySQLBackupDir (either all, or only those specified in config.Config.MySQLBackupUsersOnTargetHost)
    • [MYSQLDUMP]: -- if backup is compressed gunzip it and execute all backup.sql file in backupFolder -- create replication user -- execute START SLAVE
    • [MYDUMPER]: -- run myloader with nessesary params -- create replication user -- parse metadata file in MySQLBackupDir and execute CHANGE MASTER TO and START SLAVE (2 different cases for GTID and positional replicas – will create separate function for this)
    • [XTRABACKUP FULL\XTRABACKUP STREAM FULL TO BACKUPDIR](optionalListOfDatabases is empty, backupFolder != config.Config.MySQLDatadir, seedMethod xtrabackup or xtrabackup-stream): -- run xtrabackup –-prepare on copied backup in backupFolder -- stop MySQL -- remove everything from config.Config.MySQLDatadir -- remove ib_logfiles from innodb_log_group_home_dir -- run xtrabackup –-copy-back -–target-dir=backupdir (or may be use --move-back??) -- start MySQL -- parse xtrabackup_binlog_info in MySQL datadir and execute CHANGE MASTER TO and START SLAVE
    • [XTRABACKUP PARTIAL\XTRABACKUP STREAM PARTIAL TO BACKUPDIR](optionalListOfDatabases is not empty, backupFolder != config.Config.MySQLDatadir, seedMethod xtrabackup or xtrabackup-stream): -- run xtrabackup –-prepare on copied backup in backupFolder -- stop MySQL -- remove everything from config.Config.MySQLDatadir -- remove ib_logfiles from innodb_log_group_home_dir -- run xtrabackup –-copy-back -–target-dir=backupdir -- start MySQL -- if partial backup - run mysql_update to restore system databases -- create replication user -- parse xtrabackup_binlog_info in MySQL datadir and execute CHANGE MASTER TO and START SLAVE
    • [XTRABACKUP STREAM FULL TO DATADIR](optionalListOfDatabases is empty, backupFolder = config.Config.MySQLDatadir, seedMethod xtrabackup-stream): -- run xtrabackup –-prepare on config.Config.MySQLDatadir -- start MySQL -- parse xtrabackup_binlog_info in MySQL datadir and execute CHANGE MASTER TO and START SLAVE
    • [XTRABACKUP STREAM PARTIAL TO DATADIR](optionalListOfDatabases is not empty, backupFolder = config.Config.MySQLDatadir, seedMethod xtrabackup-stream): -- run xtrabackup –-prepare on config.Config.MySQLDatadir -- start MySQL -- run mysql_update to restore system databases -- create replication user -- parse xtrabackup_binlog_info in MySQL datadir and execute CHANGE MASTER TO and START SLAVE
    • restore users
    • [x] [AGENT] New agent API /api/cleanup/:seedId Will be used on targetHost and sourceHost in order to remove contents of config.Config.MySQLBackupdir after seed process

    All this changes are completely Backward compatible (except that for now we will need to add “lvm” param as seedMethod when we use “Seed” button on an agent page in Snapshots area) and won’t affect current agent and orchestrator workflow. 


    In order to support this new seed methods, we can reuse part of the logic of current executeSeed function - divide backup\restore flow into finite operations and use UpdateSeedStateEntry function in order to track progress.

    As a some type of draft, I suppose following scenarios for new seed methods:

    Basic checks for all types of seed methods:

    • Check that there are no active seeds for targetHost and sourceHost (query agent_seed table)
    • Check that MySQL has the same major versions on both targetHost and sourceHost
    • Check that binary logging is enabled on sourceHost
    • Check that targetHost isn’t master for some other hosts
    • Check that targetHost isn't slave for some other hosts
    • Check that there are no database connections on targetHost
    • [XTRABACKUP ONLY] Check that there is no TokuDB engine in a databases on sourceHost

    Draft for mysqldump/mydumper/xtrabackup:

    • Call agent-seed/:targetHost/:sourceHost/:seedMethod/:streamToDatadir/:optionalListOfDatabases (:streamToDatadir = false)
    • Run GetAgent function for sourceHost
    • Run GetAgent function for targetHost
    • Run basic checks
    • Get size of backup and check that we have enough space in MySQLBackupDir folders on sourceHost and targetHost: -- [MYDUMPER/MYSQLDUMP] calculate backup size it as sum of logicalSize of needed databases -- [XTRABACKUP FULL] (optionalListOfDatabases is empty) – calculate backup size as a size of MySQL datadir (use agent API /api/mysql-du) + some space for ib_logfile which will be created during xtrabackup --prepare -- [XTRABACKUP PARTIAL] calculate backup size as a sum of physicalSize of needed databases + some space for ib_logfile which will be created during xtrabackup --prepare
    • Check that we have enough space in targetHost MySQL datadir: -- [MYDUMPER/MYSQLDUMP/XTRABACKUP PARTIAL] - calculate it as a sum of physicalSize of needed databases -- [XTRABACKUP FULL] (optionalListOfDatabases is empty) calculate it as a size of MySQL datadir (use agent API /api/mysql-du) + some space for ib_logfile which will be created during xtrabackup --prepare
    • Check that MySQL on sourceHost is running
    • Start backup process on sourceHost /api/start-local-backup/:seedId/:seedMethod/:optionalListOfDatabases. This API call will return path to directory with backup
    • Start receiving on targetHost /api/receive-backup/:seedId/:seedMethod/:backupFolder. If :streamToDatadir = true than use Agent.MySQLInfo.MySQLDatadirPath as :backupFolder, else use path returned from /api/start-local-backup/:seedId/:seedMethod/:optionalListOfDatabases
    • Start sending backup archive on sourceHost /api/send-local-backup/:seedId/:targetHost/:backupFolder
    • Start restore process on targetHost /api/start-restore/:seedId/:seedMethod/:sourceHost/:sourcePort/:backupFolder/:optionalListOfDatabases. If :streamToDatadir = true than use Agent.MySQLInfo.MySQLDatadirPath as :backupFolder, else use path returned from /api/start-local-backup/:seedId/:seedMethod/:optionalListOfDatabases
    • Run /api/cleanup/:seedId on sourceHost
    • Run /api/cleanup/:seedId on targetHost
    • Run UpdateSeedComplete

    Draft for xtrabackup-stream:

    • Call agent-seed/:targetHost/:sourceHost/:seedMethod/:streamToDatadir/:optionalListOfDatabases
    • Run GetAgent function for sourceHost
    • Run GetAgent function for targetHost
    • Run basic checks
    • [IF streamToDataDir = false] Get size of backup and check that we have enough space in MySQLBackupDir folders on sourceHost and targetHost: -- [XTRABACKUP FULL] (optionalListOfDatabases is empty) – calculate backup size as a size of MySQL datadir (use agent API /api/mysql-du) + some space for ib_logfile which will be created during xtrabackup --prepare -- [XTRABACKUP PARTIAL] calculate backup size as a sum of physicalSize of needed databases + some space for ib_logfile which will be created during xtrabackup --prepare
    • Check that we have enough space in targetHost datadir: -- [XTRABACKUP PARTIAL] - calculate it as a sum of physicalSize of needed databases -- [XTRABACKUP FULL] (optionalListOfDatabases is empty) calculate it as a size of MySQL datadir (use agent API /api/mysql-du) + some space for ib_logfile which will be created during xtrabackup --prepare
    • Check that MySQL on sourceHost is running
    • Start receiving on targetHost /api/receive-backup/:seedId/:seedMethod/:backupFolder. If :streamToDatadir = true than use Agent.MySQLInfo.MySQLDatadirPath as :backupFolder, else use Agent.MySQLInfo.MySQLBackupdirPath\generate_new_folder_name
    • Start streaming backup on sourceHost /api/start-streaming-backup/:optionalListOfDatabases
    • Start restore process on targetHost /api/start-restore/:seedId/:seedMethod/:sourceHost/:sourcePort/:backupFolder/:optionalListOfDatabases. If :streamToDatadir = true than use Agent.MySQLInfo.MySQLDatadirPath as :backupFolder, else use Agent.MySQLInfo.MySQLBackupdirPath\generate_new_folder_name
    • Run /api/cleanup/:seedId on sourceHost
    • Run /api/cleanup/:seedId on targetHost
    • Run UpdateSeedComplete

    Also we will need to add these methods to agents page in orchestrator UI, but that’s the thing that I need to research a bit, because I’m not good at all in all this frontend-developers stuff :)

    Do not forget to set 700 permissions for orchestrator-agent.conf.json in order to secure passwords

  • Improve and actualize orchestrator-agent , extract many LVM commands into configuration file

    Improve and actualize orchestrator-agent , extract many LVM commands into configuration file

    • Сustomize orchestrator-agent config, adding several *Command parameters
    • add handle signal syscall.SIGHUP with config.Reload, and terminate with SIGTERM, SIGKILL
    • Improve init.d script for SIGHUP and reload support
    • extract SeedTransferPort from const to config
  • Add configuration option for location of my.cnf file

    Add configuration option for location of my.cnf file

    Percona's Ubuntu packages drop my.cnf at /etc/mysql/my.cnf.

    There's only one reference to that file directly in orchestrator-agent, but this avoids having to create a symbolic link to the correct location by exposing a configuration knob.

  • Remove cli build steps, fix dpkg name when building on Darwin

    Remove cli build steps, fix dpkg name when building on Darwin

    As far as I can tell, the CLI stuff doesn't yet exist in master for orchestrator-agent, so I've removed that from build.sh. My go-fu isn't that great, so if I'm just overlooking where it lives, let me know and I'll update that.

    Also fix build.sh so fpm pukes out the appopriate package name for the Debian build when building Linux packages on a Darwin host, orchestrator-agent_1.2.1_amd64.deb instead of orchestrator-agent_1.2.1_darwin-amd64.deb; as well as set the arch properly in the package metadata (needs to be amd64 and not darwin-amd64)

  • Move references from outbrain/orchestrator to github/orchestrator

    Move references from outbrain/orchestrator to github/orchestrator

    This at least partially fixes the build.sh script. The shell out to fpm to build the cli packages fail, I'll be submitting a second PR to address that.

PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology management, high availability, configuration management, and plugin extensions.

What is PolarDB Cluster Manager PolarDB Cluster Manager is the cluster management component of PolarDB for PostgreSQL, responsible for topology manage

Nov 9, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Oct 25, 2022
A river for elasticsearch to automatically index mysql content using the replication feed.

Mysql River Plugin for ElasticSearch The Mysql River plugin allows to hook into Mysql replication feed using the excellent python-mysql-replication an

Jun 1, 2022
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

Nov 15, 2022
Enhanced PostgreSQL logical replication

pgcat - Enhanced postgresql logical replication Why pgcat? Architecture Build from source Install Run Conflict handling Table mapping Replication iden

Dec 21, 2022
Streaming replication for SQLite.

Litestream Litestream is a standalone streaming replication tool for SQLite. It runs as a background process and safely replicates changes incremental

Jan 9, 2023
a powerful mysql toolset with Go
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

Dec 28, 2022
Sync MySQL data into elasticsearch
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

Dec 30, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Dec 30, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Jan 3, 2023
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

Nov 17, 2022
一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

Nov 25, 2022
🐳 A most popular sql audit platform for mysql
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Jan 6, 2023
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

Nov 9, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Jan 4, 2023
GitHub's Online Schema Migrations for MySQL
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

Jan 4, 2023
Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.
Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.

简介 Gaea是小米中国区电商研发部研发的基于mysql协议的数据库中间件,目前在小米商城大陆和海外得到广泛使用,包括订单、社区、活动等多个业务。Gaea支持分库分表、sql路由、读写分离等基本特性,更多详细功能可以参照下面的功能列表。其中分库分表方案兼容了mycat和kingshard两个项目的路

Dec 30, 2022
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件 English 漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球 而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 : Redis MongoDB Cl

Dec 30, 2022