Build platforms that flexibly mix SQL, batch, and stream processing paradigms

This change is 

Connects #154 Connects #159

Please review commit by commit. First commit: There is an outstanding ticket (#154) to create a loopback server for testing against a consumer. This implements a basic loopback server in the model of the broker loopback server. This is a first pass implementation and sought to solve the problems introduced in writing tests for the second commit. As this testing tool is used more I expect it will be expanded upon.

Second commit: Add a max transaction size flag to all of the apply/edit commands in gazctl.

Stop-writes attempts to monitor local disk, and if the disk fills beyond a threshold will block all attempts to write to Gazette (which requires local spooling) until disk usage has dropped.

This behavior has bitten us more than it's helped, because stop-writes often results in ensemble deadlocks which can only be resolved by letting writes proceed. It also often impacts (sometimes important) services which are innocent bystanders and not the primary culprits of disk usage.

Alternatives include monitoring at the application level, implementing quotas on disks assigned to applications, and likely other approaches.

This change is 

As an operator, I want to be able to list and/or prune fragments in the backing stores which correspond to "dead end" forks in history and represent some time range or offset range.

These forks commonly occur when gazette pods are deleted and can create a lot of clutter in the cloud storage. An operator may want to explicitly delete clean these up after an on-call issue even though they have not exceeded the configured retention.

• This functionality could be hooked into gazctl journals prune so that it automatically removes forked fragments, avoiding the caveat given in its description.

Prefer Kubernetes Service dns over env vars for service discovery because env vars aren't available across Namespaces or for ExternalName Services, and env vars are required at Pod startup, whereas dns is more resilient to Endpoint updates.

Remove 127.0.0.1 as the default for discovering etcd and gazette and expect Kubernetes first. Finally, remove _SERVICE_HOST and _SERVICE_PORT since this will clobber the dns service discovery in favor of the env vars, which we don't prefer.

This change is 

Previously, the socks server endpoint is discovered from the environment variables SOCKS_SERVER_SERVICE_HOST and SOCKS_SERVER_SERVICE_PORT.

This change keeps this behavior but also allows the endpoint to be configured via cli flag.

This change is for making it standard to discover Kubernetes Services using DNS rather than environment variables because we use ExternalName Services for cross-Namespace service discovery and ExternalName Services don't have environment variable support.

This change is 

This is a collection of patches addressing outstanding issues I'm aware of from recovery during recent net-splits.

Testing:

• I've done a bunch of flake hunting with unit tests (eg --count=X00) and feel pretty confident here.
• I've run it through integration tests on a local stack. Plan to do more manual testing here, but wanted to kick this over now.

Fixes #197, #198

This change is 

We want to run a process on gazette fragments which we'd like to have include all fragments from a previous 24hr window (e.g. 12:00:00 AM PT-11:59:59 PM PT).

Today it's difficult to determine when to run the aforementioned process because there is no guarantee as to when a spool created prior to the 24h window will close and upload the remaining events from that window to cloud storage. Especially if there is high variability in the number of writes to the consumed topic, the timing could be highly variable (I believe). Currently the only way to make a guarantee that we have included all fragments from a given window is to read all fragments until we hit one from a timestamp after the window (and perhaps fail our process if no such fragment is read).

The goal would be to have a process (perhaps a cron) which submits a request to the broker to close its current spool (even if said spool is still relatively small) and then upload that spool to cloud storage. This should provide a much easier ability to guarantee we are processing all fragments from a previous time frame.

@joshk0 could you take a look at this one please? Please lmk if I should tag someone else (although you seem to have contributed quite a lot to this file).

This change is 

I recommend combining commits for review.

In addition to enhancing gazctl, I added CircleCI branch filters so branches that start with "draft-" or "draft/" do not trigger builds. This allows developers a way to push partial work to GH without flooding their inbox with build failure notifications.

Connects #99

This change is 

As an operator, I want to provide a shard label selector, and have journal fragments of matched ShardSpec recovery logs that are not required for recovering from any of a shard’s current hints, to be deleted.

(Make gRPC api for fetching current hints?)

This PR adds a gazctl shard prune-log command, as well as a number of related underlying changes.

Log pruning identifies Fragments of a hinted recovery log which can safely be removed, while preserving the ability to play back the pruned hints.

The intention is that regular pruning would be operational-ized by pointing gazctl as .lastRecovered hints of consumer shards.

Extensive testing against the stream-sum example was performed by:

• Heavily and continuously loading a local stack
• Regularly deleting the stream-sum-summer consumer pods, to force playback and recovery.
• Regularly using the gazctl shard prune-log command to prune both .lastRecovered and also primary hint paths.
• Verifying that no playback errors occurred.

This change is 

Bumps certifi from 2019.9.11 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This change is 

The docs on deleting shard specs say that you can add delete: true under the common section. This does not work, though, since we unmarshal that field into a pc.ShardSpec, which does not contain the delete property.

We should update either the docs or the code here.

Adds a prometheus collector implementation to the broker/client package that allows tracking of disk usage across file-backed appendBuffers.

If AppendRPCs are retrying, or taking longer than normal, users can timeout waits on the AsyncAppend and alert on the rate of these timeouts to indicate that there is potential unhealthiness in the write path. While this is happening its useful to see the amount of data that the producers have buffered on disk. Given limited disk resources this is the key indicator for how much time the operator has to resolve any issues before there are real availability issues / data loss.

This change is 

Observed from a consumer application on a recent release, where gazette and the consumer were rolling in tandem (the consumer happened to pick a gazette broker that was exiting):

{"err":"beginRecovery: fetching log spec: rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.0.35:51576-\u003e10.3.247.101:8080: read: connection reset by peer","level":"error","msg":"serveStandby failed","shard":"/gazette/consumers/flow/reactor/items/ ... ","time":"2021-09-24T18:58:07Z"}

During a recent automated GKE upgrade, all brokers and Etcd pods were simultaneously signaled to exit (not ideal, but also not the issue at hand).

Etcd pods exited, and on the way out Gazette brokers observed transport-level errors which were treated as terminal, and caused a controlled but fatal shutdown across all brokers (along with a pod restart):

{"err":"service.Watch: rpc error: code = Unknown desc = closing transport due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR, debug data: ","level":"fatal","msg":"broker task failed","time":"2021-09-01T14:08:13Z"}

The shutdown was controlled -- no data loss is believed or expected to have occurred -- but it did cause cluster consistency to be lost and require operator intervention (gazctl journals reset-head).

What should happen instead

Brokers should have retried the Etcd watch on this transport-level error.