A Microservice Toolkit from The New York Times

The New York Times

Last update: Jan 7, 2023

Comments: 17

Gizmo Microservice Toolkit

This toolkit provides packages to put together server and pubsub daemons with the following features:

Standardized configuration and logging
Health check endpoints with configurable strategies
Configuration for managing pprof endpoints and log levels
Basic interfaces to define expectations and vocabulary
Structured logging containing basic request information
Useful metrics for endpoints
Graceful shutdowns

Install

Notice the capitalization!

go get github.com/NYTimes/gizmo/...

Import Path Change Notice

The New York Times recently changed the github organization from NYTimes to nytimes. This should not affect the installation as long as you use the proper casing NYTimes and not nytimes per installation instructions above.

However, the intention is to migrate the import paths to be consistent with how it's shown on GitHub. This will be a breaking change and we will introduce a major tag when we update the code. Therefore, the import path will go from github.com/NYTimes/gizmo/server to github.com/nytimes/gizmo/v2/server. This ensures that people will not have type-mismatches between import path changes.

Packages

`server`

The server package is the bulk of the toolkit and relies on server.Config to manage Server implementations.

It offers 1 server implementation:

SimpleServer, which is capable of handling basic HTTP and JSON requests via 5 of the available Service implementations: SimpleService, JSONService, ContextService, MixedService and a MixedContextService.

`server/kit`

The server/kit package embodies Gizmo's goals to combine with go-kit.

In this package you'll find:
- A more opinionated server with fewer choices.
- go-kit used for serving HTTP/JSON & gRPC used for serving HTTP2/RPC
- Monitoring, traces and metrics are automatically registered if running within App Engine, Cloud Run, Kubernetes Engine, Compute Engine or AWS EC2 Instances.
  - to change the name and version for Error reporting and Traces use SERVICE_NAME and SERVICE_VERSION environment variables.
- Logs go to stdout locally or directly to Stackdriver when in GCP.
- Using Go's 1.8 graceful HTTP shutdown.
- Services using this package are expected to deploy to GCP.

`observe`

The observe package provides observability helpers for metrics and tracing through OpenCensus

server/kit (and soon SimpleServer) utilizes this package to create a StackDriver exporter with sane defaults
GoogleProjectID, IsGAE, and IsCloudRun can help you make decisions about the underlying platform

`auth`

The auth package provides primitives for verifying inbound authentication tokens:

The PublicKeySource interface is meant to provide *rsa.PublicKeys from JSON Web Key Sets.
The Verifier struct composes key source implementations with custom decoders and verifier functions to streamline server side token verification.

`auth/gcp`

The auth/gcp package provides 2 Google Cloud Platform based auth.PublicKeySource and oauth2.TokenSource implementations:

The "Identity" key source and token source rely on GCP's identity JWT mechanism for asserting instance identities. This is the preferred method for asserting instance identity on GCP.
The "IAM" key source and token source rely on GCP's IAM services for signing and verifying JWTs. This method can be used outside of GCP, if needed and can provide a bridge for users transitioning from the 1st generation App Engine (where Identity tokens are not available) runtime to the 2nd.

The auth/gcp package also includes an Authenticator, which encapsulates a Google Identity verifier and oauth2 credentials to manage a basic web auth flow.

`config`

The config package contains a handful of useful functions to load to configuration structs from JSON files or environment variables.

There are also many structs for common configuration options and credentials of different Cloud Services and Databases.

`pubsub`

The pubsub package contains two (publisher and subscriber) generic interfaces for publishing data to queues as well as subscribing and consuming data from those queues.

There are 4 implementations of pubsub interfaces:

For pubsub via Amazon's SNS/SQS, you can use the pubsub/aws package
For pubsub via Google's Pubsub, you can use the pubsub/gcp package
For pubsub via Kafka topics, you can use the pubsub/kafka package
For publishing via HTTP, you can use the pubsub/http package

`pubsub/pubsubtest`

The pubsub/pubsubtest package contains test implementations of the pubsub.Publisher, pubsub.MultiPublisher, and pubsub.Subscriber interfaces that will allow developers to easily mock out and test their pubsub implementations.

Examples

Several reference implementations utilizing server and pubsub are available in the examples subdirectory.
There are also examples within the GoDoc: here

_{If you experience any issues please create an issue and/or reach out on the #gizmo channel in the Gophers Slack Workspace with what you've found.}

_{The Gizmo logo was based on the Go mascot designed by Renée French and copyrighted under the Creative Commons Attribution 3.0 license.}

Owner

The New York Times

https://github.com/NYTimes/gizmo https://open.nytimes.com/introducing-gizmo-aa7ea463b208

Comments

Swallowed kit test error

A call to t.Fatal() from a goroutine inside a test gets silently dropped. From go doc testing.T:

A test ends when its Test function returns or calls any of the methods FailNow, Fatal, Fatalf, SkipNow, Skip, or Skipf. Those methods, as well as the Parallel method, must be called only from the goroutine running the Test function.

This PR changes kit.Run() to use channels to send errors and readiness state, and another channel to receive quit messages from a calling goroutine. It eliminates the potential race condition of using time.Sleep to wait for a goroutine to become ready. It catches previously-swallowed errors and causes a test failure should one occur.

The requirement to use system signals to communicate with goroutines is removed.
Makes gcp attributes accessible

Just like my last pull request adding a function to Subscriber, this pull request adds a function to SubMessage making the underlying attributes sent with GCP accessible.
implement new AddPermissionWithContext func within TestSNSAPI

This will add unimplemented functions for the TestSNSAPI interface to make tests pass. If this build passes should I also add the other context funcs?
Add MultiPublisher pubsub interface

The Publisher interface only allows for publishing a single message at a time. We're working with bulk API endpoints where we need to publish 50-100 messages at once. I spoke with @jprobinson on the #gizmo channel about this contribution.

This PR introduces a MultiPublisher interface that supports a PublishMulti method that looks like this:

PublishMulti(ctx context.Context, keys []string, messages []proto.Message) error

While JP originally suggested adding this to the Publisher interface, as soon as I started digging in I realized not all publishers support this feature, like AWS SNS. See here and here)

As a result I opted to create a new interface so it is optional, and added implementations for GCP and pubsubtest packages.
Fix and refactor prometheus metrics setup
As discussed with @jprobinson - this fixes Prometheus metrics in gizmo by

not using invalid metric names but using the standard prometheus.InstrumentHandler() that will generate http_* metrics and use the full endpoint pathname as a label.

expose prometheus metrics at /metrics

🍺
changing HTTPAccessLog to be a pointer

nil = no log, "stdout" = os.Stdout, anything else logs to a file

I think this would only be a breaking change for folks who've configured the access log via code as opposed to via environment variable?
Extract GCP Monitoring
Created a new gcputil package (following ioutil/serverutil pattern)

Moved helpful initialization functions out of server/kit into the new package so that SimpleServer or really any Go program can initialize metrics/tracing.

Added a new function RegisterOpenCensus which makes it easier for the user to just call this function, without having to manually get the metadata, insert them into the opts, get the opts, insert the opts into the Exporter.

I bet server/kit can probably use RegisterOpenCensus with some refactoring in the future

fixes https://github.com/NYTimes/gizmo/issues/185
Breaking up the `config` and `pubsub` packages by dependency
One of the largest of complaints against Gizmo so far has been how the toolkit lumps all dependencies into the config and pubsub packages, often requiring users to vendor or fetch an absurd amount of packages just to work with a simple server.

This PR hopes to fix this problem by breaking out the config and pubsub packages into a set of subpackages, which follows a similar structure to Go Kit and it's implementations. The new package structure:

├── config │ ├── aws │ ├── combined │ ├── cookie │ ├── gcp │ ├── metrics │ ├── mongodb │ ├── mysql │ └── oracle ├── pubsub │ ├── aws │ ├── gcp │ ├── kafka │ └── pubsubtest ├── server └── web

Notable changes:

config.Config is now config/combined.Config

config.Server is now server.Config

If a "config" type got moved to a package with the same name, it is now likely renamed as Config.

example: config.MongoDB moved to config/mongodb.Config
PubSub GCP MaxExtension not changeable

Hi,

I'm trying to use the GCP Pubsub throught github.com/NYTimes/gizmo/pubsub/gcp but my receiver is running an import with a long duration surpassing the defaultMaxExtension = 60 * time.Second as defined on gcp.go:51. Causing the message to be resend because the receiver failed to Ack() the message.

Maybe i'm missing something but i don't see a way to change the MaxExtension on the underlying subscription's ReceiveSettings

I guess this requires a change in the code to make customisable? I'm happy to send in a pull request but would like to hear if it's needed an how you would like to see it implemented?

Adding a new `server.Router` interface and `config.Server.RouterType` option with 2 implementations

Gizmo users who need the performance of julienschmidt/httprouter more than the handy URL matching of Gorilla web toolkit's mux, can now configure the server.SimplerServer and server.RPCServer to use an alternate Router implementation.

A new attribute in config.Server, RouterType, accepts the value "gorilla" (the default) to use Gorilla mux.Router or "fast" to use httprouter.Router.

To keep the use of idiomatic http.Handlers throughout Gizmo, instead of using httprouter.Handle, a new middleware function HTTPToFastRoute injects the httprouter.Params into a Gorilla request context. To retrieve the parameters, a new web.Vars(r *http.Request) map[string]string function is available and can be used like Gorilla's mux.Vars. The GorillaRouter also copies the values from mux.Vars into web.Vars so endpoints can be agnostic to which implementation is being used.

New benchmarks with the "Fast" prefix have been added to compare the two router types:

BenchmarkFastSimpleServer_NoParam-2               500000              3705 ns/op
BenchmarkFastSimpleServer_WithParam-2             300000              4653 ns/op
BenchmarkSimpleServer_NoParam-2                   200000              6282 ns/op
BenchmarkSimpleServer_WithParam-2                 200000              7823 ns/op
BenchmarkFastJSONServer_JSONPayload-2             300000              5210 ns/op
BenchmarkFastJSONServer_NoParam-2                 500000              2680 ns/op
BenchmarkFastJSONServer_WithParam-2               500000              3092 ns/op
BenchmarkJSONServer_JSONPayload-2                 200000              7838 ns/op
BenchmarkJSONServer_NoParam-2                     500000              3562 ns/op
BenchmarkJSONServer_WithParam-2                   300000              4122 ns/op
BenchmarkFastContextSimpleServer_NoParam-2        500000              3883 ns/op
BenchmarkFastContextSimpleServer_WithParam-2      300000              4791 ns/op
BenchmarkContextSimpleServer_NoParam-2            200000              6469 ns/op
BenchmarkContextSimpleServer_WithParam-2          200000              7838 ns/op

SB-10000: Adding app- and pipeline-ids to server
Helix jira ticket SB-10000: https://jira.nyt.net/browse/SB-10000

Pull Request Checklist

[x] Ticket number has been referenced

[x] Description has been provided

[x] Any necessary document changes have been made

[x] Testing instructions have been provided

[x] Unit tests have passed

[x] go fmt, go vet and golint have been run

[x] Link to the PR has been added to the associated ticket

Description

Adding two types of IDs to gizmo:

One type application-specific. It generates a random (v4) UUID and optionally prepends a prefix. An interface has been defined so we can specify other schemes if other use cases arise.

The other type is for the entire pipeline, and is meant to trace a call from its initial request at some top-level application, all the way through.

To support these, middleware has been added that, in both cases, sets the X-Request-ID header (in the case of application IDs, only if the header is not present) to the request and response, and writes it to the context.

Dependencies were updated as a part of running make test (as far as I can tell.) Happy to revert the go.mod and go.sum files if needs be.
Bump github.com/aws/aws-sdk-go from 1.31.3 to 1.33.0
Bumps github.com/aws/aws-sdk-go from 1.31.3 to 1.33.0.

Changelog

Sourced from github.com/aws/aws-sdk-go's changelog.

Release v1.33.0 (2020-07-01)

Service Client Updates

service/appsync: Updates service API and documentation

service/chime: Updates service API and documentation

This release supports third party emergency call routing configuration for Amazon Chime Voice Connectors.

service/codebuild: Updates service API and documentation

Support build status config in project source

service/imagebuilder: Updates service API and documentation

service/rds: Updates service API

This release adds the exceptions KMSKeyNotAccessibleFault and InvalidDBClusterStateFault to the Amazon RDS ModifyDBInstance API.

service/securityhub: Updates service API and documentation

SDK Features

service/s3/s3crypto: Introduces EncryptionClientV2 and DecryptionClientV2 encryption and decryption clients which support a new key wrapping algorithm kms+context. (#3403)

DecryptionClientV2 maintains the ability to decrypt objects encrypted using the EncryptionClient.

Please see s3crypto documentation for migration details.

Release v1.32.13 (2020-06-30)

Service Client Updates

service/codeguru-reviewer: Updates service API and documentation

service/comprehendmedical: Updates service API

service/ec2: Updates service API and documentation

Added support for tag-on-create for CreateVpc, CreateEgressOnlyInternetGateway, CreateSecurityGroup, CreateSubnet, CreateNetworkInterface, CreateNetworkAcl, CreateDhcpOptions and CreateInternetGateway. You can now specify tags when creating any of these resources. For more information about tagging, see AWS Tagging Strategies.

service/ecr: Updates service API and documentation

Add a new parameter (ImageDigest) and a new exception (ImageDigestDoesNotMatchException) to PutImage API to support pushing image by digest.

service/rds: Updates service documentation

Documentation updates for rds

Release v1.32.12 (2020-06-29)

Service Client Updates

service/autoscaling: Updates service documentation and examples

Documentation updates for Amazon EC2 Auto Scaling.

service/codeguruprofiler: Updates service API, documentation, and paginators

service/codestar-connections: Updates service API, documentation, and paginators

service/ec2: Updates service API, documentation, and paginators

Virtual Private Cloud (VPC) customers can now create and manage their own Prefix Lists to simplify VPC configurations.

Release v1.32.11 (2020-06-26)

Service Client Updates

service/cloudformation: Updates service API and documentation

ListStackInstances and DescribeStackInstance now return a new StackInstanceStatus object that contains DetailedStatus values: a disambiguation of the more generic Status value. ListStackInstances output can now be filtered on DetailedStatus using the new Filters parameter.

service/cognito-idp: Updates service API

... (truncated)

Commits

b5d0d96 Release v1.33.0 (2020-07-01)

4705b39 Fix CHANGELOG_PENDING.md entry (#3404)

dda576e service/cloudfront/sign: Fix Documentation Typo (#3402)

35fa6dd service/s3/s3crypto: V2 Client Release (#3403)

9dbc703 Release v1.32.13 (2020-06-30) (#3400)

6ec1b39 Release v1.32.12 (2020-06-29) (#3397)

15f0e61 Release v1.32.11 (2020-06-26) (#3395)

cd54c31 Release v1.32.10 (2020-06-25) (#3393)

8932aba Release v1.32.9 (2020-06-24) (#3391)

ea6bf35 Release v1.32.8 (2020-06-23) (#3388)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.
Bump github.com/aws/aws-sdk-go from 1.31.3 to 1.40.13
Bumps github.com/aws/aws-sdk-go from 1.31.3 to 1.40.13.

Release notes

Sourced from github.com/aws/aws-sdk-go's releases.

Release v1.40.13 (2021-08-02)

Service Client Updates

service/greengrassv2: Updates service API and documentation

service/ssm-contacts: Updates service API and documentation

Release v1.40.12 (2021-07-30)

Service Client Updates

service/appsync: Updates service API and documentation

service/elasticloadbalancingv2: Updates service documentation

service/sagemaker: Updates service API, documentation, and waiters

API changes with respect to Lambda steps in model building pipelines. Adds several waiters to async Sagemaker Image APIs. Add more instance types to AppInstanceType field

service/secretsmanager: Updates service API and documentation

Add support for KmsKeyIds in the ListSecretVersionIds API response

Release v1.40.11 (2021-07-29)

Service Client Updates

service/chime: Updates service API and documentation

Adds support for live transcription of meetings with Amazon Transcribe and Amazon Transcribe Medical. The new APIs, StartMeetingTranscription and StopMeetingTranscription, control the generation of user-attributed transcriptions sent to meeting clients via Amazon Chime SDK data messages.

service/ec2: Updates service API and documentation

This release adds support for G4ad xlarge and 2xlarge instances powered by AMD Radeon Pro V520 GPUs and AMD 2nd Generation EPYC processors

service/iot: Updates service API and documentation

Increase maximum credential duration of role alias to 12 hours.

service/iotsitewise: Updates service API and documentation

service/savingsplans: Updates service API

Release v1.40.10 (2021-07-28)

Service Client Updates

service/cloudformation: Updates service API and documentation

SDK update to support Importing existing Stacks to new/existing Self Managed StackSet - Stack Import feature.

service/sso-admin: Updates service documentation

Release v1.40.9 (2021-07-27)

Service Client Updates

service/batch: Updates service API and documentation

Add support for ListJob filters

service/iotanalytics: Updates service API and documentation

service/iotwireless: Updates service API and documentation

service/models.lex.v2: Updates service waiters

service/quicksight: Updates service API and documentation

Add support to use row-level security with tags when embedding dashboards for users not provisioned in QuickSight

... (truncated)

Changelog

Sourced from github.com/aws/aws-sdk-go's changelog.

Release v1.40.13 (2021-08-02)

Service Client Updates

service/greengrassv2: Updates service API and documentation

service/ssm-contacts: Updates service API and documentation

Release v1.40.12 (2021-07-30)

Service Client Updates

service/appsync: Updates service API and documentation

service/elasticloadbalancingv2: Updates service documentation

service/sagemaker: Updates service API, documentation, and waiters

API changes with respect to Lambda steps in model building pipelines. Adds several waiters to async Sagemaker Image APIs. Add more instance types to AppInstanceType field

service/secretsmanager: Updates service API and documentation

Add support for KmsKeyIds in the ListSecretVersionIds API response

Release v1.40.11 (2021-07-29)

Service Client Updates

service/chime: Updates service API and documentation

Adds support for live transcription of meetings with Amazon Transcribe and Amazon Transcribe Medical. The new APIs, StartMeetingTranscription and StopMeetingTranscription, control the generation of user-attributed transcriptions sent to meeting clients via Amazon Chime SDK data messages.

service/ec2: Updates service API and documentation

This release adds support for G4ad xlarge and 2xlarge instances powered by AMD Radeon Pro V520 GPUs and AMD 2nd Generation EPYC processors

service/iot: Updates service API and documentation

Increase maximum credential duration of role alias to 12 hours.

service/iotsitewise: Updates service API and documentation

service/savingsplans: Updates service API

Release v1.40.10 (2021-07-28)

Service Client Updates

service/cloudformation: Updates service API and documentation

SDK update to support Importing existing Stacks to new/existing Self Managed StackSet - Stack Import feature.

service/sso-admin: Updates service documentation

Release v1.40.9 (2021-07-27)

Service Client Updates

service/batch: Updates service API and documentation

Add support for ListJob filters

service/iotanalytics: Updates service API and documentation

service/iotwireless: Updates service API and documentation

service/models.lex.v2: Updates service waiters

service/quicksight: Updates service API and documentation

Add support to use row-level security with tags when embedding dashboards for users not provisioned in QuickSight

... (truncated)

Commits

56b1540 Release v1.40.13 (2021-08-02) (#4041)

ee7b112 Release v1.40.12 (2021-07-30) (#4036)

c48ce71 Release v1.40.11 (2021-07-29) (#4035)

abb47fb service/s3: Update unit test to clarify implemented behavior (#4033)

a4a27c7 Release v1.40.10 (2021-07-28) (#4032)

5f4eb6a Release v1.40.9 (2021-07-27) (#4031)

9e26171 Release v1.40.8 (2021-07-26) (#4026)

4c222f5 Release v1.40.7 (2021-07-23) (#4024)

43d05f6 Release v1.40.6 (2021-07-22) (#4019)

287bf61 Release v1.40.5 (2021-07-21) (#4017)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

@dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

Update frequency (including time of day and day of week)

Pull request limits (per update run and/or open at any time)

Out-of-range updates (receive only lockfile updates, if desired)

Security updates (receive only security updates, if desired)
Bump cloud.google.com/go from 0.57.0 to 0.89.0
Bumps cloud.google.com/go from 0.57.0 to 0.89.0.

Release notes

Sourced from cloud.google.com/go's releases.

google-cloud-go v0.89.0

Features

assuredworkloads: Add EU Regions And Support compliance regime (a52baa4)

datacatalog: Added support for BigQuery connections entries feat: Added support for BigQuery routines entries feat: Added usage_signal field feat: Added labels field feat: Added ReplaceTaxonomy in Policy Tag Manager Serialization API feat: Added support for public tag templates feat: Added support for rich text tags docs: Documentation improvements (a52baa4)

datafusion: start generating apiv1 (e55a016)

iap: start generating apiv1 (e55a016)

internal/carver: add tooling to help carve out sub-modules (#4417) (a7e28f2)

networkconnectivity: Add files for Network Connectivity v1 API. (a52baa4)

retail: Add restricted Retail Search features for Retail API v2. (a52baa4)

secretmanager: In Secret Manager, users can now use filter to customize the output of ListSecrets/ListSecretVersions calls (a52baa4)

securitycenter: add finding_class and indicator fields in Finding (a52baa4)

speech: add total_billed_time response field. fix!: phrase_set_id is required field in CreatePhraseSetRequest. fix!: custom_class_id is required field in CreateCustomClassRequest. (a52baa4)

storagetransfer: start generating apiv1 (#4505) (f2d531d)

Bug Fixes

internal/gapicgen: exec Stdout already set (#4509) (41246e9)

internal/gapicgen: tidy all after dep bump (#4515) (9401be5), refs #4434

google-cloud-go v0.88.0

⚠ BREAKING CHANGES

cloudbuild/apiv1: Proto had a prior definitions of WorkerPool resources which were never supported. This change replaces those resources with definitions that are currently supported.

Features

cloudbuild/apiv1: add a WorkerPools API (19ea3f8)

cloudbuild/apiv1: Implementation of Build Failure Info: - Added message FailureInfo field (19ea3f8)

osconfig/agentendpoint: OSConfig AgentEndpoint: add basic os info to RegisterAgentRequest, add WindowsApplication type to Inventory (8936bc3)

resourcesettings: Publish Cloud ResourceSettings v1 API (43ad3cb)

Bug Fixes

internal/godocfx: set exit code, print cmd output, no go get ... (#4445) (cc70f77)

internal: detect module for properly generating docs URLs (#4460) (1eaba8b), refs #4447

kms: Updating WORKSPACE files to use the newest version of the Typescript generator. (8936bc3)

google-cloud-go v0.87.0

Features

container: allow updating security group on existing clusters (528ffc9)

... (truncated)

Changelog

Sourced from cloud.google.com/go's changelog.

0.89.0 (2021-07-29)

Features

assuredworkloads: Add EU Regions And Support compliance regime (a52baa4)

datacatalog: Added support for BigQuery connections entries feat: Added support for BigQuery routines entries feat: Added usage_signal field feat: Added labels field feat: Added ReplaceTaxonomy in Policy Tag Manager Serialization API feat: Added support for public tag templates feat: Added support for rich text tags docs: Documentation improvements (a52baa4)

datafusion: start generating apiv1 (e55a016)

iap: start generating apiv1 (e55a016)

internal/carver: add tooling to help carve out sub-modules (#4417) (a7e28f2)

networkconnectivity: Add files for Network Connectivity v1 API. (a52baa4)

retail: Add restricted Retail Search features for Retail API v2. (a52baa4)

secretmanager: In Secret Manager, users can now use filter to customize the output of ListSecrets/ListSecretVersions calls (a52baa4)

securitycenter: add finding_class and indicator fields in Finding (a52baa4)

speech: add total_billed_time response field. fix!: phrase_set_id is required field in CreatePhraseSetRequest. fix!: custom_class_id is required field in CreateCustomClassRequest. (a52baa4)

storagetransfer: start generating apiv1 (#4505) (f2d531d)

Bug Fixes

internal/gapicgen: exec Stdout already set (#4509) (41246e9)

internal/gapicgen: tidy all after dep bump (#4515) (9401be5), refs #4434

0.88.0 (2021-07-22)

⚠ BREAKING CHANGES

cloudbuild/apiv1: Proto had a prior definitions of WorkerPool resources which were never supported. This change replaces those resources with definitions that are currently supported.

Features

cloudbuild/apiv1: add a WorkerPools API (19ea3f8)

cloudbuild/apiv1: Implementation of Build Failure Info: - Added message FailureInfo field (19ea3f8)

osconfig/agentendpoint: OSConfig AgentEndpoint: add basic os info to RegisterAgentRequest, add WindowsApplication type to Inventory (8936bc3)

resourcesettings: Publish Cloud ResourceSettings v1 API (43ad3cb)

Bug Fixes

internal/godocfx: set exit code, print cmd output, no go get ... (#4445) (cc70f77)

internal: detect module for properly generating docs URLs (#4460) (1eaba8b), refs #4447

kms: Updating WORKSPACE files to use the newest version of the Typescript generator. (8936bc3)

0.87.0 (2021-07-13)

Features

container: allow updating security group on existing clusters (528ffc9)

... (truncated)

Commits

a3e97da chore: release 0.89.0 (#4486)

9401be5 fix(internal/gapicgen): tidy all after dep bump (#4515)

a52baa4 chore(all): auto-regenerate gapics (#4510)

663c899 feat(bigquery/storage/managedwriter): naming and doc improvements (#4508)

41246e9 fix(internal/gapicgen): exec Stdout already set (#4509)

d8ec92b test(bigquery/storage/managedwriter): test all stream types (#4507)

c6cf659 fix(bigquery/storage/managedwriter): fix double-close error, add tests (#4502)

9923fd1 chore(all): auto-regenerate gapics (#4499)

f2d531d feat(storagetransfer): start generating apiv1 (#4505)

35ceae2 fix(bigtable/bttest): Emulator too lenient for empty RowMutation (#4359)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

@dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

Update frequency (including time of day and day of week)

Pull request limits (per update run and/or open at any time)

Out-of-range updates (receive only lockfile updates, if desired)

Security updates (receive only security updates, if desired)
Bump google.golang.org/api from 0.25.0 to 0.52.0
Bumps google.golang.org/api from 0.25.0 to 0.52.0.

Release notes

Sourced from google.golang.org/api's releases.

google-api-go-client v0.52.0

Features

all: auto-regenerate discovery clients , refs #1137 #1135 #1134 #1133 #1131 #1130 #1128

google-api-go-client v0.51.0

Features

all: auto-regenerate discovery clients , refs #1126 #1125 #1122 #1121 #1119 #1118 #1116 #1115 #1114 #1113 #1112 #1110 #1109 #1108 #1107 #1106 #1105 #1104 #1102 #1101 #1100 #1098

google-api-go-client v0.50.0

Features

all: auto-regenerate discovery clients , refs #1097 #1095 #1093 #1090 #1089 #1088 #1086

Bug Fixes

internal: fix self-signed detection with scopes (#1096) (ff1d20b), refs #1092

google-api-go-client v0.49.0

Features

all: auto-regenerate discovery clients , refs #1085 #1084 #1082 #1080 #1079 #1078 #1076 #1073 #1071 #1070 #1068 #1067 #1066 #1064 #1063 #1060

internal: [AIP-4111] support scopes for self-signed JWT auth flow (#1075) (29cab68)

google-api-go-client v0.48.0

Features

all: auto-regenerate discovery clients , refs #1059 #1057 #1056 #1054 #1053 #1052 #1051 #1050 #1046 #1045 #1044 #1043 #1042 #1039 #1033 #1032 #1030

Bug Fixes

transport: perform mTLS endpoint switching when custom http client is provided (#1038) (c07165f)

google-api-go-client v0.47.0

Features

... (truncated)

Changelog

Sourced from google.golang.org/api's changelog.

0.52.0 (2021-07-29)

Features

all: auto-regenerate discovery clients , refs #1137 #1135 #1134 #1133 #1131 #1130 #1128

0.51.0 (2021-07-22)

Features

all: auto-regenerate discovery clients , refs #1126 #1125 #1122 #1121 #1119 #1118 #1116 #1115 #1114 #1113 #1112 #1110 #1109 #1108 #1107 #1106 #1105 #1104 #1102 #1101 #1100 #1098

0.50.0 (2021-06-30)

Features

all: auto-regenerate discovery clients , refs #1097 #1095 #1093 #1090 #1089 #1088 #1086

Bug Fixes

internal: fix self-signed detection with scopes (#1096) (ff1d20b), refs #1092

0.49.0 (2021-06-23)

Features

all: auto-regenerate discovery clients , refs #1085 #1084 #1082 #1080 #1079 #1078 #1076 #1073 #1071 #1070 #1068 #1067 #1066 #1064 #1063 #1060

internal: [AIP-4111] support scopes for self-signed JWT auth flow (#1075) (29cab68)

0.48.0 (2021-06-07)

Features

all: auto-regenerate discovery clients , refs #1059 #1057 #1056 #1054 #1053 #1052 #1051 #1050 #1046 #1045 #1044 #1043 #1042 #1039 #1033 #1032 #1030

Bug Fixes

transport: perform mTLS endpoint switching when custom http client is provided (#1038) (c07165f)

0.47.0 (2021-05-19)

Features

... (truncated)

Commits

3e2b6a2 chore: release 0.52.0 (#1129)

f9e05da feat(all): auto-regenerate discovery clients (#1137)

f2e3432 feat(all): auto-regenerate discovery clients (#1135)

108d63c feat(all): auto-regenerate discovery clients (#1134)

9de99be chore(all): update all (#1132)

f9af97b feat(all): auto-regenerate discovery clients (#1133)

053a596 feat(all): auto-regenerate discovery clients (#1131)

26f335b feat(all): auto-regenerate discovery clients (#1130)

db470df feat(all): auto-regenerate discovery clients (#1128)

81b294f test(downscope): add integration tests for token downscoping with Credential ...

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

@dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

Update frequency (including time of day and day of week)

Pull request limits (per update run and/or open at any time)

Out-of-range updates (receive only lockfile updates, if desired)

Security updates (receive only security updates, if desired)
Bump cloud.google.com/go/pubsub from 1.3.1 to 1.13.0
Bumps cloud.google.com/go/pubsub from 1.3.1 to 1.13.0.

Release notes

Sourced from cloud.google.com/go/pubsub's releases.

bigquery/v1.13.0

Support retries for specific http2 transport race.

Remove unused datasource client from bigquery/datatransfer.

Adds support for authorized User Defined Functions (UDFs).

Documentation improvements.

Various updates to autogenerated clients.

spanner spanner/v1.13.0

Features

spanner/spannertest: implement ANY_VALUE aggregation function (#3428) (e16c3e9)

spanner/spannertest: implement FULL JOIN (#3218) (99f7212)

spanner/spannertest: implement SELECT ... FROM UNNEST(...) (#3431) (deb466f)

spanner/spannertest: support array literals (#3438) (69e0110)

spanner/spannertest: support AVG aggregation function (#3286) (4788415)

spanner/spannertest: support Not Null constraint (#3491) (c36aa07)

spanner/spannertest: support UPDATE DML (#3201) (1dec6f6)

spanner/spansql: define structures and parse UPDATE DML statements (#3192) (23b6904)

spanner/spansql: support DATE and TIMESTAMP literals (#3557) (1961930)

spanner/spansql: support for parsing generated columns (#3373) (9b1d06f)

spanner/spansql: support NUMERIC data type (#3411) (1bc65d9)

spanner: Add a DirectPath fallback integration test (#3487) (de821c5)

spanner: attempt DirectPath by default (#3516) (bbc61ed)

spanner: include User agent (#3465) (4e1ef1b)

spanner: run E2E test over DirectPath (#3466) (18e3a4f)

spanner: support NUMERIC in mutations (#3328) (fa90737)

Bug Fixes

spanner: fix session leak (#3461) (11fb917), refs #3460

storage storage/v1.13.0

Features

storage: add missing StorageClass in BucketAttrsToUpdate (#3038) (2fa1b72)

storage: add projection parameter for BucketHandle.Objects() (#3549) (9b9c3dc)

Bug Fixes

storage: fix endpoint selection logic (#3172) (99edf0d)

pubsub pubsub/v1.13.0

... (truncated)

Commits

4983913 chore: release pubsub 1.13.0 (#4461)

7047808 feat(spanner/spansql): support table_hint_expr at from_clause on query_statem...

b7ce742 fix(pubsublite): set a default grpc connection pool size of 8 (#4462)

f1b7c8b feat(pubsub/pstest): add ability to create a pstest server listening on (#4459)

691e923 fix(bigtable): emulator crashes in SampleRowKeys (#4455)

66fa147 chore(all): update all (#4456)

1110dcf test(spanner): fix the failed TestColumnTypeErr test (#4450)

8f3275c test(spanner): fix errors not reporting when emulator test passes (#4451)

64fff7a chore(compute): add integration tests for compute (#4421)

60160af chore(storage): switch gapic to apiv2 (#4440)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

@dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

Update frequency (including time of day and day of week)

Pull request limits (per update run and/or open at any time)

Out-of-range updates (receive only lockfile updates, if desired)

Security updates (receive only security updates, if desired)
Bump github.com/go-kit/kit from 0.9.0 to 0.11.0
Bumps github.com/go-kit/kit from 0.9.0 to 0.11.0.

Release notes

Sourced from github.com/go-kit/kit's releases.

v0.11.0

A new release with several improvements and enhancements. The first one in a long while! Huge thanks to @sagikazarmark for putting in most of the gruntwork to make it happen! You're a superstar.

The biggest thing: package log has been extracted to a separate repository and module, go-kit/log. This means that if you or your project was importing go-kit/kit just to get package log, you can significantly reduce your go.mod and dep graph by switching to the new module. Note that we have no current plans to alias the existing go-kit/kit/log to the new go-kit/log module and package, nor to deprecate the current package in favor of the new one. They are two distinct packages with no defined relationship to each other. This may change in the future.

Major changes:

The log package was extracted to a separate repository

Examples were moved to a separate repository

Deprecated kitgen was removed

Thanks to the 22 contributors who contributed to this release! 🏌️‍♂️

Bug fixes

metrics/cloudwatch: log CloudWatch response error (#961) (thanks @Trane9991)

log: defer mutex unlocks for panic safety in SyncLogger (#974)

util/conn: close old connection before reconnect (#982) (thanks @chikaku)

log/term: fix build on GOOS=js GOARCH=wasm (#993) (thanks @mvdan)

transport/http/jsonrpc: move the ClientAfter calls to before the decode (#1008) (thanks @directionless)

sd/etcdv3: fix etcdv3 client won't return error when no endpoint is available (#1009) (thanks @wayjam)

metrics/generic: fix uint64 alignment (#1007) (thanks @ldez)

log: fix stdlibadapter when prefixed (#1036) (thanks @soven)

log: capture newlines in log stdlib (#1041) (thanks @SuperQ)

Enhancements

metrics/cloudwatch: use batch values API for CloudWatch PutMetric data call (#960) (thanks @Trane9991)

log: allow to use specific logrus level in the adaptor (#962) (thanks @Trane9991)

transport/http: add NewExplicitClient (#971)

transport/http/jsonrpc: add RequestID in error body when using the DefaultErrorEncoder (#969) (thanks @esenac)

transport/http/jsonrpc: add Version to JSON-RPC client request (#990) (thanks @shirolimit)

log: add WithSuffix to append key-value pairs to those passed to Log (#992) (thanks @vinayvinay)

sd/consul: improve inconsistent Consul SD index handling (#999) (thanks @vinayvinay)

all: dependency updates (#1029, #1095, #1097, #1098, #1106, #1118, #1115, #1119, #1124) (thanks @ChrisHines, @Enrico204, @sagikazarmark)

tracing/opencensus: add support for JSONRPC (#1022) (thanks @ryan-lang)

tracing/opentracing: improve endpoint middleware options (#1072) (thanks @alebabai)

auth/jwt: fix repetition of the word "token" in JWT (#1070) (thanks @amidam)

sd/zk: replace unmaintained zk library with drop-in replacement (#1120) (thanks @sagikazarmark)

cmd/kitgen: remove deprecated kitgen (#1121) (thanks @sagikazarmark)

Documentation, examples, tests

readme: change godoc to pkg.go.dev (#963) (thanks @relunctance)

readme: add links to generator tools (#964)

metrics/cloudwatch: fix bad Gauge test (#975) (thanks @Trane9991)

readme: update the link and description for go-micro (#989) (thanks @asim)

examples: add missing "to" preposition (#1014)

tracing/opencensus: fix failing tests (#1021) (thanks @ryan-lang)

log: fix doc comment (#1028) (thanks @vrazdalovschi)

... (truncated)

Commits

a6c5d58 Merge pull request #1129 from sagikazarmark/improve-example-references

4c47fd8 remove examples from gitignore

908c5cf docs: fix example links

d19ee33 Merge pull request #1128 from robbert229/patch-1

ccf3d8d fix a broken link to the addsvc example

f80eb06 Merge pull request #1121 from sagikazarmark/remove-kitgen

32681cc remove deprecated kitgen

2ca6ab2 Merge pull request #1112 from sagikazarmark/opentelemetry

a119c95 Merge pull request #1122 from sagikazarmark/nats-test-panic

2216160 Merge pull request #1124 from sagikazarmark/update-dependencies

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

@dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

Update frequency (including time of day and day of week)

Pull request limits (per update run and/or open at any time)

Out-of-range updates (receive only lockfile updates, if desired)

Security updates (receive only security updates, if desired)

A Microservice Toolkit from The New York Times

Gizmo Microservice Toolkit

Install

Import Path Change Notice

Packages

Examples

Owner

The New York Times

Comments

Swallowed kit test error

Makes gcp attributes accessible

implement new AddPermissionWithContext func within TestSNSAPI

Add MultiPublisher pubsub interface

Fix and refactor prometheus metrics setup

changing HTTPAccessLog to be a pointer

Extract GCP Monitoring

Breaking up the `config` and `pubsub` packages by dependency

PubSub GCP MaxExtension not changeable

Adding a new `server.Router` interface and `config.Server.RouterType` option with 2 implementations

SB-10000: Adding app- and pipeline-ids to server

Pull Request Checklist

Description

Bump github.com/aws/aws-sdk-go from 1.31.3 to 1.33.0

Release v1.33.0 (2020-07-01)

Service Client Updates

SDK Features

Release v1.32.13 (2020-06-30)

Service Client Updates

Release v1.32.12 (2020-06-29)

Service Client Updates

Release v1.32.11 (2020-06-26)

Service Client Updates

Bump github.com/aws/aws-sdk-go from 1.31.3 to 1.40.13

Release v1.40.13 (2021-08-02)

Service Client Updates

Release v1.40.12 (2021-07-30)

Service Client Updates

Release v1.40.11 (2021-07-29)

Service Client Updates

Release v1.40.10 (2021-07-28)

Service Client Updates

Release v1.40.9 (2021-07-27)

Service Client Updates

Release v1.40.13 (2021-08-02)

Service Client Updates

Release v1.40.12 (2021-07-30)

Service Client Updates

Release v1.40.11 (2021-07-29)

Service Client Updates

Release v1.40.10 (2021-07-28)

Service Client Updates

Release v1.40.9 (2021-07-27)

Service Client Updates

Bump cloud.google.com/go from 0.57.0 to 0.89.0

google-cloud-go v0.89.0

Features

Bug Fixes

google-cloud-go v0.88.0

⚠ BREAKING CHANGES

Features

Bug Fixes

google-cloud-go v0.87.0

Features

0.89.0 (2021-07-29)

Features

Bug Fixes

0.88.0 (2021-07-22)

⚠ BREAKING CHANGES

Features

Bug Fixes

0.87.0 (2021-07-13)

Features

Bump google.golang.org/api from 0.25.0 to 0.52.0

google-api-go-client v0.52.0

Features

google-api-go-client v0.51.0

Features

google-api-go-client v0.50.0

Features

Bug Fixes