Universal code search (self-hosted)

Last update: Jan 9, 2023

Comments: 17

Sourcegraph OSS edition is a fast, open-source, fully-featured code search and navigation engine. Enterprise editions are available.

Features

Fast global code search with a hybrid backend that combines a trigram index with in-memory streaming.
Code intelligence for many languages via the Language Server Index Format.
Enhances GitHub, GitLab, Phabricator, and other code hosts and code review tools via the Sourcegraph browser extension.
Integration with third-party developer tools via the Sourcegraph extension API.

Try it yourself

Try out the public instance on any open-source repository at sourcegraph.com.
Install the free and open-source browser extension.
Spin up your own instance with the quickstart installation guide.
File feature requests and bug reports in our issue tracker.
Visit about.sourcegraph.com for more information about product features.

Development

Prerequisites

Git
Go (1.15 or later)
Docker
PostgreSQL (v11 or higher)
Node.js (version 8 or 10)
Redis
Yarn

For a detailed guide to installing prerequisites, see these instructions.

Installation

Prebuilt Docker images are the fastest way to use Sourcegraph Enterprise. See the quickstart installation guide.

To use Sourcegraph OSS:

Initialize the PostgreSQL database
Ensure Docker is running
Configure the HTTPS reverse proxy
Start the development server
```
./dev/start.sh
```

Sourcegraph should now be running at https://sourcegraph.test:3443.

For detailed instructions and troubleshooting, see the local development documentation.

Documentation

The doc directory has additional documentation for developing and understanding Sourcegraph:

Project FAQ
Architecture: high-level architecture
Database setup: database best practices
General style guide
Go style guide
Documentation style guide
GraphQL API: useful tips when modifying the GraphQL API
Contributing

License

Sourcegraph OSS is available freely under the Apache 2 license. Sourcegraph OSS comprises all files in this repository except those in the enterprise/ and client/web/src/enterprise directories.

All files in the enterprise/ and client/web/src/enterprise/ directories are subject to the Sourcegraph Enterprise license.

Owner

Sourcegraph

Code search and navigation for teams (self-hosted, OSS)

https://github.com/sourcegraph/sourcegraph https://sourcegraph.com

Comments

Core Services: 3.13 Tracking Issue
Status

This tracking issue is approved.

Availability

Period is the January 21st until February 17th. Please put in the days you won't be working and the number of working days for the period.

@keegancsmith Not working: 4th & 5th February, Availability: 18d

@kzh Availability 20d

@mrnugget - Not working: 6th & 7th February (presumably), Availability: 18d

@rvantonder 19d

@ryanslade Availability 20d

@tsenart 5d (for coding)

@unknwon Not working: Jan 24th (Chinese New Year), Availability: 19d

Planned work

@keegancsmith: 8.00d

[x] e2e: log all e2e failures into central location for triage #8026 1d

[x] gitserver: Correctly report if repos are cloning after rebalance #7970 1d 🐛🕵️

[x] download button not shown for images #7924 0.5d

[x] gitserver: implement back-off on reclone operation #7804 1d 👩🐛

[x] Test and document src-expose on ec2 with docker-compose and startup #7803 1d 👩

[x] gitserver should auto-recover from corrupt repository clones #6676 1d 👩🐛🕵️

[x] repos: Stop reading repo.enabled #3971 0.5d 🧶

[x] Support internal CA or self-signed TLS certificates for all external communication #71 2d 👩

[ ] ~zoekt: support indexing multiple branches in zoekt-archive-index~ #7930 2d 👩🕵️

[ ] ~Ref glob search perf followups~ #7642 2d 🧶

[ ] ~Code host fingerprinter~ #7464 1d 🕵️

[ ] ~Exclude directories from search results~ #6870 2d 👩

[ ] ~Gitlab OAuth certificate signed by unknown authority~ #4652 0.5d 🐛👩

[ ] ~Support SSH key credentials directly in external service configs~ #3924 2d 👩

[ ] ~docs: Ensure new arch for repo-updater is reflected in dev docs~ #3911 0.5d 🧶

[ ] ~docs: Document using an alternate clone URL for repos~ #658 0.5d 🧶

@kzh: 6.50d

[x] Add BitBucket build status support in bitbucket-server-plugin #8386 ?d 👩

[x] a8n/core: Support updating changesets on BitbucketServer #7762 1d 🛠️

[x] a8n: Move "default branch" check to execution of CampaignJob #7725 2d 🛠️

[x] a8n/core: Extend CreateCampaignInput to accept Branch and persist it #7687 2d 🛠️

[x] Password-reset link not rendered for non-admin users #7520 0.5d 🐛

[x] Error fetching commit author info from GraphQL API #5335 1d 👩🐛

[ ] ~Simplify Bitbucket Server plugin interaction with Sourcegraph~ #7824 3d

[ ] ~Forbidden - CSRF token invalid~ #7368 2d 🐛

[ ] ~Address repohascommitafter search filter performance bottleneck~ #6745 0.5d

@mrnugget: 10.00d

[x] a8n: failed creating PRs because 'assignee' doesn't exist #8113 1d 🐛

[x] Creating a manual campaign -> unexpected draft mode #8009 ?d 👩

[x] a8n: Updating partially published Campaigns is not supported #7915 2d 🐛

[x] Automation should never miss search results #7861 3d 🐛🧶

[x] a8n: Finish implementation of src actions exec #7850 3d 🛠️

[x] Send pings about Automation usage to Sourcegraph #7711 1d

[ ] ~Write C# LSIF indexer~ #7760 ?d 🛠️

[x] ~a8n Spike: Add ability to "stack diffs" (RFC 92)~ #7721 3d 🛠️

@rvantonder: 6.00d

[x] Trailing backslash at end of expression: `` when searching for valid regexp #7604 1d 🐛👩

[x] Enforce warnings for unsupported parameters and filters in structural search #7293 1d 🧶

[x] Backend returns invalid filter chip with space "lang:ignore list" #6498 2d 👩🐛

[x] Impossible to search for colons in literal search #6490 2d 🐛

[x] ~search: design proposals for AND / OR / NOT expressions~ #8346 4d

[ ] ~Extract and dump type information for Dart LSIF~ #7908 4d 🛠️

[ ] ~search: Implement AND / OR / NOT for file contents~ #7823 ?d

[ ] ~search filter archived/fork "yes/no/only" values are confusing~ #7003 4d 👩🛠️

@ryanslade: 8.00d

[x] a8n: exposed token in the error message if create campaign failed #8105 ?d 🐛

[x] sourcegraph/security-issues #60 ?d 🐛🔒

[x] Redact GitLab access token from error log message #8078 ?d 🐛

[x] Add setting to enable read-only mode for automation for all Sourcegraph users #8013 ?d 👩

[x] a8n/auth: Regex search/replace doesn't work in private repos #7914 1d 🐛

[x] Remove published_at fields from Campaign and ChangesetJob #7782 2d 🧶

[x] a8n/core: GraphQL API for an extended campaign list view #7553 2d 🛠️

[x] a8n/core: Expose labels of all changesets in campaign in GraphQL API #7549 1d

[x] a8n/core: Expose CI status on changesets #7093 2d 🛠️

[ ] ~a8n/core: Extend GraphQL API to include participant users involved in the campaign~ #7552 1d 🛠️

[ ] ~a8n/core: Expose comments of all changesets in campaign in GraphQL API~ #7548 2d 🛠️

[ ] ~Diagnose and fix slow /.api/registry/extensions endpoint~ #7544 1d 🧶

[ ] ~a8n: Heuristic syncing of Changesets and ChangesetEvents~ #6388 3d 🧶

@tsenart: 3.50d

[x] Use basic auth instead of query params for auth in github-proxy #8310 0.5d 🧶

[ ] Core Services: 3.13 Tracking Issue #7719 ?d

[x] Prevent upgrading more than one minor version at a time #7702 1d 🐛

[x] Redis AOF file grows unbounded due to frequent container restarts #3300 2d

[x] redis: Log spam warning when redis is loading the dataset in memory #2904 ?d 🐛👩

[ ] ~Reject invalid externalURL with non-/ path~ #7884 0.5d 🐛

@unknwon: 16.00d

[x] External services documentation needs to be updated #8288 0.5d

[x] Make read:org permissions optional / dependent upon whether allowOrgs is specified #8163 1d 👩

[x] sourcegraph/security-issues #61 0.5d 👩🔒

[x] Bitbucket Cloud external service doesn't respect custom URLs #7990 1d 👩

[x] authz code logs warnings about Redis returning nil values #7912 0.5d 🐛

[x] RFC 40: Move call to the Authz methods into application layer #7878 1d 🧶

[x] sourcegraph/security-issues #55 1d 🔒

[x] RFC 40: Add unit tests for GraphQL APIs #7748 1d 🧶

[x] sourcegraph/security-issues #54 1d 🔒

[x] Enable HTTP Strict Transport Security (HSTS) on sourcegraph.com (and all subdomains) #7660 0.5d 🔒

[x] It is not clear how to configure Sourcegraph to just access public repositories #7587 0.5d 🐛

[x] Enforce minimum password requirements #7521 0.5d 🔒

[x] RFC 40: Handle user deletion #7302 1d

[x] Match highlight at the end of a long result snippet freezes browser tab #6489 1d 🐛

[x] Querying repository by name with github.com prefix always returns empty string for name #5125 1.5d 🐛

[x] unused site setting git.cloneURLToRepositoryName #3768 0.5d 🧶

[x] postgresql: (Auto)tune DB pool sizes based on max_connections #3473 3d 👩

[ ] ~Make new pricing tiers reflected in feature access~ #7927 1d 🐛

[x] ~RFC 40: Add Prometheus metrics~ #7827 1d 🛠️

[x] ~sourcegraph/security-issues~ #53 ?d 🔒

[ ] ~Auth: explicit session invalidation~ #1126 2d 🔒

Legend

👩 Customer issue

🐛 Bug

🧶 Technical debt

🛠️ Roadmap

🕵️ Spike

🔒 Security issue
Keep repository set in sync with config
User story

As a site admin, I configure Sourcegraph with a personal access token. I want to share Sourcegraph with my team, but I don't want Sourcegraph to be configured with all of the repositories that my token has access to. Either I update the token, or I limit the scope of synced repositories (i.e. via a search query or regex) and the repositories that are no longer accessible immediately disappear from my Sourcegraph instance.

Product changes

Mirrored repositories will be always kept in sync with associated external service configuration.

Mirrored repositories will be configured exclusively via associated external service configuration.

Enabled and disabled state will be deprecated and removed from the UI.

Explicit deletion of individual repositories will be removed from the UI.

Planned work for 3.3

[x] Deployment

[x] Instrument Syncer with Prometheus metrics (#3398)

[x] Instrument server endpoints with Prometheus metrics (#3398)

[x] Instrument new code with tracing. (#3398)

[x] Documentation

[x] Update GitHub external service documentation.

[x] Document new repositoryQuery default. (#2895)

[x] Document new exclude setting.

[x] Update GitLab external service documentation (#3042)

[x] Update Bitbucket Server external service documentation.

[x] Document new repositoryQuery setting.

[x] Document new repos setting.

[x] Document new exclude setting.

[x] Document new required username field.

[x] Write blog post about how the new syncer works.

[x] User Interface

[x] Implement new settings page in development by @francisschmaltz.

[x] Change repos listing page to support repos with and without enabled state and delete buttons.

Sourcegraph Engineer / Design Spec Link: https://zpl.io/anpXvGr

Public / Non-Engineer Design Link: https://sketch.cloud/s/wwoGV/Axae1bR/play serving requests.

[x] Common

[x] Ensure enabled state migration is safe to run many times (#3046)

[x] Retry migrations instead of exiting. Verify that Sourcegraph is usable without repo-updater (#3046)

[x] Trigger full sync when an external service is deleted. (#3290)

[x] Ensure that database is migrated once the frontend is ready. Right now waiting for the frontend doesn't ensure that migrations were ran. We either change the wait for frontend method to wait for migrations too, or make repo-updater also run migrations. (#3286)

[x] GraphQL API's setRepositoryEnabled deprecation path (#3162, #3252)

[x] Handle repo renames in the scheduler (#2609)

[x] Default feature flag to on.

[x] Use centralized service discovery mechanism to retrieve Postgres connection data. (cc: @slimsag)

[x] GitHub

[x] Implement GitHubSource and enable new syncer for GitHub external services (#2266)

[x] Implement github.exclude to support enabled state deprecation. (#2649)

[x] Change default github.repositoryQuery to ["none"] and migrate existing configs (#2785, #2792)

[x] Implement enabled state deprecation migration of GitHub external services (#2887)

[x] Migrate all disabled repos to github.exclude list.

[x] Migrate all enabled repos that aren't yielded by any GitHub source to github.repos list.

[x] Deprecate github.initialRepositoryEnablement setting.

[x] Show error to admin for repositoryQuery with more than 1000 results (#2562)

[x] repositoryQuery may return incomplete results (#3363)

[x] Gracefully handle non-existent repos in repos list. (#3027)

[x] Update UI to remove enabled state for GitHub repos.

[x] Update UI to remove explicit deletion for GitHub repos.

[x] GitLab

[x] Implement GitLabSource and enable new syncer for GitLab external services (#3008)

[x] Implement gitlab.projects to support enabled state deprecation. (#3027)

Gracefully handle non-existent projects in projects list.

[x] Implement gitlab.exclude to support enabled state deprecation. (#3014)

[x] Change default gitlab.projectQuery to ["none"] and migrate existing configs. (#3028)

[x] Extend enabled state deprecation migration with GitLab external services. (#3029)

[x] Update UI to remove enabled state for GitLab repos.

[x] Update UI to remove explicit deletion for GitLab repos.

Stretch goals for 3.3

[ ] Implement dry-run mode in Syncer.Sync to support #2866

[x] Bitbucket Server

[x] Implement BitbucketServerSource and enable new syncer for BitbucketServer external services (#3097)

[x] Implement bitbucketserver.repos to support enabled state deprecation. (#3097)

Gracefully handle non-existent repos in repos list.

[x] Implement bitbucketserver.exclude to support enabled state deprecation. (#3097)

[x] Implement bitbucketserver.repositoryQuery, set the default to ["none"] and migrate existing configs to have the previous semantically equivalent default of ["?visibility=private", "?visibility=public"] (#3097)

[x] Migrate existing Bitbucket Server repos external_id's from projectName/repoName to repoID to support renames. (#3097)

[x] Implement enabled state deprecation migration of BitbucketServer external services (#3097)

[x] Migrate all disabled repos to bitbucketserver.exclude list.

[x] Migrate all enabled repos that aren't yielded by any BitbucketServer source to bitbucketserver.repos list.

[x] Deprecate bitbucketserver.initialRepositoryEnablement setting.

[x] Update UI to remove enabled state for BitbucketServer repos.

[x] Update UI to remove explicit deletion for BitbucketServer repos.

Backlog

[ ] Create Grafana dashboard for repo-updater

[ ] Ensure that sourcegraph.com code paths work with the feature flag on. How does manually adding a repo when directly visiting one interact with the new Syncer?

[ ] Search and replace everywhere: s/Github/GitHub/g

[ ] Test: Server HTTP endpoints

[ ] Test: Update from diff in scheduler

[ ] GitHub supports query configuration to replace repositoryQuery #2562

Customers affected

Too many to name individually, but a highlight:

https://app.hubspot.com/contacts/2762526/company/732945850 said 90% of their site admin's time was managing repositories
Discussion: Reduce Renovate PR spam

At this moment we have 40 open PRs and 20 of them are from Renovate, mostly Go dependencies. This seems exacerbated by the fact that we have dependencies on pinned on commit hashes instead of semantic versions with release notes (I am not sure if semver tags are available for various dependencies). These PRs have been open for a month, which indicates to me that the value of these PRs is very low (otherwise we would have merged them). I view these PRs as noise in our open PR list that should be eliminated.

I would be in favor of completely removing the dependency updating for Go. Generally, I don't see a reason to recreationally update dependencies unless the updates contain features or bug fixes that we explicitly need.

Combining all Go dependencies into a single PR would also solve the PR spam problem while still giving us information about what dependencies are out of date. I would still have the same reservations about regularly merging said PR as expressed above.

I would be interested to hear everyone's opinion. cc @slimsag @beyang @tsenart @keegancsmith
Cloud: 3.18 Tracking issue
Status

This tracking issue is being worked on.

Context

This iteration is the first we're working officially as the cloud team on items from the RFC 151 roadmap, which we have no OKRs for yet.

However, there's still work outside of this roadmap that needs to be completed, and as such has been prioritised in this iteration, namely:

@unknwon is working on RFC 167: Enforce product tiers, starting with Spike: Enforcing product tiers #11577 and ensuing work is addressing the OKR Product: Deliver on projects to increase and support sales efforts. KR: License keys enable features for each license tier.

@unknwon is wrapping up work from RFC 137: Better automated testing. There's no specific OKR for this effort.

@keegancsmith is wrapping up work from RFC 136: Navigating Version Contexts with Index multiple (non-master) branches #6728 which directly addresses Product: Deliver on projects to increase and support sales efforts. KR: Users can search across contexts of their organization’s code as outlined in RFC 136.

@keegancsmith will be working streaming search support in the context of Sourcegraph Cloud roadmap afterwards — no OKR for this yet.

@ryanslade main focus will be on scaling repo-updater as part of the Sourcegraph Cloud roadmap.

@daxmc99 main focus will be on RFC 174: HA Postgres for Sourcegraph Cloud #11496

@asdine main focus will be on Store git commits in Postgres as part of the Sourcegraph Cloud roadmap.

Availability

Period is from 20th of June to 20th of July. Please write the days you won't be working and the number of working days for the period.

@keegancsmith: TBD

@tsenart: 21d

@unknwon: 19d (National holidays June 25-26)

@ryanslade: 21d

@daxmc99: 21d

@asdine: 20d

Chayim Kirshen (Security Engineer starting June 29th): TBD

Workload

@asdine: 5.00d

[x] repository status page with "cloning" filter is extremely slow #11029 4d

[x] cloud: Repo-updater store repositories clone status in repo table #11867 :shipit:

[ ] mail package deprecated #10703 🧶

[ ] Ability to disable SMTP TLS verification #10702 1d 👩

[ ] Split resolveRepositories code #10489 :shipit:🧶

@beyang

[ ] Sourcegraph.com behaves like customer instances for a subset of repositories #10821

@chayim

[ ] Chayim Kirshen On-boarding #11785

@daxmc99: 14.00d

[ ] RFC 174: HA Postgres for Sourcegraph Cloud #11496 10d

[ ] Add operational playbook for updating external database versions #1099

[x] add terraform configuration for sourcegraph.com's infrastructure #10455 2d

[x] Terraform migration: Migrate all non-postgres stateful data #11759 1d

[x] Terraform migration: Update docs #1115

[ ] RFC 174: Terraform Cloud SQL #11794 1d

[ ] Sourcegraph.com - add redis-store & precise-code-intel-bundle-manager snapshotting #10450

@keegancsmith: 10.00d

[x] Index multiple (non-master) branches #6728 10d 👩

@ryanslade: 4.00d

[ ] Scale repo-updater WIP #11493 🛠️

[x] RFC 132: Use rate limit data when scheduling changeset syncs #9105 2d

[x] src actions exec can cause frontend to be OOMKilled #10540 2d 🐛

[x] Diff query for unknown revision is slow #11654

@tsenart

[ ] ~Add 'activity histogram' to admin usage stats + rearrange~ #11063 2d

[x] engineering: Welcome to the Cloud team, @asdine! #1106 :shipit:

[x] team: Welcome to the Cloud, Dax #1057 :shipit:

@unknwon: 13.50d

[ ] Simplify PermsStore logic with intarray extension #11768 2d

[ ] Use Postgres intarray to store permissions object_ids #11400 3d 🧶

[x] gqltest: add dev docs #11578 0.5d

[x] RFC 142: Migrate backend e2e test with integration tests #10068 5d 🛠️

[x] Dotcom site-admin overview page breaks with anonymous user ID #11801 0.5d 🐛

[x] Codecov: use go-acc for better code coverage on master #11781 1d

[x] authz: ReposIDsWithOldestPerms breaks when repos are hard-deleted from repo table #11769 0.5d 🐛

[x] Add integration tests that saves rows to user_external_accounts then grant permissions #11614 1d 🧶

[ ] ~Enforcing product tiers~ #11577 5d 🕵️

[ ] ~e2e: remove tests covered by dev/gqltest~ #11762 1d

[ ] ~gqltest: re-enable in CI on every branch~ #11579 2d

[ ] ~Spike: Pure SQL authz flow~ #11767 3d 🕵️

Legend

👩 Customer issue

🐛 Bug

🧶 Technical debt

🛠️ Roadmap

🕵️ Spike

🔒 Security issue

:shipit: Pull Request
Core Services: 3.16 Tracking Issue
Status

This tracking issue is ready.

Availability

Period is the April 21st until May 18th. Please put in the days you won't be working and the number of working days for the period.

@keegancsmith: 18d (27 April and 1 May are holidays) minus ~multiple half-days due to childcare

@mrnugget: 19d (1 May is national holiday) minus ~multiple half-days due to childcare

@rvantonder: 12d (Not working 22 April through 1 May)

@ryanslade: 18d (27 April and 1 May are holidays)

@tsenart: 8d

@unknwon: 17d (May 1st to May 5th are national holidays)

@asdine: 19d

Work

@Unassigned

[ ] ~Add docs for connecting to Azure DevOps as a code host~ #8647 1d

[ ] ~src-expose: document usage with docker compose and kubernetes~ #9884 0.5d

@asdine: 2.00d

[x] https://sourcegraph.com/github.com throws a 500 #9274 🐛

[ ] ~Track and/or query usage in pings~ #9177 5d

[x] indexed-search: Performance improvements for concatenated regex queries #8800 👩

[x] Reject invalid externalURL with non-/ path #7884 2d 🐛

[x] Large files not being returned for indexed filepath searches #10382 🐛👩

[x] Improve performance for concatened queries #43 :shipit:

@keegancsmith

[x] Search OKRs Q2 #10025

[x] proposal: indexed search uses shallow git clones #9888

[ ] ~src-expose: document usage with docker compose and kubernetes~ #9884 0.5d

[x] Proposal: remove discussions code #9649 🧶

[ ] ~Add ability for src-expose to periodically run a command (without then running git commit)~ #9643

[x] GraphQL API Search result line numbers #9587 👩

[ ] ~Add docs for connecting to Azure DevOps as a code host~ #8647 1d

[ ] ~Index multiple (non-master) branches~ #6728 10d 👩

[x] dev/watchman: merge file change events while running handle changed #10119 :shipit:

[x] debugproxies: Nicer display names for statefulset pods #9950 :shipit:

@mrnugget: 10.50d

[x] Improve Campaigns documentation #9913 3d 🧶

[x] Run Campaigns user tests with Sourcegraph colleagues #9912 4d 🛠️

[x] Use the same code for burndown chart and derived Changeset State #9101 1.5d 🧶

[x] Expose combined diffStat on Campaign(Plan) #7092 2d

@nicksnyder: 0.50d

[x] sourcegraph/security-issues #63 0.5d

@rvantonder: 2.50d

[x] ~Implement repo or-expressions for multi repo/multi rev~ #10114 4d 👩

[x] ~spike: investigate supporting restricted or-expressions for repos~ #10113 1d 👩🕵️

[x] ~Write RFC for syntax that supports multiple repos and revisions to support RFC 136~ #10111 1.5d 👩

[x] Fix validation GitLab exclude repos #10096 🐛👩

[x] Add search option where mixed-case patterns imply case-sensitivity #10057

[x] Resolve field aliases in and/or queries #10054 0.5d 🐛

[x] search UI should be clear when forks or archived repositories were excluded #9928 2d 🐛

@ryanslade: 8.50d

[x] Merged GitHub changeset shown as "closed" until next sync when webhooks are activated #10071 1d 🐛

[x] Bitbucket Server webhooks miss needs work => unapproved #9139 2d 🐛

[x] Querying for cloning repositories is very slow #8725 1d 🐛

[x] Ensure that we can match webhook requests to exactly one external service #9008 1d 🐛

[x] Include "Campaign credits" in pings #10073 0.5d

[x] ~Validate unique rate limit configuration~ #10026 0.5d

[ ] ~Use internal rate limiters for all source requests~ #9953 2d

[ ] ~RFC 132: Use rate limit data when scheduling changeset syncs~ #9105 1d

[x] Add tests for the webhook handler that persists data #9104 2d 🧶

[x] Automatically remove Bitbucket Server webhooks if the plugin.webhook property has been removed #8914 1d 🛠️

[ ] ~Reduce usage of raw string for external service kind~ #10296 1d 🧶

@tsenart: 4.00d

[x] RFC 145: Event-level data from customers export button #10434 4d

[x] RFC 145: Usage stats download button #10475 :shipit:

[ ] ~Add ThirdPartyLicensesDistributedTools.csv~ #9975

[ ] ~Automate creation of the Go third-party licenses CSV~ #9973 2d

[ ] ~Use comma delimiters in ThirdPartyLicensesGo.csv~ #9972 0.5d

[x] registry: Instrument API with Prometheus metrics #10628 :shipit:

[x] tracking-issue: Update GitHub action #10158 :shipit:

[x] tracking-issue: Implement simplified workflow #10157 :shipit:

[x] Tracking issues page #786 :shipit:

@unknwon: 9.00d

[x] github: support only list private affiliated repositories #10184 0.5d

[ ] ~RFC 142: Migrate backend e2e test with unit tests~ #10067 4d 🛠️

[ ] Get RFC 142 approved based on info from auditing #10066 3d

[x] ~authz: better heuristic of NextSyncAt by PermsSyncer~ #9098 1d

[x] Admin button for refreshing permissions for a user / repo #8990 1d

[ ] ~Use golden accounts for VCRs in tests~ #8976 2d 🧶

[ ] ~Use serializable transactions for updating permissions~ #7877 2d 🕵️

[x] Display last updated time of permissions in repo and user settings page #8989 1.5d

[x] Add backend tests for blob redirects #10260 0.5d

[x] sourcegraph/security-issues #63 0.5d

[x] Expand CODEWONERS line *git* to be more precise #10047 0.5d

[x] Audit existing backend e2e tests #9924 1d

[x] GraphQL APIs are blocking forever in OSS version #3847 0.5d

[ ] ~Identify code owners for files currently owned by nobody~ #10064 0.5d

[ ] ~Add Grafana alert to authzFilter panel~ #9923 1d

[ ] ~Add ability for src-expose to periodically run a command (without then running git commit)~ #9643

[ ] Turn on background permissions syncing by default #10657 0.5d

Legend

👩 Customer issue

🐛 Bug

🧶 Technical debt

🛠️ Roadmap

🕵️ Spike

🔒 Security issue

:shipit: Pull Request
Distribution: 3.19 Tracking issue
Plan

Support new and existing deployments

This is an ongoing expense, we anticipate this taking no more than 10d of work spread across the entire team.

Reduce upgrade overhead

Upgrading Kubernetes deployments requires customers spend a lot of engineering time to converge our released Kubernetes manifests with their fork as documented in RFC-141.

We will finish the Dhall investigation and make a decision by the end of 3.19.

@ggilmore

@uwedeportivo

Increase our e2e test frequency

To increase our release cadence, we need to be able to run e2e tests more frequently. This is currently not possible as our CI infrastructure causes tests to be unreliable.

@slimsag

@davejrt

@pecigonzalo

Support per-team alerts

To allow teams to support and monitor the services and features they ship, we need to be able to route alerts to the relevant teams as described in RFC-189.

@bobheadxi

@davejrt

@slimsag

Availability

Period is from July 20th to August 19th (23 working days). Please write the days you won't be working and the number of working days for the period.

Dave: 22d (off July20)

Robert: 23d

Workload

@bobheadxi: 2.50d

[x] Approved: Proposal: RFC-189: Support per-team alerts and on-call rotations #12010

[x] monitoring: migrate existing alert rules to generator #12117

[x] Dogfood the monitoring we ship with Sourcegraph #5370 2d

[x] ~cadvisor: investigate collecting IO metrics~ #12163

[x] Better account for peaks / max resource usage in monitoring #12032 0.5d

[x] Make license check not fail when our node version is upgraded #12318

[x] monitoring: generate alerts as native prometheus alerts, let alert_count depend on alert rules #12336

[x] sourcegraph/server:3.18 provisioning indicators panel missing #12421 🐛

[x] deploy-sourcegraph: prometheus missing in up metric #12482

[x] monitoring: no easy way to trigger alerts for testing #12423

[x] monitoring: link back to grafana service dashboards in alert notifications #12235

[x] monitoring: better advice for alerts that do not have an entry in alert_solutions #12236

[x] frontend: hard errors alerts as ratio/percentage instead of absolute-value #12158

[x] monitoring: change relevant hard threshold alerts to ratio-based alerts #12865

[x] monitoring: index_queue_growth_rate firing without hitting threshold #12868 🐛

[ ] ~monitoring: remove custom alertmanager from cloud~ #12160

[x] monitoring: difficult to understand 7d provisioning panel #12692

@davejrt

[ ] ~Bare-metal Buildkite agents capable of running Docker and VMs~ #12101

[ ] ~deploy-sourcegraph-dhall: Add CI job to generate diff between master and feature branch~ #12716

[ ] ~baremetal buildkite agent networking / instability issues~ #12996

@daxmc99

[x] Sourcegraph.com - add redis-store & precise-code-intel-bundle-manager snapshotting #10450

@efritz

[x] sourcegraph/customer #72 🐛👩

@ggilmore

[ ] ~deploy-sourcegraph-dhall: incrementally migrate sourcegraph.com services over to dhall~ #12113

[ ] ~[draft] deploy-sourcegraph-dhall: implement migrate-to-non-root overlay~ #12283

[ ] ~deploy-sourcegraph-dhall: grafana: implement new configuration logic for k8s.sgdev.org~ #12109

[ ] ~deploy-sourcegraph-dhall: frontend: implement new configuration logic for k8s.sgdev.org~ #12105

@keegancsmith

[x] sourcegraph/customer #69 👩

@pecigonzalo

[ ] ~Reduce the impact of unplanned work~ #11904

[x] Migrate terraform state to GCP #12496

@slimsag: 7.50d

[x] sourcegraph/customer #62 2d 👩

[x] Formalize managed instances #12495 5d

[x] sourcegraph/customer #73 👩

[x] sourcegraph/customer #53 0.5d 👩

[x] sourcegraph/customer #66 👩

[ ] ~sourcegraph/customer~ #74 👩

[x] sourcegraph/customer #85 👩

[ ] ~sourcegraph/customer~ #90 🐛👩

[ ] ~Run e2e tests on bare-metal Buildkite agents on every commit to master (non-blocking)~ #12339

[ ] ~Run e2e "regression" tests on bare-metal Buildkite agents on every commit to master (non-blocking)~ #12340

[ ] distribution: add monitoring architecture page #1221 :shipit:

[ ] distribution roadmap #1104 :shipit:

[ ] Document when to introduce new services or not #5487 :shipit:

@uwedeportivo: 8.00d

[x] sourcegraph/customer #65 👩

[x] deploy-sourcegraph-dhall: symbols: implement generate reading from config #12076 0.5d

[x] deploy-sourcegraph-dhall: searcher: implement generate reading from config #12075 0.5d

[x] deploy-sourcegraph-dhall: replacer: implement generate reading from config #12074 0.5d

[x] deploy-sourcegraph-dhall: repo-updater: implement generate reading from config #12073 0.5d

[x] deploy-sourcegraph-dhall: query-runner: implement generate reading from config #12072 0.5d

[x] deploy-sourcegraph-dhall: precise-code-intel: implement generate reading from config #12071 0.5d

[x] deploy-sourcegraph-dhall: postgres: implement generate reading from config #12070 0.5d

[x] deploy-sourcegraph-dhall: jaeger: implement generate reading from config #12069 0.5d

[x] deploy-sourcegraph-dhall: github-proxy: implement generate reading from config #12066 0.5d

[x] deploy-sourcegraph-dhall: cadvisor: implement generate reading from config #12065 0.5d

[x] ci-db-backcomp.sh is not branch-aware #12302

[x] sourcegraph/customer #57 3d 👩

Legend

👩 Customer issue

🐛 Bug

🧶 Technical debt

🛠️ Roadmap

🕵️ Spike

🔒 Security issue

:shipit: Pull Request
cookies will require attribute value for SameSite attribute soon

A customer running Sourcegraph under a different domain got this warning.

My guess the Sourcegraph instance references sourcegraph.com which sets cookies.

Modern browsers will soon require this. Chrome probably very soon. Our cookies coming from sourcegraph.com (docs, extensions ?) need to have SameSite="None" and "Secure".
Sourcegraph stopped cloning and indexing repos
Question description

We had 15-16k repos and added additional 24k small repos to sourcegraph. We have two clusters, and it seems that both stopped cloning repos. There are only few repositories cloned in past 24 hours.

> (sg-aws-us-east-1) > kubectl logs --since 24h -l app=gitserver -c gitserver | grep 'cloning repo' | wc -l 30 > (sg-aws-us-east-1) >kubectl logs --since 24h -l app=gitserver -c gitserver | grep 'cloning repo' | wc -l 47

I'm able to clone manually from gitserver pod. Could cloning process be impeded by something else? Maybe indexing is not catching up? Something else?

Additional context

We're running v3.9.4-1.
Tracking Issue: Flaky CI builds

In the past few weeks we've seen a lot of CI failures that were fixed with simple retries. In other words: they're flaky.

Flaky tests & builds lead to a lot of noise, which leads to us to losing confidence in our tests & builds, which leads to us ignoring them. Let's fix them.

Plan

As a first step, we stop those flaky builds from disappearing in Buildkite and Slack and use this issue to catalog flaky CI failures (those that were fixed by retrying them).

That makes them not only more visible, but also easier to classify (do e2e tests fail? linters? infrastructure?) and thus hopefully easier to fix.

Then we try to fix them and check them off in this list.

Flaky builds

| # | Failing CI step | Reason for failure | Fix | Builds | | - | --------------- | ------------------ | ----- | ------------- | | 0 | ~e2e tests~ | ~Login failed~ | PR #5711 | Link | | 1 | ~golangci-lint~ | ~A warning? Or agent lost?~ buildkite-agents OOM | Most likely: https://github.com/sourcegraph/infrastructure/pull/1652 | 43568, 44421, 43333, 42891, 43639 | | 2 | ~golangci-lint~ | ~"Error return value not checked" - did not pop up in PR build~ | It didn't pop up in PR build, because on master golangci-lint was passed the correct rev. Not a flake. | 43420 | 3 | go generate | build output "/buildkite/builds/buildkite-agent-5898f757cb-hbk2l-1/sourcegraph/.golang/sourcegraph/bin/server" already exists and is not an object file | Could be fixed by fix for 1, since OOM error prevented build cleanup from happening | 43411 | | 4 | ./dev/check-all.sh | ./todo-security.sh: No such file or directory | - | 43198, 43366, 44936 | | 5 | e2e tests | themes did not load (reported in logs that repos are not found) | - | 43747 | 6 | e2e tests | Saved searches › Save search from saved searches page | - | 43824 | 7 | e2e tests | Repository component › symbol sidebar › displays valid symbols at different file depths for Go (./examples/cmd/webapp-opentracing/main.go.go) | - | 43827 | 8 | ~e2e tests~ | ~http://localhost:7080 was not accessible within 60s~ | Fixed https://github.com/sourcegraph/sourcegraph/pull/5960 | 43896, 44308, 44309, 44603, 44859-1, 44859-2 | | 9 | e2e tests | http://localhost:7080 was not accessible within 60s. | Seems like it wasn't fixed by 8 | 45931, 46347 | 10 | e2e tests | e2e test suite › Visual tests › Search results file | | 46136, 46532 | 11 | e2e tests | e2e test suite › Repository component › hovers › Blob › jump to definition › noops when on the definition | | 46055 | 12 | e2e tests | e2e test suite › Repository component › rev resolution › updates rev with switcher | | 45917 failed. | 13 | e2e tests | e2e test suite › Repository component › hovers › Blob › gets displayed and updates URL when clicking on a token and e2e test suite › Repository component › hovers › Blob › gets displayed when navigating to a URL with a token position | | 46176 | 14 | symbols test | FAIL: TestSearch/0 > search_test.go:205: {foo false false false [] false false 0 true false} failed: non-200 response: code=503 body=context deadline exceeded | | 46932 (failed), 46944 (passed - same commit) | 15 | Start Docker image for e2e tests | Error response from daemon: reference does not exist open /var/lib/docker/tmp/docker-import-163672497/repositories: no such file or directory | | 16 | Flaky LSIF tests | BeforeAll failed, references only returned if dumps visible at tip failed | | 50253 | | 17 | Flaky eslint test | dev/ci/yarn-run.sh build-ts all:eslint fails with module not found | | 50468 | 18 | e2e tests |e2e test suite › Campaigns › Create campaign preview for credentials campaign type | | #1485 | 19 | e2e tests |e2e test suite › External services › External service add, edit, delete | | #51680

If you run into a flaky build: please add it to this list!
Add a request for feedback in the current IntelliJ extension

This would be a great way to recruit users for feedback for validating the approach of the improved extension. It'd be great if this was semi-automated, as an example a a popup / toast would appear after interacting with the extension for random users etc.
Reduce number of queries for pings
Context at https://github.com/sourcegraph/sourcegraph/issues/10598

Important context from that issue:

The major offender is we do daily, weekly and monthly for each type of events we care, e.g. code intel would at least call eventLogs.countPerPeriodBySQL 27 times:

https://github.com/sourcegraph/sourcegraph/blob/c3f41185f29f4f2d48007acce73d5795dfd52a5f/cmd/frontend/internal/usagestats/codeintel.go#L85-L95

@efritz and @attfarhan quick questions about usage data:

How are we currently actually using this data? And I don't mean "how will we use it in the future", I'm wondering if and how we are actually using it currently.

When we do use it more in the future, what will it be used for? Are the arguments made in the original RFCs still valid?

Proposals:

fetch only ONE daily result for all usage queries (currently many of them fetch both today's and yesterday's data to avoid missing users at the end of the day)

kill SWAUs data entirely — e.g. users by software development lifecycle stage (CC @christinaforney do we have any plans to use this? I know I'm not today)

fetch only weekly? Or monthly? data for most codeintel and search metrics (event counts, users, and latencies). Do we actually care about daily data here? The questions above from @efritz and @attfarhan will tell us if we mostly care about weekly or monthly.

CC @ebrodymoore
faster function-incomplete local dev builds without cgo

The squirrel package requires cgo, which significantly increases the build (linking) time of the frontend build (by about 2x). During local development, omitting the local code intel functionality that requires cgo speeds up the build significantly. For production builds, cgo is still needed.

To use this, just build with CGO_ENABLED=0. Before this commit, that produces a compile error. After this commit, that compiles, but search-based code intel will show an error in the hovers. As such, and because we generally want dev to not diverge from prod builds, this will remain an undocumented dev setting until further notice.

Test plan

The squirrel test suite will catch any issues.
Code Insights: Fix code insights creation UI pages headers
Prior to this PR all code insights creation UI pages had chaotic headers text with no structure, in this PR we add link breadcrumbs pattern for the insight creation UI pages

Test plan

Visit all code insight creation UI pages (track changes, detect and track, language insight)

Check that all links in the page header work properly

App preview:

Web

Check out the client app preview documentation to learn more.
Code Insights: Adjust insight routes
Fixes https://github.com/sourcegraph/sourcegraph/issues/45225

Test plan

Open code insight dashboard page (custom and all insight tabs)

Open independent code insight page (click on the code insight card title)

Open all code insight creation pages

Make sure that all pages above are reachable

App preview:

Web

Check out the client app preview documentation to learn more.
WIP: Docs update for new Kustomize setup
WIP: Docs update for the new Kustomize

This is a docs update for the new Kustomize deployment method that will be introduced in https://github.com/sourcegraph/deploy-sourcegraph/pull/4215

[x] Introduced the new Kustomize setup and

[x] Steps to deploy using the new Kustomize overlays

[x] Instruction on configuring deployment using pre-configured components

[x] Instruction on making custom changes with configurable components

[x] Update current Kubernetes installation guide with the new deployment steps

TODO:

[ ] Instructions to migrate from current Kustomize setup to the new one

[x] Instructions on comparing resources output between two overlays

[x] Instructions on comparing the generated manifests between the old and new version

[ ] Add examples on how to create custom overlays for different environments

[ ] List services that require RBAC (cadvisor etc)

Test plan

Run yarn docsite:serve to stand up docs site

Clone the deploy-sourcegraph repo and cd into latest bee/newBase branch to see the lists of resources used in this docs update

Visit http://localhost:5080/admin/deploy/kubernetes/kustomize/configure in your browser and follow the instruction to build a Kustomize Overlay with custom configurations

Visit http://localhost:5080/admin/deploy/kubernetes/kustomize and follow the instructions to deploy Sourcegraph using an Overlay
build(deps): bump tokio from 1.17.0 to 1.24.1 in /docker-images/syntax-highlighter
Bumps tokio from 1.17.0 to 1.24.1.

Release notes

Sourced from tokio's releases.

Tokio v1.24.1

This release fixes a compilation failure on targets without AtomicU64 when using rustc older than 1.63. (#5356)

#5356: tokio-rs/tokio#5356

Tokio v1.24.0

The highlight of this release is the reduction of lock contention for all I/O operations (#5300). We have received reports of up to a 20% improvement in CPU utilization and increased throughput for real-world I/O heavy applications.

Fixed

rt: improve native AtomicU64 support detection (#5284)

Added

rt: add configuration option for max number of I/O events polled from the OS per tick (#5186)

rt: add an environment variable for configuring the default number of worker threads per runtime instance (#4250)

Changed

sync: reduce MPSC channel stack usage (#5294)

io: reduce lock contention in I/O operations (#5300)

fs: speed up read_dir() by chunking operations (#5309)

rt: use internal ThreadId implementation (#5329)

test: don't auto-advance time when a spawn_blocking task is running (#5115)

#5186: tokio-rs/tokio#5186 #5294: tokio-rs/tokio#5294 #5284: tokio-rs/tokio#5284 #4250: tokio-rs/tokio#4250 #5300: tokio-rs/tokio#5300 #5329: tokio-rs/tokio#5329 #5115: tokio-rs/tokio#5115 #5309: tokio-rs/tokio#5309

Tokio v1.23.1

This release forward ports changes from 1.18.4.

Fixed

net: fix Windows named pipe server builder to maintain option when toggling pipe mode (#5336).

#5336: tokio-rs/tokio#5336

Tokio v1.23.0

Fixed

net: fix Windows named pipe connect (#5208)

io: support vectored writes for ChildStdin (#5216)

io: fix async fn ready() false positive for OS-specific events (#5231)

... (truncated)

Commits

31c7e82 chore: prepare Tokio v1.24.1 (#5357)

8d8db27 tokio: add load and compare_exchange_weak to loom StaticAtomicU64 (#5356)

dfe252d chore: prepare Tokio v1.24.0 release (#5353)

21b233f test: bump version of async-stream (#5347)

7299304 Merge branch 'tokio-1.23.x' into master

1a997ff chore: prepare Tokio v1.23.1 release

a8fe333 Merge branch 'tokio-1.20.x' into tokio-1.23.x

ba81945 chore: prepare Tokio 1.20.3 release

763bdc9 ci: run WASI tasks using latest Rust

9f98535 Merge remote-tracking branch 'origin/tokio-1.18.x' into fix-named-pipes-1.20

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.
Confusing Chart/Alert: `indexed search request errors every 5m by code`
Sourcegraph version: v3.x.x and forward

Platform information: any deployment type

The indexed search request errors every 5m by code alert may be triggered by context cancelled events during a streaming search. These events can occur when a user cancels a search thats running too long.

Example graph:

These errors are not always indicative of an error in Sourcegraph's search execution.

Expected behavior:

This alert should distinguish the types of errors that are occurring in search execution better, such that admins may distinguish better critical failure types and regular user generated events.

The alert documentation should also be improved with actionable information for admins attempting to optimize their instance.

If you would like immediate help on this, please email [email protected] (you can still create the issue, but there are no SLAs on issues like there are for support requests).

/cc @sourcegraph/search-core