Git-like capabilities for your object storage

Currently lakeFS register openapi handlers and handle all specific routes. In case of a call to /api/v1/test, the unknown path under the API prefix, the mux will serve the request by the UI handler and return a valid HTML (UI) page.

The expected behaviour is to return a non-2xx status code with JSON error - prefered the internal error format, so the developer will handle an error and not fail to parse the response in case of a bad API call.

This might be intended and just needs an update on your website but I ran into a hard to debug issue trying to sync data into LakeFS with RClone. Note, I've followed the instructions previously on the site and this worked so not sure what LakeFS version it broke in. The behavior seems to point towards the client side however I haven't updated RClone at all since it did previously work.

LakeFS Version: 0.48.0(Also reproduced on 0.47.0) RClone Version:

rclone v1.56.0
- os/version: darwin 11.5.2 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.6
- go/linking: dynamic
- go/tags: none

Error trying to copy local data into LakeFS using the suggested configuration:

.... removed ...

2021/09/02 21:07:25 INFO  :
Transferred:   	   14.481Ki / 14.481 KiByte, 100%, 0 Byte/s, ETA -
Errors:                 2 (retrying may help)
Elapsed time:         1.2s

2021/09/02 21:07:25 Failed to sync with 2 errors: last error was: s3 upload: 403 Forbidden: <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><Resource></Resource><Region>us-gov-west-1</Region><RequestId>5d580752-d488-4f4f-976b-358729075279</RequestId><HostId>D071307137586F18</HostId></Error>

To test that the credentials were correct I went the other way and used LakeFS as a source after adding a file via the UI

❯ rclone ls lakefs:aif-xxxx/main/ -vv

2021/09/02 21:10:46 DEBUG : Setting --ca-cert "/Users/e379822/certs/lm_ca.pem" from environment variable RCLONE_CA_CERT="/Users/e379822/certs/lm_ca.pem"
2021/09/02 21:10:46 DEBUG : rclone: Version "v1.56.0" starting with parameters ["rclone" "ls" "lakefs:aif-xxxx/main/" "-vv"]
2021/09/02 21:10:46 DEBUG : Creating backend with remote "lakefs:aif-xxxx/main/"
2021/09/02 21:10:46 DEBUG : Using config file from "/Users/e379822/.config/rclone/rclone.conf"
2021/09/02 21:10:46 DEBUG : fs cache: renaming cache item "lakefs:aif-xxxx/main/" to be canonical "lakefs:aif-xxxx/main"
        4 test.txt
2021/09/02 21:10:47 DEBUG : 6 go routines active

What put me onto it was the signature signing is this log line in the LakeFS stating it was using SigV4. time="2021-09-03T01:51:31Z" level=warning msg="error verifying credentials for key" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:54" authenticator=sigv4 error=SignatureDoesNotMatch key=AKIAJ6UDLXIPOISF7LKQ

Also verified by dumping the RClone headers that it was using SigV4 authentication by dumping the headers.

2021/09/02 20:56:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 20:56:06 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 20:56:06 DEBUG : HTTP REQUEST (req 0xc000ad0100)
2021/09/02 20:56:06 DEBUG : PUT /aif-xxxx/main/test/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJ6UDLXIPOISF7LKQ%2F20210903%2Fus-gov-west-1%2Fs3%2Faws4_request&X-Amz-Date=20210903T025606Z&X-Amz-Expires=900&X-Amz-SignedHeaders=content-md5%3Bcontent-type%3Bhost%3Bx-amz-acl%3Bx-amz-meta-mtime&X-Amz-Signature=c3b12b3bb96069f6102df42eb22b4a64d7bc728800e53fee1b6372547710fdeb HTTP/1.1
Host: s3.lakefs.ai.us.lmco.com:443
User-Agent: rclone/v1.56.0
Content-Length: 4
content-md5: uh8lEfwwQjvbsYP+M/PdDw==
content-type: text/plain; charset=utf-8
x-amz-acl: private
x-amz-meta-mtime: 1630636753.967220656
Accept-Encoding: gzip

Setting the V2 auth in rclone does fix this issue:

❯ rclone sync -v test lakefs:aif-xxxx/main/test/ --s3-v2-auth --dump headers

....

2021/09/02 21:16:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 21:16:33 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/09/02 21:16:33 DEBUG : HTTP REQUEST (req 0xc0005a4200)
2021/09/02 21:16:33 DEBUG : PUT /aif-xxxxx/main/test/test.txt HTTP/1.1
Host: s3.lakefs.ai.us.lmco.com:443
User-Agent: rclone/v1.56.0
Content-Length: 4
Authorization: XXXX
Content-Md5: uh8lEfwwQjvbsYP+M/PdDw==
Content-Type: text/plain; charset=utf-8
Date: Fri, 03 Sep 2021 03:16:33 UTC
X-Amz-Acl: private
X-Amz-Meta-Mtime: 1630636753.967220656
Accept-Encoding: gzip

.... 
 
2021/09/02 21:16:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/09/02 21:16:34 DEBUG : test.txt: md5 = ba1f2511fc30423bdbb183fe33f3dd0f OK
2021/09/02 21:16:34 INFO  : test.txt: Copied (new)
2021/09/02 21:16:34 DEBUG : Waiting for deletions to finish
2021/09/02 21:16:34 INFO  :
Transferred:   	    4.827Ki / 4.827 KiByte, 100%, 0 Byte/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         1.5s

2021/09/02 21:16:34 DEBUG : 9 go routines active

It's an easy enough work around to add v2_auth = true, however I wanted to report it in case it is a bug since you have SigV4 listed as supported here

Fixes #2886

Command used to ingest:

 LAKECTL_S3_ENDPOINT_URL="http://127.0.0.1:9000"
  ./lakectl ingest --from s3://src-buck --to lakefs://dest-buck/main/ --intg minio 

Refer users to the documentation from the webui. For example, the Create-Repo dialog can link to the quickstart-create-repo doc page. Inspiration taken from Airbyte; while choosing the postgres Source:

In some places (docs, UI and other text files) we use the abbreviation "i.e." where "e.g." actually makes more sense. "e.g." means "For example", while "i.e." means "That is".

Find all places where we use "i.e." and replace it (only if applicable)

Clarification edit: the issue here is to find the places where "i.e." appears by mistake, not all of its appearances.

Have the completions code complete our "nouns" when relevant. Example:

lakectl fs upload --source <tab> // should suggest files
lakectl repo delete <tab> // should suggest repositories names

Currently there is no way for the user to perform a diff between a branch -including uncommitted changes- and another branch.

In git, the command git diff <branch name> will diff the current HEAD including uncommitted changes with the given branch. (git diff HEAD <branch name> will show the diff between the HEAD, without the uncommitted changes, and the given branch).

Decide which syntax will provide the desired result: related to #1305.

Hello, I'd like to expand the documentation on "self-hosted" instances. But for this, it would probably be beneficial if I got it to work first.

My issue is that I cannot access it through rclone. I am pretty sure that the issue lies with my nginx-config but I can't really spot it.

LakeFS is running on docker as suggested in the example.

Here is my nginx config:

	server {
		listen 443 ssl http2;
		listen [::]:443 ssl http2;
		server_name s3.my-domain.de
		server_name ~^(?<bucket>[a-zA-Z0-9\-_]+)\.s3\.my-domain\.de$;

		# … SSL Stuff

		if ($bucket = false) {
			return 301 https://my-domain.de/s3-note;
		}
		
		location / {
			proxy_http_version     1.1;
			proxy_set_header       Host $bucket.s3.my-domain.de;
			proxy_set_header       Authorization '';
			proxy_hide_header      x-amz-id-2;
			proxy_hide_header      x-amz-request-id;
			proxy_hide_header      Set-Cookie;
			proxy_ignore_headers   "Set-Cookie";
			proxy_buffering        off;
			proxy_intercept_errors on;
			proxy_pass             http://127.0.0.1:8000/$uri;
		}
	}

The rclone error Message is:

$ rclone copy . storage://testbucket/master -P
2021-03-31 10:24:42 ERROR : : error reading destination directory: RequestError: send request failed
caused by: Get "/": stopped after 10 redirects

Hi,

I follow the quickstart, by downloading the docker-compose.yml, running docker-compose up, then I go to http://127.0.0.1:8000/setup, enter a username 'maarten' and get:

lakefs_1    |      ██╗      █████╗ ██╗  ██╗███████╗███████╗███████╗
lakefs_1    |      ██║     ██╔══██╗██║ ██╔╝██╔════╝██╔════╝██╔════╝
lakefs_1    |      ██║     ███████║█████╔╝ █████╗  █████╗  ███████╗
lakefs_1    |      ██║     ██╔══██║██╔═██╗ ██╔══╝  ██╔══╝  ╚════██║
lakefs_1    |      ███████╗██║  ██║██║  ██╗███████╗██║     ███████║
lakefs_1    |      ╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝╚═╝     ╚══════╝
lakefs_1    |
lakefs_1    | │
lakefs_1    | │ If you're running lakeFS locally for the first time,
lakefs_1    | │     complete the setup process at http://127.0.0.1:8000/setup
lakefs_1    | │
lakefs_1    |
lakefs_1    | │
lakefs_1    | │ For more information on how to use lakeFS,
lakefs_1    | │     check out the docs at https://docs.lakefs.io/quickstart/repository
lakefs_1    | │
lakefs_1    |
lakefs_1    | time="2020-11-12T13:41:32Z" level=info msg="schema migrated" func="db.(*DatabaseMigrator).Migrate" file="build/db/migration.go:49" direction=up host="127.0.0.1:8000" method=POST path=/api/v1/setup_lakefs request_id=e4eacba2-ea5a-491b-a77b-28821f2e72df service_name=rest_api took=3.740318ms
lakefs_1    | time="2020-11-12T13:41:32Z" level=error msg="SQL query failed with error" func="db.(*dbTx).Get" file="build/db/tx.go:83" args="[Admins 2020-11-12 13:41:32.291128212 +0000 UTC m=+8.923394987]" error="scany: rows final error: ERROR: duplicate key value violates unique constraint \"auth_groups_unique_display_name\" (SQLSTATE 23505)" query="INSERT INTO auth_groups (display_name, created_at) VALUES ($1, $2) RETURNING id" took="4.118µs" type=get

Am I doing something wrong, or is the docker image broken?

cheers,

Maarten

When running lakectl log on a nonexistent repository, a misleading message appears.

Example: lakectl log lakefs://notarepo@master

Output: Error executing command: branch 'master' not found

Expected: Error executing command: repo 'notarepo' not found

Reported on Slack.

Few hundred thousands objects merge fails after 30 seconds with (UI):

time="2021-11-10T15:33:02Z" level=trace msg="HTTP call ended" func=pkg/httputil.TracingMiddleware.func1.1 file="build/pkg/httputil/tracing.go:149" host=xxxx<http:///> method=POST path=/api/v1/repositories/aif-xxxx-xxxx/refs/xxxx-xxxx-processing-a6c0f/merge/main request_body="[123 125]" request_id=da585880-e027-41bb-b3cc-b64bfa08a818 response_body="{\"message\":\"merge in CommitManager: apply ns=s3://lakefs-data/aif-xxxx-xxxx<s3://lakefs-data/xxx> id=3fe239d80f74e" response_headers="map[Content-Type:[application/json] X-Request-Id:[da585880-e027-41bb-b3cc-b64bfa08a818]]" sent_bytes=0 service_name=rest_api status_code=500 took=30.008900206s

lakectl:

Command

/home/lakefs $ lakectl merge lakefs://aif-xxx-xxx/xxx-xxx-processing-a6c0f lakefs://aif-xxx-xxx/main -c /tmp/.lakectl.yaml --l
og-level trace
DEBU[0000]/build/cmd/lakectl/cmd/root.go:67 github.com/treeverse/lakefs/cmd/lakectl/cmd.glob..func66() loaded configuration from file                fields.file=/tmp/.lakectl.yaml file=/tmp/.lakectl.yaml
Source: lakefs://aif-xxx-xxx/xxx-xxx-processing-a6c0f
Destination: lakefs://aif-xxx-xxx/main
504 Gateway Timeout

Logs from Pod:

time="2021-11-10T18:19:39Z" level=trace msg="HTTP call ended" func=pkg/httputil.TracingMiddleware.func1.1 file="build/pkg/httputil/tracing.go:149" host=lakefs.ai.us.lmco.com method=POST path=/api/v1/repositories/aif-xxx-xxx/refs/xxx-xxx-processing-a6c0f/merge/main request_body="[123 125]" request_id=fc0348ba-0c85-4674-921f-253dd6839b66 response_body="{\"message\":\"merge in CommitManager: apply ns=s3://lakefs-data/aif-xxx-xxx id=3fe239d80f74e" response_headers="map[Content-Type:[application/json] X-Request-Id:[fc0348ba-0c85-4674-921f-253dd6839b66]]" sent_bytes=0 service_name=rest_api stat us_code=500 took=29.992150768s

Feature Request

Currently it is possible to create lakefs resources (such as repository, branches, users, credentials, branch protection rules, garbage policies, ...) using:

• Web UI
• CLI (lakectl)
• Java/Python clients

It could be really nice to have a terraform provider like github terraform provider to do some infrastructure as code with LakeFS

Example

We could stuff like:

# Init provider
provider "lakefs" {
  host = "https://lakefs.example.com/api/v1"
  access_key = "AKIAlakefs12345EXAMPLE"
  secret_key  = "abc/lakefs/1234567bPxRfiCYEXAMPLEKEY"
}

# User for_each to create users and attach them to admin group
resource "lakefs_user" "data_engineers" {
  for_each = local.data_engineers
  username = each.value
}

resource "lakefs_group_membership" {
  for_each = local.data_engineers
  group = "Admins"
  username = lakefs_user.data_engineers[each.value].id
} 

# Create repo and add garbage policy
resource "lakefs_repository" "my_repo" {
  name = "my-repo"
  storage_namespace = "s3://${aws_s3_bucket.my_bucket.name}/my-repo/"
}

resource "lakefs_repository_garbage_policy" "my_repo" {
  repository = lakefs_repository.my_repo.id
  policy = { ... }
}
... 

• Avoid temporary files
• Don't read entire "rclone check" output into memory
• Detect "rclone check" success/failure directly rather than via error output

This is part of #4841 but obviously not a fix for it.

Bumps json5 from 2.2.1 to 2.2.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.