Let’s say a S3 prefix is being used by multiple services. I want to update certain object content locally and write it back to the same S3 object key IFF no other service touch that one before my work. Would it possible with this CLI?

Like for my own work, I interact with that S3 prefix with this CLI, will I be able to easily pull and tell objects added/updated/deleted by others?

Curious how does the versioning work in such case?

$ art init s3://art-vcs/datasets/foobarbar
$ art push
upload objects: (125/125), speed: 1.07MB/s
0 modified(M), 125 added(+), 0 deleted(-), 0 renamed(R)
create commit: c1ff40606ec9d01eb3d33bddfc049f672eab5d69
update ref: latest -> c1ff40606ec9d01eb3d33bddfc049f672eab5d69

support partial download

# get
art get -o output repo -- path/to/file1 path/to/file2 data/

# pull
art pull -- path/to/partia
art pull v0.1.0 -- path/to/partia ...

art status shows the differences between the latest remote repo and the local workspace.

Show status of the workspace

Usage:
  art status

Examples:
	# check current status
	art status

The original flow to download a repo to a workspace is art init + art pull.

We add a art clone command to make it simpler.

Usage:
  art clone <repository> [<dir>]

Examples:
  # clone a workspace with local repository
  art clone /path/to/mydataset

  # clone a workspace with s3 repository
  art clone s3://mybucket/path/to/mydataset

Add linker flags -s and -w to reduce binary size by removing debug and DWARF symbols

Saved around 6M before compression.

Ref: https://pkg.go.dev/cmd/link

-s Omit the symbol table and debug information. -w Omit the DWARF symbol table.

Comparison

# Before
$ go build -o bin/avc -ldflags ' -X github.com/infuseai/artivc/cmd.tagVersion= -X github.com/infuseai/artivc/cmd.gitCommit=6895c74b6ca090770e0338447bf62d2c7a2bb1e3 -X github.com/infuseai/artivc/cmd.gitTreeState=clean ' main.go
$ stat -f%z bin/avc
29522322

# After
$ go build -o bin/avc -ldflags ' -X github.com/infuseai/artivc/cmd.tagVersion= -X github.com/infuseai/artivc/cmd.gitCommit=6895c74b6ca090770e0338447bf62d2c7a2bb1e3 -X github.com/infuseai/artivc/cmd.gitTreeState=dirty -s -w ' main.go
$ stat -f%z bin/avc
23132178

Signed-off-by: Ash Wu [email protected]

Repo URL

avc init gs://<bucket>/<path>

Configuration

1. Use default application credential by

   gcloud auth application-default login  
   

2. or explicit credentials

   GOOGLE_APPLICATION_CREDENTIALS=<service account json>
   

It seems to me that avc log doesn't really require going back to the server for default output. It would be useful to either default to local-only or to have an optional argument that instructs it to only look at the local store.

avc doesn't track the deletion of files - that is, when you remove a file, it merely removes it from the current commit set. This can be problematic when you have multiple collaborating users or multiple working directories.

Example:

1. On computer X, create file A in avc repository and push it.
2. On computer Y, pull the repository. Note that file A exists.
3. On computer X, remove file A and push the update. A is not in the commit and isn't in the working directory.
4. on computer Y, pull the repository. Note that file A still exists. If Y pushes without manually removing file A, file A will reappear in the repository.

I'm new using the tools, I think is really interesting and useful. While using it I got confused by the command "put" and "push" I understand that both will upload the data to the repository but I'm not sure when should I use either of them.

Some additional metadata would be useful:

• the user who pushed a commit
• the user and date associated with pushing a specific version of a file or directory

In principal, the date is retrievable by following the history back for a particular commit. The username is not persisted, though and it might be better to keep it as per-blob metadata or carry forward with each commit. As an alternative, each blob in a commit could point back to the commit that originated it.

It would be nice to have a progress bar for the push commands. We are testing it out on one of our large medical imaging dataset (0.5M objects = 0.6TB) and the data repository took 7h to push.

I imagine this could probably extend to the other commands as well.