https://jira.mongodb.org/browse/EVG-5856

After some conversations, for now we'd like this setup script to be user-defined and in the future we can look at project-defined (for hosts spawned with task data). The reason this setup script is separate from userdata is because, for hosts spawned with task data, the script should be run after artifacts are loaded, which is evidently not necessarily true for userdata. (Although we also load artifacts in userdata?)

EVG-16035

Description

Currently, when a task spawns a host that AWS terminates immediately after giving it to us we fail the task on host.list. This will instead insert a new host intent so we try again.

We will insert a new intent only if the original host is still "starting". If the host went to running and then terminated the task could have moved on and this would be a waste. I've never actually seen a "running" instance terminate unexpectedly, but just in case. We also skip this if the task is not running or aborted because it would be a waste.

I'm undecided about limiting the number of times we loop on this. I didn't do it yet, but if someone feels strongly I'm game to add it. I think it's safe to assume it's a transient error since we know the instance was created once successfully (so it's not a user error). Even if the error does persist it won't loop infinitely because host.list will eventually reach its timeout.

Testing

I manually terminated a task's host in staging (very carefully!) and Evergreen created a second host for the task.

EVG-14977

Description

This was kept hidden till now so that users don't start using it before it is ready. While the feature is not yet fully ready to role out, at this point, a user using it won't break it, and they'll already see the downstream tasks on the patch page.

Testing

none

Isolating users' data in a portable volume will facilitate migrating between instances.

Since the script to mount a volume will depend on the distro (for instance, nitro based instances will rename block devices), add a field to the distro for the script with a placeholder for device name. Only those distros which set the script will have the option of mounting the volume.

The UI will validate that the size requested is between 1 and 100 GB. These seemed like reasonable limits to hardcode, but I'd be open to to other limits or to making it configurable from the admin settings.

I couldn't decide the best way to go about this so I'm just putting it up.

Editing the description here will be fine for commit queue patches since they populate the description before this.

diffData.base includes "@{upstream} which is why I don't know that gitCommitMessages can be used. Also is using this base correct?

If we can use gitCommitMessages then I would just strings.Split(commitsMsgs, "\n")[0].

Given a time range, we filter patches by Alias, Time, and Project. I'm filtering patches instead of versions because patches have the actual logic I'm looking for, and for commit queue this seems 1 to 1 (FinalizePatch creates a version with the same ID as the patch, and this is called for both PR and CLI).

From these patches I reconstruct the commit queue item and re-add it to the project.

Because this ranges first through projects and then through project patches, it's slightly inefficient; an alternative may be to get the patches for all projects and keep a hash map of projectID -> CQ, but this would only minimize the number of patch filter calls we have to make, not the number of patches we have to range over.

While I think the logic with unstartedTasks is also wrong, I think the actual case brought up in the ticket comes from the first part of this line statusTask.Status != evergreen.TaskFailed.

I'm not 100% sure what you were trying to do in the original ticket @ybrill but given the description about SystemFailed I think we should only set display task as finished early in the system failed case (if at all).

@john-m-liu Relating to the questions asked yesterday. It seems this is sufficient to get the page I care about working; wondering whether this can be done in a better way/if we want to use this corpsecure login more generally in EVG?

If a user is opted into spruce v1, they will be redirected from evergreen.mongodb.com/patch/:id to spruce.mongodb.com/patch/:id/configure

This allows us to redirect users to Spruce when they click on a patch link from the CLI

Having favorited projects also in all projects caused duplicate key issues when rendering the data in the projects dropdown. This PR filters favorite projects out of all projects.

This fixes printing errors when used in a format string such as:

			grip.Infof("Spawn host created with ID '%s'. Visit the hosts page in Evergreen to check on its status.", host.Id)

Now correctly prints:

evergreen-ci/evergreen master Δ ./clients/darwin_amd64/evergreen host create -d rhel70-small -k mathewrobinson
A new version is available. Run './clients/darwin_amd64/evergreen get-update --install' to download and install it.
[evergreen] 2018/04/20 12:45:00 [p=info]: Spawn host created with ID 'evg-rhel70-small-20180420164500-1674881231148019645'. Visit the hosts page in Evergreen to check on its status.

EVG-7440

Description

< add description, context, thought process, etc >

Testing

Tested in staging. Confirmed that restarting a PR version changes all builds to status "pending" in the GitHub UI, and that manually restarting two or more separate builds accurately changes each corresponding build variant in the GitHub UI to "pending" as well.