(This is an enhancement)

It would be really cool if it were possible to automatically enable https through the new Let's Encrypt service, since it's now trusted by all major browsers. It's also part of the Linux Foundation so it's rooted in the Open Source community.

This is an experiment to test out using squashfs based images which have many benefits over the AUFS diffs currently in use:

• Easy to distribute
• Easy to mount
• Easy to checksum / verify
• Easy to share between jobs (mounted once, uses the kernel file-cache)

The aim is to also use squashfs for slugs so they benefit from the above too.

image format

An image is a JSON manifest which contains a list of layers which should be mounted, e.g.:

{
  "version": "v1",
  "layers": [
    {
      "id": "3fdfac2f9355dfdb1e3b1af1afdf143acabd555dcd8fc6b80c1ea62423ac2122a634b119140663ee213c0d621d1e51de40ac9cdf8cbaa24ca7a27593ba7d05ff.squashfs",
      "type": "squashfs"
    },
    {
      "id": "8d4f80ddb31de6c9a5413c1f0ca1fb4585866e2b966591491ccc0363f2760a644d1512786eb8f5500a873d2da4f0636fa72202b4fcd68e823be8489ac543bd32.squashfs",
      "type": "squashfs"
    },
    {
      "id": "385d3eda8d4823312f6e0abcc7925795dd5b587ed333b1d29ebb83fa50dbbbbe1697dcb3e55e01eb9f4d85e76cbdbac3f907d8ff122dd09e27e56cdad8f573d5.squashfs",
      "type": "squashfs"
    }
  ]
}

mounting an image

Images are mounted using overlayfs with a lowerdir for each of the squashfs layers from the manifest (requires Linux >= 3.19, see here), and a temporary upperdir.

storage

There is a root directory (hard-coded as /var/lib/flynn/image) with 4 sub-directories:

• manifests - image manifests
• layers - squashfs layer files
• mnt - mounted squashfs layers + mounted overlayfs job mounts
• tmp - temporary files

Images are stored as SHA512.json, layers as SHA512.squashfs, with images supporting "tagging" via symlinks (e.g. manifests/cedarish.json -> manifests/1062edd28317eff4f58be2da2fde33c6fb8477bfe7b788f7c0ce10a83e8a297c.json)

convert docker -> squashfs

I've added util/imagebuilder which takes a reference to a Docker image and creates squashfs layers for each tagged layer in the image's history.

For example flynn/slugrunner is converted to three layers: ubuntu, cedarish and the slugrunner scripts, whereas flynn/controller becomes two layers: busybox + controller binaries.

The generated layers are tagged with the ID of the corresponding Docker image to avoid converting already converted images.

We probably want to support building squashfs layers from scratch, but converting from Docker images gives us a good starting point.

Flynn partially boots but I had some instances of files being missing, but thought it was worth getting feedback on what has been done so far.

/cc @titanous @josephglanville

Hi,

I've read through the documentation and think I'm following the right set of steps to install Flynn, but I can't seem to get layer 1 started.

• I created an Ubuntu 14.04 VM called and added a proxy to /etc/environment to pull code from external repos.
• The program apt-get and update manager use a different proxy setting and will not obey the environment variables. Create a file called 95proxies in /etc/apt/apt.conf.d/, and include the proxy using the correct syntax.
• Reboot to let this take effect.
• Get the Flynn install script:

curl -fsSL -o install-flynn https://dl.flynn.io/install-flynn

Copy it to install-flynn.sh, chmod +x, and make an edit to allow it to go through the proxy by specifying --keyserver-options http-proxy=myproxy

• Run the install script:

$ sudo ./install-flynn.sh

That results in the happy sight:

===> 16:04:22.212 setting libvirt defaults ===> 16:04:22.216 installing Upstart job ===> 16:04:22.241 installation complete!

• Start Flynn layer 0 and query its state with these commands $ sudo start flynn-host $ sudo status flynn-host

Flynn didn’t start so I dumped the log to find out what was going on:

$ sudo cat /var/log/upstart/flynn-host.log now=2015-10-02T20:34:27+0000 app=host at=start now=2015-10-02T20:34:27+0000 size=2773551513 at=zpool_size app=host fn=main DBUG[10-02|16:34:27] zfs get all -Hp flynn-default package=zfs 16:34:27.956398 host.go:212: could not initialize default provider: exit status 1: "/sbin/zfs zfs get all -Hp flynn-default" => The ZFS modules are not loaded. Try running '/sbin/modprobe zfs' as root to load them.

• I followed the instructions and it worked: $ sudo /sbin/modprobe zfs $ sudo start flynn-host flynn-host start/running, process 19378 $ sudo status flynn-host flynn-host start/running, process 19378
• Obtain a cluster token.

$ sudo flynn-host init --init-discovery https://discovery.flynn.io/clusters/9bae27e6-1cb2-4486-8791-d6d9dcf00000

• Init the other cluster members to the same ID using this command: sudo flynn-host init --discovery https://discovery.flynn.io/clusters/53e8402e-030f-4861-95ba-d5b5a91b5902
• To spoof the DNS records I plan to use a combination of a hosts file and dnsmasq during an evaluation. xip.io is not reachable behind my corporate firewall, so it is not an option.

$ sudo vi /etc/hosts 127.0.0.1 demo.localflynn.com

sudo apt-get dnsmasq sudo vi /etc/dnsmasq.conf and inserted this line: address=/demo.localflynn.com/127.0.0.1 sudo reboot

now ping martin.demo.localflynn.com gets a response.

• The next step is to start layer 1 using the bootstrap command. So I did that via this command: $ sudo su

  CLUSTER_DOMAIN=demo.localflynn.com \

  flynn-host bootstrap
  --min-hosts 3
  --discovery https://discovery.flynn.io/clusters/9bae27e6-1cb2-4486-8791-d6d9dcf00000

But at the preset time that times out waiting for the hosts to come online: 12:57:47.557283 check online-hosts 12:58:18.853217 check online-hosts error: timed out waiting for 3 hosts to come online (currently 0 online)

I'm at a bit of a loss as I think I've done everything right, but I don't know what I'm missing. Any help is appreciated.

Martin

My apps seem to still be running but pretty much everything else doesn't work such as

flynn ps backup dashboard can't collect debug info

even fix hosts end on an error of 14:00:30.676978 host.go:136: already have enough instances, unable to fix

is there anything I can do that will help fix it at least so that I can backup to a new cluster

I'm trying to get this running on a new server using the cloud installer and it seem to get fairly far into the install until I get the error

Error: bootstrap: online-hosts check error: timed out waiting for 1 hosts to come online (currently 0 online)
The following hosts were discovered but remained unreachable:

but there is nothing after that so can't tell what the hosts are that's discovered. I've probably got something on my server setup wrong but not sure what.

This happen whilst trying to push an update, I didn't get any errors i just got stuck half way through. Now flynn commands don't work and I don't think flynn-host collect-debug-info is working either as that been sat on getting scheduler state for 30 mins.

Store routing state in a PostgreSQL database. Closes #549.

TODO

• [x] fix broken tests.
• [x] remove etcd datastore.
• [x] investigate slow tests (14s on CI).

/cc @bgentry

This probably still needs a lot of work, but I've got what I think is a good core of a solution for #2633. @titanous, can you give it a quick glance, tell me if you see problems with the changes thus far.

https://gist.github.com/temujin9/2f5e675bfb8fdf96cf3a

The hard drive fills up rather quickly (I suspect a container started logging a lot), and there are no good solutions documented for cleanup.

The current scheduler implementation feels slightly hacked together and there are a number of open issues relating to it's short comings (#1182, #1434, #686, #129, #54, #1340).

We should refactor the scheduler to both fix these issues and make it more flexible to handle different scheduling policies (e.g. resource based, tag limiting etc.).

Here are the main responsibilities of the scheduler:

• Keep an accurate state of running jobs in memory (being handled in #1373), and ensure it matches the requested scale
• React to formation changes by adding / removing jobs from hosts
• Restart crashed jobs using an exponential backoff policy
• Update the jobs_cache DB table with job states (including ones scheduled to start in the future)

Some improvements to be made based on the current implementation:

• Support arbitrary constraints when running new jobs (e.g. available resources, host tags, data locality etc.)
• Actively keep running jobs in sync with formations, using a deterministic algorithm to determine which jobs to remove if necessary (so if multiple schedulers are scheduling, they will both start new jobs then race to stop the same duplicate job, thus minimising the impact of #1182)
• Actively keep jobs_cache table in sync with cluster state
• When the leader crashes, maintain the backoff policy from the crashed scheduler
• Simplify "omni" jobs by modifying formation process counts on host change so that they always match the actual number of jobs of that type (it is confusing that scheduler=2 could mean "2 jobs per host" rather than just "2 jobs in the cluster")
• Make unit testing simpler (all scheduler testing is currently at the integration level)

/cc @titanous

Having a lot of trouble with one of our clusters suddenly. flynn-host is unable to communicate with the local machine;

flynn-host list
22:33:51.177421 host.go:166: Error sending HTTP request, errors: Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111/services/flynn-host/instances: dial tcp 127.0.0.1:1111: getsockopt: connection refused,Get http://127.0.0.1:1111