:::: MENU ::::

How to debug the CF/PAAS deploy process

The new breed of PAAS systems are all converging on a common deployment model.

  1. A CLI tool uploads your code / executables to the PAAS.
  2. The PAAS launches a clean “LXC based” staging container VM and invokes a buildpack on your code / executable
    1. The buildpack bin/detect ‘s whether it knows how to work with your code type (eg, it looks for a Gemfile, or a .java class).  If no, it fails.
    2. The buildpack bin/compile ‘s your code, combining it with any runtime dependancies – i.e a specific JVM or Mono runtime – and any related libraries – eg, what is specified in your .nuget package or Gemfile.  This process basically results in an “app” folder which contains all the binaries your app requires to run.
    3. The buildpack bin/release ‘s by specifying the “startup command”, and any ENV vars that should be set
  3. All of this output is then zipped up into your app.tgz and added to the PAAS blobstore.  That done, the staging container is deleted.
  4. The PAAS then fires up as many runtime “LXC based” container VMs as you have specified, unzips your app.tgz into your $HOME folder, loads the ENV vars and runs the start command specified in step 2.3
  5. The PAAS monitors your start command – if it exits it will automatically rerun step 4 to give you a fresh runtime container.
  6. The PAAS also monitors the host machines running the containers – if any of those fail it will restart all the affected runtime containers on a new host machine.  This step happens more frequently than you think (typically nightly), because its also the way that the PAAS keeps the host systems operating systems updated.

Each PAAS (Heroku, Cloud Foundry, flynn.io) has custom components that orchestrate everything, but there is a healthy open source community creating buildpacks for the languages and runtimes near and dear to their hearts; and these typically work (with minor modifications) on any of the PAASes.

Yours truly has now written 4 buildpacks for Cloud Foundry:

  1. https://github.com/mrdavidlaing/stackato-buildpack-wordpress <- WordPress running on Facebook’s HipHop PHP runtime
  2. https://github.com/cloudfoundry-community/nginx-buildpack <- Static HTML sites running on Nginx
  3. https://github.com/cloudfoundry-community/.net-buildpack <- .NET apps running on Mono
  4. https://github.com/mrdavidlaing/java-buildpack-with-Procfile-container <- Extension of the CF Java buildpack

One of the major pain points in the process is debugging the staging and runtime containers because:

  1. You can’t SSH into them to poke around and explore
  2. In the event of a catastrophic failure you the container (and its logs) get deleted before you can extract any of the files.

So, my holiday project was to try and build something to make debugging the deployment process easier.

The result is https://github.com/cloudfoundry-community/container-info-buildpack – a buildpack that exposes information about the staging and runtime containers via a web-app.  See the README.md for details on how to use it.

This little experiment has been received with enthusiasm by the CF dev community; so I think I’ve identified a common pain point.

In its development I learnt about two useful things:

  1. pstree -a <- lists all the processes currently running in your shell.
  2. forego <- a speedy and memory efficient Go implementation of foreman
  3. openresty <- a collection of nginx modules that turn nginx into a simple (and very efficient) app server, with the ability to script logic using LUA.

I’m currently experimenting with being able to wrap this “info” buildpack around another buildpack, so you can

  1. Gather additional debugging info when deploying an app – say a mono app based on https://github.com/cloudfoundry-community/.net-buildpack
  2. Re-run the staging and runtime processes without having to redeploy your app.



So, what do you think ?