We are the Dev Teams of
  • brands
  • ebay_main
  • ebay
  • mobile
<
>
BLOG

Availability Zone based Terraforming on Openstack

by Holger Riegel
in How We Do It

An article by Holger RiegelhriQwC2kegeWl@e9rVbay4xD.coV0mRTv and Hans-Joachim Skwirblieshskwir1Qlblies@G7Hebay.cQgomQq

While HashiCorp's Terraform is well suited to script the setup of your platform in an openstack cloud and can be used out of the box in testing environments it can have some shortcomings for production environments. This article shows you why the usage of Terraform in a production environment must be considered carefully and how we use terraform in a production ready way.

Here at eBay eCG we have a private cloud implemented with openstack (https://www.openstack.org/). We use terraform (https://www.terraform.io) to set up and maintain all openstack resources.

Terraform is an excellent tool when it comes to scripting a platform's infrastructure setup.

For us advantages of Terraform (version 0.6.16) are:

  •  changes to your platform can be seen by simply diffing an old version of your terraform script with a newer version
  • when checked in into a version control system like git, you have the whole history of platform changes on an infrastructure level
  • you don't have to memorize lengthy sequences of clicks
  • Terraform supports multiple providers. Here we only need the openstack provider.

Shortcomings of Terraform that we would like to overcome:

  • Changing one line in the terraform scripts can lead to your platform being wiped out completely and set up from scratch afterwards.   This might not always be avoidable with the approach Terraform is taking.
  • During re-setup of a server class it is likely to happen that all servers of this server class are down for a short moment. For example all your web servers might be down at once making your entire web site unavailable for a short amount of time.
  • One little mistake in the Terraform scripts can make your platform unusable for a certain amount of time. While this might be tolerable for testing environments, it's certainly not desired for production environments.

One additional thought concerning the shortcomings: Terraform provides the command "terraform plan" that will show you in very detail which resources are created, changed or deleted during the next "terraform apply" run, so you can reason about the changes. On the one hand this is very good to get an impression of what will happen to your platform with the next terraform "apply run", on the other hand with the platform becoming more complex it might be hard to foresee and understand all the implications of the platform changes.

So "terraform plan" will not suffice to have a reliable production platform that is managed by terraform scripts.

Overcoming the shortcomings

To mitigate the shortcomings of terraform we implemented the idea of letting terraform manage only a subset of our platform resources with one terraform run.

This way the risk of unintentionally ruining the entire platform is greatly reduced. It would be possible to roll out changes for a subset of the platform only and test the changes before the entire platform is updated finally. But how to determine how to slice the subsets? We opted for choosing all resources within one availability zone for each terraform run. An availability zone contains a set of resources that are physically separated from resources in other availability zones. In legacy terms you can think of an availability zone as a rack in a datacenter. Defining availability zones makes maintenance of the cloud's underlying hardware easier.

We postulate some requirements for our approach:

  • requirement of balance: each availability zone has the identical number of machines like all the other availability zones. This way each zone holds a small copy of the entire platform. If one availability zone is out of order (e.g. it is undergoing scheduled maintenance), the other zones can always deliver all the functionality of the platform. User impact should be zero.
    Simple example: Lets say we have 3 zones, and the machines called "nginx01", "nginx02", "nginx03" are our static HTML servers. So nginx01 will be placed in zone 1, nginx02 in zone 2 and nginx03 in zone 3. 
    What if we reach the capacity limit of our nginx servers and we need more machines to 
    satisfy all the requests of our customers? 
    Normally we could add one more machine only, but doing this here would violate the requirement of balance. So what we need to do is to add nginx04, nginx05 and nginx06, each located in another zone.  In this way the platform is balanced again, and it doesn't matter which zone is chosen for maintenance.
  • requirement of DRY principle: our code should not repeat the same structures over and over again. So we should use the very same scripts for each availability zone. The only difference should be a terraform command line parameter with the zone number specifying for which zone the script should be run.
  • requirement of zone transparency: the zone should not be reflected in the server name. So the names of machines in the same class of servers should be consecutively numbered. So all nginx servers should be numbered like nginx01, nginx02, nginx03 and so on. 

Implementation

So the basic idea is to have one set of scripts that is parameterized with the zone number and can be run for each zone separately.

But what about the commonly used parts of our openstack infrastructure? Like openstack load balancers (LBAAS)? Load balancers are a not zone aware ressource, so should it be defined in our terraform script for each zone? 

We choose to have the not zone aware ressources (aka common ressources) in a separate terraform script. So there is a common set of ressources like in the picture below, and the zone aware resources in each availability zone may depend on those common resources like in the following picture: 

This set up has implications for the order in which the terraform scripts are run: we first run the terraform scripts for the common ressources, then in arbitrary order for each availability zone the scripts for the zone ressources. The following picture summarizes this:

Workflow of terraform runs

So the concept is quite clear by now, let's dive deeper into the implementation. Let's start with the question, how the zone terraform scripts can reference common resources. Our approach is to collect values generated by the common terraform scripts in a terraform variables file and to pass this variables file to the terraform scripts of each zone. We make use of the terraform output feature (see https://www.terraform.io/intro/getting-started/outputs.html). So we collect all the important common information in variables. For example the load balancer id is taken from a resource declared like this:

resource "openstack_lb_monitor_v1" "monitor" {
  type           = "TCP"
  delay          = 10
  timeout        = 5
  max_retries    = 3
  admin_state_up = "true"
}

resource "openstack_lb_pool_v1" "pool" {
  name        = "nginx"
  protocol    = "TCP"
  subnet_id   = "${var.lb_subnet_id}"
  lb_method   = "ROUND_ROBIN"
  monitor_ids = ["${openstack_lb_monitor_v1.monitor.id}"]
}

resource "openstack_lb_vip_v1" "vip" {
  name        = "nginx"
  subnet_id   = "${var.lb_subnet_id}"
  protocol    = "TCP"
  port        = 80
  pool_id     = "${openstack_lb_pool_v1.pool.id}"
  floating_ip = "${var.floating_ip}"
}

and the output value containing the load balancer id is defined like this (see https://www.terraform.io/docs/configuration/interpolation.html for terraform's interpolation syntax):

output "nginx_lb_pool_id" {
  value = "${openstack_lb_pool_v1.pool.id}"
}

Please keep in mind: if the pool is defined within a terraform module, we first have to
do an output statement in our module like the one above and then an output statement in
our main.tf, like so:

output "nginx_lb_pool_id" {
  value = "${module.nginx.nginx_lb_pool_id}"
}

Finally we wrote a simple bash wrapper script for calling terraform:

TERRAFORM_STATE_FILE="./terraform.tfstate"
VARIABLES_FILE="shared-zone-variables.tfvars"

terraout() {
        echo "#generated by tf.sh" > $VARIABLES_FILE
        echo "#variables listed here are the same for all zones" >> $VARIABLES_FILE
        echo >> $VARIABLES_FILE
        terraform output -no-color -state "${TERRAFORM_STATE_FILE}" | awk -F' = ' '{printf "%s=\"%s\"\n", $1, $2}' >> $VARIABLES_FILE
}

CMD=$1
terraform get
terraform $CMD -var-file tfvars/other-variables.tfvars -state "${TERRAFORM_STATE_FILE}"
TERRA_RETURN=$?

if [ "$TERRA_RETURN" -eq 0  ] && [ "$CMD" == "apply" ] ; then
        terraout
fi

exit $TERRA_RETURN

When terraform is successfully run with command "apply" a second terraform run is done 
with command "output", see the bash function terraout(). The id of the loadbalancer is written out to a file called "shared-zone-variables.tfvars". The script is doing some reformatting, so that it can be sourced in by other terraform scripts. Basically the variable value must be enclosed in quotation marks. So the "shared-zone-variables.tfvars" should contain lines like: 

nginx_lb_pool_id="12345678-abcd-efgh-ijkl-mnopqrstuvwx"

This file is referenced for the zone terraform runs like so:

terraform get
terraform $CMD -var "zone_no=$zone" -var-file ../shared-zone-variables.tfvars \
          -var-file ../tfvars/other-variables.tfvars -state ./terraform_zone$zone.tfstate

Where the variable $CMD holds the terraform command like "plan" or "apply" and the variable $zone contains the zone number. You can see that the generated shared-zone-variable.tfvars file is referenced. In this way the variable nginx_lb_pool_id can be used in the terraform zone scripts. The zone aware terraform file looks like this (simplified):

resource "openstack_compute_instance_v2" "nginx" {
  count             = "${var.instances_per_zone}"
  name              = "nginx${format("%02d",count.index * var.zone_total + var.zone_no)}"
  image_name        = "${var.image_name}"
  flavor_name       = "${var.flavor_name}"
  security_groups   = ["${split(",", var.security_groups)}"]
  availability_zone = "zone${var.zone_no}"

  network {
    name           = "${var.network_name}"
    access_network = true
  }

  key_pair = "${var.keypair_name}"

  scheduler_hints {
    group = "${openstack_compute_servergroup_v2.nginx.id}"
  }

  user_data = "${element(template_cloudinit_config.combined.*.rendered, count.index)}"
}

resource "openstack_lb_member_v1" "member" {
  count          = "${var.instances_per_zone}"
  pool_id        = "${var.nginx_lb_pool_id}"
  address        = "${element(openstack_compute_instance_v2.nginx.*.access_ip_v4, count.index)}"
  port           = 80
  admin_state_up = "true"
}

Here the instance of an nginx machine is created. Look at the pool_id line in the openstack_lb_member_v1" ressource. Here the nginx_lb_pool_id from the common resources is referenced (marked with blue background in the above script).

Other interesting variables (marked with orange background in the above script):

  • instances_per_zone: how many nginx machines we have per zone. Please note there is no variable defining the overall number of servers in one server class. Using the instances_per_zone variable ensure the requirement of balance is met. The overall number of servers in one server class can be computed by multiplying the instances_per_zone variable with the zone_total variable.
  • zone_no: the zone_number the terraform run was called with
  • zone_total: the total number of zones. we have 3 availability zones, so the value of this variable is always 3 for our platform.

You can see that the machine name is computed with the help of zone_total and zone_no.
This is due to the requirement of transparency. In this way the numbering of machine names of the same class of servers is always consecutive.

It could be discussed - and finally it's a matter of taste - to reflect the availability zone somehow in the server name, for instance like "nginx201" for the first nginx server in availability zone 2, or make the zone explicit as an DNS subdomain like "nginx01.zone2". Pro for the the consecutive numbering is simplicity of server names and the pro for the explicit naming approach is you immediately know the zone for each server. But as I said in the end it's a matter of taste to stick to the requirement of 
transparency.

Conclusion - TL/DR

By organizing a platform's resources in a common and a zone aware part and using terraform's features like its output facility along with a simple wrapper script it's possible to use terraform even in a production environment throughout the entire platform lifecycle. The proposed solution adheres to the requirements of balance and the good old DRY principle - making the platform more robust and easier to manage.

terrraform, openstack, availability zone, devops

?>