wiki:CloudPlugin

Version 23 (modified by Rob Guttman, 13 years ago) (diff)

--

Orchestrates AWS cloud resources using boto and pychef

Notice: This plugin is unmaintained and available for adoption.

WARNING! This plugin is in alpha.

I recommend you not use this plugin yet on production instances. Although you likely won't lose ec2 instances or chef data, it's still quite immature and the config interface is likely to change. It would be best to try out on dev instances to help test the plugin and generate ideas for new features. I'm actively putting it through its paces now at Juju.

Description

This plugin is meant to fill the coordination gap between AWS and Chef, especially:

  • Launch an ec2 instance and apply one or more chef roles
  • Terminate ec2 instances and delete its chef node

There are additional features and quite possibly more to come, but my intention is to keep the scope down to what's needed to orchestrate between AWS and Chef (and not re-implement either the AWS Management Console or the Chef Server webui).

This plugin relies heavily upon boto and pychef. Special thanks to coderanger for his work and responsiveness on pychef.

Configuration

  1. Install the plugin (after downloading and unzipping):
    cd cloudplugin/0.12
    sudo python setup.py bdist_egg
    sudo cp dist/TracCloud*.egg /your/trac/location/plugins/
    

See TracPlugins for more installation details and options. You'll likely need to restart Trac's web server after installation.

  1. Enable the plugin in trac.ini:
    [components]
    cloud.* = enabled
    

You can alternatively use the Trac Web Admin GUI to enable any or all rules.

  1. Configure the main trac.ini section:
    [cloud]
    label = Cloud
    aws_key = <aws-access-key>
    aws_secret = <aws-secret-key>
    aws_keypair = <ec2-key-pair>
    chef_instancedata_file = /path/to/instance-data.json
    

The label option above is the name to be used in the top navigation - see screenshot above.

AWS credentials

The aws_* options are your AWS credentials and the key pair used by your ec2 instances.

Chef configuration

The chef_instancedata_file option is the absolute path to the ec2 instance/user data generated by chef's knife command:

% knife ec2 instance data role[<desired-role>] > /path/to/instance-data.json

The plugin will use this file for the instance data replacing the role information as needed. As the file contains the validation.pem data, be sure to secure it accordingly. In cases where you need different credentials and/or chefserver url, you can use the following option to specify a base directory which must contain the typical .chef directory and its contents plus (optionally) a validation.pem file within the .chef directory:

[cloud]
..
chef_base_path = /path/to/dir/that/contains/.chef

It's currently assumed that you're using one of Opscode's AMIs or a homegrown AMI derived from it. These AMIs use the instance's id as the node name. However, you may have recipes which for one reason or another change the node's name after the first chef-client run. If so, you need to specify the boto field that maps to the final node name using this config option:

[cloud]
..
boto_field_node_name = private_dns_name

Adding this option will cause the plugin to create a new node of that name upon launching a new ec2 instance and once it checks in will remove the original node. Otherwise you'll have two chef nodes for the same ec2 instance - something you likely don't want. Yeah, this is a bit complicated - an area ripe for improvement.

The plugin is intended to support multiple AWS resource types in addition to instances as needed for orchestration with chef. You specify the default resource to view using this config option:

[cloud]
..
default_resource = instance

Each AWS resource type such as instance above is configured independently. See the examples section below for how to specify these resources.

Bugs/Feature Requests

Existing bugs and feature requests for CloudPlugin are here.

If you have any issues, create a new ticket.

Download

Download the zipped source from [download:cloudplugin here].

Source

You can check out CloudPlugin from here using Subversion, or browse the source with Trac.

Example

Each supported AWS resource is configured independently in a separate section in trac.ini. (The plugin currently only supports ec2 instances.) Each section must be named as follows:

[cloud.<resource_name>]
...

For example, a section named [cloud.instance] defines a resource named instance. The resource name is used in the uri, so for example, this instance resource would be accessed at:

http://trac.domain.com/cloud/instance

You can name resources whatever you want as long as they're unique.

Instances

Here's how to configure ec2 instance resources. It's best to explain the configuration by example and piecemeal:

[cloud.instance]
class = Ec2Instance
title = Instances
order = 1
label = Instance
description = AWS EC2 instances.
  • class - must map to the exact python class name for that instance. For ec2 instances, it's Ec2Instance.
  • title - used in the sub-navigation (aka contextual) menu - see screenshot above.
  • order - defines the order in the contextual menu (much like custom field ordering).
  • label - used in several views to describe a single one of these resources.
  • description - used in the grid view - see screenshot above.

Now it gets a bit more intense (the following should really all be on one line in trac.ini):

[cloud.instance]
..
fields = name < NameHandler, ec2.instance_id as Instance ID, run_list_ as Roles < RunListHandler, created_by as Created By < AuthorHandler,
  created_at_ as Created At < EpochHandler, ohai_time_ as Last Check-in < AgoEpochHandler, ec2.instance_type as Instance Type,
  ec2.hostname as Private Hostname, ec2.public_hostname_ as Public Hostname < SshHandler, ec2.placement_availability_zone as Availability Zone,
  ec2.ami_id as Image ID, supervisord.ssl_port_ as supervisord < HttpsHandler

Whoa! The fields option is a list of those chef attributes (aka fields) where you want to provide either a display name, a format handler, or both. So for example, ec2.instance_type as Instance Type above will use 'Instance Type' wherever the name of the ec2.instance_type field is to be displayed. This as notation should be familiar to those who have written Trac report SQL (although no quotes should be used here).

Only slightly more complicated are handlers to format fields. So for example, ohai_time_ as Last Check-in < AgoEpochHandler has a display name as we've seen plus the < AgoEpochHandler directs the plugin to use a specific handler to format the ohai_time field's value. In this example, the AgoEpochHandler handler converts an epoch to a string format such as "0:08:36 ago" meaning "8 minutes and 36 seconds ago" as shown in the screenshot above. But wait, what's up with the underscore after the ohai_time_? In most cases when using a handler, you should postfix the field's name with an underscore (_) so that the re-formatted value doesn't get written back to the field's value. [I'm likely to get rid of the underscore business to make it simpler, but that's what it is for now. - rhg] See the handlers.py file for a full list of provided field handlers.

grid_index = node
grid_columns = name, run_list_, created_by, ohai_time_, ec2.instance_type, ec2.placement_availability_zone
grid_sort = created_at_
grid_asc = 0

The grid_* options configure the grid view of the resource - i.e., the view shown in the screenshot above:

  • grid_index - must map to the chef search index name. For ec2 instances, this is node.
  • grid_columns - the list of chef attributes to display in order. The column names will use any display names from fields and its cell values will be formatted using any handlers from fields.
  • grid_sort - the default sort attribute/field. You can resort the grid by clicking on the column name.
  • grid_asc - the default sort direction. You can change the sort direction by clicking on the same column name.
crud_resource = nodes
crud_view = name, ec2.instance_id, run_list_, created_by, created_at_, ohai_time_, ec2.instance_type, ec2.hostname, ec2.public_hostname_, ec2.placement_availability_zone, ec2.ami_id, supervisord.ssl_port_
crud_new = run_list_||, created_by, ec2.instance_type|, ec2.ami_id|, ec2.placement_availability_zone|us-east-1a|us-east-1b|us-east-1c|us-east-1d
crud_edit = run_list_||, created_by, ec2.instance_type*, ec2.ami_id*, ec2.placement_availability_zone*

The crud_* options configure aspects for creating, reading updating, and deleting (i.e., CRUD) the resource:

  • crud_resource - must map to the chef resource name. For ec2 instances, this is nodes.
  • crud_view - the list of chef attributes to display in order. The field names will use any display names from fields and its cell values will be formatted using any handlers from fields.
  • crud_new and crud_edit - the list of augmented chef attributes to display for editing in order. The field names will use any display names from fields and its cell values will be formatted using any handlers from fields.

The field names in crud_new and crud_edit are augmented to provide additional metadata so the plugin knows how to properly configure the form. So for example:

ec2.placement_availability_zone|us-east-1a|us-east-1b|us-east-1c|us-east-1d

The pipe (|) after the field name indicates that the form should use a select/dropdown menu for this field. If the select/dropdown should support multiple selections, then use two pipes instead of one as in run_list_||. The remainder of the definition is a pipe-delimited list of options for this field (much like how custom field options are defined). If there's no list of options, then the plugin will search a chefserver data bag of the same name as the field for the list of options. So for example, ec2.instance_type| will direct the plugin to search for a data bag on the chefserver named ec2_instance_type (periods are invalid and so are replaced with underscores). The individual data bag items should be formatted like this:

{
  "id": "m1_large",
  "value": "m1.large",
  "name": "m1.large - Large (64-bit)",
  "order": 3
}

The name will be used as the option's displayed name and the value will be used as the option's value. If order fields are provided, then they'll be used to order the options accordingly (much like custom field ordering).

Lastly, if you want to have a field displayed but as read-only, then the field name should be immediately postfixed with an asterisk as in ec2.ami_id* above. The crud_new config example above results in the following screenshot:

Any regular (i.e., non-handled) fields in the crud_new and crud_edit lists will get added as attributes to the chef node such as created_by above.

Recent Changes

14004 by rjollos on 2014-07-12 18:27:48
Correct license keyword argument in setup.py. Refs #11832.
13996 by rjollos on 2014-06-25 00:56:32
Fixed author in COPYING file. Refs #11832.
13994 by rjollos on 2014-06-25 00:51:36
Changed to 3-Clause BSD license with permission of author. Refs #11832.
(more)

Author/Contributors

Author: robguttman
Maintainer: robguttman
Contributors:

Attachments (4)

Download all attachments as: .zip