[[PageOutline(2-5,Contents,pullout)]] = Orchestrates AWS cloud resources using boto and pychef = == '''WARNING! This plugin is in alpha.''' == I recommend you ''not'' use this plugin yet on production instances. Although you likely won't lose ec2 instances or chef data, it's still quite immature. It would be best to try out on dev instances to help test the plugin and generate ideas for new features. - robguttman, 2011 Jan 28 == Description == This plugin is meant to fill the coordination gap between AWS and Chef, especially: * Launch an ec2 instance and apply one or more chef roles * Terminate ec2 instances and delete its chef node [[Image(instance-grid.png)]] There are additional features and quite possibly more to come, but my intention is to keep the scope down to what's needed to orchestrate between AWS and Chef (and not re-implement either the AWS Management Console or the Chef Server webui). This plugin relies heavily upon [http://boto.cloudhackers.com/ boto] and [https://github.com/coderanger/pychef pychef]. Special thanks to coderanger for his work and responsiveness on pychef. == Configuration == 1. Install the plugin (after downloading and unzipping): {{{ cd cloudplugin/0.12 sudo python setup.py bdist_egg sudo cp dist/TracCloud*.egg /your/trac/location/plugins/ }}} See [http://trac.edgewall.org/wiki/TracPlugins TracPlugins] for more installation details and options. You'll likely need to restart Trac's web server after installation. 2. Enable the plugin in {{{trac.ini}}}: {{{ [components] cloud.* = enabled }}} You can alternatively use the Trac Web Admin GUI to enable any or all rules. 3. Configure the main {{{trac.ini}}} section: {{{ [cloud] label = Cloud aws_key = aws_secret = aws_keypair = chef_instancedata_file = /path/to/instance-data.json }}} The {{{label}}} option above is the name to be used in the top navigation - see screenshot above. === AWS credentials === The {{{aws_*}}} options are your AWS credentials and the key pair used by your ec2 instances. === Chef configuration === The {{{chef_instancedata_file}}} option is the absolute path to the ec2 instance/user data generated by chef's knife command: {{{ % knife ec2 instance data role[] > /path/to/instance-data.json }}} The plugin will use this file for the instance data replacing the role information as needed. As the file contains the {{{validation.pem}}} data, be sure to secure it accordingly. In cases where you need different credentials and/or chefserver url, you can use the following option to specify a base directory which must contain the typical {{{.chef}}} directory and its contents plus (optionally) a {{{validation.pem}}} file within the {{{.chef}}} directory: {{{ [cloud] .. chef_base_path = /path/to/dir/that/contains/.chef }}} It's currently assumed that you're using one of Opscode's AMIs or a homegrown AMI derived from it. These AMIs use the instance's id as the node name. However, you may have recipes which for one reason or another change the node's name after the first chef-client run. If so, you need to specify the boto field that maps to the final node name using this config option: {{{ [cloud] .. boto_field_node_name = private_dns_name }}} Adding this option will cause the plugin to create a new node of that name upon launching a new ec2 instance and once it checks in will remove the original node. Otherwise you'll have two chef nodes for the same ec2 instance - something you likely don't want. Yeah, this is a bit complicated - an area ripe for improvement. The plugin is intended to support multiple AWS resource types in addition to instances as needed for orchestration with chef. You specify the default resource to view using this config option: {{{ [cloud] .. default_resource = instance }}} Each AWS resource type such as {{{instance}}} above is configured independently. See the examples section [wiki:CloudPlugin#Examples below] for how to specify these resources. == Bugs/Feature Requests == Existing bugs and feature requests for CloudPlugin are [report:9?COMPONENT=CloudPlugin here]. If you have any issues, create a [http://trac-hacks.org/newticket?component=CloudPlugin&owner=robguttman new ticket]. == Download == Download the zipped source from [download:cloudplugin here]. == Source == You can check out CloudPlugin from [http://trac-hacks.org/svn/cloudplugin here] using Subversion, or [source:cloudplugin browse the source] with Trac. == Example == Each supported AWS resource is configured independently in a separate section in {{{trac.ini}}}. (The plugin currently only supports ec2 instances.) Each section must be named as follows: {{{ [cloud.] ... }}} For example, a section named {{{[cloud.instance]}}} defines a resource named {{{instance}}}. The resource name is used in the uri, so for example, this {{{instance}}} resource would be accessed at: {{{ http://trac.domain.com/cloud/instance }}} You can name resources whatever you want as long as they're unique. === Instances === Here's how to configure ec2 instance resources. It's best to explain the configuration by example and piecemeal: {{{ [cloud.instance] class = Ec2Instance title = Instances order = 1 label = Instance description = AWS EC2 instances. }}} * {{{class}}} - must map to the exact python class name for that instance. For ec2 instances, it's {{{Ec2Instance}}}. * {{{title}}} - used in the sub-navigation (aka contextual) menu - see screenshot above. * {{{order}}} - defines the order in the contextual menu (much like custom field ordering). * {{{label}}} - used in several views to describe a single one of these resources. * {{{description}}} - used in the grid view - see screenshot above. Now it gets a bit more intense (the following should really all be on one line in {{{trac.ini}}}): {{{ [cloud.instance] .. fields = name < NameHandler, ec2.instance_id as Instance ID, run_list_ as Roles < RunListHandler, created_by as Created By < AuthorHandler, created_at_ as Created At < EpochHandler, ohai_time_ as Last Check-in < AgoEpochHandler, ec2.instance_type as Instance Type, ec2.hostname as Private Hostname, ec2.public_hostname_ as Public Hostname < SshHandler, ec2.placement_availability_zone as Availability Zone, ec2.ami_id as Image ID, supervisord.ssl_port_ as supervisord < HttpsHandler }}} Whoa! The {{{fields}}} option is a list of those chef attributes (aka fields) where you want to provide either a display name, a format handler, or both. So for example, {{{ec2.instance_type as Instance Type}}} above will use 'Instance Type' wherever the name of the {{{ec2.instance_type}}} field is to be displayed. Only slightly more complicated are handlers to format fields. So for example, {{{ohai_time_ as Last Check-in < AgoEpochHandler}}} has a display name as we've seen plus the {{{< AgoEpochHandler}}} directs the plugin to use a specific handler to format the {{{ohai_time}}} field's value. In this example, the {{{AgoEpochHandler}}} handler converts an epoch to a string format such as "0:08:36 ago" meaning "8 minutes and 36 seconds ago" as shown in the screenshot above. But wait, what's up with the underscore after the {{{ohai_time_}}}? In most cases when using a handler, you should postfix the field's name with an underscore ({{{_}}}) so that the re-formatted value doesn't get written back to the field's value. ''[I'm likely to get rid of the underscore business to make it simpler, but that's what it is for now. - rhg]'' See the {{{handlers.py}}} file for a full list of provided field handlers. {{{ grid_index = node grid_columns = name, run_list_, created_by, ohai_time_, ec2.instance_type, ec2.placement_availability_zone grid_sort = created_at_ grid_asc = 0 }}} The {{{grid_*}}} options configure the grid view of the resource - i.e., the view shown in the screenshot above: * {{{grid_index}}} - must map to the chef search index name. For ec2 instances, this is {{{node}}}. * {{{grid_columns}}} - the list of chef attributes to display in order. The column names will use any display names from {{{fields}}} and its cell values will be formatted using any handlers from {{{fields}}}. * {{{grid_sort}}} - the default sort attribute/field. You can resort the grid by clicking on the column name. * {{{grid_asc}}} - the default sort direction. You can change the sort direction by clicking on the same column name. {{{ crud_resource = nodes crud_view = name, ec2.instance_id, run_list_, created_by, created_at_, ohai_time_, ec2.instance_type, ec2.hostname, ec2.public_hostname_, ec2.placement_availability_zone, ec2.ami_id, supervisord.ssl_port_ crud_new = run_list_||, created_by, ec2.instance_type|, ec2.ami_id|, ec2.placement_availability_zone|us-east-1a|us-east-1b|us-east-1c|us-east-1d crud_edit = run_list_||, created_by, ec2.instance_type*, ec2.ami_id*, ec2.placement_availability_zone* }}} The {{{crud_*}}} options configure aspects for creating, reading updating, and deleting (i.e., CRUD) the resource: * {{{crud_resource}}} - must map to the chef resource name. For ec2 instances, this is {{{nodes}}}. * {{{crud_view}}} - the list of chef attributes to display in order. The field names will use any display names from {{{fields}}} and its cell values will be formatted using any handlers from {{{fields}}}. * {{{crud_new}}} and {{{crud_edit}}} - the list of augmented chef attributes to display for editing in order. The field names will use any display names from {{{fields}}} and its cell values will be formatted using any handlers from {{{fields}}}. The field names in {{{crud_new}}} and {{{crud_edit}}} are augmented to provide additional metadata so the plugin knows how to properly configure the form. So for example: {{{ ec2.placement_availability_zone|us-east-1a|us-east-1b|us-east-1c|us-east-1d }}} The pipe ({{{|}}}) after the field name indicates that the form should use a select/dropdown menu for this field. If the select/dropdown should support multiple selections, then use two pipes instead of one as in {{{run_list_||}}}. The remainder of the definition is a pipe-delimited list of options for this field (much like how custom field options are defined). If there's no list of options, then the plugin will search a chefserver data bag of the same name as the field for the list of options. So for example, {{{ec2.instance_type|}}} will direct the plugin to search for a data bag on the chefserver named {{{ec2_instance_type}}} (periods are invalid and so are replaced with underscores). The individual data bag items should be formatted like this: {{{ { "id": "m1_large", "value": "m1.large", "name": "m1.large - Large (64-bit)", "order": 3 } }}} The {{{name}}} will be used as the option's displayed name and the {{{value}}} will be used as the option's value. If {{{order}}} fields are provided, then they'll be used to order the options accordingly (much like custom field ordering). Lastly, if you want to have a field displayed but as read-only, then the field name should be immediately postfixed with an asterisk as in {{{ec2.ami_id*}}} above. == Recent Changes == [[ChangeLog(cloudplugin, 3)]] == Author/Contributors == '''Author:''' [wiki:robguttman] [[BR]] '''Maintainer:''' [wiki:robguttman] [[BR]] '''Contributors:'''