Changes between Version 23 and Version 24 of CloudPlugin


Ignore:
Timestamp:
Feb 4, 2011, 5:51:07 PM (13 years ago)
Author:
Rob Guttman
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • CloudPlugin

    v23 v24  
    33= Orchestrates AWS cloud resources using boto and pychef =
    44
    5 == '''WARNING!  This plugin is in alpha.''' ==
    6 I recommend you ''not'' use this plugin yet on production instances.  Although you likely won't lose ec2 instances or chef data, it's still quite immature and the ''config interface is likely to change''.  It would be best to try out on dev instances to help test the plugin and generate ideas for new features.  I'm actively putting it through its paces now at [http://www.juju.com Juju].
     5== '''WARNING!  This plugin is still in alpha.''' ==
     6I recommend you ''not'' use this plugin yet on production instances.  Although you likely won't lose ec2 instances or chef data, it's still quite immature.  It would be best to try out on dev instances to help test the plugin and generate ideas for new features.  I'm actively putting it through its paces now at [http://www.juju.com Juju].
    77
    8 - robguttman, 2011 Jan 28
     8- robguttman, 2011 Feb 4
    99
    1010
     
    1313
    1414 * Launch an ec2 instance and apply one or more chef roles
    15  * Terminate ec2 instances and delete its chef node
     15 * Terminate ec2 instances and delete its chef node (and client)
    1616
    1717[[Image(instance-grid.png)]]
    1818
    1919There are additional features and quite possibly more to come, but my intention is to keep the scope down to what's needed to orchestrate between AWS and Chef (and not re-implement either the AWS Management Console or the Chef Server webui).
     20
     21This plugin is highly configurable but is opinionated in some ways.  Some of these opinions can be relaxed (submit a ticket!) but they currently are:
     22
     23 * Uses [http://wiki.opscode.com/display/chef/Knife+Bootstrap knife bootstrap] to bootstrap instances - and so your instances' OS must support this method.
     24 * Assumes the same credentials and security (e.g., keypair) is used for all instances.
     25 * Allows one or more roles (but no recipes) to define a node's run_list.
    2026
    2127This plugin relies heavily upon [http://boto.cloudhackers.com/ boto] and [https://github.com/coderanger/pychef pychef].  Special thanks to coderanger for his work and responsiveness on pychef.
     
    4046    You can alternatively use the Trac Web Admin GUI to enable any or all rules.
    4147
    42  3. Configure the main {{{trac.ini}}} section:
     48 3. Configure the main {{{trac.ini}}} section.  A minimal configuration would look like this:
    4349    {{{
    4450    [cloud]
    45     label = Cloud
    4651    aws_key = <aws-access-key>
    4752    aws_secret = <aws-secret-key>
    48     aws_keypair = <ec2-key-pair>
    49     chef_instancedata_file = /path/to/instance-data.json
     53    aws_keypair = <ec2-keypair-name>
     54    aws_keypair_pem = /path/to/keypair.pem
     55    aws_username = <ec2-username>
     56    chef_base_path = /path/to/.chef
    5057    }}}
    5158
    52 The {{{label}}} option above is the name to be used in the top navigation - see screenshot above.
     59=== AWS access configuration ===
     60The {{{aws_*}}} options (see example above) are for your AWS credentials and security information.  For more info, see [http://docs.amazonwebservices.com/AWSSimpleQueueService/2007-05-01/SQSDeveloperGuide/AWSCredentials.html here] and [http://docs.amazonwebservices.com/AWSSecurityCredentials/1.0/AboutAWSCredentials.html here].
    5361
    54 === AWS credentials ===
    55 The {{{aws_*}}} options are your AWS credentials and the key pair used by your ec2 instances.
     62=== Chef access configuration ===
     63Your Trac server must already have been configured using {{{knife configure}}}.  For more info see [http://wiki.opscode.com/display/chef/Bootstrap+Chef+RubyGems+Installation#BootstrapChefRubyGemsInstallation-CreateaKnifeClientforYourLaptop%2FDesktop here].  Set the {{{chef_base_path}}} option to the standard chef directory that contains the {{{knife.rb}}} and client pem files needed to connect with the chef server.  Common settings for this option are {{{/etc/chef}}} or {{{/home/<username>/.chef}}}.  Be sure that the {{{validation.pem}}} file is located where the {{{knife.rb}}} file says it should be.
    5664
    57 === Chef configuration ===
    58 The {{{chef_instancedata_file}}} option is the absolute path to the ec2 instance/user data generated by chef's knife command:
    59 {{{
    60 % knife ec2 instance data role[<desired-role>] > /path/to/instance-data.json
    61 }}}
    62 
    63 The plugin will use this file for the instance data replacing the role information as needed.  As the file contains the {{{validation.pem}}} data, be sure to secure it accordingly.  In cases where you need different credentials and/or chefserver url, you can use the following option to specify a base directory which must contain the typical {{{.chef}}} directory and its contents plus (optionally) a {{{validation.pem}}} file within the {{{.chef}}} directory:
    64     {{{
    65     [cloud]
    66     ..
    67     chef_base_path = /path/to/dir/that/contains/.chef
    68     }}}
    69 
    70 It's currently assumed that you're using one of Opscode's AMIs or a homegrown AMI derived from it.  These AMIs use the instance's id as the node name.  However, you may have recipes which for one reason or another change the node's name after the first chef-client run.  If so, you need to specify the boto field that maps to the final node name using this config option:
    71     {{{
    72     [cloud]
    73     ..
    74     boto_field_node_name = private_dns_name
    75     }}}
    76 
    77 Adding this option will cause the plugin to create a new node of that name upon launching a new ec2 instance and once it checks in will remove the original node.  Otherwise you'll have two chef nodes for the same ec2 instance - something you likely don't want.  Yeah, this is a bit complicated - an area ripe for improvement.
    78 
    79 The plugin is intended to support multiple AWS resource types in addition to instances as needed for orchestration with chef.  You specify the default resource to view using this config option:
    80     {{{
    81     [cloud]
    82     ..
    83     default_resource = instance
    84     }}}
    85 
    86 Each AWS resource type such as {{{instance}}} above is configured independently.  See the examples section [wiki:CloudPlugin#Examples below] for how to specify these resources.
    87  
    8865
    8966== Bugs/Feature Requests ==
     
    10380You can check out CloudPlugin from [http://trac-hacks.org/svn/cloudplugin here] using Subversion, or [source:cloudplugin browse the source] with Trac.
    10481
    105 == Example ==
    106 Each supported AWS resource is configured independently in a separate section in {{{trac.ini}}}.  (The plugin currently only supports ec2 instances.)  Each section must be named as follows:
     82== Examples ==
     83This plugin is designed to support multiple AWS resource types, the most notable being ec2 instances.
     84
     85=== Instances ===
     86The configuration for the ec2 instance resource type has reasonable defaults all of which can be overridden in its own {{{trac.ini}}} section:
     87
    10788{{{
    108 [cloud.<resource_name>]
     89[cloud.instance]
    10990...
    11091}}}
    11192
    112 For example, a section named {{{[cloud.instance]}}} defines a resource named {{{instance}}}.  The resource name is used in the uri, so for example, this {{{instance}}} resource would be accessed at:
    113 {{{
    114 http://trac.domain.com/cloud/instance
    115 }}}
     93The different configuration options are broken out below.
    11694
    117 You can name resources whatever you want as long as they're unique.
    118 
    119 === Instances ===
    120 Here's how to configure ec2 instance resources.  It's best to explain the configuration by example and piecemeal:
    121 
     95==== Labels and such ====
    12296{{{
    12397[cloud.instance]
    124 class = Ec2Instance
    12598title = Instances
    12699order = 1
     
    129102}}}
    130103
    131  * {{{class}}} - must map to the exact python class name for that instance.  For ec2 instances, it's {{{Ec2Instance}}}.
    132  * {{{title}}} - used in the sub-navigation (aka contextual) menu - see screenshot above.
    133  * {{{order}}} - defines the order in the contextual menu (much like custom field ordering).
    134  * {{{label}}} - used in several views to describe a single one of these resources.
    135  * {{{description}}} - used in the grid view - see screenshot above.
     104The {{{title}}} and {{{order}}} options are used for contextual navigation - see screenshot above.  The {{{order}}} should start with {{{1}}}.  If {{{order}}} is set to an empty string then the resource will not be shown in the contextual menu (but is still accessible by its url).  The {{{label}}} value is displayed in various buttons and forms for the name of a single resource item.  The {{{description}}} option is displayed in the grid view - see screenshot above - much like a Trac report description.
    136105
    137 Now it gets a bit more intense (the following should really all be on one line in {{{trac.ini}}}):
    138 
     106==== Field Definitions ====
     107The fields shown in each view can be configured.  Fields are first defined much like Trac custom fields but prefixed with {{{field.}}}:
    139108{{{
    140109[cloud.instance]
    141 ..
    142 fields = name < NameHandler, ec2.instance_id as Instance ID, run_list_ as Roles < RunListHandler, created_by as Created By < AuthorHandler,
    143   created_at_ as Created At < EpochHandler, ohai_time_ as Last Check-in < AgoEpochHandler, ec2.instance_type as Instance Type,
    144   ec2.hostname as Private Hostname, ec2.public_hostname_ as Public Hostname < SshHandler, ec2.placement_availability_zone as Availability Zone,
    145   ec2.ami_id as Image ID, supervisord.ssl_port_ as supervisord < HttpsHandler
     110 ..
     111field.name = text
     112field.name.label = Name
     113field.name.handler = NameHandler
     114field.run_list = multiselect
     115field.run_list.label = Roles
     116field.run_list.databag = roles  # special token
     117field.run_list.handler = RunListHandler
     118field.ec2.placement_availability_zone = select
     119field.ec2.placement_availability_zone.label = Availability Zone
     120field.ec2.placement_availability_zone.options = No preference|us-east-1a|us-east-1b|us-east-1c|us-east-1d
     121 .. many more fields ..
    146122}}}
    147123
    148 Whoa!  The {{{fields}}} option is a list of those chef attributes (aka fields) where you want to provide either a display name, a format handler, or both.  So for example, {{{ec2.instance_type as Instance Type}}} above will use 'Instance Type' wherever the name of the {{{ec2.instance_type}}} field is to be displayed.  This {{{as}}} notation should be familiar to those who have written Trac report SQL (although no quotes should be used here).
     124Each {{{field.*}}} option must map exactly to its chef [http://wiki.opscode.com/display/chef/Attributes attribute] name.  For example, {{{ec2.placement_availability_zone}}} above maps to the {{{ec2}}} attribute namespace and the {{{placement_availability_zone}}} attribute within that namespace in chef.  It's defined as a {{{select}}} field with a display label and several options separated by a pipe ({{{|}}}) just like Trac custom fields.  There are some important differences to Trac custom fields, however:
    149125
    150 Only slightly more complicated are handlers to format fields.  So for example, {{{ohai_time_ as Last Check-in < AgoEpochHandler}}} has a display name as we've seen plus the {{{< AgoEpochHandler}}} directs the plugin to use a specific handler to format the {{{ohai_time}}} field's value.  In this example, the {{{AgoEpochHandler}}} handler converts an epoch to a string format such as "0:08:36 ago" meaning "8 minutes and 36 seconds ago" as shown in the screenshot above.  But wait, what's up with the underscore after the {{{ohai_time_}}}?  In most cases when using a handler, you should postfix the field's name with an underscore ({{{_}}}) so that the re-formatted value doesn't get written back to the field's value.  ''[I'm likely to get rid of the underscore business to make it simpler, but that's what it is for now. - rhg]''  See the {{{handlers.py}}} file for a full list of provided field handlers.
     126 * A field can define a data bag instead of options
     127 * A field can define a "field handler"
    151128
     129===== Data Bags for options =====
     130You may prefer to define a select (or multiselect) field's options by using a chef [http://wiki.opscode.com/display/chef/Data+Bags data bag] instead of listing them in the trac.ini file.  For example, a field defined like this:
    152131{{{
    153 grid_index = node
    154 grid_columns = name, run_list_, created_by, ohai_time_, ec2.instance_type, ec2.placement_availability_zone
    155 grid_sort = created_at_
     132[cloud.instance]
     133 ..
     134field.ec2.ami_id = select
     135field.ec2.ami_id.label = Image ID
     136field.ec2.ami_id.databag = ec2_ami_id
     137}}}
     138
     139will query chef for and use the {{{ec2_ami_id}}} data bag items.  The individual data bag items should be formatted like this:
     140{{{
     141{
     142  "id": "ami-xxxxxxxx",
     143  "value": "ami-xxxxxxxx",
     144  "name": "ami-xxxxxxxx (64-bit)",
     145  "order": 1
     146}
     147}}}
     148
     149The {{{name}}} will be used as the option's displayed name and the {{{value}}} will be used as the option's value.  If an {{{order}}} field is provided, it will be used to order the options accordingly (much like custom field ordering).
     150
     151Specifying {{{roles}}} as a data bag has a special meaning (see the {{{run_list}}} field example above) - chef will be queried for the list of [http://wiki.opscode.com/display/chef/Roles roles] instead of a data bag.
     152
     153===== Field handlers =====
     154A field handler converts a raw chef attribute value into something more human-readable and vice-versa.  For example, the {{{RunListHandler}}} defined above for the {{{run_list}}} field extracts roles from a chef node's run_list and displays them in an easy-to-read way.  It also converts data the other direction - i.e., it converts the list of roles from a web form into the correct format to set the node's {{{run_list}}}.
     155
     156Other field handler examples include displaying a url to a web site at a specific port, converting epoch's to dates and times, etc.  See [browser:cloudplugin/0.12/cloud/handlers.py handlers.py] for the full list of provided field handlers and [browser:cloudplugin/0.12/cloud/defaults.py defaults.py] for examples of which fields they're used for by default
     157
     158==== Views ====
     159In addition to field definitions, you can define which fields should be viewed in which views, their order, and whether or not a field should be read-only in that view.  Example:
     160{{{
     161[cloud.instance]
     162 ..
     163crud_view = name, ec2.instance_type, ohai_time, ..
     164crud_new = roles, created_by, created_at, ..
     165crud_edit = roles, created_by, created_at*, ..
     166
     167grid_columns = name, ec2.instance_type, ohai_time, ..
     168grid_sort = ohai_time
    156169grid_asc = 0
    157170}}}
    158171
    159 The {{{grid_*}}} options configure the grid view of the resource - i.e., the view shown in the screenshot above:
    160  * {{{grid_index}}} - must map to the chef search index name.  For ec2 instances, this is {{{node}}}.
    161  * {{{grid_columns}}} - the list of chef attributes to display in order.  The column names will use any display names from {{{fields}}} and its cell values will be formatted using any handlers from {{{fields}}}.
    162  * {{{grid_sort}}} - the default sort attribute/field.  You can resort the grid by clicking on the column name.
    163  * {{{grid_asc}}} - the default sort direction.  You can change the sort direction by clicking on the same column name.
     172The {{{crud_view}}}, {{{crud_new}}}, {{{crud_edit}}}, and {{{grid_columns}}} options are lists of field names in the order to be displayed in their respective views.  ("crud" stands for "create, read, update, delete".)  An asterisk ({{{*}}}) appended to a field name (only applicable for {{{crud_new}}} and {{{crud_edit}}}) indicates that the field should be read-only in that view.  All fields in the {{{crud_new}}} and {{{crud_edit}}} views will be used to set or update the chef resource item.
    164173
    165 {{{
    166 crud_resource = nodes
    167 crud_view = name, ec2.instance_id, run_list_, created_by, created_at_, ohai_time_, ec2.instance_type, ec2.hostname, ec2.public_hostname_, ec2.placement_availability_zone, ec2.ami_id, supervisord.ssl_port_
    168 crud_new = run_list_||, created_by, ec2.instance_type|, ec2.ami_id|, ec2.placement_availability_zone|us-east-1a|us-east-1b|us-east-1c|us-east-1d
    169 crud_edit = run_list_||, created_by, ec2.instance_type*, ec2.ami_id*, ec2.placement_availability_zone*
    170 }}}
    171 
    172 The {{{crud_*}}} options configure aspects for creating, reading updating, and deleting (i.e., CRUD) the resource:
    173  * {{{crud_resource}}} - must map to the chef resource name.  For ec2 instances, this is {{{nodes}}}.
    174  * {{{crud_view}}} - the list of chef attributes to display in order.  The field names will use any display names from {{{fields}}} and its cell values will be formatted using any handlers from {{{fields}}}.
    175  * {{{crud_new}}} and {{{crud_edit}}} - the list of augmented chef attributes to display for editing in order.  The field names will use any display names from {{{fields}}} and its cell values will be formatted using any handlers from {{{fields}}}.
    176 
    177 The field names in {{{crud_new}}} and {{{crud_edit}}} are augmented to provide additional metadata so the plugin knows how to properly configure the form.  So for example:
    178 {{{
    179 ec2.placement_availability_zone|us-east-1a|us-east-1b|us-east-1c|us-east-1d
    180 }}}
    181 
    182 The pipe ({{{|}}}) after the field name indicates that the form should use a select/dropdown menu for this field.  If the select/dropdown should support multiple selections, then use two pipes instead of one as in {{{run_list_||}}}.  The remainder of the definition is a pipe-delimited list of options for this field (much like how custom field options are defined).  If there's no list of options, then the plugin will search a chefserver data bag of the same name as the field for the list of options.  So for example, {{{ec2.instance_type|}}} will direct the plugin to search for a data bag on the chefserver named {{{ec2_instance_type}}} (periods are invalid and so are replaced with underscores).  The individual data bag items should be formatted like this:
    183 {{{
    184 {
    185   "id": "m1_large",
    186   "value": "m1.large",
    187   "name": "m1.large - Large (64-bit)",
    188   "order": 3
    189 }
    190 }}}
    191 
    192 The {{{name}}} will be used as the option's displayed name and the {{{value}}} will be used as the option's value.  If {{{order}}} fields are provided, then they'll be used to order the options accordingly (much like custom field ordering).
    193 
    194 Lastly, if you want to have a field displayed but as read-only, then the field name should be immediately postfixed with an asterisk as in {{{ec2.ami_id*}}} above.  The {{{crud_new}}} config example above results in the following screenshot:
    195 [[Image(instance-new.png)]]
    196 
    197 Any regular (i.e., non-handled) fields in the {{{crud_new}}} and {{{crud_edit}}} lists will get added as attributes to the chef node such as {{{created_by}}} above.
     174The {{{grid_sort}}} and {{{grid_asc}}} are for default sorting in the grid view.
    198175
    199176