All notes
Chef

Development Kit



# https://docs.chef.io/quick_start.html
# Download the Chef development kit: https://downloads.chef.io/chefdk/.

# Generate a cookbook:
chef generate app first_cookbook
# Change into the first_cookbook directory.

# Update the cookbooks/recipes/default.rb recipe in the generated cookbook to contain:
# file "#{ENV['HOME']}/test.txt" do
#   content 'This file was created by Chef!'
# end

chef-client --local-mode --override-runlist first_cookbook
# This will create a file named test.txt at the home path on your machine.

Configuration

docs.chef.io: config.rb.

Default location: "~/.chef/config.rb".

.d directories may exist in any location where the "client.rb", "config.rb", or "solo.rb" files are present, such as:

/etc/chef/client.d
/etc/chef/config.d
~/chef/solo.d

# Run the chef-client in local mode.
local_mode true

# The name of the node. This may be a username with permission to authenticate to the Chef server or it may be the name of the machine from which knife is run. For example:
node_name 'user_name'
# or:
node_name 'machine_name'

no_proxy 'localhost, 10.0.1.35, *.example.com, *.dev.example.com'

Commands

chef

docs.chef.io: ctl chef.


chef env

# Run commands with the PATH environment variable and the GEM_HOME and GEM_PATH
chef exec SUBCOMMAND

########## gem
chef gem GEM_COMMAND GEM_OPTIONS

# List all of the installed gems
chef gem list
# Show a gem
chef gem list chef-dk

# Search for locally installed gems
chef gem list knife
# Search for remote gems available for installation:
chef gem search kitchen

chef gem install knife-config
chef gem uninstall knife-config

# View the contents of a gem
chef gem content knife-config

########## generate

chef generate app APP_NAME
chef generate attribute COOKBOOK_PATH NAME
chef generate cookbook COOKBOOK_PATH/COOKBOOK_NAME
# Generate a file in the /files directory
chef generate file COOKBOOK_PATH NAME
# Generate a lightweight resource and provider in the /resources and /providers directory.
chef generate lwrp COOKBOOK_PATH NAME
chef generate recipe COOKBOOK_PATH NAME
chef generate repo REPO_NAME
chef generate template COOKBOOK_PATH NAME

chef-client

docs.chef.io: ctl chef_client.


# -c CONFIG, --config CONFIG

# -o RUN_LIST_ITEM, --override-runlist RUN_LIST_ITEM

# --once (chef-client only). Run the chef-client only once and cancel interval and splay options.

# --recipe-url=RECIPE_URL
# -r RECIPE_URL (chef-solo only), --recipe-url RECIPE_URL. The URL location from which a remote cookbook tar.gz is to be downloaded.
# -r RUN_LIST_ITEM (chef-client only), --runlist RUN_LIST_ITEM. Permanently replace the current run-list with the specified run-list items.

# -i SECONDS, --interval SECONDS. The frequency (in seconds) at which the chef-client runs. Default value: 1800 (30 min).
# -s SECONDS, --splay SECONDS. A random number between zero and splay that is added to interval. Use splay to help balance the load on the Chef server by ensuring that many chef-client runs are not occuring at the same interval.

# -z, --local-mode. (chef-client only). See the setion "local mode" below.

# -j PATH, --json-attributes PATH. Used to setup the first client run. For all the future runs with option -i the attributes are expected to be persisted in the chef-server.
# "_default" is the name of the environment that is assigned to the node.
chef-client -j /etc/chef/file.json --environment _default
# file.json:
# {
#   "resolver": {
#     "nameservers": [ "10.0.0.1" ],
#     "search":"int.example.com"
#   },
#   "run_list": [ "recipe[resolver]" ]
# }

# -W, --why-run. Run the executable in why-run mode (similar to dry-run in other softwares).

# chef-client only:
# -k KEY_FILE, --client_key KEY_FILE. Default value: /etc/chef/client.pem.
# -K KEY_FILE, --validation_key KEY_FILE. The location of the file that contains the key used when a chef-client is registered with a Chef server. Default value: /etc/chef/validation.pem.

# -o RunlistItem,RunlistItem...,   Replace current run list with specified items
chef-solo -c solo.rb -o "recipe[myProj::development]" -j custom-attributes.json -l debug

-d, --daemonize                  Daemonize the process. (chef-solo only).
-E, --environment ENVIRONMENT    Set the Chef Environment on the node

-g, --group GROUP                Group to set privilege to
-u, --user USER                  User to set privilege to

-l, --log_level LEVEL            Set the log level (debug, info, warn, error, fatal)
-L, --logfile LOGLOCATION        Set the log file location, defaults to STDOUT.

-N, --node-name NODE_NAME        The node name for this client

Runlists

docs.chef.io: run_lists.


'recipe[COOKBOOK::RECIPE],COOKBOOK::RECIPE,role[NAME]'

Local mode

This allows all commands that work against the Chef server to also work against the local chef-repo, by means of chef-zero.

Local mode will store temporary and cache files under the "chef_repo_path/.cache" directory by default. This allows a normal user to run the chef-client in local mode without requiring root access.

chef-zero

chef-zero is a very lightweight Chef server that runs in-memory on the local machine. chef-zero is very useful for quickly testing and validating the behavior of the chef-client, cookbooks, recipes, and run-lists before uploading that data to the actual Chef server.

Config file: "/etc/chef/client.rb". On Microsoft Windows "C:\chef\client.rb". Use "--config" to override.

chef-solo

docs.chef.io: ctl_chef_solo.

Similar to "chef-client" command options. See its section.

Config file: "/etc/chef/solo.rb". Or use the "--config" to override.

chef-shell

docs.chef.io: chef shell.

chef-shell is a recipe debugging tool that allows the use of breakpoints within recipes.


# -s, --solo
# -z, --client
# -c CONFIG
# -S SERVER_URL
chef-shell

Configuration

If chef-shell is started using production as the named configuration, the chef-shell will load a configuration file from ~/.chef/production/chef_shell.rb
If a named configuration is not provided, chef-shell will attempt to load the chef-shell.rb file from ~/.chef/chef_shell.rb
If a chef-shell.rb file is not found, chef-shell will attempt to load the client.rb / solo.rb.

knife



Quick start

knife

"knife" is a command-line tool that provides an interface between a local chef-repo and the Chef server.

Librarian-Chef

github.com: librarian-chef.

Librarian-Chef is a bundler for your Chef-based infrastructure repositories. You can use it to resolve your infrastructure's cookbook dependencies, fetch them, and install them into your infrastructure repository.

Cheffile



site "https://supermarket.getchef.com/api/v1"

cookbook "ntp"
cookbook "timezone", "0.0.1"

cookbook "rvm",
  :git => "https://github.com/fnichol/chef-rvm",
  :ref => "v0.7.1" # The specified Git tag. Actually we can use anything that Git will recognize as a ref. This includes any branch name, tag name, SHA, or SHA unique prefix.
  :path => "pathA/pathB" # Only use the specified subdirectory. Good for a single repository with many cookbooks in it.

cookbook "cloudera",
  :path => "vendor/cookbooks/cloudera-cookbook"

Command line



########## Install

gem install librarian-chef

cd ~/path/to/chef-repo

# Librarian-Chef will always reinstall the cookbooks listed the Cheffile.lock into ./cookbooks/.
git rm -r cookbooks
echo /cookbooks >> .gitignore
# The ./tmp/ directory is used for tempfiles and caches.
echo /tmp >> .gitignore

# Make a Cheffile:
librarian-chef init

librarian-chef install [--clean] [--verbose] # It also writes the complete resolution into Cheffile.lock.

# Get an overview of your Cheffile.lock with:
librarian-chef show

########## Update

# Find out which dependencies are outdated and may be updated:
librarian-chef outdated [--verbose]
# Update the version of a dependency:
librarian-chef update ntp timezone monit [--verbose]

########## Upload

# Upload the cookbooks to your chef-server:
knife cookbook upload --all

########## Configuration

# The local config (./.librarian/chef/config)
# The global config (~/.librarian/chef/config)

librarian-chef config

librarian-chef config KEY VALUE --global/local
librarian-chef config KEY --global/local --delete

Ohai

Ohai is a tool that is used to collect system configuration data, which is provided to the chef-client for use within cookbooks. docs.chef.io: ohai.

The types of attributes Ohai collects include but are not limited to: Operating System Network Memory Disk CPU Kernel

docs.chef.io: automatic attributes by Ohai:

node['platform']	The platform on which a node is running. This attribute helps determine which providers will be used.
node['platform_version']	The version of the platform. This attribute helps determine which providers will be used.
node['ipaddress']	The IP address for a node. If the node has a default route, this is the IPV4 address for the interface. If the node does not have a default route, the value for this attribute should be nil. The IP address for default route is the recommended default value.
node['macaddress']	The MAC address for a node, determined by the same interface that detects the node['ipaddress'].
node['fqdn']	The fully qualified domain name for a node. This is used as the name of a node unless otherwise set.
node['hostname']	The host name for the node.
node['domain']	The domain for the node.
node['recipes']	A list of recipes associated with a node (and part of that node’s run-list).
node['roles']	A list of roles associated with a node (and part of that node’s run-list).
node['ohai_time']	The time at which Ohai was last run. This attribute is not commonly used in recipes, but it is saved to the Chef server and can be accessed using the knife status subcommand.


Chef::Log.debug "Current node IP: #{node['ipaddress']}, Hostname: #{node['hostname']}, Domain: #{node['domain']}"
# Current node IP: 8.8.8.8, Hostname: me-1234, Domain: wangchaofeng.com
# Hostname+Domain: me-1234.wangchaofeng.com

Cookbooks

Structure

chef generate cookbook chefdocs:

/chefdocs
  /.git
  .gitignore
  .kitchen.yml
  Berksfile
  chefignore
  metadata.rb
  README.md
  /recipes
    default.rb

metadata.rb

docs.chef.io: config rb metadata.

A metadata.json file will be compiled from metadata.rb. The json can be edited, should temporary changes be required. Any permanent changes to cookbook metadata should be done in the metadata.rb.

Pessimistic version constraint

If we’re on version 2.2.3 of a cookbook, we know that the API will be stable until the 3.0.0 release. Using traditional operators, we’d write this as >= 2.2.0, < 3.0. Instead, we can write this by combining a tilde “~” and right angle bracket “>” followed by the major and minor version numbers. For example: ~> 2.2  (Approximately greater than)

To match any 12.x version of the chef-client, but not 11.x or 13.x:
chef_version '~> 12'

Example metadata.rb



description 'A fancy cookbook that manages a herd of cats!'
long_description <<-EOH
  MARKDOWN is supported here.
EOH

maintainer 'Adam Jacob'
maintainer_email '[email protected]'

# Used by Chef Supermarket.
issues_url 'https://github.com/chef-cookbooks/chef-client/issues'
source_url 'https://github.com/chef-cookbooks/chef-client'

license 'Apache v2.0'
license 'GPL v3'
license 'MIT'
license 'Proprietary - All Rights Reserved'

# Name of the cookbook.
name 'cats'

version '2.0.0'

supports 'ubuntu', '>= 12.04'

depends 'cats', '< 1.0'

ohai_version "~> 8"

# Specifies a gem dependency to be installed via the chef_gem resource after all cookbooks are synchronized
gem "poise"
gem "chef-sugar"
gem "chef-provisioning"

# Add a recipe, definition, or resource that is provided by this cookbook, should the auto-populated list be insufficient.
# For recipes:
provides 'cats::sleep'
provides 'cats::eat'
# For definitions:
provides 'here(:kitty, :time_to_eat)'
# For resources:
provides 'service[snuggle]'

Attributes

docs.chef.io: attributes.

Types

Six types (lower precedence first): default, force_default, normal, override, force_override, automatic (from Ohai and un-modifiable).

Five locations (lower precedence first): Nodes (collected by Ohai at the start of each chef-client run), Attribute files (attributes/ in cookbooks), Recipes (in cookbooks), Environments, Roles.

Best practices

SO: using attributes in chef.

In general you should never need to touch automatic, normal, force_default or force_override levels of attributes. You should also avoid setting attributes in recipe code. You should move setting attributes in recipes to attribute files. What this leaves is these places to set attributes:

* in the initial -j argument (sets normal attributes, you should limit using this to setting the run_state, over using this is generally smell)
* in the role file as default or override precedence levels (careful with this one though because roles are not versioned and if you touch these attributes a lot you will cause production issues)
* in the cookbook attributes file as default or override precedence levels (this is where you should set most of your attributes)
* in environment files as default or override precedence levels (can be useful for settings like DNS servers in a datacenter, although you can use roles and/or cookbooks for this as well)

Libraries

"/libraries" is used to store common helper functions, e.g. to be reused across recipes or to make it easier to manage code.

docs.chef.io: libraries.

Recipes

SO: should I use include_recipe or add the recipe to run-list.

Any recipe should be able to run on an empty machine on its own. So if some recipe A depends on recipe B run before it, always use include_recipe.
Q: Does Chef immediately execute a recipe when it readsinclude_recipe foo?
A: No, Chef run consists of 2 stages, first it reads all the recipes and constructs a collection of resources to manage, and only then executes the resources.

Resources

docs.chef.io: resource.


directory '/tmp/folder' do
  owner 'root'
  group 'root'
  mode '0755'
  action :create
end
# The chef-client will look up the provider for the directory resource (Chef::Provider::Directory here), call load_current_resource to create a directory["/tmp/folder"] resource, and then do the specified action (:create).

log

docs.chef.io: resource log.



# Chef::Log.FUN. FUN could be: fatal, error, warn, info, debug.

unless node['splunk']['upgrade_enabled']
  Chef::Log.fatal('The chef-splunk::upgrade recipe was added to the node,')
  Chef::Log.fatal('but the attribute `node["splunk"]["upgrade_enabled"]` was not set.')
  Chef::Log.fatal('I am bailing here so this node does not upgrade.')
  raise
end

# Set default logging level
log 'a string to log'

# Set debug logging level
log 'a debug string' do
  level :debug
end

# Add a message to a log file
log 'message' do
  message 'This is the message that will be added to the log.'
  level :info
end

service

docs.chef.io: resource service.



# Actions: disable, enable, reload, start, restart, stop, nothing (default).

service "tomcat" do
  action :start
end

# Use a pattern
service 'samba' do
  pattern 'smbd'
  action [:enable, :start]
end

# Use the retries and providers common attributes
service 'some_service' do
  provider Chef::Provider::Service::Upstart
  action [ :enable, :start ]
  retries 3
end

#----- supports

service 'clamd' do
  supports status: true, restart: true, reload: true
  action :enable
end

templates

docs.chef.io: resource template.



template '/etc/motd' do
  source 'motd.erb'
  owner 'root'
  group 'root'
  mode '0755'

  # notifies :action, 'resource[name]', :timer
  notifies :reload, 'service[syslog-ng]', :delayed
end

notifies

timer:

:before
  Specifies that the action on a notified resource should be run before processing the resource block in which the notification is located.

:delayed
  Default. Specifies that a notification should be queued up, and then executed at the very end of the chef-client run.

:immediate, :immediately
  Specifies that a notification should be run immediately, per resource notified.

path

The full path to the file, including the file name and its extension. Defaults to 'name' if not specified.

cookbook_file

docs.chef.io: resource cookbook file.



cookbook_file '/var/www/customers/public_html/index.php' do
  source 'index.php'
  owner 'web_admin'
  group 'web_admin'
  mode '0755'
  action :create
end

execute

resource: execute.


execute 'apache_configtest' do
  command '/usr/sbin/apachectl configtest'
end

# command   String, Array # defaults to 'name' if not specified. E.g.:
execute "git init" do
  cwd "#{source_path}/secrets"
  not_if ::File.directory?("#{source_path}/secrets/.git")
end

file



file '/var/www/customers/public_html/index.php' do
  content '<html>This is a placeholder for the home page.</html>'
  mode '0755'
  owner 'web_admin'
  group 'web_admin'
end

link

docs.chef.io: resource link.


link '/tmp/file' do
  to '/etc/file'
  link_type :hard
end

# creates a symbolic link:
link '/tmp/file' do
  to '/etc/file'
end

Resources instead of recipe

Q: About chef’s best practice, “Avoid unnecessary coupling of recipes.“, which means to use include_recipe sparingly. Is it correct or reasonable? Can’t I make it modular by extracting come common steps into separate recipes?

szymon [2:37 PM] 
@owen263 if you have a code which is common and used in various places, best practice is to extract it to resource.

Custom resources

docs.chef.io: custom resources.

A custom resource is defined as a Ruby file and located in a cookbook’s "/resources".


# "name_property: true" allows the value for this property to be infered from the 'name' of the resource block. E.g. the resource file below is named "httpd.rb", so new_resource.instance_name is "httpd".
property :instance_name, String, name_property: true
property :port, Integer, required: true

action :create do
  package 'httpd' do
    action :install
  end

  template "/lib/systemd/system/httpd-#{new_resource.instance_name}.service" do
    source 'httpd.service.erb'
    variables(
      instance_name: new_resource.instance_name
    )
    owner 'root'
    group 'root'
    mode '0644'
    action :create
  end

  template "/etc/httpd/conf/httpd-#{new_resource.instance_name}.conf" do
    source 'httpd.conf.erb'
    variables(
      instance_name: new_resource.instance_name,
      port: new_resource.port
    )
    owner 'root'
    group 'root'
    mode '0644'
    action :create
  end

  directory "/var/www/vhosts/#{new_resource.instance_name}" do
    recursive true
    owner 'root'
    group 'root'
    mode '0755'
    action :create
  end

  service "httpd-#{new_resource.instance_name}" do
    action [:enable, :start]
  end

end

Final Cookbook Directory:

/website
  metadata.rb
  /recipes
    default.rb
  README.md
  /resources
    httpd.rb
  /templates
    httpd.conf.erb
    httpd.service.erb

The custom resource may be used in a recipe.


website_httpd 'httpd_site' do
  port 81
  action :create
end

Concepts

Policy

Roles

docs.chef.io: roles.

Each role consists of zero (or more) attributes and a run-list. Each node can have zero (or more) roles assigned to it.

Get node's current roles


# https://stackoverflow.com/questions/43730503/chef-get-nodes-current-role
# https://docs.chef.io/attributes.html#automatic-ohai
my_roles = node['roles'] #or node[:roles]

How to manage roles

digitalOcean.com.

Related to the idea of a role is the concept of Chef environments. For instance, one environment may be called "testing" and another may be called "production".



# Create a role under ~/chef-repo/roles/web_server.rb

name "web_server"
description "A role to configure our front-line web servers"
run_list "recipe[apt]", "recipe[nginx]"
env_run_lists "production" => ["recipe[nginx::config_prod]"], "testing" => ["recipe[nginx::config_test]"]
default_attributes "nginx" => { "log_location" => "/var/log/nginx.log" }
override_attributes "nginx" => { "gzip" => "on" }

Another method is to use Json instead of Ruby DSL, initial version may be created by knife role create test (two extra pieces of information: json_class and chef_type):


{
  "name": "web_server",
  "description": "A role to configure our front-line web servers",
  "json_class": "Chef::Role",
  "default_attributes": {
    "nginx": {
      "log_location": "/var/log/nginx.log"
    }
  },
  "override_attributes": {
    "nginx": {
      "gzip": "on"
    }
  },
  "chef_type": "role",
  "run_list": [
    "recipe[apt]",
    "recipe[nginx]"
  ],
  "env_run_lists": {
    "production": [
      "recipe[nginx::config_prod]"
    ],
    "testing": [
      "recipe[nginx::config_test]"
    ]
  }
}


##### Upload / Download role from server

# Upload role to server:
knife role from file path/to/role/file

# Show role from server:
knife role show web_server -Fjson > path/to/save/to

##### Edit node's role

knife node list

# And then we would give a command like:
knife node edit node_name
# You can add a role to the run_list
# "run_list": [
#     "role[web_server]"
# ]

knife search "role:database_server AND chef_environment:prod" -a name

environments

digitalOcean.

Create "~/chef-repo/environments/development.rb".



name "development"
description "The master development branch"
cookbook_versions({
    "nginx" => "<= 1.1.0",
    "apt" => "= 0.0.1"
})
override_attributes ({
    "nginx" => {
        "listen" => [ "80", "443" ]
    },
    "mysql" => {
        "root_pass" => "root"
    }
})


knife environment from file ~/chef-repo/environments/development.rb

# Get the environment file off of the server:
knife environment show development -Fjson > ~/chef-repo/environments/development.json

##### Setting Environments in Nodes

# To edit a node called client1
knife node edit client1
# Change "chef_environment"'s value in it.