The Cluster Guy

New Pacemaker Release Series

Thu, 11 Mar 2010 17:55:12 +0100

A number of new branches have been created in the last few days which are integral to how we plan to add new features in a controlled manner.

Current set of branches:

1.0 - The existing stable series
1.1 - The current feature series
1.2 - The next stable series (expected Q4 2010)
devel - Where new features are added

The idea is that 1.0 will continue receive only bugfixes (the amount hopefully continuing to reduce over time) and all new features go first into devel.

1.1 will be the merge point, primarily receiving bug fixes for the new features we pulled in from devel (at least two weeks before a 1.1 release).

The intention is to pursue bi-monthly releases of 1.1 over the next 6-12 months until we can offer a set of compelling new features, a sane way to use them, and rock-solid stability.

At that time, we will freeze development and release it as 1.2.0

But Can I use it in Production?

Cluster admins are naturally a cautious bunch, so in order to make 1.1 more palatable, we have also created two new schemas.

Schemas can be thought of as a contract between the developers and admins. They are a declaration of what we support.

The set is the new set of validation types:

pacemaker-1.0
pacemaker-1.1
pacemaker-1.2

pacemaker-1.0 is (and will remain) identical to what people are using now

When new features are added, their configuration syntax will first appear in pacemaker-1.1. If, based on your feedback, a feature’s syntax needs to be modified we will change it here and solicit further feedback.

Once a feature has been locked down and become stable, its configuration pieces will be added to pacemaker-1.2. Once a feature appears in pacemaker-1.2, we will not change its syntax in any 1.1 or 1.2 release.

What does this mean?

It means that when used with the pacemaker-1.0 or pacemaker-1.2 schemas, 1.1 can be safely used in production in the same manner as 1.0 is today.

If the existing syntax is all you need, consider 1.1 with the pacemaker-1.0 schema.
If you want to try a new stable feature, use 1.1 with the pacemaker-1.2 schema.
If you want to try a new experimental feature, use 1.1 with the pacemaker-1.1 schema.

Is 1.0 Still Supported?

Yes. 1.0 i still supported and will recieve all relevant bug fixes until at least 2012. See our releases page.

How can I Install it?

Users of RPM-based distributions will be able to install in the usual manner. However, to avoid forcing everyone to upgrade, packages will be located under:

http://www.clusterlabs.org/rpm-next

At this time the intention is to only produce packages for the most recent openSUSE (11.2), Fedora (12), and EPEL (5) based distributions. If you would like to try 1.1 but your distribution/version is not listed, please contact the project and we’ll see what we can do.

Packages should be available in the next week “or so”.

Identified Areas of Development

Existing

New stonith daemon (configuration is unchanged), including:
- support for RHCS Stonith Agents
- cluster-wide notifications (that allow the cluster to make more intelligent decisions)
- ability to perform multiple fencing operations in parallel (faster recovery)
Service placement influenced by the physical resources (RAM, CPU, etc) required by the service (and offered by the host)
A new tool for simulating failures and the cluster’s reaction to them
Tell the cluster to serialize an otherwise unrelated a set of resource actions (eg. Xen migrations)

Planned

Failover domains, an easy way to specify ordered preferences for a set of hosts
ACLs, the ability to restrict read/write access to the configuration based on users and groups
Improvements to the System Health feature
Freeze/thaw, an easy way to ensure a consistent backup of data in use by distributed systems.
The ability to monitor services on non-cluster nodes (such as VMs)

Pacemaker removed from OBS

Thu, 04 Feb 2010 11:42:00 +0100

Today I removed Pacemaker from server:ha-clustering on the openSUSE build service.

I lost patience with the service some time ago and the project has been providing pre-built packages from cluster labs ever since (see our install page for more details).

It seems no-one else has had the time or patience to keep the build service updated since my departure so, after noticing their age and the fact that they no longer even build on the majority of targets, I made the decision to remove them.

Hopefully this will help avoid confusing those wanting the latest Pacemaker software.

Pacemaker 1.0.7 Released

Mon, 18 Jan 2010 12:38:47 +0100

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In this release, we’ve made a number improvements to clone handling - particularly the way ordering constraints are processed - as well as some really nice improvements to the shell.

The next 1.0 release is anticipated to be in mid-March. We will be switching to a bi-monthly release schedule to begin focusing on development for the next stable series (more details soon). If you have feature requests, now is the time to voice them and/or provide patches :-)

Pre-built packages for Pacemaker and it’s immediate dependancies are currently building and will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area shortly.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	193
Diff	220 files changed, 15933 insertions(+), 8782 deletions(-)

Changes of note since Pacemaker-1.0.6

High: PE: Bug 2213 - Ensure groups process location constraints so that clone-node-max works for cloned groups
High: PE: Bug lf#2153 - non-clones should not restart when clones stop/start on other nodes
High: PE: Bug lf#2209 - Clone ordering should be able to prevent startup of dependant clones
High: PE: Bug lf#2216 - Correctly identify the state of anonymous clones when deciding when to probe
High: PE: Bug lf#2225 - Operations that require fencing should wait for ‘stonith_complete’ not ‘all_stopped’.
High: PE: Bug lf#2225 - Prevent clone peers from stopping while another is instance is (potentially) being fenced
High: PE: Correctly anti-colocate with a group
High: PE: Correctly unpack ordering constraints for resource sets to avoid graph loops
High: Tools: crm: load help from crm_cli.txt
High: Tools: crm: resource sets (bnc#550923)
High: Tools: crm: support for comments (LF 2221)
High: Tools: crm: support for description attribute in resources/operations (bnc#548690)
High: Tools: hb2openais: add EVMS2 CSM processing (and other changes) (bnc#548093)
High: Tools: hb2openais: do not allow empty rules, clones, or groups (LF 2215)
High: Tools: hb2openais: refuse to convert pure EVMS volumes
High: cib: Ensure the loop for login message terminates
High: cib: Finally fix reliability of receiving large messages over remote plaintext connections
High: cib: Fix remote notifications
High: cib: For remote connections, default to CRM_DAEMON_USER since thats the only one that the cib can validate the password for using PAM
High: cib: Remote plaintext - Retry sending parts of the message that did not fit the first time
High: crmd: Ensure batch-limit is correctly enforced
High: crmd: Ensure we have the latest status after a transition abort
High (bnc#547579,547582): Tools: crm: status section editing support
High: shell: Add allow-migrate as allowed meta-attribute (bnc#539968)
Medium: Build: Do not automatically add -L/lib, it could cause 64-bit arches to break
Medium: PE: Bug lf#2206 - rsc_order constraints always use score at the top level
Medium: PE: Only complain about target-role=master for non m/s resources
Medium: PE: Prevent non-multistate resources from being promoted through target-role
Medium: PE: Provide a default action for resource-set ordering
Medium: PE: Silently fix requires=fencing for stonith resources so that it can be set in op_defaults
Medium: Tools: Bug lf#2286 - Allow the shell to accept template parameters on the command line
Medium: Tools: Bug lf#2307 - Provide a way to determin the nodeid of past cluster members
Medium: Tools: crm: add update method to template apply (LF 2289)
Medium: Tools: crm: direct RA interface for ocf class resource agents (LF 2270)
Medium: Tools: crm: direct RA interface for stonith class resource agents (LF 2270)
Medium: Tools: crm: do not add score which does not exist
Medium: Tools: crm: do not consider warnings as errors (LF 2274)
Medium: Tools: crm: do not remove sets which contain id-ref attribute (LF 2304)
Medium: Tools: crm: drop empty attributes elements
Medium: Tools: crm: exclude locations when testing for pathological constraints (LF 2300)
Medium: Tools: crm: fix exit code on single shot commands
Medium: Tools: crm: fix node delete (LF 2305)
Medium: Tools: crm: implement -F (—force) option
Medium: Tools: crm: rename status to cibstatus (LF 2236)
Medium: Tools: crm: revisit configure commit
Medium: Tools: crm: stay in crm if user specified level only (LF 2286)
Medium: Tools: crm: verify changes on exit from the configure level
Medium: ais: Some clients such as gfs_controld want a cluster name, allow one to be specified in corosync.conf
Medium: cib: Clean up logic for receiving remote messages
Medium: cib: Create valid notification control messages
Medium: cib: Indicate where the remote connection came from
Medium: cib: Send password prompt to stderr so that stdout can be redirected
Medium: cts: Fix rsh handling when stdout is not required
Medium: doc: Fill in the section on removing a node from an AIS-based cluster
Medium: doc: Update the docs to reflect the 0.6/1.0 rolling upgrade problem
Medium: doc: Use Publican for docbook based documentation
Medium: fencing: stonithd: add metadata for stonithd instance attributes (and support in the shell)
Medium: fencing: stonithd: ignore case when comparing host names (LF 2292)
Medium: tools: Make crm_mon functional with remote connections
Medium: xml: Add stopped as a supported role for operations
Medium: xml: Bug bnc#552713 - Treat node unames as text fields not IDs
Medium: xml: Bug lf#2215 - Create an always-true expression for empty rules when upgrading from 0.6

Ubuntu looking for Pacemaker testers

Fri, 15 Jan 2010 18:22:52 +0100

Ubuntu is looking to switch its supported cluster stack to Corosync+Pacemaker and has put out a “Call for testers”.
Check out the link if this is something you’re interested in.

Pre-Announce: End of Pacemaker 0.6 support is near

Tue, 12 Jan 2010 13:18:00 +0100

Unless there are violent objections, I plan to officially stop supporting 0.6 at the end of February.

Since I’ve not seeing any bugs reported for some time, it seems that anyone still using 0.6 is happy with it for their workload.

Also, 1.0 has been out for over a year now and contains significant improvements over 0.6 including

A unified shell that hides the XML scaffolding
Migration thresholds that are easy to configure and understand
Failures can be ignored after a specified period of time
Ability to specify defaults for resource an operation parameters
Man pages for all CLI tools
Up-to-date online documentation

The online documentation has more details on whats new/different in Appendix C and detailed instructions for upgrading in Appendix E.

New Documentation Formats

Mon, 16 Nov 2009 16:56:08 +0100

I’m pleased to report that the core Pacemaker documentation is now available in PDF, HTML (chunked and single page) and even TXT formats.

The old Pages.app sources have been replaced with DocBook which allows them to be:

published in a variety of formats
kept under version control
included in the packages
updated by anybody

Additionally, we’re using Publican to produce the final result so supporting multiple languages should be now possible. Let us know if you’re interested in doing some translation :-)

The primary location for Pacemaker documentation will remain http://www.clusterlabs.org/wiki/Documentation however there is also a index of the generated documentation at http://www.clusterlabs.org/doc/ which includes the date and version from which it was generated.

Pacemaker 1.0.6 Released

Mon, 02 Nov 2009 11:19:53 +0100

The next installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In addition to further polishing of the crm shell and CLI tools, this is the first release to support CoroSync (version 1.1.2 or greater is required).

The ”Pacemaker Explained” reference has also been converted to docbook and is included as part of the tarball (and pre-built packages if the relevant stylesheets are present at build time).

Pre-built packages for Pacemaker and it’s immediate dependancies will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the OpenSUSE Build Service in the next couple of days depending in how overloaded it is.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	185
Diff	331 files changed, 13858 insertions(+), 3277 deletions(-)

Project Administrivia

We may switch to a bi-monthly release cycle. If you have any thoughts on this (for or against), please get in touch.

Changes of note since Pacemaker-1.0.5

High: cib: Correctly clean up when both plaintext and tls remote ports are requested
High: ais: Avoid excessive load by checking for dead children every 1s (instead of 100ms)
High: ais: Bug lf#2199 - Prevent expected-quorum-votes from being populated with garbage
High: ais: Bug rh#525589 - Prevent shutdown deadlocks when running on CoroSync
High: ais: Gracefully handle changes to the AIS nodeid
High: ais: Prevent deadlock - dont try to release IPC message if the connection failed
High: ais: Ubuntu needs a leading zero for directory modes
High: cib: For validation errors, send back the full CIB so the client can display the errors
High: cib: Prevent use-after-free for remote plaintext connections
High: cib: Repair the ability to connect to the cluster from non-cluster machines
High: Core: Bug lf#2169 - Allow dtd/schema validation to be disabled
High: crmd: Bug bnc#527530 - Wait for the transition to complete before leaving S_TRANSITION_ENGINE
High: crmd: Bug lf#2201 - Guard against possible cause of a segfault
High: crmd: Prevent use-after-free with LOG_DEBUG_3
High: Extras: Add sctp support to the controld RA
High: PE: Bug bnc#515172 - Provide better defaults for lt(e) and gt(e) comparisions
High: PE: Bug lf#2106 - Not all anonymous clone children are restarted after configuration change
High: PE: Bug lf#2170 - stop-all-resources option had no effect
High: PE: Bug lf#2171 - Prevent groups from starting if they depend on a complex resource which can’t
High: PE: Bug lf#2197 - Allow master instances placemaker to be influenced by colocation constraints
High: PE: Disable resource management if stonith-enabled=true and no stonith resources are defined
High: PE: Don’t include master score if it would prevent allocation
High: PE: Make sure promote/demote pseudo actions are created correctly
High: PE: Prevent target-role from promoting more than master-max instances
High: shell: Add allow-migrate as allowed meta-attribute (bnc#539968)
High: tools: bnc#547579,547582 - crm: status section editing support
High: Tools: crm: add semantic checks depending on the meta-data from resource agents
High: Tools: crm: improve processing of group edit and constraints
High: Tools: crm: improve the edit command
High: Tools: pingd - Fix a number of critical bugs (patch via Kazunori INOUE)
Med: xml: Mask the “symmetrical” attribute on rsc_colocation constraints (bnc#540672)
Medium (bnc#520707): Tools: crm: new templates ocfs2 and clvm
Medium (LF 2164): Tools: hb_report: expand the crm status command
Medium (LF 2184): Tools: crm: extend ptest command
Medium (LF 2185): Tools: crm: add resource promote/demote commands
Medium (LF 2198): Tools: crm: add node fence command
Medium: ais: Attempt to enable core file generation if it was disabled
Medium: ais: Include version details in plugin name
Medium: Build: Re-enable asciidoc documentation
Medium: Build: Shell templates arent documentation
Medium: cib: Remove delay for remote plaintext connections
Medium: Core: Disable syslog for any process that doesn’t want its arguments logged
Medium: crmd: Requery the resource metadata after every start operation
Medium: cts: add —benchmark for scalability tests
Medium: cts: Prepare for corosync testing
Medium: Extra: Include SNMP MIB file for crm_mon (from Michael Schwartzkopff)
Medium: PE: Bug lf#2178 - Indicate unmanaged clones
Medium: PE: Bug lf#2180 - Include node information for all failed ops
Medium: PE: Bug lf#2189 - Incorrect error message when unpacking simple ordering constraint
Medium: PE: Correctly log resources that would like to start but can’t
Medium: PE: Correctly log the state of orphaned clone instances
Medium: PE: If no migrate_(from|to) action is defined, look for migrate instead
Medium: PE: Only re-instate target-role if it is less than the calculated one
Medium: PE: Provide details for the maintenance-mode option
Medium: PE: Stop ptest from logging to syslog
Medium: Tools: attrd_updater - Suppress all logging with —quiet
Medium: Tools: crm: add extra flag to CibObject for invalid objects
Medium: Tools: crm: do return cached resources dom node
Medium: Tools: crm: expand template documentation
Medium: Tools: crm: first child of a removed parent inherits constraints
Medium: Tools: crm_attribute - Suppress all logging with —quiet
Medium: Tools: crm_shadow - log diffs to stdout instead of stderr
Medium: Tools: Use -q as the short form for —quiet (for consistency)

Advisory: Don't use Pacemaker on Corosync (yet)

Tue, 06 Oct 2009 16:20:00 +0200

I spent some time looking into the state of the Pacemaker/Corosync integration today and I can only recommend Pacemaker users stay on the previous version of OpenAIS (aka. Whitetank).

In a nutshell, shutdown is utterly broken.

r2140 of Corosync removed the shutdown worker thread which allowed plugins such as Pacemaker to continue sending and receiving cluster messages.
Without it, Corosync waits for Pacemaker to finish and Pacemaker waits for the messages it tried to send to arrive and be acted upon. Needless to say no-one makes any progress.

Stay tuned, now that integration testing has started it shouldn’t take too long to get everything sorted out.

Update

Since writing this, the necessary testing has been done and Pacemaker is now supported on Corosync provided you have corosync >= 1.1.2 and pacemaker >= 1.0.6

Clusters From Scratch

Mon, 21 Sep 2009 10:57:27 +0200

The first of a new series of step-by-step guides for Pacemaker.

This installment covers installation, the creation of an active/passive cluster and its conversion to active/active.

Technologies used include:

Fedora 11 as the host operating system
OpenAIS to provide messaging and membership services,
Pacemaker to perform resource management,
DRBD as a cost-effective alternative to shared storage,
OCFS2 as the cluster filesystem (in active/active mode)
The crm shell for displaying the configuration and making changes
Apache as the example service.

The PDF is available from our Documentation page or directly via http://www.clusterlabs.org/mediawiki/images/9/9d/Clusters_from_Scratch_-_Apache_on_Fedora11.pdf

Future guides are anticipated to include MySQL, mail servers and asymmetrical clusters. Feedback and suggestions for additional topics are welcome.

Version Control Prompt

Mon, 21 Sep 2009 09:41:04 +0200

I find it convenient to include current SCM data before my regular Bash prompt (reduces the chance of “accidents”). Perhaps someone else will find it useful too.

function prompt-pre-exec() {
    scm=""
    repo_root=$(hg root 2>/dev/null)
    if [ -e CVS ]; then
        scm=":: cvs ::"

    elif [ -e .svn ]; then
        scm=":: svn : ${prompt_hl}r$(svn info | grep Revision | sed s/.*:\ //)${prompt_n} ($(svn info | grep Date | sed s/.*\(\//)"

    elif [ -e .gitignore ]; then
        repo_branch=`git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/\1#/'`
        scm=`git show --pretty="format: : git : ${prompt_hl}${repo_branch}%h${prompt_n} : %an, %cr\n:: %s\n" | head -n 2`

    elif [ x != "x$repo_root" ]; then
        repo_cs=$(hg id -i)
        scm=`hg log --template " : hg : ${repo_root##*/} : ${prompt_hl}${repo_cs}${prompt_n} {tags} : {author|user}, {date|age} ago\n:: {desc|firstline|strip}\n" -r ${repo_cs%%+}`
    fi

    if [ "x$scm" != x ]; then
        # Trailing \n characters don't seem to expanded 
        scm="$scm
"
    fi
    export scm    
}

if [ x"$-" = "xhimBH" ]; then
  # Execute the following function before displaying the prompt
  export PROMPT_COMMAND='prompt-pre-exec'

  # Use \[ and \] to exclude the color code from the line wrapping calculations 
  export PS1='${scm}[\@] \u@\h \[${prompt_hl}\]\w #\[${prompt_n}\] '
fi

Then to add color, simply define prompt_hl and prompt_n. I use

export prompt_n="^[^E[00m^]"      # Default color
export prompt_hl="^[^E[01;32m^]"  # Highlight codes

To enter ^[ in emacs, type Ctrl-q then Ctrl-[. Likewise ^E is Ctrl-q Ctrl-e.

Configuring Heartbeat v1 Was So Simple

Thu, 03 Sep 2009 11:32:51 +0200

…because it couldn’t do anything.

People who loved how simple Heartbeat v1 was to configure often complain how complex Pacemaker is.

But the key differences between the two configurations are driven by the very features that haresources-based clusters couldn’t provide.

Granted we made a mess of things with the original XML syntax.: When the job of writing the CRM/Pacemaker was first pitched to me, I was promised an all-singing, all-dancing GUI that would hide the ugly XML from end users.; So we focused on power and expressiveness and little thought was given to it’s usability.

However with the release of Pacemaker 1.0 in October 2008, not only has the XML been cleaned up, but Dejan has written a consolidated cluster shell that hides all the XML anyway.

As a result almost everything you thought about Pacemaker’s complexity is no longer true and it’s just as suited to simple two-node setups as it is to large active/active configurations.

Establishing A Baseline

Here’s a sample haresources file from the linux-ha.org website:

linuxha1 IPaddr::192.168.85.3 httpd smb

The pattern here is:

$preferred_node $script::$parameter $script $script

Which is mostly sufficient for a two node cluster, because anything not running on $preferred_node is running on the only other machine in the cluster.

Deficiencies

Of course there were some obvious limitations built into Heartbeat v1 that Pacemaker was designed to address.

Couldn’t support more than two nodes
Couldn’t detect or recover from resource-level failures
Limited to sets of resources with a strict linear stop/start order

Power Creates Complexity, but Not That Much

Non-linear Resource Model

Consider the following environment:

Resource B depends on Resource A
Resource C depends on Resource A
Resource B and Resource C are independent

There is no way to truly express this with haresources.

In order to have a more powerful resource model, you need to be able to refer multiple times to resources. So now they need a name and a more flexible way to group them.

New Resource Syntax

This is Pacemaker’s XML equivalent of the IPaddr resource above, which I’ve imaginatively called IP:

This is also the point at which many people run away screaming. There’s really no need for this though, almost all of it is scaffolding.

Here’s how I created the above XML:

# crm configure primitive IP ocf:heartbeat:IPaddr params ip=192.168.85.3

Which is composed of

primitive ::= The type of resource object that we’re creating.
IP ::= Our name for the resource
IPaddr ::= The script to call
ocf ::= The standard it conforms to
ip=192.168.85.3 ::= Parameter(s) as name/value pairs

To create the other two members of the group, I ran:

# crm configure primitive http lsb::httpd
# crm configure primitive samba lsb::smb

Admit it, that was pretty easy :-)

To group them together, as-per the haresources example, is also trivial. Just provide a name for the group and a list of members:

# crm configure group v1-group IP http samba

Thats it! Here’s the result:

# crm configure show
primitive IP ocf:heartbeat:IPaddr params ip="192.168.85.3"
primitive http lsb:httpd
primitive samba lsb:smb 
group v1-group IP http samba

Resource Recovery

In order to detect resource failure, the cluster needs to check its health periodically. But what action should it call and how often? There is nowhere in haresources to specify this. Not cleanly anyway.

By way of example, here’s what it looks like in Pacemaker if you want to monitor the IP address every 5 minutes and apache/samba once a minute:

# crm configure show
primitive IP ocf:heartbeat:IPaddr params ip="192.168.85.3" op monitor interval=5min
primitive http lsb:httpd op monitor interval=60s
primitive samba lsb:smb op monitor interval=60s
group v1-group IP http samba

More Than Two Nodes

Supporting more than two nodes means that can no longer specify a preferred node like in v1. Thats not enough to tell the cluster where to put the resource after $preferred_node fails. Instead you need an ordered list.

Here’s an example of how we might specify that we prefer linuxha1 over linuxha2 over linuxha3:

# crm configure location prefer-ha1 v1-group 5000: linuxha1
# crm configure location prefer-ha2 v1-group 500:  linuxha2
# crm configure location prefer-ha3 v1-group 50:   linuxha3

The numbers (5000, 500, 50) are scores that indicate a relative preference for running on the three nodes.

To finish off, here’s the Pacemaker equivalent of just the original haresources example:

primitive IP ocf:heartbeat:IPaddr params ip="192.168.85.3"
primitive http lsb:httpd
primitive samba lsb:smb

group v1-group IP http samba
location prefer-ha1 v1-group 5000: linuxha1

See, its really not that scary and, as we saw, easily extendable to a more complex clustering environment if needed.

Failback? Sure, but to Where?

Imagine the scenario, linuxha1 failed and the group moved to linuxha2. Then linuxha2 failed, and now the resource group is running on linuxha3. What happens when the other two nodes recover?

In v1, this is controlled by the auto_failback directive. But when there are more than two nodes, where does back refer to? It could mean linuxha2, since that was the last place it was running. It could also mean linuxha1, since that is the most preferred node under normal circumstances.

The sanest way to resolve this ambiguity is to invert the option and make it a preference for keeping the resource where it is.

Thus the resource-stickiness option was born and, unlike auto_failback, it can be specified per-resource (with a global default). Which makes sense, since the cost of stopping and relocating an IP address is significantly less than that of an Oracle database.

Pacemaker even allows the administrator to have different fail-back policies apply during “core” and “non-core” hours. But more on that another day.

Another Documentation Update

Tue, 25 Aug 2009 17:01:07 +0200

Quick FYI… I’ve made some more improvements to the Configuration Explained PDF http://clusterlabs.org/mediawiki/images/f/fb/Configuration_Explained.pdf

Changes include:

Fixed a number of date based rule examples
Updated details on the stonith-enabled option
Fixed the URL for the obtaining the XSLT conversion script
Explanations of the possible values for the target-role and multiple-active, resource options
Explanations of the possible values for the on-fail and requires operation options
Explanations of the possible values for the operation option of expressions and date_expressions
Rewrote section on Moving Resources due to Connectivity Changes to use the new, more reliable ping RA
Referenced the original definition of various resource options instead of repeating them
Added text for obtaining detailed information on the meaning of stonith parameters

Dev Repository Recreated

Thu, 13 Aug 2009 10:50:56 +0200

For a variety of reasons, the Pacemaker dev repository has been recreated and its history pruned of non-pacemaker related changes.

Any existing clones need to be removed and re-cloned as the all the changeset hashes have changed and it is fundamentally a different repository.

The good news though, is that with the help of some Mercurial hacking, any links using the “old” hashes will still work.

Pacemaker 1.0.5 Released

Tue, 11 Aug 2009 22:50:00 +0200

I’m back from vacation so it’s time for another Pacemaker bug-fix release. Testing went flawlessly and so without further ado, here it is…

Pre-built packages for Pacemaker and it’s immediate dependancies will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the OpenSUSE Build Service once it catches up.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	90
Diff	185 files changed, 11051 insertions(+), 1206 deletions(-)

Project Administrivia

The next release will be in early September and there will focus on performance. Specifically I’ll be looking at CPU usage, logging and performance in larger clusters (>8 nodes).

Changes of note since Pacemaker-1.0.4

Update source tarball to revision: 462f1569a437 (stable-1.0) tip
High (bnc#507255): Tools: crm: implement date expressions
High: ais: Fix cluster connection when using corosync 1.0
High: ais: fix compilation against latest corosync
High: Build: Fix compilation when snmp and esmtp are not available
High: Core: Show help text and exit with rc 1 if option processing failed
High: crmd: Terminate if we are ever evicted from the membership
High: crmd: Unset any existing DC value before querying for a new one
High: lrm: Look in the correct location for stonith agents
High: PE: Bug 2160 - Dont shuffle clones due to colocation
High: PE: Bug bnc#515172 - Correctly process location constraint rules which contain multiple expressions
High: PE: Bug bnc#515172 - Fix the boolean-op attribute of rules
High: PE: Fix reload for master/slave resources
High: PE: New implementation of the resource migration (not stop/start) logic
High: PE: Only prevent migration if the clone dependancy is stopping/starting on the target node
High: Tools: crm: new display type (uppercase keywords)
High: Tools: crm: support for color output
High: Tools: crm_resource - Advertise —move instead of —migrate
High: Tools: Differentiate between —help and an unknown option
Medium: cib: Supply an empty status section for replace operations
Medium: crmd: Note that dc-deadtime can be used to mask the brokeness of some switches
Medium: Extra: Add tools, an RA and tests for the System Health feature written by Mark Hamzy
Medium: Extra: New node connectivity RA that uses system ping and attrd_updater
Medium: PE: Prevent use-of-NULL in find_first_action()
Medium: Tools: crm: fix the verify exit code properly
Medium: Tools: crm_resource - Prevent use-of-NULL by requiring a resource name for the -A and -a options

Poll: Which Distro do you use for Pacemaker?

Tue, 11 Aug 2009 15:32:22 +0200

Please let us know by filling out the following poll: http://www.clusterlabs.org/wiki/UsagePoll

Any version of Pacemaker counts, as do versions of Heartbeat running the CRM.

Choose the Right Hardware

Fri, 07 Aug 2009 10:02:27 +0200

Recently I was asked to help diagnose a cluster that was behaving incredibly badly.

In this case, it was a 2-node cluster based on OpenAIS and they were simulating failures. They’d initiate a failure and sure enough, the other node would recognize and respond appropriately.
So far so good.

The problems started when the failed node rebooted.

When the failed node returned, it came up alone and proceeded to fence the surviving one. Which came up alone and shot its peer. Which came up alone and shot its peer.

You get the picture.

Somehow what should have been a simple recovery had escalated into a STONITH death-match.

To make a long story short, the root of the problem was their managed switch. The switch, a SMC 8824M, was not correctly handling multicast.

When a node rebooted, and only when the node rebooted, it took approximately two minutes for the multicast group to properly re-form. This was long enough for the returning node to declare itself a master, fence its peer(s) and begin starting services. At which point it too would get shot.

The solution was to adjust the dc-deadtime option. dc-deadtime tells the cluster how long to wait before deciding it is all by itself. Setting this to an interval long enough for the switch to reliably re-form the multicast group prevented the death-match from occurring.

To tell the cluster to wait two minutes, run:

crm configure property dc-deadtime=2min

In this instance however, the customer was lucky. Simply rebooting the switch caused the problem to go away.

Which I’m sure made their finance department happy too.

Lessons learned

Switches can have bugs too
Choose your hardware carefully, make sure its suitable for the task it is intended for.

Updated Documentation

Thu, 06 Aug 2009 08:48:08 +0200

The Configuration Explained PDF has been updated for 1.0.4/5 http://clusterlabs.org/mediawiki/images/f/fb/Configuration_Explained.pdf

Changes include:

Instructions for enabling resource migration
Clarified the difference between moving and migrating a resource
Clarified how the semantics of resources sets differ from that of groups
Fixed instructions for enabling Pacemaker for Heartbeat
Updated syntax in examples
- Rules missing scores
- Invalid date expressions

Pacemaker 1.0.5: Testing In Progress

Mon, 03 Aug 2009 12:57:13 +0200

A quick note to say that 1.0.5 testing officially started today. Release testing usually takes 1-2 weeks.

Currently queued changes for this release:

High (bnc#507255): Tools: crm: implement date expressions
High: Build: Fix compilation when snmp and esmtp are not available
High: Core: Show help text and exit with rc 1 if option processing failed
High: PE: Bug 2160 - Dont shuffle clones due to colocation
High: PE: Bug bnc#515172 - Correctly process location constraint rules which contain multiple expressions
High: PE: Bug bnc#515172 - Fix the boolean-op attribute of rules
High: PE: Fix reload for master/slave resources
High: PE: New implementation of the resource migration (not stop/start) logic
High: PE: Only prevent migration if the clone dependancy is stopping/starting on the target node
High: Tools: Differentiate between —help and an unknown option
High: Tools: crm: new display type (uppercase keywords)
High: Tools: crm: support for color output
High: ais: Fix cluster connection when using corosync 1.0
High: crmd: Terminate if we are ever evicted from the membership
High: crmd: Unset any existing DC value before querying for a new one
High: lrm: Look in the correct location for stonith agents
Medium: Extra: Add tools, an RA and tests for the System Health feature written by Mark Hamzy
Medium: PE: Prevent use-of-NULL in find_first_action()
Medium: Tools: crm_resource - Prevent use-of-NULL by requiring a resource name for the -A and -a options
Medium: cib: Supply an empty status section for replace operations

Pacemaker in Fedora 12

Fri, 31 Jul 2009 13:55:53 +0200

Good news for Fedora fans, we’ve successfully navigated the required red tape and Pacemaker will ship in Fedora 12.

Hopefully Debian and Ubuntu will not be far behind.

Resource Migration and Regression Testing

Wed, 22 Jul 2009 16:00:00 +0200

Yesterday I was working on a migration bug.

It didn’t take long to identify or fix, and afterwards I was terribly pleased with myself.
The fix was simple, elegant and allowed the cluster to use migration (instead of stop then start) more often.

Why had I not seen how easy it was sooner?

Unfortunately it was because I’d ignored half the problem.

One decision I’m particularly happy I made place 6 years ago, is the one when I ensured Pacemaker’s Policy Engine could be used outside of a running cluster. Combined with some additional output options, this makes it possible to have a suite (224 right at this moment) of regression tests that catch this sort of idiocy before it ever affects an actual user.

The Policy Engine is by far the most complex part of the system, and it’s totally infeasible to test by hand even a small fraction what the regression suite can (and in under 30s too!).

This is why I’m so confident when I say that each release is better than the last. Once a Policy Engine bug gets fixed, it stays fixed.

The benefits of offline testing also occur much earlier in the process. The Policy Engine keeps a rolling list of the cluster states it performed calculations on and our test reporting tool collects these. So when users report a Policy Engine bug, there is no need to reproduce the issue and afterwards we can conclusively show (using pretty dot graphs of the cluster’s old and new behavior) that the issue is resolved.

So I sat down again today and made sure I’d thought the whole problem through - so that the next version would be a complete solution. You can see my notes below if you’re interested.

And now I’m off to implement it (and some extra tests :-)

Migration Scenario Notes

Cluster Setup

primitive(A) depends on clone(B)

Resource Activity During Move: A(node-1 to node-2)

time	node-1	node-2	node-3
t0	A.stop
t1	B.stop		B.stop
t2		B.start	B.start
t3		A.start

Resource Activity During Migration: A(node-1 to node-2)

time	node-1	node-2	node-3
t0		B.start	B.start
t1	A.stop*
t2		A.start**
t3	B.stop		B.stop

Node *: Rewritten to be a migrate-to operation
Node **: Rewritten to be a migrate-from operation

Constraints

The following constraints already exist in the system. The ‘ok’ and ‘fail’ column refers to whether they still hold for migration.

A.stop -> A.start - ok
B.stop -> B.start - fail
A.stop -> B.stop - ok
B.start -> A.start - ok
B.stop -> A.start - fail
A.stop -> B.start - fail

Scenarios

B unchanged - ok
B stopping only - fail - possible after reversing constraint 5
B starting only - fail - possible after reversing constraint 6***
B stoping and starting - fail - constraint 2 is unfixable
B restarting but only on N2 - fail - as per case 4 but even less likely

Note ***: This is what the existing implementation does