The Cluster Guy

Pacemaker 1.0.9 Released

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

Coinciding with 1.0.9 is a new version of Corosync (1.2.5). Included in both are some important fixes that should resolve most of the startup issues people have been seeing. Also included in this release are the fixes for issues reported by Valgrind and Coverity.

As per our release calendar, the next 1.0 release is planned for mid-September and 1.1.3 will be available in late July.

I’d like to particularly thank Keisuke MORI for his help with this release. Keisuke-san has taken on the role of Patch Manager for 1.0, so it is because of his hard work that we have backports of all the bugfixes from 1.1 :-)

This change has enabled me to focus on 1.1 and, I hope, be slightly more responsive to bug reports and questions on the mailing list(s).

Pre-built packages for Pacemaker and it’s immediate dependancies are available immediately for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area.

Regular updaters may also have noticed the expanded version scheme used by packages on clusterlabs.org. The build scripts now automatically bump the version numbers when rebuilding the stack. This usually occurs when new versions of corosync, cluster-glue or heartbeat come out.

Versions are now of the form: x.y.x-a.b

• x.y.z is the upstream version (this is the only time the tarball is changed)
• a indicates the number of spec file changes (ie. changes to dependancies)
• b indicates how many times the package has been rebuilt with unchanged tarballs and spec files

So the following version: pacemaker-1.0.9-1.4 would mean the fourth rebuild of the initial spec file for the upstream version 1.0.9 of Pacemaker.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changes of note since Pacemaker-1.0.8

• High: ais: Ensure the list of active processes sent to clients is always up-to-date
• High: ais: Fix previous commit, actually return a result in get_process_list()
• High: ais: Fix two more uses of getpwnam() in non-thread-safe locations
• High: ais: Look for the correct conf variable for turning on file logging
• High: ais: Need to find a better and thread-safe way to set core_uses_pid. Disable for now.
• High: ais: Use the threadsafe version of getpwnam
• High: cib: Also free query result for xpath operations that return more than one hit
• High: cib: Fix the application of unversioned diffs
• High: cib: Remove old developmental error logging
• High: Core: Bug lf#2414 - Prevent use-after-free reported by valgrind when doing xpath based deletions
• High: Core: Fix memory leak in replace_xml_child() reported by valgrind
• High: Core: fix memory leaks exposed by valgrind
• High: crmd: Bug 2401 - Improved detection of partially active peers
• High: crmd: Bug lf#2379 - Ensure the cluster terminates when the PE is not available
• High: crmd: Bug lf#2414 - Prevent use-after-free of the PE connection after it dies
• High: crmd: Bug lf#2439 - cancel_op() can also return HA_RSCBUSY
• High: crmd: Bug lf#2439 - Handle asynchronous notification of resource deletion events
• High: crmd: Do not allow the target_rc to be misused by resource agents
• High: crmd: Do not ignore action timeouts based on FSA state
• High: crmd: Ensure we dont get stuck in S_PENDING if we loose an election to someone that never talks to us again
• High: crmd: Fix memory leaks exposed by valgrind
• High: crmd: Remove race condition that could lead to multiple instances of a clone being active on a machine
• High: crmd: Send erase_status_tag() calls to the local CIB when the DC is fenced, since there is no DC to accept them
• High: PE: Bug lf#1959 - Fail unmanaged resources should not prevent other services from shutting down
• High: PE: Bug lf#2383 - Combine failcounts for all instances of an anonymous clone on a host
• High: PE: Bug lf#2384 - Fix intra-set colocation and ordering
• High: PE: Bug lf#2403 - Enforce mandatory promotion (colocation) constraints
• High: PE: Bug lf#2412 - Correctly locate clone instances by their prefix
• High: PE: Bug lf#2422 - Ordering dependencies on partially active groups not observed properly
• High: PE: Bug lf#2424 - Use notify oepration definition if it exists in the configuration
• High: PE: Bug lf#2433 - No services should be stopped until probes finish
• High: PE: Do not be so quick to pull the trigger on nodes that are coming up
• High: PE: Fix colocation for interleaved clones
• High: PE: Fix colocation with partially active groups
• High: PE: Fix memory leaks reported by valgrind
• High: PE: Make the current data set a global variable so it does not need to be passed around everywhere
• High: PE: Prevent endless loop when looking for operation definitions in the configuration
• High: PE: Rewrite native_merge_weights() to avoid Fix use-after-free
• High: Shell: always reload status if working with the cluster (bnc#590035)
• High: Tools: crm_mon - fix memory leaks exposed by valgrind
• Medium: ais: Correctly set logfile permissions in all cases
• Medium: ais: create the final directory too for resource agents (bnc#603190)
• Medium: ais: Make sure debug messages make it into the logfiles too
• Medium: Build: Do not enable the -ansi compiler option by default, prevents use of strtoll()
• Medium: cib: Bug lf#2352 - Changes to group order are not detected or broadcast to peers
• Medium: cib: Correctly free the cib contents at signoff when in file-based mode
• Medium: cib: xpath - Allow all hits to be deleted, allow the no_children option to return multiple hits
• Medium: PE: Bug lf#2391 - Ensure important options (notify, unique, etc) are always exposed during resource operations
• Medium: PE: Bug lf#2410 - Do not complain about missing agents during probes of a-symetric clusters
• Medium: PE: Bug lf#2426 - stop-all-resources should not apply to stonith resources
• Medium: PE: Bug lf#2435 - Support colocation sets with negative scores
• Medium: PE: Check for use-of-NULL in dump_node_scores()
• Medium: PE: Do not overwrite existing meta attributes (like timeout) for notify operations
• Medium: PE: Ensure deallocated resources are stopped
• Medium: PE: If there are no compatible peers when interleaving clones, ensure the instance is stopped
• Medium: PE: Ignore colocation weights from clone instances
• Medium: RA: SystemHealth: exit properly when the required software is not installed (bnc#587940)
• Medium: Shell: do not error on missing resource agent with asymmetrical clusters (lf#2410)
• Medium: Shell: do not verify empty configurations (bnc#602711)
• Medium: shell: find hb_delnode in correct directory
• Medium: Shell: observe op_defaults when verifying primitives (bnc#590033)
• Medium: Shell: on no id match the first of property-like elements (lf#2420)
• Medium: Shell: skip resource checks for property-like elements (lf#2420)
• Medium: Shell: verify meta attributes and properties (bnc#589867)
• Medium: Shell: verify only changed elements on commit (bnc#590033)
• Medium: Tools: crm_mon: refresh screen on terminal resize (bnc#589811)

Feature Spotlight: Utilization

New in 1.1 is the ability for Pacemaker to factor the system resources (RAM, CPU, etc) into its placement algorithms.

First, simply define the system resources provided by your nodes. We’ll use cores in this example, but you can literally use any name you care to dream up.

crm configure node pcmk-1 utilization cores=2
crm configure node pcmk-2 utilization cores=4

Then we tell the cluster how many cores are needed by each VM.

crm configure primitive kvm-small ocf:heartbeat:VirtualDomain utilization cores=1
crm configure primitive kvm-medium ocf:heartbeat:VirtualDomain utilization cores=2
crm configure primitive kvm-big ocf:heartbeat:VirtualDomain utilization cores=3

Finally, we tell Pacemaker how to use the information:

crm configure property placement-strategy=utilization

Now Pacemaker will ensure the load from your virtual machines will be distributed “evenly” throughout the cluster - without the need for convoluted sets of colocation constraints.

Download 1.1 today from: http://www.clusterlabs.org/rpm-next/

Limitations

This type of problem Pacemaker is dealing with here is known as the knapsack problem and falls into the NP-complete category of computer science problems - which is fancy way of saying “takes a really long time to solve”.

Clearly in a HA cluster, its not acceptable to spend minutes, let alone hours or days, finding an optional solution while services remain unavailable.

So instead of trying to solve the problem completely, Pacemaker uses a best effort algorithm for determining which node should host a particular service. This means it arrives at a “solution” much faster than traditional linear programming algorithms, but my do so at the price of leaving some services stopped.

In the contrived example above:

• kvm-small would be allocated to pcmk-1
• kvm-medium would be allocated to pcmk-2
• kvm-large would remain inactive

Which is not ideal.

Strategies for Dealing with the Limitations

• Ensure you have sufficient physical capacity It might sounds obvious, but if the physical capacity of your nodes is (close to) maxed out by the cluster under normal conditions, then failover isn’t going to go well. Even without the Utilization feature, you’ll start hitting timeouts and getting secondary “failures”.
• Build some buffer into the capabilities advertised by the nodes Advertise slightly more resources than we physically have on the (usually valid) assumption that a VM will not use 100% of the configured number of cores/RAM/etc all the time. This practice is also known as “over commit”.
• Specify service priorities If the cluster is going to sacrifice services, it should be the ones you care (comparatively) about the least.

Pacemaker ships as part of Ubuntu 10.4 - Lucid Lynx

Ubuntu LTS 10.04 now comes with full support for Pacemaker on Corosync and Heartbeat: http://fghaas.wordpress.com/2010/05/03/ubuntu-10-04-with-full-cluster-stack-support/

Kudos to everyone involved!

New Pacemaker Packages

I’ve begun uploading 1.0.8-3 to the clusterlabs.org servers.

Upon closer inspection, it became apparent that the 1.0.8-2 packages were built with the wrong tarball and this led to some substantial problems with the shell.

To rectify this, I’ve built 1.0.8-3. This new version uses the original 1.0.8 tarball and an updated spec file (to fix the snmp dependancies).

Apologies for the inconvenience.

Pacemaker in Debian

Good news for Debian fans, Pacemaker has officially made it into Sid.

According to their blog post, it should also be available as an official backport for existing Debian stable releases “soon”.

Website Updates

The http://www.clusterlabs.org server has been migrated and now features a new splash-page and a custom skin for the wiki.

Hopefully the splash page will be a more helpful entry point for people exploring the project for the first time.

http://clusterlabs.org will still show the old site until the weekend (when I switch over the mail server too).

Pacemaker 1.0.8 Released

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In this release, apart from various bug-fixes, Dejan has split the shell up into modules. It is anticipated that this will make it easier to maintain moving forward.

We are now following the published release schedule on the clusterlabs wiki. The next release is planned for mid-June and our main focus is now on features for 1.1/1.2 (see the previous post).

Pre-built packages for Pacemaker and it’s immediate dependancies are currently building and will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area shortly.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changes of note since Pacemaker-1.0.7

• High: Agents: ping - Prevent shell expansion of ‘*’ when there are files in /var/lib/heartbeat/cores/root (Patch from Sébastien PRUDHOMME)
• High: ais: Bug lf#2340 - Force rogue child processes to terminate after waiting 2.5 minutes
• High: ais: Bug lf#2359 - Default expected votes to 2 inside Corosync/OpenAIS plugin
• High: ais: Bug lf#2359 - expected-quorum-votes not correctly updated after membership change
• High: ais: Bug rhbz#525552 - Move non-threadsafe calls to setenv() to after the fork()
• High: crmd: Bug bnc#578644 - Improve handling of cancelled operations caused by resource cleanup
• High: PE: Bug lf#2317 - Avoid needless restart of primitive depending on a clone
• High: PE: Bug lf#2358 - Fix master-master anti-colocation
• High: PE: Bug lf#2361 - Ensure clones observe mandatory ordering constraints if the LHS is unrunnable
• High: PE: Correctly implement optional colocation between primitives and clone resources
• High: Shell: add support for xml in cli
• High: Shell: check timeouts also against the default-action-timeout property
• High: Shell: edit multiple meta_attributes sets in resource management (lf#2315)
• High: Shell: improve configure commit (lf#2336)
• High: Shell: new cibstatus import command (bnc#585471)
• High: Shell: restore error reporting in options
• High: Shell: update previous node lookup procedure to include the id where necessary
• High: Shell: move scores from resource sets to the constraint element (lf#2331)
• High: Shell: recovery from bad/outdated help index file
• Medium: ais: getpwnam() is also not thread safe, move after the call to fork()
• Medium: ais: Set permissions to allow ‘to_file’ logging to function correctly
• Medium: Core: Give signal handlers higher priority - patch based on Lars Ellenbergs work
• Medium: crmd: Bug bnc#578644 - Do not send operation updates for deleted resources
• Medium: PE: Bug bnc#586710 - Make sure migration ops use the correct meta options (eg. timeouts)
• Medium: PE: Deprecate the lifetime tag in constraints
• Medium: Tools: attrd - Only ignore the update if the attributes value is completely stable (ie. supplied, current, and stored all match)
• Medium: Tools: Bug lf#2302 - Use the same resource printing logic for html and non-html output
• Medium: Tools: Bug LF#2312 - crm_mon - Prevent zombie child processes when using custom traps (Patch from Bernd Schubert)
• Medium: Tools: Bug lf#2330 - Add a blank line after the subject to indicate the beginning of the mail body
• Medium: Tools: Bug lf#2330 - Move the blank line before the body text instead
• Medium: Tools: Bug lf#2330 - Use \r in addition to \n for line endings
• Medium: Tools: crm_mon - Add support for older versions of SNMP - Patch derived from the work of sato yuki
• Medium: Tools: crm_mon - Display the true fail-count, not the effective value
• Medium: Tools: crm_mon - Use node uname in snmp/smtp/etc events
• Medium: Tools: hb2openais: add support for corosync (and more)
• Medium: Shell: add option to control sorting of cib elements (lf#2290)
• Medium: Shell: do not cache node and resource ids (lf#2368)
• Medium: Shell: fix commit for new clones of new groups (bnc#585471)
• Medium: Shell: help: unsort help items
• Medium: Shell: implement lifetime for rsc migrate and node standby (lf#2353)
• Medium: Shell: load update should update existing elements
• Medium: Shell: node attributes update in configure (bnc#582767)
• Medium: Shell: parse lists not tupples
• Medium: Shell: Repair “cib cibstatus op” functionality (bnc#585641)
• Medium: Shell: repair node show (thanks to T. Schraitle) (bnc#587883)
• Medium: Shell: repare clone/ms cleanup (nbc#583288)
• Medium: Shell: catch IOErrors when opening files
• Medium: Shell: check for duplicate children when creating groups (lf#2326)
• Medium: Shell: do not allow score-attribute in orders
• Medium: Shell: do not fiddle with cib when there is no cib (bnc#575701)
• Medium: Shell: do not produce empty resource sets when adding roles/actions
• Medium: Shell: do not verify empty configurations (lf#2316)
• Medium: Shell: fix CIB upgrade command (bnc#578637)
• Medium: Shell: fix exit code for template apply
• Medium: Shell: fix reference replacement in resource sets
• Medium: Shell: install crm_cli.txt also in the datadir
• Medium: Shell: use the CRM_HELP_FILE variable if set

New Pacemaker Release Series

A number of new branches have been created in the last few days which are integral to how we plan to add new features in a controlled manner.

Current set of branches:

• 1.0 - The existing stable series
• 1.1 - The current feature series
• 1.2 - The next stable series (expected Q4 2010)
• devel - Where new features are added

The idea is that 1.0 will continue receive only bugfixes (the amount hopefully continuing to reduce over time) and all new features go first into devel.

1.1 will be the merge point, primarily receiving bug fixes for the new features we pulled in from devel (at least two weeks before a 1.1 release).

The intention is to pursue bi-monthly releases of 1.1 over the next 6-12 months until we can offer a set of compelling new features, a sane way to use them, and rock-solid stability.

At that time, we will freeze development and release it as 1.2.0

But Can I use it in Production?

Cluster admins are naturally a cautious bunch, so in order to make 1.1 more palatable, we have also created two new schemas.

Schemas can be thought of as a contract between the developers and admins. They are a declaration of what we support.

The set is the new set of validation types:

• pacemaker-1.0
• pacemaker-1.1 
• pacemaker-1.2

pacemaker-1.0 is (and will remain) identical to what people are using now

When new features are added, their configuration syntax will first appear in pacemaker-1.1. If, based on your feedback, a feature’s syntax needs to be modified we will change it here and solicit further feedback.

Once a feature has been locked down and become stable, its configuration pieces will be added to pacemaker-1.2. Once a feature appears in pacemaker-1.2, we will not change its syntax in any 1.1 or 1.2 release.

What does this mean?

It means that when used with the pacemaker-1.0 or pacemaker-1.2 schemas, 1.1 can be safely used in production in the same manner as 1.0 is today.

• If the existing syntax is all you need, consider 1.1 with the pacemaker-1.0 schema.
• If you want to try a new stable feature, use 1.1 with the pacemaker-1.2 schema.
• If you want to try a new experimental feature, use 1.1 with the pacemaker-1.1 schema.

Is 1.0 Still Supported?

Yes. 1.0 i still supported and will recieve all relevant bug fixes until at least 2012. See our releases page.

How can I Install it?

Users of RPM-based distributions will be able to install in the usual manner. However, to avoid forcing everyone to upgrade, packages will be located under:

http://www.clusterlabs.org/rpm-next

At this time the intention is to only produce packages for the most recent openSUSE (11.2), Fedora (12), and EPEL (5) based distributions. If you would like to try 1.1 but your distribution/version is not listed, please contact the project and we’ll see what we can do.

Packages should be available in the next week “or so”.

Identified Areas of Development

Existing

1. New stonith daemon (configuration is unchanged), including:
   • support for RHCS Stonith Agents
   • cluster-wide notifications (that allow the cluster to make more intelligent decisions)
   • ability to perform multiple fencing operations in parallel (faster recovery)
2. Service placement influenced by the physical resources (RAM, CPU, etc) required by the service (and offered by the host)
3. A new tool for simulating failures and the cluster’s reaction to them
4. Tell the cluster to serialize an otherwise unrelated a set of resource actions (eg. Xen migrations)

Planned

1. Failover domains, an easy way to specify ordered preferences for a set of hosts
2. ACLs, the ability to restrict read/write access to the configuration based on users and groups
3. Improvements to the System Health feature
4. Freeze/thaw, an easy way to ensure a consistent backup of data in use by distributed systems.
5. The ability to monitor services on non-cluster nodes (such as VMs)

Pacemaker removed from OBS

Today I removed Pacemaker from server:ha-clustering on the openSUSE build service.

I lost patience with the service some time ago and the project has been providing pre-built packages from cluster labs ever since (see our install page for more details).

It seems no-one else has had the time or patience to keep the build service updated since my departure so, after noticing their age and the fact that they no longer even build on the majority of targets, I made the decision to remove them.

Hopefully this will help avoid confusing those wanting the latest Pacemaker software.

Pacemaker 1.0.7 Released

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In this release, we’ve made a number improvements to clone handling - particularly the way ordering constraints are processed - as well as some really nice improvements to the shell.

The next 1.0 release is anticipated to be in mid-March. We will be switching to a bi-monthly release schedule to begin focusing on development for the next stable series (more details soon). If you have feature requests, now is the time to voice them and/or provide patches :-)

Pre-built packages for Pacemaker and it’s immediate dependancies are currently building and will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area shortly.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changes of note since Pacemaker-1.0.6

• High: PE: Bug 2213 - Ensure groups process location constraints so that clone-node-max works for cloned groups
• High: PE: Bug lf#2153 - non-clones should not restart when clones stop/start on other nodes
• High: PE: Bug lf#2209 - Clone ordering should be able to prevent startup of dependant clones
• High: PE: Bug lf#2216 - Correctly identify the state of anonymous clones when deciding when to probe
• High: PE: Bug lf#2225 - Operations that require fencing should wait for ‘stonith_complete’ not ‘all_stopped’.
• High: PE: Bug lf#2225 - Prevent clone peers from stopping while another is instance is (potentially) being fenced
• High: PE: Correctly anti-colocate with a group
• High: PE: Correctly unpack ordering constraints for resource sets to avoid graph loops
• High: Tools: crm: load help from crm_cli.txt
• High: Tools: crm: resource sets (bnc#550923)
• High: Tools: crm: support for comments (LF 2221)
• High: Tools: crm: support for description attribute in resources/operations (bnc#548690)
• High: Tools: hb2openais: add EVMS2 CSM processing (and other changes) (bnc#548093)
• High: Tools: hb2openais: do not allow empty rules, clones, or groups (LF 2215)
• High: Tools: hb2openais: refuse to convert pure EVMS volumes
• High: cib: Ensure the loop for login message terminates
• High: cib: Finally fix reliability of receiving large messages over remote plaintext connections
• High: cib: Fix remote notifications
• High: cib: For remote connections, default to CRM_DAEMON_USER since thats the only one that the cib can validate the password for using PAM
• High: cib: Remote plaintext - Retry sending parts of the message that did not fit the first time
• High: crmd: Ensure batch-limit is correctly enforced
• High: crmd: Ensure we have the latest status after a transition abort
• High (bnc#547579,547582): Tools: crm: status section editing support
• High: shell: Add allow-migrate as allowed meta-attribute (bnc#539968)
• Medium: Build: Do not automatically add -L/lib, it could cause 64-bit arches to break
• Medium: PE: Bug lf#2206 - rsc_order constraints always use score at the top level
• Medium: PE: Only complain about target-role=master for non m/s resources
• Medium: PE: Prevent non-multistate resources from being promoted through target-role
• Medium: PE: Provide a default action for resource-set ordering
• Medium: PE: Silently fix requires=fencing for stonith resources so that it can be set in op_defaults
• Medium: Tools: Bug lf#2286 - Allow the shell to accept template parameters on the command line
• Medium: Tools: Bug lf#2307 - Provide a way to determin the nodeid of past cluster members
• Medium: Tools: crm: add update method to template apply (LF 2289)
• Medium: Tools: crm: direct RA interface for ocf class resource agents (LF 2270)
• Medium: Tools: crm: direct RA interface for stonith class resource agents (LF 2270)
• Medium: Tools: crm: do not add score which does not exist
• Medium: Tools: crm: do not consider warnings as errors (LF 2274)
• Medium: Tools: crm: do not remove sets which contain id-ref attribute (LF 2304)
• Medium: Tools: crm: drop empty attributes elements
• Medium: Tools: crm: exclude locations when testing for pathological constraints (LF 2300)
• Medium: Tools: crm: fix exit code on single shot commands
• Medium: Tools: crm: fix node delete (LF 2305)
• Medium: Tools: crm: implement -F (—force) option
• Medium: Tools: crm: rename status to cibstatus (LF 2236)
• Medium: Tools: crm: revisit configure commit
• Medium: Tools: crm: stay in crm if user specified level only (LF 2286)
• Medium: Tools: crm: verify changes on exit from the configure level
• Medium: ais: Some clients such as gfs_controld want a cluster name, allow one to be specified in corosync.conf
• Medium: cib: Clean up logic for receiving remote messages
• Medium: cib: Create valid notification control messages
• Medium: cib: Indicate where the remote connection came from
• Medium: cib: Send password prompt to stderr so that stdout can be redirected
• Medium: cts: Fix rsh handling when stdout is not required
• Medium: doc: Fill in the section on removing a node from an AIS-based cluster
• Medium: doc: Update the docs to reflect the 0.6/1.0 rolling upgrade problem
• Medium: doc: Use Publican for docbook based documentation
• Medium: fencing: stonithd: add metadata for stonithd instance attributes (and support in the shell)
• Medium: fencing: stonithd: ignore case when comparing host names (LF 2292)
• Medium: tools: Make crm_mon functional with remote connections
• Medium: xml: Add stopped as a supported role for operations
• Medium: xml: Bug bnc#552713 - Treat node unames as text fields not IDs
• Medium: xml: Bug lf#2215 - Create an always-true expression for empty rules when upgrading from 0.6