The Cluster Guy

Pacemaker 1.0.12 Released

Thu, 24 Nov 2011 04:43:15 +0100

Thanks once again to the efforts of Keisuke MORI from NTT, the latest bug fixes have been back-ported from 1.1 and another instalment of the Pacemaker 1.0 release series is now ready for general consumption.

Changesets	96
Diff	121 files changed, 8617 insertions(+), 988 deletions(-)

Important changes since Pacemaker-1.0.11 include:

cib: Call gnutls_bye() and shutdown() when disconnecting from remote TLS connections
cib: Remove disconnected remote connections from mainloop
crmd: Cancel timers for actions that were pending on dead nodes
crmd: Do not wait for actions that were pending on dead nodes
crmd: Ensure we do not attempt to perform action on failed nodes
PE: Correctly recognise which recurring operations are currently active
PE: Demote from Master does not clear previous errors
PE: Ensure restarts due to definition changes cause the start action to be re-issued not probes
PE: Ensure role is preserved for unmanaged resources
PE: Ensure unmanaged resources have the correct role set so the correct monitor operation is chosen
PE: Move master based on failure of colocated group
pengine: Correctly determine the state of multi-state resources with a partial operation history
PE: Only allocate master/slave resources once
Shell: implement -w,—wait option to wait for the transition to finish
Shell: repair template list command

You also can see the full changelog,

I have updated the release calendar and the next 1.0.x release is planned for mid-May 2012.

The source tarball is also available directly from GitHub.

Pre-built packages for Pacemaker are available immediately for current openSUSE (12.1, 11.4, 11.3) and Fedora (16, 15, 14) releases as well as EPEL-5 from the ClusterLabs Build Area.

Users of more most distributions are encouraged to use the latest 1.1.x release - either from the 1.1 Build Area or from the distribution directly.

General installation instructions are available at from the ClusterLabs wiki.

New Version Control System

Thu, 13 Oct 2011 04:25:01 +0200

Since September, Pacemaker has started using Git for the 1.1 and devel trees.

There were some minor technical advantages over Mercurial (which I still personally prefer), but mostly the decision was driven by the pain associated with switching between SCMs multiple times a day.

The majority of development now happens on GitHub, which has some great features for reviewing patches and general collaboration.

The Pacemaker tree is also periodically sync’d to the Cluster Labs server in case GitHub is unavailable for any reason.

For those new to Git, GitHub has many tips for setting up Git, creating a local copy of the Pacemaker repo to work in, submitting your changes upstream (we use the Fork + Pull Model), and other assorted resources.

Be sure to configure email and user information so you get credit for your hard work too!

New Issue Tracker

Thu, 13 Oct 2011 03:32:48 +0200

Since it’s clearly not acceptable for our issue tracker to be offline for months at a time, it is time to replace the Bugzilla instance hosted by the Linux Foundation with something else.

One candidate that came close was the github issue tracker, but alas it doesn’t support attachments. The end result is that we now have an instance of Bugzilla v4 at:

http://bugs.clusterlabs.org

Bug numbers start at 5000.
This avoids clashing with older ones and may enable us to import the old ones if it ever comes back up again. I would advise people to assume this wont happen and to re-create any unresolved issues.

Pacemaker 1.0.11 Released

Mon, 02 May 2011 13:08:51 +0200

The latest installment of the Pacemaker 1.0 release series is now ready for general consumption.

Changesets	85
Diff	500 files changed, 69642 insertions(+), 58270 deletions(-)

Thanks once again to the efforts of Keisuke MORI and NTT, the latest bug fixes have been back-ported from 1.1

Important changes since Pacemaker-1.0.10 include:

cib: Repair the processing of updates sent from peer nodes
crmd: All pending operations should be recorded, even recurring ones with high start delays
crmd: Bug lf#2509 - Watch for config option changes from the CIB even if we’re not the DC
crmd: Bug lf#2528 - Introduce a slight delay when creating a transition to allow attrd time to perform its updates
crmd: Bug lf#2545 - Ensure notify variables are accurate for stop operations
crmd: Bug lf#2559 - Fail actions that were scheduled for a failed/fenced node
crmd: Cancel recurring operations while we’re still connected to the lrmd
crmd: Don’t abort transitions when probes are completed on a node
crmd: Ensure the CIB is always writable on the DC by removing a timing hole
crmd: Update failcount for failed promote and demote operations
PE: Bug lf#2495 - Prevent segfault by validating the contents of ordering sets
PE: Bug lf#2508 - Correctly reconstruct the status of anonymous cloned groups
PE: Bug lf#2544 - Prevent unstable clone placement by factoring in the current node’s score before all others
PE: Bug lf#2554 - target-role alone is not sufficient to promote resources
PE: Ensure fencing of the DC preceeds the STONITH_DONE operation
PE: Ensure that fencing has completed for stop actions on stonith-dependent resources (lf#2551)
PE: Prevent clones from being stopped because resources colocated with them cannot be active
PE: Prevet use-after-free resulting from unintended recursion when chosing a node to promote master/slave resources
Shell: don’t create empty optional sections (bnc#665131)
Tools: Bug lf#2528 - Make progress when attrd_updater is called repeatedly within the dampen interval but with the same value
Tools: Prevent crm_resource commands from being lost due to the use of cib_scope_local

You also can see the full changelog,

As per our release calendar, the next 1.0.x release is planned for mid-September.

The source tarball is also available directly from Mercurial.

Pre-built packages for Pacemaker and it’s immediate dependancies are available immediately for openSUSE 11.2, 11.3, Fedora-13 and EPEL-5 from the ClusterLabs Build Area.

Users of more recent distributions are encouraged to use the latest 1.1.x - either from the 1.1 Build Area or the distribution directly.

General installation instructions are available at from the ClusterLabs wiki.

Pacemaker 1.1.5 Released

Wed, 23 Feb 2011 12:36:01 +0100

The latest installment of the Pacemaker 1.1 release series is now ready for general consumption.

Changesets	184
Diff	605 files changed, 46103 insertions(+), 26417 deletions(-)

As well as the usual round of bug fixes, see the full changelog, S.U.S.E. has implemented support for ACLs. This means that you can now delegate permission to control parts of the cluster (as defined by you) to non-root users.

ACLs are still disabled by default, but you can read their documentation, provide feedback and decide if its something you want to use.

As per our release calendar, the next 1.1 release is planned for mid-April and 1.0.11 should be available in March depending on how quickly we can get the bugfixes from 1.1 backported.

Pre-built packages for Pacemaker and it’s immediate dependancies are available immediately for openSUSE 11.3, Fedora-14 and EPEL-5 from the ClusterLabs Build Area.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

New Logo?

Fri, 12 Nov 2010 12:07:00 +0100

One unexpected outcome from the recent Linux Plumbers conference was the contribution of a new logo to the project by NTT.

Quite possibly you’re now wondering how this logo relates at all to clustering and the Pacemaker project. Don’t worry, they came up with a backstory too!

In various forms of racing there is quite often someone/something setting a benchmark time or speed. This entity is often referred to as the pace-setter, pacemaker, or colloquially as a “rabbit”.

The logo is therefor a stylized pair of rabbit ears and the implication is that we’re setting new standards for cluster resource management.

As well as the logo, NTT also contributed some very professional looking banner images they’d created a Japanese cluster site they’ve been busy building up. Even if you can’t speak Japanese, be sure to check out the shiny intro movie on the front page!

I quite like the logo and the message, but I’m interested in the community’s reaction. I’ve created an online poll, be sure to let us know what you think.

Pacemaker Release Roundup

Fri, 12 Nov 2010 11:02:39 +0100

It may have seemed quiet since July, but things were actually so busy that I couldn’t find the time to publicize our new releases.

First up, the long awaited 1.0.10 is finally here. Thanks once again to the hard work of Keisuke MORI from NTT, 1.0.10 contains all the bug fixes from the recent 1.1.3 and 1.1.4 releases. You can preview the list of updates with the new online change log.

In addition to general bugfixes, the big news in 1.1.3 was the addition of a master control process and support for cman. Cman support allows us to run on top of a traditional RHCS cluster stack - replacing just the rgmanager component (more details on this in a subsequent post).

1.1.3 also introduced a new logging system inspired by the kernel and a PoC from Lars Ellenberg. It enables us to selectively enable logs for specific files, functions and even individual lines. Eventually this should result in less being logged by default.

The successor to 1.1.3 was all about performance. In 1.1.4 we managed to speed up the CIB and Policy Engine by about 80% each. So if you have 100’s of resources, you really want to be using this version (the changes were far too invasive to consider including in a 1.0 release).

Packages for all three releases are available from the rpm and rpm-next repositories on clusterlabs.org

In other news, I have also recently updated the release calendar for 2011.

Pacemaker, Heartbeat, Corosync, WTF?

Thu, 07 Oct 2010 13:32:43 +0200

One question I still get a lot is what all these projects are/do and how they all relate.

Here is the list of the possible components that might make up a Pacemaker install is:

Pacemaker - Resource manager
Corosync - Messaging layer
Heartbeat - Also a messaging layer
Resource Agents - Scripts that know how to control various services

Pacemaker is the thing that starts and stops services (like your database or mail server) and contains logic for ensuring both that they’re running, and that they’re only running in one location (to avoid data corruption).

But it can’t do that without the ability to talk to instances of itself on the other node(s), which is where Heartbeat and/or Corosync come in.

Think of Heartbeat and Corosync as dbus but between nodes. Somewhere that any node can throw messages on and know that they’ll be received by all its peers. This bus also ensures that everyone agrees who is (and is not) connected to the bus and tells Pacemaker when that list changes.

For two nodes Pacemaker could just as easily use sockets, but beyond that the complexity grows quite rapidly and is very hard to get right - so it really makes sense to use existing components that have proven to be reliable.

You only need one of them though :-)

Finally, in order to avoid teaching Pacemaker about every possible service that people might want to make highly available, we make use of the OCF standard to hide the details in scripts - which we call Resource Agents. Any series of command-line actions can be easily turned into a resource agent by adding them to an existing template.

However a collection of the most commonly useful ones are made available as part of the Resource Agents project.

And of course pre-built packages for all these come with most of the popular Linux distributions, including Fedora, openSUSE, SLES >= 10, RHEL >= 6, Debian, and Ubuntu.

Large Cluster Performance

Mon, 04 Oct 2010 13:26:01 +0200

Over the last few days, I’ve spent a bunch of time improving Pacemaker’s performance in large clusters.

This involved profiling the CIB and Policy Engine, identifying and optimizing hotspots and improving algorithm designs.

Since most of my work is done in virtual machines, it wasn’t possible to use oprofile. Strictly speaking oprofile worked, but without hardware performance counters the results weren’t very helpful. I also tried gprof, but that is more about counting calls rather than time spent.

Eventually I switched to callgrind and when combined with a tool Tim found called Gprof2Dot and/or kcachegrind, finally got the data I was looking for.

To do your own profiling, simply set PCMK_callgrind_enabled to either yes or to the name of a Pacemaker daemon you wish to profile. Eg. PCMK_callgrind_enabled=cib

Overall, the CIB (which is the main bottleneck in a large cluster) and the Policy Engine are about 70% faster.

The improvements will be available with 1.1.4 is released next month, or from our 1.1 code repository right now.

A summary of the various changes and description of future work is below. Any assistance in further optimization would be appreciated :-)

— Andrew

PE

Use case:
* 100 nodes * 100 clones, clone-max=100 (10,000 effective resources) * 100 resource location constraints

Baseline: with probes 20-30 minutes Baseline: without probes 28s

Phase 1

Use hashtables instead of lists for stores the available nodes for a resource New time without probes: 18s

Phase 2

Defer creation of deletion,promote and demote constraints until they are needed New time without probes: 13s

Phase 3

Use g_list_prepend() instead of g_list_append() for the list of ordering constraints New time without probes: 5s

Phase 4

New algorithm for determining which clone instances need probing New time with probes: 31s

Future work

Further improve the algorithm for determining which resources need to be probed
Further optimize the algorithm for enforcing ordering constraints

CIB

The CIB was harder to profile. Rather than give it one large task to chew through and see how long it took using a few printf’s to provide granularity, I had to run it through a profiler while it was operating in a real cluster and see where most of the time was being spent.

Phase 1

Remove most uses of cib_msg_copy(), reduced the amount of needless copying.

Phase speedup: 10%

Phase 2

Compression costs a LOT, don’t do it unless we’re hitting message limits. For now, use 256k as the threshold at which compression kicks in. The previous limit was 10k, compressing 184 of 1071 messages accounted for 23% of the total CPU used by the cib.

Each time we validated the CIB, we were re-reading and re-parsing the RelaxNG schema, which accounted for 28% of the CIB’s CPU usage on the DC. We now read it once and cache the result for the life of the CIB process.

Phase speedup: 51%

Phase 3

Push detection of group and set ordering changes to (the less busy) slave instances. This detection was costing 15% of the CIB’s total CPU time on the DC.

Phase speedup: 15%

Future work

The majority of CPU spent by the CIB is in post-processing.

Detecting what changed so we can minimize the network load: diff_xml_object, 35.5% CPU time
Calculating the current digest so peers can verify the diffs and detect ordering changes: calculate_xml_digest, 31% CPU time

Introducing the Pacemaker Master Control Process for Corosync-based Clusters

Thu, 05 Aug 2010 11:31:19 +0200

The latest addition to the Pacemaker 1.1 series is a master control process (MCP) and associated init script.

This means that Pacemaker is now started/stopped independently of the messaging layer. We anticipate that this should result in a simpler and more reliable startup/shutdown procedure when used in combination with Corosync.

Forking inside a multi-threaded process like Corosync causes all sorts of pain. This has been problematic for Pacemaker as it needs a number of daemons to be spawned.

Likewise, Corosync was never designed for staggered shutdown - something previously needed in order to prevent the cluster from leaving before Pacemaker could stop all active resources.

By moving this functionality into the MCP, the whole system should become more reliable

It should be noted that when using the MCP, Corosync will refuse to shutdown if Pacemaker is still running. Pacemaker will also naturally fail to start if Corosync isn’t active yet.

So, starting with 1.1.3, the following Corosync-based options are possible:

corosync + pacemaker plugin (v0)
corosync + pacemaker plugin (v1) + mcp
corosync + cpg + cman + mcp
corosync + cpg + quorumd + mcp

Option ‘1’ corresponds to what people have been using since openais/corosync started being supported. If Pacemaker starts being supported in RHEL6, its probably going to look like option ‘3’. Option ‘4’ is what we’re all working towards.

Anyone having startup or shutdown problems (with Pacemaker 1.1 or 1.0) should immediately move to clusters based on option ‘2’ or ‘3’.

Both involve the new master control process and therefor benefit from the more reliable startup/shutdown design.

Additionally, ‘3’ uses CPG for messaging (whereas ‘2’ still uses the plugin which makes it compatible with option nodes running ‘1’).

Unfortunately option ‘4’ isn’t fully baked yet, there’s still a few kinks in the pacemaker/quorumd interaction to be worked out. This will happen in the coming months, however any assistance in this process would be highly appreciated.

To use option ‘2’, simply change: ver: 0 to ver: 1 in the pacemaker service block of corosync.conf.

To use option ‘3’, you can either: * use cluster.conf and service cman start or, * add the cman bits to corosync.conf.

Using cluster.conf is the preferred approach. Its far easier to maintain and start automatically starts the necessary pieces for using GFS2.

Alternative 1 - Sample cluster.conf for a two-node cluster

Alternative 2 - Sample corosync.conf additions for a two node CMAN cluster

Be sure to set nodename appropriately for each host.

cluster {
    name: beekhof

    clusternodes {
            clusternode {
                    votes: 1
                    nodeid: 1
                    name: pcmk-1
            }
            clusternode {
                    votes: 1
                    nodeid: 2
                    name: pcmk-2
            }
    }
    cman {
            expected_votes: 2
            cluster_id: 123
            nodename: pcmk-1
            two_node: 1
            max_queued: 10
    }
}

service {
    name: corosync_cman
    ver: 0
}

quorum {
    provider: quorum_cman
}

Pacemaker 1.0.9 Released

Wed, 23 Jun 2010 15:46:14 +0200

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

Coinciding with 1.0.9 is a new version of Corosync (1.2.5). Included in both are some important fixes that should resolve most of the startup issues people have been seeing. Also included in this release are the fixes for issues reported by Valgrind and Coverity.

As per our release calendar, the next 1.0 release is planned for mid-September and 1.1.3 will be available in late July.

I’d like to particularly thank Keisuke MORI for his help with this release. Keisuke-san has taken on the role of Patch Manager for 1.0, so it is because of his hard work that we have backports of all the bugfixes from 1.1 :-)

This change has enabled me to focus on 1.1 and, I hope, be slightly more responsive to bug reports and questions on the mailing list(s).

Pre-built packages for Pacemaker and it’s immediate dependancies are available immediately for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area.

Regular updaters may also have noticed the expanded version scheme used by packages on clusterlabs.org. The build scripts now automatically bump the version numbers when rebuilding the stack. This usually occurs when new versions of corosync, cluster-glue or heartbeat come out.

Versions are now of the form: x.y.x-a.b

x.y.z is the upstream version (this is the only time the tarball is changed)
a indicates the number of spec file changes (ie. changes to dependancies)
b indicates how many times the package has been rebuilt with unchanged tarballs and spec files

So the following version: pacemaker-1.0.9-1.4 would mean the fourth rebuild of the initial spec file for the upstream version 1.0.9 of Pacemaker.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	152
Diff	266 files changed, 14324 insertions(+), 3842 deletions(-)

Changes of note since Pacemaker-1.0.8

High: ais: Ensure the list of active processes sent to clients is always up-to-date
High: ais: Fix previous commit, actually return a result in get_process_list()
High: ais: Fix two more uses of getpwnam() in non-thread-safe locations
High: ais: Look for the correct conf variable for turning on file logging
High: ais: Need to find a better and thread-safe way to set core_uses_pid. Disable for now.
High: ais: Use the threadsafe version of getpwnam
High: cib: Also free query result for xpath operations that return more than one hit
High: cib: Fix the application of unversioned diffs
High: cib: Remove old developmental error logging
High: Core: Bug lf#2414 - Prevent use-after-free reported by valgrind when doing xpath based deletions
High: Core: Fix memory leak in replace_xml_child() reported by valgrind
High: Core: fix memory leaks exposed by valgrind
High: crmd: Bug 2401 - Improved detection of partially active peers
High: crmd: Bug lf#2379 - Ensure the cluster terminates when the PE is not available
High: crmd: Bug lf#2414 - Prevent use-after-free of the PE connection after it dies
High: crmd: Bug lf#2439 - cancel_op() can also return HA_RSCBUSY
High: crmd: Bug lf#2439 - Handle asynchronous notification of resource deletion events
High: crmd: Do not allow the target_rc to be misused by resource agents
High: crmd: Do not ignore action timeouts based on FSA state
High: crmd: Ensure we dont get stuck in S_PENDING if we loose an election to someone that never talks to us again
High: crmd: Fix memory leaks exposed by valgrind
High: crmd: Remove race condition that could lead to multiple instances of a clone being active on a machine
High: crmd: Send erase_status_tag() calls to the local CIB when the DC is fenced, since there is no DC to accept them
High: PE: Bug lf#1959 - Fail unmanaged resources should not prevent other services from shutting down
High: PE: Bug lf#2383 - Combine failcounts for all instances of an anonymous clone on a host
High: PE: Bug lf#2384 - Fix intra-set colocation and ordering
High: PE: Bug lf#2403 - Enforce mandatory promotion (colocation) constraints
High: PE: Bug lf#2412 - Correctly locate clone instances by their prefix
High: PE: Bug lf#2422 - Ordering dependencies on partially active groups not observed properly
High: PE: Bug lf#2424 - Use notify oepration definition if it exists in the configuration
High: PE: Bug lf#2433 - No services should be stopped until probes finish
High: PE: Do not be so quick to pull the trigger on nodes that are coming up
High: PE: Fix colocation for interleaved clones
High: PE: Fix colocation with partially active groups
High: PE: Fix memory leaks reported by valgrind
High: PE: Make the current data set a global variable so it does not need to be passed around everywhere
High: PE: Prevent endless loop when looking for operation definitions in the configuration
High: PE: Rewrite native_merge_weights() to avoid Fix use-after-free
High: Shell: always reload status if working with the cluster (bnc#590035)
High: Tools: crm_mon - fix memory leaks exposed by valgrind
Medium: ais: Correctly set logfile permissions in all cases
Medium: ais: create the final directory too for resource agents (bnc#603190)
Medium: ais: Make sure debug messages make it into the logfiles too
Medium: Build: Do not enable the -ansi compiler option by default, prevents use of strtoll()
Medium: cib: Bug lf#2352 - Changes to group order are not detected or broadcast to peers
Medium: cib: Correctly free the cib contents at signoff when in file-based mode
Medium: cib: xpath - Allow all hits to be deleted, allow the no_children option to return multiple hits
Medium: PE: Bug lf#2391 - Ensure important options (notify, unique, etc) are always exposed during resource operations
Medium: PE: Bug lf#2410 - Do not complain about missing agents during probes of a-symetric clusters
Medium: PE: Bug lf#2426 - stop-all-resources should not apply to stonith resources
Medium: PE: Bug lf#2435 - Support colocation sets with negative scores
Medium: PE: Check for use-of-NULL in dump_node_scores()
Medium: PE: Do not overwrite existing meta attributes (like timeout) for notify operations
Medium: PE: Ensure deallocated resources are stopped
Medium: PE: If there are no compatible peers when interleaving clones, ensure the instance is stopped
Medium: PE: Ignore colocation weights from clone instances
Medium: RA: SystemHealth: exit properly when the required software is not installed (bnc#587940)
Medium: Shell: do not error on missing resource agent with asymmetrical clusters (lf#2410)
Medium: Shell: do not verify empty configurations (bnc#602711)
Medium: shell: find hb_delnode in correct directory
Medium: Shell: observe op_defaults when verifying primitives (bnc#590033)
Medium: Shell: on no id match the first of property-like elements (lf#2420)
Medium: Shell: skip resource checks for property-like elements (lf#2420)
Medium: Shell: verify meta attributes and properties (bnc#589867)
Medium: Shell: verify only changed elements on commit (bnc#590033)
Medium: Tools: crm_mon: refresh screen on terminal resize (bnc#589811)

Feature Spotlight: Utilization

Tue, 04 May 2010 10:00:00 +0200

New in 1.1 is the ability for Pacemaker to factor the system resources (RAM, CPU, etc) into its placement algorithms.

First, simply define the system resources provided by your nodes. We’ll use cores in this example, but you can literally use any name you care to dream up.

crm configure node pcmk-1 utilization cores=2
crm configure node pcmk-2 utilization cores=4

Then we tell the cluster how many cores are needed by each VM.

crm configure primitive kvm-small ocf:heartbeat:VirtualDomain utilization cores=1
crm configure primitive kvm-medium ocf:heartbeat:VirtualDomain utilization cores=2
crm configure primitive kvm-big ocf:heartbeat:VirtualDomain utilization cores=3

Finally, we tell Pacemaker how to use the information:

crm configure property placement-strategy=utilization

Now Pacemaker will ensure the load from your virtual machines will be distributed “evenly” throughout the cluster - without the need for convoluted sets of colocation constraints.

Download 1.1 today from: http://www.clusterlabs.org/rpm-next/

Limitations

This type of problem Pacemaker is dealing with here is known as the knapsack problem and falls into the NP-complete category of computer science problems - which is fancy way of saying “takes a really long time to solve”.

Clearly in a HA cluster, its not acceptable to spend minutes, let alone hours or days, finding an optional solution while services remain unavailable.

So instead of trying to solve the problem completely, Pacemaker uses a best effort algorithm for determining which node should host a particular service. This means it arrives at a “solution” much faster than traditional linear programming algorithms, but my do so at the price of leaving some services stopped.

In the contrived example above:

kvm-small would be allocated to pcmk-1
kvm-medium would be allocated to pcmk-2
kvm-large would remain inactive

Which is not ideal.

Strategies for Dealing with the Limitations

Ensure you have sufficient physical capacity It might sounds obvious, but if the physical capacity of your nodes is (close to) maxed out by the cluster under normal conditions, then failover isn’t going to go well. Even without the Utilization feature, you’ll start hitting timeouts and getting secondary “failures”.
Build some buffer into the capabilities advertised by the nodes Advertise slightly more resources than we physically have on the (usually valid) assumption that a VM will not use 100% of the configured number of cores/RAM/etc all the time. This practice is also known as “over commit”.
Specify service priorities If the cluster is going to sacrifice services, it should be the ones you care (comparatively) about the least.

Pacemaker ships as part of Ubuntu 10.4 - Lucid Lynx

Mon, 03 May 2010 11:10:00 +0200

Ubuntu LTS 10.04 now comes with full support for Pacemaker on Corosync and Heartbeat: http://fghaas.wordpress.com/2010/05/03/ubuntu-10-04-with-full-cluster-stack-support/

Kudos to everyone involved!

New Pacemaker Packages

Wed, 07 Apr 2010 16:40:55 +0200

I’ve begun uploading 1.0.8-3 to the clusterlabs.org servers.

Upon closer inspection, it became apparent that the 1.0.8-2 packages were built with the wrong tarball and this led to some substantial problems with the shell.

To rectify this, I’ve built 1.0.8-3. This new version uses the original 1.0.8 tarball and an updated spec file (to fix the snmp dependancies).

Apologies for the inconvenience.

Pacemaker in Debian

Wed, 07 Apr 2010 15:50:21 +0200

Good news for Debian fans, Pacemaker has officially made it into Sid.

According to their blog post, it should also be available as an official backport for existing Debian stable releases “soon”.

Website Updates

Wed, 24 Mar 2010 09:37:39 +0100

The http://www.clusterlabs.org server has been migrated and now features a new splash-page and a custom skin for the wiki.

Hopefully the splash page will be a more helpful entry point for people exploring the project for the first time.

http://clusterlabs.org will still show the old site until the weekend (when I switch over the mail server too).

Pacemaker 1.0.8 Released

Tue, 16 Mar 2010 21:25:35 +0100

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In this release, apart from various bug-fixes, Dejan has split the shell up into modules. It is anticipated that this will make it easier to maintain moving forward.

We are now following the published release schedule on the clusterlabs wiki. The next release is planned for mid-June and our main focus is now on features for 1.1/1.2 (see the previous post).

Pre-built packages for Pacemaker and it’s immediate dependancies are currently building and will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area shortly.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	181
Diff	329 files changed, 22172 insertions(+), 12297 deletions(-) The size of the diff is significantly impacted by the rearrangement of the shell

Changes of note since Pacemaker-1.0.7

High: Agents: ping - Prevent shell expansion of ‘*’ when there are files in /var/lib/heartbeat/cores/root (Patch from Sébastien PRUDHOMME)
High: ais: Bug lf#2340 - Force rogue child processes to terminate after waiting 2.5 minutes
High: ais: Bug lf#2359 - Default expected votes to 2 inside Corosync/OpenAIS plugin
High: ais: Bug lf#2359 - expected-quorum-votes not correctly updated after membership change
High: ais: Bug rhbz#525552 - Move non-threadsafe calls to setenv() to after the fork()
High: crmd: Bug bnc#578644 - Improve handling of cancelled operations caused by resource cleanup
High: PE: Bug lf#2317 - Avoid needless restart of primitive depending on a clone
High: PE: Bug lf#2358 - Fix master-master anti-colocation
High: PE: Bug lf#2361 - Ensure clones observe mandatory ordering constraints if the LHS is unrunnable
High: PE: Correctly implement optional colocation between primitives and clone resources
High: Shell: add support for xml in cli
High: Shell: check timeouts also against the default-action-timeout property
High: Shell: edit multiple meta_attributes sets in resource management (lf#2315)
High: Shell: improve configure commit (lf#2336)
High: Shell: new cibstatus import command (bnc#585471)
High: Shell: restore error reporting in options
High: Shell: update previous node lookup procedure to include the id where necessary
High: Shell: move scores from resource sets to the constraint element (lf#2331)
High: Shell: recovery from bad/outdated help index file
Medium: ais: getpwnam() is also not thread safe, move after the call to fork()
Medium: ais: Set permissions to allow ‘to_file’ logging to function correctly
Medium: Core: Give signal handlers higher priority - patch based on Lars Ellenbergs work
Medium: crmd: Bug bnc#578644 - Do not send operation updates for deleted resources
Medium: PE: Bug bnc#586710 - Make sure migration ops use the correct meta options (eg. timeouts)
Medium: PE: Deprecate the lifetime tag in constraints
Medium: Tools: attrd - Only ignore the update if the attributes value is completely stable (ie. supplied, current, and stored all match)
Medium: Tools: Bug lf#2302 - Use the same resource printing logic for html and non-html output
Medium: Tools: Bug LF#2312 - crm_mon - Prevent zombie child processes when using custom traps (Patch from Bernd Schubert)
Medium: Tools: Bug lf#2330 - Add a blank line after the subject to indicate the beginning of the mail body
Medium: Tools: Bug lf#2330 - Move the blank line before the body text instead
Medium: Tools: Bug lf#2330 - Use \r in addition to \n for line endings
Medium: Tools: crm_mon - Add support for older versions of SNMP - Patch derived from the work of sato yuki
Medium: Tools: crm_mon - Display the true fail-count, not the effective value
Medium: Tools: crm_mon - Use node uname in snmp/smtp/etc events
Medium: Tools: hb2openais: add support for corosync (and more)
Medium: Shell: add option to control sorting of cib elements (lf#2290)
Medium: Shell: do not cache node and resource ids (lf#2368)
Medium: Shell: fix commit for new clones of new groups (bnc#585471)
Medium: Shell: help: unsort help items
Medium: Shell: implement lifetime for rsc migrate and node standby (lf#2353)
Medium: Shell: load update should update existing elements
Medium: Shell: node attributes update in configure (bnc#582767)
Medium: Shell: parse lists not tupples
Medium: Shell: Repair “cib cibstatus op” functionality (bnc#585641)
Medium: Shell: repair node show (thanks to T. Schraitle) (bnc#587883)
Medium: Shell: repare clone/ms cleanup (nbc#583288)
Medium: Shell: catch IOErrors when opening files
Medium: Shell: check for duplicate children when creating groups (lf#2326)
Medium: Shell: do not allow score-attribute in orders
Medium: Shell: do not fiddle with cib when there is no cib (bnc#575701)
Medium: Shell: do not produce empty resource sets when adding roles/actions
Medium: Shell: do not verify empty configurations (lf#2316)
Medium: Shell: fix CIB upgrade command (bnc#578637)
Medium: Shell: fix exit code for template apply
Medium: Shell: fix reference replacement in resource sets
Medium: Shell: install crm_cli.txt also in the datadir
Medium: Shell: use the CRM_HELP_FILE variable if set

New Pacemaker Release Series

Thu, 11 Mar 2010 17:55:12 +0100

A number of new branches have been created in the last few days which are integral to how we plan to add new features in a controlled manner.

Current set of branches:

1.0 - The existing stable series
1.1 - The current feature series
1.2 - The next stable series (expected Q4 2010)
devel - Where new features are added

The idea is that 1.0 will continue receive only bugfixes (the amount hopefully continuing to reduce over time) and all new features go first into devel.

1.1 will be the merge point, primarily receiving bug fixes for the new features we pulled in from devel (at least two weeks before a 1.1 release).

The intention is to pursue bi-monthly releases of 1.1 over the next 6-12 months until we can offer a set of compelling new features, a sane way to use them, and rock-solid stability.

At that time, we will freeze development and release it as 1.2.0

But Can I use it in Production?

Cluster admins are naturally a cautious bunch, so in order to make 1.1 more palatable, we have also created two new schemas.

Schemas can be thought of as a contract between the developers and admins. They are a declaration of what we support.

The set is the new set of validation types:

pacemaker-1.0
pacemaker-1.1
pacemaker-1.2

pacemaker-1.0 is (and will remain) identical to what people are using now

When new features are added, their configuration syntax will first appear in pacemaker-1.1. If, based on your feedback, a feature’s syntax needs to be modified we will change it here and solicit further feedback.

Once a feature has been locked down and become stable, its configuration pieces will be added to pacemaker-1.2. Once a feature appears in pacemaker-1.2, we will not change its syntax in any 1.1 or 1.2 release.

What does this mean?

It means that when used with the pacemaker-1.0 or pacemaker-1.2 schemas, 1.1 can be safely used in production in the same manner as 1.0 is today.

If the existing syntax is all you need, consider 1.1 with the pacemaker-1.0 schema.
If you want to try a new stable feature, use 1.1 with the pacemaker-1.2 schema.
If you want to try a new experimental feature, use 1.1 with the pacemaker-1.1 schema.

Is 1.0 Still Supported?

Yes. 1.0 i still supported and will recieve all relevant bug fixes until at least 2012. See our releases page.

How can I Install it?

Users of RPM-based distributions will be able to install in the usual manner. However, to avoid forcing everyone to upgrade, packages will be located under:

http://www.clusterlabs.org/rpm-next

At this time the intention is to only produce packages for the most recent openSUSE (11.2), Fedora (12), and EPEL (5) based distributions. If you would like to try 1.1 but your distribution/version is not listed, please contact the project and we’ll see what we can do.

Packages should be available in the next week “or so”.

Identified Areas of Development

Existing

New stonith daemon (configuration is unchanged), including:
- support for RHCS Stonith Agents
- cluster-wide notifications (that allow the cluster to make more intelligent decisions)
- ability to perform multiple fencing operations in parallel (faster recovery)
Service placement influenced by the physical resources (RAM, CPU, etc) required by the service (and offered by the host)
A new tool for simulating failures and the cluster’s reaction to them
Tell the cluster to serialize an otherwise unrelated a set of resource actions (eg. Xen migrations)

Planned

Failover domains, an easy way to specify ordered preferences for a set of hosts
ACLs, the ability to restrict read/write access to the configuration based on users and groups
Improvements to the System Health feature
Freeze/thaw, an easy way to ensure a consistent backup of data in use by distributed systems.
The ability to monitor services on non-cluster nodes (such as VMs)

Pacemaker removed from OBS

Thu, 04 Feb 2010 11:42:00 +0100

Today I removed Pacemaker from server:ha-clustering on the openSUSE build service.

I lost patience with the service some time ago and the project has been providing pre-built packages from cluster labs ever since (see our install page for more details).

It seems no-one else has had the time or patience to keep the build service updated since my departure so, after noticing their age and the fact that they no longer even build on the majority of targets, I made the decision to remove them.

Hopefully this will help avoid confusing those wanting the latest Pacemaker software.

Pacemaker 1.0.7 Released

Mon, 18 Jan 2010 12:38:47 +0100

The latest installment of the Pacemaker 1.0 stable series is now ready for general consumption.

In this release, we’ve made a number improvements to clone handling - particularly the way ordering constraints are processed - as well as some really nice improvements to the shell.

The next 1.0 release is anticipated to be in mid-March. We will be switching to a bi-monthly release schedule to begin focusing on development for the next stable series (more details soon). If you have feature requests, now is the time to voice them and/or provide patches :-)

Pre-built packages for Pacemaker and it’s immediate dependancies are currently building and will be available for openSUSE, SLES, Fedora, RHEL, CentOS from the ClusterLabs Build Area shortly.

Debian users should check for updates Martin’s repo over the coming days and Ubuntu fans can visit LaunchPad for 8.04 and 9.10 packages.

The source tarball is also available directly from Mercurial.

General installation instructions are available at from the ClusterLabs wiki.

Release Statistics

Changesets	193
Diff	220 files changed, 15933 insertions(+), 8782 deletions(-)

Changes of note since Pacemaker-1.0.6

High: PE: Bug 2213 - Ensure groups process location constraints so that clone-node-max works for cloned groups
High: PE: Bug lf#2153 - non-clones should not restart when clones stop/start on other nodes
High: PE: Bug lf#2209 - Clone ordering should be able to prevent startup of dependant clones
High: PE: Bug lf#2216 - Correctly identify the state of anonymous clones when deciding when to probe
High: PE: Bug lf#2225 - Operations that require fencing should wait for ‘stonith_complete’ not ‘all_stopped’.
High: PE: Bug lf#2225 - Prevent clone peers from stopping while another is instance is (potentially) being fenced
High: PE: Correctly anti-colocate with a group
High: PE: Correctly unpack ordering constraints for resource sets to avoid graph loops
High: Tools: crm: load help from crm_cli.txt
High: Tools: crm: resource sets (bnc#550923)
High: Tools: crm: support for comments (LF 2221)
High: Tools: crm: support for description attribute in resources/operations (bnc#548690)
High: Tools: hb2openais: add EVMS2 CSM processing (and other changes) (bnc#548093)
High: Tools: hb2openais: do not allow empty rules, clones, or groups (LF 2215)
High: Tools: hb2openais: refuse to convert pure EVMS volumes
High: cib: Ensure the loop for login message terminates
High: cib: Finally fix reliability of receiving large messages over remote plaintext connections
High: cib: Fix remote notifications
High: cib: For remote connections, default to CRM_DAEMON_USER since thats the only one that the cib can validate the password for using PAM
High: cib: Remote plaintext - Retry sending parts of the message that did not fit the first time
High: crmd: Ensure batch-limit is correctly enforced
High: crmd: Ensure we have the latest status after a transition abort
High (bnc#547579,547582): Tools: crm: status section editing support
High: shell: Add allow-migrate as allowed meta-attribute (bnc#539968)
Medium: Build: Do not automatically add -L/lib, it could cause 64-bit arches to break
Medium: PE: Bug lf#2206 - rsc_order constraints always use score at the top level
Medium: PE: Only complain about target-role=master for non m/s resources
Medium: PE: Prevent non-multistate resources from being promoted through target-role
Medium: PE: Provide a default action for resource-set ordering
Medium: PE: Silently fix requires=fencing for stonith resources so that it can be set in op_defaults
Medium: Tools: Bug lf#2286 - Allow the shell to accept template parameters on the command line
Medium: Tools: Bug lf#2307 - Provide a way to determin the nodeid of past cluster members
Medium: Tools: crm: add update method to template apply (LF 2289)
Medium: Tools: crm: direct RA interface for ocf class resource agents (LF 2270)
Medium: Tools: crm: direct RA interface for stonith class resource agents (LF 2270)
Medium: Tools: crm: do not add score which does not exist
Medium: Tools: crm: do not consider warnings as errors (LF 2274)
Medium: Tools: crm: do not remove sets which contain id-ref attribute (LF 2304)
Medium: Tools: crm: drop empty attributes elements
Medium: Tools: crm: exclude locations when testing for pathological constraints (LF 2300)
Medium: Tools: crm: fix exit code on single shot commands
Medium: Tools: crm: fix node delete (LF 2305)
Medium: Tools: crm: implement -F (—force) option
Medium: Tools: crm: rename status to cibstatus (LF 2236)
Medium: Tools: crm: revisit configure commit
Medium: Tools: crm: stay in crm if user specified level only (LF 2286)
Medium: Tools: crm: verify changes on exit from the configure level
Medium: ais: Some clients such as gfs_controld want a cluster name, allow one to be specified in corosync.conf
Medium: cib: Clean up logic for receiving remote messages
Medium: cib: Create valid notification control messages
Medium: cib: Indicate where the remote connection came from
Medium: cib: Send password prompt to stderr so that stdout can be redirected
Medium: cts: Fix rsh handling when stdout is not required
Medium: doc: Fill in the section on removing a node from an AIS-based cluster
Medium: doc: Update the docs to reflect the 0.6/1.0 rolling upgrade problem
Medium: doc: Use Publican for docbook based documentation
Medium: fencing: stonithd: add metadata for stonithd instance attributes (and support in the shell)
Medium: fencing: stonithd: ignore case when comparing host names (LF 2292)
Medium: tools: Make crm_mon functional with remote connections
Medium: xml: Add stopped as a supported role for operations
Medium: xml: Bug bnc#552713 - Treat node unames as text fields not IDs
Medium: xml: Bug lf#2215 - Create an always-true expression for empty rules when upgrading from 0.6