Product SiteDocumentation Site

Pacemaker Development

Working with the Pacemaker Code Base

Edition 2

Written by the Pacemaker project contributors

Legal Notice

Copyright © 2016-2020 The Pacemaker project contributors.
The text of and illustrations in this document are licensed under version 4.0 or later of the Creative Commons Attribution-ShareAlike International Public License ("CC-BY-SA")[1].
In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
In addition to the requirements of this license, the following activities are looked upon favorably:
  1. If you are distributing Open Publication works on hardcopy or CD-ROM, you provide email notification to the authors of your intent to redistribute at least thirty days before your manuscript or media freeze, to give the authors time to provide updated documents. This notification should describe modifications, if any, made to the document.
  2. All substantive modifications (including deletions) be either clearly marked up in the document or else described in an attachment to the document.
  3. Finally, while it is not mandatory under this license, it is considered good form to offer a free copy of any hardcopy or CD-ROM expression of the author(s) work.


This document has guidelines and tips for developers interested in editing Pacemaker source code and submitting changes for inclusion in the project.

Table of Contents

1. Frequently Asked Questions
2. General Guidelines for All Languages
2.1. Copyright
3. C Coding Guidelines
3.1. Style Guidelines
3.1.1. C Boilerplate
3.1.2. Line Formatting
3.1.3. Pointers
3.1.4. Function Definitions
3.1.5. Control Statements (if, else, while, for, switch)
3.1.6. Operators
3.2. Best Practices
3.2.1. New Struct and Enum Members
3.2.2. Documentation
3.2.3. Symbol Naming
3.2.4. Memory Allocation
3.2.5. Logging
3.2.6. Regular Expressions
3.2.7. vim Settings
4. Python Coding Guidelines
4.1. Python Boilerplate
4.2. Python Compatibility
4.2.1. Python Future Imports
4.2.2. Other Python Compatibility Requirements
4.2.3. Python Usages to Avoid
4.3. Formatting Python Code
5. Evolution of the project
5.1. Foreword
5.2. Common ancestor: heartbeat project
5.2.1. Influence of heartbeat on Pacemaker
5.3. Notable Restructuring Steps in the Codebase
6. Advanced Hacking on the Project
6.1. Foreword
6.2. Debugging
6.3. Working with mocked daemons
A. Revision History

List of Examples

2.1. Copyright Notice Format

Chapter 1. Frequently Asked Questions

Who is this document intended for?
Anyone who wishes to read and/or edit the Pacemaker source code. Casual contributors should feel free to read just this FAQ, and consult other sections as needed.
Where is the source code for Pacemaker?
The source code for Pacemaker is kept on GitHub, as are all software projects under the ClusterLabs umbrella. Pacemaker uses Git for source code management. If you are a Git newbie, the gittutorial(7) man page is an excellent starting point. If you’re familiar with using Git from the command line, you can create a local copy of the Pacemaker source code with: git clone
What are the different Git branches and repositories used for?
  • The master branch is the primary branch used for development.
  • The 2.0 branch contains the latest official release, and normally does not receive any changes. During the release cycle, it will contain release candidates for the next official release, and will receive only bug fixes.
  • The 1.1 branch is similar to both the master and 2.0 branches, but for the 1.1 release series. The 1.1 branch receives only backports of certain bug fixes and backward-compatible features from the master branch. During the release cycle, it will contain release candidates for the next official 1.1 release.
  • The 1.0 repository is a frozen snapshot of the 1.0 release series, and is no longer developed.
  • Messages will be posted to the mailing list during the release cycle, with instructions about which branches to use when submitting requests.
How do I build from the source code?
See in the main checkout directory.
What coding style should I follow?
You’ll be mostly fine if you simply follow the example of existing code. When unsure, see the relevant section of this document for language-specific recommendations. Pacemaker has grown and evolved organically over many years, so you will see much code that doesn’t conform to the current guidelines. We discourage making changes solely to bring code into conformance, as any change requires developer time for review and opens the possibility of adding bugs. However, new code should follow the guidelines, and it is fine to bring lines of older code into conformance when modifying that code for other reasons.
How should I format my Git commit messages?
See existing examples in the git log. The first line should look like change-type: affected-code: explanation where change-type should be Fix or Bug for most bug fixes, Feature for new features, Log for changes to log messages or handling, Doc for changes to documentation or comments, or Test for changes in CTS and regression tests. You will sometimes see Low, Med (or Mid) and High used instead for bug fixes, to indicate the severity. The important thing is that only commits with Feature, Fix, Bug, or High will automatically be included in the change log for the next release. The affected-code is the name of the component(s) being changed, for example, controller or libcrmcommon (it’s more free-form, so don’t sweat getting it exact). The explanation briefly describes the change. The git project recommends the entire summary line stay under 50 characters, but more is fine if needed for clarity. Except for the most simple and obvious of changes, the summary should be followed by a blank line and then a longer explanation of why the change was made.
How can I test my changes?
Most importantly, Pacemaker has regression tests for most major components; these will automatically be run for any pull requests submitted through GitHub. Additionally, Pacemaker’s Cluster Test Suite (CTS) can be used to set up a test cluster and run a wide variety of complex tests. This document will have more detail on testing in the future.
What is Pacemaker’s license?
Except where noted otherwise in the file itself, the source code for all Pacemaker programs is licensed under version 2 or later of the GNU General Public License (GPLv2+), its headers and libraries under version 2.1 or later of the less restrictive GNU Lesser General Public License (LGPLv2.1+), its documentation under version 4.0 or later of the Creative Commons Attribution-ShareAlike International Public License (CC-BY-SA-4.0), and its init scripts under the Revised BSD license. If you find any deviations from this policy, or wish to inquire about alternate licensing arrangements, please e-mail the mailing list. Licensing issues are also discussed on the ClusterLabs wiki.
How can I contribute my changes to the project?
Contributions of bug fixes or new features are very much appreciated! Patches can be submitted as pull requests via GitHub (the preferred method, due to its excellent features), or e-mailed to the mailing list as an attachment in a format Git can import. Authors may only submit changes that they have the right to submit under the open source license indicated in the affected files.
What if I still have questions?
Ask on the mailing list for development-related questions, or on the mailing list for general questions about using Pacemaker. Developers often also hang out on freenode’s #clusterlabs IRC channel.

Chapter 2. General Guidelines for All Languages

Table of Contents

2.1. Copyright

2.1. Copyright

When copyright notices are added to a file, they should look like this:

Example 2.1. Copyright Notice Format

Copyright YYYY[-YYYY] the Pacemaker project contributors
The version control history for this file may have further details.
The first YYYY is the year the file was originally published. The original date is important for two reasons: when two entities claim copyright ownership of the same work, the earlier claim generally prevails; and copyright expiration is generally calculated from the original publication date. [2]
If the file is modified in later years, add -YYYY with the most recent year of modification. Even though Pacemaker is an ongoing project, copyright notices are about the years of publication of specific content.
Copyright notices are intended to indicate, but do not affect, copyright ownership, which is determined by applicable laws and regulations. Authors may put more specific copyright notices in their commit messages if desired.

[2] See the U.S. Copyright Office’s "Compendium of U.S. Copyright Office Practices", particularly "Chapter 2200: Notice of Copyright", sections 2205.1(A) and 2205.1(F), or "Updating Copyright Notices" for a more readable summary.

Chapter 3. C Coding Guidelines

3.1. Style Guidelines

Pacemaker is a large, distributed project accepting contributions from developers with a wide range of skill levels and organizational affiliations, and maintained by multiple people over long periods of time. The guidelines in this section are not technically better than alternative approaches, but make project management easier.
Many of these simply ensure stylistic consistency, which makes reading, writing, and reviewing code easier.

3.1.1. C Boilerplate

Every C file should start with a short copyright notice:
 * Copyright <YYYY[-YYYY]> the Pacemaker project contributors
 * The version control history for this file may have further details.
 * This source code is licensed under <LICENSE> WITHOUT ANY WARRANTY.
<LICENSE> should follow the policy set forth in the COPYING file, generally one of "GNU General Public License version 2 or later (GPLv2+)" or "GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)".
Header files should additionally protect against multiple inclusion by defining a unique symbol.
#  define MY_HEADER_NAME__H

// header code here

#endif // MY_HEADER_NAME__H
Public API header files should additionally declare "C" compatibility for inclusion by C++, and give a Doxygen file description. For example:
#ifdef __cplusplus
extern "C" {

 * \file
 * \brief My brief description here
 * \ingroup core

// header code here

#ifdef __cplusplus

3.1.2. Line Formatting

  • Indentation must be 4 spaces, no tabs.
  • Do not leave trailing whitespace.
  • Lines should be no longer than 80 characters unless limiting line length significantly impacts readability.

3.1.3. Pointers

  • The * goes by the variable name, not the type:
char *foo;
  • Use a space before the * and after the closing parenthesis in a cast:
char *foo = (char *) bar;

3.1.4. Function Definitions

  • In the function definition, put the return type on its own line, and place the opening brace by itself on a line.
  • For functions with enough arguments that they must break to the next line, align arguments with the first argument.
  • When a function argument is a function itself, use the pointer form.
static int
function_name(int bar, const char *a, const char *b,
              const char *c, void (*d)())
  • If a function name gets really long, start the arguments on their own line with 8 spaces of indentation:
static int
        int bar, const char *a, const char *b,
        const char *c, const char *d)

3.1.5. Control Statements (if, else, while, for, switch)

  • The keyword is followed by one space, then left parenthesis without space, condition, right parenthesis, space, opening bracket on the same line. else and else if are on the same line with the ending brace and opening brace, separated by a space.
  • Always use braces around control statement blocks, even if they only contain one line. This makes code review diffs smaller if a line gets added in the future, and avoids any chance of bad indenting making a line incorrectly appear to be part of the block.
  • Do not put assignments in if or while conditionals. This ensures that the developer’s intent is always clear, making code reviews easier and reducing the chance of using assignment where comparison is intended.
a = f();
if (a < 0) {
} else if (some_other_condition) {
} else {
  • In a switch statement, case is indented one level, and the body of each case is indented by another level. The opening brace is on the same line as switch.
switch (expression) {
    case 0:
    case 1:

3.1.6. Operators

  • Operators have spaces from both sides.
  • Do not rely on operator precedence; use parentheses when mixing operators with different priority.
  • No space is used after opening parenthesis and before closing parenthesis.
x = a + b - (c * d);

3.2. Best Practices

The guidelines in this section offer technical advantages.

3.2.1. New Struct and Enum Members

In the public APIs, always add new struct members to the end of the struct. This allows us to maintain backward API/ABI compatibility (as long as the application being linked allocates structs via API functions).
This generally applies to enum values as well, as the compiler will define enum values to 0, 1, etc., in the order given, so inserting a value in the middle will change the numerical values of all later values, making them backward-incompatible. However, if enum numerical values are explicitly specified rather than left to the compiler, new values can be added anywhere.

3.2.2. Documentation

All public API header files, functions, structs, enums, etc., should be documented with Doxygen comment blocks, as Pacemaker’s online API documentation is automatically generated via Doxygen. It is helpful to document private symbols in the same way, with an \internal tag in the Doxygen comment.

3.2.3. Symbol Naming

  • All file and function names should be unique across the entire project, to allow for individual tracing via PCMK_trace_files and PCMK_trace_functions, as well as making detail logs easier to follow.
  • Any exposed symbols in libraries (non-static function names, type names, etc.) must begin with a prefix appropriate to the library, for example, crm_, pe_, st_, lrm_. This reduces the chance of naming collisions with software linked against the library.
  • Time intervals are sometimes represented in Pacemaker code as user-defined text specifications (e.g. "10s"), other times as an integer number of seconds or milliseconds, and still other times as a string representation of an integer number. Variables for these should be named with an indication of which is being used (e.g. interval_spec, interval_ms, or interval_ms_s instead of interval).

3.2.4. Memory Allocation

Always use calloc() rather than malloc(). It has no additional cost on modern operating systems, and reduces the severity of uninitialized memory usage bugs.

3.2.5. Logging

  • When format strings are used for derived data types whose implementation may vary across platforms (pid_t, time_t, etc.), the safest approach is to use %lld in the format string, and cast the value to (long long).
  • Be sure not to pass NULL as an argument to satisfy %s format specifier in logging and more generally printf style functions. When mere <null> string is a sufficient output representation in such case, there is crm_str convenient macro. Ternary operator is an apparent choice otherwise.

3.2.6. Regular Expressions

  • Use REG_NOSUB with regcomp() whenever possible, for efficiency.
  • Be sure to use regfree() appropriately.

3.2.7. vim Settings

Developers who use vim to edit source code can add the following settings to their ~/.vimrc file to follow Pacemaker C coding guidelines:
" follow Pacemaker coding guidelines when editing C source code files
filetype plugin indent on
au FileType c   setlocal expandtab tabstop=4 softtabstop=4 shiftwidth=4 textwidth=80
autocmd BufNewFile,BufRead *.h set filetype=c
let c_space_errors = 1

Chapter 4. Python Coding Guidelines

4.1. Python Boilerplate

If a Python file is meant to be executed (as opposed to imported), it should have a .in extension, and its first line should be:
which will be replaced with the appropriate python executable when Pacemaker is built. To make that happen, add an AC_CONFIG_FILES() line to, and add the file name without .in to .gitignore (see existing examples).
After the above line if any, every Python file should start like this:

# Pacemaker targets compatibility with Python 2.7 and 3.2+
from __future__ import print_function, unicode_literals, absolute_import, division

__copyright__ = "Copyright <YYYY[-YYYY]> the Pacemaker project contributors"
<BRIEF-DESCRIPTION> is obviously a brief description of the file’s purpose. The string may contain any other information typically used in a Python file docstring.
The import statement is discussed further in Section 4.2.1, “Python Future Imports”.
<LICENSE> should follow the policy set forth in the COPYING file, generally one of "GNU General Public License version 2 or later (GPLv2+)" or "GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)".

4.2. Python Compatibility

Pacemaker targets compatibility with Python 2.7, and Python 3.2 and later. These versions have added features to be more compatible with each other, allowing us to support both the 2 and 3 series with the same code. It is a good idea to test any changes with both Python 2 and 3.

4.2.1. Python Future Imports

The future imports used in Section 4.1, “Python Boilerplate” mean:
  • All print statements must use parentheses, and printing without a newline is accomplished with the end=' ' parameter rather than a trailing comma.
  • All string literals will be treated as Unicode (the u prefix is unnecessary, and must not be used, because it is not available in Python 3.2).
  • Local modules must be imported using from . import (rather than just import). To import one item from a local module, use from .modulename import (rather than from modulename import).
  • Division using / will always return a floating-point result (use // if you want the integer floor instead).

4.2.2. Other Python Compatibility Requirements

  • When specifying an exception variable, always use as instead of a comma (e.g. except Exception as e or except (TypeError, IOError) as e). Use e.args to access the error arguments (instead of iterating over or subscripting e).
  • Use in (not has_key()) to determine if a dictionary has a particular key.
  • Always use the I/O functions from the io module rather than the native I/O functions (e.g. rather than open()).
  • When opening a file, always use the t (text) or b (binary) mode flag.
  • When creating classes, always specify a parent class to ensure that it is a "new-style" class (e.g. class Foo(object): rather than class Foo:)
  • Be aware of the bytes type added in Python 3. Many places where strings are used in Python 2 use bytes or bytearrays in Python 3 (for example, the pipes used with subprocess.Popen()). Code should handle both possibilities.
  • Be aware that the items(), keys(), and values() methods of dictionaries return lists in Python 2 and views in Python 3. In many case, no special handling is required, but if the code needs to use list methods on the result, cast the result to list first.
  • Do not raise or catch strings as exceptions (e.g. raise "Bad thing").
  • Do not use the cmp parameter of sorting functions (use key instead, if needed) or the __cmp__() method of classes (implement rich comparison methods such as __lt__() instead, if needed).
  • Do not use the buffer type.
  • Do not use features not available in all targeted Python versions. Common examples include:
    • The html, ipaddress, and UserDict modules
    • The function
    • The subprocess.DEVNULL constant
    • subprocess module-specific exceptions

4.2.3. Python Usages to Avoid

Avoid the following if possible, otherwise research the compatibility issues involved (hacky workarounds are often available):
  • long integers
  • octal integer literals
  • mixed binary and string data in one data file or variable
  • metaclasses
  • locale.strcoll and locale.strxfrm
  • the configparser and ConfigParser modules
  • importing compatibility modules such as six (so we don’t have to add them to Pacemaker’s dependencies)

4.3. Formatting Python Code

  • Indentation must be 4 spaces, no tabs.
  • Do not leave trailing whitespace.
  • Lines should be no longer than 80 characters unless limiting line length significantly impacts readability. For Python, this limitation is flexible since breaking a line often impacts readability, but definitely keep it under 120 characters.
  • Where not conflicting with this style guide, it is recommended (but not required) to follow PEP 8.
  • It is recommended (but not required) to format Python code such that pylint --disable=line-too-long,too-many-lines,too-many-instance-attributes,too-many-arguments,too-many-statements produces minimal complaints (even better if you don’t need to disable all those checks).

Chapter 5. Evolution of the project

5.1. Foreword

This section is currently not meant as a definite summary of how Pacemaker got into where it stands now, but rather to provide few valuable pointers an entusiasts (presumably software archeologist type of person) may find useful. Moreover, well-intentioned contributors to Pacemaker project may want to review them occasionally since, as the famous quote has it, "those who do not learn history are doomed to repeat it".
For anything more talkative with less emphasis on actual code, other places will serve better for the time being (and if not, should not be too hard to volunteer extensions to those writeups):

5.2. Common ancestor: heartbeat project

Pacemaker can be considered as a spin-off from ‘heartbeat’, original comprehensive HA suite started by Alan Robertson, and some portions of code are shared, at least on the conceptual level if not verbatim, till today, even if the effective percentage continually declines. Note that till Pacemaker 2.0, it also used to stand for one (and initially the only) of supported messaging back-ends (removal of this support made for one such notable drop of reused code), see also pre-2.0 commit 55ab749bf.
The codebase for heartbeat used to be hosted at, but since that does not appear reliably available recently, an archive checkout from 2016 is shared at as a dedicated read-only repository, and anchored there, the most notable commits are:
Regarding the Pacemaker’s split from heartbeat, it evolved stepwise (as opposed to one-off cut), and the last step of full dependency is depicted in The Corosync Cluster Engine paper, fig. 10. This article also provides a good reference regarding wider historical context of the tangentially (and deeper in some cases) meeting components around that time.

5.2.1. Influence of heartbeat on Pacemaker

On a closer look, we can identify these things in common:
  • extensive use of data types and functions of GLib
  • Cluster Testing System (CTS) is inherited from initial implementation by Alan Robertson

Chapter 6. Advanced Hacking on the Project

6.1. Foreword

This section aims to be a gentle introduction (or perhaps, rather a summarization of advanced techniques we developed for backreferences) to how deal with the Pacemaker internals effectively. for instance, how to:
  • debug with an ease
  • verify various interesting interaction-based properties
or simply put, all that is in the interest of the core contributors on the project to know, master, and (preferably) also evolve — way beyond what is in the presumed repertoire of a generic contributor role, which is detailed in other sections of this guide.
Therefore, if you think you will not benefit from any such details in the scope of this section, feel free to skip it.

6.2. Debugging

In the GNU userland tradition, preferred way of debugging is based on gdb (directly or via specific frontends atop) that is widely available on platforms (semi)supported with Pacemaker itself.
To make some advanced debugging easier, we maintain a script defining some useful helpers in extra/gdb/gdbhelpers file, which you can make available in the debugging session easily when invoking it as gdb -x <path-to-gdbhelpers> ....
From within the debugger, you can then invoke the new pcmk command that will guide you regarding other helper functions available, so we won’t replicate that here.

6.3. Working with mocked daemons

Since the Pacemaker run-time consists of multiple co-operating daemons as detailed elsewhere, tracking down the interaction details amongst them can be rather cumbersome. Since rebuilding existing daemons in a more modular way as opposed to clusters of mutually dependent functions, we elected to grow separate bare-bones counterparts built evolutionary as skeletons just to get the basic (long-term stabilized) communication with typical daemon clients going, and to add new modules in their outer circles (plus minimalistic hook support at those cores) on a demand-driven basis.
The code for these is located at maint/mocked; for instance, based-notifyfenced.c module of based.c skeleton mocking pacemaker-based daemon was exactly to fulfill investigation helper role (the case at hand was also an impulse to kick off this very sort of maintenance support material, to begin with).
Non-trivial knowledge of Pacemaker internals and other skills are needed to use such devised helpers, but given the other way around, some sorts of investigation may be even heftier, it may be the least effort choice. And when that’s the case, advanced contributors are expected to contribute their own extensions they used to validate the reproducibility/actual correctness of the fix along the actual code modifications. This way, the rest of the development teams is not required to deal with elaborate preconditions, be at guess, or even forced to use a blind faith regarding the causes, consequences and validity regarding the raised issues/fixes, for the greater benefit of all.

Appendix A. Revision History

Revision History
Revision 1-0Tue Jul 26 2016Andrew Beekhof, Ken Gaillot
Convert coding guidelines and developer FAQ to Publican document
Revision 1-1Mon Aug 29 2016Ken Gaillot
Add Python coding guidelines, and more about licensing
Revision 2-0Fri Jan 12 2018Ken Gaillot
Drop support for Python 2.6
Revision 2-1Tue Sep 18 2018Jan Pokorný
Start documenting notable evolutionary points
Revision 2-2Fri Dec 7 2018Ken Gaillot
Update FAQ and C guidelines
Revision 2-3Mon May 13 2019Ken Gaillot, Jan Pokorný
Update copyright notice policy, and some external references
Revision 2-4Fri 17 May 2019Jan Pokorný
Start capturing hacking howto for advanced contributors
Revision 2-5Thu 5 Mar 2020Jan Pokorný
Use crm_str