The Book of Db2 Pacemaker – Chapter 3:  Pacemaker Resource Model: Into the Pacemaker-Verse

Let me openly admit that my first treatment of this topic was a very thorough top-down, lecture style, technical approach … until Michael Keaton inspired me to do it differently with his recent appearance in “The Flash” movie where he used 3 to 4 uncooked Spaghetti to explain the mind-bending “Multiverse” concept !  It was then I decided to borrow a page and use the following analogy to explain the resource model concept in Pacemaker.

In simple terms, it’s “Cause and Effect”.  The patient in the diagram above is in need of medical care to treat whatever symptoms he exhibits.  The various questions (circled in black) represent different types of monitoring setup (the Cause) at different frequency with the potential actions in red (the Effect).  The dotted green lines represent dependency or relationship between certain symptoms.  For example, the body temperature will be continuously monitored to increase/stop the Drug C through IV if the temperature reaches certain threshold.  A slower or ceased heartbeat will likely trigger an alarm to the nurse station for immediate CPR.  The bottom line – the patient has a complete medical plan with a single goal in mind – keep him alive with precision monitoring and prescriptive actions to remedy any situation.

The complete medical plan is essentially what a resource model is all about.  It is designed with target monitoring with prescriptive actions.  Below is a simple mapping between a Pacemaker cluster (left side in green) and the patient’s medical plan (right side in orange)

The components being monitored are referred to as “Resources”.  The various machineries setup to monitor the components are the “monitoring” logic in the Resource Agent.  The actions to be carried out based on the monitoring results are the “actions” implemented in the Resource Agent. 

Now, you may ask – “Where’s Db2 in all of this?”  Well, you don’t see the references of Db2 entities in the middle column above because in Pacemaker terms, HADR databases and Db2 instances are all just “resources”.  “Instances” and “Databases” are simply Db2 concepts or naming convention placed on top to better identify them in the resource model and for DBA to determine their inter-relationships.

Let’s take a step back and summarize all the ingredients in a Db2 HA resource model with Pacemaker:

·       Resources – these are components needed to be monitored to ensure the health of the cluster.

·       Resource agents – these are opensource UNIX shell scripts modified and maintained by Db2 (users should NOT tamper with this or risk voiding your support contract) for monitoring the resources and actions to recover from specific failure.  It is important to highlight that for Db2 Integrated Pacemaker solution, only the Db2 supplied version of the resource agents can be used.  The vanilla opensource one(s) will not work.

·       Constraints – these are relationships among the resources defined by Db2.  They have profound impact on the recovery behaviours.  There are three types of Pacemaker constraints that Db2 utilizes to build our resource model (for all high availability configurations – HADR, MF, and in future, DPF, and pureScale).

1.     Location constraint – restricts where a resource can run.

2.     Colocation constraint – enforces the location of a resource with another.

3.     Ordering constraint – defines a dependency of one resource on another.

So far, everything explained is generic to all Db2 High Availability models – HADR, Mutual Failover (also DPF and pureScale in future).  It’s time to zero into the HADR resource model and walk through a few resource outputs to familiarize with the concepts.

Db2 HADR Instance and Network adapter resources with Location Constraint

The combined constraints essentially tell Pacemaker the following:

Ethernet adapter “enc4209” which is physically located in host svlxtorc is preferred to start on host svlxtorc only.  There is no need to attempt to start on host svlxtord because we know it is going to fail.  The same is true for the resources of the two instances, “alan”, and “svtdbm” on svlxtorc.

Db2 HADR Virtual IP resource with Ordering Constraint

The order constraint is used to create a dependency so that the VIP resource cannot be started until the corresponding database resource is online on the host.

Db2 HADR Primary and Standby VIP with Colocation Constraint

The colocation constraint is created when we want one resource to always be online on the same host as another resource.  The example above is our Virtual IP resources, the desired behaviour is:

•        The primary VIP resource to run on the same host as the associated database that has the primary role (Master)

•        The standby VIP resource to run on the same host as the associated database that has the standby role (Slave)

Putting all ingredients and the constraints of all resources together, you can easily infer the failure behaviour when certain resource fails.  The two diagrams below summarize the various relationships in both HADR and Mutual Failover resource model. For details on these two configurations, refer to Configuring high availability with the Db2 cluster manager utility (db2cm).

HADR resources dependency diagram

Here’s what the HADR resource model gives you:

·       The instance resource (the lone green box, second level from the top) sits at the top of the resource model.  Its state (online or offline) corresponds to whether the Db2 instance (db2sysc) is running on the host or not. 

·       The instance resource state depends solely on whether the Db2 system process (db2sysc) is active or not (depicted by the lone blue box at the third row in the diagram).  Even though the HADR DB resources sit at the same level as db2sysc, it simply describes the standard Db2 process model relationship between the Db2 instance and the databases. (A database can only be activated with a running instance.) But in the resource model, there is no constraint setup between HADR DB and Instance resource.  On the other hand, this also means that even if all HADR DBs are disabled, the instance will not failover automatically.

·       If the instance resource fails, all HADR DBs previously with primary roles under the dead instance will be brought up as primary on the new primary (old standby) host.

·       For each HADR database, it requires the specified ethernet adapter to be functioning (for log shipping purpose).  If that adapter malfunctions or if the database becomes non-responsive, a takeover will happen, and the primary role fails over to the standby host.

·       The primary VIP resource flows with the database with primary role.  The standby VIP resource flows with the database with standby role.

From the resource dependency diagram, one can deduce when an automated database failover will occur.  It is summarized in the table below.

Mutual Failover (MF) resources dependency diagram

Here’s what the Mutual Failover resource model gives you:

·       The partition resource is the parent of all other resources.  This means if ANY of the dependent resources fail, the whole partition fails over to the standby host.

·       Different from HADR, mount monitoring is supported in this resource model.

·       The partition resource depends on a functioning instance directory mount point (that’s where the sqllib resides).

·       The mount points of database log path and mirrored log path are part of the resource model.  The partition stays up as long as one of these two log paths are accessible.  This is one of the new valued HA features with Pacemaker over the old TSA model.

·       The primary VIP resource flows with the partition resource (unlike in HADR where it flows with the database with primary role).

From the resource dependency diagram, one can deduce when a partition failover will occur.  It is summarized in the table below.

This brings me to the end of content in this blog.  What do you think?  Have I over-simplified the anatomy of Db2 Pacemaker resource model?  I certainly hope not as the engineering team’s goal since day one was to make this as simple and as easily understandable as possible.  There might be a few quirks that I wasn’t able to cover, but hey … that’s why this is Chapter 3 and not the finale … there’s always the next Chapter to look forward to.   Speaking of next chapter … Let me end with a teaser to the next title in the series – The Book of Db2 Pacemaker – Chapter 4: The Quorumania!

Once again, if you have a specific topic that you would like to see, feel free to reach out to me.

Quick reference to this series:

Chapter 1 - Red pill or Blue pill ?

Chapter 2 - Pacemaker Cluster … Assemble!

Alan Lee joined IBM in 1999 and spent his first two years as the UNIX development infrastructure team lead focusing on improving overall development efficiency. He has been with the Db2 kernel development team since 2001. His early contributions spanned across various layers in Db2 kernel - operating system services, file system features exploitations, pureScale infrastructure, object storage support in AWS and IBM Cloud. His latest achievement included bringing pureScale to AWS Marketplace, reducing deployment from weeks/months to minutes. Development roles aside, he led the Db2 LUW Support organization in all geos, significantly improved its support Net Promoter Score (NPS) by double digit within a year. Currently, he is the product owner of Db2 pureScale & high availability and upline development manager of Security and Db2/SAP partnership.