DB2 data sharing recovery and duplexing

Two kinds of coupling facility trouble

IBM splits CF problems into:

• Structure failure — the structure is damaged but the CF LPAR may still be up
• Connectivity failure — this z/OS image cannot talk to the CF (channel, power, deactivated CF partition, CFCPC/LPAR failure)

A channel (link) failure is statistically more common than a whole CF death if you have only one link. Dual CF links and a preference list of two CFs in the CFRM policy are the hardware story; duplexing is the software twin.

Duplexing

Duplexing means z/OS maintains two allocated instances of a structure. Writes go to both. If one instance fails, Db2 deallocates the bad copy and keeps running on the good one. That switch is usually fast. You then repair the CF and reestablish duplexing (SETXCF START,POL or the duplexing rebuild Db2/z/OS drives when DUPLEX(ENABLED) is in the policy).

Group buffer pool duplexing

Changed pages that would otherwise live in only one GBP are also in the secondary. Castout still runs from the primary; the secondary is for failover, not a second castout engine. GBP duplexing protects against losing changed pages that have not yet been written to disk. It costs extra CF cache bandwidth. Many production groups duplex GBP0 (catalog and directory) and the busy user GBPs.

SCA and lock structure duplexing

The shared communications area (SCA) holds group-wide control information. LOCK1 holds the lock table and modify-lock list. Duplexing them avoids bringing members down when a single CF or link is lost. Simplexed SCA rebuild from virtual storage is usually quick. Simplexed lock rebuild is also often quick, but elapsed time grows with the number of modify locks held at the failure.

If Db2 cannot rebuild simplexed SCA and LOCK1 from member storage into an alternate CF (preference list + enough CF storage), IBM documents that all active members abend and you perform a group restart so the structures can be rebuilt from the logs. That is the nightmare scenario duplexing is meant to avoid.

Structure rebuild

A rebuild allocates a new structure (often on another CF) and copies or reconstructs content. Operators can request it with SETXCF START,REBUILD. Automatic rebuild depends on:

• An active sysplex failure management (SFM) policy
• The CFRM preference list having a CF that still has connectivity and space
• REBUILDPERCENT compared with the weighted loss of connectivity

Rebuild of a simplexed GBP is the painful case: Db2 tries to allocate the current size on the alternate CF. If storage cannot be allocated, changed pages are written to disk instead of staying in the GBP. AUTOREC(YES) on the GBP can start automatic recovery; otherwise you may need START DATABASE to pull pages from the log. Plan CF SIZE so a rebuild target actually fits.

REBUILDPERCENT

REBUILDPERCENT is an SFM policy keyword, not a Db2 DSNZPARM. z/OS computes a weighted loss of systems’ connectivity to the structure. Roughly:

• Active SFM policy and weighted loss not less than REBUILDPERCENT → automatic rebuild (good path; look for DSN7503I-class messages)
• Weighted loss is less than REBUILDPERCENT, or no SFM policy → affected members may come down (DSN7501A / reason codes such as 00F70600 for SCA). Structure can remain allocated. Operations fix the link, restart the member on a connected system, or rebuild manually onto another CF.

The exact tables differ slightly for SCA, LOCK1, and non-duplexed GBPs, but the idea is the same: SFM + REBUILDPERCENT decide “rebuild in place” versus “sacrifice the member.” Duplexed structures often switch to the copy with good connectivity instead of taking that rebuild-or-die choice.

Failover and member recovery

Member failure (abend, z/OS outage) is not the same as CF failure. Work should move: sysplex distributor / DVIPA for DDF, CICSPlex for CICS, WLM for batch. The failed member still holds retained locks (modify locks left in LOCK1) until restart recovery runs. Other members cannot update those protected pages. Restart the member as soon as the LPAR is up. Deferring restart of objects with retained page set P-locks can put pages on the logical page list (LPL) so the rest of the page set is not locked out—an advanced recovery tactic, not a first-day habit.

Group restart is heavier: the group cannot form until SCA/LOCK1 are rebuilt from logs. Disaster-recovery playbooks also talk about a single surviving member at a remote site (lock structure + SCA still required; GBPs can be small after you are truly one-system).

Logs are shared in the sense that every member can read the others’ logs. Recovery uses LRSN (log record sequence number), not a single-subsystem RBA, when you name a point in time.

Data sharing performance

Recovery design and performance share the same structures. Watch:

• CF service times — sync requests that convert to async are a yellow flag
• Lock false contention — grow LOCK1 lock table
• GBP directory reclaims / XI — directory too small or too much inter-member update interest
• Castout thresholds — CLASST, GBPOOLT so the GBP does not fill with changed pages
• Page set GBP-dependency — objects ping-ponging between members

Application COMMIT frequency still matters: long units of work hold modify locks in the record list and make a lock-structure rebuild slower. Binding with a reasonable isolation (CS, currently committed) reduces real contention that recovery later has to wait on.

Data sharing troubleshooting starter kit

1
2
3
4
5
6
7
-DISPLAY GROUP DETAIL
-DISPLAY GBPOOL(GBP0) GDETAIL
-DISPLAY BUFFERPOOL(BP0) GDETAIL
D XCF,STR
D XCF,STR,STRNAME=DSNCAT_LOCK1
D XCF,STR,STRNAME=DSNCAT_SCA
D XCF,STR,STRNAME=DSNCAT_GBP0

• Member missing from DISPLAY GROUP → restart or coupling problem, not “a bad SQL”
• Retained locks after an abend → restart that member; do not REORG the table as a first guess
• Pages in LPL/GRECP → START DATABASE with the recovery option your runbook names
• Sudden CF sync time spike → RMF CF activity, link errors, CF CPU, duplexing rebuild in progress
• Keep CFRM policy, SFM policy, and IPL parameters in the same change as a new CF

Explain It Like I'm Five

The coupling facility is a shared whiteboard. Duplexing is writing on two whiteboards in two rooms so if one room’s lights go out, the other still has the notes. Rebuild is grabbing a new whiteboard and copying everything over. REBUILDPERCENT is the fire-alarm rule: if enough kitchens lose the hallway, automatically copy the board; if only one kitchen loses it, that kitchen sits down until someone fixes the door. Failover is sending the cooks to another kitchen. Retained locks are “nobody else may touch this pot until the missing cook comes back and says what happened.”

Exercises

1. List GBP, SCA, and LOCK1 and say whether your site duplexes each one.
2. Explain why a simplexed GBP failure is usually worse than a duplexed GBP failure.
3. In your own words, what question does REBUILDPERCENT answer?
4. Why must a crashed member be restarted even if WLM already moved the workload?
5. Write the DISPLAY and D XCF commands you would run after a CF link alert.