DB2 disaster recovery and continuous availability

Layers of availability

IBM’s backup/recovery/restart chapter says unplanned outages are hard to avoid entirely; a good strategy reduces elapsed time. You can recover data to current or to an earlier point (table spaces, indexes, partitions, data sets) and you can back up an entire subsystem or data sharing group. Disaster recovery is that last sentence plus geography.

CF failure and CF connectivity

A Coupling Facility can fail, or a member can lose connectivity (links, CFLEVEL mismatch, CFRM policy) while the CF hardware is fine. Effects depend on which structure was hit:

• Duplexed GBP — switch to secondary; applications continue; rebuild directory over time
• Simplex GBP — recover changed pages that were not cast out; GBP recovery, possible stalls for GBP-dependent objects
• SCA or LOCK1 with an alternate CF — structure rebuild; members may pause but the group can survive
• SCA or LOCK1 with no rebuild path — all members abend; group restart

Connectivity planning includes enough CF links, a preference list with a second CF, and enough SIZE on the backup CF. D XCF,STR shows policy and allocated CF names. If you stretch data sharing across sites, CF access time becomes a performance and timeout problem as well as an HA problem — that is why many multi-site designs use GDPS with a primary CF locality rather than chatty GBPs across a long distance.

Failover

Failover means another resource takes over:

• Member failover — group attach / Sysplex Distributor / CICS workload routing sends new work to surviving members; restart the failed member to free retained locks
• Structure failover — duplex secondary becomes primary, or rebuild on another CF
• Disk failover — HyperSwap / Metro Mirror swaps to the secondary volume set
• Site failover — GDPS or a rehearsed remote IPL of z/OS and START DB2 against mirrored or restored volumes

Failover that is not rehearsed is a hope, not a plan. Track RTO (how long until service returns) and RPO (how much committed work you can lose). Synchronous mirroring aims at near-zero RPO; asynchronous mirroring trades RPO for distance.

Disaster recovery and remote site recovery

Remote site recovery is restoring Db2 after the local site is lost. You must bring up a consistent set of:

• User data volumes (or restored image copies)
• Active/archive logs and BSDS
• Catalog and directory (they are just more page sets, but they must match the logs)
• IRLM, ZPARMs, and CFRM if you restart as a group

Classic tape/vault DR: ship image copies and archive logs, recover objects or the system at the remote site, accept hours of RTO. System-level backups speed the restore. Disk mirroring plus GDPS can make the remote volumes already current. Conditional restart and log truncation appear when the remote log is not a perfect dual of the primary.

For utilities that were running, IBM documents restarting them on another z/OS system and using remote-site procedures so a local disaster does not leave half-applied REORG work as the only copy.

System-level backup and FlashCopy

The BACKUP SYSTEM utility invokes z/OS DFSMShsm to copy the volumes on which Db2 data and log information reside. You can back up a subsystem or a data sharing group. Later, RESTORE SYSTEM recovers that subsystem or group.

• All data sets you copy must be SMS-managed
• You must define copy pools (and copy pool backup storage groups) before you run BACKUP SYSTEM
• In data sharing, no failed or abnormally quiesced members may exist or the request fails
• History is recorded in the BSDS
• BACKUP SYSTEM does not reset COPY-pending — take an image copy for that
• Authorization: privilege set including SYSCTRL or SYSADM

Phases are UTILINIT, COPY, UTILTERM. FULL copies data and log copy pools; DATA ONLY copies data. RESTORE SYSTEM can restore the data copy pool and apply log, or LOG ONLY. BACKUP SYSTEM is less disruptive than SET LOG SUSPEND and has group scope; SET LOG SUSPEND is member scope.

FlashCopy is the storage instantaneous-copy service HSM/DSS calls. That is why a system-level backup can complete without a long Db2 outage: the split is a few seconds of establish, then background copy. Protect against media failure by putting copy pool and backup on separate extent pools / media within the rules of your DS8000 (FlashCopy typically cannot cross certain storage-facility boundaries).

From Db2 9 onward, system-level backups can also feed object-level RECOVER if SYSTEM_LEVEL_BACKUPS is YES — useful when you do not want to restore the whole system for one table space.

1
2
3
4
5
6
//SYSIN    DD *
  BACKUP SYSTEM FULL
/*
//SYSIN    DD *
  RESTORE SYSTEM
/*

GDPS

GDPS (Geographically Dispersed Parallel Sysplex) is IBM’s product family for multi-site z/OS availability. In a Db2 conversation it means: disk replication (Metro Mirror, Global Mirror, and related), planned and unplanned site switches, and automation so operators are not inventing START DB2 order at 3 a.m. Continuous availability designs often combine:

• Data sharing members on more than one CEC (local HA)
• Duplexed GBPs and isolated or duplexed SCA/LOCK1
• Synchronous disk mirroring between nearby sites (near-zero RPO)
• GDPS HyperSwap or site takeover runbooks

GDPS does not replace image copies or BACKUP SYSTEM for logical damage (dropped table, bad application). Mirroring happily mirrors a DROP. You still need point-in-time recovery skills.

Continuous availability versus DR drills

Continuous availability is the goal that users never notice member restart, CF takeover, or even a site switch. It costs extra CFs, extra CECs, extra links, and operational discipline (no single-CF production, no “we will duplex later”).

Disaster recovery accepts an outage window. You still need current copies, tested RESTORE SYSTEM / RECOVER jobs, documented conditional restart, and a remote CFRM policy if the remote sysplex is not a mirror of the primary. Practice the drill: a backup you have never restored is not a backup.

Group buffer pools sit in both stories. Locally they are the cache that must survive CF failure (duplex them). At a remote site they are empty structures you allocate after the disks are there — you do not “restore a GBP”; you restart members and let them rebuild interest and cache.

Explain It Like I'm Five

High availability is having two ovens in the same kitchen so dinner continues if one oven breaks. Disaster recovery is having a second house across town with a copy of the cookbook and the freezer. FlashCopy is a magic photocopier that copies the whole freezer in a blink. GDPS is the moving company that already parked a second freezer at the other house and knows the driving directions when the first house floods. Continuous availability is cooking in both houses so guests never wait. Restart recovery is just turning the same oven back on after a fuse blew.

Exercises

1. Write RTO and RPO targets for (a) one member abend, (b) one CF loss, (c) site loss.
2. Why must copy pools exist before BACKUP SYSTEM, and why must data sets be SMS-managed?
3. Explain why disk mirroring does not protect you from a mistaken DROP TABLE.
4. List what a remote site needs besides table space image copies.
5. When is duplexing a GBP enough, and when do you still need GDPS?