DB2 restart and restart recovery

Unplanned outages happen. High availability on DB2 for z/OS is not “never fail” — it is “come back consistent, fast, and with the rest of the group still serving work.” This page covers Db2 restart, the restart recovery process that uses checkpoints and logs, member restart versus group restart in data sharing, and quiesce as a planned stop.

High availability
Progress0 of 0 lessons

Db2 restart

You start and stop Db2 with the operator commands -START DB2 and -STOP DB2. Before a normal stop, the system takes a shutdown checkpoint. That checkpoint plus the recovery log (inventoried in the BSDS) give Db2 what it needs to restart. After an IPL, START DB2 is issued from a z/OS console or SDSF — DB2I cannot start a subsystem that is not up.

text
1
2
3
4
5
-START DB2 -START DB2 ACCESS(MAINT) -START DB2 PARM(DSNZPACC) -STOP DB2 MODE(QUIESCE) -STOP DB2 MODE(FORCE)
  • ACCESS(RW) — normal: TSO, batch, CICS, IMS, DDF can connect after restart completes
  • ACCESS(MAINT) — restrict connections so you can repair before opening the shop
  • PARM(member) — which DSNZPARM load module to use
  • LIGHT(YES | NO | NOINDOUBTS) — data sharing light restart: bring a member up quickly (often on another LPAR) to recover retained locks without a full workload restart

IRLM must be started first or START DB2 fails. Automatic restart (ARM / automation) is IBM’s recommendation for failed data-sharing members so humans are not the recovery clock.

Restart recovery

If a system failure occurs, a restart of Db2 restores data integrity. A Db2 subsystem or an attached subsystem (CICS, IMS) might fail. In either case Db2 automatically restarts, backs out uncommitted changes, and completes processing of committed changes that had not yet been externalized to disk. That is restart recovery — not the RECOVER utility, and not disaster recovery to another site.

Normal restart phases
PhaseWhat Db2 does
Log initializationOpen BSDS and logs; find the last checkpoint RBA/LRSN
Current status rebuild (CSR)Identify in-flight URs, page sets with pending writes, exception states
Forward-log recoveryRedo: reapply log from checkpoint to end of log for committed work still in buffers
Backward-log recoveryUndo: back out uncommitted units of recovery
End restartSubsystem becomes available (ACCESS RW or MAINT); remaining work such as postponed abort may continue

Watch the DSNR-prefixed messages. DSNR002I / DSNR006I are expected progress. DSNR004I points at trouble in forward-log recovery; DSNR005I in backward-log recovery; DSNJ001I during log initialization or CSR. If a portion of the log is unreadable, you may need conditional restart (DSNJU003 CRESTART) — a last-resort procedure that can skip work and must be planned with IBM documentation, not improvised.

Dual BSDS and dual logs let you restore dual operation while Db2 continues to run if one copy is lost. Restart recovery assumes those logs exist. If you lose too much log, you are in system recovery (image copies, RESTORE SYSTEM), not a simple START DB2.

Redo and undo

From the last checkpoint, forward recovery redoes logged changes so disk matches committed work that was still sitting in virtual buffers. Backward recovery undoes in-flight units of recovery (URs) so a half-updated row is not visible. Indoubt URs (two-phase commit with CICS, IMS, RRS, or remote Db2) are handled with heuristic or coordinator-driven resolution after restart — they are not silently committed.

Long URs can make backward recovery slow. Some shops allow postponed abort: Db2 comes up and finishes backing out huge URs in the background so the subsystem is available sooner. That is a ZPARM / restart option trade-off, not a way to skip recovery.

Db2 member restart

In a data sharing group you can stop and start an individual member while the other members continue to run. Startup for each member is similar to a non-data-sharing subsystem: BSDS, logs, checkpoint, redo, undo. The difference is retained locks and the coupling facility:

  • When a member fails, IRLM/XES can retain its update locks so other members cannot overlay uncommitted pages
  • Those locks remain until the failed member restarts and recovers (or a LIGHT restart of that member releases them)
  • Other members keep running — that is the HA value of data sharing

IBM recommends an automated procedure to restart failed members. Leaving a member down holds retained locks and can look like “the database is hung” when it is really one dead member’s locks.

text
1
2
3
-DIS GROUP -START DB2 -START DB2 LIGHT(YES)

DISPLAY GROUP shows which members are ACTIVE, FAILED, or QUIESCED. A FAILED member needs restart. LIGHT(YES) is for “get the retained locks released on another LPAR,” not for serving the day’s CICS workload on that member.

Group restart

Group restart is the rare event that critical resources in a coupling facility are lost and cannot be rebuilt. The SCA (shared communications area) and the lock structure (LOCK1) are required for the group to function. If they disappear and z/OS cannot rebuild them on an alternate CF, all members terminate abnormally. Group restart rebuilds the lost information from individual member logs.

Unlike data recovery, this information can be applied in any order — there is no need to merge log records in LRSN sequence the way RECOVER does — so Db2 can run many restart phases for individual members in parallel. Elapsed time is dominated by the member with the longest CSR, and if the lock structure was lost, the member with the longest forward-log recovery.

If you do not START DB2 for every member, the started members perform group restart on behalf of the non-starting members by reading those members’ logs (peer CSR / peer forward recovery). You should still restart all non-quiesced members together: retained locks are held for members that stay down, which hurts data availability.

  • Lost SCA — CSR rebuilds the SCA (exception statuses, BSDS names, recovery information)
  • Lost lock structure — CSR reacquires page-set P-locks; forward-log recovery rebuilds retained locks
  • Lost both — combination of the above

The real HA design is to avoid group restart: failure-isolated CFs, structure rebuild onto an alternate CF, and duplexing. Group restart is the safety net when those designs were not in place or both CFs were lost.

Quiesce

Beginners meet three different “quiesce” ideas. This page cares about the first two:

STOP DB2 options
OptionMeaning
MODE(QUIESCE)Wait for threads; shutdown checkpoint; clean stop
MODE(FORCE)Roll back inflight work more aggressively; faster stop, more restart work
CASTOUT(YES|NO)Whether to cast out GBP-dependent pages as part of stop (data sharing)

MODE(QUIESCE) is the planned stop: let threads finish, take a shutdown checkpoint, leave a clean restart. In data sharing, IBM also uses quiesce for a member you stop and leave dormant — for example extra capacity you only need in December. That member still appears in DISPLAY GROUP with status QUIESCED until you start it again. A quiesced member can remain dormant indefinitely. Permanently deleting a member from the group is a separate procedure.

Do not confuse that with the QUIESCE utility, which writes a consistency point (RBA/LRSN) into SYSIBM.SYSCOPY for listed table spaces so RECOVER can use a timestamp. And do not confuse it with -STOP DATABASE or restricted ACCESS, which are object-level, not member-level.

BACKUP SYSTEM requires that no failed or abnormally quiesced members exist in the group, or the backup request fails. Planned QUIESCED members are the dormant kind; a crashed member left FAILED is what blocks a system-level backup.

Explain It Like I'm Five

Db2 keeps a diary (the log) and occasionally writes “I got this far” in the diary (a checkpoint). If the lights go out, restart reads the last “I got this far,” finishes homework that was already marked complete (redo), and erases homework that was only half done (undo). In a classroom with several Db2 teachers (members), one teacher can go home while the others keep teaching — that is member restart. If the shared whiteboard in the hallway (SCA or lock structure) is destroyed and cannot be replaced, every teacher has to stop and rebuild the whiteboard from their diaries — that is group restart. Quiesce is sending a teacher on planned leave so they are not in the building, but their name is still on the staff list.

Exercises

  1. List the restart phases in order and name which one undoes in-flight URs.
  2. Write the operator commands to stop a member cleanly and to start it with ACCESS(MAINT).
  3. Why should automation restart a FAILED member instead of waiting for the next shift?
  4. When would you START DB2 LIGHT(YES) instead of a normal member restart?
  5. Explain why losing GBP0 is a different problem from losing LOCK1.

Quiz

Test Your Knowledge

1. What does restart recovery do after an abnormal termination?

  • Only runs COPY
  • Uses the shutdown or system checkpoint and the recovery log to complete committed work (redo) and back out uncommitted work (undo)
  • Drops the catalog
  • Starts DDF only

2. When is group restart required?

  • Every time you STOP DB2 MODE(QUIESCE)
  • When critical coupling facility structures (SCA and/or lock structure) are lost and cannot be rebuilt, so members terminate abnormally
  • Only after RUNSTATS
  • Only to start IRLM

3. What is a quiesced data sharing member?

  • A member that no longer appears in DISPLAY GROUP
  • A member stopped so it is dormant (temporary or indefinite); DISPLAY GROUP shows status QUIESCED until you START it again
  • A member that owns all GBPs
  • A utility-only address space

4. Which command starts the subsystem after an IPL or after STOP DB2?

  • BIND PACKAGE
  • -START DB2 from a z/OS console (or SDSF), because Db2 is not up yet
  • Only SPUFI
  • -DISPLAY DATABASE

5. What is the difference between member restart and group restart?

  • There is no difference
  • Member restart recovers one failed member while others keep running; group restart is a coordinated recovery of SCA/lock information from all members’ logs
  • Group restart only applies to DDF
  • Member restart always rebuilds GBP0

Frequently Asked Questions