Message Persistence Internals

Application developers set MQPMO_DEFAULT and hope messages survive restart; message persistence internals explain what the queue manager actually does with those bytes on disk. Persistent messages for a local queue live in queue data files managed by the queue manager, updated in coordination with the log described in the logging internals tutorial. Non-persistent messages trade durability for latency by staying in memory paths whenever possible. Beginners confuse DEFPSIST on the queue with the persistence option on an individual put—both matter. Capacity planners who ignore file sizes under queues and log directories fill disks and stop all persistent traffic. This tutorial covers distributed queue file layout at conceptual level, pages and indexing intuition, default persistence attributes, put and get paths for persistent versus non-persistent, depth limits and file extension, browsing and destructive read effects on storage, syncpoint commit and backout at storage layer, performance implications, z/OS contrast with coupling facility shared queues, and operational commands to inspect depth and file problems without corrupting data by manual deletion.

From MQPUT to Disk

1. Application PUT with persistence on enters queue manager agent.
2. Log records describe message addition and index update.
3. On commit, log forced; queue file pages updated with message body and metadata.
4. MQMD fields including MsgId, priority, expiry stored with message.
5. GET with syncpoint may delete or mark entry per browse options.

Queue File Concepts

DEFPSIST and Put Options

DEFINE QLOCAL PAYMENT DEFPSIST YES makes persistence the default for applications that do not specify otherwise—appropriate for payment queues. DEFPSIST NO suits telemetry where loss on crash is acceptable. MQPMO_LOGICAL_ORDER and persistence flags on individual put override defaults. Mixing styles on one queue confuses testers who see some messages survive restart and some do not.

1
2
3
4
5
6
7
DEFINE QLOCAL('ORDERS.IN') REPLACE +
  DEFPSIST(YES) MAXDEPTH(100000) MAXMSGL(4194304) +
  DESCR('Persistent order queue')
 
# Put options (conceptual):
# Persistent + syncpoint → survives restart after commit
# Non-persistent → fast, lost on crash

Explainer: Safe Deposit Box Versus Sticky Note

Persistent messages go into the bank safe deposit box (queue file plus log). Non-persistent messages are sticky notes on a whiteboard—they are fast to write but gone if someone erases the board during a power cut.

MAXDEPTH and Disk Space

MAXDEPTH limits message count, not bytes directly—deep queues of large messages still need gigabytes on disk. Monitor queue directory filesystem separately from log filesystem if split. CURDEPTH from DISPLAY QSTATUS shows count; estimate bytes with average message size sampling.

Browse Versus Destructive Get

Browse leaves message in file; destructive get removes entry and frees space eventually after commit. Long browse locks can block space reuse. Transactional browse under syncpoint behaves per get options—read IBM details for MQGMO_BROWSE_FIRST and friends.

Non-Persistent Fast Path

High-rate telemetry may use non-persistent messages and NO DEFPSIST to avoid log churn. Accept data loss on failure. Never use for money movement without explicit business sign-off.

z/OS Shared Queues Contrast

Queue sharing groups store shared messages in coupling facility structures with different internals than single queue files on Linux. Persistence and availability semantics feel similar to applications but operations use different DISPLAY and structure sizing commands—see z/OS track tutorials.

Failure Symptoms

• MQRC STORAGE_NOT_AVAILABLE — disk full or quota.
• Messages missing after restart — were non-persistent or uncommitted.
• Queue file damage — requires IBM recovery tools; do not delete files.

Step-by-Step: Observe Persistence in Lab

1. Put ten persistent messages; commit; note CURDEPTH.
2. endmqm; strmqm; verify depth still ten.
3. Put ten non-persistent; kill -9 queue manager; restart; compare depth.
4. Record disk usage in queues directory before and after purge.

Explain Like I'm Five: Message Persistence Internals

Persistence means writing the letter in a locked filing cabinet, not just leaving it on a desk where the wind can blow it away.

Practice Exercises

Exercise 1

DISPLAY QLOCAL and explain DEFPSIST for three queues in your lab.

Exercise 2

Estimate disk for 1 million 4KB persistent messages.

Exercise 3

Document difference you saw between persistent and non-persistent after unclean shutdown test.

Frequently Asked Questions

Test Your Knowledge