SSL Handshake Failures

SSL handshake failures stop IBM MQ channels before CHLAUTH, before sequence numbers, and before any business message moves. Operators see BINDING or RETRY with errors referencing SSL, TLS, GSKit, or cipher names in the error log. Because TLS is symmetric, both queue managers and any middleboxes must agree on protocol version, cipher suite, and trust anchors. Beginners often configure SSLCIPH on the sender only, install a new personal certificate but forget the intermediate CA, or enable SSLCAUTH(REQUIRED) while the partner still runs OPTIONAL. This tutorial explains handshake mechanics in plain language, the role of SSLKEYR and partner trust stores, diagnosing cipher and certificate problems, mutual TLS requirements, expiry calendars, and how to separate handshake failures from bind and SSLPEERMAP failures that happen one step later.

What Happens During the Handshake

After TCP connect, the TLS client hello lists supported protocols and ciphers. The server hello selects a mutually acceptable option, sends its certificate chain, and may request a client certificate if SSLCAUTH requires it. Keys are derived and encrypted application data can flow. IBM MQ uses the GSKit (or platform TLS stack) underneath channel attributes SSLCIPH and SSLCAUTH. If no cipher overlaps, handshake aborts. If the receiver does not trust the sender cert signer, handshake aborts. If the client cert is required but missing, handshake aborts. None of that is fixed by increasing SHORTRTY.

SSLCIPH and Cipher Negotiation

SSLCIPH on DEFINE CHANNEL lists the cipher specification IBM MQ uses for that channel. Sender and receiver must negotiate a suite both support. A channel defined with a strong AES-GCM suite fails against a legacy partner limited to older RSA suites until one side is updated or a transitional cipher is agreed for a migration window. Document approved cipher lists per security policy; do not copy random values from old guides. After ALTER CHANNEL SSLCIPH, restart or cycle the channel instance per platform guidance and test in lab with the same GSKit levels as production.

Certificates and SSLKEYR

1
2
3
4
5
ALTER QMGR SSLKEYR('/var/mqm/qmgrs/QM1/ssl/key.kdb')
DEFINE CHANNEL('SECURE.TO.PARTNER') CHLTYPE(SDR) TRPTYPE(TCP) +
  SSLCIPH(ANY_TLS12_OR_HIGHER) SSLCAUTH(REQUIRED) CONNAME('partner(1414)')
* Partner RCVR must trust your cert CA and present client cert if mutual TLS
DISPLAY CHANNEL('SECURE.TO.PARTNER') SSLCIPH SSLCAUTH

The key repository holds the queue manager personal certificate and trusted CAs. Partners import your CA or leaf as they require. When you renew, update the repository and verify the label referenced by the channel if your platform uses certificate labels. Missing intermediate certificates cause trust failures even when the leaf is valid—export the full chain from PKI and add intermediates to SSLKEYR. Monitor notAfter dates with the same rigor as application certs.

SSLCAUTH: OPTIONAL Versus REQUIRED

SSLCAUTH(OPTIONAL) on RCVR allows partners to connect with server-side TLS only—the client cert is not mandatory. SSLCAUTH(REQUIRED) demands a client certificate for mutual TLS. If your RCVR requires mutual TLS but the partner SDR has no client cert configured, handshake fails. Align policy on both sides before go-live. SVRCONN channels for applications follow the same rules: microservices need certs in their trust stores when REQUIRED is set.

Diagnosing Handshake Versus Later Failures

• Handshake failure—errors in log during TLS, often before CHLAUTH mentioned.
• SSLPEERMAP failure—TLS completes; CHLAUTH block with SSLPEER in log.
• Bind name failure—TLS completes; channel name mismatch in LASTCHLERR.
• Connection refused—no TLS at all; listener or firewall.

Use openssl s_client or platform TLS test tools against the listener port to see certificate chains outside MQ when logs are unclear. Compare subject and issuer to your SSLKEYR contents. Load balancers that terminate TLS must either pass through mutual TLS or present certificates that match what partners expect—misconfigured SSL offloading is a frequent handshake culprit.

Platform and Policy Notes

Distributed and z/OS MQ share concepts but repository paths and commands differ—read the manual for your platform. Corporate policies may disable TLS 1.0 and 1.1; older partners need upgrades, not weaker MQ security. FIPS mode restricts available ciphers further. Document every exception with risk acceptance. After OS patches that update GSKit, regression-test critical channels in a maintenance window even when MQ version unchanged.

Renewal Runbook Highlights

1. Generate new cert and chain in PKI with lead time.
2. Import to SSLKEYR on all queue managers using the label.
3. Update partner trust stores before switching production cert.
4. Test handshake in lab with DISPLAY CHSTATUS.
5. Switch production cert; monitor for 24 hours.
6. Update SSLPEERMAP if DN changed—separate from handshake.

Explainer: Secret Handshake Before Talking

TLS handshake is two people agreeing on a secret code and showing ID cards before discussing business. If they speak different code languages (cipher mismatch) or one ID is expired, they never get to the business conversation (MQ bind).

Explain Like I'm Five: SSL Handshake Failures

You and your friend tried to set up a secret whistle code before playing, but you picked different whistles and could not understand each other—so the game never started.

Practice Exercises

Exercise 1

List SSLCIPH values for two sides and identify whether ANY_TLS12_OR_HIGHER helps or hurts interoperability.

Exercise 2

Write a cert renewal checklist including SSLKEYR, partner trust, and SSLPEERMAP review.

Exercise 3

Given a log snippet with cipher error, distinguish handshake from SSLPEERMAP failure in three sentences.

Frequently Asked Questions

Test Your Knowledge