CICS MRO Resource Management

CICS Multi-Region Operation (MRO) Resource Management is a critical aspect of managing distributed CICS environments. MRO enables CICS regions running in the same z/OS image or sysplex to communicate and share resources efficiently. This guide covers the fundamentals of MRO resource management, including region types, interregion communication, resource sharing strategies, connection management, and performance optimization.

What is MRO (Multi-Region Operation)?

Multi-Region Operation (MRO) is a CICS facility that enables CICS regions in the same z/OS image or sysplex to communicate with each other without requiring SNA (System Network Architecture) networking facilities. MRO uses interregion communication (IRC) to provide fast, efficient communication between regions.

Key characteristics of MRO:

• No SNA Required: MRO does not require ACF/Communications Server or SNA networking facilities
• Same Sysplex: Regions must be in the same z/OS image or sysplex
• Fast Communication: Uses efficient interregion communication mechanisms
• Resource Sharing: Enables sharing of files, programs, and other resources across regions
• Workload Distribution: Allows distributing work across multiple regions for scalability

Explain It Like I'm 5 Years Old

Imagine you have a big office building with many different departments. Each department (that's like a CICS region) has its own room, but they all work in the same building (the sysplex). MRO is like having a special phone system that lets all these departments talk to each other really quickly without needing to go outside the building.

Some departments have the computers where people type (TOR - Terminal-Owning Region), some have the programs that do the work (AOR - Application-Owning Region), and some have the filing cabinets with all the data (FOR - File-Owning Region). MRO lets them all work together smoothly, like when someone in the typing department needs information from a filing cabinet in another department - they can just call and get it quickly!

MRO Region Types and Roles

In an MRO environment, CICS regions can be assigned different roles based on the resources they own and manage. Understanding these roles is essential for effective resource management.

Combining Roles

A single CICS region can perform multiple roles. For example, a region might be both a TOR and an AOR, or an AOR and a FOR. The roles are not mutually exclusive - they describe the primary function of a region but don't prevent it from performing other functions.

1
2
3
4
5
6
7
8
9
Example Configuration:
- Region CICS1: TOR (handles terminals) + AOR (runs applications)
- Region CICS2: AOR (runs applications) + FOR (owns files)
- Region CICS3: FOR (owns critical databases)
 
This allows:
- Terminals connect to CICS1
- Applications run in CICS1 and CICS2
- Data access goes to CICS3 when needed via function shipping

Interregion Communication (IRC)

Interregion Communication (IRC) is the internal CICS support that enables MRO. IRC provides the low-level communication mechanism that allows regions to exchange data and coordinate work.

IRC Implementation Methods

IRC can be implemented using different mechanisms depending on your z/OS configuration:

• Type 3 SVC (Supervisory Call): Traditional method using system calls for communication
• MVS Cross-Memory Services: Uses z/OS cross-memory facilities for efficient data transfer
• XCF (Cross-System Coupling Facility): For sysplex environments, uses coupling facility for communication

The choice of implementation method depends on your z/OS version, sysplex configuration, and performance requirements. XCF/MRO is typically the most efficient for sysplex environments.

MRO Facilities and Resource Sharing

MRO provides several facilities that enable different types of resource sharing and communication between regions:

Function Shipping Example

Function shipping allows a program in one region to access a resource owned by another region. Here's how it works:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
*> Program running in AOR (Application-Owning Region)
IDENTIFICATION DIVISION.
PROGRAM-ID. CUSTINQ.
 
DATA DIVISION.
WORKING-STORAGE SECTION.
01 CUSTOMER-RECORD.
   05 CUST-ID        PIC X(8).
   05 CUST-NAME      PIC X(30).
   05 CUST-BALANCE   PIC S9(9)V99 COMP-3.
 
PROCEDURE DIVISION.
    *> This file is owned by FOR (File-Owning Region)
    *> CICS automatically ships the request to the FOR
    EXEC CICS READ
         DATASET('CUSTFILE')
         INTO(CUSTOMER-RECORD)
         RIDFLD(CUST-ID)
    END-EXEC.
 
    *> Process the customer data
    DISPLAY 'Customer: ' CUST-NAME
    DISPLAY 'Balance: ' CUST-BALANCE
    
    EXEC CICS RETURN END-EXEC.

When this program runs in the AOR and tries to read CUSTFILE, CICS detects that the file is owned by the FOR. It automatically ships the READ request to the FOR, where a mirror transaction (CSMI) processes it, and the result is shipped back to the AOR. This is transparent to the application program.

Configuring MRO Connections

MRO connections are defined using CONNECTION resource definitions. Connections specify how regions communicate and what resources they can access.

Connection Resource Definition

Connections can be defined using CEDA (CICS resource definition) or RDO commands:

1
2
3
4
5
6
7
8
9
10
CEDA DEFINE CONNECTION(MROCONN) GROUP(MROGRP)
DESCRIPTION('MRO Connection to Remote Region')
NETNAME(REMOTEREG)
PROTOCOL(MRO)
ACCESSMETHOD(IRC)
SESSIONLIMIT(100)
MAXSESSIONS(200)
TIMEOUT(300)
SECURITY(USERID)
STATUS(ENABLED)

Key connection attributes:

• NETNAME: The name by which the remote region is known in the network
• PROTOCOL: Set to MRO for Multi-Region Operation
• ACCESSMETHOD: Specifies IRC (Interregion Communication)
• SESSIONLIMIT: Maximum number of concurrent sessions
• MAXSESSIONS: Maximum sessions that can be established
• TIMEOUT: Timeout value for connection operations
• SECURITY: Security options (USERID, PASSWORD, etc.)

Local and Remote Connections

In MRO, you define both local and remote connections:

• Local Connection: Defines how this region appears to other regions (its NETNAME)
• Remote Connection: Defines how to connect to other regions in the CICSplex

1
2
3
4
5
6
7
8
9
10
11
12
13
14
*> Define local connection (how this region appears to others)
CEDA DEFINE CONNECTION(LOCAL) GROUP(MROGRP)
NETNAME(MYREGION)
PROTOCOL(MRO)
ACCESSMETHOD(IRC)
STATUS(ENABLED)
 
*> Define remote connection (how to reach another region)
CEDA DEFINE CONNECTION(REMOTE1) GROUP(MROGRP)
NETNAME(REGION1)
PROTOCOL(MRO)
ACCESSMETHOD(IRC)
SESSIONLIMIT(50)
STATUS(ENABLED)

Transaction Tracking in MRO

When transactions execute across multiple regions in an MRO environment, CICS provides transaction tracking to link related work together. This is essential for monitoring, auditing, and problem determination.

Unit of Work ID (UOWID)

Each transaction is assigned a Unit of Work ID (UOWID), which is a time value derived from the initial task attach time. The UOWID flows with the transaction as it moves between regions, allowing you to track all pieces of work related to the same logical transaction.

UOWID characteristics:

• Unique identifier for a transaction across all regions
• Based on the timestamp when the transaction started
• Combined with NETNAME to form a unique identifier
• Written to SMF records for tracking and analysis

Transaction Records Across Regions

When a transaction executes across multiple regions (e.g., via transaction routing or function shipping), each region writes its own transaction record. These records can be linked together using UOWID and NETNAME:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Example: Transaction AUPD routed from TOR to AOR, then function-shipped to FOR
 
TOR Record:
- UOWID: 3577:58:58.68
- NETNAME: STOR
- Transaction: AUPD
- Program: DFHCRP (relay program)
- Duration: 00:00:00.349
 
AOR Record:
- UOWID: 3577:58:58.68 (same UOWID)
- NETNAME: STOR (originating region)
- Transaction: AUPD
- Program: DFH$AALL (application program)
- Duration: 00:00:00.302
 
FOR Record:
- UOWID: 3577:58:58.68 (same UOWID)
- NETNAME: STOR (originating region)
- Transaction: CSMI (mirror transaction)
- Program: DFHMIR
- Duration: 00:00:00.036
- File operations: 1 GET

Resource Management Strategies

Effective resource management in MRO requires careful planning of where resources are located and how they are accessed. Here are key strategies:

1. Locate Programs Near Their Data

To minimize function shipping overhead, place application programs in the same region as the data they primarily access. This reduces cross-region communication and improves performance.

1
2
3
4
5
6
7
8
9
Good Design:
- Customer Inquiry Program → Place in FOR with customer files
- Order Processing Program → Place in FOR with order files
- Report Generation → Can be in separate AOR, uses function shipping
 
Poor Design:
- All programs in AOR, all files in FOR
- Every file access requires function shipping
- High overhead and slower performance

2. Balance Workload Across Regions

Distribute transaction processing across multiple AORs to balance workload and improve scalability:

• Route different transaction types to different AORs
• Use dynamic routing based on workload
• Consider transaction affinity (keep related work together)
• Monitor and adjust routing based on performance metrics

3. Optimize Connection Pooling

Configure connection pools appropriately to balance resource usage and performance:

• SESSIONLIMIT: Set based on expected concurrent transactions
• MAXSESSIONS: Allow for peak load scenarios
• Connection Timeout: Balance between resource cleanup and connection reuse
• Monitor Usage: Track actual session usage and adjust limits

4. Manage Resource Ownership

Clearly define resource ownership to avoid conflicts and optimize access:

• Each file should have a single owning region (FOR)
• Define remote file definitions in non-owning regions
• Use appropriate access methods (function shipping vs. local access)
• Consider data locality when designing applications

Performance Optimization

Optimizing MRO performance requires understanding the overhead of interregion communication and minimizing it where possible.

Minimize Function Shipping

Function shipping has overhead - each request must be serialized, transmitted, processed, and the result returned. Minimize function shipping by:

• Locating programs in the same region as their primary data
• Using DPL instead of function shipping when appropriate
• Batching operations when possible
• Avoiding function shipping in tight loops

Optimize Session Management

Efficient session management reduces overhead:

• Reuse sessions when possible
• Configure appropriate session limits
• Monitor session queue depths
• Use timeout delete to clean up unused sessions

Monitor Interregion Traffic

Use CICS monitoring facilities to track interregion communication:

• Monitor CMF records for interregion activity
• Track function shipping counts and response times
• Identify bottlenecks in interregion communication
• Use CICS Performance Analyzer for detailed analysis

Security Considerations

MRO resource management must consider security implications:

• Connection Security: Configure appropriate security options (USERID, PASSWORD, etc.)
• Resource Access: Ensure proper security definitions for remote resources
• User Context: User IDs and security context flow with transactions across regions
• Audit Trail: Transaction tracking provides audit capability across regions

Exercises

Exercise 1: Understanding Region Roles

Scenario: You have a CICSplex with three regions:

• Region CICS1: Handles all 3270 terminals
• Region CICS2: Runs customer inquiry and update programs
• Region CICS3: Owns all VSAM customer files

Questions:

• What role does each region play (TOR, AOR, FOR)?
• When a user enters transaction CUST on a terminal in CICS1, what happens?
• How does the customer inquiry program in CICS2 access customer data in CICS3?
• What would be the UOWID flow for this transaction?

Exercise 2: Connection Configuration

Task: Design connection definitions for an MRO environment with:

• One TOR region (NETNAME: CICSTOR)
• Two AOR regions (NETNAME: CICSAOR1, CICSAOR2)
• One FOR region (NETNAME: CICSFOR)

Requirements:

• TOR must connect to both AORs
• Both AORs must connect to FOR
• Session limits: 50 for TOR-AOR, 100 for AOR-FOR
• Timeout: 300 seconds

Write the CEDA commands to define these connections.

Exercise 3: Performance Optimization

Scenario: You notice high function shipping activity between your AOR and FOR regions. Analysis shows that 80% of function-shipped requests are for the CUSTFILE.

Questions:

• What are the potential performance impacts of this high function shipping?
• What strategies could you use to reduce function shipping for CUSTFILE?
• What are the trade-offs of each strategy?
• How would you measure the improvement?

Best Practices

• Plan Resource Location: Design your region structure and resource ownership before implementation
• Minimize Cross-Region Communication: Locate programs near their data to reduce function shipping
• Monitor Performance: Use CMF and performance tools to track interregion activity
• Balance Workload: Distribute transactions across multiple AORs for scalability
• Configure Connections Properly: Set appropriate session limits and timeouts
• Use Transaction Tracking: Leverage UOWID for monitoring and problem determination
• Test Thoroughly: Test MRO configurations in a development environment first
• Document Configuration: Maintain clear documentation of region roles and connections

Common Issues and Troubleshooting

Connection Failures

Symptoms: Transactions fail with connection errors, regions cannot communicate

Solutions:

• Verify connection definitions are installed in both regions
• Check NETNAME matches between local and remote definitions
• Ensure IRC is properly configured in both regions
• Check for security violations in connection establishment
• Review system logs for IRC-related errors

Performance Degradation

Symptoms: Slow transaction response times, high CPU usage for interregion communication

Solutions:

• Analyze CMF records to identify high function shipping
• Consider relocating programs to reduce cross-region access
• Review session limits and connection pool sizes
• Check for network or IRC bottlenecks
• Optimize transaction routing to balance workload

Resource Conflicts

Symptoms: File access errors, resource contention, inconsistent data

Solutions:

• Ensure clear resource ownership (one FOR per resource)
• Verify remote file definitions are correct
• Check for proper enqueue/dequeue handling
• Review syncpoint coordination across regions