DFSORT Hipersorting

What Is Hiperspace?

Hiperspace is a z/OS storage facility that provides a separate address space for data. Historically it was associated with expanded storage (a type of memory on older mainframe configurations). On current 64-bit z/OS, the hiperspace services are often implemented using central storage rather than a separate expanded storage tier. From a DFSORT perspective, hiperspace is another place the product can put sort work: instead of (or in addition to) using memory objects in the step’s central storage, it can use hiperspace. The exact behavior and tuning parameters depend on your DFSORT (or ICETOOL) version and z/OS release; the concepts here apply to the general idea of “hipersorting” and the common OPTION parameters you will see in documentation and in shops.

Hiperspace vs Memory Objects

DFSORT has two main ways to hold sort work in “fast” storage: memory objects and hiperspace. Memory objects are blocks of central storage (main memory) in your step’s address space; their size is controlled by OPTION SIZE or MOSIZE. Hiperspace is a different storage type the product can use. The product may use different internal algorithms for sorting in memory objects versus in hiperspace, which can lead to different CPU times and sometimes different elapsed times. When you look at SMF (e.g. type 30) records, hiperspace CPU may be reported in a separate field (e.g. SMF30HPT); for a fair comparison between a run that uses hiperspace and one that does not, you should include both central storage CPU and hiperspace CPU. In practice, many sites let DFSORT choose automatically; others explicitly set HIPRMAX=0 and MOSIZE=MAX to force memory-object-only behavior for consistency or tuning.

Key OPTION Parameters

The main parameters that affect hiperspace and memory object usage are HIPRMAX, MOSIZE, and in some products DSPSIZE. Their exact names and syntax can vary by product (DFSORT, ICETOOL, Beyond Sort); the following describes the typical meaning.

HIPRMAX

HIPRMAX controls how much hiperspace DFSORT may use. HIPRMAX=OPTIMAL tells the product to choose an optimal amount—often maximizing hiperspace when it is beneficial. HIPRMAX=0 disables hiperspace entirely; then all sort work that would have gone to hiperspace must use memory objects (and possibly sortwork). Some products allow a numeric value to cap hiperspace at a specific size. When you set HIPRMAX=0, it is common to also set MOSIZE=MAX (or a large MOSIZE) so that the product has a clear limit for central storage and can complete the sort using memory objects and sortwork.

MOSIZE

MOSIZE (memory object size) limits the size of the memory objects DFSORT uses in central storage. It can be specified as a percentage of available storage (e.g. MOSIZE=50) or as a fixed value in MB. When you want to favor hiperspace, you might set MOSIZE=0 and HIPRMAX=OPTIMAL so the product uses hiperspace for sort work. When you want to favor memory objects only, you set HIPRMAX=0 and MOSIZE=MAX (or a large value). So HIPRMAX and MOSIZE work together: one pushes toward hiperspace, the other toward central-storage memory objects.

DSPSIZE

DSPSIZE controls dataspace allocation in products that use dataspaces for sort work. Dataspaces are another z/OS storage construct. The exact effect of DSPSIZE is product-dependent; see your DFSORT or ICETOOL documentation. In many discussions of “hipersorting,” the focus is on HIPRMAX and MOSIZE; DSPSIZE is relevant when the product uses dataspaces as well.

Example: Disabling Hiperspace

To force DFSORT to use only memory objects (no hiperspace), you can specify OPTION HIPRMAX=0 and MOSIZE=MAX. The following is illustrative; exact syntax may vary by product and release.

1
2
  OPTION HIPRMAX=0,MOSIZE=MAX
  SORT FIELDS=(1,10,CH,A)

This tells the product: do not use hiperspace (HIPRMAX=0), and use as much central storage as allowed for memory objects (MOSIZE=MAX). All sort work then goes to memory objects and, when they are full, to sortwork datasets. This can be useful when you want consistent behavior, when you are debugging, or when your site prefers to manage only central storage and sortwork.

Example: Maximizing Hiperspace

To let the product maximize hiperspace usage, you might specify OPTION HIPRMAX=OPTIMAL and MOSIZE=0 (so that the product does not reserve a large amount for memory objects and can favor hiperspace). Again, syntax is product-dependent.

1
2
  OPTION HIPRMAX=OPTIMAL,MOSIZE=0
  SORT FIELDS=(1,10,CH,A)

With this, DFSORT chooses how much hiperspace to use (OPTIMAL) and does not allocate a large memory object size (MOSIZE=0), so it may use hiperspace for a large portion of the sort work. Whether this is faster than memory-object-only depends on your system, data size, and the product’s algorithms; IBM generally recommends letting the product choose automatically unless you have a specific reason to override.

Performance and CPU Time

Because DFSORT can use different code paths for memory objects versus hiperspace, CPU time can differ between a run that uses hiperspace and one that uses only memory objects. If you are comparing performance, be sure to include hiperspace CPU (e.g. SMF30HPT in SMF type 30) in the total CPU for the hiperspace run; otherwise the comparison is misleading. Elapsed time can also differ due to different I/O and merge behavior. Many sites run with default or OPTIMAL settings and only override when they need reproducible behavior (e.g. HIPRMAX=0,MOSIZE=MAX) or when tuning for a specific workload.

Interaction with Sortwork and FILSZ

Hiperspace and memory objects are both “fast” storage for sort work. When either (or both) is full, DFSORT still writes runs to sortwork (SORTWK) and merges from sortwork. So hipersorting does not replace the need for sortwork when the data is large; it only changes where the first tier of sort work lives (hiperspace and/or memory objects). FILSZ is used to plan sortwork allocation; it does not directly control hiperspace or MOSIZE. For a large sort, you still need sufficient sortwork and a realistic FILSZ; hiperspace and MOSIZE affect how much work stays in fast storage before spilling to sortwork.

z/OS 64-Bit and “Expanded Storage”

On 64-bit z/OS, expanded storage as a separate hardware tier was removed. Hiperspace services were re-implemented to use central storage. So when DFSORT “uses hiperspace” on current z/OS, the backing storage is typically central storage, not a separate expanded-storage pool. The logical distinction (memory objects vs hiperspace) still exists in the product’s options and algorithms, but the underlying resource is often the same type of memory. This is why documentation may say “hiperspace uses central storage” on 64-bit z/OS.

Recommendation: Let DFSORT Choose

IBM recommends allowing DFSORT to automatically decide which storage method to use. The product evaluates available resources, file size, and the functions used (e.g. INREC, OUTREC, SUM) and can select an appropriate mix of memory objects and hiperspace. Explicit overrides (HIPRMAX=0 or HIPRMAX=OPTIMAL with MOSIZE=0) are useful when you need consistent behavior, when you are doing performance comparison, or when your site has a standard (e.g. “no hiperspace” for easier capacity planning). Otherwise, omitting these options or using OPTIMAL is usually the best approach.

Explain It Like I'm Five

Imagine you have two kinds of desks to sort papers: a big desk in your room (memory) and a special desk in another room (hiperspace). The big desk is “memory objects”; the special desk is “hiperspace.” The grown-ups (DFSORT) can use one or both. If you say “don’t use the other room” (HIPRMAX=0), they only use your big desk and, when it’s full, they put piles in boxes (sortwork). If you say “use whichever is best” (OPTIMAL), they decide. Sometimes using the special desk takes a different amount of “work time” (CPU) than using only your desk, so when we compare we have to count both. On new systems, the “other room” is actually part of the same building (central storage), but we still call it hiperspace because the rules for using it are a bit different.

Exercises

1. You want to force your sort to use only central storage (no hiperspace). What OPTION parameters would you set and what do they do?
2. When comparing CPU time of a job that uses hiperspace versus one that does not, what SMF field might you need to include and why?
3. Why might a site choose HIPRMAX=0 and MOSIZE=MAX even if OPTIMAL could use hiperspace?