Change Detection

What Is Change Detection?

In many reports you have records grouped by a key: department, account, region, or date. Readers find it easier to scan when each group is visually separated—for example, a blank line or a short header line when the department changes. Change detection is the idea of noticing "this record has a different department (or key) than the previous one" and then doing something special for that record: output a blank line before it, output a header line, or start a new sequence number. DFSORT does not compare arbitrary pairs of records; it processes records in order. So the "previous" record is always the one that was just processed. For that to correspond to "same group" or "new group," records must already be in order by the control field. That is why you sort by the control field first.

Sort First by the Control Field

Suppose your input has records with department code in positions 1–5 and employee name in 6–25. If you want a blank line every time the department changes, you must first sort by department so that all DEPT1 records are together, then all DEPT2, and so on. Example:

1
  SORT FIELDS=(1,5,CH,A)

After this sort, the first record with a new value in bytes 1–5 is exactly the first record of a new department. DFSORT can then use WHEN=GROUP BEGIN= or SECTIONS to detect that "change" and apply different formatting or insert section headers/trailers.

WHEN=GROUP and BEGIN=

WHEN=GROUP is an IFTHEN clause that works on groups of records. You define what a group is by specifying at least one of:

• BEGIN=(logexp) — A logical expression that is true when the current record starts a new group. For change detection, this is often "control field not equal to previous record." DFSORT supports comparing with the previous record in the sort order so that the first record of each new key value satisfies BEGIN.
• END=(logexp) — True when the record ends a group.
• RECORDS=n — Groups of exactly n records (e.g. RECORDS=3 for groups of three).

When BEGIN= is used for change detection, the expression typically refers to the control field. When it evaluates to true, the current record is the first of a new group. You can then use BUILD= or OVERLAY= in that IFTHEN to format this record differently (e.g. add a constant or a blank line before the data), or use PUSH=(position:SEQ=length) to add a sequence number that starts at 1 for each new group. ID=length adds a group identifier that increments for each new group. Exact syntax (e.g. how to express "different from previous") is product-specific; see your IBM DFSORT documentation for WHEN=GROUP and BEGIN=.

Example: Per-Group Sequence Number

You want a sequence number that resets to 1 for each new department (bytes 1–5). Sort by department, then use WHEN=GROUP with BEGIN= (condition that is true for the first record of each department) and PUSH with SEQ=:

1
2
3
  SORT FIELDS=(1,5,CH,A)
  OUTREC IFTHEN=(WHEN=GROUP,BEGIN=(1,5,CH,NE,PREV),
    PUSH=(26:SEQ=5))

Here, BEGIN=(1,5,CH,NE,PREV) means "bytes 1–5 (character) are not equal to the previous record"—so the first record of each new department. PREV is a DFSORT keyword for the previous record's value. PUSH=(26:SEQ=5) overlays a 5-byte zoned decimal sequence number at position 26, starting at 1 for each new group. The exact syntax for PREV and BEGIN may vary by product level; check your manual.

OUTFIL SECTIONS for Section Headers and Trailers

Another way to react to "change" is OUTFIL SECTIONS=(start,length,format). SECTIONS groups records by the specified key. For each group, you can have:

• A section header — Printed before the first record of the group (e.g. "*** DEPARTMENT xxx ***").
• A section trailer — Printed after the last record of the group (e.g. TRAILER3 with a subtotal or count for that group).

So when the key changes, DFSORT automatically starts a new section and can output a header and/or trailer for the previous section and then for the new one. This is ideal for control-break reports where you want a title and a subtotal line for each department (or other key). Example:

1
2
3
4
  SORT FIELDS=(1,5,CH,A)
  OUTFIL FNAMES=REPORT,SECTIONS=(1,5,CH),
    HEADER1=(1:'Department Report'),
    TRAILER3=(1:'Subtotal for group:',...)

SECTIONS=(1,5,CH) groups by the first 5 bytes. TRAILER3 is the section trailer (after each group). The exact syntax for TRAILER3 and subtotals (e.g. summing a numeric field in the section) is product-dependent; see your OUTFIL and report-writing documentation.

Adding a Blank Line Between Groups

To insert a blank line when the control field changes, you need to output an extra record before the first record of each new group (except possibly the very first group). One approach is to use OUTFIL with SECTIONS and a section header that is a blank line. Another is to use OUTREC IFTHEN WHEN=GROUP BEGIN= and output two lines for the "first of group" record: one blank line and one data line. Implementation details depend on whether your product allows multiple output records per input record in that context; some shops use a separate pass or ICETOOL. A simple conceptual approach: when BEGIN is true, BUILD (or OVERLAY) could include a new-line or separator; the exact method (e.g. writing a literal blank record) depends on DFSORT version and options. Check your manual for "insert record when group changes" or "blank line between groups."

Comparison: WHEN=GROUP vs SECTIONS

• WHEN=GROUP — Flexible group definition (BEGIN/END/RECORDS). Good for per-group sequence numbers (SEQ=), group IDs (ID=), and conditional formatting of the first (or last) record of a group. You build the logic inside IFTHEN.
• SECTIONS — Groups by a key and attaches section-level HEADER and TRAILER (e.g. TRAILER3). Good for report-style output where each group has a header line and a trailer line (e.g. subtotal). Less flexible for arbitrary "compare to previous" logic but simpler for standard control-break reports.

Explain It Like I'm Five

Imagine a list of kids by class: first all the Grade 1 kids, then Grade 2, then Grade 3. You want to put a blank line and a little title whenever the grade changes. Change detection is when the program looks at each line and says: "Is this grade different from the line above?" When the answer is yes, we just started a new grade—so we add a blank line and write "Grade 2" (or whatever). To make that work, the list must already be sorted by grade. DFSORT does the same thing: sort by the key (department, account, etc.), then when the key changes, add a blank line or a header or a per-group number so the report is easy to read.

Exercises

1. Sort a file by bytes 10–13 (CH). What SORT FIELDS would you use? Why must the file be sorted before you can detect "change" in that field?
2. Look up your DFSORT manual: does your level support WHEN=GROUP with BEGIN= and PREV (previous record)? What is the exact syntax for "current record's bytes 1–5 not equal to previous record's bytes 1–5"?
3. If you want a subtotal line after each department, would you use WHEN=GROUP or SECTIONS? Why?
4. What is the purpose of TRAILER3 in OUTFIL when SECTIONS is used?