Easytrieve DUPLICATE, FIRST-DUP, and LAST-DUP

Record-Relational Condition Syntax

Broadcom documents IF [NOT] DUPLICATE, FIRST-DUP, or LAST-DUP followed by file-name, PRIMARY, or SECONDARY. The subject identifies which file's neighboring records are compared. NOT reverses the result. Unlike field comparisons, no field name or EQ operator appears in the condition because the JOB's file key supplies the relationship.

1
2
3
4
5
6
7
8
9
10
11
IF DUPLICATE SORT1
   DUP-COUNT = DUP-COUNT + 1
END-IF
 
IF FIRST-DUP SORT1
   DISPLAY 'START OF DUPLICATE GROUP' CUSTOMER-ID
END-IF
 
IF LAST-DUP SORT1
   DISPLAY 'END OF DUPLICATE GROUP' CUSTOMER-ID
END-IF

What Counts as a Duplicate?

A duplicate is based on the processing key, not the complete record. Suppose input is sorted by CUSTOMER-ID. Two records with the same CUSTOMER-ID but different dates and amounts are duplicates for this condition. If the key is CUSTOMER-ID plus INVOICE-DATE, those same records may become unique. Always state the key in program comments and report headings so readers know what “duplicate” means.

DUPLICATE

DUPLICATE is true if either neighboring record shares the key. It therefore identifies every record in a repeated-key group, including first, middle, and last. Counting every record where DUPLICATE is true gives the number of records participating in duplicate groups, not the number of extra records and not the number of groups. Define the metric before coding.

1
2
3
4
IF DUPLICATE SORT1
   RECORDS-IN-DUP-GROUPS = RECORDS-IN-DUP-GROUPS + 1
   PRINT DUPLICATE-RPT
END-IF

FIRST-DUP

FIRST-DUP is true only for the first record of a group containing at least two equal keys. It is ideal for initializing group counters, printing a duplicate-group heading, or incrementing the number of duplicate groups. It is false for a unique record, even though that record is also the first and last occurrence of its key in ordinary language.

1
2
3
4
5
6
7
8
IF FIRST-DUP SORT1
   DUP-GROUP-COUNT = DUP-GROUP-COUNT + 1
   GROUP-RECORD-COUNT = 0
END-IF
 
IF DUPLICATE SORT1
   GROUP-RECORD-COUNT = GROUP-RECORD-COUNT + 1
END-IF

LAST-DUP

LAST-DUP marks the final record of a repeated-key group. By then, group accumulators contain all members if they were updated on each duplicate record. Use LAST-DUP to write one group trailer, output a consolidated record, or reset state for the next key. It is also true under DUPLICATE, so order branches carefully if only one action should run.

1
2
3
4
5
6
7
8
9
IF DUPLICATE SORT1
   GROUP-AMOUNT = GROUP-AMOUNT + TRANSACTION-AMOUNT
END-IF
 
IF LAST-DUP SORT1
   OUTPUT-KEY = CUSTOMER-ID
   OUTPUT-AMOUNT = GROUP-AMOUNT
   PUT GROUP-OUTPUT
END-IF

Finding the Start of Every Key Group

FIRST-DUP alone misses unique keys. NOT DUPLICATE identifies unique keys but misses the start of repeated groups. Broadcom documents their combination as the test for a new key: FIRST-DUP file OR NOT DUPLICATE file. This becomes true once per key, whether that key appears once or many times.

1
2
3
4
IF FIRST-DUP SORT1 OR NOT DUPLICATE SORT1
   UNIQUE-KEY-COUNT = UNIQUE-KEY-COUNT + 1
   CURRENT-KEY = CUSTOMER-ID
END-IF

UNIQUE-KEY-COUNT now counts distinct processing keys, not input records.

Finding the End of Every Key Group

The matching end-of-key formula is LAST-DUP file OR NOT DUPLICATE file. LAST-DUP handles repeated groups; NOT DUPLICATE handles a one-record group. Use this condition to emit one consolidated output per key or print a group total after the last contributing record.

1
2
3
4
5
6
GROUP-TOTAL = GROUP-TOTAL + TRANSACTION-AMOUNT
 
IF LAST-DUP SORT1 OR NOT DUPLICATE SORT1
   PERFORM WRITE-GROUP-TOTAL
   GROUP-TOTAL = 0
END-IF

Input Must Be Sequenced by the Intended Key

Record-relational conditions compare adjacent records under Easytrieve's current processing key. If duplicate customer IDs are separated by unrelated records in unsorted input, they are not neighbors and cannot form one duplicate group. Sort first or use a synchronized keyed JOB. If the SORT key differs from the business definition of duplicate, the output will be internally consistent but business-wrong.

1
2
3
4
5
6
SORT CUSTOMER-FILE TO SORT1 USING (CUSTOMER-ID) NAME CUSTOMER-SORT
 
JOB INPUT SORT1
  IF DUPLICATE SORT1
     PRINT DUP-RPT
  END-IF

Include secondary key components when deterministic order inside each group matters, such as transaction date and sequence. The duplicate relationship can remain based on a shorter synchronized key depending on the JOB definition; verify the actual syntax and intent.

Synchronized Input with PRIMARY and SECONDARY

Multi-file JOB INPUT can refer to a named file or the PRIMARY and SECONDARY roles. IF DUPLICATE PRIMARY tests repeated keys within the primary input. IF DUPLICATE SORT2 tests the named secondary. This is separate from MATCHED, which compares key presence across files. A file can contain duplicate keys and also match records in another file.

1
2
3
4
5
6
7
8
9
10
JOB INPUT (SORT1 KEY (STATE-CODE), +
           SORT2 KEY (STATE-CODE))
 
  IF DUPLICATE PRIMARY
     PRIMARY-DUP-COUNT = PRIMARY-DUP-COUNT + 1
  END-IF
 
  IF DUPLICATE SORT2
     SECONDARY-DUP-COUNT = SECONDARY-DUP-COUNT + 1
  END-IF

DUPLICATE vs MATCHED

DUPLICATE vs Duplicate-Key I/O Status

A VSAM PUT into a unique KSDS can fail because the key already exists. That is an I/O operation result available through FILE-STATUS and product-specific conditions. IF DUPLICATE input-file does not inspect the target catalog or predict whether PUT succeeds. It only compares current input with neighboring input records. Validate input groups and still check output FILE-STATUS.

Keeping One Record per Key

Deduplication requires a rule for which record wins. “Keep first” writes when FIRST-DUP OR NOT DUPLICATE. “Keep last” writes when LAST-DUP OR NOT DUPLICATE. But first and last depend on sort sequence. Sort newest date first to keep newest with a first-record rule, or oldest first to keep oldest. Document tie-breakers so repeated runs are deterministic.

1
2
3
4
5
6
7
IF FIRST-DUP SORT1 OR NOT DUPLICATE SORT1
   PUT UNIQUE-OUTPUT FROM SORT1
END-IF
 
IF DUPLICATE SORT1 AND NOT FIRST-DUP SORT1
   PRINT REJECTED-DUP-RPT
END-IF

This keeps the first sorted record and reports subsequent members. Confirm whether the last member should also be rejected; it is DUPLICATE and not FIRST-DUP, so it is included.

Counting Duplicate Records Correctly

Different stakeholders ask for different counts. Records participating in duplicates: increment for every DUPLICATE. Duplicate groups: increment for FIRST-DUP. Extra records beyond one per key: increment for DUPLICATE AND NOT FIRST-DUP. Distinct keys: increment for FIRST-DUP OR NOT DUPLICATE. Give each counter a precise name.

Duplicate Group Totals

Accumulate every record in the group, including the first and last. If the requirement includes unique keys too, do not place accumulation only inside IF DUPLICATE. Instead add every record and use the end-of-key formula to emit. Reset immediately after output so the next key starts at zero.

EOF and Boundary Look-Ahead

Easytrieve's runtime supplies previous and next relationships for these conditions, including groups at file boundaries. Do not implement a second manual EOF look-ahead on top of FIRST-DUP and LAST-DUP unless another requirement needs it. EOF answers whether a file is exhausted; LAST-DUP answers whether the current repeated key group ends here. A unique last record is NOT DUPLICATE, not LAST-DUP.

Reports and Audit Evidence

A useful duplicate report prints the key, group position, source sequence, differing fields, and selected winner. Merely listing “duplicate found” is insufficient for data remediation. Derive a status field such as FIRST DUPLICATE, MIDDLE DUPLICATE, LAST DUPLICATE, or UNIQUE before PRINT. Keep rejected records in a controlled output dataset when business decisions may be reviewed later.

Version Notes

DUPLICATE, FIRST-DUP, and LAST-DUP are documented in Broadcom Easytrieve Report Generator 11.6 and earlier CA-Easytrieve Plus synchronized-processing material. Older local macros may use names such as FIRSTDUP in comments or wrappers, but the documented language keywords include hyphens: FIRST-DUP and LAST-DUP. Compile migrated source and review the release's reserved-word list.

Common DUPLICATE Mistakes

1. Running duplicate logic on input not sorted by the intended key.
2. Assuming DUPLICATE compares the complete record.
3. Counting DUPLICATE records when the requirement asks for duplicate groups.
4. Using FIRST-DUP alone and missing unique key groups.
5. Using LAST-DUP to identify a unique final record.
6. Confusing record duplicates with a VSAM duplicate-key write status.
7. Keeping “first” without defining the sort order that makes it the desired winner.

Explain It Like I'm Five

Imagine cards sorted by last name. If the card before or after yours has the same last name, your card is DUPLICATE by that key. The first Smith card is FIRST-DUP, and the last Smith card is LAST-DUP. One Jones card by itself is NOT DUPLICATE. If the cards are not sorted, two Smith cards can be far apart and the neighbor check cannot know they belong to one group.

Exercises

1. Sort transactions by customer and print every record in duplicate groups.
2. Count duplicate groups, duplicate-participating records, and extra records separately.
3. Write one output record at the end of every key group, including unique keys.
4. Keep the newest record per key by choosing a sort order and a boundary condition.
5. Explain the difference between DUPLICATE, MATCHED, and EOF in synchronized input.
6. Create an audit report labeling first, middle, last, and unique records.