Easytrieve Sort Duplicate Handling

SORT Retains Equal Keys

1
2
3
4
SORT TRANS TO SORTWRK USING (ACCT-NO, POST-DATE)
 
* Input: three rows ACCT 100 same date different amounts
* Output: three adjacent rows — not one merged row

The sort utility orders by ACCT-NO major and POST-DATE minor. Rows with identical keys on both fields appear consecutively but all survive. To drop extras, filter in JOB or BEFORE—not by omitting USING.

Why Sort Before Duplicate Logic

Sort keys should include the business identity you use for duplicate tests—often ACCT-NO alone or ACCT-NO plus POST-DATE if duplicates are defined at that grain.

DUPLICATE, FIRST-DUP, LAST-DUP

1
2
3
4
JOB INPUT SORTWRK
  IF FIRST-DUP SORTWRK OR NOT DUPLICATE SORTWRK
     PRINT DETAIL-RPT
  END-IF

FIRST-DUP OR NOT DUPLICATE is the classic new-key test: print the first row of each duplicate group and any unique row. FIRST-DUP alone omits unique singles depending on logic—study Broadcom truth tables for your release. LAST-DUP keeps the final row in a group—useful after sorting date descending within account so last equals newest.

Last-Wins After Sort Pattern

1
2
3
4
5
6
SORT MASTER TO SORTWRK USING (CUST-ID, EFF-DATE D)
 
JOB INPUT SORTWRK
  IF LAST-DUP SORTWRK OR NOT DUPLICATE SORTWRK
     PUT OUTFILE
  END-IF

CUST-ID ascending groups customers. EFF-DATE D puts newest effective date first within customer—but LAST-DUP after that sort picks the first physical row in the group which is the newest date. Verify with sample data; some shops prefer FIRST-DUP after ascending date to keep oldest. Sort direction and FIRST/LAST choice must agree on business meaning.

Summing Duplicate Groups

1
2
3
4
5
6
7
8
9
JOB INPUT SORTWRK
  IF FIRST-DUP SORTWRK OR NOT DUPLICATE SORTWRK
     WS-TOTAL = AMT
  ELSE
     WS-TOTAL = WS-TOTAL + AMT
  END-IF
  IF LAST-DUP SORTWRK OR NOT DUPLICATE SORTWRK
     PRINT SUM-RPT
  END-IF

Accumulate amounts between FIRST-DUP and LAST-DUP inclusive, then PRINT one summary line per key group. Control reports with SUM may alternatively total detail if you PRINT every row—choose one pattern to avoid double counting.

BEFORE SELECT and Duplicates

BEFORE can drop duplicate input before sort by tracking keys in working storage—heavy for large files. SELECT executed twice on the same record in one prescreen pass still includes that record once in sort output per Broadcom. SELECT then STOP on the same pass excludes the record. Use BEFORE to filter bad rows, not as a substitute for sort-key design.

Not the Same as VSAM DUPLICATE Status

Indexed WRITE with existing key may set DUPLICATE file status—that is keyed I/O. Record-relational DUPLICATE compares sequential adjacent records in JOB after SORT. KSDS generic duplicate reads are yet another topic. Keep vocabulary separate in design documents.

Reporting Duplicate Clusters

1
2
3
4
JOB INPUT SORTWRK
  IF DUPLICATE SORTWRK
     PRINT DUP-AUDIT
  END-IF

Audit reports flag rows participating in duplicate groups—every row after the first in a group satisfies DUPLICATE when sorted. Pair with FIRST-DUP counts on SYSPRINT via DISPLAY for operational metrics.

Common Duplicate Mistakes

• Expecting SORT to collapse equal keys automatically.
• Testing DUPLICATE before sorting—groups incomplete.
• Wrong key grain—ACCT-DATE duplicate definition but sort only on ACCT.
• FIRST-DUP and descending date chosen inconsistently.
• Confusing record DUPLICATE with VSAM WRITE status.

Testing Duplicate Handling

1. Build file with twin keys and unique keys; count output rows after SORT.
2. Verify FIRST-DUP prints once per twin group.
3. Verify LAST-DUP after date D picks expected survivor.
4. Sum pattern matches manual calculator total per key.
5. Audit report lists only intended duplicate participants.

Explain It Like I'm Five

Sorting duplicate keys is lining up kids with the same birthday in a row. SORT does not send extras home—it only makes them stand together. FIRST-DUP points to the first kid in the birthday group; LAST-DUP points to the last. You decide who gets a prize—not the sort teacher.

Exercises

1. Sort three duplicate-key rows; confirm three out.
2. Write JOB that PRINTs only FIRST-DUP per ACCT-NO.
3. Design last-wins PUT with EFF-DATE D sort.
4. Summarize amounts per key with FIRST/LAST pattern.
5. Explain SORT versus JOB roles in deduplication.