Easytrieve Data Validation Projects

Validation Job Architecture

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
JOB
PERFORM INIT-VALIDATE
 
JOB INPUT PARTNER-FILE
  ADD 1 TO READ-COUNT
  PERFORM VALIDATE-RECORD
  IF VAL-STATUS NE ZERO
     PERFORM HANDLE-VALIDATION-ERROR
  ELSE
     WRITE CLEAN-FILE
     ADD 1 TO OK-COUNT
  END-IF
END-JOB
 
REPORT VALIDATION-EXCEPTIONS
  JOB INPUT ERROR-FILE
  CONTROL ERROR-CODE
  LINE ERR-KEY ERR-CODE ERR-MSG
END-REPORT
 
PERFORM TERM-VALIDATE
STOP

CLEAN-FILE feeds downstream load only when OK-COUNT positive and trailer checks pass. ERROR-FILE drives exception REPORT grouped by ERROR-CODE for analyst triage. INIT-VALIDATE loads TABLE rules effective today—rate tables, valid code lists.

Validation Rule Categories

Control Total Validation

After detail loop, PERFORM VALIDATE-TRAILER compares accumulated AMOUNT-SUM to trailer field from last record or separate trailer READ. Count validation compares READ-COUNT to header expected count when provided. Mismatch sets severity 4—STOP before CLEAN-FILE released to load JCL. Document penny tolerance if floating rounding differs—usually zero tolerance for money on mainframe integer arithmetic.

1
2
3
4
5
6
7
8
9
10
11
12
VALIDATE-TRAILER. PROC
  IF READ-COUNT NE HDR-REC-COUNT
     MOVE 80 TO VAL-STATUS
     PERFORM LOG-FATAL
     STOP
  END-IF
  IF AMOUNT-SUM NE TRL-AMOUNT
     MOVE 81 TO VAL-STATUS
     PERFORM LOG-FATAL
     STOP
  END-IF
END-PROC

Duplicate Detection

Sequential duplicate keys: compare current key to SAVE-KEY; if equal set duplicate error. Full-file duplicate detection requires SORT on key then JOB INPUT with CONTROL or compare adjacent records. Choose pattern based on partner SLA—within-file duplicates only versus database uniqueness simulation.

JCL Gate Pattern

1
2
3
4
//STEP10   EXEC PGM=EZTPA00,REGION=0M
//         DD ... PARTNER file
//STEP20   EXEC PGM=EZTPA00,REGION=0M,COND=(0,NE,STEP10)
//         DD ... LOAD uses CLEAN from STEP10

Load step runs only when validation return code zero. Some shops also test ERROR-FILE empty with IDCAMS or separate Easytrieve COUNT job. Document RC meanings in run book—RC 4 warning with rejects, RC 8 fatal trailer failure.

Partner File SLA

Validation jobs enforce contractual data quality: file must arrive by 06:00, trailer mandatory, maximum reject rate 0.1 percent. TERM-VALIDATE DISPLAY reject percentage; IF over threshold STOP and page on-call. ERROR-FILE returned to partner through secure transfer completes feedback loop. Track error code trends weekly—rising code 41 means master sync problem not partner typo.

Exception Report Design

REPORT VALIDATION-EXCEPTIONS uses CONTROL ERROR-CODE so analysts see sections per failure type. LINE includes record key, message, snippet of offending field. TITLE carries partner name and run date from reusable report framework. Limit detail lines per code if volume huge—summary LINE with COUNT per code in AFTER-BREAK plus sample first ten keys.

Reprocess Workflow

1. Partner receives ERROR-FILE layout specification.
2. Partner resubmits corrected full file or delta correction.
3. Validation reruns; compare OK-COUNT to prior run.
4. Merge policy: only CLEAN-FILE from successful run loads production.
5. Retain ERROR-FILE GDG for audit retention period.

Inline Versus Standalone Validation

Audit and Compliance

Log READ-COUNT OK-COUNT ERR-COUNT to immutable audit file with timestamp and program version. Exception REPORT PDF archived to document management. Validation rules versioned in Library with change ticket reference in INIT DISPLAY. Regulators ask prove what rules applied on trade date—version stamp answers.

Common Mistakes

• Load runs before validation COND—bad data in production.
• No exception REPORT—only ERROR-FILE analysts cannot read.
• Trailer check omitted when header promises count.
• Reject threshold zero without operations agreement—every typo stops chain.
• Duplicate validation logic diverging from ETL VALIDATE PROC—single Library source truth.

Explain It Like I'm Five

Before groceries go in the fridge, someone checks the delivery: eggs not cracked, list matches the receipt, nothing expired. Bad items go back to the store list. Data validation is that check for computer files. The fridge only opens when the good groceries pass—downstream load step is the fridge.

Exercises

1. Write validation JOB skeleton with CLEAN and ERROR outputs plus REPORT.
2. Define five error codes for a partner invoice file with severities.
3. Sketch JCL with COND gate before load step.
4. Describe trailer validation PROC for count and amount.
5. Compare standalone versus inline validation for your scenario.