Data consolidation combines information from multiple sources into unified datasets. Learn to merge files, aggregate records, and consolidate data for comprehensive business reporting and analysis.
12345678910111213141516171819202122*> Merge two sorted files by customer ID OPEN INPUT FILE1 FILE2 OPEN OUTPUT MERGED-FILE READ FILE1 AT END SET EOF1 TO TRUE READ FILE2 AT END SET EOF2 TO TRUE PERFORM UNTIL EOF1 AND EOF2 IF EOF1 PERFORM WRITE-FILE2-RECORD ELSE IF EOF2 PERFORM WRITE-FILE1-RECORD ELSE IF CUSTOMER-ID1 < CUSTOMER-ID2 PERFORM WRITE-FILE1-RECORD ELSE IF CUSTOMER-ID1 > CUSTOMER-ID2 PERFORM WRITE-FILE2-RECORD ELSE PERFORM MERGE-MATCHING-RECORDS END-IF END-PERFORM CLOSE FILE1 FILE2 MERGED-FILE.
Merge sorted files by comparing key fields. Handle three cases: file1 record only, file2 record only, and matching records. When records match, combine or choose the appropriate data based on business rules.
12345678910111213141516171819202122232425WORKING-STORAGE SECTION. 01 AGGREGATE-RECORD. 05 CUSTOMER-ID PIC 9(6). 05 TOTAL-AMOUNT PIC 9(9)V99 VALUE 0. 05 RECORD-COUNT PIC 9(6) VALUE 0. 05 FIRST-DATE PIC 9(8). 05 LAST-DATE PIC 9(8). PROCEDURE DIVISION. PERFORM UNTIL EOF READ INPUT-FILE AT END SET EOF TO TRUE NOT AT END IF CUSTOMER-ID = PREVIOUS-CUSTOMER-ID ADD TRANSACTION-AMOUNT TO TOTAL-AMOUNT ADD 1 TO RECORD-COUNT IF TRANSACTION-DATE > LAST-DATE MOVE TRANSACTION-DATE TO LAST-DATE END-IF ELSE PERFORM WRITE-AGGREGATE-RECORD PERFORM INITIALIZE-AGGREGATE END-IF END-READ END-PERFORM PERFORM WRITE-FINAL-AGGREGATE.
Aggregate records by grouping them by key fields and calculating totals, counts, and other summary statistics. Use control break logic to detect when to write aggregated records.
12345678910111213141516171819202122WORKING-STORAGE SECTION. 01 SUMMARY-RECORD. 05 SUMMARY-KEY PIC X(10). 05 SUMMARY-TOTALS. 10 TOTAL-SALES PIC 9(9)V99 VALUE 0. 10 TOTAL-COST PIC 9(9)V99 VALUE 0. 10 TOTAL-PROFIT PIC 9(9)V99 VALUE 0. 10 RECORD-COUNT PIC 9(6) VALUE 0. PROCEDURE DIVISION. PERFORM UNTIL EOF READ DETAIL-FILE AT END SET EOF TO TRUE NOT AT END MOVE REGION-CODE TO SUMMARY-KEY ADD SALES-AMOUNT TO TOTAL-SALES ADD COST-AMOUNT TO TOTAL-COST ADD 1 TO RECORD-COUNT END-READ END-PERFORM COMPUTE TOTAL-PROFIT = TOTAL-SALES - TOTAL-COST WRITE SUMMARY-RECORD.
Summarize detailed data into summary records by calculating totals, averages, and other aggregate measures. Group data by summary keys and perform calculations across all records in each group.
123456789101112*> Consolidate customer data from multiple sources PERFORM LOAD-CUSTOMER-MASTER PERFORM LOAD-CUSTOMER-TRANSACTIONS PERFORM LOAD-CUSTOMER-PREFERENCES PERFORM CONSOLIDATE-CUSTOMER-DATA VARYING I FROM 1 BY 1 UNTIL I > CUSTOMER-COUNT MOVE CUSTOMER-ID(I) TO SEARCH-KEY PERFORM FIND-TRANSACTION-DATA PERFORM FIND-PREFERENCE-DATA PERFORM BUILD-CONSOLIDATED-RECORD END-PERFORM.
Consolidate data from multiple sources by matching records across different files. Load reference data into memory, then match and combine related records to create comprehensive consolidated records.
123456789101112131415161718192021WORKING-STORAGE SECTION. 01 DUPLICATE-HANDLING. 05 CURRENT-KEY PIC X(10). 05 PREVIOUS-KEY PIC X(10) VALUE SPACES. 05 DUPLICATE-COUNT PIC 9(4) VALUE 0. PROCEDURE DIVISION. PERFORM UNTIL EOF READ INPUT-FILE AT END SET EOF TO TRUE NOT AT END MOVE RECORD-KEY TO CURRENT-KEY IF CURRENT-KEY = PREVIOUS-KEY ADD 1 TO DUPLICATE-COUNT PERFORM HANDLE-DUPLICATE-RECORD ELSE PERFORM WRITE-CONSOLIDATED-RECORD MOVE CURRENT-KEY TO PREVIOUS-KEY MOVE 0 TO DUPLICATE-COUNT END-IF END-READ END-PERFORM.
Handle duplicate records during consolidation by detecting when consecutive records have the same key. Choose appropriate strategy: keep first, keep last, merge data, or flag duplicates for manual review.
1234567891011121314151617CONSOLIDATE-RECORD. IF CUSTOMER-ID IS NUMERIC AND CUSTOMER-ID > 0 IF CUSTOMER-NAME NOT = SPACES IF TRANSACTION-AMOUNT IS NUMERIC PERFORM ADD-TO-CONSOLIDATION ELSE DISPLAY 'Invalid transaction amount for customer: ' CUSTOMER-ID ADD 1 TO INVALID-COUNT END-IF ELSE DISPLAY 'Missing customer name for ID: ' CUSTOMER-ID ADD 1 TO INVALID-COUNT END-IF ELSE DISPLAY 'Invalid customer ID: ' CUSTOMER-ID ADD 1 TO INVALID-COUNT END-IF.
Validate data during consolidation to ensure data quality. Check for required fields, valid data types, and business rule compliance. Log validation errors and maintain counts for reporting.
12345678910111213FINALIZE-CONSOLIDATION. DISPLAY 'Data Consolidation Summary' DISPLAY '========================' DISPLAY 'Total input records: ' INPUT-COUNT DISPLAY 'Records consolidated: ' CONSOLIDATED-COUNT DISPLAY 'Duplicate records: ' DUPLICATE-COUNT DISPLAY 'Invalid records: ' INVALID-COUNT DISPLAY 'Records written: ' OUTPUT-COUNT DISPLAY 'Processing time: ' PROCESSING-TIME ' seconds' IF INVALID-COUNT > 0 DISPLAY 'Warning: ' INVALID-COUNT ' invalid records found' END-IF.
Generate consolidation reports showing processing statistics, record counts, and data quality metrics. Include totals for input, output, duplicates, and invalid records for audit and monitoring purposes.
1234567891011121314151617*> Use efficient sorting and merging *> Sort files by consolidation keys before processing *> Use appropriate buffer sizes for large files *> Consider memory-based consolidation for small datasets PERFORM UNTIL EOF READ INPUT-FILE AT END SET EOF TO TRUE NOT AT END IF RECORD-COUNT < MAX-MEMORY-RECORDS PERFORM ADD-TO-MEMORY-TABLE ELSE PERFORM PROCESS-MEMORY-TABLE PERFORM INITIALIZE-MEMORY-TABLE END-IF END-READ END-PERFORM PERFORM PROCESS-FINAL-MEMORY-TABLE.
Optimize consolidation performance by using efficient algorithms, appropriate buffer sizes, and memory-based processing for small datasets. Consider sorting strategies and I/O optimization techniques.