removing duplicates from a group
Moderators: Frank Yaeger, Moderator Group
-
- Member
- Posts: 6
- Joined: Tue Dec 01, 2015 11:59 pm
removing duplicates from a group
Hi,
Can you please help with removing duplicate block..
Input file:
HDR1
DTL1
DTL2
DTL3
HDR2
DTL1
DTL2
HDR1 -- Duplicate
DTL1
DTL2
DTL3
HDR1
DTL1
HDR2
DTL2
DTL3
Output Expected:
HDR1
DTL1
DTL2
DTL3
HDR2
DTL1
DTL2
HDR1
DTL1
HDR2
DTL2
DTL3
Can you please help with removing duplicate block..
Input file:
HDR1
DTL1
DTL2
DTL3
HDR2
DTL1
DTL2
HDR1 -- Duplicate
DTL1
DTL2
DTL3
HDR1
DTL1
HDR2
DTL2
DTL3
Output Expected:
HDR1
DTL1
DTL2
DTL3
HDR2
DTL1
DTL2
HDR1
DTL1
HDR2
DTL2
DTL3
-
- Active Member
- Posts: 732
- Joined: Thu May 24, 2012 4:07 am
-
- Member
- Posts: 6
- Joined: Tue Dec 01, 2015 11:59 pm
HDR.. DTL is a block.
If that repeats (color coded) , it has to be removed.
Input file:
HDR1 -- Original block blk #1
DTL1
DTL2
DTL3
HDR2 ---------------------- blk #2
DTL1
DTL2
HDR1 --------- Duplicate Blk#3
DTL1
DTL2
DTL3
HDR1 ---------------------- blk #4
DTL1
HDR2 ---------------------- blk #5
DTL2
DTL3
If that repeats (color coded) , it has to be removed.
Input file:
HDR1 -- Original block blk #1
DTL1
DTL2
DTL3
HDR2 ---------------------- blk #2
DTL1
DTL2
HDR1 --------- Duplicate Blk#3
DTL1
DTL2
DTL3
HDR1 ---------------------- blk #4
DTL1
HDR2 ---------------------- blk #5
DTL2
DTL3
-
- Active Member
- Posts: 732
- Joined: Thu May 24, 2012 4:07 am
-
- Member
- Posts: 6
- Joined: Tue Dec 01, 2015 11:59 pm
The color coding is not for the sort but for the members for better understanding of the problem in hand.
Just to rephrase my question -
Can the sort remove duplicate on the (HDR-DTL) block level instead of individual record ?
Note: 'HDR' denotes the beginning of the block and that block continues as 'DTL' till the next HDR (or end of file) is encountered.
To answer your question - how is sort supposed to know if it was duplicate ?
Sort needs to accumulate the record as block and then compare with every block to see if it had any match. If the match is positive, don't write the block to the output.
Just to rephrase my question -
Can the sort remove duplicate on the (HDR-DTL) block level instead of individual record ?
Note: 'HDR' denotes the beginning of the block and that block continues as 'DTL' till the next HDR (or end of file) is encountered.
To answer your question - how is sort supposed to know if it was duplicate ?
Sort needs to accumulate the record as block and then compare with every block to see if it had any match. If the match is positive, don't write the block to the output.
-
- Active Member
- Posts: 732
- Joined: Thu May 24, 2012 4:07 am
-
- Member
- Posts: 6
- Joined: Tue Dec 01, 2015 11:59 pm
Number of detail records can be anywhere from 1 to n ( based on the prod data - 'n' could be 50,000 records) very dynamic .
LRECL - 61 , RECFM- FB
Currently, this is a production issue we found out recently and trying to address. So, if the fix is deployed - it should filter out all the duplicate blocks.
Why duplicating ?
When the user scans the shipment twice (or multiple times), the scanned data will be sent from DC to mainframe as many times. This human error happens once a while.
LRECL - 61 , RECFM- FB
Currently, this is a production issue we found out recently and trying to address. So, if the fix is deployed - it should filter out all the duplicate blocks.
Why duplicating ?
When the user scans the shipment twice (or multiple times), the scanned data will be sent from DC to mainframe as many times. This human error happens once a while.
-
- Active Member
- Posts: 732
- Joined: Thu May 24, 2012 4:07 am
-
- Member
- Posts: 6
- Joined: Tue Dec 01, 2015 11:59 pm
-
- Active Member
- Posts: 732
- Joined: Thu May 24, 2012 4:07 am
Reply
Hi William Collins, I have not replied this thread for making it Spam. I am not in the need to do this. I have heard many thing about Forums. So y i joined.
-
- Active Member
- Posts: 732
- Joined: Thu May 24, 2012 4:07 am
FREE TUTORIALS
Tutorials
Free tutorials from mainframegurukul
- JCL Tutorial
Covers all important JCL concepts. - Cobol Tutorial
This tutorials covers all Cobol Topics from STRING to COMP-3. - DB2 Tutorial
DB2 Tutorial focuses on DB2 COBOL Programming. - SORT Tutorial
This Tutorial covers all important aspects of DFSORT with examples - CICS Tutorial
This CICS tutorial covers CICS concepts and CICS Basics, CICS COBOL Programming.
Interview
Mainframe Interview questions
- Cobol Interview Questions
50+ Interview Questions - JCL Interview Questions
50+ Interview Questions - DB2 Interview Questions
100+ Interview Questions - CICS Interview Questions
70+ Interview Questions - VSAM Interview Questions
27 Interview Questions
Other References
Mainframe Tools and others
- XPEDITER Reference
Explains how we can debug a program - FILEAID Reference
Explains how to browse , edit and delete datasets - Change Man Reference
Quick Start tutorial on Changeman - Abend Reference
Important Abend codes explained - FaceBook Page
MainframeGurukul FaceBook Page - LinkedIn Page
MainframeGurkul Linkedin Page