removing duplicates from a group

In this Mainframe Forum - You can post your queries on DFSORT, ICETOOL , SyncSort & JCL Utilities

Moderators: Frank Yaeger, Moderator Group

Post Reply
karthi5555
Member
Posts: 6
Joined: Tue Dec 01, 2015 11:59 pm

removing duplicates from a group

Post by karthi5555 » Fri Dec 11, 2015 9:01 pm

Hi,

Can you please help with removing duplicate block..

Input file:
HDR1
DTL1
DTL2
DTL3
HDR2
DTL1
DTL2
HDR1 -- Duplicate
DTL1
DTL2
DTL3

HDR1
DTL1
HDR2
DTL2
DTL3

Output Expected:
HDR1
DTL1
DTL2
DTL3
HDR2
DTL1
DTL2
HDR1
DTL1
HDR2
DTL2
DTL3

William Collins
Active Member
Posts: 732
Joined: Thu May 24, 2012 4:07 am

Post by William Collins » Fri Dec 11, 2015 9:49 pm

How do you know it is duplicate?

karthi5555
Member
Posts: 6
Joined: Tue Dec 01, 2015 11:59 pm

Post by karthi5555 » Mon Dec 14, 2015 9:43 am

HDR.. DTL is a block.
If that repeats (color coded) , it has to be removed.

Input file:

HDR1 -- Original block blk #1
DTL1
DTL2
DTL3

HDR2 ---------------------- blk #2
DTL1
DTL2
HDR1 --------- Duplicate Blk#3
DTL1
DTL2
DTL3

HDR1 ---------------------- blk #4
DTL1
HDR2 ---------------------- blk #5
DTL2
DTL3

William Collins
Active Member
Posts: 732
Joined: Thu May 24, 2012 4:07 am

Post by William Collins » Mon Dec 14, 2015 12:26 pm

OK, but SORT has no support for colour-recognition. How is SORT supposed to know the group is duplicate? Note, this is the same question, you provided nothing with this "cute" answer.

karthi5555
Member
Posts: 6
Joined: Tue Dec 01, 2015 11:59 pm

Post by karthi5555 » Tue Dec 15, 2015 2:51 am

The color coding is not for the sort but for the members for better understanding of the problem in hand.

Just to rephrase my question -
Can the sort remove duplicate on the (HDR-DTL) block level instead of individual record ?

Note: 'HDR' denotes the beginning of the block and that block continues as 'DTL' till the next HDR (or end of file) is encountered.

To answer your question - how is sort supposed to know if it was duplicate ?
Sort needs to accumulate the record as block and then compare with every block to see if it had any match. If the match is positive, don't write the block to the output.

William Collins
Active Member
Posts: 732
Joined: Thu May 24, 2012 4:07 am

Post by William Collins » Tue Dec 15, 2015 4:13 am

At last.

What is the maximum number of detail records? LRECL and RECFM?

Is this a once-off (data fix) or a regulat task (if yes, why?).

karthi5555
Member
Posts: 6
Joined: Tue Dec 01, 2015 11:59 pm

Post by karthi5555 » Tue Dec 15, 2015 8:47 pm

Number of detail records can be anywhere from 1 to n ( based on the prod data - 'n' could be 50,000 records) very dynamic .

LRECL - 61 , RECFM- FB

Currently, this is a production issue we found out recently and trying to address. So, if the fix is deployed - it should filter out all the duplicate blocks.

Why duplicating ?
When the user scans the shipment twice (or multiple times), the scanned data will be sent from DC to mainframe as many times. This human error happens once a while.

srijayam
Member
Posts: 2
Joined: Tue Jan 05, 2016 1:36 pm
Contact:

Post by srijayam » Tue Jan 05, 2016 2:14 pm

As a beginner like me, this is so helpful.

William Collins
Active Member
Posts: 732
Joined: Thu May 24, 2012 4:07 am

Post by William Collins » Tue Jan 05, 2016 3:10 pm

srijayam,

That's a lie. You are simply posting that to introduce SPAM to the site.

karthi5555
Member
Posts: 6
Joined: Tue Dec 01, 2015 11:59 pm

Post by karthi5555 » Thu Jan 14, 2016 11:50 pm

I am looking forward to getting an answer for this post..
Please .. Any directions/suggestions would help ..

William Collins
Active Member
Posts: 732
Joined: Thu May 24, 2012 4:07 am

Post by William Collins » Fri Jan 15, 2016 4:08 pm

If there is nothing to distinguish one group of records from another, then every group in the file has to be compare to every other group in the file. Can you show better examples of the data? Needs a key to be able to work.

srijayam
Member
Posts: 2
Joined: Tue Jan 05, 2016 1:36 pm
Contact:

Reply

Post by srijayam » Mon Jan 25, 2016 4:24 pm

Hi William Collins, I have not replied this thread for making it Spam. I am not in the need to do this. I have heard many thing about Forums. So y i joined.

William Collins
Active Member
Posts: 732
Joined: Thu May 24, 2012 4:07 am

Post by William Collins » Mon Jan 25, 2016 7:28 pm

Untrue, and you prove it again.

If it were true you were here innocently, you wouldn't have that spam link, would you? If you are here to be serious, get rid of the link. If not here to be serious, get rid of yourself.

academyindia4

Topic deleted by Admin

Post by academyindia4 » Mon Jan 25, 2016 9:41 pm

<< Content deleted By Admin >>

Post Reply

FREE TUTORIALS

Tutorials
Free tutorials from mainframegurukul
  • JCL Tutorial
    Covers all important JCL concepts.
  • Cobol Tutorial
    This tutorials covers all Cobol Topics from STRING to COMP-3.
  • DB2 Tutorial
    DB2 Tutorial focuses on DB2 COBOL Programming.
  • SORT Tutorial
    This Tutorial covers all important aspects of DFSORT with examples
  • CICS Tutorial
    This CICS tutorial covers CICS concepts and CICS Basics, CICS COBOL Programming.
Interview
Mainframe Interview questions



Other References
Mainframe Tools and others