[Solved-2 Solutions] Flatten tuple like a bag in pig ?
Why Flatten
- flatten can also be applied to a tuple. In this case, it does not produce a cross product; instead, it elevates each field in the tuple to a top-level field. Again, empty tuples will remove the entire record. If the fields in a bag or tuple that is being flattened have names, Pig will carry those names along.
Problem:
- Dataset looks like the following:
( A, (1,2) )
( B, (2,9) )
- We would like to "flatten" the tuples in Pig, basically repeating each record for each value found in the inner-tuple, such that the expected output is:
( A, 1 )
( A, 2 )
( B, 2 )
( B, 9 )
We know this is possible when the tuples (1,2) and (2,9) are bags instead.
Solution 1:
input: (1-2-3, abc)
(4-5-6, xyz)
desired output:
(1, abc)
(2, abc)
(3, abc)
(4, xyz)
(5, xyz)
(6, xyz)
Initially, we used STRSPLIT that generates a tuple resulting in the similar input as above but was unsuccessful.
output = FOREACH input GENERATE FLATTEN(TOBAG(STRSPLIT($0, '-'))), $1
Output :
(1,2,3,abc)
(4,5,6,xyz)
- We can also use tokenize and that’s produces the desired result
output = FOREACH input GENERATE FLATTEN(TOKENIZE(REPLACE($0,'-', ' '))), $1;Solution 2:
- The TOBAG() function of Pig Latin converts one or more expressions to individual tuples. And these tuples are placed in a bag.
Syntax
- Given below is the syntax of the TOBAG() function.
TOBAG(expression [, expression ...])Here is the solution that use the TOBAG operator ?
A = LOAD 'data.txt' AS (a:chararray,b:(b1:chararray,b2:chararray));
B = FOREACH A GENERATE a, TOBAG(b.b1,b.b2);
C = FOREACH B GENERATE a, FLATTEN($1);