jump to navigation

Pipes to my rescue April 10, 2007

Posted by suniljagadish in Unix.
add a comment

I came across a situatuion to very simply join 2 files on my machine, wherein one is a huge gzip file (16GB after compression) and the other is flat text file. Unzipping the .gz was not a good idea. So, I had to zcat the .gz and pipe it into a the join command to get my job done. The join command requires both the file names to be specified and does not allow something like:

$ zcat myfile.gz | join myotherfile.txt

It expects the command to be used as:

$ join file1 file2

I was too lazy to write a Perl script or even awk code to do this. Given that I have been using Unix extensively only recently, I didn’t know that some commands allow you to do this:

$ join file1 -
where, “-” means- take the input from the standard input.

That solves my problem!

All I did was:

$ zcat myfile.gz | join myotherfile.txt – > output.txt