Pipes to my rescue April 10, 2007
Posted by suniljagadish in Unix.add a comment
I came across a situatuion to very simply join 2 files on my machine, wherein one is a huge gzip file (16GB after compression) and the other is flat text file. Unzipping the .gz was not a good idea. So, I had to zcat the .gz and pipe it into a the join command to get my job done. The join command requires both the file names to be specified and does not allow something like:
$ zcat myfile.gz | join myotherfile.txt
It expects the command to be used as:
$ join file1 file2
I was too lazy to write a Perl script or even awk code to do this. Given that I have been using Unix extensively only recently, I didn’t know that some commands allow you to do this:
$ join file1 -
where, “-” means- take the input from the standard input.
That solves my problem!
All I did was:
$ zcat myfile.gz | join myotherfile.txt – > output.txt