In Linux, when we work from the command line, it is often very convenient to be able to execute several commands in a single line. For example if you want to download, decompress and open the resulting folder, you would execute a single command line as follows:
sudo wget https://github.com/git/git/archive/master.zip -O git.zip ; unzip git.zip ; cd git-*
instead of running several of these commands separately and with one command per line. It is nevertheless important in the execution of these commands that each succeeds so that the next one can proceed.
Sometimes however you may need to run several commands simultaneously for one reason or another, It is the aim of this article to show you how to pull this off using GNU Parallel.
What is GNU Parallel
GNU parallel is a shell program for Linux and other unix operating systems that allows the user to execute shell scripts simultaneously. Operations are divided between the CPU threads of the computer but it can also be shared between several computers. GNU parallel is free of charge and was written by Ole Tange in Perl.
GNU installation
GNU Parallel can be installed on almost any Linux distro. I will be using Ubuntu for the demonstration. In case you are using another distribution, you will have to adapt the installation command to your platform.
Read: How to use the APT command on Ubuntu/Debian Linux systems
As GNU Parallel is available in the Ubuntu standard repository, the installation procedure is fairly simple. In your terminal, run the command:
sudo apt install parallel
GNU parallel linux installation
Now you are all set to start using GNU Parallel.
Read: How to tackle multiple tasks at once on Ubuntu 22.04
Using GNU Parallel
Example 1 :
Let us say you have a text file as follows:
cat file.txt
Now we want to compress each of the png files above using GNU Parallel. This can be done as follows :
cat file.txt | parallel -j 4 gzip -k {} [linux parallel command]
Where you can see that all the files mentioned in the file.txt file were compressed.
Example 2 :
Another example in which the command fetches all logo*.png files and zip them :
find -type f -name logo’*.png’ | parallel gzip –best
At the top, you can see all logo*.png files before running the gzip command on them using the GNU parallel tool. At the end, all of these png files were compressed.
When the results of find are piped to parallel, each resulting item on each output line is considered as one argument by the command GNU parallel is handling.
Read: How to run a command without having to wait in Linux/Ubuntu
Example 3 :
If you have many commands that you need to run simultaneously, you can insert them all in a file before handing them over to GNU parallel as follows:
First, using cat or your preferred command , create a text file in which you append one command on each line, for instance :
cat tasks
gzip image.png
convert dog.png
dog.jpg
opusenc dogbarking.wav
ffmepg -i myvideo.avi -v:b 12000k myvideo.mp4
bzip2 myimages.tar
Then let GNU Parallel perform its job using the command :
parallel –jobs 6 < tasks
The –jobs option tells GNU Parallel about the number of allowed commands be to run. This will run all the commands specified in the file tasks in Parallel. If the number of jobs that exist exceed the number of jobs allowed, GNU Parallel will maintain a queue until all jobs have been executed.
GNU Parallel will start a job and executes each command in the file separately. Parallel will proceed with this as long as it is allowed to create new jobs without crippling the system. As soons as some jobs are finished, Parallel will create new ones, until all the tasks handed over to it have been processed. What took 5 minutes before, might now take only 1 or 2 with Parallel.
Example 4 :
find name “*.jpg” | parallel -I% –max-args 1 convert % %.png
Here we have the find command which fetches all files with the extension jpg before piping them to the parallel command which converts each image into the png format.
Read: How to tweet from the Linux Command Line
-I% is a placeholder named % which will act as a single file handed over by find command. If you do not do this, you will need to manually issue a new command for every result of find.
–max-args 1 specifies the rate at which a new object from the queue is requested by GNU Parallel. The rate is limited to 1 in the example above as the command that is run by GNU Parallel requires only one file. The rate would be limited to 3 for instance in case the command requires three files for instance.
convert % %.png is needed to convert the jpg images to png. This is the command to be run in parallel.
Conclusion :
GNU Parallel is a flexible, simple yet powerful tool. If you master its usage, you can gain a lot of time especially when you work on independent tasks.
We have provided only simple use cases in the article but be assured that it can be used with far more complex real life scenarios.
You can refer to its main page in which it contains many useful examples like Bash scripts handling or remote execution over SSH.
You can also watch the tutorial series on Youtube by the creator of the tool.
If you have other useful examples using GNU Parallel , feel free to mention them in the comment section below.
If you like the content, we would appreciate your support by buying us a coffee. Thank you so much for your visit and support.
