How to Execute Shell Commands in Python

John on September 22, 2020

Sometimes one needs to run a shell command in Python. There are many reasons for this, usually, it is to access the functionality of a command-line based third-party program though it can be for other useful tasks such as running maintenance on the server.

In this tutorial, we will learn how to execute shell commands in Python.

Import the os package

The easiest way to run a shell command is with the os native Python package. os provides a collection of methods for working with the file system including running shell commands. Put the following at the top of your program to import the os package.

import os

Running a Command using os.system()

Now the os package has been imported we can use the system() method to execute a command. We will store the response from the command in a variable and print it. The example below is executing an ls command to list files and directories in the current working directory.

import os

output = os.system("ls")

print(output)

0

If you are running your program inside Jupyter notebook the ls command will return 0 for success or -1 for an error; the actual output of the command will be shown in the Jupyter terminal window.

Using the system() method works great if we just need to run a command and know if it was successful – to get output data we will need to open a pipe.

Open a Pipe to get the Output from a Terminal Command

To get the output from a terminal command in Python using os we can open a pipe using the popen() method. This will create a stream that can be read with the read() method. Let's run the same command (ls) as we did in the first example and get the output inside the Python program.

stream = os.popen("ls")
output = stream.read()
print(output)

enumerate.ipynb
example.json
fruit.json
new_dir
sentence.txt
shell commands.ipynb

Get the Output of a Terminal Command as an Array

The read() method will collect the whole output and return it as a string. To return each line as an element of an array use the readlines() method.

stream = os.popen("ls")
output = stream.readlines()
print(output)

['enumerate.ipynb\n', 'example.json\n', 'fruit.json\n', 'new_dir\n','sentence.txt\n', 'shell commands.ipynb\n']

To remove the \n (newlines) and extra empty spaces iterate through the lines using a for loop and use the .strip() method before appending the line to a new array.

stream = os.popen("ls")
temp = stream.readlines()
output = []

for l in temp:
    output.append(l.strip())

print(output)

['enumerate.ipynb', 'example.json', 'fruit.json', 'new_dir','sentence.txt', 'shell commands.ipynb']

Using the subprocess Package

The most flexible way of running commands in Python is by using the subprocess package, which can be used to chain inputs/outputs among various other functionalities.  To use it import it at the top of your program.

import subprocess

Run a Command with subprocess

The easiest way to use subprocess is with the run() method. It will return an object containing the command that was run and the return code.

success = subprocess.run(["ls", "-l"])

print(success)
print(success.returncode)

CompletedProcess(args=['ls', '-l'], returncode=0)
0

note - to run multiple commands or use command arguments, they must be passed into subprocess as an array of strings.

Get the Output of a Command from subprocess

To get the output of a command with subprocess pass stdout=subprocess.PIPE as an argument of subprocess.run(). This essentially creates a property on the output object containing the returned data from the terminal command. We can get this property on the output of subprocess to get the output of the command.

success = subprocess.run(["ls"], stdout=subprocess.PIPE)
print(success.stdout)

b'enumerate.ipynb\nexample.json\nfruit.json\nnew_dir\nshell commands.ipynb\n'

Plain Text Output from subprocess

By default subprocess.pipe() returns data in a byte format, which in most cases isn't very useful. To get a formatted plain text output pass text=True as an argument of subprocess.run().

success = subprocess.run(["ls"], stdout=subprocess.PIPE, text=True)
print(success.stdout)

enumerate.ipynb
example.json
fruit.json
new_dir
shell commands.ipynb

Provide an Input to the Command with subprocess

It is possible to send data to use in the terminal command by passing input="" as an argument subprocess.run().

success = subprocess.run(["ls"], input="some data")

Don't Display Any Output in the Console

To not display any output in the console pass stdout=subprocess.DEVNULL as an argument of subprocess.run().

response = subprocess.run(["ls", "-l"], stdout=subprocess.DEVNULL)

Using Popen in subprocess

Let's try the Popen() method to get the output from a command, which provides more shell interaction options. To run open a pipe and get an output in Python we will have to subprocess.pipe() as the second and third arguments.

process = subprocess.Popen("ls", stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output = process.communicate()
print(output)

(b'enumerate.ipynb\nexample.json\nfruit.json\nnew_dir\nsentence.txt\nshell commands.ipynb\n', b'')

To get the output from the pipe opened in Popen(), the communicate() method must be called.

You will see that the output is in the bytes format. To fix that pass universal_newlines=True as an argument in Popen().

process = subprocess.Popen("ls", stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)

Read more about the Popen constructor for more details about the extra functionality it has.

Conclusion

You now know how to run shell commands in Python using two different packages and get the output from the command if needed.