An Easy Way to Remove Tourists from Photos
Quick Success Data Journalism
If you take photographs of famous landmarks or do any astrophotography, you're probably aware of image stacking. This is a process where you take many stationary pictures and then stack (average) them together to remove noise or other unwanted elements.
These techniques can remove anything that moves at a photo site, including people. Adobe Photoshop, for example, has a "crowd removal" script that magically vanishes nonstationary objects. It relies on a statistical average known as the median, which is simply the "middle" value in a list of numbers arranged from smallest to largest.
The process requires multiple photos. These are preferably taken with a tripod-mounted camera so that the objects you want to remove change positions from one image to the next, while the background remains constant. You typically need 10 to 30 pictures taken about 20 seconds apart, or similarly spaced frames extracted from a video.
With the mean, you sum numbers and divide by the total. With the median, you sort numbers and choose the middle value.
In the following figure, a row of five images is shown with the same pixel location outlined in each. The fourth image captured a black raven flying by. If you stack with the mean, the bird's presence lingers. But do a median stack on the images – that is, sort the red, green, and blue channels and take the middle values – and you get the background value for each channel (255). No trace of the bird remains.
When you average using the median, spurious values get pushed to the ends of the list. This makes it easy to remove outliers, such as satellites in astronomical photos, and tourists in landmark photos, so long as the number of images containing an outlier (in the same location) is less than half the number of images.
Armed with this knowledge, we can write an image-stacking program. For prototyping, we'll use five synthetic images of the moon, each "ruined" by a passing plane. We'll use URL addresses to load these directly from a GitHub Repository.
Our goal will be to write a Python program that completely removes the airplane from the shot, leaving only an image of the moon. I guarantee this will be a lot cheaper than a PhotoShop subscription!
The Code
The following code was written in JupyterLab and is presented on a cell-by-cell basis. The first part uses the Pillow library; the second repeats the process using the more sophisticated OpenCV library.
Importing Libraries
There are multiple ways to stack images with Python. We'll start by taking the simplest route and using the lightweight (and free) third-party Python module called Pillow. It's the successor project to the Python Imaging Library (PIL), which was discontinued in 2011.
The Pillow module "forked" the PIL repository and upgraded the code for Python 3. You can use Pillow on Windows, macOS, and Linux, and it supports many image formats including PNG, JPEG, GIF, BMP, and TIFF. It offers standard image manipulation procedures, such as altering individual pixels, masking, handling transparency, filtering and enhancing, and adding text. But the real strength of Pillow is its ability to edit many images with ease.
Installing Pillow is easy. To use the pip tool, visit this site. For Anaconda, use conda install pillow.
NOTE: Most major Linux distributions include Pillow in packages that previously contained PIL, so you may already have Pillow on your system. Regardless of your platform, if PIL is already installed, you'll need to uninstall it before installing Pillow.
To read the images from a URL address, we'll need the requests library for sending HTTP/1.1 requests, and the BytesIO class from the io module.
When you download an image using requests, the image data is received as a binary stream. To open this binary stream as an image using the PIL library, it must be converted into a file-like object. The BytesIO class provides a way to work with binary data (bytes) in memory as if it were a file.
The io module comes pre-installed with Python, but you'll need to install requests using either:
pip install requests
or
conda install requests
We'll also need Python's numerical package, NumPy, and its standard visualization package, Matplotlib. You can find installation instructions in the previous hyperlinks.
import numpy as np
import matplotlib.pyplot as plt
import requests
from io import BytesIO
from PIL import ImageFor backward compatibility, Pillow is still imported as PIL.
Loading and Converting the Images
The next cell loads the five images from GitHub using Pillow's Image.open() method and stores them in a list. This method returns an object of class type PIL.PngImagePlugin.PngImageFile.
To calculate the median values for the images, we'll need to convert them into NumPy arrays using NumPy's array() method.
# Load images:
image_urls = ['https://bit.ly/3Z1A7VL',
'https://bit.ly/3z9G48n',
'https://bit.ly/3zckWyl',
'https://bit.ly/3zcl0hz',
'https://bit.ly/3Xb5tXi']
images = []
for url in image_urls:
response = requests.get(url)
img = Image.open(BytesIO(response.content))
images.append(img)
# Convert images to numpy arrays
image_arrays = [np.array(img) for img in images]Note how we applied the BytesIO class within the Image.open() method, as the method expects a file-like object.
Working with Images Stored on Your Machine
When working with images stored on your computer, you don't need the previous code for importing from a URL address. You can just reference your folder path or move the images into the same folder as your notebook or script. Here's how this would look if you had the moon pictures stored on your machine:
# How to load images stored in the same folder as your code:
image_files = ['moon_1.png', 'moon_2.png',
'moon_3.png', 'moon_4.png',
'moon_5.png']
images = [Image.open(img) for img in image_files]We'll continue to use the URL approach for the rest of this project.
Creating the Median Image
The code in the next cell takes our list of images (stored as NumPy arrays), stacks them, computes the median value for each pixel, and then converts this stacked array back into PIL format for display.
# Stack images along a new dimension:
stacked_images = np.stack(image_arrays, axis=-1)
# Compute the median along the new dimension:
median_image = np.median(stacked_images, axis=-1).astype(np.uint8)
# Convert the median image back to PIL format:
median_image_pil = Image.fromarray(median_image)In the first step, the np.stack() NumPy method joins the sequence of arrays along a new axis. An axis is just a dimension along which NumPy performs operations. A 1D array, for example, would be a linear sequence of numbers. A 2D array would be a matrix. Here, the axis=-1 argument specifies that the new axis should be added as the last dimension of the resulting array.
So, if you have multiple 2D images (grayscale) or 3D images (color), this line will combine them into a single array with an additional dimension. This new dimension will contain all the different pictures, letting you perform operations across them.
Next, we compute the median along this new dimension using NumPy's median() method. I don't need to tell you what that does. The output is a single image in NumPy's [uint8](https://numpy.org/doc/stable/user/basics.types.html) data type.
Finally, we use the PIL Image.fromarray() method to convert the median-stacked image into PIL format for display.
Displaying the Stacked Image
The last cell displays the image using Matplotlib's imshow() method.
# Display the result in Jupyter Notebook:
plt.imshow(median_image_pil)
plt.axis('off') # Hide the axis
plt.show()Here's the result. Despite every input image containing an airplane silhouette, the final image is pure moon.
A nice thing about his approach is that every part of this moon image is truly an image of the moon, unlike some AI techniques that "guess" at obscured backgrounds. Of course, if you have only a single image, the AI approach is your only solution.
Stacking images with OpenCV
OpenCV is Python's most popular computer vision library. It's a lot more involved than Pillow, but if you're an OpenCV user, here's one way to do it (this assumes the images are stored in the same folder as your code):
import cv2
import numpy as np
import matplotlib.pyplot as plt
# Load the images:
image_files = ['moon_1.png', 'moon_2.png',
'moon_3.png', 'moon_4.png',
'moon_5.png']
images = [cv2.imread(img) for img in image_files]
# Stack images along a new dimension:
stacked_images = np.stack(images, axis=-1)
# Compute the median along the new dimension:
median_image = np.median(stacked_images, axis=-1).astype(np.uint8)
# Display the result in Jupyter Notebook:
plt.imshow(cv2.cvtColor(median_image, cv2.COLOR_BGR2RGB))
plt.axis('off') # Hide the axis
plt.show()Since OpenCV treats images as NumPy arrays by default, there are fewer conversions – and less code – required than with Pillow.
Removing Tourists
So how does this code work for removing tourists? Here's an example from the Taj Mahal:
There are limits to this technique. If you've ever been to the Taj Mahal, Leaning Tower of Pisa, or many other famous landmarks, you surely noticed that they weren't just crowded, they were supersaturated! Getting a tourist-free photo under those conditions requires another old (and painful) photographic technique: showing up at dawn!
The Recap
Median stacking lets you remove transient elements, such as wandering tourists, from static images. These transient elements are treated as outliers and the number of images containing an outlier (in the same location) should be less than half the total number of images.
In this project, we used Python's Pillow and OpenCV libraries to perform median stacking. For more image-stacking programs, check out my book, Impractical Python Projects, available online and in bookstores like Barnes and Noble.
Impractical Python Projects: Playful Programming Activities to Make You Smarter
Thanks!
Thanks for reading and clapping and please follow me for more Quick Success Data Science and Quick Success Data Journalism articles in the future.
