Introducing NumPy, Part 1: Understanding Arrays

Quick Success Data Science

Welcome to Introducing NumPy, a four-part series for Python (or NumPy) beginners. The aim is to demystify NumPy by showcasing its core functionalities, supplemented with tables and hands-on examples of key methods and attributes.

NumPy, short for Numerical Python, serves as Python's foundational library for numerical computing. It extends Python's mathematical capability and forms the basis of many scientific and mathematical packages.

NumPy augments the built-in tools in the Python Standard Library, which can be too simple for many data analysis calculations. Using NumPy, you can perform fast operations, including mathematics, sorting, selecting, I/O, discrete Fourier transforms, basic linear algebra, basic statistical operations, shape manipulation, random simulation, and more.

At the heart of NumPy lies the array data structure, a grid of values that forms the core of its functionality. By leveraging precompiled C code, NumPy enhances the performance of slower algorithms and executes complex mathematical computations with great efficiency. By supporting multidimensional Arrays and array-based operations, NumPy simplifies handling extensive, uniform datasets, ranging from millions to billions of samples.

NumPy is a big library, so I've broken this series into four parts:

• Part 1: Understanding Arrays
• Part 2: Indexing Arrays
• Part 3: Manipulating Arrays
• Part 4: Doing Math with Arrays

In addition to this series, you can find both "quickstart" and more detailed tutorials and guides at NumPy's official site.

Installing NumPy

NumPy is open-source and easy to install.

With conda, use:

conda install numpy

With pip, use:

pip install numpy

Introducing the Array

In computer science, an array is a data structure that contains a group of elements (values or variables) of the same size and data type (referred to as dytpes in NumPy). An array can be indexed by a tuple of nonnegative integers, by Booleans, by another array, or by integers.

Here's an example of a two-dimensional array of integers, comprising a grid of two rows and three columns. Because arrays use square brackets, they look a lot like Python lists:

In [1]: import numpy as np

In [2]: arr = np.array([[0, 1, 2],
   ...:                 [3, 4, 5]])

In [3]: arr

Out[3]: 
array([[0, 1, 2],
       [3, 4, 5]])

You can use standard indexing and slicing techniques to select an element from this array. For example, to select element 2, you first index the row and then the column, using [0][2] (remember: Python starts counting at 0, not 1).

There are several reasons why you should work with arrays. Accessing individual elements by index is extremely efficient, making runtimes constant regardless of the array size. Arrays let you perform complex computations on entire blocks of data without the need to loop through and access each element one at a time. As a result, NumPy-based algorithms run orders of magnitude faster than those in native Python.

In addition to being faster, arrays store data in contiguous memory blocks, resulting in a significantly smaller memory footprint than built-in Python sequences, like lists. A list, for example, is basically an array of pointers to (potentially) heterogeneous Python objects stored in non-contiguous blocks, making it much less compact than a NumPy array. Consequently, arrays are often the preferred data structure for storing data reliably and efficiently. The popular OpenCV computer vision library, for example, manipulates and stores digital images as NumPy arrays.

Describing Arrays Using Dimension and Shape

Understanding arrays requires knowledge of their layout. The number of dimensions in an array is the number of indexes needed to select an element from the array. You can think of a dimension as an array's axis.

The number of dimensions in an array, also called its rank, can be used to describe the array. The following figure is a graphical example of one-, two-, and three-dimensional arrays.

The shape of an array is a tuple of integers representing the size of the array along each dimension, starting with the first dimension (axis 0). Example shape tuples are shown below each array in the previous figure. The number of integers in these tuples equals the array's rank.

A one-dimensional array, also referred to as a vector, has a single axis. This is the simplest form of an array and is the NumPy equivalent to Python's list data type. Here's an example of the code used to create the 1D array in the figure:

In [4]: arr_1d = np.array([5, 4, 9])

Arrays with more than one dimension are arrays within arrays. An array with both rows and columns is called a 2D array. The 2D array in the figure has a shape tuple of (2, 3) because the length of its first axis (0) is 2 and the length of its second axis (1) is 3.

A 2D array is used to represent a matrix. Remember from math class that these are rectangular grids of elements such as numbers or algebraic expressions, arranged in rows and columns and enclosed by square brackets. Matrices store data elegantly and compactly, and despite containing many elements, each matrix is treated as one unit.

Here's an example of the code used to create the 2D array in the figure:

In [5]: arr_2d = np.array([[4.1, 2.0, 6.7],
   ...:                    [0.3, 9.4, 2.2]])

Notice how this resembles a nested list.

An array with three or more dimensions is called a tensor. As mentioned earlier, arrays can have any number of dimensions. Here's the code for the 3D array in the figure:

In [6]: arr_3d = np.array([[[1, 0, 1, 1],
   ...:                     [0, 1, 1, 1],
   ...:                     [1, 1, 0, 1]],
   ...:                     
   ...:                    [[0, 0, 0, 0],
   ...:                     [0, 0, 0, 0],
   ...:                     [1, 1, 0, 1]]])

Tensors can be difficult to visualize in a two-dimensional display, but Python tries to help you out. Note how a blank line separates the two stacked matrices that comprise the 3D grid:

In [7]: arr_3d

Out[7]: 
array([[[1, 0, 1, 1],
        [0, 1, 1, 1],
        [1, 1, 0, 1]],

       [[0, 0, 0, 0],
        [0, 0, 0, 0],
        [1, 1, 0, 1]]])

You can also determine the rank of an array by counting the number of square brackets at the start of the output. Three square brackets in a row indicate a 3D array.

Creating Arrays

NumPy handles arrays through its ndarray class, also known by the alias array. The "nd" in the name is short for N-dimensional, as this class can handle any number of dimensions.

NumPy ndarrays have a fixed size at creation and can't grow like a Python list or tuple. Changing the size of an ndarray creates a new array and deletes the original.

You should know numpy.array is not the same as array.array, found in the Python Standard Library. The latter is only a one-dimensional array with limited functionality compared to NumPy arrays.

NumPy comes with several built-in functions for creating ndarrays. These let you create arrays outright or convert existing sequence data types, like tuples and lists, to arrays. The following table lists some of the more common creation functions. We'll look at some of these in more detail later. You can find a complete listing of creation functions in the official docs.

Because arrays must contain data of the same type, the array needs to know the dtype that's being passed to it. You'll have the choice of letting the functions infer the most suitable dtype (although you'll want to check the result) or providing the dtype explicitly as an additional argument.

Some commonly used dtypes are listed below. The "Code" column lists the shorthand arguments you can pass to the functions in single quotes, such as dtype= '8', in place of dtype= 'int64'. For a full list of supported data types, visit the NumPy docs.

For string and Unicode dtypes, the length of the longest string or Unicode object must be included in the dtype argument. For example, if the longest string in a dataset has 12 characters, the assigned dtype should be 'S12'. This is necessary because all the ndarray elements should have the same size. There's no way to create variable-length strings, so you must ensure that enough memory is allocated to hold every possible string in the dataset. NumPy can calculate this for you using existing input, such as when converting a list of strings to an array.

Because the amount of memory used by the dtypes is automatically assigned (or can be input), NumPy knows how much memory to allocate when creating ndarrays. The choices in the previous table give you plenty of control over how data is stored in memory, but don't let that intimidate you. Most of the time, all you'll need to know is the basic type of data you're using, such as a float or integer.

How NumPy Allocates Memory

The genius of NumPy is in how it allocates memory. The following figure shows information stored in a 3×4 2D array of numbers from 0 to 11, represented by the "Python View" diagram at the bottom of the figure.

The values of an ndarray are stored as a contiguous block of memory in your computer's RAM, as shown by the "Memory Block" diagram in the previous figure. This is efficient, as processors prefer items in memory to be in chunks rather than randomly scattered about. The latter occurs when you store data in Python datatypes like lists, which keep track of pointers to objects in memory, creating "overhead" that slows down processing.

To help NumPy interpret the bytes in memory, the dtype object stores additional information about the layout of the array, such as the size of the data (in bytes) and the byte order of the data. Because we're using the int32 dtype in the example, each number occupies 4 bytes of memory (32 bits/8 bits per byte).

The ndarray strides attribute is a tuple of the number of bytes to step in each dimension when traversing an array. This tuple informs NumPy on how to convert from the contiguous "Memory Block" to the "Python View" array shown in the figure.

In the figure, the memory block consists of 48 bytes (12 integers x 4 bytes each), stored one after the other. The array strides indicate how many bytes must be skipped in memory to move to the next position along a certain axis. For example, we must skip 4 bytes (1 integer) to reach the next column, but 16 bytes (4 integers) to move to the same position in the next row. Thus, the strides for the array are (16, 4).

Using the array() Function

The simplest way to create an array is to pass the NumPy array() function a sequence, such as a list, which is then converted into an ndarray. Let's do that now to create a 1D array. We'll begin by importing NumPy using the alias np (this is by convention and will reduce the amount of typing needed to call NumPy functions):

In [8]: import numpy as np

In [9]: arr1d = np.array([1, 2, 3, 4])

In [10]: type(arr1d)

Out[10]: numpy.ndarray

In [11]: print(arr1d)
[1 2 3 4]

You can also create an ndarray by passing the array() function a variable, like this:

In [12]: my_sequence = [1, 2, 3, 4]

In [13]: arr1d = np.array(my_sequence)

In [14]: arr1d

Out[14]: array([1, 2, 3, 4])

To create a multidimensional array, pass array() a nested sequence, where each nested sequence is the same length. Here's an example that uses a list containing three nested lists to create a 2D array:

In [15]: arr2d = np.array([[0, 1, 2], [3, 4, 5], [6, 7, 8]])

In [16]: print(arr2d)
[[0 1 2]
 [3 4 5]
 [6 7 8]]

Each nested list became a new row in the 2D array. To build the same array from tuples, you would replace all the square brackets [] in line In [15] with parentheses ().

When you print an array, NumPy displays it with the following layout: the last axis is printed from left to right, the second-to-last is printed from top to bottom, and the rest are also printed from top to bottom, with each slice separated from the next by an empty line. So, 1D arrays are printed as rows, 2D arrays as matrices, and 3D arrays as lists of matrices.

Now, let's check some of the 2D array's attributes, such as its shape:

In [17]: arr2d.shape

Out[17]: (3, 3)

its number of dimensions:

In [18]: arr2d.ndim

Out[18]: 2

and its strides:

In [19]: arr2d.strides

Out[19]: (12, 4)

Although the items in an array must be the same data type, this doesn't mean that you can't pass these items to the array() function within a mixture of sequence types, such as tuples and lists:

In [20]: mixed_input = np.array([[0, 1, 2], (3, 4, 5), [6, 7, 8]])

In [21]: mixed_input

Out[21]: 
array([[0, 1, 2],
       [3, 4, 5],
       [6, 7, 8]])

This worked because NumPy reads the data type of the elements in a sequence rather than the data type of the sequence itself. You won't have the same luck, however, if you try to pass nested lists of different lengths:

In [22]: arr2d = np.array([[0, 1, 2], [3, 4, 5], [6, 7]])

VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences 
(which is a list-or-tuple of lists-or-tuples-or ndarrays with different 
lengths or shapes) is deprecated. If you meant to do this, you must specify 
'dtype=object' when creating the ndarray.
 arr2d = np.array([[0, 1, 2], [3, 4, 5], [6, 7]])

You can avoid this warning by changing the dtype to object, as follows:

In [23]: arr2d = np.array([[0, 1, 2], [3, 4, 5], [6, 7]], dtype='object')

In [24]: print(arr2d)
[list([0, 1, 2]) list([3, 4, 5]) list([6, 7])]

You now have a 1D array of list objects rather than the 2D array of integers you probably wanted.

Just as with mathematical matrices, arrays need to have the same number of rows and columns if you plan to use them for mathematical calculations (there's some flexibility to this, using "broadcasting," but we'll save it for Part 4 of this series).

Now, let's look at arrays with more than two dimensions. The array() function transforms sequences of sequences into two-dimensional arrays; sequences of sequences of sequences into three-dimensional arrays; and so on. So, to make a 3D array, you need to pass the function multiple nested sequences. Here's an example using nested lists:

In [25]: arr3d = np.array([[[0, 0, 0],
    ...:                   [1, 1, 1]],
    ...:                  [[2, 2, 2],
    ...:                   [3, 3, 3]]])

In [26]: arr3d

Out[26]: 
array([[[0, 0, 0],
        [1, 1, 1]],

       [[2, 2, 2],
        [3, 3, 3]]])

Here, we passed the function a list containing two nested lists that each contained two nested lists. Notice how the output array has a blank line in the middle. This visually separates the two stacked 2D arrays created by the function.

Keeping track of all those brackets when creating high-dimension arrays can be cumbersome and dangerous to your eyesight. Fortunately, NumPy provides additional methods for creating arrays that can be more convenient than the array() function. We'll look at some of these in the next sections.

Using the arange() Function

To create arrays that hold sequences of numbers, NumPy provides the arange() function, which works like Python's built-in range() function, only it returns an array rather than an immutable sequence of numbers.

The arange() function takes similar arguments to range(). Here, we make a 1D array of the integers from 0 to 9:

In [27]: arr1d = np.arange(10)

In [28]: arr1d

Out[28]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

We can also add a start, stop, and step argument to create an array of even numbers between 0 and 10:

In [29]: arr1d_step = np.arange(0, 10, 2)

In [30]: arr1d_step

Out[30]: array([0, 2, 4, 6, 8])

Next, we start the sequence at 5 and stop at 9:

In [31]: arr1d_start_5 = np.arange(5, 10)

In [32]: arr1d_start_5

Out[32]: array([5, 6, 7, 8, 9])

Whereas range() always produces a sequence of integers, arange() lets you specify the data type of the numbers in the array. Here, we use double-precision floating-point numbers:

In [33]: arr1d_float = np.arange(10, dtype='float64')

In [34]: arr1d_float.dtype
Out[34]: dtype('float64')

Interestingly, arange() accepts a float for the step parameter:

In [35]: arr1d_float_step = np.arange(0, 3, 0.3)

In [36]: arr1d_float_step
Out[36]: array([0. , 0.3, 0.6, 0.9, 1.2, 1.5, 1.8, 2.1, 2.4, 2.7])

When arange() is used with floating-point arguments, it's usually not possible to predict the number of elements obtained, due to the finite floating-point precision. For this reason, it's better to use the NumPy linspace() function, which receives as an argument the number of elements desired instead of the step argument. We'll look at linspace() shortly.

With the arange() and reshape() functions, you can create a multidimensional array – and generate a lot of data – with a single line of code. The arange() function creates a 1D array, and reshape() divides this linear array into different parts as specified by a shape argument. Here's an example using the 3D shape tuple (2, 2, 4):

In [37]: arr3d = np.arange(16).reshape(2, 2, 4)

In [38]: print(arr3d)
[[[ 0  1  2  3]
  [ 4  5  6  7]]

 [[ 8  9 10 11]
  [12 13 14 15]]]

Because arrays need to be symmetrical, the product of the shape tuple must equal the size of the array. In this case, (8, 2, 1) and (4, 2, 2) will work, but (2, 3, 4) will raise an error because the resulting array has 24 elements, whereas you specified 16 (np.arange(16)):

In [39]: arr3d = np.arange(16).reshape(2, 3, 4)

ValueError Traceback (most recent call last)
 in 
----> 1 arr3d = np.arange(16).reshape(2, 3, 4)

ValueError: cannot reshape array of size 16 into shape (2,3,4)

Using the linspace() Function

The NumPy linspace() function creates an ndarray of evenly spaced numbers within a defined interval. It's the arange() function with a num (number of samples) argument rather than a step argument. The num argument determines how many elements will be in the array, and the function calculates the intervening numbers so that the intervals between them are the same.

Suppose that you want an array of size 6 with values between 0 and 20. All you need to do is pass the function a start, stop, and num value, as follows, using keyword arguments for clarity:

In [40]: np.linspace(start=0, stop=20, num=6)

Out[40]: array([ 0.,  4.,  8., 12., 16., 20.])

This produced a 1D array of six floating-point values, with all the values evenly spaced. Note that the stop value (20) is included in the array.

You can force the function to not include the endpoint by setting the Boolean parameter endpoint to False:

In [41]: np.linspace(0, 20, 6, endpoint=False)

Out[41]: 
array([ 0.        ,  3.33333333,  6.66666667, 10.        , 
       13.33333333, 16.66666667])

If you want to retrieve the size of the intervals between values, set the Boolean parameter retstep to True. This returns the step value:

In [42]: arr1d, step = np.linspace(0, 20, 6, retstep=True)

In [43]: step

Out[43]: 4.0

By default, the linspace() function returns a dtype of float64. You can override this by passing it a dtype argument:

In [44]: np.linspace(0, 20, 6, dtype='int64')

Out[44]: array([ 0,  4,  8, 12, 16, 20], dtype=int64)

Be careful when changing the data type. The result may no longer be a linear space due to rounding.

As with arange(), you can reshape the array on the fly. Here, we produce a 2D array with the same linspace() arguments:

In [45]: np.linspace(0, 20, 6).reshape(2, 3)

Out[45]: 
array([[ 0.,  4.,  8.],
       [12., 16., 20.]])

It's possible to create sequences with uneven spacing. The np.logspace() function, for example, creates a logarithmic space with numbers evenly spaced on a log scale.

The linspace() function lets you control the number of elements in an array, something that can be challenging to do when using arange(). Arrays of evenly spaced numbers are useful when working with mathematical functions of continuous variables. Linear spaces also come in handy when you need to sample an object – such as a waveform – evenly. To see some useful examples of linspace() in action, visit the Real Python website.

Creating Prefilled Arrays

For convenience, NumPy lets you create ndarrays using prefilled zeros, ones, random values, or values of your choosing. You can even create an empty array with no predefined values. These arrays are commonly used when you need a structure for holding computation results, for training machine learning applications, for creating image masks, for performing linear algebra, and so on.

To create a zero-filled array, simply pass a shape tuple to the zero() function, as follows:

In [46]: np.zeros((3, 3))

Out[46]: 
array([[0., 0., 0.],
       [0., 0., 0.],
       [0., 0., 0.]])

To create an array filled with ones, repeat the process with the ones() function:

In [47]: np.ones((3, 3))

Out[47]: 
array([[1., 1., 1.],
       [1., 1., 1.],
       [1., 1., 1.]])

The np.eye() function creates an array where all items are equal to zero, except for the _k_th diagonal, whose values are equal to one:

In [48]: np.eye(N=3, M=3, k=0)

Out[48]: 
array([[1., 0., 0.],
       [0., 1., 0.],
       [0., 0., 1.]])

By default, these functions return float64 values, but you can override this using a dtype argument, such as dtype=int.

To fill an array with a custom value and data type, use the full() function with the following syntax:

In [49]: np.full((3, 3), fill_value=5, dtype='int64')

Out[49]: 
array([[5, 5, 5],
       [5, 5, 5],
       [5, 5, 5]], dtype=int64)

The empty() function returns a new ndarray of a given shape filled with uninitialized arbitrary placeholder data of the given data type:

np.empty((2, 3, 2))

Out[50]: 
array([[[9.29913537e-312, 3.16202013e-322],
        [0.00000000e+000, 0.00000000e+000],
        [1.78022341e-306, 3.60032036e+179]],

       [[1.50399158e+161, 3.54295288e-033],
        [4.47072617e-062, 1.08464142e-042],
        [4.27721310e-033, 7.79859012e-043]]])

According to the documentation, empty() does not initialize the values of the array like other creation functions, such as zeros(), and may therefore be marginally faster. However, the values stored in the newly allocated array are arbitrary. For reproducible behavior, you must set each element of the array before reading.

Finally, you can generate arrays of pseudo-random numbers using NumPy. For floating-point values between 0 and 1, just pass random() a shape tuple:

In [51]: np.random.random((3, 3))

Out[51]: 
array([[0.63057625, 0.86598533, 0.21743918],
       [0.43408773, 0.79878953, 0.36728046],
       [0.32102443, 0.20350924, 0.58266535]])

In addition, you can generate random integers, sample values from a "standard normal" distribution, shuffle an existing array's contents in place, and more. We'll look at some of these options later in this series, and you can find the official documentation here.

If you're confused about the difference between empty() and random(), empty() does not initialize values in the array, whereas random() does. So, you should use empty() when you plan to fill the array with specific values later and use random() when you need an array of preset random values.

Accessing Array Attributes

As objects, ndarrays have attributes accessible through dot notation. We've looked at some of these already, and you can find more listed in the following table:

For example, to get the shape of the arr1d object, enter the following:

In [52]: arr1d = np.arange(0, 4)

In [53]: arr1d.shape

Out[53]: (4,)

As a 1D array, there's only one axis and thus only one index. Note the comma after the index, which tells Python that this is a tuple data type and not just an integer in parentheses.

The size of the array is the total number of elements it contains. This is the same as the product of the elements returned by shape. To get the array's size, enter the following:

In [54]: arr1d.size

Out[54]: 4

To get the array's dtype, enter:

In [55]: arr1d.dtype

Out[55]: dtype('int32')

Note that, even if you have a 64-bit machine, the default dtype for numbers may be 32-bit, such as int32 or float32. To ensure that you're using 64-bit numbers, you can specify the dtype when creating the array, as follows (for int64):

In [56]: test = np.array([5, 4, 9], dtype='int64')

In [57]: test.dtype

Out[57]: dtype('int64')

To get the array's strides, access the strides attribute with dot notation:

In [58]: arr1d.strides

Out[58]: (4,)

When using strings in arrays, the dtype needs to include the length of the longest string. NumPy can generally figure this out on its own, as follows:

In [59]: arr1d_str = np.array(['wheat', 'soybeans', 'corn'])

In [60]: arr1d_str.dtype

Out[60]: dtype('

In [61]: arr1d_str.dtype.name

Out[61]: 'str256'

In [62]: arr1d_str.itemsize

Out[62]: 32

In [1]: import numpy as np 

In [2]: np.zeros((10, 10))