Securing your Containerised Models and Workloads

Author:Murphy | View: 29546 | Time: 2025-03-23 12:20:40

Containerisation is now the de facto means of deploying many applications, with Docker being the forefront software driving its adoption. With its popularity also comes the increased risk of attacks [1]. Hence it will serve us well to secure our docker applications. The most fundamental means of doing this is to ensure that we set the user within our containers as a non-root user.

CONTENTS
========

Why use non-root?

What you can & cannot do as a default non-root user

The Four Scenarios
  1) Serve a model from host (Read Only)
  2) Run data processing pipelines (Write within Container)
  3) Libraries automatically writing files (Write within Container)
  4) Save trained models (Write to Host)

Summary

Why use Non-Root?

Or rather, why not use the root user? Let's take an example of a dummy architecture like the one below.

Security is often viewed in a multi-layered approach. If an attacker manages to enter a container, the permissions it has as a user will be the first layer of defence. If the container user is assigned to have root access, the attacker can have free control of everything within the container. With such broad access, it can also exploit any potential vulnerabilities present and using that, potentially escape out to the host, and gain full access access to all connecting systems. The consequences are severe, including the following:

retrieve the secrets stored
intercept and disrupt your traffic
run malicious services like crypto-mining
gain access to any connecting sensitive services like databases

Damn, that sounds really scary! Well, the solution is simple, change your containers to a non-root user!

Before we even go to the rest of the article, if you do not have a good grasp of Linux permissions and access rights, do take a look at my previous article [2].

What You Can & Cannot Do as a Default Non-Root User

Let's attempt to create a simple Docker application with a default non-root user. Use the Dockerfile below.

# Dockerfile
FROM python:3.11-slim

WORKDIR /app

# create a dummy py file
RUN echo "print('I can run an existing py file')" > example.py

# create & switch to non-root user
RUN adduser --no-create-home nonroot
USER nonroot

Build the image and create a container with it.

docker build -t test .
docker run -it test bash

Now that you are inside the container, let's try a few commands. So what are the things that you cannot do? You can see that all kinds of writing and installation permissions are not allowed.

On the opposite spectrum, we can run all kinds of read permissions.

Because we have python installed, it is a little unique. If we ls -l $(which python) we can see that the python interpreter has full permissions. Thus, it can execute existing python files like the example.py file we created initially in the Dockerfile. We can even enter into the python console and run simple commands. However, as other system write permissions have been removed when we switch to the non-root user, you can see that we cannot create and modify the scripts, or use python to run write commands.

While system-wide restrictions are good for security, there will be many instances whereby write permissions for specific files and directories are required, and we need to cater for such allowances.

In the following sections, I will give examples of four scenarios in a Machine Learning operations lifecycle. With these examples, one should be able to gain an understanding of how to implement for most other instances.

The Four Scenarios

1) Serve a Model from Host – Read Only

When serving a model, it involves an inference and serving script to load the model and expose it via an API (e.g., Flask, FastAPI) to accept inputs. The model is sometimes loaded from the host machine, and separated from the image so that the image size is optimally small, and any reload of the image will be optimally quick without repeated model downloads. The model is then passed into the container via a bind-mount **** volume, to be loaded and served.

This is probably the least cumbersome way to implement a non-root user because only read permission is required, which by default is granted to all users. Below is a sample Dockerfile of how that is done.

# Dockerfile
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip3 install --no-cache-dir --upgrade pip~=23.2.1 
    && pip3 install --no-cache-dir -r requirements.txt

COPY ./project/ /app

# add non-root user ---------------------
RUN adduser --no-create-home nonroot
# switch from root to non-root user -----
USER nonroot

CMD ["python", "inference.py"]

It has two simple commands to first create a new system user called nonroot. Second, it is then switched from the root to the nonrootuser just before the last CMD line. This is important as a default non-root user does not have any write and execute permissions, so it cannot install, copy or manipulate files that are required as seen in the earlier steps.

Now that we know how to assign a non-root user in Docker, let's go to the next step.

2) Run Data Processing Pipelines – Write within Container

Sometimes, we just want to store temporary files to execute some jobs, let's say, some data preprocessing work. This consists of adding and deleting files. We can do such tasks within the container since the files are not persistent.

However, we will need write permissions if we are using a non-root user. To do that, we will need to use the command chown (change owner) and assign ownership to the nonroot user for the specific folder where write access is required. With that done, we can then switch the user to nonroot .

# Dockerfile

# ....

# add non-root user & grant ownership to processing folder
RUN adduser --no-create-home nonroot && 
    mkdir processing && 
    chown nonroot processing

# switch from root to non-root user
USER nonroot

CMD ["python", "preprocess.py"]

3) Libraries Automatically Writing Files – Write within Container

The previous example shows how to write files which we created ourselves. However, it is also common for the libraries you use to create files and directories automatically. You will only know they are created when you try running the container and it is denied permission to write.

I will show you two such examples, one from supervisor, which is used to manage multiple processes, and another from huggingface-hub, for downloading models from huggingface. Permission errors like these will be seen if we switch to a non-root user.

For the two supervisor files, we can create them as empty files first, and assign ownership rights to them. For the huggingface-hub download issue, it has already been hinted in the error log that we can change the download directory via the TRANSFORMERS_CACHE variable, hence we can first assign the directory variable, create the directory, and then assign ownership.

# Dockerfile

# ....

# add non-root user ................
# change huggingface dl dir
ENV TRANSFORMERS_CACHE=/app/model

RUN adduser --no-create-home nonroot && 
    # create supervisor files & huggingfacehub dir
    touch /app/supervisord.log /app/supervisord.pid && 
    mkdir $TRANSFORMERS_CACHE && 
    # grant supervisor & huggingfacehub write access
    chown nonroot /app/supervisord.log && 
    chown nonroot /app/supervisord.pid && 
    chown nonroot $TRANSFORMERS_CACHE
USER nonroot

CMD ["supervisord", "-c", "conf/supervisord.conf"]

Of course, there will be other examples that may slightly differ from what I show here, but the concept of allowing the least permissions to write will be the same.

4) Save Trained Models – Write to Host

Let's say we are using a container to train a model, and we want that model to be written to the host, e.g., to be picked up by another task for benchmarking or deployment. For this instance, we will need to write the model file out by linking a container directory to the host directory, also known as a bind mount.

First, we need to create a group and user for nonroot, specifying a unique ID for each, where for this case, we use 1001 (any number from 1000 is fine). Then, a model directory to store the model is created.

A difference here compared to Scenario 2 is that chown is not required for the model directory to write. Why?

# Dockerfile

# ....
# add non-root group/user & create model folder
ENV UID=1001
RUN addgroup --gid $UID nonroot && 
    adduser --uid $UID --gid $UID --no-create-home nonroot && 
    mkdir model

# switch from root to non-root user
USER nonroot

CMD ["python", "train.py"]

This is because the permission of the bind-mounted directory is determined by the host directory. Hence, we need to again create the same user in the host, ensuring that the user id is the same. The model directory is then created in the host and the nonroot user is granted the owner permissions.

# in host terminal

# add the same user & group
addgroup --gid 1001
adduser --uid 1001 --gid 1001 --no-create-home nonroot
# create model dir to bind-mount & make nonroot an owner
mkdir /home/model
chown nonroot /home/model

Bind mount is usually specified in the docker-compose.yml file or docker run command to enable more flexibility. Below is an example of the former.

version: "3.5"

services:
    modeltraining:
        container_name: modeltraining
        build:
            dockerfile: Dockerfile
        volumes:
            - type: bind
              source: /home/model # host dir
              target: /app/model  # container dir

And for the latter:

docker run -d --name modeltraining -v /home/model:/app/model

Run either of them, and you will see that your non-root user can execute the script without any issues.

Summary

We have seen how we can assign non-root user and still make the containers work with their desired tasks. This is mainly relevant when specific write permissions are required. We just need to know two fundamental concepts.

For writing permissions in the container, chown in the Dockerfile
For writing permissions for a bind-mount, create the same non-root user in the host and chown in the host directory

If you need to enter into the docker container and run some tests as a root user, we can use the following command.