Probabilistic Logistic Regression with TensorFlow

Author:Murphy | View: 20469 | Time: 2025-03-23 20:02:00

Introduction

This article belongs to the series "Probabilistic Deep Learning". This weekly series covers probabilistic approaches to deep learning. The main goal is to extend deep learning models to quantify uncertainty, i.e., know what they do not know.

In this article, we will introduce the concept of probabilistic logistic regression, a powerful technique that allows for the inclusion of uncertainty in the prediction process. We will explore how this approach can lead to more robust and accurate predictions, especially in cases where the data is noisy, or the model is overfitting. Additionally, by incorporating a prior distribution on the model parameters, we can regularize the model and prevent overfitting. This approach serves as a great first step into the exciting world of Bayesian Deep Learning.

Articles published so far:

Figure 1: The motto for today: lines can separate more things than we give them credit for (source)

As usual, the code is available on my GitHub.

Preliminary Work

In our previous article in this series we built the Naive Bayes algorithm from scratch and used it to classify wine samples based on selected characteristics. This time, we will be using a probabilistic logistic regression approach. Since we already followed the end-to-end approach, I will skip most of the Exploratory Data Analysis section and the class prior distribution definition.

The only thing to note is that there is a difference in the features that we selected for this model.

Figure 2: Target samples distribution by alcohol and hue.

We will use hue and flavanoids as our independent variables. Notice how these features are more effective in separating the target variable than alcohol and hue.

Figure 3: Target samples distribution by flavanoids and hue.

Probabilistic Logistic Regression

Logistic regression is a widely-used statistical method for binary classification, which is used to model the relationship between a binary response variable and one or more predictors. Logistic regression can be used to model the probability of a binary outcome as a function of the predictor variables. The traditional logistic regression model is a deterministic model, which assumes that the relationship between the predictor variables and the response variable is fixed and known. However, in many real-world applications, the true relationship between the predictors and the response is uncertain, and it is more appropriate to use a probabilistic approach.

Probabilistic logistic regression models the relationship between the predictor variables and the binary response variable using a probabilistic framework, and is able to account for uncertainty in the data and the model parameters. This is achieved by placing probability distributions over the model parameters, rather than assuming fixed values for them. In this way, probabilistic logistic regression models can provide more accurate predictions and better uncertainty quantification compared to traditional logistic regression models.

One of the most popular probabilistic models for logistic regression is the Bayesian logistic regression model. These models are based on Bayes' theorem, which states that the posterior probability of a model parameter given the data is proportional to the product of the likelihood of the data given the parameter and the prior probability of the parameter. Often, Bayesian logistic regression models use a conjugate prior distribution for the model parameters, which allows for closed-form solutions for the posterior distributions. This enables the computation of the probability of the response variable given the predictor variables, which is known as the posterior predictive distribution.

Likelihood

In this section, we present a method for computing the class-conditional densities in the probabilistic approach to logistic regression. Our method is based on the maximum likelihood estimate for the means, which is given by