What's Logistic Regression?

Table of Contents

Previously, I talked about how linear regression worked and implemented it in Python from scratch. This post is about understanding and implementing logistic regression from scratch. I will talk about what logistic regression is, how it works, and why we use it.

Intro

Unfortunately or fortunately, the field of ML cannot be discussed without understanding the underlying mathematics on which it is built.

Similar to linear regression, we are approximating some parameters and then using them to make predictions. However, unlike linear regression, we are not predicting a particular value but rather trying to classify an input into a category.

Therefore, the name is actually misleading. It is a classification algorithm, more specifically, a binary classification algorithm. The output is not a continuous value; rather, it is a binary output such as 0/1 or yes/no.

Math

Idea

The core idea is to take a linear combination

$$ z=wx+b $$

and pass it through

$$ g(z)=\frac{1}{1+e^{-z}}, $$

which is known as the sigmoid or logistic function.

The sigmoid function takes the real-valued continuous output $z$ and compresses it to a value in $(0,1)$. This value can be interpreted as a probability:

$$ g(z)=P(y=1\mid x;w,b) $$
Note

It is the probability that the class is $y=1$ when the input feature is $x$, given the parameters $w$ and $b$.

We can also have more than one feature. For example, if we have two input features,

$$ z=w_1x_1+w_2x_2+b. $$

Therefore,

$$ g(z)=\frac{1}{1+e^{-(w_1x_1+w_2x_2+b)}}. $$

This can be generalised to $n$ features using vectors:

$$ \begin{aligned} z&=\vec{w}\cdot\vec{x}+b,\\ g(z)&=\frac{1}{1+e^{-(\vec{w}\cdot\vec{x}+b)}}. \end{aligned} $$

Making Predictions

Now that we have the output of $g(z)$ from the sigmoid function, where $0<g(z)<1$, we can apply a threshold to make predictions:

$$ \begin{aligned} g(z)\geq0.5&\implies\hat{y}=1,\\ g(z)<0.5&\implies\hat{y}=0. \end{aligned} $$

Here, we use $0.5$ as the threshold.

Note

Decision Boundary

$$ >\vec{w}\cdot\vec{x}+b=0\implies g(z)=0.5. >$$

This is called the decision boundary. It is the line where the model is almost neutral between $y=0$ and $y=1$.

It splits the input space into two regions. Points on one side are classified as $0$, while points on the other side are classified as $1$.

Cost Function

The cost function measures how well the parameters fit the training data. A reasonable idea might be to use the same cost function as linear regression, namely MSE:

$$ C(w,b)=\frac{1}{m}\sum_{i=1}^{m}\left(g(z_i)-y_i\right)^2. $$

Although MSE can technically be used with binary labels, combining it with the sigmoid function produces an objective that is generally non-convex in the model parameters. Consequently, gradient descent may encounter local minima or flat regions.

A new cost function is needed, or more specifically, a new loss function $L(g(z_i),y_i)$ is needed, such that the resulting cost function $C(w,b)$ is convex.

Loss Function

The loss function should satisfy the following requirements:

  1. Penalise predictions that are far from the true label.
  2. Reward predictions that are close to the true label.

Mathematically, we use the following piecewise function:

$$ L(g(z_i),y_i)= \begin{cases} -\log(g(z_i)) & \text{if } y_i=1,\\ -\log(1-g(z_i)) & \text{if } y_i=0. \end{cases} $$

These are the graphs of the two functions:

Binary cross-entropy loss

If $y_i=1$ and the model predicts $0.1$, then

$$ -\log(0.1)\approx 1, $$

which is a high loss. If $y_i=0$ and the model predicts $0.95$, then

$$ -\log(1-0.95)=\log(20), $$

which is also a high loss.

This can be rewritten as:

$$ L(g(z_i),y_i)=-y_i\log(g(z_i))-(1-y_i)\log(1-g(z_i)). $$

Notice that this is the same as the piecewise function. When $y_i=0$, the first term disappears, and when $y_i=1$, the second term disappears.

Now that we have the cost function, we can write it as follows:

$$ C(w,b)=\frac{1}{m}\sum_{i=1}^{m} \left[-y_i\log(g(z_i))-(1-y_i)\log(1-g(z_i))\right]. $$

Minimizing the Cost Function

The process of minimizing this cost function is the same as in linear regression: we apply gradient descent. At each iteration,

$$ \begin{aligned} w&=w-\alpha\frac{\partial C(w,b)}{\partial w},\\ b&=b-\alpha\frac{\partial C(w,b)}{\partial b}. \end{aligned} $$

Before putting this into code, let us simplify the derivatives of $C(w,b)$ with respect to $w$ and $b$:

$$ \begin{aligned} z_i&=\vec{w}\cdot\vec{x_i}+b,\\ g(z_i)&=\frac{1}{1+e^{-(\vec{w}\cdot\vec{x_i}+b)}},\\ L(g(z_i),y_i)&=-y_i\log(g(z_i))-(1-y_i)\log(1-g(z_i)),\\ C(w,b)&=\frac{1}{m}\sum_{i=1}^{m}L(g(z_i),y_i). \end{aligned} $$

Differentiating the loss function,

$$ \begin{aligned} \frac{\partial L}{\partial w_j} &=\frac{\partial L}{\partial g_i} \cdot\frac{\partial g_i}{\partial w_j}\\ &=\frac{\partial L}{\partial g_i} \cdot\frac{\partial g_i}{\partial z_i} \cdot\frac{\partial z_i}{\partial w_j}\\ &=\left( \frac{g_i-y_i}{\cancel{g_i(1-g_i)}} \right) \cdot\cancel{g_i(1-g_i)} \cdot x_{i,j}. \end{aligned} $$

Therefore,

$$ \frac{\partial C}{\partial w_j} =\frac{1}{m}\sum_{i=1}^{m}(g_i-y_i)x_{i,j}. $$

Similarly, for $b$,

$$ \frac{\partial C}{\partial b} =\frac{1}{m}\sum_{i=1}^{m}(g_i-y_i). $$

We do not have to worry about indexing across the input features because there is one weight for each feature, along with one bias term.

Cooking the Code

Dataset

Our dataset has five fields, but we will only use Age and EstimatedSalary to predict Purchased. However, before using the data, we first standardise the continuous features so that they have comparable scales.

Load Data

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

df = pd.read_csv("User_Data.csv")

print(df.head())
print(df.columns)

Extract Fields

X = df[["Age", "EstimatedSalary"]].to_numpy(dtype=np.float64)
y = df["Purchased"].to_numpy(dtype=np.float64)

Split into Training and Test Sets

rng = np.random.default_rng(42)

indices = np.arange(len(X))
rng.shuffle(indices)

test_size = int(0.2 * len(X))

test_indices = indices[:test_size]
train_indices = indices[test_size:]

X_train = X[train_indices]
X_test = X[test_indices]

y_train = y[train_indices]
y_test = y[test_indices]

Calculate the Mean and Standard Deviation

mean = X_train.mean(axis=0)
std = X_train.std(axis=0)

if np.any(std == 0):
    raise ValueError("At least one feature has zero variance.")

X_train = (X_train - mean) / std
X_test = (X_test - mean) / std

Visualisation with a Scatter Plot

plt.scatter(
    X_train[y_train == 0, 0],
    X_train[y_train == 0, 1],
    color="tab:red",
    label="Class 0",
    s=20
)

plt.scatter(
    X_train[y_train == 1, 0],
    X_train[y_train == 1, 1],
    color="tab:green",
    label="Class 1",
    s=20
)

plt.xlabel("Standardised Age")
plt.ylabel("Standardised Estimated Salary")
plt.legend()
plt.show()

Sigmoid Function

def sigmoid(z):
    if z >= 0:
        return 1 / (1 + np.exp(-z))
    else:
        exp_z = np.exp(z)
        return exp_z / (1 + exp_z)

Cost Function

def cost_func(X, y, w, b):
    m = X.shape[0]
    cost_sum = 0.0
    epsilon = 1e-15

    for i in range(m):
        z = np.dot(w, X[i]) + b
        g = sigmoid(z)

        g = np.clip(g, epsilon, 1 - epsilon)

        cost_sum += (
            -y[i] * np.log(g)
            -(1 - y[i]) * np.log(1 - g)
        )

    return cost_sum / m

Gradient Function

def gradient_func(X, y, w, b):
    m, n = X.shape

    grad_w = np.zeros(n)
    grad_b = 0.0

    for i in range(m):
        z = np.dot(w, X[i]) + b
        g = sigmoid(z)
        error = g - y[i]

        grad_b += error

        for j in range(n):
            grad_w[j] += error * X[i, j]

    grad_w /= m
    grad_b /= m

    return grad_b, grad_w

Gradient Descent Function

def grad_des(X, y, alpha, iterations):
    _, n = X.shape

    w = np.zeros(n)
    b = 0.0

    for i in range(iterations):
        grad_b, grad_w = gradient_func(X, y, w, b)

        w -= alpha * grad_w
        b -= alpha * grad_b

        if i % 1000 == 0:
            print(f"{i}: {cost_func(X, y, w, b)}")

    return w, b

Prediction Function

def predict(X, w, b):
    m = X.shape[0]
    predictions = np.zeros(m)

    for i in range(m):
        z = np.dot(w, X[i]) + b
        g = sigmoid(z)

        predictions[i] = 1 if g >= 0.5 else 0

    return predictions

Training and Testing

learning_rate = 0.001
iterations = 10000

final_w, final_b = grad_des(
    X_train,
    y_train,
    learning_rate,
    iterations
)

predictions_train = predict(X_train, final_w, final_b)
predictions_test = predict(X_test, final_w, final_b)

print("Prediction shapes:")
print(predictions_train.shape, y_train.shape)
print(predictions_test.shape, y_test.shape)

accuracy_train = np.mean(predictions_train == y_train) * 100
accuracy_test = np.mean(predictions_test == y_test) * 100

print(f"Training accuracy: {accuracy_train:.2f}%")
print(f"Test accuracy: {accuracy_test:.2f}%")

Plotting descion boundary

x_values = np.linspace(
    X_train[:, 0].min(),
    X_train[:, 0].max(),
    100
)

y_values = -(final_w[0] * x_values + final_b) / final_w[1]

plt.scatter(
    X_train[y_train == 0, 0],
    X_train[y_train == 0, 1],
    color="tab:red",
    label="Class 0"
)

plt.scatter(
    X_train[y_train == 1, 0],
    X_train[y_train == 1, 1],
    color="tab:green",
    label="Class 1"
)

plt.plot(
    x_values,
    y_values,
    color="black",
    label="Decision boundary"
)

plt.xlabel("Standardised Age")
plt.ylabel("Standardised Estimated Salary")
plt.legend()
plt.show()

This is a complete basic implementation of binary logistic regression using NumPy. The model learns a linear decision boundary and uses the sigmoid function to convert the linear score into a probability. In practical applications, we would also consider validation data, regularisation, class imbalance, numerical stability, and metrics beyond accuracy.