Skip to main content

Logistic Regression Derivatives (Simple Explanation)

This page explains how to compute derivatives in logistic regression using a computation graph, broken down into easy-to-understand steps.

🧬 Goal​

We want to find out how to adjust the weights and bias in a logistic regression model so it gets better at making predictions. We use something called derivatives (slopes) to guide the learning process.

🔄 Forward Pass (Prediction)​

1. Compute the score z:​

z = w_1 x_1 + w_2 x_2 + b

This is a weighted sum of the inputs.

2. Apply the sigmoid function:​

a = \sigma(z) = \frac{1}{1 + e^{-z}}

This turns the score into a probability a (e.g., how likely is this email to be spam?).

3. Compute the loss:​

\mathcal{L}(a, y)

This tells us how wrong our prediction a is, compared to the true label y.

🔄 Backward Pass (Learning)​

We now work backwards to calculate how much each part contributed to the error.

Step 1: From Loss to Sigmoid Output​

\frac{\partial \mathcal{L}}{\partial a} = -\frac{y}{a} + \frac{1 - y}{1 - a}

This tells us how much the loss changes when a changes.

Step 2: From Sigmoid Output to z​

\frac{\partial a}{\partial z} = a(1 - a)

This is the derivative of the sigmoid function. So:

\frac{\partial \mathcal{L}}{\partial z} = \frac{\partial \mathcal{L}}{\partial a} \cdot \frac{\partial a}{\partial z}

Step 3: From z to Weights and Bias​

Since z = w_1 x_1 + w_2 x_2 + b, we have:

\frac{\partial z}{\partial w_1} = x_1\qquad
\frac{\partial z}{\partial w_2} = x_2\qquad
\frac{\partial z}{\partial b} = 1

So:

\frac{\partial \mathcal{L}}{\partial w_1} = \frac{\partial \mathcal{L}}{\partial z} \cdot x_1
\frac{\partial \mathcal{L}}{\partial w_2} = \frac{\partial \mathcal{L}}{\partial z} \cdot x_2
\frac{\partial \mathcal{L}}{\partial b} = \frac{\partial \mathcal{L}}{\partial z}

🎓 Gradient Descent Update​

We update our weights and bias using these gradients:

w_1 := w_1 - \alpha \cdot \frac{\partial \mathcal{L}}{\partial w_1}
w_2 := w_2 - \alpha \cdot \frac{\partial \mathcal{L}}{\partial w_2}
b := b - \alpha \cdot \frac{\partial \mathcal{L}}{\partial b}

Where \alpha is the learning rate (controls the step size).

✅ Summary​

  • Forward pass: Make a prediction using weights and sigmoid.
  • Loss: Measure how wrong the prediction was.
  • Backward pass: Compute how much each weight and bias contributed to the error.
  • Update: Adjust weights and bias to reduce future errors.

This is the heart of training a logistic regression model using derivatives and gradient descent.