Logistic Regression

Understanding Logistic Regression

Despite its name, logistic regression is actually a classification algorithm, not a regression algorithm. It's used to predict binary outcomes (yes/no, spam/not spam, pass/fail) or multi-class categories.

Logistic regression uses the logistic (sigmoid) function to transform linear combinations of features into probabilities between 0 and 1. These probabilities are then converted into class predictions.

How Logistic Regression Works

Unlike linear regression which predicts continuous values, logistic regression:

Outputs probabilities: Returns values between 0 and 1
Uses sigmoid function: Transforms linear combinations into probabilities
Makes binary decisions: Typically uses 0.5 as the threshold
Works for classification: Predicts categories, not numbers

The sigmoid function ensures that no matter what input you give, the output is always between 0 and 1, making it perfect for probability estimation.

Binary Classification Example

Let's see logistic regression in action with a simple binary classification problem:

binary_classification.py
# Binary classification with logistic regression
from sklearn.linear_model import LogisticRegression
import numpy as np

# Training data: [hours_studied] -> passed (1) or failed (0)
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]])
y = np.array([0, 0, 0, 0, 1, 1, 1, 1])  # 0 = failed, 1 = passed

# Create and train the model
model = LogisticRegression()
model.fit(X, y)

print("Training data:")
for hours, result in zip(X, y):
    status = "Passed" if result == 1 else "Failed"
    print(f"  {hours[0]} hours: {status}")

# Make predictions
new_hours = np.array([[4.5], [6.5]])
predictions = model.predict(new_hours)
probabilities = model.predict_proba(new_hours)

print("\nPredictions:")
for hours, pred, prob in zip(new_hours, predictions, probabilities):
    status = "Pass" if pred == 1 else "Fail"
    print(f"  {hours[0]} hours: {status} (probability: {prob[1]:.2f})")

Understanding Probabilities

Logistic regression outputs probabilities, not just predictions. This gives you more information:

probabilities.py
# Understanding probability outputs
from sklearn.linear_model import LogisticRegression
import numpy as np

X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([0, 0, 0, 1, 1, 1])

model = LogisticRegression()
model.fit(X, y)

# Get probabilities for predictions
test_X = np.array([[2.5], [4.5], [5.5]])
probabilities = model.predict_proba(test_X)
predictions = model.predict(test_X)

print("Predictions with probabilities:")
for x, pred, prob in zip(test_X, predictions, probabilities):
    print(f"x={x[0]}: Class {pred}, Prob(Class 0)={prob[0]:.3f}, Prob(Class 1)={prob[1]:.3f}")

print("\nNote: Probabilities sum to 1.0")
print("Default threshold is 0.5 - if Prob(Class 1) > 0.5, predict Class 1")

Multi-class Classification

Logistic regression can also handle multiple classes (not just binary):

multiclass.py
# Multi-class classification
from sklearn.linear_model import LogisticRegression
import numpy as np

# Three classes: 0 = Low, 1 = Medium, 2 = High
X = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9]])
y = np.array([0, 0, 0, 1, 1, 1, 2, 2, 2])

# Train model (automatically handles multi-class)
model = LogisticRegression(multi_class='multinomial')
model.fit(X, y)

print("Multi-class predictions:")
test_X = np.array([[2.5], [4.5], [7.5]])
predictions = model.predict(test_X)
probabilities = model.predict_proba(test_X)

for x, pred, prob in zip(test_X, predictions, probabilities):
    classes = ["Low", "Medium", "High"]
    print(f"x={x[0]}: {classes[pred]} (probs: {prob})")

Real-World Applications

Logistic regression is widely used for:

Email Spam Detection: Classify emails as spam or not spam
Medical Diagnosis: Predict disease presence based on symptoms
Credit Approval: Decide whether to approve a loan
Image Classification: Classify images into categories
Sentiment Analysis: Classify text as positive or negative

Advantages and Limitations

Logistic regression has several advantages:

Interpretable: Easy to understand and explain
Fast: Quick to train and make predictions
Probabilistic: Provides probability estimates, not just predictions
No feature scaling needed: Works well without normalization

However, it has limitations:

Assumes linear relationships: May not capture complex patterns
Requires feature engineering: May need polynomial features for non-linear data
Can be sensitive to outliers: Extreme values can affect the model

💡 When to Use Logistic Regression

Use logistic regression when you need interpretability, have linearly separable classes, or want probability estimates. For complex non-linear relationships, consider decision trees or neural networks.

🎉

Lesson Complete!

Great work! Continue to the next lesson.