L20

10/10/26 Sat 10:36am

Logistic Regression

It is a supervised learning algorithm and is used for Classification problems. Whereas Dependant variable is categorical and binary.

$$P=\frac{1}{1+e^{-z}}$$

where

$$z=wx+b$$

Gradient Decent Algorithm for Regression

Let $\hat{y}$ be the predicted output from the model, and $y$ be the actual output.
Error $e$ is given by

$$e=\hat{y}-y$$

The Objective function $J$ is given by

$$J=\frac{1}{2}{e}^2$$

The gradient Descents are

$$\frac{\delta J}{\delta w},\frac{\delta J}{\delta b}$$

and the weights

$$w_{\text{new}}=w_{\text{old}}-\eta \frac{\delta J}{\delta w}$$ $$b_{\text{new}}=b_{old}-\eta \frac{\delta J}{\delta b}$$

where $\eta$ is the learning rate

$$\eta =0.1(\text{usually})$$

now

$$\frac{\delta J}{\delta w}=\frac{\delta J}{\delta e}\times \frac{\delta e}{\delta w}=e\times x$$ $$\boxed{⟹\frac{\delta J}{\delta w}=ex}$$

similarly

$$\frac{\delta J}{\delta b}=\frac{\delta J}{\delta e}\times \frac{\delta e}{\delta b}=e\times 1$$ $$\boxed{⟹\frac{\delta J}{\delta b}=e}$$

Gradient Decent for Classification

The objective function $J$ is given by

$$J=-\left[y\ln P+(1-y)\ln (1-P)\right]$$ $$\frac{\delta J}{\delta w}\&\frac{\delta J}{\delta b}$$ $$\frac{\delta J}{\delta w}=\frac{\delta J}{\delta P}\times \frac{\delta P}{\delta z}\times \frac{\delta z}{\delta w}$$ $$\frac{\delta J}{\delta b}=\frac{\delta J}{\delta P}.\frac{\delta P}{\delta z}.\frac{\delta z}{\delta b}$$

now

$$\frac{\delta J}{\delta P}=-\left[\frac{y}{P}-\frac{1-y}{1-P}\right]=\frac{P-y}{P(1-P)}$$ $$\frac{\delta P}{\delta z}=\frac{e^{-z}}{(1+e^{-z})}$$

Q&A

Question

At 80 percent loading the measured temperature rises 30 $\text{celsius}$ for a motor. Initially $w=0.5$ and $b=0.1$ with a learning rate ($\eta$) of 0.1 perform one update, calculate the new temperature rise and estimate the transformer temperature at 25 $\text{celsius}$ ambient.

$$x=\frac{\text{loading percentage}}{100}$$ $$y=\frac{\text{Temp Rise}}{50}$$ $$\hat{y}=wx+b$$

Solution:

$$x=\frac{80}{100}=0.8$$ $$y=\frac{30}{50}=0.6$$

x	y	$\hat{y}=wx+b=0.5x+b$	e=$\hat{y}-y$
0.8	0.6	0.5	-0.1
now using

$$\frac{\delta J}{\delta w}=ex⟹\frac{\delta J}{\delta w}=-0.1\times 0.8=-0.08$$

and

An industrial pump uses logsictal regression to find out normal ( y = 0 ) or faulty (y = 1) based vibration rms. the vibration measurement is given by the following formula

$$x-\frac{v-\mu }{s}$$

where the training data $\mu =2$mm/s $s=1$mm/s the model estimates fault probability as

$$P=\frac{1}{1-e^{wx+b}}$$

and the current parameters w= 0.4 and $b=-0.6$

A pump confirmed to be faulty with a vibration of 4 mm/s $\therefore$ this training sample has target variable y = 1. Using binary cross entropy loss and a learning rate $\eta$ = 0.1 calculate the following

1. the standardized input x
2. Fault probabilty and loss before updating the model
3. Gradient and perform one gradient desecnt update of w and b
4. Using the updated parameters recacluate the fault probability and loss for the sample
5. Classify the sample using the rule fault if $P\ge 0.5$ otherwise normal