Let’s first consider a situation where you need to predict something based on the previous data. For example, the rent of house based on number of rooms, area etc. If some new data is provided, the model should be able to predict the rent for the given input data.
Let’s say the independent variables be X representing all of the independent data (number of rooms). Let the dependent variable be Y representing all of the dependent data.
![\[Y = X W\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-a4463efd2fb5c81730a025de4fbd03d7_l3.png "Rendered by QuickLaTeX.com")
W represents the coefficients (also called as weights) which needs to be learned or determined.
Analytical Method
If X is a full-rank matrix, you could just take inverse to find out the weights.
![\[W = X^{-1}Y\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-44d0f43a5625097772a870a4cf1d7d13_l3.png "Rendered by QuickLaTeX.com")
If 
doesn’t exist then use projection theorem (pseudo inverse).
![\[W = (X^{T}X)^{-1}X^{T}Y\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-f4fa08f5570d855e4bf32a012ff28aef_l3.png "Rendered by QuickLaTeX.com")
![\[W = X \setminus Y\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-f05e892dc457ad845380b4f3658bf432_l3.png "Rendered by QuickLaTeX.com")
Iterative Method
Let 
be the predicted output based on W. Let the loss/error function be defined by
![\[L = \frac{1}{2} \sum_{i=0}^{i=n-1} (Y_i - Y_i^{'})^2\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-ec0862fabe76120e2963a3d43e6b500c_l3.png "Rendered by QuickLaTeX.com")
![\[L = \frac{1}{2} (Y - Y^{'})^{T} (Y - Y')\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-24f924f8d0e66d34e2549762d78501f2_l3.png "Rendered by QuickLaTeX.com")
![\[L = \frac{1}{2} \sum_{i=0}^{i=n-1} (Y_{i} - W^TX_{i})^2\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-3754797f0872e14bef87b5ac182dcaf7_l3.png "Rendered by QuickLaTeX.com")
![\[\frac{dL}{dW} = \frac{1}{2} 2 (Y_{i} - Y_{i}^{'}) (- X_{i})\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-7d0977c305028427cf642062cec3eab1_l3.png "Rendered by QuickLaTeX.com")
![\[\frac{dL}{dW} = - X_{i}(Y_i - Y_{i}^{'})\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-4eaff0612419387a71a13b77bd974c98_l3.png "Rendered by QuickLaTeX.com")
The objective is to minimize the loss function. Because the L function is convex, the minima is given by 
.
Iteratively, if you change W in the opposite direction of the derivative 
, the weight converges to the pseudo inverse value.
![\[W = W - \alpha \frac{dL}{dW}\]](http://www.ssravisutha.com/wp-content/ql-cache/quicklatex.com-401b76907da5e1a8128294b8fcd403fd_l3.png "Rendered by QuickLaTeX.com")
There are multiple update strategies – online/stochastic update, batch update and mini-batch update.