Terminologies in Regression:

Before learning regression concepts we recommend you to go through the requisite mathematics such as vectors & Linear Algebra etc. Follow this link to check the necessary mathematical pre-requisites.

Simple Linear Regression:

Ideally the regression equation should look like this:

In reality we get an equation that has some error associated with it. Such as:

Predicted/Fitted Value:

Residual: 

is the error term in the regression equation & is calculated as 

OLS( Ordinary Least Squares Regression ):

The residual sum of squares is: 

The method of minimizing the sum of squared residuals is termed as Ordinary Least Squared Regression.

Multiple linear Regression:

When there are multiple predictors, the equation is simply extended to accommodate

them:

RMSE & Residual Standard Error.

R-Squared: 

This value represents proportion of variation explained by the model & takes value between 0 & 1. Formula is:

It gives the "Goodness of fit" of a model. The ideal fit should be 1 & the closer to 1 value we get, the better.

t-statistic & p-value:

t-statistic has an inverse relation with p-value. p-value measures the extent to which a coefficient is significant. So the higher the t-statistic is & hence lower the p-value is, the more significant the predictor.  

link p value concept here............. 

Nonlinear Regression:

Its when the relation between the response(predicted value) & the predictor value is not linear. We can introduce a higher order polynomial term in the regression equation to make it nonlinear. 

Sometimes introducing a higher order term into the regression makes it more wiggly. In that case we specify knots & introduce something called a "Spline". It will then be called a spline regression. 

Sometimes that also becomes challenging as we may not know where to introduce the knot. In such conditions we may need to use GAM (Generalized Additive Models). This helps smoothen the curve & captures the relations perfectly. The mathematics behind this is beyond the scope of this article.