Revision sheet: Mastering Multiple Regression and ANOVA

📋 Course Outline

1. Multiple regression model
2. Regression fit and residuals
3. Regression inference and variance explained
4. Causality and multivariate relationships
5. Categorical predictors and dummy coding
6. General linear model
7. ANOVA via regression
8. One-way ANOVA example

📖 1. Multiple regression model

🔑 Key Concepts & Definitions

• Multiple regression model : A multiple regression model expresses the conditional mean of a numerical outcome as a linear function of several predictors.
• Regression predictors : Regression predictors are the multiple explanatory variables used to build the linear equation for the response’s expected value.
• Flat surface plane : In two-predictor regression, the model corresponds to a plane determined by separate slopes for each predictor.

📝 Essential Points

• The multiple regression equation is $E(y)=a+b_1x_1+b_2x_2+\dots+b_kx_k$, where $k$ is the number of predictors.
• With two predictors, the model can be visualized as a plane with two slopes, one for $x_1$ and one for $x_2$.
• To represent a model with $k$ predictors needs $k+1$ dimensions for the corresponding geometric depiction.

💡 Memory Hook

Think “plane for 2 slopes”: more predictors mean more dimensions ($k+1$).

📖 2. Regression fit and residuals

🔑 Key Concepts & Definitions

• Residual : A residual is the prediction error for one observation, computed as the difference between the observed and predicted values.
• Sum of squared errors : Sum of squared errors is the total squared residual size, measuring overall prediction disagreement with the data.
• Least squares criterion : The least squares criterion selects regression coefficients that achieve the smallest sum of squared errors.

📝 Essential Points

• For one observation, the residual is $e=y-\hat{y}$ where $y$ is observed and $\hat{y}$ is predicted.
• Sum of squared errors is $SSE=\sum( y-\hat{y})^2$ and quantifies total prediction error magnitude.
• The estimated coefficients $a$ and $b_k$ are chosen so the SSE is minimized (least squares).
• In the worked example, person 10 has $\hat{y}=24.348$ and residual $e=23-24.348=-1.348$, giving $e^2\approx1.823$.

💡 Memory Hook

Residuals are signed errors; SSE squares them so opposite errors don’t cancel.

📖 3. Regression inference and variance explained

🔑 Key Concepts & Definitions

• Total sum of squares : Total sum of squares measures variation in the outcome around the best null prediction without predictors.
• Regression sum of squares : Regression sum of squares measures variation in the outcome explained by the fitted regression model.
• Coefficient of multiple determination : The coefficient of multiple determination $R^2$ is the proportion of total outcome variance explained by the model.

📝 Essential Points

• The total sum of squares (TSS) quantifies variation in $y$ around $\bar{y}$, the best prediction with no $x$ variables.
• The model partitions variation into $RSS$ (explained) and $SSE$ (unexplained) so that $R^2=RSS/TSS=(TSS-SSE)/TSS$.
• In the example, $R^2=0.339$, meaning the model explains about $34\%$ of the variance versus using $\hat{y}$ from the null model.
• For regression to be useful, it should improve prediction over both the null model and simple (single-predictor) models.

💡 Memory Hook

$R^2$ is the “share of TSS left after SSE”: explained over total.

📖 4. Causality and multivariate relationships

🔑 Key Concepts & Definitions

• Causality : Causality is the claim that changes in a variable produce changes in an outcome rather than merely coinciding with it.
• Association : Association is a statistical relationship between a predictor and an outcome, possibly without implying cause.
• Confounding : Confounding occurs when controlling for another variable eliminates or reduces the apparent effect of a predictor.

📝 Essential Points

• Three conditions for causality are required: association between $x$ and $y$, correct time order ($x$ precedes $y$), and elimination of alternative explanations like confounding.
• A spurious association example is that taller children can look better at math because older grade level and more education may explain both.
• In multivariate regression, confounding is when the effect of $x_1$ is reduced or eliminated after controlling for $x_2$.
• Suppression is when the effect of $x_1$ becomes larger or evident after controlling for $x_2$.
• Interaction (moderation) occurs when the effect of $x_1$ changes size depending on the value of $x_2$.

💡 Memory Hook

Association + time order + no confounders is the “causality checklist.”

📖 5. Categorical predictors and dummy coding

🔑 Key Concepts & Definitions

• Dummy coding : Dummy coding represents categorical variable levels using indicator variables that take values 0 or 1.
• Reference category : The reference category is the omitted level in dummy-coded regression so other levels are compared to it.
• Collinearity : Collinearity is redundancy among predictors, which can arise if too many dummy variables for one categorical factor are included.

📝 Essential Points

• Dummy coding uses one dummy variable per categorical level, but only enter one less than the number of levels into the regression.
• Each dummy variable contains only 0s and 1s, and the set of dummy values uniquely identifies the original category level.
• Only two of three dummy variables are used for a 3-level categorical IV; the missing category is the reference category.
• The slope and intercept interpretations depend on (0,1) coding: with a (0,1) dummy IV, the intercept equals the mean for the reference group.
• If the (0,1) category codes change, regression still works but the intercept no longer equals the mean for the intended group.

💡 Memory Hook

Omit one dummy: fewer predictors than levels to avoid collinearity; that omitted level is the reference.

📖 6. General linear model

🔑 Key Concepts & Definitions

• General linear model : The General Linear Model (GLM) is a system of linear models providing the shared framework behind multiple regression and ANOVA.
• GLM hypothesis tests : GLM includes common hypothesis tests matched to its underlying model structure, such as t-tests, F-tests, and ANOVA.
• System of linear models : A system of linear models is a unified modeling structure where different designs can be analyzed using linear-equation methods.

📝 Essential Points

• Regression and ANOVA share the same underlying statistical model, the GLM.
• Using regression to do ANOVA is a foundation for ANCOVA that mixes numeric and categorical IVs.
• Dichotomous predictors can be handled by regression by using dummy coding, even though they are not normally analyzed as a categorical DV.

💡 Memory Hook

GLM is the “bridge”: regression and ANOVA are two faces of the same linear framework.

📖 7. ANOVA via regression

🔑 Key Concepts & Definitions

• ANOVA via regression : ANOVA via regression reformulates group mean comparisons as a regression model using dummy-coded categorical predictors.
• ANCOVA foundation : ANCOVA foundation means regression-based ANOVA prepares you to analyze models combining covariates and categorical factors later.
• Factor notation : Factor notation in regression (like i.party in Stata) tells software to treat a variable as categorical and create appropriate dummy variables.

📝 Essential Points

• ANOVA is used for comparing means across groups, and regression can perform the same test when the categorical IV is dummy coded.
• For k categorical levels, you include multiple dummy variables for that factor in the regression model to represent the factor information.
• Nominal category labels cannot be entered directly in regression because they violate the required linearity assumption.
• Stata factor notation: using $i.$ before a variable name treats it as categorical and uses a default (lowest coded) reference category.

💡 Memory Hook

ANOVA by regression = group means encoded as dummies; F-test stays the “overall ANOVA” test.

📖 8. One-way ANOVA example

🔑 Key Concepts & Definitions

• One-way ANOVA : One-way ANOVA compares the outcome’s mean across multiple groups defined by a single categorical independent variable.
• Political party dummy coding : Dummy coding political party creates indicators so each party’s mean ideology can be compared to the reference party.
• Pairwise comparisons : Pairwise comparisons are contrasts between specific group means derived from regression coefficients for the dummy variables.

📝 Essential Points

• In PoliticalIdeology.dta, the RQ asks whether mean ideology differs by party across Democrat, Independent, and Republican groups.
• The regression form uses ideology as DV and two dummy codes (e.g., regress ideology p1 p2) with the omitted party as reference.
• The overall test is significant: $F(2,2365)=318.79$ with $p<0.001$, with a large effect size reported as $21\%$ variance explained.
• Regression coefficients for dummies are pairwise differences versus the reference category: p1 compares Democrats vs Republicans and p2 compares Independents vs Republicans.
• The constant equals the mean ideology of the reference group (Republicans in the example).
• One pairwise comparison is missing from the coefficient table because only two dummies are entered; you need additional work to get Democrats vs Independents.

💡 Memory Hook

With 3 parties, two dummy lines appear: each dummy compares one party to the reference, leaving one contrast out of the basic output.

📊 Synthesis Tables

Regression vs ANOVA via dummies

Aspect	Regression output	ANOVA analogue
Overall test	F-test for the dummy predictors together	ANOVA F-test across groups
Specific comparisons	Dummy coefficient tests vs reference category	ANOVA contrasts between group means

⚠️ Common Pitfalls & Confusions

1. Confusing association with causality can lead you to claim causation without time order and without ruling out confounding.
2. Including a dummy variable for every category level (instead of one less) can introduce collinearity and distort coefficient interpretation.
3. Interpreting the intercept as a general overall mean rather than the mean of the reference group when predictors are dummy coded (0,1).
4. Treating nominal categorical labels as numeric predictors can break the linearity assumption used by standard regression.

✅ Exam Checklist

1. Write the multiple regression model $E(y)=a+b_1x_1+\dots+b_kx_k$ and state what $k$ means.
2. Compute a residual as $e=y-\hat{y}$ and interpret its meaning as a prediction error.
3. Use $SSE=\sum(y-\hat{y})^2$ and explain that least squares selects coefficients that minimize SSE.
4. Define TSS, RSS, SSE, and compute $R^2=RSS/TSS=(TSS-SSE)/TSS$.
5. State the two regression inference types: the global test of the full additive model and tests of each unique predictor coefficient controlling for others.
6. State the null and alternative hypotheses for the global F-test: $H_0:b_1=\dots=b_k=0$ versus $H_a:$ at least one $b_k\neq0$.
7. Explain the causality checklist: association, correct time order, and elimination of alternative explanations like confounding.
8. Distinguish confounding, suppression, and interaction (moderation) as described by how effects change after adding a second variable.
9. Apply dummy coding rules: 0/1 dummies, omit one less than the number of levels, and identify the reference category.
10. Interpret dummy-variable regression coefficients as pairwise differences vs the reference group and identify that one comparison is missing for $3$ groups with $2$ dummies.
11. Connect one-way ANOVA to regression via dummy predictors by identifying the overall F-test and the coefficient contrasts.
12. Recall how Stata factor notation ($i.$ and ib#.) sets the categorical treatment and the reference category.