Lernzettel: Regression Analysis Fundamentals

📋 Course Outline

1. Unit goals and structure
2. Assessment tasks and weekly exercises
3. Regression concepts and notation
4. Simple linear regression example
5. Model fit and prediction intervals
6. Regression assumptions in simple models
7. Multiple regression workflow
8. Data screening and descriptives
9. Collinearity and predictor relations
10. Multiple regression diagnostics

📖 1. Unit goals and structure

🔑 Key Concepts & Definitions

• Generalized Linear Model GLM : A generalized linear framework for statistical analyses that underpins the unit’s approach.
• Unit instructors : The teaching team for PSYU3349, including the Lecturer & Convenor and Co-Convenor who oversee instruction and coordination.
• Unit schedule by weeks : A week-by-week progression that pairs readings with topics leading from regression basics toward broader regression-related methods.
• Live lectures and Echo360 : A delivery setup where lectures run in person, are live-streamed, and are later available as video recordings via Echo360.

📝 Essential Points

• The main goal is to provide a framework for statistical analyses using the Generalized Linear Model, or GLM.
• The unit runs live lectures every Friday 11 am–2 pm in Price Lecture Theatre and is also live streamed for later viewing on Echo360.
• Weekly practical exercises are completed on iLearn and weekly practical classes (starting Week 2) review answers and allow questions.
• There are three assessment types: a data-analysis task (20%), a practical project (40%), and a final exam (40%).
• Each weekly exercise is based on previous lecture material or earlier revision, not the current week.

💡 Memory Hook

GLM is the backbone: all lectures, exercises, and the unit’s main goal keep returning to GLM.

📖 2. Assessment tasks and weekly exercises

🔑 Key Concepts & Definitions

• Weekly exercises : Weekly exercises are iLearn tasks linked to previous lecture material and/or earlier revision, not the current week.
• Data-analysis task : The data-analysis task is a compulsory on-campus e-task worth 20% of the final unit mark.
• Practical project : The practical project is a statistics application to a psychology question worth 40% of the final unit mark and submitted via TurnItIn on iLearn.
• Final exam : The final exam is a cumulative multiple-choice assessment worth 40% of the final unit mark with a 2-hour time limit.
• iLearn : iLearn is the platform where weekly exercises, project details, task materials, and notices are provided for PSYU3349.

📝 Essential Points

• Each weekly exercise is based on prior lecture content or earlier weeks’ revision, not the current week’s lecture.
• Weekly exercises are best completed before your tutorial session because you can’t rely on last-minute time for technical problems.
• The data-analysis task runs in the Week 7 tutorial session, is completed in Stata on your laptop, and must be done at your enrolled session.
• The practical project is due in Week 8 and includes late penalties as stated in the unit guide, with exact date/time TBC.
• The final exam is multiple-choice, cumulative across the unit, uses 2 hours, and allows 4 single-sided or 2 double-sided A4 notes plus a calculator.

📖 3. Regression concepts and notation

🔑 Key Concepts & Definitions

• Regression : A statistical method that predicts a numeric outcome y from one or more numeric predictors x.
• Residual (error) : The residual is the difference between an observed outcome y and its predicted value $\hat{y}$, capturing unexplained variation.
• Regression sum of squares : Regression sum of squares is the part of variability in y that is explained by the fitted regression line.
• R2 : R2 is an effect-size quantity that summarizes how much variation in y the regression model explains.
• Conditional vs marginal distributions : Conditional distributions describe outcomes given specific x values, while marginal distributions describe overall outcomes without conditioning on x.

📝 Essential Points

• Regression predicts y from x but is not causal, so experimental research is needed to infer causation.
• A simple linear regression model is written as $\hat{y}=\alpha+\beta x$ with an intercept at x=0 and slope giving direction.
• The error term can be expressed as $\epsilon=y-\hat{y}$, so smaller residuals mean closer fit to the regression line.
• Least squares fits the line by minimizing residual (unexplained) sum of squares and correspondingly maximizing regression (explained) sum of squares.
• Different but related notation is used: sample-based relationships use $a,b$ and data-based indices $i$, while the population mean function is $E(y)=a+bx$.

💡 Memory Hook

Prediction ≠ causation: regression finds predictable association (y from x), experiments establish causality.

📖 4. Simple linear regression example

🔑 Key Concepts & Definitions

• Simple linear regression : A simple linear regression model expresses the mean outcome y as a straight-line function of one numeric predictor x.
• Prediction equation : A prediction equation is the fitted regression line written with estimated parameters to produce y from any chosen x.
• Intercept term : The intercept is the fitted value of y when x = 0 in the regression line.
• Slope term : The slope is the fitted mean change in y for a 1-unit increase in x.

📝 Essential Points

• In Stata, the fitted simple linear regression is obtained with the command regress PSYU3349 TUTORIALS.
• For regress PSYU3349 TUTORIALS, the estimated intercept is 73.266.
• For regress PSYU3349 TUTORIALS, the estimated slope is 0.545.
• The fitted prediction equation from that regression is y = 73.27 + 0.55x.
• The slope interpretation here is that PSYU3349 SNG increases by 0.55 points per additional attended tutorial session.

💡 Memory Hook

Think “line = intercept + slope × x”: plug x into $\hat{y}$ to predict y, and slope tells the per-unit change.

📖 5. Model fit and prediction intervals

🔑 Key Concepts & Definitions

• Conditional distribution of y : A conditional distribution is the spread of y values around the regression line for a fixed value of x.
• Conditional mean function : The conditional mean function gives the expected value of y at each x as a + bx (equivalently E(y) = a + bx).
• Conditional standard deviation : The conditional standard deviation is the estimated spread of y values around the regression line at a fixed x, derived from the model’s residual variance.
• Prediction interval : A prediction interval is an estimated range of likely y values for a given x, centered on the conditional mean and using the conditional spread.

📝 Essential Points

• The goal of model fit is to reduce variation of y around the regression line compared with variation around the mean of y.
• The conditional spread around the fitted mean is summarized by SD of the fitted y distribution, computed from residual variance as sqrt(SSE/(n−2)).
• The factor n−2 arises because the linear model estimates two parameters, the intercept a and the slope b.
• When the conditional distribution is approximately normal, about 95% of observations fall within mean ± 2·SD.
• In the tutorials example at x = 3, the estimated conditional mean is 74.92 with SD = 9.08, giving an interval [56.76, 93.08].

💡 Memory Hook

95% rule: conditional normality → predict y as conditional mean ± 2·conditional SD.

📖 6. Regression assumptions in simple models

🔑 Key Concepts & Definitions

• Linear relationship : A regression assumption that the mean of y changes approximately linearly with x across the observed range.
• Independence of observations : A regression assumption that residuals from different observations do not systematically depend on each other.
• Normal residuals : A regression assumption that residuals are approximately normally distributed for valid inference.
• Homoscedasticity : A regression assumption that the conditional variance of y around the regression line is constant for all x.

📝 Essential Points

• Residuals can vary because the x–y relationship is only approximately linear, other factors affect y, and there may be random or systematic measurement error.
• A regression model uses 2 estimated parameters (a and b), so the residual variance estimate uses $n-2$ degrees of freedom.
• With conditional SD $\hat\sigma=9.08$, about 95% of observations at a fixed x should fall in mean ± $2\hat\sigma$, giving [56.76, 93.08] when the conditional mean is 74.92.
• Heteroscedasticity shows up as a “fanning” pattern on a residual-vs-fitted (rvf) plot, indicating non-constant conditional variance.
• The conditional distribution of y at fixed x differs from the marginal distribution because it measures spread around the regression line rather than around the overall mean.

💡 Memory Hook

Assumptions checklist: Linear + Independent residuals + Normal residuals + Homoscedastic (equal spread) → use rvf fanning to spot the variance problem.

📖 7. Multiple regression workflow

🔑 Key Concepts & Definitions

• Statistical control : Multiple regression fits 2+ predictors together to separate their overlapping contributions and identify unique relationships in correlational data.
• Observation-to-predictor ratio : The observation-to-predictor ratio compares sample size n to the number of predictors p to judge whether the model is adequately sized.
• Univariate data description : Univariate data description summarizes each variable on its own using graphical and numerical summaries before checking relationships.
• Bivariate graphical data description : Bivariate graphical description uses plots to examine how the DV relates to each IV and how IVs relate to each other.
• Correlation matrix : A correlation matrix reports pairwise correlations (with significance markers) to support checks on how predictors and the DV move together.

📝 Essential Points

• A 9-step workflow is to recognize multiple regression, state research questions, understand population/data, do univariate summaries, do bivariate plots, do bivariate numerical checks, fit full model, reduce it, then fit the final model and report it.
• For sample size planning use n > 5p as a minimum and n > 10p as desirable, and in the example n = 40 with p = 2 gives n = 20p.
• Univariate summaries include distribution “5 points” such as central tendency, variability, skewness, kurtosis, and modality for each variable separately.
• Bivariate graphical description includes DV vs each IV plots, IV vs IV plots, and a scatterplot matrix (graph matrix) to quickly inspect relationships and potential outliers.
• For collinearity screening in step 6, strong predictor correlations are suggested by heuristics of |r| > 0.7 for possible collinearity and |r| > 0.8 for definite collinearity.

💡 Memory Hook

Workflow shortcut: RQ → Univariate → Bivariate (plots + correlations) → Fit (full then reduced) → Final report.

📖 8. Data screening and descriptives

🔑 Key Concepts & Definitions

• Scatterplot matrix : A scatterplot matrix is a grid of pairwise scatterplots used to visually inspect relationships among a set of variables, including the DV versus each IV.
• Tolerance : Tolerance is the proportion of variance in one IV not explained by the remaining IVs when that IV is regressed on all others.
• Variance inflation factor : Variance inflation factor (VIF) quantifies how much multicollinearity inflates uncertainty by using the inverse of tolerance.

📝 Essential Points

• For univariate histograms, comment on central tendency, variability, skew, kurtosis, and modality, using the same 5-point structure across variables.
• A common sample-size guideline for $n$ observations and $p$ predictors is at least $n>5p$ (preferably $n>10p$) to support later multiple regression.
• Bivariate graphical screening includes plotting the DV against each IV and plotting IVs against each other, then checking for outliers and unusual patterns.
• As an initial heuristic for collinearity, consider $|r|>0.7$ as possible collinearity and $|r|>0.8$ as more definite when using IV–IV correlations.
• Tolerance and VIF are linked by $VIF=1/tolerance$, with tolerance values $<0.1$ indicating an IV is hard to distinguish from other IVs and VIF values $>10$ indicating concern.

💡 Memory Hook

5-point distribution checklist: Center–Spread–Skew–Kurtosis–Modality.

📖 9. Collinearity and predictor relations

🔑 Key Concepts & Definitions

• Collinearity : Collinearity occurs when two or more independent variables are so correlated that one can be predicted accurately from the others.
• Multicollinearity : Multicollinearity exists when an independent variable can be reproduced from a linear combination of the remaining independent variables.
• Cohen correlation cut-offs : Cohen’s cut-offs classify correlation strength using approximate ranges for weak, moderate, and strong relationships by |r|.

📝 Essential Points

• Collinearity/multicollinearity arises with multiple IVs and the non-orthogonality typical of observational (non-experimental) studies.
• Strong correlations between IVs are sufficient to suggest collinearity and multicollinearity.
• Multicollinearity can be indicated by large correlations among IVs and by large changes in coefficients or standard errors when adding a new IV.
• A bivariate heuristic is that collinearity is possible when |r| > 0.7 and definite when |r| > 0.8.
• Tolerance is computed as 1/VIF, where tolerance < 0.1 indicates an IV is hard to distinguish from a combination of other IVs.
• Bivariate correlations may miss collinearity because one IV can be a non-obvious linear combination of multiple IVs, so use VIF/tolerance for proper assessment.

💡 Memory Hook

If VIF gets big, it means your coefficient variance is inflated; tolerance is the “leftover variance,” so small tolerance means severe overlap.

📖 10. Multiple regression diagnostics

🔑 Key Concepts & Definitions

• Residual independence : Residual independence means the regression residuals do not have systematic dependence across observations.
• Normal probability plot : A normal probability plot is a Q–Q style display used to judge whether residuals follow an approximately normal distribution.
• Shapiro-Wilk test : The Shapiro-Wilk test is a statistical test used to evaluate whether residuals are consistent with normality.
• rvf plot : An rvf plot graphs residuals against predicted values to check whether residual variance stays roughly constant.
• rvpplot residual-vs-predictor : An rvpplot plots residuals against a specific predictor to assess linearity for that predictor.

📝 Essential Points

• Residual independence is treated as met when the sampling method implies independent errors.
• Normality of residuals is assessed using a normal probability plot and the Shapiro-Wilk test on residuals, with a non-significant p-value supporting normality.
• Homoscedasticity is checked with an rvfplot by looking for no obvious change in residual spread across predicted values.
• Linearity is globally screened with an rvf plot but it cannot guarantee linearity at every predictor value.
• A more thorough linearity check in multiple regression is to use rvpplot for each IV to look for trends in residuals.

💡 Memory Hook

Use 4 checks: independent errors → normal residuals (plot + Shapiro) → constant variance (rvf) → linearity (rvpplot per IV).

📅 Key Dates

Date	Event
February 27	Week 1: Agresti 9 (revision) & 11; Multiple regression
March 6	Week 2: Agresti 12.1-12.4; ANOVA by regression I
March 20	Week 4: Agresti 13.1-13.2; ANCOVA
April 24	Week 8: Agresti 14.1 & notes; Model reduction
May 8	Week 9: Agresti 15.1-15.3; Categorical data & logistic regression II
June 5	Week 13: n/a; Review

📊 Synthesis Tables

Conditional vs marginal distributions

Type	What it measures	Center/spread reference
Marginal distribution of y	Spread or variance of scores around mean %𝑦	Around the overall mean (not conditioning on x)
Conditional distribution of y	Spread or variance of y scores around the regression line for any given value of x	Around the regression line at fixed x (conditional upon x)

⚠️ Common Pitfalls & Confusions

1. Confusing prediction with causation: regression predicts y from x but cannot determine causality; only experimental research is said to infer causal relationships.
2. Mixing up conditional vs marginal spread: conditional spread is around the regression line for fixed x, while marginal spread is around the overall mean of y.
3. Using the wrong degrees of freedom for conditional SD: the SD estimate uses sqrt(SSE/(n−2)) because two parameters (a and b) are estimated.
4. Assuming bivariate correlations are enough for collinearity: one IV can be a non-obvious linear combination of others, so tolerance/VIF are needed.
5. Reversing the interpretation of tolerance and VIF: tolerance = 1/VIF and tolerance < 0.1 (VIF > 10) indicates an IV is hard to distinguish / of concern.
6. Interpreting partial regression coefficients without “holding other predictors constant”: in multiple regression, b1/b2 are effects holding the other IV(s) fixed.
7. Centering confusion: centering is rescaling an IV by subtracting its mean (so intercept becomes more sensible) and it does not change the apparent relationship (scatterplots stay the same except origin).

✅ Exam Checklist

1. State the unit’s main goal as providing a framework for statistical analyses using the Generalized Linear Model (GLM).
2. Identify the live lecture time/location and how recordings are accessed (Echo360 on iLearn).
3. Describe the 3 assessment types with their weightings and key logistics (data-analysis task Week 7 on-campus; practical project Week 8; final exam cumulative MCQ, 2 hours, notes allowed).
4. Explain regression research wording and the distinction: regression predicts y from x but is not causal without experimental research.
5. Given simple linear regression notation, write the prediction equation form (prediction equation) and interpret slope/intercept as in the tutorial example.
6. Compute/interpret conditional SD and the approximate 95% interval (mean ± 2·SD) for a fixed x using the tutorial-style numbers.
7. List the simple regression assumptions (linear relationship, independence, normal residuals, homoscedasticity) and the specific informal/formal checks named (rvf plot fanning; normal probability plot + Shapiro-Wilk).
8. For multiple regression workflow, reproduce the 9 steps in order (including the univariate, bivariate graphical, bivariate numerical/correlation matrix, full model, reduce, final model/report).
9. Apply the observation-to-predictor ratio rule for sample size planning (minimum n > 5p; desirable n > 10p) and use the example phrased as n = 20p when p = 2 and n = 40.
10. Use the collinearity heuristics and proper assessment: interpret |r| > 0.7 possible and |r| > 0.8 definite, then connect VIF/tolerance with tolerance = 1/VIF and tolerance < 0.1 (VIF > 10).
11. Run and describe the regression diagnostics procedures: normality (normal probability plot + Shapiro-Wilk), homoscedasticity (rvfplot/no change in residual spread), and linearity checks (rvfplot global; rvpplot residuals vs each IV).
12. Interpret multiple regression coefficients correctly: explain partial regression coefficients as effects holding the other predictor(s) constant and note the y-intercept depends on the fixed values of IVs, motivating centering.