Introduction

---

The purpose of this paper is to offer a simpler and more basic relationship between the number of independent variables (k) and the size of a sample (n) used in building the multiple regression model. This should prove useful in particular to those with limited knowledge of advanced statistical concepts.

In simple regression, the model focuses on only one predictor to explain the behavior and variability of a dependent variable by examining the variability of this one predictor. The most basic and common model is the simple linear regression one. Certain assumptions are made when using a simple linear regression to model the relationship between the dependent variable (response) and the independent variable (predictor).These assumptions are conditions necessary to draw inferences regarding the model, such as the normal distribution of the errors and the independence of the observations.

A common but important misunderstanding among students and less experienced practitioners who want to evolve the simple regression model to a multiple regression one is the practice of adding another variable while maintaining the same sample size; the misconception is to overlook the challenge posed by multiple predictors such as multicollinearity and the insignificance of such new predictors in the model. The result often appears satisfactory to the user, since they see an increase in the mathematical results, such as the coefficient of determination R² and the p-value of the hypothesis testing to determine the significance of the model. However, this creates pitfalls and misinterpretation of the real outcome of the outputs and may lead to incorrect use of the model that is built.

Thus, the focus of this paper is to address these issues, and to present a new and simple relationship that explains how to manage the size of a sample for every additional predictor.

This relationship will be analyzed, and results obtained will be presented and discussed. Since I start the framework by evolving the simple linear regression model, I limit my independent variables to ten (k 10). In addition, I vary the level of significance (α) to show the consistency of the model.

The analysis is done by observing the behavior of the coefficients of multiple regression, such as the F-statistic and the coefficient of determination R², to make sure that the relationship developed satisfy the significance condition of the testing of the model.

This paper serves as a guiding tool for students and practitioners who may lack the required analytical background and skills yet who need statistical analysis for decision-making, hence the effort to steer away from the vague and complex mathematical models. I aim to connect these different concepts in a more straightforward and accessible way to help this large audience obtain more reliable results. The new developed model has the linear form: n = b₁ (k) + b₀

Contemporary advanced statistical software can be used to answer questions similar to the one addressed here using the complex concepts mentioned earlier, such as the power of the test, the acceptable probability of Type II error, the effect size, and other variables. However, since I am focusing on users and practitioners who do not have the necessary analytical or computing skills, I recognized the need to bridge this gap, leading me to the direction I took.

Prior Literature

Regression analysis is one of the most widely used statistical concepts in almost every domain for the purpose of forecasting and prediction, to find which independent variables are better predictors for a selected dependent variable. Given the contemporary capacity to collect more data, add more predictors, and to build and interpret more complex models, multiple regression has become a more widely used application by a majority of practitioners.

The earliest form of regression was the method of least squares presented by Legendre [19], which is an algebraic technique for fitting linear equations to data. Gauss [13] claimed that he was the first one to come up with the least squares work, where he took it beyond Legendre and succeeded in connecting the method with the principles of probability and normal distribution.

Bartko et al. [2] focused on the importance of statistical power accompanied by nomograms for determining sample size and statistical power for the student’s t-tests, whereas Cohen [6,7] and Erdfelder et al. [9] addressed the continued neglect of statistical power analysis in the research by providing a convenient presentation of required sample sizes. Effect-size indexes and conventional values are given for small, medium, and large effects.

I also find an abundance of work, dating back to Fisher [10], and not limited to Bland and Altman [4]; Rovine and Von Eye [24]; and Rodgers and Nicewander [23] who all addressed different aspects of the simple linear regression model. Fisher focused on the frequency distribution of the values of the correlation coefficient in samples from large populations, whereas Rodgers and Nicewander presented thirteen different formulas, each of which represents a different computational and conceptual definition of the correlation coefficient, r.Rovine and Von Eye expanded on this research by presenting a fourteenth way.

Sample sizes have always been a major topic of research when regression is involved. To mention a few, Fritz and MacKinnon [11] presented the necessary sample sizes for six of the most common and most recommended tests of mediation for various combinations of parameters. Hsieh et al. [15] developed sample size formulae for comparing means or proportions in order to calculate the required sample size for a simple logistic regression model. Maas and Hox [20] used a simulation study to determine the influence of different sample sizes at the group level based on the accuracy of the estimates (regression coefficients and variances) and their standard errors. Zaarour and Melachrinoudis [27] developed a relationship between the sample size, the coefficient of determination, and the level of significance, initially in simple linear regression for relatively small sample sizes and later expanded the relationship to include the number of independent variables in multiple regression.

Moving from the basic simple linear regression model to the more complex multiple regression model creates a few important issues. One of these issues is how to deal with the initial sample size. Oliker [22] stated that the problem of determining the number of independent variables is considered under the assumption that the dependent and independent variables have a joint normal distribution, whereas Knofczynski and Mundfrom [18], and Beaujean [3] used a Monte Carlo simulation to model different scenarios to examine the smallest sample sizes for multiple scenarios for each number of independent variables.

This research primarily focuses on the sample size when we decide to increase the number of independent variables is increased. Extensive research has been conducted in this area. Gatsonis and Sampson [12] provided a formula to calculate the minimum sample size required to ensure the reliability of a multiple linear regression model, as did Mendoza and Stafford [21]. Shieh [25,26] presented multiple relationships in separate research papers. Kelly [17] and Algina and Keselman [1], on the other hand, focused on the relationship between the sample size and the squared multiple correlation coefficients in multiple linear regression. Gatsonis and Sampson used the exact power of the test, rather than an approximation, as well as allowing for the calculation of the minimum sample size for a variety of effect sizes rather than just a single effect size. Mendoza and Stafford also enabled the calculation of the minimum sample size by allowing a variety of effect sizes, but they also considered the distinction between the fixed and the random regression models. Shieh incorporated the correlation between the independent variables, since multicollinearity renders the model less efficient, and additionally took into consideration the possibility of the data being clustered, in cases of observational studies. Kelly stated that the minimum sample size required for a multiple regression model to be reliable depends on the squared multiple correlations in addition to the power of the test and the expected variance of the dependent variable. Furthermore, Bujang et al. [5] developed a relationship for the minimum sample size necessary for the multiple regression model in both experimental and non-experimental studies, using the Power and Sample Size Software.

As mentioned earlier, regression analysis is one of the most applied statistical methods in all scientific fields. Thus, we find research that covers the various aspects in all sectors, including engineering and the medical field. Addressing the determination of the sample size in linear regression analysis, Jan and Shieh [16] improved on the approximate formula of Colosimo et al. [8] by developing a more exact approach that outperformed the approximate methods and offered a useful tool in planning validation studies. The stochastic nature of predictors is taken into account by assuming that they have an identical and normal distribution, whereas the Colosimo et al. approach was adopted a direct replacement of mean values for the predictors.

However, in most of these relationships, the focus is on finding the appropriate sample size by using the desired power of the test, the effect size, and the acceptable probability of a Type II error, in addition to other complex variables which are usually challenging for users who do not have the required advanced mathematical proficiencies.

In this paper, I simplify the relationship between the sample size (n) and the number of independent variables (k) by using just the k to calculate the n. I develop a simple linear relationship between n and k and address the challenge when limited resources are available in collecting a sample. I also tested the model using different levels of significance (α) to ensure reliability and consistency of the results.

Model Development and Solution Procedure

In this section, I introduce the necessary variables used in simple linear regression, then I discuss what it entails to transition to a multiple regression model and highlight the main pitfalls arising from this. Lastly, I introduce a simple relationship to calculate the required sample size, given the number of independent variables, and discuss the results and outputs.

Introduction to Simple Linear Regression Factors and Specifics

Starting with the coefficient of determination, R², which is the ratio of the explained variation to the total variation. R² is used to explain the variability of the dependent variable by considering the variability of the independent variable. It is the used to analyze the fit of the data, using an in-sample performance measure. In addition, to deal with statistical significance, hypothesis testing will have to be performed. One way to obtain this would be to use a statistics similar to the test statistic F. For the simple linear regression case, k = 1 and n ≥3. Hence the relationship between the test statistic F and the R² is:

Equally, given the significant F_α for a specific (α), I can calculate the values of the critical coefficient of determination R² to be compared to the R² values. Starting with the simple linear regression case, I will consider the following:

● According to the central limit theorem, the random sample is considered significant (n ≥ 30), in order for the sample statistics to behave normally.

I will be dealing with relatively small sample sizes since I do not address the work of big data and allowing for our targeted audience’s analytical and computing skills level.

It is worth noting here that I am not referring to the adjusted coefficient of determination Ra2 (which will be discussed later) used with multiple regression, but instead I examine the critical R² values that would render the model significant. As a result, one can always determine the smallest sample size needed to obtain a significant model given a certain level of significance α, where R² needs to be equal or higher to a critical value R². Figure 1 shows these values of n for three specific α values. Consequently, for any value of R² that we obtain from running our simple linear regression model, we can compare the sample size used to a critical sample size nα. If nis at least equal to the critical value nα, the model will be considered significant.

Figure 1

Transitioning into the Multiple Regression Model

In this section, I address the issues and pitfalls that arise from adding additional predictors to the simple regression model. Once I address those, I introduce in the next section the relationship developed in this paper that will give us a direct and simple insight into how big the sample size needs to be to deal with the additional independent variables.

The complexity of analyzing all the issues in multiple regression is beyond the scope of this paper and its targeted audience. Following is a list of some of those issues:

● For relatively small samples, it is not advisable to keep adding predictors, without changing the size of the sample, as the gap between the coefficient of determination R² and the adjusted one R_a² will become more and more significant.

● The mathematical improvement of the outputs is deceiving since it does not necessarily reflect improvement of the model.

● The strong correlation amongst the independent variables, multicollinearity, renders the model less reliable.

● Understanding the desired difference between the true and estimated values of the model.

● Understanding the desired power of the test, and the acceptable probability of the type II error.

● How much to increase the sample size by with every additional k.

When dealing with multiple predictors, I have to ensure that they are not strongly correlated, to avoid redundancy. In addition, merely because the overall model passes the significance test, does not mean that all the predictors are significant and necessary.

Evolving from the simple regression to the multiple regression mode should only be undertaken if it is done correctly. Because our focus in this research is on how to deal with the sample size while making sure the model is significant and avoiding the multicollinearity and the insignificant predictors, the following assumptions are made before moving into the next section:

● No collinearity concerns.

● All added predictors are significant.

● The gap between the coefficient of determination and the adjusted one is accounted for.

Equation (1) can is extended as follows for multiple regression:

Our purpose in this research is to try to find a new simpler equation that can give us the smallest sample size given the number of independent variables considered in the model, regardless of what level of significance I am dealing with.

Relationship between n and k

As mentioned in the Prior Literature section, there has been a solid amount of work done in finding a relationship between the size of a sample and the number of independent variables considered. Most of these relationships are measured based on their effect size. An effect size is typically a value based on a sample estimate of its corresponding population parameter. In addition, the acceptable probability of the Type I (α) and Type II (β) errors are also considered. As the power of the test (1- β) increases with different levels of α, the sample size will also increase.

I offer a new and simple relationship of n and k, and discuss its impact on the effect size, such as R², or on the test Statistic F. I start the analysis by looking at the formula for the adjusted coefficient of determination:

which can be reworked to the following:

R² assumes all the independent variables affect the result of the model, whereas R_a² considers only those variables that actually have an influence on the performance of the model. This is a preliminary endeavor to adjust for the fact that when we add another predictor to a multiple regression, the value of R² will get larger mathematically, regardless of whether the model becomes more significant. Even though we use the adjusted R_a² as an out-of-sample performance measure, a smaller gap between the two coefficients is more desirable. Thus, looking back at Equation (4) in order to minimize the gap between the two coefficients:

The notion that the degree of freedom (n – 1) is larger than the number of independent variables is not significant on its own. However, the question that really needs to be solved in this research is: how big n needs to be given a k. I start by simulating data for equation (3) to observe for what sample sizes and number of independent variables, does the adjusted coefficient of determination (R_a²) takes on non-negative values given different values of R².

I chose the following ranges for the simulation:

● n ≥ 3

● 1 ≤ k ≤ 10

● 0.01 ≤ R² ≤ 0.99, with an increment of 0.01.

Using Excel, I developed 99 different sheets to take into consideration the 99 different values of R². In each sheet (value of R²), I considered the 10 different values of k and the different values of n. As a result, I obtained a total of more than 200,000 values of R_a². After applying the condition: n -1 ≥ k, in equation (5) and eliminating all the negative values of R_a² , I analyzed the remaining values in the following manner: For each value of R² and k, I found the smallest value of n where the Ra2 starts being non-negative.

Table (1) shows an excerpt of an example of the process with one specific value of R² = 0.05. The complete table is provided in (Appendix A). Looking at the first row, we notice that for k = 1, n needs to be at least 22 for R_a² to take on a non-negative value. Not only we are able to find all the n values for a given k, but we also notice a specific trend in the values obtained. We will go over the patterns obtained for all the 99 iterations calculated in the next section.

I simulated the same work for the 99 different values of R². Table (2) and Figure (2) show a summary of the results of the simulation. Results from Table (1) are highlighted in Table(2).

Table 1

Table 2

I can see clear patterns of how the sample size behaves when I increase k for the different values of R². For example, when looking at R² = 0.01, the pattern is: n = 100(k) + 2. When R² =0.05, n takes on the following form: n = 20(k) + 2, Or when R²= 0.10, the equation becomes: n = 10(k) + 2. Table (3) shows a summary of some of these patterns:

I can see a pattern with every iteration of the R², thus the need to develop a general form for these patterns and create an equation that can be used to calculate the sample size as a function of k, regardless of the values of R².

General Form

These patterns found in the 99 different iterations all have the form n = b₁ (k) + b₀, where the slope b₁ gets smaller as the R² gets larger. Table (4) shows an excerpt of the 99 different slopes obtained. A detailed table is provided in (Appendix B) showing the 99 different slope obtained for all the values of R².

Figure 2

Table 3

Table 4

Looking at the distribution of the slopes, the histogram is highly skewed to the right, with the value 10 being the cutoff point where the classes start to dip to the right. In addition, Figure (3) shows the scatter plot obtained graphing all the slopes obtained from the different values of R².

The slope: b₁ = 10 is where the kink is observed. This breakthrough allowed for the formulation of the desired model. To complete this model, I needed to calculate the y-intercept. Taking into account the assumptions of normality and having the desired sample size of n ≥ 30 when starting with only one predictor, the y-intercept value was determined to be 20, hence giving us the following new and simplified relationship between n and k:

Figure 3

I ran a sensitivity analysis to observe the results of the model. I did this for four different levels of significance to make sure that the model cannot be impacted by changing the alpha. The same behavior emerges regardless of what level of significance used. Figure (4) shows an example of this behavior when ∝ = 0.05. I compared the behavior of the critical value of the F-statistic to the different values of F based on the alpha given. The new simplified relationship between n and k passes the significance test and provides a linear form to calculate the sample size needed given the number of predictors used.

It is vital to mention that the general form of n = 10(k) + 20 not only provides the simplicity and the significance needed to deal with the challenge of increasing the sample size when adding predictors for the multiple regression model but can also handle any number of k, not just up to ten predictors.

Figure 4

I ran additional Sensitivity Analysis to compare our results to previous models, such as the one from Green [14], who developed a conservative linear form equation: n ≥ 8(k) + 50 and the one from Zaarour et al. [27] who developed a lower bound equation of the coefficient of determination as a function of n and k: R² > k / (n-1) for multiple regression. I tested my model, and I was able to confirm that my results showed improvement to Green’s well-known rule of thumb as well as satisfied the lower bound condition of R² of Zaarour’s function.

Table (5) and Figure (5) show the results when comparing the new developed model to Green’s conservative model. I am able to obtain significant results by using a less conservative equation. One of the benefits is that we do not need as big sample sizes (up to k = 15) to obtain the desired results. This is important, especially if we have limited resources to collecting data. It is worth mentioning again that this work deals with relatively small samples due to limitations of resources.

Table 5

Figure 5

Concluding Remarks

The pivotal question when dealing with multiple regression is how much the sample size must be increased by when adding independent variables. This paper offers a crucial answer in a model that illustrates that sample size is a function of only the number of independent variables, which answers the challenge of what to do with the sample size and how much to increase it by for each additional predictor. The model is developed and tested with different values of the significance level (∝) to make sure we have consistency in the results.

The purpose of the work is to offer a simpler and more basic relationship between the number of independent variables (k) and the size of a sample (n) used in building the multiple regression model. The framework builds the simple linear regression model up, so independent variables are limited to ten (k 10), and the level of significance (∝), is varied to show the consistency of the model.

This paper serves as a guiding tool for students and practitioners who may not have the necessary analytical background and expand from linear regression models to multiple regression models without the aid of highly advanced software that requires specific skills, yet they use statistical analysis for decision-making. This also explains my effort to steer away from vague and complex mathematical models. The model has a linear form n = b₁(k) + b₀

Table 6

Table 7

References

---

1. Algina J, Olejnik S (2000) Determining sample size for accurate estimation of the squared multiple correlation coefficient. Multivariate Behavioral Research 35: 119-36.
2. Bartko JJ, Pulver AE, Carpenter WT (1988) The Power of Analysis: Statistical Perspectives. Part II. Psychiatry Research 23: 301-9.
3. Beaujean AA (2014) Sample Size Determination for Regression Models Using Monte Carlo Methods in R. Practical Assessment, Research & Evaluation 19: 1-16.
4. Bland JM, Altman DG (1996) Measurement Error and Correlation Coefficients. BMJ 313-412.
5. Bujang MA, Sa’at N, Sidik TMITAB (2017) Determination of Minimum Sample Size Requirement for Multiple Linear Regression and Analysis of Covariance Based on Experimental and Non-experimental Studies. Epidemiology Biostatistics and Public Health 14: 1-9.
6. Cohen J (1992) A Power Primer. Psychological Bulletin 1: 155-9.
7. Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences. Hillsdale: Erlbaum.
8. Colosimo EA, Cruz FR, Miranda JLO, Woensel TV (2007) Sample Size Calculation for Method Validation using Linear Regression. Journal of Statistical Computation and Simulation, 77: 505-16.
9. Erdfelder E, Faul F, Buchner A (1996) GPOWER: A General Power Analysis Program. Behavior Research Methods, Instruments, and Computers 28: 1-11.
10. Fisher RA (1915) Frequency Distribution of the Values of the Correlation Coefficient in Samples from an Indefinitely Large Population. Biometrika 4: 507-21.
11. Fritz MS, MacKinnon DP (2007) Required Sample Size to Detect the Mediated Effect. Psychological Science 3: 233-9.
12. Gatsonis C, Sampson AR (1989) Multiple correlation: Exact power and sample size calculations. Psychological Bulletin 106: 516-24.
13. Gauss CF (1809) Theoria motus corporum coelestium in sectionibus conicis solem Ambientum.
14. Green SB (1991) How Many Subjects does it Take to do a Regression Analysis? Multivariate Behavioral Research 26: 499‐510.
15. Hsieh FY, Bloch DA, Larsen MD (1998) A Simple Method of Sample Size Calculation for Linear and Logistic Regression. Statistics in Medicine 17: 1623-34.
16. Jan SL, Shieh G (2019) Sample Size Calculations for Model Validation in Linear Regression Analysis. BioMed Central: Medical Research Methodology 19.
17. Kelly K (2008) Sample Size Planning for the Squared Multiple Correlation Coefficient: Accuracy in Parameter Estimation via Narrow Confidence Intervals. Multivariate Behavioral Research 43: 524-55.
18. Knofczynski GT, Mundfrom D (2007) Sample Sizes when Using Multiple Linear Regression for Prediction. Educational and Psychological Measurement 68: 431.
19. Legendre AM (1805) Nouvelles méthodes pour la détermination des orbites des comètes.Paris: F. Didot.
20. Maas CJM, Hox JJ (2005) Sufficient Sample Sizes for Multilevel Modeling. Methodology 3: 86-92.
21. Mendoza JL, Stafford KL (2001) Confidence intervals, power calculation, and sample size estimation for the squared multiple correlation coefficient under the fixed and random regression models: A computer program and useful standard tables. Educational and Psychological Measurement 61: 650-67.
22. Oliker VI (1978) On the Relationship Between the Sample Size and the Number of Variables in a Linear Regression Mode. Communication in Statistics – Theory and Methods 7: 505-16.
23. Rodgers JL, Nicewander WA (1988) Thirteen Ways to Look at the Correlation Coefficient. The American Statistician 1: 59-66.
24. Rovine MJ, Von Eye A (1997) A 14th Way to Look at a Correlation Coefficient: Correlation as the Proportion of Matches. The American Statistician 1: 42-6.
25. Shieh GA (2006) Exact interval estimation, power calculation, and sample size determination in normal correlation analysis. Psychometrika 71: 529-40.
26. Shieh GA (2001) Unified Approach to Power Calculation and Sample Size Determination for Random Regression Models. Psychometrika 72: 347-60.
27. Zaarour N, Melachrinoudis E (2019) What’s in a Coefficient? The “Not so Simple Interpretation of R2, for Relatively Small Sample Sizes. Journal of Education and Training Studies 7: 27-40.