You're asking one of the most crucial (and often most challenging) questions in power analysis! The effect size is the engine of your sample size calculation. Get it wrong, and your entire power analysis is flawed.

Let's break down how to specify the expected effect size, covering your two stated approaches and adding a third common one, along with best practices.

1. Specifying Expected Effect Size for Power Analysis

The effect size quantifies the strength of a phenomenon or the magnitude of a difference/relationship. Unlike p-values, which depend on sample size, effect sizes are standardized and independent of it, making them essential for comparing findings across studies and for power analysis.

There are three primary ways to specify an effect size:

A. Based on Prior Research (Empirical Approach)

This is generally the most robust and preferred method. You leverage existing knowledge to estimate the likely magnitude of the effect you expect to find.

• Conduct a Thorough Literature Search:


• Look for studies highly similar to yours in terms of population, intervention/exposure, outcome measures, and design.


• Prioritize Meta-Analyses: If a meta-analysis on your topic exists, it provides a pooled, more reliable estimate of the effect size, often the best starting point.


• Identify Direct Effect Sizes: Look for studies that explicitly report effect sizes like:


• Cohen's d (for comparing two means, e.g., t-tests)


• Pearson's r (for correlation)


• Odds Ratio (OR) or Relative Risk (RR) (for dichotomous outcomes, e.g., logistic regression)


• Eta-squared ($\eta^2$) or Partial Eta-squared ($\eta_p^2$) (for ANOVA)


• Cramer's V or Phi ($\phi$) (for chi-square tests)


• Regression Coefficients (unstandardized and standardized)


• Convert Reported Statistics to Effect Sizes: If studies don't directly report effect sizes, you can often calculate them from other reported statistics:


• From Means and Standard Deviations: If a study reports means and standard deviations for two groups, you can calculate Cohen's d:

• From t-statistics: For an independent samples t-test:

• From F-statistics (ANOVA): For a one-way ANOVA with two groups, $F = t^2$, so you can use the t-stat formula. More generally, you can calculate partial eta-squared ($ \etap^2 $) from F and df values, and then convert $ \etap^2 $ to Cohen's f or d (though this can be more complex).


• From Chi-square statistics: Calculate Phi ($\phi$) for 2x2 tables:

• Evaluate the Quality and Relevance:


• Sample Size: Larger, well-conducted studies tend to provide more stable estimates.


• Methodological Rigor: Critically appraise the studies. Are they reliable? Are there biases?


• Relevance: How similar are the populations, interventions, and outcome measures to your own study? A slightly different population or intervention might have a different effect size.


• Replication: If multiple studies report similar effect sizes, your estimate becomes more confident. If they vary widely, consider why and perhaps choose a more conservative (smaller) estimate.


• Synthesize and Choose: Based on your review, select the most plausible effect size. If there's a range, consider running a sensitivity analysis (see Section D) using a few different values.

B. Smallest Effect of Interest (Substantive/Clinical/Practical Significance)

This approach focuses on what effect size would be considered meaningful or important in a real-world context, regardless of whether prior research has demonstrated it. This is especially critical in fields like medicine, policy, or education, where an effect might be statistically significant but practically trivial.

• Define "Meaningful":


• Clinical Importance: What is the minimum change in an outcome measure that would be considered beneficial for a patient? (e.g., a 5-point reduction on a pain scale, a 10% reduction in blood pressure, an extra 3 months of life). This is often called the Minimum Clinically Important Difference (MCID).


• Policy Relevance: What change would justify the cost or effort of implementing a new policy?


• Educational Impact: What improvement in test scores would be considered worthwhile for a new teaching method?


• Cost-Benefit: Would the observed effect justify the resources expended?


• Consult Experts and Stakeholders:


• Engage with clinicians, policy makers, educators, or other domain experts. Ask them: "What is the smallest effect of X on Y that you would care about or consider important enough to act on?"


• Gather their consensus or a range of opinions.


• Review Existing Benchmarks or Guidelines:


• Some fields have established thresholds for what constitutes a meaningful change.


• For example, in public health, a certain reduction in disease incidence might be a recognized target.


• Translate to a Standardized Effect Size: Once you have a raw "smallest meaningful difference" (e.g., 5 points on a scale), you'll need an estimate of the standard deviation of that outcome measure in your target population to convert it into a standardized effect size (like Cohen's d). You might get this SD from:


• Prior research on similar populations.


• Pilot data.


• Educated guess.

C. Conventions/Rules of Thumb (Cohen's Guidelines)

When prior research is very scarce, conflicting, or when defining a "smallest effect of interest" is difficult, researchers sometimes resort to conventional guidelines for "small," "medium," and "large" effect sizes, often attributed to Jacob Cohen.

Use with Extreme Caution! These are general benchmarks and may not be appropriate for your specific field or outcome. Always prefer empirical evidence or a substantive rationale if possible.

• Cohen's d (difference between two means):


• Small (0.2): A difference that is barely perceptible. Roughly a 15% overlap between groups.


• Medium (0.5): A difference large enough to be visible to the naked eye. Roughly a 33% overlap.


• Large (0.8): A substantially noticeable difference. Roughly a 47% overlap.


• Pearson's r (correlation):


• Small (0.1): Accounts for 1% of the variance.


• Medium (0.3): Accounts for 9% of the variance.


• Large (0.5): Accounts for 25% of the variance.

• In truly exploratory studies where no prior data exists.


• As a sensitivity check: "What sample size would I need to detect even a 'small' effect?"


• As a last resort when all other methods fail.

D. Best Practices & Key Considerations

• Justify Your Choice: Regardless of how you derive it, explicitly state and defend the chosen effect size in your study protocol or grant application. Explain your reasoning.


• Conservatism: If you are uncertain, err on the side of a smaller effect size. A smaller effect size requires a larger sample size to achieve the same power, making your study less likely to be underpowered.


• Sensitivity Analysis: This is highly recommended. Don't just pick one effect size. Calculate sample size for a range of plausible effect sizes (e.g., the smallest effect of interest, an effect from prior research, and a slightly more optimistic one). This shows you've considered uncertainty and allows you to understand the trade-offs.


• Pilot Studies: If your study is completely novel, a pilot study can provide preliminary data to estimate the effect size and variance, which can then inform a more accurate power analysis for the main study.


• Type of Effect Size: Ensure the effect size you choose or calculate is appropriate for the statistical test you plan to use (e.g., Cohen's d for t-tests, r for correlations, OR for logistic regression, etc.). Power analysis software (like G*Power or packages in R) will require specific effect size metrics depending on the test.

In summary, when specifying your effect size:

• Priority 1: Find it from prior, high-quality research (especially meta-analyses) that is highly relevant to your study.


• Priority 2: If prior research is limited or shows trivial effects, define the smallest effect size that is clinically or practically meaningful in your field.


• Priority 3 (Last Resort): Use Cohen's conventions as a very rough guide, but acknowledge their limitations.