Reliability and Validity in Research (2026): Types, Threats & How to Demonstrate Them

Your examiners will ask one question above all others when reading your methodology chapter: “How do you know your measurements actually measured what you say they measured, and would they produce the same results if someone else repeated them?” These twin demands — that your instruments are consistent and that they are accurate — are what researchers call reliability and validity in research. Getting them wrong does not merely weaken one section of your thesis; it undermines every finding that follows.

This guide gives you the conceptual framework, the standard threshold values, the classical threats you must address, and the reporting language that satisfies doctoral examiners at institutions following Creswell, Shadish, and Lincoln & Guba traditions — all in a single, chapter-ready reference.

If you have not yet decided whether your study will be quantitative, qualitative, or mixed, start with our primer on qualitative vs quantitative research to orient the concepts here within your broader design.

Reliability vs Validity: Core Definitions

Think of reliability as a bathroom scale that always shows the same reading for the same object regardless of the day or who stands on it. Think of validity as whether that scale is actually measuring your weight — not the weight of your clothes, not fluid retention, but the underlying construct you care about. A scale that consistently over-reads by 5 kg is reliable but not valid. A scale whose readings fluctuate randomly is neither reliable nor valid. You need both.

In the measurement literature this relationship is expressed with a sporting analogy: a player who consistently hits the left edge of a target is reliable but not valid (biased); a player who scatters shots around the bullseye is valid on average but not reliable (inconsistent). The distinction matters practically because the strategies for improving each are different — reliability problems are usually addressed through instrument design, rater training, and sample size; validity problems require reconceptualisation of the construct or revision of the measurement approach.

Types of Reliability

Test-Retest Reliability

Test-retest reliability assesses the stability of a measure over time. You administer the same instrument to the same participants on two occasions — separated by an interval long enough that memory carry-over is minimised, but short enough that the underlying trait should not have genuinely changed — and correlate the two sets of scores. An intraclass correlation coefficient (ICC) or Pearson correlation of ≥ 0.70 is generally considered acceptable; ≥ 0.80 indicates good stability for most social science instruments. A two-to-four-week interval is a commonly used default for attitude scales in education and psychology research.

Inter-Rater Reliability

When data are produced by human observers or coders — qualitative coding, rubric-based assessment, clinical rating scales — you need inter-rater reliability (IRR) to confirm that different raters produce consistent scores. The appropriate statistic depends on the measurement level: Cohen’s Kappa (κ) for categorical data, weighted Kappa for ordinal categories, and the ICC for continuous ratings. For a detailed walkthrough of calculation and interpretation, see this guide to inter-rater reliability and Cohen’s Kappa. Following Landis and Koch (1977), κ ≥ 0.61 is considered substantial agreement and κ ≥ 0.81 almost perfect. If initial agreement falls below thresholds, conduct a calibration session in which raters discuss borderline cases, revise the coding scheme if needed, and re-code a subsample.

Internal Consistency

Internal consistency captures the degree to which items on a multi-item scale measure the same underlying construct. The most widely reported statistic is Cronbach’s alpha (α). Conventional thresholds are:

However, alpha assumes tau-equivalence — that every item loads equally on the latent factor — an assumption rarely met in practice. McDonald’s omega (ω) derives estimates from actual factor loadings and is increasingly preferred in methods-rigorous journals. Reporting both α and ω is recommended in 2026 when item loadings are unequal.

Parallel-Forms Reliability

Parallel-forms reliability is assessed when two equivalent versions of the same instrument exist. Participants complete both, and the correlation between scores indicates equivalence. This is used primarily in standardised testing contexts where memory effects from a single administration would inflate test-retest estimates.

Types of Validity

Content Validity

Content validity asks whether your instrument’s items adequately sample the full domain of the construct. It is established through expert judgement — a panel of five to ten subject-matter experts who rate each item on relevance and clarity. Lawshe’s Content Validity Ratio (CVR) or the Content Validity Index (CVI) quantify panel agreement; a CVI of ≥ 0.80 is the commonly cited acceptability threshold.

Construct Validity

Construct validity encompasses all evidence that your instrument actually measures the theoretical construct it purports to measure. It is examined through two complementary routes:

• Convergent validity: Scores should correlate positively with other instruments measuring the same construct.
• Discriminant validity: Scores should correlate weakly with instruments measuring theoretically distinct constructs.

In structural equation modelling (SEM), convergent validity is evaluated using Average Variance Extracted (AVE ≥ 0.50), and discriminant validity via the Fornell-Larcker criterion. Confirmatory Factor Analysis (CFA) is the primary tool for evaluating construct validity formally.

Criterion Validity

Criterion validity assesses whether your measure predicts or correlates with an external criterion:

• Concurrent validity: The measure correlates with an established criterion administered at the same time.
• Predictive validity: Scores at Time 1 predict an outcome measured at Time 2.

Internal Validity

Internal validity is the degree to which a study can support a causal inference — that changes in the independent variable caused changes in the dependent variable. Campbell and Stanley’s foundational framework identifies the most common threats:

• History: External events between pre- and post-measurement that affect the outcome independently of the intervention.
• Maturation: Natural changes in participants over time that mimic a treatment effect.
• Testing effects: Familiarity with the instrument from prior exposure improves performance independently of the intervention.
• Instrumentation: Changes in the measurement tool or rater behaviour over data collection.
• Statistical regression: Extreme scorers at pre-test tend toward the mean at post-test due to measurement error.
• Selection bias: Pre-existing differences between groups that explain post-intervention differences.
• Attrition: Non-random dropout of participants that alters group composition.

External Validity

External validity addresses the extent to which findings can be generalised beyond the immediate study context — across different populations, settings, and time points. This is directly linked to your choice of sampling methods in research: probability sampling supports stronger external validity claims than convenience or purposive sampling.

Threats to Reliability and Validity: Consolidated Overview

Demonstrating Reliability & Validity in Quantitative Research

Claiming reliability and validity without evidence is insufficient for a thesis or dissertation. Your methodology chapter must present a coherent demonstration strategy. Best practice in 2026 typically follows these steps:

1. Instrument selection or construction: When using an established scale, cite psychometric evidence from the original validation study and any relevant cross-sample adaptations. When constructing a new instrument, begin with a systematic item-generation phase grounded in existing theory.
2. Expert review for content validity: Convene a panel of experts who rate each item on relevance and clarity. Calculate the CVI and report it. Revise or remove items falling below threshold.
3. Pilot testing: Administer to a small sample (n ≥ 30) drawn from the same population as your main study. Compute alpha and omega; identify and revise items with item-total correlations below 0.30.
4. Confirmatory Factor Analysis: For instruments claiming to measure a specific latent construct, CFA provides the most rigorous evidence of construct validity. Report CFI ≥ 0.95, RMSEA ≤ 0.06, SRMR ≤ 0.08, standardised factor loadings, and AVE.
5. Inter-rater reliability (where applicable): Calculate and report the appropriate IRR statistic before main data collection begins. Document calibration procedures in your methods.

Qualitative Trustworthiness: The Lincoln & Guba Framework

Quantitative concepts of reliability and validity do not transfer cleanly to qualitative inquiry, where the goal is rich, contextually grounded understanding rather than statistical replication. Lincoln and Guba (1985) proposed four parallel criteria for establishing trustworthiness, each mapping conceptually onto a quantitative counterpart but demonstrated through procedural rather than statistical means.

Credibility

Credibility concerns whether your findings accurately reflect participants’ perspectives. The most powerful strategy is member checking (respondent validation): returning interpreted data or preliminary themes to participants and asking whether the interpretation resonates with their experience. Combined with triangulation — using multiple data sources, methods, or investigators — member checking substantially strengthens credibility claims. Participant disagreement with your interpretations is not a problem; it is analytically informative.

Transferability

Unlike external validity in quantitative research, transferability does not claim that findings generalise statistically to a defined population. Instead, the researcher provides thick description — sufficiently detailed contextual information about the setting, participants, and conditions — to enable readers to judge whether findings might be applicable to their own contexts. The responsibility for transfer is shared: you create the basis for it; readers evaluate its relevance.

Dependability and Confirmability

Both criteria are supported through maintaining an audit trail — a transparent, documented record of research design decisions, data collection procedures, analytic choices, and how interpretations evolved throughout the inquiry. A reflexivity statement acknowledging your positionality and how prior assumptions may have shaped interpretation is standard practice for confirmability in qualitative studies that employ thematic analysis and other interpretive frameworks.

How to Report in Your Methodology Chapter

The research methodology chapter of a dissertation or thesis should present reliability and validity evidence systematically under a dedicated subsection — typically titled “Reliability and Validity of Instruments” or, for qualitative studies, “Rigour and Trustworthiness.”

For quantitative studies, include:

• The statistical evidence collected (alpha, omega, ICC, CFA fit indices, CVI scores)
• The benchmark thresholds applied, with citations for each (e.g., Nunnally, 1978 for alpha; Landis & Koch, 1977 for Kappa)
• Whether evidence derives from the original validation study, a prior study with your population, or your own pilot
• Any remediation steps taken when initial values were subthreshold

For qualitative studies, include:

• Explicit reference to Lincoln and Guba’s trustworthiness framework
• The specific strategies employed for each of the four criteria and when in the research process they were applied
• A reflexivity statement acknowledging your role as the primary instrument of data collection and analysis

Frequently Asked Questions