Moning Lyu

Self-report questionnaires on individual differences (IDs) are central to second language acquisition (SLA) research, yet the qualitative procedures that generate their item pools remain largely unexamined. Drawing on argument-based validation theory and cross-disciplinary measurement guidelines, this study offers the first systematic methodological synthesis of how that formative phase is reported. We reviewed 85 peer-reviewed SLA studies published between 1983 and 2025 that developed or substantially revised an ID scale, coding each along four dimensions: transparency, theoretical grounding, methodological rigor, and content-validity evidence. Four research questions examined how qualitative item development was documented, which methods and frameworks were employed, how item inclusion was justified, and how content validity was assessed. Results expose a pervasive reporting gap: Only 25% of studies described any qualitative data collection, 16% referenced an explicit theoretical framework, 2% linked draft items to participant quotations, and only one study reported a numerical content validity index. Details on coder agreement, reflexivity, and response-process evidence were nearly absent. These gaps threaten construct representativeness, invite cultural bias, and undermine validation claims. We translate our findings into a seven-section reporting checklist for the qualitative phase, with implications for editorial policy, doctoral training, and future meta-analyses.