More details, less variability? A crossover design study on the impact of information granularity on ChatGPT’s training program stability

This study aimed to evaluate how varying levels of information granularity affect the output variability and multidimensional quality — including Personality, Effectiveness, Safety, and Feasibility — of ChatGPT-generated training programs. A crossover design was used to compare simple and detailed input prompts, with each prompt input into GPT-4 (accessed via ChatGPT) four times to generate eight training programs. The training programs were anonymized by the research team and subsequently evaluated in a blinded manner by 11 experts (mean age = 35.4 years, average of 18.1 years of practical experience in the field of sport and exercise science). Output variability was assessed using the coefficient of variation (CV%), and quality ratings were based on a custom 15-item scale covering Personality, Effectiveness, Safety, and Feasibility. Differences in expert ratings across training programs were examined using repeated-measures ANOVA, with Friedman tests applied as sensitivity analyses to test the robustness of the results. Training programs generated from detailed input prompts consistently received higher expert ratings across all dimensions. CV% was generally lower under the detailed input prompts, indicating more stable outputs. Significant main effects of information granularity were found in Personality, Safety, Feasibility, and overall scores (all p < 0.05), though not in Effectiveness. Notably, repeated inputs of the same information granularity still yielded structurally and qualitatively different outputs, highlighting residual variability even under controlled conditions. Information granularity plays a crucial role in shaping the quality and stability of AI-generated training programs. Providing detailed, structured input enhances personalization, reduces output fluctuation, and improves alignment with exercise science principles.