Firms increasingly fine-tune generative AI on A/B test data to produce marketing content (headlines, subject lines, advertisements) that resonates with audiences. But fine-tuning on outcomes teaches models what content performs well without teaching them why, producing reward hacking: in our setting, standard fine-tuning inflates ``shocking'' from 0.7% to 43.7% of generated headlines and independently amplifies hyperbolic language and narrows vocabulary. We propose knowledge-guided alignment, which conditions fine-tuning on a structured knowledge block K encoding validated behavioral hypotheses about why content engages (such as curiosity gaps, protagonist framing, or audience-relevant specificity). While the DPO temperature beta constrains how far the model moves from its reference policy, K constrains where it moves, toward hypothesis-consisten generation. K is discovered endogenously from the A/B data: a reasoning LLM proposes candidate hypotheses from small mini-batches of preference pairs, each candidate is validated by its effect on generation quality across the full training distribution, and a combinatorial search selects the optimal set. Across 23,437 A/B-tested headlines, knowledge-guided alignment is preferred by human evaluators 38% vs. 31.5% for standard fine-tuning (p < 0.001) and reduces clickbait, hyperbole, and vocabulary collapse without targeting any explicitly. The gains are largest when training data is scarce. No alternative (human expertise, zero-shot LLM knowledge, four state-of-the-art hypothesis generators, or direct clickbait penalization) achieves simultaneous improvement across all evaluation dimensions.

ljudy@ceibs.edu