Trustpilot enhanced the ranking of reviews on business profile pages using LADDER, a method that employs LLMs for the efficient annotation of data and the training of a point-wise LTR algorithm. This approach increased the relevance of the top four reviews, reducing ‚See all reviews‘ clicks by 5% and improving the user experience. Attendees will learn how to use LLMs to scale ranking data annotation efficiently and validate LTR models. We will discuss the results and provide actionable insights for improving ranking systems.