Preheat your oven to 350°F (180°C) and grease a 9x5-inch loaf pan, lining it with parchment paper.
In a medium mixing bowl, whisk together the all-purpose flour and baking powder until well combined.
In a large mixing bowl, combine granulated sugar, culinary lavender, and lemon zest. Rub the mixture until fragrant.
Add softened butter to the flavored sugar mix and cream together using an electric mixer until light and fluffy.
Add three large eggs one at a time, mixing well after each addition. Fold in fresh lemon juice.
Mix half of the flour mixture into the wet ingredients, then add room temperature milk and mix until smooth. Finally, add the remaining flour mixture and stir gently.
Pour the batter into the prepared loaf pan and spread it out evenly.