Really appreciate the emphasis on a strong product strategy to get the most out of your A/B testing efforts. I'd argue that this is especially important when it comes to those long-term metrics; once a product is in place, it's easy to find a revenue-adjacent metric to optimize and feel like you're making progress. Knowing which tradeoffs are the wrong tradeoffs, and therefore which long-term counter metrics to track, requires a clear strategy of where the business is trying to grow.
Really appreciate the emphasis on a strong product strategy to get the most out of your A/B testing efforts. I'd argue that this is especially important when it comes to those long-term metrics; once a product is in place, it's easy to find a revenue-adjacent metric to optimize and feel like you're making progress. Knowing which tradeoffs are the wrong tradeoffs, and therefore which long-term counter metrics to track, requires a clear strategy of where the business is trying to grow.
Great post.
But you CAN test strategy rigorously and with statistical validity.
People just don't bother.