Experimenting with MLFlow and Microsoft Fabric
A Huge thanks to Martim Chaves who co-authored this post and developed the example scripts.
It's no secret that Machine Learning (ML) systems require careful tuning to become truly useful, and it would be an extremely rare occurrence for a model to work perfectly the first time it's run!
When first starting out on your ML journey, an easy trap to fall into is to try lots of different things to improve performance, but not recording these configurations along the way. This then makes it difficult to know which configuration (or combination of configurations) had the best performance.
When developing models, there are lots of "knobs" and "levers" that can be adjusted, and often the best way to improve is to try different configurations and see which one works best. These things include improving the features being used, trying different model architectures, adjusting the model's hyperparameters, and others. Experimentation needs to be systematic, and the results need to be logged. That's why having a good setup to carry out these experiments is fundamental in the development of any practical ML System, in the same way that source control is fundamental for code.
This is where experiments come in to play. Experiments are a way to keep track of these different configurations, and the results that come from them.
What's great about experiments in Fabric is that they are actually a wrapper for MLFlow, a hugely popular, open-source platform for managing the end-to-end machine learning lifecycle. This means that we can use all of the great features that Mlflow has to offer, but with the added benefit of not having to worry about setting up the infrastructure that a collaborative MLFlow environment would require. This allows us to focus on the fun stuff
