Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA

Auto-WEKA, which is created to help users by automatically fetching through the space of WEKA’s learning algorithms and respective hyperparameter techniques to improve the performance, using different optimization methods.

Progressively, researchers of Machine Learning tools who  are non-experts requiring off-the-rack solutions?

The Machine Learning community has brought available a wide variety of powerful learning algorithms and feature selection methods through open-source packages, such as WEKA and mlr. These packages give an option for the user to make choices such as: selecting a learning algorithm and setting hyperparameters to the model. It is challenging to pick the right choice, so many times users leave to select algorithms based on reputation or set hyperparameters to default values. 

Certainly, adopting such an approach will yield performance far worse than that of the best method and hyperparameter settings.

A likely explanation is that it is very challenging to search the combined space of learning algorithms and their hyperparameters: the response function is noisy and the space can be highly dimensional, involving both categorical and continuous choices, and on top of it contains hierarchical dependencies.

Another related concept of work is on meta-learning procedures that extract characteristics of the dataset, such as the computation of so-called landmarking algorithms, to predict which algorithm or hyperparameter configuration will perform well. 

 The most preferable is Bayesian optimization procedures.

To demonstrate the feasibility of an automatic approach to solving many problems, we built Auto-WEKA, which solves problems for the learners and feature engineers which are implemented in the WEKA Machine Learning package. 

Meta-methods take a single base classifier and its parameters as input, and ensemble methods take any number of base learners as input. These ensemble methods are amazing where we have many settings related to tuning hyperparameters. All ML algorithms do not apply to all datasets as reasons can be anything. For any given dataset, our Auto-WEKA implementation automatically only considers the subset of applicable learners.

From: Auto-WEKA: Automatic Model Selection and Hyperparameter Optimization in WEKA

Auto-WEKA which works with tuning hyperparameter values moves straight up to the fidelity of the machine. We can highlight that this combined hyperparameter space is far bigger than an easy union of the bottom learner's hyperparameter spaces since the ensemble methods allow 5 independent base learners. The meta, ensemble methods together with feature selection contribute to moving forward to the entire size of AutoWEKA’s hyperparameter space. 

CIFAR-10-Small is a subset of CIFAR-10, where we have 50,000 training data points.

An approach that the user might take is to perform a 5-fold cross-validation on the training set for every technique with unmodified hyperparameters, and choose the classifier with the smallest average misclassification error across folds.

An Auto-WEKA is amazing at optimizing its given objective function however, this cannot be enough to conclude that it fits models that work well. Because the number of hyperparameters of a machine learning algorithm grows, so does its potential for overfitting. The cross-validation will considerably increase the robustness of Autoweka against overfitting, but as its hyperparameter tuning factor is more far-reaching than that of any other normal classification algorithms, it's very important to analyze whether overfitting poses a controversy in such a scenario.

Conclusion:

The automated process of picking algorithms, regulating the hyperparameters, monotonous modeling building, and evaluation of models built are going to be the next phase of Auto ML. We can say that " it can't be neither an automatic data science nor automated development in AI", it is concluded as “transforming model building” Currently, selecting the “best” algorithm to use as per dataset requires a level of intuition or expertise about the information. Data scientists leverage their experience to experiment with different combinations of models and hyperparameter values to attain the best accuracy.Click Here Data Science Course

AutoML will lessen our dependence on intuition by iteratively trying out an algorithm, scoring its performance, and selecting and refining other models. In other words, it'll automate the machine learning process of the information science workflow.

AutoML will become mainstream and help to accelerate the model-building process.