Automatic Machine Learning (AutoML)

Surveys

• AutoML principles and practices - Organizes automated learning by search space, search algorithm, and evaluation strategy, connecting pipeline configuration with architecture search and the underlying bilevel objective. Shen et al. 2018.
• Hyperparameter optimization survey - Compares optimization approaches for common learning algorithms, summarizes software, and discusses strengths, limitations, and unresolved research questions. Yang and Shami 2020.
• AutoML survey - Reviews automation across data preparation, feature engineering, hyperparameter optimization, and neural architecture search; distinguishes one-shot search and joint architecture/hyperparameter optimization. He et al. 2021.

Hyperparameter search

• Sequential model-based optimization - Introduces sequential expected-improvement methods for conditional hyperparameter spaces, including tree-structured Parzen estimation; evaluates them against random search. Bergstra et al. 2011.
• Random search - Samples configurations instead of enumerating a grid. Its analysis explains why grids waste trials when only a few hyperparameters strongly affect validation performance. Bergstra and Bengio 2012.
• Practical Bayesian optimization - Models validation performance with a Gaussian process and selects subsequent trials from its posterior; also considers variable evaluation costs and parallel experiments. Snoek et al. 2012.
• Neural surrogates for Bayesian optimization - Uses a neural network to learn basis functions for Bayesian regression, replacing a Gaussian-process surrogate when many evaluations make its fitting cost restrictive. Snoek et al. 2015.
• Hyperband - Allocates increasing resources to randomly sampled configurations and stops weak candidates early; searches across different tradeoffs between breadth and evaluation budget. Li et al. 2018.
• BOHB - Combines model-guided configuration selection with Hyperband's budget allocation, joining informed search with evaluations at multiple resource levels. Falkner et al. 2018.
• ASHA - Uses asynchronous resource allocation and aggressive early stopping for parallel hyperparameter tuning, allowing search to exploit many workers without synchronized evaluation rounds. Li et al. 2020.

Pipeline search and ensembles

• Auto-sklearn - Combines Bayesian pipeline search with learning from previous datasets and ensembles of evaluated models; jointly selects preprocessing, classifiers, and hyperparameters. Feurer et al. 2015.
  • Auto-sklearn 2.0 - Adds portfolio-based meta-learning and bandit budget allocation, then selects among AutoML strategies to reduce the configuration burden on users. Feurer et al. 2022.
• AlphaD3M - Uses a pipeline grammar and model-based reinforcement learning with pretrained guidance to synthesize classification and regression pipelines; the paper reports public code, models, and data. Drori et al. 2019.
• AutoGluon-Tabular - Builds multilayer stacked ensembles for tabular prediction, emphasizing combinations of models rather than spending the entire budget on selecting one configuration. Erickson et al. 2020.
• AutoML-DSGE - Evolves scikit-learn classification pipelines with Dynamic Structured Grammatical Evolution, constraining candidate pipelines through a grammar. Assunção et al. 2020.
• Lale - Unifies manual and automated pipeline construction through high-level Python interfaces, including topology search, higher-order operators, and consistency checks. Baudart et al. 2020.
• Auto-PyTorch - Jointly optimizes network architecture and training hyperparameters using multiple evaluation budgets, portfolio warm starts, and ensembles with tabular baselines; introduces LCBench learning curves. Zimmer et al. 2021.
• FLAML - Adapts learner and hyperparameter search to trial cost and model error, using resource-aware search strategies for tabular AutoML under limited computation budgets. Wang et al. 2021.
• GAMA - Provides modular pipeline-search and post-processing components with logging and visualization, supporting controlled comparisons between AutoML strategies. Gijsbers and Vanschoren 2021.
• VolcanoML - Decomposes large pipeline-search spaces into smaller searches and composes them into execution plans, separating search-space structure from optimization execution. Li et al. 2021.

Pretrained models and agent-driven automation

• TabPFN - Pretrains on synthetic prediction tasks, then conditions predictions on a labeled table. Amortizes learning across datasets; the paper evaluates small tables up to 10,000 rows, not unrestricted tabular prediction. Hollmann et al. 2025.
• MLZero - Uses data-inspection agents, library knowledge and execution history to generate and revise ML pipelines. Evaluated on MLE-bench Lite and an author-created multimodal benchmark. Fang et al. 2025; code.

Neural architecture search

• Reinforcement-learning NAS - Trains a recurrent controller to generate network descriptions, using candidate validation accuracy as a reinforcement-learning reward. Zoph and Le 2017.
• ENAS - Searches for subgraphs of a shared computation graph. Parameter sharing amortizes candidate training, while a policy-gradient controller selects architectures using validation rewards. Pham et al. 2018.
• Regularized evolution - Uses tournament-based architecture evolution with an age preference for younger candidates; studies search efficiency through controlled comparisons with reinforcement learning. Real et al. 2019.
• DARTS - Continuously relaxes discrete architecture choices so gradients can optimize the search representation, replacing evolutionary or reinforcement-learning search over discrete candidates. Liu et al. 2019.
• Petridish - Grows networks by adding shortcut connections, motivated by forward stage-wise regression and gradient boosting; jointly trains candidate connections before selecting a subset. Hu et al. 2019.
• MnasNet - Searches mobile architectures using both accuracy and measured device latency in its objective; hardware measurements replace operation counts as the deployment-cost signal. Tan et al. 2019.
• AgEBO-Tabular - Combines aging evolution for architectures with asynchronous Bayesian tuning of data-parallel training settings, jointly adapting network structure and training configuration. Egele et al. 2021.
• Zero-cost NAS proxies - Ranks candidate architectures using inexpensive initialization-time scores, including pruning-inspired proxies computed from one minibatch; proxy rankings still require validation against trained models. Abdelfattah et al. 2021.
• AIRA-Compose - Agents assemble fixed computational primitives, then scale selected architectures. At 1B parameters and 37.5B tokens, AIRAformer-D improves over Composer; attention-heavy winners differ from the best equal-FLOP designs. Three training seeds. Pepe et al. 2026.

Model structure and algorithm discovery

• Compositional kernel search - Searches Gaussian-process kernels built from sums and products of elementary kernels, yielding decompositions of regression functions into interpretable components. Duvenaud et al. 2013.
• Automatic statistician - Searches compositional Gaussian-process regression models and generates figures and natural-language descriptions of trends, periodicity, and changepoints. Lloyd et al. 2014.
• AutoML-Zero - Evolves learning programs from basic mathematical operations, searching model structure and learning rules together; demonstrations use constrained tasks rather than unrestricted autonomous research. Real et al. 2020.

Evaluation

• NAS-Bench-101 - Provides repeated evaluations of a fixed convolutional architecture space on CIFAR-10, allowing reproducible search comparisons through table queries rather than retraining each candidate. Ying et al. 2019.
• Random-search NAS baselines - Li and Talwalkar find early-stopped random search competitive with ENAS on PTB and CIFAR-10. Weight sharing improves their baseline further; comparisons need repeated runs, matched training recipes and explicit search cost. Li and Talwalkar 2019.
• NAS-Bench-201 - Records training results for all 15,625 cells in a fixed search space on three datasets. Enables controlled comparisons of search algorithms; ranking performance within this space does not establish superiority across architectures or training recipes. Dong and Yang 2020.
• AMLB - Benchmarks AutoML systems across classification and regression tasks, measuring predictive quality, inference time, and failures; task subsets can produce different framework rankings. Gijsbers et al. 2024.
• Budget-sensitive AutoML evaluation - Across 104 tasks, framework rankings remain fairly stable down to five-minute budgets. Early stopping has less consistent effects; implementations stop individual models, search generations or ensembles. Jurado et al. 2025.
• MLE-bench - Benchmarks agents on ML competitions requiring data preparation, model training and experimental iteration; examines resource limits, pretraining contamination and grading against competition metrics. Chan et al. 2025.
• TabArena - Curates IID tabular tasks and compares defaults, tuning and ensembles. The best deep models match or beat boosted trees with tuning and ensembling at larger budgets; foundation models cover subsets in this paper snapshot. Erickson et al. 2025.
• CARPS - Connects optimizers to black-box, multi-fidelity and multi-objective benchmarks through one interface. Selects representative task subsets to reduce evaluation cost; records trajectories for analysis. Benjamins et al. 2025; code.
• BeyondArena - Across 142 datasets, default tabular foundation models lead on small IID tasks; tuned, ensembled conventional models lead on grouped/temporal tasks. Above 100k training rows, missing TabPFN/TabDPT scores use a random-forest fallback. Purucker et al. 2026.
• Test-time compute for tabular models - On TabArena, greedy selection improves TabPFN-3 while uniform averaging of the same 96 configurations worsens error. Selection gains shrink when fewer out-of-fold labels guide the ensemble. Ning et al. 2026.

Software

Optimization and experiment software

• Hyperopt - Describes search-space specification, sequential model-based optimization, and parallel execution in Python, with analysis of completed tuning trials. Bergstra et al. 2013.
• Optuna - Defines conditional search spaces in Python; samplers propose parameters and pruners stop unpromising trials. Supports parallel studies and inspection of trial histories. Akiba et al. 2019; docs.
• SMAC3 - Bayesian optimization for conditional configuration spaces, multiple fidelities, objectives and problem instances. Provides ask-and-tell and warm-start interfaces. Lindauer et al. 2022; docs.
• Ray Tune - Separates training scripts from model-selection algorithms through a common interface, supporting distributed search and scheduling across compute clusters. Liaw et al. 2018.
• Katib - Presents a framework-agnostic tuning service designed around scalability, fault tolerance, extensibility, and multiple users in cloud-native deployments. George et al. 2020.
• Ax - Adaptive experimentation platform for defining, managing, and running optimization studies, including field experiments. docs.
• BoTorch - PyTorch library for Bayesian optimization research; integrates with Ax for experiment management and data handling. docs.
• KerasTuner - Defines hyperparameters during model construction and searches them with Bayesian optimization, Hyperband, or random search. docs.

Pipeline and domain software

• TPOT - Optimizes machine-learning pipelines through genetic programming; its graph-based rewrite supports flexible search spaces and multiple optimization objectives. docs.
• AutoKeras - Keras-based AutoML interfaces for fitting models from data, with task-specific estimators and architecture search. docs.
• H2O AutoML - Automates supervised model training and tuning in H2O, including ensembles of candidate models. docs.
• AutoGBT - Automatically tunes gradient-boosted trees for streaming classification under concept drift; developed for the lifelong AutoML challenge. code.
• Ludwig - Declarative framework for training and fine-tuning models from configuration files, including language, multimodal, and tabular models. code.
• AutoGluon - Pipeline interfaces for tabular, time-series and multimodal prediction. Tabular presets combine conventional learners and pretrained models; model choices, hardware needs and checkpoint licenses vary by preset. docs.
• FLAML - Searches learners and configurations under time or trial budgets, with optional stacked ensembles. Supports prediction-time and custom metric constraints; search cost and deployment constraints are separate controls. docs.

Archived and legacy implementations

• NNI - Toolkit for hyperparameter search, neural architecture search and model compression. Microsoft archived the repository in September 2024; preserved documentation describes the archived implementation. code.
• AdaNet - TensorFlow framework combining neural-subnetwork growth with ensemble learning. The repository was archived in April 2026; the method remains a research reference, while this implementation is read-only. code.
• HpBandSter - Distributed Hyperband/BOHB implementation. Maintainers explicitly state that it is no longer maintained and point to SMAC3 and DEHB as successors; retain for historical reproduction. code.

Related topics

• Optimization
• Neural Networks
• Ensemble Learning
• Feature Selection