DBIS

AutoML refers to the automation of machine learning (ML) pipelines, including preprocessing, hyperparameter optimization, model selection, and more, to maximize accuracy and performance. AutoML approaches enable users to train models that perform reasonably well without requiring deep knowledge of ML methods or extensive manual experimentation.
Recently, AutoML has been extended through the introduction of LLM-based agents. These agents refine user input to coordinate individual pipeline steps and optimize results, replacing the need for domain experts. However, existing Agent AutoML solutions primarily focus on large-scale models with high computational requirements, often making them inaccessible due to financial or hardware constraints. In contrast, smaller, open-source models can run locally, increasing accessibility for users with limited resources. Additionally, domain-specific models (e.g., private classification models in health sciences) may already exist and perform adequately for common subtasks, potentially reducing the need for training a new model from scratch for a given task.
Agent frameworks such as AutoGen and MetaGPT already provide the ability to include non LLM or ML related code, while Tools like HuggingFace AutoTrain already support finetuning a pre-existing model from their ecosystem. AutoML methods in general have also employed Transfer Learning to finetune existing models. However, the combination of AutoML with Agents that consider local pre-existing models has not yet been explored. Agents need to be aware of available methods by characterizing them and making them comparable using an ontology, such as ML-Schema.

The subject of this thesis is twofold: First to use small large language models in an agent framework for AutoML and second to integrate pre-existing classification models into the LLM decision making using a Machine Learning ontology.  The resulting framework should be evaluated against the optimal version of the pre-existing models for classification tasks of popular datasets.

Goals & Objectives:

• Extending an Auto-ML or Multi-Agent framework with the capability to draw from (local) pre-existing classification models mapped onto categories from a pre-existing ML ontology, e.g. via GraphRAG
• Using small open-source LLMs for classification tasks within the framework and evaluate their performance against the models’ own evaluation datasets and standard benchmark datasets.
• Analyze the advantages and limitations of this approach compared to existing AutoML solutions.

Challenges:
While the actual extension to pre-existing models may be straightforward, to allow for flexible addition and correct use by the Agents can be difficult: If an Agent does not have enough knowledge about an approach it may use it wrongly or not at all.

Related Literature:

• https://arxiv.org/abs/2410.02958
• https://arxiv.org/abs/2410.17238
• https://arxiv.org/abs/2412.14222
• https://arxiv.org/abs/2309.01125