In life sciences, giant amounts of raw data are accumulated. At the same time, advanced data science techniques like deep learning are getting more accessible. In general, supervised learning methods are capable to handle, process and ultimately draw meaningful information from these large-scale datasets. However, the available biomedical data is mainly unstructured and lacks large-scale, high-quality, training data that is correctly labeled. This issue impacts nearly all applications of supervised machine learning to the life sciences, from automatic image interpretation to electronic health records to genomics to objective diagnostic or monitoring tools. While the challenges of training complex, high-parameter deep learning models from a limited number of samples is obvious, uncertainty in the labels of those samples can be just as problematic. In this project students developed a structured, formalized framework for a closed-loop machine learning system, which includes expert label collection and label meta information computation (like quality measures). Different (semi-)supervised machine learning approaches that leverage those labels were compared. Ultimately, the system incorporates an active learning enabled feedback loop which partly steers the label collection component. The whole work was evaluated in the use case of clinical research in the field of Parkinsons Disease.