In this work, we have introduced a novel end-to-end pipeline that enhances classification by incorporating ROIs generated by a weakly supervised method as an auxiliary task in a deep multitask learning framework. We have demonstrated that our approach outperforms conventional methods that only rely on classification. By using weakly supervised segmentation, we are able to leverage pixel-wise information, which ultimately leads to improved classification performance. Published in MICCAI Workshop Proceedings.