Synthetic data-augmented explainable Vision Transformer for colorectal cancer diagnosis via surface tactile imaging

In this work, we present a synthetic data augmented explainable Vision Transformer (ViT) framework designed for the informed and intuitive early diagnosis of colorectal cancer (CRC) polyps. The framework uses textural images — generated by our recently developed vision-based tactile sensor (called HySenSe) and augmented by synthetically generated images from a diffusion model pipeline, to output class-based probabilities of potential CRC polyp types. Additionally, it provides local relevancy-based heatmaps to assist clinicians by highlighting key areas of interest in the tactile images representing CRC polyp textures. We benchmark each aspect of this framework through: (i) Inception Scores for the synthetic images generated by the diffusion pipeline, (ii) Performance evaluation and sensitivity analyses on the effects of synthetic data addition on model generalizability compared with other state-of-the-art architectures, (iii) Dimensionality reduction techniques to confirm the suitability of synthetically generated images, and (iv) Comparison of two independent approaches visualizing explainability.