In various scenarios, such as providing daily assistance to the elderly, aiding individuals recovering from severe injuries, or supporting commercial and recycling tasks, one can envision a robotic assistant handling garments by fetching, folding, and organizing clothes. In this project, we will focus on a key technical prerequisite for realizing this challenging scenario: advancing current computer vision techniques to enhance understanding and reasoning about the garment being manipulated. Specifically, we willaddressthe challenging tasks of recognizing a garment and its semantic parts in the visual content. The deep semantic understanding of garments is particularly challenging as garment objects depict complex topological and physical behaviors, involving strong self-occlusions and deformations of fabrics.

Recent successful image segmentation techniques in computer vision, such as SAM (Segment Anything Model) do not easily generalize to the detection of semantic parts, since the boundary assumption in object detection does not apply in semantic part detection where the presence of explicit boundaries is only optional. While its adaptation through prompting to specific domains like medical imagery have been demonstrated, SAM has shown exceptional proficiency only in certain objects and modalities, exhibiting inadequacy or ineffectiveness in other contexts. Advancing beyond FCN (Fully Convolutional Network) based per-pixel labeling, recent developments have introduced specialized context modules and various self-attention mechanisms, enabling the incorporation of contextual information. However, these models may not be particularly beneficial in our targeted settings, where the context is often limited. Methods specially developed for garment semantics mostly deal with 2D static images depicting clothes close to their canonical states seen by clear views, and focus on the segmentation of individual clothes rather than their semantic parts.