Human-agent interaction (HAI) is an increasingly important research area that investigates how humans interact with intelligent agents. Effective HAI relies heavily on social signal understanding, particularly visual social signals that convey human attention, intention, and affect, enabling agents to perceive, interpret, and respond appropriately to human behaviour. Recent advances in artificial intelligence (AI) have substantially improved visual perception and representation learning, allowing machines to detect and model visual social behavioural cues with increasing accuracy. However, translating these cues into socially meaningful interpretations remains a significant challenge, often resulting in unnatural interactions and limited empathic responsiveness in HAI systems. In this paper, we coined the concept of computational social vision (CSV) spanning the domain of both psychology and computing science to understand social signals. We summarize visual-cognitive computing techniques for this purpose with an emphasis on how social vision and artificial intelligence technologies support the inference of latent social states from dynamic behavioural evidence. To structure the field, we organise representative methods according to the semantic roles of visual behaviours, including both head-centric and body-centric social vision approaches. Finally, we discuss socially interactive embodied systems, analyzing how agents, such as social robots and virtual humans, adaptively perceive their environments and interact with humans as well as the future trend.