Individuals with a Schizophrenia Spectrum Disorder (SSD) often demonstrate marked deficits in social cognition, namely the ability to understand the emotions and perspectives of others. Social cognition deficits are strongly, and uniquely, correlated with poor functional outcomes, such as illness relapse, disability, and unemployment. At present, there is no effective behavioral or pharmacological treatment for social cognition deficits in SSD. This constitutes a major unmet treatment need, as SSDs are among the top causes of disability in adolescence and adulthood in OECD countries; in Canada alone, the cost to the economy is approximately 10 billion dollars per year. Here, we use data from the large, multi-centre, NIMH-funded ‘Social Processes Initiative in the Neurobiology of the Schizophrenia(s)’ (SPINS) clinical trial, designed to illuminate the brain-basis of social cognition deficits across the SSD spectrum; a needed first step to inform future intervention efforts. This dataset is unique for its size (N=480; 300 SSD and 180 healthy controls (HC), ages (18-55), multi-modal advanced imaging protocol, and the extent of its characterization of social cognition. Specifically, we report findings from the ‘Empathic Accuracy’ (EA) task, in which participants provide real-time valence rating of emotional stimuli (videos of actors recounting positive and negative memories), while undergoing fMRI (4D data, 3D in space and 1D in time). Our objective is to delineate abnormalities in dynamic neural activity associated with impaired performance on the EA task, using an unsupervised learning technique well-suited for psychiatric time series data. Our methods are as follows. First, we isolate brain activity associated with poor EA task performance across patients. Second, we apply contrastive predictive coding algorithm, an unsupervised representation learning technique which can learn representations by predicting the future in latent space by using a powerful autoregressive model using a probabilistic contrastive loss. The loss induces the latent space to capture information that is maximally useful to predict future samples. We also use each participant’s accuracy score on the EA task, the diagnostic class labels of SSD and HC, as an additional supervision to train the model to obtain more refined representations. We initally use a strided convolutional network as an encoder, genc in our experiment. We then use a GRU RNN for the autoregressive part of the model, gar. The output of the GRU at every timestep is used as the contextfrom which we predict 6 timesteps in the future using the contrastive loss. The described research applies an unsupervised representation learning technique to time series data from an emotional fMRI task, in a large SSD and HC population. Our goal is to discover meaningful bio-markers that underlie social cognitive impairment in SSD.