Learning general representations from unlabeled data
Self-supervised learning (SSL) is a paradigm in which AI models leverage large volumes of unlabeled data by learning directly from the inherent structure of the data itself. Rather than depending on human annotations, models solve auxiliary tasks automatically generated that allow them to acquire general and transferable representations.
This approach is fundamental to the development of Foundation Models and constitutes a key piece in the evolution of General Purpose AI Systems (GPAIS). The key is that the learned representations are not task-specific, but rather capture reusable semantic structure in multiple contexts with minimal subsequent supervision.
SSL methods are organized around different strategies for generating the supervision signal directly from the data
Models learn by bringing similar views of the same data close in vector space (positive pairs) and pushing apart views from different data (negative pairs). One of the most influential SSL paradigms and the core of models like SimCLR, MoCo or DINOv2.
Models learn to reconstruct parts that are deleted or masked from a sample. Inspired by BERT, this technique does not require defining data augmentations and is very effective for learning rich representations in both image and text.
Applying SSL to tabular data is an open challenge given its heterogeneous nature: mix of numeric, categorical, and ordinal variables. The representations obtained are key for knowledge fusion and cross-domain generalization.
In the group we work on two complementary SSL lines, each oriented to a data type where traditional supervised methods find their limits
We apply Contrastive Learning to problems where labeled data is scarce and variability between subjects is high. The central idea: two distinct augmentations of the same signal should produce close representations, while signals from different sources should be separated in the latent space.
Our main focus is EEG signals, where we have developed frequency-aware contrastive loss functions that incorporate knowledge of brain frequency bands (delta, theta, alpha, beta, gamma) to learn more transferable representations in epileptic seizure detection.
Most advances in SSL have occurred in image and text. Applying it to tabular data —present in industry, health, and finance— remains an open challenge due to its heterogeneous nature: columns with very different types, scales, and semantics.
Models like TabPFN have demonstrated the potential of tabular foundation models, but they require labeled data. Our research seeks to obtain general purpose representations that serve as a basis for tasks like knowledge fusion, data reduction, and adaptation with minimal supervision.
Contrastive learning is structured around the construction and comparison of pairs, guided by a specific loss function
Two views of the same example obtained through transformations or data augmentations. The model learns to produce close representations, capturing invariance to irrelevant changes for the task.
Views from different examples. The model learns to separate them in vector space, forcing representations to capture semantically significant differences between instances.
The most commonly used loss function maximizes similarity between positive pairs relative to negative ones. Variants such as NT-Xent and the frequency-aware loss for EEG extend this framework to specific domains.
In both biomedical signals and tabular data, SSL faces a series of open challenges that define the frontiers of research
Negatives that are too easy provide no learning signal; false negatives —semantically similar instances treated as distinct— degrade representations. Defining appropriate augmentations for temporal signals or EEG requires deep domain knowledge.
Unlike images or text, tabular data mix numeric, categorical, and ordinal variables with very different semantics across columns. Designing effective self-supervised objectives in this context remains an open problem.
Without careful selection of negatives or with inadequate architectures, the model can collapse and assign the same representation to all examples, losing all discriminative capacity.
Tabular data are difficult to homogenize for training foundation models at scale, unlike what happens in image or text. The lack of common benchmarks makes it difficult to compare methods.
Self-supervised representations must be sufficiently general to transfer across patients, facilities, or datasets. Domain shift is one of the main obstacles for clinical and industrial deployment.
Contrastive Learning requires large batches or extensive queues of negatives. Adapting these frameworks to federated environments —where clinical data cannot be centralized— adds additional memory and communication constraints.
In the GPAIS group at the IAFER research chair, we are leading research in SSL applied to two domains where labeled data is scarce and generalization is critical: biomedical signals and heterogeneous tabular data.
SSL has a transversal impact in domains where unlabeled data abound but annotations are expensive, scarce, or inaccessible: