I'm a Data Scientist and Data Engineer with a background in Physics, working with large-scale data processing, distributed computing, Python-based data pipelines, and quantitative data analysis.
I currently work at LIneA, where I design and develop scalable workflows for complex scientific datasets. My work involves data ingestion, transformation, integration, validation, quality assurance, exploratory analysis, crossmatching, deduplication, and distributed execution across interactive Jupyter environments and multi-node HPC clusters.
I work primarily with Python, SQL, Dask, Jupyter, SLURM, and HPC systems, with a strong focus on scalability, performance, memory efficiency, data quality, reproducibility, testing, and maintainable software.
My current work is applied to large-scale astronomy, including projects connected to the Rubin Observatory / LSST ecosystem and international scientific collaborations, but the underlying challenges are broadly applicable to data science and data engineering.
- Designing scalable Python data pipelines
- Processing and analyzing datasets ranging from millions to hundreds of millions of records
- Performing exploratory and quantitative analysis using distributions, summary statistics, percentiles, and large-scale visualization
- Building distributed workflows with Dask and SLURM
- Integrating and standardizing heterogeneous data sources
- Developing data validation, quality assurance, profiling, and deduplication workflows
- Building Python packages, command-line tools, automated tests, and CI workflows
- Contributing to open-source scientific software
A distributed Python pipeline for generating large-scale HiPS catalogs using Dask, LSDB, and HPC infrastructure.
The project supports configurable selection strategies, YAML-based configuration, command-line execution, parallel processing, automated validation, and reproducible catalog generation. It is also published on PyPI.
A distributed pipeline for integrating dozens of heterogeneous datasets into consolidated data products.
The workflow includes schema harmonization, quality metadata standardization, scalable spatial crossmatching, deterministic deduplication, validation, and memory-efficient distributed processing with Python, Dask, LSDB, HATS, and HPC infrastructure.
A distributed workflow for interactive analysis and visualization of datasets containing up to 691 million records.
The architecture integrates JupyterLab with a multi-node HPC cluster using Dask and SLURMCluster, combining distributed computation with HoloViews, Bokeh, and Datashader for responsive exploration of datasets that exceed single-machine memory.
I also taught a short course on this workflow and published the training materials openly.
Designed and implemented the .concat API in the open-source LSDB library, enabling concatenation of distributed catalogs stored in the HATS format.
The contribution included feature design, implementation, integration with the existing catalog architecture, edge-case handling, and development of a comprehensive unit-test suite to validate correctness and reliability.
Data Science & Analysis
Python · Pandas · Jupyter · Exploratory Data Analysis · Descriptive Statistics · Data Visualization
Data Engineering
SQL · Parquet · Data Pipelines · Data Integration · Data Quality · Data Validation
Distributed Computing & HPC
Dask · SLURM · Distributed Computing · Parallel Computing · HPC
Scientific & Large-Scale Data
LSDB · HATS · HoloViews · Bokeh · Datashader
Software Engineering
Git · GitHub · Testing · CI · CLI Development · Configuration-Driven Workflows
I'm currently expanding my data engineering background through formal studies in:
- Database systems and SQL
- Data modeling
- Data warehousing
- Cloud data engineering
- AWS
This complements my professional experience with large-scale and distributed data processing.