Data Engineer

About

We are a proprietary quantitative trading firm across equities, derivatives, and crypto. This role owns the end-to-end data layer—from ingestion to delivery—ensuring all research and trading systems run on clean, reliable, and reproducible data. This is a core engineering role, responsible for the firm's data backbone, not analytics or dashboards.

The Role

You will be responsible for building and maintaining the data backbone of a quantitative trading system, ensuring that all downstream research and execution rely on high-quality, consistent data. This is not a cloud ETL, BI, dashboarding, Excel reporting, strategy development, or notebook-only role.

Responsibilities

• Build and maintain data pipelines integrating data from broker APIs, exchanges, third party providers, and internal datasets, ensuring reliability and scheduling.
• Clean and standardize data, handling corporate actions, symbol mapping, missing/inconsistent time series, and reconciling multiple sources into unified datasets with consistent schemas.
• Design and manage data storage architecture using Parquet, PostgreSQL, ClickHouse, and CSV with efficient partitioning, indexing and querying.
• Develop data quality and validation systems to detect anomalies, missing data, and duplicates; ensure versioned and reproducible datasets.
• Build and maintain an internal data access layer and unified data layer for consistent retrieval of live and historical data with consistent schemas so that researchers can access data without directly querying databases for seamless strategy execution ensuring alignment between real-time and back testing datasets.
• Manage changes in upstream data sources (APIs, websites, schemas) to ensure pipeline continuity and minimal disruption.
• Ensure pipeline reliability on Linux systems with job scheduling, monitoring, logging, debugging failures, and maintaining fault-tolerant, observable, and recoverable systems.
• Enable robust data backfilling and reprocessing for regenerating datasets when source data or processing logic changes.
• Guide and review work of data analyst interns, ensuring structured, consistent, and integrated data outputs.
• Work closely with quant department to streamline data access, troubleshoot issues, and improve data reliability, enabling faster and more accurate strategy development.

Requirements

• Bachelor's or Master's in computer science, engineering, or related field with 3 – 8 years of hands-on experience in Data Engineering or Data Infrastructure Roles.
• Strong hands-on Python expertise with pandas, NumPy, requests/httpx, file handling, logging, exception handling, retries, and rate-limit handling.
• Proven experience building and maintaining production-grade data pipelines, especially for time-series and messy real-world data (missing values, schema inconsistencies).
• Hands-on experience with PostgreSQL, ClickHouse (or similar OLAP systems), and Parquet, along with building robust data ingestion pipelines via APIs and webscraping.
• Comfortable in Linux environments, with the ability to debug pipelines, manage logs, and handle scheduling (cron/systemd).
• Clear understanding of live vs historical data systems, including latency, consistency, and alignment across both environments.
• Proven ability to design reliable, reproducible, and high-quality data systems, with experience of enforcing data standards and mentoring junior team members.
• Knowledge of Indian/US Stock Markets is beneficial but not necessary.

Nice to Have

Experience with PySpark, Kafka, Databricks, Snowflake, BigQuery, AWS Glue, or Azure Data Factory is optional only, not the main requirement.

Position Details

Location: Ahmedabad (on-site)
Type: Full-time
Experience: 3 – 8 years