Idexcel technologies - data engineer - etl/pyspark
BangaloreIdexcel Technologies Private Limited
Description : Databricks (Spark) : - Develop scalable ETL/ELT pipelines using PySpark (RDD/DataFrame APIs), Delta Lake, Auto Loader (cloudFiles), and Structured Streaming.- Optimize jobs : partitioning, bucketing, Z-Ordering, OPTIMIZE + VACUUM, broadcast joins, AQE, checkpointing.- Manage Unity Catalog : catalogs/schemas/tables, data lineage, [...]
Category IT & Telecommunications