View all jobs
About Authority Partners:
Authority Partners is a leading and global IT services company with over 26 years of experience, serving clients within the financial, healthcare, insurance, mortgage, and technology sectors among many others. We are committed to building relationships with IT professionals who possess integrity, dedication, teamwork, and the desire to learn and grow with us. We specialize in placement for long-term contracts and permanent opportunities and helping our consultants achieve their career success.
Authority Partners is an equal opportunity employer and is committed to providing a workplace that is free from all types of discrimination, as well as from abusive or offensive behavior and harassment.
Senior Databricks Engineer
Remote, AnywhereAuthority Partners is seeking an experienced Data Engineer to join our team. You will design, expand, and optimize data pipelines and architectures to support cross-functional teams. The ideal candidate is skilled in building and enhancing data systems, supporting developers, analysts, and scientists, and ensuring seamless data delivery across projects.
Up for the challenge? Then read on and apply!
Responsibilities:
Up for the challenge? Then read on and apply!
Responsibilities:
-
Design, build, and maintain scalable data pipelines for ingesting, transforming, and delivering structured and unstructured data.
-
Collaborate with stakeholders in an Agile environment to gather requirements and design data solutions and technology roadmaps aligned with business goals.
-
Define and implement best practices for data quality, governance, and security within Azure Databricks.
-
Provide expertise on advanced concepts including medallion architecture, Delta Live Tables (DLT), and Unity Catalog.
-
Lead and participate in code reviews, documentation, and knowledge-sharing sessions.
-
Identify opportunities to add value through effective information management.
-
Build relationships with key stakeholders responsible for information and performance management.
What you have:
-
7+ years of experience in enterprise data engineering, designing, and building scalable data platforms.
-
Strong hands-on experience with Azure Databricks and PySpark for developing and optimizing data pipelines and platforms.
-
Extensive experience with Azure Databricks fundamentals and architecture, including cluster configuration, performance tuning, and optimization of Data Engineering and SQL Warehouse modules.
-
Experience building scalable solutions for data migration, batch processing, and real-time data ingestion.
-
Strong expertise in architecting big data solutions using Data Lakes, Data Warehouses, and Data Products.
You stand out if you have:
-
Experience or knowledge of integrating Databricks with Collibra for metadata ingestion, lineage propagation, and data quality rule mapping.
-
Client management skills
Education:
-
Bachelor’s degree in Computer Science or a related field, or equivalent education and experience.
About Authority Partners:
Authority Partners is a leading and global IT services company with over 26 years of experience, serving clients within the financial, healthcare, insurance, mortgage, and technology sectors among many others. We are committed to building relationships with IT professionals who possess integrity, dedication, teamwork, and the desire to learn and grow with us. We specialize in placement for long-term contracts and permanent opportunities and helping our consultants achieve their career success.
Authority Partners is an equal opportunity employer and is committed to providing a workplace that is free from all types of discrimination, as well as from abusive or offensive behavior and harassment.