Job Description
One of our large clients in the energy industry is seeking a hands on Senior Databricks Engineer to support a large-scale modernization initiative focused on migrating complex pricing, forecasting, and trading models to Databricks. This role will primarily support the Pricing workstream, which is the most complex and business-critical component of the program. The consultant will partner closely with technical leads, data engineers, and business stakeholders to translate existing Python and PySpark pricing models into scalable, production-ready Databricks solutions.
This individual must be highly technical and execution-focused, with proven experience building and deploying enterprise Databricks solutions rather than simply designing architectures or providing theoretical guidance. Success in this role requires strong collaboration skills, as the engineer will work directly with pricing model owners and business users to gather requirements, validate model outputs, and ensure successful migration and adoption.
Key Responsibilities:
• Lead the migration of existing Python and PySpark pricing models into the Databricks platform.
• Design, develop, test, and deploy scalable Databricks solutions supporting pricing analytics and business operations.
• Build and maintain Databricks notebooks and collaborative development workflows.
• Develop and optimize large-scale data ingestion pipelines from multiple enterprise systems.
• Translate legacy pricing logic into modern cloud-native Databricks implementations while maintaining business accuracy.
• Configure and maintain connectivity between Databricks and Azure SQL Server / Azure SQL Managed Instance environments.
• Support data governance and cataloging efforts using Unity Catalog.
• Create reusable Python libraries and package them as wheel files for deployment across Databricks environments.
• Implement job orchestration, workflow automation, and scheduled processing within Databricks.
• Utilize Databricks Repos and CI/CD pipelines to support source control, testing, and deployment practices.
• Participate in code reviews and promote engineering best practices across the development team.
• Partner directly with business stakeholders to gather requirements, document functionality, and validate migrated pricing models.
• Troubleshoot production issues and optimize performance of data pipelines and model execution.
• Collaborate with architects, technical leads, and business teams to ensure successful project delivery.
• Contribute to the overall cloud modernization strategy supporting Pricing, Forecasting, and Trading capabilities.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Required Skills & Experience
• 5+ years of experience developing data and analytics solutions using Python.
• 3+ years of hands-on Databricks implementation experience in enterprise environments.
• Strong expertise with PySpark and distributed data processing.
• Experience building production Databricks notebooks and workflows.
• Hands-on experience with Databricks job scheduling, orchestration, and automation.
• Experience configuring and managing Unity Catalog.
• Strong experience connecting Databricks to Azure SQL Server and Azure SQL Managed Instance data sources.
• Experience packaging, publishing, and consuming reusable Python wheel packages.
• Experience leveraging Databricks Repos, Git integration, and CI/CD deployment practices.
• Strong understanding of data engineering best practices, ETL/ELT design, and cloud data architectures.
• Proven experience migrating legacy applications, analytics platforms, or data workloads into Databricks.
• Excellent communication and stakeholder management skills.
• Experience collaborating directly with business users to gather requirements and perform solution validation.
Nice to Have Skills & Experience
• Experience with Azure Synapse Analytics.
• Experience within energy, utilities, commodity trading, retail energy, or pricing organizations.
• Knowledge of forecasting, pricing, risk management, or trading analytics platforms.
• Experience supporting large-scale cloud transformation initiatives.
• Familiarity with Azure Data Factory and other Azure data services.
• Understanding of DevOps methodologies and Infrastructure as Code practices.
Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401k retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.