Senior Data/ML Engineer - Databricks Forecasting Platform Remote

We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This role focuses on modernizing a Databricks-based system, reducing technical debt, and enhancing testing, observability, and reliability.

Location: Remote (North America preferred, open to global remote with North American hours)

Responsibilities

• Review the current forecasting platform architecture and identify areas for improvement
• Refactor existing Databricks, Python, and PySpark implementations
• Move business logic out of Databricks notebooks and into reusable Python modules or packages
• Improve separation of concerns between orchestration and core business logic
• Establish stronger engineering standards and define quality benchmarks for the platform
• Implement or improve automated testing practices and validation mechanisms for forecasting workflows
• Build or improve monitoring and observability, increasing visibility into how predictions are generated
• Help monitor model behavior and operational health over time
• Improve reliability of scheduled training workflows, reducing manual intervention on failure
• Improve failure handling, retries, and overall workflow resilience
• Maintain and extend existing forecasting capabilities as needed

Requirements

• Strong professional experience with Databricks, including workspaces, notebooks, scheduled workflows, and CI/CD processes
• Strong Python engineering experience, including designing reusable modules or packages
• Strong PySpark experience with production data pipelines or distributed data processing
• Experience refactoring production code and improving maintainability
• Familiarity with time series forecasting concepts and workflows
• Ability to understand and work effectively within an existing, unfamiliar codebase
• Experience improving software quality, testing strategy, and engineering standards
• Experience implementing automated testing practices
• Experience improving monitoring, observability, or operational visibility for production systems
• Strong judgment around technical debt, refactoring priorities, and maintainable architecture
• Ability to work with existing systems rather than only building from scratch