The U.S. Pharmacopeial Convention (USP) is seeking a Senior Data Engineer to design, build, and maintain scalable, production-grade data pipelines across our AWS-based data platform. As a key member of our core data platform team, you will contribute to the continued evolution of our AWS Lakehouse architecture and help shape best practices.
Requirements
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent combination of education and experience.
- Minimum 7+ years of data engineering experience, with strong expertise in AWS services and hands-on work with big data and distributed data technologies in the AWS cloud environment.
- Strong proficiency in Python and SQL with experience in PySpark/Spark.
- Hands-on experience building and operating data pipelines on AWS (Glue, Step Functions, Lambda, S3, Athena, SNS, SQS, Redshift etc.).
- Demonstrated experience with event-driven pipeline architectures.
- Experience ingesting and processing vendor-supplied data files on a scale.
- Practical experience implementing Medallion (Bronze/Silver/Gold) architecture in a data lake or Lakehouse environment.
- Experience with Terraform for cloud infrastructure provisioning.
- Experience with CI/CD tooling (e.g., Bitbucket, GitHub, AWS CodePipeline) for data pipeline deployments.
- Strong grasp of data warehousing concepts (star schema, dimensional modeling, SCD).
- Experience in integrating data with Microsoft Power BI or similar BI tools.
- Experience with UNIX/Linux including basic commands and shell scripting.
- Production experience with monitoring, troubleshooting, and on-call support for data pipelines.
- Experience with Agile engineering practices.
- Hands-on integration experience with Oracle EBS as a data source.
- Experience with data quality frameworks (e.g., Great Expectations, dbt tests).
- Experience with AWS CDK or CloudFormation in addition to Terraform.
- Knowledge of data catalog and data lineage tools (ex. Alation, Collibra).
- AWS certifications (e.g., AWS Certified Data Engineer, AWS Certified Solutions Architect).
Benefits
- Company-paid time off
- Comprehensive healthcare options
- Retirement savings
