Reporting to the Manager, BI Development, the Data Engineer will be responsible for integrating enterprise wide data from Cofense’s Cyber Security based Applications, Micro-services and other disparate data sources. Essential Duties/Responsibilities include designing Data Platform and Architecture, developing Data Pipelines on Cloud Technologies, and securing data end-to-end by complying data privacy rules.
Requirements
- Substantial experience with SQL and no-SQL OLTP databases and OLAP data warehousing technologies, especially AWS Aurora
- Experience data modeling and building data pipelines for multi-product and/or multi-department organizations
- Develop Data Pipelines on Cloud Technologies like Azure/AWS with well-defined tool frameworks
- Able to develop ETL code to stream data from disparate (structured and semi-structured) SaaS product data stores to Data Lake/Data Warehouse using Python, Azure/AWS Data Lake services
- Ability to write complex SQL scripts and automate them using Python
- Develop test cases and unit tests for key implementations of Data Platform by adhering to software engineering best practices and standards
- Secure data end-to-end by complying data privacy rules while developing processes to move data across Applications/Data Sources and Data Lake/Data Warehouse, as well as while delivering data through SQL clients and BI tools.
- Experience building adhoc reporting APIs and service layer on top of underlying OLTP and OLAP databases
- Help integrate Data Platform with BI tools like Power BI, Tableau, Splunk etc.
- Ability to develop and interpret Entity Relationship Diagrams (ERD) across data sets in relational database systems as well as non-relational Data stores
- Able to do Data Mining and Identify trends, patterns, anomalies in complex data sets across multiple data sources/systems and present results without ambiguity
- Develop data transformations to generate Facts, Summaries, Key metrics by applying business rulesets and aggregations using Python, SQL and other transformation tools
- Able to review current processes related to data ingestion, transformation and statistical analysis and re-engineer them
- Collaborate with business users across Cofense’s departments in defining requirements, prioritize project work and deliver them timely
- Expertise in SQL skills for data transformations, statistical analysis, and troubleshooting across more than one Database Platforms (MySQL, PostgreSQL, Redshift, Azure SQL Warehouse etc.)
- Expert in writing complex SQL scripts and automate them using Python
- Knowledge of Data management on NoSQL DBs like DynamoDB, Mongo, and know-how of Big Data tools Hadoop, Spark, Kafka/Kinesis/SQS/Azure Queues or other messaging tools is huge plus
- Analytical skills, with good at finding data trends/outliers, anomalies, and articulate complex information or data points with Business Users, Management, and individuals
- Enthusiasm to work with lot of data across disparate data sources and Databases
- Has strong sense of engineering craftsmanship, takes pride in the code they write.
- Has a sense of intellectual curiosity and a burning desire to learn is self-driven, actively looks for ways to contribute, and knows how to get things done
- Is deliriously customer-focused both internal and external customers
- Sees big picture impact and relationships among and across work units
- Identifies complex technical problems and tries to resolve with minimal help
Benefits
- 401k Matching
- Retirement Plan
- Tuition Reimbursement
