- Design, construct, and optimize ETL pipelines utilizing AWS Glue 3.0+ and PySpark.
- Implement scalable and safe knowledge lakes utilizing Amazon S3, following bronze/silver/gold zoning.
- Write performant SQL utilizing AWS Athena (Presto) with CTEs, window features, and aggregations.
- Take full possession from ingestion → transformation → validation → metadata → documentation → dashboard-ready output.
- Construct pipelines that aren’t simply performant, however audit-ready and metadata-rich from the primary model.
- Combine classification tags and possession metadata into all columns utilizing AWS Glue Catalog tagging conventions.
- Guarantee no pipeline strikes to QA or BI crew with out validation logs and field-level metadata accomplished.
- Develop job orchestration workflows utilizing AWS Step Capabilities built-in with EventBridge or CloudWatch.
- Handle schemas and metadata utilizing AWS Glue Knowledge Catalog.
- Take full possession from ingestion → transformation → validation → metadata → documentation → dashboard-ready output.
- Guarantee no pipeline strikes to QA or BI crew with out validation logs and field-level metadata accomplished.
- Implement knowledge high quality utilizing Nice Expectations, with checks for null %, ranges, and referential guidelines.
- Guarantee knowledge lineage with OpenMetadata or Amundsen and add metadata classifications (e.g., PII, KPIs).
- Collaborate with knowledge scientists on ML pipelines, dealing with JSON/Parquet I/O and have engineering.
- Should perceive tips on how to put together flattened, filterable datasets for BI instruments like Sigma, Energy BI, or Tableau.
- Interpret enterprise metrics corresponding to forecasted income, margin traits, occupancy/utilization, and volatility.
- Work with consultants, QA, and enterprise groups to finalize KPIs and logic.
- Construct pipelines that aren’t simply performant, however audit-ready and metadata-rich from the primary model.
- Combine classification tags and possession metadata into all columns utilizing AWS Glue Catalog tagging conventions.
- This isn’t only a coding function. We count on the candidate to suppose like a knowledge architect inside their module – designing pipelines that scale, deal with exceptions, and align to evolving KPIs.
- Advertisement -