RAW TO RELIABLEby Lee YaoData Engineer & Analytics Systems Builder
Menu

WORK / 01

RAW TO RELIABLE / WORK / SYSTEMS INDEX

Data Systems & Analytics Work

A portfolio of data platforms, analytics pipelines, orchestration workflows, dimensional models, data-quality systems, and decision-ready outputs.

Each project is organized around the engineering problem, system architecture, data flow, modeling approach, quality controls, implementation decisions, and resulting analytical outputs.

Raw Inputs / Pipeline / Validation / Modeling / Analytics / Decision Support

Projects
05
Featured
03
Current Build
00
Public Repositories
05

PROJECT REGISTRY / DATA ENGINEERING / ANALYTICS ENGINEERING

05 SYSTEMS

  1. 01

    Financial Data Platform

    Completed

    Featured Work

    Tableau SEC Company Financial Performance dashboard showing Amazon annual revenue, net income, profitability margins, assets, liabilities, and debt-to-assets ratio through 2025.
    ANALYTICS OUTPUT

    Final Tableau dashboard using Snowflake financial marts. The screenshot shows one company selection and annual reporting periods.

    SEC Financial Data Platform

    Completed Oct 2026 · 100 CIKs requested

    Real SEC company facts move through Python ingestion, validation, Snowflake and dbt marts into anomaly flags and a Tableau dashboard; a Kafka and Spark stream processes filing events.

    Engineering Problem

    SEC financial records arrive through raw API responses with inconsistent tags, reporting periods, statement types, and filing structures. The project organizes those records into validated, analytics-ready financial datasets and supports anomaly monitoring across company metrics.

    Focus

    • Batch ingestion
    • Financial-data normalization
    • Statement classification
    • Data quality
    • dbt financial marts
    • Rule-based anomaly detection
    • Filing-event streaming

    Technologies

    PythonAWS S3 (historical)KafkaSpark Structured StreamingSnowflakedbtAirflowGreat ExpectationsTableauSQLDocker
    View RepositoryView Case Study

  2. 02

    Streaming-Oriented Data Pipeline

    Completed

    Featured Work

    CMS Medicare Streaming Data Pipeline

    CMS Public Data API · Hospital inpatient charges

    Kafka and Spark Structured Streaming process public Medicare hospital charge records into Snowflake snapshots and dbt models for DRG charge-to-payment analysis.

    Engineering Problem

    Public provider/DRG records need consistent processing to compare submitted charges with Medicare payment measures. The implemented stream builds DRG aggregates and snapshot-based analytical models; Airflow orchestration is documented separately.

    Focus

    • Kafka ingestion
    • Stateful Spark micro-batches
    • Snowflake aggregate snapshots
    • DRG outlier and severity rules
    • Documented Airflow workflow

    Technologies

    KafkaSpark Structured StreamingApache AirflowSnowflakedbtPythonDockerSQL
    View RepositoryView Case Study

  3. 03

    Analytics Engineering Platform

    Completed

    Featured Work

    Olist Supply Chain & AI Reviews Intelligence

    Mar 2026 – Apr 2026

    An e-commerce analytics project connecting Snowflake and dbt models, Cortex-enriched Portuguese reviews, Power BI reporting, and a separate Neo4j graph investigation.

    Engineering Problem

    The Olist dataset spans orders, customers, sellers, products, payments, reviews, and delivery activity. The project unifies those sources into dimensional models and analytical marts while extending structured reporting with sentiment classification and graph-based relationship analysis.

    Focus

    • Local Prefect flow
    • Order-item modeling
    • Portuguese review enrichment
    • Seller sentiment analysis
    • Graph investigation

    Technologies

    PythonPrefectDockerAWS S3SnowflakedbtPower BISnowflake CortexNeo4jPandasSQL
    View RepositoryView Case Study

  4. 04

    Batch Data Platform

    Completed

    US DOT Flights Data Engineering Pipeline

    US DOT / BTS · Flight On-Time Performance

    Parameterized US DOT flight-data ingestion with Python, AWS S3, Snowflake, dbt dimensional modeling, and Airflow orchestration for carrier performance and delay attribution.

    Engineering Problem

    Periodic BTS flight records need repeatable acquisition, warehouse loading, and consistent analytical grains. Five manually triggered Airflow tasks coordinate ingestion, loading, dbt models, and tests, with stage-specific rerun protections.

    Focus

    • Monthly batch ingestion
    • Airflow orchestration
    • Partitioned S3 / Snowflake loading
    • Dimensional modeling
    • Carrier performance & delay causes
    • Rerun safeguards

    Technologies

    Apache AirflowDocker ComposePythonAWS S3SnowflakedbtSQL
    View RepositoryView Case Study

  5. 05

    Analytics & BI Pipeline

    Completed

    NYC Yellow Taxi Analytics Engineering

    NYC Yellow Taxi / Q1 2019

    An analytics engineering pipeline for NYC Yellow Taxi trip data using DuckDB, PostgreSQL, dbt, dimensional modeling, data-quality flags, and Power BI reporting.

    Engineering Problem

    Historical trip records require explicit timestamp filters, validity rules, geography roles, and analytical grains before operational reporting. The project defines trip-level and daily SQL marts with recorded Power BI outputs.

    Focus

    • DuckDB ingestion
    • dbt transformations
    • Data-quality flags
    • Dimensional modeling
    • Trip and daily grains
    • Weighted metrics
    • Power BI reporting

    Technologies

    DuckDBPostgreSQLdbtDockerPower BISQL
    View RepositoryView Case Study
Return to Homepage