Leaf Analytics Engine
This repository contains Leaf’s shipper-data processing logic. This logic is currently set up as a series of data pipelines conceptually separated into ETL and Adapt:
The ETL pipeline ingests raw shipment data from Leaf’s clients and produces an enhanced (and uniform) version of those shipment data that can be used in Adapt.
The Adapt pipeline identifies business opportunities and generates data assets consumed by downstream business consumers (CX, Engineering).
The pipelines are loosely coupled with and run through Prefect.
Contents:
Recent Changes
Merged in ADAPT-671_Warriors_Logistics_20240917 (pull request #514) by Jacob Bennett at 2024-10-01 04:26:44
ADAPT-671 Warriors Logistics 20240917 * First pass through mappings. * Adds functions to handle window dates outside of 90/180/270/360 for linehaul/total_cost/linehaul_shipper values. These come up when window sizes don’t match with one of those four. Approved-by: Scott Hutchinson
Merged in feature/add-company-dashboard-adapt-task (pull request #506) by Scott Hutchinson at 2024-09-13 16:16:21
Add Dashboard generation pipeline * Update load_actual_mkt_lh calcs * Add seasonality query * Update forecast logic to use zero fuel, distance normalized * Remove dat reefer calcs * Add dashboard_params section to default param file * Add serializer for checkpoints * Add config option for checkpoint serializer * Add stream handler to prefect logger * Add mapped tasks and database read tasks * Add test mapping for ball, to be removed later * Update compute in dashboard pipeline * Remove old adapt dashboard pipeline file * Fix read_db gtm_loads backfill * Add dashboard params to default param file * Update dashboard pipeline with new task types * Remove legacy dashboard code * Merge branch ‘dev’ into feature/add-company-dashboard-adapt-task * Add ball specific mapper, remove test.py * Add logging info for custom functions in dashboard mapper * Merge branch ‘dev’ into feature/add-company-dashboard-adapt-task * Add market_rates and reference_miles queries * Add temp task for new rate calculation before refactor * Merge branch ‘dev’ into feature/add-company-dashboard-adapt-task * Remove new forecast calculations from mapper * Merged dev into feature/add-company-dashboard-adapt-task Approved-by: Jacob Bennett
Merged in ADAPT-666_Ball_update_20240904 (pull request #507) by Jacob Bennett at 2024-09-11 17:15:07
Ball 2024-09-04 updated. * Ball 2024-09-04 updated. DAT rates in for adapt and then removed. Insert mostly done, but constellation tables are still waiting on a PR in the migrations repo to fix a few bugs. * Merged dev into ADAPT-666_Ball_update_20240904 Approved-by: Scott Hutchinson
Merged in ADAPT-669_Novamex_rerun_20240909 (pull request #509) by Jacob Bennett at 2024-09-10 13:49:30
Novamex rerun 20240909 * Rerun of novamex as previous run was about 1yr old and had missing info from various files. Updates in mapping and params but no other changes. Approved-by: Scott Hutchinson
Merged in ADAPT-668_Oshkosh_rerun_20240905 (pull request #508) by Jacob Bennett at 2024-09-09 16:11:53
Oshkosh Corporation 2024-09-05 * Updated run for generating dashboards. Previous run was over 2 years ago. Approved-by: Scott Hutchinson
Merged in ADAPT-660_Hubbell_20240725 (pull request #500) by Jacob Bennett at 2024-08-27 21:13:29
Full Hubbell run. * Full Hubbell run. TL, no peg info so ran as zero peg. * Rerun with FSC info * Hubbell rerun with batch_date of 2024-08-07. Minor fixes and adjustments * Merged dev into ADAPT-660_Hubbell_20240725 * Merged dev into ADAPT-660_Hubbell_20240725 * Merged dev into ADAPT-660_Hubbell_20240725 Approved-by: Scott Hutchinson
Merged in ADAPT-647_Ball_20240426 (pull request #481) by Jacob Bennett at 2024-08-27 21:03:45
Run with updated Ball data, DAT rates used as Ball only gives averages. * Run with updated Ball data, DAT rates used as Ball only gives averages. * Merge branch ‘dev’ into ADAPT-647_Ball_20240426 Approved-by: Scott Hutchinson
Merged in ADAPT-664_Leaf_internal_active_20240820_ytd (pull request #504) by Jacob Bennett at 2024-08-27 20:38:05
ADAPT-664 Leaf internal active 20240820 ytd * Leaf active 2024 YTD with additional spend data from PGA. * Merge branch ‘dev’ into ADAPT-645_Leaf_Type_1_Active_20240424 * leaf active internal run 2024-08-20 ytd. Approved-by: Scott Hutchinson
Merged in standard-textile-20240615 (pull request #492) by Jacob Bennett at 2024-08-27 20:05:59
ADAPT-656 Standard textile 20240615 * Add standard textile mappings and params * Full run and insert. * Merged dev into standard-textile-20240615 Approved-by: Scott Hutchinson
Merged in fix/assert_zips_for_rate_pipeline (pull request #505) by Jacob Bennett at 2024-08-27 16:46:09
Adds assert prior to calculating market rates as nulls will break merges * Adds assert prior to calculating market rates as nulls will break merges * Change assert statement to not isna().any() Approved-by: Scott Hutchinson
- Commit:
- 09c5f82