01. Code used in Pipeline: pl_stg_processing_nyctaxi - alexracz73/Fabric_E2E_Project GitHub Wiki

Overall Pipeline

---

Latest Processed Data

For the Script Activity “Latest Processed Data”

select top 1 
latest_processed_pickup 
from metadata.processing_log 
where table_processed = 'staging_nyctaxi_yellow'
order by latest_processed_pickup desc;

---

v_end_date

Pipeline expression for v_end_date Set Variable activity

@addToTime(concat(variables('v_date'), '-01'),1,'Month')

---

SP Removing Outlier Dates

For the Stored Procedure Activity “SP Removing Outlier Dates”.

Create the Stored Procedure stg.data_cleaning_stg in the Data Warehouse using the code below.

create procedure stg.data_cleaning_stg
@end_date datetime2,
@start_date datetime2
as
delete from stg.nyctaxi_yellow where tpep_pickup_datetime < @start_date or tpep_pickup_datetime > @end_date;