Build Data Model

This is the data processing component of my engineering thesis, "Development and analysis of a Process Mining based model representing public transportation in Warsaw" (2023). It turns raw GPS pings from Warsaw's public transit vehicles into an event log and a set of helper tables that can be fed into a process mining tool such as Celonis.

I wrote this a few years ago and it is not something I plan to keep developing. It ran as a single Azure Function, triggered once a day, processing the previous day's data.

Background

Warsaw's public transport publishes live vehicle GPS coordinates, but not which stop a vehicle just passed or which direction it is travelling. This code reconstructs that missing information: it matches raw coordinates to timetable routes, detects when a vehicle reaches a stop, groups stops into full rides, and matches each ride against the official timetable to calculate delays. Weather data for the relevant time and location is added to each row afterward.

Full details on the requirements, architecture, and data model are in the two PDFs included in this repository:

• Barankiewicz Alicja - Engineering thesis.pdf, the full thesis text
• Appendix 1 - Data model tables and algorithms.pdf, table schemas and algorithm listings

How it works

The core of the project is a small finite state automaton that walks through one vehicle's GPS pings in time order and figures out what the vehicle is doing. Each vehicle has one of these states at any point:

• UNKNOWN: the vehicle's direction and next stop are not yet known. The algorithm looks at the vehicle's recent positions, compares them against the two possible directions of its route, and picks the direction the vehicle appears to be moving toward.
• A or B: the vehicle is following a known direction of its route. Each new GPS point is checked against the next stop on the route using basic geometry, treating each stop as a small circle and checking whether the vehicle's path crosses it. A crossing means the stop was visited, so it gets logged and the vehicle moves on to expect the next stop. Reaching the last stop of the route flips the vehicle over to the opposite direction and starts a new ride.

If a vehicle goes quiet for more than 20 minutes, or a gap between two GPS points is unusually long, the state resets to UNKNOWN and the algorithm has to re-detect the direction from scratch. This means only rides that were fully and cleanly detected from start to end make it into the final data. Anything partial gets discarded, which was a deliberate choice to keep the resulting data model reliable.

Once a ride is detected, it gets matched against the official timetable to see whether it ran on time and to calculate delays at each stop. Weather data for the ride's stops and times is then attached as extra context.

The whole process runs per vehicle in parallel, and results from all vehicles are combined into one event log plus a handful of supporting tables (rides, tracks, stops, cities, day types).

Project structure

• __init__.py, the Azure Function entry point. Downloads raw data, kicks off processing, uploads results.
• preprocess.py, orchestrates the run: groups GPS data by vehicle, runs the detection algorithm across vehicles, adds weather context.
• analyze_gps.py, the state machine described above.
• classes.py, the domain objects (Vehicle, Route, Ride, RideHistory, weather API wrapper).
• helper_functions.py, geometry helpers used to detect stop visits.
• helper_tables.py, logic for incrementally updating the supporting tables.
• config.py / imports.py, shared settings and imports.

Running

This was built to run as an Azure Function on a daily timer (see function.json), reading from and writing to Azure Blob Storage. There is a local mode (LOCAL flag in config.py) that reads and writes plain CSV and parquet files from disk instead, which is what was used during development and testing.

It needs an Azure Storage connection string and an OpenWeatherMap API key to fetch weather history, both read from environment variables.

There is no dependency file included. Direct dependencies visible from the imports are pandas, numpy, pyarrow, requests, pytz, and the Azure Functions and Storage SDKs.

Status

This project is finished and archived. I am not maintaining it.