Currently, the operator makes an API call pulling 100 campaigns by recent activity into memory per call. After 50 calls it writes the data to S3 and then sets output = [].
[https://github.com/airflow-plugins/hubspot_plugin/blob/master/operators/hubspot_to_s3_operator.py#L379](reset here).
It then never writes the last batch of 5000 campaigns to S3 and then iterates over each of the last 5000 campaigns to make an individual api call for each campaign to grab the detailed campaign info.
So if you have more than 5000 campaigns, it will only create a detailed campaign file for the modulus 5000 of the number of campaigns.
Currently, the operator makes an API call pulling 100 campaigns by recent activity into memory per call. After 50 calls it writes the data to S3 and then sets
output = [].[https://github.com/airflow-plugins/hubspot_plugin/blob/master/operators/hubspot_to_s3_operator.py#L379](reset here).
It then never writes the last batch of 5000 campaigns to S3 and then iterates over each of the last 5000 campaigns to make an individual api call for each campaign to grab the detailed campaign info.
So if you have more than 5000 campaigns, it will only create a detailed campaign file for the modulus 5000 of the number of campaigns.