Discussions
Unable to get yesterday's data during midnight-2am
What is the correct way to obtain "all of yesterday's data" for a frequently running cron job?
Suppose a cron job is running a few times a day, asking for the day's data. When a new day begins at midnight UTC, my assumption is that I need to make requests using days_ago=1
to obtain all of yesterday. In practice I've run into some data issues. If I make that request between 0-2am, the data I get back only spans 00:00-22:19 and is missing everything after 22:20. So I should wait a bit (give the historical db some time to catch up), but how long? Since Dialpad also caches my requests, asking for the same report for the next ~2hrs gives back the same (incomplete) report back. The problem seems to fix itself after ~3hours where asking for the same report generates a brand new report, with the missing data filled in.
My current solution is to avoid asking for reports between 0-3am because I expect data to be incomplete. Does anyone know either (a) how to retrieve "yesterday's data" completely just after midnight UTC, or (b) some guidance on how long to wait after midnight before making the first request.