I have had a script running for a few months but ran into an issue today in a load job from a pandas df, with a stamp column
df.published_at[0]
gives
Timestamp('2022-04-28 20:59:51-0700', tz='pytz.FixedOffset(-420)')
When I try to load to BigQuery through a load job, I get the following error:
[{'reason': 'invalidQuery', 'location': 'query', 'message': 'Cannot return an invalid timestamp value of 1651204791000000000 microseconds relative to the Unix epoch. The range of valid timestamp values is [0001-01-01 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field published_at'}]
It seems that BigQuery is somehow reading that timestamp as Unix nanoseconds (1651204791000000000), not microseconds (which would be 1651204791000000) which is putting it out of the range of acceptable values. Why is it doing that?
I used a workaround to just use a string for that column before the load job, and the BQ schema accepts it as a timestamp. I'm just curious why this issue might have come up now and not previously?
