0
Follow
0
View

BigQuery load job from pandas dataframe timestamp column reading as unix nanoseconds, not microseconds

cryice520 注册会员
2023-01-24 21:44

I come here 5 months later (29 September 2022) because I have the exact same problem.

I'm trying to load data to BigQuery, from python, with client.load_table_from_json. One of my columns is a "processed_at" column which stores time objects (dtype: datetime64[ns, UTC]). I specify the right type in my table_schema :

table_schema = [
    bigquery.SchemaField("processed_at", "TIMESTAMP", mode="NULLABLE")
]

I get this error :

BadRequest: 400 Cannot return an invalid stamp value of 1664454374000000000 microseconds relative to the Unix epoch.
The range of valid timestamp values is [0001-01-01 00:00:00, 9999-12-31 23:59:59.999999]; error in writing field processed_at

BigQuery really seems to think in microsecondes, instead of nanoseconds, and thus all my datetimes fall out of range.

I will try to cast them as strings, thanks for the workaround.

About the Author

Question Info

Publish Time
2023-01-24 21:44
Update Time
2023-01-24 21:44