I want to get the p50FileSize metric from after a OPTIMIZE query returns. Currently it seems the only way to accomplish this is along the lines of:
spark.sql("OPTIMIZE .delta 'path'")
dt = DeltaTable.forPath(spark, 'path')
Then get the history and operations from dt
A quicker way would be to pull this data immediately after the query:
data = Spark.sql("OPTIMIZE .delta 'path'")
And grab metrics from data, but there are no metrics for p25FileSize, p50FileSize, p75FileSize. Is there a way to get these metrics without calling the .history() command?
