As I understand your query, you are trying to convert date datatype columns into timestamp datatype in your dataframe. Please let me know if that is not the ask here. You can use to_timestamp() function to convert String to Timestamp (TimestampType) in PySpark. The converted time wou...
Apache Spark, Apache Impala) that still write parquet files the old way, with TIMESTAMPS as INT96 (with no logical data type) per default because of the larger precision / range. This is also true for prominent Microsoft/Azure services (e.g. Azure Synapse, Azure HDI, Microsoft Fabric, ...