Extract data from JSON strings using SQL operators (GA) Changes can be efficiently applied from the narrow table to the wider table.įor details, see Change data feed. It is a common practice to maintain two versions of the same table one narrow table as the source of truth and a wider table with additional data. Maintaining sync between replicas of two different tables representing the same data.Getting rows that were updated, inserted, or deleted greatly improves the performance of the downstream job consuming the output of the merge as entire files need not be processed and deduplicated now. Efficient downstream consumption of merge, updates, and deletes.You can query these changes through SQL and DataFrame and DataStream readers.
When enabled, the runtime records additional information regarding row-level changes for every write operation on the table. The change data feed of a Delta table represents the row-level changes between different versions of the table. Incrementally ingest updates and deletions in Delta tables using a change data feed (Public Preview) The runtime can also detect when tables are MERGE intensive and autotunes file sizes.įor more fine-grained tuning, you can set your target size as a property, and all data layout optimization operations (forĮxample, Optimize, Optimized Writes, and Auto Compaction) will make a best-effort attempt to generate files of that size. Operations (for example, upserting of change data into tables). You can now set a table property to instruct Delta Lake to select file sizes suitable for tables with frequent MERGE Tune the size of files in Delta tables to suit the needs of your workloads The Delta Lake CLONE and RESTORE commands now return metrics when they complete, so you can easily understand how much data was cloned or recovered as a result of these operations. Easily understand how much Delta Lake data was cloned or recovered See Audit information and Operation metrics keys. VACUUM commits to the Delta transaction log now contain information about files removed and directories vacuumed. See Schema inference and evolution in Auto Loader. Auto Loader will also “rescue” data that was unexpected (for example, of differing data types) in a JSON blob column, that you can choose to access later by using the semi-structured data access APIs. In addition, Auto Loader can detect the introduction of new columns to your data and restart, so that you never lose data again. New Auto Loader options enable easier schema evolutionĪuto Loader no longer requires you to specify a schema input to get started when ingesting JSON data. Extract data from JSON strings using SQL operators (GA).Incrementally ingest updates and deletions in Delta tables using a change data feed (Public Preview).Tune the size of files in Delta tables to suit the needs of your workloads.Easily understand how much Delta Lake data was cloned or recovered.New Auto Loader options enable easier schema evolution.The following release notes provide information about Databricks Runtime 8.2, powered by Apache Spark 3.1.1. Databricks released this image in April 2021.
0 kommentar(er)
