'The Design and Implementation of Presto Local Cache at Uber' by Alluxio uber localcache
we introduced Uber’s Presto use cases and how we collaborated to implement Alluxio local cache to overcome different challenges in accelerating Presto queries. The second part discusses the improvements to the local cache metadata.First, we want to prevent stale caching. The underlying data files might be changed by the third-party frameworks. Note that this situation might be rare in Hive tables but very common in Hudi tables.
Third, metadata should be recoverable after server restart. We have stored metadata in the local cache in memory instead of disk, which makes it impossible to recover metadata when the server is down and restarted.Therefore, we propose the File Level Metadata, which holds and keeps the last modified time and the scope of each data file we cached. The file-level metadata store should be persistent on disk so the data will not disappear after restarting.
When calling openFile, Alluxio creates a new instance of PrestoCacheContext, which holds the HiveFileContext and has the scope , quota, cache identifier , and other information. We will pass this cache context to the local file system. Alluxio can thus manage metadata and collect metrics.In addition to passing data from Presto to Alluxio, we can also call back to Presto.
México Últimas Noticias, México Titulares
Similar News:También puedes leer noticias similares a ésta que hemos recopilado de otras fuentes de noticias.
Speeding Up Uber’s Presto with Alluxio - Part 1 | HackerNoonReduce HDFS Latency with Alluxio Managed SSD Cache and Presto Consistent Hashing-Based Soft Affinity Scheduling
Leer más »
Speeding Up Uber’s Presto with Alluxio - Part 1 | HackerNoonReduce HDFS Latency with Alluxio Managed SSD Cache and Presto Consistent Hashing-Based Soft Affinity Scheduling
Leer más »
Astounding Stories of Super-Science, February 1930 - Table of Links | HackerNoon'Astounding Stories of Super-Science, February 1930 - Table of Links' by hackernoon sciencefiction astoundingstories
Leer más »
![Tell Us About Your Gaming Experiences [Gaming Writing Prompt] | HackerNoon](https://i.headtopics.com/images/2022/6/3/hackernoon/tell-us-about-your-gaming-experiences-gaming-writing-prompt-hackernoon-1532451992135778318.webp?w=640)
Tell Us About Your Gaming Experiences [Gaming Writing Prompt] | HackerNoon
Leer más »
The Economic Consequences of the Peace: Chapter III - The Conference | HackerNoonThe Economic Consequences of the Peace Chapter III: The Conference by John Maynard Keynes is part of HackerNoon’s Book Blog Post series.
Leer más »
Progressive Delivery, Feature Flags, and Experiments: The Story of a New Dashboard | HackerNoon'Progressive Delivery, Feature Flags, and Experiments: The Story of a New Dashboard'cc: bugsnag goodcompany
Leer más »