Log management is one of the most important tasks that every autonomy program needs to master. Test fleets collect on average four terabytes of drive data per vehicle per day, while production fleets (i.e., vehicles purchased by individual consumers) can generate millions of events per day. This firehose of data has enormous potential to power an autonomy program’s development efforts.
Due to the costs and risks involved in real-world testing, it is crucial that autonomy programs collect and manage their drive data effectively. For example, to operate a test fleet, autonomy programs must purchase and maintain vehicles and sensors and pay a team of safety operators. Additionally, just one critical mistake during real-world testing can put human lives at risk. Autonomy programs should thus implement practices to scale their data collection efficiently, create a pipeline for effective drive data processing, and build scalable workflows that extract the maximum value from all collected data.
Applied Intuition’s log management handbook discusses the technical building blocks, ideal workflows, and cost management strategies of an expansive drive data management process. This blog post is the first in a three-part series providing a short introduction to these topics.
