{"article":{"slug":"how-shredding-json-is-giving-logfire-1000x-query-speedups","title":"How shredding JSON is giving Logfire 1000x query speedups","subtitle":null,"summary":"Adrian Garcia Badaracco explains how Pydantic Logfire uses dynamic shredding of semi-structured OpenTelemetry attributes into separate Parquet columns, so queries on JSON attributes that once timed out now finish in under a second, plus plans for indexes and usage-based shredding.","content_type":"blog_post","language":"en","canonical_url":"https://pydantic.dev/articles/dynamic-shredding-2026-01-26","author":{"name":"Adrian Garcia Badaracco","url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Pydantic","url":"https://pydantic.dev/","listing_slug":null,"listing":null},"topics":[{"name":"Databases","slug":"databases","url":"https://listedarticles.com/topics/databases"},{"name":"Observability","slug":"observability","url":"https://listedarticles.com/topics/observability"},{"name":"Performance","slug":"performance","url":"https://listedarticles.com/topics/performance"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":3025,"reading_minutes":13,"published_at":"2026-01-29T00:00:00.000Z","added_at":"2026-10-11T08:12:45.751Z","updated_at":"2026-10-11T08:12:45.751Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/how-shredding-json-is-giving-logfire-1000x-query-speedups","markdown_url":"https://listedarticles.com/articles/how-shredding-json-is-giving-logfire-1000x-query-speedups.md","example":false,"citation":"Adrian Garcia Badaracco, Pydantic. \"How shredding JSON is giving Logfire 1000x query speedups.\" 29 Jan 2026. https://pydantic.dev/articles/dynamic-shredding-2026-01-26 (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://pydantic.dev/articles/dynamic-shredding-2026-01-26"},"body_markdown":"## Preface\n\nQuerying data in [Logfire](https://pydantic.dev/logfire) is about to get dramatically faster. Consider >1,000 times faster in some cases. Queries that previously timed out after 30 seconds now complete in under a second.\n\n> \"Seeing much faster query times! Feels basically instant now\", says an early access customer\n\nThis technical blog post details how we achieved this through dynamic shredding, an optimization for querying semi-structured attributes data. We'll begin to roll out this feature to all customers in February 2026.\n\n## Logfire Data Model\n\nOpenTelemetry data (the data that Logfire ingests) is semi-structured data.\nIt contains a mix of:\n\n* Fixed schema fields (timestamp, span name, etc.)\n* Semi-structured attributes (HTTP status codes, user IDs, regions, etc.)\n\nSemi-structured data is sent via `attributes` (per-span key-value pairs) or resource attributes (`otel_resource_attributes` in Logfire; these are per-resource key-value pairs where a resource is a service/k8s pod/etc, essentially an instance of your application or service).\n\nWhen you write code like:\n\n```\napp = FastAPI()\n\n@app.get('/user/{user_id}')\nasync def handle_request(user_id: str) -> dict[str, str]:\n with logfire.span(f'handle_request for user {user_id}', user_id=user_id):\n return {\"user_id\": user_id}\n```\n\nThis generates an export along the lines of:\n\n```\n{\n \"resource\": {\n \"attributes\": {\n \"service.name\": \"my-web-service\",\n \"region\": \"us-west-2\"\n }\n },\n \"spans\": [\n // Manual span created by user code\n {\n \"span_name\": \"handle_request for user user123\",\n \"start_timestamp\": \"2024-01-01T12:00:00Z\",\n \"end_timestamp\": \"2024-01-01T12:00:01Z\",\n \"message\": \"handle_request for user user123\",\n \"trace_id\": \"0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1\",\n \"span_id\": \"abcdef1234567890\",\n \"parent_span_id\": \"1234567890abcdef\",\n \"attributes\": {\n \"user_id\": \"user123\",\n \"http.status_code\": 200,\n },\n },\n // Automatic span created by OpenTelemetry FastAPI instrumentation\n {\n \"span_name\": \"GET /user/{user_id}\",\n \"start_timestamp\": \"2024-01-01T12:00:00Z\",\n \"end_timestamp\": \"2024-01-01T12:00:01Z\",\n \"message\": \"GET /user/user123\",\n \"trace_id\": \"0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1\",\n \"span_id\": \"1234567890abcdef\",\n \"parent_span_id\": null,\n \"attributes\": {\n \"http.method\": \"GET\",\n \"http.response\": 200,\n },\n }\n ]\n}\n```\n\nI'm using JSON here for illustration, although we support various wire formats (OTLP protobuf, JSON, etc.); most data is sent as protobuf over gRPC.\nThere are also some details like timestamps being represented as integers (epoch nanos) rather than ISO strings and an extra level of nesting from scopes that I've omitted for clarity.\n\nA couple of things to note here:\n\n1. Some fields like `start_timestamp`, `end_timestamp`, `span_name`, and `message` are fixed-schema fields that exist for every span\n and are strongly typed.\n2. Attributes are semi-structured and can vary widely between spans.\n Some attributes like `http.status_code` and `http.method` are common and have semantic meaning in the OpenTelemetry spec,\n while others like `user_id` are application-specific and could be an integer in some places and a string in others.\n\nWe ultimately store this data as Parquet files in object storage, with a columnar schema roughly like:\n\n```\nstart_timestamp: Timestamp(Microsecond, \"+00:00\")\nend_timestamp: Timestamp(Microsecond, \"+00:00\")\nspan_name: Utf8View\nmessage: Utf8View\ntrace_id: FixedSizeBinary(16)\nspan_id: FixedSizeBinary(8)\nparent_span_id: FixedSizeBinary(8)\nattributes: Utf8View (JSON)\notel_resource_attributes: Utf8View (JSON)\n```\n\nSome background on columnar storage, Parquet and zone map pruning is beneficial to understand the rest of this blog post. I recommend reading [MotherDuck's excellent overview of the topic](https://motherduck.com/learn-more/columnar-storage-guide/).\n\n### How we started: fast JSON parsing\n\nPydantic started as a library to validate and parse JSON data efficiently.\nWhen we built Logfire, we initially stored all attributes as JSON blobs in a single column and relied on [Jiter](https://github.com/pydantic/jiter) (one of the fastest JSON parsers out there in any language) to rip through the JSON data as quickly as possible.\nThis is the layout detailed above (`attributes: Utf8View (JSON)`).\n\nThis approach was simple but made a fundamental mistake: we were optimizing to do work faster, but we couldn't really avoid doing the work.\nCompression is also suboptimal since a single varying attribute makes Parquet level dictionary compression ineffective and even zstd struggles with highly variable data, all the JSON syntax overhead, etc.\n\nThe biggest problem really ends up being extra IO: since most of our data is stored on object storage, extra IO from downloading large JSON blobs, and especially the latency of starting these downloads, ends up dominating query runtimes much more so than any CPU work.\n\n### Shredding columns to save IO and compute\n\nTo make queries fast, we need to avoid unnecessary work.\nFor normal columns DataFusion / Parquet avoid unnecessary work by using columnar storage (allowing reading only the columns you need)\nand zone maps / min-max statistics (allowing skipping entire row groups or pages that don't match your filter predicates).\n\nSo when you write a query like:\n\n```\nSELECT count(*) FROM records WHERE duration < 0;\n```\n\nThe system can efficiently read just the `duration` column and skip row groups where the max duration is < 0 based on the stored statistics.\nIn this case this is a valid query, duration is a f64 so in theory it could have values <0.\nIn practice though duration is always >= 0 so this query will be very fast: based on statistics we know no rows match and we can return 0 immediately.\n\nSo how can we enable this for semi-structured data like attributes stored as JSON?\n\nOur initial solution was *static shredding*: we manually picked a small set of commonly queried attributes\n(like `http.status_code`, `http.method`, etc.) and extracted these into their own strongly typed columns during ingestion.\n\nSo when we receive:\n\n| attributes |\n| --- |\n| {'http.response.status\\_code': 200, 'url.full': 'https:‎//example.com?foo=bar'} |\n| {'my\\_llm\\_response': 'large text'} |\n| {'user\\_id': '123'} |\n\nAssuming our hardcoded shredding configuration extracts `http.response.status_code`, `url.full`, and `db.query.text`, we would store:\n\n| \\_lf\\_attributes | \\_lf\\_attributes\\_http.response.status\\_code | \\_lf\\_attributes\\_url.full | \\_lf\\_attributes\\_db.query.text |\n| --- | --- | --- | --- |\n| null | 200 | https:‎//example.com?foo=bar | null |\n| {'my\\_llm\\_response': 'large text'} | null | null | null |\n| {'user\\_id': '123'} | null | null | null |\n\nNow we can run queries like:\n\n```\nSELECT count(*) FROM records WHERE _lf_attributes_http.response.status_code = 500;\n```\n\nWe can use statistics to prune files / row groups / pages that don't match,\nand then scan just the `_lf_attributes_http.response.status_code` column which is strongly typed (int64) and compresses well.\n\nOf course you as a user don't have to use the `_lf_attributes_http.response.status_code` column directly, you can write:\n\n```\nSELECT count(*) FROM records WHERE attributes->'http.response.status_code' = 500;\n```\n\nAnd we rewrite this under the hood to use the shredded column.\n\nI thought it worth mentioning the `_lf_attributes` prefix: one of the challenges with the current static shredding approach\nis that these columns are user-visible, e.g. if you run `select * from records limit 10` you'll see these columns.\nSince these are an internal implementation detail, we'd rather not leak them to you.\nWe want to keep the logical layout of `attributes` as a single JSON column from your perspective,\nwhile optimizing the physical layout under the hood.\n\n### The Bad: Limited Flexibility, High Cost for Unshredded Attributes and Overheads\n\nThere are several issues with this scheme:\n\n* **Limited Flexibility**: Only a small, hardcoded set of attributes are shredded. If users want to query other attributes (e.g., `user_id`, `session_id`, `region`), they pay the full cost of JSON storage and parsing. Queries on these attributes can get slow on projects with 100s of GBs of data. We can't easily add new shredded attributes without code changes and deployments. This is particularly painful with indexing; we only have indexes on the hardcoded set of shredded attributes.\n* **High Cost for Unshredded Attributes**: Queries that filter on unshredded attributes (e.g., `attributes->>'user_id' = 'user123'`) require scanning the entire JSON column, leading to high I/O, poor compression, and expensive JSON parsing. This results in slow queries and high resource usage. Since data is stored in chunks, even a single large unshredded attribute can bloat the entire column, making all queries expensive even if they are not touching that attribute.\n* **User-visible columns**: The shredded columns are visible to users, leading to confusion and clutter in query results. Users may not understand the purpose of these internal columns, and they can complicate query writing and result interpretation.\n* **Overheads**: As you can see, there are several columns that may be all `null`s (e.g. if the application doesn't do DB queries, the `_lf_attributes_db.query.text` column is always null). Although Parquet / Arrow / DataFusion are quite efficient at handling nulls, there is still some overhead in storage and processing for these unused columns.\n\nEvery query touching `attributes` that isn't filtering on the small hardcoded set of shredded attributes pays these costs.\n\nFor example, consider the query:\n\n```\nSELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';\n```\n\nThis query needs to scan the entire `attributes` JSON column since `my_llm_response` is not shredded.\nEven though `my_llm_response` is a large attribute it may be a *rare* attribute that only appears in a small fraction of rows.\nThink about it this way: you probably have many more spans that don't involve LLM calls than ones that do, even in an agentic application.\n\n```\n┌─────────────────────────────────────────┐\n│ Cost 1: Download Entire _lf_attributes │\n│ (could be GBs of data) │\n└─────────────────────────────────────────┘\n ↓\n┌──────────────────────────────────────────┐\n│ Cost 2: Poor Compression │\n│ Variable structure → zstd 10% efficiency│\n└──────────────────────────────────────────┘\n ↓\n┌─────────────────────────────────────────┐\n│ Cost 3: JSON Parsing │\n│ Jiter does lazy parsing, worst case │\n│ must parse entire row. │\n└─────────────────────────────────────────┘\n ↓\n┌─────────────────────────────────────────┐\n│ Cost 4: In-Memory Operations │\n│ Create copies of extracted values │\n│ Do comparisons in memory │\n└─────────────────────────────────────────┘\n```\n\n**The Result**: this query has to parse and scan potentially GBs of JSON data, much of which is irrelevant to the query, just to get the few rows that contain `my_llm_response`.\n\nOr consider the query:\n\n```\nSELECT * FROM events WHERE attributes->>'user_id' = 'user123';\n```\n\nThis query now has the opposite problem: even if 1/3 of rows contain a `user_id` attribute, we have to scan the entire JSON column to find them,\nwhich includes those large `my_llm_response` attributes that bloat the column size.\n\n## The Solution: Dynamic Shredding\n\nRather than storing all attributes in a single JSON column, we automatically extract the **most frequently accessed attributes** into separate, strongly typed columns.\nThis happens automatically during data ingestion and compaction—no configuration needed.\n\nLet's compare the physical layout of data before and after dynamic shredding.\n\n**Before (Current)**:\n\n| \\_lf\\_attributes | \\_lf\\_attributes\\_http.response.status\\_code | \\_lf\\_attributes\\_url.full | \\_lf\\_attributes\\_db.query.text |\n| --- | --- | --- | --- |\n| null | 200 | https:‎//example.com?foo=bar | null |\n| {'my\\_llm\\_response': 'large text'} | null | null | null |\n| {'user\\_id': '123'} | null | null | null |\n\n**After (With Dynamic Shredding)**:\n\n| \\_lf\\_attributes | \\_lf\\_attributes\\_http.response.status\\_code | \\_lf\\_attributes\\_url.full | \\_lf\\_attributes\\_my\\_llm\\_response | \\_lf\\_attributes\\_user\\_id |\n| --- | --- | --- | --- | --- |\n| null | 200 | https:‎//example.com?foo=bar | null | null |\n| null | null | null | large text | null |\n| null | null | null | null | 123 |\n\nThis layout is generated dynamically for each Parquet file we write out based on the actual data we see.\nWe don't have to include all null columns and we can prioritize shredding the most frequently seen or largest attributes.\n\nNow when the query above runs:\n\n```\nSELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';\n```\n\nIt can efficiently scan just the `_lf_attributes_my_llm_response` column:\n\n```\n┌─────────────────────────────────────────┐\n│ Benefit 1: Index pruning │\n│ Zone maps and full text search indexes │\n│ let us skip entire files / row groups │\n└─────────────────────────────────────────┘\n ↓\n┌──────────────────────────────────────────┐\n│ Benefit 2: Download Only Relevant Column│\n│ (_lf_attributes_my_llm_response) │\n└──────────────────────────────────────────┘\n ↓\n┌─────────────────────────────────────────┐\n│ Benefit 3: Better Compression │\n│ English text string data │\n│ via zstd + dictionary encoding │\n│ Maybe FastLanes in future? │\n└─────────────────────────────────────────┘\n ↓\n┌─────────────────────────────────────────┐\n│ Benefit 4: Zero JSON Parsing │\n│ Data already extracted & typed │\n│ No deserialization, zero copy into mem │\n└─────────────────────────────────────────┘\n```\n\nThis query now benefits from the full efficiency of progressive narrowing / pruning and columnar storage.\nIt is as fast as if `my_llm_response` were a first-class column in the schema.\n\n## The Hard Path to Dynamic Shredding\n\nWe've been working on this feature for nearly 6 months now.\nIt's required a lot of upstream changes in DataFusion to enable expression pushdown and predicate pushdown for JSON accessors.\nThis is the blessing and the curse of working with an upstream open source project: we get to build on a solid foundation, but we also have to wait for upstream changes to land and get released.\nWe've worked on this feature in a way that benefits not only us but anyone using DataFusion with semi-structured or nested data, including Parquet/Arrow struct columns and the newly introduced [Json Variant type](https://parquet.apache.org/docs/file-format/types/variantencoding/).\n\nWe've had to refactor core parts of the DataFusion execution engine to push down expressions into the Parquet reader so that it can use the physical file layout to optimize expression evaluation (in our case to decide if we need to read from the `_lf_attributes` remainder column or if there is a shredded column we can use instead).\nThis work is done and has fully landed as of DataFusion 52.1.0 (Jan 2026).\n\nThe remaining work involves actually getting these expressions into the scan operators, e.g. for cases like:\n\n```\nSELECT attributes->>'http.response.status_code', count(*)\nFROM records\nGROUP BY attributes->>'http.response.status_code';\n```\n\nPulling out the `attributes->>'http.response.status_code'` expression from the GROUP BY clause and pushing it down into the Parquet scan so that we can use the shredded column directly is a bit tricky, and involves complex expression rewriting and optimization passes, but we expect to land this work in DataFusion by Feb 2026.\nThe upshot of this is that we've been able to contribute major improvements to DataFusion that benefit a broad community of users.\n\n## Minimizing Breakage for Users\n\nOur plan to land this while minimizing bugs and breakage for users is to first rewrite our query path (non-destructive, most of this has already been done) to use dynamic shredding but keep the static shredding in the write path.\nAlong with this change we'll eliminate and isolate all notion of hardcoded shredded columns from the codebase into a self-contained section of the write path.\nEssentially our initial heuristic for dynamic shredding will mimic the current static shredding configuration.\nThis lets us test out the vast majority of these changes without making any irreversible changes (like writing out a new physical layout).\n\nOnce we have confidence that the dynamic shredding query path is solid, we can flip the switch in the write path to enable dynamic shredding using the discovery algorithm.\n\n## How do we decide what to shred?\n\nAnalysis of our production data has shown that 99% of our existing Parquet files have fewer than 128 unique top-level keys in the `attributes` JSON column.\nThe P50 is closer to 32 unique keys.\n\nWe've thus chosen a default of extracting the top 128 largest (by total size across all rows) keys from each JSON column during ingestion/compaction.\n\nThis automatic discovery approach balances the tradeoff between:\n\n* Extracting enough attributes to cover common query patterns and maximize performance benefits\n* Avoiding excessive column proliferation and overhead from shredding too many attributes\n\nFor example, if you accidentally start sending logs like `logfire.span('foo', **{uuid.uuid4(): 'some value' for _ in range(1000)})`, we don't want to shred all 1000 dynamically generated keys.\nThis algorithm is robust to even such pathological cases because it will shred the non-unique keys that dominate the data while putting the rest of the \"accidental\" dynamically generated keys into the reduced JSON column.\nQueries against these rare keys will still work, just with the usual JSON parsing overhead.\nQueries against the rest of the data will remain fast.\n\nHaving a hard cap on the number of shredded columns lets us have a controlled overhead in terms of Parquet column count which avoids bloating the schema and incurring excessive overheads in parsing of Parquet metadata.\nOur empirical measurements shows that ~ 1000 columns is not a problem for Parquet / DataFusion, but more than that and you start to see a noticeable performance overhead.\n\n## How are data types determined?\n\nAs we discover which attributes to shred, we also need to determine the data type for each shredded column.\nWe do this by keeping track of the current planned output data type for each candidate attribute as we scan through the data and widening the type as needed.\nThe basic rule is `Int` -> `Float` -> `JSON`, `Bool` -> `JSON` and `String` -> `JSON`.\nThere is an important distinction between `String` and `JSON`.\nConsider the values `[{\"user_id\": \"123\"}, {\"user_id\": 456}]`.\nIn this case we would store the data as JSON which practically speaking is a `Utf8View` column containing JSON strings: [`\"123\"`, `456`].\nThis means JSON parsing is still needed to read from this column but we still benefit from columnar storage, compression, and to some extent pruning.\nIf on the other hand we have `[\"123\", \"456\"]` we can store this as a `Utf8View` column directly without JSON parsing: [`123`, `456`].\n\n## Future work\n\n### Using the Parquet Variant type\n\nOur shredded data layout predates both the Parquet Variant type and [ClickHouse's JSON support](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse)\nbut is very similar to these approaches.\n\nOur long term plan is to adopt the Parquet Variant type for storing the remaining unshredded attributes, which will give us essentially the same architecture but using an industry standard data layout that other query engines can also read and write, and moving complexity from our code to DataFusion / Parquet itself.\n\nThe DataFusion community (us included) is still working through some of the details of the Parquet Variant type.\nNamely we need to incorporate the variant UDFs into DataFusion and need to add support for statistics of nested struct types to enable pruning.\n\n### Dynamic indexing\n\nWe will initially have basic support for zone map pruning on shredded columns, but in the future we would like to add more index types like:\n\n* Inverted indexes for text columns to speed up `LIKE` and full-text search queries, e.g. so you can quickly filter LLM responses for keywords.\n* Bloom filters for high-cardinality columns to speed up equality filters.\n\n### Usage based shredding\n\nOur initial approach will be to shred based on attribute size/frequency in the data.\nThis lets us self-contain the heuristic within the write path for each Parquet file.\nIn the future we may want to incorporate query usage patterns into the shredding decision.\nFor example if we see that `user_id` is frequently queried but is not a large attribute, we may want to shred it anyway.\nThis would require tracking query patterns over time and feeding this back into the ingestion/compaction pipeline.\nWe would also use this approach to decide what indexing strategies to pursue for each shredded column.\n\n## Conclusion\n\nIn the next few months you'll probably see your Logfire queries get orders of magnitude faster.\nYou don't need to do anything on your end, just continue to use [Logfire](http://pydantic.dev/logfire?utm_source=adrian_blogpost) as you currently do!\n\n## References and related work\n\n* [DataFusion Blog: Parquet Pushdown](https://datafusion.apache.org/blog/2025/03/21/parquet-pushdown/)\n* [ClickHouse JSON Type Reference](https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouse)\n* [Parquet Variant type](https://parquet.apache.org/docs/file-format/types/variantencoding/)","body_html":"

Preface

\n

Querying data in Logfire is about to get dramatically faster. Consider >1,000 times faster in some cases. Queries that previously timed out after 30 seconds now complete in under a second.

\n

"Seeing much faster <customer_attribute> query times! Feels basically instant now", says an early access customer

\n

This technical blog post details how we achieved this through dynamic shredding, an optimization for querying semi-structured attributes data. We'll begin to roll out this feature to all customers in February 2026.

\n

Logfire Data Model

\n

OpenTelemetry data (the data that Logfire ingests) is semi-structured data.\nIt contains a mix of:

\n\n

Semi-structured data is sent via attributes (per-span key-value pairs) or resource attributes (otel_resource_attributes in Logfire; these are per-resource key-value pairs where a resource is a service/k8s pod/etc, essentially an instance of your application or service).

\n

When you write code like:

\n
app = FastAPI()\n\n@app.get('/user/{user_id}')\nasync def handle_request(user_id: str) -> dict[str, str]:\n  with logfire.span(f'handle_request for user {user_id}', user_id=user_id):\n    return {"user_id": user_id}
\n

This generates an export along the lines of:

\n
{\n  "resource": {\n    "attributes": {\n      "service.name": "my-web-service",\n      "region": "us-west-2"\n    }\n  },\n  "spans": [\n    // Manual span created by user code\n    {\n      "span_name": "handle_request for user user123",\n      "start_timestamp": "2024-01-01T12:00:00Z",\n      "end_timestamp": "2024-01-01T12:00:01Z",\n      "message": "handle_request for user user123",\n      "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1",\n      "span_id": "abcdef1234567890",\n      "parent_span_id": "1234567890abcdef",\n      "attributes": {\n        "user_id": "user123",\n        "http.status_code": 200,\n      },\n    },\n    // Automatic span created by OpenTelemetry FastAPI instrumentation\n    {\n      "span_name": "GET /user/{user_id}",\n      "start_timestamp": "2024-01-01T12:00:00Z",\n      "end_timestamp": "2024-01-01T12:00:01Z",\n      "message": "GET /user/user123",\n      "trace_id": "0194c7b1e5f4a3b2c1d0e6f7a8b9c0d1",\n      "span_id": "1234567890abcdef",\n      "parent_span_id": null,\n      "attributes": {\n        "http.method": "GET",\n        "http.response": 200,\n      },\n    }\n  ]\n}
\n

I'm using JSON here for illustration, although we support various wire formats (OTLP protobuf, JSON, etc.); most data is sent as protobuf over gRPC.\nThere are also some details like timestamps being represented as integers (epoch nanos) rather than ISO strings and an extra level of nesting from scopes that I've omitted for clarity.

\n

A couple of things to note here:

\n

  1. Some fields like start_timestamp, end_timestamp, span_name, and message are fixed-schema fields that exist for every span

    and are strongly typed.

  2. Attributes are semi-structured and can vary widely between spans.

    Some attributes like http.status_code and http.method are common and have semantic meaning in the OpenTelemetry spec,\n while others like user_id are application-specific and could be an integer in some places and a string in others.

\n

We ultimately store this data as Parquet files in object storage, with a columnar schema roughly like:

\n
start_timestamp: Timestamp(Microsecond, "+00:00")\nend_timestamp: Timestamp(Microsecond, "+00:00")\nspan_name: Utf8View\nmessage: Utf8View\ntrace_id: FixedSizeBinary(16)\nspan_id: FixedSizeBinary(8)\nparent_span_id: FixedSizeBinary(8)\nattributes: Utf8View (JSON)\notel_resource_attributes: Utf8View (JSON)
\n

Some background on columnar storage, Parquet and zone map pruning is beneficial to understand the rest of this blog post. I recommend reading MotherDuck's excellent overview of the topic.

\n

How we started: fast JSON parsing

\n

Pydantic started as a library to validate and parse JSON data efficiently.\nWhen we built Logfire, we initially stored all attributes as JSON blobs in a single column and relied on Jiter (one of the fastest JSON parsers out there in any language) to rip through the JSON data as quickly as possible.\nThis is the layout detailed above (attributes: Utf8View (JSON)).

\n

This approach was simple but made a fundamental mistake: we were optimizing to do work faster, but we couldn't really avoid doing the work.\nCompression is also suboptimal since a single varying attribute makes Parquet level dictionary compression ineffective and even zstd struggles with highly variable data, all the JSON syntax overhead, etc.

\n

The biggest problem really ends up being extra IO: since most of our data is stored on object storage, extra IO from downloading large JSON blobs, and especially the latency of starting these downloads, ends up dominating query runtimes much more so than any CPU work.

\n

Shredding columns to save IO and compute

\n

To make queries fast, we need to avoid unnecessary work.\nFor normal columns DataFusion / Parquet avoid unnecessary work by using columnar storage (allowing reading only the columns you need)\nand zone maps / min-max statistics (allowing skipping entire row groups or pages that don't match your filter predicates).

\n

So when you write a query like:

\n
SELECT count(*) FROM records WHERE duration < 0;
\n

The system can efficiently read just the duration column and skip row groups where the max duration is < 0 based on the stored statistics.\nIn this case this is a valid query, duration is a f64 so in theory it could have values <0.\nIn practice though duration is always >= 0 so this query will be very fast: based on statistics we know no rows match and we can return 0 immediately.

\n

So how can we enable this for semi-structured data like attributes stored as JSON?

\n

Our initial solution was static shredding: we manually picked a small set of commonly queried attributes\n(like http.status_code, http.method, etc.) and extracted these into their own strongly typed columns during ingestion.

\n

So when we receive:

\n
attributes
{'http.response.status_code': 200, 'url.full': 'https:‎//example.com?foo=bar'}
{'my_llm_response': 'large text'}
{'user_id': '123'}
\n

Assuming our hardcoded shredding configuration extracts http.response.status_code, url.full, and db.query.text, we would store:

\n
_lf_attributes_lf_attributes_http.response.status_code_lf_attributes_url.full_lf_attributes_db.query.text
null200https:‎//example.com?foo=barnull
{'my_llm_response': 'large text'}nullnullnull
{'user_id': '123'}nullnullnull
\n

Now we can run queries like:

\n
SELECT count(*) FROM records WHERE _lf_attributes_http.response.status_code = 500;
\n

We can use statistics to prune files / row groups / pages that don't match,\nand then scan just the _lf_attributes_http.response.status_code column which is strongly typed (int64) and compresses well.

\n

Of course you as a user don't have to use the _lf_attributes_http.response.status_code column directly, you can write:

\n
SELECT count(*) FROM records WHERE attributes->'http.response.status_code' = 500;
\n

And we rewrite this under the hood to use the shredded column.

\n

I thought it worth mentioning the _lf_attributes prefix: one of the challenges with the current static shredding approach\nis that these columns are user-visible, e.g. if you run select * from records limit 10 you'll see these columns.\nSince these are an internal implementation detail, we'd rather not leak them to you.\nWe want to keep the logical layout of attributes as a single JSON column from your perspective,\nwhile optimizing the physical layout under the hood.

\n

The Bad: Limited Flexibility, High Cost for Unshredded Attributes and Overheads

\n

There are several issues with this scheme:

\n\n

Every query touching attributes that isn't filtering on the small hardcoded set of shredded attributes pays these costs.

\n

For example, consider the query:

\n
SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';
\n

This query needs to scan the entire attributes JSON column since my_llm_response is not shredded.\nEven though my_llm_response is a large attribute it may be a rare attribute that only appears in a small fraction of rows.\nThink about it this way: you probably have many more spans that don't involve LLM calls than ones that do, even in an agentic application.

\n
┌─────────────────────────────────────────┐\n│  Cost 1: Download Entire _lf_attributes │\n│  (could be GBs of data)                 │\n└─────────────────────────────────────────┘\n                    ↓\n┌──────────────────────────────────────────┐\n│  Cost 2: Poor Compression                │\n│  Variable structure → zstd 10% efficiency│\n└──────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────┐\n│  Cost 3: JSON Parsing                   │\n│  Jiter does lazy parsing, worst case    │\n│  must parse entire row.                 │\n└─────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────┐\n│  Cost 4: In-Memory Operations           │\n│  Create copies of extracted values      │\n│  Do comparisons in memory               │\n└─────────────────────────────────────────┘
\n

The Result: this query has to parse and scan potentially GBs of JSON data, much of which is irrelevant to the query, just to get the few rows that contain my_llm_response.

\n

Or consider the query:

\n
SELECT * FROM events WHERE attributes->>'user_id' = 'user123';
\n

This query now has the opposite problem: even if 1/3 of rows contain a user_id attribute, we have to scan the entire JSON column to find them,\nwhich includes those large my_llm_response attributes that bloat the column size.

\n

The Solution: Dynamic Shredding

\n

Rather than storing all attributes in a single JSON column, we automatically extract the most frequently accessed attributes into separate, strongly typed columns.\nThis happens automatically during data ingestion and compaction—no configuration needed.

\n

Let's compare the physical layout of data before and after dynamic shredding.

\n

Before (Current):

\n
_lf_attributes_lf_attributes_http.response.status_code_lf_attributes_url.full_lf_attributes_db.query.text
null200https:‎//example.com?foo=barnull
{'my_llm_response': 'large text'}nullnullnull
{'user_id': '123'}nullnullnull
\n

After (With Dynamic Shredding):

\n
_lf_attributes_lf_attributes_http.response.status_code_lf_attributes_url.full_lf_attributes_my_llm_response_lf_attributes_user_id
null200https:‎//example.com?foo=barnullnull
nullnullnulllarge textnull
nullnullnullnull123
\n

This layout is generated dynamically for each Parquet file we write out based on the actual data we see.\nWe don't have to include all null columns and we can prioritize shredding the most frequently seen or largest attributes.

\n

Now when the query above runs:

\n
SELECT * FROM events WHERE attributes->>'my_llm_response' like '%important%';
\n

It can efficiently scan just the _lf_attributes_my_llm_response column:

\n
┌─────────────────────────────────────────┐\n│  Benefit 1: Index pruning               │\n│  Zone maps and full text search indexes │\n│  let us skip entire files / row groups  │\n└─────────────────────────────────────────┘\n                    ↓\n┌──────────────────────────────────────────┐\n│  Benefit 2: Download Only Relevant Column│\n│  (_lf_attributes_my_llm_response)        │\n└──────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────┐\n│  Benefit 3: Better Compression          │\n│  English text string data               │\n│  via zstd + dictionary encoding         │\n│  Maybe FastLanes in future?             │\n└─────────────────────────────────────────┘\n                    ↓\n┌─────────────────────────────────────────┐\n│  Benefit 4: Zero JSON Parsing           │\n│  Data already extracted & typed         │\n│  No deserialization, zero copy into mem │\n└─────────────────────────────────────────┘
\n

This query now benefits from the full efficiency of progressive narrowing / pruning and columnar storage.\nIt is as fast as if my_llm_response were a first-class column in the schema.

\n

The Hard Path to Dynamic Shredding

\n

We've been working on this feature for nearly 6 months now.\nIt's required a lot of upstream changes in DataFusion to enable expression pushdown and predicate pushdown for JSON accessors.\nThis is the blessing and the curse of working with an upstream open source project: we get to build on a solid foundation, but we also have to wait for upstream changes to land and get released.\nWe've worked on this feature in a way that benefits not only us but anyone using DataFusion with semi-structured or nested data, including Parquet/Arrow struct columns and the newly introduced Json Variant type.

\n

We've had to refactor core parts of the DataFusion execution engine to push down expressions into the Parquet reader so that it can use the physical file layout to optimize expression evaluation (in our case to decide if we need to read from the _lf_attributes remainder column or if there is a shredded column we can use instead).\nThis work is done and has fully landed as of DataFusion 52.1.0 (Jan 2026).

\n

The remaining work involves actually getting these expressions into the scan operators, e.g. for cases like:

\n
SELECT attributes->>'http.response.status_code', count(*)\nFROM records\nGROUP BY attributes->>'http.response.status_code';
\n

Pulling out the attributes->>'http.response.status_code' expression from the GROUP BY clause and pushing it down into the Parquet scan so that we can use the shredded column directly is a bit tricky, and involves complex expression rewriting and optimization passes, but we expect to land this work in DataFusion by Feb 2026.\nThe upshot of this is that we've been able to contribute major improvements to DataFusion that benefit a broad community of users.

\n

Minimizing Breakage for Users

\n

Our plan to land this while minimizing bugs and breakage for users is to first rewrite our query path (non-destructive, most of this has already been done) to use dynamic shredding but keep the static shredding in the write path.\nAlong with this change we'll eliminate and isolate all notion of hardcoded shredded columns from the codebase into a self-contained section of the write path.\nEssentially our initial heuristic for dynamic shredding will mimic the current static shredding configuration.\nThis lets us test out the vast majority of these changes without making any irreversible changes (like writing out a new physical layout).

\n

Once we have confidence that the dynamic shredding query path is solid, we can flip the switch in the write path to enable dynamic shredding using the discovery algorithm.

\n

How do we decide what to shred?

\n

Analysis of our production data has shown that 99% of our existing Parquet files have fewer than 128 unique top-level keys in the attributes JSON column.\nThe P50 is closer to 32 unique keys.

\n

We've thus chosen a default of extracting the top 128 largest (by total size across all rows) keys from each JSON column during ingestion/compaction.

\n

This automatic discovery approach balances the tradeoff between:

\n\n

For example, if you accidentally start sending logs like logfire.span('foo', **{uuid.uuid4(): 'some value' for _ in range(1000)}), we don't want to shred all 1000 dynamically generated keys.\nThis algorithm is robust to even such pathological cases because it will shred the non-unique keys that dominate the data while putting the rest of the "accidental" dynamically generated keys into the reduced JSON column.\nQueries against these rare keys will still work, just with the usual JSON parsing overhead.\nQueries against the rest of the data will remain fast.

\n

Having a hard cap on the number of shredded columns lets us have a controlled overhead in terms of Parquet column count which avoids bloating the schema and incurring excessive overheads in parsing of Parquet metadata.\nOur empirical measurements shows that ~ 1000 columns is not a problem for Parquet / DataFusion, but more than that and you start to see a noticeable performance overhead.

\n

How are data types determined?

\n

As we discover which attributes to shred, we also need to determine the data type for each shredded column.\nWe do this by keeping track of the current planned output data type for each candidate attribute as we scan through the data and widening the type as needed.\nThe basic rule is Int -> Float -> JSON, Bool -> JSON and String -> JSON.\nThere is an important distinction between String and JSON.\nConsider the values [{"user_id": "123"}, {"user_id": 456}].\nIn this case we would store the data as JSON which practically speaking is a Utf8View column containing JSON strings: ["123", 456].\nThis means JSON parsing is still needed to read from this column but we still benefit from columnar storage, compression, and to some extent pruning.\nIf on the other hand we have ["123", "456"] we can store this as a Utf8View column directly without JSON parsing: [123, 456].

\n

Future work

\n

Using the Parquet Variant type

\n

Our shredded data layout predates both the Parquet Variant type and ClickHouse's JSON support\nbut is very similar to these approaches.

\n

Our long term plan is to adopt the Parquet Variant type for storing the remaining unshredded attributes, which will give us essentially the same architecture but using an industry standard data layout that other query engines can also read and write, and moving complexity from our code to DataFusion / Parquet itself.

\n

The DataFusion community (us included) is still working through some of the details of the Parquet Variant type.\nNamely we need to incorporate the variant UDFs into DataFusion and need to add support for statistics of nested struct types to enable pruning.

\n

Dynamic indexing

\n

We will initially have basic support for zone map pruning on shredded columns, but in the future we would like to add more index types like:

\n\n

Usage based shredding

\n

Our initial approach will be to shred based on attribute size/frequency in the data.\nThis lets us self-contain the heuristic within the write path for each Parquet file.\nIn the future we may want to incorporate query usage patterns into the shredding decision.\nFor example if we see that user_id is frequently queried but is not a large attribute, we may want to shred it anyway.\nThis would require tracking query patterns over time and feeding this back into the ingestion/compaction pipeline.\nWe would also use this approach to decide what indexing strategies to pursue for each shredded column.

\n

Conclusion

\n

In the next few months you'll probably see your Logfire queries get orders of magnitude faster.\nYou don't need to do anything on your end, just continue to use Logfire as you currently do!

\n\n","headings":[{"level":2,"text":"Preface","id":"preface"},{"level":2,"text":"Logfire Data Model","id":"logfire-data-model"},{"level":3,"text":"How we started: fast JSON parsing","id":"how-we-started-fast-json-parsing"},{"level":3,"text":"Shredding columns to save IO and compute","id":"shredding-columns-to-save-io-and-compute"},{"level":3,"text":"The Bad: Limited Flexibility, High Cost for Unshredded Attributes and Overheads","id":"the-bad-limited-flexibility-high-cost-for-unshredded-attributes-"},{"level":2,"text":"The Solution: Dynamic Shredding","id":"the-solution-dynamic-shredding"},{"level":2,"text":"The Hard Path to Dynamic Shredding","id":"the-hard-path-to-dynamic-shredding"},{"level":2,"text":"Minimizing Breakage for Users","id":"minimizing-breakage-for-users"},{"level":2,"text":"How do we decide what to shred?","id":"how-do-we-decide-what-to-shred"},{"level":2,"text":"How are data types determined?","id":"how-are-data-types-determined"},{"level":2,"text":"Future work","id":"future-work"},{"level":3,"text":"Using the Parquet Variant type","id":"using-the-parquet-variant-type"},{"level":3,"text":"Dynamic indexing","id":"dynamic-indexing"},{"level":3,"text":"Usage based shredding","id":"usage-based-shredding"},{"level":2,"text":"Conclusion","id":"conclusion"},{"level":2,"text":"References and related work","id":"references-and-related-work"}]}}