If you are still reaching for pandas + Jupyter by reflex, your stack is a generation behind.

Polars

A DataFrame library written in Rust with a query optimizer. The API is similar to pandas but eager and lazy execution are first-class. On 10M-row joins it is routinely 5-20x faster.

DuckDB

An in-process analytical SQL engine. Read Parquet, CSV, JSON or Arrow directly with zero ETL. Pair it with Polars and you have a local data warehouse on your laptop.

Marimo

A reactive notebook that stores files as plain .py. No hidden state, real git diffs, deployable as an app. Jupyter still has the mindshare, but Marimo wins on every objective axis.

A typical pipeline

import polars as pl
import duckdb
df = pl.scan_parquet("events/*.parquet")
agg = duckdb.sql("SELECT country, count(*) FROM df GROUP BY 1").pl()

Three tools, no Spark cluster, no Airflow, and it scales further than most people need.