The NumPy and pandas Mental Model
NumPy and pandas are usually taught as two separate libraries with two separate APIs, but underneath the syntax they share one model.
//
Search across all documentation pages
NumPy and pandas are usually taught as two separate libraries with two separate APIs, but underneath the syntax they share one model.
Numeric data lives in contiguous, homogeneously-typed arrays, and almost every operation you write is a request to run a loop in compiled code instead of in the Python interpreter.
This page is about that shared model - what vectorization and broadcasting actually do underneath the syntax, how pandas' Series, DataFrame, and Index build labeled structure on top of NumPy's raw arrays, and where Polars' lazy engine breaks from this model entirely.
Data Analysis Basics walks through the concrete API for a first analysis; this page is about the machine underneath it.
NaNs when indexes don't match, and neither NumPy nor eager pandas plans ahead the way a lazy engine does.A NumPy ndarray is a block of contiguous memory holding values of a single dtype - all int64, all float64, or all of some other fixed-width type.
That homogeneity is what makes vectorization possible: because every element has the same type and size, NumPy can hand the whole array to a compiled loop (a ufunc, short for universal function) instead of asking the Python interpreter to visit each element one at a time.
A Python for loop over a list pays interpreter overhead - type checks, dynamic dispatch, object boxing - on every single iteration; a vectorized NumPy call pays that overhead once, then runs a tight C loop over raw bytes.
Broadcasting is the rule that lets arrays of different shapes combine without you writing an explicit loop or copy: when shapes don't match, NumPy compares them from the trailing dimension inward, and any dimension of size 1 is stretched to match its counterpart - conceptually, not physically, since no memory is actually duplicated.
A simple analogy: broadcasting is like handing every seat in a stadium the same flyer without printing one per seat - the flyer's content is reused, only the "position" changes.
pandas takes this array model and adds two things NumPy doesn't have: labels and heterogeneity across columns.
A Series is one NumPy array (or an Arrow-backed array) plus an Index - a labeled axis that gives each element an identity beyond its position.
A DataFrame is best understood as a dictionary of Series that all share the same row Index, which is why each column can hold its own dtype even though every value within a column still has to share one.
This is the detail that trips up newcomers coming from NumPy: pandas arithmetic and joins operate on alignment by label first, and position is almost incidental.
Vectorization and broadcasting explain why NumPy math is fast, but the interesting mechanics happen at the boundary between "vectorized" and "not."
Every NumPy ufunc call has to decide, before running, whether the inputs' shapes are broadcast-compatible; if they aren't, you get a ValueError, not silent wrong output, which is one of the model's better safety properties.
pandas layers alignment on top of that: adding two Series first reindexes both to the union of their indexes, and any label present in one but not the other produces NaN rather than raising - a design choice that trades safety for convenience, and one that silently manufactures missing data if you weren't expecting a mismatch.
That same alignment step is also why a DataFrame operation can be much slower than the equivalent raw NumPy array operation on the same data - the labels have to be reconciled every time, not just the numbers.
Memory layout matters too: NumPy arrays are contiguous and can be row-major (C order) or column-major (Fortran order), and an operation that walks against the grain of that layout pays a cache-locality penalty even though the vectorized loop still "works."
pandas' internal block manager historically stored same-dtype columns together for exactly this reason, and modern pandas (2.2+) increasingly backs columns with Arrow arrays, which changes the missing-value story: Arrow-backed nullable dtypes carry an explicit validity bitmap instead of relying on NaN, so integer and string columns can hold real missing values without upcasting to float64 the way classic NumPy-backed pandas had to.
Indexing is where the model's sharp edges show up most: .loc operates in label space, .iloc in positional space, and chained indexing (df[df.x > 0]['y'] = 1) can silently write to a temporary copy instead of the original frame, which is the mechanism behind SettingWithCopyWarning.
import numpy as np
a = np.array([[1, 2, 3], [4, 5, 6]]) # shape (2, 3)
b = np.array([10, 20, 30]) # shape (3,)
# b is broadcast across each row: shapes (2,3) and (3,) align on the
# trailing dimension, so b is conceptually repeated for every row
result = a + b # shape (2, 3), no explicit loop, no b copies storedThe array-and-alignment model scales well until data stops fitting comfortably in memory, and that's exactly where Polars' design departs from it.
Polars keeps the same columnar, Arrow-based memory idea, but its LazyFrame API does not execute anything when you write a transformation - it builds a query plan, and only collect() triggers execution.
That deferred model lets Polars' query optimizer push filters and column selections down before any I/O happens, so scanning a Parquet file with scan_parquet() and a .filter() can skip reading whole row groups or columns the plan proves are unnecessary.
pandas has no equivalent planning stage: every .assign(), .merge(), or .groupby() call executes immediately and materializes its full result, which is simple to reason about but means an unnecessary intermediate DataFrame is still built even if a later step throws most of it away.
Polars' eager API exists too and behaves more like pandas - execute immediately - but the lazy API is where the architectural difference actually pays off, particularly on data too large to comfortably hold multiple intermediate copies of.
None of this makes NumPy or eager pandas obsolete: for data that already fits in memory with a handful of transform steps, an optimizer has little to optimize, and the simplicity of immediate execution is often worth more than the deferred-planning overhead.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
NumPy ndarray | Fastest possible dense numeric math, minimal overhead | No labels, no heterogeneous columns, awkward for tabular/joined data | Pure numeric arrays: images, matrices, signal data |
| pandas (eager) | Rich labeled API, huge ecosystem, mature I/O | Executes every step immediately - no cross-step optimization; memory-bound | Exploratory analysis, small-to-medium tabular data |
| Polars (eager) | Faster than pandas on many ops via multithreading + Arrow | Still materializes every intermediate result | Medium data where you want speed without changing API style |
| Polars (lazy) | Optimizer fuses/pushes down filters and projections before execution | Query plan indirection adds a mental step; some pandas idioms don't map 1:1 | Large files/datasets, pipelines with several chained transforms |
NaNs a raw NumPy operation on the same numbers never would.collect() still has to run eventually..loc resolves labels, .iloc resolves integer positions, and on a DataFrame with a non-default or reordered Index these two can return entirely different rows for the "same" numeric argument.NaN for floats, which forced integer columns with missing data to upcast to float64; Arrow-backed nullable dtypes in pandas 2.2+ finally give integers and strings a real missing marker without that upcast.a + b on two NumPy arrays is dramatically faster than a Python for loop doing the same additions.Broadcasting is the rule that lets NumPy combine arrays of different but compatible shapes by conceptually (not physically) stretching any dimension of size 1 to match its counterpart, comparing shapes from the trailing dimension inward.
NaN in the result rather than raising.No - a DataFrame is closer to a dictionary of Series that share one row Index, which is why each column can hold its own dtype (string, int, datetime) even though a single NumPy array can only hold one dtype for the whole block.
.loc selects by label (what's in the Index), .iloc selects by integer position (0-based, ignoring labels).RangeIndex these often coincide, which hides the distinction until the Index is reordered, filtered, or non-integer..sort_values() or .reset_index() is a frequent source of "off by a row" bugs.It signals that a chained operation (like df[mask]['col'] = value) may have written to a temporary intermediate object instead of the original DataFrame, because pandas can't always guarantee whether an intermediate selection returned a view or a copy - the fix is to select and assign in a single .loc[mask, 'col'] = value call.
Classic NumPy-backed integer arrays have no bit pattern reserved for "missing," so pandas historically had to upcast an integer column to float64 the moment a NaN appeared, since only floats have a native missing-value representation - Arrow-backed nullable integer dtypes (pandas 2.2+) avoid this by carrying a separate validity mask.
Polars is built on Apache Arrow's columnar memory format from the ground up, which enables efficient multithreading and easier zero-copy interop with other Arrow-based tools; pandas has been migrating toward optional Arrow-backed dtypes since 2.2 but its default block manager still has NumPy-era roots.
pl.DataFrame) executes each operation immediately, like pandas.pl.LazyFrame) builds an unexecuted query plan and waits for .collect().Reach for Polars when data is large enough that scan-time filter/projection pushdown or multithreaded execution meaningfully matters, or when a pipeline chains many transforms over files bigger than comfortably fits in memory multiple times over; for small, exploratory, or ecosystem-heavy work (visualization libraries, older tooling) pandas' maturity often wins.
No - most Polars adopters use both: pandas for exploratory work, legacy integrations, and libraries that expect a pandas DataFrame, and Polars for the specific large or performance-sensitive stages of a pipeline, converting between the two with to_pandas()/from_pandas() when needed.
Because the underlying operation might be a single vectorized ufunc call in one case and a Python-level .apply() with a per-row function call in the other; both can look like df['x'].something(...), but only the vectorized path avoids per-element interpreter overhead.
.apply()/Python loop (interpreted, slow) before you write it.Stack versions: This page was written for Python 3.14 (stable) / 3.13 (maintenance), pandas 2.2+, and Polars 1.x.
Reviewed by Chris St. John·Last updated Jul 15, 2026