KoliStat blog

the-stats-duck v0.7 — a regression in every GROUP BY

2026-07-06T00:00:00.000Z

the-stats-duck — our open-source DuckDB extension for statistics — just shipped v0.7.0, and it's now a one-line install from the DuckDB community extensions (see below).

v0.6 taught DuckDB to fit a regression: lm took an R-style formula and gave you back a coefficient table for the whole query. Good for one model. But clinical trial work is rarely just one model — it's usually one model per site, per arm, per cohort — and the standard errors you report had better survive a reviewer who knows the residuals aren't textbook-tidy.

So v0.7's headline is lm_fit: ordinary least squares as a DuckDB aggregate. One regression per GROUP BY group, with robust and cluster-robust standard errors on request.

One regression per group

Because lm_fit is an aggregate, GROUP BY does the fanning-out for free. Here it fits body_mass_g ~ flipper_length_mm + bill_length_mm separately for each penguin species, in a single query — live, in your browser:

Three species, three fitted models, one pass over the data. Change GROUP BY species to GROUP BY island and you get a model per island instead — the aggregate doesn't care.

The shape of it

lm_fit takes the response and the predictor row, and returns a STRUCT:

lm_fit(y, x [, vcov [, cluster [, add_intercept]]])

y — the response value (one per row).
x — the predictor row as a LIST(DOUBLE), e.g. [flipper_length_mm, bill_length_mm]. An intercept is prepended automatically unless add_intercept is set to false.
The result STRUCT carries a coefficients list — one {term, estimate, std_error, t_statistic, p_value} per predictor — alongside n, k, df_residual, r_squared, adj_r_squared, sigma, f_statistic, f_p_value, and the vcov_type actually used.

Predictors come back by position — (Intercept), then x1, x2, … in list order — so you unnest the coefficients to get a tidy row per term. On a tiny dataset that looks like this:

SELECT (u).term, (u).estimate, (u).std_error
FROM (
  SELECT unnest((lm_fit(y, [x1])).coefficients) AS u
  FROM (VALUES
      (2.1, 1.0), (3.9, 2.0), (6.2, 3.0), (7.8, 4.0),
      (10.1, 5.0), (12.2, 6.0), (13.8, 7.0), (16.1, 8.0)
  ) AS t(y, x1)
);
--   term         estimate  std_error
--   (Intercept)  0.0357    0.1404
--   x1           1.9976    0.0278

lm_fit is the aggregate, positional sibling of v0.6's formula-driven lm/lm_summary. Reach for lm when you want a formula over one table; reach for lm_fit when you want a model per group, robust standard errors, or both.

Standard errors that survive review

The point estimates are just OLS. What changes with the vcov argument is the standard errors — and with them the t-statistics and p-values. That's usually the number a methodologist actually argues about.

-- Heteroskedasticity-consistent (HC3) — the conservative small-sample default
SELECT (u).term, (u).std_error
FROM (SELECT unnest((lm_fit(y, [x1, x2], 'HC3')).coefficients) AS u FROM mytable);
--   term         std_error
--   (Intercept)  3.2802
--   x1           0.4749
--   x2           0.3839

The full menu:

'const' — classical OLS (the default).
'HC0'–'HC3' — heteroskedasticity-consistent. 'HC1' is Stata's , robust; 'HC3' is the conservative choice when n is small.
'CR0' / 'CR1' — cluster-robust, for data with repeated measurement inside a subject, site, or center. Pass a per-row cluster key ('cluster' is accepted as an alias for 'CR1', the Stata / statsmodels default), and inference uses a t(G − 1) reference with G clusters:

SELECT (u).term, (u).estimate, (u).std_error
FROM (
  SELECT unnest(
    (lm_fit(response, [dose], 'CR1', subject_id::VARCHAR)).coefficients
  ) AS u
  FROM observations
);

Only the covariance estimator moves — the coefficients are identical across all of them. Every estimator is validated against statsmodels' cov_type='HC*' and 'cluster'.

One nicety for the per-group case: a degenerate group — fewer rows than predictors, a singular design, or fewer than two clusters — returns NULL for that group instead of aborting the whole query. You get every model that can be fit and NULLs where one can't, which is exactly what you want when you've just fanned out across 200 sites.

bootstrap(), now identical on every machine

A quieter but important fix: a seeded bootstrap() now returns the exact same resamples on every operating system. Before v0.7, the same seed could draw a different set of resamples on Windows, macOS, and Linux — so a bootstrap result you shared with a colleague wouldn't reproduce number-for-number on their machine. The estimates were always statistically sound; they just weren't bit-for-bit identical across platforms.

Now they are: give bootstrap() a seed and anyone who reruns it, on any OS, gets the same resamples and the same interval. If you like this kind of bug, we wrote up two related RNG traps from the sampler work, and the full reasoning behind this one is in the engineering note.

Under the hood: a shared numerical core

lm_fit is built on a new, well-tested numerical core — the matrix machinery that regression and robust standard errors actually run on (the least-squares solve, the matrix inverses, and the "sandwich" that turns residuals into robust standard errors). It's deliberately self-contained and reusable, so the same carefully-checked code can back more of the extension over time: lm_fit is the first to use it, and the existing lm / lm_summary move onto it next. More of the statistics now stands on one foundation, validated the same way.

Go play

the-stats-duck is MIT-licensed and now lives in the DuckDB community extensions — so installing it is one line in any DuckDB session, no building from source:

INSTALL stats_duck FROM community;
LOAD stats_duck;

It's the statistics engine under Bedevere and KoliLang; the code and full v0.7.0 release notes live on GitHub. Drop it into any DuckDB session and your GROUP BY grows a regression department.

And if there's an estimator or test you need that isn't there yet, get in touch — commissioned functions ship in the open-source extension, so everyone gets them.

Not dead. What? Doing your regressions.

Story of an interesting bug, found (and solved) by Claude Code

2026-07-03T00:00:00.000Z

Most of the-stats-duck — our open-source DuckDB extension for statistics — is written with Claude Code doing the typing. This post is the story of the most interesting bug of the v0.6.0 release: the kind that returns plausible-looking output while being statistically wrong, the kind that slips past every code review and could sit in production for months.

It turned out to be two bugs, stacked. One was ours. One was the C++ standard library's. Claude Code wrote the first one, then found and fixed both — and the diagnostic trail is a nice little detective story, so we wrote it down.

The setup

v0.6.0 added the random-sampling family: rnorm, rt, rchisq, rf, rgamma, rbeta, rexp, rweibull, rlnorm, rpois. The design is classic inverse-CDF sampling: draw a uniform $u \sim \mathrm{Uniform}(0,1)$ , push it through the distribution's quantile function, done. The quantile functions already existed and were cross-checked against R — the r* family just had to compose them with a uniform generator.

The first draft of rexp(rate) looked like every other distribution scalar in the codebase:

static void RExp1Exec(DataChunk &args, ExpressionState &, Vector &result) {
    UnaryExecutor::ExecuteWithNulls<double, double>(
        args.data[0], result, args.size(),
        [](double rate, ValidityMask &mask, idx_t idx) {
            return SafeSample(
                [&](double u) { return stats_duck::ExponentialQuantile(u, rate); },
                mask, idx);
        });
}

If you have ever written a DuckDB scalar function, this is exactly what you would write. UnaryExecutor is the officially sanctioned helper that deals with all the vector plumbing for you. The function was marked VOLATILE, the flag that tells DuckDB "this is not a pure function, don't optimise calls away." Every box ticked.

It compiled. It ran. It returned random-looking numbers.

A mean that won't sit still

Then Claude Code ran the smoke test it runs on every sampler — check the moments against theory:

SELECT round(avg(rexp(1.0)), 2) AS mean,
       round(max(rexp(1.0)), 1) AS mx
FROM range(500000);

An exponential with rate λ = 1 has mean 1.0, and the expected maximum of 500,000 independent draws is a textbook extreme-value fact: about $\ln(500{,}000) + \gamma \approx 13.7$ .

We got a mean drifting between 0.92 and 1.1 across runs, and a max of about 8.

Not absurd numbers. Numbers a tired human would wave through — random is random, right? But the mean of 500k draws should sit within a hair of 1.00, and a max of 8 is a right tail that has been amputated. Something was wrong, and one more observation made it interesting: rnorm() — the zero-argument form — had perfect moments, while rgamma(2, 3) was slightly off. The error correlated with which executor pattern the function used, not with which distribution it sampled.

500,000 rows, 245 random numbers

The killer diagnostic was one query:

SELECT count(*) AS total, count(DISTINCT v) AS distinct_cnt
FROM (SELECT rexp(1.0) AS v FROM range(500000));

total: 500000
distinct_cnt: 245

Half a million rows. 245 unique values. And the ratio gives the whole game away: 500,000 / 245 ≈ 2041 — which is DuckDB's vector size of 2048. Each chunk of 2048 rows was getting exactly one random draw, copied 2048 times.

Here is why. DuckDB executes queries in chunks of up to 2048 rows, and UnaryExecutor has a fast path for constant inputs: when the argument is the literal 1.0, the executor reads the constant once, invokes your lambda once, and broadcasts the single result across the chunk. For a pure function, that is a perfectly sound optimisation. For a sampling function, it is catastrophic — and it also explains the clue above: rnorm() takes no arguments, so it never went through UnaryExecutor and had a hand-written per-row loop all along.

But we marked it VOLATILE!

We did. And this is the part worth writing on a wall:

FunctionStability::VOLATILE instructs the planner — don't constant-fold, don't common-subexpression-eliminate. It says nothing to the executor, whose constant-input fast path is a runtime optimisation that fires whether the function is volatile or not.

So VOLATILE correctly stops rnorm() + rnorm() from being folded into 2 * rnorm() at plan time, and then the executor cheerfully replicates one draw across each vector at run time anyway. Two different layers, two different notions of "don't cache this."

The fix: drop the helper and write the per-row loop explicitly via UnifiedVectorFormat, which still handles constant / flat / dictionary input encodings — but in a loop we control:

template <typename Quantile>
static inline void RunPerRow1(DataChunk &args, Vector &result, Quantile &&q) {
    idx_t count = args.size();
    UnifiedVectorFormat a0;
    args.data[0].ToUnifiedFormat(count, a0);
    auto v0 = UnifiedVectorFormat::GetData<double>(a0);
    result.SetVectorType(VectorType::FLAT_VECTOR);
    auto out = FlatVector::GetData<double>(result);
    auto &mask = FlatVector::Validity(result);
    for (idx_t i = 0; i < count; i++) {
        auto i0 = a0.sel->get_index(i);
        if (!a0.validity.RowIsValid(i0)) {
            mask.SetInvalid(i);
            (void)NextUniform(); // keep RNG advance independent of NULLs
            out[i] = 0.0;
            continue;
        }
        out[i] = SafeSampleOne(
            [&](double u) { return q(u, v0[i0]); }, mask, i);
    }
}

(Note the small courtesy in the NULL branch: the RNG advances anyway, so the stream of draws doesn't depend on where the NULLs happen to sit.)

After the fix: 500,000 rows, 500,000 distinct values, mean ≈ 0.997. Bug one down.

The clipped tail

Except the story wasn't over. The distribution's broad shape was now right, but the right tail of rexp(1.0) still came up light. Claude Code compared tail counts against what Exp(1) theory demands for 500k draws:

Statistic	Expected (Exp(1), n=500k)	Observed
`count(v > 5)`	≈ 3,370	4,096
`count(v > 8)`	≈ 167	0
`count(v > 10)`	≈ 23	0
Implied max of `u` (= 1 − exp(−max v))	≈ 1 − 1.5×10⁻⁶	≈ 0.999665

The largest uniform draw in half a million samples was about 0.999665. The probability of that happening by chance is $(0.999665)^{500{,}000} \approx e^{-167} \approx 10^{-72}$ . That is not bad luck. Something was clipping u from above.

The uniform generator was this:

std::uniform_real_distribution<double> dist(
    std::nextafter(0.0, 1.0),
    std::nextafter(1.0, 0.0));
double u = dist(rng);

Textbook C++. Deliberately an open interval (0, 1), so the inverse-CDF transforms downstream — log(1-u), qnorm(u) — can never hit the endpoints. A code reviewer nods and moves on.

But on the MSVC STL our Windows builds ship against, that distribution does not deliver uniform draws across its declared support — the upper tail is squeezed in. Same story for std::generate_canonical<double, 53>. We didn't excavate the implementation to pin the exact mechanism; there are years-old reports against MSVC's <random> with similar symptoms, and we needed a working generator more than a culprit.

So the fix sidesteps the library entirely: take one 64-bit draw from mt19937_64, keep the top 53 bits, scale. Every representable double in [0, 1) at 53-bit precision gets equal probability mass:

static inline double NextUniform() {
    auto &rng = TLSRng();
    uint64_t bits = rng() >> 11;            // keep the top 53 bits
    double u = static_cast<double>(bits) * (1.0 / static_cast<double>(1ULL << 53));
    if (u <= 0.0) {
        u = std::numeric_limits<double>::min();
    }
    return u;
}

The upper bound is 1 − 2⁻⁵³ by construction, so no nudging needed there; the lower side is bumped off exact zero to keep log(0) and qnorm(0) out of reach. After this, the tail counts match Exp(1) theory to three significant figures out to the 10⁻⁵ quantile — and the smoke-test max lands right where extreme-value theory says it should, around 13.7.

(In hindsight, that very first max ≈ 8 was both bugs compressed into a single number: 245 effective draws would put the expected maximum near 6, and the clipped generator couldn't produce anything above 8.0 regardless.)

Both bugs pass code review

What makes this pair worth a blog post is that neither bug is visible in the code.

The first draft of rexp is idiomatic DuckDB — it uses the recommended executor helper and the documented volatility flag. The uniform generator is idiomatic C++ — std::uniform_real_distribution is the API every textbook tells you to use. You could put both snippets in front of ten strong engineers and collect ten approvals.

The bugs only exist statistically. And statistics is exactly the thing a review diff doesn't show. Worse: checking moments alone isn't enough either — a generator that clips its upper tail keeps the mean and standard deviation close to theory while silently ruining every Monte Carlo workload that cares about tails, which is roughly all of them.

The thing that caught both was a boring, mechanical verification loop: after implementing, check the output against theory — moments, then distinct counts, then tail counts — and refuse to shrug when 500k draws average 0.92 instead of 1.00. This is precisely the kind of discipline an agent is good at, because it never gets bored and never decides "eh, random is random." Claude Code wrote the naive first draft, like most of us would have. It also ran the checks that most of us, honestly, would have skipped — then chased the anomaly down through DuckDB's executor internals into the standard library, and wrote the engineering note so the next contributor — human or agent — starts from the pattern that works.

The checklist

Distilled from the wreckage, for anyone shipping sampling functions anywhere:

count(*) vs count(DISTINCT v). If a function is supposed to vary per row and these disagree, you have an executor caching problem. Full stop.
Tail counts against theoretical probabilities. count(v > threshold) vs n · P(V > threshold). Catches uniform-generator bias that moment checks can't see.
Inverse round-trip. pnorm(rnorm()) must be Uniform(0, 1): mean 0.5, stddev $1/\sqrt{12}$ .
Run it more than once. Thread-local RNGs seed per thread, so symptoms drift between runs. One weird number might be sampling noise; a moving target across reruns is a strong signal.

Trust, but count(DISTINCT)

The fixes landed before v0.6.0 shipped, so every released build samples correctly. Don't take our word for it — this is the shipped build, running in your browser right now:

SELECT
  count(*) AS total
  , count(DISTINCT v) AS distinct_cnt
  , round(avg(v), 3) AS mean
  , round(max(v), 1) AS mx
FROM (SELECT rexp(1.0) AS v FROM range(500000));

Half a million rows, half a million distinct values, mean within a hair of 1, max where the theory puts it.

the-stats-duck is MIT-licensed and runs anywhere DuckDB runs — it's the statistics engine under Bedevere and KoliLang. The full engineering note, with the complete diagnostic trail, lives in the repo. And if there's a distribution or test you need that isn't there yet, get in touch — commissioned functions ship in the open-source extension, so everyone gets them. Carefully sampled.

the-stats-duck v0.6.0 — statistics that live in your SQL

2026-06-22T00:00:00.000Z

the-stats-duck — our open-source DuckDB extension — just shipped v0.6.0, and in keeping with the whole operation, the release is named i-m-not-dead. It very much isn't!

In case you haven't met it before: the-stats-duck is the statistics engine that Bedevere and KoliLang lean on. It allows DuckDB to do real statistics — distributions, tests, regression, even plots — without ever leaving SQL. MIT-licensed, and it runs anywhere DuckDB runs — including a browser!

Which is quite handy, because every demo below is a live Bedevere instance running the-stats-duck right here in your browser. The dataset is the famous Palmer Penguins — change the SQL and re-run it.

SELECT * FROM 'penguins';

Profile a whole table in one shot: `meta()`

The first thing anyone would usually do with a new dataset is squint at it. meta() squints for you the full profile — one row per column:

SELECT
  column_name, kind, n_missing
  , n_distinct, mean, median, stddev, top
FROM meta('penguins')
;

It overlaps with DuckDB's built-in SUMMARIZE, but meta() is a table function — so you can join it, filter it, and compose it in CTEs.¹ Do you need to know "how many numeric columns, and how many missing values total?" That's just aggregation over meta().

Regression without leaving SQL: `lm` + an R-style formula

Yes, ordinary least squares, in a SQL query, with the formula syntax you already know:

SELECT *
FROM lm_summary(
  'penguins'
  , formula := 'body_mass_g ~ flipper_length_mm + bill_length_mm'
);

lm() gives you the coefficient table (estimate, std_error, t_statistic, p_value per term); lm_summary() gives the model-level line (R², adjusted R², F and its p-value, residual df, sigma). The formula language handles additive predictors and intercept removal (- 1); interactions and inline transforms aren't in v0.6 yet, but we might add them soon!

Under the hood it's a Cholesky solve of X'X, complete-case filtered, and cross-verified against R on cars and mtcars to four decimals.

Confidence intervals by brute force: `bootstrap()`

Don't want to assume normality? Resample:

WITH b AS (
  SELECT bootstrap(body_mass_g, 'mean', 2000, 42) AS samples
  FROM penguins
)
SELECT
  list_aggregate(samples, 'quantile_cont', 0.025) AS lo
  , list_aggregate(samples, 'quantile_cont', 0.975) AS hi
FROM b
;

bootstrap(value, statistic, n_iters [, seed]) resamples with replacement and hands back a LIST<DOUBLE> of the statistic from each resample (mean, median, stddev, …). Pair it with list_aggregate(list, 'quantile_cont', p) for a percentile interval. Pass a seed and it's reproducible — even across GROUP BY groups.

Charts, straight from SQL: `VISUALIZE … DRAW`

the-stats-duck also speaks a small — for now — plot grammar (ggsql) that compiles to Vega-Lite, which Bedevere renders. v0.6.0 adds violin marks, 2-D facets (FACET BY row, col), and a per-layer STAT smooth (LOESS) — so DRAW point DRAW line STAT smooth is your scatter-with-trend in one line.

VISUALIZE
  species AS x
  , body_mass_g AS y
FROM penguins
DRAW violin
TITLE 'Body Mass (g) by species'
;

Scatter with a fitted line: `STAT smooth`

The other half of v0.6.0's chart story is the per-layer STAT modifier — appended after a mark as DRAW <mark> STAT <name>. It transforms that layer only, so you can stack a raw mark under a statistical one. The canonical use is a scatter with a LOESS overlay:

VISUALIZE
  bill_depth_mm AS x
  , bill_length_mm AS y
  , species AS color
FROM penguins
DRAW point
DRAW line STAT smooth
SCALE x ZERO false
SCALE x LABEL 'Bill Depth (mm)'
SCALE y ZERO false
SCALE y LABEL 'Bill Length (mm)'
TITLE 'Penguins Bill Depth VS Bill Length (Smooth)'
;

Three modifiers ship. smooth injects a Vega-Lite LOESS transform on (x, y), grouped by color when it's mapped — one fitted curve per species above. summary rewrites the layer's data SQL to AVG(y) GROUP BY x [, color, facet, facet2] ORDER BY x, collapsing each cell to its mean. identity is the explicit no-op. smooth is rejected on marks that already emit their own transform — regression, density, violin, histogram — which bake the statistic in themselves.

A straight-line fit: `DRAW regression`

Where STAT smooth traces the data's local shape, the regression mark draws a single straight line — a least-squares y ~ x fit, grouped by color. Same axes as above, but each species gets a linear trend instead of a LOESS curve:

VISUALIZE
  bill_depth_mm AS x
  , bill_length_mm AS y
  , species AS color
FROM penguins
DRAW point
DRAW regression
SCALE x ZERO false
SCALE x LABEL 'Bill Depth (mm)'
SCALE y ZERO false
SCALE y LABEL 'Bill Length (mm)'
TITLE 'Penguins Bill Depth VS Bill Length (Regression)'
;

Ten new distributions (and the full d/p/q/r)

The distribution zoo grew: negative binomial, hypergeometric, Weibull, log-normal, and Poisson, each with the R-style d / p / q triple — plus a complete set of random samplers: rnorm, rt, rchisq, rf, rgamma, rbeta, rexp, rweibull, rlnorm, rpois. Every one cross-checked against R to six decimals. Simulate, fit, and test without round-tripping to another language.

Here's the Poisson(λ = 3) probability mass function — dpois evaluated over 0…10 and drawn as a bar chart, all in SQL:

WITH pois AS (
  SELECT k, dpois(k, 3) AS pmf
  FROM range(0, 11) AS t(k)
)
VISUALIZE
  k AS x
  , pmf AS y
FROM pois
DRAW bar
;

And it got fast: `read_stat` is 52x quicker

Last, and this one's for the XPT crowd: reading SAS / SPSS / Stata files used to be accidentally O(N²): the reader re-parsed the file from byte zero for every 2,048-row chunk! 😱

v0.6.0 parses each file exactly once into a buffered, spillable column store and streams from it. Here are the numbers:

a 200k-row, 7 MB XPT: ~67 s → ~1.3 s (52x)
the real CDISC pilot qs.xpt (122k rows): ~39 s → ~1.1 s (36x)
1.6M rows: ~70 min → ~15 s

Go play

the-stats-duck is open source: github.com/KoliStat/the-stats-duck (full v0.6.0 notes here). It powers Bedevere and KoliLang, but it's just a DuckDB extension: drop it into any DuckDB and your SQL grows a statistics department.

Not dead. Doing your statistics.

A reader rightly points out that DuckDB's SUMMARIZE can be used in a CTE too — you just wrap it in a FROM subquery: WITH some_table AS (FROM range(10)), x AS (FROM (SUMMARIZE some_table)) FROM x;. The practical difference is ergonomics: because meta() is a table function, it composes (join, filter, CTE) without the FROM (…) wrapper. ↩

You don't know XPT files

2026-06-17T00:00:00.000Z

...unless you belong to the magical world of clinical trials! ¹ ²

Every clinical trial headed for the FDA produces a fresh pile of XPT files. It's nearly 40 years old, stores its numbers in IBM hexadecimal, and caps variable names at 8 characters. It is also not going anywhere, because regulators still mandate it.

Show them to me!

First of all, would you like to open an XPT file? Well, it just happened that I launched a web app that does exactly that (among many other wonderful things, of course), and it happens that this might be the blog of the website that tries to promote it, in some way. If you're not interested in the app but you stick around, you might still learn a thing or two from this post.

The app is Bedevere Wise, named after the acute and science-fond knight of the Round Table. Here is how it works: you take your XPT file, you drop it into the app, and voilà, magic!

Well, you can also try to drop many other kinds of files used in the clinical trial world, like CSV, SAS7BDAT, Stata and even Excel! Or you can let the app browse a folder for you. And everything stays in your browser! Even though, for the most picky ones, a desktop version of the app is on its way to be released soon — perfect even for the most air-gapped, strictly-secure, highly-regulated environments!

In case you are curious to try, but you don't have an XPT file handy, here is a GitHub repo with a full clinical trial dataset and plenty of XPT files that you can download.

Or, if you just want a couple of tiny files to poke at, grab these unimpressive, synthetic, test samples:

So, what are these damn files?

I'd like to start by saying, "According to Wikipedia...", but unfortunately, there's no page on Wikipedia about the topic. However, there is a page about the SAS Transport File Format (XPORT) Family in the Digital Preservation section of The Library of Congress of the United States. The page says:

The SAS Transport File Format is an openly documented specification maintained by SAS, a commercial company with a variety of software products for statistics and business analytics, including the application now known as SAS/STAT, which originated in the late 1960s as SAS (an acronym for Statistical Analysis System) at North Carolina State University. The transport format was originally developed in the late 1980s when the corporate entity was known as SAS Institute, Inc. and the software as SAS, to support data transfers between statistical software systems, especially between SAS applications running on different operating systems. SAS considers it non-proprietary. ³

Basically, it's a dinosaur of the computer age! An almost 40-year-old tabular data format, designed to store things that can be represented as tables, with headers and columns that are either text or numbers.

And given the age, it comes as no surprise that XPT files are on the list, since over their long life they simply became ubiquitous in the pharma/clinical trials world. Hence the reason to publicly preserve the format.

And the reason they became — and stay — so ubiquitous? Regulation. If you want to submit the data of a clinical trial to the FDA (and it's a similar story for Japan's PMDA), you don't really get to choose your file format: submission datasets have to be provided as SAS Transport Format Version 5. Yes, the .xpt we've been talking about. So every study headed for a regulatory submission produces a fresh pile of XPT files. And that pile isn't shrinking any time soon.

It shouldn't surprise you, then, that many other, more familiar file formats are in the Library of Congress too, like PDF, JSON or ZIP. The reason is quite simple: those formats have become essential in our everyday lives, not only for people in IT or software engineers, but for everyone. Even though most people don't know all the technical details and the specifics of the format, everyone relies on them, and there must be ways to know how to interact with those formats. They became public protocols, like the ones we use to send internet packets or to open web pages.

And XPT files are no less than that! You — small pharma company — want to establish a formal communication with the FDA? Then, you speak their language and submit XPT files.

Wait, I thought it was all IEEE 754!

If you managed to stick around up to this point, you might either be a bit of a masochist, an LLM crawler, or simply interested in some more technical details. And well, from a software engineering perspective, there is at least one worth mentioning: numeric values in the XPT format are stored using the IBM float representation (yes, there is a page on Wikipedia about it). So, in order to be used by modern programming languages, they must be converted into the more convenient IEEE 754 floating point standard. So no, even in 2026 it's not all IEEE 754. ⁴

If you speak Python, here is the code used by Pandas, which I first ~~copied~~ re-implemented in one of my Go libraries. Since that implementation is a bit too long and detailed, I'll use a shorter version here:

# A numeric whose first byte is one of these *and* whose remaining
# 7 bytes are all zero is a SAS missing value rather than a real number:
#     '.'      -> 0x2E         ordinary missing
#     '_'      -> 0x5F         special missing  ._
#     'A'..'Z' -> 0x41..0x5A   special missings .A .. .Z
MISSING_FIRST = {0x2E, 0x5F} | set(range(0x41, 0x5A + 1))


def ibm_to_double(raw):
    """
    Decode up to 8 bytes of IBM hex float.
    
    Returns a Python float for real numbers, or None for a SAS missing
    value. (The specific missing code, '.'/'_'/'A'..'Z', is available
    as the byte `raw[0]` if you need to distinguish them.)
    """
    
    # short numerics carry only the high-order bytes
    b = raw.ljust(8, b"\x00")
    
    # SAS missing value: a marker byte followed by nothing but zeros.
    if b[1:] == b"\x00" * 7 and b[0] in MISSING_FIRST:
      return None
    
    n = int.from_bytes(b, "big")
    
    # check for 0
    if n == 0:
      return 0.0
    
    sign = -1.0 if n >> 63 else 1.0
    exponent = (n >> 56) & 0x7F            # excess-64, power of 16
    fraction = n & 0x00FFFFFFFFFFFFFF      # low 56 bits = integer mantissa
    
    return sign * fraction * 16.0 ** (exponent - 78)

ibm1 = bytes.fromhex("41 50 00 00 00 00 00 00")
print(ibm_to_double(ibm1)) # 5.0

ibm2 = bytes.fromhex("C1 50 00 00 00 00 00 00")
print(ibm_to_double(ibm2)) # -5.0

ibm3 = bytes.fromhex("41 13 c0 83 12 6e 97 8e")
print(ibm_to_double(ibm3)) # 1.2345000000000002

ibm4 = bytes.fromhex("3d 81 72 5b 67 2e e3 40")
print(ibm_to_double(ibm4)) # 0.00012345

Ok, even if you are familiar with Python, that might look a bit too low-level, but that's how it is: moving from one numeric format to another means getting your hands dirty shifting bits around. Still, it's not as long and complicated as the Pandas implementation; here we are going straight to the point.

Decoding a number, bit by bit

First of all, how is the numeric value actually represented? Looking at Wikipedia, we know that it can be represented with 4 bytes (32 bits), laid out like this:

1 7 24

Sign 1 bit · Exponent 7 bits · Fraction 24 bits

However, there might be cases in which the fraction uses 4 more bytes (56 bits in total), because the value is stored using the double precision format⁵:

1 7 56

Sign 1 bit · Exponent 7 bits · Fraction 56 bits

Nothing really changes, since we read the most significant digits from left to right, and that's why we can just pad the bytes on the right with 0s until we get 8 bytes in total⁶:

b = raw.ljust(8, b"\x00")

Next step: we check whether the number is a SAS missing value (the format's take on Not-a-Number). That's as simple as checking whether the first byte is one of the MISSING_FIRST special values while the remaining 7 bytes are all 0s. Then we check whether the number is exactly 0 — nothing special here either, all bits must be 0. To check that, we read the value as a big-endian integer, which will also come in handy for the next step.

Now that we handled the special cases, we can do the actual conversion. From Wikipedia, we get:

(-1)^{\text{sign}} \times 0.\text{significand} \times 16^{\text{exponent} - 64}

which is almost what we have in the last instruction of our ibm_to_double function:

return sign * fraction * 16.0 ** (exponent - 78)

Where does that 78 come from, instead of the 64 in the formula? It's the same thing, just accounted for differently. Wikipedia treats the 56 fraction bits as 0.significand, i.e. a binary fraction between 0 and 1. We, on the other hand, are lazy and read those same 56 bits as a plain integer (that's the big-endian int.from_bytes from before). Reading 56 bits as an integer instead of a fraction means multiplying by 2^56, which is 16^14, so we have to compensate by lowering the exponent by 14: 64 + 14 = 78. No magic, just moving the same factor from one place to another.

Sign. If the first bit is 1, the number is negative, so we shift away everything but that first bit and use it in an if/else. Take the first example in the code, 41 50 00 00 00 00 00 00, which represents 5.0. The first byte 41 is 0100 0001 in binary. Shift everything 63 places to the right with the >> operator, and we're left with 63 0s plus that original first bit — still a 0 — so the number is positive. If we consider the second example instead, C1 50 00 00 00 00 00 00, the first byte C1 in binary is 1100 0001, and since the only thing that changed was the first bit, we get -5.0.

Exponent. We've already seen the shift used to select the first bit of our value. This expression

exponent = (n >> 56) & 0x7F

selects the first byte instead (>> 56) and ignores the first bit (& 0x7F), which is the sign, of course, and we don't need it here. If we select the first byte from the first and the second examples we have 0100 0001 and 1100 0001, which, after the bitwise and with 0x7F (0111 1111), both give 0100 0001, which in base 10 is 65. Then 65 - 78 is -13, which doesn't really make sense on its own, since the number we want is 5.0 or -5.0.

Fraction. That -13 only starts to make sense once we bring in the fraction — everything but the first byte, obtained with & 0x00FFFFFFFFFFFFFF: 00 50 00 00 00 00 00 00, which in base 10 is 22517998136852480.

So, if we put everything together, we get

1.0 \times 22517998136852480 \times 16^{-13}

which is exactly 5.0! The magic of floating point representation!

Everything else

Good! Now that we've sorted that out, everything else about the XPT format is quite straightforward. There are a handful of version numbers floating around (5, 6, 8, 9), but really just two families: the classic v5/6 — 8-character variable names, which is what we'll parse here — and v8/9, which mostly exists to allow longer variable names (through a few extra header records). Either way it's the same idea: a stack of fixed-layout records that look more or less like this (without the newline characters):

HEADER RECORD*******LIBRARY HEADER RECORD!!!!!!!000000000000000000000000000000
SAS     SAS     SASLIB  9.4     X64_10PR                        25NOV23:15:18:17
25NOV23:15:18:17
HEADER RECORD*******MEMBER  HEADER RECORD!!!!!!!000000000000000001600000000140
HEADER RECORD*******DSCRPTR HEADER RECORD!!!!!!!000000000000000000000000000000
SAS     VALUES  SASDATA 9.4     X64_10PR                        25NOV23:15:18:17
25NOV23:15:18:17
HEADER RECORD*******NAMESTR HEADER RECORD!!!!!!!000000000400000000000000000000

Take that second record — the one starting with SAS — it's the library's real header, and it decodes into a row of fixed-width fields:

SASmarker · 8 SASmarker · 8 SASLIBmarker · 8 9.4version · 8 X64_10PROS · 8 ·····padding · 24 25NOV23:15:18:17created · 16

The real header record: 80 bytes total — most fields are 8 bytes wide, with 24 bytes of padding and a 16-byte timestamp.

Apart from the dates (creation and last update), operating system, and SAS version, what matters here are the 4th and the last records:

...
HEADER RECORD*******MEMBER  HEADER RECORD!!!!!!!000000000000000001600000000140
...
HEADER RECORD*******NAMESTR HEADER RECORD!!!!!!!000000000400000000000000000000

The number 140 at the end of the 4th record tells us the number of bytes for the variable descriptor structure, while the last record tells us how many variables to expect (4 in this case). If that number 4, floating among a bunch of 0s, looks weird to you, you're not alone. Why did they decide to put it there, and not at the very end? Who knows...

Ok, now that we know how many variables we need to read next, we can start parsing them into this structure:

struct NAMESTR {
    short ntype;    /* VARIABLE TYPE: 1=NUMERIC, 2=CHAR */
    short nhfun;    /* HASH OF NNAME (always 0) */
    short nlng;     /* LENGTH OF VARIABLE IN OBSERVATION */
    short nvar0;    /* VARNUM */
    char8 nname;    /* NAME OF VARIABLE*/
    char40 nlabel;  /* LABEL OF VARIABLE */
    char8 nform;    /* NAME OF FORMAT */
    short nfl;      /* FORMAT FIELD LENGTH OR 0 */
    short nfd;      /* FORMAT NUMBER OF DECIMALS */
    short nfj;      /* 0=LEFT JUSTIFICATION, 1=RIGHT JUST */
    char nfill[2];  /* (UNUSED, FOR ALIGNMENT AND FUTURE) */
    char8 niform;   /* NAME OF INPUT FORMAT */
    short nifl;     /* INFORMAT LENGTH ATTRIBUTE */
    short nifd;     /* INFORMAT NUMBER OF DECIMALS */
    long npos;      /* POSITION OF VALUE IN OBSERVATION */
    char rest[52];  /* remaining fields are irrelevant */
};

I know, that's C, a language that's too scary for many programmers nowadays!

But that's how it's reported in the technical documents, and if you want to read an XPT file in a different language (or even C itself), you need to know how short a short is and how long a long is.

After we read the variable descriptors, we get another header record:

HEADER RECORD*******OBS     HEADER RECORD!!!!!!!000000000000000000000000000000

and after that, if we got the type and size of each variable right, we can finally read the observations! (row by row)

Final Thoughts

If you made it this far, you now know more about IBM hexadecimal floats than you probably ever wanted to. The good news: the whole point of Bedevere Wise is that you don't have to. Drop your XPT file in and read it right away — no bit-shifting required. And it's about to get better: the next release makes reading XPT files faster and lets you export to XPT as well, so you can round-trip them instead of just opening them.

There are some well-known limitations though. If you pay attention to the variable descriptor structure, you might notice this:

struct NAMESTR {
    /* ... */
    char8 nname;    /* NAME OF VARIABLE*/
    /* ... */
};

which means that variable names can't be longer than 8 characters — at least in the classic v5/6 files (the ones regulators want), since v8/9 lifts the limit, as we saw. Yes, that's quite a constraint, but that's how it was designed! And, if you're familiar with clinical trials, I'm sure you've seen cryptic names like these: R2A1LO, AENTMTFL, or LBNRIND. Now you know why.

Did I mention XPT files enough?

Actually, if .xpt rings a bell and you've never been near a clinical trial, you might know it from Mozilla: XPCOM used .xpt files (XPConnect Typelibs) to store compiled interface metadata. If that's you: I'm sorry, this post really isn't for you. :( ↩
Or maybe you belong to an economics department. ↩
I was quite amazed myself to discover how easily I could grab the technical documents that I needed when I first started implementing my first XPT parser in Go, before the LLM/agentic Age (2 BVC, Before Vibe Coding, to be precise; here is the commit). ↩
Ok, I already know what to expect here: yes, for some applications, there might be more appropriate floating point representations. ↩
Strictly speaking, XPT always uses the 8-byte IBM double layout and just truncates it to fewer bytes. A truncated double happens to be bit-for-bit identical to IBM single precision — both share the 1-bit sign and 7-bit exponent, and the double simply carries more fraction bits — which is exactly why right-padding the missing low bytes with zeros is lossless. ↩
Numeric values aren't always 4 or 8 bytes: a variable can be any length from 2 to 8 bytes, set per variable by the nlng field in the NAMESTR struct you'll meet below. Shorter numerics keep only the high-order bytes, so padding back out to 8 recovers the full double. ↩

Welcome to the KoliStat blog

2026-06-10T00:00:00.000Z

KoliStat is the umbrella over three products built around the same question: what would clinical and tabular data tooling look like if it were built today, with the web as the runtime and DuckDB performance underneath?

KoliLang — the browser-native SAS engine: closed-source, compiled to WebAssembly, built to match SAS byte-for-byte, and live at editor.kolistat.com.
Bedevere — the tabular data visualizer and DuckDB SQL editor.
the-stats-duck — the open-source DuckDB extension that supplies the statistics.

This blog is for the engineering notes that don't fit on a product page: pieces of the SAS dialect we had to model from scratch, DuckDB extension internals, WASM ABI design decisions, the macro evaluator, the PROC FREQ output spec, and why a particular byte-parity bug took two weeks to find.

Cadence will be irregular. The work itself drives the topics: when something interesting lands in KoliLang, Bedevere, or the-stats-duck, it gets a post. Subscribe via the Atom feed or just check back — no newsletter, no comment section. That part is by design.