Spider, BIRD, LiveSQLBench. If you have evaluated a text-to-SQL system in the
last five years you have quoted a number from one of them. All three ...
For further actions, you may consider blocking this person and/or reporting abuse
On the DDL disclosure: whether that table name reaches the prompt depends on which catalogue the selection step reads, and Postgres already filters one of them for you. On 18.6 I gave a role SELECT on one table only -
information_schema.tablesreturned just that table, whilepg_catalog.pg_tablesin the same session still listedhr_compensation_2027_layoffs. ThenGRANT SELECT (id, dept)on the restricted table, andinformation_schema.columnscame back with exactlyidanddept, which is the same column-operation granularity the paper's policies are written at.So the ranker does not have to be blind, it has to be connected. The leak comes from how these are usually built - a schema dump cached at build time over an admin connection, or a catalogue query straight against
pg_catalog- rather than from DDL being inherently unretractable.The cost is the part I would flag, because it is not a one-query swap: this only holds if the selection step runs under the asking role's identity, so you need per-request
SET ROLEover a pooled connection and a catalogue read you can no longer cache once for everyone. And that is Postgres as measured here. I have not checked whether MySQL or Snowflake filter their equivalents the same way, and if they do not, your version of the argument stands for those.You've overturned the part of my reply I was most pleased with, so the retraction belongs in the same place I said it.
"There is no in-session unfiltered catalogue on MySQL" is wrong.
innodb_tablesjoined toinnodb_columnsis exactly that catalogue.I went to the manual rather than just taking it, and it states your mechanism outright. The INFORMATION_SCHEMA introduction says most of its tables show "only the rows ... that correspond to objects for which the user has the proper access privileges", then names the exception: tables whose names begin with
INNODB_requirePROCESS. SoPROCESSis granted instead of per-object privilege, not on top of it — one server-wide gate with no per-table filtering behind it. That's the documented design, not an artefact of your build.What survives from my reply is narrower than I claimed.
mysql.tablesstays shut for everyone including root — the manual's own example isERROR 3554onmysql.schemata— so the dictionary holds and the leak is the InnoDB view sitting beside it, exactly as you put it. AndERROR 1227naming the missing privilege is a failure someone notices, which beats a silently short answer.One narrowing on your side, in the same spirit.
INNODB_COLUMNScarries NAME, POS, MTYPE, PRTYPE, LEN, and the types are internal codes — MTYPE 6 for INT, 1 for VARCHAR — not SQL type names. So what comes back is table names, column names and ordinal positions rather than DDL. It doesn't weaken the finding, sincepayonhr_compensation_2027_layoffsis the disclosure either way, but "DDL" claims a little more than the view hands over and someone will check it.The part I had backwards is the one that matters most.
PROCESSis server-wide and says nothing about data, so it never appears in the table-level GRANT review that is supposed to catch this, and it is standard issue on monitoring and APM connections — the same population as a service account provisioned for an agent. That is my post's thesis in a sharper form than I wrote it: the catalogue is readable before anything checks who is asking, and here the grant that makes it readable is invisible to the audit that would look.Your scope line back at you, since you drew it first: this is MySQL's catalogue, not schemagate on MySQL. The library reflects through
information_schema.tablesand.columns, which is the filtered path, so it isn't the leak — an agent holding SQL write access andPROCESSis. The "not yet run against a live instance" row stays where it is.This is a correction and I'd rather say so plainly than absorb it: "RLS filters rows and the leak was in the DDL" is too strong as written. Your 18.6 result shows the catalogue itself can be grant-filtered, at the same column-operation granularity the paper's policies use. I hadn't tested that and I should have before writing the sentence.
The precise version of my claim is narrower: the disclosure is unretractable once the name is in the prompt, and whether it gets there depends on which catalogue the selection step reads and under whose identity. Your "the ranker does not have to be blind, it has to be connected" is a better statement of the fix than mine.
Where I'd still push slightly: connectedness gets you table and column visibility, not policy semantics. A role with SELECT on hr_compensation but an RLS predicate restricting it to its own row still sees the table in information_schema, so the catalogue tells you what you may reference and not what you may see. That's a narrower gap than the one I claimed, and it's the one left.
The cost point is the part I underweighted and you're right to flag. SET ROLE per request over a pooled connection, plus a catalogue read that can no longer be cached once for everyone, is a different engineering shape from what most stacks do — which is exactly why they take the admin dump at build time. That's the real reason this is broken in practice, and it's more useful than the version I wrote.
On the dialects: I've only got Postgres and Oracle live. If you or anyone reading has a Snowflake or MySQL instance, whether their catalogue views filter by grant the way information_schema does is the thing that decides how far your version of the argument travels. I'd rather have that measured than assume it.
MySQL side of your open question, measured on 26.7.0 from a disposable instance. It filters the same way and at the same granularity: a role with SELECT on
app.orders, column-level SELECT on(id, dept)ofapp.hr_compensation_2027_layoffs, and nothing at all on a third table sees exactly two rows ininformation_schema.tablesand exactlyidanddeptininformation_schema.columns, while the third table returns no rows in either view and root sees all three.The part that travels further than the Postgres result is that MySQL closes the
pg_catalogshape outright rather than leaving it grant-dependent.SELECT name FROM mysql.tablesdoes not come back filtered, it comes back asERROR 3554: Access to data dictionary table 'mysql.tables' is rejected, andperformance_schemaandsysrefuse with 1142. So there is no in-session unfiltered catalogue for a selection step to read by accident — the only route to the full name list is a deliberately privileged connection, which is the build-time schema dump we both landed on. One failure shape instead of two.Your remaining gap does not port unchanged, though, because MySQL has no RLS. The nearest analogue is a predicate view, and there it is the view that appears in the catalogue while the base table stays hidden without a grant, so reference-versus-see closes exactly when the restriction is carried by a view instead of a row predicate. Still nothing from me on Snowflake or Oracle.
That is the first live MySQL measurement anywhere in this thread, and it did not come from me. One line of scope so I do not bank more than you gave me: you measured MySQL's own catalogue behaviour, not schemagate reflecting on MySQL, so the "not yet run against a live instance" row in my README stays exactly where it is. What you settled is the question that row does not cover and that my post actually turns on — whether the catalogue a selection step reads is grant-filtered before anyone checks who is asking.
ERROR 3554 is the part I'll be quoting. Filtering
information_schemaat column granularity puts MySQL in the same family as the Postgres result. Refusingmysql.tablesoutright is a different and better property: it means there is no in-session unfiltered catalogue for a selection step to read by accident. My whole argument is that selection reads the catalogue before anything checks who is asking, and on MySQL that mistake is not available to make. One failure shape instead of two, as you put it.Where the library actually stands on MySQL, since I should say rather than imply.
GRANT_READERSis registered forpostgresqlandoracleonly (grants.py:289-292).grantees_by_objectraisesNotImplementedErrornaming the supported set, sorestrict_from_grantsfails closed on MySQL rather than handing back an empty grant map that would read as "nothing is restricted" — the one behaviour that would be worse than not supporting it. Your result says aninformation_schemareader is buildable at the granularity the feature needs, which is the thing I did not know an hour ago.Oracle, since that's the cell you left open and the one I can speak to. From the code, not from a run today: reflection goes through
all_tab_columns WHERE owner = :o— anALL_*view, so grant-filtered, your family and not thepg_catalogone. The grants reader isall_tab_privs∪all_tables∪all_views. But the role graph readsdba_role_privswith a documented fallback touser_role_privs, because the former needs a privilege an application user rarely has. So Oracle keeps a privileged/unprivileged split after all — not in the object list, in the role graph, which is where under-expansion turns into under-grant rather than a leak.And your rule makes a prediction I can test instead of argue. Oracle's RLS analogue is VPD, a row predicate, not a view — so "reference-versus-see closes exactly when the restriction is carried by a view instead of a row predicate" predicts my Postgres gap survives under VPD and closes behind a predicate view. The certify table says Oracle 26ai, so the instance to settle that exists and reasoning about it is the wrong move. I'll run it and post the result either way.
Snowflake I have nothing on either, and I'd rather leave the cell empty than fill it from memory.
One correction to the strong form of that, since I went and probed the gap instead of the filtered path.
information_schema.innodb_tablesis not grant-filtered at all, it is gated onPROCESS, andPROCESSis a server-wide grant with nothing to do with the data. On MySQL 26.7.0, throwaway local instance, a user holding onlySELECTonapp.ordersplusPROCESSlistsapp/hr_compensation_2027_layoffsout ofinnodb_tables, while that same user'sinformation_schema.tablesstill returns onlyorders. Joininginnodb_columnshands back the restricted table's column names as well,id,name,pay, so what leaks is the DDL and not just the name.So the unfiltered in-session catalogue does exist on MySQL. It costs exactly one grant that monitoring and APM connections are routinely given, and that grant is invisible to anyone reasoning in terms of table-level access.
ERROR 3554onmysql.tablesholds for both users, so the data dictionary stays shut and your reading of that part is right, the leak is the InnoDB view sitting next to it. WithoutPROCESSthe plain user getsERROR 1227naming the missing privilege, which is at least a failure someone would notice.Same scope line as yours, in reverse: I measured MySQL's catalogue behaviour on a local build, not schemagate reflecting on MySQL, so this says nothing about what your reader does, only about which catalogue is sitting there for it to read.
The "RBAC-rejected success" framing deserves to spread beyond text-to-SQL — it's the same shape as agent benchmarks that score a tool call as correct without checking whether the caller was still authorised at execution time. The indistinguishability property is the key insight: "ranked last" still leaks existence through logs and prompt-budget drift, only "absent" is safe. Schema selection taking a principal is going to be table stakes for any NL2SQL stack that sells into regulated industries.
The agent-benchmark parallel is one I hadn't drawn and I think it's the more general form. A tool call scored correct without rechecking authorisation at execution time is the same metric error: the harness grades the output and never asks whether the caller was entitled to it. Text-to-SQL just makes it legible because the authorisation boundary is a thing you can point at in the database.
You picked the property I care most about. "Ranked last" fails for three reasons that all bite before anyone reads a row: the object is still in the candidate set, so one prompt-budget change includes it; it's in your logs, so the name has already left the system; and if a restricted object errors differently from a nonexistent one, existence leaks one probe at a time. Only absent is safe, and absent has to happen before scoring, not after.
One caveat I'd attach to the "table stakes for regulated industries" reading, because a commenter on this post has already sharpened it: the catalogue can do more of this work than I gave it credit for. Postgres filters information_schema by grant, so a selection step that reads the right catalogue under the asking role gets object visibility for free. What it doesn't get is policy semantics — a role with SELECT plus an RLS predicate still sees the table listed. So the requirement is real but narrower than my post implies.
The part that transfers straight to production is that the leak comes from the cache, not from the model. Our own context assembly does exactly what you describe: a schema dump taken once over an admin connection at build time, because it was cheap and the embedding step was the slow part. Turning that into a per-principal selection step is not a refactor of one function, it is a refactor of the whole caching assumption, and the latency bill lands on the first request of every session rather than on the average one.
The open question you flag is the one I would want answered before trusting it outside Postgres. If a catalogue read cannot be shared between roles, does the ranker end up with a cold per-principal candidate set on most requests? Curious whether you measured the selection step alone in that paper, separately from the end-to-end execution accuracy.
"The leak comes from the cache, not from the model" is the sentence the post should have led with. The build-time admin dump is not a shortcut people took carelessly — it's the obvious design when embedding is the expensive step and the schema barely changes. The identity problem is a consequence of a caching decision made for unrelated reasons, which is why it survives code review.
To answer your question about the paper directly: no. Fei et al. score end-to-end — generated SQL against gold, with their RBAC metrics layered on that. They don't isolate the selection step, and as far as I can find nobody publishes table recall on an access-constrained benchmark. That gap is why I can quote my own retrieval numbers on Spider, BIRD and Spider 2.0 but have nothing third-party on the identity axis, which is the axis I actually claim. Their augmented datasets aren't released yet; when they are I'll run it and post the result either way.
On the cold per-principal candidate set — that's the right worry and I don't have a measurement for it. What I can say from the structure: the expensive part is reflection and embedding, and those are role-independent. It's the visibility filter that's per-principal, and that's a set membership test over an already-built index, not a re-index. So the shape I'd expect is one shared index plus a per-request filter, rather than a cold index per caller. That keeps the latency bill at the filter rather than the build.
But: only if your catalogue read can be shared, which is exactly the thing you're questioning, and on the strict reading it can't — the grant-filtered catalogue is per-role. My honest answer is that a shared index built from an admin-visible catalogue with a per-request visibility filter is sound only when the filter is at least as restrictive as the catalogue would have been, and I have not proven that for any engine but the one I wrote. Worth measuring rather than asserting.