Mew: Minimal Table Retrieval over Data Lakes

Mew answers a question over a data lake in two stages. Offline, it profiles every table with an LLM, embeds columns and tables into FAISS indexes, and builds a join graph. Online, for each question it generates an answer-table sketch, retrieves and aligns candidate tables, grounds the sketch in the top-10 candidates with one LLM call that also picks focus tables, and then selects a small connected set of tables with a join-aware greedy algorithm.

Environment Setup

Create a Python environment and install dependencies

conda create -n mew python=3.12 -y
conda activate mew
pip install -r requirements.txt
export OPENAI_API_KEY=...      # not needed if you pass --llm-base-url to use an open model

Original Datasets

ID	OSS Component Name	Modified	Copyright Holder	Upstream Link	License
1	BIRD (dev set)	No	Alibaba Research	link	CC BY-SA 4.0
2	Spider (dev set)	No	Yale University	link	CC BY-SA 4.0
3	birdwild / spiderwild*	Yes	The University of Queensland	link	Not specified

* We modified birdwild and spiderwild only by unioning tables that share the same schema, producing BirdOpen and SpiderOpen. BirdOpen-small and SpiderOpen-small are random samples of these data lakes.

Open Source Software (OSS) Components: Baselines

ID	OSS Component Name	Modified	Copyright Holder	Upstream Link	License
1	DMRAL	No	The University of Queensland	link	Not specified
2	CORE-T	No	UKP Lab, TU Darmstadt	link	Apache-2.0
3	REAR	No	Arizona State University	link	CC-BY-4.0
4	JAR	No	MIT	link	Not specified

Pipeline (Offline → Online → Evaluate)

Run the steps for one benchmark. B is any value of --benchmark above. Every script accepts --help.

B=bird

1. Build semantic profiles

LLM descriptions of every table and column (from the schema and the first five rows) plus value statistics.

python offline_profiling.py --benchmark $B

Output in datasets/$B/: semantic_profiles.pkl, data_profiles.pkl, datalake_cache.pkl.

Argument	Description	Default
--benchmark	benchmark to profile	bird
--model-name	LLM for the descriptions	gpt-5.4-mini
--llm-base-url	OpenAI-compatible server for an open model	OpenAI API
--resume-semantic	only profile tables missing from an existing semantic_profiles.pkl	off
--output-dir	write the profiles elsewhere (use for a non-default LLM)	datasets/<benchmark>

---

2. Build embeddings and indexes

Embeds every column and table and writes the FAISS indexes.

python offline_embeddings.py --benchmark $B

Output in datasets/$B/: column_index.faiss, column_embedding_index.faiss, table_index.faiss, their *_metadata.pkl files and profile_inverted_index.pkl.

Argument	Description	Default
--benchmark	benchmark	bird
--embedding-model	SentenceTransformer model	BAAI/bge-base-en-v1.5
--semantic-profile-path	profiles to embed	datasets/<benchmark>/semantic_profiles.pkl
--out-dir	where to write the indexes	datasets/<benchmark>
--fp32	encode in float32 (needed for EmbeddingGemma)	off

---

3. Build the join graph

Edges between tables from value overlap and column-embedding similarity.

python offline_graph.py --benchmark $B

Output: join_graph.pkl in the embedding directory.

Argument	Description	Default
--benchmark	benchmark	bird
--embedding-dir	folder with the step-2 files; join_graph.pkl is written here	datasets/<benchmark>
--workers	processes for column-pair scoring	1

---

4. Run Mew

Runs Mew on every question of datasets/$B/gt.csv. prerender_sample_rows.py is an optional speed-up that reads the first rows of every table once; online_retrieval.py uses its output automatically.

python prerender_sample_rows.py --benchmark $B
python online_retrieval.py --benchmark $B --decomposition-cache-tag sketch1

Output: datasets/$B/online_refined/top30_..._combinedcall__coretstyle_k10.json with, per question, the sketch, retrieved_tables, focused_tables and Mew's final joined_tables, plus a *_runtime_summary.json with time and token counts.

Argument	Description	Default
--benchmark	benchmark	bird
--llm-model	LLM for the sketch and the grounding call	gpt-5.4-mini
--llm-base-url	OpenAI-compatible server for an open model	OpenAI API
--embedding-model, --embedding-dir	must match steps 2 and 3	bge-base, datasets/<benchmark>
--top-k-tables	stage-2 pool size	25
--online-refine-candidate-k	candidates shown to the LLM	10
--decomposition-cache-tag	names the sketch cache; a new tag makes fresh LLM calls	none
--online-refine-cache-tag	names the result file and refinement cache	coretstyle_k10
--question-workers	parallel worker processes	1

---

5. Evaluate retrieval

Precision, recall and F1 at Ideal-K, difficulty breakdown, returned-set size, runtime and tokens, for Mew and the baselines; then F1 by number of joinable distractors, number of joins and difficulty.

python evaluate_retrieval.py  --benchmark $B
python evaluate_robustness.py --benchmark $B       

The baselines (DMRAL, CORE-T, REAR, JAR) are run with their own code; evaluate_retrieval.py reads their outputs from datasets/<benchmark>/rear.json, jar_contriever.txt, coret_*.json and $MEW_BASELINES_ROOT/multitab_qa_dmral/code/<benchmark>/output/retrieved_for_eval.json.

Argument	Description	Default
--benchmark	benchmark	bird
--difficulty	evaluate_retrieval.py: all, simple, moderate, challenging or a comma-separated list	all
--online-refined-result-path	evaluate_retrieval.py: evaluate a different Mew result file	the step-4 file
--only-noise, --only-joins, --only-difficulty	evaluate_robustness.py: run one analysis	all three
--summary-csv	evaluate_robustness.py: print one CSV of F1@Ideal-K per bucket and exit	off

---

Other experiments

Downstream text-to-SQL (BIRD benchmarks)

Gives each method's retrieved tables to a text-to-SQL LLM, runs the SQL over the CSV tables in SQLite, and compares the answer with the gold answer. The second command builds the comparison table.

python downstream/nl2sql_retrieval_impact_eval.py --benchmark bird_opendata --model-name gpt-5.4-mini \
       --method "Mew=<mew result json>" --method "REAR=rear" --output downstream/<name>.json
python downstream/build_downstream_table_overlap.py --config bird_opendata_gpt5mini

Argument	Description	Default
--method label=source	repeatable; source is a Mew result JSON (optionally path::field), rear, jar or an oracle such as gt_tables	the benchmark's standard baselines
--model-name, --model-backend	text-to-SQL LLM; openai or local (Hugging Face model)	gpt-5.4-mini, openai
--config (second script)	bird_opendata_llama3b, bird_opendata_llama8b or bird_opendata_gpt5mini; or --json label=path:record_label for your own files	none

Distractor study

Adds d distractor tables to the gold tables of each question and measures execution accuracy. It needs the step-4 Mew result and a stage-1 candidate file, made with python online_retrieval.py --benchmark bird_opendata --zero-shot --stage1-source-diagnostic --online-refine-cache-tag distractorstudy_stage1raw.

python distractor_study.py --benchmark bird_opendata

Arguments: --d-values (default 1,2,4,8), --skip-distractor-types (subset of random,similar,joinable,dup), --min-related-tabs (3), --model-name, --output.

Ablations (on bird)

Each ablation reruns online_retrieval.py with one factor changed and reuses the sketches of the step-4 run, so run step 4 first and pass the same tag as SKETCH_TAG.

SKETCH_TAG=sketch1 ./run_updated_mew_ablations.sh k          # candidates shown to the LLM: 5, 20, 25, 30
SKETCH_TAG=sketch1 ./run_updated_mew_ablations.sh pool       # stage-2 pool size: 10, 25, 50
SKETCH_TAG=sketch1 ./run_updated_mew_ablations.sh stage      # no refinement (--zero-shot), no focus tables (--no-focus)
SKETCH_TAG=sketch1 ./run_updated_mew_ablations.sh embedding  # UAE and EmbeddingGemma (build their indexes first)
./run_llm_ablation.sh qwen7b Qwen/Qwen2.5-7B-Instruct        # whole pipeline with an open LLM
python summarize_ablations.py --study k|pool|stage|embedding|llm|all

The embedding runs need offline_embeddings.py --out-dir datasets/bird/emb_<name> and offline_graph.py --embedding-dir datasets/bird/emb_<name> first (<name> is uae or gemma; EmbeddingGemma needs --fp32 and torch ≥ 2.6). run_llm_ablation.sh takes a run name, a model name and optionally GPU ids; it starts the model with vLLM (pip install vllm) and rebuilds the offline stage with it, or set LLM_BASE_URL to use a server you already run. Optional environment variables for the driver: BENCH, GPUS, PYTHON_BIN, PYTHON_BIN_TORCH26.

---

Quick start example

B=bird
python offline_profiling.py  --benchmark $B
python offline_embeddings.py --benchmark $B
python offline_graph.py      --benchmark $B
python prerender_sample_rows.py --benchmark $B
python online_retrieval.py --benchmark $B --decomposition-cache-tag sketch1
python evaluate_retrieval.py --benchmark $B

Citation

The benchmarks come from the following work.

@inproceedings{li2023bird,
  title     = {Can {LLM} Already Serve as A Database Interface? A {BIg} Bench for Large-Scale Database Grounded Text-to-{SQL}s},
  author    = {Li, Jinyang and Hui, Binyuan and Qu, Ge and Yang, Jiaxi and Li, Binhua and Li, Bowen and Wang, Bailin and Qin, Bowen and Geng, Ruiying and Huo, Nan and Zhou, Xuanhe and Ma, Chenhao and Li, Guoliang and Chang, Kevin C. C. and Huang, Fei and Cheng, Reynold and Li, Yongbin},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2023}
}

@inproceedings{yu2018spider,
  title     = {Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-{SQL} Task},
  author    = {Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2018},
  pages     = {3911--3921}
}

@inproceedings{luo2026decomposition,
  title     = {Decomposition-Driven Multi-Table Retrieval and Reasoning for Numerical Question Answering},
  author    = {Feng Luo and Hai Lan and Hui Luo and Zhifeng Bao and Xiaoli Wang and J. Shane Culpepper and Shazia Wasim Sadiq},
  booktitle = {{ICDE}},
  year      = {2026},
  pages     = {1872-1885}
}

Disclosures:

This software may include, incorporate, or access open source software (OSS) components, datasets and other third party components, including those identified below. The license terms respectively governing the datasets and third-party components continue to govern those portions, and you agree to those license terms may limit any distribution, use, and copying. You may use any OSS components under the terms of their respective licenses, which may include BSD 3, Apache 2.0, and other licenses. In the event of conflicts between Megagon Labs, Inc. (“Megagon”) license conditions and the OSS license conditions, the applicable OSS conditions governing the corresponding OSS components shall prevail. You agree not to, and are not permitted to, distribute actual datasets used with the OSS components listed below. You agree and are limited to distribute only links to datasets from known sources by listing them in the datasets overview table below. You agree that any right to modify datasets originating from parties other than Megagon are governed by the respective third party’s license conditions. You agree that Megagon grants no license as to any of its intellectual property and patent rights. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS (INCLUDING MEGAGON) “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. You agree to cease using, incorporating, and distributing any part of the provided materials if you do not agree with the terms or the lack of any warranty herein. While Megagon makes commercially reasonable efforts to ensure that citations in this document are complete and accurate, errors may occur. If you see any error or omission, please help us improve this document by sending information to contact_oss@megagon.ai.