“Chat with your data” is one of the most common pitches in AI software. You upload a spreadsheet or connect a database, type a question such as “which region had the biggest drop in sales last quarter?” and receive a table or chart. It feels like magic, but the mechanism is ordinary and inspectable: the model translates your question into code, a normal engine runs that code and the model explains the output. Knowing which step does what tells you where mistakes enter and how to catch them before a wrong number lands in a slide deck.

Step one: understand the question and the data

The tool first needs to know what your data looks like. It reads table names, column names, data types and often a handful of sample rows, then includes that description in the prompt it sends to a language model. This is why clear column names matter so much. A column called rev_adj_q3 is a puzzle to a model, while revenue_adjusted_q3_usd is a hint.

Some products add a semantic layer: definitions that spell out what “active customer” or “net revenue” means in your company. When such a layer exists, the model is far less likely to invent its own definition.

Step two: write code

The language model then writes a program. There are two common targets.

  • SQL. For databases, the model generates a query. This is called text-to-SQL, and it has been a research area for years. Academic benchmarks such as Spider, a 2018 dataset of human-written questions and queries across many databases, and BIRD, a 2023 benchmark built around larger, messier databases, show how performance drops as schemas get bigger and questions get more ambiguous. Benchmark scores are not a guarantee about your data, but they illustrate why messy real-world databases are hard.
  • Python or spreadsheet formulas. For uploaded files, tools typically write and run code in a sandbox, for example using a data library to clean, join and plot. Products such as Julius AI describe this kind of chat-with-a-spreadsheet workflow, and Hex combines notebooks, SQL and Python with AI that writes queries and code from natural language, according to their descriptions.

The important point is that the arithmetic happens in the database or interpreter, not in the language model. That is a strength: counting 10 million rows is reliable when a database does it. It also means errors are usually logic errors in the code, not mental-math slips.

Step three: run, inspect and explain

The tool executes the code, collects results and often asks the model to summarize them or choose a chart. Many tools show the code or query so you can read it. If one does not, treat its answers with extra caution.

Where it goes wrong

Most failures fall into a few patterns.

  1. Wrong definition. “Customers” could mean accounts, users or paying accounts. The model picks one without telling you.
  2. Bad joins. Joining tables on the wrong key duplicates rows and inflates totals. This is one of the most common silent errors in any analysis.
  3. Dropped data. Filters that remove nulls, outliers or a date range you did not intend.
  4. Ambiguous time. Fiscal versus calendar quarters, time zones and “last month” on the first of the month.
  5. Unsupported conclusions. The model describes a pattern as a cause, or reads a trend into noise, because summaries sound confident.
  6. Data leakage into prompts. Sample rows sent to a cloud model may contain personal or confidential information, so check what the vendor sends and stores.

A verification checklist

You can use the same habits you would use for a colleague’s analysis.

  • Read the query or code. Look at filters, joins and groupings. If you cannot read it, ask the tool to explain it line by line.
  • Run a sanity total. Compare a headline number, such as total revenue, against a figure you already trust.
  • Test a known answer. Ask a question whose answer you know, and see whether the tool gets it right.
  • Check row counts. After each join or filter, confirm the row count makes sense.
  • Ask for the assumption list. Prompt the tool to state definitions and filters it used.
  • Keep a human sign-off for numbers that drive decisions or external reporting.

For teams building their own analytics assistant, our guide on building an evaluation set for LLM apps explains how to test question-and-answer pairs systematically, which applies directly to text-to-SQL.

When these tools help most

AI analysis tools are strongest for exploration and first drafts: quickly profiling a new file, trying chart ideas or writing boilerplate SQL that an analyst then reviews. They are weakest when the question is ambiguous, the data is poorly documented or the cost of a mistake is high. Costs also matter: heavy use of cloud models is billed by usage in many products, as we explain in how AI pricing works.

What to watch next

  • Wider use of semantic layers that give models agreed definitions of metrics.
  • Tools that show confidence or highlight assumptions alongside results.
  • Better benchmarks that reflect messy enterprise schemas and ambiguous questions.
  • On-device or self-hosted options for sensitive data.
  • Features that automatically test an answer against a second method.

Key takeaways

  • AI analytics tools translate questions into SQL or Python; a database or interpreter does the calculation.
  • Clear column names and agreed metric definitions improve accuracy.
  • Typical errors are wrong definitions, bad joins, dropped rows and overconfident summaries.
  • Read the generated query, sanity-check totals and test a known answer before trusting a result.
  • Treat the output as a draft analysis that needs review for important decisions.

Frequently asked questions

Does the AI do the math itself?

Usually not. The model writes code or a query, and a database or interpreter performs the calculation, which is why exact counts are reliable when the code is right.

Is text-to-SQL accurate enough for production?

It depends on schema size, question clarity and safeguards. Research benchmarks show accuracy falls on complex, large databases, so teams typically add review steps and definitions.

Is it safe to upload company data?

Check the vendor’s data handling terms, what is sent to the model and whether you can use a private deployment. Remove or mask personal data when possible.