One of the most interesting and perplexing aspects of new generative LLM models is the way we interact with them. When you go to use one of these tools, you’re greeted with a text input where you can type in anything you want. At first it feels magical - “I can ask it anything” - but the more you interact with the model, the more it becomes clear that it’s not so simple. Asking a question in a different way, or adding or removing context, can provide a better response. Attempting to get the results you want can drive you to extremes - we’ve seen people promise to tip the LLM or punish it in an attempt to improve the output.
Fortunately, there are better prompting methods we can leverage than just promising the LLM money. Using LLMs to generate SQL queries has been widely researched, as detailed in A Survey on Employing Large Language Models for Text-to-SQL Tasks by Liang Shi, Zhengju Tang and Zhi Yang here. The paper covers a variety of topics, from evaluation data sets, to prompt engineering and fine-tuning.
The most interesting part to me was the section on prompt engineering, which defined a few areas to improve performance. Within the four areas of Basic Structure, Supplementary Knowledge, Example Selection and Reasoning, the authors also defined approaches specific to Text2SQL applications.
In previous experiments converting natural language queries to SQL, I barely scratched the surface. When developing my basic agent, I only using used a question structure that included the entire database schema in the prompt. In future posts, I will explore other methods such as providing SQL knowledge, updating basic structure, and testing Chain of Thought reasoning.