When Officials Prompt Chatbots

Some of the guidance and training programs we analyzed advise certain prompting behaviors, sometimes even offering model prompts. Many of these advise adding contextual information to prompts. This can relate to the intended form of output (“be concise”, “cite specific evidence”) and/or the conditions in which the output is being produced (“this is for Minister X”, “act as an experienced policy researcher in social welfare”). However, adding further context to prompts can actually heighten risks, sometimes in surprising ways. Some research has suggested, for example, that asking chatbots to “be more concise” can lead to more inaccuracies.

We therefore conducted small-scale tests of LLMs, seeing how they responded to the sort of prompts that we expect officials might write, such as formulating positions or summarizing evidence. We say “the sort of prompts we expect officials might write” because, as noted in our discussion of FOIs, getting “real world” examples of prompts is challenging. Some guidelines do provide suggestions (even templates) of how officials should write prompts, often suggesting additional contextual information (such as “I need to brief Minister XZY”). However, we cannot be sure how often this advice is actually used in practice. Nonetheless, the prompts we developed generally did include additional contextual information of the prompts suggested in guidance.

To test the effects of this type of contextual information, we varied it – for instance using names of different ministers, varying sources of arguments, or phrasing requests differently – and observed how answers changed. In order to test a wide range of possibilities and analyze numerous variants, we queried the LLMs multiple times via automation. This was done using the official application programming interfaces (APIs) provided by the companies and assessing answers through a combination of LLMs and manual human inspection of some answers (enabled by customizing the PETRI framework).