Key Insights

• •

  Large language models are stateless and rely entirely on the provided context window for each response.

• •

  The context window is essentially a text file aggregating system instructions, user queries, documents, and other relevant data.

• •

  All modern AI techniques (retrieval, augmented generation, tool use) aim to improve the quality and relevance of this context window.

• •

  Tool use allows the model to autonomously decide when to invoke external APIs or services based on the input query.

• •

  Agent models enhance tool use by planning and executing multiple tool calls in an iterative, reasoning loop until a task is complete.

• •

  Post-training with billions of examples significantly improves models' abilities in reasoning, tool use, and planning.

• •

  Designing AI products should begin with user needs and the necessary context rather than starting with complex agent architectures.

• •

  Prompt structure, including semantic content and organization (e.g., XML tags), helps the model parse context effectively but is flexible.

• •

  Token limits constrain the context window size; modern models like Google’s can handle up to a million tokens, enabling very large context inputs.

• •

  User-specific data (e.g., PTO policy, employee info) can be integrated into the context window dynamically through backend queries to provide accurate personalized responses.

Notable Quotes