The HumanEval dataset is a collection of 164 hand-written programming problems that require synthesizing programs from docstrings. The pass@k metric is a measure of how often a model generates a correct solution within the top-k candidates. The HumanEval dataset and the pass@k metric are designed to evaluate the performance of large language models (LLMs) in code generation tasks by focusing on functional correctness rather than text similarity.

The HumanEval dataset and the pass@k metric can be used to test the ability of LLMs to understand natural language, generate code, and solve programming challenges. They can also be used to encourage the development of more robust and generalizable models that can handle diverse and complex problems. Moreover, the HumanEval dataset and the pass@k metric can be used to enhance the user experience of code generation tools, such as Codex, by providing accurate and reliable solutions.

• Here is an example problem from the HumanEval dataset, taken from the math domain¹:

Write a function that takes a positive integer n and returns the nth Fibonacci number.

The Fibonacci sequence is defined as follows: F(0) = 0, F(1) = 1, and F(n) = F(n-1) + F(n-2) for n > 1.

• A) def fibonacci(n: int) -> int: if n == 0: return 0 elif n == 1: return 1 else: return fibonacci(n-1) + fibonacci(n-2)
• B) def fibonacci(n: int) -> int: a = 0 b = 1 for i in range(n): a, b = b, a + b return a
• C) def fibonacci(n: int) -> int: return round(((1 + 5**0.5) / 2)n / 50.5)
• D) def fibonacci(n: int) -> int: return n * fibonacci(n-1) if n > 1 else n

The correct answer is A, B, or C, as they all implement the Fibonacci sequence correctly, while D implements the factorial function instead.

• The original paper by Chen et al. that introduces the dataset and the metric.
• The GitHub repository by OpenAI that contains the dataset, the code, and the models for HumanEval.
• The Papers With Code website that lists the papers and code related to the HumanEval dataset and task.
• The Deepgram blog post that explains the HumanEval dataset and the pass@k metric and their contribution to LLM evaluation.
• TriviaQA