MBPP is a dataset of around 1,000 Python programming problems that are solvable by entry-level programmers. The problems cover topics such as programming fundamentals, standard library functionality, and text processing. Each problem consists of a natural language description, a code solution, and three automated test cases.

The dataset can be used to evaluate the capabilities of code-generation models, such as large language models, to synthesize short Python programs from natural language descriptions. It can also be used to enhance the user experience of code generation tools, such as Codex, by providing accurate and reliable solutions.

Here are some examples of problems from the MBPP dataset, taken from different topics:

• Write a function that takes a string and returns the number of vowels in it.
  • Python

    def count_vowels(s: str) -> int:
        vowels = "aeiouAEIOU"
        count = 0
        for c in s:
            if c in vowels:
                count += 1
        return count
     Test cases:

    • count_vowels("hello") == 2
    • count_vowels("AEIOU") == 5
    • count_vowels("bcdfghjkl") == 0

• Write a function that takes a list of numbers and returns the second largest number in the list.
  • Python

    def second_largest(nums: list) -> int:
        largest = max(nums)
        second = min(nums)
        for n in nums:
            if n > second and n < largest:
                second = n
        return second
    

    AI-generated code. Review and use carefully.
  • Test cases:
    • second_largest([1, 2, 3, 4, 5]) == 4
    • second_largest([10, 9, 8, 7, 6]) == 9
    • second_largest([5, 5, 4, 3, 2]) == 4
• Write a function that takes a string and returns a new string with the first and last characters swapped.
  • Python

    def swap_ends(s: str) -> str:
        if len(s) < 2:
            return s
        else:
            return s[-1] + s[1:-1] + s[0]
    

    Test cases:

    • swap_ends("hello") == "oellh"
    • swap_ends("a") == "a"
    • swap_ends("ab") == "ba"

• The Hugging Face website that provides the dataset download link and the documentation for loading and using the dataset.
• The TensorFlow website that provides the dataset download link and the documentation for loading and using the dataset.
• The original paper by Austin et al. that introduces the dataset and the methodology used to create it.
• The GitHub repository by OpenAI that contains the dataset, the code, and the models for HumanEval.