Skip to content

Latest commit

 

History

History
286 lines (272 loc) · 7.85 KB

91.md

File metadata and controls

286 lines (272 loc) · 7.85 KB
Raw

Results of Code LLama 34B on HumanEval/91

Date: 01 September, 2023
Model: codellama-34b-instruct
Temperature: 0.2
Passing runs: 0
Total runs: 10
Success rate: 0%

Prompt

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """

The above is an incomplete Python code fragment. Return the complete and correct code with no additional text.

Correct solution

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """import re
    sentences = re.split(r'[.?!]\s*', S)
    return sum(sentence[0:2] == 'I ' for sentence in sentences)

Unit tests

def check(candidate):

    # Check some simple cases
    assert candidate("Hello world") == 0, "Test 1"
    assert candidate("Is the sky blue?") == 0, "Test 2"
    assert candidate("I love It !") == 1, "Test 3"
    assert candidate("bIt") == 0, "Test 4"
    assert candidate("I feel good today. I will be productive. will kill It") == 2, "Test 5"
    assert candidate("You and I are going for a walk") == 0, "Test 6"

    # Check some edge cases that are easy to work out by hand.
    assert True, "This prints if this assert fails 2 (also good for debugging!)"

Runs

❌ Run 1

Start time: 2023-09-01 02:47:09
Duration: 7.53s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 2

Start time: 2023-09-01 02:50:11
Duration: 7.52s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 3

Start time: 2023-09-01 02:53:06
Duration: 7.45s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split(('.', '?', '!')):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 4

Start time: 2023-09-01 02:56:05
Duration: 9.37s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 5

Start time: 2023-09-01 02:58:58
Duration: 8.93s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 6

Start time: 2023-09-01 03:02:05
Duration: 8.13s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 7

Start time: 2023-09-01 03:05:07
Duration: 8.67s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 8

Start time: 2023-09-01 03:08:07
Duration: 8.17s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 9

Start time: 2023-09-01 03:11:05
Duration: 8.46s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') or S.split('?') or S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count

❌ Run 10

Start time: 2023-09-01 03:14:06
Duration: 7.78s
Result: Fail

def is_bored(S):
    """
    You'll be given a string of words, and your task is to count the number
    of boredoms. A boredom is a sentence that starts with the word "I".
    Sentences are delimited by '.', '?' or '!'.
   
    For example:
    >>> is_bored("Hello world")
    0
    >>> is_bored("The sky is blue. The sun is shining. I love this weather")
    1
    """
    count = 0
    for sentence in S.split('.') + S.split('?') + S.split('!'):
        if sentence.startswith('I'):
            count += 1
    return count