Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TDL-19801: Tap fetch data for sub-questions #62

Merged
merged 17 commits into from
Aug 29, 2022
Merged

TDL-19801: Tap fetch data for sub-questions #62

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
6bce5af
TDL-19801: Tap does not support fetching data for the questions neste…
Jul 7, 2022
a4af60c
addressed the comments
Jul 11, 2022
8f835a1
add unittest
Jul 11, 2022
46202a5
modify funciton comment
Jul 11, 2022
6fb58f7
formated test_case value in unittest
Jul 11, 2022
08e7b86
formated expected_case
Jul 11, 2022
ec5bd1e
Updated schema in questions.json
Jul 12, 2022
74aa166
resolved build fail error
Jul 12, 2022
fea7596
resolve build fail error
Jul 12, 2022
15141fe
updated unittest
Jul 26, 2022
627aaaa
updated setup.py
Jul 28, 2022
7e7dc9a
add parameterized
Jul 28, 2022
72551c6
change start date in start_date_test
Jul 29, 2022
1230dcc
make change in bookmark test
Jul 29, 2022
db99dc0
Merge branch 'crest-master' into TDL-19801-Tap-Does-Not-Support-Fetch…
prijendev Aug 29, 2022
4b9515c
Updated unit test case.
prijendev Aug 29, 2022
630dfa3
Updated schemas and keyerror.
prijendev Aug 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions tap_typeform/schemas/questions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,13 @@
"ref": {
"type": ["null", "string"],
"selected": true
},
"sub_questions": {
"type": ["null", "array"],
"items": {
"type": ["null", "object"],
"properties": {}
}
}
},
"key_properties": ["form_id","question_id"]
Expand Down
26 changes: 25 additions & 1 deletion tap_typeform/streams.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,23 @@ def get_landings(atx, form_id, bookmark):

yield from items

def fetch_sub_questions(row):
'''This function fetches records for each sub_question in a question group and returns a list of fetched sub_questions'''
sub_questions = [] #Creating blank list to accommodate each sub-question's records

#If question group has no sub_questions, return blank list
if row['properties'].get('fields') == None :
return None

#Appending each sub-question to the list
for question in row['properties']['fields']:
sub_questions.append({
"question_id": question['id'],
"title": question['title'],
"ref": question['ref']
})
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#If question group has no sub_questions, return blank list
if row['properties'].get('fields') == None :
return None
#Appending each sub-question to the list
for question in row['properties']['fields']:
sub_questions.append({
"question_id": question['id'],
"title": question['title'],
"ref": question['ref']
})
#Appending each sub-question to the list
for question in row['properties'].get('fields'):
sub_questions.append({
"question_id": question['id'],
"title": question['title'],
"ref": question['ref']
})

We can remove the if condition because for loop will not iterate on None or [] so no need to handle it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the "if" condition and used the "get" method as per suggestion.


return sub_questions

def sync_form_definition(atx, form_id):
with singer.metrics.job_timer('form definition '+form_id):
Expand All @@ -156,12 +173,19 @@ def sync_form_definition(atx, form_id):
# we only care about a few fields in the form definition
# just those that give an analyst a reference to the submissions
for row in data:
sub_questions ={} #Creating a blank dictionary to store records of sub_questions,if any

#If type of question is group, i.e. it has sub_questions, then fetch those sub_questions
if row['type'] == 'group':
sub_questions['sub_questions'] = fetch_sub_questions(row)

#If sub_questions are fetched then add those in this field and display the same, else don't display this field
definition_data_rows.append({
"form_id": form_id,
"question_id": row['id'],
"title": row['title'],
"ref": row['ref']
})
}|sub_questions)

write_records(atx, 'questions', definition_data_rows)

Expand Down