TDL-19801: Tap fetch data for sub-questions #62

jtilala · 2022-07-07T09:46:13Z

Description of change

Added field sub_questions for the Question group to resove issue #59.
Added function fetch_sub_questions to fetch data for sub-questions.

Manual QA steps

Run sync mode to check whether we are getting data for sub-questions or not

Risks

Rollback steps

revert this branch

…d within a Question Group.

savan-chovatiya · 2022-07-07T10:20:32Z

tap_typeform/streams.py

+    #If question group has no sub_questions, return blank list
+    if row['properties'].get('fields') == None :
+        return None
+
+    #Appending each sub-question to the list
+    for question in row['properties']['fields']:
+        sub_questions.append({
+            "question_id": question['id'],
+            "title": question['title'],
+            "ref": question['ref']
+            })


Suggested change

#If question group has no sub_questions, return blank list

if row['properties'].get('fields') == None :

return None

#Appending each sub-question to the list

for question in row['properties']['fields']:

sub_questions.append({

"question_id": question['id'],

"title": question['title'],

"ref": question['ref']

})

#Appending each sub-question to the list

for question in row['properties'].get('fields'):

sub_questions.append({

"question_id": question['id'],

"title": question['title'],

"ref": question['ref']

})

We can remove the if condition because for loop will not iterate on None or [] so no need to handle it.

Removed the "if" condition and used the "get" method as per suggestion.

savan-chovatiya · 2022-07-11T12:42:31Z

tests/unittests/test_sub_questions.py

+    test_cases = [{'case1a': 'value1a', 'case1b': 'value1b', 'case1c': 'value1c', 'properties': {'description': 'Group question', 'case1': 'value1','case2':'value2', 'fields': [{'id': 'id1', 'title': 'title1', 'ref': 'ref1'}, {'id': 'id2','title': 'title2', 'ref': 'title2'}]}, 'type': 'group'}, {'case2a': 'value2a', 'case2b': 'value2b', 'case2c': 'value2c', 'properties': {'description': 'Group question', 'case3':'value3', 'case4':'value4'}, 'type': 'group'}]
+    expected_case = [{'question_id': 'id1', 'title': 'title1', 'ref': 'ref1'}, {'question_id': 'id2', 'title': 'title2', 'ref': 'title2'}]


Divide this line into multiline so every test scenario can be readable.

Formated test_cases and expected_case in unit tests for better readability.

tap_typeform/streams.py

RushiT0122 · 2022-07-26T12:01:11Z

tests/unittests/test_sub_questions.py

+    for test_case in test_cases :
+        if test_case ['properties'].get('fields'):
+            assert expected_case == fetch_sub_questions(test_case)
+        else:
+            assert [] == fetch_sub_questions(test_case)


If first assert fails, test will exit without executing next test. Please use parameterised unittests.

…ing-Data-For-The-Questions-Nested-Within-A-Question-Group

* Tdl 19964 add missing tap tester tests (#65) * TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config * TDL-19801: Tap fetch data for sub-questions (#62) * TDL-19801: Tap does not support fetching data for the questions nested within a Question Group. * addressed the comments * add unittest * modify funciton comment * formated test_case value in unittest * formated expected_case * Updated schema in questions.json * resolved build fail error * resolve build fail error * updated unittest * updated setup.py * add parameterized * change start date in start_date_test * make change in bookmark test * Updated unit test case. * Updated schemas and keyerror. Co-authored-by: Jay Tilala <[email protected]> Co-authored-by: prijendev <[email protected]> * TDL-19959 added missing fields (#63) * added missing fields * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * TDL-19964 Added missing tap-tester commits * updated discover and schema file * updated tap-tester tests * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * updated bookmarks tap-tester test * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * Updated and removed extra comments * fixed the issue for Keyerror of form_id in answers stream * updated unittests * updated start date as per format * added configurable page size param * updated pagination test case * handled page_size for 0 and updated unittests * resolved review comments * resolved bug fixes for pagination and removed incompleted_forms_only param * updated array type schema * added new stream unsubmitted landings * added new stream in tap-tester * updated indentation * resolved PR review comments * updated indentation to use 2 spaces * added page_size in example config * added back incremental_range in the tap-tester * fixed the cci issues * added missing fields to a dict * added missing assertion for all fields * updated comment * raised exc instead of fatal error message and updated unittests * removed get_logger() * added logger instead of print * Updated schema for questions. * Removed duplicate assertion in all_fields test. Co-authored-by: prijendev <[email protected]> * Updated schema for answers stream. Co-authored-by: namrata270998 <[email protected]> Co-authored-by: jtilala <[email protected]> Co-authored-by: Jay Tilala <[email protected]>

* Tdl 19964 add missing tap tester tests (#65) * TDL-19964 Added missing tap-tester commits * updated tap-tester tests * updated bookmarks tap-tester test * Updated and removed extra comments * updated start date as per format * updated pagination test case * resolved review comments * added new stream in tap-tester * fixed the cci issues * added missing assertion for all fields * removed get_logger() * added logger instead of print * TDL-19957 Update dict based to class based implementation (#64) * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * updated discover and schema file * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * fixed the issue for Keyerror of form_id in answers stream * updated unittests * added configurable page size param * handled page_size for 0 and updated unittests * resolved bug fixes for pagination and removed incompleted_forms_only param * added new stream unsubmitted landings * resolved PR review comments * added page_size in example config * added back incremental_range in the tap-tester * raised exc instead of fatal error message and updated unittests * resolved PR comments * fixed the issue when page_size not passed in config * TDL-19801: Tap fetch data for sub-questions (#62) * TDL-19801: Tap does not support fetching data for the questions nested within a Question Group. * addressed the comments * add unittest * modify funciton comment * formated test_case value in unittest * formated expected_case * Updated schema in questions.json * resolved build fail error * resolve build fail error * updated unittest * updated setup.py * add parameterized * change start date in start_date_test * make change in bookmark test * Updated unit test case. * Updated schemas and keyerror. Co-authored-by: Jay Tilala <[email protected]> Co-authored-by: prijendev <[email protected]> * TDL-19959 added missing fields (#63) * added missing fields * TDL-19957 update dict based to class based * updated while condition * updated while condition * removed incremental_range from REQUIRED_CONFIG_KEYS * TDL-19964 Added missing tap-tester commits * updated discover and schema file * updated tap-tester tests * updated to replication_key instead of keys * updated schemas and added comments * added unittests for code coverage * updated bookmarks tap-tester test * added unittests for sync.py * added parameterized in setup.py * added parameterized in config.yml and updated unittests * updated the replication key to list instead of a single key * Updated and removed extra comments * fixed the issue for Keyerror of form_id in answers stream * updated unittests * updated start date as per format * added configurable page size param * updated pagination test case * handled page_size for 0 and updated unittests * resolved review comments * resolved bug fixes for pagination and removed incompleted_forms_only param * updated array type schema * added new stream unsubmitted landings * added new stream in tap-tester * updated indentation * resolved PR review comments * updated indentation to use 2 spaces * added page_size in example config * added back incremental_range in the tap-tester * fixed the cci issues * added missing fields to a dict * added missing assertion for all fields * updated comment * raised exc instead of fatal error message and updated unittests * removed get_logger() * added logger instead of print * Updated schema for questions. * Removed duplicate assertion in all_fields test. Co-authored-by: prijendev <[email protected]> * Updated schema for answers stream. * update version and changelog * changelog update for v2.0.0 * changelog update for v2.0.0 * minor changelog update Co-authored-by: namrata270998 <[email protected]> Co-authored-by: jtilala <[email protected]> Co-authored-by: Jay Tilala <[email protected]> Co-authored-by: prijendev <[email protected]> Co-authored-by: kethan1122 <[email protected]> Co-authored-by: rdeshmukh15 <[email protected]>

TDL-19801: Tap does not support fetching data for the questions neste…

6bce5af

…d within a Question Group.

jtilala requested review from dbshah1212 and savan-chovatiya July 7, 2022 09:46

savan-chovatiya reviewed Jul 7, 2022

View reviewed changes

Jay Tilala added 3 commits July 11, 2022 14:43

addressed the comments

a4af60c

add unittest

8f835a1

modify funciton comment

46202a5

savan-chovatiya reviewed Jul 11, 2022

View reviewed changes

Jay Tilala added 2 commits July 11, 2022 18:17

formated test_case value in unittest

6fb58f7

formated expected_case

08e7b86

savan-chovatiya approved these changes Jul 11, 2022

View reviewed changes

dbshah1212 reviewed Jul 11, 2022

View reviewed changes

tap_typeform/streams.py Outdated Show resolved Hide resolved

Updated schema in questions.json

ec5bd1e

dbshah1212 approved these changes Jul 12, 2022

View reviewed changes

Jay Tilala added 2 commits July 12, 2022 10:47

resolved build fail error

74aa166

resolve build fail error

fea7596

jtilala requested review from cosimon, KrisPersonal, nitingaikwad1, RushiT0122 and shantanu73 July 13, 2022 05:09

KrisPersonal approved these changes Jul 26, 2022

View reviewed changes

RushiT0122 requested changes Jul 26, 2022

View reviewed changes

updated unittest

15141fe

jtilala requested review from RushiT0122 and dbshah1212 July 26, 2022 13:05

RushiT0122 approved these changes Jul 28, 2022

View reviewed changes

RushiT0122 self-requested a review July 28, 2022 07:47

Jay Tilala added 2 commits July 28, 2022 13:22

updated setup.py

627aaaa

add parameterized

7e7dc9a

Jay Tilala added 2 commits July 29, 2022 19:30

change start date in start_date_test

72551c6

make change in bookmark test

1230dcc

RushiT0122 approved these changes Aug 1, 2022

View reviewed changes

dbshah1212 approved these changes Aug 3, 2022

View reviewed changes

prijendev changed the base branch from master to crest-master August 29, 2022 08:14

prijendev added 3 commits August 29, 2022 14:12

Merge branch 'crest-master' into TDL-19801-Tap-Does-Not-Support-Fetch…

db99dc0

…ing-Data-For-The-Questions-Nested-Within-A-Question-Group

Updated unit test case.

4b9515c

Updated schemas and keyerror.

630dfa3

prijendev merged commit df040a5 into crest-master Aug 29, 2022

prijendev mentioned this pull request Sep 2, 2022

Crest master #66

Merged

kethan1122 deleted the TDL-19801-Tap-Does-Not-Support-Fetching-Data-For-The-Questions-Nested-Within-A-Question-Group branch April 28, 2023 09:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TDL-19801: Tap fetch data for sub-questions #62

TDL-19801: Tap fetch data for sub-questions #62

jtilala commented Jul 7, 2022 •

edited

Loading

savan-chovatiya Jul 7, 2022

jtilala Jul 11, 2022

savan-chovatiya Jul 11, 2022

jtilala Jul 11, 2022

RushiT0122 Jul 26, 2022

		test_cases = [{'case1a': 'value1a', 'case1b': 'value1b', 'case1c': 'value1c', 'properties': {'description': 'Group question', 'case1': 'value1','case2':'value2', 'fields': [{'id': 'id1', 'title': 'title1', 'ref': 'ref1'}, {'id': 'id2','title': 'title2', 'ref': 'title2'}]}, 'type': 'group'}, {'case2a': 'value2a', 'case2b': 'value2b', 'case2c': 'value2c', 'properties': {'description': 'Group question', 'case3':'value3', 'case4':'value4'}, 'type': 'group'}]
		expected_case = [{'question_id': 'id1', 'title': 'title1', 'ref': 'ref1'}, {'question_id': 'id2', 'title': 'title2', 'ref': 'title2'}]

TDL-19801: Tap fetch data for sub-questions #62

TDL-19801: Tap fetch data for sub-questions #62

Conversation

jtilala commented Jul 7, 2022 • edited Loading

Description of change

Manual QA steps

Risks

Rollback steps

savan-chovatiya Jul 7, 2022

Choose a reason for hiding this comment

jtilala Jul 11, 2022

Choose a reason for hiding this comment

savan-chovatiya Jul 11, 2022

Choose a reason for hiding this comment

jtilala Jul 11, 2022

Choose a reason for hiding this comment

RushiT0122 Jul 26, 2022

Choose a reason for hiding this comment

jtilala commented Jul 7, 2022 •

edited

Loading