About how to generate tables in database while running the shaduler

[priya23595]

How are datasets tables in the Postgres Database generated based on events and dimensions CSV. Need to know the flow for it
can u help me to understand

[DevashishRaghav]

Hey Priya
Please check this deck: Link

[bommalla]

@priya23595 Every grammar file should have one data file then only datasets are created in the database otherwise it won't created.

1. The adaptor API will pick the files from the bucket and create one process_input folder and store the all event files with program name separately in the process input folder.
2. The "Run Latest Code local" nifi processor take input files + dimensions based on that mapping will happens laters Nifi having certain datasets tables list with queries are executed and storing output to respective dataset table

[priya23595]

I will Go through this link, thank you
But we have some questions can we connect in second half

[RNarmatha]

Hi Priya,
Can you please post your questions here . So that our team can get in touch with you

[priya23595]

How to get automatically generate the tables in database while run the scheduler after ingestion the data? what exactly code for this need to understand

[RNarmatha]

https://github.com/Sunbird-cQube/processing-ms/tree/staging/impl/c-qube

Please look into this .

[priya23595]

can we connect? about this understanding

[DevashishRaghav]

Hey Priya
Sure we can connect but before that, we will soon put up the recordings of training sessions on microsite. To get the maximum value from our interaction over a call, I think it would be great if you go through them first.
Thanks

[priya23595]

ok sir no problem please share this training videos soon
thank you

[priya23595]

Please can u share the link of microsite so that i can go through the training sessions recording videos
Thank you

[DevashishRaghav]

Here is the link:
https://cqube.sunbird.org/

Under the 'Recordings of Training' section, you will find the link to the Google folder where you can access all recordings.

[priya23595]

-Hello can we connect about ingesting and uploading the project elated csv files on minio bucket and visualise the data
-Please can u help ,me to ingesting the new dimension and event project related files and visualise the same
can we connect?

[RNarmatha]

Hi @priya23595 , For the ingestion , we have the steps in the microsite . Kindly follow the steps

[priya23595]

and also want to know about code changes for other data

[RNarmatha]

For this can you please elaborate on this point

[priya23595]

-I am having new data for ingestion ,after ingesting the data we need to run scheduler it will run the script file and generating the tables in database and then it will visualise it on dashboard.
-For this new data what exactly code change we need to do in script or about the generating the tables needs to understand the code

[RNarmatha]

@dhanush-2397 - Could you please help on this ?

[dhanush-2397]

@priya23595 If the data is similiar to the schema and the grammar files present then we don't need any code changes => (VSK programs already present in the code base),
If you are working on your own use case then you need to define event grammars,dimension grammars and then modify the config.json file to ingest the data.

[priya23595]

In Which micro-services this config file is present

[priya23595]

yes am Working for our own use-cases related data files can u tell me the steps of how to define the event grammar and dimension grammars?

[RNarmatha]

Please refer this link to define the event and dimensions

[priya23595]

I am not able to delete previously ingested the event and dimension data from dimension grammar and event grammar tables can u help to delete this data to defining new data

[bommalla]

Please follow the below steps to truncate the data.

1. First check the all the containers: sudo docker ps -a
2. Go inside nifi container: sudo docker exec -it nifi_app bash
3. Then navigate the below mentioned path: Sunbird-cQube-processing-ms/impl/c-qube/inest
4. Then run the below command : yarn cli nuke-db

[priya23595]

Hello
I Am Done with creating the schema but when am uploading with national program or in new programs then it will get error

[priya23595]

My Email_Id:
[email protected]

Thank you

[priya23595]

hello
vim natitonwideusers-dimention.grammar.csv Index,,, string,string,string,string state_id,state_name,"suborg_id",suborg_name

am creating grammar file using vim command but it not get created plzz help me to correct this command

[bommalla]

@Priyanka Chincholi ***@***.***> Please follow the below steps. 1. First need install vim by using below command apt-get install vim 2. vim <grammar file name> 3. copy all the grammar file content and paste it 4. Then save the file by using the below command press esc button :wq! Thanks & Regards Rajesh

…

On Fri, Aug 11, 2023 at 4:56 PM Priyanka Chincholi ***@***.***> wrote: hello vim natitonwideusers-dimention.grammar.csv Index,,, string,string,string,string state_id,state_name,"suborg_id",suborg_name am creating grammar file using vim command but it not get created plzz help me to correct this command — Reply to this email directly, view it on GitHub <#809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUXQBUXE4FS5TAAVEFAIUCDXUYJHZANCNFSM6AAAAAA2VR6Y3U> . You are receiving this because you commented.Message ID: ***@***.***>

-- Regards, *Rajesh Bommalla* Sr.Data Engineer *M* +91 9908089539 *E* ***@***.*** www.tibilsolutions.com [image: LinkedIn] <https://www.linkedin.com/company/1913934> [image: Facebook] <https://www.facebook.com/TibilSolutions> [image: Twitter] <https://twitter.com/TibilSolutions> [image: Insta] <https://www.instagram.com/tibilsolutions/> *This message (including any attachments) may contain confidential, proprietary, privileged, and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution, or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.*

[priya23595]

vim natitonwideusers-dimention.grammar.csv
Index,,,
string,string,string,string
state_id,state_name,suborg_id,suborg_name

natitonwideusers-dimention.grammar.csv Grammar file name and its content is this correct

[priya23595]

All Grammar files are created now am going to start ingesting the data

[priya23595]

Hello @RNarmatha , @bommalla
Am creating the grammar files of custom data and Uploading the all dimension and event files but in the database Dataset is not generating
after running the scheduler APIs,Can u tell me why this is not generating please refer attachment ?

[bommalla]

@Priyanka Chincholi ***@***.***> Please check whether the nationwide users' grammar file is properly created or not ? Please send me the nationwide users event data.I'll check it from my end. Thanks & Regards

…

On Thu, Aug 17, 2023 at 12:14 PM Priyanka Chincholi < ***@***.***> wrote: Hello @RNarmatha <https://github.com/RNarmatha> , @bommalla <https://github.com/bommalla> Am creating the grammar files of custom data and Uploading the all dimension and event files but in the database Dataset is not generating after running the scheduler APIs,Can u tell me why this is not generating please refer attachment ? [image: Screenshot from 2023-08-17 12-10-53] <https://user-images.githubusercontent.com/136596601/261220990-f28e95f0-c4ea-4282-b0ea-4c198ce21e8d.png> [image: Screenshot from 2023-08-17 12-10-58] <https://user-images.githubusercontent.com/136596601/261221022-01f8f068-4f8d-443e-be81-d0a1fa6b0621.png> [image: Screenshot from 2023-08-17 12-11-40] <https://user-images.githubusercontent.com/136596601/261221042-82b59817-0f1e-420c-bdfe-73df02edbded.png> — Reply to this email directly, view it on GitHub <#809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUXQBUVPYL6QDCTEHOTRIBDXVW4WRANCNFSM6AAAAAA2VR6Y3U> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, *Rajesh Bommalla* Sr.Data Engineer *M* +91 9908089539 *E* ***@***.*** www.tibilsolutions.com [image: LinkedIn] <https://www.linkedin.com/company/1913934> [image: Facebook] <https://www.facebook.com/TibilSolutions> [image: Twitter] <https://twitter.com/TibilSolutions> [image: Insta] <https://www.instagram.com/tibilsolutions/> *This message (including any attachments) may contain confidential, proprietary, privileged, and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution, or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.*

[priya23595]

natitonwidecontent-event-data.csv
natitonwideusers-event-data.csv

[priya23595]

Grammar file is Created correctly please check on your side

[bommalla]

@Priyanka Chincholi ***@***.***> Please check the all diemasion and event data files if there is foreign key relation or not. if not datasets are not created. Thanks & Regards Rajesh

…

On Thu, Aug 17, 2023 at 4:16 PM Priyanka Chincholi ***@***.***> wrote: Grammar file is Created correctly please check on your side — Reply to this email directly, view it on GitHub <#809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUXQBUVLXOKXX3RKF34JALTXVXY6XANCNFSM6AAAAAA2VR6Y3U> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, *Rajesh Bommalla* Sr.Data Engineer *M* +91 9908089539 *E* ***@***.*** www.tibilsolutions.com [image: LinkedIn] <https://www.linkedin.com/company/1913934> [image: Facebook] <https://www.facebook.com/TibilSolutions> [image: Twitter] <https://twitter.com/TibilSolutions> [image: Insta] <https://www.instagram.com/tibilsolutions/> *This message (including any attachments) may contain confidential, proprietary, privileged, and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution, or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.*

[priya23595]

hello sir
can we connect for the showing u the grammar files and issue while visualising the data

[bommalla]

Sure. @Priyanka Chincholi ***@***.***> I scheduled a call at 2:30PM today. Thanks & Regards Rajesh

…

On Wed, Aug 30, 2023 at 12:30 PM Priyanka Chincholi < ***@***.***> wrote: hello sir can we connect for the showing u the grammar files and issue while visualising the data — Reply to this email directly, view it on GitHub <#809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUXQBUUFHRRBE3PH2EGZXZDXX3QI5ANCNFSM6AAAAAA2VR6Y3U> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Regards, *Rajesh Bommalla* Sr.Data Engineer *M* +91 9908089539 *E* ***@***.*** www.tibilsolutions.com [image: LinkedIn] <https://www.linkedin.com/company/1913934> [image: Facebook] <https://www.facebook.com/TibilSolutions> [image: Twitter] <https://twitter.com/TibilSolutions> [image: Insta] <https://www.instagram.com/tibilsolutions/> *This message (including any attachments) may contain confidential, proprietary, privileged, and/or private information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution, or other use of this message or any attachments by an individual or entity other than the intended recipient is prohibited.*

[priya23595]

ok sir

[priya23595]

can we connect at 3 because am having meeting at 2.30

[RNarmatha]

Hi Priyanka, Sure We will connect at 3 pm Thanks Narmatha

…

On Wed, Aug 30, 2023 at 2:24 PM Priyanka Chincholi ***@***.***> wrote: can we connect at 3 because am having meeting at 2.30 — Reply to this email directly, view it on GitHub <#809 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A2WM7ZU4V7AEFFD5CIZAKIDXX35S3ANCNFSM6AAAAAA2VR6Y3U> . You are receiving this because you were mentioned.Message ID: ***@***.***>