added_multiprocessing

[abhishekgupta368]

#6 adding multiprocessing

Label: medium

Feature added:
I add multiprocessing in a file using pools

[rajatkb]

• Remove header data added by spyder editor. We don't need those in this project.
• Change implementation to use Process class not thread pools. Thread pools are not interruptable i.e cannot be scheduled when in I/O wait.
• Do not bring change to code structure. You can create a sepparate utility class focused on Process based execution and then use that poool based execution to pass our function here along with args in kwargs fashion. This will allow us to isolate issues of Multiprocessing to one single file.

[rajatkb]

• You are over complicating the implementation.
• I will add an interface to be implemented, follow that it will help you to implement the functionality
• have a habit of adding doc string to your code. I did not saw doc string for the new function and classes.

[rajatkb]

@abhishekgupta368 I will provide an interface make sure to implement that.

[rajatkb]

@abhishekgupta368 discuss the implementation during meetup, please. It's not what we require.

[abhishekgupta368]

No, there is problem in my git terminal due to which you got problematic file.
I am working on it.

[abhishekgupta368]

this is a error, I got:

Traceback (most recent call last):
  File "C:/Users/Lenovo/Desktop/open_src_notify/Conference-Notify/Scrapper-Service/app.py", line 95, in <module>
    lambda : scrapper(  log_level = log_level,
  File "C:\Users\Lenovo\Desktop\open_src_notify\Conference-Notify\Scrapper-Service\process\mutiprocessing.py", line 13, in execute_process
    p.start()
  File "C:\Users\Lenovo\Anaconda3\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\Lenovo\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Lenovo\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Lenovo\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\Lenovo\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function <lambda> at 0x000001C85F8E5BF8>: attribute lookup <lambda> on __main__ failed
2020-03-09 11:56:48,340 - __main__ - INFO - Scrapping done from all Scrapper plugins

[rajatkb]

Ya that's a legit error, because we are passing lambda functions which are not exactly pickable. i.e you cannot pickle them or covert to binary representation of objects. So hence the issue. Minor modifications should be able to help. You see if there is some hacky way of fixing this. 😅 I am looking into it too.

[abhishekgupta368]

ok

[rajatkb]

• No use of logger for task-related instantiation, the logger information is provided for usage , use accordingly.
• There is a merge conflict, resolve it ASAP

[rajatkb]

• Can you put some log messages specific to this class. Which will inform us about current stage of the processes running.
• Log the stats
• add doc string to the classes and function. use vscode extension docstring for generating the format.

[rajatkb]

Ok , looks good. merging.

[abhishekgupta368]

hard work paid off