Add ability to have default config options for spark

[c2zwdjnlcg]

Adds a session_configs_defaults config option that gets merged with session_configs

This way users can use the configure magic to change session configs but any settings set in as defaults are preserved unless explicitly overridden by the user

Checklist

• Wrote a description of my changes above
• Formatted my code with black
• Added a bullet point for my changes to the top of the CHANGELOG.md file
• Added or modified unit tests to reflect my changes
• Manually tested with a notebook
• If adding a feature, there is an example notebook and/or documentation in the README.md file

[devstein]

@c2zwdjnlcg Thanks for making a PR! This is a common feature request from users.

I like the implementation -- it's simple, but satisfies the requirements.

[Carreau]

I don't have much experience with sparkmagic, but things seem ok to me.

If anything maybe a docstring in def _recursive_merge that warns the both(?) arguments are mutated and that it is recommended to deepcopy them before passing it in – which you do when you use it. But it my not be obvious at first glance.

[c2zwdjnlcg]

I don't have much experience with sparkmagic, but things seem ok to me.

If anything maybe a docstring in def _recursive_merge that warns the both(?) arguments are mutated and that it is recommended to deepcopy them before passing it in – which you do when you use it. But it my not be obvious at first glance.

Added a docstring to recommend deepcopy

[devstein]

@c2zwdjnlcg Thanks for contributing!

[c2zwdjnlcg]

@devstein thanks for merging. When is the next release scheduled for?

[devstein]

@c2zwdjnlcg I need to fix an outstanding issue with CI failing (help is welcome!).

Once that's fix I'll create a new release. I'm busy with other work, so my best guess is 2-3 weeks

[c2zwdjnlcg]

thanks, i opened #831 hopefully that helps