Update the spark version to the current version

[mikealfare]

resolves #1054

Problem

• we're using an old runtime version of Spark for some tests
• there's an issue in spacy that requires pydanic>=1.10.8 (see FIXED: Pydantic issubclass error for python 3.8 and 3.9 explosion/spaCy#12659)
• there's an issue in databricks.koalas that requires an update in handling numpy updates; however, this library is no longer supported

Solution

• update the version in the tests
• update pydanic<1.10.3 to pydanic>=1.10.8
• add a docstring explaining all of this
• exclude the koalas test

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the dbt-spark contributing guide.

[VersusFacit]

Really appreciate the comments.

[github-actions]

The backport to 1.8.latest failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-1.8.latest 1.8.latest
# Navigate to the new working tree
cd .worktrees/backport-1.8.latest
# Create a new branch
git switch --create backport-1055-to-1.8.latest
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 824ca0f2249d145234f21d7e4066e033a273e2e2
# Push it to GitHub
git push --set-upstream origin backport-1055-to-1.8.latest
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-1.8.latest

Then, create a pull request where the base branch is 1.8.latest and the compare/head branch is backport-1055-to-1.8.latest.