Skip to content

Latest commit

 

History

History
444 lines (335 loc) · 17.8 KB

CHANGELOG-bdc.md

File metadata and controls

444 lines (335 loc) · 17.8 KB

Change Log for BDC

Version 1.39.0

  • Within the a notebook section in build.yaml, you can now substitute ${source_path_no_ext} into ${dest}, as long as the source path is relative and not absolute. For instance, if your source is Day-1/Lesson-01.py, you can use a dest value of $target_lang/source_path_no_ext.$target_extension to produce the destination path, which (in a typical build) will expand to Scala/Day-1/Lesson-01.scala and Python/Day-1/Lesson-01.py. This addresses Issue #90.

Version 1.38.0

  • Can now substitute $course_name, $course_version and/or $course_id (which is $course_name-$course_version) into the student_dbc and instructor_dbc fields. Addresses Issue #93.

Version 1.37.0

  • Added bdc_load_build() function, to allow other tools (like course) to extract build information.
  • Moved working_directory() function to db_edu_util library, to remove duplication between bdc and course.

Version 1.36.0

  • Added ability to tag the Git repo with a tag constructed from the course's name and version, as found in build.yaml. See Issue #78

Version 1.35.0

  • bdc --download now detects notebooks that are marked skip: true and emits a proper message, instead of confusing skipped notebooks with extra notebooks not in the build file. Addresses Issue #83.
  • Fixed rendering of copyright symbol in generated version notebook. Addresses
    Issue #77.
  • The generated version notebook now contains the most recent Git commit from the current branch of the repository that contains the build file. If the build file isn't in a repository, then bdc emits an error, but keeps going, writing "Unknown" as the Git commit string. Addresses Issue #79

Version 1.34.0

  • Added generation of an index.html file for consolidated instructor notes, as requested in Issue 75.

Version 1.33.0

  • Added the ability to mark a notebook as not to be built. The notebook is excluded from all build profiles, but included in uploads and downloads. This change addresses Issue #73.

Version 1.32.1

  • Fixed a bug in the handling of only_in_profile in notebook and misc_file sections, leading to files so marked not being copied.

Version 1.32.0

  • Removed direct support for DB_SHARD_HOME and the home configuration option, since the databricks package supports both.

Version 1.31.0

  • Updated to use new databricks.Workspace class to interact with the Databricks workspace.

Version 1.30.0

  • Now implemented in Python 3. Will not work with Python 2.
  • Added support for arbitrary build profiles.
  • Added support for instructor note consolidation.
  • Added support for an include directive in the build.yaml file (e.g., #include "file.yaml")

Version 1.29.0

  • bdc now invokes databricks via its API, rather than by shelling out.

Version 1.28.0

  • A file in the misc_files section can now be marked with only_in_profile.

Version 1.27.0

  • bdc now ignores notebooks that are empty files or contain no cells, spitting out a warning.
  • gendbc is now written in Python. Changed bdc to call it as a Python function, instead of invoking the JVM to run the old command line Scala version.
  • Refactored bdc so that its functionality is available as a library, as well as from the command line.

Version 1.26.1

  • Now uses namedtuple._replace to copy a namedtuple while replacing one value, instead of custom code.

Version 1.26.0

  • bdc --upload and bdc --download now support multiple instances of a source file. The source file will be uploaded to all the target places. Upon download, only the first instance will be downloaded. Use with care!
  • The build file is now validated before a build, upload, or download is run.
  • Added --check (-C) argument that can be used to validate a build file without running a build.
  • markdown.html_stylesheet path is now assumed to be relative to the build file, unless it's absolute (which isn't recommended).

Version 1.25.0

  • In misc_files, if the destination does not have an extension, it is now assumed to be a directory, and dest_is_dir is inferred to be true. You can force it to be false, if need be, but only if the destination doesn't have an extension.

Version 1.24.1

  • Fixed a misleading error message when misc_files specifies a target destination of directory, but dest_is_dir isn't set.

Version 1.24.0

  • The zipfile and dest in the bundle section can now substitute the current output profile ("amazon" or "azure").

Version 1.23.2

  • Fixed handling of DB_SHARD_HOME to ensure that the environment variable actually has a non-empty value, not just that it is present.

Version 1.23.1

  • Fixed to pass notebook_defaults.variables variables into the master parser, making them available to Markdown cell templates.

Version 1.23.0

  • Fixed a (newly introduced) bug that caused an abort when copying instructor notes.
  • misc_files templates are now Mustache templates, not Python string templates.
  • Instructor notes and guides are now converted to HTML and PDF, where appropriate, just like other docs.

Version 1.22.0

  • Added ability for files in misc_files section to be templates, with variables substituted in them. See the sample build.yaml for details.
  • Added ability to generate PDF from a Markdown or HTML miscellaneous file.
  • Added a bundle section, allowing a zip file of built materials to be generated automatically. See the sample build.yaml for details.
  • The master (for a notebook or in notebook_defaults) now supports an enable_templates flag. If set to true, Markdown cells in the notebook are treated as Mustache templates by the master parser. (The flag is false by default.)
  • It is now possible to specify the name of the student DBC, via a new student_dbc build parameter; it defaults to Labs.dbc.
  • Similarly, it is now possible to specify the name of the student DBC, via a new instructor_dbc build parameter; it defaults to Instructor-Labs.dbc.
  • Added some parse-time validation of the source files (and required README.md and LICENSE.md) for the datasets section.
  • The README.md and LICENSE.md files for each data set are also converted to HTML and PDF and copied.
  • HTML generated from Markdown now gets anchor links for each generated HTML header.
  • course_info now supports a title attribute.

Version 1.21.0

  • Added course_info.type build setting, which can be ilt or self-paced. This build.yaml setting is now required.

Version 1.20.0

  • Added --info and --shell command line parameters.

Version 1.19.0

  • Added ability to specify debug: true in a master section to enable master parse-level debug messages for individual notebooks.

Version 1.18.2

Fixed bug relating to upload and download capability: If two notebooks with separate profiles ("amazon" and "azure") map to the same dest value, bdc would overwrite one of them during the upload and would arbitrarily choose one on the download. Now, it adds an "az" or "am" qualifier to the uploaded file. For instance, assume build.yaml has these two notebooks (and assume typical values in notebook_defaults):

  - src: 02-ETL-Process-Overview-az.py
    dest: ${target_lang}/02-ETL-Process-Overview.py
    only_in_profile: azure

  - src: 02-ETL-Process-Overview-am.py
    dest: ${target_lang}/02-ETL-Process-Overview.py
    only_in_profile: amazon

Both notebooks map to the same build destination. bdc --upload will upload 02-ETL-Process-Overview-az.py as 01-az-ETL-Process-Overview.py, and it will upload 02-ETL-Process-Overview-am.py as 01-am-ETL-Process-Overview.py.

bdc always applies the am or az prefix, if only_in_profile is specified, even if there are no destination conflicts. The prefix is placed after any numerals in the destination file name; if there are no numerals, it's placed at the beginning.

Version 1.18.1

  • Fixed bug: databricks command profile wasn't being passed all the places it should've been.

Version 1.18.0

  • --upload and --download now honor a --dbprofile option to specify the authentication profile to use with the databricks-cli. This option corresponds directly to the --profile argument to the databricks command.

Version 1.17.0

  • Added support for Amazon and Azure target profiles.

Version 1.16.0

  • Variables can now be defined in the notebook_defaults section and in the individual notebooks. These variables are expanded at notebook processing time, so they can access variables like ${notebook_type} and ${target_lang}. They can also override variables in the build-wide "variables" section.

Version 1.15.0

  • The ternary IF variable substitution syntax now supports simple variable substitutions within the comparison string, the "true" string, and the "false" string. Double quotes are still required, and only simple substitutions are permitted (i.e., ternary IFs and replacements are not). Examples:
${foo == "$bar" ? "It matches $$bar." : "It's $foo, not $bar"}
         ^    ^   ^                 ^   ^                   ^
         Note that the double quotes are REQUIRED

${x == "abc${foo}def" ? "YES" : "NO."}

${x == "01-abc" ? "${bar[0]}" : "${bar[-1]}"}
  • Similarly, the replacement string in a substitution edit can contain simple variable substitutions (but not ternary IFs and replacements). Examples:
${file/^\d+/$x/g}
${foo/\d/ABC${bar[0]}DEF/g}

  • Variables can now be subscripted, Python-style, as long as they use the brace (${var}) syntax. Examples: ${foo[0]}, ${foo[-1]}, ${foo[2:3]}, ${foo[:]}, ${foo[:-1]}, ${foo[1:]}

  • Character escaping changes:

    • To escape a $, use \$ or $$.
    • To escape a double quote, use \".

  • Fixed a bug: Escaped "$" (i.e., "$$") sequences weren't properly being unescaped.

Version 1.14.0

  • Variable substitution now supports a simple inline variable edit capability. General format: ${var/regex/replacement/flags} where regex is a regular expression, replacement is a replacement string, and flags can be i (case-insensitive), g (substitute all occurrences, not just the first), or ig (both). The delimiter can be either "/" or "|", and the delimiter can be escaped with a backslash, if necessary. Examples: Regular expression groups can be substituted using $1, $2, etc.
# Replace all occurrences of "letters/numbers" with "FOOBAR"
${foo|[a-z]+/\d+|FOOBAR|g}

Version 1.13.0

  • Variable substitution now supports a C-like ternary if syntax. For instance:
${variable == "foo" ? "Got foo" : "No foo"}
  • Added doctests to bdc/bdcutil.py. Just run the module to run the tests.

Version 1.12.2

  • Revised default Version-x.x.x file, removing an excess new line.

Version 1.12.1

  • Fixed upload and download capabilities to handle new (nonexistent) notebooks better.

Version 1.12.0

  • Added top_dbc_folder_name to build.yaml, allowing specification of the topmost folder in the generated DBC. Defaults to the course name. See the sample build.yaml for full details.
  • --upload and --download now support relative shard paths, but only if either environment variable "DB_SHARD_HOME" is set or ~/.databrickscfg has a home setting in the DEFAULT section. See the README for details.

Version 1.11.0

  • Added ability to enable or disable a footer that is automatically added to each generated notebook. The feature is controlled by a per-notebook footer option in the ". (See the sample build.yaml). The feature is on by default, and the default footer is a Databricks copyright.
  • You can now set master.heading.path and master.footer.path to the string "DEFAULT" to force the internal default to be used, which is useful if overriding a non-default value in the notebook_defaults setting.
  • Added course_info.copyright_year configuration item, to set the copyright year. Defaults to current year.
  • Added master_parse_min_version, which is required for any course that uses the master parser.

Version 1.10.1

  • Fixed --upload and --download, which broke due to all the changes in 1.10.0.
  • Added Python 2 check. (Python 3 is no longer supported.)

Version 1.10.0

  • bdc now generates a version-stamped notebook, with version information, at the top level of the generated build, providing an easy way for students to determine the course version, even if they rename the folder after import.
  • bdc no longer includes the version number in the top-level DBC folder.
  • A new bdc_min_version configuration item is now required. It identifies the minimum version of bdc required to parse a particular build.yaml. See the sample build.yaml for full details.
  • The student and instructor subdirectories (in the DBC) can now be configured by student_dir and instructor_dir, respectively. If not specified, they default to student_dir=StudentFile and instructor_dir=InstructorFiles.
  • Added variables section to build.yaml, allowing definition of arbitrary variables for substitution. See the sample build.yaml for details.
  • bdc now automatically generates a top-level version notebook in the DBC files.
  • Added a notebook_defaults section to capture default notebook settings. You can specify default dest patterns and the defaults for master in this new section.
  • Added a $target_extension substitution, allowing you to substitute the post-master parse target file extension into a notebook destination, if master parsing is enabled. (The extension is substituted without the leading ".".)
  • Added a $notebook_type substitution, allowing you to substitute the type of the notebook (answers, exercises or instructor) into the notebook destination, if master parsing is enabled.
  • Added an optional notebook_type_name section that allows you to define alternate strings for the $notebook_type substitution. See the sample build.yaml for details.
  • Changed the substitution of $extension so that it does not include the leading ".".
  • Added master.instructor, master.answers and master.exercises settings, allowing control over whether or not to generate instructor, answer and exercises notebooks for a given source notebook. All are true, by default.
  • Removed the top-level notebook_heading configuration item. The notebook heading (both path and the enabled/disabled setting) can now be specified on a per-notebook basis. Defaults can be set in the new notebook_defaults section. See the sample build.yaml for full details.
  • The target master parse language is no longer automatically inserted. If you enable master parsing, and you specify more than one language, you must use an explicit substitution of $target_lang in the notebook destination; otherwise, bdc will abort. If you only have a single language, you can omit $target_lang.
  • Removed the use of a leading "/" in a notebook destination as a means to suppress the automatic insertion of the target language.
  • Moved some general-purpose functions into separate bdcutil module.

Version 1.9.0

  • The master configuration file (bdc.cfg) is no longer used. bdc locates gendbc via the path, and it allows specification of the output directory via a new -d (or --dest) option.
  • Updated documentation in the README.

Version 1.8.0

  • Added ability to upload and download entire course via databricks CLI.
  • The master configuration file argument is now optional and defaults to ~/.bdc.cfg. Specify an alternate master configuration file with the -c option.
  • Relative directory names as destinations in build.yaml (e.g., dest: '..') are no longer supported. If you want to suppress the insertion of a target language, use a destination path that starts with "/".

Version 1.7.0

  • Added --list-notebooks option, providing a quick way to get a listing of all the notebooks in a course.

Version 1.6.0

  • Changed to support notebook heading changes in master parser. Notebook heading is automatically added by the build tool, unless the notebook_heading.enabled parameter is set to false.

Version 1.5.0

  • Updated to support notebook_heading override parameter in the master section for a notebook. This parameter, if defined, must point to a file containing Markdown and/or HTML, to be used to replace cells with the NOTEBOOK_HEADING command. It corresponds to the --notebook-heading master parse command-line parameter, and it's optional.

Version 1.4.1

  • Emit tool name (bdc) as prefix on verbose messages.
  • When -v is specified, invoke master parser with new verbose argument.

Version 1.4.0

  • Updated to work with newest version of master parser, which produces three kinds of notebooks (instructor, exercises, answers).
  • Updated to copy exercises and answers notebooks to the student labs section, and the instructor notebooks to the instructor labs section.
  • Removed student and answers keywords from course configuration master section. All notebook types are now generated unconditionally.
  • Fixed handling of destination directories.
  • Allow use of ${target_lang} in master parse destination configuration (dest keyword).
  • Added skip keyword, allowing files to be "commented out" easily.
  • Added change log.