+
+
+
+
alto2txt2fixture is a standalone tool to convert alto2txt XML output and other related datasets into JSON (and where feasible CSV) data with corresponding relational IDs to ease general use and ingestion into a relational database.
We target the the JSON produced for importing into lwmdb: a database built using the Django python webframework database fixture structure.
We provide a command line interface to process alto2txt XML files stored locally (or mounted via azure blobfuse), and for additional public data we automate a means of downloading those automatically.
We recommend downloading a copy of the reposity or using git clone. From a local copy use poetry to install dependencies:
If you would like to test, render documentation and/or contribute to the code included dev dependencies in a local install:
To processing newspaper metadata with a local copy of alto2txt XML results, it's easiest to have that data in the same folder as your alto2txt2fixture checkout and poetry installed folder. One arranged, you should be able to begin the JSON converstion with
To generate related data in JSON and CSV form, assuming you have an internet collection and access to a living-with-machines azure account, the following will download related data into JSON and CSV files. The JSON results should be consistent with lwmdb tables for ease of import.
Entry point for alto2txt2fixture.parse to convert alto2txt XML -> JSON.
This module defines the run function which is the main driver for the entire +process.
+It imports various functions from other modules and uses them to route and
+parse XML data generated by alto2txt.
The following steps are performed in the run function:
+alto2txt data into subdirectories with
+ structured files.JSON files.If the script is run as a main program (i.e. if the name of the script is
+__main__), the run() function is executed.
Note: at present this does not include any functunality in create_adjacent_tables.py
Manage command line arguments for run()
This constructs an ArgumentParser instance to manage
+configurating calls of run() to manage newspaper
+XML to JSON converstion.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
argv |
+
+ list[str] | None
+ |
+
+
+
+ If |
+
+ None
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ Namespace
+ |
+
+
+
+ A |
+
alto2txt2fixture/__main__.py44 + 45 + 46 + 47 + 48 + 49 + 50 + 51 + 52 + 53 + 54 + 55 + 56 + 57 + 58 + 59 + 60 + 61 + 62 + 63 + 64 + 65 + 66 + 67 + 68 + 69 + 70 + 71 + 72 + 73 + 74 + 75 + 76 + 77 + 78 + 79 + 80 + 81 + 82 + 83 + 84 + 85 + 86 + 87 + 88 + 89 + 90 + 91 + 92 + 93 + 94 + 95 + 96 + 97 + 98 + 99 +100 +101 +102 +103 +104 +105 +106 +107 +108 +109 +110 +111 +112 +113 +114 +115 | |
Manage running newspaper XML to JSON conversion.
First parse_args is called for command line arguments including:
collectionsoutputmountpointIf any of these arguments are specified, they will be used, otherwise they
+will default to the values in the settings module.
The show_setup function is then called to display the configurations
+being used.
The route function is then called to route the alto2txt files into
+subdirectories with structured files.
The parse function is then called to parse the resulting JSON files.
Finally, the clear_cache function is called to clear the cache
+(pending the user's confirmation).
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
local_args |
+
+ list[str] | None
+ |
+
+
+
+ Options passed to |
+
+ None
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ None + |
+
alto2txt2fixture/__main__.py118 +119 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +140 +141 +142 +143 +144 +145 +146 +147 +148 +149 +150 +151 +152 +153 +154 +155 +156 +157 +158 +159 +160 +161 +162 +163 +164 +165 +166 +167 +168 +169 +170 +171 +172 +173 +174 +175 +176 +177 +178 +179 +180 +181 +182 +183 +184 +185 +186 +187 +188 +189 +190 +191 +192 +193 +194 +195 +196 +197 +198 +199 +200 +201 +202 +203 | |
file_rename_table(
+ paths_dict: dict[os.PathLike, os.PathLike],
+ compress_format: ArchiveFormatEnum = COMPRESSION_TYPE_DEFAULT,
+ title: str = FILE_RENAME_TABLE_TITLE_DEFAULT,
+ prefix: str = "",
+ renumber: bool = True,
+) -> Table
+Create a rich.Table of rename configuration.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
paths_dict |
+
+ dict[os.PathLike, os.PathLike]
+ |
+
+
+
+ dict[os.PathLike, os.PathLike],
+Original and renumbered |
+ + required + | +
compress_format |
+
+ ArchiveFormatEnum
+ |
+
+
+
+ Which |
+
+ COMPRESSION_TYPE_DEFAULT
+ |
+
title |
+
+ str
+ |
+
+
+
+ Title of returned |
+
+ FILE_RENAME_TABLE_TITLE_DEFAULT
+ |
+
prefix |
+
+ str
+ |
+
+
+
+
|
+
+ ''
+ |
+
renumber |
+
+ bool
+ |
+
+
+
+ Whether an |
+
+ True
+ |
+
alto2txt2fixture/cli.pyfunc_table(
+ func: Callable,
+ values: dict,
+ title: str = "",
+ extra_dict: dict[str, Any] = {},
+) -> Table
+Geneate rich Table from func signature and help attr.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
func |
+
+ Callable
+ |
+
+
+
+ Function whose |
+ + required + | +
values |
+
+ dict
+ |
+
+
+
+
|
+ + required + | +
title |
+
+ str
+ |
+
+
+
+
|
+
+ ''
+ |
+
extra_dict |
+
+ dict[str, Any]
+ |
+
+
+
+ A |
+
+ {}
+ |
+
>>> def test_func(
+... var_a: Annotated[str, typer.Option(help="Example")] = "Default"
+... ) -> None:
+... test_func_table: Table = func_table(test_func, values=vars())
+... console.print(test_func_table)
+>>> if is_platform_win:
+... pytest.skip('fails on certain Windows root paths: issue #56')
+>>> test_func()
+ test_func config
+┏━━━━━━━━━━┳━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓
+┃ Variable ┃ Type ┃ Value ┃ Notes ┃
+┡━━━━━━━━━━╇━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩
+│ var_a │ str │ Default │ Example │
+└──────────┴──────┴─────────┴─────────┘
+alto2txt2fixture/cli.py296 +297 +298 +299 +300 +301 +302 +303 +304 +305 +306 +307 +308 +309 +310 +311 +312 +313 +314 +315 +316 +317 +318 +319 +320 +321 +322 +323 +324 +325 +326 +327 +328 +329 +330 +331 +332 +333 +334 +335 +336 +337 +338 +339 +340 +341 +342 +343 +344 +345 +346 +347 +348 +349 +350 +351 +352 +353 +354 +355 +356 +357 +358 +359 +360 | |
plaintext(
+ path: Annotated[Path, typer.Argument(help="Path to raw plaintext files")],
+ save_path: Annotated[
+ Path, typer.Option(help="Path to save json export files")
+ ] = Path(DEFAULT_PLAINTEXT_FIXTURE_OUTPUT),
+ data_provider_code: Annotated[
+ str, typer.Option(help="Data provider code use existing config")
+ ] = "",
+ extract_path: Annotated[
+ Path, typer.Option(help="Folder to extract compressed raw plaintext to")
+ ] = Path(DEFAULT_EXTRACTED_SUBDIR),
+ initial_pk: Annotated[
+ int,
+ typer.Option(help="First primary key to increment json export from"),
+ ] = DEFAULT_INITIAL_PK,
+ records_per_json: Annotated[
+ int, typer.Option(help="Max records per json fixture")
+ ] = DEFAULT_MAX_PLAINTEXT_PER_FIXTURE_FILE,
+ digit_padding: Annotated[
+ int,
+ typer.Option(help="Padding '0's for indexing json fixture filenames"),
+ ] = FILE_NAME_0_PADDING_DEFAULT,
+ compress: Annotated[
+ bool, typer.Option(help="Compress json fixtures")
+ ] = False,
+ compress_path: Annotated[
+ Path, typer.Option(help="Folder to compress json fixtueres to")
+ ] = Path(COMPRESSED_PATH_DEFAULT),
+ compress_format: Annotated[
+ ArchiveFormatEnum,
+ typer.Option(case_sensitive=False, help="Compression format"),
+ ] = COMPRESSION_TYPE_DEFAULT,
+) -> None
+Create a PlainTextFixture and save to save_path.
alto2txt2fixture/cli.py40 + 41 + 42 + 43 + 44 + 45 + 46 + 47 + 48 + 49 + 50 + 51 + 52 + 53 + 54 + 55 + 56 + 57 + 58 + 59 + 60 + 61 + 62 + 63 + 64 + 65 + 66 + 67 + 68 + 69 + 70 + 71 + 72 + 73 + 74 + 75 + 76 + 77 + 78 + 79 + 80 + 81 + 82 + 83 + 84 + 85 + 86 + 87 + 88 + 89 + 90 + 91 + 92 + 93 + 94 + 95 + 96 + 97 + 98 + 99 +100 +101 +102 | |
rename(
+ path: Annotated[Path, typer.Argument(help="Path to files to manage")],
+ folder: Annotated[
+ Path, typer.Option(help="Path under `path` for new files")
+ ] = Path(),
+ renumber: Annotated[
+ bool, typer.Option(help="Show changes without applying")
+ ] = False,
+ regex: Annotated[str, typer.Option(help="Regex to filter files")] = "*.txt",
+ padding: Annotated[
+ int, typer.Option(help="Digits to pad file name")
+ ] = FILE_NAME_0_PADDING_DEFAULT,
+ prefix: Annotated[str, typer.Option(help="Prefix for new file names")] = "",
+ dry_run: Annotated[
+ bool, typer.Option(help="Show changes without applying")
+ ] = True,
+ compress: Annotated[
+ bool, typer.Option(help="Whether to compress files")
+ ] = False,
+ compress_format: Annotated[
+ ArchiveFormatEnum,
+ typer.Option(case_sensitive=False, help="Compression format"),
+ ] = COMPRESSION_TYPE_DEFAULT,
+ compress_suffix: Annotated[
+ str, typer.Option(help="Compressed file name suffix")
+ ] = "",
+ compress_folder: Annotated[
+ Path, typer.Option(help="Optional folder to differ from renaming")
+ ] = COMPRESSED_PATH_DEFAULT,
+ delete_uncompressed: Annotated[
+ bool, typer.Option(help="Delete unneeded files after compression")
+ ] = False,
+ log_level: Annotated[
+ int, typer.Option(help="Set logging level for debugging")
+ ] = WARNING,
+ force: Annotated[
+ bool, typer.Option(--force, help="Force run without prompt")
+ ] = False,
+) -> None
+Manage file names and compression.
+ +alto2txt2fixture/cli.py105 +106 +107 +108 +109 +110 +111 +112 +113 +114 +115 +116 +117 +118 +119 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +140 +141 +142 +143 +144 +145 +146 +147 +148 +149 +150 +151 +152 +153 +154 +155 +156 +157 +158 +159 +160 +161 +162 +163 +164 +165 +166 +167 +168 +169 +170 +171 +172 +173 +174 +175 +176 +177 +178 +179 +180 +181 +182 +183 +184 +185 +186 +187 +188 +189 +190 +191 +192 +193 +194 +195 +196 +197 +198 +199 +200 +201 +202 +203 +204 +205 +206 +207 +208 +209 +210 +211 +212 +213 +214 +215 +216 +217 +218 +219 | |
show_fixture_tables(
+ run_settings: dotdict = settings,
+ print_in_call: bool = True,
+ data_provider_index: str = DATA_PROVIDER_INDEX,
+) -> list[Table]
+Print fixture tables specified in settings.fixture_tables in rich.Table format.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
run_settings |
+
+ dotdict
+ |
+
+
+
+
|
+
+ settings
+ |
+
print_in_call |
+
+ bool
+ |
+
+
+
+ whether to print to console (will use |
+
+ True
+ |
+
data_provider_index |
+
+ str
+ |
+
+
+
+ key to index |
+
+ DATA_PROVIDER_INDEX
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ list[Table]
+ |
+
+
+
+ A |
+
>>> fixture_tables: list[Table] = show_fixture_tables(
+... settings,
+... print_in_call=False)
+>>> len(fixture_tables)
+1
+>>> fixture_tables[0].title
+'dataprovider'
+>>> [column.header for column in fixture_tables[0].columns]
+['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']
+>>> fixture_tables = show_fixture_tables(settings)
+<BLANKLINE>
+...dataprovider...Heritage...│ bl_hmd...│ hmd...
+It is possible for the example test to fail in different screen sizes. Try +increasing the window or screen width of terminal used to check before +raising an issue.
+alto2txt2fixture/cli.pyGenerate a rich.table.Table for printing configuration to console.
alto2txt2fixture/cli.pyReturns a list with corrected data from a provided dictionary.
+ +alto2txt2fixture/create_adjacent_tables.pycsv2json_list(
+ csv_path: PathLike,
+ output_path: Path = OUTPUT,
+ saved: list[Path] | None = None,
+ indent: int = JSON_INDENT,
+) -> list
+Save csv_path as a json file and return as a list.
Managing Pandas DataFrame nan values via suggestion:
+https://stackoverflow.com/a/62691803/678486
alto2txt2fixture/create_adjacent_tables.pydownload_data(
+ files_dict: RemoteDataFilesType = {},
+ overwrite: bool = OVERWRITE,
+ exclude: list[str] = [],
+) -> None
+Download files in files_dict, overwrite if specified.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
files_dict |
+
+ RemoteDataFilesType
+ |
+
+
+
+
|
+
+ {}
+ |
+
overwrite |
+
+ bool
+ |
+
+
+
+
|
+
+ OVERWRITE
+ |
+
exclude |
+
+ list[str]
+ |
+
+
+
+
|
+
+ []
+ |
+
>>> from os import chdir
+>>> tmp_path: Path = getfixture('tmp_path')
+>>> set_path: Path = chdir(tmp_path)
+>>> download_data(exclude=["mitchells", "Newspaper-1", "linking"])
+Excluding mitchells...
+Excluding Newspaper-1...
+Excluding linking...
+Downloading cache...dict_admin_counties.json
+100% ... 37/37 bytes
+Downloading cache...dict_countries.json
+100% ... 33.2/33.2 kB
+Downloading cache...dict_historic_counties.json
+100% ... 41.4/41.4 kB
+Downloading cache...nlp_loc_wikidata_concat.csv
+100% ... 59.8/59.8 kB
+Downloading cache...wikidata_gazetteer_selected_columns.csv
+100% ... 47.8/47.8 MB
+alto2txt2fixture/create_adjacent_tables.py229 +230 +231 +232 +233 +234 +235 +236 +237 +238 +239 +240 +241 +242 +243 +244 +245 +246 +247 +248 +249 +250 +251 +252 +253 +254 +255 +256 +257 +258 +259 +260 +261 +262 +263 +264 +265 +266 +267 +268 +269 +270 +271 +272 +273 +274 +275 +276 +277 +278 +279 +280 +281 +282 +283 +284 +285 +286 +287 +288 +289 +290 +291 +292 +293 +294 +295 +296 +297 +298 | |
Get a list from a string, which contains
Return a dict of csv and json paths for each module_name table.
The csv and json paths
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
names |
+
+ Sequence[str]
+ |
+
+
+
+ iterable of names of each |
+ + required + | +
module_name |
+
+ str
+ |
+
+
+
+ name of module each name is part of, that is added as a prefix + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ TableOutputConfigType
+ |
+
+
+
+ A |
+
>>> pprint(get_outpaths_dict(MITCHELLS_TABELS, "mitchells"))
+{'Entry': {'csv': 'mitchells.Entry.csv', 'json': 'mitchells.Entry.json'},
+ 'Issue': {'csv': 'mitchells.Issue.csv', 'json': 'mitchells.Issue.json'},
+ 'PoliticalLeaning': {'csv': 'mitchells.PoliticalLeaning.csv',
+ 'json': 'mitchells.PoliticalLeaning.json'},
+ 'Price': {'csv': 'mitchells.Price.csv', 'json': 'mitchells.Price.json'}}
+alto2txt2fixture/create_adjacent_tables.pyrun(
+ files_dict: dict = {},
+ files_to_download_overwrite: bool = OVERWRITE,
+ saved: list[PathLike] = SAVED,
+ time_stamp: str = "",
+ output_path: Path = OUTPUT,
+) -> list[PathLike]
+Download, process and link files_dict to json and csv.
This will require access to https://zooniversedata.blob.core.windows.net/downloads/.
alto2txt2fixture/create_adjacent_tables.py301 +302 +303 +304 +305 +306 +307 +308 +309 +310 +311 +312 +313 +314 +315 +316 +317 +318 +319 +320 +321 +322 +323 +324 +325 +326 +327 +328 +329 +330 +331 +332 +333 +334 +335 +336 +337 +338 +339 +340 +341 +342 +343 +344 +345 +346 +347 +348 +349 +350 +351 +352 +353 +354 +355 +356 +357 +358 +359 +360 +361 +362 +363 +364 +365 +366 +367 +368 +369 +370 +371 +372 +373 +374 +375 +376 +377 +378 +379 +380 +381 +382 +383 +384 +385 +386 +387 +388 +389 +390 +391 +392 +393 +394 +395 +396 +397 +398 +399 +400 +401 +402 +403 +404 +405 +406 +407 +408 +409 +410 +411 +412 +413 +414 +415 +416 +417 +418 +419 +420 +421 +422 +423 +424 +425 +426 +427 +428 +429 +430 +431 +432 +433 +434 +435 +436 +437 +438 +439 +440 +441 +442 +443 +444 +445 +446 +447 +448 +449 +450 +451 +452 +453 +454 +455 +456 +457 +458 +459 +460 +461 +462 +463 +464 +465 +466 +467 +468 +469 +470 +471 +472 +473 +474 +475 +476 +477 +478 +479 +480 +481 +482 +483 +484 +485 +486 +487 +488 +489 +490 +491 +492 +493 +494 +495 +496 +497 +498 +499 +500 +501 +502 +503 +504 +505 +506 +507 +508 +509 +510 +511 +512 +513 +514 +515 +516 +517 +518 +519 +520 +521 +522 +523 +524 +525 +526 +527 +528 +529 +530 +531 +532 +533 +534 +535 +536 +537 +538 +539 +540 +541 +542 +543 +544 +545 +546 +547 +548 +549 +550 +551 +552 +553 +554 +555 +556 +557 +558 +559 +560 +561 +562 +563 +564 +565 +566 +567 +568 +569 +570 +571 +572 +573 +574 +575 +576 +577 +578 +579 +580 +581 +582 +583 +584 +585 +586 +587 +588 +589 +590 +591 +592 +593 +594 +595 +596 +597 +598 +599 +600 +601 +602 +603 +604 +605 +606 +607 +608 +609 +610 +611 +612 +613 +614 +615 +616 +617 +618 +619 +620 +621 +622 +623 +624 +625 +626 +627 +628 +629 +630 +631 +632 +633 +634 +635 +636 +637 +638 +639 +640 +641 +642 +643 +644 +645 +646 +647 +648 +649 +650 +651 +652 +653 +654 +655 +656 +657 +658 +659 +660 +661 +662 +663 +664 +665 +666 +667 +668 +669 +670 +671 +672 +673 +674 +675 +676 +677 +678 +679 +680 +681 +682 +683 +684 +685 +686 +687 +688 +689 +690 +691 +692 +693 +694 +695 +696 +697 +698 +699 +700 +701 +702 +703 +704 +705 +706 +707 +708 +709 +710 +711 +712 +713 +714 +715 +716 +717 +718 +719 +720 +721 +722 +723 +724 +725 +726 +727 +728 +729 +730 +731 +732 +733 +734 +735 +736 +737 +738 +739 +740 +741 +742 +743 +744 +745 +746 +747 +748 +749 +750 +751 +752 +753 +754 +755 +756 +757 +758 +759 +760 +761 +762 +763 +764 +765 +766 +767 +768 +769 +770 +771 | |
get_jisc_title(
+ title: str,
+ issue_date: str,
+ jisc_papers: pd.DataFrame,
+ input_sub_path: str,
+ publication_code: str,
+ abbr: str | None = None,
+) -> str
+Match a newspaper title with jisc_papers records.
Takes an input_sub_path, a publication_code, and an (optional)
+abbreviation for any newspaper to locate the title in the
+jisc_papers DataFrame. jisc_papers is usually loaded via the
+setup_jisc_papers function.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
title |
+
+ str
+ |
+
+
+
+ target newspaper title + |
+ + required + | +
issue_date |
+
+ str
+ |
+
+
+
+ target newspaper issue_date + |
+ + required + | +
jisc_papers |
+
+ pd.DataFrame
+ |
+
+
+
+
|
+ + required + | +
input_sub_path |
+
+ str
+ |
+
+
+
+ path of files to narrow down query input_sub_path + |
+ + required + | +
publication_code |
+
+ str
+ |
+
+
+
+ unique codes to match newspaper records + |
+ + required + | +
abbr |
+
+ str | None
+ |
+
+
+
+ an optional abbreviation of the newspaper title + |
+
+ None
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+ Matched |
+
Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+ A string estimating the JISC equivalent newspaper title + |
+
alto2txt2fixture/jisc.py90 + 91 + 92 + 93 + 94 + 95 + 96 + 97 + 98 + 99 +100 +101 +102 +103 +104 +105 +106 +107 +108 +109 +110 +111 +112 +113 +114 +115 +116 +117 +118 +119 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +140 +141 +142 +143 +144 +145 +146 +147 +148 +149 +150 +151 +152 +153 +154 +155 +156 +157 +158 +159 +160 +161 +162 +163 +164 +165 +166 +167 +168 +169 +170 +171 +172 +173 +174 +175 +176 +177 +178 +179 +180 +181 +182 +183 +184 +185 +186 +187 | |
Create a DataFrame with information in JISC_PAPERS_CSV in settings.
Returns:
+| Type | +Description | +
|---|---|
+ pd.DataFrame
+ |
+
+
+
+
|
+
alto2txt2fixture/jisc.py10 +11 +12 +13 +14 +15 +16 +17 +18 +19 +20 +21 +22 +23 +24 +25 +26 +27 +28 +29 +30 +31 +32 +33 +34 +35 +36 +37 +38 +39 +40 +41 +42 +43 +44 +45 +46 +47 +48 +49 +50 +51 +52 +53 +54 +55 +56 +57 +58 +59 +60 +61 +62 +63 +64 +65 +66 +67 +68 +69 +70 +71 +72 +73 +74 +75 +76 +77 +78 +79 +80 +81 +82 +83 +84 +85 +86 +87 | |
Print msg in colorama Force.RED and exit()
If silent exit() after call, else raise RuntimeError if crash=True.
alto2txt2fixture/log.pyfixtures(
+ filelist: list = [],
+ model: str = "",
+ translate: dict = {},
+ rename: dict = {},
+ uniq_keys: list = [],
+) -> Generator[FixtureDict, None, None]
+Generates fixtures for a specified model using a list of files.
+This function takes a list of files and generates fixtures for a specified +model. The fixtures can be used to populate a database or perform other +data-related operations.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
filelist |
+
+ list
+ |
+
+
+
+ A list of files to process and generate fixtures from. + |
+
+ []
+ |
+
model |
+
+ str
+ |
+
+
+
+ The name of the model for which fixtures are generated. +translate: A nested dictionary representing the translation mapping +for fields. The structure of the translator follows the format: + +The translated fields will be used as keys, and their +corresponding primary keys (obtained from the provided files) will +be used as values in the generated fixtures. + |
+
+ ''
+ |
+
rename |
+
+ dict
+ |
+ + + | +
+ {}
+ |
+
uniq_keys |
+
+ list
+ |
+
+
+
+ A list of fields that need to be considered for +uniqueness in the fixtures. If specified, the fixtures will yield +only unique items based on the combination of these fields. + |
+
+ []
+ |
+
Yields:
+| Type | +Description | +
|---|---|
+ FixtureDict
+ |
+
+
+
+
|
+
Returns:
+| Type | +Description | +
|---|---|
+ Generator[FixtureDict, None, None]
+ |
+
+
+
+ This function generates fixtures but does not return any value. + |
+
alto2txt2fixture/parser.py65 + 66 + 67 + 68 + 69 + 70 + 71 + 72 + 73 + 74 + 75 + 76 + 77 + 78 + 79 + 80 + 81 + 82 + 83 + 84 + 85 + 86 + 87 + 88 + 89 + 90 + 91 + 92 + 93 + 94 + 95 + 96 + 97 + 98 + 99 +100 +101 +102 +103 +104 +105 +106 +107 +108 +109 +110 +111 +112 +113 +114 +115 +116 +117 +118 +119 +120 +121 +122 +123 +124 +125 +126 +127 +128 +129 +130 +131 +132 +133 +134 +135 +136 +137 +138 +139 +140 +141 +142 +143 +144 +145 +146 +147 +148 +149 +150 +151 +152 +153 +154 +155 +156 | |
get_fields(
+ file: Union[Path, str, dict],
+ translate: dict = {},
+ rename: dict = {},
+ allow_null: bool = False,
+) -> dict
+Retrieves fields from a file and performs modifications and checks.
+This function takes a file (in various formats: Path, str, or dict)
+and processes its fields. It retrieves the fields from the file and
+performs modifications, translations, and checks on the fields.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
file |
+
+ Union[Path, str, dict]
+ |
+
+
+
+ The file from which the fields are retrieved. + |
+ + required + | +
translate |
+
+ dict
+ |
+ + + | +
+ {}
+ |
+
rename |
+
+ dict
+ |
+ + + | +
+ {}
+ |
+
allow_null |
+
+ bool
+ |
+
+
+
+ Determines whether to allow |
+
+ False
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ A dictionary representing the retrieved fields from the file, +with modifications and checks applied. + |
+
Raises:
+| Type | +Description | +
|---|---|
+ RuntimeError
+ |
+
+
+
+ If the file type is unsupported or if an error occurs +during field retrieval or processing. + |
+
alto2txt2fixture/parser.py258 +259 +260 +261 +262 +263 +264 +265 +266 +267 +268 +269 +270 +271 +272 +273 +274 +275 +276 +277 +278 +279 +280 +281 +282 +283 +284 +285 +286 +287 +288 +289 +290 +291 +292 +293 +294 +295 +296 +297 +298 +299 +300 +301 +302 +303 +304 +305 +306 +307 +308 +309 +310 +311 +312 +313 +314 +315 +316 +317 +318 +319 +320 +321 +322 +323 +324 +325 +326 +327 +328 +329 +330 +331 +332 +333 +334 +335 +336 +337 +338 +339 +340 +341 +342 +343 +344 +345 +346 +347 +348 +349 +350 +351 +352 +353 +354 +355 +356 +357 +358 +359 +360 +361 +362 +363 +364 +365 +366 +367 +368 +369 +370 +371 +372 +373 +374 +375 +376 +377 +378 +379 | |
Retrieves a specific key from a file and returns its value.
+This function reads a file and extracts the value of a specified +key. If the key is not found or an error occurs while processing +the file, a warning is printed, and an empty string is returned.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
item |
+
+ Path
+ |
+
+
+
+ The file from which the key is extracted. + |
+ + required + | +
x |
+
+ str
+ |
+
+
+
+ The key to be retrieved from the file. + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+ The value of the specified key from the file. + |
+
alto2txt2fixture/parser.pyConverts a list of fields into a nested dictionary representing a +translator.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
fields |
+
+ list[TranslatorTuple]
+ |
+
+
+
+ A list of tuples representing fields to be translated. + |
+
+ [TranslatorTuple('', '', [])]
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+ + + | +
alto2txt2fixture/parser.pyParses files from collections and generates fixtures for various models.
+This function processes files from the specified collections and generates
+fixtures for different models, such as newspapers.dataprovider,
+newspapers.ingest, newspapers.digitisation, newspapers.newspaper,
+newspapers.issue, and newspapers.item.
It performs various steps, such as file listing, fixture generation, +translation mapping, renaming fields, and saving fixtures to files.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
collections |
+
+ list
+ |
+
+
+
+ A list of collections from which files are +processed and fixtures are generated. + |
+ + required + | +
cache_home |
+
+ str
+ |
+
+
+
+ The directory path where the collections are located. + |
+ + required + | +
output |
+
+ str
+ |
+
+
+
+ The directory path where the fixtures will be saved. + |
+ + required + | +
max_elements_per_file |
+
+ int
+ |
+
+
+
+ The maximum number of elements per file +when saving fixtures. + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ This function generates fixtures but does not return any value. + |
+
alto2txt2fixture/parser.py382 +383 +384 +385 +386 +387 +388 +389 +390 +391 +392 +393 +394 +395 +396 +397 +398 +399 +400 +401 +402 +403 +404 +405 +406 +407 +408 +409 +410 +411 +412 +413 +414 +415 +416 +417 +418 +419 +420 +421 +422 +423 +424 +425 +426 +427 +428 +429 +430 +431 +432 +433 +434 +435 +436 +437 +438 +439 +440 +441 +442 +443 +444 +445 +446 +447 +448 +449 +450 +451 +452 +453 +454 +455 +456 +457 +458 +459 +460 +461 +462 +463 +464 +465 +466 +467 +468 +469 +470 +471 +472 +473 +474 +475 +476 +477 +478 +479 +480 +481 +482 +483 +484 +485 +486 +487 +488 +489 +490 +491 +492 +493 +494 +495 +496 +497 +498 +499 +500 +501 +502 +503 +504 +505 +506 +507 +508 +509 +510 +511 +512 +513 +514 +515 +516 +517 +518 +519 +520 +521 +522 +523 +524 +525 +526 +527 +528 +529 +530 +531 +532 +533 +534 +535 +536 +537 +538 +539 +540 +541 +542 +543 +544 +545 +546 +547 +548 +549 +550 +551 +552 +553 +554 +555 +556 +557 +558 +559 +560 | |
Resets the fixture directory by removing all JSON files inside it.
+This function takes a directory path (output) as input and removes all
+JSON files within the directory.
Prior to removal, it prompts the user for confirmation to proceed. If the +user confirms, the function clears the fixture directory by deleting the +JSON files.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
output |
+
+ str | Path
+ |
+
+
+
+ The directory path of the fixture directory to be reset. + |
+ + required + | +
Raises:
+| Type | +Description | +
|---|---|
+ RuntimeError
+ |
+
+
+
+ If the |
+
alto2txt2fixture/parser.pyGenerates unique items from a list of files based on specified keys.
+This function takes a list of files and yields unique items based on a
+combination of keys. The keys are extracted from each file using the
+get_key_from function, and duplicate items are ignored.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
filelist |
+
+ list
+ |
+
+
+
+ A list of files from which unique items are +generated. + |
+ + required + | +
keys |
+
+ list
+ |
+
+
+
+ A list of keys used for uniqueness. Each key specifies +a field to be used for uniqueness checking in the generated +items. + |
+
+ []
+ |
+
Yields:
+| Type | +Description | +
|---|---|
+ Any
+ |
+
+
+
+ A unique item from |
+
alto2txt2fixture/parser.py
+ Bases: TypedDict
A dict of lwmdb.fulltext.models.Fulltext fixture structure.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
path |
+
+ PathLike
+ |
+
+
+
+ Plaintext file path. + |
+
compressed_path |
+
+ PathLike | None
+ |
+
+
+
+ If |
+
primary_key |
+
+ int
+ |
+
+
+
+ An |
+
dataclass
+
+
+Manage exporting plaintext alto2txt results for lwmdb.fulltext.models.Fulltext.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
path |
+
+ PathLike
+ |
+
+
+
+ PathLike source for fixtures as either a folder or file. + |
+
data_provider_code |
+
+ str | None
+ |
+
+
+
+ A short string to uniquely identify |
+
files |
+
+ tuple[PathLike, ...] | None
+ |
+
+
+
+ A iterable |
+
compressed_glob_regex |
+
+ str
+ |
+
+
+
+ A Regular Expression to filter plaintext files from uncompressed
+ |
+
data_provider |
+
+ DataProviderFixtureDict | None
+ |
+
+
+
+ If available a |
+
model_str |
+
+ str
+ |
+
+
+
+ The name of the fulltext app has a fulltext model class specified in
+lwmdb.fulltext.models.fulltext. A sql table is generated from
+on that fulltext class and the json fixture structure generated
+from this class is where records will be stored.
+ |
+
extract_subdir |
+
+ PathLike
+ |
+
+
+
+ Folder to extract |
+
plaintext_extension |
+
+ str
+ |
+
+
+
+ What file extension to use to filter |
+
data_provider_code_dict |
+
+ dict[str, DataProviderFixtureDict]
+ |
+
+
+
+ A |
+
max_plaintext_per_fixture_file |
+
+ int
+ |
+
+
+
+ A maximum number of fixtures per fixture file, designed to configure +chunking fixtures. + |
+
saved_fixture_prefix |
+
+ str
+ |
+
+
+
+ A |
+
export_directory |
+
+ PathLike
+ |
+
+
+
+ Directory to save all exported fixtures to. + |
+
initial_pk |
+
+ int
+ |
+
+
+
+ Default begins at 1, can be set to another number if needed to
+add to add more to pre-existing set of records up to a given |
+
json_0_file_name_padding |
+
+ int
+ |
+
+
+
+ Number of |
+
_disk_usage |
+
+ int
+ |
+
+
+
+ Available harddrive space. Designed to help mitigate decompressing too +many files for available disk space. + |
+
self._uncompressed_source_file_dict |
+
+ int
+ |
+
+
+
+ A dictionary of extracted plaintext to compressed source file. This is
+a field in |
+
>>> path = getfixture('bl_lwm')
+>>> logger.setLevel(INFO)
+>>> plaintext_bl_lwm = PlainTextFixture(
+... data_provider_code='bl_lwm',
+... path=path,
+... compressed_glob_regex="*_plaintext.zip",
+... )
+>>> plaintext_bl_lwm
+<PlainTextFixture(path='...bl_lwm')>
+>>> plaintext_bl_lwm.info()
+<BLANKLINE>
+...PlainTextFixture for 2 'bl_lwm' files...
+┌─────────────────────┬────────────────────────────────...┐
+│ Path │ '...bl_lwm' ...│
+│ Compressed Files │ '...bl_lwm...0003079-test_plain...│
+│ │ '...bl_lwm...0003548-test_plain...│
+│ Extract Path │ '...bl_lwm...extracted' ...│
+│ Uncompressed Files │ None ...│
+│ Data Provider │ 'Living with Machines' ...│
+│ Initial Primary Key │ 1 ...│
+│ Max Rows Per JSON │ 100 ...│
+│ JSON File Name 0s │ 6 ...│
+└─────────────────────┴────────────────────────────────...┘
+>>> plaintext_bl_lwm.free_hd_space_in_GB > 1
+True
+>>> plaintext_bl_lwm.extract_compressed()
+<BLANKLINE>
+...Extract path:...'...lwm...extracted...
+...Extracting:...'...lwm...00030...
+...Extracting:...'...lwm...00035...
+...%...[...]...
+>>> plaintext_bl_lwm.delete_decompressed()
+Deleting all files in:...'...bl_lwm...tracted'
+property
+
+
+Return a tuple of all self.files with known archive file names.
property
+
+
+Return self.data_provider code attributre or None.
self.data_provider_code>>> bl_hmd = PlainTextFixture(
+... path=".",
+... data_provider_code="bl_hmd")
+>>> bl_hmd.data_provider_name
+'Heritage Made Digital'
+>>> bl_lwm = PlainTextFixture(
+... path='.',
+... data_provider=NEWSPAPER_DATA_PROVIDER_CODE_DICT['bl_lwm'],
+... )
+>>> bl_lwm.data_provider_name
+'Living with Machines'
+>>> plaintext_fixture = PlainTextFixture(
+... path=".")
+<BLANKLINE>
+...`.data_provider` and `.data_provider_code`...
+...are...'None'...in...<PlainTextFixture(path='.')>...
+>>> plaintext_fixture.data_provider_name
+`
+property
+
+
+If self._exported_json_paths return Generator of those paths.
Yields:
+| Type | +Description | +
|---|---|
| + | +
+
+
+ Each path from |
+
property
+
+
+Path any compressed files would be extracted to.
+property
+
+
+Return remaing hard drive space estimate in gigabytes.
+property
+
+
+property
+
+
+Return a tuple of all self.files with self.plaintext_extension.
property
+
+
+Return truncated self.compressed_files file names or empty str.
property
+
+
+Return truncated self.extract_path or empty str.
property
+
+
+Return truncated self.plaintext_provided_uncompressed file names or empty str.
property
+
+
+If self.compressed_files is in zip, return info, else None.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')
+>>> zipfile_info_list: list[ZipInfo] = list(plaintext_bl_lwm.zipinfo)
+Getting zipfile info from <PlainTextFixture(path='...bl_lwm')>
+>>> zipfile_info_list[0][-1].filename
+'0003079...1898...0204...0003079_18980204_sect0001.txt'
+>>> zipfile_info_list[-1][-1].filename
+'0003548...1904...0707...0003548_19040707_art0059.txt'
+>>> zipfile_info_list[0][-1].file_size
+70192
+>>> zipfile_info_list[0][-1].compress_size
+39911
+Manage populating additional attributes if necessary.
+ +alto2txt2fixture/plaintext.pyReturn class name with count and DataProvider if available.
alto2txt2fixture/plaintext.pycompress_json_exports(
+ output_path: PathLike | None = None, format: ArchiveFormatEnum | None = None
+) -> tuple[Path, ...]
+Compress self._exported_json_paths to format.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
output_path |
+
+ PathLike | None
+ |
+
+
+
+
|
+
+ None
+ |
+
format |
+
+ ArchiveFormatEnum | None
+ |
+
+
+
+ What compression format to use from |
+
+ None
+ |
+
Neither output_path nor format overwrite the related attributes
+of self.
>>> if is_platform_win:
+... pytest.skip('decompression fails on Windows: issue #55')
+>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')
+<BLANKLINE>
+...
+>>> compressed_paths: Path = plaintext_bl_lwm.compress_json_exports(
+... format='tar')
+<BLANKLINE>
+...Compressing...'...01.json' to...'tar'...in:...
+>>> compressed_paths
+(...Path('.../plaintext_fixture-000001.json.tar'),)
+alto2txt2fixture/plaintext.pyRemove all files in self.extract_path.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')
+<BLANKLINE>
+...Extract path:...'...bl_lwm...extracted'...
+>>> plaintext_bl_lwm.delete_decompressed()
+Deleting all files in:...
+>>> plaintext_bl_lwm.delete_decompressed()
+<BLANKLINE>
+...Extract path empty:...'...bl_lwm...extracted'...
+alto2txt2fixture/plaintext.pyexport_to_json_fixtures(
+ output_path: PathLike | None = None,
+ prefix: str | None = None,
+ json_0_file_name_padding: int | None = None,
+) -> None
+Iterate over self.plaintext_paths exporting to json django fixtures.
For errors running on windows see: +#55
+Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
output_path |
+
+ PathLike | None
+ |
+
+
+
+ Folder to save all |
+
+ None
+ |
+
prefix |
+
+ str | None
+ |
+
+
+
+ Any |
+
+ None
+ |
+
json_0_file_name_padding |
+
+ int | None
+ |
+
+
+
+ Number of |
+
+ None
+ |
+
>>> if is_platform_win:
+... pytest.skip('decompression fails on Windows: issue #55')
+>>> bl_lwm: Path = getfixture("bl_lwm")
+>>> first_lwm_plaintext_json_dict: PlaintextFixtureDict = (
+... getfixture("first_lwm_plaintext_json_dict")
+... )
+>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')
+<BLANKLINE>
+...Extract path:...bl_lwm...extracted...
+>>> plaintext_bl_lwm.export_to_json_fixtures(output_path=bl_lwm / "output")
+<BLANKLINE>
+Compressed configs...%...[...]
+>>> len(plaintext_bl_lwm._exported_json_paths)
+1
+>>> plaintext_bl_lwm._exported_json_paths
+(...Path(...plaintext_fixture-000001.json...),)
+>>> import json
+>>> exported_json = json.loads(
+... plaintext_bl_lwm._exported_json_paths[0].read_text()
+... )
+>>> exported_json[0]['pk'] == first_lwm_plaintext_json_dict['pk']
+True
+>>> exported_json[0]['model'] == first_lwm_plaintext_json_dict['model']
+True
+>>> (exported_json[0]['fields']['text'] ==
+... first_lwm_plaintext_json_dict['fields']['text'])
+True
+>>> (exported_json[0]['fields']['path'] ==
+... str(first_lwm_plaintext_json_dict['fields']['path']))
+True
+>>> (exported_json[0]['fields']['compressed_path'] ==
+... str(first_lwm_plaintext_json_dict['fields']['compressed_path']))
+True
+>>> exported_json[0]['fields']['created_at']
+'20...'
+>>> (exported_json[0]['fields']['updated_at'] ==
+... exported_json[0]['fields']['updated_at'])
+True
+alto2txt2fixture/plaintext.py610 +611 +612 +613 +614 +615 +616 +617 +618 +619 +620 +621 +622 +623 +624 +625 +626 +627 +628 +629 +630 +631 +632 +633 +634 +635 +636 +637 +638 +639 +640 +641 +642 +643 +644 +645 +646 +647 +648 +649 +650 +651 +652 +653 +654 +655 +656 +657 +658 +659 +660 +661 +662 +663 +664 +665 +666 +667 +668 +669 +670 +671 +672 +673 +674 +675 +676 +677 +678 +679 +680 +681 +682 +683 +684 +685 +686 +687 +688 +689 +690 | |
Extract self.compressed_files to self.extracted_subdir_name.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')
+>>> plaintext_bl_lwm.extract_compressed()
+<BLANKLINE>
+...Extract path:...'...bl_lwm...extracted'...
+>>> filter_sect1_txt: list[str] = [txt_file for txt_file in
+... plaintext_bl_lwm._uncompressed_source_file_dict.keys()
+... if txt_file.name.endswith('204_sect0001.txt')]
+>>> len(filter_sect1_txt)
+1
+>>> plaintext_bl_lwm._uncompressed_source_file_dict[
+... filter_sect1_txt[0]
+... ]
+<BLANKLINE>
+...Path('...bl_lwm...0003079-test_plaintext.zip')
+>>> plaintext_bl_lwm.delete_decompressed()
+Deleting all files in:...'...bl_lwm...tracted'
+alto2txt2fixture/plaintext.pyReturn a generator of all plaintext files for potential fixtures.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')
+<BLANKLINE>
+...Extract path:...bl_lwm...extracted...
+>>> plaintext_paths = plaintext_bl_lwm.plaintext_paths()
+>>> first_path_fixture_dict = next(iter(plaintext_paths))
+>>> first_path_fixture_dict['path'].name
+'0003079_18980107_art0001.txt'
+>>> first_path_fixture_dict['compressed_path'].name
+'0003079-test_plaintext.zip'
+>>> len(plaintext_bl_lwm._pk_plaintext_dict)
+1
+>>> plaintext_bl_lwm._pk_plaintext_dict[
+... first_path_fixture_dict['path']
+... ] # This demonstrates the `pk` begins from 1 following `SQL` standards
+1
+alto2txt2fixture/plaintext.pyGenerate fixture dicts from self.plaintext_paths.
For errors running on windows see: +#55
+>>> if is_platform_win:
+... pytest.skip('decompression fails on Windows: issue #55')
+>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')
+<BLANKLINE>
+...Extract path:...bl_lwm...extracted...
+>>> paths_dict = list(plaintext_bl_lwm.plaintext_paths_to_dicts())
+Compressed configs :...%...[ ... it/s ]
+>>> plaintext_bl_lwm.delete_decompressed()
+Deleting all files in: '...tracted'
+alto2txt2fixture/plaintext.pyset_exported_json_paths(
+ export_directory: PathLike | None,
+ saved_fixture_prefix: str | None,
+ overwrite: bool = False,
+) -> None
+Set self._exported_json_paths for use with self.exported_json_paths.
If provided export_directory and saved_fixture_prefix will
+overwite those attributes on self.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
export_directory |
+
+ PathLike | None
+ |
+
+
+
+
|
+ + required + | +
saved_fixture_prefix |
+
+ str | None
+ |
+
+
+
+
|
+ + required + | +
overwrite |
+
+ bool
+ |
+
+
+
+ Force replace |
+
+ False
+ |
+
>>> if is_platform_win:
+... pytest.skip('decompression fails on Windows: issue #55')
+>>> tmp_path = getfixture('tmp_path')
+>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')
+<BLANKLINE>
+...
+>>> tuple(plaintext_bl_lwm.exported_json_paths)
+(...Path('...plaintext_fixture-000001.json'),)
+>>> plaintext_bl_lwm.set_exported_json_paths(tmp_path, 'check-prefix')
+Traceback (most recent call last):
+ ...
+ValueError: Cannot overwrite 'self._exported_json_paths' without
+'overwrite' = True. Current 'self._exported_json_paths':
+(...Path('...plaintext_fixture-000001.json'),)
+>>> plaintext_bl_lwm.set_exported_json_paths(tmp_path,
+... 'check-prefix', overwrite=True)
+<BLANKLINE>
+...Force change '._exported_json_paths' in...<PlainTextFixture...
+>>> plaintext_bl_lwm.export_directory == tmp_path
+True
+>>> plaintext_bl_lwm.saved_fixture_prefix
+'check-prefix'
+alto2txt2fixture/plaintext.py719 +720 +721 +722 +723 +724 +725 +726 +727 +728 +729 +730 +731 +732 +733 +734 +735 +736 +737 +738 +739 +740 +741 +742 +743 +744 +745 +746 +747 +748 +749 +750 +751 +752 +753 +754 +755 +756 +757 +758 +759 +760 +761 +762 +763 +764 +765 +766 +767 +768 +769 +770 +771 +772 +773 +774 +775 +776 +777 +778 +779 +780 +781 +782 +783 | |
Archive(
+ path: str | Path,
+ collection: str = "",
+ report_id: str | None = None,
+ jisc_papers: pd.DataFrame | None = None,
+ json_indent: int = JSON_INDENT,
+)
+Manage extracting information from a ZIP archive.
+The Archive class represents a zip archive of XML files. The class is used
+to extract information from a ZIP archive, and it contains several methods
+to process the data contained in the archive.
open(Archive) context manager
Archive can be opened with a context manager, which creates a meta +object, with timings for the object. When closed, it will save the +meta JSON to the correct paths.
+Attributes:
+| Name | +Type | +Description | +
|---|---|---|
path |
+
+ Path
+ |
+
+
+
+ The path to the zip archive. + |
+
collection |
+
+ str
+ |
+
+
+
+ The collection of the XML files in the archive. Default is "". + |
+
report |
+
+ Path
+ |
+
+
+
+ The file path of the report file for the archive. + |
+
report_id |
+
+ str
+ |
+
+
+
+ The report ID for the archive. If not provided, a random UUID is +generated. + |
+
report_parent |
+
+ Path
+ |
+
+
+
+ The parent directory of the report file for the archive. + |
+
jisc_papers |
+
+ pd.DataFrame
+ |
+
+
+
+ A DataFrame of JISC papers. + |
+
size |
+
+ str | float
+ |
+
+
+
+ The size of the archive, in human-readable format. + |
+
size_raw |
+
+ str | float
+ |
+
+
+
+ The raw size of the archive, in bytes. + |
+
roots |
+
+ Generator[ET.Element, None, None]
+ |
+
+
+
+ The root elements of the XML documents contained in the archive. + |
+
meta |
+
+ dotdict
+ |
+
+
+
+ Metadata about the archive, such as its path, size, and number of contents. + |
+
json_indent |
+
+ int
+ |
+
+
+
+ Indentation formatting of |
+
Raises:
+| Type | +Description | +
|---|---|
+ RuntimeError
+ |
+
+
+
+ If the |
+
Constructor method.
+ +alto2txt2fixture/router.pyproperty
+
+
+Property that calls the get_documents method
property
+
+
+Returns the list of files in the zip file
+A generator that yields instances of the Document class for each XML +file in the ZIP archive.
+It uses the tqdm library to display a progress bar in the terminal
+while it is running.
If the contents of the ZIP file are not empty, the method creates an
+instance of the Document class by passing the root element of the XML
+file, the collection name, meta information about the archive, and the
+JISC papers data frame (if provided) to the constructor of the
+Document class. The instance of the Document class is then
+returned by the generator.
Yields:
+| Type | +Description | +
|---|---|
+ Document
+ |
+
+
+
+
|
+
alto2txt2fixture/router.pyYields the root elements of the XML documents contained in the archive.
+ +alto2txt2fixture/router.pyThe Cache class provides a blueprint for creating and managing cache data. +The class has several methods that help in getting the cache path, +converting the data to a dictionary, and writing the cache data to a file.
+It is inherited by many other classes in this document.
+ +Initializes the Cache class object.
+ +alto2txt2fixture/router.pyReturns the cache path, which is used to store the cache data. +The path is normally constructed using some of the object's +properties (collection, kind, and id) but can be changed when +inherited.
+ +alto2txt2fixture/router.pyWrites the cache data to a file at the specified cache path. The cache +data is first converted to a dictionary using the as_dict method. If +the cache path already exists, the function returns True.
+ +alto2txt2fixture/router.pyA Collection represents a group of newspaper archives from any passed +alto2txt metadata output.
+A Collection is initialised with a name and an optional pandas DataFrame
+of JISC papers. The archives property returns an iterable of the
+Archive objects within the collection.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
name |
+
+ str
+ |
+
+
+
+ Name of the collection (default "hmd") + |
+
jisc_papers |
+
+ pandas.DataFrame
+ |
+
+
+
+ DataFrame of JISC papers, optional + |
+
Constructor method.
+ +alto2txt2fixture/router.py
+ Bases: Cache
The DataProvider class extends the Cache class and represents a newspaper +data provider. The class has several properties and methods that allow +creation of a data provider object and the manipulation of its data.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
collection |
+
+ str
+ |
+
+
+
+ A string representing publication collection + |
+
kind |
+
+ str
+ |
+
+
+
+ Indication of object type, defaults to |
+
providers_meta_data |
+
+ list[FixtureDict]
+ |
+
+
+
+ structured dict of metadata for known collection sources + |
+
collection_type |
+
+ str
+ |
+
+
+
+ related data sources and potential linkage source + |
+
index_field |
+
+ str
+ |
+
+
+
+ field name for querying existing records + |
+
>>> from pprint import pprint
+>>> hmd = DataProvider("hmd")
+>>> hmd.pk
+2
+>>> pprint(hmd.as_dict())
+{'code': 'bl_hmd',
+ 'collection': 'newspapers',
+ 'legacy_code': 'hmd',
+ 'name': 'Heritage Made Digital',
+ 'source_note': 'British Library-funded digitised newspapers provided by the '
+ 'British Newspaper Archive'}
+Constructor method.
+ +alto2txt2fixture/router.pyproperty
+
+
+Return self.providers_meta_data[self.collection] or {}.
property
+
+
+Return self.providers_meta_data[self.collection] or {}.
property
+
+
+Return pk if provided via providers_meta_data, else None.
property
+
+
+Return all self.index_field values from providers_meta_data.
Return a dict of the data provider object.
Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ Dictionary representation of the DataProvider object + |
+
alto2txt2fixture/router.py
+ Bases: Cache
The Digitisation class extends the Cache class and represents a newspaper +digitisation. The class has several properties and methods that allow +creation of an digitisation object and the manipulation of its data.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
root |
+
+ ET.Element
+ |
+
+
+
+ An xml element that represents the root of the publication + |
+
collection |
+
+ str
+ |
+
+
+
+ A string that represents the collection of the publication + |
+
Constructor method.
+ +alto2txt2fixture/router.pyclass-attribute
+ instance-attribute
+
+
+A string that represents the type of the object, set to +"digitisation".
+A method that returns a dictionary representation of the digitisation +object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ Dictionary representation of the Digitising object + |
+
alto2txt2fixture/router.pyThe Document class is a representation of a document that contains +information about a publication, newspaper, item, digitisation, and +ingest. This class holds all the relevant information about a document in +a structured manner and provides properties that can be used to access +different aspects of the document.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
collection |
+
+ str | None
+ |
+
+
+
+ A string that represents the collection of the publication + |
+
root |
+
+ ET.Element | None
+ |
+
+
+
+ An |
+
zip_file |
+
+ str | None
+ |
+
+
+
+ A path to a valid |
+
jisc_papers |
+
+ pd.DataFrame | None
+ |
+
+
+
+ A |
+
meta |
+
+ dotdict | None
+ |
+
+
+
+ TODO + |
+
Constructor method.
+ +alto2txt2fixture/router.py
+ Bases: Cache
The Ingest class extends the Cache class and represents a newspaper ingest. +The class has several properties and methods that allow the creation of an +ingest object and the manipulation of its data.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
root |
+
+ ET.Element
+ |
+
+
+
+ An xml element that represents the root of the publication + |
+
collection |
+
+ str
+ |
+
+
+
+ A string that represents the collection of the publication + |
+
Constructor method.
+ +alto2txt2fixture/router.pyclass-attribute
+ instance-attribute
+
+
+A string that represents the type of the object, set to "ingest".
+A method that returns a dictionary representation of the ingest +object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ Dictionary representation of the Ingest object + |
+
alto2txt2fixture/router.pyIssue(
+ publication: ET.Element,
+ newspaper: Optional[Newspaper] = None,
+ collection: str = "",
+ input_sub_path: str = "",
+ meta: dotdict = dotdict(),
+)
+
+ Bases: Cache
The Issue class extends the Cache class and represents a newspaper issue. +The class has several properties and methods that allow the creation of an +issue object and the manipulation of its data.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
root |
+ + | +
+
+
+ An xml element that represents the root of the publication + |
+
newspaper |
+
+ Newspaper | None
+ |
+
+
+
+ The parent newspaper + |
+
collection |
+
+ str
+ |
+
+
+
+ A string that represents the collection of the publication + |
+
input_sub_path |
+
+ str
+ |
+
+
+
+ TODO + |
+
meta |
+
+ dotdict
+ |
+
+
+
+ TODO + |
+
Constructor method.
+ +alto2txt2fixture/router.pyproperty
+
+
+Sets up and saves the issue code for easy access as property.
+property
+
+
+Sets up and saves the issue date for easy access as property.
+class-attribute
+ instance-attribute
+
+
+A string that represents the type of the object, set to "issue".
+A method that returns a dictionary representation of the issue +object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ Dictionary representation of the Issue object + |
+
alto2txt2fixture/router.pyReturns the path to the cache file for the issue object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ Path
+ |
+
+
+
+ Path to the cache file for the issue object + |
+
alto2txt2fixture/router.pyItem(
+ root: ET.Element,
+ issue_code: str = "",
+ digitisation: dict = {},
+ ingest: dict = {},
+ collection: str = "",
+ newspaper: Optional[Newspaper] = None,
+ meta: dotdict = dotdict(),
+)
+
+ Bases: Cache
The Newspaper class extends the Cache class and represents a newspaper +item, i.e. an article. The class has several properties and methods that +allow the creation of an article object and the manipulation of its data.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
root |
+
+ ET.Element
+ |
+
+
+
+ An xml element that represents the root of the publication + |
+
issue_code |
+
+ str
+ |
+
+
+
+ A string that represents the issue code + |
+
digitisation |
+
+ dict
+ |
+
+
+
+ TODO + |
+
ingest |
+
+ dict
+ |
+
+
+
+ TODO + |
+
collection |
+
+ str
+ |
+
+
+
+ A string that represents the collection of the publication + |
+
newspaper |
+
+ Newspaper | None
+ |
+
+
+
+ The parent newspaper + |
+
meta |
+
+ dotdict
+ |
+
+
+
+ TODO + |
+
Constructor method.
+ +alto2txt2fixture/router.pyproperty
+
+
+Sets up and saves the item code for easy access as property.
+property
+
+
+Sets up and saves the issue XML item for easy access as a property.
+class-attribute
+ instance-attribute
+
+
+A string that represents the type of the object, set to "item".
+A method that returns a dictionary representation of the item object +(i.e. article).
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ Dictionary representation of the Item object + |
+
alto2txt2fixture/router.pyReturns the path to the cache file for the item (article) object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ Path
+ |
+
+
+
+ Path to the cache file for the article object + |
+
alto2txt2fixture/router.pySpecial cache-write function that appends rather than writes at the +end of the process.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ None. + |
+
alto2txt2fixture/router.pyNewspaper(
+ root: ET.Element,
+ collection: str = "",
+ meta: dotdict = dotdict(),
+ jisc_papers: Optional[pd.DataFrame] = None,
+)
+
+ Bases: Cache
The Newspaper class extends the Cache class and represents a newspaper.
+The class has several properties and methods that allow the creation of a +newspaper object and the manipulation of its data.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
root |
+ + | +
+
+
+ An xml element that represents the root of the publication. + |
+
collection |
+ + | +
+
+
+ A string that represents the collection of the publication. + |
+
meta |
+ + | +
+
+
+ A dotdict object that holds metadata about the publication. + |
+
jisc_papers |
+ + | +
+
+
+ A pandas DataFrame object for JISC paper information. + |
+
Constructor method.
+ +alto2txt2fixture/router.pyclass-attribute
+ instance-attribute
+
+
+A string that represents the type of the object, set to "newspaper".
+property
+
+
+Returns the nested directories in which we want to save the cache file.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ list
+ |
+
+
+
+ List of the desired directories in descending order + |
+
property
+
+
+A property that returns the code of the publication.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+ The code of the publication + |
+
property
+
+
+A property that returns the title of the newspaper.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+ The title of the newspaper + |
+
A method that returns a dictionary representation of the newspaper +object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ Dictionary representation of the Newspaper object + |
+
alto2txt2fixture/router.pyReturns the path to the cache file for the newspaper object.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ Path
+ |
+
+
+
+ Path to the cache file for the newspaper object + |
+
alto2txt2fixture/router.pyA method that returns the publication code from the input sub-path of +the publication process.
+ + + +Returns:
+| Type | +Description | +
|---|---|
+ str | None
+ |
+
+
+
+ The code of the publication + |
+
alto2txt2fixture/router.pyroute(
+ collections: list,
+ cache_home: str,
+ mountpoint: str,
+ jisc_papers_path: str,
+ report_dir: str,
+) -> None
+This function is responsible for setting up the path for the alto2txt +mountpoint, setting up the JISC papers and routing the collections for +processing.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
collections |
+
+ list
+ |
+
+
+
+ List of collection names + |
+ + required + | +
cache_home |
+
+ str
+ |
+
+
+
+ Directory path for the cache + |
+ + required + | +
mountpoint |
+
+ str
+ |
+
+
+
+ Directory path for the alto2txt mountpoint + |
+ + required + | +
jisc_papers_path |
+
+ str
+ |
+
+
+
+ Path to the JISC papers + |
+ + required + | +
report_dir |
+
+ str
+ |
+
+
+
+ Path to the report directory + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ None + |
+
alto2txt2fixture/router.py1135 +1136 +1137 +1138 +1139 +1140 +1141 +1142 +1143 +1144 +1145 +1146 +1147 +1148 +1149 +1150 +1151 +1152 +1153 +1154 +1155 +1156 +1157 +1158 +1159 +1160 +1161 +1162 +1163 +1164 +1165 +1166 +1167 +1168 +1169 +1170 +1171 +1172 +1173 +1174 +1175 +1176 +1177 +1178 +1179 +1180 +1181 +1182 +1183 +1184 +1185 +1186 +1187 +1188 +1189 +1190 +1191 +1192 +1193 +1194 +1195 +1196 +1197 +1198 | |
The settings module provides configuration for running alto2txt2fixture.
Most of these are managed within the settings variable within this module.
Note
+See the command line interface parameters documentation for means of modifying settings when run.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
JSON_INDEX |
+ + | +
+
+
+ Amount of indentation to include in output |
+
DATA_PROVIDER_INDEX |
+
+ Final[str]
+ |
+
+
+
+ The |
+
NEWSPAPER_COLLECTION_METADATA |
+
+ Final[list[DataProviderFixtureDict]]
+ |
+
+
+
+ A list of |
+
SETUP_TITLE |
+
+ str
+ |
+
+
+
+ the title printed at the commandline via |
+
settings |
+
+ dotdict
+ |
+
+
+
+ a |
+
+ Bases: TypedDict
Fields within the fields portion of a FixtureDict to fit lwmdb.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
name |
+
+ str
+ |
+
+
+
+ The name of the collection data source. For |
+
code |
+
+ str | NEWSPAPER_OCR_FORMATS
+ |
+
+
+
+ A short slug-like, url-compatible (replace spaces with |
+
legacy_code |
+
+ LEGACY_NEWSPAPER_OCR_FORMATS | None
+ |
+
+
+
+ Either blank or a legacy slug-like, url-compatible (replace spaces with
+ |
+
collection |
+
+ str
+ |
+
+
+
+ Data Provider type. + |
+
source_note |
+
+ str | None
+ |
+
+
+
+ A sentence about the data provider. + |
+
+ Bases: FixtureDictBaseClass
A dict structure for DataProvider sources in line with lwmdb.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
pk |
+
+ int
+ |
+
+
+
+ an id to uniquely define and query each entry + |
+
model |
+
+ str
+ |
+
+
+
+ what model a given record is for + |
+
fields |
+
+ DataProviderFieldsDict
+ |
+
+
+
+ a |
+
+ Bases: FixtureDictBaseClass
A dict structure to ease use as a json database fixture.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
pk |
+ + | +
+
+
+ an id to uniquely define and query each entry + |
+
model |
+ + | +
+
+
+ what model a given record is for + |
+
fields |
+
+ dict[str, Any]
+ |
+
+
+
+ a |
+
+ Bases: TypedDict
A base dict structure for json fixtures.
+ Bases: FixtureDictBaseClass
A dict structure for Fulltext sources in line with lwmdb.
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
model |
+
+ str
+ |
+
+
+
+
|
+
fields |
+
+ PlaintextFixtureFieldsDict
+ |
+
+
+
+ a |
+
pk |
+
+ int
+ |
+
+
+
+
|
+
No pk is included. By not specifying one, django should generate new onces during
+import.
+ Bases: TypedDict
A typed dict for Plaintext Fixutres to match lwmdb.Fulltext model
Attributes:
+| Name | +Type | +Description | +
|---|---|---|
text |
+
+ str
+ |
+
+
+
+ Plaintext, potentially quite large newspaper articles. +May have unusual or unreadable sequences of characters +due to issues with Optical Character Recognition quality. + |
+
path |
+
+ str
+ |
+
+
+
+ Path of provided plaintext file. If |
+
compressed_path |
+
+ str | None
+ |
+
+
+
+ The path of a compressed data source, the extraction of which provides
+access to |
+
+ Bases: NamedTuple
A named tuple of fields for translation.
+ + + +Attributes:
+| Name | +Type | +Description | +
|---|---|---|
start |
+
+ str
+ |
+
+
+
+ A string representing the starting field name. + |
+
finish |
+
+ str | list
+ |
+
+
+
+ A string or list specifying the field(s) to be translated. +If it is a string, the translated field +will be a direct mapping of the specified field in +each item of the input list. +If it is a list, the translated field will be a +hyphen-separated concatenation of the specified fields +in each item of the input list. + |
+
lst |
+
+ list[dict]
+ |
+
+
+
+ A list of dictionaries representing the items to be
+translated. Each dictionary should contain the necessary
+fields for translation, with the field names specified in
+the |
+
+ Bases: dict
dot.notation access to dictionary attributes
+ + + + +
+ Bases: NamedTuple
Type hint for nametuple returned from disk_usage.
check_newspaper_collection_configuration(
+ collections: Iterable[str] = settings.COLLECTIONS,
+ newspaper_collections: Iterable[
+ FixtureDict
+ ] = NEWSPAPER_COLLECTION_METADATA,
+ data_provider_index: str = DATA_PROVIDER_INDEX,
+) -> set[str]
+Check the names in collections match the names in newspaper_collections.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
collections |
+
+ Iterable[str]
+ |
+
+
+
+ Names of newspaper collections, defaults to |
+
+ settings.COLLECTIONS
+ |
+
newspaper_collections |
+
+ Iterable[FixtureDict]
+ |
+
+
+
+ Newspaper collections in a list of |
+
+ NEWSPAPER_COLLECTION_METADATA
+ |
+
data_provider_index |
+
+ str
+ |
+
+
+
+
|
+
+ DATA_PROVIDER_INDEX
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ set[str]
+ |
+
+
+
+ A set of |
+
>>> check_newspaper_collection_configuration()
+set()
+>>> unmatched: set[str] = check_newspaper_collection_configuration(
+... ["cat", "dog"])
+<BLANKLINE>
+...Warning: 2 `collections` not in `newspaper_collections`: ...
+>>> unmatched == {'dog', 'cat'}
+True
+Note
+Set orders are random so checking unmatched == {'dog, 'cat'} to
+ensure correctness irrespective of order in the example above.
alto2txt2fixture/utils.pyClears the cache directory by removing all .json files in it.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
dir |
+
+ str | Path
+ |
+
+
+
+ The path of the directory to be cleared. + |
+ + required + | +
alto2txt2fixture/utils.pycompress_fixture(
+ path: PathLike,
+ output_path: PathLike | str = settings.OUTPUT,
+ suffix: str = "",
+ format: str | ArchiveFormatEnum = ZIP_FILE_EXTENSION,
+ force_overwrite: bool = False,
+ dry_run: bool = False,
+) -> Path
+Compress exported fixtures files using make_archive.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
path |
+
+ PathLike
+ |
+
+
+
+
|
+ + required + | +
output_path |
+
+ PathLike | str
+ |
+
+
+
+ Compressed file name (without extension specified from |
+
+ settings.OUTPUT
+ |
+
format |
+
+ str | ArchiveFormatEnum
+ |
+
+
+
+ A |
+
+ ZIP_FILE_EXTENSION
+ |
+
suffix |
+
+ str
+ |
+
+
+
+
|
+
+ ''
+ |
+
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')
+<BLANKLINE>
+...
+>>> tmp_path = getfixture('tmp_path')
+>>> json_path: Path = next(plaintext_bl_lwm.exported_json_paths)
+>>> assert 'pytest-of' in str(json_path)
+>>> compressed_path: Path = compress_fixture(path=json_path,
+... output_path=tmp_path,
+... dry_run=True)
+<BLANKLINE>
+...creating...'...plain...-...01.json.zip...'...addin...
+...'plain...01.json'...to...it...
+>>> compressed_path.exists()
+False
+>>> compressed_path: Path = compress_fixture(path=json_path,
+... output_path=tmp_path,
+... dry_run=False)
+<BLANKLINE>
+...creating...'...plain...-...01.json.zip...'...addin...
+...'plain...01.json'...to...it...
+>>> from zipfile import ZipFile, ZipInfo
+>>> zipfile_info_list: list[ZipInfo] = ZipFile(
+... tmp_path / 'plaintext_fixture-000001.json.zip'
+... ).infolist()
+>>> len(zipfile_info_list)
+1
+>>> Path(zipfile_info_list[0].filename).name
+'plaintext_fixture-000001.json'
+alto2txt2fixture/utils.py1114 +1115 +1116 +1117 +1118 +1119 +1120 +1121 +1122 +1123 +1124 +1125 +1126 +1127 +1128 +1129 +1130 +1131 +1132 +1133 +1134 +1135 +1136 +1137 +1138 +1139 +1140 +1141 +1142 +1143 +1144 +1145 +1146 +1147 +1148 +1149 +1150 +1151 +1152 +1153 +1154 +1155 +1156 +1157 +1158 +1159 +1160 +1161 +1162 +1163 +1164 +1165 +1166 +1167 +1168 +1169 +1170 +1171 +1172 +1173 +1174 +1175 +1176 +1177 +1178 +1179 +1180 +1181 +1182 +1183 +1184 +1185 +1186 +1187 +1188 +1189 +1190 +1191 +1192 +1193 +1194 +1195 +1196 +1197 +1198 +1199 +1200 +1201 +1202 +1203 +1204 +1205 +1206 +1207 +1208 +1209 +1210 +1211 +1212 +1213 +1214 +1215 +1216 +1217 +1218 +1219 +1220 +1221 +1222 +1223 +1224 +1225 +1226 +1227 +1228 +1229 +1230 +1231 +1232 | |
Copy files from copy_path_dict keys to values.
>>> tmp_path: Path = getfixture('tmp_path')
+>>> for i in range(4):
+... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)
+>>> pprint(sorted(tmp_path.iterdir()))
+[...Path('...test_file-0.txt'),
+ ...Path('...test_file-1.txt'),
+ ...Path('...test_file-2.txt'),
+ ...Path('...test_file-3.txt')]
+>>> output_path = tmp_path / 'save'
+>>> output_path.mkdir(exist_ok=True)
+>>> copy_dict_paths(
+... glob_path_rename_by_0_padding(tmp_path,
+... glob_regex_str="*.txt",
+... output_path=output_path))
+<BLANKLINE>
+...Specified...'...save'...for...saving...file...copies...
+...'...-0...txt'...to...'...-00...txt...'...
+...'...-1...txt'...to...'...-01...txt...'
+...'...-2...txt'...to...'...-02...txt...'
+...'...-3...txt'...to...'...-03...txt...'
+>>> pprint(sorted(tmp_path.iterdir()))
+[...Path('...save'),
+ ...Path('...test_file-0.txt'),
+ ...Path('...test_file-1.txt'),
+ ...Path('...test_file-2.txt'),
+ ...Path('...test_file-3.txt')]
+>>> pprint(sorted((tmp_path / 'save').iterdir()))
+ [...Path('...test_file-00.txt'),
+ ...Path('...test_file-01.txt'),
+ ...Path('...test_file-02.txt'),
+ ...Path('...test_file-03.txt')]
+alto2txt2fixture/utils.pyCreate a lookup dictionary from a list of dictionaries.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
lst |
+
+ list
+ |
+
+
+
+ A list of dictionaries that should be used to generate the lookup. + |
+
+ []
+ |
+
on |
+
+ list
+ |
+
+
+
+ A list of keys from the dictionaries in the list that should be used as the keys in the lookup. + |
+
+ []
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ dict
+ |
+
+
+
+ The generated lookup dictionary. + |
+
alto2txt2fixture/utils.pydict_from_list_fixture_fields(
+ fixture_list: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,
+ field_name: str = DATA_PROVIDER_INDEX,
+) -> dict[str, FixtureDict]
+Create a dict from fixture_list with attr_name as key.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
fixture_list |
+
+ Iterable[FixtureDict]
+ |
+
+
+
+
|
+
+ NEWSPAPER_COLLECTION_METADATA
+ |
+
field_name |
+
+ str
+ |
+
+
+
+ key for values within |
+
+ DATA_PROVIDER_INDEX
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ dict[str, FixtureDict]
+ |
+
+
+
+ A |
+
alto2txt2fixture/utils.pyexport_fixtures(
+ fixture_tables: dict[str, Sequence[FixtureDict]],
+ path: str | PathLike = settings.FIXTURE_TABLES_OUTPUT,
+ prefix: str = "test-",
+ add_created: bool = True,
+ formats: Sequence[EXPORT_FORMATS] = settings.FIXTURE_TABLES_FORMATS,
+ file_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,
+) -> None
+Export fixture_tables in formats.
This is still in experimental phase of development and not recommended +for production.
+Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
fixture_tables |
+
+ dict[str, Sequence[FixtureDict]]
+ |
+
+
+
+
|
+ + required + | +
path |
+
+ str | PathLike
+ |
+
+
+
+
|
+
+ settings.FIXTURE_TABLES_OUTPUT
+ |
+
prefix |
+
+ str
+ |
+
+
+
+
|
+
+ 'test-'
+ |
+
formats |
+
+ Sequence[EXPORT_FORMATS]
+ |
+
+
+
+
|
+
+ settings.FIXTURE_TABLES_FORMATS
+ |
+
file_name_0_padding |
+
+ int
+ |
+
+
+
+ Zeros to prefix the number of each fixture file name. + |
+
+ FILE_NAME_0_PADDING_DEFAULT
+ |
+
>>> test_fixture_tables: dict[str, FixtureDict] = {
+... 'test0': NEWSPAPER_COLLECTION_METADATA,
+... 'test1': NEWSPAPER_COLLECTION_METADATA}
+>>> export_fixtures(test_fixture_tables, path='tests/')
+<BLANKLINE>
+...Warning: Saving test0...
+...Warning: Saving test1...
+>>> from pandas import read_csv
+>>> fixture0_json = load_json('tests/test-test0-000001.json')
+>>> fixture0_df = read_csv('tests/test-test0-000001.csv')
+>>> fixture1_json = load_json('tests/test-test1-000001.json')
+>>> fixture1_df = read_csv('tests/test-test1-000001.csv')
+>>> fixture0_json == fixture1_json
+True
+>>> all(fixture0_df == fixture1_df)
+True
+>>> all(field in fixture0_json[0]['fields']
+... for field in ['created_at', 'updated_at'])
+True
+>>> fixture0_json[1]['pk']
+2
+>>> fixture0_json[1]['fields'][DATA_PROVIDER_INDEX]
+'hmd'
+>>> fixture0_df[['pk', DATA_PROVIDER_INDEX]].iloc[1].to_list()
+[2, 'hmd']
+alto2txt2fixture/utils.py943 + 944 + 945 + 946 + 947 + 948 + 949 + 950 + 951 + 952 + 953 + 954 + 955 + 956 + 957 + 958 + 959 + 960 + 961 + 962 + 963 + 964 + 965 + 966 + 967 + 968 + 969 + 970 + 971 + 972 + 973 + 974 + 975 + 976 + 977 + 978 + 979 + 980 + 981 + 982 + 983 + 984 + 985 + 986 + 987 + 988 + 989 + 990 + 991 + 992 + 993 + 994 + 995 + 996 + 997 + 998 + 999 +1000 +1001 +1002 +1003 +1004 +1005 +1006 +1007 +1008 +1009 +1010 +1011 +1012 +1013 | |
filter_json_fields(
+ json_results: list | dict | None = None,
+ file_path: PathLike | None = None,
+ fields: Sequence[str] = [],
+ value: Hashable = "",
+ **kwargs: Hashable
+) -> dict | list
+Return keys and values from json_dict where any fields equal value.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
file_path |
+
+ PathLike | None
+ |
+
+
+
+ The file |
+
+ None
+ |
+
fields |
+
+ Sequence[str]
+ |
+
+
+
+ Which fields to check equal |
+
+ []
+ |
+
value |
+
+ Hashable
+ |
+
+
+
+ Value to filter by + |
+
+ ''
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ dict | list
+ |
+
+
+
+ A |
+
Raises:
+| Type | +Description | +
|---|---|
+ ValueError
+ |
+
+
+
+
|
+
>>> entry_fixture: dict = [
+... {"pk": 4889, "model": "mitchells.entry",
+... "fields": {"title": "BIRMINGHAM POST .",
+... "price_raw": ['2d'],
+... "year": 1920,
+... "date_established_raw": "1857",
+... "persons": [], "newspaper": ""}},
+... {"pk": 9207, "model": "mitchells.entry",
+... "fields": {"title": "ULVERSTONE ADVERTISER .",
+... "price_raw": ['2 ½ d', '3 ½ d'],
+... "year": 1856,
+... "date_established_raw": "1848",
+... "persons": ['Stephen Soulby'],
+... "newspaper": "",}},
+... {"pk": 15, "model": "mitchells.entry",
+... "fields": {"title": "LLOYD'S WEEKLY LONDON NEWSPAPER .",
+... "price_raw": ['2d', '3d'],
+... "year": 1857,
+... "date_established_raw": "November , 1842",
+... "persons": ['Mr. Douglas Jerrold', 'Edward Lloyd'],
+... "newspaper": 1187}}
+... ]
+>>> pprint(filter_json_fields(entry_fixture,
+... fields=("newspaper", "persons"),
+... value=""))
+[{'fields': {'date_established_raw': '1857',
+ 'newspaper': '',
+ 'persons': [],
+ 'price_raw': ['2d'],
+ 'title': 'BIRMINGHAM POST .',
+ 'year': 1920},
+ 'model': 'mitchells.entry',
+ 'pk': 4889},
+ {'fields': {'date_established_raw': '1848',
+ 'newspaper': '',
+ 'persons': ['Stephen Soulby'],
+ 'price_raw': ['2 ½ d', '3 ½ d'],
+ 'title': 'ULVERSTONE ADVERTISER .',
+ 'year': 1856},
+ 'model': 'mitchells.entry',
+ 'pk': 9207}]
+alto2txt2fixture/utils.py490 +491 +492 +493 +494 +495 +496 +497 +498 +499 +500 +501 +502 +503 +504 +505 +506 +507 +508 +509 +510 +511 +512 +513 +514 +515 +516 +517 +518 +519 +520 +521 +522 +523 +524 +525 +526 +527 +528 +529 +530 +531 +532 +533 +534 +535 +536 +537 +538 +539 +540 +541 +542 +543 +544 +545 +546 +547 +548 +549 +550 +551 +552 +553 +554 +555 +556 +557 +558 +559 +560 +561 +562 +563 +564 +565 +566 +567 +568 +569 +570 +571 +572 +573 +574 +575 | |
fixture_fields(
+ fixture_dict: FixtureDict, include_pk: bool = True, as_dict: bool = False
+) -> tuple[str, ...] | dict[str, Any]
+Generate a tuple of FixtureDict field names.
This is not in the utils module to avoid a circular import.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
fixture_dict |
+
+ FixtureDict
+ |
+
+
+
+ A |
+ + required + | +
include_pk |
+
+ bool
+ |
+
+
+
+ Whether to include the |
+
+ True
+ |
+
>>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0])
+('pk', 'name', 'code', 'legacy_code', 'collection', 'source_note')
+>>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0], include_pk=False)
+('name', 'code', 'legacy_code', 'collection', 'source_note')
+>>> hmd_dict: dict[str, Any] = fixture_fields(
+... NEWSPAPER_COLLECTION_METADATA[1], as_dict=True)
+>>> hmd_dict['code']
+'bl_hmd'
+>>> hmd_dict['pk']
+2
+>>> hmd_dict = fixture_fields(
+... NEWSPAPER_COLLECTION_METADATA[1], include_pk=False, as_dict=True)
+>>> 'pk' in hmd_dict
+False
+alto2txt2fixture/utils.pyfixture_or_default_dict(
+ key: str,
+ fixture_dict: dict[str, FixtureDict],
+ default_dict: FixtureDict | dict = {},
+) -> FixtureDict | dict
+Return a FixtureDict from fixture_list via key index, else default_dict.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
key |
+
+ str
+ |
+
+
+
+ a |
+ + required + | +
fixture_dict |
+
+ dict[str, FixtureDict]
+ |
+
+
+
+ a |
+ + required + | +
default_dict |
+
+ FixtureDict | dict
+ |
+
+
+
+ a |
+
+ {}
+ |
+
>>> newspaper_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields(
+... NEWSPAPER_COLLECTION_METADATA)
+>>> hmd_dict: FixtureDict = fixture_or_default_dict(
+... 'hmd', newspaper_dict
+... )
+>>> hmd_dict == newspaper_dict['hmd']
+True
+>>> fixture_or_default_dict(
+... 'hmd', NEWSPAPER_COLLECTION_METADATA
+... )
+{}
+>>> fixture_or_default_dict(
+... 'hmd', NEWSPAPER_COLLECTION_METADATA, {'a': 'default'}
+... )
+{'a': 'default'}
+alto2txt2fixture/utils.pyfixtures_dict2csv(
+ fixtures: Iterable[FixtureDict] | Generator[FixtureDict, None, None],
+ prefix: str = "",
+ output_path: PathLike | str = settings.OUTPUT,
+ index: bool = False,
+ max_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,
+ file_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,
+) -> None
+Saves fixtures generated by a generator to separate separate CSV files.
This function takes an Iterable or Generator of fixtures and saves to
+separate CSV files. The fixtures are saved in batches, where each batch
+is determined by the max_elements_per_file parameter.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
fixtures |
+
+ Iterable[FixtureDict] | Generator[FixtureDict, None, None]
+ |
+
+
+
+ An |
+ + required + | +
prefix |
+
+ str
+ |
+
+
+
+ A string prefix to be added to the file names of the +saved fixtures. + |
+
+ ''
+ |
+
output_path |
+
+ PathLike | str
+ |
+
+
+
+ Path to folder fixtures are saved to + |
+
+ settings.OUTPUT
+ |
+
max_elements_per_file |
+
+ int
+ |
+
+
+
+ Maximum |
+
+ settings.MAX_ELEMENTS_PER_FILE
+ |
+
file_name_0_padding |
+
+ int
+ |
+
+
+
+ Zeros to prefix the number of each fixture file name. + |
+
+ FILE_NAME_0_PADDING_DEFAULT
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ This function saves fixtures to files and does not return a value. + |
+
>>> tmp_path: Path = getfixture('tmp_path')
+>>> from pandas import read_csv
+>>> fixtures_dict2csv(NEWSPAPER_COLLECTION_METADATA,
+... prefix='test', output_path=tmp_path)
+>>> imported_fixture = read_csv(tmp_path / 'test-000001.csv')
+>>> imported_fixture.iloc[1]['pk']
+2
+>>> imported_fixture.iloc[1][DATA_PROVIDER_INDEX]
+'hmd'
+alto2txt2fixture/utils.py870 +871 +872 +873 +874 +875 +876 +877 +878 +879 +880 +881 +882 +883 +884 +885 +886 +887 +888 +889 +890 +891 +892 +893 +894 +895 +896 +897 +898 +899 +900 +901 +902 +903 +904 +905 +906 +907 +908 +909 +910 +911 +912 +913 +914 +915 +916 +917 +918 +919 +920 +921 +922 +923 +924 +925 +926 +927 +928 +929 +930 +931 +932 +933 +934 +935 +936 +937 +938 +939 +940 | |
free_hd_space_in_GB(
+ disk_usage_tuple: DiskUsageTuple | None = None, path: PathLike | None = None
+) -> float
+Return remaing hard drive space estimate in gigabytes.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
disk_usage_tuple |
+
+ DiskUsageTuple | None
+ |
+
+
+
+ A |
+
+ None
+ |
+
path |
+
+ PathLike | None
+ |
+
+
+
+ A |
+
+ None
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ float
+ |
+
+
+
+ A |
+
alto2txt2fixture/utils.pygen_fixture_tables(
+ fixture_tables: dict[str, list[FixtureDict]] = {},
+ include_fixture_pk_column: bool = True,
+) -> Generator[Table, None, None]
+Generator of rich.Table instances from FixtureDict configuration tables.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
fixture_tables |
+
+ dict[str, list[FixtureDict]]
+ |
+
+
+
+
|
+
+ {}
+ |
+
include_fixture_pk_column |
+
+ bool
+ |
+
+
+
+ whether to include the |
+
+ True
+ |
+
>>> table_name: str = "data_provider"
+>>> tables = tuple(
+... gen_fixture_tables(
+... {table_name: NEWSPAPER_COLLECTION_METADATA}
+... ))
+>>> len(tables)
+1
+>>> assert tables[0].title == table_name
+>>> [column.header for column in tables[0].columns]
+['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']
+alto2txt2fixture/utils.pyThis function takes in a Path object path and returns a list of lists
+of zipfiles sorted and chunked according to certain conditions defined
+in the settings object (see settings.CHUNK_THRESHOLD).
Note: the function will also skip zip files of a certain file size, which
+can be specified in the settings object (see settings.SKIP_FILE_SIZE).
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
path |
+
+ Path
+ |
+
+
+
+ The input path where the zipfiles are located + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ list
+ |
+
+
+
+ A list of lists of |
+
alto2txt2fixture/utils.pyGet a string key from a dictionary using values from specified keys.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
x |
+
+ dict
+ |
+
+
+
+ A dictionary from which the key is generated. + |
+
+ dict()
+ |
+
on |
+
+ list
+ |
+
+
+
+ A list of keys from the dictionary that should be used to +generate the key. + |
+
+ []
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+ The generated string key. + |
+
alto2txt2fixture/utils.pyProvides the path to any given lockfile, which controls whether any +existing files should be overwritten or not.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
collection |
+
+ str
+ |
+
+
+
+ Collection folder name + |
+ + required + | +
kind |
+
+ NewspaperElements
+ |
+
+
+
+ Either |
+ + required + | +
dic |
+
+ dict
+ |
+
+
+
+ A dictionary with required information for either |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ Path
+ |
+
+
+
+ Path to the resulting lockfile + |
+
alto2txt2fixture/utils.pyReturn datetime.now() as either a string or datetime object.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
as_str |
+
+ bool
+ |
+
+
+
+ Whether to return |
+
+ False
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ datetime.datetime | str
+ |
+
+
+
+
|
+
alto2txt2fixture/utils.pyConverts an input value into a Path object if it's not already one.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str | Path
+ |
+
+
+
+ The input value, which can be a string or a Path object. + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ Path
+ |
+
+
+
+ The input value as a Path object. + |
+
alto2txt2fixture/utils.pyReturns a nice string for any given file size.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str | Path
+ |
+
+
+
+ Path to read the size from + |
+ + required + | +
raw |
+
+ bool
+ |
+
+
+
+ Whether to return the file size as total number of bytes or +a human-readable MB/GB amount + |
+
+ False
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ str | float
+ |
+
+
+
+ Return |
+
alto2txt2fixture/utils.pyReturn ordered glob, filtered out any pesky, unwanted .DS_Store from macOS.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str
+ |
+
+
+
+ Path to a directory to filter + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ list
+ |
+
+
+
+ Sorted list of files contained in the provided path without the ones + |
+
+ list
+ |
+
+
+
+ whose names start with a |
+
alto2txt2fixture/utils.pyglob_path_rename_by_0_padding(
+ path: PathLike,
+ output_path: PathLike | None = None,
+ glob_regex_str: str = "*",
+ padding: int | None = 0,
+ match_int_regex: str = PADDING_0_REGEX_DEFAULT,
+ index: int = -1,
+) -> dict[PathLike, PathLike]
+Return an OrderedDict of replacement 0-padded file names from path.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
path |
+
+ PathLike
+ |
+
+
+
+
|
+ + required + | +
output_path |
+
+ PathLike | None
+ |
+
+
+
+
|
+
+ None
+ |
+
glob_regex_str |
+
+ str
+ |
+
+
+
+
|
+
+ '*'
+ |
+
padding |
+
+ int | None
+ |
+
+
+
+ How many digits (0s) to pad |
+
+ 0
+ |
+
match_int_regex |
+
+ str
+ |
+
+
+
+ Regular expression for matching numbers in |
+
+ PADDING_0_REGEX_DEFAULT
+ |
+
index |
+
+ int
+ |
+
+
+
+ Which index of number in |
+
+ -1
+ |
+
>>> tmp_path: Path = getfixture('tmp_path')
+>>> for i in range(4):
+... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)
+>>> pprint(sorted(tmp_path.iterdir()))
+[...Path('...test_file-0.txt'),
+ ...Path('...test_file-1.txt'),
+ ...Path('...test_file-2.txt'),
+ ...Path('...test_file-3.txt')]
+>>> pprint(glob_path_rename_by_0_padding(tmp_path))
+{...Path('...test_file-0.txt'): ...Path('...test_file-00.txt'),
+ ...Path('...test_file-1.txt'): ...Path('...test_file-01.txt'),
+ ...Path('...test_file-2.txt'): ...Path('...test_file-02.txt'),
+ ...Path('...test_file-3.txt'): ...Path('...test_file-03.txt')}
+alto2txt2fixture/utils.py1491 +1492 +1493 +1494 +1495 +1496 +1497 +1498 +1499 +1500 +1501 +1502 +1503 +1504 +1505 +1506 +1507 +1508 +1509 +1510 +1511 +1512 +1513 +1514 +1515 +1516 +1517 +1518 +1519 +1520 +1521 +1522 +1523 +1524 +1525 +1526 +1527 +1528 +1529 +1530 +1531 +1532 +1533 +1534 +1535 +1536 +1537 +1538 +1539 +1540 +1541 +1542 +1543 +1544 +1545 +1546 +1547 +1548 +1549 +1550 +1551 +1552 +1553 +1554 +1555 +1556 +1557 +1558 +1559 +1560 +1561 +1562 +1563 +1564 +1565 +1566 +1567 +1568 +1569 +1570 +1571 +1572 +1573 +1574 +1575 +1576 +1577 | |
int_from_str(
+ s: str, index: int = -1, regex: str = PADDING_0_REGEX_DEFAULT
+) -> tuple[str, int]
+Return matched (or None) regex from s by index index.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
s |
+
+ str
+ |
+
+
+
+
|
+ + required + | +
index |
+
+ int
+ |
+
+
+
+ Which index of number in |
+
+ -1
+ |
+
regex |
+
+ str
+ |
+
+
+
+ Regular expression for matching numbers in |
+
+ PADDING_0_REGEX_DEFAULT
+ |
+
alto2txt2fixture/utils.pylist_json_files(
+ p: str | Path,
+ drill: bool = False,
+ exclude_names: list = [],
+ include_names: list = [],
+) -> Generator[Path, None, None] | list[Path]
+List json files under the path specified in p.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str | Path
+ |
+
+
+
+ The path to search for |
+ + required + | +
drill |
+
+ bool
+ |
+
+
+
+ A flag indicating whether to drill down the subdirectories
+or not. Default is |
+
+ False
+ |
+
exclude_names |
+
+ list
+ |
+
+
+
+ A list of file names to exclude from the search +result. Default is an empty list + |
+
+ []
+ |
+
include_names |
+
+ list
+ |
+
+
+
+ A list of file names to include in search result.
+If provided, the |
+
+ []
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ Generator[Path, None, None] | list[Path]
+ |
+
+
+
+ A list of |
+
alto2txt2fixture/utils.pyEasier access to reading json files.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str | Path
+ |
+
+
+
+ Path to read |
+ + required + | +
crash |
+
+ bool
+ |
+
+
+
+ Whether the program should crash if there is a |
+
+ False
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ dict | list
+ |
+
+
+
+ The decoded |
+
+ dict | list
+ |
+
+
+
+ if the file cannot be decoded and |
+
alto2txt2fixture/utils.pyload_multiple_json(
+ p: str | Path,
+ drill: bool = False,
+ filter_na: bool = True,
+ crash: bool = False,
+) -> list
+Load multiple json files and return a list of their content.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str | Path
+ |
+
+
+
+ The path to search for |
+ + required + | +
drill |
+
+ bool
+ |
+
+
+
+ A flag indicating whether to drill down the subdirectories
+or not. Default is |
+
+ False
+ |
+
filter_na |
+
+ bool
+ |
+
+
+
+ A flag indicating whether to filter out the content that
+is |
+
+ True
+ |
+
crash |
+
+ bool
+ |
+
+
+
+ A flag indicating whether to raise an exception when an
+error occurs while loading a |
+
+ False
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ list
+ |
+
+
+
+ A |
+
alto2txt2fixture/utils.pyWrites a '.' to a lockfile, after making sure the parent directory exists.
+ + + +Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
lockfile |
+
+ Path
+ |
+
+
+
+ The path to the lock file to be created + |
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ None + |
+
alto2txt2fixture/utils.pyReturn a sorted tuple of Paths in path using glob_regex_str.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
path |
+
+ PathLike
+ |
+
+
+
+ Patch to search via |
+ + required + | +
glob_regex_str |
+
+ str
+ |
+
+
+
+ Regular expression to use with |
+
+ '*'
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ tuple[PathLike, ...]
+ |
+
+
+
+
|
+
>>> bl_lwm = getfixture("bl_lwm")
+>>> pprint(path_globs_to_tuple(bl_lwm, '*text.zip'))
+(...Path('...bl_lwm...0003079-test_plaintext.zip'),
+ ...Path('...bl_lwm...0003548-test_plaintext.zip'))
+>>> pprint(path_globs_to_tuple(bl_lwm, '*.txt'))
+(...Path('...bl_lwm...0003079_18980121_sect0001.txt'),
+ ...Path('...bl_lwm...0003548_19040707_art0037.txt'))
+alto2txt2fixture/utils.pypaths_with_newlines(
+ paths: Iterable[PathLike], truncate: bool = False, **kwargs: bool
+) -> str
+Return a str of paths separated by a \n.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')
+>>> print(paths_with_newlines(plaintext_bl_lwm.compressed_files))
+'...bl_lwm...0003079-test_plaintext.zip'
+'...bl_lwm...0003548-test_plaintext.zip'
+>>> print(
+... paths_with_newlines(plaintext_bl_lwm.compressed_files,
+... truncate=True)
+... )
+<BLANKLINE>
+...
+'...0003079-test_plaintext.zip'
+'...0003548-test_plaintext.zip'
+alto2txt2fixture/utils.pyrename_by_0_padding(
+ file_path: PathLike,
+ match_str: str | None = None,
+ match_int: int | None = None,
+ padding: int = FILE_NAME_0_PADDING_DEFAULT,
+ replace_count: int = 1,
+ exclude_parents: bool = True,
+ reverse_int_match: bool = False,
+) -> Path
+Return file_path with 0 padding Path change.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
file_path |
+
+ PathLike
+ |
+
+
+
+
|
+ + required + | +
match_str |
+
+ str | None
+ |
+
+
+
+
|
+
+ None
+ |
+
match_int |
+
+ int | None
+ |
+
+
+
+
|
+
+ None
+ |
+
padding |
+
+ int
+ |
+
+
+
+ How many digits (0s) to pad |
+
+ FILE_NAME_0_PADDING_DEFAULT
+ |
+
exclude_parents |
+
+ bool
+ |
+
+
+
+ Only rename parts of |
+
+ True
+ |
+
reverse_int_match |
+
+ bool
+ |
+
+
+
+ Whether to match from the end of the |
+
+ False
+ |
+
>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',
+... match_str='05', match_int=5)
+<BLANKLINE>
+...Path('a/path/to/3/fixture-03-000005.txt')...
+>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',
+... match_str='03')
+<BLANKLINE>
+...Path('a/path/to/3/fixture-000003-05.txt')...
+>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',
+... match_str='05', padding=0)
+<BLANKLINE>
+...Path('a/path/to/3/fixture-03-5.txt')...
+>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',
+... match_int=3)
+<BLANKLINE>
+...Path('a/path/to/3/fixture-0000003-05.txt')...
+>>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',
+... match_int=3, padding=2,
+... exclude_parents=False)
+<BLANKLINE>
+...Path('a/path/to/03/f-03-05-0003.txt')...
+>>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',
+... match_int=3, padding=2,
+... exclude_parents=False,
+... replace_count=3, )
+<BLANKLINE>
+...Path('a/path/to/03/f-003-05-00003.txt')...
+alto2txt2fixture/utils.py1404 +1405 +1406 +1407 +1408 +1409 +1410 +1411 +1412 +1413 +1414 +1415 +1416 +1417 +1418 +1419 +1420 +1421 +1422 +1423 +1424 +1425 +1426 +1427 +1428 +1429 +1430 +1431 +1432 +1433 +1434 +1435 +1436 +1437 +1438 +1439 +1440 +1441 +1442 +1443 +1444 +1445 +1446 +1447 +1448 +1449 +1450 +1451 +1452 +1453 +1454 +1455 +1456 +1457 +1458 +1459 +1460 +1461 +1462 +1463 +1464 +1465 +1466 +1467 +1468 +1469 +1470 +1471 +1472 +1473 +1474 +1475 +1476 +1477 +1478 +1479 +1480 +1481 +1482 +1483 +1484 +1485 +1486 +1487 +1488 | |
save_fixture(
+ generator: Sequence | Generator = [],
+ prefix: str = "",
+ output_path: PathLike | str = settings.OUTPUT,
+ max_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,
+ add_created: bool = True,
+ json_indent: int = JSON_INDENT,
+ file_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,
+) -> None
+Saves fixtures generated by a generator to separate JSON files.
+This function takes a generator and saves the generated fixtures to
+separate JSON files. The fixtures are saved in batches, where each batch
+is determined by the max_elements_per_file parameter.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
generator |
+
+ Sequence | Generator
+ |
+
+
+
+ A generator that yields the fixtures to be saved. + |
+
+ []
+ |
+
prefix |
+
+ str
+ |
+
+
+
+ A string prefix to be added to the file names of the +saved fixtures. + |
+
+ ''
+ |
+
output_path |
+
+ PathLike | str
+ |
+
+
+
+ Path to folder fixtures are saved to + |
+
+ settings.OUTPUT
+ |
+
max_elements_per_file |
+
+ int
+ |
+
+
+
+ Maximum |
+
+ settings.MAX_ELEMENTS_PER_FILE
+ |
+
add_created |
+
+ bool
+ |
+
+
+
+ Whether to add |
+
+ True
+ |
+
json_indent |
+
+ int
+ |
+
+
+
+ Number of indent spaces per line in saved |
+
+ JSON_INDENT
+ |
+
file_name_0_padding |
+
+ int
+ |
+
+
+
+ Zeros to prefix the number of each fixture file name. + |
+
+ FILE_NAME_0_PADDING_DEFAULT
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ This function saves the fixtures to files but does not return + |
+
+ None
+ |
+
+
+
+ any value. + |
+
>>> tmp_path: Path = getfixture('tmp_path')
+>>> save_fixture(NEWSPAPER_COLLECTION_METADATA,
+... prefix='test', output_path=tmp_path)
+>>> imported_fixture = load_json(tmp_path / 'test-000001.json')
+>>> imported_fixture[1]['pk']
+2
+>>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]
+'hmd'
+>>> 'created_at' in imported_fixture[1]['fields']
+True
+alto2txt2fixture/utils.py781 +782 +783 +784 +785 +786 +787 +788 +789 +790 +791 +792 +793 +794 +795 +796 +797 +798 +799 +800 +801 +802 +803 +804 +805 +806 +807 +808 +809 +810 +811 +812 +813 +814 +815 +816 +817 +818 +819 +820 +821 +822 +823 +824 +825 +826 +827 +828 +829 +830 +831 +832 +833 +834 +835 +836 +837 +838 +839 +840 +841 +842 +843 +844 +845 +846 +847 +848 +849 +850 +851 +852 +853 +854 +855 +856 +857 +858 +859 +860 +861 +862 +863 +864 +865 +866 +867 | |
truncate_path_str(
+ path: PathLike,
+ max_length: int = MAX_TRUNCATE_PATH_STR_LEN,
+ folder_filler_str: str = INTERMEDIATE_PATH_TRUNCATION_STR,
+ head_parts: int = TRUNC_HEADS_PATH_DEFAULT,
+ tail_parts: int = TRUNC_TAILS_PATH_DEFAULT,
+ path_sep: str = sep,
+ _force_type: Type[Path] | Type[PureWindowsPath] = Path,
+) -> str
+If len(text) > max_length return text followed by trail_str.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
path |
+
+ PathLike
+ |
+
+
+
+
|
+ + required + | +
max_length |
+
+ int
+ |
+
+
+
+ maximum length of |
+
+ MAX_TRUNCATE_PATH_STR_LEN
+ |
+
folder_filler_str |
+
+ str
+ |
+
+
+
+ what to fill intermediate path names with + |
+
+ INTERMEDIATE_PATH_TRUNCATION_STR
+ |
+
head_parts |
+
+ int
+ |
+
+
+
+ how many parts of |
+
+ TRUNC_HEADS_PATH_DEFAULT
+ |
+
tail_parts |
+
+ int
+ |
+
+
+
+ how many parts from the |
+
+ TRUNC_TAILS_PATH_DEFAULT
+ |
+
path_sep |
+
+ str
+ |
+
+
+
+ what |
+
+ sep
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ str
+ |
+
+
+
+
|
+
For errors running on windows see: +#56
+>>> love_shadows: Path = (
+... Path('Standing') / 'in' / 'the' / 'shadows'/ 'of' / 'love.')
+>>> truncate_path_str(love_shadows)
+'Standing...love.'
+>>> truncate_path_str(love_shadows, max_length=100)
+'Standing...in...the...shadows...of...love.'
+>>> truncate_path_str(love_shadows, folder_filler_str="*")
+'Standing...*...*...*...*...love.'
+>>> root_love_shadows: Path = Path(sep) / love_shadows
+>>> truncate_path_str(root_love_shadows, folder_filler_str="*")
+<BLANKLINE>
+...
+'...Standing...*...*...*...*...love.'
+>>> if is_platform_win:
+... pytest.skip('fails on certain Windows root paths: issue #56')
+>>> truncate_path_str(root_love_shadows,
+... folder_filler_str="*", tail_parts=3)
+<BLANKLINE>
+...
+'...Standing...*...*...shadows...of...love.'...
+alto2txt2fixture/utils.py1263 +1264 +1265 +1266 +1267 +1268 +1269 +1270 +1271 +1272 +1273 +1274 +1275 +1276 +1277 +1278 +1279 +1280 +1281 +1282 +1283 +1284 +1285 +1286 +1287 +1288 +1289 +1290 +1291 +1292 +1293 +1294 +1295 +1296 +1297 +1298 +1299 +1300 +1301 +1302 +1303 +1304 +1305 +1306 +1307 +1308 +1309 +1310 +1311 +1312 +1313 +1314 +1315 +1316 +1317 +1318 +1319 +1320 +1321 +1322 +1323 +1324 +1325 +1326 +1327 +1328 +1329 +1330 +1331 +1332 +1333 +1334 +1335 +1336 +1337 +1338 +1339 +1340 +1341 +1342 +1343 +1344 +1345 +1346 +1347 +1348 +1349 +1350 +1351 +1352 +1353 +1354 +1355 +1356 +1357 +1358 +1359 +1360 +1361 +1362 +1363 +1364 +1365 +1366 +1367 +1368 | |
Return a tuple of valid compression paths in files.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
files |
+
+ Sequence[PathLike]
+ |
+
+
+
+
|
+ + required + | +
Returns:
+| Type | +Description | +
|---|---|
+ list[PathLike]
+ |
+
+
+
+ A list of files that could be decompressed. + |
+
alto2txt2fixture/utils.pywrite_json(
+ p: str | Path,
+ o: dict,
+ add_created: bool = True,
+ json_indent: int = JSON_INDENT,
+) -> None
+Easier access to writing json files. Checks whether parent exists.
Parameters:
+| Name | +Type | +Description | +Default | +
|---|---|---|---|
p |
+
+ str | Path
+ |
+
+
+
+ Path to write |
+ + required + | +
o |
+
+ dict
+ |
+
+
+
+ Object to write to |
+ + required + | +
add_created |
+
+ bool
+ |
+
+
+
+ If set to True will add |
+
+ True
+ |
+
json_indent |
+
+ int
+ |
+
+
+
+ What indetation format to write out |
+
+ JSON_INDENT
+ |
+
Returns:
+| Type | +Description | +
|---|---|
+ None
+ |
+
+
+
+ None + |
+
>>> tmp_path: Path = getfixture('tmp_path')
+>>> path: Path = tmp_path / 'test-write-json-example.json'
+>>> write_json(p=path,
+... o=NEWSPAPER_COLLECTION_METADATA,
+... add_created=True)
+>>> imported_fixture = load_json(path)
+>>> imported_fixture[1]['pk']
+2
+>>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]
+'hmd'
+alto2txt2fixture/utils.py332 +333 +334 +335 +336 +337 +338 +339 +340 +341 +342 +343 +344 +345 +346 +347 +348 +349 +350 +351 +352 +353 +354 +355 +356 +357 +358 +359 +360 +361 +362 +363 +364 +365 +366 +367 +368 +369 +370 +371 +372 +373 +374 +375 +376 +377 +378 +379 +380 +381 +382 +383 +384 +385 +386 +387 +388 +389 +390 +391 +392 +393 +394 +395 +396 +397 +398 +399 | |
poetry to runThe program should run automatically with the following command:
+ +Alternatively, if you want to add optional parameters and don’t want to use the standard poetry script to run, you can use the (somewhat convoluted) poetry run alto2txt2fixture/run.py and provide any optional parameters. You can see a list of all the “Optional parameters” below. For example, if you want to only include the hmd collection:
If you find yourself in trouble with poetry, the program should run perfectly fine on its own, assuming the dependencies are installed. The same command, then, would be:
Note
+See the list under [tool.poetry.dependencies] in pyproject.toml for a list of dependencies that would need to be installed for alto2txt2fixture to work outside a python poetry environment.
The program has a number of optional parameters that you can choose to include or not. The table below describes each parameter, how to pass it to the program, and what its defaults are.
+| Flag | +Description | +Default value | +
|---|---|---|
-c, --collections |
+Which collections to process in the mounted alto2txt directory | +hmd, lwm, jisc, bna |
+
-o, --output |
+Into which directory should the processed files be put? | +./output/fixtures/ |
+
-m, --mountpoint |
+Where is the alto2txt directories mounted? | +./input/alto2txt/ |
+
-t, --test-config |
+Print the config table but do not run | +False |
+
alto2txt2fixture is a standalone tool to convert alto2txt XML output and other related datasets into JSON (and where feasible CSV) data with corresponding relational IDs to ease general use and ingestion into a relational database.
We target the the JSON produced for importing into lwmdb: a database built using the Django python webframework database fixture structure.
We provide a command line interface to process alto2txt XML files stored locally (or mounted via azure blobfuse), and for additional public data we automate a means of downloading those automatically.
We recommend downloading a copy of the reposity or using git clone. From a local copy use poetry to install dependencies:
$ cd alto2txt2fixture\n$ poetry install\nIf you would like to test, render documentation and/or contribute to the code included dev dependencies in a local install:
$ poetry install --with dev\nTo processing newspaper metadata with a local copy of alto2txt XML results, it's easiest to have that data in the same folder as your alto2txt2fixture checkout and poetry installed folder. One arranged, you should be able to begin the JSON converstion with
$ poetry run a2t2f-news\nTo generate related data in JSON and CSV form, assuming you have an internet collection and access to a living-with-machines azure account, the following will download related data into JSON and CSV files. The JSON results should be consistent with lwmdb tables for ease of import.
$ poetry run a2t2f-adj\npoetry to run","text":"The program should run automatically with the following command:
$ poetry run a2t2f-news\nAlternatively, if you want to add optional parameters and don\u2019t want to use the standard poetry script to run, you can use the (somewhat convoluted) poetry run alto2txt2fixture/run.py and provide any optional parameters. You can see a list of all the \u201cOptional parameters\u201d below. For example, if you want to only include the hmd collection:
$ poetry run alto2txt2fixture/run.py --collections hmd\nIf you find yourself in trouble with poetry, the program should run perfectly fine on its own, assuming the dependencies are installed. The same command, then, would be:
$ python alto2txt2fixture/run.py --collections hmd\nNote
See the list under [tool.poetry.dependencies] in pyproject.toml for a list of dependencies that would need to be installed for alto2txt2fixture to work outside a python poetry environment.
The program has a number of optional parameters that you can choose to include or not. The table below describes each parameter, how to pass it to the program, and what its defaults are.
Flag Description Default value-c, --collections Which collections to process in the mounted alto2txt directory hmd, lwm, jisc, bna -o, --output Into which directory should the processed files be put? ./output/fixtures/ -m, --mountpoint Where is the alto2txt directories mounted? ./input/alto2txt/ -t, --test-config Print the config table but do not run False"},{"location":"running.html#successfully-running-the-program-an-example","title":"Successfully running the program: An example","text":""},{"location":"understanding-results.html","title":"Understanding the Results","text":""},{"location":"understanding-results.html#the-resulting-file-structure","title":"The resulting file structure","text":"The examples below follow standard settings
If you choose other settings for when you run the program, your output directory may look different from the information on this page.
"},{"location":"understanding-results.html#reports","title":"Reports","text":"Reports are automatically generated with a unique hash as the overarching folder structure. Inside the reports directory, you\u2019ll find a JSON file for each alto2txt directory (organised by NLP identifier).
The report structure, thus, looks like this:
The JSON file has some good troubleshooting information. You\u2019ll find that the contents are structured as a Python dictionary (or JavaScript Object). Here is an example:
Here is an explanation of each of the keys in the dictionary:
Key Explanation Data typepath The input path for the zip file that is being converted. string bytes The size of the input zip file represented in bytes. integer size The size of the input zip file represented in a human-readable string. string contents #TODO #3 integer start Date and time when processing started (see also end below). datestring newspaper_paths #TODO #3 list (string) publication_codes A list of the NLPs that are contained in the input zip file. list (string) issue_paths A list of all the issue paths that are contained in the cache directory. list (string) item_paths A list of all the item paths that are contained in the cache directory. list (string) end Date and time when processing ended (see also start above). datestring seconds Seconds that the script spent interpreting the zip file (should be added to the microseconds below). integer microseconds Microseconds that the script spent interpreting the zip file (should be added to the seconds above). integer"},{"location":"understanding-results.html#fixtures","title":"Fixtures","text":"The most important output of the script is contained in the fixtures directory. This directory contains JSON files for all the different columns in the corresponding Django metadata database (i.e. DataProvider, Digitisation, Ingest, Issue, Newspaper, and Item). The numbering at the end of each file indicates the order of the files as they are divided into a maximum of 2e6 elements*:
Each JSON file contains a Python-like list (JavaScript Array) of dictionaries (JavaScript Objects), which have a primary key (pk), the related database model (in the example below the Django newspapers app\u2019s newspaper table), and a nested dictionary/Object which contains all the values for the database\u2019s table entry:
* The maximum elements per file can be adjusted in the settings.py file\u2019s settings object\u2019s MAX_ELEMENTS_PER_FILE value.
Entry point for alto2txt2fixture.parse to convert alto2txt XML -> JSON.
This module defines the run function which is the main driver for the entire process.
It imports various functions from other modules and uses them to route and parse XML data generated by alto2txt.
The following steps are performed in the run function:
alto2txt data into subdirectories with structured files.JSON files.If the script is run as a main program (i.e. if the name of the script is __main__), the run() function is executed.
Note: at present this does not include any functunality in create_adjacent_tables.py
parse_args(argv: list[str] | None = None) -> Namespace\nManage command line arguments for run()
This constructs an ArgumentParser instance to manage configurating calls of run() to manage newspaper XML to JSON converstion.
Parameters:
Name Type Description Defaultargv list[str] | None If None treat as equivalent of ['--help], if alistofstrpass those options toArgumentParser`
None Returns:
Type DescriptionNamespace A Namespace dict-like configuration for run()
alto2txt2fixture/__main__.py def parse_args(argv: list[str] | None = None) -> Namespace:\n\"\"\"Manage command line arguments for `run()`\n This constructs an `ArgumentParser` instance to manage\n configurating calls of `run()` to manage `newspaper`\n `XML` to `JSON` converstion.\n Arguments:\n argv:\n If `None` treat as equivalent of ['--help`],\n if a `list` of `str` pass those options to `ArgumentParser`\n Returns:\n A `Namespace` `dict`-like configuration for `run()`\n \"\"\"\nargv = None if not argv else argv\nparser = ArgumentParser(\nprog=\"a2t2f-news\",\ndescription=\"Process alto2txt XML into and Django JSON Fixture files\",\nepilog=(\n\"Note: this is still in beta mode and contributions welcome\\n\\n\" + __doc__\n),\nformatter_class=RawTextHelpFormatter,\n)\nparser.add_argument(\n\"-c\",\n\"--collections\",\nnargs=\"+\",\nhelp=\"<Optional> Set collections\",\nrequired=False,\n)\nparser.add_argument(\n\"-m\",\n\"--mountpoint\",\ntype=str,\nhelp=\"<Optional> Mountpoint\",\nrequired=False,\n)\nparser.add_argument(\n\"-o\",\n\"--output\",\ntype=str,\nhelp=\"<Optional> Set an output directory\",\nrequired=False,\n)\nparser.add_argument(\n\"-t\",\n\"--test-config\",\ndefault=False,\nhelp=\"Only print the configuration\",\naction=BooleanOptionalAction,\n)\nparser.add_argument(\n\"-f\",\n\"--show-fixture-tables\",\ndefault=True,\nhelp=\"Print included fixture table configurations\",\naction=BooleanOptionalAction,\n)\nparser.add_argument(\n\"--export-fixture-tables\",\ndefault=True,\nhelp=\"Experimental: export fixture tables prior to data processing\",\naction=BooleanOptionalAction,\n)\nparser.add_argument(\n\"--data-provider-field\",\ntype=str,\ndefault=DATA_PROVIDER_INDEX,\nhelp=\"Key for indexing DataProvider records\",\n)\nreturn parser.parse_args(argv)\nrun(local_args: list[str] | None = None) -> None\nManage running newspaper XML to JSON conversion.
First parse_args is called for command line arguments including:
collectionsoutputmountpointIf any of these arguments are specified, they will be used, otherwise they will default to the values in the settings module.
The show_setup function is then called to display the configurations being used.
The route function is then called to route the alto2txt files into subdirectories with structured files.
The parse function is then called to parse the resulting JSON files.
Finally, the clear_cache function is called to clear the cache (pending the user's confirmation).
Parameters:
Name Type Description Defaultlocal_args list[str] | None Options passed to parse_args()
None Returns:
Type DescriptionNone None
Source code inalto2txt2fixture/__main__.py def run(local_args: list[str] | None = None) -> None:\n\"\"\"Manage running newspaper `XML` to `JSON` conversion.\n First `parse_args` is called for command line arguments including:\n - `collections`\n - `output`\n - `mountpoint`\n If any of these arguments are specified, they will be used, otherwise they\n will default to the values in the `settings` module.\n The `show_setup` function is then called to display the configurations\n being used.\n The `route` function is then called to route the alto2txt files into\n subdirectories with structured files.\n The `parse` function is then called to parse the resulting JSON files.\n Finally, the `clear_cache` function is called to clear the cache\n (pending the user's confirmation).\n Arguments:\n local_args: Options passed to `parse_args()`\n Returns:\n None\n \"\"\"\nargs: Namespace = parse_args(argv=local_args)\nif args.collections:\nCOLLECTIONS = [x.lower() for x in args.collections]\nelse:\nCOLLECTIONS = settings.COLLECTIONS\nif args.output:\nOUTPUT = args.output.rstrip(\"/\")\nelse:\nOUTPUT = settings.OUTPUT\nif args.mountpoint:\nMOUNTPOINT = args.mountpoint.rstrip(\"/\")\nelse:\nMOUNTPOINT = settings.MOUNTPOINT\nshow_setup(\nCOLLECTIONS=COLLECTIONS,\nOUTPUT=OUTPUT,\nCACHE_HOME=settings.CACHE_HOME,\nMOUNTPOINT=MOUNTPOINT,\nJISC_PAPERS_CSV=settings.JISC_PAPERS_CSV,\nREPORT_DIR=settings.REPORT_DIR,\nMAX_ELEMENTS_PER_FILE=settings.MAX_ELEMENTS_PER_FILE,\n)\nif args.show_fixture_tables:\n# Show a table of fixtures used, defaults to DataProvider Table\nshow_fixture_tables(settings, data_provider_index=args.data_provider_field)\nif args.export_fixture_tables:\nexport_fixtures(\nfixture_tables=settings.FIXTURE_TABLES,\npath=OUTPUT,\nformats=settings.FIXTURE_TABLES_FORMATS,\n)\nif not args.test_config:\n# Routing alto2txt into subdirectories with structured files\nroute(\nCOLLECTIONS,\nsettings.CACHE_HOME,\nMOUNTPOINT,\nsettings.JISC_PAPERS_CSV,\nsettings.REPORT_DIR,\n)\n# Parsing the resulting JSON files\nparse(\nCOLLECTIONS,\nsettings.CACHE_HOME,\nOUTPUT,\nsettings.MAX_ELEMENTS_PER_FILE,\n)\nclear_cache(settings.CACHE_HOME)\nfile_rename_table(\npaths_dict: dict[os.PathLike, os.PathLike],\ncompress_format: ArchiveFormatEnum = COMPRESSION_TYPE_DEFAULT,\ntitle: str = FILE_RENAME_TABLE_TITLE_DEFAULT,\nprefix: str = \"\",\nrenumber: bool = True,\n) -> Table\nCreate a rich.Table of rename configuration.
Parameters:
Name Type Description Defaultpaths_dict dict[os.PathLike, os.PathLike] dict[os.PathLike, os.PathLike], Original and renumbered paths dict
compress_format ArchiveFormatEnum Which ArchiveFormatEnum for compression
COMPRESSION_TYPE_DEFAULT title str Title of returned Table
FILE_RENAME_TABLE_TITLE_DEFAULT prefix str str to add in front of every new path
'' renumber bool Whether an int in each path will be renumbered.
True Source code in alto2txt2fixture/cli.py def file_rename_table(\npaths_dict: dict[os.PathLike, os.PathLike],\ncompress_format: ArchiveFormatEnum = COMPRESSION_TYPE_DEFAULT,\ntitle: str = FILE_RENAME_TABLE_TITLE_DEFAULT,\nprefix: str = \"\",\nrenumber: bool = True,\n) -> Table:\n\"\"\"Create a `rich.Table` of rename configuration.\n Args:\n paths_dict: dict[os.PathLike, os.PathLike],\n Original and renumbered `paths` `dict`\n compress_format:\n Which `ArchiveFormatEnum` for compression\n title:\n Title of returned `Table`\n prefix:\n `str` to add in front of every new path\n renumber:\n Whether an `int` in each path will be renumbered.\n \"\"\"\ntable: Table = Table(title=title)\ntable.add_column(\"Current File Name\", justify=\"right\", style=\"cyan\")\ntable.add_column(\"New File Name\", style=\"magenta\")\ndef final_file_name(name: os.PathLike) -> str:\nreturn (\nprefix\n+ str(Path(name).name)\n+ (f\".{compress_format}\" if compress_format else \"\")\n)\nfor old_path, new_path in paths_dict.items():\nname: str = final_file_name(new_path if renumber else old_path)\ntable.add_row(Path(old_path).name, name)\nreturn table\nfunc_table(\nfunc: Callable,\nvalues: dict,\ntitle: str = \"\",\nextra_dict: dict[str, Any] = {},\n) -> Table\nGeneate rich Table from func signature and help attr.
Parameters:
Name Type Description Defaultfunc Callable Function whose args and type hints will be converted to a table.
values dict dict of variables covered in func signature. local() often suffices.
title str str for table title.
'' extra_dict dict[str, Any] A dict of additional rows to add to the table. For each key, value pair: if the value is a tuple, it will be expanded to match the Type, Value, and Notes columns; else the Type will be inferred and Notes left blank.
{} Example >>> def test_func(\n... var_a: Annotated[str, typer.Option(help=\"Example\")] = \"Default\"\n... ) -> None:\n... test_func_table: Table = func_table(test_func, values=vars())\n... console.print(test_func_table)\n>>> if is_platform_win:\n... pytest.skip('fails on certain Windows root paths: issue #56')\n>>> test_func()\n test_func config\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Variable \u2503 Type \u2503 Value \u2503 Notes \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 var_a \u2502 str \u2502 Default \u2502 Example \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nalto2txt2fixture/cli.py def func_table(\nfunc: Callable, values: dict, title: str = \"\", extra_dict: dict[str, Any] = {}\n) -> Table:\n\"\"\"Geneate `rich` `Table` from `func` signature and `help` attr.\n Args:\n func:\n Function whose `args` and `type` hints will be converted\n to a table.\n values:\n `dict` of variables covered in `func` signature.\n `local()` often suffices.\n title:\n `str` for table title.\n extra_dict:\n A `dict` of additional rows to add to the table. For each\n `key`, `value` pair: if the `value` is a `tuple`, it will\n be expanded to match the `Type`, `Value`, and `Notes`\n columns; else the `Type` will be inferred and `Notes`\n left blank.\n Example:\n ```pycon\n >>> def test_func(\n ... var_a: Annotated[str, typer.Option(help=\"Example\")] = \"Default\"\n ... ) -> None:\n ... test_func_table: Table = func_table(test_func, values=vars())\n ... console.print(test_func_table)\n >>> if is_platform_win:\n ... pytest.skip('fails on certain Windows root paths: issue #56')\n >>> test_func()\n test_func config\n \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n \u2503 Variable \u2503 Type \u2503 Value \u2503 Notes \u2503\n \u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n \u2502 var_a \u2502 str \u2502 Default \u2502 Example \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n ```\n \"\"\"\ntitle = title if title else f\"{func.__name__} config\"\nfunc_signature: dict = get_type_hints(func, include_extras=True)\ntable: Table = Table(title=title)\ntable.add_column(\"Variable\", justify=\"right\", style=\"cyan\")\ntable.add_column(\"Type\", style=\"yellow\")\ntable.add_column(\"Value\", style=\"magenta\")\ntable.add_column(\"Notes\")\nfor var, info in func_signature.items():\ntry:\nvar_type, annotation = get_args(info)\nvalue: Any = values[var]\nif value in (\"\", \"\"):\nvalue = \"''\"\ntable.add_row(str(var), var_type.__name__, str(value), annotation.help)\nexcept ValueError:\ncontinue\nfor key, val in extra_dict.items():\nif isinstance(val, tuple):\ntable.add_row(key, *val)\nelse:\ntable.add_row(key, type(val).__name__, str(val))\nreturn table\nplaintext(\npath: Annotated[Path, typer.Argument(help=\"Path to raw plaintext files\")],\nsave_path: Annotated[\nPath, typer.Option(help=\"Path to save json export files\")\n] = Path(DEFAULT_PLAINTEXT_FIXTURE_OUTPUT),\ndata_provider_code: Annotated[\nstr, typer.Option(help=\"Data provider code use existing config\")\n] = \"\",\nextract_path: Annotated[\nPath, typer.Option(help=\"Folder to extract compressed raw plaintext to\")\n] = Path(DEFAULT_EXTRACTED_SUBDIR),\ninitial_pk: Annotated[\nint,\ntyper.Option(help=\"First primary key to increment json export from\"),\n] = DEFAULT_INITIAL_PK,\nrecords_per_json: Annotated[\nint, typer.Option(help=\"Max records per json fixture\")\n] = DEFAULT_MAX_PLAINTEXT_PER_FIXTURE_FILE,\ndigit_padding: Annotated[\nint,\ntyper.Option(help=\"Padding '0's for indexing json fixture filenames\"),\n] = FILE_NAME_0_PADDING_DEFAULT,\ncompress: Annotated[\nbool, typer.Option(help=\"Compress json fixtures\")\n] = False,\ncompress_path: Annotated[\nPath, typer.Option(help=\"Folder to compress json fixtueres to\")\n] = Path(COMPRESSED_PATH_DEFAULT),\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\n) -> None\nCreate a PlainTextFixture and save to save_path.
alto2txt2fixture/cli.py @cli.command()\ndef plaintext(\npath: Annotated[Path, typer.Argument(help=\"Path to raw plaintext files\")],\nsave_path: Annotated[\nPath, typer.Option(help=\"Path to save json export files\")\n] = Path(DEFAULT_PLAINTEXT_FIXTURE_OUTPUT),\ndata_provider_code: Annotated[\nstr, typer.Option(help=\"Data provider code use existing config\")\n] = \"\",\nextract_path: Annotated[\nPath, typer.Option(help=\"Folder to extract compressed raw plaintext to\")\n] = Path(DEFAULT_EXTRACTED_SUBDIR),\ninitial_pk: Annotated[\nint, typer.Option(help=\"First primary key to increment json export from\")\n] = DEFAULT_INITIAL_PK,\nrecords_per_json: Annotated[\nint, typer.Option(help=\"Max records per json fixture\")\n] = DEFAULT_MAX_PLAINTEXT_PER_FIXTURE_FILE,\ndigit_padding: Annotated[\nint, typer.Option(help=\"Padding '0's for indexing json fixture filenames\")\n] = FILE_NAME_0_PADDING_DEFAULT,\ncompress: Annotated[bool, typer.Option(help=\"Compress json fixtures\")] = False,\ncompress_path: Annotated[\nPath, typer.Option(help=\"Folder to compress json fixtueres to\")\n] = Path(COMPRESSED_PATH_DEFAULT),\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\n) -> None:\n\"\"\"Create a PlainTextFixture and save to `save_path`.\"\"\"\nplaintext_fixture = PlainTextFixture(\npath=path,\ndata_provider_code=data_provider_code,\nextract_subdir=extract_path,\nexport_directory=save_path,\ninitial_pk=initial_pk,\nmax_plaintext_per_fixture_file=records_per_json,\njson_0_file_name_padding=digit_padding,\njson_export_compression_format=compress_format,\njson_export_compression_subdir=compress_path,\n)\nplaintext_fixture.info()\nwhile (\nnot plaintext_fixture.compressed_files\nand not plaintext_fixture.plaintext_provided_uncompressed\n):\ntry_another_compressed_txt_source: bool = Confirm.ask(\nf\"No .txt files available from extract path: \"\nf\"{plaintext_fixture.trunc_extract_path_str}\\n\"\nf\"Would you like to extract fixtures from a different path?\",\ndefault=\"n\",\n)\nif try_another_compressed_txt_source:\nnew_extract_path: str = Prompt.ask(\"Please enter a new extract path\")\nplaintext_fixture.path = Path(new_extract_path)\nelse:\nreturn\nplaintext_fixture.info()\nplaintext_fixture.extract_compressed()\nplaintext_fixture.export_to_json_fixtures()\nif compress:\nplaintext_fixture.compress_json_exports()\nrename(\npath: Annotated[Path, typer.Argument(help=\"Path to files to manage\")],\nfolder: Annotated[\nPath, typer.Option(help=\"Path under `path` for new files\")\n] = Path(),\nrenumber: Annotated[\nbool, typer.Option(help=\"Show changes without applying\")\n] = False,\nregex: Annotated[str, typer.Option(help=\"Regex to filter files\")] = \"*.txt\",\npadding: Annotated[\nint, typer.Option(help=\"Digits to pad file name\")\n] = FILE_NAME_0_PADDING_DEFAULT,\nprefix: Annotated[str, typer.Option(help=\"Prefix for new file names\")] = \"\",\ndry_run: Annotated[\nbool, typer.Option(help=\"Show changes without applying\")\n] = True,\ncompress: Annotated[\nbool, typer.Option(help=\"Whether to compress files\")\n] = False,\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\ncompress_suffix: Annotated[\nstr, typer.Option(help=\"Compressed file name suffix\")\n] = \"\",\ncompress_folder: Annotated[\nPath, typer.Option(help=\"Optional folder to differ from renaming\")\n] = COMPRESSED_PATH_DEFAULT,\ndelete_uncompressed: Annotated[\nbool, typer.Option(help=\"Delete unneeded files after compression\")\n] = False,\nlog_level: Annotated[\nint, typer.Option(help=\"Set logging level for debugging\")\n] = WARNING,\nforce: Annotated[\nbool, typer.Option(--force, help=\"Force run without prompt\")\n] = False,\n) -> None\nManage file names and compression.
Source code inalto2txt2fixture/cli.py @cli.command()\ndef rename(\npath: Annotated[Path, typer.Argument(help=\"Path to files to manage\")],\nfolder: Annotated[\nPath, typer.Option(help=\"Path under `path` for new files\")\n] = Path(),\nrenumber: Annotated[\nbool, typer.Option(help=\"Show changes without applying\")\n] = False,\nregex: Annotated[str, typer.Option(help=\"Regex to filter files\")] = \"*.txt\",\npadding: Annotated[\nint, typer.Option(help=\"Digits to pad file name\")\n] = FILE_NAME_0_PADDING_DEFAULT,\nprefix: Annotated[str, typer.Option(help=\"Prefix for new file names\")] = \"\",\ndry_run: Annotated[bool, typer.Option(help=\"Show changes without applying\")] = True,\ncompress: Annotated[bool, typer.Option(help=\"Whether to compress files\")] = False,\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\ncompress_suffix: Annotated[\nstr, typer.Option(help=\"Compressed file name suffix\")\n] = \"\",\ncompress_folder: Annotated[\nPath, typer.Option(help=\"Optional folder to differ from renaming\")\n] = COMPRESSED_PATH_DEFAULT,\ndelete_uncompressed: Annotated[\nbool, typer.Option(help=\"Delete unneeded files after compression\")\n] = False,\nlog_level: Annotated[\nint, typer.Option(help=\"Set logging level for debugging\")\n] = WARNING,\nforce: Annotated[\nbool, typer.Option(\"--force\", help=\"Force run without prompt\")\n] = False,\n) -> None:\n\"\"\"Manage file names and compression.\"\"\"\nlogger.level = log_level\nfolder_path: Path = Path(path) / folder\ncompress_path: Path = Path(path) / compress_folder\ntry:\npaths_dict: dict[os.PathLike, os.PathLike] = glob_path_rename_by_0_padding(\npath=path,\noutput_path=folder,\nglob_regex_str=regex,\npadding=padding,\n)\nassert paths_dict\nexcept (ValueError, AssertionError, IndexError) as err:\nconsole.print(f\"Error: '{err}'\\nTried reading from path: '{path}'\")\nraise typer.Abort()\nfiles_count: int = len(paths_dict)\nif not compress and not force:\ncompress = Confirm.ask(\nf\"Compress all ({files_count}) output file(s)?\",\ndefault=\"n\",\n)\nif compress and not force:\ncompress_format = Prompt.ask(\n\"Compression format\",\nchoices=list(str(format) for format in ArchiveFormatEnum),\ndefault=compress_format,\n)\nextra_info_dict: dict[str, int | os.PathLike] = {\n\"rename_path\": folder_path,\n\"compress_path\": compress_path,\n\"HD Space (GB)\": int(free_hd_space_in_GB()),\n}\nconfig_table: Table = func_table(\nrename,\nvalues=locals(),\nextra_dict=extra_info_dict,\n)\nconsole.print(config_table)\nfile_names_table: Table = file_rename_table(\npaths_dict,\ncompress_format=compress_format,\ntitle=FILE_RENAME_TABLE_TITLE_DEFAULT,\nprefix=prefix,\nrenumber=renumber,\n)\nconsole.print(file_names_table)\nif dry_run:\nif not force:\nrenumber = Confirm.ask(\nf\"Copy {'and compress ' if compress else ''}\"\nf\"{files_count} files \"\nf\"from:\\n\\t'{path}'\\nto:\\n\\t'{folder_path}'\\n\"\n)\nif not delete_uncompressed:\ndelete_uncompressed = Confirm.ask(\nf\"Delete all uncompressed, renamed files \"\nf\"in:\\n'{folder_path}'\\n\"\nf\"after compression to '{compress_format}' format in:\"\nf\"\\n'{compress_path}'\\n\",\ndefault=\"n\",\n)\nif renumber:\ncopy_dict_paths(paths_dict)\nif compress:\nfor old_path, new_path in paths_dict.items():\nfile_path: Path = Path(new_path) if renumber else Path(old_path)\ncompress_fixture(\nfile_path,\noutput_path=compress_path,\nsuffix=compress_suffix,\nformat=compress_format,\n)\nif delete_uncompressed and renumber:\nconsole.print(f\"Deleting {new_path}\")\nPath(new_path).unlink()\nshow_fixture_tables(\nrun_settings: dotdict = settings,\nprint_in_call: bool = True,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> list[Table]\nPrint fixture tables specified in settings.fixture_tables in rich.Table format.
Parameters:
Name Type Description Defaultrun_settings dotdict alto2txt2fixture run configuration
settings print_in_call bool whether to print to console (will use console variable if so)
True data_provider_index str key to index dataprovider from NEWSPAPER_COLLECTION_METADATA
DATA_PROVIDER_INDEX Returns:
Type Descriptionlist[Table] A list of rich.Table renders from configurations in run_settings.FIXTURE_TABLES
>>> fixture_tables: list[Table] = show_fixture_tables(\n... settings,\n... print_in_call=False)\n>>> len(fixture_tables)\n1\n>>> fixture_tables[0].title\n'dataprovider'\n>>> [column.header for column in fixture_tables[0].columns]\n['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\n>>> fixture_tables = show_fixture_tables(settings)\n<BLANKLINE>\n...dataprovider...Heritage...\u2502 bl_hmd...\u2502 hmd...\nIt is possible for the example test to fail in different screen sizes. Try increasing the window or screen width of terminal used to check before raising an issue.
Source code inalto2txt2fixture/cli.py def show_fixture_tables(\nrun_settings: dotdict = settings,\nprint_in_call: bool = True,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> list[Table]:\n\"\"\"Print fixture tables specified in ``settings.fixture_tables`` in `rich.Table` format.\n Arguments:\n run_settings: `alto2txt2fixture` run configuration\n print_in_call: whether to print to console (will use ``console`` variable if so)\n data_provider_index: key to index `dataprovider` from ``NEWSPAPER_COLLECTION_METADATA``\n Returns:\n A `list` of `rich.Table` renders from configurations in ``run_settings.FIXTURE_TABLES``\n Example:\n ```pycon\n >>> fixture_tables: list[Table] = show_fixture_tables(\n ... settings,\n ... print_in_call=False)\n >>> len(fixture_tables)\n 1\n >>> fixture_tables[0].title\n 'dataprovider'\n >>> [column.header for column in fixture_tables[0].columns]\n ['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\n >>> fixture_tables = show_fixture_tables(settings)\n <BLANKLINE>\n ...dataprovider...Heritage...\u2502 bl_hmd...\u2502 hmd...\n ```\n Note:\n It is possible for the example test to fail in different screen sizes. Try\n increasing the window or screen width of terminal used to check before\n raising an issue.\n \"\"\"\nif run_settings.FIXTURE_TABLES:\nif \"dataprovider\" in run_settings.FIXTURE_TABLES:\ncheck_newspaper_collection_configuration(\nrun_settings.COLLECTIONS,\nrun_settings.FIXTURE_TABLES[\"dataprovider\"],\ndata_provider_index=data_provider_index,\n)\nconsole_tables: list[Table] = list(\ngen_fixture_tables(run_settings.FIXTURE_TABLES)\n)\nif print_in_call:\nfor console_table in console_tables:\nconsole.print(console_table)\nreturn console_tables\nelse:\nreturn []\nshow_setup(clear: bool = True, title: str = SETUP_TITLE, **kwargs: str) -> None\nGenerate a rich.table.Table for printing configuration to console.
alto2txt2fixture/cli.py def show_setup(clear: bool = True, title: str = SETUP_TITLE, **kwargs) -> None:\n\"\"\"Generate a `rich.table.Table` for printing configuration to console.\"\"\"\nif clear and os.name == \"posix\":\nos.system(\"clear\")\nelif clear:\nos.system(\"cls\")\ntable = Table(title=title)\ntable.add_column(\"Setting\", justify=\"right\", style=\"cyan\", no_wrap=True)\ntable.add_column(\"Value\", style=\"magenta\")\nfor key, value in kwargs.items():\ntable.add_row(str(key), str(value))\nconsole.print(table)\nreturn\ncorrect_dict(o: dict) -> list\nReturns a list with corrected data from a provided dictionary.
Source code inalto2txt2fixture/create_adjacent_tables.py def correct_dict(o: dict) -> list:\n\"\"\"Returns a list with corrected data from a provided dictionary.\"\"\"\nreturn [(k, v[0], v[1]) for k, v in o.items() if not v[0].startswith(\"Q\")] + [\n(k, v[1], v[0]) for k, v in o.items() if v[0].startswith(\"Q\")\n]\ncsv2json_list(\ncsv_path: PathLike,\noutput_path: Path = OUTPUT,\nsaved: list[Path] | None = None,\nindent: int = JSON_INDENT,\n) -> list\nSave csv_path as a json file and return as a list.
Managing Pandas DataFrame nan values via suggestion: https://stackoverflow.com/a/62691803/678486
alto2txt2fixture/create_adjacent_tables.py def csv2json_list(\ncsv_path: PathLike,\noutput_path: Path = OUTPUT,\nsaved: list[Path] | None = None,\nindent: int = JSON_INDENT,\n) -> list:\n\"\"\"Save `csv_path` as a `json` file and return as a `list`.\n Note:\n Managing `Pandas` `DataFrame` `nan` values via suggestion:\n https://stackoverflow.com/a/62691803/678486\n \"\"\"\njson_data: list[dict[str, Any]] = []\ndf: pd.DataFame = (\npd.read_csv(csv_path, index_col=0).fillna(np.nan).replace([np.nan], [None])\n)\nif \"political_leanings\" in df.columns:\ndf[\"political_leanings\"] = df[\"political_leanings\"].apply(json.loads)\nif \"prices\" in df.columns:\ndf[\"prices\"] = df[\"prices\"].apply(json.loads)\nmodel: str = Path(csv_path).stem.lower()\nfor pk, row in df.iterrows():\nfields: dict[str, Any] = row.to_dict()\njson_data.append({\"pk\": pk, \"model\": model, \"fields\": fields})\njson_path: Path = Path(output_path) / f\"{Path(csv_path).stem}.json\"\njson_path.parent.mkdir(parents=True, exist_ok=True)\njson_path.write_text(json.dumps(json_data, indent=indent))\nif saved and isinstance(saved, list):\nsaved.append(json_path)\nreturn json_data\ndownload_data(\nfiles_dict: RemoteDataFilesType = {},\noverwrite: bool = OVERWRITE,\nexclude: list[str] = [],\n) -> None\nDownload files in files_dict, overwrite if specified.
Parameters:
Name Type Description Defaultfiles_dict RemoteDataFilesType dict of related files to download
{} overwrite bool bool to overwrite LOCAL_CACHE files or not
OVERWRITE exclude list[str] list of files to exclude from files_dict
[] Example >>> from os import chdir\n>>> tmp_path: Path = getfixture('tmp_path')\n>>> set_path: Path = chdir(tmp_path)\n>>> download_data(exclude=[\"mitchells\", \"Newspaper-1\", \"linking\"])\nExcluding mitchells...\nExcluding Newspaper-1...\nExcluding linking...\nDownloading cache...dict_admin_counties.json\n100% ... 37/37 bytes\nDownloading cache...dict_countries.json\n100% ... 33.2/33.2 kB\nDownloading cache...dict_historic_counties.json\n100% ... 41.4/41.4 kB\nDownloading cache...nlp_loc_wikidata_concat.csv\n100% ... 59.8/59.8 kB\nDownloading cache...wikidata_gazetteer_selected_columns.csv\n100% ... 47.8/47.8 MB\nalto2txt2fixture/create_adjacent_tables.py def download_data(\nfiles_dict: RemoteDataFilesType = {},\noverwrite: bool = OVERWRITE,\nexclude: list[str] = [],\n) -> None:\n\"\"\"Download files in ``files_dict``, overwrite if specified.\n Args:\n files_dict: `dict` of related files to download\n overwrite: `bool` to overwrite ``LOCAL_CACHE`` files or not\n exclude: `list` of files to exclude from ``files_dict``\n Example:\n ```pycon\n >>> from os import chdir\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> set_path: Path = chdir(tmp_path)\n >>> download_data(exclude=[\"mitchells\", \"Newspaper-1\", \"linking\"])\n Excluding mitchells...\n Excluding Newspaper-1...\n Excluding linking...\n Downloading cache...dict_admin_counties.json\n 100% ... 37/37 bytes\n Downloading cache...dict_countries.json\n 100% ... 33.2/33.2 kB\n Downloading cache...dict_historic_counties.json\n 100% ... 41.4/41.4 kB\n Downloading cache...nlp_loc_wikidata_concat.csv\n 100% ... 59.8/59.8 kB\n Downloading cache...wikidata_gazetteer_selected_columns.csv\n 100% ... 47.8/47.8 MB\n ```\n \"\"\"\nif not files_dict:\nfiles_dict = deepcopy(FILES)\nfor data_source in exclude:\nif data_source in files_dict:\nprint(f\"Excluding {data_source}...\")\nfiles_dict.pop(data_source, 0)\nelse:\nlogger.warning(\nf'\"{data_source}\" not an option to exclude from {files_dict}'\n)\n# Describe whether local file exists\nfor k in files_dict.keys():\nfiles_dict[k][\"exists\"] = files_dict[k][\"local\"].exists()\nfiles_to_download = [\n(v[\"remote\"], v[\"local\"], v[\"exists\"])\nfor v in files_dict.values()\nif \"exists\" in v and not v[\"exists\"] or overwrite\n]\nfor url, out, exists in files_to_download:\nrmtree(Path(out), ignore_errors=True) if exists else None\nprint(f\"Downloading {out}\")\nPath(out).parent.mkdir(parents=True, exist_ok=True)\nassert isinstance(url, str)\nwith urlopen(url) as response, open(out, \"wb\") as out_file:\ntotal: int = int(response.info()[\"Content-length\"])\nwith Progress(\n\"[progress.percentage]{task.percentage:>3.0f}%\",\nBarColumn(), # removed bar_width=None to avoid too long when resized\nDownloadColumn(),\n) as progress:\ndownload_task = progress.add_task(\"Download\", total=total)\nfor chunk in response:\nout_file.write(chunk)\nprogress.update(download_task, advance=len(chunk))\nget_list(x)\nGet a list from a string, which contains as separator. If no string is encountered, the function returns an empty list. Source code in alto2txt2fixture/create_adjacent_tables.py
def get_list(x):\n\"\"\"Get a list from a string, which contains <SEP> as separator. If no\n string is encountered, the function returns an empty list.\"\"\"\nreturn x.split(\"<SEP>\") if isinstance(x, str) else []\nget_outpaths_dict(\nnames: Sequence[str], module_name: str\n) -> TableOutputConfigType\nReturn a dict of csv and json paths for each module_name table.
The csv and json paths
Parameters:
Name Type Description Defaultnames Sequence[str] iterable of names of each module_name's component. Main target is csv and json table names
module_name str name of module each name is part of, that is added as a prefix
requiredReturns:
Type DescriptionTableOutputConfigType A TableOutputConfigType: a dict of table names and output csv and json filenames.
>>> pprint(get_outpaths_dict(MITCHELLS_TABELS, \"mitchells\"))\n{'Entry': {'csv': 'mitchells.Entry.csv', 'json': 'mitchells.Entry.json'},\n 'Issue': {'csv': 'mitchells.Issue.csv', 'json': 'mitchells.Issue.json'},\n 'PoliticalLeaning': {'csv': 'mitchells.PoliticalLeaning.csv',\n 'json': 'mitchells.PoliticalLeaning.json'},\n 'Price': {'csv': 'mitchells.Price.csv', 'json': 'mitchells.Price.json'}}\nalto2txt2fixture/create_adjacent_tables.py def get_outpaths_dict(names: Sequence[str], module_name: str) -> TableOutputConfigType:\n\"\"\"Return a `dict` of `csv` and `json` paths for each `module_name` table.\n The `csv` and `json` paths\n Args:\n names: iterable of names of each `module_name`'s component. Main target is `csv` and `json` table names\n module_name: name of module each name is part of, that is added as a prefix\n Returns:\n A ``TableOutputConfigType``: a `dict` of table ``names`` and output\n `csv` and `json` filenames.\n Example:\n ```pycon\n >>> pprint(get_outpaths_dict(MITCHELLS_TABELS, \"mitchells\"))\n {'Entry': {'csv': 'mitchells.Entry.csv', 'json': 'mitchells.Entry.json'},\n 'Issue': {'csv': 'mitchells.Issue.csv', 'json': 'mitchells.Issue.json'},\n 'PoliticalLeaning': {'csv': 'mitchells.PoliticalLeaning.csv',\n 'json': 'mitchells.PoliticalLeaning.json'},\n 'Price': {'csv': 'mitchells.Price.csv', 'json': 'mitchells.Price.json'}}\n ```\n \"\"\"\nreturn {\nname: OutputPathDict(\ncsv=f\"{module_name}.{name}.csv\",\njson=f\"{module_name}.{name}.json\",\n)\nfor name in names\n}\nrun(\nfiles_dict: dict = {},\nfiles_to_download_overwrite: bool = OVERWRITE,\nsaved: list[PathLike] = SAVED,\ntime_stamp: str = \"\",\noutput_path: Path = OUTPUT,\n) -> list[PathLike]\nDownload, process and link files_dict to json and csv.
This will require access to https://zooniversedata.blob.core.windows.net/downloads/.
alto2txt2fixture/create_adjacent_tables.py def run(\nfiles_dict: dict = {},\nfiles_to_download_overwrite: bool = OVERWRITE,\nsaved: list[PathLike] = SAVED,\ntime_stamp: str = \"\",\noutput_path: Path = OUTPUT,\n) -> list[PathLike]:\n\"\"\"Download, process and link ``files_dict`` to `json` and `csv`.\n Note:\n This will require access to `https://zooniversedata.blob.core.windows.net/downloads/`.\n \"\"\"\n# Ensure time_stamp from the point of calling `run`\nif not time_stamp:\ntime_stamp = get_now(as_str=False).strftime(TIME_FORMAT)\n# Ensure an independent deepcopy of FILES to avoid modifying subsequent runs\nif not files_dict:\nfiles_dict = deepcopy(FILES)\n# Download non-existing files\ndownload_data(files_dict=files_dict, overwrite=files_to_download_overwrite)\n# Create the output directory (defined in output_path)\noutput_path.mkdir(exist_ok=True, parents=True)\n# Read all the Wikidata Q values from Mitchells\nassert \"local\" in files_dict[\"mitchells\"]\nmitchells_df = pd.read_csv(files_dict[\"mitchells\"][\"local\"], index_col=0)\nmitchell_wikidata_mentions = sorted(\nlist(mitchells_df.PLACE_PUB_WIKI.unique()),\nkey=lambda x: int(x.replace(\"Q\", \"\")),\n)\n# Set up wikidata_gazetteer\ngaz_cols = [\"wikidata_id\", \"english_label\", \"latitude\", \"longitude\", \"geonamesIDs\"]\nassert \"local\" in files_dict[\"wikidata_gazetteer_selected_columns\"]\nwikidata_gazetteer = pd.read_csv(\nfiles_dict[\"wikidata_gazetteer_selected_columns\"][\"local\"], usecols=gaz_cols\n)\nwikidata_gazetteer.rename(\n{\n\"wikidata_id\": \"place_wikidata_id\",\n\"english_label\": \"place_label\",\n\"geonamesIDs\": \"geonames_ids\",\n},\naxis=1,\ninplace=True,\n)\n# Read in + fix all dictionaries\ndict_historic_counties = json.loads(\nPath(files_dict[\"dict_historic_counties\"][\"local\"]).read_text()\n)\ndict_admin_counties = json.loads(\nPath(files_dict[\"dict_admin_counties\"][\"local\"]).read_text()\n)\ndict_countries = json.loads(Path(files_dict[\"dict_countries\"][\"local\"]).read_text())\ndict_historic_counties = correct_dict(dict_historic_counties)\ndict_admin_counties = correct_dict(dict_admin_counties)\ndict_countries = correct_dict(dict_countries)\n# Create assisting frames\nhistorical_counties_df = pd.DataFrame(\ndict_historic_counties,\ncolumns=[\"place_wikidata_id\", \"hcounty_label\", \"hcounty_wikidata_id\"],\n)\nadmin_county_df = pd.DataFrame(\ndict_admin_counties,\ncolumns=[\n\"place_wikidata_id\",\n\"admin_county_label\",\n\"admin_county_wikidata_id\",\n],\n)\ncountries_df = pd.DataFrame(\ndict_countries,\ncolumns=[\"place_wikidata_id\", \"country_label\", \"country_wikidata_id\"],\n)\nwikidata_gazetteer = wikidata_gazetteer[\nwikidata_gazetteer.place_wikidata_id.isin(mitchell_wikidata_mentions)\n].sort_values(\"place_wikidata_id\")\nwikidata_gazetteer[\"place_pk\"] = np.arange(1, len(wikidata_gazetteer) + 1)\nwikidata_gazetteer = wikidata_gazetteer[\n[\"place_pk\"] + [x for x in wikidata_gazetteer.columns if not x == \"place_pk\"]\n]\n# Merge wikidata_gazetteer with all the assisting frames (and rename the\n# resulting columns)\nwikidata_gazetteer = pd.merge(\nwikidata_gazetteer, historical_counties_df, on=\"place_wikidata_id\", how=\"left\"\n)\nwikidata_gazetteer = pd.merge(\nwikidata_gazetteer, admin_county_df, on=\"place_wikidata_id\", how=\"left\"\n)\nwikidata_gazetteer = pd.merge(\nwikidata_gazetteer, countries_df, on=\"place_wikidata_id\", how=\"left\"\n)\nwikidata_gazetteer.rename(\n{\n\"admin_county_label\": \"admin_county__label\",\n\"admin_county_wikidata_id\": \"admin_county__wikidata_id\",\n\"hcounty_label\": \"historic_county__label\",\n\"hcounty_wikidata_id\": \"historic_county__wikidata_id\",\n\"country_label\": \"country__label\",\n\"country_wikidata_id\": \"country__wikidata_id\",\n},\naxis=1,\ninplace=True,\n)\n# Split back up into dataframes specific for the tables\nhistoric_county_table = (\nwikidata_gazetteer[[\"historic_county__label\", \"historic_county__wikidata_id\"]]\n.drop_duplicates()\n.copy()\n)\nhistoric_county_table = historic_county_table.replace({\"\": np.nan}).dropna()\nhistoric_county_table[\"historic_county__pk\"] = np.arange(\n1, len(historic_county_table) + 1\n)\nadmin_county_table = (\nwikidata_gazetteer[[\"admin_county__label\", \"admin_county__wikidata_id\"]]\n.drop_duplicates()\n.copy()\n)\nadmin_county_table = admin_county_table.replace({\"\": np.nan}).dropna()\nadmin_county_table[\"admin_county__pk\"] = np.arange(1, len(admin_county_table) + 1)\ncountry_table = (\nwikidata_gazetteer[[\"country__label\", \"country__wikidata_id\"]]\n.drop_duplicates()\n.copy()\n)\ncountry_table = country_table.replace({\"\": np.nan}).dropna()\ncountry_table[\"country__pk\"] = np.arange(1, len(country_table) + 1)\n# Set up place_table from wikidata_gazetteer\nplace_table = wikidata_gazetteer.copy()\nplace_table = (\npd.merge(\nplace_table,\nhistoric_county_table,\non=[\"historic_county__label\", \"historic_county__wikidata_id\"],\nhow=\"left\",\n)\n.drop([\"historic_county__label\", \"historic_county__wikidata_id\"], axis=1)\n.rename({\"historic_county__pk\": \"historic_county_id\"}, axis=1)\n)\nplace_table = (\npd.merge(\nplace_table,\nadmin_county_table,\non=[\"admin_county__label\", \"admin_county__wikidata_id\"],\nhow=\"left\",\n)\n.drop([\"admin_county__label\", \"admin_county__wikidata_id\"], axis=1)\n.rename({\"admin_county__pk\": \"admin_county_id\"}, axis=1)\n)\nplace_table = (\npd.merge(\nplace_table,\ncountry_table,\non=[\"country__label\", \"country__wikidata_id\"],\nhow=\"left\",\n)\n.drop([\"country__label\", \"country__wikidata_id\"], axis=1)\n.rename({\"country__pk\": \"country_id\"}, axis=1)\n)\nplace_table.fillna(\"\", inplace=True)\nplace_table.set_index(\"place_pk\", inplace=True)\nplace_table.rename(\n{\"place_label\": \"label\", \"place_wikidata_id\": \"wikidata_id\"},\naxis=1,\ninplace=True,\n)\nplace_table[\"historic_county_id\"] = (\nplace_table[\"historic_county_id\"]\n.replace(r\"^\\s*$\", 0, regex=True)\n.astype(int)\n.replace(0, \"\")\n)\nplace_table[\"admin_county_id\"] = (\nplace_table[\"admin_county_id\"]\n.replace(r\"^\\s*$\", 0, regex=True)\n.astype(int)\n.replace(0, \"\")\n)\nplace_table[\"country_id\"] = (\nplace_table[\"country_id\"]\n.replace(r\"^\\s*$\", 0, regex=True)\n.astype(int)\n.replace(0, \"\")\n)\nplace_table.index.rename(\"pk\", inplace=True)\nplace_table.rename(\n{\n\"historic_county_id\": \"historic_county\",\n\"admin_county_id\": \"admin_county\",\n\"country_id\": \"country\",\n},\naxis=1,\ninplace=True,\n)\nhistoric_county_table.set_index(\"historic_county__pk\", inplace=True)\nhistoric_county_table.rename(\n{x: x.split(\"__\")[1] for x in historic_county_table.columns},\naxis=1,\ninplace=True,\n)\nhistoric_county_table.index.rename(\"pk\", inplace=True)\nadmin_county_table.set_index(\"admin_county__pk\", inplace=True)\nadmin_county_table.rename(\n{x: x.split(\"__\")[1] for x in admin_county_table.columns}, axis=1, inplace=True\n)\nadmin_county_table.index.rename(\"pk\", inplace=True)\ncountry_table.set_index(\"country__pk\", inplace=True)\ncountry_table.rename(\n{x: x.split(\"__\")[1] for x in country_table.columns}, axis=1, inplace=True\n)\ncountry_table.index.rename(\"pk\", inplace=True)\n# Adding created_at, updated_at to all the gazetteer tables\nplace_table[\"created_at\"] = time_stamp\nplace_table[\"updated_at\"] = time_stamp\nadmin_county_table[\"created_at\"] = time_stamp\nadmin_county_table[\"updated_at\"] = time_stamp\nhistoric_county_table[\"created_at\"] = time_stamp\nhistoric_county_table[\"updated_at\"] = time_stamp\ncountry_table[\"created_at\"] = time_stamp\ncountry_table[\"updated_at\"] = time_stamp\n# Save CSV files for gazetteer tables\nplace_table.to_csv(output_path / GAZETTEER_OUT_FILENAMES[PLACE][\"csv\"])\nadmin_county_table.to_csv(\noutput_path / GAZETTEER_OUT_FILENAMES[ADMIN_COUNTY][\"csv\"]\n)\nhistoric_county_table.to_csv(\noutput_path / GAZETTEER_OUT_FILENAMES[HISTORIC_COUNTY][\"csv\"]\n)\ncountry_table.to_csv(output_path / GAZETTEER_OUT_FILENAMES[COUNTRY][\"csv\"])\nsaved.extend(\n[\noutput_path / GAZETTEER_OUT_FILENAMES[PLACE][\"csv\"],\noutput_path / GAZETTEER_OUT_FILENAMES[ADMIN_COUNTY][\"csv\"],\noutput_path / GAZETTEER_OUT_FILENAMES[HISTORIC_COUNTY][\"csv\"],\noutput_path / GAZETTEER_OUT_FILENAMES[COUNTRY][\"csv\"],\n]\n)\n# Fix up Mitchells (already loaded)\nmitchells_df[\"politics\"] = mitchells_df.POLITICS.apply(get_list)\nmitchells_df[\"persons\"] = mitchells_df.PERSONS.apply(get_list)\nmitchells_df[\"organisations\"] = mitchells_df.ORGANIZATIONS.apply(get_list)\nmitchells_df[\"price\"] = mitchells_df.PRICE.apply(get_list)\nmitchells_df.rename(\n{\n\"ID\": \"mpd_id\",\n\"TITLE\": \"title\",\n\"politics\": \"political_leaning_raw\",\n\"price\": \"price_raw\",\n\"YEAR\": \"year\",\n\"PLACE_PUB_WIKI\": \"place_of_publication_id\",\n\"ESTABLISHED_DATE\": \"date_established_raw\",\n\"PUBLISED_DATE\": \"day_of_publication_raw\",\n},\naxis=1,\ninplace=True,\n)\ndrop_cols = [\n\"CHAIN_ID\",\n\"POLITICS\",\n\"PERSONS\",\n\"ORGANIZATIONS\",\n\"PRICE\",\n\"PLACE_PUB\",\n\"PLACE_PUB_COORD\",\n\"PLACES\",\n\"PLACES_TRES\",\n\"TEXT\",\n]\nmitchells_df.drop(columns=drop_cols, inplace=True)\n# Create derivative tables (from Mitchells) = political_leanings, prices,\n# issues\npolitical_leanings = sorted(\nlist(set([y.strip() for x in mitchells_df.political_leaning_raw for y in x]))\n)\npolitical_leanings_table = pd.DataFrame()\npolitical_leanings_table[\"political_leaning__pk\"] = np.arange(\n1, len(political_leanings) + 1\n)\npolitical_leanings_table[\"political_leaning__label\"] = political_leanings\nexport = political_leanings_table.copy()\nexport[\"created_at\"] = time_stamp\nexport[\"updated_at\"] = time_stamp\nexport.set_index(\"political_leaning__pk\", inplace=True)\nexport.index.rename(\"pk\", inplace=True)\nexport.rename(\n{x: x.split(\"__\")[1] if len(x.split(\"__\")) > 1 else x for x in export.columns},\naxis=1,\ninplace=True,\n)\nexport.to_csv(output_path / MITCHELLS_OUT_FILENAMES[POLITICAL_LEANING][\"csv\"])\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[POLITICAL_LEANING][\"csv\"])\nprices = sorted(list(set([y.strip() for x in mitchells_df.price_raw for y in x])))\nprices_table = pd.DataFrame()\nprices_table[\"price__pk\"] = np.arange(1, len(prices) + 1)\nprices_table[\"price__label\"] = prices\nexport = prices_table.copy()\nexport[\"created_at\"] = time_stamp\nexport[\"updated_at\"] = time_stamp\nexport.set_index(\"price__pk\", inplace=True)\nexport.index.rename(\"pk\", inplace=True)\nexport.rename(\n{x: x.split(\"__\")[1] if len(x.split(\"__\")) > 1 else x for x in export.columns},\naxis=1,\ninplace=True,\n)\nexport.to_csv(output_path / MITCHELLS_OUT_FILENAMES[PRICE][\"csv\"])\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[PRICE][\"csv\"])\nissues = sorted(list(mitchells_df.year.unique()))\nissues_table = pd.DataFrame()\nissues_table[\"issue__pk\"] = np.arange(1, len(issues) + 1)\nissues_table[\"issue__year\"] = issues\nexport = issues_table.copy()\nexport[\"created_at\"] = time_stamp\nexport[\"updated_at\"] = time_stamp\nexport.set_index(\"issue__pk\", inplace=True)\nexport.index.rename(\"pk\", inplace=True)\nexport.rename(\n{x: x.split(\"__\")[1] if len(x.split(\"__\")) > 1 else x for x in export.columns},\naxis=1,\ninplace=True,\n)\nexport.to_csv(output_path / MITCHELLS_OUT_FILENAMES[ISSUE][\"csv\"])\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[ISSUE][\"csv\"])\n# Set up linking on Mitchells dataframe\nlinking_df = pd.read_csv(\nfiles_dict[\"linking\"][\"local\"],\nindex_col=0,\ndtype={\"NLP\": str},\nusecols=[\n\"NLP\",\n\"Title\",\n\"AcquiredYears\",\n\"Editions\",\n\"EditionTitles\",\n\"City\",\n\"Publisher\",\n\"UnavailableYears\",\n\"Collection\",\n\"UK\",\n\"Complete\",\n\"Notes\",\n\"County\",\n\"HistoricCounty\",\n\"First date held\",\n\"Publication title\",\n\"link_to_mpd\",\n],\n)\nlinking_df[\"NLP\"] = linking_df.index\nlinking_df.rename(\n{\"link_to_mpd\": \"mpd_id\", \"NLP\": \"newspaper\"}, axis=1, inplace=True\n)\n# Link Mitchells with all the other data\nmitchells_df = pd.merge(mitchells_df, linking_df, on=\"mpd_id\", how=\"inner\")\n# Create entry_table\nentry_table = mitchells_df.copy()\nentry_table[\"place_of_circulation_raw\"] = \"\"\nentry_table[\"publication_district_raw\"] = \"\"\nentry_table[\"publication_county_raw\"] = \"\"\n# TODO: What happened to the three columns above? (Check w Kaspar?)\n# Only keep relevant columns\nentry_table = entry_table[\n[\n\"title\",\n\"political_leaning_raw\",\n\"price_raw\",\n\"year\",\n\"date_established_raw\",\n\"day_of_publication_raw\",\n\"place_of_circulation_raw\",\n\"publication_district_raw\",\n\"publication_county_raw\",\n\"organisations\",\n\"persons\",\n\"place_of_publication_id\",\n\"newspaper\",\n]\n]\n# Fix refs to political_leanings_table\nrev = political_leanings_table.set_index(\"political_leaning__label\")\nentry_table[\"political_leanings\"] = entry_table.political_leaning_raw.apply(\nlambda x: [rev.at[y, \"political_leaning__pk\"] for y in x]\n)\n# Fix refs to prices_table\nrev = prices_table.set_index(\"price__label\")\nentry_table[\"prices\"] = entry_table.price_raw.apply(\nlambda x: [rev.at[y.strip(), \"price__pk\"] for y in x]\n)\n# Fix refs to issues_table\nrev = issues_table.set_index(\"issue__year\")\nentry_table[\"issue\"] = entry_table.year.apply(lambda x: rev.at[x, \"issue__pk\"])\n# Fix refs to place_table\nrev = place_table.copy()\nrev[\"place__pk\"] = rev.index\nrev.set_index(\"wikidata_id\", inplace=True)\nentry_table[\"place_of_publication\"] = entry_table.place_of_publication_id.apply(\ntest_place, rev=rev\n)\nentry_table.drop(columns=[\"place_of_publication_id\"], inplace=True)\n# Set up ref to newspapers\nrev = json.loads(files_dict[\"Newspaper-1\"][\"local\"].read_text())\nrev = [dict(pk=v[\"pk\"], **v[\"fields\"]) for v in rev]\nrev = pd.DataFrame(rev)\nrev.set_index(\"publication_code\", inplace=True)\nentry_table[\"newspaper\"] = entry_table.newspaper.str.zfill(7)\nentry_table[\"newspaper\"] = entry_table.newspaper.apply(test_paper, rev=rev)\n# Create PK for entries\nentry_table[\"pk\"] = np.arange(1, len(entry_table) + 1)\n# Sort columns in entries file\nentry_table = entry_table[\n[\"pk\"] + [col for col in entry_table.columns if not col == \"pk\"]\n]\n# Add created_at, modified_at to entry_table\nentry_table[\"created_at\"] = time_stamp\nentry_table[\"updated_at\"] = time_stamp\n# Export entry_table\nentry_table.set_index(\"pk\").to_csv(\noutput_path / MITCHELLS_OUT_FILENAMES[ENTRY][\"csv\"]\n)\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[ENTRY][\"csv\"])\n# ######\u00a0NOW WE CAN EASILY CREATE JSON files_dict\nfor csv_file_path in output_path.glob(\"*.csv\"):\ncsv2json_list(csv_file_path, output_path=output_path)\nprint(\"Finished - saved files:\")\nprint(\"- \" + \"\\n- \".join([str(x) for x in saved]))\nreturn saved\nget_jisc_title(\ntitle: str,\nissue_date: str,\njisc_papers: pd.DataFrame,\ninput_sub_path: str,\npublication_code: str,\nabbr: str | None = None,\n) -> str\nMatch a newspaper title with jisc_papers records.
Takes an input_sub_path, a publication_code, and an (optional) abbreviation for any newspaper to locate the title in the jisc_papers DataFrame. jisc_papers is usually loaded via the setup_jisc_papers function.
Parameters:
Name Type Description Defaulttitle str target newspaper title
requiredissue_date str target newspaper issue_date
requiredjisc_papers pd.DataFrame DataFrame of jisc_papers to match
input_sub_path str path of files to narrow down query input_sub_path
requiredpublication_code str unique codes to match newspaper records
requiredabbr str | None an optional abbreviation of the newspaper title
None Returns:
Type Descriptionstr Matched title str or abbr.
Returns:
Type Descriptionstr A string estimating the JISC equivalent newspaper title
Source code inalto2txt2fixture/jisc.py def get_jisc_title(\ntitle: str,\nissue_date: str,\njisc_papers: pd.DataFrame,\ninput_sub_path: str,\npublication_code: str,\nabbr: str | None = None,\n) -> str:\n\"\"\"\n Match a newspaper ``title`` with ``jisc_papers`` records.\n Takes an ``input_sub_path``, a ``publication_code``, and an (optional)\n abbreviation for any newspaper to locate the ``title`` in the\n ``jisc_papers`` `DataFrame`. ``jisc_papers`` is usually loaded via the\n ``setup_jisc_papers`` function.\n Args:\n title: target newspaper title\n issue_date: target newspaper issue_date\n jisc_papers: `DataFrame` of `jisc_papers` to match\n input_sub_path: path of files to narrow down query input_sub_path\n publication_code: unique codes to match newspaper records\n abbr: an optional abbreviation of the newspaper title\n Returns:\n Matched ``title`` `str` or ``abbr``.\n Returns:\n A string estimating the JISC equivalent newspaper title\n \"\"\"\n# First option, search the input_sub_path for a valid-looking publication_code\ng = PUBLICATION_CODE.findall(input_sub_path)\nif len(g) == 1:\npublication_code = g[0]\n# Let's see if we can find title:\ntitle = (\njisc_papers[\njisc_papers.publication_code == publication_code\n].title.to_list()[0]\nif jisc_papers[\njisc_papers.publication_code == publication_code\n].title.count()\n== 1\nelse title\n)\nreturn title\n# Second option, look through JISC papers for best match (on publication_code if we have it, but abbr more importantly if we have it)\nif abbr:\n_publication_code = publication_code\npublication_code = abbr\nif jisc_papers.abbr[jisc_papers.abbr == publication_code].count():\ndate = datetime.strptime(issue_date, \"%Y-%m-%d\")\nmask = (\n(jisc_papers.abbr == publication_code)\n& (date >= jisc_papers.start_date)\n& (date <= jisc_papers.end_date)\n)\nfiltered = jisc_papers.loc[mask]\nif filtered.publication_code.count() == 1:\npublication_code = filtered.publication_code.to_list()[0]\ntitle = filtered.title.to_list()[0]\nreturn title\n# Last option: let's find all the possible titles in the jisc_papers for the abbreviation, and if it's just one unique title, let's pick it!\nif abbr:\ntest = list({x for x in jisc_papers[jisc_papers.abbr == abbr].title})\nif len(test) == 1:\nreturn test[0]\nelse:\nmask1 = (jisc_papers.abbr == publication_code) & (\njisc_papers.publication_code == _publication_code\n)\ntest1 = jisc_papers.loc[mask1]\ntest1 = list({x for x in jisc_papers[jisc_papers.abbr == abbr].title})\nif len(test) == 1:\nreturn test1[0]\n# Fallback: if abbreviation is set, we'll return that:\nif abbr:\n# For these exceptions, see issue comment:\n# https://github.com/alan-turing-institute/Living-with-Machines/issues/2453#issuecomment-1050652587\nif abbr == \"IPJL\":\nreturn \"Ipswich Journal\"\nelif abbr == \"BHCH\":\nreturn \"Bath Chronicle\"\nelif abbr == \"LSIR\":\nreturn \"Leeds Intelligencer\"\nelif abbr == \"AGER\":\nreturn \"Lancaster Gazetter, And General Advertiser For Lancashire West\"\nreturn abbr\nraise RuntimeError(f\"Title {title} could not be found.\")\nsetup_jisc_papers(path: str = settings.JISC_PAPERS_CSV) -> pd.DataFrame\nCreate a DataFrame with information in JISC_PAPERS_CSV in settings.
Returns:
Type Descriptionpd.DataFrame DataFrame with all JISC titles.
alto2txt2fixture/jisc.py def setup_jisc_papers(path: str = settings.JISC_PAPERS_CSV) -> pd.DataFrame:\n\"\"\"\n Create a `DataFrame` with information in `JISC_PAPERS_CSV` in settings.\n Returns:\n `DataFrame` with all JISC titles.\n \"\"\"\nif not Path(path).exists():\nraise RuntimeError(\nf\"Could not find required JISC papers file. Put {Path(path).name} in {Path(path).parent} or correct the settings with a different path.\"\n)\nmonths = {\n\"Jan\": 1,\n\"Feb\": 2,\n\"Mar\": 3,\n\"Apr\": 4,\n\"May\": 5,\n\"Jun\": 6,\n\"June\": 6,\n\"Jul\": 7,\n\"July\": 7,\n\"Aug\": 8,\n\"Sep\": 9,\n\"Sept\": 9,\n\"Oct\": 10,\n\"Nov\": 11,\n\"Dec\": 12,\n\"Dec.\": 12,\n}\njisc_papers = pd.read_csv(\npath,\nusecols=[\n\"Newspaper Title\",\n\"NLP\",\n\"Abbr\",\n\"StartD\",\n\"StartM\",\n\"StartY\",\n\"EndD\",\n\"EndM\",\n\"EndY\",\n],\n)\njisc_papers[\"start_date\"] = jisc_papers.apply(\nlambda x: datetime(\nyear=int(x.StartY),\nmonth=months[x.StartM.strip(\".\").strip()],\nday=int(x.StartD),\n),\naxis=1,\n)\njisc_papers[\"end_date\"] = jisc_papers.apply(\nlambda x: datetime(\nyear=int(x.EndY), month=months[x.EndM.strip(\".\").strip()], day=int(x.EndD)\n),\naxis=1,\n)\njisc_papers.drop(\n[\"StartD\", \"StartM\", \"StartY\", \"EndD\", \"EndM\", \"EndY\"],\naxis=\"columns\",\ninplace=True,\n)\njisc_papers.rename(\n{\"Newspaper Title\": \"title\", \"NLP\": \"publication_code\", \"Abbr\": \"abbr\"},\naxis=1,\ninplace=True,\n)\njisc_papers[\"title\"] = jisc_papers[\"title\"].apply(\nlambda x: \"The \" + x[:-5] if x.strip()[-5:].lower() == \", the\" else x\n)\njisc_papers[\"publication_code\"] = jisc_papers[\"publication_code\"].apply(\nlambda x: str(x).zfill(7)\n)\nreturn jisc_papers\nerror(msg: str, crash: bool = True, silent: bool = True) -> None\nPrint msg in colorama Force.RED and exit()
If silent exit() after call, else raise RuntimeError if crash=True.
alto2txt2fixture/log.py def error(msg: str, crash: bool = True, silent: bool = True) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.RED` and `exit()`\n If `silent` `exit()` after call, else `raise` `RuntimeError` if ``crash=True``.\"\"\"\nif crash and silent:\nprint(f\"{Fore.RED}{msg}{Style.RESET_ALL}\")\nexit()\nelif crash:\nraise RuntimeError(msg) from None\nprint(f\"{Fore.RED}{msg}{Style.RESET_ALL}\")\nreturn\ninfo(msg: str) -> None\nPrint msg in colorama Force.CYAN colour.
alto2txt2fixture/log.py def info(msg: str) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.CYAN` colour.\"\"\"\nprint(f\"{Fore.CYAN}{msg}{Style.RESET_ALL}\")\nreturn\nsuccess(msg: str) -> None\nPrint msg in colorama Force.GREEN colour.
alto2txt2fixture/log.py def success(msg: str) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.GREEN` colour.\"\"\"\nprint(f\"{Fore.GREEN}{msg}{Style.RESET_ALL}\")\nreturn\nwarning(msg: str) -> None\nPrint msg in colorama Force.YELLOW colour.
alto2txt2fixture/log.py def warning(msg: str) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.YELLOW` colour.\"\"\"\nprint(f\"{Fore.YELLOW}Warning: {msg}{Style.RESET_ALL}\")\nreturn\nfixtures(\nfilelist: list = [],\nmodel: str = \"\",\ntranslate: dict = {},\nrename: dict = {},\nuniq_keys: list = [],\n) -> Generator[FixtureDict, None, None]\nGenerates fixtures for a specified model using a list of files.
This function takes a list of files and generates fixtures for a specified model. The fixtures can be used to populate a database or perform other data-related operations.
Parameters:
Name Type Description Defaultfilelist list A list of files to process and generate fixtures from.
[] model str The name of the model for which fixtures are generated. translate: A nested dictionary representing the translation mapping for fields. The structure of the translator follows the format:
{\n'part1': {\n'part2': {\n'translated_field': 'pk'\n}\n}\n}\n'' rename dict A nested dictionary representing the field renaming mapping. The structure of the dictionary follows the format:
{\n'part1': {\n'part2': 'new_field_name'\n}\n}\n{} uniq_keys list A list of fields that need to be considered for uniqueness in the fixtures. If specified, the fixtures will yield only unique items based on the combination of these fields.
[] Yields:
Type DescriptionFixtureDict FixtureDict from model, pk and dict of fields.
Returns:
Type DescriptionGenerator[FixtureDict, None, None] This function generates fixtures but does not return any value.
Source code inalto2txt2fixture/parser.py def fixtures(\nfilelist: list = [],\nmodel: str = \"\",\ntranslate: dict = {},\nrename: dict = {},\nuniq_keys: list = [],\n) -> Generator[FixtureDict, None, None]:\n\"\"\"\n Generates fixtures for a specified model using a list of files.\n This function takes a list of files and generates fixtures for a specified\n model. The fixtures can be used to populate a database or perform other\n data-related operations.\n Args:\n filelist: A list of files to process and generate fixtures from.\n model: The name of the model for which fixtures are generated.\n translate: A nested dictionary representing the translation mapping\n for fields. The structure of the translator follows the format:\n ```python\n {\n 'part1': {\n 'part2': {\n 'translated_field': 'pk'\n }\n }\n }\n ```\n The translated fields will be used as keys, and their\n corresponding primary keys (obtained from the provided files) will\n be used as values in the generated fixtures.\n rename: A nested dictionary representing the field renaming\n mapping. The structure of the dictionary follows the format:\n ```python\n {\n 'part1': {\n 'part2': 'new_field_name'\n }\n }\n ```\n The fields specified in the dictionary will be renamed to the\n provided new field names in the generated fixtures.\n uniq_keys: A list of fields that need to be considered for\n uniqueness in the fixtures. If specified, the fixtures will yield\n only unique items based on the combination of these fields.\n Yields:\n `FixtureDict` from ``model``, ``pk`` and `dict` of ``fields``.\n Returns:\n This function generates fixtures but does not return any value.\n \"\"\"\nfilelist = sorted(filelist, key=lambda x: str(x).split(\"/\")[:-1])\ncount = len(filelist)\n# Process JSONL\nif [x for x in filelist if \".jsonl\" in x.name]:\npk = 0\n# In the future, we might want to show progress here (tqdm or suchlike)\nfor file in filelist:\nfor line in file.read_text().splitlines():\npk += 1\nline = json.loads(line)\nyield FixtureDict(\npk=pk,\nmodel=model,\nfields=dict(**get_fields(line, translate=translate, rename=rename)),\n)\nreturn\nelse:\n# Process JSON\npks = [x for x in range(1, count + 1)]\nif len(uniq_keys):\nuniq_files = list(uniq(filelist, uniq_keys))\ncount = len(uniq_files)\nzipped = zip(uniq_files, pks)\nelse:\nzipped = zip(filelist, pks)\nfor x in tqdm(\nzipped, total=count, desc=f\"{model} ({count:,} objs)\", leave=False\n):\nyield FixtureDict(\npk=x[1],\nmodel=model,\nfields=dict(**get_fields(x[0], translate=translate, rename=rename)),\n)\nreturn\nget_fields(\nfile: Union[Path, str, dict],\ntranslate: dict = {},\nrename: dict = {},\nallow_null: bool = False,\n) -> dict\nRetrieves fields from a file and performs modifications and checks.
This function takes a file (in various formats: Path, str, or dict) and processes its fields. It retrieves the fields from the file and performs modifications, translations, and checks on the fields.
Parameters:
Name Type Description Defaultfile Union[Path, str, dict] The file from which the fields are retrieved.
requiredtranslate dict A nested dictionary representing the translation mapping for fields. The structure of the translator follows the format:
{\n'part1': {\n'part2': {\n'translated_field': 'pk'\n}\n}\n}\n{} rename dict A nested dictionary representing the field renaming mapping. The structure of the dictionary follows the format:
{\n'part1': {\n'part2': 'new_field_name'\n}\n}\n{} allow_null bool Determines whether to allow None values for relational fields. If set to True, relational fields with missing values will be assigned None. If set to False, an error will be raised.
False Returns:
Type Descriptiondict A dictionary representing the retrieved fields from the file, with modifications and checks applied.
Raises:
Type DescriptionRuntimeError If the file type is unsupported or if an error occurs during field retrieval or processing.
Source code inalto2txt2fixture/parser.py def get_fields(\nfile: Union[Path, str, dict],\ntranslate: dict = {},\nrename: dict = {},\nallow_null: bool = False,\n) -> dict:\n\"\"\"\n Retrieves fields from a file and performs modifications and checks.\n This function takes a file (in various formats: `Path`, `str`, or `dict`)\n and processes its fields. It retrieves the fields from the file and\n performs modifications, translations, and checks on the fields.\n Args:\n file: The file from which the fields are retrieved.\n translate: A nested dictionary representing the translation mapping\n for fields. The structure of the translator follows the format:\n ```python\n {\n 'part1': {\n 'part2': {\n 'translated_field': 'pk'\n }\n }\n }\n ```\n The translated fields will be used to replace the original fields\n in the retrieved fields.\n rename: A nested dictionary representing the field renaming\n mapping. The structure of the dictionary follows the format:\n ```python\n {\n 'part1': {\n 'part2': 'new_field_name'\n }\n }\n ```\n The fields specified in the dictionary will be renamed to the\n provided new field names in the retrieved fields.\n allow_null: Determines whether to allow ``None`` values for\n relational fields. If set to ``True``, relational fields with\n missing values will be assigned ``None``. If set to ``False``, an\n error will be raised.\n Returns:\n A dictionary representing the retrieved fields from the file,\n with modifications and checks applied.\n Raises:\n RuntimeError: If the file type is unsupported or if an error occurs\n during field retrieval or processing.\n \"\"\"\nif isinstance(file, Path):\ntry:\nfields = json.loads(file.read_text())\nexcept Exception as e:\nraise RuntimeError(f\"Cannot interpret JSON ({e}): {file}\")\nelif isinstance(file, str):\nif \"\\n\" in file:\nraise RuntimeError(\"File has multiple lines.\")\ntry:\nfields = json.loads(file)\nexcept json.decoder.JSONDecodeError as e:\nraise RuntimeError(f\"Cannot interpret JSON ({e}): {file}\")\nelif isinstance(file, dict):\nfields = file\nelse:\nraise RuntimeError(f\"Cannot process type {type(file)}.\")\n# Fix relational fields for any file\nfor key in [key for key in fields.keys() if \"__\" in key]:\nparts = key.split(\"__\")\ntry:\nbefore = fields[key]\nif before:\nbefore = before.replace(\"---\", \"/\")\nloc = translate.get(parts[0], {}).get(parts[1], {})\nfields[key] = loc.get(before)\nif fields[key] is None:\nraise RuntimeError(\nf\"Cannot translate fields.{key} from {before}: {loc}\"\n)\nexcept AttributeError:\nif allow_null:\nfields[key] = None\nelse:\nprint(\n\"Content had relational fields, but something went wrong in parsing the data:\"\n)\nprint(\"file\", file)\nprint(\"fields\", fields)\nprint(\"KEY:\", key)\nraise RuntimeError()\nnew_name = rename.get(parts[0], {}).get(parts[1], None)\nif new_name:\nfields[new_name] = fields[key]\ndel fields[key]\nfields[\"created_at\"] = NOW_str\nfields[\"updated_at\"] = NOW_str\ntry:\nfields[\"item_type\"] = str(fields[\"item_type\"]).upper()\nexcept KeyError:\npass\ntry:\nif fields[\"ocr_quality_mean\"] == \"\":\nfields[\"ocr_quality_mean\"] = 0\nexcept KeyError:\npass\ntry:\nif fields[\"ocr_quality_sd\"] == \"\":\nfields[\"ocr_quality_sd\"] = 0\nexcept KeyError:\npass\nreturn fields\nget_key_from(item: Path, x: str) -> str\nRetrieves a specific key from a file and returns its value.
This function reads a file and extracts the value of a specified key. If the key is not found or an error occurs while processing the file, a warning is printed, and an empty string is returned.
Parameters:
Name Type Description Defaultitem Path The file from which the key is extracted.
requiredx str The key to be retrieved from the file.
requiredReturns:
Type Descriptionstr The value of the specified key from the file.
Source code inalto2txt2fixture/parser.py def get_key_from(item: Path, x: str) -> str:\n\"\"\"\n Retrieves a specific key from a file and returns its value.\n This function reads a file and extracts the value of a specified\n key. If the key is not found or an error occurs while processing\n the file, a warning is printed, and an empty string is returned.\n Args:\n item: The file from which the key is extracted.\n x: The key to be retrieved from the file.\n Returns:\n The value of the specified key from the file.\n \"\"\"\nresult = json.loads(item.read_text()).get(x, None)\nif not result:\nprint(f\"[WARN] Could not find key {x} in {item}\")\nresult = \"\"\nreturn result\nget_translator(\nfields: list[TranslatorTuple] = [TranslatorTuple(\"\", \"\", [])]\n) -> dict\nConverts a list of fields into a nested dictionary representing a translator.
Parameters:
Name Type Description Defaultfields list[TranslatorTuple] A list of tuples representing fields to be translated.
[TranslatorTuple('', '', [])] Returns:
Type Descriptiondict A nested dictionary representing the translator. The structure of the dictionary follows the format:
{\n'part1': {\n'part2': {\n'translated_field': 'pk'\n}\n}\n}\n>>> fields = [\n... TranslatorTuple(\n... start='start__field1',\n... finish='field1',\n... lst=[{\n... 'fields': {'field1': 'translation1'},\n... 'pk': 1}],\n... )]\n>>> get_translator(fields)\n{'start': {'field1': {'translation1': 1}}}\nalto2txt2fixture/parser.py def get_translator(\nfields: list[TranslatorTuple] = [TranslatorTuple(\"\", \"\", [])]\n) -> dict:\n\"\"\"\n Converts a list of fields into a nested dictionary representing a\n translator.\n Args:\n fields: A list of tuples representing fields to be translated.\n Returns:\n A nested dictionary representing the translator. The structure of\n the dictionary follows the format:\n ```python\n {\n 'part1': {\n 'part2': {\n 'translated_field': 'pk'\n }\n }\n }\n ```\n Example:\n ```pycon\n >>> fields = [\n ... TranslatorTuple(\n ... start='start__field1',\n ... finish='field1',\n ... lst=[{\n ... 'fields': {'field1': 'translation1'},\n ... 'pk': 1}],\n ... )]\n >>> get_translator(fields)\n {'start': {'field1': {'translation1': 1}}}\n ```\n \"\"\"\n_ = dict()\nfor field in fields:\nstart, finish, lst = field\npart1, part2 = start.split(\"__\")\nif part1 not in _:\n_[part1] = {}\nif part2 not in _[part1]:\n_[part1][part2] = {}\nif isinstance(finish, str):\n_[part1][part2] = {o[\"fields\"][finish]: o[\"pk\"] for o in lst}\nelif isinstance(finish, list):\n_[part1][part2] = {\n\"-\".join([o[\"fields\"][x] for x in finish]): o[\"pk\"] for o in lst\n}\nreturn _\nparse(\ncollections: list, cache_home: str, output: str, max_elements_per_file: int\n) -> None\nParses files from collections and generates fixtures for various models.
This function processes files from the specified collections and generates fixtures for different models, such as newspapers.dataprovider, newspapers.ingest, newspapers.digitisation, newspapers.newspaper, newspapers.issue, and newspapers.item.
It performs various steps, such as file listing, fixture generation, translation mapping, renaming fields, and saving fixtures to files.
Parameters:
Name Type Description Defaultcollections list A list of collections from which files are processed and fixtures are generated.
requiredcache_home str The directory path where the collections are located.
requiredoutput str The directory path where the fixtures will be saved.
requiredmax_elements_per_file int The maximum number of elements per file when saving fixtures.
requiredReturns:
Type DescriptionNone This function generates fixtures but does not return any value.
Source code inalto2txt2fixture/parser.py def parse(\ncollections: list, cache_home: str, output: str, max_elements_per_file: int\n) -> None:\n\"\"\"\n Parses files from collections and generates fixtures for various models.\n This function processes files from the specified collections and generates\n fixtures for different models, such as `newspapers.dataprovider`,\n `newspapers.ingest`, `newspapers.digitisation`, `newspapers.newspaper`,\n `newspapers.issue`, and `newspapers.item`.\n It performs various steps, such as file listing, fixture generation,\n translation mapping, renaming fields, and saving fixtures to files.\n Args:\n collections: A list of collections from which files are\n processed and fixtures are generated.\n cache_home: The directory path where the collections are located.\n output: The directory path where the fixtures will be saved.\n max_elements_per_file: The maximum number of elements per file\n when saving fixtures.\n Returns:\n This function generates fixtures but does not return any value.\n \"\"\"\nglobal CACHE_HOME\nglobal OUTPUT\nglobal MAX_ELEMENTS_PER_FILE\nCACHE_HOME = cache_home\nOUTPUT = output\nMAX_ELEMENTS_PER_FILE = max_elements_per_file\n# Set up output directory\nreset_fixture_dir(OUTPUT)\n# Get file lists\nprint(\"\\nGetting file lists...\")\ndef issues_in_x(x):\nreturn \"issues\" in str(x.parent).split(\"/\")\ndef newspapers_in_x(x):\nreturn not any(\n[\ncondition\nfor y in str(x.parent).split(\"/\")\nfor condition in [\n\"issues\" in y,\n\"ingest\" in y,\n\"digitisation\" in y,\n\"data-provider\" in y,\n]\n]\n)\nall_json = [\nx for y in collections for x in (Path(CACHE_HOME) / y).glob(\"**/*.json\")\n]\nall_jsonl = [\nx for y in collections for x in (Path(CACHE_HOME) / y).glob(\"**/*.jsonl\")\n]\nprint(f\"--> {len(all_json):,} JSON files altogether\")\nprint(f\"--> {len(all_jsonl):,} JSONL files altogether\")\nprint(\"\\nSetting up fixtures...\")\n# Process data providers\ndef data_provider_in_x(x):\nreturn \"data-provider\" in str(x.parent).split(\"/\")\ndata_provider_json = list(\nfixtures(\nmodel=\"newspapers.dataprovider\",\nfilelist=[x for x in all_json if data_provider_in_x(x)],\nuniq_keys=[\"name\"],\n)\n)\nprint(f\"--> {len(data_provider_json):,} DataProvider fixtures\")\n# Process ingest\ndef ingest_in_x(x):\nreturn \"ingest\" in str(x.parent).split(\"/\")\ningest_json = list(\nfixtures(\nmodel=\"newspapers.ingest\",\nfilelist=[x for x in all_json if ingest_in_x(x)],\nuniq_keys=[\"lwm_tool_name\", \"lwm_tool_version\"],\n)\n)\nprint(f\"--> {len(ingest_json):,} Ingest fixtures\")\n# Process digitisation\ndef digitisation_in_x(x):\nreturn \"digitisation\" in str(x.parent).split(\"/\")\ndigitisation_json = list(\nfixtures(\nmodel=\"newspapers.digitisation\",\nfilelist=[x for x in all_json if digitisation_in_x(x)],\nuniq_keys=[\"software\"],\n)\n)\nprint(f\"--> {len(digitisation_json):,} Digitisation fixtures\")\n# Process newspapers\nnewspaper_json = list(\nfixtures(\nmodel=\"newspapers.newspaper\",\nfilelist=[file for file in all_json if newspapers_in_x(file)],\n)\n)\nprint(f\"--> {len(newspaper_json):,} Newspaper fixtures\")\n# Process issue\ntranslate = get_translator(\n[\nTranslatorTuple(\n\"publication__publication_code\", \"publication_code\", newspaper_json\n)\n]\n)\nrename = {\"publication\": {\"publication_code\": \"newspaper_id\"}}\nissue_json = list(\nfixtures(\nmodel=\"newspapers.issue\",\nfilelist=[file for file in all_json if issues_in_x(file)],\ntranslate=translate,\nrename=rename,\n)\n)\nprint(f\"--> {len(issue_json):,} Issue fixtures\")\n# Create translator/clear up memory before processing items\ntranslate = get_translator(\n[\n(\"issue__issue_identifier\", \"issue_code\", issue_json),\n(\"digitisation__software\", \"software\", digitisation_json),\n(\"data_provider__name\", \"name\", data_provider_json),\n(\n\"ingest__lwm_tool_identifier\",\n[\"lwm_tool_name\", \"lwm_tool_version\"],\ningest_json,\n),\n]\n)\nrename = {\n\"issue\": {\"issue_identifier\": \"issue_id\"},\n\"digitisation\": {\"software\": \"digitisation_id\"},\n\"data_provider\": {\"name\": \"data_provider_id\"},\n\"ingest\": {\"lwm_tool_identifier\": \"ingest_id\"},\n}\nsave_fixture(newspaper_json, \"Newspaper\")\nsave_fixture(issue_json, \"Issue\")\ndel newspaper_json\ndel issue_json\ngc.collect()\nprint(\"\\nSaving...\")\nsave_fixture(digitisation_json, \"Digitisation\")\nsave_fixture(ingest_json, \"Ingest\")\nsave_fixture(data_provider_json, \"DataProvider\")\n# Process items\nitem_json = fixtures(\nmodel=\"newspapers.item\",\nfilelist=all_jsonl,\ntranslate=translate,\nrename=rename,\n)\nsave_fixture(item_json, \"Item\")\nreturn\nreset_fixture_dir(output: str | Path) -> None\nResets the fixture directory by removing all JSON files inside it.
This function takes a directory path (output) as input and removes all JSON files within the directory.
Prior to removal, it prompts the user for confirmation to proceed. If the user confirms, the function clears the fixture directory by deleting the JSON files.
Parameters:
Name Type Description Defaultoutput str | Path The directory path of the fixture directory to be reset.
requiredRaises:
Type DescriptionRuntimeError If the output directory is not specified as a string.
alto2txt2fixture/parser.py def reset_fixture_dir(output: str | Path) -> None:\n\"\"\"\n Resets the fixture directory by removing all JSON files inside it.\n This function takes a directory path (``output``) as input and removes all\n JSON files within the directory.\n Prior to removal, it prompts the user for confirmation to proceed. If the\n user confirms, the function clears the fixture directory by deleting the\n JSON files.\n Args:\n output: The directory path of the fixture directory to be reset.\n Raises:\n RuntimeError: If the ``output`` directory is not specified as a string.\n \"\"\"\nif not isinstance(output, str):\nraise RuntimeError(\"`output` directory needs to be specified as a string.\")\noutput = Path(output)\ny = input(\nf\"This command will automatically empty the fixture directory ({output.absolute()}). \"\n\"Do you want to proceed? [y/N]\"\n)\nif not y.lower() == \"y\":\noutput.mkdir(parents=True, exist_ok=True)\nreturn\nprint(\"\\nClearing up the fixture directory\")\n# Ensure directory exists\noutput.mkdir(parents=True, exist_ok=True)\n# Drop all JSON files\n[x.unlink() for x in Path(output).glob(\"*.json\")]\nreturn\nuniq(filelist: list, keys: list = []) -> Generator[Any, None, None]\nGenerates unique items from a list of files based on specified keys.
This function takes a list of files and yields unique items based on a combination of keys. The keys are extracted from each file using the get_key_from function, and duplicate items are ignored.
Parameters:
Name Type Description Defaultfilelist list A list of files from which unique items are generated.
requiredkeys list A list of keys used for uniqueness. Each key specifies a field to be used for uniqueness checking in the generated items.
[] Yields:
Type DescriptionAny A unique item from filelist.
alto2txt2fixture/parser.py def uniq(filelist: list, keys: list = []) -> Generator[Any, None, None]:\n\"\"\"\n Generates unique items from a list of files based on specified keys.\n This function takes a list of files and yields unique items based on a\n combination of keys. The keys are extracted from each file using the\n ``get_key_from`` function, and duplicate items are ignored.\n Args:\n filelist: A list of files from which unique items are\n generated.\n keys: A list of keys used for uniqueness. Each key specifies\n a field to be used for uniqueness checking in the generated\n items.\n Yields:\n A unique item from `filelist`.\n \"\"\"\nseen = set()\nfor item in filelist:\nkey = \"-\".join([get_key_from(item, x) for x in keys])\nif key not in seen:\nseen.add(key)\nyield item\nelse:\n# Drop it if duplicate\npass\nUseful regular expressions, intially just PUBLICATION_CODE.
Bases: TypedDict
A dict of lwmdb.fulltext.models.Fulltext fixture structure.
Attributes:
Name Type Descriptionpath PathLike Plaintext file path.
compressed_path PathLike | None If path is within a compressed file, compressed_path is that source. Else None.
primary_key int An int >= 1 for a SQL table primary key (pk).
dataclass","text":"Manage exporting plaintext alto2txt results for lwmdb.fulltext.models.Fulltext.
Attributes:
Name Type Descriptionpath PathLike PathLike source for fixtures as either a folder or file.
data_provider_code str | None A short string to uniquely identify DataProviders, primarily to match sources in self.data_provider_code_dict.
files tuple[PathLike, ...] | None A iterable PathLike collection of to either read as plaintext or decomepress to extract plaintext.
compressed_glob_regex str A Regular Expression to filter plaintext files from uncompressed self.files, more specifically self.compressed_files.
data_provider DataProviderFixtureDict | None If available a DataProviderFixtureDict for DataProvider metadata. By default all options are stored in self.data_provider_code_dict.
model_str str The name of the lwmdb django model the fixture is for. This is of the form app_name.model_name. Following the default config for lwmdb:
FULLTEXT_DJANGO_MODEL: Final[str] = \"fulltext.fulltext\"\nfulltext app has a fulltext model class specified in lwmdb.fulltext.models.fulltext. A sql table is generated from on that fulltext class and the json fixture structure generated from this class is where records will be stored. extract_subdir PathLike Folder to extract self.compressed_files to.
plaintext_extension str What file extension to use to filter plaintext files.
data_provider_code_dict dict[str, DataProviderFixtureDict] A dict of metadata for preconfigured DataProvider records in lwmdb.
max_plaintext_per_fixture_file int A maximum number of fixtures per fixture file, designed to configure chunking fixtures.
saved_fixture_prefix str A str to prefix all saved json fixture filenames.
export_directory PathLike Directory to save all exported fixtures to.
initial_pk int Default begins at 1, can be set to another number if needed to add to add more to pre-existing set of records up to a given pk
json_0_file_name_padding int Number of 0s to prefix file name numbering.
_disk_usage int Available harddrive space. Designed to help mitigate decompressing too many files for available disk space.
self._uncompressed_source_file_dict int A dictionary of extracted plaintext to compressed source file. This is a field in json fixture records.
>>> path = getfixture('bl_lwm')\n>>> logger.setLevel(INFO)\n>>> plaintext_bl_lwm = PlainTextFixture(\n... data_provider_code='bl_lwm',\n... path=path,\n... compressed_glob_regex=\"*_plaintext.zip\",\n... )\n>>> plaintext_bl_lwm\n<PlainTextFixture(path='...bl_lwm')>\n>>> plaintext_bl_lwm.info()\n<BLANKLINE>\n...PlainTextFixture for 2 'bl_lwm' files...\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500...\u2510\n\u2502 Path \u2502 '...bl_lwm' ...\u2502\n\u2502 Compressed Files \u2502 '...bl_lwm...0003079-test_plain...\u2502\n\u2502 \u2502 '...bl_lwm...0003548-test_plain...\u2502\n\u2502 Extract Path \u2502 '...bl_lwm...extracted' ...\u2502\n\u2502 Uncompressed Files \u2502 None ...\u2502\n\u2502 Data Provider \u2502 'Living with Machines' ...\u2502\n\u2502 Initial Primary Key \u2502 1 ...\u2502\n\u2502 Max Rows Per JSON \u2502 100 ...\u2502\n\u2502 JSON File Name 0s \u2502 6 ...\u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500...\u2518\n>>> plaintext_bl_lwm.free_hd_space_in_GB > 1\nTrue\n>>> plaintext_bl_lwm.extract_compressed()\n<BLANKLINE>\n...Extract path:...'...lwm...extracted...\n...Extracting:...'...lwm...00030...\n...Extracting:...'...lwm...00035...\n...%...[...]...\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in:...'...bl_lwm...tracted'\nproperty","text":"compressed_files: tuple[PathLike, ...]\nReturn a tuple of all self.files with known archive file names.
property","text":"data_provider_name: str | None\nReturn self.data_provider code attributre or None.
self.data_provider_code>>> bl_hmd = PlainTextFixture(\n... path=\".\",\n... data_provider_code=\"bl_hmd\")\n>>> bl_hmd.data_provider_name\n'Heritage Made Digital'\n>>> bl_lwm = PlainTextFixture(\n... path='.',\n... data_provider=NEWSPAPER_DATA_PROVIDER_CODE_DICT['bl_lwm'],\n... )\n>>> bl_lwm.data_provider_name\n'Living with Machines'\n>>> plaintext_fixture = PlainTextFixture(\n... path=\".\")\n<BLANKLINE>\n...`.data_provider` and `.data_provider_code`...\n...are...'None'...in...<PlainTextFixture(path='.')>...\n>>> plaintext_fixture.data_provider_name\n`
"},{"location":"reference/alto2txt2fixture/plaintext.html#alto2txt2fixture.plaintext.PlainTextFixture.exported_json_paths","title":"exported_json_pathsproperty","text":"exported_json_paths: Generator[Path, None, None]\nIf self._exported_json_paths return Generator of those paths.
Yields:
Type DescriptionEach path from self._exported_json_paths
>>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> tuple(plaintext_bl_lwm.exported_json_paths)\n(...Path('...plaintext_fixture-000001.json'),)\nproperty","text":"extract_path: Path\nPath any compressed files would be extracted to.
"},{"location":"reference/alto2txt2fixture/plaintext.html#alto2txt2fixture.plaintext.PlainTextFixture.free_hd_space_in_GB","title":"free_hd_space_in_GBproperty","text":"free_hd_space_in_GB: float\nReturn remaing hard drive space estimate in gigabytes.
"},{"location":"reference/alto2txt2fixture/plaintext.html#alto2txt2fixture.plaintext.PlainTextFixture.info_table","title":"info_tableproperty","text":"info_table: Table\nGenerate a rich.table.Table of config information.
>>> hmd_plaintext_fixture = PlainTextFixture(\n... path=\".\",\n... data_provider_code=\"bl_hmd\")\n>>> table = hmd_plaintext_fixture.info_table\n>>> table.title\n\"PlainTextFixture for 0 'bl_hmd' files\"\nproperty","text":"plaintext_provided_uncompressed: tuple[PathLike, ...]\nReturn a tuple of all self.files with self.plaintext_extension.
property","text":"trunc_compressed_file_names_str: str\nReturn truncated self.compressed_files file names or empty str.
property","text":"trunc_extract_path_str: str\nReturn truncated self.extract_path or empty str.
property","text":"trunc_uncompressed_file_names_str: str\nReturn truncated self.plaintext_provided_uncompressed file names or empty str.
property","text":"zipinfo: Generator[list[ZipInfo], None, None]\nIf self.compressed_files is in zip, return info, else None.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n>>> zipfile_info_list: list[ZipInfo] = list(plaintext_bl_lwm.zipinfo)\nGetting zipfile info from <PlainTextFixture(path='...bl_lwm')>\n>>> zipfile_info_list[0][-1].filename\n'0003079...1898...0204...0003079_18980204_sect0001.txt'\n>>> zipfile_info_list[-1][-1].filename\n'0003548...1904...0707...0003548_19040707_art0059.txt'\n>>> zipfile_info_list[0][-1].file_size\n70192\n>>> zipfile_info_list[0][-1].compress_size\n39911\n__len__() -> int\nReturn the number of files to process.
Source code inalto2txt2fixture/plaintext.py def __len__(self) -> int:\n\"\"\"Return the number of files to process.\"\"\"\nreturn len(self.files) if self.files else 0\n__post_init__() -> None\nManage populating additional attributes if necessary.
Source code inalto2txt2fixture/plaintext.py def __post_init__(self) -> None:\n\"\"\"Manage populating additional attributes if necessary.\"\"\"\nself._check_and_set_files_attr(force=True)\nself._check_and_set_data_provider(force=True)\nself._disk_usage: DiskUsageTuple = disk_usage(self.path)\nself._uncompressed_source_file_dict: OrderedDict[\nPathLike, PathLike\n] = OrderedDict()\nself._pk_plaintext_dict: OrderedDict[PathLike, int] = OrderedDict()\n__repr__() -> str\nReturn class name with path attribute.
alto2txt2fixture/plaintext.py def __repr__(self) -> str:\n\"\"\"Return `class` name with `path` attribute.\"\"\"\nreturn f\"<{type(self).__name__}(path='{self.path}')>\"\n__str__() -> str\nReturn class name with count and DataProvider if available.
alto2txt2fixture/plaintext.py def __str__(self) -> str:\n\"\"\"Return class name with count and `DataProvider` if available.\"\"\"\nreturn (\nf\"{type(self).__name__} \"\nf\"for {len(self)} \"\nf\"{self._data_provider_code_quoted_with_trailing_space}files\"\n)\ncompress_json_exports(\noutput_path: PathLike | None = None, format: ArchiveFormatEnum | None = None\n) -> tuple[Path, ...]\nCompress self._exported_json_paths to format.
Parameters:
Name Type Description Defaultoutput_path PathLike | None Path to save compressed json files to. Uses self.json_export_compression_subdir if None is passed.
None format ArchiveFormatEnum | None What compression format to use from ArchiveFormatEnum. Uses self.json_export_compression_format if None is passed.
None Note Neither output_path nor format overwrite the related attributes of self.
>>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> compressed_paths: Path = plaintext_bl_lwm.compress_json_exports(\n... format='tar')\n<BLANKLINE>\n...Compressing...'...01.json' to...'tar'...in:...\n>>> compressed_paths\n(...Path('.../plaintext_fixture-000001.json.tar'),)\nalto2txt2fixture/plaintext.py def compress_json_exports(\nself,\noutput_path: PathLike | None = None,\nformat: ArchiveFormatEnum | None = None,\n) -> tuple[Path, ...]:\n\"\"\"Compress `self._exported_json_paths` to `format`.\n Args:\n output_path:\n `Path` to save compressed `json` files to. Uses\n `self.json_export_compression_subdir` if `None` is passed.\n format:\n What compression format to use from `ArchiveFormatEnum`. Uses\n `self.json_export_compression_format` if `None` is passed.\n Note:\n Neither `output_path` nor `format` overwrite the related attributes\n of `self`.\n Returns: The the `output_path` passed to save compressed `json`.\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n <BLANKLINE>\n ...\n >>> compressed_paths: Path = plaintext_bl_lwm.compress_json_exports(\n ... format='tar')\n <BLANKLINE>\n ...Compressing...'...01.json' to...'tar'...in:...\n >>> compressed_paths\n (...Path('.../plaintext_fixture-000001.json.tar'),)\n ```\n \"\"\"\noutput_path = (\nPath(self.json_export_compression_subdir)\nif not output_path\nelse Path(output_path)\n)\nformat = self.json_export_compression_format if not format else format\ncompressed_paths: list[Path] = []\nfor json_path in self.exported_json_paths:\ncompressed_paths.append(\ncompress_fixture(json_path, output_path=output_path, format=format)\n)\nreturn tuple(compressed_paths)\ndelete_decompressed(ignore_errors: bool = True) -> None\nRemove all files in self.extract_path.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...'...bl_lwm...extracted'...\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in:...\n>>> plaintext_bl_lwm.delete_decompressed()\n<BLANKLINE>\n...Extract path empty:...'...bl_lwm...extracted'...\nalto2txt2fixture/plaintext.py def delete_decompressed(self, ignore_errors: bool = True) -> None:\n\"\"\"Remove all files in `self.extract_path`.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...'...bl_lwm...extracted'...\n >>> plaintext_bl_lwm.delete_decompressed()\n Deleting all files in:...\n >>> plaintext_bl_lwm.delete_decompressed()\n <BLANKLINE>\n ...Extract path empty:...'...bl_lwm...extracted'...\n ```\n \"\"\"\nif self.extract_path.exists():\nconsole.print(f\"Deleting all files in: '{self.extract_path}'\")\nrmtree(self.extract_path, ignore_errors=ignore_errors)\nelse:\nconsole.log(f\"Extract path empty: '{self.extract_path}'\")\nexport_to_json_fixtures(\noutput_path: PathLike | None = None,\nprefix: str | None = None,\njson_0_file_name_padding: int | None = None,\n) -> None\nIterate over self.plaintext_paths exporting to json django fixtures.
For errors running on windows see: #55
Parameters:
Name Type Description Defaultoutput_path PathLike | None Folder to save all json fixtures in.
None prefix str | None Any str prefix for saved fixture files.
None json_0_file_name_padding int | None Number of 0s to prefix file name numbering.
None Example >>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> bl_lwm: Path = getfixture(\"bl_lwm\")\n>>> first_lwm_plaintext_json_dict: PlaintextFixtureDict = (\n... getfixture(\"first_lwm_plaintext_json_dict\")\n... )\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...bl_lwm...extracted...\n>>> plaintext_bl_lwm.export_to_json_fixtures(output_path=bl_lwm / \"output\")\n<BLANKLINE>\nCompressed configs...%...[...]\n>>> len(plaintext_bl_lwm._exported_json_paths)\n1\n>>> plaintext_bl_lwm._exported_json_paths\n(...Path(...plaintext_fixture-000001.json...),)\n>>> import json\n>>> exported_json = json.loads(\n... plaintext_bl_lwm._exported_json_paths[0].read_text()\n... )\n>>> exported_json[0]['pk'] == first_lwm_plaintext_json_dict['pk']\nTrue\n>>> exported_json[0]['model'] == first_lwm_plaintext_json_dict['model']\nTrue\n>>> (exported_json[0]['fields']['text'] ==\n... first_lwm_plaintext_json_dict['fields']['text'])\nTrue\n>>> (exported_json[0]['fields']['path'] ==\n... str(first_lwm_plaintext_json_dict['fields']['path']))\nTrue\n>>> (exported_json[0]['fields']['compressed_path'] ==\n... str(first_lwm_plaintext_json_dict['fields']['compressed_path']))\nTrue\n>>> exported_json[0]['fields']['created_at']\n'20...'\n>>> (exported_json[0]['fields']['updated_at'] ==\n... exported_json[0]['fields']['updated_at'])\nTrue\nalto2txt2fixture/plaintext.py def export_to_json_fixtures(\nself,\noutput_path: PathLike | None = None,\nprefix: str | None = None,\njson_0_file_name_padding: int | None = None,\n) -> None:\n\"\"\"Iterate over `self.plaintext_paths` exporting to `json` `django` fixtures.\n Note:\n For errors running on windows see:\n [#55](https://github.com/Living-with-machines/alto2txt2fixture/issues/55)\n Args:\n output_path:\n Folder to save all `json` fixtures in.\n prefix:\n Any `str` prefix for saved fixture files.\n json_0_file_name_padding:\n Number of `0`s to prefix file name numbering.\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> bl_lwm: Path = getfixture(\"bl_lwm\")\n >>> first_lwm_plaintext_json_dict: PlaintextFixtureDict = (\n ... getfixture(\"first_lwm_plaintext_json_dict\")\n ... )\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...bl_lwm...extracted...\n >>> plaintext_bl_lwm.export_to_json_fixtures(output_path=bl_lwm / \"output\")\n <BLANKLINE>\n Compressed configs...%...[...]\n >>> len(plaintext_bl_lwm._exported_json_paths)\n 1\n >>> plaintext_bl_lwm._exported_json_paths\n (...Path(...plaintext_fixture-000001.json...),)\n >>> import json\n >>> exported_json = json.loads(\n ... plaintext_bl_lwm._exported_json_paths[0].read_text()\n ... )\n >>> exported_json[0]['pk'] == first_lwm_plaintext_json_dict['pk']\n True\n >>> exported_json[0]['model'] == first_lwm_plaintext_json_dict['model']\n True\n >>> (exported_json[0]['fields']['text'] ==\n ... first_lwm_plaintext_json_dict['fields']['text'])\n True\n >>> (exported_json[0]['fields']['path'] ==\n ... str(first_lwm_plaintext_json_dict['fields']['path']))\n True\n >>> (exported_json[0]['fields']['compressed_path'] ==\n ... str(first_lwm_plaintext_json_dict['fields']['compressed_path']))\n True\n >>> exported_json[0]['fields']['created_at']\n '20...'\n >>> (exported_json[0]['fields']['updated_at'] ==\n ... exported_json[0]['fields']['updated_at'])\n True\n ```\n \"\"\"\noutput_path = self.export_directory if not output_path else output_path\nprefix = self.saved_fixture_prefix if not prefix else prefix\njson_0_file_name_padding = (\nself.json_0_file_name_padding\nif not json_0_file_name_padding\nelse json_0_file_name_padding\n)\nsave_fixture(\nself.plaintext_paths_to_dicts(),\nprefix=prefix,\noutput_path=output_path,\nadd_created=True,\nmax_elements_per_file=self.max_plaintext_per_fixture_file,\nfile_name_0_padding=json_0_file_name_padding,\n)\nself.set_exported_json_paths(\nexport_directory=output_path, saved_fixture_prefix=prefix\n)\nextract_compressed() -> None\nExtract self.compressed_files to self.extracted_subdir_name.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n>>> plaintext_bl_lwm.extract_compressed()\n<BLANKLINE>\n...Extract path:...'...bl_lwm...extracted'...\n>>> filter_sect1_txt: list[str] = [txt_file for txt_file in\n... plaintext_bl_lwm._uncompressed_source_file_dict.keys()\n... if txt_file.name.endswith('204_sect0001.txt')]\n>>> len(filter_sect1_txt)\n1\n>>> plaintext_bl_lwm._uncompressed_source_file_dict[\n... filter_sect1_txt[0]\n... ]\n<BLANKLINE>\n...Path('...bl_lwm...0003079-test_plaintext.zip')\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in:...'...bl_lwm...tracted'\nalto2txt2fixture/plaintext.py def extract_compressed(self) -> None:\n\"\"\"Extract `self.compressed_files` to `self.extracted_subdir_name`.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n >>> plaintext_bl_lwm.extract_compressed()\n <BLANKLINE>\n ...Extract path:...'...bl_lwm...extracted'...\n >>> filter_sect1_txt: list[str] = [txt_file for txt_file in\n ... plaintext_bl_lwm._uncompressed_source_file_dict.keys()\n ... if txt_file.name.endswith('204_sect0001.txt')]\n >>> len(filter_sect1_txt)\n 1\n >>> plaintext_bl_lwm._uncompressed_source_file_dict[\n ... filter_sect1_txt[0]\n ... ]\n <BLANKLINE>\n ...Path('...bl_lwm...0003079-test_plaintext.zip')\n >>> plaintext_bl_lwm.delete_decompressed()\n Deleting all files in:...'...bl_lwm...tracted'\n ```\n \"\"\"\nself.extract_path.mkdir(parents=True, exist_ok=True)\nconsole.log(f\"Extract path: '{self.extract_path}'\")\nfor compressed_file in tqdm(\nself.compressed_files,\ntotal=len(self.compressed_files),\n):\nlogger.info(f\"Extracting: '{compressed_file}' ...\")\nunpack_archive(compressed_file, self.extract_path)\nfor path in sorted(self.extract_path.glob(self.plaintext_glob_regex)):\nif path not in self._uncompressed_source_file_dict:\nself._uncompressed_source_file_dict[path] = compressed_file\ninfo() -> None\nPrint self.info_table to the console.
alto2txt2fixture/plaintext.py def info(self) -> None:\n\"\"\"Print `self.info_table` to the `console`.\"\"\"\nconsole.print(self.info_table)\nplaintext_paths(reset_cache = False) -> Generator[FulltextPathDict, None, None]\nReturn a generator of all plaintext files for potential fixtures.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...bl_lwm...extracted...\n>>> plaintext_paths = plaintext_bl_lwm.plaintext_paths()\n>>> first_path_fixture_dict = next(iter(plaintext_paths))\n>>> first_path_fixture_dict['path'].name\n'0003079_18980107_art0001.txt'\n>>> first_path_fixture_dict['compressed_path'].name\n'0003079-test_plaintext.zip'\n>>> len(plaintext_bl_lwm._pk_plaintext_dict)\n1\n>>> plaintext_bl_lwm._pk_plaintext_dict[\n... first_path_fixture_dict['path']\n... ] # This demonstrates the `pk` begins from 1 following `SQL` standards\n1\nalto2txt2fixture/plaintext.py def plaintext_paths(\nself, reset_cache=False\n) -> Generator[FulltextPathDict, None, None]:\n\"\"\"Return a generator of all `plaintext` files for potential fixtures.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...bl_lwm...extracted...\n >>> plaintext_paths = plaintext_bl_lwm.plaintext_paths()\n >>> first_path_fixture_dict = next(iter(plaintext_paths))\n >>> first_path_fixture_dict['path'].name\n '0003079_18980107_art0001.txt'\n >>> first_path_fixture_dict['compressed_path'].name\n '0003079-test_plaintext.zip'\n >>> len(plaintext_bl_lwm._pk_plaintext_dict)\n 1\n >>> plaintext_bl_lwm._pk_plaintext_dict[\n ... first_path_fixture_dict['path']\n ... ] # This demonstrates the `pk` begins from 1 following `SQL` standards\n 1\n ```\n \"\"\"\nif self.compressed_files and not self.extract_path.exists():\nconsole.print(\n\"Compressed files not yet extracted. Try `extract_compression()`.\"\n)\nelse:\ni: int = 0\npk: int\nif self._uncompressed_source_file_dict:\nfor i, uncompressed_tuple in enumerate(\ntqdm(\nself._uncompressed_source_file_dict.items(),\ndesc=\"Compressed configs :\",\ntotal=len(self._uncompressed_source_file_dict),\n)\n):\npk = i + self.initial_pk # Most `SQL` `pk` begins at 1\nself._pk_plaintext_dict[uncompressed_tuple[0]] = pk\nyield FulltextPathDict(\npath=uncompressed_tuple[0],\ncompressed_path=uncompressed_tuple[1],\nprimary_key=pk,\n)\nif self.plaintext_provided_uncompressed:\nfor j, path in enumerate(\ntqdm(\nself.plaintext_provided_uncompressed,\ndesc=\"Uncompressed configs:\",\ntotal=len(self.plaintext_provided_uncompressed),\n)\n):\npk = j + i + self.initial_pk\nself._pk_plaintext_dict[path] = pk\nyield FulltextPathDict(\npath=path, compressed_path=None, primary_key=pk\n)\nplaintext_paths_to_dicts() -> Generator[PlaintextFixtureDict, None, None]\nGenerate fixture dicts from self.plaintext_paths.
For errors running on windows see: #55
Example>>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...bl_lwm...extracted...\n>>> paths_dict = list(plaintext_bl_lwm.plaintext_paths_to_dicts())\nCompressed configs :...%...[ ... it/s ]\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in: '...tracted'\nalto2txt2fixture/plaintext.py def plaintext_paths_to_dicts(self) -> Generator[PlaintextFixtureDict, None, None]:\n\"\"\"Generate fixture dicts from `self.plaintext_paths`.\n Note:\n For errors running on windows see:\n [#55](https://github.com/Living-with-machines/alto2txt2fixture/issues/55)\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...bl_lwm...extracted...\n >>> paths_dict = list(plaintext_bl_lwm.plaintext_paths_to_dicts())\n Compressed configs :...%...[ ... it/s ]\n >>> plaintext_bl_lwm.delete_decompressed()\n Deleting all files in: '...tracted'\n ```\n \"\"\"\ntext: str\nerror_str: str | None = None\nfor plaintext_path_dict in self.plaintext_paths():\nerror_str = None\ntry:\ntext = Path(plaintext_path_dict[\"path\"]).read_text()\nexcept UnicodeDecodeError as err:\nlogger.warning(err)\ntext = \"\"\nerror_str = str(err)\nfields: PlaintextFixtureFieldsDict = PlaintextFixtureFieldsDict(\ntext=text,\npath=str(plaintext_path_dict[\"path\"]),\ncompressed_path=str(plaintext_path_dict[\"compressed_path\"]),\nerrors=error_str,\n)\nyield PlaintextFixtureDict(\nmodel=self.model_str,\nfields=fields,\npk=plaintext_path_dict[\"primary_key\"],\n)\nset_exported_json_paths(\nexport_directory: PathLike | None,\nsaved_fixture_prefix: str | None,\noverwrite: bool = False,\n) -> None\nSet self._exported_json_paths for use with self.exported_json_paths.
If provided export_directory and saved_fixture_prefix will overwite those attributes on self.
Parameters:
Name Type Description Defaultexport_directory PathLike | None Path to check for saved json files.
saved_fixture_prefix str | None str to prefix each exported json file with.
overwrite bool Force replace self._exported_json_paths if already set.
False Example >>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> tmp_path = getfixture('tmp_path')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> tuple(plaintext_bl_lwm.exported_json_paths)\n(...Path('...plaintext_fixture-000001.json'),)\n>>> plaintext_bl_lwm.set_exported_json_paths(tmp_path, 'check-prefix')\nTraceback (most recent call last):\n...\nValueError: Cannot overwrite 'self._exported_json_paths' without\n'overwrite' = True. Current 'self._exported_json_paths':\n(...Path('...plaintext_fixture-000001.json'),)\n>>> plaintext_bl_lwm.set_exported_json_paths(tmp_path,\n... 'check-prefix', overwrite=True)\n<BLANKLINE>\n...Force change '._exported_json_paths' in...<PlainTextFixture...\n>>> plaintext_bl_lwm.export_directory == tmp_path\nTrue\n>>> plaintext_bl_lwm.saved_fixture_prefix\n'check-prefix'\nalto2txt2fixture/plaintext.py def set_exported_json_paths(\nself,\nexport_directory: PathLike | None,\nsaved_fixture_prefix: str | None,\noverwrite: bool = False,\n) -> None:\n\"\"\"Set `self._exported_json_paths` for use with `self.exported_json_paths`.\n Note:\n If provided `export_directory` and `saved_fixture_prefix` will\n overwite those attributes on `self.`\n Params:\n export_directory:\n `Path` to check for saved `json` files.\n saved_fixture_prefix:\n `str` to prefix each exported `json` file with.\n overwrite:\n Force replace `self._exported_json_paths` if already set.\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> tmp_path = getfixture('tmp_path')\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n <BLANKLINE>\n ...\n >>> tuple(plaintext_bl_lwm.exported_json_paths)\n (...Path('...plaintext_fixture-000001.json'),)\n >>> plaintext_bl_lwm.set_exported_json_paths(tmp_path, 'check-prefix')\n Traceback (most recent call last):\n ...\n ValueError: Cannot overwrite 'self._exported_json_paths' without\n 'overwrite' = True. Current 'self._exported_json_paths':\n (...Path('...plaintext_fixture-000001.json'),)\n >>> plaintext_bl_lwm.set_exported_json_paths(tmp_path,\n ... 'check-prefix', overwrite=True)\n <BLANKLINE>\n ...Force change '._exported_json_paths' in...<PlainTextFixture...\n >>> plaintext_bl_lwm.export_directory == tmp_path\n True\n >>> plaintext_bl_lwm.saved_fixture_prefix\n 'check-prefix'\n ```\n \"\"\"\nif hasattr(self, \"_exported_json_paths\"):\nif overwrite:\nlogger.info(f\"Force change '._exported_json_paths' in {repr(self)}\")\nelse:\nraise ValueError(\nf\"Cannot overwrite 'self._exported_json_paths' without \"\nf\"'overwrite' = True. Current 'self._exported_json_paths':\\n \"\nf\"{pformat(self._exported_json_paths)}\"\n)\nself.export_directory = (\nexport_directory if export_directory else self.export_directory\n)\nself.saved_fixture_prefix = (\nsaved_fixture_prefix if saved_fixture_prefix else self.saved_fixture_prefix\n)\nself._exported_json_paths = path_globs_to_tuple(\nself.export_directory, f\"**/{self.saved_fixture_prefix}*.json\"\n)\nArchive(\npath: str | Path,\ncollection: str = \"\",\nreport_id: str | None = None,\njisc_papers: pd.DataFrame | None = None,\njson_indent: int = JSON_INDENT,\n)\nManage extracting information from a ZIP archive.
The Archive class represents a zip archive of XML files. The class is used to extract information from a ZIP archive, and it contains several methods to process the data contained in the archive.
open(Archive) context manager
Archive can be opened with a context manager, which creates a meta object, with timings for the object. When closed, it will save the meta JSON to the correct paths.
Attributes:
Name Type Descriptionpath Path The path to the zip archive.
collection str The collection of the XML files in the archive. Default is \"\".
report Path The file path of the report file for the archive.
report_id str The report ID for the archive. If not provided, a random UUID is generated.
report_parent Path The parent directory of the report file for the archive.
jisc_papers pd.DataFrame A DataFrame of JISC papers.
size str | float The size of the archive, in human-readable format.
size_raw str | float The raw size of the archive, in bytes.
roots Generator[ET.Element, None, None] The root elements of the XML documents contained in the archive.
meta dotdict Metadata about the archive, such as its path, size, and number of contents.
json_indent int Indentation formatting of json output
Raises:
Type DescriptionRuntimeError If the path does not exist.
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\npath: str | Path,\ncollection: str = \"\",\nreport_id: str | None = None,\njisc_papers: pd.DataFrame | None = None,\njson_indent: int = JSON_INDENT,\n):\n\"\"\"Constructor method.\"\"\"\nself.path: Path = Path(path)\nif not self.path.exists():\nraise RuntimeError(\"Path does not exist.\")\nself.size: str | float = get_size_from_path(self.path)\nself.size_raw: str | float = get_size_from_path(self.path, raw=True)\nself.zip_file: zipfile.ZipFile = zipfile.ZipFile(self.path)\nself.collection: str = collection\nself.roots: Generator[ET.Element, None, None] = self.get_roots()\nself.meta: dotdict = dotdict(\npath=str(self.path),\nbytes=self.size_raw,\nsize=self.size,\ncontents=len(self.filelist),\n)\nif not report_id:\nself.report_id: str = str(uuid.uuid4())\nelse:\nself.report_id = report_id\nself.jisc_papers: pd.DataFrame = jisc_papers\nself.report_parent: Path = Path(f\"{REPORT_DIR}/{self.report_id}\")\nself.report: Path = (\nself.report_parent / f\"{self.path.stem.replace('_metadata', '')}.json\"\n)\nself.json_indent: int = json_indent\nproperty","text":"documents\nProperty that calls the get_documents method
property","text":"filelist\nReturns the list of files in the zip file
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Archive.__len__","title":"__len__","text":"__len__()\nThe number of files inside the zip archive.
Source code inalto2txt2fixture/router.py def __len__(self):\n\"\"\"The number of files inside the zip archive.\"\"\"\nreturn len(self.filelist)\nget_documents() -> Generator[Document, None, None]\nA generator that yields instances of the Document class for each XML file in the ZIP archive.
It uses the tqdm library to display a progress bar in the terminal while it is running.
If the contents of the ZIP file are not empty, the method creates an instance of the Document class by passing the root element of the XML file, the collection name, meta information about the archive, and the JISC papers data frame (if provided) to the constructor of the Document class. The instance of the Document class is then returned by the generator.
Yields:
Type DescriptionDocument Document class instance for each unzipped XML file.
alto2txt2fixture/router.py def get_documents(self) -> Generator[Document, None, None]:\n\"\"\"\n A generator that yields instances of the Document class for each XML\n file in the ZIP archive.\n It uses the `tqdm` library to display a progress bar in the terminal\n while it is running.\n If the contents of the ZIP file are not empty, the method creates an\n instance of the ``Document`` class by passing the root element of the XML\n file, the collection name, meta information about the archive, and the\n JISC papers data frame (if provided) to the constructor of the\n ``Document`` class. The instance of the ``Document`` class is then\n returned by the generator.\n Yields:\n ``Document`` class instance for each unzipped `XML` file.\n \"\"\"\nfor xml_file in tqdm(\nself.filelist,\ndesc=f\"{Path(self.zip_file.filename).stem} ({self.meta.size})\",\nleave=False,\ncolour=\"green\",\n):\nwith self.zip_file.open(xml_file) as f:\nxml = f.read()\nif xml:\nyield Document(\nroot=ET.fromstring(xml),\ncollection=self.collection,\nmeta=self.meta,\njisc_papers=self.jisc_papers,\n)\nget_roots() -> Generator[ET.Element, None, None]\nYields the root elements of the XML documents contained in the archive.
Source code inalto2txt2fixture/router.py def get_roots(self) -> Generator[ET.Element, None, None]:\n\"\"\"\n Yields the root elements of the XML documents contained in the archive.\n \"\"\"\nfor xml_file in tqdm(self.filelist, leave=False, colour=\"blue\"):\nwith self.zip_file.open(xml_file) as f:\nxml = f.read()\nif xml:\nyield ET.fromstring(xml)\nCache()\nThe Cache class provides a blueprint for creating and managing cache data. The class has several methods that help in getting the cache path, converting the data to a dictionary, and writing the cache data to a file.
It is inherited by many other classes in this document.
Initializes the Cache class object.
Source code inalto2txt2fixture/router.py def __init__(self):\n\"\"\"\n Initializes the Cache class object.\n \"\"\"\npass\n__str__() -> str\nReturns the string representation of the cache data as a dictionary.
Source code inalto2txt2fixture/router.py def __str__(self) -> str:\n\"\"\"\n Returns the string representation of the cache data as a dictionary.\n \"\"\"\nreturn str(self.as_dict())\nas_dict() -> dict\nConverts the cache data to a dictionary and returns it.
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n Converts the cache data to a dictionary and returns it.\n \"\"\"\nreturn {}\nget_cache_path() -> Path\nReturns the cache path, which is used to store the cache data. The path is normally constructed using some of the object's properties (collection, kind, and id) but can be changed when inherited.
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the cache path, which is used to store the cache data.\n The path is normally constructed using some of the object's\n properties (collection, kind, and id) but can be changed when\n inherited.\n \"\"\"\nreturn Path(f\"{CACHE_HOME}/{self.collection}/{self.kind}/{self.id}.json\")\nwrite_to_cache(json_indent: int = JSON_INDENT) -> Optional[bool]\nWrites the cache data to a file at the specified cache path. The cache data is first converted to a dictionary using the as_dict method. If the cache path already exists, the function returns True.
Source code inalto2txt2fixture/router.py def write_to_cache(self, json_indent: int = JSON_INDENT) -> Optional[bool]:\n\"\"\"\n Writes the cache data to a file at the specified cache path. The cache\n data is first converted to a dictionary using the as_dict method. If\n the cache path already exists, the function returns True.\n \"\"\"\npath = self.get_cache_path()\ntry:\nif path.exists():\nreturn True\nexcept AttributeError:\nerror(\nf\"Error occurred when getting cache path for \"\nf\"{self.kind}: {path}. It was not of expected \"\nf\"type Path but of type {type(path)}:\",\n)\npath.parent.mkdir(parents=True, exist_ok=True)\nwith open(path, \"w+\") as f:\nf.write(json.dumps(self.as_dict(), indent=json_indent))\nreturn\nCollection(name: str = 'hmd', jisc_papers: Optional[pd.DataFrame] = None)\nA Collection represents a group of newspaper archives from any passed alto2txt metadata output.
A Collection is initialised with a name and an optional pandas DataFrame of JISC papers. The archives property returns an iterable of the Archive objects within the collection.
Attributes:
Name Type Descriptionname str Name of the collection (default \"hmd\")
jisc_papers pandas.DataFrame DataFrame of JISC papers, optional
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, name: str = \"hmd\", jisc_papers: Optional[pd.DataFrame] = None):\n\"\"\"Constructor method.\"\"\"\nself.name: str = name\nself.jisc_papers: pd.DataFrame | None = jisc_papers\nself.dir: Path = Path(f\"{MNT}/{self.name}-alto2txt/metadata\")\nself.zip_files: list[Path] = sorted(\nlist(self.dir.glob(\"*.zip\")), key=lambda x: x.stat().st_size\n)\nself.zip_file_count: int = sum([1 for _ in self.dir.glob(\"*.zip\")])\nself.report_id: str = str(uuid.uuid4())\nself.empty: bool = self.zip_file_count == 0\nDataProvider(collection: str)\n Bases: Cache
The DataProvider class extends the Cache class and represents a newspaper data provider. The class has several properties and methods that allow creation of a data provider object and the manipulation of its data.
Attributes:
Name Type Descriptioncollection str A string representing publication collection
kind str Indication of object type, defaults to data-provider
providers_meta_data list[FixtureDict] structured dict of metadata for known collection sources
collection_type str related data sources and potential linkage source
index_field str field name for querying existing records
Example>>> from pprint import pprint\n>>> hmd = DataProvider(\"hmd\")\n>>> hmd.pk\n2\n>>> pprint(hmd.as_dict())\n{'code': 'bl_hmd',\n 'collection': 'newspapers',\n 'legacy_code': 'hmd',\n 'name': 'Heritage Made Digital',\n 'source_note': 'British Library-funded digitised newspapers provided by the '\n 'British Newspaper Archive'}\nConstructor method.
Source code inalto2txt2fixture/router.py def __init__(self, collection: str):\n\"\"\"Constructor method.\"\"\"\nself.collection: str = collection\nproperty","text":"meta_data: FixtureDict | dict\nReturn self.providers_meta_data[self.collection] or {}.
property","text":"meta_data_fields: FixtureDict | dict\nReturn self.providers_meta_data[self.collection] or {}.
property","text":"pk: int | None\nReturn pk if provided via providers_meta_data, else None.
property","text":"providers_index_dict: dict[str, FixtureDict]\nReturn all self.index_field values from providers_meta_data.
as_dict() -> dict\nReturn a dict of the data provider object.
Returns:
Type Descriptiondict Dictionary representation of the DataProvider object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n Return a `dict` of the data provider object.\n Returns:\n Dictionary representation of the DataProvider object\n \"\"\"\nif self.meta_data:\nreturn {\n\"name\": self.meta_data_fields[\"name\"],\n\"code\": self.meta_data_fields[\"code\"],\n\"legacy_code\": self.collection,\n\"source_note\": self.meta_data_fields[\"source_note\"],\n\"collection\": self.collection_type,\n}\nelse:\nreturn {\n\"name\": self.collection,\n\"code\": slugify(self.collection),\n\"source_note\": \"\",\n\"legacy_code\": None,\n\"collection\": self.collection_type,\n}\nDigitisation(root: ET.Element, collection: str = '')\n Bases: Cache
The Digitisation class extends the Cache class and represents a newspaper digitisation. The class has several properties and methods that allow creation of an digitisation object and the manipulation of its data.
Attributes:
Name Type Descriptionroot ET.Element An xml element that represents the root of the publication
collection str A string that represents the collection of the publication
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, root: ET.Element, collection: str = \"\"):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nself.root: ET.Element = root\nself.collection: str = collection\nclass-attribute instance-attribute","text":"kind = 'digitisation'\nA string that represents the type of the object, set to \"digitisation\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Digitisation.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the digitisation object.
Returns:
Type Descriptiondict Dictionary representation of the Digitising object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the digitisation\n object.\n Returns:\n Dictionary representation of the Digitising object\n \"\"\"\ndic = {\nx.tag: x.text or \"\"\nfor x in self.root.findall(\"./process/*\")\nif x.tag\nin [\n\"xml_flavour\",\n\"software\",\n\"mets_namespace\",\n\"alto_namespace\",\n]\n}\nif not dic.get(\"software\"):\nreturn {}\nreturn dic\nDocument(*args, **kwargs)\nThe Document class is a representation of a document that contains information about a publication, newspaper, item, digitisation, and ingest. This class holds all the relevant information about a document in a structured manner and provides properties that can be used to access different aspects of the document.
Attributes:
Name Type Descriptioncollection str | None A string that represents the collection of the publication
root ET.Element | None An XML element that represents the root of the publication
zip_file str | None A path to a valid zip file
jisc_papers pd.DataFrame | None A pandas DataFrame object that holds information about the JISC papers
meta dotdict | None TODO
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, *args, **kwargs):\n\"\"\"Constructor method.\"\"\"\nself.collection: str | None = kwargs.get(\"collection\")\nif not self.collection or not isinstance(self.collection, str):\nraise RuntimeError(\"A valid collection must be passed\")\nself.root: ET.Element | None = kwargs.get(\"root\")\nif not self.root or not isinstance(self.root, ET.Element):\nraise RuntimeError(\"A valid XML root must be passed\")\nself.zip_file: str | None = kwargs.get(\"zip_file\")\nif self.zip_file and not isinstance(self.zip_file, str):\nraise RuntimeError(\"A valid zip file must be passed\")\nself.jisc_papers: pd.DataFrame | None = kwargs.get(\"jisc_papers\")\nif not isinstance(self.jisc_papers, pd.DataFrame):\nraise RuntimeError(\n\"A valid DataFrame containing JISC papers must be passed\"\n)\nself.meta: dotdict | None = kwargs.get(\"meta\")\nself._publication_elem = None\nself._input_sub_path = None\nself._ingest = None\nself._digitisation = None\nself._item = None\nself._issue = None\nself._newspaper = None\nself._data_provider = None\nproperty","text":"publication: ET.Element\nThis property returns an ElementTree object representing the publication information in the XML document.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Ingest","title":"Ingest","text":"Ingest(root: ET.Element, collection: str = '')\n Bases: Cache
The Ingest class extends the Cache class and represents a newspaper ingest. The class has several properties and methods that allow the creation of an ingest object and the manipulation of its data.
Attributes:
Name Type Descriptionroot ET.Element An xml element that represents the root of the publication
collection str A string that represents the collection of the publication
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, root: ET.Element, collection: str = \"\"):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nself.root: ET.Element = root\nself.collection: str = collection\nclass-attribute instance-attribute","text":"kind = 'ingest'\nA string that represents the type of the object, set to \"ingest\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Ingest.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the ingest object.
Returns:
Type Descriptiondict Dictionary representation of the Ingest object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the ingest\n object.\n Returns:\n Dictionary representation of the Ingest object\n \"\"\"\nreturn {\nf\"lwm_tool_{x.tag}\": x.text or \"\"\nfor x in self.root.findall(\"./process/lwm_tool/*\")\n}\nIssue(\npublication: ET.Element,\nnewspaper: Optional[Newspaper] = None,\ncollection: str = \"\",\ninput_sub_path: str = \"\",\nmeta: dotdict = dotdict(),\n)\n Bases: Cache
The Issue class extends the Cache class and represents a newspaper issue. The class has several properties and methods that allow the creation of an issue object and the manipulation of its data.
Attributes:
Name Type Descriptionroot An xml element that represents the root of the publication
newspaper Newspaper | None The parent newspaper
collection str A string that represents the collection of the publication
input_sub_path str TODO
meta dotdict TODO
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\npublication: ET.Element,\nnewspaper: Optional[Newspaper] = None,\ncollection: str = \"\",\ninput_sub_path: str = \"\",\nmeta: dotdict = dotdict(),\n):\n\"\"\"Constructor method.\"\"\"\nself.publication: ET.Element = publication\nself.newspaper: Newspaper | None = newspaper\nself.collection: str = collection\nself.input_sub_path: str = input_sub_path\nself.meta: dotdict = meta\nself._issue = None\nself._issue_date = None\npath: str = str(self.get_cache_path())\nif not self.meta.issue_paths:\nself.meta.issue_paths = [path]\nelif path not in self.meta.issue_paths:\nself.meta.issue_paths.append(path)\nproperty","text":"issue_code: str\nSets up and saves the issue code for easy access as property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Issue.issue_date","title":"issue_dateproperty","text":"issue_date: str\nSets up and saves the issue date for easy access as property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Issue.kind","title":"kindclass-attribute instance-attribute","text":"kind = 'issue'\nA string that represents the type of the object, set to \"issue\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Issue.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the issue object.
Returns:
Type Descriptiondict Dictionary representation of the Issue object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the issue\n object.\n Returns:\n Dictionary representation of the Issue object\n \"\"\"\nif not self._issue:\nself._issue = dict(\nissue_code=self.issue_code,\nissue_date=self.issue_date,\npublication__publication_code=self.newspaper.publication_code,\ninput_sub_path=self.input_sub_path,\n)\nreturn self._issue\nget_cache_path() -> Path\nReturns the path to the cache file for the issue object.
Returns:
Type DescriptionPath Path to the cache file for the issue object
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the path to the cache file for the issue object.\n Returns:\n Path to the cache file for the issue object\n \"\"\"\njson_file = f\"/{self.newspaper.publication_code}/issues/{self.issue_code}.json\"\nreturn Path(\nf\"{CACHE_HOME}/{self.collection}/\"\n+ \"/\".join(self.newspaper.number_paths)\n+ json_file\n)\nItem(\nroot: ET.Element,\nissue_code: str = \"\",\ndigitisation: dict = {},\ningest: dict = {},\ncollection: str = \"\",\nnewspaper: Optional[Newspaper] = None,\nmeta: dotdict = dotdict(),\n)\n Bases: Cache
The Newspaper class extends the Cache class and represents a newspaper item, i.e. an article. The class has several properties and methods that allow the creation of an article object and the manipulation of its data.
Attributes:
Name Type Descriptionroot ET.Element An xml element that represents the root of the publication
issue_code str A string that represents the issue code
digitisation dict TODO
ingest dict TODO
collection str A string that represents the collection of the publication
newspaper Newspaper | None The parent newspaper
meta dotdict TODO
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\nroot: ET.Element,\nissue_code: str = \"\",\ndigitisation: dict = {},\ningest: dict = {},\ncollection: str = \"\",\nnewspaper: Optional[Newspaper] = None,\nmeta: dotdict = dotdict(),\n):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nif not isinstance(newspaper, Newspaper):\nraise RuntimeError(\"Expected newspaper to be of type router.Newspaper\")\nself.root: ET.Element = root\nself.issue_code: str = issue_code\nself.digitisation: dict = digitisation\nself.ingest: dict = ingest\nself.collection: str = collection\nself.newspaper: Newspaper | None = newspaper\nself.meta: dotdict = meta\nself._item_elem = None\nself._item_code = None\nself._item = None\npath: str = str(self.get_cache_path())\nif not self.meta.item_paths:\nself.meta.item_paths = [path]\nelif path not in self.meta.item_paths:\nself.meta.item_paths.append(path)\nproperty","text":"item_code: str\nSets up and saves the item code for easy access as property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Item.item_elem","title":"item_elemproperty","text":"item_elem\nSets up and saves the issue XML item for easy access as a property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Item.kind","title":"kindclass-attribute instance-attribute","text":"kind = 'item'\nA string that represents the type of the object, set to \"item\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Item.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the item object (i.e. article).
Returns:
Type Descriptiondict Dictionary representation of the Item object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the item object\n (i.e. article).\n Returns:\n Dictionary representation of the Item object\n \"\"\"\nif not self._item:\nself._item = {\nf\"{x.tag}\": x.text or \"\"\nfor x in self.item_elem.findall(\"*\")\nif x.tag\nin [\n\"title\",\n\"word_count\",\n\"ocr_quality_mean\",\n\"ocr_quality_sd\",\n\"plain_text_file\",\n\"item_type\",\n]\n}\nself._item[\"title\"] = self._item.get(\"title\", \"\")[:2097151]\nself._item = {\n\"item_code\": self.item_code,\n\"word_count\": self._item.get(\"word_count\", 0),\n\"title\": self._item.get(\"title\"),\n\"item_type\": self._item.get(\"item_type\"),\n\"input_filename\": self._item.get(\"plain_text_file\", \"\"),\n\"ocr_quality_mean\": self._item.get(\"ocr_quality_mean\", 0),\n\"ocr_quality_sd\": self._item.get(\"ocr_quality_sd\", 0),\n\"digitisation__software\": self.digitisation.id,\n\"ingest__lwm_tool_identifier\": self.ingest.id,\n\"issue__issue_identifier\": self.issue_code,\n\"data_provider__name\": self.collection,\n}\nreturn self._item\nget_cache_path() -> Path\nReturns the path to the cache file for the item (article) object.
Returns:
Type DescriptionPath Path to the cache file for the article object
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the path to the cache file for the item (article) object.\n Returns:\n Path to the cache file for the article object\n \"\"\"\nreturn Path(\nf\"{CACHE_HOME}/{self.collection}/\"\n+ \"/\".join(self.newspaper.number_paths)\n+ f\"/{self.newspaper.publication_code}/items.jsonl\"\n)\nwrite_to_cache(json_indent = JSON_INDENT) -> None\nSpecial cache-write function that appends rather than writes at the end of the process.
Returns:
Type DescriptionNone None.
Source code inalto2txt2fixture/router.py def write_to_cache(self, json_indent=JSON_INDENT) -> None:\n\"\"\"\n Special cache-write function that appends rather than writes at the\n end of the process.\n Returns:\n None.\n \"\"\"\npath = self.get_cache_path()\npath.parent.mkdir(parents=True, exist_ok=True)\nwith open(path, \"a+\") as f:\nf.write(json.dumps(self.as_dict(), indent=json_indent) + \"\\n\")\nreturn\nNewspaper(\nroot: ET.Element,\ncollection: str = \"\",\nmeta: dotdict = dotdict(),\njisc_papers: Optional[pd.DataFrame] = None,\n)\n Bases: Cache
The Newspaper class extends the Cache class and represents a newspaper.
The class has several properties and methods that allow the creation of a newspaper object and the manipulation of its data.
Attributes:
Name Type Descriptionroot An xml element that represents the root of the publication.
collection A string that represents the collection of the publication.
meta A dotdict object that holds metadata about the publication.
jisc_papers A pandas DataFrame object for JISC paper information.
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\nroot: ET.Element,\ncollection: str = \"\",\nmeta: dotdict = dotdict(),\njisc_papers: Optional[pd.DataFrame] = None,\n):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nself.publication = root.find(\"./publication\")\nself.input_sub_path = root.find(\"./process/input_sub_path\").text\nself.issue_date = self.publication.find(\"./issue/date\").text\nself.collection = collection\nself.meta = meta\nself.jisc_papers = jisc_papers\nself._newspaper = None\nself._title = None\nself._publication_code = None\npath = str(self.get_cache_path())\nif not self.meta.newspaper_paths:\nself.meta.newspaper_paths = []\nelif path not in self.meta.newspaper_paths:\nself.meta.newspaper_paths.append(path)\nif not self.meta.publication_codes:\nself.meta.publication_codes = [self.publication_code]\nelif self.publication_code not in self.meta.publication_codes:\nself.meta.publication_codes.append(self.publication_code)\nself.zip_file = Path(meta.path).name\nclass-attribute instance-attribute","text":"kind = 'newspaper'\nA string that represents the type of the object, set to \"newspaper\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.number_paths","title":"number_pathsproperty","text":"number_paths: list\nReturns the nested directories in which we want to save the cache file.
Returns:
Type Descriptionlist List of the desired directories in descending order
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.publication_code","title":"publication_codeproperty","text":"publication_code: str\nA property that returns the code of the publication.
Returns:
Type Descriptionstr The code of the publication
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.title","title":"titleproperty","text":"title: str\nA property that returns the title of the newspaper.
Returns:
Type Descriptionstr The title of the newspaper
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the newspaper object.
Returns:
Type Descriptiondict Dictionary representation of the Newspaper object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the newspaper\n object.\n Returns:\n Dictionary representation of the Newspaper object\n \"\"\"\nif not self._newspaper:\nself._newspaper = dict(\n**dict(publication_code=self.publication_code, title=self.title),\n**{\nx.tag: x.text or \"\"\nfor x in self.publication.findall(\"*\")\nif x.tag in [\"location\"]\n},\n)\nreturn self._newspaper\nget_cache_path() -> Path\nReturns the path to the cache file for the newspaper object.
Returns:
Type DescriptionPath Path to the cache file for the newspaper object
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the path to the cache file for the newspaper object.\n Returns:\n Path to the cache file for the newspaper object\n \"\"\"\njson_file = f\"/{self.publication_code}/{self.publication_code}.json\"\nreturn Path(\nf\"{CACHE_HOME}/{self.collection}/\" + \"/\".join(self.number_paths) + json_file\n)\npublication_code_from_input_sub_path() -> str | None\nA method that returns the publication code from the input sub-path of the publication process.
Returns:
Type Descriptionstr | None The code of the publication
Source code inalto2txt2fixture/router.py def publication_code_from_input_sub_path(self) -> str | None:\n\"\"\"\n A method that returns the publication code from the input sub-path of\n the publication process.\n Returns:\n The code of the publication\n \"\"\"\ng = PUBLICATION_CODE.findall(self.input_sub_path)\nif len(g) == 1:\nreturn g[0]\nreturn None\nroute(\ncollections: list,\ncache_home: str,\nmountpoint: str,\njisc_papers_path: str,\nreport_dir: str,\n) -> None\nThis function is responsible for setting up the path for the alto2txt mountpoint, setting up the JISC papers and routing the collections for processing.
Parameters:
Name Type Description Defaultcollections list List of collection names
requiredcache_home str Directory path for the cache
requiredmountpoint str Directory path for the alto2txt mountpoint
requiredjisc_papers_path str Path to the JISC papers
requiredreport_dir str Path to the report directory
requiredReturns:
Type DescriptionNone None
Source code inalto2txt2fixture/router.py def route(\ncollections: list,\ncache_home: str,\nmountpoint: str,\njisc_papers_path: str,\nreport_dir: str,\n) -> None:\n\"\"\"\n This function is responsible for setting up the path for the alto2txt\n mountpoint, setting up the JISC papers and routing the collections for\n processing.\n Args:\n collections: List of collection names\n cache_home: Directory path for the cache\n mountpoint: Directory path for the alto2txt mountpoint\n jisc_papers_path: Path to the JISC papers\n report_dir: Path to the report directory\n Returns:\n None\n \"\"\"\nglobal CACHE_HOME\nglobal MNT\nglobal REPORT_DIR\nCACHE_HOME = cache_home\nREPORT_DIR = report_dir\nMNT = Path(mountpoint) if isinstance(mountpoint, str) else mountpoint\nif not MNT.exists():\nerror(\nf\"The mountpoint provided for alto2txt does not exist. \"\nf\"Either create a local copy or blobfuse it to \"\nf\"`{MNT.absolute()}`.\"\n)\njisc_papers = setup_jisc_papers(path=jisc_papers_path)\nfor collection_name in collections:\ncollection = Collection(name=collection_name, jisc_papers=jisc_papers)\nif collection.empty:\nerror(\nf\"It looks like {collection_name} is empty in the \"\nf\"alto2txt mountpoint: `{collection.dir.absolute()}`.\"\n)\nfor archive in collection.archives:\nwith archive as _:\n[\n(\ndoc.item.write_to_cache(),\ndoc.newspaper.write_to_cache(),\ndoc.issue.write_to_cache(),\ndoc.data_provider.write_to_cache(),\ndoc.ingest.write_to_cache(),\ndoc.digitisation.write_to_cache(),\n)\nfor doc in archive.documents\n]\nreturn\nThe settings module provides configuration for running alto2txt2fixture.
Most of these are managed within the settings variable within this module.
Note
See the command line interface parameters documentation for means of modifying settings when run.
Attributes:
Name Type DescriptionJSON_INDEX Amount of indentation to include in output JSON files
DATA_PROVIDER_INDEX Final[str] The field used to index DataProvider records
NEWSPAPER_COLLECTION_METADATA Final[list[DataProviderFixtureDict]] A list of FixtureDicts specifying speific newspaper data providers
SETUP_TITLE str the title printed at the commandline via cli.show_setup() function
settings dotdict a docdict configuration for running newspaper portions of alto2txt2fixture
Bases: TypedDict
Fields within the fields portion of a FixtureDict to fit lwmdb.
Attributes:
Name Type Descriptionname str The name of the collection data source. For lwmdb this should be less than 600 characters.
code str | NEWSPAPER_OCR_FORMATS A short slug-like, url-compatible (replace spaces with -) str to uniquely identify a data provider in urls, api calls etc. Designed to fit NEWSPAPER_OCR_FORMATS and any future slug-like codes.
legacy_code LEGACY_NEWSPAPER_OCR_FORMATS | None Either blank or a legacy slug-like, url-compatible (replace spaces with -) str originally used by alto2txt following LEGACY_NEWSPAPER_OCR_FORMATSNEWSPAPER_OCR_FORMATS.
collection str Data Provider type.
source_note str | None A sentence about the data provider.
"},{"location":"reference/alto2txt2fixture/types.html#alto2txt2fixture.types.DataProviderFixtureDict","title":"DataProviderFixtureDict","text":" Bases: FixtureDictBaseClass
A dict structure for DataProvider sources in line with lwmdb.
Attributes:
Name Type Descriptionpk int an id to uniquely define and query each entry
model str what model a given record is for
fields DataProviderFieldsDict a DataProviderFieldsDict
Bases: FixtureDictBaseClass
A dict structure to ease use as a json database fixture.
Attributes:
Name Type Descriptionpk an id to uniquely define and query each entry
model what model a given record is for
fields dict[str, Any] a dict of record information conforming to model table
Bases: TypedDict
A base dict structure for json fixtures.
Bases: FixtureDictBaseClass
A dict structure for Fulltext sources in line with lwmdb.
Attributes:
Name Type Descriptionmodel str str in django fixture spec to indicate what model a record is for
fields PlaintextFixtureFieldsDict a PlaintextFixtureFieldsDict dict instance
pk int int id for fixture record
No pk is included. By not specifying one, django should generate new onces during import.
Bases: TypedDict
A typed dict for Plaintext Fixutres to match lwmdb.Fulltext model
Attributes:
Name Type Descriptiontext str Plaintext, potentially quite large newspaper articles. May have unusual or unreadable sequences of characters due to issues with Optical Character Recognition quality.
path str Path of provided plaintext file. If compressed_path is None, this is the original relative Path of the plaintext file.
compressed_path str | None The path of a compressed data source, the extraction of which provides access to plaintext files.
Bases: NamedTuple
A named tuple of fields for translation.
Attributes:
Name Type Descriptionstart str A string representing the starting field name.
finish str | list A string or list specifying the field(s) to be translated. If it is a string, the translated field will be a direct mapping of the specified field in each item of the input list. If it is a list, the translated field will be a hyphen-separated concatenation of the specified fields in each item of the input list.
lst list[dict] A list of dictionaries representing the items to be translated. Each dictionary should contain the necessary fields for translation, with the field names specified in the start parameter.
Bases: dict
dot.notation access to dictionary attributes
"},{"location":"reference/alto2txt2fixture/utils.html","title":"utils","text":""},{"location":"reference/alto2txt2fixture/utils.html#alto2txt2fixture.utils.DiskUsageTuple","title":"DiskUsageTuple","text":" Bases: NamedTuple
Type hint for nametuple returned from disk_usage.
check_newspaper_collection_configuration(\ncollections: Iterable[str] = settings.COLLECTIONS,\nnewspaper_collections: Iterable[\nFixtureDict\n] = NEWSPAPER_COLLECTION_METADATA,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> set[str]\nCheck the names in collections match the names in newspaper_collections.
Parameters:
Name Type Description Defaultcollections Iterable[str] Names of newspaper collections, defaults to settings.COLLECTIONS
settings.COLLECTIONS newspaper_collections Iterable[FixtureDict] Newspaper collections in a list of FixtureDict format. Defaults to settings.FIXTURE_TABLE['dataprovider]
NEWSPAPER_COLLECTION_METADATA data_provider_index str dict fields key used to check matchiching collections name
DATA_PROVIDER_INDEX Returns:
Type Descriptionset[str] A set of collections without a matching newspaper_collections record.
>>> check_newspaper_collection_configuration()\nset()\n>>> unmatched: set[str] = check_newspaper_collection_configuration(\n... [\"cat\", \"dog\"])\n<BLANKLINE>\n...Warning: 2 `collections` not in `newspaper_collections`: ...\n>>> unmatched == {'dog', 'cat'}\nTrue\nNote
Set orders are random so checking unmatched == {'dog, 'cat'} to ensure correctness irrespective of order in the example above.
alto2txt2fixture/utils.py def check_newspaper_collection_configuration(\ncollections: Iterable[str] = settings.COLLECTIONS,\nnewspaper_collections: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> set[str]:\n\"\"\"Check the names in `collections` match the names in `newspaper_collections`.\n Arguments:\n collections:\n Names of newspaper collections, defaults to ``settings.COLLECTIONS``\n newspaper_collections:\n Newspaper collections in a list of `FixtureDict` format. Defaults\n to ``settings.FIXTURE_TABLE['dataprovider]``\n data_provider_index:\n `dict` `fields` `key` used to check matchiching `collections` name\n Returns:\n A set of ``collections`` without a matching `newspaper_collections` record.\n Example:\n ```pycon\n >>> check_newspaper_collection_configuration()\n set()\n >>> unmatched: set[str] = check_newspaper_collection_configuration(\n ... [\"cat\", \"dog\"])\n <BLANKLINE>\n ...Warning: 2 `collections` not in `newspaper_collections`: ...\n >>> unmatched == {'dog', 'cat'}\n True\n ```\n !!! note\n Set orders are random so checking `unmatched == {'dog, 'cat'}` to\n ensure correctness irrespective of order in the example above.\n \"\"\"\nnewspaper_collection_names: tuple[str, ...] = tuple(\ndict_from_list_fixture_fields(\nnewspaper_collections, field_name=data_provider_index\n).keys()\n)\ncollection_diff: set[str] = set(collections) - set(newspaper_collection_names)\nif collection_diff:\nwarning(\nf\"{len(collection_diff)} `collections` \"\nf\"not in `newspaper_collections`: {collection_diff}\"\n)\nreturn collection_diff\nclear_cache(dir: str | Path) -> None\nClears the cache directory by removing all .json files in it.
Parameters:
Name Type Description Defaultdir str | Path The path of the directory to be cleared.
required Source code inalto2txt2fixture/utils.py def clear_cache(dir: str | Path) -> None:\n\"\"\"\n Clears the cache directory by removing all `.json` files in it.\n Args:\n dir: The path of the directory to be cleared.\n \"\"\"\ndir = get_path_from(dir)\ny = input(\nf\"Do you want to erase the cache path now that the \"\nf\"files have been generated ({dir.absolute()})? [y/N]\"\n)\nif y.lower() == \"y\":\ninfo(\"Clearing up the cache directory\")\nfor x in dir.glob(\"*.json\"):\nx.unlink()\ncompress_fixture(\npath: PathLike,\noutput_path: PathLike | str = settings.OUTPUT,\nsuffix: str = \"\",\nformat: str | ArchiveFormatEnum = ZIP_FILE_EXTENSION,\nforce_overwrite: bool = False,\ndry_run: bool = False,\n) -> Path\nCompress exported fixtures files using make_archive.
Parameters:
Name Type Description Defaultpath PathLike Path to file to compress
output_path PathLike | str Compressed file name (without extension specified from format).
settings.OUTPUT format str | ArchiveFormatEnum A str of one of the registered compression formats. By default Python provides zip, tar, gztar, bztar, and xztar. See ArchiveFormatEnum variable for options checked.
ZIP_FILE_EXTENSION suffix str str to add to comprssed file name saved. For example: if path = plaintext_fixture-1.json and suffix=_compressed, then the saved file might be called plaintext_fixture-1_compressed.json.zip
'' Example >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> tmp_path = getfixture('tmp_path')\n>>> json_path: Path = next(plaintext_bl_lwm.exported_json_paths)\n>>> assert 'pytest-of' in str(json_path)\n>>> compressed_path: Path = compress_fixture(path=json_path,\n... output_path=tmp_path,\n... dry_run=True)\n<BLANKLINE>\n...creating...'...plain...-...01.json.zip...'...addin...\n...'plain...01.json'...to...it...\n>>> compressed_path.exists()\nFalse\n>>> compressed_path: Path = compress_fixture(path=json_path,\n... output_path=tmp_path,\n... dry_run=False)\n<BLANKLINE>\n...creating...'...plain...-...01.json.zip...'...addin...\n...'plain...01.json'...to...it...\n>>> from zipfile import ZipFile, ZipInfo\n>>> zipfile_info_list: list[ZipInfo] = ZipFile(\n... tmp_path / 'plaintext_fixture-000001.json.zip'\n... ).infolist()\n>>> len(zipfile_info_list)\n1\n>>> Path(zipfile_info_list[0].filename).name\n'plaintext_fixture-000001.json'\nalto2txt2fixture/utils.py def compress_fixture(\npath: PathLike,\noutput_path: PathLike | str = settings.OUTPUT,\nsuffix: str = \"\",\nformat: str | ArchiveFormatEnum = ZIP_FILE_EXTENSION,\n# base_dir: PathLike | None = None,\nforce_overwrite: bool = False,\ndry_run: bool = False,\n) -> Path:\n\"\"\"Compress exported `fixtures` files using `make_archive`.\n Args:\n path:\n `Path` to file to compress\n output_path:\n Compressed file name (without extension specified from `format`).\n format:\n A `str` of one of the registered compression formats. By default\n `Python` provides `zip`, `tar`, `gztar`, `bztar`, and `xztar`.\n See `ArchiveFormatEnum` variable for options checked.\n suffix:\n `str` to add to comprssed file name saved.\n For example: if `path = plaintext_fixture-1.json` and\n `suffix=_compressed`, then the saved file might be called\n `plaintext_fixture-1_compressed.json.zip`\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n <BLANKLINE>\n ...\n >>> tmp_path = getfixture('tmp_path')\n >>> json_path: Path = next(plaintext_bl_lwm.exported_json_paths)\n >>> assert 'pytest-of' in str(json_path)\n >>> compressed_path: Path = compress_fixture(path=json_path,\n ... output_path=tmp_path,\n ... dry_run=True)\n <BLANKLINE>\n ...creating...'...plain...-...01.json.zip...'...addin...\n ...'plain...01.json'...to...it...\n >>> compressed_path.exists()\n False\n >>> compressed_path: Path = compress_fixture(path=json_path,\n ... output_path=tmp_path,\n ... dry_run=False)\n <BLANKLINE>\n ...creating...'...plain...-...01.json.zip...'...addin...\n ...'plain...01.json'...to...it...\n >>> from zipfile import ZipFile, ZipInfo\n >>> zipfile_info_list: list[ZipInfo] = ZipFile(\n ... tmp_path / 'plaintext_fixture-000001.json.zip'\n ... ).infolist()\n >>> len(zipfile_info_list)\n 1\n >>> Path(zipfile_info_list[0].filename).name\n 'plaintext_fixture-000001.json'\n ```\n \"\"\"\npath = Path(path)\ncurrent_dir: Path = Path()\nroot_dir: str | None = None\nbase_dir: str | None = None\nif not path.exists():\nraise ValueError(f\"Cannot compress not existent 'path': {path}\")\nif isinstance(format, str):\ntry:\nformat = ArchiveFormatEnum(format)\nexcept ValueError:\nraise ValueError(\nf\"format '{format}' not valid, \"\nf\"options are:'\\n{pformat(ARCHIVE_FORMATS)}\"\n)\nchdir(str(Path(path).parent))\nif path.is_file():\nroot_dir = str(Path(path).parent)\nbase_dir = path.name\n# chdir(str(Path(path).parent))\nelif path.is_dir():\n# chdir(str(path))\nroot_dir = path.name\nelse:\nraise ValueError(f\"Path type {type(path)} is not supported.\")\nsave_file_name: Path = Path(Path(path).stem + suffix + \"\".join(Path(path).suffixes))\nsave_path: Path = Path(output_path) / save_file_name\n# root_dir: Path = save_path.parent\nif Path(str(save_path) + f\".{format}\").exists():\nerror_message: str = f\"Path to save to already exists: '{save_path}'\"\nif force_overwrite:\nlogger.warn(error_message)\nlogger.warn(f\"Overwriting '{save_path}'\")\nelse:\nraise ValueError(error_message)\nlogger.info(f\"Compressing '{path}' to '{format}' in: '{save_path.parent}'\")\n# if Path(path).is_file():\n# logger.info(f\"'path' to {format} is a file. Setting 'base_dir' to: '{path}'\")\n# base_dir = path\narchive_path: Path = Path(\nmake_archive(\nbase_name=str(save_path),\nformat=str(format),\nroot_dir=root_dir,\nbase_dir=base_dir,\n# root_dir=str(Path()),\n# base_dir=path,\ndry_run=dry_run,\nlogger=logger,\n)\n)\nchdir(current_dir)\nreturn archive_path\ncopy_dict_paths(copy_path_dict: dict[PathLike, PathLike]) -> None\nCopy files from copy_path_dict keys to values.
>>> tmp_path: Path = getfixture('tmp_path')\n>>> for i in range(4):\n... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n>>> pprint(sorted(tmp_path.iterdir()))\n[...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n>>> output_path = tmp_path / 'save'\n>>> output_path.mkdir(exist_ok=True)\n>>> copy_dict_paths(\n... glob_path_rename_by_0_padding(tmp_path,\n... glob_regex_str=\"*.txt\",\n... output_path=output_path))\n<BLANKLINE>\n...Specified...'...save'...for...saving...file...copies...\n...'...-0...txt'...to...'...-00...txt...'...\n...'...-1...txt'...to...'...-01...txt...'\n...'...-2...txt'...to...'...-02...txt...'\n...'...-3...txt'...to...'...-03...txt...'\n>>> pprint(sorted(tmp_path.iterdir()))\n[...Path('...save'),\n ...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n>>> pprint(sorted((tmp_path / 'save').iterdir()))\n [...Path('...test_file-00.txt'),\n ...Path('...test_file-01.txt'),\n ...Path('...test_file-02.txt'),\n ...Path('...test_file-03.txt')]\nalto2txt2fixture/utils.py def copy_dict_paths(copy_path_dict: dict[PathLike, PathLike]) -> None:\n\"\"\"Copy files from `copy_path_dict` `keys` to `values`.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> for i in range(4):\n ... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n >>> pprint(sorted(tmp_path.iterdir()))\n [...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n >>> output_path = tmp_path / 'save'\n >>> output_path.mkdir(exist_ok=True)\n >>> copy_dict_paths(\n ... glob_path_rename_by_0_padding(tmp_path,\n ... glob_regex_str=\"*.txt\",\n ... output_path=output_path))\n <BLANKLINE>\n ...Specified...'...save'...for...saving...file...copies...\n ...'...-0...txt'...to...'...-00...txt...'...\n ...'...-1...txt'...to...'...-01...txt...'\n ...'...-2...txt'...to...'...-02...txt...'\n ...'...-3...txt'...to...'...-03...txt...'\n >>> pprint(sorted(tmp_path.iterdir()))\n [...Path('...save'),\n ...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n >>> pprint(sorted((tmp_path / 'save').iterdir()))\n [...Path('...test_file-00.txt'),\n ...Path('...test_file-01.txt'),\n ...Path('...test_file-02.txt'),\n ...Path('...test_file-03.txt')]\n ```\n \"\"\"\nfor current_path, copy_path in copy_path_dict.items():\nlogger.info(f\"Copying '{current_path}' to '{copy_path}'\")\nPath(copy_path).parent.mkdir(exist_ok=True)\ncopyfile(current_path, copy_path)\ncreate_lookup(lst: list = [], on: list = []) -> dict\nCreate a lookup dictionary from a list of dictionaries.
Parameters:
Name Type Description Defaultlst list A list of dictionaries that should be used to generate the lookup.
[] on list A list of keys from the dictionaries in the list that should be used as the keys in the lookup.
[] Returns:
Type Descriptiondict The generated lookup dictionary.
Source code inalto2txt2fixture/utils.py def create_lookup(lst: list = [], on: list = []) -> dict:\n\"\"\"\n Create a lookup dictionary from a list of dictionaries.\n Args:\n lst: A list of dictionaries that should be used to generate the lookup.\n on: A list of keys from the dictionaries in the list that should be used as the keys in the lookup.\n Returns:\n The generated lookup dictionary.\n \"\"\"\nreturn {get_key(x, on): x[\"pk\"] for x in lst}\ndict_from_list_fixture_fields(\nfixture_list: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,\nfield_name: str = DATA_PROVIDER_INDEX,\n) -> dict[str, FixtureDict]\nCreate a dict from fixture_list with attr_name as key.
Parameters:
Name Type Description Defaultfixture_list Iterable[FixtureDict] list of FixtureDict with attr_name key fields.
NEWSPAPER_COLLECTION_METADATA field_name str key for values within fixture_list fields.
DATA_PROVIDER_INDEX Returns:
Type Descriptiondict[str, FixtureDict] A dict where extracted field_name is key for related FixtureDict values.
>>> fixture_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields()\n>>> fixture_dict['hmd']['pk']\n2\n>>> fixture_dict['hmd']['fields'][DATA_PROVIDER_INDEX]\n'hmd'\n>>> fixture_dict['hmd']['fields']['code']\n'bl_hmd'\nalto2txt2fixture/utils.py def dict_from_list_fixture_fields(\nfixture_list: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,\nfield_name: str = DATA_PROVIDER_INDEX,\n) -> dict[str, FixtureDict]:\n\"\"\"Create a `dict` from ``fixture_list`` with ``attr_name`` as `key`.\n Args:\n fixture_list: `list` of `FixtureDict` with ``attr_name`` key `fields`.\n field_name: key for values within ``fixture_list`` `fields`.\n Returns:\n A `dict` where extracted `field_name` is key for related `FixtureDict` values.\n Example:\n ```pycon\n >>> fixture_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields()\n >>> fixture_dict['hmd']['pk']\n 2\n >>> fixture_dict['hmd']['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n >>> fixture_dict['hmd']['fields']['code']\n 'bl_hmd'\n ```\n \"\"\"\nreturn {record[\"fields\"][field_name]: record for record in fixture_list}\nexport_fixtures(\nfixture_tables: dict[str, Sequence[FixtureDict]],\npath: str | PathLike = settings.FIXTURE_TABLES_OUTPUT,\nprefix: str = \"test-\",\nadd_created: bool = True,\nformats: Sequence[EXPORT_FORMATS] = settings.FIXTURE_TABLES_FORMATS,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None\nExport fixture_tables in formats.
This is still in experimental phase of development and not recommended for production.
Parameters:
Name Type Description Defaultfixture_tables dict[str, Sequence[FixtureDict]] dict of table name (eg: dataprovider) and FixtureDict
path str | PathLike Path to save exports in
settings.FIXTURE_TABLES_OUTPUT prefix str str to prefix export file names with
'test-' formats Sequence[EXPORT_FORMATS] list of EXPORT_FORMATS to export
settings.FIXTURE_TABLES_FORMATS file_name_0_padding int Zeros to prefix the number of each fixture file name.
FILE_NAME_0_PADDING_DEFAULT Example >>> test_fixture_tables: dict[str, FixtureDict] = {\n... 'test0': NEWSPAPER_COLLECTION_METADATA,\n... 'test1': NEWSPAPER_COLLECTION_METADATA}\n>>> export_fixtures(test_fixture_tables, path='tests/')\n<BLANKLINE>\n...Warning: Saving test0...\n...Warning: Saving test1...\n>>> from pandas import read_csv\n>>> fixture0_json = load_json('tests/test-test0-000001.json')\n>>> fixture0_df = read_csv('tests/test-test0-000001.csv')\n>>> fixture1_json = load_json('tests/test-test1-000001.json')\n>>> fixture1_df = read_csv('tests/test-test1-000001.csv')\n>>> fixture0_json == fixture1_json\nTrue\n>>> all(fixture0_df == fixture1_df)\nTrue\n>>> all(field in fixture0_json[0]['fields']\n... for field in ['created_at', 'updated_at'])\nTrue\n>>> fixture0_json[1]['pk']\n2\n>>> fixture0_json[1]['fields'][DATA_PROVIDER_INDEX]\n'hmd'\n>>> fixture0_df[['pk', DATA_PROVIDER_INDEX]].iloc[1].to_list()\n[2, 'hmd']\nalto2txt2fixture/utils.py def export_fixtures(\nfixture_tables: dict[str, Sequence[FixtureDict]],\npath: str | PathLike = settings.FIXTURE_TABLES_OUTPUT,\nprefix: str = \"test-\",\nadd_created: bool = True,\nformats: Sequence[EXPORT_FORMATS] = settings.FIXTURE_TABLES_FORMATS,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None:\n\"\"\"Export ``fixture_tables`` in ``formats``.\n Note:\n This is still in experimental phase of development and not recommended\n for production.\n Args:\n fixture_tables:\n `dict` of table name (eg: `dataprovider`) and `FixtureDict`\n path:\n `Path` to save exports in\n prefix:\n `str` to prefix export file names with\n formats:\n `list` of `EXPORT_FORMATS` to export\n file_name_0_padding:\n Zeros to prefix the number of each fixture file name.\n Example:\n ```pycon\n >>> test_fixture_tables: dict[str, FixtureDict] = {\n ... 'test0': NEWSPAPER_COLLECTION_METADATA,\n ... 'test1': NEWSPAPER_COLLECTION_METADATA}\n >>> export_fixtures(test_fixture_tables, path='tests/')\n <BLANKLINE>\n ...Warning: Saving test0...\n ...Warning: Saving test1...\n >>> from pandas import read_csv\n >>> fixture0_json = load_json('tests/test-test0-000001.json')\n >>> fixture0_df = read_csv('tests/test-test0-000001.csv')\n >>> fixture1_json = load_json('tests/test-test1-000001.json')\n >>> fixture1_df = read_csv('tests/test-test1-000001.csv')\n >>> fixture0_json == fixture1_json\n True\n >>> all(fixture0_df == fixture1_df)\n True\n >>> all(field in fixture0_json[0]['fields']\n ... for field in ['created_at', 'updated_at'])\n True\n >>> fixture0_json[1]['pk']\n 2\n >>> fixture0_json[1]['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n >>> fixture0_df[['pk', DATA_PROVIDER_INDEX]].iloc[1].to_list()\n [2, 'hmd']\n ```\n \"\"\"\nfor table_name, records in fixture_tables.items():\nwarning(\nf\"Saving {table_name} fixture in {formats} formats \"\nf\"to {path} *without* checks...\"\n)\nif \"json\" in formats:\nsave_fixture(\nrecords,\nprefix=f\"{prefix}{table_name}\",\noutput_path=path,\nadd_created=add_created,\nfile_name_0_padding=file_name_0_padding,\n)\nif \"csv\" in formats:\nfixtures_dict2csv(records, prefix=f\"{prefix}{table_name}\", output_path=path)\nfilter_json_fields(\njson_results: list | dict | None = None,\nfile_path: PathLike | None = None,\nfields: Sequence[str] = [],\nvalue: Hashable = \"\",\n**kwargs: Hashable\n) -> dict | list\nReturn keys and values from json_dict where any fields equal value.
Parameters:
Name Type Description Defaultfile_path PathLike | None The file path to load based on extension and filter
None fields Sequence[str] Which fields to check equal value
[] value Hashable Value to filter by
'' Returns:
Type Descriptiondict | list A dict of records indexed by pk which fit filter criteria
Raises:
Type DescriptionValueError file_path must have a .json suffix
>>> entry_fixture: dict = [\n... {\"pk\": 4889, \"model\": \"mitchells.entry\",\n... \"fields\": {\"title\": \"BIRMINGHAM POST .\",\n... \"price_raw\": ['2d'],\n... \"year\": 1920,\n... \"date_established_raw\": \"1857\",\n... \"persons\": [], \"newspaper\": \"\"}},\n... {\"pk\": 9207, \"model\": \"mitchells.entry\",\n... \"fields\": {\"title\": \"ULVERSTONE ADVERTISER .\",\n... \"price_raw\": ['2 \u00bd d', '3 \u00bd d'],\n... \"year\": 1856,\n... \"date_established_raw\": \"1848\",\n... \"persons\": ['Stephen Soulby'],\n... \"newspaper\": \"\",}},\n... {\"pk\": 15, \"model\": \"mitchells.entry\",\n... \"fields\": {\"title\": \"LLOYD'S WEEKLY LONDON NEWSPAPER .\",\n... \"price_raw\": ['2d', '3d'],\n... \"year\": 1857,\n... \"date_established_raw\": \"November , 1842\",\n... \"persons\": ['Mr. Douglas Jerrold', 'Edward Lloyd'],\n... \"newspaper\": 1187}}\n... ]\n>>> pprint(filter_json_fields(entry_fixture,\n... fields=(\"newspaper\", \"persons\"),\n... value=\"\"))\n[{'fields': {'date_established_raw': '1857',\n 'newspaper': '',\n 'persons': [],\n 'price_raw': ['2d'],\n 'title': 'BIRMINGHAM POST .',\n 'year': 1920},\n 'model': 'mitchells.entry',\n 'pk': 4889},\n {'fields': {'date_established_raw': '1848',\n 'newspaper': '',\n 'persons': ['Stephen Soulby'],\n 'price_raw': ['2 \u00bd d', '3 \u00bd d'],\n 'title': 'ULVERSTONE ADVERTISER .',\n 'year': 1856},\n 'model': 'mitchells.entry',\n 'pk': 9207}]\nalto2txt2fixture/utils.py def filter_json_fields(\njson_results: list | dict | None = None,\nfile_path: PathLike | None = None,\nfields: Sequence[str] = [],\nvalue: Hashable = \"\",\n**kwargs,\n) -> dict | list:\n\"\"\"Return `keys` and `values` from `json_dict` where any `fields` equal `value`.\n Args:\n file_path: The file `path` to load based on extension and filter\n fields: Which fields to check equal `value`\n value: Value to filter by\n Returns:\n A `dict` of records indexed by `pk` which fit filter criteria\n Raises:\n ValueError: ``file_path`` must have a `.json` `suffix`\n Example:\n ```pycon\n >>> entry_fixture: dict = [\n ... {\"pk\": 4889, \"model\": \"mitchells.entry\",\n ... \"fields\": {\"title\": \"BIRMINGHAM POST .\",\n ... \"price_raw\": ['2d'],\n ... \"year\": 1920,\n ... \"date_established_raw\": \"1857\",\n ... \"persons\": [], \"newspaper\": \"\"}},\n ... {\"pk\": 9207, \"model\": \"mitchells.entry\",\n ... \"fields\": {\"title\": \"ULVERSTONE ADVERTISER .\",\n ... \"price_raw\": ['2 \\u00bd d', '3 \\u00bd d'],\n ... \"year\": 1856,\n ... \"date_established_raw\": \"1848\",\n ... \"persons\": ['Stephen Soulby'],\n ... \"newspaper\": \"\",}},\n ... {\"pk\": 15, \"model\": \"mitchells.entry\",\n ... \"fields\": {\"title\": \"LLOYD'S WEEKLY LONDON NEWSPAPER .\",\n ... \"price_raw\": ['2d', '3d'],\n ... \"year\": 1857,\n ... \"date_established_raw\": \"November , 1842\",\n ... \"persons\": ['Mr. Douglas Jerrold', 'Edward Lloyd'],\n ... \"newspaper\": 1187}}\n ... ]\n >>> pprint(filter_json_fields(entry_fixture,\n ... fields=(\"newspaper\", \"persons\"),\n ... value=\"\"))\n [{'fields': {'date_established_raw': '1857',\n 'newspaper': '',\n 'persons': [],\n 'price_raw': ['2d'],\n 'title': 'BIRMINGHAM POST .',\n 'year': 1920},\n 'model': 'mitchells.entry',\n 'pk': 4889},\n {'fields': {'date_established_raw': '1848',\n 'newspaper': '',\n 'persons': ['Stephen Soulby'],\n 'price_raw': ['2 \\u00bd d', '3 \\u00bd d'],\n 'title': 'ULVERSTONE ADVERTISER .',\n 'year': 1856},\n 'model': 'mitchells.entry',\n 'pk': 9207}]\n ```\n \"\"\"\nif not json_results:\nassert file_path\ntry:\nassert Path(file_path).suffix == \".json\"\nexcept AssertionError:\nraise ValueError(f\"{file_path} must be `json` format.\")\njson_results = load_json(Path(file_path), **kwargs)\nassert json_results\nif isinstance(json_results, dict):\nreturn {\nk: v\nfor k, v in json_results.items()\nif any(v[\"fields\"][field] == value for field in fields)\n}\nelse:\nreturn [\nv\nfor v in json_results\nif any(v[\"fields\"][field] == value for field in fields)\n]\nfixture_fields(\nfixture_dict: FixtureDict, include_pk: bool = True, as_dict: bool = False\n) -> tuple[str, ...] | dict[str, Any]\nGenerate a tuple of FixtureDict field names.
This is not in the utils module to avoid a circular import.
Parameters:
Name Type Description Defaultfixture_dict FixtureDict A FixtureDict instance to extract names from fields
include_pk bool Whether to include the pk (primary key) column
True Example >>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0])\n('pk', 'name', 'code', 'legacy_code', 'collection', 'source_note')\n>>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0], include_pk=False)\n('name', 'code', 'legacy_code', 'collection', 'source_note')\n>>> hmd_dict: dict[str, Any] = fixture_fields(\n... NEWSPAPER_COLLECTION_METADATA[1], as_dict=True)\n>>> hmd_dict['code']\n'bl_hmd'\n>>> hmd_dict['pk']\n2\n>>> hmd_dict = fixture_fields(\n... NEWSPAPER_COLLECTION_METADATA[1], include_pk=False, as_dict=True)\n>>> 'pk' in hmd_dict\nFalse\nalto2txt2fixture/utils.py def fixture_fields(\nfixture_dict: FixtureDict, include_pk: bool = True, as_dict: bool = False\n) -> tuple[str, ...] | dict[str, Any]:\n\"\"\"Generate a tuple of `FixtureDict` `field` names.\n Note:\n This is not in the `utils` module to avoid a circular import.\n Args:\n fixture_dict: A `FixtureDict` instance to extract names from `fields`\n include_pk: Whether to include the `pk` (primary key) column\n Example:\n ```pycon\n >>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0])\n ('pk', 'name', 'code', 'legacy_code', 'collection', 'source_note')\n >>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0], include_pk=False)\n ('name', 'code', 'legacy_code', 'collection', 'source_note')\n >>> hmd_dict: dict[str, Any] = fixture_fields(\n ... NEWSPAPER_COLLECTION_METADATA[1], as_dict=True)\n >>> hmd_dict['code']\n 'bl_hmd'\n >>> hmd_dict['pk']\n 2\n >>> hmd_dict = fixture_fields(\n ... NEWSPAPER_COLLECTION_METADATA[1], include_pk=False, as_dict=True)\n >>> 'pk' in hmd_dict\n False\n ```\n \"\"\"\nfields: OrderedDict[str, Any] = OrderedDict(fixture_dict[\"fields\"])\nif include_pk:\nfields[\"pk\"] = fixture_dict[\"pk\"]\nfields.move_to_end(\"pk\", last=False)\nif as_dict:\nreturn fields\nelse:\nreturn tuple(fields.keys())\nfixture_or_default_dict(\nkey: str,\nfixture_dict: dict[str, FixtureDict],\ndefault_dict: FixtureDict | dict = {},\n) -> FixtureDict | dict\nReturn a FixtureDict from fixture_list via key index, else default_dict.
Parameters:
Name Type Description Defaultkey str a str to query fixture_dict with
fixture_dict dict[str, FixtureDict] a dict of str to FixtureDict, often generated by dict_from_list_fixture_fields
default_dict FixtureDict | dict a dict to return if key is not in fixture_dict index
{} Example >>> newspaper_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields(\n... NEWSPAPER_COLLECTION_METADATA)\n>>> hmd_dict: FixtureDict = fixture_or_default_dict(\n... 'hmd', newspaper_dict\n... )\n>>> hmd_dict == newspaper_dict['hmd']\nTrue\n>>> fixture_or_default_dict(\n... 'hmd', NEWSPAPER_COLLECTION_METADATA\n... )\n{}\n>>> fixture_or_default_dict(\n... 'hmd', NEWSPAPER_COLLECTION_METADATA, {'a': 'default'}\n... )\n{'a': 'default'}\nalto2txt2fixture/utils.py def fixture_or_default_dict(\nkey: str,\nfixture_dict: dict[str, FixtureDict],\ndefault_dict: FixtureDict | dict = {},\n) -> FixtureDict | dict:\n\"\"\"Return a `FixtureDict` from ``fixture_list`` via ``key`` index, else ``default_dict``.\n Args:\n key:\n a `str` to query ``fixture_dict`` with\n fixture_dict:\n a `dict` of `str` to `FixtureDict`, often generated by\n ``dict_from_list_fixture_fields``\n default_dict:\n a `dict` to return if ``key`` is not in ``fixture_dict`` index\n Example:\n ```pycon\n >>> newspaper_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields(\n ... NEWSPAPER_COLLECTION_METADATA)\n >>> hmd_dict: FixtureDict = fixture_or_default_dict(\n ... 'hmd', newspaper_dict\n ... )\n >>> hmd_dict == newspaper_dict['hmd']\n True\n >>> fixture_or_default_dict(\n ... 'hmd', NEWSPAPER_COLLECTION_METADATA\n ... )\n {}\n >>> fixture_or_default_dict(\n ... 'hmd', NEWSPAPER_COLLECTION_METADATA, {'a': 'default'}\n ... )\n {'a': 'default'}\n ```\n \"\"\"\nif key in fixture_dict:\nreturn fixture_dict[key]\nelse:\nreturn default_dict\nfixtures_dict2csv(\nfixtures: Iterable[FixtureDict] | Generator[FixtureDict, None, None],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nindex: bool = False,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None\nSaves fixtures generated by a generator to separate separate CSV files.
This function takes an Iterable or Generator of fixtures and saves to separate CSV files. The fixtures are saved in batches, where each batch is determined by the max_elements_per_file parameter.
Parameters:
Name Type Description Defaultfixtures Iterable[FixtureDict] | Generator[FixtureDict, None, None] An Iterable or Generator of the fixtures to be saved.
prefix str A string prefix to be added to the file names of the saved fixtures.
'' output_path PathLike | str Path to folder fixtures are saved to
settings.OUTPUT max_elements_per_file int Maximum JSON records saved in each file
settings.MAX_ELEMENTS_PER_FILE file_name_0_padding int Zeros to prefix the number of each fixture file name.
FILE_NAME_0_PADDING_DEFAULT Returns:
Type DescriptionNone This function saves fixtures to files and does not return a value.
Example>>> tmp_path: Path = getfixture('tmp_path')\n>>> from pandas import read_csv\n>>> fixtures_dict2csv(NEWSPAPER_COLLECTION_METADATA,\n... prefix='test', output_path=tmp_path)\n>>> imported_fixture = read_csv(tmp_path / 'test-000001.csv')\n>>> imported_fixture.iloc[1]['pk']\n2\n>>> imported_fixture.iloc[1][DATA_PROVIDER_INDEX]\n'hmd'\nalto2txt2fixture/utils.py def fixtures_dict2csv(\nfixtures: Iterable[FixtureDict] | Generator[FixtureDict, None, None],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nindex: bool = False,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None:\n\"\"\"Saves fixtures generated by a generator to separate separate `CSV` files.\n This function takes an `Iterable` or `Generator` of fixtures and saves to\n separate `CSV` files. The fixtures are saved in batches, where each batch\n is determined by the ``max_elements_per_file`` parameter.\n Args:\n fixtures:\n An `Iterable` or `Generator` of the fixtures to be saved.\n prefix:\n A string prefix to be added to the file names of the\n saved fixtures.\n output_path:\n Path to folder fixtures are saved to\n max_elements_per_file:\n Maximum `JSON` records saved in each file\n file_name_0_padding:\n Zeros to prefix the number of each fixture file name.\n Returns:\n This function saves fixtures to files and does not return a value.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> from pandas import read_csv\n >>> fixtures_dict2csv(NEWSPAPER_COLLECTION_METADATA,\n ... prefix='test', output_path=tmp_path)\n >>> imported_fixture = read_csv(tmp_path / 'test-000001.csv')\n >>> imported_fixture.iloc[1]['pk']\n 2\n >>> imported_fixture.iloc[1][DATA_PROVIDER_INDEX]\n 'hmd'\n ```\n \"\"\"\ninternal_counter: int = 1\ncounter: int = 1\nlst: list = []\nfile_name: str\ndf: DataFrame\nPath(output_path).mkdir(parents=True, exist_ok=True)\nfor item in fixtures:\nlst.append(fixture_fields(item, as_dict=True))\ninternal_counter += 1\nif internal_counter > max_elements_per_file:\ndf = DataFrame.from_records(lst)\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.csv\"\ndf.to_csv(Path(output_path) / file_name, index=index)\n# Save up some memory\ndel lst\ngc.collect()\n# Re-instantiate\nlst = []\ninternal_counter = 1\ncounter += 1\nelse:\ndf = DataFrame.from_records(lst)\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.csv\"\ndf.to_csv(Path(output_path) / file_name, index=index)\nfree_hd_space_in_GB(\ndisk_usage_tuple: DiskUsageTuple | None = None, path: PathLike | None = None\n) -> float\nReturn remaing hard drive space estimate in gigabytes.
Parameters:
Name Type Description Defaultdisk_usage_tuple DiskUsageTuple | None A NamedTuple normally returned from disk_usage() or None.
None path PathLike | None A path to pass to disk_usage if disk_usage_tuple is None.
None Returns:
Type Descriptionfloat A float from dividing the disk_usage_tuple.free value by BYTES_PER_GIGABYTE
>>> space_in_gb = free_hd_space_in_GB()\n>>> space_in_gb > 1 # Hopefully true wherever run...\nTrue\nalto2txt2fixture/utils.py def free_hd_space_in_GB(\ndisk_usage_tuple: DiskUsageTuple | None = None, path: PathLike | None = None\n) -> float:\n\"\"\"Return remaing hard drive space estimate in gigabytes.\n Args:\n disk_usage_tuple:\n A `NamedTuple` normally returned from `disk_usage()` or `None`.\n path:\n A `path` to pass to `disk_usage` if `disk_usage_tuple` is `None`.\n Returns:\n A `float` from dividing the `disk_usage_tuple.free` value by `BYTES_PER_GIGABYTE`\n Example:\n ```pycon\n >>> space_in_gb = free_hd_space_in_GB()\n >>> space_in_gb > 1 # Hopefully true wherever run...\n True\n ```\n \"\"\"\nif not disk_usage_tuple:\nif not path:\npath = Path(getcwd())\ndisk_usage_tuple = disk_usage(path=path)\nassert disk_usage_tuple\nreturn disk_usage_tuple.free / BYTES_PER_GIGABYTE\ngen_fixture_tables(\nfixture_tables: dict[str, list[FixtureDict]] = {},\ninclude_fixture_pk_column: bool = True,\n) -> Generator[Table, None, None]\nGenerator of rich.Table instances from FixtureDict configuration tables.
Parameters:
Name Type Description Defaultfixture_tables dict[str, list[FixtureDict]] dict where key is for Table title and value is a FixtureDict
{} include_fixture_pk_column bool whether to include the pk field from FixtureDict
True Example >>> table_name: str = \"data_provider\"\n>>> tables = tuple(\n... gen_fixture_tables(\n... {table_name: NEWSPAPER_COLLECTION_METADATA}\n... ))\n>>> len(tables)\n1\n>>> assert tables[0].title == table_name\n>>> [column.header for column in tables[0].columns]\n['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\nalto2txt2fixture/utils.py def gen_fixture_tables(\nfixture_tables: dict[str, list[FixtureDict]] = {},\ninclude_fixture_pk_column: bool = True,\n) -> Generator[Table, None, None]:\n\"\"\"Generator of `rich.Table` instances from `FixtureDict` configuration tables.\n Args:\n fixture_tables: `dict` where `key` is for `Table` title and `value` is a `FixtureDict`\n include_fixture_pk_column: whether to include the `pk` field from `FixtureDict`\n Example:\n ```pycon\n >>> table_name: str = \"data_provider\"\n >>> tables = tuple(\n ... gen_fixture_tables(\n ... {table_name: NEWSPAPER_COLLECTION_METADATA}\n ... ))\n >>> len(tables)\n 1\n >>> assert tables[0].title == table_name\n >>> [column.header for column in tables[0].columns]\n ['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\n ```\n \"\"\"\nfor name, fixture_records in fixture_tables.items():\nfixture_table: Table = Table(title=name)\nfor i, fixture_dict in enumerate(fixture_records):\nif i == 0:\n[\nfixture_table.add_column(name)\nfor name in fixture_fields(fixture_dict, include_fixture_pk_column)\n]\nrow_values: tuple[str, ...] = tuple(\nstr(x) for x in (fixture_dict[\"pk\"], *fixture_dict[\"fields\"].values())\n)\nfixture_table.add_row(*row_values)\nyield fixture_table\nget_chunked_zipfiles(path: Path) -> list\nThis function takes in a Path object path and returns a list of lists of zipfiles sorted and chunked according to certain conditions defined in the settings object (see settings.CHUNK_THRESHOLD).
Note: the function will also skip zip files of a certain file size, which can be specified in the settings object (see settings.SKIP_FILE_SIZE).
Parameters:
Name Type Description Defaultpath Path The input path where the zipfiles are located
requiredReturns:
Type Descriptionlist A list of lists of zipfiles, each inner list represents a chunk of zipfiles.
alto2txt2fixture/utils.py def get_chunked_zipfiles(path: Path) -> list:\n\"\"\"This function takes in a `Path` object `path` and returns a list of lists\n of `zipfiles` sorted and chunked according to certain conditions defined\n in the `settings` object (see `settings.CHUNK_THRESHOLD`).\n Note: the function will also skip zip files of a certain file size, which\n can be specified in the `settings` object (see `settings.SKIP_FILE_SIZE`).\n Args:\n path: The input path where the zipfiles are located\n Returns:\n A list of lists of `zipfiles`, each inner list represents a chunk of\n zipfiles.\n \"\"\"\nzipfiles = sorted(\npath.glob(\"*.zip\"),\nkey=lambda x: x.stat().st_size,\nreverse=settings.START_WITH_LARGEST,\n)\nzipfiles = [x for x in zipfiles if x.stat().st_size <= settings.SKIP_FILE_SIZE]\nif len(zipfiles) > settings.CHUNK_THRESHOLD:\nchunks = array_split(zipfiles, len(zipfiles) / settings.CHUNK_THRESHOLD)\nelse:\nchunks = [zipfiles]\nreturn chunks\nget_key(x: dict = dict(), on: list = []) -> str\nGet a string key from a dictionary using values from specified keys.
Parameters:
Name Type Description Defaultx dict A dictionary from which the key is generated.
dict() on list A list of keys from the dictionary that should be used to generate the key.
[] Returns:
Type Descriptionstr The generated string key.
Source code inalto2txt2fixture/utils.py def get_key(x: dict = dict(), on: list = []) -> str:\n\"\"\"\n Get a string key from a dictionary using values from specified keys.\n Args:\n x: A dictionary from which the key is generated.\n on: A list of keys from the dictionary that should be used to\n generate the key.\n Returns:\n The generated string key.\n \"\"\"\nreturn f\"{'-'.join([str(x['fields'][y]) for y in on])}\"\nget_lockfile(collection: str, kind: NewspaperElements, dic: dict) -> Path\nProvides the path to any given lockfile, which controls whether any existing files should be overwritten or not.
Parameters:
Name Type Description Defaultcollection str Collection folder name
requiredkind NewspaperElements Either newspaper or issue or item
dic dict A dictionary with required information for either kind passed
Returns:
Type DescriptionPath Path to the resulting lockfile
Source code inalto2txt2fixture/utils.py def get_lockfile(collection: str, kind: NewspaperElements, dic: dict) -> Path:\n\"\"\"\n Provides the path to any given lockfile, which controls whether any\n existing files should be overwritten or not.\n Args:\n collection: Collection folder name\n kind: Either `newspaper` or `issue` or `item`\n dic: A dictionary with required information for either `kind` passed\n Returns:\n Path to the resulting lockfile\n \"\"\"\np: Path\nbase = Path(f\"cache-lockfiles/{collection}\")\nif kind == \"newspaper\":\np = base / f\"newspapers/{dic['publication_code']}\"\nelif kind == \"issue\":\np = base / f\"issues/{dic['publication__publication_code']}/{dic['issue_code']}\"\nelif kind == \"item\":\ntry:\nif dic.get(\"issue_code\"):\np = base / f\"items/{dic['issue_code']}/{dic['item_code']}\"\nelif dic.get(\"issue__issue_identifier\"):\np = base / f\"items/{dic['issue__issue_identifier']}/{dic['item_code']}\"\nexcept KeyError:\nerror(\"An unknown error occurred (in get_lockfile)\")\nelse:\np = base / \"lockfile\"\np.parent.mkdir(parents=True, exist_ok=True) if settings.WRITE_LOCKFILES else None\nreturn p\nget_now(as_str: bool = False) -> datetime.datetime | str\nReturn datetime.now() as either a string or datetime object.
Parameters:
Name Type Description Defaultas_str bool Whether to return now time as a str or not, default: False
False Returns:
Type Descriptiondatetime.datetime | str datetime.now() in pytz.UTC time zone as a string if as_str, else as a datetime.datetime object.
alto2txt2fixture/utils.py def get_now(as_str: bool = False) -> datetime.datetime | str:\n\"\"\"\n Return `datetime.now()` as either a string or `datetime` object.\n Args:\n as_str: Whether to return `now` `time` as a `str` or not, default: `False`\n Returns:\n `datetime.now()` in `pytz.UTC` time zone as a string if `as_str`, else\n as a `datetime.datetime` object.\n \"\"\"\nnow = datetime.datetime.now(tz=pytz.UTC)\nif as_str:\nreturn str(now)\nelse:\nassert isinstance(now, datetime.datetime)\nreturn now\nget_path_from(p: str | Path) -> Path\nConverts an input value into a Path object if it's not already one.
Parameters:
Name Type Description Defaultp str | Path The input value, which can be a string or a Path object.
requiredReturns:
Type DescriptionPath The input value as a Path object.
Source code inalto2txt2fixture/utils.py def get_path_from(p: str | Path) -> Path:\n\"\"\"\n Converts an input value into a Path object if it's not already one.\n Args:\n p: The input value, which can be a string or a Path object.\n Returns:\n The input value as a Path object.\n \"\"\"\nif isinstance(p, str):\np = Path(p)\nif not isinstance(p, Path):\nraise RuntimeError(f\"Unable to handle type: {type(p)}\")\nreturn p\nget_size_from_path(p: str | Path, raw: bool = False) -> str | float\nReturns a nice string for any given file size.
Parameters:
Name Type Description Defaultp str | Path Path to read the size from
requiredraw bool Whether to return the file size as total number of bytes or a human-readable MB/GB amount
False Returns:
Type Descriptionstr | float Return str followed by MB or GB for size if not raw otherwise float.
alto2txt2fixture/utils.py def get_size_from_path(p: str | Path, raw: bool = False) -> str | float:\n\"\"\"\n Returns a nice string for any given file size.\n Args:\n p: Path to read the size from\n raw: Whether to return the file size as total number of bytes or\n a human-readable MB/GB amount\n Returns:\n Return `str` followed by `MB` or `GB` for size if not `raw` otherwise `float`.\n \"\"\"\np = get_path_from(p)\nbytes = p.stat().st_size\nif raw:\nreturn bytes\nrel_size: float | int | str = round(bytes / 1000 / 1000 / 1000, 1)\nassert not isinstance(rel_size, str)\nif rel_size < 0.5:\nrel_size = round(bytes / 1000 / 1000, 1)\nrel_size = f\"{rel_size}MB\"\nelse:\nrel_size = f\"{rel_size}GB\"\nreturn rel_size\nglob_filter(p: str) -> list\nReturn ordered glob, filtered out any pesky, unwanted .DS_Store from macOS.
Parameters:
Name Type Description Defaultp str Path to a directory to filter
requiredReturns:
Type Descriptionlist Sorted list of files contained in the provided path without the ones
list whose names start with a .
alto2txt2fixture/utils.py def glob_filter(p: str) -> list:\n\"\"\"\n Return ordered glob, filtered out any pesky, unwanted .DS_Store from macOS.\n Args:\n p: Path to a directory to filter\n Returns:\n Sorted list of files contained in the provided path without the ones\n whose names start with a `.`\n \"\"\"\nreturn sorted([x for x in get_path_from(p).glob(\"*\") if not x.name.startswith(\".\")])\nglob_path_rename_by_0_padding(\npath: PathLike,\noutput_path: PathLike | None = None,\nglob_regex_str: str = \"*\",\npadding: int | None = 0,\nmatch_int_regex: str = PADDING_0_REGEX_DEFAULT,\nindex: int = -1,\n) -> dict[PathLike, PathLike]\nReturn an OrderedDict of replacement 0-padded file names from path.
Parameters:
Name Type Description Defaultpath PathLike PathLike to source files to rename.
output_path PathLike | None PathLike to save renamed files to.
None glob_regex_str str str to match files to rename within path.
'*' padding int | None How many digits (0s) to pad match_int with.
0 match_int_regex str Regular expression for matching numbers in s to pad. Only rename parts of Path(file_path).name; else replace across Path(file_path).parents as well.
PADDING_0_REGEX_DEFAULT index int Which index of number in s to pad with 0s. Like numbering a list, 0 indicates the first match and -1 indicates the last match.
-1 Example >>> tmp_path: Path = getfixture('tmp_path')\n>>> for i in range(4):\n... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n>>> pprint(sorted(tmp_path.iterdir()))\n[...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n>>> pprint(glob_path_rename_by_0_padding(tmp_path))\n{...Path('...test_file-0.txt'): ...Path('...test_file-00.txt'),\n ...Path('...test_file-1.txt'): ...Path('...test_file-01.txt'),\n ...Path('...test_file-2.txt'): ...Path('...test_file-02.txt'),\n ...Path('...test_file-3.txt'): ...Path('...test_file-03.txt')}\nalto2txt2fixture/utils.py def glob_path_rename_by_0_padding(\npath: PathLike,\noutput_path: PathLike | None = None,\nglob_regex_str: str = \"*\",\npadding: int | None = 0,\nmatch_int_regex: str = PADDING_0_REGEX_DEFAULT,\nindex: int = -1,\n) -> dict[PathLike, PathLike]:\n\"\"\"Return an `OrderedDict` of replacement 0-padded file names from `path`.\n Params:\n path:\n `PathLike` to source files to rename.\n output_path:\n `PathLike` to save renamed files to.\n glob_regex_str:\n `str` to match files to rename within `path`.\n padding:\n How many digits (0s) to pad `match_int` with.\n match_int_regex:\n Regular expression for matching numbers in `s` to pad.\n Only rename parts of `Path(file_path).name`; else\n replace across `Path(file_path).parents` as well.\n index:\n Which index of number in `s` to pad with 0s.\n Like numbering a `list`, 0 indicates the first match\n and -1 indicates the last match.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> for i in range(4):\n ... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n >>> pprint(sorted(tmp_path.iterdir()))\n [...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n >>> pprint(glob_path_rename_by_0_padding(tmp_path))\n {...Path('...test_file-0.txt'): ...Path('...test_file-00.txt'),\n ...Path('...test_file-1.txt'): ...Path('...test_file-01.txt'),\n ...Path('...test_file-2.txt'): ...Path('...test_file-02.txt'),\n ...Path('...test_file-3.txt'): ...Path('...test_file-03.txt')}\n ```\n \"\"\"\ntry:\nassert Path(path).exists()\nexcept AssertionError:\nraise ValueError(f'path does not exist: \"{Path(path)}\"')\npaths_tuple: tuple[PathLike, ...] = path_globs_to_tuple(path, glob_regex_str)\ntry:\nassert paths_tuple\nexcept AssertionError:\nraise FileNotFoundError(\nf\"No files found matching 'glob_regex_str': \"\nf\"'{glob_regex_str}' in: '{path}'\"\n)\npaths_to_index: tuple[tuple[str, int], ...] = tuple(\nint_from_str(str(matched_path), index=index, regex=match_int_regex)\nfor matched_path in paths_tuple\n)\nmax_index: int = max(index[1] for index in paths_to_index)\nmax_index_digits: int = len(str(max_index))\nif not padding or padding < max_index_digits:\npadding = max_index_digits + 1\nnew_names_dict: dict[PathLike, PathLike] = {}\nif output_path:\nif not Path(output_path).is_absolute():\noutput_path = Path(path) / output_path\nlogger.debug(f\"Specified '{output_path}' for saving file copies\")\nfor i, old_path in enumerate(paths_tuple):\nmatch_str, match_int = paths_to_index[i]\nnew_names_dict[old_path] = rename_by_0_padding(\nold_path, match_str=str(match_str), match_int=match_int, padding=padding\n)\nif output_path:\nnew_names_dict[old_path] = (\nPath(output_path) / Path(new_names_dict[old_path]).name\n)\nreturn new_names_dict\nint_from_str(\ns: str, index: int = -1, regex: str = PADDING_0_REGEX_DEFAULT\n) -> tuple[str, int]\nReturn matched (or None) regex from s by index index.
Parameters:
Name Type Description Defaults str str to match and via regex.
index int Which index of number in s to pad with 0s. Like numbering a list, 0 indicates the first match and -1 indicates the last match.
-1 regex str Regular expression for matching numbers in s to pad.
PADDING_0_REGEX_DEFAULT Example >>> int_from_str('a/path/to/fixture-03-05.txt')\n('05', 5)\n>>> int_from_str('a/path/to/fixture-03-05.txt', index=0)\n('03', 3)\nalto2txt2fixture/utils.py def int_from_str(\ns: str,\nindex: int = -1,\nregex: str = PADDING_0_REGEX_DEFAULT,\n) -> tuple[str, int]:\n\"\"\"Return matched (or None) `regex` from `s` by index `index`.\n Params:\n s:\n `str` to match and via `regex`.\n index:\n Which index of number in `s` to pad with 0s.\n Like numbering a `list`, 0 indicates the first match\n and -1 indicates the last match.\n regex:\n Regular expression for matching numbers in `s` to pad.\n Example:\n ```pycon\n >>> int_from_str('a/path/to/fixture-03-05.txt')\n ('05', 5)\n >>> int_from_str('a/path/to/fixture-03-05.txt', index=0)\n ('03', 3)\n ```\n \"\"\"\nmatches: list[str] = [match for match in findall(regex, s) if match]\nmatch_str: str = matches[index]\nreturn match_str, int(match_str)\nlist_json_files(\np: str | Path,\ndrill: bool = False,\nexclude_names: list = [],\ninclude_names: list = [],\n) -> Generator[Path, None, None] | list[Path]\nList json files under the path specified in p.
Parameters:
Name Type Description Defaultp str | Path The path to search for json files
drill bool A flag indicating whether to drill down the subdirectories or not. Default is False
False exclude_names list A list of file names to exclude from the search result. Default is an empty list
[] include_names list A list of file names to include in search result. If provided, the exclude_names argument will be ignored. Default is an empty list
[] Returns:
Type DescriptionGenerator[Path, None, None] | list[Path] A list of Path objects pointing to the found json files
alto2txt2fixture/utils.py def list_json_files(\np: str | Path,\ndrill: bool = False,\nexclude_names: list = [],\ninclude_names: list = [],\n) -> Generator[Path, None, None] | list[Path]:\n\"\"\"\n List `json` files under the path specified in ``p``.\n Args:\n p: The path to search for `json` files\n drill: A flag indicating whether to drill down the subdirectories\n or not. Default is ``False``\n exclude_names: A list of file names to exclude from the search\n result. Default is an empty list\n include_names: A list of file names to include in search result.\n If provided, the ``exclude_names`` argument will be ignored.\n Default is an empty list\n Returns:\n A list of `Path` objects pointing to the found `json` files\n \"\"\"\nq: str = \"**/*.json\" if drill else \"*.json\"\nfiles = get_path_from(p).glob(q)\nif exclude_names:\nfiles = list({x for x in files if x.name not in exclude_names})\nelif include_names:\nfiles = list({x for x in files if x.name in include_names})\nreturn sorted(files)\nload_json(p: str | Path, crash: bool = False) -> dict | list\nEasier access to reading json files.
Parameters:
Name Type Description Defaultp str | Path Path to read json from
crash bool Whether the program should crash if there is a json decode error, default: False
False Returns:
Type Descriptiondict | list The decoded json contents from the path, but an empty dictionary
dict | list if the file cannot be decoded and crash is set to False
alto2txt2fixture/utils.py def load_json(p: str | Path, crash: bool = False) -> dict | list:\n\"\"\"\n Easier access to reading `json` files.\n Args:\n p: Path to read `json` from\n crash: Whether the program should crash if there is a `json` decode\n error, default: ``False``\n Returns:\n The decoded `json` contents from the path, but an empty dictionary\n if the file cannot be decoded and ``crash`` is set to ``False``\n \"\"\"\np = get_path_from(p)\ntry:\nreturn json.loads(p.read_text())\nexcept json.JSONDecodeError:\nmsg = f\"Error: {p.read_text()}\"\nerror(msg, crash=crash)\nreturn {}\nload_multiple_json(\np: str | Path,\ndrill: bool = False,\nfilter_na: bool = True,\ncrash: bool = False,\n) -> list\nLoad multiple json files and return a list of their content.
Parameters:
Name Type Description Defaultp str | Path The path to search for json files
drill bool A flag indicating whether to drill down the subdirectories or not. Default is False
False filter_na bool A flag indicating whether to filter out the content that is None. Default is True.
True crash bool A flag indicating whether to raise an exception when an error occurs while loading a json file. Default is False.
False Returns:
Type Descriptionlist A list of the content of the loaded json files.
alto2txt2fixture/utils.py def load_multiple_json(\np: str | Path,\ndrill: bool = False,\nfilter_na: bool = True,\ncrash: bool = False,\n) -> list:\n\"\"\"\n Load multiple `json` files and return a list of their content.\n Args:\n p: The path to search for `json` files\n drill: A flag indicating whether to drill down the subdirectories\n or not. Default is `False`\n filter_na: A flag indicating whether to filter out the content that\n is `None`. Default is `True`.\n crash: A flag indicating whether to raise an exception when an\n error occurs while loading a `json` file. Default is `False`.\n Returns:\n A `list` of the content of the loaded `json` files.\n \"\"\"\nfiles = list_json_files(p, drill=drill)\ncontent = [load_json(x, crash=crash) for x in files]\nreturn [x for x in content if x] if filter_na else content\nlock(lockfile: Path) -> None\nWrites a '.' to a lockfile, after making sure the parent directory exists.
Parameters:
Name Type Description Defaultlockfile Path The path to the lock file to be created
requiredReturns:
Type DescriptionNone None
Source code inalto2txt2fixture/utils.py def lock(lockfile: Path) -> None:\n\"\"\"\n Writes a '.' to a lockfile, after making sure the parent directory exists.\n Args:\n lockfile: The path to the lock file to be created\n Returns:\n None\n \"\"\"\nlockfile.parent.mkdir(parents=True, exist_ok=True)\nlockfile.write_text(\"\")\nreturn\npath_globs_to_tuple(\npath: PathLike, glob_regex_str: str = \"*\"\n) -> tuple[PathLike, ...]\nReturn a sorted tuple of Paths in path using glob_regex_str.
Parameters:
Name Type Description Defaultpath PathLike Patch to search via glob
glob_regex_str str Regular expression to use with glob at path
'*' Returns:
Type Descriptiontuple[PathLike, ...] tuple of matching paths.
>>> bl_lwm = getfixture(\"bl_lwm\")\n>>> pprint(path_globs_to_tuple(bl_lwm, '*text.zip'))\n(...Path('...bl_lwm...0003079-test_plaintext.zip'),\n ...Path('...bl_lwm...0003548-test_plaintext.zip'))\n>>> pprint(path_globs_to_tuple(bl_lwm, '*.txt'))\n(...Path('...bl_lwm...0003079_18980121_sect0001.txt'),\n ...Path('...bl_lwm...0003548_19040707_art0037.txt'))\nalto2txt2fixture/utils.py def path_globs_to_tuple(\npath: PathLike, glob_regex_str: str = \"*\"\n) -> tuple[PathLike, ...]:\n\"\"\"Return a sorted `tuple` of `Path`s in `path` using `glob_regex_str`.\n Args:\n path:\n Patch to search via `glob`\n glob_regex_str:\n Regular expression to use with `glob` at `path`\n Returns:\n `tuple` of matching paths.\n Example:\n ```pycon\n >>> bl_lwm = getfixture(\"bl_lwm\")\n >>> pprint(path_globs_to_tuple(bl_lwm, '*text.zip'))\n (...Path('...bl_lwm...0003079-test_plaintext.zip'),\n ...Path('...bl_lwm...0003548-test_plaintext.zip'))\n >>> pprint(path_globs_to_tuple(bl_lwm, '*.txt'))\n (...Path('...bl_lwm...0003079_18980121_sect0001.txt'),\n ...Path('...bl_lwm...0003548_19040707_art0037.txt'))\n ```\n \"\"\"\nreturn tuple(sorted(Path(path).glob(glob_regex_str)))\npaths_with_newlines(\npaths: Iterable[PathLike], truncate: bool = False, **kwargs: bool\n) -> str\nReturn a str of paths separated by a \\n.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n>>> print(paths_with_newlines(plaintext_bl_lwm.compressed_files))\n'...bl_lwm...0003079-test_plaintext.zip'\n'...bl_lwm...0003548-test_plaintext.zip'\n>>> print(\n... paths_with_newlines(plaintext_bl_lwm.compressed_files,\n... truncate=True)\n... )\n<BLANKLINE>\n...\n'...0003079-test_plaintext.zip'\n'...0003548-test_plaintext.zip'\nalto2txt2fixture/utils.py def paths_with_newlines(\npaths: Iterable[PathLike], truncate: bool = False, **kwargs\n) -> str:\n\"\"\"Return a `str` of `paths` separated by a `\\\\n`.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n >>> print(paths_with_newlines(plaintext_bl_lwm.compressed_files))\n '...bl_lwm...0003079-test_plaintext.zip'\n '...bl_lwm...0003548-test_plaintext.zip'\n >>> print(\n ... paths_with_newlines(plaintext_bl_lwm.compressed_files,\n ... truncate=True)\n ... )\n <BLANKLINE>\n ...\n '...0003079-test_plaintext.zip'\n '...0003548-test_plaintext.zip'\n ```\n \"\"\"\nif truncate:\nreturn \"\\n\".join(f\"'{truncate_path_str(f, **kwargs)}'\" for f in paths)\nelse:\nreturn \"\\n\".join(f\"'{f}'\" for f in paths)\nrename_by_0_padding(\nfile_path: PathLike,\nmatch_str: str | None = None,\nmatch_int: int | None = None,\npadding: int = FILE_NAME_0_PADDING_DEFAULT,\nreplace_count: int = 1,\nexclude_parents: bool = True,\nreverse_int_match: bool = False,\n) -> Path\nReturn file_path with 0 padding Path change.
Parameters:
Name Type Description Defaultfile_path PathLike PathLike to rename.
match_str str | None str to match and replace with padded match_int
None match_int int | None int to pad and replace match_str
None padding int How many digits (0s) to pad match_int with.
FILE_NAME_0_PADDING_DEFAULT exclude_parents bool Only rename parts of Path(file_path).name; else replace across Path(file_path).parents as well.
True reverse_int_match bool Whether to match from the end of the file_path.
False Example >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_str='05', match_int=5)\n<BLANKLINE>\n...Path('a/path/to/3/fixture-03-000005.txt')...\n>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_str='03')\n<BLANKLINE>\n...Path('a/path/to/3/fixture-000003-05.txt')...\n>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_str='05', padding=0)\n<BLANKLINE>\n...Path('a/path/to/3/fixture-03-5.txt')...\n>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_int=3)\n<BLANKLINE>\n...Path('a/path/to/3/fixture-0000003-05.txt')...\n>>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n... match_int=3, padding=2,\n... exclude_parents=False)\n<BLANKLINE>\n...Path('a/path/to/03/f-03-05-0003.txt')...\n>>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n... match_int=3, padding=2,\n... exclude_parents=False,\n... replace_count=3, )\n<BLANKLINE>\n...Path('a/path/to/03/f-003-05-00003.txt')...\nalto2txt2fixture/utils.py def rename_by_0_padding(\nfile_path: PathLike,\nmatch_str: str | None = None,\nmatch_int: int | None = None,\npadding: int = FILE_NAME_0_PADDING_DEFAULT,\nreplace_count: int = 1,\nexclude_parents: bool = True,\nreverse_int_match: bool = False,\n) -> Path:\n\"\"\"Return `file_path` with `0` `padding` `Path` change.\n Params:\n file_path:\n `PathLike` to rename.\n match_str:\n `str` to match and replace with padded `match_int`\n match_int:\n `int` to pad and replace `match_str`\n padding:\n How many digits (0s) to pad `match_int` with.\n exclude_parents:\n Only rename parts of `Path(file_path).name`; else\n replace across `Path(file_path).parents` as well.\n reverse_int_match:\n Whether to match from the end of the `file_path`.\n Example:\n ```pycon\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_str='05', match_int=5)\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-03-000005.txt')...\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_str='03')\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-000003-05.txt')...\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_str='05', padding=0)\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-03-5.txt')...\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_int=3)\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-0000003-05.txt')...\n >>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n ... match_int=3, padding=2,\n ... exclude_parents=False)\n <BLANKLINE>\n ...Path('a/path/to/03/f-03-05-0003.txt')...\n >>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n ... match_int=3, padding=2,\n ... exclude_parents=False,\n ... replace_count=3, )\n <BLANKLINE>\n ...Path('a/path/to/03/f-003-05-00003.txt')...\n ```\n \"\"\"\nif match_int is None and match_str in (None, \"\"):\nraise ValueError(f\"At least `match_int` or `match_str` required; both None.\")\nelif match_str and not match_int:\nmatch_int = int(match_str)\nelif match_int is not None and not match_str:\nassert str(match_int) in str(file_path)\nmatch_str = int_from_str(\nstr(file_path),\nindex=-1 if reverse_int_match else 0,\n)[0]\nassert match_int is not None and match_str is not None\nif exclude_parents:\nreturn Path(file_path).parent / Path(file_path).name.replace(\nmatch_str, str(match_int).zfill(padding), replace_count\n)\nelse:\nreturn Path(\nstr(file_path).replace(\nmatch_str, str(match_int).zfill(padding), replace_count\n)\n)\nsave_fixture(\ngenerator: Sequence | Generator = [],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nadd_created: bool = True,\njson_indent: int = JSON_INDENT,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None\nSaves fixtures generated by a generator to separate JSON files.
This function takes a generator and saves the generated fixtures to separate JSON files. The fixtures are saved in batches, where each batch is determined by the max_elements_per_file parameter.
Parameters:
Name Type Description Defaultgenerator Sequence | Generator A generator that yields the fixtures to be saved.
[] prefix str A string prefix to be added to the file names of the saved fixtures.
'' output_path PathLike | str Path to folder fixtures are saved to
settings.OUTPUT max_elements_per_file int Maximum JSON records saved in each file
settings.MAX_ELEMENTS_PER_FILE add_created bool Whether to add created_at and updated_at timestamps
True json_indent int Number of indent spaces per line in saved JSON
JSON_INDENT file_name_0_padding int Zeros to prefix the number of each fixture file name.
FILE_NAME_0_PADDING_DEFAULT Returns:
Type DescriptionNone This function saves the fixtures to files but does not return
None any value.
Example>>> tmp_path: Path = getfixture('tmp_path')\n>>> save_fixture(NEWSPAPER_COLLECTION_METADATA,\n... prefix='test', output_path=tmp_path)\n>>> imported_fixture = load_json(tmp_path / 'test-000001.json')\n>>> imported_fixture[1]['pk']\n2\n>>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n'hmd'\n>>> 'created_at' in imported_fixture[1]['fields']\nTrue\nalto2txt2fixture/utils.py def save_fixture(\ngenerator: Sequence | Generator = [],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nadd_created: bool = True,\njson_indent: int = JSON_INDENT,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None:\n\"\"\"Saves fixtures generated by a generator to separate JSON files.\n This function takes a generator and saves the generated fixtures to\n separate JSON files. The fixtures are saved in batches, where each batch\n is determined by the ``max_elements_per_file`` parameter.\n Args:\n generator:\n A generator that yields the fixtures to be saved.\n prefix:\n A string prefix to be added to the file names of the\n saved fixtures.\n output_path:\n Path to folder fixtures are saved to\n max_elements_per_file:\n Maximum `JSON` records saved in each file\n add_created:\n Whether to add `created_at` and `updated_at` `timestamps`\n json_indent:\n Number of indent spaces per line in saved `JSON`\n file_name_0_padding:\n Zeros to prefix the number of each fixture file name.\n Returns:\n This function saves the fixtures to files but does not return\n any value.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> save_fixture(NEWSPAPER_COLLECTION_METADATA,\n ... prefix='test', output_path=tmp_path)\n >>> imported_fixture = load_json(tmp_path / 'test-000001.json')\n >>> imported_fixture[1]['pk']\n 2\n >>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n >>> 'created_at' in imported_fixture[1]['fields']\n True\n ```\n \"\"\"\ninternal_counter = 1\ncounter = 1\nlst = []\nfile_name: str\nPath(output_path).mkdir(parents=True, exist_ok=True)\nfor item in generator:\nlst.append(item)\ninternal_counter += 1\nif internal_counter > max_elements_per_file:\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.json\"\nwrite_json(\np=Path(f\"{output_path}/file_name\"),\no=lst,\nadd_created=add_created,\njson_indent=json_indent,\n)\n# Save up some memory\ndel lst\ngc.collect()\n# Re-instantiate\nlst = []\ninternal_counter = 1\ncounter += 1\nelse:\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.json\"\nwrite_json(\np=Path(f\"{output_path}/{file_name}\"),\no=lst,\nadd_created=add_created,\njson_indent=json_indent,\n)\nreturn\ntruncate_path_str(\npath: PathLike,\nmax_length: int = MAX_TRUNCATE_PATH_STR_LEN,\nfolder_filler_str: str = INTERMEDIATE_PATH_TRUNCATION_STR,\nhead_parts: int = TRUNC_HEADS_PATH_DEFAULT,\ntail_parts: int = TRUNC_TAILS_PATH_DEFAULT,\npath_sep: str = sep,\n_force_type: Type[Path] | Type[PureWindowsPath] = Path,\n) -> str\nIf len(text) > max_length return text followed by trail_str.
Parameters:
Name Type Description Defaultpath PathLike PathLike object to truncate
max_length int maximum length of path to allow, anything belond truncated
MAX_TRUNCATE_PATH_STR_LEN folder_filler_str str what to fill intermediate path names with
INTERMEDIATE_PATH_TRUNCATION_STR head_parts int how many parts of path from the root to keep. These must be int >= 0
TRUNC_HEADS_PATH_DEFAULT tail_parts int how many parts from the path tail the root to keep. These must be int >= 0
TRUNC_TAILS_PATH_DEFAULT path_sep str what str to replace path parts with if over max_length
sep Returns:
Type Descriptionstr text truncated to max_length (if longer than max_length), with with folder_filler_str for intermediate folder names
For errors running on windows see: #56
Example>>> love_shadows: Path = (\n... Path('Standing') / 'in' / 'the' / 'shadows'/ 'of' / 'love.')\n>>> truncate_path_str(love_shadows)\n'Standing...love.'\n>>> truncate_path_str(love_shadows, max_length=100)\n'Standing...in...the...shadows...of...love.'\n>>> truncate_path_str(love_shadows, folder_filler_str=\"*\")\n'Standing...*...*...*...*...love.'\n>>> root_love_shadows: Path = Path(sep) / love_shadows\n>>> truncate_path_str(root_love_shadows, folder_filler_str=\"*\")\n<BLANKLINE>\n...\n'...Standing...*...*...*...*...love.'\n>>> if is_platform_win:\n... pytest.skip('fails on certain Windows root paths: issue #56')\n>>> truncate_path_str(root_love_shadows,\n... folder_filler_str=\"*\", tail_parts=3)\n<BLANKLINE>\n...\n'...Standing...*...*...shadows...of...love.'...\nalto2txt2fixture/utils.py def truncate_path_str(\npath: PathLike,\nmax_length: int = MAX_TRUNCATE_PATH_STR_LEN,\nfolder_filler_str: str = INTERMEDIATE_PATH_TRUNCATION_STR,\nhead_parts: int = TRUNC_HEADS_PATH_DEFAULT,\ntail_parts: int = TRUNC_TAILS_PATH_DEFAULT,\npath_sep: str = sep,\n_force_type: Type[Path] | Type[PureWindowsPath] = Path,\n) -> str:\n\"\"\"If `len(text) > max_length` return `text` followed by `trail_str`.\n Args:\n path:\n `PathLike` object to truncate\n max_length:\n maximum length of `path` to allow, anything belond truncated\n folder_filler_str:\n what to fill intermediate path names with\n head_parts:\n how many parts of `path` from the root to keep.\n These must be `int` >= 0\n tail_parts:\n how many parts from the `path` tail the root to keep.\n These must be `int` >= 0\n path_sep:\n what `str` to replace `path` parts with if over `max_length`\n Returns:\n `text` truncated to `max_length` (if longer than `max_length`),\n with with `folder_filler_str` for intermediate folder names\n Note:\n For errors running on windows see:\n [#56](https://github.com/Living-with-machines/alto2txt2fixture/issues/56)\n Example:\n ```pycon\n >>> love_shadows: Path = (\n ... Path('Standing') / 'in' / 'the' / 'shadows'/ 'of' / 'love.')\n >>> truncate_path_str(love_shadows)\n 'Standing...love.'\n >>> truncate_path_str(love_shadows, max_length=100)\n 'Standing...in...the...shadows...of...love.'\n >>> truncate_path_str(love_shadows, folder_filler_str=\"*\")\n 'Standing...*...*...*...*...love.'\n >>> root_love_shadows: Path = Path(sep) / love_shadows\n >>> truncate_path_str(root_love_shadows, folder_filler_str=\"*\")\n <BLANKLINE>\n ...\n '...Standing...*...*...*...*...love.'\n >>> if is_platform_win:\n ... pytest.skip('fails on certain Windows root paths: issue #56')\n >>> truncate_path_str(root_love_shadows,\n ... folder_filler_str=\"*\", tail_parts=3)\n <BLANKLINE>\n ...\n '...Standing...*...*...shadows...of...love.'...\n ```\n \"\"\"\npath = _force_type(normpath(path))\nif len(str(path)) > max_length:\ntry:\nassert not (head_parts < 0 or tail_parts < 0)\nexcept AssertionError:\nlogger.error(\nf\"Both index params for `truncate_path_str` must be >=0: \"\nf\"(head_parts={head_parts}, tail_parts={tail_parts})\"\n)\nreturn str(path)\noriginal_path_parts: tuple[str, ...] = path.parts\nhead_index_fix: int = 0\nif path.is_absolute() or path.drive:\nhead_index_fix += 1\nfor part in original_path_parts[head_parts + head_index_fix :]:\nif not part:\nhead_index_fix += 1\nelse:\nbreak\nlogger.debug(\nf\"Adding {head_index_fix} to `head_parts`: {head_parts} \"\nf\"to truncate: '{path}'\"\n)\nhead_parts += head_index_fix\ntry:\nassert head_parts + tail_parts < len(str(original_path_parts))\nexcept AssertionError:\nlogger.error(\nf\"Returning untruncated. Params \"\nf\"(head_parts={head_parts}, tail_parts={tail_parts}) \"\nf\"not valid to truncate: '{path}'\"\n)\nreturn str(path)\ntail_index: int = len(original_path_parts) - tail_parts\nreplaced_path_parts: tuple[str, ...] = tuple(\npart if (i < head_parts or i >= tail_index) else folder_filler_str\nfor i, part in enumerate(original_path_parts)\n)\nreplaced_start_str: str = \"\".join(replaced_path_parts[:head_parts])\nreplaced_end_str: str = path_sep.join(\npath for path in replaced_path_parts[head_parts:]\n)\nreturn path_sep.join((replaced_start_str, replaced_end_str))\nelse:\nreturn str(path)\nvalid_compression_files(files: Sequence[PathLike]) -> list[PathLike]\nReturn a tuple of valid compression paths in files.
Parameters:
Name Type Description Defaultfiles Sequence[PathLike] Sequence of files to filter compression types from.
Returns:
Type Descriptionlist[PathLike] A list of files that could be decompressed.
Example>>> valid_compression_files([\n... 'cat.tar.bz2', 'dog.tar.bz3', 'fish.tgz', 'bird.zip',\n... 'giraffe.txt', 'frog'\n... ])\n['cat.tar.bz2', 'fish.tgz', 'bird.zip']\nalto2txt2fixture/utils.py def valid_compression_files(files: Sequence[PathLike]) -> list[PathLike]:\n\"\"\"Return a `tuple` of valid compression paths in `files`.\n Args:\n files:\n `Sequence` of files to filter compression types from.\n Returns:\n A list of files that could be decompressed.\n Example:\n ```pycon\n >>> valid_compression_files([\n ... 'cat.tar.bz2', 'dog.tar.bz3', 'fish.tgz', 'bird.zip',\n ... 'giraffe.txt', 'frog'\n ... ])\n ['cat.tar.bz2', 'fish.tgz', 'bird.zip']\n ```\n \"\"\"\nreturn [\nfile\nfor file in files\nif \"\".join(Path(file).suffixes) in VALID_COMPRESSION_FORMATS\n]\nwrite_json(\np: str | Path,\no: dict,\nadd_created: bool = True,\njson_indent: int = JSON_INDENT,\n) -> None\nEasier access to writing json files. Checks whether parent exists.
Parameters:
Name Type Description Defaultp str | Path Path to write json to
o dict Object to write to json file
add_created bool If set to True will add created_at and updated_at to the dictionary's fields. If created_at and updated_at already exist in the fields, they will be forcefully updated.
True json_indent int What indetation format to write out JSON file in
JSON_INDENT Returns:
Type DescriptionNone None
Example>>> tmp_path: Path = getfixture('tmp_path')\n>>> path: Path = tmp_path / 'test-write-json-example.json'\n>>> write_json(p=path,\n... o=NEWSPAPER_COLLECTION_METADATA,\n... add_created=True)\n>>> imported_fixture = load_json(path)\n>>> imported_fixture[1]['pk']\n2\n>>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n'hmd'\nalto2txt2fixture/utils.py def write_json(\np: str | Path, o: dict, add_created: bool = True, json_indent: int = JSON_INDENT\n) -> None:\n\"\"\"\n Easier access to writing `json` files. Checks whether parent exists.\n Args:\n p: Path to write `json` to\n o: Object to write to `json` file\n add_created:\n If set to True will add `created_at` and `updated_at`\n to the dictionary's fields. If `created_at` and `updated_at`\n already exist in the fields, they will be forcefully updated.\n json_indent:\n What indetation format to write out `JSON` file in\n Returns:\n None\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> path: Path = tmp_path / 'test-write-json-example.json'\n >>> write_json(p=path,\n ... o=NEWSPAPER_COLLECTION_METADATA,\n ... add_created=True)\n >>> imported_fixture = load_json(path)\n >>> imported_fixture[1]['pk']\n 2\n >>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n ```\n `\n \"\"\"\np = get_path_from(p)\nif not (isinstance(o, dict) or isinstance(o, list)):\nraise RuntimeError(f\"Unable to handle data of type: {type(o)}\")\ndef _append_created_fields(o: dict):\n\"\"\"Add `created_at` and `updated_at` fields to a `dict` with `FixtureDict` values.\"\"\"\nreturn dict(\n**{k: v for k, v in o.items() if not k == \"fields\"},\nfields=dict(\n**{\nk: v\nfor k, v in o[\"fields\"].items()\nif not k == \"created_at\" and not k == \"updated_at\"\n},\n**{\"created_at\": NOW_str, \"updated_at\": NOW_str},\n),\n)\ntry:\nif add_created and isinstance(o, dict):\no = _append_created_fields(o)\nelif add_created and isinstance(o, list):\no = [_append_created_fields(x) for x in o]\nexcept KeyError:\nerror(\"An unknown error occurred (in write_json)\")\np.parent.mkdir(parents=True, exist_ok=True)\np.write_text(json.dumps(o, indent=json_indent))\nreturn\nThe installation process should be fairly easy to take care of, using poetry:
$ poetry install\nHowever, this is only the first step in the process. As the script works through the alto2txt collections, you will either need to choose the slower option \u2014 mounting them to your computer (using blobfuse) \u2014\u00a0or the faster option \u2014 downloading the required zip files from the Azure storage to your local hard drive. In the two following sections, both of those options are described.
alto2txt to the program","text":""},{"location":"tutorial/first-steps.html#downloading-local-copies-of-alto2txt-on-your-computer","title":"Downloading local copies of alto2txt on your computer","text":"This option will take up a lot of hard drive space
As of the time of writing, downloading all of alto2txt\u2019s metadata takes up about 185GB on your local drive.
You do not have to download all of the collections or all of the zip files for each collection, as long as you are aware that the resulting fixtures will be limited in scope.
"},{"location":"tutorial/first-steps.html#step-1-log-in-to-azure-using-microsoft-azure-storage-explorer","title":"Step 1: Log in to Azure using Microsoft Azure Storage Explorer","text":"Microsoft Azure Storage Explorer (MASE) is a great and free tool for downloading content off Azure. Your first step is to download and install this product on your local computer.
Once you have opened MASE, you will need to sign into the appropriate Azure account.
"},{"location":"tutorial/first-steps.html#step-2-download-the-alto2txt-blob-container-to-your-hard-drive","title":"Step 2: Download thealto2txt blob container to your hard drive","text":"On your left-hand side, you should see a menu where you can navigate to the correct \u201cblob container\u201d: Living with Machines > Storage Accounts > alto2txt > Blob Containers:
You will want to replicate the same structure as the Blob Container itself in a folder on your hard drive:
Once you have the structure set up, you are ready to download all of the files needed. For each of the blob containers, make sure that you download the metadata directory only onto your computer:
Select all of the files and press the download button:
Make sure you save all the zip files inside the correct local folder:
The \u201cActivities\u201d bar will now show you the progress and speed:
"},{"location":"tutorial/first-steps.html#mounting-alto2txt-on-your-computer","title":"Mountingalto2txt on your computer","text":"This option will only work on a Linux or UNIX computer
If you have a mac, your only option is the one below.
"},{"location":"tutorial/first-steps.html#step-1-install-blobfuse","title":"Step 1: Install BlobFuse","text":"Follow the instructions for installing BlobFuse and the instructions for how to prepare your drive for mounting.
"},{"location":"tutorial/first-steps.html#step-2-set-up-sas-tokens","title":"Step 2: Set up SAS tokens","text":"Follow the instructions for setting up access to your Azure storage account.
"},{"location":"tutorial/first-steps.html#step-3-mount-your-blobs","title":"Step 3: Mount your blobs","text":"TODO #3: Write this section.
Note that you can also search on the internet for ideas on how to create local scripts to facilitate easier connection next time.
"}]} \ No newline at end of file diff --git a/search/search_index.json b/search/search_index.json new file mode 100644 index 0000000..12055a1 --- /dev/null +++ b/search/search_index.json @@ -0,0 +1 @@ +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"index.html","title":"alto2txt2fixture","text":"alto2txt2fixture is a standalone tool to convert alto2txt XML output and other related datasets into JSON (and where feasible CSV) data with corresponding relational IDs to ease general use and ingestion into a relational database.
We target the the JSON produced for importing into lwmdb: a database built using the Django python webframework database fixture structure.
We provide a command line interface to process alto2txt XML files stored locally (or mounted via azure blobfuse), and for additional public data we automate a means of downloading those automatically.
We recommend downloading a copy of the reposity or using git clone. From a local copy use poetry to install dependencies:
$ cd alto2txt2fixture\n$ poetry install\nIf you would like to test, render documentation and/or contribute to the code included dev dependencies in a local install:
$ poetry install --with dev\nTo processing newspaper metadata with a local copy of alto2txt XML results, it's easiest to have that data in the same folder as your alto2txt2fixture checkout and poetry installed folder. One arranged, you should be able to begin the JSON converstion with
$ poetry run a2t2f-news\nTo generate related data in JSON and CSV form, assuming you have an internet collection and access to a living-with-machines azure account, the following will download related data into JSON and CSV files. The JSON results should be consistent with lwmdb tables for ease of import.
$ poetry run a2t2f-adj\npoetry to run","text":"The program should run automatically with the following command:
$ poetry run a2t2f-news\nAlternatively, if you want to add optional parameters and don\u2019t want to use the standard poetry script to run, you can use the (somewhat convoluted) poetry run alto2txt2fixture/run.py and provide any optional parameters. You can see a list of all the \u201cOptional parameters\u201d below. For example, if you want to only include the hmd collection:
$ poetry run alto2txt2fixture/run.py --collections hmd\nIf you find yourself in trouble with poetry, the program should run perfectly fine on its own, assuming the dependencies are installed. The same command, then, would be:
$ python alto2txt2fixture/run.py --collections hmd\nNote
See the list under [tool.poetry.dependencies] in pyproject.toml for a list of dependencies that would need to be installed for alto2txt2fixture to work outside a python poetry environment.
The program has a number of optional parameters that you can choose to include or not. The table below describes each parameter, how to pass it to the program, and what its defaults are.
Flag Description Default value-c, --collections Which collections to process in the mounted alto2txt directory hmd, lwm, jisc, bna -o, --output Into which directory should the processed files be put? ./output/fixtures/ -m, --mountpoint Where is the alto2txt directories mounted? ./input/alto2txt/ -t, --test-config Print the config table but do not run False"},{"location":"running.html#successfully-running-the-program-an-example","title":"Successfully running the program: An example","text":""},{"location":"understanding-results.html","title":"Understanding the Results","text":""},{"location":"understanding-results.html#the-resulting-file-structure","title":"The resulting file structure","text":"The examples below follow standard settings
If you choose other settings for when you run the program, your output directory may look different from the information on this page.
"},{"location":"understanding-results.html#reports","title":"Reports","text":"Reports are automatically generated with a unique hash as the overarching folder structure. Inside the reports directory, you\u2019ll find a JSON file for each alto2txt directory (organised by NLP identifier).
The report structure, thus, looks like this:
The JSON file has some good troubleshooting information. You\u2019ll find that the contents are structured as a Python dictionary (or JavaScript Object). Here is an example:
Here is an explanation of each of the keys in the dictionary:
Key Explanation Data typepath The input path for the zip file that is being converted. string bytes The size of the input zip file represented in bytes. integer size The size of the input zip file represented in a human-readable string. string contents #TODO #3 integer start Date and time when processing started (see also end below). datestring newspaper_paths #TODO #3 list (string) publication_codes A list of the NLPs that are contained in the input zip file. list (string) issue_paths A list of all the issue paths that are contained in the cache directory. list (string) item_paths A list of all the item paths that are contained in the cache directory. list (string) end Date and time when processing ended (see also start above). datestring seconds Seconds that the script spent interpreting the zip file (should be added to the microseconds below). integer microseconds Microseconds that the script spent interpreting the zip file (should be added to the seconds above). integer"},{"location":"understanding-results.html#fixtures","title":"Fixtures","text":"The most important output of the script is contained in the fixtures directory. This directory contains JSON files for all the different columns in the corresponding Django metadata database (i.e. DataProvider, Digitisation, Ingest, Issue, Newspaper, and Item). The numbering at the end of each file indicates the order of the files as they are divided into a maximum of 2e6 elements*:
Each JSON file contains a Python-like list (JavaScript Array) of dictionaries (JavaScript Objects), which have a primary key (pk), the related database model (in the example below the Django newspapers app\u2019s newspaper table), and a nested dictionary/Object which contains all the values for the database\u2019s table entry:
* The maximum elements per file can be adjusted in the settings.py file\u2019s settings object\u2019s MAX_ELEMENTS_PER_FILE value.
Entry point for alto2txt2fixture.parse to convert alto2txt XML -> JSON.
This module defines the run function which is the main driver for the entire process.
It imports various functions from other modules and uses them to route and parse XML data generated by alto2txt.
The following steps are performed in the run function:
alto2txt data into subdirectories with structured files.JSON files.If the script is run as a main program (i.e. if the name of the script is __main__), the run() function is executed.
Note: at present this does not include any functunality in create_adjacent_tables.py
parse_args(argv: list[str] | None = None) -> Namespace\nManage command line arguments for run()
This constructs an ArgumentParser instance to manage configurating calls of run() to manage newspaper XML to JSON converstion.
Parameters:
Name Type Description Defaultargv list[str] | None If None treat as equivalent of ['--help], if alistofstrpass those options toArgumentParser`
None Returns:
Type DescriptionNamespace A Namespace dict-like configuration for run()
alto2txt2fixture/__main__.py def parse_args(argv: list[str] | None = None) -> Namespace:\n\"\"\"Manage command line arguments for `run()`\n This constructs an `ArgumentParser` instance to manage\n configurating calls of `run()` to manage `newspaper`\n `XML` to `JSON` converstion.\n Arguments:\n argv:\n If `None` treat as equivalent of ['--help`],\n if a `list` of `str` pass those options to `ArgumentParser`\n Returns:\n A `Namespace` `dict`-like configuration for `run()`\n \"\"\"\nargv = None if not argv else argv\nparser = ArgumentParser(\nprog=\"a2t2f-news\",\ndescription=\"Process alto2txt XML into and Django JSON Fixture files\",\nepilog=(\n\"Note: this is still in beta mode and contributions welcome\\n\\n\" + __doc__\n),\nformatter_class=RawTextHelpFormatter,\n)\nparser.add_argument(\n\"-c\",\n\"--collections\",\nnargs=\"+\",\nhelp=\"<Optional> Set collections\",\nrequired=False,\n)\nparser.add_argument(\n\"-m\",\n\"--mountpoint\",\ntype=str,\nhelp=\"<Optional> Mountpoint\",\nrequired=False,\n)\nparser.add_argument(\n\"-o\",\n\"--output\",\ntype=str,\nhelp=\"<Optional> Set an output directory\",\nrequired=False,\n)\nparser.add_argument(\n\"-t\",\n\"--test-config\",\ndefault=False,\nhelp=\"Only print the configuration\",\naction=BooleanOptionalAction,\n)\nparser.add_argument(\n\"-f\",\n\"--show-fixture-tables\",\ndefault=True,\nhelp=\"Print included fixture table configurations\",\naction=BooleanOptionalAction,\n)\nparser.add_argument(\n\"--export-fixture-tables\",\ndefault=True,\nhelp=\"Experimental: export fixture tables prior to data processing\",\naction=BooleanOptionalAction,\n)\nparser.add_argument(\n\"--data-provider-field\",\ntype=str,\ndefault=DATA_PROVIDER_INDEX,\nhelp=\"Key for indexing DataProvider records\",\n)\nreturn parser.parse_args(argv)\nrun(local_args: list[str] | None = None) -> None\nManage running newspaper XML to JSON conversion.
First parse_args is called for command line arguments including:
collectionsoutputmountpointIf any of these arguments are specified, they will be used, otherwise they will default to the values in the settings module.
The show_setup function is then called to display the configurations being used.
The route function is then called to route the alto2txt files into subdirectories with structured files.
The parse function is then called to parse the resulting JSON files.
Finally, the clear_cache function is called to clear the cache (pending the user's confirmation).
Parameters:
Name Type Description Defaultlocal_args list[str] | None Options passed to parse_args()
None Returns:
Type DescriptionNone None
Source code inalto2txt2fixture/__main__.py def run(local_args: list[str] | None = None) -> None:\n\"\"\"Manage running newspaper `XML` to `JSON` conversion.\n First `parse_args` is called for command line arguments including:\n - `collections`\n - `output`\n - `mountpoint`\n If any of these arguments are specified, they will be used, otherwise they\n will default to the values in the `settings` module.\n The `show_setup` function is then called to display the configurations\n being used.\n The `route` function is then called to route the alto2txt files into\n subdirectories with structured files.\n The `parse` function is then called to parse the resulting JSON files.\n Finally, the `clear_cache` function is called to clear the cache\n (pending the user's confirmation).\n Arguments:\n local_args: Options passed to `parse_args()`\n Returns:\n None\n \"\"\"\nargs: Namespace = parse_args(argv=local_args)\nif args.collections:\nCOLLECTIONS = [x.lower() for x in args.collections]\nelse:\nCOLLECTIONS = settings.COLLECTIONS\nif args.output:\nOUTPUT = args.output.rstrip(\"/\")\nelse:\nOUTPUT = settings.OUTPUT\nif args.mountpoint:\nMOUNTPOINT = args.mountpoint.rstrip(\"/\")\nelse:\nMOUNTPOINT = settings.MOUNTPOINT\nshow_setup(\nCOLLECTIONS=COLLECTIONS,\nOUTPUT=OUTPUT,\nCACHE_HOME=settings.CACHE_HOME,\nMOUNTPOINT=MOUNTPOINT,\nJISC_PAPERS_CSV=settings.JISC_PAPERS_CSV,\nREPORT_DIR=settings.REPORT_DIR,\nMAX_ELEMENTS_PER_FILE=settings.MAX_ELEMENTS_PER_FILE,\n)\nif args.show_fixture_tables:\n# Show a table of fixtures used, defaults to DataProvider Table\nshow_fixture_tables(settings, data_provider_index=args.data_provider_field)\nif args.export_fixture_tables:\nexport_fixtures(\nfixture_tables=settings.FIXTURE_TABLES,\npath=OUTPUT,\nformats=settings.FIXTURE_TABLES_FORMATS,\n)\nif not args.test_config:\n# Routing alto2txt into subdirectories with structured files\nroute(\nCOLLECTIONS,\nsettings.CACHE_HOME,\nMOUNTPOINT,\nsettings.JISC_PAPERS_CSV,\nsettings.REPORT_DIR,\n)\n# Parsing the resulting JSON files\nparse(\nCOLLECTIONS,\nsettings.CACHE_HOME,\nOUTPUT,\nsettings.MAX_ELEMENTS_PER_FILE,\n)\nclear_cache(settings.CACHE_HOME)\nfile_rename_table(\npaths_dict: dict[os.PathLike, os.PathLike],\ncompress_format: ArchiveFormatEnum = COMPRESSION_TYPE_DEFAULT,\ntitle: str = FILE_RENAME_TABLE_TITLE_DEFAULT,\nprefix: str = \"\",\nrenumber: bool = True,\n) -> Table\nCreate a rich.Table of rename configuration.
Parameters:
Name Type Description Defaultpaths_dict dict[os.PathLike, os.PathLike] dict[os.PathLike, os.PathLike], Original and renumbered paths dict
compress_format ArchiveFormatEnum Which ArchiveFormatEnum for compression
COMPRESSION_TYPE_DEFAULT title str Title of returned Table
FILE_RENAME_TABLE_TITLE_DEFAULT prefix str str to add in front of every new path
'' renumber bool Whether an int in each path will be renumbered.
True Source code in alto2txt2fixture/cli.py def file_rename_table(\npaths_dict: dict[os.PathLike, os.PathLike],\ncompress_format: ArchiveFormatEnum = COMPRESSION_TYPE_DEFAULT,\ntitle: str = FILE_RENAME_TABLE_TITLE_DEFAULT,\nprefix: str = \"\",\nrenumber: bool = True,\n) -> Table:\n\"\"\"Create a `rich.Table` of rename configuration.\n Args:\n paths_dict: dict[os.PathLike, os.PathLike],\n Original and renumbered `paths` `dict`\n compress_format:\n Which `ArchiveFormatEnum` for compression\n title:\n Title of returned `Table`\n prefix:\n `str` to add in front of every new path\n renumber:\n Whether an `int` in each path will be renumbered.\n \"\"\"\ntable: Table = Table(title=title)\ntable.add_column(\"Current File Name\", justify=\"right\", style=\"cyan\")\ntable.add_column(\"New File Name\", style=\"magenta\")\ndef final_file_name(name: os.PathLike) -> str:\nreturn (\nprefix\n+ str(Path(name).name)\n+ (f\".{compress_format}\" if compress_format else \"\")\n)\nfor old_path, new_path in paths_dict.items():\nname: str = final_file_name(new_path if renumber else old_path)\ntable.add_row(Path(old_path).name, name)\nreturn table\nfunc_table(\nfunc: Callable,\nvalues: dict,\ntitle: str = \"\",\nextra_dict: dict[str, Any] = {},\n) -> Table\nGeneate rich Table from func signature and help attr.
Parameters:
Name Type Description Defaultfunc Callable Function whose args and type hints will be converted to a table.
values dict dict of variables covered in func signature. local() often suffices.
title str str for table title.
'' extra_dict dict[str, Any] A dict of additional rows to add to the table. For each key, value pair: if the value is a tuple, it will be expanded to match the Type, Value, and Notes columns; else the Type will be inferred and Notes left blank.
{} Example >>> def test_func(\n... var_a: Annotated[str, typer.Option(help=\"Example\")] = \"Default\"\n... ) -> None:\n... test_func_table: Table = func_table(test_func, values=vars())\n... console.print(test_func_table)\n>>> if is_platform_win:\n... pytest.skip('fails on certain Windows root paths: issue #56')\n>>> test_func()\n test_func config\n\u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n\u2503 Variable \u2503 Type \u2503 Value \u2503 Notes \u2503\n\u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n\u2502 var_a \u2502 str \u2502 Default \u2502 Example \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\nalto2txt2fixture/cli.py def func_table(\nfunc: Callable, values: dict, title: str = \"\", extra_dict: dict[str, Any] = {}\n) -> Table:\n\"\"\"Geneate `rich` `Table` from `func` signature and `help` attr.\n Args:\n func:\n Function whose `args` and `type` hints will be converted\n to a table.\n values:\n `dict` of variables covered in `func` signature.\n `local()` often suffices.\n title:\n `str` for table title.\n extra_dict:\n A `dict` of additional rows to add to the table. For each\n `key`, `value` pair: if the `value` is a `tuple`, it will\n be expanded to match the `Type`, `Value`, and `Notes`\n columns; else the `Type` will be inferred and `Notes`\n left blank.\n Example:\n ```pycon\n >>> def test_func(\n ... var_a: Annotated[str, typer.Option(help=\"Example\")] = \"Default\"\n ... ) -> None:\n ... test_func_table: Table = func_table(test_func, values=vars())\n ... console.print(test_func_table)\n >>> if is_platform_win:\n ... pytest.skip('fails on certain Windows root paths: issue #56')\n >>> test_func()\n test_func config\n \u250f\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2533\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2513\n \u2503 Variable \u2503 Type \u2503 Value \u2503 Notes \u2503\n \u2521\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2547\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2529\n \u2502 var_a \u2502 str \u2502 Default \u2502 Example \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n ```\n \"\"\"\ntitle = title if title else f\"{func.__name__} config\"\nfunc_signature: dict = get_type_hints(func, include_extras=True)\ntable: Table = Table(title=title)\ntable.add_column(\"Variable\", justify=\"right\", style=\"cyan\")\ntable.add_column(\"Type\", style=\"yellow\")\ntable.add_column(\"Value\", style=\"magenta\")\ntable.add_column(\"Notes\")\nfor var, info in func_signature.items():\ntry:\nvar_type, annotation = get_args(info)\nvalue: Any = values[var]\nif value in (\"\", \"\"):\nvalue = \"''\"\ntable.add_row(str(var), var_type.__name__, str(value), annotation.help)\nexcept ValueError:\ncontinue\nfor key, val in extra_dict.items():\nif isinstance(val, tuple):\ntable.add_row(key, *val)\nelse:\ntable.add_row(key, type(val).__name__, str(val))\nreturn table\nplaintext(\npath: Annotated[Path, typer.Argument(help=\"Path to raw plaintext files\")],\nsave_path: Annotated[\nPath, typer.Option(help=\"Path to save json export files\")\n] = Path(DEFAULT_PLAINTEXT_FIXTURE_OUTPUT),\ndata_provider_code: Annotated[\nstr, typer.Option(help=\"Data provider code use existing config\")\n] = \"\",\nextract_path: Annotated[\nPath, typer.Option(help=\"Folder to extract compressed raw plaintext to\")\n] = Path(DEFAULT_EXTRACTED_SUBDIR),\ninitial_pk: Annotated[\nint,\ntyper.Option(help=\"First primary key to increment json export from\"),\n] = DEFAULT_INITIAL_PK,\nrecords_per_json: Annotated[\nint, typer.Option(help=\"Max records per json fixture\")\n] = DEFAULT_MAX_PLAINTEXT_PER_FIXTURE_FILE,\ndigit_padding: Annotated[\nint,\ntyper.Option(help=\"Padding '0's for indexing json fixture filenames\"),\n] = FILE_NAME_0_PADDING_DEFAULT,\ncompress: Annotated[\nbool, typer.Option(help=\"Compress json fixtures\")\n] = False,\ncompress_path: Annotated[\nPath, typer.Option(help=\"Folder to compress json fixtueres to\")\n] = Path(COMPRESSED_PATH_DEFAULT),\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\n) -> None\nCreate a PlainTextFixture and save to save_path.
alto2txt2fixture/cli.py @cli.command()\ndef plaintext(\npath: Annotated[Path, typer.Argument(help=\"Path to raw plaintext files\")],\nsave_path: Annotated[\nPath, typer.Option(help=\"Path to save json export files\")\n] = Path(DEFAULT_PLAINTEXT_FIXTURE_OUTPUT),\ndata_provider_code: Annotated[\nstr, typer.Option(help=\"Data provider code use existing config\")\n] = \"\",\nextract_path: Annotated[\nPath, typer.Option(help=\"Folder to extract compressed raw plaintext to\")\n] = Path(DEFAULT_EXTRACTED_SUBDIR),\ninitial_pk: Annotated[\nint, typer.Option(help=\"First primary key to increment json export from\")\n] = DEFAULT_INITIAL_PK,\nrecords_per_json: Annotated[\nint, typer.Option(help=\"Max records per json fixture\")\n] = DEFAULT_MAX_PLAINTEXT_PER_FIXTURE_FILE,\ndigit_padding: Annotated[\nint, typer.Option(help=\"Padding '0's for indexing json fixture filenames\")\n] = FILE_NAME_0_PADDING_DEFAULT,\ncompress: Annotated[bool, typer.Option(help=\"Compress json fixtures\")] = False,\ncompress_path: Annotated[\nPath, typer.Option(help=\"Folder to compress json fixtueres to\")\n] = Path(COMPRESSED_PATH_DEFAULT),\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\n) -> None:\n\"\"\"Create a PlainTextFixture and save to `save_path`.\"\"\"\nplaintext_fixture = PlainTextFixture(\npath=path,\ndata_provider_code=data_provider_code,\nextract_subdir=extract_path,\nexport_directory=save_path,\ninitial_pk=initial_pk,\nmax_plaintext_per_fixture_file=records_per_json,\njson_0_file_name_padding=digit_padding,\njson_export_compression_format=compress_format,\njson_export_compression_subdir=compress_path,\n)\nplaintext_fixture.info()\nwhile (\nnot plaintext_fixture.compressed_files\nand not plaintext_fixture.plaintext_provided_uncompressed\n):\ntry_another_compressed_txt_source: bool = Confirm.ask(\nf\"No .txt files available from extract path: \"\nf\"{plaintext_fixture.trunc_extract_path_str}\\n\"\nf\"Would you like to extract fixtures from a different path?\",\ndefault=\"n\",\n)\nif try_another_compressed_txt_source:\nnew_extract_path: str = Prompt.ask(\"Please enter a new extract path\")\nplaintext_fixture.path = Path(new_extract_path)\nelse:\nreturn\nplaintext_fixture.info()\nplaintext_fixture.extract_compressed()\nplaintext_fixture.export_to_json_fixtures()\nif compress:\nplaintext_fixture.compress_json_exports()\nrename(\npath: Annotated[Path, typer.Argument(help=\"Path to files to manage\")],\nfolder: Annotated[\nPath, typer.Option(help=\"Path under `path` for new files\")\n] = Path(),\nrenumber: Annotated[\nbool, typer.Option(help=\"Show changes without applying\")\n] = False,\nregex: Annotated[str, typer.Option(help=\"Regex to filter files\")] = \"*.txt\",\npadding: Annotated[\nint, typer.Option(help=\"Digits to pad file name\")\n] = FILE_NAME_0_PADDING_DEFAULT,\nprefix: Annotated[str, typer.Option(help=\"Prefix for new file names\")] = \"\",\ndry_run: Annotated[\nbool, typer.Option(help=\"Show changes without applying\")\n] = True,\ncompress: Annotated[\nbool, typer.Option(help=\"Whether to compress files\")\n] = False,\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\ncompress_suffix: Annotated[\nstr, typer.Option(help=\"Compressed file name suffix\")\n] = \"\",\ncompress_folder: Annotated[\nPath, typer.Option(help=\"Optional folder to differ from renaming\")\n] = COMPRESSED_PATH_DEFAULT,\ndelete_uncompressed: Annotated[\nbool, typer.Option(help=\"Delete unneeded files after compression\")\n] = False,\nlog_level: Annotated[\nint, typer.Option(help=\"Set logging level for debugging\")\n] = WARNING,\nforce: Annotated[\nbool, typer.Option(--force, help=\"Force run without prompt\")\n] = False,\n) -> None\nManage file names and compression.
Source code inalto2txt2fixture/cli.py @cli.command()\ndef rename(\npath: Annotated[Path, typer.Argument(help=\"Path to files to manage\")],\nfolder: Annotated[\nPath, typer.Option(help=\"Path under `path` for new files\")\n] = Path(),\nrenumber: Annotated[\nbool, typer.Option(help=\"Show changes without applying\")\n] = False,\nregex: Annotated[str, typer.Option(help=\"Regex to filter files\")] = \"*.txt\",\npadding: Annotated[\nint, typer.Option(help=\"Digits to pad file name\")\n] = FILE_NAME_0_PADDING_DEFAULT,\nprefix: Annotated[str, typer.Option(help=\"Prefix for new file names\")] = \"\",\ndry_run: Annotated[bool, typer.Option(help=\"Show changes without applying\")] = True,\ncompress: Annotated[bool, typer.Option(help=\"Whether to compress files\")] = False,\ncompress_format: Annotated[\nArchiveFormatEnum,\ntyper.Option(case_sensitive=False, help=\"Compression format\"),\n] = COMPRESSION_TYPE_DEFAULT,\ncompress_suffix: Annotated[\nstr, typer.Option(help=\"Compressed file name suffix\")\n] = \"\",\ncompress_folder: Annotated[\nPath, typer.Option(help=\"Optional folder to differ from renaming\")\n] = COMPRESSED_PATH_DEFAULT,\ndelete_uncompressed: Annotated[\nbool, typer.Option(help=\"Delete unneeded files after compression\")\n] = False,\nlog_level: Annotated[\nint, typer.Option(help=\"Set logging level for debugging\")\n] = WARNING,\nforce: Annotated[\nbool, typer.Option(\"--force\", help=\"Force run without prompt\")\n] = False,\n) -> None:\n\"\"\"Manage file names and compression.\"\"\"\nlogger.level = log_level\nfolder_path: Path = Path(path) / folder\ncompress_path: Path = Path(path) / compress_folder\ntry:\npaths_dict: dict[os.PathLike, os.PathLike] = glob_path_rename_by_0_padding(\npath=path,\noutput_path=folder,\nglob_regex_str=regex,\npadding=padding,\n)\nassert paths_dict\nexcept (ValueError, AssertionError, IndexError) as err:\nconsole.print(f\"Error: '{err}'\\nTried reading from path: '{path}'\")\nraise typer.Abort()\nfiles_count: int = len(paths_dict)\nif not compress and not force:\ncompress = Confirm.ask(\nf\"Compress all ({files_count}) output file(s)?\",\ndefault=\"n\",\n)\nif compress and not force:\ncompress_format = Prompt.ask(\n\"Compression format\",\nchoices=list(str(format) for format in ArchiveFormatEnum),\ndefault=compress_format,\n)\nextra_info_dict: dict[str, int | os.PathLike] = {\n\"rename_path\": folder_path,\n\"compress_path\": compress_path,\n\"HD Space (GB)\": int(free_hd_space_in_GB()),\n}\nconfig_table: Table = func_table(\nrename,\nvalues=locals(),\nextra_dict=extra_info_dict,\n)\nconsole.print(config_table)\nfile_names_table: Table = file_rename_table(\npaths_dict,\ncompress_format=compress_format,\ntitle=FILE_RENAME_TABLE_TITLE_DEFAULT,\nprefix=prefix,\nrenumber=renumber,\n)\nconsole.print(file_names_table)\nif dry_run:\nif not force:\nrenumber = Confirm.ask(\nf\"Copy {'and compress ' if compress else ''}\"\nf\"{files_count} files \"\nf\"from:\\n\\t'{path}'\\nto:\\n\\t'{folder_path}'\\n\"\n)\nif not delete_uncompressed:\ndelete_uncompressed = Confirm.ask(\nf\"Delete all uncompressed, renamed files \"\nf\"in:\\n'{folder_path}'\\n\"\nf\"after compression to '{compress_format}' format in:\"\nf\"\\n'{compress_path}'\\n\",\ndefault=\"n\",\n)\nif renumber:\ncopy_dict_paths(paths_dict)\nif compress:\nfor old_path, new_path in paths_dict.items():\nfile_path: Path = Path(new_path) if renumber else Path(old_path)\ncompress_fixture(\nfile_path,\noutput_path=compress_path,\nsuffix=compress_suffix,\nformat=compress_format,\n)\nif delete_uncompressed and renumber:\nconsole.print(f\"Deleting {new_path}\")\nPath(new_path).unlink()\nshow_fixture_tables(\nrun_settings: dotdict = settings,\nprint_in_call: bool = True,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> list[Table]\nPrint fixture tables specified in settings.fixture_tables in rich.Table format.
Parameters:
Name Type Description Defaultrun_settings dotdict alto2txt2fixture run configuration
settings print_in_call bool whether to print to console (will use console variable if so)
True data_provider_index str key to index dataprovider from NEWSPAPER_COLLECTION_METADATA
DATA_PROVIDER_INDEX Returns:
Type Descriptionlist[Table] A list of rich.Table renders from configurations in run_settings.FIXTURE_TABLES
>>> fixture_tables: list[Table] = show_fixture_tables(\n... settings,\n... print_in_call=False)\n>>> len(fixture_tables)\n1\n>>> fixture_tables[0].title\n'dataprovider'\n>>> [column.header for column in fixture_tables[0].columns]\n['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\n>>> fixture_tables = show_fixture_tables(settings)\n<BLANKLINE>\n...dataprovider...Heritage...\u2502 bl_hmd...\u2502 hmd...\nIt is possible for the example test to fail in different screen sizes. Try increasing the window or screen width of terminal used to check before raising an issue.
Source code inalto2txt2fixture/cli.py def show_fixture_tables(\nrun_settings: dotdict = settings,\nprint_in_call: bool = True,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> list[Table]:\n\"\"\"Print fixture tables specified in ``settings.fixture_tables`` in `rich.Table` format.\n Arguments:\n run_settings: `alto2txt2fixture` run configuration\n print_in_call: whether to print to console (will use ``console`` variable if so)\n data_provider_index: key to index `dataprovider` from ``NEWSPAPER_COLLECTION_METADATA``\n Returns:\n A `list` of `rich.Table` renders from configurations in ``run_settings.FIXTURE_TABLES``\n Example:\n ```pycon\n >>> fixture_tables: list[Table] = show_fixture_tables(\n ... settings,\n ... print_in_call=False)\n >>> len(fixture_tables)\n 1\n >>> fixture_tables[0].title\n 'dataprovider'\n >>> [column.header for column in fixture_tables[0].columns]\n ['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\n >>> fixture_tables = show_fixture_tables(settings)\n <BLANKLINE>\n ...dataprovider...Heritage...\u2502 bl_hmd...\u2502 hmd...\n ```\n Note:\n It is possible for the example test to fail in different screen sizes. Try\n increasing the window or screen width of terminal used to check before\n raising an issue.\n \"\"\"\nif run_settings.FIXTURE_TABLES:\nif \"dataprovider\" in run_settings.FIXTURE_TABLES:\ncheck_newspaper_collection_configuration(\nrun_settings.COLLECTIONS,\nrun_settings.FIXTURE_TABLES[\"dataprovider\"],\ndata_provider_index=data_provider_index,\n)\nconsole_tables: list[Table] = list(\ngen_fixture_tables(run_settings.FIXTURE_TABLES)\n)\nif print_in_call:\nfor console_table in console_tables:\nconsole.print(console_table)\nreturn console_tables\nelse:\nreturn []\nshow_setup(clear: bool = True, title: str = SETUP_TITLE, **kwargs: str) -> None\nGenerate a rich.table.Table for printing configuration to console.
alto2txt2fixture/cli.py def show_setup(clear: bool = True, title: str = SETUP_TITLE, **kwargs) -> None:\n\"\"\"Generate a `rich.table.Table` for printing configuration to console.\"\"\"\nif clear and os.name == \"posix\":\nos.system(\"clear\")\nelif clear:\nos.system(\"cls\")\ntable = Table(title=title)\ntable.add_column(\"Setting\", justify=\"right\", style=\"cyan\", no_wrap=True)\ntable.add_column(\"Value\", style=\"magenta\")\nfor key, value in kwargs.items():\ntable.add_row(str(key), str(value))\nconsole.print(table)\nreturn\ncorrect_dict(o: dict) -> list\nReturns a list with corrected data from a provided dictionary.
Source code inalto2txt2fixture/create_adjacent_tables.py def correct_dict(o: dict) -> list:\n\"\"\"Returns a list with corrected data from a provided dictionary.\"\"\"\nreturn [(k, v[0], v[1]) for k, v in o.items() if not v[0].startswith(\"Q\")] + [\n(k, v[1], v[0]) for k, v in o.items() if v[0].startswith(\"Q\")\n]\ncsv2json_list(\ncsv_path: PathLike,\noutput_path: Path = OUTPUT,\nsaved: list[Path] | None = None,\nindent: int = JSON_INDENT,\n) -> list\nSave csv_path as a json file and return as a list.
Managing Pandas DataFrame nan values via suggestion: https://stackoverflow.com/a/62691803/678486
alto2txt2fixture/create_adjacent_tables.py def csv2json_list(\ncsv_path: PathLike,\noutput_path: Path = OUTPUT,\nsaved: list[Path] | None = None,\nindent: int = JSON_INDENT,\n) -> list:\n\"\"\"Save `csv_path` as a `json` file and return as a `list`.\n Note:\n Managing `Pandas` `DataFrame` `nan` values via suggestion:\n https://stackoverflow.com/a/62691803/678486\n \"\"\"\njson_data: list[dict[str, Any]] = []\ndf: pd.DataFame = (\npd.read_csv(csv_path, index_col=0).fillna(np.nan).replace([np.nan], [None])\n)\nif \"political_leanings\" in df.columns:\ndf[\"political_leanings\"] = df[\"political_leanings\"].apply(json.loads)\nif \"prices\" in df.columns:\ndf[\"prices\"] = df[\"prices\"].apply(json.loads)\nmodel: str = Path(csv_path).stem.lower()\nfor pk, row in df.iterrows():\nfields: dict[str, Any] = row.to_dict()\njson_data.append({\"pk\": pk, \"model\": model, \"fields\": fields})\njson_path: Path = Path(output_path) / f\"{Path(csv_path).stem}.json\"\njson_path.parent.mkdir(parents=True, exist_ok=True)\njson_path.write_text(json.dumps(json_data, indent=indent))\nif saved and isinstance(saved, list):\nsaved.append(json_path)\nreturn json_data\ndownload_data(\nfiles_dict: RemoteDataFilesType = {},\noverwrite: bool = OVERWRITE,\nexclude: list[str] = [],\n) -> None\nDownload files in files_dict, overwrite if specified.
Parameters:
Name Type Description Defaultfiles_dict RemoteDataFilesType dict of related files to download
{} overwrite bool bool to overwrite LOCAL_CACHE files or not
OVERWRITE exclude list[str] list of files to exclude from files_dict
[] Example >>> from os import chdir\n>>> tmp_path: Path = getfixture('tmp_path')\n>>> set_path: Path = chdir(tmp_path)\n>>> download_data(exclude=[\"mitchells\", \"Newspaper-1\", \"linking\"])\nExcluding mitchells...\nExcluding Newspaper-1...\nExcluding linking...\nDownloading cache...dict_admin_counties.json\n100% ... 37/37 bytes\nDownloading cache...dict_countries.json\n100% ... 33.2/33.2 kB\nDownloading cache...dict_historic_counties.json\n100% ... 41.4/41.4 kB\nDownloading cache...nlp_loc_wikidata_concat.csv\n100% ... 59.8/59.8 kB\nDownloading cache...wikidata_gazetteer_selected_columns.csv\n100% ... 47.8/47.8 MB\nalto2txt2fixture/create_adjacent_tables.py def download_data(\nfiles_dict: RemoteDataFilesType = {},\noverwrite: bool = OVERWRITE,\nexclude: list[str] = [],\n) -> None:\n\"\"\"Download files in ``files_dict``, overwrite if specified.\n Args:\n files_dict: `dict` of related files to download\n overwrite: `bool` to overwrite ``LOCAL_CACHE`` files or not\n exclude: `list` of files to exclude from ``files_dict``\n Example:\n ```pycon\n >>> from os import chdir\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> set_path: Path = chdir(tmp_path)\n >>> download_data(exclude=[\"mitchells\", \"Newspaper-1\", \"linking\"])\n Excluding mitchells...\n Excluding Newspaper-1...\n Excluding linking...\n Downloading cache...dict_admin_counties.json\n 100% ... 37/37 bytes\n Downloading cache...dict_countries.json\n 100% ... 33.2/33.2 kB\n Downloading cache...dict_historic_counties.json\n 100% ... 41.4/41.4 kB\n Downloading cache...nlp_loc_wikidata_concat.csv\n 100% ... 59.8/59.8 kB\n Downloading cache...wikidata_gazetteer_selected_columns.csv\n 100% ... 47.8/47.8 MB\n ```\n \"\"\"\nif not files_dict:\nfiles_dict = deepcopy(FILES)\nfor data_source in exclude:\nif data_source in files_dict:\nprint(f\"Excluding {data_source}...\")\nfiles_dict.pop(data_source, 0)\nelse:\nlogger.warning(\nf'\"{data_source}\" not an option to exclude from {files_dict}'\n)\n# Describe whether local file exists\nfor k in files_dict.keys():\nfiles_dict[k][\"exists\"] = files_dict[k][\"local\"].exists()\nfiles_to_download = [\n(v[\"remote\"], v[\"local\"], v[\"exists\"])\nfor v in files_dict.values()\nif \"exists\" in v and not v[\"exists\"] or overwrite\n]\nfor url, out, exists in files_to_download:\nrmtree(Path(out), ignore_errors=True) if exists else None\nprint(f\"Downloading {out}\")\nPath(out).parent.mkdir(parents=True, exist_ok=True)\nassert isinstance(url, str)\nwith urlopen(url) as response, open(out, \"wb\") as out_file:\ntotal: int = int(response.info()[\"Content-length\"])\nwith Progress(\n\"[progress.percentage]{task.percentage:>3.0f}%\",\nBarColumn(), # removed bar_width=None to avoid too long when resized\nDownloadColumn(),\n) as progress:\ndownload_task = progress.add_task(\"Download\", total=total)\nfor chunk in response:\nout_file.write(chunk)\nprogress.update(download_task, advance=len(chunk))\nget_list(x)\nGet a list from a string, which contains as separator. If no string is encountered, the function returns an empty list. Source code in alto2txt2fixture/create_adjacent_tables.py
def get_list(x):\n\"\"\"Get a list from a string, which contains <SEP> as separator. If no\n string is encountered, the function returns an empty list.\"\"\"\nreturn x.split(\"<SEP>\") if isinstance(x, str) else []\nget_outpaths_dict(\nnames: Sequence[str], module_name: str\n) -> TableOutputConfigType\nReturn a dict of csv and json paths for each module_name table.
The csv and json paths
Parameters:
Name Type Description Defaultnames Sequence[str] iterable of names of each module_name's component. Main target is csv and json table names
module_name str name of module each name is part of, that is added as a prefix
requiredReturns:
Type DescriptionTableOutputConfigType A TableOutputConfigType: a dict of table names and output csv and json filenames.
>>> pprint(get_outpaths_dict(MITCHELLS_TABELS, \"mitchells\"))\n{'Entry': {'csv': 'mitchells.Entry.csv', 'json': 'mitchells.Entry.json'},\n 'Issue': {'csv': 'mitchells.Issue.csv', 'json': 'mitchells.Issue.json'},\n 'PoliticalLeaning': {'csv': 'mitchells.PoliticalLeaning.csv',\n 'json': 'mitchells.PoliticalLeaning.json'},\n 'Price': {'csv': 'mitchells.Price.csv', 'json': 'mitchells.Price.json'}}\nalto2txt2fixture/create_adjacent_tables.py def get_outpaths_dict(names: Sequence[str], module_name: str) -> TableOutputConfigType:\n\"\"\"Return a `dict` of `csv` and `json` paths for each `module_name` table.\n The `csv` and `json` paths\n Args:\n names: iterable of names of each `module_name`'s component. Main target is `csv` and `json` table names\n module_name: name of module each name is part of, that is added as a prefix\n Returns:\n A ``TableOutputConfigType``: a `dict` of table ``names`` and output\n `csv` and `json` filenames.\n Example:\n ```pycon\n >>> pprint(get_outpaths_dict(MITCHELLS_TABELS, \"mitchells\"))\n {'Entry': {'csv': 'mitchells.Entry.csv', 'json': 'mitchells.Entry.json'},\n 'Issue': {'csv': 'mitchells.Issue.csv', 'json': 'mitchells.Issue.json'},\n 'PoliticalLeaning': {'csv': 'mitchells.PoliticalLeaning.csv',\n 'json': 'mitchells.PoliticalLeaning.json'},\n 'Price': {'csv': 'mitchells.Price.csv', 'json': 'mitchells.Price.json'}}\n ```\n \"\"\"\nreturn {\nname: OutputPathDict(\ncsv=f\"{module_name}.{name}.csv\",\njson=f\"{module_name}.{name}.json\",\n)\nfor name in names\n}\nrun(\nfiles_dict: dict = {},\nfiles_to_download_overwrite: bool = OVERWRITE,\nsaved: list[PathLike] = SAVED,\ntime_stamp: str = \"\",\noutput_path: Path = OUTPUT,\n) -> list[PathLike]\nDownload, process and link files_dict to json and csv.
This will require access to https://zooniversedata.blob.core.windows.net/downloads/.
alto2txt2fixture/create_adjacent_tables.py def run(\nfiles_dict: dict = {},\nfiles_to_download_overwrite: bool = OVERWRITE,\nsaved: list[PathLike] = SAVED,\ntime_stamp: str = \"\",\noutput_path: Path = OUTPUT,\n) -> list[PathLike]:\n\"\"\"Download, process and link ``files_dict`` to `json` and `csv`.\n Note:\n This will require access to `https://zooniversedata.blob.core.windows.net/downloads/`.\n \"\"\"\n# Ensure time_stamp from the point of calling `run`\nif not time_stamp:\ntime_stamp = get_now(as_str=False).strftime(TIME_FORMAT)\n# Ensure an independent deepcopy of FILES to avoid modifying subsequent runs\nif not files_dict:\nfiles_dict = deepcopy(FILES)\n# Download non-existing files\ndownload_data(files_dict=files_dict, overwrite=files_to_download_overwrite)\n# Create the output directory (defined in output_path)\noutput_path.mkdir(exist_ok=True, parents=True)\n# Read all the Wikidata Q values from Mitchells\nassert \"local\" in files_dict[\"mitchells\"]\nmitchells_df = pd.read_csv(files_dict[\"mitchells\"][\"local\"], index_col=0)\nmitchell_wikidata_mentions = sorted(\nlist(mitchells_df.PLACE_PUB_WIKI.unique()),\nkey=lambda x: int(x.replace(\"Q\", \"\")),\n)\n# Set up wikidata_gazetteer\ngaz_cols = [\"wikidata_id\", \"english_label\", \"latitude\", \"longitude\", \"geonamesIDs\"]\nassert \"local\" in files_dict[\"wikidata_gazetteer_selected_columns\"]\nwikidata_gazetteer = pd.read_csv(\nfiles_dict[\"wikidata_gazetteer_selected_columns\"][\"local\"], usecols=gaz_cols\n)\nwikidata_gazetteer.rename(\n{\n\"wikidata_id\": \"place_wikidata_id\",\n\"english_label\": \"place_label\",\n\"geonamesIDs\": \"geonames_ids\",\n},\naxis=1,\ninplace=True,\n)\n# Read in + fix all dictionaries\ndict_historic_counties = json.loads(\nPath(files_dict[\"dict_historic_counties\"][\"local\"]).read_text()\n)\ndict_admin_counties = json.loads(\nPath(files_dict[\"dict_admin_counties\"][\"local\"]).read_text()\n)\ndict_countries = json.loads(Path(files_dict[\"dict_countries\"][\"local\"]).read_text())\ndict_historic_counties = correct_dict(dict_historic_counties)\ndict_admin_counties = correct_dict(dict_admin_counties)\ndict_countries = correct_dict(dict_countries)\n# Create assisting frames\nhistorical_counties_df = pd.DataFrame(\ndict_historic_counties,\ncolumns=[\"place_wikidata_id\", \"hcounty_label\", \"hcounty_wikidata_id\"],\n)\nadmin_county_df = pd.DataFrame(\ndict_admin_counties,\ncolumns=[\n\"place_wikidata_id\",\n\"admin_county_label\",\n\"admin_county_wikidata_id\",\n],\n)\ncountries_df = pd.DataFrame(\ndict_countries,\ncolumns=[\"place_wikidata_id\", \"country_label\", \"country_wikidata_id\"],\n)\nwikidata_gazetteer = wikidata_gazetteer[\nwikidata_gazetteer.place_wikidata_id.isin(mitchell_wikidata_mentions)\n].sort_values(\"place_wikidata_id\")\nwikidata_gazetteer[\"place_pk\"] = np.arange(1, len(wikidata_gazetteer) + 1)\nwikidata_gazetteer = wikidata_gazetteer[\n[\"place_pk\"] + [x for x in wikidata_gazetteer.columns if not x == \"place_pk\"]\n]\n# Merge wikidata_gazetteer with all the assisting frames (and rename the\n# resulting columns)\nwikidata_gazetteer = pd.merge(\nwikidata_gazetteer, historical_counties_df, on=\"place_wikidata_id\", how=\"left\"\n)\nwikidata_gazetteer = pd.merge(\nwikidata_gazetteer, admin_county_df, on=\"place_wikidata_id\", how=\"left\"\n)\nwikidata_gazetteer = pd.merge(\nwikidata_gazetteer, countries_df, on=\"place_wikidata_id\", how=\"left\"\n)\nwikidata_gazetteer.rename(\n{\n\"admin_county_label\": \"admin_county__label\",\n\"admin_county_wikidata_id\": \"admin_county__wikidata_id\",\n\"hcounty_label\": \"historic_county__label\",\n\"hcounty_wikidata_id\": \"historic_county__wikidata_id\",\n\"country_label\": \"country__label\",\n\"country_wikidata_id\": \"country__wikidata_id\",\n},\naxis=1,\ninplace=True,\n)\n# Split back up into dataframes specific for the tables\nhistoric_county_table = (\nwikidata_gazetteer[[\"historic_county__label\", \"historic_county__wikidata_id\"]]\n.drop_duplicates()\n.copy()\n)\nhistoric_county_table = historic_county_table.replace({\"\": np.nan}).dropna()\nhistoric_county_table[\"historic_county__pk\"] = np.arange(\n1, len(historic_county_table) + 1\n)\nadmin_county_table = (\nwikidata_gazetteer[[\"admin_county__label\", \"admin_county__wikidata_id\"]]\n.drop_duplicates()\n.copy()\n)\nadmin_county_table = admin_county_table.replace({\"\": np.nan}).dropna()\nadmin_county_table[\"admin_county__pk\"] = np.arange(1, len(admin_county_table) + 1)\ncountry_table = (\nwikidata_gazetteer[[\"country__label\", \"country__wikidata_id\"]]\n.drop_duplicates()\n.copy()\n)\ncountry_table = country_table.replace({\"\": np.nan}).dropna()\ncountry_table[\"country__pk\"] = np.arange(1, len(country_table) + 1)\n# Set up place_table from wikidata_gazetteer\nplace_table = wikidata_gazetteer.copy()\nplace_table = (\npd.merge(\nplace_table,\nhistoric_county_table,\non=[\"historic_county__label\", \"historic_county__wikidata_id\"],\nhow=\"left\",\n)\n.drop([\"historic_county__label\", \"historic_county__wikidata_id\"], axis=1)\n.rename({\"historic_county__pk\": \"historic_county_id\"}, axis=1)\n)\nplace_table = (\npd.merge(\nplace_table,\nadmin_county_table,\non=[\"admin_county__label\", \"admin_county__wikidata_id\"],\nhow=\"left\",\n)\n.drop([\"admin_county__label\", \"admin_county__wikidata_id\"], axis=1)\n.rename({\"admin_county__pk\": \"admin_county_id\"}, axis=1)\n)\nplace_table = (\npd.merge(\nplace_table,\ncountry_table,\non=[\"country__label\", \"country__wikidata_id\"],\nhow=\"left\",\n)\n.drop([\"country__label\", \"country__wikidata_id\"], axis=1)\n.rename({\"country__pk\": \"country_id\"}, axis=1)\n)\nplace_table.fillna(\"\", inplace=True)\nplace_table.set_index(\"place_pk\", inplace=True)\nplace_table.rename(\n{\"place_label\": \"label\", \"place_wikidata_id\": \"wikidata_id\"},\naxis=1,\ninplace=True,\n)\nplace_table[\"historic_county_id\"] = (\nplace_table[\"historic_county_id\"]\n.replace(r\"^\\s*$\", 0, regex=True)\n.astype(int)\n.replace(0, \"\")\n)\nplace_table[\"admin_county_id\"] = (\nplace_table[\"admin_county_id\"]\n.replace(r\"^\\s*$\", 0, regex=True)\n.astype(int)\n.replace(0, \"\")\n)\nplace_table[\"country_id\"] = (\nplace_table[\"country_id\"]\n.replace(r\"^\\s*$\", 0, regex=True)\n.astype(int)\n.replace(0, \"\")\n)\nplace_table.index.rename(\"pk\", inplace=True)\nplace_table.rename(\n{\n\"historic_county_id\": \"historic_county\",\n\"admin_county_id\": \"admin_county\",\n\"country_id\": \"country\",\n},\naxis=1,\ninplace=True,\n)\nhistoric_county_table.set_index(\"historic_county__pk\", inplace=True)\nhistoric_county_table.rename(\n{x: x.split(\"__\")[1] for x in historic_county_table.columns},\naxis=1,\ninplace=True,\n)\nhistoric_county_table.index.rename(\"pk\", inplace=True)\nadmin_county_table.set_index(\"admin_county__pk\", inplace=True)\nadmin_county_table.rename(\n{x: x.split(\"__\")[1] for x in admin_county_table.columns}, axis=1, inplace=True\n)\nadmin_county_table.index.rename(\"pk\", inplace=True)\ncountry_table.set_index(\"country__pk\", inplace=True)\ncountry_table.rename(\n{x: x.split(\"__\")[1] for x in country_table.columns}, axis=1, inplace=True\n)\ncountry_table.index.rename(\"pk\", inplace=True)\n# Adding created_at, updated_at to all the gazetteer tables\nplace_table[\"created_at\"] = time_stamp\nplace_table[\"updated_at\"] = time_stamp\nadmin_county_table[\"created_at\"] = time_stamp\nadmin_county_table[\"updated_at\"] = time_stamp\nhistoric_county_table[\"created_at\"] = time_stamp\nhistoric_county_table[\"updated_at\"] = time_stamp\ncountry_table[\"created_at\"] = time_stamp\ncountry_table[\"updated_at\"] = time_stamp\n# Save CSV files for gazetteer tables\nplace_table.to_csv(output_path / GAZETTEER_OUT_FILENAMES[PLACE][\"csv\"])\nadmin_county_table.to_csv(\noutput_path / GAZETTEER_OUT_FILENAMES[ADMIN_COUNTY][\"csv\"]\n)\nhistoric_county_table.to_csv(\noutput_path / GAZETTEER_OUT_FILENAMES[HISTORIC_COUNTY][\"csv\"]\n)\ncountry_table.to_csv(output_path / GAZETTEER_OUT_FILENAMES[COUNTRY][\"csv\"])\nsaved.extend(\n[\noutput_path / GAZETTEER_OUT_FILENAMES[PLACE][\"csv\"],\noutput_path / GAZETTEER_OUT_FILENAMES[ADMIN_COUNTY][\"csv\"],\noutput_path / GAZETTEER_OUT_FILENAMES[HISTORIC_COUNTY][\"csv\"],\noutput_path / GAZETTEER_OUT_FILENAMES[COUNTRY][\"csv\"],\n]\n)\n# Fix up Mitchells (already loaded)\nmitchells_df[\"politics\"] = mitchells_df.POLITICS.apply(get_list)\nmitchells_df[\"persons\"] = mitchells_df.PERSONS.apply(get_list)\nmitchells_df[\"organisations\"] = mitchells_df.ORGANIZATIONS.apply(get_list)\nmitchells_df[\"price\"] = mitchells_df.PRICE.apply(get_list)\nmitchells_df.rename(\n{\n\"ID\": \"mpd_id\",\n\"TITLE\": \"title\",\n\"politics\": \"political_leaning_raw\",\n\"price\": \"price_raw\",\n\"YEAR\": \"year\",\n\"PLACE_PUB_WIKI\": \"place_of_publication_id\",\n\"ESTABLISHED_DATE\": \"date_established_raw\",\n\"PUBLISED_DATE\": \"day_of_publication_raw\",\n},\naxis=1,\ninplace=True,\n)\ndrop_cols = [\n\"CHAIN_ID\",\n\"POLITICS\",\n\"PERSONS\",\n\"ORGANIZATIONS\",\n\"PRICE\",\n\"PLACE_PUB\",\n\"PLACE_PUB_COORD\",\n\"PLACES\",\n\"PLACES_TRES\",\n\"TEXT\",\n]\nmitchells_df.drop(columns=drop_cols, inplace=True)\n# Create derivative tables (from Mitchells) = political_leanings, prices,\n# issues\npolitical_leanings = sorted(\nlist(set([y.strip() for x in mitchells_df.political_leaning_raw for y in x]))\n)\npolitical_leanings_table = pd.DataFrame()\npolitical_leanings_table[\"political_leaning__pk\"] = np.arange(\n1, len(political_leanings) + 1\n)\npolitical_leanings_table[\"political_leaning__label\"] = political_leanings\nexport = political_leanings_table.copy()\nexport[\"created_at\"] = time_stamp\nexport[\"updated_at\"] = time_stamp\nexport.set_index(\"political_leaning__pk\", inplace=True)\nexport.index.rename(\"pk\", inplace=True)\nexport.rename(\n{x: x.split(\"__\")[1] if len(x.split(\"__\")) > 1 else x for x in export.columns},\naxis=1,\ninplace=True,\n)\nexport.to_csv(output_path / MITCHELLS_OUT_FILENAMES[POLITICAL_LEANING][\"csv\"])\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[POLITICAL_LEANING][\"csv\"])\nprices = sorted(list(set([y.strip() for x in mitchells_df.price_raw for y in x])))\nprices_table = pd.DataFrame()\nprices_table[\"price__pk\"] = np.arange(1, len(prices) + 1)\nprices_table[\"price__label\"] = prices\nexport = prices_table.copy()\nexport[\"created_at\"] = time_stamp\nexport[\"updated_at\"] = time_stamp\nexport.set_index(\"price__pk\", inplace=True)\nexport.index.rename(\"pk\", inplace=True)\nexport.rename(\n{x: x.split(\"__\")[1] if len(x.split(\"__\")) > 1 else x for x in export.columns},\naxis=1,\ninplace=True,\n)\nexport.to_csv(output_path / MITCHELLS_OUT_FILENAMES[PRICE][\"csv\"])\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[PRICE][\"csv\"])\nissues = sorted(list(mitchells_df.year.unique()))\nissues_table = pd.DataFrame()\nissues_table[\"issue__pk\"] = np.arange(1, len(issues) + 1)\nissues_table[\"issue__year\"] = issues\nexport = issues_table.copy()\nexport[\"created_at\"] = time_stamp\nexport[\"updated_at\"] = time_stamp\nexport.set_index(\"issue__pk\", inplace=True)\nexport.index.rename(\"pk\", inplace=True)\nexport.rename(\n{x: x.split(\"__\")[1] if len(x.split(\"__\")) > 1 else x for x in export.columns},\naxis=1,\ninplace=True,\n)\nexport.to_csv(output_path / MITCHELLS_OUT_FILENAMES[ISSUE][\"csv\"])\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[ISSUE][\"csv\"])\n# Set up linking on Mitchells dataframe\nlinking_df = pd.read_csv(\nfiles_dict[\"linking\"][\"local\"],\nindex_col=0,\ndtype={\"NLP\": str},\nusecols=[\n\"NLP\",\n\"Title\",\n\"AcquiredYears\",\n\"Editions\",\n\"EditionTitles\",\n\"City\",\n\"Publisher\",\n\"UnavailableYears\",\n\"Collection\",\n\"UK\",\n\"Complete\",\n\"Notes\",\n\"County\",\n\"HistoricCounty\",\n\"First date held\",\n\"Publication title\",\n\"link_to_mpd\",\n],\n)\nlinking_df[\"NLP\"] = linking_df.index\nlinking_df.rename(\n{\"link_to_mpd\": \"mpd_id\", \"NLP\": \"newspaper\"}, axis=1, inplace=True\n)\n# Link Mitchells with all the other data\nmitchells_df = pd.merge(mitchells_df, linking_df, on=\"mpd_id\", how=\"inner\")\n# Create entry_table\nentry_table = mitchells_df.copy()\nentry_table[\"place_of_circulation_raw\"] = \"\"\nentry_table[\"publication_district_raw\"] = \"\"\nentry_table[\"publication_county_raw\"] = \"\"\n# TODO: What happened to the three columns above? (Check w Kaspar?)\n# Only keep relevant columns\nentry_table = entry_table[\n[\n\"title\",\n\"political_leaning_raw\",\n\"price_raw\",\n\"year\",\n\"date_established_raw\",\n\"day_of_publication_raw\",\n\"place_of_circulation_raw\",\n\"publication_district_raw\",\n\"publication_county_raw\",\n\"organisations\",\n\"persons\",\n\"place_of_publication_id\",\n\"newspaper\",\n]\n]\n# Fix refs to political_leanings_table\nrev = political_leanings_table.set_index(\"political_leaning__label\")\nentry_table[\"political_leanings\"] = entry_table.political_leaning_raw.apply(\nlambda x: [rev.at[y, \"political_leaning__pk\"] for y in x]\n)\n# Fix refs to prices_table\nrev = prices_table.set_index(\"price__label\")\nentry_table[\"prices\"] = entry_table.price_raw.apply(\nlambda x: [rev.at[y.strip(), \"price__pk\"] for y in x]\n)\n# Fix refs to issues_table\nrev = issues_table.set_index(\"issue__year\")\nentry_table[\"issue\"] = entry_table.year.apply(lambda x: rev.at[x, \"issue__pk\"])\n# Fix refs to place_table\nrev = place_table.copy()\nrev[\"place__pk\"] = rev.index\nrev.set_index(\"wikidata_id\", inplace=True)\nentry_table[\"place_of_publication\"] = entry_table.place_of_publication_id.apply(\ntest_place, rev=rev\n)\nentry_table.drop(columns=[\"place_of_publication_id\"], inplace=True)\n# Set up ref to newspapers\nrev = json.loads(files_dict[\"Newspaper-1\"][\"local\"].read_text())\nrev = [dict(pk=v[\"pk\"], **v[\"fields\"]) for v in rev]\nrev = pd.DataFrame(rev)\nrev.set_index(\"publication_code\", inplace=True)\nentry_table[\"newspaper\"] = entry_table.newspaper.str.zfill(7)\nentry_table[\"newspaper\"] = entry_table.newspaper.apply(test_paper, rev=rev)\n# Create PK for entries\nentry_table[\"pk\"] = np.arange(1, len(entry_table) + 1)\n# Sort columns in entries file\nentry_table = entry_table[\n[\"pk\"] + [col for col in entry_table.columns if not col == \"pk\"]\n]\n# Add created_at, modified_at to entry_table\nentry_table[\"created_at\"] = time_stamp\nentry_table[\"updated_at\"] = time_stamp\n# Export entry_table\nentry_table.set_index(\"pk\").to_csv(\noutput_path / MITCHELLS_OUT_FILENAMES[ENTRY][\"csv\"]\n)\nsaved.append(output_path / MITCHELLS_OUT_FILENAMES[ENTRY][\"csv\"])\n# ######\u00a0NOW WE CAN EASILY CREATE JSON files_dict\nfor csv_file_path in output_path.glob(\"*.csv\"):\ncsv2json_list(csv_file_path, output_path=output_path)\nprint(\"Finished - saved files:\")\nprint(\"- \" + \"\\n- \".join([str(x) for x in saved]))\nreturn saved\nget_jisc_title(\ntitle: str,\nissue_date: str,\njisc_papers: pd.DataFrame,\ninput_sub_path: str,\npublication_code: str,\nabbr: str | None = None,\n) -> str\nMatch a newspaper title with jisc_papers records.
Takes an input_sub_path, a publication_code, and an (optional) abbreviation for any newspaper to locate the title in the jisc_papers DataFrame. jisc_papers is usually loaded via the setup_jisc_papers function.
Parameters:
Name Type Description Defaulttitle str target newspaper title
requiredissue_date str target newspaper issue_date
requiredjisc_papers pd.DataFrame DataFrame of jisc_papers to match
input_sub_path str path of files to narrow down query input_sub_path
requiredpublication_code str unique codes to match newspaper records
requiredabbr str | None an optional abbreviation of the newspaper title
None Returns:
Type Descriptionstr Matched title str or abbr.
Returns:
Type Descriptionstr A string estimating the JISC equivalent newspaper title
Source code inalto2txt2fixture/jisc.py def get_jisc_title(\ntitle: str,\nissue_date: str,\njisc_papers: pd.DataFrame,\ninput_sub_path: str,\npublication_code: str,\nabbr: str | None = None,\n) -> str:\n\"\"\"\n Match a newspaper ``title`` with ``jisc_papers`` records.\n Takes an ``input_sub_path``, a ``publication_code``, and an (optional)\n abbreviation for any newspaper to locate the ``title`` in the\n ``jisc_papers`` `DataFrame`. ``jisc_papers`` is usually loaded via the\n ``setup_jisc_papers`` function.\n Args:\n title: target newspaper title\n issue_date: target newspaper issue_date\n jisc_papers: `DataFrame` of `jisc_papers` to match\n input_sub_path: path of files to narrow down query input_sub_path\n publication_code: unique codes to match newspaper records\n abbr: an optional abbreviation of the newspaper title\n Returns:\n Matched ``title`` `str` or ``abbr``.\n Returns:\n A string estimating the JISC equivalent newspaper title\n \"\"\"\n# First option, search the input_sub_path for a valid-looking publication_code\ng = PUBLICATION_CODE.findall(input_sub_path)\nif len(g) == 1:\npublication_code = g[0]\n# Let's see if we can find title:\ntitle = (\njisc_papers[\njisc_papers.publication_code == publication_code\n].title.to_list()[0]\nif jisc_papers[\njisc_papers.publication_code == publication_code\n].title.count()\n== 1\nelse title\n)\nreturn title\n# Second option, look through JISC papers for best match (on publication_code if we have it, but abbr more importantly if we have it)\nif abbr:\n_publication_code = publication_code\npublication_code = abbr\nif jisc_papers.abbr[jisc_papers.abbr == publication_code].count():\ndate = datetime.strptime(issue_date, \"%Y-%m-%d\")\nmask = (\n(jisc_papers.abbr == publication_code)\n& (date >= jisc_papers.start_date)\n& (date <= jisc_papers.end_date)\n)\nfiltered = jisc_papers.loc[mask]\nif filtered.publication_code.count() == 1:\npublication_code = filtered.publication_code.to_list()[0]\ntitle = filtered.title.to_list()[0]\nreturn title\n# Last option: let's find all the possible titles in the jisc_papers for the abbreviation, and if it's just one unique title, let's pick it!\nif abbr:\ntest = list({x for x in jisc_papers[jisc_papers.abbr == abbr].title})\nif len(test) == 1:\nreturn test[0]\nelse:\nmask1 = (jisc_papers.abbr == publication_code) & (\njisc_papers.publication_code == _publication_code\n)\ntest1 = jisc_papers.loc[mask1]\ntest1 = list({x for x in jisc_papers[jisc_papers.abbr == abbr].title})\nif len(test) == 1:\nreturn test1[0]\n# Fallback: if abbreviation is set, we'll return that:\nif abbr:\n# For these exceptions, see issue comment:\n# https://github.com/alan-turing-institute/Living-with-Machines/issues/2453#issuecomment-1050652587\nif abbr == \"IPJL\":\nreturn \"Ipswich Journal\"\nelif abbr == \"BHCH\":\nreturn \"Bath Chronicle\"\nelif abbr == \"LSIR\":\nreturn \"Leeds Intelligencer\"\nelif abbr == \"AGER\":\nreturn \"Lancaster Gazetter, And General Advertiser For Lancashire West\"\nreturn abbr\nraise RuntimeError(f\"Title {title} could not be found.\")\nsetup_jisc_papers(path: str = settings.JISC_PAPERS_CSV) -> pd.DataFrame\nCreate a DataFrame with information in JISC_PAPERS_CSV in settings.
Returns:
Type Descriptionpd.DataFrame DataFrame with all JISC titles.
alto2txt2fixture/jisc.py def setup_jisc_papers(path: str = settings.JISC_PAPERS_CSV) -> pd.DataFrame:\n\"\"\"\n Create a `DataFrame` with information in `JISC_PAPERS_CSV` in settings.\n Returns:\n `DataFrame` with all JISC titles.\n \"\"\"\nif not Path(path).exists():\nraise RuntimeError(\nf\"Could not find required JISC papers file. Put {Path(path).name} in {Path(path).parent} or correct the settings with a different path.\"\n)\nmonths = {\n\"Jan\": 1,\n\"Feb\": 2,\n\"Mar\": 3,\n\"Apr\": 4,\n\"May\": 5,\n\"Jun\": 6,\n\"June\": 6,\n\"Jul\": 7,\n\"July\": 7,\n\"Aug\": 8,\n\"Sep\": 9,\n\"Sept\": 9,\n\"Oct\": 10,\n\"Nov\": 11,\n\"Dec\": 12,\n\"Dec.\": 12,\n}\njisc_papers = pd.read_csv(\npath,\nusecols=[\n\"Newspaper Title\",\n\"NLP\",\n\"Abbr\",\n\"StartD\",\n\"StartM\",\n\"StartY\",\n\"EndD\",\n\"EndM\",\n\"EndY\",\n],\n)\njisc_papers[\"start_date\"] = jisc_papers.apply(\nlambda x: datetime(\nyear=int(x.StartY),\nmonth=months[x.StartM.strip(\".\").strip()],\nday=int(x.StartD),\n),\naxis=1,\n)\njisc_papers[\"end_date\"] = jisc_papers.apply(\nlambda x: datetime(\nyear=int(x.EndY), month=months[x.EndM.strip(\".\").strip()], day=int(x.EndD)\n),\naxis=1,\n)\njisc_papers.drop(\n[\"StartD\", \"StartM\", \"StartY\", \"EndD\", \"EndM\", \"EndY\"],\naxis=\"columns\",\ninplace=True,\n)\njisc_papers.rename(\n{\"Newspaper Title\": \"title\", \"NLP\": \"publication_code\", \"Abbr\": \"abbr\"},\naxis=1,\ninplace=True,\n)\njisc_papers[\"title\"] = jisc_papers[\"title\"].apply(\nlambda x: \"The \" + x[:-5] if x.strip()[-5:].lower() == \", the\" else x\n)\njisc_papers[\"publication_code\"] = jisc_papers[\"publication_code\"].apply(\nlambda x: str(x).zfill(7)\n)\nreturn jisc_papers\nerror(msg: str, crash: bool = True, silent: bool = True) -> None\nPrint msg in colorama Force.RED and exit()
If silent exit() after call, else raise RuntimeError if crash=True.
alto2txt2fixture/log.py def error(msg: str, crash: bool = True, silent: bool = True) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.RED` and `exit()`\n If `silent` `exit()` after call, else `raise` `RuntimeError` if ``crash=True``.\"\"\"\nif crash and silent:\nprint(f\"{Fore.RED}{msg}{Style.RESET_ALL}\")\nexit()\nelif crash:\nraise RuntimeError(msg) from None\nprint(f\"{Fore.RED}{msg}{Style.RESET_ALL}\")\nreturn\ninfo(msg: str) -> None\nPrint msg in colorama Force.CYAN colour.
alto2txt2fixture/log.py def info(msg: str) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.CYAN` colour.\"\"\"\nprint(f\"{Fore.CYAN}{msg}{Style.RESET_ALL}\")\nreturn\nsuccess(msg: str) -> None\nPrint msg in colorama Force.GREEN colour.
alto2txt2fixture/log.py def success(msg: str) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.GREEN` colour.\"\"\"\nprint(f\"{Fore.GREEN}{msg}{Style.RESET_ALL}\")\nreturn\nwarning(msg: str) -> None\nPrint msg in colorama Force.YELLOW colour.
alto2txt2fixture/log.py def warning(msg: str) -> None:\n\"\"\"Print ``msg`` in `colorama` `Force.YELLOW` colour.\"\"\"\nprint(f\"{Fore.YELLOW}Warning: {msg}{Style.RESET_ALL}\")\nreturn\nfixtures(\nfilelist: list = [],\nmodel: str = \"\",\ntranslate: dict = {},\nrename: dict = {},\nuniq_keys: list = [],\n) -> Generator[FixtureDict, None, None]\nGenerates fixtures for a specified model using a list of files.
This function takes a list of files and generates fixtures for a specified model. The fixtures can be used to populate a database or perform other data-related operations.
Parameters:
Name Type Description Defaultfilelist list A list of files to process and generate fixtures from.
[] model str The name of the model for which fixtures are generated. translate: A nested dictionary representing the translation mapping for fields. The structure of the translator follows the format:
{\n'part1': {\n'part2': {\n'translated_field': 'pk'\n}\n}\n}\n'' rename dict A nested dictionary representing the field renaming mapping. The structure of the dictionary follows the format:
{\n'part1': {\n'part2': 'new_field_name'\n}\n}\n{} uniq_keys list A list of fields that need to be considered for uniqueness in the fixtures. If specified, the fixtures will yield only unique items based on the combination of these fields.
[] Yields:
Type DescriptionFixtureDict FixtureDict from model, pk and dict of fields.
Returns:
Type DescriptionGenerator[FixtureDict, None, None] This function generates fixtures but does not return any value.
Source code inalto2txt2fixture/parser.py def fixtures(\nfilelist: list = [],\nmodel: str = \"\",\ntranslate: dict = {},\nrename: dict = {},\nuniq_keys: list = [],\n) -> Generator[FixtureDict, None, None]:\n\"\"\"\n Generates fixtures for a specified model using a list of files.\n This function takes a list of files and generates fixtures for a specified\n model. The fixtures can be used to populate a database or perform other\n data-related operations.\n Args:\n filelist: A list of files to process and generate fixtures from.\n model: The name of the model for which fixtures are generated.\n translate: A nested dictionary representing the translation mapping\n for fields. The structure of the translator follows the format:\n ```python\n {\n 'part1': {\n 'part2': {\n 'translated_field': 'pk'\n }\n }\n }\n ```\n The translated fields will be used as keys, and their\n corresponding primary keys (obtained from the provided files) will\n be used as values in the generated fixtures.\n rename: A nested dictionary representing the field renaming\n mapping. The structure of the dictionary follows the format:\n ```python\n {\n 'part1': {\n 'part2': 'new_field_name'\n }\n }\n ```\n The fields specified in the dictionary will be renamed to the\n provided new field names in the generated fixtures.\n uniq_keys: A list of fields that need to be considered for\n uniqueness in the fixtures. If specified, the fixtures will yield\n only unique items based on the combination of these fields.\n Yields:\n `FixtureDict` from ``model``, ``pk`` and `dict` of ``fields``.\n Returns:\n This function generates fixtures but does not return any value.\n \"\"\"\nfilelist = sorted(filelist, key=lambda x: str(x).split(\"/\")[:-1])\ncount = len(filelist)\n# Process JSONL\nif [x for x in filelist if \".jsonl\" in x.name]:\npk = 0\n# In the future, we might want to show progress here (tqdm or suchlike)\nfor file in filelist:\nfor line in file.read_text().splitlines():\npk += 1\nline = json.loads(line)\nyield FixtureDict(\npk=pk,\nmodel=model,\nfields=dict(**get_fields(line, translate=translate, rename=rename)),\n)\nreturn\nelse:\n# Process JSON\npks = [x for x in range(1, count + 1)]\nif len(uniq_keys):\nuniq_files = list(uniq(filelist, uniq_keys))\ncount = len(uniq_files)\nzipped = zip(uniq_files, pks)\nelse:\nzipped = zip(filelist, pks)\nfor x in tqdm(\nzipped, total=count, desc=f\"{model} ({count:,} objs)\", leave=False\n):\nyield FixtureDict(\npk=x[1],\nmodel=model,\nfields=dict(**get_fields(x[0], translate=translate, rename=rename)),\n)\nreturn\nget_fields(\nfile: Union[Path, str, dict],\ntranslate: dict = {},\nrename: dict = {},\nallow_null: bool = False,\n) -> dict\nRetrieves fields from a file and performs modifications and checks.
This function takes a file (in various formats: Path, str, or dict) and processes its fields. It retrieves the fields from the file and performs modifications, translations, and checks on the fields.
Parameters:
Name Type Description Defaultfile Union[Path, str, dict] The file from which the fields are retrieved.
requiredtranslate dict A nested dictionary representing the translation mapping for fields. The structure of the translator follows the format:
{\n'part1': {\n'part2': {\n'translated_field': 'pk'\n}\n}\n}\n{} rename dict A nested dictionary representing the field renaming mapping. The structure of the dictionary follows the format:
{\n'part1': {\n'part2': 'new_field_name'\n}\n}\n{} allow_null bool Determines whether to allow None values for relational fields. If set to True, relational fields with missing values will be assigned None. If set to False, an error will be raised.
False Returns:
Type Descriptiondict A dictionary representing the retrieved fields from the file, with modifications and checks applied.
Raises:
Type DescriptionRuntimeError If the file type is unsupported or if an error occurs during field retrieval or processing.
Source code inalto2txt2fixture/parser.py def get_fields(\nfile: Union[Path, str, dict],\ntranslate: dict = {},\nrename: dict = {},\nallow_null: bool = False,\n) -> dict:\n\"\"\"\n Retrieves fields from a file and performs modifications and checks.\n This function takes a file (in various formats: `Path`, `str`, or `dict`)\n and processes its fields. It retrieves the fields from the file and\n performs modifications, translations, and checks on the fields.\n Args:\n file: The file from which the fields are retrieved.\n translate: A nested dictionary representing the translation mapping\n for fields. The structure of the translator follows the format:\n ```python\n {\n 'part1': {\n 'part2': {\n 'translated_field': 'pk'\n }\n }\n }\n ```\n The translated fields will be used to replace the original fields\n in the retrieved fields.\n rename: A nested dictionary representing the field renaming\n mapping. The structure of the dictionary follows the format:\n ```python\n {\n 'part1': {\n 'part2': 'new_field_name'\n }\n }\n ```\n The fields specified in the dictionary will be renamed to the\n provided new field names in the retrieved fields.\n allow_null: Determines whether to allow ``None`` values for\n relational fields. If set to ``True``, relational fields with\n missing values will be assigned ``None``. If set to ``False``, an\n error will be raised.\n Returns:\n A dictionary representing the retrieved fields from the file,\n with modifications and checks applied.\n Raises:\n RuntimeError: If the file type is unsupported or if an error occurs\n during field retrieval or processing.\n \"\"\"\nif isinstance(file, Path):\ntry:\nfields = json.loads(file.read_text())\nexcept Exception as e:\nraise RuntimeError(f\"Cannot interpret JSON ({e}): {file}\")\nelif isinstance(file, str):\nif \"\\n\" in file:\nraise RuntimeError(\"File has multiple lines.\")\ntry:\nfields = json.loads(file)\nexcept json.decoder.JSONDecodeError as e:\nraise RuntimeError(f\"Cannot interpret JSON ({e}): {file}\")\nelif isinstance(file, dict):\nfields = file\nelse:\nraise RuntimeError(f\"Cannot process type {type(file)}.\")\n# Fix relational fields for any file\nfor key in [key for key in fields.keys() if \"__\" in key]:\nparts = key.split(\"__\")\ntry:\nbefore = fields[key]\nif before:\nbefore = before.replace(\"---\", \"/\")\nloc = translate.get(parts[0], {}).get(parts[1], {})\nfields[key] = loc.get(before)\nif fields[key] is None:\nraise RuntimeError(\nf\"Cannot translate fields.{key} from {before}: {loc}\"\n)\nexcept AttributeError:\nif allow_null:\nfields[key] = None\nelse:\nprint(\n\"Content had relational fields, but something went wrong in parsing the data:\"\n)\nprint(\"file\", file)\nprint(\"fields\", fields)\nprint(\"KEY:\", key)\nraise RuntimeError()\nnew_name = rename.get(parts[0], {}).get(parts[1], None)\nif new_name:\nfields[new_name] = fields[key]\ndel fields[key]\nfields[\"created_at\"] = NOW_str\nfields[\"updated_at\"] = NOW_str\ntry:\nfields[\"item_type\"] = str(fields[\"item_type\"]).upper()\nexcept KeyError:\npass\ntry:\nif fields[\"ocr_quality_mean\"] == \"\":\nfields[\"ocr_quality_mean\"] = 0\nexcept KeyError:\npass\ntry:\nif fields[\"ocr_quality_sd\"] == \"\":\nfields[\"ocr_quality_sd\"] = 0\nexcept KeyError:\npass\nreturn fields\nget_key_from(item: Path, x: str) -> str\nRetrieves a specific key from a file and returns its value.
This function reads a file and extracts the value of a specified key. If the key is not found or an error occurs while processing the file, a warning is printed, and an empty string is returned.
Parameters:
Name Type Description Defaultitem Path The file from which the key is extracted.
requiredx str The key to be retrieved from the file.
requiredReturns:
Type Descriptionstr The value of the specified key from the file.
Source code inalto2txt2fixture/parser.py def get_key_from(item: Path, x: str) -> str:\n\"\"\"\n Retrieves a specific key from a file and returns its value.\n This function reads a file and extracts the value of a specified\n key. If the key is not found or an error occurs while processing\n the file, a warning is printed, and an empty string is returned.\n Args:\n item: The file from which the key is extracted.\n x: The key to be retrieved from the file.\n Returns:\n The value of the specified key from the file.\n \"\"\"\nresult = json.loads(item.read_text()).get(x, None)\nif not result:\nprint(f\"[WARN] Could not find key {x} in {item}\")\nresult = \"\"\nreturn result\nget_translator(\nfields: list[TranslatorTuple] = [TranslatorTuple(\"\", \"\", [])]\n) -> dict\nConverts a list of fields into a nested dictionary representing a translator.
Parameters:
Name Type Description Defaultfields list[TranslatorTuple] A list of tuples representing fields to be translated.
[TranslatorTuple('', '', [])] Returns:
Type Descriptiondict A nested dictionary representing the translator. The structure of the dictionary follows the format:
{\n'part1': {\n'part2': {\n'translated_field': 'pk'\n}\n}\n}\n>>> fields = [\n... TranslatorTuple(\n... start='start__field1',\n... finish='field1',\n... lst=[{\n... 'fields': {'field1': 'translation1'},\n... 'pk': 1}],\n... )]\n>>> get_translator(fields)\n{'start': {'field1': {'translation1': 1}}}\nalto2txt2fixture/parser.py def get_translator(\nfields: list[TranslatorTuple] = [TranslatorTuple(\"\", \"\", [])]\n) -> dict:\n\"\"\"\n Converts a list of fields into a nested dictionary representing a\n translator.\n Args:\n fields: A list of tuples representing fields to be translated.\n Returns:\n A nested dictionary representing the translator. The structure of\n the dictionary follows the format:\n ```python\n {\n 'part1': {\n 'part2': {\n 'translated_field': 'pk'\n }\n }\n }\n ```\n Example:\n ```pycon\n >>> fields = [\n ... TranslatorTuple(\n ... start='start__field1',\n ... finish='field1',\n ... lst=[{\n ... 'fields': {'field1': 'translation1'},\n ... 'pk': 1}],\n ... )]\n >>> get_translator(fields)\n {'start': {'field1': {'translation1': 1}}}\n ```\n \"\"\"\n_ = dict()\nfor field in fields:\nstart, finish, lst = field\npart1, part2 = start.split(\"__\")\nif part1 not in _:\n_[part1] = {}\nif part2 not in _[part1]:\n_[part1][part2] = {}\nif isinstance(finish, str):\n_[part1][part2] = {o[\"fields\"][finish]: o[\"pk\"] for o in lst}\nelif isinstance(finish, list):\n_[part1][part2] = {\n\"-\".join([o[\"fields\"][x] for x in finish]): o[\"pk\"] for o in lst\n}\nreturn _\nparse(\ncollections: list, cache_home: str, output: str, max_elements_per_file: int\n) -> None\nParses files from collections and generates fixtures for various models.
This function processes files from the specified collections and generates fixtures for different models, such as newspapers.dataprovider, newspapers.ingest, newspapers.digitisation, newspapers.newspaper, newspapers.issue, and newspapers.item.
It performs various steps, such as file listing, fixture generation, translation mapping, renaming fields, and saving fixtures to files.
Parameters:
Name Type Description Defaultcollections list A list of collections from which files are processed and fixtures are generated.
requiredcache_home str The directory path where the collections are located.
requiredoutput str The directory path where the fixtures will be saved.
requiredmax_elements_per_file int The maximum number of elements per file when saving fixtures.
requiredReturns:
Type DescriptionNone This function generates fixtures but does not return any value.
Source code inalto2txt2fixture/parser.py def parse(\ncollections: list, cache_home: str, output: str, max_elements_per_file: int\n) -> None:\n\"\"\"\n Parses files from collections and generates fixtures for various models.\n This function processes files from the specified collections and generates\n fixtures for different models, such as `newspapers.dataprovider`,\n `newspapers.ingest`, `newspapers.digitisation`, `newspapers.newspaper`,\n `newspapers.issue`, and `newspapers.item`.\n It performs various steps, such as file listing, fixture generation,\n translation mapping, renaming fields, and saving fixtures to files.\n Args:\n collections: A list of collections from which files are\n processed and fixtures are generated.\n cache_home: The directory path where the collections are located.\n output: The directory path where the fixtures will be saved.\n max_elements_per_file: The maximum number of elements per file\n when saving fixtures.\n Returns:\n This function generates fixtures but does not return any value.\n \"\"\"\nglobal CACHE_HOME\nglobal OUTPUT\nglobal MAX_ELEMENTS_PER_FILE\nCACHE_HOME = cache_home\nOUTPUT = output\nMAX_ELEMENTS_PER_FILE = max_elements_per_file\n# Set up output directory\nreset_fixture_dir(OUTPUT)\n# Get file lists\nprint(\"\\nGetting file lists...\")\ndef issues_in_x(x):\nreturn \"issues\" in str(x.parent).split(\"/\")\ndef newspapers_in_x(x):\nreturn not any(\n[\ncondition\nfor y in str(x.parent).split(\"/\")\nfor condition in [\n\"issues\" in y,\n\"ingest\" in y,\n\"digitisation\" in y,\n\"data-provider\" in y,\n]\n]\n)\nall_json = [\nx for y in collections for x in (Path(CACHE_HOME) / y).glob(\"**/*.json\")\n]\nall_jsonl = [\nx for y in collections for x in (Path(CACHE_HOME) / y).glob(\"**/*.jsonl\")\n]\nprint(f\"--> {len(all_json):,} JSON files altogether\")\nprint(f\"--> {len(all_jsonl):,} JSONL files altogether\")\nprint(\"\\nSetting up fixtures...\")\n# Process data providers\ndef data_provider_in_x(x):\nreturn \"data-provider\" in str(x.parent).split(\"/\")\ndata_provider_json = list(\nfixtures(\nmodel=\"newspapers.dataprovider\",\nfilelist=[x for x in all_json if data_provider_in_x(x)],\nuniq_keys=[\"name\"],\n)\n)\nprint(f\"--> {len(data_provider_json):,} DataProvider fixtures\")\n# Process ingest\ndef ingest_in_x(x):\nreturn \"ingest\" in str(x.parent).split(\"/\")\ningest_json = list(\nfixtures(\nmodel=\"newspapers.ingest\",\nfilelist=[x for x in all_json if ingest_in_x(x)],\nuniq_keys=[\"lwm_tool_name\", \"lwm_tool_version\"],\n)\n)\nprint(f\"--> {len(ingest_json):,} Ingest fixtures\")\n# Process digitisation\ndef digitisation_in_x(x):\nreturn \"digitisation\" in str(x.parent).split(\"/\")\ndigitisation_json = list(\nfixtures(\nmodel=\"newspapers.digitisation\",\nfilelist=[x for x in all_json if digitisation_in_x(x)],\nuniq_keys=[\"software\"],\n)\n)\nprint(f\"--> {len(digitisation_json):,} Digitisation fixtures\")\n# Process newspapers\nnewspaper_json = list(\nfixtures(\nmodel=\"newspapers.newspaper\",\nfilelist=[file for file in all_json if newspapers_in_x(file)],\n)\n)\nprint(f\"--> {len(newspaper_json):,} Newspaper fixtures\")\n# Process issue\ntranslate = get_translator(\n[\nTranslatorTuple(\n\"publication__publication_code\", \"publication_code\", newspaper_json\n)\n]\n)\nrename = {\"publication\": {\"publication_code\": \"newspaper_id\"}}\nissue_json = list(\nfixtures(\nmodel=\"newspapers.issue\",\nfilelist=[file for file in all_json if issues_in_x(file)],\ntranslate=translate,\nrename=rename,\n)\n)\nprint(f\"--> {len(issue_json):,} Issue fixtures\")\n# Create translator/clear up memory before processing items\ntranslate = get_translator(\n[\n(\"issue__issue_identifier\", \"issue_code\", issue_json),\n(\"digitisation__software\", \"software\", digitisation_json),\n(\"data_provider__name\", \"name\", data_provider_json),\n(\n\"ingest__lwm_tool_identifier\",\n[\"lwm_tool_name\", \"lwm_tool_version\"],\ningest_json,\n),\n]\n)\nrename = {\n\"issue\": {\"issue_identifier\": \"issue_id\"},\n\"digitisation\": {\"software\": \"digitisation_id\"},\n\"data_provider\": {\"name\": \"data_provider_id\"},\n\"ingest\": {\"lwm_tool_identifier\": \"ingest_id\"},\n}\nsave_fixture(newspaper_json, \"Newspaper\")\nsave_fixture(issue_json, \"Issue\")\ndel newspaper_json\ndel issue_json\ngc.collect()\nprint(\"\\nSaving...\")\nsave_fixture(digitisation_json, \"Digitisation\")\nsave_fixture(ingest_json, \"Ingest\")\nsave_fixture(data_provider_json, \"DataProvider\")\n# Process items\nitem_json = fixtures(\nmodel=\"newspapers.item\",\nfilelist=all_jsonl,\ntranslate=translate,\nrename=rename,\n)\nsave_fixture(item_json, \"Item\")\nreturn\nreset_fixture_dir(output: str | Path) -> None\nResets the fixture directory by removing all JSON files inside it.
This function takes a directory path (output) as input and removes all JSON files within the directory.
Prior to removal, it prompts the user for confirmation to proceed. If the user confirms, the function clears the fixture directory by deleting the JSON files.
Parameters:
Name Type Description Defaultoutput str | Path The directory path of the fixture directory to be reset.
requiredRaises:
Type DescriptionRuntimeError If the output directory is not specified as a string.
alto2txt2fixture/parser.py def reset_fixture_dir(output: str | Path) -> None:\n\"\"\"\n Resets the fixture directory by removing all JSON files inside it.\n This function takes a directory path (``output``) as input and removes all\n JSON files within the directory.\n Prior to removal, it prompts the user for confirmation to proceed. If the\n user confirms, the function clears the fixture directory by deleting the\n JSON files.\n Args:\n output: The directory path of the fixture directory to be reset.\n Raises:\n RuntimeError: If the ``output`` directory is not specified as a string.\n \"\"\"\nif not isinstance(output, str):\nraise RuntimeError(\"`output` directory needs to be specified as a string.\")\noutput = Path(output)\ny = input(\nf\"This command will automatically empty the fixture directory ({output.absolute()}). \"\n\"Do you want to proceed? [y/N]\"\n)\nif not y.lower() == \"y\":\noutput.mkdir(parents=True, exist_ok=True)\nreturn\nprint(\"\\nClearing up the fixture directory\")\n# Ensure directory exists\noutput.mkdir(parents=True, exist_ok=True)\n# Drop all JSON files\n[x.unlink() for x in Path(output).glob(\"*.json\")]\nreturn\nuniq(filelist: list, keys: list = []) -> Generator[Any, None, None]\nGenerates unique items from a list of files based on specified keys.
This function takes a list of files and yields unique items based on a combination of keys. The keys are extracted from each file using the get_key_from function, and duplicate items are ignored.
Parameters:
Name Type Description Defaultfilelist list A list of files from which unique items are generated.
requiredkeys list A list of keys used for uniqueness. Each key specifies a field to be used for uniqueness checking in the generated items.
[] Yields:
Type DescriptionAny A unique item from filelist.
alto2txt2fixture/parser.py def uniq(filelist: list, keys: list = []) -> Generator[Any, None, None]:\n\"\"\"\n Generates unique items from a list of files based on specified keys.\n This function takes a list of files and yields unique items based on a\n combination of keys. The keys are extracted from each file using the\n ``get_key_from`` function, and duplicate items are ignored.\n Args:\n filelist: A list of files from which unique items are\n generated.\n keys: A list of keys used for uniqueness. Each key specifies\n a field to be used for uniqueness checking in the generated\n items.\n Yields:\n A unique item from `filelist`.\n \"\"\"\nseen = set()\nfor item in filelist:\nkey = \"-\".join([get_key_from(item, x) for x in keys])\nif key not in seen:\nseen.add(key)\nyield item\nelse:\n# Drop it if duplicate\npass\nUseful regular expressions, intially just PUBLICATION_CODE.
Bases: TypedDict
A dict of lwmdb.fulltext.models.Fulltext fixture structure.
Attributes:
Name Type Descriptionpath PathLike Plaintext file path.
compressed_path PathLike | None If path is within a compressed file, compressed_path is that source. Else None.
primary_key int An int >= 1 for a SQL table primary key (pk).
dataclass","text":"Manage exporting plaintext alto2txt results for lwmdb.fulltext.models.Fulltext.
Attributes:
Name Type Descriptionpath PathLike PathLike source for fixtures as either a folder or file.
data_provider_code str | None A short string to uniquely identify DataProviders, primarily to match sources in self.data_provider_code_dict.
files tuple[PathLike, ...] | None A iterable PathLike collection of to either read as plaintext or decomepress to extract plaintext.
compressed_glob_regex str A Regular Expression to filter plaintext files from uncompressed self.files, more specifically self.compressed_files.
data_provider DataProviderFixtureDict | None If available a DataProviderFixtureDict for DataProvider metadata. By default all options are stored in self.data_provider_code_dict.
model_str str The name of the lwmdb django model the fixture is for. This is of the form app_name.model_name. Following the default config for lwmdb:
FULLTEXT_DJANGO_MODEL: Final[str] = \"fulltext.fulltext\"\nfulltext app has a fulltext model class specified in lwmdb.fulltext.models.fulltext. A sql table is generated from on that fulltext class and the json fixture structure generated from this class is where records will be stored. extract_subdir PathLike Folder to extract self.compressed_files to.
plaintext_extension str What file extension to use to filter plaintext files.
data_provider_code_dict dict[str, DataProviderFixtureDict] A dict of metadata for preconfigured DataProvider records in lwmdb.
max_plaintext_per_fixture_file int A maximum number of fixtures per fixture file, designed to configure chunking fixtures.
saved_fixture_prefix str A str to prefix all saved json fixture filenames.
export_directory PathLike Directory to save all exported fixtures to.
initial_pk int Default begins at 1, can be set to another number if needed to add to add more to pre-existing set of records up to a given pk
json_0_file_name_padding int Number of 0s to prefix file name numbering.
_disk_usage int Available harddrive space. Designed to help mitigate decompressing too many files for available disk space.
self._uncompressed_source_file_dict int A dictionary of extracted plaintext to compressed source file. This is a field in json fixture records.
>>> path = getfixture('bl_lwm')\n>>> logger.setLevel(INFO)\n>>> plaintext_bl_lwm = PlainTextFixture(\n... data_provider_code='bl_lwm',\n... path=path,\n... compressed_glob_regex=\"*_plaintext.zip\",\n... )\n>>> plaintext_bl_lwm\n<PlainTextFixture(path='...bl_lwm')>\n>>> plaintext_bl_lwm.info()\n<BLANKLINE>\n...PlainTextFixture for 2 'bl_lwm' files...\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500...\u2510\n\u2502 Path \u2502 '...bl_lwm' ...\u2502\n\u2502 Compressed Files \u2502 '...bl_lwm...0003079-test_plain...\u2502\n\u2502 \u2502 '...bl_lwm...0003548-test_plain...\u2502\n\u2502 Extract Path \u2502 '...bl_lwm...extracted' ...\u2502\n\u2502 Uncompressed Files \u2502 None ...\u2502\n\u2502 Data Provider \u2502 'Living with Machines' ...\u2502\n\u2502 Initial Primary Key \u2502 1 ...\u2502\n\u2502 Max Rows Per JSON \u2502 100 ...\u2502\n\u2502 JSON File Name 0s \u2502 6 ...\u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500...\u2518\n>>> plaintext_bl_lwm.free_hd_space_in_GB > 1\nTrue\n>>> plaintext_bl_lwm.extract_compressed()\n<BLANKLINE>\n...Extract path:...'...lwm...extracted...\n...Extracting:...'...lwm...00030...\n...Extracting:...'...lwm...00035...\n...%...[...]...\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in:...'...bl_lwm...tracted'\nproperty","text":"compressed_files: tuple[PathLike, ...]\nReturn a tuple of all self.files with known archive file names.
property","text":"data_provider_name: str | None\nReturn self.data_provider code attributre or None.
self.data_provider_code>>> bl_hmd = PlainTextFixture(\n... path=\".\",\n... data_provider_code=\"bl_hmd\")\n>>> bl_hmd.data_provider_name\n'Heritage Made Digital'\n>>> bl_lwm = PlainTextFixture(\n... path='.',\n... data_provider=NEWSPAPER_DATA_PROVIDER_CODE_DICT['bl_lwm'],\n... )\n>>> bl_lwm.data_provider_name\n'Living with Machines'\n>>> plaintext_fixture = PlainTextFixture(\n... path=\".\")\n<BLANKLINE>\n...`.data_provider` and `.data_provider_code`...\n...are...'None'...in...<PlainTextFixture(path='.')>...\n>>> plaintext_fixture.data_provider_name\n`
"},{"location":"reference/alto2txt2fixture/plaintext.html#alto2txt2fixture.plaintext.PlainTextFixture.exported_json_paths","title":"exported_json_pathsproperty","text":"exported_json_paths: Generator[Path, None, None]\nIf self._exported_json_paths return Generator of those paths.
Yields:
Type DescriptionEach path from self._exported_json_paths
>>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> tuple(plaintext_bl_lwm.exported_json_paths)\n(...Path('...plaintext_fixture-000001.json'),)\nproperty","text":"extract_path: Path\nPath any compressed files would be extracted to.
"},{"location":"reference/alto2txt2fixture/plaintext.html#alto2txt2fixture.plaintext.PlainTextFixture.free_hd_space_in_GB","title":"free_hd_space_in_GBproperty","text":"free_hd_space_in_GB: float\nReturn remaing hard drive space estimate in gigabytes.
"},{"location":"reference/alto2txt2fixture/plaintext.html#alto2txt2fixture.plaintext.PlainTextFixture.info_table","title":"info_tableproperty","text":"info_table: Table\nGenerate a rich.table.Table of config information.
>>> hmd_plaintext_fixture = PlainTextFixture(\n... path=\".\",\n... data_provider_code=\"bl_hmd\")\n>>> table = hmd_plaintext_fixture.info_table\n>>> table.title\n\"PlainTextFixture for 0 'bl_hmd' files\"\nproperty","text":"plaintext_provided_uncompressed: tuple[PathLike, ...]\nReturn a tuple of all self.files with self.plaintext_extension.
property","text":"trunc_compressed_file_names_str: str\nReturn truncated self.compressed_files file names or empty str.
property","text":"trunc_extract_path_str: str\nReturn truncated self.extract_path or empty str.
property","text":"trunc_uncompressed_file_names_str: str\nReturn truncated self.plaintext_provided_uncompressed file names or empty str.
property","text":"zipinfo: Generator[list[ZipInfo], None, None]\nIf self.compressed_files is in zip, return info, else None.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n>>> zipfile_info_list: list[ZipInfo] = list(plaintext_bl_lwm.zipinfo)\nGetting zipfile info from <PlainTextFixture(path='...bl_lwm')>\n>>> zipfile_info_list[0][-1].filename\n'0003079...1898...0204...0003079_18980204_sect0001.txt'\n>>> zipfile_info_list[-1][-1].filename\n'0003548...1904...0707...0003548_19040707_art0059.txt'\n>>> zipfile_info_list[0][-1].file_size\n70192\n>>> zipfile_info_list[0][-1].compress_size\n39911\n__len__() -> int\nReturn the number of files to process.
Source code inalto2txt2fixture/plaintext.py def __len__(self) -> int:\n\"\"\"Return the number of files to process.\"\"\"\nreturn len(self.files) if self.files else 0\n__post_init__() -> None\nManage populating additional attributes if necessary.
Source code inalto2txt2fixture/plaintext.py def __post_init__(self) -> None:\n\"\"\"Manage populating additional attributes if necessary.\"\"\"\nself._check_and_set_files_attr(force=True)\nself._check_and_set_data_provider(force=True)\nself._disk_usage: DiskUsageTuple = disk_usage(self.path)\nself._uncompressed_source_file_dict: OrderedDict[\nPathLike, PathLike\n] = OrderedDict()\nself._pk_plaintext_dict: OrderedDict[PathLike, int] = OrderedDict()\n__repr__() -> str\nReturn class name with path attribute.
alto2txt2fixture/plaintext.py def __repr__(self) -> str:\n\"\"\"Return `class` name with `path` attribute.\"\"\"\nreturn f\"<{type(self).__name__}(path='{self.path}')>\"\n__str__() -> str\nReturn class name with count and DataProvider if available.
alto2txt2fixture/plaintext.py def __str__(self) -> str:\n\"\"\"Return class name with count and `DataProvider` if available.\"\"\"\nreturn (\nf\"{type(self).__name__} \"\nf\"for {len(self)} \"\nf\"{self._data_provider_code_quoted_with_trailing_space}files\"\n)\ncompress_json_exports(\noutput_path: PathLike | None = None, format: ArchiveFormatEnum | None = None\n) -> tuple[Path, ...]\nCompress self._exported_json_paths to format.
Parameters:
Name Type Description Defaultoutput_path PathLike | None Path to save compressed json files to. Uses self.json_export_compression_subdir if None is passed.
None format ArchiveFormatEnum | None What compression format to use from ArchiveFormatEnum. Uses self.json_export_compression_format if None is passed.
None Note Neither output_path nor format overwrite the related attributes of self.
>>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> compressed_paths: Path = plaintext_bl_lwm.compress_json_exports(\n... format='tar')\n<BLANKLINE>\n...Compressing...'...01.json' to...'tar'...in:...\n>>> compressed_paths\n(...Path('.../plaintext_fixture-000001.json.tar'),)\nalto2txt2fixture/plaintext.py def compress_json_exports(\nself,\noutput_path: PathLike | None = None,\nformat: ArchiveFormatEnum | None = None,\n) -> tuple[Path, ...]:\n\"\"\"Compress `self._exported_json_paths` to `format`.\n Args:\n output_path:\n `Path` to save compressed `json` files to. Uses\n `self.json_export_compression_subdir` if `None` is passed.\n format:\n What compression format to use from `ArchiveFormatEnum`. Uses\n `self.json_export_compression_format` if `None` is passed.\n Note:\n Neither `output_path` nor `format` overwrite the related attributes\n of `self`.\n Returns: The the `output_path` passed to save compressed `json`.\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n <BLANKLINE>\n ...\n >>> compressed_paths: Path = plaintext_bl_lwm.compress_json_exports(\n ... format='tar')\n <BLANKLINE>\n ...Compressing...'...01.json' to...'tar'...in:...\n >>> compressed_paths\n (...Path('.../plaintext_fixture-000001.json.tar'),)\n ```\n \"\"\"\noutput_path = (\nPath(self.json_export_compression_subdir)\nif not output_path\nelse Path(output_path)\n)\nformat = self.json_export_compression_format if not format else format\ncompressed_paths: list[Path] = []\nfor json_path in self.exported_json_paths:\ncompressed_paths.append(\ncompress_fixture(json_path, output_path=output_path, format=format)\n)\nreturn tuple(compressed_paths)\ndelete_decompressed(ignore_errors: bool = True) -> None\nRemove all files in self.extract_path.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...'...bl_lwm...extracted'...\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in:...\n>>> plaintext_bl_lwm.delete_decompressed()\n<BLANKLINE>\n...Extract path empty:...'...bl_lwm...extracted'...\nalto2txt2fixture/plaintext.py def delete_decompressed(self, ignore_errors: bool = True) -> None:\n\"\"\"Remove all files in `self.extract_path`.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...'...bl_lwm...extracted'...\n >>> plaintext_bl_lwm.delete_decompressed()\n Deleting all files in:...\n >>> plaintext_bl_lwm.delete_decompressed()\n <BLANKLINE>\n ...Extract path empty:...'...bl_lwm...extracted'...\n ```\n \"\"\"\nif self.extract_path.exists():\nconsole.print(f\"Deleting all files in: '{self.extract_path}'\")\nrmtree(self.extract_path, ignore_errors=ignore_errors)\nelse:\nconsole.log(f\"Extract path empty: '{self.extract_path}'\")\nexport_to_json_fixtures(\noutput_path: PathLike | None = None,\nprefix: str | None = None,\njson_0_file_name_padding: int | None = None,\n) -> None\nIterate over self.plaintext_paths exporting to json django fixtures.
For errors running on windows see: #55
Parameters:
Name Type Description Defaultoutput_path PathLike | None Folder to save all json fixtures in.
None prefix str | None Any str prefix for saved fixture files.
None json_0_file_name_padding int | None Number of 0s to prefix file name numbering.
None Example >>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> bl_lwm: Path = getfixture(\"bl_lwm\")\n>>> first_lwm_plaintext_json_dict: PlaintextFixtureDict = (\n... getfixture(\"first_lwm_plaintext_json_dict\")\n... )\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...bl_lwm...extracted...\n>>> plaintext_bl_lwm.export_to_json_fixtures(output_path=bl_lwm / \"output\")\n<BLANKLINE>\nCompressed configs...%...[...]\n>>> len(plaintext_bl_lwm._exported_json_paths)\n1\n>>> plaintext_bl_lwm._exported_json_paths\n(...Path(...plaintext_fixture-000001.json...),)\n>>> import json\n>>> exported_json = json.loads(\n... plaintext_bl_lwm._exported_json_paths[0].read_text()\n... )\n>>> exported_json[0]['pk'] == first_lwm_plaintext_json_dict['pk']\nTrue\n>>> exported_json[0]['model'] == first_lwm_plaintext_json_dict['model']\nTrue\n>>> (exported_json[0]['fields']['text'] ==\n... first_lwm_plaintext_json_dict['fields']['text'])\nTrue\n>>> (exported_json[0]['fields']['path'] ==\n... str(first_lwm_plaintext_json_dict['fields']['path']))\nTrue\n>>> (exported_json[0]['fields']['compressed_path'] ==\n... str(first_lwm_plaintext_json_dict['fields']['compressed_path']))\nTrue\n>>> exported_json[0]['fields']['created_at']\n'20...'\n>>> (exported_json[0]['fields']['updated_at'] ==\n... exported_json[0]['fields']['updated_at'])\nTrue\nalto2txt2fixture/plaintext.py def export_to_json_fixtures(\nself,\noutput_path: PathLike | None = None,\nprefix: str | None = None,\njson_0_file_name_padding: int | None = None,\n) -> None:\n\"\"\"Iterate over `self.plaintext_paths` exporting to `json` `django` fixtures.\n Note:\n For errors running on windows see:\n [#55](https://github.com/Living-with-machines/alto2txt2fixture/issues/55)\n Args:\n output_path:\n Folder to save all `json` fixtures in.\n prefix:\n Any `str` prefix for saved fixture files.\n json_0_file_name_padding:\n Number of `0`s to prefix file name numbering.\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> bl_lwm: Path = getfixture(\"bl_lwm\")\n >>> first_lwm_plaintext_json_dict: PlaintextFixtureDict = (\n ... getfixture(\"first_lwm_plaintext_json_dict\")\n ... )\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...bl_lwm...extracted...\n >>> plaintext_bl_lwm.export_to_json_fixtures(output_path=bl_lwm / \"output\")\n <BLANKLINE>\n Compressed configs...%...[...]\n >>> len(plaintext_bl_lwm._exported_json_paths)\n 1\n >>> plaintext_bl_lwm._exported_json_paths\n (...Path(...plaintext_fixture-000001.json...),)\n >>> import json\n >>> exported_json = json.loads(\n ... plaintext_bl_lwm._exported_json_paths[0].read_text()\n ... )\n >>> exported_json[0]['pk'] == first_lwm_plaintext_json_dict['pk']\n True\n >>> exported_json[0]['model'] == first_lwm_plaintext_json_dict['model']\n True\n >>> (exported_json[0]['fields']['text'] ==\n ... first_lwm_plaintext_json_dict['fields']['text'])\n True\n >>> (exported_json[0]['fields']['path'] ==\n ... str(first_lwm_plaintext_json_dict['fields']['path']))\n True\n >>> (exported_json[0]['fields']['compressed_path'] ==\n ... str(first_lwm_plaintext_json_dict['fields']['compressed_path']))\n True\n >>> exported_json[0]['fields']['created_at']\n '20...'\n >>> (exported_json[0]['fields']['updated_at'] ==\n ... exported_json[0]['fields']['updated_at'])\n True\n ```\n \"\"\"\noutput_path = self.export_directory if not output_path else output_path\nprefix = self.saved_fixture_prefix if not prefix else prefix\njson_0_file_name_padding = (\nself.json_0_file_name_padding\nif not json_0_file_name_padding\nelse json_0_file_name_padding\n)\nsave_fixture(\nself.plaintext_paths_to_dicts(),\nprefix=prefix,\noutput_path=output_path,\nadd_created=True,\nmax_elements_per_file=self.max_plaintext_per_fixture_file,\nfile_name_0_padding=json_0_file_name_padding,\n)\nself.set_exported_json_paths(\nexport_directory=output_path, saved_fixture_prefix=prefix\n)\nextract_compressed() -> None\nExtract self.compressed_files to self.extracted_subdir_name.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n>>> plaintext_bl_lwm.extract_compressed()\n<BLANKLINE>\n...Extract path:...'...bl_lwm...extracted'...\n>>> filter_sect1_txt: list[str] = [txt_file for txt_file in\n... plaintext_bl_lwm._uncompressed_source_file_dict.keys()\n... if txt_file.name.endswith('204_sect0001.txt')]\n>>> len(filter_sect1_txt)\n1\n>>> plaintext_bl_lwm._uncompressed_source_file_dict[\n... filter_sect1_txt[0]\n... ]\n<BLANKLINE>\n...Path('...bl_lwm...0003079-test_plaintext.zip')\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in:...'...bl_lwm...tracted'\nalto2txt2fixture/plaintext.py def extract_compressed(self) -> None:\n\"\"\"Extract `self.compressed_files` to `self.extracted_subdir_name`.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n >>> plaintext_bl_lwm.extract_compressed()\n <BLANKLINE>\n ...Extract path:...'...bl_lwm...extracted'...\n >>> filter_sect1_txt: list[str] = [txt_file for txt_file in\n ... plaintext_bl_lwm._uncompressed_source_file_dict.keys()\n ... if txt_file.name.endswith('204_sect0001.txt')]\n >>> len(filter_sect1_txt)\n 1\n >>> plaintext_bl_lwm._uncompressed_source_file_dict[\n ... filter_sect1_txt[0]\n ... ]\n <BLANKLINE>\n ...Path('...bl_lwm...0003079-test_plaintext.zip')\n >>> plaintext_bl_lwm.delete_decompressed()\n Deleting all files in:...'...bl_lwm...tracted'\n ```\n \"\"\"\nself.extract_path.mkdir(parents=True, exist_ok=True)\nconsole.log(f\"Extract path: '{self.extract_path}'\")\nfor compressed_file in tqdm(\nself.compressed_files,\ntotal=len(self.compressed_files),\n):\nlogger.info(f\"Extracting: '{compressed_file}' ...\")\nunpack_archive(compressed_file, self.extract_path)\nfor path in sorted(self.extract_path.glob(self.plaintext_glob_regex)):\nif path not in self._uncompressed_source_file_dict:\nself._uncompressed_source_file_dict[path] = compressed_file\ninfo() -> None\nPrint self.info_table to the console.
alto2txt2fixture/plaintext.py def info(self) -> None:\n\"\"\"Print `self.info_table` to the `console`.\"\"\"\nconsole.print(self.info_table)\nplaintext_paths(reset_cache = False) -> Generator[FulltextPathDict, None, None]\nReturn a generator of all plaintext files for potential fixtures.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...bl_lwm...extracted...\n>>> plaintext_paths = plaintext_bl_lwm.plaintext_paths()\n>>> first_path_fixture_dict = next(iter(plaintext_paths))\n>>> first_path_fixture_dict['path'].name\n'0003079_18980107_art0001.txt'\n>>> first_path_fixture_dict['compressed_path'].name\n'0003079-test_plaintext.zip'\n>>> len(plaintext_bl_lwm._pk_plaintext_dict)\n1\n>>> plaintext_bl_lwm._pk_plaintext_dict[\n... first_path_fixture_dict['path']\n... ] # This demonstrates the `pk` begins from 1 following `SQL` standards\n1\nalto2txt2fixture/plaintext.py def plaintext_paths(\nself, reset_cache=False\n) -> Generator[FulltextPathDict, None, None]:\n\"\"\"Return a generator of all `plaintext` files for potential fixtures.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...bl_lwm...extracted...\n >>> plaintext_paths = plaintext_bl_lwm.plaintext_paths()\n >>> first_path_fixture_dict = next(iter(plaintext_paths))\n >>> first_path_fixture_dict['path'].name\n '0003079_18980107_art0001.txt'\n >>> first_path_fixture_dict['compressed_path'].name\n '0003079-test_plaintext.zip'\n >>> len(plaintext_bl_lwm._pk_plaintext_dict)\n 1\n >>> plaintext_bl_lwm._pk_plaintext_dict[\n ... first_path_fixture_dict['path']\n ... ] # This demonstrates the `pk` begins from 1 following `SQL` standards\n 1\n ```\n \"\"\"\nif self.compressed_files and not self.extract_path.exists():\nconsole.print(\n\"Compressed files not yet extracted. Try `extract_compression()`.\"\n)\nelse:\ni: int = 0\npk: int\nif self._uncompressed_source_file_dict:\nfor i, uncompressed_tuple in enumerate(\ntqdm(\nself._uncompressed_source_file_dict.items(),\ndesc=\"Compressed configs :\",\ntotal=len(self._uncompressed_source_file_dict),\n)\n):\npk = i + self.initial_pk # Most `SQL` `pk` begins at 1\nself._pk_plaintext_dict[uncompressed_tuple[0]] = pk\nyield FulltextPathDict(\npath=uncompressed_tuple[0],\ncompressed_path=uncompressed_tuple[1],\nprimary_key=pk,\n)\nif self.plaintext_provided_uncompressed:\nfor j, path in enumerate(\ntqdm(\nself.plaintext_provided_uncompressed,\ndesc=\"Uncompressed configs:\",\ntotal=len(self.plaintext_provided_uncompressed),\n)\n):\npk = j + i + self.initial_pk\nself._pk_plaintext_dict[path] = pk\nyield FulltextPathDict(\npath=path, compressed_path=None, primary_key=pk\n)\nplaintext_paths_to_dicts() -> Generator[PlaintextFixtureDict, None, None]\nGenerate fixture dicts from self.plaintext_paths.
For errors running on windows see: #55
Example>>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n<BLANKLINE>\n...Extract path:...bl_lwm...extracted...\n>>> paths_dict = list(plaintext_bl_lwm.plaintext_paths_to_dicts())\nCompressed configs :...%...[ ... it/s ]\n>>> plaintext_bl_lwm.delete_decompressed()\nDeleting all files in: '...tracted'\nalto2txt2fixture/plaintext.py def plaintext_paths_to_dicts(self) -> Generator[PlaintextFixtureDict, None, None]:\n\"\"\"Generate fixture dicts from `self.plaintext_paths`.\n Note:\n For errors running on windows see:\n [#55](https://github.com/Living-with-machines/alto2txt2fixture/issues/55)\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_extracted')\n <BLANKLINE>\n ...Extract path:...bl_lwm...extracted...\n >>> paths_dict = list(plaintext_bl_lwm.plaintext_paths_to_dicts())\n Compressed configs :...%...[ ... it/s ]\n >>> plaintext_bl_lwm.delete_decompressed()\n Deleting all files in: '...tracted'\n ```\n \"\"\"\ntext: str\nerror_str: str | None = None\nfor plaintext_path_dict in self.plaintext_paths():\nerror_str = None\ntry:\ntext = Path(plaintext_path_dict[\"path\"]).read_text()\nexcept UnicodeDecodeError as err:\nlogger.warning(err)\ntext = \"\"\nerror_str = str(err)\nfields: PlaintextFixtureFieldsDict = PlaintextFixtureFieldsDict(\ntext=text,\npath=str(plaintext_path_dict[\"path\"]),\ncompressed_path=str(plaintext_path_dict[\"compressed_path\"]),\nerrors=error_str,\n)\nyield PlaintextFixtureDict(\nmodel=self.model_str,\nfields=fields,\npk=plaintext_path_dict[\"primary_key\"],\n)\nset_exported_json_paths(\nexport_directory: PathLike | None,\nsaved_fixture_prefix: str | None,\noverwrite: bool = False,\n) -> None\nSet self._exported_json_paths for use with self.exported_json_paths.
If provided export_directory and saved_fixture_prefix will overwite those attributes on self.
Parameters:
Name Type Description Defaultexport_directory PathLike | None Path to check for saved json files.
saved_fixture_prefix str | None str to prefix each exported json file with.
overwrite bool Force replace self._exported_json_paths if already set.
False Example >>> if is_platform_win:\n... pytest.skip('decompression fails on Windows: issue #55')\n>>> tmp_path = getfixture('tmp_path')\n>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> tuple(plaintext_bl_lwm.exported_json_paths)\n(...Path('...plaintext_fixture-000001.json'),)\n>>> plaintext_bl_lwm.set_exported_json_paths(tmp_path, 'check-prefix')\nTraceback (most recent call last):\n...\nValueError: Cannot overwrite 'self._exported_json_paths' without\n'overwrite' = True. Current 'self._exported_json_paths':\n(...Path('...plaintext_fixture-000001.json'),)\n>>> plaintext_bl_lwm.set_exported_json_paths(tmp_path,\n... 'check-prefix', overwrite=True)\n<BLANKLINE>\n...Force change '._exported_json_paths' in...<PlainTextFixture...\n>>> plaintext_bl_lwm.export_directory == tmp_path\nTrue\n>>> plaintext_bl_lwm.saved_fixture_prefix\n'check-prefix'\nalto2txt2fixture/plaintext.py def set_exported_json_paths(\nself,\nexport_directory: PathLike | None,\nsaved_fixture_prefix: str | None,\noverwrite: bool = False,\n) -> None:\n\"\"\"Set `self._exported_json_paths` for use with `self.exported_json_paths`.\n Note:\n If provided `export_directory` and `saved_fixture_prefix` will\n overwite those attributes on `self.`\n Params:\n export_directory:\n `Path` to check for saved `json` files.\n saved_fixture_prefix:\n `str` to prefix each exported `json` file with.\n overwrite:\n Force replace `self._exported_json_paths` if already set.\n Example:\n ```pycon\n >>> if is_platform_win:\n ... pytest.skip('decompression fails on Windows: issue #55')\n >>> tmp_path = getfixture('tmp_path')\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n <BLANKLINE>\n ...\n >>> tuple(plaintext_bl_lwm.exported_json_paths)\n (...Path('...plaintext_fixture-000001.json'),)\n >>> plaintext_bl_lwm.set_exported_json_paths(tmp_path, 'check-prefix')\n Traceback (most recent call last):\n ...\n ValueError: Cannot overwrite 'self._exported_json_paths' without\n 'overwrite' = True. Current 'self._exported_json_paths':\n (...Path('...plaintext_fixture-000001.json'),)\n >>> plaintext_bl_lwm.set_exported_json_paths(tmp_path,\n ... 'check-prefix', overwrite=True)\n <BLANKLINE>\n ...Force change '._exported_json_paths' in...<PlainTextFixture...\n >>> plaintext_bl_lwm.export_directory == tmp_path\n True\n >>> plaintext_bl_lwm.saved_fixture_prefix\n 'check-prefix'\n ```\n \"\"\"\nif hasattr(self, \"_exported_json_paths\"):\nif overwrite:\nlogger.info(f\"Force change '._exported_json_paths' in {repr(self)}\")\nelse:\nraise ValueError(\nf\"Cannot overwrite 'self._exported_json_paths' without \"\nf\"'overwrite' = True. Current 'self._exported_json_paths':\\n \"\nf\"{pformat(self._exported_json_paths)}\"\n)\nself.export_directory = (\nexport_directory if export_directory else self.export_directory\n)\nself.saved_fixture_prefix = (\nsaved_fixture_prefix if saved_fixture_prefix else self.saved_fixture_prefix\n)\nself._exported_json_paths = path_globs_to_tuple(\nself.export_directory, f\"**/{self.saved_fixture_prefix}*.json\"\n)\nArchive(\npath: str | Path,\ncollection: str = \"\",\nreport_id: str | None = None,\njisc_papers: pd.DataFrame | None = None,\njson_indent: int = JSON_INDENT,\n)\nManage extracting information from a ZIP archive.
The Archive class represents a zip archive of XML files. The class is used to extract information from a ZIP archive, and it contains several methods to process the data contained in the archive.
open(Archive) context manager
Archive can be opened with a context manager, which creates a meta object, with timings for the object. When closed, it will save the meta JSON to the correct paths.
Attributes:
Name Type Descriptionpath Path The path to the zip archive.
collection str The collection of the XML files in the archive. Default is \"\".
report Path The file path of the report file for the archive.
report_id str The report ID for the archive. If not provided, a random UUID is generated.
report_parent Path The parent directory of the report file for the archive.
jisc_papers pd.DataFrame A DataFrame of JISC papers.
size str | float The size of the archive, in human-readable format.
size_raw str | float The raw size of the archive, in bytes.
roots Generator[ET.Element, None, None] The root elements of the XML documents contained in the archive.
meta dotdict Metadata about the archive, such as its path, size, and number of contents.
json_indent int Indentation formatting of json output
Raises:
Type DescriptionRuntimeError If the path does not exist.
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\npath: str | Path,\ncollection: str = \"\",\nreport_id: str | None = None,\njisc_papers: pd.DataFrame | None = None,\njson_indent: int = JSON_INDENT,\n):\n\"\"\"Constructor method.\"\"\"\nself.path: Path = Path(path)\nif not self.path.exists():\nraise RuntimeError(\"Path does not exist.\")\nself.size: str | float = get_size_from_path(self.path)\nself.size_raw: str | float = get_size_from_path(self.path, raw=True)\nself.zip_file: zipfile.ZipFile = zipfile.ZipFile(self.path)\nself.collection: str = collection\nself.roots: Generator[ET.Element, None, None] = self.get_roots()\nself.meta: dotdict = dotdict(\npath=str(self.path),\nbytes=self.size_raw,\nsize=self.size,\ncontents=len(self.filelist),\n)\nif not report_id:\nself.report_id: str = str(uuid.uuid4())\nelse:\nself.report_id = report_id\nself.jisc_papers: pd.DataFrame = jisc_papers\nself.report_parent: Path = Path(f\"{REPORT_DIR}/{self.report_id}\")\nself.report: Path = (\nself.report_parent / f\"{self.path.stem.replace('_metadata', '')}.json\"\n)\nself.json_indent: int = json_indent\nproperty","text":"documents\nProperty that calls the get_documents method
property","text":"filelist\nReturns the list of files in the zip file
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Archive.__len__","title":"__len__","text":"__len__()\nThe number of files inside the zip archive.
Source code inalto2txt2fixture/router.py def __len__(self):\n\"\"\"The number of files inside the zip archive.\"\"\"\nreturn len(self.filelist)\nget_documents() -> Generator[Document, None, None]\nA generator that yields instances of the Document class for each XML file in the ZIP archive.
It uses the tqdm library to display a progress bar in the terminal while it is running.
If the contents of the ZIP file are not empty, the method creates an instance of the Document class by passing the root element of the XML file, the collection name, meta information about the archive, and the JISC papers data frame (if provided) to the constructor of the Document class. The instance of the Document class is then returned by the generator.
Yields:
Type DescriptionDocument Document class instance for each unzipped XML file.
alto2txt2fixture/router.py def get_documents(self) -> Generator[Document, None, None]:\n\"\"\"\n A generator that yields instances of the Document class for each XML\n file in the ZIP archive.\n It uses the `tqdm` library to display a progress bar in the terminal\n while it is running.\n If the contents of the ZIP file are not empty, the method creates an\n instance of the ``Document`` class by passing the root element of the XML\n file, the collection name, meta information about the archive, and the\n JISC papers data frame (if provided) to the constructor of the\n ``Document`` class. The instance of the ``Document`` class is then\n returned by the generator.\n Yields:\n ``Document`` class instance for each unzipped `XML` file.\n \"\"\"\nfor xml_file in tqdm(\nself.filelist,\ndesc=f\"{Path(self.zip_file.filename).stem} ({self.meta.size})\",\nleave=False,\ncolour=\"green\",\n):\nwith self.zip_file.open(xml_file) as f:\nxml = f.read()\nif xml:\nyield Document(\nroot=ET.fromstring(xml),\ncollection=self.collection,\nmeta=self.meta,\njisc_papers=self.jisc_papers,\n)\nget_roots() -> Generator[ET.Element, None, None]\nYields the root elements of the XML documents contained in the archive.
Source code inalto2txt2fixture/router.py def get_roots(self) -> Generator[ET.Element, None, None]:\n\"\"\"\n Yields the root elements of the XML documents contained in the archive.\n \"\"\"\nfor xml_file in tqdm(self.filelist, leave=False, colour=\"blue\"):\nwith self.zip_file.open(xml_file) as f:\nxml = f.read()\nif xml:\nyield ET.fromstring(xml)\nCache()\nThe Cache class provides a blueprint for creating and managing cache data. The class has several methods that help in getting the cache path, converting the data to a dictionary, and writing the cache data to a file.
It is inherited by many other classes in this document.
Initializes the Cache class object.
Source code inalto2txt2fixture/router.py def __init__(self):\n\"\"\"\n Initializes the Cache class object.\n \"\"\"\npass\n__str__() -> str\nReturns the string representation of the cache data as a dictionary.
Source code inalto2txt2fixture/router.py def __str__(self) -> str:\n\"\"\"\n Returns the string representation of the cache data as a dictionary.\n \"\"\"\nreturn str(self.as_dict())\nas_dict() -> dict\nConverts the cache data to a dictionary and returns it.
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n Converts the cache data to a dictionary and returns it.\n \"\"\"\nreturn {}\nget_cache_path() -> Path\nReturns the cache path, which is used to store the cache data. The path is normally constructed using some of the object's properties (collection, kind, and id) but can be changed when inherited.
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the cache path, which is used to store the cache data.\n The path is normally constructed using some of the object's\n properties (collection, kind, and id) but can be changed when\n inherited.\n \"\"\"\nreturn Path(f\"{CACHE_HOME}/{self.collection}/{self.kind}/{self.id}.json\")\nwrite_to_cache(json_indent: int = JSON_INDENT) -> Optional[bool]\nWrites the cache data to a file at the specified cache path. The cache data is first converted to a dictionary using the as_dict method. If the cache path already exists, the function returns True.
Source code inalto2txt2fixture/router.py def write_to_cache(self, json_indent: int = JSON_INDENT) -> Optional[bool]:\n\"\"\"\n Writes the cache data to a file at the specified cache path. The cache\n data is first converted to a dictionary using the as_dict method. If\n the cache path already exists, the function returns True.\n \"\"\"\npath = self.get_cache_path()\ntry:\nif path.exists():\nreturn True\nexcept AttributeError:\nerror(\nf\"Error occurred when getting cache path for \"\nf\"{self.kind}: {path}. It was not of expected \"\nf\"type Path but of type {type(path)}:\",\n)\npath.parent.mkdir(parents=True, exist_ok=True)\nwith open(path, \"w+\") as f:\nf.write(json.dumps(self.as_dict(), indent=json_indent))\nreturn\nCollection(name: str = 'hmd', jisc_papers: Optional[pd.DataFrame] = None)\nA Collection represents a group of newspaper archives from any passed alto2txt metadata output.
A Collection is initialised with a name and an optional pandas DataFrame of JISC papers. The archives property returns an iterable of the Archive objects within the collection.
Attributes:
Name Type Descriptionname str Name of the collection (default \"hmd\")
jisc_papers pandas.DataFrame DataFrame of JISC papers, optional
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, name: str = \"hmd\", jisc_papers: Optional[pd.DataFrame] = None):\n\"\"\"Constructor method.\"\"\"\nself.name: str = name\nself.jisc_papers: pd.DataFrame | None = jisc_papers\nself.dir: Path = Path(f\"{MNT}/{self.name}-alto2txt/metadata\")\nself.zip_files: list[Path] = sorted(\nlist(self.dir.glob(\"*.zip\")), key=lambda x: x.stat().st_size\n)\nself.zip_file_count: int = sum([1 for _ in self.dir.glob(\"*.zip\")])\nself.report_id: str = str(uuid.uuid4())\nself.empty: bool = self.zip_file_count == 0\nDataProvider(collection: str)\n Bases: Cache
The DataProvider class extends the Cache class and represents a newspaper data provider. The class has several properties and methods that allow creation of a data provider object and the manipulation of its data.
Attributes:
Name Type Descriptioncollection str A string representing publication collection
kind str Indication of object type, defaults to data-provider
providers_meta_data list[FixtureDict] structured dict of metadata for known collection sources
collection_type str related data sources and potential linkage source
index_field str field name for querying existing records
Example>>> from pprint import pprint\n>>> hmd = DataProvider(\"hmd\")\n>>> hmd.pk\n2\n>>> pprint(hmd.as_dict())\n{'code': 'bl_hmd',\n 'collection': 'newspapers',\n 'legacy_code': 'hmd',\n 'name': 'Heritage Made Digital',\n 'source_note': 'British Library-funded digitised newspapers provided by the '\n 'British Newspaper Archive'}\nConstructor method.
Source code inalto2txt2fixture/router.py def __init__(self, collection: str):\n\"\"\"Constructor method.\"\"\"\nself.collection: str = collection\nproperty","text":"meta_data: FixtureDict | dict\nReturn self.providers_meta_data[self.collection] or {}.
property","text":"meta_data_fields: FixtureDict | dict\nReturn self.providers_meta_data[self.collection] or {}.
property","text":"pk: int | None\nReturn pk if provided via providers_meta_data, else None.
property","text":"providers_index_dict: dict[str, FixtureDict]\nReturn all self.index_field values from providers_meta_data.
as_dict() -> dict\nReturn a dict of the data provider object.
Returns:
Type Descriptiondict Dictionary representation of the DataProvider object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n Return a `dict` of the data provider object.\n Returns:\n Dictionary representation of the DataProvider object\n \"\"\"\nif self.meta_data:\nreturn {\n\"name\": self.meta_data_fields[\"name\"],\n\"code\": self.meta_data_fields[\"code\"],\n\"legacy_code\": self.collection,\n\"source_note\": self.meta_data_fields[\"source_note\"],\n\"collection\": self.collection_type,\n}\nelse:\nreturn {\n\"name\": self.collection,\n\"code\": slugify(self.collection),\n\"source_note\": \"\",\n\"legacy_code\": None,\n\"collection\": self.collection_type,\n}\nDigitisation(root: ET.Element, collection: str = '')\n Bases: Cache
The Digitisation class extends the Cache class and represents a newspaper digitisation. The class has several properties and methods that allow creation of an digitisation object and the manipulation of its data.
Attributes:
Name Type Descriptionroot ET.Element An xml element that represents the root of the publication
collection str A string that represents the collection of the publication
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, root: ET.Element, collection: str = \"\"):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nself.root: ET.Element = root\nself.collection: str = collection\nclass-attribute instance-attribute","text":"kind = 'digitisation'\nA string that represents the type of the object, set to \"digitisation\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Digitisation.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the digitisation object.
Returns:
Type Descriptiondict Dictionary representation of the Digitising object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the digitisation\n object.\n Returns:\n Dictionary representation of the Digitising object\n \"\"\"\ndic = {\nx.tag: x.text or \"\"\nfor x in self.root.findall(\"./process/*\")\nif x.tag\nin [\n\"xml_flavour\",\n\"software\",\n\"mets_namespace\",\n\"alto_namespace\",\n]\n}\nif not dic.get(\"software\"):\nreturn {}\nreturn dic\nDocument(*args, **kwargs)\nThe Document class is a representation of a document that contains information about a publication, newspaper, item, digitisation, and ingest. This class holds all the relevant information about a document in a structured manner and provides properties that can be used to access different aspects of the document.
Attributes:
Name Type Descriptioncollection str | None A string that represents the collection of the publication
root ET.Element | None An XML element that represents the root of the publication
zip_file str | None A path to a valid zip file
jisc_papers pd.DataFrame | None A pandas DataFrame object that holds information about the JISC papers
meta dotdict | None TODO
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, *args, **kwargs):\n\"\"\"Constructor method.\"\"\"\nself.collection: str | None = kwargs.get(\"collection\")\nif not self.collection or not isinstance(self.collection, str):\nraise RuntimeError(\"A valid collection must be passed\")\nself.root: ET.Element | None = kwargs.get(\"root\")\nif not self.root or not isinstance(self.root, ET.Element):\nraise RuntimeError(\"A valid XML root must be passed\")\nself.zip_file: str | None = kwargs.get(\"zip_file\")\nif self.zip_file and not isinstance(self.zip_file, str):\nraise RuntimeError(\"A valid zip file must be passed\")\nself.jisc_papers: pd.DataFrame | None = kwargs.get(\"jisc_papers\")\nif not isinstance(self.jisc_papers, pd.DataFrame):\nraise RuntimeError(\n\"A valid DataFrame containing JISC papers must be passed\"\n)\nself.meta: dotdict | None = kwargs.get(\"meta\")\nself._publication_elem = None\nself._input_sub_path = None\nself._ingest = None\nself._digitisation = None\nself._item = None\nself._issue = None\nself._newspaper = None\nself._data_provider = None\nproperty","text":"publication: ET.Element\nThis property returns an ElementTree object representing the publication information in the XML document.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Ingest","title":"Ingest","text":"Ingest(root: ET.Element, collection: str = '')\n Bases: Cache
The Ingest class extends the Cache class and represents a newspaper ingest. The class has several properties and methods that allow the creation of an ingest object and the manipulation of its data.
Attributes:
Name Type Descriptionroot ET.Element An xml element that represents the root of the publication
collection str A string that represents the collection of the publication
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(self, root: ET.Element, collection: str = \"\"):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nself.root: ET.Element = root\nself.collection: str = collection\nclass-attribute instance-attribute","text":"kind = 'ingest'\nA string that represents the type of the object, set to \"ingest\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Ingest.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the ingest object.
Returns:
Type Descriptiondict Dictionary representation of the Ingest object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the ingest\n object.\n Returns:\n Dictionary representation of the Ingest object\n \"\"\"\nreturn {\nf\"lwm_tool_{x.tag}\": x.text or \"\"\nfor x in self.root.findall(\"./process/lwm_tool/*\")\n}\nIssue(\npublication: ET.Element,\nnewspaper: Optional[Newspaper] = None,\ncollection: str = \"\",\ninput_sub_path: str = \"\",\nmeta: dotdict = dotdict(),\n)\n Bases: Cache
The Issue class extends the Cache class and represents a newspaper issue. The class has several properties and methods that allow the creation of an issue object and the manipulation of its data.
Attributes:
Name Type Descriptionroot An xml element that represents the root of the publication
newspaper Newspaper | None The parent newspaper
collection str A string that represents the collection of the publication
input_sub_path str TODO
meta dotdict TODO
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\npublication: ET.Element,\nnewspaper: Optional[Newspaper] = None,\ncollection: str = \"\",\ninput_sub_path: str = \"\",\nmeta: dotdict = dotdict(),\n):\n\"\"\"Constructor method.\"\"\"\nself.publication: ET.Element = publication\nself.newspaper: Newspaper | None = newspaper\nself.collection: str = collection\nself.input_sub_path: str = input_sub_path\nself.meta: dotdict = meta\nself._issue = None\nself._issue_date = None\npath: str = str(self.get_cache_path())\nif not self.meta.issue_paths:\nself.meta.issue_paths = [path]\nelif path not in self.meta.issue_paths:\nself.meta.issue_paths.append(path)\nproperty","text":"issue_code: str\nSets up and saves the issue code for easy access as property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Issue.issue_date","title":"issue_dateproperty","text":"issue_date: str\nSets up and saves the issue date for easy access as property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Issue.kind","title":"kindclass-attribute instance-attribute","text":"kind = 'issue'\nA string that represents the type of the object, set to \"issue\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Issue.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the issue object.
Returns:
Type Descriptiondict Dictionary representation of the Issue object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the issue\n object.\n Returns:\n Dictionary representation of the Issue object\n \"\"\"\nif not self._issue:\nself._issue = dict(\nissue_code=self.issue_code,\nissue_date=self.issue_date,\npublication__publication_code=self.newspaper.publication_code,\ninput_sub_path=self.input_sub_path,\n)\nreturn self._issue\nget_cache_path() -> Path\nReturns the path to the cache file for the issue object.
Returns:
Type DescriptionPath Path to the cache file for the issue object
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the path to the cache file for the issue object.\n Returns:\n Path to the cache file for the issue object\n \"\"\"\njson_file = f\"/{self.newspaper.publication_code}/issues/{self.issue_code}.json\"\nreturn Path(\nf\"{CACHE_HOME}/{self.collection}/\"\n+ \"/\".join(self.newspaper.number_paths)\n+ json_file\n)\nItem(\nroot: ET.Element,\nissue_code: str = \"\",\ndigitisation: dict = {},\ningest: dict = {},\ncollection: str = \"\",\nnewspaper: Optional[Newspaper] = None,\nmeta: dotdict = dotdict(),\n)\n Bases: Cache
The Newspaper class extends the Cache class and represents a newspaper item, i.e. an article. The class has several properties and methods that allow the creation of an article object and the manipulation of its data.
Attributes:
Name Type Descriptionroot ET.Element An xml element that represents the root of the publication
issue_code str A string that represents the issue code
digitisation dict TODO
ingest dict TODO
collection str A string that represents the collection of the publication
newspaper Newspaper | None The parent newspaper
meta dotdict TODO
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\nroot: ET.Element,\nissue_code: str = \"\",\ndigitisation: dict = {},\ningest: dict = {},\ncollection: str = \"\",\nnewspaper: Optional[Newspaper] = None,\nmeta: dotdict = dotdict(),\n):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nif not isinstance(newspaper, Newspaper):\nraise RuntimeError(\"Expected newspaper to be of type router.Newspaper\")\nself.root: ET.Element = root\nself.issue_code: str = issue_code\nself.digitisation: dict = digitisation\nself.ingest: dict = ingest\nself.collection: str = collection\nself.newspaper: Newspaper | None = newspaper\nself.meta: dotdict = meta\nself._item_elem = None\nself._item_code = None\nself._item = None\npath: str = str(self.get_cache_path())\nif not self.meta.item_paths:\nself.meta.item_paths = [path]\nelif path not in self.meta.item_paths:\nself.meta.item_paths.append(path)\nproperty","text":"item_code: str\nSets up and saves the item code for easy access as property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Item.item_elem","title":"item_elemproperty","text":"item_elem\nSets up and saves the issue XML item for easy access as a property.
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Item.kind","title":"kindclass-attribute instance-attribute","text":"kind = 'item'\nA string that represents the type of the object, set to \"item\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Item.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the item object (i.e. article).
Returns:
Type Descriptiondict Dictionary representation of the Item object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the item object\n (i.e. article).\n Returns:\n Dictionary representation of the Item object\n \"\"\"\nif not self._item:\nself._item = {\nf\"{x.tag}\": x.text or \"\"\nfor x in self.item_elem.findall(\"*\")\nif x.tag\nin [\n\"title\",\n\"word_count\",\n\"ocr_quality_mean\",\n\"ocr_quality_sd\",\n\"plain_text_file\",\n\"item_type\",\n]\n}\nself._item[\"title\"] = self._item.get(\"title\", \"\")[:2097151]\nself._item = {\n\"item_code\": self.item_code,\n\"word_count\": self._item.get(\"word_count\", 0),\n\"title\": self._item.get(\"title\"),\n\"item_type\": self._item.get(\"item_type\"),\n\"input_filename\": self._item.get(\"plain_text_file\", \"\"),\n\"ocr_quality_mean\": self._item.get(\"ocr_quality_mean\", 0),\n\"ocr_quality_sd\": self._item.get(\"ocr_quality_sd\", 0),\n\"digitisation__software\": self.digitisation.id,\n\"ingest__lwm_tool_identifier\": self.ingest.id,\n\"issue__issue_identifier\": self.issue_code,\n\"data_provider__name\": self.collection,\n}\nreturn self._item\nget_cache_path() -> Path\nReturns the path to the cache file for the item (article) object.
Returns:
Type DescriptionPath Path to the cache file for the article object
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the path to the cache file for the item (article) object.\n Returns:\n Path to the cache file for the article object\n \"\"\"\nreturn Path(\nf\"{CACHE_HOME}/{self.collection}/\"\n+ \"/\".join(self.newspaper.number_paths)\n+ f\"/{self.newspaper.publication_code}/items.jsonl\"\n)\nwrite_to_cache(json_indent = JSON_INDENT) -> None\nSpecial cache-write function that appends rather than writes at the end of the process.
Returns:
Type DescriptionNone None.
Source code inalto2txt2fixture/router.py def write_to_cache(self, json_indent=JSON_INDENT) -> None:\n\"\"\"\n Special cache-write function that appends rather than writes at the\n end of the process.\n Returns:\n None.\n \"\"\"\npath = self.get_cache_path()\npath.parent.mkdir(parents=True, exist_ok=True)\nwith open(path, \"a+\") as f:\nf.write(json.dumps(self.as_dict(), indent=json_indent) + \"\\n\")\nreturn\nNewspaper(\nroot: ET.Element,\ncollection: str = \"\",\nmeta: dotdict = dotdict(),\njisc_papers: Optional[pd.DataFrame] = None,\n)\n Bases: Cache
The Newspaper class extends the Cache class and represents a newspaper.
The class has several properties and methods that allow the creation of a newspaper object and the manipulation of its data.
Attributes:
Name Type Descriptionroot An xml element that represents the root of the publication.
collection A string that represents the collection of the publication.
meta A dotdict object that holds metadata about the publication.
jisc_papers A pandas DataFrame object for JISC paper information.
Constructor method.
Source code inalto2txt2fixture/router.py def __init__(\nself,\nroot: ET.Element,\ncollection: str = \"\",\nmeta: dotdict = dotdict(),\njisc_papers: Optional[pd.DataFrame] = None,\n):\n\"\"\"Constructor method.\"\"\"\nif not isinstance(root, ET.Element):\nraise RuntimeError(f\"Expected root to be xml.etree.Element: {type(root)}\")\nself.publication = root.find(\"./publication\")\nself.input_sub_path = root.find(\"./process/input_sub_path\").text\nself.issue_date = self.publication.find(\"./issue/date\").text\nself.collection = collection\nself.meta = meta\nself.jisc_papers = jisc_papers\nself._newspaper = None\nself._title = None\nself._publication_code = None\npath = str(self.get_cache_path())\nif not self.meta.newspaper_paths:\nself.meta.newspaper_paths = []\nelif path not in self.meta.newspaper_paths:\nself.meta.newspaper_paths.append(path)\nif not self.meta.publication_codes:\nself.meta.publication_codes = [self.publication_code]\nelif self.publication_code not in self.meta.publication_codes:\nself.meta.publication_codes.append(self.publication_code)\nself.zip_file = Path(meta.path).name\nclass-attribute instance-attribute","text":"kind = 'newspaper'\nA string that represents the type of the object, set to \"newspaper\".
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.number_paths","title":"number_pathsproperty","text":"number_paths: list\nReturns the nested directories in which we want to save the cache file.
Returns:
Type Descriptionlist List of the desired directories in descending order
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.publication_code","title":"publication_codeproperty","text":"publication_code: str\nA property that returns the code of the publication.
Returns:
Type Descriptionstr The code of the publication
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.title","title":"titleproperty","text":"title: str\nA property that returns the title of the newspaper.
Returns:
Type Descriptionstr The title of the newspaper
"},{"location":"reference/alto2txt2fixture/router.html#alto2txt2fixture.router.Newspaper.as_dict","title":"as_dict","text":"as_dict() -> dict\nA method that returns a dictionary representation of the newspaper object.
Returns:
Type Descriptiondict Dictionary representation of the Newspaper object
Source code inalto2txt2fixture/router.py def as_dict(self) -> dict:\n\"\"\"\n A method that returns a dictionary representation of the newspaper\n object.\n Returns:\n Dictionary representation of the Newspaper object\n \"\"\"\nif not self._newspaper:\nself._newspaper = dict(\n**dict(publication_code=self.publication_code, title=self.title),\n**{\nx.tag: x.text or \"\"\nfor x in self.publication.findall(\"*\")\nif x.tag in [\"location\"]\n},\n)\nreturn self._newspaper\nget_cache_path() -> Path\nReturns the path to the cache file for the newspaper object.
Returns:
Type DescriptionPath Path to the cache file for the newspaper object
Source code inalto2txt2fixture/router.py def get_cache_path(self) -> Path:\n\"\"\"\n Returns the path to the cache file for the newspaper object.\n Returns:\n Path to the cache file for the newspaper object\n \"\"\"\njson_file = f\"/{self.publication_code}/{self.publication_code}.json\"\nreturn Path(\nf\"{CACHE_HOME}/{self.collection}/\" + \"/\".join(self.number_paths) + json_file\n)\npublication_code_from_input_sub_path() -> str | None\nA method that returns the publication code from the input sub-path of the publication process.
Returns:
Type Descriptionstr | None The code of the publication
Source code inalto2txt2fixture/router.py def publication_code_from_input_sub_path(self) -> str | None:\n\"\"\"\n A method that returns the publication code from the input sub-path of\n the publication process.\n Returns:\n The code of the publication\n \"\"\"\ng = PUBLICATION_CODE.findall(self.input_sub_path)\nif len(g) == 1:\nreturn g[0]\nreturn None\nroute(\ncollections: list,\ncache_home: str,\nmountpoint: str,\njisc_papers_path: str,\nreport_dir: str,\n) -> None\nThis function is responsible for setting up the path for the alto2txt mountpoint, setting up the JISC papers and routing the collections for processing.
Parameters:
Name Type Description Defaultcollections list List of collection names
requiredcache_home str Directory path for the cache
requiredmountpoint str Directory path for the alto2txt mountpoint
requiredjisc_papers_path str Path to the JISC papers
requiredreport_dir str Path to the report directory
requiredReturns:
Type DescriptionNone None
Source code inalto2txt2fixture/router.py def route(\ncollections: list,\ncache_home: str,\nmountpoint: str,\njisc_papers_path: str,\nreport_dir: str,\n) -> None:\n\"\"\"\n This function is responsible for setting up the path for the alto2txt\n mountpoint, setting up the JISC papers and routing the collections for\n processing.\n Args:\n collections: List of collection names\n cache_home: Directory path for the cache\n mountpoint: Directory path for the alto2txt mountpoint\n jisc_papers_path: Path to the JISC papers\n report_dir: Path to the report directory\n Returns:\n None\n \"\"\"\nglobal CACHE_HOME\nglobal MNT\nglobal REPORT_DIR\nCACHE_HOME = cache_home\nREPORT_DIR = report_dir\nMNT = Path(mountpoint) if isinstance(mountpoint, str) else mountpoint\nif not MNT.exists():\nerror(\nf\"The mountpoint provided for alto2txt does not exist. \"\nf\"Either create a local copy or blobfuse it to \"\nf\"`{MNT.absolute()}`.\"\n)\njisc_papers = setup_jisc_papers(path=jisc_papers_path)\nfor collection_name in collections:\ncollection = Collection(name=collection_name, jisc_papers=jisc_papers)\nif collection.empty:\nerror(\nf\"It looks like {collection_name} is empty in the \"\nf\"alto2txt mountpoint: `{collection.dir.absolute()}`.\"\n)\nfor archive in collection.archives:\nwith archive as _:\n[\n(\ndoc.item.write_to_cache(),\ndoc.newspaper.write_to_cache(),\ndoc.issue.write_to_cache(),\ndoc.data_provider.write_to_cache(),\ndoc.ingest.write_to_cache(),\ndoc.digitisation.write_to_cache(),\n)\nfor doc in archive.documents\n]\nreturn\nThe settings module provides configuration for running alto2txt2fixture.
Most of these are managed within the settings variable within this module.
Note
See the command line interface parameters documentation for means of modifying settings when run.
Attributes:
Name Type DescriptionJSON_INDEX Amount of indentation to include in output JSON files
DATA_PROVIDER_INDEX Final[str] The field used to index DataProvider records
NEWSPAPER_COLLECTION_METADATA Final[list[DataProviderFixtureDict]] A list of FixtureDicts specifying speific newspaper data providers
SETUP_TITLE str the title printed at the commandline via cli.show_setup() function
settings dotdict a docdict configuration for running newspaper portions of alto2txt2fixture
Bases: TypedDict
Fields within the fields portion of a FixtureDict to fit lwmdb.
Attributes:
Name Type Descriptionname str The name of the collection data source. For lwmdb this should be less than 600 characters.
code str | NEWSPAPER_OCR_FORMATS A short slug-like, url-compatible (replace spaces with -) str to uniquely identify a data provider in urls, api calls etc. Designed to fit NEWSPAPER_OCR_FORMATS and any future slug-like codes.
legacy_code LEGACY_NEWSPAPER_OCR_FORMATS | None Either blank or a legacy slug-like, url-compatible (replace spaces with -) str originally used by alto2txt following LEGACY_NEWSPAPER_OCR_FORMATSNEWSPAPER_OCR_FORMATS.
collection str Data Provider type.
source_note str | None A sentence about the data provider.
"},{"location":"reference/alto2txt2fixture/types.html#alto2txt2fixture.types.DataProviderFixtureDict","title":"DataProviderFixtureDict","text":" Bases: FixtureDictBaseClass
A dict structure for DataProvider sources in line with lwmdb.
Attributes:
Name Type Descriptionpk int an id to uniquely define and query each entry
model str what model a given record is for
fields DataProviderFieldsDict a DataProviderFieldsDict
Bases: FixtureDictBaseClass
A dict structure to ease use as a json database fixture.
Attributes:
Name Type Descriptionpk an id to uniquely define and query each entry
model what model a given record is for
fields dict[str, Any] a dict of record information conforming to model table
Bases: TypedDict
A base dict structure for json fixtures.
Bases: FixtureDictBaseClass
A dict structure for Fulltext sources in line with lwmdb.
Attributes:
Name Type Descriptionmodel str str in django fixture spec to indicate what model a record is for
fields PlaintextFixtureFieldsDict a PlaintextFixtureFieldsDict dict instance
pk int int id for fixture record
No pk is included. By not specifying one, django should generate new onces during import.
Bases: TypedDict
A typed dict for Plaintext Fixutres to match lwmdb.Fulltext model
Attributes:
Name Type Descriptiontext str Plaintext, potentially quite large newspaper articles. May have unusual or unreadable sequences of characters due to issues with Optical Character Recognition quality.
path str Path of provided plaintext file. If compressed_path is None, this is the original relative Path of the plaintext file.
compressed_path str | None The path of a compressed data source, the extraction of which provides access to plaintext files.
Bases: NamedTuple
A named tuple of fields for translation.
Attributes:
Name Type Descriptionstart str A string representing the starting field name.
finish str | list A string or list specifying the field(s) to be translated. If it is a string, the translated field will be a direct mapping of the specified field in each item of the input list. If it is a list, the translated field will be a hyphen-separated concatenation of the specified fields in each item of the input list.
lst list[dict] A list of dictionaries representing the items to be translated. Each dictionary should contain the necessary fields for translation, with the field names specified in the start parameter.
Bases: dict
dot.notation access to dictionary attributes
"},{"location":"reference/alto2txt2fixture/utils.html","title":"utils","text":""},{"location":"reference/alto2txt2fixture/utils.html#alto2txt2fixture.utils.DiskUsageTuple","title":"DiskUsageTuple","text":" Bases: NamedTuple
Type hint for nametuple returned from disk_usage.
check_newspaper_collection_configuration(\ncollections: Iterable[str] = settings.COLLECTIONS,\nnewspaper_collections: Iterable[\nFixtureDict\n] = NEWSPAPER_COLLECTION_METADATA,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> set[str]\nCheck the names in collections match the names in newspaper_collections.
Parameters:
Name Type Description Defaultcollections Iterable[str] Names of newspaper collections, defaults to settings.COLLECTIONS
settings.COLLECTIONS newspaper_collections Iterable[FixtureDict] Newspaper collections in a list of FixtureDict format. Defaults to settings.FIXTURE_TABLE['dataprovider]
NEWSPAPER_COLLECTION_METADATA data_provider_index str dict fields key used to check matchiching collections name
DATA_PROVIDER_INDEX Returns:
Type Descriptionset[str] A set of collections without a matching newspaper_collections record.
>>> check_newspaper_collection_configuration()\nset()\n>>> unmatched: set[str] = check_newspaper_collection_configuration(\n... [\"cat\", \"dog\"])\n<BLANKLINE>\n...Warning: 2 `collections` not in `newspaper_collections`: ...\n>>> unmatched == {'dog', 'cat'}\nTrue\nNote
Set orders are random so checking unmatched == {'dog, 'cat'} to ensure correctness irrespective of order in the example above.
alto2txt2fixture/utils.py def check_newspaper_collection_configuration(\ncollections: Iterable[str] = settings.COLLECTIONS,\nnewspaper_collections: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,\ndata_provider_index: str = DATA_PROVIDER_INDEX,\n) -> set[str]:\n\"\"\"Check the names in `collections` match the names in `newspaper_collections`.\n Arguments:\n collections:\n Names of newspaper collections, defaults to ``settings.COLLECTIONS``\n newspaper_collections:\n Newspaper collections in a list of `FixtureDict` format. Defaults\n to ``settings.FIXTURE_TABLE['dataprovider]``\n data_provider_index:\n `dict` `fields` `key` used to check matchiching `collections` name\n Returns:\n A set of ``collections`` without a matching `newspaper_collections` record.\n Example:\n ```pycon\n >>> check_newspaper_collection_configuration()\n set()\n >>> unmatched: set[str] = check_newspaper_collection_configuration(\n ... [\"cat\", \"dog\"])\n <BLANKLINE>\n ...Warning: 2 `collections` not in `newspaper_collections`: ...\n >>> unmatched == {'dog', 'cat'}\n True\n ```\n !!! note\n Set orders are random so checking `unmatched == {'dog, 'cat'}` to\n ensure correctness irrespective of order in the example above.\n \"\"\"\nnewspaper_collection_names: tuple[str, ...] = tuple(\ndict_from_list_fixture_fields(\nnewspaper_collections, field_name=data_provider_index\n).keys()\n)\ncollection_diff: set[str] = set(collections) - set(newspaper_collection_names)\nif collection_diff:\nwarning(\nf\"{len(collection_diff)} `collections` \"\nf\"not in `newspaper_collections`: {collection_diff}\"\n)\nreturn collection_diff\nclear_cache(dir: str | Path) -> None\nClears the cache directory by removing all .json files in it.
Parameters:
Name Type Description Defaultdir str | Path The path of the directory to be cleared.
required Source code inalto2txt2fixture/utils.py def clear_cache(dir: str | Path) -> None:\n\"\"\"\n Clears the cache directory by removing all `.json` files in it.\n Args:\n dir: The path of the directory to be cleared.\n \"\"\"\ndir = get_path_from(dir)\ny = input(\nf\"Do you want to erase the cache path now that the \"\nf\"files have been generated ({dir.absolute()})? [y/N]\"\n)\nif y.lower() == \"y\":\ninfo(\"Clearing up the cache directory\")\nfor x in dir.glob(\"*.json\"):\nx.unlink()\ncompress_fixture(\npath: PathLike,\noutput_path: PathLike | str = settings.OUTPUT,\nsuffix: str = \"\",\nformat: str | ArchiveFormatEnum = ZIP_FILE_EXTENSION,\nforce_overwrite: bool = False,\ndry_run: bool = False,\n) -> Path\nCompress exported fixtures files using make_archive.
Parameters:
Name Type Description Defaultpath PathLike Path to file to compress
output_path PathLike | str Compressed file name (without extension specified from format).
settings.OUTPUT format str | ArchiveFormatEnum A str of one of the registered compression formats. By default Python provides zip, tar, gztar, bztar, and xztar. See ArchiveFormatEnum variable for options checked.
ZIP_FILE_EXTENSION suffix str str to add to comprssed file name saved. For example: if path = plaintext_fixture-1.json and suffix=_compressed, then the saved file might be called plaintext_fixture-1_compressed.json.zip
'' Example >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n<BLANKLINE>\n...\n>>> tmp_path = getfixture('tmp_path')\n>>> json_path: Path = next(plaintext_bl_lwm.exported_json_paths)\n>>> assert 'pytest-of' in str(json_path)\n>>> compressed_path: Path = compress_fixture(path=json_path,\n... output_path=tmp_path,\n... dry_run=True)\n<BLANKLINE>\n...creating...'...plain...-...01.json.zip...'...addin...\n...'plain...01.json'...to...it...\n>>> compressed_path.exists()\nFalse\n>>> compressed_path: Path = compress_fixture(path=json_path,\n... output_path=tmp_path,\n... dry_run=False)\n<BLANKLINE>\n...creating...'...plain...-...01.json.zip...'...addin...\n...'plain...01.json'...to...it...\n>>> from zipfile import ZipFile, ZipInfo\n>>> zipfile_info_list: list[ZipInfo] = ZipFile(\n... tmp_path / 'plaintext_fixture-000001.json.zip'\n... ).infolist()\n>>> len(zipfile_info_list)\n1\n>>> Path(zipfile_info_list[0].filename).name\n'plaintext_fixture-000001.json'\nalto2txt2fixture/utils.py def compress_fixture(\npath: PathLike,\noutput_path: PathLike | str = settings.OUTPUT,\nsuffix: str = \"\",\nformat: str | ArchiveFormatEnum = ZIP_FILE_EXTENSION,\n# base_dir: PathLike | None = None,\nforce_overwrite: bool = False,\ndry_run: bool = False,\n) -> Path:\n\"\"\"Compress exported `fixtures` files using `make_archive`.\n Args:\n path:\n `Path` to file to compress\n output_path:\n Compressed file name (without extension specified from `format`).\n format:\n A `str` of one of the registered compression formats. By default\n `Python` provides `zip`, `tar`, `gztar`, `bztar`, and `xztar`.\n See `ArchiveFormatEnum` variable for options checked.\n suffix:\n `str` to add to comprssed file name saved.\n For example: if `path = plaintext_fixture-1.json` and\n `suffix=_compressed`, then the saved file might be called\n `plaintext_fixture-1_compressed.json.zip`\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext_json_export')\n <BLANKLINE>\n ...\n >>> tmp_path = getfixture('tmp_path')\n >>> json_path: Path = next(plaintext_bl_lwm.exported_json_paths)\n >>> assert 'pytest-of' in str(json_path)\n >>> compressed_path: Path = compress_fixture(path=json_path,\n ... output_path=tmp_path,\n ... dry_run=True)\n <BLANKLINE>\n ...creating...'...plain...-...01.json.zip...'...addin...\n ...'plain...01.json'...to...it...\n >>> compressed_path.exists()\n False\n >>> compressed_path: Path = compress_fixture(path=json_path,\n ... output_path=tmp_path,\n ... dry_run=False)\n <BLANKLINE>\n ...creating...'...plain...-...01.json.zip...'...addin...\n ...'plain...01.json'...to...it...\n >>> from zipfile import ZipFile, ZipInfo\n >>> zipfile_info_list: list[ZipInfo] = ZipFile(\n ... tmp_path / 'plaintext_fixture-000001.json.zip'\n ... ).infolist()\n >>> len(zipfile_info_list)\n 1\n >>> Path(zipfile_info_list[0].filename).name\n 'plaintext_fixture-000001.json'\n ```\n \"\"\"\npath = Path(path)\ncurrent_dir: Path = Path()\nroot_dir: str | None = None\nbase_dir: str | None = None\nif not path.exists():\nraise ValueError(f\"Cannot compress not existent 'path': {path}\")\nif isinstance(format, str):\ntry:\nformat = ArchiveFormatEnum(format)\nexcept ValueError:\nraise ValueError(\nf\"format '{format}' not valid, \"\nf\"options are:'\\n{pformat(ARCHIVE_FORMATS)}\"\n)\nchdir(str(Path(path).parent))\nif path.is_file():\nroot_dir = str(Path(path).parent)\nbase_dir = path.name\n# chdir(str(Path(path).parent))\nelif path.is_dir():\n# chdir(str(path))\nroot_dir = path.name\nelse:\nraise ValueError(f\"Path type {type(path)} is not supported.\")\nsave_file_name: Path = Path(Path(path).stem + suffix + \"\".join(Path(path).suffixes))\nsave_path: Path = Path(output_path) / save_file_name\n# root_dir: Path = save_path.parent\nif Path(str(save_path) + f\".{format}\").exists():\nerror_message: str = f\"Path to save to already exists: '{save_path}'\"\nif force_overwrite:\nlogger.warn(error_message)\nlogger.warn(f\"Overwriting '{save_path}'\")\nelse:\nraise ValueError(error_message)\nlogger.info(f\"Compressing '{path}' to '{format}' in: '{save_path.parent}'\")\n# if Path(path).is_file():\n# logger.info(f\"'path' to {format} is a file. Setting 'base_dir' to: '{path}'\")\n# base_dir = path\narchive_path: Path = Path(\nmake_archive(\nbase_name=str(save_path),\nformat=str(format),\nroot_dir=root_dir,\nbase_dir=base_dir,\n# root_dir=str(Path()),\n# base_dir=path,\ndry_run=dry_run,\nlogger=logger,\n)\n)\nchdir(current_dir)\nreturn archive_path\ncopy_dict_paths(copy_path_dict: dict[PathLike, PathLike]) -> None\nCopy files from copy_path_dict keys to values.
>>> tmp_path: Path = getfixture('tmp_path')\n>>> for i in range(4):\n... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n>>> pprint(sorted(tmp_path.iterdir()))\n[...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n>>> output_path = tmp_path / 'save'\n>>> output_path.mkdir(exist_ok=True)\n>>> copy_dict_paths(\n... glob_path_rename_by_0_padding(tmp_path,\n... glob_regex_str=\"*.txt\",\n... output_path=output_path))\n<BLANKLINE>\n...Specified...'...save'...for...saving...file...copies...\n...'...-0...txt'...to...'...-00...txt...'...\n...'...-1...txt'...to...'...-01...txt...'\n...'...-2...txt'...to...'...-02...txt...'\n...'...-3...txt'...to...'...-03...txt...'\n>>> pprint(sorted(tmp_path.iterdir()))\n[...Path('...save'),\n ...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n>>> pprint(sorted((tmp_path / 'save').iterdir()))\n [...Path('...test_file-00.txt'),\n ...Path('...test_file-01.txt'),\n ...Path('...test_file-02.txt'),\n ...Path('...test_file-03.txt')]\nalto2txt2fixture/utils.py def copy_dict_paths(copy_path_dict: dict[PathLike, PathLike]) -> None:\n\"\"\"Copy files from `copy_path_dict` `keys` to `values`.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> for i in range(4):\n ... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n >>> pprint(sorted(tmp_path.iterdir()))\n [...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n >>> output_path = tmp_path / 'save'\n >>> output_path.mkdir(exist_ok=True)\n >>> copy_dict_paths(\n ... glob_path_rename_by_0_padding(tmp_path,\n ... glob_regex_str=\"*.txt\",\n ... output_path=output_path))\n <BLANKLINE>\n ...Specified...'...save'...for...saving...file...copies...\n ...'...-0...txt'...to...'...-00...txt...'...\n ...'...-1...txt'...to...'...-01...txt...'\n ...'...-2...txt'...to...'...-02...txt...'\n ...'...-3...txt'...to...'...-03...txt...'\n >>> pprint(sorted(tmp_path.iterdir()))\n [...Path('...save'),\n ...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n >>> pprint(sorted((tmp_path / 'save').iterdir()))\n [...Path('...test_file-00.txt'),\n ...Path('...test_file-01.txt'),\n ...Path('...test_file-02.txt'),\n ...Path('...test_file-03.txt')]\n ```\n \"\"\"\nfor current_path, copy_path in copy_path_dict.items():\nlogger.info(f\"Copying '{current_path}' to '{copy_path}'\")\nPath(copy_path).parent.mkdir(exist_ok=True)\ncopyfile(current_path, copy_path)\ncreate_lookup(lst: list = [], on: list = []) -> dict\nCreate a lookup dictionary from a list of dictionaries.
Parameters:
Name Type Description Defaultlst list A list of dictionaries that should be used to generate the lookup.
[] on list A list of keys from the dictionaries in the list that should be used as the keys in the lookup.
[] Returns:
Type Descriptiondict The generated lookup dictionary.
Source code inalto2txt2fixture/utils.py def create_lookup(lst: list = [], on: list = []) -> dict:\n\"\"\"\n Create a lookup dictionary from a list of dictionaries.\n Args:\n lst: A list of dictionaries that should be used to generate the lookup.\n on: A list of keys from the dictionaries in the list that should be used as the keys in the lookup.\n Returns:\n The generated lookup dictionary.\n \"\"\"\nreturn {get_key(x, on): x[\"pk\"] for x in lst}\ndict_from_list_fixture_fields(\nfixture_list: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,\nfield_name: str = DATA_PROVIDER_INDEX,\n) -> dict[str, FixtureDict]\nCreate a dict from fixture_list with attr_name as key.
Parameters:
Name Type Description Defaultfixture_list Iterable[FixtureDict] list of FixtureDict with attr_name key fields.
NEWSPAPER_COLLECTION_METADATA field_name str key for values within fixture_list fields.
DATA_PROVIDER_INDEX Returns:
Type Descriptiondict[str, FixtureDict] A dict where extracted field_name is key for related FixtureDict values.
>>> fixture_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields()\n>>> fixture_dict['hmd']['pk']\n2\n>>> fixture_dict['hmd']['fields'][DATA_PROVIDER_INDEX]\n'hmd'\n>>> fixture_dict['hmd']['fields']['code']\n'bl_hmd'\nalto2txt2fixture/utils.py def dict_from_list_fixture_fields(\nfixture_list: Iterable[FixtureDict] = NEWSPAPER_COLLECTION_METADATA,\nfield_name: str = DATA_PROVIDER_INDEX,\n) -> dict[str, FixtureDict]:\n\"\"\"Create a `dict` from ``fixture_list`` with ``attr_name`` as `key`.\n Args:\n fixture_list: `list` of `FixtureDict` with ``attr_name`` key `fields`.\n field_name: key for values within ``fixture_list`` `fields`.\n Returns:\n A `dict` where extracted `field_name` is key for related `FixtureDict` values.\n Example:\n ```pycon\n >>> fixture_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields()\n >>> fixture_dict['hmd']['pk']\n 2\n >>> fixture_dict['hmd']['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n >>> fixture_dict['hmd']['fields']['code']\n 'bl_hmd'\n ```\n \"\"\"\nreturn {record[\"fields\"][field_name]: record for record in fixture_list}\nexport_fixtures(\nfixture_tables: dict[str, Sequence[FixtureDict]],\npath: str | PathLike = settings.FIXTURE_TABLES_OUTPUT,\nprefix: str = \"test-\",\nadd_created: bool = True,\nformats: Sequence[EXPORT_FORMATS] = settings.FIXTURE_TABLES_FORMATS,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None\nExport fixture_tables in formats.
This is still in experimental phase of development and not recommended for production.
Parameters:
Name Type Description Defaultfixture_tables dict[str, Sequence[FixtureDict]] dict of table name (eg: dataprovider) and FixtureDict
path str | PathLike Path to save exports in
settings.FIXTURE_TABLES_OUTPUT prefix str str to prefix export file names with
'test-' formats Sequence[EXPORT_FORMATS] list of EXPORT_FORMATS to export
settings.FIXTURE_TABLES_FORMATS file_name_0_padding int Zeros to prefix the number of each fixture file name.
FILE_NAME_0_PADDING_DEFAULT Example >>> test_fixture_tables: dict[str, FixtureDict] = {\n... 'test0': NEWSPAPER_COLLECTION_METADATA,\n... 'test1': NEWSPAPER_COLLECTION_METADATA}\n>>> export_fixtures(test_fixture_tables, path='tests/')\n<BLANKLINE>\n...Warning: Saving test0...\n...Warning: Saving test1...\n>>> from pandas import read_csv\n>>> fixture0_json = load_json('tests/test-test0-000001.json')\n>>> fixture0_df = read_csv('tests/test-test0-000001.csv')\n>>> fixture1_json = load_json('tests/test-test1-000001.json')\n>>> fixture1_df = read_csv('tests/test-test1-000001.csv')\n>>> fixture0_json == fixture1_json\nTrue\n>>> all(fixture0_df == fixture1_df)\nTrue\n>>> all(field in fixture0_json[0]['fields']\n... for field in ['created_at', 'updated_at'])\nTrue\n>>> fixture0_json[1]['pk']\n2\n>>> fixture0_json[1]['fields'][DATA_PROVIDER_INDEX]\n'hmd'\n>>> fixture0_df[['pk', DATA_PROVIDER_INDEX]].iloc[1].to_list()\n[2, 'hmd']\nalto2txt2fixture/utils.py def export_fixtures(\nfixture_tables: dict[str, Sequence[FixtureDict]],\npath: str | PathLike = settings.FIXTURE_TABLES_OUTPUT,\nprefix: str = \"test-\",\nadd_created: bool = True,\nformats: Sequence[EXPORT_FORMATS] = settings.FIXTURE_TABLES_FORMATS,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None:\n\"\"\"Export ``fixture_tables`` in ``formats``.\n Note:\n This is still in experimental phase of development and not recommended\n for production.\n Args:\n fixture_tables:\n `dict` of table name (eg: `dataprovider`) and `FixtureDict`\n path:\n `Path` to save exports in\n prefix:\n `str` to prefix export file names with\n formats:\n `list` of `EXPORT_FORMATS` to export\n file_name_0_padding:\n Zeros to prefix the number of each fixture file name.\n Example:\n ```pycon\n >>> test_fixture_tables: dict[str, FixtureDict] = {\n ... 'test0': NEWSPAPER_COLLECTION_METADATA,\n ... 'test1': NEWSPAPER_COLLECTION_METADATA}\n >>> export_fixtures(test_fixture_tables, path='tests/')\n <BLANKLINE>\n ...Warning: Saving test0...\n ...Warning: Saving test1...\n >>> from pandas import read_csv\n >>> fixture0_json = load_json('tests/test-test0-000001.json')\n >>> fixture0_df = read_csv('tests/test-test0-000001.csv')\n >>> fixture1_json = load_json('tests/test-test1-000001.json')\n >>> fixture1_df = read_csv('tests/test-test1-000001.csv')\n >>> fixture0_json == fixture1_json\n True\n >>> all(fixture0_df == fixture1_df)\n True\n >>> all(field in fixture0_json[0]['fields']\n ... for field in ['created_at', 'updated_at'])\n True\n >>> fixture0_json[1]['pk']\n 2\n >>> fixture0_json[1]['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n >>> fixture0_df[['pk', DATA_PROVIDER_INDEX]].iloc[1].to_list()\n [2, 'hmd']\n ```\n \"\"\"\nfor table_name, records in fixture_tables.items():\nwarning(\nf\"Saving {table_name} fixture in {formats} formats \"\nf\"to {path} *without* checks...\"\n)\nif \"json\" in formats:\nsave_fixture(\nrecords,\nprefix=f\"{prefix}{table_name}\",\noutput_path=path,\nadd_created=add_created,\nfile_name_0_padding=file_name_0_padding,\n)\nif \"csv\" in formats:\nfixtures_dict2csv(records, prefix=f\"{prefix}{table_name}\", output_path=path)\nfilter_json_fields(\njson_results: list | dict | None = None,\nfile_path: PathLike | None = None,\nfields: Sequence[str] = [],\nvalue: Hashable = \"\",\n**kwargs: Hashable\n) -> dict | list\nReturn keys and values from json_dict where any fields equal value.
Parameters:
Name Type Description Defaultfile_path PathLike | None The file path to load based on extension and filter
None fields Sequence[str] Which fields to check equal value
[] value Hashable Value to filter by
'' Returns:
Type Descriptiondict | list A dict of records indexed by pk which fit filter criteria
Raises:
Type DescriptionValueError file_path must have a .json suffix
>>> entry_fixture: dict = [\n... {\"pk\": 4889, \"model\": \"mitchells.entry\",\n... \"fields\": {\"title\": \"BIRMINGHAM POST .\",\n... \"price_raw\": ['2d'],\n... \"year\": 1920,\n... \"date_established_raw\": \"1857\",\n... \"persons\": [], \"newspaper\": \"\"}},\n... {\"pk\": 9207, \"model\": \"mitchells.entry\",\n... \"fields\": {\"title\": \"ULVERSTONE ADVERTISER .\",\n... \"price_raw\": ['2 \u00bd d', '3 \u00bd d'],\n... \"year\": 1856,\n... \"date_established_raw\": \"1848\",\n... \"persons\": ['Stephen Soulby'],\n... \"newspaper\": \"\",}},\n... {\"pk\": 15, \"model\": \"mitchells.entry\",\n... \"fields\": {\"title\": \"LLOYD'S WEEKLY LONDON NEWSPAPER .\",\n... \"price_raw\": ['2d', '3d'],\n... \"year\": 1857,\n... \"date_established_raw\": \"November , 1842\",\n... \"persons\": ['Mr. Douglas Jerrold', 'Edward Lloyd'],\n... \"newspaper\": 1187}}\n... ]\n>>> pprint(filter_json_fields(entry_fixture,\n... fields=(\"newspaper\", \"persons\"),\n... value=\"\"))\n[{'fields': {'date_established_raw': '1857',\n 'newspaper': '',\n 'persons': [],\n 'price_raw': ['2d'],\n 'title': 'BIRMINGHAM POST .',\n 'year': 1920},\n 'model': 'mitchells.entry',\n 'pk': 4889},\n {'fields': {'date_established_raw': '1848',\n 'newspaper': '',\n 'persons': ['Stephen Soulby'],\n 'price_raw': ['2 \u00bd d', '3 \u00bd d'],\n 'title': 'ULVERSTONE ADVERTISER .',\n 'year': 1856},\n 'model': 'mitchells.entry',\n 'pk': 9207}]\nalto2txt2fixture/utils.py def filter_json_fields(\njson_results: list | dict | None = None,\nfile_path: PathLike | None = None,\nfields: Sequence[str] = [],\nvalue: Hashable = \"\",\n**kwargs,\n) -> dict | list:\n\"\"\"Return `keys` and `values` from `json_dict` where any `fields` equal `value`.\n Args:\n file_path: The file `path` to load based on extension and filter\n fields: Which fields to check equal `value`\n value: Value to filter by\n Returns:\n A `dict` of records indexed by `pk` which fit filter criteria\n Raises:\n ValueError: ``file_path`` must have a `.json` `suffix`\n Example:\n ```pycon\n >>> entry_fixture: dict = [\n ... {\"pk\": 4889, \"model\": \"mitchells.entry\",\n ... \"fields\": {\"title\": \"BIRMINGHAM POST .\",\n ... \"price_raw\": ['2d'],\n ... \"year\": 1920,\n ... \"date_established_raw\": \"1857\",\n ... \"persons\": [], \"newspaper\": \"\"}},\n ... {\"pk\": 9207, \"model\": \"mitchells.entry\",\n ... \"fields\": {\"title\": \"ULVERSTONE ADVERTISER .\",\n ... \"price_raw\": ['2 \\u00bd d', '3 \\u00bd d'],\n ... \"year\": 1856,\n ... \"date_established_raw\": \"1848\",\n ... \"persons\": ['Stephen Soulby'],\n ... \"newspaper\": \"\",}},\n ... {\"pk\": 15, \"model\": \"mitchells.entry\",\n ... \"fields\": {\"title\": \"LLOYD'S WEEKLY LONDON NEWSPAPER .\",\n ... \"price_raw\": ['2d', '3d'],\n ... \"year\": 1857,\n ... \"date_established_raw\": \"November , 1842\",\n ... \"persons\": ['Mr. Douglas Jerrold', 'Edward Lloyd'],\n ... \"newspaper\": 1187}}\n ... ]\n >>> pprint(filter_json_fields(entry_fixture,\n ... fields=(\"newspaper\", \"persons\"),\n ... value=\"\"))\n [{'fields': {'date_established_raw': '1857',\n 'newspaper': '',\n 'persons': [],\n 'price_raw': ['2d'],\n 'title': 'BIRMINGHAM POST .',\n 'year': 1920},\n 'model': 'mitchells.entry',\n 'pk': 4889},\n {'fields': {'date_established_raw': '1848',\n 'newspaper': '',\n 'persons': ['Stephen Soulby'],\n 'price_raw': ['2 \\u00bd d', '3 \\u00bd d'],\n 'title': 'ULVERSTONE ADVERTISER .',\n 'year': 1856},\n 'model': 'mitchells.entry',\n 'pk': 9207}]\n ```\n \"\"\"\nif not json_results:\nassert file_path\ntry:\nassert Path(file_path).suffix == \".json\"\nexcept AssertionError:\nraise ValueError(f\"{file_path} must be `json` format.\")\njson_results = load_json(Path(file_path), **kwargs)\nassert json_results\nif isinstance(json_results, dict):\nreturn {\nk: v\nfor k, v in json_results.items()\nif any(v[\"fields\"][field] == value for field in fields)\n}\nelse:\nreturn [\nv\nfor v in json_results\nif any(v[\"fields\"][field] == value for field in fields)\n]\nfixture_fields(\nfixture_dict: FixtureDict, include_pk: bool = True, as_dict: bool = False\n) -> tuple[str, ...] | dict[str, Any]\nGenerate a tuple of FixtureDict field names.
This is not in the utils module to avoid a circular import.
Parameters:
Name Type Description Defaultfixture_dict FixtureDict A FixtureDict instance to extract names from fields
include_pk bool Whether to include the pk (primary key) column
True Example >>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0])\n('pk', 'name', 'code', 'legacy_code', 'collection', 'source_note')\n>>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0], include_pk=False)\n('name', 'code', 'legacy_code', 'collection', 'source_note')\n>>> hmd_dict: dict[str, Any] = fixture_fields(\n... NEWSPAPER_COLLECTION_METADATA[1], as_dict=True)\n>>> hmd_dict['code']\n'bl_hmd'\n>>> hmd_dict['pk']\n2\n>>> hmd_dict = fixture_fields(\n... NEWSPAPER_COLLECTION_METADATA[1], include_pk=False, as_dict=True)\n>>> 'pk' in hmd_dict\nFalse\nalto2txt2fixture/utils.py def fixture_fields(\nfixture_dict: FixtureDict, include_pk: bool = True, as_dict: bool = False\n) -> tuple[str, ...] | dict[str, Any]:\n\"\"\"Generate a tuple of `FixtureDict` `field` names.\n Note:\n This is not in the `utils` module to avoid a circular import.\n Args:\n fixture_dict: A `FixtureDict` instance to extract names from `fields`\n include_pk: Whether to include the `pk` (primary key) column\n Example:\n ```pycon\n >>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0])\n ('pk', 'name', 'code', 'legacy_code', 'collection', 'source_note')\n >>> fixture_fields(NEWSPAPER_COLLECTION_METADATA[0], include_pk=False)\n ('name', 'code', 'legacy_code', 'collection', 'source_note')\n >>> hmd_dict: dict[str, Any] = fixture_fields(\n ... NEWSPAPER_COLLECTION_METADATA[1], as_dict=True)\n >>> hmd_dict['code']\n 'bl_hmd'\n >>> hmd_dict['pk']\n 2\n >>> hmd_dict = fixture_fields(\n ... NEWSPAPER_COLLECTION_METADATA[1], include_pk=False, as_dict=True)\n >>> 'pk' in hmd_dict\n False\n ```\n \"\"\"\nfields: OrderedDict[str, Any] = OrderedDict(fixture_dict[\"fields\"])\nif include_pk:\nfields[\"pk\"] = fixture_dict[\"pk\"]\nfields.move_to_end(\"pk\", last=False)\nif as_dict:\nreturn fields\nelse:\nreturn tuple(fields.keys())\nfixture_or_default_dict(\nkey: str,\nfixture_dict: dict[str, FixtureDict],\ndefault_dict: FixtureDict | dict = {},\n) -> FixtureDict | dict\nReturn a FixtureDict from fixture_list via key index, else default_dict.
Parameters:
Name Type Description Defaultkey str a str to query fixture_dict with
fixture_dict dict[str, FixtureDict] a dict of str to FixtureDict, often generated by dict_from_list_fixture_fields
default_dict FixtureDict | dict a dict to return if key is not in fixture_dict index
{} Example >>> newspaper_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields(\n... NEWSPAPER_COLLECTION_METADATA)\n>>> hmd_dict: FixtureDict = fixture_or_default_dict(\n... 'hmd', newspaper_dict\n... )\n>>> hmd_dict == newspaper_dict['hmd']\nTrue\n>>> fixture_or_default_dict(\n... 'hmd', NEWSPAPER_COLLECTION_METADATA\n... )\n{}\n>>> fixture_or_default_dict(\n... 'hmd', NEWSPAPER_COLLECTION_METADATA, {'a': 'default'}\n... )\n{'a': 'default'}\nalto2txt2fixture/utils.py def fixture_or_default_dict(\nkey: str,\nfixture_dict: dict[str, FixtureDict],\ndefault_dict: FixtureDict | dict = {},\n) -> FixtureDict | dict:\n\"\"\"Return a `FixtureDict` from ``fixture_list`` via ``key`` index, else ``default_dict``.\n Args:\n key:\n a `str` to query ``fixture_dict`` with\n fixture_dict:\n a `dict` of `str` to `FixtureDict`, often generated by\n ``dict_from_list_fixture_fields``\n default_dict:\n a `dict` to return if ``key`` is not in ``fixture_dict`` index\n Example:\n ```pycon\n >>> newspaper_dict: dict[str, FixtureDict] = dict_from_list_fixture_fields(\n ... NEWSPAPER_COLLECTION_METADATA)\n >>> hmd_dict: FixtureDict = fixture_or_default_dict(\n ... 'hmd', newspaper_dict\n ... )\n >>> hmd_dict == newspaper_dict['hmd']\n True\n >>> fixture_or_default_dict(\n ... 'hmd', NEWSPAPER_COLLECTION_METADATA\n ... )\n {}\n >>> fixture_or_default_dict(\n ... 'hmd', NEWSPAPER_COLLECTION_METADATA, {'a': 'default'}\n ... )\n {'a': 'default'}\n ```\n \"\"\"\nif key in fixture_dict:\nreturn fixture_dict[key]\nelse:\nreturn default_dict\nfixtures_dict2csv(\nfixtures: Iterable[FixtureDict] | Generator[FixtureDict, None, None],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nindex: bool = False,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None\nSaves fixtures generated by a generator to separate separate CSV files.
This function takes an Iterable or Generator of fixtures and saves to separate CSV files. The fixtures are saved in batches, where each batch is determined by the max_elements_per_file parameter.
Parameters:
Name Type Description Defaultfixtures Iterable[FixtureDict] | Generator[FixtureDict, None, None] An Iterable or Generator of the fixtures to be saved.
prefix str A string prefix to be added to the file names of the saved fixtures.
'' output_path PathLike | str Path to folder fixtures are saved to
settings.OUTPUT max_elements_per_file int Maximum JSON records saved in each file
settings.MAX_ELEMENTS_PER_FILE file_name_0_padding int Zeros to prefix the number of each fixture file name.
FILE_NAME_0_PADDING_DEFAULT Returns:
Type DescriptionNone This function saves fixtures to files and does not return a value.
Example>>> tmp_path: Path = getfixture('tmp_path')\n>>> from pandas import read_csv\n>>> fixtures_dict2csv(NEWSPAPER_COLLECTION_METADATA,\n... prefix='test', output_path=tmp_path)\n>>> imported_fixture = read_csv(tmp_path / 'test-000001.csv')\n>>> imported_fixture.iloc[1]['pk']\n2\n>>> imported_fixture.iloc[1][DATA_PROVIDER_INDEX]\n'hmd'\nalto2txt2fixture/utils.py def fixtures_dict2csv(\nfixtures: Iterable[FixtureDict] | Generator[FixtureDict, None, None],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nindex: bool = False,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None:\n\"\"\"Saves fixtures generated by a generator to separate separate `CSV` files.\n This function takes an `Iterable` or `Generator` of fixtures and saves to\n separate `CSV` files. The fixtures are saved in batches, where each batch\n is determined by the ``max_elements_per_file`` parameter.\n Args:\n fixtures:\n An `Iterable` or `Generator` of the fixtures to be saved.\n prefix:\n A string prefix to be added to the file names of the\n saved fixtures.\n output_path:\n Path to folder fixtures are saved to\n max_elements_per_file:\n Maximum `JSON` records saved in each file\n file_name_0_padding:\n Zeros to prefix the number of each fixture file name.\n Returns:\n This function saves fixtures to files and does not return a value.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> from pandas import read_csv\n >>> fixtures_dict2csv(NEWSPAPER_COLLECTION_METADATA,\n ... prefix='test', output_path=tmp_path)\n >>> imported_fixture = read_csv(tmp_path / 'test-000001.csv')\n >>> imported_fixture.iloc[1]['pk']\n 2\n >>> imported_fixture.iloc[1][DATA_PROVIDER_INDEX]\n 'hmd'\n ```\n \"\"\"\ninternal_counter: int = 1\ncounter: int = 1\nlst: list = []\nfile_name: str\ndf: DataFrame\nPath(output_path).mkdir(parents=True, exist_ok=True)\nfor item in fixtures:\nlst.append(fixture_fields(item, as_dict=True))\ninternal_counter += 1\nif internal_counter > max_elements_per_file:\ndf = DataFrame.from_records(lst)\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.csv\"\ndf.to_csv(Path(output_path) / file_name, index=index)\n# Save up some memory\ndel lst\ngc.collect()\n# Re-instantiate\nlst = []\ninternal_counter = 1\ncounter += 1\nelse:\ndf = DataFrame.from_records(lst)\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.csv\"\ndf.to_csv(Path(output_path) / file_name, index=index)\nfree_hd_space_in_GB(\ndisk_usage_tuple: DiskUsageTuple | None = None, path: PathLike | None = None\n) -> float\nReturn remaing hard drive space estimate in gigabytes.
Parameters:
Name Type Description Defaultdisk_usage_tuple DiskUsageTuple | None A NamedTuple normally returned from disk_usage() or None.
None path PathLike | None A path to pass to disk_usage if disk_usage_tuple is None.
None Returns:
Type Descriptionfloat A float from dividing the disk_usage_tuple.free value by BYTES_PER_GIGABYTE
>>> space_in_gb = free_hd_space_in_GB()\n>>> space_in_gb > 1 # Hopefully true wherever run...\nTrue\nalto2txt2fixture/utils.py def free_hd_space_in_GB(\ndisk_usage_tuple: DiskUsageTuple | None = None, path: PathLike | None = None\n) -> float:\n\"\"\"Return remaing hard drive space estimate in gigabytes.\n Args:\n disk_usage_tuple:\n A `NamedTuple` normally returned from `disk_usage()` or `None`.\n path:\n A `path` to pass to `disk_usage` if `disk_usage_tuple` is `None`.\n Returns:\n A `float` from dividing the `disk_usage_tuple.free` value by `BYTES_PER_GIGABYTE`\n Example:\n ```pycon\n >>> space_in_gb = free_hd_space_in_GB()\n >>> space_in_gb > 1 # Hopefully true wherever run...\n True\n ```\n \"\"\"\nif not disk_usage_tuple:\nif not path:\npath = Path(getcwd())\ndisk_usage_tuple = disk_usage(path=path)\nassert disk_usage_tuple\nreturn disk_usage_tuple.free / BYTES_PER_GIGABYTE\ngen_fixture_tables(\nfixture_tables: dict[str, list[FixtureDict]] = {},\ninclude_fixture_pk_column: bool = True,\n) -> Generator[Table, None, None]\nGenerator of rich.Table instances from FixtureDict configuration tables.
Parameters:
Name Type Description Defaultfixture_tables dict[str, list[FixtureDict]] dict where key is for Table title and value is a FixtureDict
{} include_fixture_pk_column bool whether to include the pk field from FixtureDict
True Example >>> table_name: str = \"data_provider\"\n>>> tables = tuple(\n... gen_fixture_tables(\n... {table_name: NEWSPAPER_COLLECTION_METADATA}\n... ))\n>>> len(tables)\n1\n>>> assert tables[0].title == table_name\n>>> [column.header for column in tables[0].columns]\n['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\nalto2txt2fixture/utils.py def gen_fixture_tables(\nfixture_tables: dict[str, list[FixtureDict]] = {},\ninclude_fixture_pk_column: bool = True,\n) -> Generator[Table, None, None]:\n\"\"\"Generator of `rich.Table` instances from `FixtureDict` configuration tables.\n Args:\n fixture_tables: `dict` where `key` is for `Table` title and `value` is a `FixtureDict`\n include_fixture_pk_column: whether to include the `pk` field from `FixtureDict`\n Example:\n ```pycon\n >>> table_name: str = \"data_provider\"\n >>> tables = tuple(\n ... gen_fixture_tables(\n ... {table_name: NEWSPAPER_COLLECTION_METADATA}\n ... ))\n >>> len(tables)\n 1\n >>> assert tables[0].title == table_name\n >>> [column.header for column in tables[0].columns]\n ['pk', 'name', 'code', 'legacy_code', 'collection', 'source_note']\n ```\n \"\"\"\nfor name, fixture_records in fixture_tables.items():\nfixture_table: Table = Table(title=name)\nfor i, fixture_dict in enumerate(fixture_records):\nif i == 0:\n[\nfixture_table.add_column(name)\nfor name in fixture_fields(fixture_dict, include_fixture_pk_column)\n]\nrow_values: tuple[str, ...] = tuple(\nstr(x) for x in (fixture_dict[\"pk\"], *fixture_dict[\"fields\"].values())\n)\nfixture_table.add_row(*row_values)\nyield fixture_table\nget_chunked_zipfiles(path: Path) -> list\nThis function takes in a Path object path and returns a list of lists of zipfiles sorted and chunked according to certain conditions defined in the settings object (see settings.CHUNK_THRESHOLD).
Note: the function will also skip zip files of a certain file size, which can be specified in the settings object (see settings.SKIP_FILE_SIZE).
Parameters:
Name Type Description Defaultpath Path The input path where the zipfiles are located
requiredReturns:
Type Descriptionlist A list of lists of zipfiles, each inner list represents a chunk of zipfiles.
alto2txt2fixture/utils.py def get_chunked_zipfiles(path: Path) -> list:\n\"\"\"This function takes in a `Path` object `path` and returns a list of lists\n of `zipfiles` sorted and chunked according to certain conditions defined\n in the `settings` object (see `settings.CHUNK_THRESHOLD`).\n Note: the function will also skip zip files of a certain file size, which\n can be specified in the `settings` object (see `settings.SKIP_FILE_SIZE`).\n Args:\n path: The input path where the zipfiles are located\n Returns:\n A list of lists of `zipfiles`, each inner list represents a chunk of\n zipfiles.\n \"\"\"\nzipfiles = sorted(\npath.glob(\"*.zip\"),\nkey=lambda x: x.stat().st_size,\nreverse=settings.START_WITH_LARGEST,\n)\nzipfiles = [x for x in zipfiles if x.stat().st_size <= settings.SKIP_FILE_SIZE]\nif len(zipfiles) > settings.CHUNK_THRESHOLD:\nchunks = array_split(zipfiles, len(zipfiles) / settings.CHUNK_THRESHOLD)\nelse:\nchunks = [zipfiles]\nreturn chunks\nget_key(x: dict = dict(), on: list = []) -> str\nGet a string key from a dictionary using values from specified keys.
Parameters:
Name Type Description Defaultx dict A dictionary from which the key is generated.
dict() on list A list of keys from the dictionary that should be used to generate the key.
[] Returns:
Type Descriptionstr The generated string key.
Source code inalto2txt2fixture/utils.py def get_key(x: dict = dict(), on: list = []) -> str:\n\"\"\"\n Get a string key from a dictionary using values from specified keys.\n Args:\n x: A dictionary from which the key is generated.\n on: A list of keys from the dictionary that should be used to\n generate the key.\n Returns:\n The generated string key.\n \"\"\"\nreturn f\"{'-'.join([str(x['fields'][y]) for y in on])}\"\nget_lockfile(collection: str, kind: NewspaperElements, dic: dict) -> Path\nProvides the path to any given lockfile, which controls whether any existing files should be overwritten or not.
Parameters:
Name Type Description Defaultcollection str Collection folder name
requiredkind NewspaperElements Either newspaper or issue or item
dic dict A dictionary with required information for either kind passed
Returns:
Type DescriptionPath Path to the resulting lockfile
Source code inalto2txt2fixture/utils.py def get_lockfile(collection: str, kind: NewspaperElements, dic: dict) -> Path:\n\"\"\"\n Provides the path to any given lockfile, which controls whether any\n existing files should be overwritten or not.\n Args:\n collection: Collection folder name\n kind: Either `newspaper` or `issue` or `item`\n dic: A dictionary with required information for either `kind` passed\n Returns:\n Path to the resulting lockfile\n \"\"\"\np: Path\nbase = Path(f\"cache-lockfiles/{collection}\")\nif kind == \"newspaper\":\np = base / f\"newspapers/{dic['publication_code']}\"\nelif kind == \"issue\":\np = base / f\"issues/{dic['publication__publication_code']}/{dic['issue_code']}\"\nelif kind == \"item\":\ntry:\nif dic.get(\"issue_code\"):\np = base / f\"items/{dic['issue_code']}/{dic['item_code']}\"\nelif dic.get(\"issue__issue_identifier\"):\np = base / f\"items/{dic['issue__issue_identifier']}/{dic['item_code']}\"\nexcept KeyError:\nerror(\"An unknown error occurred (in get_lockfile)\")\nelse:\np = base / \"lockfile\"\np.parent.mkdir(parents=True, exist_ok=True) if settings.WRITE_LOCKFILES else None\nreturn p\nget_now(as_str: bool = False) -> datetime.datetime | str\nReturn datetime.now() as either a string or datetime object.
Parameters:
Name Type Description Defaultas_str bool Whether to return now time as a str or not, default: False
False Returns:
Type Descriptiondatetime.datetime | str datetime.now() in pytz.UTC time zone as a string if as_str, else as a datetime.datetime object.
alto2txt2fixture/utils.py def get_now(as_str: bool = False) -> datetime.datetime | str:\n\"\"\"\n Return `datetime.now()` as either a string or `datetime` object.\n Args:\n as_str: Whether to return `now` `time` as a `str` or not, default: `False`\n Returns:\n `datetime.now()` in `pytz.UTC` time zone as a string if `as_str`, else\n as a `datetime.datetime` object.\n \"\"\"\nnow = datetime.datetime.now(tz=pytz.UTC)\nif as_str:\nreturn str(now)\nelse:\nassert isinstance(now, datetime.datetime)\nreturn now\nget_path_from(p: str | Path) -> Path\nConverts an input value into a Path object if it's not already one.
Parameters:
Name Type Description Defaultp str | Path The input value, which can be a string or a Path object.
requiredReturns:
Type DescriptionPath The input value as a Path object.
Source code inalto2txt2fixture/utils.py def get_path_from(p: str | Path) -> Path:\n\"\"\"\n Converts an input value into a Path object if it's not already one.\n Args:\n p: The input value, which can be a string or a Path object.\n Returns:\n The input value as a Path object.\n \"\"\"\nif isinstance(p, str):\np = Path(p)\nif not isinstance(p, Path):\nraise RuntimeError(f\"Unable to handle type: {type(p)}\")\nreturn p\nget_size_from_path(p: str | Path, raw: bool = False) -> str | float\nReturns a nice string for any given file size.
Parameters:
Name Type Description Defaultp str | Path Path to read the size from
requiredraw bool Whether to return the file size as total number of bytes or a human-readable MB/GB amount
False Returns:
Type Descriptionstr | float Return str followed by MB or GB for size if not raw otherwise float.
alto2txt2fixture/utils.py def get_size_from_path(p: str | Path, raw: bool = False) -> str | float:\n\"\"\"\n Returns a nice string for any given file size.\n Args:\n p: Path to read the size from\n raw: Whether to return the file size as total number of bytes or\n a human-readable MB/GB amount\n Returns:\n Return `str` followed by `MB` or `GB` for size if not `raw` otherwise `float`.\n \"\"\"\np = get_path_from(p)\nbytes = p.stat().st_size\nif raw:\nreturn bytes\nrel_size: float | int | str = round(bytes / 1000 / 1000 / 1000, 1)\nassert not isinstance(rel_size, str)\nif rel_size < 0.5:\nrel_size = round(bytes / 1000 / 1000, 1)\nrel_size = f\"{rel_size}MB\"\nelse:\nrel_size = f\"{rel_size}GB\"\nreturn rel_size\nglob_filter(p: str) -> list\nReturn ordered glob, filtered out any pesky, unwanted .DS_Store from macOS.
Parameters:
Name Type Description Defaultp str Path to a directory to filter
requiredReturns:
Type Descriptionlist Sorted list of files contained in the provided path without the ones
list whose names start with a .
alto2txt2fixture/utils.py def glob_filter(p: str) -> list:\n\"\"\"\n Return ordered glob, filtered out any pesky, unwanted .DS_Store from macOS.\n Args:\n p: Path to a directory to filter\n Returns:\n Sorted list of files contained in the provided path without the ones\n whose names start with a `.`\n \"\"\"\nreturn sorted([x for x in get_path_from(p).glob(\"*\") if not x.name.startswith(\".\")])\nglob_path_rename_by_0_padding(\npath: PathLike,\noutput_path: PathLike | None = None,\nglob_regex_str: str = \"*\",\npadding: int | None = 0,\nmatch_int_regex: str = PADDING_0_REGEX_DEFAULT,\nindex: int = -1,\n) -> dict[PathLike, PathLike]\nReturn an OrderedDict of replacement 0-padded file names from path.
Parameters:
Name Type Description Defaultpath PathLike PathLike to source files to rename.
output_path PathLike | None PathLike to save renamed files to.
None glob_regex_str str str to match files to rename within path.
'*' padding int | None How many digits (0s) to pad match_int with.
0 match_int_regex str Regular expression for matching numbers in s to pad. Only rename parts of Path(file_path).name; else replace across Path(file_path).parents as well.
PADDING_0_REGEX_DEFAULT index int Which index of number in s to pad with 0s. Like numbering a list, 0 indicates the first match and -1 indicates the last match.
-1 Example >>> tmp_path: Path = getfixture('tmp_path')\n>>> for i in range(4):\n... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n>>> pprint(sorted(tmp_path.iterdir()))\n[...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n>>> pprint(glob_path_rename_by_0_padding(tmp_path))\n{...Path('...test_file-0.txt'): ...Path('...test_file-00.txt'),\n ...Path('...test_file-1.txt'): ...Path('...test_file-01.txt'),\n ...Path('...test_file-2.txt'): ...Path('...test_file-02.txt'),\n ...Path('...test_file-3.txt'): ...Path('...test_file-03.txt')}\nalto2txt2fixture/utils.py def glob_path_rename_by_0_padding(\npath: PathLike,\noutput_path: PathLike | None = None,\nglob_regex_str: str = \"*\",\npadding: int | None = 0,\nmatch_int_regex: str = PADDING_0_REGEX_DEFAULT,\nindex: int = -1,\n) -> dict[PathLike, PathLike]:\n\"\"\"Return an `OrderedDict` of replacement 0-padded file names from `path`.\n Params:\n path:\n `PathLike` to source files to rename.\n output_path:\n `PathLike` to save renamed files to.\n glob_regex_str:\n `str` to match files to rename within `path`.\n padding:\n How many digits (0s) to pad `match_int` with.\n match_int_regex:\n Regular expression for matching numbers in `s` to pad.\n Only rename parts of `Path(file_path).name`; else\n replace across `Path(file_path).parents` as well.\n index:\n Which index of number in `s` to pad with 0s.\n Like numbering a `list`, 0 indicates the first match\n and -1 indicates the last match.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> for i in range(4):\n ... (tmp_path / f'test_file-{i}.txt').touch(exist_ok=True)\n >>> pprint(sorted(tmp_path.iterdir()))\n [...Path('...test_file-0.txt'),\n ...Path('...test_file-1.txt'),\n ...Path('...test_file-2.txt'),\n ...Path('...test_file-3.txt')]\n >>> pprint(glob_path_rename_by_0_padding(tmp_path))\n {...Path('...test_file-0.txt'): ...Path('...test_file-00.txt'),\n ...Path('...test_file-1.txt'): ...Path('...test_file-01.txt'),\n ...Path('...test_file-2.txt'): ...Path('...test_file-02.txt'),\n ...Path('...test_file-3.txt'): ...Path('...test_file-03.txt')}\n ```\n \"\"\"\ntry:\nassert Path(path).exists()\nexcept AssertionError:\nraise ValueError(f'path does not exist: \"{Path(path)}\"')\npaths_tuple: tuple[PathLike, ...] = path_globs_to_tuple(path, glob_regex_str)\ntry:\nassert paths_tuple\nexcept AssertionError:\nraise FileNotFoundError(\nf\"No files found matching 'glob_regex_str': \"\nf\"'{glob_regex_str}' in: '{path}'\"\n)\npaths_to_index: tuple[tuple[str, int], ...] = tuple(\nint_from_str(str(matched_path), index=index, regex=match_int_regex)\nfor matched_path in paths_tuple\n)\nmax_index: int = max(index[1] for index in paths_to_index)\nmax_index_digits: int = len(str(max_index))\nif not padding or padding < max_index_digits:\npadding = max_index_digits + 1\nnew_names_dict: dict[PathLike, PathLike] = {}\nif output_path:\nif not Path(output_path).is_absolute():\noutput_path = Path(path) / output_path\nlogger.debug(f\"Specified '{output_path}' for saving file copies\")\nfor i, old_path in enumerate(paths_tuple):\nmatch_str, match_int = paths_to_index[i]\nnew_names_dict[old_path] = rename_by_0_padding(\nold_path, match_str=str(match_str), match_int=match_int, padding=padding\n)\nif output_path:\nnew_names_dict[old_path] = (\nPath(output_path) / Path(new_names_dict[old_path]).name\n)\nreturn new_names_dict\nint_from_str(\ns: str, index: int = -1, regex: str = PADDING_0_REGEX_DEFAULT\n) -> tuple[str, int]\nReturn matched (or None) regex from s by index index.
Parameters:
Name Type Description Defaults str str to match and via regex.
index int Which index of number in s to pad with 0s. Like numbering a list, 0 indicates the first match and -1 indicates the last match.
-1 regex str Regular expression for matching numbers in s to pad.
PADDING_0_REGEX_DEFAULT Example >>> int_from_str('a/path/to/fixture-03-05.txt')\n('05', 5)\n>>> int_from_str('a/path/to/fixture-03-05.txt', index=0)\n('03', 3)\nalto2txt2fixture/utils.py def int_from_str(\ns: str,\nindex: int = -1,\nregex: str = PADDING_0_REGEX_DEFAULT,\n) -> tuple[str, int]:\n\"\"\"Return matched (or None) `regex` from `s` by index `index`.\n Params:\n s:\n `str` to match and via `regex`.\n index:\n Which index of number in `s` to pad with 0s.\n Like numbering a `list`, 0 indicates the first match\n and -1 indicates the last match.\n regex:\n Regular expression for matching numbers in `s` to pad.\n Example:\n ```pycon\n >>> int_from_str('a/path/to/fixture-03-05.txt')\n ('05', 5)\n >>> int_from_str('a/path/to/fixture-03-05.txt', index=0)\n ('03', 3)\n ```\n \"\"\"\nmatches: list[str] = [match for match in findall(regex, s) if match]\nmatch_str: str = matches[index]\nreturn match_str, int(match_str)\nlist_json_files(\np: str | Path,\ndrill: bool = False,\nexclude_names: list = [],\ninclude_names: list = [],\n) -> Generator[Path, None, None] | list[Path]\nList json files under the path specified in p.
Parameters:
Name Type Description Defaultp str | Path The path to search for json files
drill bool A flag indicating whether to drill down the subdirectories or not. Default is False
False exclude_names list A list of file names to exclude from the search result. Default is an empty list
[] include_names list A list of file names to include in search result. If provided, the exclude_names argument will be ignored. Default is an empty list
[] Returns:
Type DescriptionGenerator[Path, None, None] | list[Path] A list of Path objects pointing to the found json files
alto2txt2fixture/utils.py def list_json_files(\np: str | Path,\ndrill: bool = False,\nexclude_names: list = [],\ninclude_names: list = [],\n) -> Generator[Path, None, None] | list[Path]:\n\"\"\"\n List `json` files under the path specified in ``p``.\n Args:\n p: The path to search for `json` files\n drill: A flag indicating whether to drill down the subdirectories\n or not. Default is ``False``\n exclude_names: A list of file names to exclude from the search\n result. Default is an empty list\n include_names: A list of file names to include in search result.\n If provided, the ``exclude_names`` argument will be ignored.\n Default is an empty list\n Returns:\n A list of `Path` objects pointing to the found `json` files\n \"\"\"\nq: str = \"**/*.json\" if drill else \"*.json\"\nfiles = get_path_from(p).glob(q)\nif exclude_names:\nfiles = list({x for x in files if x.name not in exclude_names})\nelif include_names:\nfiles = list({x for x in files if x.name in include_names})\nreturn sorted(files)\nload_json(p: str | Path, crash: bool = False) -> dict | list\nEasier access to reading json files.
Parameters:
Name Type Description Defaultp str | Path Path to read json from
crash bool Whether the program should crash if there is a json decode error, default: False
False Returns:
Type Descriptiondict | list The decoded json contents from the path, but an empty dictionary
dict | list if the file cannot be decoded and crash is set to False
alto2txt2fixture/utils.py def load_json(p: str | Path, crash: bool = False) -> dict | list:\n\"\"\"\n Easier access to reading `json` files.\n Args:\n p: Path to read `json` from\n crash: Whether the program should crash if there is a `json` decode\n error, default: ``False``\n Returns:\n The decoded `json` contents from the path, but an empty dictionary\n if the file cannot be decoded and ``crash`` is set to ``False``\n \"\"\"\np = get_path_from(p)\ntry:\nreturn json.loads(p.read_text())\nexcept json.JSONDecodeError:\nmsg = f\"Error: {p.read_text()}\"\nerror(msg, crash=crash)\nreturn {}\nload_multiple_json(\np: str | Path,\ndrill: bool = False,\nfilter_na: bool = True,\ncrash: bool = False,\n) -> list\nLoad multiple json files and return a list of their content.
Parameters:
Name Type Description Defaultp str | Path The path to search for json files
drill bool A flag indicating whether to drill down the subdirectories or not. Default is False
False filter_na bool A flag indicating whether to filter out the content that is None. Default is True.
True crash bool A flag indicating whether to raise an exception when an error occurs while loading a json file. Default is False.
False Returns:
Type Descriptionlist A list of the content of the loaded json files.
alto2txt2fixture/utils.py def load_multiple_json(\np: str | Path,\ndrill: bool = False,\nfilter_na: bool = True,\ncrash: bool = False,\n) -> list:\n\"\"\"\n Load multiple `json` files and return a list of their content.\n Args:\n p: The path to search for `json` files\n drill: A flag indicating whether to drill down the subdirectories\n or not. Default is `False`\n filter_na: A flag indicating whether to filter out the content that\n is `None`. Default is `True`.\n crash: A flag indicating whether to raise an exception when an\n error occurs while loading a `json` file. Default is `False`.\n Returns:\n A `list` of the content of the loaded `json` files.\n \"\"\"\nfiles = list_json_files(p, drill=drill)\ncontent = [load_json(x, crash=crash) for x in files]\nreturn [x for x in content if x] if filter_na else content\nlock(lockfile: Path) -> None\nWrites a '.' to a lockfile, after making sure the parent directory exists.
Parameters:
Name Type Description Defaultlockfile Path The path to the lock file to be created
requiredReturns:
Type DescriptionNone None
Source code inalto2txt2fixture/utils.py def lock(lockfile: Path) -> None:\n\"\"\"\n Writes a '.' to a lockfile, after making sure the parent directory exists.\n Args:\n lockfile: The path to the lock file to be created\n Returns:\n None\n \"\"\"\nlockfile.parent.mkdir(parents=True, exist_ok=True)\nlockfile.write_text(\"\")\nreturn\npath_globs_to_tuple(\npath: PathLike, glob_regex_str: str = \"*\"\n) -> tuple[PathLike, ...]\nReturn a sorted tuple of Paths in path using glob_regex_str.
Parameters:
Name Type Description Defaultpath PathLike Patch to search via glob
glob_regex_str str Regular expression to use with glob at path
'*' Returns:
Type Descriptiontuple[PathLike, ...] tuple of matching paths.
>>> bl_lwm = getfixture(\"bl_lwm\")\n>>> pprint(path_globs_to_tuple(bl_lwm, '*text.zip'))\n(...Path('...bl_lwm...0003079-test_plaintext.zip'),\n ...Path('...bl_lwm...0003548-test_plaintext.zip'))\n>>> pprint(path_globs_to_tuple(bl_lwm, '*.txt'))\n(...Path('...bl_lwm...0003079_18980121_sect0001.txt'),\n ...Path('...bl_lwm...0003548_19040707_art0037.txt'))\nalto2txt2fixture/utils.py def path_globs_to_tuple(\npath: PathLike, glob_regex_str: str = \"*\"\n) -> tuple[PathLike, ...]:\n\"\"\"Return a sorted `tuple` of `Path`s in `path` using `glob_regex_str`.\n Args:\n path:\n Patch to search via `glob`\n glob_regex_str:\n Regular expression to use with `glob` at `path`\n Returns:\n `tuple` of matching paths.\n Example:\n ```pycon\n >>> bl_lwm = getfixture(\"bl_lwm\")\n >>> pprint(path_globs_to_tuple(bl_lwm, '*text.zip'))\n (...Path('...bl_lwm...0003079-test_plaintext.zip'),\n ...Path('...bl_lwm...0003548-test_plaintext.zip'))\n >>> pprint(path_globs_to_tuple(bl_lwm, '*.txt'))\n (...Path('...bl_lwm...0003079_18980121_sect0001.txt'),\n ...Path('...bl_lwm...0003548_19040707_art0037.txt'))\n ```\n \"\"\"\nreturn tuple(sorted(Path(path).glob(glob_regex_str)))\npaths_with_newlines(\npaths: Iterable[PathLike], truncate: bool = False, **kwargs: bool\n) -> str\nReturn a str of paths separated by a \\n.
>>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n>>> print(paths_with_newlines(plaintext_bl_lwm.compressed_files))\n'...bl_lwm...0003079-test_plaintext.zip'\n'...bl_lwm...0003548-test_plaintext.zip'\n>>> print(\n... paths_with_newlines(plaintext_bl_lwm.compressed_files,\n... truncate=True)\n... )\n<BLANKLINE>\n...\n'...0003079-test_plaintext.zip'\n'...0003548-test_plaintext.zip'\nalto2txt2fixture/utils.py def paths_with_newlines(\npaths: Iterable[PathLike], truncate: bool = False, **kwargs\n) -> str:\n\"\"\"Return a `str` of `paths` separated by a `\\\\n`.\n Example:\n ```pycon\n >>> plaintext_bl_lwm = getfixture('bl_lwm_plaintext')\n >>> print(paths_with_newlines(plaintext_bl_lwm.compressed_files))\n '...bl_lwm...0003079-test_plaintext.zip'\n '...bl_lwm...0003548-test_plaintext.zip'\n >>> print(\n ... paths_with_newlines(plaintext_bl_lwm.compressed_files,\n ... truncate=True)\n ... )\n <BLANKLINE>\n ...\n '...0003079-test_plaintext.zip'\n '...0003548-test_plaintext.zip'\n ```\n \"\"\"\nif truncate:\nreturn \"\\n\".join(f\"'{truncate_path_str(f, **kwargs)}'\" for f in paths)\nelse:\nreturn \"\\n\".join(f\"'{f}'\" for f in paths)\nrename_by_0_padding(\nfile_path: PathLike,\nmatch_str: str | None = None,\nmatch_int: int | None = None,\npadding: int = FILE_NAME_0_PADDING_DEFAULT,\nreplace_count: int = 1,\nexclude_parents: bool = True,\nreverse_int_match: bool = False,\n) -> Path\nReturn file_path with 0 padding Path change.
Parameters:
Name Type Description Defaultfile_path PathLike PathLike to rename.
match_str str | None str to match and replace with padded match_int
None match_int int | None int to pad and replace match_str
None padding int How many digits (0s) to pad match_int with.
FILE_NAME_0_PADDING_DEFAULT exclude_parents bool Only rename parts of Path(file_path).name; else replace across Path(file_path).parents as well.
True reverse_int_match bool Whether to match from the end of the file_path.
False Example >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_str='05', match_int=5)\n<BLANKLINE>\n...Path('a/path/to/3/fixture-03-000005.txt')...\n>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_str='03')\n<BLANKLINE>\n...Path('a/path/to/3/fixture-000003-05.txt')...\n>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_str='05', padding=0)\n<BLANKLINE>\n...Path('a/path/to/3/fixture-03-5.txt')...\n>>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n... match_int=3)\n<BLANKLINE>\n...Path('a/path/to/3/fixture-0000003-05.txt')...\n>>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n... match_int=3, padding=2,\n... exclude_parents=False)\n<BLANKLINE>\n...Path('a/path/to/03/f-03-05-0003.txt')...\n>>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n... match_int=3, padding=2,\n... exclude_parents=False,\n... replace_count=3, )\n<BLANKLINE>\n...Path('a/path/to/03/f-003-05-00003.txt')...\nalto2txt2fixture/utils.py def rename_by_0_padding(\nfile_path: PathLike,\nmatch_str: str | None = None,\nmatch_int: int | None = None,\npadding: int = FILE_NAME_0_PADDING_DEFAULT,\nreplace_count: int = 1,\nexclude_parents: bool = True,\nreverse_int_match: bool = False,\n) -> Path:\n\"\"\"Return `file_path` with `0` `padding` `Path` change.\n Params:\n file_path:\n `PathLike` to rename.\n match_str:\n `str` to match and replace with padded `match_int`\n match_int:\n `int` to pad and replace `match_str`\n padding:\n How many digits (0s) to pad `match_int` with.\n exclude_parents:\n Only rename parts of `Path(file_path).name`; else\n replace across `Path(file_path).parents` as well.\n reverse_int_match:\n Whether to match from the end of the `file_path`.\n Example:\n ```pycon\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_str='05', match_int=5)\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-03-000005.txt')...\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_str='03')\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-000003-05.txt')...\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_str='05', padding=0)\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-03-5.txt')...\n >>> rename_by_0_padding('a/path/to/3/fixture-03-05.txt',\n ... match_int=3)\n <BLANKLINE>\n ...Path('a/path/to/3/fixture-0000003-05.txt')...\n >>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n ... match_int=3, padding=2,\n ... exclude_parents=False)\n <BLANKLINE>\n ...Path('a/path/to/03/f-03-05-0003.txt')...\n >>> rename_by_0_padding('a/path/to/3/f-03-05-0003.txt',\n ... match_int=3, padding=2,\n ... exclude_parents=False,\n ... replace_count=3, )\n <BLANKLINE>\n ...Path('a/path/to/03/f-003-05-00003.txt')...\n ```\n \"\"\"\nif match_int is None and match_str in (None, \"\"):\nraise ValueError(f\"At least `match_int` or `match_str` required; both None.\")\nelif match_str and not match_int:\nmatch_int = int(match_str)\nelif match_int is not None and not match_str:\nassert str(match_int) in str(file_path)\nmatch_str = int_from_str(\nstr(file_path),\nindex=-1 if reverse_int_match else 0,\n)[0]\nassert match_int is not None and match_str is not None\nif exclude_parents:\nreturn Path(file_path).parent / Path(file_path).name.replace(\nmatch_str, str(match_int).zfill(padding), replace_count\n)\nelse:\nreturn Path(\nstr(file_path).replace(\nmatch_str, str(match_int).zfill(padding), replace_count\n)\n)\nsave_fixture(\ngenerator: Sequence | Generator = [],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nadd_created: bool = True,\njson_indent: int = JSON_INDENT,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None\nSaves fixtures generated by a generator to separate JSON files.
This function takes a generator and saves the generated fixtures to separate JSON files. The fixtures are saved in batches, where each batch is determined by the max_elements_per_file parameter.
Parameters:
Name Type Description Defaultgenerator Sequence | Generator A generator that yields the fixtures to be saved.
[] prefix str A string prefix to be added to the file names of the saved fixtures.
'' output_path PathLike | str Path to folder fixtures are saved to
settings.OUTPUT max_elements_per_file int Maximum JSON records saved in each file
settings.MAX_ELEMENTS_PER_FILE add_created bool Whether to add created_at and updated_at timestamps
True json_indent int Number of indent spaces per line in saved JSON
JSON_INDENT file_name_0_padding int Zeros to prefix the number of each fixture file name.
FILE_NAME_0_PADDING_DEFAULT Returns:
Type DescriptionNone This function saves the fixtures to files but does not return
None any value.
Example>>> tmp_path: Path = getfixture('tmp_path')\n>>> save_fixture(NEWSPAPER_COLLECTION_METADATA,\n... prefix='test', output_path=tmp_path)\n>>> imported_fixture = load_json(tmp_path / 'test-000001.json')\n>>> imported_fixture[1]['pk']\n2\n>>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n'hmd'\n>>> 'created_at' in imported_fixture[1]['fields']\nTrue\nalto2txt2fixture/utils.py def save_fixture(\ngenerator: Sequence | Generator = [],\nprefix: str = \"\",\noutput_path: PathLike | str = settings.OUTPUT,\nmax_elements_per_file: int = settings.MAX_ELEMENTS_PER_FILE,\nadd_created: bool = True,\njson_indent: int = JSON_INDENT,\nfile_name_0_padding: int = FILE_NAME_0_PADDING_DEFAULT,\n) -> None:\n\"\"\"Saves fixtures generated by a generator to separate JSON files.\n This function takes a generator and saves the generated fixtures to\n separate JSON files. The fixtures are saved in batches, where each batch\n is determined by the ``max_elements_per_file`` parameter.\n Args:\n generator:\n A generator that yields the fixtures to be saved.\n prefix:\n A string prefix to be added to the file names of the\n saved fixtures.\n output_path:\n Path to folder fixtures are saved to\n max_elements_per_file:\n Maximum `JSON` records saved in each file\n add_created:\n Whether to add `created_at` and `updated_at` `timestamps`\n json_indent:\n Number of indent spaces per line in saved `JSON`\n file_name_0_padding:\n Zeros to prefix the number of each fixture file name.\n Returns:\n This function saves the fixtures to files but does not return\n any value.\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> save_fixture(NEWSPAPER_COLLECTION_METADATA,\n ... prefix='test', output_path=tmp_path)\n >>> imported_fixture = load_json(tmp_path / 'test-000001.json')\n >>> imported_fixture[1]['pk']\n 2\n >>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n >>> 'created_at' in imported_fixture[1]['fields']\n True\n ```\n \"\"\"\ninternal_counter = 1\ncounter = 1\nlst = []\nfile_name: str\nPath(output_path).mkdir(parents=True, exist_ok=True)\nfor item in generator:\nlst.append(item)\ninternal_counter += 1\nif internal_counter > max_elements_per_file:\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.json\"\nwrite_json(\np=Path(f\"{output_path}/file_name\"),\no=lst,\nadd_created=add_created,\njson_indent=json_indent,\n)\n# Save up some memory\ndel lst\ngc.collect()\n# Re-instantiate\nlst = []\ninternal_counter = 1\ncounter += 1\nelse:\nfile_name = f\"{prefix}-{str(counter).zfill(file_name_0_padding)}.json\"\nwrite_json(\np=Path(f\"{output_path}/{file_name}\"),\no=lst,\nadd_created=add_created,\njson_indent=json_indent,\n)\nreturn\ntruncate_path_str(\npath: PathLike,\nmax_length: int = MAX_TRUNCATE_PATH_STR_LEN,\nfolder_filler_str: str = INTERMEDIATE_PATH_TRUNCATION_STR,\nhead_parts: int = TRUNC_HEADS_PATH_DEFAULT,\ntail_parts: int = TRUNC_TAILS_PATH_DEFAULT,\npath_sep: str = sep,\n_force_type: Type[Path] | Type[PureWindowsPath] = Path,\n) -> str\nIf len(text) > max_length return text followed by trail_str.
Parameters:
Name Type Description Defaultpath PathLike PathLike object to truncate
max_length int maximum length of path to allow, anything belond truncated
MAX_TRUNCATE_PATH_STR_LEN folder_filler_str str what to fill intermediate path names with
INTERMEDIATE_PATH_TRUNCATION_STR head_parts int how many parts of path from the root to keep. These must be int >= 0
TRUNC_HEADS_PATH_DEFAULT tail_parts int how many parts from the path tail the root to keep. These must be int >= 0
TRUNC_TAILS_PATH_DEFAULT path_sep str what str to replace path parts with if over max_length
sep Returns:
Type Descriptionstr text truncated to max_length (if longer than max_length), with with folder_filler_str for intermediate folder names
For errors running on windows see: #56
Example>>> love_shadows: Path = (\n... Path('Standing') / 'in' / 'the' / 'shadows'/ 'of' / 'love.')\n>>> truncate_path_str(love_shadows)\n'Standing...love.'\n>>> truncate_path_str(love_shadows, max_length=100)\n'Standing...in...the...shadows...of...love.'\n>>> truncate_path_str(love_shadows, folder_filler_str=\"*\")\n'Standing...*...*...*...*...love.'\n>>> root_love_shadows: Path = Path(sep) / love_shadows\n>>> truncate_path_str(root_love_shadows, folder_filler_str=\"*\")\n<BLANKLINE>\n...\n'...Standing...*...*...*...*...love.'\n>>> if is_platform_win:\n... pytest.skip('fails on certain Windows root paths: issue #56')\n>>> truncate_path_str(root_love_shadows,\n... folder_filler_str=\"*\", tail_parts=3)\n<BLANKLINE>\n...\n'...Standing...*...*...shadows...of...love.'...\nalto2txt2fixture/utils.py def truncate_path_str(\npath: PathLike,\nmax_length: int = MAX_TRUNCATE_PATH_STR_LEN,\nfolder_filler_str: str = INTERMEDIATE_PATH_TRUNCATION_STR,\nhead_parts: int = TRUNC_HEADS_PATH_DEFAULT,\ntail_parts: int = TRUNC_TAILS_PATH_DEFAULT,\npath_sep: str = sep,\n_force_type: Type[Path] | Type[PureWindowsPath] = Path,\n) -> str:\n\"\"\"If `len(text) > max_length` return `text` followed by `trail_str`.\n Args:\n path:\n `PathLike` object to truncate\n max_length:\n maximum length of `path` to allow, anything belond truncated\n folder_filler_str:\n what to fill intermediate path names with\n head_parts:\n how many parts of `path` from the root to keep.\n These must be `int` >= 0\n tail_parts:\n how many parts from the `path` tail the root to keep.\n These must be `int` >= 0\n path_sep:\n what `str` to replace `path` parts with if over `max_length`\n Returns:\n `text` truncated to `max_length` (if longer than `max_length`),\n with with `folder_filler_str` for intermediate folder names\n Note:\n For errors running on windows see:\n [#56](https://github.com/Living-with-machines/alto2txt2fixture/issues/56)\n Example:\n ```pycon\n >>> love_shadows: Path = (\n ... Path('Standing') / 'in' / 'the' / 'shadows'/ 'of' / 'love.')\n >>> truncate_path_str(love_shadows)\n 'Standing...love.'\n >>> truncate_path_str(love_shadows, max_length=100)\n 'Standing...in...the...shadows...of...love.'\n >>> truncate_path_str(love_shadows, folder_filler_str=\"*\")\n 'Standing...*...*...*...*...love.'\n >>> root_love_shadows: Path = Path(sep) / love_shadows\n >>> truncate_path_str(root_love_shadows, folder_filler_str=\"*\")\n <BLANKLINE>\n ...\n '...Standing...*...*...*...*...love.'\n >>> if is_platform_win:\n ... pytest.skip('fails on certain Windows root paths: issue #56')\n >>> truncate_path_str(root_love_shadows,\n ... folder_filler_str=\"*\", tail_parts=3)\n <BLANKLINE>\n ...\n '...Standing...*...*...shadows...of...love.'...\n ```\n \"\"\"\npath = _force_type(normpath(path))\nif len(str(path)) > max_length:\ntry:\nassert not (head_parts < 0 or tail_parts < 0)\nexcept AssertionError:\nlogger.error(\nf\"Both index params for `truncate_path_str` must be >=0: \"\nf\"(head_parts={head_parts}, tail_parts={tail_parts})\"\n)\nreturn str(path)\noriginal_path_parts: tuple[str, ...] = path.parts\nhead_index_fix: int = 0\nif path.is_absolute() or path.drive:\nhead_index_fix += 1\nfor part in original_path_parts[head_parts + head_index_fix :]:\nif not part:\nhead_index_fix += 1\nelse:\nbreak\nlogger.debug(\nf\"Adding {head_index_fix} to `head_parts`: {head_parts} \"\nf\"to truncate: '{path}'\"\n)\nhead_parts += head_index_fix\ntry:\nassert head_parts + tail_parts < len(str(original_path_parts))\nexcept AssertionError:\nlogger.error(\nf\"Returning untruncated. Params \"\nf\"(head_parts={head_parts}, tail_parts={tail_parts}) \"\nf\"not valid to truncate: '{path}'\"\n)\nreturn str(path)\ntail_index: int = len(original_path_parts) - tail_parts\nreplaced_path_parts: tuple[str, ...] = tuple(\npart if (i < head_parts or i >= tail_index) else folder_filler_str\nfor i, part in enumerate(original_path_parts)\n)\nreplaced_start_str: str = \"\".join(replaced_path_parts[:head_parts])\nreplaced_end_str: str = path_sep.join(\npath for path in replaced_path_parts[head_parts:]\n)\nreturn path_sep.join((replaced_start_str, replaced_end_str))\nelse:\nreturn str(path)\nvalid_compression_files(files: Sequence[PathLike]) -> list[PathLike]\nReturn a tuple of valid compression paths in files.
Parameters:
Name Type Description Defaultfiles Sequence[PathLike] Sequence of files to filter compression types from.
Returns:
Type Descriptionlist[PathLike] A list of files that could be decompressed.
Example>>> valid_compression_files([\n... 'cat.tar.bz2', 'dog.tar.bz3', 'fish.tgz', 'bird.zip',\n... 'giraffe.txt', 'frog'\n... ])\n['cat.tar.bz2', 'fish.tgz', 'bird.zip']\nalto2txt2fixture/utils.py def valid_compression_files(files: Sequence[PathLike]) -> list[PathLike]:\n\"\"\"Return a `tuple` of valid compression paths in `files`.\n Args:\n files:\n `Sequence` of files to filter compression types from.\n Returns:\n A list of files that could be decompressed.\n Example:\n ```pycon\n >>> valid_compression_files([\n ... 'cat.tar.bz2', 'dog.tar.bz3', 'fish.tgz', 'bird.zip',\n ... 'giraffe.txt', 'frog'\n ... ])\n ['cat.tar.bz2', 'fish.tgz', 'bird.zip']\n ```\n \"\"\"\nreturn [\nfile\nfor file in files\nif \"\".join(Path(file).suffixes) in VALID_COMPRESSION_FORMATS\n]\nwrite_json(\np: str | Path,\no: dict,\nadd_created: bool = True,\njson_indent: int = JSON_INDENT,\n) -> None\nEasier access to writing json files. Checks whether parent exists.
Parameters:
Name Type Description Defaultp str | Path Path to write json to
o dict Object to write to json file
add_created bool If set to True will add created_at and updated_at to the dictionary's fields. If created_at and updated_at already exist in the fields, they will be forcefully updated.
True json_indent int What indetation format to write out JSON file in
JSON_INDENT Returns:
Type DescriptionNone None
Example>>> tmp_path: Path = getfixture('tmp_path')\n>>> path: Path = tmp_path / 'test-write-json-example.json'\n>>> write_json(p=path,\n... o=NEWSPAPER_COLLECTION_METADATA,\n... add_created=True)\n>>> imported_fixture = load_json(path)\n>>> imported_fixture[1]['pk']\n2\n>>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n'hmd'\nalto2txt2fixture/utils.py def write_json(\np: str | Path, o: dict, add_created: bool = True, json_indent: int = JSON_INDENT\n) -> None:\n\"\"\"\n Easier access to writing `json` files. Checks whether parent exists.\n Args:\n p: Path to write `json` to\n o: Object to write to `json` file\n add_created:\n If set to True will add `created_at` and `updated_at`\n to the dictionary's fields. If `created_at` and `updated_at`\n already exist in the fields, they will be forcefully updated.\n json_indent:\n What indetation format to write out `JSON` file in\n Returns:\n None\n Example:\n ```pycon\n >>> tmp_path: Path = getfixture('tmp_path')\n >>> path: Path = tmp_path / 'test-write-json-example.json'\n >>> write_json(p=path,\n ... o=NEWSPAPER_COLLECTION_METADATA,\n ... add_created=True)\n >>> imported_fixture = load_json(path)\n >>> imported_fixture[1]['pk']\n 2\n >>> imported_fixture[1]['fields'][DATA_PROVIDER_INDEX]\n 'hmd'\n ```\n `\n \"\"\"\np = get_path_from(p)\nif not (isinstance(o, dict) or isinstance(o, list)):\nraise RuntimeError(f\"Unable to handle data of type: {type(o)}\")\ndef _append_created_fields(o: dict):\n\"\"\"Add `created_at` and `updated_at` fields to a `dict` with `FixtureDict` values.\"\"\"\nreturn dict(\n**{k: v for k, v in o.items() if not k == \"fields\"},\nfields=dict(\n**{\nk: v\nfor k, v in o[\"fields\"].items()\nif not k == \"created_at\" and not k == \"updated_at\"\n},\n**{\"created_at\": NOW_str, \"updated_at\": NOW_str},\n),\n)\ntry:\nif add_created and isinstance(o, dict):\no = _append_created_fields(o)\nelif add_created and isinstance(o, list):\no = [_append_created_fields(x) for x in o]\nexcept KeyError:\nerror(\"An unknown error occurred (in write_json)\")\np.parent.mkdir(parents=True, exist_ok=True)\np.write_text(json.dumps(o, indent=json_indent))\nreturn\nThe installation process should be fairly easy to take care of, using poetry:
$ poetry install\nHowever, this is only the first step in the process. As the script works through the alto2txt collections, you will either need to choose the slower option \u2014 mounting them to your computer (using blobfuse) \u2014\u00a0or the faster option \u2014 downloading the required zip files from the Azure storage to your local hard drive. In the two following sections, both of those options are described.
alto2txt to the program","text":""},{"location":"tutorial/first-steps.html#downloading-local-copies-of-alto2txt-on-your-computer","title":"Downloading local copies of alto2txt on your computer","text":"This option will take up a lot of hard drive space
As of the time of writing, downloading all of alto2txt\u2019s metadata takes up about 185GB on your local drive.
You do not have to download all of the collections or all of the zip files for each collection, as long as you are aware that the resulting fixtures will be limited in scope.
"},{"location":"tutorial/first-steps.html#step-1-log-in-to-azure-using-microsoft-azure-storage-explorer","title":"Step 1: Log in to Azure using Microsoft Azure Storage Explorer","text":"Microsoft Azure Storage Explorer (MASE) is a great and free tool for downloading content off Azure. Your first step is to download and install this product on your local computer.
Once you have opened MASE, you will need to sign into the appropriate Azure account.
"},{"location":"tutorial/first-steps.html#step-2-download-the-alto2txt-blob-container-to-your-hard-drive","title":"Step 2: Download thealto2txt blob container to your hard drive","text":"On your left-hand side, you should see a menu where you can navigate to the correct \u201cblob container\u201d: Living with Machines > Storage Accounts > alto2txt > Blob Containers:
You will want to replicate the same structure as the Blob Container itself in a folder on your hard drive:
Once you have the structure set up, you are ready to download all of the files needed. For each of the blob containers, make sure that you download the metadata directory only onto your computer:
Select all of the files and press the download button:
Make sure you save all the zip files inside the correct local folder:
The \u201cActivities\u201d bar will now show you the progress and speed:
"},{"location":"tutorial/first-steps.html#mounting-alto2txt-on-your-computer","title":"Mountingalto2txt on your computer","text":"This option will only work on a Linux or UNIX computer
If you have a mac, your only option is the one below.
"},{"location":"tutorial/first-steps.html#step-1-install-blobfuse","title":"Step 1: Install BlobFuse","text":"Follow the instructions for installing BlobFuse and the instructions for how to prepare your drive for mounting.
"},{"location":"tutorial/first-steps.html#step-2-set-up-sas-tokens","title":"Step 2: Set up SAS tokens","text":"Follow the instructions for setting up access to your Azure storage account.
"},{"location":"tutorial/first-steps.html#step-3-mount-your-blobs","title":"Step 3: Mount your blobs","text":"TODO #3: Write this section.
Note that you can also search on the internet for ideas on how to create local scripts to facilitate easier connection next time.
"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml new file mode 100644 index 0000000..0283b42 --- /dev/null +++ b/sitemap.xml @@ -0,0 +1,88 @@ + +The installation process should be fairly easy to take care of, using poetry:
However, this is only the first step in the process. As the script works through the alto2txt collections, you will either need to choose the slower option — mounting them to your computer (using blobfuse) — or the faster option — downloading the required zip files from the Azure storage to your local hard drive. In the two following sections, both of those options are described.
alto2txt to the programalto2txt on your computerThis option will take up a lot of hard drive space
+As of the time of writing, downloading all of alto2txt’s metadata takes up about 185GB on your local drive.
You do not have to download all of the collections or all of the zip files for each collection, as long as you are aware that the resulting fixtures will be limited in scope.
+Microsoft Azure Storage Explorer (MASE) is a great and free tool for downloading content off Azure. Your first step is to download and install this product on your local computer.
+Once you have opened MASE, you will need to sign into the appropriate Azure account.
+alto2txt blob container to your hard driveOn your left-hand side, you should see a menu where you can navigate to the correct “blob container”: Living with Machines > Storage Accounts > alto2txt > Blob Containers:
You will want to replicate the same structure as the Blob Container itself in a folder on your hard drive:
+
Once you have the structure set up, you are ready to download all of the files needed. For each of the blob containers, make sure that you download the metadata directory only onto your computer:
Select all of the files and press the download button:
+
Make sure you save all the zip files inside the correct local folder:
+
The “Activities” bar will now show you the progress and speed:
+
alto2txt on your computerThis option will only work on a Linux or UNIX computer
+If you have a mac, your only option is the one below.
+Follow the instructions for installing BlobFuse and the instructions for how to prepare your drive for mounting.
+Follow the instructions for setting up access to your Azure storage account.
+TODO #3: Write this section.
+Note that you can also search on the internet for ideas on how to create local scripts to facilitate easier connection next time.
+ + + + + + +The examples below follow standard settings
+If you choose other settings for when you run the program, your output directory may look different from the information on this page.
+Reports are automatically generated with a unique hash as the overarching folder structure. Inside the reports directory, you’ll find a JSON file for each alto2txt directory (organised by NLP identifier).
The report structure, thus, looks like this:
+
The JSON file has some good troubleshooting information. You’ll find that the contents are structured as a Python dictionary (or JavaScript Object). Here is an example:
Here is an explanation of each of the keys in the dictionary:
+| Key | +Explanation | +Data type | +
|---|---|---|
path |
+The input path for the zip file that is being converted. | +string |
+
bytes |
+The size of the input zip file represented in bytes. | +integer |
+
size |
+The size of the input zip file represented in a human-readable string. | +string |
+
contents |
+#TODO #3 | +integer |
+
start |
+Date and time when processing started (see also end below). |
+datestring |
+
newspaper_paths |
+#TODO #3 | +list (string) |
+
publication_codes |
+A list of the NLPs that are contained in the input zip file. | +list (string) |
+
issue_paths |
+A list of all the issue paths that are contained in the cache directory. | +list (string) |
+
item_paths |
+A list of all the item paths that are contained in the cache directory. | +list (string) |
+
end |
+Date and time when processing ended (see also start above). |
+datestring |
+
seconds |
+Seconds that the script spent interpreting the zip file (should be added to the microseconds below). |
+integer |
+
microseconds |
+Microseconds that the script spent interpreting the zip file (should be added to the seconds above). |
+integer |
+
The most important output of the script is contained in the fixtures directory. This directory contains JSON files for all the different columns in the corresponding Django metadata database (i.e. DataProvider, Digitisation, Ingest, Issue, Newspaper, and Item). The numbering at the end of each file indicates the order of the files as they are divided into a maximum of 2e6 elements*:
Each JSON file contains a Python-like list (JavaScript Array) of dictionaries (JavaScript Objects), which have a primary key (pk), the related database model (in the example below the Django newspapers app’s newspaper table), and a nested dictionary/Object which contains all the values for the database’s table entry:
* The maximum elements per file can be adjusted in the settings.py file’s settings object’s MAX_ELEMENTS_PER_FILE value.