Not perfect yet. Still some tiny issues missing and some re-reconciling/correcting UI needed. Should be ready tonight? Left on this pull still a few DPM()s for my own enjoyment.

@alliomeria gosh. I made it i think?
OF course the UI can get some better things and we need to add a LOT of options to avoid "overwriting" all your lovely Manually LoD. But gosh. This is so good!!

And our CSV headers are not (lowercase/normalized) And gosh. Well. But this seems to work. Last commit for tonight.

@alliomeria i will need you to test this when you can. This is finally working as expected. For DCMNY this gives us initially 714 headers! but then the cleanup reduces things significantly to lower numbers (e.g for gsmt:icc to 87 or so)
We get more data of course now!! (because of children really providing their correct data when not collapsing).

Uff. Was super hard to find my own error but the gist for my own future me:

Batch driven Plugins like the SolrImport one provide the headers via their ::getInfo method but because we do not really know until the batch ends how many of the headers will really survive the cleanup we do not set them into the actual Config (config that is stored in each AMI set) which really means we have to manually pass them to the batch!

Good, so good

…ering

On Books with pages

@alliomeria this should fix the double encoding issue, but
just to be sure, could you try Solr import 7 with a collection that has complex weird characters in its descriptions/titles (maybe NYHS?)
and also with a google sheet using, e.g Japanese + "" and & and ' somewher?

WE can test tomorrow together. Thing is i removed a lot of code that i "for some reason" had added, so wonder if i will not break now something somewhere else, and thus "testing" is required!

I was resetting the array to give this a default value. happens that i needed the first key, not the first value and now that our "labels" are Capitalized and not == to the value (e.g Direct v/s direct) i was breaking the Form State. What a mess!

- Can not handle CSV rows that are either indexed arrays (default) or associative ones.
- ::createAmiSet() can be passed a Name (title/label) via the \stdClass $data using $data->name. used for now on the CSV export but as easy-peasy as it gets to be added to the AMI normal Multi Step one (next pull?)

And mostly the logic generated by \Drupal\views_bulk_operations\Form\ConfigureAction::buildForm where form_state does not survive the ajax triggered dynamic fields that have internally a submit (e.g OpenStreetmaps). So will mark as experimental until i find a strange/dark hack to get this rolling. webform elements, gosh

Quite smart may i say! So much and so little code really. Test it, let me know how it goes

- This adds new methods for offsetting and paging CSV data (was a pain because JSON pretty printed has breaklines). Since i was here i also added options for getting headers (or not) for offset and paged CSV access
- Adds an actual pager to the Edit Reconciling page
- Nothing of this is Done-done. I need to now change the "saving" algorithm to allow partial / offset saves to happen. Probably using the already created KeyValue for the set instead of reading the actual Form data as we did with large full sets and also need the "this was fixed" checkbox to be added to a new Column

More tomorrow @alliomeria !

@alliomeria paging is now working again.
I was doing a wrong "check if this needs a header row" thing and ending
without a header row which the Reconcile form needed. Good!

@patdunlavey not the last pull of this branch (LOD had to be postponed for tonight) but direct should work now. Please give it a spin? (i will do some extra double checking of data after things, but should "mostly" workout.

Allison says..

After dealing with Direct Ingest i broke Twig templates.... @patdunlavey ... you will see this fail... should back again to its normal behavior.

It was getting weird to have such cross dependencies. Now both deal with their own concerns and there is complementation. This changes a lot

Also:
- Json Decoding Cells now deals with Smart Quote exceptions and tries to be as smart as possible in case of errors. But if the key is one of our core ones ap: or as: then in case of failure we reset to NULL value. This allows us to avoid breaking code in case one our controlled values via a Twig template OR a Direct ingest ends with wrong data.

@todo. Implement a JSON-SCHEMA level validation for our own internal control vocabulary (WIP!)

… numbers

@alliomeria when you come back. Solr exact fetch is fixed, also fixes the real time report to the number of rows requested. Tested on collapsed/uncollapsed and with offsets to check if a set of consequent harvests with  offsets was correct and not missing anything. All checks out!

…the options

This is basically the same as setting type = "node" but allows us in the future to also allow certain Actions to run on e.g AMI Sets

@alliomeria how to test?

- git pull this branch
- Clear Caches
- Edit your Search and replace View. Remove the "Global: Views bulk operations" field (instead of installing/reinstalling the module, this seems to be the best option a.k.a as TRICK to make actions show again)
- Add it again. The list should be refreshed. Check the ones you want to use
- Save the view
- test!

Happens that i uncommented a line i had commented because i used to be smart. Not anymore. If you add a #name property to an AJAX driven form with select, the original value gets cached for ever and never changes even if you submit it differently. How do i know this? Just because i tested it. There is NO documentation. Wonder if that is also the issue with Webform based find and replace??

Internally our CSV to data structure is already normalized for lower case (we need to document this better). So for Solr import, also inmediatelly set the column names to lower case and in general enforce checking against lowercases when doing validation. Does not require any change really on Twig templates because for Twig "data.HoLa" is the same as "data.hola". Just saying @alliomeria

all ZIP files will go eventually  (once the AMI set entity is saved or when uploaded directly via "Edit" of a Set, into private://ami/zip

Since we can not predict the final extension of a download upfront without download i check now using gob() for the first part of the possible filename.
If there is already a single (only a single one) File with the same starting future name (which is consistent since we use the URL of the file to generate it) then we reuse the already downloaded one. Will add also a button that deletes all files for an AMI set in the future and one that forces the download even if there.

Also this piece of code ensures ZIP files (After first upload) are moved into private://ami/zip once the AMI set is saved

And first pass works, like a charm!
Ok, need to add
1.- Config entry to set "what is many files" and what not.
2.- Better reuse of as:technical metadata, so we do not reprocess that (file re use after download is already in place).

Probably the best thing ever here. @aksm @alliomeria

@alliomeria can you test with the same CSV with 387 objects?
And please twice? First time may be slow-ish, second time quite fast

also reprocessing of cached TECHMD, etc

@dmer @patdunlavey can you check this code please (as you know release process so as soon as you can). I wonder if you removed moderation workflows which enables by default revisions or if you have not set revisions at the bundle level?) But this will force that.

Works. Looks good. I'm good. A good person right @alliomeria and @aksm ?
Revisit CSV offsets once we are on PHP 8.1 I found a PHP BUG!!

…hing

Need to check now CSV write!

I have been willing to write the first PHP remote streamer library from forever and i never have the time. So for now, for the people that decide to put ALLL on S3 we have to download the file.. no other option.
So we inject the StrawberryfieldFileMetadataService to reuse the ::ensureFileAvailability function.
But we can not delete the file here since we may need it for the rest of our times.

But.. VBO actions is not respecting the Facet selection when using the "Select all" option... We should make sure that is clear? Documentation? Eventually patch VBO or, alter the form and remove Select All completely?