Stability improvements #24
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Improved error handling. Separate parallel tasks for File watch and processing files Checks for File reads likes valid directory path, file format etc Pre-condition checks before commit/abort the transactions. Added more logs for debugging. Signed-off-by: Shwetha N <[email protected]>
Support ingestion from multiple directories
Refactor DB interaction. Signed-off-by: Shwetha N <[email protected]>
File Ingestion and Processing refactor Signed-off-by: Shwetha N <[email protected]>
Fix logging in case of invalid files In case of invalid files, the failed files will be moved to a separate Failed_Files folder next to the database. This ensures the warning message for invalid files is shown only once.
Fix failing unit tests by fixing parameters.
… latest to reduce vulnerability issues (#22) Add spotbug and checkstyle support, updated gradle and some jars to latest to reduce vulnerability issues.
Updates the code to extract the parent directory for Failed_Files folder. Path.getParent() method is used to make it platform independent.
…before processing (#28) Delete filename from pending table if actual file doesn't exist just before processing. Signed-off-by: Kuldeep Kumar <[email protected]>
Updated configuration properties for all file types(CSV, Parquet and Raw) Updated wiki for configuration parameters, description and default values. Signed-off-by: Shwetha N <[email protected]>
Optimize CsvFileIngestService and RawFileIngestService. CSV Events are now batched in a list and entire batch is converted to json. Each record is parsed into a hashmap before being added to the batch. For Raw file events, BufferedInputStream is used now Also the service class names in test scripts are fixed and new script is added for testing csv files.
PR updates the APP_VERSION for pravega-sensor-collector builds to 0.2.18. The build-installer.sh script is also modified to publish an executable jar.
Move completed files to a different directory
Introduces new unit tests for the Pravega sensor collector to ensure robust and reliable data collection from sensors. The tests cover various scenarios, including normal operation, edge cases, and potential error conditions. Signed-off-by: Shwetha N <[email protected]>
…indows (#34) Add garbage collection of completed files by moving it into completed_files directory
shshashwat
reviewed
Dec 14, 2023
|
|
||
| 2. Modify [PravegaSensorCollectorApp.xml](windows-service/PravegaSensorCollectorApp.xml). Check PRAVEGA_SENSOR_COLLECTOR_RAW1_PRAVEGA_CONTROLLER_URI. | ||
| 4. Add the path to the jar file `pravega-sensor-collector/build/libs/pravega-sensor-collector-${APP_VERSION}.jar` generated by build-installer.sh in PravegaSensorCollectorApp.xml. |
dada-dell-emc
approved these changes
Jan 5, 2024
abhinb
approved these changes
Jan 5, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cumulative Merge of stability improvements
All tests should pass