Skip to content

Commit

Permalink
Merge branch 'master' into feature/call-number-browse-v2
Browse files Browse the repository at this point in the history
  • Loading branch information
SvitlanaKovalova1 committed Jan 6, 2025
2 parents 31e6e18 + f32d6af commit c6dc1cb
Show file tree
Hide file tree
Showing 64 changed files with 1,141 additions and 897 deletions.
4 changes: 4 additions & 0 deletions NEWS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@

### Features
* Move Instance sub-entities population from database trigger to code ([MSEARCH-887](https://folio-org.atlassian.net/browse/MSEARCH-887))
* Update reindex merge failed status only for failed entity type ([MSEARCH-909](https://folio-org.atlassian.net/browse/MSEARCH-909))
* Extend reindex range tables with status, fail_cause columns ([MSEARCH-870](https://folio-org.atlassian.net/browse/MSEARCH-870))
* Implement scheduled indexing for instance sub-resources ([MSEARCH-922](https://folio-org.atlassian.net/browse/MSEARCH-922))
* Call Numbers Browse: Implement Database Structure and Logic for Managing Call Numbers ([MSEARCH-862](https://folio-org.atlassian.net/browse/MSEARCH-862))
* Call Numbers Browse: Implement Call Number Browse Config ([MSEARCH-863](https://folio-org.atlassian.net/browse/MSEARCH-863))
* Call Numbers Browse: Implement Indexing and Re-indexing Mechanisms for Call-Numbers ([MSEARCH-864](https://folio-org.atlassian.net/browse/MSEARCH-864))
Expand All @@ -21,6 +24,7 @@
* Fix old browse config returned on get after upsert ([MSEARCH-897](https://folio-org.atlassian.net/browse/MSEARCH-897))
* Fix generation of IDs ranges in Reindex Upload for Subject, Classification and Contributor ([MSEARCH-907](https://folio-org.atlassian.net/browse/MSEARCH-907))
* Remove browse config caching ([MSEARCH-897](https://folio-org.atlassian.net/browse/MSEARCH-897))
* Fix the "Invalid reference" appears after updating ownership ([MSEARCH-915](https://folio-org.atlassian.net/browse/MSEARCH-915))

### Tech Dept
* Description ([ISSUE](https://folio-org.atlassian.net/browse/ISSUE))
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ and [Cross-cluster replication](https://docs.aws.amazon.com/opensearch-service/l
| REINDEX_MERGE_RANGE_PUBLISHER_RETRY_INTERVAL_MS | 1000 | The retry interval in ms for reindex merge range request publishing. |
| REINDEX_MERGE_RANGE_PUBLISHER_RETRY_ATTEMPTS | 5 | The maximum number of retries for reindex merge range request publishing. |
| MAX_SEARCH_BATCH_REQUEST_IDS_COUNT | 20_000 | Defines maximum batch request IDs count for searching consolidated items/holdings in consortium |
| INSTANCE_CHILDREN_INDEX_DELAY_MS | 60000 | Defines the delay for scheduler that indexes subjects/contributors/classifications in a background |

The module uses system user to communicate with other modules from Kafka consumers.
For production deployments you MUST specify the password for this system user via env variable:
Expand Down
35 changes: 10 additions & 25 deletions descriptors/ModuleDescriptor-template.json
Original file line number Diff line number Diff line change
Expand Up @@ -1054,36 +1054,11 @@
"value": "",
"description": "Replication factor for `search.reindex.range-index` topic."
},
{
"name": "KAFKA_INDEX_SUB_RESOURCE_TOPIC_PARTITIONS",
"value": "50",
"description": "Amount of partitions for `search.index.sub-resource` topic."
},
{
"name": "KAFKA_INDEX_SUB_RESOURCE_TOPIC_REPLICATION_FACTOR",
"value": "",
"description": "Replication factor for `search.index.sub-resource` topic."
},
{
"name": "KAFKA_REINDEX_RECORDS_CONCURRENCY",
"value": "2",
"description": "Custom number of kafka concurrent threads for `inventory.reindex-records` message consuming."
},
{
"name": "KAFKA_INDEX_SUB_RESOURCE_CONCURRENCY",
"value": "2",
"description": "Custom number of kafka concurrent threads for `search.index.sub-resource` message consuming."
},
{
"name": "KAFKA_CONSUMER_INDEX_SUB_RESOURCE_MAX_POLL_RECORDS",
"value": "200",
"description": "Maximum number of records returned in a single call to poll() for instance sub-resource events."
},
{
"name": "KAFKA_CONSUMER_INDEX_SUB_RESOURCE_MAX_POLL_INTERVAL_MS",
"value": "600000",
"description": "Maximum processing time allowed for consumer on instance sub-resource events."
},
{
"name": "KAFKA_CONSUMER_MAX_POLL_RECORDS",
"value": "200",
Expand Down Expand Up @@ -1161,6 +1136,16 @@
"value": "smile",
"description": "Format for passing data to elasticsearch (json/smile)"
},
{
"name": "INSTANCE_CHILDREN_INDEX_DELAY_MS",
"value": "60000",
"description": "Scheduler delay for indexing subjects/contributors/classifications"
},
{
"name": "INSTANCE_CHILDREN_INDEX_BATCH_SIZE",
"value": "500",
"description": "Batch size for indexing subjects/contributors/classifications"
},
{
"name": "INITIAL_LANGUAGES",
"value": "eng",
Expand Down
16 changes: 8 additions & 8 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.3.5</version>
<version>3.4.0</version>
<relativePath />
</parent>

Expand Down Expand Up @@ -39,27 +39,27 @@
<folio-isbn-utils.version>1.8.0-SNAPSHOT</folio-isbn-utils.version>
<folio-cql2pgjson.version>35.3.0</folio-cql2pgjson.version>
<opensearch.version>2.18.0</opensearch.version>
<mapstruct.version>1.6.2</mapstruct.version>
<apache-commons-io.version>2.17.0</apache-commons-io.version>
<mapstruct.version>1.6.3</mapstruct.version>
<apache-commons-io.version>2.18.0</apache-commons-io.version>
<apache-commons-collections.version>4.4</apache-commons-collections.version>
<marc4j.version>2.9.5</marc4j.version>
<lombok.version>1.18.34</lombok.version>
<lombok.version>1.18.36</lombok.version>
<lombok.mapstruct-binding.version>0.2.0</lombok.mapstruct-binding.version>
<streamex.version>0.8.3</streamex.version>

<!-- Plugins versions -->
<folio-module-descriptor-validator.version>1.0.0</folio-module-descriptor-validator.version>
<maven-openapi-generator-plugin.version>7.9.0</maven-openapi-generator-plugin.version>
<folio-module-descriptor-validator.version>1.0.1</folio-module-descriptor-validator.version>
<maven-openapi-generator-plugin.version>7.10.0</maven-openapi-generator-plugin.version>
<maven-copy-rename-plugin.version>1.0.1</maven-copy-rename-plugin.version>
<maven-build-helper-plugin.version>3.6.0</maven-build-helper-plugin.version>
<maven-clean-plugin.version>3.4.0</maven-clean-plugin.version>
<maven-resources-plugin.version>3.3.1</maven-resources-plugin.version>
<maven-checkstyle-plugin.version>3.6.0</maven-checkstyle-plugin.version>
<maven-compiler-plugin.version>3.13.0</maven-compiler-plugin.version>
<maven-surefire-plugin.version>3.5.2</maven-surefire-plugin.version>
<maven-failsafe-plugin.version>3.5.1</maven-failsafe-plugin.version>
<maven-failsafe-plugin.version>3.5.2</maven-failsafe-plugin.version>
<maven-release-plugin.version>3.1.1</maven-release-plugin.version>
<checkstyle.version>10.20.0</checkstyle.version>
<checkstyle.version>10.21.0</checkstyle.version>
</properties>

<dependencies>
Expand Down
2 changes: 2 additions & 0 deletions src/main/java/org/folio/search/SearchApplication.java
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,13 @@
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;
import org.springframework.cloud.openfeign.EnableFeignClients;
import org.springframework.scheduling.annotation.EnableScheduling;

/**
* Folio search application.
*/
@EnableCaching
@EnableScheduling
@EnableFeignClients
@SpringBootApplication
public class SearchApplication {
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,12 @@ public static class IndexingSettings {
* Data format to use for passing data to elasticsearch.
*/
private IndexingDataFormat dataFormat;

/**
* Scheduler delay for indexing subjects/contributors/classifications.
* */
private long instanceChildrenIndexDelayMs;

}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
package org.folio.search.configuration.properties;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;

@Data
@Component
@NoArgsConstructor
@AllArgsConstructor(staticName = "of")
@ConfigurationProperties(prefix = "folio.system")
public class SystemProperties {

private String schemaName;
private String changeLog;
}
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.logging.log4j.message.FormattedMessage;
import org.folio.search.domain.dto.ResourceEvent;
import org.folio.search.model.event.SubResourceEvent;
import org.folio.search.model.types.ResourceType;
import org.folio.search.service.ResourceService;
import org.folio.search.service.config.ConfigSynchronizationService;
Expand Down Expand Up @@ -70,31 +69,6 @@ public void handleInstanceEvents(List<ConsumerRecord<String, ResourceEvent>> con

}

/**
* Handles instance sub-resource events and indexes them after extracting from instance resource..
*
* @param consumerRecords - list of consumer records from Apache Kafka to process.
*/
@KafkaListener(
id = KafkaConstants.INSTANCE_SUB_RESOURCE_LISTENER_ID,
containerFactory = "subResourceListenerContainerFactory",
topicPattern = "#{folioKafkaProperties.listener['index-sub-resource'].topicPattern}",
groupId = "#{folioKafkaProperties.listener['index-sub-resource'].groupId}",
concurrency = "#{folioKafkaProperties.listener['index-sub-resource'].concurrency}")
public void handleInstanceChildrenEvents(List<ConsumerRecord<String, SubResourceEvent>> consumerRecords) {
log.info("Processing instance sub resource events from kafka events [number of events: {}]",
consumerRecords.size());
var batch = consumerRecords.stream()
.map(ConsumerRecord::value)
.toList();
var batchByTenant = batch.stream().collect(Collectors.groupingBy(ResourceEvent::getTenant));
batchByTenant.forEach((tenant, resourceEvents) -> executionService.executeSystemUserScoped(tenant, () -> {
folioMessageBatchProcessor.consumeBatchWithFallback(resourceEvents, KAFKA_RETRY_TEMPLATE_NAME,
resourceService::indexInstanceSubResources, KafkaMessageListener::logFailedEvent);
return null;
}));
}

/**
* Handles authority record events and indexes them using event body.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ private void process(String tenant, List<ResourceEvent> batch) {
.map(ResourceEvent::getId)
.toList();
if (!idsToDrop.isEmpty()) {
repository.deleteEntities(idsToDrop);
deleteEntities(tenant, recordCollection.getKey(), repository, idsToDrop);
}

var noShadowCopiesInstanceEvents = recordByOperation.values().stream().flatMap(Collection::stream).toList();
Expand All @@ -127,6 +127,14 @@ private void process(String tenant, List<ResourceEvent> batch) {
}
}

private void deleteEntities(String tenant, String resourceType, MergeRangeRepository repository, List<String> ids) {
if (ResourceType.HOLDINGS.getName().equals(resourceType) || ResourceType.ITEM.getName().equals(resourceType)) {
repository.deleteEntitiesForTenant(ids, tenant);
} else {
repository.deleteEntities(ids);
}
}

private boolean isInstanceEvent(ResourceEvent event) {
var resourceName = event.getResourceName();
return ResourceType.INSTANCE.getName().equals(resourceName)
Expand Down

This file was deleted.

3 changes: 3 additions & 0 deletions src/main/java/org/folio/search/model/entity/TenantEntity.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
package org.folio.search.model.entity;

public record TenantEntity(String id, String centralId, boolean active) { }
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import java.util.UUID;
import lombok.Data;
import org.folio.search.model.types.ReindexEntityType;
import org.folio.search.model.types.ReindexRangeStatus;

@Data
public class MergeRangeEntity {
Expand All @@ -15,6 +16,8 @@ public class MergeRangeEntity {
public static final String RANGE_UPPER_COLUMN = "upper";
public static final String CREATED_AT_COLUMN = "created_at";
public static final String FINISHED_AT_COLUMN = "finished_at";
public static final String STATUS_COLUMN = "status";
public static final String FAIL_CAUSE_COLUMN = "fail_cause";

private final UUID id;
private final ReindexEntityType entityType;
Expand All @@ -23,5 +26,7 @@ public class MergeRangeEntity {
private final String upperId;
private final Timestamp createdAt;
private Timestamp finishedAt;
private final ReindexRangeStatus status;
private final String failCause;

}
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
import java.util.UUID;
import lombok.Data;
import org.folio.search.model.types.ReindexEntityType;
import org.folio.search.model.types.ReindexRangeStatus;

@Data
public class UploadRangeEntity {
Expand All @@ -14,12 +15,16 @@ public class UploadRangeEntity {
public static final String UPPER_BOUND_COLUMN = "upper";
public static final String CREATED_AT_COLUMN = "created_at";
public static final String FINISHED_AT_COLUMN = "finished_at";
public static final String STATUS_COLUMN = "status";
public static final String FAIL_CAUSE_COLUMN = "fail_cause";

private final UUID id;
private final ReindexEntityType entityType;
private final String lower;
private final String upper;
private final Timestamp createdAt;
private Timestamp finishedAt;
private final ReindexRangeStatus status;
private final String failCause;

}
2 changes: 2 additions & 0 deletions src/main/java/org/folio/search/model/service/ResultList.java
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ public class ResultList<T> {
/**
* Paged result data.
*/
@JsonAlias({"classificationTypes", "identifierTypes", "alternativeTitleTypes", "callNumberTypes", "locations",
"loccamps", "loclibs", "locinsts"})
private List<T> result = Collections.emptyList();

// The `key` is required per contract
Expand Down
28 changes: 28 additions & 0 deletions src/main/java/org/folio/search/model/types/ReindexRangeStatus.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package org.folio.search.model.types;

import lombok.Getter;

@Getter
public enum ReindexRangeStatus {
SUCCESS("Success"),
FAIL("Fail");

private final String value;

ReindexRangeStatus(String value) {
this.value = value;
}

public static ReindexRangeStatus valueOfNullable(String value) {
if (value == null) {
return null;
}

for (ReindexRangeStatus b : ReindexRangeStatus.values()) {
if (b.name().equalsIgnoreCase(value)) {
return b;
}
}
throw new IllegalArgumentException("Unexpected value '" + value + "'");
}
}
Loading

0 comments on commit c6dc1cb

Please sign in to comment.