Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup: New option **--full-apply-filters** that can be used with **--full** option to apply filters anyway #962

Merged
merged 1 commit into from
Dec 31, 2024
Merged

Backup: New option **--full-apply-filters** that can be used with **--full** option to apply filters anyway #962

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@ Note: Can be used with `sfdx plugins:install sfdx-hardis@beta` and docker image
- Command updates
- [hardis:project:deploy:smart](https://sfdx-hardis.cloudity.com/hardis/project/deploy/smart/): Refactor deployment errors parsing: use JSON output instead of text output
- [hardis:org:test:apex](https://sfdx-hardis.cloudity.com/hardis/org/test/apex/): Display the number of failed tests in messages and notifications
- [hardis:org:monitor:backup](https://sfdx-hardis.cloudity.com/hardis/org/monitor/backup/): New option **--exclude-namespaces** that can be used with **--full** option
- [hardis:org:monitor:backup](https://sfdx-hardis.cloudity.com/hardis/org/monitor/backup/):
- New option **--exclude-namespaces** that can be used with **--full** option
- New option **--full-apply-filters** that can be used with **--full** option to apply filters anyway

- Core enhancements & fixes
- Obfuscate some data from text log files
Expand Down
62 changes: 41 additions & 21 deletions src/commands/hardis/org/monitor/backup.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,10 +40,15 @@ You can remove more metadata types from backup, especially in case you have too

Activate it with **--full** parameter, or variable MONITORING_BACKUP_MODE_FULL=true

Ignores filters (namespaces items & manifest/package-skip-items.xm) to retrieve ALL metadatas, including those you might not care about (reports, translations...)
Ignores filters (namespaces items & manifest/package-skip-items.xml) to retrieve ALL metadatas, including those you might not care about (reports, translations...)

As we can retrieve only 10000 files by call, the list of all metadatas will be chunked to make multiple calls (and take more time than filtered mode)

- if you use \`--full-apply-filters\` , manifest/package-skip-items.xml and MONITORING_BACKUP_SKIP_METADATA_TYPES filters will be applied anyway
- if you use \`--exclude-namespaces\` , namespaced items will be ignored

_With those both options, it's like if you are not using --full, but with chunked metadata download_

## In CI/CD

This command is part of [sfdx-hardis Monitoring](${CONSTANTS.DOC_URL_ROOT}/salesforce-monitoring-metadata-backup/) and can output Grafana, Slack and MsTeams Notifications.
Expand Down Expand Up @@ -79,6 +84,11 @@ If Flow history doc always display a single state, you probably need to update y
default: false,
description: 'If mode --full is activated, exclude namespaced metadatas',
}),
"full-apply-filters": Flags.boolean({
char: "z",
default: false,
description: 'If mode --full is activated, apply filters of manifest/package-skip-items.xml and MONITORING_BACKUP_SKIP_METADATA_TYPES anyway',
}),
outputfile: Flags.string({
char: 'f',
description: 'Force the path and name of output report file. Must end with .csv',
Expand Down Expand Up @@ -108,7 +118,9 @@ If Flow history doc always display a single state, you probably need to update y
protected full: boolean = false;
protected maxByChunk: number = 3000;
protected excludeNamespaces: boolean = false;
protected fullApplyFilters: boolean = false;

protected packageXmlToRemove: string | null = null;
protected extractPackageXmlChunks: any[] = [];
protected currentPackage: any = {};
protected currentPackageLen = 0;
Expand All @@ -126,6 +138,7 @@ If Flow history doc always display a single state, you probably need to update y
this.full = flags.full || (process.env?.MONITORING_BACKUP_MODE_FULL === "true" ? true : false);
this.maxByChunk = flags["max-by-chunk"] || 3000;
this.excludeNamespaces = flags["exclude-namespaces"] === true ? true : false;
this.fullApplyFilters = flags["full-apply-filters"] === true ? true : false;
this.outputFile = flags.outputfile || null;
this.debugMode = flags.debug || false;

Expand Down Expand Up @@ -256,11 +269,14 @@ If Flow history doc always display a single state, you probably need to update y
private async extractMetadatasFull(packageXmlFullFile: string, flags) {
let packageXmlToExtract = packageXmlFullFile;
// Filter namespaces if requested in the command
if (this.excludeNamespaces || process.env?.SFDX_HARDIS_BACKUP_EXCLUDE_NAMESPACES === "true") {
if (this.excludeNamespaces || process.env?.SFDX_HARDIS_BACKUP_EXCLUDE_NAMESPACES === "true" || this.fullApplyFilters) {
packageXmlToExtract = await this.buildFilteredManifestsForRetrieve(packageXmlFullFile);
const packageXmlFullFileWithoutNamespace = 'manifest/package-all-org-items-except-namespaces.xml';
const namespacesToFilter = (this.excludeNamespaces || process.env?.SFDX_HARDIS_BACKUP_EXCLUDE_NAMESPACES === "true") ? this.namespaces : [];
await filterPackageXml(packageXmlFullFile, packageXmlFullFileWithoutNamespace, {
removeNamespaces: this.namespaces,
removeNamespaces: namespacesToFilter,
removeStandard: false,
removeFromPackageXmlFile: this.packageXmlToRemove,
updateApiVersion: CONSTANTS.API_VERSION,
});
packageXmlToExtract = packageXmlFullFileWithoutNamespace;
Expand Down Expand Up @@ -348,17 +364,32 @@ If Flow history doc always display a single state, you probably need to update y
}

private async extractMetadatasFiltered(packageXmlFullFile: string, flags) {
const packageXmlBackUpItemsFile = await this.buildFilteredManifestsForRetrieve(packageXmlFullFile);

// Apply filters to package.xml
uxLog(this, c.cyan(`Reducing content of ${packageXmlFullFile} to generate ${packageXmlBackUpItemsFile} ...`));
await filterPackageXml(packageXmlFullFile, packageXmlBackUpItemsFile, {
removeNamespaces: this.namespaces,
removeStandard: true,
removeFromPackageXmlFile: this.packageXmlToRemove,
updateApiVersion: CONSTANTS.API_VERSION,
});

// Retrieve sfdx sources in local git repo
await this.retrievePackageXml(packageXmlBackUpItemsFile, flags);
}

private async buildFilteredManifestsForRetrieve(packageXmlFullFile: string) {
const packageXmlBackUpItemsFile = 'manifest/package-backup-items.xml';
const packageXmlSkipItemsFile = 'manifest/package-skip-items.xml';
let packageXmlToRemove: string | null = null;
if (fs.existsSync(packageXmlSkipItemsFile)) {
uxLog(
this,
c.grey(
`${packageXmlSkipItemsFile} has been found and will be use to reduce the content of ${packageXmlFullFile} ...`
)
);
packageXmlToRemove = packageXmlSkipItemsFile;
this.packageXmlToRemove = packageXmlSkipItemsFile;
}

// Add more metadata types to ignore using global variable MONITORING_BACKUP_SKIP_METADATA_TYPES
Expand All @@ -371,27 +402,16 @@ If Flow history doc always display a single state, you probably need to update y
)
);
let packageSkipItems = {};
if (fs.existsSync(packageXmlToRemove || '')) {
packageSkipItems = await parsePackageXmlFile(packageXmlToRemove || '');
if (fs.existsSync(this.packageXmlToRemove || '')) {
packageSkipItems = await parsePackageXmlFile(this.packageXmlToRemove || '');
}
for (const metadataType of additionalSkipMetadataTypes.split(',')) {
packageSkipItems[metadataType] = ['*'];
}
packageXmlToRemove = 'manifest/package-skip-items-dynamic-do-not-update-manually.xml';
await writePackageXmlFile(packageXmlToRemove, packageSkipItems);
this.packageXmlToRemove = 'manifest/package-skip-items-dynamic-do-not-update-manually.xml';
await writePackageXmlFile(this.packageXmlToRemove, packageSkipItems);
}

// Apply filters to package.xml
uxLog(this, c.cyan(`Reducing content of ${packageXmlFullFile} to generate ${packageXmlBackUpItemsFile} ...`));
await filterPackageXml(packageXmlFullFile, packageXmlBackUpItemsFile, {
removeNamespaces: this.namespaces,
removeStandard: true,
removeFromPackageXmlFile: packageXmlToRemove,
updateApiVersion: CONSTANTS.API_VERSION,
});

// Retrieve sfdx sources in local git repo
await this.retrievePackageXml(packageXmlBackUpItemsFile, flags);
return packageXmlBackUpItemsFile;
}

private async retrievePackageXml(packageXmlBackUpItemsFile: string, flags: any) {
Expand Down
Loading