Skip to content

Commit

Permalink
doc: update readme & pack
Browse files Browse the repository at this point in the history
  • Loading branch information
@edfus
edfus committed Jun 4, 2021
1 parent afb7693 commit 7cc66de
Show file tree
Hide file tree
Showing 9 changed files with 217 additions and 99 deletions.
158 changes: 99 additions & 59 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ A partial replacement is replacing only the 1st parenthesized capture group subs
Take the following snippet converting something like `import x from "../src/x.mjs"` into `import x from "../build/x.mjs"` as an example:

```js
import { sed as updateFileContent } from "stream-editor" ;
import { sed as updateFileContent } from "stream-editor";

updateFileContent({
file: "index.mjs",
Expand All @@ -55,6 +55,43 @@ Special replacement patterns (parenthesized capture group placeholders) are well

You can specify a truthy `isFullReplacement` to perform a full replacment instead.

### Asynchronous replacement

Yes, asynchronous function replacements just work like a charm.

```js
import { streamEdit } from "stream-editor";

const filepath = "./index.js";
const dest = "./build/index.js";

streamEdit({
from: fs.createReadStream(filepath),
to: fs.createWriteStream(dest),
replace: [
{
// match ${{ import("module.mjs") }} | ${{ expr( 1 + 1 ) }}
match: /\$\{\{\s*([A-Z_-]+?)\s*\(\s*(.+?)\s*\)\s*\}\}/i,
replacement: async (whole, method, input) => {
switch (method.toUpperCase()) {
case "IMPORT":
input = input.replace(/^["']|["']$/g, "");
const importFilePath = path.join(path.dirname(filepath), input);
return fs.promises.readFile(importFilePath, "utf-8");
case "EXPR":
return (async () => String(await eval(input)))();
default:
throw new Error(`unknown method ${method} in ${whole}`);
}
}
}
],
defaultOptions: {
isFullReplacement: true
}
});
```

### Substituting texts within files in streaming fashion

This package will create readable and writable streams connected to a single file at the same time, while disallowing any write operations to advance further than the current reading index. This feature is based on [rw-stream](https://github.com/signicode/rw-stream)'s great work.
Expand Down Expand Up @@ -221,62 +258,65 @@ Currently, stream-editor has zero dependency.

See <https://github.com/edfus/stream-editor/tree/master/test>.

```plain text
Normalize & Replace
√ can handle sticky regular expressions
√ can handle string match with special characters
√ can handle partial replacement with placeholders
√ can handle non-capture-group parenthesized pattern: Assertions
√ can handle non-capture-group parenthesized pattern: Round brackets
√ can handle pattern starts with a capture group
√ can handle malformed (without capture groups) partial replacement
√ can await replace partially with function
√ recognize $\d{1,3} $& $` $' and check validity (throw warnings)
Edit streams
√ should check arguments
√ should warn unknown/unneeded options
√ should respect FORCE_COLOR, NO_COLOR, NODE_DISABLE_COLORS
√ should pipe one Readable to multiple dumps (59ms)
√ should replace CRLF with LF
√ should have replaced /dum(b)/i to dumpling (while preserving dum's case)
√ should have global and local limits on replacement amount
√ should have line buffer maxLength
√ should edit and combine multiple Readable into one Writable
√ has readableObjectMode
√ can signal an unsuccessful substitution using beforeCompletion
√ can declare a limit below which a substitution is considered failed for a search
truncation & limitation
√ truncating the rest when limitations reached
√ not: self rw-stream
√ not: piping stream
transcoding
√ gbk to utf8 buffer
√ gbk to hex with HWM
error handling
√ destroys streams properly when one of them closed prematurely
√ destroys streams properly if errors occurred during initialization
√ multiple-to-one: can correctly propagate errors emitted by readableStreams
√ multiple-to-one: can handle prematurely destroyed readableStreams
√ multiple-to-one: can correctly propagate errors emitted by writableStream
√ multiple-to-one: can handle prematurely ended writableStream
√ multiple-to-one: can handle prematurely destroyed writableStream
√ one-to-multiple: can correctly propagate errors emitted by writableStreams
√ one-to-multiple: can handle prematurely ended writableStreams
√ one-to-multiple: can handle prematurely destroyed writableStreams
√ can handle errors thrown from postProcessing
√ can handle errors thrown from join functions
√ can handle errors thrown from replacement functions
corner cases
√ can handle empty content
√ can handle non-string in regular expression split result
try-on
√ can handle files larger than 64KiB
42 passing (325ms)
```plain text
Normalize & Replace
√ can handle sticky regular expressions
√ can handle string match with special characters
√ can handle partial replacement with placeholders
√ can handle non-capture-group parenthesized pattern: Assertions
√ can handle non-capture-group parenthesized pattern: Round brackets
√ can handle pattern starts with a capture group
√ can handle malformed (without capture groups) partial replacement
√ can await replace partially with function
√ recognize $\d{1,3} $& $` $' and check validity (throw warnings)
√ produce the same result as String.prototype.replace
Edit streams
√ should check arguments
√ should warn unknown/unneeded options
√ should respect FORCE_COLOR, NO_COLOR, NODE_DISABLE_COLORS
√ should pipe one Readable to multiple dumps (64ms)
√ should replace CRLF with LF
√ should have replaced /dum(b)/i to dumpling (while preserving dum's case)
√ should have global and local limits on replacement amount
√ should have line buffer maxLength
√ should edit and combine multiple Readable into one Writable
√ has readableObjectMode
√ can handle async replacements
√ can signal an unsuccessful substitution using beforeCompletion
√ can declare a limit below which a substitution is considered failed for a search
truncation & limitation
√ truncating the rest when limitations reached
√ not: self rw-stream
√ not: piping stream
transcoding
√ gbk to utf8 buffer
√ gbk to hex with HWM
error handling
√ destroys streams properly when one of them closed prematurely
√ destroys streams properly if errors occurred during initialization
√ multiple-to-one: can correctly propagate errors emitted by readableStreams
√ multiple-to-one: can handle prematurely destroyed readableStreams
√ multiple-to-one: can correctly propagate errors emitted by writableStream
√ multiple-to-one: can handle prematurely ended writableStream
√ multiple-to-one: can handle prematurely destroyed writableStream
√ one-to-multiple: can correctly propagate errors emitted by writableStreams
√ one-to-multiple: can handle prematurely ended writableStreams
√ one-to-multiple: can handle prematurely destroyed writableStreams
√ can handle errors thrown from postProcessing
√ can handle errors thrown from join functions
√ can handle errors thrown from replacement functions
corner cases
√ can handle empty content
√ can handle regular expressions that always match
√ can handle non-string in a regExp separator's split result
try-on
√ can handle files larger than 64KiB
45 passing (332ms)
```

## API
Expand All @@ -294,7 +334,7 @@ An object input with one or more following options is acceptable to `streamEdit`
| name | alias | expect | safe to ignore | default |
| :--: | :-: | :-----: | :-: | :--: |
| search | match | `string` \| `RegExp` | ✔ | none |
| replacement | x | `string` \| `(wholeMatch, ...args) => string` | ✔ | none |
| replacement | x | `string` \| `[async] (wholeMatch, ...args) => string` | ✔ | none |
| limit | x | `number` | ✔ | `Infinity` |
| maxTimes | x | `number` | ✔ | `Infinity` |
| minTimes | x | `number` | ✔ | `0` |
Expand Down Expand Up @@ -378,7 +418,7 @@ interface SearchAndReplaceOptions extends BasicReplaceOptions {
* a function that returns the replacement text can be passed.
*
* Special replacement patterns (parenthesized capture group placeholders)
* are well supported.
* / async replacement functions are well supported.
*
* For a partial replacement, $& (also the 1st supplied value to replace
* function) and $1 (the 2nd param passed) always have the same value,
Expand Down
7 changes: 4 additions & 3 deletions build/index.d.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,15 +57,16 @@ interface BasicReplaceOption extends BasicInfrequentReplaceOption {
* a function that returns the replacement text can be passed.
*
* Special replacement patterns (parenthesized capture group placeholders)
* are well supported.
* / async replacement functions are well supported.
*
* For a partial replacement, $& (also the 1st supplied value to replace
* function) and $1 (the 2nd param passed) always have the same value,
* supplying the matched substring in the parenthesized capture group
* you specified.
*/
replacement?: string | ((wholeMatch: string, ...args: string[]) => string);

replacement?:
string | ((wholeMatch: string, ...args: string[]) => string)
| ((wholeMatch: string, ...args: string[]) => Promise<string>)
}

interface SearchAndReplaceOption extends BasicReplaceOption {
Expand Down
64 changes: 51 additions & 13 deletions build/index.js
View file
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ function getProcessOptions(options) {
flags
),
replacement:
(wholeMatch, precededPart, substrMatch, ...rest) => {
async (wholeMatch, precededPart, substrMatch, ...rest) => {
let _replacement = replacement;
if (specialTreatmentNeeded) {
let i = 0;
Expand All @@ -270,12 +270,12 @@ function getProcessOptions(options) {
const userDefinedGroups = [substrMatch].concat(rest.slice(0, i));

if (isFunction) {
// partial replacement with a function
_replacement = replacement(
// function as a partial replacement
_replacement = await replacement(
substrMatch, ...userDefinedGroups, wholeMatch.indexOf(substrMatch), wholeMatch
);
} else {
// has capture group placeHolder
// is string & may have capture group placeHolders
_replacement = _replacement.replace(
captureGroupPlaceholdersPatternGlobal,
$n => {
Expand Down Expand Up @@ -403,16 +403,54 @@ function getProcessOptions(options) {
}) // do not insert a semicolon here
);

const processFunc = (part, EOF) => {
if (typeof part !== "string")
return ""; // For cases like "Adbfdbdafb".split(/(?=([^,\n]+(,\n)?|(,\n)))/)
const processFunc = async (part, EOF) => {
if (typeof part !== "string") {
return postProcessing(part); // For cases like "Adbfdbdafb".split(/(?=([^,\n]+(,\n)?|(,\n)))/)
}

for (const rule of replaceSet) {
let ret;
const resultIndices = [];
const resultPromises = [];

const { pattern, replacement: asyncReplace } = rule;
let trapWatchDog_i = -1;

while ((ret = pattern.exec(part)) !== null) {
if(trapWatchDog_i === pattern.lastIndex) {
pattern.lastIndex++;
continue;
}

replaceSet.forEach(rule => {
part = part.replace(
rule.pattern,
rule.replacement
);
});
trapWatchDog_i = pattern.lastIndex;

const startIndex = ret.index;
const endIndex = ret.index + ret[0].length;
const replacedResultPromise = asyncReplace(
...ret, ret.index, ret.input, ret.groups
); // the sync or async replacement function

resultIndices.push({
startIndex,
endIndex
});
resultPromises.push(replacedResultPromise);
}

const results = await Promise.all(resultPromises);

let lastIndex = 0;
let greedySnake = "";

for (let i = 0; i < results.length; i++) {
greedySnake = greedySnake.concat(
part.slice(lastIndex, resultIndices[i].startIndex).concat(results[i])
);
lastIndex = resultIndices[i].endIndex;
}

part = greedySnake.concat(part.slice(lastIndex, part.length));
}

return postProcessing(part, EOF);
};
Expand Down
64 changes: 51 additions & 13 deletions build/index.mjs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -255,7 +255,7 @@ function getProcessOptions(options) {
flags
),
replacement:
(wholeMatch, precededPart, substrMatch, ...rest) => {
async (wholeMatch, precededPart, substrMatch, ...rest) => {
let _replacement = replacement;
if (specialTreatmentNeeded) {
let i = 0;
Expand All @@ -269,12 +269,12 @@ function getProcessOptions(options) {
const userDefinedGroups = [substrMatch].concat(rest.slice(0, i));

if (isFunction) {
// partial replacement with a function
_replacement = replacement(
// function as a partial replacement
_replacement = await replacement(
substrMatch, ...userDefinedGroups, wholeMatch.indexOf(substrMatch), wholeMatch
);
} else {
// has capture group placeHolder
// is string & may have capture group placeHolders
_replacement = _replacement.replace(
captureGroupPlaceholdersPatternGlobal,
$n => {
Expand Down Expand Up @@ -402,16 +402,54 @@ function getProcessOptions(options) {
}) // do not insert a semicolon here
);

const processFunc = (part, EOF) => {
if (typeof part !== "string")
return ""; // For cases like "Adbfdbdafb".split(/(?=([^,\n]+(,\n)?|(,\n)))/)
const processFunc = async (part, EOF) => {
if (typeof part !== "string") {
return postProcessing(part); // For cases like "Adbfdbdafb".split(/(?=([^,\n]+(,\n)?|(,\n)))/)
}

for (const rule of replaceSet) {
let ret;
const resultIndices = [];
const resultPromises = [];

const { pattern, replacement: asyncReplace } = rule;
let trapWatchDog_i = -1;

while ((ret = pattern.exec(part)) !== null) {
if(trapWatchDog_i === pattern.lastIndex) {
pattern.lastIndex++;
continue;
}

replaceSet.forEach(rule => {
part = part.replace(
rule.pattern,
rule.replacement
);
});
trapWatchDog_i = pattern.lastIndex;

const startIndex = ret.index;
const endIndex = ret.index + ret[0].length;
const replacedResultPromise = asyncReplace(
...ret, ret.index, ret.input, ret.groups
); // the sync or async replacement function

resultIndices.push({
startIndex,
endIndex
});
resultPromises.push(replacedResultPromise);
}

const results = await Promise.all(resultPromises);

let lastIndex = 0;
let greedySnake = "";

for (let i = 0; i < results.length; i++) {
greedySnake = greedySnake.concat(
part.slice(lastIndex, resultIndices[i].startIndex).concat(results[i])
);
lastIndex = resultIndices[i].endIndex;
}

part = greedySnake.concat(part.slice(lastIndex, part.length));
}

return postProcessing(part, EOF);
};
Expand Down
Loading

0 comments on commit 7cc66de

Please sign in to comment.