Skip to content

Commit

Permalink
[KYUUBI #6400] Fix memory leak when using saveToFile
Browse files Browse the repository at this point in the history
# 🔍 Description
## Issue References 🔗

Fix memory leak when using saveToFile mode.

FYI: https://stackoverflow.com/questions/45649044/scala-stream-iterate-and-memory-management

Stream is IterableAgain, which means, that it will keep all the elements you iterate through in case you want to see them again.

## Describe Your Solution 🔧

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

## Types of changes 🔖

- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Test Plan 🧪

#### Behavior Without This Pull Request ⚰️

#### Behavior With This Pull Request 🎉

#### Related Unit Tests

---

# Checklist 📝

- [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)

**Be nice. Be informative.**

Closes #6400 from turboFei/memory_leak.

Closes #6400

cdea358 [Wang, Fei] fix memory leak

Authored-by: Wang, Fei <[email protected]>
Signed-off-by: Wang, Fei <[email protected]>
  • Loading branch information
turboFei committed May 21, 2024
1 parent 522a28e commit 97aa4f7
Showing 1 changed file with 3 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,9 @@ class FetchOrcStatement(spark: SparkSession) {
val iterRow = orcIter.map(value =>
unsafeProjection(deserializer.deserialize(value)))
.map(value => toRowConverter(value))
new IterableFetchIterator[Row](iterRow.toIterable)
new IterableFetchIterator[Row](new Iterable[Row] {
override def iterator: Iterator[Row] = iterRow
})
}

def close(): Unit = {
Expand Down

0 comments on commit 97aa4f7

Please sign in to comment.