Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On ongoing branch of making everything async #565

Draft
wants to merge 33 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
3a820c5
async Deflate. Start of writer
adamhathcock Jan 24, 2021
bcdfd99
async dispose and fix tests?
adamhathcock Jan 24, 2021
2b4f029
more async await
adamhathcock Feb 1, 2021
9cf8a3d
more awaits
adamhathcock Feb 6, 2021
cdba5ec
AsyncEnumerable usage in entries
adamhathcock Feb 7, 2021
d234f2d
First pass of trying tar
adamhathcock Feb 7, 2021
7d2dc58
More API fixes
adamhathcock Feb 7, 2021
c6a011d
Fixed reader issue
adamhathcock Feb 7, 2021
9738b81
Fix rewindable stream and encoding tests
adamhathcock Feb 7, 2021
f40d334
Tar and Xz mostly work
adamhathcock Feb 7, 2021
813bd5a
Async open entry
adamhathcock Feb 8, 2021
ef3d4da
Fix test and some zip writing
adamhathcock Feb 8, 2021
d6fe729
create async
adamhathcock Feb 8, 2021
db02e8b
Minor fixes
adamhathcock Feb 8, 2021
949e903
More LZMA conversion going, BZip2 not for now
adamhathcock Feb 13, 2021
1f37ced
AsyncStream everything
adamhathcock Feb 13, 2021
fe4cc8e
Zip LZMA write will roundtrip
adamhathcock Feb 13, 2021
ea688e1
Writer problems still :(
adamhathcock Feb 13, 2021
1520917
AsyncStream for BZip2
adamhathcock Feb 13, 2021
7e9fb64
Minor changes
adamhathcock Feb 14, 2021
be34fe2
Merge branch 'master' into async
adamhathcock Feb 14, 2021
5c11075
Updates for merge
adamhathcock Feb 14, 2021
8cdc49c
ReadByteAsync
adamhathcock Feb 14, 2021
14e6d95
More clean up doesn’t help
adamhathcock Feb 14, 2021
d9c53e1
ZLIbStreamfile fixes?
adamhathcock Feb 20, 2021
30da0b9
Fixed Gzip by reverting EmitHeaderAsync
adamhathcock Feb 20, 2021
0c35abd
Explicit exception for read shortcut
adamhathcock Feb 20, 2021
eeb6761
Reuse gzip header reading
adamhathcock Feb 20, 2021
249f11f
Rework some zip writing
adamhathcock Feb 21, 2021
997c11e
Bug fix on counting
adamhathcock Feb 21, 2021
5cfc608
More fixes?
adamhathcock Feb 21, 2021
dd710ec
fixed read only sub stream
adamhathcock Feb 21, 2021
e08e4e5
Enabling Bzip2 but something else is broken
adamhathcock Feb 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions src/SharpCompress/Archives/AbstractWritableArchive.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using SharpCompress.Common;
using SharpCompress.Readers;
using SharpCompress.Writers;
Expand Down Expand Up @@ -137,11 +139,11 @@ private bool DoesKeyMatchExisting(string key)
return false;
}

public void SaveTo(Stream stream, WriterOptions options)
public async Task SaveToAsync(Stream stream, WriterOptions options)
Copy link

@JanHyka JanHyka Feb 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CancellationToken parameter should be passed through whole call chain and exposed to library consumers (as non-mandatory parameter on top level). Otherwise caller won't be able to cancel running operation (timeouts).

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's definitely something I want to do. I might miss more. Need to find a Rider plugin to help me catch it.

{
//reset streams of new entries
newEntries.Cast<IWritableArchiveEntry>().ForEach(x => x.Stream.Seek(0, SeekOrigin.Begin));
SaveTo(stream, options, OldEntries, newEntries);
await SaveToAsync(stream, options, OldEntries, newEntries);
}

protected TEntry CreateEntry(string key, Stream source, long size, DateTime? modified,
Expand All @@ -157,7 +159,8 @@ protected TEntry CreateEntry(string key, Stream source, long size, DateTime? mod
protected abstract TEntry CreateEntryInternal(string key, Stream source, long size, DateTime? modified,
bool closeStream);

protected abstract void SaveTo(Stream stream, WriterOptions options, IEnumerable<TEntry> oldEntries, IEnumerable<TEntry> newEntries);
protected abstract Task SaveToAsync(Stream stream, WriterOptions options, IEnumerable<TEntry> oldEntries, IEnumerable<TEntry> newEntries,
CancellationToken cancellationToken = default);

public override void Dispose()
{
Expand Down
36 changes: 17 additions & 19 deletions src/SharpCompress/Archives/GZip/GZipArchive.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using SharpCompress.Common;
using SharpCompress.Common.GZip;
using SharpCompress.Readers;
Expand Down Expand Up @@ -82,17 +84,15 @@ public static bool IsGZipFile(FileInfo fileInfo)
return IsGZipFile(stream);
}

public void SaveTo(string filePath)
public Task SaveToAsync(string filePath)
{
SaveTo(new FileInfo(filePath));
return SaveToAsync(new FileInfo(filePath));
}

public void SaveTo(FileInfo fileInfo)
public async Task SaveToAsync(FileInfo fileInfo)
{
using (var stream = fileInfo.Open(FileMode.Create, FileAccess.Write))
{
SaveTo(stream, new WriterOptions(CompressionType.GZip));
}
using var stream = fileInfo.Open(FileMode.Create, FileAccess.Write);
await SaveToAsync(stream, new WriterOptions(CompressionType.GZip));
}

public static bool IsGZipFile(Stream stream)
Expand Down Expand Up @@ -139,24 +139,22 @@ protected override GZipArchiveEntry CreateEntryInternal(string filePath, Stream
return new GZipWritableArchiveEntry(this, source, filePath, size, modified, closeStream);
}

protected override void SaveTo(Stream stream, WriterOptions options,
IEnumerable<GZipArchiveEntry> oldEntries,
IEnumerable<GZipArchiveEntry> newEntries)
protected override async Task SaveToAsync(Stream stream, WriterOptions options,
IEnumerable<GZipArchiveEntry> oldEntries,
IEnumerable<GZipArchiveEntry> newEntries,
CancellationToken cancellationToken = default)
{
if (Entries.Count > 1)
{
throw new InvalidOperationException("Only one entry is allowed in a GZip Archive");
}
using (var writer = new GZipWriter(stream, new GZipWriterOptions(options)))

await using var writer = new GZipWriter(stream, new GZipWriterOptions(options));
foreach (var entry in oldEntries.Concat(newEntries)
.Where(x => !x.IsDirectory))
{
foreach (var entry in oldEntries.Concat(newEntries)
.Where(x => !x.IsDirectory))
{
using (var entryStream = entry.OpenEntryStream())
{
writer.Write(entry.Key, entryStream, entry.LastModifiedTime);
}
}
await using var entryStream = entry.OpenEntryStream();
await writer.WriteAsync(entry.Key, entryStream, entry.LastModifiedTime, cancellationToken);
}
}

Expand Down
3 changes: 2 additions & 1 deletion src/SharpCompress/Archives/IWritableArchive.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using System;
using System.IO;
using System.Threading.Tasks;
using SharpCompress.Writers;

namespace SharpCompress.Archives
Expand All @@ -10,7 +11,7 @@ public interface IWritableArchive : IArchive

IArchiveEntry AddEntry(string key, Stream source, bool closeStream, long size = 0, DateTime? modified = null);

void SaveTo(Stream stream, WriterOptions options);
Task SaveToAsync(Stream stream, WriterOptions options);

/// <summary>
/// Use this to pause entry rebuilding when adding large collections of entries. Dispose when complete. A using statement is recommended.
Expand Down
13 changes: 6 additions & 7 deletions src/SharpCompress/Archives/IWritableArchiveExtensions.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
using System;
using System.IO;
using System.Threading.Tasks;
using SharpCompress.Writers;

namespace SharpCompress.Archives
Expand All @@ -18,17 +19,15 @@ public static void AddEntry(this IWritableArchive writableArchive,
fileInfo.LastWriteTime);
}

public static void SaveTo(this IWritableArchive writableArchive, string filePath, WriterOptions options)
public static Task SaveToAsync(this IWritableArchive writableArchive, string filePath, WriterOptions options)
{
writableArchive.SaveTo(new FileInfo(filePath), options);
return writableArchive.SaveToAsync(new FileInfo(filePath), options);
}

public static void SaveTo(this IWritableArchive writableArchive, FileInfo fileInfo, WriterOptions options)
public static async Task SaveToAsync(this IWritableArchive writableArchive, FileInfo fileInfo, WriterOptions options)
{
using (var stream = fileInfo.Open(FileMode.Create, FileAccess.Write))
{
writableArchive.SaveTo(stream, options);
}
await using var stream = fileInfo.Open(FileMode.Create, FileAccess.Write);
await writableArchive.SaveToAsync(stream, options);
}

public static void AddAllFromDirectory(
Expand Down
30 changes: 14 additions & 16 deletions src/SharpCompress/Archives/Tar/TarArchive.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using SharpCompress.Common;
using SharpCompress.Common.Tar;
using SharpCompress.Common.Tar.Headers;
Expand Down Expand Up @@ -59,10 +61,9 @@ public static bool IsTarFile(FileInfo fileInfo)
{
return false;
}
using (Stream stream = fileInfo.OpenRead())
{
return IsTarFile(stream);
}

using Stream stream = fileInfo.OpenRead();
return IsTarFile(stream);
}

public static bool IsTarFile(Stream stream)
Expand Down Expand Up @@ -170,20 +171,17 @@ protected override TarArchiveEntry CreateEntryInternal(string filePath, Stream s
closeStream);
}

protected override void SaveTo(Stream stream, WriterOptions options,
IEnumerable<TarArchiveEntry> oldEntries,
IEnumerable<TarArchiveEntry> newEntries)
protected override async Task SaveToAsync(Stream stream, WriterOptions options,
IEnumerable<TarArchiveEntry> oldEntries,
IEnumerable<TarArchiveEntry> newEntries,
CancellationToken cancellationToken = default)
{
using (var writer = new TarWriter(stream, new TarWriterOptions(options)))
await using var writer = new TarWriter(stream, new TarWriterOptions(options));
foreach (var entry in oldEntries.Concat(newEntries)
.Where(x => !x.IsDirectory))
{
foreach (var entry in oldEntries.Concat(newEntries)
.Where(x => !x.IsDirectory))
{
using (var entryStream = entry.OpenEntryStream())
{
writer.Write(entry.Key, entryStream, entry.LastModifiedTime, entry.Size);
}
}
await using var entryStream = entry.OpenEntryStream();
await writer.WriteAsync(entry.Key, entryStream, entry.LastModifiedTime, cancellationToken);
}
}

Expand Down
19 changes: 11 additions & 8 deletions src/SharpCompress/Archives/Zip/ZipArchive.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using SharpCompress.Common;
using SharpCompress.Common.Zip;
using SharpCompress.Common.Zip.Headers;
Expand Down Expand Up @@ -163,23 +165,24 @@ protected override IEnumerable<ZipArchiveEntry> LoadEntries(IEnumerable<ZipVolum
}
}

public void SaveTo(Stream stream)
public Task SaveToAsync(Stream stream)
{
SaveTo(stream, new WriterOptions(CompressionType.Deflate));
return SaveToAsync(stream, new WriterOptions(CompressionType.Deflate));
}

protected override void SaveTo(Stream stream, WriterOptions options,
IEnumerable<ZipArchiveEntry> oldEntries,
IEnumerable<ZipArchiveEntry> newEntries)
protected override async Task SaveToAsync(Stream stream, WriterOptions options,
IEnumerable<ZipArchiveEntry> oldEntries,
IEnumerable<ZipArchiveEntry> newEntries,
CancellationToken cancellationToken = default)
{
using (var writer = new ZipWriter(stream, new ZipWriterOptions(options)))
await using (var writer = new ZipWriter(stream, new ZipWriterOptions(options)))
{
foreach (var entry in oldEntries.Concat(newEntries)
.Where(x => !x.IsDirectory))
{
using (var entryStream = entry.OpenEntryStream())
await using (var entryStream = entry.OpenEntryStream())
{
writer.Write(entry.Key, entryStream, entry.LastModifiedTime);
await writer.WriteAsync(entry.Key, entryStream, entry.LastModifiedTime, cancellationToken);
}
}
}
Expand Down
18 changes: 11 additions & 7 deletions src/SharpCompress/Common/ExtractionMethods.cs
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;

namespace SharpCompress.Common
{
Expand All @@ -8,10 +10,11 @@ internal static class ExtractionMethods
/// <summary>
/// Extract to specific directory, retaining filename
/// </summary>
public static void WriteEntryToDirectory(IEntry entry,
public static async ValueTask WriteEntryToDirectoryAsync(IEntry entry,
string destinationDirectory,
ExtractionOptions? options,
Action<string, ExtractionOptions?> write)
Func<string, ExtractionOptions?, CancellationToken, ValueTask> write,
CancellationToken cancellationToken = default)
{
string destinationFileName;
string file = Path.GetFileName(entry.Key);
Expand Down Expand Up @@ -52,19 +55,20 @@ public static void WriteEntryToDirectory(IEntry entry,
{
throw new ExtractionException("Entry is trying to write a file outside of the destination directory.");
}
write(destinationFileName, options);
await write(destinationFileName, options, cancellationToken);
}
else if (options.ExtractFullPath && !Directory.Exists(destinationFileName))
{
Directory.CreateDirectory(destinationFileName);
}
}

public static void WriteEntryToFile(IEntry entry, string destinationFileName,
public static async ValueTask WriteEntryToFileAsync(IEntry entry, string destinationFileName,
ExtractionOptions? options,
Action<string, FileMode> openAndWrite)
Func<string, FileMode, CancellationToken, ValueTask> openAndWrite,
CancellationToken cancellationToken = default)
{
if (entry.LinkTarget != null)
if (entry.LinkTarget is not null)
{
if (options?.WriteSymbolicLink is null)
{
Expand All @@ -85,7 +89,7 @@ public static void WriteEntryToFile(IEntry entry, string destinationFileName,
fm = FileMode.CreateNew;
}

openAndWrite(destinationFileName, fm);
await openAndWrite(destinationFileName, fm, cancellationToken);
entry.PreserveExtractionOptions(destinationFileName, options);
}
}
Expand Down
4 changes: 3 additions & 1 deletion src/SharpCompress/Common/GZip/GZipEntry.cs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;

namespace SharpCompress.Common.GZip
{
Expand Down Expand Up @@ -41,8 +42,9 @@ internal GZipEntry(GZipFilePart filePart)

internal override IEnumerable<FilePart> Parts => _filePart.AsEnumerable<FilePart>();

internal static IEnumerable<GZipEntry> GetEntries(Stream stream, OptionsBase options)
internal static async IAsyncEnumerable<GZipEntry> GetEntries(Stream stream, OptionsBase options)
{
await Task.CompletedTask;
yield return new GZipEntry(new GZipFilePart(stream, options.ArchiveEncoding));
}
}
Expand Down
Loading