Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LifecycleRule.NoncurrentVersionExpiration.NewerNoncurrentVersions is not populating in PutLifecycleConfiguration request XML #3580

Closed
1 task
vkadiwala-conga opened this issue Dec 17, 2024 · 6 comments
Closed
1 task

LifecycleRule.NoncurrentVersionExpiration.NewerNoncurrentVersions is not populating in PutLifecycleConfiguration request XML #3580

vkadiwala-conga opened this issue Dec 17, 2024 · 6 comments
Assignees
@ashishdhingra
Labels
bug This issue is a bug. p2 This is a standard priority issue queued s3

Comments

@vkadiwala-conga
Copy link

vkadiwala-conga commented Dec 17, 2024 •
edited by ashishdhingra
Loading

Describe the bug

This is the rule I am creating on bucket through .Net AWSSDK.S3

var removeOlderVersionRule = new LifecycleRule
{
    Id = $"RemoveOlderVersionsBeyond2Versions",
    Status = LifecycleRuleStatus.Enabled,
    Filter = new LifecycleFilter()
    {
        LifecycleFilterPredicate = new LifecyclePrefixPredicate()
        {
            Prefix = ""
        }
    },
    NoncurrentVersionExpiration = new LifecycleRuleNoncurrentVersionExpiration
    {
        NoncurrentDays = 1,
        NewerNoncurrentVersions = 2
    }
};

While calling AmazonS3Client.PutLifecycleConfigurationAsync method below XML is getting generated

<LifecycleConfiguration
	xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<NoncurrentVersionExpiration>
			<NoncurrentDays>1</NoncurrentDays>
		</NoncurrentVersionExpiration>
		<ID>RemoveOlderVersionsBeyond2Versions</ID>
		<Filter>
			<Prefix />
		</Filter>
		<Status>Enabled</Status>
	</Rule>
</LifecycleConfiguration>

As you see, NewerNoncurrentVersions property is not getting generated in request XML and hence when the rule is applied it will not set the value
image

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

It should add the xml element NewerNoncurrentVersions in NoncurrentVersionExpiration element while calling AmazonS3Client.PutLifecycleConfigurationAsync method

<LifecycleConfiguration
	xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<NoncurrentVersionExpiration>
			<NoncurrentDays>1</NoncurrentDays>
			<NewerNoncurrentVersions>5</NewerNoncurrentVersions>
		</NoncurrentVersionExpiration>
		<ID>RemoveOlderVersionsBeyond2Versions</ID>
		<Filter>
			<Prefix />
		</Filter>
		<Status>Enabled</Status>
	</Rule>
</LifecycleConfiguration>

Current Behavior

While calling AmazonS3Client.PutLifecycleConfigurationAsync method below XML is getting generated

<LifecycleConfiguration
	xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<NoncurrentVersionExpiration>
			<NoncurrentDays>1</NoncurrentDays>
		</NoncurrentVersionExpiration>
		<ID>RemoveOlderVersionsBeyond2Versions</ID>
		<Filter>
			<Prefix />
		</Filter>
		<Status>Enabled</Status>
	</Rule>
</LifecycleConfiguration>

Reproduction Steps

var removeOlderVersionRule = new LifecycleRule
{
    Id = $"RemoveOlderVersionsBeyond2Versions",
    Status = LifecycleRuleStatus.Enabled,
    Filter = new LifecycleFilter()
    {
        LifecycleFilterPredicate = new LifecyclePrefixPredicate()
        {
            Prefix = ""
        }
    },
    NoncurrentVersionExpiration = new LifecycleRuleNoncurrentVersionExpiration
    {
        NoncurrentDays = 1,
        NewerNoncurrentVersions = 2
    }
};

var bucketLifecycleConfiguration = new LifecycleConfiguration
{
    Rules = new List<LifecycleRule> { removeOlderVersionRule }
};

var putRequest = new PutLifecycleConfigurationRequest
{
    BucketName = bucketName,
    Configuration = bucketLifecycleConfiguration
};

await AmazonS3Client.PutLifecycleConfigurationAsync(putRequest);

Possible Solution

Possible reason is in class PutLifecycleConfigurationRequestMarshaller.cs it is not creating an XML element for NewerNoncurrentVersions which has to be done like NoncurrentDays
image

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

It is happening in all versions of AWSSDK.S3 starting from Version="3.7.405.9" to Version="4.0.0-preview.4"

Targeted .NET Platform

.Net 8.0

Operating System and version

Windows 11 Enterprise
@vkadiwala-conga vkadiwala-conga added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 17, 2024
@vkadiwala-conga
Copy link
Author

Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/intro-lifecycle-rules.html#intro-lifecycle-rules-actions

image

@ashishdhingra ashishdhingra self-assigned this Dec 17, 2024
@ashishdhingra ashishdhingra added needs-reproduction This issue needs reproduction. s3 p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Dec 17, 2024
@ashishdhingra
Copy link
Contributor

Per S3 API: NoncurrentVersionExpiration, it does support NewerNoncurrentVersions property. However, it's missing in PutLifecycleConfigurationRequestMarshaller.

Per Changelog, S3 (3.7.404.7) had fix: Add missing NewerNoncurrentVersions to PutLifecycleConfigurationRequestMarshall in 70a6e1b. However, it was done only for NoncurrentVersionTransition here (per S3 API: NoncurrentVersionTransition, not NoncurrentVersionExpiration.

@ashishdhingra ashishdhingra added needs-review and removed needs-reproduction This issue needs reproduction. labels Dec 17, 2024
@ashishdhingra ashishdhingra changed the title LifecycleRule.NoncurrentVersionExpiration.NewerNoncurrentVersions is not populating in Put request xml and LifecycleRule.NoncurrentVersionExpiration.NewerNoncurrentVersions is not populating in PutLifecycleConfiguration request XML Dec 17, 2024
@ashishdhingra ashishdhingra mentioned this issue Dec 17, 2024
Merged
10 tasks
@vkadiwala-conga
Copy link
Author

vkadiwala-conga commented Dec 19, 2024 •
edited
Loading

@ashishdhingra in addition to this, if we retrieve the rules and without any change if we call AmazonS3Client.PutLifecycleConfigurationAsync method, it throws the exception

Step 1: Create below lifecycle configuration rule
image

Step 2: Get Lifecycle configuration rule using AmazonS3Client.GetLifecycleConfigurationAsync(configurationRequest)
It return below xml in the response:

<?xml version="1.0" encoding="UTF-8"?>
<LifecycleConfiguration
	xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<ID>RemoveOlderVersionsBeyond2Versions</ID>
		<Filter/>
		<Status>Enabled</Status>
		<NoncurrentVersionExpiration>
			<NoncurrentDays>1</NoncurrentDays>
		</NoncurrentVersionExpiration>
	</Rule>
</LifecycleConfiguration>

and in LifeCycleConfigurationRule it will set Filter = null
image

Step 3: Call AmazonS3Client.PutLifecycleConfigurationAsync without changing any configuration or rule which was received in Step#2. It will generate below xml and add in request payload

<LifecycleConfiguration
	xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Rule>
		<NoncurrentVersionExpiration>
			<NoncurrentDays>1</NoncurrentDays>
		</NoncurrentVersionExpiration>
		<ID>RemoveOlderVersionsBeyond2Versions</ID>
		<Status>Enabled</Status>
	</Rule>
</LifecycleConfiguration>

As it is not generating the Filter element which was received in response in Step#2, hence it throws exception "Amazon.S3.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema"

Amazon.S3.AmazonS3Exception: The XML you provided was not well-formed or did not validate against our published schema
 ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown.
   at Amazon.Runtime.HttpWebRequestMessage.ProcessHttpResponseMessage(HttpResponseMessage responseMessage)
   at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken)
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream)
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.S3Express.S3ExpressPreSigner.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)
   at AWSClientTestApp.Controllers.S3TestController.GetAndUpdateBucketPolicy(String bucketName) in C:\vivek_git\AWSClientTestApp\AWSClientTestApp\Controllers\S3TestController.cs:line 378
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(ActionContext actionContext, IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)

@ashishdhingra
Copy link
Contributor

@vkadiwala-conga For the other issue that you reported in previous comment, it appears to be similar to aws/aws-tools-for-powershell#367. Per my discussion with the team, it would be fixed in next major version of AWS SDK and PowerShell, which are currently in preview. There is a workaround proposed in aws/aws-tools-for-powershell#367 (comment), which is to parse response from GetLifecycleConfigurationAsync() to see if Filter property is null and then set it to empty collection if that is the case. As a general guideline, it is also recommended to use Prefix at LifecycleRule level to optimize costs.

@ashishdhingra
Copy link
Contributor

Fix released in AWSSDK.S3 (3.7.410.9)

@github-actions GitHub Actions
Copy link

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ashishdhingra ashishdhingra

Labels
bug This issue is a bug. p2 This is a standard priority issue queued s3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vkadiwala-conga @ashishdhingra