S3 Random Upload Errors

[jay-herrera]

Before opening, please confirm:

• I have searched for duplicate or closed issues and discussions.
• I have read the guide for submitting bug reports.
• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

React

Amplify APIs

Storage

Amplify Version

v6

Amplify Categories

auth, storage

Backend

Amplify Gen 2 (Preview)

Environment information

# Put output below this line
  System:
    OS: macOS 14.6.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 301.13 MB / 64.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.0.0 - ~/.nvm/versions/node/v20.0.0/bin/node
    npm: 9.6.4 - ~/.nvm/versions/node/v20.0.0/bin/npm
    bun: 1.0.29 - /opt/homebrew/bin/bun
  Browsers:
    Chrome: 129.0.6668.100
    Safari: 17.6
  npmPackages:
    %name%:  0.1.0 
    @aws-amplify/backend: ^1.4.0 => 1.4.0 
    @aws-amplify/backend-cli: ^1.2.9 => 1.2.9 
    @aws-amplify/ui-react: ^6.5.3 => 6.5.3 
    @aws-amplify/ui-react-internal:  undefined ()
    @aws-amplify/ui-react-server:  undefined ()
    @eslint/js: ^9.11.1 => 9.12.0 
    @types/react: ^18.3.10 => 18.3.11 
    @types/react-dom: ^18.3.0 => 18.3.0 
    @vitejs/plugin-react: ^4.3.2 => 4.3.2 
    aws-amplify: ^6.6.4 => 6.6.4 
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/data:  undefined ()
    aws-amplify/data/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    aws-cdk: ^2.161.1 => 2.161.1 
    aws-cdk-lib: ^2.161.1 => 2.161.1 
    constructs: ^10.3.0 => 10.3.0 
    esbuild: ^0.24.0 => 0.24.0 (0.23.1, 0.21.5)
    eslint: ^9.11.1 => 9.12.0 
    eslint-plugin-react-hooks: ^5.1.0-rc.0 => 5.1.0-rc-fb9a90fa48-20240614 
    eslint-plugin-react-refresh: ^0.4.12 => 0.4.12 
    globals: ^15.9.0 => 15.11.0 (11.12.0, 14.0.0)
    react: ^18.3.1 => 18.3.1 
    react-dom: ^18.3.1 => 18.3.1 
    react-dropzone: ^14.2.9 => 14.2.9 
    tsx: ^4.19.1 => 4.19.1 
    typescript: ^5.6.3 => 5.6.3 (4.4.4, 4.9.5)
    typescript-eslint: ^8.7.0 => 8.8.1 
    vite: ^5.4.8 => 5.4.8 
  npmGlobalPackages:
    corepack: 0.17.2
    npm: 9.6.4
    yalc: 1.0.0-pre.53

Describe the bug

Hey, guys! I have been having some issues with random failures when uploading to S3 using uploadData from a web browser. I have created a POC with amplify that is just auth and an S3 bucket. The request does not seem to actually reach the S3 endpoint, and does not return a response. I assume that the error is caused by a timeout because no response is ever received.

I do not see the failed requests in the S3 Access Logs. On safari, I get a "Network Error - Failed to load resource: The network connection was lost" emitted, and chrome emits "ERR_CONNECTION_RESET". In order to rule out cors errors, and permission errors, I gave the broadest possible cors configuration, and gave all S3 actions access to my auth role.

I have also tried just using the S3 client directly. I have also tried using a bucket not attached to Amplify. I have tried different networks, different devices,, and different file types. The error is emmitted from xhr-http-handler, but there is no more detailed information. I am at my wits' end, looking everywhere for at least a thread to pull on, but everything has been fruitless so far.

Expected behavior

S3 Uploads work reliably

Reproduction steps

1. Create S3 Bucket
2. Try to upload file using UploadData
3. Errors will happen randomly only sometimes

Code Snippet

// Put your code below this line.
import {useDropzone} from 'react-dropzone'
import amplifyOutputs from '../amplify_outputs.json'
import {useState} from 'react'
import {uploadData} from 'aws-amplify/storage'
import {Authenticator} from '@aws-amplify/ui-react'
import {Amplify} from 'aws-amplify'
import './App.css'

Amplify.configure(amplifyOutputs)

const s3FileType = {
	gif: 'image/gif',
	jpeg: 'image/jpeg',
	jpg: 'image/jpeg',
	mov: 'video/mp4',
	mp4: 'video/mp4',
	png: 'image/png',
	svg: 'image/svg+xml'
}

const generateUniqueFileName = (fileName: string) => {
	const split = fileName.split('.')

	return `${split[0]}-${generateRandomId()}.${split[split.length - 1]}` // examplefile-ogly4y8m.png
}

const generateRandomId = () => {
	return Math.random().toString(36).slice(2, 7)
}

function App() {
	const [files, setFiles] = useState<File[]>([])
	const [loading, setLoading] = useState(false)
	const {getInputProps, getRootProps} = useDropzone({
		accept: {
			'image/png': ['.png', '.jpg', '.jpeg', '.gif', '.svg', '.webp'],
			'text/plain': ['.txt'],
			'video/mp4': ['.mp4', '.mov', '.webm']
		},
		maxFiles: 10,
		onDrop: (acceptedFiles) => {
			setFiles(acceptedFiles)
		}
	})

	return (
		<Authenticator>
			{({signOut, user}) => (
				<>
					<h3>Hello {user?.username}</h3>
					<button onClick={signOut}>Sign out</button>
					<h1>S3 Upload POC</h1>
					<div className='card'>
						{loading ? (
							<h2>Uploading...</h2>
						) : (
							<>
								<button {...getRootProps()}>
									<input
										{...getInputProps({
											name: '',
											type: 'file',
											multiple: true,
											style: {display: 'none'}
										})}
									/>
									Upload Files
								</button>

								<button
									onClick={async () => {
										if (!files.length) {
											return null
										}
										setLoading(true)
										try {
											const uploads = await Promise.all(
												files.map(async (file) => {
													const fileName = generateUniqueFileName(file.name)
													const ext =
														(file.name
															.split('.')
															.pop() as keyof typeof s3FileType) || 'png'
													return uploadData({
														options: {
															contentType: s3FileType[ext]
														},
														path: ({identityId}) => {
															return `${identityId}/${fileName}`
														},
														data: file
													}).result
												})
											)
											console.log(uploads)
										} catch (e) {
											alert('Error uploading files')
										}
										setLoading(false)
										setFiles([])
									}}
								>
									Upload To S3
								</button>
							</>
						)}
						<ul>
							{files.map((file) => (
								<li key={file.name}>{file.name}</li>
							))}
						</ul>
					</div>
				</>
			)}
		</Authenticator>
	)
}

export default App

Log output

// Put your logs below this line
[Error] [ERROR] 21:42.29 xhr-http-handler - Network Error
	(anonymous function) (user-script:29:1:901)
	_log (index-BTpIe6pi.js:44:18368)
	error (index-BTpIe6pi.js:44:18779)
	(anonymous function) (index-BTpIe6pi.js:47:31016)

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[HuiSF]

Hi @jay-herrera A network error along ERR_CONNECTION_RESET is mostly caused by your application cannot establish a connection to your S3 bucket endpoint.

Could you follow this article to ensure the connectivity from your development environment to your S3 bucket endpoint? The region where your bucket was created in may also be a factor if the region is far from your location.

In addition could you also provide a sample of the failed request, including the URL, request headers and response (if any), you can find these information from developer tool, network tab.

[jay-herrera]

@HuiSF Hey! I will follow that article and report back:)

[jay-herrera]

@HuiSF
Ok, I have done a few things.

1. I followed that article and noticed that the server connection closed at around 23 seconds every time. Here is my output:
   Trying 52.216.34.154...
   Connected to s3-r-w.us-east-1.amazonaws.com.
   Escape character is '^]'.
   Connection closed by foreign host.
   telnet 443 0.01s user 0.01s system 0% cpu 23.163 total

2. I patched the aws-amplify library to give more detailed errors for the xhr failure. You can find the logs attached as xhr.log. This did not give much further insight, as it appeared that no response is ever returned to the xhr request. The error was a "Progress Event", which indicates that the problem did not occur in the initial sennding of the request.

3. I set up a man in the middle proxy to analyze the traffic. You can find the result attached as mitm.txt, however, you may want to use mitmproxy to view it, because it's not encoded in a readable format. The result also indicated that the server had closed the connection before the multipart upload was finished.

I think that #12912 may be related.
xhr.log
mitm.txt

[AllanZhengYP]

Hi @jay-herrera

Thanks for getting back at us with the details. I tried to replay with the mitm proxy but I got no luck. The requests are all timed out now. The xhr log is more helpful, but it indicates the browser emitting net::ERR_CONNECTION_RESET error.

There are various reasons can cause this issue and most of them are related to the network connectivity. Here the AWS SDK provides some step to trouble shoot this error as well.

Did you start to see this error after upgrading to v6? Or did you start to see this error after change of network condition?

[jay-herrera]

@AllanZhengYP After some further investigation and much trial and much trial and error, I think I've found a solution that seems to be working very well. I did have to use s3 through the aws-sdk, then I set all of the native retries to only 1 maxAttempt. Then I wrapped everything in this retry wrapper:

export const wrapRetry = async <T>(
	fn: (..._any: any[]) => Promise<T>,
	maxAttempts: number,
	backoffFunction: (attemptNumber: number) => number
) => {
	for (let attempt = 0; attempt < maxAttempts; attempt++) {
		try {
			await new Promise((resolve) => {
				return setTimeout(resolve, backoffFunction(attempt))
			})

			return await fn()
		} catch (err: any) {
			if (attempt === maxAttempts - 1) {
				throw err
			}
			console.error(err, typeof err, JSON.stringify(err))

			if (err.httpStatusCode === 400 || err.httpStatusCode === 403) {
				throw err
			}
			console.info(`Attempt ${attempt + 1} failed. Retrying...`)
		}
	}

	throw new Error('Maximum number of attempts exceeded.')
}

I've learned a few things from this process:
-I am uncertain if uploadData implements a retry strategy directly
-If it does, the strategy does not recognize "net::ERR_CONNECTION_RESET", "ERR_NO_RESPONSE", or "503 SLOWDOWN" as retryable errors, although experimentally, these errors are likely to succeed on a retry
-Using backoff functions for retries has a huge effect on how likely a retry is to succeed

[cwomack]

@jay-herrera, thank you for the additional context here. We're investigating the retry handler further and will mark this as a bug.

[HuiSF]

Hi @jay-herrera We released v6.6.7 which fixes retry is not occurring as expected, which should mitigate the issue you are facing, please upgrade and test again, thanks!

[cwomack]

Closing this issue as it should now be resolved per the above comment! If anyone still experiences this on v6.6.7+, please feel free to reply back and let us know any reproduction steps.

Thank you!