将大文件上传到Amazon S3时出现问题

Z时代
2024-01-10
分类：问答

我尝试使用Amazon-

SDK（Java）示例代码S3TransferProgressSample.java将大文件上传到Amazon-S3存储（也在AWS文档中发布在此处）。

但是，当我尝试上传11 GB的文件时，上传会卡在不同的位置，并显示错误消息：

Unable to upload file to Amazon S3: Unable to upload part: Unable toexecute HTTP request: Unbuffered entity enclosing request can not be repeated " (attached screenshot).

看起来在发生IOException之后，SDK无法重试该请求（请参见下文）。

有人遇到吗？解决此问题的最佳实践是什么？任何代码表示赞赏。

INFO: Received successful response: 200, AWS Request ID: 2B66E7669E24DA75 Jan 15, 2011 6:44:46 AM com.amazonaws.http.HttpClient execute INFO: Sending Request: PUT s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters: (uploadId: m9MqxzD484Ys1nifnX._IzJBGbCFIoT_zBg0xdd6kkZ4TAtmcG0lXQOE.LeiSEuqn6NjcosIQLXJeKzSnKllmw--, partNumber: 1494, ) Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error** Jan 15, 2011 6:45:10 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Jan 15, 2011 6:45:12 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error** Jan 15, 2011 6:45:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry **INFO: I/O exception (java.net.SocketException) caught when processing request: Connection reset by peer: socket write error** Jan 15, 2011 6:45:13 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Jan 15, 2011 6:45:13 AM com.amazonaws.http.HttpClient execute **WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated.** Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:14 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:15 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:16 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:17 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute WARNING: Unable to execute HTTP request: Unbuffered entity enclosing request can not be repeated. Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute .... Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient handleResponse **INFO: Received successful response: 204, AWS Request ID: E794B8FCA4C3D007** Jan 15, 2011 6:45:21 AM com.amazonaws.http.HttpClient execute ... Jan 15, 2011 6:45:19 AM com.amazonaws.http.HttpClient execute INFO: Sending Request: DELETE s3.amazonaws.com /test_file_upload/autogenerated.txt Parameters: ... Jan 15, 2011 6:47:01 AM com.amazonaws.http.HttpClient handleErrorResponse INFO: Received error response: Status Code: 404, AWS Request ID: 0CE25DFE767CC595, AWS Error Code: NoSuchUpload, AWS Error Message: The specified upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.

回答：

尝试使用底层API。

当出现问题时，这将使您有更多的控制权，因为它们可能与11GB文件有关。

往返S3的请求有时会失败。使用低级API，如果上传失败，您将可以重试其中的一部分。

重构一下Amazon文档中的示例：

// Step 2: Upload parts.
long filePosition = 0;
for (int i = 1; filePosition < contentLength; i++) {
    // Last part can be less than 5 MB. Adjust part size.
    partSize = Math.min(partSize, (contentLength - filePosition));
    // Create request to upload a part.
    UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(existingBucketName).withKey(keyName)
                .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                .withFileOffset(filePosition)
                .withFile(file)
                .withPartSize(partSize);
    // repeat the upload until it succeeds.
    boolean anotherPass;  
        do {
              anotherPass = false;  // assume everythings ok
              try {
                  // Upload part and add response to our list.
                  partETags.add(s3Client.uploadPart(uploadRequest).getPartETag());
              } catch (Exception e) {
                    anotherPass = true; // repeat
              }
        } while (anotherPass);
     filePosition += partSize;
}
   // Step 3: complete.
   CompleteMultipartUploadRequest compRequest = new 
                     CompleteMultipartUploadRequest(
                                existingBucketName, 
                                keyName, 
                                initResponse.getUploadId(), 
                                partETags);
   s3Client.completeMultipartUpload(compRequest);