大网数据广告活动
服务器运维问答

Copy s3 contents of bucket, including all versions

· 52CE 我爱测

I have a few S3 buckets that I want to hand access over to another organisation, as am handing over an existing hosting client to them. After looking through the docs, if I no longer want to be responsible for the bucket, it seem…

问题原文

原帖链接 · Chris Adams · CC BY-SA 3.0

I have a few S3 buckets that I want to hand access over to another organisation, as am handing over an existing hosting client to them.

After looking through the docs, if I no longer want to be responsible for the bucket, it seems the simplest option is to copy the contents of the new bucket across to a new bucket controlled by the new organisation, and make any existing apps write files to the new bucket from now on.

The bucket isn't very big:

aws s3 ls --human-readable --recursive --summarize s3://some-client-bucket

 # (snip… lots of files listed, all less than 10mb)

 # Total Objects: 22764
 # Total Size: 2.4 GiB

But some the bucket is versioned, and I have daily snapshots of some files going back the last year which I also want to be able to transfer across.

Is there a straightforward way to do this?

I understand how I might copy the current contents of a bucket to to new one controlled by another organisation after they grant me access using something like:

aws s3 sync s3://some-client-bucket s3://new-client-bucket --recursive

However, I don't think this will move across the versions too, and I'm I've been relying on S3's per file versioning rather than timestamping the files myself.

Do I have to jerry-rig some script to:

  • download each versioned file
  • rename it with a timestamp
  • upload it to the new bucket

Or is there some nifty extra feature in S3 to do this automagically for me?

社区高分解答

原帖链接 · Kons · CC BY-SA 4.0

This one is an old question, but I just happened to come against the same requirement. Today there is a fairly easy way to copy all files from one versioned bucket to another. I came up with the following PowerShell script to do it:

$sourceBucket = <source bucket>
$destinationBucket = <destination bucket>
$nextKeyMarker = $null
$nextVersionIdMarker = $null
do {
    $objectVersions = Get-S3Version -BucketName $sourceBucket `
                                    -KeyMarker $nextKeyMarker `
                                    -VersionIdMarker $nextVersionIdMarker
    $nextKeyMarker = $objectVersions.NextKeyMarker
    $nextVersionIdMarker = $objectVersions.NextVersionIdMarker
    $objectVersions.Versions | foreach { 
        Copy-S3Object -BucketName $sourceBucket `
                      -DestinationBucket $destinationBucket `
                      -Key $_.Key `
                      -DestinationKey $_.Key `
                      -VersionId $_.VersionId `
                      -CannedACLName bucket-owner-full-control
    }
} while ($objectVersions.IsTruncated -eq "True")
  1. This should work no matter the number of objects in the bucket as it copies object versions in bulks of 1000.
  2. The CannedACLName parameter is used to transfer ownership of the objects to the new AWS account where the destination bucket belongs. Without this, the owner of the copied objects will still be the account of the source bucket, making all copied objects inaccessible by the new owner account. If you are copying objects between buckets within the same account, you don't need to specify this.

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2017-01-09)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0 / CC BY-SA 4.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。