debeltoni/super-giant/s3-bucket-ops

s5cmd Ops

Use this skill to interact with S3 via s5cmd for listing, copying, moving, and syncing artifacts.

Core rules

• The /proj/giant-data global data dir should be a local copy of the s3://giant-data/ bucket. Paths should correspond between local and S3.
• use only s5cmd for S3 operations.
• Always quote S3 URIs with double quotes.
• s5cmd mv and s5cmd cp require wildcards when moving directories. Use "s3://giant-data/path/*".
• Avoid destructive deletes unless explicitly requested.
• Avoid using s5cmd sync to or from the S3 bucket unless necessary. cp commands have less side effects. If you have to for whatever reason to use sync, use the --size-only flag because the user's S3 bucket is actually R2 which behaves badly with timestamps.
• The deafult and only bucket is "s3://giant-data/".

Common commands

Populat credentials:

source ~/.env-R2 
# NEVER CAT OR READ THIS FILE!

List buckets or prefixes:

s5cmd ls
s5cmd ls "s3://giant-data/prefix/"

Move a folder (use wildcard):

s5cmd mv "s3://giant-data/src/*" "s3://bucket/dst/"

Copy a single file:

s5cmd cp "s3://giant-data/path/file" "/local/path/"

Sync a folder to local:

s5cmd sync --size-only "s3://giant-data/prefix/" "/local/dir/"

Notes

• Keep local destinations under the global data root when syncing training artifacts.
• When unsure of a path, list parent prefixes first.