Skip to content

How to solve RClone S3 Storage is not working as expected bug when backing up Discord Data?

Example Command


rclone sync ~/DiscordScraping/DiscordData raw-discord-data:raw-discord-data --s3-list-chunk 500

Logs

➜  ~ rclone sync ~/DiscordScraping/DiscordData raw-discord-data:raw-discord-data --s3-list-chunk 500      
2023/10/03 17:37:03 ERROR : $GUILD_ID/$Guild_Name - OFFICIAL ▬▬▬▬▬▬▬▬▬▬ - 🚨│minor-announcements [1048586302980886560].json_Files/267198358dbc223d888e0cb1cf2eb234-21157.png: Failed to copy: expected element type <Error> but have <html>
2023/10/03 17:37:26 ERROR : $GUILD_ID/$Guild_Name - OFFICIAL ▬▬▬▬▬▬▬▬▬▬ - 🚨│minor-announcements [1048586302980886560].json_Files/334ac5f99e4355495ade4cc2fb64311d-B3947.png: Failed to copy: expected element type <Error> but have <html>

Research

➜  ~ rclone sync ~/DiscordScraping/DiscordData raw-discord-data:raw-discord-data   --s3-copy-cutoff 512M
2023/10/03 17:54:44 ERROR : $GUILD_ID/$Guild_Name - ↣ Lux Official ↢ - 🔊┃𝖠𝗇𝗇𝗈𝗎𝗇𝖼𝖾𝗆𝖾𝗇𝗍𝗌 [1017402092052287568].json_Files/c65850262478d9a6448cb534c5db56fe-423FD.png: Failed to copy: expected element type <Error> but have <html>
  • Potential Work Around's
    • Pipe rclone output to file, parse out error files, create tar archive for files, upload to S3
      • This involves capturing and processing logs which is a skill I want to work on
    • Create tar archives for each guild and upload those to S3 in addition to the individual files
      • This option will take up a lot of additional storage we still want the raw attachments to be still available on object storage