logo elektroda
logo elektroda
X
logo elektroda
Advanced Search
logo elektroda

Automatic backup - rclone - sample script - how to protect yourself against file loss?

p.kaczmarek2 2253 2
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
Reply Cool? Ranking DIY | New topic
Notify about new articles
  • Rclone logo with a symbol of three interlocked circles.
    In this topic I will present rclone - a simple tool to synchronize/backup files. I will show here a ready-to-use script that allows you to make a copy of selected folders with just one click and the entire copy will work on an incremental basis, so it will be a much better solution than the classic CTRL + C and CTRL + V still used by so many people, also technical ones.

    Why is rclone needed at all?
    Nowadays, electronics are permanently connected to the computer. We store all data digitally - whether there are diagrams, catalog notes, or firmware designs for microcontrollers and documentation. However, all these files are at risk of being lost when least expected. Digital data carriers may fail at any time and then potential data recovery, even if possible, is difficult and problematic. Even if the media does not fail, we are also at risk of ransomware viruses... Of course rclone this is just one of the options, and I usually recommend using many at the same time (just as you should use many backup copies - the 3-2-1 rule - i.e. "you should have 3 copies, on 2 different media, where 1 of them is on a different location"), but this topic will only talk about rclone.

    Where to get rclone?
    The tool in question is cross-platform and available for download completely free of charge, including: here:
    https://rclone.org/downloads/
    We simply download and unpack the package for our platform.
    This way we get the expected exe:
    Screenshot of a folder with rclone files in Windows.
    However, it is not a window program, only a command line tool, so you will need to write a script...

    My rclone script
    The documentation is available on the download page, but for simplicity, I will give you my ready-made script here:
    Code: text Expand Select all Copy to clipboard
    
rclone.exe sync W:/ X:/W --progress --transfers=64 --delete-excluded --exclude "Library/**" --exclude "Debug/**" --exclude "Release/**" --exclude "Android/**" --exclude "$RECYCLE.BIN/**" --exclude "Virtual Machines/**" --exclude "GIT/**" --exclude "C-Sky/**" --exclude ".venv"  --exclude "NoBackupToDelete/**" --delete-before
pause

    Once the above script is run, it will map the contents of the directory IN:/ In the catalogue X:/W . After its execution, the target directory will be a 1:1 representation of the source directory, which means that if we delete something from the source directory, it will also disappear in the target directory. The copy progress will be visible in the console pane. The maximum number of file transfers at once here is 64. Additionally, the script excludes several folders from synchronization to which paths are provided - including the trash bin (RECYCLE.BIN). Additionally, it includes an option --delete-before , which specifies that files deleted on the source are deleted on the copy before new files are transferred to it.

    How to use this script?
    Basically, there are several possibilities:
    - you can simply have a second disk in your computer for backups (in laptops you can sometimes replace the CD/DVD drive with a disk compartment), for example HDD, it does not have to be a fast disk
    - you can connect an external drive and manually run the script
    - you can also map the network drive, then rclone will also work
    Script calling can also be automatic or manual. It all depends on what we want to protect ourselves from. Having a copy on a second disk on the computer will not save us, for example, against ransomware...

    What are the advantages and disadvantages of this script?
    Let`s start with the advantages:
    + simplicity, much simpler than copy & paste
    + only what has changed compared to the previous copy is copied
    + deleted files on the source are also deleted on the target
    But this solution also has disadvantages that we must be aware of:
    - if we edit an important document and overwrite it and make a copy, the previous version will be overwritten (this solution is not SVN/GIT, it does not store the history of changes)
    - if we unknowingly delete something and run sync, it will also be deleted from the copy
    - if, for example, something encrypts our files, as above, synchronization will spoil the copies

    In addition, there is one more problem, but it is not directly related to this script - copying large numbers of small files takes a long time. If we want to keep, for example, a GIT repository with code for a longer period of time, it is worth considering using a RAR archive with repair data . One large RAR archive will copy much faster than many small files.

    Nevertheless, I think the script is useful (or at least it is a step forward compared to the "CTRL + C and CTRL + V" methods) - do you use this type of solutions? Or maybe you can suggest some way to improve this script? I invite you to discuss.

    Cool? Ranking DIY
    Helpful post? Buy me a coffee.
    About Author
    p.kaczmarek2
    Moderator Smart Home  
    Offline 
    p.kaczmarek2 wrote 11928 posts with rating 9987, helped 572 times. Been with us since 2014 year.
  • ADVERTISEMENT
  • #2 21024402
    karroryfer
    Level 12  
    actually, it`s interesting that there is no (or at least I don`t know of) a tool that, in the case of changed files, would allow incremental addition of only changes
    (more precisely, there are such, but they are rather larger backup systems, not synchronization systems)
  • #3 21025644
    Sam Sung
    Level 33  
    You can make an incremental backup, e.g. using Wildebeest tar , encryption GnuPG .
    Code: text Expand Select all Copy to clipboard
    tar c -g ~/backup/moje_dane1.snar --totals -X ~/backup/exclude -C ~/moje_dane1 . | pbzip2 | gpg2 --compress-algo=none --encrypt --default-recipient-self > ~/backup/moje_dane1.poziom_N.bz2

    The main advantage of this whole thing rclone - as I understand it - there is uploading files to the cloud, because you can copy files between disks or computers in the local network using a regular rsync `em. However, if we generate encrypted incremental backups, e.g. as shown above, then using rclone to upload them to cloud accounts might make sense.
    Restoring such a backup looks like this - from an empty directory for each file in the order of increments:
    Code: text Expand Select all Copy to clipboard
    gpg2 --decrypt ~/backup/moje_dane1.poziom_N.bz2 | pbzip2 -d | (cd ~/moje_dane1 && tar x -g /dev/null)

    Instead of providing a litany of --exclude arguments, in ~/backup/exclude we enter, for example:
    Code: text Expand Select all Copy to clipboard
    *~
*.class
*.o
*.iso
*.img
bzImage
node_modules
output

    (I suspect that rclone it can do that too and you don`t have to write --exclude a dozen times)
    Using rclone to simply copy files from disk to disk is style over substance. Alternatively, this shows Windows` poverty in the field of batch processing. Maybe some cmd.exe or PowerShell expert will save Windows` honor and show how to do such a trivial thing properly, i.e. without rclone? :)
Reply Cool? Ranking DIY | New topic
Notify about new articles
Report a violation of the law
ADVERTISEMENT