logo elektroda
logo elektroda
X
logo elektroda

Windows 10 and rsync, the efficient and fast way to back up our data

p.kaczmarek2 6945 11
ADVERTISEMENT
Treść została przetłumaczona polish » english Zobacz oryginalną wersję tematu
  • Screenshot showing Ubuntu installation via WSL in the Windows PowerShell window. .
    I will show here how to easily set up Linux rsync on Windows via WSL and then use it to make a copy of our files to a local NAS server via SMB, although analogously copies can also be made to a regular external hard drive. In a way, this method will automate what we often do quite unnecessarily manually, so it will ultimately save us time in keeping our files safe. In addition, the whole thing will work more efficiently, thanks to the fact that rsync only sends changed files and does not blindly copy everything.

    Today's electronics are inextricably linked to IT. It's really hard to imagine sifting through tonnes of catalogue notes in the form of physical binders taking up half our cupboard, and even harder to imagine designing a circuit fully on paper - from schematic to PCB design, which is also increasingly at least two layers. For this reason, I thought it worthwhile here to present a method for automatically creating copies via the tool rsync . Rsync is a console tool, normally available on Linux systems, but with a little effort it can be run on Windows too. I'll outline the whole process here step by step and perhaps save someone from losing files.

    Step 1 - Install WSL <span>
    These Windows steps are skipped if we are on Linux! .
    For some time now, Microsoft has offered us the possibility to install WSL, the so-called Windows Subsystem for Linux. WSL allows us to install a Linux distribution (such as Ubuntu, OpenSUSE, Kali, Debian, Arch Linux, etc.) on Windows without having to resort to less efficient and inconvenient solutions such as dual boot or there virtual machine. This allows us to run Linux programs from within our ten (or eleven...).
    Before installation, we need to check that we meet the requirements.
    You need Windows 10 version2004 or higher (Build 19041 or higher) or Windows 11. If we don't meet this, we do an upgrade.
    Then we open PowerShell - as administrator:
    PowerShell context menu with Run as Administrator option highlighted in Windows. .
    We type wsl --install :
    PowerShell window running as administrator with the command wsl --install typed. .
    The laborious installation process will then begin:
    Windows PowerShell window with the WSL installation command active and a message about installing Virtual Machine Platform. .
    Waiting:
    Screenshot of Windows PowerShell window running wsl --install command, installation progress at 74.1%. .
    In my case it continued to install Ubuntu by default, but the installation was stuck at 0.6%:
    Screenshot of WSL and Ubuntu installation in a Windows PowerShell window. .
    I aborted the process and performed the necessary reboot of the machine. Reboot can be done with the command wsl --install -d Ubuntu :
    Windows PowerShell window running the command to install Ubuntu via WSL, showing download progress at 16.8%..
    After a while, our Linux system will install and our Ubuntu prompt will start.

    Step 2 - we are finalizing Ubuntu on Windows .
    It's time to configure our Windows Ubuntu. We provide a username and password:
    Computer screen with PowerShell running Ubuntu installation via WSL and Ubuntu window asking to create a UNIX user. .
    Ubuntu is picky, I had to enter the name in lower case:
    PowerShell window installing WSL and Ubuntu, with Ubuntu terminal prompting for a UNIX username during initial setup. .
    We also set a password, I myself chose a fairly short four-letter one:
    Ubuntu installation screen in PowerShell on Windows with user setup in WSL. .


    Step 3 - getting familiar with Ubuntu .
    From now on, we can type "WSL" on the Windows bar and run the Linux command line this way:
    Screenshot of the Windows search showing the WSL app with a penguin icon in the results. .
    In my case it looks like this:
    Ubuntu 20.04 terminal launched in WSL showing the welcome message and system summary. .
    The obvious implication of this is that you need to know the Linux commands. That is, we operate via cd (change folder), mkdir (create), chmod (set permissions), etc, etc. Our disks can be found in the mnt folder:
    Screenshot of Ubuntu terminal running in WSL on Windows, showing filesystem navigation commands and directory listing. .
    For example, below I have looked at my W drive - lower case there. All files are visible:
    WSL terminal window on Windows displaying directory structure and filenames on mounted W drive with visible commands and text files. .

    Step 4 - plug the backup media into the WSL .
    Now we still need to access our media from the backup in WSL. Initially I wanted to use the normal Linux way of SMB, i.e. mapping the Samba resource to mnt via mount, but that didn't want to work here.
    Eventually I decided that I would map my server on Windows as a network drive .
    This is performed in This Computer, here:
    A screenshot from Windows Explorer highlighting the Map network drive option in the top toolbar. .
    We click through the wizard:
    Network location wizard window in Windows, with Next and Cancel buttons visible. .
    We enter the address of the share in the format <span>share:
    Network drive mapping window in Windows with options for selecting drive letter and network folder. .
    Below you have an example of what this looked like for me:
    Windows network drive mapping window with selected drive letter and network folder address. .
    It's worth ticking the box from automatically connecting after a reboot.
    Then in my case the disk did not appear at first, but after the reboot everything was ok:
    Windows “This PC” window with a mapped network drive “Backup (\NAS19)” as drive X: highlighted. .
    If you want to copy to a regular drive connected via USB it is even simpler - you don't need to map the network drive.

    Step 5 - rsync command .
    The name rsync comes from Remote Sync - meaning remote synchronisation. It is not a simple command cp , but much more. Rsync offers a lot of switches and possibilities, which you can read about in the documentation:
    https://linux.die.net/man/1/rsync
    Rsync is a much more efficient solution than blindly copying data. Rsync analyses the files and only transfers what has changed. Rsync can be easily resumed and paused. Rsync also compresses data on transfer.
    I didn't even have to install rsync itself. I only had to mount the newly added network share to WSL by first creating a folder for it in mnt (as root) with the mkdir command:
    WSL terminal window showing permission denied error for mkdir, then successful directory creation with sudo. .
    And then by executing mount:
    
    sudo mount -t drvfs X: /mnt/x
    
    .
    WSL terminal window showing the command to mount drive X: to /mnt/x using sudo mount -t drvfs. .
    As a result, WSL also started to see files from my mapped network drive.
    Then all I had to do was use the rsync command.
    I recommend this for a trial:
    
    rsync -av --delete --dry-run /mnt/w /mnt/x/b/w
    
    .
    The basic syntax is:
    
    rsync [przełączniki] [źródło] [cel]
    
    .
    This will make a copy of the files from the source on the specified target.
    Let's decipher the arguments one by one:
    - a is the archive mode (recursive synchronisation with the option to preserve attributes)
    - v - verbose mode (displaying progress and processing information)
    - delete - deletes files on the target location that do not exist on the source location
    - dry-run - enables simulation mode, where rsync displays what would happen, but does not perform the actual synchronisation, so you can experiment to your heart's content
    This is what it looked like for me:
    WSL terminal showing example use of the rsync command to simulate copying files between directories. .
    rsync will also give information about deleted files. And of course - the process can be interrupted and resumed, the whole thing works "smart" and does not copy blindly, it is an efficient solution.


    Summary
    Installing WSL on Windows really opens up a lot of possibilities and allows you to run a lot of convenient, previously rather inaccessible applications comfortably on "windows". Rsync is one of them and does quite well, especially as it synchronises everything locally. There are no clouds, Boxes, Drives or thereabouts commercial intermediaries, additionally constantly trying to persuade us to buy additional storage space. For this reason, I believe that the solution presented here is attractive to those who value privacy and want to have everything under their control.
    Finally, let me remind you of the important 3-2-1 rule:
    - 3 means that all our data should be kept in at least 3 copies (3 x backup)
    - 2 means that the copies should be on at least two different media
    - 1 means that at least one copy should be in a different location (physically, so-called "offsite").. off-site backup)
    In my case there are a bit more of these copies, as I also regularly burn DVDs with progress shots, but these 3 (or is it 4?) copies are the bare minimum when something is important.
    That's about it from my side. If something is not clear, please ask in the forum, I will help. .
    I hope to save at least one person from data loss this way....
    Do any readers use rsync? Or do you have any other methods to protect against data loss? .
    PS: With these at least three copies it's not a joke - just recently on our forum I read a story of a user who supposedly made a copy, but when the main media broke down (one with SMR - but that's another matter...) this copy failed him.... so three copies is the minimum.
    PS2: Just for the record - nothing replaces a version control system, but if it's about e.g. photos etc, then rather rsync will suffice

    Cool? Ranking DIY
    Helpful post? Buy me a coffee.
    About Author
    p.kaczmarek2
    Moderator Smart Home
    Offline 
    p.kaczmarek2 wrote 11800 posts with rating 9921, helped 563 times. Been with us since 2014 year.
  • ADVERTISEMENT
  • #2 20681510
    GGK
    Level 12  
    Such a copy does not protect against ransomware unfortunately. You can play around and set the mapping of the remote share before and after synchronisation. Possibly introduce rotation. We use the free veeam, which works superbly and has deduplication, conoresion and versioning.
  • #3 20681555
    p.kaczmarek2
    Moderator Smart Home
    It depends. In my case, the remote server is only switched on occasionally, so there's a period of time where I can figure out if something is wrong. I also have extra copies.

    A version control system is essential for e.g. code, but for data such as images, pdfs, etc., in my case rsync is sufficient.
    Helpful post? Buy me a coffee.
  • #4 20681844
    acctr
    Level 38  
    I have set up a backup and data management system for my own use. I use rsync, but this tool alone is not enough.
    There are problems such as duplicate files and strategies for managing them, synchronising backups (especially when it comes to photos and getting rid of unnecessary files), undoing changes, logging exactly what is being done, busy reports, etc.
    I base my solution on file hashes. All operations of adding, moving and deleting files are first performed on the hashes, then execution scripts are generated and scripts for undoing changes should something go wrong.
    Technically it is a set of tools written in Python, Bash, Java and C.
    Helpful post? Buy me a coffee.
  • ADVERTISEMENT
  • #6 20681970
    p.kaczmarek2
    Moderator Smart Home
    @simw good point.

    @acctr yes, I play like that too, I have at least a multithreaded, just based on hash tables (but the first level of hashes is... file length in bytes), deduplication program:
    Screenshot of a file deduplication program showing a summary of files, directories, and data usage. .
    Helpful post? Buy me a coffee.
  • ADVERTISEMENT
  • #7 20682157
    ex-or
    Level 28  
    I use rsync to back up my Linux machines. The root branch is dumped in its entirety (without /tmp and the like) - disaster recovery is easy and fast, which I have tested several times.
    The filesystems of bekapped machines are based on LVM, and bekap is done on snapshot so it can be done during normal operation while maintaining consistency. The transfer is done via SSH, which slows down the transfer a bit (a direct connection with the rsyncd daemon is fastest) but allows for flexible configuration, and is the most secure. Mounting or mapping network shares is rather a pain, neither convenient nor secure. On the NAS side, there is simple archive versioning based on hard links, which on the one hand allows to maintain almost any number of versions, on the other hand does not bloat the archive volume excessively. Bekap can be run by cron from the NAS server or from a client machine.
  • ADVERTISEMENT
  • #8 20682232
    acctr
    Level 38  
    ex-or wrote:
    I use rsync to backup Linux machines. The root branch is dumped in its entirety (without /tmp and the like) - disaster recovery is easy and fast, which I have tested several times.
    .
    I use the dedicated Timeshift tool for this.
    Helpful post? Buy me a coffee.
  • #9 20683068
    jvoytech
    Level 21  
    If by chance someone has msys2 installed on windows, rsync is installed with the command "pacman -S rsync"
  • #10 20683076
    acctr
    Level 38  
    jvoytech wrote:
    rsync is installed with the command "pacman -S rsync"
    .
    And this is not the package manager in Arch linux? In ubuntu/mint we have apt.
    Helpful post? Buy me a coffee.
  • #11 20683084
    p.kaczmarek2
    Moderator Smart Home
    Looks like @jvoytech is right. On my msys2 this command works. Thanks, I learned something new.

    File Explorer window with the msys64 folder open and a terminal window showing rsync installation in MSYS2. .
    Helpful post? Buy me a coffee.
  • #12 20683143
    simw
    Level 27  
    p.kaczmarek2 wrote:
    It looks like @jvoytech is right. With me on msys2 this command works. Thanks, I learned something new.
    .
    It seems to be similar with Cygwin64.
    It too is in packages and can be installed.
    All in all no surprise, after all it's a linux toolkit for windows.
    Cygwin Setup window displaying a package selection list, rsync searched. .

Topic summary

The discussion focuses on using rsync for efficient data backup on Windows 10 via Windows Subsystem for Linux (WSL). Users highlight the advantages of rsync, such as only transferring changed files, which saves time and resources. However, concerns about ransomware protection and the need for additional tools for managing duplicates and logging are raised. Alternatives like Veeam and Timeshift are mentioned for more comprehensive backup solutions. Users also discuss the availability of rsync through other platforms like msys2 and Cygwin, emphasizing its versatility in backup strategies.
Summary generated by the language model.
ADVERTISEMENT