How to delta/incrementally copy folders with many files with an integrity check on Windows

rsync is a utility for efficiently transferring and synchronizing files between a computer and an external hard drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operating systems. The rsync algorithm is a type of delta encoding, and is used for minimizing network usage. Zlib may be used for additional data compression, and SSH or stunnel can be used for security.

Rsync is typically used for synchronizing files and directories between two different systems. For example, if the command rsync local-file user@remote-host:remote-file is run, rsync will use SSH to connect as user to remote-host. Once connected, it will invoke the remote host's rsync and then the two programs will determine what parts of the local file need to be transferred so that the remote file matches the local one.


Great overview of technical working of rsync
http://tutorials.jenkov.com/rsync/overview.html

Get Rsync for Windows, open source code project. It's a self contained exe.
https://www.itefix.net/cwrsync

For an easy to use similar project with GUI use SyncToy 2.1 free from Microsoft. 

https://www.microsoft.com/en-us/download/details.aspx?id=15155




Sample command 

$ rsync -av Documents/* /tmp/documents

In the command above, the option:

1. -a – means archive mode
2. -v – means verbose, showing details of ongoing operations

All Cmd Line Options

rsync  version 3.1.3  protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, no xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
     --info=FLAGS            fine-grained informational verbosity
     --debug=FLAGS           fine-grained debug verbosity
     --msgs2stderr           special output handling for debugging
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place (SEE MAN PAGE)
     --append                append data onto shorter files
     --append-verify         like --append, but with old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the source tree
     --munge-links           munge symlinks to make them safer (but unusable)
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve the file's executability
     --chmod=CHMOD           affect file and/or directory permissions
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
 -J, --omit-link-times       omit symlinks from --times
     --super                 receiver attempts super-user activities
 -S, --sparse                turn sequences of nulls into sparse blocks
     --preallocate           allocate dest files before writing them
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (without delta-xfer algorithm)
     --checksum-choice=STR   choose the checksum algorithms
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on the remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that already exist on receiver
     --remove-source-files   sender removes synchronized files (non-dirs)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during the transfer
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --ignore-missing-args   ignore missing source args without error
     --delete-missing-args   delete missing source args from destination
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of directories even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --usermap=STRING        custom username mapping
     --groupmap=STRING       custom groupname mapping
     --chown=USER:GROUP      simple username/groupname mapping
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
 -M, --remote-option=OPTION  send OPTION to the remote side only
     --size-only             skip files that match in size
 -@, --modify-window=NUM     set the accuracy for mod-time comparisons
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare destination files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with a suffix in LIST
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --include-from=FILE     read include patterns from FILE
     --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --address=ADDRESS       bind address for outgoing socket to daemon
     --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
     --blocking-io           use blocking I/O for the remote shell
     --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
     --out-format=FORMAT     output updates using the specified FORMAT
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT
     --password-file=FILE    read daemon-access password from FILE
     --list-only             list the files instead of copying them
     --bwlimit=RATE          limit socket I/O bandwidth
     --outbuf=N|L|B          set output buffering to None, Line, or Block
     --write-batch=FILE      write a batched update to FILE
     --only-write-batch=FILE like --write-batch but w/o updating destination
     --read-batch=FILE       read a batched update from FILE
     --protocol=NUM          force an older protocol version to be used
     --iconv=CONVERT_SPEC    request charset conversion of filenames
     --checksum-seed=NUM     set block/file checksum seed (advanced)
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers

How to fix PDFs ending in .EXE

Update : Metadata Consulting [dot] ca: How to fix right-to-left files - bulk command line tool

Malware writers can trick you in 2 ways into thinking your file is a "PDF looking" file. 

Firstly, a maliciously constructed “.exe” can be built to display an PDF icon, so it looks like PDF default reader will open this file. If the filename is really long then, you can't see the extension (see image below). 

2ndly and may not be so obvious, malicious PDF filename is constructed as with a right-to-left override character is such a way that the file ends ".pdf" extension, but really is an ".exe".  


So in example below, the 2nd file looks like a ".txt" file, but is really a ".docx" file (the 1st file). The 1st file has been cleansed of the RTL Unicode character, and ends in ".docx". 

The PDF file is actually an ".exe" file, but looks like it will open with default PDF reader. 

Download RTLExample.7z ( it includes the above files with PDF ".exe" example. The files contain no viruses or malware. The PDF is safe ".exe", and just opens this page in Chrome). This may read as malicious files by your anti-virus software as it contains the left-to-right characters in filename. You can create these files yourself as well. 

Here's the same files as viewed from the command (cmd.exe) line. The box character represents the RTL character.

Note: Detection of malicious file is never done by a filename alone, so a good antivirus will flag the contents of this file, for known signatures. BUT you can remove the annoying RTL character with the free tool below! 

How is RTLO being abused by malware writers?

In apps that support Unicode like Window Explorer, the right-to-left override malware method uses  a RTL Unicode character, that will reverse the order of the characters that follow it. It's used mainly for Middle Eastern/Asian languages that you read right-to-left.

RTLO can be used to spoof fake extensions. To do this we need a hidden RTL Unicode character in the file name.

What is “Right-to-Left Override” RTLO?

The RTLO method is used to hide the true type of a file, so it might trick you into open text file (.txt) which really is a Word file (.docx) with malicious malware. More recently this file could hide a .wav file. Audio files such .wav file are being embedded with malware, is on the forefront of malware maliciousness. Read about that on my post here.

The method exploits a feature built into Windows Explorer. Since Microsoft Windows does a great job of supporting different languages from around the world, some of those languages that are written from right-to-left (RTL). 

Let’s say you want to use a right-to-left written language, like Hebrew or Arabic, on a site combined with a left-to-right written language like English or French. In this case, you would want bidirectional script support.

Bidirectional script support is the capability of a computer system to correctly display bi-directional text. In HTML we can use Unicode right-to-left marks and left-to-right marks to override the HTML bidirectional algorithm when it produces undesirable results:

left-to-right mark: ‎ (U+200E) Unicode character
right-to-left mark: ‏ (U+200F) Unicode character

How do you fix files that have the RTLO or other bad characters ? 

Here's a tool I built to clean up Right-to-Left Mark (and many others) and Unicode Control Characters from your files. It's super fast, small and written in native C++.

Download touchRTL.7z (you need https://www.7-zip.org/ to unpack). Free for personal use, but will open this page.

To get touchPRO.7z, use contact form, Pro has flags to remove Unicode spaces and punctuations (math symbols, currency, open closing braces, and accent marks).  

Just run this command and it will recursively rename filenames to remove those characters under the specified directory name. If directory name, contains spaces you need quotes.

touchRTL -v -l -R "directory name"

where

Usage: touchRTL [-acdhlmprRtuvx] [-r REFFILE | -t TIME | -d DATETIME] FILE...

A FILE argument that does not exist is created empty, unless -c or --no-create
is supplied.

  -a, --access-time        change only the file access time
  -c, --no-create          do not create any new files - If the file exists, touch will update the access time, else will do nothing.
  -l, --RTL                remove Unicode control & format characters (esp. infamous right-to-left) from filename (ditto -u)
  -m, --modif-time         change only the file modification time
  -p, --pause-exit         pause on exit (non-GNU extra)
  -R, --recursive          recursively touch files in specified directory and all subdirectories (non-GNU extra)
  -u, --uni-cntrl-chars    remove Unicode control & format characters (esp. infamous right-to-left) from filename (ditto -l)
  -v, --verbose            output the result of every file processed (non-GNU extra)
  -x, --creation-time      change only the file creation time (non-GNU extra)
  -r, --reference REFFILE  use this file's times instead of current time

  -t, --time TIMESTAMP     use [[CC]YY]MMDDhhmm[.ss] instead of current time
                           Where
                            CC: First two digit of the year
                            YY: Last two digits of the year
                            MM: Month (two-digit numeric month)
                            DD: Day (two-digit numeric day i.e. day of month)
                            hh: Hour
                            mm: Minutes
                            ss: Seconds
                           [] indicates that field is optional

  -d, --date DATETIME      use YYYY-MM-DDThh:mm:ss[.ms] instead of current time (non-GNU, does not parse string)
                           accepted "2033-04-01T07:07:07", "2033-04-01 07:07:07", "2033-04-01 07:07:07.1200"

  -h, --help               Display this help and exit.
  --version                Display version information and exit.

Note: -d and -t options accept different time-date formats.

Copyright © 2019 Metadata Consulting <metadataconsult@gmail.com> - https://metadataconsulting.ca/
Open source by Stephane Duguay <s@binarez.com> - https://www.binarez.com/touch_dot_exe/

Amazon Phishing Email - Fraud Activity Payment - Purchase of "AMAZON PRIME" on Amazon Market Place

For the record, this is an Amazon phishing email attempt that is recently going around and made it through spam filters. What to do?  Report them, goto bottom of page.

From : ​bi​ll​i​ng-​pr​ob​lem@amazon.com​   Subject : RE: [Payment Update] Case ID: #07777777 - Fraud Activity Payment - Purchase of "AMAZON PRIME" on Amazon Market Place on 22 October 2019 / Pago de actividad fraudulenta: compra de "AMAZON PRIME" en Amazon Market Place Reply-to : ​bi​ll​i​ng-​pr​ob​lem@amazon.com​ Amazon-Statement-ID-#07777777.docx Here's  a preview of text in Word Document

Dear Customer, 
 

We believe that an unauthorized party may have accessed your account. To protect your information, we have:  
 

Disabled the password to your account. 

Reversed any modifications made by this party. 

Canceled any pending orders. You can ignore any confirmation emails that you received for these orders. 

If appropriate, refunded purchases to your payment instrument. 

Restored any gift card balance that may have been used. It may take 2 to 3 days for the gift card balance to be available in your account. 
 

Just log in https://amazon.com and follow the instructions in your account notifications to see what information you need to provide. Please send the missing information by 24 October 2019. 
 
 
Sincerely,  
 
 
Amazon.com  
========================= 

SPAM/ PHISHING LINKs;  

1. https://t.umblr.com/redirect?z=https%3A%2F%2Famazon-customer-service.accountunderreview.com%2F%3Famalsoleh&t=OsdfsfasdfasdfasfH:DSFDKJFHQ%3D%3D&b=t%)*(&)(DFDFDF-SDFDFD&p=https%3A%2F%2Frakaap182713.tumblr.com%2Fpost%2sdfdf9090892%2Fhttpsamazon-customer-serviceaccountunderreview&m=1

How to tell this is a Phishing email ?

1. Check email address in full, if it's not from originating company then it's phishing.
2. Hover over all links in email, if it's not from the amazon.com site then forget it.
   
   
3. The best way is to look at message source, see below.

How to examine Email Message Source ?

Now lets look at message source

1. Outlook.com->Actions->View Message Source. 
2. Gmail.com->More (down arrow to top right)->Show original.

Check for suspicious links, anything that does not originate from apple.com.

Report Phishing Email (not as Spam)

1. Outlook.com->Junk (at Top)->Phishing Scam
2. Gmail.com->More (downarrow to top right)->Report Phishing 

Report Phishing URLs at Google now 

If you have recievied this email take further action now by click these links

1. https://www.google.com/safebrowsing/report_phish/

Report phishing at Microsoft and government agencies

1. http://www.microsoft.com/security/online-privacy/phishing-faq.aspx

Report phishing emails to Amazon 

Send the e-mail to stop-spoofing@amazon.com

Note: Sending this suspicious e-mail as an attachment is the best way for us to track it.