Around 70 percent of all Microsoft patches were fixes for memory safety bugs over the last 12 years

Why are malware viruses so wide spread?

This is the perennial question, that will not die. Here's one major reason why.

Microsoft security engineer Matt Miller from Microsoft Security Response Center gave a presentation at security conference stated it very succinctly in his slides, that over the last 12 years, around 70 percent of all Microsoft patches were fixes for memory safety bugs (see slide 10). 

The reason for this high percentage is because Windows has been written mostly in C and C++, two "memory-unsafe" programming languages that allow developers fine-grained control of the memory addresses where their code can be executed. 

Memory-unsafe bugs happen when software, accidentally or intentionally, accesses system memory in a way that exceeds its allocated size and memory addresses, accessing other parts of the system to gain elevated privileges or impregnate custom malware code into that adjacent memory space.


Memory-unsafe bugs have similar terms such as buffer overflow, race condition, page fault, null pointer, stack exhaustion, heap exhaustion/corruption, use after free, or double free --all describe memory safety vulnerabilities. 

There is a tension for whom the responsibility resides with, the language/compiler or the operating system. 

The C philosophy is always trust the programmer. And also not checking bounds allows a C program to run faster. The problem is that C/C++ doesn't actually do any boundary checking with regards to arrays. It depends on the OS to ensure that you are accessing valid memory.

There is only one solution and that is to re-write the entire Windows Operating system in a memory-safe language like Rust. That is starting to be addressed. 

Here's a brief review of one unsafe memory bug,  image courtesy of stack overflow.

Source Slides : https://github.com/Microsoft/MSRC-Security-Research/blob/master/presentations/2019_02_BlueHatIL/2019_01%20-%20BlueHatIL%20-%20Trends%2C%20challenge%2C%20and%20shifts%20in%20software%20vulnerability%20mitigation.pdf

Stack Overflow Attack Source Slides
https://www.slideshare.net/gumption/buffer-overflow-attacks-7024353

What files to clean for Windows 10 1809 Disk Cleanup Categories

Here's a list of files to clean-up your Windows 10 Version 1809 (October 2018 Update) using Disk Cleanup. Disk Cleanup has many new file deletion categories, such as Windows Defender files, see below.

However, Microsoft has a Metro version of Disk Cleanup up called Storage Sense which is a simplified interface of Disk Cleanup. It amalgamates some of the categories below, but retains many.

Storage Sense

1. Open Settings -> Click on System -> Click on Storage
2. Under "Storage Sense" click the Free up space now option, to list same categories as below.

How to Use

Disk Cleanup wizard detects outdated files that can be delete, so if the category appears you can select those files to be deleted. By category, I mean potential files to be deleted under the "Files to delete:" heading in the bordered box below. 

These potential files to be deleted will be enumerated each time you run Disk Cleanup, so the categories will change each time you run it and and be different on other computers.

Click "View Files" button to examine the files that are to be deleted in each category.

Click "Clean up system files" button to to analyze the selected drive and display what Windows system can be cleaned up. A progress bar is shown during this process. Wait for this to finish.

When done, Disk Cleanup shows the total amount of space that can be freed up. Then, in the 'Files to delete' section you see different types of files that can be deleted. 

This will include categories such as 'Downloaded Program Files', 'Recycle Bin', 'System error' files, 'Temporary files' and others. For each category of items, you see how much space they occupy at the moment. 

Extra : Click 'Clean up system files' button (to save allot of gigs of space) - see below.

Detailed Explanation of all 'Files to Delete' Categories:

Category	Description
Compress System Disk	WARNING : THIS COMPRESSES YOUR DISK, it does not clean intermediate compressed files. THIS SLOWS YOUR FILE SYSTEM DOWN. Consult with an admin before using or extensively research this option before using. Turning this off has caused issues before. It's a one way street generally. Save storage space by allowing Windows to compress the  contents of your drive . You will be able to access your files as normal , and the compression will be automatic and transparent to you . Depending on how much into is on your drive, compression  may take a while. If you need to decompress the drive or any  Folders at a later date you can do so with Windows File Explorer. @C:\Program Files\rempl\strgsnsaddons.dll,-1009
BranchCache	Files created by BranchCache service for caching data.   BranchCache is a wide area network (WAN) bandwidth optimization technology that is included in some editions of the Windows Server 2012 and Windows 8 operating systems, as well as in some editions of Windows Server 2008 R2 and Windows 7. To optimize WAN bandwidth when users access content on remote servers, BranchCache copies content from your main office or hosted cloud content servers and caches the content at branch office locations, allowing client computers at branch offices to access the content locally rather than over the WAN.
File History Files	File History saves copies of your files so you can get them back if they're lost or damaged. It automatically backs up files in the background and lets you restore them from a simple, time-based interface.
DirectX Shader Cache	Clean up files created by the graphics system which can speed up application load time and improve responsiveness. They will be re-generated as needed.
Diagnostic Data Viewer	In 2018 Microsoft released the Diagnostic Data Viewer (DDV) which is a tool that lets you review the raw diagnostic data Windows is sending to Microsoft. Now you can also view Office diagnostic data using the same viewer.
Downloads	Warning: These are files in your personal Downloads folder. It's important to understand that the "Downloads" option is unchecked by default and it's a helpful feature to those that use downloads folder for temporary files. On the other hand, if you use it as a place to store needed files, make sure the option is unchecked or you will lose the content stored in the folder. Folder:C:\Users\{UserName}\Downloads File List:*.* @C:\WINDOWS\System32\DATACLEN.DLL, -1045
Hello Face Remove	Warning: Using this option seems to disable Windows Hello https://h30434.www3.hp.com/t5/Notebook-Operating-System-and-Recovery/Hello-Face-stopped-working-after-a-disk-cleanup/td-p/7028729 When you uninstall the optional Windows Hello Face component. Signing in to Windows using facial recognition might not work. You'll still be able to log in with your Windows Hello PIN or a username and password combination. To tum this back on in the  future, go to Settings, search for Add an optional feature, select  Add a feature, and then select Windows Hello Face. @C:\Program Files\rempl\strgsnsaddons.dll,-1029
Language Pack Files	Remove unused language resource files, including keyboards, speech inputs, etc.
Mixed Reality	Removes Mixed Reality viewer files. @C:\Program Files\rempl\strgsnsaddons.dll,-1023
OneDrive File Remove	Removes temporary Onedrive offline files in Onedrive folders marked as "online-only" to free up disk space. Note: You can always explicitly set a folder as 'Always keep offline' (even if you haven't used those files in a long time) and it won't affect those folders. @C:\Program Files\rempl\strgsnsaddons.dll,-1013
Downloaded Files	Duplicate of Downloads option @C:\Program Files\rempl\strgsnsaddons.dll,-101
Delete all System Restore Points	This will deleting all previous restore points. Restore points allow you to go back to a previous install state. It can be risky to remove all restore points. If you feel confident, and need the space. Then after option, create a restore point afterwards immediately. Note: You can also save space by deleting all previous restore points except for the last one. To do that, run Disk Cleanup and after it scans your drive, select the More Options tab. Then under the System Restore and Shadow Copies section click Clean up and then the verification message
Language Pack	Remove unused language resource files, including keyboards, speech inputs, etc.
Old ChkDsk Files	Check Disk (chkdsk) is a command line that you use to recover files from your hard disk, generally caused by surface errors due to aging, bumping and smoke, that cause bad sectors (lost data) to appear. Chkdsk recovers what it can of these sectors in files which was written over the bad sector into files ending in .CHK. You can open and read the contents using Notepad or even better Notepad++. Often the contents are not worth keeping but they can be. Chkdsk files are indicative that a drive is starting to fail. If new bad sectors continue to appear you should replace the drive or you risk losing all your files. "FileList"="*.CHK" "Folder"="?:\\FOUND.000|?:\\FOUND.001|?:\\FOUND.002|?:\\FOUND.003|?:\\FOUND.004|?:\\FOUND.005| ?:\\FOUND.006|?:\\FOUND.007|?:\\FOUND.008|?:\\FOUND.009"
Delivery Optimization Files	Delivery optimization files are files that were previously downloaded to your computer and can be deleted if currently unused by the Delivery Optimization service. Windows Update Delivery Optimization lets you get Windows updates and Windows Store apps from sources in addition to Microsoft. This can help you get updates and apps more quickly if you have a limited or unreliable Internet connection. And if you own more than one PC, it can reduce the amount of Internet bandwidth needed to keep all of your PCs up-to-date. Delivery Optimization also sends updates and apps from your PC to other PCs on your local network or PCs on the Internet. @C:\WINDOWS\system32\domgmt.dll
Content Indexer Cleaner	The Windows search indexer is constantly running in the background to make file searches as quick as possible. "Folder"="?:\\Catalog.wci" File List:*.*
Temporary Setup Files	These files should no longer be needed. They were originally created by a setup program that is no longer running. Located in directory C:\\Windows\\msdownld.tmp|?:\\msdownld.tmp "FileList"="*.tmp" @C:\WINDOWS\system32\setupcln.dll, -1001
Download Program Files	Downloaded Program Files are ActiveX controls and Java applets downloaded automatically from the Internet with you view certain pages. They are temporarily stored in the Downloaded Program Files folder on your hard disk. @C:\Windows\System32\occache.dll,- 1071
Temporary Internet Files	The Temporary Internet Files folder contains webpages stored on your hard disk for quick viewing. Your personalized settings for webpages will be left intact.
Offline Web Pages	Offline pages are webpages that are stored on your computer so you can view them without being connected to the Internet. If you delete these pages now, you can still view your favorites offline later by synchronizing then. Your personalized settings for webpages will be left intact.
Debug Dump Files	Files created by Windows.
RetailDemo Offline Content	Windows 10 includes a Retail Demo experience mode. This feature basically is useful for and meant for retail store staff who want to demo Windows 10 to customers.
Recycle Bin	The Recycle Bin contains files you have deleted from your computer.
Setup Log Files	Files created by Windows. "FileList"="setup*.log|setup*.old|setuplog.txt|winnt32.log" "Folder"="%WINDIR%"
System Error Memory Dump Files	Remove system error memory dump files.
System Error Minidump Files	Remove system error minidump files.
Temporary Files	Programs sometimes stores temporary information in the TEMP folder. Before a program closes, it usually deleted this information. You can safely delete temporary files that have not been modified in over a week. Folder:C:\Users\Markus\AppData \Local\Temp|C:\WINDOWS\Temp|C:\WINDOWS\Logs|C:\WINDOWS\System32\LogFiles File List:*.*
Temporary Sync Files	Remove Windows Media Sync files. You can use Windows Media Player to copy music, videos, and pictures from your Player Library to a portable device, such as a compatible MP3 player. This process is called syncing. These are temp files in caches create in this process which are delete
Thumbnails	Windows keeps a copy of all your picture, video, and document thumbnails so they can be displayed quickly when you open a folder. If you delete these thumbnails, they will be automatically recreated as needed.
User File Versions	Windows stores file versions temporarily on this disk before copying them to the designated File History disk. If you delete these files, you will lose some file history. File History is Windows 10’s main backup tool, originally introduced in Windows 8. Despite the name, File History isn’t just a way to restore previous versions of files–it’s a fully-featured backup tool, it's suppose to be a clone of Apple Time Machine.
Windows Error Report Files (4 types)	Files used for Windows Error Reporting (WER). These are logs of errors (program crashes mostly) that were reported to Microsoft by the Windows Error Reporting service.

'Clean up system files' button you see the following dialog box

This will include categories such as'Windows ESD Installation Files', 'Windows Defender', and others. 

Windows Update Cleanup only appears in the list when the Disk Cleanup wizard detects Windows updates that you don't need on your system.  This category will generally save you the greatest amount of space. All of these are okay to delete, that is the purpose of this wizard. 

Recommendation:  Carefully select categories to delete all the files, but review the categories below for further details, some have irreversible effects.

Detailed Explanation of 'Clean up system files' categories

Category	Description
Windows ESD installation Files (since Win 10)	You will need these files to Reset or Refresh your PC. Windows ESD Files was introduced to upgrade to Windows 10, behind the scenes. Windows ESD Files are files used for a upgrade to a new version of Windows. ESD stands for Electronic Software Delivery and delivers files in an encypted (.esd) format. This then contains a .wim file. A Windows IMage (.wim) file contains one or more compressed Windows images. Each Windows image in a .wim file contains a list of all of the components, settings, and packages available with that Windows image. Install.wim file in its turn contains everything needed for a complete Windows installation. You can convert the Windows 10 .esd file to make your own ISO disk to upgrade any PC later!
Temporary Windows installation files	Installation files used by Windows setup. These files are left over from the installation process and can be safely deleted.
Previous Windows installation(s)	Files from a previous Windows installation. Files and folders that may conflict with the installation of Windows have been moved to folders named Windows.old. You can access data from the previous Windows installations in this folder. @C:\WINDOWS\system32\setupcln.dll,-1002
Update package Backup Files	Windows saves old versions of files that have been updated by an Update package. If you delete the files, you won't be able to uninstall the Update package later.
Windows Update Cleanup	Windows keeps copies of all installed updates from Windows Update, even after installing newer versions of updates that are no longer needed and taking up space. (You might need to restart your computer.) @C:\WINDOWS\system32\scavengeui.dll,-1002
Device driver packages	Windows keeps copies of all previously installed device driver packages    from Windows Update and other sources even after installing newer versions of drivers. This task will remove older versions of drivers that are no longer needed. The most current version of each driver package will be kept. @C:\WINDOWS\system32\pnpclean.dll, -102
Windows Defender Antivirus	Non critical files used by Windows Defender Antivirus All files in these locations will be deletedFolder:C:\ProgramData\Microsoft\Windows Defender\LocalCopy|C:\ProgramData\Microsoft\Windows Defender\Support File List:*.*@C:\Program Files\WindowsDefender\MpAsDesc.dll,-380
Files Discarded by Windows Upgrade	Files from a previous Windows installation. As a precaution, Windows upgrade keeps a copy of any files that were not moved to the new version of Windows and were not identified as Windows system files. If you are sure that no user's personal files are missing after the upgrade, you can delete these files. @C:\WINDOWS\system32\setupcln.dll, -1005 Setup Directories:$WINDOWS.~Q;$INPLACE.~TR;$Windows.~LS
Windows Upgrade Log Files	Windows upgrade log files contain information that can help identify and    troubleshoot problems that occur during Windows installation, upgrade, or servicing. Deleting these files can make it difficult to troubleshoot installation issues. @C:\WINDOWS\System32\DATACLEN.DLL,-1010Folder:C:\WINDOWS File List:setup*.log|setup*.old|setuplog.txt|winnt32.log
Service Pack Backup Files	Windows saves old versions of files that have been updated by a service pack. If you delete the files, you won't be able to uninstall the service pack later. @C:\WINDOWS\system32\scavengeui.dll,-1000

When you click OK, Disk Cleanup will prompt you to confirm that you want to permanently delete the selected files. 

If you have never done, you'll be surprised at the Gigs of space freed up. 

How to delta/incrementally copy folders with many files with an integrity check on Windows

rsync is a utility for efficiently transferring and synchronizing files between a computer and an external hard drive and across networked computers by comparing the modification times and sizes of files. It is commonly found on Unix-like operating systems. The rsync algorithm is a type of delta encoding, and is used for minimizing network usage. Zlib may be used for additional data compression, and SSH or stunnel can be used for security.

Rsync is typically used for synchronizing files and directories between two different systems. For example, if the command rsync local-file user@remote-host:remote-file is run, rsync will use SSH to connect as user to remote-host. Once connected, it will invoke the remote host's rsync and then the two programs will determine what parts of the local file need to be transferred so that the remote file matches the local one.

Great overview of technical working of rsync
http://tutorials.jenkov.com/rsync/overview.html

Get Rsync for Windows, open source code project. It's a self contained exe.
https://www.itefix.net/cwrsync

For an easy to use similar project with GUI use SyncToy 2.1 free from Microsoft. 

https://www.microsoft.com/en-us/download/details.aspx?id=15155




Sample command 

$ rsync -av Documents/* /tmp/documents

In the command above, the option:

1. -a – means archive mode
2. -v – means verbose, showing details of ongoing operations

All Cmd Line Options

rsync  version 3.1.3  protocol version 31
Copyright (C) 1996-2018 by Andrew Tridgell, Wayne Davison, and others.
Web site: http://rsync.samba.org/
Capabilities:
    64-bit files, 64-bit inums, 32-bit timestamps, 64-bit long ints,
    socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,
    append, no ACLs, no xattrs, iconv, symtimes, prealloc

rsync comes with ABSOLUTELY NO WARRANTY.  This is free software, and you
are welcome to redistribute it under certain conditions.  See the GNU
General Public Licence for details.

rsync is a file transfer program capable of efficient remote update
via a fast differencing algorithm.

Usage: rsync [OPTION]... SRC [SRC]... DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
  or   rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
  or   rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
  or   rsync [OPTION]... [USER@]HOST:SRC [DEST]
  or   rsync [OPTION]... [USER@]HOST::SRC [DEST]
  or   rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.

Options
 -v, --verbose               increase verbosity
     --info=FLAGS            fine-grained informational verbosity
     --debug=FLAGS           fine-grained debug verbosity
     --msgs2stderr           special output handling for debugging
 -q, --quiet                 suppress non-error messages
     --no-motd               suppress daemon-mode MOTD (see manpage caveat)
 -c, --checksum              skip based on checksum, not mod-time & size
 -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
     --no-OPTION             turn off an implied OPTION (e.g. --no-D)
 -r, --recursive             recurse into directories
 -R, --relative              use relative path names
     --no-implied-dirs       don't send implied dirs with --relative
 -b, --backup                make backups (see --suffix & --backup-dir)
     --backup-dir=DIR        make backups into hierarchy based in DIR
     --suffix=SUFFIX         set backup suffix (default ~ w/o --backup-dir)
 -u, --update                skip files that are newer on the receiver
     --inplace               update destination files in-place (SEE MAN PAGE)
     --append                append data onto shorter files
     --append-verify         like --append, but with old data in file checksum
 -d, --dirs                  transfer directories without recursing
 -l, --links                 copy symlinks as symlinks
 -L, --copy-links            transform symlink into referent file/dir
     --copy-unsafe-links     only "unsafe" symlinks are transformed
     --safe-links            ignore symlinks that point outside the source tree
     --munge-links           munge symlinks to make them safer (but unusable)
 -k, --copy-dirlinks         transform symlink to a dir into referent dir
 -K, --keep-dirlinks         treat symlinked dir on receiver as dir
 -H, --hard-links            preserve hard links
 -p, --perms                 preserve permissions
 -E, --executability         preserve the file's executability
     --chmod=CHMOD           affect file and/or directory permissions
 -o, --owner                 preserve owner (super-user only)
 -g, --group                 preserve group
     --devices               preserve device files (super-user only)
     --specials              preserve special files
 -D                          same as --devices --specials
 -t, --times                 preserve modification times
 -O, --omit-dir-times        omit directories from --times
 -J, --omit-link-times       omit symlinks from --times
     --super                 receiver attempts super-user activities
 -S, --sparse                turn sequences of nulls into sparse blocks
     --preallocate           allocate dest files before writing them
 -n, --dry-run               perform a trial run with no changes made
 -W, --whole-file            copy files whole (without delta-xfer algorithm)
     --checksum-choice=STR   choose the checksum algorithms
 -x, --one-file-system       don't cross filesystem boundaries
 -B, --block-size=SIZE       force a fixed checksum block-size
 -e, --rsh=COMMAND           specify the remote shell to use
     --rsync-path=PROGRAM    specify the rsync to run on the remote machine
     --existing              skip creating new files on receiver
     --ignore-existing       skip updating files that already exist on receiver
     --remove-source-files   sender removes synchronized files (non-dirs)
     --del                   an alias for --delete-during
     --delete                delete extraneous files from destination dirs
     --delete-before         receiver deletes before transfer, not during
     --delete-during         receiver deletes during the transfer
     --delete-delay          find deletions during, delete after
     --delete-after          receiver deletes after transfer, not during
     --delete-excluded       also delete excluded files from destination dirs
     --ignore-missing-args   ignore missing source args without error
     --delete-missing-args   delete missing source args from destination
     --ignore-errors         delete even if there are I/O errors
     --force                 force deletion of directories even if not empty
     --max-delete=NUM        don't delete more than NUM files
     --max-size=SIZE         don't transfer any file larger than SIZE
     --min-size=SIZE         don't transfer any file smaller than SIZE
     --partial               keep partially transferred files
     --partial-dir=DIR       put a partially transferred file into DIR
     --delay-updates         put all updated files into place at transfer's end
 -m, --prune-empty-dirs      prune empty directory chains from the file-list
     --numeric-ids           don't map uid/gid values by user/group name
     --usermap=STRING        custom username mapping
     --groupmap=STRING       custom groupname mapping
     --chown=USER:GROUP      simple username/groupname mapping
     --timeout=SECONDS       set I/O timeout in seconds
     --contimeout=SECONDS    set daemon connection timeout in seconds
 -I, --ignore-times          don't skip files that match in size and mod-time
 -M, --remote-option=OPTION  send OPTION to the remote side only
     --size-only             skip files that match in size
 -@, --modify-window=NUM     set the accuracy for mod-time comparisons
 -T, --temp-dir=DIR          create temporary files in directory DIR
 -y, --fuzzy                 find similar file for basis if no dest file
     --compare-dest=DIR      also compare destination files relative to DIR
     --copy-dest=DIR         ... and include copies of unchanged files
     --link-dest=DIR         hardlink to files in DIR when unchanged
 -z, --compress              compress file data during the transfer
     --compress-level=NUM    explicitly set compression level
     --skip-compress=LIST    skip compressing files with a suffix in LIST
 -C, --cvs-exclude           auto-ignore files the same way CVS does
 -f, --filter=RULE           add a file-filtering RULE
 -F                          same as --filter='dir-merge /.rsync-filter'
                             repeated: --filter='- .rsync-filter'
     --exclude=PATTERN       exclude files matching PATTERN
     --exclude-from=FILE     read exclude patterns from FILE
     --include=PATTERN       don't exclude files matching PATTERN
     --include-from=FILE     read include patterns from FILE
     --files-from=FILE       read list of source-file names from FILE
 -0, --from0                 all *-from/filter files are delimited by 0s
 -s, --protect-args          no space-splitting; only wildcard special-chars
     --address=ADDRESS       bind address for outgoing socket to daemon
     --port=PORT             specify double-colon alternate port number
     --sockopts=OPTIONS      specify custom TCP options
     --blocking-io           use blocking I/O for the remote shell
     --stats                 give some file-transfer stats
 -8, --8-bit-output          leave high-bit chars unescaped in output
 -h, --human-readable        output numbers in a human-readable format
     --progress              show progress during transfer
 -P                          same as --partial --progress
 -i, --itemize-changes       output a change-summary for all updates
     --out-format=FORMAT     output updates using the specified FORMAT
     --log-file=FILE         log what we're doing to the specified FILE
     --log-file-format=FMT   log updates using the specified FMT
     --password-file=FILE    read daemon-access password from FILE
     --list-only             list the files instead of copying them
     --bwlimit=RATE          limit socket I/O bandwidth
     --outbuf=N|L|B          set output buffering to None, Line, or Block
     --write-batch=FILE      write a batched update to FILE
     --only-write-batch=FILE like --write-batch but w/o updating destination
     --read-batch=FILE       read a batched update from FILE
     --protocol=NUM          force an older protocol version to be used
     --iconv=CONVERT_SPEC    request charset conversion of filenames
     --checksum-seed=NUM     set block/file checksum seed (advanced)
 -4, --ipv4                  prefer IPv4
 -6, --ipv6                  prefer IPv6
     --version               print version number
(-h) --help                  show this help (-h is --help only if used alone)

Use "rsync --daemon --help" to see the daemon-mode command-line options.
Please see the rsync(1) and rsyncd.conf(5) man pages for full documentation.
See http://rsync.samba.org/ for updates, bug reports, and answers