Sunday, July 31, 2016

Outlook Hotmail stopped syncing, missing folders

Microsoft is migrating all Outlook.com mailboxes to new Office 365 Exchange Online mail servers, accounts will be moved starting Q1 2016 in the USA. The migration of other countries is targeted to be completed by the end of 2016. 


How can you tell your mailbox has been migrated?
















What are the negative affects of this migration? 



  1. First sign you have migrated is your Outlook Hotmail Connector Version 14.0.6123.5001 will stop syncing. They have ditched support for the Hotmail Connector, which implies no more PST files, now you are forced to use a OST file. OST files do not contain all you emails, its just a cache so you cannot back up all your full emails anymore. You cannot copy a OST file and expect a full back-up.
  2. Disappearing or Missing Folder in Outlook

    During the migration, none of the folders are being replicated.
    I lost all my folder structures and rules. 




Office 365 Exchange Online now uses OST vs PST file? 


There are major differences between OST and PST containers for you emails. PST files are used to download all your emails so you can make a full back-up, OST is only a recent used cache of emails. You can read full details here.

But to summarize the primary differences between the two types of Outlook data files are as follows:

Outlook Data Files (.pst) are used for POP3, IMAP, and web-based mail accounts. When you want to create archives or back up your Outlook folders and items on your computer you can use (.pst) files.

Your items can also be moved or archived to an Outlook Data File (.pst). Because a .pst file is kept on your computer, it is not subject to mailbox size limits on the mail server. By moving items to a .pst file on your computer, you can free up storage space in the mailbox on your mail server. 


You had the potential to move all you emails into (.pst) file and make a full back-up of your emails. 


Outlook Data Files (.ost) are used when you have an Exchange account and want to work offline or use or use the default Cached Exchange Mode. Outlook Data Files (.ost) are always copies of items that are saved on a mail server and don't have all of your emails downloaded like a Outlook Data Files (.pst), but only recent items are cached to facilitate synchronization.  

When you work offline, folders that are synchronized are determined by Send/Receive groups. By using Send/Receive groups, you can choose which folders are synchronized and kept current so that when a connection to the server is not possible or you choose to work offline, you can continue to work with those items. You can also specify that updates to the Address Book be downloaded during synchronization.


You can configure Outlook to keep a local copy of your items on your computer in an Outlook data file that is named an offline Outlook Data File (.ost). This allows you to use Cached Exchange Mode or to work offline when a connection to the Exchange computer may not be possible or wanted. 

With Cached Exchange Mode, you do not have to set up Send/Receive groups, choose folders that you want to be available offline, and then keep those folders synchronized. 

The .ost file is synchronized with the items on the server that runs Exchange. Because your data remains on the Exchange server, you can re-create this .ost file on your new computer without having to back up the .ost file.


How to back-up all your emails from an OST file or Exchange account ?



  1. Open Outlook, 
  2. Export the OST file to a PST file using the Outlook export wizard making sure that you select the of the folder tree as the starting point along with ensuring to that "Include Subfolders" is checked. 
  3. When you're done, open the PST file you just created in Outlook to ensure that everything was properly done and all the data you're expecting to be there - is there. 
  4. When done checking the data, right click on the of the PST file you just added back --> select "Close".








Saturday, July 30, 2016

RegtoText - NEW command-line executable converts Windows Registry file to human readable text

RegtoText is a command line utility that converts a Windows Registry exported file (.reg) into a human readable text (.txt) file. Hex numbers are converted into ASCII characters when possible. Conversion can be challenging since registry key can accept any binary format, so heuristic and probabilistic methods are used to decode 
values when possible to ASCII.



💀 WARNING: 
RegtoText is a paid utility and an old out-of-date limited demo was released to www.softpedia.com site only. Any other sites and the "cracked version" will contain malware under the hood. 

Jan 2020 - Now has a separate website at - http://regtotext.blogspot.com/


This tool target as forensic (FBI,CIA, Antivirus Co)/management/educational tool to quickly search and eyeball the entire registry file for encoded values that are suspicious. Registry keys could hold persistant malware signatures (like Poweliks), back-doors or simply hidden secret messages. Most commonly, keys contain   foreign languages encodings that can be spotted more efficiently with human eyes. Furthermore, once this file is decoded using RegtoText, it is searchable as a human readable text file and can be indexed in any internal forensic exploit search engine/database.















RegtoText converts a Registry exported file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
Windows Registry Editor Version 5.00

HKEY_CURRENT_USER\Control Panel\Appearance\New Schemes\4\Sizes\0]
"DisplayName"="@themeui.dll,-2019"
"Flat Menus"=dword:00000000
"Font #0"=hex:f5,ff,ff,ff,00,00,00,00,00,00,00,00,00,00,00,00,bc,02,00,00,00,\
  00,00,00,00,00,00,00,54,00,61,00,68,00,6f,00,6d,00,61,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,\
  00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00,00
"Size #0"=hex(b):01,00,00,00,00,00,00,00
"Size #9"=hex(b):00,00,00,00,00,00,00,00
"Color #0"=dword:00c8d0d4
"LegacyName"="@themeui.dll,-854"

into a human readable text file either UTF-8 or ASCII (for ASCII ?? and odd characters are removed)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
RegtoText Windows Registry Conversion Version 5.00 (sample UTF-8 export)

[HKEY_CURRENT_USER\Control Panel\Appearance\New Schemes\4\Sizes\0]
"DisplayName"="@themeui.dll,-2019"
"Flat Menus"=[REG_DWORD] 0
"Font #0"=[REG_BINARY] ??      '     Tahoma                         
"Size #0"=[REG_QWORD] 1
"Size #9"=[REG_QWORD] 0
"Color #0"=[REG_DWORD] 13160660
"LegacyName"="@themeui.dll,-854"


REGTOTEXT HELP


.FILENAME
RegToText.exe 

.VERSION

Version 2.2016.214.373 (Update Jan 2018)

.SYNOPSIS 
Parses a valid Windows registry exported file (.reg) and translates indecipherable hex and decimal values into a human readable text file.

.PURPOSE
The aim of this command-line executable is to make a human readable registry file. This greatly aids in searching and understanding the Windows Registry, key for developers.

.DESCRIPTION 
RegToText windows console application deciphers unreadable portions of registry file to text. Firstly, it checks for a valid Windows registry file ending with file extension (.reg). Then it validates file header for ""Windows Registry Editor Version 5.00"" for Windows 2000, ME, XP,7, Vista, 8, 8.1, 10+, Server 2003+ or ""REGEDIT4"" for Windows 98, NT 4.0 and Server 2000-. Passing this, the process will begin to translates all the hexadecimal and decimal values into output Unicode text file. Output is written out in 250 line chunks. Upon premature or cancellation, output file will contain up-to the last chunk written out. Output encoding can be UTF-8 or ASCII. Output encoding can be UTF-8 or ASCII. All non-printable characters are cleansed for ASCII, read ENCODING notes for details. Encoding choice can drastically affect file output size. 

All the registry types are translated listed as follows;


"value" alias hex(1) Default or blank String value data with escape characters
hex alias hex(3) REG_BINARY  Binary data (any arbitrary data)
dword alias hex(4)
REG_DWORD / REG_DWORD_LITTLE_ENDIAN
A 32-bit unsigned integer coded in little-endian format
hex(0) REG_NONE No type (the stored value, if any)
hex(1) REG_SZ A string value, normally stored and exposed in UTF-16LE (when using the Unicode version of Win32 API functions), usually terminated by a NUL character
hex(2) REG_EXPAND_SZ An “expandable” string value that can contain environment variables, normally stored and exposed in UTF-16LE, usually terminated by a NUL character
hex(3) REG_BINARY Binary data (any arbitrary data) including variable byte encoded UTF8 values
hex(4)
REG_DWORD_LITTLE_ENDIAN equivalent to REG_DWORD 
A 32-bit unsigned integer coded in little-endian format
hex(5) REG_DWORD_BIG_ENDIAN A 32-bit unsigned integer coded in big-endian format
hex(6) REG_LINK A symbolic link (UNICODE) to another Registry key, specifying a root key and the path to the target key
hex(7) REG_MULTI_SZ A multi-string value, which is an ordered list of non-empty strings, normally stored and exposed in UTF-16LE, each one terminated by a NUL character, the list being normally terminated by a second NUL character.
hex(8) REG_RESOURCE_LIST A resource list
hex(9) REG_FULL_RESOURCE_DESCRIPTOR A resource descriptor
hex(a) REG_RESOURCE_REQUIREMENTS_LIST A resource requirements list
hex(b) REG_QWORD_LITTLE_ENDIAN equivalent to REG_QWORD 
A 64-bit integer little-endian (introduced in Windows XP)

.LIMITATIONS

  1. Does not decode Darwin Descriptors (DD), perhaps in future enterprise version. Vote for it in comments section.(DD is basically an obfuscated GUID a very basic tool is available for download here)
  2. Does not unpack packed GUIDs, perhaps in future enterprise version. Vote for it in comments section.(here's a Powershell script to available for download here)
  3. Does not ROT-13 decode the UserAssist key (HCU\Software\Microsoft\Windows\CurrentVersion \Explorer\UserAssist). Vote for it in comments section.(decode at http://rot13.com/)
  4. Does not perform language encoding translations. Vote for it in comments section.
  5. REG_RESOURCE_LIST, REG_FULL_RESOURCE_DESCRIPTOR,REG_RESOURCE_REQUIREMENTS_LIST are dumped. Coming soon, properly decodes REG_RESOURCE_LIST data structure from registry which is based on CM_RESOURCE_LIST structure that is a list that contains many CM_FULL_RESOURCE_DESCRIPTORs.

    Update Aug 2019
    - Separate tool to decrypt 
    REG_RESOURCE_LIST available.

.REQUIREMENTS
32-bit app which requires .NET Framework 4 Client Profile.

.64 BIT REGISTRY KEYS

The registry in 64-bit versions of Windows is divided into 32-bit and 64-bit keys. Many of the 32-bit keys have the same names as their 64-bit counterparts, and vice versa.
To open registry 64bit version run %systemroot%\syswow64\regedit from CMD as Admin.

Refer to https://support.microsoft.com/en-ca/kb/305097 to extract 64-bit keys to .reg file. RegtoText decodes 64-bit keys as 32-bit keys. There is no difference.

.HIDDEN REGISTRY KEYS - SECURITY & SAM
Some of the security and core system related keys are hidden from user even when part of an administrator group cannot see these special keys.

Here are some of the such hidden registry keys

HKEY_LOCAL_MACHINE\SECURITY
HKEY_LOCAL_MACHINE\SAM

SECURITY registry key stores all the system policy and LSA secrets related information.  SAM registry key has details for user accounts along with LM/NTLM password hashes for each user.

There are many ways we can view these hidden registry keys. We can use psexec.exe tool (part of pstools package from sysinternals) to launch the regedit.exe as system account as shown below.


psexec.exe -s -i regedit.exe

.ENCODING
Characters outside the ASCII or UTF8 range are stripped. NON-PRINTABLE less than decimal 31 are stripped for ASCII encoding. UTF8 preserves more of the original source content, but a cost of larger output file size. More importantly, UTF8 encoding will pass allot of unreadable characters and non-printable characters that may cause issues when scrolling large files in text editors. ASCII allows for maximum readability with intention of one line per registry key and side benifit of space savings. Large files over 1G benefit tremendously when loading ACSII text editors for scrolling and searching.

.TEXT EDITORS

Notepad and Notepad++ will not load 1 G+ files. Textpad (memory lim), Notepad Light (upto 2G) and UltraEdit (claims 2^64-1G) will load file over 1G+ files.

.PERFORMANCE 
Tested on 4.25M rows in 18 mins, 24 secs. Processing 921,572 subkeys and 2,344,590 key/value pairs.

.USAGE
RegtoText.exe [/h] [/v] [/s] inputfile.reg [/o:filename.txt] [/e:{UT8F|ASCII}]

.ARGUMENTS
[drive:][path]inputfile.reg            1st argument required
                                       Input registry file. If path omitted, default to current path. 
.FLAGS
(order not important)
/h|/help                               Help
/v|/version                            Version
/s|/silent                             Silent
/l|/license                            License
/e|/encoding:{UTF8|ASCII}              Output encoding. If omitted, default value:'UTF8'.

/o|/output:[drive:][path]filename.txt  Output text file. If omitted, default value:'inputfile.txt

.INPUT
Must be valid exported registry file from REGEDIT.exe ending in .reg

.OUTPUT
Creates a Unicode text file ending in .txt extension. If exists prompt to delete ? No, creates a timestamped file. Hexadecimal and decimal values are decoded using according /e flag.

.EXAMPLE 
regtotext c:\Users\MDC\Documents\myfullregistryBCK.reg /e:ASCII

.AUTHOR 
metadataconsult@gmail.com (Metadata Consulting, ON, CDN) July 30, 2016

.LICENSE
Read Full License Agreement use /l FLAG OR pipe into a text file using 'regtotext /l > RTTLic.txt' to read in Notepad.


Download RegtoText demo version 2 (update Jan 2018). Read demo license.


For a commercial licensed version contact metadataconsult@gmail.com



Commercial version sample run on a new Windows 10 Pro install with Office 2016. 


Windows 10 Pro Registry Subkey Depth Frequency Distribution

Windows 10 Pro Registry Subkey Depth Frequency Distribution graph is determined by counting number of keys seperated by "\" backslashes. This also in including the hive key. Hive key is the first key in registry path and starts with KHEY, in example registy export below it's HKEY_LOCAL_MACHINE.

The subkey depth of the example exported registry path below is 8.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\VolumeCaches\Windows Error Reporting System Queue Files]


The number of subkeys or registry "paths" in a new Windows 10 Pro with Office is approx. 1M. 

The average subkey depth 8.2, and largest subkey depth group of 6 at 233,583 and a standard deviation of 2.510586.


Windows Registry Value Types Distribution


Not well publicized, but here's is a Windows Registry Value Types Distribution numbers and graph for an Windows 7 Ultimate registry developer machine. To understand the registry in a detail look at registry at http://www.techsupportalert.com/content/deeper-windows-registry.htm or search this blog with "Registry"

Exported DataType Registry Data Type Counts
"value" REG_SZ 1417270
dword: REG_DWORD 531122
hex: REG_BINARY 347749
hex(2): REG_EXPAND_SZ 28945
hex(7): REG_MULTI_SZ 24957
hex(b): REG_QWORD 4561
hex(0): REG_NONE 1271
hex(8): REG_RESOURCE_LIST 259
hex(a): REG_RESOURCE_REQUIREMENTS_LIST 244
hex(9): REG_FULL_RESOURCE_DESCRIPTOR 31
hex(6): REG_LINK 0
Type Frequency 2356409



All the registry types are translated listed as follows;


"value" alias hex(1)Default or blankString value data with escape characters
hex alias hex(3)REG_BINARY Binary data (any arbitrary data)
dword alias hex(4)
REG_DWORD / REG_DWORD_LITTLE_ENDIAN
A 32-bit unsigned integer coded in little-endian format
hex(0)REG_NONENo type (the stored value, if any)
hex(1)REG_SZA string value, normally stored and exposed in UTF-16LE (when using the Unicode version of Win32 API functions), usually terminated by a NUL character
hex(2)REG_EXPAND_SZAn “expandable” string value that can contain environment variables, normally stored and exposed in UTF-16LE, usually terminated by a NUL character
hex(3)REG_BINARYBinary data (any arbitrary data) including variable byte encoded UTF8 values
hex(4)
REG_DWORD_LITTLE_ENDIAN equivalent to REG_DWORD 
A 32-bit unsigned integer coded in little-endian format
hex(5)REG_DWORD_BIG_ENDIANA 32-bit unsigned integer coded in big-endian format
hex(6)REG_LINKA symbolic link (UNICODE) to another Registry key, specifying a root key and the path to the target key
hex(7)REG_MULTI_SZA multi-string value, which is an ordered list of non-empty strings, normally stored and exposed in UTF-16LE, each one terminated by a NUL character, the list being normally terminated by a second NUL character.
hex(8)REG_RESOURCE_LISTA resource list
hex(9)REG_FULL_RESOURCE_DESCRIPTORA resource descriptor
hex(a)REG_RESOURCE_REQUIREMENTS_LISTA resource requirements list
hex(b)REG_QWORD_LITTLE_ENDIAN equivalent to REG_QWORD 
A 64-bit integer little-endian (introduced in Windows XP)