Thursday, June 8, 2017

Get most recent wget for windows and get response headers from server only

What is Wget?  

GNU Wget is a free software package for dowloading files from using internet using HTTP, HTTPS and FTP, the most widely-used Internet protocols.


GNU Wget has many features to make retrieving large files or mirroring entire web or FTP sites easy, including:
  • Can resume aborted downloads, using REST and RANGE
  • Can use filename wild cards and recursively mirror directories
  • NLS-based message files for many different languages
  • Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally
  • Runs on most UNIX-like operating systems as well as Microsoft Windows
  • Supports HTTP proxies
  • Supports HTTP cookies
  • Supports persistent HTTP connections
  • Unattended / background operation

How to get most recent build of wget for Windows? 

Jernej Simoncic is an active maintainer and provider of wget for Windows at 


Get latest version 1.19.1


I am a fan of individual utilities, instead of downloading entire subsystems. Cygwin provides this utility, but the default bare bones Cygwin install is 101 MB, but for other packages the full installation can reach 114 Gb and options are numerous. A little prohibitive, for a quick and dirty.

How to get response headers from server only? 

This will dump the http server response headers and not download the home page (index.html).

 
1
wget --server-response --spider geniebouchard.com

Results for http://geniebouchard.com we can see it is redirected and to suspicious ww1.geniebouchard.com. 

 Also we these are not properly hidden headers, revealing juicy details for hackers

  • Server: Apache/2.4.6 (CentOS) PHP/5.4.16
  • X-Powered-By: PHP/5.6.29-1~dotdeb+7.1


Results of wget for http://geniebouchard.com/

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
Spider mode enabled. Check if remote file exists.

--2017-06-08 12:22:09--  http://geniebouchard.com/
Resolving geniebouchard.com (geniebouchard.com)... 69.39.236.56
Connecting to geniebouchard.com (geniebouchard.com)|69.39.236.56|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.1 302 Found
  Date: Thu, 08 Jun 2017 16:38:46 GMT
  Server: Apache/2.4.6 (CentOS) PHP/5.4.16
  X-Powered-By: PHP/5.4.16
  Location: http://ww1.geniebouchard.com
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: text/html; charset=UTF-8

Location: http://ww1.geniebouchard.com 
[following]
Spider mode enabled. Check if remote file exists.

--2017-06-08 12:22:09--  http://ww1.geniebouchard.com/
Resolving ww1.geniebouchard.com (ww1.geniebouchard.com)... 72.52.4.90
Connecting to ww1.geniebouchard.com (ww1.geniebouchard.com)|72.52.4.90|:80... connected.
HTTP request sent, awaiting response...
  HTTP/1.0 200 OK
  Date: Thu, 08 Jun 2017 16:21:48 GMT
  Server: Apache/2.2.22 (Debian)
  X-Powered-By: PHP/5.6.29-1~dotdeb+7.1
  Expires: Mon, 26 Jul 1997 05:00:00 GMT
  Last-Modified: Thu, 08 Jun 2017 16:21:48 GMT
  Cache-Control: no-store, no-cache, must-revalidate
  Cache-Control: post-check=0, pre-check=0
  Pragma: no-cache
  Vary: Accept-Encoding
  Content-Type: text/html; charset=UTF-8
  X-Cache: MISS from 550555
  Cneonction: close
  Connection: Keep-Alive
  Set-Cookie: NSC_tfep-83+63+5+01-91=ffffffff516a73d445525d5f4f58455e445a4a423660;path=/;httponly
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.



How to block X-Powered-By and Server response headers? 

 In your .htaccess file you can try, but depends on your server config and if server was built with mod_headers

 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Hide X-Powered-By and Server headers, sent by downstream application servers:
# Note you need both below as the "always" one doesn't work with Jboss for some reason

ServerSignature Off

<IfModule mod_headers.c>
 Header unset Server
 Header always unset X-Powered-By
 Header unset X-Powered-By
 Header unset X-CF-Powered-By
 Header unset X-Mod-Pagespeed
 Header unset X-Pingback
</IfModule>

Posted by metadataconsulting (profile) at 12:20 PM

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)