Directories discovery

There are few apps allowing for a web directories mapping. Most of them perform the same function, but they vary in extra functionality. It is up to the tester to pick up the favorite one. Here I will try to paint a picture of what is available and what the tools are capable of doing. I will try to stack them side by side and compare the output as well as effectiveness of discovery. After all, rich directories discovery is as good as the word list used for the job.

Tests were conducted using identical parameters: word list, threads and IP

Target: HTB 10.10.10.6 - popcorn

OWASP OpenDoor:

Command run:

python3 opendoor.py --host 10.10.10.6 --scan=directories -t 50

Notable mentions:

+ It notifies about the denied access to the directories + Built in word list + Thread control and html report specification + Sub domain scan + URL parsing

Cons:

-- no option to exclude status codes or perform recursive scan -- slowest scan ( limited to 25 threads )

OpenDoor options:

└──╼ $python3 opendoor.py -h
usage: opendoor.py [-h] [--host HOST] [-p PORT] [-m METHOD] [-t THREADS]
[-d DELAY] [--timeout TIMEOUT] [-r RETRIES]
[--accept-cookies] [--debug DEBUG] [--tor]
[--torlist TORLIST] [--proxy PROXY] [-s SCAN] [-w WORDLIST]
[--reports REPORTS] [--reports-dir REPORTS_DIR]
[--random-agent] [--random-list] [--prefix PREFIX]
[-e EXTENSIONS] [-i IGNORE_EXTENSIONS] [--sniff SNIFF]
[--update] [--version] [--examples] [--docs]
[--wizard [WIZARD]]
optional arguments:
-h, --help show this help message and exit
required named options:
--host HOST Target host (ip); --host http://example.com
Application tools:
--update Update from CVS
--version Get current version
--examples Examples of usage
--docs Read documentation
--wizard [WIZARD] Run wizard scanner from your config
Debug tools:
--debug DEBUG Debug level 1 - 3
Reports tools:
--reports REPORTS Scan reports (json,std,txt,html)
--reports-dir REPORTS_DIR
Path to custom reports dir
Request tools:
-p PORT, --port PORT Custom port (Default 80)
-m METHOD, --method METHOD
Request method (use HEAD as default)
-d DELAY, --delay DELAY
Delay between requests threading
--timeout TIMEOUT Request timeout (30 sec default)
-r RETRIES, --retries RETRIES
Max retries to reconnect (default 3)
--accept-cookies Accept and route cookies from responses
--tor Using built-in proxylist
--torlist TORLIST Path to custom proxylist
--proxy PROXY Custom permanent proxy server
--random-agent Randomize user-agent per request
Sniff tools:
--sniff SNIFF Response sniff plugins
(indexof,collation,file,skipempty)
Stream tools:
-t THREADS, --threads THREADS
Allowed threads
Wordlist tools:
-s SCAN, --scan SCAN Scan type scan=directories or scan=subdomains
-w WORDLIST, --wordlist WORDLIST
Path to custom wordlist
--random-list Shuffle scan list
--prefix PREFIX Append path prefix to scan host
-e EXTENSIONS, --extensions EXTENSIONS
Force use selected extensions for scan session -e
php,json e.g
-i IGNORE_EXTENSIONS, --ignore-extensions IGNORE_EXTENSIONS
Force ignore extensions for scan session -i aspx,jsp
e.g

Download

Cansina

Command run:

python ./cansina.py -u 10.10.10.6 -p ./directories.dat --persist -t 50 --show-type --full-path -b 403,404

word list directories.dat taken from Open Door directory

Notable mentions:

+ Clean, minimalist output. + Ability to exclude status codes (-b for ban 403, 404) + Detailed information about discovered links ( code, size, line, time ) + Option to resume the scan + Extensive options and use of recursive scan

Cons:

-- No HTML output option

Cansina options

└──╼ $python ./cansina.py -h
_____ _
/ ____| (_)
| | __ _ _ __ ___ _ _ __ __ _
| | / _` | '_ \/ __| | '_ \ / _` |
| |___| (_| | | | \__ \ | | | | (_| |
\_____\__,_|_| |_|___/_|_| |_|\__,_|
usage: cansina.py -u url -p payload [options]
Cansina is a web content discovery tool. It makes requests and analyze the
responses trying to figure out whether the resource is or not accessible.
optional arguments:
-h, --help show this help message and exit
-A AUTHENTICATION Basic Authentication (e.g: user:password)
-C COOKIES your cookies (e.g: key:value)
-D Check for fake 404 (warning: machine decision)
-H Make HTTP HEAD requests
-P PROXIES Set a http and/or https proxy (ex:
http://127.0.0.1:8080,https://...
-S Remove ending slash for payloads
-T REQUEST_DELAY Time (a float number, e.g: 0.25 or 1.75) between
requests
-U Make payload requests upper-case
-a USER_AGENT The preferred user-agent (default provided)
-b BANNED List of banned response codes
-B UNBANNED List of unbanned response codes, mark all response as
invalid without unbanned response codes, higher
priority than banned
-c CONTENT Inspect content looking for a particular string
-d DISCRIMINATOR If this string if found it will be treated as a 404
-e EXTENSION Extension list to use e.g: php,asp,...(default none)
-p PAYLOAD A single file, a file with filenames (.payload) or a
directory (will do *.txt)
-s SIZE_DISCRIMINATOR
Will skip pages with this size in bytes (or a list of
sizes 0,500,1500...)
-t THREADS Number of threads (default 4)
-u TARGET Target url
-r RESUME Resume a session
-R Parse robots.txt and check its contents
--recursive Recursive descend on path directories
--persist Use HTTP persistent connections
--full-path Show full path instead of only resources
--show-type Show content-type in results
--no-follow Do not follow redirections
--line CONTINUE_LINE Continue payload in line <n>
--headers HEADERS Set personalized headers: key=value;key=value...
--capitalize Transform 'word' into 'Word'.
--strip-extension Strip word extension: word.ext into word
--alpha Filter non alphanumeric words from wordlist
License, requests, etc: https://github.com/deibit/cansina

Download

Dirsearch

Command run:

./dirsearch.py -u 10.10.10.6 -e html -t 50 -w ./directories.dat -x 403

Notable mentions:

+ Can be very quick in directories discovery + Extensive options in regards to fine-tuning and reports + Recursive search

Cons:

-- No options for URL parsing -- Need to specify an extension to run

Dirsearch options:

└──╼ $./dirsearch.py -h
Usage: dirsearch.py [-u|--url] target [-e|--extensions] extensions [options]
Options:
-h, --help show this help message and exit
Mandatory:
-u URL, --url=URL URL target
-L URLLIST, --url-list=URLLIST
URL list target
-e EXTENSIONS, --extensions=EXTENSIONS
Extension list separated by comma (Example: php,asp)
Dictionary Settings:
-w WORDLIST, --wordlist=WORDLIST
-l, --lowercase
-f, --force-extensions
Force extensions for every wordlist entry (like in
DirBuster)
General Settings:
-s DELAY, --delay=DELAY
Delay between requests (float number)
-r, --recursive Bruteforce recursively
--suppress-empty, --suppress-empty
--scan-subdir=SCANSUBDIRS, --scan-subdirs=SCANSUBDIRS
Scan subdirectories of the given -u|--url (separated
by comma)
--exclude-subdir=EXCLUDESUBDIRS, --exclude-subdirs=EXCLUDESUBDIRS
Exclude the following subdirectories during recursive
scan (separated by comma)
-t THREADSCOUNT, --threads=THREADSCOUNT
Number of Threads
-x EXCLUDESTATUSCODES, --exclude-status=EXCLUDESTATUSCODES
Exclude status code, separated by comma (example: 301,
500)
-c COOKIE, --cookie=COOKIE
--ua=USERAGENT, --user-agent=USERAGENT
-F, --follow-redirects
-H HEADERS, --header=HEADERS
Headers to add (example: --header "Referer:
example.com" --header "User-Agent: IE"
--random-agents, --random-user-agents
Connection Settings:
--timeout=TIMEOUT Connection timeout
--ip=IP Resolve name to IP address
--proxy=HTTPPROXY, --http-proxy=HTTPPROXY
Http Proxy (example: localhost:8080
--max-retries=MAXRETRIES
-b, --request-by-hostname
By default dirsearch will request by IP for speed.
This forces requests by hostname
Reports:
--simple-report=SIMPLEOUTPUTFILE
Only found paths
--plain-text-report=PLAINTEXTOUTPUTFILE
Found paths with status codes
--json-report=JSONOUTPUTFILE

Download

Gobuster

Command run:

gobuster -u 10.10.10.6 -w ./directories.dat -t 50 -s 200 -e

Notable mentions:

+ Very quick in directories discovery + Ability to skip SSL certificate verification + Good fine tuning and URL parsing + Included in Kali and Parrot OS by default + DNS brute forcing with wildcard detection

Cons:

-- Does not seem to keep alive connections resulting in canceled requests from time to time

Gobuster options:

└──╼ $gobuster -h
Usage of gobuster:
-P string
Password for Basic Auth (dir mode only)
-U string
Username for Basic Auth (dir mode only)
-a string
Set the User-Agent string (dir mode only)
-c string
Cookies to use for the requests (dir mode only)
-cn
Show CNAME records (dns mode only, cannot be used with '-i' option)
-e Expanded mode, print full URLs
-f Append a forward-slash to each directory request (dir mode only)
-fw
Force continued operation when wildcard found
-i Show IP addresses (dns mode only)
-k Skip SSL certificate verification
-l Include the length of the body in the output (dir mode only)
-m string
Directory/File mode (dir) or DNS mode (dns) (default "dir")
-n Don't print status codes
-np
Don't display progress
-o string
Output file to write results to (defaults to stdout)
-p string
Proxy to use for requests [http(s)://host:port] (dir mode only)
-q Don't print the banner and other noise
-r Follow redirects
-s string
Positive status codes (dir mode only) (default "200,204,301,302,307,403")
-t int
Number of concurrent threads (default 10)
-to duration
HTTP Timeout in seconds (dir mode only) (default 10s)
-u string
The target URL or Domain
-v Verbose output (errors)
-w string
Path to the wordlist
-x string
File extension(s) to search for (dir mode only)

Download

OWASP Dirbuster

Command run:

dirbuster

Notable mentions:

+ Can be very precise with the specified word list + Result tree and thread control as well as extension specification + CLI and GUI options available. + Recursive search and good thread control + Comes pre-installed with Kali Linux and Parrot OS

Cons:

-- Can sometimes crash (freeze)

Dirbuster options:

└──╼ $dirbuster -h
DirBuster - 1.0-RC1
Usage: java -jar DirBuster-1.0-RC1 -u <URL http://example.com/> [Options]
Options:
-h : Display this help message
-H : Start DirBuster in headless mode (no gui), report will be auto saved on exit
-l <Word list to use> : The Word list to use for the list based brute force. Default: /home/rafaleon/Practice/dirsearch/directory-list-2.3-small.txt
-g : Only use GET requests. Default Not Set
-e <File Extention list> : File Extention list eg asp,aspx. Default: php
-t <Number of Threads> : Number of connection threads to use. Default: 10
-s <Start point> : Start point of the scan. Default: /
-v : Verbose output, Default: Not set
-P : Don't Parse html, Default: Not Set
-R : Don't be recursive, Default: Not Set
-r <location> : File to save report to. Default: /home/rafaleon/Practice/dirsearch/DirBuster-Report-[hostname]-[port].txt
Examples:
Run DirBuster in headless mode
java -jar DirBuster-1.0-RC1.jar -H -u https://www.target.com/
Start GUI with target prepopulated
java -jar DirBuster-1.0-RC1.jar -u https://www.target.com/

Download

Conclusions:

Directories discovery is a major part of a security engagement. Ability to find directories not exposed to public eye but searchable by pentesting tools can discover critical information about the web infrastructure of the target in scope. It is up to security researcher to find the best tool for the job and combine the right word list that will present the most fruitful results.

Security Word list for every occasion: