Directories discovery
There are few apps allowing for a web directories mapping. Most of them perform the same function, but they vary in extra functionality. It is up to the tester to pick up the favorite one. Here I will try to paint a picture of what is available and what the tools are capable of doing. I will try to stack them side by side and compare the output as well as effectiveness of discovery. After all, rich directories discovery is as good as the word list used for the job.
Tests were conducted using identical parameters: word list, threads and IP
Target: HTB 10.10.10.6 - popcorn

OWASP OpenDoor:

Command run:
python3 opendoor.py --host 10.10.10.6 --scan=directories -t 50
Notable mentions:
+ It notifies about the denied access to the directories + Built in word list + Thread control and html report specification + Sub domain scan + URL parsing
Cons:
-- no option to exclude status codes or perform recursive scan -- slowest scan ( limited to 25 threads )
OpenDoor options:
1
└──╼ $python3 opendoor.py -h
2
usage: opendoor.py [-h] [--host HOST] [-p PORT] [-m METHOD] [-t THREADS]
3
[-d DELAY] [--timeout TIMEOUT] [-r RETRIES]
4
[--accept-cookies] [--debug DEBUG] [--tor]
5
[--torlist TORLIST] [--proxy PROXY] [-s SCAN] [-w WORDLIST]
6
[--reports REPORTS] [--reports-dir REPORTS_DIR]
7
[--random-agent] [--random-list] [--prefix PREFIX]
8
[-e EXTENSIONS] [-i IGNORE_EXTENSIONS] [--sniff SNIFF]
9
[--update] [--version] [--examples] [--docs]
10
[--wizard [WIZARD]]
11
12
optional arguments:
13
-h, --help show this help message and exit
14
15
required named options:
16
--host HOST Target host (ip); --host http://example.com
17
18
Application tools:
19
--update Update from CVS
20
--version Get current version
21
--examples Examples of usage
22
--docs Read documentation
23
--wizard [WIZARD] Run wizard scanner from your config
24
25
Debug tools:
26
--debug DEBUG Debug level 1 - 3
27
28
Reports tools:
29
--reports REPORTS Scan reports (json,std,txt,html)
30
--reports-dir REPORTS_DIR
31
Path to custom reports dir
32
33
Request tools:
34
-p PORT, --port PORT Custom port (Default 80)
35
-m METHOD, --method METHOD
36
Request method (use HEAD as default)
37
-d DELAY, --delay DELAY
38
Delay between requests threading
39
--timeout TIMEOUT Request timeout (30 sec default)
40
-r RETRIES, --retries RETRIES
41
Max retries to reconnect (default 3)
42
--accept-cookies Accept and route cookies from responses
43
--tor Using built-in proxylist
44
--torlist TORLIST Path to custom proxylist
45
--proxy PROXY Custom permanent proxy server
46
--random-agent Randomize user-agent per request
47
48
Sniff tools:
49
--sniff SNIFF Response sniff plugins
50
(indexof,collation,file,skipempty)
51
52
Stream tools:
53
-t THREADS, --threads THREADS
54
Allowed threads
55
56
Wordlist tools:
57
-s SCAN, --scan SCAN Scan type scan=directories or scan=subdomains
58
-w WORDLIST, --wordlist WORDLIST
59
Path to custom wordlist
60
--random-list Shuffle scan list
61
--prefix PREFIX Append path prefix to scan host
62
-e EXTENSIONS, --extensions EXTENSIONS
63
Force use selected extensions for scan session -e
64
php,json e.g
65
-i IGNORE_EXTENSIONS, --ignore-extensions IGNORE_EXTENSIONS
66
Force ignore extensions for scan session -i aspx,jsp
67
e.g
68
Copied!

Cansina

Command run:
python ./cansina.py -u 10.10.10.6 -p ./directories.dat --persist -t 50 --show-type --full-path -b 403,404
word list directories.dat taken from Open Door directory
Notable mentions:
+ Clean, minimalist output. + Ability to exclude status codes (-b for ban 403, 404) + Detailed information about discovered links ( code, size, line, time ) + Option to resume the scan + Extensive options and use of recursive scan
Cons:
-- No HTML output option
Cansina options
1
└──╼ $python ./cansina.py -h
2
_____ _
3
/ ____| (_)
4
| | __ _ _ __ ___ _ _ __ __ _
5
| | / _` | '_ \/ __| | '_ \ / _` |
6
| |___| (_| | | | \__ \ | | | | (_| |
7
\_____\__,_|_| |_|___/_|_| |_|\__,_|
8
9
10
usage: cansina.py -u url -p payload [options]
11
12
Cansina is a web content discovery tool. It makes requests and analyze the
13
responses trying to figure out whether the resource is or not accessible.
14
15
optional arguments:
16
-h, --help show this help message and exit
17
-A AUTHENTICATION Basic Authentication (e.g: user:password)
18
-C COOKIES your cookies (e.g: key:value)
19
-D Check for fake 404 (warning: machine decision)
20
-H Make HTTP HEAD requests
21
-P PROXIES Set a http and/or https proxy (ex:
22
http://127.0.0.1:8080,https://...
23
-S Remove ending slash for payloads
24
-T REQUEST_DELAY Time (a float number, e.g: 0.25 or 1.75) between
25
requests
26
-U Make payload requests upper-case
27
-a USER_AGENT The preferred user-agent (default provided)
28
-b BANNED List of banned response codes
29
-B UNBANNED List of unbanned response codes, mark all response as
30
invalid without unbanned response codes, higher
31
priority than banned
32
-c CONTENT Inspect content looking for a particular string
33
-d DISCRIMINATOR If this string if found it will be treated as a 404
34
-e EXTENSION Extension list to use e.g: php,asp,...(default none)
35
-p PAYLOAD A single file, a file with filenames (.payload) or a
36
directory (will do *.txt)
37
-s SIZE_DISCRIMINATOR
38
Will skip pages with this size in bytes (or a list of
39
sizes 0,500,1500...)
40
-t THREADS Number of threads (default 4)
41
-u TARGET Target url
42
-r RESUME Resume a session
43
-R Parse robots.txt and check its contents
44
--recursive Recursive descend on path directories
45
--persist Use HTTP persistent connections
46
--full-path Show full path instead of only resources
47
--show-type Show content-type in results
48
--no-follow Do not follow redirections
49
--line CONTINUE_LINE Continue payload in line <n>
50
--headers HEADERS Set personalized headers: key=value;key=value...
51
--capitalize Transform 'word' into 'Word'.
52
--strip-extension Strip word extension: word.ext into word
53
--alpha Filter non alphanumeric words from wordlist
54
55
License, requests, etc: https://github.com/deibit/cansina
56
Copied!
Download

Dirsearch

Command run:
./dirsearch.py -u 10.10.10.6 -e html -t 50 -w ./directories.dat -x 403
Notable mentions:
+ Can be very quick in directories discovery + Extensive options in regards to fine-tuning and reports + Recursive search
Cons:
-- No options for URL parsing -- Need to specify an extension to run
Dirsearch options:
1
└──╼ $./dirsearch.py -h
2
Usage: dirsearch.py [-u|--url] target [-e|--extensions] extensions [options]
3
4
Options:
5
-h, --help show this help message and exit
6
7
Mandatory:
8
-u URL, --url=URL URL target
9
-L URLLIST, --url-list=URLLIST
10
URL list target
11
-e EXTENSIONS, --extensions=EXTENSIONS
12
Extension list separated by comma (Example: php,asp)
13
14
Dictionary Settings:
15
-w WORDLIST, --wordlist=WORDLIST
16
-l, --lowercase
17
-f, --force-extensions
18
Force extensions for every wordlist entry (like in
19
DirBuster)
20
21
General Settings:
22
-s DELAY, --delay=DELAY
23
Delay between requests (float number)
24
-r, --recursive Bruteforce recursively
25
--suppress-empty, --suppress-empty
26
--scan-subdir=SCANSUBDIRS, --scan-subdirs=SCANSUBDIRS
27
Scan subdirectories of the given -u|--url (separated
28
by comma)
29
--exclude-subdir=EXCLUDESUBDIRS, --exclude-subdirs=EXCLUDESUBDIRS
30
Exclude the following subdirectories during recursive
31
scan (separated by comma)
32
-t THREADSCOUNT, --threads=THREADSCOUNT
33
Number of Threads
34
-x EXCLUDESTATUSCODES, --exclude-status=EXCLUDESTATUSCODES
35
Exclude status code, separated by comma (example: 301,
36
500)
37
-c COOKIE, --cookie=COOKIE
38
--ua=USERAGENT, --user-agent=USERAGENT
39
-F, --follow-redirects
40
-H HEADERS, --header=HEADERS
41
Headers to add (example: --header "Referer:
42
example.com" --header "User-Agent: IE"
43
--random-agents, --random-user-agents
44
45
Connection Settings:
46
--timeout=TIMEOUT Connection timeout
47
--ip=IP Resolve name to IP address
48
--proxy=HTTPPROXY, --http-proxy=HTTPPROXY
49
Http Proxy (example: localhost:8080
50
--max-retries=MAXRETRIES
51
-b, --request-by-hostname
52
By default dirsearch will request by IP for speed.
53
This forces requests by hostname
54
55
Reports:
56
--simple-report=SIMPLEOUTPUTFILE
57
Only found paths
58
--plain-text-report=PLAINTEXTOUTPUTFILE
59
Found paths with status codes
60
--json-report=JSONOUTPUTFILE
61
Copied!
Download

Gobuster

Command run:
gobuster -u 10.10.10.6 -w ./directories.dat -t 50 -s 200 -e
Notable mentions:
+ Very quick in directories discovery + Ability to skip SSL certificate verification + Good fine tuning and URL parsing + Included in Kali and Parrot OS by default + DNS brute forcing with wildcard detection
Cons:
-- Does not seem to keep alive connections resulting in canceled requests from time to time
Gobuster options:
1
└──╼ $gobuster -h
2
Usage of gobuster:
3
-P string
4
Password for Basic Auth (dir mode only)
5
-U string
6
Username for Basic Auth (dir mode only)
7
-a string
8
Set the User-Agent string (dir mode only)
9
-c string
10
Cookies to use for the requests (dir mode only)
11
-cn
12
Show CNAME records (dns mode only, cannot be used with '-i' option)
13
-e Expanded mode, print full URLs
14
-f Append a forward-slash to each directory request (dir mode only)
15
-fw
16
Force continued operation when wildcard found
17
-i Show IP addresses (dns mode only)
18
-k Skip SSL certificate verification
19
-l Include the length of the body in the output (dir mode only)
20
-m string
21
Directory/File mode (dir) or DNS mode (dns) (default "dir")
22
-n Don't print status codes
23
-np
24
Don't display progress
25
-o string
26
Output file to write results to (defaults to stdout)
27
-p string
28
Proxy to use for requests [http(s)://host:port] (dir mode only)
29
-q Don't print the banner and other noise
30
-r Follow redirects
31
-s string
32
Positive status codes (dir mode only) (default "200,204,301,302,307,403")
33
-t int
34
Number of concurrent threads (default 10)
35
-to duration
36
HTTP Timeout in seconds (dir mode only) (default 10s)
37
-u string
38
The target URL or Domain
39
-v Verbose output (errors)
40
-w string
41
Path to the wordlist
42
-x string
43
File extension(s) to search for (dir mode only)
44
Copied!
Download

OWASP Dirbuster

Command run:
1
dirbuster
Copied!
Notable mentions:
+ Can be very precise with the specified word list + Result tree and thread control as well as extension specification + CLI and GUI options available. + Recursive search and good thread control + Comes pre-installed with Kali Linux and Parrot OS
Cons:
-- Can sometimes crash (freeze)
Dirbuster options:
1
└──╼ $dirbuster -h
2
DirBuster - 1.0-RC1
3
Usage: java -jar DirBuster-1.0-RC1 -u <URL http://example.com/> [Options]
4
5
Options:
6
-h : Display this help message
7
-H : Start DirBuster in headless mode (no gui), report will be auto saved on exit
8
-l <Word list to use> : The Word list to use for the list based brute force. Default: /home/rafaleon/Practice/dirsearch/directory-list-2.3-small.txt
9
-g : Only use GET requests. Default Not Set
10
-e <File Extention list> : File Extention list eg asp,aspx. Default: php
11
-t <Number of Threads> : Number of connection threads to use. Default: 10
12
-s <Start point> : Start point of the scan. Default: /
13
-v : Verbose output, Default: Not set
14
-P : Don't Parse html, Default: Not Set
15
-R : Don't be recursive, Default: Not Set
16
-r <location> : File to save report to. Default: /home/rafaleon/Practice/dirsearch/DirBuster-Report-[hostname]-[port].txt
17
18
Examples:
19
20
Run DirBuster in headless mode
21
java -jar DirBuster-1.0-RC1.jar -H -u https://www.target.com/
22
23
Start GUI with target prepopulated
24
java -jar DirBuster-1.0-RC1.jar -u https://www.target.com/
Copied!
Download
Conclusions:
Directories discovery is a major part of a security engagement. Ability to find directories not exposed to public eye but searchable by pentesting tools can discover critical information about the web infrastructure of the target in scope. It is up to security researcher to find the best tool for the job and combine the right word list that will present the most fruitful results.
Security Word list for every occasion:
GitHub - danielmiessler/SecLists: SecLists is the security tester's companion. It's a collection of multiple types of lists used during security assessments, collected in one place. List types include usernames, passwords, URLs, sensitive data patterns, fuzzing payloads, web shells, and many more.
GitHub