Uro – Declutters Url Lists For Crawling/Pentesting

Utilizing a URL checklist for safety testing might be painful as there are quite a lot of URLs which have uninteresting/duplicate content material; uro goals to unravel that.

It does not make any http requests to the URLs and removes: – incremental urls e.g. /web page/1/ and /web page/2/ – weblog posts and related human written content material e.g. /posts/a-brief-history-of-time – urls with similar path however parameter worth distinction e.g. /web page.php?id=1 and /web page.php?id=2 – photographs, js, css and different “ineffective” recordsdata

Set up

The really helpful strategy to set up uro is as follows:

pipx set up uro

Be aware: If you’re utilizing an older model of python, use pip as an alternative of pipx

Fundamental Utilization

The quickest strategy to embody uro in your workflow is to feed it information via stdin and print it to your terminal.

cat urls.txt | uro

Superior utilization

Studying urls from a file (-i/–input)

uro -i enter.txt

Writing urls to a file (-o/–output)

If the file already exists, uro won’t overwrite the contents. In any other case, it’s going to create a brand new file.

uro -i enter.txt -o output.txt

Whitelist (-w/--whitelist)

uro will ignore all different extensions besides those offered.

uro -w php asp html

Be aware: Extensionless pages e.g. /books/1 will nonetheless be included. To take away them too, use --filter hasext.

Blacklist (-b/--blacklist)

uro will ignore the given extensions.

uro -b jpg png js pdf

Be aware: uro has an inventory of “ineffective” extensions which it removes by default; that checklist will probably be overridden by no matter extensions you present via blacklist possibility. Extensionless pages e.g. /books/1 will nonetheless be included. To take away them too, use --filter hasext.

Filters (-f/–filters)

For granular management, uro helps the next filters:

1. hasparams: solely output urls which have question parameters e.g. http://instance.com/web page.php?id=
2. noparams: solely output urls that haven’t any question parameters e.g. http://instance.com/web page.php
3. hasext: solely output urls which have extensions e.g. http://instance.com/web page.php
4. noext: solely output urls that haven’t any extensions e.g. http://instance.com/web page
5. allexts: do not take away any web page based mostly on extension e.g. hold .jpg which might be eliminated in any other case
6. keepcontent: hold human written content material e.g. blogs.
7. keepslash: do not take away trailing slash from urls e.g. http://instance.com/web page/
8. vuln: solely output urls with parameters which are know to be susceptible. Extra data.

Instance: uro --filters hasexts hasparams