Bug › Module 4 › Lesson 3
URL Discovery
URL discovery finds endpoints and parameters — crawl and archive only in-scope $SCOPE_HOST assets.
Visual · bounty_url_discovery
URL discovery literacy. $SCOPE_HOST only. Original Cyberlium.
Opening
More URLs is not more permission. Scope and rate limits govern every fetch.
URL discovery combines crawling, archived URLs, JavaScript parsing, and wordlists to find endpoints, APIs, and parameters on in-scope hosts. Bounty hunters use tools conceptually — katana, gau, wayback URLs — filtered to $SCOPE_HOST hostnames at policy rates. Crawling out-of-scope CDN or third-party widgets wastes time and violates rules. Cyberlium teaches URL discovery literacy without crawling random sites or classmate projects. You log sources and stay within scope. Next: Recon Lab.
1. Sources for URL discovery
Wayback/archived URLs for in-scope $SCOPE_HOST pages, sitemap.xml on listed domains, robots.txt, linked JS bundles on in-scope apps, and wordlists run against confirmed in-scope hosts only.
Each source gets a note line: passive vs active, hostname, scope confirmation date.
Command guide
Try these commands — Sources for URL discovery
═══ TOOLS & WEBSITES ═══ Browse / read these (authorized learning only — stay in YOUR lab / program scope)
Httpx — https://github.com/projectdiscovery/httpx (probe hosts from scope) Status codes — map live endpoints
═══ INSTALL ═══
Linux (Debian/Ubuntu):
Command — copy this
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest sudo apt install curl
macOS:
Command — copy this
brew install httpx
Windows:
Command — copy this
go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest
Built-in (PowerShell: Invoke-WebRequest)
═══ LINUX / macOS ═══
Command — copy this
source "$HOME/cyberlium-lab/t20-scope.env" printf '%s ' "$SCOPE_HOST" > "$HOME/cyberlium-lab/t20-hosts.txt" httpx -l "$HOME/cyberlium-lab/t20-hosts.txt" -status-code -title -silent 2>/dev/null | head -15 || curl -sS -I "https://$SCOPE_HOST/" | head -10
Primary tools to practice this lesson: httpx, curl. Reference sites: Httpx (https://github.com/projectdiscovery/httpx); Status codes. Run every command in the box — install first, then the usage lines — only on YOUR lab / program scope.
2. Crawl discipline and rate limits
Respect program rate limits during crawls. Avoid aggressive recursive crawls on production unless policy allows. Prefer targeted path wordlists on known in-scope apps.
Tutorial crawls on public shop domains get rewritten to $SCOPE_HOST mentally before copying habits.
3. URL lists feed asset mapping — stay in scope
Discovered URLs become inventory rows next module — but only for in-scope hosts. Third-party iframe URLs are noted as out-of-scope, not crawled deeply.
Ship: URL discovery source table for $SCOPE_HOST — source, passive/active, scope check. Next: Recon Lab.
4. What you ship: URL discovery source table for $SCOPE_HOST
Sources, passive/active tag, scope check column. $SCOPE_HOST hostnames only. chmod 600.
5. What you record before the next lesson
Date. URL discovery table. $SCOPE_HOST named. File t20-m04-l03-url-discovery.txt chmod 600.
6. Wrong vs right: random sites vs in-scope program
Worked failure — same MSF word, opposite target. Right never needs a café Wi-Fi or classmate laptop.
Wrong
Crawl every link on the internet from one in-scope page. Archive classmate deploy URLs.
Right
Write URL discovery table for $SCOPE_HOST. Next: Recon Lab.
Mission: template URL discovery sources
1) List three URL sources for $SCOPE_HOST. 2) Tag passive vs active. 3) Add scope check column. 4) chmod 600. Never crawl unconfirmed hostnames.
Stuck? Ask Cyberlium AI Mentor
URL floods without scope filter create bans — filter first.
Knowledge Check
APPLY: URL discovery in Cyberlium applies to:
Multiple choice
Knowledge Check
APPLY: True or False: Aggressive crawls ignore program rate limits.
True or False
Knowledge Check
APPLY: Third-party iframe URL on $SCOPE_HOST page:
Multiple choice