OSINT › Module 7 › Lesson 1
Wayback Machine and urlscan.io
Archives remember pages you unpublished — YOUR URLs only, plus GATE then
Visual · archive_wayback_urlscan
Wayback and urlscan.io are libraries for URLs YOU own.
Opening
Deleting a page is not the same as deleting every copy. Archives keep receipts.
The Internet Archive Wayback Machine at https://web.archive.org/ stores public snapshots. Its CDX API lists capture timestamps for a URL. urlscan.io at https://urlscan.io/ records how a URL rendered for a scan you are allowed to request. Cyberlium uses both on URLs YOU own — your site, your GitHub Pages, or the loopback toy — not a classmate's profile and not a stranger's login page. Original Cyberlium Topic 12.
1. Wayback is a time machine for public URLs, not a stalking kit
If YOU published a page, Wayback may still have a copy after you took it down. That is why defenders check their own sites. The CDX API is a listing helper: timestamps and original URLs. You substitute a site YOU own, or you skip. A classmate's name in a CDX query is still people-search you were told not to do in Module 3.
urlscan.io is a rendering/scan service. Submit YOUR URLs after you create YOUR account if the UI requires it. Do not submit a victim's bank login “to see the DOM.” Do not treat urlscan as a webcam hunter.
Command guide
Wayback is a time machine for public URLs, not a stalking kit
═══ INSTALL ═══
Linux (Debian/Ubuntu):
Command — copy this
sudo apt install curl
macOS: Built-in
Windows: Built-in (PowerShell: Invoke-WebRequest)
═══ LINUX / macOS ═══
Wayback Machine — check archived snapshots
Command — copy this
curl -s 'https://web.archive.org/web/timemap/link/example.com' | head -20 curl -s 'https://archive.org/wayback/available?url=example.com' | python3 -m json.tool
urlscan.io — search for domain scans
Command — copy this
curl -s 'https://urlscan.io/api/v1/search/?q=domain:example.com' | python3 -m json.tool | head -40
2. robots.txt and sitemap.xml are invitations you already published
curl YOUR site's /robots.txt and /sitemap.xml (or the loopback toy later) is reading files the server offered. That is not a stealth crawl of someone else's admin panel. If you have no site, write that and wait for the toy in the lab.
Do not weaponize Disallow lines as a “hidden treasure map” against a company you do not own. Do not hydra paths you saw in a sitemap.
3. GATE still is not an archive target
If 192.168.0.1 was Router Admin, you STOPPED. Do not urlscan your home gateway. Do not put a router login in Wayback on purpose. Archives of YOUR public blog are in-scope. LAN admin panels are not.
Teaching bind remains Classmates do not need your toy on 0.0.0.0.
4. What you ship: named archive tools plus YOUR-URL-or-skip
Bookmarks for Wayback, CDX, urlscan.io. CDX literacy curl recorded. robots.txt shape noted. chmod 600. No stranger submissions. SAFE
5. What you record before the next lesson
Date. Wayback/urlscan = YOUR URLs. DEMO GATE. NEVER stalk archives of classmates. File t12-m07-l01-wayback-urlscan.txt.
6. Wrong vs right: strangers vs identifiers YOU own
Worked failure — same OSINT word, opposite target. Right never needs a classmate or a dump site.
Wrong
urlscan a classmate's private app. CDX a stranger's login. nmap /24 to find “more sites to archive.”
Right
Official Wayback and urlscan on URLs YOU own, or skip. Next: Wappalyzer WhatWeb BuiltWith Observatory.
Mission: treat archives as copies of YOUR publishing
1) / STOP if router. 2) Bookmark Wayback and urlscan.io. 3) Run the CDX literacy curl or skip; write YOUR-URL-or-skip. Never submit strangers. Never nmap the LAN.
Stuck? Ask Cyberlium AI Mentor
If CDX times out, use https://web.archive.org/ in a browser for YOUR URL — not a dump site.
Knowledge Check
APPLY: You want to see old copies of YOUR blog. First official stop:
Multiple choice
Knowledge Check
APPLY: True or False: urlscan.io is in-scope for any URL on the internet because the service is public.
True or False
Knowledge Check
APPLY: curl http://192.168.0.1/ is TP-Link Router Admin. You:
Multiple choice