urlscan.io Blog

urlscan.io has always been a powerful tool for scanning and investigating suspicious websites. Our platform is used by hundreds of customers and tens of thousands of community users to scan suspicious URLs. Up until now, the majority of these scans were initiated by customers.

Today we are announcing the general availability of urlscan Observe, our new and integrated hands-off monitoring system on the urlscan Pro platform. urlscan Observe ties together our extensive data collection with our notification and scanning features to drive fast and automated monitoring of suspected malicious infrastructure.

urlscan Observe: The idea

urlscan Observe aims to fill two gaps in existing automation workflows:

• Automatically discovering interesting things such as domains, hostnames, IPs, or URLs.
• Automatically monitoring these things for activity and changes.

Using the example of domains used for phishing and brand impersonation gives a good overview of the challenges involved. Proactively looking out for suspicious domains is something that a lot of our customers are already doing, and there are a variety of commercial and Open Source tools available for surfacing these. The easiest way to get started would be to start monitoring free data sources like certstream for TLS hostnames of interest. While finding suspicious domains might be relatively easy, the hard part is what happens next: Monitoring these domains to see if they ever go live.

We have built urlscan Observe to actively monitor observables, scan them using our web scanning engine, resolve DNS records, capture any observations, alert you about major changes and show the whole timeline in a dedicated UI.

Monitoring Example: Suspicious domains

To understand the different stages in the lifecycle of a piece of infrastructure, let’s have a look at a fictional albeit common timeline of a suspicious domain and what we can observe about it:

• 1:12am We observe the domain for the first time in a DNS zonefile. It does not have any DNS A/AAAA records yet.
• 2:40pm The domain starts to resolve to an IPv4 address.
• 2:45pm We observe the first TLS certificate for this domain in a Certificate Transparency (CT) log.
• 2:55pm The domain starts responding to HTTP requests. It only carries an empty landing page.
• 3:12pm The domain starts serving a directory index listing on HTTP.
• 3:36pm The domain starts serving a phishing site via HTTP.
• 6:15pm The domain has deactivated by the hosting company and is now serving an empty placeholder page.
• 8:50pm The domain stops resolving via DNS.

This timeline of events tells the story of when and how the malicious domain was first set up, how quickly it went live and how long it took until it was taken down again. At every step of this lifecycle one has to monitor the domain (DNS, HTTP, TLS certs) and compare observations with previous time intervals to figure out if anything about the domain has recently changed. This is exactly what urlscan Observe will automatically do for you going forward.

urlscan Observe workflow

urlscan Observe monitors Observables such as hostnames, domains, IPs and URLs as part of Incidents within urlscan Pro. There are two ways to create these incidents:

• You can manually create an incident by supplying your own hostname, domain, IP, or URL.
• You can set up Saved Searches and a Subscription within urlscan Pro to automatically create incidents for new observables.

Using Saved Searches is a really powerful tool because you can write a query that matches interesting observables in our Real-Time Newly Observed Hostnames & Domains Feed. This feed captures hundreds of thousands of new domains and millions of new unique hostnames every day. Using our Search API you can write a query that matches hostnames of interest, either by strings within the hostname or by certain infrastructure attributes such as NS, MX or other DNS records.

You can work with incidents from within the urlscan Pro UI, but you can also set up Alerting Channels to be notified whenever there are new incidents and changes to existing incidents. As part of urlscan Observe we have overhauled our notification system and can now send out notifications via E-Mail and Webhooks.

Availability

urlscan Observe is available starting today and is included for all customers on our Professional, Enterprise and Ultimate plans.

If you want to learn about urlscan Pro platform and how it might be valuable for your organisation feel free to reach out to us! We offer free trials with no strings attached. We would be happy to give you a passionate demo of what our platform can do for you. Reach out to us at sales@urlscan.io.