urlscan.io Blog

Today we are officially launching our real-time feed and search index of newly observed hostnames and domains on urlscan Pro. This is a huge step forward since it will allow customers to proactively look for new domains and hostnames that might be of interest to them, even if these hostnames were not previously scanned as a full-blown website through urlscan.io.

The hostnames data source tracks newly observed domains and hostnames sourced from multiple data sources including Certificate Transparency, passive DNS monitoring, existing scan data, zonefiles, etc. This data source lets customers:

• Search for new domains and hostnames even if they were not scanned on urlscan.io.
• Search for new domains and hostnames using keywords.
• Search for new domains based on infrastructure (such as A, MX, or NS records).
• Enumerate subdomains and hostnames for specific domains.
• Enumerate hostnames / domains on a specific IP address.
• Perform reconnaissance and lightweight attack surface discovery.
• Export results via an API call or as a CSV file via the UI.
• Create saved searches and email alerts. (Q1 / 2023)

Additionally, the certificates data source contains TLS certificates that have been collected from various Certificate Transparency (CT) logs. With these additions we are bringing in more of the discovery, alerting and analysis workflow into urlscan Pro that customers previously had to combine using auxiliary tools and platforms.

Hostnames & Domains by the numbers

Currently our hostnames data sources contains 1.5 billion unique hostnames across 310 million unique registered domains. More than half of these domains already have a DNS resolution attached to them.

The certificates datasource contains more than 5 billion TLS certificates issued through Certificate Transparency logs.

Both of these data source operate in real time, with new records taking no more than a minute before they appear in the search results.

Next Steps

Making these data sources available for searching is just the first step. Over the next year, these new additions will come together to form our fully automated hands-off monitoring pipeline. This new pipeline will allow our customers to set up keywords of interest within urlscan Pro and we will take care of the rest:

• We will watch for and alert on newly observed hostnames matching these keywords.
• We will try to resolve these hostnames via DNS until they have a DNS record.
• We will scan for potential websites on these hosts and alert customers when a site goes live.
• We will monitor the page content for changes, such as when a page goes from landing page to fully set up website.

Furthermore these data sources will be more tightly integrated into the urlscan Pro platform. We will add the same set of convenience features customers have been used to, including Saved Searches and E-Mail Subscriptions for new hits.

Pricing & Availability

Access to these data sources and the upcoming alerting pipeline is free of charge for customers on active subscriptions. For new customers and subscription renewals customers can choose to add this functionality as a separate subscription module.