Introduction
Welcome to the DomainResearch API documentation! By accessing our endpoints, you’ll get information on various domains in our database.
We have language bindings in Shell and Python! We’ll surely have some more soon:) On the right, you can see code examples and it is super easy to switch them using tabs in the top right.
Authentication
To authorize, pass the credentials with each request:
import requests
r = requests.get(
'https://api_endpoint_here/',
auth=('user', 'password'),
...)
curl -u "user:password" "https://api_endpoint_here/" ...
Or, use an authentication token instead in the HTTP header:
import requests
r = requests.get(
'https://api_endpoint_here/',
headers={"X-Auth-Token", "db7b9833"},
...)
curl -H "X-Auth-Token: db7b9833" "https://api_endpoint_here/" ...
DomainResearch API provides authentication by login/password and by using tokens. The second method is preferable if you want to share access to API via your own account, but don’t want to share a plain text password. You can create up to 5 authentication tokens on the My Account page.
Search endpoint
import requests
r = requests.post(
'https://domainresearch.domaincrawler.com/api/v2/search/',
auth=('user', 'password'),
json={
"query":
[
{
"field": "title",
"values":[
{
"query_type": "all_words",
"value": "airpods"
}
]
},
{
"field": "apps",
"values": [
{
"query_type": "eq",
"value": {
"categories": "Ecommerce",
"name": [
"OpenCart",
"Shopify"
]
}
}
]
},
{
"field": "tld",
"values": [
{
"query_type": "in",
"value": [
"com",
"shop"
]
}
],
}
],
"limit": 100,
"format": "json"
})
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/v2/search/' \
-H "Content-Type: application/json" \
--data-raw '{
"format": "json",
"from": 0,
"limit": 100,
"query": [
{
"field": "title",
"values": [
{
"query_type": "all_words",
"value": "airpods"
}
]
},
{
"field": "apps",
"values": [
{
"query_type": "eq",
"value": {
"categories": "Ecommerce",
"name": [
"OpenCart",
"Shopify"
]
}
}
]
},
{
"field": "tld",
"values": [
{
"query_type": "in",
"value": [
"com",
"shop"
]
}
]
}
]
}'
The above command returns JSON structured like this:
{
"domains": [
{
"domain": "example.com",
"registration_date": "2020-05-30T16:37:37.000Z",
...
},
...
],
"tlds": {
"com": 22,
"store": 2
},
"total": 24
}
Description
This is the main endpoint for searching through the data. It returns domains list for the given search query.
HTTP Request
POST https://domainresearch.domaincrawler.com/api/v2/search/
JSON parameters
Parameter | Value | Description |
---|---|---|
query | {object with filters} | Put desired filters here. See below for the detailed description about each filter. |
limit | 100 | Set the limit for the results. Put 0 to disable limit. |
format | “csv”, “json”, “xlsx” | Specify the response format. |
fields | [array of strings] | Optional. Specify output fields. |
sort | string | Optional. Defaults to domain |
Output fields
You can choose what fields should be present in a response. domain
is always included
Default output fields:
- dns
- registrar
- registrant
- rc (equal to Redirect)
All available output fields:
- tld
- parked
- registration_date
- expiration_date
- last_update_date
- created_at_utc (equal to Addition Date)
- dns
- dnssec
- cert_info
- status
- reseller
- registrar
- registrar_id
- registrant
- registrant_phone
- registrant_fax
- registrant_email
- registrant_country
- rc (equal to Redirect)
- asn
- aso
- geo
- program (equal to Google Analytics/Google Ads)
- html_tags
- protocol (HTTP or HTTPS protocol our crawlers used to access a website)
Sorting
You can specify a field to sort results by.
To sort in descending order, put -
in the beginning, e.g. -registration_date
.
Defaults to domain
.
Available fields to sort by:
- domain
- tld
- registration_date
- expiration_date
- last_update_date
- created_at_utc
- status
- registrar
- registrar_id
- registrant
- registrant_phone
- registrant_fax
- registrant_email
- registrant_country
- asn
- aso
- geo
Extra options
Some filters support additional search options.
To use them, add extra_options
object in filter values
.
Fuzzy search
This option allows for search of strings that slightly differ from the specified string.
An edit distance is the number of one-character changes needed to turn one term into another. These changes can include:
- Changing a character (box → fox)
- Removing a character (black → lack)
- Inserting a character (sic → sick)
- Transposing two adjacent characters (act → cat)
{
"extra_options": [
{
"edit_distance": "Auto", // can be "Auto", 1, or 2
"name": "fuzzy_search",
"value": true
}
]
}
// Example for HTML Content filter
{
"field": "html_content",
"values": [
{
"query_type": "all_words",
"value": "airpods",
"extra_options": [
{
"edit_distance": "Auto",
"name": "fuzzy_search",
"value": true
}
]
}
]
}
HTML Content Filter
{
"field": "html_content", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "all_words",
"value": "airpods"
},
...
]
}
This filter allows full-text search in website Title, Meta Description, Meta Keywords, H1 Text.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
Title Filter
{
"field": "title", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "all_words",
"value": "airpods"
},
...
]
}
This filter allows full-text search in website Title.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
Meta Keywords Filter
{
"field": "meta_keywords", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "all_words",
"value": "airpods"
},
...
]
}
This filter allows full-text search in website Meta Keywords.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
Meta Description Filter
{
"field": "meta_description", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "all_words",
"value": "airpods"
},
...
]
}
This filter allows full-text search in website Meta Description.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
Meta Author Filter
{
"field": "meta_author", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "exact_phrase",
"value": "Michael Scott"
},
...
]
}
This filter allows full-text search in website Meta Author.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
Meta Content Language
{
"field": "meta_content_language", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "exact_phrase",
"value": "sv-se"
},
...
]
}
This filter allows full-text search in website Meta Content Language.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
h1 Text Filter
{
"field": "h1_text", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "all_words",
"value": "airpods"
},
...
]
}
This filter allows full-text search in website h1 Text.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Supports fuzzy search.
Apps Filter
{
"field": "apps", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator.
{
"query_type": "eq",
"value": {
"categories": "Ecommerce",
"name": [
"OpenCart",
"Shopify"
]
}
}
]
}
This filter allows search for domains that are using a specified technology.
Please, view DomainResearch GUI version to see all possible technologies categories and names:
query_type | description |
---|---|
eq | if you want to search for exact match |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
TLD Filter
{
"field": "tld", // filter name
"values": [
{
"value": [
"query_type": "in",
"com",
"shop"
]
}
]
}
This filter allows search by a domain’s TLD.
query_type | description |
---|---|
in | the list of TLDs to search for |
Domain Name Filter
{
"field": "domain", // filter name
"values": [
{
"query_type": "starts_with",
"value": "cooldomain"
}
]
}
This filter allows searching a substring in domain name using substring or regular expression.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s name |
contains | if you want to search for a domain’s name that contains a specified string |
exclude | if you want to exclude results that contain a domain’s name with given value |
regex | if you want to search for a domain’s name that matches a specified regular expression |
wildcard | if you want to search for a domain’s name that matches a specified wildcard |
starts_with | if you want to search for a domain’s name that starts with a specified string |
ends_with | if you want to search for a domain’s name that ends with a specified string |
For eq
query type it supports fuzzy search.
Domain Length Filter
{
"field": "domain_length", // filter name
"values": [
{
"query_type": "gt",
"value": "5"
}
]
}
This filter allows filtering by a domain length.
query_type | description |
---|---|
eq | if you want to search for domains with an exact length |
lt | if you want to search for domains with a length less than a given value |
lte | if you want to search for domains with a length less than or equal to a given value |
gt | if you want to search for domains with a length greater than a given value |
gte | if you want to search for domains with a length greater than or equal to a given value |
Redirect Filter
{
"field": "redirect", // filter name
"values": [
{
"query_type": "eq",
"value": "domain.com"
}
]
}
This filter allows filtering by a domain redirect target. Search occurs in each step of a redirect chain, so intermediate redirect targets will be processed too.
query_type | description |
---|---|
eq | if you want to search for a domain in the redirect chain |
contains | if you want to search for a domain in the redirect chain that contains a specified string |
starts_with | if you want to search for a domain in the redirect chain that starts with a specified string |
ends_with | if you want to search for a domain in the redirect chain that ends with a specified string |
exclude | if you want to exclude results that contain a domain in the redirect chain |
regex | if you want to search for a domain in the redirect chain that matches a specified regular expression |
wildcard | if you want to search for a domain in the redirect chain that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Creation Date Filter
{
"field": "registration_date", // filter name
"values": [
{
"query_type": "gte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a domain Creation Date.
query_type | description |
---|---|
eq | if you want to search for domains with an exact creation date |
lt | if you want to search for domains with a creation date less than a given value |
lte | if you want to search for domains with a creation date less than or equal to a given value |
gt | if you want to search for domains with a creation date greater than a given value |
gte | if you want to search for domains with a creation date greater than or equal to a given value |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Expiration Date Filter
{
"field": "expiration_date", // filter name
"values": [
{
"query_type": "lte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a domain Expiration Date.
query_type | description |
---|---|
eq | if you want to search for domains with an exact expiration date |
lt | if you want to search for domains with a expiration date less than a given value |
lte | if you want to search for domains with a expiration date less than or equal to a given value |
gt | if you want to search for domains with a expiration date greater than a given value |
gte | if you want to search for domains with a expiration date greater than or equal to a given value |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Update Date Filter
{
"field": "last_update_date", // filter name
"values": [
{
"query_type": "gte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a domain Update Date.
query_type | description |
---|---|
eq | if you want to search for domains with an exact update date |
lt | if you want to search for domains with an update date less than a given value |
lte | if you want to search for domains with an update date less than or equal to a given value |
gt | if you want to search for domains with an update date greater than a given value |
gte | if you want to search for domains with an update date greater than or equal to a given value |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Addition Date Filter
{
"field": "addition_date", // filter name
"values": [
{
"query_type": "gte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a date when a domain was added to the IV database. For example, it allows to get recently added domains when using “gt” or “gte” query type.
query_type | description |
---|---|
eq | if you want to search for domains with an addition update date |
lt | if you want to search for domains with an addition date less than a given value |
lte | if you want to search for domains with an addition date less than or equal to a given value |
gt | if you want to search for domains with an addition date greater than a given value |
gte | if you want to search for domains with an addition date greater than or equal to a given value |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrar ID Filter
{
"field": "registrar_id", // filter name
"values": [
{
"query_type": "in",
"value": [
"123",
"456"
]
}
]
}
This filter allows search by a domain’s Registrar ID.
query_type | description |
---|---|
in | if you want to search for a domain’s registrar id from a specified list |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrar Name Filter
{
"field": "registrar", // filter name
"values": [
{
"query_type": "contains",
"value": "Registrar name"
}
]
}
This filter allows search by a domain’s Registrar Name.
query_type | description |
---|---|
contains | if you want to search for a domain’s registrar name that contains a specified string |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant Name Filter
{
"field": "registrant", // filter name
"values": [
{
"query_type": "eq",
"value": "John Smith"
}
]
}
This filter allows search by a domain’s Registrant Name.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant name |
contains | if you want to search for a domain’s registrant name that contains a specified string |
starts_with | if you want to search for a domain’s registrant name that starts with a specified string |
ends_with | if you want to search for a domain’s registrant name that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant name with given value |
regex | if you want to search for a domain’s registrant name that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant name that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
For eq
query type it supports fuzzy search.
Registrant Country Filter
{
"field": "registrant_country", // filter name
"values": [
{
"query_type": "eq",
"value": "US"
}
]
}
This filter allows search by a domain’s Registrant Country.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrar name |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant Phone Filter
{
"field": "registrant_phone", // filter name
"values": [
{
"query_type": "eq",
"value": "+1.2346789012"
}
]
}
This filter allows search by a domain’s Registrant Phone.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant phone |
contains | if you want to search for a domain’s registrant phone that contains a specified string |
starts_with | if you want to search for a domain’s registrant phone that starts with a specified string |
ends_with | if you want to search for a domain’s registrant phone that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant phone with given value |
regex | if you want to search for a domain’s registrant phone that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant phone that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant Fax Filter
{
"field": "registrant_fax", // filter name
"values": [
{
"query_type": "eq",
"value": "+1.2346789012"
}
]
}
This filter allows search by a domain’s Registrant Fax.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant fax |
contains | if you want to search for a domain’s registrant fax that contains a specified string |
starts_with | if you want to search for a domain’s registrant fax that starts with a specified string |
ends_with | if you want to search for a domain’s registrant fax that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant fax with given value |
regex | if you want to search for a domain’s registrant fax that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant fax that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant Email Filter
{
"field": "registrant_email", // filter name
"values": [
{
"query_type": "eq",
"value": "user@domain.com"
}
]
}
This filter allows search by a domain’s Registrant Email.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant email |
contains | if you want to search for a domain’s registrant email that contains a specified string |
starts_with | if you want to search for a domain’s registrant email that starts with a specified string |
ends_with | if you want to search for a domain’s registrant email that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant email with given value |
regex | if you want to search for a domain’s registrant email that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant email that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant City Filter
{
"field": "registrant_city", // filter name
"values": [
{
"query_type": "eq",
"value": "Tempe"
}
]
}
This filter allows search by a domain’s Registrant City.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant city |
contains | if you want to search for a domain’s registrant city that contains a specified string |
starts_with | if you want to search for a domain’s registrant city that starts with a specified string |
ends_with | if you want to search for a domain’s registrant city that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant city with given value |
regex | if you want to search for a domain’s registrant city that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant city that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant State Filter
{
"field": "registrant_state", // filter name
"values": [
{
"query_type": "eq",
"value": "Arizona"
}
]
}
This filter allows search by a domain’s Registrant State.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant state |
contains | if you want to search for a domain’s registrant state that contains a specified string |
starts_with | if you want to search for a domain’s registrant state that starts with a specified string |
ends_with | if you want to search for a domain’s registrant state that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant state with given value |
regex | if you want to search for a domain’s registrant state that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant state that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant Postal Code Filter
{
"field": "registrant_postal_code", // filter name
"values": [
{
"query_type": "eq",
"value": "85284"
}
]
}
This filter allows search by a domain’s Registrant Postal Code.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s registrant postal code |
contains | if you want to search for a domain’s registrant postal code that contains a specified string |
starts_with | if you want to search for a domain’s registrant postal code that starts with a specified string |
ends_with | if you want to search for a domain’s registrant postal code that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s registrant postal code with given value |
regex | if you want to search for a domain’s registrant postal code that matches a specified regular expression |
wildcard | if you want to search for a domain’s registrant postal code that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Registrant Street Filter
{
"field": "registrant_street", // filter name
"values": [
{
"query_type": "any_word",
"value": "Sunnyside"
}
]
}
This filter allows full-text search in Registrant Street.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Reseller Name Filter
{
"field": "reseller", // filter name
"values": [
{
"query_type": "wildcard",
"value": "*Test*"
}
]
}
This filter allows search by a domain’s Registrant Email.
query_type | description |
---|---|
eq | if you want to search for exact match for a domain’s reseller name |
contains | if you want to search for a domain’s reseller name that contains a specified string |
starts_with | if you want to search for a domain’s reseller name that starts with a specified string |
ends_with | if you want to search for a domain’s reseller name that ends with a specified string |
exclude | if you want to exclude results that contain a domain’s reseller name with given value |
regex | if you want to search for a domain’s reseller name that matches a specified regular expression |
wildcard | if you want to search for a domain’s reseller name that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNS A Record Filter
{
"field": "A", // filter name
"values": [
{
"query_type": "wildcard",
"value": "1.1.???.*"
}
]
}
This filter allows search by a DNS “A” record.
query_type | description |
---|---|
eq | if you want to search for exact match for a DNS A record |
contains | if you want to search for a DNS A record that contains a specified string |
starts_with | if you want to search for a DNS A record that starts with a specified string |
ends_with | if you want to search for a DNS A record that ends with a specified string |
exclude | if you want to exclude results that contain a DNS A record with given value |
regex | if you want to search for a DNS A record that matches a specified regular expression |
wildcard | if you want to search for a DNS A record that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNS AAAA Record Filter
{
"field": "AAAA", // filter name
"values": [
{
"query_type": "wildcard",
"value": "2001:0db8:0000:0000:0000:0000:3257:9652"
}
]
}
This filter allows search by a DNS “AAAA” record.
query_type | description |
---|---|
eq | if you want to search for exact match for a DNS AAAA record |
contains | if you want to search for a DNS AAAA record that contains a specified string |
starts_with | if you want to search for a DNS AAAA record that starts with a specified string |
ends_with | if you want to search for a DNS AAAA record that ends with a specified string |
exclude | if you want to exclude results that contain a DNS AAAA record with given value |
regex | if you want to search for a DNS AAAA record that matches a specified regular expression |
wildcard | if you want to search for a DNS AAAA record that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNS MX Record Filter
{
"field": "MX", // filter name
"values": [
{
"query_type": "wildcard",
"value": "mx*.domain.com"
}
]
}
This filter allows search by a DNS “MX” record.
query_type | description |
---|---|
eq | if you want to search for exact match for a DNS MX record |
contains | if you want to search for a DNS MX record that contains a specified string |
starts_with | if you want to search for a DNS MX record that starts with a specified string |
ends_with | if you want to search for a DNS MX record that ends with a specified string |
exclude | if you want to exclude results that contain a DNS MX record with given value |
regex | if you want to search for a DNS MX record that matches a specified regular expression |
wildcard | if you want to search for a DNS MX record that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNS NS Record Filter
{
"field": "NS", // filter name
"values": [
{
"query_type": "wildcard",
"value": "*.cloudflare.com."
}
]
}
This filter allows search by a DNS “NS” record.
query_type | description |
---|---|
eq | if you want to search for exact match for a DNS NS record |
contains | if you want to search for a DNS NS record that contains a specified string |
starts_with | if you want to search for a DNS NS record that starts with a specified string |
ends_with | if you want to search for a DNS NS record that ends with a specified string |
exclude | if you want to exclude results that contain a DNS NS record with given value |
regex | if you want to search for a DNS NS record that matches a specified regular expression |
wildcard | if you want to search for a DNS NS record that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNS TXT Filter
{
"field": "TXT", // filter name
"values": [
{
"query_type": "starts_with",
"value": [
"google-site-verification",
"facebook-domain-verification"
]
}
]
}
This filter allows search by some DNS TXT values. Possible options:
- google-site-verification
- facebook-domain-verification
- atlassian-domain-verification
- adobe-idp-site-verification
- ahrefs-site-verification
- cisco-ci-domain-verification
Filter will return domains that have these options present in DNS TXT.
query_type | description |
---|---|
starts_with | if you want to search for a value that starts with a specified string |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNS SPF Records Filter
{
"field": "TXT_SPF", // filter name
"values": [
{
"query_type": "startswith_v=spf",
"value": "zendesk"
}
]
}
This filter allows search by some DNS TXT SPF records.
For example, putting “zendesk” in this filter will return all domains that have a Zendesk record in their TXT SPF records.
query_type | description |
---|---|
startswith_v=spf | if you want to search for DNS SPF record that starts with a specified string |
DNS TXT Full-text Search filter
{
"field": "TXT_full_text_search", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
"query_type": "all_words",
"value": "google"
},
...
]
}
This filter allows a full-text search in a DNS “TXT” record.
query_type | description |
---|---|
all_words | if you want to search for the all words |
any_word | if you want to search for any word |
exact_phrase | if you want to search the phrase |
exclude | if you want to exclude websites that contain specified word in this field(s) |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
DNSSEC filter
{
"field": "dnssec", // filter name
"values": [
{
"query_type": "has_value",
"value": true
},
...
]
}
This filter allows to filter domains by DNSSEC presence.
query_type | description |
---|---|
has_value | shows only domains that has/doesn’t have a value for this field. Accepts boolean values |
Parked Domains Filter
{
"field": "parked_domains", // filter name
"values": [
{
"query_type": "eq",
"value": "exclude_parked"
}
]
}
This filter allows search for parked domains.
query_type | description |
---|---|
eq | set “exclude_parked” in the value field to exclude parked domains from the search results. Or set “only_parked” to search for only parked domains. |
Due to the current implementation, queries with this filter can be quite slow. This will be fixed in the future updates.
Reachable/Unreachable Domains Filter
{
"field": "status", // filter name
"values": [
{
"query_type": "eq",
"value": "only_reachable"
}
]
}
This filter allows search for reachable/unreachable domains.
query_type | description |
---|---|
eq | set “only_reachable” in the value field to exclude unreachable domains from the search results (i.e. those that had HTTP response codes equal to 1xx, 4xx, 5xx or that didn’t respond at all). Or set “only_unreachable” to search for only unreachable domains |
Registered/Not registered Domains Filter
{
"field": "registered", // filter name
"values": [
{
"query_type": "eq",
"value": "registered"
}
]
}
This filter allows search for registered/not registered domains.
query_type | description |
---|---|
eq | set “registered” in the value field to show only registered domains. Or set “not_registered” to search for only not registered domains |
SSL Certificate Issuer Filter
{
"field": "cert_issuer", // filter name
"values": [
{
"query_type": "contains",
"value": "let's encrypt"
}
]
}
This filter allows search by SSL certificate issuer.
query_type | description |
---|---|
eq | if you want to search for exact match for a SSL certificate issuer |
contains | if you want to search for a SSL certificate issuer that contains a specified string |
starts_with | if you want to search for a SSL certificate issuer that starts with a specified string |
ends_with | if you want to search for a SSL certificate issuer that ends with a specified string |
exclude | if you want to exclude results that contain a SSL certificate issuer with given value |
regex | if you want to search for a SSL certificate issuer that matches a specified regular expression |
wildcard | if you want to search for a SSL certificate issuer that matches a specified wildcard |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Autonomous System Number Filter
{
"field": "asn", // filter name
"values": [
{
"query_type": "eq",
"value": "12345"
}
]
}
This filter allows search by Autonomous System Number.
query_type | description |
---|---|
eq | if you want to search for domains with an exact ASN value |
lt | if you want to search for domains with an ASN value less than a given value |
lte | if you want to search for domains with an ASN value less than or equal to a given value |
gt | if you want to search for domains with an ASN value greater than a given value |
gte | if you want to search for domains with an ASN value greater than or equal to a given value |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Autonomous System Organization Filter
{
"field": "aso", // filter name
"values": [
{
"query_type": "wildcard",
"value": "* AG"
}
]
}
This filter allows search by Autonomous System Organization.
query_type | description |
---|---|
eq | if you want to search for exact match for an ASO value |
contains | if you want to search for an ASO value that contains a specified string |
exclude | if you want to exclude results that contain an ASO value with given value |
regex | if you want to search for an ASO value that matches a specified regular expression |
wildcard | if you want to search for an ASO value that matches a specified wildcard |
starts_with | if you want to search for an ASO value that starts with a specified string |
ends_with | if you want to search for an ASO value that ends with a specified string |
GeoIP Filter
{
"field": "geo", // filter name
"values": [
{
"query_type": "eq",
"value": "DE"
}
]
}
This filter allows search by GeoIP country code.
query_type | description |
---|---|
eq | if you want to search for exact match for a country code |
contains | if you want to search for a country code that contains a specified string |
exclude | if you want to exclude results that contain a country code with given value |
regex | if you want to search for a country code that matches a specified regular expression |
wildcard | if you want to search for a country code that matches a specified wildcard |
starts_with | if you want to search for a country code that starts with a specified string |
ends_with | if you want to search for a country code that ends with a specified string |
Google Analytics Filter
{
"field": "google_analytics", // filter name
"values": [
{
"query_type": "eq",
"value": "UA-10876-1"
}
]
}
This filter allows search by Google Analytics ID.
query_type | description |
---|---|
eq | if you want to search for exact match for a Google Analytics ID |
contains | if you want to search for a Google Analytics ID that contains a specified string |
exclude | if you want to exclude results that contain a Google Analytics ID with given value |
regex | if you want to search for a Google Analytics ID that matches a specified regular expression |
wildcard | if you want to search for a Google Analytics ID that matches a specified wildcard |
starts_with | if you want to search for a Google Analytics ID that starts with a specified string |
ends_with | if you want to search for a Google Analytics ID that ends with a specified string |
has_value | if you want to show only websites that has/doesn’t have a value for this field. Accepts boolean values |
Hosting Name Filter
{
"field": "hosting_name", // filter name
"values": [
{
"query_type": "eq",
"value": "cloudflare"
}
]
}
This filter allows search by Hosting Name.
query_type | description |
---|---|
eq | if you want to search for exact match for a hosting name |
Response Description
JSON
Key | Description |
---|---|
domains | The array with found domains. |
tlds | The number of found domains per TLD. |
total | The total number of found domains (even if “limit” is set, this number shows the non-limited number of found domains). |
CSV/XLSX
CSV/XLSX output provides a list of found domains with selected fields.
Domain Details endpoint
Description
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/domain_details/example.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/domain_details/example.com'
The above command returns JSON structured like this:
{
"domain": "example.com",
"dns": {
"2021-02-22 12:45:50": {
"A": [
"93.184.216.34"
],
"AAAA": [
"2606:2800:220:1:248:1893:25c8:1946"
],
"MX": [
"."
],
"NS": [
"a.iana-servers.net.",
"b.iana-servers.net."
],
"TXT": [
"v=spf1 -all",
"8j5nfqld20zpcyr8xjw0ydcfq9rk8hgm"
]
},
"2021-02-26 02:44:32": {
"A": [
"93.184.216.34"
],
"AAAA": [
"2606:2800:220:1:248:1893:25c8:1946"
],
"MX": [
"."
],
"NS": [
"a.iana-servers.net.",
"b.iana-servers.net."
],
"TXT": [
"v=spf1 -all",
"8j5nfqld20zpcyr8xjw0ydcfq9rk8hgm"
]
}
},
"apps": {
"2021-02-22 12:45:50": [
{
"categories": [
"IaaS"
],
"name": "Amazon ECS"
},
{
"categories": [
"PaaS"
],
"name": "Amazon Web Services"
},
{
"categories": [
"Containers"
],
"name": "Docker"
}
],
"2021-02-26 02:44:32": [
{
"categories": [
"Containers"
],
"name": "Docker"
},
{
"categories": [],
"name": "Linux"
},
{
"categories": [
"IaaS"
],
"name": "Amazon ECS"
},
{
"categories": [
"PaaS"
],
"name": "Amazon Web Services"
}
]
},
...
}
This endpoint allows to get all the data about specified domain.
It also accepts subdomains, but even then it returns data about a main domain, not a subdomain.
HTTP Request
GET https://domainresearch.domaincrawler.com/api/domain_details/<domain_name>
Response Description
Response is a big JSON file with various keys. Each key describes a data point. Each value is an object for a corresponding data point, that contains dates when the data was downloaded and value for each date. The result JSON is too big to be shown in the documentation, see a small example with just a few datapoints included at right.
Historical Data endpoints
Description
These endpoints provide historical data for a specified domain.
To get the historical data, first, you need to get the list of dates for which we have corresponding historical data.
For WHOIS you can skip this step if you want to get latest records (see WHOIS data
section below).
Dates list
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/dns/dates/?domain=domaincrawler.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/dns/dates/?domain=domaincrawler.com'
The above command returns JSON structured like this:
{
"resource": [
{
"date": "2021-09-13T17:56:32.431000"
},
{
"date": "2021-09-08T15:00:27.276000"
},
{
"date": "2021-09-06T01:59:48.942000"
},
...
]
}
HTTP Request
GET https://domainresearch.domaincrawler.com/api/<data_type>/dates/?domain=<domain_name>
Description
Possible data types are:
dns
- for DNS recordswhois
- for WHOIS recordscipa
- for other records (HTML, SSL, apps, robots.txt, etc.)
A response will contain all the dates for the corresponding data type in a descending order.
Historical data
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/dns/?domain=domaincrawler.com&date=2020-07-02T00:33:34.641000',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/dns/?domain=domaincrawler.com&date=2020-07-02T00:33:34.641000'
The above command returns JSON structured like this:
{
"resource": {
"date": "2020-07-02T00:33:34.641000",
"domain": "com.domaincrawler",
"values": {
"A": [
"80.248.226.34"
],
"SOA": null,
"TXT": null,
"AAAA": null,
"CNAME": null,
"NS": [
"ns2.ettnet.se.",
"ns.ettnet.se."
],
"MX": [
"smtp-in.sto-hy.se.stejtech.net.",
"smtp-in.sto-ste.se.stejtech.net."
]
}
}
}
HTTP Request
GET https://domainresearch.domaincrawler.com/api/<data_point>/?domain=<domain_name>&date=<date_from_dates_list>
Description
Possible data points are:
dns
whois
robots_txt
program
- for Google Analytics and Google Ads datahumans_txt
html_tags
- for HTML content datacert_info
- for SSL dataapps
ads_txt
WHOIS data
For WHOIS data it is possible to get latest saved WHOIS without sending data
argument:
GET https://domainresearch.domaincrawler.com/api/whois/?domain=<domain_name>
It is also possible to get the latest saved WHOIS responses for multiple domains (up to 1000) in one request via whois_list
endpoint:
POST https://domainresearch.domaincrawler.com/api/whois_list/
A list of domains must be provided in request body in JSON format with Content-Type: application/json
header set.
// JSON request body example
{
"domains": [
"domain1.com",
"domain2.com",
...
]
}
// cURL request example
curl -X POST -u "user:password" https://domainresearch.domaincrawler.com/api/whois_list/ -H "Content-Type: application/json" -d '{"domains": ["domain1.com", "domain2.com"]}'
Hosting Name endpoint
Description
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/hosting_name/domaincrawler.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/hosting_name/domaincrawler.com'
The above command returns JSON structured like this:
{
"domain": "domaincrawler.com",
"hosting": "Internet Vikings International AB"
}
or in a case of error
{
"error": "hosting_unknown",
"msg": "Can't retrieve hosting"
}
The endpoint provides a domain hosting name.
HTTP Request
GET https://domainresearch.domaincrawler.com/api/hosting_name/<domain_name>
Response Description
Returns JSON with domain
and hosting
fields.
In the case if domain isn’t found or hosted returns error hosting_unknown
.
Hosting Details endpoint
Description
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/hosting_details/domaincrawler.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/hosting_details/domaincrawler.com'
The above command returns plain text response from Regional Internet Registry
The endpoint provides a domain hosting details.
HTTP Request
GET https://domainresearch.domaincrawler.com/api/hosting_details/<domain_name>
Response Description
Returns plain text response from Regional Internet Registry.
Bulk Domains Search endpoint
Description
import requests
r = requests.post(
'https://domainresearch.domaincrawler.com/api/v2/bulk_domains/',
auth=('user', 'password'),
json={
"domains": [
"example.com",
"domaincrawler.com"
],
"fields": [
"parked",
"dns",
"hosting",
"registrar",
"registrant",
"rc"
],
"sort": "domain",
"format": "json",
"from": 0,
"limit": 100
}
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/v2/bulk_domains/' \
-H "Content-Type: application/json" \
--data-raw '{
"domains": [
"example.com",
"domaincrawler.com"
],
"fields": [
"parked",
"dns",
"hosting",
"registrar",
"registrant",
"rc"
],
"sort": "domain",
"format": "json",
"from": 0,
"limit": 100
}'
The above command returns data associated with the list of domains.
The endpoint provides batch loading data for the list of domains.
HTTP Request
POST https://domainresearch.domaincrawler.com/api/v2/bulk_domains/
Response Description
Returns data associated with the list of domains.
Output fields
You can choose what fields should be present in a response. domain
is always included
All available output fields:
- tld
- parked
- registration_date
- expiration_date
- last_update_date
- created_at_utc (equal to Addition Date)
- dns
- cert_info
- status
- reseller
- registrar
- registrar_id
- registrant
- registrant_phone
- registrant_fax
- registrant_email
- registrant_country
- rc (equal to Redirect)
- apps
- asn
- aso
- geo
- program (equal to Google Analytics/Google Ads)
- html_tags
Sorting
You can specify a field to sort results by.
To sort in descending order, put -
in the beginning, e.g. -registration_date
.
Defaults to domain
.
Available fields to sort by:
- domain
- tld
- registration_date
- expiration_date
- last_update_date
- created_at_utc
- status
- registrar
- registrar_id
- registrant
- registrant_phone
- registrant_fax
- registrant_email
- registrant_country
- asn
- aso
- geo
Errors
Error Code | Meaning |
---|---|
400 | Bad Request – Check you JSON arguments. |
401 | Unauthorized – Login or password is wrong. |
404 | Not Found – The specified endpoint could not be found. |
405 | Method Not Allowed – You tried to access endpoint with an invalid method. |
500 | Internal Server Error – We had a problem with our server. Try again later. |