Introduction
Welcome to the DomainResearch API documentation! By accessing our endpoints, you’ll get information on various domains in our database.
We have language bindings in Shell and Python! We’ll surely have some more soon:) On the right, you can see code examples and it is super easy to switch them using tabs in the top right.
Authentication
DomainResearch API uses basic authentication to allow access to the API
To authorize, pass the credentials with each request:
import requests
r = requests.get('https://api_endpoint_here/', auth=('user', 'password'), ...)
curl -u "user:password" "https://api_endpoint_here/" ...
Make sure to replace
user
andpassword
with your credentials.
Search endpoint
import requests
r = requests.post(
'https://domainresearch.domaincrawler.com/api/v2/search/',
auth=('user', 'password'),
json={
"query":
[
{
"field": "title",
"values":[
{
"query_type": "all_words",
"value": "airpods"
}
]
},
{
"field": "apps",
"values": [
{
"query_type": "eq",
"value": {
"categories": "Ecommerce",
"name": [
"OpenCart",
"Shopify"
]
}
}
]
},
{
"field": "tld",
"values": [
{
"query_type": "in",
"value": [
"com",
"shop"
]
}
],
}
],
"limit": 100,
"format": "json"
})
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/v2/search/' \
-H "Content-Type: application/json" \
--data-raw '{
"format": "json",
"from": 0,
"limit": 100,
"query": [
{
"field": "title",
"values": [
{
"query_type": "all_words",
"value": "airpods"
}
]
},
{
"field": "apps",
"values": [
{
"query_type": "eq",
"value": {
"categories": "Ecommerce",
"name": [
"OpenCart",
"Shopify"
]
}
}
]
},
{
"field": "tld",
"values": [
{
"query_type": "in",
"value": [
"com",
"shop"
]
}
]
}
]
}'
The above command returns JSON structured like this:
{
"domains": [
{
"domain": "example.com",
"registration_date": "2020-05-30T16:37:37.000Z",
...
},
...
],
"tlds": {
"com": 22,
"store": 2
},
"total": 24
}
Description
This is the main endpoint for searching through the data. It returns domains list for the given search query.
HTTP Request
POST https://domainresearch.domaincrawler.com/api/v2/search/
JSON parameters
Parameter | Value | Description |
---|---|---|
query | {object with filters} | Put desired filters here. See below for the detailed description about each filter. |
limit | 100 | Set the limit for the results. Put 0 to disable limit. |
format | “csv”, “json”, “xlsx” | Specify the response format. |
fields | [array of strings] | Optional. Specify output fields. |
sort | string | Optional. Defaults to domain |
Output fields
You can choose what fields should be present in a response. domain
is always included
Default output fields:
- dns
- registrar
- registrant
- rc (equal to Redirect)
All available output fields:
- tld
- parked
- registration_date
- expiration_date
- last_update_date
- created_at_utc (equal to Addition Date)
- dns
- cert_info
- status
- registrar
- registrar_id
- registrant
- registrant_phone
- registrant_fax
- registrant_email
- registrant_country
- rc (equal to Redirect)
- asn
- program (equal to Google Analytics/Google Ads)
Sorting
You can specify a field to sort results by.
To sort in descending order, put -
in the beginning, e.g. -registration_date
.
Defaults to domain
.
Available fields to sort by:
- domain
- tld
- registration_date
- expiration_date
- last_update_date
- created_at_utc
- status
- registrar
- registrar_id
- registrant
- registrant_phone
- registrant_fax
- registrant_email
- registrant_country
- asn
Extra options
Some filters support additional search options.
To use them, add extra_options
object in filter values
.
Fuzzy search
This option allows for search of strings that slightly differ from the specified string.
An edit distance is the number of one-character changes needed to turn one term into another. These changes can include:
- Changing a character (box → fox)
- Removing a character (black → lack)
- Inserting a character (sic → sick)
- Transposing two adjacent characters (act → cat)
{
"extra_options": [
{
"edit_distance": "Auto", // can be "Auto", 1, or 2
"name": "fuzzy_search",
"value": true
}
]
}
// Example for HTML Content filter
{
"field": "html_content",
"values": [
{
"query_type": "all_words",
"value": "airpods",
"extra_options": [
{
"edit_distance": "Auto",
"name": "fuzzy_search",
"value": true
}
]
},
]
}
HTML Content Filter
{
"field": "html_content", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
// Specify "all_words" if you want to search for the all words,
// "any_word" if you want to search for any word,
// "exact_phrase" if you want to search the phrase
// "exclude" if you want to exclude websites that contain specified word in this field(s).
"query_type": "all_words",
"value": "airpods" // value
},
...
]
}
This filter allows full-text search in website Title, Meta Description, Meta Keywords, H1 Text.
Supports fuzzy search.
Title Filter
{
"field": "title", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
// Specify "all_words" if you want to search for the all words,
// "any_word" if you want to search for any word,
// "exact_phrase" if you want to search the phrase
// "exclude" if you want to exclude websites that contain specified word in this field(s).
"query_type": "all_words",
"value": "airpods" // value
},
...
]
}
This filter allows full-text search in website Title.
Supports fuzzy search.
Meta Keywords Filter
{
"field": "meta_keywords", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
// Specify "all_words" if you want to search for the all words,
// "any_word" if you want to search for any word,
// "exact_phrase" if you want to search the phrase
// "exclude" if you want to exclude websites that contain specified word in this field(s).
"query_type": "all_words",
"value": "airpods" // value
},
...
]
}
This filter allows full-text search in website Meta Keywords.
Supports fuzzy search.
Meta Description Filter
{
"field": "meta_description", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
// Specify "all_words" if you want to search for the all words,
// "any_word" if you want to search for any word,
// "exact_phrase" if you want to search the phrase
// "exclude" if you want to exclude websites that contain specified word in this field(s).
"query_type": "all_words",
"value": "airpods" // value
},
...
]
}
This filter allows full-text search in website Meta Description.
Supports fuzzy search.
h1 Text Filter
{
"field": "h1_text", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator
{
// Specify "all_words" if you want to search for the all words,
// "any_word" if you want to search for any word,
// "exact_phrase" if you want to search the phrase
// "exclude" if you want to exclude websites that contain specified word in this field(s).
"query_type": "all_words",
"value": "airpods" // value
},
...
]
}
This filter allows full-text search in website h1 Text.
Supports fuzzy search.
Apps Filter
{
"field": "apps", // filter name
"values": [ // specify one or more filter values. All values are combined with AND operator.
{
"query_type": "eq", // only "eq" query type is supported
"value": {
"categories": "Ecommerce",
// specify one or more techonologies for the category.
// All values are combined with OR operator.
"name": [
"OpenCart",
"Shopify"
]
}
}
]
}
This filter allows search for domains that are using a specified technology.
Please, view DomainResearch GUI version to see all possible technologies categories and names:
TLD Filter
{
"field": "tld", // filter name
"values": [
{
"query_type": "in", // only "in" query type is supported
// Specify one or more TLDs.
// All values are combined with OR operator.
"value": [
"com",
"shop"
]
}
]
}
This filter allows search by a domain’s TLD.
Domain Name Filter
{
"field": "domain", // filter name
"values": [
{
// available query types are: "eq", "starts_with", "ends_with", "contains", "exclude" and "regex"
// queries that have wildcards in the beginning of the string
// (i.e. "endswith", "contains", and "regex" that doesn't start from the beginning) can be quite slow
"query_type": "starts_with",
"value": "cooldomain"
}
]
}
This filter allows searching a substring in domain name using substring or regular expression.
For eq
query type it supports fuzzy search.
Domain Length Filter
{
"field": "domain_length", // filter name
"values": [
{
// available query types are: "eq", "lt", "lte", "gt", "gte"
"query_type": "gt",
"value": "5"
}
]
}
This filter allows filtering by a domain length.
Redirect Filter
{
"field": "redirect", // filter name
"values": [
{
// "eq" and "exclude" query types are supported
"query_type": "eq",
"value": "domain.com"
}
]
}
This filter allows filtering by a domain redirect target. Filter includes “www.” automatically, e.g. search for “example.com” will find all domains that are redirect to example.com
or www.example.com
.
Creation Date Filter
{
"field": "registration_date", // filter name
"values": [
{
// available query types are: "eq", "lt", "lte", "gt", "gte"
"query_type": "gte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a domain Creation Date.
Expiration Date Filter
{
"field": "expiration_date", // filter name
"values": [
{
// available query types are: "eq", "lt", "lte", "gt", "gte"
"query_type": "lte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a domain Expiration Date.
Update Date Filter
{
"field": "last_update_date", // filter name
"values": [
{
// available query types are: "eq", "lt", "lte", "gt", "gte"
"query_type": "gte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a domain Update Date.
Addition Date Filter
{
"field": "addition_date", // filter name
"values": [
{
// available query types are: "eq", "lt", "lte", "gt", "gte"
"query_type": "gte",
"value": "2021-05-14"
}
]
}
This filter allows filtering by a date when a domain was added to the IV database. For example, it allows to get recently added domains when using “gt” or “gte” query type.
Registrar ID Filter
{
"field": "registrar_id", // filter name
"values": [
{
// only "in" query type is supported
"query_type": "in",
// Specify one or more Registrars IDs.
// All values are combined with OR operator.
"value": [
"123",
"456"
]
}
]
}
This filter allows search by a domain’s Registrar ID.
Registrar Name Filter
{
"field": "registrar", // filter name
"values": [
{
// only "contains" query type is supported
"query_type": "contains",
"value": "Registrar name"
}
]
}
This filter allows search by a domain’s Registrar Name.
Registrant Name Filter
{
"field": "registrant", // filter name
"values": [
{
// "eq", "contains", "starts_with", "ends_with", "regex" qyery types are supported
"query_type": "eq",
"value": "John Smith"
}
]
}
This filter allows search by a domain’s Registrant Name.
For eq
query type it supports fuzzy search.
Registrant Country Filter
{
"field": "registrant_country", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
// 2-digit country code
"value": "US"
}
]
}
This filter allows search by a domain’s Registrant Country.
Registrant Phone Filter
{
"field": "registrant_phone", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
// Phones are parsed from different WHOISes, and stored in the different formats.
// We don't change them to the single format yet, so this filter won't find the number
// if it is stored in the different format. This will be fixed in the future updates.
"value": "+1.2346789012"
}
]
}
This filter allows search by a domain’s Registrant Phone.
Registrant Fax Filter
{
"field": "registrant_fax", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
// Phones are parsed from different WHOISes, and stored in the different formats.
// We don't change them to the single format yet, so this filter won't find the number
// if it is stored in the different format. This will be fixed in the future updates.
"value": "+1.2346789012"
}
]
}
This filter allows search by a domain’s Registrant Fax.
Registrant Email Filter
{
"field": "registrant_email", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
"value": "user@domain.com"
}
]
}
This filter allows search by a domain’s Registrant Email.
DNS A Record Filter
{
"field": "A", // filter name
"values": [
{
// "wildcard" query type is supported.
// "?" matches any single character,
// "*" can match zero or more characters, including an empty one.
"query_type": "wildcard",
"value": "1.1.1.*"
}
]
}
This filter allows search by a DNS “A” record.
DNS AAAA Record Filter
{
"field": "AAAA", // filter name
"values": [
{
// "wildcard" query type is supported.
// "?" matches any single character,
// "*" can match zero or more characters, including an empty one.
"query_type": "wildcard",
"value": "2001:0db8:0000:0000:0000:0000:3257:9652"
}
]
}
This filter allows search by a DNS “AAAA” record.
DNS MX Record Filter
{
"field": "MX", // filter name
"values": [
{
// "wildcard" query type is supported.
// "?" matches any single character,
// "*" can match zero or more characters, including an empty one.
"query_type": "wildcard",
"value": "mx*.domain.com"
}
]
}
This filter allows search by a DNS “MX” record.
DNS NS Record Filter
{
"field": "NS", // filter name
"values": [
{
// "wildcard" query type is supported.
// "?" matches any single character,
// "*" can match zero or more characters, including an empty one.
"query_type": "wildcard",
"value": "*.cloudflare.com."
}
]
}
This filter allows search by a DNS “NS” record.
DNS TXT Options Filter
{
"field": "TXT", // filter name
"values": [
{
// only "startswith" query type is supported
"query_type": "startswith",
// possible options are:
// google-site-verification
// facebook-domain-verification
// atlassian-domain-verification
// adobe-idp-site-verification
// ahrefs-site-verification
// cisco-ci-domain-verification
// All options are combined with AND operator.
"value": [
"google-site-verification",
"facebook-domain-verification"
]
}
]
}
This filter allows search by some DNS TXT values. Possible options:
- google-site-verification
- facebook-domain-verification
- atlassian-domain-verification
- adobe-idp-site-verification
- ahrefs-site-verification
- cisco-ci-domain-verification
Filter will return domains that have these options present in DNS TXT.
DNS SPF Records Filter
{
"field": "TXT_SPF", // filter name
"values": [
{
// only "startswith_v=spf" query type is supported
"query_type": "startswith_v=spf",
"value": "zendesk"
}
]
}
This filter allows search by some DNS TXT SPF records.
For example, putting “zendesk” in this filter will return all domains that have a Zendesk record in their TXT SPF records.
Parked Domains Filter
{
"field": "parked_domains", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
// Specify "exclude_parked" to exclude parked domains from a search;
// "only_parked" to include only parked domains to a search.
"value": "exclude_parked"
}
]
}
This filter allows search for parked domains.
Due to the current implementation, queries with this filter can be quite slow. This will be fixed in the future updates.
Reachable/Unreachable Domains Filter
{
"field": "status", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
// value can be "only_reachable" or "only_unreachable"
// "only_reachable" shows domains that were reachable by our crawler,
// i.e. had HTTP response codes equal to 2xx or 3xx
// "only_unreachable" shows domains that were unreachable by our crawler,
// i.e. had HTTP response codes equal to 1xx, 4xx, 5xx or that didn't responded at all
"value": "only_reachable"
}
]
}
This filter allows search for reachable/unreachable domains.
SSL Certificate Issuer Filter
{
"field": "cert_issuer", // filter name
"values": [
{
// only "contains" query type is supported
"query_type": "contains",
"value": "let's encrypt"
}
]
}
This filter allows search by SSL certificate issuer.
Autonomous System Number Filter
{
"field": "asn", // filter name
"values": [
{
// only "eq" query type is supported
"query_type": "eq",
"value": "12345"
}
]
}
This filter allows search by Autonomous System Number.
Response Description
JSON
Key | Description |
---|---|
domains | The array with found domains. |
tlds | The number of found domains per TLD. |
total | The total number of found domains (even if “limit” is set, this number shows the non-limited number of found domains). |
CSV/XLSX
CSV/XLSX output provides a list of found domains with selected fields.
Domain Details endpoint
Description
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/domain_details/example.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/domain_details/example.com'
The above command returns JSON structured like this:
{
"domain": "example.com",
"dns": {
"2021-02-22 12:45:50": {
"A": [
"93.184.216.34"
],
"AAAA": [
"2606:2800:220:1:248:1893:25c8:1946"
],
"MX": [
"."
],
"NS": [
"a.iana-servers.net.",
"b.iana-servers.net."
],
"TXT": [
"v=spf1 -all",
"8j5nfqld20zpcyr8xjw0ydcfq9rk8hgm"
]
},
"2021-02-26 02:44:32": {
"A": [
"93.184.216.34"
],
"AAAA": [
"2606:2800:220:1:248:1893:25c8:1946"
],
"MX": [
"."
],
"NS": [
"a.iana-servers.net.",
"b.iana-servers.net."
],
"TXT": [
"v=spf1 -all",
"8j5nfqld20zpcyr8xjw0ydcfq9rk8hgm"
]
}
},
"apps": {
"2021-02-22 12:45:50": [
{
"categories": [
"IaaS"
],
"name": "Amazon ECS"
},
{
"categories": [
"PaaS"
],
"name": "Amazon Web Services"
},
{
"categories": [
"Containers"
],
"name": "Docker"
}
],
"2021-02-26 02:44:32": [
{
"categories": [
"Containers"
],
"name": "Docker"
},
{
"categories": [],
"name": "Linux"
},
{
"categories": [
"IaaS"
],
"name": "Amazon ECS"
},
{
"categories": [
"PaaS"
],
"name": "Amazon Web Services"
}
]
},
...
}
This endpoint allows to get all the data about specified domain.
It also accepts subdomains, but even then it returns data about a main domain, not a subdomain.
HTTP Request
GET https://domainresearch.domaincrawler.com/api/domain_details/<domain_name>
Response Description
Response is a big JSON file with various keys. Each key describes a data point. Each value is an object for a corresponding data point, that contains dates when the data was downloaded and value for each date. The result JSON is too big to be shown in the documentation, see a small example with just a few datapoints included at right.
Historical Data endpoints
Description
These endpoints provide historical data for a specified domain.
To get the historical data, first, you need to get the list of dates for which we have corresponding historical data.
For WHOIS you can skip this step if you want to get latest records (see WHOIS data
section below).
Dates list
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/dns/dates/?domain=domaincrawler.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/dns/dates/?domain=domaincrawler.com'
The above command returns JSON structured like this:
{
"resource": [
{
"date": "2021-09-13T17:56:32.431000"
},
{
"date": "2021-09-08T15:00:27.276000"
},
{
"date": "2021-09-06T01:59:48.942000"
},
...
]
}
HTTP Request
GET https://domainresearch.domaincrawler.com/api/<data_type>/dates/?domain=<domain_name>
Description
Possible data types are:
dns
- for DNS recordswhois
- for WHOIS recordscipa
- for other records (HTML, SSL, apps, robots.txt, etc.)
A response will contain all the dates for the corresponding data type in a descending order.
Historical data
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/dns/?domain=domaincrawler.com&date=2020-07-02T00:33:34.641000',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/dns/?domain=domaincrawler.com&date=2020-07-02T00:33:34.641000'
The above command returns JSON structured like this:
{
"resource": {
"date": "2020-07-02T00:33:34.641000",
"domain": "com.domaincrawler",
"values": {
"A": [
"80.248.226.34"
],
"SOA": null,
"TXT": null,
"AAAA": null,
"CNAME": null,
"NS": [
"ns2.ettnet.se.",
"ns.ettnet.se."
],
"MX": [
"smtp-in.sto-hy.se.stejtech.net.",
"smtp-in.sto-ste.se.stejtech.net."
]
}
}
}
HTTP Request
GET https://domainresearch.domaincrawler.com/api/<data_point>/?domain=<domain_name>&date=<date_from_dates_list>
Description
Possible data points are:
dns
whois
robots_txt
program
- for Google Analytics and Google Ads datahumans_txt
html_tags
- for HTML content datacert_info
- for SSL dataapps
ads_txt
WHOIS data
For WHOIS data it is possible to get latest saved WHOIS without sending data
argument:
GET https://domainresearch.domaincrawler.com/api/whois/?domain=<domain_name>
It is also possible to get the latest saved WHOIS responses for multiple domains (up to 1000) in one request via whois_list
endpoint:
POST https://domainresearch.domaincrawler.com/api/whois_list/
A list of domains must be provided in request body in JSON format with Content-Type: application/json
header set.
// JSON request body example
{
"domains": [
"domain1.com",
"domain2.com",
...
]
}
// cURL request example
curl -X POST -u "user:password" https://domainresearch.domaincrawler.com/api/whois_list/ -H "Content-Type: application/json" -d '{"domains": ["domain1.com", "domain2.com"]}'
Hosting Name endpoint
Description
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/hosting_name/domaincrawler.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/hosting_name/domaincrawler.com'
The above command returns JSON structured like this:
{
"domain": "domaincrawler.com",
"hosting": "Internet Vikings International AB"
}
or in a case of error
{
"error": "hosting_unknown",
"msg": "Can't retrieve hosting"
}
The endpoint provides a domain hosting name.
HTTP Request
GET https://domainresearch.domaincrawler.com/api/hosting_name/<domain_name>
Response Description
Returns JSON with domain
and hosting
fields.
In the case if domain isn’t found or hosted returns error hosting_unknown
.
Hosting Details endpoint
Description
import requests
r = requests.get(
'https://domainresearch.domaincrawler.com/api/hosting_details/domaincrawler.com',
auth=('user', 'password')
)
print(r.json())
curl -u "user:password" 'https://domainresearch.domaincrawler.com/api/hosting_details/domaincrawler.com'
The above command returns plain text response from Regional Internet Registry
The endpoint provides a domain hosting details.
HTTP Request
GET https://domainresearch.domaincrawler.com/api/hosting_details/<domain_name>
Response Description
Returns plain text response from Regional Internet Registry.
Errors
Error Code | Meaning |
---|---|
400 | Bad Request – Check you JSON arguments. |
401 | Unauthorized – Login or password is wrong. |
404 | Not Found – The specified endpoint could not be found. |
405 | Method Not Allowed – You tried to access endpoint with an invalid method. |
500 | Internal Server Error – We had a problem with our server. Try again later. |