Shadow Language
The Shadow Language is used in the analysis system to make queries. The same language is also used to create detections.
General Syntax
The syntax of the language is similar to the Elastic KQL and VirusTotal query language. For each query the desired entity can be specified. Currently, there are five entities available which are documented further down. Following the entity, corresponding filter fields can be specified. Each entity has different filter fields, which are explained in more detail in the corresponding entity documentation.
If no entity is specified, the parser tries to determine the entities automatically.
The language uses an implicit AND. If desired fields are to be linked with OR, this must be explicitly specified.
If the query contains multiple entities, the partial queries must be explicitly separated using parenthesis expressions.
Example Queries
Example Domain Query for getting subdomains of google.com:
entity:domain domain:*.google.com
This can also be written as domain:*.google.com
Example Whois Query:
entity:whois registrar:WEBCC name_server:*.cloudflare.com creation_date:2022-07-01+
This can also be written as registrar:*WEBCC* name_server:*.cloudflare.com creation_date:2022-07-01+
Example Certificate Query:
entity:cert issuer:*ZeroSSL* seen:2023-08-01+ NOT domain:www.*
Example DNS Query:
entity:dns a_record:'142.250.203.*' domain:*.ch
This can also be written as a_record:'142.250.203.*' domain:*.ch
Example HTTP Query:
entity:http title:'*Swisscom*' NOT domain:*.swisscom.ch
NOT domain:swisscom.ch NOT domain:*.swisscom.com NOT domain:swisscom.com
Example Fuzzy Domain Query on Whois:
entity:whois fuzzy_domain:amazon.com creation_date:2023-09-01+
Example Whois DNS Joined Query:
(entity:whois registrar:'NameCheap, Inc.' domain:'shadow*')
AND (entity:dns ns_record:'*dreamhost.com.')
This can also be written as registrar:*NameCheap* domain:shadow* ns_record:*dreamhost.com.
Entities
The currently available entities are described below. Each entity has different fields that can be used to filter data.
Domain Entity
The domain entity contains the cumulated domain names from all data sources and their first and last seen timestamp.
The following fields can be used with this entity:
| Field | Description |
|---|---|
domain |
The domain name |
Special Fields:
| Field | Description |
|---|---|
fuzzy_domain |
Finds domains similar to the given word |
Whois Entity
The whois entity contains query Whois information. Following fields can be used with this entity:
| Field | Description |
|---|---|
domain |
The domain name |
creation_date |
The date and time when the domain was registered |
updated_date |
The date and time when the whois record was last updated |
expired_date |
The date and time when the domain will expire or expired |
statuses |
Current status |
name_server |
First configured name servers when the domain was registered |
dnssec |
description |
registrar |
The registrar where the domain was registered |
registrar_country |
The country of the registrar |
registrar_abuse_contact_phone |
The abuse phone contact of the registrar |
registrar_abuse_contact_email |
The abuse email contact of the registrar |
registrar_iana_id |
The IANA id of the registrar |
registrar_url |
The URL of the registrar |
registrar_whois_server |
The whois server provided by the registrar |
registrant_name |
The name of the registrant |
registrant_org |
The organization of the registrant |
registrant_street |
The street address of the registrant |
registrant_city |
The city of the registrant |
registrant_state |
The state of the registrant |
registrant_postal |
The postal code of the registrant |
registrant_country |
The country of the registrant |
registrant_phone |
The phone number of the registrant |
registrant_email |
The email address of the registrant |
tech_name |
The name of the technical contact |
tech_org |
The organization of the technical contact |
tech_street |
The street address of the technical contact |
tech_city |
The city of the technical contact |
tech_state |
The state of the technical contact |
tech_postal |
The poscal code of the technical contact |
tech_country |
The country of the technical contact |
tech_phone |
The phone number of the technical contact |
tech_email |
The email address of the technical contact |
admin_name |
The name of the administrative contact |
admin_org |
The organization of the administrative contact |
admin_street |
The street address of the administrative contact |
admin_city |
The city of the administrative contact |
admin_state |
The state of the administrative contact |
admin_postal |
The postcal code of the administrative contact |
admin_country |
The country of the administrative contact |
admin_phone |
The phone number of the administrative contact |
admin_email |
The email address of the administrative contact |
Special Fields:
| Field | Description |
|---|---|
contains_hyphen |
Indicates if domain contains hyphen |
fuzzy_domain |
Finds domains similar to the given word |
similar_to_top |
Finds domains which are similar to the top domains |
starts_with_nr |
Indicates if the domain starts with a number |
contains_word_from_top |
The domain contains a word from a top domain |
ends_with_nr |
Indicates if the domain ends with a number |
contains_nr |
Indicates if the domain contains a number |
domain_length |
The domain length |
tag |
Tag set on the domain |
popularity_rank |
The popularity rank of the domain |
Certificate Entity
The cert entity contains TLS certificate information. Following fields can be used with this entity:
| Field | Description |
|---|---|
cert_index |
description |
domain |
The full domain |
l1d |
The top level domain |
l2d |
The domain name |
authorityInfoAccess |
The authority info access |
authorityKeyIdentifier |
The authority key identifier |
basicConstraints |
The basic constraints |
certificatePolicies |
The certificate policies |
extendedKeyUsage |
The extended key usage |
keyUsage |
The key usage |
subjectAltName |
The subject alternative name |
crlDistributionPoints |
The CRL distribution points |
extra |
description |
issuerAltName |
The issuer alternative name |
fingerprint |
The fingerprint of the certificate |
issuer |
The name of the issuer |
issuer_email_address |
The email address of the issuer |
not_after |
The date and time until the certificate is valid |
not_before |
The date and time from when the certificate is valid |
serial_number |
The serial number of the certificate |
signature_algorithm |
The used signature algorithm |
subject |
The certificate subject |
subject_email_address |
The subject email address |
seen |
The date and time when the certificate was first seen |
source_name |
The certificate transparency log source |
source_url |
The certificate transparency log url |
Special Fields:
| Field | Description |
|---|---|
fuzzy_domain |
Finds domains similar to the given word |
tag |
Tag set on the domain |
DNS Entity
The dns entity contains all kinds of DNS information. Following fields can be used with this entity:
| Field | Description |
|---|---|
domain |
The domain name |
a_record |
The address record |
a_record_ttl |
The time to live of the address record |
aaaa_record |
the IPv6 address record |
aaaa_record_ttl |
The time to live of the IPv6 address record |
caa_record |
The certificate authority record |
caa_record_ttl |
The time to live of the certificate authority record |
cname_record |
The canonical name record |
cname_record_ttl |
The time to live of the canonical name record |
dname_record |
The DNS resource record |
dname_record_ttl |
The time to live of the DNS resource record |
mx_record |
The mail exchange resource record |
mx_record_ttl |
The time to live of the mail exchange resource record |
ns_record |
The name server record |
ns_record_ttl |
The time to live of the name server record |
soa_record |
The start of authority record |
soa_record_ttl |
The time to live of the start of authority record |
txt_record |
The text record |
txt_record_ttl |
The time to live of the text record |
Special Fields:
| Field | Description |
|---|---|
fuzzy_domain |
Finds domains similar to the given word |
ip_country |
IP location country |
asn |
The asn the IP is hosted in |
mail_server_provider |
The email server provider |
tag |
Tag set on the domain |
HTTP Entity
The http entity contains all kinds of HTTP information. Following fields can be used with this entity:
| Field | Description |
|---|---|
domain |
The domain name |
hash |
The content SHA256 hash |
header |
The header key name |
header_value |
The header value |
ip |
The IP address connected to |
port |
The port connected to |
title |
The title of the website |
meta |
The meta key name |
meta_value |
The meta value |
jarm |
The jarm fingerprint |