Welcome to the NoSpaceships Ltd dns-logger!
This document details features provided by the NoSpaceships Ltd dns-logger and information related to its operation.
This document is aimed at all users of the dns-logger.
Almost all network transactions begin with a DNS query. This gives the DNS infrastructure insight into what services devices are wishing to connect with. This data has great value from multiple perspectives.
Operations teams can understand what services devices are accessing and how often. The data can be used to better schedule change requests, or to understand what devices are accessing a service which is to be decommissioned.
Security teams can know when devices wish to connect with domains linked to malicious activity. The same is also true for IP addresses, since DNS query responses will contain IP addresses. Response IP addresses can also give security teams’ data which can be used to pivot, e.g. a domain resolves to an IP address linked to malicious activity, the IP address can be used to identify other domains resolving the same IP address, and therefore possibly identifying other linked domains. The same is true for IP addresses, that is using the IP addresses to identify domains which resolve to them.
The dns-logger uses existing operating system packet capture features to obtain a real-time stream of DNS query request and response messages, both UDP and TCP. Messages include client and server information, detail about the question being asked, and specific response detail such as any IP addresses returned to a client.
The dns-logger maintains a memory based queue of the DNS messages and does not store data on disk, allowing it to be used on fairly busy DNS servers. Messages are streamed to one of a several configured servers in real-time, with messages being formatted using Syslog or JSON. The dns-logger automatically fails-over to another available target when one becomes unavailable.
The dns-logger has no third party dependencies, it does not require WinPCAP or npcap for example, and simply uses existing operating system provided features to capture network data.
The dns-logger aims to be the light-weight, vendor-agnostic, fast and effecient method administrators use to collect DNS activity in their environment, with fast and simple installation, and easy integraton with existing SIEM and Syslog infrastructures.
During installation the dns-logger will make a number of changes to the system.
On Linux platforms the following changes will be made:
dns-logger.service
file is placed under the /usr/lib/systemd/system
directory, otherwise a
dns-logger
file is placed under the /etc/init.d
directory, in both cases
the service is installed, enabled, and startedOn Windows platforms the following changes will be made:
dns-logger
will be created, this will be set to
start automatically, and startedFor both platforms above, the steps taken during installation are reversed.
Since the dns-logger will run on a DNS server it requires no disaster recovery or redundancy configuration, if a DNS server is unavailable so is the dns-logger instance for that server.
The dns-logger forwards DNS activity to one of several configured targets. It uses a failover strategy to continue sending messages to the next target in the case a target is unavailable.
For example, if the dns-logger had the targets 1.1.1.1
and 2.2.2.2
configured, and both were available, then the dns-logger on start will connect
and send messages to 1.1.1.1
. If that target then becomes unavailable, the
connection to it will timeout and the dns-logger will then connect to
2.2.2.2
. Even if 1.1.1.1
then becomes available, the dns-logger will
continue to send messages to 2.2.2.2
until the dns-logger is restarted, or
until 2.2.2.2
becomes unavailable. To force the dns-logger to fail-back to
1.1.1.1
the 2.2.2.2
would have to be made unavailable.
The dns-logger captures DNS activity from and to clients, and other DNS servers. Activity is formatted as messages, and then forwarded on to one of several configured targets. The dns-logger supports several formats for these messages. The first is a new-line delimited JSON format, and the second is a RFC 5424 Syslog format.
For all formats, currently a message is logged for each DNS query request (where the DNS header opcode is QUERY and the DNS header qr field is 0), and a message is logged for each DNS query response (where the DNS header opcode is QUERY and the DNS header qr field is 1).
JSON
The JSON message format is utilised by specifying the value json
for the
message-format
attribute in the <install>/config/dns-logger.config
file.
Any changes to this file require a restart of the dns-logger service.
When this format is specified, a stringified JSON string is sent to a target. Each JSON object is terminated with a new-line character.
For DNS query request messages, the following fields are logged:
timestamp
- A string containing the date and time in ISO format of when
the message was seen by the dns-logger, e.g. 2018-03-13T16:59:49.477080Z
host
- A string containing the hostname of the host which generated the
message, e.g. dns1
program
- A string containing the name of the program which generated the
message, this will always be dns-logger
pid
- The process ID of the dns-logger instance which generated the
message, e.g. 4296
client-ip
- A string containing the dotted quad formatted IP address of
the client which initiated the DNS query request, e.g. 192.168.1.16
client-port
- An unsigned integer containing the UDP or TCP port for the
client which initiated the DNS query request, e.g. 55777
server-ip
- A string containing the dotted quad formatted IP address of
the server to which the DNS query request was sent, e.g. 192.168.1.200
server-port
- An unsigned integer containing the UDP or TCP port for the
server to which the DNS query request was sent, for most users this will
always be 53
protocol
- Either the string udp
or the string tcp
indicating the
transport protocol used for the messagemessage-type
- Always the string query-request
to indicate the message
is a DNS query request messagemessage-size
- An unsigned integer specifying the total size of the DNS
message from the wire minus all other headers such as Ethernet, IP and
TCP, i.e. it covers only the DNS message header and the DNS message body
e.g. 205
flags
- An object containing the values for specific flag fields from the
DNS message header, the key is the flag name as two characters and its value
will be either 1
or 0
, currently only the rd
(Recursion Desired) flag
is included for DNS query request messages, e.g. {"rd":1}
question-name
- A string containing the DNS question the DNS query message
relates too, this field will encode specific characters using so called
percent encoding, whereby a character is replaced with the %
character
followed by two hexadecimal characters specifying the value of the replaced
character in hexadecimal, the characters replaced are mostly non-printable
characters and characters that would interfere with the message format, e.g.
the following, while not realistic, begins with the /
character and
demonstrates the encoding: %2fwww.nospaceships.com
question-type
- A string containing the DNS query type in its string form,
e.g. the string A
, when a DNS query type is unknown the string TYPEX
will be used where X
is the DNS query type code as an integer, e.g. for
A
this would be TYPE1
question-class
- A string containing the DNS query class in its string
form, e.g. the string IN
, when a DNS query class is unknown the string
CLASSX
will be used where X
is the DNS query class code as an integer,
e.g. for IN
this would be CLASS1
section-counts
- An array containing three unsigned integers, the first
specifies the total answer DNS resource records in the DNS query request,
the second specifies the total number of authority DNS resource records in
the DNS query request, and the third specifies the total number of
additional DNS resource records in the DNS query request, e.g. [0,0,1]
The following is an example DNS query request (new-lines have been added for clarity, normally this message would be formatted on a single line followed by a single new-line character):
{
"timestamp": "2018-03-13T18:04:10.4907087Z",
"host": "t20",
"program": "dns-logger",
"pid": 9676,
"client-ip": "192.168.1.199",
"client-port": 55298,
"server-ip": "192.168.1.200",
"server-port": 53,
"protocol": "udp",
"message-type": "query-request",
"message-size": 27,
"flags": {
"rd": 1
},
"question-name": "bbc.co.uk",
"question-type": "A",
"question-class": "IN",
"section-counts": [
0,
0,
0
]
}
For DNS query response messages, the following fields are logged:
timestamp
- A string containing the date and time in ISO format of when
the message was seen by the dns-logger, e.g. 2018-03-13T16:59:49.477080Z
host
- A string containing the hostname of the host which generated the
message, e.g. dns1
program
- A string containing the name of the program which generated the
message, this will always be dns-logger
pid
- The process ID of the dns-logger instance which generated the
message, e.g. 4296
client-ip
- A string containing the dotted quad formatted IP address of
the client the DNS query response was sent to, e.g. 192.168.1.16
client-port
- An unsigned integer containing the UDP or TCP port for the
client the DNS query response was sent to, e.g. 55777
server-ip
- A string containing the dotted quad formatted IP address of
the server from which the DNS query response was sent, e.g. 192.168.1.200
server-port
- An unsigned integer containing the UDP or TCP port for the
server from which the DNS query response was sent, for most users this will
always be 53
protocol
- Either the string udp
or the string tcp
indicating the
transport protocol used for the messagemessage-type
- Always the string query-response
to indicate the message
is a DNS query response messagemessage-size
- An unsigned integer specifying the total size of the DNS
message from the wire minus all other headers such as Ethernet, IP and
TCP, i.e. it covers only the DNS message header and the DNS message body
e.g. 451
flags
- An object containing the values for specific flag fields from the
DNS message header, the key is the flag name as two characters and its value
will be either 1
or 0
, currently the aa
(Authoritative Answer), tc
(Truncated Response), rd
(Recursion Desired) and ra
(Recursion Available)
flags are included for DNS query response messages, e.g.
{"aa":1,"tc":0,"rd":1,"rd":1}
question-name
- A string containing the DNS question the DNS query response
relates too, this field will encode specific characters using so called
percent encoding, whereby a character is replaced with the %
character
followed by two hexadecimal characters specifying the value of the replaced
character in hexadecimal, the characters replaced are mostly non-printable
characters and characters that would interfere with the message format, e.g.
the following, while not realistic, begins with the /
character and
demonstrates the encoding - %2fwww.nospaceships.com
question-type
- A string containing the DNS query type in its string form,
e.g. the string A
, when a DNS query type is unknown the string TYPEX
will be used where X
is the DNS query type code as an integer, e.g. for
A
this would be TYPE1
question-class
- A string containing the DNS query class in its string
form, e.g. the string IN
, when a DNS query class is unknown the string
CLASSX
will be used where X
is the DNS query class code as an integer,
e.g. for IN
this would be CLASS1
section-counts
- An array containing three unsigned integers, the first
specifies the total answer DNS resource records in the DNS query response,
the second specifies the total number of authority DNS resource records in
the DNS query response, and the third specifies the total number of
additional DNS resource records in the DNS query response, e.g. [3,2,2]
response-code
- A string containing the DNS query response code in its
string form, e.g. the string NXDOMAIN
, when a DNS query response code is
unknown the string RCODEX
will be used where X
is the DNS query response
code as an integer, e.g. for NXDOMAIN
this would be RCODE3
answer-ips
- An array of strings, each string is the dotted quad formatted
IP address from DNS resource records in the answer section in the DNS
query response if any are found, e.g. ["1.2.3.4","5.6.7.7"]
, this can be
useful from a security perspective since it exposes IP addresses DNS clients
may then go on and connect toThe following is an example DNS query response (new-lines have been added for clarity, normally this message would be formatted on a single line followed by a single new-line character):
{
"timestamp": "2018-03-13T18:04:10.5555385Z",
"host": "t20",
"program": "dns-logger",
"pid": 9676,
"client-ip": "192.168.1.199",
"client-port": 55298,
"server-ip": "192.168.1.200",
"server-port": 53,
"protocol": "udp",
"message-type": "query-response",
"message-size": 259,
"response-code": "NOERROR",
"flags": {
"aa": 0,
"tc": 0,
"rd": 1,
"ra": 1
},
"question-name": "bbc.co.uk",
"question-type": "A",
"question-class": "IN",
"section-counts": [
4,
4,
4
],
"answer-ips": [
"151.101.64.81",
"151.101.0.81",
"151.101.192.81",
"151.101.128.81"
]
}
Syslog
The Syslog message format is utilised by specifying the value syslog
for the
message-format
attribute in the <install>/config/dns-logger.config
file.
Any changes to this file require a restart of the dns-logger service.
When this format is specified, Syslog messages similar to the ISC BIND querylog format will be generated. Each Syslog message is terminated with a new-line character. There are a few differences between the ISC BIND querylog format and the format generated by the dns-logger:
+
or -
that can appear after the
DNS query type in the message) and the TCP flag (the T
which can also
appear after the DNS query type in the message) are logged, no other flags
are loggedDNS query request messages will look like the following:
<135>1 2018-03-13T18:19:40.6749336Z t20 dns-logger 14940 - - client 192.168.1.199#58105: query: bbc.co.uk IN A + (192.168.1.200) 27bytes 0 0 0
This message is broken down as follows:
<priority>1 <timestamp> <host> <program> <pid> - - client <client-ip>#<client-port>: query: <question-name> <question-class> <question-type> [+-][T] (<server-ip>) <message-size>bytes <section-counts>
These fields are defined as follows:
priority
- The syslog priority, the syslog-facility
and syslog-severity
attributes defined in the <install>/config/dns-logger.config
file are used
to construct the value for this fieldtimestamp
- A string containing the date and time in ISO format when the
message was seen by the dns-logger, e.g. 2018-03-13T16:59:49.477080Z
host
- A string containing the hostname of the host which generated the
message, e.g. dns1
program
- A string containing the name of the program which generated the
message, this will always be dns-logger
pid
- The process ID of the dns-logger instance which generated the
message, e.g. 4296
client-ip
- A string containing the dotted quad formatted IP address of
the client which initiated the DNS query request, e.g. 192.168.1.16
client-port
- An unsigned integer containing the UDP or TCP port for the
client which initiated the DNS query request, e.g. 55777
question-name
- A string containing the DNS question the DNS query request
relates too, this field will encode specific characters using so called
percent encoding, whereby a character is replaced with the %
character
followed by two hexadecimal characters specifying the value of the replaced
character in hexadecimal, the characters replaced are mostly non-printable
characters and characters that would interfere with the message format, e.g.
the following, while not realistic, begins with the /
character and
demonstrates the encoding - %2fwww.nospaceships.com
question-class
- A string containing the DNS query class in its string
form, e.g. the string IN
, when a DNS query class is unknown the string
CLASSX
will be used where X
is the DNS query class code as an integer,
e.g. for IN
this would be CLASS1
question-type
- A string containing the DNS query type in its string form,
e.g. the string A
, when a DNS query type is unknown the string TYPEX
will be used where X
is the DNS query type code as an integer, e.g. for
A
this would be TYPE1
[+-]
- Either the character -
or the character +
indicating whether
the DNS client requested recursion or notprotocol
- If TCP was utilised to perform the query the character T
will appear directory following the previous field, i.e. there will be no
white-space between themserver-ip
- A string containing the dotted quad formatted IP address of
the server to which the DNS query request was sent, e.g. 192.168.1.200
message-size
- An unsigned integer specifying the total size of the DNS
message from the wire minus all other headers such as Ethernet, IP and
TCP, i.e. it covers on the DNS message header and the DNS message body
e.g. 205
section-counts
- Three unsigned integers which are space separated, the
first specifies the total answer DNS resource records in the DNS query
request, the second specifies the total number of authority DNS resource
records in the DNS query request, and the third specifies the total number
of additional DNS resource records in the DNS query request, e.g. 0 0 1
DNS query response messages will look like the following:
<135>1 2018-03-13T18:19:40.7193243Z t20 dns-logger 14940 - - client 192.168.1.199#58105: response: bbc.co.uk IN A + (192.168.1.200) 259bytes 4 4 4 NOERROR 151.101.192.81 151.101.128.81 151.101.64.81 151.101.0.81
This message is broken down as follows:
<priority>1 <timestamp> <host> <program> <pid> - - client <client-ip>#<client-port>: response: <question-name> <question-class> <question-type> [+-][T] (<server-ip>) <message-size>bytes <section-counts> <response-code> <answer-ips>
These fields are defined as follows:
priority
- The syslog priority, the syslog-facility
and syslog-severity
attributes defined in the <install>/config/dns-logger.config
file are used
to construct the value for this fieldtimestamp
- A string containing the date and time in ISO format when the
message was seen by the dns-logger, e.g. 2018-03-13T16:59:49.477080Z
host
- A string containing the hostname of the host which generated the
message, e.g. dns1
program
- A string containing the name of the program which generated the
message, this will always be dns-logger
pid
- The process ID of the dns-logger instance which generated the
message, e.g. 4296
client-ip
- A string containing the dotted quad formatted IP address of
the client to which the DNS query response was sent, e.g. 192.168.1.16
client-port
- An unsigned integer containing the UDP or TCP port for the
client to which the DNS query response was sent, e.g. 55777
question-name
- A string containing the DNS question the DNS query response
relates too, this field will encode specific characters using so called
percent encoding, whereby a character is replaced with the %
character
followed by two hexadecimal characters specifying the value of the replaced
character in hexadecimal, the characters replaced are mostly non-printable
characters and characters that would interfere with the message format, e.g.
the following, while not realistic, begins with the /
character and
demonstrates the encoding - %2fwww.nospaceships.com
question-class
- A string containing the DNS query class in its string
form, e.g. the string IN
, when a DNS query class is unknown the string
CLASSX
will be used where X
is the DNS query class code as an integer,
e.g. for IN
this would be CLASS1
question-type
- A string containing the DNS query type in its string form,
e.g. the string A
, when a DNS query type is unknown the string TYPEX
will be used where X
is the DNS query type code as an integer, e.g. for
A
this would be TYPE1
[+-]
- Either the character -
or the character +
indicating whether
the DNS client requested recursion or notprotocol
- If TCP was utilised to perform the query the character T
will appear directory following the previous field, i.e. there will be no
white-space between themserver-ip
- A string containing the dotted quad formatted IP address of
the server from which the DNS query response was sent, e.g. 192.168.1.200
message-size
- An unsigned integer specifying the total size of the DNS
message from the wire minus all other headers such as Ethernet, IP and
TCP, i.e. it covers on the DNS message header and the DNS message body
e.g. 205
section-counts
- Three unsigned integers which are space separated, the
first specifies the total answer DNS resource records in the DNS query
response, the second specifies the total number of authority DNS resource
records in the DNS query response, and the third specifies the total number
of additional DNS resource records in the DNS query response, e.g. 0 0 1
response-code
- A string containing the DNS query response code in its
string form, e.g. the string NXDOMAIN
, when a DNS query response code is
unknown the string RCODEX
will be used where X
is the DNS query response
code as an integer, e.g. for NXDOMAIN
this would be RCODE3
answer-ips
- A space separated list of dotted quad formatted IP address
from each DNS resource records in the answer section in the DNS query
response if any are found, e.g. 1.2.3.4 5.6.7.7
, this can be useful from a
security perspective since it exposes IP addresses DNS clients may then go
on and connect toThe dns-logger program that provides the dns-logger background service can be passed a parameter to instruct it to print the messages it would normally send to a configured target to standard output instead. This feature can be used to verify an installation is working.
To have the dns-logger print messages to standard output specify the
--run-stdout
command line parameter:
Linux
cd <install>
./dns-logger.bin --run-stdout config/dns-logger.config
Windows
cd <install>
dns-logger.exe --run-stdout config\dns-logger.config
NOTE The dns-logger service does NOT need to be stopped to utilise this feature, and if started can be left running.
Simply issue a CTRL+C
to stop the dns-logger.
The above example uses the same configuration file used by the dns-logger background service, therefore messages will be formatted using the same configuration. An alternative configuration can be specified, which can contain different configuration.
NOTE This method could also be used for ad-hoc DNS activity capture when troubleshooting specific issues.