Raw Logs (paid feature). Export logs to your storage

What is a Raw Logs feature?
Export to an S3 storage
    • Amazon storage
    • Non-Amazon storage
Export to an FTP/SFTP storage
Export time intervals
Log path example
Log format
Log example
Log fields

What is a Raw Logs feature?

Raw Logs is an option that enables an automatic export of CDN resource logs to your storage. Logs contain information about user requests sent to cache servers and pre-cache servers (if origin shielding is enabled). 

Note: The feature is paid. To activate, contact us via support@gcore.com. After activation, enable "Raw Logs" in your control panel and configure export to S3, FTP, or SFTP storage.

Export logs to an S3 storage

Amazon storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers. 

3. For storage type, select "Amazon".

4. Specify your access key ID. In your Amazon personal account, it is called "AWS access Key ID". You can find it using the instruction. An access key ID and secret access key are required to configure log export to your storage.

5. Specify your secret access key. In your Amazon personal account, it is called "AWS secret access Key". You can find it using the instruction.

6. Specify your AWS region — the location of a server where your storage is hosted. This is optional: for most storages, the region is determined automatically. You can leave the field empty. But we recommend filling it out to ensure that your logs are exported successfully.

7. Choose how to organize storage: put logs of all CDN resources into one bucket or use separate buckets for each CDN resource. 

8. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. Specify a folder name if you want to export logs to a specific folder within a bucket.  

9. Click "Save changes".

Non-Amazon storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. For storage type, select "Other".

4. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Hostname" field.

5. Specify your access key ID. Along with a secret access key, it is required to configure log export to your storage. If you use Gcore S3 storage, you can find its access key ID in your personal account in the "Access key" field.

6. Specify your secret access key. If you use Gcore S3 storage, you can find its secret access key in your personal account in the "Secret key" field.

7. Specify a bucket hostname — a bucket ID that is used by your S3 storage system in the {bucket_name}.{hostname} format. It is required to ensure that logs are exported to a correct bucket within a storage. This field is optional: for some storages, a bucket hostname is determined automatically. If you use Gcore or Yandex.Cloud storage, a bucket hostname is required. A bucket hostname of the Gcore storage looks as follows: {bucket name}.{hostname from step 3}. For example: examplename.s-ed1.cloud.gcore.lu. A bucket hostname of Yandex.Cloud storage looks as follows: {bucket name}.{Yandex.Cloud hostname}}. For example: examplename.storage.yandexcloud.net.

8. Specify a region — location ID of a server where your storage is hosted. This is optional: for some storages, the region is determined automatically. You can leave the field empty. If you use Gcore S3 storage, a location ID is required. You can find it in the "Details" of the storage. Your location ID is a part of your hostname to the first dot.

9. Choose how to organize storage: put logs of all CDN resources into one bucket or to use separate buckets for each CDN resource.

10. Specify bucket(s) for log export. Make sure you indicate an existing bucket. Otherwise, your logs cannot be exported. If you want to export logs to a specific folder within a bucket, specify a folder name.

11. Click "Save changes".

Export logs to an FTP/SFTP storage

1. Leave the box "Do not send empty logs" checked if you don't want to receive empty logs. If you want to receive empty logs, uncheck it.

2. If you use our shielding option, you can see a checkbox "Add logs from origin shielding". Check the box if you want to receive logs from both edge servers and pre-cache shielding servers.

3. Specify a hostname — a name that is assigned to a storage server within a network and is used instead of an IP address. If you use Gcore SFTP storage, you can find its hostname in the "Details" of the storage in the "Hostname" field. It looks as follows: ams.origin.gcdn.co. Additionally, you can specify an FTP or SFTP storage port by adding a colon after the hostname. For example: ams.origin.gcore.co:2200.

4. Specify a storage username. If you use Gcore SFTP storage, you can find its username in the "Details" of your storage in the "Storage/User name" field.

5. Enter your storage password. 

6. Specify a folder for export. If you use Gcore SFTP storage, specify the root (home) folder where other folders originate from. You can find its name in the "Details" of your SFTP storage at the end of the "Upload path" field.

If you use an SFTP storage from another provider, clarify whether a root folder that includes other folders is created by default. If not, leave the field empty. If yes, specify a folder name. 

7. Choose how to organize storage: put logs of all CDN resources into one folder or to use separate folders for each CDN resource. Then specify a folder name. If you specify a non-existent folder, logs will be exported to a root folder. 

8. Click "Save changes".

Export time intervals

Logs are exported at the end of each hour. If you activate Raw Logs at 00:30, the first logs will be exported between 00:45 and 01:00, and the next ones — between 01:45 and 02:00.

If CDN servers are not requested and the box "Do not send empty logs" is unchecked, an empty log file (± 20 bytes) will be sent to your storage.

You can see the status of the Raw Logs option in your control panel:

• "Pending" is a status for the time interval between the connection to a storage and the very first log export
• "OK" is a status showing that logs are exported from at least one CDN server
• "Failed" is a status indicating that an error occurred while connecting to a storage or that the service failed to export logs within 24 hours
• "Pause" is a status showing that the option is paused

Log path example

s3://log-bucket-name/2019/08/20/15/nodename_primarycname.domain.ru_access.log.gz

Log format

"$remote_addr" "-" "$remote_user" "[$time_local]" "$request" "$status"
"$body_bytes_sent" "$http_referer" "$http_user_agent" "$bytes_sent"
"$edgename" "$scheme" "$host" "$request_time"
"$upstream_response_time" "$request_length" "$http_range" "[$responding_node]"
"$upstream_cache_status" "$upstream_response_length" "$upstream_addr"
"$gcdn_api_client_id" "$gcdn_api_resource_id" "$uid_got" "$uid_set"
"$geoip_country_code" "$geoip_city" "$shield_type" "$server_addr" "$server_port"
"$upstream_status" "-" "$upstream_connect_time" "$upstream_header_time"
"$shard_addr" "$geoip2_data_asnumber" "$connection" "$connection_requests"
"$request_id" "$http_x_forwarded_proto" "$http_x_forwarded_request_id" "$ssl_cipher"
"$ssl_session_id" "$ssl_session_reused"
"$sent_http_content_type" "$tcpinfo_rtt" "$tcpi_total_retrans" "$tcpinfo_snd_cwnd" 

Please don’t be surprised if you see a field that is not listed above. We occasionally add new fields. If some fields are added to logs, you will receive an email about it. New fields are added to the end of the line.

Log example

"0.0.0.0" "-" "-" "[26/Apr/2019:09:47:40 +0000]" "GET /ContentCommon/images/image.png HTTP/1.1"
"200" "1514283" "https://example.com/videos/10" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36"
"1514848" "[dh-up-gc18]" "https" "origin.cdn.com" "1.500" "0.445" "157" "bytes=0-1901653" "[dh]"
"MISS" "10485760" "0.0.0.0:80" "2510" "7399" "-" "-" "KZ" "-" "shield_no" "0.0.0.0" "80" "206" "-" "0.000"
"0.200" "0.0.0.0" "asnumber" "106980391" "1" "c1c0f12ab35b7cccccd5dc0a454879c5" "-" "-"
"ECDHE-RSA-AES256-GCM-SHA384" "28a4184139cb43cdc79006cf2d1a4ac93bdc****" "r"
"application/json" "21" "10" "45"

Log fields

Not all fields are important. Some of them relate to our internal CDN system and are not meaningful for you. In the table below, we have highlighted such system fields in italics. Other fields can be helpful for traffic analysis or statistics. 

Field	Log value example	Description
$remote_addr	0.0.0.0	User's IP address
$remote_user (internal system variable)	-	Username used in Basic authentication
[$time_local]	[26/Apr/2019:09:47:40 +0000]	Local time in Common Log Format
$request	GET /ContentCommon/images/image.png HTTP/1.1	HTTP method, requested file path, HTTP version
$status	200	Response status code from a CDN server
$body_bytes_sent	1514283	Number of bytes sent to a user, excluding the response header size
$http_referer	https://example.com/videos/10	Referrer - a URL requested by a user
$http_user_agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 YaBrowser/16.10.0.2309 Safari/537.36	User agent that was used to send a request (browser or other application)
$bytes_sent	1514848	Number of bytes sent to a user
$edgename	[dh-up-gc18]	CDN server that forwarded a requested file
$scheme	https	Protocol (HTTP or HTTPS) of a request
$host	cdn.example.com	Requested hostname of a CDN resource
$request_time	1.500	Request processing time in seconds (accurate to milliseconds); time elapsed between the first bytes of a request were processed and logging after the last bytes were sent to a user
$upstream_response_time	0.445	Number of seconds (accurate to milliseconds) it took to receive a response from an origin. In case of multiple responses, commas and colons are used
$request_length	157	Request length (including request line, header, and request body)
$http_range	bytes=0-1901653	File fragment size in a Range request
[$responding_node]	dh	Responding data center
$upstream_cache_status	MISS	Status of a requested file in CDN cache: HIT is a status of a response served from CDN cache. STALE is a status of an outdated response that failed to update because an origin was not responding or responding incorrectly. UPDATING is a status of an outdated response that is still updating since a previous request. REVALIDATED is a status of a response that is identical to the one on an origin based on the proxy_cache_revalidate directive. EXPIRED is a status of a response that has expired in cache, but still matches the one on an origin. A request has been sent to an origin for re-caching. MISS is a status of a response that has been served directly from an origin, rather than from cache BYPASS is a status of a response for the first file request after clearing the cache. Note: this status appears when the file is requested by each CDN server. When one CDN server requests a file for the first time, it will have the BYPASS status. When the same server requests the file again, the status will be changed to HIT. When another CDN server requests the file, it will again have the BYPASS status.
$upstream_response_length	10485760	Response length from an origin in bytes. In case of multiple responses, commas and colons are used
$upstream_addr	0.0.0.0:80	Origin's IP address and port
$gcdn_api_client_id (internal system variable)	123	Your ID in our system
$gcdn_api_resource_id (internal system variable)	01	Your CDN-resource ID in our system
$uid_got (internal system variable)	-	Cookie name and received user ID
$uid_set (internal system variable)	-	Cookie name and provided user ID
$geoip_country_code	KZ	User’s country code according to the ISO 3166 standard (Alpha-2 code).
$geoip_city	-	User’s city code
$shield_type (internal system variable)	shield_no	This field shows whether the shielding option is enabled: shield_old - enabled shield_no - disabled
$server_addr (internal system variable)	0.0.0.0	IP address of an Anycast zone or CDN server
$server_port (internal system variable)	80	Requested port
$upstream_status	206	Origin response code
$upstream_connect_time	0.000	Number of seconds (accurate to milliseconds) it took to access an origin server
$upstream_header_time	0.200	Number of seconds (accurate to milliseconds) it took to receive a response header from an origin server
$shard_addr (internal system variable)	0.0.0.0	IP address of a CDN server that was first to accept a request if the Cache Sharding feature is enabled
$geoip2_data_asnumber	asnumber	Number of an autonomous system that sent a request
$connection (internal system variable)	2897494295	Connection serial number
$connection_requests (internal system variable)	1	Current number of requests made through a connection
$request_id (internal system variable)	c1c0f12ab35b7cccccd5dc0a454879c5	Unique request identifier generated from 16 random bytes, in hexadecimal form
$http_x_forwarded_proto	-	Initial protocol of an incoming request (HTTP or HTTPS)
$http_x_forwarded_request_id (internal system variable)	-	Initial ID of an incoming request
$ssl_cipher (internal system variable)	ECDHE-RSA-AES256-GCM-SHA384	Cipher name used for an established SSL connection
$ssl_session_id (internal system variable)	28a4184139cb43cdc79006cf2d1a4ac93bdc****	Session ID of an established SSL connection
$ssl_session_reused (internal system variable)	r	The filed shows whether a session was reused (“r” ) or not (“.”)
$sent_http_content_type	application/json	Value of the Content-Type HTTP header, indicating the MIME type of a transmitted file
$tcpinfo_rtt	21	Average time (latency) it takes to transfer a packet to/from a server. The unit of time is microseconds.
$tcpi_total_retrans	10	Total number of retransmitted packets over the life of the connection.
$tcpinfo_snd_cwnd	45	Size of the TCP Congestion window, i.e., the maximum number of TCP segments that the connection can send before an acknowledgment is required.