Skip to content

Latest commit

 

History

History
156 lines (131 loc) · 18 KB

README.md

File metadata and controls

156 lines (131 loc) · 18 KB

AWS MSK Kafka Cluster Terraform module

Terraform module which creates AWS MSK (Managed Streaming for Kafka) resources.

Usage

See examples directory for working examples to reference:

module "kafka-cluster" {
  source                 = "./msk-cluster"
  name                   = "Kafka"
  number_of_broker_nodes = 3
  enhanced_monitoring    = "PER_TOPIC_PER_PARTITION"
  environment = "dev"

  broker_node_client_subnets  = ["subnet-a", "subnet-b", "subnet-c"] #Adding your subnet
  broker_node_ebs_volume_size = 20
  broker_node_instance_type   = "kafka.t3.small"
  broker_node_security_groups = ["sg-default"] #Adding your security group

  encryption_in_transit_client_broker = "TLS"
  encryption_in_transit_in_cluster    = true

  configuration_name        = "example-configuration"
  configuration_description = "Example configuration"
  configuration_server_properties = {
    "auto.create.topics.enable" = true
    "delete.topic.enable"       = true
  }

  jmx_exporter_enabled    = true
  node_exporter_enabled   = true
  cloudwatch_logs_enabled = true
  # s3_logs_enabled         = true
  # s3_logs_bucket          = "S3-bucket-msk-log"

}

Examples

Examples codified under the examples are intended to give users references for how to use the module(s) as well as testing/validating changes to the source code of the module. If contributing to the project, please be sure to make any appropriate updates to the relevant examples to allow maintainers to test your changes and to keep the examples up to date for users. Thank you!

Requirements

Name Version
terraform >= 0.13.1
aws >= 3.71

Providers

Name Version
aws >= 3.71

Modules

No modules.

Resources

Name Type
aws_appautoscaling_policy.this resource
aws_appautoscaling_target.this resource
aws_cloudwatch_log_group.this resource
aws_msk_cluster.this resource
aws_msk_configuration.this resource
aws_msk_scram_secret_association.this resource
aws_mskconnect_custom_plugin.this resource
aws_mskconnect_worker_configuration.this resource

Inputs

Name Description Type Default Required
broker_node_client_subnets A list of subnets to connect to in client VPC (documentation) list(string) [] no
broker_node_ebs_volume_size The size in GiB of the EBS volume for the data drive on each broker node number null no
broker_node_instance_type Specify the instance type to use for the kafka brokers. e.g. kafka.m5.large. (Pricing info) string null no
broker_node_security_groups A list of the security groups to associate with the elastic network interfaces to control who can communicate with the cluster list(string) [] no
client_authentication_sasl_iam Enables IAM client authentication bool false no
client_authentication_sasl_scram Enables SCRAM client authentication via AWS Secrets Manager bool false no
client_authentication_tls_certificate_authority_arns List of ACM Certificate Authority Amazon Resource Names (ARNs) list(string) [] no
cloudwatch_log_group_kms_key_id The ARN of the KMS Key to use when encrypting log data string null no
cloudwatch_log_group_name Name of the Cloudwatch Log Group to deliver logs to string null no
cloudwatch_log_group_retention_in_days Specifies the number of days you want to retain log events in the log group number 0 no
cloudwatch_logs_enabled Indicates whether you want to enable or disable streaming broker logs to Cloudwatch Logs bool false no
configuration_description Description of the configuration string null no
configuration_name Name of the configuration string null no
configuration_server_properties Contents of the server.properties file. Supported properties are documented in the MSK Developer Guide map(string) {} no
connect_custom_plugin_timeouts Timeout configurations for the connect custom plugins map(string) 
{
  "create": null
}
 no
connect_custom_plugins Map of custom plugin configuration details (map of maps) any {} no
connect_worker_config_description A summary description of the worker configuration string null no
connect_worker_config_name The name of the worker configuration string null no
connect_worker_config_properties_file_content Contents of connect-distributed.properties file. The value can be either base64 encoded or in raw format string null no
create Determines whether cluster resources will be created bool true no
create_cloudwatch_log_group Determines whether to create a CloudWatch log group bool true no
create_connect_worker_configuration Determines whether to create connect worker configuration bool false no
create_schema_registry Determines whether to create a Glue schema registry for managing Avro schemas for the cluster bool true no
create_scram_secret_association Determines whether to create SASL/SCRAM secret association bool false no
encryption_at_rest_kms_key_arn You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest string null no
encryption_in_transit_client_broker Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT, and PLAINTEXT. Default value is TLS string null no
encryption_in_transit_in_cluster Whether data communication among broker nodes is encrypted. Default value: true bool null no
enhanced_monitoring Specify the desired enhanced MSK CloudWatch monitoring level. See Monitoring Amazon MSK with Amazon CloudWatch string null no
firehose_delivery_stream Name of the Kinesis Data Firehose delivery stream to deliver logs to string null no
firehose_logs_enabled Indicates whether you want to enable or disable streaming broker logs to Kinesis Data Firehose bool false no
jmx_exporter_enabled Indicates whether you want to enable or disable the JMX Exporter bool false no
kafka_version Specify the desired Kafka software version string null no
name Name of the MSK cluster string "msk" no
node_exporter_enabled Indicates whether you want to enable or disable the Node Exporter bool false no
number_of_broker_nodes The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets number null no
s3_logs_bucket Name of the S3 bucket to deliver logs to string null no
s3_logs_enabled Indicates whether you want to enable or disable streaming broker logs to S3 bool false no
s3_logs_prefix Prefix to append to the folder name string null no
scaling_max_capacity Max storage capacity for Kafka broker autoscaling number 250 no
scaling_role_arn The ARN of the IAM role that allows Application AutoScaling to modify your scalable target on your behalf. This defaults to an IAM Service-Linked Role string null no
scaling_target_value The Kafka broker storage utilization at which scaling is initiated number 70 no
schema_registries A map of schema registries to be created map(any) {} no
schemas A map schemas to be created within the schema registry map(any) {} no
scram_secret_association_secret_arn_list List of AWS Secrets Manager secret ARNs to associate with SCRAM list(string) [] no
tags A map of tags to assign to the resources created map(string) {} no
timeouts Create, update, and delete timeout configurations for the cluster map(string) {} no

Outputs

Name Description
appautoscaling_policy_arn The ARN assigned by AWS to the scaling policy
appautoscaling_policy_name The scaling policy's name
appautoscaling_policy_policy_type The scaling policy's type
arn Amazon Resource Name (ARN) of the MSK cluster
bootstrap_brokers Comma separated list of one or more hostname:port pairs of kafka brokers suitable to bootstrap connectivity to the kafka cluster
bootstrap_brokers_plaintext Comma separated list of one or more hostname:port pairs of kafka brokers suitable to bootstrap connectivity to the kafka cluster. Contains a value if encryption_in_transit_client_broker is set to PLAINTEXT or TLS_PLAINTEXT
bootstrap_brokers_sasl_iam One or more DNS names (or IP addresses) and SASL IAM port pairs. This attribute will have a value if encryption_in_transit_client_broker is set to TLS_PLAINTEXT or TLS and client_authentication_sasl_iam is set to true
bootstrap_brokers_sasl_scram One or more DNS names (or IP addresses) and SASL SCRAM port pairs. This attribute will have a value if encryption_in_transit_client_broker is set to TLS_PLAINTEXT or TLS and client_authentication_sasl_scram is set to true
bootstrap_brokers_tls One or more DNS names (or IP addresses) and TLS port pairs. This attribute will have a value if encryption_in_transit_client_broker is set to TLS_PLAINTEXT or TLS
configuration_arn Amazon Resource Name (ARN) of the configuration
configuration_latest_revision Latest revision of the configuration
connect_custom_plugins A map of output attributes for the connect custom plugins created
connect_worker_configuration_arn The Amazon Resource Name (ARN) of the worker configuration
connect_worker_configuration_latest_revision An ID of the latest successfully created revision of the worker configuration
current_version Current version of the MSK Cluster used for updates, e.g. K13V1IB3VIYZZH
log_group_arn The Amazon Resource Name (ARN) specifying the log group
schema_registries A map of output attributes for the schema registries created
schemas A map of output attributes for the schemas created
scram_secret_association_id Amazon Resource Name (ARN) of the MSK cluster
zookeeper_connect_string A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster. The returned values are sorted alphabetically
zookeeper_connect_string_tls A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster via TLS. The returned values are sorted alphabetically