Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service by AWS consulting services that enables users to ingest and process streaming data in real time using Apache Kafka. With Amazon MSK, organizations can capture and process log and event streams to derive insights from their data. Using Amazon MSK, developers and DevOps managers can run Apache Kafka applications and Kafka Connect connectors on AWS without the need to manage the infrastructure and operations of Apache Kafka. Amazon MSK provisions and configures Apache Kafka cluster to ensure high availability, handles server patches and upgrades, monitors, and alarms, and scales clusters to support the load. This allows users to focus on their applications and data processing rather than the underlying infrastructure.
Get started with Amazon MSK
To create a simple MSK cluster within three Brokers and three availability zones you have to create an Amazon Elastic compute cloud or EC2 instance that will allow you to send commands to our MSK cluster and to produce and consume data from the cluster. The MSK cluster and EC2 instance will be hosted within the same virtual private cloud or VPC on AWS.
Step 1. Go to Amazon MSK console.
Step 2. Click on the Create cluster button.
Step 3. Select the Custom Create method so that you can specify all the details of the cluster.
Step 4. Give a name for the cluster.
Step 5. Select the Provision as the cluster type. This will allow us to specify the number of Brokers and the amount of storage per Broker.
Step 6. Select a broker.
Step 7. Scroll down and reduce the storage volume to 100 GB.
Step 8. Click the Next button.
Step 9. Select the VPC that has been created. Then, specify the three zones and subnets.
Step 10. Scroll down and click the Create link for Amazon EC2. Enter the necessary information and click the Create Security Group button.
Step 11. Go back to Amazon MSK. Then, click the Browse button to connect the security group with the MSK cluster.
Step 12. Select the name of your cluster and click the Choose button.
Step 13. Select the default security group and click the remove button. Then click the Next button.
Step 14. Scroll down and select Plaintext traffic. Then click the Next button. Again, click the Next button. Review the settings and click the Create cluster button.
Step 15. Navigate to the Amazon EC2 and launch an EC2 instance.
Step 16. Go to the Amazone MSK, click the Properties tab, scroll down to the Networking section, click the link under Security groups applied, and point it towards the EC2 instance created before.
Step 17. Edit inbound rules and go to instance details in the EC2 management console. Change the action to modify the IAM role. And update the IAM role.
Step 18. Click the instance link, and click the Connect button. Again, click the Connect button. Use AWS CLI to confirm the connection. Then, install Java and Kafka.
Step 19. Copy the bootstrap server’s plain text and paste it into the commands.
Step 20. After inserting the plain text, paste it into the CLI. First, create a topic and then generate a list of topics. Create a producer to produce data on the topic.
Conclusion
Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed service that simplifies the deployment and management of Apache Kafka infrastructure on AWS. It allows users to focus on developing and running their applications while AWS takes care of the operational aspects of Apache Kafka.
Metclouds’ team of experts helps you build and run real-time data streaming applications and pipelines without the need to manage your own Kafka infrastructure.