How to manage units
Unit management guide for scaling and running admin utility scripts.
Replication and Scaling
Increasing the number of Apache Kafka Brokers can be achieved by adding more units to the Charmed Apache Kafka application, for example:
juju add-unit kafka -n <num_brokers_to_add>
For more information on how to manage units, please refer to the Juju documentation
It is important to note that when adding more units, the Apache Kafka cluster will not automatically rebalance existing topics and partitions. New storage and new brokers will be used only when new topics and new partitions are created.
Partition reassignment can still be done manually by the admin user by using the
charmed-kafka.reassign-partitions Charmed Apache Kafka bin utility script. Please refer to
its documentation for more information.
Scaling down is currently not supported in the charm automation.
If partition reassignment is not manually performed before scaling down in order
to make sure the decommissioned units do not hold any data, your cluster may
suffer to data loss.
Running Apache Kafka admin utility scripts
Apache Kafka ships with bin/*.sh commands to do various administrative tasks such as:
bin/kafka-config.shto update cluster configurationbin/kafka-topics.shfor topic managementbin/kafka-acls.shfor management of ACLs of Apache Kafka users
Please refer to the upstream Apache Kafka project,
for a full list of the bash commands available in Apache Kafka distributions. Also, you can
use --help argument for printing a short summary of the argument for a given
bash command.
The most important commands are also exposed via the Charmed Apache Kafka snap,
accessible via charmed-kafka.<command>. Please refer to this table for
more information about the mapping between the Apache Kafka bin commands and the snap entrypoints.
Before running bash scripts, make sure that some listeners have been correctly opened by creating appropriate integrations. Please refer to this table for more information about how listeners are opened based on relations. To simply open a SASL/SCRAM listener, just integrate a client application using the data integrator, as described in the How to manage app guide.
To run most of the scripts, you need to provide:
- the Apache Kafka service endpoints, generally referred to as bootstrap servers
- authentication information
Juju admins of the Apache Kafka deployment
For Juju admins of the Apache Kafka deployment, the bootstrap servers information can be obtained using:
BOOTSTRAP_SERVERS=$(juju run kafka/leader get-admin-credentials | grep "bootstrap.servers" | cut -d "=" -f 2)
Admin client authentication information is stored in the
/var/snap/charmed-kafka/common/etc/kafka/client.properties file is present on every Apache Kafka
broker. The content of the file can be accessed using:
juju ssh kafka/leader `cat /etc/kafka/client.properties`
This file can be provided to the Apache Kafka bin commands via the --command-config
argument. Note that client.properties may also refer to other files (
e.g. truststore and keystore for TLS-enabled connections). Those
files also need to be accessible and correctly specified.
Commands can also be run within a Apache Kafka Broker, since both the authentication file (along with the truststore if needed) and the Charmed Apache Kafka snap are already present.
Example (listing topics)
For instance, to list the current topics on the Apache Kafka cluster, you can run:
juju ssh kafka/leader 'charmed-kafka.topics --bootstrap-server $BOOTSTRAP_SERVERS --list --command-config /var/snap/charmed-kafka/common/etc/kafka/client.properties'
Juju external users
For external users managed by the Data Integrator Charm, the endpoints and credentials can be fetched using the dedicated action
juju run data-integrator/leader get-credentials --format yaml
The client.properties file can be generated by substituting the relevant information in the
file available on the brokers at /var/snap/charmed-kafka/current/etc/kafka/client.properties
To do so, fetch the information using juju commands:
BOOTSTRAP_SERVERS=$(juju run data-integrator/leader get-credentials --format yaml | yq .kafka.endpoints )
USERNAME=$(juju run data-integrator/leader get-credentials --format yaml | yq .kafka.username )
PASSWORD=$(juju run data-integrator/leader get-credentials --format yaml | yq .kafka.password )
Then copy the /var/snap/charmed-kafka/current/etc/kafka/client.properties and substitute the following lines:
...
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="<USERNAME>" password="<PASSWORD>";
...
bootstrap.servers=<BOOTSTRAP_SERVERS>