crate.operator.scale module

class crate.operator.scale.ScaleSubHandler(namespace, name, ref, context, depends_on=None, run_on_dep_failures=False)

Bases: StateBasedSubHandler

Parameters

  • namespace (str) –

  • name (str) –

  • ref (str) –

  • context (dict) –

async handle(namespace, name, spec, old, diff, logger, **kwargs)
Parameters
async crate.operator.scale.check_for_deallocation(cursor, node_names, logger)

Wait until the nodes node_names have no more shards.

Parameters

  • cursor (Cursor) – A database cursor to a current and open database connection.

  • node_names (List[str]) – A list of CrateDB node names. These are the names that are known to CrateDB, e.g. data-hot-2 or master-1.

  • logger (Logger) –

async crate.operator.scale.check_nodes_present_or_gone(connection_factory, old_replicas, new_replicas, node_prefix, namespace, name, logger)
Parameters

  • connection_factory – A callable that allows the operator to connect to the database. We regularly need to reconnect to ensure the connection wasn’t closed because it was opened to a CrateDB node that was shut down since the connection was opened.

  • old_replicas (int) – The number of replicas in a StatefulSet before scaling.

  • new_replicas (int) – The number of replicas in a StatefulSet after scaling.

  • node_prefix (str) – The prefix of the node names in CrateDB.

  • namespace (str) –

  • name (str) –

  • logger (Logger) –

Raises

A kopf.TemporaryError when nodes are missing (scale up) or still available (scale down).

async crate.operator.scale.deallocate_nodes(conn_factory, new_num_data_nodes, excess_nodes, logger)

Trigger deallocation of data from nodes excess_nodes.

When the function exits cleanly, the nodes are deallocated. WHen the function raises a kopf.TemporaryError, some shards still need to be relocated.

Parameters

  • conn_factory – A function that establishes a database connection to the CrateDB cluster used for SQL queries.

  • new_num_data_nodes (int) – The total number of data nodes in the CrateDB cluster without the excess_nodes.

  • excess_nodes (List[str]) – The list of CrateDB nodes to deallocate.

  • logger (Logger) –

Raises

A kopf.TemporaryError when shards need to be relocated.

Return type

None

crate.operator.scale.get_container(statefulset)

Return the 'crate' container inside a StatefulSet

Parameters

statefulset (V1StatefulSet) – The StatefulSet where to lookup the CrateDB container in.

Return type

V1Container

Returns

The CrateDB container in the provided StatefulSet.

crate.operator.scale.parse_replicas(r)

Parse the number_of_replicas CrateDB table setting into an integer.

The input string could be in one of these forms:

  • 'n' (e.g. '3')

  • 'm-n' (e.g. '1-3')

  • 'n-all' (e.g. '1-all')

  • 'all'

When all is given, it’s treated as sys.maxsize.

Parameters

r (str) – The value for number_of_replicas.

Return type

int

Returns

The maximum number of replicas for a table.

crate.operator.scale.patch_command(old_command, total_nodes)

Iterate through the old_command items and update them with total_nodes where required.

Return the list making up the new CrateDB command.

Parameters

  • old_command (List[str]) – The command used to start-up CrateDB inside a Kubernetes container. This consists of the path to the Docker entrypoint script, the crate command argument and any additional settings.

  • total_nodes (int) – The number of nodes that will be in the CrateDB cluster. From that, the quorum is derived as well.

Return type

List[str]

Returns

The list forming the new CrateDB command.

async crate.operator.scale.reset_allocation(namespace, pod_name, has_ssl)

Reset all temporary node deallocations to none.

Note

Ideally, we’d be using the system user to reset the allocation exclusions. However, due to a bug, this isn’t possible in CrateDB <= 4.1.6. We therefore fall back to the “exec-in-container” approach that we also use during cluster bootstrapping.

Parameters

  • namespace (str) – The Kubernetes namespace for the CrateDB cluster.

  • pod_name (str) – The pod name of one of the eligible master nodes in the cluster. Used to exec into.

  • has_ssl (bool) – When True, crash will establish a connection to the CrateDB cluster from inside the crate container using SSL/TLS. This must match how the cluster is configured, otherwise crash won’t be able to connect, since non-encrypted connections are forbidden when SSL/TLS is enabled, and encrypted connections aren’t possible when no SSL/TLS is configured.

Return type

None

async crate.operator.scale.scale_cluster(apps, core, namespace, name, old, master_diff_item, data_diff_items, logger)

Scale cluster name according to the given master_diff_item and data_diff_items.

Parameters

  • apps (AppsV1Api) – An instance of the Kubernetes Apps V1 API.

  • core (CoreV1Api) – An instance of the Kubernetes Core V1 API.

  • namespace (str) – The Kubernetes namespace for the CrateDB cluster.

  • name (str) – The CrateDB custom resource name defining the CrateDB cluster.

  • old (Body) – The old resource body.

  • master_diff_item (Optional[DiffItem]) – An optional change indicating how many master nodes a cluster should have.

  • data_diff_items (Optional[Diff]) – An optional list of changes made to the individual data node specifications.

  • logger (Logger) –

async crate.operator.scale.scale_cluster_patch_total_nodes(apps, namespace, name, spec, total_nodes)

Update all StatefulSets to ensure intermittent node restarts will not reset the desired total nodes and quorum in a cluster.

Parameters

  • apps (AppsV1Api) – An instance of the Kubernetes Apps V1 API.

  • namespace (str) – The Kubernetes namespace for the CrateDB cluster.

  • name (str) – The CrateDB custom resource name defining the CrateDB cluster.

  • spec (Spec) – The spec field from the new CrateDB cluster custom object.

  • total_nodes (int) – The total number of nodes in the CrateDB cluster after scaling all StatefulSets.

Return type

None

async crate.operator.scale.update_statefulset(apps, namespace, sts_name, statefulset, replicas, total_nodes)

Call the Kubernetes API and update the container command and the replicas.

Parameters

  • apps (AppsV1Api) – An instance of the Kubernetes Apps V1 API.

  • namespace (Optional[str]) – The Kubernetes namespace for the CrateDB cluster.

  • sts_name (Optional[str]) – The name for the Kubernetes StatefulSet to update.

  • replicas (Optional[int]) – The new number of replicas for the StatefulSet. Can be None to not change the number of replicas for the StatefulSet.

  • total_nodes (int) – The number of nodes that will be in the CrateDB cluster.

  • statefulset (Optional[V1StatefulSet]) –