Despite the prevalence of search results pointing to the unofficial Google Cloud Platform representations of the Google Marketplace version of the Elastic Search Docker containers, these resources are incomplete, out of date, and serve to lock you in to the Google consumer products divisions.
Yes: Elastic Search maintains a deployable Docker container, with caveats.
# DB: current docker 9/21/23; see https://hub.docker.com/_/elasticsearch/
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.8.2
To run on Google Compute Engine requires several small modifications to its setup. These persist of appropriating the correct open ports of firewalls and setting configurations to operating system kernel so the search engine operates. Lastly, to deploy the search engine requires environment flags. It is a system of steps that should all the user to launch nearly any container.
Google deploys containers into machines running its proprietary Container Optimized OS, which restricts system and kernel configurability operations for security reasons. For this reason steps must be baked in at deployment.
The next steps will assume you have created a project on Google Cloud Console and will follow instructions to authorize its APIs as necessary.
LGTM DB: up on my local machine
# LGTM DB 9/21/23: works on my local machine.
# Elastic Search uses port 9200 for client HTTP and 9300 for internode comms
# xpack.security.enabled=false disables security and auth for non-prod usage
# discovery.type=single-node creates one node and will not search for others
docker run --rm -p 9200:9200 -p 9300:9300 -e "xpack.security.enabled=false" -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:8.8.2
# DB 9/21/23: simple client call to the docker for the search system health
curl -XGET http://127.0.0.1:9200/_cluster/health?pretty
Let us repeat the equivalent steps on one Google Compute Engine machine.
how-to: up on Google Compute Engine with Docker deployment
What base command?
gcloud compute instances create-with-container
create-with-container command deploys, then runs, the Docker container into one Container-Optimized OS machine. In this OS, exposing ports of the host VM will have a one-to-one mapping to exposing container ports; in other words, creating a firewall rule to allow TCP traffic through 9200 is sufficient to, and will automatically, expose the container port 9200.
Google Compute Engine opens firewall traffic through the use of applying labels to each machine, while assigning each label a particular firewall use case. Two firewall rules,
https-server, are supplied to projects by default to allow TCP traffic on ports 80 and 8080 respectively, although each must still be applied to the container in order to open the ports. Elastic Search needs neither of these ports, though does need to allow port 9200 and so we will create that firewall rule and its label here. The
create-with-container command will apply labels as the
🚦🟡 Yield: What base command?
gcloud compute firewall-rules create
# LGTM DB 9/21/23: create project-wide label that opens port 9200 for ingress
gcloud compute firewall-rules create elasticsearch-ingress --allow=tcp:9200 --source-ranges="0.0.0.0/0" --description="open ports for elasticsearch client HTTP ingress" --direction=INGRESS
# LGTM DB 9/21/23: adding tag both opens port on VM and exposes on container
Elastic Search uses a kernel-level map system to hold its files in memory to store its indices, and the system configuration usually needs to be modified to allow Elastic Search to perform its standard operations. The underlying cause of this failure will lead to shutdowns from memory exceptions, or even more likely an opaque
Error: elasticsearch exited unexpectedly.
The Container-Optimized OS does not allow many operating system level Linux-like commands, including the use
sysctl which is required to fix this problem by running the command
sysctl -w vm.max_map_count=262144. Container-Optimized OS does include what is referred to as its
toolbox functionality for some installation though this is not the case here. Instead, the solution is to recognize that Linux-like commands can be run before its
create-with-container deployment of the Docker container into the OS. It uses either the
--metadata-from-file=startup-script= arguments to pass the following short script:
# LGTM DB 9/21/23: set sysctl configuration in its system conf file; and load.
# this allows OS modification because Container-Optimized OS constraints added.
# this flag will be added to the gcloud create-with-container command
--metadata=startup-script="echo 'vm.max_map_count=262144' > /etc/sysctl.conf; sysctl -p;"
docker run -e flag creates an environment variable inside the Docker container and is equivalent to the
--container-env flag on
create-with-container command. These are environment variables; command line arguments and the
--container-arg flag pass argument directly into the
ENTRYPOINT command of the Dockerfile, and has strict formatting rules.
# LGTM DB 9/21/23: adds container `docker run -e` environment runtime flags
all-together: zero to Elastic Search server in the cloud
This one command will create a Google Compute Engine default machine named
unnamed-es-vm, run the
sysctl startup script, download the Elastic Search Docker to the virtual machine, allow ingress by TCP into port 9200 for HTTP access, and run
docker run command with environment flags.
The startup and launch takes about eighty seconds for the server to be up.
Each command requires
auth login and the
--project <name> flag as well.
# LGTM DB 9/21/23: creates and starts up an Elastic Search server
gcloud compute instances create-with-container unnamed-es-vm --container-image docker.elastic.co/elasticsearch/elasticsearch:8.8.2 --tags elasticsearch-ingress --zone us-west1-c --metadata=startup-script="echo 'vm.max_map_count=262144' > /etc/sysctl.conf; sysctl -p;" --container-env xpack.security.enabled=false,discovery.type=single-node
We can watch the virtual machine during its startup with this command:
# LGTM DB 9/21/23: display in local terminal the current startup output log
gcloud compute instances get-serial-port-output unnamed-es-vm
One of the benefits of this simple approach is that the virtual machine can be paused at no cost; and snapshots can be used to restore its index state.
whats-next: want to work on more; or, did this not work for you?
starlight.ai email addresses online and reach out whenever.