Building in GitLab
To learn how to use containers and Docker on your local machine, refer to our tutorial section.
We use our own installation of GitLab for Source Code Management, Continuous Integration automation, containers registry and other development lifecycle tasks. It fully uses Nautilus Cluster resources, which provides our users unlimited storage and fast builds. All data from our GitLab except container images are backed up nightly to Google storage, which means there's almost zero chance that you might lose your code in our repository.
Step 1: Create a Git repo
- To use our GItLab installation, register at https://gitlab.nrp-nautilus.io
- Use GitLab for storing your code like any git repository. Here's GitLab basics guide.
- Create a new project in your GitLab account
Step 2: Use Containers Registry
What makes GitLab especially useful for kubernetes cluster in integration with Containers Registry. You can store your containers directly in our cluster and avoid slow downloads from DockerHub (although you're still free to do that as well).
If you wish to use our registry, in your https://gitlab.nrp-nautilus.io project go to Packages & Registries -> Container Registry
menu and read instructions on how to use one.
Step 3: Continuous Integration automation
To fully unleash the GitLab powers, introduce yourself to Continuous Integration automation and more advanced DevOps article.
- Create the
.gitlab-ci.yml
file in your project, see Quick start guide. The runners are already configured.
There's a list of CI templates available for most common languages. - If you need to build your Dockerfile and create a container from it, adjust this
.gitlab-ci.yml
template:
image: docker:git
default:
tags:
- docker
before_script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
stages:
- build-and-push
build-and-push-job:
stage: build-and-push
script:
- cd $CI_PROJECT_DIR && docker build . -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE:latest
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
Here's also the variant for using kaniko, which now has severe speed problems pushing to gitlab later resolved by using the ENV variable:
image: gcr.io/kaniko-project/executor:debug
stages:
- build-and-push
build-and-push-job:
stage: build-and-push
variables:
GODEBUG: "http2client=0"
script:
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --cache=true --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:8} --destination $CI_REGISTRY_IMAGE:latest
- Go to
CI / CD -> Jobs
tab to see in amazement your job running and image being uploaded to your registry. - From the
Packages -> Containers Registry
tab get the URL of your image to be included in your pod definition:
spec:
containers:
- name: my-container
image: gitlab-registry.nrp-nautilus.io/<your_group>/<your_project>:<optional_tag>
Using sysbox-provided docker
image: docker:git
default:
tags:
- sysbox
services:
- name: docker:dind
variables:
DOCKER_HOST: tcp://docker:2376/
DOCKER_TLS_CERTDIR: "/certs"
DOCKER_TLS_VERIFY: 1
DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
build-jupyter-base:
before_script:
- until docker info; do sleep 1; done
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab-registry.nrp-nautilus.io
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE:latest
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
Build better containers
Make yourself familiar with Docker containers best practices
Use multi-stage builds when necessary
Use S3 to store large files collections and access those during builds
Refer to S3 documentation
Other development information
Check out this guide from the Netherlands eScience Center for best practices in developing academic code.