Syntax highlighting of 35ff99b ~( devops/giltab/ci)

# Gitlab-CI

[TOC]

## Refs

[GitLab Continuous Integration (GitLab CI/CD)](https://docs.gitlab.com/ce/ci/README.html)

[Configuration of your jobs with .gitlab-ci.yml](https://docs.gitlab.com/runner/install/docker.html#docker-image-installation-and-configuration)

[Keyword references](https://docs.gitlab.com/ee/ci/yaml/) | [include merge](https://docs.gitlab.com/ee/ci/yaml/includes.html#merge-method-for-include)

[Predefined variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) | [Gitlab Ci/CD Variables](https://docs.gitlab.com/ee/ci/variables/) | [File Type](https://docs.gitlab.com/ee/ci/variables/#cicd-variable-types)

[GitLab Runner](https://docs.gitlab.com/runner/)

[GitLab CI: Run jobs sequentially, in parallel or build a custom pipeline](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/)

[Linter](https://gitlab.isoit.ru/ci/lint)

[Regexp in only](https://docs.gitlab.com/ce/ci/yaml/#only-and-except-simplified)

[Container Scanning](https://docs.gitlab.com/ee/user/application_security/container_scanning/)

[CI/CD Examples](https://docs.gitlab.com/ee/ci/examples/)


[API Docs](https://docs.gitlab.com/ee/api/) |  [Piplines](https://docs.gitlab.com/ee/api/pipelines.html) | [Lint](https://docs.gitlab.com/ee/api/lint.html)

[GitLab CI/CD artifacts reports types](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html)

## Recipes

### Workflow

[Configuring a GitLab CI pipeline for Rails, PostgreSQL, rspec, and rubocop](https://medium.com/@steven.jackson/configuring-a-gitlab-ci-pipeline-for-rails-postgresql-and-rspec-57c411400172)

[Simple GitLab Containuous deployment with docker compose, docker machine and Gitlab CI](https://medium.com/@Empanado/simple-continuous-deployment-with-docker-compose-docker-machine-and-gitlab-ci-9047765322e1) | [Образ для статьи](https://hub.docker.com/r/jonaskello/docker-and-compose/)

```yaml
workflow:
  rules:
    - if: $CI_MERGE_REQUEST_TITLE =~ /^WIP:/
      when: never
    - if: $CI_MERGE_REQUEST_ID
    - if: $CI_COMMIT_BRANCH == "master"
    - if: $CI_PIPELINE_SOURCE == "schedule"
    - if: $CI_PIPELINE_SOURCE == "web"
```

### Rollback

Откат деплоя, если он завершился с ошибкой
```yaml
deploy_job:
  stage: deploy
  script:
    - kubectl apply -f frontend-deployment.yaml
    - kubectl rollout status deployment/frontend --timeout=60s

rollback_deploy_job:
  stage: rollback
  script:
    - kubectl rollout undo deployment/frontend
  when: on_failure
```

## docker-образ

Можно использовать CI для сборки docker-образа. Для этого необходимо чтобы *runner* функционировал в *привилегированном* режиме:
```bash
sudo cat /srv/gitlab-runner/config/config.toml
# ...
#[[runners]]
#  name = "runner-2"
#  url = "https://gitlab.isoit.ru"
#  token = "123123..."
#  executor = "docker"
#  [runners.docker]
#    tls_verify = false
#    image = "docker:18.04"
#    privileged = true 	 <------------------------
#    disable_cache = false
#    volumes = ["/cache"]
#    shm_size = 0
#  [runners.cache]
```
Если `privileged = false`, то необходимо руками изменить значение и перезапустить *runner*-контейнер.

В *gitlab-ci*конфигурации, приведенной ниже, используется образ *Docker in docker (did)*. Контейнер, запущенный из данного образа, как раз собирает *внутри себя* образ на этапе *build*. Если сразу не "отгрузить" собранный образ в *registry*, то он будет удален вместе с контейнером *did*. Другими словами, мы получим лишь факт того, что образ может быть собран по *Dockerfile*.

## Troubleshooting

Если при запуске сценария в *runner* выдается
```bash
docker info  
# Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
```
то необходимо скорректировать файл конфигурации *runner*-а
```yml
#/srv/gitlab-runner/config/config.toml
...
[[runners]]
  name = "runner-2"
  url = "https://gitlab.isoit.ru"
  token = "68ab9e5d61d96660616e7b385e7257"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:18.04"
    privileged = true
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] # <----------
    shm_size = 0
  [runners.cache]
```
и перезапустить соответствующий контейнер *runner*

[stackoverflow](https://stackoverflow.com/questions/42867039/gitlab-ci-runner-cant-connect-to-unix-var-run-docker-sock-in-kubernetes)

## Образец конфигурации
```yml
stages:
  - test
  - build

test:
  stage: test
  image: "phusion/passenger-ruby25:0.9.30"
  services:
    - postgres:10.3

  variables:
    POSTGRES_DB: platform_test
    POSTGRES_USER: postgres
    POSTGRES_PASSWORD: dbpassword

  before_script:
    - apt update -qq && apt install -y -qq libpq-dev
    - ruby -v
    - which ruby
    - gem install bundler --no-ri --no-rdoc
    - RAILS_ENV=test bundle install --local
    - cp config/database.yml.gitlab-test config/database.yml
    - RAILS_ENV=test bundle exec rake db:create db:schema:load

  script:
    - RAILS_ENV=test bundle exec rspec

  only:
    - develop

build:
  stage: build
  image: "docker:stable"
  # When using dind, it's wise to use the overlayfs driver for
  # improved performance.
  variables:
    DOCKER_DRIVER: overlay2
  services:
    - docker:dind
  before_script:
    - docker info
  script:
    - docker build -t waltix-platform .
  only:
    - develop
```

## Запуск runner-а

https://docs.gitlab.com/runner/install/docker.html

https://docs.gitlab.com/runner/configuration/advanced-configuration.html

### Ruby

```bash
sudo mkdir -p /srv/gitlab-runner/ruby25/config

docker run -d --name gitlab-runner-ruby25 \
	--restart always \
	-v /srv/gitlab-runner/ruby25/config:/etc/gitlab-runner \
	-v /var/run/docker.sock:/var/run/docker.sock \
	gitlab/gitlab-runner:11.8

docker exec -it gitlab-runner-ruby25 gitlab-runner register
```

Файл конфигурации 
*/srv/gitlab-runner/ruby25/config/config.toml*
```toml
concurrent = 4
check_interval = 0

[[runners]]
  name = "ruby25-passenger"
  url = "https://gitlab.isoit.ru"
  token = "secret"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.isoit.ru:5000/cicdd/img/pr25"
    privileged = false
    disable_cache = false
    volumes = ["/cache", "/bundle"]
    shm_size = 0
  [runners.cache]
```

### Docker

```bash
sudo mkdir -p /srv/gitlab-runner/docker/config
docker run -d --name gitlab-runner-docker \
	--restart always \
	-v /srv/gitlab-runner/docker/config:/etc/gitlab-runner \
	-v /var/run/docker.sock:/var/run/docker.sock \
	gitlab/gitlab-runner:11.8
docker exec -it gitlab-runner-docker gitlab-runner register
```

Файл конфигурации 
*/srv/gitlab-runner/docker/config/config.toml*
```toml
concurrent = 2
check_interval = 0

[[runners]]
  name = "docker"
  url = "https://gitlab.isoit.ru"
  token = "secret"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.isoit.ru:5000/cicdd/img/docker:stable"
    privileged = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.cache]
```

### Docker Compose

```bash
sudo mkdir -p /srv/gitlab-runner/docker-compose/config
docker run -d --name gitlab-runner-docker-compose \
	--restart always \
	-v /srv/gitlab-runner/docker-compose/config:/etc/gitlab-runner \
	-v /var/run/docker.sock:/var/run/docker.sock \
	gitlab/gitlab-runner:11.8
docker exec -it gitlab-runner-docker-compose gitlab-runner register
```
Файл конфигурации 
*/srv/gitlab-runner/docker-compose/config/config.toml*
```toml
concurrent = 1
check_interval = 0

[[runners]]
  name = "docker-compose"
  url = "https://gitlab.isoit.ru"
  token = "b16382f2cbbe52f4f05b9da6774916"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.isoit.ru:5000/cicdd/img/docker-and-compose"
    privileged = false
    disable_cache = false
    volumes = ["/cache", "/minio:/minio", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
  [runners.cache]
```

### Python

```bash
sudo mkdir -p /srv/gitlab-runner/python36/config
docker run -d --name gitlab-runner-python36 \
	--restart always \
	-v /srv/gitlab-runner/python36/config:/etc/gitlab-runner \
	-v /var/run/docker.sock:/var/run/docker.sock \
	gitlab/gitlab-runner:11.8
docker exec -it gitlab-runner-python36 gitlab-runner register
```

Файл конфигурации */srv/gitlab-runner/python36/config/config.toml*
```toml
concurrent = 2
check_interval = 0

[[runners]]
  name = "python36"
  url = "https://gitlab.isoit.ru"
  token = "secret"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.isoit.ru:5000/cicdd/img/py36"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
```

### Node

```bash
sudo mkdir -p /srv/gitlab-runner/node9/config
docker run -d --name gitlab-runner-node9 \
	--restart always \
	-v /srv/gitlab-runner/node9/config:/etc/gitlab-runner \
	-v /var/run/docker.sock:/var/run/docker.sock \
	gitlab/gitlab-runner:11.8
docker exec -it gitlab-runner-node9 gitlab-runner register
```

## Запуск runner-а локально

https://substrakt.com/how-to-debug-gitlab-ci-builds-locally/
https://docs.gitlab.com/runner/install/linux-manually.html

Предполагается, что Docker уже установлен

```bash
# скачиваем исполняеиый файл runner-а
sudo wget -O /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
# назначаем права на запуск
sudo chmod +x /usr/local/bin/gitlab-runner
# создаем специального пользователя
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
# предположение:
# runner как сервис устанавливать необязательно, 
# т.к. мы будем пользовать его "вручную"
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner 
sudo gitlab-runner start
# --- Запуск этапа сборки (stage)
# выполнять из директории в которой лежит .gitlab-ci.yml
sudo gitlab-runner exec docker test
```

По всей видимости, при запуске команды `gitlab-runner exec` игнорируется файл с настройками `config.tomp`. Поэтому необходимо конфигурацию передавать через аргументы...

Кроме того необходимо учитывать, что каждая операция выполняется в собственной изолированной среде. Общие данные необходимо "складировать" в явно указанный том, в примере ниже это */cache*.
```bash
gitlab-runner exec docker build \
	--docker-volumes /cache:/cache \
	--docker-volumes /var/run/docker.sock:/var/run/docker.sock \
	--docker-privileged=true
```

На данный момент при локальном запуске runner-а нет доступа к секретным переменным. Как временное решение - использование секции *variables*.
```yml
build:
  stage: build

  image: "registry.isoit.ru:5000/deployer/img/docker:stable"

  # When using dind, it's wise to use the overlayfs driver for
  # improved performance.
  variables:
    MCI: registry.isoit.ru:5000/cicdd/img/mc:latest
    MS: http://10.0.100.234:9000
    # MINIO_USER: secret variable
    # MINIO_PASSWORD: secret variable
    DOCKER_DRIVER: overlay2
    DOCKER_REGISTRY: registry.isoit.ru:5000
    # DOCKER_PASSWORD: secret variable
    BUILD_IMAGE: registry.isoit.ru:5000/zoid/waltix-platform-demo:develop

  services:
    - "registry.isoit.ru:5000/deployer/img/docker:dind" 

  before_script:
    - docker info
    - docker run --rm -v /cache/.mc:/root/.mc $MCI config host add minio $MS $MINIO_USER $MINIO_PASSWORD

  script:
    - docker run --rm -v /cache/.mc:/root/.mc -v /cache/tmp:/tmp $MCI cp minio/ds4-config/.vimrc /tmp
    - docker run --rm -v /cache/.mc:/root/.mc -v /cache/tmp:/tmp $MCI find minio/docker-certs --name "*.pem" --exec "mc cp {} /tmp" 
    - ls -la /cache/tmp
    - docker login -u zoid -p $DOCKER_PASSWORD $DOCKER_REGISTRY
    - docker build -t $BUILD_IMAGE  .
    - docker push $BUILD_IMAGE

  only:
    - develop

```

## Gitlab Coverage Badge

https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/


## Troubleshooting

### unknown command "sh" for "some-command"

При запуске Helm скрипта в раннере получаем ошибку
```bash
Error: unknown command "sh" for "helm"
```
Проблема описана здесь - https://stackoverflow.com/questions/67289837/gitlab-issue-while-running-helm-command-as-error-unknown-command-sh-for-he
Необходимо удалить entrypoint для job
```yaml
package:
  stage: build
  image:
    name: alpine/helm:3.7.1
    entrypoint: [""]
  script:
    - helm package .
  # ...
```


### container is marked for removal
```
ERROR: Preparation failed: Error response from daemon: container is marked for removal and cannot be started
```
https://gitlab.com/gitlab-org/gitlab-runner/issues/2400