diff --git a/.README.md.swp b/.README.md.swp deleted file mode 100644 index b5952d1..0000000 Binary files a/.README.md.swp and /dev/null differ diff --git a/README.md b/README.md index b416ec7..ff6629e 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,11 @@ Ansible을 이용하여 Datasaker Agent를 설치할 수 있습니다. ## Requirements - Ansible v2.6+가 필요합니다. -- 대부분의 데비안 리눅스 배포판을 지원합니다. +- 대부분의 Debian Linux 배포판을 지원합니다. +- 대부분의 Redhat Linux 배포판을 지원합니다. - Amazon Linux 2 배포판을 지원합니다. + ## Installation Ansible Galaxy에서 Datasaker role을 설치합니다. @@ -17,6 +19,8 @@ ansible-galaxy install dsk_bot.datasaker ``` 에이전트를 배포하기 위하여 Ansible playbook을 작성합니다. +****`dsk-log-agent` 설치 시 `fluent-bit` 이 자동으로 설치됩니다.*** + 아래는 기본 설치에 대한 예시입니다. @@ -28,7 +32,7 @@ ansible-galaxy install dsk_bot.datasaker - role: dsk_bot.datasaker vars: datasaker_api_key: "" - datasaker_agents: ["dsk-node-agent"] + datasaker_agents: ["dsk-node-agent","dsk-log-agent"] ``` #### Docker Agent Default Install Example ```yml @@ -43,99 +47,133 @@ ansible-galaxy install dsk_bot.datasaker ### 필수 설정 -| 변수명 | 설명 | -|--------------------------------------------|--------------------------------------------------| +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| |`datasaker_api_key`|API Key를 입력합니다.| -|`datasaker_agents`| 각 호스트에 설치하고자 하는 Host Agent 리스트입니다.
`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`
(Default) `dsk-node-agent`| -|`datasaker_docker_agents`| 각 호스트에 설치하고자 하는 Docker Container Agent 리스트입니다.
Docker Container Agents를 넣으면 Host Agent 설치는 자동으로 비활성화 됩니다.
`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`
(Default) `dsk-docker-node-agent`| +|`datasaker_agents`| 각 호스트에 설치하고자 하는 Host Agent 리스트입니다.
`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`
| `dsk-node-agent`| +|`datasaker_docker_agents`| 각 호스트에 설치하고자 하는 Docker Container Agent 리스트입니다.
Docker Container Agents를 넣으면 Host Agent 설치는 자동으로 비활성화 됩니다.
`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`
| `dsk-docker-node-agent`| ### Docker Container Agent 설정 -| 변수명 | 설명 | -|--------------------------------------------|--------------------------------------------------| -|`datasaker_docker_config_path`| Datasaker Global Config 위치 설정.
(Default) `~/.datasaker`| -|`datasaker_docker_global_config`| Datasaker Global Config 이름 설정.
(Default) `~/.datasaker/config.yml`| -|`docker_default_path`| Datasaker Docker Log Agent에 마운트할 Docker Log 수집 위치 설정.
(Default) `/var/lib/docker/containers/`| -|`datasaker_docker_path`| Datasaker Docker Agent Container 위치 설정.
(Default) `/var/datasaker`| -|`container_agent_restart_policy`| `dsk-container-agent` Container Restart Policy 설정.
(Default) `always`| -|`node_agent_restart_policy`| `dsk-node-agent` Container Restart Policy 설정.
(Default) `always`| -|`trace_agent_restart_policy`| `dsk-trace-agent` Container Restart Policy 설정.
(Default) `always`| -|`log_agent_restart_policy`| `dsk-log-agent` Container Restart Policy 설정.
(Default) `always`| -|`postgres_agent_restart_policy`| `dsk-postgres-agent` Container Restart Policy 설정.
(Default) `always`| -|`plan_postgres_agent_restart_policy`| `dsk-plan-postgres-agent` Container Restart Policy 설정.
(Default) `always`| -|`container_agent_log_level`| `dsk-container-agent` Log Level 설정.
(Default) `INFO`| -|`node_agent_log_level`| `dsk-node-agent` Log Level 설정.
(Default) `INFO`| -|`trace_agent_log_level`| `dsk-trace-agent` Log Level 설정.
(Default) `INFO`| -|`log_agent_log_level`| `dsk-log-agent` Log Level 설정.
(Default) `INFO`| -|`postgres_agent_log_level`| `dsk-postgres-agent` Log Level 설정.
(Default) `INFO`| -|`plan_postgres_agent_log_level`| `dsk-plan-postgres-agent` Log Level 설정.
(Default) `INFO`| +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`datasaker_docker_config_path`| Datasaker Global Config 위치 설정.
| `~/.datasaker`| +|`datasaker_docker_global_config`| Datasaker Global Config 이름 설정.
| `~/.datasaker/config.yml`| +|`docker_default_path`| Datasaker Docker Log Agent에 마운트할 Docker Log 수집 위치 설정.
| `/var/lib/docker/containers/`| +|`datasaker_docker_path`| Datasaker Docker Agent Container 위치 설정.
| `/var/datasaker`| +|`container_agent_restart_policy`| `dsk-container-agent` Container Restart Policy 설정.
| `always`| +|`node_agent_restart_policy`| `dsk-node-agent` Container Restart Policy 설정.
| `always`| +|`trace_agent_restart_policy`| `dsk-trace-agent` Container Restart Policy 설정.
| `always`| +|`log_agent_restart_policy`| `dsk-log-agent` Container Restart Policy 설정.
| `always`| +|`postgres_agent_restart_policy`| `dsk-postgres-agent` Container Restart Policy 설정.
| `always`| +|`plan_postgres_agent_restart_policy`| `dsk-plan-postgres-agent` Container Restart Policy 설정.
| `always`| +|`container_agent_log_level`| `dsk-container-agent` Log Level 설정.
| `INFO`| +|`node_agent_log_level`| `dsk-node-agent` Log Level 설정.
| `INFO`| +|`trace_agent_log_level`| `dsk-trace-agent` Log Level 설정.
| `INFO`| +|`log_agent_log_level`| `dsk-log-agent` Log Level 설정.
| `INFO`| +|`postgres_agent_log_level`| `dsk-postgres-agent` Log Level 설정.
| `INFO`| +|`plan_postgres_agent_log_level`| `dsk-plan-postgres-agent` Log Level 설정.
| `INFO`| - + ### Datasaker Agent 상세 설정 - Host Agent 와 Docker Container Agent는 같은 설정값을 사용합니다. -| 변수명 | 설명 | -|--------------------------------------------|--------------------------------------------------| -|`trace_sampling_rate`| `dsk-trace-agent` 에서 collector에 적용되는 샘플링 비율 설정.
- 100 이상일 때 모든 데이터가 수집.
(Default) `10`| -|`log_collects`| `dsk-log-agent` 에서 로그 수집 컬렉션 구성 설정. 리스트의 각 항목에는 아래 항목들이 포함. | -|`log_collects[*].paths`| `dsk-log-agent` 에서 로그 수집을 위한 경로 설정.
(Default) [`host-agent`]=`/var/log/*.log`, [`docker-agent`]=`/var/log/sample/*/*.log` | -|`log_collects[*].exclude_paths`| `dsk-log-agent` 에서 로그 컬렉션에서 제외할 경로 설정. 값을 지정하지 않으면 수집 경로에 설정한 모든 로그 수집.
(Default) `None` | -|`log_collects[*].keywords`| `dsk-log-agent` 에서 로그를 필터링하기 위한 키워드 설정. 값을 지정하지 않으면 모든 로그 수집.
(Default) `None` | -|`log_collects[*].tag`| `dsk-log-agent` 에서 사용자 설정 태그.
(Default) `Default` | -|`log_collects[*].service.name`| `dsk-log-agent` 에서 수집하는 서비스명 설정.
(Default) `default` | -|`log_collects[*].service.category`| `dsk-log-agent` 에서 수집하는 서비스 분류값 설정.
`app` `database` `syslog` `etc` (Default) `etc` | -|`log_collects[*].service.type`| `dsk-log-agent` 에서 수집하는 서비스 타입 설정.
`postgres` `mysql` `java` `etc` (Default) `etc` | -|`log_collects[*].service.address`| `dsk-log-agent` 에서 수집하는 서비스 타입이 Database인 경우에 작성
- 데이터베이스 host 및 port 정보 | -|`postgres_user_name`| `dsk-postgres-agent`에 Postgres user ID 설정.
(Default) `None` | -|`postgres_user_password`| `dsk-postgres-agent`에 Postgres user password 설정.
(Default) `None` | -|`postgres_database_address`| `dsk-postgres-agent`에 Postgres address 설정.
(Default) `None` | -|`postgres_database_port`| `dsk-postgres-agent`에 Postgres port 설정.
(Default) `None` | -|`plan_postgres_user_name`| `dsk-plan-postgres-agent`에 Plan Postgres user ID 설정.
(Default) `None` | -|`plan_postgres_user_password`| `dsk-plan-postgres-agent`에 Plan Postgres user password 설정.
(Default) `None` | -|`plan_postgres_database_address`| `dsk-plan-postgres-agent`에 Plan Postgres address 설정.
(Default) `None` | -|`plan_postgres_database_port`| `dsk-plan-postgres-agent`에 Plan Postgres port 설정.
(Default) `None` | -|`plan_postgres_database_name`| `dsk-plan-postgres-agent`에 Plan Postgres database 설정.
(Default) `None` | -|`plan_postgres_scrape_interval`| `dsk-plan-postgres-agent`에 Plan Postgres scrape interval 설정.
(Default) `30s` | -|`plan_postgres_scrape_timeout`| `dsk-plan-postgres-agent`에 Plan Postgres scrape timeout 설정.
(Default) `5s` | -|`plan_postgres_slow_query_standard`| `dsk-plan-postgres-agent`에 Plan Postgres slow query standard 설정.
(Default) `5s` | -|`plan_postgres_executor_number`| `dsk-plan-postgres-agent`에 Plan Postgres executor number 설정.
(Default) `10` | -|`plan_postgres_sender_number`| `dsk-plan-postgres-agent`에 Plan Postgres sender number 설정.
(Default) `10` | -|`plan_postgres_activity_query_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres activity query buffer 설정.
(Default) `50` | -|`plan_postgres_plan_sender_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres plan sender buffer 설정.
(Default) `50` | +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`trace_sampling_rate`| `dsk-trace-agent` 에서 collector에 적용되는 샘플링 비율 설정.
- 100 이상일 때 모든 데이터가 수집.
| `10`| +|`logs[*].service`|로그 수집 대상의 서비스 이름 설정.|`default`| +|`logs[*].tag`|로그 수집 대상의 태그 설정.|`None`| +|`logs[*].keyword`|로그 수집 키워드 설정. 키워드가 포함된 로그만 수집.|`None`| +|`logs[*].multiline.format`|멀티라인 로그 포맷 설정 (예 : go, java, ruby, python).|`None`| +|`logs[*].multiline.pattern`|멀티라인 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}).|`None`| +|`logs[*].masking[*].pattern`|마스킹할 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}) 사용자 커스텀 정규식 패턴 사용 가능.|`None`| +|`logs[*].masking[*].replace`|마스킹 패턴이 대체될 문자열 설정. (예 : ******).|`None`| +|`logs[*].collect.type`|로그 수집 방법 설정 (`file`, `driver` 중 하나의 값을 작성).|`file`| +|`logs[*].collect.category`|서비스 분류 설정 (`app`, `database`, `syslog`, `etc` 중 하나의 값을 작성).|`etc`| +|`logs[*].collect.address`|데이터베이스 host 및 port 정보 설정 (서비스 분류가 database인 경우 설정).|`None`| +|`logs[*].collect.file.paths`|로그 수집 대상 경로 설정. 예 : /var/log/sample/*.log.|`['/var/log/*.log']`| +|`logs[*].collect.file.exclude_paths`|로그 수집 제외 대상 경로 설정.|`None`| +|`postgres_user_name`| `dsk-postgres-agent`에 Postgres user ID 설정.
| `None` | +|`postgres_user_password`| `dsk-postgres-agent`에 Postgres user password 설정.
| `None` | +|`postgres_database_address`| `dsk-postgres-agent`에 Postgres address 설정.
| `None` | +|`postgres_database_port`| `dsk-postgres-agent`에 Postgres port 설정.
| `None` | +|`plan_postgres_user_name`| `dsk-plan-postgres-agent`에 Plan Postgres user ID 설정.
| `None` | +|`plan_postgres_user_password`| `dsk-plan-postgres-agent`에 Plan Postgres user password 설정.
| `None` | +|`plan_postgres_database_address`| `dsk-plan-postgres-agent`에 Plan Postgres address 설정.
| `None` | +|`plan_postgres_database_port`| `dsk-plan-postgres-agent`에 Plan Postgres port 설정.
| `None` | +|`plan_postgres_database_name`| `dsk-plan-postgres-agent`에 Plan Postgres database 설정.
| `None` | +|`plan_postgres_scrape_interval`| `dsk-plan-postgres-agent`에 Plan Postgres scrape interval 설정.
| `30s` | +|`plan_postgres_scrape_timeout`| `dsk-plan-postgres-agent`에 Plan Postgres scrape timeout 설정.
| `5s` | +|`plan_postgres_slow_query_standard`| `dsk-plan-postgres-agent`에 Plan Postgres slow query standard 설정.
| `5s` | +|`plan_postgres_executor_number`| `dsk-plan-postgres-agent`에 Plan Postgres executor number 설정.
| `10` | +|`plan_postgres_sender_number`| `dsk-plan-postgres-agent`에 Plan Postgres sender number 설정.
| `10` | +|`plan_postgres_activity_query_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres activity query buffer 설정.
| `50` | +|`plan_postgres_plan_sender_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres plan sender buffer 설정.
| `50` | -#### Ansible Playbook 상세 설정 Example +#### Ansible Playbook 상세 설정 Example (Linux) +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_api_key: "" + datasaker_docker_agents: + - "dsk-node-agent" + - "dsk-trace-agent" + - "dsk-log-agent" + - "dsk-postgres-agent" + postgres_user_name: sample + postgres_user_password: 1q2w3e4r + postgres_database_address: 0.0.0.0 + postgres_database_port: 5432 + plan_postgres_user_name: sample + plan_postgres_user_password: 1q2w3e4r + plan_postgres_database_address: 0.0.0.0 + plan_postgres_database_name: sample + plan_postgres_database_port: 5432 + logs: + - collect: + type: file + file: + paths: + - /var/log/*.log + - /datasaker/log/*.log +``` + +#### Ansible Playbook 상세 설정 Example (Docker) ```yml - hosts: servers become: true @@ -157,34 +195,25 @@ ansible-galaxy install dsk_bot.datasaker plan_postgres_database_address: 0.0.0.0 plan_postgres_database_name: sample plan_postgres_database_port: 5432 - log_collects: - - paths: - - "/var/log/sample/*/*.log" - exclude_paths: [] - keywords: [] - tag: "Default" - service: - name: "default" - category: "etc" - type: "etc" - - paths: - - "/var/log/sample/b4d5ac015a5a*/*.log" - service: - name: "docker-test" - category: "database" - type: "postgres" - address: "0.0.0.0:5432" + logs: + - collect: + type: file + file: + paths: + - /var/log/*.log + - /datasaker/log/*.log ``` + ## Uninstallation Datasaker Agent를 제거 할 수 있습니다. datasaker_clean은 uninstall이 `True`로 설정되어야 합니다. -| 변수명 | 설명 | -|--------------------------------------------|--------------------------------------------------| -|`uninstall`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent만 제거.
(Default) `False`| -|`datasaker_clean`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent 와 생성 된 폴더 및 설정 파일까지 제거.
(Default) `False`| +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`uninstall`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent만 제거.
| `False`| +|`datasaker_clean`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent 와 생성 된 폴더 및 설정 파일까지 제거.
| `False`| #### Datasaker Agents Uninstall Example diff --git a/README_en.md b/README_en.md index 4815220..6d447c4 100644 --- a/README_en.md +++ b/README_en.md @@ -8,6 +8,7 @@ The Ansible Datasaker role installs and configures the Datasaker Agent and integ - Requires Ansible v2.6+. - Supports most Debian Linux distributions. +- Supports most RedHat Linux distributions. - Supports Amazon Linux 2 distributions. ### Installation @@ -20,6 +21,10 @@ ansible-galaxy install dsk_bot.datasaker To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key to your playbook: +*** When installing `dsk-log-agent`, `fluent-bit` is automatically installed. *** + +In this example: + ###### Host Agent Default Install Example ```yml - hosts: servers @@ -28,7 +33,7 @@ To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key - role: dsk_bot.datasaker vars: datasaker_api_key: "" - datasaker_agents: ["dsk-node-agent"] + datasaker_agents: ["dsk-node-agent","dsk-log-agent"] ``` ###### Docker Agent Default Install Example ```yml @@ -43,97 +48,134 @@ To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key #### Base Role variables -| Variable | Description | -|--------------------------------------------|--------------------------------------------------| +| Variable | Description | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| |`datasaker_api_key`|Your Datasaker API key.| -|`datasaker_agents`|Set to Datasaker Host Agent.
`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`
(Default) `dsk-node-agent`| -|`datasaker_docker_agents`|Set to Datasaker Docker Agent.
`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`
(Default) `dsk-docker-node-agent`| +|`datasaker_agents`|Set to Datasaker Host Agent.
`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`
| `dsk-node-agent`| +|`datasaker_docker_agents`|Set to Datasaker Docker Agent.
`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`
| `dsk-docker-node-agent`| #### Docker Agent Role variables -| Variable | Description | -|--------------------------------------------|--------------------------------------------------| -|`datasaker_docker_config_path`| Override the datasaker global config path.
(Default) `~/.datasaker`| -|`datasaker_docker_global_config`| Override the datasaker global config file name.
(Default) `~/.datasaker/config.yml`| -|`docker_default_path`| Override the docker containers path.
(Default) `/var/lib/docker/containers/`| -|`datasaker_docker_path`| Override the datasaker docker agent containers path.
(Default) `/var/datasaker`| -|`container_agent_restart_policy`| Override the restart policy for a `dsk-container-agent` container
(Default) `always`| -|`node_agent_restart_policy`| Override the restart policy for a `dsk-node-agent` container
(Default) `always`| -|`trace_agent_restart_policy`| Override the restart policy for a `dsk-trace-agent` container
(Default) `always`| -|`log_agent_restart_policy`| Override the restart policy for a `dsk-log-agent` container
(Default) `always`| -|`postgres_agent_restart_policy`| Override the restart policy for a `dsk-postgres-agent` container
(Default) `always`| -|`plan_postgres_agent_restart_policy`| Override the restart policy for a `dsk-plan-postgres-agent` container
(Default) `always`| -|`container_agent_log_level`| Override the `dsk-container-agent` log level
(Default) `INFO`| -|`node_agent_log_level`| Override the `dsk-node-agent` log level
(Default) `INFO`| -|`trace_agent_log_level`| Override the `dsk-trace-agent` log level
(Default) `INFO`| -|`log_agent_log_level`| Override the `dsk-log-agent` log level
(Default) `INFO`| -|`postgres_agent_log_level`| Override the `dsk-postgres-agent` log level
(Default) `INFO`| -|`plan_postgres_agent_log_level`| Override the `dsk-plan-postgres-agent` log level
(Default) `INFO`| +| Variable | Description | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`datasaker_docker_config_path`| Override the datasaker global config path.
| `~/.datasaker`| +|`datasaker_docker_global_config`| Override the datasaker global config file name.
| `~/.datasaker/config.yml`| +|`docker_default_path`| Override the docker containers path.
| `/var/lib/docker/containers/`| +|`datasaker_docker_path`| Override the datasaker docker agent containers path.
| `/var/datasaker`| +|`container_agent_restart_policy`| Override the restart policy for a `dsk-container-agent` container
| `always`| +|`node_agent_restart_policy`| Override the restart policy for a `dsk-node-agent` container
| `always`| +|`trace_agent_restart_policy`| Override the restart policy for a `dsk-trace-agent` container
| `always`| +|`log_agent_restart_policy`| Override the restart policy for a `dsk-log-agent` container
| `always`| +|`postgres_agent_restart_policy`| Override the restart policy for a `dsk-postgres-agent` container
| `always`| +|`plan_postgres_agent_restart_policy`| Override the restart policy for a `dsk-plan-postgres-agent` container
| `always`| +|`container_agent_log_level`| Override the `dsk-container-agent` log level
| `INFO`| +|`node_agent_log_level`| Override the `dsk-node-agent` log level
| `INFO`| +|`trace_agent_log_level`| Override the `dsk-trace-agent` log level
| `INFO`| +|`log_agent_log_level`| Override the `dsk-log-agent` log level
| `INFO`| +|`postgres_agent_log_level`| Override the `dsk-postgres-agent` log level
| `INFO`| +|`plan_postgres_agent_log_level`| Override the `dsk-plan-postgres-agent` log level
| `INFO`| - + #### Agents Setting Role variables -| Variable | Description | -|--------------------------------------------|--------------------------------------------------| -|`trace_sampling_rate`| Override The `dsk-trace-agent` sampling rate applied to the collector.
- When set to 100 or higher, all data is collected.
(Default) `10`| -|`log_collects`| An array of log collection configurations. Each item in the array includes the following. | -|`log_collects[*].paths`| An array of paths for log collection.
(Default) [`host-agent`]=`/var/log/*.log`, [`docker-agent`]=`/var/log/sample/*/*.log` | -|`log_collects[*].exclude_paths`| An array of paths to be excluded from the log collection. If the array is empty, no paths will be excluded.
(Default) `None` | -|`log_collects[*].keywords`| An array of keywords for filtering the logs. If the array is empty, no keyword filtering will be applied.
(Default) `None` | -|`log_collects[*].tag`| The tag for the log collection item.
(Default) `Default` | -|`log_collects[*].service.name`| The name of the service.
(Default) `default` | -|`log_collects[*].service.category`| The category of the service.
(Default) `etc` | -|`log_collects[*].service.type`| The type of the service.
(Default) `etc` | -|`log_collects[*].service.address`| The address of the service.
This field is optional and is used only for certain services that require an address. | -|`postgres_user_name`| Enter the Postgres user ID.
(Default) `None` | -|`postgres_user_password`| Enter the Postgres user password.
(Default) `None` | -|`postgres_database_address`| Enter the Postgres address.
(Default) `None` | -|`postgres_database_port`| Enter the Postgres port.
(Default) `None` | -|`plan_postgres_user_name`| Enter the Plan Postgres user ID.
(Default) `None` | -|`plan_postgres_user_password`| Enter the Plan Postgres user password.
(Default) `None` | -|`plan_postgres_database_address`| Enter the Plan Postgres address.
(Default) `None` | -|`plan_postgres_database_port`| Enter the Plan Postgres port.
(Default) `None` | -|`plan_postgres_database_name`| Enter the Plan Postgres database.
(Default) `None` | -|`plan_postgres_scrape_interval`| Override the Plan Postgres scrape interval
(Default) `30s` | -|`plan_postgres_scrape_timeout`| Override the Plan Postgres scrape timeout
(Default) `5s` | -|`plan_postgres_slow_query_standard`| Override the Plan Postgres slow query standard
(Default) `5s` | -|`plan_postgres_executor_number`| Override the Plan Postgres executor number
(Default) `10` | -|`plan_postgres_sender_number`| Override the Plan Postgres sender number
(Default) `10` | -|`plan_postgres_activity_query_buffer`| Override the Plan Postgres activity query buffer
(Default) `50` | -|`plan_postgres_plan_sender_buffer`| Override the Plan Postgres plan sender buffer
(Default) `50` | +| Variable | Description | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`trace_sampling_rate`| Override The `dsk-trace-agent` sampling rate applied to the collector.
- When set to 100 or higher, all data is collected.
| `10`| +|`logs[*].service`|Defines the service name of the log collection target.|`default`| +|`logs[*].tag`|Sets the tag of the log collection target.|`None`| +|`logs[*].keyword`|Sets the keyword for log collection. Only logs that include the keyword are collected.|`None`| +|`logs[*].multiline.format`|Sets the multiline log format (e.g.: go, java, ruby, python).|`None`| +|`logs[*].multiline.pattern`|Sets the multiline log pattern. (e.g.: ^\d{4}-\d{2}-\d{2}).|`None`| +|`logs[*].masking[*].pattern`|Sets the log pattern to be masked. (e.g.: ^\d{4}-\d{2}-\d{2}) User-defined regular expression patterns are possible.|`None`| +|`logs[*].masking[*].replace`|Sets the string that the masking pattern will be replaced with. (e.g.: *****).|`None`| +|`logs[*].collect.type`|Sets the method of log collection (Choose one from file, driver).|`file`| +|`logs[*].collect.category`|Sets the service category (Choose one from app, database, syslog, etc).|`etc`| +|`logs[*].collect.address`|Sets the database host and port information (required if service category is database).|`None`| +|`logs[*].collect.file.paths`|Sets the paths for log collection. Example: /var/log/sample/.log.|`['/var/log/*.log']`| +|`logs[*].collect.file.exclude_paths`|Sets the paths to be excluded from log collection.|`None`| +|`postgres_user_name`| Enter the Postgres user ID.
| `None` | +|`postgres_user_password`| Enter the Postgres user password.
| `None` | +|`postgres_database_address`| Enter the Postgres address.
| `None` | +|`postgres_database_port`| Enter the Postgres port.
| `None` | +|`plan_postgres_user_name`| Enter the Plan Postgres user ID.
| `None` | +|`plan_postgres_user_password`| Enter the Plan Postgres user password.
| `None` | +|`plan_postgres_database_address`| Enter the Plan Postgres address.
| `None` | +|`plan_postgres_database_port`| Enter the Plan Postgres port.
| `None` | +|`plan_postgres_database_name`| Enter the Plan Postgres database.
| `None` | +|`plan_postgres_scrape_interval`| Override the Plan Postgres scrape interval
| `30s` | +|`plan_postgres_scrape_timeout`| Override the Plan Postgres scrape timeout
| `5s` | +|`plan_postgres_slow_query_standard`| Override the Plan Postgres slow query standard
| `5s` | +|`plan_postgres_executor_number`| Override the Plan Postgres executor number
| `10` | +|`plan_postgres_sender_number`| Override the Plan Postgres sender number
| `10` | +|`plan_postgres_activity_query_buffer`| Override the Plan Postgres activity query buffer
| `50` | +|`plan_postgres_plan_sender_buffer`| Override the Plan Postgres plan sender buffer
| `50` | -###### Agent Variables Example + +In this example: + +###### Ansible Playbook Setting Example (Linux) +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_api_key: "" + datasaker_docker_agents: + - "dsk-node-agent" + - "dsk-trace-agent" + - "dsk-log-agent" + - "dsk-postgres-agent" + postgres_user_name: sample + postgres_user_password: 1q2w3e4r + postgres_database_address: 0.0.0.0 + postgres_database_port: 5432 + plan_postgres_user_name: sample + plan_postgres_user_password: 1q2w3e4r + plan_postgres_database_address: 0.0.0.0 + plan_postgres_database_name: sample + plan_postgres_database_port: 5432 + logs: + - collect: + type: file + file: + paths: + - /var/log/*.log + - /datasaker/log/*.log +``` + +###### Ansible Playbook Setting Example (Docker) ```yml - hosts: servers become: true @@ -155,28 +197,25 @@ To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key plan_postgres_database_address: 0.0.0.0 plan_postgres_database_name: sample plan_postgres_database_port: 5432 - log_collects: - - paths: - - "/var/log/sample/*/*.log" - exclude_paths: [] - keywords: [] - tag: "Default" - service: - name: "default" - category: "etc" - type: "etc" - - paths: - - "/var/log/sample/b4d5ac015a5a*/*.log" - service: - name: "docker-test" - category: "database" - type: "postgres" - address: "0.0.0.0:5432" + logs: + - collect: + type: file + file: + paths: + - /var/log/*.log + - /datasaker/log/*.log ``` ## Uninstallation -However for more control over the uninstall parameters, the following code can be used. +Datasaker Agent can be uninstalled. +For this, datasaker_clean should be set to True. + +| Variable | Description | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`uninstall`| Only removes the agents specified in `datasaker_agents` or `datasaker_docker_agents`.
| `False`| +|`datasaker_clean`| Removes the agents specified in `datasaker_agents` or `datasaker_docker_agents` along with any generated folders and configuration files.
| `False`| + In this example: ```yml diff --git a/README_ko.md b/README_ko.md new file mode 100644 index 0000000..ff6629e --- /dev/null +++ b/README_ko.md @@ -0,0 +1,236 @@ +# Ansible Datasaker Role + +Ansible을 이용하여 Datasaker Agent를 설치할 수 있습니다. + +## Requirements + +- Ansible v2.6+가 필요합니다. +- 대부분의 Debian Linux 배포판을 지원합니다. +- 대부분의 Redhat Linux 배포판을 지원합니다. +- Amazon Linux 2 배포판을 지원합니다. + + +## Installation + +Ansible Galaxy에서 Datasaker role을 설치합니다. + +```shell +ansible-galaxy install dsk_bot.datasaker +``` + +에이전트를 배포하기 위하여 Ansible playbook을 작성합니다. +****`dsk-log-agent` 설치 시 `fluent-bit` 이 자동으로 설치됩니다.*** + + +아래는 기본 설치에 대한 예시입니다. + +#### Host Agent Default Install Example +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_api_key: "" + datasaker_agents: ["dsk-node-agent","dsk-log-agent"] +``` +#### Docker Agent Default Install Example +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_api_key: "" + datasaker_docker_agents: ["dsk-docker-node-agent","dsk-docker-log-agent"] +``` + +### 필수 설정 + +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`datasaker_api_key`|API Key를 입력합니다.| +|`datasaker_agents`| 각 호스트에 설치하고자 하는 Host Agent 리스트입니다.
`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`
| `dsk-node-agent`| +|`datasaker_docker_agents`| 각 호스트에 설치하고자 하는 Docker Container Agent 리스트입니다.
Docker Container Agents를 넣으면 Host Agent 설치는 자동으로 비활성화 됩니다.
`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`
| `dsk-docker-node-agent`| + + +### Docker Container Agent 설정 +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`datasaker_docker_config_path`| Datasaker Global Config 위치 설정.
| `~/.datasaker`| +|`datasaker_docker_global_config`| Datasaker Global Config 이름 설정.
| `~/.datasaker/config.yml`| +|`docker_default_path`| Datasaker Docker Log Agent에 마운트할 Docker Log 수집 위치 설정.
| `/var/lib/docker/containers/`| +|`datasaker_docker_path`| Datasaker Docker Agent Container 위치 설정.
| `/var/datasaker`| +|`container_agent_restart_policy`| `dsk-container-agent` Container Restart Policy 설정.
| `always`| +|`node_agent_restart_policy`| `dsk-node-agent` Container Restart Policy 설정.
| `always`| +|`trace_agent_restart_policy`| `dsk-trace-agent` Container Restart Policy 설정.
| `always`| +|`log_agent_restart_policy`| `dsk-log-agent` Container Restart Policy 설정.
| `always`| +|`postgres_agent_restart_policy`| `dsk-postgres-agent` Container Restart Policy 설정.
| `always`| +|`plan_postgres_agent_restart_policy`| `dsk-plan-postgres-agent` Container Restart Policy 설정.
| `always`| +|`container_agent_log_level`| `dsk-container-agent` Log Level 설정.
| `INFO`| +|`node_agent_log_level`| `dsk-node-agent` Log Level 설정.
| `INFO`| +|`trace_agent_log_level`| `dsk-trace-agent` Log Level 설정.
| `INFO`| +|`log_agent_log_level`| `dsk-log-agent` Log Level 설정.
| `INFO`| +|`postgres_agent_log_level`| `dsk-postgres-agent` Log Level 설정.
| `INFO`| +|`plan_postgres_agent_log_level`| `dsk-plan-postgres-agent` Log Level 설정.
| `INFO`| + + + +### Datasaker Agent 상세 설정 +- Host Agent 와 Docker Container Agent는 같은 설정값을 사용합니다. + +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`trace_sampling_rate`| `dsk-trace-agent` 에서 collector에 적용되는 샘플링 비율 설정.
- 100 이상일 때 모든 데이터가 수집.
| `10`| +|`logs[*].service`|로그 수집 대상의 서비스 이름 설정.|`default`| +|`logs[*].tag`|로그 수집 대상의 태그 설정.|`None`| +|`logs[*].keyword`|로그 수집 키워드 설정. 키워드가 포함된 로그만 수집.|`None`| +|`logs[*].multiline.format`|멀티라인 로그 포맷 설정 (예 : go, java, ruby, python).|`None`| +|`logs[*].multiline.pattern`|멀티라인 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}).|`None`| +|`logs[*].masking[*].pattern`|마스킹할 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}) 사용자 커스텀 정규식 패턴 사용 가능.|`None`| +|`logs[*].masking[*].replace`|마스킹 패턴이 대체될 문자열 설정. (예 : ******).|`None`| +|`logs[*].collect.type`|로그 수집 방법 설정 (`file`, `driver` 중 하나의 값을 작성).|`file`| +|`logs[*].collect.category`|서비스 분류 설정 (`app`, `database`, `syslog`, `etc` 중 하나의 값을 작성).|`etc`| +|`logs[*].collect.address`|데이터베이스 host 및 port 정보 설정 (서비스 분류가 database인 경우 설정).|`None`| +|`logs[*].collect.file.paths`|로그 수집 대상 경로 설정. 예 : /var/log/sample/*.log.|`['/var/log/*.log']`| +|`logs[*].collect.file.exclude_paths`|로그 수집 제외 대상 경로 설정.|`None`| +|`postgres_user_name`| `dsk-postgres-agent`에 Postgres user ID 설정.
| `None` | +|`postgres_user_password`| `dsk-postgres-agent`에 Postgres user password 설정.
| `None` | +|`postgres_database_address`| `dsk-postgres-agent`에 Postgres address 설정.
| `None` | +|`postgres_database_port`| `dsk-postgres-agent`에 Postgres port 설정.
| `None` | +|`plan_postgres_user_name`| `dsk-plan-postgres-agent`에 Plan Postgres user ID 설정.
| `None` | +|`plan_postgres_user_password`| `dsk-plan-postgres-agent`에 Plan Postgres user password 설정.
| `None` | +|`plan_postgres_database_address`| `dsk-plan-postgres-agent`에 Plan Postgres address 설정.
| `None` | +|`plan_postgres_database_port`| `dsk-plan-postgres-agent`에 Plan Postgres port 설정.
| `None` | +|`plan_postgres_database_name`| `dsk-plan-postgres-agent`에 Plan Postgres database 설정.
| `None` | +|`plan_postgres_scrape_interval`| `dsk-plan-postgres-agent`에 Plan Postgres scrape interval 설정.
| `30s` | +|`plan_postgres_scrape_timeout`| `dsk-plan-postgres-agent`에 Plan Postgres scrape timeout 설정.
| `5s` | +|`plan_postgres_slow_query_standard`| `dsk-plan-postgres-agent`에 Plan Postgres slow query standard 설정.
| `5s` | +|`plan_postgres_executor_number`| `dsk-plan-postgres-agent`에 Plan Postgres executor number 설정.
| `10` | +|`plan_postgres_sender_number`| `dsk-plan-postgres-agent`에 Plan Postgres sender number 설정.
| `10` | +|`plan_postgres_activity_query_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres activity query buffer 설정.
| `50` | +|`plan_postgres_plan_sender_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres plan sender buffer 설정.
| `50` | + +#### Ansible Playbook 상세 설정 Example (Linux) +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_api_key: "" + datasaker_docker_agents: + - "dsk-node-agent" + - "dsk-trace-agent" + - "dsk-log-agent" + - "dsk-postgres-agent" + postgres_user_name: sample + postgres_user_password: 1q2w3e4r + postgres_database_address: 0.0.0.0 + postgres_database_port: 5432 + plan_postgres_user_name: sample + plan_postgres_user_password: 1q2w3e4r + plan_postgres_database_address: 0.0.0.0 + plan_postgres_database_name: sample + plan_postgres_database_port: 5432 + logs: + - collect: + type: file + file: + paths: + - /var/log/*.log + - /datasaker/log/*.log +``` + +#### Ansible Playbook 상세 설정 Example (Docker) +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_api_key: "" + datasaker_docker_agents: + - "dsk-docker-node-agent" + - "dsk-docker-trace-agent" + - "dsk-docker-log-agent" + - "dsk-docker-postgres-agent" + postgres_user_name: sample + postgres_user_password: 1q2w3e4r + postgres_database_address: 0.0.0.0 + postgres_database_port: 5432 + plan_postgres_user_name: sample + plan_postgres_user_password: 1q2w3e4r + plan_postgres_database_address: 0.0.0.0 + plan_postgres_database_name: sample + plan_postgres_database_port: 5432 + logs: + - collect: + type: file + file: + paths: + - /var/log/*.log + - /datasaker/log/*.log +``` + + +## Uninstallation + +Datasaker Agent를 제거 할 수 있습니다. +datasaker_clean은 uninstall이 `True`로 설정되어야 합니다. + +| 변수명 | 설명 | Default | +|--------------------------------------------|--------------------------------------------------|--------------------------------------------------| +|`uninstall`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent만 제거.
| `False`| +|`datasaker_clean`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent 와 생성 된 폴더 및 설정 파일까지 제거.
| `False`| + +#### Datasaker Agents Uninstall Example + +```yml +- hosts: servers + become: true + roles: + - role: dsk_bot.datasaker + vars: + datasaker_agents: [""] + uninstall: True + datasaker_clean: True +``` + diff --git a/defaults/main.yml b/defaults/main.yml index 5ff7208..806647b 100644 --- a/defaults/main.yml +++ b/defaults/main.yml @@ -26,7 +26,7 @@ datagate_loggate_timeout: 5s datasaker_api_url: api.kr.datasaker.io datasaker_api_send_interval: 1m -datasaker_yum_repo: "http://nexus.exem-oss.org/repository/datasaker-amazonlinux-2023" +datasaker_yum_repo: "https://nexus.exem-oss.org/repository/" datasaker_yum_enabled: yes datasaker_yum_gpgcheck: no @@ -71,5 +71,7 @@ log_agent_log_level: "INFO" postgres_agent_log_level: "INFO" plan_postgres_agent_log_level: "INFO" +log_agent_port: 21212 + uninstall: False datasaker_clean: False diff --git a/files/libpq-13.5-1.el8.x86_64.rpm b/files/libpq-13.5-1.el8.x86_64.rpm new file mode 100644 index 0000000..0a31fa7 Binary files /dev/null and b/files/libpq-13.5-1.el8.x86_64.rpm differ diff --git a/files/libpq-devel-13.5-1.el8.x86_64.rpm b/files/libpq-devel-13.5-1.el8.x86_64.rpm new file mode 100644 index 0000000..7d02cdc Binary files /dev/null and b/files/libpq-devel-13.5-1.el8.x86_64.rpm differ diff --git a/meta/.galaxy_install_info b/meta/.galaxy_install_info index bee023e..027e9b7 100644 --- a/meta/.galaxy_install_info +++ b/meta/.galaxy_install_info @@ -1,2 +1,2 @@ -install_date: Thu Jun 15 08:26:46 2023 -version: 1.0.1 +install_date: Wed Jun 28 05:49:19 2023 +version: 1.0.2 diff --git a/meta/main.yml b/meta/main.yml index 2194fef..2eb305e 100644 --- a/meta/main.yml +++ b/meta/main.yml @@ -17,6 +17,11 @@ galaxy_info: - name: EL versions: - '7' + - '8' + - '9' + - name: Amazon Linux + - '2' + - '2023' galaxy_tags: - monitoring dependencies: [] diff --git a/tasks/dsk-debian-pkg.yml b/tasks/dsk-debian-pkg.yml index c4acb12..a70a13c 100644 --- a/tasks/dsk-debian-pkg.yml +++ b/tasks/dsk-debian-pkg.yml @@ -1,7 +1,14 @@ --- -- set_fact: - datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}" - when: ansible_facts.distribution_version != "18.04" +# - set_fact: +# datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}" +# when: ansible_facts.distribution_version != "18.04" +- name: Set os-specific variables + set_fact: + os_specific: "{% if ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '7' %}datasaker-redhat-7 + {% elif ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '8' %}datasaker-redhat-8 + {% elif ansible_facts['os_family'] == 'Amazon' and ansible_facts['distribution_major_version'] == '2' %}datasaker-amazonlinux-2 + {% elif ansible_facts['os_family'] == 'Debian'%}debian-repos + {% else %}unknown{% endif %}" - name: "Create temporary directory for key manipulation" tempfile: @@ -53,7 +60,7 @@ update_cache: yes force: yes -- name: Make Datasaker Log Directory +- name: "Make Datasaker Log Directory" ansible.builtin.file: path: "{{ datasaker_host_log_path }}/{{ item }}" state: directory @@ -77,13 +84,15 @@ name: "td-agent" state: latest update_cache: yes - when: '"dsk-log-agent" in datasaker_agents' + when: + - '"dsk-log-agent" in datasaker_agents' + - log_agent_image_tag != "latest" -- name: Reload systemd configuration +- name: "Reload systemd configuration" ansible.builtin.systemd: daemon_reload: True -- name: Restart dsk-agent service +- name: "Restart dsk-agent service" ansible.builtin.systemd: name: "{{ item }}" enabled: true diff --git a/tasks/dsk-docker-log-agent.yml b/tasks/dsk-docker-log-agent.yml index 11a68a9..cb5de50 100644 --- a/tasks/dsk-docker-log-agent.yml +++ b/tasks/dsk-docker-log-agent.yml @@ -19,13 +19,14 @@ src: docker-log-agent-config.yml.j2 dest: "{{ datasaker_docker_path }}/dsk-docker-log-agent-config.yml" -- name: Run Datasaker Log Agent +- name: "Run Datasaker Log Agent" docker_container: name: dsk-docker-log-agent image: "datasaker/dsk-log-agent:{{ log_agent_image_tag }}" state: started - detach: true restart_policy: "{{ log_agent_restart_policy }}" + ports: + - "{{ log_agent_port }}:{{ log_agent_port }}" env: DKS_LOG_LEVEL: "{{ log_agent_log_level }}" command: "-global.config=/etc/datasaker/global-config.yml -agent.config=/etc/datasaker/dsk-log-agent/agent-config.yml -mount.volume=true" @@ -33,5 +34,18 @@ - "{{ datasaker_docker_path }}:/var/datasaker/" - "{{ datasaker_docker_global_config }}:/etc/datasaker/global-config.yml:ro" - "{{ datasaker_docker_path }}/dsk-docker-log-agent-config.yml:/etc/datasaker/dsk-log-agent/agent-config.yml:ro" - - "{{ item }}" - loop: "{{ custom_log_volume | default(['{{ docker_default_path }}:/var/log/sample/:ro']) }}" \ No newline at end of file + - "{{ item }}:{{ item }}:ro" + with_items: "{{ custom_log_volume | default(['{{ docker_default_path }}']) }}" + +- name: "Run APP with fluentd logging" + docker_container: + name: "{{ app_name }}" + image: "{{ app_image }}" + state: started + log_options: + log-driver: fluentd + log-opt: + fluentd-address: "{{ log_agent_port }}:{{ log_agent_port }}" + when: + - app_name is defined + - app_image is defined \ No newline at end of file diff --git a/tasks/dsk-log-agent.yml b/tasks/dsk-log-agent.yml index cebebea..74527a0 100644 --- a/tasks/dsk-log-agent.yml +++ b/tasks/dsk-log-agent.yml @@ -17,4 +17,49 @@ - name: "Setting dsk-log-agent config" template: src: log-agent-config.yml.j2 - dest: "{{ datasaker_host_path }}/dsk-log-agent/agent-config.yml" \ No newline at end of file + dest: "{{ datasaker_host_path }}/dsk-log-agent/agent-config.yml" + +- name: "Install fluent-bit" + block: + - name: "Install fluent-bit" + shell: curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh + rescue: + - name: "Copy RPM files" + copy: + src: "{{ item }}" + dest: "/tmp/" + with_items: + - "{{ role_path }}/files/libpq-13.5-1.el8.x86_64.rpm" + - "{{ role_path }}/files/libpq-devel-13.5-1.el8.x86_64.rpm" + when: + - ansible_facts['os_family'] == 'RedHat' + - ansible_facts['distribution_major_version'] == '8' + - name: "Install libpq-13.5-1" + command: rpm -ivh /tmp/{{ item }} + with_items: + - libpq-13.5-1.el8.x86_64.rpm + - libpq-devel-13.5-1.el8.x86_64.rpm + when: + - ansible_facts['os_family'] == 'RedHat' + - ansible_facts['distribution_major_version'] == '8' + - template: + src: fluent-bit-repo.yml.j2 + dest: "/etc/yum.repos.d/fluent-bit.repo" + when: + - ansible_facts['os_family'] == 'RedHat' + - ansible_facts['distribution_major_version'] == '7' + - yum: + name: "{{ item }}" + state: present + update_cache: yes + with_items: + - fluent-bit + - file: + path: /tmp/{{ item }} + state: absent + with_items: + - libpq-13.5-1.el8.x86_64.rpm + - libpq-devel-13.5-1.el8.x86_64.rpm + when: + - ansible_facts['os_family'] == 'RedHat' + - ansible_facts['distribution_major_version'] == '8' \ No newline at end of file diff --git a/tasks/dsk-redhat-pkg.yml b/tasks/dsk-redhat-pkg.yml index 17de2da..08d2a64 100644 --- a/tasks/dsk-redhat-pkg.yml +++ b/tasks/dsk-redhat-pkg.yml @@ -1,14 +1,22 @@ --- +- name: Set os-specific variables + set_fact: + os_specific: "{% if ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '7' %}datasaker-redhat-7 + {% elif ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '8' %}datasaker-redhat-8 + {% elif ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '9' %}datasaker-redhat-9 + {% elif ansible_facts['os_family'] == 'Amazon' and ansible_facts['distribution_major_version'] == '2' %}datasaker-amazonlinux-2 + {% else %}unknown{% endif %}" + - name: "Add datasaker repository" yum_repository: name: datasaker description: datasaker-repo - baseurl: "{{ datasaker_yum_repo }}" + baseurl: "{{ datasaker_yum_repo }}{{ os_specific }}" enabled: "{{ datasaker_yum_enabled }}" gpgcheck: "{{ datasaker_yum_gpgcheck }}" -- set_fact: - datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}" +# - set_fact: +# datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}" - name: "Check datasaker Agent" include_tasks: check-agent.yml @@ -28,7 +36,9 @@ name: "td-agent" state: latest update_cache: yes - when: '"dsk-log-agent" in datasaker_agents' + when: + - '"dsk-log-agent" in datasaker_agents' + - log_agent_image_tag != "latest" - name: "Install datasaker agent" yum: @@ -40,4 +50,12 @@ notify: - Reload systemd configuration - Restart {{ item }} service - ignore_errors: true \ No newline at end of file + ignore_errors: true + +- name: "Setting dsk-log-agent config" + template: + src: log-agent-config.yml.j2 + dest: "{{ datasaker_host_path }}/dsk-log-agent/agent-config.yml" + when: + - '"dsk-log-agent" in datasaker_agents' + - log_agent_image_tag == "latest" \ No newline at end of file diff --git a/templates/docker-log-agent-config.yml.j2 b/templates/docker-log-agent-config.yml.j2 index 263947e..d502480 100644 --- a/templates/docker-log-agent-config.yml.j2 +++ b/templates/docker-log-agent-config.yml.j2 @@ -1,32 +1,51 @@ agent: metadata: - agent_name: "{{ log_agent_name | default('dsk-log-agent') }}" - collect: -{% for collect_item in log_collects | default([{'paths': ['/var/log/sample/*/*.log'], 'exclude_paths': [], 'keywords': [], 'tag': 'Default', 'service': {'name': 'default', 'category': 'etc', 'type': 'etc'}}]) %} - - paths: -{% for path in collect_item.paths %} - - "{{ path }}" + agent_name: {{ metadata.agent_name | default('dsk-log-agent') }} + cluster_id: {{ metadata.cluster_id | default('unknown') }} + logs: +{% for log in logs | default([{'service': 'default', 'collect': {'type': 'file', 'category': 'etc', 'file': {'paths': ['/var/log/*.log']}}}]) %} + - service: {{ log.service | default('default') }} + tag: +{% for tag in log.tag | default([]) %} + - {{ tag }} {% endfor %} - exclude_paths: -{% if collect_item.exclude_paths | default([]) %} -{% for exclude_path in collect_item.exclude_paths %} - - "{{ exclude_path }}" + keyword: +{% for keyword in log.keyword | default([]) %} + - {{ keyword }} +{% endfor %} + multiline: + format: {{ log.multiline.format | default('') }} + pattern: +{% for pattern in log.multiline.pattern | default([]) %} + - {{ pattern }} +{% endfor %} + masking: +{% for mask in log.masking | default([]) %} + - pattern: {{ mask.pattern }} + replace: {{ mask.replace }} +{% endfor %} + collect: +{% if log.collect.type == "file" %} + type: {{ log.collect.type | default('file') }} + category: {{ log.collect.category | default('etc') }} + address: {{ log.collect.address | default('') }} + file: + paths: +{% for path in log.collect.file.paths %} + - {{ path }} +{% endfor %} + exclude_paths: +{% for exclude_path in log.collect.file.exclude_paths | default([]) %} + - {{ exclude_path }} {% endfor %} {% endif %} - keywords: -{% if collect_item.keywords | default([]) %} -{% for keyword in collect_item.keywords %} - - "{{- keyword -}}" +{% if log.collect.type == "driver" %} + type: {{ log.collect.type | default('driver') }} + category: {{ log.collect.category | default('etc') }} + driver: + containers: +{% for container in (log.collect.docker_driver.container if 'docker_driver' in log.collect else ['*']) %} + - "{{ container }}" {% endfor %} {% endif %} -{% if collect_item.tag is defined %} - tag: "{{ collect_item.tag }}" -{% endif %} - service: - name: "{{ collect_item.service.name }}" - category: "{{ collect_item.service.category }}" - type: "{{ collect_item.service.type }}" -{% if collect_item.service.address is defined %} - address: "{{ collect_item.service.address }}" -{% endif %} {% endfor %} \ No newline at end of file diff --git a/templates/fluent-bit-repo.yml.j2 b/templates/fluent-bit-repo.yml.j2 new file mode 100644 index 0000000..0d0945d --- /dev/null +++ b/templates/fluent-bit-repo.yml.j2 @@ -0,0 +1,7 @@ +[fluent-bit] +name = Fluent Bit +baseurl = https://packages.fluentbit.io/centos/7/$basearch/ +gpgcheck=1 +gpgkey=https://packages.fluentbit.io/fluentbit.key +repo_gpgcheck=1 +enabled=1 \ No newline at end of file diff --git a/templates/log-agent-config.yml.j2 b/templates/log-agent-config.yml.j2 index d6c235e..a0e8269 100644 --- a/templates/log-agent-config.yml.j2 +++ b/templates/log-agent-config.yml.j2 @@ -1,32 +1,42 @@ agent: metadata: - agent_name: "{{ log_agent_name | default('dsk-log-agent') }}" - collect: -{% for collect_item in log_collects | default([{'paths': ['/var/log/*.log'], 'exclude_paths': [], 'keywords': [], 'tag': 'Default', 'service': {'name': 'default', 'category': 'etc', 'type': 'etc'}}]) %} - - paths: -{% for path in collect_item.paths %} - - "{{ path }}" + agent_name: {{ metadata.agent_name | default('dsk-log-agent') }} + cluster_id: {{ metadata.cluster_id | default('unknown') }} + logs: +{% for log in logs | default([{'service': 'default', 'collect': {'type': 'file', 'category': 'etc', 'file': {'paths': ['/var/log/*.log']}}}]) %} + - service: {{ log.service | default('default') }} + tag: +{% for tag in log.tag | default([]) %} + - {{ tag }} {% endfor %} - exclude_paths: -{% if collect_item.exclude_paths | default([]) %} -{% for exclude_path in collect_item.exclude_paths %} - - "{{ exclude_path }}" + keyword: +{% for keyword in log.keyword | default([]) %} + - {{ keyword }} {% endfor %} -{% endif %} - keywords: -{% if collect_item.keywords | default([]) %} -{% for keyword in collect_item.keywords %} - - "{{- keyword -}}" + multiline: + format: {{ log.multiline.format | default('') }} + pattern: +{% for pattern in log.multiline.pattern | default([]) %} + - {{ pattern }} +{% endfor %} + masking: +{% for mask in log.masking | default([]) %} + - pattern: {{ mask.pattern }} + replace: {{ mask.replace }} +{% endfor %} + collect: +{% if log.collect.type == "file" %} + type: {{ log.collect.type | default('file') }} + category: {{ log.collect.category | default('etc') }} + address: {{ log.collect.address | default('') }} + file: + paths: +{% for path in log.collect.file.paths %} + - {{ path }} +{% endfor %} + exclude_paths: +{% for exclude_path in log.collect.file.exclude_paths | default([]) %} + - {{ exclude_path }} {% endfor %} -{% endif %} -{% if collect_item.tag is defined %} - tag: "{{ collect_item.tag }}" -{% endif %} - service: - name: "{{ collect_item.service.name }}" - category: "{{ collect_item.service.category }}" - type: "{{ collect_item.service.type }}" -{% if collect_item.service.address is defined %} - address: "{{ collect_item.service.address }}" {% endif %} {% endfor %} \ No newline at end of file