sa_8001/datasaker-agent-ansible
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

log agent 수정

Browse Source
This commit is contained in:
havelight-ee
2023-08-02 11:04:04 +09:00
parent ba2259b7b8
commit ffa0348d68
16 changed files with 706 additions and 273 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
.README.md.swp
View File

Binary file not shown.

237
README.md
View File

@@ -5,9 +5,11 @@ Ansible을 이용하여 Datasaker Agent를 설치할 수 있습니다.
## Requirements
- Ansible v2.6+가 필요합니다.
- 대부분의 데비안 리눅스 배포판을 지원합니다.
- 대부분의 Debian Linux 배포판을 지원합니다.
- 대부분의 Redhat Linux 배포판을 지원합니다.
- Amazon Linux 2 배포판을 지원합니다.
## Installation
Ansible Galaxy에서 Datasaker role을 설치합니다.
@@ -17,6 +19,8 @@ ansible-galaxy install dsk_bot.datasaker
```
에이전트를 배포하기 위하여 Ansible playbook을 작성합니다.
****`dsk-log-agent` 설치 시 `fluent-bit` 이 자동으로 설치됩니다.***
아래는 기본 설치에 대한 예시입니다.
@@ -28,7 +32,7 @@ ansible-galaxy install dsk_bot.datasaker
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_agents: ["dsk-node-agent"]
datasaker_agents: ["dsk-node-agent","dsk-log-agent"]
```
#### Docker Agent Default Install Example
```yml
@@ -43,99 +47,133 @@ ansible-galaxy install dsk_bot.datasaker
### 필수 설정
| 변수명 | 설명 |
|--------------------------------------------|--------------------------------------------------|
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datasaker_api_key`|API Key를 입력합니다.|
|`datasaker_agents`| 각 호스트에 설치하고자 하는 Host Agent 리스트입니다. <br>`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`<br>(Default) `dsk-node-agent`|
|`datasaker_docker_agents`| 각 호스트에 설치하고자 하는 Docker Container Agent 리스트입니다. <br>Docker Container Agents를 넣으면 Host Agent 설치는 자동으로 비활성화 됩니다. <br>`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`<br>(Default) `dsk-docker-node-agent`|
|`datasaker_agents`| 각 호스트에 설치하고자 하는 Host Agent 리스트입니다. <br>`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`<br>| `dsk-node-agent`|
|`datasaker_docker_agents`| 각 호스트에 설치하고자 하는 Docker Container Agent 리스트입니다. <br>Docker Container Agents를 넣으면 Host Agent 설치는 자동으로 비활성화 됩니다. <br>`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`<br>| `dsk-docker-node-agent`|
<!--
#### Datasaker 공통 설정
| 변수명 | 설명 |
|--------------------------------------------|--------------------------------------------------|
|`datagate_url`| Datasaker Agent가 전송하는 Datasaker Datagate URL 설정. <br>(Default) `gate.kr.datasaker.io`|
|`datagate_trace_url`| `dsk-trace-agent` Datagate URL 설정. <br>(Default) `datagate_url`|
|`datagate_trace_port`| `dsk-trace-agent` Datagate Port 설정. <br>(Default) `31300`|
|`datagate_trace_timeout`| `dsk-trace-agent` Data 전송 만료 시간 설정. <br>(Default) `5s`|
|`datagate_manifest_url`| `dsk-manifest-agent` Datagate URL 설정. <br>(Default) `datagate_url`|
|`datagate_manifest_port`| `dsk-manifest-agent` Datagate Port 설정. <br>(Default) `31301`|
|`datagate_manifest_timeout`| `dsk-manifest-agent` Data 전송 만료 시간 설정. <br>(Default) `5s`|
|`datagate_metric_url`| `dsk-metric-agent` Datagate URL 설정. <br>(Default) `datagate_url`|
|`datagate_metric_port`| `dsk-metric-agent` Datagate Port 설정. <br>(Default) `31302`|
|`datagate_metric_timeout`| `dsk-metric-agent` Data 전송 만료 시간 설정. <br>(Default) `5s`|
|`datagate_plan_url`| `dsk-plan-agent` Datagate URL 설정. <br>(Default) `datagate_url`|
|`datagate_plan_port`| `dsk-plan-agent` Datagate Port 설정. <br>(Default) `31303`|
|`datagate_plan_timeout`| `dsk-plan-agent` Data 전송 만료 시간 설정. <br>(Default) `5s`|
|`datagate_loggate_url`| `dsk-log-agent` Datagate URL 설정. <br>(Default) `datagate_url`|
|`datagate_loggate_port`| `dsk-log-agent` Datagate Port 설정. <br>(Default) `31304`|
|`datagate_loggate_timeout`| `dsk-log-agent` Data 전송 만료 시간 설정. <br>(Default) `5s`|
|`datasaker_api_url`| Datasaker API Server URL. <br>(Default) `api.kr.datasaker.io`|
|`datasaker_api_send_interval`| Datasaker API Server Data 전송 만료 시간 설정. <br>(Default) `1m`|
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datagate_url`| Datasaker Agent가 전송하는 Datasaker Datagate URL 설정. <br>| `gate.kr.datasaker.io`|
|`datagate_trace_url`| `dsk-trace-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_trace_port`| `dsk-trace-agent` Datagate Port 설정. <br>| `31300`|
|`datagate_trace_timeout`| `dsk-trace-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_manifest_url`| `dsk-manifest-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_manifest_port`| `dsk-manifest-agent` Datagate Port 설정. <br>| `31301`|
|`datagate_manifest_timeout`| `dsk-manifest-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_metric_url`| `dsk-metric-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_metric_port`| `dsk-metric-agent` Datagate Port 설정. <br>| `31302`|
|`datagate_metric_timeout`| `dsk-metric-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_plan_url`| `dsk-plan-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_plan_port`| `dsk-plan-agent` Datagate Port 설정. <br>| `31303`|
|`datagate_plan_timeout`| `dsk-plan-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_loggate_url`| `dsk-log-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_loggate_port`| `dsk-log-agent` Datagate Port 설정. <br>| `31304`|
|`datagate_loggate_timeout`| `dsk-log-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datasaker_api_url`| Datasaker API Server URL. <br>| `api.kr.datasaker.io`|
|`datasaker_api_send_interval`| Datasaker API Server Data 전송 만료 시간 설정. <br>| `1m`|
-->
### Docker Container Agent 설정
| 변수명 | 설명 |
|--------------------------------------------|--------------------------------------------------|
|`datasaker_docker_config_path`| Datasaker Global Config 위치 설정. <br> (Default) `~/.datasaker`|
|`datasaker_docker_global_config`| Datasaker Global Config 이름 설정. <br> (Default) `~/.datasaker/config.yml`|
|`docker_default_path`| Datasaker Docker Log Agent에 마운트할 Docker Log 수집 위치 설정. <br> (Default) `/var/lib/docker/containers/`|
|`datasaker_docker_path`| Datasaker Docker Agent Container 위치 설정. <br> (Default) `/var/datasaker`|
|`container_agent_restart_policy`| `dsk-container-agent` Container Restart Policy 설정. <br> (Default) `always`|
|`node_agent_restart_policy`| `dsk-node-agent` Container Restart Policy 설정. <br> (Default) `always`|
|`trace_agent_restart_policy`| `dsk-trace-agent` Container Restart Policy 설정. <br> (Default) `always`|
|`log_agent_restart_policy`| `dsk-log-agent` Container Restart Policy 설정. <br> (Default) `always`|
|`postgres_agent_restart_policy`| `dsk-postgres-agent` Container Restart Policy 설정. <br> (Default) `always`|
|`plan_postgres_agent_restart_policy`| `dsk-plan-postgres-agent` Container Restart Policy 설정. <br> (Default) `always`|
|`container_agent_log_level`| `dsk-container-agent` Log Level 설정. <br> (Default) `INFO`|
|`node_agent_log_level`| `dsk-node-agent` Log Level 설정. <br> (Default) `INFO`|
|`trace_agent_log_level`| `dsk-trace-agent` Log Level 설정. <br> (Default) `INFO`|
|`log_agent_log_level`| `dsk-log-agent` Log Level 설정. <br> (Default) `INFO`|
|`postgres_agent_log_level`| `dsk-postgres-agent` Log Level 설정. <br> (Default) `INFO`|
|`plan_postgres_agent_log_level`| `dsk-plan-postgres-agent` Log Level 설정. <br> (Default) `INFO`|
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datasaker_docker_config_path`| Datasaker Global Config 위치 설정. <br> | `~/.datasaker`|
|`datasaker_docker_global_config`| Datasaker Global Config 이름 설정. <br> | `~/.datasaker/config.yml`|
|`docker_default_path`| Datasaker Docker Log Agent에 마운트할 Docker Log 수집 위치 설정. <br> | `/var/lib/docker/containers/`|
|`datasaker_docker_path`| Datasaker Docker Agent Container 위치 설정. <br> | `/var/datasaker`|
|`container_agent_restart_policy`| `dsk-container-agent` Container Restart Policy 설정. <br> | `always`|
|`node_agent_restart_policy`| `dsk-node-agent` Container Restart Policy 설정. <br> | `always`|
|`trace_agent_restart_policy`| `dsk-trace-agent` Container Restart Policy 설정. <br> | `always`|
|`log_agent_restart_policy`| `dsk-log-agent` Container Restart Policy 설정. <br> | `always`|
|`postgres_agent_restart_policy`| `dsk-postgres-agent` Container Restart Policy 설정. <br> | `always`|
|`plan_postgres_agent_restart_policy`| `dsk-plan-postgres-agent` Container Restart Policy 설정. <br> | `always`|
|`container_agent_log_level`| `dsk-container-agent` Log Level 설정. <br> | `INFO`|
|`node_agent_log_level`| `dsk-node-agent` Log Level 설정. <br> | `INFO`|
|`trace_agent_log_level`| `dsk-trace-agent` Log Level 설정. <br> | `INFO`|
|`log_agent_log_level`| `dsk-log-agent` Log Level 설정. <br> | `INFO`|
|`postgres_agent_log_level`| `dsk-postgres-agent` Log Level 설정. <br> | `INFO`|
|`plan_postgres_agent_log_level`| `dsk-plan-postgres-agent` Log Level 설정. <br> | `INFO`|
<!--|`datasaker_docker_user`| Datasaker Docker Container Directory Ownership 설정. <br> (Default) `datasaker`|
|`datasaker_docker_group`| Datasaker Docker Container Directory Group 설정. <br> (Default) `datasaker`|
|`datasaker_docker_user_uid`| Datasaker Docker Container Agent User UID 설정 <br> (Default) `202306`|
|`datasaker_docker_user_gid`| Datasaker Docker Container Agent User GID 설정 <br> (Default) `202306`|
|`container_agent_image_tag`| `dsk-container-agent` Image tag 설정. <br> (Default) `latest`|
|`node_agent_image_tag`| `dsk-node-agent` Image tag 설정. <br> (Default) `latest`|
|`trace_agent_image_tag`| `dsk-trace-agent` Image tag 설정. <br> (Default) `latest`|
|`log_agent_image_tag`| `dsk-log-agent` Image tag 설정. <br> (Default) `latest`|
|`postgres_agent_image_tag`| `dsk-postgres-agent` Image tag 설정. <br> (Default) `latest`|
|`plan_postgres_agent_image_tag`| `dsk-plan-postgres-agent` Image tag 설정. <br> (Default) `latest`|-->
<!--|`datasaker_docker_user`| Datasaker Docker Container Directory Ownership 설정. <br> | `datasaker`|
|`datasaker_docker_group`| Datasaker Docker Container Directory Group 설정. <br> | `datasaker`|
|`datasaker_docker_user_uid`| Datasaker Docker Container Agent User UID 설정 <br> | `202306`|
|`datasaker_docker_user_gid`| Datasaker Docker Container Agent User GID 설정 <br> | `202306`|
|`container_agent_image_tag`| `dsk-container-agent` Image tag 설정. <br> | `latest`|
|`node_agent_image_tag`| `dsk-node-agent` Image tag 설정. <br> | `latest`|
|`trace_agent_image_tag`| `dsk-trace-agent` Image tag 설정. <br> | `latest`|
|`log_agent_image_tag`| `dsk-log-agent` Image tag 설정. <br> | `latest`|
|`postgres_agent_image_tag`| `dsk-postgres-agent` Image tag 설정. <br> | `latest`|
|`plan_postgres_agent_image_tag`| `dsk-plan-postgres-agent` Image tag 설정. <br> | `latest`|-->
### Datasaker Agent 상세 설정
- Host Agent 와 Docker Container Agent는 같은 설정값을 사용합니다.
| 변수명 | 설명 |
|--------------------------------------------|--------------------------------------------------|
|`trace_sampling_rate`| `dsk-trace-agent` 에서 collector에 적용되는 샘플링 비율 설정.<br>- 100 이상일 때 모든 데이터가 수집.<br> (Default) `10`|
|`log_collects`| `dsk-log-agent` 에서 로그 수집 컬렉션 구성 설정. 리스트의 각 항목에는 아래 항목들이 포함. |
|`log_collects[*].paths`| `dsk-log-agent` 에서 로그 수집을 위한 경로 설정. <br> (Default) [`host-agent`]=`/var/log/*.log`, [`docker-agent`]=`/var/log/sample/*/*.log` |
|`log_collects[*].exclude_paths`| `dsk-log-agent` 에서 로그 컬렉션에서 제외할 경로 설정. 값을 지정하지 않으면 수집 경로에 설정한 모든 로그 수집. <br> (Default) `None` |
|`log_collects[*].keywords`| `dsk-log-agent` 에서 로그를 필터링하기 위한 키워드 설정. 값을 지정하지 않으면 모든 로그 수집. <br> (Default) `None` |
|`log_collects[*].tag`| `dsk-log-agent` 에서 사용자 설정 태그. <br> (Default) `Default` |
|`log_collects[*].service.name`| `dsk-log-agent` 에서 수집하는 서비스명 설정. <br> (Default) `default` |
|`log_collects[*].service.category`| `dsk-log-agent` 에서 수집하는 서비스 분류값 설정. <br> `app` `database` `syslog` `etc` (Default) `etc` |
|`log_collects[*].service.type`| `dsk-log-agent` 에서 수집하는 서비스 타입 설정. <br> `postgres` `mysql` `java` `etc` (Default) `etc` |
|`log_collects[*].service.address`| `dsk-log-agent` 에서 수집하는 서비스 타입이 Database인 경우에 작성 <br>- 데이터베이스 host 및 port 정보 |
|`postgres_user_name`| `dsk-postgres-agent`에 Postgres user ID 설정. <br> (Default) `None` |
|`postgres_user_password`| `dsk-postgres-agent`에 Postgres user password 설정. <br> (Default) `None` |
|`postgres_database_address`| `dsk-postgres-agent`에 Postgres address 설정. <br> (Default) `None` |
|`postgres_database_port`| `dsk-postgres-agent`에 Postgres port 설정. <br> (Default) `None` |
|`plan_postgres_user_name`| `dsk-plan-postgres-agent`에 Plan Postgres user ID 설정. <br> (Default) `None` |
|`plan_postgres_user_password`| `dsk-plan-postgres-agent`에 Plan Postgres user password 설정. <br> (Default) `None` |
|`plan_postgres_database_address`| `dsk-plan-postgres-agent`에 Plan Postgres address 설정. <br> (Default) `None` |
|`plan_postgres_database_port`| `dsk-plan-postgres-agent`에 Plan Postgres port 설정. <br> (Default) `None` |
|`plan_postgres_database_name`| `dsk-plan-postgres-agent`에 Plan Postgres database 설정. <br> (Default) `None` |
|`plan_postgres_scrape_interval`| `dsk-plan-postgres-agent`에 Plan Postgres scrape interval 설정. <br> (Default) `30s` |
|`plan_postgres_scrape_timeout`| `dsk-plan-postgres-agent`에 Plan Postgres scrape timeout 설정. <br> (Default) `5s` |
|`plan_postgres_slow_query_standard`| `dsk-plan-postgres-agent`에 Plan Postgres slow query standard 설정. <br> (Default) `5s` |
|`plan_postgres_executor_number`| `dsk-plan-postgres-agent`에 Plan Postgres executor number 설정. <br> (Default) `10` |
|`plan_postgres_sender_number`| `dsk-plan-postgres-agent`에 Plan Postgres sender number 설정. <br> (Default) `10` |
|`plan_postgres_activity_query_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres activity query buffer 설정. <br> (Default) `50` |
|`plan_postgres_plan_sender_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres plan sender buffer 설정. <br> (Default) `50` |
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`trace_sampling_rate`| `dsk-trace-agent` 에서 collector에 적용되는 샘플링 비율 설정.<br>- 100 이상일 때 모든 데이터가 수집.<br> | `10`|
|`logs[*].service`|로그 수집 대상의 서비스 이름 설정.|`default`|
|`logs[*].tag`|로그 수집 대상의 태그 설정.|`None`|
|`logs[*].keyword`|로그 수집 키워드 설정. 키워드가 포함된 로그만 수집.|`None`|
|`logs[*].multiline.format`|멀티라인 로그 포맷 설정 (예 : go, java, ruby, python).|`None`|
|`logs[*].multiline.pattern`|멀티라인 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}).|`None`|
|`logs[*].masking[*].pattern`|마스킹할 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}) 사용자 커스텀 정규식 패턴 사용 가능.|`None`|
|`logs[*].masking[*].replace`|마스킹 패턴이 대체될 문자열 설정. (예 : ******).|`None`|
|`logs[*].collect.type`|로그 수집 방법 설정 (`file`, `driver` 중 하나의 값을 작성).|`file`|
|`logs[*].collect.category`|서비스 분류 설정 (`app`, `database`, `syslog`, `etc` 중 하나의 값을 작성).|`etc`|
|`logs[*].collect.address`|데이터베이스 host 및 port 정보 설정 (서비스 분류가 database인 경우 설정).|`None`|
|`logs[*].collect.file.paths`|로그 수집 대상 경로 설정. 예 : /var/log/sample/*.log.|`['/var/log/*.log']`|
|`logs[*].collect.file.exclude_paths`|로그 수집 제외 대상 경로 설정.|`None`|
|`postgres_user_name`| `dsk-postgres-agent`에 Postgres user ID 설정. <br> | `None` |
|`postgres_user_password`| `dsk-postgres-agent`에 Postgres user password 설정. <br> | `None` |
|`postgres_database_address`| `dsk-postgres-agent`에 Postgres address 설정. <br> | `None` |
|`postgres_database_port`| `dsk-postgres-agent`에 Postgres port 설정. <br> | `None` |
|`plan_postgres_user_name`| `dsk-plan-postgres-agent`에 Plan Postgres user ID 설정. <br> | `None` |
|`plan_postgres_user_password`| `dsk-plan-postgres-agent`에 Plan Postgres user password 설정. <br> | `None` |
|`plan_postgres_database_address`| `dsk-plan-postgres-agent`에 Plan Postgres address 설정. <br> | `None` |
|`plan_postgres_database_port`| `dsk-plan-postgres-agent`에 Plan Postgres port 설정. <br> | `None` |
|`plan_postgres_database_name`| `dsk-plan-postgres-agent`에 Plan Postgres database 설정. <br> | `None` |
|`plan_postgres_scrape_interval`| `dsk-plan-postgres-agent`에 Plan Postgres scrape interval 설정. <br> | `30s` |
|`plan_postgres_scrape_timeout`| `dsk-plan-postgres-agent`에 Plan Postgres scrape timeout 설정. <br> | `5s` |
|`plan_postgres_slow_query_standard`| `dsk-plan-postgres-agent`에 Plan Postgres slow query standard 설정. <br> | `5s` |
|`plan_postgres_executor_number`| `dsk-plan-postgres-agent`에 Plan Postgres executor number 설정. <br> | `10` |
|`plan_postgres_sender_number`| `dsk-plan-postgres-agent`에 Plan Postgres sender number 설정. <br> | `10` |
|`plan_postgres_activity_query_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres activity query buffer 설정. <br> | `50` |
|`plan_postgres_plan_sender_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres plan sender buffer 설정. <br> | `50` |
#### Ansible Playbook 상세 설정 Example
#### Ansible Playbook 상세 설정 Example (Linux)
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_docker_agents:
- "dsk-node-agent"
- "dsk-trace-agent"
- "dsk-log-agent"
- "dsk-postgres-agent"
postgres_user_name: sample
postgres_user_password: 1q2w3e4r
postgres_database_address: 0.0.0.0
postgres_database_port: 5432
plan_postgres_user_name: sample
plan_postgres_user_password: 1q2w3e4r
plan_postgres_database_address: 0.0.0.0
plan_postgres_database_name: sample
plan_postgres_database_port: 5432
logs:
- collect:
type: file
file:
paths:
- /var/log/*.log
- /datasaker/log/*.log
```
#### Ansible Playbook 상세 설정 Example (Docker)
```yml
- hosts: servers
become: true
@@ -157,34 +195,25 @@ ansible-galaxy install dsk_bot.datasaker
plan_postgres_database_address: 0.0.0.0
plan_postgres_database_name: sample
plan_postgres_database_port: 5432
log_collects:
- paths:
- "/var/log/sample/*/*.log"
exclude_paths: []
keywords: []
tag: "Default"
service:
name: "default"
category: "etc"
type: "etc"
- paths:
- "/var/log/sample/b4d5ac015a5a*/*.log"
service:
name: "docker-test"
category: "database"
type: "postgres"
address: "0.0.0.0:5432"
logs:
- collect:
type: file
file:
paths:
- /var/log/*.log
- /datasaker/log/*.log
```
## Uninstallation
Datasaker Agent를 제거 할 수 있습니다.
datasaker_clean은 uninstall이 `True`로 설정되어야 합니다.
| 변수명 | 설명 |
|--------------------------------------------|--------------------------------------------------|
|`uninstall`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent만 제거. <br> (Default) `False`|
|`datasaker_clean`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent 와 생성 된 폴더 및 설정 파일까지 제거. <br> (Default) `False`|
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`uninstall`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent만 제거. <br> | `False`|
|`datasaker_clean`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent 와 생성 된 폴더 및 설정 파일까지 제거. <br> | `False`|
#### Datasaker Agents Uninstall Example

239
README_en.md
View File

@@ -8,6 +8,7 @@ The Ansible Datasaker role installs and configures the Datasaker Agent and integ
- Requires Ansible v2.6+.
- Supports most Debian Linux distributions.
- Supports most RedHat Linux distributions.
- Supports Amazon Linux 2 distributions.
### Installation
@@ -20,6 +21,10 @@ ansible-galaxy install dsk_bot.datasaker
To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key to your playbook:
*** When installing `dsk-log-agent`, `fluent-bit` is automatically installed. ***
In this example:
###### Host Agent Default Install Example
```yml
- hosts: servers
@@ -28,7 +33,7 @@ To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_agents: ["dsk-node-agent"]
datasaker_agents: ["dsk-node-agent","dsk-log-agent"]
```
###### Docker Agent Default Install Example
```yml
@@ -43,97 +48,134 @@ To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key
#### Base Role variables
| Variable | Description |
|--------------------------------------------|--------------------------------------------------|
| Variable | Description | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datasaker_api_key`|Your Datasaker API key.|
|`datasaker_agents`|Set to Datasaker Host Agent.<br>`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`<br>(Default) `dsk-node-agent`|
|`datasaker_docker_agents`|Set to Datasaker Docker Agent.<br>`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`<br>(Default) `dsk-docker-node-agent`|
|`datasaker_agents`|Set to Datasaker Host Agent.<br>`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`<br>| `dsk-node-agent`|
|`datasaker_docker_agents`|Set to Datasaker Docker Agent.<br>`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`<br>| `dsk-docker-node-agent`|
<!--
#### Agent Global Config Role variables
| Variable | Description |
|--------------------------------------------|--------------------------------------------------|
|`datagate_url`|The site of the Datasaker intake to send Agent data to. <br>(Default) `gate.kr.datasaker.io`|
|`datagate_trace_url`|Override the `dsk-trace-agent` datagate url. <br>(Default) `datagate_url`|
|`datagate_trace_port`|Override the `dsk-trace-agent` datagate port. <br>(Default) `31300`|
|`datagate_trace_timeout`|Override the `dsk-trace-agent` data expiration time. <br>(Default) `5s`|
|`datagate_manifest_url`|Override the `dsk-manifest-agent` datagate url. <br>(Default) `datagate_url`|
|`datagate_manifest_port`|Override the `dsk-manifest-agent` datagate port. <br>(Default) `31301`|
|`datagate_manifest_timeout`|Override the `dsk-manifest-agent` data expiration time. <br>(Default) `5s`|
|`datagate_metric_url`|Override the `dsk-metric-agent` datagate url. <br>(Default) `datagate_url`|
|`datagate_metric_port`|Override the `dsk-metric-agent` datagate port. <br>(Default) `31302`|
|`datagate_metric_timeout`|Override the `dsk-metric-agent` data expiration time. <br>(Default) `5s`|
|`datagate_plan_url`|Override the `dsk-plan-agent` datagate url. <br>(Default) `datagate_url`|
|`datagate_plan_port`|Override the `dsk-plan-agent` datagate port. <br>(Default) `31303`|
|`datagate_plan_timeout`|Override the `dsk-plan-agent` data expiration time. <br>(Default) `5s`|
|`datagate_loggate_url`|Override the `dsk-log-agent` datagate url. <br>(Default) `datagate_url`|
|`datagate_loggate_port`|Override the `dsk-log-agent` datagate port. <br>(Default) `31304`|
|`datagate_loggate_timeout`|Override the `dsk-log-agent` data expiration time. <br>(Default) `5s`|
|`datasaker_api_url`|Override the datasaker api server url. <br>(Default) `api.kr.datasaker.io`|
|`datasaker_api_send_interval`|Override the datasaker api server data expiration time. <br>(Default) `1m`|
| Variable | Description | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datagate_url`|The site of the Datasaker intake to send Agent data to. <br>| `gate.kr.datasaker.io`|
|`datagate_trace_url`|Override the `dsk-trace-agent` datagate url. <br>| `datagate_url`|
|`datagate_trace_port`|Override the `dsk-trace-agent` datagate port. <br>| `31300`|
|`datagate_trace_timeout`|Override the `dsk-trace-agent` data expiration time. <br>| `5s`|
|`datagate_manifest_url`|Override the `dsk-manifest-agent` datagate url. <br>| `datagate_url`|
|`datagate_manifest_port`|Override the `dsk-manifest-agent` datagate port. <br>| `31301`|
|`datagate_manifest_timeout`|Override the `dsk-manifest-agent` data expiration time. <br>| `5s`|
|`datagate_metric_url`|Override the `dsk-metric-agent` datagate url. <br>| `datagate_url`|
|`datagate_metric_port`|Override the `dsk-metric-agent` datagate port. <br>| `31302`|
|`datagate_metric_timeout`|Override the `dsk-metric-agent` data expiration time. <br>| `5s`|
|`datagate_plan_url`|Override the `dsk-plan-agent` datagate url. <br>| `datagate_url`|
|`datagate_plan_port`|Override the `dsk-plan-agent` datagate port. <br>| `31303`|
|`datagate_plan_timeout`|Override the `dsk-plan-agent` data expiration time. <br>| `5s`|
|`datagate_loggate_url`|Override the `dsk-log-agent` datagate url. <br>| `datagate_url`|
|`datagate_loggate_port`|Override the `dsk-log-agent` datagate port. <br>| `31304`|
|`datagate_loggate_timeout`|Override the `dsk-log-agent` data expiration time. <br>| `5s`|
|`datasaker_api_url`|Override the datasaker api server url. <br>| `api.kr.datasaker.io`|
|`datasaker_api_send_interval`|Override the datasaker api server data expiration time. <br>| `1m`|
-->
#### Docker Agent Role variables
| Variable | Description |
|--------------------------------------------|--------------------------------------------------|
|`datasaker_docker_config_path`| Override the datasaker global config path. <br> (Default) `~/.datasaker`|
|`datasaker_docker_global_config`| Override the datasaker global config file name. <br> (Default) `~/.datasaker/config.yml`|
|`docker_default_path`| Override the docker containers path. <br> (Default) `/var/lib/docker/containers/`|
|`datasaker_docker_path`| Override the datasaker docker agent containers path. <br> (Default) `/var/datasaker`|
|`container_agent_restart_policy`| Override the restart policy for a `dsk-container-agent` container <br> (Default) `always`|
|`node_agent_restart_policy`| Override the restart policy for a `dsk-node-agent` container <br> (Default) `always`|
|`trace_agent_restart_policy`| Override the restart policy for a `dsk-trace-agent` container <br> (Default) `always`|
|`log_agent_restart_policy`| Override the restart policy for a `dsk-log-agent` container <br> (Default) `always`|
|`postgres_agent_restart_policy`| Override the restart policy for a `dsk-postgres-agent` container <br> (Default) `always`|
|`plan_postgres_agent_restart_policy`| Override the restart policy for a `dsk-plan-postgres-agent` container <br> (Default) `always`|
|`container_agent_log_level`| Override the `dsk-container-agent` log level <br> (Default) `INFO`|
|`node_agent_log_level`| Override the `dsk-node-agent` log level <br> (Default) `INFO`|
|`trace_agent_log_level`| Override the `dsk-trace-agent` log level <br> (Default) `INFO`|
|`log_agent_log_level`| Override the `dsk-log-agent` log level <br> (Default) `INFO`|
|`postgres_agent_log_level`| Override the `dsk-postgres-agent` log level <br> (Default) `INFO`|
|`plan_postgres_agent_log_level`| Override the `dsk-plan-postgres-agent` log level <br> (Default) `INFO`|
| Variable | Description | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datasaker_docker_config_path`| Override the datasaker global config path. <br> | `~/.datasaker`|
|`datasaker_docker_global_config`| Override the datasaker global config file name. <br> | `~/.datasaker/config.yml`|
|`docker_default_path`| Override the docker containers path. <br> | `/var/lib/docker/containers/`|
|`datasaker_docker_path`| Override the datasaker docker agent containers path. <br> | `/var/datasaker`|
|`container_agent_restart_policy`| Override the restart policy for a `dsk-container-agent` container <br> | `always`|
|`node_agent_restart_policy`| Override the restart policy for a `dsk-node-agent` container <br> | `always`|
|`trace_agent_restart_policy`| Override the restart policy for a `dsk-trace-agent` container <br> | `always`|
|`log_agent_restart_policy`| Override the restart policy for a `dsk-log-agent` container <br> | `always`|
|`postgres_agent_restart_policy`| Override the restart policy for a `dsk-postgres-agent` container <br> | `always`|
|`plan_postgres_agent_restart_policy`| Override the restart policy for a `dsk-plan-postgres-agent` container <br> | `always`|
|`container_agent_log_level`| Override the `dsk-container-agent` log level <br> | `INFO`|
|`node_agent_log_level`| Override the `dsk-node-agent` log level <br> | `INFO`|
|`trace_agent_log_level`| Override the `dsk-trace-agent` log level <br> | `INFO`|
|`log_agent_log_level`| Override the `dsk-log-agent` log level <br> | `INFO`|
|`postgres_agent_log_level`| Override the `dsk-postgres-agent` log level <br> | `INFO`|
|`plan_postgres_agent_log_level`| Override the `dsk-plan-postgres-agent` log level <br> | `INFO`|
<!--|`datasaker_docker_user`| Override Owner in the datasaker docker agent containers directory. <br> (Default) `datasaker`|
|`datasaker_docker_group`| Override Group in the datasaker docker agent containers directory. <br> (Default) `datasaker`|
|`datasaker_docker_user_uid`| Override uid in the datasaker user. <br> (Default) `202306`|
|`datasaker_docker_user_gid`| Override gid in the datasaker user. <br> (Default) `202306`|
|`container_agent_image_tag`| Override the `dsk-container-agent` Image tag. <br> (Default) `latest`|
|`node_agent_image_tag`| Override the `dsk-node-agent` Image tag. <br> (Default) `latest`|
|`trace_agent_image_tag`| Override the `dsk-trace-agent` Image tag. <br> (Default) `latest`|
|`log_agent_image_tag`| Override the `dsk-log-agent` Image tag. <br> (Default) `latest`|
|`postgres_agent_image_tag`| Override the `dsk-postgres-agent` Image tag. <br> (Default) `latest`|
|`plan_postgres_agent_image_tag`| Override the `dsk-plan-postgres-agent` Image tag. <br> (Default) `latest`|-->
<!--|`datasaker_docker_user`| Override Owner in the datasaker docker agent containers directory. <br> | `datasaker`|
|`datasaker_docker_group`| Override Group in the datasaker docker agent containers directory. <br> | `datasaker`|
|`datasaker_docker_user_uid`| Override uid in the datasaker user. <br> | `202306`|
|`datasaker_docker_user_gid`| Override gid in the datasaker user. <br> | `202306`|
|`container_agent_image_tag`| Override the `dsk-container-agent` Image tag. <br> | `latest`|
|`node_agent_image_tag`| Override the `dsk-node-agent` Image tag. <br> | `latest`|
|`trace_agent_image_tag`| Override the `dsk-trace-agent` Image tag. <br> | `latest`|
|`log_agent_image_tag`| Override the `dsk-log-agent` Image tag. <br> | `latest`|
|`postgres_agent_image_tag`| Override the `dsk-postgres-agent` Image tag. <br> | `latest`|
|`plan_postgres_agent_image_tag`| Override the `dsk-plan-postgres-agent` Image tag. <br> | `latest`|-->
#### Agents Setting Role variables
| Variable | Description |
|--------------------------------------------|--------------------------------------------------|
|`trace_sampling_rate`| Override The `dsk-trace-agent` sampling rate applied to the collector.<br>- When set to 100 or higher, all data is collected.<br> (Default) `10`|
|`log_collects`| An array of log collection configurations. Each item in the array includes the following. |
|`log_collects[*].paths`| An array of paths for log collection. <br> (Default) [`host-agent`]=`/var/log/*.log`, [`docker-agent`]=`/var/log/sample/*/*.log` |
|`log_collects[*].exclude_paths`| An array of paths to be excluded from the log collection. If the array is empty, no paths will be excluded. <br> (Default) `None` |
|`log_collects[*].keywords`| An array of keywords for filtering the logs. If the array is empty, no keyword filtering will be applied. <br> (Default) `None` |
|`log_collects[*].tag`| The tag for the log collection item. <br> (Default) `Default` |
|`log_collects[*].service.name`| The name of the service. <br> (Default) `default` |
|`log_collects[*].service.category`| The category of the service. <br> (Default) `etc` |
|`log_collects[*].service.type`| The type of the service. <br> (Default) `etc` |
|`log_collects[*].service.address`| The address of the service. <br> This field is optional and is used only for certain services that require an address. |
|`postgres_user_name`| Enter the Postgres user ID. <br> (Default) `None` |
|`postgres_user_password`| Enter the Postgres user password. <br> (Default) `None` |
|`postgres_database_address`| Enter the Postgres address. <br> (Default) `None` |
|`postgres_database_port`| Enter the Postgres port. <br> (Default) `None` |
|`plan_postgres_user_name`| Enter the Plan Postgres user ID. <br> (Default) `None` |
|`plan_postgres_user_password`| Enter the Plan Postgres user password. <br> (Default) `None` |
|`plan_postgres_database_address`| Enter the Plan Postgres address. <br> (Default) `None` |
|`plan_postgres_database_port`| Enter the Plan Postgres port. <br> (Default) `None` |
|`plan_postgres_database_name`| Enter the Plan Postgres database. <br> (Default) `None` |
|`plan_postgres_scrape_interval`| Override the Plan Postgres scrape interval <br> (Default) `30s` |
|`plan_postgres_scrape_timeout`| Override the Plan Postgres scrape timeout <br> (Default) `5s` |
|`plan_postgres_slow_query_standard`| Override the Plan Postgres slow query standard <br> (Default) `5s` |
|`plan_postgres_executor_number`| Override the Plan Postgres executor number <br> (Default) `10` |
|`plan_postgres_sender_number`| Override the Plan Postgres sender number <br> (Default) `10` |
|`plan_postgres_activity_query_buffer`| Override the Plan Postgres activity query buffer <br> (Default) `50` |
|`plan_postgres_plan_sender_buffer`| Override the Plan Postgres plan sender buffer <br> (Default) `50` |
| Variable | Description | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`trace_sampling_rate`| Override The `dsk-trace-agent` sampling rate applied to the collector.<br>- When set to 100 or higher, all data is collected.<br> | `10`|
|`logs[*].service`|Defines the service name of the log collection target.|`default`|
|`logs[*].tag`|Sets the tag of the log collection target.|`None`|
|`logs[*].keyword`|Sets the keyword for log collection. Only logs that include the keyword are collected.|`None`|
|`logs[*].multiline.format`|Sets the multiline log format (e.g.: go, java, ruby, python).|`None`|
|`logs[*].multiline.pattern`|Sets the multiline log pattern. (e.g.: ^\d{4}-\d{2}-\d{2}).|`None`|
|`logs[*].masking[*].pattern`|Sets the log pattern to be masked. (e.g.: ^\d{4}-\d{2}-\d{2}) User-defined regular expression patterns are possible.|`None`|
|`logs[*].masking[*].replace`|Sets the string that the masking pattern will be replaced with. (e.g.: *****).|`None`|
|`logs[*].collect.type`|Sets the method of log collection (Choose one from file, driver).|`file`|
|`logs[*].collect.category`|Sets the service category (Choose one from app, database, syslog, etc).|`etc`|
|`logs[*].collect.address`|Sets the database host and port information (required if service category is database).|`None`|
|`logs[*].collect.file.paths`|Sets the paths for log collection. Example: /var/log/sample/.log.|`['/var/log/*.log']`|
|`logs[*].collect.file.exclude_paths`|Sets the paths to be excluded from log collection.|`None`|
|`postgres_user_name`| Enter the Postgres user ID. <br> | `None` |
|`postgres_user_password`| Enter the Postgres user password. <br> | `None` |
|`postgres_database_address`| Enter the Postgres address. <br> | `None` |
|`postgres_database_port`| Enter the Postgres port. <br> | `None` |
|`plan_postgres_user_name`| Enter the Plan Postgres user ID. <br> | `None` |
|`plan_postgres_user_password`| Enter the Plan Postgres user password. <br> | `None` |
|`plan_postgres_database_address`| Enter the Plan Postgres address. <br> | `None` |
|`plan_postgres_database_port`| Enter the Plan Postgres port. <br> | `None` |
|`plan_postgres_database_name`| Enter the Plan Postgres database. <br> | `None` |
|`plan_postgres_scrape_interval`| Override the Plan Postgres scrape interval <br> | `30s` |
|`plan_postgres_scrape_timeout`| Override the Plan Postgres scrape timeout <br> | `5s` |
|`plan_postgres_slow_query_standard`| Override the Plan Postgres slow query standard <br> | `5s` |
|`plan_postgres_executor_number`| Override the Plan Postgres executor number <br> | `10` |
|`plan_postgres_sender_number`| Override the Plan Postgres sender number <br> | `10` |
|`plan_postgres_activity_query_buffer`| Override the Plan Postgres activity query buffer <br> | `50` |
|`plan_postgres_plan_sender_buffer`| Override the Plan Postgres plan sender buffer <br> | `50` |
###### Agent Variables Example
In this example:
###### Ansible Playbook Setting Example (Linux)
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_docker_agents:
- "dsk-node-agent"
- "dsk-trace-agent"
- "dsk-log-agent"
- "dsk-postgres-agent"
postgres_user_name: sample
postgres_user_password: 1q2w3e4r
postgres_database_address: 0.0.0.0
postgres_database_port: 5432
plan_postgres_user_name: sample
plan_postgres_user_password: 1q2w3e4r
plan_postgres_database_address: 0.0.0.0
plan_postgres_database_name: sample
plan_postgres_database_port: 5432
logs:
- collect:
type: file
file:
paths:
- /var/log/*.log
- /datasaker/log/*.log
```
###### Ansible Playbook Setting Example (Docker)
```yml
- hosts: servers
become: true
@@ -155,28 +197,25 @@ To deploy the Datasaker Agent on hosts, add the Datasaker role and your API key
plan_postgres_database_address: 0.0.0.0
plan_postgres_database_name: sample
plan_postgres_database_port: 5432
log_collects:
- paths:
- "/var/log/sample/*/*.log"
exclude_paths: []
keywords: []
tag: "Default"
service:
name: "default"
category: "etc"
type: "etc"
- paths:
- "/var/log/sample/b4d5ac015a5a*/*.log"
service:
name: "docker-test"
category: "database"
type: "postgres"
address: "0.0.0.0:5432"
logs:
- collect:
type: file
file:
paths:
- /var/log/*.log
- /datasaker/log/*.log
```
## Uninstallation
However for more control over the uninstall parameters, the following code can be used.
Datasaker Agent can be uninstalled.
For this, datasaker_clean should be set to True.
| Variable | Description | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`uninstall`| Only removes the agents specified in `datasaker_agents` or `datasaker_docker_agents`. <br>| `False`|
|`datasaker_clean`| Removes the agents specified in `datasaker_agents` or `datasaker_docker_agents` along with any generated folders and configuration files. <br> | `False`|
In this example:
```yml

236
README_ko.md Normal file
View File

@@ -0,0 +1,236 @@
# Ansible Datasaker Role
Ansible을 이용하여 Datasaker Agent를 설치할 수 있습니다.
## Requirements
- Ansible v2.6+가 필요합니다.
- 대부분의 Debian Linux 배포판을 지원합니다.
- 대부분의 Redhat Linux 배포판을 지원합니다.
- Amazon Linux 2 배포판을 지원합니다.
## Installation
Ansible Galaxy에서 Datasaker role을 설치합니다.
```shell
ansible-galaxy install dsk_bot.datasaker
```
에이전트를 배포하기 위하여 Ansible playbook을 작성합니다.
****`dsk-log-agent` 설치 시 `fluent-bit` 이 자동으로 설치됩니다.***
아래는 기본 설치에 대한 예시입니다.
#### Host Agent Default Install Example
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_agents: ["dsk-node-agent","dsk-log-agent"]
```
#### Docker Agent Default Install Example
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_docker_agents: ["dsk-docker-node-agent","dsk-docker-log-agent"]
```
### 필수 설정
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datasaker_api_key`|API Key를 입력합니다.|
|`datasaker_agents`| 각 호스트에 설치하고자 하는 Host Agent 리스트입니다. <br>`dsk-node-agent` `dsk-trace-agent` `dsk-log-agent` `dsk-postgres-agent` `dsk-plan-postgres-agent`<br>| `dsk-node-agent`|
|`datasaker_docker_agents`| 각 호스트에 설치하고자 하는 Docker Container Agent 리스트입니다. <br>Docker Container Agents를 넣으면 Host Agent 설치는 자동으로 비활성화 됩니다. <br>`dsk-docker-node-agent` `dsk-docker-trace-agent` `dsk-docker-log-agent` `dsk-docker-postgres-agent`<br>| `dsk-docker-node-agent`|
<!--
#### Datasaker 공통 설정
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datagate_url`| Datasaker Agent가 전송하는 Datasaker Datagate URL 설정. <br>| `gate.kr.datasaker.io`|
|`datagate_trace_url`| `dsk-trace-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_trace_port`| `dsk-trace-agent` Datagate Port 설정. <br>| `31300`|
|`datagate_trace_timeout`| `dsk-trace-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_manifest_url`| `dsk-manifest-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_manifest_port`| `dsk-manifest-agent` Datagate Port 설정. <br>| `31301`|
|`datagate_manifest_timeout`| `dsk-manifest-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_metric_url`| `dsk-metric-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_metric_port`| `dsk-metric-agent` Datagate Port 설정. <br>| `31302`|
|`datagate_metric_timeout`| `dsk-metric-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_plan_url`| `dsk-plan-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_plan_port`| `dsk-plan-agent` Datagate Port 설정. <br>| `31303`|
|`datagate_plan_timeout`| `dsk-plan-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datagate_loggate_url`| `dsk-log-agent` Datagate URL 설정. <br>| `datagate_url`|
|`datagate_loggate_port`| `dsk-log-agent` Datagate Port 설정. <br>| `31304`|
|`datagate_loggate_timeout`| `dsk-log-agent` Data 전송 만료 시간 설정. <br>| `5s`|
|`datasaker_api_url`| Datasaker API Server URL. <br>| `api.kr.datasaker.io`|
|`datasaker_api_send_interval`| Datasaker API Server Data 전송 만료 시간 설정. <br>| `1m`|
-->
### Docker Container Agent 설정
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`datasaker_docker_config_path`| Datasaker Global Config 위치 설정. <br> | `~/.datasaker`|
|`datasaker_docker_global_config`| Datasaker Global Config 이름 설정. <br> | `~/.datasaker/config.yml`|
|`docker_default_path`| Datasaker Docker Log Agent에 마운트할 Docker Log 수집 위치 설정. <br> | `/var/lib/docker/containers/`|
|`datasaker_docker_path`| Datasaker Docker Agent Container 위치 설정. <br> | `/var/datasaker`|
|`container_agent_restart_policy`| `dsk-container-agent` Container Restart Policy 설정. <br> | `always`|
|`node_agent_restart_policy`| `dsk-node-agent` Container Restart Policy 설정. <br> | `always`|
|`trace_agent_restart_policy`| `dsk-trace-agent` Container Restart Policy 설정. <br> | `always`|
|`log_agent_restart_policy`| `dsk-log-agent` Container Restart Policy 설정. <br> | `always`|
|`postgres_agent_restart_policy`| `dsk-postgres-agent` Container Restart Policy 설정. <br> | `always`|
|`plan_postgres_agent_restart_policy`| `dsk-plan-postgres-agent` Container Restart Policy 설정. <br> | `always`|
|`container_agent_log_level`| `dsk-container-agent` Log Level 설정. <br> | `INFO`|
|`node_agent_log_level`| `dsk-node-agent` Log Level 설정. <br> | `INFO`|
|`trace_agent_log_level`| `dsk-trace-agent` Log Level 설정. <br> | `INFO`|
|`log_agent_log_level`| `dsk-log-agent` Log Level 설정. <br> | `INFO`|
|`postgres_agent_log_level`| `dsk-postgres-agent` Log Level 설정. <br> | `INFO`|
|`plan_postgres_agent_log_level`| `dsk-plan-postgres-agent` Log Level 설정. <br> | `INFO`|
<!--|`datasaker_docker_user`| Datasaker Docker Container Directory Ownership 설정. <br> | `datasaker`|
|`datasaker_docker_group`| Datasaker Docker Container Directory Group 설정. <br> | `datasaker`|
|`datasaker_docker_user_uid`| Datasaker Docker Container Agent User UID 설정 <br> | `202306`|
|`datasaker_docker_user_gid`| Datasaker Docker Container Agent User GID 설정 <br> | `202306`|
|`container_agent_image_tag`| `dsk-container-agent` Image tag 설정. <br> | `latest`|
|`node_agent_image_tag`| `dsk-node-agent` Image tag 설정. <br> | `latest`|
|`trace_agent_image_tag`| `dsk-trace-agent` Image tag 설정. <br> | `latest`|
|`log_agent_image_tag`| `dsk-log-agent` Image tag 설정. <br> | `latest`|
|`postgres_agent_image_tag`| `dsk-postgres-agent` Image tag 설정. <br> | `latest`|
|`plan_postgres_agent_image_tag`| `dsk-plan-postgres-agent` Image tag 설정. <br> | `latest`|-->
### Datasaker Agent 상세 설정
- Host Agent 와 Docker Container Agent는 같은 설정값을 사용합니다.
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`trace_sampling_rate`| `dsk-trace-agent` 에서 collector에 적용되는 샘플링 비율 설정.<br>- 100 이상일 때 모든 데이터가 수집.<br> | `10`|
|`logs[*].service`|로그 수집 대상의 서비스 이름 설정.|`default`|
|`logs[*].tag`|로그 수집 대상의 태그 설정.|`None`|
|`logs[*].keyword`|로그 수집 키워드 설정. 키워드가 포함된 로그만 수집.|`None`|
|`logs[*].multiline.format`|멀티라인 로그 포맷 설정 (예 : go, java, ruby, python).|`None`|
|`logs[*].multiline.pattern`|멀티라인 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}).|`None`|
|`logs[*].masking[*].pattern`|마스킹할 로그 패턴 설정. (예 : ^\d{4}-\d{2}-\d{2}) 사용자 커스텀 정규식 패턴 사용 가능.|`None`|
|`logs[*].masking[*].replace`|마스킹 패턴이 대체될 문자열 설정. (예 : ******).|`None`|
|`logs[*].collect.type`|로그 수집 방법 설정 (`file`, `driver` 중 하나의 값을 작성).|`file`|
|`logs[*].collect.category`|서비스 분류 설정 (`app`, `database`, `syslog`, `etc` 중 하나의 값을 작성).|`etc`|
|`logs[*].collect.address`|데이터베이스 host 및 port 정보 설정 (서비스 분류가 database인 경우 설정).|`None`|
|`logs[*].collect.file.paths`|로그 수집 대상 경로 설정. 예 : /var/log/sample/*.log.|`['/var/log/*.log']`|
|`logs[*].collect.file.exclude_paths`|로그 수집 제외 대상 경로 설정.|`None`|
|`postgres_user_name`| `dsk-postgres-agent`에 Postgres user ID 설정. <br> | `None` |
|`postgres_user_password`| `dsk-postgres-agent`에 Postgres user password 설정. <br> | `None` |
|`postgres_database_address`| `dsk-postgres-agent`에 Postgres address 설정. <br> | `None` |
|`postgres_database_port`| `dsk-postgres-agent`에 Postgres port 설정. <br> | `None` |
|`plan_postgres_user_name`| `dsk-plan-postgres-agent`에 Plan Postgres user ID 설정. <br> | `None` |
|`plan_postgres_user_password`| `dsk-plan-postgres-agent`에 Plan Postgres user password 설정. <br> | `None` |
|`plan_postgres_database_address`| `dsk-plan-postgres-agent`에 Plan Postgres address 설정. <br> | `None` |
|`plan_postgres_database_port`| `dsk-plan-postgres-agent`에 Plan Postgres port 설정. <br> | `None` |
|`plan_postgres_database_name`| `dsk-plan-postgres-agent`에 Plan Postgres database 설정. <br> | `None` |
|`plan_postgres_scrape_interval`| `dsk-plan-postgres-agent`에 Plan Postgres scrape interval 설정. <br> | `30s` |
|`plan_postgres_scrape_timeout`| `dsk-plan-postgres-agent`에 Plan Postgres scrape timeout 설정. <br> | `5s` |
|`plan_postgres_slow_query_standard`| `dsk-plan-postgres-agent`에 Plan Postgres slow query standard 설정. <br> | `5s` |
|`plan_postgres_executor_number`| `dsk-plan-postgres-agent`에 Plan Postgres executor number 설정. <br> | `10` |
|`plan_postgres_sender_number`| `dsk-plan-postgres-agent`에 Plan Postgres sender number 설정. <br> | `10` |
|`plan_postgres_activity_query_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres activity query buffer 설정. <br> | `50` |
|`plan_postgres_plan_sender_buffer`| `dsk-plan-postgres-agent`에 Plan Postgres plan sender buffer 설정. <br> | `50` |
#### Ansible Playbook 상세 설정 Example (Linux)
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_docker_agents:
- "dsk-node-agent"
- "dsk-trace-agent"
- "dsk-log-agent"
- "dsk-postgres-agent"
postgres_user_name: sample
postgres_user_password: 1q2w3e4r
postgres_database_address: 0.0.0.0
postgres_database_port: 5432
plan_postgres_user_name: sample
plan_postgres_user_password: 1q2w3e4r
plan_postgres_database_address: 0.0.0.0
plan_postgres_database_name: sample
plan_postgres_database_port: 5432
logs:
- collect:
type: file
file:
paths:
- /var/log/*.log
- /datasaker/log/*.log
```
#### Ansible Playbook 상세 설정 Example (Docker)
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_api_key: "<YOUR_API_KEY>"
datasaker_docker_agents:
- "dsk-docker-node-agent"
- "dsk-docker-trace-agent"
- "dsk-docker-log-agent"
- "dsk-docker-postgres-agent"
postgres_user_name: sample
postgres_user_password: 1q2w3e4r
postgres_database_address: 0.0.0.0
postgres_database_port: 5432
plan_postgres_user_name: sample
plan_postgres_user_password: 1q2w3e4r
plan_postgres_database_address: 0.0.0.0
plan_postgres_database_name: sample
plan_postgres_database_port: 5432
logs:
- collect:
type: file
file:
paths:
- /var/log/*.log
- /datasaker/log/*.log
```
## Uninstallation
Datasaker Agent를 제거 할 수 있습니다.
datasaker_clean은 uninstall이 `True`로 설정되어야 합니다.
| 변수명 | 설명 | Default |
|--------------------------------------------|--------------------------------------------------|--------------------------------------------------|
|`uninstall`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent만 제거. <br> | `False`|
|`datasaker_clean`| `datasaker_agents` 또는 `datasaker_docker_agents` 에 작성된 Agent 와 생성 된 폴더 및 설정 파일까지 제거. <br> | `False`|
#### Datasaker Agents Uninstall Example
```yml
- hosts: servers
become: true
roles:
- role: dsk_bot.datasaker
vars:
datasaker_agents: ["<AGENT_NAME>"]
uninstall: True
datasaker_clean: True
```
<!--
## Troubleshooting
### Debian stretch
**Note:**
-->

4
defaults/main.yml
View File

@@ -26,7 +26,7 @@ datagate_loggate_timeout: 5s
datasaker_api_url: api.kr.datasaker.io
datasaker_api_send_interval: 1m
datasaker_yum_repo: "http://nexus.exem-oss.org/repository/datasaker-amazonlinux-2023"
datasaker_yum_repo: "https://nexus.exem-oss.org/repository/"
datasaker_yum_enabled: yes
datasaker_yum_gpgcheck: no
@@ -71,5 +71,7 @@ log_agent_log_level: "INFO"
postgres_agent_log_level: "INFO"
plan_postgres_agent_log_level: "INFO"
log_agent_port: 21212
uninstall: False
datasaker_clean: False

BIN
files/libpq-13.5-1.el8.x86_64.rpm Normal file
View File

Binary file not shown.

BIN
files/libpq-devel-13.5-1.el8.x86_64.rpm Normal file
View File

Binary file not shown.

4
meta/.galaxy_install_info
View File

@@ -1,2 +1,2 @@
install_date: Thu Jun 15 08:26:46 2023
version: 1.0.1
install_date: Wed Jun 28 05:49:19 2023
version: 1.0.2

5
meta/main.yml
View File

@@ -17,6 +17,11 @@ galaxy_info:
- name: EL
versions:
- '7'
- '8'
- '9'
- name: Amazon Linux
- '2'
- '2023'
galaxy_tags:
- monitoring
dependencies: []

23
tasks/dsk-debian-pkg.yml
View File

@@ -1,7 +1,14 @@
---
- set_fact:
datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}"
when: ansible_facts.distribution_version != "18.04"
# - set_fact:
# datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}"
# when: ansible_facts.distribution_version != "18.04"
- name: Set os-specific variables
set_fact:
os_specific: "{% if ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '7' %}datasaker-redhat-7
{% elif ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '8' %}datasaker-redhat-8
{% elif ansible_facts['os_family'] == 'Amazon' and ansible_facts['distribution_major_version'] == '2' %}datasaker-amazonlinux-2
{% elif ansible_facts['os_family'] == 'Debian'%}debian-repos
{% else %}unknown{% endif %}"
- name: "Create temporary directory for key manipulation"
tempfile:
@@ -53,7 +60,7 @@
update_cache: yes
force: yes
- name: Make Datasaker Log Directory
- name: "Make Datasaker Log Directory"
ansible.builtin.file:
path: "{{ datasaker_host_log_path }}/{{ item }}"
state: directory
@@ -77,13 +84,15 @@
name: "td-agent"
state: latest
update_cache: yes
when: '"dsk-log-agent" in datasaker_agents'
when:
- '"dsk-log-agent" in datasaker_agents'
- log_agent_image_tag != "latest"
- name: Reload systemd configuration
- name: "Reload systemd configuration"
ansible.builtin.systemd:
daemon_reload: True
- name: Restart dsk-agent service
- name: "Restart dsk-agent service"
ansible.builtin.systemd:
name: "{{ item }}"
enabled: true

22
tasks/dsk-docker-log-agent.yml
View File

@@ -19,13 +19,14 @@
src: docker-log-agent-config.yml.j2
dest: "{{ datasaker_docker_path }}/dsk-docker-log-agent-config.yml"
- name: Run Datasaker Log Agent
- name: "Run Datasaker Log Agent"
docker_container:
name: dsk-docker-log-agent
image: "datasaker/dsk-log-agent:{{ log_agent_image_tag }}"
state: started
detach: true
restart_policy: "{{ log_agent_restart_policy }}"
ports:
- "{{ log_agent_port }}:{{ log_agent_port }}"
env:
DKS_LOG_LEVEL: "{{ log_agent_log_level }}"
command: "-global.config=/etc/datasaker/global-config.yml -agent.config=/etc/datasaker/dsk-log-agent/agent-config.yml -mount.volume=true"
@@ -33,5 +34,18 @@
- "{{ datasaker_docker_path }}:/var/datasaker/"
- "{{ datasaker_docker_global_config }}:/etc/datasaker/global-config.yml:ro"
- "{{ datasaker_docker_path }}/dsk-docker-log-agent-config.yml:/etc/datasaker/dsk-log-agent/agent-config.yml:ro"
- "{{ item }}"
loop: "{{ custom_log_volume | default(['{{ docker_default_path }}:/var/log/sample/:ro']) }}"
- "{{ item }}:{{ item }}:ro"
with_items: "{{ custom_log_volume | default(['{{ docker_default_path }}']) }}"
- name: "Run APP with fluentd logging"
docker_container:
name: "{{ app_name }}"
image: "{{ app_image }}"
state: started
log_options:
log-driver: fluentd
log-opt:
fluentd-address: "{{ log_agent_port }}:{{ log_agent_port }}"
when:
- app_name is defined
- app_image is defined

45
tasks/dsk-log-agent.yml
View File

@@ -18,3 +18,48 @@
template:
src: log-agent-config.yml.j2
dest: "{{ datasaker_host_path }}/dsk-log-agent/agent-config.yml"
- name: "Install fluent-bit"
block:
- name: "Install fluent-bit"
shell: curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh
rescue:
- name: "Copy RPM files"
copy:
src: "{{ item }}"
dest: "/tmp/"
with_items:
- "{{ role_path }}/files/libpq-13.5-1.el8.x86_64.rpm"
- "{{ role_path }}/files/libpq-devel-13.5-1.el8.x86_64.rpm"
when:
- ansible_facts['os_family'] == 'RedHat'
- ansible_facts['distribution_major_version'] == '8'
- name: "Install libpq-13.5-1"
command: rpm -ivh /tmp/{{ item }}
with_items:
- libpq-13.5-1.el8.x86_64.rpm
- libpq-devel-13.5-1.el8.x86_64.rpm
when:
- ansible_facts['os_family'] == 'RedHat'
- ansible_facts['distribution_major_version'] == '8'
- template:
src: fluent-bit-repo.yml.j2
dest: "/etc/yum.repos.d/fluent-bit.repo"
when:
- ansible_facts['os_family'] == 'RedHat'
- ansible_facts['distribution_major_version'] == '7'
- yum:
name: "{{ item }}"
state: present
update_cache: yes
with_items:
- fluent-bit
- file:
path: /tmp/{{ item }}
state: absent
with_items:
- libpq-13.5-1.el8.x86_64.rpm
- libpq-devel-13.5-1.el8.x86_64.rpm
when:
- ansible_facts['os_family'] == 'RedHat'
- ansible_facts['distribution_major_version'] == '8'

26
tasks/dsk-redhat-pkg.yml
View File

@@ -1,14 +1,22 @@
---
- name: Set os-specific variables
set_fact:
os_specific: "{% if ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '7' %}datasaker-redhat-7
{% elif ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '8' %}datasaker-redhat-8
{% elif ansible_facts['os_family'] == 'RedHat' and ansible_facts['distribution_major_version'] == '9' %}datasaker-redhat-9
{% elif ansible_facts['os_family'] == 'Amazon' and ansible_facts['distribution_major_version'] == '2' %}datasaker-amazonlinux-2
{% else %}unknown{% endif %}"
- name: "Add datasaker repository"
yum_repository:
name: datasaker
description: datasaker-repo
baseurl: "{{ datasaker_yum_repo }}"
baseurl: "{{ datasaker_yum_repo }}{{ os_specific }}"
enabled: "{{ datasaker_yum_enabled }}"
gpgcheck: "{{ datasaker_yum_gpgcheck }}"
- set_fact:
datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}"
# - set_fact:
# datasaker_agents: "{{ datasaker_agents | reject('equalto', 'dsk-log-agent') | list }}"
- name: "Check datasaker Agent"
include_tasks: check-agent.yml
@@ -28,7 +36,9 @@
name: "td-agent"
state: latest
update_cache: yes
when: '"dsk-log-agent" in datasaker_agents'
when:
- '"dsk-log-agent" in datasaker_agents'
- log_agent_image_tag != "latest"
- name: "Install datasaker agent"
yum:
@@ -41,3 +51,11 @@
- Reload systemd configuration
- Restart {{ item }} service
ignore_errors: true
- name: "Setting dsk-log-agent config"
template:
src: log-agent-config.yml.j2
dest: "{{ datasaker_host_path }}/dsk-log-agent/agent-config.yml"
when:
- '"dsk-log-agent" in datasaker_agents'
- log_agent_image_tag == "latest"

63
templates/docker-log-agent-config.yml.j2
View File

@@ -1,32 +1,51 @@
agent:
metadata:
agent_name: "{{ log_agent_name | default('dsk-log-agent') }}"
agent_name: {{ metadata.agent_name | default('dsk-log-agent') }}
cluster_id: {{ metadata.cluster_id | default('unknown') }}
logs:
{% for log in logs | default([{'service': 'default', 'collect': {'type': 'file', 'category': 'etc', 'file': {'paths': ['/var/log/*.log']}}}]) %}
- service: {{ log.service | default('default') }}
tag:
{% for tag in log.tag | default([]) %}
- {{ tag }}
{% endfor %}
keyword:
{% for keyword in log.keyword | default([]) %}
- {{ keyword }}
{% endfor %}
multiline:
format: {{ log.multiline.format | default('') }}
pattern:
{% for pattern in log.multiline.pattern | default([]) %}
- {{ pattern }}
{% endfor %}
masking:
{% for mask in log.masking | default([]) %}
- pattern: {{ mask.pattern }}
replace: {{ mask.replace }}
{% endfor %}
collect:
{% for collect_item in log_collects | default([{'paths': ['/var/log/sample/*/*.log'], 'exclude_paths': [], 'keywords': [], 'tag': 'Default', 'service': {'name': 'default', 'category': 'etc', 'type': 'etc'}}]) %}
- paths:
{% for path in collect_item.paths %}
- "{{ path }}"
{% if log.collect.type == "file" %}
type: {{ log.collect.type | default('file') }}
category: {{ log.collect.category | default('etc') }}
address: {{ log.collect.address | default('') }}
file:
paths:
{% for path in log.collect.file.paths %}
- {{ path }}
{% endfor %}
exclude_paths:
{% if collect_item.exclude_paths | default([]) %}
{% for exclude_path in collect_item.exclude_paths %}
- "{{ exclude_path }}"
{% for exclude_path in log.collect.file.exclude_paths | default([]) %}
- {{ exclude_path }}
{% endfor %}
{% endif %}
keywords:
{% if collect_item.keywords | default([]) %}
{% for keyword in collect_item.keywords %}
- "{{- keyword -}}"
{% if log.collect.type == "driver" %}
type: {{ log.collect.type | default('driver') }}
category: {{ log.collect.category | default('etc') }}
driver:
containers:
{% for container in (log.collect.docker_driver.container if 'docker_driver' in log.collect else ['*']) %}
- "{{ container }}"
{% endfor %}
{% endif %}
{% if collect_item.tag is defined %}
tag: "{{ collect_item.tag }}"
{% endif %}
service:
name: "{{ collect_item.service.name }}"
category: "{{ collect_item.service.category }}"
type: "{{ collect_item.service.type }}"
{% if collect_item.service.address is defined %}
address: "{{ collect_item.service.address }}"
{% endif %}
{% endfor %}

7
templates/fluent-bit-repo.yml.j2 Normal file
View File

@@ -0,0 +1,7 @@
[fluent-bit]
name = Fluent Bit
baseurl = https://packages.fluentbit.io/centos/7/$basearch/
gpgcheck=1
gpgkey=https://packages.fluentbit.io/fluentbit.key
repo_gpgcheck=1
enabled=1

58
templates/log-agent-config.yml.j2
View File

@@ -1,32 +1,42 @@
agent:
metadata:
agent_name: "{{ log_agent_name | default('dsk-log-agent') }}"
agent_name: {{ metadata.agent_name | default('dsk-log-agent') }}
cluster_id: {{ metadata.cluster_id | default('unknown') }}
logs:
{% for log in logs | default([{'service': 'default', 'collect': {'type': 'file', 'category': 'etc', 'file': {'paths': ['/var/log/*.log']}}}]) %}
- service: {{ log.service | default('default') }}
tag:
{% for tag in log.tag | default([]) %}
- {{ tag }}
{% endfor %}
keyword:
{% for keyword in log.keyword | default([]) %}
- {{ keyword }}
{% endfor %}
multiline:
format: {{ log.multiline.format | default('') }}
pattern:
{% for pattern in log.multiline.pattern | default([]) %}
- {{ pattern }}
{% endfor %}
masking:
{% for mask in log.masking | default([]) %}
- pattern: {{ mask.pattern }}
replace: {{ mask.replace }}
{% endfor %}
collect:
{% for collect_item in log_collects | default([{'paths': ['/var/log/*.log'], 'exclude_paths': [], 'keywords': [], 'tag': 'Default', 'service': {'name': 'default', 'category': 'etc', 'type': 'etc'}}]) %}
- paths:
{% for path in collect_item.paths %}
- "{{ path }}"
{% if log.collect.type == "file" %}
type: {{ log.collect.type | default('file') }}
category: {{ log.collect.category | default('etc') }}
address: {{ log.collect.address | default('') }}
file:
paths:
{% for path in log.collect.file.paths %}
- {{ path }}
{% endfor %}
exclude_paths:
{% if collect_item.exclude_paths | default([]) %}
{% for exclude_path in collect_item.exclude_paths %}
- "{{ exclude_path }}"
{% for exclude_path in log.collect.file.exclude_paths | default([]) %}
- {{ exclude_path }}
{% endfor %}
{% endif %}
keywords:
{% if collect_item.keywords | default([]) %}
{% for keyword in collect_item.keywords %}
- "{{- keyword -}}"
{% endfor %}
{% endif %}
{% if collect_item.tag is defined %}
tag: "{{ collect_item.tag }}"
{% endif %}
service:
name: "{{ collect_item.service.name }}"
category: "{{ collect_item.service.category }}"
type: "{{ collect_item.service.type }}"
{% if collect_item.service.address is defined %}
address: "{{ collect_item.service.address }}"
{% endif %}
{% endfor %}