Name Status User Plist
elasticsearch started gigas /Users/gigas/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plist
kibana started gigas /Users/gigas/Library/LaunchAgents/homebrew.mxcl.kibana.plist
logstash error gigas /Users/gigas/Library/LaunchAgents/homebrew.mxcl.logstash.plist
별도 설정파일없이 -f 옵션으로 시작하고 있기 때문에 brew services 로 등록되지는 않습니다.
/usr/local/bin/logstash -f syslog.conf
Using JAVA_HOME defined java: /Library/Java/JavaVirtualMachines/jdk-10.0.2.jdk/Contents/Home
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/local/Cellar/logstash/7.12.0/libexec/logs which is now configured via log4j2.properties
[2021-04-15T12:21:23,862][INFO ][logstash.runner ] Log4j configuration path used is: /usr/local/Cellar/logstash/7.12.0/libexec/config/log4j2.properties
[2021-04-15T12:21:23,874][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 10.0.2+13 on 10.0.2+13 +indy +jit [darwin-x86_64]"}
[2021-04-15T12:21:24,004][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-04-15T12:21:24,036][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"31d249d3-662b-4761-a658-482e72562d5c", :path=>"/usr/local/Cellar/logstash/7.12.0/libexec/data/uuid"}
[2021-04-15T12:21:24,642][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-04-15T12:21:25,664][INFO ][org.reflections.Reflections] Reflections took 38 ms to scan 1 urls, producing 23 keys and 47 values
[2021-04-15T12:21:26,763][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
[2021-04-15T12:21:26,924][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"}
[2021-04-15T12:21:27,123][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2021-04-15T12:21:27,126][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2021-04-15T12:21:27,137][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2021-04-15T12:21:27,202][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-04-15T12:21:27,262][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
[2021-04-15T12:21:27,263][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2021-04-15T12:21:27,293][INFO ][logstash.outputs.elasticsearch][main] Installing elasticsearch template to _template/logstash
[2021-04-15T12:21:27,362][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["/usr/local/etc/logstash/syslog.conf"], :thread=>"#<Thread:0x39654f6f run>"}
[2021-04-15T12:21:28,430][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.07}
[2021-04-15T12:21:28,709][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/local/Cellar/logstash/7.12.0/libexec/data/plugins/inputs/file/.sincedb_7a95d797537896d7bc2d7c8449373a40", :path=>["/var/log/*.log", "/var/log/messages", "/var/log/syslog"]}
[2021-04-15T12:21:28,740][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2021-04-15T12:21:28,774][INFO ][filewatch.observingtail ][main][976adb32015ef413e59b22b156956b8bec525ed6230bfb57972bc53d08535932] START, creating Discoverer, Watch with file and sincedb collections
[2021-04-15T12:21:28,792][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
"syslog_severity_code" => 5,
"message" => "Apr 15 12:21:35 gigasui-MacBookPro Google Chrome Helper[6300]: Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835",
"syslog_pid" => "6300",
"received_at" => "2021-04-15T03:21:36.161Z",
"@timestamp" => 2021-04-15T03:21:35.000Z,
"syslog_facility" => "user-level",
"syslog_message" => "Libnotify: notify_register_coalesced_registration failed with code 9 on line 2835",
"@version" => "1",
"syslog_hostname" => "gigasui-MacBookPro",
"host" => "gigasui-MacBookPro.local",
"syslog_program" => "Google Chrome Helper",
"received_from" => "gigasui-MacBookPro.local",
"path" => "/var/log/system.log",
"syslog_facility_code" => 1,
"syslog_severity" => "notice",
"type" => "syslog",
"syslog_timestamp" => "Apr 15 12:21:35"
}
편한 관리를 위해 brew services로 등록해보겠습니다.
pipeline.yml 에 설정된 conf 파일을 읽도록 처리가 되었기 때문에 bres services list 명령어로 상태 관리를 할 수 있습니다.
brew services start logstash
그리고 Logstash 자체도 REST API를 제공하고 있습니다.
정상적으로 실행이 되었는지 확인하기 위해 기본 설정된 9600 포트로 접속해봅니다.
curl localhost:9600/_node?pretty
Logstash설정이 완료 되었기 때문에 Kibana를 통해 시각화가 필요합니다.
설정에 되어있는 index가 자동으로 생성되지만 수동으로 등록 할 수도 있습니다.
Kibana 스택 관리 메뉴에서 index를 등록합니다.
http://localhost:5601/app/home#/
conf 파일의 output에 설정된 elasticsearch-> index 로 색인 등록을 할 수 있습니다.
syslog-demo를 입력하면 해당 인덱스를 찾았다는 문구가 보여지게 됩니다.
확인 후 Next step 버튼을 클릭합니다.
시간 필터 필드 이름으로 @timestamp 필드를 선택하고 'Create index pattern' 버특을 클릭합니다.
==> Successfully started `kibana` (label: homebrew.mxcl.kibana)
Kibana를 시작하고 모든 ELK 서비스가 실행 중인지 확인합니다.
logstash 서비스 상태는 error로 나오는데 다음글에서 다루겠습니다.
brew services list
Name Status User Plist
elasticsearch started gigas /Users/gigas/Library/LaunchAgents/homebrew.mxcl.elasticsearch.plist
kibana started gigas /Users/gigas/Library/LaunchAgents/homebrew.mxcl.kibana.plist
logstash error gigas /Users/gigas/Library/LaunchAgents/homebrew.mxcl.logstash.plist
