LinuxÏÂÈÕÖ¾ÖÎÀíÓëÆÊÎöµÄ×î¼Ñ¹¤¾ßºÍÊÖÒÕ
linuxÏÂÈÕÖ¾ÖÎÀíÓëÆÊÎöµÄ×î¼Ñ¹¤¾ßºÍÊÖÒÕ
µ¼ÑÔ£º
ÔÚLinuxϵͳÖУ¬ÈÕÖ¾ÊǺÜÊÇÖ÷ÒªµÄ×é³É²¿·Ö¡£ËüÃǼͼÁËϵͳµÄÔËÐÐ״̬ºÍÊÂÎñ£¬ÎªÏµÍ³ÖÎÀíÔ±ÌṩÁËÒªº¦µÄÐÅÏ¢À´ÅŲé¹ÊÕϺÍÆÊÎöϵͳÐÔÄÜ¡£¿ÉÊÇ£¬Ëæ×ÅЧÀÍÆ÷¹æÄ£µÄÔöÌíºÍÈÕÖ¾Á¿µÄÒ»Ö±ÔöÌí£¬ÊÖ¶¯ÖÎÀíºÍÆÊÎöÈÕÖ¾±äµÃ²»¿ÉÐС£Òò´Ë£¬Ñ°ÕÒÒ»ÖÖ¸ßЧºÍ¿É¿¿µÄÈÕÖ¾ÖÎÀíºÍÆÊÎö¹¤¾ß±äµÃÖÁ¹ØÖ÷Òª¡£±¾ÎĽ«ÏÈÈݼ¸ÖÖÔÚLinuxÏÂÆÕ±éʹÓõÄ×î¼Ñ¹¤¾ßºÍÊÖÒÕ¡£
syslog-ng
syslog-ngÊÇÒ»¸ö¹¦Ð§Ç¿Ê¢µÄÈÕÖ¾ÍøÂçºÍת·¢¹¤¾ß£¬ÓÃÓÚÖÎÀíºÍÆÊÎöϵͳÈÕÖ¾¡£Ëü¾ßÓÐÎÞаµÄÉèÖÃÑ¡ÏÄܹ»´ÓÖÖÖÖȪԴÍøÂçÈÕÖ¾£¬²¢½«ËüÃÇ·¢Ë͵½Ö¸¶¨µÄÄ¿µÄ¡£ÏÂÃæÊÇÒ»¸öʹÓÃsyslog-ngÍøÂçºÍת·¢ÈÕÖ¾µÄʾÀýÉèÖÃÎļþ£º
source s_network { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); }; destination d_file { file("/var/log/mylog.log"); }; log { source(s_network); destination(d_file); };
µÇ¼ºó¸´ÖÆ
ÒÔÉÏÉèÖý«¼àÌýËùÓÐÍøÂçÉϵÄ514¶Ë¿Ú£¬²¢½«ÎüÊÕµ½µÄÈÕÖ¾ÉúÑĵ½/var/log/mylog.logÎļþÖС£Í¨¹ýsyslog-ngµÄÉèÖã¬Äú¿ÉÒÔƾ֤×Ô¼ºµÄÐèÇóÎÞаµØÖÎÀíºÍת·¢ÈÕÖ¾¡£
Logstash
LogstashÊÇÒ»¸öÇ¿Ê¢µÄ¿ªÔ´ÈÕÖ¾ÍøÂç¡¢´¦ÀíºÍ´«Ê乤¾ß¡£ËüÄܹ»Í¨¹ýÖÖÖÖÊäÈë²å¼þÍøÂçÈÕÖ¾Êý¾Ý£¬È»ºó¾ÓɹýÂ˺ʹ¦Àíºó½«Æä·¢Ë͵½Êä³ö²å¼þµÄÄ¿µÄλÖá£ÏÂÃæÊÇÒ»¸öʹÓÃLogstashÍøÂçºÍÆÊÎöApache»á¼ûÈÕÖ¾µÄʾÀýÉèÖãº
input { file { path => "/var/log/apache2/access.log" start_position => "beginning" } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } date { match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] } } output { elasticsearch { hosts => ["localhost:9200"] index => "apache-access-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } }
µÇ¼ºó¸´ÖÆ
ÒÔÉÏÉèÖý«´ÓÖ¸¶¨Â·¾¶ÍøÂçApache»á¼ûÈÕÖ¾£¬²¢Ê¹ÓÃGrokģʽƥÅäºÍDate²å¼þ¶ÔÈÕÖ¾¾ÙÐÐÆÊÎöºÍת»»¡£È»ºó£¬Ëü½«Í¨¹ýElasticsearch²å¼þ½«´¦ÀíºóµÄÈÕÖ¾·¢Ë͵½ElasticsearchЧÀÍÆ÷£¬²¢½«ÆäË÷Òýµ½ÈÕÆÚÃûÌõÄË÷ÒýÖС£
Elasticsearch
ElasticsearchÊÇÒ»¸öÂþÑÜʽµÄËÑË÷ºÍÆÊÎöÒýÇ棬ÌØÊâÊÊÊÊÓÃÓÚ´æ´¢ºÍÆÊÎö´ó×ÚµÄÈÕÖ¾Êý¾Ý¡£ËüÄܹ»¸ßЧµØË÷ÒýºÍËÑË÷Êý¾Ý£¬²¢ÌṩÁËÎÞаµÄÅÌÎʺ;ۺϹ¦Ð§¡£ÏÂÃæÊÇÒ»¸öʹÓÃElasticsearch¾ÙÐмòÆÓÈÕÖ¾ËÑË÷ºÍ¾ÛºÏµÄʾÀý´úÂ룺
# ËÑË÷ËùÓк¬ÓС°error¡±µÄÈÕÖ¾ GET /mylog/_search { "query": { "match": { "message": "error" } } } # ¾ÛºÏͳ¼Æÿ¸ö¼¶±ðµÄÈÕÖ¾ÊýÄ¿ GET /mylog/_search { "size": 0, "aggs": { "log_level": { "terms": { "field": "level.keyword" } } } }
µÇ¼ºó¸´ÖÆ
ÒÔÉÏ´úÂ뽫ÔÚÃûΪ”mylog”µÄË÷ÒýÖÐËÑË÷°üÀ¨”error”Òªº¦×ÖµÄÈÕÖ¾£¬²¢Í³¼Æÿ¸öÈÕÖ¾¼¶±ðµÄÊýÄ¿¡£
×ܽ᣺
ÈÕÖ¾ÖÎÀíºÍÆÊÎö¹ØÓÚϵͳÖÎÀíºÍ¹ÊÕÏÅŲéÖÁ¹ØÖ÷Òª¡£±¾ÎÄÏÈÈÝÁËÔÚLinuxƽ̨ÏÂ×î¼ÑµÄÈÕÖ¾ÖÎÀíºÍÆÊÎö¹¤¾ßºÍÊÖÒÕ£¬°üÀ¨syslog-ng¡¢LogstashºÍElasticsearch¡£Í¨¹ýºÏÀíÉèÖúÍʹÓÃÕâЩ¹¤¾ß£¬¿ÉÒÔ¸ßЧµØÖÎÀíºÍÆÊÎöϵͳÈÕÖ¾£¬ÌáÉýϵͳÐÔÄܺ͹ÊÕÏÅŲéµÄÄÜÁ¦¡£Ï£Íû±¾ÎĶԶÁÕßÔÚLinuxÈÕÖ¾ÖÎÀíºÍÆÊÎö·½ÃæÓÐËù×ÊÖú¡£
ÒÔÉϾÍÊÇLinuxÏÂÈÕÖ¾ÖÎÀíÓëÆÊÎöµÄ×î¼Ñ¹¤¾ßºÍÊÖÒÕµÄÏêϸÄÚÈÝ£¬¸ü¶àÇë¹Ø×¢±¾ÍøÄÚÆäËüÏà¹ØÎÄÕ£¡