大网数据广告活动
服务器运维问答

How can I combine two commands to tally data from compressed and uncompressed log files?

· 52CE 我爱测

I am running this command to get a tally of failed login attempts from a server’s auth.log and it works well: sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | un…

问题原文

原帖链接 · Giacomo1968 · CC BY-SA 3.0

I am running this command to get a tally of failed login attempts from a server’s auth.log and it works well:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

But the issue is—like every server in the world—I have some GZipped archives created by log rotation that won’t be parsed by this command:

-rw-r----- 1 syslog adm  7822722 Oct 31 13:44 /var/log/auth.log
-rw-r----- 1 syslog adm 12532511 Oct 25 06:59 /var/log/auth.log.1
-rw-r----- 1 syslog adm  2250939 Oct 18 06:55 /var/log/auth.log.2.gz
-rw-r----- 1 syslog adm  2139669 Oct 11 07:06 /var/log/auth.log.3.gz
-rw-r----- 1 syslog adm  2769919 Oct  4 06:54 /var/log/auth.log.4.gz

To deal with those I have a variant of the command that uses zcat instead of cat but it’s a bit of a nuisance to have to run two commands to get this data. Is there any way to combine the cat and zcat commands into one command so I can get combined results? I know I could write some Bash script to filter through compressed versus uncompressed files, but I use this current one-liner on various servers and I want to have a one-liner I can quickly refer to and use when needed.

More details.

I’ve been doing some security auditing and firewall tweaking and have this command I run that gives me a nice tally of “Failed password” attempts from the auth.log files on an Ubuntu Linux machine:

sudo cat /var/log/auth.{log,log.1} | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

It works great! The output looks something like this:

 5909 Oct 18
13444 Oct 19
  351 Oct 20
  162 Oct 21
  499 Oct 22
  377 Oct 23
  145 Oct 24
10897 Oct 25
   76 Oct 26
   54 Oct 27
  310 Oct 28
 1024 Oct 29
  208 Oct 30
   30 Oct 31

And while this works well for uncompressed logs, since the logs rotate and get compressed there are always some GZip compressed files that would be nice to tally up as well. So I run this variant of the above command that uses zcat:

sudo zcat -q /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

And the output is something like this:

gzip: /var/log/auth.log: not in gzip format

gzip: /var/log/auth.log.1: not in gzip format
  10413 Sep 27
  15977 Sep 28
  12297 Sep 29
  14438 Sep 30
  23394 Oct 1
  12912 Oct 2
  15844 Oct 3
  19697 Oct 4
  15350 Oct 5
  12358 Oct 6
  12692 Oct 7
   8377 Oct 8
  10875 Oct 9
    565 Oct 10
  16027 Oct 11
  10422 Oct 12
   6808 Oct 13
  26891 Oct 14
   9493 Oct 15
   5138 Oct 16
   9415 Oct 17
   2226 Oct 18

As you can see, the output works when it works, but it would be nice if the cat and zcat variants of this command could simply be combined into one command. How can that be done?

Bonus Points:

These are not critical issues, but it would be nice if somehow they could be addressed in a solution:

  • Note that at the top of the zcat output there are two errors from zcat attempting to process two uncompressed files. Suppressing that would be nice.
  • Also note how the cat and zcat commands show data for October 18th; a day when log rotation happened. Would there be any way to add up those two values in the command? If not, I’m fine with having two lines of October 18th data with different values I would have to add up manually.

已采纳解答

原帖链接 · Brian · CC BY-SA 3.0

Can use zgrep which will decompress if necessary and therefore works with both straight text and compressed input. As well grep/zgrep can process multiple files directly which is needed in this case since mixed compressed and text to standard input doesn't always work as intended. Supress filenames in the grep output with -h or --no-filename.

sudo zgrep -h 'Failed password' /var/log/auth.* | grep sshd | awk '{print $1,$2}' | sort -k 1,1M -k 2n | uniq -c

Man page:

ZGREP(1)                                                              ZGREP(1)

NAME
       zgrep - search possibly compressed files for a regular expression

SYNOPSIS
       zgrep [ grep_options ] [ -e ] pattern filename...

DESCRIPTION
       Zgrep  invokes grep on compressed or gzipped files.  All options specified
       are passed directly to grep.  If no file is specified, then  the  standard
       input  is  decompressed if necessary and fed to grep.  Otherwise the given
       files are uncompressed if necessary and fed to grep.

       If the GREP environment variable is set, zgrep uses it as the grep program
       to be invoked.

AUTHOR
       Charles Levert ([email omitted])

SEE ALSO
       grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)

摘自公开归档,保留原文语言,版式经整理,外部媒体与邮箱地址已移除。

资料来源与适用范围

资料来源
Server Fault
适用范围
以原帖系统、软件版本与发布日期为准(2015-10-31)
内容语言
中文;命令、路径和错误标识保留技术原文
本地收录日期
2026-11-15
授权说明
CC BY-SA 3.0

正文与所需资源由本站本地存储。查看来源及更新记录 ↗

请按适用版本和实际日志确认原因;本站没有对每一种系统或设备逐项执行文中的操作。