caboose

19 Responses to “sed, awk for rails devs”

1. Greg Says:
   July 1st, 2007 at 10:49 PM

   sed/awk tutorial? Yes, please! These are already so helpful and that stuff always seems like magic to me…

2. Tien Dung Says:
   July 2nd, 2007 at 12:38 AM

   Yes. especially related to Rail tutorial :)

3. Cristi Balan Says:
   July 2nd, 2007 at 02:04 AM

   That last thingy is missing the actual adding. Working version is:

   svn st | grep ? | sed “s/^? //” | xargs svn add
   

   Also, it does exactly the same thing as:

   svn add * –force
   

4. Matti Kotsalainen Says:
   July 2nd, 2007 at 02:20 AM

   Wow. I hadn’t bothered to look at those weird *nix tools like awk and sed but your example proved me wrong. I found a great awk-tutorial here. http://www.vectorsite.net/tsawk.html

5. Vrensk Says:
   July 2nd, 2007 at 04:36 AM

   Awk and sed are powerful, but as a Ruby (and former Perl) programmer, I’d rather draw on the tools that I know than feel lost once I try to do something too advanced. Following the lead from Perl, Ruby has internalised the most important capabilities of sed and awk. Thus

   cat log/development.log | grep Completed | awk ’ { print “[” $3 “] - ” $0 }’ | sort -nr
   

   can be written as

   cat log/development.log | ruby -nae ‘puts(“[” + $F[2] + “] - ” + $_) if /Completed/’ | sort -nr
   

   or even

   ruby -nae ‘puts(“[” + $F[2] + “] - ” + $_) if /Completed/’ log/development.log | sort -nr
   

   Note that the indexes are different: To Ruby ‘$0’ is the first split field, but to awk it’s the entire line. The Ruby (or Perl) flag combo “nae” can be said to start “awk-mode”: ‘n’ assumes while gets; ...; end loop around your program, ‘a’ auto-splits the current line into $F, and ‘e’ lets you specify a program line by line.

6. Vrensk Says:
   July 2nd, 2007 at 04:41 AM

   [Apparently I have insulted smartypants and am now incurring its wrath.] Awk and sed are powerful, but as a Ruby (and former Perl) programmer, I’d rather draw on the tools that I know than feel lost once I try to do something too advanced. Following the lead from Perl, Ruby has internalised the most important capabilities of sed and awk. Thus

       cat log/development.log | grep Completed | awk ' { print "[" $3 "] - " $0 }' | sort -nr
   

   can be written as

       cat log/development.log | ruby -nae 'puts("[" + $F[2] + "] - " + $_) if /Completed/' | sort -nr
   

   or even

       ruby -nae 'puts("[" + $F[2] + "] - " + $_) if /Completed/' log/development.log | sort -nr
   

   Note that the indexes are different: To Ruby ‘$0’ is the first split field, but to awk it’s the entire line. The Ruby (or Perl) flag combo “nae” can be said to start “awk-mode”: ‘n’ assumes while gets; ...; end loop around your program, ‘a’ auto-splits the current line into $F, and ‘e’ lets you specify a program line by line.

7. Jeremy Says:
   July 2nd, 2007 at 08:29 AM

   For committing files with a ?:

   svn st | grep ? | awk ‘{ print $2 }’ | xargs svn add
   

   For finding slow actions:

   cat production.log | grep ‘(0 reqs\/sec)’ | awk ‘{ print $3 ” - ” $0 }’ | sort -nr
   

   Jeremy

8. Morgan Says:
   July 2nd, 2007 at 02:34 PM

   Greetings, Oddly I found that the ‘sort -nr’ command doesn’t work because you added the [] around it. (Ignoring that copy-and-past with the generated ‘smartquotes’ doesn’t work either.)

   I had to do this to get double-digit seconds to sort properly (mind you this is with GNU coreutils sort, and should work similarly with the older textutils):

   cat log/development.log |
     grep Completed |
     awk ‘{ print “[” $3 “] - ” $0 }’ |
     sort -nr –key=1.1 |
     head
   

   Now double (and triple!) digit seconds are usually a sign that a service being called is failing, but it’s still the right thing to sort them to the top.

   Since ‘[’ is a non-numeric character, all the lines which start with it (which is all of them) are sorted normally, which is to say that [189.0] comes after [2.00].

   Hope this helps, and comes out formatted okay…

   – Morgan Schweers, CyberFOX!

9. Morgan Says:
   July 2nd, 2007 at 02:40 PM

   Greetings,

   Ouch.

   Uh…on the awk line, that’s apos, quote, quote, quote, quote, apos.

   On the sort line that’s a double-dash, and could probably be replaced with ‘-k’ and strike the =.

   Smartypants is still not so smart when it comes to code, it looks like. :(

   – Morgan

10. Helder Says:
    July 2nd, 2007 at 06:00 PM

    I’d very much like and sed & awk tutorial, if you please :)

11. Johannes Says:
    July 3rd, 2007 at 12:39 AM

    Don’t use cat something | grep whatever, it makes unix-admins cry.

    grep Completed log/development.log | awk ’ { print “[” $3 “] - ” $0 }’ | sort -nr

    See? Less pipes == more goody. Also, I’d go the ruby/perl route, too - why learn awk & sed when you have ruby (or perl - i like perl for stuff like that)?

12. Morgan Says:
    July 3rd, 2007 at 11:29 AM

    Greetings, Comparing the ruby/sort command by vrensk above using i386-linux ruby 1.8.5 (running in a VM on an old XEON box) on a 203K line development.log file, it consistently takes 883ms.

    With the grep/awk/sort it consistently takes 250ms.

    At that size, it doesn’t matter much, but if it scales at that same ratio (I don’t know)… In general you use grep/awk/sort because using a generalized programming language for this is like swatting flies with a tank.

    Use all the arrows in your quiver, not just one ‘uber’ arrow.

    – Morgan

    p.s. Humorous note, using cat and the additional pipe adds 200ms to the cat/grep/awk/sort pipe sequence, and a whopping 2 seconds to the cat/ruby/sort sequence. Definitely makes the unix admins cry. :)

13. Justin Says:
    July 5th, 2007 at 09:45 PM

    sed/awk tutorial: yes please!

14. Adi Azar Says:
    July 8th, 2007 at 11:21 PM

    Yes.

15. Pratik Says:
    July 9th, 2007 at 01:30 AM

    Just wanted to point out that the example you gave, could better be rewritten as:

    cat log/development.log | awk ’/Completed/ { print “[” $3 “] - ” $0 }’ | sort -nr

16. Mike Says:
    July 9th, 2007 at 04:07 PM

    Sed/awk tutorial would be awesome…yes!

17. emxnslsboc Says:
    July 31st, 2007 at 06:10 AM

    Hello! Good Site! Thanks you! rqdcfnymajpta

18. bquposizdw Says:
    July 31st, 2007 at 07:33 AM

    Thanks for this site! pq.qsvmk.cn u.zfwek.cn ggb.punwg.cn wp.mywev.cn tyvqk.lqwtm.cn szboi.tsdva.cn q.afsyf.cn csket.sryqx.cn af.fizyn.cn njs.fbnqg.cn uz.omemg.cn bh.mywev.cn pdgby.tsdva.cn zc.zlhmj.cn yj.osqkp.cn mjd.axvbe.cn o.hefeb.cn mnjmi.atvfm.cn apihx.hrzkp.cn cr.ypcoj.cn su.rnhmw.cn sgk.scvez.cn rnepf.fqtpq.cn loh.xlerr.cn fdnx.qzuxl.cn vg.mlubj.cn

19. zujbhvhiri Says:
    July 31st, 2007 at 08:57 AM

    Thanks for this site! xs.gmgpu.cn qgx.vpnds.cn ntema.zxzjg.cn evr.ofbqu.cn fb.vryyd.cn ylx.zibam.cn nm.zkijp.cn vf.wcknd.cn vp.wcknd.cn xxe.hypgr.cn ofb.oemjq.cn vxy.xopvz.cn u.wcknd.cn xnk.dhiam.cn b.avcqq.cn g.ohcxe.cn omlyy.yrhtb.cn o.gmrxs.cn nhiyx.ogpbw.cn tk.bxeci.cn ix.arphl.cn cyuye.jekxc.cn v.sfjld.cn g.bjjaf.cn

Sorry, comments are closed for this article.