Charlie Harvey

Bash Tip: Remove a line from middle of file

This is a thing that I’ve wanted to do for ages, and it came up for me during this week'd episode of Hak5, which I am getting to be a big fan of by the way. The material under discussion was what to do when one of your known hosts changes. Darren was saying he was a bit slack and just did an rm of his whole known_hosts file.

As you may know, when a host key has changed, you'll get a warning like this telling you so in no uncertain terms $ ssh -o stricthostkeychecking=ask ssh.example.com @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA host key has just been changed. The fingerprint for the RSA key sent by the remote host is 4a:ad:fe:96:de:02:95:f1:76:34:23:12:e4:78:12:bb. Please contact your system administrator. Add correct host key in /home/xahria/.ssh/known_hosts to get rid of this message. Offending key in /home/xahria/.ssh/known_hosts:8 RSA host key for localhost has changed and you have requested strict checking. Host key verification failed. As you can see the offending key is on line 8 of our known hosts file. So I’d like to be able to remove that line.

Now there is probably a tool for this or something, but I thought it was a nice quick one liner to do in bash, a-la Commandline Fu. My first shot used nested backticks and piping the output of wc -l to bc. Fugly. I think this version is a little nicer.

My test data is a file t with contents1 2 3 4 5 Exciting stuff. My goal is to rewrite the file so that it loses line 3.

Here’s how I do it. (x=3; head -n$(($x-1)) t; tail -n$((`wc -l t | cut -d ' ' -f 1`-$x)) t) > t2 && cat t2 > t And lets just check it works: $ cat t 1 2 4 5

There is quite a lot going on so let's step through it.

  • First of all we are using parentheses to group the commands that do the magic (we will come back to them in a moment). This, we assume works, and so the result is printed to a temp file, t2.
  • We then use && to wait for the previous commands to complete and cat t2 to t. We could just have well used cp.
  • Back to the magic. head prints the first lines of a file and tail the last lines. You can specify how many lines with the -n option. So a line out of the middle of a file is the output of head -n (line number minus 1) plus tail (length of file minus line number).
  • We assign the line number to x and use the ; seperator to tell Bash we are going to do another thing in this line. We use Bash’s $((2+2)) arithmetic syntax to work out line number - 1. We use the number we get by doing this as the argument for tail -n
  • Working out the length of the file is harder. The wc utility comes to our aid. wc -l tells the number of lines in a file. Unfortunately it also prints the file name, so we use cut -d ' ' -f 1 (for the first field using a space character as a delimeter) to tidy it up. This we wrap in backticks so it is evaluated first. We then wrap the whole lot in a $(()) and subtract the line number (x) from it. The resulting number is our -n for tail.
  • At this stage we can be confident that the content of the commands within the brackets is what we want and we're ready to feed it to t2.

In the real world, you would use the "Offending key" line number from the SSH warning as your x and ~/.ssh/known_hosts as your t. I suppose you could make a proper script of this if you wanted. Being a perl geek, I would perl -nle 'print if(++$i!=3)' known_hostsBut playing Bash is fun sometimes, eh?


Comments

  • Be respectful. You may want to read the comment guidelines before posting.
  • You can use Markdown syntax to format your comments. You can only use level 5 and 6 headings.
  • You can add class="your language" to code blocks to help highlight.js highlight them correctly.

Privacy note: This form will forward your IP address, user agent and referrer to the Akismet, StopForumSpam and Botscout spam filtering services. I don’t log these details. Those services will. I do log everything you type into the form. Full privacy statement.