Apple, Computers, Linux, Open-source, Software

Tell Tar to Auto Compress Those Files!

Just discovered a handy shortcut when working with the GNU utility “tar”. Like many other Unix utilities, the switches that you can pass tar change its behavior. To create a “plain” tar file (compressing multiple files down to a single tape archive format – similar to zip) you can execute the following:

bsimpson@Saturn:/tmp$ echo "test" > test
bsimpson@Saturn:/tmp$ tar -cf test.tar test
bsimpson@Saturn:/tmp$ file test*
test:     ASCII text
test.tar: POSIX tar archive (GNU)

The “c” switch creates a new archive, and the “f” switch tells it that the name of the new archive follows. Similarly, we can untar this using the “x” switch, mutually exclusive with “c” for its create counterpart.

We can also apply compression to these new archives:

bsimpson@Saturn:/tmp$ echo "test" > test
bsimpson@Saturn:/tmp$ tar -czf test.tar.gz test
bsimpson@Saturn:/tmp$ file test*
test:        ASCII text
test.tar.gz: gzip compressed data, from Unix, last modified: Tue Jan 11 22:31:34 2011

Or alternately, we can provide the bzip2 compression:

bsimpson@Saturn:/tmp$ echo "test" > test
bsimpson@Saturn:/tmp$ tar -cjf test.tar.bz2 test
bsimpson@Saturn:/tmp$ file test*
test:         ASCII text
test.tar.bz2: bzip2 compressed data, block size = 900k

All well and good, however tar allows you to specify the compression (or lack thereof) based on the new filename. You call it what you want, and tar will figure out what compression to apply. Consider the following:

bsimpson@Saturn:/tmp$ echo "test" > test
bsimpson@Saturn:/tmp$ tar -caf test.tar test
bsimpson@Saturn:/tmp$ tar -caf test.tar.gz test
bsimpson@Saturn:/tmp$ tar -caf test.tar.bz2 test
bsimpson@Saturn:/tmp$ file test*
test:         ASCII text
test.tar:     POSIX tar archive (GNU)
test.tar.bz2: bzip2 compressed data, block size = 900k
test.tar.gz:  gzip compressed data, from Unix, last modified: Tue Jan 11 22:24:43 2011

As a side note, these do not stack. For example “test.tar.gz.bz2” just produces a bzip2 encoded ASCII test file. If you *really* wanted this, you could use the pipe command to chain your compressions.

Hope this saves some time!

Advertisements
Computers, Family, Hardware, Linux, Open-source, Personal, Software, Windows

Bringing the Dead Back to Life

destroyeddriveNo, not zombies. A few days ago, a friend of mine brought me a computer and told me that it was running REALLY slow. I did some investigation, and quickly discovered after running some diagnostics that the hard drive was on the fritz. They had been wanting a new hard drive for a while anyways. One that was bigger, faster, and one that worked. I went to Newegg.com, bought a replacement SATA drive, and after arrival, began the task of migrating to this new drive.

The installation of Windows that was on the old drive was fine (aside from the drive itself causing errors). I didn’t want to reinstall Windows, as it is such a painful and time-consuming process. Hunt down drivers, copy over the documents, reconfigure settings, install the old software again. No thanks. Instead, I decided to explore my options with the open source utility dd. Why not Norton Ghost, or one of the dozens of other programs? They all cost money, and I suspect some just use a ported version of dd to do their dirty work behind the scenes. I am cutting out the middle-man (and the cost) and going straight for dd.

dd is a common Unix program whose primary purpose is the low-level copying and conversion of raw data” – Wikipedia

dd allows a user to copy data from a source to a destination bit for bit – an exact mirror image. Even better, if the device has corruptions, you can use dd_rescue. The concept is the same, however this utility assumes that the source is likely damaged and does some special handling for bad sectors.

Making the backup

To make your disk copy, boot the source computer to a Linux live CD such as Ubuntu. A live CD may or may not mount your hard drive inside of the operating system. If it does, you want to unmount this device. In Ubuntu, the mount directory is in “/media”. If you browse for files here, and see a Windows partition, unmount this first. This ensures that there are no write operations occurring during the copy of the disk.

Next, fire up dd_rescue and use the following syntax:

sudo dd_rescue /dev/sda - | ssh user@remote_server "dd of=/path/to/save/backup"
  • sudo runs this command with elevated permissions
  • dd_rescue needs the two parameters of <source> and <destination>
  • Our source is /dev/sda. Note that we are copying the entire drive (sda) instead of a partition (such as sda1, sda2, etc). Your device may differ from “sda”.
  • Our destination is , which is shorthand for standard out. This would direct the binary content to the screen
  • The “|” symbol (pipe) is redirecting the previous output to a new process
  • ssh is a secure way to copy files from one machine to another. In this case, we are specifying a username and a remote server (ip)
  • Finally, “dd” is executed on the remote server with the parameter “of” set to the path where the image will be saved to

Note that you will need to have sudo access, an ssh account on another machine, and permissions to write to the path you will be saving to. Also, make sure that your replacement drive is the same size, or larger than your source drive. Make sense right?

This process will take a while, depending on the damage to the drive, and the size, network speed, etc. I maxed out my router at around 11 MBps (100 Mbps). dd_rescue will provide output, and let you know when it is complete. An 80 GB hard drive took about 2 hours to complete.

Restoring the Backup

Once this completes, you are ready to shut down the source machine, and swap out the bad hard drive for the new hard drive. Reboot to your live CD, and run the following command (taking some values from earlier):

ssh user@remote_server "dd if=/path/to/save/backup"| sudo dd of=/dev/sda
  • We are using ssh again, this time to pull the image back across to the client
  • On the remote server, we are executing dd if. Note the if. We are using this backup as the input file, instead of the output file (of)
  • We are piping this output into our local machine
  • Using sudo, we execute dd with the of parameter. This will do a bit by bit copy of the image to the destination media.

Note that again you want to ensure that the target is not mounted in any way. Also, we are doing the entire hard drive, so I am just using “/dev/sda”

Guess what?! This process will take a while, just like the copy operation before.

Expand the New Partition

Note that if you have a replacement drive that is larger, you will need to expand the Windows partition. This makes sense, since an 80GB drive has less bits than a 250GB drive. When the image (that is 80GB) finishes copying, the rest of the hard drive is completely untouched. To address this behavior, and be able to use the rest of the replacement hard drive, we will need to expand this partition.

You may need to run a “chkdsk /R” a few times in Windows if your hard drive has any bad sectors. As a note, new drives usually have a few bad sectors that need to be identified and disabled. After you run chkdsk, fire up your Live CD again, and launch gparted. This is a graphical tool for managing hard drive partitions. You should see the unallocated space at the end of your drive. Click the preceding partition (NTFS, etc) and choose to resize this partition, taking up all of the unallocated space. Apply your changes, and in a while, you have a full sized hard drive partition for Windows.

If you receive an error about bad sectors preventing ntfsresize from running from gparted, you may not be able to continue. Despite running chkdsk and restarting multiple times as instructed, I was not able to continue in gparted due to this message. There is switch –bad-sectors that can be called for ntfsresize arguments, however the GUI gparted does not let you do this. I first tried setting an alias, but it seems the program references the full path of the command, so the alias did not work.  Finally, I arrived at this solution:

# sudo su
# mv /usr/sbin/ntfsresize /usr/sbin/ntfsresize.real
# touch /usr/sbin/ntfsresize
# chmod a+x /usr/sbin/ntfsresize

Now, edit the file ntfsresize file you just created and do something like the following code:

#!/bin/bash
ntfsresize.real $@ --bad-sectors
exit 0
  • We are becoming root with sudo su
  • We move the existing ntfsresize file to another filename (ntfsresize.real)
  • We then create a new file named ntfsresize with execute permissions, and edit this file
  • Inside this file, we call the original ntfsresize, with our –bad-sectors switch, and $@. This is a way to pass any arguments sent to our script on to the ntfsresize.real script.

After this, run gparted, and you should be able to continue. A resize from 80GB to 250GB took about an hour to complete.

References: