iproutes

routes

With regard to static routing, consider the above diagram. We have three separate networks: 192.168.1.0, 192.168.2.0, and 192.168.3.0. At first, network hosts (routers, computers, etc.) can only communicate with other hosts that are on their own network. For instance, the computer named James has a single interface on network 192.168.1.0, so that’s the only network that it can ‘see’. Initially, it will only be able to communicate with Router A.

Router A has network interfaces on the 192.168.1.0 and 192.168.2.0 networks, so those are the two networks that it can ‘see’. These are the only networks Router A ‘knows’ about, so it can only communicate with hosts on the 192.168.1.0 and 192.168.2.0 networks. So Router A doesn’t even ‘know’ that the 192.168.3.0 network exists.

Similarly, Router B can ‘see’ networks 192.168.2.0 and 192.168.3.0. When you enter a route into the table, you’re telling a host that there’s a new network it can get to, and you’re giving it the address of a gateway that it can use to get to the new network.

So to be able to contact Jesus (or any other host on the 192.168.3.0 network) from Router A, you’d enter the command:

ip route 192.168.3.0 255.255.255.0 192.168.2.2
             ^             ^             ^
           network        mask         gateway

This works because Router B can ‘see’ both Router A and Jesus. Thanks to this routing table entry when Router A wants to reach the 192.168.3.0 network, it knows it can get there via Router B at 192.168.2.2, so it sends the packet to Router B. Router B can see the 192.168.3.0 network directly, so it forwards the packet along to Jesus at 192.168.3.11.

So, now we know how to direct router A to the 192.168.3.0 network. But what if we want James to also be able to reach the 192.168.3.0 network? Well, Router A already knows how to get there, and James can already ‘see’ Router A, since they’re both on network 192.168.1.0. So we can just tell James to use Router A as its gateway to the 192.168.3.0 network. If James were a router instead of a computer, we’d use the command:

ip route 192.168.3.0 255.255.255.0 192.168.1.1
             ^             ^             ^
           network        mask         gateway

James would then be able to contact Jesus (or any host on the 192.168.3.0) network by forwarding the packet to 192.168.1.1 (Router A), which would then forward the packet to 192.168.2.2 (Router B) which would then forward the packet to its destination (Jesus in this case) via its directly connected interface.

Now, for Jesus to be able to respond to James, Jesus would need to have Router B set up as its gateway to the 192.168.1.0 network, and Router B would have to have Router A set up as its gateway to the 192.168.1.0 network. Then, any host on the 192.168.1.0 network would have a path to the 192.168.3.0 network and vice versa.

(reference: https://serverfault.com/questions/171551/help-me-understand-the-ip-route-command-for-cisco-routers )

using variables on the command line

Following on from my last example of copying a SSH public key to a remote computer, this is something I need to do when setting up a new computer. Setting up private/public keys for SSH just make logging in that little bit smoother.

When you need to rerun the command, you need to load it up, edit it and resubmit it. Unfortunately (although it’s probably possible) I don’t know an easy way to bring up a previous command and edit it in-line so that I can send it again without actually sending the command again before doing so.

Instead, Load a variable into the command line and change it next time.

-- 11:03:01 -- MBP:~ madivad$ ssh minixbmc
Password:
Last login: Mon Apr 25 18:23:18 2016
minixbmc:~ madivad$ exit
logout
Connection to minixbmc closed.
-- 11:03:17 -- MBP:~ madivad$ remote=minixbmc
-- 11:03:26 -- MBP:~ madivad$  history | grep remote
  439  remote=he1000
  440  cat ~/.ssh/id_rsa.pub | ssh madivad@$remote "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
  502  remote=minixbmc
-- 11:03:34 -- MBP:~ madivad$ !440
cat ~/.ssh/id_rsa.pub | ssh madivad@$remote "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"
Password:
-- 11:03:40 -- MBP:~ madivad$ ssh minixbmc
Last login: Tue Apr 26 11:03:12 2016 from mbp.fritz.box
minixbmc:~ madivad$

For example, in the above session, for simple commands, I would being the history file up, reissue line 440, then edit, then issue it again. In this situation, it would have the effect of loading the key again, and that’s not what I want to do.

  • Breaking it down, I logged into the remote machine and realised a password was needed,
  • I logged out,
  • I set the “remote” variable,
  • looked for the relevant history command (I knew it had the word “remote” on it),
  • I re-issued that line, and
  • then tested the login.
  • No password was needed, the command was a success.

This could be done with other things as well where you’re always changing one element on the line (or multiple elements, and use multiple variables).

For a more simple and silly example, let’s create a quick update and install script for ubuntu:

upstall=’htop multiwatch’
sudo apt update && sudo apt install $upstall

Instead of typing the whole line next time, I can just type the new apps to install in the “upstall” variable and reissue the command (in this case, using arrow up a couple of times, or grabbing the index from the history file).

$ sudo apt update && sudo apt install $upstall
[sudo] password for madivad:
Hit:1 http://au.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://au.archive.ubuntu.com/ubuntu xenial-updates InRelease [92.2 kB]
Hit:3 http://au.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [92.2 kB]
Fetched 184 kB in 1s (101 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
byobu is already the newest version (5.106-0ubuntu1).
htop is already the newest version (2.0.1-1).
multiwatch is already the newest version (1.0.0-rc1+really1.0.0-1).
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

If I then later want do another update and install something else, I can re-set the “upstall” variable and arrow up or grab it out of history.

11:53:44 madivad@he1000:~$ upstall=jq
12:03:44 madivad@he1000:~$ sudo apt update && sudo apt install $upstall
Hit:1 http://au.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://au.archive.ubuntu.com/ubuntu xenial-updates InRelease [92.2 kB]
Hit:3 http://au.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [92.2 kB]
Fetched 184 kB in 2s (91.0 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
jq is already the newest version (1.5+dfsg-1).
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.
I'm a simple man, I like simplicity. And although there are probably better ways to do this, for the time being, this is how I'm getting the job done. It works well for me, but I'm open to any suggestions and/or improvements.

As I said, not the best example, but hopefully you get the idea.

SSH key pairs and getting it onto the server

cat ~/.ssh/id_rsa.pub | ssh someuser@someserver "mkdir -p ~/.ssh && cat >>  ~/.ssh/authorized_keys"

usual assumptions:

  • change `someuser` and `someserver` as appropriate
  • usual public key file and location on the client
  • usual authorisation file location on the server

What files is my program trying to access?

In this example I’m using hashdeep. I’m redirecting the output of two hash sets to two different files. I am doing that with the following command:

hashdeep -rj0 /path-to-drive-1 > hashes.drive1

and

hashdeep -rj0 /path-to-drive-2 > hashes.drive2

I have those running in their own terminal windows. I then optionally have another two windows open running a tail on them so I can monitor the files:

tail -f hashes.drive1

The hard drives are located in an external multi-bay enclosure and all hard drive LEDs are flashing away like mad. A good sign. But every now and then I’ll run an ‘ls’ to see where the files are at (checking file size) or alternatively (and usually better but more resource intensive) a line count of the hash files. Given I know how many files there should be, the line count gives a fair indication of the progress of the whole process.

wc -l hashes.drive*

In today’s example I was simply doing a file size comparison of the two hashes vs a known hashset of one of the drives that was a month old. The sizes should be relatively similar. I was getting results similar to:

madivad@server:~$ ls -al hashes*
-rw-rw-r-- 1 madivad madivad 330483319 Feb 11 09:26 hash.drive1.1602
-rw-rw-r-- 1 madivad madivad 341570757 Mar 23 12:09 hash.drive1.1603
-rw-rw-r-- 1 madivad madivad 243344728 Mar 23 11:18 hash.drive2.1603

The fact that drive1.1603 is larger is of no consequence, there are just more files to consider.

After running the above check for sometime, I realised that one of the files (in this case drive1.1603) had stalled for several hours. I’m not exactly sure when it seemed to stop growing, but doing a tail of the file confirmed it was stopped. The last output was an inconsequential .DS_Store file roughly 6K in size. After physically monitoring it for some time I began to get concerned about this. I could see the all 4 RAID drives getting activity, but nothing was being recorded. The 5th drive, the backup, was hashing away without a problem and the log file was growing as expected.

After some quick research I came across this stack exchange Q&A ( How do I know which file a program is trying to access? )

The first answer provided a solution that worked best with my scenario:

lsof -c hashdeep

I’d never seen this output before but very quickly I could see the important pieces of information it had dumped out. Namely:

madivad@server:~$ lsof -c hashdeep
COMMAND  PID  USER    FD TYPE DEVICE     SIZE/OFF  NODE       NAME
hashdeep 2539 madivad 1w REG  252,0     243344728  5535319    /home/madivad/hash.drive1.1603
hashdeep 2539 madivad 3r REG  259,0  499418030080  113639426  /path1/largeFiles/a-very-big-image-of-500GB.img
hashdeep 2552 madivad 1w REG  252,0     341611062  5535320    /home/madivad/hash.drive2.1603
hashdeep 2552 madivad 3r REG  8,33     3152347139  126025746  /path2/misc/random.file

The ‘w’ of FD with ‘1w’ signifies the file is being written and that the file being written was hash.drive1.1603

The ‘r’ of FD with ‘3r’ signifies the file is being read for hashing purposes, and that file is a very large file that I know is around 500GB. Running the command again shows me the second file being read in had changed, yet the first had stayed the same.

Given the file is very large and will take considerable time to hash and that the hard drive LEDs are flashing, I realised all was good in the world and I could move on with the days activities.

UPDATE: after reading the man page on lsof I found a better way to monitor the continual progress of it was to run it with the -r “repeat” switch which defaults to 15 seconds, which could be updated more or less frequently by adding a numerical component:

lsof -r 5 -c hashdeep

ZFS on Ubuntu 14.04

I’ve completed a fresh install of Ubuntu on an old box that use to contain solaris. The system disk died and upon installing a new (old) drive and then installing ubuntu, I found that two of the disks still in the computer were part of a zfs raid. I’m not sure how many disks were in the raid, but I’m curious as to what was on this file system that has been shutdown in inaccessible for more than 7-8 years. (I later confirmed that most files on the system are from 2008 and earlier).

There were only 4 available sata connectors to the board. Two were used for the drives in there, it didn’t take long to find the matching drives to plug in.

Installing ZFS

I began with installing zfs by following the instructions here:
Install ZFS on Ubuntu—Server as Code

Installing SSH Server

I also had to install an SSH Server because this is on a box located remotely. Follow any generic install of Open SSH Server. The one I used was SSH/OpenSSH/Configuring.

Because my server is behind a firewall and not publicly accessible, I haven’t worried too much about logon via SSH Keys. I have done it before, but this box is only going to be a temporary install, but I do recommend you do that. A couple of other related tutorials to passwordless ssh key access to servers:
https://help.ubuntu.com/community/SSH/OpenSSH/Keys << a good resource, includes troubleshooting
http://www.mccarroll.net/blog/rpi_cluster2/index.html
https://www.howtoforge.com/tutorial/ssh-and-scp-with-public-key-authentication/
https://www.raspberrypi.org/documentation/remote-access/ssh/passwordless.md

Installing Samba

I also installed samba, following these instructions How to Create a Network Share Via Samba Via CLI (Command-line interface/Linux Terminal) – Uncomplicated, Simple and Brief Way!

With SSH, Samba and the ZFS modules installed, configured and running… let’s try and rebuild this raid :)

Let’s have a look!

disks:

lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 111.8G  0 disk 
├─sda1                         8:1    0   243M  0 part /boot
├─sda2                         8:2    0     1K  0 part 
└─sda5                         8:5    0 111.6G  0 part 
  ├─ubuntu--vg-root (dm-0)   252:0    0 110.6G  0 lvm  /
  └─ubuntu--vg-swap_1 (dm-1) 252:1    0  1016M  0 lvm  [SWAP]
sdb                            8:16   0 698.7G  0 disk 
├─sdb1                         8:17   0 698.6G  0 part 
└─sdb9                         8:25   0     8M  0 part 
sdc                            8:32   0 698.7G  0 disk 
├─sdc1                         8:33   0 698.6G  0 part 
└─sdc9                         8:41   0     8M  0 part 
sdd                            8:48   0 698.7G  0 disk 
├─sdd1                         8:49   0 698.6G  0 part 
└─sdd9                         8:57   0     8M  0 part 
sde                            8:64   0 698.7G  0 disk 
├─sde1                         8:65   0 698.6G  0 part 
└─sde9                         8:73   0     8M  0 part 
sr0                           11:0    1  1024M  0 rom

and for pools specifically:

$ sudo zpool import
   pool: solaraid
     id: 10786192747791980338
  state: ONLINE
 status: The pool is formatted using a legacy on-disk version.
 action: The pool can be imported using its name or numeric identifier, though
	some features will not be available without an explicit 'zpool upgrade'.
 config:

	solaraid                                       ONLINE
	  raidz1-0                                     ONLINE
	    ata-WDC_WD7500AACS-00C7B0_WD-WCASN         ONLINE
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT         ONLINE
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT         ONLINE
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT         ONLINE

Ok, this PC only had 4x SATA drives and it appears I’ve found the correct drives. Things are looking good from the start.

Let’s do it!

:~$ sudo zpool import solaraid
:~$ sudo zpool status
  pool: solaraid
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
	continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Wed Nov 11 21:37:59 2015
    11.5M scanned out of 1.63T at 1.15M/s, 412h58m to go
    2.60M resilvered, 0.00% done
config:

	NAME                                           STATE     READ WRITE CKSUM
	solaraid                                ONLINE       0     0     0
	  raidz1-0                              ONLINE       0     0     0
	    ata-WDC_WD7500AACS-00C7B0_WD-WCASN  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     2  (resilvering)

errors: No known data errors

YOUCH! 412HOURS… That’s 17 days! I gave it a couple of seconds to stabilise and ran it again, and came up with an error:

:~$ sudo zpool status
  pool: solaraid
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 2.60M in 0h0m with 0 errors on Wed Nov 11 21:38:24 2015
config:

	NAME                                           STATE     READ WRITE CKSUM
	solaraid                                ONLINE       0     0     0
	  raidz1-0                              ONLINE       0     0     0
	    ata-WDC_WD7500AACS-00C7B0_WD-WCASN  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     2

errors: No known data errors

Reboot and Ubuntu bootup fail

(long winded fluff about nothing, you can scroll down to “Resilvering” if you’d like to skip this)

I do recall I had issues with this raid, I could certainly go the route of upgrading it first and trying it again, but once I did a

tree -L 2 /solaraid/

and seeing things I had long thought were gone, I’m going to back this up first :)

The only problem is, the raid is installed in a box with only a 10/100 network card :(

I’ll let it run overnight taking off only what I need, and see how we go. This has been a good find

At this point I was operating in the house and the server is located off-site. I had several ssh/terminal windows open to the box and as I was working away I kept getting the message to reboot the system. I issued the relevant reboot command and set off a ping to tell me when it came back online… It didn’t come back online.

I went to the server and found it was still booting. This was after more than half an hour and eventually it gave up and crashed and restarted again.

For the next couple of hours I could not get the drive to boot and I was blaming the old boot drive, but after eventually getting into the “Try Ubuntu” mode from the DVD I found that one of the drives were not being reported in the system. Another was coming up as totally unknown and two were seen as part of a set. It took several hours to get to the bottom of it. Eventually thru the BIOS I could see one of the drives weren’t being detected.

A couple of sata cable changes and swapping power cables around and I was back in business.

Resilvering

:~$ sudo zpool status
  pool: solaraid
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    1.73G scanned out of 1.63T at 32.3M/s, 14h40m to go
    12.5K repaired, 0.10% done
config:

	NAME                                           STATE     READ WRITE CKSUM
	solaraid                                ONLINE       0     0     0
	  raidz1-0                              ONLINE       0     0     0
	    ata-WDC_WD7500AACS-00C7B0_WD-WCASN  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0    13  (repairing)
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0

errors: No known data errors

Over the next few minutes I kept polling the status and it was picking up speed.

  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    24.3G scanned out of 1.63T at 84.5M/s, 5h31m to go
    12.5K repaired, 1.46% done
  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    32.1G scanned out of 1.63T at 97.0M/s, 4h47m to go
    12.5K repaired, 1.93% done
  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    56.9G scanned out of 1.63T at 102M/s, 4h29m to go
    12.5K repaired, 3.42% done
  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    184G scanned out of 1.63T at 137M/s, 3h4m to go
    184K repaired, 11.01% done

It’s starting to slow down again, and we’re seeing more errors!

  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    195G scanned out of 1.63T at 93.6M/s, 4h28m to go
    354K repaired, 11.68% done

The next day and something had gone wrong. I’m still unsure what happened, but the whole `solaraid` drive became unresponsive.. Where it had got to the evening(/morning before) at 195GB in the resilvering is where it was when I checked it later today. And the drive was otherwise not responding. I remotely tried to reboot and again it hanged.

At this present time, I’m still putting it down to hardware, but it’s really unknown what’s at the root of the problem.

It’s been back up and running for a while now and it’s present status is:

  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    1.41T scanned out of 1.63T at 160M/s, 0h23m to go
    1.23M repaired, 86.89% done
config:

	NAME                                    STATE     READ WRITE CKSUM
	solaraid                                ONLINE       0     0     0
	  raidz1-0                              ONLINE       0     0     0
	    ata-WDC_WD7500AACS-00C7B0_WD-WCASN  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0   251  (repairing)
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     6  (repairing)

errors: No known data errors

When I captured the above, I hadn’t realised the process was almost finished until I pasted and read over it.

  scan: scrub in progress since Thu Nov 12 03:12:54 2015
    1.62T scanned out of 1.63T at 158M/s, 0h0m to go
    1.29M repaired, 99.46% done

As I write this the process has been running for exactly 24 hours. We have 0.5% left.

The final capture:

:/solaraid$ sudo zpool status 
  pool: solaraid
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: scrub repaired 1.29M in 12h0m with 0 errors on Thu Nov 12 15:13:05 2015
config:

	NAME                                    STATE     READ WRITE CKSUM
	solaraid                                ONLINE       0     0     0
	  raidz1-0                              ONLINE       0     0     0
	    ata-WDC_WD7500AACS-00C7B0_WD-WCASN  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0     0
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0   291
	    ata-WDC_WD7500AAKS-00RBA0_WD-WCAPT  ONLINE       0     0    12

errors: No known data errors

The resilvering has checked every checksum in 1.6TB of data and repaired 1.29MB. The process took exactly 12 hours (with a reboot thrown in there JUST to push the boundaries a little bit).

Next we’re to get any data off that we want… Let’s grab some directory information
To be continued…