Starting a new duplicati backup

The Duplicati backup software is a nice cross platform backup solution. There are times when one wants to start a new backup with the same settings as another backup. Duplicati doesn’t support 2 backups going to the same directory, so one needs to make sure to specify a new destination for the new backup.

  1. Export the backup
    1. Click on the backup
    2. Click Export
    3. Click Export button (accept message about passwords)
    4. Save file to somewhere on your computer
  2. Disable scheduled backups of this job for now
    1. Click on the backup
    2. Click Edit
    3. Click Schedule
    4. Uncheck Automatically run backups
    5. Click Options
    6. Save
  3. Create a new backup from the old configuration
    1. Click on Add backup
    2. Import from file
    3. Select the file on your computer
    4. Import file
    5. Change the name from “backup name” to “{backup name} {today’s date}”
    6. Click on Destination
    7. Change the end of the Path on server to be the current date (this makes sure that we start in a new directory)
    8. Test connection
    9. OK to create
    10. Schedule
    11. Make sure there is a schedule set
    12. Options
    13. Save
  4. Delete the exported backup file from your computer so that the password isn’t left around for someone to find
  5. Start the backup

Once you are certain that the backup is working you should clean up the old backup from the destination to save space. Here are instructions for doing this when the backup destination is Nextcloud.

  1. Click on the old, broken backup
  2. Click Delete
  3. Click Delete backup (leave delete remote files unchecked, we will do that manually)
  4. Delete the remote files manually
    1. Visit the Nextcloud installation.
      1. Look in the backup configuration for the destination.
      2. Pay attention to the Server and port and the SSL checkbox.
      3. Pay attention to the “Path on server” and keep everything before “/remote.php”
      4. Put the following in the address bar of your web browser https://{host}:{port}/{path}. Use “http” if “SSL” is not checked.
    2. Login with the username and password that is configured in the backup.
    3. Check the box next to the directory from the OLD backup.
    4. Click the 3 dots next to Actions
    5. Click delete
    6. Wait for it to finish
    7. Use the gear in the top right to log out

Fixing duplicati unexpected difference in fileset

Duplicati is a nice backup tool that is cross platform. However sometimes it has issues. This post gives one solution for dealing with the error “Unexpected difference in fileset version”.

  1. Open the duplicati web page
  2. Pay attention to the fileset version number in the error message.
  3. Dismiss the error
  4. Click on the troubled backup
  5. Click on commandline
  6. Change the command at the top to delete
  7. Clear out the commandline arguments field
  8. Put “–version=XX” in the commandline arguments field (change XX to version specified in the error message)
  9. Click Run “delete” command now at the bottom right
  10. Wait
  11. Execute a new backup and see if there are errors
  12. Repeat if necessary

Tracking packages installed

To help with restores or migration to another piece of hardware I find it useful to keep track of which packages are installed on my Linux systems. This isn’t very difficult, but it has taken a bit of experimenting to come up with with something that works fairly well and is automated.

Create in /etc/systemd/system/status-email-root@.service. This is a helper service for sending emails on service failures.

[Unit]
Description=status email for %i to Jon

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/systemd-email.sh root %i

/usr/local/sbin/systemd-email.sh looks like this

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script directory"


/usr/sbin/sendmail -t <<ERRMAIL
To: $1
From: systemd <root@$HOSTNAME>
Subject: $2
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$(systemctl status --full "$2")
ERRMAIL

Create in /etc/systemd/system/track-installed-packages.service with the content. Note the “showmanual” argument to “apt-mark”. This makes sure to only output the packages that were manually installed and not include those that were installed as dependencies. This keeps the list short and if dependencies change later keeps from installed unneeded packages later.

[Unit]
Description=track installed packages
OnFailure=status-email-root@%n.service

[Service]
Type=oneshot
ExecStart=/usr/bin/apt-mark showmanual
StandardOutput=file:/home/installed-packages.txt

Now create /etc/systemd/system/track-installed-packages.timer

[Unit]
Description=track installed packages daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Now enable the timer with

systemctl daemon-reload
systemctl enable track-installed-packages.timer
systemctl start track-installed-packages.timer

Now if you backup /home, you’ll always have the most recent list of installed packages.

I have a similar pair of systemd files for snaps that calls “snap list” and outputs to “/home/snap-packages.txt”. Snap doesn’t appear to have an option to specify only manually installed snaps. This may also error out if there are no snaps installed.

Configure Linux Jenkins node

I have been setting up a few Jenkins nodes lately and decided that I should write up the configuration that I’m using to share with others.

Create the node in Jenkins

The first thing to do is to create the node in Jenkins. Start by logging into your Jenkins host, then visit the “Manage Jenkins” link. Once there, visit “Manage Nodes” and then click “New Node” on the left.

Give your node a name. It’s a good idea to avoid spaces and special characters. I use letters, numbers, underscores and hyphens. Select “Permanent Agent” and then “OK”.

Here you need to specify the working directory, labels and the usage. I usually set the usage to only build jobs with a matching label expression. This is useful when setting up nodes per job to make sure that the node doesn’t get used for other random jobs. You may also want to specify an email address to notify when the node goes online and/or offline.

Once you have saved the configuration you will see a page specifying that the agent is offline and how to launch it. The important piece of information here is the secret. This will be a very long string of letters and numbers.

Linux Setup

First create a user in Linux that the node will run as. This user should not have any special privileges.

sudo adduser JENKINS_BUILD_USER

Replace “JENKINS_BUILD_USER” with the username that you are using. By default this user has a locked password so no one can login as this user.

In “/home/JENKINS_BUILD_USER” create the file “start-jenkins-node.sh” to start the node

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script directory"

jenkins_host=JENKINS_HOST
jenkins_node_name=NODE_NAME
jenkins_node_secret=SECRET

cd "${mydir}"
# --no-check-certificate is needed if the certificate store does not recognize the jenkins host certificate
try wget https://${jenkins_host}/jnlpJars/agent.jar -O agent.jar

# -noCertificateCheck is needed if the certificate isn't recognized
nohup java -jar agent.jar -jnlpUrl https://${jenkins_host}/computer/${jenkins_node_name}/slave-agent.jnlp -secret ${jenkins_node_secret} -workDir "${HOME}" > "${HOME}"/jenkins-node.log 2>&1

Replace JENKINS_HOST with the hostname that Jenkins is running on. This script assumes that Jenkins is running at hte root of your server. If that’s not the case you’ll want to append the base path to the end of JENKINS_HOST. Replace NODE_NAME with the name of the node and SECRET with the secret from the node configuration on the Jenkins host.

Mark the file executable.

chmod +x /home/JENKINS_BUILD_USER/start-jenkins-node.sh

Create “/etc/systemd/system/jenkins_node.service”

[Service]
Type=simple
ExecStart=/home/JENKINS_BUILD_USER/start-jenkins-node.sh
WorkingDirectory=/home/JENKINS_BUILD_USER
Restart=always
RestartSec=60
User=JENKINS_BUILD_USER

[Unit]
After=network-online.target
Wants=network-online.target

[Install]
WantedBy=default.target

Replace JENKINS_BUILD_USER with the user that you created. Then you can enable and start the service with

sudo systemctl daemon-reload
sudo systemctl enable jenkins_node
sudo systemctl start jenkins_node

At this point you should see your node online in Jenkins and you are ready to use it for jobs.

My initial experience with Google Inbox

This past week I finally decided to try out Google Inbox. The feature that really drew me to it was the ability to snooze emails. This feature allows you to make an email leave your inbox and come back at some later date and time. This is a really cool feature and a nice way to delay dealing with an email until you need to. In addition to this it is really easy to create filters that add emails to bundles (labels). These bundles can be set to appear in the inbox or not and you have some control over when the bundles appear in the inbox. When a bundle appears in the inbox it shows up as a wide message, once opened you see all of the messages in the bundle. This is a nice way to be able to group messages; you can see your labels in the inbox in a compact fashion. You can also decide which bundles will trigger notifications in the android app.

After using Inbox for about a week, I’ve decided to go back to using GMail. Here are my reasons:

  1. The keyboard shortcuts in the web interface are lacking.
    • No keyboard shortcut to goto a label/bundle
    • No keyboard shortcut to type in the name of a bundle to move to. There is a shortcut ‘.’ to open the move to menu though.
  2. The bar on the left side showing the bundles don’t show how many unread messages are in the bundle
  3. When you choose to have bundles show up in the inbox you can select as the messages arrive, once a day (7:00) and once a week (Monday 7:00). I would really like to be able to at least pick the time for once a day and once a week. It would be nice to be able to pick the day on the once a week.
  4. I like to make sure all messages that I keep are assigned at least one label. The Inbox interface doesn’t allow me to see what labels have been applied to a message. This makes me very concerned that I will loose messages by them being archived and not assigned any labels. GMail’s search interface is great, but I really like to be able to find my messages by label.

If Google fixes these features I will give Inbox a try again, until then I’m sticking with GMail.

IPv6 on Comcast Residential

Comcast has now opened up their IPv6 service to residential customers. If you have a supported modem from Comcast and a device connected to it that understands IPv6 you can connect. You might ask why would I want to setup IPv6? And that is a good question. One reason is to stay up to date with current networking technology. Another reason is that we’re running out of IPv4 addresses and we will eventually need to switch to IPv6. Currently many sites on the Internet are supporting IPv4 and IPv6 to help with the adoption of IPv6. Another reason for IPv6 support is that this setup can give you a subnet of public IPv6 addresses to use in your house. Meaning that you can allow computers on your internal network to be accessible from the outside world. Of course this also means that you could potentially open up your computers to the outside world, so you need to be careful and setup your firewall to keep your internal computers secure unless you want them accessible. This also removes any issues with NAT as IPv6 doesn’t have any NAT support.

For my setup I have a compatible modem from Comcast and a Linux computer as my router. My Linux computer is running Ubuntu. These instructions are specific to my setup, but should be able to be used by others running most any Linux distribution.

The first thing you should do is secure your network from IPv6 so that something doesn’t get in while you’re setting things up. Here is my IPv6 firewall setup, it’s very similar to my IPv4 setup, except the port numbers for DHCP are different. Outbound traffic is allowed and inbound traffic is denied. I’ve also disabled forwarding of traffic, this prevents inbound traffic directly to the internet work. This script needs to be located at “/usr/local/sbin/firewall-ipv6-start” for the radvd script at the end of this post to work properly.

#!/bin/sh 

IPTABLES=/sbin/ip6tables
INET_IFACE="eth0"
LAN_IFACE="eth1"
LO_IFACE="lo"

$IPTABLES -F
$IPTABLES -X

# accept everything by default
$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P FORWARD DROP

${IPTABLES} -A FORWARD \
 -m state --state RELATED,ESTABLISHED \
 -m comment --comment "allow inbound traffic for established and related connections" \
 -j ACCEPT
${IPTABLES} -A FORWARD \
 -i ${LAN_IFACE} -o ${INET_IFACE} \
 -m comment --comment "allow all Internet bound traffic from the internal network" \
 -j ACCEPT
${IPTABLES} -A FORWARD -p ipv6-icmp \
 -m comment --comment "forward any ICMP traffic" \
 -j ACCEPT

${IPTABLES} -A INPUT \
 -m state --state RELATED,ESTABLISHED \
 -m comment --comment "allow inbound traffic for established and related connections" \
 -j ACCEPT

${IPTABLES} -A INPUT \
 -i ${LO_IFACE} \
 -m comment --comment "allow any local-only traffic" \
 -j ACCEPT

${IPTABLES} -A INPUT \
 -p ipv6-icmp \
 -m comment --comment "allow ICMP traffic from anywhere" \
 -j ACCEPT

${IPTABLES} -A INPUT -i ${INET_IFACE} \
 -p udp -m udp --dport 546 \
 -m comment --comment "Accept DHCP traffic" \
 -j ACCEPT

${IPTABLES} -A INPUT -i ${INET_IFACE} \
 -p udp -m udp --dport 547 \
 -m comment --comment "Accept DHCP traffic" \
 -j ACCEPT
 

The remainder of this post is based upon this post on using DHCPv6 with prefix delegation.

The next thing we need to do is get an address from Comcast along with a prefix (subnet) to hand out to the computers on the internal network. Comcast doesn’t appear to be using router advertisements for IPv6, so we’ll need to use DHCP over IPv6. For this I could use the ISC DHCP server that I’m using for IPv4, but it doesn’t support prefix delegation which I need to give the other computers in my house IPv6 addresses. For this I installed wide dhcp client. Ubuntu includes this in the package wide-dhcpv6-client. A side advantage to using a different DHCP client for IPv6 is that you can turn it off to disable IPv6 support without messing with your IPv4 network. Once you install the client edit /etc/wide-dhcpv6/dhcp6c.conf to look like this. You will need to modify the interface used and possibly the sla-len. I found the sla-len by trial and error. You won’t get a prefix if the value is incorrect.

interface eth0 { # external facing interface (WAN)
 send ia-na 1;
 send ia-pd 1;

 request domain-name-servers;
 request domain-name;

 script "/etc/wide-dhcpv6/dhcp6c-script";
};

id-assoc pd 1 {
 prefix-interface eth1 { #internal facing interface (LAN)
 sla-id 0; # subnet. Combined with ia-pd to configure the subnet for this interface.
 ifid 1; # IP address "postfix". if not set it will use EUI-64 address of the 
         # interface. Combined with SLA-ID'd prefix to create full IP address of interface.
 sla-len 0; # prefix bits assigned. Take the prefix size you're assigned
            # (something like /48 or /56) and subtract it from 64. 
            # In my case I was assigned a /64, thus the value is 0
 };
};

id-assoc na 1 {
 # id-assoc for external interface
};

When you start the wide DHCP client and all is happy you will find that your external interface has an address. In my case it starts with 2001:558:6014. See the output of “ip addr show dev eth0” changing the interface as appropriate. Below is the output for my system with the IP addresses masked out.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
 link/ether 00:d0:b7:3f:4d:18 brd ff:ff:ff:ff:ff:ff
 inet XXX.XXX.XXX.XXX/22 brd 255.255.255.255 scope global eth0
 valid_lft forever preferred_lft forever
 inet6 2001:558:6014:XXXX:XXXX:XXXX:XXXX:XXXX/128 scope global 
 valid_lft forever preferred_lft forever
 inet6 fe80::XXXX:XXXX:XXXX:XXXX/64 scope link 
 valid_lft forever preferred_lft forever

Once this is setup you can ping IPv6 addresses from your router. You can test this with “ping6 google.com”.

Now to allow your local network talk to the Internet via IPv6 you’ll need to allow forwarding and then assign them IPv6 addresses.

First we’ll tell the kernel to allow forwarding by modifying adding the file 70-ipv6-routing.conf to /etc/sysctl.d. Note that net.ipv6.conf.all.accept_ra is set to 2. Any other value will not work due to how the router advertisements are handled.

# only set this on the external interface, otherwise we don't get a
# default route for IPv6
net.ipv6.conf.EXT_IFACE.accept_ra=2
net.ipv6.conf.EXT_IFACE.forwarding=0

net.ipv6.conf.INT_IFACE.accept_ra=1
net.ipv6.conf.INT_IFACE.forwarding=1

net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.autoconf=1

 
Once you change these values you will need to reboot or use the sysctl utility to set them immediately.

Now to hand out IPv6 addresses to the rest of the network. This will be done by setting up radvd. The package ‘radvd’ on Unbuntu contains this daemon. Once installed you can setup /etc/radvd.conf for the prefix that Comcast gave you. However when your IP address changes you’ll need to update the file. So instead I have created a script that can be run from wide dhcp client. Put the following in “/usr/local/sbin/update-ipv6-setup.sh” and add a call to this script from the end of /etc/wide-dhcpv6/dhcp6c-script. You’ll need to change the interface in this script to be your internal interface.

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script dir
ectory"

prefix=$(ip -6 addr show dev eth1 scope global \
 | grep inet6 \
 | awk '{print $2}') \
 || fatal "Unable to get prefix"

cat > /etc/radvd.conf.new <<EOF

interface eth1
{
 AdvSendAdvert on;
 AdvIntervalOpt on;
 MinRtrAdvInterval 60;
 MaxRtrAdvInterval 300;
 AdvLinkMTU 1280;
 AdvOtherConfigFlag on;
 AdvHomeAgentFlag off;
 
 prefix ${prefix}
 {
 AdvOnLink on;
 AdvAutonomous on;
 AdvRouterAddr on;
 };
};

EOF

diff /etc/radvd.conf.new /etc/radvd.conf > /dev/null
if [ $? -ne 0]; then 
 # only move if there are differences
  try mv -f /etc/radvd.conf.new /etc/radvd.conf
  try service radvd restart
fi
try /usr/local/sbin/firewall-ipv6-start

Now you have IPv6 setup on your router and your home network. I found that Linux, Windows and Mac automatically recognize the IPv6 router advertisements and grab addresses and setup routes appropriately.

Resources:

 

Skitch for easy sharing of drawings

I use a Mac at work and was recently introduced to Skitch in a training class. This application is really handy if you need to share images or screen shots with others. Skitch allows you to easily take a screen shot of what you’re doing and then annotate it to state what is important in the image. You can also easily do simple drawings and then share them. Once you have an image that you like you can just drag it to your email program, or if you sign up for an account with Skitch you can have the image uploaded there and accessible from their website.

Finally able to use my iPod Touch without Windows or Mac

I’ve finally made it. I can now use my iPod Touch just with Linux. Apple adding over the air updates really helped, but I still couldn’t sync my music. I also found that pulling large videos off via Dropbox didn’t work as the app would time out. I had been trying to get gtkpod to work, but kept running into a problem with the database checksum on the music database. Recently I found Phone Drive and this has solved my problem. This app lets me copy images and videos out of my camera roll into the app and then the app will bring up a web server and an ftp server on the device. I can then browse the files from my web browser and download the files to my desktop. I can also put files up this way. Phone Drive also has a built in music player that will play any directory as a playlist and supports users adding their own playlists. So I wrote a little script (below) that will generate a playlist for all music and one for each artist on my computer and then I can upload those and all of my music via ftp. I did have a little trouble with some ftp programs that want to open multiple connections. In the end I used ncftp as it doesn’t try to open multiple connections and it has the ability to upload directories recursively. I did find one oddity that I needed to turn off the auto conversion of ascii files otherwise playlist files got their line endings changed and Phone Drive wouldn’t recognize them.

So my current list of apps that I regularly use is this:

  • Calendar syncing with Google, either via Active Sync or via caldav (this allows the colors to sync)
  • Contacts syncing with Google as an Exchange account
  • GMail app for mail from Google as I prefer the way Google does conversations
  • Appigo Todo syncing with Toodledo
  • PlainText for notes
  • Dropbox for keeping family pictures and moving images between my device and my computer
  • MiniKeePass on my device and KeePassX syncing through Dropbox
  • Podcaster for audio and video podcasts
  • ReadItLater for offline access to web pages
#!/bin/sh
debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script directory"
cd "${mydir}/mp3"
# generate artist playlists
find . -maxdepth 1 -type d -print0 | while read -d $'\0' dir
 do
 short_dir="${dir#./}"
 if [ "${short_dir}" != "." ]; then
 playlist="${mydir}/itouch-playlists/aaa_${short_dir}.m3u"
 log "Processing ${short_dir}"
 printf "" > "${playlist}"
 find "${short_dir}" -type f -name '*.mp3' -printf "/music/%p\n" >> "${playlist}"
 mv "${playlist}" "${dir}"
 fi
done
# generate all music playlist
log "Generating all music playlist"
find . -type f -name '*.mp3' \
 -fprintf "${mydir}/mp3/aaa_all_music-new.m3u" "/music/%P\n"

 

Appigo Todo now supports Dropbox

On my iPod Touch I’m using Appigo’s Todo application. I had been syncing the data to Toodledo so that I had a backup in case something happened to my itouch. Now Appigo’s Todo app supports Dropbox. The advantage here is that the place that I’m syncing to, dropbox, supports all of the features of the Todo app. This is important for recovery. Toodledo doesn’t support hierarchical todo items. However when syncing with dropbox it’s in Appigo’s own format so all features are supported. And it turns out the native format is an sqlite database, so I can pull all of the items out of it in case Appigo goes away!

2/20/2012 update. I’ve gone back to Toodledo.com for syncing as the Dropbox sync for the todo app was just too slow. The Toodledo sync is faster, but it’s not a smart as it keeps trying to sync when I’m offline and then popping up an error dialog. I’ve contacted Appigo about this, but haven’t seen a fix yet.

Passwords for iTouch and iPhone

Until recently I’ve been using MyKeePass to store passwords on my iTouch. This allows me to use KeePassX on my desktop. What I’d really like though is Dropbox integration. Well recently on Hak5 they said that there was a free app for IOS that understood KeePass password databases and MyKeePass isn’t free. So I went looking and found MiniKeePass and discovered that it’s open-source and free. On top of that it integrates nicely with Dropbox! You still need to manually sync the database, but it’s really easy if you keep it in Dropbox. When you make changes on your desktop, you just open the Dropbox application on your iTouch and open the database and it imports right into MiniKeePass, then delete the old database from MiniKeePass. When you make changes on your iTouch, you can export it to Dropbox and overwrite the one there so that your desktop sees the changes.