Ramblings of an Eggplant

X2Go compositing issues

2023-02-13T07:04:00-06:00

I use X2Go to login to some of my systems when I need to access a graphical desktop. To keep the load on the system low I use XFCE as the desktop environment. Recently I found that when I logged into one of my systems running Ubuntu 22.04 that applications didn't appear to want to launch. When checking top in a terminal on the remote system I found x2goagent at 100% CPU. After a couple of attempts I came across this post on the a Debian users mailing list that solved my problem.

Setting "use_compositing" to "false" in ~/.config/xfce4/xfconf/xfce-perchannel-xml/xfwm4.xml solved the issue.

GitHub pull requests and issues

2023-02-10T18:34:00-06:00

I have resisted using pull requests on GitHub for my own repositories because it seemed like I need to have have both an Issue and a Pull Request for each branch. I create Issues for fixes and features that I want to implement. During development I create a branch for the Issue and then once it's finished and tested I manually merge with a merge commit and use the message of the merge commit to close the Issue.

Recently I came across an article about converting a GitHub Issue to a Pull Request. This solved my issue of having both an Issue and a Pull Request for each feature. I can create my Issues like normal and create a branch. Then when I push the branch I execute the following:

hub pull-request -i <issue number> -b main

This converts the specified Issue number into a Pull Request and associates it with the current branch. I need the "-b main" because I have my push.default value set to "nothing" and it appears that "hub" can't find the main branch with that setup.

I've found a couple of benefits of using this pattern. One is that I can merge pull requests using the GitHub mobile app on my phone. The other is that now it's really easy to review the changes made for a feature and keep the discussion about the issue in the same place.

Using password protected zip files in Mac

2022-02-10T20:51:00-06:00

In an earlier post I explained how to use password protected zip files in Windows. This post is the Mac version. 7-Zip is still the file format, however the application is Keka instead of 7-Zip. This is because the 7-Zip application for Mac is command line only and therefore not quite as user-friendly.

Install Keka

Download Keka from https://www.keka.io or the Mac App store.

If you downloaded the file from, do the standard Mac thing and open the dmg file and drag the application to the Applications folder.

Creating a password protected file

Start by putting the files that you want to send in a folder by themselves, even if it’s only a single file. This makes it easy to keep track of what is being sent. It also helps the recipient as they will get the same folder name when they uncompress the file.

Open Keka. Specify 7Z and enter a password

Drag and drop the folder onto the window

Pick filename

Click Compress

Attach this file to an email and send it with a simple message. DO NOT PUT THE PASSWORD IN THE EMAIL. Contact the recipient through another means, phone, text, encrypted chat (WhatsApp) to send the password.

Opening a password protected file

The recipient saves the file to their computer. You will need to pay attention to where the file is saved. It might be in Documents, it might be in Downloads or some other location depending on your email client.

Open Finder and navigate to where the file is saved. Right click on 7z file -> Open With Keka. Don't double click on the file to open it. This will use the built-in Mac utility that doesn't understand the encryption on the file. You will be prompted for a password, but it'll never work.

Pick a location. Click Extract. You will then be prompted to enter the password. When it finishes you will have a folder with the files that were sent.

Using password protected zip files in windows

2020-11-17T21:35:00-06:00

Creating zip files with a password is a fairly easy way to add some security to information that you are sending via email. It's not really good encryption, but it will keep many people out.

I suggest using 7-Zip. It's an open-source and free application that runs on Windows, Linux and Mac. 7-zip has some better password encryption options than the zip programs built into some operating systems. This does mean that both the sender and recipient need to install 7-Zip, but I think it's worth it.

If you are on a Mac, see my post for that platform.

Install 7-Zip

Do this on both the sending computer and the recipient computer.

Download 7-Zip for your computer from their download page.

The first table on the page is the most recent stable version. Pick the 64-bit exe. In this screen shot it is the second row. This should work for most anyone these days. Pay attention to where the file is saved, it is probably in Downloads.

Open File Explorer and navigate to the downloaded file. Double click on the file to start the install. The file will be named 7z1900-64.exe, where the 1900 may be different. There will be a black and white icon next to it.

During the install, the defaults are all fine. You may need to enter an administrator password for the install to finish.

Creating a password protected file

Start by putting the files that you want to send in a folder by themselves, even if it's only a single file. This makes it easy to keep track of what is being sent. It also helps the recipient as they will get the same folder name when they uncompress the file.

Open Windows Explorer and navigate to the folder that you want to send. Now right click on the folder that you put all of your files in. Then click on 7-Zip and then Add to archive... If the 7-Zip option isn't in the right-click menu, it is likely the case that you installed the 32-bit version rather than the 64-bit version. Uninstall 7-Zip and then go back to the download page and make sure you get the 64-bit version.

In the archive window there is a section in the lower right to enter a password. You need to enter it twice to be sure it was typed correctly. Leave everything else at the defaults. This will use AES-256 encryption, lower right corner, to encrypt the file. The format will be 7-Zip, so the recipient needs 7-Zip to open it. Then click OK. This will create a file with the name that you specified with a 7z extension next to the folder. If you have extensions turned off you may just see the name of the file. In the image below the file to send is the one with the paper icon named "stuff to send.7z". This was created from the yellow folder "stuff to send".

Opening a password protected file

Open File Explorer and navigate to the downloaded file. Then right click on the file and select 7-Zip, then Extract Here... You will then be prompted for the password. When it finishes you will have a folder with the files that were sent.

Starting a new duplicati backup

2020-08-02T17:55:00-05:00

The Duplicati backup software is a nice cross platform backup solution. There are times when one wants to start a new backup with the same settings as another backup. Duplicati doesn't support 2 backups going to the same directory, so one needs to make sure to specify a new destination for the new backup.

Export the backup
1. Click on the backup
2. Click Export
3. Click Export button (accept message about passwords)
4. Save file to somewhere on your computer
Disable scheduled backups of this job for now
1. Click on the backup
2. Click Edit
3. Click Schedule
4. Uncheck Automatically run backups
5. Click Options
6. Save
Create a new backup from the old configuration
1. Click on Add backup
2. Import from file
3. Select the file on your computer
4. Import file
5. Change the name from "backup name" to "{backup name} {today's date}"
6. Click on Destination
7. Change the end of the Path on server to be the current date (this makes sure that we start in a new directory)
8. Test connection
9. OK to create
10. Schedule
11. Make sure there is a schedule set
12. Options
13. Save
Delete the exported backup file from your computer so that the password isn't left around for someone to find
Start the backup

Cleanup of old backups

Once you are certain that the backup is working you should clean up the old backup from the destination to save space. Here are instructions for doing this when the backup destination is Nextcloud.

Delete the backup from the server

Click on the backup you want to delete
Click Edit
Click on 2 Destination
The Storage type will be WebDAV - if not, this won't work
Pay attention to the Server and port and the SSL checkbox.
In "Path on server" everything before "/remote.php" is the path
Put the following in the address bar of your web browser https://{host}:{port}/{path}.
- Use "http" instead of "https" if "SSL" is not checked.
Login with the username and password that is configured in the backup.
Check the box next to the directory from the OLD backup.
Click the 3 dots next to Actions
Click delete
Wait for it to finish
Use the gear in the top right to log out

Alternatively I have also found that one can click the checkbox in Duplicati to delete the files from the server when deleting the configuration (see below) and that will remove the files from the server in most cases.

Delete the backup configuration

Click on the backup you want to delete
Click Delete
Click Delete backup (leave delete remote files unchecked, this was done above)

Fixing duplicati unexpected difference in fileset

2020-08-02T17:48:00-05:00

Duplicati is a nice backup tool that is cross platform. However sometimes it has issues. This post gives one solution for dealing with the error "Unexpected difference in fileset version".

Open the duplicati web page
Pay attention to the fileset version number in the error message.
Dismiss the error
Click on the troubled backup
Click on commandline
Change the command at the top to delete
Clear out the commandline arguments field
Put "--version=XX" in the commandline arguments field (change XX to version specified in the error message)
Click Run "delete" command now at the bottom right
Wait
Execute a new backup and see if there are errors
Repeat if necessary

Tracking packages installed

2020-07-19T11:35:00-05:00

To help with restores or migration to another piece of hardware I find it useful to keep track of which packages are installed on my Linux systems. This isn't very difficult, but it has taken a bit of experimenting to come up with with something that works fairly well and is automated.

Create in /etc/systemd/system/status-email-root@.service. This is a helper service for sending emails on service failures.

[Unit]
Description=status email for %i to Jon

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/systemd-email.sh root %i

/usr/local/sbin/systemd-email.sh looks like this

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script directory"


/usr/sbin/sendmail -t <<ERRMAIL
To: $1
From: systemd <root@$HOSTNAME>
Subject: $2
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8

$(systemctl status --full "$2")
ERRMAIL

Create in /etc/systemd/system/track-installed-packages.service with the content. Note the "showmanual" argument to "apt-mark". This makes sure to only output the packages that were manually installed and not include those that were installed as dependencies. This keeps the list short and if dependencies change later keeps from installed unneeded packages later.

[Unit]
Description=track installed packages
OnFailure=status-email-root@%n.service

[Service]
Type=oneshot
ExecStart=/usr/bin/apt-mark showmanual
StandardOutput=file:/home/installed-packages.txt

Now create /etc/systemd/system/track-installed-packages.timer

[Unit]
Description=track installed packages daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Now enable the timer with

systemctl daemon-reload
systemctl enable track-installed-packages.timer
systemctl start track-installed-packages.timer

Now if you backup /home, you'll always have the most recent list of installed packages.

I have a similar pair of systemd files for snaps that calls "snap list" and outputs to "/home/snap-packages.txt". Snap doesn't appear to have an option to specify only manually installed snaps. This may also error out if there are no snaps installed.

Configure Linux Jenkins node

2020-07-11T18:28:00-05:00

I have been setting up a few Jenkins nodes lately and decided that I should write up the configuration that I'm using to share with others.

Create the node in Jenkins

The first thing to do is to create the node in Jenkins. Start by logging into your Jenkins host, then visit the "Manage Jenkins" link. Once there, visit "Manage Nodes" and then click "New Node" on the left.

Give your node a name. It's a good idea to avoid spaces and special characters. I use letters, numbers, underscores and hyphens. Select "Permanent Agent" and then "OK".

Here you need to specify the working directory, labels and the usage. I usually set the usage to only build jobs with a matching label expression. This is useful when setting up nodes per job to make sure that the node doesn't get used for other random jobs. You may also want to specify an email address to notify when the node goes online and/or offline.

Once you have saved the configuration you will see a page specifying that the agent is offline and how to launch it. The important piece of information here is the secret. This will be a very long string of letters and numbers.

Linux Setup

First create a user in Linux that the node will run as. This user should not have any special privileges.

sudo adduser JENKINS_BUILD_USER

Replace "JENKINS_BUILD_USER" with the username that you are using. By default this user has a locked password so no one can login as this user.

In "/home/JENKINS_BUILD_USER" create the file "start-jenkins-node.sh" to start the node

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script directory"

jenkins_host=JENKINS_HOST
jenkins_node_name=NODE_NAME
jenkins_node_secret=SECRET

cd "${mydir}"
# --no-check-certificate is needed if the certificate store does not recognize the jenkins host certificate
try wget https://${jenkins_host}/jnlpJars/agent.jar -O agent.jar

# -noCertificateCheck is needed if the certificate isn't recognized
nohup java -jar agent.jar -jnlpUrl https://${jenkins_host}/computer/${jenkins_node_name}/slave-agent.jnlp -secret ${jenkins_node_secret} -workDir "${HOME}" > "${HOME}"/jenkins-node.log 2>&1

Replace JENKINS_HOST with the hostname that Jenkins is running on. This script assumes that Jenkins is running at hte root of your server. If that's not the case you'll want to append the base path to the end of JENKINS_HOST. Replace NODE_NAME with the name of the node and SECRET with the secret from the node configuration on the Jenkins host.

Mark the file executable.

chmod +x /home/JENKINS_BUILD_USER/start-jenkins-node.sh

Create "/etc/systemd/system/jenkins_node.service"

[Service]
Type=simple
ExecStart=/home/JENKINS_BUILD_USER/start-jenkins-node.sh
WorkingDirectory=/home/JENKINS_BUILD_USER
Restart=always
RestartSec=60
User=JENKINS_BUILD_USER

[Unit]
After=network-online.target
Wants=network-online.target

[Install]
WantedBy=default.target

Replace JENKINS_BUILD_USER with the user that you created. Then you can enable and start the service with

sudo systemctl daemon-reload
sudo systemctl enable jenkins_node
sudo systemctl start jenkins_node

At this point you should see your node online in Jenkins and you are ready to use it for jobs.

My initial experience with Google Inbox

2015-10-24T21:33:00-05:00

This past week I finally decided to try out Google Inbox. The feature that really drew me to it was the ability to snooze emails. This feature allows you to make an email leave your inbox and come back at some later date and time. This is a really cool feature and a nice way to delay dealing with an email until you need to. In addition to this it is really easy to create filters that add emails to bundles (labels). These bundles can be set to appear in the inbox or not and you have some control over when the bundles appear in the inbox. When a bundle appears in the inbox it shows up as a wide message, once opened you see all of the messages in the bundle. This is a nice way to be able to group messages; you can see your labels in the inbox in a compact fashion. You can also decide which bundles will trigger notifications in the android app.

After using Inbox for about a week, I've decided to go back to using GMail. Here are my reasons:

The keyboard shortcuts in the web interface are lacking.
- No keyboard shortcut to goto a label/bundle
- No keyboard shortcut to type in the name of a bundle to move to. There is a shortcut '.' to open the move to menu though.
The bar on the left side showing the bundles don't show how many unread messages are in the bundle
When you choose to have bundles show up in the inbox you can select as the messages arrive, once a day (7:00) and once a week (Monday 7:00). I would really like to be able to at least pick the time for once a day and once a week. It would be nice to be able to pick the day on the once a week.
I like to make sure all messages that I keep are assigned at least one label. The Inbox interface doesn't allow me to see what labels have been applied to a message. This makes me very concerned that I will loose messages by them being archived and not assigned any labels. GMail's search interface is great, but I really like to be able to find my messages by label.

If Google fixes these features I will give Inbox a try again, until then I'm sticking with GMail.

IPv6 on Comcast Residential

2014-07-04T13:32:00-05:00

Comcast has now opened up their IPv6 service to residential customers. If you have a supported modem from Comcast and a device connected to it that understands IPv6 you can connect. You might ask why would I want to setup IPv6? And that is a good question. One reason is to stay up to date with current networking technology. Another reason is that we're running out of IPv4 addresses and we will eventually need to switch to IPv6. Currently many sites on the Internet are supporting IPv4 and IPv6 to help with the adoption of IPv6. Another reason for IPv6 support is that this setup can give you a subnet of public IPv6 addresses to use in your house. Meaning that you can allow computers on your internal network to be accessible from the outside world. Of course this also means that you could potentially open up your computers to the outside world, so you need to be careful and setup your firewall to keep your internal computers secure unless you want them accessible. This also removes any issues with NAT as IPv6 doesn't have any NAT support.

For my setup I have a compatible modem from Comcast and a Linux computer as my router. My Linux computer is running Ubuntu. These instructions are specific to my setup, but should be able to be used by others running most any Linux distribution.

The first thing you should do is secure your network from IPv6 so that something doesn't get in while you're setting things up. Here is my IPv6 firewall setup, it's very similar to my IPv4 setup, except the port numbers for DHCP are different. Outbound traffic is allowed and inbound traffic is denied. I've also disabled forwarding of traffic, this prevents inbound traffic directly to the internet work. This script needs to be located at "/usr/local/sbin/firewall-ipv6-start" for the radvd script at the end of this post to work properly.

#!/bin/sh

IPTABLES=/sbin/ip6tables
INET_IFACE="eth0"
LAN_IFACE="eth1"
LO_IFACE="lo"

$IPTABLES -F
$IPTABLES -X

# accept everything by default
$IPTABLES -P INPUT DROP
$IPTABLES -P OUTPUT ACCEPT
$IPTABLES -P FORWARD DROP

${IPTABLES} -A FORWARD   
-m state --state RELATED,ESTABLISHED   
-m comment --comment "allow inbound traffic for established and related connections"   
-j ACCEPT
${IPTABLES} -A FORWARD   
-i ${LAN_IFACE} -o ${INET_IFACE}   
-m comment --comment "allow all Internet bound traffic from the internal network"   
-j ACCEPT
${IPTABLES} -A FORWARD -p ipv6-icmp   
-m comment --comment "forward any ICMP traffic"   
-j ACCEPT

${IPTABLES} -A INPUT   
-m state --state RELATED,ESTABLISHED   
-m comment --comment "allow inbound traffic for established and related connections"   
-j ACCEPT

${IPTABLES} -A INPUT   
-i ${LO_IFACE}   
-m comment --comment "allow any local-only traffic"   
-j ACCEPT

${IPTABLES} -A INPUT   
-p ipv6-icmp   
-m comment --comment "allow ICMP traffic from anywhere"   
-j ACCEPT

${IPTABLES} -A INPUT -i ${INET_IFACE}   
-p udp -m udp --dport 546   
-m comment --comment "Accept DHCP traffic"   
-j ACCEPT

${IPTABLES} -A INPUT -i ${INET_IFACE}   
-p udp -m udp --dport 547   
-m comment --comment "Accept DHCP traffic"   
-j ACCEPT

The remainder of this post is based upon this post on using DHCPv6 with prefix delegation.

The next thing we need to do is get an address from Comcast along with a prefix (subnet) to hand out to the computers on the internal network. Comcast doesn't appear to be using router advertisements for IPv6, so we'll need to use DHCP over IPv6. For this I could use the ISC DHCP server that I'm using for IPv4, but it doesn't support prefix delegation which I need to give the other computers in my house IPv6 addresses. For this I installed wide dhcp client. Ubuntu includes this in the package wide-dhcpv6-client. A side advantage to using a different DHCP client for IPv6 is that you can turn it off to disable IPv6 support without messing with your IPv4 network. Once you install the client edit /etc/wide-dhcpv6/dhcp6c.conf to look like this. You will need to modify the interface used and possibly the sla-len. I found the sla-len by trial and error. You won't get a prefix if the value is incorrect.

interface eth0 { # external facing interface (WAN)
 send ia-na 1;
 send ia-pd 1;

 request domain-name-servers;
 request domain-name;

 script "/etc/wide-dhcpv6/dhcp6c-script";
};

id-assoc pd 1 {
 prefix-interface eth1 { #internal facing interface (LAN)
 sla-id 0; # subnet. Combined with ia-pd to configure the subnet for this interface.
 ifid 1; # IP address "postfix". if not set it will use EUI-64 address of the 
         # interface. Combined with SLA-ID'd prefix to create full IP address of interface.
 sla-len 0; # prefix bits assigned. Take the prefix size you're assigned
            # (something like /48 or /56) and subtract it from 64. 
            # In my case I was assigned a /64, thus the value is 0
 };
};

id-assoc na 1 {
 # id-assoc for external interface
};

When you start the wide DHCP client and all is happy you will find that your external interface has an address. In my case it starts with 2001:558:6014. See the output of "ip addr show dev eth0" changing the interface as appropriate. Below is the output for my system with the IP addresses masked out.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
 link/ether 00:d0:b7:3f:4d:18 brd ff:ff:ff:ff:ff:ff
 inet XXX.XXX.XXX.XXX/22 brd 255.255.255.255 scope global eth0
 valid_lft forever preferred_lft forever
 inet6 2001:558:6014:XXXX:XXXX:XXXX:XXXX:XXXX/128 scope global 
 valid_lft forever preferred_lft forever
 inet6 fe80::XXXX:XXXX:XXXX:XXXX/64 scope link 
 valid_lft forever preferred_lft forever

Once this is setup you can ping IPv6 addresses from your router. You can test this with "ping6 google.com".

Now to allow your local network talk to the Internet via IPv6 you'll need to allow forwarding and then assign them IPv6 addresses.

First we'll tell the kernel to allow forwarding by modifying adding the file 70-ipv6-routing.conf to /etc/sysctl.d. Note that net.ipv6.conf.all.accept_ra is set to 2. Any other value will not work due to how the router advertisements are handled.

# only set this on the external interface, otherwise we don't get a
# default route for IPv6
net.ipv6.conf.EXT_IFACE.accept_ra=2
net.ipv6.conf.EXT_IFACE.forwarding=0

net.ipv6.conf.INT_IFACE.accept_ra=1
net.ipv6.conf.INT_IFACE.forwarding=1

net.ipv6.conf.all.forwarding=1
net.ipv6.conf.all.autoconf=1

Once you change these values you will need to reboot or use the sysctl utility to set them immediately.

Now to hand out IPv6 addresses to the rest of the network. This will be done by setting up radvd. The package 'radvd' on Unbuntu contains this daemon. Once installed you can setup /etc/radvd.conf for the prefix that Comcast gave you. However when your IP address changes you'll need to update the file. So instead I have created a script that can be run from wide dhcp client. Put the following in "/usr/local/sbin/update-ipv6-setup.sh" and add a call to this script from the end of /etc/wide-dhcpv6/dhcp6c-script. You'll need to change the interface in this script to be your internal interface.

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }
try() { "$@" || fatal "'$@' failed"; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script dir
ectory"

prefix=$(ip -6 addr show dev eth1 scope global   
| grep inet6   
| awk '{print $2}')   
|| fatal "Unable to get prefix"

cat > /etc/radvd.conf.new <<EOF

interface eth1
{
 AdvSendAdvert on;
 AdvIntervalOpt on;
 MinRtrAdvInterval 60;
 MaxRtrAdvInterval 300;
 AdvLinkMTU 1280;
 AdvOtherConfigFlag on;
 AdvHomeAgentFlag off;

 prefix ${prefix}
 {
 AdvOnLink on;
 AdvAutonomous on;
 AdvRouterAddr on;
 };
};

EOF

diff /etc/radvd.conf.new /etc/radvd.conf > /dev/null
if [ $? -ne 0]; then 
 # only move if there are differences
  try mv -f /etc/radvd.conf.new /etc/radvd.conf
  try service radvd restart
fi
try /usr/local/sbin/firewall-ipv6-start

Now you have IPv6 setup on your router and your home network. I found that Linux, Windows and Mac automatically recognize the IPv6 router advertisements and grab addresses and setup routes appropriately.

Resources:

Skitch for easy sharing of drawings

2012-02-20T11:14:00-06:00

I use a Mac at work and was recently introduced to Skitch in a training class. This application is really handy if you need to share images or screen shots with others. Skitch allows you to easily take a screen shot of what you're doing and then annotate it to state what is important in the image. You can also easily do simple drawings and then share them. Once you have an image that you like you can just drag it to your email program, or if you sign up for an account with Skitch you can have the image uploaded there and accessible from their website.

Finally able to use my iPod Touch without Windows or Mac

2012-02-20T11:10:00-06:00

I've finally made it. I can now use my iPod Touch just with Linux. Apple adding over the air updates really helped, but I still couldn't sync my music. I also found that pulling large videos off via Dropbox didn't work as the app would time out. I had been trying to get gtkpod to work, but kept running into a problem with the database checksum on the music database. Recently I found Phone Drive and this has solved my problem. This app lets me copy images and videos out of my camera roll into the app and then the app will bring up a web server and an ftp server on the device. I can then browse the files from my web browser and download the files to my desktop. I can also put files up this way. Phone Drive also has a built in music player that will play any directory as a playlist and supports users adding their own playlists. So I wrote a little script (below) that will generate a playlist for all music and one for each artist on my computer and then I can upload those and all of my music via ftp. I did have a little trouble with some ftp programs that want to open multiple connections. In the end I used ncftp as it doesn't try to open multiple connections and it has the ability to upload directories recursively. I did find one oddity that I needed to turn off the auto conversion of ascii files otherwise playlist files got their line endings changed and Phone Drive wouldn't recognize them.

So my current list of apps that I regularly use is this:

Calendar syncing with Google, either via Active Sync or via caldav (this allows the colors to sync)
Contacts syncing with Google as an Exchange account
GMail app for mail from Google as I prefer the way Google does conversations
Appigo Todo syncing with Toodledo
PlainText for notes
Dropbox for keeping family pictures and moving images between my device and my computer
MiniKeePass on my device and KeePassX syncing through Dropbox
Podcaster for audio and video podcasts
ReadItLater for offline access to web pages

#!/bin/sh

debug() { ! "${log_debug-false}" || log "DEBUG: $*" >&2; }
log() { printf '%s\n' "$*"; }
warn() { log "WARNING: $*" >&2; }
error() { log "ERROR: $*" >&2; }
fatal() { error "$*"; exit 1; }

mydir=$(cd "$(dirname "$0")" && pwd -L) || fatal "Unable to determine script directory"

cd "${mydir}/mp3"

# generate artist playlists
find . -maxdepth 1 -type d -print0 | while read -d $'\0' dir
 do
 short_dir="${dir#./}"
 if [ "${short_dir}" != "." ]; then
 playlist="${mydir}/itouch-playlists/aaa_${short_dir}.m3u"
 log "Processing ${short_dir}"
 printf "" > "${playlist}"
 find "${short_dir}" -type f -name '*.mp3' -printf "/music/%p\n" >> "${playlist}"
 mv "${playlist}" "${dir}"
 fi
done

# generate all music playlist
log "Generating all music playlist"
find . -type f -name '*.mp3'   
-fprintf "${mydir}/mp3/aaa_all_music-new.m3u" "/music/%P\n"

Appigo Todo now supports Dropbox

2011-09-04T20:47:00-05:00

On my iPod Touch I'm using Appigo's Todo application. I had been syncing the data to Toodledo so that I had a backup in case something happened to my itouch. Now Appigo's Todo app supports Dropbox. The advantage here is that the place that I'm syncing to, dropbox, supports all of the features of the Todo app. This is important for recovery. Toodledo doesn't support hierarchical todo items. However when syncing with dropbox it's in Appigo's own format so all features are supported. And it turns out the native format is an sqlite database, so I can pull all of the items out of it in case Appigo goes away!

2/20/2012 update. I've gone back to Toodledo.com for syncing as the Dropbox sync for the todo app was just too slow. The Toodledo sync is faster, but it's not a smart as it keeps trying to sync when I'm offline and then popping up an error dialog. I've contacted Appigo about this, but haven't seen a fix yet.

Passwords for iTouch and iPhone

2011-08-07T18:27:00-05:00

Until recently I've been using MyKeePass to store passwords on my iTouch. This allows me to use KeePassX on my desktop. What I'd really like though is Dropbox integration. Well recently on Hak5 they said that there was a free app for IOS that understood KeePass password databases and MyKeePass isn't free. So I went looking and found MiniKeePass and discovered that it's open-source and free. On top of that it integrates nicely with Dropbox! You still need to manually sync the database, but it's really easy if you keep it in Dropbox. When you make changes on your desktop, you just open the Dropbox application on your iTouch and open the database and it imports right into MiniKeePass, then delete the old database from MiniKeePass. When you make changes on your iTouch, you can export it to Dropbox and overwrite the one there so that your desktop sees the changes.

Backing up cloud data

2011-06-05T18:32:00-05:00

This year I replaced my Palm Treo with an iPod Touch. To make sure that I could access all of my data on my iPod as well as on my Linux desktop (and anywhere else for that matter) I moved all of my data to the cloud. This is great for accessibility, however I still want backups in case something happens to the cloud provider. Remember that all data needs to be backed up or you run the risk of losing it.

The 3 cloud providers that I am currently using and want backups for are Google Contacts, Google Calendar and Toodledo. My backups are done from my Linux desktop, so I'm using unix tools to do the work. I'm sure people can find replacements under Windows as well.

I wrote a script for each job (based upon various solutions that I found on the net). The data for each cloud service is backed up into a directory named by the date and I choose to keep 30 days worth of backups, so that I can go back and pull information that I may have accidentally changed or deleted.

First up is Google Calendar.

#!/bin/bash -e
root=${HOME}/backup/pim/calendar
date_dir=`date +%Y%m%d`
base=${root}/${date_dir}
mkdir -p ${base}
find ${root} -maxdepth 1 -type d -mtime +30 -exec rm -rf \{\} \;
CURL="curl -s -S"
${CURL} -o ${base}/cal.xml "PRIVATE_URL_FROM_GOOGLE"

Here I setup some variables to determine the name of the directory based upon date. I then delete anything older than 30 days. The real key here is the last line. This tells curl to download the calendar and save it locally as "cal.xml". The PRIVATE_URL_FROM_GOOGLE is found by going to calendar settings for the calendar in Google. At the bottom is the private address with 3 links for XML, ICAL, and HTML. I choose to download the XML version, but you can pick any that you want to work with. Since this script has your private calendar URL, you should protect it from others as they can change your calendar with this URL.

Backing up Google Contacts is a little harder because you have to login to get the contacts. I found a script to do this searching Google. I found a script written in python that dumps the contacts in Google's XML format. I don't know the best way to load back from this format, but I'm sure I could figure it out if needed.

#!/usr/bin/env python

import gdata.contacts.service

gd_client = gdata.contacts.service.ContactsService()
gd_client.ClientLogin('user@gmail.com', 'password')

query = gdata.contacts.service.ContactsQuery()
query.max_results = 1000 # change for max contacts returned

feed = gd_client.GetContactsFeed(query.ToUri())
print feed

Then I wrote a wrapper script like the one for Google Calendar that puts the data where I want it.

#!/bin/bash -e
root=${HOME}/backup/pim/contacts
date_dir=`date +%Y%m%d`
base=${root}/${date_dir}
mkdir -p ${base}
find ${root} -maxdepth 1 -type d -mtime +30 -exec rm -rf \{\} \;
export base
${HOME}/bin/export-gcontact.py > ${base}/contacts.xml

I'm sure I could have done this in python too, but I wanted to keep all of my scripts as much the same as possible.

The last cloud service I needed a backup of was Toodledo. This one uses curl like the one for Google Calendar. Except that it does the login too. This script has curl post to the login page, save the cookies, then visit the XML export page. Since the cookie is saved, we don't get prompted for a login again. I tried to do this trick on the Google Contacts page as well, but it doesn't work because Google embeds a random hash in the login page as a hidden form field that needs to be passed on for the login to work. Anyway here's my Toodledo backup script:

#!/bin/bash -e
root=${HOME}/backup/pim/tododate_dir=`date +%Y%m%d`base=${root}/${date_dir}mkdir -p ${base}
cd ${base}
curl \  
  --output /dev/null  \ 
  --form 'email=YOUR_EMAIL'

--form 'pass=YOUR_PASSWORD' \ --form 'remember=1' \ --cookie-jar ${root}/cookies.txt \ http://www.toodledo.com/signin.php \ --output toodledo.xml \ http://www.toodledo.com/xml.php

Note that this script has your username and password in it, so you'll want to set the permissions on the script to keep others out.

To backup my email from gmail, I've found offlineimap to be a great application and easy to setup.

Passwords on the iPod Touch

2011-03-03T20:07:00-06:00

On my Palm I used Keyring to keep track of my passwords. This has a nice interface in jpilot to access it on my desktop. When it came to my iPod Touch I needed to find something else to use. I did some searching and found that there was an open source password program called KeePass that had applications for many platforms (including Windows, Linux, Mac and IOS). It turns out that there is an old version of the database (1.x) and a new version (2.x). There isn't an application for the new format on Linux yet, so I choose to use the old database format and the desktop application KeePassX. There are two applications for IOS listed on the KeePass website: MyKeePass and iKeepass. I chose MyKeePass because it supports both the old and new formats and seems to have been updated more recently.

MyKeePass can store multiple databases and those databases can be loaded from a public website, or from it's own web server. You can tell the application you want to import a database file and it starts up a mini web server that just has an upload page and tells you the URL to visit. If your computer is on the same network you can visit this URL and then upload the database and it's on your device. This is pretty nice. It would even better if MyKeePass could read the database file from Dropbox, but that functionality doesn't exist yet. You can have MyKeePass reference a public URL and that could be a public file in Dropbox, but that seems a little to open for me. I'd prefer it to be a private folder on Dropbox that I store my database in.

I found an XSLT to convert the output of the keyring XML exporter to the 2.x format of KeePass, but not to the 1.x format. So I hacked on the 2.x XSLT some and created my version for 1.x. This is a fork of the original that I hope will get pulled into the main.

Offline web pages on iPod Touch

2011-02-27T21:00:00-06:00

One of the things that I really liked about my Palm was using Plucker to view web pages when I'm not connected to the Internet. I've found a few apps for the iPod Touch that do this and here are the ones I've tried and what I think about them.

I read all of my news using Google Reader and so I wanted an app that would sync with it and keep the articles offline so that I could read them later. I tried Byline and MobileRSS. Both are good RSS readers, however I've found that I prefer MobileRSS. The syncing of all messages and ability to scan through all messages easily is great. Also MobileRSS allows one to send web pages to multiple other services, twitter, email, safari, Instapaper, Read It Later, etc. I'm currently using the free version and it works nicely. I'll probably upgrade to the pro version to get rid of the ads eventually.

For generic web pages I started out using iWebSaver favelet. This favelet takes a given web page and sends it through the iWebSaver favelet site and produces a data page that can be stored on your IOS device. This data page is then opened in Safari when you want to view it. The advantage to this is that you have the whole page as is downloaded onto your device. The problem is that it takes a long time to load and doesn't remember where you are in the page.

Next I tried Instapaper. This has worked very well for most pages. I've been very happy with the scaled down version of web pages. However I found that my Bible study had a problem with rendering. I couldn't see the days. I'm guessing this is some CSS magic that gets stripped out by Instapaper as extra visuals.

I then gave Read It Later a try. At first I thought that it wasn't going to help because the stripped down page looked the same as Instapaper and my Bible study still didn't work. However then I noticed a button at the bottom of the page with a 'T' on it. It turns out that this toggles the page from the stripped down text version to the full page. With a little reading through the options I found that if I disable auto picking of the "right" format to download I can have Read It Later always download both the text and full versions of the page. Then when I'm offline I can toggle between the two. This allows me to view most pages in text and then pages like my Bible study as the full page.

Palm Contacts to Google

2011-02-23T20:56:00-06:00

I have already been using Google to store some of my contacts for syncing using Zindus. So it seemed pretty natural to use Google to store all of my contacts. Using Google means that my information is available on any platform. Google provides a nice sync interface for the iPod touch that works well in an offline mode, so I was happy with it.

Originally I started writing an application in python to convert my Palm contacts to Google. As I got close to having this done I discovered that the Birthday and Anniversary fields couldn't get populated via the python API from Google (it may work now, but didn't work then). So I started looking for other options.

It turns out that Google allows you to export your contacts in a CSV format and can import in that CSV format as well. So I entered a few contacts with all of the fields that I wanted and then did an export to see what I got. The CSV file was pretty standard, but had lots of extra columns. So I tried trimming it down and discovered that I actually need to have most of the columns, even if they're blank, otherwise Google wouldn't recognize some of the other non-blank columns. This seems kind of odd, but my guess is that Google is looking for some set of column names to determine if it's a CSV file from Outlook or one from Google.

So now I needed to get my contacts into the Google CSV format. I started by exporting from jpilot and then trying to reorganize the data. This got old real fast. So I wrote another application in C++ using the pilot link library that would output my CSV files. I've put this application up on github so that anyone else can use it as well.

After running this application I had a nice CSV file that I was able to import into Google and all of my contacts were now available to my iTouch. I did end up hacking the IM conversion as I didn't have many contacts with IM information in my Palm. Otherwise the conversion went pretty well.

Convert Palm Calendar to Google

2011-02-17T20:34:00-06:00

I chose to use Google for my calendar. I already use Google for some of my mail and I sync contacts with Thunderbird using Zindus. Using Google means that my information is available on any platform. Google provides a nice sync interface for the iPod touch.

One annoying thing about the Google calendar integration with the iPod Touch is the lack of the ability to choose the color of the calendar. You need to choose to sync one calendar, then if the calendar color isn't what you want, tell google to not sync the calendar, wait 10 minutes, then tell it to sync again. Repeat until you get the color you want.

I started writing a python program to do the calendar conversion as jppy gave me acces to the Palm database and Google has a python version of their API. However I found the python API to be lagging behind the Java API in features supported, such as birthday and anniversary information. So I decided to use Java to interact with Google. In initial testing this worked well, however I needed a way to access my Palm database from Java.

Instead of direct access I choose to write a C++ program that read the Palm database using the pilot link library and then output using Google protocol buffers. This kept me from having to rewrite the code to parse the palm database and give me a format that I could read in most any language. The C++ program is available on github as palm-export. The Java program for importing into Google calendar is also on github as calendar-import.

As of this writing I can import all calendar information except recurrence exceptions. This is when you have a repeating event and have chosen to cancel some of the events in the series. Google supports this concept and I have even gotten it to work for some events. However since I haven't gotten it to work for all and I can't tell when it's not working, I came up with a workaround. Instead of creating the exception I create a new single event for the exception with "EXCEPTION:" prepended to the event name. This way I can recognize when one is canceled and I can just cancel the single event in Google, which works fine.

After the import I found a few of my appointments off by an hour. I suspect some timezones on recurring events didn't get in correctly, so you'll want to check appointments that are created with timezones in your Palm.

A couple of things to note. The iTouch doesn't support all repeat types nor all alarm durations that the Palm does, however Google does. What this means is that these will show up as "Custom" in the iTouch and cannot be entered directly. So when you want to use an alarm or repeat type that the iTouch doesn't support you'll need to use the Google web interface to the calendar and then wait for the data to sync, which is pretty fast.

Recently (2/17/2011) I've noted that Google sometimes has trouble loading all of my calendars so they will disappear from the iTouch and then come back later. I'm hoping this is just a temporary situation and will be corrected in the future.

Replacing my Palm

2011-02-16T21:21:00-06:00

I have been using a Palm as my PDA for a number of years. With the decline of the Palm platform I have been looking for a replacement PDA. The smartphones aren't an answer for me as I really don't want to pay for a data plan. I was looking at the Archos 43, however I haven't been able to get my hands on one to see how well it works. In particular if the resistive touchscreen is reasonable. Furthermore Best Buy removed the Archos 43 from their website after listing it for 2 months. So I broke down and bought and iPod Touch.

I run Linux at home, so a major concern is how to sync data. In this post and following posts I will describe the tools that I've chosen to use and why; as well as how I imported my old data. Hopefully this will help others that are considering making such a change or looking to convert from one PDA to another or just want to know what tools others are using.

I also wanted to make sure that all of my information is available offline as well like my Palm. So I have mapped each of my Palm apps to the apps on my iTouch and then either web applications or a desktop application to view the data on my Linux computer.

Todo: Appigo todo backed in Toodledo to have an online backup. This is also a handy way to edit a bunch of todo items. Note that the free account doesn't support the hierarchies of the Appigo application, but at least the data is there. To import data here I just manually entered the todo items.
Memo: Appigo notebook backed in Toodledo. I tried Evernote, but it was overkill for my needs and I wasn't able to easily import my information from my Palm. To import data I just copied and pasted from jPilot into Toodledo. 3/20 update: I've switched to PlainText for my memos app. It's really nice because it stores each memo as a text file in Dropbox and then it can be easily edited on my desktop.
pFuel: Gas Cubby is a nice application for keeping track of gas mileage and maintenance reminders. It also integrates very well with Appigo's todo application so that maintenance reminders can be turned into todo items. Importing data here was just a matter of exporting from pFuel through the palm memos application and then massaging the CSV file to match what Gas Cubby wants for their input format.
plucker (offline web pages): See this post for details.
MyBible: Laridian has made MyBible for the Palm and PocketBible for the iPod Touch (and other IOS devices). I was able to transfer all of my Bibles easily. This worked out very well.
Keyring: MyKeePass - more information on this in a future post
Titrax: HoursTracker is a nice application for keeping track of time on projects.
Dropbox: I didn't have this on my Palm, but it's pretty handy for keeping photos as the photo album application won't let me create albums.
Podcasts: Podcaster. I tried iTunes, but you can't subscribe to a podcast on the device. Podcaster allows me to do this and has a nice refresh feature to grab all of my latest podcasts.
Calendar: Google Calendar - more details in a later post
Contacts: Google Contacts - more details in a later post

Setting up Neatx

2010-10-17T20:24:00-05:00

I've been using the free version of the NX server from nomachine to access my Linux hosts and it's been working really well. I've set it up at work to allowed shared access to our compute server, however we've run into an issue here. The free version limits how many people can use the server. Now we really don't need the support and this isn't like a big production shop where we need a rock solid version, so I started looking for open source alternatives. I found FreeNX, but development there seems to have paused for now. I then found Neatx. It's hosted on google code and appears to be a little rough around the edges, but people claim it works.

There isn't an installer, but I'm a developer, so I figured this shouldn't be too bad. Turns out that after about 2 hours this evening I was able to get it going. Here are the steps that I followed. Note that I'm doing this on opensuse 11.3. The instructions should work elsewhere though.

Install nxclient, nxnode, and nxserver from nomachine's website. I grabbed the RPMs for opensuse and they installed just fine. I had originally done this because these were the binaries I was using, as it turns out I need some libraries from there for neatx to run properly.
Copy the libraries off to the side to use later. If someone knows how to get around this step, I'd be very happy.
- sudo cp -r /usr/NX/lib /usr/local/nx-lib
Remove nxclient, nxnode, and nxserver. We don't need them now.
mkdir nx
cd nx
Download the following source files from nomachine's opensource site. My downloads were version 3.4.0, but there may be a newer version out when you read this.
- nxagent-3.4.0-11.tar.gz
- nxauth-3.4.0-3.tar.gz
- nxcomp-3.4.0-7.tar.gz
- nxcompext-3.4.0-1.tar.gz
- nxcompshad-3.4.0-3.tar.gz
- nxproxy-3.4.0-2.tar.gz
- nx-X11-3.4.0-4.tar.gz
Untar the sources: for i in *.tar.gz; do tar -xzf \$i; done
cd nx-X11
make World
sudo cp programs/Xserver/nxagent /usr/local/bin/nxagent.real
Now here is where we use the libraries. I tried finding all of the libraries in the sources that I built, but they weren't enough, something was still missing. So you need to create /usr/local/bin/nxagent as a script with the following content:
- #!/bin/bash
- export LD_LIBRARY_PATH=/usr/local/nx-lib
- exec /usr/local/bin/nxagent.real "\$@"
Download neatx from svn. Note that there aren't any packages when I did this.
- svn co http://neatx.googlecode.com/svn/trunk/neatx neatx-read-only
Follow the instructions in neatx-read-only/neatx/INSTALL
Fix the instructions on nx's home dir
- sudo chown -R nx ~nx
Edit /usr/local/etc/neatx.conf to match your system. Some paths will likely need to be changed.

You should now have neatx setup. Connect to it from any NX client and give it a try. The default neatx.conf has the loglevel set to debug. Leave this on until you've successfully connected, then change it to info to keep your logs from getting real full. I spent a lot of time looking through the output of the debug logs for errors in my setup. Neatx is written in python, although you don't need to know the language to be able to debug most of the errors.

Tips for brother printers

2010-10-02T18:30:00-05:00

Brother makes great, inexpensive laser printers. However the technology that they use to determine when you need new toner is a little too conservative. I've found that it states that I need more toner when there is still quite a bit of toner left. Unfortunately at this point the printer will refuse to print anymore pages unless you replace the toner, or make it believe there is more toner. Being the frugal person that I am, I like to print all the way to the end of my toner. Here's what I've done to be able to print all of the way to the end of a toner cartridge.

If you have a black and white laser printer, then you need some black electrical tape. Other tape may work, but this is what I've used. You pull out the toner cartridge and note that on each end of the cartridge you'll see a round, clear plastic window. The printer shoots some light down this and if the light reaches the other end, you're told that you need more toner. If you put a piece of black tape across these windows then the printer believes that the toner cartridge is still full. Granted now the only way to know that you're low on toner or out is that things don't print so well. However you are able to print to the end of the toner cartridge.

If you have a color printer, the process is a little different. I purchased a HL-4040CDN and found that the black tape trick didn't work. Instead I dug up these instructions at fix your own printer.

Open the front cover of the printer
Press and hold the cancel button
Press the reprint button while still holding cancel
Here is the reset menu
Go to the appropriate cartridge on the menu and reset it and you're done!

FYI... Pressing the "Go" button and the up arrow gives you the parts life reset menu (drum, laser, fuser, etc.).

Don't delay upgrades

2010-08-31T20:40:00-05:00

I usually try and upgrade my computers fairly quickly once a new operating system or a new version of an application comes out, even if there aren't particularly new features that I'm looking for. However sometimes I've been a little slow to upgrade some servers because I'm too busy with other stuff or it's too hard to schedule downtime. I was reminded this week why it's a good idea to do upgrades sooner rather than later.

I had some computers that I had gotten behind on upgrading the operating system by about 1 year. The security patches were applied, but there was a newer version of the OS and I just didn't have the time to take care of it. Well, since then I moved out of that job, but still depended on the server. Since then it's up to someone else to upgrade this system and they're much like me, very busy with other things. So time goes on and now this server is 3 years behind on the OS upgrade and there are some major changes in the OS.

Now it's time to replace the hardware of the machine. Since it's a Linux machine the standard answer is just move the drives and keep going. I suspected there might be problems, so I left my number with my replacement. He started the replacement and well, it didn't go smoothly. As it turns out the new hardware wasn't quite supported by the older OS, such that the system would partially boot, but not completely. So we ended up doing a full upgrade across 3 minor and 1 major versions of the OS and then fixing up all of the little things that broke along the way.

In the end this probably took longer than it would have to do along the way because the system configuration would have been fresher in our minds. Plus the changes wouldn't have been so drastic and things would likely have migrated much easier.

So remember to make time to upgrade your systems right away.

Migrating bacula from MySQL to PostgreSQL

2010-06-25T19:59:00-05:00

So I've been looking to migrate my bacula installation from MySQL to PostgreSQL. Personally I like PostgreSQL better and the claims on the bacula-users list were that it's faster. So I did a bunch of reading, and then testing the database conversion and finally have made it through the process. Here is the results of how to do it. The system that I did the migration on is an OpenSUSE 11.2 system.

I first upgraded my install from bacula 3.0.3 to 5.0.0, still using MySQL. This was a pretty straightforward process. A little different because the packing of the RPMS changed between 3.0.3 and 5.x.

Backup my existing config files and the database.

tar -czf /root/bacula-backup.tar.gz /etc/bacula
mysqldump -u bacula -ppassword bacula > bacula-3.0-mysql.sql

Remove the old RPMs

zypper remove bacula bacula-bat bacula-updatedb bacula-server

Install the new RPMs

zypper install bacula-console bacula-console-bat bacula-director-mysql bacula-storage-mysql bacula-client bacula-director-mysql

Update the database

/usr/lib/bacula/update_mysql_tables -u bacula -ppassword bacula

The next step was the hard one, converting the database. I used a post from the bacula-users list to come up with the appropriate mysqldump line. I then repeatedly created the postgreSQL database and tried to import the dump until it imported without serious errors. In the end this is the pipeline that created a good dump:

mysqldump -t -n -c --compatible=postgresql --skip-quote-names --skip-opt --disable-keys --lock-tables -u bacula -ppassword bacula   
 | grep -v "INSERT INTO Status"   
 | sed -e 's/0000-00-00 00:00:00/1970-01-01 00:00:00/g'   
 | sed -e 's/\\0//' > fixed-bacula-backup.sql

The mysqldump line is pretty much what was in the mailing list post, except that I did all of the tables at once. The grep is to get rid of inserts into the status table. I was having issues with duplicate keys and such and the bacula-users list assured me that this table is created by the make_postgresql_tables script. The first sed line is to fix some bad dates. MySQL allows a date of all zeros, PostgreSQL doesn't, so I just bumped the 0 dates to the beginning of the unix epoch. The second sed line removes the extra null characters that showed up on all of the inserts into the log table. I'm not sure what caused these, but PostgreSQL doesn't like to import them and this made it much happier.

I then setup a .pgpass file in root's home directory so that I could secure the postgreSQL database with a password and not put it in my bacula config files. You can learn about the pgpass file in the postgreSQL documentation.

Next it's just a matter of creating the PostgreSQL tables as the postgres user (or some other user with postgresql superuser privileges)

./create_postgresql_database
./make_postgresql_tables
./grant_postgresql_privileges

And then loading in the data. This load took a little over an hour and a half on my system, so be prepared to wait a bit.

psql -Ubacula bacula < fixed-bacula-backup.sql

Now one needs to reset the sequences that postgreSQL uses to autocreate ids. I started with the instructions in the bacula manual, but needed to add a couple of missing sequences.

SELECT SETVAL('basefiles_baseid_seq', (SELECT MAX(baseid) FROM basefiles));
SELECT SETVAL('client_clientid_seq', (SELECT MAX(clientid) FROM client));
SELECT SETVAL('file_fileid_seq', (SELECT MAX(fileid) FROM file));
SELECT SETVAL('filename_filenameid_seq', (SELECT MAX(filenameid) FROM filename));
SELECT SETVAL('fileset_filesetid_seq', (SELECT MAX(filesetid) FROM fileset));
SELECT SETVAL('job_jobid_seq', (SELECT MAX(jobid) FROM job));
SELECT SETVAL('jobmedia_jobmediaid_seq', (SELECT MAX(jobmediaid) FROM jobmedia));
SELECT SETVAL('media_mediaid_seq', (SELECT MAX(mediaid) FROM media));
SELECT SETVAL('path_pathid_seq', (SELECT MAX(pathid) FROM path));
SELECT SETVAL('basefiles_baseid_seq', (SELECT MAX(baseid) FROM basefiles));
SELECT SETVAL('client_clientid_seq', (SELECT MAX(clientid) FROM client));
SELECT SETVAL('file_fileid_seq', (SELECT MAX(fileid) FROM file));
SELECT SETVAL('filename_filenameid_seq', (SELECT MAX(filenameid) FROM filename));
SELECT SETVAL('fileset_filesetid_seq', (SELECT MAX(filesetid) FROM fileset));
SELECT SETVAL('job_jobid_seq', (SELECT MAX(jobid) FROM job));
SELECT SETVAL('jobmedia_jobmediaid_seq', (SELECT MAX(jobmediaid) FROM jobmedia));
SELECT SETVAL('media_mediaid_seq', (SELECT MAX(mediaid) FROM media));
SELECT SETVAL('path_pathid_seq', (SELECT MAX(pathid) FROM path));
SELECT SETVAL('pool_poolid_seq', (SELECT MAX(poolid) FROM pool));

Updates I needed to add:

SELECT SETVAL('device_deviceid_seq', (SELECT MAX(deviceid) FROM device));
SELECT SETVAL('location_locationid_seq', (SELECT MAX(locationid) FROM location));
SELECT SETVAL('locationlog_loclogid_seq', (SELECT MAX(loclogid) FROM locationlog));
SELECT SETVAL('log_logid_seq', (SELECT MAX(logid) FROM log));
SELECT SETVAL('mediatype_mediatypeid_seq', (SELECT MAX(mediatypeid) FROM mediatype));
SELECT SETVAL('storage_storageid_seq', (SELECT MAX(storageid) FROM storage));

After that I needed to modify the Catalog section in my bacula-dir.conf file to use localhost for the "DB Address", remove the mysql socket reference and remove the password reference.

I also needed to modify the backup catalog command to be this (all on one line):

RunBeforeJob = "/usr/lib/bacula/make_catalog_backup bacula bacula \"\" localhost"

Computer companies being cheap can be annoying

2010-06-15T22:09:00-05:00

I have a 15" Macbook Pro unibody for work. Recently I was looking at the hard drive specs as I needed to upgrade another users laptop. Turns out that the SATA controller is capable of 3Gbps, however the drive from Apple is only capable of 1.5Gbps. What's really annoying about this is that I specifically requested a 7200RPM drive from Apple to get some extra performance. Granted that the interface usually isn't the bottleneck here, but still it's annoying.

Don't change 2 things at once

2010-06-09T02:03:00-05:00

So I'm working on a project that has a number of components to it. In particular the data is all stored in a MySQL database. For various reasons we wanted to convert this to a PostgreSQL database. So off I went working on a branch to make the changes and test the system. Meanwhile other parts of the system are changing as well, in particular the size of the input data. When it comes time to merge I get everything setup and then merge the changes in and all the tests pass, so I commit.

Then we notice that the nightly performance run on the continuous integration server is really slow, taking 2 hours instead of 15 minutes. We had noticed there being some slowness on logins before, but now the logins were slow and the software being tested was really slow. So we go about testing the CI server and find that openSUSE has been kind enough to keep track of the MAC address from the original system (we had installed on a different drive and chassis and then moved to this identical chassis). This caused the IPv6 link local address of this machine to match that of the previous system, which happened to be on the same network. This is bad, so we changed the settings back to the right MAC address and things were better, but still slow. So we decided to reinstall the OS, there are only a couple of directories of data to save, so no big deal.

After the reinstall, logins are faster, but the performance test is still slow. So we blame this on the changes to the input data. So over the next month or so we optimize the handling of the input data and find ways to reduce the input data some, but not back to the original size. The performance test is better, but still not back where it should be, we're down to about 1.5 hours now.

Other tasks kept me busy, so I didn't get back to this for another couple of weeks. At that point I'm running the performance test on another development machine and it's running in 15 minutes! This is great! But it's still running slow on the CI server. What's different? So I start checking versions of software and all and everything matches up. What I do notice is differences in the drives. So I get another drive installed in the CI server so I can test different configurations of drives for the PostgreSQL data directory. Here are the results:

XFS (logbufs=8): ~4 hours to finish
ext4: ~1 hour 50 minutes to finish
ext3: 15 minutes to finish
ext3 on LVM: 15 minutes to finish
reiserfs: ~1 hour 50 minutes
ext3 barrier=1: ~15 minutes
ext4 nobarrier: ~15 minutes
jfs: ~15 minutes

So as you can see, the filesystem really makes a difference. Turns out the development machine was using ext3 on LVM and the CI server was running ext4. After posting to the postgres-performance mailing list about this, it turns out that I either get speed or safety. With ext3, if the power goes out, I could have a corrupted database, with ext4 this isn't likely to happen. Given that I'm doing research here and if the power goes out during a test we have a lot bigger problems, I switched to ext3 and left it at that.

Now if I had just changed one thing (the database), rather than the database, the input data and the CI server setup around the same time, I probably would have caught this much sooner. It also would help to have my development systems setup with not only the same software, but the same filesystems too.

About Me

2010-06-08T12:39:00-05:00

Jon Schewe is an alumnus of Michigan Technological University. I graduated in 1998 with a BS in Computer Science. I'm currently working for BBN as a research scientist. For those that may be interested, my resume is available upon request.

What does it mean to be a "Research Scientist"? Well, it means I write lots of software that does of different things. I write software that controls oil refineries, I write network security software, I write software that controls robots. Some of my software runs on PCs, some on servers and some on embedded devices. Usually the software I write isn't going to be in a product for 3 to 5 years, if at all. It's fun to be a research scientist because I get lots of new challenges each day and I usually get a new project (or two) every year.