Salik Rafiq's Linux Diary

Saving away outgoing emails in Exim

2011-12-16T11:55:00.000Z

In our office we use internet fax by myfax.com who are simply excellent btw. Stay away from the other guys.

Incoming faxes are dropped into a common folder via a bit of procmail script in a common user's home folder. This user has shared folder setup, see my other post on setting up shared folders in this way.

However, from time to time there is a need for staff to see a fax, which has been sent by another colleague. Usually one colleague had to forward the e-mail to the other so they could view it.

Today, I decided that was a small simple issue that could easily be resolved. We use Exim4 as our e-mail transport. Exim4 is excellent, but I find its documentation a little on the terse side. But I persevered and found two blog entries which helped me out immensely.

The first blog explains how to setup a filter file to truly blind copy all outgoing mail from a user to another user. However for Ubuntu users the file setup mentioned will not work. Ubuntu uses the flat-file configuration to make life simpler for user, and it does! So this blog entry sorts it all out for us.

Now if you'd rather copy it to the shared folder, and your user's have rights you can use the Exim save command documented here. I plopped it through as e-mail and let procmail take care to drop it into the correct box.

I hope you find this of great use to you and if you have other solutions, feel free to let me know by posting below.

Random crash of Backup server

2011-09-18T14:26:00.002+01:00

I have been trying to nail down an issue with my backup server. I thought I had solved it with a boot option of noapic , because the server worked fine after this setup. It booted reliably up each time.

Until yesterday, when it did something that it had done before. That is the NIC seems to turn off all by itself and then the OS sort of hangs. Before this point the server had sent out its e-mails and even started a couple of backups. I'm not able to login at the console, so the only option I have is a power off and then a boot.

Checking through the logs reveals nothing. DMESG, syslog, messages reveals nothing. No panic, nothing. All I can see is that after sometime backuppc can no longer ping machines and then backuppc soon stops - perhaps because the server is now hung. Pinging the backup server does not work either, so the server really is locked up.

It is very annoying to say the least. The previous server was rock-solid in this regard. It was extremely slow, but at least it booted and stayed up. This maybe because it had a more modern BIOS than the current unit. Which makes me think I will now have to hunt down a updated BIOS.

This really is the first Ubuntu/Linux unreliability I have had in over four (4) years of using Linux.

If anyone has some place to begin, please don't hesitate to comment. I am running Ubuntu 10.04 LTS server edition. Its only purpose is to run Backuppc and this server is woken by WOL each night to start the backup and then shutsdown early morning when all backups are done.

edit: Just to let you know that this appeared to be a hardware issue and I have switched everything over to the original backup machine. Which is much slower but at least works. I now need to wonder what the problem is.

Fixing the CloneZilla GRUB issue

2011-08-30T18:28:00.001+01:00

Yesterday I wrote how I was experiencing some troubles rebooting a system after using clonezilla to clone the current drive onto a fresh HD. The old drive was 11 years old. It probably could have gone on for a couple mor years, but why take a chance.

A lot of the "fixes" for this issue are based on the older GRUB and these commands are simply not available in GRUB 2.0. These fixes ask you to use command like



find /boot/grub/stage1

boot (hd0,0)

setup hd(0)

Those are not there in GRUB 2.0

I managed to fix my issue by getting the machine to boot in a live disk. Well actually I used the older drive as the boot drive and the new drive as the secondary drive. With this setup I mounted the partition where the /boot folder was. In my case it was within the / mount point.



mount /dev/sdb1 /mnt

And then run the grub-install command as such



grub-install --no-floppy --root-directory=/mnt /dev/sdb

The process completes and the machine will reboot immediately - meaning that I don't have a chance to shutdown and remove the older drive.

I got the machine up now, but the NIC was not being recognised. It was odd, the card had power, but a lspci did not list it as a device. A quick reseat of the card solved that issue and my machine was back up.

I need to keep this notes for the next machine...the NAS, eeek!

Conezilla and GRUB a match

2011-08-29T23:37:00.000+01:00

The backup server in the office is obviously a quite critical machine. Its the only thing preventing user's from permanently losing their data. The machine was an old machine that I had bought from a charity. I replaced the powesupply and the internal fans and upgrade the NIC and installed a RAID.

Well I had been checking the drive health by SMART using the LINUX package smartctl. I don't fully understand the statistics in this package but it did look like the drive was wearing and due for a replacement. So I'd heard of Clonezilla and thought this was the perfect solution.

I had a 160GB drive I bought over a year ago for just this purpose. So I installed it and turned off the RAID while I figured things out.

I fired up clonezilla and went through the steps. I ran into a couple of hitches along the way.
Clonezilla failed to copy one of the partitions because it wasn't clean. I had to fsck that partition and start over.

The second was that I asked Clonezilla to copy over the GRUB in the MBR - but this also failed as I was still on GRUB 1.x and Clonezilla wanted to put in GRUB 2.0. So on the reboot the machine did not boot. It dumped me into a min-GRUB command line. This didn't have all the commands of the OS version. So I was a bit stuck.

Now the frustrating bit. Somehow as I was rejigging the cables and devices the machine failed to POST, which was sad indeed. I have a spare but this sad as this machine really had been a workhorse for four years now.

I now have to use another unused machine (lucky me) to take over the backup duties. I did try the Ubuntu repair-system process but this failed to install GRUB for some reason. Which does worry me a lot. Obviously another way would be to re-install Ubuntu fresh to the disk and clone over the partitions. That would be a last resort.

Any advice would be most welcome.

Is this so long for Steve Jobs?

2011-08-26T11:12:00.000+01:00

The news this week that the very ill Steve Jobs has left Apple I'm sure caught everyone off guard. However he has been ill for quite some time and had handed over the day-today running of the company long ago. His main contributions were in product ideas and announcements.

I've grown up with Apple. I used an Apple II+ and Apple IIe in high school and have used their original Macintosh computers in University. I remember how they marketed the Macintoshes as a kind of transportable computer with ads of students carrying them around. I really doubt anyone actually did.

But the Mac had flaws, its was expensive, there was no way to expand any part of it. If you didn't like the monitor tough, you wanted more memory hmmmm, colour? go away. A radical departure from the Apple II line which was expandable. Its main flaw was a shortage of business applications and when competing with IBM and Microsoft that was a killer. Steve Wozniak (remember him) left Apple over the killing of the Apple II/III line.

Shortly after that Jobs left, but he shortly reappeared as the founder of NeXT which developed a cube which ran a graphical UNIX OS. Apple was now floundering, with little direction and computers which looked like IBM PC's and still a lack of applications. Apple was bleeding money it had wasted on Newton and the never-ending development of a new OS. Steve Jobs saw his opportunity and convinced Apple to buy NeXT, for its OS, and he was back at the helm of his baby.

Jobs came back with a new Mac, then spotted the potable MP3 market and selling music online and the rest is history. While Apple's computers are still very much a niche market - I don't really think they sell very many - Apple is more of a desirable gadget company now. I wonder how Woz thinks about that?

Will Apple still be able to innovate, well surely it wasn't all just down to one man. I certainly hope not.

Backuppc+Samba and Backup Operators confusion

2011-08-19T13:29:00.000+01:00

I have been using the excellent backup server Backuppc and if you need a network backup facility which can backup Windows, Unix and Linux systems give it a look.

It does have one flaw in it however and this may catch you out if you get confused by the Backup Operators group.

On every Windows machine there is a group called Backup Operators which if look at the description says:

Backup Operators can override security restrictions for the sole purpose of backing up or restoring files

So you think to yourself, "Great I can put the backuppc user into that group on the Samba PDC and I'll be able to backup all the files I want".

Unfortunately it does not work out that way, for two reasons:

The domain Backup Operators can only be used on domain controllers.
Backuppc uses smbclient to connect and not Windows Backup API.

A look on technet explains the privileges the Backup Operator group has. So putting the backuppc user into the domain Backup Operators group does not give it privileges to backup all the PCs.

And putting the backuppc user into the Backup Operators group will not solve the problem either because of no. 2. Samba's smbclient will login as a normal user and the OS will use the file system ACLs to determine rights. Those rights to read all files regardless of the ACL permissions applies only when the Windows Backup API is being used.

So you have two choices really.

Add read, traverse, list folder contents for the Backup Operators group to the files and folders you want the backup to read. And place the backuppc user into the Backup Operators group on the PC.
Use the administrator account

The first choice is obviously very tedious and error prone and the second choice has some security risks with it. I chose the latter and I suspect most users will too.

I have read something about Zmanada's Windows Client for Amanda but I'm not sure if it uses the Backup API or not and would solve this problem. It is something for me to look into.

Backuppc and waking machines

2011-08-13T19:36:00.002+01:00

In our office there are 9 machines, all of which are backed up nightly with Backuppc. However I did often notice that machines did not get backed up, usually because the machine was not turned on. This was becoming a problem and over half of the machines were not being backed up nightly and some had not been done for many weeks. The nagging e-mails that Backuppc sends out were also being ignored as well.

I needed a way to ensure the machines were on, at least when it was their turn to be backed up. Enter the linux program etherwake. Etherwake sends out a magic packed to a specific machine by it MAC address. Typically machines will not wake when they are pinged, only a specific so-called "magic packet" will wake a machine.

Backuppc does not however have anything currently builtin which invokes etherwake before attempting to wake it. It also has a habit of doing a nmblookup before backup as well, which will most likely fail if the machine has been off for some time.

What I did was replace the ping command in backuppc with a bash shell script and disable the nmblookup. Here's how I did it.

First write a script in bash and place where you like. It will be someplace where the backuppc user will have rights to access to it. I put the script in the backuppc user's home.



#!/bin/bash



#this script is totally designed for the backuppc ping command

#which is the first thing it does before it starts a backup

#this is a substitute which pings the machine, if it is not

#awake then it wakes it using a magic packet - using the wol.bsh script

#then pings again to make sure



PING=/bin/ping

ARG1=$1

ARG2=$2

WAKEHOST=$3

ETHWAKE=/usr/sbin/etherwake

SLEEPTIME=3m



logger "Backuppc pinging  $1 $2 $3"





function fwol {

        TO_WAKEUP=$1

        sudo $ETHWAKE $1

        if [ $? -eq 0 ]

        then

           WOL_RES="OK"

        else

           WOL_RES="FAIL"

        fi

}



$PING $ARG1 $ARG2 $WAKEHOST >>/dev/null 2>&1



if [ $? -ne 0 ]; then

        fwol $WAKEHOST

        if [ "$WOL_RES" = "FAIL" ]; then

                exit 1

        fi

        sleep $SLEEPTIME

        $PING $ARG1 $ARG2 $WAKEHOST

        if [ $? -eq 0 ]

        then

           logger "success waking $WAKEHOST."

        else

           logger "unable to wake $WAKEHOST."

           exit 1

        fi

else

   $PING $ARG1 $ARG2 $WAKEHOST

fi



exit 0

I saved this a wolping.bsh. Essentially this first checks if the machine is up by pinging it. If it does respond then it simply drops out and pings again. Otherwise it invokes etherwake, waits for three minutes and then ping again. Note that in the backuppc code it will invoke the ping command twice, the first time as a wakeup and second the check the roundtrip time. Which is why I first send the ping to /dev/null and then ping again, I don't want the first output read by backuppc.

Now in backuppc in the server config, you will need to do the following:

Set PingPath to the path to where you saved the script above
Set NmbLookupFindHostCmd on Backup Settings to blank

Removing the NmbLookup will disable this "feature" and prevent backuppc from reporting the machine down before it wakes up.

That's it for the backuppc configuration. There is one last item which is needed. Remember etherwake only understands MAC addresses, but backuppc is invoking our script with a host name. So how can you make a host name to a MAC address?

Etherwake will search a /etc/ethers file when it is given a host name. So create one, you'll need to obtain the MAC addresses of all the machines you want to respond and add them into a ethers file as MAC hostname pairs, similar to a hosts file, eg:



00:3B:56:89:1A:22   myhost

And that's all there is to it. I now even have the backup server woken up by the router when needed. There is no reason to have any more machines on than necessary.

Next post I'll get into how you set this up on Linux and Windows to respond to these WOL/magic packet requests.

Webdav "improvements" in Win7 SP1

2011-07-29T22:31:00.001+01:00

I previously wrote about the problems I have had using webdav in Windows 7 to my webdav server. If you haven't read it follow this link.

Essentialy using the Window 7 webDAV client to a server using basic authentication over SSL did not work. No matter how hard you tried.

This situation has only been slight rectified in Win7 SP1, but with other issues. When I tried to create the drive mapping to the server I get some real weirdness.

When the mapping is created I get an error that the mapping already exists, which is very weird. A look under My Computer reveals three identical links have been created.
Using one of the links does connect to the server and I can see the folders and files. But I cannot navigate through the folders, nor can I open and of the files! Which makes it quite useless.

I have attempted to look for a solution to this issue, but nothing has turned up so. Its entirely possible that I'm the only one encountering this problem. For the moment I'll still keep using the 3rd party client I have been using for the past year.

Anyone with a solution please feel free to comment.

Thinking of the CRM future.

2011-05-16T12:52:00.000+01:00

To be honest business really has fallen flat here at Chameeya. I've no one to blame but myself really. That's one of the Botty Rules:

As a business owner I am 100% responsible for everything.

It seems the database programmer market really has fallen through the floor. This may be because businesses are cutting back on development budgets and/or shifting work to other areas such as Eastern Europe or South Asia.

So rather than put my head in my hands...well okay so I have done that. I am looking for new opportunities. As I look around I can see that eCommerce, Website, CRM and web applications are still in demand. In fact finding people with good PHP, Drupal, Wordpress skills are in high demand. I have known from my mentor that small businesses need help with CRM and contacting and maintaining contact with their businesses.

I've begun thinking that learning PHP and a package like SugarCRM or ZohoCRM might be worthwhile. I would still need to market it however. Really marketing is my week point. The rate at which I blog probably gives you an idea how bad I am! I'm still in the database area, CRM's rely heavily on databases, and its still programming. Customers will need customisation and that's where my programming skills come into play. And they will need maintenance contracts. Smaller customers can use an Access system, and larger customers can use a in-house or hosted SugarCRM system.

And when you think about it, businesses really need to keep in touch with contacts and customers more than ever now. Now is the time when they need a CRM more than anyone else.

Does this sound like a good vision of the future. I'm seriously asking for advice. seriously.

Why server virtualisation is important

2010-12-02T14:24:00.000Z

The other day our electricity bill arrived and uh, well it was quite large. It quite shocked us. And it also showed us that we were using a lot more power than we were last year. I began to think what had changed which so increased our power consumption over the last year.

In the past year I had just started to leave the servers, all five of them, on 24/7. I am hypothesising that that is the difference. I don't have a energy meter on each server but some have 4 fans and five hard disks. In theory they should spin down over time, but whenever I've checked it hasn't happened. And with a small office the servers are only utilized 25% of time, max.

So now I have begun a crusade to reduce our power consumption. I've already shutdown the Asterisk server and have all phones going direct to the VOIP provider. I have shutdown the Windows server which hadn't been used in two months. I moved services from the backup server to the NAS server. I configured the backup server to turn off at 8am everyday - I just have to remember to turn it on before I go.

Now I need to move services from my webmail server to the NAS machine. That will make one machine responsible for DNS, DHCP, LDAP, Samba, e-mail, IMAP, WEBDAV and Squirrel mail.

When I need a server turned on, to do Oracle or Umbraco or SQL Server testing, then I'll just turn it on when needed. I don't know what possessed me to build so many servers...no real use is it? This is why virtualisation is so important. Under utilised machines now become utilized and less servers means less power.

Glad I thought of it! (lol) Just being cheeky.

CRMs and just the basics please

2010-10-22T13:55:00.001+01:00

I've been speaking with my mentor last week about the work I do. Mainly it was about straightening out my skill set towards the market needs. As I market towards small business, it really is important to leave all the techno-lingo out of the marketing. That works fine for larger consulting houses who are looking for help - ie: a contractor. But it doesn't work for a small businessman who knows nothing about technology. They just want something to solve their need.

Lately I've been doing more and more simple systems in Access - mainly with CRM capabilities. Access is perfect for a single man shop. There is no installation requirement, it can use the Access runtime component. And as long as you don't need it to integrate with e-mail, its works a charm. Plus it can be extended to do so many other things.

For the web one can go and get ZOHO CRM which is a very simple system with basic customisability. And free for first three users and reasonable charges there after. Plus a developer can addon any forms and other customisation. This is a new avenue for Chameeya. It still uses a database in the back - so we utilise our database programming skills - and a UI - so we utilise our UI skills.

So many websites go on an on about technology - and I'm no different. This is probably because we are essentially geeks and think (or expect) everyone else to have an understanding of computers. Time for me to update and simplify the website.

My take on the Cloud (pt1)

2010-10-19T11:21:00.000+01:00

The Cloud. You knew I eventually had to get there didn't you? What is all this hubub about the "Cloud". Well I'll try and explain what I think, without creating yet another definition. I think there are about 68 definitions now. Well According to Simon Wardley there are.

The Cloud is the next step in the gradual evolution of computer services from innovative tool to commodity. These days hardware is relatively cheap. CPU cycles are relatively cheap. Gone are the days when computers were only used by wizards with special hats. Now anyone has access to CPU cycles, and for cheap.

Will we all end up eventually using the Cloud, whether we know it or not? I think that is eventually the end game. We'll have small local physical hardware with virtual desktops. The software is there now to do it. We're just waiting for the infrastructure to do it. The average connection speed in the UK is so slow it makes something like that undoable.

Currently we'll just have to put up with browser-based apps with about 75% functionality of the desktop apps. Developers will use Clouds for development. Teams will be dispersed around the world and VPNs will die a most honourable death.

We here at Chameeya have embraced this concept by signing up with Azure and SQL Azure. We see this as the way forward. Particularily for small firms which can now offer big iron when needed and no iron when not. Azure is a wonderful platform and its right that Microsoft has "bet their company on it". Though I don't think they have....

Finally think I've got Dovecot sorted (I think!)

2010-10-15T12:49:00.000+01:00

In the last blog I wrote on - at some length I might add - about setting up public shared folders in Dovecot and about just how hard that is! There are some annoyances with Dovecot, but that's the evolution of something free (the old "you get what you pay for" line). I haven't tried Dovecot 2.0 yet. Perhaps in 18 months when the next LTS of Ubuntu is available I'll get there.

After getting the basics of the configuration working I set out to start using and testing the setup. What I found initially is that Dovecot will use the root folder permissions as the permissions for manipulating files. This is particularily problematic when sharing folders as typically you need to enable group permissions. Eg:


rwx-----  user  mail  .
rwxrwx--  user  Pubmail .

In the dovecot error log, or just in mail.err, you'll see plenty of informative error messages about fchown, dotlock etc.. and in there will be information about the group being used Eg:


dovecot: IMAP(infocss): fchown(/home/infocss/Mail/subscriptions.lock, -1, 8(mail)) failed: Operation not permitted (egid=513(Domain Users), group based on /home/infocss/Mail)

In this case you can see that it used the permissions of the group to perform the operation and this failed. That group doesn't have the permissions - from Dovecot - to perform that operation.

The only way I have found to "fix" this is to allow additional groups to have privileged permissions within Dovecot. This is in addition to the group permissions you need on the files - though I must add that I haven't tried reducing the file permissions. I'm afraid of breaking something that works!

To do this you need to modify a line in the config file, as described in the Dovecot Wiki.


...
mail_access_groups = Group1,Group2
...

The wiki, while telling you that you can have multiple groups here, doesn't tell you whether the line is comma delimiter or space delimited. It is comma delimited.

That should get your public shared boxes working. Though I still wonder how this setup will work with use shared boxes. I still had tremendous problems with that.

In the last post I pondered about the ACL and how group will work there. Dovecot only really recognises a user's primary group but not supplementary ones. The "workaround" to this, and to get ACLs really working is to use Post Login Scripting. This is thankfully explained here.

I haven't tried this. Presumably the script can be put in an appropriate place - /usr/local/bin ??

The guys at Chameeya now have full access to the info e-mail box to answer and respond to customer enquiries.

Shared folders in dovecot (aka how to hide useful information)

2010-10-11T16:48:00.002+01:00

Along with the installation of squirrelmail in the office it became inconvenient for people to monitor the public e-mail addresses of the business. You know the info@companyname.com and support@companyname.com through the web interface. The user would have to log out and then in with the shared account - annoying.

In most IMAP systems where is a way to implement shared folders so user's can stored documents or monitor public e-mail accounts. For example you have a public support or sales account and you'd like the entire team or company access to these e-mails.

I use dovecot IMAP server v 1.2.9 which came with Ubuntu 10.04. The latest version is 2.0 but Ubuntu 10.04 is not using it yet. These instructions may apply to 1.2+ but you'd be best going to the dovecot wiki instead. As well I tried the shared folders setup but I had all sorts of permissions issues and one issue with the shared dict file which I could not resolve so I gave up. I assume it is a bug.

Firstly I am sharing a user's account e-mail and as stated above. This would have been the right way to do this, but I could not get it work.

First make a backup of the dovecot.conf file



cd /etc/dovecot

cp dovecot.conf dovecot.conf.bak

Then open the dovecot.conf file using your favorite editor and scroll down to find the following section



# REMEMBER: If you add any namespaces, the default namespace must be added

# explicitly, ie. mail_location does nothing unless you have a namespace

# without a location setting. Default namespace is simply done by having a

# namespace with empty prefix.

#namespace private {

# Hierarchy separator to use. You should use the same separator for all

# namespaces or some clients get confused. '/' is usually a good one.

# The default however depends on the underlying mail storage format.

#separator =

This is the default in dovecot, there is no namespaces at all. To implement a public (or shared) namespace, you will NEED to implement the private namespace. To do this uncomment the appropriate lines so that the following are set:



namespace private {

separator = /

prefix = 

inbox = yes

subscriptions = yes

}

Don't worry about the commented lines. Leave them in there, these are the only lines you need to change.

Now you need to add in the public namespace for the public folders to sit underneath. Note that I couldn't get this work directly to a e-mail folder. I'll get into what I did below along with the messy filesystem permissions and ACLs.



namespace public {

separator = /

prefix = Public/

location = maildir:/var/mail/Public

subscriptions = no

hidden = no

list = children

}

So this is a public namespace, the choices are private, shared and public. The separator, for subscriptions and e-mail clients is '/' which is the most common. In the subscriptions list the prefix for this mail boxes is 'Public'. The public folder is located at /var/mail/Public. The subscriptions are handled by the 'parent' namespace ie: the user's subscription list. This is probably what you want. And the Public folder is only listed if there are folders underneath being shared. Note that the actual path /var/mail/Public is just a container for the maildir folders underneath. The wiki for dovecot give more information on your choices here.

Then you'll need to add in two more lines to enable the ACL which will allow you to restrict/enable access to the shared mailboxes to the specific users you want.



protocol imap {

...

mail_plugins = acl imap_acl

}

...

plugin {

...

acl = vfile

}

Now things are setup in the imap server, but don't restart it yet! Now we need to get the filesystem and acl files right.

I then create the folder and shared subfolders.



cd /var/mail

mkdir Public

chgrp mail Public

touch dovecot-acl

I then symlink in the maildir folder I want users to have access to. I did this because the account already existed and I didn't want to effect any symlinks in the system pointing (and delivering e-mail) to this folder.



ln -s infobox ./Public/.info

Now I need to give users access the Public folder - but not yet the subfolders. The wiki misses some important points here. Firstly you need to allow users to read/list the public folder. I do this by editing the dovecot-acl file and allowing anyone read access



cd Public

nano dovecot-acl

add the following line



anyone lr

You need the put the same file in the maildir folders (not the cur/new/tmp folders) and any subfolders you want users to access. The wiki gives good explanation of what goes in the acl mine looked like



user=me keilrswtx

owner akxeilprwts

I could not get the group option working. Though I think if you read this then usage of the mail_access_groups flag may fix it. I haven't tried it.

Now the last bit is to give the correct file permissions for access. I did this with file system acl (using setfacl and getfacl). I won't get into the details here as they are well documented in many places.

This is what I did.



cd /var/mail/Public

chmod o+rx .

setfacl -dm g:GRP:rwx .info

cd .info

setfacl -dm g:GRP:rwx cur

setfacl -dm g:GRP:rwx new

setfacl -dm g:GRP:rwx tmp

setfacl -m g:GRP:rw cur/*

setfacl -m g:GRP:rw new/*

setfacl -m g:GRP:rw tmp/*

Now I can restart the dovecot server



service dovecot restart

Using my e-mail client I can see and subscribe to the public folders and read the e-mails.

Job done!

Lets not pressure higher ed too much guys

2010-10-01T12:21:00.000+01:00

Yesterday I have the pleasure of attending a workshop on getting business and education together on providing the right skills to business.

This was an excellent day and I met a lot of people and learned a lot. A few key points came out of this:

There needs to be identification of those students that want to either be an entrepreneur, employees or academics. There needs to be separate tracks for these, two at minimum.
Work placements need to be longer. I suggested something along the lines of the co-op program where students are on work placement (paid) for six months at a time, interleaved with course work.
Lecturers should also take work placements. This gets them out of the "Ivory Towers" to see how things are really done
Students need to be given advice on aptitude, communication and attitude. Strong communication - particularly face to face and working with others is extremely important. These skills seem to be in decline or were never taught in the first place. Computer Science programmes rarely teach documentation and communication skills. They should, these are essential to work in any environment.

I was bemused to hear of one local employer that was complaining that they could not find people with a specific skill set. He complained that this was due to cost. Universities were teaching using freely available technologies like Java whereas he uses more expensive Microsoft technologies. This really takes the disconnect to the extreme

I have always been passionate that universities cannot cater for every business need. If they teach fundamental skills these are easily transferable to any technology. Being an experienced programmer myself I know that I can easily learn another computer technology in days. This is simply a case where an employer is shirking their training responsibilities.

As I say to any employer, if you want specific skills from universities, then put your money where your mouth is and fund them! They might just prove more useful to your business, and the student, then your Ferrari.

There is a reason SquirrelMail is so popular

2010-10-01T11:50:00.000+01:00

As my wife now has two offices it was necessary to tie them together. I think I have spoken out this in previous posts. Previously I have spoke about getting WebDAV working on her server and the problems encountered with Windows 7. Now to get e-mail working.

My ISP, a strangly named, Big Wet Fish, offers me webmail using a very light and simple UI named squirrelmail. Which anyone can download freely from squirrelmail.org. This product is also extremely easy to install using the supplied quick install instructions.

I followed these very closely. I already had Apache and PHP and a Dovecot IMAP server, so I simply substituted where necessary. I also setup Apache with a virtual server with SSL for external access.

SquirrelMail is so quick and light that even over the slow connection on the server side (1Mbps upload and 3.5Mbps download) that its hardly noticable with the quickness of squirrelmail. Highly recommended.

Windows 7, WebDav and basic authentication

2010-09-22T16:13:00.000+01:00

In a recent post I gave some tips and experience on setting up a WebDAV server on lighttpd. Now one thing you noticed is that the authentication was basic. This means that the password is sent clear to the server from the client. Unfortunately most web servers which use LDAP as the authentication and authorization back end can only do so using basic authentication.

This is fine using Windows XP. Its WebDAV client doesn't care. In Vista it will work with basic as long as the connection is encrypted. In Windows 7 it will only work with digest authentication whether or not the connection is encrypted. Well that is my expereince. If someone has a way to get the Win7 WebDAV client to work over a SSL link to a WebDAV server which uses basic authentication please tell us.

As it is we must now use 3rd party WebDAV clients. Typically you have to pay in order to get one that is at least as good as the Windows supplied client.

Microsoft please sort this issue out and take it back to the Vista functionality.

A simple e-mail client and WebDAV

2010-09-15T12:29:00.000+01:00

Last week I explored two fabulous products which has made one of my clients very happy. This client has two separate offices with separate broadband connections. Now we're talking about a three person operation so a micro business really. Can't really afford a dedicated leased line between the two offices. ADSL broadband will have to do.

I began looking at the services which are needed. Currently e-mail and files access. I had two options really. VPN and or web services. VPN is dead simple to implement and maintaining passwords will be simpler with a domain. But over a slow connection VPN really drags. So I implemented that on either end and it is working using ipSec on Vista. I followed the instructions here.

I decided then also to implement web services for those times when a user simply wants to come in and check their e-mail or grab a file off the server. Turns out there are two simple products to do this, SquirrelMail and WebDAV.

WebDAV is more or less "builtin" to every Linux distribution out there. If you have lighttpd or apache on your machine you can turn on WebDAV and bingo you're set. WebDAV is defined as

Web-based Distributed Authoring and Versioning, or WebDAV, is a set of extensions to the Hypertext Transfer Protocol (HTTP) that allows computer-users to edit and manage files collaboratively on remote World Wide Web servers. RFC 4918 defines the extensions

This can be bolted on to almost every webserver - even IIS!. I used lighttpd and all it takes is to follow the following steps:

apt-get install lighttpd lighttpd-mod-webdav apache2-utils

Then enable the webdav and authentication modules in lighttpd

lighty-enable-mod auth
lighty-enable-mod webdav

Then edit your /etc/lighttpd/lighttpd.conf file and uncomment the mod_auth and mod_webdav lines from the file. It should look similar to this:

server.modules              = (
            "mod_access",
            "mod_alias",
            "mod_accesslog",
            "mod_compress",
#           "mod_rewrite",
#           "mod_redirect",
#           "mod_evhost",
#           "mod_usertrack",
#           "mod_rrdtool",
           "mod_webdav",
#           "mod_expire",
#           "mod_flv_streaming",
#           "mod_evasive"
)

Then add something similar into the file to setup the WebDAV location:

alias.url = ( "/webdav" => "/var/www/webee" )
$HTTP["url"] =~ "^/webdav($|/)" {
webdav.activate = "enable"
webdav.is-readonly = "disable"
webdav.sqlite-db-name = "/var/run/lighttpd/lighttpd.webdav_lock.db"
auth.backend = "htpasswd"
auth.backend.htpasswd.userfile = "/var/www/webee/passwd.dav"
auth.require = ( "" => ( "method" => "basic",
"realm" => "webdav",
"require" => "valid-user" ) )
}

Then restart lighty

sudo service lighttpd restart

You can use Windows to setup a WebDAV connection, the process differs from XP to Vista and Windows 7. You can find some instructions here - just ignore the top bit. I find that on XP certain files (typically .pdf files) cannot be opened. And in Vista you don't double-click to open a folder or a file. You have to use the right-mouse context menu and select open.

Now that gets you an unencrypted WebDAV connection. For some this maybe okay, but for professional installations you won't want important client files flying over public wires unencrypted. For this we'll need to turn on lighttpd's SSL capabilities. This is fairly simple, First we need to get lighty to listen on the appropriate port and select the proper certificate. Let's start with the certificate first.

sudo openssl req -new -x509 -keyout /etc/ssl/certs/webdav.pem -out /etc/ssl/certs/webdav.pem -days 3650 -nodes

Fill in the questions as appropriate. Make sure to put in your external web address - the one you are using either from your website host or from DynDNS (or similar) - when it asks for YOUR name. Note that the private and public key are in the same file.

Open lighttpd.conf again and add this into it.

$SERVER["socket"] == ":443" {
ssl.engine                  = "enable" 
ssl.pemfile                 = "/etc/ssl/certs/webdav.pem" 
}

This gets lighty listening on port 443, the standard one for SSL connections. You can change this port whatever you like. I like to use something else to help prevent hacking into my network.

Now to get this enables on the WebDAV piece we add these changes to the original setup.


alias.url = ( "/webdav" => "/var/www/webee" )
$HTTP["scheme"] == "https" {
server.document-root = "/var/www/webee"
webdav.activate = "enable"
webdav.is-readonly = "disable"
webdav.sqlite-db-name = "/var/run/lighttpd/lighttpd.webdav_lock.db"
auth.backend = "htpasswd"
auth.backend.htpasswd.userfile = "/var/www/web1/passwd.dav"
auth.require = ( "" => ( "method" => "basic",
"realm" => "webdav",
"require" => "valid-user" ) )
}

This is slighty different to the previous setup as lighty is setup to switch into HTTPS and WebDAV when the user connects to the webdav folder. To connect using a client you'll need to change the address to https:///.

Now restart lighty

sudo service lighttpd restart

One issue you will run into is that the webserver is running under the www-data account and group and all your files will need to be accessible by this account. You have a couple of options here:

Change the running account of lightty to a user with the same group privileges as your other users. A decent option if the server will only be handling WebDAV.
Add each of your users' into the www-data group. Not as nice as the above option and can give users abilities to delete or over write files in other parts of your web server.
Use ACL's to allow additional write privileges for the sets of users in www-data and other groups. Probably tricky to manage the privileges and ensure they are inherited properly.

Being a bit lazy I am trying the last option.

There you have it. very simple really. I've run out of room and will have to discuss squirrelmail in a further post. The guys here at Chameeya are very chuffed at this setup. Wow wee!

Comparing CMSs is very hard to do

2010-09-03T11:59:00.001+01:00

I have a requirement from my wife to build a case management system for her with a bundling capability. Now rather than start from scratch, which is tempting, I would look into what's available now.

There are legal case management systems around, though far too expensive for a small operation like my wife's. I did find one that was free (yay!) but development had stopped on it. None the less, no reason I couldn't have a look and try it out and use it as a template. The LCM from these guys. free for non-profit and NGO's and recommends using CiviCRM. But having tried CiviCRM I'm not sure that I agree. CiviCRM will need customisation before it can do all of what LCM can do.

Then I started looking at other platforms to build or modify a solution on. I looked at Dot Net Nuke. This is a highly popular solution built on .Net, but after reading through the experiences of application developers it does seem to be difficult to extend. So I have crossed it off my list. Next came Umbraco and mojoPortal. This looks highly promising. Both are light and put almost zero constraints on controls and layouts, which is perfect really. Both are using (or will soon use) the ASP.Net MVC model which is another winner. And both use SQL Server which I am highly skilled.

Another choice will will work on all platforms is Nuxeo DM. This is a enterprise CM tool which runs on Java is highly standards compliant. Full open source and nothing is held back, unlike Alfresco or KnowledgeTree. This is a great product but is perhaps overkill for what she needs.

My conclusion is that it is a race between Umbraco and mojoPortal. And one that I think Umbraco might just win. But the race ain't over till the fat lady sings. I am terrible the guys at chameeya will be throwing paper balls at me now.

Final Asterisk clean up

2010-08-27T17:08:00.001+01:00

This time I'm going to go through the final pieces of the Asterisk clean up on my Ubuntu 10.04 setup.

There were two annyoing things left behind by my install and setup of Asterisk.

When I shutdown freepbx (aka amportal) the Asterisk instance would crash and then restart
Ubuntu would want to update asterisk-mysql package

I'm going to show you how I got around these two issues. Firstly the amportal startup string and how to kill that nasty Asterisk crash and core dump.

The script I use to startup freepbx is below


#!/bin/bash
### BEGIN INIT INFO
# Provides:          asterisk
# Required-Start:    \$network \$syslog
# Required-Stop:     \$network \$syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Asterisk daemon.
# Description:       This script handles start/stop states of asterisk.
### END INIT INFO

set -e
set -a
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DESC="Asterisk"
NAME=amportal
DAEMON=/usr/sbin/$NAME

#test -x $DAEMON || exit 0

d_start() {
    amportal start
}
d_stop() {
    amportal stop
}
d_reload() {
    amportal restart
}
case "$1" in
start)
    echo -n "Starting $DESC: $NAME"
    d_start
    echo "."
;;
stop)
    echo -n "Stopping $DESC: $NAME"
    d_stop
    echo "."
;;
restart|force-reload)
    echo -n "Restarting $DESC: $NAME"
    d_stop
    sleep 10
    d_start
    echo "."
;;
*)
    echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
    exit 3
;;
esac
exit 0

Save this to /etc/init.d/amportal and then do the following


update_rc.d amportal defaults 80 20
update_rc.d -f asterisk remove

so that FreePBX will come up when you boot and remove the Asterisk from starting on its own. You could also use the sysv-rc-conf tool which allows you to do this visually.
You notice that this runs the amportal script which is located at /usr/local/sbin. Inside of there andother script is invoke called freepbx_engine. This is typically located at /var/lib/asterisk/bin. Inside there you will see lines like


killall -9 safe_asterisk
.
.
.
/usr/sbin/safe_asterisk -U asterisk ...
.
.
.
/usr/sbin/asterisk -rx "core stop gracefully" | grep -v "No such command"
/usr/sbin/asterisk -rx "stop gracefully" | grep -v -E "No such command|deprecated"

What is Safe_Asterisk? This is a script which starts a loop and runs Asterisk inside the loop. If Asterisk was to die for some reason then it detects this and restarts Asterisk.

There are two problems which I had with this. Firstly safe_asterisk starts Asterisk in the foreground, so my startup script would appear to hang after Asterisk started. Second when I killed the script (by using Ctrl+C), Asterisk would stop and crash and then restart and sometimes it would fail to restart and keep re-trying.

Getting Asterisk to start in the background was simple enough, but the crashes and attempted restarts still happened when I shutdown freepbx.

The solution I found was to do utilize the existing Asterisk startup script. I replaced the above lines with this


#killall -9 safe_asterisk
killall -9 asterisk
.
.
.
#/usr/sbin/safe_asterisk -U asterisk -G $AMPASTERISKGROUP
/etc/init.d/asterisk start
.
.
.
#/usr/sbin/asterisk -rx "core stop gracefully" | grep -v "No such command"
#/usr/sbin/asterisk -rx "stop gracefully" | grep -v -E "No such command|deprecated"
/etc/init.d/asterisk stop

That solved all my problems with the starting and stopping of freepbx.

Now, if you've added on the Asterisk Addon from my previous post. You may find that next time you do a apt-get or aptitude to update your OS that it wants to pull down an "updated" asterisk-mysql. Do not do this! It will overwrite the addons and they will then not load. To keep from being prompted you need to pin this package at the version that its on and you will not be prompted anymore. This is thanks to this page.

echo asterisk-mysql hold | dpkg --set-selections

and to check that worked you do


dpkg --get-selections | grep asterisk

asterisk                                        install
asterisk-config                                 install
asterisk-mysql                                  hold
asterisk-sounds-extra                           install
asterisk-sounds-main                            install

what more could a man want!

I hope that helps a lot more of you with using FreePBX which just gets better and better.

Is it better to extend or rewrite

2010-08-24T16:09:00.001+01:00

I have been dealing with this conondrum for sometime now. The problem, well its not really a problem. Lets say that its an opportunity then. I have this opportunity to develop some software for a small legal practise I know of. They are very small, tiny is closer to the description as it consists of a single solicitor and spouse as the unpaid 'volunteer' helping out.

This person approached me a while back to come up with a small and cheap system which would provide basic needs of case management. They also want something that can build bundles a lot easier than the very manual process the have now.

So over the past while I have umm'd and ahhh'd over this. Being a software programmer, but mainly in desktop and database applications I have no contact at all with the legal case management area. So do what everyone else in my situation does. I google.

After a lot of looking around, I've come to the conclusion that I can either take a CMS and extend it to perform legal case management. Or I can write on from scratch. The latter does sound tempting as tying yourself into an existing framework can hamper some future customisations one wishes to do. Then there is technology, .Net or LAMP? (well there are more choices than that really).

Not being an expert in web technologies I have to decide this without really knowing anything. Who do I ask? Maybe somebody in LinkedIn knows?

My choices are:

CiviCRM and extend it.
DotNetNuke and extend
Build from scratch on LAMP
Build from scratch on ASP.Net

The technology choices are quite open as I'd have to learn ASP.Net or PHP or whatever anyway. Though I'd rather stay away from MySQL and stick with something better like SQL Server and I do like Linq. So one choice made, it must be on ASP.Net!

Then I found this book Pro ASP.NET 4.0 CMS. I started to think. Yeah, all the source, no licences at all. Sounds good. Sounds very good.

Me and gang at Chameeya are gonna give this a go.

Getting those Asterisk Addons into Ubuntu

2010-08-18T16:14:00.000+01:00

I have this VOIP server in my office which is running FreePBX and Asterisk. Well its not in my office, its in a server room down the hall. Its a small AMD Duron of about 1Ghz, which is enough.

FreePBX is a great GUI and management interface for Asterisk and I encourage anyone to use it. I originally used the AsteriskGUI, but that was buggy and had severe limitations. Also, development had basically stopped on it. You know when Digium (the sponsors of Asterisk) start shipping FreePBX with Asterisk in its AsteriskNOW package - their own GUI is dead.

It took a bit of trial and error to get FreePBX installed on Ubuntu 10.04 - as there is no package for it. I also managed to get it working with lighty (lighttpd) which saved a lot of memory. And as an additional tip, disable innoDB in MySQL which will save you even more memory.

By default Asterisk logs all the calls to a CSV file. These files are very ugly and impossible to read. There is a panel in FreePBX which allows you to interrogate CDR or call records. This panel queries a MySQL database. This database is part of the FreePBX install script which managed to work for me. But to log to the database Asterisk must have the cdr_mysql addon. This does not come straight with a ubuntu package. There are a couple of steps you need to go through.

First, do not go out an grab the svn sources from digium site and compile and install. This will not work. You will most likely get these types of errors in your Asterisk full log:


Module 'app_addon_sql_mysql.so' was not compiled with the same compile-time options as this version of Asterisk.
Module 'app_addon_sql_mysql.so' will not be initialized as it may cause instability.
Module 'app_addon_sql_mysql.so' could not be loaded.

Follow these steps though and you will have success:


~# apt-get build-dep asterisk-mysql
~# apt-get -b source asterisk-mysql
~# dpkg -i asterisk-mysql_1.6.2.0-1_i386.deb

So what does this do? The first step downloads all the build dependencies for the add ons. The next builds the addons from the source. The last step installs the addons into the Asterisk modules folder. Thanks to the guys as Launchpad for helping everyone sort this out. See here.

Then restart your Asterisk server and you are done. I think next time I'll show you how to get around the core dump when you shutdown Asterisk using the scripts which come with FreePBX.

Now if we could only nail down the call quality, me and the other guys at chameeya would be so happy.

Even Linux needs to scan for Windows Viruses

2010-08-17T14:26:00.000+01:00

What! you say. Why would Linux need to scan for Windows viruses, surely they don't affect Linux. Well, yes it is true that a standard Linux O/S setup cannot be affected by Windows viruses. However, almost all setups I read about with Linux involve either acting as a server for Windows machines or as a dual boot/wine host.

In these cases it is essential to run a antivirus tool to prevent your Windows machines from being infected by files sitting on a samba share, shared e-mail server or wine. In my case, my office server stores Windows documents on samba shares and shares e-mail via dovecot IMAP server. The e-mails are fetched from various sources using fetchmail and passed through procmail for delivery to the correct mail folder.

The most common antivirus package on Linux is ClamAV. There are others available from:

I will be speaking about my setup with ClamAV. ClamAV is the most common anti-virus you will find on a Linux setup. This is because it is completely open source and free and available as part of the major Linux distributions. I use Ubuntu throughout my business and ClamAV is included in the main repositories.

To get started the simplest thing to do is install:


sudo apt-get install clamav-base freshclam-daemon clamav-daemon clamassassin

let's go through this. We install the base tools and software (clam-base), then a daemon to download the latest virus and malware signatures (freshclam-daemon), the clamav daemon - very useful for scanning e-mail and a script for processing e-mail virus scans (clamassassin).

There are two ways to scan your files, with clamscan or with clamdscan which uses the daemon. Using clamdscan is usually faster as all the virus definitions are loaded in memory. However, the daemon runs under its own user (clamav) and you need to add it to any groups necessary to scan your files.

I use clamscan within a script file which is then invoked through cron to provide a nightly scan with a e-mail summary.

To scan e-mail I needed to add in a procmail recipe. Procmail is a wonderful piece of software which could use a better web page. Procmail delivers mail based on a set of rules/patterns which are set in a global and user's rules file.

My current setup I have fetchmail bringing down e-mail from my ISP and various other sources. Procmail which is dropping the e-mails into the correct folders. Exim which is currently only acting as a local and forwarding e-mail server. There is a way to tie in ClamAV into Exim but this would only catch outgoing e-mails which I'm not worried about. The only place appropriate was in Procmail.

After some looking around I came up with this procmail recipe:

MAILDIR=/var/spool/mail
DEFAULT=$MAILDIR/$LOGNAME/
SUBJ_=`formail -xSubject: | expand | sed -e 's/^[ ]*//g' -e 's/[ ]*$//g'`

# check if there is a virus and indicate in subject, don't actually delete it!
:0fw
| /usr/bin/clamassassin

:0
* ^X-Virus-Status: Yes
{
  :0: fhw
   | formail -I"Subject: [**VIRUS**] ${SUBJ_}"
  :0: fhw
   | formail -A"X-VIRUS-INFO: BLOCKED BY CLAMASSASSIN"
  LOG="VIRUS "
  :0:
  $DEFAULT/.virus
}

To test it go to http://eicar.org/anti_virus_test_file.htm where you can download a sample virus and then send it to yourself. The e-mail should be trapped and dropped into your virus folder with the subject altered.

My colleagues appreciated not having to worry any longer about e-mails with suspect viruses coming through the e-mail system. I also have made some adjustments to our Windows e-mail clients for any remote accounts accessed by IMAP (like info and enquiries). Some e-mail clients like to cache the e-mails and attachments from these accounts and this bypasses our e-mail scanner. I have turned off that caching because as you can imagine these accounts get a lot of spam and malicious attachments.

We here at chameeya now feel much more secure. We are even improving in other areas by removing local admin rights for many users and ensuring everyone is logging in with domain accounts.

Now to see if I can get a proxy server to keep out all the porn!

The future wave of SQL development?

2010-08-05T14:56:00.001+01:00

Last month Microsoft released the SQL Azure management tool currently code named project Houston. You can read more about it here. Houston is a Silverlight tool which provides a GUI to managed and create items within your SQL Azure database instance. I would encourage everyone to watch the demos and get hooked.

Houston will allow those developers not familiar with SQL Server to build tables, views and stored procedures quickly as part of their web project. Though if someone can code a stored procedure they wouldn't really be a simple developer would they?

It is, in a sense, a kind of Access-ish type of interface to SQL Azure. SQL Azure really does allow a small operation like us to build robust applications for enterprises, hosted in the cloud and managed by us. No need for us to rent expensive servers and setup all the software on our own or, eek!, build our own with the expense and SLA requirements. The cloud commoditises all that and lets us develop and someone else manage. But with the additional flexibility that a customer can turn it off when not needed. So you only pay for the CPU and storage you are using.

This month I hope to attend several more seminars on SQL Server. And at the end of September there is SQLbits. For anyone who hasn't been or heard of SQLBits, this is a community conference which has now expanded beyond its original Saturday format. Check out their website and register. This time they are in York.

Access reporting using sub-reports

2010-07-23T15:18:00.000+01:00

Access isn't well regarded for its reporting. In my experience a lot of users use Excel as the Access reporting tool. But there is one category where Access excels above Excel. (nice pun eh?). That is sub reporting. This is where two separate pieces of information must be brought together on a Access report.

An example is the personnel assignment report below:

In this report there are three different pieces of information brought together.

Individual basic details
The professional qualifications
Individual assignment history

This normally cannot be done in Access. So how does on accomplish this. With sub reports.

Now look at the same report in design view:

Its a bit messy - working with sub-reports always is. But here you can see that for each person in the detail section are two sub-reports. One is their profile, skills, wage number, education. And second their work assignment history. These sub-reports are always stored as separate reports within Access.

This does help make it easier to layout the reports.

Lets take a look at how the sub-reports are linked to the main report. There must be a unique way to join the two. otherwise the sub-reports will display either no data or all data. We want the sub-report to only show the related data. Lets take a look at the assignment history sub report.

To do this you must have the subreport item selected. Don't click on the top left as that it is the report. When you've got it right you'll see small resize handles around the sides of the subreport.

In this case you can see the sub-report field resourceID it linked with the main report field ID. This ensures you are reporting on the right data.

And you can carry this forward with sub-reports containing sub-reports. I'm not sure how far it goes as I've only gone to the sub-sub-report level myself.

This is just a sample of the project that my colleagues and I at chameeya take on on a regular basis.