Month: March 2010

F1 race opinions – Australia

I had planned on watching the race live last night from 11PM to 1AM, but after a fairly long day and too much pizza I went to bed before it started and was therefore able to watch it this morning on the DVR while avoiding the internet (and hence results), luckily I was awake enough to watch the whole thing and it really was a fun race.

Perhaps though it is time to change the definition of ‘winner’ though ?

Over the weekend Lewis Hamilton manage to score more ‘Hero Points’ than any driver that I can remember since the late, great James Hunt and therefore should be considered the winner this weekend.

The viewers were also winners this time around after the snooze-fest last time around. This race had pretty much everything, it started out wet/damp, then there was the almost expected first lap crash that shuffled the field. A brief session behind the safety car followed and then the race proper began. During which time the track dried out and the field got shuffled a little and there was more overtaking for position than anyone could have predicted.

Read More

Heading back to work….

Today is the last day of my enforced ‘vacation’, I have an official start date of ‘Thursday” for my new job.

It has been a bit of an adventure to say the least. The day before the lay-off I applied for a job at a company that is fairly close to my old office, the position sounded interesting and the company looks strong and forward thinking. I heard nothing. I interviewed elsewhere and found that although there are jobs out there, there are few that really excited me. In the suddenly busy days and changed routines I completely forgot about the job that I had applied for.

Now at this point I suggest that you read back a little more, one of the first things I did when faced with job hunting was to re-write my resume. The prior version was rambling and full of way too many detailed descriptions. I read somewhere that the average recruiter will merely glance at a resume and scan it for applicable skills, so the absolute best way to get their attention is to use simple, short, concise prose and perhaps lots of bullets. Knowing that given the chance I can sit and write a novel when a simple yes/no is needed, you can probably understand when I say that my resume was more like a bound volume than a couple of page introduction to my career.

So I carefully re-wrote it and created a format that I felt was eye catching enough to get noticed, but not so much that it was for all the wrong reasons, I did leave out the fluffy bunnies, but the cool font and clever layout remained. I had a set of great interviews at a small company that I felt went well, but no offer other than ‘you are #1’ was forthcoming, they expanded their search, but I remained #1, but they are still looking elsewhere.

The level of interest in my newly concise resume was substantial, quickly, including a call from an agency in Florida. I nearly ignored the message because I did not understand half of it – a combination of a quick talker and a bad line nearly conspired to kill it dead.

As soon as I spoke to the caller, Donna, I realized that this was a very, very good opportunity and potentially a very, very good fit. I e-mailed her a clean version of my resume and then things started to happen, quickly.

At some point I realized that I had applied for this job about a month previous !

I interviewed on the phone a couple of days later and once more felt that the opportunity was extremely good. I also felt that I had ‘clicked’. Very quickly I was called in and spent several hours over two days talking with potential colleagues. Then it was a question of getting the offer out and accepted and background checks etc.

So I start tomorrow, which is great. I have spent way too much time fiddling with the suspension on the MINI and re-writing css for my web sites and changing hosts for cncpics.com….

How does my mind work ?

I spent some time today sorting out my music collection. It was a mess with a lot of duplicates and even missing files, thankfully all of our music is stored on the server, so my personal collection, which resides on an internal disk that is accessible from both Windows and Linux (iTunes and Rhythm Box) is merely a subset of the main library.

So any missing files are at least accessible.

Booting into Windows and using iTunes to manage the music makes some sense due to the ease of use of the application and the extra facilities over RB. The major advantage is that drag and drop will simply copy and catalogue files and place them into the correct structure.

However my quad core 64Bit 8GB machine actually struggles to run iTunes at a decent rate, adding or deleting more than a couple of hundred files in a single hit it will report that it is out of memory. Secondly there is little to no scripting abilities in Vista compared to Linux.

After a couple of hours of frustration, performance issues and more frustration I switched to the land of $ prompts.

To find missing files I simply compared the number of files in the album folders using ls – | grep wc -l (number of lines) and then also looked for dupes with a few simple grep’s.

Once I had the missing and dupes under a reasonable level of control I started to look at the albums that I had in the collection and deleted ones that I do not want.

Again this was easier from the file manager than from RB (or iTunes).

Cleaning up this way reduced my local storage nicely and made sure that I only had whole albums (or CD singles) as far as possible. Running RB to show the results and to sync with my iPhone was pretty gratifying.

It also sparked a memory.

I rarely listen to the radio, but one morning a few months ago I was using ooTunes to listen to BBC Radio 1, a British pop channel and they played a track called ‘Paper Planes by a band called MIA when I heard it I bought the album on iTunes within a few minutes. The song was oddly familiar but I could not for the life of me figure out where I had heard it. Until today.

It turns out that I already had the song, albeit in a massively re-mixed form as it is used in Slumdog Millionaire, one of my favorite movies of last year.

How do I know this ?

There was a missing track in the Slumdog OST folder and my script highlighted it.

Somehow my brain had made the link when I had heard the un-remixed version on the ‘radio’ hence the familiarity.

I really prefer the original version over the remix though.

Now if only I could use this association and recognition skill for something useful…..

Oh and at the end of the scripting and messing, I booted back into Vista and imported the new library back into the iTunes library and all was well in the world…..

Getting Familiar with Homer J.

During the rebuild of cncpics.com I took a decent look at the log file.

It looks like there are a large number of robots that are attempting to harvest the images, causing huge amounts of database lookup and then eventually an invalid URL type error.

Because of the way that the gallery software works, anything that looks like a gallery URL is handled by the database.

By ‘fishing’ for images the robots are causing a lot of database reads and  log file writes. Which adds to the load on the server. Which in turn reduces the performance for the legitimate users.

To alleviate the load, we have changed the URL pattern for albums and images. All of the old links will land on the ‘404’ page.

http://cncpics.com/404

While I realize that most of the 404’s are not humans, it amuses me to think that the robots are getting familiar with Homer J.

Network Violation – again

Is there no end to the stupidity that is rampant within godaddy ?

This morning while waiting for my tea to brew I checked my e-mail. Once more there was a ‘network violation’ notice from GoDaddy.

Dear Sir/Madam,

It has come to our attention that your dotdotnetdot.net hosting account is being used as a file repository. This is a direct violation of our Terms of Service. All files that are stored on a hosting account, must be used for a working traditional website. You may review the Storage and Security section of our Web Site and Virtual Dedicated Server Service Agreement for further clarification.

You have 48 hours to remove all the files not being referenced to a website. Specifically, the following are currently in violation:

cncpicsinfo.tgz 91GB

Failure to comply will result in access to the files and/or directories in question to be removed on your behalf. If you are interested in keeping file backups, you will need to purchase the Online File Folder, a Virtual Dedicated Server, or a Dedicated Server.

Obviously I am a bad boy, again.

But look a little closer and ask yourself, why would I stash a 91GB Zipped/Tar’d file on my OTHER hosting account ? The hosting account that hosts this blog and the associated files ?

It seems that GoDaddy took it upon theirselves to create an archive of the cncpics site for me, I did not ask them to do this, nor do I need this. To make matters even worse I cannot get to the file to either download it nor delete it.

Yet I am in violation, again.

GoDaddy’s tech support are telling me that they cannot remove the file without my authorization, which I have attempted to do, but they still cannot do it and their ‘senior support team’ are working on the problem.

let me give you a hint guys, you need to use the ‘rm’ command to delete it

something like this

rm cncpicsinfo.tgz

Not exactly a difficult task, yet there ‘senior tech support team’ have been working on it for an hour straight.

Just for the hell of it I just kicked off and FTP download of the file – all 91GB, if the tech support crew do not manage to master the complexities of the ‘rm’ command within the next 40hrs and 32 minutes, then the file will be resident on my PC.

*sigh

Dear Sir/Madam,

It has come to our attention that your dotdotnetdot.net hosting account is being used as a file repository. This is a direct violation of our Terms of Service. All files that are stored on a hosting account, must be used for a working traditional website. You may review the Storage and Security section of our Web Site and Virtual Dedicated Server Service Agreement for further clarification.

You have 48 hours to remove all the files not being referenced to a website. Specifically, the following are currently in violation:

cncpicsinfo.tgz 91GB

Failure to comply will result in access to the files and/or directories in question to be removed on your behalf. If you are interested in keeping file backups, you will need to purchase the Online File Folder, a Virtual Dedicated Server, or a Dedicated Server.

Learning through adversity

As I have documented extensively over the last three days, cncpics.com has been causing me a lot of grief.  To re-cap, the mega post of photo’s combined with a failed cron job several weeks ago combined in a ‘perfect storm’ to seriously annoy the godaddy network admins.

I had three choices.

1 – find a new and inexpensive host and hope that they would be able to deal with the traffic that came their way.

2 – give up and walk away from something that I have put serious time/effort and passion into for many years

3 – do it correctly and get a (virtual) dedicated server to handle the load and fix the issues.

Obviously I chose to walk away from it. NOT.

Initially, because it was cheap and had a good reputation, I  created a domain on a basic hosting platform using Siteground, they are a lot more techno than GoDaddy and I thought that it might me a risk going to them, but for under $20 it was worth a shot. Initially it seemed that this may just be the right host, PHP5 support would allow me to move to Gallery 3 and the control panel is full of cool stuff.

Problems with the database restore should have told me that I was wrong, but the curve-ball of getting an old backup from godaddy threw me and I pretty much worked through the night last night.

When I finally got the site up with the last two weeks of images loaded, the impact of the viewers from around the world simply swamped the poor host and they throttled the scripts.

In short they are a great host, but not tough enough, I will be using the hosting space for this blog and my personal sites once I have got my new host up and running.

So, my new host…..

Following some advice on fatwallet.com I am now the proprietor of an extremely cool Virtual Dedicated Server. I am not going to say where just yet as I want them to prove that they can handle the traffic before I give them a shout out, but I certainly got a ‘deal’.

Initial impressions are very, very very good indeed.

The MYSQL database that took ‘all night’ to create last night on siteground took 4 minutes and 15 seconds here.

Approximately 120 times faster.

FTP is uploaded at a solid 1.4MB vs 240KB on the old host.

But the best part is that we are no longer worried too much about processor and memory use unless we try to exceed the apportioned amounts.

So, the new URL that I am building on is cncpicsarchive.com all other domains point here.

As I intimated with the title of this post, I have learnt a lot during this fiasco.

Read More

cncpics – progress at last

It has been a long night.

As I write this it is close to 5am – I left the mysql upload running at 11pm last night and a couple of hours ago dragged myself out of bed to see what was happening. It is mostly good news, but it is also a little too early to celebrate.

The database is uploaded

The site actually works

FTP is running and uploading the pictures.

The bad news is that the FTP Process is going to take some time, some considerable time. As yet I have not ‘done the math’ but I would estimate that the upload is going to take a couple of days at least.

Secondly (more bad news) the image builds are taking longer than the page is permitting, therefore the first view of any image gives a blank, this has happened before, but I am not sure what the solution was at the time so I need to research it a little. It may be related to the stress that I am putting the server under by uploading images etc. But it also may be that the server is simply not up to the task in hand. There is only one way to find out I suppose.

Looking at the resource usage it shows that FTP is really consuming a lot of CPU – so maybe this is it. I am not sure.

I am going to let it upload for a while and sleep a little.

Check out the potential new site

http://mehpics.com

cncpics – Wrong database.

*sigh

Where to begin ?

Because I was in a catch-22 with Go-Daddy, I found a new fast, friendly host and created a new domain. Purely for testing purposes. The idea was that I could setup a clone of cncpics.com and then once it was all working, simply make some dns changes and all would be good.

Behind the scenes of cncpics.com there is a heavily customized version of an off-the-shelf gallery product which uses a mysql database to store data. It is not enough to simply grab a copy of the photographs and install a nice shiney new copy of gallery, because they I would need to add 45,000 photographs to the gallery and this is a heck of a task.

So the three components are the code, the photo’s and the database.

The code and the database are fairly easy to FTP from my old host, but because I am in ‘violation’ I was unable to get to the control panel to backup the databases.

So at 10AM I asked godaddy for a backup of the databases. The third person I spoke to told me that they where doing the job and that they were no ‘all there’. So I FTP’d them to my windows box and tried to upload them to the new host. Sadly there are a few hundred thousand records in some of the tables and the restore failed. So I jumped over to Linux land and found a tool that will no only connect to the remote database, but it will also load the data in much smaller chunks.

The downside is that this takes a long time and if there is any other traffic on the internet it is slower too.

Many hours later the database restore has finished and after a few minor tweeks to a configuration file the site worked.

Except the galleries ended in the middle of November 2009.

The database backup was several months old.

Somehow I was calm when I called godaddy, somehow I persuaded the support rep to snag a new backup and somehow it looks like he did a good job.

I am obviously now a long way behind, I need to leave the database upload running overnight and then most of tomorrow will be taken up with allowing the photographs to upload too.

Then it will be more configuration changes and more tweeking,

Hopefully on Friday I will be able to tell Godaddy that I no longer need their services….

CNCPICS – an update

Further to my logic catch22 yesterday, it looks like cncpics.com is going to be broken for a while.

Godaddy – the host of both that site and this blog (for the moment) are incapable of keeping a promise, they promised me late last night that they would not take the site off line and would give me time this morning to work through the issues.

The issues are fairly simple, for reasons unknown the caching mechanism is creating a lot of file in the, well, cache area. This is not a bad thing generally as the cache helps the performance.

However, the host has a strict 1024 file limit in any one folder. This limit was broken this week by a 925 picture collection – including the various derivatives the cache folders spiked up to 2-3000 files.

There is a simple answer, all I need is access to the gallery so that I can remove a bunch of photo’s from this week. Once this happens the cache growth will go away everything will return to normal. except this morning the account is suspended.

However, the issues is made a whole lot worse, very, very quickly. As soon as an admin notices that there are 1025 files in a folder, they instantly block access to the entire cache folder hierachy. Which leads the software to dump endless files in the $home folder, which leads to 16k files there, which leads to the account suspension.

Long story short, I need to re-host the site, I am done with godaddy.

I also cannot move the domains elsewhere for 60 day as I recently moved them to godaddy in a fit of ‘trying to keep everything together’.

I have signed up for a new more powerful host elsewhere and I am testing gallery 3

mehpics.com

The biggest issue of course is that uploading 50gb’s of images takes some time and regardless of where I find to host, the upload gap is going to be substantial.

So, my plan is thus

  • Upload the gallery code to the new host – done
  • Upload the 34mb database backup to the new host – in progress
  • Make the gallery work – meh
  • upload the images – in progress
  • convert to gallery 3  (optional step)

Beyond that I need to move this blog and gallery and a couple of others that share this hosting space.

Stay tuned for details…..

Access Denied – catch 22

This morning I sat having coffee with a friend when my phone displayed a voicemail

Hi. This is Amanda calling from go daddy hosting at (480) 505-8712
to say we sent you an email regarding your hosting account
recipient keep X dot in for which requires your attention. Thank you.

As this is not the first such call I have had from them I waited until I got home to deal with it.

It appears that the cache had built up to the point of impacting performance, again.

So they simply blocked access to my hosting account.

The first call had the following logic problem

We cannot allow you access to your account until the cache is deleted

But I cannot delete the cache without access to the account

We cannot allow you access to your account until the cache is deleted

But……..

As you can see, this loop has no end and the first support rep was adamant that I needed to delete the files before they would unlock the domain.

The second guy was better, he wanted to create an archive of all of the files that were in the problem folders for me.

this would take a few minutes.

a couple of hours elapsed and I got hold of an exceptionally feisty young man that basically called me a liar and suggested that I needed a dedicated server

Finally I got hold of a much more sensible person called ‘Jared’ that was able to tell me what the files were and to get the ones that I no longer needed, deleted.

i am now waiting for the site to come back and/or more logic errors – I am not sure which will happen though….