A rather dull worKLOG. This is just a scratchpad for solutions to IT problems that might be useful to someone else. Expect no opinions, no brilliant insights and definitely no pictures of pets or children. Expect stack traces, code snippets and other hints for the Google Indexer.

Monday, August 13, 2007

Parallels screws up Windows

Since Parallels 3 has come out there have been a lot of complaints that it can screw up your Windows XP installation, resulting in an error like:

/system32/ntoskrnl.exe file is missing

Parallels has finally posted a fix:
http://kb.parallels.com/entry/30/516/

However, it doesn't tell you how to get your Parallels VM to boot from the Windows CD in the first place. The trick is to start the VM, then go to edit/Virtual Machine, and in there you'll find an option to change the Boot order to CD first.

In my case, this only partially fixed the problem, perhaps because I'd previously botched things by following other advice on the internet. The solution for me was to completely deinstall parallels (using the _installer_, weirdly), then reinstalling it. Be warned that this might then need you to reactivate your windows installation.

J1 visas at the London US embassy

The US embassy website at http://london.usembassy.gov makes applying for a J1 visa appear to be an intimidating and unpleasant process. It's big on the "don'ts" - Don't bring a mobile phone, don't expect to be allowed inside with a bag, don't arrive too early or too late. Then there's the forms: half a dead tree's worth, requiring you to list your experience in nuclear, chemical and biological weapons, every foreign country you've been to in the last ten years (the form was designed by an American...right? They only give you a box big enough to write "Mexico and Canada".)
With a start like that, it's hardly surprising that the actual experience is, in fact, a whole lot better, at least if you're white, middle class and British.

It's still expensive though: for my family of 3 the bill was:
$100 X 3 visa fees
$100 SEVIS fee
£15 mail courier
£170 transport to, from and around london
£15 phone call to make the appointments
------
£400

So, it's worth getting the paperwork right first time.

Ingredients for a J1 (and associated J2) visas:
  • DS2019 forms for everyone - these should be provided by your sponsor
  • An appointment at the Embassy in London (the Consulate in Edinburgh doesn't do visas. I wonder what they DO do.)
    • The appointments are booked via a phone line at £1.20 per minute
  • After you make the appointment, you will receive via email an appointment letter and a payment receipt for each member of your family
  • Visa application forms for everybody - this is form DS156, which you complete online, and then print out.
  • Contact forms DS158 for everybody. This is a PDF file that you print out, and then complete (unless you're lucky enough to have a PDF editor).
  • Form DS157 for males of military age (and assorted other categories such as scientists). This is where you have fun listing your chemical warfare experience. It's also a PDF that you print out.
  • A SEVIS fee receipt - this is a receipt that you've coughed up the $100 to take part in the program. You can print this off from the web after you've filled in the online form. Note that they don't tell you that you can use this as official proof until you've already agreed to pay an additional $30 to have the certificate couriered over from the US.
  • passports for all the family
  • One passport photo for each family member.

What happens at the embassy
You show up at the embassy shortly before the time on your appointment letter and join a queue. There's one queue (red) for US citizens doing US-citizen things and one queue (yellow) for the rest of us (though when I was there, there was noone in the red queue). At some point a friendly english girl comes down the line and checks your appointment letter, crosses you off a list of expected people, and ushers you on to the next queue at the security checkpoint. At around your allotted time you are waved forward into the checkpoint, which is a portacabin containing an airport style xray machine and a cheery Australian chap. Your documents are looked at again, and you have to put your stuff through the xray machine. There's nothing here that's any more scary than going into an airport departure lounge. Off you trot round the outside of the embassy following the "visas" sign, up some steps and to reception, where your appointment letters are checked again, and you're given a sticker with a number on it. You go into a waiting room and, well, wait. Eventually your number is called and you are told to go to a counter window, where all your documents are taken and examined. These booths are also manned by surprisingly friendly (since it must be a tedious job) Brits and Australians. At this point you get to play with the finger print machine. You're given a blue form with your sticker attached and told to go and sit down and wait again. Tum-de-tum. After an hour or so you get called again, this time to the booths down the corridor for your "interview". If you were expecting an interview behind closed doors with a guy with a rubber hose and a desk light then you'll be pleasantly surprised. Again, the interview takes place at a counter window, in my case with a charming and friendly girl who ought to have been working for the tourist board, not the embassy. This job only seems to be trusted to Americans though, since they presumably have a bigger incentive to keep ne'er-do-wells out of their country. You will be asked about your research project, but we're talking about delivering your elevator pitch here...it's not a viva exam. Click click, stamp stamp. Your visa has been granted. The whole interview took about 5 minutes. Including waiting time, the process takes about 2 hours (though I was told it was a particularly light day). From the order the numbers were being called it was clear that some people had been waiting considerably longer, presumably for being too muslim, too dark skinned or having too interesting a police record. Finally, you take your blue form to a counter at the back of the waiting room where you hand over £15 pounds to pay for the safe return of your passports. I guess you could skip this bit, but it would render the trip a little pointless.

Frequently Asked Questions
Well, I was was asking them, anyway.
  • How long does the appointment phone call last?
    • At £1.20 a minute, you'd expect them to milk it for all the cash they could get, but mine proceeded at such a rapid pace you'd think it was them paying, not me. Everything was done in about 10 minutes.
  • Do children need to attend?
    • Apparently, children under 14 don't need to go.
  • Can children attend?
    • Yes. They advise that the embassy isn't the most fun place in the world for a young child, but you're not prohibited from taking child + pram into the embassy.
      The pram will probably need to go through the xray though.
  • What about my mobile phone, key fob etc....?
    • The website says these aren't allowed. It's lying. When you go through security they'll be taken off you and stored till you return.
  • The website says that the visa photos need to be 2 inches square.
    • Normal UK passport sized photos seem to be acceptable.
  • Are food and drink allowed in?
    • They say that you can't take liquids in, and I've no reason to doubt this. However, there's a cafe inside that serves tea, coffee and pop. Beware that it closes at 3ish though.
  • Can I take a bag in?
    • The website says that you can't take a large rucksack in....I think they mean one of those 70 litre jobs. I took in a reasonably large briefcase. From the setup I expect that they'll tolerate anything you could take as hand luggage on a plane, though their patience might be more limited on busy days.

Thursday, August 02, 2007

Faster pussycat

"...but Java's so slow...."

1) Java's bytecodes are interpreted by a virtual machine, therefore Java can never be as fast as a statically compiled language such as C
2) Java's VM can perform dynamic optimization in ways not available to a statically optimizing C compiler, therefore Java can outperform C.

The web is full of opinions on which is the fastest language and why. There are plenty of benchmarks proving once and for all one view or the other. However, you never really believe it unless you've tried it yourself. So here, for the first time ever is conclusive proof that really there ain't much in it one way or another.

My completely unscientific test case: an algorithm to sum the first N Fibonacci numbers. The numbers themselves are calculated by a deliberately naive recursive algorithm - don't try this at home. The running time grows exponentially with N. However, it is easy to get a good range of running times. The algorithm can be coded almost identically, line for line, in my 3 target languages of Java, C, Python.

Here are the results:










NJava (client vm)Java (server vm)C (gcc standard options)C (gcc -O3)Python
00.20.2000
100.20.2000
200.20.2000
300.30.2001.4
350.50.40.215
402.54.81.8171
45*24.520.65219.8
50272232600227


*For N>=45 the result overflows when using 32 bit integers. Here's some results using Java longs and C long longs (64 bits):





NJava (server vm)C (gcc -O3)
452123.5
50221265


No doubt you can find algorithms (and better C compilers) that change the results. But change them enough for you to care?



Details:
The tests were run on Ubuntu Linux running in a Parallels VM on a 2007 model Macbook Pro with a couple of gig of RAM.
The Java code was compiled and run with Java 6, the C code with gcc, and the Python code run with Python 2.4.3
Times are the sum of user and system time in secs, recorded to the nearest 0.1 of a sec, and the tests were run a handful of times until I reckoned the time looked about right. Times include any VM startup times.

Thursday, June 07, 2007

Boxplots for beginners

How to create Boxplots in R
Refs:
http://en.wikipedia.org/wiki/Boxplot
http://www.r-project.org/

Suppose you have data such as:
number_of_threads thread time
1 1 10
2 1 20.5
2 2 19.5
3 1 30
3 2 25
3 3 35

This table represents made up data describing some system testing, where the system is exercised with 1,2,3... concurrent users. You want to produce a boxplot showing the timings for each number of concurrent users. Here's the magic incantation in R:
Assuming the above file is saved to data.txt
1) Use R's Misc menu to change to the correct working directory
2) Load the data into a dataframe with
myresults = read.table("data.txt", header=TRUE)
3) Generate the boxplot with
boxplot( time ~ number_of_threads, myresults)

To jazz it up with some axis labels:

boxplot( time ~ number_of_threads, myresults, xlab="Concurrent users", ylab="time")

Adding several boxplots on the same axis:
boxplot( time ~ number_of_threads, myresults, add=TRUE)
*Warning* - if you don't see your new boxplot, then it's because for some nutty reason the plot doesn't automatically rescale, and you need to set the scale explicitly with ylim=c(0,40) or similar.
Making the boxes a particular colour:
boxplot( time ~ number_of_threads, myresults, col="yellow")
and adding a legend:
legend(2,9m c("plot1","plot2"), fill=c("red","yellow"))

Wednesday, June 06, 2007

http://www.flickr.com/photos/josiefraser/532394229/in/set-72157600317386945/

Our Emerge slide.

Tuesday, February 06, 2007

Converting jpgs to ps or eps

A bit of googling finds a number of commercial and shareware packages that will do this task, but you can also do:

jpegtopnm MyDog.jpg | pnmtops > MyDog.ps

on a unix machine.

http://en.wikipedia.org/wiki/Portable_Pixmap_file_format

Wednesday, January 31, 2007

Moving mail to gmail

I've decided to move completely over to gmail so that I don't have to configure my mail clients to deal with umpteen different accounts. I also want to store all my legacy mail so it's all searchable in the one place. How do I upload my Thunderbird mail files to gmail?
This: http://www.marklyon.org/gmail/download.htm seems to do the job. It's a small python app that goes through your Thunderbird mbox files and forwards them all. It's slow, and it's going to mess up your emails' timestamps, but it does the job.

Friday, January 19, 2007

Weekly summary

* Helping Isa debug PLASTIC functionality in VOSpec

http://esavo02.esac.esa.int:8080/vospec/
* Prep for MSc
* Installed latest STILTS
* Looking into an example for VisIVO paper
* New Plastic group in Apps WG
* Uni of Queensland Connection


--
------------------------------------------------------------------------
AstroGrid/VOTech
&
WFAU, Institute for Astronomy, Edinburgh
Skype:johndavidtaylor <skype:johndavidtaylor?chat>

------------------------------------------------------------------------
*Gratuitous advertising:*
Plastic - http://plastic.sourceforge.net | AstroRuntime -

http://www2.astrogrid.org/desktop
AstroGrid - http://www.astrogrid.org | WFAU -

http://www.roe.ac.uk/ifa/wfau/

Thursday, January 11, 2007

How can a bash or shell script find its own directory?

It's often desirable to write a small bash script to launch an
application. However, you want the application to be portable, so you
don't want to hard-wire absolute file locations into the script.
Unfortunately, this means that if you execute the script from somewhere
other than the directory which holds the script, then it won't locate
the files. A bit of googling suggests that the general problem of
making a script aware of the directory in which it resides is quite
hard. However, the following works for me:
cwd=`dirname $0`
old=$pwd
cd $cwd
#launch your application here, for example
nohup python main.py &
#where main.py is in the same directory as the script
cd $old

Subclipse, Eclipse plugin does not work on Ubuntu

You've installed Eclipse, the subclipse subversion plugin, set up your
repository locations and are ready to go. Left click on your favourite
subversion repository to see the contents. Uh-oh:
'
Folder " does not exist remotely.
'

This is because the JVM that ships with Ubuntu (GNU gij) doesn't even
remotely work.
OK that's very unfair. It does seem to screw up Eclipse and Subclipse
though.
To fix this problem, you need to download the latest Sun JDK install it
somewhere convenient, and (importantly) make sure that this JDK's bin
folder is on the front of your path. If you've got it set correctly
then executing java -version at the command line should give you the
reassuring
"
java version "1.6.0"
....
java HotSpot(TM) Client VM.....

"

Thursday, January 04, 2007

"Multiproject" site generation in Maven 2.0.4

Yet again Maven has shat upon me from on high by randomly updating a
plugin and changing its behaviour. Previously, if you had a project
configured as a "multiproject" build of modules, the way to include a
link in your parent project's docs to the child documentation was to add
${modules} into the parent's site.xml. No longer! Try this now, and
your mvn will spit out some garbage about your site.xml being malformed
xml, when it plainly isn't. No, now the secret incantation is to put a
menu item in thus: <menu ref="modules"/>. The same goes for ${reports}
and ${parent} presumably. The new syntax is certainly nicer, if you
don't mind the extra wear and tear on your fingers, but I wish they
had just deprecated the old syntax, rather than secretly removing it.

Problems with Image loading in Java


I'm trying to load images in Java for use in icons with the following:
new ImageIconr(url)

Mostly, this works fine. But just once in a while it fails with:

Uncaught error fetching image:
java.lang.SecurityException
at java.lang.SecurityManager.checkPermission(SecurityManager.java:562)
at java.lang.SecurityManager.checkConnect(SecurityManager.java:1078)
at sun.awt.image.URLImageSource.checkSecurity(URLImageSource.java:81)
at sun.awt.image.ImageRepresentation.imageComplete(ImageRepresentation.j
ava:597)
at sun.awt.image.InputStreamImageSource.errorConsumer(InputStreamImageSo
urce.java:131)
at sun.awt.image.InputStreamImageSource.setDecoder(InputStreamImageSourc
e.java:331)
at sun.awt.image.InputStreamImageSource.doFetch(InputStreamImageSource.j
ava:252)
at sun.awt.image.ImageFetcher.fetchloop(ImageFetcher.java:172)
at sun.awt.image.ImageFetcher.run(ImageFetcher.java:136)

What's worse, surrounding the code with a try-catch block doesn't save you - the application simply locks up. Note that this
only happens when running under a SecurityManager such as under WebStart. It seems there's a bug in the JDK
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4881229
which means that if the URL gets redirected with an http 302, then an internal thread in the JDK will fail. This
is pretty hopeless - can't fix it, can't trap it.

The workaround is to use a different part of the API. Instead of constructing an Icon from a URL, you first get an Image using
Image im = ImageIO.read(url);
then construct an icon from that:
new Icon(im);

Wednesday, December 13, 2006

Subversion on sourceforge refuses to make copies

If you find that SourceForge's subversion system is buggered (again) and
refuses to let you make copies, it could be because you hadn't noticed
their not-very-well-advertised switch of access URL to:

"On November 31, 2006 the access method for Subversion changed. This
document reflects those changes. The old method had numerous problems,
including spurious 50x error messages and other issues that kept it from
functioning fully. This newly documented access method solves many, if
not all of the issues with the old mechanism.

Users of the old method
(https://svn.sourceforge.net/svnroot/PROJECTNAME) should switch to the
new access method
(https://PROJECTNAME.svn.sourceforge.net/svnroot/PROJECTNAME) using
these steps:

1. Make a copy of your local working copy.
2. Run 'svn info' at the root of the repository content, it should
display a line that appears similar to: URL:
https://svn.sourceforge.net/svnroot/PROJECTNAME/trunk
3. Run the following command at the root of the working copy: svn
switch --relocate
https://svn.sourceforge.net/svnroot/PROJECTNAME/trunk
https://PROJECTNAME.svn.sourceforge.net/svnroot/PROJECTNAME/trunk

SVN is a complex tool."

Right, and it doesn't get any easier when SourceForge keep arseing about
with it.

http://sourceforge.net/docman/display_doc.php?docid=31070&group_id=1#notice

Monday, December 11, 2006

AstroGrid registry bug with namespaces

We've recently upgrade our WFAU publishing registry to version Version
2006.3.03r. This has introduced a bug where previously registered
entries cannot be edited. When you submit the edits you get a:

env:ServerServer Error: No namespace defined for prefix vororg.exist.xmldb.LocalXPathQueryService.compile(LocalXPathQueryService.java:213) org.exist.xmldb.LocalXPathQueryService.doQuery(LocalXPathQueryService.java:280) org.exist.xmldb.LocalXPathQueryService.query(LocalXPathQueryService.java:112) org.exist.xmldb.LocalXPathQueryService.query(LocalXPathQueryService.java:103) org.astrogrid.xmldb.client.XMLDBServiceImpl.queryXQuery(XMLDBServiceImpl.java:235) org.astrogrid.registry.server.xmldb.XMLDBRegistry.query(XMLDBRegistry.java:71) org.astrogrid.registry.server.admin.AuthorityListManager.getManagedAuthorities(AuthorityListManager.java:118) org.astrogrid.registry.server.admin.AuthorityListManager.populateManagedMaps(AuthorityListManager.java:106) org.astrogrid.registry.server.admin.RegistryAdminService.updateResource(RegistryAdminService.java:282) org.apache.jsp.admin.addResourceEntry_jsp._jspService(addResourceEntry_jsp.java:488) org.apache.jasper.runtime.HttpJspBase.service(H
ttpJspBase.java:94) javax.servlet.http.HttpServlet.service(HttpServlet.java:802) org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324) org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292) org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236) javax.servlet.http.HttpServlet.service(HttpServlet.java:802) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp
Valve.java:214) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) org.apache.catal
ina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.c
atalina.core.ContainerBase.invoke(ContainerBase.java:929) org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) java.lang.Thread.run(Thread.java:595)

The fix is to copy the
xmlns:vr="http://www.ivoa.net/xml/VOResource/v0.10" from the
vor:Resource into the root vor:VOResources element.

VOMon - the VO service monitor has been released:

VOMon - the VO service monitor has been released:

http://vomon.sourceforge.net

VOMon is a mashup of the Astro Runtime
<http://www2.astrogrid.org/desktop/astro-runtime> and MARS
<http://leapfrog-mars.sourceforge.net/>, glued together with Python.
VOMon uses the Astro Runtime to query a Registry to obtain all the
services that match an admin-specified query. By default it queries the
central AstroGrid registry, but can be configured to query any
IVOA-compliant registry. This list of services is updated every hour or
so, so that VOMon will always have a reasonably up to date list of
services to monitor. The services are passed to MARS which does the
actual monitoring. Currently this monitoring consists of simply
pinging the accessURL and checking the response. Some services such as
Cone and SIAP, have default parameters appended to the accessURL to
ensure they do not fail. Once the VOSI interface is widely supported,
VOMon will be upgraded to use that. VOMon can either output a report to
be hosted on a web server, or present the user with a GUI interface.

An instance has been deployed at:
http://thor.roe.ac.uk/vomon/status.xml

Tuesday, December 05, 2006

Weekly Summary


* Prepared proposal for JISC CoP participation
* Plastic Website maintenance http://plastic.sourceforge.net and
general email business
* Finally, published IVOA Plastic Note:
http://ivoa.net/Documents/latest/PlasticDesktopInterop.html

Wednesday, November 15, 2006

Weekly update 15/11/06

* Submitted ADASS paper
* Currently moving the fastem algorithm off hempriggs. In the process
I'm upgrading to a new version supplied by CMU that doesn't require user
intervention when things go wrong, thus making the web service more
reliable. Unfortunately, this new version has a slightly different
interface, so it's not a straight drop in replacement.
* Given access to all CMU's algorithms
* Tony has asked me to lead a bid to join the U&I CofP. Email
attached. Deadline is next week, shouldn't take too much work.
Co-conspirators are Elizabeth A and Norman.
* Upgraded the Workbench Plastic library and made some fixes

Thursday, November 09, 2006

Cygwin/X - logging into remote linux machines from Windows

Logging into linux machines from Windows can be done in various ways,
such as using ssh via PuTTY. In alternative is to use Cygwin, which not
only gives you a fairly complete Linux emulation on your windows box,
but also makes it easy to use X-forwarding, and thus use remote
GUI-based applications.

Step 1) Install Cygwin. The installer will give you a choice of
packages to install. Make sure you install the x11 packages
Step 2) Start the x server by running [cygwin
installation]\usr\X11R6\bin\startxwin.bat
Step 3) From the resulting xterm window, log into the remote machine
with ssh -X (remote machine name)
Any remotely started GUI apps should now magically appear on your
Windows box.

If you get an error of the form
warning, error event received:
X Error of failed request: BadAtom (invalid Atom parameter)
Major opcode of failed request: 18 (X_ChangeProperty)
Atom id in failed request: 0x1b3
Serial number of failed request: 944
Current serial number in output stream: 946

when you try to start the GUI app, try using -Y instead of -X.

Wednesday, November 08, 2006

Increasing the memory available to Eclipse.

To increase the memory allocated to the JVM for Eclipse, edit the
eclipse.ini file in your Eclipse directory to be something like:
-vmargs -Xms40m -Xmx512m