A rather dull worKLOG. This is just a scratchpad for solutions to IT problems that might be useful to someone else. Expect no opinions, no brilliant insights and definitely no pictures of pets or children. Expect stack traces, code snippets and other hints for the Google Indexer.

Wednesday, December 13, 2006

Subversion on sourceforge refuses to make copies

If you find that SourceForge's subversion system is buggered (again) and
refuses to let you make copies, it could be because you hadn't noticed
their not-very-well-advertised switch of access URL to:

"On November 31, 2006 the access method for Subversion changed. This
document reflects those changes. The old method had numerous problems,
including spurious 50x error messages and other issues that kept it from
functioning fully. This newly documented access method solves many, if
not all of the issues with the old mechanism.

Users of the old method
(https://svn.sourceforge.net/svnroot/PROJECTNAME) should switch to the
new access method
(https://PROJECTNAME.svn.sourceforge.net/svnroot/PROJECTNAME) using
these steps:

1. Make a copy of your local working copy.
2. Run 'svn info' at the root of the repository content, it should
display a line that appears similar to: URL:
https://svn.sourceforge.net/svnroot/PROJECTNAME/trunk
3. Run the following command at the root of the working copy: svn
switch --relocate
https://svn.sourceforge.net/svnroot/PROJECTNAME/trunk
https://PROJECTNAME.svn.sourceforge.net/svnroot/PROJECTNAME/trunk

SVN is a complex tool."

Right, and it doesn't get any easier when SourceForge keep arseing about
with it.

http://sourceforge.net/docman/display_doc.php?docid=31070&group_id=1#notice

Monday, December 11, 2006

AstroGrid registry bug with namespaces

We've recently upgrade our WFAU publishing registry to version Version
2006.3.03r. This has introduced a bug where previously registered
entries cannot be edited. When you submit the edits you get a:

env:ServerServer Error: No namespace defined for prefix vororg.exist.xmldb.LocalXPathQueryService.compile(LocalXPathQueryService.java:213) org.exist.xmldb.LocalXPathQueryService.doQuery(LocalXPathQueryService.java:280) org.exist.xmldb.LocalXPathQueryService.query(LocalXPathQueryService.java:112) org.exist.xmldb.LocalXPathQueryService.query(LocalXPathQueryService.java:103) org.astrogrid.xmldb.client.XMLDBServiceImpl.queryXQuery(XMLDBServiceImpl.java:235) org.astrogrid.registry.server.xmldb.XMLDBRegistry.query(XMLDBRegistry.java:71) org.astrogrid.registry.server.admin.AuthorityListManager.getManagedAuthorities(AuthorityListManager.java:118) org.astrogrid.registry.server.admin.AuthorityListManager.populateManagedMaps(AuthorityListManager.java:106) org.astrogrid.registry.server.admin.RegistryAdminService.updateResource(RegistryAdminService.java:282) org.apache.jsp.admin.addResourceEntry_jsp._jspService(addResourceEntry_jsp.java:488) org.apache.jasper.runtime.HttpJspBase.service(H
ttpJspBase.java:94) javax.servlet.http.HttpServlet.service(HttpServlet.java:802) org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:324) org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:292) org.apache.jasper.servlet.JspServlet.service(JspServlet.java:236) javax.servlet.http.HttpServlet.service(HttpServlet.java:802) org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:237) org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:157) org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapp
Valve.java:214) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:198) org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:152) org.apache.catal
ina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:540) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:118) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:102) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) org.apache.catalina.core.StandardValveContext.invokeNext(StandardValveContext.java:104) org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:520) org.apache.c
atalina.core.ContainerBase.invoke(ContainerBase.java:929) org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:160) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:799) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:705) org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:577) org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683) java.lang.Thread.run(Thread.java:595)

The fix is to copy the
xmlns:vr="http://www.ivoa.net/xml/VOResource/v0.10" from the
vor:Resource into the root vor:VOResources element.

VOMon - the VO service monitor has been released:

VOMon - the VO service monitor has been released:

http://vomon.sourceforge.net

VOMon is a mashup of the Astro Runtime
<http://www2.astrogrid.org/desktop/astro-runtime> and MARS
<http://leapfrog-mars.sourceforge.net/>, glued together with Python.
VOMon uses the Astro Runtime to query a Registry to obtain all the
services that match an admin-specified query. By default it queries the
central AstroGrid registry, but can be configured to query any
IVOA-compliant registry. This list of services is updated every hour or
so, so that VOMon will always have a reasonably up to date list of
services to monitor. The services are passed to MARS which does the
actual monitoring. Currently this monitoring consists of simply
pinging the accessURL and checking the response. Some services such as
Cone and SIAP, have default parameters appended to the accessURL to
ensure they do not fail. Once the VOSI interface is widely supported,
VOMon will be upgraded to use that. VOMon can either output a report to
be hosted on a web server, or present the user with a GUI interface.

An instance has been deployed at:
http://thor.roe.ac.uk/vomon/status.xml

Tuesday, December 05, 2006

Weekly Summary


* Prepared proposal for JISC CoP participation
* Plastic Website maintenance http://plastic.sourceforge.net and
general email business
* Finally, published IVOA Plastic Note:
http://ivoa.net/Documents/latest/PlasticDesktopInterop.html

Wednesday, November 15, 2006

Weekly update 15/11/06

* Submitted ADASS paper
* Currently moving the fastem algorithm off hempriggs. In the process
I'm upgrading to a new version supplied by CMU that doesn't require user
intervention when things go wrong, thus making the web service more
reliable. Unfortunately, this new version has a slightly different
interface, so it's not a straight drop in replacement.
* Given access to all CMU's algorithms
* Tony has asked me to lead a bid to join the U&I CofP. Email
attached. Deadline is next week, shouldn't take too much work.
Co-conspirators are Elizabeth A and Norman.
* Upgraded the Workbench Plastic library and made some fixes

Thursday, November 09, 2006

Cygwin/X - logging into remote linux machines from Windows

Logging into linux machines from Windows can be done in various ways,
such as using ssh via PuTTY. In alternative is to use Cygwin, which not
only gives you a fairly complete Linux emulation on your windows box,
but also makes it easy to use X-forwarding, and thus use remote
GUI-based applications.

Step 1) Install Cygwin. The installer will give you a choice of
packages to install. Make sure you install the x11 packages
Step 2) Start the x server by running [cygwin
installation]\usr\X11R6\bin\startxwin.bat
Step 3) From the resulting xterm window, log into the remote machine
with ssh -X (remote machine name)
Any remotely started GUI apps should now magically appear on your
Windows box.

If you get an error of the form
warning, error event received:
X Error of failed request: BadAtom (invalid Atom parameter)
Major opcode of failed request: 18 (X_ChangeProperty)
Atom id in failed request: 0x1b3
Serial number of failed request: 944
Current serial number in output stream: 946

when you try to start the GUI app, try using -Y instead of -X.

Wednesday, November 08, 2006

Increasing the memory available to Eclipse.

To increase the memory allocated to the JVM for Eclipse, edit the
eclipse.ini file in your Eclipse directory to be something like:
-vmargs -Xms40m -Xmx512m

Weekly Summary

AG@WFAU Operations
=================
* Helped test ukidss
* Suggested compression as a solution to the large metadoc file problem.
* Moved test-registry
* China-AGVO operational
Datamining
=======
* Swotting up on datamining
* Been in touch with Andrew Moore about fastem - applied for access to
software

Plastic
====
* ADASS followup
* Been in touch with Ruby people
* TKH unavailable
* Contacted Jason about site spruce up but heard nothing yet

Other
=====
* Preparing Eclipse lecture
* Setting up my Eclipse to do C++

AOB
deadline for adass manuscripts is Monday

Tuesday, November 07, 2006

Setting up C++ in Eclipse on Windows

Although Eclipse works most naturally as a Java IDE, it doesn't do too
badly for other languages, and has the advantage that you don't need to
learn multiple IDEs and can keep all your code together in a single
environment.
Steps in setting up C++ in Eclipse, on Windows:
* If it's not already installed, get the CDT plugin - this is available
as part of the standard Callisto bundle from the Eclipse website.
(help/software updates/find and install/add new remote site
http://download.eclipse.org/tools/cdt/releases/callisto)
* The CDT doesn't come with its own compilers etc - you'll need to
download them separately. One option is to install Cygwin, making sure
that you're ticked all the boxes to get make and gcc. Important: don't
forget to add cygwin/bin to your windows path. The problem is, that any
executables you then make will need the cygwin1.dll to be bundled with
them, which is hardly ideal.
* An alternative, mentioned in
http://wiki.fredlackey.com/pmwiki.php?n=HelpfulInfo.DevelopmentEclipseCppCofnig
and others is to install MinGW from
http://prdownloads.sourceforge.net/mingw/MinGW-5.0.3.exe?download which
will produce exes with no dependencies. Again, make sure it's on your path.
* Managed vs Standard projects. If you create a managed project then
Eclipse will generate a simple makefile for you, otherwise you are
expected to create a makefile yourself, and will spend many happy hours
hitting the "build" button and wondering why nothing happens.

Monday, November 06, 2006

xml-rpc from C++ on Windows pt I

As part of the Plastic http://plastic.sourceforge.net project I'm
looking at making xml-rpc calls from different languages. Current focus
is on C/C++ since it opens up the possibility of using SWIG to get
Plastic access from a large range of other languages "for free".

My setup:
OS: Windows XP
IDE: Eclipse, with the CDT plugin
Tools: g++ and make, as part of the MinGW and MSYS systems

The most promising xml-rpc library is this one:
http://xmlrpcpp.sourceforge.net/links.html. Promising because a) I've
got it working before b) It has no dependencies c) It does client and
server (necessary for a fully-fledged plastic application).

Steps
1) Download and install the xml-rpc library files into a suitable folder.
2) Add this folder to the list of source folders in project
properties/c++ build/Tool settings/compiler/directories
3) Add the _WINDOWS flag to the compiler/preprocessor
If you hit build now, you'll get linker errors as listed below. The
problem is that although you get the winsock header files included on
the path automatically, you need to explicitly specify the corresponding
library to the linker. The library in question is probably called
something like: libws2_32.a and can be found in the MinGW/lib folder.
4) project properties/c++ build/Tool settings/linker/libraries/ add
library ..... and here's the real doozy....you add ws2_32 (!!!not!!!
libws2_32.a .... the linker prepends the lib and adds the .a for you).

Hit build, and you're away. All the above assumes that you have already
written your xml-rpc client of course....

=================================================================
The linker errors (for google's benefit):
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:45:
undefined reference to `WSAStartup@8'
./xmlrpc/XmlRpcSocket.o(.text+0x5d): In function
`ZN6XmlRpc12XmlRpcSocket6socketEv':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:80:
undefined reference to `socket@12'
./xmlrpc/XmlRpcSocket.o(.text+0x8e): In function
`ZN6XmlRpc12XmlRpcSocket5closeEi':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:89:
undefined reference to `closesocket@4'
./xmlrpc/XmlRpcSocket.o(.text+0xbc): In function
`ZN6XmlRpc12XmlRpcSocket14setNonBlockingEi':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:103:
undefined reference to `ioctlsocket@12'
./xmlrpc/XmlRpcSocket.o(.text+0x10b): In function
`ZN6XmlRpc12XmlRpcSocket12setReuseAddrEi':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:115:
undefined reference to `setsockopt@20'
./xmlrpc/XmlRpcSocket.o(.text+0x157): In function
`ZN6XmlRpc12XmlRpcSocket4bindEii':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:126:
undefined reference to `htonl@4'
./xmlrpc/XmlRpcSocket.o(.text+0x16b):E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:127:
undefined reference to `htons@4'
./xmlrpc/XmlRpcSocket.o(.text+0x18c):E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:128:
undefined reference to `bind@12'
./xmlrpc/XmlRpcSocket.o(.text+0x1bc): In function
`ZN6XmlRpc12XmlRpcSocket6listenEii':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:136:
undefined reference to `listen@8'
./xmlrpc/XmlRpcSocket.o(.text+0x1fa): In function
`ZN6XmlRpc12XmlRpcSocket6acceptEi':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:151:
undefined reference to `accept@12'
./xmlrpc/XmlRpcSocket.o(.text+0x23a): In function
`ZN6XmlRpc12XmlRpcSocket7connectEiRSsi':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:164:
undefined reference to `gethostbyname@4'
./xmlrpc/XmlRpcSocket.o(.text+0x290):E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:169:
undefined reference to `htons@4'
./xmlrpc/XmlRpcSocket.o(.text+0x2b1):E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:173:
undefined reference to `connect@12'
./xmlrpc/XmlRpcSocket.o(.text+0x383): In function
`ZN6XmlRpc12XmlRpcSocket6nbReadEiRSsPb':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:191:
undefined reference to `recv@16'
./xmlrpc/XmlRpcSocket.o(.text+0x49e): In function
`ZN6XmlRpc12XmlRpcSocket7nbWriteEiRSsPi':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:223:
undefined reference to `send@16'
./xmlrpc/XmlRpcSocket.o(.text+0x517): In function
`ZN6XmlRpc12XmlRpcSocket8getErrorEv':
E:/jdt/data/experiments/PlasticListener2/Debug/../xmlrpc/XmlRpcSocket.cpp:248:
undefined reference to `WSAGetLastError@0'

--
------------------------------------------------------------------------
AstroGrid/VOTech
&
WFAU, Institute for Astronomy, Edinburgh
Skype:johndavidtaylor <skype:johndavidtaylor?chat>

------------------------------------------------------------------------
*Gratuitous advertising:*
Plastic - http://plastic.sourceforge.net | AstroRuntime -
http://www2.astrogrid.org/desktop
AstroGrid - http://www.astrogrid.org | WFAU -
http://www.roe.ac.uk/ifa/wfau/

Wednesday, November 01, 2006

Weekly Summary 1/11/06

  • Debate about coverage in registry and possible solutions
  • Visit to Cambridge to cover Taverna
    • Link to Guy's summary
    • Side debate on merits of SourceForge over managing our own code
  • Discovered that with the exception of 2MASS, our cone searches aren't showing in AstroScope - should do soon following redeploy
  • Discovered many missing services from AstroGrid's registry...Kevin fixing....
  • Swotting up on Data Mining
  • Downloaded VOTechBroker
  • Following up ADASS

Tuesday, October 31, 2006

Weasl - Wfau Edinburgh Astro Service Locator

At the last IVOA meeting in Russia, David Schade argued that "position
is the least important attribute of an object". He was talking about
astronomical objects of course, but it's probably true of VO services
too. Nevertheless, I wrote this:

http://thor.roe.ac.uk/weasl/index.html

It's built from the Astro Server Runtime, a google map, a GeoEncoder
service and some sticky tape.

It's not particularly slick at the moment - the server just writes out a webpage with all the javascript required to place the points on the map. It would be far better to make it more AJAXy and have the webpage request the points. Could make it a lot more dynamic then.

Gotchas
=======
None really. The ASR made it very easy, and Google is, of course, wonderful.
Registry Access
---------------
All done through the ASR. The trick was to check the type of all the returned Resources to see if it's a Service. Only then do you get access to the attributes you need.
Goole Map
---------
The map javascript is pretty good, though it gets a bit unresponsive above a hundred or so points. One gotcha is that the <div> that contains the
map worked fine as
<div id="map" height="" width=""></div>
but if you compressed the tag into
<div id="map" height="" width=""/>
the browser nested any following divs inside it, with hilarious consequences.

Friday, October 27, 2006

Getting Java Server Pages up and running in Tomcat

Trying to get JSF up and running to use as a front end to WEASL (coming
soon). Huge problems with missing classes when running the latest JSF
(1.2) in Tomcat 5.0.28, so fallen back to 1.1. Apparently 1.2 requires
a version of JSP that Tomcat doesn't yet support. Trying out the
examples in Core JSF (http://www.horstmann.com/corejsf/)
but got an NPE;
27-Oct-2006 00:30:41 org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Allocate exception for servlet Faces Servlet
java.lang.NullPointerException
at javax.faces.webapp.FacesServlet.init(FacesServlet.java:165)
at
org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1105)
at
org.apache.catalina.core.StandardWrapper.allocate(StandardWrapper.java:757)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:130)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

The solution to this is simply to remove the working directory from
under Tomcat and restart the webapp. All will then be well.

Wednesday, October 25, 2006

Two weekly summary

* ADASS
* 6DF database problem (see bugzilla
http://www.astrogrid.org/bugzilla/show_bug.cgi?id=1913)
* Upgraded Plastic library in AstroWeka, got the code into SVN and fixed
the Ant build script
--
------------------------------------------------------------------------
AstroGrid/VOTech
&
WFAU, Institute for Astronomy, Edinburgh
Skype:johndavidtaylor <skype:johndavidtaylor?chat>

------------------------------------------------------------------------
*Gratuitous advertising:*
Plastic - http://plastic.sourceforge.net | AstroRuntime -
http://www2.astrogrid.org/desktop
AstroGrid - http://www.astrogrid.org | WFAU -
http://www.roe.ac.uk/ifa/wfau/

Wednesday, October 11, 2006

(Two) Weekly Summary 11/10/06

* Started trying to collate all the apps we do in DS6 here:
http://wiki.eurovotech.org/twiki/bin/view/VOTech/DS6Software
and
chivvy people along into having a proper release process.
* Looked at Maven 2.1-SNAPSHOT in the hope that it has solved the many
bugs in Maven 2.0.4 that are stopping me getting _my_ software into shape
* Maven is still a disaster for site generation. Refactored the Plastic
site to remove as many subprojects as possible in the hope that this
will help.
* In depth look at Mirage
* Installed STILTS-1.3 and got authorityId changed to Starlink, which we
manage here at ROE.
* Checked all our dbs except usnob, conesearch and adql through workbench.
* Created bugzilla entries for all our components to keep on top of
issues. See
http://www.astrogrid.org/bugzilla/editproducts.cgi?action=edit&product=WFAU
* Released PAM - the Plastic Application Monitor plastic.sourceforge.net/tupperware/plastic-application-monitor/
* Continued drafting Plastic Note
* ADASS preparation.

Thursday, October 05, 2006

Maven site generation still buggered

I'd hoped that some of the issues with maven site generation
http://jontayler.wordpress.com/2006/07/26/porting-the-plastic-site-over-to-maven-2/
might be fixed by now. There seems to be no hurry to release the next
version of Maven
(see http://www.mail-archive.com/users@maven.apache.org/msg51783.html)
so I ventured into bleeding-edge territory and downloaded the latest
SNAPSHOT from
http://maven.zones.apache.org/~maven/builds/branches/maven-2.0.x/

Neither the site:staging bug nor the
windows-maven-site-ignores-settings.xml bug appear fixed. Admittedly,
these are bugs in plugins, rather than the core, so perhaps I just need
to force it to use the latest plugins. How hard can that be?

--


It's reassuring that I'm not the only one losing patience with the whole Maven endeavour:
http://www.ctoforaday.com/archives/000050.html

Tuesday, October 03, 2006

Recursive IFRAME IIb



Even trying to use two pages to do the recursion doesn't fool Firefox.

Recursive IFRAME II

IFrames

Inspired by this:
http://communitygrids.blogspot.com/2006_03_01_communitygrids_archive.html
which is an interesting blog in its own right. I was tempted to see if you can do a recursive IFrame.




Quite sensibly - Firefox spots the recursion and just presents a white box.

Animated GIFS for WFAU

Mark has been working on some rather neat animated GIFS for WFAU. Hope to embed these in our registry entries - that should blow CASU off the page in AstroScope.
http://eurovotech.org/wfau/logos/wfau_logos.htm

Will get similar done for Plastic....

Sunday, October 01, 2006

Evaluating Skype's security

http://www.skype.com/security/files/2005-031%20security%20evaluation.pdf

Friday, September 29, 2006

System Tests for AG@ROE

One of the biggest things we have to address if we are to persuade astros that the VO is a serious project is reliability. Currently too many services fall over or don't meet interoperability standards fully.
At Edinburgh we are hosting the full range of VO Services from datasets such as the WFCAM Science Archive to applications such as the Anomaly Detector and STILTS to infrastructure such as MySpace. We have to make sure that we know that systems have fallen over before our customers do.
We are going to need a multilayered solution, from simple-minded pinging of webservices such as that provided by MARS (currently down, awaiting a java upgrade) to the heartBeat service in the VOSupportInterfaces currently being developed by the IVOA GWS group. Going to greater depth, I think we need automated system tests of all of our user-facing components, testing them in the same way as a user would. These system tests would be run (say) once a day, rather than providing the real-time feedback offered by the other techniques. For example, MySpace should be tested by creating folders and files, storing and retrieving data. A dataset should be tested by running a standard conesearch etc.

A convenient way to run these tests is via the Astro Server Runtime developed by Noel Winstanley. This offers an easy API to all the VO services we deploy here at Edinburgh and allows me to write the tests in my language-of-choice. From my previous experience of running the Integration&Release process, I've decided to go with JUnit tests wrapped in a Maven project run by a cron job. This is because JUnit tests are easy to write, and Maven provides easy facilities to publish reports to the web on the success rate of the tests.

Status of the System Tests project:
  • Tests for Community and MySpace written
  • Tests for conesearches fleshed out
  • Code has been put into cvs under: http://www.astrogrid.org/viewcvs/test/AgAtRoe/
  • Bugs in the Workbench and Astro Server Runtime have been identified that block further development
  • Need to restructure build to use the full, un-Proguarded ASR.

Tuesday, September 26, 2006

(Two) Weekly Summary 27/9/06

  • One Week at IVOA Moscow (Notes)
  • Released a draft of the Plastic spec for the IVOA note, which received quite a thorough going over from MT.
  • Started work on System tests for AG@ROE

Anomaly Detector problem

There seems to be a problem running the Anomaly Detector algorithm at
http://eurovotech.org/twiki/bin/view/VOTech/AnomalyDetection

This algorithm is currently in the ROE Test registry, not in the main AstroGrid registry. Unfortunately the new workbench seems to be disregarding registry settings and using Leicester, regardless. This might be linked to the change over to use the streaming registry client.

Useful page:
Workbench configuration options.

(Problem now fixed in HEAD)

Wednesday, September 13, 2006

(Two) Weekly Summary 13/9/06

  • In Strasbourg at VOTech meeting.
    • Progress on messages for clustering and spectra
    • approval from MT and TB for IVOA note
    • Interesting tool from Bruno...might help with SIAP service for SSA
    • Information from Guy on secure SIAP - seems that TLS is the way forward, provided we can find a container that can deal with certificates.
  • I've asked Pierre Fernique (deputising for Mark Allen) for a slot at the IVOA meeting - it seems they're not inundated with speakers at the apps IG.
  • Marked MSc dissertation.



Actions Items:
--------------

(68.2) JDT to give an ROE coffee talk with a demo of AstroGrid
- maybe also including access to UKIDSS data.
[open: decided to postpone this until after registration of users]

(69.2) JDT to pass this up to Noel Winstanley and/or Kevin Benson.
[closed: see AG@ROE below]
Follow up: JDT to fix registry entries

(70.4) JDT to contact Guy Rixon to check that their plans for wrapping
STIL(TS) as CEA services are consistent.
[closed: JDT decided that the best thing to do would be to deploy here
a clone of the Cambridge set-up. STILTS can auto-generate the required
application description, so that it is possible to have the description
for the two installations being identical, which would enable JES to
pick on or other of them.]

Followup: JDT has now cloned the STILTS installation at the ROE. This is using STILTS' auto-generated app-description. There are a couple of minor issues: some of the description metadata is html-formatted, and doesn't play nicely with the Workbench, and one of the parameter types has been incorrectly specified. [the constant specifying the STILTS command]. One of the disadvantages of cloning the Cambridge installation is that we in Edinburgh don't get any credit - it's got IoA logos all over it.

(70.5) JDT to contact Phil Nicholson to ask about his plans for
a standard SIAP component.
[closed: see AG@ROE below]

Follow up: see JDT comments above.

AOB
---

IBM hardware
JDT mentioned that there had been an email circulated about the possibility of
some surplus IBM hardware becoming available on site soon.
Any news?

Tuesday, August 29, 2006

Thunderbird mail forwarding from a filter broken

I want to use a spare gmail account to backup all my incoming and outgoing mail in Thunderbird. Outgoing is easy enough - for each account under tools/accounts/Copies & Folders you add a line to the Bcc section. To set up backup of incoming mail, I created a filter to act on the inbox and find "read" messages, and forward them. This ought to work but doesn't....I just get an alert popup box with "sending of messages failed." It seems I'm not the only one.

Java signed applet security insanity

I've been looking into using signed applets as a javascript-plastic bridge. The applets have to be signed so that they can make RMI connections to the Plastic Hub - this works well. Then through the wonders of LiveConnect you can call methods on the applet from JavaScript, and thanks to a magical object called JSObject you can make callbacks back to the JavaScript from the Java. This also, mostly, works well. However, try deploying the applet anywhere but localhost, and some of the JavaScript->Java calls fail with security exceptions (see below). Not all, just some. The reason? Apparently, if the JavaScript itself isn't signed, then any method calls originating in the JavaScript are treated as if the applet itself isn't signed - you lose all your signed applet privs. What kind of deranged madman thought that up? It's useless as a security benefit - all you do is put the controversial stuff into a separate thread, so that the JavaScript to Java calls merely set and get properties...so it's trivially subverted. Instead it just makes your code more complicated than it need be, and confounds the poor programmer with mysterious errors.

Implementation note: it's not good enough just to spawn an new thread in the method called by JavaScript, as this will inherit the security of the calling thread. Instead you need to instantiate a worker thread in (say) the start() method of the applet, and have the JavaScript-called method hand tasks to this worker. You can see a (rather naive) implementation here.

-----------
An alternative solution:
Put the "untrusted code" in:
  somemethod() {
...normal code here...
String user = (String) AccessController.doPrivileged(
new PrivilegedAction() {
public Object run() {
return System.getProperty("user.name");
}
}
);




Stacktrace for google's benefit:
java.security.PrivilegedActionException: java.lang.reflect.InvocationTargetException
at java.security.AccessController.doPrivileged(Native Method)
at sun.plugin.liveconnect.SecureInvocation$2.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.plugin.liveconnect.SecureInvocation.CallMethod(Unknown Source)
at sun.plugin.liveconnect.SecureInvocation.access$300(Unknown Source)
at sun.plugin.liveconnect.SecureInvocation$CallMethodThread.run(Unknown Source)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.plugin.javascript.invoke.JSInvoke.invoke(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at sun.plugin.javascript.JSClassLoader.invoke(Unknown Source)
at sun.plugin.liveconnect.PrivilegedCallMethodAction.run(Unknown Source)
... 6 more
Caused by: java.security.AccessControlException: access denied (java.net.SocketPermission 10.0.0.8:2234 connect,resolve)
at java.security.AccessControlContext.checkPermission(Unknown Source)
at java.security.AccessController.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkPermission(Unknown Source)
at java.lang.SecurityManager.checkConnect(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.checkConnectPermission(Unknown Source)
at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
at sun.rmi.server.UnicastRef.invoke(Unknown Source)
at net.ladypleaser.rmilite.impl.RemoteInvocationHandlerImpl_Stub.invoke(Unknown Source)
at net.ladypleaser.rmilite.impl.LocalInvocationHandlerImpl.invokeRemote(LocalInvocationHandlerImpl.java:53)
at net.ladypleaser.rmilite.impl.LocalInvocationHandlerImpl.invoke(LocalInvocationHandlerImpl.java:41)
at $Proxy0.requestToSubset(Unknown Source)
at org.votech.plastic.managers.PlasticApplication.sendMessage(PlasticApplication.java:166)
at org.votech.plastic.managers.PlasticApplication.sendMessage(PlasticApplication.java:140)
at uk.ac.roe.PlasticHubApplet.sendLoadVOTable(Unknown Source)
... 16 more

Wednesday, August 23, 2006

How to get an applet to repaint

You have an applet. In response to some event you want to change all the components it contains. However, your changes don't appear...why?
If I was using a JFrame, I'd liberally throw in a few pack() statements, which causes the JFrame to resize itself based on the new components, and redraw, but you don't have this option with an applet. After some javadocing, it seems that a combination of validate(), invalidate() and repaint() ought to do it. Some experiments later, it seems that
  • invalidate does nothing
  • validate shows up the new components, but over the old ones
  • repaint erases everything
  • invalidate followed by validate is the same as validate
  • invalidate followed by repaint is the same as repaint

  • validate followed by repaint does the trick

Copying a list of files in Ant

One might think that such an everyday task as copying a list of files would be trivial in Ant. Not so - it's a bloody nightmare.
  1. First attempt: creating a FileList would seem to be what you want. Nope - the Copy task doesn't support FileLists. Why?????
  2. Second attempt: create a FileSet (with the usual substitution of angle brackets....would it be too much to ask to have a blog site that doesn't stuff up xml?):


  3. [copy todir="foo"]
    [fileset dir="bar"]
    [filename name="pathfrombarto/file"/]
    [filename name="differentpathfrombarto/file2"/]
    [/fileset]
    [/copy]

    Nope - the filenames are mutually exclusive - the fileset does an "and" on them, and since your files won't match both names, it copies nothing. Ridiculous.
In any case, the fact that you're forced to use a FileSet and its pattern matching means that if you have a large set of files below "dir" (e.g. a Maven repository) then the above takes an age.

The solution, incidentally, is:

[copy todir="foo"]
[fileset dir="bar"]
[or]
[filename name="pathfrombarto/file"/]
[filename name="differentpathfrombarto/file2"/]
[/or]
[/fileset]
[/copy]

Though it still takes an age to process.

Monday, August 21, 2006

Weekly Summary 30/8/06

  • Guy has created a new community with password change facility - passed I&R and ready to install.
    • Tried to install, but it won't self-register: see bug 1789
    • Guy has put a fix on the merge queue for this...in the meantime we can hack the reg entry....done...appears to work well. Except.....you can change your password with any old password.
    • Mark to reinstall when it passes I&R? How did this error slip through I&R in the first place?
  • Submitted abstract to ADASS 2006 http://www.adass.org:8080/Conferences/2006/Venue/people/participants/abstract?abstract_id=82
  • Planning: using Open Workbench http://www.openworkbench.org/ to create GANTT charts and managed dependencies between tasks.
  • Downloaded and installed Eirik http://wiki.eurovotech.org/twiki/bin/view/VOTech/EirikDemo on a Debian machine. Plastic and AR functionality appear fine. Couldn’t test further due to Window size issues. Installation straightforward - install debian version of R first, then hand edit the Eirik startup script to find the location or libR.so.
    • Tried to install Eirik on Shuna (Bob’s machine). Since I’m not root this involved building R from source using ./config and make. This is fairly easy to do, following the instructions, but care must be taken in the config stage to request a shared library libR.so to be built, and also to set –with-readline=no on Shuna. Unfortunately getting Eirik to run on Shuna wasn’t do-able due to assorted missing shared libs on this machine.
    • However, following several iterations, did get a windows version running.
  • Hardened the hub to concurrency problems and clients passing shit values.
  • Jeff and others are experiencing some AR problems - AR is refusing RMI connections. I also have the same problem from time to time and it’s usually cured with a reboot. Mark’s Plaskit hub doesn’t suffer from this. Suspect it’s something to do with the way the AR deals with IP addresses - a network outtage is enough to screw it up.
  • Finished Anomaly Detection pipeline
    • Looking at deploying stilts at the same time. There are problems with the existings STILTS deployments discussed here. (Fixed by Kona). Get Kona/Mark to install?
    • Standard and Advanced interfaces now done, and registered. All works smoothly, but not very robust to errors in data since fastem locks up waiting for a user response....not a great idea for a CEA app.
    • Created an applet to allow users to simply change their workbench registry settings. This was also a learning exercise for creating a plastic-applet.
    • Instructions for use are here.
  • Finished a prototype Plastic-compatible web page...this is aimed at our own WSA, but also to get DataScope on board.
  • Received request from Noel to smarten up our metadata - we need to discuss logos.
  • Checked Son Of GAIA for Nigel, and found that I can’t run the webstart version either due to missing classes. Probably due to the 1.5 java libraries having renamed some classes. Reported to ussc@star.rl.ac.uk...address is defunct...emailed Mark Taylor...apparently you need to install the JAI package first (not documented!).
  • Assisted Brian getting his weka cea apps deployed to our test registry. We encountered a lot of out-of-memory errors in the registry, and also a particularly annoying bug necessitating the reconfiguration of the registry and changing its authority Id.
  • Assisted Brian changing the registry his workbench uses.
  • Realised that if we have a test registry, we need a test JES, so installed one here. The official release of JES is lacking an astrogrid-security jar, which you have to add by hand to WEB-INF/lib. Catherine has now released a new build, which Mark should test.
  • Thanks to Brian, reproduced the mysterious "Found Resource Document, but had no AccessURL" that has been plaguing systems that harvest from our publishing registry and try to access our date sets. Bugzilla report. It appears to be due to our use of Java 1.5. I've found that most of our old datasets suffer from the problem, but that it doesn't occur in the latest snapshot build of the workbench. Hence we could ignore it - it will go away when we reregister our new services anyway. Left it in Mark and Kevin's capable hands.
  • Republished Plastic site with google analytics code throughout
    • mvn 2.0.5 not yet released, and the bugs in 2.0.4 make site publication a painful process as
      • site:deploy doesn't work in windows
      • site:stage doesn't copy the contents of directories, effectively making multiproject site deployment useless



Actions
(68.2) JDT to give an ROE coffee talk with a demo of AstroGrid
- maybe also including access to UKIDSS data.
pending...


(69.1) JDT and KEA to produce a plan for providing password-changing
functionality for ROE account holders registering for AG.
DONE - see above

(69.2) JDT to pass this up to Noel Winstanley and/or Kevin Benson. DONE - see above

ACTION (70.1) MSH to install the new version of Community.
done by JDT

ACTION: (70.4) JDT to contact Guy Rixon to check that their plans for wrapping
STIL(TS) as CEA services are consistent.
DONE - see above

Thoughts:
Need better cover for MSH's absence

Creating the Anomaly Detector CEC

I'm installing Andy Connolly's Expectation Maximization/Anomaly Detection algorithm into an AstroGrid CEC (Common Execution Connector) to make it available to the VO. Along the way I've been using the splendid STILTS utility to do all the VOTable conversions.
Things I've learned along the way
  • STILTS is sufficient to take a table in assorted different formats, and extract a user-defined set of columns (specified by position or name), and export the results as an EM-friendly space-separated text file. It is also smart enough to take the generated list of row numbers and p_values and do an exact cross match on the original table...thus generating a clone of the original table, sorted by p_value and with the p_values attached as an extra column. In short, it does all the table manipulation needed to transform a VO-standard table into a proprietary format and back again, with only a little help from awk.
  • Any processes started by the CEC must NOT ask the user for input. The current EM-algorithm prompts the user for input if it fails for some reason...this seriously screws up your CEC. The solution is to add a test on the return value from the STILTS preprocessing step, before the EM-algorith is called:
    • #echo $?
      #Try to protect fastem from bad data
      if [ "$?" -ne "0" ]; then
      echo "STILTS failed to prepare data. Aborting"
      exit 1
      fi

  • You can only specify a registry template for a single CEA application. This means that if you're CEC supports several applications, you need to edit the registry description by hand after it's been generated.
  • You can specify an optional argument in the CEC-CL config with (insert)
  • The CEC-CL only supports the following kinds of parameters:
    • -key value
    • positional
    • key=value
  • Unfortunately the em algorithm takes parameters as
    • key value
  • You can get round this by pattern matching on the params with something like ${@//=/ } which takes all the parameters in one go and replaces all = with ' '

Friday, August 18, 2006

New week, new blog

Still looking for the perfect home for my blog. All I want is something that's quick and responsive that I can use to jot down notes from work to give my ageing memory a break. Plus...when you spend all day wrestling maven into submission, it's possible that what you've learned just might save someone else some grey hairs.

So, I'm defecting from wordpress: http://jontayler.wordpress.com/ to here.