Science With the Virtual Observatory
2006 Summer School

Exercise 4: Build a cone search service using a local database

Contributors to this document: Brian Kent (Cornell University) and Gretchen Greene (STSCI)

In this exercise, a simple ConeSearch service (specification) will be setup and configured locally. Participants will understand the basics of the VO ConeSearch protocol and how to use it to query a database on their local laptops and return a XML/VOTable file based on an input of RA/DEC coordinates and a search radius. The resulting service will be tested with a verification service.

Overview

A ConeSearch service is conceptually a simple idea - a user provides three parameters - right ascension, declination and a search radius (in decimal degrees). An XML/VOTable file is returned with objects and any data pertaining to those objects within the search radius. By the end of the exercise, students will understand how to create a database, table, and import data into MySQL, manipulate the java code to access the database, start Tomcat to run a local webserver, and utilize the code in a java servlet page to return an XML file.

Useful links

Java Database Connectivity (JDBC)
Apache Tomcat
ConeSearch Specification
Implementation of an XML Schema describing a Cone Search Service by Ray Plante.
Cornell Digital HI Archive(with ConeSearch)
Registry Search of ResourceType CONE

First let's examine a simple coneshell and test the web server setup.

Simple Cone Shell and server startup

Tomcat

First lets make sure our tomcat works. We should check that CATALINA_HOME is properly set.

> dir "%CATALINA_HOME%"
or on UNIX
>  ls $CATALINA_HOME

OK if that returns a listing we should be able to use our script to start tomcat:

> starttomcat

This will fail if tomcat is already running on the port. The port is set in the server.xml file in the conf directory under CATALINA_HOME. The default is 8080 - but you may change it. As a reminder, there are additional utilies for managing the server.

> bouncetomcat (to stop and start) or
				> stoptomcat

If you are having a service error, it may be due to a bad reset or not waiting for the server to complete initialization. In the case of error, sometimes try to reset the server using the utilities provided making sure you allow a few seconds for the initialization to complete. The Tomcat server may be verified at

http://localhost:8080/

Java Server Pages (JSP)

JSP is a very quick way to develop highly functional web pages. Effectively each page becomes a java class (a servlet) which is compiled once by the server. When coding in JSP we may intermix java code with actual output such as XML or HTML to create our pages. The .NET framework also now provides a facility like this which are its ASPX pages.

Lets look at the cone service. Effectively it is a web application which takes 3 parameters: RA,DEC,SR. It then needs to do a lookup using this information and provide the answer in a VOTABLE. So a call to a cone service should look like

   coneservice?RA=108&DEC=1.5&SR=0.5

So an easy way to do this would be to make a simple JSP page and have it pull these parameters from the request. Such a page is provided in the distribution package in dev/conserver/coneShell.jsp.

We will walk through the code before we deploy it. This is a nice easy JSP so for deployment all we need to do is copy it to a web directory. The default root of the tomcat installation is in the webapps/ROOT directory so we need to :

> copy coneShell.jsp "%WEB_DEPLOY%"\ROOT
or on UNIX
> cp coneShell.jsp $WEB_DEPLOY/ROOT

Then this will be available at

http://localhost:8080/coneShell.jsp?SR=1&DEC=180&RA=1

But there are no data there yet!

Importing data into MySQL

We will use a portion of an HI catalog of galaxies as out dataset (Springob et al. 2005, ApJSS, 160,149). The csv catalog can be found in $NVOSS_HOME/java/data/agcgalaxies.csv. First we will begin by starting the MySQL server.

> MySQL 5.0\bin\mysqld
or on UNIX
> mysqld

This will start MySQL in the background. Now, login to MySQL (we'll use the root account for simplicity, but you would not want to do this a live server.

> MySQL 5.0\bin\mysql -u root
or on UNIX
> mysql -u root

You will now be presented with a MySQL prompt. Please refer to Tom McGlynn's discussion on SQL for synatx and basic commands. The first step is to create a database called nvoss and use it:

msyql> create database nvoss;
Query OK, 1 row affected (0.04 sec)

mysql> use nvoss;
Database changed
mysql>

Next, within the database nvoss we can create a table called galaxies:


mysql> create TABLE galaxies (
mysql> agcnumber int NOT NULL,
mysql> ra double,
mysql> decl double,
mysql> velocity int);

Check the table structure via:


mysql> describe galaxies;

There are different ways of getting data into your table. Using a comma-separated value ASCII file is usually easiest. A convienient way to load the data is via:


mysql> load data infile 'directory-to-/java/data/agcgalaxies.csv' into table galaxies fields terminated by ',';

One can now issue a simple SQL query at the prompt to examine the data...


mysql> SELECT * from galaxies;

We'll now move on to the java code and associated methods...

Java code

The java code with the associated methods for this exercise can be found in $NVOSS_HOME/java/src/coneserver/src/sumsch/VOTCone.java The database is queried, and returned objects are written to a VOTable file. The class VOTCone calls the IVOA VOTWrap search method which will determine the angular distance between the position passed to the conesearch (using the java distance method), and an object within the table along a great circle. Those objects are returned to the java server page and displayed in XML.

We will now switch to the main coneserver directory and compile the coneserver code:


> cd $NVOSS_HOME/java/src/coneserver
> ant compile

We can now run a simple ant test, which will return messages establishing a connection and termination to MySQL.


> ant test

The final step is to deploy the module to the server, also by using ant. First, a small modfication needs to be made to the web.xml configuration file located in $NVOSS_HOME/java/src/coneserver/conf/. The env-entry value needs to be changed to your java data directory:


<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE web-app
     PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN"
    "http://java.sun.com/dtd/web-app_2_3.dtd">
<web-app>
<env-entry>
  <env-entry-name>dataPath</env-entry-name>
  <env-entry-type>java.lang.String</env-entry-type>
  <env-entry-value>/nvoss2006/java/data/</env-entry-value>
 </env-entry>

</web-app>

Return to the coneserver directory, and deploy the servlet:


> ant deploy

Give it a try!

The coneserver module will be deployed to $CATALINA_HOME/webapps/ROOT/cone/. We can now test the coneserver in our web browser at the address http://localhost:8080/cone/cone.jsp?RA=180&DEC=12&SR=2 which will return the file header, object numbers, coordinates, and heliocentric velocities. A further shell can be created to read the XML and display the table in a nicer format. The conesearch server can now be accessed via any valid ConeSearch client application (see client application talk).

As the ConeServer is now running on your machine, you can use a client application to access the Cone Search Service. Any client application in any language on any platform will work. We can use the Python cone client located $NVOSS_HOME/python/samples/ConeSearchEx.py that uses the class ConeSearch.py.


> python ConeSearchEx.py coneSearchRaw http://localhost:8080/cone/cone.jsp 180.0 12.5 2.0 1

In addition, one can validate a ConeSearch service with the NCSA Cone Search validator.

Try it here with the Cornell Digital HI Archive written in PHP.

The NVO Summer School is made possible through the support of the National Science Foundation and the National Aeronautics and Space Administration.

Science With the Virtual Observatory 2006 Summer School