Science With the Virtual Observatory
2006 Summer School

Building a Java Skynode

Course Description

Distributed databases in astronomy now have the advantages of providing access to large datasets via network protocols and have opened the window to cross correlating and querying datasets across wavelengths and multi-parameter space.  In this course the student will learn how to build and configure a local fullskynode server, register their service address in a VO Registry, and perform advanced queries on their fullskynodes in the OpenSkyQuery portal.  Although this FullSkynode exercise demonstrates the access to a MySQL database,  the implementation and utilities in the nvoss software fullskynode package can be used with other database connections.

1. Required Software and prerequisites

In order to perform this exercise the following software installations are required.  You may go to the software installation pages on the NVOSS 2006 Software setup for specific location and download instructions. 

Setup the NVOSS Environment

 

The NVOSS setup script has been run. 

               

\nvoss2006\setup

 

Note:  As an alternative to running the setup scripts a manual path set can get you in trouble on Windows machines if you say

set nvoss_home="..." rather than NVOSS_HOME.  Although Windows seems to be case-insensitive, the Ant scripts

run here are not.

 

Database Configuration

 

A running MySQL database which has the SPOCS database described in the Introduction to SQL installed.

The addcolumns.sql and fillcolumns.sql scripts in $NVOSS_HOME/sql/SPOCS/mySQL are assumed to have

already been run to create the ra and dec columns in the SPOCS table.

 

 

Tomcat Server

 

The TOMCAT server supplied with the NVOSS library is running.  You can just use the server command defined in the

NVOSS setup script.  So at the command prompt type

 

        starttomcat

 

 

The SkyNode will be installed as a CGI application in this server and the TOMCAT configuration has been modified to support CGI.  The ANT install (or test)

will create the CGI directory if you don't have any other CGI scripts.

 

Perl

 

A version of Perl is installed and in the command path. Any version of Perl > 5.0 should be fine.  Perl is used only as a simple wrapper for the Java CGI task.  TOMCAT

assumes that Perl is to be used for invoking CGI scripts.

2. Getting Started

We will be working in the NVOSS_HOME/java/src/fullskynode directory.  In this exercise you will need to following utilities on your system to perform the exercise steps.

·         An editor

·         Browser window

·         Command line window

3. Configuration for Local Skynode

Let’s go to the directory and look at the files.  The README and USAGE files are reference documents for this exercise.  The other files are required for the actual skynode server build.  We will start by editing the ant script file,  build.xml.  This will allow us to customize the build to our local machine (i.e.  the skynode server) and make sure your skynode is distinct from all other Skynodes.  If you not following the defaults for the Summer School SkyNode demonstration you may need to alter some of the default settings in the build.xml AND the vo.settings files.

In your editor,  open the build.xml file.  You will need to change the two values:  the SkyNode ID and the SkyNode Host.  The ID is an identifier string which is used to distinguish your Full SkyNode from all others.  We suggest the value TSTVO_iii where iii are your initials.  The SkyNode Host is the hostname or IP of this machine on which both the MySQL and TOMCAT servers are running.  Here is an example of the build.xml file where the initials GRG are used and the ip address of a local server is filled in.

 

<?xml version="1.0" ?>

<project name="skynode" default="install" basedir=".">

<!-- 

 Need this for separators where the \ will be escaped.

  -->

<condition property="sep" value="\\" else="${file.separator}">

  <equals arg1="\" arg2="${file.separator}" />

  </condition>

  <property environment="env" />

  <property name="lib" value=".${file.separator}lib" />

  <property name="classes" value=".${file.separator}classes" />

<!-- 

 The name of the SkyNode as seen in OpenSkyQuery.
         The SkyNode will need to be registered, and the
         ShortName of the registered node must match
         this value.  For the NVO summer school we recommend
         using your initials to distinguish your SkyNode from
         other Students, e.g., TSTVO_TAM.  This value is also
  specified as the ShortName in the VO registry.
 
    ************************* MUST CHANGE **********************      

 -->

  <property name="skynode_id" value="TSTVO_GRG" />

<!-- 

 The host name or IP of the machine on which the SkyNode is
         to run.  That's probably this host...
  
    ************************* MUST CHANGE *********************** 

-->

  <property name="skynode_host" value="192.168.0.2" />

- <!-- 

 The port number of the URL of the SkyNode  

  -->

  <property name="skynode_port" value="8080" />

- <!-- 

 The directory at which the URL appears to run.
         If the server redirects requests this may appear
  unrelated to the directory in which the CGI script is
  actually placed.
     

  -->

 

 

Suggestions for finding the name/ip of the current computer.

  • Windows :  run ipconfig at the command prompt
  • Mac:  Apple Menu>System Preferences>Network>TCP/IP.
  • Linux:  At the command line type /sbin/ifconfig and you will get a listing of information. Your IP is shown after inet addr. you may be able to use the hostname command on Unix systems

 

For a SkyNode to work it must be on a machine that is visible to the portals that are to use it.  A SkyNode will generally not work if it is behind a firewall or NAT server.  Systems on DHCP servers can work if the DHCP server supplies routable addresses to its clients. Again, if you are not following the recommended settings in this exercise you may need to alter default settings in the build.xml and the vo.settings files.  When building a full skynode with access to another database you will need to change the vo.settings jdbcURL and table access.

 

[NOTE for port number change:   If you need to run the Tomcat server on a different port,  you can change the port number for the skynode_port in build.xml file and also in the Tomcat server.xml]

 

[NOTE for Windows Users: If your Windows installation uses a SystemRoot other than c:\windows, then you need to update the sn.pl file to set the correct root before installing.]

4. Create Skynode USER for MySQL

In building a skynode server,  you are providing access to your local database from the outside world.  A standard practice for basic database security (although this is not required for the skynode implementation) is to allow remote users to have only the permissions they need to access the Skynode.  The adduser.sql script should be run at the MySQL command line (using the MySQL root account).  It adds a user 'skynode'.  Users would be well advised to change the password used here and in the vo.settings file.

 mysql –u root –ppwd < adduser.sql

 

Note that this is run from the MySQL root account, though it will create the MySQL skynode account.  Also recall that there is no space between the -p and the root password. You may also run from the mysql prompt (MySQL started from the fullskynode directory):

 

mysql > \. Adduser.sql

 

 5. Build Skynode server

The build.xml ant script provides all the targets and instructions for building and deploying the skynode server to the Tomcat cgi location. You do not have to create any directories or copy files, the ant script takes care of that for you. At the command prompt execute the following:

ant install

There is a test target available which will test the build configuration.  If desirable to check your setup run the ant test target.

ant test

 6. Register Skynode

Skynodes are ‘discovered’ via the OpenSkyQuery portal access to a NVO Registry.  A test registry is provided for this demonstration and also other development skynodes here. 

http://dev.voservices.net/registryFT

 

The most direct approach we will follow for this demonstration is to edit an existing skynode service.  So we can go directly to the following link:

 

http://dev.voservices.net/registryFT/UpdateRegistry.aspx?InsertMode=t&SearchIdentifier=ivo%3A%2F%2Fnasa.heasarc%2Ftest%2Fskynode%2Ftest_tam2&btnSubmit=Search

 

You need to change 3 fields:

 

  The ShortName must be changed to the same string you used, TSTVO_iii (where iii are your initials)

  as the SkyNode ID entered in the configuration step 3 build.xml file.

 

  The ServiceURL must be changed to point to your SkyNode.  By default that

  should be http://your.ip.address:8080/cgi-bin/sn.pl.

 

  The Identifier field must be changed to a unique string. We suggest you just change

  the final TSTVO_TAM2 to whatever your SkyNode ID is.

 

Enter a password in your registration form (bottom left) and then submit the resource.  You will need to use this password for any future skynode resource metadata changes. . 

 

If you do a keyword search on your skynode id it may not show up immediately due to registry indexes which rebuild periodically, you may perform an advanced search using the search form and enter

 

Shortname like ‘TSTVO_iii’                      where iii are the initials used in the build configuration. 

 

This is an example from the registry search form.

 

 

Please enter your SQL predicate here using VO MetaData e.g.
ResourceType like 'CONE' and contentlevel like '%research%'

Custom Predicate

-OR-

Choose

 

 

7. Run the Skynode in OpenSkyQuery

If your registration succeeded and we know it did, you should see the SkyNode in the list of nodes in the Advanced Query at

 

http://dev.voservices.net/openskyqueryFT

8. Perform Query

Since we are working on a local network,  we have setup a local OpenSkyQuery portal from where you can perform test queries on your MySQL database.

Go to the local NVOSS OSQ portal site http://192.168.1.33/openSkyQueryFT

 

and go to the Advanced Query page by clicking on the top menu tab.

You can now test your skynode setup by performing a few basic queries.   Here are some example queries you may follow for your testing.

SPOCS_Query1  

-- Q1: Find objects with declination values between 0 and 1 deg.
-- Demonstrates single node query in OSQ with java full skynode

SELECT b.ra,b.dec FROM TSTVO_tamnoao:spocs b where b.dec > 0 and b.dec<1

SPOCS_Query2  

-- Q2: Perform cross match between full skynode databases.
-- Demonstrates cross match between the spoc table and sao catalog
-- which contains ~ 200000 brightest stars in the sky 

SELECT b.ra,b.dec
   FROM TSTVO_saonoao:sao a, TSTVO_tamnoao:spocs b
      where XMATCH(a,b) < 3.5 and b.dec>0 and b.dec<1

SPOCS_Query3  

-- Q3: Find all solar analogs
-- Demonstrates selection of objects based on filtering criteria.  Similar query demonstrated with MySQL interface
-- Note to use care with case sensitivity in field names.
SELECT s.ID, s.Name, s.Teff, s.logg, s.Metal FROM TSTVO_tamnoao:spocs s
   where   s.Teff BETWEEN 5720 AND 5820
   AND     s.logg BETWEEN 4.34 AND 4.54
   AND     s.Metal BETWEEN -0.1 AND +0.1
      ORDER BY s.Teff

By now you have seen several examples of using SQL and MySQL and cross-correlation capabilities using OSQ.  There are limitations to the OSQ portal which you may encounter.  One feature to keep in mind is that the OSQ portal invokes an Execution Plan that uses query optimization and communication between the individual skynodes.  Therefore,  if you perform a query which prioritizes an external skynode to perform an execution plan,  that node may not communicate to nodes within a local network depending on the external access permissions.  The queries above have been tested and will work within the nvoss skynode network.

9. Student Exercise

This exercise will require you to use your knowledge of SQL,  MySQL, and now fullskynodes to find what appears to be a positional error in the Spocs Database.  When the SPOCS data was created ASCII positions that had been encoded as hh mm ss.s dd mm ss were coverted to six integer and real columns.   For positions between declination 0 and declination -1, the sign of the declination is lost since the datatype are not float, and the declination appears to be positive.  So for the objects apparently in the range 0<dec<1 we don't know which are at the correct position and which have had the dec inverted.  Let's  use SkyNodes to fix this. The first student to discover which objects in the spocs table are incorrect will win a prize (TBD).  GOODLUCK!!!  Here are a few helpful hints that may give you some clues how to use the skynode correlation functions to find matching and non matching sources.

Hint 1: Query example 1 above will show you how to limit a declination range search.

Hint 2: The SAO catalog contains ~ 200000 of the brightest stars in the sky and is a useful catalog for performing optical counterpart cross-matches to the SPOCS tables. Query 2 above shows how to run the cross match query.  

Hint 3: You may find the anticorrelation function useful, i.e. XMATCH(!a,b) to check sources that do not match between catalogs. Using a restriction on range of search would be advisable.

Hint 4: To check for a specific source in a catalog you may use the Region function.

SELECT a.ra,a.dec

  FROM

  TSTVO_saonoao:sao a

  WHERE Region('CIRCLE J2000 raValue decValue searchTolerance')

10. Links of interest

IVOA TWiki site

·         VO Query Working Group

o        http://www.ivoa.net/twiki/bin/view/IVOA/IvoaVOQL

Astronomical databases on-line using SQL interfaces

·         The Sloan Digital Sky Survey - SDSS

o        http://skyserver.sdss.org/

·         Galaxy Evolution Explorer - Galex 

o        http://galex.stsci.edu/

·         SuperCOSMOS Science Archive - SSA

o        http://surveys.roe.ac.uk/

SDSS SQL Tutorial

·         SQL Introduction

o        http://cas.sdss.org/dr4/en/help/docs/sql_help.asp

·         Sample Queries

o        http://cas.sdss.org/dr4/en/help/docs/realquery.asp

The NVO Summer School is made possible through the support of the National Science Foundation and the National Aeronautics and Space Administration.