|
Science With the Virtual Observatory |
In this exercise, we will gain experience in setting up and using a VOSpace implementation. The NVOSS software package contains both a VOSpace server and client and we shall be using both to retrieve data from other VO services and expose it to other users.
The VOSpace implementation included in the NVOSS software package consists of:
a VOSpace web service - this can either be run as a standalone server (preferred method) or as a servlet running in Tomcat. The service is built on XFire 1.2RC1 Java SOAP framework (http://xfire.codehaus.org) and XmlBeans 2.1.0 (http://xmlbeans.apache.org).
a VOSpace metadata store - this is a lightweight abstraction layer that sits on top of an existing database or other data store. Metastore implementations are provided for both MySQL and Virtuoso (http://virtuoso.openlinksw.com/wiki/main/) open source databases.
an HTTP data server - the VOSpace implementation supports HTTP PUT and GET (v1.1) as transfer protocols. A simple HTTP server based on Apache HttpCore (http://jakarta.apache.org/httpcomponents/http-core/) is provided: it interfaces with the VOSpace metadata store and employs one-time URLs.
a VOSpace client - this is a simple command-line client to both the web service and the HTTP data server. It is built on Apache Axis 1.2.1 Final.
First we need to build the Java source code for the VOSpace server and client:
> source $NVOSS_HOME/bin/setup.csh
Unfortunately we need to apply a patch to the distributed NVOSS code: the patch is available here. The patch is applied thus:
> cd $NVOSS_HOME/java/src/vospace > jar xvf <path/to/vospace-bugfix.jar> > ant jar
We also need to create a MySQL database to hold VOSpace metadata:
> mysql -u root -p<password> < schema.sql
There are a number of properties that we need to set to configure the service and these are all in the file vospace.properties:
Parameter |
Description |
Suggested value |
|---|---|---|
space.identifier |
The IVOA identifier for the service |
ivo://nvoss.org/vospace/<initials> |
space.baseuri |
The root directory where the service will store data. Note: it must be prefaced with the file:// scheme identifer. |
file:///tmp |
server.space.port |
This is the port that the service will run on. |
8008 |
server.meta.dbuid |
The username used to access MySQL database. |
root |
server.meta.dbpwd |
The password used to access the MyDQL database. |
<password> |
server.http.url |
The external address of the HTTP data server. |
The IP address of your laptop |
server.http.port |
The port that the HTTP data server runs on. |
7007 |
server.http.basedir |
The root directory that the HTTP data server will use to store files: it should be the same as space.baseuri except without the file:// scheme identifier. |
/tmp |
Note: if you are using a firewall, you will need to open the ports on which you are running the service and the HTTP data server.
The service can be started by running the vospaceserver script:
> cd $NVOSS_HOME/java/src/vospace
> ./vospaceserver
DriverManager.getConnection("jdbc:mysql://localhost/vospace")
trying driver[className=com.mysql.jdbc.Driver,com.mysql.jdbc.Driver@ec6696]
getConnection returning driver[className=com.mysql.jdbc.Driver,com.mysql.jdbc.Driver@ec6696]
2006-09-10 22:56:01,524 [Thread-0] INFO edu.caltech.nvo.vospace.server.HttpDataServer - Accepting connections on port 7007
2006-09-10 22:56:03,352 [main] INFO org.mortbay.http.HttpServer - Version Jetty/5.1.3
2006-09-10 22:56:03,362 [main] INFO org.mortbay.util.Container - Started org.mortbay.jetty.servlet.ServletHandler@8efa2f
2006-09-10 22:56:03,363 [main] INFO org.mortbay.util.Container - Started ServletHttpContext[/,/]
2006-09-10 22:56:03,369 [main] INFO org.mortbay.http.SocketListener - Started SocketListener on 0.0.0.0:8080
2006-09-10 22:56:03,370 [main] INFO org.mortbay.util.Container - Started org.mortbay.jetty.Server@a3da69
You will probably want to run this in its own window.
The client can be run using the vospaceclient script: the syntax is:
> cd $NVOSS_HOME/java/src/vospace > ./vospaceclient <method> <service endpoint>
where the <service endpoint> is the endpoint URL for the VOSpace service you want to call. For the NVOSS VOSpace instances, this will be:
http://<IP address>/VOSpaceServiceImpl
The available methods are:
Method |
Description |
|---|---|
getviews |
gets a list of the available import and export data formats for the space |
getprotocols |
gets a list of supported transfer protocols for the space |
getproperties |
gets a list of the properties for the space |
createnode |
creates a new node in the space |
deletenode |
deletes a node from the space |
movenode |
moves a node within the space |
copynode |
copies a node within the space |
getnode |
gets the details for a specific node |
setnode |
sets the property values for a specific node |
listnodes |
lists the nodes in the space |
pushtospace |
gets a URL to send data to the space |
pulltospace |
imports data into the space |
pullfromspace |
gets a URL that data can be read from |
pushfromspace |
gets the space to send data to a remote location |
httpput |
performs an HTTP PUT operation |
httpget |
performs an HTTP GET operation |
Most of the methods will prompt you for further information.
In this first example, we will save the results of a cone search query into our local VOSpace service.
The first thing we want to do is to find a cone search service that we can use so we want to go to a registry and get a list of cone search services. You can either use the STScI registry or the local version of Carnivore.
If you are using the STScI registry then select the CONTENTS tab and then the CONE ResourceType. If you are using CARNIVORE, select the Summary tab and then Browse Cone Search. Once you have your list, pick a cone search and look at the XML record for the service - the value we are after is the <accessURL>.
I'm going to use the GALEX cone search at http://galex.stsci.edu/gxWS/ConeSearch/gxConeSearch.aspx? in this example. Now remember that the cone search call also requires the verbs: ra, dec and sr so my complete URL will be:
http://galex.stsci.edu/gxWS/ConeSearch/gxConeSearch.aspx?ra=120&dec=20&sr=0.5
Now cone search services return VOTables so we want to make sure that our VOSpace service supports the VOTable data format:
> ./vospaceclient getviews http://localhost:8008/VOSpaceServiceImpl Get Views ========= The available import data formats for the space are: ivo://net.ivoa.vospace/views/image/png-1.0 ivo://net.ivoa.vospace/views/image/jpeg-1.0 ivo://net.ivoa.vospace/views/image/fits-image ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table The available export data formats for the space are: ivo://net.ivoa.vospace/views/image/png-1.0 ivo://net.ivoa.vospace/views/image/jpeg-1.0 ivo://net.ivoa.vospace/views/image/fits-image ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table
Note the URI for VOTable: ivo://net.ivoa.vospace/views/tabular/votable-1.1
We are also going to be retrieving the cone search results using a HTTP GET call so we want to check that our VOSpace service supports this protocol:
> ./vospaceclient getprotocols http://localhost:8008/VOSpaceServiceImpl Get Protocols ============= The transfer protocols for which the space can act as a client are: ivo://net.ivoa.vospace/protocol/http-get ivo://net.ivoa.vospace/protocol/http-put The transfer protocols for which the space can act as a server are: ivo://net.ivoa.vospace/protocol/http-get ivo://net.ivoa.vospace/protocol/http-put
so the URI for HTTP GET is: ivo://net.ivoa.vospace/protocol/http-get.
We're now ready to tell the VOSpace service to call this cone search service and pull the results over to it:
>./vospaceclient pulltospace http://localhost:8008/VOSpaceServiceImpl Pull data to space ================== Define the node into which data is to be imported: What type of node (plain, data, unstructured, structured):
I'm going to use a structured node:
What type of node (plain, data, unstructured, structured): structured What is the identifier for the node (hitting return will cause the space to generate an identifier):
Now we need to fill in the identifier for this node: remember that this needs to conform to the VOSpace identifier syntax (vos://[service]/[name]). When you configured your VOSpace service, you specified an IVOA identifier for your service - the value of the space.identifier parameter. If you used the suggested value of ivo://nvoss.org/vospace/<initials> then the [service] part of the VOSpace identifier will be: nvoss.org!vospace!<initials> The [name] part can be whatever you want but I'm going to use votable1:
What is the identifier for the node (hitting return will cause the space to generate an identifier): vos://nvoss.org!vospace!mjg/votable1 Now for the node properties: Identifying URI for property (blank means no more properties):
I'm not going to define any properties on this node so I just hit Return
Now specify the transfer details: What is the identifier for the data format:
The data format is VOTable so we just put in the URI we noted above:
What is the identifier for the data format: ivo://net.ivoa.vospace/views/tabular/votable-1.1 Now for the protocols: Identifying URI for protocol (blank means no more protocols):
and the protocol we are using is HTTP GET:
Identifying URI for protocol (blank means no more protocols): ivo://net.ivoa.vospace/protocol/http-get What is the endpoint for this protocol:
and this is where we put the complete URL for the cone search that we constructed above:
What is the endpoint for this protocol: http://galex.stsci.edu/gxWS/ConeSearch/gxConeSearch.aspx?ra=120&dec=20&sr=0.5 Identifying URI for protocol (blank means no more protocols):
We're not going to specify any alternate transfer protocols because there are none for a cone search so we just hit RETURN.
The node into which data was imported in now defined as: Identifier: vos://nvoss.org!vospace!mjg/votable1 Type: StructuredDataNodeType Properties: No properties are defined. Accepted data formats: ivo://net.ivoa.vospace/views/image/png-1.0 ivo://net.ivoa.vospace/views/image/jpeg-1.0 ivo://net.ivoa.vospace/views/image/fits-image ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table Provided data formats: ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table
Well, this seems to have worked OK but let's just check. In the vospace.properties file, we specified the root directory where the service will store data in the space.baseuri parameter (the suggested value is file:///tmp) so let's just have a look in this directory:
> ls /tmp nvoss.org_vospace_mjg_votable1
and we can check that it looks reasonable as well:
> head /tmp/nvoss.org_vospace_mjg_votable1
<?xml version="1.0" encoding="utf-8"?>
<VOTABLE xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://galex.stsci.edu">
<DESCRIPTION> Results from MAST/GALEX Service </DESCRIPTION>
<DEFINITIONS>
<COOSYS ID="J2000" equinox="2000." epoch="2000." system="eq_FK5" />
</DEFINITIONS>
<INFO name="SUCCESS" value="Parameters validated" />
<RESOURCE type="results">
<DESCRIPTION>GALEX Conesearch Service Results</DESCRIPTION>
<INFO name="QUERY_STATUS" value="OK" />
In this second example, we're going to transfer some data from one VOSpace to another so what I would like you to do is form groups of three. Now each of you is going to transfer data from the VOSpace of the first of the other two in the group to the VOSpace of the second of the other two in the group: so if Auberon, Berengaria and Cholmondeley form a group, Auberon is going to transfer data from Berengaria's VOSpace to Cholmondeley's, Berengaria data from Cholmondeley's VOSpace to Auberon's and Cholmondeley from Auberon's to Berengaria's.
Firstly you're going to have note what the IP addresses of each of the VOSpaces are, for example:
Person |
IP address |
|---|---|
Auberon |
192.168.1.201 |
Berengaria |
192.168.1.202 |
Cholmondeley |
192.168.1.203 |
Since the only data in any of our VOSpaces is the result of a cone search, we already know that the data format is going to be VOTable so the URI is: ivo://net.ivoa.vospace/views/tabular/votable-1.1.
We're going to get one VOSpace to push data to another so the transfer protocol is HTTP PUT. If we were really dealing with a set of different VOSpace implementations, we would check first of all that each of the two VOSpaces could deal with HTTP PUT (although the VOSpace implementation will do this automatically as runtime). However, since we are all using the same implementation, we just need to check one:
> ./vospaceclient getprotocols http://localhost:8008/VOSpaceServiceImpl Get Protocols ============= The transfer protocols for which the space can act as a client are: ivo://net.ivoa.vospace/protocol/http-get ivo://net.ivoa.vospace/protocol/http-put The transfer protocols for which the space can act as a server are: ivo://net.ivoa.vospace/protocol/http-get ivo://net.ivoa.vospace/protocol/http-put
so the URI of the protocol is: ivo://net.ivoa.vospace/protocol/http-put.
Finally we need to know what the identifier of the data we are transferring is:
Person |
Source identifier |
Destination identifier |
|---|---|---|
Auberon |
vos://nvoss.org!vospace!berengaria/votable1 | vos://nvoss.org!vospace!cholmondeley/votable2 |
Berengaria |
vos://nvoss.org!vospace!cholmondeley/votable1 | vos://nvoss.org!vospace!auberon/votable2 |
Cholmondeley |
vos://nvoss.org!vospace!auberon/votable1 | vos://nvoss.org!vospace!berengaria/votable2 |
Of course, in real life you would have done a listing of the VOSpace from which you are transferring data.
We're now going to follow what Auberon does - firstly he needs to talk to Cholmondeley's VOSpace to tell it that data is going to pushed to it and get an HTTP endpoint to receive it:
> ./vospaceclient pushtospace http://192.168.1.202:8008/VOSpaceServiceImpl Push data to space ================== Define the node into which data is to be imported: What type of node (plain, data, unstructured, structured): structured What is the identifier for the node (hitting return will cause the space to generate an identifier): vos://nvoss.org!vospace!cholmondeley/vospace2 Now for the node properties: Identifying URI for property (blank means no more properties): Now specify the transfer details: What is the identifier for the data format: ivo://net.ivoa.vospace/views/tabular/votable-1.1 Now for the protocols: Identifying URI for protocol (blank means no more protocols): ivo://net.ivoa.vospace/protocol/http-put Identifying URI for protocol (blank means no more protocols): The node into which data is to be imported is now defined as: Identifier: vos://nvoss.org!vospace!cholmondeley/vospace2 Type: StructuredDataNodeType Properties: No properties are defined. Accepted data formats: ivo://net.ivoa.vospace/views/image/png-1.0 ivo://net.ivoa.vospace/views/image/jpeg-1.0 ivo://net.ivoa.vospace/views/image/fits-image ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table Provided data formats: ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table The operational details of the transfer are: Format: ivo://net.ivoa.vospace/views/tabular/votable-1.1 Protocol: ivo://net.ivoa.vospace/protocol/http-put Endpoint: http://192.168.1.202:7007/ece6fa21-ec58-4ef2-9e37-dfb778de4e74
Now that Auberon has the transfer details, he just needs to tell Berengaria's VOSpace to push the data from itself to the HTTP PUT endpoint that Cholmondeley's VOSpace provided him with:
> ./vospaceclient pushfromspace http://192.168.1.201:8008/VOSpaceServiceImpl Push data from space ==================== What is the identifier of the node from which data is to be exported: vos://nvoss.org!vospace!berengaria/vospace1 Now specify the transfer details: What is the identifier for the data format: ivo://net.ivoa.vospace/views/tabular/votable-1.1 Now for the protocols: Identifying URI for protocol (blank means no more protocols): ivo://net.ivoa.vospace/protocol/http-put What is the endpoint for this protocol: http://192.168.1.202:7007/ece6fa21-ec58-4ef2-9e37-dfb778de4e74 Identifying URI for protocol (blank means no more protocols):
Auberon can now do a listing of Cholmondeley's VOSpace to see if the data is there:
> ./vospaceclient listnodes http://192.168.1.202:8008/VOSpaceServiceImpl List Nodes ========== What level of detail do you want in the listing (min|properties|max): max Identifier: vos://nvoss.org!vospace!cholmondeley/votable2 Type: StructuredDataNodeType Properties: No properties are defined. Accepted data formats: ivo://net.ivoa.vospace/views/image/png-1.0 ivo://net.ivoa.vospace/views/image/jpeg-1.0 ivo://net.ivoa.vospace/views/image/fits-image ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table Provided data formats: ivo://net.ivoa.vospace/views/tabular/votable-1.1 ivo://net.ivoa.vospace/views/tabular/fits-table
and/or he can ask Cholmondeley to check that his data is in the VOSpace and looks OK.
The NVO Summer School is made possible through the support of the National Science Foundation and the National Aeronautics and Space Administration.
 |
 |