Create generated SPARQL search pages for the info-filter, so that they can be directly linked to from the trac.

Update the search link to the real RDS/WIP rdl endpoint search.

Changes made to the storage manager to support the PCA-RDL conversion need to be tested for the effects on the servlet deployment.

Support for templates/relationships and class of relationship required in default presentation xsls.

Step-by-step content is available via the command line interface, but not via the web interface - enable it for web.

Copyright needs to be declared properly for each fragment, and we need a uniform policy for copyright on contributions.

Provide an automated means of exploring all of the links - maybe just use 3rd party tools for this.

The InfoFilter needs a better way of getting the model and/or we should make sure that the base URI of the model in Jena/MySQL is set up properly.

Errors and warnings currently go to the console or to the app. server logging mechanism - should intercept errors and warnings etc. for use by the application for inclusion in reporting (perhaps pass variable as document node).

SOAP implementation requires SOAP Fault handling for exceptions not related to output stream.

The stability of the MySQL driver is a bit of an open question - its prone to catastrophic and permanent failure (until VM restart) if it blows its heap during a query. Why heap is blown is unknown - it could be the Joseki/Jena infrastructure, or it could be the MySQL client side state - probably the former.

Jena needs better database connection recovery for MySQL - solution right now is to reboot Jetty when it messes up.

Presentation performance is adequate for items with a small number associated instances, but is very poor when the number of related things increases - need to find a way of streamlining the SPARQL queries to reduce latency. Can also improve the template compilation. Note that transforms can be done in parallel, so that is a way of dealing with latency too.

A "Clear Model" button is needed in the GWT UI. Also need the union/add/replace option on the upload dialog.

Current implementation blurs models and datasets, and in fact may well mis-use model names in the API. Need to correct the uploader to use a dataset with default model rather than plain model, if that's what's happenning.

Currently, the implementation only supports manually mapping the Joseki services - we need to find a way of mapping them automatically out of a single database.

Modify the MySQL specific model setup scripts in the Jena source to generate the required full text indexing over the two different model columns (short & expanded). Consider integrating SDB?

Provision for reporting a suite of WSDL files concerning an endpoint to Web Servics clients.

Make the server the authoritative source of DNS records for the ids-adi.org domain by installing a BIND service and documenting configuration. Will also need to transfer domain name ownership to IDS-ADI.

Provide documentation for building mappings such as having http://dm.rdswip.ids-adi.org/data proxy opaquely to http://rdswip.ids-adi.org/servlet/endpoint/dm.

Configure the logging for Jetty, J2SE1.4, Log4J and all the other logging components to route to the syslog - currently its routing to stdout/stderr and then getting forwarded to the syslog.

Load the latest PCA RDL data into the test endpoint. Loaded data - took approx 1hr - no timeouts or other hassle - inserts into mysql are the limiting factor and seem to pro-rate the upload speed, keeping the connection active.

The upload form needs a drop down with union/add/replace options to allow the data to join the existing data, without creating duplicates, add to the existing data (duplicate triples allowed) or completely remove and replace the existing data. Actually though, it looks like union is what occurs automatically, unlike RAP, so "add" is currently just like "union" right now.

The result page for the upload form is currently empty - would be nice to say "loaded xxx triples" or some such. Ended up implementing it so that it reports any triples removed and the net triples added.

Current servlets require individual configuration options - should merge them into the StoreManager I think. Note: couldn't merge into StoreManager due to changes in the servlet API that restrict servlets from finding other instantiated servlets by name - used context params instead.

Automated SOAP support on any SPARQL endpoint. Added the SOAP support, but didn't test - just used known operable code.

Load the latest PCA RDL data into the test endpoint, but with changes to use the endpoint as the base URI. Also modified the SQL tables backing this endpoint to support searching on the object column.

There's a problem with MySQL handling content from Jena that is flagged poorly with respect to character encoding. Turns out this was solved by restarted mysqld (and jetty didn't recover).

Build some infrastructure to take a URI, apply a set of SPARQL queries to the inherent endpoint and project the results using XSLT into a general XML format, with final XHTML presentation via the scripts similar to the IRM ones. All SPARQL & XSLT should reside in the presentation tree. Working now - very neat little framework - all SPARQL and XSLT integrated and control the XML production from the endpoint.

Test SOAP support using standard client.

Would be very useful to allow the info filter to run in standalone mode, as a .jar outside of the server. Done and used in the test scripts committed to svn.

Improved performance dramatically by optimizing the order of the statements in a SPARQL join - shouldn't (theoretically) have any effect, so this is exposing either an issue in the ARQ Jena implementation or MySQL.

Imported part2 directly into the PCA RDL for ease of use - future this should be managed with named graphs or something similar.

Modified implementation so that it is a real, user-oriented search, not a SPARQL query. Modified the scripts to provide an entry point for the search.

Modified implementation so that it is a real, user-oriented search, not a SPARQL query. Modified the scripts to provide an entry point for the search.

Would be useful to have a way of extracting the steps of the filter as a single streamed zip file, complete with error output, ordered by step number. Actually implemented this as "application/zip" or as "multipart/mixed" so that since most of the data is text it can be delivered either way - very large numbers of "steps" are typical (eg. I think electric motor gives 240 or so).

Extensive VM caching for objects frequently re-used within the context of a single query (compiled XSLTs). Extensive VM caching for objects reused between queries (ARQ queries). Extensive FS caching for SPARQL/XSLT transform results. All caching set up to be automatically cleared on app. server restart. Considerable performance gains. FS caching is incomplete implementation - relies on SHA1 too much. All caching needs an external bypass & flush.

Cache flush implemented at /servlets/admin/flush-cache.

App. server needs remote restart capability for authorized security principals implemented at user level (not root). Implemented - see /presentation/admin/index.html for details.

Also need to put hash collision detection in SHA1 FS caching implementation - done, using simple raw binary data bucket indexing to store full key. Position of key determines name of resource.

Caching system operates, but needs a bypass option passed through UI. Implemented as a cache parameter that can take the following words:

• normal: read/write cache
• bypass: do not use or change cache
• fresh: update cache, do not use values from cache
• fetch: use values from cache, do not update cache

Also, Pragma and Cache-Control HTTP headers with a "no-cache" value will implicitly add a cache=fresh parameter pair. For most browsers, this means that holding down the shift button and pressing refresh will have the expected behaviour (clears any existing cache entries and posts new cache entries).

Added idemopotency control over generated content (ability to suppress meta content about the generation, like dates/times), in order to provide for controled testing of the caching system while using productions that normally include aforementioned meta content.

Modified the main CSS file - table and list content is looking a little better. Also added (bogus) copyright info and some generated summary info - could probably add more summary info.

Added a whole bunch of infrastructure to better support authentication at every level of the OS and services. Basically it is LDAP integrated with PAM, NSS & Apache and configured with PHPLDAPAdmin (in Apache). Lots of little config details to keep track of. Theoretically, this could be opened up for other purposes (eg. authentication for trac server).

An automated, incremental off-site backup system has been put in place for the critical content filesystems.

SSL Certificate purchased from Thawte and installed. All authenticated services moved to https://secure.ids-adi.org, including exposed repository. Shared structures set up for consistent security application between vhosts. Repository authorization rules centrally sourced.

Added some color; softened the titles, changed the body to a sans-serif font, made links more obvious.

Provision for reporting a suite of WSDL files concerning an endpoint to Web Servics clients. Added - still imperfect - need a way of mapping the client request URI (I think). Any endpoint served up by the joseki-soap-lite code should respond to both ?wsdl and /wsdl.

Fixed SOAP failures introduced by WSDL Support.

Added main web page logo image, background image and favourites icon.

Fleshed out generated pages with consistent heading fragments, and other pages with some semi-consistent headings and icons from the materials above. Altered CSS to suite.

SQL Server is now backed up off-site daily (in addition to critical content filesystems as above).

Disabled the header fragment in the generated pages since it seemed to be causing strange server side deadlocks (in Apache).

Added a page describing the general layout and assembly of the system as an operating whole.

Added a page describing how to administer accounts on the system, for the four most typical operations: new user, add to group, remove from group and disable user.

InfoFilter Cache is bleeding queries between endpoints - needs to be sealed up. Fixed and tested. Results from different SPARQL query endpoints are independent now, however note that different invoking endpoints shared cached results from the same query endpoints.

Some extra functionality requested on the ID generator, including: an operating unbind; a popup menu for base URIs; a config xml that shows the configured base URIs and other info; and an info option that shows what has been said about a given ID. All of these have been implemented now, plus there has been a change in the permissions for identifiers - only the original requester can revoke now, and only until they have fixated. Once they have fixated, then other people can bind.

PCA-RDL to RDS/WIP Converter largely completed and entered into testing phase on clean room system.

Infrastructural refactoring for PCA-RDL converter to support config, logging, RDF writing and non-heap mapping. Solve scalability issues with Jena and configuration and reporting issues with existing IDS-ADI framework.

With the converter code ready and tested on the clean-room install, ran the converter on the server machine with fresh data as at 2008-08-18 from PCA. RDS/WIP SPARQL endpoints are now live for raw HTTP access.

Enabled SOAP for RDS/WIP endpoints.

The InfoFilter-based UI (XSLT+SPARQL) is now in place for the RDS/WIP. There is a rudimentary search functionality too, but performance is poor - should be able to be improved with DB tweaking. There is also a lot of "junk" in the data that will have to be carved out with another pass of the converter at some point.