## SBMT Guide to Adding Data

Please update this document as needed to clarify steps and add details, or when procedures change. 

##### Reference documents
* releaseinstructions.md
* SBMT-Guide.docx (out of date)

##### Directories
Place data received from scientists into the `/project/sbmtpipeline/sbmt/rawdata` directory. Generate intermediate output from scripts to directory `/project/sbmtpipeline/sbmt/processed` before moving to live data directories.

User directories are located at `/project/sbmtpipeline/sbmt/users`. 

Directory structure for bodies and images is as follows (TBD: update when sbmt2 server is complete):

      BODY_DIR/
      	<shape and coloring files>
      	IMAGER_DIR/
      		imagelist.txt
      		sumfiles/
      		infofiles/
      		images/
      		gallery/

### Adding a Body

##### Client

1. Update configuration (SmallBodyViewConfig.java) to add menu items and tabs.

##### Server

1. Run gravity script to generate shape model resolutions and compute plate colorings and coordinate grid at each resolution level.
2. Update /sbmt/misc/server/sbmt/internal/content/references.md with body constants used and data source.

* A Bennu-specific gravity script is checked in at /sbmt/misc/programs/gravity/runBennuGravity.sh. Data directory paths, rotation rate, and reference potential must be updated for other bodies. 

* The gravity script calls the DistributedGravity java class, which is located in the altwg-java-tools.jar library, maintained by Ray Espiritu and located at http://hardin/svn/sis/projects/osirisrex/ola/altwg/trunk/java-tools. Contact Ray before making any changes to his codebase.

##### Test/Checklist

1. Select the body from the menu and verify resolutions, plate colorings, coordinate grid, and other features on the body tab.

### Adding Images

##### Client

1. Create subclass of PerspectiveImage for the new imager. Override methods that set sumfile path and infofile path. Override other methods as needed. 
2. Update the configuration (SmallBodyViewConfig.java) to add a new imager. Specify image path and search query type. Search query may be either FixedListQuery (use initially for testing images in local .neartool/cache directory, no search capability) or GenericPhpQuery (for database image search). Update DatabaseGeneratorSQL.java to add directory path and image list name for database searches.
2. For database queries, set the default search configuration (SmallBodyViewConfig.java) to return all images when no region is selected (i.e. set the start and end date wider than the data time range, the S/C distance and resolution upper bounds for the furthest image, etc).

##### Image Lists

Both fixed list and database queries require a text file containing a list of images. The formats differ and are as follows: 

     * The FixedListQuery file name must be "imagelist.txt" and must reside in the imager path. 
     * This file name is hard-coded in SBMT.
     * Data format:
     *      <fitsFileName> <imageTime>
     * For example:
     *      mc0_0034950778_0x536_sci_1.fit 2007-02-28T06:41:14.454
     *      mc0_0034951618_0x536_sci_1.fit 2007-02-28T06:55:14.454
     *      ...
     *
     * The database image list file name is specified in the DatabaseGeneratorSQL.RunInfo enum.
     * Data format:
     *      <relative-path>/<fitsFileName>
     * Where relative-path is relative to the SBMT root data path. For example:
     *      /GASKELL/ITOKAWA/AMICA/images/st_0009955932_v.fit
     *      /GASKELL/ITOKAWA/AMICA/images/st_0015411418_w.fit
     *      ...

The database is at http://ripley.jhuapl.edu/phpMyAdmin/. 
Note that the DatabaseGeneratorSQL will NOT store data for images that do not intercept the body.

##### Server

1. Place images in live directory specified in client.
2. Create an image list for the query type specified in client.
3. For database query, run the shell script DatabaseGenerator.sh located in SBMT build's bin/ directory. Execute without parameters for usage. The default is to generate tables with names ending in "\_beta". Once tested with the SBMT Beta run configuration, rename tables to remove the "\_beta" ending. 
4. Update /sbmt/misc/server/sbmt/internal/content/references.md with image data source.
5. Create the image gallery thumbnails by running the repository script sbmt/misc/programs/fits2thumbnails.py.
6. Create the image gallery webpage by running the repository script sbmt/misc/programs/make\_gallery\_webpage/make\_gallery\_webpage.py.  (TBD: verify this step)

_For GASKELL (sumfile) pointing:_

1. Place sumfiles in the live data directory specified in the client.

_For SPICE (infofile) pointing:_

1. Collect the required SPICE kernels. For PDS images (Planetary Data System), the kernels and their locations will be specified in the image archive.  
2. Execute a copy of script create\_info\_files.cpp. There are multiple, customized versions of this script located in the repository at sbmt/misc/programs. Each mission's script is slightly different. Scripts compile.sh and run.sh also exist to compile and execute the C++ code. 
3. Move the resulting infofiles to the live data directory specified in the client.

##### Test/Checklist

1. Confirm that images are mapped onto the body correctly: Selecting a high resolution for the body and map images that contain distinguishable features. Toggle the "Show" and "Boundary" columns in the image search results. The image should map identically onto the features. 
2. For database query type only, verify that the region search displays only images that overlap the selected region. Also test a few of the query filters (date range, S/C distance, etc) and verify the search results. Verify that any images that did NOT make it into the database do not contain the body. This can be done with a FITS viewer. DatabaseGeneratorSQL will print to screen those images.
3. From the GUI menu, create an image gallery from the search results and verify that it automatically brings up a web page containing the images in the search results.
4. From the SBMT website, verify that the full image gallery is viewable.

### Adding Lidar Data