Setting up the Hayabusa 2 Deployment Server (in Aizu, Japan) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ======================================================== Deployment Server Access ======================================================== - Log on to the deployment server as “administrator” (i.e. with sudo access) - log on to a DMZ machine (which allows ssh to external sites) - from the DMZ machine, ssh in to the deployment server machine at Aizu $ ssh apl@hyb2sbmt.u-aizu.ac.jp ———————————————————————————--------- Install 3rd-party software on server ———————————————————————————--------- - Logged in to the deployment server machine, proceed to set up the Apache 2 web server: - install the apache web server (This was already done for me by the Aizu administrator, but if it hadn’t, I would have done this:) $ sudo apt-get update $ sudo apt-get install apache2 - Verify that the apache web server is running by bringing up the URL http://hyb2sbmt.u-aizu.ac.jp/ in a browser You should see the standard default apache web server page - Create a new site configuration for the web server $ pushd /etc/apache2 $ cd sites-available/ - Copy over the default site configuration to a new “sbmt” one and modify $ sudo cp 000-default.conf 001-sbmt.conf $ sudo vi 001-sbmt.conf - Edit the new virtual host module (001-sbmt.conf) to point to a new web server subdirectory: DocumentRoot /var/www/sbmt - Replace the default module with the new one and restart the web server $ sudo a2ensite /etc/apache2/sites-available/001-sbmt.conf $ sudo a2dissite /etc/apache2/sites-available/000-default.conf $ sudo service apache2 reload - Copy the default web server page to the new sbmt server directory and give sbmt access to it $ cd /var/www $ sudo cp -r html sbmt $ suso chown sbmt $ sudo chgrp sbmt sbmt $ sudo chmod g+w sbmt $ sudo chown sbmt index.html $ sudo chgrp sbmt index.html $ sudo chmod g+w sbmt/index.html - Give the admin account access to the sbmt group and verify $ sudo usermod -a -G sbmt apl $ groups apl - Enable htaccess control in the web server $ cd /etc/apache2 $ sudo vi apache2.conf - Make sure the .htaccess file name is set: AccessFileName .htaccess - Set the sbmt directory to allow override: Options Indexes FollowSymLinks AllowOverride All Require all granted - Restart the web server sudo service apache2 restart - From the sbmt user account, modify the content of the sbmt server index.html page to show that it is the SBMT deployment server page $ exit $ ssh sbmt@hyb2sbmt.u-aizu.ac.jp $ cd /var/www/sbmt $ vi index.html modify the index.html file HTML text to say “Small Body Mapping Tool - Hayabusa 2 Deployment Server” - Bring up the web server http://hyb2sbmt.jhuapl.edu/ in a browser to verify it ———————————————————————————--------- Set up access control ———————————————————————————--------- - Create a new .htacces file to control access $ cd /var/www/sbmt $ vi .htaccess - Add the following contents to the .htaccess file: AuthUserFile /var/www/sbmt/.htpasswd-sbmt AuthType Basic AuthName "Please Enter SBMT Password" Require valid-user Order Allow,Deny require user ernstcm1 require user peachjm1 require user zimmemi1 require user turnerj1 require user nguyel1 require user steelrj1 require user hayabusa2 require user naru.hirata Satisfy any - Create a new .htpasswd-sbmt file $ vi .htpasswd-sbmt - Add the following content to the .htpasswd-sbmt file: ernstcm1:VZvknzxEu.XGo barnoos1:/IDi916LPUFsQ naru.hirata:$apr1$f.LcEpkY$BxqH9EL4gsnYj5Fs3C.nj. turnerj1:cb6KKiFtTB8E. nguyel1:0sFjqKJ3n2a26 zimmemi1:$apr1$.nmOwGMk$hD54U3o.m8KbGCcG7gnPK. steelrj1:$apr1$HdEL6Ma2$9QRMZ1t0n8Z9.Zpez5yIV1 hayabusa2:$apr1$4kuBOJt6$rSXX8t8d2mTxZJwDFNFJh/ peachjm1:$apr1$vYb4rTIN$AMOvdiTm9.avxR./Uxenj0 - In a web browser, try to access the http://hyb2sbmt.u-aizu.ac.jp/ and verify that it prompts you for a password - Enter your SBMT password and verify that you can see the server web page - Create the sbmt data subdirectories $ mkdir sbmt $ cd sbmt $ mkdir data $ cd data ———————————————————————————--------- Upload simulated data to server for testing ———————————————————————————--------- - From the APL DMZ machine, zip up and scp the Earth and Ryugu data and copy to the Aizu server $ cd /project/nearsdc/data $ zip -r earth earth/hayabusa2 $ zip -r ryugu.zip ryugu $ scp earth.zip sbmt@hyb2sbmt.u-aizu.ac.jp:/var/www/sbmt/data (and enter the password) $ scp ryg.zip sbmt@hyb2sbmt.u-aizu.ac.jp:/var/www/sbmt/data (and enter the password) - From the Aizu server, as sbmt user, unzip the data files: $ cd /var/www/sbmt/data $ unzip earth.zip $ unzip ryugu.zip - Test the Hayabusa2 Deploy release of SBMT and verify that you can access the Earth and Ryugu Bodies ———————————————————————————--------- Install Image Database ———————————————————————————--------- - Set up the image database and confirm it is running $ sudo apt install mysql-server Specify a MySQL password. I am using the same password as the "sbmt" password on the deployed server. $ sudo netstat -tap | grep mysql - You should see the following response: tcp 0 0 localhost:mysql *:* LISTEN 22355/mysqld - Install php $ sudo apt install php $ sudo apt-get install libapache2-mod-php $ sudo systemctl restart mysql.service - Install phpmyadmin $ sudo apt-get update $ sudo apt-get install phpmyadmin php-mbstring php-gettext In the installation screen that appears, you MUST select "apache2" for server by hitting the space bar (it is NOT selected by default) and select to configure the database with dbconfig-common Specify the login password. I am using the same password as the "sbmt" password on the deployed server. $ sudo phpenmod mcrypt $ sudo phpenmod mbstring $ sudo systemctl restart apache2 You can verify that phpMyAdmin is running by browsing to the phpMyAdmin web page: http://hyb2sbmt.u-aizu.ac.jp/phpmyadmin and logging in as "root", using the same password as the "sbmt" password (on the deployed erver). - Create the database “sbmt” using phmMyAdmin Logged in to phpMyAdmin as root, select the “Databases” tab Enter “sbmt” for the database name and press the “Create” button - Create additional users and give them full database privileges so they can populate the database Logged in to phpMyAdmin as root, select the “User Accounts” tab Click the “Add user account” link Enter the user name, password and re-type password. The password will be used by users to log in to phpMyAdmin as well as to populate the database with the GenerateDatabaseSql utility. Leave the Host name field as “Any host” (%) For the Global privileges field, check the “Check all” checkbox Press the “Go” button - Create an “smbt” database user for search only purposes. This will be used by the PHP search script. When an image search is executed by the SBMT client. Logged in to phpMyAdmin as root, select the “User Accounts” tab Click the “Add user account” link Enter “sbmt” for the user name, give it a password that is different from the other account passwords (since this password will be stored in the clear) and re-type password. The password will be used by the PHP script to connect to the database to do queries. Leave the Host name field as “Any host” (%) For the privileges section, only check the Data “select” and “file” checkboxes. Press the “Go” button - Modify Apache web server to allow PHP access $ sudo vi /etc/apache2/mods-enabled/dir.conf In dir.conf, add “index.php” to the DirectoryIndex list: DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm $ sudo systemctl restart apache2 - Edit the MySQL configuration file to allow remote access $ sudo mv /etc/mysql/my.cnf /etc/mysql/my-orig.cnf $ sudo vi /etc/mysql/my.cnf Add the following text to the my.cnf file: [mysql] user = mysql socket = /var/run/mysqld/mysqld.sock port = 3306 bind-address = 10.109.29.242 …where the bind-address value is the IP address of the server machine. Restart MySQL: $ sudo /etc/init.d/mysql restart ????????????????????????? - Open up port on firewall if it is blocking access $ sudo /sbin/iptables -A INPUT -i eth0 -p tcp --destination-port 3306 -j ACCEPT ????????????????????????? - Modify the searchimages.php script to access the Hayabusa2 server database and upload to server This script has to be tailored for each SBMT server with the database name and credentials. Also, the internal version of the script uses an out-of-date PHP mysql access library, so newer SBMT servers need to use the new version. I have created subfolders in the SBMT repository under the the “misc/server/query” folder for each mission to store the PHP script for its server. Use the script in the “hayabusa2” subfolder for the Hayabusa2 servers. Copy the Hayabusa2 “searchimages.php” file into the /var/www/sbmt/sbmt/query directory on the Hayabusa2 server. For other servers than Hayabusa2, modify the $host and $password string variabls as appropriate. ———————————————————————————--------- Upload new data to the server ———————————————————————————--------- Once the server is up and running, new model, image and related data can be uploaded by running scripts on both the APL and Aizu sides. - Run the Ryugu data processing scripts on the APL DMZ linux machines 1) Log in to a DMZ machine (e.g. kenny.jhuapl.edu) using your own (not sbmt) credentials ssh kenny.jhuapl.edu 2) Change directories to the Ryugu raw data directory $ cd /project/sbmtpipeline/rawdata/ryugu 3) Run the delivery2rawdata script for any model data (shape, coloring, etc.) $ ./delivery2rawdata-ryugu.sh where is the name of the model as delivered in the /project/sbmtpipeline/deliveries/ryugu directory (e.g. NASA-001), and is the 8-digit string representing the deliver date (e.g 20180702) 4) Run the delivery2rawdata script for any shared image data $ ./delivery2rawdata-ryugu.sh shared 5) Run the rawdata2processed script for any model data $ ./rawdata2processed-ryugu-rt.sh (e.g. ./rawdata2processed-ryugu-rt.sh nasa-001 20180702) - Upload the data to the Aizu server 1) Log back on to the DMZ linux account with the "smbt" credentials (you will need to get the password) $ ssh sbmt@kenny.jhuapl.edu 2) Navigate to the /project/sbmtpipeline/rawdata/ryugu directory and run the remote deploy scripts $ cd /project/sbmtpipeline/rawdata/ryugu $ ./processed2deployed-ryugu-remote.sh aizu for model data, (e.g. ./processed2deployed-ryugu-remote.sh nasa-001 20180702) $ ./processed2deployed-ryugu-remote.sh shared aizu for shared image data. Note that aizu is the only remote location parameter that works currently, but this you still must enter it as a parameter. - Process the data on the Aizu server 1) From the DMZ machines, logged in with the "sbmt" credentials, ssh in to the Aizu server $ ssh sbmt@hyb2sbme.u-aizu.ac.jp No password should be required, because this account on kenny has shared SSH keys. 2) Navigate to the scripts directory and run the server-side model data processing script $ cd /var/www/sbmt/sbmt/scripts/ $ ./processed2deployed-ryugu-aizu.sh (e.g. ./processed2deployed-ryugu-aizu.sh nasa-002 20180724) This sets up the appropriate soft links within the model data directory. Note that the shared image data does not require any server-side processing script. 3) For new models that haven't been deployed before, make the model "live" by setting a soft link $ cd /var/www/sbmt/sbmt/data/ryugu $ ln -s - (e.g. ln -s nasa-001-20180628 nasa-001) ———————————————————————————--------- Populate the database with image data ———————————————————————————--------- The image database populating tool cannot, apparently, be run remotely. Also, it must be altered every time a new model is added. So, each new version must be built and copied up to the Aizu server for running locally. - Modify the DatabaseGeneratorSql program. 1) In Eclipse, open the DatabaseGeneratorSql.java class and modify the RunInfo enum to add an enum statement for each new model. The name of the enum will be passed as a parameter when running the program from the command line. The first parameter of the enum constructor is the ViewConfig of the model. The second parameter is the absolute path on the Aizu server of the imagelist-fullpath-sum.txt image list file. The third parameter is the database table name suffix for the model. The fourth parameter is the relative path of the image list for remote access, which is not used for the Aizu models, since they can not be accessed remotely. 2) Check the new version of the DatabaseGeneratorSql tool in to the code repository. - Build the DatabaseGeneratorSql software 1) Log on to the kenny machine on the DMZ using the "sbmt" credentials $ ssh sbmt@kenny 2) Create a personal build directory to build the new version $ mkdir /project/sbmt2/users/ 3) Navigate to your personal build directory and check out the sbmt and saavtk projects $ cd /project/sbmt2/users/ $ git clone http://hardin:8080/scm/git/vtk/saavtk --branch sbmt1dev-redmine- $ git clone http://hardin:8080/scm/git/sbmt --branch sbmt1dev-redmine- 4) Start the bash shell and set the ROOT environment variables to your personal build directory $ /bin/bash $ export SBMTROOT=/project/sbmt2/users//sbmt $ export SAAVTKROOT=/project/sbmt2/users//saavtk 5) Set the mission of the source code to the HAYABUSA1_DEPLOY mission $ $SBMTROOT/misc/scripts/set-released-mission.sh HAYABUSA2_DEPLOY sbmthyb2 6) Make a release in order to build the DatabaseGeneratorSql code $ cd sbmt $ make clean $ make release 7) Copy the release libs and jars up to the Aizu server $ rsync -vrt build/jar jar sbmt@hyb2sbmt.u-aizu.ac.jp:/var/www/sbmt/sbmt/build $ rsync -vrt lib sbmt@hyb2sbmt.u-aizu.ac.jp:/var/www/sbmt/sbmt/ - Run the Database Generator on the Aizu Server 1) Using your Hayabusa2 server database credential, log on to the Hayabusa2 phpMyAdmin web page at: http://hyb2sbmt.u-aizu.ac.jp/phpmyadmin 2) Navigate to the "sbmt" database and verify that the table corresponding to your new model doesn't exist, or already exists if you are updating the images for an existing model. 3) Log on to the Aizu server from the DMZ linux account $ ssh sbmt@hyb2sbme.u-aizu.ac.jp 4) Set the database login credentials From your sbmt account on the Aizu server, check to see if there is a .sbmthyb2 directory - If it doesn't have a .sbmthyb2 directory, create one. - In that directory, add a text file (if it doesn't exist already) called "mysql-login.txt" containing the following two lines: smbt 5) Navigate to the scripts directory and run the database generator $ cd /var/www/sbmt/sbmt/scripts/ $ ./run-hayabusa2-deploy-program.sh edu.jhuapl.sbmt.tools.DatabaseGeneratorSql --debug --append-tables GASKELL where is the enumeration value that you added to the DatabaseGeneratorSql program. The run-hayabusa2-deploy-program.sh script is an Aizu server specific version of the run-on-linux.sh script used for running SMBT standalone tools on the DMZ linux machines. 6) Also, populate the shared Spice-based (INFO-file-based) image data, if there has been any new images $ ./run-hayabusa2-deploy-program.sh edu.jhuapl.sbmt.tools.DatabaseGeneratorSql --debug --append-tables SPICE RYUGU_SHARED There are also some scripts in the same directory for populating specific models (e.g. generate-db-hyb2-shared.sh), which can also be used. 7) Test the client - From Eclipse, run the "SmallBodyMappingToolAPL - Hayabusa2 - Deploy" runconfig - Go to the new body model - Select the ONC image tab and do some searches for images (SPC and Spice based) - Verify that images are returned by the image search and that they load correctly