Smart Server Defense
Powered by BlueIce3
Configuration Documentation

This page has extra information on the configuration, training, and execution of the BlueIce2 core AI, details on how to test the BlueIce2 MySQL database setup, as well as info on the configuration of the different components.
Default SSH Port: 50001 (AWS AMI Defaults to 22 automatic SSHD port checks are now disabled by default.)
Default MySQL Port: 50004 (localhost only)
Default Apache Port: 80
Contents
Checking The Base Installation   top
Below is a listing of the core requirements, base installation, for the BlueIce3. You can use this list to check your server configuration and make sure the requisite software is installed. Some of the packages are required for the BlueIce3 suite, some are part of an effort to make a useful bare bones system with BlueIce3 and expected server functionality ready out of the box.
The core software requirements, base installation, are as follows...
  1. Directory Structure: Check that the following directories have been create in the root directory where the install script resides.
    ./apps/python/BlueIce2: BlueIce2 AI software.
    ./db_baks: Repository for database backups.
    ./apps/python/BlueIceUtils: Automation tools for BlueIce2.
    ./apps/scripts/SmartServerDefenseCfg: Documentation, web site for BlueIce2.

    The base install directory structure is as follows.
    blueice3
    -> 0_5_0_9 : version code dir
        -> blueice3_install : install script, local
        -> db_baks : blueice2 mysql db backups, local
        -> apps
            -> python
                -> BlueIce2 : blueice2 AI core
                    -> checkpoints : Tensorflow checkpoints saves
                    -> data : main app data dir
                        -> access_logs : training, apache access logs
                        -> config : sample config file dir
                        -> evaluate : normal apache access logs to evaluate
                        -> other_vhosts_access : training, apache other vhosts access logs
                        -> output : output directory for writing the bad IP address list
                -> BlueIce2Utils : blueice2utils support app
                    -> data : main app data dir
                        -> config : sample config file dir
                    -> test : main unit test dir
            -> scripts
                -> SmartServerDefenseCfg : server configuration project (meta)
                    -> blueice3_install : install script, latest
                    -> db_baks : blueice2 mysql db backups, latest
                    -> db_migrations :  blueice2 migration files for new db features
                    -> db_unit_tests : SQL unit tests for the db stored procedure, etc
                    -> errata : Various system sample files, vimrc, motd-00-header, etc
                    -> SmartServerDefenseWeb : documentation and blocked ip listing
            
  2. Apache2: Check that the following packages have been installed.
    apache2

  3. PHP: Check that the following packages have been installed.
    php7.1
    php7.1-common
    libapache2-mod-php7.1
    php7.1-cli
    php7.1-mcrypt
    php7.1-pgsql
    php7.1-curl
    php7.1-mysql

  4. PERL: Check that the following packages have been installed.
    perl

  5. GIT: Check that the following packages have been installed.
    git

  6. MySQL: Check that the following packages have been installed.
    mysql-server
    mysql-client
    php7.1-mysql

  7. GIT Repos (DEPRECATED): Check that the GIT repositories have been initialized. The proper GIT credentials for the dummy account can be found in the install script.
    git clone "https://USER:PASS@github.com/[account]/BlueIce2.git" ./apps/python/BlueIce2
    git clone "https://USER:PASS@github.com/[account]/BlueIce2Utils.git" ./apps/python/BlueIceUtils
    git clone "https://USER:PASS@github.com/[account]/SmartServerDefenseCfg.git" ./apps/scripts/SmartServerDefenseCfg

  8. CURL: Check that the following packages have been installed.
    curl
    libcurl3
    php7.1-curl

  9. MUTT: Check that the following packages have been installed.
    mutt

  10. UNZIP: Check that the following packages have been installed.
    unzip

  11. MLOCATE: Check that the following packages have been installed.
    mlocate

  12. PYTHON: Check that the following packages have been installed.
    python
    python-dev
    python-pip

  13. PYTHON LIBS: Check that the following python libraries have been installed via pip.
    tensorflow
    PyMySql
Checking The MySQL Configuration   top

App Config Files
The next thing we'll look into is how to double check the base configuration. We'll start with the configuration files that each GIT based application uses. These are created by the install process and should contain the application database account and some environment information depending on the config file in question.
Verify the location of the different config files, there should be a config.txt in each of the directories listed below.
./apps/python/BlueIce2/
./apps/python/BlueIce2Utils/
./apps/scripts/SmartServerDefenseCfg/SmartServerDefenseWeb/

The default config file entries for each application are as follows. The database connection information should match the information contained in the install script. While the settings can be changed they must be consistent between all three projects. The values below are not real running values.

Default Password Formulas...
#!/bin/bash
#Script for generating the default local application database password.
HOST=`hostname`
echo -n "bi2usr_${HOST}" | md5sum | cut -c1-32

#Script for generating the default root database password.
echo -n "root_${HOST}" | md5sum | cut -c1-32

For BlueIce2...
dbconn,dbServer=localhost
dbconn,dbPort=50004
dbconn,dbUser=bi2usr
dbconn,dbPassword=71uTupTu6
dbconn,dbName=blueice2
blueice2utilsexe=../BlueIce2Utils/Main.py
dbValidWebFiles=True
dbTrainingFiles=True
dbLogResult=True

For BlueIce2Utils...
dbconn,dbServer=localhost
dbconn,dbPort=50004
dbconn,dbUser=bi2usr
dbconn,dbPassword=71uTupTu6
dbconn,dbName=blueice2
blueice2exe=../BlueIce2/Main.py

For SmartServerDefenseCfg...
dbconn,dbServer=localhost
dbconn,dbPort=50004
dbconn,dbUser=bi2usr
dbconn,dbPassword=71uTupTu6
dbconn,dbName=blueice2
Python App to Database Connectivity   top
Next up we'll test our database connections from the different python applications. There is a web site and web service tier with some functionality but we are leaving it out of the initial release to focus on more core aspects of the project.
In order to test the database connection for the BlueIce2Utils application run the following command.
cd ./apps/python/BueIce2Utils
python Main.py -j testDbConn
You should see output like the following. If not you need to double check that the app database account was created successfully. Check that all conig.txt files have the same login credentials and make sure that those credentials work. Recreate the database user account if you have to.
BlueIce2Utils Application Version: 0.4.0.9_python2.7
Found job: testDbConn
Running job: testDbConn
Description: (u'Hello World!', 253, None, 12, 12, 31, False)
Row: Hello World!
Process finished with exit code 0
In order to test the database connection for the BlueIce2 application run the following command.
cd ./apps/python/BlueIce2
python Main.py -dbTestConnOnly
You should see output like the following. If not you need to double check that the app database account was created successfully. Check that all conig.txt files have the same login credentials and make sure that those credentials work. Recreate the database user account if you have to.
Testing database connection...
Description: (u'Hello World!', 253, None, 12, 12, 31, False)
Row: Hello World!
Process finished with exit code 0
Checking The Database Setup   top
Once we have made sure our app database account is working properly we'll want to check the database is setup correctly by running a series of unit tests on the different database tables. If things don't look good we'll restore the latest db backup from the project directory to our local MySQL server.
cd ./apps/scripts/SmartServerDefenseCfg/db_baks/
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_apache_log_files_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_bi3_input_clients.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_bi3_installed_versions.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_bi3_output_clients.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_blocked_ips_details_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_blocked_ips_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_blocked_ips_types_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_sshd_log_files_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_sshd_training_files_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_training_files_table.sql
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./unit_tests_valid_web_files_table.sql
The full list of SQL unit tests are as follows, the tests run stored procedures and perform simple assertions on the tables. There should be no 'Error' text in there output. Successfully performing the tests means that the stored procedures and underlying tables, SQL, etc, are functioning properly.
  • unit_tests_apache_log_files_table.sql
  • unit_tests_bi3_input_clients_table.sql
  • unit_tests_bi3_installed_versions_table.sql
  • unit_tests_bi3_output_clients_table.sql
  • unit_tests_blocked_ips_details_table.sql
  • unit_tests_blocked_ips_table.sql
  • unit_tests_blocked_ips_types_table.sql
  • unit_tests_sshd_log_files_table.sql
  • unit_tests_sshd_training_files_table.sql
  • unit_tests_training_files_table.sql
  • unit_tests_valid_web_files_table.sql
Each script should complete without error. We recommend using non-standard ports for the MySQL server and also only allowing localhost connection for the app database account. If you need to restore the database to its default run the following commands.
cd ./apps/scripts/SmartServerDefenseCfg/db_baks/
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost -e "DROP DATABASE blueice2;"
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost -e "CREATE DATABASE blueice2;"
mysql -D blueice2 -u bi2usr -p -P 50004 -h localhost < ./blueice2_bak_2018-03-29.sql
Verify the unit test scripts work with the restored database, also verify the python application database connectivity, as detailed above.
Checking The Web Service Setup   top
If you have run the extra steps necessary to setup and start the BlueIce3 website and web service tier this section will show you how to verify that it's up and running properly. First you can check the base URL, the default is to use the root URL for the current web server. You should see a copy of the smart server defense website and associated documentation.
http://localhost/SmartServerDefenseWeb/
http://[my IP address]/SmartServerDefenseWeb/
To check the web service tier access it without any arguments and you'll get a listing of available web service methods and their call signature.
http://localhost/SmartServerDefenseWeb/websvc.php
http://[my IP address]/SmartServerDefenseWeb/websvc.php
For more advanced verification of the web service tier and its functionality you can run the included unit tests listed below.
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_ApacheLogFiles.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_Bi3InputClients.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_Bi3InstalledVersions.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_Bi3OutputClients.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_BlockedIps.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_BlockedIpsDetails.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_BlockedIpsTypes.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_Other.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_SimpleCommands.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_SshdLogFiles.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_SshdTrainingFiles.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_TrainingFiles.php
http://localhost/SmartServerDefenseWeb/tests/php/UnitTests_ValidWebFiles.php
Each script should complete its run without printing an error output from any of the assertions. Running the unit tests will alter the data in your database so be careful when you use them, it's a good idea to backup the blueice2 database if you're not working with a fresh installation and have data you want to keep.
Python App Console Arguments   top

BlueIce2
We'll quickly go over the available command line arguments for BlueIce2. The options are as listed below. There cannot be any spaces between each console argument flag and its value if there is one. All directory scanning will ignore directories of the name 'unprocessed' and any local files.
  • -dbServer=[value]: Sets the database server, overriding what has been loaded from the local config.txt if available.
  • -dbPort=[value]: Sets the database port number, overriding what has been loaded from the local config.txt if available.
  • -dbUser=[value]: Sets the database user, overriding what has been loaded from the local config.txt if available.
  • -dbPassword=[value]: Sets the database password, overriding what has been loaded from the local config.txt if available.
  • -dbName=[value]: Sets the database name, overriding what has been loaded from the local config.txt if available.
  • -dataDir=[value]: Sets the location of the data directory, default value is './data'.
  • -apacheLogFile=[value]: Sets the location of the Apache server's access log file, default value is './data/evaluate/apache_eval_log.txt'. This is a copy of the current active log file or a sym link to the current active log file depending on your automation implementation.
  • -sshdLogFile=[value]: Sets the location of the system auth log file, default value is './data/evaluate/sshd_eval_log.txt'. This is a copy of the current active log file or a sym link to the current active log file depending on your automation implementation.
  • -webDir=[value]: Sets the location of the web document root, uses this directory for a file system scan when no database link to valid web files is provided.
  • -ipOutputFile=[value]: Sets the location for the IP output file, this file contains the list of IP addresses that should be blocked based on the last, incremental, log scan. The default value for the output file is './data/output/ip_block_list.txt'.
  • -lastLogRow=[value]: Sets the value for the last log row that was read, if the current log size is smaller than the last known log size it is assumed the log has been cycled and this value will be forced to 0.
  • -siteName=[value]: Sets the name of the site used to pull valid web files for that web site. In this way blueice2 lends itself to being run with separate configurations for each web site you want protected. Default value is 'default'.
  • -dbValidWebFiles: Toggles a flag that forces blueice2 to load its valid web file list from the database.
  • -dbTrainingFiles: Toggles a flag that forces blueice2 to load its training file list from the database.
  • -dbTestConnOnly: Toggles a flag that forces blueice2 only run a database connectivity test.
  • -skipEvalCallBack: Toggles a flag that forces blueice2 to skip the eval callback method.
  • -dbLogResult: Toggles a flag that forces blueice2 to record the last read log row of the run in the apache_log_files table. The next run will pick up after this row, unless a log cycle was detected then it will always start at row 0.
  • -logBatchSize=[value]: Sets the batch size of apache log files to process. Example, setting a value of 256 means that the program will wait until 256 logs rows have been read in before processing them with the AI engine. Recommended values for basic servers are 256, 512, 1024.
  • -exeToRun=[value]: Specifies which executable configuration to run. Adding executable configurations is supported but requires adding to the blueice2 python programming.
    • blue_ice_log_reg_apache_url: Runs blueice2 expecting to check url data against a valid list of current web files.
    • blue_ice_log_reg_apache: Alias for 'blue_ice_log_reg_apache_url'.
    • blue_ice_log_reg_apache_nourl: Out-of-the-box configuration ignores the URL and uses other log data to make a determination.
    • blue_ice_log_reg_sshd: Out-of-the-box configuration for processing SSHD logs and finding attacking IP addresses.
  • -silent: Turns on silent mode, this flag overrides the verbose flag. Silent mode will prevent as much logging as possible.
  • -verbose: Turns on verbose mode, this flag will enable extra logging where available.
BlueIce2Utils   top
The latest version of the BlueIce3 suite includes an updated command runner for the BlueIce2Utils python tool. Since there is a lot of functionality that this tool needs to support we abstracted the command interaction such that it supports JSON command descriptors. The legacy command set is also supported via the legacy CLI arguments as well as via the new JSON command descriptors. For a reference of the legacy CLI arguments go here. For examples of the CLI commands in use please review the python unit test files mentioned in the next section. All directory scanning will ignore directories of the name 'unprocessed' and any hidden files.
Main method entries are in some cases redundant, they will map to all of the original CLI commands that this tool could run as well as new commands that were only defined and not directly mapped to a CLI argument, as well as all the direct JSON commands listed below.
The available command line arguments for BlueIce2Utils. The options are as listed below. All '-j' job commands have been moved to the JSON command argument syntax listed below.
  • -dbServer=[value]: Sets the database server, overriding what has been loaded from the local config.txt if available.
  • -dbPort=[value]: Sets the database port number, overriding what has been loaded from the local config.txt if available.
  • -dbUser=[value]: Sets the database user, overriding what has been loaded from the local config.txt if available.
  • -dbPassword=[value]: Sets the database password, overriding what has been loaded from the local config.txt if available.
  • -dbName=[value]: Sets the database name, overriding what has been loaded from the local config.txt if available.
  • -neverBlock=[value]: A string with a list of IPs to never block, or a path to a file with each IP address on it's own line.
  • -ipOutputFile=[value]: Sets the location for the IP output file, this file contains the list of IP addresses that should be blocked based on the last, incremental, log scan.
  • -apacheLogFile=[value]: Sets the location of the Apache server's access log file, default value is './data/evaluate/apache_eval_log.txt'. This is a copy of the current active log file or a sym link to the current active log file depending on your automation implementation.
The updated JSON command argument list.
An example of running JSON command descriptors with the BlueIce2Utils program is as follows. You can chain a series of commands together with a JSON array. You can also reference values and counts, sizes, lengths from the results of previously run commands in the command series you've specified. Think of it as a simple session that exists during the current command series run. The unit tests for this console app show the syntax for every JSON command you can run. Similarly the web service, if you are using BlueIce3 to share blocked IP information run the same JSON command arguments with a few extra details. You can view the unit tests for the web service to see the syntax used to call every command the web service supports.
cd ./apps/python/BlueIce2Utils
python Main.py -rj "[{\"cmd\":\"AddActivateApacheLogFile\",\"targetFilePath\":\"./test_files/testing1.log\",\"targetFileName\":\"testing1.log\",\"fileSize\":100,\"lastModifiedOn\":\"2018-02-20 10:07:50\",\"lastLogRowRead\":10}]"
    
BlueIce3 Web Service Commands   top
The latest version of the BlueIce3 suite includes a web service based command runner for the PHP web service tier. This means that you can run almost all of the same commands as the BlueIce2Utils command line application with direct web service calls. There are a few commands that are spcific to the web service interaction, and all commands have a privacy setting that is not shown below. Most commands run as 'private' which requires the web service call to come from the local server or a trusted IP address. You can add entries to this list by editing the line in 'websvc.php' shown below.
$privateIpList = ["127.0.0.1", "localhost"];
Some web service commands require that the request be 'host locked', this means that the post has both of the receiving server's web service keys. This is to help protect against abusing the web service tier. An example of a JSON command post to the web service tier is as follows. The two host keys are defined by the receiving server, in this case the target server is registering an out going IP client. Notice that both of it's web service keys must be registered.
{
    "action": "rj",
    "key": "a420a25e6d541d79315b937d41f71ec0",
    "hostkey1": "2aac2e895b4d180c869fa0d37fba86ce",
    "hostkey2": "820c901b085022904d0aae84b4557956",
    "params":
    {
        "data": 
        [
            {
                "cmd": "DeleteAllBi3OutputClients"
            },
            {
                "cmd": "AddActivateBi3OutputClient",
                "ipAddress": "127.0.0.1",
                "webSvcKey1": "2aac2e895b4d180c869fa0d37fba86ce",
                "webSvcKey2": "820c901b085022904d0aae84b4557956"
            }
        ]
    }
}
The commands available to the service tier are as follows and can be viewed by accessing the websvc.php page directly with no request data.
  • listBlockedIps: Lists the contents of this server's blocked IP table.
  • publishBlockedIps: Publishes the current server's list of blocked IP addresses to each server in the output client table.
  • -rj: Run the json command object provided.
  • getLogs: Returns the current logs array of the web service tier.
  • getLastLog: Returns the last log entry of the web service tier.
  • getLastError: Returns the last error message that occurred on the web service tier.
  • getErrors: Returns the errors array of the web service tier.
  • clearLogs: Clears the logs array.
  • clearErrors: Clears the errors array.
  • getRpcDesc: Returns a JSON description of the top level web service calls.
The JSON commands arguments that the '-rj' web service call supports are as follows:
Check the contents of the 'tests' folder for examples on how to run web service commands in PHP and an examples of the syntax of all available commands.
Checking The BlueIce2 Configuration   top

Checking AI Training Files
Next up we'll go over the BlueIce2 configuration to make sure our tensorflow AI app is properly trained and configured. Let's check the training file listing. Run the following commands.
cd ./apps/python/BlueIce2Utils
python Main.py -j getTrainingFiles
You should see a listing of training files that have been stored in the database. The default location for training files is ./apps/python/BlueIce2/data/access_logs/ for regular apache access logs and ./apps/python/BlueIce2/data/other_vhosts_access/ for other vhosts log files. Training files must be marked up in order to be processed by BlueIce2.
Rows the have a preceeding '1 - - ' are considered valid access attempts, rows with e preceeding '0 - - ' are considered attack access attempts. To gaurantee that BlueIce2 retrains itself you must clear all the files contained in the checkpoints directory of the BlueIce2 application.
cd ./apps/python/BlueIce2/checkpoints
rm *
rm *.*
The next time BlueIce2 is ran it will pick up the new list of training files from the database, requires the proper command line arguments, or the local data directory, automatic, and process them. You'll notice that the checkpoints directory will get filled with new files when BlueIce2 runs through new training data.
To verify the python installation as well as the database connectivity and current state you can run the included python unit test files. They are located in the BlueIce2Utils project folder under 'tests' sub folder. The available unit test files are listed below, be careful they can and will alter your current database data.
  • UnitTests_ApacheLogFiles.py
  • UnitTests_Bi3InputClients.py
  • UnitTests_Bi3InstalledVersions.py
  • UnitTests_Bi3OutputClients.py
  • UnitTests_BlockedIps.py
  • UnitTests_BlockedIpsDetails.py
  • UnitTests_BlockedIpsTypes.py
  • UnitTests_Other.py
  • UnitTests_SshdLogFiles.py
  • UnitTests_SshdTrainingFiles.py
  • UnitTests_TrainingFiles.py
  • UnitTests_ValidWebFiles.py
For an example of each and every CLI JSON command, as well as examples on how to use the new command system to run a unit test, dereference variables, and perform comparisons please review the files below found in the BlueIce2Utils folder in the 'data/test_json_files' sub folder.
  • UnitTests_ApacheLogFiles_json.txt
  • UnitTests_Bi3InputClients_json.txt
  • UnitTests_Bi3InstalledVersions_json.txt
  • UnitTests_Bi3OutputClients_json.txt
  • UnitTests_BlockedIpsDetails_json.txt
  • UnitTests_BlockedIpsTypes_json.txt
  • UnitTests_BlockedIps_json.txt
  • UnitTests_Other_json.txt
  • UnitTests_SshdLogFiles_json.txt
  • UnitTests_SshdTrainingFiles_json.txt
  • UnitTests_TrainingFiles_json.txt
  • UnitTests_ValidWebFiles_json.txt
  • simple_json_command_test.txt
Adding AI Training Files   top
If your BlueIce2 installation is new or you would like to add new training files then follow the steps below. Because of the correlation between the access log signature and the intent of the access you can keep the default training files in the access_logs, other_vhosts_access, and sshd_auth_logs folders.
Place your marked up, '1-- ' or '0-- ' prefixed rows, log files into the access_logs folder if the original log file came from apache's regular access log file. Place your marked up log files in the other_vhosts_access folder if the log file came form apache's other vhosts access log file. Follow the directions above for setting the appropriate apache log format. SSHD logs from the '/var/log/auth.X' system logs can be placed in the 'sshd_auth_logs' folder. The vim command for adding a prefix to each line is listed below this can be used to setup each log file to be prepared as a training file. Please view existing training files as an example. Directory scanning will ignore folders named 'unprocessed' and any hidden files.
:%s/^/0-- /
Alternately if the marked up log files are in a location where they can be accessed by BlueIce2 you can store the training files in the database. To store training files in the database use the following commands.
cd ./apps/python/BlueIce2Utils
python Main.py -j storeTrainingFiles [path to apache training file directory #1]
python Main.py -j storeTrainingFiles [path to apache training file directory #2]
python Main.py -rj "{\"cmd\": \"storeSshdTrainingFiles\", \"targetDir\": \"[path to SSHD training file directory #1]\"}"
Remember if you use the database for training files you will need to use the '-dbTrainingFiles' argument when running BlueIce2, otherwise BlueIce2 will automatically load the files in the access_logs, other_vhosts_access, and sshd_auth_logs directories. To clear the entries in the training_files tables run the following command.
cd ./apps/python/BlueIce2Utils
python Main.py -j clearTrainingFiles
python Main.py -rj "{\"cmd\": \"clearSshdTrainingFiles\"}"
Once you have your training files setup properly either on the file system or in the database you are ready to run BlueIce2. To use the database for training files pass in the -dbTrainingFiles argument otherwise just use the local data directories. Use the -skipEvalCallBack argument to prevent BlueIce2 from evaluating apache or SSHD log files and generating an output IP list.
You will have to run BlueIce2 for each AI model you want to train and update, please follow the instruction here to clean out previous checkpoints from older model training if need be.
Training the AI   top
In order to help process the marked up apache logs we need to provide our Tensor Flow AI not only with training files but also with a list of valid web files. There are two ways to tell BlueIce2 about valid web files. The first approach passes a web directory argument into BlueIce2 so that it can scan that directory for valid web URLs. The commands for this approach are as follows.
cd ./apps/python/BlueIce2
python Main.py -dbTrainingFiles -skipEvalCallBack -webDir /var/www/html/myWebSite/
To use the database to store our list of valid web files run the following commands. And then use the -dbValidWebFiles argument when running BlueIce2.
cd ./apps/python/BlueIce2Utils
python Main.py -j storeValidWebFiles /var/www/html/myWebSite default_site
To clear the existing valid web files entries use the following command.
cd ./apps/python/BlueIce2Utils
python Main.py -j clearValidWebFiles
To run BlueIce2 and use the database as the source for valid web files use the following commands.
cd ./apps/python/BlueIce2
python Main.py -dbTrainingFiles -skipEvalCallBack -dbValidWebFiles
You'll need to make one adjustment to the small_interval_blueice3 script or run the script creation step again. Make sure you change the exeToRun argument text to 'blue_ice_log_reg_apache_nourl' to 'blue_ice_log_reg_apache' this will allow the URL to be part of the AI decision making process. The purpose of using the URL in the AI logic is that it can learn to allow valid mistakes, i.e. a valid mistake and not an attack attempt. To run the AI model for the SSHD authorization scanning process set the exeToRun to 'blue_ice_log_reg_sshd'.
Running the AI and Generating a Block List   top
Once we have our training files in place either via the MySQL BlueIce2 database or the file system and we've ran BlueIce2 to complete the training you should have seen an output similar to the one depicted below. Now we can actually use our trained AI engine to evaluate an active log file.
Found tensor dimension: 7x2
Weight: (7, 2)
Bias: (2,)
('Loss: ', [1.9658128])
Accuracy: 0.9838
Custom Evaluation: blue_ice
Test1: [Bad] False
Test2: [Good] True
Test3: [Bad] False
Test4: [Good] True
Process finished with exit code 0
That's a 98% accuracy after training on a few thousand rows of data. The rating is based on the test set taken from the training set, so we are comparing apples to apples. Strangely enough the included training file won't perform terribly on your own system. But you'll want to include your own training files, clear out all the existing check points, and re-run the training. You'll want to make sure you've included a list of valid web files and directory roots. BlueIce2 uses these when checking the apache log file's URL attribute.
Be sure to keep BlueIce2 up to date by adding new marked up training files into the database or file system and by updating the valid web files listing when the web site is updated. It is not recommended to use the file system scan approach in a scheduled task. Below is the command for running BlueIce2 and allowing the program to evaluate ./evaluate/apache_eval_log.txt and generate a list of IP addresses to block. For a database approach...
cd ./apps/python/BlueIce2
python Main.py -dbLogResult -dbTrainingFiles -dbValidWebFiles
For a database driven set of training files and a live file system scan for valid web files...
cd ./apps/python/BlueIce2
python Main.py -dbLogResult -dbTrainingFiles -webDir /var/www/html/
For file system driven approach...
cd ./apps/python/BlueIce2
python Main.py -dbLogResult -webDir /var/www/html/
You'll notice the output IP address file has been updated. You should see something like the following list of attacking IP addresses and a row count marker on the last row. The file is located here ./apps/python/BlueIce2/data/output/ip_block_list.txt.
46.17.47.89
115.231.218.25
35.154.189.116
139.162.124.167
189.233.243.115
91.200.12.95
---RowCount:446
Important Support Scripts   top
Some important support scripts that are installed along with BlueIce3 are as follows.
  • show_default_db_passwords: Shows all the unique password and web service keys set with the BlueIce3 installation.
  • test_db_blueice3: Tests the connection to the database.
  • cycle_root_mail: Cycles the /var/mail/root file if it has reached too large a size.
Clean BlueIce3   top
We'll quickly go over the available command line arguments for cleaning out any an instance of BlueIce3 and resetting it for a clean run.
[full path]/blueice3_install -i -int -apkg -j 28
To clear out the TensorFlow checkpoints folder run the following commands. Command #32 is only available in the newer software suite so you may need to run the delete commands by hand if you're using version 0.4.0.9.
[full path]/blueice3_install -i -int -apkg -j 32
or...
cd [full path]/apps/python/BlueIce2
rm ./checkpoints/*
rm ./checkpoints/*.*
PHP Unit Test Web Service Keys   top
The web service access key is defined by the config file's 'websvckey1' entry. You may need to update the secure key on the PHP unit tests to run them on your server. The default PHP unit tests have the following format, the host keys are only necessary for service calls that require a higher level of security.
{
"action": "rj",
"key": "a420a25e6d541d79315b937d41f71ec0",
"hostkey1": "2aac2e895b4d180c869fa0d37fba86ce",
"hostkey2": "820c901b085022904d0aae84b4557956",
"params": {}
}
The web service call has three main sections, the action to run, this is the higher level web service method to execute, the security key section, and the parameter section. The security key section may not be required for certain calls, check the web service call description page for more details, you can access this information by passing a '?help=on' argument to the web service page, 'websvc.php'. If a secure key is required the 'websvckey1' entry in the destination server's config file is needed to access web service methods. If a host unlock is required you must also pass in the 'websvchostkey1' and 'websvchostkey2' in that order. The 'show_default_db_passwords' script will show you all the local server's access keys.
Register BlueIce3 Input/Output Servers   top
In order to enable BlueIce3 blocked IP address sharing you need to register input and output clients. The destination output server should be configured to have the sending server in its input client's table. To register a client use the following command against the BlueIce3Utils job runner command line argument. An example JSON argument is as follows.
{
"cmd": "AddActivateBi3OutputClient",
"ipAddress": "127.0.0.1",
"webSvcKey1": "2aac2e895b4d180c869fa0d37fba86ce",
"webSvcKey2": "820c901b085022904d0aae84b4557956"
},
{
"cmd": "AddActivateBi3InputClient",
"ipAddress": "127.0.0.1",
"webSvcKey1": "2aac2e895b4d180c869fa0d37fba86ce",
"webSvcKey2": "820c901b085022904d0aae84b4557956"
}
The 'websvckey1' and 'websvckey2' parameters are meant to be populated by the target server's 'websvchostkey1' and 'websvchostkey2' configuration keys. The 'show_default_db_passwords' script will show you all the local server's access keys.
Copyright © 2018    Middlemind LLC.    Victor G. Brusca