ElasticSearch README¶
As mentioned earlier, Foursight maintains two connections - one to AWS S3 and another to an Elasticsearch instance, also running on AWS. The idea is that we use Elasticsearch to optimize certain Foursight requests such as getting the main and history pages, which are very slow when grabbing all of the data sequentially from S3. S3 serves as a backup for Elasticsearch. You can think of ES as if it is an optimized cache for S3. All GETs
will first check ES and then go to S3 if the given ID is not found.
ES Mapping¶
The current mapping can be found in mapping.json. At this time the mapping is simply a modified version of the dynamic mapping, which is automatically disabled when creating indexes using ES connection. This mapping is subject to change.
ES Migration¶
The primary method for migration checks from S3 to ES is to use migration.py. When run with no arguments, this script will migrate all checks from S3 to ES. This script makes use of a check that can be found in es_checks.py.
ES Connection¶
The ES Connection object can be found in es_connection.py with associated tests in test_es_connection.py. It implements the AbstractConnection ‘interface’ defined in abstract_connection.py. RunResult
has been refactored to utilize this connection to post all results to both S3 and ES. It will by default check ES first and fallback to S3.