# Using Big Data to Explore the Impact of Economic

As the world plugs into the internet, 'big data' is becoming more mainstream. With such abundance of data it is important to adequately collect, filter, manage and finance in particular, where the key asymmetrical information needed to stay ahead of competition is buried within a mountain of data.

## Financial Data

> Bid/ask price data along with the associated volume was collected live from the broker pricing feeds for each instrument. Then stored in the RAM of a kdb+ database and written to disk every 24 hours.

## Data Scraping

```
01101010
01001010
1100110
1000000

0101001001
000101010
000101100
010101001
```

Due to limitations in our setup, data files were limited to around 6 Gbps, so different methods of compression and filtering were used.
Using a data scraper, economic news data were pulled from 'forexfactory.com'. This data included date-time, importance predicted, and actual values. Using its time-stamp it was cross-referenced with the financial data.

## Data Restructuring

## Data Cleansing

Data cleansing is an important aspect of dealing with big data. As no data sources are perfect, defects are inevitable and to draw any reliable results they must be corrected. In our case, both data sources had incorrect time-zones, invalidating all initial results. To correct this, the data scraper was altered to convert date-time stamps to UTC.

Holes were also found in the time-series financial data as a result of database software patches and price feed drop-outs. These holes were filled with an equivalent data source (Dukascopy), by back-loading data into the kdb+ database.

## Data Visualization