Greetings everyone!
In today’s post I’m going to show you how you can get 99% modeling quality when backtesting using MT4’s “Strategy Tester.” If you’re a MT4 backtesting veteran you no doubt know that using regular M1 historic data the highest modeling quality you can get is 90%, but what you may or may not know is that the accuracy of your backtests can be pushed even higher, BUT you need raw tick data.
How does one obtain such tick data? Well, I asked myself that very same question, and I’ve looked everywhere for a free source of historical forex tick data and after extensive searching I found it. The only problem is that the tick data is not readily usable by MT4 and as such requires a rather elaborate process of conversion and also a few other tricks to force MT4 to use the converted tick data. That’s where this tutorial or “how-to” comes in. So pay close attention as you MUST follow each step precisely.
You may be asking yourself why would one want to get 99% modeling quality. The answer has to do with scalping. When backtesting an expert advisor that does not attempt to scalp the market the default MT4 history data provides an acceptably accurate level of simulation. However when you’re trying to backtest a scalping EA even the slightest variation in the price feed can strongly affect performance.
Before I move on I’d like to mention that a large portion of this tutorial was taken from Birt’s EA Review website, so please thank him by dropping by this website, subscribe to his RSS feed, etc. The conversion scripts are his too! Massive kudos to Birt for writing them.
Ok, now let’s move on. First off we need the forex tick data, and so far the best free source I’ve found is provided by the Swiss forex broker know as Dukascopy. This broker has a pretty good reputation from what I read on various internet forums and review sites, but for the moment our only main interest is getting our hands on their high quality tick data. Unlike many other brokers or data feed providers, Dukascopy offers FREE tick data for most most currency pairs spanning from April 2007 to now. Even though the tick data is being offered at no cost, it's commercial-quality data and it approaches the ideal of 99% modeling quality. I’ve been able to get 99% modeling quality in my backtesting on a very consistent basis. The tick data is updated hourly directly from the Dukascopy data feed server. The bottom line is that this kicks ass! Now let’s hurry up and take advantage of this wonderful gift!
Necessary Tools:
Since the data is downloaded in files that span only 1 hour, some tools are necessary to download and parse it. These tools are a basically a collection of PHP scripts. Feel free to do whatever you want with the code, but don't forget to give credit where it's due – meaning to Birt.
You can download the script archive here.
You will find 3 scripts inside:
- A script for downloading the Dukascopy data, suggestively named "download_dukascopy_data.php". As a courtesy for Dukascopy who is graciously providing free data, even though the script doesn't download existing data, it still requests missing files, so, to avoid stressing their server, please set the dates in the $currencies array at the beginning of the script to the date of the last download; they're using the standard linux seconds-since-1970 format, look up mktime to figure it out.
- A script for processing the downloaded data, which assumes that it's in the same directory with the previous script and that the data was downloaded there (process_dukascopy_data.php); this one needs some parameters, run it without any for a description or check out the next script
- A small linux shell script that will process all the downloaded data
A minimum requirement to run these scripts is a command line PHP with the CURL extension. If you're using a windows box you’re going to need to download the windows version of PHP. Don’t worry though as a download link will be provided.
Windows download & convert to CSV how-to
Start by going to the windows PHP download section and fetching the latest binary version as a zip file.
Once you've done that, unpack it to c:\php\ and also unpack the scripts from dukascopy_php_scripts.zip in the same directory.
Rename c:\php\php.ini-development to c:\php\php.ini.
Edit c:\php\php.ini, search for
;extension=php_curl.dll
and remove the semicolon in front of the line and add an "ext/" in front of "php_curl.dll" so that it looks like this:
extension=ext/php_curl.dll
Save the file and exit.
Edit 21.04.2010: if you run into a zip error and your PHP installation has an ext/php_zip.dll, apply the above for extension=ext/php_zip.dll.
Click start->run and type
cmd
then click ok (or alternatively type
cmd
then hit enter in the windows 7/vista "search programs and files" box in the start menu).
Type
cd \php
in the command window.
Type
php download_dukascopy_data.php
Have a coffee. Have another coffee. Go sleep. Go to work. Go to the gym. Go to a club. Wait some more. I'm not kidding, it takes ages. You can check the progress by watching the currency pair directories get filled. You can change the array at the beginning of the file to switch the order or to remove some pairs you don't want. The number next to each pair is the timestamp at which to start downloading. If you get any strange errors, run the process again when it's finished – it will only download any files that were missed in the first step due to network errors.
Edit 23.11.2009: if you want to easily convert from a regular date to a unix timestamp, you can use Epoch Converter, a very easy to use online tool.
When the download is finished, assuming you wanted to get the EURUSD data up to 01.11.2009, you'd type
php process_dukascopy_data.php EURUSD 200702 200911 EURUSD.csv
and the output will be placed in EURUSD.csv.
Alternatively, you can type
process.bat
which will batch process all the currency data. It's mostly safe to ignore the error spam at this step.
This should be it, if everything went ok you should have your CSV files in the same c:\php folder.
Warning: make sure you have enough space on your harddisk. The downloaded tick files have over 7.5 GB while the CSV files have close to 60 GB at the time of this writing.
Using the tick data
Now, in order to use the CSV files, you need to perform some steps
- Convert the data to FXT & HST
- Copy the FXT & HST to the correct folders
- Run MT4 with a loader to allow loading your custom FXT file, then start backtesting
I'll go through each step below:
Converting the data to an MT4 format
To convert the data to an FXT file (and the accompanying HST files) usable in Metatrader 4, you need a new script that you can download here.
This script is based on GainData2fxt.mq4, an older script by an anonymous author and on FXTHeader.mqh, written by stringo@codebase.mql4.com.
This script has been updated to allow switching the GMT offset of the resulting FXT & HST files and to allow exporting the real spread in the file.
You have to perform the following steps in order to convert the CSV data:
- Copy the files from the dukascopy_mql_scripts_22.04.2010.zip archive to your MT4 installation. There's a directory structure inside, make sure they land in the proper places
- Move the tick data file (the CSV file) to experts\files
- Open a chart for the pair that you have data for (if you have EURUSD.csv you MUST open an EURUSD chart)
- Select the timeframe that you wish to generate the FXT for (if you want to backtest on H1, then select H1 as the chart timeframe)
- Double click the Dukascopy2FXT script in the navigation panel (it's in the scripts section) and configure the parameters in the window that pops up.
- Configure the parameters.
For CsvFile you don't need to write anything if the file is named just like the symbol and has a CSV extension; otherwise, just type the file name there.
Select "true" for CreateHst only if you want the script to create HST files. You do not need to perform this step more than once (leave it on "false" if you're regenerating data for a different timeframe), because if this parameter is set to true, history will be generated for the whole span of the CSV file and for all timeframes, regardless of the start/end date you configure (note that the FXT file is still limited by those).
By using the Spread parameter, you can configure the desired spread of the FXT file. If left at 0, it will use the current spread that your broker has. A special note here would be that many brokers are widening their spreads during the weekends, so pay close attention. If set to anything other than zero, it will use the number as the spread; if your broker is a 5 digit broker, you have to enter 20 for a spread of 2 pips, while if your broker is a 4 digit broker, you would just have to enter 2. Small trick to change the spread without regenerating an FXT: open the existing FXT file with a hex editor; the spread is at offset 0xFC.
The StartDate and EndDate fields control the span of the FXT file. There is a programming design bug in MT4, where the client is unable to read FXT files larger than 2GB and seeing that the tick data is rather huge, it's very likely that your resulting file might be larger than that if you don't pay attention. I added a "block" so the creation of the FXT file stops at 2GB, but keep in mind that if you don't use set limits, you will probably not get the resulting data range that you desire. A good rule of the thumb is to use ~12 month periods for EURUSD and 16 month periods for any other currency. If the currency CSV file is below 2GB, most likely your resulting FXT file will be below 2GB as well and you're probably safe to generate it for the whole range available in the CSV. The format of the StartDate and EndDate fields is YYYY.MM.DD and if left blank the script will just use the earliest, respectively the latest date available in the CSV file. - Click "Ok". Once you do that, the data generation process will start and it will typically take 5 to 30 minutes, depending on the data range and volume, possibly even more if you're using a slow machine. When the processing is finished, you will get an alert and some data will be printed in the Experts log.
- Some people experienced miscellaneous issues. If your particular case is not listed here, feel free to drop a comment, I might be able to help.
Among the common problems that the people encounter it's worth mentioning the Windows 7 / Windows Vista users that run MT4 from their Program Files folder and that are unable to find the resulting files. The issue is caused by the fact that they have UAC enabled and these operating systems use folder virtualization. For Windows 7, the resulting files can be found in c:\ProgramData\ typically, while for Windows Vista they can be found in c:\Users\username\AppData\Local\VirtualStore\Program Files\. If they're not there, just search for *.FXT, they must be somewhere. To get around this problem, either copy your MT4 folder in a location that's not protected by UAC or simply disable UAC (type UAC in the start->search box thing and follow the on-screen process).
Another issue experienced by some users is the lack of a MIN_LOT and LOT_STEP for the symbol in the resulted FXT file. This is caused by generating the FXT file with a MT4 client that has never been connected to a broker. When generating a file, do that on an MT4 client that has been connected to your broker at least once. Aka don't use a "virgin" installation that has never seen a demo account.
Copying the data to the correct location
After the script above has finished running, you will end up with a lot of files in your "experts\files" folder. At this point, exit the MT4 terminal then proceed to moving any .HST files from "experts\files" to "history\your_server_name". Pay very close attention if you have multiple server directories in your "history" folder – you will have to move them into the one that's correct for active account!
Note: unless you modify the CSV or get a new one, the HST files only need to be generated and moved once.
After moving the HST files, move the generated FXT file from "experts\files" to "tester\history".
Using the data in a backtest
Since the recent MT4 clients no longer offer the ability to use custom FXT files, a special loader is needed. You can download it here, it works only with the MT4 version commonly known as 225 (version 4.00 build 225).
To use the files you created in a backtest, you will have to copy the exe inside fxt_loader.zip into the folder of your MT4 client and instead of running terminal.exe, run "fxt loader.exe". If you're worried about running executable files and you can do a manual patch, here are the offsets and data:
hex offset -> old byte -> new byte
1402FC -> EE -> 00
1402FD -> 02 -> 00
14041B -> 9D -> 00
14041C -> 01 -> 00
140521 -> 97 -> 00
What the loader does is disabling the MT4 default behavior to regenerate the backtest files every time you start a new test – so, if a file is already there, a new one will not be generated. By placing the FXT file you generated with the script in the tester\history folder, you ensure that it's the file it is going to use for your backtest; of course, the selected backtest currency pair and timeframe must coincide with what you generated the FXT with.
As an alternative to the loader above, you can use this loader which, in addition to what the other one does, will also let you run multiple instances of the terminal from the same installation folder. Read the comments below for more info and the patched offsets.
Also there is a new build 226 loader. It also includes an MQL script for the people who don't want to run EXE files or who have particularly dumb antivirus packages; if you want to use the script version, start MT4 then put the script on any chart to enable the FXT functionality. However, keep in mind that while you can start more than an instance of the same MT4 client with the loader, you can't do so with the script. If you want the patch offsets, they can be found in the script. Note that running the script on a build other than 226 (225 for example) will surely make it crash, so double check what build you're running before you use it. On a separate note, you can make a similar script for build 225 by using the offsets posted above.
How to verify it worked?
Easy! If your test is running and the file in tester\history was not recreated, it means everything is OK and you’re in business!
GMT offset
This is a sensitive issue for many EAs. The Dukascopy data seems to be GMT-based all around the year, so the GMT offset that you need to use is 0. As of 21.04.2010, the Dukascopy and Oanda MQL scripts have an external parameter that lets you specify the GMT offset you would like for the resulting data, both HST and FXT.
Oanda tick data
If you're one of the lucky people who can get the Oanda tick data (it's only available to people who have an account there funded with over 1k USD), get it in CSV format and use the Oanda2FXT MQL script. It works just like the Dukascopy2FXT script described above.
Variable spread backtesting
If you want your backtest to be even more accurate, as of 21.04.2010 you also have the option to use the real spread from the tick data. All you have to do is:
1. Make sure you're using the latest Dukascopy MQL script or the latest Oanda MQL script.
2. When running the script, set UseRealSpread to true.
3. When loading the terminal, use the variable spread loader (which is different from the other loaders).
Note that as of now, this only works on Metatrader 4 build 225. I don't think I'll be making one for 226 anytime soon, so if there's anyone who fancies asm and thinks they can handle porting it to 226, here's what I changed:
00401323 DA0D 36134000 FIMUL DWORD PTR DS:[401336]
00401329 DB17 FIST DWORD PTR DS:[EDI]
0040132B E9 26020000 JMP terminal.00401556
00401336 A0860100 //100000
0040133A ACC52737 //0.00001
00401556 DA27 FISUB DWORD PTR DS:[EDI]
00401558 D815 3A134000 FCOM DWORD PTR DS:[40133A]
0040155E EB 12 JMP SHORT terminal.00401572
00401572 DFE0 FSTSW AX
00401574 F6C4 01 TEST AH,1
00401577 E9 E5020000 JMP terminal.00401861
00401861 DC42 1C FADD QWORD PTR DS:[EDX+1C]
00401864 74 06 JE SHORT terminal.0040186C
00401866 DC83 20030000 FADD QWORD PTR DS:[EBX+320]
0040186C C3 RETN
00541209 E8 1501ECFF CALL terminal.00401323
0054120E 90 NOP
Downloads:
For easy reference here are all the files referred to in this post.
Dukascopy data download / parse scripts
Dukascopy MQL process scripts
MT4 build 225 FXT loader
MT4 build 225 FXT multiloader
MT4 build 225 FXT multiloader with variable spread
MT4 build 226 FXT multiloader and script
Oanda MQL Process script
I hope you found this tutorial useful! Thanks goes out once again to Birt for writing the PHP scripts and the how-to guide. If you wish to chat with other forex traders about MT4 backtesting in general or specifically about this method of achieving 99% modeling quality I strongly recommend you join the Forex Nirvana forum. The specific forum section dedicated MT4 backtesting is located here:
http://www.forexnirvana.com/f41/
Happy backtesting everyone!
Cheers,
Alan
Share and Enjoy:
No comments:
Post a Comment