Dukascopy Review: 3 Key Findings for 2020 - ForexBrokers.com

Releasing a Decade of Forex Tick Data I Crawled and Converted

Releasing a Decade of Forex Tick Data I Crawled and Converted

Intro:

In my exploration of the world of big data and I became curious about tick data. Unfortunately, market data is almost always behind a paywall or de-sampled to the point of uselessness. After discovering the Dukascopy API, I knew I wanted to make this data available for all in a more accessible format. Over the course of a few months, I downloaded, cleaned, parsed, and compressed over a decade of Forex tick data on 37 currency pairs and commodities. Today I am happy to finally release the final result of my work to the DataHoarder community!

Download Links:

Warning: I have rented a seedbox for the next 3 months from seedbox.io but I have been having some issues. If you have any issues with the torrent please leave a comment. Also, PLEASE SEED when you are done. This is quite a large data set and I can only push so much data on my own.
Torrent File: https://drive.google.com/file/d/18ymZWeFLJK7FggK_iiWZ-TxgWIVdJVvv/view?usp=sharingCompanion Blog Post: https://www.driftinginrecursion.com/post/dukascopy_opensource_data/

Stats Overview:

Totals Quantities
Total Files 463
Total Line Count 8,495,770,706
Total Data Points 33,983,082,824
Total Decompressed Size 501 GB
Total Compressed Size 61 GB

About the Data:

The data was collected from https://www.dukascopy.com/ via a public API that allows for the download of tick data on the hour level. These files come in the form of a .bi5 file. The data starts as early as 2004 all the way to 2019.
These files were decompressed, then merged into yearly CSV’s named in the following convention. “AUDCHF_tick_UTC+0_00_2011.csv” or ‘Pair_Resolution_Timezone_Year.csv’
These CSV’s are split into 3 categories “Majors”, “Crosses”, “Commodities”.
Majors, Crosses, and Commodities have had their timestamps modified so that they are in the official UTC ISO standard. This was originally done for a Postgresql database that quickly became obsolesced. Any files that have been modified are appended with a “-Parse”. These timestamps have been modified in the following format.
Millisecond timestamps to UTC +00:00 time [2017.01.01 22:37:08.014] -- [2017-01-01T22:37:08.014+00:00]
https://preview.redd.it/x6g277skfiu51.png?width=1399&format=png&auto=webp&s=35cd6735c1826424580919ac3377612377a3107c

User Resources:

For those looking to use this data in a live context or update it frequently, I have included a number of tools for both Windows and Linux that will be useful.

Windows

The ~/dukascopy/resources/windows folder contains a third party tool written in java that can download and convert Dukascopy’s .bi5 files. I have also included the latest zstd binaries from Zstandard Github page.

Linux

Linux is my daily driver in 99% of cases, so I have developed all my scraping tools using Linux only tools. In the ~/dukascopy/resources/linux folder you will find a number of shell script and pyhton3 files that I used to collect this data. There are quite a few files in this directory but I will cover the core ones below.

download-day.py:

This file is used to download a single symbol for a single day and then convert and merge all 24 .bi5 files into a single CSV.

download-year.py

This file is used to download a single symbol for a full year and then convert and merge all .bi5 files into a single CSV.

dukascopy.py

This file contains all the core logic for downloading and converting data from dukascopy.

utc-timestamp-convert.py

This tad slow but works well enough. It requires the pandas project and parses timestamps into the UTC ISO standard. This is useful for those looking to maintain the format of new files with the those in this repo, or those looking to use this in a SQL database.
submitted by jtimperio to DataHoarder [link] [comments]

How reliable are free sources for historical data?

Has anyone used free sources like histdata.com, dukascopy, forexsb to train models or backtest their algos?
I am working on training a model to trade forex pairs. Still experimenting with the idea and don’t want to invest in an data api before I have a solid strategy.
Wondering if I can make use of this data to test the model with this data and move on to test the model with something paid afterwards OR just go with a single data source as best practices would say.
Thanks in advance!
submitted by vikas-sharma to algotrading [link] [comments]

What forex pairs order do you trade? Do you stick with standard pairs from your broker or do you trade usd vs other currencies?

I am running backtest on forex pairs. I am using dukascopy data with the standard pairs (audusd, eurusd, usdcad, etc).
Does it make a difference how I order the pairs because it makes a difference for my results.
For example if I only backtesting usd pairs.
Should I trade pairs as my broker presents them? audusd, eurusd, gbpusd, nzdusd, usdcad, usdchf, usdjpy.
Should I invert some pairs so I am trading usd against the other currencies so my buy and sell signals are against the dollar instead of a mix match. usdeur, usdgbp, usdcad, etc.
Trying to wrap my head around what this means. I would think the dollar correlation s would have some effect and I would want to pair everything against the dollar but I have been getting better results when I keep them as the broker has them.
submitted by BrockSamson83 to algotrading [link] [comments]

Dukascopy forex data

I've been trying to get data from the Dukascopy forex historicals for quite some time now, and I'd like to summarize what I've done so far, and what I still need, in order to help anyone else that also wants to use it.
First, just downloading the data is a pain. The URL that you have to get it from is
 https://datafeed.dukascopy.com/datafeed/{PAIR}/{YEAR}/{MONTH}/{DAY}/{HOUR}h_ticks.bi5 {PAIR} is the currency pair, for example "AUDUSD", "EURUSD", or "USDJPY" {YEAR} is the year, for example "2010", "2014", or "2017" {MONTH} is the month, a two digit number. For some reason, months are zero-indexed. For example, "00" corresponds to January, "05" is June, "11" is December. {DAY} is the day of the month, and as far as I can tell, it is NOT zero-indexed. Again, it is two digits wide. {HOUR} is the hour of the day. For some reason, Dukascopy stores each hour of the day separately. It is zero-indexed, so "00" to "23" 
Now that you have a *.bi5 file, you have to extract it. *.bi5 files are lzma compressed files, so find a way to extract them. I used 7z command line.
Now once you've extracted it, you'll notice it's still a binary file. The data is stored in 20 byte wide rows, with each 4 byte segment corresponding to a piece of data. Example:
[ TIME ] [ ASKP ] [ BIDP ] [ ASKV ] [ BIDV ] 0000 0800 0002 2f51 0002 2f47 4096 6666 4013 3333 TIME is a 32-bit big-endian integer representing the number of milliseconds that have passed since the beginning of this hour. ASKP is a 32-bit big-endian integer representing the asking price of the pair, multiplied by 100,000. BIDP is a 32-bit big-endian integer representing the bidding price of the pair, multiplied by 100,000. ASKV is a 32-bit big-endian floating point number representing the asking volume, divided by 1,000,000. BIDV is a 32-bit big-endian floating point number representing the bidding volume, divided by 1,000,000. 
This is how far I've gotten so far before I noticed that something is wrong. The contents of the *.bi5 file do not match the contents of the file that you can download from the official front-end, here: https://www.dukascopy.com/swiss/english/marketwatch/historical/ .
For example, the January 8, 2010 *.csv file does not match in any way with the *.bi5 file of the corresponding day. Does anyone know what I am doing wrong?
EDIT: Another question is about the hours: what time zone are these files relative to? It seems that the data starts showing up from the last two hours of Sunday, going through the week, and then stopping some time before Friday ends, all relative to whatever timezone this is in.
submitted by Allurisk to algotrading [link] [comments]

Getting Started

Hey guys! I found a super cool list of everything a new forex trader would need to get started! Originally made by to nate1357. Link to original thread http://redd.it/328cjr
Free Resources
Education:
www.babypips.com/school
www.informedtrades.com/f7
www.forex4noobs.com/forex-education
www.en.tradimo.com/learn/forex-trading
www.youtube.com/useTheTradeitsimple
www.traderscalm.com
www.orderflowtrading.com/LearnOrderFlow.aspx
www.profitube.com
Calendars:
www.forexfactory.com/calendar.php
www.dailyfx.com/calendar
www.fxstreet.com/economic-calendar
www.forexlive.com/EconomicCalendar
www.myfxbook.com/forex-economic-calendar
www.investing.com/economic-calendar
Free News Websites:
www.forexlive.com - Daily live news, analysis and resources
www.financemagnates.com - FX industry news and updates
www.fxstreet.com - Daily news, analysis and resources
www.forextell.com
www.forexcup.com/news
www.bloomberg.com/markets
Forums:
www.reddit.com/forex
www.forums.babypips.com/
www.forexfactory.com/forum.php
www.elitetrader.com/et/index.php
www.forex-tsd.com/
www.fxgears.com/forum/index.php
www.trade2win.com/boards
Margin / pip / position size calculators
www.myfxbook.com/forex-calculators
Brokerages:
There are many factors to consider when choosing a brokerage. Regulations typically force US traders to only trade at US brokerages, while international traders have more choice. After considering location you need to consider how much capital you will start trading with as many have minimum deposit levels. Once you’ve narrowed that down you can compared spreads and execution. ECN brokers execute your orders straight through to their liquidity providers, while market maker brokers may pair up your trades with other clients. Market maker brokers typically will partially hedge your positions on the interbank market. Many consider this to be a conflict of interest and prefer to trade at an ECN broker who would have an active motive to see you succeed. Lastly, brokers run inherently risky business models so it is important to consider the risk of bankruptcy.
www.forexpeacearmy.com - Aggregates broker reviews. Be warned though that people only seem to make bad reviews.
www.myfxbook.com/forex-broker-spreads - Live comparison of executable spreads
United States & International-
-Interactive Brokers
International Only-
-LMAX (whitelabel DarwinEx)
*DMA broker based in the UK. Note that as a DMA broker LMAX eliminates the ability for LPs to last-look transactions. This may result in reduced liquidity during volatile times as liquidity providers would be likely not to risk posting liquidity to LMAX's pool. *Tight spreads *Minimum deposit $10,000 *Fairly well diversified
-Dukascopy
*ECN based in Switzerland, but available elsewhere depending on local regulations.
*Tight spreads *Minimum deposit $100 *Fairly well diversified
-IC Markets *ECN based in Australia *Fair spreads on standard account, tight spreads on professional accounts. *Minimum deposit $200 *Fairly well diversified
-Pepperstone
*ECN broker based in Australia. *Fair spreads on standard account, tight spreads on professional accounts. *Minimum deposit $200 *Not well diversified
Software / Apps:
Desktop/mobile
Terminology/Acronyms:
www.forexlive.com/ForexJargon - Common terms and acronyms
FAQ:
I need to exchange money, how do I do it?
This isn’t what this sub is for. Your best bet is using your bank or an online exchange service. Be prepared to pay a hefty fee.
I have money in one currency and need to exchange it into another sometime in the future, should I wait?
Don’t ask us this. We speculate intraday in FX and shouldn’t be relied on to tell you what’s best for you. Exchange the money when you need it.
I have an FX account, should I start trading demo or live?
This is highly debatable. You should definitely demo trade until you have mastered how to use the trading platform on desktop and mobile. After that it’s up to you. Many think that the psychology of trading live vs demo trading is massively different. So it may pay to learn to trade live. Just be warned that most FX traders lose almost their entire first account so start with a low affordable balance.
What’s money management?
Money management is a form of risk management and is arguably the most important aspect of your trading when it comes to long term survival. You should always enter trades with a stop loss - the distance of the stop allows you to calculate how large of a percent of your account balance will be lost if your trade stops out. You can run a monte carlo simulation to figure out the risk of having a number of trades go against you in a row to drain your account. The general rule is that you should only risk losing 1-4% of your account per trade entered.
More on this here: www.investopedia.com/articles/forex/06/fxmoneymgmt.asp[35]
www.swing-trade-stocks.com/money-management.html[36]
What about automated trading?
Retail FX traders have been known to program “Expert Advisors” (EAs) to automate trading. It’s generally advisable to stay away from that until you’re very experienced. Never buy an EA from a developer because the vast majority of them are scams.
What indicators are best?
That’s up to you to test and find out. Many in this forum dislike oscillating indicators since they fail to capture the essence of what moves price. With experience you will discover what works best for you. In my experience indicators that are most popular with professional traders are those that provide trading “levels” such as pivot points, fibonacci, moving averages, trendlines, etc.
What timeframe should I trade?
Price action can vary in different timeframes. In longer term timeframes the price action and fundamentals are much more clear. Unfortunately it would take a very long time to figure out whether or not what you’re doing is successful on longer timeframes. In shorter timeframes you can often tell very quickly if what you’re doing is profitable. Unfortunately there’s a lot more “noise” on these levels which can prove deceptive for those trying to learn. Therefore the best bet is to use a multi-timeframe analysis, working from top-down to come up with trades.
Should I trade using fundamental analysis (FA) of technical analysis (TA)?
This is a long standing argument in these forums and elsewhere. I’ll settle it here - you should have an understanding of both. Yes there are traders who blindly ignore one of the other but a truly well rounded trader should understand and implement both into the analysis. The market is driven in the longer term through FA. But TA is necessary to give traders a place to enter and exit trades from a psychological risk/reward standpoint.
I’ve heard trading Binary Options is an easy way to make money?
The general advice is to stay away from binaries. The structure of binary options is so that when you lose the broker wins. This incentive has created a very scammy industry where there are few legitimate binary options brokers. In addition in order to be profitable in binaries you have to win 55-65% of the time. That’s a much higher premium over spot FX.
Am I actually exchanging currencies?
Yes and no. Your broker handles spot FX is currency pairs. Although they make an exchange at the settlement date they treat your position in your account as a virtual currency pair. Think of it like a contract where you can only buy or sell it as a pair. In this sense you are always long one currency while short another. You are merely speculating that one currency will appreciate or depreciate vs another.
Why didn't my order fill?
Even if price appears to cross over a line on your chart it does not guarantee a fill. Different charting platforms chart different prices - some chart the bid price, some the ask price and some the midpoint price. To fill a limit order price needs to cross your limit's price plus the spread at the time that it is crossing. If it does not equal or exceed the spread then it will not fill. Be wary that in general spreads are not fixed. So what may fill at one time may not at another.
submitted by ClassicalAnt6 to TeamOceanSky [link] [comments]

Trading with Doc 10/06: Forex Pairs that Signal other Instruments DukasCopy Review By FXEmpire.com Dukascopy - YouTube Dukascopy Webinars - YouTube Exotic Currency Pairs

Founded in 1998, Dukascopy is an online forex & CFD broker headquartered in Geneva, Switzerland. They have 2 subsidiaries operating in the online trading space: Dukascopy Europe IBS AS based in Latvia and Dukascopy Japan K.K. based out of Tokyo. The two subsidiaries are regulated by the financial regulators in their respective countries, and trading conditions therefore vary depending on which ... Dukascopy forex broker has been operating on the international financial markets since 2000. forex dukascopy is a Swiss regulated Brokerage House. The company offers its customers unique access to the first Decentralized Marketplace in the world, joining the best liquidity of centralized... ForexBroker is your one stop portal for comparing the best Forex Brokers, Stock Brokers and Trading Platforms in the world! About us ... They currently have more than 64 currency pairs to invest in. Regulation Dukascopy is an international business and they have offices in the likes of Dubai, Tokyo, Kuala Lumpur, Moscow, Kiev, Riga, Shanghai, Hong Kong and Geneva. They are extremely well ... Dukascopy forex pairs. Folglich ist dukascopy bank sa gegenpartei fur alle kundentransaktionen. Further 4 types of commodities 12 types of stock market indices cfds and more than 300 global stocks can be utilized. Ihre transaktionen werden auf dem swfx swiss forex marketplace ausgefuhrt welcher ein von dukascopy bank sa betriebener ecn ist. Die dukascopy bank sa ubermittelt auftrage in ihrem ... Dukascopy's most popular Forex Pairs; Over 50 built-in tools and Indicators; 9 timeframes; Innovative Video-identification of the customer. Open Live Account online Video-identification will take up to 15 minutes with Dukascopy Connect, free and safe mobile app and mobile payment system developed by Dukascopy. only 15 minutes Powered by Dukascopy Connect. Only three simple steps: Fill in the ... Dukascopy offers a total of 653 symbols to trade across multiple markets covering forex and CFD traders with 60 forex pairs and 593 CFDs. Dukascopy also offers cryptocurrency trading with CFDs on significant crypto assets, and the actual underlying (non-CFD) across its own launched cryptocurrencies. Dukascopy Bank: Dukascopy Bank has over 148 million Swiss francs (CHF) in assets on its balance ... Managed Forex Accounts, introducing forex brokers, Currency Forex Data Feed and News Currency Forex Trading Platform provided on-line by Dukascopy.com All trading related information on the Dukascopy website is not intended to solicit residents of Belgium, Israel, Russian Federation and Canada (including Québec).

[index] [3261] [24804] [19553] [28572] [14429] [14388] [25234] [3734] [18095] [18752]

Trading with Doc 10/06: Forex Pairs that Signal other Instruments

Dukascopy Bank SA provides free training and analytics into the Forex training world DocDow is considered the most experienced trader from the Dukascopy Community. He shares his trading numbers daily and has an hour-long chat with other traders. ----- SEE MORE at the Dukascopy ... Founded in 1998, Dukascopy is a one of a kind Swiss online forex broker and the bank that offers its clients online trading services for currency pairs, Contract for Differences (CFDs) and binary ... Miren Dukascopy TV en su idioma: ... [EXOTIC FOREX PAIRS]: why you shouldn't trade them-PIPS OVER MONEY 2019 - Duration: 8:52. Pips Over Money 677 views. 8:52. Market Drivers. Oil War And Italy ... DocDow is considered the most experienced trader from the Dukascopy Community. He shares his trading numbers daily and has an hour-long chat with other traders. ----- SEE MORE at the Dukascopy ... DocDow is considered the most experienced trader from the Dukascopy Community. He shares his trading numbers daily and has an hour-long chat with other traders. ----- SEE MORE at the Dukascopy ... New review API update note for Oanda and Dukascopy forex trading brokers by Bryan Downing. 27:05. Dukascopy vs Oanda vs FXPro forex API trading by Bryan Downing. 18:28. Playing with great progress ...

http://arab-binary-option.doubnirinlie.ga