python gui programming

Sorry for kicking the hornets nest, but I really am enjoying this enthusiasm. From what I have read, there are also ways to utilize excel with python as well (using Py instead of VBA I think?). Not to force the issue with Python (although I think it's a very powerful language), but maybe this discussion will lead to using the best tools to accomplish what everyone is interested in. I use Slack and am involved in a project developing an algorithmic trading system using a tool called Zorro (programming with C Lite). In that project, we are able to share code, articles, etc. using Slack without any problems. I run Slack both on my Mac & Windows desktops as well as on my iPhone where I can get notified of discussions & messages. As far as code sharing on Slack, different channels can be set up so that one channel becomes a repository for code ideas (or maybe one for excel & one for Python) and the other channel is a discussion channel. But the code can be shared directly in the channel and it's all searchable as well. All of this however, can be done here in the Aeromir forums as well, but is a bit difficult to track on the iPhone.

All that to say that I am open to whatever you all decide as I am really interested to learn and improve my efficiencies in managing my trades ... hopefully that will also translate into more and consistent profits with better decision making. One thing I have learned is that putting processes into an algorithmic framework helps to solidify the process and also identify shortcomings that can be tweaked in the process. I use the term algorithm in this sense not as an app or automation/robot, but more of a definition of a trading process and decision making framework from start to finish. I also trade FX which requires a very rigid process, especially in terms of risk & money management. Options however, I have found at times to be a bit more nuanced, especially when trying to get filled on orders, especially 4 legged condors that need to be split in two separate orders and trying to identify appropriate stops, conditional orders, etc. is definitely a moving target with options because there are so many factors you can key off of such as underlying price, the greeks, T+ projections, etc. This is why I have expressed my desire to have a dashboard type decision frame work that I can use for each trade based on my set of rules (or at least rules that others have defined as best). I envision being able to enter a ticker and the associated strikes and dates that bring in the data to a model for a specific trade I want to put on or have put on, and see at a glance all the relevent metrics such as breakeven, and even an associated price if I wanted to close or open the trade with 4 legs or 2 legs, additional strategies that can be used, with subtotals and grand totals for each component and the entire trade. At this point, I am not really needing to analyze a trade such what the greeks look like or stepping out for future projections, although that could be something to consider down the line. I am not sure how difficult that would be.

Again, sorry for the brain dump.

So any suggestions about how/where to start? Probably need to define what a first end state might look like and work towards that goal. I am happy to bug test, provide ideas, help code, run errands ... whatever is needed. ;-)

Thanks,

Steve
 
By the way, I read an interesting book called "Algorithmic Trading with Interactive Brokers ... Python & C++". It had some interesting ideas about setting up code for options.
 
Yes I know about those post from the CD days I perticularly like the Kevin Lee videos that he started but than he got busy with other stuff so that was pretty much abandoned
I know about RTD as I use it to track my trades but I never got beyond that into serious programing but if someone wants to start something with excel programing I am all ears
 
Marcas,

is using Python via an API for accessing the data faster than setting up a DDE connection with Excel? If you didn't use Excel as the dashboard, what would / could be used instead? Also, to your point, right now the lines don't need to look pretty in my opinion; and we aren't trying to build something like ONE, but more utilities to help consolidate everything we need as traders into a single location for decision making. I have a Mac, a Windows 10 PC and a headless Mac Mini, along with two screens and single keyboard and mouse. I use TOS and soon will also use IB TWS for options as well. I run ONE on the PC along with TOS for the data feed, but I also run TOS on my MacBook which I prefer because of the OS and Interface of OSX. With ONE running on one screen and TOS on the other, I can visualize what I am doing while making trades, but TOS is similar to ONE so its a bit redundant.
 
Sorry for kicking the hornets nest, but I really am enjoying this enthusiasm. From what I have read, there are also ways to utilize excel with python as well (using Py instead of VBA I think?). Not to force the issue with Python (although I think it's a very powerful language), but maybe this discussion will lead to using the best tools to accomplish what everyone is interested in. I use Slack and am involved in a project developing an algorithmic trading system using a tool called Zorro (programming with C Lite). In that project, we are able to share code, articles, etc. using Slack without any problems. I run Slack both on my Mac & Windows desktops as well as on my iPhone where I can get notified of discussions & messages. As far as code sharing on Slack, different channels can be set up so that one channel becomes a repository for code ideas (or maybe one for excel & one for Python) and the other channel is a discussion channel. But the code can be shared directly in the channel and it's all searchable as well. All of this however, can be done here in the Aeromir forums as well, but is a bit difficult to track on the iPhone.

All that to say that I am open to whatever you all decide as I am really interested to learn and improve my efficiencies in managing my trades ... hopefully that will also translate into more and consistent profits with better decision making. One thing I have learned is that putting processes into an algorithmic framework helps to solidify the process and also identify shortcomings that can be tweaked in the process. I use the term algorithm in this sense not as an app or automation/robot, but more of a definition of a trading process and decision making framework from start to finish. I also trade FX which requires a very rigid process, especially in terms of risk & money management. Options however, I have found at times to be a bit more nuanced, especially when trying to get filled on orders, especially 4 legged condors that need to be split in two separate orders and trying to identify appropriate stops, conditional orders, etc. is definitely a moving target with options because there are so many factors you can key off of such as underlying price, the greeks, T+ projections, etc. This is why I have expressed my desire to have a dashboard type decision frame work that I can use for each trade based on my set of rules (or at least rules that others have defined as best). I envision being able to enter a ticker and the associated strikes and dates that bring in the data to a model for a specific trade I want to put on or have put on, and see at a glance all the relevent metrics such as breakeven, and even an associated price if I wanted to close or open the trade with 4 legs or 2 legs, additional strategies that can be used, with subtotals and grand totals for each component and the entire trade. At this point, I am not really needing to analyze a trade such what the greeks look like or stepping out for future projections, although that could be something to consider down the line. I am not sure how difficult that would be.

Again, sorry for the brain dump.

So any suggestions about how/where to start? Probably need to define what a first end state might look like and work towards that goal. I am happy to bug test, provide ideas, help code, run errands ... whatever is needed. ;-)

Thanks,

Steve
Steve:
I appreciate your "brain dump"! We have similar thoughts on process.
IMHO: The most difficult task (for me) has been determining precisely what I want! Implementing is the easy part.

Regarding Py in Excel instead of VB.... Not a recommendation, but observe this product: "https://www.pyxll.com/", but it seems to pricey for me.

However, you should be aware that there are other packages for Python, that allow you to accomplish similar thing external to Excel. I use Perl do do this, but there are Python packages that likely eclipse what I can do in perl. (My expertise is in Perl, and only limited python so far). For me, trying to support and continue to develop in VB is a non-starter, so I develop in Perl or some Python when code development is warranted.

I like using Slack, as you can create channels for specific things. The Free version of Slack seems plenty fine for me.
Marcas has a few slack channels, there are some Aeromir and older CD channels that can be used, or can create some.
I have a few slack channels (very low usage) that I can send out invitations to join, but none currently for code development.
I do have one for others to try/test something that I wrote for directional trading with options, but that is a bit off topic.
 
I appreciate all thoughts. Like Gary, I can't say they are strange to me :) Also agree with G. that deciding what is needed is the hardest part but only after you deep into coding waters. Before opposite seems to be true.

My experience so far with cooperation: it is hard to do. There is enthusiasm out there but goals are different for each trader. Algos, currencies, daytrading, Boxcar trade etc. I don't blame anybody, I myself don't have enough time to work on projects of others where I see no practical application for my trading style. That's why I'd rather focus on more universal applications, like one I showed in last TG1. In the future I can show some others if there is such desire. Or separate Zoom meeting can be organized as not everybody is interested in this stuff. Designing smth like ONE is not that hard, problem is (for me) data.

Re Forum / Slack. I found Slack more practical, but I also like to talk about other things beside trading. Not everybody likes that. If there is a will, you can start wherever you want. I'm planing to utilize old CapitalDiscussion Slack because I like it for few reasons, but by all means use what is comforable to you. Slack has this feller that you cant block off unwanted topics. Just to make it clear, trading is more important to me than coding and almost all coding on my side has some trading philosophy behind it (right or wrong).

As a start, if you, SG, provide me some data (like options chains or something) and tell me what your idea is, I can show you what can be done in py regarding dashboard in just few lines of code. Then, maybe, Jim will improve things in Java (I think Java have better graphic libraries), and then all can be implemented into Excel. Just a thought.

is using Python via an API for accessing the data faster than setting up a DDE connection with Excel?

I do not have much experience with Excel but I dare to say that API method is superior to RTD. By far.

marcas, i'm in favor of sharing code and ideas. i find slack a little more inconvenient than this forum but i can
deal with it. i can't share my programs, for various reasons

Sure, that's another thing, sharing code is not the goal. But sharing ideas and helping each other solving coding and trading roadblocks is, imo.
 
Last edited:
What are CD Channels? Is everything set up so we can continue the discussion in Slack?

Maybe we should set some goals.

It sounds like using an API is the best route, so that would be the first thing to establish with a data source. I have access to IB/TWS & TOS data. What does everyone else have? What would be a good test to get this part going so that once we have established our data connection, we can take it to the next step. I am thinking a test that everyone can try out if they want to.

For next steps, I would think setting up some kind of dashboard where the data would be displayed. I don't know if it's excel or something else. Then we would have the the backend (data) connecting to the beginnings of a front end (dashboard). As for the dashboard, would it have tabs for each individual underlying? Would it have one main connection point as a repository for the data which can then be linked to throughout the rest of the dashboard?

Once the front and back are connected, then we can decide what we want to use the dashboard for. Do we have static strategies setup like the Boxcar, etc. where we pretty much know the rules & guidelines for the trade or do we try and provide the capability to have functions such as a field for the underlying, DTE's and Strikes, which will in turn populate the data in the fields we want to display in our decision support dashboard?

I am thinking out loud here and probably being very rudimentary, and much of this has probably been done already. I am thinking if we take each step one by one so everyone can weigh in on what works best (or has worked best based on past experience), then maybe we can come up with a consensus of how to put together something that gives all of us flexibility and something useful.

I kind of view the dashboard as a sort of toolbox for day to day management of our option investment processes.

Am I off base? If someone else has suggestions of a better way, I have no pride of authorship of any of this.
 
Speaking for myself personally I would prefer to use TOS and RTD with excel I think there are a lot more folks familiar with excel than with API
Sure API may be the best route but not if you have to learn that first so I think half the audience would be gone leaving only the few who know how to use API than if you use Python or any other programing language more people would drop off leaving just a few experts talking among themselves
Just my thoughts
 
That makes sense. DDE & API are just the method to connect to get the data into the dashboard and the API way may be more complicated. However, if we initially get the data into the dashboard using DDE, there is nothing to prevent those who want to use API to do so, right? It's just a data connection.

I guess the question I have is, will the way the data comes into the dashboard affect how the dashboard is put together or how we use the data? Is this an important decision to have consensus on before we start to decide what it is we want to be looking at for our decisions?

As for the dashboard, here is an example of what I just now experienced when trying to close out a PDS for SPX today. In an excel sheet, I entered the trade price I initially paid for the PDS with the prices of both legs. Then using TOS, I entered an order to close the position. In order to figure out what limit price I wanted to use to place the trade, I moved the price up a bit, then hit the confirm and send button to see what the value is for the closing trade. I then went back and forth doing the same thing until I arrived at a limit price that gave me the dollar amount I was looking for to close the trade. It seems to me that based on real time updated pricing in a dashboard, I should be able to get a pretty good idea what limit price I should need to enter for any position I have at a glance. ONE gives me a total cost for the trade from inception of the trade and I can see what my breakeven would be, but it doesn't tell me what the cost would be for adjusting part of the trade. With these boxcar trades for example, I am frequently just closing the PDS and letting the PCS expire so that I close the PDS for a profit (hopefully) and get to pocket the full credit I received from the PCS to begin with. Maybe there is a better way, but I find myself having a blank excel sheet to use as a calculator and then go back and forth to figure out something that should be a fairly quick decision.
 
Some details that may allow one to get better feel for some decision making on which direction (coding/DataSource, etc) they may consider next.
Data source:
Some details for choosing RTD via TOS/Excel.
1) RTD is limited by quantiy of RTD links.
The limitation is from your PC (memory and speed), your Excel (excel overhead and possible version limitation), and your TOS (primarily iits changing Memory requirements). If you are NOT pulling all chains, or you only reference your positions, then this is NOT an issue. If your number of RTD links (# opras X #fields each) is < 10K or so, this should not be an issue. (I seem to recall I noticed issues with #links above 40,000. -- I do about 30K each minute and have no issues for a number of years-- This is part of my current process for supporting the GTool RTT reports.
From this morning's automated setup:
"Extraction PHASE: RTD_Responses = "27177", RTDrows=391, waiting for 27177 responses, Retrys=3." <-- my debug message for setting up the daily RTD mechanism for SPX options. -- This is limited to options < 1 year DTE.
2) RTD resulting values are not guaranteed to be atomic with respect to other values if you use the default automatic 2 second refresh. You can address this by setting the refresh to no refresh, then have your VBA (or your external Perl or Python program) force a refresh, then wait about 2 seconds, then process the refreshed data. See "https://news.cqg.com/blogs/2011/04/adjusting-rtd-interval-throttle-microsoft-excel" for manually altering this. Note: this can also be accomplished from Perl/Python (altering the interval, and forcing the refresh).

The RTD approach cannot be fully automated (un-attended), but can get close enough. I have been doing it for a few years, but need to bounce TOS at least once per week as TOS is the unreliable link, especially when they do updates. So, if you close out your TOS, and fire up a fresh on on Sunday afternoons for the coming week, it has become reliable.

Some details for choosing the TDA-API for your real-time data.
1) Simplicity: Use the get chains, which will pull all chains for the underlying in one operation, which provides atomic snapshot and no discovery of which options exists is necessary. -- The API call for SPX consumes about 2 seconds on my PC, which is acceptable for me.
2) Chose what information you wish to use/keep and discard the rest.
3) You may put this data anywhere you chose. If you want it in Excel, you can do that. (I do not put this data in Excel, but into a MySQL database).
4) Language: Python works great and is what I and Marcas use.
5) Is not dependent on any manual efforts or TOS.
6) I currently use this to pull all chains for 14 underlyings each minute to my SQL database. This is not possible with RTD as the qty of links and massive amount of data would overwhelm both TOS and Excel (and maybe your PC memory).

To re-iterate: I currently do both. (RTD for GTool reports and Aeromir market info updates), and API for populating my DB and for other tool development.
 
Maybe we should set some goals.
(...)
For next steps(...)

SG, I like what you are saying
but
keep in mind that others may have different goals and they wont be interested in investing much time in your project. I rather suggest to focus on building small chunks of code everybody can use. For example, to pull data from somewhere (Yaho! maybe..), to convert data to useful format, to access data in effective way, to develop method to quickly find optimum trade setup, to visualize expiration graph of a given trade, etc. When those blocks are ready each can punt them together the way they want and modify the way they want for individual use. I think learning to code by doing is a decent way to learn (for our purposes) and one can learn a a lot about trading this way as well. I don't see creating any big piece of universal software feasible at this moment.
I'm definitely for small steps but if I can help with bigger project, I will. Your idea of 'overall dashboard' is doable but I bet that when you are half the way, your goals will change :). If you focus on building small blocks you can quickly rearrange them instead of starting from the scratch (which is not that a bad idea after all).

CD channels I mentioned are on old Capita Discussion Slack. It is mostly abandoned and not used anymore, it can be utilized until better solution is found. I like slack because of easiness of use, posting pics and chunks of code included. Tom has administrative privileges for CD Slack and can create new channels (if needed at all). I suppose for initial cooperation it is fine, but not necessary, one can create yet another Slack workgroup or use any other means. Whatever works.

Re source of data. Beside API and Excel there are other methods as well. One of the simplest one (for TOS users) is to simply go to Trade tab and experiment with exporting chains to csv file on HDD. Al least this way you can quickly put hands on some data to work with. But even that is not necessary - useful utilities can be constructed without need for any data (like TOST). If there is such a need I can show example of simple code utilizing TDA-API and crunching a bit what TDA offers. I don't know if I will be available for next TG1 nor what the agenda is ,but probably I can present it there or on separate meeting. Or maybe it is to early for that.

I suppose there is majority of traders willing to advance with Excel. I recommend to just start working. Find methodology to pull single options chain to spreadsheet or to monitor your open position in Excel (if that is what you want to do).
 
i started learning python last year and as a script language it's pretty powerful, but i'm
used to gui programming in java. i looked at tkinter and it is ok but very tedious. are there other
python programmers in this forum? what gui packages are you using? i need a gui editor like
what's available in java and android.

PyQt Designer is a standalone GUI layout tool that is well-supported. It includes a reliable code generator that generates pretty efficient code. You simply layout your application window, graphically and then generate code. You insert your program functions in the "put your code here" stubs for the buttons, menus, and other active interface objects, and hook up your events. PyQt Designer has a complete API wrapper library around the Qt C++ classes and objects supports all the Qt GUIs classes.

Rich, there is plenty of info you provided. Wonder if you use QT in your coding practice?

Yes, I use Python and PyQt when I want stand-alone application interfaces. When I want to learn and prototype things, I use Jupyter Notebook until it is ready to package. Jupyter is great for many things, especially documentation, collaborating, and sharing computational documents. But since it is browser-based, it is a bit too loose for packaged production applications. If you want a tight GUI application like ONE, you need to use a real GUI toolkit (or be an expert in Javascript, CSS, HTML) or advanced Python web framework toolkits with GUIs (like Django or Flask).

I, of course, checked out QT but saying truth it was bit to much for my needs. At some point I had to answer myself question, if I want to be a coder or trader... amount of time needed to only get familiar with many aspects of coding was humongous...
As I remember I also didn't like licensing of QT but it could change since it was a while ago I was looking.

I skimmed through some material you linked and must admit that it is much better quality then when I was researching.
I will look up what QT can do and re-evaluate.
Thanks a lot for info.

One q:
Can you tell if I can use interactive Plotly graphs (html) in QT windows without much gymnastics?

Plotly can work in an IDE. I am not sure if Dash works outside of a browser. Plotly and Dash are designed to provide their own Reactive interface like RShiny. Plotly works inside Jupyter. I suppose you could get it to work using a web server behind a PyQt window.

From the Plotly website:

"plotly enables Python users to create beautiful interactive web-based visualizations that can be displayed in Jupyter notebooks, saved to standalone HTML files, or served as part of pure Python-built web applications using Dash."

With PyQt, you code in a Python API wrapper around the Qt API. You don't have to touch C++. NO license fee is required for PyQt, since PyQt is an API layer (wrapper) over an embedded FOSS version of Qt. PyQt is free, and maintained by a significant community of coders.

Are you a coder, or trader, or a trader who needs to code?

People (traders) often say "I am a trader, not a programmer" or "I am a trader, and I don't want to be a programmer."
Or "I am not good at programming, and I don't have time to learn, because it will take too much time away from my trading." (that was me.)

However, when the trading support tools that you need to be a better trader are NOT AVAILABLE, you have several alternatives:
1) pay someone else, according to YOUR specifications, and wait for them to develop it for you.​
2) wait for someone else to develop it, according to THEIR specifications (and they may or may not share it with you for free),​
3) pay for a commercial product that does mostly what you want (but probably not everything (suboptimal fit).​
4) develop it yourself
5) business as usual == use the same old (crappy) tools you've been using (Excel anyone?)

My opinion on Excel:

I have used Excel since the early 1990s. It does some things very well. I can do well enough with formulas in cells, but I REALLY don't like programming in Excel VBA. I might be considered an Excel Power User, but not an Excel expert developer.

The problem with the Microsoft Office approach is that the VBA object models and APIs are all sufficiently different from one another (Excel VBA != Word VBA != Access VBA != PowerPoint VBA != OutLook VBA, and OneNote has no VBA) to make it a nightmare to develop sophisticated tools like OptionNetExplorer. You might ask "Who would want to (or could) develop ONE in Excel?" (Perhaps a masochist, or a non-trader who's dedicated his/her life to Microsoft Office programming.) If you make it an "all in one" collection of utilities, eventually you have a BRITTLE maintenance nightmare. The brittleness of Excel really shows itself in production applications. Simple utilities in Excel are fine, but to link these little utilities together into a full blown application, then you are stuck importing and exporting data. When you try to use Excel as a database, you quickly scale beyond what Excel was designed for. In the 2010s+ you can use PowerPivot, PowerQuery, PowerBI, but that means more complexity and (slower) layers on top of SQL Server and Excel. It becomes a house of cards.

Excel is its own visual programming language and object model with many quirks and limitations that take years to become painfully visible. Having spent 2 decades using Excel and never really wanting to invest the time to become a VBA expert, I could rant for hours about MS-Office development shortcomings.

I still use Excel because I am lazy and stuck in old habits for certain tasks I've done in Excel for 20 years, like formatting and browsing datasets in Excel before I decide what to do with it. Excel is really great for re-using other people's one-off worksheets for finance work of all kinds.

Into Python

I finally bit the bullet and learned Python after years of procrastinating about mastering a programming language. I had been exposed to it around 2005 when working on website development, but found it (and PHP) too "script-like" and unstructured at that time (they were not yet fully object-oriented). I committed to mastering it about 4.5 years ago by taking a training job teaching others in data science and analytics.

First I carefully assessed whether Python could be "the one". R and Perl were not it, because they are not general-purpose enough. I wanted to invest in one (and only one) coding language to master before I retire in about 10 years. I needed a language that I could use for everything (Excel replacements, analytics dashboards, databases, GUIs, websites, security, big data, little data, analytics, visualization, artwork, statistics, simulation tools, games, reporting, etc.).

I found Python is straightforward to learn, with a clean and consistent object model. There is a delightful (and mind-boggling) collection of packages available for any type of application imaginable. And there are many 10,000s more that you probably have never imagined. It keeps improving by the month with more technical communities adopting it. It has unstoppable snowball momentum. 253,938 projects as of today (see https://pypi.org/ ). In 2018 there were 68 more packages being added every day. The Python language by itself doesn't do everything well, but with core Python libraries (PANDAS, matplotlib, statmodels, SciKit-Learn, Jupyter Notebook) it does analytics and most data operations well.

Alas, Python may eventually be replaced by Julia, which combines the best features of Python, R, C, Haskell, and MatLab. Julia is a pure functional vector-based, high-performance language with full object-oriented support. However, Julia is still far from mainstream and requires more computer science training than Python to really use its features well.

Incidentally the Jupyter Notebook is an acronym of Julia, Python, and R, spelled (JU-lia) + (PYT-hon) + (R) ==> JuPYTeR

I apologize if I didn't answer all your questions and rambled too much. I know I have strong opinions. I don't want others to go through too much pain, like I have been through with Excel and other programming languages.
 
Last edited:
Some details that may allow one to get better feel for some decision making on which direction (coding/DataSource, etc) they may consider next.
Data source:
Some details for choosing RTD via TOS/Excel.
1) RTD is limited by quantiy of RTD links.
The limitation is from your PC (memory and speed), your Excel (excel overhead and possible version limitation), and your TOS (primarily iits changing Memory requirements). If you are NOT pulling all chains, or you only reference your positions, then this is NOT an issue. If your number of RTD links (# opras X #fields each) is < 10K or so, this should not be an issue. (I seem to recall I noticed issues with #links above 40,000. -- I do about 30K each minute and have no issues for a number of years-- This is part of my current process for supporting the GTool RTT reports.
From this morning's automated setup:
"Extraction PHASE: RTD_Responses = "27177", RTDrows=391, waiting for 27177 responses, Retrys=3." <-- my debug message for setting up the daily RTD mechanism for SPX options. -- This is limited to options < 1 year DTE.
2) RTD resulting values are not guaranteed to be atomic with respect to other values if you use the default automatic 2 second refresh. You can address this by setting the refresh to no refresh, then have your VBA (or your external Perl or Python program) force a refresh, then wait about 2 seconds, then process the refreshed data. See "https://news.cqg.com/blogs/2011/04/adjusting-rtd-interval-throttle-microsoft-excel" for manually altering this. Note: this can also be accomplished from Perl/Python (altering the interval, and forcing the refresh).

The RTD approach cannot be fully automated (un-attended), but can get close enough. I have been doing it for a few years, but need to bounce TOS at least once per week as TOS is the unreliable link, especially when they do updates. So, if you close out your TOS, and fire up a fresh on on Sunday afternoons for the coming week, it has become reliable.

Some details for choosing the TDA-API for your real-time data.
1) Simplicity: Use the get chains, which will pull all chains for the underlying in one operation, which provides atomic snapshot and no discovery of which options exists is necessary. -- The API call for SPX consumes about 2 seconds on my PC, which is acceptable for me.
2) Chose what information you wish to use/keep and discard the rest.
3) You may put this data anywhere you chose. If you want it in Excel, you can do that. (I do not put this data in Excel, but into a MySQL database).
4) Language: Python works great and is what I and Marcas use.
5) Is not dependent on any manual efforts or TOS.
6) I currently use this to pull all chains for 14 underlyings each minute to my SQL database. This is not possible with RTD as the qty of links and massive amount of data would overwhelm both TOS and Excel (and maybe your PC memory).

To re-iterate: I currently do both. (RTD for GTool reports and Aeromir market info updates), and API for populating my DB and for other tool development.

Gary, do you run your program continuously to pull all option chain data into your MySQL database? Or do you run it only long enough each day to pull the option chain data needed to do your (daily or on-demand) analysis for the GTool reports?

Is it a "MySQL" database, or is your SQL something else like PostgreSQL or SQL Server?

I thought you used Perl or PHP, not Python for the GTool reporting?
 
Gary, do you run your program continuously to pull all option chain data into your MySQL database? Or do you run it only long enough each day to pull the option chain data needed to do your (daily or on-demand) analysis for the GTool reports?

Is it a "MySQL" database, or is your SQL something else like PostgreSQL or SQL Server?

I thought you used Perl or PHP, not Python for the GTool reporting?

A small background first:
I experimented and used a number of data sources. LiveVol subscription, RTD (TOS-Excel), and TDA-API. of these RTD and TDA-API are real-time. The RTD has "wart-like" variation in that it can be used with OnDemand for some historic data collectiion.
Rather than have each tool/application have do deal with all these permutations, I decided to isolate the data collection from the data consumption to simplify my life and allow data consumption tasks be developed independently from the collection tasks by using an SQL Database.
I chose MariaDB which was recommended by a college. It is adequate for what I am now doing and serves that purpose well. It is free and supported via both Perl and Python. I use both (Python for the API, and Perl for the Livevol and RTD ) for data collection to the DB.
So, I have different code which takes each data source and adds them to the SQL Databases.
For the Real-Time, I collect the data at the top of every minuite (within a few seconds) and add to the db with timestamps. I collect the data beginning at 6:31AM Pacific thru 13:15 Pacific. I do have one database that is used when I need after hours data that is only used as rqd to handle the non-systematic cases.

To answer your question, for my TDA-API mechanism, I pull all chains (not all data). I collect BID, ASK, and Open Interest. I do my own option calcs which saves database space. (note the Open Interest is one sample per day, so space not concern for that item)

The GTool accesses data by referencing the most recent timestamps, which is always within 60-seconds of real time (when the market is open). The GTool currently references the SQL data which was populated by another process collecting data via RTD. That process collects all SPX chains with less than 365 days DTE.
The GTool is primarily Perl.

Side note regarding Database size: mySQL (aka MariaDB) is NOT thrifty with db space! So, one must carefully consider db content. Tradeoffs on DB size and in-line computations may need to be considered. - Prior to switching to the DB, I was using ziped CSV files to conserve space and speed file read. This worked very well. Switching to the DB supported laughable compression (something like 30% if my memory is correct), while my csv zips were 80 to 90% compressed.
 
With PyQt, you code in a Python API wrapper around the Qt API. You don't have to touch C++. NO license fee is required for PyQt, since PyQt is an API layer (wrapper) over an embedded FOSS version of Qt. PyQt is free, and maintained by a significant community of coders.

according to their web site pyqt is free but licensed under gpl. if you're comfortable with gpl, that's fine. i'm not
for a lot of what i do. of course, you can ignore the license for your own use but user beware.
 
Rich, first of all you are not rumbling. You touched many topics but this is fine and understandable. All you said makes sense. I agree with what you said in general; in details, I suspect, there can be some differences.

What you said about PyQT makes me wanting to try it out… eventually. For what I need now, Plotly is a perfect match, or to be on save side, seems to be so. Inside Jupyter combination of widgets and Plotly provides me interactivity I want. With Dash I can go to browser but I can do that even with Plotly alone (to some degree). So I’m fine for now. QT5 would be preferred to write code for sale. Commercial license seems to be quite prohibitive for small customer base. But, as said, I will try it, especially if they have well working QTDesigner.

“Plotly can work in an IDE. I am not sure if Dash works outside of a browser.”

No, not to my knowledge. But mix of Plotly and widgets can give you viable alternative in Jupyter, or even better as you have quick access to code at any moment.


Are you a coder, or trader, or a trader who needs to code?

People (traders) often say "I am a trader, not a programmer" or "I am a trader, and I don't want to be a programmer."
Or "I am not good at programming, and I don't have time to learn, because it will take too much time away from my trading." (that was me.)

Those lines are familiar. I used them until I realized that without own research I will start chasing my tale. I do like coding (does Pascal means anything to you :) ), but I also know that to learn it to any decent degree will consume a lot of my (and my family’s) time. I’m not ready for that. Luckily it is not needed for trading. I only want to find satisfactory, no BS, answers to my question, I want to understands things better. So, in a sense, one doesn’t have to become programmer in a strict sense to trade.
This can be matter of semantic but I think you know what I want to say.


However, when the trading support tools that you need to be a better trader are NOT AVAILABLE (...)

I totally agree. One practically can not advance much as a trader this days without good, dedicated software. Software must work in the loop with mindset. Software, alone ofc, is not enough.


My opinion on Excel:

You have much more experience that I do but it seems to be constant motive with Microsoft. “Find people’s needs and quickly fill them” and try to fix problems later. But it is not horrible for trader to start with Excel, better that than nothing and can be quite fruitful.


Into Python

You don’t have to convince me, although I only skimmed on the surface of Py. I use something like 5 packages in total (not exactly true) and have no idea of what is offered out there. I can complain that I have hard time to keep up with changes in those 5. I don't seek out new stuff. For example, I was using Mathplotlib and I was fine with it, eventually (few years I think) I looked into Plotly and I was hooked. I learned to do what I wanted, sometimes on deep levels (for me). I did know about Plotly.express but it was lacking… I couldn’t make it to behave… until I found out that for px I need tidy df :). Now, when I know that, I let GO (graphical objects, basic of Plotly) go. Yet another change... but px is so awesome... The only think I’m still missing is getting data extracted from mouse click on the graph to clipboard…

Coding, when you learn basics, can be fun and can pull you in... I do remember though that I code for trading, not otherwise, primary to learn, secondary to speed up decision process. I’m not a programmer in a professional way. If I can do it, everybody can. It’s worth the effort, if you are serious about trading (but not absolutely necessary, imo). I'm also convinced that working together, or simply in connection with others, can be more beneficial (and bring more fun). This aint easy though...
 
Some really great comments and am enjoying reading this thread. Hopefully I will be able to eventually contribute as much as others have in the future.

I like the sound of the speed and robustness of TDA-API. I also came across a Python wrapper for TDA-API this weekend. Is that what you all use or is it necessary?

My comments about using Excel is in no way to get into VBA programming. In all the years since MS released visual basic, I have never taken the time or interest to learn to program it. The extent of my programming in excel is entering functions in cells to perform math and manipulation on the data for output that I need. I have also used it with various plugins such as a replacement for the Solver function for more robust stochastic modeling. I only reference using excel as a dashboard because I can visually picture all the "cells" arranged with information to help me make decisions and the data in those cells being a result of either my changing a value in another cell (I.e. option strike, underlying price, etc.) or because the real time data, based on my inputs, has changed. I don't have any idea how this might look using Python. I like garyw's definition of data collection and data consumption. I view excel in this context as data consumption, only because I have no other way to describe it not knowing what is available to replace it.

I fully agree that a modular approach is better and I am definitely not looking to build any ultimate tool. Just hoping to assemble individual tools that when combined will help me to trade better and make more money. This is the reason I was trying to break it down to the backend being data collection and the dashboard being data consumption.

I have taken a couple of courses on learning how to program with Python and given some examples, I could probably use it to get where I want to go. This has pretty much always been the way I have programmed, using examples, books, etc. to cobble together what I am trying to get to. Marcas its interesting you mention Pascal because that was one of the first languages I learned when I began working with a database system called Paradox, which it's language was based on Pascal. I used Paradox when I was working as an investment manager to download our client data from their custodian bank and reformat it for our internal uses such as reports and performance measurement calculations.

I think the long and short of it for my needs that may or may not benefit someone else, although I am always willing to share and collaborate since I am not in this to build something to sell; is to find examples of these components that others have created for similar uses. I like the idea of being able to get the option chain data, since my use for all this will be related to trading options.

A couple of questions

How can I access the CD Slack server?

What is GTools?

Thanks
 
@ SG:

What is GTool? : A tool I wrote to aid in the selecting RTT trade entries. Old presentation: "
" <-- The name G-Tool came later.
The reports from the GTool come out twice daily and are available to subscribers of the service.
 
Hopefully I will be able to eventually contribute as much as others have in the future.

I think this is no problem. As long as you want to put your some effort and don't expect others to do work for you, I think it's fine. You may have some interesting ideas to share that can benefit everybody.


I like the sound of the speed and robustness of TDA-API. I also came across a Python wrapper for TDA-API this weekend. Is that what you all use or is it necessary?

No, wrapper is not necessary. There are couple tutorials about how to start with TDA-API out there (can't say about quality), there are also ready solutions to be found. Wrappers are not necessary.
Seems that I will have time tomorrow morning and would be able to show simple example of tdaapi application on tg1.


My comments about using Excel is in no way to get into VBA programming. In all the years since MS released visual basic, I have never taken the time or interest to learn to program it.

As I said, what I can say about Excel is mostly opinion,without much practice. I can do basic stuff and I suppose I can learn more if need arises, which I do nor expect. I also played with Excel's VBA a little but can not provide any advice.
For what you aiming for - dashboard, Excel is fine. I suppose you don't need anything else.
Problems can arise if you start playing with bigger data. It could changed by now, but while ago loading full SPX 1 day option chain from csv file to spreadsheet took quite a while. Jupyter (not python) can access huge csv files in real time and you can do it as fast in py. Not a big deal until you have to do import few times a day... So, Excel is fine, but know it's limitations. Or maybe it would be a good idea to have some other tools handy to improve Excel experience? But those a questions only you can answer.

I have taken a couple of courses on learning how to program with Python and given some examples, I could probably use it to get where I want to go. This has pretty much always been the way I have programmed, using examples, books, etc. to cobble together what I am trying to get to. Marcas its interesting you mention Pascal because that was one of the first languages I learned when I began working with a database system called Paradox, which it's language was based on Pascal. I used Paradox when I was working as an investment manager to download our client data from their custodian bank and reformat it for our internal uses such as reports and performance measurement calculations.

I think you are in very good position to start this game. I expected that you didn't see a single line of py code :)

I also started with Pascal (more or less). I'm missing "?" as print command.. :), but my exposure to coding stopped right there. When I started a new, I knew nothing.
I wouldn't be surprised that Pascal is still in use in banking, their software is really, really old.

Anyway, here is invit to CD Slack. I'm not quite sure if this is _the_ way to go.
Presently I'm, practically, alone there although I suspect that few people still have it on, as a matter of inertia. I'd rather be in place where more people are active. And do not expect any Excel help from me, just basic Py. Beside I'm not much after coding if it doesn't relay to trading. Trading is the first thing, coding comes second for me. (That means considering those two things.)
join.slack.com/t/capitaldiscussions/shared_invite/zt-39q2blap-yi9HXvJxO4pMyGZaYyMt_A
you need an e-mail to join.
 
Last edited:
SG, I will help you to go through, just not today. I have some sort of emergency going on. I suggest to remove your e-mail from public view. Even if it is dummy mail you don't want it to be flooded with spam. Scraper bots are vigilant. I will pm you hopefully tomorrow. If you manage to get to CD Slack just say hi and I know you are there. in general or programming channel.
 
Top
Contact Us