|
Class: 11-12-07 New Ensign Training |
| Playback | Data Base | Obsolete Tick Pool | Refresh Speed |
| Auto Refresh | Study Grids | Price Histogram |
As many of you now know we released the version that had been in beta testing for the past 3 months. We appreciate all those who participated in the testing and that group was up to 178 users at the time of the release. Many had been beta testers from back in August, and they are the real troopers that helped us find and fix many things prior to this official release. Over 325 more use got the upgrade Saturday and Sunday and used the new Ensign for the first time today, so as you can imagine, we had lots of questions about new features, and changes in the way things are done, etc.
Hopefully the day went well for the majority of them, and those needing help were serviced in a prompt and professional manner. Some may have elected to retrograde to the last official release version without the new features, which can be done using a link on the Download page of our web site.
Let me cover some of the topics most commonly asked about.
Probably the top inquiry is about the change to the Playback feature in the new Ensign. There are two web pages with material to read and I will post those links now.
The first link is to class material about playback from a month ago, and the 2nd link are the Playback instructions which have been updated to show the new screens for the new Ensign. The basic change is that the new Ensign will play back the actual tick files that are now a foundation in our database redesign. The old Ensign would playback special playback files that had to be build from the tick pool or from a data vendor refresh. The new Ensign has eliminated that step of building special playback files, and plays the tick information directly back.
Yes, this change does obsolete the library of old playback files you may have collected in the \Playback folder, but the advantages will be accepted and appreciated once you get used to the change. Also, an advantage is you can readily play back any symbol you have been following because there will exist tick database files that you collected or refreshed. You do not have an extra step to take of downloading a playback file or building one.
In the past Ensign has maintained some playback files on our web site for 5 symbols, and we will continue to do so for another month or so for the benefit of those on the old Ensign. In parallel, we will post some tick database files for those who might want older files, or would have difficulty in getting a tick file from a Refresh. This latter step should be simple enough that the need for Ensign to have tick files on the web site would be diminished.
As shown in the playback documentation you enter a symbol and can either use the calendar or the Browse button to pick the date to playback. You do need the tick file in the database for the date you want to playback. If you have files for earlier dates, then the chart will be loaded with prebuilt bars for the days ahead of the play date. You will notice a huge speed improvement in how long it takes to open the chart with the prebuilt bars. It is super fast now. And the DEMO symbol chart will look identical to the actual chart for the bars.
jg: will it be possible to write an ESPL script to poke old playback files into the tick database in order for them to be available for playback ?
Good question and it is already on my list of things to consider as a built in function, like a conversion. There is great value in the libraries of playback files you have, and it would be a pretty simple step for me to read a playback and write a tick file. So let me do it as time and priorities permit. Now if I can just live on less sleep I can get more done....lol. But I like the idea and it will have a benefit in this regard.
In the past there has been an occasion inquiry if a playback file would build some bars to heal a chart or provide more back data. Well when I make the playback to tick conversion feature, then the answer would be yes. You get the data into the tick and minute files, and then that resource can be used to build bars for more back history. It has value and should be a feature I add to the database manager form, to get data by importing a playback file.
Any questions about playback that are still in your mind?
sportsrep3: will the playback on the old version continue to work?
Yes playback in old Ensign will continue to work and use playback files. However, there is an issue of mixing old and new Ensign doing same playback in a chat room setting. The two do not stay in time sync. The new Ensign is superb at playing back at the intended pace, and old Ensign slips in keeping on pace in fast markets. Old Ensign experiences time creep until old and new are not in sync with each other. Sorry about that, but over the next month or so most users will migrate to new Ensign and that issue will diminish.
In the new Ensign the equivalent archive of files for playback is now the tick database. There is a tick file for each day, organized in folders for each symbol. The new version does not use anything in the old C:\Ensign\Playback folder, unless I add the conversion feature to make a playback file into a tick file.
dougfir: I have ASCII tick data going back 4 years, can I import/convert that to a database readable form?topset: can the tickfile be exported as well as imported?
zbullzbear: who has 'database' we or Ensign?
A tick file can be opened as a chart and the chart can export. I do not yet have features to export or import directly to the database files, but intend for such features to be added. It just takes time to continue to build on the foundation that is in place at the present time. Also, I want to add tools for editing directly in the database files, possibly use an edit on the chart to make the same edit in the database. Right now the solution is to try to refresh the database file and clean up a spike by getting a better bar from the vendor refresh.
taku: did I read that it is wise to set # of bars to 65000
No taku, you did not see any suggestion like that. In fact, I would suggest the opposite. The database files are independent of the chart files. Both exist in parallel. A chart has a Max Bars setting and that is used to retain X number of recent bars. As new bars are added to the chart the oldest are dropped. The database, is always accumulating and never dropping. So data you collect in August can still be there come next July. 2000 is a better number to use for chart Max Bars.
If and when you happen to need more bars in the chart for some purpose, you can at that time change Max Bars to be a higher value, position chart so first bar in file is in the chart view which is easily done with Shift = keys, and then use the pop-up menu for Rebuild Visible, and the chart will then go add more bars to the left of the chart that it extracts from the database files. This can happen very quickly. So the database file is a great resource for having back data, and not depend on a refresh to try to replace it.
Now on a related thread of conversation, lets say you missed some tick data a few days back. In the database you go to the day in question and refresh that one day with ticks or minute bars. And the refresh of the specific day then rebuilds the open chart to include the new data. I helped a user today with this very need, and made a web page out of the process so it is now training material for the next person. http://ensign.editme.com/q1774
See this example where today, the 12th, the users wanted to get ticks for the 8th. This little example gives some familiarity of using the database manager form, which I most often get to from the chart's pop-up menu but it is also available from menu File | File Manager | Database Manager.
zbullzbear: who maintain the folders?
You can do nothing and let Ensign make the folders and add files from the ticks for the symbols you have on your Setup | Manager list. Or you can spend some time seeing what the database manager is doing. Files and folders can be deleted. Files can be refreshed if they lack completeness. Refresh can be requested for dates not yet in the database.
wout: fine! what symbols are provided?
Te symbol universe depends on your data provider, and if you have no provider use the Ensign Internet selection as your source and then all data comes from the DTN Market Access source. The DTN source has all US futures, stocks and indexes. Foreign stocks for daily is available from other Internet sources.
Let me post a link showing the refresh that is available. http://ensign.editme.com/eoddata You can download ticks for any of the exchanges listed in the table in the Tick column that show a yes, which is US stocks, Indexes, futures, Canadian stocks and futures, IPE, LIFFE, Eurex, etc. And we can do the same with the Forex currencies for our FXCM feed which is included.
The database concept is really less of a mystery than you realize. In the old Ensign we had a tick pool which was a collection of files holding all ticks as sequentially received for all symbols followed. This collection was in a folder called \AllTicks. Its design was a linked thread of ticks with the entry point to a symbol's thread in the quote table.
If the computer crashed form a power outage, then the important pointers into the table in the quote record were not saved on the hard disk and the current pool would then be useless or corrupt. In the new design the database has replaced the tick pool. Here we distribute the ticks to their owner's folder by symbol, and no longer have any linked thread of ticks.
The advantage are that we can find all ticks need to bring a chart current much quicker, and if the computer crashes, nothing is obsolete because it was already saved in the right place and not subject to pointer to a linked list. So we do a bit more work to save the ticks but pick up the benefit on the other side that there is less work to read them.
The intra-day charts all read or are brought current from a minute database. Lets use a 60-m chart as an example. You open the chart, it reads the 60-min chart file, then sees where it was last updated, and consults the minute database to bring the chart current. Perhaps the chart is through 2 days ago, so the the minute files for the last 3 days would be opened, 60-min bars built from the 1-min records, and then the updated chart file is resaved so we can eliminate the rebuild of these bars the next time the chart is opened. This process happen faster than ever before, and the speed improvement is noticeable and appreciated.
Shar: Howard, do we get fills for Daily, Wkly, Monthly from Internet Service? And what larger minute charts? We don't have to do those day by day like the tick charts, do we?
The daily, weekly and monthly charts operate the same as before in that the chart file for these IS the database. What you have in the daily, weekly and monthly folder is what you have.
This brings us to another topic of a common question. Some say the refresh is slower on the new Ensign. Let me discuss so you see what is happening, and I will again use the 60-min chart as an example. In the old Ensign many of you would open a 60-min chart, see it needed data a couple days back and you would use Alt-7 to do a refresh Maximum request.
About 4 seconds later you would have the chart repaint with the data, and you gave it no more thought. Well, doing the same thing now seems slower for these reasons. First, because we have a 1-min database we want to benefit. Any intra-day chart refresh is going to ask for 1-min bars though 60-min bars are wanted. Thus the reply will be 60 times bigger to ask for X days of 1-min bars than X days of 60-min bars.
When the refresh is received, we update the 1-min database and then update the chart with bars build from the 1-min refresh. Also, Alt-7 for Max refresh gets more from DTN Market Access than ever before. Before max would be 4 months and now it is 8 months.
But consider this, ONCE you have done a max refresh, you should not do it again. You already have the data in the database and continuing to refresh maximum adds no benefit to you. That is like ripping out the driveway and repouring concrete each time you want to take the car out for a spin. We wise.... only refresh data you don't have.
Look at the database content. We show a list of dates in the database, and for any given date, you can click on the file and see a row of boxes indicating the presence of data in each half hour period. It gives a good idea if the file content is complete. If it is complete leave it alone.
So today, being your first day, a max refresh or a new chart might have taken longer to refresh because of the detail being obtained. From now on, 1 day refreshes should suffice, or 2 day refreshes will suffice. Use the bigger refresh requests as needed.
Now I will give training on the topic of automatic refreshes. The objective is to make Ensign smarter and the user needing to give less thought to the chart content. One of the things implemented to that end was to have Ensign do refreshing before you need the chart. Ensign does it in advance in anticipation of your wanting it.
We have two new check box options on the Setup | Charts form for starting an Auto Refresh of the minute and/or the tick databases when the program connects.
Let me use Buffy and her eSignal connection as an example. Say she is on-line until 9 pm at night, and then exits Ensign and turns it back on at 7 am the next day. Ensign's databases know they have data through 9 pm, so upon connecting if the auto refresh options are enabled, the program will start going through the symbol list on the Setup | Manager form and request minute data from 9 pm through 7 am, and then when that process is finished, it will request tick data for the same time period.
The minute data base is refreshed first because that completes most quickly. Buffy might start to open some charts, and they may or may not be refreshed, yet, but an unrefreshed chart is detected, and its refresh request is then moved to the top of the queue. Eventually all pending requests on the queue are serviced, and hopefully Buffy never sees the rest of the day a single refresh request because the databases are current already. So, Buffy, how has that felt to you.... good karma??
@Buffy2: I see this all the time Howard as I go back to old version to get a template or something I forgot to bring into the clean beta install I opted to do. It is a great feature for sure. I just wait until top left says done after opening workspace. At some point I go to dbm and refresh days not locked which locks them making them read only and pick up the ticks I may of missed due to whatever
Now the auto refresh is a good intention, but unfortunately it is tainted by which data vendor you use. And the taint is for the IB users, whose refresh from IB is plagued by 3 issues.
1st issue is that any refresh request from IB is really slow to get a reply.
2nd issue is that the have a cap of 2000 record returned on any reply so we have to break up a refresh for a given period into multiple requests. All that takes longer to get the total reply.
3rd hindrance for IB refresh is they have imposed a pacing restriction of 60- requests in a 10-min period. and since a period refresh might have been broken into multiple requests, the 60- limit gets used up pretty fast. So I do NOT encourage the auto refresh options for program run to be checked for IB users. Otherwise the refresh resource can be bottlenecked for a couple hours waiting for the 10 min limit period to expire.
taku: no limits on refresh from DTN , correct?
DTN does not have any such limits. So if you can, IB users should do their refreshing from the DTN source.
The last topic for training is that of study grids. Please read the material about the powerful new flexible grids in the new design. http://ensign.editme.com/c070815#grids
Each study has a new grid selection on its property form to replace the check box list. The new grid object will show on the chart property form. If you see a sub-window such as #1 lacking the grid you like, go to the chart object list and see the listing of Grid for Window 1. Select it and configure it with the lines, styles, and line values you seek. It is more flexible than ever before. You can have different line styles and colors for each grid line now. And if you will scroll down in the documentation there are examples of colorful fills to give some cool effect for the grid.
This new feature may need a bit of configuration for defaults you like because the defaults that show might not be what you expect. But I am showing where you can do some fixing to get it the way you like it. If the grid setting is what you want for the default, then check the Use as Default box before closing the grid form. The Grid button found on the study property will take you directly the Grid object property form to bypass getting the using the chart object list. Either way is ending up at the same place which is the object's property form.
Dogfoot: hi, I just signed up for Ensign , looking forward to MP tpo's and related functions. Can I hold my breath? Trading friend of odds_on and FuturesTrader71.
Dogfoot, the MP tool in Ensign is the price histogram and considerable improvement was made for the new Ensign. Here is a page of documentation from an earlier class. http://ensign.editme.com/c070829#histogram
More improvements have been suggested and I agree they are worthy of adding as time and priorities permit. I doubt any price histogram changes will be made until after Thanksgiving as the coming week looks booked and then I will be away for extra time for Thanksgiving next week. PH is a high priority on top of things to work on in Dec.
Have a good evening and thanks for your patience on this unusual day when so many made the switch to the new Ensign all at once.
al_gorithm: Very nice work Howard.
Disciplined: The new Ensign version is really very fast and very smooth Howard. Thanks for this big improvement
claude: I agree. With IB it might take a little more work but it worked great for me today
drh: take a break ...great job ... I am wore out trying to keep current....:)....i started out with Howard at 6:00 am this morning...great job ...my friend
wout: well Howard, I am off, has been very nice to have your class here and hope to meet you again. You have done a wonderful work!!!! THANKS
@Buffy2: thank you so much Howard for the extra support that was needed also