Save and restore arrays

[dragancesu]

Saving options should be in every program

A long time ago i started to create a program to help me with that, and it waited until this topic came up

Allows you to store options such as Checkbox, ComboBox, Radio group and Text field

Program serves to enter the parameters to be stored, then generates the program to be included in the
application (options.prg and _options.dbf)

in attachment is source and exe

optdemo.zip is demo data, generate program look like excels options (copy and delete indexes)

[RPC]

Hi Dragancesu
Thanks for this program. I will definitely try it.

[edk]

Hi Edward
You are right. I am getting the same message for 17 dec 2015. But I can download for 19 feb 2015
Further when I download from the website at
https://www1.nseindia.com/products/cont ... eve_eq.htm
with Select Report as "Bhavcopy"
Date 17-12-2015
it downloads(in two formats csv and dbf) and provides links, when I hover over the link I can see the download link as
https://www1.nseindia.com/content/histo ... av.csv.zip
when I click the link the file downloads but again when I copy paste the link in browser I get the same message of Access denied.

In fact I can download this way from 3 nov 1994 on wards.

Please check
Thanks

Hi.
Strange thing indeed. I did various tests and it follows that if there are at least two links it does not work properly. Even if the search results are displayed, typing in the same window in the browser address of the copied link results in a 403 error. Only using the link by clicking opens the target in the new window, which is the .zip file, and then the server allows it to be downloaded.
I analyzed headers, cookies and sessions and I can't find how the link from the new window is "authorizing".
I was looking for different methods but all return a 403 error.

In the end I did the automation of Internet Explorer, but this requires confirmation of downloading the file. I don't know how to automate this process.
Maybe it will help a bit:

Code: Select all

#include "HMG.CH"

FUNCTION main()

SET DATE BRIT
SET CENTURY ON

DEFINE WINDOW testunzip MAIN AT 0,0 WIDTH 200 HEIGHT 200
       DEFINE DATEPICKER datepicker_1
	           ROW 20
			   COL 20
			   WIDTH 100
			   HEIGHT 20
			   ONLOSTFOCUS getDay()
	   END DATEPICKER

       DEFINE BUTTON Button_1
          ROW 100
          COL 20
          WIDTH 100
          HEIGHT 30
          CAPTION "Download" 
		  ACTION Download()
      END BUTTON	

      DEFINE LABEL Lbl_1
         ROW 45
         COL 20
         WIDTH 150
         HEIGHT 20
		 VALUE ""
      END LABEL		 
	  
	  
END WINDOW
testunzip.center
testunzip.activate
RETURN nil	  


FUNCTION Download()
LOCAL cBhavFile, cFileNm, cUrl, cFile, dDate
      dDate := testunzip.datepicker_1.value
      cBhavFile := 'cm' + ntoc( DAY( dDate ), 10, 2, '0' ) + UPPER( LEFT( CMONTH( dDate ), 3 ) ) + ntoc( YEAR( dDate ), 10, 4, '0' ) + 'bhav.csv'
      cFileNm := cBhavFile + '.zip'
      *cUrl := 'https://www1.nseindia.com/content/historical/EQUITIES/' + ntoc( YEAR( dDate ), 10, 4, 0 ) + '/' + UPPER( LEFT( CMONTH( dDate ), 3 ) ) + '/' + cFileNm
      cUrl := "https://www1.nseindia.com/ArchieveSearch?h_filetype=eqbhav&date=" + hb_DtoC( dDate , "dd-mm-YYYY" ) + "&section=EQ"

      cFile := getcurrentfolder() + '\' + cFileNm
      lGotFile := downloadfile( cUrl, cFile, dDate, "NSE" )
      IF !lGotFile
         msginfo( {"NSE Bhavcopy file not found for ", dDate })
         
      ELSE
         IF FILE( cFile )
         
		 UNCOMPRESS cFile
		 msgdebug("File uncompressed")
         
		 ENDIF
     ENDIF
RETURN nil

FUNCTION GetDay()
testUnZip.Lbl_1.value := cdow(testUnZip.datepicker_1.value)
RETURN nil

FUNCTION DownloadFile( cUrl, cFile, dDate, cExchange )

LOCAL aRetorno, nHandle, nCont, lOk, oIE
LOCAL link, Alllinks

   lOk := .f.
   
   BEGIN SEQUENCE WITH {|o| break(o)}
	oIE := Win_OleCreateObject( "InternetExplorer.Application" )
	oIE:Visible := .F.		//<= set .T. to show IE
	oIE:Navigate( cURL )
 
	Do While oIE:readyState() <> 4
		Do Events
	EndDo
	
	Do While oIE:Busy()
		DO Events
	enddo

	AllLinks:=oIE:Document:getElementsByTagName('A')

	FOR EACH link IN AllLinks

		*msgdebug( link:href )
		*msgdebug( link:innertext )

		IF link:innertext == hb_FNameNameExt (cFile)
			link:Click(0)		//emulate click on href
		ENDIF
		
	NEXT 
	
     oIE:Quit()

     
     RECOVER USING oErr

	    	IF VALTYPE(oIE:Document)='O'
	    
	    		Msgbox ( oIE:Document:documentElement:outerHTML() )
	    		oIE:Quit()
	    		
	    	ENDIF
     
END SEQUENCE

	oIE := Nil
 
	/*
   IF EMPTY( aRetorno ) .OR. '!DOCTYPE' $ aRetorno
      //msgstop( "file not found on the server" )
      //BREAK
      RETURN lOk
   ENDIF
 
   nHandle := FCREATE( cFile )
   IF VALTYPE( aRetorno ) == "C"
      FWRITE( nHandle, aRetorno )
   ELSE
      FOR nCont = 1 TO LEN( aRetorno )
         FWRITE( nHandle, CHR( aRetorno[ nCont ] ) )
      NEXT
   ENDIF
   FCLOSE( nHandle )

   lOk := FILE( cFile )
   */

RETURN lOk

[RPC]

Hi Edward
You have really taken a lot of effort in helping me. I am really grateful to you for that.
I tried your program. It gives message "NSE Bhavcopy file not found for..." . This is happening because return value from Downloadfile function is lOk which never gets .t. value. Being very dumb in understanding your program I do not know where to turn it to .t. Please advise.
I tried oIE:Visible := .t. The IE is displayed only for fraction of second I can't see what it displays.
I tried the url you have used in program. It is perfectly working right from 3 nov 1994. But how to download the zip file through the program ? I couldn't ascertain if link:Click(0) is working.
Many thanks for your help

[edk]

I think I know what the problem is. You need to allow IE to pop up windows.
Set oIE:visible := .T.
Try to pause after the link:Click(0), e.g. Msgbox(), you should be asked whether to allow pop-ups, allow allways.

allow.png (27.67 KiB) Viewed 2165 times

Afrer next run, then a window should appear asking you what to do with the file.

save.png (11.91 KiB) Viewed 2165 times

And here you have to manually indicate the place of saving. I don't know how to automate this stage, that's why I don't return the .T. value, because I don't know if the user saved the file. Maybe you could listen to the folder if the file appeared in the desired location?

[RPC]

Hi Edward
Ok I got it. Since I want to download the file through the program and it could be many files at a time, clicking for download every time is not feasible. After downloading the file it needs to be processed further. At my computer I would know the download path but at a different computer the path may be different. So this may not help much.

Do you think there may be API for this. There is a program at https://github.com/hemenkapadia/getbhav ... 3.0.2a.zip
which is free of cost(Donationware) and a discussion forum at
https://github.com/hemenkapadia/getbhavcopy/issues and source code at
https://github.com/hemenkapadia/getbhav ... 3.0.2a.zip
https://github.com/hemenkapadia/getbhav ... .2a.tar.gz
which downloads the files from www,nseindia.com site. Since the program is free of cost I presume the person is using a free API to download these files. Is it possible to find out the API from the source code ?
Can you please look into this.
I know you have spent a lot of time on this, but please see if you can help further.
Many thanks.

[RPC]

Hi Edward
In case you want to try Getbhavcopy following are the options settings

[edk]

It seems to me that this application also has a problem downloading NSE Equity Bhavcopy file for December 17, 2015.
I tried to download files from both servers:

Code: Select all

Message	*********  Starting Getbhavcopy  *********
Message	Settings Saved
Message	*********  Downloading Data for 17-12-2015  *********
Message	Downloading NSE Equity Bhavcopy
Error	Error downloading NSE Equity Bhavcopy file
Error	Serwer zdalny zwrócił błąd: (404) Nie znaleziono.
Message	Downloading index data for NSENIFTY
Message	Downloading index data for NIFTYJUNIOR
Message	Downloading index data for NIFTYMIDCAPLIQ15
Message	Downloading index data for NSE100
Message	Downloading index data for NIFTY200
Message	Downloading index data for NSE500
Message	Downloading index data for NSEMIDCAP150
Message	Downloading index data for MIDCAP50
Message	Downloading index data for NSEMIDCAP
Message	Downloading index data for NIFTYSMALLCAP250
Message	Downloading index data for NIFTYSMALLCAP50
Message	Downloading index data for NSESMLCAP100
Error	NSE server did not return data for index NSESMLCAP100
Message	Downloading index data for NIFTYLARGEMIDCAP250
Message	Downloading index data for NIFTYMIDSMALLCAP400
Message	Downloading index data for NIFTYAUTO
Message	Downloading index data for BANKNIFTY
Message	Downloading index data for NIFTYFINSERVICE
Message	Downloading index data for NIFTYFMGC
Message	Downloading index data for NSEIT
Message	Downloading index data for NIFTYMEDIA
Message	Downloading index data for NIFTYMETAL
Message	Downloading index data for NIFTYPHARMA
Message	Downloading index data for NIFTYPVTBANK
Message	Downloading index data for NIFTYPSUBANK
Message	Downloading index data for NIFTYREALTY
Message	Downloading index data for NIFTYCOMMODITIES
Message	Downloading index data for NIFTYCONSUMPTION
Message	Downloading index data for NIFTYCPSE
Message	Downloading index data for NIFTYENERGY
Message	Downloading index data for NIFTY100ESG
Message	Downloading index data for NIFTY100ENHESG
Error	NSE server did not return data for index NIFTY100ENHESG
Message	Downloading index data for NIFTYINFRA
Message	Downloading index data for NIFTYMNC
Message	Downloading index data for NIFTYPSE
Message	Downloading index data for NIFTYSMEEMERGE
Error	NSE server did not return data for index NIFTYSMEEMERGE
Message	Downloading index data for NIFTYSERVSECTOR
Message	Downloading index data for NIFTYSHARIAH25
Message	Downloading index data for NIFTY50SHARIAH
Message	Downloading index data for NIFTY500SHARIAH
Message	Downloading index data for NIFTYABGROUP
Message	Downloading index data for NIFTYMAHINDRA
Message	Downloading index data for NIFTYTATA
Message	Downloading index data for NIFTYTATA25CAP
Message	Downloading index data for NIFTYABGROUP
Message	Downloading index data for NIFTYLIQ15
Message	Downloading index data for NIFTY500VALUE50
Message	Downloading index data for NIFTYALPHALOWVOL30
Message	Downloading index data for NIFTYQUALLOWVOL30
Message	Downloading index data for NIFTYALPHAQUALLOWVOL30
Message	Downloading index data for NIFTYALPHAQUALVALLOWVOL30
Message	Downloading index data for NIFTY50EQUALWEIGHT
Message	Downloading index data for NIFTY100EQUALWEIGHT
Message	Downloading index data for NIFTY100LOWVOL30
Message	Downloading index data for NSEDEFTY
Message	Downloading index data for NIFTY50DIVPOINT
Message	Downloading index data for NIFTYDIVOPPS50
Message	Downloading index data for NIFTYALPHA50
Message	Downloading index data for NIFTY50ARBITRAGE
Message	Downloading index data for NIFTY50FUTINDEX
Message	Downloading index data for NIFTY50FUTTRINDEX
Message	Downloading index data for NIFTYHIGHBETA50
Message	Downloading index data for NIFTYLOWVOL50
Message	Downloading index data for NIFTY200QUALITY30
Message	Downloading index data for NIFTY50VALUE20
Message	Downloading index data for NIFTYGROWSECT15
Message	Downloading index data for NIFTY50TR2XLEV
Message	Downloading index data for NIFTY50PR2XLEV
Message	Downloading index data for NIFTY50TR1XINV
Message	Downloading index data for NIFTY50PR1XINV
Message	Downloading index data for NIFTY-GS-COMPSITE
Message	Downloading index data for NIFTY-GS-4-8YR
Message	Downloading index data for NIFTY-GS-8-13YR
Message	Downloading index data for NIFTY-GS-10YR
Message	Downloading index data for NIFTY-GS-10YR-CLN
Message	Downloading index data for NIFTY-GS-11-15YR
Message	Downloading index data for NIFTY-GS-15YRPLUS
Message	Downloading index data for VIX
Message	Processing downloaded data
Error	NSE Processor cannot open temporary bhavcopy file
Error	Nie można odnaleźć pliku 'C:\Dwn\Getbhavcopy_3.0.2a\temp\NSEbhav.gbc'.
Message	********* Data download complete *********

Code: Select all

Message	*********  Starting Getbhavcopy  *********
Message	*********  Downloading Data for 17-12-2015  *********
Message	Downloading NSE Equity Bhavcopy
Error	Error downloading NSE Equity Bhavcopy file
Error	Serwer zdalny zwrócił błąd: (403) Zabronione.
Message	Downloading index data for NSENIFTY
Error	Error downloading Index Data for NSENIFTY
Error	Połączenie podstawowe zostało zakończone: Wystąpił nieoczekiwany błąd przy odbiorze.
Message	Downloading index data for NIFTYJUNIOR
Error	Error downloading Index Data for NIFTYJUNIOR
Error	Połączenie podstawowe zostało zakończone: Wystąpił nieoczekiwany błąd przy odbiorze.
Message	Downloading index data for NIFTYMIDCAPLIQ15

Ask nseindia.com if they do have an official Web API?

[RPC]

Hi Edward
I could download for 17 dec 2015

Code: Select all

Message	*********  Starting Getbhavcopy  *********
Message	*********  Downloading Data for 17-12-2015  *********
Message	Downloading NSE Equity Bhavcopy
Message	Downloading index data for NSENIFTY
Message	Downloading index data for NIFTYJUNIOR
Message	Downloading index data for NIFTYMIDCAPLIQ15
Message	Downloading index data for NSE100
Message	Downloading index data for NIFTY200
Message	Downloading index data for NSE500
Message	Downloading index data for NSEMIDCAP150
Message	Downloading index data for MIDCAP50
Message	Downloading index data for NSEMIDCAP
Message	Downloading index data for NIFTYSMALLCAP250
Message	Downloading index data for NIFTYSMALLCAP50
Message	Downloading index data for NSESMLCAP100
Error	NSE server did not return data for index NSESMLCAP100
Message	Downloading index data for NIFTYLARGEMIDCAP250
Message	Downloading index data for NIFTYMIDSMALLCAP400
Message	Downloading index data for NIFTYAUTO
Message	Downloading index data for BANKNIFTY
Message	Downloading index data for NIFTYFINSERVICE

I have observed Bhavcopy generally downloads without problem but index data sometimes has problem downloading as it has happened with 'NSEAMLCAP100' in above download.
The second server is not working so even I cannot download anything from there.
I will mail nseindia.com for official web API , but I doubt if they will reply.
Is there a way we can find API ?
Thanks a lot for all the help

[edk]

I am sorry, but in this application there is still a problem with downloading files from December 2015 on my computer, could it be possible only from the IP address of India?
Program sources are incomprehensible to me, they look like some html templates for doc. Forgive me, but I can't help you with these sources.