Fail to run the IEX Downloader Parser
I clone the repo provided by our professor and tried to run through the code. I could not run through the vm_go.sh. file, saying I don't have a default VM provider. Then, I skipped the VM part and executed the download.sh and parse_all.sh file separately. The download.sh file worked normally. However, the data it downloads was huge. Data on 2022-08-01 is about 4 GB in compressed format.
The following command in .sh file generated an error
iexdownloaderparser % ./parse_all.sh
PCAP_FILE=data/iex_downloads/DEEP/data_feeds_20220801_20220801_IEXTP1_DEEP1.0.pcap.gz PCAP_DATE=20220801
tcpdump: unknown file format
Documents/Courses/UIUC/group_05_project/iexdownloaderparser/src/parse_iex_pcap.py:608: SyntaxWarning: "is" with a literal. Did you mean "=="?
if stocks_to_output is "ALL":
interested in SPY
Starting parsing @ 2023-10-24 17:25:23.460082
Traceback (most recent call last):
File "Documents/Courses/UIUC/group_05_project/iexdownloaderparser/src/parse_iex_pcap.py", line 646, in <module>
parser.parse(max_packets_to_parse)
File "Documents/Courses/UIUC/group_05_project/iexdownloaderparser/src/parse_iex_pcap.py", line 149, in parse
(magic_number, version_major, version_minor, this_zone, sigfigs, snaplen, network) = struct.unpack("IHHiIII", byte)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 24 bytes
It seems either tcpdump or parse_iex_pcap.py was wrong. Then, I trouble shooted by go through this process step by step. It turns out gunzip and tcpdump worked normally. However, I cannot finish the tcpdump because the file is too large.